Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
Add architecture improvements and comprehensive documentation
swati510f627e7010a0e
diff --git a/docs/architecture-guide.md b/docs/architecture-guide.mdnew file mode 100644index 000000000..e17dea3de--- /dev/null+++ b/docs/architecture-guide.md@@ -0,0 +1,1179 @@+# Repowise Architecture — Complete Guide++This document covers the entire Repowise architecture beyond the graph layer: ingestion, generation, persistence, providers, server, and MCP tools. For graph algorithms specifically, see `graph-algorithms-guide.md`.++---++## Table of Contents++1. [The Big Picture](#1-the-big-picture)+2. [Ingestion Pipeline](#2-ingestion-pipeline)+3. [Generation Pipeline](#3-generation-pipeline)+4. [Persistence Layer (Three Stores)](#4-persistence-layer-three-stores)+5. [LLM Provider Abstraction](#5-llm-provider-abstraction)+6. [Rate Limiter](#6-rate-limiter)+7. [Server Layer](#7-server-layer)+8. [MCP Tools](#8-mcp-tools)+9. [Frontend (Web UI)](#9-frontend-web-ui)++---++## 1. The Big Picture++Repowise takes a codebase and produces a living wiki — AI-generated documentation that stays current as the code changes.++```+Your Codebase+ │+ ▼+┌─────────────────────────────────────────────┐+│ INGESTION │+│ FileTraverser → ASTParser → GraphBuilder │+│ GitIndexer → ChangeDetector │+│ DeadCodeAnalyzer │+└──────────────────┬──────────────────────────┘+ │ ParsedFiles, Graph, GitMetadata+ ▼+┌─────────────────────────────────────────────┐+│ GENERATION │+│ ContextAssembler → Jinja2 Templates │+│ PageGenerator → LLM Provider │+│ JobSystem (checkpoints, resumability) │+└──────────────────┬──────────────────────────┘+ │ GeneratedPages+ ▼+┌─────────────────────────────────────────────┐+│ PERSISTENCE │+│ SQL Store ─── Full-Text Search (FTS5) │+│ Vector Store (LanceDB / pgvector) │+│ Graph Store (NetworkX → SQL tables) │+└──────────────────┬──────────────────────────┘+ │+ ┌──────────┼──────────┐+ ▼ ▼ ▼+ CLI Server MCP Server+ (repowise) (FastAPI) (for AI editors)+ │+ ▼+ Web UI+ (Next.js 15)+```++**Why this architecture?** Each layer has a single responsibility and a clear boundary:++- **Ingestion** knows about code but nothing about LLMs or databases+- **Generation** knows about LLMs but not about HTTP or the web UI+- **Persistence** knows about storage but not about how data was created+- **Server** knows about HTTP but delegates all logic to core++This means you can swap the LLM provider, change the database, or replace the frontend without touching the core logic.++---++## 2. Ingestion Pipeline++The ingestion pipeline turns raw source code into structured data. It has five components that run in sequence.++### 2.1 FileTraverser — "What files exist?"++**The problem:** A repository contains thousands of files, but most are irrelevant — `node_modules/`, compiled binaries, lock files, generated code. You need to find the files worth documenting.++**How it works:**++FileTraverser walks the directory tree and applies three filter layers:++```+All files in repo+ │+ ▼+ Layer 1: Blocked directories+ (.git, node_modules, __pycache__, .venv, dist, build, ...)+ │+ ▼+ Layer 2: .gitignore + .repowiseIgnore+ (using pathspec library for glob matching)+ │+ ▼+ Layer 3: File-level filters+ - Blocked extensions (.pyc, .so, .exe, .wasm)+ - Blocked patterns (*.min.js, package-lock.json)+ - Generated file detection (checks first 512 bytes for markers like "DO NOT EDIT")+ - Size limit (default 500 KB)+ │+ ▼+ Surviving files → FileInfo objects+```++**Language detection** uses a multi-step fallback:++1. Check special filenames first (`Dockerfile` → dockerfile, `Makefile` → makefile)+2. Check extension map (`.py` → python, `.tsx` → typescript, etc.)+3. If extension unknown: check for null bytes (binary → skip)+4. If text: read first 200 chars for shebang (`#!/usr/bin/env python` → python)++**File classification** adds boolean flags to each FileInfo:++| Flag | How it's detected | Example |+|------|------------------|---------|+| `is_test` | `test_` prefix, `_test` suffix, or inside `/tests/` directory | `test_auth.py`, `auth.spec.ts` |+| `is_entry_point` | Filename in known set: `main.py`, `index.ts`, `app.py`, `server.go` | `src/main.py` |+| `is_config` | Language is yaml, toml, json, dockerfile, or makefile | `docker-compose.yml` |+| `is_api_contract` | Language is proto/graphql, or name contains "openapi"/"swagger" | `api.proto`, `openapi.yaml` |++**Why this matters:** These flags drive downstream decisions. Entry points are never flagged as dead code. Test files don't get full wiki pages. Config files skip AST parsing.++### 2.2 ASTParser — "What's inside each file?"++**The problem:** You need to extract functions, classes, imports, and exports from source code in 9+ languages without writing a parser per language.++**The solution: tree-sitter queries.**++Tree-sitter is a parser generator that produces concrete syntax trees for any language. Repowise uses it with a **unified query architecture**:++```+Source code (Python) tree-sitter grammar S-expression query (.scm)+────────────────── + ───────────────────── + ──────────────────────────+def calculate(x): python grammar (function_definition+ return x * 2 name: (identifier) @symbol.name+ parameters: (parameters) @symbol.params+ ) @symbol.def+```++Every language uses the same capture names:+- `@symbol.def` — the full definition node (provides line numbers)+- `@symbol.name` — the name identifier+- `@symbol.params` — parameter list+- `@symbol.modifiers` — decorators, visibility keywords+- `@import.statement` — full import node+- `@import.module` — the module path being imported++**Adding a new language** requires only two things:+1. Write one `.scm` query file in `packages/core/queries/`+2. Add one entry to the `LANGUAGE_CONFIGS` dict++No new Python classes, no language-specific if/elif chains.++**What the parser extracts:**++For each file, the parser produces a `ParsedFile` containing:++```python+ParsedFile+├── symbols: [Symbol] # functions, classes, methods, interfaces, enums+│ ├── name: "calculate"+│ ├── kind: "function"+│ ├── signature: "def calculate(x: int) -> int"+│ ├── visibility: "public" # language-specific rules+│ ├── start_line, end_line # for source extraction+│ ├── docstring: "..."+│ ├── decorators: ["@cache"]+│ ├── is_async: True/False+│ └── parent_name: "Calculator" # if it's a method+├── imports: [Import]+│ ├── module_path: "utils.helpers"+│ ├── imported_names: ["calculate", "Config"]+│ └── is_relative: True/False+├── exports: ["calculate", "Config"]+└── content_hash: SHA-256 of raw bytes+```++**Visibility detection** is language-specific because every language has different conventions:++| Language | Rule | Example |+|----------|------|---------|+| Python | Underscore prefix → private, dunder → public | `_helper` is private, `__init__` is public |+| Go | Uppercase first letter → public | `Calculate` is public, `calculate` is private |+| TypeScript | `private`/`protected` keyword in modifiers | `private _cache` is private |+| Rust | `pub` keyword in modifiers | `pub fn new()` is public |+| Java | `private`/`protected`/`public` keyword | `public void run()` is public |+| C/C++ | All public by default | Everything is public |++**Parent detection** (for methods) also varies:++| Strategy | Languages | How it works |+|----------|-----------|-------------|+| Nesting | Python, TypeScript, Java | Walk up the AST to find enclosing class node |+| Receiver | Go | Extract type from `func (r *Router) Handle()` receiver |+| Impl block | Rust | Find enclosing `impl Router { }` ancestor |++**Special handlers** exist for languages without tree-sitter support:++- **OpenAPI** (YAML/JSON) — extracts HTTP operations and schema definitions+- **Dockerfile** — extracts FROM, ENTRYPOINT, CMD, EXPOSE as symbols+- **Makefile** — extracts targets as functions, `include` as imports++### 2.3 GraphBuilder — "How do files depend on each other?"++See `graph-algorithms-guide.md` for full coverage. The key point: GraphBuilder takes ParsedFiles and resolves their imports into actual file paths, creating a directed dependency graph.++After building the static import graph, GraphBuilder also adds **framework-aware synthetic edges** via `add_framework_edges()`. This detects common framework patterns and creates edges for dependencies that static import resolution misses:++- **pytest**: `conftest.py` → test files in the same/child directories+- **Django**: `admin.py` → `models.py`, `urls.py` → `views.py`, `forms.py` → `models.py`, `serializers.py` → `models.py` (same directory)+- **FastAPI**: files with `include_router()` → the router modules they include+- **Flask**: files with `register_blueprint()` → the blueprint modules they register++Framework detection uses the existing `detect_tech_stack()` function (which scans `pyproject.toml`, `package.json`, etc.) to know which frameworks are present.++**Import resolution** is the tricky part — each language has different rules:++```+Python relative: "from .sibling import x" → resolve dots + walk up directories+Python absolute: "from pkg.calc import x" → try pkg/calc.py, pkg/calc/__init__.py+TypeScript: "./utils" → try utils.ts, utils.tsx, utils/index.ts+Go: "github.com/foo/bar" → match last segment "bar" by stem+Generic fallback: stem matching → "calculator" matches calculator.py+```++### 2.4 GitIndexer — "What's the history of each file?"++**The problem:** Code structure (imports, symbols) is only half the story. You also need to know: who owns this file? How often does it change? Is it a hotspot? What files change together?++**What it extracts per file:**++```+Timeline+├── commit_count_total, commit_count_90d, commit_count_30d+├── first_commit_at, last_commit_at, age_days+└── merge_commit_count_90d++Ownership+├── primary_owner (by git blame — who wrote the most lines)+├── recent_owner (most commits in last 90 days)+├── contributor_count+├── bus_factor (minimum contributors for 80% of commits)+└── top_authors_json (top 5 by commit count)++Churn+├── lines_added_90d, lines_deleted_90d+├── avg_commit_size+├── churn_percentile (rank among all files, 0.0-1.0)+├── is_hotspot (churn_percentile >= 0.75 AND active in 90d)+└── is_stable (> 10 total commits BUT 0 in 90d)++Significant Commits+└── Top 10 commits that matter (filtered)+ - Skip: merge commits, bot authors, < 12 chars+ - Skip: "Bump version", "chore:", "ci:", "style:" (unless decision signal)+ - Decision signals: "migrate", "refactor", "deprecate", "rewrite", etc.+```++**Bus factor** deserves explanation. It answers: "if people leave, when does knowledge become dangerously concentrated?"++```+Example: file with 100 commits+ Alice: 60 commits (60%)+ Bob: 25 commits (25%)+ Carol: 10 commits (10%)+ Dave: 5 commits (5%)++Walk from top: Alice alone = 60%. Not 80%. Add Bob: 85%. >= 80%.+Bus factor = 2.++If both Alice AND Bob leave, knowledge drops below 20%.+Bus factor of 1 = single point of failure.+```++**Co-change detection** finds files that change together:++```+Commit 1: [auth.py, config.py, tests/test_auth.py]+Commit 2: [auth.py, config.py]+Commit 3: [auth.py, config.py, middleware.py]++auth.py ↔ config.py: co-changed 3 times → strong signal+auth.py ↔ middleware.py: co-changed 1 time → weak signal+```++Recent co-changes count more than old ones. The score uses exponential decay:++```+score = Σ exp(-age_days / 180)+```++A co-change yesterday contributes ~1.0. A co-change 6 months ago contributes ~0.37. A co-change a year ago contributes ~0.13. This ensures the signal reflects current coupling patterns, not ancient history.++**Why a single git log call?** A naive approach would run `git log` per-file (10,000 files → 10,000 processes). Instead, GitIndexer runs one `git log --name-only` for the whole repo and parses all commits in a single pass. O(1) git processes instead of O(N).++### 2.5 ChangeDetector — "What changed since last run?"++**The problem:** After the initial indexing, you don't want to re-generate documentation for every file. You only want to update what changed.++**How it works:**++```+Step 1: git diff HEAD~1..HEAD → list of changed files++Step 2: For each changed file, parse old and new versions++Step 3: Detect symbol renames+ Old file has: calculate(), Config+ New file has: compute(), Config+ "calculate" removed, "compute" added, same kind (function),+ similar line position → likely rename (confidence scored via Levenshtein)++Step 4: Cascade through the graph+ auth.py changed+ → Who imports auth.py? → [main.py, middleware.py] (1-hop, full regen)+ → Who imports those? → [app.py, cli.py] (2-hop, decay only)+ → Co-change partners? → [config.py] (co-change, decay)++Step 5: Budget-sort regeneration list+ Sort by PageRank (most important first)+ Cap at cascade_budget (adaptive: 10 for 1 file, 30 for 2-5, up to 50 max)+ Remaining pages → decay their confidence score without regenerating+```++**Why cascade?** If `utils.py` changes, the documentation for `service.py` (which imports `utils.py`) is now potentially wrong — it might reference old function names or behavior. The cascade ensures dependent documentation stays accurate.++**Why budget?** Without a budget, changing a widely-imported file like `config.py` would trigger regeneration of hundreds of pages. The budget ensures a bounded cost per update, with PageRank-based prioritization so the most important pages get refreshed first.++---++## 3. Generation Pipeline++The generation pipeline takes ingested data and produces wiki pages using LLMs.++### 3.1 Page Types and Levels++Repowise generates **10 types of pages** in **8 ordered levels**:++```+Level 0: api_contract ← API definitions (OpenAPI, Proto, GraphQL)+Level 1: symbol_spotlight ← Individual important symbols+Level 2: file_page ← Individual code files+Level 3: scc_page ← Circular dependency documentation+Level 4: module_page ← Directory-level summaries+Level 5: cross_package ← Inter-package boundaries (monorepo only)+Level 6: repo_overview ← Repo-wide summary+ architecture_diagram ← Mermaid dependency diagram+Level 7: infra_page ← Dockerfiles, Makefiles, Terraform+ diff_summary ← Change summaries+```++**Why levels?** Later levels depend on earlier levels. A module_page (level 4) references file_pages (level 2). The repo_overview (level 6) references module_pages (level 4). By generating in order, later pages can include summaries of earlier pages as context for the LLM.++**Within each level**, pages generate concurrently (controlled by a semaphore, default concurrency = 5).++### 3.2 Page Budget — "How many pages should we generate?"++Not every file deserves its own wiki page. The budget controls cost:++```python+budget = max(50, int(total_files * 0.10)) # at most 10% of files, minimum 50+```++The budget is allocated across page types:++```+Fixed overhead (always generated):+ + all API contract files+ + all SCC cycles (size > 1)+ + all modules (top-level directories)+ + repo_overview + architecture_diagram++Remaining budget split between:+ file_pages: top 10% files by PageRank (capped by remaining)+ symbol_spotlights: top 10% public symbols by PageRank (capped by remaining)+```++**Example for a 500-file repo:**++```+Budget = max(50, 500 × 0.10) = 50 pages++Fixed overhead:+ 3 API contracts + 2 SCC cycles + 8 modules + 2 overview = 15++Remaining = 50 - 15 = 35+ file_pages: top 10% of 500 = 50, capped at 35 → 25 file pages+ symbol_spotlights: top 10% of symbols, capped at 10+```++### 3.3 Significant File Selection — "Which files get their own page?"++A file gets a Level 2 page if it passes `_is_significant_file()`:++```+Is it a package __init__.py with symbols? → YES (module interface)+Is it an entry point? → YES+Is its PageRank >= threshold (top percentile)? → YES+Is its betweenness centrality > 0? → YES (bridge file)+Does it have >= 1 symbol? → Required (unless entry point or high PageRank)++Everything else → included only in module-level summaries+```++**Why betweenness overrides PageRank here:** A bridge file might not be widely imported (low PageRank), but it's the only connection between two subsystems. Documenting it helps developers understand the coupling point.++### 3.4 ContextAssembler — "What context does the LLM need?"++Each page type has a corresponding context assembler method that builds a dataclass of everything the LLM needs to know. The context is then rendered into a Jinja2 template to form the user prompt.++**File page context example:**++```+FilePageContext+├── file_path, language+├── symbols (public first, then private)+│ └── Each with: name, kind, signature, docstring, visibility+├── imports, exports+├── dependencies (what I import) + dependents (who imports me)+├── file_source_snippet (raw code, trimmed to token budget)+├── pagerank_score, betweenness_score, community_id+├── git_metadata (ownership, churn, significant commits)+├── co_change_pages (files that change with this one)+├── dead_code_findings (unused symbols in this file)+├── depth: "minimal" | "standard" | "thorough"+├── dependency_summaries (summaries of already-generated pages)+└── rag_context (related pages from vector search)+```++**Token budgeting** ensures context fits the LLM window:++```+Token estimate = len(text) // 4 (rough approximation, no tiktoken dependency)++Budget = 48,000 tokens per page++Allocation priority:+1. File path + language + symbol signatures (always included)+2. Symbol docstrings (if budget allows)+3. Import list (capped at 30) (if budget allows)+4. Source code snippet (remainder of budget)+ → If source > 40% of budget → use structural summary instead+```++### 3.5 Generation Depth — "How detailed should docs be?"++Each file gets a depth level that tells the LLM how much detail to produce:++```+"thorough" if:+ - File is a hotspot (high churn + high commits)+ - OR has > 100 total commits AND > 10 in last 90 days+ - OR has >= 8 significant commits+ - OR has co-change partners++"minimal" if:+ - File is stable (many total commits but none recently)+ - AND PageRank < 0.3+ - AND commit count < 5++"standard" otherwise+```++**Why?** Hotspot files change often and are touched by many developers — they need comprehensive documentation explaining edge cases, rationale, and usage patterns. Stable low-importance files just need a brief API summary. This saves LLM tokens and keeps the wiki focused.++### 3.6 PageGenerator — "How are pages actually generated?"++The `generate_all()` method orchestrates everything:++```+async def generate_all():++ 1. Compute graph metrics (PageRank, betweenness, communities, SCCs)+ 2. Compute page budget and thresholds+ 3. For each level 0-7:+ a. Build list of (page_id, coroutine) pairs+ b. Skip pages already completed (for resumability)+ c. Run all coroutines concurrently (semaphore-limited)+ d. For each completed page:+ - Embed in vector store (for RAG context in later levels)+ - Extract summary (for dependency context in later levels)+ - Record in checkpoint (for resumability)+ 4. Return all generated pages+```++**Each page generation follows the same pattern:**++```+1. Assemble context → ContextAssembler.assemble_*() → dataclass+2. Render template → Jinja2 template → user_prompt string+3. Check cache → SHA256(model + type + prompt) → hit = skip LLM+4. Call LLM → provider.generate(system, user) → markdown response+5. Wrap result → GeneratedPage dataclass+6. Validate output → Cross-check backtick refs against AST symbols+```++**Step 6 (LLM output validation)** extracts all backtick-quoted names from the generated markdown (e.g., `` `calculate` ``, `` `Config` ``) and compares them against actual symbol names, exports, and imports from the ParsedFile. References that don't match any known name are logged as hallucination warnings and stored in `page.metadata["hallucination_warnings"]`. Common keywords and builtins are excluded from checking.++**After `generate_all()` completes**, the `update` command prints a **Generation Report** — a rich table showing pages by type, token counts (input/output/cached), estimated cost, elapsed time, stale page count, and hallucination warning count.++**System prompts are constant per page type.** This is intentional — Anthropic's prompt caching gives ~10% cost reduction when the system prompt is identical across requests. Since all file_pages share the same system prompt, the cache hit rate is high.++### 3.7 Job System — "What if generation crashes halfway?"++Generating documentation for a large repo might take hours and involve hundreds of LLM calls. If the process crashes at page 150 of 300, you don't want to start over.++**How it works:**++```+Checkpoint file: .repowise/jobs/{job_id}.json++{+ "status": "running",+ "total_pages": 300,+ "completed_pages": 150,+ "completed_page_ids": ["file_page:src/auth.py", "file_page:src/api.py", ...],+ "failed_page_ids": ["file_page:src/broken.py"],+ "current_level": 2+}+```++State machine:++```+pending → running → completed+ ↓+ failed+ ↓+ paused → running (resume)+```++On resume, `generate_all()` reads `completed_page_ids` and skips those pages. Generation continues from where it left off.++**Why checkpoint per page (not per level)?** A level might have 200 pages. Crashing at page 199 and restarting from page 1 of that level wastes 198 LLM calls. Per-page checkpointing means at most 1 wasted call on crash.++### 3.8 Freshness and Confidence Decay++Pages go stale as code changes. Repowise tracks this:++```+confidence = 1.0 at generation time+ → decays linearly to 0.0 over expiry_threshold_days (default 30)++freshness_status:+ "fresh" → source hash matches AND age < staleness_threshold (7 days)+ "stale" → source hash changed OR age >= 7 days (but < 30)+ "expired" → age >= 30 days (always regenerate)+```++**Source hash = SHA256(user_prompt).** If the code changes, the assembled context changes, the rendered template changes, and the hash changes. This detects staleness even without checking git — if the inputs to the LLM would be different, the page is stale.++### 3.9 RAG Integration — "Can pages reference each other?"++When generating a file_page, the system queries the vector store for related pages:++```+Query terms = file's exports or top 3 public symbol names+Results = top 3 semantically similar pages (excluding self)+Injected into ctx.rag_context as snippets+```++**Why?** If `service.py` imports `repository.py`, and we already generated a page for `repository.py`, the LLM documenting `service.py` can reference the repository documentation for accurate cross-references.++### 3.10 Editor File Generation (CLAUDE.md)++Repowise generates `CLAUDE.md` files for AI coding assistants. These are NOT LLM-generated — they're assembled from structured data:++```+CLAUDE.md contains:+ - Architecture summary (from repo_overview page)+ - Key modules table (sorted by PageRank)+ - Entry points list+ - Tech stack (categorized by type)+ - Hotspot files table (high churn)+ - Repowise MCP tools (recommended workflow)+ - Build commands (from project config)+```++**Marker-based merge** preserves user customizations:++```markdown+# CLAUDE.md+<!-- Your custom instructions here — Repowise never touches this -->++Your hand-written notes, conventions, team rules...++<!-- REPOWISE:START — Do not edit below this line. Auto-generated by Repowise. -->+[Auto-generated content updated on each run]+<!-- REPOWISE:END -->+```++On update: only content between markers is replaced. Everything above/outside is preserved.++---++## 4. Persistence Layer (Three Stores)++Repowise uses three independent storage systems because each answers a fundamentally different type of question.++### 4.1 SQL Store — "What exists?"++**Tech:** SQLAlchemy 2.0 + SQLite (default) or PostgreSQL++**What it stores:**++| Table | Contents | Key |+|-------|----------|-----|+| `repositories` | Registered repos (name, path, branch, sync state) | `local_path` |+| `wiki_pages` | Generated documentation | `page_id = "{type}:{path}"` |+| `wiki_page_versions` | Historical snapshots of pages | auto-incremented |+| `graph_nodes` | File nodes with computed metrics | `(repo_id, node_id)` |+| `graph_edges` | Import dependencies between nodes | `(repo_id, source, target)` |+| `wiki_symbols` | Functions, classes, methods | `(repo_id, symbol_id)` |+| `git_metadata` | Per-file ownership, churn, co-changes | `(repo_id, file_path)` |+| `generation_jobs` | Job status and progress | UUID |+| `dead_code_findings` | Unreachable files, unused exports | UUID |+| `decision_records` | Architectural Decision Records (ADRs) | composite natural key |++**Page versioning:**++```+First upsert → insert page, version=1, no PageVersion created+Second upsert → archive old content as PageVersion, increment to version=2+Third upsert → archive again, increment to version=3+```++This means: version 1 is the current content (in `wiki_pages`), older versions are in `wiki_page_versions`. You get history without doubling storage on the first write.++**Why natural keys?** Page IDs are `"{page_type}:{target_path}"` (e.g., `"file_page:src/auth.py"`). This means you can upsert a page knowing only its type and path — no need to query the database for an auto-generated ID first. Same for repositories (keyed by `local_path`) and decisions (keyed by title + source + evidence_file).++### 4.2 Full-Text Search — "Which pages mention X?"++**Two backends** (auto-detected from database dialect):++| Backend | Technology | How it works |+|---------|-----------|-------------|+| SQLite | FTS5 virtual table | `page_fts(page_id, title, content)` with BM25-like ranking |+| PostgreSQL | GIN index + tsvector | `to_tsvector('english', title || content)` with ts_rank |++**Query processing:**+1. Strip English stop words ("the", "is", "at", etc.)+2. Join remaining terms with OR (broad recall)+3. Support prefix matching: `"pay*"` matches "payment", "payload"++**Why a separate search index?** SQL LIKE queries don't rank results by relevance and are slow on large text columns. FTS5/tsvector indexes are optimized for text search and return results ranked by relevance.++### 4.3 Vector Store — "What's semantically similar to X?"++**Three implementations** (chosen at startup based on config):++| Implementation | Storage | Use case |+|---------------|---------|----------|+| InMemoryVectorStore | Python dict | Tests, tiny repos |+| LanceDBVectorStore | Local files (`.repowise/lancedb/`) | Default for production |+| PgVectorStore | PostgreSQL pgvector extension | When using PostgreSQL |++**How it works:**++```+1. Text → Embedder → float vector (e.g., 1536 dimensions)+2. Vector stored alongside page_id and metadata+3. Search: query text → embed → find nearest vectors by cosine similarity+4. Return top-k most similar pages+```++**All vectors are L2-normalized** (unit length). This means cosine similarity = dot product, which is cheaper to compute.++**Embedder options:**+- MockEmbedder: SHA-256 → 8-dim vector (deterministic, no API calls)+- OpenAI: `text-embedding-3-small` (1536 dims)+- Gemini: configurable dimensions++**Why a vector store alongside FTS?** FTS matches keywords — it finds "authentication" when you search "authentication." Vector search matches meaning — it finds "login flow" and "credential validation" when you search "authentication" because those concepts are semantically close in embedding space. Both are useful; Repowise tries vector search first and falls back to FTS.++### How the three stores work together++```+Page Generation:+ LLM produces markdown+ → SQL: upsert_page(page_id, content, metadata)+ → FTS: index(page_id, title, content)+ → Vector: embed_and_upsert(page_id, content, metadata)++Search query "how does auth work?":+ → Vector store: top 5 semantically similar pages+ → FTS fallback: keyword match if vector search fails+ → SQL: fetch full page content for results++Graph query:+ → SQL: read GraphNode/GraphEdge rows+ → Reconstruct NetworkX graph for algorithms+ → Return computed paths, neighborhoods, metrics+```++---++## 5. LLM Provider Abstraction++Repowise supports 6 LLM providers behind a single interface.++### The interface++```python+class BaseProvider:+ async def generate(system_prompt, user_prompt, max_tokens, temperature) → GeneratedResponse+ provider_name → str # "anthropic", "openai", etc.+ model_name → str # "claude-sonnet-4-6", "gpt-4o", etc.++GeneratedResponse:+ content: str # markdown output+ input_tokens: int+ output_tokens: int+ cached_tokens: int # Anthropic prompt cache hits+```++**Why an abstraction?** You might want to use Claude for production documentation but Ollama for local development. Or switch from OpenAI to Gemini when pricing changes. The abstraction means the generation pipeline doesn't care which LLM it's talking to.++### Provider implementations++| Provider | Key Feature | Default Model |+|----------|------------|---------------|+| **Anthropic** | Prompt caching (~10% cost savings), retry with backoff | claude-sonnet-4-6 |+| **OpenAI** | OpenAI-compatible endpoint support (works with proxies) | gpt-5.4-nano |+| **Gemini** | Runs sync SDK in thread pool (async wrapper) | gemini-3.1-flash-lite |+| **Ollama** | Local inference, no API key, air-gap compatible | configurable |+| **LiteLLM** | 100+ LLMs via unified API (Groq, Together, Azure, etc.) | configurable |+| **Mock** | Deterministic responses for testing | — |++### Provider Registry++```python+provider = get_provider("anthropic", api_key="sk-...", model="claude-sonnet-4-6")+# Automatically attaches a RateLimiter with Anthropic defaults (50 RPM, 100k TPM)+```++The registry uses **lazy imports** — `pip install repowise-core` works without installing the Anthropic or OpenAI SDK. The SDK is only imported when you actually request that provider.++### Prompt caching (Anthropic)++Anthropic caches the system prompt server-side. Since all file_pages share the same system prompt, subsequent requests within a session use the cached version at reduced token cost. Repowise exploits this by keeping system prompts constant per page type.++---++## 6. Rate Limiter++LLM APIs have rate limits (e.g., Anthropic allows 50 requests/minute). The rate limiter prevents 429 errors.++### How it works++A **sliding-window** approach tracking both requests per minute (RPM) and tokens per minute (TPM):++```+┌─── 60-second window ──────────────────────────────┐+│ req req req req req ...... req [new req?] │+│ ↑ recorded timestamps │+└───────────────────────────────────────────────────┘++Can I make a new request?+ 1. Prune timestamps older than 60 seconds+ 2. Count remaining: if < RPM limit → RPM OK+ 3. Sum tokens in window: if + estimated < TPM limit → TPM OK+ 4. Both OK → proceed. Otherwise → sleep until a slot opens.+```++**Why sleep outside the lock?**++```python+async with self._lock:+ # Check limits+ if not ok:+ sleep_time = calculate_wait()+# Release lock THEN sleep+await asyncio.sleep(sleep_time)+```++If we held the lock during sleep, all other coroutines would block waiting for the lock. By releasing before sleeping, other coroutines can check their own limits independently. This prevents thundering herd — multiple coroutines don't all wake up at the same instant.++### Default limits per provider++| Provider | RPM | TPM |+|----------|-----|-----|+| Anthropic | 50 | 100,000 |+| OpenAI | 60 | 150,000 |+| Gemini | 60 | 1,000,000 |+| Ollama | 1,000 | 10,000,000 |++Ollama limits are essentially unlimited (local inference). Gemini's TPM is high because token counting is more generous.++### Retry on 429++If the API returns 429 despite rate limiting (e.g., shared quota), exponential backoff kicks in:++```+Attempt 1: wait 2^1 + jitter = ~2.5 seconds+Attempt 2: wait 2^2 + jitter = ~4.7 seconds+Attempt 3: wait 2^3 + jitter = ~8.3 seconds+Max: 64 seconds+```++---++## 7. Server Layer++### 7.1 FastAPI App++The server is a FastAPI application created via `create_app()` factory pattern.++**Startup (lifespan manager):**+1. Initialize SQLAlchemy async engine + session factory+2. Create FTS index (idempotent)+3. Build embedder from environment (REPOWISE_EMBEDDER → mock/gemini/openai)+4. Create vector store (InMemory or LanceDB)+5. Start APScheduler background jobs+6. Bridge state to MCP server (shared DB + stores)++**12 router groups:**++| Router | Path | Purpose |+|--------|------|---------|+| health | `/health` | Liveness check |+| repos | `/api/repos` | CRUD repositories |+| pages | `/api/pages` | Wiki pages + versioning + regeneration |+| search | `/api/search` | Full-text + semantic search |+| jobs | `/api/jobs` | Job status + SSE progress stream |+| symbols | `/api/symbols` | Symbol index lookup |+| graph | `/api/graph` | Dependency graph export + pathfinding |+| git | `/api/repos/{id}/git-*` | Hotspots, ownership, git summary |+| dead_code | `/api/dead-code` | Dead code findings |+| decisions | `/api/repos/{id}/decisions` | Architectural decision records |+| webhooks | `/api/webhooks` | GitHub + GitLab push handlers |+| chat | `/api/repos/{id}/chat` | Agentic chat with tool use |++### 7.2 Webhooks — Incremental Updates++When code is pushed to GitHub/GitLab, a webhook triggers incremental documentation updates:++```+GitHub push event+ │+ ▼+ Verify HMAC-SHA256 signature+ │+ ▼+ Store WebhookEvent in DB+ │+ ▼+ Find matching Repository by URL+ │+ ▼+ Create GenerationJob:+ mode: "incremental"+ config: {before: "abc123", after: "def456"}+ │+ ▼+ Background worker picks up job:+ 1. git diff before..after → changed files+ 2. Re-parse changed files+ 3. ChangeDetector cascades through graph+ 4. Regenerate affected pages (budget-limited)+ 5. Update SQL + FTS + Vector stores+```++### 7.3 Scheduler++Two recurring background jobs (APScheduler):++| Job | Interval | Purpose |+|-----|----------|---------|+| Staleness checker | 15 min | Find stale/expired pages across all repos |+| Polling fallback | 15 min | Compare stored HEAD commit vs actual git HEAD (catches missed webhooks) |++### 7.4 Doctor and Three-Store Repair++`repowise doctor` runs health checks: git repo, `.repowise/` dir, database, state.json, providers, stale pages, and **three-store consistency** (SQL vs vector store vs FTS index).++With `--repair`, it fixes detected mismatches:+- Re-embeds pages missing from the vector store+- Re-indexes pages missing from the FTS index+- Deletes orphaned entries from vector store and FTS that no longer exist in SQL++This is powered by `list_page_ids()` on vector stores and `list_indexed_ids()` on FullTextSearch — methods that return the set of page IDs each store knows about, enabling set-difference consistency checks.++### 7.5 Chat — Agentic Loop++The chat endpoint runs an agentic loop where the LLM can call Repowise tools:++```+User: "How does auth work in this codebase?"+ │+ ▼+ LLM receives: system prompt (with repo context) + 8 tool schemas+ │+ ▼+ Iteration 1: LLM calls search_codebase("authentication")+ → Returns 5 relevant pages+ │+ ▼+ Iteration 2: LLM calls get_context(["src/auth/login.py"])+ → Returns docs + ownership + decisions+ │+ ▼+ Iteration 3: LLM synthesizes answer+ → Streams text response to user+ │+ ▼+ Save conversation to DB (for continuity)+```++Max 10 iterations per request. Streamed via SSE (Server-Sent Events).++---++## 8. MCP Tools++MCP (Model Context Protocol) lets AI coding assistants (Claude Code, Cursor, Windsurf, Cline) call Repowise tools directly. There are 8 tools, each answering a specific question.++### Tool 1: `get_overview` — "What is this codebase?"++**When to use:** First time exploring an unfamiliar repo.++**Returns:** Architecture summary, module map, entry points, and git health metrics (hotspot count, average bus factor, churn trend).++**Git health computation:**++```+churn_trend:+ recent_rate = total_commits_30d / 30 days+ baseline_rate = (total_commits_90d - total_commits_30d) / 60 days++ recent_rate > baseline × 1.3 → "increasing" (accelerating development)+ recent_rate < baseline × 0.7 → "decreasing" (cooling down)+ otherwise → "stable"+```++### Tool 2: `get_context` — "What does this code do?"++**When to use:** Before modifying a file, to understand its documentation, ownership, and governing decisions.++**Target resolution** (4-phase fallback):++```+Input: "auth.py"+ 1. Try exact: file_page:auth.py → found? return+ 2. Try module: target_path = auth.py → found? return+ 3. Try symbol: ILIKE '%auth.py%' → found? return+ 4. Try file: target_path = auth.py → found? return+ 5. Fallback: check GitMetadata exists → suggest fuzzy matches+```++**Returns:** Documentation, symbol list, importers, ownership (primary + recent owner), last change, governing decisions, freshness score.++### Tool 3: `get_risk` — "Will changing this break anything?"++**When to use:** Before a refactor, to assess blast radius.++**Risk type classification:**++```+bug-prone: >= 40% of commits match (fix|bug|patch|hotfix|revert|crash|error)+churn-heavy: churn_percentile >= 0.7+bus-factor-risk: bus_factor == 1 AND total_commits > 20+high-coupling: dependents >= 5+stable: none of the above+```++**Impact surface** — BFS up 2 hops through reverse dependencies, ranked by PageRank:++```+auth.py changed+ → middleware.py imports auth.py (PageRank 0.05)+ → main.py imports middleware.py (PageRank 0.08, entry point)+ → app.py imports main.py (PageRank 0.12)++Impact surface: [app.py, main.py, middleware.py] (top 3 by criticality)+```++### Tool 4: `get_why` — "Why was this built this way?"++**When to use:** When you find surprising code and need to understand the rationale.++**Four modes:**++| Mode | Trigger | Returns |+|------|---------|---------|+| Health dashboard | No query, no targets | Decision counts, stale decisions, ungoverned hotspots |+| Path analysis | Target provided | Decisions governing that file + origin story + alignment score |+| Semantic search | Query provided | Decisions matching the query semantically |+| Target-aware search | Both | Decisions matching query that also govern targets |++**Origin story** — reconstructs the history of a file:++```+Who created it? → first commit author+Who maintains it now? → primary owner by blame+Key decisions? → commits with "migrate", "refactor", "deprecate" matched to ADRs+Alignment? → "high" if active decisions exist and siblings are similarly governed+```++### Tool 5: `search_codebase` — "Where is X implemented?"++**When to use:** Looking for specific functionality across the codebase.++**Search strategy:**++```+1. Wait for vector store ready (background async loading, up to 30s)+2. Semantic search (embedding similarity)+ → Fetch 3× limit (for filtering headroom)+ → 8-second timeout+3. Fallback to FTS if semantic fails+4. Boost recently-modified files:+ - Modified in last 30 days: +1.0× to relevance+ - Modified in last 90 days: +0.5×+5. Normalize confidence relative to top result+```++### Tool 6: `get_dependency_path` — "How does A connect to B?"++**When to use:** Understanding how two seemingly unrelated files are connected.++**When a path exists:** Returns the chain of imports (BFS shortest path).++**When no path exists:** Returns rich diagnostic context:++```+visual_context:+ reverse_path: Does B → A exist? (dependency flows the other way)+ common_ancestors: Nodes reachable from both A and B (via undirected BFS)+ shared_neighbors: Files that both A and B directly connect to+ community_analysis: Are A and B in the same community?+ bridge_suggestions: High-PageRank files connecting both communities++co_change_signal: If no import link but frequent co-changes → logical coupling+```++### Tool 7: `get_dead_code` — "What can we safely delete?"++**When to use:** Cleanup sprints, reducing maintenance burden.++**Findings organized in three tiers:**++| Tier | Confidence | Meaning |+|------|-----------|---------|+| High | >= 0.8 | Almost certainly dead. No importers, no recent commits, old. |+| Medium | 0.5 - 0.8 | Probably dead. No importers but has some recent activity. |+| Low | < 0.5 | Suspicious. Might be dynamically loaded or framework-used. |++**Supports rollup by directory or owner** — so you can say "show me all dead code owned by Alice in the payments/ directory."++### Tool 8: `get_architecture_diagram` — "Show me the structure"++**When to use:** Getting a visual understanding of the codebase architecture.++**Three scopes:**++| Scope | What you get |+|-------|-------------|+| `"repo"` | Full architecture diagram (from pre-generated page) |+| `"module"` | Subgraph of a specific module and its dependencies |+| `"file"` | Single file with its imports and importers |++Returns Mermaid syntax that renders as a flowchart in any Mermaid-compatible viewer.++---++## 9. Frontend (Web UI)++### Tech Stack++```+Next.js 15 + React 19 + TypeScript+Tailwind CSS v4 + Radix UI primitives+SWR (data fetching) + SSE (live job progress)+React Flow (graph visualization) + ELK.js (hierarchical layout)+Shiki (syntax highlighting) + next-mdx-remote (wiki rendering)+Recharts (charts) + Mermaid (diagrams)+Framer Motion (animations)+```++### Pages++| Route | Purpose |+|-------|---------|+| `/` | Dashboard — repo list, recent jobs, aggregate stats |+| `/repos/[id]` | Repository overview |+| `/repos/[id]/wiki/[...slug]` | Wiki page viewer (MDX with Mermaid + syntax highlighting) |+| `/repos/[id]/graph` | Interactive dependency graph (5 view modes) |+| `/repos/[id]/search` | Full-text and semantic search |+| `/repos/[id]/symbols` | Symbol index (sortable, filterable table) |+| `/repos/[id]/coverage` | Documentation freshness dashboard |+| `/repos/[id]/ownership` | Code ownership breakdown |+| `/repos/[id]/hotspots` | High-churn files |+| `/repos/[id]/dead-code` | Dead code findings with bulk actions |+| `/repos/[id]/decisions` | Architectural decision records |+| `/settings` | Global and per-repo settings |++### Graph Visualization++**5 view modes:**++| Mode | What it shows | Layout |+|------|-------------|--------|+| Module | Directory-level nodes (click to drill down) | ELK hierarchical |+| Full | File-level dependency graph | ELK hierarchical |+| Architecture | Entry points + 3-hop reachability | ELK hierarchical |+| Dead code | Unreachable nodes highlighted | ELK hierarchical |+| Hot files | High-churn files + their dependencies | ELK hierarchical |++**ELK.js** (Eclipse Layout Kernel) is used instead of force-directed layout because dependency graphs are DAGs — they have a natural top-to-bottom flow. ELK's layered algorithm respects this directionality, producing clean hierarchical layouts instead of the tangled hairballs that force-directed layouts create for DAGs.++**Interactive features:**+- **Drill-down**: In module view, click a module to expand into sub-modules+- **Path finder**: Select two nodes, find the shortest dependency path+- **Context menu**: Right-click for view docs, explore, set as path endpoint+- **Color modes**: By language, PageRank, or community+- **Ego sidebar**: Click a node to see its stats, git metadata, and connections+- **MiniMap**: Color-coded by doc coverage or importance++---++## Putting It All Together++Here's the complete flow from `repowise init` to serving documentation:++```+repowise init ./my-project+ │+ ▼+ FileTraverser: walk directory, apply filters+ → 500 files discovered+ │+ ▼+ ASTParser: tree-sitter queries per file+ → 500 ParsedFiles with 3,000 symbols and 2,000 imports+ │+ ▼+ GraphBuilder: resolve imports, build directed graph+ → 500 nodes, 2,000 edges+ → Compute: PageRank, betweenness, SCCs, communities+ │+ ▼+ GitIndexer: mine commit history+ → Ownership, churn, bus factor, co-changes per file+ │+ ▼+ DeadCodeAnalyzer: graph reachability from entry points+ → 15 unreachable files, 30 unused exports+ │+ ▼+ PageGenerator.generate_all():+ → Level 0: 3 API contracts+ → Level 1: 10 symbol spotlights+ → Level 2: 25 file pages+ → Level 3: 2 SCC cycle pages+ → Level 4: 8 module pages+ → Level 5: 3 cross-package pages+ → Level 6: repo overview + architecture diagram+ → Level 7: 2 infra pages+ Total: 55 pages generated via LLM+ │+ ▼+ Persistence:+ → SQL: 55 pages + 500 graph nodes + 2,000 edges + 500 git metadata rows+ → FTS: 55 pages indexed for keyword search+ → Vector: 55 pages embedded for semantic search+ → Graph: metrics stored in graph_nodes table+ │+ ▼+ CLAUDE.md: generated with architecture summary, module map, hotspots+ │+ ▼+ Ready to serve:+ → repowise serve → REST API + Web UI at http://localhost:8877+ → repowise mcp → MCP server for Claude Code / Cursor / Cline+ → repowise search → CLI search across documentation+ → repowise update → Incremental sync after code changes+```diff --git a/docs/critical-analysis.md b/docs/critical-analysis.mdnew file mode 100644index 000000000..e7b5ff690--- /dev/null+++ b/docs/critical-analysis.md@@ -0,0 +1,552 @@+# Repowise — Critical Analysis++An honest assessment of what Repowise solves, where it can fail, how likely those failures are, and what would make it better.++---++## Table of Contents++1. [What Problem Does Repowise Solve?](#1-what-problem-does-repowise-solve)+2. [How Useful Is It?](#2-how-useful-is-it)+3. [Where the Architecture Can Fail](#3-where-the-architecture-can-fail)+4. [Improvement Suggestions](#4-improvement-suggestions)++---++## 1. What Problem Does Repowise Solve?++### The core problem: Code knowledge is invisible and perishable++When a developer joins a team, they face a codebase with thousands of files and no map. They ask:++- "What does this file do?" → Read the code, guess, or interrupt someone.+- "Why is it built this way?" → Nobody remembers. The original author left.+- "If I change this, what breaks?" → No way to know without deep expertise.+- "What's dead code vs critical infrastructure?" → Looks the same in the file tree.++This knowledge exists — scattered across git history, code comments, Slack threads, and people's heads. But it's not **accessible** at the moment of need.++### What Repowise does about it++Repowise automates the creation and maintenance of codebase intelligence:++| Problem | Repowise Solution | How |+|---------|-------------------|-----|+| No documentation | Auto-generated wiki pages | Tree-sitter parsing + LLM generation |+| Don't know what's important | PageRank scoring | Graph algorithms on import dependencies |+| Don't know who owns what | Ownership tracking | Git blame + commit history analysis |+| Don't know what's risky to change | Risk assessment | Churn × centrality × bus factor |+| Don't know what's dead | Dead code detection | Graph reachability + git activity |+| Don't know why decisions were made | Decision archaeology | Inline markers + git commit mining + README extraction |+| Documentation goes stale | Incremental updates | Change detection + cascade propagation |+| AI assistants lack codebase context | MCP tools | 8 specialized tools for Claude Code, Cursor, etc. |++### What makes it non-trivial++Many tools generate documentation. Repowise's differentiators are:++1. **Graph-aware intelligence.** It doesn't just document files — it understands how they connect. PageRank identifies what matters. Betweenness identifies bottlenecks. Communities identify subsystems. This structural understanding drives everything from doc priority to risk assessment.++2. **Temporal intelligence.** Git history adds a time dimension. A file's importance isn't just structural (who imports it) but behavioral (how often it changes, who touches it, what changes alongside it). Co-change detection finds coupling invisible in the import graph.++3. **Decision archaeology.** No other tool automatically discovers *why* code is built the way it is by mining commits, comments, and documentation. This is the hardest kind of knowledge to preserve and the most valuable.++4. **Incremental maintenance with cascade.** The cascade algorithm is genuinely clever — it propagates staleness through the dependency graph with a bounded budget, ensuring the most important docs stay fresh without unbounded cost.++---++## 2. How Useful Is It?++### High value scenarios++| Scenario | Value | Why |+|----------|-------|-----|+| New developer onboarding | **Very High** | Replaces weeks of "reading code and asking questions" with searchable documentation, architecture diagrams, and ownership maps |+| Pre-refactor impact analysis | **Very High** | `get_risk` tells you exactly what depends on what you're changing, who owns it, and how volatile it is — before you touch anything |+| AI-assisted development | **High** | MCP tools give Claude Code / Cursor deep codebase context. Instead of the AI guessing, it queries real docs, ownership, and decisions |+| Dead code cleanup | **High** | Automated detection with confidence tiers saves engineering time. The "safe to delete" flag reduces review burden |+| Knowledge preservation | **High** | When senior developers leave, their decision rationale is captured in the wiki and decision records instead of leaving with them |+| Code review context | **Medium** | Reviewer can check the wiki for file context, ownership, and governing decisions before reviewing a PR |+| Compliance and auditing | **Medium** | Decision records with provenance (source, evidence commits, confidence) provide audit trails |++### Moderate value scenarios++| Scenario | Value | Limitation |+|----------|-------|-----------|+| Small repos (< 50 files) | **Low** | You can read 50 files yourself. The overhead of running Repowise isn't justified. |+| Rapidly prototyping repos | **Low** | Code changes so fast that documentation is stale within hours. The incremental system helps but can't keep up with constant rewrites. |+| Non-code-heavy repos | **Low** | Repos that are mostly config, data, or documentation get little value from AST parsing and dependency graphs. |++### Quantitative value estimate++For a **500-file repo with 10 developers**:++```+Without Repowise:+ New developer onboarding: ~2 weeks of asking questions+ Finding file owners: ~10 min per file (git blame, asking around)+ Pre-refactor impact: ~1-2 hours of manual grep + tracing+ Dead code discovery: ~1-2 days of manual audit per quarter++With Repowise:+ New developer onboarding: ~2 days reading wiki + using search+ Finding file owners: instant (ownership data in every page)+ Pre-refactor impact: ~30 seconds (get_risk tool call)+ Dead code discovery: instant (pre-computed with confidence tiers)++Time saved per developer per quarter: ~15-25 hours+For 10 developers: ~150-250 hours/quarter+```++The ROI depends on the LLM cost of `repowise init` (potentially $5-50 depending on provider and repo size) versus the engineering hours saved.++---++## 3. Where the Architecture Can Fail++### Failure Category 1: Data Consistency (Three-Store Split)++**The problem:** Repowise writes to three independent stores — SQL, vector store, and graph — without atomic transactions across them. They can get out of sync.++**Failure scenario:**++```+1. LLM generates page content ✓+2. Embed in vector store ✓+3. Upsert in SQL database ✗ (database connection drops)+4. Index in FTS (never reached)++State: Vector store has the page. SQL doesn't. FTS doesn't.+Result: Semantic search finds the page, but clicking it returns 404.+```++**Reverse scenario:**++```+1. LLM generates page content ✓+2. Embed in vector store ✗ (embedding API timeout)+3. Upsert in SQL database ✓+4. Index in FTS ✓++State: SQL and FTS have the page. Vector store doesn't.+Result: Keyword search works. Semantic search misses the page.+```++**Probability: Medium (5-15% over a large init run).** Vector store embedding involves an external API call (OpenAI/Gemini). Over hundreds of pages, at least one is likely to fail. The failure is silent (logged at debug level), so the user may not notice.++**Impact: Low-Medium.** Individual pages missing from one store doesn't break the system — it degrades search quality. But the user has no way to detect or repair the inconsistency.++**Why it happens:** True distributed transactions across SQLite + LanceDB + an embedding API are impractical. The architecture prioritizes availability (keep going if one store fails) over consistency (all stores agree).++---++### Failure Category 2: LLM Output Quality++**The problem:** Generated documentation is only as good as the LLM's understanding. The LLM sees assembled context (symbols, imports, source code snippets), not the running system.++**Failure scenarios:**++| Scenario | Probability | Impact |+|----------|------------|--------|+| LLM hallucinates function behavior that doesn't match the code | **Medium (10-20%)** | Misleading docs. Developer trusts wiki and writes incorrect code. |+| LLM misunderstands complex metaprogramming, decorators, or dynamic dispatch | **High (30%+)** for such files | Missing or wrong symbol documentation |+| LLM generates vague "this module handles X" without specifics | **Medium (15-25%)** for low-context files | Docs exist but aren't useful |+| LLM fails to converge on architecture diagram | **Low (< 5%)** | Falls back to no diagram |++**Why it's hard to fix:** The LLM doesn't execute the code. It infers behavior from static text. For straightforward code (clear function names, type annotations, docstrings), accuracy is high. For metaprogramming-heavy code (Python decorators that transform functions, Ruby method_missing, JavaScript Proxy objects), the LLM may produce plausible but wrong descriptions.++**Compounding problem:** Once an incorrect doc is embedded in the vector store, it becomes RAG context for other pages. If `auth.py`'s docs incorrectly describe its behavior, `service.py`'s docs (which reference `auth.py` via dependency summaries) inherit the error.++---++### Failure Category 3: Stale Documentation Drift++**The problem:** Even with incremental updates, documentation can drift from reality.++**How staleness accumulates:**++```+Day 1: repowise init → all docs fresh+Day 3: Developer changes config.py → cascade updates 30 pages+ But cascade_budget = 30, and 50 pages actually depend on config.py+ 20 pages are marked "decay_only" (stale, not regenerated)+Day 5: Developer changes utils.py → cascade updates 30 different pages+ The 20 stale pages from Day 3 are still stale+Day 10: Those 20 pages hit staleness_threshold (7 days)+ But nobody runs repowise update again+Day 30: Those pages hit expiry_threshold and would be regenerated+ IF someone runs repowise update+```++**Probability: High (almost certain) for repos without automated CI integration.** If `repowise update` only runs when a developer remembers to run it, documentation drift is inevitable.++**Impact: Medium.** Stale docs are worse than no docs — they give a false sense of confidence. The freshness indicator (confidence score) mitigates this, but only if developers check it.++---++### Failure Category 4: Graph Accuracy (Import Resolution)++**The problem:** The dependency graph is only as accurate as import resolution. Missed or wrong imports lead to wrong PageRank, wrong cascade propagation, and wrong dead code detection.++**Where import resolution fails:**++| Pattern | Example | What Repowise misses |+|---------|---------|---------------------|+| Dynamic imports | `importlib.import_module(name)` | Entire dependency edge |+| Conditional imports | `if TYPE_CHECKING: import X` | Edge exists but is typing-only, not runtime |+| Re-exports through `__init__.py` | `from .utils import *` | May miss specific names in wildcard |+| Barrel files (TypeScript) | `export * from './module'` | Specific names re-exported |+| Build-time code generation | Protobuf → `_pb2.py` files | Generated files might be gitignored |+| Runtime plugin loading | Django apps, pytest plugins | Framework-mediated imports |+| Monkeypatching | `module.func = my_func` | No import edge at all |++**Probability: Medium-High (20-40% of edges affected in large repos).** Most codebases use some dynamic imports. The more framework-heavy the code (Django, Flask, FastAPI with dependency injection), the more imports are invisible to static analysis.++**Impact cascades:**++```+Missing edge A → B means:+ - A's documentation doesn't mention dependency on B+ - B's PageRank is too low (missing one vote)+ - Changing B doesn't cascade to A's docs (becomes stale silently)+ - B might be falsely flagged as dead code (in_degree appears lower)+```++**Partial mitigation:** Co-change edges partially compensate. If A and B always change together (captured from git history), they get a co-change edge even without an import edge. But co-change edges don't affect PageRank (deliberately excluded).++---++### Failure Category 5: Scale and Performance++**The problem:** Several components are in-memory and O(n²) or worse.++| Component | Complexity | Breaks at | What happens |+|-----------|-----------|-----------|-------------|+| NetworkX graph in memory | O(N + E) memory | ~100K files with dense edges | OOM crash during `graph.build()` |+| Betweenness centrality (exact) | O(N × E) time | >30K nodes | Falls back to sampling (k=500), approximate results |+| PageRank convergence | O(E × iterations) | Unusual graph topologies | `PowerIterationFailedConvergence` → falls back to uniform scores |+| Symbol rename detection | O(removed × added) per file | Files with 500+ symbols | Slow rename detection, but no hard limit |+| InMemory vector search | O(N × D) per query | >10K pages | Seconds per search, kills UX |+| `git log --name-only` | O(commits) | >500K commits | Parsing huge git log output |+| Co-change pair computation | O(files_per_commit²) per commit | Commits touching 100+ files | Quadratic pair generation |++**Probability:**++- For repos < 10K files: **Very Low** (< 1%). Everything fits comfortably.+- For repos 10K-50K files: **Low** (5-10%). Might see betweenness approximation and slower git indexing.+- For repos 50K-100K files: **Medium** (20-30%). Memory pressure on graph, long git indexing times.+- For repos > 100K files: **High** (50%+). Likely needs architectural changes (streaming graph, approximate algorithms, sharded processing).++---++### Failure Category 6: Decision Mining Accuracy++**The problem:** LLM-based decision extraction from git commits and READMEs produces false positives and false negatives.++**False positives (phantom decisions):**++```+Commit: "Migrate test fixtures to new format"+LLM extracts: "Decision: Migrate to new test format"+Reality: This was a routine chore, not an architectural decision+Confidence: 0.70 (git_archaeology)+```++**Probability: Medium (15-25% of git-mined decisions).** The word "migrate" triggers extraction, but not every migration is an architectural decision.++**False negatives (missed decisions):**++```+Commit: "Rewrote the query layer because the ORM couldn't handle+ the new partitioning scheme"+→ Might be missed if commit doesn't use signal keywords+```++Decisions expressed in natural language without keywords like "migrate", "switch to", "replace" won't be found.++**Impact: Medium.** False positives clutter the decision list (proposed status helps — developers can dismiss them). False negatives mean some decisions are invisible (the harder problem).++---++### Failure Category 7: Job System and Resumability++**The problem:** The checkpoint file (JSON on disk) and the database can get out of sync.++**Failure scenario:**++```+1. Page generated successfully+2. job_system.complete_page(job_id, page_id) → JSON file updated ✓+3. Process crashes before session.commit()+4. Database doesn't have the page++On resume:+5. Job checkpoint says page_id is complete (skip it)+6. But database doesn't have it+7. Page is permanently missing+```++**Probability: Low (1-3% per large init run).** Requires a crash between two specific lines. But over hundreds of pages, the window is hit occasionally.++**Impact: Low.** The missing page is a gap in documentation. It won't be regenerated because the checkpoint says it's done. Manual intervention (clearing the checkpoint) is needed.++---++### Failure Category 8: Security and Sensitive Data++**The problem:** Repowise sends source code to external LLM APIs.++| Risk | Probability | Severity |+|------|------------|----------|+| Source code sent to LLM API (Anthropic, OpenAI) includes secrets | **Low** (file size + gitignore filtering helps) | **High** if secrets leak |+| `.env` files parsed and sent as context | **Very Low** (excluded by traverser) | **Critical** |+| API keys in code comments sent to LLM | **Low-Medium** | **High** |+| Generated docs stored in plain text in `.repowise/wiki.db` | **Certain** (by design) | **Medium** (local file, but if shared...) |+| Vector embeddings encode semantic content of code | **Certain** | **Low** (embeddings aren't reversible to code, but contain signal) |++**Mitigation:** Ollama provider supports fully offline operation. For sensitive codebases, this is the only safe option.++---++### Summary: Failure probability matrix++| Failure Category | Probability | Impact | Detection | Recovery |+|-----------------|------------|--------|-----------|----------|+| Three-store inconsistency | Medium | Low-Medium | Hard (silent failures) | Manual re-index |+| LLM hallucination in docs | Medium | Medium | Hard (looks plausible) | Re-generate page |+| Documentation staleness | High | Medium | Easy (confidence score) | Run update |+| Import resolution gaps | Medium-High | Medium | Hard (invisible edges) | None (fundamental limit) |+| Scale/performance limits | Low-High (repo dependent) | High | Easy (OOM, timeout) | Architecture changes |+| Decision mining false positives | Medium | Low | Easy (review proposed) | Dismiss via CLI |+| Job checkpoint desync | Low | Low | Hard (silent gap) | Clear checkpoint |+| Sensitive data exposure | Low | High | Medium (audit logs) | Use Ollama |++---++## 4. Improvement Suggestions++> **Note:** Five of the improvements below have been implemented. They are marked with **(IMPLEMENTED)** and describe what was built rather than what should be built.++### Priority 1: Data Consistency (High impact, Medium effort) — IMPLEMENTED++**Problem:** Three stores can get out of sync.++**Implemented: `repowise doctor --repair`**++The `doctor` command now checks SQL↔Vector Store and SQL↔FTS consistency by comparing page ID sets across all three stores. With `--repair`, it re-embeds missing vector entries, re-indexes missing FTS entries, and deletes orphans. This is powered by new `list_page_ids()` methods on all vector store implementations and `list_indexed_ids()` on FullTextSearch.++**Original suggestion (not yet implemented): Write-ahead log for multi-store writes**++```+Before:+ 1. embed_to_vector_store(page) # may fail+ 2. upsert_to_sql(page) # may fail+ 3. index_to_fts(page) # may fail++After:+ 1. write_to_wal(page_id, content, metadata) # single append+ 2. embed_to_vector_store(page) # may fail+ 3. upsert_to_sql(page) # may fail+ 4. index_to_fts(page) # may fail+ 5. mark_wal_entry_complete(page_id)++On startup:+ for entry in wal where not complete:+ retry all three stores+```++This doesn't give you atomic transactions, but it gives you **eventual consistency** — every page that was generated will eventually land in all three stores, even if the process crashes mid-write.++**Simpler alternative:** Add a `repowise doctor --repair` command that compares SQL pages vs vector store entries vs FTS index and re-syncs any mismatches. Not real-time, but catches drift.++### Priority 2: LLM Output Validation (High impact, Low effort) — IMPLEMENTED++**Problem:** LLM output is trusted without validation.++**Implemented:** `_validate_symbol_references()` in `page_generator.py` extracts all backtick-quoted names from generated markdown via regex, cross-checks them against actual symbols, exports, and imports from the ParsedFile. Warnings are logged via structlog and stored in `page.metadata["hallucination_warnings"]`. Common language keywords and builtins are excluded via `_BACKTICK_SKIP`.++**Additional suggestions (not yet implemented):**++1. **Structural validation:** After LLM generates a page, verify it contains expected sections (e.g., file_page should have at least a title and a symbol table). Reject and retry if malformed.++2. **Factual cross-check:** Compare function names in the generated docs against actual symbol names from AST parsing. If the docs reference a function that doesn't exist in the file, flag as hallucination.++ ```python+ actual_symbols = {s.name for s in parsed_file.symbols}+ mentioned_symbols = extract_code_references(generated_content)+ hallucinated = mentioned_symbols - actual_symbols+ if hallucinated:+ log.warning("Possible hallucination", symbols=hallucinated)+ # Option: re-generate with stronger prompt, or add warning to page+ ```++3. **Confidence-weighted display:** Show LLM confidence alongside generated content. Pages generated with "minimal" depth or low-context could display a "low detail" indicator.++### Priority 3: Dynamic Import Detection (High impact, High effort) — PARTIALLY IMPLEMENTED++**Problem:** Static import resolution misses dynamic imports, framework-mediated loading, and plugin systems.++**Implemented:** `add_framework_edges()` on `GraphBuilder` adds synthetic edges with `edge_type="framework"` for four frameworks:+- **pytest**: conftest.py → test files in same/child directories+- **Django**: admin→models, urls→views, forms→models, serializers→models+- **FastAPI**: `include_router()` calls → router modules+- **Flask**: `register_blueprint()` calls → blueprint modules++Framework detection uses regex on source code and the existing `detect_tech_stack()` from manifest files. Called automatically during `repowise init` and `repowise update`.++**Additional suggestions (not yet implemented):**++1. **More framework-aware heuristics:** Additional patterns to detect:++ ```python+ # Django: settings.INSTALLED_APPS → app module imports+ # Flask: app.register_blueprint(bp) → blueprint module+ # FastAPI: app.include_router(router) → router module+ # pytest: conftest.py fixtures → test files using them+ ```++ Each framework pattern is a known import mechanism that can be statically detected even though it's not a Python `import` statement.++2. **Runtime trace integration (optional):** For Python, run `sys.settrace()` during tests to capture actual imports. Merge runtime edges with static edges. This captures every dynamic import that's exercised by tests. Downside: requires running tests, which may be slow or unavailable.++3. **Heuristic edge recovery from co-change:** If two files always co-change AND are in related directories AND have overlapping symbol names, add a weak synthetic import edge. This is noisy but better than nothing for dynamic-import-heavy codebases.++### Priority 4: Scale Architecture (Medium impact, High effort)++**Problem:** In-memory graph, in-memory vector search, and O(n²) algorithms limit scale.++**Suggestions:**++1. **Streaming graph construction:** Instead of building the full NetworkX graph in memory, use a SQLite-backed graph where nodes and edges are stored in tables. Compute metrics incrementally or with SQL-based algorithms.++ ```+ Current: Build graph in memory → compute metrics → persist to SQLite+ Improved: Build directly in SQLite → compute metrics with SQL queries+ (PageRank via iterative SQL)+ ```++2. **Approximate graph metrics at scale:** For repos > 50K files, consider:+ - **PageRank:** Use the power iteration directly on the edge list (no need for full graph in memory). Can be done with SQL or sparse matrix operations.+ - **Betweenness:** Already approximated with sampling. Could use a more efficient algorithm like Brandes' with better sampling strategies.+ - **Communities:** Louvain is already fast. No change needed.++3. **Chunk-based vector search:** Replace InMemoryVectorStore default with LanceDB even for small repos. LanceDB handles scaling internally with IVF-PQ indexes. The in-memory implementation should be test-only.++4. **Parallel git indexing with process pool:** The current ThreadPoolExecutor(20) is limited by Python's GIL for CPU-bound work and GitPython's thread safety for I/O. Consider using `multiprocessing` for git operations, or calling `git log` as a subprocess and parsing output.++### Priority 5: Staleness Prevention (Medium impact, Medium effort) — PARTIALLY IMPLEMENTED++**Problem:** Documentation drifts if nobody runs `repowise update`.++**Suggestions:**++1. **CI/CD integration as first-class:** Provide a GitHub Action and GitLab CI template that runs `repowise update` on every push to main. This makes maintenance automatic rather than opt-in.++ ```yaml+ # .github/workflows/repowise.yml+ on:+ push:+ branches: [main]+ jobs:+ update-docs:+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ with: { fetch-depth: 0 }+ - run: pip install repowise-cli+ - run: repowise update --cascade-budget 50+ - run: repowise serve --export # publish to static site+ ```++2. **Staleness alerts in CLAUDE.md:** When generating CLAUDE.md, include a staleness summary:++ ```markdown+ ⚠ 15 pages are stale (last updated > 7 days ago)+ Run `repowise update` to refresh documentation.+ ```++ AI coding assistants reading CLAUDE.md would see this and could suggest running an update.++3. **Smarter cascade budgeting — IMPLEMENTED:** `compute_adaptive_budget()` in `change_detector.py` scales budget by change magnitude: 1 file→10, 2-5→30, 6+→min(n×3, 50). Hard cap at 50. The `--cascade-budget` CLI flag defaults to auto when unset, and users can still override manually.++### Priority 6: Decision System Refinement (Medium impact, Low effort)++**Problem:** False positives in git archaeology; no user feedback loop.++**Suggestions:**++1. **Feedback-weighted confidence:** When a developer dismisses a proposed decision (via `repowise decision dismiss`), record the false positive pattern. Over time, learn which commit message patterns produce false positives and lower their confidence.++ ```+ Dismissed: "Migrate test fixtures to new format" (git_archaeology)+ Pattern learned: "migrate" + "test" + "fixture" → lower confidence by 0.15+ ```++2. **Decision coverage in file pages:** When generating a file_page, include a section on governing decisions. This surfaces decisions where developers actually need them — in the documentation for the file they're about to modify.++3. **Decision expiry:** Auto-deprecate decisions that haven't been confirmed and have high staleness scores for > 90 days. Currently decisions stay "active" forever unless manually changed.++### Priority 7: Observability and Repair (Low impact, Low effort) — IMPLEMENTED++**Problem:** Failures are silent. Users don't know when something is degraded.++**Implemented:**++1. **`repowise doctor --repair`:** Extended with three-store consistency checks (SQL↔Vector, SQL↔FTS) and automated repair (re-embed, re-index, delete orphans). See Priority 1.++2. **Generation report:** `GenerationReport` dataclass in `report.py` with `render_report()` prints a rich table after `repowise update` showing: pages by type, input/output/cached tokens, estimated cost, elapsed time, stale page count, hallucination warning count, and decisions extracted.++**Not yet implemented:**++3. **Structured error log:** `.repowise/errors.json` with timestamped, categorized errors. The `doctor` command would read this to suggest repairs.++### Priority 8: Testing and Validation (Low impact, High effort)++**Suggestions:**++1. **Golden test suite:** For 3-5 well-known open source repos (different sizes, languages, frameworks), generate documentation and manually validate key pages. Store as golden snapshots. Run against each new Repowise version to detect regressions.++2. **Scale benchmark:** Run `repowise init` against repos of increasing size (1K, 5K, 10K, 50K, 100K files) and record: wall time, peak memory, LLM token cost, page quality (manual sample). Publish as a benchmark table.++3. **Import resolution accuracy test:** For a known repo, manually annotate all import relationships. Compare against Repowise's resolved graph. Compute precision (no false edges) and recall (no missing edges). This quantifies the fundamental limitation of static import resolution.++---++## Summary++### What Repowise gets right++- **Graph-first architecture** is the right foundation. Import graphs capture real structural dependencies that keyword search and file-listing tools miss entirely.+- **Temporal intelligence** (git history, co-change, staleness) adds a dimension that pure static analysis can't provide.+- **Bounded cascade** is an elegant solution to the exponential propagation problem. PageRank-sorted budgets ensure the most valuable docs stay fresh.+- **Multi-source decision extraction** is genuinely novel. No other open-source tool mines architectural intent from commits, comments, and documentation simultaneously.+- **MCP integration** makes the intelligence immediately useful inside AI coding workflows, which is where developers increasingly spend their time.++### What has been addressed++Five of the top priorities have been implemented:++| Risk | Status | Implementation |+|------|--------|---------------|+| Three-store consistency | **Fixed** | `doctor --repair` checks SQL↔Vector↔FTS and auto-repairs |+| Dynamic import blindness | **Partially fixed** | Framework edges for pytest, Django, FastAPI, Flask |+| LLM output validation | **Fixed** | Symbol cross-check on every file_page generation |+| Cascade budget rigidity | **Fixed** | Adaptive budget scales 10-50 based on change magnitude |+| Silent failures | **Fixed** | Generation report after every update run |++### What still needs work++1. **CI-first staleness prevention** — the biggest adoption risk. A GitHub Action template exists in `docs/` but isn't yet a first-class integration.+2. **Scale architecture** — in-memory graph and O(n²) algorithms still limit repos > 50K files.+3. **Decision auto-expiry** — decisions with high staleness for 90+ days should auto-deprecate (partially designed, not yet in crud.py).+4. **Runtime trace integration** — would capture dynamic imports exercised by tests, complementing the static framework heuristics.++### Overall assessment++Repowise solves a real, painful problem. The architecture is well-designed for its intended scale (repos up to ~50K files). The graph-based approach is fundamentally sound — it captures structural relationships that simpler tools miss. The incremental update system with adaptive cascade budgets is sophisticated and practical.++The recent improvements address the top risks: three-store consistency is now detectable and repairable, LLM hallucinations are flagged via symbol cross-checking, framework-mediated imports are captured for the most common Python frameworks, and generation runs produce clear cost/quality summaries.++Remaining risks are: adoption requiring CI integration for automatic maintenance, scale limits for very large monorepos, and decision system maturity. None of these are fundamental to the architecture — they're engineering hardening, not redesign.++For a team with a 500-10,000 file codebase that wants to reduce onboarding time, improve refactoring safety, and preserve architectural knowledge, Repowise delivers substantial value. For very small repos (< 50 files) or very large monorepos (> 100K files), the value proposition is weaker — the first doesn't need it, the second might outgrow it.diff --git a/docs/deep-dives-guide.md b/docs/deep-dives-guide.mdnew file mode 100644index 000000000..6e0ad662f--- /dev/null+++ b/docs/deep-dives-guide.md@@ -0,0 +1,1072 @@+# Repowise Deep Dives — Complete Guide++This document covers systems that are referenced but not fully explained in `architecture-guide.md` and `graph-algorithms-guide.md`. Each section is self-contained with full intuition, implementation details, and the math behind the algorithms.++---++## Table of Contents++1. [Dead Code Detection](#1-dead-code-detection)+2. [Decision Records (ADR) System](#2-decision-records-adr-system)+3. [Search and Vector Store Internals](#3-search-and-vector-store-internals)+4. [Incremental Updates and Webhooks](#4-incremental-updates-and-webhooks)+5. [Change Cascade Algorithm](#5-change-cascade-algorithm)++---++## 1. Dead Code Detection++### What problem does it solve?++Every codebase accumulates files and functions that nothing uses anymore. A refactor removes the last caller of `old_parser.py` but nobody deletes the file. Over months, these dead files pile up — increasing maintenance burden, confusing new developers, and inflating CI times.++Repowise's dead code analyzer finds these automatically using **pure graph traversal + git metadata**. No LLM calls. Runs in under 10 seconds.++### The four detection strategies++#### Strategy 1: Unreachable Files++**Question:** "Is anything importing this file?"++**Algorithm:**++```+For each node in the dependency graph:+ Skip if: external package, non-code language, entry point, test file,+ fixture directory, __init__.py, config file, migration, etc.++ if in_degree(node) == 0:+ This file has zero importers → candidate for dead code+```++`in_degree` is just the count of incoming edges — files that import this one. If nobody imports it and it's not an entry point or test, it's suspicious.++**Why in_degree alone isn't enough:**++Consider `plugin_auth.py`. Nothing imports it directly because the plugin framework loads it dynamically at runtime via `importlib.import_module()`. The graph doesn't capture dynamic imports because they don't appear in the AST as static import statements.++Repowise handles this with multiple layers of filtering:++**Layer 1 — Structural exclusions** (never flagged):++| Exclusion | Why |+|-----------|-----|+| Entry points (`main.py`, `index.ts`, `app.py`) | They're where execution starts, nothing imports them |+| Test files | Tests import production code, not the other way around |+| `__init__.py` | Package initializers, loaded by Python automatically |+| Config files (`setup.py`, `next.config.js`, `vite.config.ts`) | Loaded by frameworks |+| Migrations (`*migrations*`) | Run by migration tools, not imported |+| Schema/seed files | Data definitions, not code |+| Fixture directories (`fixtures/`, `testdata/`, `sample_repo/`) | Test data, not production code |+| Non-code languages (JSON, YAML, Markdown, SQL, Terraform) | No import semantics |+| API contracts (proto, graphql marked `is_api_contract`) | Consumed by code generators |++**Layer 2 — Dynamic pattern matching:**++```python+_DEFAULT_DYNAMIC_PATTERNS = (+ "*Plugin", # Plugin discovery systems+ "*Handler", # Event handler registration+ "*Adapter", # Adapter patterns+ "*Middleware", # Middleware chains+ "register_*", # Registration functions+ "on_*", # Event callbacks+)+```++Files matching these patterns aren't marked `safe_to_delete` even if confidence is high, because they're likely loaded dynamically.++**Layer 3 — Confidence scoring with git metadata:**++This is where it gets interesting. A file with zero importers might be dead, or it might be actively used via dynamic loading. Git history helps distinguish:++```+if no_commits_in_90_days AND file_older_than_180_days:+ confidence = 1.0 # Almost certainly dead+ reasoning: Nobody imports it AND nobody has touched it in 6 months++elif no_commits_in_90_days:+ confidence = 0.7 # Probably dead+ reasoning: Nobody imports it AND no recent activity, but file isn't ancient++else: # has recent commits+ confidence = 0.4 # Suspicious but uncertain+ reasoning: Nobody imports it BUT someone is actively changing it+ (maybe dynamically loaded, maybe a script run manually)+```++**Intuition:** If a file is truly dead, it stops receiving commits. Active files — even dynamically loaded ones — still get bug fixes and updates. The combination of in_degree=0 (structural signal) and no-recent-commits (behavioral signal) gives high confidence.++**safe_to_delete computation:**++```+safe_to_delete = (confidence >= 0.7) AND (not matches_dynamic_patterns)+```++A file is only marked safe to delete if we're confident it's dead AND it doesn't look like a plugin/handler/adapter that might be dynamically loaded.++#### Strategy 2: Unused Exports++**Question:** "Is this public function/class imported by anything?"++This is more granular than unreachable files. A file might be imported, but specific exports within it might be unused.++**Algorithm:**++```+For each file in the graph:+ Skip if: external, non-code, test, fixture, never-flag pattern++ For each PUBLIC symbol in the file:+ Skip if: has framework decorator (pytest.fixture, pytest.mark)+ Skip if: matches dynamic pattern (*Handler, register_*, etc.)++ has_importers = False+ For each file that imports this file (predecessors):+ Check edge's imported_names list+ if symbol_name in imported_names OR "*" in imported_names:+ has_importers = True+ break++ if not has_importers:+ → This export is unused+```++**Edge data is key here.** Each edge in the graph stores `imported_names` — the specific names imported across that edge. For example:++```python+# In service.py:+from auth import login, validate_token++# Edge: service.py → auth.py, imported_names = ["login", "validate_token"]+```++If `auth.py` also exports `reset_password` but no edge's `imported_names` includes it, then `reset_password` is an unused export.++**Confidence scoring for unused exports:**++```+if symbol_name ends with _DEPRECATED, _LEGACY, or _COMPAT:+ confidence = 0.3 # Already marked as legacy by developer++elif file_has_other_importers: # file is used, but this symbol isn't+ confidence = 1.0 # Very suspicious — file is active but this export isn't++else: # file itself has no importers+ confidence = 0.7 # File and symbol both unused+```++**Why the file-imported distinction matters:**++If `auth.py` is imported by 10 files but none of them import `reset_password`, that's a strong signal — developers actively use this file but skip this function. Confidence = 1.0.++If `auth.py` itself has zero importers, the unused export is less interesting — the whole file is dead, and the unused export is just a consequence. Confidence = 0.7.++**safe_to_delete for exports:**++```+safe = (confidence >= 0.7) AND (complexity_estimate < 5)+```++Low-complexity symbols (simple functions, constants) are safer to remove than complex ones that might have non-obvious side effects.++#### Strategy 3: Unused Internals++**Question:** "Is this private function called within its own file?"++**Status: Not implemented** (returns empty list). The comment says "Higher false positive rate — off by default."++**Why it's hard:** Private functions might be called via string dispatch, decorators, metaclasses, or closures that the AST parser doesn't trace. False positives here are more disruptive than for public symbols because developers expect private functions to be internal and are less likely to question the analyzer.++#### Strategy 4: Zombie Packages++**Question:** "Does any other package in this monorepo actually use this package?"++**Algorithm:**++```+# Group files by top-level directory (= package)+packages = group_by(all_files, first_path_segment)++# Only applies to monorepos (2+ packages)+if len(packages) < 2: return []++For each package:+ has_external_importers = False+ For each file in this package:+ For each predecessor (file importing this one):+ if predecessor is from a DIFFERENT package:+ has_external_importers = True+ break++ if not has_external_importers:+ → This package is a zombie (nothing outside it uses it)+```++**Example:**++```+packages/+├── auth/ # imported by api/ and cli/+│ ├── login.py+│ └── jwt.py+├── api/ # imported by cli/+│ └── routes.py+├── cli/ # entry point, imports auth/ and api/+│ └── main.py+└── legacy-reports/ # NOTHING outside this package imports it+ ├── generator.py+ └── formatter.py+```++`legacy-reports/` is a zombie package. Its files might import each other internally, but no other package depends on it.++**Confidence:** Always 0.5 (medium). Packages might be used as standalone entry points, scripts, or tooling that the dependency graph doesn't capture.++**safe_to_delete:** Always `False`. Deleting an entire package is too risky for automatic recommendation.++### How findings flow to the user++```+DeadCodeAnalyzer.analyze()+ │+ ├── _detect_unreachable_files() → findings with confidence + safe_to_delete+ ├── _detect_unused_exports() → findings with confidence + safe_to_delete+ ├── _detect_zombie_packages() → findings at 0.5 confidence, never safe+ │+ ▼ apply min_confidence filter (default 0.4)+ │+ ▼ persist to dead_code_findings table+ │+ ├── REST API: /api/dead-code (filter by kind, confidence, status)+ ├── MCP Tool: get_dead_code (grouped into tiers: high/medium/low)+ └── Web UI: tabbed view with bulk resolve/acknowledge/false-positive+```++**The MCP tool groups findings into three action tiers:**++| Tier | Confidence | Action |+|------|-----------|--------|+| **High** (>= 0.8) | Almost certainly dead | Start here. Safe quick wins. |+| **Medium** (0.5 - 0.8) | Probably dead | Review with team before deleting. |+| **Low** (< 0.5) | Suspicious | Investigate — might be dynamic loading. |++### Incremental dead code analysis++When files change (via `repowise update`), full re-analysis is wasteful. `analyze_partial()` only checks affected files:++```python+def analyze_partial(affected_files, config):+ for node in affected_files:+ if node not in graph: continue+ if in_degree(node) == 0 and not entry_point and not test:+ → Check if this file became unreachable due to the change+```++This is O(affected_files) instead of O(all_files). Only detects newly-unreachable files — it doesn't re-check unused exports or zombie packages (those need the full graph).++---++## 2. Decision Records (ADR) System++### What problem does it solve?++Code tells you **what** the system does. Comments sometimes tell you **how**. But almost nothing tells you **why** — why was this approach chosen over alternatives? What constraints forced this design? What was the tradeoff?++When those decisions live only in someone's head or a forgotten Slack thread, every new developer has to reverse-engineer the reasoning. Worse, they might unknowingly undo a deliberate tradeoff, reintroducing a problem that was already solved.++Repowise's decision system **automatically discovers architectural decisions** from four sources, stores them as structured records, tracks their staleness as code evolves, and surfaces them when developers need context.++### How decisions are discovered++#### Source 1: Inline Markers (confidence 0.95)++Developers sometimes leave breadcrumbs in code:++```python+# WHY: We use bcrypt instead of argon2 because our deployment target+# doesn't have the argon2 C bindings available.+password_hash = bcrypt.hashpw(password, bcrypt.gensalt())++# DECISION: Rate limiting is done at the application layer, not the+# load balancer, because we need per-user limits, not per-IP.+```++The extractor scans source files for regex markers:++```+# WHY: ...+# DECISION: ...+# TRADEOFF: ...+# ADR: ...+# RATIONALE: ...+# REJECTED: ...+```++When found, it captures a ±20-line context window around the marker and sends it to the LLM for structuring into a decision record (title, context, decision, rationale, alternatives, consequences).++**Why 0.95 confidence?** The developer explicitly wrote a decision marker. The intent is unambiguous. 0.95 instead of 1.0 because the LLM structuring might misinterpret the context.++#### Source 2: Git Archaeology (confidence 0.70-0.85)++Most decisions aren't marked in code. They're implicit in commit messages:++```+commit abc123: "Migrate from REST to GraphQL for the admin API — reduces+round trips from 12 to 3 for the dashboard view"++commit def456: "Switch from moment.js to date-fns — moment is 300KB,+date-fns is 15KB with tree-shaking"+```++The extractor scores commits by **decision signal keywords**:++```+"migrate", "switch to", "replace", "refactor to", "deprecate",+"remove", "adopt", "introduce", "upgrade", "rewrite", "extract",+"split", "convert", "transition", "revert"+```++Commits with these keywords are batched (groups of 5) and sent to the LLM to identify which ones represent actual architectural decisions vs routine changes.++**Why variable confidence (0.70-0.85)?** Git commits are noisier than inline markers. A commit saying "migrate database" could be a major architectural decision or just a routine migration script. The LLM assesses this and assigns confidence.++#### Source 3: README Mining (confidence 0.60)++Documentation files often contain architectural rationale:++```markdown+## Architecture++We use a message queue between the API and worker services because...++### Why SQLite?++For single-tenant deployments, PostgreSQL is overkill. SQLite gives us...+```++The extractor processes README.md, ARCHITECTURE.md, CONTRIBUTING.md, DESIGN.md, DECISIONS.md, and `docs/*.md` (up to 10 files, 50KB each).++**Why 0.60 confidence?** README content is often aspirational or outdated. It describes what the code *should* be, not necessarily what it *is* today. Lower confidence reflects this uncertainty.++#### Source 4: CLI Capture (confidence 1.0)++```bash+repowise decision add+```++Interactive prompt for manual entry. The developer directly states the decision — no extraction uncertainty.++**All four sources run in parallel** via `asyncio.gather()`. If one source fails (e.g., LLM timeout during git archaeology), the others still complete.++### Decision data model++Each decision record stores:++```+DecisionRecord+├── title: "Migrate admin API from REST to GraphQL"+├── status: "active" | "proposed" | "deprecated" | "superseded"+├── context: "Dashboard required 12 API calls to render..."+├── decision: "Use GraphQL for the admin API"+├── rationale: "Reduces round trips from 12 to 3..."+├── alternatives: ["Keep REST with batching", "Use gRPC"]+├── consequences: ["Need GraphQL schema maintenance", "Client complexity increases"]+├── affected_files: ["src/admin/schema.py", "src/admin/resolvers.py"]+├── affected_modules: ["admin"]+├── tags: ["api", "performance"]+├── source: "git_archaeology"+├── evidence_file: "src/admin/schema.py"+├── evidence_commits: ["abc123"]+├── confidence: 0.80+├── staleness_score: 0.15+└── superseded_by: null+```++**Deduplication key:** `(repository_id, title, source, evidence_file)`. The same decision discovered from two sources (e.g., inline marker + readme mention) creates separate records intentionally — this preserves provenance and lets you see where each piece of evidence came from.++### Staleness computation++Decisions go stale when the code they govern changes but the decision itself doesn't get updated. The staleness algorithm detects this drift.++**Per-file score:**++For each file in the decision's `affected_files`:++```+if file has no git metadata:+ file_score = 1.0 (can't verify → assume stale)++elif file's last commit is BEFORE the decision was created:+ file_score = 0.0 (file hasn't changed since decision was made → still fresh)++else: # file changed AFTER the decision+ base = min(1.0, (commit_count_90d / 15) × 0.7+ + (age_days / 365) × 0.3)++ conflict_boost = 0.0+ For each significant commit AFTER the decision:+ if commit message contains conflict keywords:+ ("replace", "remove", "deprecate", "migrate away",+ "drop", "revert", "undo", "disable", "eliminate")+ AND shares 2+ meaningful words with decision text:+ conflict_boost = 0.3++ file_score = min(1.0, base + conflict_boost)+```++**Breaking down the base score:**++- **70% weight on recent activity:** `commit_count_90d / 15`. If 15+ commits in 90 days, this maxes out at 1.0. Files with heavy churn since the decision was made are likely to have drifted from the original intent.++- **30% weight on age:** `age_days / 365`. Decisions older than a year get a staleness penalty simply because codebases evolve. Even without heavy churn, a year-old decision might not reflect current reality.++**The conflict boost:**++The most interesting part. If a commit message after the decision contains words like "replace", "remove", "deprecate" AND shares meaningful words with the decision text itself, that's a strong signal that someone is actively working against the decision.++**Example:**++```+Decision: "Use bcrypt for password hashing" (created 2025-06-01)+Commit (2026-01-15): "Replace bcrypt with argon2 for password hashing"++The commit contains "replace" (conflict keyword) and shares "bcrypt",+"password", "hashing" with the decision text.+→ conflict_boost = 0.3+→ This decision is likely stale (someone replaced what it decided)+```++**Aggregate score:** Average across all affected files, rounded to 3 decimals.++**Interpretation:**+- **0.0 - 0.3:** Fresh. Code hasn't materially changed since the decision.+- **0.3 - 0.5:** Moderate. Some drift, worth a review.+- **0.5 - 1.0:** Stale. High churn and/or explicit contradictory commits.++### Ungoverned hotspot detection++**Question:** "Which files change a lot but have no documented decisions explaining why?"++```+hotspot_files = files where churn_percentile >= 0.75 AND commit_count_90d > 0++governed_files = union of all affected_files across active decisions++ungoverned_hotspots = hotspot_files - governed_files+```++These are the most dangerous files in the codebase: they change frequently (risky) and nobody has documented why they're designed the way they are (opaque). New developers are most likely to introduce bugs here.++### Alignment scoring++When you query `get_why("src/auth/login.py")`, the system computes an **alignment score** — how well-governed is this file?++**Algorithm:**++```+1. Find all decisions governing this file+ (file in affected_files OR module in affected_modules)++2. If no decisions → score = "none"+ "This file is ungoverned — no documented rationale."++3. Count statuses:+ active_count = decisions with status "active"+ deprecated_count = decisions with status "deprecated" or "superseded"+ stale_count = decisions with staleness_score > 0.5+ proposed_count = decisions with status "proposed"++4. Compute sibling coverage:+ sibling_files = other files in the same directory+ sibling_decisions = decisions governing siblings+ coverage = |shared_decisions| / |sibling_decisions|++5. Score decision tree:+ ┌─ All deprecated, no active → "low" (technical debt)+ ├─ >= 50% stale → "low" (rationale may be invalid)+ ├─ Has active + sibling_coverage >= 0.5 → "high" (well-governed)+ ├─ Has active + sibling_coverage < 0.5 → "medium" (unique pattern)+ ├─ Has active, no siblings → "high"+ ├─ Only proposed → "medium" (unreviewed)+ └─ Mixed → "medium"+```++**Why sibling coverage matters:**++If `auth/login.py` is governed by a decision about "Use JWT for authentication" and its sibling `auth/jwt.py` is also governed by the same decision, that's a well-structured module where files share consistent architectural direction. Coverage >= 50% → "high" alignment.++If `auth/login.py` has a unique decision that no sibling shares, it might be an outlier — the decision applies narrowly, or the file doesn't fit the module's pattern. Coverage < 50% → "medium."++### Origin story++When you look up a file's decisions, the system also builds an **origin story** — a narrative reconstruction of how this file came to be:++```+Origin Story for src/auth/login.py:++Created: 2024-03-15 (732 days ago)+Created by: Alice Chen (47% of commits)+Last change: 2026-03-20 by Bob Kim+Commits: 89 total, 12 in last 90 days++Key commits:+ - abc123 (2024-03-15): "Initial auth module with JWT" → [Alice]+ - def456 (2024-08-22): "Migrate from session cookies to JWT" → [Alice]+ - ghi789 (2025-11-03): "Add MFA support to login flow" → [Bob]++Linked decisions:+ - "Use JWT for authentication" (active, confidence 0.85)+ Evidence commits: abc123, def456 (messages share "JWT" keyword)+ - "Add multi-factor authentication" (active, confidence 0.75)+ Evidence commits: ghi789 (message shares "MFA" keyword)+```++**Commit-decision linkage** works by keyword overlap: if a commit message shares 2+ meaningful words (after removing stop words) with a decision's text, they're linked as evidence.++### Decision lifecycle++```+proposed → active → deprecated+ ↓ ↓+ superseded ← superseded_by link++CLI commands:+ repowise decision add → creates with status "active"+ repowise decision confirm → proposed → active+ repowise decision deprecate → sets status "deprecated"+ repowise decision dismiss → deletes a proposed decision+ repowise decision health → shows stale, ungoverned, proposed+```++---++## 3. Search and Vector Store Internals++### The problem with keyword search++If you search for "authentication" with keyword matching, you find pages containing the word "authentication." But you miss pages about "login flow", "credential validation", or "session management" — concepts that are semantically identical but use different words.++Vector search solves this by comparing **meaning**, not characters.++### How vector search works — from text to numbers++**Step 1: Embedding.** Convert text to a vector (list of numbers):++```+"authentication module" → [0.12, -0.45, 0.78, ..., 0.03] (1536 numbers)+"login credential check" → [0.11, -0.43, 0.80, ..., 0.05] (1536 numbers)+"database connection" → [0.67, 0.22, -0.15, ..., 0.91] (1536 numbers)+```++The embedding model (OpenAI, Gemini, or mock) maps semantically similar text to nearby vectors. "Authentication" and "login" end up close together. "Database connection" ends up far away.++**Step 2: Normalize.** All vectors are L2-normalized to unit length:++```+normalized = vector / ||vector||++where ||vector|| = sqrt(v[0]² + v[1]² + ... + v[n]²)+```++After normalization, every vector has length 1.0. This is crucial because it makes cosine similarity equal to the dot product, which is cheaper to compute:++```+cosine_similarity(a, b) = (a · b) / (||a|| × ||b||)++If ||a|| = 1 and ||b|| = 1, then:+cosine_similarity(a, b) = a · b = Σ(a[i] × b[i])+```++**Step 3: Store.** Save each vector alongside its page_id and metadata.++**Step 4: Search.** Embed the query, compute similarity against all stored vectors, return top-k.++### The three vector store implementations++#### InMemoryVectorStore++Simplest implementation. Stores vectors in a Python dict:++```python+_store: dict[page_id] → (vector, metadata)+```++Search computes cosine similarity against every vector:++```python+def _cosine(a, b):+ dot = sum(x * y for x, y in zip(a, b))+ norm_a = sqrt(sum(x * x for x in a))+ norm_b = sqrt(sum(x * x for x in b))+ return dot / (norm_a * norm_b) if (norm_a * norm_b) > 0 else 0.0+```++**Time complexity:** O(N × D) per search, where N = number of pages, D = vector dimensions.++For 55 pages with 1536 dimensions, this is ~84,000 multiplications per search — trivial. For 100,000 pages, you'd want something smarter. That's where LanceDB comes in.++#### LanceDBVectorStore++Embedded vector database stored as local files in `.repowise/lancedb/`. Uses Apache Arrow columnar format with an IVF-PQ (Inverted File Index with Product Quantization) index for fast approximate nearest neighbor search.++**Schema:**++```+page_id: string+vector: list<float32>[dim]+title: string+page_type: string+target_path: string+content_snippet: string (first 200 chars)+```++**Upsert strategy:**++```python+# LanceDB 0.12+: atomic merge_insert+table.merge_insert("page_id")+ .when_matched_update_all() # existing page → update vector + metadata+ .when_not_matched_insert_all() # new page → insert+ .execute([row])++# Fallback for older LanceDB: two operations+table.delete(f"page_id = '{safe_id}'") # delete old+table.add([row]) # insert new+```++**Why merge_insert matters:** Without it, there's a window between delete and add where the page doesn't exist. If a search runs during that window, it misses the page. merge_insert is atomic — the old and new versions are swapped in one operation.++#### PgVectorStore++Uses PostgreSQL's pgvector extension. Stores embeddings directly in the `wiki_pages` table:++```sql+-- Upsert+UPDATE wiki_pages SET embedding = CAST('[0.12,-0.45,...]' AS vector) WHERE id = 'page_id';++-- Search (cosine distance operator <=>)+SELECT id, title, content, page_type, target_path,+ 1 - (embedding <=> CAST('[0.11,-0.43,...]' AS vector)) AS score+FROM wiki_pages+WHERE embedding IS NOT NULL+ORDER BY embedding <=> CAST('[0.11,-0.43,...]' AS vector)+LIMIT 10;+```++**The `<=>` operator** computes cosine distance (0 = identical, 2 = opposite). Subtracting from 1 converts to similarity.++**Why raw SQL instead of ORM?** The `embedding` column is a pgvector type, which isn't declared in the SQLAlchemy ORM model. This keeps the models dialect-neutral (they work with both SQLite and PostgreSQL). The pgvector column is added by an Alembic migration that only runs on PostgreSQL.++### Full-text search (FTS)++Vector search is powerful but slow (needs embeddings, API calls). Full-text search is fast and works with exact keywords.++#### SQLite FTS5++**Index creation:**++```sql+CREATE VIRTUAL TABLE page_fts USING fts5(+ page_id UNINDEXED, -- stored but not searchable+ title, -- searchable+ content -- searchable+);+```++**Query construction:**++The user's query is transformed into an FTS5 MATCH expression:++```+Input: "Python decorator pattern"++Step 1: Tokenize → ["python", "decorator", "pattern"]+Step 2: Remove stop words → ["python", "decorator", "pattern"] (none removed)+Step 3: Add prefix matching → "python"* OR "decorator"* OR "pattern"*+```++The `*` suffix enables prefix matching: `"auth"*` matches "authentication", "authorization", "authoring".++OR between terms gives **broad recall** — a page matching any term is returned. FTS5's built-in BM25 ranking naturally boosts pages matching **more** terms.++**Another example:**++```+Input: "the async await system"++Step 1: Tokenize → ["the", "async", "await", "system"]+Step 2: Remove stop words → ["async", "await", "system"] (removes "the")+Step 3: → "async"* OR "await"* OR "system"*+```++127 stop words are removed (a, an, the, is, are, was, were, be, been, have, has, had, do, does, did, will, would, etc.).++**Edge case:** If all words are stop words (e.g., "a the is"), the query falls back to exact phrase matching: `"a the is"`.++#### PostgreSQL++Uses `to_tsvector('english', ...)` for document representation and `plainto_tsquery('english', ...)` for queries. PostgreSQL handles stemming (running → run), stop word removal, and ranking via `ts_rank()`.++A GIN index makes searches fast without scanning every row.++### Search ranking in the MCP tool++The MCP `search_codebase` tool applies additional ranking on top of raw search scores:++**Step 1: Try semantic search** (vector store, 8-second timeout)++**Step 2: Fallback to FTS** if semantic fails or returns empty++**Step 3: Freshness boost** — recently modified files rank higher:++```+if file has commits in last 30 days: recency = 1.0+elif file has commits in last 90 days: recency = 0.5+else: recency = 0.0++boosted_score = raw_score × (1 + 0.2 × recency)+```++A file modified yesterday gets a 20% boost. A file untouched for a year gets no boost.++**Why boost freshness?** If you're searching for "authentication" and two pages match equally, the one that was recently updated is more likely to be accurate and relevant to current development.++**Step 4: Normalize confidence** relative to the best result:++```+confidence = relevance_score / max_relevance_score+```++The top result gets confidence ≈ 1.0. Other results are proportionally lower. This gives the user a relative quality signal without needing to interpret raw cosine similarity values.++---++## 4. Incremental Updates and Webhooks++### The problem++Initial documentation generation is expensive — potentially hundreds of LLM calls. After that, you want to keep docs fresh as code changes, but regenerating everything on every commit is wasteful and slow.++Repowise's incremental update system regenerates **only what changed** and its dependencies.++### Three triggers++#### Trigger 1: CLI update command++```bash+repowise update+```++Reads `.repowise/state.json` to find the last synced commit, diffs against current HEAD, and regenerates affected pages.++#### Trigger 2: Filesystem watcher++```bash+repowise watch --debounce 2000+```++Uses `watchdog` library to monitor the filesystem. When files change:++1. Add changed paths to a set+2. Start a debounce timer (default 2 seconds)+3. If more changes arrive, reset the timer+4. When the timer fires (filesystem quiet for 2 seconds), run `repowise update`++**Why debounce?** Saving a file in an editor often triggers multiple filesystem events (write, metadata change, backup file creation). Without debouncing, each save would trigger 3-4 redundant update runs. The 2-second quiet period waits until all save-related events settle.++#### Trigger 3: Webhooks++GitHub and GitLab can send POST requests when code is pushed:++**GitHub verification:**++```+Webhook arrives with:+ Body: {...}+ Header: X-Hub-Signature-256: sha256=abc123...++Server computes:+ expected = HMAC-SHA256(secret, body)++Verification:+ hmac.compare_digest(expected, received) # constant-time comparison+```++`compare_digest` is critical for security — it takes the same time regardless of where the strings differ, preventing timing attacks that could leak the secret byte by byte.++**GitLab verification:**++```+Header: X-Gitlab-Token: my-secret-token+Compare: hmac.compare_digest(expected_token, received_token)+```++Simpler — just a shared token, no HMAC computation.++**After verification:**++```+1. Store raw webhook event in database (audit trail)+2. Find matching repository by URL+3. Create GenerationJob:+ - status: "pending"+ - mode: "incremental"+ - config: {before: "old_commit_sha", after: "new_commit_sha"}+4. Link webhook event to job+```++### The update pipeline++```+1. LOAD STATE+ Read last_sync_commit from .repowise/state.json+ │+2. DIFF+ ChangeDetector.get_changed_files(last_sync_commit, HEAD)+ → List of FileDiff objects (added, deleted, modified, renamed)+ │+3. RE-INGEST+ Re-run FileTraverser + ASTParser + GraphBuilder on entire repo+ (Need fresh graph to compute cascades correctly)+ │+4. RE-INDEX GIT+ GitIndexer.index_changed_files() — only for changed files+ Update churn percentiles, hotspot flags+ │+5. DETECT DECISIONS+ Scan changed files for inline decision markers (WHY:, DECISION:, etc.)+ Update decision staleness scores+ │+6. CASCADE ANALYSIS+ ChangeDetector.get_affected_pages(file_diffs, graph, cascade_budget)+ → regenerate: pages to fully regenerate (budget-limited)+ → rename_patch: pages with symbol renames (text replacement)+ → decay_only: pages to mark stale without regeneration+ │+7. GENERATE+ PageGenerator.generate_all() with only affected files+ │+8. PERSIST+ Upsert pages, git metadata, decisions+ Update FTS index, vector store+ Update CLAUDE.md if enabled+ │+9. SAVE STATE+ Write current HEAD to .repowise/state.json+```++### SSE progress streaming++Long-running jobs report progress via Server-Sent Events:++```+Client: GET /api/jobs/{job_id}/stream+ Accept: text/event-stream++Server: (every 1 second)+ event: progress+ data: {"job_id": "abc", "status": "running", "completed_pages": 12, "total_pages": 55}++ event: progress+ data: {"job_id": "abc", "status": "running", "completed_pages": 25, "total_pages": 55}++ ...++ event: done+ data: {"job_id": "abc", "status": "completed", "completed_pages": 55, "total_pages": 55}+```++The server checks for client disconnection each iteration and stops streaming when the client goes away. Headers include `Cache-Control: no-cache` and `X-Accel-Buffering: no` (prevents nginx from buffering the stream).++### Scheduler polling fallback++Webhooks can fail (network issues, misconfiguration, GitHub outages). The scheduler runs two background jobs every 15 minutes:++**Job 1: Staleness checker** — finds stale/expired pages across all repos and logs them.++**Job 2: Polling fallback** — for each local repo, compares the stored HEAD commit against actual `git rev-parse HEAD`. If they differ, a webhook was missed. (Currently logs only; full auto-sync is future work.)++---++## 5. Change Cascade Algorithm++### The problem++Changing `utils.py` doesn't just affect `utils.py`'s documentation. Every file that imports `utils.py` might now have incorrect documentation too — it might reference old function names, outdated behavior, or changed signatures.++But propagating changes through the entire dependency graph could trigger hundreds of regenerations. You need a smart cascade with a budget.++### The algorithm++```+Input:+ file_diffs: list of changed files with their old/new parsed versions+ graph: dependency graph (directed, import edges)+ cascade_budget: max pages to fully regenerate (adaptive, hard cap 50)++Output:+ regenerate: set of page IDs for full LLM regeneration+ rename_patch: set of page IDs needing text replacement (for renames)+ decay_only: set of page IDs to mark stale without regeneration+```++**Step 1: Direct changes**++```+directly_changed = {file.path for file in file_diffs}+```++These always get regenerated (they changed, their docs are definitely wrong).++**Step 2: 1-hop cascade (reverse dependencies)**++```+one_hop = set()+for file in directly_changed:+ for predecessor in graph.predecessors(file):+ # predecessor imports file → predecessor's docs may reference file+ one_hop.add(predecessor)+one_hop -= directly_changed # don't double-count+```++**Example:**++```+auth.py changed+graph.predecessors("auth.py") = ["main.py", "middleware.py", "api.py"]++one_hop = {"main.py", "middleware.py", "api.py"}+```++These files import `auth.py`. If `auth.py` renamed a function, their documentation might reference the old name.++**Step 3: Symbol rename detection**++For each changed file, compare old and new parsed versions to detect renames:++```+Old file symbols: [calculate, Config, validate]+New file symbols: [compute, Config, validate]++"calculate" removed, "compute" added, both are functions+Name similarity = SequenceMatcher("calculate", "compute").ratio() = 0.63+Line proximity = start lines within ±5 → bonus = 0.2+Combined = 0.63 + 0.2 = 0.83 (above threshold 0.65)++→ Detected rename: calculate → compute+```++Files referencing the old name need a text patch (string replacement in the existing doc) rather than a full regeneration.++**Step 4: Co-change decay**++```+co_change = set()+for file in directly_changed:+ for partner in graph edges with edge_type="co_changes":+ if partner is file's co-change partner:+ co_change.add(partner)+co_change -= directly_changed+co_change -= one_hop+```++Files that frequently change alongside the modified files are marked for decay — their docs might be stale but the evidence is weaker (correlation, not causation).++**Step 5: 2-hop weak cascade (for renames only)**++```+two_hop = set()+for file in one_hop:+ if file has symbol renames:+ for predecessor in graph.predecessors(file):+ two_hop.add(predecessor)+two_hop -= directly_changed+two_hop -= one_hop+```++If a rename propagated to a 1-hop file, files importing that 1-hop file might also need updating. But this is speculative — marked for decay only.++**Step 6: Budget application**++```+candidates = directly_changed ∪ one_hop+sorted_by_pagerank = sort(candidates, key=pagerank, descending)++regenerate = sorted_by_pagerank[:cascade_budget]+decay_only = sorted_by_pagerank[cascade_budget:] ∪ two_hop ∪ co_change+```++**Why sort by PageRank?** If the budget is 50 and there are 80 candidates, you want to regenerate the 50 most important files first. A high-PageRank file is imported by many others — if its documentation is wrong, the error propagates further. Low-PageRank leaf files can wait.++**Adaptive budget:** The budget is no longer fixed. `compute_adaptive_budget()` scales it based on change magnitude:++| Files changed | Budget |+|---------------|--------|+| 0 | 0 |+| 1 | 10 |+| 2-5 | 30 |+| 6+ | min(n × 3, 50) |++Hard cap at 50. Users can override with `--cascade-budget N`.++### Worked example++```+Codebase: 200 files, adaptive cascade_budget = 30 (3 files changed)++Developer changes: config.py++graph.predecessors("config.py") = [+ "auth.py", "api.py", "db.py", "cache.py", "logger.py",+ "middleware.py", "scheduler.py", "worker.py", "mailer.py",+ "validator.py", "serializer.py", "router.py"+] # 12 files++directly_changed = {"config.py"} # 1 file+one_hop = {12 files above} # 12 files+candidates = 1 + 12 = 13 files (under budget of 30)++config.py renamed: LOG_LEVEL → LOGGING_LEVEL+→ rename_patch candidates: files importing LOG_LEVEL++co_change partners of config.py: ["docker-compose.yml", ".env.example"]+→ decay_only++Result:+ regenerate: 13 pages (all fit within budget)+ rename_patch: pages referencing LOG_LEVEL+ decay_only: docker-compose.yml, .env.example docs+```++Now imagine `config.py` is imported by 80 files:++```+candidates = 1 + 80 = 81 files (over budget of 30)++Sort by PageRank:+ Top 30: main.py, auth.py, api.py, ... (highest PageRank)+ Remaining 51: leaf files, utilities, tests++Result:+ regenerate: 30 pages (budget-limited, most important first)+ decay_only: 51 pages (confidence decayed, regenerated on next run)+```++The 51 files that didn't make the cut have their confidence score reduced. They'll show as "stale" in the UI and be prioritized for regeneration on the next update.++### Confidence decay++Pages in `decay_only` don't get regenerated but their freshness score changes:++```+confidence decays linearly:+ 1.0 at generation time → 0.0 after expiry_threshold_days (default 30)++freshness_status:+ "fresh" if hash matches AND age < 7 days+ "stale" if hash changed OR age >= 7 days+ "expired" if age >= 30 days (forces regeneration on next run)+```++This creates a natural queue: pages that got bumped from the cascade budget eventually hit "expired" status and get regenerated in a future update cycle, ensuring nothing stays stale forever.diff --git a/docs/graph-algorithms-guide.md b/docs/graph-algorithms-guide.mdnew file mode 100644index 000000000..7b55e7e79--- /dev/null+++ b/docs/graph-algorithms-guide.md@@ -0,0 +1,618 @@+# Graph Algorithms in Repowise — Complete Guide++This document covers every graph algorithm used in Repowise: what it does, the intuition behind it, the actual math, and why Repowise chose it.++---++## The Foundation: What Is the Graph?++Before any algorithm runs, Repowise builds a **directed graph** from your codebase.++- Each **node** is a source file (e.g., `auth/login.py`)+- Each **edge** is an import (e.g., `login.py` imports `utils.py` → edge from `login.py` to `utils.py`)+- Edge direction matters: `A → B` means "A depends on B", not the other way around++There are three types of edges:++| Edge type | Source | Example |+|-----------|--------|---------|+| `imports` (default) | Static import resolution from AST | `from auth import login` |+| `co_changes` | Git co-change analysis | `auth.py` and `config.py` frequently change together |+| `framework` | Framework-aware synthetic edges | pytest conftest→tests, Django admin→models, FastAPI include_router→routers, Flask register_blueprint→blueprints |++Framework edges are detected automatically when the tech stack includes Django, FastAPI, Flask, or pytest. They capture real runtime dependencies that static import resolution misses (e.g., `conftest.py` fixtures are loaded by pytest, not imported directly by test files).++This is the raw material. Every algorithm below operates on this graph.++---++## 1. PageRank++### What question does it answer?++**"How important is this file to the codebase?"**++A file is important if many files depend on it, especially if _those_ files are also important.++### The intuition++Imagine a new developer joins the team. They pick a random file, start reading it, and follow one of its imports to the next file. They keep doing this — open a file, pick a random import, jump there. Over weeks of doing this, some files keep appearing again and again. Those files are the important ones.++PageRank simulates exactly this. It's a model of a random person browsing through the import graph, counting how often they land on each file.++### Why not just count importers?++Consider this graph:++```+main.py ──→ auth.py ──→ crypto.py+app.py ──→ auth.py+cli.py ──→ auth.py+test.py ──→ auth.py+```++`crypto.py` has only 1 importer (`auth.py`). `auth.py` has 4 importers. Simple count says `auth.py` is 4x more important.++But `crypto.py` is imported by `auth.py`, which is itself highly imported. So indirectly, all 4 files that need `auth.py` also transitively need `crypto.py`. A vote from `auth.py` should count more than a vote from `test.py` because `auth.py` is itself important.++PageRank captures this transitive importance. Simple import-counting does not.++### The math++Every file starts with a score of `1/N` where N is the total number of files. Then we iterate:++```+score(B) = (1 - α) / N + α × Σ [ score(A) / out_degree(A) ]+ for each A that imports B+```++Breaking this down:++- **`(1 - α) / N`** — the "teleport" component. Everyone gets this baseline. With α = 0.85, this is `0.15 / N`.+- **`α × Σ [ score(A) / out_degree(A) ]`** — the "follow edges" component. For each file A that imports B, B gets a share of A's score divided by how many files A imports.++**Why divide by `out_degree(A)`?** If `auth.py` imports 10 files, it splits its "vote" evenly across all 10. If it only imports 2 files, each gets a bigger share. A focused file that imports few things gives a stronger signal per import.++**Example iteration:**++Suppose 3 files, all start at score = 1/3 = 0.333:++```+A ──→ B ──→ C+```++Round 1:+- `score(A)` = 0.15/3 + 0 = 0.05 (nobody imports A)+- `score(B)` = 0.15/3 + 0.85 × (0.333/1) = 0.05 + 0.283 = 0.333+- `score(C)` = 0.15/3 + 0.85 × (0.333/1) = 0.05 + 0.283 = 0.333++Round 2 (using new scores):+- `score(A)` = 0.05 + 0 = 0.05+- `score(B)` = 0.05 + 0.85 × (0.05/1) = 0.05 + 0.0425 = 0.0925+- `score(C)` = 0.05 + 0.85 × (0.333/1) = 0.05 + 0.283 = 0.333++After many rounds, scores stabilize. C ends up highest (it's the most depended-on), A is lowest (nothing depends on it), B is in the middle.++### The damping factor α = 0.85++This is the probability of following an import edge vs. teleporting to a random file.++**Why teleport at all?** Two problems without it:++**Problem 1 — Disconnected components.** If your codebase has two isolated clusters with no imports between them, the random walker gets trapped in whichever cluster it starts in. Files in the other cluster get score = 0. That's wrong — they're still important within their cluster.++```+# Cluster A # Cluster B+a1 ──→ a2 ──→ a3 b1 ──→ b2 ──→ b3+ ↑ │ ↑ │+ └────────────┘ └────────────┘++No edges between clusters. Without teleport, starting+in A means you never visit B. B's scores = 0.+```++**Problem 2 — Dead ends.** A file that imports nothing traps the walker. There are no edges to follow.++```+helpers.py ──→ (nothing)+```++The 15% teleport chance solves both: the walker occasionally jumps to a random file, so every file gets visited and scores are always nonzero.++**Why 0.85 specifically?** This is the original value from the Google PageRank paper. It's been validated across decades of graph analysis. Higher values (0.95) give more weight to the actual link structure but take longer to converge and are more sensitive to oddities. Lower values (0.5) spread scores out too evenly, making everything look similar. 0.85 is the standard tradeoff.++### Convergence failure++Sometimes the iterative computation oscillates and doesn't settle on stable scores. Repowise handles this:++```python+try:+ return nx.pagerank(filtered, alpha=alpha)+except nx.PowerIterationFailedConvergence:+ return {node: 1.0 / n for node in filtered.nodes()}+```++If PageRank doesn't converge, every file gets equal score `1/N`. This is safe — no file gets unfairly prioritized.++### Why co-change edges are excluded++Repowise's graph has two edge types:+1. **Import edges**: `auth.py` imports `utils.py` (structural dependency)+2. **Co-change edges**: `auth.py` and `config.py` often change in the same commit (behavioral correlation)++PageRank runs only on import edges. Why?++Co-change is noisy and doesn't indicate dependency. Examples of co-change that would corrupt PageRank:++- A developer renames a constant across 20 files in one commit. None of those files structurally depend on each other through that rename.+- Every bug fix touches `handler.py` and `test_handler.py`. The test file isn't architecturally important just because it changes alongside the handler.+- A release process updates `CHANGELOG.md`, `version.py`, and `setup.cfg` together. None import each other.++If co-change edges fed into PageRank, files that happen to change alongside many others would appear "important" even when nothing imports them. PageRank should answer "if this file breaks, what else breaks?" — that's a structural question, answered only by import edges.++### How Repowise uses PageRank++1. **Documentation priority**: High PageRank files get wiki pages generated first. If budget is limited, the most depended-on files are documented.+2. **Generation depth**: Low PageRank + low git churn → "minimal" docs (saves LLM tokens and cost).+3. **Significant file selection**: Files above the PageRank threshold get detailed Level 2 wiki pages. Files below get summarized in module-level pages.+4. **CLAUDE.md generation**: When generating editor context files, files are sorted by PageRank descending — the most important files appear first for AI coding assistants.++---++## 2. Betweenness Centrality++### What question does it answer?++**"How critical is this file as a bridge between different parts of the codebase?"**++A file with high betweenness sits on the shortest paths connecting many pairs of files. If you removed it, parts of the codebase would become more disconnected.++### The intuition++Think of a road network. Some roads are highways that connect neighborhoods. If a highway closes, traffic between those neighborhoods has to take long detours. Other roads are cul-de-sacs — nobody drives through them to get somewhere else.++Betweenness centrality measures which files are the "highways." They may not be the most imported files (that's PageRank), but they're the ones that connect otherwise separate areas.++### How it differs from PageRank++Consider this graph:++```+ ┌── feature_a1.py+input.py ─┤── feature_a2.py ──→ bridge.py ──→ db_query.py ──→ models.py+ └── feature_a3.py ──→ schema.py+```++- `input.py` has high **PageRank** (many things depend on it indirectly)+- `bridge.py` has high **betweenness** (it's the only path connecting the feature cluster to the database cluster)+- `models.py` has high PageRank too (widely imported) but maybe low betweenness if there are other paths++**PageRank says "I'm important." Betweenness says "I'm a bottleneck."**++### The math++For each pair of nodes (s, t) in the graph:+1. Find all shortest paths from s to t+2. Count how many of those shortest paths pass through node v+3. Divide by the total number of shortest paths from s to t++```+betweenness(v) = Σ [ σ_st(v) / σ_st ]+ s≠v≠t++where:+ σ_st = total number of shortest paths from s to t+ σ_st(v) = number of those shortest paths that pass through v+```++Then normalize to [0, 1] by dividing by `(N-1)(N-2)` — the maximum possible pairs.++**Worked example:**++```+A ──→ B ──→ D+A ──→ C ──→ D+```++Shortest paths:+- A→D: two paths (A→B→D and A→C→D), length 2 each+- A→B: one path (A→B)+- A→C: one path (A→C)+- B→D: one path (B→D)+- C→D: one path (C→D)++For node B:+- Path A→D: 2 shortest paths total, 1 passes through B. Contribution: 1/2.+- Path C→D: 1 shortest path, none pass through B. Contribution: 0.+- B doesn't count as intermediate for paths starting/ending at B.++`betweenness(B)` = 1/2 (before normalization)++`betweenness(C)` = 1/2 (symmetric — same structure)++Both B and C are equally "bridge-like" because there are two parallel paths through the graph.++Now if we remove C:+```+A ──→ B ──→ D+```+`betweenness(B)` = 1 — B is the only bridge. All traffic flows through it.++### The complexity problem and sampling++The standard algorithm (Brandes' algorithm) is O(N × E) where N = nodes and E = edges. For a 50,000-file repo with 200,000 edges, that's 10 billion operations.++Repowise solves this with **sampling**:++```python+if n > 30_000: # _LARGE_REPO_THRESHOLD+ k = min(500, n)+ return nx.betweenness_centrality(g, k=k, normalized=True)+```++Instead of computing exact betweenness using all N×N pairs, it samples 500 random source nodes and approximates. Research shows this gives good results — the ranking of "which files are most bridge-like" stays accurate even with sampling, just the exact scores shift slightly.++For repos under 30,000 nodes, exact computation is used.++### How Repowise uses betweenness++1. **Significant file selection**: A file with high betweenness gets Level 2 docs even if its PageRank isn't high. Bridge files deserve documentation because they're coupling points — if someone changes them, ripple effects cross component boundaries.+2. **File page context**: Betweenness is passed to the LLM in the file page template. The LLM can note "this file bridges the auth and database modules" in its generated documentation.+3. **Risk assessment**: High betweenness + high git churn = dangerous file. It's a bottleneck that changes frequently — a maintenance risk.++---++## 3. Strongly Connected Components (SCCs)++### What question does it answer?++**"Which files form circular dependency cycles?"**++### The intuition++In a healthy codebase, dependencies flow in one direction: `main → service → repository → model`. You can always tell which module is "higher" or "lower" in the stack.++A circular dependency breaks this. If `models.py` imports `serializers.py` and `serializers.py` imports `models.py`, there's no clear hierarchy. You can't understand one without the other. You can't test one without the other. You can't deploy one without the other.++An SCC is a maximal group of files where every file can reach every other file by following imports. If A can reach B and B can reach A, they're in the same SCC.++### The math (Tarjan's Algorithm)++The algorithm works via depth-first search (DFS) with two key numbers per node:++1. **Discovery index**: When you first visit a node, stamp it with an incrementing counter+2. **Low-link value**: The smallest discovery index reachable from this node via DFS edges + one back-edge++The algorithm:++```+1. Start DFS from any unvisited node+2. Push each visited node onto a stack+3. For each neighbor:+ - If unvisited: recurse, then update current node's low-link = min(own low-link, child's low-link)+ - If on stack: update current node's low-link = min(own low-link, neighbor's discovery index)+4. After processing all neighbors: if low-link == discovery index, this node is the "root" of an SCC.+ Pop everything from the stack down to this node — that's one SCC.+```++**Worked example:**++```+A ──→ B ──→ C ──→ A (cycle: A, B, C)+ │+ └──→ D ──→ E (no cycle)+```++DFS starting at A:+- Visit A (discovery=0, low=0), push A+- Visit B (discovery=1, low=1), push B+- Visit C (discovery=2, low=2), push C+ - C has edge to A (on stack): C.low = min(2, 0) = 0+ - C has edge to D: visit D+ - Visit D (discovery=3, low=3), push D+ - Visit E (discovery=4, low=4), push E+ - E has no unvisited neighbors+ - E.low == E.discovery → E is an SCC root. Pop E. SCC: {E}+ - D.low == D.discovery → D is an SCC root. Pop D. SCC: {D}+ - Back to C: C.low = 0+- Back to B: B.low = min(1, 0) = 0+- Back to A: A.low = 0, A.low == A.discovery → SCC root. Pop C, B, A. SCC: {A, B, C}++Result: Three SCCs: {A,B,C}, {D}, {E}. Only {A,B,C} has size > 1, meaning it's a circular dependency.++**Time complexity:** O(N + E) — each node and edge is visited exactly once. Very fast.++### What counts as a circular dependency++- **SCC of size 1**: Normal. A file that doesn't import itself. Not a cycle.+- **SCC of size > 1**: Circular dependency. These files form a cycle and can't be separated.++### How Repowise uses SCCs++1. **Dedicated wiki pages**: Each SCC with size > 1 gets its own `scc_page` in the wiki, documenting which files are in the cycle, what they import from each other, and why it's worth addressing.+2. **Architecture diagram**: Circular dependencies are highlighted in the generated architecture overview so the team can see coupling hotspots.+3. **Codebase health signal**: Many or large SCCs indicate tightly coupled code. The repo overview includes a count of circular dependencies.+4. **SCC IDs in node metadata**: Each file is tagged with an `scc_id`. Files in the same SCC share the same ID, which helps the frontend visualize which groups of files are tangled together.++---++## 4. Louvain Community Detection++### What question does it answer?++**"Which files naturally cluster together into subsystems?"**++### The intuition++In any codebase, files tend to cluster. The database layer files import each other heavily. The API route files import each other. But the database layer and the API routes have fewer connections between them.++Community detection finds these clusters automatically. It looks at the density of edges and groups files that are more connected to each other than to the rest of the graph.++Think of it as automated "package detection." Even if your code doesn't use clean package boundaries, community detection reveals the actual subsystem structure from the import graph.++### Why undirected?++Repowise converts the directed graph to undirected before running Louvain:++```python+communities = nx.community.louvain_communities(g.to_undirected(), seed=42)+```++Why? Import direction doesn't matter for clustering. If `auth.py` imports `crypto.py`, they're related — regardless of which one depends on which. The question isn't "who depends on whom" (that's PageRank's job) but "who belongs with whom."++### The math: Modularity++Louvain optimizes a metric called **modularity** (Q). Modularity measures the difference between actual intra-community edges and what you'd expect by random chance.++```+Q = (1/2m) × Σ [ A_ij - (k_i × k_j)/(2m) ] × δ(c_i, c_j)+ i,j++where:+ m = total number of edges+ A_ij = 1 if edge exists between i and j, 0 otherwise+ k_i = degree of node i (number of edges)+ k_j = degree of node j+ δ(c_i, c_j) = 1 if i and j are in the same community, 0 otherwise+```++**What each part means:**++- **`A_ij`**: Does an actual edge exist between file i and file j? (1 = yes, 0 = no)+- **`(k_i × k_j) / (2m)`**: If edges were distributed randomly, what's the probability of an edge between i and j? Files with many connections are more likely to connect by chance.+- **`A_ij - (k_i × k_j)/(2m)`**: The "surprise factor." Positive if there's an edge where you wouldn't expect one. Negative if there's no edge where you'd expect one.+- **`δ(c_i, c_j)`**: Only count pairs in the same community.++So modularity = sum of "surprising connections" within communities. High Q means communities have more internal edges than random chance would predict. That's what we want.++**Q ranges from -0.5 to 1.0.** Real-world values of 0.3 to 0.7 indicate meaningful community structure.++### The Louvain algorithm++The Louvain algorithm maximizes Q using a two-phase greedy approach:++**Phase 1 — Local moves:**+1. Start with each node in its own community (N communities)+2. For each node, compute the modularity gain of moving it to each neighbor's community+3. Move the node to the community that gives the biggest gain (if positive)+4. Repeat until no node wants to move++**The modularity gain formula:**++```+ΔQ = [ (Σ_in + k_i,in) / (2m) - ((Σ_tot + k_i) / (2m))² ]+ - [ (Σ_in / (2m)) - (Σ_tot / (2m))² - (k_i / (2m))² ]++where:+ Σ_in = sum of edge weights inside the target community+ Σ_tot = sum of all edge weights of nodes in the target community+ k_i = degree of node i+ k_i,in = sum of edges from node i to nodes in the target community+```++You don't need to memorize this. The key insight: it's cheap to compute (looks only at the node's neighbors), which is why Louvain is fast.++**Phase 2 — Aggregation:**+1. Collapse each community into a single super-node+2. Sum the edges between communities to create a smaller graph+3. Go back to Phase 1 on the smaller graph++This repeats until Q stops improving. Each round of aggregation discovers larger-scale communities. It naturally produces a hierarchy: first small clusters, then clusters-of-clusters.++**Time complexity:** Nearly O(N) in practice (each node is moved a small constant number of times). This makes it suitable for large codebases.++**Why `seed=42`?** Louvain is non-deterministic — the order you process nodes affects the result. Fixing the random seed makes results reproducible across runs. 42 is a conventional choice (from The Hitchhiker's Guide).++### How Repowise uses communities++1. **Architecture diagram**: Communities are listed in the generated architecture overview template, showing the LLM which files form natural subsystems so it can describe the high-level structure.+2. **Frontend graph visualization**: The graph UI has a "color by community" mode. Each community gets a distinct color, making clusters visually obvious.+3. **File page context**: Each file's community_id is included in the LLM prompt template when generating file documentation, helping the LLM explain which subsystem a file belongs to.+4. **Path finder diagnostics**: When no shortest path exists between two files, the system checks if they're in the same community. If they're in different communities, it suggests "bridge nodes" — files that have connections to both communities.++---++## 5. Shortest Path (BFS)++### What question does it answer?++**"How does file A depend on file B, through which intermediate files?"**++### The intuition++Given two files, trace the chain of imports connecting them. Like asking "how do I get from this API endpoint to that database model?" — the answer is a chain: `endpoint.py → service.py → repository.py → model.py`.++### The math++Repowise uses `nx.shortest_path()` which runs **Breadth-First Search (BFS)** on unweighted graphs:++1. Start at the source node. Mark it as visited. Distance = 0.+2. Visit all its neighbors. Mark them. Distance = 1.+3. Visit all _their_ unvisited neighbors. Distance = 2.+4. Continue until you reach the target or exhaust the graph.++BFS guarantees the first path found is the shortest (fewest hops). Time complexity: O(N + E).++### When no path exists: Visual context fallback++If no directed path exists from A to B, Repowise doesn't just say "not found." It computes diagnostic context:++1. **Reverse path check**: Maybe B → A exists (the dependency flows the other way).+2. **Nearest common ancestors**: Convert to undirected, find nodes reachable from both A and B, pick the closest ones. These are files that both A and B relate to even though they don't directly connect.+3. **Shared neighbors**: Files that are directly connected to both A and B.+4. **Community analysis**: Are A and B in the same community? If not, suggest bridge nodes (high-PageRank files connected to both communities).++This is a practical design choice — rather than a binary "connected or not," the system gives you the relationships that do exist.++---++## 6. Ego Graph (N-hop Neighborhood)++### What question does it answer?++**"What is the local context around this file?"**++### The intuition++When you're exploring a file, you want to see its neighborhood: what it imports, what imports it, and maybe one more level out. The ego graph gives you a focused subgraph centered on one file.++### The math++Uses `nx.ego_graph(graph, node, radius=hops, undirected=True)`:++1. Start at the center node+2. Find all nodes within `hops` steps in either direction (ignoring edge direction — both importers and importees count)+3. Return the subgraph induced by those nodes++With hops=2 (default), you see:+- The center file+- Everything it imports + everything that imports it (1-hop)+- Everything those files import or are imported by (2-hop)++This is BFS with a depth limit, time complexity O(branching_factor^hops).++---++## 7. Single-Source Shortest Path Length (BFS with cutoff)++### What question does it answer?++**"What is reachable from this entry point, and how far away is each file?"**++### Used in: Entry-point architecture view++```python+paths = nx.single_source_shortest_path_length(graph, ep.node_id, cutoff=3)+```++Starting from each entry point (e.g., `main.py`, `app.py`), this computes the distance to every reachable file within 3 hops. The union of all reachable files forms the "architecture view" — the parts of the codebase that are actually exercised from entry points.++Files NOT reachable from any entry point within 3 hops are candidates for dead code or library code that's only indirectly used.++This is standard BFS, truncated at depth 3.++---++## 8. In-Degree Analysis (Dead Code Detection)++### What question does it answer?++**"Is this file used by anything?"**++### The intuition++The simplest graph metric: count the number of incoming edges. If no file imports this file (in_degree = 0), it might be dead code.++### The math++```+in_degree(v) = |{ u : edge(u, v) exists }|+```++Just count the edges pointing into each node. O(1) per node lookup.++### Why it's not that simple++In-degree = 0 doesn't always mean dead code. Repowise applies several filters:++- **Entry points** (`main.py`, `app.py`, `index.ts`): Nothing imports them, but they're the starting points. Not dead.+- **Test files**: Tests import production code, not the other way around. Low in-degree is expected.+- **Config files** (`__init__.py`, `setup.py`, `next.config.js`): Loaded by frameworks, not via import statements.+- **Non-code files** (JSON, YAML, Markdown): No import semantics at all.+- **Framework patterns** (`*Handler`, `*Plugin`, `*Middleware`): Loaded dynamically, not via imports.++After filtering, Repowise scores confidence using git metadata:+- **No commits in 90 days + older than 180 days**: confidence = 1.0 (almost certainly dead)+- **No commits in 90 days**: confidence = 0.7 (probably dead)+- **Recent commits but no importers**: confidence = 0.4 (suspicious but maybe actively used via dynamic loading)++---++## How the Algorithms Connect++These algorithms aren't isolated — they feed into each other to create a complete picture:++```+ Import Graph+ │+ ┌──────────────────┼──────────────────┐+ │ │ │+ PageRank Betweenness SCCs+ "How important?" "How bridge-like?" "Any cycles?"+ │ │ │+ └──────┬───────────┘ │+ │ │+ Significant File Cycle Documentation+ Selection (Level 2) (SCC Pages)+ │+ ▼+ Doc Generation+ (priority order)+++ Community Detection+ "What clusters exist?"+ │+ ┌────────┼────────┐+ │ │ │+ Arch Frontend Path Finder+ Diagram Coloring Bridge Suggestions+++ In-Degree+ "Is anything importing this?"+ │+ Dead Code Detection+ (filtered by entry points, tests, configs)+++ Shortest Path / Ego / BFS+ "How are specific files connected?"+ │+ API Endpoints + Frontend Graph Views+```++### Why these specific algorithms?++| Algorithm | Alternative considered | Why this one wins |+|-----------|----------------------|-------------------|+| PageRank | Simple import count | Captures transitive importance, not just direct |+| Betweenness | Closeness centrality | Betweenness directly identifies bottlenecks; closeness measures average distance which is less actionable |+| Tarjan's SCC | Brute-force cycle detection | O(N+E) vs O(N³). For large codebases, brute-force is impractical |+| Louvain | Spectral clustering, Girvan-Newman | Louvain is nearly O(N) and handles large graphs. Girvan-Newman is O(N²E), too slow. Spectral requires matrix decomposition, overkill for this use case |+| BFS shortest path | Dijkstra | Edges are unweighted (an import is an import). BFS is optimal for unweighted graphs and simpler than Dijkstra |++---++## Complexity Summary++| Algorithm | Time Complexity | Space | Repowise Optimization |+|-----------|----------------|-------|-----------------------|+| PageRank | O(E × iterations) ≈ O(E × 50) | O(N) | Converge fallback to uniform |+| Betweenness | O(N × E) | O(N + E) | Sample k=500 for repos > 30k nodes |+| SCCs (Tarjan) | O(N + E) | O(N) | None needed — already linear |+| Louvain | ~O(N) empirically | O(N + E) | Seed=42 for determinism |+| BFS shortest path | O(N + E) | O(N) | Cutoff=3 for entry-point views |+| In-degree | O(1) per node | O(1) | None needed |++For a 10,000-file repo with 40,000 edges, all metrics compute in under 5 seconds. For a 100,000-file repo, the sampling optimizations keep betweenness under 30 seconds while the rest stay fast.diff --git a/packages/cli/src/repowise/cli/commands/doctor_cmd.py b/packages/cli/src/repowise/cli/commands/doctor_cmd.pyindex 5eddfaa60..5a26f96c6 100644--- a/packages/cli/src/repowise/cli/commands/doctor_cmd.py+++ b/packages/cli/src/repowise/cli/commands/doctor_cmd.py@@ -22,7 +22,8 @@ def _check(name: str, ok: bool, detail: str = "") -> tuple[str, str, str]:@click.command("doctor")@click.argument("path", required=False, default=None)-def doctor_command(path: str | None) -> None:[email protected]("--repair", is_flag=True, default=False, help="Attempt to fix detected mismatches.")+def doctor_command(path: str | None, repair: bool) -> None:"""Run health checks on the wiki setup."""repo_path = resolve_repo_path(path)checks: list[tuple[str, str, str]] = []@@ -102,6 +103,7 @@ def doctor_command(path: str | None) -> None:checks.append(_check("Providers", False, str(e)))# 6. Stale page count+ stale_count = 0if db_ok and page_count > 0:try:@@ -131,6 +133,88 @@ def doctor_command(path: str | None) -> None:except Exception:checks.append(_check("Stale pages", True, "Could not check"))+ # 7-8. Three-store consistency (SQL vs Vector Store vs FTS)+ missing_from_vector: set[str] = set()+ orphaned_vector: set[str] = set()+ missing_from_fts: set[str] = set()+ orphaned_fts: set[str] = set()++ if db_ok and page_count > 0:+ try:++ async def _check_stores():+ from repowise.core.persistence import (+ FullTextSearch,+ create_engine,+ create_session_factory,+ get_repository_by_path,+ get_session,+ list_pages,+ )+ from repowise.core.persistence.vector_store import (+ InMemoryVectorStore,+ LanceDBVectorStore,+ )+ from repowise.core.providers.embedding.base import MockEmbedder++ url = get_db_url_for_repo(repo_path)+ engine = create_engine(url)+ sf = create_session_factory(engine)++ # Get all SQL page IDs+ async with get_session(sf) as session:+ repo = await get_repository_by_path(session, str(repo_path))+ if not repo:+ await engine.dispose()+ return set(), set(), set(), set()+ pages = await list_pages(session, repo.id, limit=10000)+ sql_ids = {p.page_id for p in pages}++ # Check vector store+ vs_ids: set[str] = set()+ lance_dir = repowise_dir / "lancedb"+ if lance_dir.exists():+ try:+ embedder = MockEmbedder()+ vs = LanceDBVectorStore(str(lance_dir), embedder=embedder)+ vs_ids = await vs.list_page_ids()+ await vs.close()+ except Exception:+ pass # LanceDB not available++ m_vec = sql_ids - vs_ids if vs_ids else set()+ o_vec = vs_ids - sql_ids if vs_ids else set()++ # Check FTS+ fts = FullTextSearch(engine)+ try:+ fts_ids = await fts.list_indexed_ids()+ except Exception:+ fts_ids = set()+ m_fts = sql_ids - fts_ids if fts_ids else set()+ o_fts = fts_ids - sql_ids if fts_ids else set()++ await engine.dispose()+ return m_vec, o_vec, m_fts, o_fts++ missing_from_vector, orphaned_vector, missing_from_fts, orphaned_fts = run_async(+ _check_stores()+ )++ vec_ok = not missing_from_vector and not orphaned_vector+ vec_detail = "in sync" if vec_ok else (+ f"{len(missing_from_vector)} missing, {len(orphaned_vector)} orphaned"+ )+ checks.append(_check("SQL ↔ Vector Store", vec_ok, vec_detail))++ fts_ok = not missing_from_fts and not orphaned_fts+ fts_detail = "in sync" if fts_ok else (+ f"{len(missing_from_fts)} missing, {len(orphaned_fts)} orphaned"+ )+ checks.append(_check("SQL ↔ FTS Index", fts_ok, fts_detail))+ except Exception:+ checks.append(_check("Store consistency", True, "Could not check"))+# Displaytable = Table(title="repowise Doctor")table.add_column("Check", style="cyan")@@ -145,3 +229,100 @@ def doctor_command(path: str | None) -> None:console.print("[bold green]All checks passed![/bold green]")else:console.print("[bold yellow]Some checks failed.[/bold yellow]")++ # --repair: fix detected mismatches+ has_mismatches = missing_from_fts or orphaned_fts or missing_from_vector or orphaned_vector+ if repair and has_mismatches:+ console.print("\n[bold]Repairing store mismatches...[/bold]")++ async def _repair():+ from repowise.core.persistence import (+ FullTextSearch,+ create_engine,+ create_session_factory,+ get_session,+ list_pages,+ )++ url = get_db_url_for_repo(repo_path)+ engine = create_engine(url)+ sf = create_session_factory(engine)+ repaired = 0++ # Repair FTS: re-index missing pages, delete orphaned+ if missing_from_fts or orphaned_fts:+ fts = FullTextSearch(engine)+ await fts.ensure_index()+ if missing_from_fts:+ # Fetch full page data for missing pages+ async with get_session(sf) as session:+ from repowise.core.persistence.models import Page+ from sqlalchemy import select++ rows = await session.execute(+ select(Page).where(Page.page_id.in_(list(missing_from_fts)))+ )+ for page in rows.scalars().all():+ await fts.index(page.page_id, page.title, page.content)+ repaired += 1+ for pid in orphaned_fts:+ await fts.delete(pid)+ repaired += 1++ # Repair vector store: re-embed missing pages, delete orphaned+ lance_dir = repowise_dir / "lancedb"+ if lance_dir.exists() and (missing_from_vector or orphaned_vector):+ try:+ from repowise.core.persistence.vector_store import LanceDBVectorStore+ from repowise.core.providers.embedding.base import MockEmbedder++ import os++ # Resolve embedder — try real embedder, fall back to mock+ embedder_name = os.environ.get("REPOWISE_EMBEDDER", "mock").lower()+ if embedder_name == "mock":+ embedder = MockEmbedder()+ else:+ # Use mock for repair to avoid API costs; user can re-run+ # repowise reindex for real embeddings+ embedder = MockEmbedder()++ vs = LanceDBVectorStore(str(lance_dir), embedder=embedder)++ if missing_from_vector:+ async with get_session(sf) as session:+ from repowise.core.persistence.models import Page+ from sqlalchemy import select++ rows = await session.execute(+ select(Page).where(+ Page.page_id.in_(list(missing_from_vector))+ )+ )+ for page in rows.scalars().all():+ await vs.embed_and_upsert(+ page.page_id,+ page.content,+ {+ "title": page.title,+ "page_type": page.page_type,+ "target_path": page.target_path,+ },+ )+ repaired += 1++ for pid in orphaned_vector:+ await vs.delete(pid)+ repaired += 1++ await vs.close()+ except Exception as exc:+ console.print(f"[yellow]Vector repair skipped: {exc}[/yellow]")++ await engine.dispose()+ return repaired++ repaired_count = run_async(_repair())+ console.print(f"[bold green]Repaired {repaired_count} entries.[/bold green]")+ elif repair and not has_mismatches:+ console.print("[green]Nothing to repair.[/green]")diff --git a/packages/cli/src/repowise/cli/commands/init_cmd.py b/packages/cli/src/repowise/cli/commands/init_cmd.pyindex a286cd174..a50cafbc6 100644--- a/packages/cli/src/repowise/cli/commands/init_cmd.py+++ b/packages/cli/src/repowise/cli/commands/init_cmd.py@@ -570,6 +570,16 @@ def init_command(# Build graphprogress.update(task_graph, visible=True)graph_builder.build()++ # Add framework-aware synthetic edges (conftest, Django, FastAPI, Flask)+ try:+ from repowise.core.generation.editor_files.tech_stack import detect_tech_stack++ tech_items = detect_tech_stack(repo_path)+ graph_builder.add_framework_edges([item.name for item in tech_items])+ except Exception:+ pass # framework edge detection is best-effort+progress.update(task_graph, completed=1)# Wait for git indexing to complete — show a spinner so the userdiff --git a/packages/cli/src/repowise/cli/commands/update_cmd.py b/packages/cli/src/repowise/cli/commands/update_cmd.pyindex 64468b2a3..954897754 100644--- a/packages/cli/src/repowise/cli/commands/update_cmd.py+++ b/packages/cli/src/repowise/cli/commands/update_cmd.py@@ -24,7 +24,7 @@ from repowise.cli.helpers import (@click.option("--provider", "provider_name", default=None, help="LLM provider name.")@click.option("--model", default=None, help="Model identifier override.")@click.option("--since", default=None, help="Base git ref to diff from (overrides state).")[email protected]("--cascade-budget", type=int, default=30, help="Max pages to regenerate per run.")[email protected]("--cascade-budget", type=int, default=None, help="Max pages to regenerate (auto-scaled if unset).")@click.option("--dry-run", is_flag=True, default=False, help="Show affected pages without regenerating.")@@ -33,7 +33,7 @@ def update_command(provider_name: str | None,model: str | None,since: str | None,- cascade_budget: int,+ cascade_budget: int | None,dry_run: bool,) -> None:"""Incrementally update wiki pages for files changed since last sync."""@@ -113,6 +113,17 @@ def update_command(passgraph_builder.build()+ # Add framework-aware synthetic edges (conftest, Django, FastAPI, Flask)+ try:+ from repowise.core.generation.editor_files.tech_stack import detect_tech_stack++ tech_items = detect_tech_stack(repo_path)+ fw_count = graph_builder.add_framework_edges([item.name for item in tech_items])+ if fw_count:+ console.print(f"Framework edges added: [cyan]{fw_count}[/cyan]")+ except Exception:+ pass # framework edge detection is best-effort+# Re-index git metadata for changed filesgit_meta_map: dict[str, dict] = {}try:@@ -132,7 +143,12 @@ def update_command(except Exception as exc:console.print(f"[yellow]Git re-index skipped: {exc}[/yellow]")- # Determine affected pages+ # Determine affected pages (auto-scale budget if not explicitly set)+ if cascade_budget is None:+ from repowise.core.ingestion.change_detector import compute_adaptive_budget++ cascade_budget = compute_adaptive_budget(file_diffs, len(file_infos))+ console.print(f"Adaptive cascade budget: [cyan]{cascade_budget}[/cyan]")affected = detector.get_affected_pages(file_diffs, graph_builder.graph(), cascade_budget)console.print(f"Pages to regenerate: [cyan]{len(affected.regenerate)}[/cyan]")@@ -305,6 +321,20 @@ def update_command(save_state(repo_path, state)elapsed = time.monotonic() - start- console.print(- f"[bold green]Updated {len(generated_pages)} pages in {elapsed:.1f}s[/bold green]"- )++ # Print generation report+ try:+ from repowise.core.generation.report import GenerationReport, render_report++ report = GenerationReport.from_pages(+ generated_pages,+ stale_count=len(affected.decay_only),+ decisions_count=len(new_decision_markers),+ elapsed=elapsed,+ )+ render_report(report, console)+ except Exception:+ # Fallback to simple message if report fails+ console.print(+ f"[bold green]Updated {len(generated_pages)} pages in {elapsed:.1f}s[/bold green]"+ )diff --git a/packages/core/src/repowise/core/generation/page_generator.py b/packages/core/src/repowise/core/generation/page_generator.pyindex 11f7d6989..5f8a0ca0e 100644--- a/packages/core/src/repowise/core/generation/page_generator.py+++ b/packages/core/src/repowise/core/generation/page_generator.py@@ -15,6 +15,7 @@ from __future__ import annotationsimport asyncioimport hashlib+import reimport uuidfrom collections.abc import Callablefrom datetime import UTC, datetime@@ -812,7 +813,7 @@ class PageGenerator:log.debug("rag.search_failed", path=parsed.file_info.path, error=str(e))user_prompt = self._render("file_page.j2", ctx=ctx)response = await self._call_provider("file_page", user_prompt, str(uuid.uuid4()))- return self._build_generated_page(+ page = self._build_generated_page("file_page",parsed.file_info.path,f"File: {parsed.file_info.path}",@@ -820,6 +821,17 @@ class PageGenerator:compute_source_hash(user_prompt),GENERATION_LEVELS["file_page"],)+ # Cross-check LLM output against actual symbols+ hal_warnings = _validate_symbol_references(response.content, parsed)+ if hal_warnings:+ log.warning(+ "hallucination_check",+ path=parsed.file_info.path,+ count=len(hal_warnings),+ refs=hal_warnings[:5],+ )+ page.metadata["hallucination_warnings"] = hal_warnings+ return pageasync def _call_provider(self,@@ -938,3 +950,58 @@ def _is_significant_file(return is_entry or pr >= pr_thresholdreturn True+++# ---------------------------------------------------------------------------+# LLM output validation+# ---------------------------------------------------------------------------++# Common words that appear in backticks but are not code symbols.+_BACKTICK_SKIP = frozenset(+ {+ "True", "False", "None", "null", "undefined", "self", "cls", "this",+ "str", "int", "float", "bool", "list", "dict", "set", "tuple",+ "bytes", "object", "type", "Any", "Optional", "Union",+ "async", "await", "return", "yield", "import", "from",+ "class", "def", "if", "else", "for", "while", "try", "except",+ "raise", "with", "pass", "break", "continue", "lambda",+ "in", "not", "and", "or", "is", "del", "assert", "finally",+ "elif", "as", "pip", "npm", "go", "rust", "python", "node",+ }+)++# Regex: single-backtick references that look like identifiers.+_BACKTICK_REF_RE = re.compile(r"(?<!`)` *([A-Za-z_]\w*(?:\.\w+)*) *`(?!`)")+++def _validate_symbol_references(+ content: str,+ parsed: ParsedFile,+) -> list[str]:+ """Cross-check backtick-quoted names in LLM output against actual symbols.++ Returns a list of warning strings for references that don't match any+ known symbol, export, or import in the ParsedFile.+ """+ refs = set(_BACKTICK_REF_RE.findall(content))+ if not refs:+ return []++ known: set[str] = set()+ for s in parsed.symbols:+ known.add(s.name)+ known.add(s.qualified_name)+ known.update(parsed.exports)+ for imp in parsed.imports:+ if imp.module_path:+ known.add(imp.module_path.split(".")[-1])+ known.update(imp.imported_names)++ warnings: list[str] = []+ for ref in refs:+ if ref in _BACKTICK_SKIP or len(ref) < 2:+ continue+ base = ref.split(".")[-1]+ if ref not in known and base not in known:+ warnings.append(ref)+ return warningsdiff --git a/packages/core/src/repowise/core/generation/report.py b/packages/core/src/repowise/core/generation/report.pynew file mode 100644index 000000000..bac973d56--- /dev/null+++ b/packages/core/src/repowise/core/generation/report.py@@ -0,0 +1,99 @@+"""Generation report — structured summary of a generation run.++Provides token accounting, page breakdown by type, and cost estimation.+"""++from __future__ import annotations++from collections import Counter+from dataclasses import dataclass, field++from .models import GeneratedPage+++@dataclass+class GenerationReport:+ """Summary produced after ``generate_all`` completes."""++ pages_by_type: dict[str, int] = field(default_factory=dict)+ total_input_tokens: int = 0+ total_output_tokens: int = 0+ total_cached_tokens: int = 0+ stale_page_count: int = 0+ dead_code_findings_count: int = 0+ decisions_extracted: int = 0+ elapsed_seconds: float = 0.0+ hallucination_warning_count: int = 0++ @classmethod+ def from_pages(+ cls,+ pages: list[GeneratedPage],+ *,+ stale_count: int = 0,+ dead_code_count: int = 0,+ decisions_count: int = 0,+ elapsed: float = 0.0,+ ) -> GenerationReport:+ by_type = dict(Counter(p.page_type for p in pages))+ hal_count = sum(+ 1 for p in pages if p.metadata.get("hallucination_warnings")+ )+ return cls(+ pages_by_type=by_type,+ total_input_tokens=sum(p.input_tokens for p in pages),+ total_output_tokens=sum(p.output_tokens for p in pages),+ total_cached_tokens=sum(p.cached_tokens for p in pages),+ stale_page_count=stale_count,+ dead_code_findings_count=dead_code_count,+ decisions_extracted=decisions_count,+ elapsed_seconds=elapsed,+ hallucination_warning_count=hal_count,+ )++ @property+ def total_pages(self) -> int:+ return sum(self.pages_by_type.values())++ def estimated_cost_usd(+ self,+ input_rate: float = 3.0,+ output_rate: float = 15.0,+ ) -> float:+ """Estimated USD cost. Rates are per 1M tokens (Sonnet 4 defaults)."""+ return (+ self.total_input_tokens * input_rate+ + self.total_output_tokens * output_rate+ ) / 1_000_000+++def render_report(report: GenerationReport, console: object) -> None:+ """Print a rich table summarising the generation run."""+ from rich.table import Table # deferred so core has no hard rich dep++ table = Table(title="Generation Report", show_lines=False)+ table.add_column("Metric", style="cyan")+ table.add_column("Value", justify="right")++ for ptype, count in sorted(report.pages_by_type.items()):+ table.add_row(f" {ptype}", str(count))+ table.add_row("[bold]Total pages[/bold]", f"[bold]{report.total_pages}[/bold]")+ table.add_row("Input tokens", f"{report.total_input_tokens:,}")+ table.add_row("Output tokens", f"{report.total_output_tokens:,}")+ if report.total_cached_tokens:+ table.add_row("Cached tokens", f"{report.total_cached_tokens:,}")+ table.add_row("Est. cost", f"${report.estimated_cost_usd():.4f}")+ table.add_row("Elapsed", f"{report.elapsed_seconds:.1f}s")+ if report.stale_page_count:+ table.add_row("Stale pages", f"[yellow]{report.stale_page_count}[/yellow]")+ if report.dead_code_findings_count:+ table.add_row("Dead code findings", str(report.dead_code_findings_count))+ if report.decisions_extracted:+ table.add_row("Decisions extracted", str(report.decisions_extracted))+ if report.hallucination_warning_count:+ table.add_row(+ "Hallucination warnings",+ f"[yellow]{report.hallucination_warning_count}[/yellow]",+ )++ console.print(table) # type: ignore[union-attr]diff --git a/packages/core/src/repowise/core/ingestion/change_detector.py b/packages/core/src/repowise/core/ingestion/change_detector.pyindex ab26f4ef4..e97143828 100644--- a/packages/core/src/repowise/core/ingestion/change_detector.py+++ b/packages/core/src/repowise/core/ingestion/change_detector.py@@ -76,6 +76,26 @@ class AffectedPages:decay_only: list[str] # pages to mark stale without immediate regeneration+def compute_adaptive_budget(file_diffs: list[FileDiff], total_files: int) -> int:+ """Compute a cascade budget scaled to the magnitude of the change.++ Small changes get a small budget to avoid unnecessary LLM calls.+ Large refactors get a proportionally larger budget so important+ dependent pages are regenerated in the same run. Hard cap at 50.++ Returns an integer cascade budget.+ """+ n = len(file_diffs)+ if n == 0:+ return 0+ if n == 1:+ return 10+ if n <= 5:+ return 30+ # 6+ files: scale proportionally, hard cap at 50+ return min(n * 3, 50, total_files)++# ---------------------------------------------------------------------------# ChangeDetector# ---------------------------------------------------------------------------diff --git a/packages/core/src/repowise/core/ingestion/graph.py b/packages/core/src/repowise/core/ingestion/graph.pyindex 672f270d5..19a50f9c7 100644--- a/packages/core/src/repowise/core/ingestion/graph.py+++ b/packages/core/src/repowise/core/ingestion/graph.py@@ -387,6 +387,171 @@ class GraphBuilder:# Re-add co_changes edgesself.add_co_change_edges(updated_meta, min_count)+ # ------------------------------------------------------------------+ # Framework-aware synthetic edges+ # ------------------------------------------------------------------++ def add_framework_edges(self, tech_stack: list[str] | None = None) -> int:+ """Add synthetic edges for framework-mediated relationships.++ Detects common patterns (conftest fixtures, Django settings/admin/urls,+ FastAPI include_router, Flask register_blueprint) and creates directed+ edges with ``edge_type="framework"``. These edges participate in+ PageRank (they represent real runtime dependencies).++ Returns the number of edges added.+ """+ count = 0+ path_set = set(self._parsed_files.keys())++ # Always run: pytest conftest detection+ count += self._add_conftest_edges(path_set)++ stack_lower = {s.lower() for s in (tech_stack or [])}++ if "django" in stack_lower:+ count += self._add_django_edges(path_set)+ if "fastapi" in stack_lower or "starlette" in stack_lower:+ count += self._add_fastapi_edges(path_set)+ if "flask" in stack_lower:+ count += self._add_flask_edges(path_set)++ if count:+ log.info("Framework edges added", count=count)+ return count++ def _add_edge_if_new(self, source: str, target: str) -> bool:+ """Add a framework edge if no edge already exists. Returns True if added."""+ if source == target:+ return False+ if self._graph.has_edge(source, target):+ return False+ self._graph.add_edge(source, target, edge_type="framework", imported_names=[])+ return True++ def _add_conftest_edges(self, path_set: set[str]) -> int:+ """conftest.py → test files in the same or child directories."""+ import re as _re++ count = 0+ conftest_paths = [p for p in path_set if Path(p).name == "conftest.py"]++ for conf in conftest_paths:+ conf_dir = Path(conf).parent.as_posix()+ prefix = f"{conf_dir}/" if conf_dir != "." else ""+ for p in path_set:+ if p == conf:+ continue+ node = self._graph.nodes.get(p, {})+ if not node.get("is_test", False):+ continue+ # Test file must be in the same or a child directory+ if p.startswith(prefix) or (prefix == "" and "/" not in p):+ if self._add_edge_if_new(p, conf):+ count += 1+ return count++ def _add_django_edges(self, path_set: set[str]) -> int:+ """Django conventions: admin→models, urls→views in the same directory."""+ import re as _re++ count = 0+ by_dir: dict[str, dict[str, str]] = {} # dir → {stem: path}+ for p in path_set:+ pp = Path(p)+ d = pp.parent.as_posix()+ by_dir.setdefault(d, {})[pp.stem] = p++ for d, stems in by_dir.items():+ # admin.py → models.py+ if "admin" in stems and "models" in stems:+ if self._add_edge_if_new(stems["admin"], stems["models"]):+ count += 1+ # urls.py → views.py+ if "urls" in stems and "views" in stems:+ if self._add_edge_if_new(stems["urls"], stems["views"]):+ count += 1+ # forms.py → models.py+ if "forms" in stems and "models" in stems:+ if self._add_edge_if_new(stems["forms"], stems["models"]):+ count += 1+ # serializers.py → models.py+ if "serializers" in stems and "models" in stems:+ if self._add_edge_if_new(stems["serializers"], stems["models"]):+ count += 1+ return count++ def _add_fastapi_edges(self, path_set: set[str]) -> int:+ """Detect include_router() calls and link app files to router modules."""+ import re as _re++ count = 0+ # Build a map from imported variable names to source file paths+ # by scanning imports across all parsed files+ var_to_file: dict[str, str] = {}+ for path, parsed in self._parsed_files.items():+ for imp in parsed.imports:+ for name in imp.imported_names:+ if name.lower().endswith("router") or name.lower().endswith("app"):+ resolved = self._resolve_import(+ imp.module_path, path, path_set,+ {Path(p).stem.lower(): p for p in path_set},+ parsed.file_info.language,+ )+ if resolved and resolved in path_set:+ var_to_file[name] = resolved++ # Scan source for include_router / add_api_route patterns+ router_re = _re.compile(r"(?:include_router|add_api_route)\s*\(\s*(\w+)")+ for path, parsed in self._parsed_files.items():+ if parsed.file_info.language != "python":+ continue+ try:+ source = Path(parsed.file_info.abs_path).read_text(errors="ignore")+ except Exception:+ continue+ for match in router_re.finditer(source):+ var_name = match.group(1)+ target = var_to_file.get(var_name)+ if target and target in path_set:+ if self._add_edge_if_new(path, target):+ count += 1+ return count++ def _add_flask_edges(self, path_set: set[str]) -> int:+ """Detect register_blueprint() calls and link app files to blueprint modules."""+ import re as _re++ count = 0+ var_to_file: dict[str, str] = {}+ for path, parsed in self._parsed_files.items():+ for imp in parsed.imports:+ for name in imp.imported_names:+ if "blueprint" in name.lower() or name.lower().endswith("bp"):+ resolved = self._resolve_import(+ imp.module_path, path, path_set,+ {Path(p).stem.lower(): p for p in path_set},+ parsed.file_info.language,+ )+ if resolved and resolved in path_set:+ var_to_file[name] = resolved++ bp_re = _re.compile(r"register_blueprint\s*\(\s*(\w+)")+ for path, parsed in self._parsed_files.items():+ if parsed.file_info.language != "python":+ continue+ try:+ source = Path(parsed.file_info.abs_path).read_text(errors="ignore")+ except Exception:+ continue+ for match in bp_re.finditer(source):+ var_name = match.group(1)+ target = var_to_file.get(var_name)+ if target and target in path_set:+ if self._add_edge_if_new(path, target):+ count += 1+ return count+def pagerank(self, alpha: float = 0.85) -> dict[str, float]:"""Return PageRank scores for each node.diff --git a/packages/core/src/repowise/core/persistence/search.py b/packages/core/src/repowise/core/persistence/search.pyindex ebf63c792..cc2023c7d 100644--- a/packages/core/src/repowise/core/persistence/search.py+++ b/packages/core/src/repowise/core/persistence/search.py@@ -217,6 +217,22 @@ class FullTextSearch:{"pid": page_id},)+ async def list_indexed_ids(self) -> set[str]:+ """Return the set of page IDs currently in the FTS index.++ Used by ``repowise doctor --repair`` to detect three-store+ inconsistencies.+ """+ if self._dialect == "sqlite":+ async with self._engine.connect() as conn:+ rows = await conn.execute(text("SELECT page_id FROM page_fts"))+ return {r[0] for r in rows.fetchall()}+ # PostgreSQL: all wiki_pages rows are automatically indexed via GIN,+ # so the set of "indexed" ids is all page ids in the table.+ async with self._engine.connect() as conn:+ rows = await conn.execute(text("SELECT id FROM wiki_pages"))+ return {r[0] for r in rows.fetchall()}+async def search(self, query: str, limit: int = 10) -> list[SearchResult]:"""Search for pages matching *query*.diff --git a/packages/core/src/repowise/core/persistence/vector_store.py b/packages/core/src/repowise/core/persistence/vector_store.pyindex e1016c189..dce05c085 100644--- a/packages/core/src/repowise/core/persistence/vector_store.py+++ b/packages/core/src/repowise/core/persistence/vector_store.py@@ -68,6 +68,14 @@ class VectorStore(ABC):"""Release any resources held by the store."""...+ async def list_page_ids(self) -> set[str]:+ """Return the set of page IDs currently stored.++ Used by ``repowise doctor --repair`` to detect three-store+ inconsistencies. Implementations may override for efficiency.+ """+ return set() # default: empty (subclasses should override)+# ---------------------------------------------------------------------------# InMemoryVectorStore@@ -134,6 +142,9 @@ class InMemoryVectorStore(VectorStore):async def close(self) -> None:self._store.clear()+ async def list_page_ids(self) -> set[str]:+ return set(self._store.keys())+def __len__(self) -> int:return len(self._store)@@ -276,6 +287,13 @@ class LanceDBVectorStore(VectorStore):self._table = Noneself._db = None+ async def list_page_ids(self) -> set[str]:+ await self._ensure_connected()+ if self._table is None:+ return set()+ rows = await self._table.query().select(["page_id"]).to_list() # type: ignore[union-attr]+ return {r["page_id"] for r in rows}+# ---------------------------------------------------------------------------# PgVectorStore@@ -363,3 +381,12 @@ class PgVectorStore(VectorStore):async def close(self) -> None:pass # session_factory manages connection lifecycle++ async def list_page_ids(self) -> set[str]:+ from sqlalchemy.sql import text as sa_text++ async with self._session_factory() as session:+ rows = await session.execute(+ sa_text("SELECT id FROM wiki_pages WHERE embedding IS NOT NULL")+ )+ return {r[0] for r in rows.fetchall()}