Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs: document git host support and the forges package (#3400)
Raghav Chamadiyaae206201bd3a
diff --git a/docs/README.md b/docs/README.mdindex fea7fecd5..0be57f110 100644--- a/docs/README.md+++ b/docs/README.md@@ -81,6 +81,12 @@ For a map of every layer and how they feed each other, read|-----|----------------|| [layers/DECISIONS.md](layers/DECISIONS.md) | Architectural decisions mined from your repo and your agent sessions |+### Use GitLab, Azure DevOps or Bitbucket++| Doc | What it covers |+|-----|----------------|+| [start/GIT_HOSTS.md](start/GIT_HOSTS.md) | What works on each git host and CI, and how to name a self-managed host |+### Run across many repos| Doc | What it covers |diff --git a/docs/architecture/README.md b/docs/architecture/README.mdindex 484473c7b..a82e90bbe 100644--- a/docs/architecture/README.md+++ b/docs/architecture/README.md@@ -14,6 +14,7 @@ curious; you don't need them to *use* repowise (start with| [decisions.md](decisions.md) | Decision capture internals: the source registry, evidence verification, confidence, currency, session mining and hook delivery || [test-intelligence.md](test-intelligence.md) | The inferred test tier: call and import walks, their measured precision and recall, the resolution-origin filter, and why nothing is stored || [language-support.md](language-support.md) | The language pipeline internals and the step-by-step recipe for adding a new language |+| [forges.md](forges.md) | Git host support: the forges package, its invariants, and the recipes for adding a host or a CI system || [agent-platform.md](agent-platform.md) | Agent identities, provider specs, integration targets and adapters, and the recipes for adding an agent integration or an agent CLI indexing backend || [graph-algorithms.md](graph-algorithms.md) | The graph algorithms (PageRank, betweenness, Tarjan SCC, Leiden/Louvain community detection, shortest path) with the math and complexity || [refactoring.md](refactoring.md) | Refactoring intelligence internals: detector registration, plan shape, ranking, REST routes, and code generation |diff --git a/docs/architecture/forges.md b/docs/architecture/forges.mdnew file mode 100644index 000000000..66afe5d47--- /dev/null+++ b/docs/architecture/forges.md@@ -0,0 +1,98 @@+# Forges: git host support++`packages/core/src/repowise/core/forges/` holds every fact repowise knows about+a git host (a forge): which hosts it claims, its URL shapes, its CI variables,+its merge-message conventions, its noreply addresses and bot accounts. Callers+ask this package and never match hosts themselves. The user-facing summary is+[GIT_HOSTS.md](../start/GIT_HOSTS.md).++## Module map++| File | Holds |+|------|-------|+| `base.py` | `ForgeKind`, `RemoteRef`, `RemoteParts`, the `Forge` protocol, `BaseForge` defaults, `CiSystem`, URL and ref helpers |+| `registry.py` | `register`, `get_forge`, `all_forges`, host resolution (override, then host claim, else generic), `forge_hosts` and `REPOWISE_FORGE_HOSTS` |+| `remote.py` | `parse_remote`, `canonical_key`, `strip_credentials`: pure string work on remote URLs |+| `detect.py` | `read_remote_url` (reads `.git/config`, follows worktrees) and `detect_forge` (remote, then CI env) |+| `changes.py` | `change_number` (the change a commit is) and `change_refs` (every change it mentions) |+| `identity.py` | `canonical_email`, `noreply_login`, `is_bot`, plus the bot names shared by every host |+| `ci.py` | `CiContext`, `detect_ci`, and CIs that belong to no forge (Jenkins) |+| `github.py`, `gitlab.py`, `azure.py`, `bitbucket.py` | One forge each: host claims, path parsing, URL builders, `CiSystem`, merge forms, identity rules |+| `generic.py` | The fallback: parses any `host/path`, builds no URLs, reads GitHub-style messages |++Users: the webhook lookup (`server/routers/webhooks.py`), `git_refs.remote_name`,+the overview remote and repository responses (`strip_credentials`),+`core/ci/base.py` (`CI_ENV_VARS`, `default_revspec`), the git indexer+(`change_number`), revert matching (`analysis/decisions/reverts.py`) and+`author_identity.py`.++## Invariants++- **Every forge fact lives in its forge's module.** A host name, URL shape,+ noreply domain, bot name or merge convention elsewhere is a bug. New host+ regexes go in this package, not in their callers.+- **Parsers do no I/O.** `remote.py`, `changes.py`, `identity.py` and `ci.py`+ read strings and env mappings only. `detect.py` reads `.git/config` directly+ instead of spawning git; `forge_hosts` reads the repo config.+- **Values are frozen, slotted dataclasses** (`RemoteRef`, `CiSystem`,+ `CiContext` and every forge). Registration replaces a forge; nothing mutates one.+- **Canonical keys** drop the scheme, userinfo, port, query, `.git`, trailing+ slashes and GitLab's `/-/` web routes, then lowercase the whole key, path+ included: GitHub and GitLab resolve repo paths case-insensitively, so a+ registered URL that differs from the webhook's only in case must still match.+ Azure's three host forms fold to `dev.azure.com/org/project/_git/repo`;+ Bitbucket Data Center keys as `host/PROJ/repo`.+- **Precision over recall.** A wrong change number is worse than none, so only+ a merge or squash form sets one; a bare `#N`, `AB#N` or a link never does.+ Every merge form needs a literal lead, exactly one capture group and no+ `IGNORECASE` flag (`lead_of` raises otherwise); write the number as+ `\d{1,9}`. For identity, a missed fold beats+ joining two people: only a forge's provable noreply form folds, and a bot+ name matches as an explicit marker or a full service name, never a substring.+- **Identity reads every forge.** An address keeps its minting forge's shape+ after a move, so `normalize_identity` gives the same answer on every forge.+- **History rules are versioned.** Changing a merge form or identity rule+ changes stored rows: bump `GIT_HISTORY_VERSION` in+ `ingestion/git_indexer/_constants.py` so the next update re-reads the full+ history once.++## Adding a forge++1. Add a member to `ForgeKind` in `base.py`.+2. Create `forges/<name>.py` with a `@dataclass(frozen=True, slots=True)`+ subclass of `BaseForge`.+3. Declare host claims in `claims_host` (exact public hosts and aliases; a+ self-managed install is reached through `forges.hosts`). Claims must stay+ disjoint from every other forge's.+4. Implement `parse` (the forge's path shape, `None` when it does not fit) and,+ if clone and web URLs differ, `canonical_path`.+5. Implement the URL builders: `commit_url`, `change_url`, `blob_url`,+ `compare_url`.+6. Set `label` and `change_term`, and override `format_change_ref` if the+ display is not `#N`.+7. Give it a `CiSystem` (markers, base branch, head sha, change number, repo+ URL) if the forge runs its own CI.+8. Declare merge forms (`merge_subject_res`, `merge_body_res`,+ `native_merge_subject_res`, `merge_body_first`) and implement+ `parse_change_refs` for mentions.+9. Declare identity rules: `noreply_re` with a `login` group and+ `noreply_fold`, `bot_name_re`, `bot_email_re`.+10. Call `register(...)` at module bottom and import the module in+ `forges/__init__.py`.+11. Add rows to each table test under `tests/unit/core/`:+ `test_forges_remote.py` (parse, key, credentials), `test_forges_urls.py`+ (URLs, label, change display), `test_forges_change_refs.py`+ (`change_number`, `change_refs`), `test_forges_identity.py` (noreply, bots),+ `test_forges_ci.py` (CI env) and `test_forges_detect.py` (remote and CI+ detection).+12. If a merge form or identity rule changes stored history, bump+ `GIT_HISTORY_VERSION`.++## Adding a CI system that is not a forge++A CI that builds repos from any host (Jenkins) gets a `CiSystem` in `ci.py`+and an entry in `_STANDALONE`; standalone CIs are tried after the forges'+own. `CI_ENV_VARS` and `CI_BASE_VARS` in `core/ci/base.py` pick up its markers+and base-branch vars automatically. A CI that is detected by presence alone,+with nothing to read, goes in `_PRESENCE_ONLY` in `core/ci/base.py`. Add its+build shapes to `CASES` in `tests/unit/core/test_forges_ci.py`.diff --git a/docs/reference/CONFIG.md b/docs/reference/CONFIG.mdindex 317c2ac80..e8e63ea39 100644--- a/docs/reference/CONFIG.md+++ b/docs/reference/CONFIG.md@@ -156,6 +156,7 @@ You can edit this file directly. Changes take effect on the next `init`,| `distill` | see below | Output distillation config || `mcp` | see below | MCP tool surface config || `refactoring` | see below | Refactoring-intelligence config |+| `forges.hosts` | `{}` | Maps a self-managed git host to its forge (`github`, `gitlab`, `azure`, `bitbucket`, `generic`). See [GIT_HOSTS.md](../start/GIT_HOSTS.md#self-managed-hosts) |`reasoning` controls documentation-generation calls for reasoning-capable chatmodels. `auto` preserves provider defaults. `off`/`none` disable Qwen3-style@@ -1016,6 +1017,7 @@ Anonymous usage telemetry is **enabled by default** (opt-out).|----------|-------------|| `REPOWISE_GIT_WINDOW_ANCHOR` | Git history windows (90-day churn, prior defects, decay, blame age) are measured from the indexed commit's committer date by default. Set to `now` to measure them from wall-clock time instead || `REPOWISE_SKIP_EDITOR_SETUP` | Truthy value stops `init` writing to your machine-wide editor config: the Claude Code / Claude Desktop MCP entry, the Claude Code hooks, and the distill rewrite-hook offer. Same switch as `init --no-editor-setup` ([CLI_REFERENCE.md](CLI_REFERENCE.md#repowise-init-path)); the env var is the one to use for CI, sandboxes, and benchmark runs that index many repos. Project-local files (`.repowise/mcp.json`, `CLAUDE.md`, Codex config) are written either way |+| `REPOWISE_FORGE_HOSTS` | Self-managed git hosts as `host=forge` pairs separated by commas (`git.corp.com=gitlab,tfs.corp.com=azure`). Wins over `forges.hosts` in `config.yaml` || `REPOWISE_CHANGELOG` | Override the changelog source used by the "what's new" check || `REPOWISE_PARSE_WORKERS` | How many processes parse files during indexing. Defaults to your CPU count capped at 8, and never exceeds the number of files to parse. Each worker is a separate interpreter holding roughly 50 MB, so lower it on a memory-constrained machine; raising it above 8 is not measurably faster |diff --git a/docs/reference/UPGRADING.md b/docs/reference/UPGRADING.mdindex 51760d434..43c0a7493 100644--- a/docs/reference/UPGRADING.md+++ b/docs/reference/UPGRADING.md@@ -68,6 +68,10 @@ repowise decision source set session --on**Your first `update` re-scores the whole repository.** The health analyzer version moved from 11 to 21 over this cycle, because new test-quality markers, a per-language assertion vocabulary, a `.tsx` grammar fix and re-ranked history gates all change how a stored health number is computed. Rather than wait out the decay timer, the first update after upgrading re-scores every file, so it takes longer than usual. On a 171-file repository that was 10.2 seconds against a 5.8 second baseline, with the next update back to normal. Nothing needs re-indexing: the store format and parser schema are unchanged.+### Git history is re-read once when its rules change++When a release changes how git history becomes stored rows (change numbers, author identity), the first `update` re-reads the full history once instead of only the changed files. It regenerates no pages and makes no model calls. See [GIT_HOSTS.md](../start/GIT_HOSTS.md#upgrading).+### `health-rules.json` globs now match like `.gitignore`Per-path rules in `.repowise/health-rules.json` used to be matched withdiff --git a/docs/scale/AUTO_SYNC.md b/docs/scale/AUTO_SYNC.mdindex 711ad157e..60c59d5bf 100644--- a/docs/scale/AUTO_SYNC.md+++ b/docs/scale/AUTO_SYNC.md@@ -127,6 +127,10 @@ With `REPOWISE_GITLAB_WEBHOOK_TOKEN` set, the server compares it with the`401 Unauthorized`. Without the token, only local callers are accepted and everyother caller gets `403 Forbidden`, as with GitHub.+Both endpoints match the payload's repository to a registered one by a+canonical key, so clone, ssh and web URLs of the same repository match, case+aside. Other hosts are listed in [GIT_HOSTS.md](../start/GIT_HOSTS.md).+---## 5. Polling Fallbackdiff --git a/docs/start/CI.md b/docs/start/CI.mdindex 970721e6b..e881e406a 100644--- a/docs/start/CI.md+++ b/docs/start/CI.md@@ -55,7 +55,8 @@ target branch from the CI's own variables (`GITHUB_BASE_REF`,does so with `--fail-above-percentile` or `--format markdown` / `github`;otherwise an omitted range still means your uncommitted work, else `HEAD`. Docdrift has no range: it checks the whole working tree, and a baseline limits it-to new findings.+to new findings. The variables each CI is detected by and read from, Azure+Pipelines included, are in [GIT_HOSTS.md](GIT_HOSTS.md#ci-systems).**History.** CI checkouts are usually shallow, which leaves no merge-base.Fetch full history (`fetch-depth: 0` on GitHub Actions, `GIT_DEPTH: 0` ondiff --git a/docs/start/GIT_HOSTS.md b/docs/start/GIT_HOSTS.mdnew file mode 100644index 000000000..13601b225--- /dev/null+++ b/docs/start/GIT_HOSTS.md@@ -0,0 +1,133 @@+# Git hosts++Repowise reads your repository from git, so indexing, the graph, health and+the MCP tools work the same on any host. What depends on the host is smaller:+recognising the remote URL, reading the CI job's target branch, finding the+pull or merge request number in commit history, telling bots from people, and+matching a webhook to a registered repository. This page says what works where.++<p>+ <strong>Hosts </strong>+ <img src="https://img.shields.io/badge/GitHub-181717?style=flat-square&logo=github&logoColor=white" alt="GitHub" />+ <img src="https://img.shields.io/badge/GitLab-FC6D26?style=flat-square&logo=gitlab&logoColor=white" alt="GitLab" />+ <img src="https://img.shields.io/badge/Azure%20DevOps-0078D7?style=flat-square" alt="Azure DevOps" />+ <img src="https://img.shields.io/badge/Bitbucket-0052CC?style=flat-square&logo=bitbucket&logoColor=white" alt="Bitbucket" />+ <img src="https://img.shields.io/badge/Any%20git%20host-F05032?style=flat-square&logo=git&logoColor=white" alt="Any git host" />+</p>+<p>+ <strong>CI </strong>+ <img src="https://img.shields.io/badge/GitHub%20Actions-2088FF?style=flat-square&logo=githubactions&logoColor=white" alt="GitHub Actions" />+ <img src="https://img.shields.io/badge/GitLab%20CI-FC6D26?style=flat-square&logo=gitlab&logoColor=white" alt="GitLab CI" />+ <img src="https://img.shields.io/badge/Azure%20Pipelines-2560E0?style=flat-square" alt="Azure Pipelines" />+ <img src="https://img.shields.io/badge/Bitbucket%20Pipelines-0052CC?style=flat-square&logo=bitbucket&logoColor=white" alt="Bitbucket Pipelines" />+ <img src="https://img.shields.io/badge/Jenkins-D24939?style=flat-square&logo=jenkins&logoColor=white" alt="Jenkins" />+</p>++## Support++| Host | Remote recognised | CI detected | PR / MR numbers from history | Webhook auto-sync |+|------|-------------------|-------------|------------------------------|-------------------|+| GitHub | Yes | GitHub Actions | Yes | `/api/webhooks/github` |+| GitHub Enterprise | With a [host override](#self-managed-hosts) | GitHub Actions | Yes | `/api/webhooks/github` |+| GitLab.com | Yes | GitLab CI | Yes | `/api/webhooks/gitlab` |+| Self-managed GitLab | Yes when a host label is `gitlab` (`gitlab.corp.com`), else with an override | GitLab CI | Yes | `/api/webhooks/gitlab` |+| Azure DevOps (`dev.azure.com`, `ssh.dev.azure.com`, `*.visualstudio.com`) | Yes, all three URL forms as one repo | Azure Pipelines | Yes | No |+| Bitbucket Cloud | Yes | Bitbucket Pipelines | Yes | No |+| Bitbucket Data Center | With an override | Jenkins or another [listed CI](#ci-systems) | Yes | No |+| Other git hosts (Gitea, cgit, plain ssh) | As a generic host | Any [listed CI](#ci-systems) | Yes | No |++Azure DevOps Server needs an override too. Bot accounts and noreply addresses+from every host are recognised on every repository, since history can predate+a move between hosts. Remote URLs shown in the dashboard and returned by the+server have their credentials removed.++## Self-managed hosts++Repowise never probes a host over the network, so a self-managed host with a+neutral name (`git.corp.com`) reads as a generic host until you name its forge.+Add the mapping to `.repowise/config.yaml`:++```yaml+forges:+ hosts:+ git.corp.com: gitlab+ ghe.corp.com: github+ tfs.corp.com: azure+ bitbucket.corp.com: bitbucket+```++`.repowise/` is local and gitignored, so for CI runners and teammates set it in+the environment instead:++```bash+REPOWISE_FORGE_HOSTS="git.corp.com=gitlab,tfs.corp.com=azure"+```++Values are `github`, `gitlab`, `azure`, `bitbucket` or `generic`; an unknown+value is ignored. Keys match on the host alone, so a scheme or port in the key+does no harm. The environment variable wins over the config file. When the+remote's host is still unknown, the CI running the job decides: a GitLab runner+(`GITLAB_CI`) marks the repository as GitLab.++## CI systems++| CI | Detected by | Base branch read from | Change number read from |+|----|-------------|-----------------------|-------------------------|+| GitHub Actions | `GITHUB_ACTIONS` | `GITHUB_BASE_REF` | `GITHUB_REF` (`refs/pull/N/merge`) |+| GitLab CI | `GITLAB_CI` | `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | `CI_MERGE_REQUEST_IID` |+| Azure Pipelines | `TF_BUILD` or `SYSTEM_TEAMFOUNDATIONCOLLECTIONURI` | `SYSTEM_PULLREQUEST_TARGETBRANCHNAME`, else `SYSTEM_PULLREQUEST_TARGETBRANCH` | `SYSTEM_PULLREQUEST_PULLREQUESTNUMBER`, else `SYSTEM_PULLREQUEST_PULLREQUESTID` |+| Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_PR_DESTINATION_BRANCH` | `BITBUCKET_PR_ID` |+| Jenkins | `JENKINS_URL` | `CHANGE_TARGET` (multibranch) | `CHANGE_ID` |+| Buildkite, TeamCity, any CI setting `CI` | `BUILDKITE`, `TEAMCITY_VERSION`, `CI` (known as CI only) | none: pass a range | none |++The CI gates diff `origin/<base branch>...HEAD` when you pass no range, with+`refs/heads/` removed from the branch. Azure Pipelines building a repository+hosted elsewhere reads `BUILD_REPOSITORY_PROVIDER` to tell which host it is.+Setup for each gate is in [CI.md](CI.md).++## What repowise reads from commit history++A commit gets a PR or MR number only when its own merge or squash message says+it is that change:++| Host | Forms that count |+|------|------------------|+| GitHub and generic hosts | Subject `Merge pull request #12 ...`; a subject ending `(#12)` or `(gh-12)` |+| GitLab | Body trailer `See merge request group/proj!12` (or its URL form); on GitLab repos, a subject ending `(!12)` |+| Azure DevOps | Subject `Merged PR 12: ...` or `Merge pull request 12 from ...` |+| Bitbucket | A subject ending `(pull request #12)`; subject `Merge pull request #12 in PROJ/repo ...` |++Every host's forms are read on every repository, the repository's own host+first. A bare `#12` never sets a number: it names an issue as often as a pull+request, and a wrong number is worse than none. Azure Boards links (`AB#12`)+and links to a pull request name some other change, so they do not count+either. A `revert:` commit that names a change (`#12`, a pull URL or `!12`) is+linked to it only when exactly one commit's merge message is that change.++Authors fold and bots drop out the same way on every host:++- GitHub noreply addresses (`[email protected]` and+ `[email protected]`) fold to one person. GitLab noreply+ addresses keep their account id and stay as written.+- Bots: names or emails with `[bot]`, names ending in `-bot`, `_bot` or ` bot`, common services+ (Dependabot, Renovate, Snyk, Codecov, Mergify, pre-commit.ci and others),+ `github-actions`, GitLab project and group access-token bots and+ `Ghost User`, Azure `... Build Service (org)` accounts and+ `bitbucket-pipelines`.++## Upgrading++The first `repowise update` after upgrading to a release that changes these+history rules re-reads the full git history once, so change numbers and+authors are rewritten for every file, not only the changed ones. It+regenerates no pages and makes no model calls. Later updates are incremental+again.++## See also++- [CI.md](CI.md) for the gates and the GitHub Action and GitLab templates+- [AUTO_SYNC.md](../scale/AUTO_SYNC.md) for webhook setup+- [SECURITY.md](../layers/SECURITY.md) for the `gitlab_token`,+ `azure_devops_pat` and `git_url_credentials` secret kinds+- [forges.md](../architecture/forges.md) for how host support is built and+ how to add a host