Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

fix(decisions): stop inline markers matching prose, and tell a failed source from an empty one (#1610)

Raghav Chamadiya93879a9ec61b
diff --git a/docs/layers/DECISIONS.md b/docs/layers/DECISIONS.md
index 71ed124d8..34e81e19a 100644
--- a/docs/layers/DECISIONS.md
+++ b/docs/layers/DECISIONS.md
@@ -46,18 +46,16 @@ get_why() # health dashboard
## Where decisions come from
-Six capture sources run at index time. Five are deterministic passes over the
-repo and its git history; the sixth is a harvest during LLM doc generation, and
-it is the only one that needs a provider.
+Five capture sources run at index time, each a pass over the repo and its git
+history.
| Source | Key | Reads | Notes |
|--------|-----|-------|-------|
| ADR files | `adr` | `adr/`, `adrs/`, `docs/adr/`, `docs/adrs/`, `docs/decisions/`, `decisions/`, `architecture/`, `doc/adr/` | Nygard and MADR headings plus YAML frontmatter, parsed without an LLM. Up to 60 files. `accepted`/`approved` map to `active`, `draft` to `proposed`, `rejected` to `deprecated`. |
-| Inline markers | `inline_marker` | `# WHY:` / `# DECISION:` / `# TRADEOFF:` / `# ADR:` / `# RATIONALE:` / `# REJECTED:` | Any comment syntax (`#`, `//`, `--`, `/*`, `*`). Up to 5 continuation lines, plus 20 lines of surrounding context. Fenced code blocks in Markdown are skipped. |
+| Inline markers | `inline_marker` | `# WHY:` / `# DECISION:` / `# TRADEOFF:` / `# ADR:` / `# RATIONALE:` / `# REJECTED:` | Any comment syntax (`#`, `//`, `--`, `/*`, `*`). The keyword is case-sensitive and must be capitalised, like `TODO:` and `FIXME:` — otherwise ordinary prose ("# Rejected: nothing to extract.") becomes an architectural decision. Up to 5 continuation lines, plus 20 lines of surrounding context. Fenced code blocks in Markdown are skipped. |
| Git archaeology | `git_archaeology` | Commit messages | Gated on 19 decision verbs (migrate, switch to, replace, adopt, deprecate, drop, rewrite, split, revert, and the rest). |
| PR bodies | `pr` | Squash-merge and PR commit bodies | A body only qualifies when it looks like a PR description (`## Why`, `## Motivation`, `## Context`, `Closes #`, `Before:` / `After:`). Up to 25 bodies. |
| Code comments | `comment` | Block comments and docstrings on high-centrality files | Bounded to 30 nodes, and to prose carrying a rationale cue ("because", "instead of", "rather than", "trade-off", "we chose", "deliberately"). Centrality-bounded on purpose: comment archaeology across a whole repo is noise. |
-| Doc generation | `llm_inferred` | The wiki generation pass | The page generator proposes decisions it inferred while writing a page. Every field must survive the grounding gate below. |
Two more sources sit outside the index-time set: `session` (mined from your
coding-agent transcripts, below) and `cli` (a decision you typed yourself, the
diff --git a/docs/reference/CLI_REFERENCE.md b/docs/reference/CLI_REFERENCE.md
index 288985309..f8a2dfdd7 100644
--- a/docs/reference/CLI_REFERENCE.md
+++ b/docs/reference/CLI_REFERENCE.md
@@ -133,7 +133,7 @@ In workspace mode, adds: repo scanning, per-repo indexing, cross-repo analysis (
1. **Everything**, index + model-written docs. After picking a provider you can answer **"Customize?"** to tune any setting before the run.
2. **Index only**, the same layers plus a wiki rendered from structure; no LLM, no key, no cost. Answer **"Customize indexing?"** to set exclude patterns, commit limit, skip-tests/infra, submodules, and fast mode.
-3. **Advanced**, full control. First choose **"Generate model-written wiki docs?"**; the prompts then split into an **Indexing** section (always) and a **Generation** section (provider, concurrency, embedder, wiki style, onboarding, decision harvesting, tiering, only when model-written docs are on).
+3. **Advanced**, full control. First choose **"Generate model-written wiki docs?"**; the prompts then split into an **Indexing** section (always) and a **Generation** section (provider, concurrency, embedder, wiki style, onboarding, tiering, only when model-written docs are on).
**Page volume.** File pages are rendered from structure and cost no model tokens, so what bounds them buys wiki size, embedding calls and less retrieval noise, never spend. Two numbers govern it, both derived from the size distribution of real indexed repos. Above **2,000** documentable files (p95), an interactive advanced-mode run offers the top 2,000 by importance instead of one page per file, quoted in pages and megabytes; nothing is capped automatically at that size. Above **4,500** file pages (about 40 MB of file layer, near p99), every run holds the bucket to 4,500 and says so in its output, because that is where the tail is the difference between a wiki that publishes and one that does not. `--max-file-pages N` sets a cap directly and `--max-file-pages 0` refuses one; either way the answer is saved as `max_file_pages` in `config.yaml`, which `update --full` and `generate` honour. A hosted or scripted caller sets the same field with no terminal involved.
@@ -161,7 +161,6 @@ All three reach the indexing knobs; the LLM-only knobs appear only when model-wr
| `--max-file-pages` | Most file pages to emit, highest importance first. Omit to let the size policy decide (see Page volume above), `0` for one page per eligible file however many that is, or a positive number for a hard cap. Saved to `config.yaml`. |
| `--coverage-report` | Test-coverage report to ingest (LCOV / Cobertura / Clover). Repeatable. Auto-discovered when omitted. This is test coverage for code-health, not a documentation-breadth knob: every code file is documented either way. |
| `--onboarding` / `--no-onboarding` | Generate the curated Onboarding collection (up to 8 overview pages). Default: on; slots without enough signal are skipped. |
-| `--harvest-decisions` / `--no-harvest-decisions` | Harvest architectural decisions during page generation (verified against source before storage). Default: on. |
| `--wiki-style` | Documentation voice/density: `comprehensive` (default), `caveman` (token-condensed, AI-first), `reference` (API-manual), `tutorial` (beginner-friendly). Interactive full runs prompt when omitted. Saved to config so `update` keeps the style. See [WIKI.md](../layers/WIKI.md#styles). |
| `--language` | Output language for generated wiki pages: `en` (default), `ar`, `de`, `es`, `fr`, `hi`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `ru`, `tr`, `zh`. Code, file paths, and symbol names stay untranslated. Saved to config so `update` keeps the language. Also asked in advanced interactive mode. To switch an existing wiki's language, set the flag and re-run `init --force`. |
| `--resume` | Continue a previous run instead of redoing it: completed phases (indexing, analysis) are skipped, the earlier run's git tier is kept, and generation writes only the pages this repo does not have yet. Use it after an interrupted run, and after one that finished with failed pages (a provider outage, rate limiting) — pages already written are skipped with no model call, so nothing is paid for twice. Matching is per page, not per model, so switching provider still keeps what the old one wrote. |
diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/command.py b/packages/cli/src/repowise/cli/commands/init_cmd/command.py
index 78095573c..ea59672a7 100644
--- a/packages/cli/src/repowise/cli/commands/init_cmd/command.py
+++ b/packages/cli/src/repowise/cli/commands/init_cmd/command.py
@@ -257,7 +257,6 @@ def _run_generation_phase(
language: str,
resolved_reasoning: str,
onboarding: bool,
- harvest_decisions: bool,
wiki_style: str,
max_file_pages: int | None,
yes: bool,
@@ -291,7 +290,6 @@ def _run_generation_phase(
language=language,
reasoning=resolved_reasoning,
enable_onboarding=onboarding,
- harvest_decisions=harvest_decisions,
wiki_style=wiki_style,
max_file_pages=max_file_pages,
)
@@ -630,17 +628,6 @@ def _run_generation_phase(
"code-health, not a documentation-breadth knob."
),
)
- "--harvest-decisions/--no-harvest-decisions",
- "harvest_decisions",
- default=True,
- help=(
- "Harvest candidate architectural decisions from LLM page generation "
- "(file pages). Each harvested decision is verified against the file's "
- "source before storage. The model emits a decision only on a genuine "
- "hit, so the token cost lands only on files that carry one. Default: on."
- ),
-)
@click.option(
"--wiki-style",
"wiki_style",
@@ -737,7 +724,6 @@ def init_command(
onboarding: bool,
max_file_pages_opt: int | None,
coverage_report: tuple[str, ...],
- harvest_decisions: bool,
wiki_style: str | None,
language_opt: str | None,
no_cost_tracking: bool,
@@ -893,7 +879,6 @@ def init_command(
resume=resume,
force=force,
onboarding=onboarding,
- harvest_decisions=harvest_decisions,
# Apply the chosen style uniformly across the workspace's repos
# (no per-repo interactive prompt in the multi-repo flow).
wiki_style=resolve_style(wiki_style).name,
@@ -1033,7 +1018,6 @@ def init_command(
reasoning = adv.get("reasoning") or reasoning
test_run = adv["test_run"]
onboarding = adv.get("onboarding", onboarding)
- harvest_decisions = adv.get("harvest_decisions", harvest_decisions)
if adv.get("wiki_style"):
wiki_style = adv["wiki_style"]
if adv.get("language"):
@@ -1409,7 +1393,6 @@ def init_command(
language=language,
resolved_reasoning=resolved_reasoning,
onboarding=onboarding,
- harvest_decisions=harvest_decisions,
wiki_style=wiki_style,
max_file_pages=max_file_pages,
yes=yes,
diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py b/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py
index bd827cf9b..1cd7a4b67 100644
--- a/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py
+++ b/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py
@@ -95,7 +95,6 @@ def _run_workspace_generation(
test_run: bool,
reasoning: str = "auto",
onboarding: bool = True,
- harvest_decisions: bool = True,
wiki_style: str = DEFAULT_STYLE,
language: str = "en",
) -> list[Any]:
@@ -113,7 +112,6 @@ def _run_workspace_generation(
max_concurrency=concurrency,
reasoning=resolve_reasoning(reasoning),
enable_onboarding=onboarding,
- harvest_decisions=harvest_decisions,
wiki_style=wiki_style,
language=language,
# No question is asked in the workspace flow, so this only picks up a cap
@@ -268,7 +266,6 @@ class _WorkspaceCtx:
yes: bool
resume: bool
onboarding: bool
- harvest_decisions: bool
wiki_style: str
language: str
resolved_reasoning: str
@@ -412,7 +409,6 @@ def _ingest_and_generate_repo(repo: Any, idx: int, total: int, ctx: _WorkspaceCt
test_run=ctx.test_run,
reasoning=ctx.resolved_reasoning,
onboarding=ctx.onboarding,
- harvest_decisions=ctx.harvest_decisions,
wiki_style=ctx.wiki_style,
language=ctx.language,
)
@@ -582,7 +578,6 @@ def _workspace_init(
resume: bool = False,
force: bool = False,
onboarding: bool = True,
- harvest_decisions: bool = True,
wiki_style: str = DEFAULT_STYLE,
language: str | None = None,
run_mode: str = "standard",
@@ -661,7 +656,7 @@ def _workspace_init(
reasoning = selection.reasoning
# Pass the resolved style so the advanced generation section doesn't
# add a per-workspace wiki-style prompt (the workspace flow applies
- # one style uniformly); onboarding / decision harvesting still apply.
+ # one style uniformly); onboarding still applies.
adv = interactive_advanced_config(
console, prompt_reasoning=False, wiki_style=wiki_style, language=language
)
@@ -676,7 +671,6 @@ def _workspace_init(
reasoning = adv.get("reasoning") or reasoning
embedder_name_resolved = resolve_embedder(adv.get("embedder") or embedder_name)
onboarding = adv.get("onboarding", onboarding)
- harvest_decisions = adv.get("harvest_decisions", harvest_decisions)
if adv.get("wiki_style"):
wiki_style = adv["wiki_style"]
if adv.get("language"):
@@ -779,7 +773,6 @@ def _workspace_init(
yes=yes,
resume=resume,
onboarding=onboarding,
- harvest_decisions=harvest_decisions,
wiki_style=wiki_style,
language=language or "en",
resolved_reasoning=resolved_reasoning,
diff --git a/packages/cli/src/repowise/cli/commands/update_cmd/persistence.py b/packages/cli/src/repowise/cli/commands/update_cmd/persistence.py
index 4003c25a8..2adb77fd9 100644
--- a/packages/cli/src/repowise/cli/commands/update_cmd/persistence.py
+++ b/packages/cli/src/repowise/cli/commands/update_cmd/persistence.py
@@ -816,10 +816,9 @@ async def _persist_full_update_async(
import dataclasses as _dc
decision_dicts.extend(_dc.asdict(d) for d in new_decision_markers)
- for page in generated_pages:
- harvested = page.metadata.get("harvested_decisions")
- if harvested:
- decision_dicts.extend(harvested)
+ # A second read folded in ``page.metadata["harvested_decisions"]``
+ # here. Nothing ever wrote that key, so it never contributed a
+ # record; it is gone along with the generation-time harvest.
if decision_dicts:
from repowise.core.persistence.crud import bulk_upsert_decisions
diff --git a/packages/cli/src/repowise/cli/ui/mode_selection.py b/packages/cli/src/repowise/cli/ui/mode_selection.py
index ee6cf108f..76d7acade 100644
--- a/packages/cli/src/repowise/cli/ui/mode_selection.py
+++ b/packages/cli/src/repowise/cli/ui/mode_selection.py
@@ -417,7 +417,7 @@ def _prompt_generation(
language: str | None = None,
) -> None:
"""Generation section: concurrency, reasoning, embedder, test run, tiering,
- onboarding, decision harvesting, wiki style, and output language.
+ onboarding, wiki style, and output language.
*wiki_style* carries an explicit ``--wiki-style`` value; when set the style
prompt is skipped so the flag wins. *language* works the same way for the
@@ -469,13 +469,6 @@ def _prompt_generation(
default=True,
)
- # Decision harvesting rides along with file-page generation; the token cost
- # lands only on files that actually carry a decision.
- result["harvest_decisions"] = click.confirm(
- " Harvest architectural decisions during generation?",
- default=True,
- )
-
# Documentation voice/density. An explicit --wiki-style wins; otherwise prompt
# here so the choice lands inside the section, before the summary panel.
result["wiki_style"] = wiki_style if wiki_style is not None else prompt_wiki_style(console)
@@ -545,7 +538,6 @@ def _summary_rows(
f"{result['language']} ({SUPPORTED_LANGUAGES.get(result['language'], '?')})",
)
add("Onboarding", "yes" if result.get("onboarding", True) else "no")
- add("Harvest decisions", "yes" if result.get("harvest_decisions", True) else "no")
add("Test run", "yes" if result["test_run"] else "no")
return summary
@@ -567,7 +559,7 @@ def interactive_advanced_config(
The indexing section (scope, run mode, exclude, git) is always prompted.
The generation section (concurrency, reasoning, embedder, onboarding,
- decision harvesting, tiering, wiki style, test run) is prompted only when
+ tiering, wiki style, test run) is prompted only when
*generate_docs* is True, so an index-only advanced run skips knobs that have
no effect. ``generate_docs`` is echoed back in the result.
@@ -575,7 +567,7 @@ def interactive_advanced_config(
``commit_limit``, ``follow_renames``, ``skip_tests``, ``skip_infra``,
``exclude``, ``include_submodules``, ``run_mode``, ``max_file_pages``,
``generate_docs`` (always), plus ``concurrency``, ``reasoning``, ``embedder``,
- ``test_run``, ``onboarding``, ``harvest_decisions``, ``wiki_style``,
+ ``test_run``, ``onboarding``, ``wiki_style``,
``language`` (docs only).
Editor integration prompts are intentionally not asked here so that full and
diff --git a/packages/core/src/repowise/core/analysis/decisions/extractor.py b/packages/core/src/repowise/core/analysis/decisions/extractor.py
index c333c76a3..87a998fbe 100644
--- a/packages/core/src/repowise/core/analysis/decisions/extractor.py
+++ b/packages/core/src/repowise/core/analysis/decisions/extractor.py
@@ -153,11 +153,52 @@ class ExtractedDecision:
source_text: str = ""
+class DecisionSourceError(RuntimeError):
+ """Every batch of a decision source failed.
+
+ Raised so :meth:`DecisionExtractor.extract_all` records the source as
+ failed rather than empty. A source that loses *some* batches still
+ returns what it has and only logs, because partial supply beats none.
+ """
+
+
+def _collect_batches(
+ source: str,
+ results: list[Any],
+) -> list[ExtractedDecision]:
+ """Flatten ``asyncio.gather(..., return_exceptions=True)`` output.
+
+ Exceptions are logged per batch. If nothing survived and there was work
+ to do, the source failed outright and says so instead of returning a
+ zero that reads like an empty repository.
+ """
+ decisions: list[ExtractedDecision] = []
+ errors: list[str] = []
+ for result in results:
+ if isinstance(result, list):
+ decisions.extend(result)
+ else:
+ errors.append(f"{type(result).__name__}: {result}")
+ logger.warning(
+ "decision_extractor.batch_failed",
+ source=source,
+ error=str(result),
+ )
+ if errors and len(errors) == len(results):
+ raise DecisionSourceError(f"all {len(errors)} batch(es) failed — {errors[0]}")
+ return decisions
+
+
@dataclass
class DecisionExtractionReport:
total_found: int
decisions: list[ExtractedDecision]
by_source: dict[str, int]
+ # Sources that raised, {source name: error text}. A source that fails
+ # returns an empty list, so without this a total outage and an honestly
+ # empty repo are the same number on screen. Callers render it; nothing
+ # else can tell the two apart.
+ failures: dict[str, str] = field(default_factory=dict)
# Every index-time capture source, in progress order. The CLI derives its
@@ -191,11 +232,20 @@ def enabled_source_names(repo_config: dict[str, Any] | None) -> tuple[str, ...]:
# Comment marker detection
# ---------------------------------------------------------------------------
+# The keyword is deliberately case-SENSITIVE. These are annotation
+# conventions, written in caps like TODO:/FIXME:/HACK:, and matching them
+# case-insensitively turns ordinary prose into architectural decisions. Two
+# real examples from this repo, both of which reached the store as `active`
+# records: a wrapped sentence whose continuation line began "# decision:
+# namespace, batched like the pages", and a test's "# Rejected: nothing to
+# extract." Across 3,860 tracked files those were the ONLY two matches — a
+# 100% false-positive rate — because no genuine marker was written in lower
+# case. A missed marker costs one record; a false positive publishes a
+# sentence fragment as a decision governing every file it touches.
MARKER_RE = re.compile(
r"^\s*(?:#|//|--|/\*|\*)\s*"
r"(?P<keyword>WHY|DECISION|TRADEOFF|ADR|RATIONALE|REJECTED)"
r"\s*:\s*(?P<text>.+)",
- re.IGNORECASE,
)
_SKIP_DIRS = frozenset(
@@ -355,6 +405,10 @@ _COMMENT_RATIONALE_CUES = (
)
_MAX_COMMENT_NODES = 30
+# Inline markers sent to the model per call. Caps prompt length only; a file
+# with more markers is sent in several calls rather than truncated.
+_MARKERS_PER_CALL = 5
+
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# Main extractor
@@ -531,24 +585,36 @@ class DecisionExtractor:
async def _structure_markers_via_llm(
self, file_path: str, markers: list[dict]
) -> list[ExtractedDecision]:
- """Use LLM to structure inline markers into decision records."""
- markers_block = ""
- for m in markers[:5]: # Batch up to 5 per call
- markers_block += (
- f"\n--- Marker ({m['keyword']}) at line {m['line']} ---\n"
- f"Text: {m['text']}\n"
- f"Surrounding code:\n{m['context'][:1500]}\n"
- )
+ """Use LLM to structure inline markers into decision records.
+
+ Markers are sent five per call. The batch size caps prompt length;
+ it is not a cap on how many markers a file may have. ``markers[:5]``
+ alone dropped every marker past the fifth: the caller invokes this
+ once per file, and the raw-marker fallback below only runs on an
+ exception, so those markers were never structured and never fell
+ back — they left no record and no log line.
+ """
+ decisions: list[ExtractedDecision] = []
+ for start in range(0, len(markers), _MARKERS_PER_CALL):
+ batch = markers[start : start + _MARKERS_PER_CALL]
+ markers_block = ""
+ for m in batch:
+ markers_block += (
+ f"\n--- Marker ({m['keyword']}) at line {m['line']} ---\n"
+ f"Text: {m['text']}\n"
+ f"Surrounding code:\n{m['context'][:1500]}\n"
+ )
- prompt = INLINE_MARKER_PROMPT.format(
- file_path=file_path,
- markers_block=markers_block,
- )
+ prompt = INLINE_MARKER_PROMPT.format(
+ file_path=file_path,
+ markers_block=markers_block,
+ )
- response = await self._provider.generate(
- _SYSTEM_PROMPT, prompt, max_tokens=2000, temperature=0.2
- )
- return self._parse_decisions_json(response.content)
+ response = await self._provider.generate(
+ _SYSTEM_PROMPT, prompt, max_tokens=2000, temperature=0.2
+ )
+ decisions.extend(self._parse_decisions_json(response.content))
+ return decisions
# ------------------------------------------------------------------
# Source 2: Git archaeology
@@ -660,14 +726,7 @@ class DecisionExtractor:
*[_process_batch(b) for b in batches],
return_exceptions=True,
)
- for result in results:
- if isinstance(result, list):
- decisions.extend(result)
- else:
- logger.warning(
- "decision_extractor.git_batch_failed",
- error=str(result),
- )
+ decisions.extend(_collect_batches("git_archaeology", list(results)))
return decisions
@@ -884,12 +943,14 @@ class DecisionExtractor:
)
source_by_sha[c["sha"]] = f"{c['subject']}\n{c['body']}"
prompt = PR_BODY_MINING_PROMPT.format(bodies_block=bodies_block)
- try:
- response = await self._provider.generate(
- _SYSTEM_PROMPT, prompt, max_tokens=2500, temperature=0.2
- )
- except Exception:
- return []
+ # Let this propagate to the gather below. Swallowed here, a total
+ # provider outage returned five empty lists, so ``_collect_batches``
+ # saw no errors and the run reported "Nothing found in: pull
+ # requests" — the exact zero-that-means-failure this change exists
+ # to remove.
+ response = await self._provider.generate(
+ _SYSTEM_PROMPT, prompt, max_tokens=2500, temperature=0.2
+ )
extracted = self._parse_decisions_json(response.content)
for d in extracted:
sha = d.evidence_commits[0] if d.evidence_commits else ""
@@ -908,16 +969,10 @@ class DecisionExtractor:
d.affected_modules = self._infer_modules(d.affected_files)
return extracted
- decisions: list[ExtractedDecision] = []
results = await asyncio.gather(
*[_process_batch(b) for b in batches], return_exceptions=True
)
- for result in results:
- if isinstance(result, list):
- decisions.extend(result)
- else:
- logger.warning("decision_extractor.pr_batch_failed", error=str(result))
- return decisions
+ return _collect_batches("pr", list(results))
# ------------------------------------------------------------------
# Source 5: Comment archaeology (centrality-bounded)
@@ -954,12 +1009,13 @@ class DecisionExtractor:
for fp, prose in batch:
comments_block += f"\n--- {fp} ---\n{prose[:1500]}\n"
prompt = COMMENT_ARCHAEOLOGY_PROMPT.format(comments_block=comments_block)
- try:
- response = await self._provider.generate(
- _SYSTEM_PROMPT, prompt, max_tokens=2500, temperature=0.2
- )
- except Exception:
- return []
+ # Was a bare ``except Exception: return []`` here with no log at
+ # all — the quietest of the three swallows, and the one that made
+ # "comment: 0" unfalsifiable. Let it propagate to the gather
+ # below, which counts it.
+ response = await self._provider.generate(
+ _SYSTEM_PROMPT, prompt, max_tokens=2500, temperature=0.2
+ )
extracted = self._parse_decisions_json(response.content)
# Best-effort attribution to the originating file by token overlap.
for d in extracted:
@@ -982,9 +1038,7 @@ class DecisionExtractor:
results = await asyncio.gather(
*[_process_batch(b) for b in batches], return_exceptions=True
)
- for result in results:
- if isinstance(result, list):
- decisions.extend(result)
+ decisions.extend(_collect_batches("comment", list(results)))
return decisions
# Above this node count, skip the iterative PageRank solve and use degree
@@ -1216,6 +1270,8 @@ class DecisionExtractor:
ungrounded LLM fields are dropped and evidence-less decisions rejected.
"""
+ failures: dict[str, str] = {}
+
async def _safe_source(name: str, coro_fn: Any) -> list[ExtractedDecision]:
try:
logger.info("decision_extractor.starting", source=name)
@@ -1223,6 +1279,10 @@ class DecisionExtractor:
logger.info("decision_extractor.finished", source=name, count=len(result))
return result
except Exception as exc:
+ # Recorded as well as logged: the CLI pins core to ERROR, so
+ # a source that dies here used to reach the user as a plain
+ # zero indistinguishable from "this repo has no ADRs".
+ failures[name] = f"{type(exc).__name__}: {exc}"
logger.warning("decision_extractor.source_failed", source=name, error=str(exc))
return []
finally:
@@ -1269,6 +1329,7 @@ class DecisionExtractor:
total_found=len(decisions),
decisions=decisions,
by_source=by_source,
+ failures=failures,
)
# ------------------------------------------------------------------
diff --git a/packages/core/src/repowise/core/analysis/decisions/provenance.py b/packages/core/src/repowise/core/analysis/decisions/provenance.py
index 27d9abb10..65a60f1bd 100644
--- a/packages/core/src/repowise/core/analysis/decisions/provenance.py
+++ b/packages/core/src/repowise/core/analysis/decisions/provenance.py
@@ -36,8 +36,7 @@ __all__ = [
# same decision, the headline fields come from the highest-rank evidence and
# the lower-rank source is retained as a corroborating evidence row (never
# discarded). ``test_name`` and ``inferred`` are placeholders for later phases
-# (LLM-docs harvest lands ``llm_inferred`` in Phase 2) but the full ladder is
-# defined now so ranks are stable across phases.
+# but the full ladder is defined now so ranks are stable across phases.
#
# A person outranks a document. ``session`` sits above ``adr`` because a
# transcript carries the user's own words about a choice they were making at the
@@ -65,7 +64,12 @@ SOURCE_RANK: dict[str, int] = {
"code_comment": 2, # legacy rows from the removed comment harvest (#751)
"test_name": 2, # placeholder — behaviour asserted by a test name
"inferred": 1, # placeholder — purely inferred, no verbatim source
- "llm_inferred": 1, # Phase 2 LLM-docs harvest
+ # Legacy rows from the generation-time harvest, which is gone: it only ever
+ # ran on file pages, and those are rendered structurally now. Deliberately
+ # NOT in RETIRED_SOURCES yet — adding it drains surviving ``proposed`` rows
+ # and the retired set is hand-mirrored in packages/ui, so that retirement is
+ # its own cross-package change rather than a side effect of this one.
+ "llm_inferred": 1,
}
MAX_SOURCE_RANK: int = max(SOURCE_RANK.values())
diff --git a/packages/core/src/repowise/core/generation/models.py b/packages/core/src/repowise/core/generation/models.py
index 959985eb8..47245a4f8 100644
--- a/packages/core/src/repowise/core/generation/models.py
+++ b/packages/core/src/repowise/core/generation/models.py
@@ -281,13 +281,6 @@ class GenerationConfig:
# an empty / nearly-empty store anyway, so the search is a wasted
# round-trip until enough content is indexed to return useful hits.
rag_min_store_size: int = 10
- # Phase 2: harvest candidate architectural decisions from Tier-1 LLM page
- # generation (file pages). On by default, escapable via
- # ``--no-harvest-decisions``. The model is instructed to emit a decision
- # block only on a genuine hit, so the output-token cost lands only on files
- # that carry a decision; harvested candidates pass the same substring gate
- # as every other source before storage.
- harvest_decisions: bool = True
# ---- In-loop self-repair (hallucinated symbol refs) ----------------
# When the post-generation validator flags at least this many backtick
# identifiers that do not exist in the documented file, the tier-1 file
@@ -526,8 +519,7 @@ class GeneratedPage:
metadata: dict[str, object] = field(default_factory=dict)
# Cross-run reuse KEY (not a plain file hash): SHA256 of the documented
# file's raw-bytes hash folded with the generation fingerprint (template,
- # system prompt, language, style, harvest flag — see
- # PageGenerator._reuse_content_hash). Empty for pages not built from a
+ # system prompt, language, style). Empty for pages not built from a
# single file (module/overview/architecture). Unlike source_hash it is
# stable across runs for an unchanged file + unchanged settings, so
# cross-run reuse can key on it even when the rendered prompt (RAG
diff --git a/packages/core/src/repowise/core/generation/page_generator/decision_harvest.py b/packages/core/src/repowise/core/generation/page_generator/decision_harvest.py
deleted file mode 100644
index 1b4fd8282..000000000
--- a/packages/core/src/repowise/core/generation/page_generator/decision_harvest.py
+++ /dev/null
@@ -1,184 +0,0 @@
-"""Phase-2 LLM-docs decision harvest.
-
-The page generator already runs an LLM over every Tier-1 file page with rich
-graph + git context. This module makes that same call *also* surface candidate
-architectural decisions at near-zero marginal cost: the system prompt is
-extended with a directive instructing the model to append a single trailing
-JSON block when — and only when — a real decision is evident, and this module
-parses that block back out, strips it from the page markdown, and turns it into
-gated ``ExtractedDecision`` candidates.
-
-Two guardrails make this safe to enable by default:
-
-1. **Emit-nothing-on-no-decision.** The directive tells the model to append the
- block only on a genuine hit, so output-token cost is incurred only on files
- that actually carry a decision — never on the long tail.
-2. **The substring gate.** Because the model is now *generating* candidates
- rather than reading them, every harvested ``decision`` / ``rationale`` /
- ``source_quote`` is run through the same anti-hallucination gate
- (:func:`decision_gate.apply_substring_gate`) against the file's verbatim
- source. Ungrounded fields are dropped and evidence-less candidates rejected.
-
-Scope: harvesting is restricted to ``file_page`` — it is the only Tier-1 page
-type with a single, clean verbatim source span (the file's own bytes) for the
-gate to verify quotes against. Module/layer pages synthesise across many files
-and have no such span, so harvesting there would defeat the gate; they are
-deliberately excluded.
-"""
-
-from __future__ import annotations
-
-import dataclasses
-import json
-import re
-
-import structlog
-
-from repowise.core.analysis.decision_extractor import ExtractedDecision
-from repowise.core.analysis.decision_gate import apply_substring_gate
-
-log = structlog.get_logger(__name__)
-
-__all__ = [
- "HARVESTABLE_PAGE_TYPES",
- "HARVEST_DIRECTIVE",
- "harvest_decisions",
- "parse_and_strip_decisions",
-]
-
-# Only page types with a single verbatim source span the gate can verify
-# against. See module docstring for why module/layer pages are excluded.
-HARVESTABLE_PAGE_TYPES: frozenset[str] = frozenset({"file_page"})
-
-# Fenced-block info string used to tag the harvest payload. A dedicated tag
-# (rather than a bare ```json fence) lets us locate the block unambiguously and
-# avoids colliding with legitimate JSON examples inside the documentation body.
-_FENCE_TAG = "repowise-decisions"
-
-# Appended to the file_page system prompt when harvesting is enabled. Kept here
-# (next to the parser) so the emit contract and the parse contract evolve
-# together. The directive is intentionally strict about emitting nothing on a
-# miss — that bound is the whole cost story (see 2D).
-HARVEST_DIRECTIVE = (
- "\n\n"
- "DECISION HARVEST (machine-read — follow exactly):\n"
- "After the markdown documentation, decide whether this file's code, "
- "structure, and the supplied git/architectural signals reveal a real, "
- "concrete architectural DECISION (a deliberate choice with a rationale — "
- "e.g. adopting a library, a pattern, a data store, a protocol, or rejecting "
- "an alternative). Most files embody no such decision.\n"
- f"- If a genuine decision is evident, append EXACTLY ONE fenced ```{_FENCE_TAG} "
- "block containing a JSON object: "
- '{"decisions": [{"title": "...", "decision": "...", "rationale": "...", '
- '"source_quote": "..."}]}. '
- "The source_quote MUST be a verbatim substring copied from this file's "
- "source — not paraphrased — and is what proves the decision is real. "
- 'Keep titles short and canonical (e.g. "Use Redis for caching").\n'
- "- If NO real decision is evident, output NOTHING after the markdown — no "
- "block, no empty block, no commentary. Inventing a decision is worse than "
- "emitting none."
-)
-
-# Match a trailing ```repowise-decisions … ``` fence anywhere in the content
-# (DOTALL so the JSON body may span lines). We take the LAST match so a fence
-# quoted inside the prose body can't shadow the real trailing payload.
-_FENCE_RE = re.compile(
- rf"```[ \t]*{re.escape(_FENCE_TAG)}[ \t]*\r?\n(?P<body>.*?)\r?\n?```",
- re.DOTALL | re.IGNORECASE,
-)
-
-
-def parse_and_strip_decisions(content: str) -> tuple[str, list[dict]]:
- """Split a harvest fence off the end of *content*.
-
- Returns ``(clean_content, decisions)`` where ``clean_content`` is the
- markdown with the fence removed (trailing whitespace trimmed) and
- ``decisions`` is the parsed ``decisions`` list (empty when the block is
- absent or malformed — a harvest failure must never corrupt the page).
- """
- matches = list(_FENCE_RE.finditer(content))
- if not matches:
- return content, []
-
- block = matches[-1]
- clean = (content[: block.start()] + content[block.end() :]).rstrip()
-
- raw = block.group("body").strip()
- try:
- payload = json.loads(raw)
- except (json.JSONDecodeError, ValueError):
- log.debug("decision_harvest.parse_failed", body_len=len(raw))
- return clean, []
-
- decisions = payload.get("decisions") if isinstance(payload, dict) else None
- if not isinstance(decisions, list):
- return clean, []
-
- out = [d for d in decisions if isinstance(d, dict) and (d.get("title") or "").strip()]
- return clean, out
-
-
-def harvest_decisions(
- content: str,
- *,
- source_text: str,
- evidence_file: str,
- affected_modules: list[str] | None = None,
-) -> tuple[str, list[dict]]:
- """Parse, strip, and gate harvested decisions for one page.
-
- Args:
- content: The raw LLM page content (may carry a trailing fence).
- source_text: The file's verbatim source — what the gate verifies
- each harvested quote against.
- evidence_file: The file path the decisions are attributed to.
- affected_modules: Optional module hints for the decisions.
-
- Returns ``(clean_content, gated_decision_dicts)``. The dicts are ready for
- ``crud.bulk_upsert_decisions`` — ``source="llm_inferred"`` (lowest rank, so
- a harvested decision never outranks a real ADR), ``status="proposed"``, and
- ``verification`` stamped by the gate. Ungrounded candidates are dropped.
- """
- clean, raw = parse_and_strip_decisions(content)
- if not raw:
- return clean, []
-
- candidates: list[ExtractedDecision] = []
- for d in raw:
- candidates.append(
- ExtractedDecision(
- title=str(d.get("title", "")).strip(),
- decision=str(d.get("decision", "")).strip(),
- rationale=str(d.get("rationale", "")).strip(),
- source_quote=str(d.get("source_quote", "")).strip(),
- affected_files=[evidence_file],
- affected_modules=list(affected_modules or []),
- source="llm_inferred",
- evidence_file=evidence_file,
- status="proposed",
- # The gate verifies the quote/decision/rationale against this.
- source_text=source_text,
- )
- )
-
- kept, rejected = apply_substring_gate(candidates)
- if rejected:
- log.info(
- "decision_harvest.gate_rejected",
- file=evidence_file,
- kept=len(kept),
- rejected=rejected,
- )
-
- # Title fallback: the gate clears any paraphrased ``decision`` that is not a
- # verbatim substring of the source, but a survivor's ``source_quote`` keeps
- # the record alive — leaving a body-less, title-only decision. The title is
- # the model's own canonical one-line summary of the choice and is always
- # present (``parse_and_strip_decisions`` requires it), so promote it into the
- # empty ``decision`` field. Deterministic, no new LLM call, and the gate's
- # rejection behavior is untouched.
- for d in kept:
- if not d.decision and d.title:
- d.decision = d.title
-
- return clean, [dataclasses.asdict(d) for d in kept]
diff --git a/packages/core/src/repowise/core/pipeline/persist.py b/packages/core/src/repowise/core/pipeline/persist.py
index be2d88ee9..dd122aa5e 100644
--- a/packages/core/src/repowise/core/pipeline/persist.py
+++ b/packages/core/src/repowise/core/pipeline/persist.py
@@ -1509,20 +1509,14 @@ async def persist_analysis(result: Any, session: Any, repo_id: str) -> None:
logger.warning("health_snapshot_skipped", error=str(_snap_err))
# ---- Decision records ----------------------------------------------------
- # Two contributors merge into one upsert: the multi-source extractor
- # (decision_report) and the Phase-2 LLM-docs harvest (ridden on each
- # generated page's metadata, already gated at generation time). Folding
- # them into a single bulk_upsert lets harvested candidates corroborate
- # extracted decisions (extra evidence row + confidence bump) or stand alone
- # as low-rank ``proposed`` records awaiting review.
+ # One contributor: the multi-source extractor. A second read used to fold
+ # ``page.metadata["harvested_decisions"]`` in from LLM page generation, but
+ # nothing ever wrote that key — harvesting only ever ran on ``file_page``,
+ # and file pages are rendered structurally now, so the directive had no
+ # host and the loop was permanently a no-op.
decision_dicts: list[dict] = []
if result.decision_report and result.decision_report.decisions:
decision_dicts.extend(dataclasses.asdict(d) for d in result.decision_report.decisions)
- if result.generated_pages:
- for page in result.generated_pages:
- harvested = page.metadata.get("harvested_decisions")
- if harvested:
- decision_dicts.extend(harvested)
# Restore records the semantic supersession detector retired before it was
# turned off. ``superseded`` is a protected status, so nothing else will
diff --git a/packages/core/src/repowise/core/pipeline/phases/analysis.py b/packages/core/src/repowise/core/pipeline/phases/analysis.py
index 00d1dd518..b8e5f94dc 100644
--- a/packages/core/src/repowise/core/pipeline/phases/analysis.py
+++ b/packages/core/src/repowise/core/pipeline/phases/analysis.py
@@ -376,13 +376,27 @@ async def _run_decision_extraction(
# ``session`` is in the label table but is not one of the
# extractor's own sources, so ``source in enabled`` leaves it out
# of this list — correct, since it is mined separately below.
+ #
+ # A source that *failed* is listed separately and as a warning.
+ # It is not "nothing found": the repo may be full of decisions
+ # nobody could read. Same shape the update path already uses for
+ # its ``degraded`` list, so both runs report an outage the same
+ # way instead of one of them printing a reassuring zero.
+ failures = getattr(report, "failures", {}) or {}
empty = [
label.removeprefix("from ")
for source, label in _DECISION_SOURCE_LABELS
- if source in enabled and not bs.get(source)
+ if source in enabled and not bs.get(source) and source not in failures
]
if empty:
progress.on_message("info", f"→ Nothing found in: {', '.join(empty)}")
+ for source, label in _DECISION_SOURCE_LABELS:
+ if source in failures:
+ progress.on_message(
+ "warning",
+ f"Decision source {label.removeprefix('from ')} failed, "
+ f"so this run found none there: {failures[source]}",
+ )
_phase_done(progress, "decisions")
return report
diff --git a/packages/core/src/repowise/core/pipeline/resume/controller.py b/packages/core/src/repowise/core/pipeline/resume/controller.py
index e83498449..20b9ce393 100644
--- a/packages/core/src/repowise/core/pipeline/resume/controller.py
+++ b/packages/core/src/repowise/core/pipeline/resume/controller.py
@@ -202,9 +202,7 @@ class ResumeController:
lets a generation interrupt resume past analysis instead of recomputing
it (decision extraction in particular can run for minutes). Best-effort:
a persistence hiccup is logged and swallowed, leaving ANALYSIS unmarked
- so the resumed run simply recomputes it. ``generated_pages`` is empty at
- this point, so harvested-decision folding still happens at the final
- end-of-run persist.
+ so the resumed run simply recomputes it.
"""
from repowise.core.persistence import get_session
diff --git a/packages/core/src/repowise/core/providers/llm/base.py b/packages/core/src/repowise/core/providers/llm/base.py
index fed71a1da..3ca9329a2 100644
--- a/packages/core/src/repowise/core/providers/llm/base.py
+++ b/packages/core/src/repowise/core/providers/llm/base.py
@@ -196,14 +196,6 @@ class GeneratedResponse:
cached_tokens: Tokens served from the provider's prompt cache (if any).
Normalised across providers by the adapter.
usage: Provider-specific usage dict (stored as-is for auditing).
- decisions: Optional structured side-channel: candidate architectural
- decisions the model surfaced while writing the page
- (Phase-2 LLM-docs harvest). Populated by the generator
- from a trailing JSON block (or native structured output)
- and stripped from ``content`` before storage; ``None``
- when nothing was harvested. Each item is a raw dict
- carrying at least ``title`` + ``source_quote``; the
- generator gates them against the file source before use.
stop_reason: Provider-neutral completion reason. Uses the same values
as streaming chat (notably ``end_turn``, ``max_tokens``,
and ``tool_use``).
@@ -216,7 +208,6 @@ class GeneratedResponse:
output_tokens: int
cached_tokens: int = 0
usage: dict[str, Any] = field(default_factory=dict)
- decisions: list[dict] | None = None
stop_reason: str | None = None
provider_stop_reason: str | None = None
diff --git a/tests/unit/analysis/test_decision_marker_batching.py b/tests/unit/analysis/test_decision_marker_batching.py
new file mode 100644
index 000000000..3619f72a9
--- /dev/null
+++ b/tests/unit/analysis/test_decision_marker_batching.py
@@ -0,0 +1,77 @@
+"""Every inline marker in a file reaches the model.
+
+``markers[:5]`` truncated the per-file list instead of batching it. The caller
+runs once per file and the raw-marker fallback only fires on an exception, so
+markers past the fifth were never structured, never fell back, and left no log
+line — they simply did not exist.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from types import SimpleNamespace
+
+from repowise.core.analysis.decisions.extractor import DecisionExtractor
+
+_MARKER_LINE_RE = re.compile(r"--- Marker \(\w+\) at line (\d+) ---")
+
+_MARKER_COUNT = 12
+
+
+def _source() -> str:
+ lines = []
+ for i in range(_MARKER_COUNT):
+ lines.append(f"def fn{i}():")
+ lines.append(f" # DECISION: choose strategy {i} because it is faster")
+ lines.append(f" return {i}")
+ return "\n".join(lines) + "\n"
+
+
+class _RecordingProvider:
+ """Answers each call with a decision per marker line in that prompt."""
+
+ def __init__(self) -> None:
+ self.prompts: list[str] = []
+
+ async def generate(self, system: str, prompt: str, **kwargs: object) -> SimpleNamespace:
+ self.prompts.append(prompt)
+ # Matched on the marker header specifically. Splitting on "at line "
+ # also hits the ±20 lines of source context each marker carries, and
+ # the resulting ValueError would be swallowed by the raw-marker
+ # fallback — the test would stay green while measuring nothing.
+ lines = [int(n) for n in _MARKER_LINE_RE.findall(prompt)]
+ return SimpleNamespace(
+ content=json.dumps(
+ [
+ {
+ "marker_line": line,
+ "title": f"Strategy at {line}",
+ "decision": f"choose strategy at {line}",
+ }
+ for line in lines
+ ]
+ )
+ )
+
+
+async def test_all_markers_are_structured_not_just_the_first_five(tmp_path):
+ (tmp_path / "app.py").write_text(_source(), encoding="utf-8")
+ provider = _RecordingProvider()
+
+ decisions = await DecisionExtractor(
+ repo_path=tmp_path, provider=provider
+ ).scan_inline_markers()
+
+ assert len(decisions) == _MARKER_COUNT
+ # 12 markers at 5 per call.
+ assert len(provider.prompts) == 3
+ # Every marker line is represented exactly once.
+ lines = sorted(d.evidence_line for d in decisions)
+ assert lines == sorted(2 + 3 * i for i in range(_MARKER_COUNT))
+ # These came from the structuring path, not the raw-marker fallback. Count
+ # and line numbers alone cannot tell the two apart: on any exception the
+ # fallback emits one record per marker with these same lines, so without
+ # this the test passes with LLM structuring entirely broken.
+ assert all(d.confidence == 0.95 for d in decisions)
+ assert all(d.title.startswith("Strategy at ") for d in decisions)
diff --git a/tests/unit/analysis/test_decision_marker_case.py b/tests/unit/analysis/test_decision_marker_case.py
new file mode 100644
index 000000000..4fe4325e6
--- /dev/null
+++ b/tests/unit/analysis/test_decision_marker_case.py
@@ -0,0 +1,73 @@
+"""Inline decision markers must be capitalised to count.
+
+The keyword used to match case-insensitively, which made every lower-case
+occurrence of "decision:" or "rejected:" in ordinary prose an architectural
+decision. On this repository that was not a tail case: across 3,860 tracked
+files the only two matches were both false positives, and both reached the
+store as ``active`` records.
+"""
+
+from __future__ import annotations
+
+import json
+from types import SimpleNamespace
+
+from repowise.core.analysis.decisions.extractor import MARKER_RE, DecisionExtractor
+
+
+class _StubProvider:
+ def __init__(self, payload: list[dict]) -> None:
+ self._payload = payload
+ self.calls = 0
+
+ async def generate(self, system: str, prompt: str, **kwargs: object) -> SimpleNamespace:
+ self.calls += 1
+ return SimpleNamespace(content=json.dumps(self._payload))
+
+
+def test_capitalised_keywords_still_match():
+ for keyword in ("WHY", "DECISION", "TRADEOFF", "ADR", "RATIONALE", "REJECTED"):
+ m = MARKER_RE.match(f" # {keyword}: pin the driver to 2.x")
+ assert m is not None, keyword
+ assert m.group("keyword") == keyword
+
+
+def test_prose_sentence_is_not_a_marker():
+ """The exact line that produced a garbled record on a live index.
+
+ ``reindex_cmd.py`` wraps the sentence "embedded into the shared page store
+ under the / decision: namespace, batched like the pages", so the
+ continuation line begins with the word "decision" and a colon.
+ """
+ assert MARKER_RE.match(" # decision: namespace, batched like the pages") is None
+
+
+def test_titlecase_label_is_not_a_marker():
+ """A test file's "# Rejected: nothing to extract." was the other one."""
+ assert MARKER_RE.match(" # Rejected: nothing to extract.") is None
+ assert MARKER_RE.match("# Why: we need this") is None
+
+
+async def test_lowercase_prose_yields_no_decisions(tmp_path):
+ """End to end: a file of prose false positives extracts nothing.
+
+ The provider is stubbed to return a decision for any call, so a non-empty
+ result here means the scan matched something it should not have.
+ """
+ (tmp_path / "reindex.py").write_text(
+ "def _reindex():\n"
+ " # Decision records embedded into the shared page store under the\n"
+ " # decision: namespace, batched like the pages. Uses embed_batch\n"
+ " # directly (which raises on failure).\n"
+ " # Rejected: nothing to extract.\n"
+ " return None\n",
+ encoding="utf-8",
+ )
+ provider = _StubProvider([{"title": "Ghost", "decision": "ghost"}])
+
+ decisions = await DecisionExtractor(
+ repo_path=tmp_path, provider=provider
+ ).scan_inline_markers()
+
+ assert decisions == []
+ assert provider.calls == 0
diff --git a/tests/unit/analysis/test_decision_source_failures.py b/tests/unit/analysis/test_decision_source_failures.py
new file mode 100644
index 000000000..f42db9c90
--- /dev/null
+++ b/tests/unit/analysis/test_decision_source_failures.py
@@ -0,0 +1,79 @@
+"""A decision source that fails is reported as failed, not as empty.
+
+Every source swallows its own exceptions and returns ``[]``, and the CLI pins
+``repowise.core`` to ERROR, so a source that died reached the user as the same
+zero a repository with genuinely no ADRs produces. The two are now distinct:
+``DecisionExtractionReport.failures`` names the sources that raised.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+
+from repowise.core.analysis.decisions.extractor import (
+ DecisionExtractor,
+ DecisionSourceError,
+ _collect_batches,
+)
+
+
+async def _boom() -> list:
+ raise RuntimeError("provider exploded")
+
+
+async def _empty() -> list:
+ return []
+
+
+def _extractor(tmp_path) -> DecisionExtractor:
+ return DecisionExtractor(repo_path=tmp_path, provider=SimpleNamespace())
+
+
+async def test_failed_source_is_named_in_failures(tmp_path, monkeypatch):
+ ex = _extractor(tmp_path)
+ monkeypatch.setattr(ex, "mine_git_archaeology", _boom)
+ monkeypatch.setattr(ex, "scan_inline_markers", _empty)
+ monkeypatch.setattr(ex, "discover_adrs", _empty)
+ monkeypatch.setattr(ex, "mine_pr_bodies", _empty)
+ monkeypatch.setattr(ex, "mine_comment_archaeology", _empty)
+
+ report = await ex.extract_all()
+
+ assert report.by_source["git_archaeology"] == 0
+ assert "git_archaeology" in report.failures
+ assert "provider exploded" in report.failures["git_archaeology"]
+ # A source that honestly found nothing is not in the failure map.
+ assert "adr" not in report.failures
+
+
+async def test_all_sources_succeeding_reports_no_failures(tmp_path, monkeypatch):
+ ex = _extractor(tmp_path)
+ for name in (
+ "scan_inline_markers",
+ "mine_git_archaeology",
+ "discover_adrs",
+ "mine_pr_bodies",
+ "mine_comment_archaeology",
+ ):
+ monkeypatch.setattr(ex, name, _empty)
+
+ report = await ex.extract_all()
+
+ assert report.failures == {}
+
+
+def test_partial_batch_failure_keeps_what_survived():
+ """Losing some batches degrades the source; it does not fail it."""
+ kept = _collect_batches("pr", [["a", "b"], RuntimeError("one batch died")])
+ assert kept == ["a", "b"]
+
+
+def test_every_batch_failing_is_a_failed_source():
+ with pytest.raises(DecisionSourceError):
+ _collect_batches("pr", [RuntimeError("a"), RuntimeError("b")])
+
+
+def test_no_batches_is_not_a_failure():
+ assert _collect_batches("pr", []) == []
diff --git a/tests/unit/cli/test_ui_package.py b/tests/unit/cli/test_ui_package.py
index 37f662285..bdba3a37c 100644
--- a/tests/unit/cli/test_ui_package.py
+++ b/tests/unit/cli/test_ui_package.py
@@ -118,7 +118,6 @@ def test_advanced_config_default_keys_no_fast(monkeypatch: pytest.MonkeyPatch) -
"embedder": "mock",
"test_run": False,
"onboarding": True,
- "harvest_decisions": True,
"wiki_style": DEFAULT_STYLE,
"language": "en",
}
@@ -148,7 +147,7 @@ def test_advanced_config_index_only_omits_generation_keys(
"embedder": "mock",
}
# The knobs that only shape a model's writing must not appear.
- for key in ("concurrency", "reasoning", "onboarding", "harvest_decisions", "wiki_style"):
+ for key in ("concurrency", "reasoning", "onboarding", "wiki_style"):
assert key not in result
diff --git a/tests/unit/generation/test_decision_harvest.py b/tests/unit/generation/test_decision_harvest.py
deleted file mode 100644
index 8b180db50..000000000
--- a/tests/unit/generation/test_decision_harvest.py
+++ /dev/null
@@ -1,186 +0,0 @@
-"""Phase-2 LLM-docs decision harvest: parsing, the anti-hallucination gate,
-and the harvest-directive toggle.
-
-These cover the critical guardrail — because the generator now *generates*
-decision candidates, a fluent-but-invented rationale/quote that is not grounded
-in the file's source must be dropped, and a candidate with no surviving
-grounded field must be rejected outright.
-"""
-
-from __future__ import annotations
-
-from unittest.mock import MagicMock
-
-from repowise.core.generation.models import GenerationConfig
-from repowise.core.generation.page_generator.core import PageGenerator
-from repowise.core.generation.page_generator.decision_harvest import (
- HARVEST_DIRECTIVE,
- harvest_decisions,
- parse_and_strip_decisions,
-)
-
-# A small verbatim file source the harvested quotes are gated against.
-_SOURCE = (
- "import redis\n\n"
- "# We use Redis as the cache backend because it gives us sub-millisecond\n"
- "# reads and a shared store across workers.\n"
- "CACHE = redis.Redis(host='localhost')\n"
-)
-
-
-def _fenced(json_body: str) -> str:
- return (
- f"# Overview\n\nThis module wires the cache.\n\n```repowise-decisions\n{json_body}\n```\n"
- )
-
-
-# ---------------------------------------------------------------------------
-# parse_and_strip_decisions
-# ---------------------------------------------------------------------------
-
-
-def test_parse_strips_fence_and_returns_payload():
- content = _fenced('{"decisions": [{"title": "Use Redis", "source_quote": "use Redis"}]}')
- clean, decisions = parse_and_strip_decisions(content)
-
- assert "repowise-decisions" not in clean
- assert "```" not in clean
- assert clean.endswith("wires the cache.")
- assert len(decisions) == 1
- assert decisions[0]["title"] == "Use Redis"
-
-
-def test_parse_no_fence_is_passthrough():
- content = "# Overview\n\nNothing to harvest here.\n"
- clean, decisions = parse_and_strip_decisions(content)
- assert clean == content
- assert decisions == []
-
-
-def test_parse_malformed_json_does_not_corrupt_page():
- content = _fenced("{not valid json")
- clean, decisions = parse_and_strip_decisions(content)
- # Fence still stripped; payload simply empty rather than raising.
- assert "repowise-decisions" not in clean
- assert decisions == []
-
-
-def test_parse_drops_titleless_items():
- content = _fenced('{"decisions": [{"source_quote": "use Redis"}, {"title": "Use Redis"}]}')
- _clean, decisions = parse_and_strip_decisions(content)
- assert [d["title"] for d in decisions] == ["Use Redis"]
-
-
-# ---------------------------------------------------------------------------
-# harvest_decisions — the gate
-# ---------------------------------------------------------------------------
-
-
-def test_grounded_decision_is_kept_as_llm_inferred():
- content = _fenced(
- '{"decisions": [{"title": "Use Redis for caching", '
- '"decision": "We use Redis as the cache backend", '
- '"rationale": "sub-millisecond reads and a shared store across workers", '
- '"source_quote": "We use Redis as the cache backend"}]}'
- )
- clean, decisions = harvest_decisions(content, source_text=_SOURCE, evidence_file="cache.py")
-
- assert "repowise-decisions" not in clean
- assert len(decisions) == 1
- d = decisions[0]
- assert d["source"] == "llm_inferred"
- assert d["status"] == "proposed"
- assert d["evidence_file"] == "cache.py"
- assert d["affected_files"] == ["cache.py"]
- # Grounded in the verbatim source → exact verdict, fields preserved.
- assert d["verification"] == "exact"
- assert d["decision"] == "We use Redis as the cache backend"
- # Transient source span must never leak into persistence.
- assert d["source_text"] == ""
-
-
-def test_hallucinated_decision_is_rejected():
- # None of the produced fields appear in the source — a pure fabrication.
- content = _fenced(
- '{"decisions": [{"title": "Adopt Kafka event bus", '
- '"decision": "All services publish domain events to Kafka", '
- '"rationale": "decouples producers from consumers at scale", '
- '"source_quote": "All services publish domain events to Kafka"}]}'
- )
- clean, decisions = harvest_decisions(content, source_text=_SOURCE, evidence_file="cache.py")
- assert decisions == [], "an ungrounded harvested decision must be dropped by the gate"
- assert "repowise-decisions" not in clean
-
-
-def test_partial_hallucination_drops_only_ungrounded_fields():
- # source_quote is grounded; rationale is invented. The decision survives
- # with the invented field cleared.
- content = _fenced(
- '{"decisions": [{"title": "Use Redis", '
- '"decision": "We use Redis as the cache backend", '
- '"rationale": "mandated by the platform compliance committee", '
- '"source_quote": "We use Redis as the cache backend"}]}'
- )
- _clean, decisions = harvest_decisions(content, source_text=_SOURCE, evidence_file="cache.py")
- assert len(decisions) == 1
- d = decisions[0]
- assert d["decision"] == "We use Redis as the cache backend"
- assert d["rationale"] == "", "the ungrounded rationale must be dropped"
-
-
-def test_title_promoted_to_decision_when_gate_clears_paraphrase():
- # The decision/rationale are real but paraphrased, so the substring gate
- # clears them; the verbatim source_quote keeps the record alive. Without the
- # title fallback this leaves a body-less, title-only record. The title (the
- # model's canonical one-line summary) must be promoted into ``decision``.
- content = _fenced(
- '{"decisions": [{"title": "Use Redis for caching", '
- '"decision": "Persistence layer selected by the platform team", '
- '"rationale": "latency targets demanded an alternative approach", '
- '"source_quote": "We use Redis as the cache backend"}]}'
- )
- _clean, decisions = harvest_decisions(content, source_text=_SOURCE, evidence_file="cache.py")
- assert len(decisions) == 1
- d = decisions[0]
- # Paraphrased prose was cleared by the gate...
- assert d["rationale"] == ""
- # ...but the record is not body-less: the title filled the empty decision.
- assert d["decision"] == "Use Redis for caching"
- # The grounding quote that kept the record alive is still recorded.
- assert d["source_quote"] == "We use Redis as the cache backend"
-
-
-def test_no_fence_yields_no_decisions():
- clean, decisions = harvest_decisions(
- "# Overview\n\nPlain page.\n", source_text=_SOURCE, evidence_file="cache.py"
- )
- assert decisions == []
- # No fence → content returned verbatim (only the fence path trims).
- assert clean == "# Overview\n\nPlain page.\n"
-
-
-# ---------------------------------------------------------------------------
-# Harvest directive toggle (the --no-harvest-decisions escape hatch)
-# ---------------------------------------------------------------------------
-
-
-def _make_generator(harvest: bool) -> PageGenerator:
- provider = MagicMock()
- provider.model_name = "mock-model"
- provider.provider_name = "mock"
- config = GenerationConfig(harvest_decisions=harvest)
- return PageGenerator(provider, MagicMock(), config)
-
-
-def test_directive_reaches_no_system_prompt():
- """Harvesting only ever ran on ``file_page``, and a file page no longer
- goes near a model, so nothing carries the directive.
-
- The parser below still has tests because the payload format is worth
- keeping intact, but there is no producer feeding it. Whether harvesting
- gets a new host or is retired is a decision of its own rather than a side
- effect of the file layer becoming structural.
- """
- gen = _make_generator(harvest=True)
- for page_type in ("module_page", "repo_overview", "architecture_diagram", "onboarding"):
- assert HARVEST_DIRECTIVE not in gen._build_system_prompt(page_type)
diff --git a/tests/unit/generation/test_page_generator.py b/tests/unit/generation/test_page_generator.py
index 27ac3ef89..ebc2a8861 100644
--- a/tests/unit/generation/test_page_generator.py
+++ b/tests/unit/generation/test_page_generator.py
@@ -695,11 +695,7 @@ async def test_generate_all_level_values_in_range():
def _gen(language: str = "en") -> PageGenerator:
- # Harvest disabled here so these assert the language-prefix logic in
- # isolation; the harvest-directive suffix is covered in test_decision_harvest.
- config = GenerationConfig(
- max_tokens=256, token_budget=500, max_concurrency=1, harvest_decisions=False
- )
+ config = GenerationConfig(max_tokens=256, token_budget=500, max_concurrency=1)
provider = MockProvider()
assembler = ContextAssembler(config)
return PageGenerator(provider, assembler, config, language=language)
@@ -736,7 +732,7 @@ def test_language_defaults_from_config_when_arg_omitted():
# Callers that only build a GenerationConfig (server regenerate, pipeline
# fallback) must still get the configured output language.
config = GenerationConfig(
- max_tokens=256, token_budget=500, max_concurrency=1, harvest_decisions=False, language="ru"
+ max_tokens=256, token_budget=500, max_concurrency=1, language="ru"
)
gen = PageGenerator(MockProvider(), ContextAssembler(config), config)
prompt = gen._build_system_prompt("module_page")
diff --git a/tests/unit/pipeline/test_decision_source_reporting.py b/tests/unit/pipeline/test_decision_source_reporting.py
new file mode 100644
index 000000000..cc2a115d5
--- /dev/null
+++ b/tests/unit/pipeline/test_decision_source_reporting.py
@@ -0,0 +1,97 @@
+"""The decisions phase says which sources failed, separately from which were empty.
+
+``_run_decision_extraction`` renders two lists. A source that found nothing
+belongs in the informational one; a source that raised belongs in the warning
+one, and must not appear in both — "Nothing found in: pull requests" is exactly
+the reassuring sentence a failed source used to produce.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+
+from repowise.core.pipeline.phases.analysis import _run_decision_extraction
+
+
+class _RecordingProgress:
+ def __init__(self) -> None:
+ self.messages: list[tuple[str, str]] = []
+
+ def on_message(self, level: str, text: str) -> None:
+ self.messages.append((level, text))
+
+ def on_phase_start(self, *_a: object, **_k: object) -> None: ...
+ def on_item_done(self, *_a: object, **_k: object) -> None: ...
+ def on_phase_done(self, *_a: object, **_k: object) -> None: ...
+
+ def text_at(self, level: str) -> str:
+ return " ".join(t for lvl, t in self.messages if lvl == level)
+
+
+class _StubExtractor:
+ """Stands in for DecisionExtractor: pr raised, git/comment found nothing."""
+
+ def __init__(self, *_a: object, **_k: object) -> None: ...
+
+ async def extract_all(self, on_step=None, enabled_sources=None):
+ return SimpleNamespace(
+ total_found=1,
+ decisions=[SimpleNamespace(source="inline_marker")],
+ by_source={
+ "inline_marker": 1,
+ "git_archaeology": 0,
+ "adr": 0,
+ "pr": 0,
+ "comment": 0,
+ },
+ failures={"pr": "RuntimeError: provider exploded"},
+ )
+
+
+def _patched(monkeypatch, tmp_path):
+ import repowise.core.analysis.decision_extractor as de
+
+ monkeypatch.setattr(de, "DecisionExtractor", _StubExtractor)
+ monkeypatch.setattr(
+ de,
+ "enabled_source_names",
+ lambda _cfg: ("inline_marker", "git_archaeology", "adr", "pr", "comment"),
+ )
+ return tmp_path
+
+
+async def test_failed_source_warns_and_is_kept_out_of_the_empty_list(_patched, monkeypatch):
+ progress = _RecordingProgress()
+
+ # The session miner is a separate pass; keep it out of this assertion.
+ import repowise.core.sessions.miners.decisions as miners
+
+ monkeypatch.setattr(miners, "session_mining_enabled", lambda _cfg: False)
+
+ await _run_decision_extraction(
+ _patched,
+ llm_client=SimpleNamespace(),
+ graph_builder=SimpleNamespace(graph=lambda: None),
+ git_meta_map={},
+ parsed_files=[],
+ progress=progress,
+ )
+
+ warnings = progress.text_at("warning")
+ info = progress.text_at("info")
+
+ # The failure is reported, names the source, and carries the reason.
+ assert "pull requests" in warnings
+ assert "provider exploded" in warnings
+ # And it is NOT also reported as an honest zero.
+ assert "Nothing found in" in info
+ nothing_line = next(
+ t for lvl, t in progress.messages if lvl == "info" and "Nothing found" in t
+ )
+ assert "pull requests" not in nothing_line
+ # Sources that genuinely found nothing still appear there.
+ assert "git history" in nothing_line
+ assert "ADR files" in nothing_line