Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

feat: 2.0 groundwork (profiles, source ranking, runs, browser lane, verification) (#48)

Jordan Gibbs5c8f90cdd3fd
diff --git a/CHANGELOG.md b/CHANGELOG.md
index da3938d..1007c23 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,73 @@
# Changelog
-## [0.8.7] - 2026-07-18
+## [Unreleased]
+
+### Per-agent models are now real config; dollar-cost claims removed
+
+- **ModelMap wired into agent rendering.** The profile's per-agent model map existed but was decorative — every installed agent's `model:` frontmatter line was hardcoded in `hooks.py`, so overriding models via a profile silently did nothing. All 16 agent templates now carry `model: << p.models.X >>`, rendered from the profile at install time, and `ModelMap` gained the two missing agents (`cite_checker`, `browser_fetcher`) plus validation (non-empty; aliases or full model IDs both accepted). Full flexibility per agent: `[profile.full]` + `models = { fetcher = "haiku" }` swaps every fetcher to Haiku on the next install/`profile use`; unspecified agents keep their defaults. Defaults are unchanged (verified byte-identical by the goldens).
+- **Model names left the prose.** Agent descriptions and skill text no longer claim "Runs on Sonnet/Opus" (or Sonnet-specific context-window sizes) — the rendered `model:` line is the single source of truth, so a model override can never be contradicted by stale prose. Install-action labels and code comments were de-modeled to match. Drift-proofed by tests: any literal `model:` line in an agent template, or any "Runs on <model>" claim in rendered output, fails the suite.
+- **Dollar-cost estimates removed from the product** (the local benchmark harness keeps its billing-aware cost math). On subscription billing, Claude Code's `cost_usd` is an API-equivalent valuation, not a charge — stating costs as prices contradicted how most users run the pipeline. Gone: the `cost_estimate` profile field and its four builtin values, the router tier table's cost column, `profile list/use` cost output, gap-fetch's "+$1-3 per run", and the source-analyst's "$2-5 per spawn" block (now "Effort discipline"). Time estimates remain. The opt-in budget governor (`run init --budget`) stays, relabeled as a ceiling on *estimated API-equivalent* spend; `run status/report` spend lines now say "API-equiv". A rendered-prompt test rejects any future `$N-M` range.
+
+### Benchmark harness: gear-aware, one-command, and citation-correct
+
+- **Fixed: wrapped runs shipped citation-free reports.** The step-1 skill has always documented "the benchmark harness sets `inline` via wrapper_contract" — but the harness never wrote that file. Every wrapped run therefore fell back to the `wikilink` default, shipping vault-internal `[[note-id]]` markers that `evaluate.py` strips before grading, so reports reached the FACT citation evaluator with **zero verifiable URLs**. Confirmed on the existing fleet: `runs_layercake/query_62` is 7,932 words with 0 numbered citations, 0 URLs. The harness now writes `research/wrapper_contract.json` (`citation_style: inline` + required terminal sections), the pipeline prompt states the requirement explicitly, and `citation-style-preservation` + `quote-integrity` joined the post-run validation rules so a regression is caught per query.
+- **Harness modernized for V8.** The pipeline prompt invoked `/research-layercake` (retired), and the startup banner checked for V7 `layercake-*` skills expecting 14 — now `/hyperresearch` and 18 `hyperresearch-*` step skills. The timeout was hardcoded at 3600s, which would have silently timed out (and discarded) every premier run; it is now gear-scaled (1h at `full`, 6h at `premier`).
+- **Gear-aware fleets.** Scale comes from the installed gear, not a flag — `_setup_run_dir` copies the project root's rendered `.claude/` + `config.toml` into every run dir. Non-`full` gears now get their own runs dir (`bench/runs_layercake-premier/`) and tag suffix so fleets never overwrite each other; `evaluate.py` gained a matching `--gear` flag (it previously hardcoded `runs_layercake` and stripped the tag by a fixed length, which a gear suffix breaks).
+- **`bench/compare.py`** — head-to-head fleet comparison on shared queries only: RACE sub-scores where graded, plus structural metrics (words, citations, citation density, unique URLs, vault sources, cost, duration) that need no API key. Flags the citation confound when comparing against pre-fix fleets.
+- **`bench/run_premier.py`** — one command: preflight (CLIs, queries, gear validity, grading keys) → set + verify gear → cost confirmation → run → RACE/FACT evaluation → comparison.
+
+### Scale gears: premier profile + `hpr profile use`
+
+- **New built-in `premier` profile** — the flat pipeline at ~2× scale: 100–130 sources (min 90), 80–160 planned searches, 14–18 wave-1 fetchers, 10 loci with a doubled depth budget (80), and a widened downstream funnel (claims 150–220, must-read 50–70, 8–16K words, 120–220 citations, raised critic caps) so the extra corpus actually reaches the page instead of stranding in the vault. Estimate: ~3–5 hours.
+- **Gears vs tiers, made explicit.** `light` and `dissertation` are run-time *tiers* (auto-classified / opt-in per query); `full` and `premier` are install-time *gears* — the profile whose numbers are rendered into the skill and agent prompts. The router now carries a "Scale gear (tier ≠ gear)" section, and the width-sweep's full-tier numbers follow the gear (`p.*`) instead of being pinned to the `full` profile (byte-identical under the default gear, verified by goldens).
+- **`hpr profile use <name>`** — the one-command gear shift: validates the profile, re-renders every installed skill/agent prompt, and persists the choice under `[pipeline] profile` in config.toml so later bare `install` runs (e.g. after upgrades) keep the gear. Refuses `light`/`dissertation` with an explanation (they're tiers). `hpr profile list` now shows descriptions, source targets, time estimates, and marks the current gear.
+- **Fix: `[profile.*]` overlays survive config saves.** `VaultConfig.save()` previously dropped user-defined profile tables — any config write (e.g. the crawl4ai auto-setup) silently destroyed custom pipeline profiles. Overlays now round-trip verbatim, including nested inline tables and array-of-array values.
+
+### Phase 5 (2.0 roadmap): verification — the layer that makes it trustworthy
+
+- **Cite-check (new step 14.5, full + dissertation).** Every citation is verified as a citation-sentence BINDING before ship. `hpr citecheck extract` parses (sentence, citation) pairs for both citation styles and mechanically auto-passes pairs whose numbers/wording the claims table confirms; dangling citations (resolving to no vault note) are instant critical findings. The sampled remainder (100% of number-bearing sentences, deterministic sampling for the rest — resume-safe, no RNG) goes to the new `hyperresearch-cite-checker` agent, whose verdicts default skeptical (`supported / partially-supported / unsupported / wrong-source`); findings feed a second, small tool-locked patcher pass. 18 step skills now.
+- **Three verification lint rules.** `quote-integrity` (error): every quoted span ≥5 words must exist verbatim in a vault note — hallucinated quotes cannot ship. `numeric-consistency` (warning): report numbers untraceable to claims or cited-note bodies are flagged for verification. `retracted-citations` (error): citing a retracted source blocks the gate unless the citation itself acknowledges the retraction (sometimes the retraction IS the story).
+- **Ship-time retraction sweep.** `hpr sources retractions --tag` re-checks every DOI-bearing note fresh (cache-bypassing), so a retraction published yesterday is caught today — including on vault sources being REUSED from prior runs.
+- **Independence audit.** `hpr sources independence` clusters derivative sources (canonical-URL identity with tracking-param stripping, near-duplicate bodies via MinHash Jaccard, shared wire-service boilerplate keyed on the body opening — outlets retitle, the wire text doesn't change) and scores members `1/cluster_size`. Step 3's consensus rule now counts independence-weighted voices, so five syndicated copies of one press release are ONE vote, not five.
+- **Run telemetry + verification battery.** `hpr run report [--all]` rolls up per-step wall-time, spend, and event counts from the manifest + events log — the feedback loop for tuning profile constants against observed yield. `hpr run verify` is the CI-able structural gate (report exists, required headings, length in profile range ±20%, citation density ≥1.5/1000 chars, tier artifacts present, scaffold-leak check, cite-check criticals resolved; exit 1 on failure) — wired into the router's final integrity gate alongside the new lint rules.
+- **Bench note.** The phase plan's nightly bench-smoke CI workflow was NOT created: `bench/` turns out to be entirely gitignored (local-only), so a workflow referencing it would be broken by construction. `hpr run verify` is the shipped, CI-able equivalent; local bench scratch (`_*.py` etc.) was tidied into `bench/archive/`.
+
+### Phase 4 (2.0 roadmap): the browser lane — blocked fetches escalate instead of dying
+
+- **Escalation queue (schema v10).** Fetches that hit login walls or bot/captcha walls are no longer discarded — the fetch gate queues them in a new race-safe `escalations` table (SQLite atomic claim semantics; the phase doc's JSON-file design was upgraded deliberately) with reason, utility score, and provenance. Error codes distinguish `AUTH_REQUIRED_ESCALATED` / `JUNK_ESCALATED` so fetcher agents don't retry. Content-quality junk (404s, empty pages) still dies — a 404 in Chrome is still a 404. Policy under `[chrome]`: `enabled`, `escalation_utility_threshold` (low-value blocked URLs are abandoned, the lane is serial and precious), `max_items_per_run`.
+- **`hpr escalation` CLI** — `list/add/claim/complete-via-ingest/human/retry/abandon`. `claim` is atomic under concurrent claimers; `ingest` is the one-shot completion (writes the vault note with `fetch_provider: chrome`, records the source row, syncs, resolves the item) so the browser agent can't half-finish bookkeeping.
+- **`hyperresearch-browser-fetcher` agent** — drains the queue by driving the user's real Chrome browser via Claude-in-Chrome (batched ToolSearch load, new tab always, one instance at a time). Playbook: infinite scroll caps, SPA expansion, PDF-viewer text layers, screenshot transcription for chart-heavy pages, and a Google Scholar lane (`reason: scholar_search` items carry a query; results ingest as structured notes, high-citation hits re-queue). **Hard scope boundary, stated in the prompt and enforced by the workflow: CAPTCHAs, 2FA, and logins are never solved automatically — items go to `needs_human`.**
+- **Human-in-the-loop checkpoint.** All `needs_human` items are consolidated into ONE user prompt at a natural pause point (never per-URL interruptions); after the human completes challenges in their own browser, `escalation retry` + one more drain. Non-interactive runs record `run block --on human-challenges` and continue with everything else; `hpr run status` shows queue depth and needs_human counts.
+- **Graceful degradation.** Without the Claude-in-Chrome extension the queue simply accumulates (visible in `run status`); the floor is the pre-4.0 status quo. Session handoff (Chrome cookies → crawl4ai profile) was evaluated and deliberately NOT implemented: HttpOnly cookies aren't scriptable from page JS, so the honest v1 is the Chrome lane itself plus the existing `hyperresearch setup` guided-login flow.
+
+### Phase 3 (2.0 roadmap): dissertation scale — run isolation, manifest, chapters
+
+- **Per-run workspaces.** Every run-scoped pipeline artifact (scaffold, decomposition, loci, comparisons, critic findings, patch/polish logs, temp scratch, canonical query) now lives under `research/runs/<vault_tag>/` — concurrent and sequential runs can never collide (closes the parallel-run race flagged in 0.8.6). Vault notes stay global; final reports stay at `research/notes/final_report_<tag>.md`. Sync never ingests run workspaces; lint rules resolve artifacts run-aware with full legacy flat-path fallback for pre-3.0 vaults; `vault-tag` collision checks cover run dirs.
+- **Run manifest + explicit resume.** `hpr run init/status/resume/abort/step/spend/event` — `run.json` records per-step + per-chapter status, spend counters, and a heartbeat; `hpr run resume` returns the exact next step and Skill invocation (replacing artifact-scan recovery, which remains as fallback). `run status` flags possibly-stalled runs.
+- **Budget governor.** `run init --budget <usd>` sets a hard ceiling; crossing it flips the run to `blocked (budget)`. The router instructs shrinking fan-out near the ceiling instead of silently skipping profile-mandated steps.
+- **`dissertation` profile + chaptered execution.** New built-in profile (opt-in only, never auto-classified): 250–450 sources across 4–10 chapters, each chapter running the proven 40–80-source pipeline envelope (steps 2–10 loop per chapter, ≤2 in flight), global reconcile/synthesis on top, 25K–80K-word chaptered output, scaled critic caps, multi-hour pacing. New step skill `hyperresearch-1-5-chapter-partition` (17 step skills now).
+- **Literature-review matrix + meta-analysis substrate.** `hpr claims matrix` generates the per-source review table (tier, venue, citations, quality, key finding) from the claims table; `hpr claims targets` groups claims by stance_target across sources with stance splits and source-attributed numbers for comparison tables.
+
+### Phase 2 (2.0 roadmap): the source-ranking engine
+
+Source quality becomes a persistent, queryable property instead of ephemeral prompt prose (schema v9, additive-only).
+
+- **Per-source quality scores.** New note columns: `doi`, `utility_score`, `citation_count`, `venue`, `is_retracted` (frontmatter-mirrored — markdown stays truth) plus derived `authority_score`, `centrality_score`, `independence`, `quality_score` (DB-cache, recomputed, survive re-sync).
+- **`hpr sources score`** — enriches DOI-bearing notes from OpenAlex/Semantic Scholar (citation counts, venue, **retraction flags**), cached in a new `api_cache` table (`[ranking] api_cache_ttl_days`, default 30). `hpr sources backfill-doi` regexes the back-catalog; new fetches capture DOIs/arXiv ids automatically, and `fetch --utility-score` persists the step-2 utility score that was previously discarded after fetch selection.
+- **`hpr graph rank`** — pure-Python PageRank over the link + provenance-breadcrumb graph; centrality here means "many independent research chains converged on this source". Also recomputed during `repair`.
+- **Composite `quality_score`** — renormalized weighted blend of tier weight, utility, citation-authority percentile (vault-relative, log-scaled), and centrality; retracted sources floored at 0.05. Weights configurable under `[ranking]`.
+- **`hpr search --ranked`** — folds `quality_score` into FTS relevance (`(0.5 + quality)` multiplier; unscored notes stay neutral). Default search behavior unchanged.
+- **Claims table** — `hpr claims ingest/list/search` persists fetcher-extracted `claims-*.json` into queryable `claims` + `claims_fts` tables keyed to source notes (idempotent by content hash). This is the substrate for phase-5 cite-checking.
+- **Semantic search (embeddings table revived)** — `hpr embed sync` + `hpr search --semantic` (RRF hybrid with FTS). Provider-pluggable under `[embeddings]`: `none` (default — zero API keys required), `voyage`, `openai`. Brute-force cosine, no vector-DB dependency.
+- **Pipeline integration** — step 2 gains step 2.7 (persist ranking signals after the last wave; retracted sources flagged before they can anchor a locus); fetcher batches carry utility scores; step 10 curates from `search --ranked` instead of orchestrator intuition.
+
+### Phase 1 (2.0 roadmap): config extraction + pipeline profiles
+
+- **Every behavioral constant is now config** (`docs/roadmap-2.0/phase-1-config-profiles.md` WS1). New `.hyperresearch/config.toml` sections: `[fetch]` (page/PDF timeouts, smart-wait polling, visible-browser domain list, image timeout), `[junk]` (content gates, binary-garbage ratio, extra signal lists), `[assets]` (max images, min image bytes), `[dedup]` (MinHash/LSH parameters, threshold), `[lint]` (extract-coverage and stale-review thresholds), plus `[search]` output defaults (`default_limit`, `chars_per_token`, `snippet_len`). All defaults reproduce prior behavior.
+- **BEHAVIOR CHANGE — PDF downloads verify TLS by default.** The PDF fetch path previously hardcoded `verify=False`, silently disabling certificate verification. New `[fetch] pdf_verify_tls` defaults to `true` (secure). Set it to `false` explicitly for cert-broken mirrors you trust.
+- **Pipeline profiles** (`hyperresearch profile list/show/validate`). Every research-scale knob — source gates, fetcher fan-out, loci caps, depth budgets, draft counts, word targets, critic caps, per-agent models — lives in a named, validated profile. Built-in `full` and `light` reproduce the shipped V8 values exactly; users override keys or define new profiles (`[profile.dissertation] extends = "full"`) in config.toml.
+- **Skill/agent prompts are now templates rendered at install.** `hyperresearch install --profile <name>` renders the 17 skills and 15 agent prompts from the chosen profile (custom `<< >>` Jinja delimiters — prompt-native `{{...}}` placeholders and JSON braces pass through untouched). Rendered files carry a `rendered from profile "..."` provenance header after the frontmatter. Golden tests pin the `full`-profile render byte-for-byte against the pre-template prompts, so profile/template drift is a test failure, not a silent prompt change.
+- **Width-sweep consistency fix** (roadmap phase-0 WS5, pulled forward): the three contradictory full-tier source-target statements (40-100 / 40–80 / 55–80) are unified to the profile value (55–80); the light target (12–20 vs 15–25) and the tier-table fetchers-per-wave (8–12 vs 10–12) are likewise unified to the table/Wave-1 values.
Community-fix release: five contributed PRs plus two maintainer follow-ups, closing #32, #33, #35, #37, and #39.
diff --git a/README.md b/README.md
index 7b3b4fa..a83d5e1 100644
--- a/README.md
+++ b/README.md
@@ -13,14 +13,24 @@
---
-**Hyperresearch turns Claude Code into a deep research agent. and currently leads the DeepResearch-Bench RACE leaderboard (benchmarked internally).** A tier-adaptive 16-step pipeline produces adversarially-audited reports with full source provenance. Every fetched source lands in a persistent, searchable vault that compounds across sessions.
+**Hyperresearch turns Claude Code into a deep research agent: one that currently leads the DeepResearch-Bench RACE leaderboard (benchmarked internally).** A tier-adaptive 16-step pipeline takes one prompt and produces an adversarially-audited report with full source provenance. Every source it reads lands in a persistent, searchable vault, so each session starts smarter than the last.
<p align="center">
- <img src="assets/benchmark.png" alt="DeepResearch-Bench top-5 — hyperresearch leads the chart ahead of Grep Deep Research, Cellcog Max, nvidia-aiq, Gemini Deep Research, and OpenAI Deep Research" width="780">
+ <img src="assets/benchmark.png" alt="DeepResearch-Bench top-5 hyperresearch leads the chart ahead of Grep Deep Research, Cellcog Max, nvidia-aiq, Gemini Deep Research, and OpenAI Deep Research" width="780">
</p>
<p align="center"><sub>Forward-looking projection from a stratified pilot against the DeepResearch-Bench leaderboard snapshot (https://huggingface.co/spaces/muset-ai/DeepResearch-Bench-Leaderboard). Third party validation is pending.</sub></p>
+## Why it wins
+
+- **250+ sources in a single run.** The `premier` scale profile targets 100–130 in the width sweep alone; citation chasing and gap-fill fetches more than double what actually lands in the corpus.
+- **Every citation is verified before the report ships.** A skeptical cite-checker audits whether each cited source actually supports its sentence. Hallucinated quotes and unacknowledged retractions are hard blocks at the gate.
+- **Syndication doesn't count as consensus.** An independence audit clusters derivative copies, so five reprints of one press release argue with the weight of one source.
+- **Adversarial by construction.** Four critics attack every draft in parallel, and a tool-locked patcher can only apply surgical edits. It physically cannot rewrite the report.
+- **Nothing is thrown away.** Every source lands in a searchable markdown-plus-SQLite vault that your next session reuses before it fetches anything new.
+- **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died.
+- **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs write 25K–80K words across chapters, from 300–450 sources.
+
## Install
```bash
@@ -30,7 +40,7 @@ pip install hyperresearch && hyperresearch install
Then `/hyperresearch <anything>` in Claude Code.
-> Python 3.11–3.13. (3.14 not yet supported — use `pyenv install 3.13`, `uv venv -p 3.13`, or `py -3.13 -m venv .venv`.)
+> Python 3.11–3.13. (3.14 not yet supported. Use `pyenv install 3.13`, `uv venv -p 3.13`, or `py -3.13 -m venv .venv`.)
>
> Power users: `hyperresearch install --global` makes `/hyperresearch` reachable from every Claude Code session anywhere, at the cost of ~15 lines in every session's system reminder. Per-project install (above) keeps unrelated CC sessions clean.
@@ -38,12 +48,13 @@ Then `/hyperresearch <anything>` in Claude Code.
## The 16-step research pipeline
-The entry skill is a thin router. It bootstraps the canonical research query, then invokes one step skill per pipeline phase via Claude Code's `Skill` tool. Each step's procedure is loaded fresh into context only when needed defeating context-rot problems that makes long pipelines silently drop steps.
+The entry skill is a thin router. It pins down the canonical research query, then invokes one step skill per phase via Claude Code's `Skill` tool. Each step's procedure loads into context only when that step actually runs. That's what stops a long pipeline from quietly dropping steps as its context rots.
| # | Step | What it does | Tiers |
|---|---|---|---|
-| 1 | Decompose | Canonical query → atomic items + coverage matrix + tier classification | both |
-| 2 | Width sweep | Multi-perspective search plan + parallel fetcher waves (Haiku) | both |
+| 1 | Decompose | Canonical query → atomic items + coverage matrix + tier classification | all |
+| 1.5 | Chapter partition | Group atomic items into 4–10 chapters; steps 2–10 then loop per chapter | dissertation |
+| 2 | Width sweep | Multi-perspective search plan + parallel fetcher waves | all |
| 3 | Contradiction graph | Pair contradictions across the corpus into ranked clusters | full |
| 4 | Loci analysis | Two parallel loci-analysts → scored loci with source budgets | full |
| 5 | Depth investigation | K parallel depth-investigators → interim notes with committed positions | full |
@@ -51,35 +62,49 @@ The entry skill is a thin router. It bootstraps the canonical research query, th
| 7 | Source tensions | Extract expert disagreements → source-tensions.json | full |
| 8 | Corpus critic | "What source would overturn this?" + targeted gap-fill fetch | full |
| 9 | Evidence digest | Top claims + verbatim quotes → evidence-digest.md | full |
-| 10 | Triple draft | Per-angle source curation + 3 parallel draft sub-orchestrators (light: single draft) | both |
+| 10 | Triple draft | Per-angle source curation + 3 parallel draft sub-orchestrators (light: single draft) | all |
| 11 | Synthesize | Plan + outline + spawn synthesizer subagent → final_report.md | full |
| 12 | Critics | 4 adversarial critics in parallel → findings JSONs | full |
| 13 | Gap-fetch | Targeted fetch wave for critic-identified vault gaps | full |
| 14 | Patcher | Surgical Edit hunks applied to draft (tool-locked Read+Edit) | full |
-| 15 | Polish | Hygiene + filler pass (tool-locked Read+Edit subagent) | both |
-| 16 | Readability audit | Recommender writes JSON suggestions; orchestrator selectively applies | both |
+| 14.5 | Cite-check | Verify citation-sentence bindings; skeptical LLM spot-check; second surgical patch pass | full |
+| 15 | Polish | Hygiene + filler pass (tool-locked Read+Edit subagent) | all |
+| 16 | Readability audit | Recommender writes JSON suggestions; orchestrator selectively applies | all |
-### Depth Modes
+### Tiers and gears: the two scale levers
-In your prompt, you can request one of two tiers and the rest of the pipeline scales accordingly. Full mode is default.
+**Tiers** route per query. Step 1 auto-classifies `light` vs `full`. `dissertation` is opt-in only; ask for it in your prompt.
-| Tier | Steps that run | Typical time |
+| Tier | What runs | Typical time |
|---|---|---|
-| `light` | bounded factual queries, surveys, comparisons — 1 → 2 → 10 → 15 → 16 | ~30–40 min |
-| `full` | deep argumentative analysis with adversarial review — all 16 steps | ~1.5–2.5 hours |
+| `light` | bounded factual queries, surveys, comparisons: 1 → 2 → 10 → 15 → 16 | ~30–40 min |
+| `full` (default) | deep argumentative analysis with adversarial review: all 16 steps + cite-check | ~1.5–2.5 h at `full` gear |
+| `dissertation` | chaptered mega-runs: 300–450 sources across 4–10 chapters, 25K–80K words | ~4–8 hours |
+
+**Gears** set the scale of the standard pipeline: the source targets, depth budgets, and word targets rendered into the step skills.
+
+```bash
+hyperresearch profile list # all profiles + descriptions + current gear
+hyperresearch profile use premier # 100–130 sources, doubled depth budget (~3–5 h)
+hyperresearch profile use full # back to the 55–80-source baseline
+```
+
+The gear persists per project and survives reinstalls. Custom gears: define `[profile.<name>]` in `.hyperresearch/config.toml` (any knob: source targets, loci caps, draft counts, word targets, per-agent models) and `profile use <name>`.
### The two load-bearing principles
1. **Patch, never regenerate.** After step 11 produces the synthesized report (or step 10 for light tier), the only modifications are surgical Edit hunks. The patcher and polish auditor are tool-locked to `[Read, Edit]` at the Claude Code allowlist level so they physically cannot Write a new draft. Per-hunk caps make "just rewrite it" mechanically impossible. Critic findings that don't fit a small hunk escalate as structural issues.
-2. **Canonical research query is gospel.** The verbatim user prompt is persisted to `research/query-<vault_tag>.md` once and re-read by every subsequent step and every spawned subagent. Wrapper requirements (save paths, citation format, terminal sections) are a separate contract.
+2. **Canonical research query is gospel.** The verbatim user prompt is persisted to `research/runs/<vault_tag>/query.md` once and re-read by every subsequent step and every spawned subagent. Wrapper requirements (save paths, citation format, terminal sections) are a separate contract.
### Subagent roster
-| Agent | Model | Role |
+Models are profile config, not hardcode. The table shows the shipped defaults, and you can override any of them in `.hyperresearch/config.toml`: `[profile.full]` with `models = { fetcher = "haiku" }` swaps every fetcher to Haiku on the next install or `profile use`.
+
+| Agent | Default model | Role |
|---|---|---|
| `hyperresearch-fetcher` | Sonnet | URL fetching via crawl4ai; runs 8–12 in parallel per wave |
-| `hyperresearch-source-analyst` | Sonnet (1M ctx) | End-to-end digest of any single long source >5000 words |
+| `hyperresearch-source-analyst` | Sonnet | End-to-end digest of any single long source >5000 words |
| `hyperresearch-loci-analyst` | Sonnet | Reads the width corpus, returns 1–8 depth loci with rationale |
| `hyperresearch-depth-investigator` | Sonnet | Investigates one locus, writes one interim note with a committed position |
| `hyperresearch-corpus-critic` | Sonnet | "What source would overturn the current direction?" pre-draft gap analysis |
@@ -90,14 +115,16 @@ In your prompt, you can request one of two tiers and the rest of the pipeline sc
| `hyperresearch-width-critic` | Opus | Topical corners the corpus supports but the draft ignores |
| `hyperresearch-instruction-critic` | Opus | Structural mismatches against the prompt's atomic items |
| `hyperresearch-patcher` | Opus | Tool-locked `[Read, Edit]`. Applies critic findings as surgical Edit hunks |
+| `hyperresearch-cite-checker` | Sonnet | Skeptically verifies sampled citation-sentence bindings before ship |
| `hyperresearch-polish-auditor` | Opus | Tool-locked `[Read, Edit]`. Cuts filler, strips hygiene leaks |
| `hyperresearch-readability-recommender` | Opus | Writes JSON suggestions for paragraph rhythm and list/table conversion |
+| `hyperresearch-browser-fetcher` | Sonnet | Drains the escalation queue by driving your real Chrome (Claude-in-Chrome) |
---
## The vault: persistent, searchable, compounding
-Hyperresearch is not a one-shot report generator like most other Deep research harnesses. Every fetched source lands in a SQLite-indexed vault that every future research session can reuse.
+Most deep research harnesses are one-shot: report out, everything else discarded. Hyperresearch keeps what it reads. Every fetched source lands in a SQLite-indexed vault that future sessions search before they fetch.
```bash
hyperresearch search "ion-trap gate fidelity" -j # Full-text search
@@ -108,26 +135,61 @@ hyperresearch graph backlinks <id> -j # Reverse links
hyperresearch lint -j # Health check (broken links, missing tags)
```
-**Markdown is truth, SQLite is cache.** Notes live as plain markdown with YAML frontmatter in `research/notes/`. The SQLite index is fully rebuildable. Delete it and `hyperresearch sync` reconstructs it from the markdown. The vault is inspectable in any editor, version-controllable in git, and readable without the tool installed.
+**Markdown is truth, SQLite is cache.** Notes live as plain markdown with YAML frontmatter in `research/notes/`. The SQLite index is fully rebuildable: delete it and `hyperresearch sync` reconstructs it from the markdown. Open the vault in any editor, version it in git. You don't need the tool installed to read your own research.
**PDFs fetch directly.** `hyperresearch fetch` auto-detects PDF URLs (arXiv, NBER, SSRN, direct `.pdf` links) and extracts full text via pymupdf. Raw PDFs land in `research/raw/<note-id>.pdf` and the note's `raw_file:` frontmatter links back.
**Provenance breadcrumbs.** Every fetched source carries a `--suggested-by` link back to whatever surfaced it. The chain forms a rooted tree from seed fetches; the `provenance` lint rule catches disconnected components.
+**Semantic search, if you want it.** `hyperresearch embed sync` populates embeddings (provider-pluggable: `voyage`, `openai`, or the default `none`, which needs zero API keys) and `search --semantic` blends vector similarity with full-text ranking.
+
+---
+
+## Source ranking: quality is persistent, not vibes
+
+Every source accumulates a composite `quality_score` built from source-type tier, fetch-time utility, citation authority (from OpenAlex / Semantic Scholar, including **retraction flags**), and vault PageRank centrality:
+
+```bash
+hyperresearch sources score -j # Enrich DOI-bearing notes: citations, venue, retractions
+hyperresearch graph rank -j # PageRank over the link + provenance graph
+hyperresearch search "q" --ranked -j # Quality-weighted full-text search
+hyperresearch sources independence -j # Cluster syndicated/derivative copies: 5 copies of one press release = 1 vote
+hyperresearch claims search "q" -j # Query extracted claims across all sources
+```
+
+Retracted sources are floored to near-zero quality, and a ship-time retraction sweep re-checks every cited DOI fresh, so a retraction published yesterday is caught today. Even on vault sources reused from old runs.
+
+---
+
+## Runs: resumable, budgeted, verified
+
+Every run owns an isolated workspace (`research/runs/<vault_tag>/`) and a manifest. Concurrent runs never collide, and a crashed run resumes exactly where it stopped:
+
+```bash
+hyperresearch run status -j # Step-by-step status, spend, escalation queue depth
+hyperresearch run resume -j # Exact next step + Skill invocation to continue
+hyperresearch run report -j # Per-step wall-time / spend / source-yield telemetry
+hyperresearch run verify <tag> -j # Ship gate: headings, length, citation density, cite-check resolution
+```
+
+`run init --budget 50` caps estimated API-equivalent spend; crossing the cap blocks the run rather than letting it quietly balloon. And before any report ships, the verification battery runs: **quote-integrity** (every quoted span must exist verbatim in a vault note), **retracted-citations** (citing a retracted source unacknowledged blocks the ship), **numeric-consistency** (numbers untraceable to evidence get flagged), plus the cite-check step's per-citation binding audit.
+
---
## What's structurally enforced
-- **Verbatim prompt as gospel** — `scaffold-prompt` lint blocks if the scaffold doesn't open with the user's exact prompt
-- **Locus coverage** — every step 4 locus must have a step 5 interim note; missing interims flag as errors
-- **Patch-only modification** — steps 14, 15, 16 are tool-locked to `[Read, Edit]`. They cannot regenerate the draft
-- **Critical findings never silently skip** — `patch-surgery` lint surfaces any critical finding the patcher couldn't apply
-- **Schema integrity** — `tier`, `content_type`, and `type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index
-- **Hygiene leaks caught on the way out** — scaffold sections, YAML frontmatter, and prompt echoes are stripped by step 15 before ship
+- **Verbatim prompt as gospel.** `scaffold-prompt` lint blocks if the scaffold doesn't open with the user's exact prompt
+- **Locus coverage.** Every step 4 locus must have a step 5 interim note; missing interims flag as errors
+- **Patch-only modification.** Steps 14, 15, 16 are tool-locked to `[Read, Edit]`. They cannot regenerate the draft
+- **Critical findings never silently skip.** `patch-surgery` lint surfaces any critical finding the patcher couldn't apply
+- **Quoted text must exist.** `quote-integrity` lint blocks any quoted span that doesn't appear verbatim in a vault note; hallucinated quotes cannot ship
+- **Retractions block the ship.** Citing a retracted source without acknowledging the retraction is a hard error at the final gate
+- **Schema integrity.** `tier`, `content_type`, and `type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index
+- **Hygiene leaks caught on the way out.** Scaffold sections, YAML frontmatter, and prompt echoes are stripped by step 15 before ship
---
-## Authenticated crawling
+## Authenticated crawling + the browser lane
Fetch from LinkedIn, Twitter, paywalled sites or anything you can log into:
@@ -137,16 +199,18 @@ hyperresearch setup # Browser opens. Log into your sites. Done.
LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills.
+**Blocked fetches escalate instead of dying.** When headless crawling hits a login wall or bot wall mid-run, the URL queues as an escalation (`hyperresearch escalation list -j`). If you have the [Claude-in-Chrome](https://claude.com/chrome) extension, the browser-fetcher agent drains the queue by driving your real, logged-in Chrome. Hard boundary: **CAPTCHAs, 2FA, and logins are never solved automatically.** They're consolidated into one message and handed to you.
+
---
## Academic APIs before web search
For any topic with a research literature, hit academic APIs BEFORE web search. They return citation-ranked canonical papers; web search returns derivative commentary.
-- **Semantic Scholar** — `https://api.semanticscholar.org/graph/v1/paper/search`
-- **arXiv** — `https://export.arxiv.org/api/query`
-- **OpenAlex** — `https://api.openalex.org/works`
-- **PubMed** — `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi`
+- **Semantic Scholar:** `https://api.semanticscholar.org/graph/v1/paper/search`
+- **arXiv:** `https://export.arxiv.org/api/query`
+- **OpenAlex:** `https://api.openalex.org/works`
+- **PubMed:** `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi`
After the academic sweep, run web searches for context, news, non-academic angles, and at least one adversarial search ("criticism of X", "limitations of X").
@@ -156,7 +220,7 @@ After the academic sweep, run web searches for context, news, non-academic angle
- It doesn't replace your judgment on which sources matter. The agent picks, you steer.
- It can't fetch what's behind a paywall you haven't logged into.
-- It runs on Anthropic models Opus + Sonnet + Haiku via the subagent roster. Costs scale with tier and corpus size. If anyone wants to port this to Codex, put up a PR!
+- It runs on Anthropic models via the subagent roster (per-agent assignments come from the profile's model map). Usage scales with tier, gear, and corpus size. If anyone wants to port this to Codex, put up a PR!
- The lint gate catches **structural** failures (missing scaffold, broken provenance, unresolved CRITICALs). It cannot guarantee factual accuracy, that's still your call.
---
diff --git a/docs/roadmap-2.0/README.md b/docs/roadmap-2.0/README.md
new file mode 100644
index 0000000..3528497
--- /dev/null
+++ b/docs/roadmap-2.0/README.md
@@ -0,0 +1,68 @@
+# Hyperresearch 2.0 Roadmap
+
+This directory holds the engineering specs for the 2.0 program: six phases that take hyperresearch from a ~80-source, ~10K-word single-report harness to a dissertation-scale research system with persistent source ranking, mechanical verification, and a browser lane for hard-to-reach sources.
+
+## The 2.0 thesis
+
+1.x won on **process discipline**: the 16-step V8 chain, adversarial critics, tool-locked patching, and lint-enforced contracts. Two audits (2026-07-19) found that the remaining gap between 1.x and "better than a human researcher" is not orchestration — it's that:
+
+1. **Quality judgment is ephemeral.** The 6-dimension utility score is used once to pick fetch targets, then discarded. Citations are numbered by first appearance. Nothing in the Python layer can answer "which sources are most load-bearing?" — the `sources` table is a dedup ledger with no score, tier, or rank column.
+2. **Scale is prose-hardcoded.** Source gates (min 45 / target 55–80), loci cap (6), depth budget (40), draft count (3), must-read bounds (20–50), word ceiling (10K) all live as literals inside skill and agent prompt text. Scaling up currently means rewriting 17 skills and 14 agent prompts.
+
+2.0 fixes both: quality judgment becomes **persistent and programmatic** (ranking, claims, verification), and scale becomes a **profile** instead of a rewrite.
+
+## Phases
+
+| Phase | Doc | One-liner |
+|---|---|---|
+| 0 | [phase-0-cleanup.md](phase-0-cleanup.md) | Clear the debt: prompts out of `hooks.py`, V7→V8 vocabulary, one fetch engine, dead code out |
+| 1 | [phase-1-config-profiles.md](phase-1-config-profiles.md) | Every magic number becomes config; pipeline profiles + templated prompts |
+| 2 | [phase-2-source-ranking.md](phase-2-source-ranking.md) | Persistent source scores, citation-graph metadata, tier-weighted retrieval, claims table |
+| 3 | [phase-3-dissertation-scale.md](phase-3-dissertation-scale.md) | Per-run workspaces, run manifest/resume, chaptered execution, hundreds of sources |
+| 4 | [phase-4-chrome-lane.md](phase-4-chrome-lane.md) | Claude-in-Chrome fetch lane: escalation queue, human-in-the-loop checkpoint, session handoff |
+| 5 | [phase-5-verification.md](phase-5-verification.md) | Cite-check, quote-integrity, retraction sweep, independence audit, telemetry, bench CI |
+
+## Dependency graph
+
+```
+Phase 0 (cleanup)
+ │
+Phase 1 (config + profiles)
+ │
+ ├──────────────┬──────────────┐
+Phase 2 Phase 4 Phase 3*
+(ranking) (chrome lane) (scale)
+ │ │
+ └──────────────┬──────────────┘
+ Phase 5
+ (verification)
+```
+
+\* Phase 3 depends on Phase 1 (profiles) and benefits strongly from Phase 2 (programmatic curation at scale); it can start after Phase 1 with Phase-2 integration deferred. Phase 4 is independent after Phase 1 and can run in parallel with 2/3. Phase 5's cite-check and retraction workstreams require Phase 2's claims table and DOI metadata; its mechanical lints (quote-integrity, numeric-consistency) only require Phase 0.
+
+## Status
+
+- [ ] Phase 0 — cleanup (partially pulled forward: width-sweep consistency fix landed with Phase 1; prompt extraction from hooks.py, V7 vocabulary unification, fetch consolidation, and dead-code removal remain open)
+- [x] Phase 1 — config + profiles (2026-07-19)
+- [x] Phase 2 — source ranking (2026-07-19)
+- [x] Phase 3 — dissertation scale (2026-07-19)
+- [x] Phase 4 — chrome lane (2026-07-19; session handoff deliberately dropped — see phase doc/CHANGELOG)
+- [x] Phase 5 — verification (2026-07-19; bench CI workflow replaced by shipped `hpr run verify` — bench/ is gitignored)
+
+## What 1.x already has (do not rebuild)
+
+Contributors should not re-implement any of the following — they exist and work:
+
+- **Multi-lens sourcing**: 4 search lenses (breadth / academic citation-chain / adversarial ≥5 searches / period-pinned primary filings), academic-APIs-before-web, Wikipedia-as-source-hub-never-cited, mandatory fetcher citation-chasing (3–8 primaries per batch) with `--suggested-by` provenance chains.
+- **Fetch-time quality gates**: pre-fetch 6-dim utility scoring, login-wall/junk/binary detection (`web/base.py`), redundancy audit tagging `derivative-of` sources.
+- **Adversarial architecture**: pre-draft corpus critic, 4 parallel post-draft critics, tool-locked patcher/polish-auditor (`[Read, Edit]`), patch-never-regenerate invariant.
+- **Process lint suite**: `scaffold-prompt`, `locus-coverage`, `patch-surgery`, `wrapper-report`, `audit-gate`, `instruction-coverage`, `citation-style-preservation`, `provenance` (`cli/lint.py`).
+- **Provenance & archival**: rooted suggestion tree, `raw_file` PDF archival via pymupdf, `sources` URL ledger.
+- **Compounding vault**: markdown-is-truth / SQLite-is-cache, FTS5 search with status-aware ranking, run-to-run safety (`archive-run`, collision-safe `vault-tag`).
+- **Authenticated crawling v1**: crawl4ai browser profiles, patchright stealth, visible-browser fallback for session-killing domains.
+
+## Conventions for these docs
+
+Every phase doc follows the same structure: Goal & non-goals → Current state (with `file:line` audit anchors, verified 2026-07-19 against v0.8.7 / commit `1dcf415`) → Workstreams (design, file-level changes, schemas, migration notes) → Dependencies → Acceptance criteria → Risks & mitigations → Effort estimates (S = hours, M = a day or two, L = several days).
+
+Line numbers drift; when executing a phase, re-verify anchors with grep before editing. The anchor's job is to make the target findable, not to be eternally exact.
diff --git a/docs/roadmap-2.0/phase-0-cleanup.md b/docs/roadmap-2.0/phase-0-cleanup.md
new file mode 100644
index 0000000..1dfcc19
--- /dev/null
+++ b/docs/roadmap-2.0/phase-0-cleanup.md
@@ -0,0 +1,113 @@
+# Phase 0 — Cleanup: clear the debt before building on it
+
+## Goal
+
+Remove the structural debt that makes every later phase harder: prompt text trapped in a 3,495-line Python module, stale V7 vocabulary in 11 of 14 agent prompts, three parallel fetch implementations with a core→cli layering inversion, and confirmed dead code. After Phase 0, prompts are diffable markdown, there is exactly one fetch engine, and the tree contains nothing that lies about the current architecture.
+
+## Non-goals
+
+No behavior changes to the pipeline, no new features, no config work (that's Phase 1). A run before and after Phase 0 should produce equivalent output. The human-facing PKM surface (`serve/`, `indexgen/`, `topic`, `template`, `git`, `watch`, `link`, `dedup`) is *not* deleted — see WS6 for the packaging decision.
+
+## Current state (audit anchors, v0.8.7)
+
+- `core/hooks.py` is 3,495 lines; ~83% (lines ~53–2953) is 15 triple-quoted agent-prompt string constants plus `HOOK_SCRIPT_TEMPLATE` (`hooks.py:2911`); only ~540 lines (2954–3495) are install logic.
+- 11 of 14 agent prompts describe the retired V7 "7-phase / Layer N" pipeline: loci-analyst (`hooks.py:76`), depth-investigator (`:283`), dialectic-critic (`:549`), depth-critic (`:672`), width-critic (`:775`), instruction-critic (`:936`), patcher (`:1219`), polish-auditor (`:1390`), source-analyst (`:2408`), fetcher (`:2561`), corpus-critic (`:2819`). Only draft-orchestrator (`:1705`), synthesizer (`:1889`), and readability-reformatter (`:2196`) speak V8 "step N". The roster comment (`:2957-2962`) and lint rule descriptions (`cli/lint.py:26-27,750,753`) also carry Layer language.
+- Three fetch implementations: `cli/fetch.py:205-611` (the feature-complete one: tier/content-type detection, `--suggested-by` breadcrumbs, idempotent dup handling), `core/fetcher.py` (`fetch_and_save`, used only by `mcp/server.py:291-298`), `cli/research.py:_save_result`. Layering inversion at `core/fetcher.py:148` (`from hyperresearch.cli.fetch import _save_assets`).
+- Triplicated visible-browser domain list: `core/fetcher.py:42-45`, `cli/fetch.py:293-296`, `cli/fetch_batch.py:58-61`. Duplicated smart-wait JS: `web/crawl4ai_provider.py:220-235` vs `:308-320`. Duplicated garbage-ratio threshold: `web/base.py:131` vs `web/crawl4ai_provider.py:65` (the comment at `:62-64` warns about exactly this drift).
+- Confirmed dead: `embeddings` table (`core/db.py:88-94`, zero readers/writers), `models/graph.py` (21 lines) and `models/search.py` (24 lines) with zero import sites, empty stub packages `graph/` and `export/`.
+- `cli/research.py` (313 lines) is superseded by the skill pipeline and only functions with exa/tavily (crawl4ai raises `NotImplementedError` for `search()`, `crawl4ai_provider.py:469`).
+- `core/enrich.py` heuristics compete with agent curation; `core/fetcher.py:115` itself comments that curation "is the agent's job." Callers: `cli/fetch_batch.py:168`, `cli/research.py:271`, `mcp/server.py:343`, `cli/repair.py:91`.
+- Width-sweep skill states the full-tier source target three inconsistent ways: "40–100" (frontmatter description), "40–80" (goal, ~line 17), "55–80" (table, ~line 222).
+- Legacy `layercake-*` migration cruft: `hooks.py:3319` (`research-layercake` in retired dirs), `:3333-3480` (`_prune_retired_agents` + legacy dir cleanup).
+
+## Workstreams
+
+### WS1 — Extract prompts from `hooks.py` to package data (L)
+
+**Design.** Agent prompts become `.md` files with YAML frontmatter (they are already written in that shape inside the strings), shipped as package data next to the existing 17 skills. New layout:
+
+```
+src/hyperresearch/skills/ # existing: 17 skill .md files + .hyperresearch/hook.js
+src/hyperresearch/agents/ # new: 15 agent .md files (loci-analyst.md, fetcher.md, ...)
+```
+
+**Changes.**
+- Move each `*_AGENT` constant's content verbatim (minus V7 fixes from WS2) into `src/hyperresearch/agents/hyperresearch-<name>.md`.
+- Add a loader mirroring `_read_skill_source` (`hooks.py:~3437`): `importlib.resources` first, source-tree fallback. Reuse, don't fork — factor a shared `_read_package_md(package, name)`.
+- `install_hooks` / `install_global_hooks` / `_install_*_agent` helpers read files instead of constants. `hooks.py` shrinks to install logic only (~500 lines).
+- `hook.js` already ships as package data (`skills/.hyperresearch/hook.js`) — keep; delete the redundant `HOOK_SCRIPT_TEMPLATE` constant if it duplicates the file, or make the file the single source.
+- Update `pyproject.toml` hatch wheel config if needed so `agents/` package data ships (packages are included via `packages = ["src/hyperresearch"]`; verify with a wheel-content test — there is precedent in the 0.8.7 packaging regression tests).
+
+**Migration.** None for users — `hyperresearch install` output is byte-identical (modulo WS2 text fixes).
+
+### WS2 — V7 → V8 vocabulary unification (M)
+
+**Design.** One vocabulary: "step N of the 16-step V8 pipeline." The Layer→step mapping for rewriting: Layer 1 → step 2 (width sweep), Layer 2 → step 4 (loci), Layer 3 → step 5 (depth), Layer 3.5 → step 6 (reconcile), Layer 3.7 → step 8 (corpus critic), Layer 4 → steps 10–11 (draft/synthesize), Layer 5 → step 12 (critics), Layer 6 → step 14 (patcher), Layer 7 → step 15 (polish).
+
+**Changes.**
+- Rewrite the pipeline-position paragraphs in the 11 stale agent files (post-WS1 they are markdown — do WS1 first, then WS2 edits the .md files).
+- Keep each agent's *description* frontmatter in sync — the `.claude/agents/` descriptions surface in Claude Code's agent list and several still say "Layer N" (e.g. corpus-critic says "Layer 3.7").
+- Fix lint rule descriptions: `cli/lint.py:26-27` ("Layer 2 … Layer 6"), `:750,753` ("Layer 1 width sweep", "Layer 3 depth investigators").
+- Fix roster comment `hooks.py:2957-2962` ("as of v7") and install labels (`:3291,:3302`).
+- Add a regression test: grep the shipped skills + agents package data for `Layer \d|7-phase|layercake` → must be zero matches (allowlist the migration-pruning code itself).
+
+### WS3 — One fetch engine (L)
+
+**Design.** `core/fetcher.py` becomes the single engine; `cli/fetch.py` and `mcp/server.py` become thin callers. The cli implementation is the feature superset, so consolidation direction is: port cli features *into* core, then delete the cli duplicate.
+
+**Changes.**
+- Move into `core/fetcher.py`: tier/content-type auto-detection (`cli/fetch.py:85-201` `_detect_tier`/`_detect_content_type`), `--suggested-by` breadcrumb prepend + idempotent duplicate handling, the `INSERT OR IGNORE` race guard, and `_save_assets` (currently `cli/fetch.py:508`, imported backwards at `core/fetcher.py:148`). Assets code moves to `core/assets.py` or into `core/fetcher.py`; core must import nothing from `cli`.
+- `fetch_and_save()` signature grows the cli-only options (suggested_by, save_assets, tags, tier/content_type overrides). `cli/fetch.py` keeps CLI parsing + output rendering only.
+- `cli/fetch_batch.py` calls the same engine per URL with one deferred sync at the end (preserve its single-batched-sync behavior).
+- Single `AUTH_AGGRESSIVE_DOMAINS` constant in `core/fetcher.py` (until Phase 1 makes it config); the other two copies import it.
+- Extract the smart-wait JS into one module-level constant in `crawl4ai_provider.py` used by both `_make_run_config` and `_fetch_visible`.
+- Single junk threshold: `crawl4ai_provider.py:65` imports the ratio-check helper from `web/base.py` instead of re-hardcoding `0.05` (the shared `binary_garbage_ratio` function already exists — share the *threshold comparison* too, e.g. `is_binary_garbage(sample)`).
+- Tests: existing `tests/` fetch tests must pass unchanged; add one test asserting `core` has no `cli` imports (walk `core/*.py` ASTs).
+
+### WS4 — Dead code removal (S)
+
+**Changes.**
+- Delete `models/graph.py`, `models/search.py`, and the empty `graph/`, `export/` packages (real functionality lives in `cli/graph.py`, `cli/export.py` — untouched).
+- **Keep** the `embeddings` table (`core/db.py:88-94`) — Phase 2 WS6 revives it. Add a comment: `-- dormant; populated by Phase 2 semantic search (docs/roadmap-2.0/phase-2-source-ranking.md)`.
+- Retire `cli/research.py`: remove the command registration from `cli/__init__.py`, delete the module, note in CHANGELOG ("`hyperresearch research` retired; use `/hyperresearch`"). Its unique helper `_extract_links_from_results` dies with it.
+- Demote `core/enrich.py` to repair-only: drop calls from `cli/fetch_batch.py:168` and `mcp/server.py:343`; keep the `cli/repair.py:91` call (repair is the "fix a neglected vault" path where heuristics beat nothing). Update the MCP `create_note` docstring accordingly.
+
+### WS5 — Consistency fixes (S)
+
+**Changes.**
+- Width-sweep skill (`src/hyperresearch/skills/hyperresearch-2-width-sweep.md` + installed copy): one source-target statement. Pick the table values (min 45 / target 55–80) as canonical; header and goal line reference them. (Phase 1 turns these into template variables; this WS just stops the file disagreeing with itself.)
+- Sunset `layercake-*` pruning (`hooks.py:3319,3333-3480`): v0.8.x installs have had two-plus release cycles to migrate. Keep `_prune_retired_agents` (it serves the current roster) but drop the layercake-specific dir list, or gate it behind a `--prune-legacy` flag. Note in CHANGELOG that pre-0.8.0 installs should run 0.8.7's installer once before upgrading.
+
+### WS6 — PKM surface packaging decision (S, decision + docs only)
+
+**Design.** `serve/` (717 lines), `indexgen/` (256), `topic`, `template`, `git`, `watch`, `link`, `dedup` are never used by the pipeline but are legitimate human-vault features. Decision: keep them in-tree, but document the boundary — add a "pipeline-critical vs. vault-convenience" module map to `CONTRIBUTING.md` so contributors know what the agent path depends on. Revisit extras-group split (`pip install hyperresearch[pkm]`) only if install size becomes a complaint; a code split now would churn imports for little gain.
+
+## Dependencies
+
+None — Phase 0 is the root of the graph. WS2 depends on WS1 (edit prompts as markdown, not as Python strings).
+
+## Acceptance criteria
+
+- [ ] `core/hooks.py` < 600 lines; all agent prompts exist as `.md` package data; `pip install` + `hyperresearch install` produces the same `.claude/` tree as before (diff-tested).
+- [ ] Zero matches for `Layer \d|7-phase|layercake` in shipped skills/agents (regression test in CI).
+- [ ] Exactly one fetch implementation; `core/` imports nothing from `cli/` (AST test); MCP `fetch_url` and CLI `fetch` both route through `core/fetcher.py`; full test suite green.
+- [ ] `models/graph.py`, `models/search.py`, `graph/`, `export/` gone; `cli/research.py` retired; `enrich` called only from `repair`.
+- [ ] Width-sweep skill states the source target once.
+- [ ] CHANGELOG entries for the retirements.
+
+## Risks & mitigations
+
+- **Prompt-extraction transcription errors** — a dropped line in a 200-line prompt silently degrades an agent. Mitigation: extraction is mechanical (script it: write each constant to file, then assert file content == constant before deleting the constant).
+- **Fetch consolidation regressions** — cli fetch has subtle behaviors (idempotent re-fetch appends breadcrumbs, orphan cleanup). Mitigation: write characterization tests against current `cli/fetch.py` behavior *before* moving code.
+- **`hyperresearch install` diff churn** for existing users — WS2 rewrites agent text, so installed agents change on upgrade. That's intended; call it out in CHANGELOG.
+
+## Effort
+
+| WS | Size |
+|---|---|
+| WS1 prompt extraction | L |
+| WS2 vocabulary | M |
+| WS3 fetch engine | L |
+| WS4 dead code | S |
+| WS5 consistency | S |
+| WS6 packaging decision | S |
diff --git a/docs/roadmap-2.0/phase-1-config-profiles.md b/docs/roadmap-2.0/phase-1-config-profiles.md
new file mode 100644
index 0000000..b18146a
--- /dev/null
+++ b/docs/roadmap-2.0/phase-1-config-profiles.md
@@ -0,0 +1,206 @@
+# Phase 1 — Config extraction + pipeline profiles
+
+## Goal
+
+Every number that governs behavior lives in config, and every number that governs *research scale* lives in a named **pipeline profile**. The 17 skills and 15 agent prompts become Jinja templates rendered from a profile at install time, so "scale up" is a profile edit, not a 30-file prose rewrite. Ship `light` and `full` profiles that reproduce today's exact behavior; later phases add `dissertation`.
+
+## Non-goals
+
+No new pipeline capabilities, no schema changes (Phase 2), no per-run workspaces (Phase 3). Rendered output under the default `full` profile must be semantically identical to today's skills/agents.
+
+## Current state (audit anchors, v0.8.7)
+
+**Python-layer hardcoded values (representative — full inventory below):**
+
+| Anchor | Value | Meaning |
+|---|---|---|
+| `web/crawl4ai_provider.py:119` | `timeout=30, verify=False` | PDF download timeout + **silent TLS-verification disable** |
+| `web/crawl4ai_provider.py:143` | `< 100` bytes | min valid PDF size |
+| `web/crawl4ai_provider.py:244` | `page_timeout=30000` | crawl page timeout |
+| `web/crawl4ai_provider.py:220-235` | 2000ms initial / 500ms poll / 2 stable / 16 checks | smart-wait DOM-stability loop |
+| `web/base.py:95` | `< 300` chars | empty-content junk gate |
+| `web/base.py:131` + `crawl4ai_provider.py:65` | ratio `> 0.05` | binary-garbage threshold |
+| `web/base.py:135` | `< 1500` chars | cookie-wall gate |
+| `web/base.py:71-73` | `< 1000` chars + keyword | login-wall gate |
+| `web/base.py:60-63,99-141` | inline keyword tuples | login/cloudflare/error/cookie signal lists |
+| `cli/fetch.py:19,21` | `MAX_IMAGES=5`, `MIN_IMAGE_BYTES=50_000` | asset caps |
+| `core/fetcher.py:42-45` | domain list | visible-browser (auth-aggressive) domains |
+| `core/similarity.py:12,29,51` | shingle n=3, num_perm=128, bands=16 | MinHash/LSH params |
+| `cli/dedup.py:17` | `LSH_THRESHOLD=200` | brute↔LSH switchover |
+| `cli/lint.py:917,975,1556` | 150 words, //3 ratio, 90 days | extract-coverage + stale-review thresholds |
+| `cli/search.py:30,32` | limit 20, 4 chars/token | search defaults |
+| `core/enrich.py:41,69` | 5 tags, 120-char summary | enrich caps |
+
+**Skill/agent-prose hardcoded parameters (the scale knobs — full inventory from the 2026-07-19 skills audit):**
+
+- Tier routing: light = steps 1→2→10→15→16; full = all 16 (router skill).
+- Source gates: light min 10 / target 15–25; full min 45 / target 55–80; "beyond ~80 diminishing returns" ceiling rationale (width-sweep ~line 227).
+- Fan-out: 40–100 planned searches; 80–120 candidate URLs; 60–100 deduped; 10–12 batches × 8–12 URLs; Wave 1 = 10–12 fetchers; Wave 2 = 3–5; Wave 3 = 2–3; ≥5 adversarial searches; 80%-of-queue wave-done threshold; ≤1 vault check/min.
+- Utility scoring: 6 dims × 0–3, max 18; ≥3 candidate URLs per atomic item floor.
+- Source-analyst: >5000-word trigger, cap 6 per query.
+- Loci: 2 analysts; clamp 6; 4 scoring dims × 0–10; total depth budget 40; brackets 30–40→≤15, 20–29→≤10, 10–19→≤5, <10→0–3.
+- Depth: K ≤ 6 investigators; default budget 10; >50% failure abort.
+- Tensions: comparisons 3–5; source-tensions 3–7 from top 8–12 full-body reads (15–20 surveyed).
+- Corpus critic: 3–8 gaps; 2–4 fill fetchers. Evidence digest: 80–120 claims cap, 30 min. Gap-fetch: cap 5.
+- Drafting: exactly 3 drafts; must-read 20–50 (argumentative 35–50 / structured 25–40 / short 20–30); word targets 500–2000 / 2000–5000 / 5000–10000; citation density ≥2 per 1000 chars; synthesizer totals 80–150 / 40–80 / 15–30 citations.
+- Critics: finding caps 12 / 12 / 10 / 15; readability recommender cap 50; polish thresholds (50-word sentences, 200-word paragraphs).
+- Model map (frozen per agent): sonnet = fetcher, source-analyst, loci-analyst, depth-investigator, corpus-critic; opus = everything else.
+
+## Workstreams
+
+### WS1 — Python settings extraction (M)
+
+**Design.** Extend `VaultConfig` (`core/config.py`) with new dataclass sections; all defaults equal to today's literals so absent config keys change nothing.
+
+```toml
+[fetch]
+page_timeout_ms = 30000
+pdf_timeout_s = 30
+pdf_verify_tls = true # NEW default: flips the current silent verify=False to secure-by-default; document the change
+min_pdf_bytes = 100
+wait_initial_ms = 2000
+poll_interval_ms = 500
+stable_checks = 2
+max_checks = 16
+visible_browser_domains = ["linkedin.com", "twitter.com", "x.com", "facebook.com", "instagram.com", "tiktok.com"]
+image_timeout_s = 15
+
+[junk]
+min_content_chars = 300
+login_wall_max_chars = 1000
+cookie_wall_max_chars = 1500
+binary_garbage_ratio = 0.05
+sample_window = 2000
+extra_login_signals = [] # appended to built-in lists, not replacing them
+extra_junk_domains = []
+
+[assets]
+max_images = 5
+min_image_bytes = 50000
+
+[dedup]
+shingle_size = 3
+minhash_perm = 128
+lsh_bands = 16
+lsh_switchover = 200
+default_threshold = 0.6
+
+[lint]
+extract_min_words = 150
+extract_coverage_divisor = 3
+stale_review_days = 90
+
+[search] # existing section gains:
+default_limit = 20
+chars_per_token = 4
+snippet_len = 200
+```
+
+**Changes.** Thread config through call sites (fetch providers get a config object at `get_provider()` time; lint rules read `vault.config`). Signal keyword lists stay as code constants but gain `extra_*` config extension points — replacing them wholesale is a footgun. **Note the one intentional behavior change:** `pdf_verify_tls` defaults to `true`, fixing the silent `verify=False` at `crawl4ai_provider.py:119`; users hitting cert-broken mirrors set it to `false` explicitly. CHANGELOG this.
+
+### WS2 — Pipeline profile schema (M)
+
+**Design.** Profiles live in `.hyperresearch/config.toml` under `[profile.<name>]`, with built-in `light` and `full` defined in code (overridable). A profile is a flat parameter set + a model map:
+
+```toml
+[profile.full] # shipped defaults == today's prose values
+steps = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
+source_min = 45
+source_target_low = 55
+source_target_high = 80
+planned_searches = [40, 100]
+candidate_urls = [80, 120]
+batch_size = [8, 12]
+wave1_fetchers = [10, 12]
+wave2_fetchers = [3, 5]
+adversarial_searches_min = 5
+utility_scoring = true
+source_analyst_cap = 6
+source_analyst_word_trigger = 5000
+loci_analysts = 2
+loci_max = 6
+depth_budget_total = 40
+depth_budget_brackets = [[30, 15], [20, 10], [10, 5], [0, 3]]
+investigator_max = 6
+comparisons_tensions = [3, 5]
+source_tensions = [3, 7]
+tension_full_reads = [8, 12]
+corpus_critic_gaps = [3, 8]
+claims_cap = [80, 120]
+claims_min = 30
+draft_count = 3
+must_read = { argumentative = [35, 50], structured = [25, 40], short = [20, 30] }
+word_targets = { short = [500, 2000], structured = [2000, 5000], argumentative = [5000, 10000] }
+citation_density_min = 2.0 # per 1000 chars
+critic_finding_caps = { dialectic = 12, depth = 12, width = 10, instruction = 15 }
+gap_fetch_cap = 5
+readability_rec_cap = 50
+vault_check_interval_s = 60
+wave_done_ratio = 0.8
+
+[profile.full.models]
+fetcher = "sonnet"
+source_analyst = "sonnet"
+loci_analyst = "sonnet"
+depth_investigator = "sonnet"
+corpus_critic = "sonnet"
+draft_orchestrator = "opus"
+synthesizer = "opus"
+critics = "opus"
+patcher = "opus"
+polish_auditor = "opus"
+readability_recommender = "opus"
+```
+
+`[profile.light]` mirrors today's light tier (steps `[1,2,10,15,16]`, source_min 10, target 15–25, wave1_fetchers [3,5], utility_scoring false, etc.). A `hpr profile show <name> -j` command prints the resolved profile for agents to read.
+
+**Loader.** `core/profiles.py`: built-in dicts + TOML overlay + validation (pydantic model; ranges as 2-lists; unknown keys warn). `hpr profile list/show/validate`.
+
+### WS3 — Prompt templating + render pipeline (L)
+
+**Design.** Skills and agent `.md` files (post-Phase-0 they're package data) become Jinja2 templates (`jinja2` is already a dependency). Literals become variables: `"Target {{ p.source_target_low }}–{{ p.source_target_high }} curated sources"`, `"spawn {{ p.wave1_fetchers[0] }}–{{ p.wave1_fetchers[1] }} fetcher subagents in ONE message"`. Model frontmatter in agent files renders from the profile's model map.
+
+**Render points.**
+- `hyperresearch install [--profile NAME]` renders templates → `.claude/skills/` + `.claude/agents/` (default profile: `full` — matching today's install output).
+- Rendered files carry a header comment: `<!-- rendered from profile "full" (hyperresearch 2.0.0) — edit the profile, not this file -->`.
+- Step 1's tier classifier maps tier → profile name; the router reads the resolved profile via `hpr profile show` rather than a prose table. Tier switching at run time = both profiles' skills are pre-rendered; the router table itself becomes generated.
+
+**Drift prevention.**
+- Test: render `full`, diff against a checked-in golden copy of the 1.x prose values → any numeric drift is a deliberate golden update.
+- Test: grep templates for bare load-bearing numbers that should be variables (maintain an allowlist for genuinely fixed numbers like "step 14").
+- Keep templates readable: only *scale* parameters templated; procedural prose stays literal.
+
+### WS4 — Tier/profile plumbing (S)
+
+- Step-1 decompose skill writes `pipeline_profile` (not just `pipeline_tier`) into `research/prompt-decomposition.json`; downstream steps read parameters from `hpr profile show <name> -j` when they need a number at run time (e.g., gap caps), rather than trusting possibly-stale rendered prose.
+- Router's tier table and time/cost estimates render from profiles.
+- `wrapper_contract.json` gains an optional `profile` override so harnesses (bench) can pin one.
+
+## Dependencies
+
+Phase 0 (WS1/WS2 there must land first — templates are edited as package-data markdown).
+
+## Acceptance criteria
+
+- [ ] All Python-layer values from the WS1 table read from config; defaults reproduce current behavior; `pdf_verify_tls` change CHANGELOG'd.
+- [ ] `hpr profile list/show/validate` work; built-in `light`/`full` resolve; user overlay merges.
+- [ ] `hyperresearch install` renders templates; rendered `full` output is semantically identical to 1.x skills/agents (golden-diff test).
+- [ ] No load-bearing magic numbers remain in templates (grep test with allowlist).
+- [ ] A custom profile with `source_min = 200` renders skills that say 200 — end-to-end template test.
+- [ ] Full test suite green.
+
+## Risks & mitigations
+
+- **Template rot** — prose edited without noticing it's a template. Mitigation: render header comment + golden tests + CONTRIBUTING note.
+- **Prompt quality regression from mechanical substitution** — a range variable can read awkwardly mid-sentence. Mitigation: hand-review every substitution site once; the golden diff makes review tractable.
+- **Config sprawl** — 40+ new keys. Mitigation: everything defaults; `config show` groups by section; docs list only the 6–8 keys users realistically touch.
+- **Run-time vs render-time skew** — an installed render from profile A while decomposition says profile B. Mitigation: router asserts rendered-profile header matches `pipeline_profile`, re-runs `install --profile` if not (install is idempotent and fast).
+
+## Effort
+
+| WS | Size |
+|---|---|
+| WS1 settings extraction | M |
+| WS2 profile schema | M |
+| WS3 templating | L |
+| WS4 plumbing | S |
diff --git a/docs/roadmap-2.0/phase-2-source-ranking.md b/docs/roadmap-2.0/phase-2-source-ranking.md
new file mode 100644
index 0000000..88f7753
--- /dev/null
+++ b/docs/roadmap-2.0/phase-2-source-ranking.md
@@ -0,0 +1,145 @@
+# Phase 2 — The source-ranking engine
+
+## Goal
+
+Make source quality a **persistent, queryable property** instead of ephemeral prompt prose. After Phase 2, every source note carries a composite quality score built from fetch-time utility, citation-graph authority, vault centrality, and independence; retrieval is tier- and score-weighted; extracted claims live in the database keyed to their sources; and semantic search works over the vault. This is the prerequisite for both dissertation-scale curation (Phase 3) and mechanical verification (Phase 5).
+
+## Non-goals
+
+No changes to how many sources are fetched (Phase 3), no citation-*checking* (Phase 5). The LLM still makes final curation judgments — Phase 2 gives it a ranked shortlist instead of a flat pile.
+
+## Current state (audit anchors, v0.8.7)
+
+- **No numeric source ranking exists in the Python layer.** The `sources` table (`core/db.py:101-110`) has columns `url, note_id, domain, fetched_at, provider, content_hash, status` — a dedup ledger only.
+- The step-2 utility score (6 dims × 0–3) is written to `research/temp/scored-urls.md`, used once for fetch selection, then **discarded** — it never travels with the note.
+- `tier` / `content_type` exist on notes (`models/note.py:30-39`, CHECK-constrained in `db.py:27-30`) but are set by agent frontmatter whim and **never enter search scoring**: `search/fts.py:143-144` selects them and ignores them; ranking is BM25 + status multipliers only (`fts.py:154-164`).
+- The `embeddings` table (`core/db.py:88-94`) is schema-defined and completely dormant — no writer, no reader.
+- Claims are extracted per-fetch into `research/temp/claims-<note-id>.json` (fetcher agent contract) and consumed by steps 3/9 as files; they are never persisted queryably.
+- Citation numbering in reports is order-of-first-appearance (draft-orchestrator/synthesizer prompts) — quality-blind.
+- Provenance graph (links + `--suggested-by` breadcrumbs) exists in the `links` table but nothing computes centrality on it.
+
+## Workstreams
+
+### WS1 — Source-score schema + utility persistence (M)
+
+**Design.** New migration (SCHEMA_VERSION 9). Scores live on **notes** (the citable unit), with fetch metadata staying on `sources`.
+
+```sql
+ALTER TABLE notes ADD COLUMN doi TEXT;
+ALTER TABLE notes ADD COLUMN utility_score REAL; -- step-2 composite (0-18), persisted at fetch
+ALTER TABLE notes ADD COLUMN authority_score REAL; -- citation-graph derived (WS2), normalized 0-1
+ALTER TABLE notes ADD COLUMN centrality_score REAL; -- vault PageRank (WS3), normalized 0-1
+ALTER TABLE notes ADD COLUMN independence REAL; -- 1.0 default; discounted for derivative-of (WS see phase 5)
+ALTER TABLE notes ADD COLUMN citation_count INTEGER; -- external, from OpenAlex/S2
+ALTER TABLE notes ADD COLUMN venue TEXT;
+ALTER TABLE notes ADD COLUMN is_retracted INTEGER DEFAULT 0;
+ALTER TABLE notes ADD COLUMN quality_score REAL; -- computed composite, see WS4
+```
+
+Frontmatter mirrors (`NoteMeta` gains optional `doi`, `utility_score`, etc.) so markdown stays truth and `sync` round-trips them; DB-only derived values (`centrality_score`, `quality_score`) are cache-resident and recomputed, not stored in frontmatter.
+
+**Utility persistence.** Step-2 skill change: when the orchestrator scores URLs, fetcher spawn prompts include the URL's utility score, and the fetcher writes it into the new note's frontmatter (`utility_score:`). `scored-urls.md` remains the working artifact; the score now survives it.
+
+**DOI capture.** Fetcher extracts DOI/arXiv id when present (meta tags, arXiv URL patterns — `crawl4ai_provider.py:44-50` already recognizes arXiv URLs); `hpr sources backfill-doi` regexes existing notes' source URLs/bodies for the 804-note back-catalog.
+
+### WS2 — External citation metadata: `hpr sources score` (M)
+
+**Design.** Batch CLI command that enriches DOI/arXiv-bearing notes from free scholarly APIs (httpx already a dependency; the CLAUDE.md academic-API endpoints are the same services, now called from Python instead of prose):
+
+- **OpenAlex** (`api.openalex.org/works/doi:<doi>`, batch via filter, `mailto` param): `cited_by_count`, venue, retraction flag, referenced_works for chaining.
+- **Semantic Scholar Graph** (`/graph/v1/paper/DOI:<doi>?fields=citationCount,venue,externalIds`): fallback + arXiv coverage.
+- **Crossref** (`api.crossref.org/works/<doi>`): `update-to` relations surface retractions/corrections (Retraction Watch data flows through Crossref since 2023).
+
+**Mechanics.** Rate limits: OpenAlex ~10 rps polite pool, S2 ~1 rps unauthenticated — batch endpoints where available, sequential with backoff otherwise. Cache responses in a new `api_cache` table (url, fetched_at, body) with a 30-day TTL so re-scoring is cheap. `authority_score` = log-scaled `citation_count` normalized within the vault (rank percentile), so a 50-citation niche paper isn't crushed by a 10k-citation classic in another field. Non-academic sources (no DOI) keep `authority_score = NULL` — the composite (WS4) falls back to tier weighting for them.
+
+**Pipeline hook.** Step 2 gains a post-wave sub-step: `hpr sources score --tag <vault_tag>` after each fetch wave (cheap; only new DOI-bearing notes hit APIs). Retraction hits (`is_retracted=1`) are surfaced immediately in the step-2 coverage check so a retracted source never anchors a locus.
+
+### WS3 — Vault centrality: `hpr graph rank` (S)
+
+**Design.** PageRank over the directed graph of `links` rows (wiki-links + suggested-by breadcrumbs, which already land in `links` because breadcrumbs are body wiki-links). Pure-Python power iteration — the vault is thousands of nodes, not millions; no new dependency. Damping 0.85, 50 iterations or 1e-6 convergence. Store to `notes.centrality_score` (normalized 0–1 by max). Expose as `hpr graph rank [-j]` and recompute automatically at the end of `sync` when links changed (cheap) or behind `--no-rank` for huge vaults.
+
+**Interpretation note for the docs/prompts:** centrality in *this* graph means "many independent research chains converged on this source" — a strong load-bearing signal precisely because fetcher chasing is citation-driven.
+
+### WS4 — Composite quality score + tier-weighted retrieval (M)
+
+**Design.** `quality_score` computed at sync/score time:
+
+```
+quality = w_tier * tier_weight[tier] # ground_truth 1.0 … commentary 0.4, unknown 0.6
+ + w_util * (utility_score / 18)
+ + w_auth * (authority_score or tier_fallback)
+ + w_cent * centrality_score
+ - retraction_penalty (hard: is_retracted → quality = 0.05)
+```
+
+Weights in config `[ranking]` (defaults: tier 0.35, utility 0.2, authority 0.25, centrality 0.2). All inputs nullable — missing components renormalize rather than zeroing.
+
+**Retrieval integration.** `search/fts.py`: after BM25 + status multipliers (`fts.py:154-164`), multiply by `(0.5 + quality_score)` when `--ranked` is passed (new flag; default off to preserve existing behavior/tests, flipped on in pipeline prompts). New `hpr search --ranked --tag <vault_tag> --for-item "<atomic item>"` becomes the curation primitive.
+
+**Pipeline integration (the payoff).** Step 10 curation changes from "orchestrator surveys the vault and hand-picks" to: `hpr note list --tag <vault_tag> --ranked -j` → top-K per atomic item by quality × relevance → orchestrator *refines* (angle differentiation, tension proponents) instead of *inventing*. Citation preference in drafts: when multiple sources support a claim, cite the highest-quality one — the ranked list makes this instruction executable rather than aspirational.
+
+### WS5 — Claims table (M)
+
+**Design.** Persist what fetchers already produce. New tables (same migration):
+
+```sql
+CREATE TABLE claims (
+ id INTEGER PRIMARY KEY,
+ note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
+ claim TEXT NOT NULL,
+ quoted_support TEXT,
+ numbers TEXT, -- JSON array
+ confidence TEXT, -- high/medium/low
+ evidence_type TEXT, -- empirical/statistical/anecdotal/opinion
+ stance_target TEXT,
+ stance TEXT,
+ vault_tag TEXT
+);
+CREATE VIRTUAL TABLE claims_fts USING fts5(claim, quoted_support, content='claims', content_rowid='id');
+```
+
+`hpr claims ingest research/temp/claims-*.json` (idempotent by note_id+claim hash) runs at the end of step 2 / step 13; `hpr claims list --note <id>`, `--search "<text>"`, `--tag <vault_tag>` query it. Steps 3 (contradiction graph) and 9 (evidence digest) can consume the DB instead of re-parsing JSON files — optional simplification, keep file compatibility for now.
+
+**Why:** this is the substrate for Phase 5's cite-check (claim ↔ sentence verification) and numeric-consistency lint, and it makes "which source best supports X" a query.
+
+### WS6 — Embeddings revival: semantic search (L)
+
+**Design.** Populate the dormant `embeddings` table (`core/db.py:88-94`). Pluggable provider via config `[embeddings]`: `provider = "none" | "voyage" | "openai" | "local"` (voyage-3-lite-class API models are cheap and good; `local` via sentence-transformers stays an optional extra to avoid a heavy default dependency). Embed title + summary + first ~1500 body chars per note; store float32 BLOBs; brute-force cosine at query time (fine to ~50k notes; no vector-DB dependency).
+
+**Surface.** `hpr embed sync` (embed new/changed notes), `hpr search --semantic "<q>"`, and hybrid `--ranked --semantic` (reciprocal-rank-fusion of FTS and cosine lists). Pipeline use: step-10 curation and step-13 gap-check ("does the vault already cover X?") — semantic match catches conceptually-relevant sources FTS keyword match misses, which matters at 300+ sources.
+
+**Fallback.** Everything degrades gracefully with `provider = "none"` (the default): no API key required for core functionality.
+
+## Dependencies
+
+- Phase 0 (single fetch engine to hook utility/DOI persistence into; prompts as editable markdown).
+- Phase 1 (skill/agent prompt changes ride the template system; `[ranking]`/`[embeddings]` config sections).
+
+## Acceptance criteria
+
+- [ ] Migration v9 applies cleanly to the live 804-note vault; `sync` round-trips new frontmatter fields.
+- [ ] After a pipeline run: every fetched note has `utility_score`; DOI-bearing notes have `citation_count`/`authority_score`; `hpr graph rank` populates centrality; `quality_score` computed for all.
+- [ ] `hpr sources score` respects rate limits, caches, and survives API outages (partial enrichment, no crash).
+- [ ] A retracted DOI (test fixture) drives `quality_score` to floor and surfaces in step-2 coverage output.
+- [ ] `hpr search --ranked` reorders results by quality (test: same BM25 score, different tiers → ground_truth first); default search unchanged.
+- [ ] `hpr claims ingest/list/search` work; claims survive re-ingest idempotently.
+- [ ] `hpr embed sync` + `--semantic` return sane neighbors on the live vault with an API provider configured; `provider="none"` leaves all other features working.
+- [ ] Step-10 skill template consumes ranked curation; bench spot-run shows no RACE regression.
+
+## Risks & mitigations
+
+- **API coverage gaps** — much of a web corpus has no DOI. Mitigation: composite renormalizes over available components; tier weighting carries non-academic sources.
+- **Score gaming the drafts** — over-weighting authority buries good practitioner sources. Mitigation: weights are config; the orchestrator refines rather than blindly takes top-K; bench comparison before/after.
+- **Embedding cost/privacy** — vault bodies leave the machine with API providers. Mitigation: opt-in (`none` default), embed only title+summary+excerpt, document it.
+- **Migration risk on live vaults** — additive ALTERs only; no table rebuilds; back up `.hyperresearch/hyperresearch.db` in the migration (the file is disposable cache anyway — worst case `sync` rebuilds).
+
+## Effort
+
+| WS | Size |
+|---|---|
+| WS1 schema + utility persistence | M |
+| WS2 citation metadata CLI | M |
+| WS3 vault PageRank | S |
+| WS4 composite + retrieval | M |
+| WS5 claims table | M |
+| WS6 embeddings | L |
diff --git a/docs/roadmap-2.0/phase-3-dissertation-scale.md b/docs/roadmap-2.0/phase-3-dissertation-scale.md
new file mode 100644
index 0000000..583a920
--- /dev/null
+++ b/docs/roadmap-2.0/phase-3-dissertation-scale.md
@@ -0,0 +1,146 @@
+# Phase 3 — Dissertation scale: chaptered execution, run isolation, hundreds of sources
+
+## Goal
+
+One run produces a PhD-dissertation-scale report: 25K–80K words, 250–450 sources, chaptered structure, literature-review matrix — in a single multi-hour session that is resumable, budget-governed, and safe to run alongside other runs. The architectural move is **hierarchy**: the proven 40–80-source pipeline becomes the per-chapter unit, with global reconcile/synthesis/critic layers above it. No single agent ever holds the whole corpus.
+
+## Non-goals
+
+No changes to the per-chapter research mechanics (steps 2–10 run as-is within a chapter). No multi-machine distribution. Browser-lane sourcing is Phase 4; verification depth is Phase 5.
+
+## Current state (audit anchors, v0.8.7)
+
+- Run artifacts write to **flat paths**: `research/scaffold.md`, `research/loci.json`, `research/comparisons.md`, `research/temp/*`, `research/critic-findings-*.json`, `research/patch-log.json`. The 0.8.6 changelog explicitly flags that two overlapping runs race on these and defers per-run dirs as "a deeper refactor."
+- Recovery is implicit: "find the highest-numbered step whose artifact exists" (router skill, Recovery section). No manifest, no spend tracking, no explicit resume command. `hpr archive-run` moves *prior* run artifacts aside; `hpr vault-tag` mints collision-safe tags.
+- Scale ceilings (all profile parameters after Phase 1): source target 55–80 with a "beyond ~80 diminishing returns" rationale (width-sweep ~line 227); loci ≤ 6; depth budget 40; 3 drafts × ≤50 must-reads (a 300-source corpus would be mostly unread by any drafter); word ceiling 10K (`argumentative`); evidence digest ≤120 claims.
+- Lint rules glob flat paths: `wrapper-report`, `patch-surgery`, `instruction-coverage` look for `research/…/final_report*.md` and flat artifact names (`cli/lint.py:516,1006,1103`).
+- Pacing constants tuned for ~1-hour runs: ≤1 vault check/min, 30–60s note appends, 80% wave-done threshold.
+
+## Workstreams
+
+### WS1 — Per-run workspaces: `research/runs/<vault_tag>/` (M)
+
+**Design.** All run-scoped artifacts move under a per-run directory; vault notes stay global (the vault is the shared, compounding asset — runs are ephemeral workspaces over it).
+
+```
+research/runs/<vault_tag>/
+ run.json # manifest (WS2)
+ query.md # replaces research/query-<tag>.md
+ scaffold.md
+ prompt-decomposition.json
+ loci.json # per-chapter under chapters/ when chaptered
+ comparisons.md
+ corpus-critic-gaps.json
+ critic-findings-*.json
+ patch-log.json polish-log.json readability-*.json
+ temp/ # everything currently in research/temp/
+ chapters/ch1/ … chapters/chN/ # WS3: per-chapter artifact sets
+ escalation-queue.json # Phase 4
+```
+
+**Changes.**
+- `Vault` gains `runs_dir` / `run_dir(tag)` properties (`core/vault.py` alongside the existing dir properties); `hpr run init <tag>` scaffolds it.
+- Skills/agents templates: every artifact path becomes `{{ run_dir }}/…` (Phase 1's template system makes this a variable substitution, not a 30-file hand edit).
+- `compute_sync_plan` (`core/sync.py`) already skips `research/` root staging files; extend the skip to `research/runs/**` except any note-shaped files intentionally placed there (keep the existing frontmatter-probe logic — reuse `_has_frontmatter`).
+- Final reports **stay** at `research/notes/final_report_<vault_tag>.md` — they are vault notes, the product.
+- Lint rules that glob flat paths get run-aware resolution: `--run <tag>` flag, defaulting to the newest dir under `research/runs/`; keep legacy flat-path fallback for pre-2.0 vaults (the 0.8.5 precedent: rules already glob `final_report*.md` with back-compat).
+- `hpr archive-run` becomes trivial (move one directory) but stays for pre-2.0 layouts.
+
+**Resolves** the 0.8.6-flagged parallel-run race: two concurrent runs own disjoint trees. Vault-level contention (SQLite writes) is already serialized by WAL + `BEGIN IMMEDIATE`.
+
+### WS2 — Run manifest + explicit resume (M)
+
+**Design.** `run.json`, written by the orchestrator at each step boundary (schema versioned):
+
+```json
+{
+ "vault_tag": "china-rail-x9f2a1", "profile": "dissertation",
+ "started_at": "...", "updated_at": "...",
+ "status": "running | paused | blocked | done | failed",
+ "steps": { "1": {"status": "done", "finished_at": "..."},
+ "2": {"status": "running", "chapter": 3} },
+ "chapters": { "ch1": {"title": "...", "status": "done", "sources": 62},
+ "ch3": {"status": "width-sweep"} },
+ "spend": { "estimated_usd": 41.2, "sources_fetched": 187, "notes_written": 214 },
+ "blocked_on": null
+}
+```
+
+**Surface.** `hpr run status [<tag>] [-j]` (human + agent readable), `hpr run resume <tag>` (prints the resume position + the exact Skill invocation to continue with — the orchestrator consumes this instead of the artifact-existence heuristic, which remains the fallback), `hpr run abort <tag>`. The router skill's Recovery section rewrites to: read `run.json` first; artifact scan only if the manifest is missing/corrupt.
+
+**Spend tracking.** The orchestrator can't see true token costs; track proxies it *can* observe (sources fetched, subagents spawned by type × profile-configured cost estimates). Good enough for the budget governor; exact accounting is out of scope.
+
+### WS3 — Chaptered execution (L — the core of this phase)
+
+**Design.** New **step 1.5 (chapter partition)**, full/dissertation profiles only, triggered when decomposition's atomic-item count or the profile demands it (`chapters = "auto"` → 1 chapter below ~12 atomic items, else 4–10 grouped by topical cohesion; `chapters = N` to force).
+
+Execution shape:
+
+```
+1 decompose → 1.5 partition
+for each chapter (2–3 chapters concurrently, profile-capped):
+ 2 width sweep → 3 contradictions → 4 loci → 5 depth → 6 reconcile
+ → 7 tensions → 8 corpus critic → 9 digest → 10 chapter draft (single draft per chapter)
+global:
+ 6g cross-CHAPTER reconcile (chapter positions → global tensions doc)
+ 11g global synthesis (chapter drafts → integrated dissertation, chapter H1s preserved)
+ 12 critics (per-chapter instruction/width; global dialectic/depth)
+ 13 gap-fetch → 14 patcher → 15 polish → 16 readability (whole document)
+```
+
+**Key decisions.**
+- **Chapter tagging:** `<vault_tag>-ch<N>` alongside `<vault_tag>` on every note, so per-chapter queries (`--tag <tag>-ch3`) and whole-run queries both work. Cross-chapter source *reuse* is free — a source fetched for ch2 is findable by ch5's coverage check via the ranked vault search (Phase 2), so chapters don't re-fetch each other's sources (dedup is by URL in `sources` anyway).
+- **Per-chapter drafting replaces the triple-draft ensemble at chapter level.** The triple-draft's purpose (angle diversity) is served globally: chapters *are* the decomposition, and global synthesis integrates across them. Profile flag `chapter_draft_count = 1` (dissertation) vs `draft_count = 3` (full, unchaptered) keeps the 1.x behavior intact for unchaptered runs.
+- **Critics at two altitudes:** instruction-critic checks the global document against `required_section_headings` + chapter contract; dialectic/depth critics run per-chapter (bounded context) with findings merged; width-critic runs globally against the full corpus tag. Critic finding caps scale by chapter count (profile).
+- **Synthesizer context limits:** the global synthesizer reads chapter drafts (each ≤ ~8K words) + global reconcile doc — bounded regardless of corpus size. For 80K-word outputs, synthesis proceeds chapter-by-chapter against the global outline (two-pass per chapter, one voice doc as the running style contract) — same Read+Write tool-lock, writing one file incrementally.
+- **Concurrency:** chapters are independent through step 9; run 2–3 concurrently (profile `chapter_concurrency`), bounded by subagent-slot pressure. WS1's per-chapter dirs make this race-free; the orchestrator interleaves via its existing task-notification loop.
+
+**New/changed skills:** `hyperresearch-1.5-chapter-partition` (new), chapter-loop control in the router, `6g`/`11g` global variants (new skills or profile-gated sections of 6/11). All numeric knobs (chapter count bounds, per-chapter source targets, concurrency) live in `[profile.dissertation]`.
+
+### WS4 — Budget governor (S)
+
+**Design.** `--budget <usd>` at invocation → `run.json.budget`. The orchestrator checks `hpr run status -j` at step boundaries; the router skill gains a "budget check" rule: remaining budget scales the *next* step's parameters (fewer wave-2 fetchers, lower depth budgets, fewer chapters) using profile-defined floors — never silently skipping tier-mandated steps, instead shrinking their fan-out and recording the decision in the manifest. Hard stop at 100%: pause with `status: blocked`, `blocked_on: "budget"`, surface to user.
+
+### WS5 — Dissertation artifacts + profile (M)
+
+- **`[profile.dissertation]`:** steps incl. 1.5; `source_min = 250`, `source_target = [300, 450]` (allocated per chapter: each chapter runs the proven 40–80 envelope); `loci_max = 20` total (≤4/chapter); `depth_budget_total = 160`; `claims_cap = [400, 600]` global (per-chapter digests + global index); `word_targets.dissertation = [25000, 80000]`; `chapter_concurrency = 2`; scaled critic caps; pacing constants (WS6).
+- **New `response_format: dissertation`** in step-1 decompose: chaptered H1s, per-chapter exec summaries, global abstract + conclusion + full bibliography.
+- **Literature-review matrix:** `hpr claims matrix --tag <vault_tag> [-o file.md]` — per-source rows (source, year, tier, method/evidence_type, key findings from claims, limitations from adversarial claims, quality_score) generated from the Phase-2 claims table; embedded as a dissertation appendix and used by chapter drafters. Pure Python + one small LLM-free aggregation; drafters add prose interpretation.
+- **Numeric meta-analysis pass:** new step-9 sub-step (dissertation profile): group claims sharing a `stance_target`/quantity across ≥5 sources → comparison table with ranges/outliers flagged → shipped into evidence digest. Extraction is from claims' `numbers` fields (already captured); no new agent, orchestrator-level aggregation via `hpr claims list -j`.
+
+### WS6 — Pacing for multi-hour runs (S)
+
+Profile-parameterize the cadence constants (vault-check interval, note-append cadence, wave-done ratio, subagent failure-retry counts) — dissertation runs check less frequently and tolerate longer waves. Add heartbeat writes to `run.json.updated_at` so `hpr run status` can flag a stalled run (no update > N minutes → `possibly-stalled` warning).
+
+## Dependencies
+
+- **Phase 1 (hard):** every scale number here is a profile parameter; path rewiring rides templates.
+- **Phase 2 (strong):** ranked curation + claims table power chapter curation, the lit-review matrix, and meta-analysis. A Phase-3-without-Phase-2 fallback (orchestrator-judgment curation per chapter) works but is explicitly degraded.
+- Phase 0 transitively.
+
+## Acceptance criteria
+
+- [ ] Two concurrent runs with different tags share a vault with zero artifact collisions (integration test: parallel `run init` + artifact writes + lint).
+- [ ] Kill a run mid-step-5; `hpr run resume` reports the correct position; the orchestrator continues without re-fetching completed chapters (manifest test + manual run).
+- [ ] Lint suite passes with run-dir layout and still passes on a legacy flat-layout vault.
+- [ ] A dissertation-profile run on a broad query produces: ≥250 sources across ≥4 chapters, chaptered ≥25K-word report, lit-review matrix appendix, all integrity gates green.
+- [ ] Budget governor: a run with `--budget` under-shoots the cap and records scaling decisions in the manifest; hard-stop pauses cleanly.
+- [ ] `full` (unchaptered) profile behavior is byte-compatible with 1.x pipeline semantics (golden bench query).
+
+## Risks & mitigations
+
+- **Coherence across chapters** — chaptered writing can read as an anthology. Mitigation: global reconcile (6g) + running voice doc in 11g + global critics; the synthesizer's existing "one prose voice, not section-grafted" contract applies per the whole document.
+- **Cost blowout** — 300+ sources ≈ $200–500/run. Mitigation: budget governor is in this phase for that reason; dissertation profile is opt-in.
+- **Orchestrator context over multi-hour runs** — even with fresh-loaded step skills, the conversation grows. Mitigation: the manifest makes recovery cheap and explicit; chapter loops re-invoke the router; consider a per-chapter sub-orchestrator agent if single-session limits bite (flagged, not committed).
+- **Migration** — pre-2.0 vaults with flat artifacts. Mitigation: lint/back-compat fallbacks (WS1); `archive-run` still handles old layouts; no forced migration.
+
+## Effort
+
+| WS | Size |
+|---|---|
+| WS1 run workspaces | M |
+| WS2 manifest + resume | M |
+| WS3 chaptered execution | L |
+| WS4 budget governor | S |
+| WS5 dissertation artifacts | M |
+| WS6 pacing | S |
diff --git a/docs/roadmap-2.0/phase-4-chrome-lane.md b/docs/roadmap-2.0/phase-4-chrome-lane.md
new file mode 100644
index 0000000..64d3c9f
--- /dev/null
+++ b/docs/roadmap-2.0/phase-4-chrome-lane.md
@@ -0,0 +1,128 @@
+# Phase 4 — The Chrome lane: hard-target fetching via Claude in Chrome
+
+## Goal
+
+Sources that headless crawling cannot reach — login-gated, bot-walled, interactive, viewer-rendered — stop being silently lost. A blocked URL escalates to a lane that drives the user's **real Chrome browser with their real logged-in sessions** (via Claude-in-Chrome MCP tools), with a human-in-the-loop checkpoint for the challenges only a human should complete. Chrome becomes the key-master; headless crawl4ai stays the high-throughput workhorse.
+
+## Non-goals / scope boundary
+
+**No automated CAPTCHA solving and no bot-detection defeat.** When the lane hits a CAPTCHA, 2FA prompt, or login it cannot legitimately pass, it pauses and hands off to the human — it does not attempt to solve, spoof, or evade. This is a hard scope boundary, not a deferred feature. Also out of scope: any credential storage by hyperresearch (sessions live in the user's browser profile, where they already are), and paywall circumvention (the lane fetches what the user's own subscriptions can see).
+
+## Current state (audit anchors, v0.8.7)
+
+- Detection already exists, recovery doesn't: `looks_like_login_wall` / `looks_like_junk` (`web/base.py:60-143`) correctly identify blocked fetches — and then the URL is **discarded**. At dissertation scale that's dozens of silently lost sources per run.
+- The visible-browser fallback (`crawl4ai_provider.py` `_fetch_visible`) and crawl4ai profiles (`~/.crawl4ai/profiles/<name>`) exist, but they are a *separate* browser identity the user must log into via `hyperresearch setup` — not their daily sessions, and prone to session kills on aggressive sites.
+- Crawl4ai saves screenshots (`screenshot=True` in run config) but **nothing reads them** — no vision extraction path exists.
+- Google Scholar (no API, aggressive headless blocking) is unreachable by the current stack — the single best citation-chaining surface there is.
+- Provider seam is clean: `WebProvider` protocol + `WebResult` dataclass (`web/base.py:36`) — a new lane slots in without touching consumers.
+- Claude-in-Chrome MCP tools (`mcp__claude-in-chrome__*`: tabs, navigate, computer, read_page, get_page_text, find, javascript_tool, read_network_requests) are available to Claude Code sessions and subagents via ToolSearch.
+
+## Architecture: Python queues, agent drives
+
+The Chrome lane cannot be a pure-Python `WebProvider` — Claude-in-Chrome is an MCP surface driven by an *agent*, not an importable library. The design splits accordingly:
+
+- **Python side:** escalation queue management, queue-item lifecycle, result ingestion (reusing `write_note` + the Phase-0 unified fetch engine's note-creation path), cookie handoff.
+- **Agent side:** a new `hyperresearch-browser-fetcher` subagent that drains the queue by driving Chrome tabs, extracts content, and calls `hpr` to ingest results.
+
+## Workstreams
+
+### WS1 — Escalation queue + ladder (M)
+
+**Design.** The fetch engine gains an outcome taxonomy instead of discard-on-failure. On `login_wall`, `junk` (bot-block subtypes: cloudflare/captcha signals already distinguished in `base.py`), or repeated fetch failure:
+
+```
+crawl4ai (headless, parallel)
+ └─ blocked → enqueue research/runs/<tag>/escalation-queue.json
+ └─ browser-fetcher agent drains via Chrome
+ └─ human checkpoint (WS3) only if Chrome also can't proceed
+```
+
+Queue item schema:
+
+```json
+{ "url": "...", "reason": "login_wall | bot_block | captcha | fetch_failed | interactive_needed",
+ "requested_by": "step-2-wave-1", "suggested_by": "<note-id>", "utility_score": 14,
+ "status": "queued | in_progress | fetched | needs_human | abandoned",
+ "attempts": 1, "note_id": null }
+```
+
+**Changes.**
+- `core/fetcher.py`: on gate rejection, write the queue entry (append-safe JSON-lines or lock-guarded JSON) instead of just erroring; CLI/JSON output reports `escalated` as a distinct outcome so fetcher agents don't retry.
+- `hpr escalation list/add/claim/complete [-j]` — small CLI for queue lifecycle (agents interact through it; no direct file editing, avoiding races).
+- Step-2 and step-13 skills: after each wave, check `hpr escalation list -j`; if high-utility items are queued (utility ≥ profile threshold), spawn the browser-fetcher to drain them while the next wave proceeds. Low-utility blocked URLs are abandoned without ceremony — the ladder climbs only when the source is worth it.
+
+### WS2 — `hyperresearch-browser-fetcher` agent (L)
+
+**Design.** New subagent definition (installed like the others; Phase-0 layout):
+
+- **Tools:** `Bash, Read, ToolSearch` + the Claude-in-Chrome MCP tools (loaded via one batched ToolSearch per session, per the MCP server's own guidance). Model: sonnet.
+- **Contract:** claim queue items via `hpr escalation claim`; for each: open a new tab (never reuse user tabs), navigate, wait for content, extract via `get_page_text`/`read_page`; for PDFs rendered in a viewer, trigger download or extract text from the viewer DOM; write the note via `hpr note new --body-file` with `fetch_provider: chrome` frontmatter + the standard provenance fields; mark item `fetched` with the note id.
+- **Interactive extraction playbook (in the prompt):** infinite scroll (scroll-until-stable with a hard cap), "load more"/accordion expansion, in-site search when the target is a query not a URL, SPA navigation waits. Session-kill awareness: prefer reading over clicking on auth-sensitive sites; never log out, never change account state; do not navigate to URLs outside the claimed item's domain except for redirects.
+- **Vision-assisted extraction:** when the page is chart/figure-heavy or text extraction is thin, screenshot and read the image, transcribing figures/axis values into the note body under a `## Extracted figures` section. (This also unlocks reading the screenshots crawl4ai already saves — add a `hpr assets path` + Read step to the source-analyst prompt as a cheap rider.)
+- **Escalation out:** on CAPTCHA/2FA/login → mark item `needs_human` with a one-line description, move on. Never attempt to solve.
+- Batch behavior: drain up to N items (profile) per spawn; sequential, one tab at a time (Chrome lane is precious/low-throughput by design).
+
+**Concurrency note:** exactly one browser-fetcher at a time (the queue `claim` op enforces it) — parallel agents fighting over one browser is chaos.
+
+### WS3 — Human-in-the-loop checkpoint (M)
+
+**Design.** When items sit at `needs_human`, the run pauses *that lane* — not the pipeline:
+
+- The orchestrator, at step boundaries, checks for `needs_human` items; if present and the item's utility justifies it, it surfaces a **single consolidated prompt** to the user: "3 sources need you: [site A: solve CAPTCHA], [site B: log in], [site C: 2FA]. Open them in Chrome, complete the challenge, then say 'done' (or 'skip')." In non-interactive (`-p`) runs, this writes `run.json.blocked_on = "human-challenges"` + a notification (stdout marker the harness can surface / OS notification where available) and continues with everything else; the queue drains on the next resume.
+- After the human completes challenges in their own browser, the browser-fetcher retries the items — the session now carries the solved state.
+- The consolidated-prompt pattern matters: one interruption per run, not one per URL. Batch `needs_human` items and raise them at natural pause points (end of wave, step boundary).
+
+**This checkpoint is the CAPTCHA/2FA/login story, in full.** The human does human things in their own browser for ten seconds; the pipeline gets the source. Document this verbatim in the agent prompt and user docs so expectations are unambiguous.
+
+### WS4 — Session handoff: Chrome → headless (M)
+
+**Design.** After an interactive login succeeds in Chrome, subsequent fetches from that domain shouldn't need the Chrome lane. `hpr session handoff <domain>`:
+
+- Export cookies for the domain from the Chrome profile into the crawl4ai profile's storage state (`~/.crawl4ai/profiles/<name>` is a Playwright `user_data_dir`; write a `storage_state.json` the provider loads, or inject cookies via Playwright's `add_cookies` on context creation — implementation detail: crawl4ai supports `storage_state` in browser config).
+- Cookie *reading* from Chrome: via the Claude-in-Chrome `javascript_tool` (`document.cookie` covers non-HttpOnly only) is insufficient — instead have the browser-fetcher trigger the handoff by exporting through Chrome's DevTools-adjacent surface if available, or fall back to instructing the user to run `hyperresearch setup` for that site once (the existing flow) now that they know it's worth it. **Design honestly around the constraint: HttpOnly cookies are not scriptable from page JS.** The realistic v1: session handoff works where the site's session survives in the crawl4ai profile after one guided login there; the Chrome lane covers the rest directly.
+- Track handoff domains in config so the fetch engine prefers headless-with-profile for them and only re-escalates on a fresh login wall.
+
+**Honest framing for the doc:** WS4 is an optimization with a hard platform constraint; the Chrome lane (WS2) is the guaranteed path. Ship WS2/WS3 first, treat WS4 as best-effort.
+
+### WS5 — Google Scholar lane (S)
+
+**Design.** A browser-fetcher playbook specialization, not new machinery: queue items of type `scholar-search` carry a query instead of a URL; the agent searches Scholar in a tab, extracts the top-N results (title, authors, year, citation count, link, "cited by" link), and returns them as a structured result note (`type: index`, tagged `scholar-results`) that step-2's search planning consumes like an academic-API response. Cited-by chaining: enqueue follow-up items for high-value hits. Rate discipline in the prompt: human-paced interactions, small N, one query at a time — this is a courtesy lane, not a scraper.
+
+### WS6 — Config + profile integration (S)
+
+`[chrome]` config: `enabled`, `escalation_utility_threshold`, `max_items_per_run`, `drain_batch_size`, `scholar_enabled`. Profiles set lane budgets (light: disabled by default; full: modest; dissertation: generous). `hpr run status` shows queue depth + `needs_human` count.
+
+## Dependencies
+
+- Phase 0 (unified fetch engine to hook the outcome taxonomy into).
+- Phase 1 (config/profile plumbing; agent prompts as templates).
+- Phase 3's run dirs give the queue its home (`research/runs/<tag>/`); if Phase 4 lands first, the queue lives at `research/temp/escalation-queue.json` with a migration note.
+- External: Claude-in-Chrome extension installed + site permissions granted (document in setup docs; the lane degrades to "queue accumulates, nothing drains" without it — surfaced in `run status`).
+
+## Acceptance criteria
+
+- [ ] A login-walled URL (test fixture site) is enqueued, not discarded; fetch JSON reports `escalated`.
+- [ ] Browser-fetcher drains a queue of public-but-interactive pages (infinite scroll fixture) into well-formed notes with `fetch_provider: chrome` and correct provenance.
+- [ ] A CAPTCHA/login page results in `needs_human` (never an attempted solve), consolidated into one user prompt; after manual completion, retry succeeds.
+- [ ] Non-interactive run with blocked items completes everything else and records `blocked_on`; resume drains after human action.
+- [ ] Scholar lane returns structured results for a query and enqueues cited-by follow-ups.
+- [ ] With Chrome unavailable, the pipeline runs exactly as 1.x (lane cleanly disabled).
+- [ ] Queue operations are race-safe under concurrent fetcher waves (claim/complete integration test).
+
+## Risks & mitigations
+
+- **Site terms-of-service and rate discipline** — the lane touches sites that resist automation. Mitigation: human-paced playbooks, per-run caps, courtesy framing in prompts, and the human checkpoint keeps a person in the loop for exactly the interactions sites gate on. The lane reads what the user's own access can see; it does not evade.
+- **Browser-agent fragility** — selectors and page structures vary wildly. Mitigation: text-extraction-first (get_page_text) over DOM surgery; per-item attempt caps; abandon gracefully (an abandoned source is where we started — lost — so the floor is the status quo).
+- **User interruption fatigue** — mitigation: utility threshold before anything reaches `needs_human`; consolidated prompts; skip is always an option.
+- **One-browser bottleneck** — the lane is serial by design. Mitigation: it only receives high-utility escalations; throughput stays with headless waves.
+
+## Effort
+
+| WS | Size |
+|---|---|
+| WS1 queue + ladder | M |
+| WS2 browser-fetcher agent | L |
+| WS3 human checkpoint | M |
+| WS4 session handoff | M (best-effort) |
+| WS5 Scholar lane | S |
+| WS6 config | S |
diff --git a/docs/roadmap-2.0/phase-5-verification.md b/docs/roadmap-2.0/phase-5-verification.md
new file mode 100644
index 0000000..557cbaf
--- /dev/null
+++ b/docs/roadmap-2.0/phase-5-verification.md
@@ -0,0 +1,105 @@
+# Phase 5 — Verification: the layer that makes it better than a human
+
+## Goal
+
+Verification density no human researcher sustains: every citation checked against its source, every quote mechanically verified verbatim, every number traced to extracted evidence, every cited DOI swept for retractions, consensus counted only across *independent* sources — plus telemetry that makes pipeline tuning data-driven and a bench gate that stops prompt regressions from shipping. This is where machine research stops being merely faster and becomes more trustworthy.
+
+## Non-goals
+
+Not a fact-checking oracle: verification confirms the report faithfully represents its sources, not that the sources are true (source *quality* is Phase 2's job; truth remains the reader's). No new sourcing (Phases 2–4).
+
+## Current state (audit anchors, v0.8.7)
+
+- **No post-hoc citation verification exists.** The critics audit coverage/argument/instructions; none checks whether citation `[14]` actually supports the sentence it's attached to. The FACT half of DeepResearch-Bench measures exactly this, externally — the pipeline never self-measures it.
+- Quotes in reports are unverifiable mechanically today; `quoted_support` exists in claims JSONs (and, after Phase 2 WS5, in the `claims` table) but nothing cross-checks report text against it.
+- Retraction status: not checked anywhere (Phase 2 WS2 adds `is_retracted` at fetch/score time; nothing re-checks at ship time or for older vault sources being reused).
+- Independence: the step-2.6 redundancy audit tags `derivative-of` for >60% shared `quoted_support` — but consensus counting ("3+ independent sources agree", step 3) trusts the corpus is independent; syndicated/churnalism chains can inflate agreement.
+- Telemetry: none. No per-step cost/time record survives a run; profile tuning is vibes. `bench/` exists (DeepResearch-Bench harness, RACE/FACT via Gemini) but is manual and cluttered with ~20 `_*.py` scratch scripts (`bench/_analyze_drafts.py`, `_grade_q9_parallel.py`, etc.).
+- Lint architecture is ready for new rules: `RULES` dict registry (`cli/lint.py:16`), severity model, `--rule` selection, JSON output — new verifications slot in as rules.
+
+## Workstreams
+
+### WS1 — Cite-check: new step 14.5 (L)
+
+**Design.** After patching (step 14), before polish (15): verify that each citation supports its sentence. Runs for full + dissertation profiles.
+
+**Mechanics.**
+1. **Extract pairs (Python):** `hpr citecheck extract <report> -j` parses the report into (sentence, citation-target) pairs for both citation styles — `[N]`→Sources-list entry→note id, and `[[note-id]]` wikilinks. Pure parsing, reuses `core/patterns.py` wiki-link regexes.
+2. **Triage (Python):** for each pair, look up the cited note's claims (Phase 2 claims table). Auto-pass tier: the sentence's numbers/entities appear in a claim's `quoted_support` or `numbers` (string/number match) → verified-mechanical. Remainder goes to the LLM tier.
+3. **LLM tier:** a new `hyperresearch-cite-checker` subagent (sonnet; Read + Bash for `note show`) receives batches of (sentence, note-id, note-body-excerpt) and returns per-pair verdicts: `supported | partially-supported | unsupported | wrong-source` (with the note id that *does* support it when findable via claims search). Sampling for scale: 100% of number-bearing and strong-claim sentences; profile-configurable sample rate (default 60%) for the rest; dissertation profile checks 100% of chapter-topic sentences.
+4. **Findings:** `research/runs/<tag>/cite-check-findings.json` in critic-findings format → the **patcher runs a second, small pass** (reuse step-14 machinery verbatim: same agent, same tool lock, appended patch-log section) fixing `unsupported`/`wrong-source` items — swap citation, soften claim to what the source supports, or escalate.
+
+**Why 14.5, not part of step 12:** critics read the pre-patch draft; cite-check must audit what will actually ship, after patch hunks moved text and citations.
+
+### WS2 — Mechanical lints: quote-integrity + numeric-consistency (M)
+
+New rules in the `RULES` registry, both pure Python, both run in the step-15 gate:
+
+- **`quote-integrity`:** every quoted span in the final report (≥ N words, default 5, config) must appear verbatim (whitespace-normalized) in some vault note body — checked via SQL `LIKE`/FTS phrase query against `note_content.body_plain`, scoped to the run's tag first, whole vault second. Unmatched → `error` with the quote and nearest fuzzy match (difflib) so the fix is fast. Kills hallucinated quotes dead.
+- **`numeric-consistency`:** every number in the report (excluding structural numbers — dates in headings, section numbers, table indices; maintain an exclusion regex) must appear in (a) a claims-table `numbers`/`quoted_support` entry, (b) a cited note's body, or (c) a computed-value allowlist the meta-analysis pass (Phase 3 WS5) emits for derived figures. Unmatched → `warning` (not error — legitimate arithmetic exists), listed with locations. Severity configurable; dissertation profile promotes to error.
+
+Both rules respect run-dir resolution (Phase 3) and legacy layout.
+
+### WS3 — Retraction sweep at ship time (S)
+
+**Design.** `hpr sources retractions --tag <vault_tag> [-j]`: for every note *cited in the final report*, re-check `is_retracted` (Phase 2's cached Crossref/OpenAlex path; TTL-bypassing `--fresh` flag for the ship-time run). New lint rule **`retracted-citations`** (`error`): a cited retracted source blocks the integrity gate; the report must either drop it or cite it *as* retracted (an explicit `(retracted)` marker in the citation satisfies the rule — sometimes the retraction is the story). Step 15 skill adds the sweep before the gate. Also valuable for **vault reuse**: old sources fetched in prior runs get re-checked when newly cited.
+
+### WS4 — Independence audit (M)
+
+**Design.** Generalize `derivative-of` into a first-class independence model feeding consensus math:
+
+- **Detection upgrades (Python assist + existing audit):** beyond >60% shared quotes — shared canonical URL after redirect resolution, near-duplicate bodies (reuse MinHash from `core/similarity.py` — it exists for dedup already), same wire-service/PR boilerplate signatures (dateline + "PRNewswire/Business Wire/AP" markers), and same-domain clustering. `hpr sources independence --tag <tag>` computes clusters and writes `independence` scores (Phase 2 column): 1.0 for cluster roots, discounted members share credit.
+- **Consensus integration:** step 3's consensus rule ("3+ independent sources agree") and step 9's digest change from counting *sources* to counting *independence-weighted clusters* — the skill templates read cluster data via `hpr sources independence -j`. Five syndicated copies of one press release = one vote.
+- **Report surface:** the synthesizer's confidence language keys off weighted counts; the polish auditor's job is unchanged.
+
+### WS5 — Run telemetry + `hpr run report` (M)
+
+**Design.** Extend the Phase-3 manifest into a queryable record:
+
+- Per-step: wall-time, subagents spawned (type × count), sources fetched/escalated/abandoned, claims ingested, findings raised/applied/skipped, estimated cost (profile cost-model × spawn counts).
+- `hpr run report <tag>` renders a post-run summary (markdown + `-j`); `hpr run report --all -j` aggregates across runs for tuning questions ("what does wave 2 actually yield?", "how often does gap-fetch fire?", "cite-check unsupported rate by profile").
+- Storage: `run.json` for live state + an append-only `research/runs/<tag>/events.jsonl` for per-event granularity (step start/end, spawn, fetch outcome). Orchestrator writes events through `hpr run event` so the format stays uniform.
+- This is the feedback loop for every profile number Phase 1 made tunable — replace inherited constants with observed yield curves.
+
+### WS6 — Bench as CI regression gate (M)
+
+**Design.**
+- **Cleanup:** move `bench/_*.py` scratch scripts (~20 files: `_analyze_drafts.py`, `_audit_references*.py`, `_grade_q9_parallel.py`, `_predict_full100.py`, etc.) into `bench/archive/` or delete; keep `harness.py`, `setup.sh`, `evaluate.sh`, README as the supported surface.
+- **Smoke tier:** `bench/harness.py --smoke` — 3 fixed queries (1 light, 1 full-EN, 1 full-ZH), assertions on *structural* outcomes that don't need Gemini: report exists, length in profile range, required headings present, citation density ≥ floor, integrity gate green, cite-check unsupported rate < threshold (WS1 gives us an internal quality metric — this is the point: CI can gate on self-measured citation faithfulness without external judges).
+- **Trigger:** manual/nightly workflow (`workflow_dispatch` + schedule) — not per-PR (each run costs real money); PRs touching `src/hyperresearch/skills/` or `agents/` get a bot comment reminding that bench-smoke hasn't validated the change until the nightly passes. Full 100-query RACE/FACT runs stay manual for release candidates.
+- **Regression memory:** smoke results append to `bench/results/history.jsonl`; the workflow diffs against the trailing median and fails on >X% structural-metric regression.
+
+## Dependencies
+
+- **Phase 2 (hard for WS1/WS3):** claims table powers triage; DOI metadata powers retractions.
+- **Phase 3 (soft):** run dirs/manifest for findings paths and telemetry home; WS2 lints work without it via legacy layout.
+- Phase 0/1 transitively (templates, config).
+
+## Acceptance criteria
+
+- [ ] Cite-check on a seeded report with 3 planted defects (unsupported claim, wrong-source citation, fabricated quote) flags all 3; patcher pass fixes or escalates them; a clean report passes with < 5% LLM-tier flags.
+- [ ] `quote-integrity` catches a planted fabricated quote; passes on a legitimate report; runs in seconds on the live vault.
+- [ ] `numeric-consistency` traces report numbers to claims/bodies; exclusion regex keeps false positives < ~10% on example-reports/.
+- [ ] A cited retracted DOI (fixture) blocks the gate; `(retracted)`-marked citation passes.
+- [ ] Independence: a fixture cluster of 4 syndicated copies + 1 original counts as ~1 consensus vote, not 5.
+- [ ] `hpr run report` produces per-step cost/time/yield for a completed run; `--all` aggregates.
+- [ ] Nightly bench-smoke workflow runs green on main and fails on a deliberately broken skill (test the gate itself once).
+- [ ] `bench/` root contains only supported files.
+
+## Risks & mitigations
+
+- **Cite-check cost** — LLM-tier at dissertation scale is thousands of pairs. Mitigation: mechanical triage first (claims matching auto-passes the bulk), sampling tiers, sonnet not opus, batched pairs per spawn.
+- **False-positive lints eroding trust** — a noisy numeric-consistency rule gets ignored. Mitigation: warning severity by default, exclusion regex tuned on `example-reports/`, promote to error only for dissertation profile after observed precision.
+- **Second patcher pass destabilizing a polished draft** — 14.5 runs *before* 15 precisely so polish sees the final text; patch-surgery lint already guards unapplied criticals.
+- **Bench flakiness in CI** — live-web runs are nondeterministic. Mitigation: structural assertions (not score thresholds), trailing-median comparison, nightly not per-PR.
+
+## Effort
+
+| WS | Size |
+|---|---|
+| WS1 cite-check | L |
+| WS2 mechanical lints | M |
+| WS3 retraction sweep | S |
+| WS4 independence | M |
+| WS5 telemetry | M |
+| WS6 bench CI | M |
diff --git a/pyproject.toml b/pyproject.toml
index e1be874..5ed1a5b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -79,6 +79,17 @@ ignore = [
"SIM105",
]
+[tool.ruff.lint.per-file-ignores]
+# Profiles/render carry deliberate en dashes (user-facing range strings and
+# the dash filter); tests assert on en-dash-rendered prompt output. The
+# "ambiguous unicode" warnings are false positives by construction there.
+"src/hyperresearch/core/profiles.py" = ["RUF001"]
+"src/hyperresearch/core/render.py" = ["RUF001", "RUF002"]
+"src/hyperresearch/core/citecheck.py" = ["RUF001"]
+"tests/test_core/test_render.py" = ["RUF001", "RUF002"]
+"tests/test_core/test_prompt_golden.py" = ["RUF001", "RUF002"]
+"tests/test_cli/test_install_profile.py" = ["RUF001", "RUF002"]
+
[tool.mypy]
python_version = "3.11"
strict = true
diff --git a/src/hyperresearch/cli/__init__.py b/src/hyperresearch/cli/__init__.py
index 9187726..d797783 100644
--- a/src/hyperresearch/cli/__init__.py
+++ b/src/hyperresearch/cli/__init__.py
@@ -127,9 +127,22 @@ app.add_typer(git_app, name="git", help="Git integration.")
app.add_typer(tag_app, name="tag", help="Tag management.")
from hyperresearch.cli.assets import app as assets_app
+from hyperresearch.cli.citecheck_cmd import app as citecheck_app
+from hyperresearch.cli.claims_cmd import app as claims_app
+from hyperresearch.cli.embed_cmd import app as embed_app
+from hyperresearch.cli.escalation_cmd import app as escalation_app
from hyperresearch.cli.link import app as link_app
+from hyperresearch.cli.profile_cmd import app as profile_app
+from hyperresearch.cli.run_cmd import app as run_app
from hyperresearch.cli.sources import app as sources_app
+app.add_typer(profile_app, name="profile", help="Pipeline profiles (scale parameters).")
+app.add_typer(claims_app, name="claims", help="Fetcher-extracted claims (ingest + query).")
+app.add_typer(embed_app, name="embed", help="Semantic-search embeddings.")
+app.add_typer(run_app, name="run", help="Per-run workspaces + manifest (init/status/resume).")
+app.add_typer(escalation_app, name="escalation", help="Browser-lane queue for blocked fetches.")
+app.add_typer(citecheck_app, name="citecheck", help="Citation-sentence binding verification.")
+
app.add_typer(sources_app, name="sources", help="Fetched web sources.")
app.add_typer(assets_app, name="assets", help="Downloaded images, screenshots, and media.")
app.add_typer(link_app, name="link", help="Auto-discover and insert wiki-links.")
diff --git a/src/hyperresearch/cli/citecheck_cmd.py b/src/hyperresearch/cli/citecheck_cmd.py
new file mode 100644
index 0000000..1955b70
--- /dev/null
+++ b/src/hyperresearch/cli/citecheck_cmd.py
@@ -0,0 +1,68 @@
+"""Cite-check commands — extract citation pairs + mechanical triage."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def citecheck_extract(
+ vault_tag: str = typer.Argument(..., help="Run tag (pairs file lands in the run workspace)"),
+ report: str | None = typer.Option(None, "--report", help="Report path (default: research/notes/final_report_<tag>.md)"),
+ sample_rate: float = typer.Option(0.6, "--sample-rate", help="Fraction of weak (non-number-bearing) needs-llm pairs to check; number-bearing sentences are always 100%"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Extract (sentence, citation) pairs from the final report and triage them.
+
+ Auto-passes pairs whose numbers/wording the cited note's claims already
+ confirm; writes the sampled needs-llm remainder + dangling citations to
+ runs/<tag>/cite-check-pairs.json for the cite-checker agent.
+ """
+ from hyperresearch.core.citecheck import write_pairs_file
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ report_path = Path(report) if report else (
+ vault.root / "research" / "notes" / f"final_report_{vault_tag}.md"
+ )
+ if not report_path.exists():
+ msg = f"report not found: {report_path}"
+ if json_output:
+ output(error(msg, "NO_REPORT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {msg}")
+ raise typer.Exit(1)
+
+ result = write_pairs_file(vault, vault_tag, report_path, sample_rate=sample_rate)
+ summary = result["summary"]
+ data = {
+ "pairs_file": str(vault.run_dir(vault_tag) / "cite-check-pairs.json"),
+ "summary": summary,
+ "sampled_for_llm": len(result["sampled_for_llm"]),
+ "dangling": len(result["dangling"]),
+ }
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(
+ f"[green]Cite-check triage:[/] {summary['total']} pairs — "
+ f"{summary['supported_mechanical']} auto-passed, "
+ f"{data['sampled_for_llm']} sampled for LLM verification, "
+ f"{summary['dangling']} dangling"
+ )
diff --git a/src/hyperresearch/cli/claims_cmd.py b/src/hyperresearch/cli/claims_cmd.py
new file mode 100644
index 0000000..7387356
--- /dev/null
+++ b/src/hyperresearch/cli/claims_cmd.py
@@ -0,0 +1,155 @@
+"""Claims commands — ingest and query fetcher-extracted claims."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def _vault_or_exit(json_output: bool):
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ return Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+
+def claims_ingest(
+ paths: list[str] = typer.Argument(None, help="claims-*.json files (default: all under research/temp/)"),
+ vault_tag: str | None = typer.Option(None, "--tag", "-t", help="vault_tag to stamp on ingested claims"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Ingest claims JSON files into the claims table (idempotent)."""
+ from hyperresearch.core.claims import ingest_claims_dir, ingest_claims_file
+
+ vault = _vault_or_exit(json_output)
+ vault.auto_sync()
+
+ if paths:
+ summary = {"files": len(paths), "ingested": 0, "skipped": 0, "errors": []}
+ for p in paths:
+ r = ingest_claims_file(vault.db, Path(p), vault_tag)
+ summary["ingested"] += r["ingested"]
+ summary["skipped"] += r["skipped"]
+ summary["errors"].extend(f"{Path(p).name}: {e}" for e in r["errors"])
+ vault.db.commit()
+ else:
+ summary = ingest_claims_dir(vault, vault_tag=vault_tag)
+
+ if json_output:
+ output(success(summary, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(
+ f"[green]Ingested:[/] {summary['ingested']} claims "
+ f"({summary['skipped']} already present) from {summary['files']} file(s)"
+ )
+ for e in summary["errors"]:
+ console.print(f" [yellow]{e}[/]")
+
+
+def claims_list(
+ note: str | None = typer.Option(None, "--note", help="Only claims from this source note"),
+ vault_tag: str | None = typer.Option(None, "--tag", "-t", help="Only claims with this vault_tag"),
+ limit: int = typer.Option(100, "--limit", "-n", help="Max claims"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """List stored claims, optionally filtered by source note or vault_tag."""
+ from hyperresearch.core.claims import list_claims
+
+ vault = _vault_or_exit(json_output)
+ rows = list_claims(vault.db, note_id=note, vault_tag=vault_tag, limit=limit)
+
+ if json_output:
+ output(success({"claims": rows}, count=len(rows), vault=str(vault.root)), json_mode=True)
+ else:
+ for r in rows:
+ console.print(f" [cyan]{r['note_id']}[/] {r['claim'][:120]}")
+ console.print(f"[dim]{len(rows)} claims[/]")
+
+
+def claims_search(
+ query: str = typer.Argument(..., help="FTS query over claims + quoted support"),
+ limit: int = typer.Option(20, "--limit", "-n", help="Max results"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Full-text search over claims — 'which source supports X' as a query."""
+ from hyperresearch.core.claims import search_claims
+
+ vault = _vault_or_exit(json_output)
+ rows = search_claims(vault.db, query, limit=limit)
+
+ if json_output:
+ output(success({"claims": rows}, count=len(rows), vault=str(vault.root)), json_mode=True)
+ else:
+ if not rows:
+ console.print("[dim]No matching claims.[/]")
+ return
+ for r in rows:
+ console.print(f" [cyan]{r['note_id']}[/] {r['claim'][:120]}")
+ if r.get("quoted_support"):
+ console.print(f" [dim]\"{r['quoted_support'][:160]}\"[/]")
+
+
+def claims_matrix(
+ vault_tag: str | None = typer.Option(None, "--tag", "-t", help="Only claims with this vault_tag"),
+ out: str | None = typer.Option(None, "--out", "-o", help="Write the markdown table to this file"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Literature-review matrix: one row per claimed source (tier, quality, key finding).
+
+ The dissertation-scale appendix artifact — weeks of human table-building,
+ generated from the claims table.
+ """
+ from hyperresearch.core.claims import literature_matrix, render_matrix_markdown
+
+ vault = _vault_or_exit(json_output)
+ rows = literature_matrix(vault.db, vault_tag=vault_tag)
+
+ if out:
+ Path(out).parent.mkdir(parents=True, exist_ok=True)
+ Path(out).write_text(render_matrix_markdown(rows), encoding="utf-8")
+
+ if json_output:
+ output(success({"rows": rows, "written_to": out}, count=len(rows), vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(render_matrix_markdown(rows))
+ if out:
+ console.print(f"[green]Written:[/] {out}")
+
+
+def claims_targets(
+ vault_tag: str | None = typer.Option(None, "--tag", "-t", help="Only claims with this vault_tag"),
+ min_sources: int = typer.Option(2, "--min-sources", help="Only targets addressed by >= N distinct sources"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Group claims by stance_target across sources — meta-analysis substrate.
+
+ Surfaces where multiple sources address the same quantity/question, with
+ stance splits and source-attributed numbers for comparison tables.
+ """
+ from hyperresearch.core.claims import group_by_target
+
+ vault = _vault_or_exit(json_output)
+ groups = group_by_target(vault.db, vault_tag=vault_tag, min_sources=min_sources)
+
+ if json_output:
+ output(success({"targets": groups}, count=len(groups), vault=str(vault.root)), json_mode=True)
+ else:
+ for g in groups:
+ console.print(f" [cyan]{g['stance_target']}[/] {g['n_sources']} sources, {g['n_claims']} claims, stances={g['stances']}")
diff --git a/src/hyperresearch/cli/dedup.py b/src/hyperresearch/cli/dedup.py
index c97513b..9200f69 100644
--- a/src/hyperresearch/cli/dedup.py
+++ b/src/hyperresearch/cli/dedup.py
@@ -13,12 +13,9 @@ from hyperresearch.core.similarity import (
)
from hyperresearch.models.output import success
-# Use MinHash+LSH for vaults above this size, O(n^2) below
-LSH_THRESHOLD = 200
-
def dedup(
- threshold: float = typer.Option(0.6, "--threshold", "-t", help="Similarity threshold (0.0-1.0)"),
+ threshold: float | None = typer.Option(None, "--threshold", "-t", help="Similarity threshold (0.0-1.0); default from [dedup] config"),
limit: int = typer.Option(20, "--limit", "-l", help="Max pairs to show"),
json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
) -> None:
@@ -35,6 +32,9 @@ def dedup(
raise typer.Exit(1)
vault.auto_sync()
+ cfg = vault.config.dedup
+ if threshold is None:
+ threshold = cfg.default_threshold
# Load all note content
rows = vault.db.execute(
@@ -54,7 +54,7 @@ def dedup(
# Build shingle sets
notes = {}
for row in rows:
- shingles = shingle(row["body_plain"])
+ shingles = shingle(row["body_plain"], n=cfg.shingle_size)
notes[row["id"]] = {
"id": row["id"],
"title": row["title"],
@@ -63,8 +63,8 @@ def dedup(
}
# Choose algorithm based on vault size
- if len(notes) >= LSH_THRESHOLD:
- pairs = _dedup_lsh(notes, threshold)
+ if len(notes) >= cfg.lsh_switchover:
+ pairs = _dedup_lsh(notes, threshold, num_perm=cfg.minhash_perm, bands=cfg.lsh_bands)
method = "minhash+lsh"
else:
pairs = _dedup_brute(notes, threshold)
@@ -111,15 +111,15 @@ def _dedup_brute(notes: dict, threshold: float) -> list[dict]:
return pairs
-def _dedup_lsh(notes: dict, threshold: float) -> list[dict]:
+def _dedup_lsh(notes: dict, threshold: float, num_perm: int = 128, bands: int = 16) -> list[dict]:
"""MinHash+LSH for large vaults — approximate but O(n)."""
# Compute signatures
signatures = {}
for nid, note in notes.items():
- signatures[nid] = minhash_signature(note["shingles"])
+ signatures[nid] = minhash_signature(note["shingles"], num_perm=num_perm)
# Find candidate pairs via LSH
- candidates = lsh_candidates(signatures)
+ candidates = lsh_candidates(signatures, bands=bands)
# Verify candidates with exact Jaccard
pairs = []
diff --git a/src/hyperresearch/cli/embed_cmd.py b/src/hyperresearch/cli/embed_cmd.py
new file mode 100644
index 0000000..7011a3a
--- /dev/null
+++ b/src/hyperresearch/cli/embed_cmd.py
@@ -0,0 +1,83 @@
+"""Embedding commands — semantic-search vector management."""
+
+from __future__ import annotations
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def embed_sync_cmd(
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Embed new/changed notes with the configured provider.
+
+ Requires [embeddings] provider = "voyage" or "openai" in config.toml
+ (plus the provider's API key env var). No-op re-runs are cheap: only
+ notes whose content changed since their last embedding are re-sent.
+ """
+ from hyperresearch.core.embed import EmbeddingError, embed_sync
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ try:
+ result = embed_sync(vault)
+ except EmbeddingError as e:
+ if json_output:
+ output(error(str(e), "EMBEDDINGS_DISABLED"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ if json_output:
+ output(success(result, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(
+ f"[green]Embedded:[/] {result['embedded']} notes "
+ f"({result['skipped']} up to date) via {result['provider']}/{result['model']}"
+ )
+
+
+def embed_status(
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Show embedding coverage for the vault."""
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ conn = vault.db
+ total = conn.execute("SELECT COUNT(*) AS c FROM notes").fetchone()["c"]
+ embedded = conn.execute("SELECT COUNT(*) AS c FROM embeddings").fetchone()["c"]
+ data = {
+ "provider": vault.config.embeddings.provider,
+ "notes": total,
+ "embedded": embedded,
+ }
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(
+ f"provider: {data['provider']} embedded: {embedded}/{total} notes"
+ )
diff --git a/src/hyperresearch/cli/escalation_cmd.py b/src/hyperresearch/cli/escalation_cmd.py
new file mode 100644
index 0000000..66ba4bc
--- /dev/null
+++ b/src/hyperresearch/cli/escalation_cmd.py
@@ -0,0 +1,276 @@
+"""Escalation commands — the browser-lane work queue.
+
+Blocked fetches land here (login walls, bot blocks, captchas); the
+hyperresearch-browser-fetcher agent drains the queue via `claim` →
+(`ingest` | `human` | `abandon`). `ingest` is the one-shot completion:
+it writes the note, records the source row, syncs, and resolves the item —
+the agent supplies content, Python does the bookkeeping.
+"""
+
+from __future__ import annotations
+
+import hashlib
+from datetime import UTC, datetime
+from pathlib import Path
+from urllib.parse import urlparse
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def _vault_or_exit(json_output: bool):
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ return Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+
+def escalation_list(
+ status: str | None = typer.Option(None, "--status", "-s", help="queued|in_progress|fetched|needs_human|abandoned"),
+ tag: str | None = typer.Option(None, "--tag", "-t", help="Only items for this vault_tag"),
+ limit: int = typer.Option(100, "--limit", "-n", help="Max items"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """List escalation queue items (highest utility first)."""
+ from hyperresearch.core.escalation import list_items, queue_stats
+
+ vault = _vault_or_exit(json_output)
+ items = list_items(vault.db, status=status, vault_tag=tag, limit=limit)
+ stats = queue_stats(vault.db, vault_tag=tag)
+
+ if json_output:
+ output(success({"items": items, "stats": stats}, count=len(items), vault=str(vault.root)), json_mode=True)
+ else:
+ for it in items:
+ score = f"u{it['utility_score']:g}" if it["utility_score"] is not None else "u?"
+ console.print(f" #{it['id']} [{it['status']}] ({it['reason']}, {score}) {it['url']}")
+ console.print(f"[dim]{stats}[/]")
+
+
+def escalation_add(
+ url: str = typer.Argument(..., help="URL (or Scholar query for --reason scholar_search)"),
+ reason: str = typer.Option(..., "--reason", "-r", help="login_wall|bot_block|captcha|fetch_failed|interactive_needed|scholar_search"),
+ tag: str | None = typer.Option(None, "--tag", "-t", help="vault_tag"),
+ utility: float | None = typer.Option(None, "--utility", help="Utility score (drives drain priority)"),
+ suggested_by: str | None = typer.Option(None, "--suggested-by", help="Note id that surfaced this URL"),
+ detail: str | None = typer.Option(None, "--detail", help="One-line context"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Manually queue an item (e.g. an interactive-only page or a Scholar search)."""
+ from hyperresearch.core.escalation import EscalationError, enqueue
+
+ vault = _vault_or_exit(json_output)
+ try:
+ item_id = enqueue(
+ vault.db, url, reason, vault_tag=tag, requested_by="manual",
+ suggested_by=suggested_by, utility_score=utility, detail=detail,
+ )
+ except EscalationError as e:
+ if json_output:
+ output(error(str(e), "ESCALATION_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ data = {"id": item_id, "already_queued": item_id is None}
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f" queued #{item_id}" if item_id else " already queued")
+
+
+def escalation_claim(
+ claimed_by: str = typer.Option("browser-fetcher", "--by", help="Claimer id"),
+ tag: str | None = typer.Option(None, "--tag", "-t", help="Only claim items for this vault_tag"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Atomically claim the next queued item (empty result when queue is drained)."""
+ from hyperresearch.core.escalation import claim_next
+
+ vault = _vault_or_exit(json_output)
+ item = claim_next(vault.db, claimed_by, vault_tag=tag)
+
+ if json_output:
+ output(success({"item": item, "queue_empty": item is None}, vault=str(vault.root)), json_mode=True)
+ else:
+ if item is None:
+ console.print("[dim]Queue empty.[/]")
+ else:
+ console.print(f" claimed #{item['id']} ({item['reason']}) {item['url']}")
+
+
+def escalation_ingest(
+ item_id: int = typer.Argument(..., help="Claimed escalation item id"),
+ title: str = typer.Option(..., "--title", help="Page title"),
+ body_file: str = typer.Option(..., "--body-file", help="File containing the extracted page content (markdown/plain text)"),
+ tags: list[str] = typer.Option([], "--tag", "-t", help="Tags (repeatable; the item's vault_tag is added automatically)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Complete a claimed item: write the note + source row, sync, resolve.
+
+ One command so the browser agent can't half-finish the bookkeeping.
+ Notes carry `fetch_provider: chrome` — browser-lane provenance is visible.
+ """
+ from hyperresearch.core.escalation import EscalationError, list_items, resolve
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.scholar import extract_doi
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ vault = _vault_or_exit(json_output)
+ conn = vault.db
+
+ matching = [it for it in list_items(conn, limit=10000) if it["id"] == item_id]
+ if not matching:
+ if json_output:
+ output(error(f"no escalation item #{item_id}", "NOT_FOUND"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] no escalation item #{item_id}")
+ raise typer.Exit(1)
+ item = matching[0]
+
+ body = Path(body_file).read_text(encoding="utf-8-sig")
+ url = item["url"]
+ domain = urlparse(url).netloc
+ now = datetime.now(UTC)
+
+ all_tags = list(tags)
+ if item["vault_tag"] and item["vault_tag"] not in all_tags:
+ all_tags.insert(0, item["vault_tag"])
+
+ extra_meta: dict = {
+ "source": url,
+ "source_domain": domain,
+ "fetched_at": now.isoformat(),
+ "fetch_provider": "chrome",
+ }
+ detected_doi = extract_doi(url, content=body)
+ if detected_doi:
+ extra_meta["doi"] = detected_doi
+ if item["utility_score"] is not None:
+ extra_meta["utility_score"] = item["utility_score"]
+
+ if item["suggested_by"]:
+ body = f"*Suggested by [[{item['suggested_by']}]]*\n\n" + body
+
+ note_path = write_note(
+ vault.notes_dir, title=title, body=body, tags=all_tags,
+ status="draft", source=url, extra_frontmatter=extra_meta,
+ )
+ note_id = note_path.stem
+
+ plan = compute_sync_plan(vault)
+ if plan.to_add or plan.to_update:
+ execute_sync(vault, plan)
+
+ content_hash = hashlib.sha256(body.encode("utf-8")).hexdigest()[:16]
+ conn.execute(
+ """INSERT OR IGNORE INTO sources (url, note_id, domain, fetched_at, provider, content_hash)
+ VALUES (?, ?, ?, ?, ?, ?)""",
+ (url, note_id, domain, now.isoformat(), "chrome", content_hash),
+ )
+ conn.commit()
+
+ try:
+ resolved = resolve(conn, item_id, "fetched", note_id=note_id)
+ except EscalationError as e:
+ if json_output:
+ output(error(str(e), "ESCALATION_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ data = {
+ "note_id": note_id,
+ "path": str(note_path.relative_to(vault.root)),
+ "word_count": len(body.split()),
+ "item": resolved,
+ }
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]Ingested:[/] #{item_id} -> note '{note_id}'")
+
+
+def escalation_human(
+ item_id: int = typer.Argument(..., help="Item id"),
+ detail: str = typer.Option(..., "--detail", help="One line: what the human must do (e.g. 'solve CAPTCHA on nature.com')"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Mark an item as needing the human (CAPTCHA / 2FA / login). NEVER solve these automatically."""
+ from hyperresearch.core.escalation import EscalationError, resolve
+
+ vault = _vault_or_exit(json_output)
+ try:
+ item = resolve(vault.db, item_id, "needs_human", detail=detail)
+ except EscalationError as e:
+ if json_output:
+ output(error(str(e), "ESCALATION_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success(item, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f" #{item_id} needs human: {detail}")
+
+
+def escalation_retry(
+ item_id: int = typer.Argument(..., help="Item id"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Re-queue a needs_human item after the human completed the challenge."""
+ from hyperresearch.core.escalation import EscalationError, resolve
+
+ vault = _vault_or_exit(json_output)
+ try:
+ item = resolve(vault.db, item_id, "queued")
+ except EscalationError as e:
+ if json_output:
+ output(error(str(e), "ESCALATION_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success(item, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f" #{item_id} re-queued")
+
+
+def escalation_abandon(
+ item_id: int = typer.Argument(..., help="Item id"),
+ detail: str | None = typer.Option(None, "--detail", help="Why (optional)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Give up on an item. The floor is the pre-4.0 status quo: the source is lost."""
+ from hyperresearch.core.escalation import EscalationError, resolve
+
+ vault = _vault_or_exit(json_output)
+ try:
+ item = resolve(vault.db, item_id, "abandoned", detail=detail)
+ except EscalationError as e:
+ if json_output:
+ output(error(str(e), "ESCALATION_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success(item, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f" #{item_id} abandoned")
diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.py
index 7a81521..f675ebf 100644
--- a/src/hyperresearch/cli/fetch.py
+++ b/src/hyperresearch/cli/fetch.py
@@ -11,14 +11,11 @@ from urllib.parse import urlparse
import typer
from hyperresearch.cli._output import console, output
+from hyperresearch.core.config import AssetSettings, FetchSettings
from hyperresearch.models.output import error, success
app = typer.Typer()
-# Max images to download per fetch
-MAX_IMAGES = 5
-# Skip small images (icons, logos, spacers, tracking pixels, nav elements)
-MIN_IMAGE_BYTES = 50_000
# URL patterns that are almost never content images
SKIP_URL_PATTERNS = (
"logo", "icon", "favicon", "badge", "avatar", "sprite", "banner",
@@ -220,6 +217,11 @@ def fetch(
"--suggested-by-reason",
help="One-line justification for why the suggesting source named this URL. Appears next to the [[source-id]] breadcrumb.",
),
+ utility_score: float | None = typer.Option(
+ None,
+ "--utility-score",
+ help="Step-2 fetch-selection utility score (0-18). Persisted to note frontmatter so the quality composite can use it after fetching.",
+ ),
json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
) -> None:
"""Fetch a URL and save its content as a research note."""
@@ -290,11 +292,7 @@ def fetch(
from urllib.parse import urlparse as _urlparse
domain = _urlparse(url).netloc.lower()
- _auth_aggressive_domains = (
- "linkedin.com", "twitter.com", "x.com", "facebook.com",
- "instagram.com", "tiktok.com",
- )
- if any(d in domain for d in _auth_aggressive_domains):
+ if any(d in domain for d in vault.config.fetch.visible_browser_domains):
visible = True
# Fetch content
@@ -303,6 +301,8 @@ def fetch(
profile=vault.config.web_profile,
magic=vault.config.web_magic,
headless=not visible,
+ settings=vault.config.fetch,
+ gates=vault.config.junk,
)
if not json_output:
console.print(f"[dim]Fetching with {prov.name}...[/]")
@@ -316,25 +316,38 @@ def fetch(
console.print(f"[red]Fetch failed:[/] {e}")
raise typer.Exit(1)
- # Detect login redirects — abort instead of saving login page junk
- if result.looks_like_login_wall(url):
+ # Detect login redirects — abort, but ESCALATE to the browser lane
+ # instead of silently losing the source.
+ if result.looks_like_login_wall(url, vault.config.junk):
+ item_id = _escalate_blocked(vault, url, "login_wall", tags, suggested_by, utility_score,
+ detail=f"login wall: {result.title}")
+ escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""
msg = (
f"Redirected to login page ({result.title}). "
"Try --visible flag (runs browser non-headless, sites are less aggressive). "
- "If that fails, re-create your login profile with 'hyperresearch setup'."
+ f"If that fails, re-create your login profile with 'hyperresearch setup'.{escalated}"
)
if json_output:
- output(error(msg, "AUTH_REQUIRED"), json_mode=True)
+ output(error(msg, "AUTH_REQUIRED_ESCALATED" if item_id else "AUTH_REQUIRED"), json_mode=True)
else:
console.print(f"[red]Auth required:[/] {msg}")
raise typer.Exit(1)
- # Detect junk pages — captcha, error pages, binary garbage, empty content
- junk_reason = result.looks_like_junk()
+ # Detect junk pages — captcha, error pages, binary garbage, empty content.
+ # Bot-detection junk (Cloudflare/captcha walls) escalates to the browser
+ # lane; content-quality junk (error pages, empty content) does not — a
+ # 404 in Chrome is still a 404.
+ junk_reason = result.looks_like_junk(vault.config.junk)
if junk_reason:
- msg = f"Skipped junk content from {url}: {junk_reason}"
+ item_id = None
+ if junk_reason.startswith("Bot detection"):
+ reason = "captcha" if "captcha" in junk_reason.lower() else "bot_block"
+ item_id = _escalate_blocked(vault, url, reason, tags, suggested_by, utility_score,
+ detail=junk_reason)
+ escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""
+ msg = f"Skipped junk content from {url}: {junk_reason}.{escalated}"
if json_output:
- output(error(msg, "JUNK_CONTENT"), json_mode=True)
+ output(error(msg, "JUNK_ESCALATED" if item_id else "JUNK_CONTENT"), json_mode=True)
else:
console.print(f"[yellow]Skipped:[/] {msg}")
raise typer.Exit(1)
@@ -354,6 +367,15 @@ def fetch(
if result.metadata.get("author"):
extra_meta["author"] = result.metadata["author"]
+ # Source-ranking capture: DOI/arXiv id + the orchestrator's utility score
+ from hyperresearch.core.scholar import extract_doi
+
+ detected_doi = extract_doi(url, result.raw_html, result.content)
+ if detected_doi:
+ extra_meta["doi"] = detected_doi
+ if utility_score is not None:
+ extra_meta["utility_score"] = utility_score
+
# Build body with backlink breadcrumb if this fetch was suggested by another note.
# Prepending `*Suggested by [[source-id]] — reason*` lines creates wiki-links the
# normal link extractor will pick up, so the new note automatically appears in
@@ -480,7 +502,10 @@ def fetch(
saved_assets: list[dict] = []
if save_assets:
assets_dir = vault.root / "research" / "assets" / note_id
- saved_assets = _save_assets(conn, result, note_id, assets_dir)
+ saved_assets = _save_assets(
+ conn, result, note_id, assets_dir,
+ settings=vault.config.assets, image_timeout_s=vault.config.fetch.image_timeout_s,
+ )
data = {
"note_id": note_id,
@@ -505,8 +530,38 @@ def fetch(
console.print(f" Assets: {len(saved_assets)} saved to research/assets/{note_id}/")
-def _save_assets(conn, result, note_id: str, assets_dir: Path) -> list[dict]:
+def _escalate_blocked(
+ vault, url: str, reason: str, tags: list[str],
+ suggested_by: list[str], utility_score: float | None,
+ detail: str | None = None,
+) -> int | None:
+ """Queue a blocked fetch for the Chrome lane. Never raises — a failed
+ enqueue must not mask the original fetch outcome."""
+ try:
+ from hyperresearch.core.escalation import maybe_enqueue_blocked_fetch
+
+ return maybe_enqueue_blocked_fetch(
+ vault, url, reason,
+ vault_tag=tags[0] if tags else None,
+ suggested_by=suggested_by[0] if suggested_by else None,
+ utility_score=utility_score,
+ detail=detail,
+ )
+ except Exception:
+ return None
+
+
+def _save_assets(
+ conn,
+ result,
+ note_id: str,
+ assets_dir: Path,
+ settings: AssetSettings | None = None,
+ image_timeout_s: int | None = None,
+) -> list[dict]:
"""Save screenshot and images to assets dir, record in DB. Returns list of saved asset info."""
+ settings = settings or AssetSettings()
+ timeout_s = image_timeout_s if image_timeout_s is not None else FetchSettings().image_timeout_s
saved: list[dict] = []
now = datetime.now(UTC).isoformat()
@@ -544,10 +599,13 @@ def _save_assets(conn, result, note_id: str, assets_dir: Path) -> list[dict]:
# Sort by score descending, take top N
candidates.sort(key=lambda m: m.get("score", 0), reverse=True)
- for img in candidates[:MAX_IMAGES]:
+ for img in candidates[: settings.max_images]:
img_url = img.get("src", "")
alt = img.get("alt", "") or ""
- asset_info = _download_image(conn, note_id, img_url, alt, assets_dir, now)
+ asset_info = _download_image(
+ conn, note_id, img_url, alt, assets_dir, now,
+ min_image_bytes=settings.min_image_bytes, timeout_s=timeout_s,
+ )
if asset_info:
saved.append(asset_info)
@@ -557,7 +615,10 @@ def _save_assets(conn, result, note_id: str, assets_dir: Path) -> list[dict]:
return saved
-def _download_image(conn, note_id: str, img_url: str, alt: str, assets_dir: Path, now: str) -> dict | None:
+def _download_image(
+ conn, note_id: str, img_url: str, alt: str, assets_dir: Path, now: str,
+ min_image_bytes: int = 50_000, timeout_s: int = 15,
+) -> dict | None:
"""Download a single image. Returns asset info dict or None if skipped."""
import urllib.request
@@ -584,14 +645,14 @@ def _download_image(conn, note_id: str, img_url: str, alt: str, assets_dir: Path
try:
req = urllib.request.Request(img_url, headers={"User-Agent": "hyperresearch/0.1"})
- with urllib.request.urlopen(req, timeout=15) as resp:
+ with urllib.request.urlopen(req, timeout=timeout_s) as resp:
data = resp.read()
content_type = resp.headers.get("Content-Type", "")
except Exception:
return None
# Skip tiny images (icons, spacers, tracking pixels)
- if len(data) < MIN_IMAGE_BYTES:
+ if len(data) < min_image_bytes:
return None
file_path.write_bytes(data)
diff --git a/src/hyperresearch/cli/fetch_batch.py b/src/hyperresearch/cli/fetch_batch.py
index 3f64262..6c3db65 100644
--- a/src/hyperresearch/cli/fetch_batch.py
+++ b/src/hyperresearch/cli/fetch_batch.py
@@ -55,21 +55,18 @@ def fetch_batch(
vault.auto_sync()
conn = vault.db
- _auth_aggressive = (
- "linkedin.com", "twitter.com", "x.com", "facebook.com",
- "instagram.com", "tiktok.com",
- )
-
def _needs_visible(fetch_url: str) -> bool:
if not vault.config.web_profile:
return False
domain = urlparse(fetch_url).netloc.lower()
- return any(d in domain for d in _auth_aggressive)
+ return any(d in domain for d in vault.config.fetch.visible_browser_domains)
prov = get_provider(
provider_name or vault.config.web_provider,
profile=vault.config.web_profile,
magic=vault.config.web_magic,
+ settings=vault.config.fetch,
+ gates=vault.config.junk,
)
# Filter out already-fetched URLs
@@ -121,6 +118,8 @@ def fetch_batch(
profile=vault.config.web_profile,
magic=vault.config.web_magic,
headless=False,
+ settings=vault.config.fetch,
+ gates=vault.config.junk,
)
for url in visible_urls:
try:
@@ -135,7 +134,7 @@ def fetch_batch(
url = result.url
# Skip login redirects
- if result.looks_like_login_wall(url):
+ if result.looks_like_login_wall(url, vault.config.junk):
if not json_output:
console.print(
f" [yellow]Auth required:[/] {url} — login page detected, skipping. "
@@ -190,10 +189,14 @@ def fetch_batch(
# Save assets if requested
if save_assets:
- from hyperresearch.cli.fetch import _save_assets
+ from hyperresearch.cli.fetch import _save_assets as _save_assets_fn
assets_dir = vault.root / "research" / "assets" / note_id
- _save_assets(conn, result, note_id, assets_dir)
+ _save_assets_fn(
+ conn, result, note_id, assets_dir,
+ settings=vault.config.assets,
+ image_timeout_s=vault.config.fetch.image_timeout_s,
+ )
created_notes.append({
"note_id": note_id,
diff --git a/src/hyperresearch/cli/graph.py b/src/hyperresearch/cli/graph.py
index 67f7a65..0d9deff 100644
--- a/src/hyperresearch/cli/graph.py
+++ b/src/hyperresearch/cli/graph.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import typer
from hyperresearch.cli._output import console, output
-from hyperresearch.models.output import success
+from hyperresearch.models.output import error, success
app = typer.Typer()
@@ -286,3 +286,54 @@ def graph_hubs(
for h in hubs:
table.add_row(h["id"], h["title"], str(h["inbound_links"]))
console.print(table)
+
+
+def graph_rank(
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+ top: int = typer.Option(10, "--top", "-n", help="Show top-N central notes"),
+) -> None:
+ """Compute vault centrality (PageRank over the link graph).
+
+ Stores normalized scores to notes.centrality_score and recomputes the
+ composite quality_score. Centrality here means "many independent research
+ chains converged on this source".
+ """
+ from hyperresearch.core.graphrank import compute_centrality
+ from hyperresearch.core.quality import compute_quality_scores
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ conn = vault.db
+ ranked = compute_centrality(conn)
+ compute_quality_scores(conn, vault.config.ranking)
+
+ rows = conn.execute(
+ "SELECT id, title, centrality_score FROM notes "
+ "WHERE centrality_score IS NOT NULL "
+ "ORDER BY centrality_score DESC LIMIT ?",
+ (top,),
+ ).fetchall()
+ top_notes = [
+ {"id": r["id"], "title": r["title"], "centrality": round(r["centrality_score"], 4)}
+ for r in rows
+ ]
+
+ if json_output:
+ output(
+ success({"ranked": ranked, "top": top_notes}, count=ranked, vault=str(vault.root)),
+ json_mode=True,
+ )
+ else:
+ console.print(f"[green]Centrality computed:[/] {ranked} notes")
+ for n in top_notes:
+ console.print(f" {n['centrality']:.4f} [cyan]{n['id']}[/] {n['title']}")
diff --git a/src/hyperresearch/cli/install.py b/src/hyperresearch/cli/install.py
index 72cd4ad..d15387b 100644
--- a/src/hyperresearch/cli/install.py
+++ b/src/hyperresearch/cli/install.py
@@ -25,23 +25,59 @@ def install(
"--steps-only",
help="Install only the 16 step skills to <PATH>/.claude/skills/. Used internally by the entry skill bootstrap on first /hyperresearch invocation in a project. Not normally invoked by users.",
),
+ profile: str | None = typer.Option(
+ None,
+ "--profile",
+ help="Pipeline profile to render skill/agent prompts from (built-in gears: full, premier; plus any [profile.*] defined in .hyperresearch/config.toml). Defaults to the gear persisted by `hyperresearch profile use` (or 'full'). See `hyperresearch profile list`.",
+ ),
) -> None:
"""Install hyperresearch: init vault + inject CLAUDE.md + install Claude Code hooks."""
import sys
from hyperresearch.core.hooks import (
_install_hyperresearch_step_skills,
+ _set_render_state,
install_global_hooks,
install_hooks,
)
+ from hyperresearch.core.profiles import ProfileError
from hyperresearch.core.vault import Vault, VaultError
+ # No explicit --profile → use the gear persisted by `hpr profile use`
+ # in the target's config (falling back to "full").
+ def _default_profile(config_path: Path | None) -> str:
+ if profile is not None:
+ return profile
+ if config_path is not None and config_path.exists():
+ from hyperresearch.core.config import VaultConfig
+
+ return VaultConfig.load(config_path).pipeline_profile
+ return "full"
+
+ # Validate the profile early so a typo fails before any files are written.
+ def _check_profile(resolved: str, config_path: Path | None) -> None:
+ from hyperresearch.core.profiles import resolve_profile
+
+ try:
+ resolve_profile(resolved, config_path)
+ except ProfileError as e:
+ if json_output:
+ output(error(str(e), "UNKNOWN_PROFILE"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
# Steps-only path: lazy install of the 16 step skills to a project's
# .claude/skills/. Called by the entry skill's bootstrap on first
# /hyperresearch in a project (after a global install). Cheap no-op
# on subsequent invocations.
if steps_only:
target = Path(path).resolve()
+ steps_config = target / ".hyperresearch" / "config.toml"
+ steps_config_path = steps_config if steps_config.exists() else None
+ steps_profile = _default_profile(steps_config_path)
+ _check_profile(steps_profile, steps_config_path)
+ _set_render_state(steps_profile, steps_config_path)
result = _install_hyperresearch_step_skills(target)
if json_output:
output(
@@ -66,7 +102,9 @@ def install(
hpr_path = _resolve_executable()
home = Path.home()
- hook_actions = install_global_hooks(home, hpr_path=hpr_path)
+ global_profile = profile if profile is not None else "full"
+ _check_profile(global_profile, None)
+ hook_actions = install_global_hooks(home, hpr_path=hpr_path, profile=global_profile)
if json_output:
output(
@@ -127,8 +165,13 @@ def install(
# Step 3: Always re-inject CLAUDE.md (updates blurb + path)
doc_actions = inject_agent_docs(root)
- # Step 4: Install Claude Code hook + skills + subagents
- hook_actions = install_hooks(root, hpr_path=hpr_path)
+ # Step 4: Install Claude Code hook + skills + subagents (rendered from the
+ # gear profile — explicit --profile, else the gear persisted in config)
+ project_config = root / ".hyperresearch" / "config.toml"
+ project_config_path = project_config if project_config.exists() else None
+ project_profile = _default_profile(project_config_path)
+ _check_profile(project_profile, project_config_path)
+ hook_actions = install_hooks(root, hpr_path=hpr_path, profile=project_profile)
# Step 3: Auto-configure crawl4ai if installed
crawl4ai_status = _setup_crawl4ai(vault)
diff --git a/src/hyperresearch/cli/lint.py b/src/hyperresearch/cli/lint.py
index 0087ef9..35db65c 100644
--- a/src/hyperresearch/cli/lint.py
+++ b/src/hyperresearch/cli/lint.py
@@ -3,6 +3,7 @@
from __future__ import annotations
import json
+import re
from datetime import UTC
import typer
@@ -28,6 +29,9 @@ RULES = {
"instruction-coverage": "Atomic items from prompt-decomposition missing from the final report (draft drifted from user's ask)",
"citation-style-preservation": "Final report carries no citations matching the declared citation_style (wikilink markers or numbered references stripped)",
"extract-coverage": "Light-tier runs with fetched sources but no paired extract notes (reading loop skipped or source-analyst not delegated for long sources)",
+ "quote-integrity": "Quoted spans in the final report that appear verbatim in no vault note (hallucinated quotes)",
+ "numeric-consistency": "Numbers in the final report untraceable to claims or cited note bodies",
+ "retracted-citations": "Final report cites a retracted source without marking it as retracted",
"orphaned-raw-files": "Files in research/raw/ with no matching note (disk leak from old note rm)",
"singleton-tags": "Tags used by only one note",
"broken-links": "Wiki-links that don't resolve",
@@ -40,6 +44,194 @@ RULES = {
}
+def _run_dirs_newest_first(vault) -> list:
+ """Run workspaces under research/runs/, newest manifest first."""
+ runs_dir = vault.root / "research" / "runs"
+ if not runs_dir.is_dir():
+ return []
+ dirs = [d for d in runs_dir.iterdir() if d.is_dir() and (d / "run.json").exists()]
+ dirs.sort(key=lambda d: (d / "run.json").stat().st_mtime, reverse=True)
+ return dirs
+
+
+def _run_artifact(vault, *relparts: str):
+ """Resolve a run-scoped pipeline artifact.
+
+ 3.0 layout: research/runs/<vault_tag>/<artifact> (newest run that has it
+ wins). Legacy pre-3.0 layout: research/<artifact>. Always returns a Path
+ (the legacy flat path when nothing exists anywhere) so `.exists()` checks
+ at call sites keep working unchanged.
+ """
+ for d in _run_dirs_newest_first(vault):
+ p = d.joinpath(*relparts)
+ if p.exists():
+ return p
+ return (vault.root / "research").joinpath(*relparts)
+
+
+def _query_files(vault) -> list:
+ """Canonical query files: runs/<tag>/query.md (3.0) + legacy query-*.md."""
+ files = []
+ for d in _run_dirs_newest_first(vault):
+ q = d / "query.md"
+ if q.exists():
+ files.append(q)
+ files.extend(sorted((vault.root / "research").glob("query-*.md")))
+ return files
+
+
+def _latest_report(vault):
+ """Newest final_report*.md, or (None, None) when no report exists."""
+ notes_dir = vault.root / "research" / "notes"
+ if not notes_dir.is_dir():
+ return None, None
+ candidates = sorted(notes_dir.glob("final_report*.md"), key=lambda p: p.stat().st_mtime)
+ if not candidates:
+ return None, None
+ p = candidates[-1]
+ try:
+ return p, p.read_text(encoding="utf-8-sig")
+ except OSError:
+ return None, None
+
+
+_QUOTE_SPAN_RE = re.compile(r"[\"“]([^\"“”]{20,600})[\"”]")
+_REPORT_NUMBER_RE = re.compile(r"\d[\d,]*\.\d+%?|\d[\d,]{3,}%?|\d[\d,]*%")
+
+
+def _report_body_only(report_text: str) -> str:
+ """Report minus the Sources/References section and citation markers."""
+ import re as _re
+
+ body = _re.split(r"^##\s+(?:Sources|References)\b", report_text, maxsplit=1, flags=_re.M | _re.I)[0]
+ return _re.sub(r"\[\d{1,3}\]", "", body)
+
+
+def _check_quote_integrity(vault, conn, report_path, report_text) -> list[dict]:
+ """Every quoted span >= 5 words must exist in some vault note body.
+
+ Uses an FTS phrase query (porter-stemmed on both sides, so inflection
+ noise doesn't false-positive) and reports the nearest fuzzy match for
+ fast fixing. Hallucinated quotes are error severity — they are the one
+ thing a research report can never ship.
+ """
+ import re as _re
+
+ issues: list[dict] = []
+ body = _report_body_only(report_text)
+ for m in _QUOTE_SPAN_RE.finditer(body):
+ quote = _re.sub(r"\s+", " ", m.group(1)).strip()
+ if len(quote.split()) < 5:
+ continue
+ phrase = quote.replace('"', " ").replace("'", "''")
+ try:
+ hit = conn.execute(
+ 'SELECT id FROM notes_fts WHERE notes_fts MATCH ? LIMIT 1',
+ (f'body_plain: "{phrase}"',),
+ ).fetchone()
+ except Exception:
+ hit = None
+ if hit:
+ continue
+ issues.append({
+ "rule": "quote-integrity",
+ "severity": "error",
+ "note_id": "<report>",
+ "note_path": str(report_path),
+ "message": (
+ f"Quoted span not found verbatim in any vault note: \"{quote[:120]}\"... "
+ "Either the quote is hallucinated/mangled or its source was never fetched. "
+ "Fix the quote or drop the quotation marks."
+ ),
+ })
+ return issues
+
+
+def _check_numeric_consistency(vault, conn, report_path, report_text) -> list[dict]:
+ """Substantive numbers in the report should be traceable to claims or
+ cited note bodies. Warning severity — legitimate derived arithmetic
+ exists; this flags candidates for the orchestrator to verify, it does
+ not block the gate."""
+ body = _report_body_only(report_text)
+ report_numbers = {n.rstrip(".,") for n in _REPORT_NUMBER_RE.findall(body)}
+ if not report_numbers:
+ return []
+
+ # Evidence blob: every claim (text, quotes, numbers) + bodies of notes
+ # with sources (the citable population).
+ parts: list[str] = []
+ for row in conn.execute("SELECT claim, quoted_support, numbers FROM claims"):
+ parts.extend(filter(None, (row["claim"], row["quoted_support"], row["numbers"])))
+ for row in conn.execute(
+ "SELECT nc.body_plain FROM note_content nc JOIN notes n ON n.id = nc.note_id "
+ "WHERE n.source IS NOT NULL"
+ ):
+ parts.append(row["body_plain"])
+ blob = " ".join(parts).replace(",", "")
+
+ issues: list[dict] = []
+ untraced = sorted(
+ n for n in report_numbers
+ if n.replace(",", "") not in blob
+ )
+ for number in untraced[:20]:
+ issues.append({
+ "rule": "numeric-consistency",
+ "severity": "warning",
+ "note_id": "<report>",
+ "note_path": str(report_path),
+ "message": (
+ f"Number '{number}' in the final report appears in no claim or cited "
+ "note body. If it is derived arithmetic, fine; otherwise verify it "
+ "against a source or remove it."
+ ),
+ })
+ if len(untraced) > 20:
+ issues.append({
+ "rule": "numeric-consistency",
+ "severity": "warning",
+ "note_id": "<report>",
+ "note_path": str(report_path),
+ "message": f"...and {len(untraced) - 20} more untraceable numbers (showing first 20).",
+ })
+ return issues
+
+
+def _check_retracted_citations(vault, conn, report_path, report_text) -> list[dict]:
+ """A cited retracted source blocks the gate — unless the citation itself
+ acknowledges the retraction (sometimes the retraction IS the story)."""
+
+ from hyperresearch.core.patterns import WIKI_LINK_RE
+
+ retracted = {
+ row["id"]: row["title"]
+ for row in conn.execute("SELECT id, title FROM notes WHERE is_retracted = 1")
+ }
+ if not retracted:
+ return []
+
+ issues: list[dict] = []
+ for m in WIKI_LINK_RE.finditer(report_text):
+ target = m.group(1).strip()
+ if target not in retracted:
+ continue
+ window = report_text[max(0, m.start() - 200): m.end() + 200].lower()
+ if "retract" in window:
+ continue
+ issues.append({
+ "rule": "retracted-citations",
+ "severity": "error",
+ "note_id": target,
+ "note_path": str(report_path),
+ "message": (
+ f"Final report cites [[{target}]] ('{retracted[target]}') which is "
+ "RETRACTED, without acknowledging the retraction. Drop the citation "
+ "or explicitly mark it (e.g. '(retracted)') where cited."
+ ),
+ })
+ return issues
+
+
@app.callback(invoke_without_command=True)
def lint(
ctx: typer.Context,
@@ -133,7 +325,7 @@ def lint(
if audit_file:
audit_path = vault.root / audit_file
else:
- audit_path = vault.root / "research" / "audit_findings.json"
+ audit_path = _run_artifact(vault, "audit_findings.json")
if audit_path.exists():
try:
audit_data = _json.loads(audit_path.read_text(encoding="utf-8"))
@@ -396,11 +588,13 @@ def lint(
)
prompt_path = vault.root / "research" / "prompt.txt"
canonical_prompt: str | None = None
- # Check for new-style query files first, fall back to legacy prompt.txt
- query_dir = vault.root / "research"
- query_files = sorted(query_dir.glob("query-*.md"))
+ # Check for query files first (runs/<tag>/query.md, then legacy
+ # research/query-*.md), fall back to legacy prompt.txt
+ query_files = _query_files(vault)
if query_files:
- # Use the most recent query file (last alphabetically)
+ # _query_files orders newest-run first; legacy globs come after.
+ # Use the first (newest) for 3.0 runs, last for pure-legacy vaults.
+ query_files = [query_files[0]] if query_files[0].name == "query.md" else query_files
try:
raw = query_files[-1].read_text(encoding="utf-8-sig").replace("\r\n", "\n")
# Strip YAML frontmatter if present
@@ -525,7 +719,7 @@ def lint(
# but still enforce scaffold-leak hygiene (always wrong regardless
# of wrapper).
prompt_path = vault.root / "research" / "prompt.txt"
- query_files_exist = bool(sorted((vault.root / "research").glob("query-*.md")))
+ query_files_exist = bool(_query_files(vault))
contract_path = vault.root / "research" / "wrapper_contract.json"
wrapper_contract: dict | None = None
if contract_path.exists():
@@ -759,7 +953,7 @@ def lint(
# breadcrumbs, reachability from seeds) still apply — they catch
# fabricated provenance. The coverage ratio does not. Skip the
# ratio checks when the vault is a hyperresearch run (loci.json exists).
- is_hyperresearch_run = (vault.root / "research" / "loci.json").exists()
+ is_hyperresearch_run = _run_artifact(vault, "loci.json").exists()
non_seed_ratio = len(non_seeds) / max(len(source_rows), 1)
if is_hyperresearch_run:
@@ -813,7 +1007,7 @@ def lint(
# the orchestrator commits to). Layer 3 must produce one interim note
# per locus, tagged `locus-<locus-name>` with `type: interim`. This
# rule catches depth investigators that failed or were skipped.
- loci_path = vault.root / "research" / "loci.json"
+ loci_path = _run_artifact(vault, "loci.json")
if loci_path.exists():
try:
loci_data = json.loads(loci_path.read_text(encoding="utf-8"))
@@ -912,9 +1106,9 @@ def lint(
# Hyperresearch runs use a different artifact shape (interim notes per
# locus, checked by `locus-coverage`). Skip this rule when
# `research/loci.json` exists — that signals a hyperresearch run.
- is_hyperresearch_run = (vault.root / "research" / "loci.json").exists()
+ is_hyperresearch_run = _run_artifact(vault, "loci.json").exists()
if not is_hyperresearch_run:
- extract_min_words = 150
+ extract_min_words = vault.config.lint.extract_min_words
source_count_row = conn.execute(
"SELECT COUNT(*) as c FROM notes n "
@@ -972,8 +1166,9 @@ def lint(
# session needs at least 1 extract — the analyst is mandatory
# on single-pass runs, not optional.
if source_count >= 1:
- required_extracts = max(1, source_count // 3)
- error_floor = max(1, source_count // 4)
+ divisor = vault.config.lint.extract_coverage_divisor
+ required_extracts = max(1, source_count // divisor)
+ error_floor = max(1, source_count // (divisor + 1))
if extract_count < required_extracts:
ratio = extract_count / source_count if source_count else 0
notes_parts = []
@@ -1008,7 +1203,7 @@ def lint(
# skipped, and conflicted findings. Skipped `critical` findings are
# blockers — the draft shipped with a critical critique unresolved.
# This rule surfaces that.
- patch_log_path = vault.root / "research" / "patch-log.json"
+ patch_log_path = _run_artifact(vault, "patch-log.json")
if patch_log_path.exists():
try:
log = json.loads(patch_log_path.read_text(encoding="utf-8"))
@@ -1064,7 +1259,7 @@ def lint(
# Only flag when there WERE findings to apply.
critic_totals = 0
for name in ("dialectic", "depth", "width"):
- cf = vault.root / "research" / f"critic-findings-{name}.json"
+ cf = _run_artifact(vault, f"critic-findings-{name}.json")
if cf.exists():
try:
cdata = json.loads(cf.read_text(encoding="utf-8"))
@@ -1112,7 +1307,7 @@ def lint(
# The instruction-critic agent (Layer 5) does the deeper structural
# audit. This lint rule is the final post-patch gate that catches
# items the critic flagged but the patcher couldn't apply.
- decomp_path = vault.root / "research" / "prompt-decomposition.json"
+ decomp_path = _run_artifact(vault, "prompt-decomposition.json")
notes_dir = vault.root / "research" / "notes"
report_candidates = sorted(
notes_dir.glob("final_report*.md"),
@@ -1208,7 +1403,7 @@ def lint(
# survive in the final report. Deliberately NOT a density floor —
# short reports and quote-heavy sections make density checks
# false-positive; a separate warning-level rule can add that later.
- decomp_path = vault.root / "research" / "prompt-decomposition.json"
+ decomp_path = _run_artifact(vault, "prompt-decomposition.json")
citation_style: str | None = None
if decomp_path.exists():
try:
@@ -1366,8 +1561,8 @@ def lint(
""").fetchone()["c"]
scaffold_count = _count_by_tag("scaffold")
- scaffold_md_exists = (vault.root / "research" / "scaffold.md").exists()
- loci_json_exists = (vault.root / "research" / "loci.json").exists()
+ scaffold_md_exists = _run_artifact(vault, "scaffold.md").exists()
+ loci_json_exists = _run_artifact(vault, "loci.json").exists()
interim_count_row = conn.execute(
"SELECT COUNT(*) as c FROM notes n WHERE n.type = 'interim'"
).fetchone()
@@ -1412,7 +1607,7 @@ def lint(
if loci_json_exists:
try:
loci_data = json.loads(
- (vault.root / "research" / "loci.json").read_text(encoding="utf-8")
+ _run_artifact(vault, "loci.json").read_text(encoding="utf-8")
)
loci_list = (
loci_data.get("loci", [])
@@ -1424,7 +1619,7 @@ def lint(
locus_count = 0
comparisons_exists = (
- vault.root / "research" / "comparisons.md"
+ _run_artifact(vault, "comparisons.md")
).exists()
if locus_count >= 1 and not comparisons_exists:
issues.append({
@@ -1553,7 +1748,7 @@ def lint(
if "stale-reviews" in rules_to_run:
from datetime import datetime, timedelta
- cutoff = (datetime.now(UTC) - timedelta(days=90)).isoformat()
+ cutoff = (datetime.now(UTC) - timedelta(days=vault.config.lint.stale_review_days)).isoformat()
for row in conn.execute(
"SELECT id, path, reviewed FROM notes "
"WHERE reviewed IS NOT NULL AND reviewed < ? AND status = 'evergreen'",
@@ -1567,6 +1762,19 @@ def lint(
"message": f"Last reviewed {row['reviewed'][:10]}. Consider re-reviewing.",
})
+ # --- Phase-5 verification rules: report content vs. vault evidence ---
+
+ _verification_rules = {"quote-integrity", "numeric-consistency", "retracted-citations"}
+ if _verification_rules & set(rules_to_run):
+ report_path, report_text = _latest_report(vault)
+ if report_text is not None:
+ if "quote-integrity" in rules_to_run:
+ issues.extend(_check_quote_integrity(vault, conn, report_path, report_text))
+ if "numeric-consistency" in rules_to_run:
+ issues.extend(_check_numeric_consistency(vault, conn, report_path, report_text))
+ if "retracted-citations" in rules_to_run:
+ issues.extend(_check_retracted_citations(vault, conn, report_path, report_text))
+
# Audit-gate self-certification post-check. For each CRITICAL finding
# that was marked `fixed_at`, we queued its implied lint rule for
# re-running via rules_to_run. Now that the full lint pass is done,
diff --git a/src/hyperresearch/cli/profile_cmd.py b/src/hyperresearch/cli/profile_cmd.py
new file mode 100644
index 0000000..babfaae
--- /dev/null
+++ b/src/hyperresearch/cli/profile_cmd.py
@@ -0,0 +1,239 @@
+"""Profile commands — inspect, validate, and switch pipeline profiles.
+
+Two distinct concepts, one command group:
+
+- **Tiers** are chosen per query at run time: `light` (auto-classified for
+ bounded queries) and `dissertation` (opt-in chaptered mega-runs). The
+ pipeline routes tiers itself.
+- **The gear** is the scale profile whose numbers are rendered into the
+ installed skill/agent prompts: `full` (standard) or `premier` (max
+ width/depth), plus any user-defined `[profile.*]` overlay. Switch gears
+ with `hpr profile use <name>` — it re-renders every installed prompt and
+ persists the choice in `.hyperresearch/config.toml`.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def _discover_vault():
+ """The enclosing vault, or None when outside one."""
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ return Vault.discover()
+ except VaultError:
+ return None
+
+
+def _config_path() -> Path | None:
+ """Config path of the enclosing vault, or None when outside a vault.
+
+ Profiles resolve fine without a vault — you just get the built-ins.
+ """
+ vault = _discover_vault()
+ return vault.config_path if vault else None
+
+
+def profile_list(
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """List available pipeline profiles (built-in + user-defined)."""
+ from hyperresearch.core.profiles import (
+ BUILTIN_PROFILES,
+ GEAR_PROFILES,
+ ProfileError,
+ list_profiles,
+ resolve_profile,
+ )
+
+ vault = _discover_vault()
+ cfg = vault.config_path if vault else None
+ current_gear = vault.config.pipeline_profile if vault else "full"
+
+ names = list_profiles(cfg)
+ rows = []
+ for n in names:
+ row: dict = {
+ "name": n,
+ "builtin": n in BUILTIN_PROFILES,
+ "kind": "gear" if (n in GEAR_PROFILES or n not in BUILTIN_PROFILES) else "tier",
+ "current_gear": n == current_gear,
+ }
+ try:
+ p = resolve_profile(n, cfg)
+ row.update(
+ description=p.description,
+ sources=list(p.source_target),
+ time_estimate=p.time_estimate,
+ )
+ except ProfileError as e:
+ row.update(description=f"INVALID: {e}", sources=None, time_estimate="")
+ rows.append(row)
+
+ if json_output:
+ output(
+ success({"profiles": rows, "current_gear": current_gear}, count=len(rows)),
+ json_mode=True,
+ )
+ return
+
+ for r in rows:
+ mark = " [green]*current gear[/]" if r["current_gear"] else ""
+ kind = r["kind"] if r["builtin"] else "user-defined gear"
+ console.print(f" [cyan]{r['name']}[/] [dim]({kind})[/]{mark}")
+ if r["description"]:
+ console.print(f" {r['description']}")
+ if r["sources"]:
+ console.print(
+ f" [dim]sources {r['sources'][0]}-{r['sources'][1]}"
+ f" {r['time_estimate']}[/]"
+ )
+ console.print(
+ "\n[dim]Tiers route per query (light auto, dissertation opt-in). "
+ "Switch the scale gear with: hyperresearch profile use <full|premier>[/]"
+ )
+
+
+def profile_use(
+ name: str = typer.Argument(..., help="Gear profile to switch to (full, premier, or a [profile.*] overlay)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Switch the pipeline scale gear: re-render installed skills/agents from this profile and persist it as the project default."""
+ from hyperresearch.core.profiles import ProfileError, resolve_profile
+
+ vault = _discover_vault()
+ if vault is None:
+ msg = "no vault here — run `hyperresearch install` first"
+ if json_output:
+ output(error(msg, "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {msg}")
+ raise typer.Exit(1)
+
+ if name in ("light", "dissertation"):
+ msg = (
+ f"'{name}' is a run-time tier, not a scale gear. "
+ "light is auto-selected for bounded queries; dissertation runs are requested "
+ "per query (or via `hyperresearch run init <tag> --profile dissertation`). "
+ "Gears control the flat pipeline's scale: full, premier, or a custom [profile.*] overlay."
+ )
+ if json_output:
+ output(error(msg, "TIER_NOT_GEAR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {msg}")
+ raise typer.Exit(1)
+
+ try:
+ profile = resolve_profile(name, vault.config_path)
+ except ProfileError as e:
+ if json_output:
+ output(error(str(e), "UNKNOWN_PROFILE"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ # Persist the gear, then re-render every installed prompt from it.
+ vault.config.pipeline_profile = name
+ vault.config.save(vault.config_path)
+
+ from hyperresearch.core.agent_docs import _resolve_executable
+ from hyperresearch.core.hooks import install_hooks
+
+ actions = install_hooks(vault.root, hpr_path=_resolve_executable(), profile=name)
+
+ data = {
+ "gear": name,
+ "description": profile.description,
+ "sources": list(profile.source_target),
+ "time_estimate": profile.time_estimate,
+ "rerendered": actions,
+ }
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ return
+
+ console.print(f"[green]Gear switched:[/] [bold]{name}[/]")
+ if profile.description:
+ console.print(f" {profile.description}")
+ console.print(
+ f" [dim]sources {profile.source_target[0]}-{profile.source_target[1]}"
+ f" {profile.time_estimate}[/]"
+ )
+ if actions:
+ console.print(f" re-rendered {len(actions)} skill/agent file(s)")
+ else:
+ console.print(" [dim]all installed prompts already at this gear[/]")
+ console.print("[dim]Takes effect on the next /hyperresearch run. Revert with: hyperresearch profile use full[/]")
+
+
+def profile_show(
+ name: str = typer.Argument(..., help="Profile name"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Show a resolved profile (built-in defaults + user overlay)."""
+ from hyperresearch.core.profiles import ProfileError, resolve_profile
+
+ try:
+ profile = resolve_profile(name, _config_path())
+ except ProfileError as e:
+ if json_output:
+ output(error(str(e), "UNKNOWN_PROFILE"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ data = profile.model_dump()
+ if json_output:
+ output(success(data), json_mode=True)
+ else:
+ console.print(f"[bold]{profile.name}[/]")
+ for key, value in data.items():
+ if key == "name":
+ continue
+ console.print(f" {key} = {value}")
+
+
[email protected]("validate")
+def profile_validate(
+ name: str | None = typer.Argument(None, help="Profile name (default: validate all)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Validate profile definitions (built-ins + user overlays)."""
+ from hyperresearch.core.profiles import ProfileError, list_profiles, resolve_profile
+
+ cfg = _config_path()
+ names = [name] if name else list_profiles(cfg)
+ results = []
+ ok = True
+ for n in names:
+ try:
+ resolve_profile(n, cfg)
+ results.append({"name": n, "valid": True, "error": None})
+ except ProfileError as e:
+ ok = False
+ results.append({"name": n, "valid": False, "error": str(e)})
+
+ if json_output:
+ if ok:
+ output(success({"profiles": results, "all_valid": True}, count=len(results)), json_mode=True)
+ else:
+ bad = "; ".join(f"{r['name']}: {r['error']}" for r in results if not r["valid"])
+ output(error(f"invalid profile(s) — {bad}", "INVALID_PROFILE"), json_mode=True)
+ else:
+ for r in results:
+ mark = "[green]ok[/]" if r["valid"] else f"[red]INVALID[/] — {r['error']}"
+ console.print(f" [cyan]{r['name']}[/]: {mark}")
+ if not ok:
+ raise typer.Exit(1)
diff --git a/src/hyperresearch/cli/repair.py b/src/hyperresearch/cli/repair.py
index a63ffc8..ccb2711 100644
--- a/src/hyperresearch/cli/repair.py
+++ b/src/hyperresearch/cli/repair.py
@@ -221,6 +221,18 @@ def repair(
if not json_output:
console.print("[dim]5/6 Skipping indexes[/]")
+ # Step 4.5: Recompute derived ranking scores (centrality + quality).
+ # Cheap, pure-local, and keeps the score cache fresh after a force-sync
+ # rebuilt the DB from markdown.
+ from hyperresearch.core.graphrank import compute_centrality
+ from hyperresearch.core.quality import compute_quality_scores
+
+ ranked = compute_centrality(vault.db)
+ compute_quality_scores(vault.db, vault.config.ranking)
+ report["centrality_ranked"] = ranked
+ if not json_output and ranked:
+ console.print(f" centrality + quality recomputed for {ranked} notes")
+
# Step 5: Update agent docs
if update_docs:
if not json_output:
diff --git a/src/hyperresearch/cli/run_cmd.py b/src/hyperresearch/cli/run_cmd.py
new file mode 100644
index 0000000..0c3fc1a
--- /dev/null
+++ b/src/hyperresearch/cli/run_cmd.py
@@ -0,0 +1,436 @@
+"""Run commands — per-run workspaces, manifest, resume, budget."""
+
+from __future__ import annotations
+
+import json
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def _vault_or_exit(json_output: bool):
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ return Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+
+def _resolve_tag(vault, tag: str | None, json_output: bool) -> str:
+ from hyperresearch.core.runs import latest_run_tag
+
+ if tag:
+ return tag
+ latest = latest_run_tag(vault)
+ if latest is None:
+ if json_output:
+ output(error("no runs exist yet", "NO_RUNS"), json_mode=True)
+ else:
+ console.print("[red]Error:[/] no runs exist yet")
+ raise typer.Exit(1)
+ return latest
+
+
+def run_init(
+ vault_tag: str = typer.Argument(..., help="Collision-safe run tag (mint via `hyperresearch vault-tag <slug>`)"),
+ profile: str = typer.Option("full", "--profile", help="Pipeline profile for this run"),
+ budget: float | None = typer.Option(None, "--budget", help="Hard ceiling on estimated API-equivalent spend (the run blocks when the estimate crosses it; a value measure, not a bill, on subscription billing)"),
+ query_file: str | None = typer.Option(None, "--query-file", help="File whose verbatim contents become runs/<tag>/query.md"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Scaffold research/runs/<vault_tag>/ with a fresh manifest (idempotent)."""
+ from pathlib import Path
+
+ from hyperresearch.core.runs import RunError, init_run
+
+ vault = _vault_or_exit(json_output)
+ query = None
+ if query_file:
+ query = Path(query_file).read_text(encoding="utf-8-sig")
+ try:
+ manifest = init_run(vault, vault_tag, profile=profile, budget_usd=budget, query=query)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ data = {"run_dir": str(vault.run_dir(vault_tag)), "manifest": manifest}
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]Run initialized:[/] {data['run_dir']}")
+ console.print(f" profile: {manifest['profile']} budget: {manifest.get('budget_usd')}")
+
+
+def run_status(
+ vault_tag: str | None = typer.Argument(None, help="Run tag (default: newest run)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Show a run's manifest + resume position + stall/budget state."""
+ from hyperresearch.core.runs import RunError, status_summary
+
+ vault = _vault_or_exit(json_output)
+ tag = _resolve_tag(vault, vault_tag, json_output)
+ try:
+ summary = status_summary(vault, tag)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ # Browser-lane queue depth for this run
+ try:
+ from hyperresearch.core.escalation import queue_stats
+
+ summary["escalations"] = queue_stats(vault.db, vault_tag=tag)
+ except Exception:
+ summary["escalations"] = None
+
+ if json_output:
+ output(success(summary, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[bold]{summary['vault_tag']}[/] ({summary['profile']}) status: {summary['status']}")
+ esc = summary.get("escalations")
+ if esc and (esc.get("queued") or esc.get("needs_human")):
+ console.print(
+ f" [yellow]escalations:[/] {esc['queued']} queued, {esc['needs_human']} need the human"
+ )
+ if summary.get("possibly_stalled"):
+ console.print(" [yellow]possibly stalled — no manifest update recently[/]")
+ resume = summary["resume"]
+ console.print(f" done: {', '.join(resume['done_steps']) or '-'}")
+ console.print(f" next: {resume['next_step'] or '(complete)'}")
+ spend = summary.get("spend", {})
+ console.print(
+ f" API-equiv spend: ~${spend.get('estimated_usd', 0)} | {spend.get('sources_fetched', 0)} sources "
+ f"| {spend.get('agents_spawned', 0)} agents"
+ )
+ if "budget_remaining_usd" in summary:
+ console.print(f" budget remaining (API-equiv): ${summary['budget_remaining_usd']}")
+
+
+def run_list(
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """List all runs, newest first."""
+ from hyperresearch.core.runs import list_runs
+
+ vault = _vault_or_exit(json_output)
+ runs = list_runs(vault)
+ rows = [
+ {
+ "vault_tag": m.get("vault_tag"),
+ "status": m.get("status"),
+ "profile": m.get("profile"),
+ "started_at": m.get("started_at"),
+ }
+ for m in runs
+ ]
+ if json_output:
+ output(success({"runs": rows}, count=len(rows), vault=str(vault.root)), json_mode=True)
+ else:
+ for r in rows:
+ console.print(f" [cyan]{r['vault_tag']}[/] {r['status']} ({r['profile']}) {r['started_at']}")
+
+
+def run_resume(
+ vault_tag: str | None = typer.Argument(None, help="Run tag (default: newest run)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Print the exact position a recovering orchestrator should continue from."""
+ from hyperresearch.core.runs import RunError, load_manifest, resume_position, set_status
+
+ vault = _vault_or_exit(json_output)
+ tag = _resolve_tag(vault, vault_tag, json_output)
+ try:
+ manifest = load_manifest(vault, tag)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ position = resume_position(manifest)
+ if manifest["status"] in ("paused", "blocked", "failed"):
+ set_status(vault, tag, "running")
+
+ data = {
+ "vault_tag": tag,
+ "run_dir": str(vault.run_dir(tag)),
+ "profile": manifest["profile"],
+ **position,
+ "skill_to_invoke": (
+ f"hyperresearch-{position['next_step'].replace('.', '-')}"
+ if position["next_step"]
+ else None
+ ),
+ }
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ if position["next_step"] is None:
+ console.print(f"[green]{tag}[/] — all profile steps complete.")
+ else:
+ console.print(f"[green]{tag}[/] — resume at step {position['next_step']}")
+ console.print(f" Skill(skill: \"{data['skill_to_invoke']}\")")
+
+
+def run_abort(
+ vault_tag: str = typer.Argument(..., help="Run tag"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Mark a run aborted (artifacts stay on disk)."""
+ from hyperresearch.core.runs import RunError, set_status
+
+ vault = _vault_or_exit(json_output)
+ try:
+ manifest = set_status(vault, vault_tag, "aborted")
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success(manifest, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[yellow]Aborted:[/] {vault_tag}")
+
+
+def run_step(
+ vault_tag: str = typer.Argument(..., help="Run tag"),
+ step: str = typer.Argument(..., help='Step id ("1", "1.5", "11g", ...)'),
+ status: str = typer.Option(..., "--status", "-s", help="pending|running|done|skipped|failed"),
+ chapter: str | None = typer.Option(None, "--chapter", help="Chapter id for chaptered steps (e.g. ch3)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Record a step-status transition in the manifest."""
+ from hyperresearch.core.runs import RunError, set_step
+
+ vault = _vault_or_exit(json_output)
+ try:
+ manifest = set_step(vault, vault_tag, step, status, chapter=chapter)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success({"steps": manifest["steps"]}, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f" step {step} -> {status}")
+
+
+def run_spend(
+ vault_tag: str = typer.Argument(..., help="Run tag"),
+ usd: float = typer.Option(0.0, "--usd", help="Estimated API-equivalent spend to add"),
+ sources: int = typer.Option(0, "--sources", help="Sources fetched to add"),
+ notes: int = typer.Option(0, "--notes", help="Notes written to add"),
+ agents: int = typer.Option(0, "--agents", help="Subagents spawned to add"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Add spend counters. Crossing --budget flips the run to blocked."""
+ from hyperresearch.core.runs import RunError, add_spend
+
+ vault = _vault_or_exit(json_output)
+ try:
+ manifest = add_spend(
+ vault, vault_tag,
+ estimated_usd=usd, sources_fetched=sources,
+ notes_written=notes, agents_spawned=agents,
+ )
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ data = {"spend": manifest["spend"], "status": manifest["status"], "blocked_on": manifest["blocked_on"]}
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f" spend: {data['spend']} status: {data['status']}")
+
+
+def run_event(
+ vault_tag: str = typer.Argument(..., help="Run tag"),
+ type_: str = typer.Option(..., "--type", help="Event type (spawn, fetch-wave, escalation, note)"),
+ data: str | None = typer.Option(None, "--data", help="JSON payload for the event"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Append an event to events.jsonl (also heartbeats the manifest)."""
+ from hyperresearch.core.runs import RunError, record_event
+
+ vault = _vault_or_exit(json_output)
+ payload = {"type": type_}
+ if data:
+ try:
+ payload.update(json.loads(data))
+ except json.JSONDecodeError as e:
+ if json_output:
+ output(error(f"--data is not valid JSON: {e}", "BAD_JSON"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] --data is not valid JSON: {e}")
+ raise typer.Exit(1)
+ try:
+ record_event(vault, vault_tag, payload)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success({"recorded": payload}, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(" event recorded")
+
+
+def run_block(
+ vault_tag: str = typer.Argument(..., help="Run tag"),
+ on: str = typer.Option(..., "--on", help="What the run is blocked on (e.g. human-challenges, budget)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Mark a run blocked (e.g. on human browser challenges). `run resume` unblocks."""
+ from hyperresearch.core.runs import RunError, set_status
+
+ vault = _vault_or_exit(json_output)
+ try:
+ manifest = set_status(vault, vault_tag, "blocked", blocked_on=on)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+ if json_output:
+ output(success({"status": manifest["status"], "blocked_on": manifest["blocked_on"]}, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[yellow]Blocked:[/] {vault_tag} on {on}")
+
+
+def run_report(
+ vault_tag: str | None = typer.Argument(None, help="Run tag (default: newest run); ignored with --all"),
+ all_runs: bool = typer.Option(False, "--all", help="Aggregate across every run"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Post-run telemetry: per-step wall-time, spend, and yield.
+
+ The feedback loop for profile tuning — replace inherited constants with
+ observed yield curves ("what does wave 2 actually produce?").
+ """
+ from hyperresearch.core.runs import RunError, list_runs, load_manifest, run_report_data
+
+ vault = _vault_or_exit(json_output)
+
+ if all_runs:
+ reports = []
+ for manifest in list_runs(vault):
+ tag = manifest.get("vault_tag")
+ if not tag or manifest.get("status") == "corrupt-manifest":
+ continue
+ try:
+ reports.append(run_report_data(vault, tag))
+ except RunError:
+ continue
+ agg = {
+ "runs": len(reports),
+ "total_estimated_usd": round(sum(r["spend"]["estimated_usd"] for r in reports), 2),
+ "total_sources": sum(r["spend"]["sources_fetched"] for r in reports),
+ "total_agents": sum(r["spend"]["agents_spawned"] for r in reports),
+ "by_run": reports,
+ }
+ if json_output:
+ output(success(agg, count=len(reports), vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[bold]{agg['runs']} runs[/] — ~${agg['total_estimated_usd']} API-equiv, "
+ f"{agg['total_sources']} sources, {agg['total_agents']} agents")
+ for r in reports:
+ console.print(f" [cyan]{r['vault_tag']}[/] {r['status']} ~${r['spend']['estimated_usd']} "
+ f"{r['spend']['sources_fetched']}src {r['total_wall_minutes']}min")
+ return
+
+ tag = _resolve_tag(vault, vault_tag, json_output)
+ try:
+ load_manifest(vault, tag)
+ report = run_report_data(vault, tag)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ if json_output:
+ output(success(report, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[bold]{report['vault_tag']}[/] ({report['profile']}) — {report['status']}, "
+ f"{report['total_wall_minutes']} min total")
+ for step in report["steps"]:
+ console.print(f" step {step['step']:>4}: {step['status']:<8} {step['minutes']} min")
+ spend = report["spend"]
+ console.print(f" API-equiv spend: ~${spend['estimated_usd']} | {spend['sources_fetched']} sources | "
+ f"{spend['agents_spawned']} agents | {spend['notes_written']} notes")
+ ev = report["events"]
+ if ev:
+ console.print(f" events: {ev}")
+
+
+def run_verify(
+ vault_tag: str | None = typer.Argument(None, help="Run tag (default: newest run)"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Structural verification battery (exit 1 on failure) — the CI-able gate.
+
+ Report exists / headings honored / length in profile range / citation
+ density above floor / tier artifacts present / cite-check resolved.
+ Pair with `hyperresearch lint -j` for the content rules.
+ """
+ from hyperresearch.core.runs import RunError, verify_run
+
+ vault = _vault_or_exit(json_output)
+ tag = _resolve_tag(vault, vault_tag, json_output)
+ try:
+ result = verify_run(vault, tag)
+ except RunError as e:
+ if json_output:
+ output(error(str(e), "RUN_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ if json_output:
+ output(success(result, vault=str(vault.root)), json_mode=True)
+ else:
+ for c in result["checks"]:
+ mark = "[green]ok[/]" if c["ok"] else "[red]FAIL[/]"
+ console.print(f" {mark} {c['name']}: {c['detail']}")
+ console.print("[green]PASSED[/]" if result["passed"] else "[red]FAILED[/]")
+ if not result["passed"]:
+ raise typer.Exit(1)
diff --git a/src/hyperresearch/cli/search.py b/src/hyperresearch/cli/search.py
index a358d9a..56f9641 100644
--- a/src/hyperresearch/cli/search.py
+++ b/src/hyperresearch/cli/search.py
@@ -27,7 +27,9 @@ def search(
has_backlinks: bool = typer.Option(False, "--has-backlinks", help="Only notes with backlinks"),
include_body: bool = typer.Option(False, "--include-body", help="Include full note body (auto-enabled with --json)"),
no_body: bool = typer.Option(False, "--no-body", help="Disable auto body inclusion in JSON mode"),
- limit: int = typer.Option(20, "--limit", "-l", help="Max results"),
+ limit: int | None = typer.Option(None, "--limit", "-l", help="Max results (default from [search] config)"),
+ ranked: bool = typer.Option(False, "--ranked", help="Fold composite source-quality scores into ranking (tier + utility + authority + centrality)"),
+ semantic: bool = typer.Option(False, "--semantic", help="Blend in embedding similarity (requires [embeddings] provider)"),
offset: int = typer.Option(0, "--offset", help="Offset for pagination"),
max_tokens: int | None = typer.Option(None, "--max-tokens", help="Truncate results to fit token budget (1 token ~ 4 chars)"),
json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
@@ -69,6 +71,8 @@ def search(
console.print(f"[red]Error:[/] {e}")
raise typer.Exit(1)
vault.auto_sync()
+ if limit is None:
+ limit = vault.config.search_default_limit
filters = SearchFilters(
tags=tag or None,
@@ -99,7 +103,8 @@ def search(
}
try:
results = search_fts(
- vault.db, query, filters=filters, limit=limit, offset=offset, ranking=ranking
+ vault.db, query, filters=filters, limit=limit, offset=offset,
+ ranking=ranking, quality_ranked=ranked,
)
except SearchQueryError as e:
if json_output:
@@ -108,6 +113,47 @@ def search(
console.print(f"[red]Error:[/] {e}")
raise typer.Exit(2) from e
+ # Hybrid semantic blend: RRF-fuse the FTS ranking with embedding cosine
+ # neighbors, then refetch metadata for any semantic-only hits.
+ if semantic:
+ from hyperresearch.core.embed import (
+ EmbeddingError,
+ reciprocal_rank_fusion,
+ semantic_search,
+ )
+
+ try:
+ sem_hits = semantic_search(vault, query, limit=limit)
+ except EmbeddingError as e:
+ if json_output:
+ output(error(str(e), "EMBEDDINGS_DISABLED"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ fused = reciprocal_rank_fusion(
+ [[r["id"] for r in results], [h["id"] for h in sem_hits]]
+ )
+ by_id = {r["id"]: r for r in results}
+ merged = []
+ for note_id, fused_score in fused[:limit]:
+ r = by_id.get(note_id)
+ if r is None:
+ row = vault.db.execute(
+ "SELECT id, title, path, status, type, tier, content_type, "
+ "quality_score, created, updated, word_count, summary "
+ "FROM notes WHERE id = ?",
+ (note_id,),
+ ).fetchone()
+ if row is None:
+ continue
+ r = dict(row)
+ r["tags"] = []
+ r["snippet"] = ""
+ r["score"] = fused_score
+ merged.append(r)
+ results = merged
+
# Auto-include body in JSON mode (agents almost always need it)
want_body = include_body or (json_output and not no_body)
if want_body and results:
@@ -119,7 +165,7 @@ def search(
# Token budget: truncate results to fit within limit
if max_tokens and results:
- budget_chars = max_tokens * 4 # ~4 chars per token
+ budget_chars = max_tokens * vault.config.search_chars_per_token
used = 0
trimmed = []
for r in results:
@@ -157,5 +203,5 @@ def search(
f"[dim]({r['status']})[/] [green]{tags_str}[/]"
)
if r.get("snippet"):
- console.print(f" [dim]{r['snippet'][:200]}[/]")
+ console.print(f" [dim]{r['snippet'][: vault.config.search_snippet_len]}[/]")
console.print()
diff --git a/src/hyperresearch/cli/setup.py b/src/hyperresearch/cli/setup.py
index 49f26f5..2936f26 100644
--- a/src/hyperresearch/cli/setup.py
+++ b/src/hyperresearch/cli/setup.py
@@ -143,8 +143,9 @@ def setup(
for action in doc_actions:
console.print(f" [green]Docs:[/] {action}")
- # Install Claude Code hook + skills + subagents
- hook_actions = install_hooks(root, hpr_path=hpr_path)
+ # Install Claude Code hook + skills + subagents (rendered from the
+ # persisted scale gear, if one was chosen via `hpr profile use`)
+ hook_actions = install_hooks(root, hpr_path=hpr_path, profile=vault.config.pipeline_profile)
for action in hook_actions:
console.print(f" [green]Hook:[/] {action}")
if not hook_actions:
diff --git a/src/hyperresearch/cli/sources.py b/src/hyperresearch/cli/sources.py
index 21a7f6e..bcfaa23 100644
--- a/src/hyperresearch/cli/sources.py
+++ b/src/hyperresearch/cli/sources.py
@@ -95,3 +95,144 @@ def source_check(
output(success(data, vault=str(vault.root)), json_mode=True)
else:
console.print(f"[dim]Not fetched:[/] {url}")
+
+
+def sources_score(
+ tag: str | None = typer.Option(None, "--tag", "-t", help="Only notes with this tag (e.g. vault_tag)"),
+ fresh: bool = typer.Option(False, "--fresh", help="Bypass the api_cache TTL and re-check already-enriched notes"),
+ limit: int | None = typer.Option(None, "--limit", "-n", help="Max notes to enrich this run"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Enrich DOI-bearing notes with citation metadata (OpenAlex/Semantic Scholar).
+
+ Populates citation_count / venue / is_retracted in frontmatter + DB, then
+ recomputes authority percentiles and composite quality scores. Responses
+ are cached in the api_cache table ([ranking] api_cache_ttl_days).
+ """
+ from hyperresearch.core.scholar import score_sources
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ result = score_sources(vault, tag=tag, fresh=fresh, limit=limit)
+
+ if json_output:
+ output(success(result, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]Scored:[/] {result['scored']} notes enriched")
+ console.print(f" Authority percentiles ranked: {result['authority_ranked']}")
+ if result["retracted"]:
+ console.print(f" [red]RETRACTED:[/] {', '.join(result['retracted'])}")
+ if result["missing"]:
+ console.print(f" [yellow]No metadata found:[/] {len(result['missing'])} notes")
+
+
[email protected]("backfill-doi")
+def sources_backfill_doi(
+ tag: str | None = typer.Option(None, "--tag", "-t", help="Only notes with this tag"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Extract DOIs from existing notes' source URLs and bodies (back-catalog)."""
+ from hyperresearch.core.scholar import backfill_dois
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ gained = backfill_dois(vault, tag=tag)
+
+ if json_output:
+ output(success({"dois_added": gained}, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]DOIs added:[/] {gained} notes")
+
+
[email protected]("retractions")
+def sources_retractions(
+ tag: str | None = typer.Option(None, "--tag", "-t", help="Only notes with this tag"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Ship-time retraction sweep: re-check every DOI-bearing note fresh.
+
+ Bypasses the api_cache TTL so a retraction published yesterday is caught
+ today. Run before the step-15 integrity gate; the `retracted-citations`
+ lint rule then blocks any unacknowledged retracted citation.
+ """
+ from hyperresearch.core.scholar import score_sources
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ result = score_sources(vault, tag=tag, fresh=True)
+
+ data = {"checked": result["scored"], "retracted": result["retracted"], "unresolved": len(result["missing"])}
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]Retraction sweep:[/] {data['checked']} DOI-bearing notes re-checked")
+ if data["retracted"]:
+ console.print(f" [red]RETRACTED:[/] {', '.join(data['retracted'])}")
+ else:
+ console.print(" no retractions found")
+
+
[email protected]("independence")
+def sources_independence(
+ tag: str | None = typer.Option(None, "--tag", "-t", help="Only notes with this tag"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Cluster derivative sources and score independence.
+
+ Five syndicated copies of one press release should count as ONE voice in
+ consensus math, not five. Clusters via canonical-URL identity, near-
+ duplicate bodies (MinHash), and shared wire-service boilerplate; writes
+ notes.independence (root 1.0, members 1/cluster_size).
+ """
+ from hyperresearch.core.independence import compute_independence
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ result = compute_independence(vault, tag=tag)
+
+ if json_output:
+ output(success(result, count=len(result["clusters"]), vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(
+ f"[green]Independence:[/] {result['scored']} notes scored, "
+ f"{len(result['clusters'])} derivative cluster(s)"
+ )
+ for c in result["clusters"]:
+ console.print(f" [{c['kind']}] root [cyan]{c['root']}[/] <- {', '.join(c['members'])}")
diff --git a/src/hyperresearch/cli/vault_tag.py b/src/hyperresearch/cli/vault_tag.py
index 6c5c122..4ac97d4 100644
--- a/src/hyperresearch/cli/vault_tag.py
+++ b/src/hyperresearch/cli/vault_tag.py
@@ -67,6 +67,14 @@ def _existing_tags(vault_root: Path, research_dir: Path) -> set[str]:
m = _REPORT_FILE_RE.match(p.name)
if m:
tags.add(m.group(1))
+ # 3.0 per-run workspaces: every research/runs/<tag>/ directory IS a tag
+ # (excluding archive-* dirs from `archive-run`, which embed old tags with
+ # a prefix that can't collide with fresh mints).
+ runs_dir = research_dir / "runs"
+ if runs_dir.is_dir():
+ for d in runs_dir.iterdir():
+ if d.is_dir() and not d.name.startswith("archive-"):
+ tags.add(d.name)
return tags
diff --git a/src/hyperresearch/core/agent_docs.py b/src/hyperresearch/core/agent_docs.py
index 711f478..92c1b08 100644
--- a/src/hyperresearch/core/agent_docs.py
+++ b/src/hyperresearch/core/agent_docs.py
@@ -27,27 +27,40 @@ This project uses hyperresearch as an agent-driven research knowledge base. The
### How to do research
-**Run a research session with `/hyperresearch <query>`.** This invokes the V8 16-step pipeline. The entry skill at `.claude/skills/hyperresearch/SKILL.md` is a thin ROUTER. The 16 step procedures live in their own skills (`hyperresearch-1-decompose` through `hyperresearch-16-readability-audit`) and are loaded fresh into context via the `Skill` tool when each step runs. This solves V7's context-compaction problem: each step's procedure lands in context only when needed. Read the entry skill before you start a research session; it explains the chain mechanics.
+**Run a research session with `/hyperresearch <query>`.** This invokes the V8 16-step pipeline. The entry skill at `.claude/skills/hyperresearch/SKILL.md` is a thin ROUTER. The step procedures live in their own skills (`hyperresearch-1-decompose` through `hyperresearch-16-readability-audit`, plus half-steps `1-5-chapter-partition` and `14-5-cite-check`) and are loaded fresh into context via the `Skill` tool when each step runs. This solves V7's context-compaction problem: each step's procedure lands in context only when needed. Read the entry skill before you start a research session; it explains the chain mechanics.
-Step 1 classifies the query into one of two tiers (`light` or `full`) and the rest of the pipeline scales accordingly — short bounded queries skip the depth investigations, critics, and patcher (~30-40 min); argumentative deep-research queries run all 16 steps with adversarial review (~1.5-2.5 hours).
+Step 1 classifies the query into a tier (`light` or `full`; `dissertation` is opt-in per run, never auto-classified) and the rest of the pipeline scales accordingly — short bounded queries skip the depth investigations, critics, and patcher (~30-40 min); argumentative deep-research queries run all 16 steps with adversarial review; dissertation runs loop steps 2-10 per chapter. Orthogonal to tiers, the installed **scale gear** (`full` ~55-80 sources, or `premier` ~100-130 sources with doubled depth budget) sets the numbers rendered into the step skills — the user switches it with `{hpr} profile use <full|premier>`; inspect with `{hpr} profile list -j`.
**Do NOT use WebFetch for source pages** — use `{hpr} fetch` instead. The skill files explain when to fetch vs. search.
+### Run management and verification
+
+Every run owns a workspace at `research/runs/<vault_tag>/` and a manifest (`run.json`) — the durable record of pipeline position and spend:
+
+```bash
+{hpr} run status -j # Newest run: step status, spend, escalation queue depth
+{hpr} run resume -j # Exact next step + Skill invocation to continue with
+{hpr} run report -j # Per-step wall-time / spend / event telemetry
+{hpr} run verify <vault_tag> -j # Ship gate: headings, length, citation density, cite-check resolution
+```
+
+Blocked fetches (login walls, bot walls, captchas) queue as escalations instead of dying: `{hpr} escalation list --status queued -j`. The browser-fetcher agent drains them via the user's real Chrome; CAPTCHAs / logins / 2FA are ALWAYS handed to the human, consolidated into one message.
+
### What the skill files own
The skill files own everything about how to research. That includes:
- The pipeline phases and what each phase does
-- Which subagents exist and what each one is for (fetcher, loci-analyst, depth-investigator, 4 critics, patcher, polish-auditor)
+- Which subagents exist and what each one is for (fetcher, source-analyst, loci-analyst, depth-investigator, corpus-critic, draft-orchestrators, synthesizer, 4 critics, patcher, cite-checker, polish-auditor, readability-recommender, browser-fetcher)
- The tool-lock invariant (patcher and polish-auditor can only Read + Edit, never Write)
- The subagent spawn contract (every Task call passes the verbatim research_query + pipeline position + inputs)
-- Artifact locations (`research/scaffold.md`, `research/prompt-decomposition.json`, `research/loci.json`, `research/comparisons.md`, interim notes, patch / polish logs)
+- Artifact locations — everything run-scoped lives under `research/runs/<vault_tag>/` (scaffold.md, prompt-decomposition.json, loci.json, comparisons.md, critic findings, patch / polish logs); final reports at `research/notes/final_report_<vault_tag>.md`
- The curation pass after every research session
If you need to know how hyperresearch works, read the skill file. This document does NOT duplicate that content — when the skill file and this file disagree, the skill file wins.
### Canonical research query
-In a normal run, the canonical research query is the user's verbatim prompt. In wrapped runs, if `research/prompt.txt` exists, that file is gospel and overrides any wrapping instructions. The pipeline persists the query as `research/query-<vault_tag>.md` with YAML frontmatter — this is the canonical query reference for all downstream layers. Wrapper requirements (save path, citation format, terminal sections) are a separate contract, captured in the scaffold — not pasted into the `## User Prompt (VERBATIM — gospel)` section.
+In a normal run, the canonical research query is the user's verbatim prompt. In wrapped runs, if `research/prompt.txt` exists, that file is gospel and overrides any wrapping instructions. The pipeline persists the query as `research/runs/<vault_tag>/query.md` with YAML frontmatter — this is the canonical query reference for all downstream steps. Wrapper requirements (save path, citation format, terminal sections) are a separate contract, captured in the scaffold — not pasted into the `## User Prompt (VERBATIM — gospel)` section.
### Academic APIs before web search
@@ -100,6 +113,8 @@ Every research session must end with a curation pass:
{hpr} note update <id> --summary "<specific summary>" --add-tag <t> -j # Add summary + tags
{hpr} lint -j # Find missing tags / summaries / broken links
{hpr} repair -j # Auto-fix broken links, rebuild indexes
+{hpr} sources score -j # Enrich DOI-bearing sources (citations, venue, retractions) + recompute quality
+{hpr} graph rank -j # Recompute vault PageRank centrality
{hpr} status -j # Overall vault health
```
diff --git a/src/hyperresearch/core/citecheck.py b/src/hyperresearch/core/citecheck.py
new file mode 100644
index 0000000..db6def3
--- /dev/null
+++ b/src/hyperresearch/core/citecheck.py
@@ -0,0 +1,213 @@
+"""Cite-check — does each citation actually support its sentence?
+
+The FACT half of research quality, self-measured before ship instead of by
+an external benchmark. Three layers:
+
+1. `extract_pairs(report_text, conn)` — pure parsing. Splits the report into
+ sentences and binds each citation marker to its sentence, for BOTH
+ citation styles: numbered `[N]` (resolved through the `## Sources`
+ section) and `[[note-id]]` wikilinks.
+
+2. `triage_pairs(pairs, conn)` — mechanical tier. A pair auto-passes when
+ the sentence's numbers or a long word-overlap window appear in the cited
+ note's claims (`quoted_support` / `numbers`) — no LLM needed for the
+ bulk. The remainder is marked `needs-llm` for the cite-checker agent.
+
+3. The `hyperresearch-cite-checker` agent (step 14.5) verifies the
+ needs-llm tail against the actual note bodies and emits findings the
+ patcher applies. Verdicts: supported | partially-supported |
+ unsupported | wrong-source.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+
+from hyperresearch.core.patterns import WIKI_LINK_RE
+
+# Sentence split: period/question/exclamation followed by space+capital, or newline.
+_SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+(?=[A-Z一-鿿])|\n+")
+_NUMBERED_CITE_RE = re.compile(r"\[(\d{1,3})\]")
+_SOURCES_ENTRY_RE = re.compile(r"^\s*\[(\d{1,3})\]\s+(.+)$")
+_NUMBER_RE = re.compile(r"\d[\d,]*\.?\d*%?")
+
+# Sentences carrying these are checked at 100% regardless of sampling.
+_STRONG_MARKERS = ("%", "$", "billion", "million", "increase", "decrease", "grew", "fell")
+
+
+def _split_sentences(text: str) -> list[str]:
+ return [s.strip() for s in _SENTENCE_SPLIT_RE.split(text) if s.strip()]
+
+
+def parse_sources_section(report_text: str, conn) -> dict[str, str | None]:
+ """Map `[N]` -> note_id by matching Sources-section URLs/titles to the vault."""
+ mapping: dict[str, str | None] = {}
+ in_sources = False
+ for line in report_text.splitlines():
+ if re.match(r"^##\s+(Sources|References)\b", line, re.IGNORECASE):
+ in_sources = True
+ continue
+ if in_sources and line.startswith("## "):
+ break
+ if not in_sources:
+ continue
+ m = _SOURCES_ENTRY_RE.match(line)
+ if not m:
+ continue
+ num, rest = m.group(1), m.group(2)
+ note_id = None
+ url_m = re.search(r"https?://\S+", rest)
+ if url_m:
+ url = url_m.group(0).rstrip(".,)")
+ row = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()
+ if row:
+ note_id = row["note_id"]
+ if note_id is None:
+ title = rest.split("http")[0].strip(" .–-")
+ if title:
+ row = conn.execute(
+ "SELECT id FROM notes WHERE title = ? COLLATE NOCASE", (title,)
+ ).fetchone()
+ if row:
+ note_id = row["id"]
+ mapping[num] = note_id
+ return mapping
+
+
+def extract_pairs(report_text: str, conn) -> list[dict]:
+ """All (sentence, note_id) citation bindings in the report.
+
+ Pairs whose citation can't be resolved to a vault note get
+ note_id=None — those are findings in themselves (dangling citation).
+ """
+ numbered_map = parse_sources_section(report_text, conn)
+ known_ids = {row["id"] for row in conn.execute("SELECT id FROM notes")}
+
+ # Strip the Sources section from the checked body
+ body = re.split(r"^##\s+(?:Sources|References)\b", report_text, maxsplit=1, flags=re.M | re.I)[0]
+
+ pairs: list[dict] = []
+ for sentence in _split_sentences(body):
+ cited: list[str | None] = []
+ for m in _NUMBERED_CITE_RE.finditer(sentence):
+ if m.group(1) in numbered_map:
+ cited.append(numbered_map[m.group(1)])
+ for m in WIKI_LINK_RE.finditer(sentence):
+ target = m.group(1).strip()
+ cited.append(target if target in known_ids else None)
+ for note_id in cited:
+ pairs.append({
+ "sentence": sentence,
+ "note_id": note_id,
+ "numbers": _NUMBER_RE.findall(sentence),
+ "strong": any(k in sentence.lower() for k in _STRONG_MARKERS) or bool(_NUMBER_RE.search(sentence)),
+ })
+ return pairs
+
+
+def _norm(text: str) -> str:
+ return re.sub(r"\s+", " ", text.lower()).strip()
+
+
+def triage_pairs(pairs: list[dict], conn) -> dict:
+ """Mechanical verification tier.
+
+ Verdicts per pair:
+ dangling — citation resolves to no vault note (finding)
+ supported-mechanical — sentence numbers / long overlap found in the
+ cited note's claims (auto-pass)
+ needs-llm — the cite-checker agent must judge it
+ """
+ claims_by_note: dict[str, list[dict]] = {}
+
+ def _claims(note_id: str) -> list[dict]:
+ if note_id not in claims_by_note:
+ rows = conn.execute(
+ "SELECT claim, quoted_support, numbers FROM claims WHERE note_id = ?",
+ (note_id,),
+ ).fetchall()
+ claims_by_note[note_id] = [dict(r) for r in rows]
+ return claims_by_note[note_id]
+
+ supported = 0
+ dangling = 0
+ needs_llm = []
+ for pair in pairs:
+ if pair["note_id"] is None:
+ pair["verdict"] = "dangling"
+ dangling += 1
+ continue
+ matched = False
+ note_claims = _claims(pair["note_id"])
+ blob = _norm(" ".join(
+ (c["claim"] or "") + " " + (c["quoted_support"] or "") + " " + (c["numbers"] or "")
+ for c in note_claims
+ ))
+ if blob:
+ nums = [n for n in pair["numbers"] if len(n.replace(",", "")) >= 2]
+ if nums and all(n.replace(",", "") in blob.replace(",", "") for n in nums):
+ matched = True
+ elif not nums:
+ # Long word-overlap window: any 6-consecutive-word shingle of the
+ # sentence found in the claims blob
+ words = _norm(pair["sentence"]).split()
+ for i in range(len(words) - 5):
+ if " ".join(words[i : i + 6]) in blob:
+ matched = True
+ break
+ if matched:
+ pair["verdict"] = "supported-mechanical"
+ supported += 1
+ else:
+ pair["verdict"] = "needs-llm"
+ needs_llm.append(pair)
+
+ return {
+ "total": len(pairs),
+ "supported_mechanical": supported,
+ "dangling": dangling,
+ "needs_llm": len(needs_llm),
+ "pairs": pairs,
+ }
+
+
+def sample_needs_llm(pairs: list[dict], sample_rate: float = 0.6) -> list[dict]:
+ """Deterministic sampling of the LLM tier: 100% of strong (number-bearing)
+ sentences, every k-th of the rest. No RNG — reproducible across resumes."""
+ out = []
+ weak_kept = 0
+ weak_seen = 0
+ keep_every = max(1, round(1 / sample_rate)) if sample_rate > 0 else 0
+ for pair in pairs:
+ if pair.get("verdict") != "needs-llm":
+ continue
+ if pair["strong"]:
+ out.append(pair)
+ elif keep_every:
+ weak_seen += 1
+ if weak_seen % keep_every == 0:
+ out.append(pair)
+ weak_kept += 1
+ return out
+
+
+def write_pairs_file(vault, vault_tag: str, report_path, sample_rate: float = 0.6) -> dict:
+ """Extract + triage + sample; write cite-check-pairs.json into the run dir."""
+ report_text = report_path.read_text(encoding="utf-8-sig")
+ pairs = extract_pairs(report_text, vault.db)
+ triaged = triage_pairs(pairs, vault.db)
+ to_check = sample_needs_llm(triaged["pairs"], sample_rate)
+
+ run_dir = vault.run_dir(vault_tag)
+ run_dir.mkdir(parents=True, exist_ok=True)
+ out = {
+ "report": str(report_path),
+ "summary": {k: v for k, v in triaged.items() if k != "pairs"},
+ "sampled_for_llm": to_check,
+ "dangling": [p for p in triaged["pairs"] if p["verdict"] == "dangling"],
+ }
+ (run_dir / "cite-check-pairs.json").write_text(
+ json.dumps(out, indent=2, ensure_ascii=False), encoding="utf-8"
+ )
+ return out
diff --git a/src/hyperresearch/core/claims.py b/src/hyperresearch/core/claims.py
new file mode 100644
index 0000000..18a4df0
--- /dev/null
+++ b/src/hyperresearch/core/claims.py
@@ -0,0 +1,263 @@
+"""Claims persistence — fetcher-extracted claims as queryable DB rows.
+
+Fetchers write `research/temp/claims-<note-id>.json` files during step 2.
+This module ingests them into the `claims` (+ `claims_fts`) tables, keyed to
+their source notes, so downstream consumers can ask "which source best
+supports X" as a query instead of re-parsing JSON files. This is the
+substrate for phase-5 cite-checking and numeric-consistency lints.
+
+Ingest is idempotent: rows are keyed by (note_id, sha256(claim)[:16]), so
+re-running over the same files is a no-op.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from datetime import UTC, datetime
+from pathlib import Path
+
+
+def _claim_hash(claim: str) -> str:
+ return hashlib.sha256(claim.strip().encode("utf-8")).hexdigest()[:16]
+
+
+def _note_id_from_filename(path: Path) -> str | None:
+ """claims-<note-id>.json -> <note-id>."""
+ stem = path.stem
+ if stem.startswith("claims-"):
+ return stem[len("claims-"):]
+ return None
+
+
+def _iter_claim_dicts(data) -> list[dict]:
+ """Accept both bare-list files and {claims: [...]} wrappers."""
+ if isinstance(data, list):
+ return [c for c in data if isinstance(c, dict)]
+ if isinstance(data, dict):
+ inner = data.get("claims", [])
+ if isinstance(inner, list):
+ return [c for c in inner if isinstance(c, dict)]
+ return []
+
+
+def ingest_claims_file(conn, path: Path, vault_tag: str | None = None) -> dict:
+ """Ingest one claims JSON file. Returns {ingested, skipped, errors}."""
+ note_id = _note_id_from_filename(path)
+ result = {"file": str(path), "note_id": note_id, "ingested": 0, "skipped": 0, "errors": []}
+ if note_id is None:
+ result["errors"].append("filename does not match claims-<note-id>.json")
+ return result
+
+ note_row = conn.execute("SELECT id FROM notes WHERE id = ?", (note_id,)).fetchone()
+ if note_row is None:
+ result["errors"].append(f"note '{note_id}' not in vault (sync first?)")
+ return result
+
+ try:
+ data = json.loads(path.read_text(encoding="utf-8-sig"))
+ except (json.JSONDecodeError, OSError) as e:
+ result["errors"].append(f"unreadable JSON: {e}")
+ return result
+
+ now = datetime.now(UTC).isoformat()
+ for c in _iter_claim_dicts(data):
+ claim_text = (c.get("claim") or c.get("text") or "").strip()
+ if not claim_text:
+ result["skipped"] += 1
+ continue
+ h = _claim_hash(claim_text)
+ numbers = c.get("numbers")
+ cur = conn.execute(
+ """INSERT OR IGNORE INTO claims
+ (note_id, claim, claim_hash, quoted_support, numbers, confidence,
+ evidence_type, stance_target, stance, vault_tag, ingested_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+ (
+ note_id,
+ claim_text,
+ h,
+ c.get("quoted_support"),
+ json.dumps(numbers) if numbers else None,
+ c.get("confidence"),
+ c.get("evidence_type"),
+ c.get("stance_target"),
+ c.get("stance"),
+ vault_tag,
+ now,
+ ),
+ )
+ if cur.rowcount:
+ claim_id = cur.lastrowid
+ conn.execute(
+ "INSERT INTO claims_fts (claim_id, claim, quoted_support) VALUES (?, ?, ?)",
+ (claim_id, claim_text, c.get("quoted_support") or ""),
+ )
+ result["ingested"] += 1
+ else:
+ result["skipped"] += 1
+ return result
+
+
+def ingest_claims_dir(vault, temp_dir: Path | None = None, vault_tag: str | None = None) -> dict:
+ """Ingest every claims-*.json under research/temp/. Returns a summary."""
+ conn = vault.db
+ if temp_dir is None:
+ temp_dir = vault.root / "research" / "temp"
+ files = sorted(temp_dir.glob("claims-*.json")) if temp_dir.is_dir() else []
+ summary = {"files": len(files), "ingested": 0, "skipped": 0, "errors": []}
+ for f in files:
+ r = ingest_claims_file(conn, f, vault_tag)
+ summary["ingested"] += r["ingested"]
+ summary["skipped"] += r["skipped"]
+ for e in r["errors"]:
+ summary["errors"].append(f"{f.name}: {e}")
+ conn.commit()
+ return summary
+
+
+def search_claims(conn, query: str, limit: int = 20) -> list[dict]:
+ """FTS search over claims + quoted support."""
+ rows = conn.execute(
+ """SELECT c.id, c.note_id, c.claim, c.quoted_support, c.numbers,
+ c.confidence, c.evidence_type, c.stance_target, c.stance, c.vault_tag
+ FROM claims_fts f JOIN claims c ON c.id = f.claim_id
+ WHERE claims_fts MATCH ?
+ ORDER BY rank LIMIT ?""",
+ (query, limit),
+ ).fetchall()
+ return [dict(r) for r in rows]
+
+
+def list_claims(
+ conn,
+ note_id: str | None = None,
+ vault_tag: str | None = None,
+ limit: int = 100,
+) -> list[dict]:
+ query = (
+ "SELECT id, note_id, claim, quoted_support, numbers, confidence, "
+ "evidence_type, stance_target, stance, vault_tag FROM claims"
+ )
+ conds, params = [], []
+ if note_id:
+ conds.append("note_id = ?")
+ params.append(note_id)
+ if vault_tag:
+ conds.append("vault_tag = ?")
+ params.append(vault_tag)
+ if conds:
+ query += " WHERE " + " AND ".join(conds)
+ query += " ORDER BY id LIMIT ?"
+ params.append(limit)
+ return [dict(r) for r in conn.execute(query, params).fetchall()]
+
+
+def literature_matrix(conn, vault_tag: str | None = None) -> list[dict]:
+ """Per-source literature-review rows: source metadata + claim digest.
+
+ The dissertation-scale artifact: one row per source note that has claims,
+ carrying tier/quality/venue plus a digest of what its claims establish.
+ Sorted by quality_score (best evidence first), None-quality last.
+ """
+ query = """
+ SELECT n.id, n.title, n.tier, n.content_type, n.venue,
+ n.citation_count, n.is_retracted, n.quality_score,
+ n.created, n.source,
+ COUNT(c.id) AS n_claims,
+ SUM(CASE WHEN c.evidence_type IN ('empirical','statistical') THEN 1 ELSE 0 END) AS n_empirical,
+ SUM(CASE WHEN c.numbers IS NOT NULL THEN 1 ELSE 0 END) AS n_quantified
+ FROM claims c JOIN notes n ON n.id = c.note_id
+ """
+ params: list = []
+ if vault_tag:
+ query += " WHERE c.vault_tag = ?"
+ params.append(vault_tag)
+ query += " GROUP BY n.id"
+ rows = [dict(r) for r in conn.execute(query, params).fetchall()]
+
+ # Attach each source's single highest-signal claim (empirical + quantified first)
+ for row in rows:
+ top = conn.execute(
+ """SELECT claim FROM claims WHERE note_id = ?
+ ORDER BY (CASE WHEN evidence_type IN ('empirical','statistical') THEN 0 ELSE 1 END),
+ (CASE WHEN numbers IS NOT NULL THEN 0 ELSE 1 END),
+ (CASE WHEN confidence = 'high' THEN 0 ELSE 1 END),
+ id
+ LIMIT 1""",
+ (row["id"],),
+ ).fetchone()
+ row["key_claim"] = top["claim"] if top else None
+
+ rows.sort(key=lambda r: (r["quality_score"] is None, -(r["quality_score"] or 0.0)))
+ return rows
+
+
+def render_matrix_markdown(rows: list[dict]) -> str:
+ """Render literature_matrix rows as a markdown table."""
+ lines = [
+ "| Source | Tier | Type | Venue | Cites | Quality | Claims (emp/quant) | Key finding |",
+ "|---|---|---|---|---|---|---|---|",
+ ]
+ for r in rows:
+ quality = f"{r['quality_score']:.2f}" if r["quality_score"] is not None else "-"
+ retracted = " **RETRACTED**" if r["is_retracted"] else ""
+ key = (r["key_claim"] or "").replace("|", "/")[:140]
+ lines.append(
+ f"| [[{r['id']}]]{retracted} | {r['tier'] or '-'} | {r['content_type'] or '-'} "
+ f"| {r['venue'] or '-'} | {r['citation_count'] if r['citation_count'] is not None else '-'} "
+ f"| {quality} | {r['n_claims']} ({r['n_empirical'] or 0}/{r['n_quantified'] or 0}) | {key} |"
+ )
+ return "\n".join(lines) + "\n"
+
+
+def group_by_target(conn, vault_tag: str | None = None, min_sources: int = 2) -> list[dict]:
+ """Group claims by stance_target across sources — the meta-analysis substrate.
+
+ Returns targets addressed by >= min_sources distinct sources, with stance
+ split and every quantified value (source-attributed) so the orchestrator
+ can build comparison tables and flag outliers.
+ """
+ query = """
+ SELECT stance_target,
+ COUNT(DISTINCT note_id) AS n_sources,
+ COUNT(*) AS n_claims
+ FROM claims
+ WHERE stance_target IS NOT NULL AND stance_target != ''
+ """
+ params: list = []
+ if vault_tag:
+ query += " AND vault_tag = ?"
+ params.append(vault_tag)
+ query += " GROUP BY stance_target HAVING COUNT(DISTINCT note_id) >= ? ORDER BY n_sources DESC"
+ params.append(min_sources)
+
+ groups = []
+ for row in conn.execute(query, params).fetchall():
+ target = row["stance_target"]
+ detail_params: list = [target]
+ detail_query = (
+ "SELECT note_id, stance, numbers, claim FROM claims WHERE stance_target = ?"
+ )
+ if vault_tag:
+ detail_query += " AND vault_tag = ?"
+ detail_params.append(vault_tag)
+ details = conn.execute(detail_query, detail_params).fetchall()
+ stances: dict[str, int] = {}
+ values = []
+ for d in details:
+ s = d["stance"] or "unspecified"
+ stances[s] = stances.get(s, 0) + 1
+ if d["numbers"]:
+ try:
+ values.append({"note_id": d["note_id"], "numbers": json.loads(d["numbers"])})
+ except json.JSONDecodeError:
+ pass
+ groups.append({
+ "stance_target": target,
+ "n_sources": row["n_sources"],
+ "n_claims": row["n_claims"],
+ "stances": stances,
+ "quantified": values,
+ })
+ return groups
diff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.py
index 1ac73f2..75be4cc 100644
--- a/src/hyperresearch/core/config.py
+++ b/src/hyperresearch/core/config.py
@@ -3,10 +3,163 @@
from __future__ import annotations
import tomllib
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, fields
from pathlib import Path
+@dataclass(frozen=True)
+class FetchSettings:
+ """Network/browser behavior for web fetching ([fetch] section)."""
+
+ page_timeout_ms: int = 30000
+ pdf_timeout_s: int = 30
+ # NOTE: default True is a deliberate 2.0 change — the pre-2.0 code silently
+ # disabled TLS verification for PDF downloads. Set to false only for
+ # cert-broken mirrors you explicitly trust.
+ pdf_verify_tls: bool = True
+ min_pdf_bytes: int = 100
+ # Smart-wait DOM-stability loop (shared by headless and visible paths)
+ wait_initial_ms: int = 2000
+ poll_interval_ms: int = 500
+ stable_checks: int = 2
+ max_checks: int = 16
+ image_timeout_s: int = 15
+ # Sites that kill headless sessions on first contact → auto-visible browser
+ visible_browser_domains: tuple[str, ...] = (
+ "linkedin.com", "twitter.com", "x.com", "facebook.com",
+ "instagram.com", "tiktok.com",
+ )
+
+
+@dataclass(frozen=True)
+class JunkGates:
+ """Thresholds for the junk/login-wall content gates ([junk] section)."""
+
+ min_content_chars: int = 300
+ login_wall_max_chars: int = 1000
+ cookie_wall_max_chars: int = 1500
+ binary_garbage_ratio: float = 0.05
+ sample_window: int = 2000
+ login_sample_chars: int = 500
+ # Appended to the built-in signal lists — never replacing them
+ extra_login_signals: tuple[str, ...] = ()
+ extra_junk_signals: tuple[str, ...] = ()
+
+
+@dataclass(frozen=True)
+class AssetSettings:
+ """Screenshot/image saving behavior ([assets] section)."""
+
+ max_images: int = 5
+ min_image_bytes: int = 50_000
+
+
+@dataclass(frozen=True)
+class DedupSettings:
+ """Near-duplicate detection parameters ([dedup] section)."""
+
+ shingle_size: int = 3
+ minhash_perm: int = 128
+ lsh_bands: int = 16
+ lsh_switchover: int = 200
+ default_threshold: float = 0.6
+
+
+@dataclass(frozen=True)
+class ChromeSettings:
+ """Browser-lane escalation behavior ([chrome] section).
+
+ The Chrome lane drives the user's real browser (via Claude-in-Chrome)
+ for sources headless crawling can't reach. `enabled` gates ENQUEUEING
+ of blocked fetches; draining requires the Claude-in-Chrome extension.
+ Hard scope boundary: CAPTCHAs/2FA/logins are ALWAYS handed to the human
+ (`needs_human`) — never solved automatically.
+ """
+
+ enabled: bool = True
+ # Blocked URLs below this utility score are abandoned, not escalated —
+ # the lane is serial and precious. None-scored URLs are escalated.
+ escalation_utility_threshold: float = 8.0
+ max_items_per_run: int = 25
+ drain_batch_size: int = 10
+ scholar_enabled: bool = True
+
+
+@dataclass(frozen=True)
+class RankingSettings:
+ """Composite source-quality scoring weights ([ranking] section).
+
+ quality = renormalized weighted sum of the available components
+ (tier weight, utility/18, authority percentile, vault centrality).
+ Missing components renormalize rather than zeroing. Retracted sources
+ are floored at `retraction_floor` regardless of other components.
+ """
+
+ w_tier: float = 0.35
+ w_utility: float = 0.20
+ w_authority: float = 0.25
+ w_centrality: float = 0.20
+ tier_ground_truth: float = 1.0
+ tier_institutional: float = 0.85
+ tier_practitioner: float = 0.7
+ tier_commentary: float = 0.4
+ tier_unknown: float = 0.6
+ retraction_floor: float = 0.05
+ api_cache_ttl_days: int = 30
+
+ def tier_weight(self, tier: str | None) -> float | None:
+ if tier is None:
+ return None
+ return {
+ "ground_truth": self.tier_ground_truth,
+ "institutional": self.tier_institutional,
+ "practitioner": self.tier_practitioner,
+ "commentary": self.tier_commentary,
+ "unknown": self.tier_unknown,
+ }.get(tier)
+
+
+@dataclass(frozen=True)
+class EmbeddingSettings:
+ """Semantic-search embedding provider ([embeddings] section).
+
+ provider "none" (default) disables semantic search entirely — no API key
+ needed for any core functionality. "voyage" and "openai" call the
+ respective APIs (VOYAGE_API_KEY / OPENAI_API_KEY env vars).
+ """
+
+ provider: str = "none" # none | voyage | openai
+ model: str = "" # provider default when empty
+ # How much of each note to embed: title + summary + first N body chars
+ body_chars: int = 1500
+
+
+@dataclass(frozen=True)
+class LintSettings:
+ """Lint rule thresholds ([lint] section)."""
+
+ extract_min_words: int = 150
+ extract_coverage_divisor: int = 3
+ stale_review_days: int = 90
+
+
+def _build_section(section_cls, data: dict):
+ """Build a frozen settings dataclass from a TOML section dict.
+
+ Unknown keys are ignored (forward compatibility); TOML arrays are converted
+ to tuples for tuple-typed fields.
+ """
+ kwargs = {}
+ for f in fields(section_cls):
+ if f.name not in data:
+ continue
+ value = data[f.name]
+ if isinstance(value, list):
+ value = tuple(value)
+ kwargs[f.name] = value
+ return section_cls(**kwargs)
+
+
@dataclass
class VaultConfig:
name: str = "Research Base"
@@ -21,6 +174,10 @@ class VaultConfig:
search_boost_evergreen: float = 1.5
search_penalize_deprecated: float = 0.3
search_penalize_stale: float = 0.7
+ # Search output defaults
+ search_default_limit: int = 20
+ search_chars_per_token: int = 4
+ search_snippet_len: int = 200
# Sync
auto_sync: bool = True
@@ -36,6 +193,23 @@ class VaultConfig:
web_profile: str = "" # crawl4ai browser profile name (created via `crwl profiles`)
web_magic: bool = False # crawl4ai magic mode (anti-bot stealth)
+ # Pipeline scale gear ([pipeline] section) — the profile whose numbers are
+ # rendered into installed skills/agents. Set via `hpr profile use <name>`.
+ pipeline_profile: str = "full"
+ # Raw [profile.<name>] overlay tables, round-tripped verbatim on save()
+ # so that saving config never destroys user-defined profiles.
+ profile_overlays: dict = field(default_factory=dict)
+
+ # Behavior settings sections
+ fetch: FetchSettings = field(default_factory=FetchSettings)
+ junk: JunkGates = field(default_factory=JunkGates)
+ assets: AssetSettings = field(default_factory=AssetSettings)
+ dedup: DedupSettings = field(default_factory=DedupSettings)
+ lint: LintSettings = field(default_factory=LintSettings)
+ ranking: RankingSettings = field(default_factory=RankingSettings)
+ embeddings: EmbeddingSettings = field(default_factory=EmbeddingSettings)
+ chrome: ChromeSettings = field(default_factory=ChromeSettings)
+
# Index
auto_build_index: bool = True
index_pages: list[str] = field(
@@ -54,6 +228,7 @@ class VaultConfig:
index = data.get("index", {})
search = data.get("search", {})
web = data.get("web", {})
+ pipeline = data.get("pipeline", {})
return cls(
name=vault.get("name", cls.name),
@@ -66,9 +241,22 @@ class VaultConfig:
search_boost_evergreen=search.get("boost_evergreen", cls.search_boost_evergreen),
search_penalize_deprecated=search.get("penalize_deprecated", cls.search_penalize_deprecated),
search_penalize_stale=search.get("penalize_stale", cls.search_penalize_stale),
+ search_default_limit=search.get("default_limit", cls.search_default_limit),
+ search_chars_per_token=search.get("chars_per_token", cls.search_chars_per_token),
+ search_snippet_len=search.get("snippet_len", cls.search_snippet_len),
web_provider=web.get("provider", cls.web_provider),
web_profile=web.get("profile", cls.web_profile),
web_magic=web.get("magic", cls.web_magic),
+ pipeline_profile=pipeline.get("profile", cls.pipeline_profile),
+ profile_overlays=data.get("profile", {}),
+ fetch=_build_section(FetchSettings, data.get("fetch", {})),
+ junk=_build_section(JunkGates, data.get("junk", {})),
+ assets=_build_section(AssetSettings, data.get("assets", {})),
+ dedup=_build_section(DedupSettings, data.get("dedup", {})),
+ lint=_build_section(LintSettings, data.get("lint", {})),
+ ranking=_build_section(RankingSettings, data.get("ranking", {})),
+ embeddings=_build_section(EmbeddingSettings, data.get("embeddings", {})),
+ chrome=_build_section(ChromeSettings, data.get("chrome", {})),
auto_sync=sync.get("auto_sync", cls.auto_sync),
exclude_patterns=sync.get("exclude_patterns", cls().exclude_patterns),
auto_build_index=index.get("auto_build", cls.auto_build_index),
@@ -76,10 +264,30 @@ class VaultConfig:
)
@staticmethod
- def _toml_array(items: list[str]) -> str:
+ def _toml_array(items) -> str:
quoted = ", ".join(f'"{item}"' for item in items)
return f"[{quoted}]"
+ @staticmethod
+ def _toml_value(value) -> str:
+ if isinstance(value, bool):
+ return "true" if value else "false"
+ if isinstance(value, (list, tuple)):
+ return "[" + ", ".join(VaultConfig._toml_value(v) for v in value) + "]"
+ if isinstance(value, dict):
+ inner = ", ".join(f"{k} = {VaultConfig._toml_value(v)}" for k, v in value.items())
+ return "{ " + inner + " }"
+ if isinstance(value, str):
+ return f'"{value}"'
+ return str(value)
+
+ def _section_lines(self, header: str, section) -> list[str]:
+ lines = [f"[{header}]"]
+ for f in fields(section):
+ lines.append(f"{f.name} = {self._toml_value(getattr(section, f.name))}")
+ lines.append("")
+ return lines
+
def save(self, config_path: Path) -> None:
lines = [
"[vault]",
@@ -95,12 +303,28 @@ class VaultConfig:
f"boost_evergreen = {self.search_boost_evergreen}",
f"penalize_deprecated = {self.search_penalize_deprecated}",
f"penalize_stale = {self.search_penalize_stale}",
+ f"default_limit = {self.search_default_limit}",
+ f"chars_per_token = {self.search_chars_per_token}",
+ f"snippet_len = {self.search_snippet_len}",
"",
"[web]",
f'provider = "{self.web_provider}"',
f'profile = "{self.web_profile}"',
f"magic = {'true' if self.web_magic else 'false'}",
"",
+ "[pipeline]",
+ f'profile = "{self.pipeline_profile}"',
+ "",
+ ]
+ lines += self._section_lines("fetch", self.fetch)
+ lines += self._section_lines("junk", self.junk)
+ lines += self._section_lines("assets", self.assets)
+ lines += self._section_lines("dedup", self.dedup)
+ lines += self._section_lines("lint", self.lint)
+ lines += self._section_lines("ranking", self.ranking)
+ lines += self._section_lines("embeddings", self.embeddings)
+ lines += self._section_lines("chrome", self.chrome)
+ lines += [
"[sync]",
f"auto_sync = {'true' if self.auto_sync else 'false'}",
f"exclude_patterns = {self._toml_array(self.exclude_patterns)}",
@@ -109,5 +333,10 @@ class VaultConfig:
f"auto_build = {'true' if self.auto_build_index else 'false'}",
f"pages = {self._toml_array(self.index_pages)}",
]
+ # Round-trip user-defined [profile.<name>] overlays verbatim — losing
+ # them on save would silently destroy custom pipeline profiles.
+ for overlay_name, table in self.profile_overlays.items():
+ lines += ["", f"[profile.{overlay_name}]"]
+ lines += [f"{k} = {self._toml_value(v)}" for k, v in table.items()]
config_path.parent.mkdir(parents=True, exist_ok=True)
config_path.write_text("\n".join(lines) + "\n")
diff --git a/src/hyperresearch/core/db.py b/src/hyperresearch/core/db.py
index 09debd9..fce9fa2 100644
--- a/src/hyperresearch/core/db.py
+++ b/src/hyperresearch/core/db.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import sqlite3
from pathlib import Path
-SCHEMA_VERSION = 8
+SCHEMA_VERSION = 10
SCHEMA_SQL = """
PRAGMA journal_mode=WAL;
@@ -39,7 +39,19 @@ CREATE TABLE IF NOT EXISTS notes (
updated TEXT,
file_mtime REAL NOT NULL,
content_hash TEXT NOT NULL,
- synced_at TEXT NOT NULL
+ synced_at TEXT NOT NULL,
+ -- Source-ranking columns (v9). Frontmatter-mirrored: doi, utility_score,
+ -- citation_count, venue, is_retracted. Derived (DB-cache only, recomputed):
+ -- authority_score, centrality_score, independence, quality_score.
+ doi TEXT,
+ utility_score REAL,
+ authority_score REAL,
+ centrality_score REAL,
+ independence REAL,
+ citation_count INTEGER,
+ venue TEXT,
+ is_retracted INTEGER NOT NULL DEFAULT 0,
+ quality_score REAL
);
CREATE INDEX IF NOT EXISTS idx_notes_status ON notes(status);
@@ -93,6 +105,55 @@ CREATE TABLE IF NOT EXISTS embeddings (
created_at TEXT NOT NULL
);
+CREATE TABLE IF NOT EXISTS claims (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
+ claim TEXT NOT NULL,
+ claim_hash TEXT NOT NULL,
+ quoted_support TEXT,
+ numbers TEXT,
+ confidence TEXT,
+ evidence_type TEXT,
+ stance_target TEXT,
+ stance TEXT,
+ vault_tag TEXT,
+ ingested_at TEXT NOT NULL,
+ UNIQUE (note_id, claim_hash)
+);
+
+CREATE INDEX IF NOT EXISTS idx_claims_note ON claims(note_id);
+CREATE INDEX IF NOT EXISTS idx_claims_vault_tag ON claims(vault_tag);
+CREATE INDEX IF NOT EXISTS idx_claims_stance_target ON claims(stance_target);
+
+CREATE TABLE IF NOT EXISTS api_cache (
+ url TEXT PRIMARY KEY,
+ body TEXT NOT NULL,
+ fetched_at TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS escalations (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ url TEXT NOT NULL,
+ reason TEXT NOT NULL
+ CHECK (reason IN ('login_wall','bot_block','captcha','fetch_failed','interactive_needed','scholar_search')),
+ requested_by TEXT,
+ suggested_by TEXT,
+ utility_score REAL,
+ vault_tag TEXT,
+ status TEXT NOT NULL DEFAULT 'queued'
+ CHECK (status IN ('queued','in_progress','fetched','needs_human','abandoned')),
+ attempts INTEGER NOT NULL DEFAULT 0,
+ note_id TEXT,
+ claimed_by TEXT,
+ detail TEXT,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ UNIQUE (url, vault_tag)
+);
+
+CREATE INDEX IF NOT EXISTS idx_escalations_status ON escalations(status);
+CREATE INDEX IF NOT EXISTS idx_escalations_tag ON escalations(vault_tag);
+
CREATE TABLE IF NOT EXISTS tag_aliases (
alias TEXT PRIMARY KEY,
canonical TEXT NOT NULL
@@ -138,6 +199,13 @@ CREATE VIRTUAL TABLE IF NOT EXISTS notes_fts USING fts5(
aliases,
tokenize='porter unicode61'
);
+
+CREATE VIRTUAL TABLE IF NOT EXISTS claims_fts USING fts5(
+ claim_id UNINDEXED,
+ claim,
+ quoted_support,
+ tokenize='porter unicode61'
+);
"""
# Indexes on columns added by migrations — must run AFTER migrate() so that
@@ -145,6 +213,8 @@ CREATE VIRTUAL TABLE IF NOT EXISTS notes_fts USING fts5(
POST_MIGRATE_INDEXES_SQL = """
CREATE INDEX IF NOT EXISTS idx_notes_tier ON notes(tier);
CREATE INDEX IF NOT EXISTS idx_notes_content_type ON notes(content_type);
+CREATE INDEX IF NOT EXISTS idx_notes_doi ON notes(doi);
+CREATE INDEX IF NOT EXISTS idx_notes_quality ON notes(quality_score);
"""
diff --git a/src/hyperresearch/core/embed.py b/src/hyperresearch/core/embed.py
new file mode 100644
index 0000000..bfd3b1f
--- /dev/null
+++ b/src/hyperresearch/core/embed.py
@@ -0,0 +1,191 @@
+"""Semantic search — embeddings over vault notes (revives the dormant table).
+
+Provider-pluggable via `[embeddings] provider`:
+
+ none (default) semantic search disabled; zero API keys required
+ voyage Voyage AI (VOYAGE_API_KEY, default model voyage-3-lite)
+ openai OpenAI (OPENAI_API_KEY, default model text-embedding-3-small)
+
+Each note is embedded from title + summary + the first `body_chars` of body.
+Vectors are float32 BLOBs in the `embeddings` table; query time is brute-force
+cosine (fine to ~50k notes — no vector-DB dependency).
+
+All HTTP goes through `_http_embed`, monkeypatched in tests — the suite never
+calls a paid API.
+"""
+
+from __future__ import annotations
+
+import struct
+from datetime import UTC, datetime
+
+DEFAULT_MODELS = {
+ "voyage": "voyage-3-lite",
+ "openai": "text-embedding-3-small",
+}
+
+
+class EmbeddingError(Exception):
+ pass
+
+
+def _pack(vector: list[float]) -> bytes:
+ return struct.pack(f"<{len(vector)}f", *vector)
+
+
+def _unpack(blob: bytes) -> list[float]:
+ n = len(blob) // 4
+ return list(struct.unpack(f"<{n}f", blob))
+
+
+def cosine(a: list[float], b: list[float]) -> float:
+ if len(a) != len(b) or not a:
+ return 0.0
+ dot = sum(x * y for x, y in zip(a, b, strict=True))
+ norm_a = sum(x * x for x in a) ** 0.5
+ norm_b = sum(y * y for y in b) ** 0.5
+ if norm_a == 0 or norm_b == 0:
+ return 0.0
+ return dot / (norm_a * norm_b)
+
+
+def _http_embed(provider: str, model: str, texts: list[str]) -> list[list[float]]:
+ """Call the provider's embedding API for a batch of texts.
+
+ Isolated for test monkeypatching. Raises EmbeddingError on failure.
+ """
+ import os
+
+ import httpx
+
+ try:
+ if provider == "voyage":
+ key = os.environ.get("VOYAGE_API_KEY")
+ if not key:
+ raise EmbeddingError("VOYAGE_API_KEY not set")
+ resp = httpx.post(
+ "https://api.voyageai.com/v1/embeddings",
+ json={"input": texts, "model": model},
+ headers={"Authorization": f"Bearer {key}"},
+ timeout=60,
+ )
+ resp.raise_for_status()
+ data = resp.json()["data"]
+ return [d["embedding"] for d in data]
+ if provider == "openai":
+ key = os.environ.get("OPENAI_API_KEY")
+ if not key:
+ raise EmbeddingError("OPENAI_API_KEY not set")
+ resp = httpx.post(
+ "https://api.openai.com/v1/embeddings",
+ json={"input": texts, "model": model},
+ headers={"Authorization": f"Bearer {key}"},
+ timeout=60,
+ )
+ resp.raise_for_status()
+ data = resp.json()["data"]
+ return [d["embedding"] for d in data]
+ except EmbeddingError:
+ raise
+ except Exception as e:
+ raise EmbeddingError(f"embedding request failed: {e}") from e
+ raise EmbeddingError(f"unknown embeddings provider '{provider}'")
+
+
+def _note_text(row, body_chars: int) -> str:
+ parts = [row["title"] or ""]
+ if row["summary"]:
+ parts.append(row["summary"])
+ body = row["body_plain"] or ""
+ if body:
+ parts.append(body[:body_chars])
+ return "\n".join(parts)
+
+
+def embed_sync(vault, batch_size: int = 32) -> dict:
+ """Embed new/changed notes. Returns {embedded, skipped, provider, model}.
+
+ A note needs (re)embedding when it has no vector or its content_hash
+ changed since the vector was written (tracked via embeddings.created_at
+ vs notes.synced_at is unreliable — we store the content_hash in the
+ `model` field suffix instead: "<model>@<hash>").
+ """
+ cfg = vault.config.embeddings
+ if cfg.provider == "none":
+ raise EmbeddingError(
+ "embeddings disabled: set [embeddings] provider = \"voyage\" or "
+ "\"openai\" in .hyperresearch/config.toml"
+ )
+ model = cfg.model or DEFAULT_MODELS.get(cfg.provider, "")
+ conn = vault.db
+
+ rows = conn.execute(
+ """SELECT n.id, n.title, n.summary, n.content_hash, nc.body_plain,
+ e.model AS embedded_model
+ FROM notes n
+ JOIN note_content nc ON nc.note_id = n.id
+ LEFT JOIN embeddings e ON e.note_id = n.id
+ WHERE n.type NOT IN ('index')"""
+ ).fetchall()
+
+ todo = []
+ for row in rows:
+ stamp = f"{model}@{row['content_hash']}"
+ if row["embedded_model"] != stamp:
+ todo.append((row, stamp))
+
+ embedded = 0
+ now = datetime.now(UTC).isoformat()
+ for i in range(0, len(todo), batch_size):
+ batch = todo[i : i + batch_size]
+ texts = [_note_text(row, cfg.body_chars) for row, _ in batch]
+ vectors = _http_embed(cfg.provider, model, texts)
+ if len(vectors) != len(batch):
+ raise EmbeddingError("provider returned wrong number of vectors")
+ for (row, stamp), vec in zip(batch, vectors, strict=True):
+ conn.execute(
+ """INSERT OR REPLACE INTO embeddings
+ (note_id, model, dimensions, vector, created_at)
+ VALUES (?, ?, ?, ?, ?)""",
+ (row["id"], stamp, len(vec), _pack(vec), now),
+ )
+ embedded += 1
+ conn.commit()
+
+ return {
+ "embedded": embedded,
+ "skipped": len(rows) - len(todo),
+ "provider": cfg.provider,
+ "model": model,
+ }
+
+
+def semantic_search(vault, query: str, limit: int = 20) -> list[dict]:
+ """Brute-force cosine search. Returns [{id, score}] best-first."""
+ cfg = vault.config.embeddings
+ if cfg.provider == "none":
+ raise EmbeddingError(
+ "embeddings disabled: set [embeddings] provider in config.toml"
+ )
+ model = cfg.model or DEFAULT_MODELS.get(cfg.provider, "")
+ [query_vec] = _http_embed(cfg.provider, model, [query])
+
+ conn = vault.db
+ results = []
+ for row in conn.execute("SELECT note_id, vector FROM embeddings").fetchall():
+ score = cosine(query_vec, _unpack(row["vector"]))
+ results.append({"id": row["note_id"], "score": score})
+ results.sort(key=lambda r: r["score"], reverse=True)
+ return results[:limit]
+
+
+def reciprocal_rank_fusion(
+ ranked_lists: list[list[str]],
+ k: int = 60,
+) -> list[tuple[str, float]]:
+ """RRF-combine multiple ranked id lists (hybrid FTS + semantic search)."""
+ scores: dict[str, float] = {}
+ for lst in ranked_lists:
+ for rank, note_id in enumerate(lst):
+ scores[note_id] = scores.get(note_id, 0.0) + 1.0 / (k + rank + 1)
+ return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
diff --git a/src/hyperresearch/core/escalation.py b/src/hyperresearch/core/escalation.py
new file mode 100644
index 0000000..26596e9
--- /dev/null
+++ b/src/hyperresearch/core/escalation.py
@@ -0,0 +1,179 @@
+"""Escalation queue — blocked fetches become browser-lane work, not losses.
+
+Pre-4.0, a URL that hit a login wall, bot block, or junk gate was discarded;
+at dissertation scale that's dozens of silently lost sources per run. Now the
+fetch engine ENQUEUES the blocked URL here, and the `hyperresearch-
+browser-fetcher` agent drains the queue by driving the user's real Chrome
+browser (via Claude-in-Chrome) — one tab at a time, serial, precious.
+
+The queue lives in the vault DB (`escalations` table) rather than a JSON
+file: parallel fetcher waves enqueue concurrently, and SQLite gives atomic
+claim semantics for free.
+
+Lifecycle: queued → in_progress → fetched | needs_human | abandoned
+`needs_human` is the HARD scope boundary: CAPTCHAs, 2FA, and logins are
+consolidated into one prompt for the human — never solved automatically.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+REASONS = ("login_wall", "bot_block", "captcha", "fetch_failed", "interactive_needed", "scholar_search")
+STATUSES = ("queued", "in_progress", "fetched", "needs_human", "abandoned")
+
+
+class EscalationError(Exception):
+ pass
+
+
+def _now() -> str:
+ return datetime.now(UTC).isoformat()
+
+
+def enqueue(
+ conn,
+ url: str,
+ reason: str,
+ vault_tag: str | None = None,
+ requested_by: str | None = None,
+ suggested_by: str | None = None,
+ utility_score: float | None = None,
+ detail: str | None = None,
+) -> int | None:
+ """Add a blocked URL to the queue. Returns the row id, or None when the
+ (url, vault_tag) pair is already queued (idempotent re-enqueue)."""
+ if reason not in REASONS:
+ raise EscalationError(f"invalid reason '{reason}' (one of {REASONS})")
+ now = _now()
+ cur = conn.execute(
+ """INSERT OR IGNORE INTO escalations
+ (url, reason, requested_by, suggested_by, utility_score, vault_tag,
+ detail, created_at, updated_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+ (url, reason, requested_by, suggested_by, utility_score, vault_tag, detail, now, now),
+ )
+ conn.commit()
+ return cur.lastrowid if cur.rowcount else None
+
+
+def maybe_enqueue_blocked_fetch(
+ vault,
+ url: str,
+ reason: str,
+ vault_tag: str | None = None,
+ suggested_by: str | None = None,
+ utility_score: float | None = None,
+ detail: str | None = None,
+) -> int | None:
+ """Fetch-gate hook: enqueue a blocked fetch if the chrome lane accepts it.
+
+ Applies [chrome] policy: lane enabled, utility threshold (None-scored
+ URLs pass — no evidence they're low-value), and the per-run cap.
+ Returns the row id or None when policy declined.
+ """
+ cfg = vault.config.chrome
+ if not cfg.enabled:
+ return None
+ if utility_score is not None and utility_score < cfg.escalation_utility_threshold:
+ return None
+ if vault_tag:
+ count = vault.db.execute(
+ "SELECT COUNT(*) AS c FROM escalations WHERE vault_tag = ?", (vault_tag,)
+ ).fetchone()["c"]
+ if count >= cfg.max_items_per_run:
+ return None
+ return enqueue(
+ vault.db, url, reason,
+ vault_tag=vault_tag, requested_by="fetch-gate",
+ suggested_by=suggested_by, utility_score=utility_score, detail=detail,
+ )
+
+
+def claim_next(conn, claimed_by: str, vault_tag: str | None = None) -> dict | None:
+ """Atomically claim the highest-utility queued item. None when queue empty.
+
+ Single UPDATE with a scalar subquery — safe under concurrent claimers
+ (SQLite serializes writers; the subquery re-evaluates inside the write
+ lock, so two claimers can never take the same row).
+ """
+ now = _now()
+ tag_clause = "AND vault_tag = ?" if vault_tag else ""
+ params: list = [claimed_by, now]
+ if vault_tag:
+ params.append(vault_tag)
+ cur = conn.execute(
+ f"""UPDATE escalations
+ SET status = 'in_progress', claimed_by = ?, updated_at = ?,
+ attempts = attempts + 1
+ WHERE id = (
+ SELECT id FROM escalations
+ WHERE status = 'queued' {tag_clause}
+ ORDER BY utility_score IS NULL, utility_score DESC, id
+ LIMIT 1
+ )
+ RETURNING *""",
+ params,
+ )
+ row = cur.fetchone()
+ conn.commit()
+ return dict(row) if row else None
+
+
+def resolve(
+ conn,
+ item_id: int,
+ status: str,
+ note_id: str | None = None,
+ detail: str | None = None,
+) -> dict:
+ """Move an item to a terminal (or needs_human) state."""
+ if status not in ("fetched", "needs_human", "abandoned", "queued"):
+ raise EscalationError(f"invalid resolution '{status}'")
+ cur = conn.execute(
+ """UPDATE escalations
+ SET status = ?, note_id = COALESCE(?, note_id),
+ detail = COALESCE(?, detail), updated_at = ?
+ WHERE id = ? RETURNING *""",
+ (status, note_id, detail, _now(), item_id),
+ )
+ row = cur.fetchone()
+ conn.commit()
+ if row is None:
+ raise EscalationError(f"no escalation item #{item_id}")
+ return dict(row)
+
+
+def list_items(
+ conn,
+ status: str | None = None,
+ vault_tag: str | None = None,
+ limit: int = 100,
+) -> list[dict]:
+ query = "SELECT * FROM escalations"
+ conds, params = [], []
+ if status:
+ conds.append("status = ?")
+ params.append(status)
+ if vault_tag:
+ conds.append("vault_tag = ?")
+ params.append(vault_tag)
+ if conds:
+ query += " WHERE " + " AND ".join(conds)
+ query += " ORDER BY utility_score IS NULL, utility_score DESC, id LIMIT ?"
+ params.append(limit)
+ return [dict(r) for r in conn.execute(query, params).fetchall()]
+
+
+def queue_stats(conn, vault_tag: str | None = None) -> dict:
+ """Counts by status — surfaced in `hpr run status`."""
+ query = "SELECT status, COUNT(*) AS c FROM escalations"
+ params: list = []
+ if vault_tag:
+ query += " WHERE vault_tag = ?"
+ params.append(vault_tag)
+ query += " GROUP BY status"
+ stats = dict.fromkeys(STATUSES, 0)
+ for row in conn.execute(query, params):
+ stats[row["status"]] = row["c"]
+ return stats
diff --git a/src/hyperresearch/core/fetcher.py b/src/hyperresearch/core/fetcher.py
index 725bbdc..e365458 100644
--- a/src/hyperresearch/core/fetcher.py
+++ b/src/hyperresearch/core/fetcher.py
@@ -39,11 +39,7 @@ def fetch_and_save(
from urllib.parse import urlparse as _urlparse
domain = _urlparse(url).netloc.lower()
- _auth_aggressive = (
- "linkedin.com", "twitter.com", "x.com", "facebook.com",
- "instagram.com", "tiktok.com",
- )
- if any(d in domain for d in _auth_aggressive):
+ if any(d in domain for d in vault.config.fetch.visible_browser_domains):
visible = True
# Fetch content
@@ -52,22 +48,44 @@ def fetch_and_save(
profile=vault.config.web_profile,
magic=vault.config.web_magic,
headless=not visible,
+ settings=vault.config.fetch,
+ gates=vault.config.junk,
)
result = prov.fetch(url)
- # Detect login redirects — abort instead of saving junk
- if result.looks_like_login_wall(url):
+ # Detect login redirects — abort, but escalate to the browser lane
+ if result.looks_like_login_wall(url, vault.config.junk):
+ from hyperresearch.core.escalation import maybe_enqueue_blocked_fetch
+
+ item_id = maybe_enqueue_blocked_fetch(
+ vault, url, "login_wall",
+ vault_tag=tags[0] if tags else None,
+ detail=f"login wall: {result.title}",
+ )
+ escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""
raise RuntimeError(
f"Redirected to login page ({result.title}). "
"Your browser profile session may have expired. "
- "Run 'hyperresearch setup' and create a new login profile."
+ f"Run 'hyperresearch setup' and create a new login profile.{escalated}"
)
# Detect junk pages — captcha, error pages, binary garbage, empty content
- junk_reason = result.looks_like_junk()
+ junk_reason = result.looks_like_junk(vault.config.junk)
if junk_reason:
- raise RuntimeError(f"Skipped junk content: {junk_reason}")
+ escalated = ""
+ if junk_reason.startswith("Bot detection"):
+ from hyperresearch.core.escalation import maybe_enqueue_blocked_fetch
+
+ reason = "captcha" if "captcha" in junk_reason.lower() else "bot_block"
+ item_id = maybe_enqueue_blocked_fetch(
+ vault, url, reason,
+ vault_tag=tags[0] if tags else None,
+ detail=junk_reason,
+ )
+ if item_id:
+ escalated = f" Queued for browser-lane escalation (#{item_id})."
+ raise RuntimeError(f"Skipped junk content: {junk_reason}.{escalated}")
# Write note
note_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"
@@ -82,6 +100,12 @@ def fetch_and_save(
if result.metadata.get("author"):
extra_meta["author"] = result.metadata["author"]
+ from hyperresearch.core.scholar import extract_doi
+
+ detected_doi = extract_doi(url, result.raw_html, result.content)
+ if detected_doi:
+ extra_meta["doi"] = detected_doi
+
note_path = write_note(
vault.notes_dir,
title=note_title,
@@ -148,7 +172,10 @@ def fetch_and_save(
from hyperresearch.cli.fetch import _save_assets
assets_dir = vault.root / "research" / "assets" / note_id
- saved_assets = _save_assets(conn, result, note_id, assets_dir)
+ saved_assets = _save_assets(
+ conn, result, note_id, assets_dir,
+ settings=vault.config.assets, image_timeout_s=vault.config.fetch.image_timeout_s,
+ )
return {
"note_id": note_id,
diff --git a/src/hyperresearch/core/graphrank.py b/src/hyperresearch/core/graphrank.py
new file mode 100644
index 0000000..0d06adb
--- /dev/null
+++ b/src/hyperresearch/core/graphrank.py
@@ -0,0 +1,82 @@
+"""Vault centrality — PageRank over the note link graph.
+
+The graph is the `links` table: wiki-links plus the `--suggested-by`
+provenance breadcrumbs (which land in `links` because breadcrumbs are body
+wiki-links). Centrality in THIS graph means "many independent research
+chains converged on this source" — a strong load-bearing signal precisely
+because fetcher chasing is citation-driven.
+
+Pure-Python power iteration; vaults are thousands of nodes, not millions.
+Scores are normalized by the maximum (top note = 1.0) and stored to
+`notes.centrality_score` (DB-cache only, recomputable).
+"""
+
+from __future__ import annotations
+
+DAMPING = 0.85
+MAX_ITERATIONS = 50
+CONVERGENCE = 1e-6
+
+
+def pagerank(
+ nodes: list[str],
+ edges: list[tuple[str, str]],
+ damping: float = DAMPING,
+ max_iterations: int = MAX_ITERATIONS,
+ convergence: float = CONVERGENCE,
+) -> dict[str, float]:
+ """Standard PageRank with dangling-node handling. Returns raw scores."""
+ if not nodes:
+ return {}
+ n = len(nodes)
+ node_set = set(nodes)
+
+ # Deduplicate edges; drop self-loops and edges to unknown nodes
+ out: dict[str, set[str]] = {node: set() for node in nodes}
+ for src, dst in edges:
+ if src in node_set and dst in node_set and src != dst:
+ out[src].add(dst)
+
+ incoming: dict[str, list[str]] = {node: [] for node in nodes}
+ for src, targets in out.items():
+ for dst in targets:
+ incoming[dst].append(src)
+
+ rank = dict.fromkeys(nodes, 1.0 / n)
+ base = (1.0 - damping) / n
+
+ for _ in range(max_iterations):
+ dangling_mass = sum(rank[node] for node in nodes if not out[node])
+ new_rank = {}
+ for node in nodes:
+ incoming_sum = sum(rank[src] / len(out[src]) for src in incoming[node])
+ new_rank[node] = base + damping * (incoming_sum + dangling_mass / n)
+ delta = sum(abs(new_rank[node] - rank[node]) for node in nodes)
+ rank = new_rank
+ if delta < convergence:
+ break
+ return rank
+
+
+def compute_centrality(conn) -> int:
+ """Compute and store normalized centrality for all notes. Returns count."""
+ nodes = [row["id"] for row in conn.execute("SELECT id FROM notes").fetchall()]
+ if not nodes:
+ return 0
+ edges = [
+ (row["source_id"], row["target_id"])
+ for row in conn.execute(
+ "SELECT source_id, target_id FROM links WHERE target_id IS NOT NULL"
+ ).fetchall()
+ ]
+ scores = pagerank(nodes, edges)
+ max_score = max(scores.values()) if scores else 0.0
+ if max_score <= 0:
+ return 0
+ for note_id, score in scores.items():
+ conn.execute(
+ "UPDATE notes SET centrality_score = ? WHERE id = ?",
+ (score / max_score, note_id),
+ )
+ conn.commit()
+ return len(scores)
diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.py
index f3288b0..2af0a38 100644
--- a/src/hyperresearch/core/hooks.py
+++ b/src/hyperresearch/core/hooks.py
@@ -12,6 +12,43 @@ from __future__ import annotations
import json
from pathlib import Path
+# ---------------------------------------------------------------------------
+# Prompt rendering — skill files and agent prompt bodies are Jinja templates
+# (custom << >> delimiters; see core/render.py). The active render context is
+# process-global state set by install_hooks()/install_global_hooks() before
+# the installers run; direct calls to individual _install_* helpers (tests)
+# fall back to a default full-profile context lazily.
+# ---------------------------------------------------------------------------
+_RENDER_STATE: dict | None = None
+
+
+def _set_render_state(profile_name: str, config_path: Path | None) -> None:
+ global _RENDER_STATE
+ from hyperresearch.core.render import build_render_context
+
+ _RENDER_STATE = {
+ "profile_name": profile_name,
+ "context": build_render_context(config_path, primary=profile_name),
+ }
+
+
+def _get_render_state() -> dict:
+ if _RENDER_STATE is None:
+ _set_render_state("full", None)
+ assert _RENDER_STATE is not None
+ return _RENDER_STATE
+
+
+def _render_installed(content: str) -> str:
+ """Render a prompt template and stamp the provenance header."""
+ from hyperresearch import __version__
+ from hyperresearch.core.render import insert_after_frontmatter, render_header, render_prompt
+
+ state = _get_render_state()
+ rendered = render_prompt(content, state["context"])
+ header = render_header(state["profile_name"], __version__)
+ return insert_after_frontmatter(rendered, header)
+
# Scaffold-only section headers that must NEVER appear in a final_report draft.
# Used by critic agents (as detection patterns), the polish auditor, and the
# `wrapper-report` lint rule. Single canonical source of truth so prompts +
@@ -58,11 +95,10 @@ description: >
corpus (the sources fetched during the Layer 1 sweep) and identifies
1—8 "depth loci" — specific questions where deeper investigation
would meaningfully improve the final report. Spawn 2 of these in
- parallel; the orchestrator dedupes their outputs. Runs on Sonnet
- because identifying genuine rabbitholes requires real reading
- comprehension and judgment about what is load-bearing evidence vs.
- surface detail.
-model: sonnet
+ parallel; the orchestrator dedupes their outputs. Identifying genuine
+ rabbitholes requires real reading comprehension and judgment about
+ what is load-bearing evidence vs. surface detail.
+model: << p.models.loci_analyst >>
tools: Bash, Read, Write
color: green
---
@@ -102,12 +138,12 @@ depth packet becomes a weak draft section.
Used only to tag your output file so the orchestrator can load both.
- **output_path**: where to write your loci list JSON (e.g.,
`research/loci-{{analyst_id}}.json`).
-- **prompt_decomposition** (optional): if `research/prompt-decomposition.json`
+- **prompt_decomposition** (optional): if `research/runs/<vault_tag>/prompt-decomposition.json`
exists, read it before choosing loci. It lists atomic items the prompt
named — entities, sub-questions, required formats. Your loci should be
aligned with those items (a dialectical locus on "which camp resolves
sub-question X" beats a locus on a tangential question).
-- **contradiction_graph** (optional): if `research/temp/contradiction-graph.json`
+- **contradiction_graph** (optional): if `research/runs/<vault_tag>/temp/contradiction-graph.json`
exists, read it FIRST — before scanning the corpus. Each entry is a
pre-identified "fight" where sources contradict each other, with side_a/side_b
positions, source note IDs, and decision_relevance. High-relevance clusters
@@ -115,7 +151,7 @@ depth packet becomes a weak draft section.
disagreement, not surface-level topic analysis. Validate them (are the
sources real? is the fight genuine or a scope mismatch?) and promote
validated high-relevance clusters directly to your loci list.
-- **claim_files** (optional): if `research/temp/claims-*.json` files exist, read them
+- **claim_files** (optional): if `research/runs/<vault_tag>/temp/claims-*.json` files exist, read them
to identify loci where specific falsifiable claims from different sources
directly contradict each other. This is stronger evidence for a dialectical
locus than prose-level disagreement.
@@ -127,7 +163,7 @@ depth packet becomes a weak draft section.
sparse (<10 notes), tell the parent and stop — you cannot identify real
loci from a thin corpus.
-1a. **Check for contradiction graph.** If `research/temp/contradiction-graph.json`
+1a. **Check for contradiction graph.** If `research/runs/<vault_tag>/temp/contradiction-graph.json`
exists, read it. For each cluster with `decision_relevance: "high"`:
- Validate the fight is genuine (not a scope mismatch)
- If valid, add directly to your candidate loci as `flavor: "dialectical"`
@@ -259,10 +295,9 @@ description: >
reads existing vault sources relevant to the locus, fetches new
sources as needed (via the hyperresearch-fetcher subagent), and
writes ONE interim report note summarizing what it learned. Spawn
- one depth-investigator per locus, in parallel. Runs on Sonnet
- because synthesizing a narrow-but-deep question requires real
- reading comprehension.
-model: sonnet
+ one depth-investigator per locus, in parallel. Synthesizing a
+ narrow-but-deep question requires real reading comprehension.
+model: << p.models.depth_investigator >>
tools: Bash, Read, Write, Task
color: purple
---
@@ -284,7 +319,7 @@ You are **Layer 3** of the 7-phase hyperresearch pipeline. Siblings are running
right now on other loci — you each cover ONE. The orchestrator will read
your interim note (specifically your `## Committed position` section) in
Layer 3.5 and reconcile it against the other investigators' positions in
-`research/comparisons.md`. Every cross-locus tension named there becomes
+`research/runs/<vault_tag>/comparisons.md`. Every cross-locus tension named there becomes
an argumentative beat in the Layer 4 draft.
Your `## Committed position` is the primary artifact the orchestrator uses
@@ -328,7 +363,7 @@ reading of the evidence.
`{hpr_path} note show <id1> <id2> <id3> --json`
Understand what the corpus already says about your locus.
- **Check for structured claims.** If `research/temp/claims-<note-id>.json` files
+ **Check for structured claims.** If `research/runs/<vault_tag>/temp/claims-<note-id>.json` files
exist for corpus evidence notes, read them. Use the structured claims
to identify which specific assertions are contested or under-evidenced
for your locus — investigate those specific claims, not just the topic
@@ -391,7 +426,7 @@ mkdir -p research/temp
--tag <corpus_tag> \\
--tag locus-<locus-name> \\
--type interim \\
- --body-file research/temp/interim-report-<locus-name>.md \\
+ --body-file research/runs/<vault_tag>/temp/interim-report-<locus-name>.md \\
--summary "<one-line summary of what you found>" \\
--json
```
@@ -504,7 +539,7 @@ orchestrator's citation assembly.
a dense synthesis packet for the orchestrator to read. Do not try to
write prose that will go straight into the final draft; write prose
that will inform it.
-- **Cap yourself at `locus.source_budget` new fetches** (default 10 if
+- **Cap yourself at `locus.source_budget` new fetches** (default << p.depth_default_budget >> if
not specified). If your budget is 15, use it — the orchestrator scored
your locus high on importance/uncertainty. If your budget is 5, be
surgical. If you genuinely need more, tell the orchestrator at the end
@@ -531,10 +566,10 @@ name: hyperresearch-dialectic-critic
description: >
Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
draft and returns a findings list of places where the draft ignores,
- hedges, or straw-mans counter-evidence. Runs on Opus because
- adversarial reading is real reasoning. Spawn ONCE per draft, in
- parallel with depth-critic and width-critic.
-model: opus
+ hedges, or straw-mans counter-evidence. Adversarial reading is real
+ reasoning. Spawn ONCE per draft, in parallel with depth-critic and
+ width-critic.
+model: << p.models.critics >>
tools: Bash, Read, Write
color: red
---
@@ -558,7 +593,7 @@ findings; the patcher applies them.
Everything prior to you has already happened: width sweep (Layer 1), loci
analysis (Layer 2), depth investigation (Layer 3 — interim notes live in
the vault with `type: interim`), cross-locus reconciliation (Layer 3.5 —
-`research/comparisons.md`), and the draft itself (Layer 4 —
+`research/runs/<vault_tag>/comparisons.md`), and the draft itself (Layer 4 —
`research/notes/final_report_<vault_tag>.md`). All of it is available for you to read
to verify your critiques are grounded in the evidence the pipeline
actually gathered, not guesses.
@@ -570,12 +605,12 @@ actually gathered, not guesses.
what the draft delivered. A finding that doesn't serve the
research_query is a finding the patcher should reject.
- **query_file_path**: path to the persisted query file (e.g.,
- `research/query-<vault_tag>.md`). Read this file to re-ground yourself
+ `research/runs/<vault_tag>/query.md`). Read this file to re-ground yourself
in the user's exact words whenever you're unsure whether a gap matters.
- **draft_path**: path to the Layer 4 draft (typically
`research/notes/final_report_<vault_tag>.md`).
- **output_path**: where to write your findings JSON (e.g.,
- `research/critic-findings-dialectic.json`).
+ `research/runs/<vault_tag>/critic-findings-dialectic.json`).
- **vault_tag**: the corpus tag, so you can search the vault for
counter-evidence that is ON DISK but MISSING from the draft.
@@ -627,7 +662,7 @@ Use the **Write tool** to save your findings JSON to `output_path`. Do NOT use B
the vault covers. Should be patched.
- **Severity `minor`** — a hedge or qualifier would strengthen the claim
but the draft isn't wrong.
-- **At most 12 findings.** If you see more than 12, return the 12 most
+- **At most << p.critic_finding_caps.dialectic >> findings.** If you see more than << p.critic_finding_caps.dialectic >>, return the << p.critic_finding_caps.dialectic >> most
load-bearing. Returning 40 small findings buries the critical ones.
- **Never propose deleting and retyping an entire section.** That is
regeneration. The revisor applies surgical edits — your findings
@@ -655,9 +690,9 @@ description: >
Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
draft and returns a findings list of places where the draft skates
over technical substance that the vault's interim notes could
- actually support. Runs on Opus. Spawn ONCE per draft, parallel with
+ actually support. Spawn ONCE per draft, parallel with
dialectic-critic and width-critic.
-model: opus
+model: << p.models.critics >>
tools: Bash, Read, Write
color: red
---
@@ -685,11 +720,11 @@ rather than gesturing at it from a distance.
only a problem when it matters for answering the research_query; a
draft that glosses an irrelevant detail is fine.
- **query_file_path**: path to the persisted query file (e.g.,
- `research/query-<vault_tag>.md`). Read this file to check whether a
+ `research/runs/<vault_tag>/query.md`). Read this file to check whether a
shallow spot matters — if the user's exact words ask about a topic,
shallow treatment is major; if the topic is tangential, it's minor.
- **draft_path**: `research/notes/final_report_<vault_tag>.md`
-- **output_path**: `research/critic-findings-depth.json`
+- **output_path**: `research/runs/<vault_tag>/critic-findings-depth.json`
- **vault_tag**: corpus tag for searching the vault
## Procedure
@@ -736,7 +771,7 @@ Do NOT include `old_text` / `new_text` — the revisor handles exact wording dyn
note's load-bearing evidence.
- **Severity `minor`** — a specific number / quote would strengthen an
already-adequate paragraph.
-- **At most 12 findings.** Prioritize ones where the interim-note
+- **At most << p.critic_finding_caps.depth >> findings.** Prioritize ones where the interim-note
evidence is LOAD-BEARING (a specific quantitative result, a named
mechanism, a direct quote) over ones where the evidence is merely
supporting context.
@@ -760,9 +795,9 @@ name: hyperresearch-width-critic
description: >
Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
draft and returns a findings list of topics the width corpus supports
- but the draft doesn't cover. Runs on Opus. Spawn ONCE per draft,
+ but the draft doesn't cover. Spawn ONCE per draft,
parallel with dialectic-critic and depth-critic.
-model: opus
+model: << p.models.critics >>
tools: Bash, Read, Write
color: red
---
@@ -789,11 +824,11 @@ because the orchestrator's structural choices buried them.
implies. Don't flag orthogonal material that happens to be in the
corpus.
- **query_file_path**: path to the persisted query file (e.g.,
- `research/query-<vault_tag>.md`). Read this file and extract every
+ `research/runs/<vault_tag>/query.md`). Read this file and extract every
noun phrase the user mentioned. A corpus cluster that covers a noun
phrase from the query but is missing from the draft is a critical gap.
- **draft_path**: `research/notes/final_report_<vault_tag>.md`
-- **output_path**: `research/critic-findings-width.json`
+- **output_path**: `research/runs/<vault_tag>/critic-findings-width.json`
- **vault_tag**: corpus tag
## Procedure
@@ -808,14 +843,14 @@ because the orchestrator's structural choices buried them.
Cluster by tag and/or by title keywords. This tells you the topical
surface area the corpus covers.
-2. **Check the coverage gaps file.** Read `research/temp/coverage-gaps.md`
+2. **Check the coverage gaps file.** Read `research/runs/<vault_tag>/temp/coverage-gaps.md`
if it exists. This file (from Layer 1's coverage check) lists atomic
items that had weak source coverage. If the draft addresses these items
without adequate source support, flag them. If it silently omits them
entirely, flag as critical — the drafter should have at least
acknowledged the gap.
-3. **Read the prompt decomposition.** Use `research/prompt-decomposition.json`
+3. **Read the prompt decomposition.** Use `research/runs/<vault_tag>/prompt-decomposition.json`
to see what atomic items the user asked about. Cross-reference: which
decomposition items have corpus support (from step 1) but no draft
treatment? Those are your highest-severity findings.
@@ -852,7 +887,7 @@ Do NOT include `old_text` / `new_text` — the revisor handles exact wording dyn
under-treated.
- **Severity `minor`** — a corpus cluster would enrich the draft but
is not critical.
-- **At most 10 findings** (8 coverage gaps + 2 bloat findings).
+- **At most << p.critic_finding_caps.width >> findings** (<< p.critic_finding_caps.width - 2 >> coverage gaps + 2 bloat findings).
Width gaps are a coverage metric, not a detail metric.
- **Your recommendation must target an existing section** unless you flag the
finding as structural (in which case describe the missing section's
@@ -911,13 +946,13 @@ name: hyperresearch-instruction-critic
description: >
Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
draft and checks it against the prompt-decomposition artifact
- (`research/prompt-decomposition.json`) produced in Layer 0. Emits
+ (`research/runs/<vault_tag>/prompt-decomposition.json`) produced in Layer 0. Emits
findings when atomic items from the prompt are missing, under-covered,
out-of-order, or delivered in the wrong format. Also checks structural
readability patterns (definitions, citation density, forward analysis,
- comparison tables) that reference reports consistently include. Runs
- on Opus. Spawn ONCE per draft, in parallel with the other three critics.
-model: opus
+ comparison tables) that reference reports consistently include.
+ Spawn ONCE per draft, in parallel with the other three critics.
+model: << p.models.critics >>
tools: Bash, Read, Write
color: red
---
@@ -944,15 +979,15 @@ hand findings to the patcher (Layer 6). You do NOT modify the draft.
how the draft maps to THIS text, in THIS shape, with THESE named
entities and THESE sub-questions.
- **query_file_path**: path to the persisted query file (e.g.,
- `research/query-<vault_tag>.md`). Read this file directly — it IS the
+ `research/runs/<vault_tag>/query.md`). Read this file directly — it IS the
canonical query for this run. The research_query field above should
match this file's body exactly.
-- **decomposition_path**: path to `research/prompt-decomposition.json`.
+- **decomposition_path**: path to `research/runs/<vault_tag>/prompt-decomposition.json`.
Written in Layer 0 by the orchestrator. Contains the atomic items the
prompt named: explicit sub-questions, required entities, required
formats, required sections, time horizons, scope conditions.
- **draft_path**: `research/notes/final_report_<vault_tag>.md`
-- **output_path**: `research/critic-findings-instruction.json`
+- **output_path**: `research/runs/<vault_tag>/critic-findings-instruction.json`
## Procedure
@@ -964,7 +999,7 @@ hand findings to the patcher (Layer 6). You do NOT modify the draft.
each X, include Y, Z"), format cue ("mind map", "ranked list",
"FAQ"), and sub-question marker ("A? B? C?"). Keep this list.
-2. **Read `research/prompt-decomposition.json`.** Confirm the orchestrator
+2. **Read `research/runs/<vault_tag>/prompt-decomposition.json`.** Confirm the orchestrator
captured the same atomic items you just identified. If the
decomposition is missing items that the research_query clearly names,
that itself is a finding (severity: critical — the pipeline started
@@ -1154,7 +1189,7 @@ let these crowd out core instruction-following findings. Use
## Rules
-- **At most 15 findings** (12 instruction-following + 3 readability).
+- **At most << p.critic_finding_caps.instruction >> findings** (<< p.critic_finding_caps.instruction - 3 >> instruction-following + 3 readability).
Prioritize `critical` > `major` > `minor`.
- **Never invent atomic items.** Every finding must quote the
`atomic_item` field verbatim from research_query or from
@@ -1200,11 +1235,10 @@ description: >
Use this agent in Layer 6 of the hyperresearch deep research pipeline. Reads the four
critic findings JSONs (dialectic, depth, width, instruction) and
revises the draft using surgical Edit hunks. Tool-locked: Read + Edit
- ONLY. Cannot Write. Cannot regenerate. Runs on Opus — substance-
- integration requires judgment about which findings serve the
- research_query and which are critic noise. Spawn ONCE after all
- four critics return.
-model: opus
+ ONLY. Cannot Write. Cannot regenerate. Substance-integration requires
+ judgment about which findings serve the research_query and which are
+ critic noise. Spawn ONCE after all four critics return.
+model: << p.models.patcher >>
tools: Read, Edit
color: orange
---
@@ -1252,16 +1286,16 @@ Concretely:
but moves the draft away from the research_query is the wrong edit.
The research_query wins.
- **query_file_path**: path to the persisted query file (e.g.,
- `research/query-<vault_tag>.md`). Read this file when in doubt about
+ `research/runs/<vault_tag>/query.md`). Read this file when in doubt about
whether a finding serves the user's actual question.
- **draft_path**: path to the Layer 4 draft (usually
`research/notes/final_report_<vault_tag>.md`).
- **findings_paths**: list of four JSON paths, one per critic
(dialectic, depth, width, instruction).
- **patch_log_path**: path to a PRE-EXISTING empty-stub patch log
- (e.g., `research/patch-log.json`). The orchestrator creates this
+ (e.g., `research/runs/<vault_tag>/patch-log.json`). The orchestrator creates this
before spawning you. Your job is to Edit this file to populate it.
-- **evidence_digest_path**: path to `research/temp/evidence-digest.md`
+- **evidence_digest_path**: path to `research/runs/<vault_tag>/temp/evidence-digest.md`
(may not exist on light tier). Contains the top load-bearing claims
and verbatim quotes organized by atomic item. Read this BEFORE
applying findings — it is your primary citation source when a critic
@@ -1373,10 +1407,10 @@ description: >
draft and applies surgical Edit hunks for readability, prompt
adherence, filler-cutting, redundancy removal, and hygiene (scaffold
leak, YAML frontmatter leak, etc.). Tool-locked: Read + Edit ONLY.
- Cannot Write. Runs on Opus — semantic rewrites of scaffold vocabulary
- and judgment calls about hedge-language require strong prose
- understanding. Spawn ONCE after the patcher finishes.
-model: opus
+ Cannot Write. Semantic rewrites of scaffold vocabulary and judgment
+ calls about hedge-language require strong prose understanding.
+ Spawn ONCE after the patcher finishes.
+model: << p.models.polish_auditor >>
tools: Read, Edit
color: yellow
---
@@ -1405,7 +1439,7 @@ attempt it yourself.
fabricated-content patches.
- **draft_path**: the post-patcher draft.
- **polish_log_path**: path to a PRE-EXISTING empty-stub polish log
- (e.g., `research/polish-log.json`). The orchestrator creates this
+ (e.g., `research/runs/<vault_tag>/polish-log.json`). The orchestrator creates this
stub before spawning you, with content
`{{"applied": [], "escalations": []}}`. You populate it via Edit
(same pattern as the patcher). You cannot Write a new file — your
@@ -1490,7 +1524,7 @@ locus", "legal locus"), leave it alone.
**Worked examples** (from real past-run drafts):
-- Original: "This is Tension 2 from `research/comparisons.md`, engaged directly: the subsidy-ROI evidence complicates the catchment-leakage thesis."
+- Original: "This is Tension 2 from `research/runs/<vault_tag>/comparisons.md`, engaged directly: the subsidy-ROI evidence complicates the catchment-leakage thesis."
Rewrite: "The subsidy-ROI evidence complicates the catchment-leakage thesis."
- Original: "Three separate loci converge on the same methodological failure mode."
@@ -1677,7 +1711,7 @@ ship or loop back for a structural fix.
# ---------------------------------------------------------------------------
# Layer 4 — draft orchestrator. One of 3 parallel sub-orchestrators that
# each produce an independent draft from a different analytical angle.
-# Opus, full tool access including Task (can spawn fetchers for additional
+# Full tool access including Task (can spawn fetchers for additional
# evidence gathering specific to its angle).
# ---------------------------------------------------------------------------
DRAFT_ORCHESTRATOR_AGENT = """\
@@ -1689,8 +1723,8 @@ description: >
source note IDs to read. Reads every note on the list via batch
`note show` (no vault surveys, no decision-making about what to read),
then writes one complete draft from the assigned angle. The main
- orchestrator synthesizes a final report from all three drafts. Runs on Opus.
-model: opus
+ orchestrator synthesizes a final report from all three drafts.
+model: << p.models.draft_orchestrator >>
tools: Bash, Read, Write
color: green
---
@@ -1703,11 +1737,11 @@ from all three drafts.
## Pipeline position
You are **step 10** of the hyperresearch V8 pipeline. Prior steps produced:
-- `research/prompt-decomposition.json` — atomic items, required_section_headings
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, required_section_headings
- Width corpus (vault notes tagged with the vault_tag)
-- `research/temp/evidence-digest.md` — top claims + verbatim quotes
-- `research/comparisons.md` (if full tier) — cross-locus tensions
-- `research/temp/source-tensions.json` (if full tier) — expert disagreements
+- `research/runs/<vault_tag>/temp/evidence-digest.md` — top claims + verbatim quotes
+- `research/runs/<vault_tag>/comparisons.md` (if full tier) — cross-locus tensions
+- `research/runs/<vault_tag>/temp/source-tensions.json` (if full tier) — expert disagreements
- Interim notes from depth investigators (if full tier)
- **A pre-curated `must_read_note_ids` list** — the orchestrator already
picked the 20-50 sources most relevant to YOUR angle. You don't choose
@@ -1723,17 +1757,17 @@ all three. Your draft is an INPUT to the synthesis, not the final output.
- **query_file_path**: path to the persisted query file.
- **vault_tag**: corpus tag.
- **draft_id**: your identifier — `"a"`, `"b"`, or `"c"`.
-- **output_path**: where to write your draft (e.g., `research/temp/draft-a.md`).
+- **output_path**: where to write your draft (e.g., `research/runs/<vault_tag>/temp/draft-a.md`).
- **analytical_angle**: a 2-3 sentence description of your assigned angle.
This is what makes your draft DIFFERENT from the other two. Lean into it.
- **must_read_note_ids**: an array of 20-50 vault note IDs. The orchestrator
pre-selected these as most relevant to your angle. **You MUST read every
one before writing.** No vault surveys, no skimming summaries, no choosing
your own sources.
-- **decomposition_path**: `research/prompt-decomposition.json`.
-- **evidence_digest_path**: `research/temp/evidence-digest.md` (if exists).
-- **comparisons_path**: `research/comparisons.md` (if exists).
-- **source_tensions_path**: `research/temp/source-tensions.json` (if exists).
+- **decomposition_path**: `research/runs/<vault_tag>/prompt-decomposition.json`.
+- **evidence_digest_path**: `research/runs/<vault_tag>/temp/evidence-digest.md` (if exists).
+- **comparisons_path**: `research/runs/<vault_tag>/comparisons.md` (if exists).
+- **source_tensions_path**: `research/runs/<vault_tag>/temp/source-tensions.json` (if exists).
- **response_format**: `"short"` / `"structured"` / `"argumentative"`.
- **citation_style**: `"wikilink"` / `"inline"` / `"none"`.
- **modality**: `"collect"` / `"synthesize"` / `"compare"` / `"forecast"`.
@@ -1743,11 +1777,11 @@ all three. Your draft is an INPUT to the synthesis, not the final output.
These are quick — get them out of the way before the heavy reading.
1. Read the query file. This is your north star.
-2. Read `research/prompt-decomposition.json`. Note every atomic item and
+2. Read `research/runs/<vault_tag>/prompt-decomposition.json`. Note every atomic item and
`required_section_headings` — you MUST honor these.
-3. Read `research/temp/evidence-digest.md` if it exists.
-4. Read `research/comparisons.md` if it exists.
-5. Read `research/temp/source-tensions.json` if it exists.
+3. Read `research/runs/<vault_tag>/temp/evidence-digest.md` if it exists.
+4. Read `research/runs/<vault_tag>/comparisons.md` if it exists.
+5. Read `research/runs/<vault_tag>/temp/source-tensions.json` if it exists.
**Do NOT survey the vault.** Do NOT run `note list`, `search ""`, or any
metadata listing command. The orchestrator already curated your reading
@@ -1857,7 +1891,7 @@ When done, tell the main orchestrator:
# Synthesizer. Step 10.3 of hyperresearch V8. Reads the 3 sub-orchestrator
# drafts plus the orchestrator's synthesis plan and outline, then writes
# a fresh integrated final report in two passes (rough integrated draft,
-# then voice/redundancy/length cleanup). Opus, tool-locked to [Read, Write].
+# then voice/redundancy/length cleanup). Tool-locked to [Read, Write].
# Single subagent, runs once.
# ---------------------------------------------------------------------------
SYNTHESIZER_AGENT = """\
@@ -1872,8 +1906,8 @@ description: >
rewrites for voice consistency, redundancy, length discipline, and
argumentative density. The final report is a fresh write in ONE prose
voice, NOT section-grafted from the inputs. Tool-locked: Read + Write
- ONLY. Cannot Bash, cannot spawn subagents. Runs on Opus.
-model: opus
+ ONLY. Cannot Bash, cannot spawn subagents.
+model: << p.models.synthesizer >>
tools: Read, Write
color: cyan
---
@@ -1916,21 +1950,21 @@ mental model; writing the final report is a fresh act.
- **research_query**: the user's original question, verbatim. GOSPEL.
- **query_file_path**: path to the persisted query file.
-- **draft_paths**: array of 3 paths — `[research/temp/draft-a.md,
- research/temp/draft-b.md, research/temp/draft-c.md]`.
-- **synthesis_plan_path**: `research/temp/synthesis-plan.md` — the
+- **draft_paths**: array of 3 paths — `[research/runs/<vault_tag>/temp/draft-a.md,
+ research/runs/<vault_tag>/temp/draft-b.md, research/runs/<vault_tag>/temp/draft-c.md]`.
+- **synthesis_plan_path**: `research/runs/<vault_tag>/temp/synthesis-plan.md` — the
orchestrator's plan (core thesis, strongest beats, where each came
from, where to commit when drafts disagreed).
-- **synthesis_outline_path**: `research/temp/synthesis-outline.md` —
+- **synthesis_outline_path**: `research/runs/<vault_tag>/temp/synthesis-outline.md` —
the orchestrator's per-section outline (1-2 sentences per H2 section
naming what evidence and argument goes there).
-- **decomposition_path**: `research/prompt-decomposition.json` — atomic
+- **decomposition_path**: `research/runs/<vault_tag>/prompt-decomposition.json` — atomic
items, required_section_headings, response_format, citation_style.
-- **comparisons_path**: `research/comparisons.md` (full tier).
-- **source_tensions_path**: `research/temp/source-tensions.json` (full tier).
-- **evidence_digest_path**: `research/temp/evidence-digest.md` — top
+- **comparisons_path**: `research/runs/<vault_tag>/comparisons.md` (full tier).
+- **source_tensions_path**: `research/runs/<vault_tag>/temp/source-tensions.json` (full tier).
+- **evidence_digest_path**: `research/runs/<vault_tag>/temp/evidence-digest.md` — top
claims with verbatim quotes and source IDs.
-- **pass1_output_path**: `research/temp/synthesis-pass1.md` — where
+- **pass1_output_path**: `research/runs/<vault_tag>/temp/synthesis-pass1.md` — where
you write the rough integrated draft (pass 1).
- **final_output_path**: `research/notes/final_report_<vault_tag>.md` — where you
write the cleaned-up final report (pass 2).
@@ -2163,7 +2197,7 @@ When done, tell the orchestrator:
# ---------------------------------------------------------------------------
-# Readability reformatter. Experimental Layer 8 agent. Opus, tool-locked
+# Readability reformatter. Experimental Layer 8 agent. Tool-locked
# to [Read, Edit]. Takes the polished report and reformats for human
# readability: breaks walls of text, adds visual hierarchy, ensures
# scannability.
@@ -2179,8 +2213,8 @@ description: >
category (merge-paragraphs / break-paragraph / make-list / make-table
/ bold-keyterms / split-sentence / remove-hr / add-whitespace).
Tool-locked to [Read, Write] — cannot Edit. The orchestrator decides
- which recommendations to apply via direct Edit calls. Runs on Opus.
-model: opus
+ which recommendations to apply via direct Edit calls.
+model: << p.models.readability_recommender >>
tools: Read, Write
color: magenta
---
@@ -2214,7 +2248,7 @@ apply. You are advisory.
- **research_query**: verbatim user question. GOSPEL.
- **draft_path**: `research/notes/final_report_<vault_tag>.md` — the polished report.
-- **recommendations_path**: `research/readability-recommendations.json`
+- **recommendations_path**: `research/runs/<vault_tag>/readability-recommendations.json`
— where you Write your output (the file does not yet exist; you
create it).
@@ -2328,7 +2362,7 @@ Write a single JSON file to `recommendations_path`. Schema:
it's the formatted list. For HR removal, it's an empty string.
- `rationale` is one sentence explaining the readability gain.
-3. **Cap your output at 50 recommendations.** Prioritize by impact:
+3. **Cap your output at << p.readability_rec_cap >> recommendations.** Prioritize by impact:
- Merge-paragraphs and break-paragraph fixes have the highest impact
(they fundamentally change paragraph rhythm)
- Make-list and make-table fixes substantially improve scannability
@@ -2375,8 +2409,8 @@ Tell the orchestrator:
# ---------------------------------------------------------------------------
# Source analyst. Leaf subagent for deep end-to-end analysis of ONE long
-# source. Runs on Sonnet (1M context window). Produces a structured
-# source-analysis note backlinked to the original.
+# source. Produces a structured source-analysis note backlinked to the
+# original.
# ---------------------------------------------------------------------------
SOURCE_ANALYST_AGENT = """\
---
@@ -2388,10 +2422,9 @@ description: >
type='source-analysis', backlinked to the original source. Use when a
single source is load-bearing AND exceeds roughly 5000 words — short
sources are already adequately covered by the fetcher's summary.
- Runs on Sonnet (1M context window). Spawn multiple in parallel for
- multiple independent long sources. Does NOT spawn any other subagents
- itself (leaf).
-model: sonnet
+ Spawn multiple in parallel for multiple independent long sources.
+ Does NOT spawn any other subagents itself (leaf).
+model: << p.models.source_analyst >>
tools: Bash, Read, Write
color: cyan
---
@@ -2409,8 +2442,8 @@ You are a leaf subagent available to the orchestrator (Layer 1-4) and
the depth investigator (Layer 3). Neither layer reads long sources
optimally: the orchestrator would consume excessive context, the
depth investigator is scoped to its locus and may miss cross-locus
-substance. You fill that gap by reading ONE source fully on Sonnet's
-1M-token context window.
+substance. You fill that gap by reading ONE source fully, end to end,
+in your own context window.
You do NOT spawn other subagents. If you need something beyond the
single source you were assigned, report back to the parent agent
@@ -2428,7 +2461,7 @@ analyst, fetch new sources, or move on.
full body.
- **output_path**: the markdown file path where you write the analysis
body BEFORE calling `note new --body-file` (e.g.,
- `research/temp/source-analysis-<source_note_id>.md`).
+ `research/runs/<vault_tag>/temp/source-analysis-<source_note_id>.md`).
- **vault_tag**: the run-level corpus tag so the new note is findable
alongside its sibling notes.
@@ -2445,11 +2478,11 @@ analyst, fetch new sources, or move on.
```bash
PYTHONIOENCODING=utf-8 {hpr_path} note show <source_note_id> -j
```
- Hold the full body in your context. Sonnet 1M lets you read up to
- roughly 750K words before truncation matters. If the source exceeds
- that (rare — most 500-page PDFs extract to <300K words), report
- back to the parent with `truncation_warning: true` and analyze
- what you could read.
+ Hold the full body in your context. Most sources fit comfortably —
+ even 500-page PDFs usually extract to <300K words. If the source
+ exceeds what your context window can hold, report back to the
+ parent with `truncation_warning: true` and analyze what you could
+ read.
3. **Read the research_query again.** Anchor your analysis to what
the user actually asked. Not every load-bearing claim in the
@@ -2532,10 +2565,10 @@ Arabic, etc.), your extracted quotes MUST be copied verbatim from
the Read tool output. Never retype or transliterate. Downstream
agents and lint rules expect exact character matches.
-## Cost discipline
+## Effort discipline
-You run on Sonnet 1M context. A full read of a 60K-word source costs
-roughly $2-5 per spawn. Do not pad: if the source's substantive
+A full read of a 60K-word source is one of the pipeline's most
+expensive single spawns. Do not pad: if the source's substantive
density is low despite its length (e.g., a long transcript that
repeats itself), your analysis should be correspondingly short. The
template sections are REQUIRED, but each section's length is
@@ -2558,7 +2591,7 @@ Return a compact status line to the parent:
# ---------------------------------------------------------------------------
-# Layer 1 — fetcher. Sonnet-powered research agent with agency to follow
+# Layer 1 — fetcher. Research agent with agency to follow
# leads to primary sources. Fetches assigned URLs + chases citation chains.
# ---------------------------------------------------------------------------
RESEARCHER_AGENT = """\
@@ -2568,9 +2601,9 @@ description: >
Research fetcher with primary-source-chasing agency. Fetches assigned URLs,
reads and summarizes content, extracts structured claims, then follows
citation chains and references to discover and fetch primary sources the
- secondary sources cite. Runs on Sonnet for better comprehension and
- judgment. Spawn multiple in parallel for bulk research.
-model: sonnet
+ secondary sources cite. Needs solid comprehension and judgment.
+ Spawn multiple in parallel for bulk research.
+model: << p.models.fetcher >>
tools: Bash, Read, Write, WebSearch
color: blue
---
@@ -2636,6 +2669,19 @@ On Windows, ALWAYS prefix commands with `PYTHONIOENCODING=utf-8`:
PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> -j
```
+### Utility flag — `--utility-score`
+
+When your assigned batch carries a utility score next to a URL (e.g.
+"https://... (utility: 14)"), pass it through so it persists into the note's
+frontmatter and feeds the vault's composite quality ranking:
+
+```bash
+PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> --utility-score 14 -j
+```
+
+DOIs and arXiv ids are captured automatically during fetch — you do not
+need to extract them yourself.
+
### Backlink flag — `--suggested-by`
When fetching a URL that was referenced by a source you already processed,
@@ -2693,7 +2739,7 @@ For each URL the parent agent gave you:
**Long source flag:** if >5000 words AND relevant, report prominently
to the parent agent for potential `hyperresearch-source-analyst` delegation.
-6. **Extract structured claims** to `research/temp/claims-<note-id>.json`:
+6. **Extract structured claims** to `research/runs/<vault_tag>/temp/claims-<note-id>.json`:
```json
{{
@@ -2737,7 +2783,7 @@ After processing ALL assigned URLs, review your leads list. This is where
you add real value — secondary sources cite primary evidence, and fetching
those primaries gives the pipeline higher-authority sources to cite.
-1. **Prioritize leads.** From your collected leads, select the **3-8 most
+1. **Prioritize leads.** From your collected leads, select the **<< p.fetcher_chase|hyphen >> most
promising** based on:
- **Authority:** government data, peer-reviewed papers, and official
reports over blog commentary or news articles
@@ -2767,11 +2813,11 @@ those primaries gives the pipeline higher-authority sources to cite.
3. **Process each discovered source** with the same full procedure as
Phase 1: read the note content with `{hpr_path} note show <id> -j`,
quality check, write summary with `{hpr_path} note update`, add tags,
- and extract structured claims to `research/temp/claims-<note-id>.json`.
+ and extract structured claims to `research/runs/<vault_tag>/temp/claims-<note-id>.json`.
Primary sources often have the specific numbers and methodological
details that secondary commentary paraphrases — extract these precisely.
-4. **Cap:** Fetch at most **8 additional primary sources** beyond your
+4. **Cap:** Fetch at most **<< p.fetcher_chase_cap >> additional primary sources** beyond your
assigned URLs. This is targeted enrichment. If you find more promising
leads than you can fetch, report the unfetched leads to the parent
agent.
@@ -2802,9 +2848,9 @@ description: >
and comparisons.md. Verifies committed positions against original
source text via note show, then asks: "what source, if found, would
overturn the current direction?" Outputs a targeted fetch list of 3-8
- high-leverage missing sources. Runs on Sonnet. Spawn ONCE before
+ high-leverage missing sources. Spawn ONCE before
drafting, after Layer 3.5 comparisons.
-model: sonnet
+model: << p.models.corpus_critic >>
tools: Bash, Read, Write
color: teal
---
@@ -2827,9 +2873,9 @@ to drafting.
- **research_query**: verbatim. GOSPEL.
- **corpus_tag**: vault tag for searching.
-- **comparisons_path**: `research/comparisons.md`
-- **loci_path**: `research/loci.json`
-- **output_path**: `research/corpus-critic-gaps.json`
+- **comparisons_path**: `research/runs/<vault_tag>/comparisons.md`
+- **loci_path**: `research/runs/<vault_tag>/loci.json`
+- **output_path**: `research/runs/<vault_tag>/corpus-critic-gaps.json`
## Procedure
@@ -2855,14 +2901,14 @@ to drafting.
scope limitation, or contradicting detail that the position ignores,
flag that as a gap — the draft would inherit the error.
-3. **Read consensus claims** from `research/temp/consensus-claims.json`
+3. **Read consensus claims** from `research/runs/<vault_tag>/temp/consensus-claims.json`
(if it exists). For each high-confidence consensus:
- Is there a plausible dissenting source you haven't looked for?
- Is the consensus supported by INDEPENDENT sources, or by
derivative sources tracing to one upstream report? Check
- `research/temp/redundancy-audit.md` if it exists.
+ `research/runs/<vault_tag>/temp/redundancy-audit.md` if it exists.
-4. **Check the redundancy audit** (`research/temp/redundancy-audit.md`).
+4. **Check the redundancy audit** (`research/runs/<vault_tag>/temp/redundancy-audit.md`).
Are any positions supported only by derivative sources? That support
is fragile — flag it.
@@ -2888,7 +2934,7 @@ to drafting.
}}
```
- **Cap: 3-8 gaps.** Only `critical` and `high` priority. Do not
+ **Cap: << p.corpus_critic_gaps|hyphen >> gaps.** Only `critical` and `high` priority. Do not
identify gaps for tangential topics — every gap must serve the
research_query.
@@ -2908,6 +2954,213 @@ to drafting.
# The hook script that gets installed
+BROWSER_FETCHER_AGENT = """\
+---
+name: hyperresearch-browser-fetcher
+description: >
+ Escalation-lane fetcher that drives the user's REAL Chrome browser (via
+ Claude-in-Chrome) for sources headless crawling cannot reach: login-gated
+ pages, bot-walled sites, interactive/infinite-scroll pages, viewer-rendered
+ PDFs, and Google Scholar searches. Drains the `hyperresearch escalation`
+ queue serially — one item, one tab, at a time. Spawn EXACTLY ONE at a time;
+ parallel instances fighting over one browser is chaos. HARD BOUNDARY:
+ never attempts to solve CAPTCHAs, 2FA, or logins — those are marked
+ needs_human and consolidated for the user.
+model: << p.models.browser_fetcher >>
+tools: Bash, Read, Write, ToolSearch
+color: orange
+---
+
+You are the hyperresearch browser-lane fetcher. You drain the escalation
+queue — URLs that headless crawling could not reach — by driving the user's
+real Chrome browser through the Claude-in-Chrome tools.
+
+## Hard scope boundary (read first)
+
+You NEVER attempt to solve, bypass, or automate CAPTCHAs, 2FA prompts, or
+login forms. The moment a page asks for something only the account owner
+should do, you run:
+
+```bash
+PYTHONIOENCODING=utf-8 {hpr_path} escalation human <id> --detail "<one line: site + what the human must do>" -j
+```
+
+and move to the next item. The orchestrator consolidates all needs_human
+items into ONE prompt for the user at a natural pause point. You read what
+the user's own access can see; you do not evade. Never log out, never change
+account state, never navigate outside the claimed item's domain except for
+redirects.
+
+## Setup (once per session)
+
+Load the Chrome tools in ONE batched ToolSearch call:
+
+ToolSearch query: "select:mcp__claude-in-chrome__tabs_context_mcp,mcp__claude-in-chrome__tabs_create_mcp,mcp__claude-in-chrome__navigate,mcp__claude-in-chrome__get_page_text,mcp__claude-in-chrome__read_page,mcp__claude-in-chrome__find,mcp__claude-in-chrome__computer"
+
+Then call tabs_context_mcp once. ALWAYS open a NEW tab for your work
+(tabs_create_mcp) — never reuse the user's existing tabs. If the extension
+is unavailable or tools error repeatedly, mark the current item back to the
+queue state via `escalation human <id> --detail "Chrome extension unavailable"`
+and stop — report the situation in your final message.
+
+## The drain loop
+
+Repeat up to your assigned batch size (default 10 items):
+
+1. **Claim:**
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation claim --by browser-fetcher --tag <vault_tag> -j
+ ```
+ `queue_empty: true` → you're done; write your summary and return.
+
+2. **Navigate** the claimed URL in your tab. Wait for content. Human-paced:
+ one page at a time, no rapid-fire requests.
+
+3. **Extract.** Prefer `get_page_text` (whole-page text) over DOM surgery.
+ Playbook for hard pages:
+ - **Infinite scroll / "load more":** scroll or click until content
+ stabilizes, hard cap ~10 interactions, then extract once.
+ - **In-page navigation (SPAs, tabs, accordions):** expand sections that
+ contain content relevant to the research query; skip nav chrome.
+ - **PDF in a viewer:** extract the viewer's text layer via get_page_text;
+ if empty, note "PDF viewer without text layer" and mark needs_human
+ with the download suggestion.
+ - **Charts/figures with thin text:** screenshot via the computer tool and
+ transcribe the load-bearing figures/axis values into a
+ `## Extracted figures` section of your writeup.
+ - **CAPTCHA / login / 2FA appears:** STOP. `escalation human` (see
+ boundary above). Next item.
+
+4. **Ingest.** Write the extracted content to a scratch file, then:
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation ingest <id> --title "<page title>" --body-file <scratch-file> --tag <topic-tag> -j
+ ```
+ One command — it writes the vault note (with `fetch_provider: chrome`
+ provenance), records the source row, syncs, and resolves the item. Do
+ NOT use `note new` or `fetch` for escalation items.
+
+5. **Genuinely unreachable** (dead page, geo-block, content gone):
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation abandon <id> --detail "<why>" -j
+ ```
+ Abandoning is fine — the floor is where we started (source lost).
+
+## Scholar items (`reason: scholar_search`)
+
+The item's `url` field is a SEARCH QUERY, not a URL. Google Scholar has no
+API and blocks headless crawlers; you are the lane.
+
+1. Open https://scholar.google.com in your tab, search the query.
+2. Extract the top ~10 results: title, authors, year, venue, citation
+ count, link, and the cited-by link.
+3. Write them as a markdown list and ingest with
+ `--title "Scholar: <query>" --tag scholar-results`.
+4. For the 2-3 highest-citation results directly relevant to the research
+ query, queue their links for a future drain:
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation add "<paper url>" --reason interactive_needed --tag <vault_tag> --suggested-by <scholar-note-id> --detail "high-citation Scholar hit" -j
+ ```
+ Cap: one query at a time, small N, human-paced. This is a courtesy lane,
+ not a scraper.
+
+## Report back
+
+Your final message is data for the orchestrator, not prose for a human:
+- items drained: N fetched / N needs_human / N abandoned
+- note ids created
+- needs_human items with their one-line details (the orchestrator will
+ consolidate these for the user)
+- anything that suggests the whole domain is unreachable (so the
+ orchestrator stops queueing it)
+"""
+
+
+CITE_CHECKER_AGENT = """\
+---
+name: hyperresearch-cite-checker
+description: >
+ Step 14.5 of the hyperresearch V8 pipeline. Verifies that each sampled
+ citation actually supports its sentence by reading the cited note's body.
+ Receives batches of (sentence, note_id) pairs the mechanical triage could
+ not auto-pass; returns per-pair verdicts (supported / partially-supported /
+ unsupported / wrong-source) as findings JSON the patcher consumes.
+ This is reading comprehension at volume, not prose judgment.
+ Never edits the report.
+model: << p.models.cite_checker >>
+tools: Bash, Read, Write
+color: red
+---
+
+You are the hyperresearch cite-checker. Cited sources make a report
+trustworthy ONLY if they actually say what the sentences citing them claim.
+You verify that binding, pair by pair.
+
+## Pipeline position
+
+You are step 14.5 of the hyperresearch V8 pipeline. The report has been
+synthesized (11), critiqued (12), and patched (14). Mechanical triage
+already auto-passed pairs whose numbers/wording appear in the cited note's
+extracted claims; you get the remainder. Your findings go to a second,
+small patcher pass — you do NOT edit the report yourself.
+
+## Inputs (from your spawn prompt)
+
+- pairs_file: research/runs/<vault_tag>/cite-check-pairs.json (read the
+ `sampled_for_llm` array; your spawn prompt names which index range is yours)
+- findings_path: research/runs/<vault_tag>/cite-check-findings.json
+- vault_tag
+
+## Procedure
+
+For each assigned pair:
+
+1. Read the cited note's body:
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} note show <note_id> -j
+ ```
+ Batch-read up to 5 ids per call when consecutive pairs cite different notes.
+
+2. Judge: does the note's content support the sentence AS WRITTEN?
+ - **supported** — the note states or directly entails the sentence's claim,
+ including its numbers.
+ - **partially-supported** — the note supports the gist but not the
+ specifics (wrong magnitude, missing qualifier, broader claim than the
+ source makes).
+ - **unsupported** — nothing in the note backs the sentence.
+ - **wrong-source** — the note doesn't back it, but another vault note
+ does. Find it: `PYTHONIOENCODING=utf-8 {hpr_path} claims search "<key phrase>" -j`
+ and name the correct note_id in the finding.
+
+ Judge the SOURCE-SENTENCE binding only. Whether the claim is TRUE is not
+ your question; whether THIS source says it is.
+
+3. Only non-`supported` verdicts become findings. Write ALL your findings in
+ ONE JSON array to your assigned findings path:
+ ```json
+ [
+ {
+ "verdict": "unsupported | partially-supported | wrong-source",
+ "severity": "critical | major",
+ "sentence": "<verbatim from the pairs file>",
+ "cited_note_id": "<id>",
+ "correct_note_id": "<id or null>",
+ "evidence": "<one sentence: what the note actually says / lacks>",
+ "suggested_fix": "<swap citation | soften claim to what the source supports | delete sentence>"
+ }
+ ]
+ ```
+ Severity: `critical` for unsupported number-bearing claims and
+ wrong-source; `major` otherwise. Write `[]` if every pair checked out.
+
+## Rules
+
+- Verdicts default SKEPTICAL: when you cannot find support in the note,
+ the verdict is unsupported — never "probably fine".
+- Do not re-litigate pairs the triage auto-passed.
+- Your final message: counts per verdict + the findings path. Data, not prose.
+"""
+
+
HOOK_SCRIPT_TEMPLATE = """\
#!/usr/bin/env node
/**
@@ -2951,9 +3204,16 @@ if (vault) {{
"""
-def install_hooks(vault_root: Path, hpr_path: str = "hyperresearch") -> list[str]:
+def install_hooks(
+ vault_root: Path,
+ hpr_path: str = "hyperresearch",
+ profile: str = "full",
+) -> list[str]:
"""Install the Claude Code hook + skills + subagents. Returns list of actions taken.
+ Skill and agent prompts are rendered from the given pipeline profile
+ (plus any `[profile.*]` overlays in the vault's config.toml).
+
Hyperresearch roster (as of v7):
fetcher (Layer 1, 3, 4), loci-analyst (Layer 2), depth-investigator (Layer 3),
source-analyst (on-demand, 1M context), corpus-critic (Layer 3.7),
@@ -2961,6 +3221,8 @@ def install_hooks(vault_root: Path, hpr_path: str = "hyperresearch") -> list[str
dialectic-critic + depth-critic + width-critic + instruction-critic (Layer 5),
patcher (Layer 6), polish-auditor (Layer 7).
"""
+ config_path = vault_root / ".hyperresearch" / "config.toml"
+ _set_render_state(profile, config_path if config_path.exists() else None)
actions = []
for installer in (
@@ -2981,6 +3243,8 @@ def install_hooks(vault_root: Path, hpr_path: str = "hyperresearch") -> list[str
lambda: _install_corpus_critic_agent(vault_root, hpr_path),
lambda: _install_draft_orchestrator_agent(vault_root, hpr_path),
lambda: _install_synthesizer_agent(vault_root, hpr_path),
+ lambda: _install_browser_fetcher_agent(vault_root, hpr_path),
+ lambda: _install_cite_checker_agent(vault_root, hpr_path),
lambda: _prune_retired_agents(vault_root),
):
result = installer()
@@ -2990,7 +3254,11 @@ def install_hooks(vault_root: Path, hpr_path: str = "hyperresearch") -> list[str
return actions
-def install_global_hooks(home: Path | None = None, hpr_path: str = "hyperresearch") -> list[str]:
+def install_global_hooks(
+ home: Path | None = None,
+ hpr_path: str = "hyperresearch",
+ profile: str = "full",
+) -> list[str]:
"""Install Claude Code skills + agents globally under ~/.claude/.
Unlike `install_hooks`, this skips:
@@ -3016,6 +3284,8 @@ def install_global_hooks(home: Path | None = None, hpr_path: str = "hyperresearc
if home is None:
home = Path.home()
+ # Global installs have no vault config — built-in profiles only.
+ _set_render_state(profile, None)
actions = []
for installer in (
@@ -3034,6 +3304,8 @@ def install_global_hooks(home: Path | None = None, hpr_path: str = "hyperresearc
lambda: _install_corpus_critic_agent(home, hpr_path),
lambda: _install_draft_orchestrator_agent(home, hpr_path),
lambda: _install_synthesizer_agent(home, hpr_path),
+ lambda: _install_browser_fetcher_agent(home, hpr_path),
+ lambda: _install_cite_checker_agent(home, hpr_path),
lambda: _prune_retired_agents(home),
lambda: _prune_global_step_skills(home),
):
@@ -3133,6 +3405,8 @@ def _write_agent_file(
agents_dir.mkdir(parents=True, exist_ok=True)
agent_path = agents_dir / filename
+ content = _render_installed(content)
+
if agent_path.exists():
existing = agent_path.read_text(encoding="utf-8")
if existing == content:
@@ -3146,7 +3420,7 @@ def _install_researcher_agent(vault_root: Path, hpr_path: str) -> str | None:
hpr_posix = hpr_path.replace("\\", "/")
content = RESEARCHER_AGENT.format(hpr_path=hpr_posix)
return _write_agent_file(
- vault_root, "hyperresearch-fetcher.md", content, "sonnet fetcher (primary-source chasing)"
+ vault_root, "hyperresearch-fetcher.md", content, "fetcher (primary-source chasing)"
)
@@ -3154,7 +3428,7 @@ def _install_loci_analyst_agent(vault_root: Path, hpr_path: str) -> str | None:
hpr_posix = hpr_path.replace("\\", "/")
content = LOCI_ANALYST_AGENT.format(hpr_path=hpr_posix)
return _write_agent_file(
- vault_root, "hyperresearch-loci-analyst.md", content, "sonnet loci analyst"
+ vault_root, "hyperresearch-loci-analyst.md", content, "loci analyst"
)
@@ -3165,7 +3439,7 @@ def _install_source_analyst_agent(vault_root: Path, hpr_path: str) -> str | None
vault_root,
"hyperresearch-source-analyst.md",
content,
- "sonnet source analyst (1M context)",
+ "source analyst (full-source deep read)",
)
@@ -3176,7 +3450,7 @@ def _install_depth_investigator_agent(vault_root: Path, hpr_path: str) -> str |
vault_root,
"hyperresearch-depth-investigator.md",
content,
- "sonnet depth investigator",
+ "depth investigator",
)
@@ -3187,7 +3461,7 @@ def _install_dialectic_critic_agent(vault_root: Path, hpr_path: str) -> str | No
vault_root,
"hyperresearch-dialectic-critic.md",
content,
- "opus dialectic critic",
+ "dialectic critic",
)
@@ -3198,7 +3472,7 @@ def _install_depth_critic_agent(vault_root: Path, hpr_path: str) -> str | None:
vault_root,
"hyperresearch-depth-critic.md",
content,
- "opus depth critic",
+ "depth critic",
)
@@ -3209,7 +3483,7 @@ def _install_width_critic_agent(vault_root: Path, hpr_path: str) -> str | None:
vault_root,
"hyperresearch-width-critic.md",
content,
- "opus width critic",
+ "width critic",
)
@@ -3221,7 +3495,7 @@ def _install_instruction_critic_agent(vault_root: Path, hpr_path: str) -> str |
vault_root,
"hyperresearch-instruction-critic.md",
content,
- "opus instruction critic",
+ "instruction critic",
)
@@ -3229,7 +3503,7 @@ def _install_patcher_agent(vault_root: Path, hpr_path: str) -> str | None:
# Patcher prompt does not reference hpr_path, but format is harmless
content = PATCHER_AGENT
return _write_agent_file(
- vault_root, "hyperresearch-patcher.md", content, "sonnet patcher (Read+Edit only)"
+ vault_root, "hyperresearch-patcher.md", content, "patcher (Read+Edit only)"
)
@@ -3240,7 +3514,7 @@ def _install_synthesizer_agent(vault_root: Path, hpr_path: str) -> str | None:
vault_root,
"hyperresearch-synthesizer.md",
content,
- "opus synthesizer (Read+Write only, two-pass)",
+ "synthesizer (Read+Write only, two-pass)",
)
@@ -3252,7 +3526,7 @@ def _install_polish_auditor_agent(vault_root: Path, hpr_path: str) -> str | None
vault_root,
"hyperresearch-polish-auditor.md",
content,
- "sonnet polish auditor (Read+Edit only)",
+ "polish auditor (Read+Edit only)",
)
@@ -3278,7 +3552,29 @@ def _install_readability_reformatter_agent(vault_root: Path, hpr_path: str) -> s
vault_root,
"hyperresearch-readability-recommender.md",
content,
- "opus readability recommender (Read+Write — writes JSON recommendations only)",
+ "readability recommender (Read+Write — writes JSON recommendations only)",
+ )
+
+
+def _install_cite_checker_agent(vault_root: Path, hpr_path: str) -> str | None:
+ hpr_posix = hpr_path.replace("\\", "/")
+ content = CITE_CHECKER_AGENT.replace("{hpr_path}", hpr_posix)
+ return _write_agent_file(
+ vault_root,
+ "hyperresearch-cite-checker.md",
+ content,
+ "cite-checker (step 14.5 — citation-sentence binding verification)",
+ )
+
+
+def _install_browser_fetcher_agent(vault_root: Path, hpr_path: str) -> str | None:
+ hpr_posix = hpr_path.replace("\\", "/")
+ content = BROWSER_FETCHER_AGENT.replace("{hpr_path}", hpr_posix)
+ return _write_agent_file(
+ vault_root,
+ "hyperresearch-browser-fetcher.md",
+ content,
+ "browser-lane fetcher (Chrome escalation queue; needs_human boundary)",
)
@@ -3288,7 +3584,7 @@ def _install_corpus_critic_agent(vault_root: Path, hpr_path: str) -> str | None:
vault_root,
"hyperresearch-corpus-critic.md",
content,
- "sonnet corpus critic (Layer 3.7)",
+ "corpus critic (Layer 3.7)",
)
@@ -3299,7 +3595,7 @@ def _install_draft_orchestrator_agent(vault_root: Path, hpr_path: str) -> str |
vault_root,
"hyperresearch-draft-orchestrator.md",
content,
- "opus draft sub-orchestrator (Layer 4)",
+ "draft sub-orchestrator (Layer 4)",
)
@@ -3404,6 +3700,7 @@ def _install_hyperresearch_skill(vault_root: Path) -> str | None:
content = _read_skill_source("hyperresearch.md")
if content is None:
return None
+ content = _render_installed(content)
skill_dir = vault_root / ".claude" / "skills" / "hyperresearch"
skill_dir.mkdir(parents=True, exist_ok=True)
@@ -3416,6 +3713,7 @@ def _install_hyperresearch_skill(vault_root: Path) -> str | None:
_HYPERRESEARCH_STEP_SKILLS = [
"hyperresearch-1-decompose",
+ "hyperresearch-1-5-chapter-partition",
"hyperresearch-2-width-sweep",
"hyperresearch-3-contradiction-graph",
"hyperresearch-4-loci-analysis",
@@ -3429,6 +3727,7 @@ _HYPERRESEARCH_STEP_SKILLS = [
"hyperresearch-12-critics",
"hyperresearch-13-gap-fetch",
"hyperresearch-14-patcher",
+ "hyperresearch-14-5-cite-check",
"hyperresearch-15-polish",
"hyperresearch-16-readability-audit",
]
@@ -3459,6 +3758,7 @@ def _install_hyperresearch_step_skills(vault_root: Path) -> str | None:
content = _read_skill_source(src_name)
if content is None:
continue
+ content = _render_installed(content)
skill_dir = skills_root / skill_name
skill_dir.mkdir(parents=True, exist_ok=True)
diff --git a/src/hyperresearch/core/independence.py b/src/hyperresearch/core/independence.py
new file mode 100644
index 0000000..3844b1a
--- /dev/null
+++ b/src/hyperresearch/core/independence.py
@@ -0,0 +1,151 @@
+"""Independence audit — syndicated copies must not multiply consensus.
+
+"3+ independent sources agree" is the pipeline's consensus rule, but a wire
+story republished by five outlets is ONE source wearing five outfits.
+This module clusters derivative sources three ways:
+
+ url — identical canonical URL (scheme/www/trailing-slash/UTM stripped)
+ body — near-duplicate bodies (>=0.7 MinHash-verified Jaccard, reusing
+ core/similarity.py)
+ wire — shared wire-service boilerplate (PRNewswire, Business Wire, ...)
+ in the opening of the text, same story cluster by title overlap
+
+Scores: cluster root (earliest fetched) keeps independence 1.0; members get
+1/cluster_size. Unclustered notes get 1.0. Stored on notes.independence
+(DB-cache, recomputable) and consumed by step 3's consensus counting and
+the quality composite.
+"""
+
+from __future__ import annotations
+
+import re
+from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
+
+from hyperresearch.core.similarity import jaccard, shingle
+
+WIRE_MARKERS = (
+ "prnewswire", "pr newswire", "business wire", "businesswire",
+ "globe newswire", "globenewswire", "(reuters)", "(ap)", "associated press",
+ "accesswire", "newsfile corp",
+)
+
+_TRACKING_PARAMS_RE = re.compile(r"^(utm_|fbclid|gclid|ref$|source$)")
+NEAR_DUP_THRESHOLD = 0.7
+
+
+def canonical_url(url: str) -> str:
+ """Normalize scheme/host/path/query so syndication mirrors collide."""
+ p = urlparse(url.strip().lower())
+ host = p.netloc.removeprefix("www.")
+ path = p.path.rstrip("/")
+ query = urlencode(sorted(
+ (k, v) for k, v in parse_qsl(p.query) if not _TRACKING_PARAMS_RE.match(k)
+ ))
+ return urlunparse(("https", host, path, "", query, ""))
+
+
+def _wire_signature(body: str, title: str) -> str | None:
+ """Wire-marker + body-opening signature for press-release clustering.
+
+ Keyed on the body head, NOT the title — outlets retitle syndicated
+ copy, but the wire text itself opens identically.
+ """
+ head = (body or "")[:1500].lower()
+ marker = next((m for m in WIRE_MARKERS if m in head), None)
+ if marker is None:
+ return None
+ tokens = re.findall(r"[a-z0-9]{4,}", head)[:10]
+ return f"{marker}|{' '.join(tokens)}"
+
+
+def compute_independence(vault, tag: str | None = None) -> dict:
+ """Cluster derivative sources, write independence scores. Returns summary."""
+ conn = vault.db
+ query = (
+ "SELECT n.id, n.source, n.title, n.created, nc.body_plain "
+ "FROM notes n JOIN note_content nc ON nc.note_id = n.id "
+ "WHERE n.source IS NOT NULL AND n.type NOT IN ('index')"
+ )
+ params: tuple = ()
+ if tag:
+ query += " AND n.id IN (SELECT note_id FROM tags WHERE tag = ?)"
+ params = (tag,)
+ rows = [dict(r) for r in conn.execute(query, params).fetchall()]
+
+ # Union-find over note ids
+ parent: dict[str, str] = {r["id"]: r["id"] for r in rows}
+ cluster_kind: dict[frozenset, str] = {}
+
+ def find(x: str) -> str:
+ while parent[x] != x:
+ parent[x] = parent[parent[x]]
+ x = parent[x]
+ return x
+
+ def union(a: str, b: str, kind: str) -> None:
+ ra, rb = find(a), find(b)
+ if ra != rb:
+ parent[rb] = ra
+ cluster_kind[frozenset((a, b))] = kind
+
+ # 1. Canonical-URL identity
+ by_url: dict[str, list[dict]] = {}
+ for r in rows:
+ by_url.setdefault(canonical_url(r["source"]), []).append(r)
+ for group in by_url.values():
+ for other in group[1:]:
+ union(group[0]["id"], other["id"], "url")
+
+ # 2. Wire-service signature
+ by_wire: dict[str, list[dict]] = {}
+ for r in rows:
+ sig = _wire_signature(r["body_plain"], r["title"])
+ if sig:
+ by_wire.setdefault(sig, []).append(r)
+ for group in by_wire.values():
+ for other in group[1:]:
+ union(group[0]["id"], other["id"], "wire")
+
+ # 3. Near-duplicate bodies (pairwise Jaccard on shingles; vaults are
+ # small enough per-tag, and dedup's MinHash/LSH path exists for scale)
+ shingles = {r["id"]: shingle(r["body_plain"] or "", n=vault.config.dedup.shingle_size) for r in rows}
+ ids = [r["id"] for r in rows]
+ for i in range(len(ids)):
+ for j in range(i + 1, len(ids)):
+ if find(ids[i]) == find(ids[j]):
+ continue
+ if jaccard(shingles[ids[i]], shingles[ids[j]]) >= NEAR_DUP_THRESHOLD:
+ union(ids[i], ids[j], "body")
+
+ # Materialize clusters; root = earliest fetched (the upstream original)
+ groups: dict[str, list[dict]] = {}
+ for r in rows:
+ groups.setdefault(find(r["id"]), []).append(r)
+
+ clusters = []
+ scored = 0
+ for members in groups.values():
+ if len(members) == 1:
+ conn.execute("UPDATE notes SET independence = 1.0 WHERE id = ?", (members[0]["id"],))
+ scored += 1
+ continue
+ members.sort(key=lambda r: r["created"] or "")
+ root = members[0]
+ share = round(1.0 / len(members), 4)
+ conn.execute("UPDATE notes SET independence = 1.0 WHERE id = ?", (root["id"],))
+ kinds = {
+ cluster_kind.get(frozenset((a["id"], b["id"])))
+ for a in members for b in members
+ if cluster_kind.get(frozenset((a["id"], b["id"])))
+ }
+ for m in members[1:]:
+ conn.execute("UPDATE notes SET independence = ? WHERE id = ?", (share, m["id"]))
+ scored += len(members)
+ clusters.append({
+ "root": root["id"],
+ "members": [m["id"] for m in members[1:]],
+ "size": len(members),
+ "kind": "+".join(sorted(k for k in kinds if k)) or "mixed",
+ })
+ conn.commit()
+ return {"scored": scored, "clusters": clusters}
diff --git a/src/hyperresearch/core/migrations.py b/src/hyperresearch/core/migrations.py
index 2a86e0d..f4cf1ac 100644
--- a/src/hyperresearch/core/migrations.py
+++ b/src/hyperresearch/core/migrations.py
@@ -158,6 +158,88 @@ def _migrate_v8_source_analysis_note_type(conn: sqlite3.Connection) -> None:
""")
+def _migrate_v9_source_ranking(conn: sqlite3.Connection) -> None:
+ """Source-ranking schema (2.0 phase 2): note score columns, claims, api_cache.
+
+ Additive only — existing rows keep NULL scores until `hpr sources score` /
+ `hpr graph rank` populate them. Idempotent via column/table existence checks.
+ """
+ existing = {row[1] for row in conn.execute("PRAGMA table_info(notes)")}
+ new_columns = {
+ "doi": "TEXT",
+ "utility_score": "REAL",
+ "authority_score": "REAL",
+ "centrality_score": "REAL",
+ "independence": "REAL",
+ "citation_count": "INTEGER",
+ "venue": "TEXT",
+ "is_retracted": "INTEGER NOT NULL DEFAULT 0",
+ "quality_score": "REAL",
+ }
+ for name, decl in new_columns.items():
+ if name not in existing:
+ conn.execute(f"ALTER TABLE notes ADD COLUMN {name} {decl}")
+
+ conn.executescript("""
+ CREATE TABLE IF NOT EXISTS claims (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
+ claim TEXT NOT NULL,
+ claim_hash TEXT NOT NULL,
+ quoted_support TEXT,
+ numbers TEXT,
+ confidence TEXT,
+ evidence_type TEXT,
+ stance_target TEXT,
+ stance TEXT,
+ vault_tag TEXT,
+ ingested_at TEXT NOT NULL,
+ UNIQUE (note_id, claim_hash)
+ );
+ CREATE INDEX IF NOT EXISTS idx_claims_note ON claims(note_id);
+ CREATE INDEX IF NOT EXISTS idx_claims_vault_tag ON claims(vault_tag);
+ CREATE INDEX IF NOT EXISTS idx_claims_stance_target ON claims(stance_target);
+
+ CREATE VIRTUAL TABLE IF NOT EXISTS claims_fts USING fts5(
+ claim_id UNINDEXED,
+ claim,
+ quoted_support,
+ tokenize='porter unicode61'
+ );
+
+ CREATE TABLE IF NOT EXISTS api_cache (
+ url TEXT PRIMARY KEY,
+ body TEXT NOT NULL,
+ fetched_at TEXT NOT NULL
+ );
+ """)
+
+
+_MIGRATE_V10_ESCALATIONS_SQL = """
+CREATE TABLE IF NOT EXISTS escalations (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ url TEXT NOT NULL,
+ reason TEXT NOT NULL
+ CHECK (reason IN ('login_wall','bot_block','captcha','fetch_failed','interactive_needed','scholar_search')),
+ requested_by TEXT,
+ suggested_by TEXT,
+ utility_score REAL,
+ vault_tag TEXT,
+ status TEXT NOT NULL DEFAULT 'queued'
+ CHECK (status IN ('queued','in_progress','fetched','needs_human','abandoned')),
+ attempts INTEGER NOT NULL DEFAULT 0,
+ note_id TEXT,
+ claimed_by TEXT,
+ detail TEXT,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ UNIQUE (url, vault_tag)
+);
+CREATE INDEX IF NOT EXISTS idx_escalations_status ON escalations(status);
+CREATE INDEX IF NOT EXISTS idx_escalations_tag ON escalations(vault_tag);
+"""
+
+
MIGRATIONS: dict[int, str | Callable[[sqlite3.Connection], None]] = {
2: """
CREATE TABLE IF NOT EXISTS tag_aliases (
@@ -202,6 +284,8 @@ CREATE INDEX IF NOT EXISTS idx_assets_type ON assets(type);
6: _migrate_v6_tier_content_type,
7: _migrate_v7_interim_note_type,
8: _migrate_v8_source_analysis_note_type,
+ 9: _migrate_v9_source_ranking,
+ 10: _MIGRATE_V10_ESCALATIONS_SQL,
}
diff --git a/src/hyperresearch/core/profiles.py b/src/hyperresearch/core/profiles.py
new file mode 100644
index 0000000..71d388f
--- /dev/null
+++ b/src/hyperresearch/core/profiles.py
@@ -0,0 +1,423 @@
+"""Pipeline profiles — the tunable scale parameters of the research pipeline.
+
+A profile is a named, validated bundle of every knob that governs research
+scale: source-count gates, fetcher fan-out, loci caps, depth budgets, draft
+counts, word targets, critic caps, and per-agent model assignments.
+
+Built-in profiles `light` and `full` reproduce the V8 pipeline's shipped
+values exactly. Users can override keys or define new profiles in
+`.hyperresearch/config.toml`:
+
+ [profile.full] # tweak a built-in
+ source_min = 60
+
+ [profile.dissertation] # define a new profile
+ extends = "full"
+ source_min = 250
+ loci_max = 20
+
+Profiles are consumed in two places:
+ 1. Prompt templating (`core/render.py`) — skill/agent prompt templates
+ reference profile values at install-render time.
+ 2. `hpr profile show <name> -j` — agents read the resolved profile at run
+ time when they need a number the rendered prose doesn't carry.
+"""
+
+from __future__ import annotations
+
+import tomllib
+from pathlib import Path
+
+from pydantic import BaseModel, ConfigDict, Field, field_validator
+
+Range = tuple[int, int]
+
+
+class ModelMap(BaseModel):
+ """Per-agent model assignments, rendered into installed agent frontmatter.
+
+ Values are whatever Claude Code accepts in an agent's `model:` line — an
+ alias (`haiku`, `sonnet`, `opus`) or a full model ID. Override per agent
+ in a `[profile.<name>]` overlay:
+
+ [profile.premier]
+ models = { fetcher = "haiku" }
+
+ Unspecified agents keep their defaults (the dict replaces wholesale, then
+ defaults fill the gaps).
+ """
+
+ model_config = ConfigDict(extra="forbid", frozen=True, protected_namespaces=())
+
+ fetcher: str = "sonnet"
+ source_analyst: str = "sonnet"
+ loci_analyst: str = "sonnet"
+ depth_investigator: str = "sonnet"
+ corpus_critic: str = "sonnet"
+ cite_checker: str = "sonnet"
+ browser_fetcher: str = "sonnet"
+ draft_orchestrator: str = "opus"
+ synthesizer: str = "opus"
+ critics: str = "opus"
+ patcher: str = "opus"
+ polish_auditor: str = "opus"
+ readability_recommender: str = "opus"
+
+ @field_validator("*")
+ @classmethod
+ def _non_empty(cls, v: str) -> str:
+ if not isinstance(v, str) or not v.strip():
+ raise ValueError("model assignment must be a non-empty string")
+ return v.strip()
+
+
+class Profile(BaseModel):
+ """One resolved pipeline profile. All ranges are (low, high) inclusive."""
+
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ name: str
+ extends: str | None = None
+ # One-line human description shown by `hpr profile list`.
+ description: str = ""
+
+ # Step routing
+ steps: tuple[int, ...]
+
+ # Chaptered execution (dissertation-scale). chapters == (0, 0) means
+ # unchaptered — the classic flat pipeline. When non-zero, the router
+ # invokes step 1.5 (chapter partition) after decompose and loops steps
+ # 2-10 per chapter, each chapter staying within chapter_source_target.
+ chapters: Range = (0, 0)
+ chapter_concurrency: int = 1
+ chapter_source_target: Range = (40, 80)
+
+ # Step 2 — width sweep
+ source_min: int
+ source_target: Range
+ planned_searches: Range
+ candidate_urls: Range
+ deduped_urls: Range
+ batch_size: Range
+ batch_count: Range
+ waves: Range
+ wave1_fetchers: Range
+ wave2_fetchers: Range
+ wave3_fetchers: Range
+ adversarial_searches_min: int
+ utility_scoring: bool
+ source_analyst_cap: int
+ source_analyst_word_trigger: int
+ fetcher_chase: Range
+ fetcher_chase_cap: int
+
+ # Steps 4-5 — loci + depth
+ loci_analysts: int
+ loci_max: int
+ depth_budget_total: int
+ # (min_composite_score, max_source_budget) rows, highest bracket first
+ depth_budget_brackets: tuple[Range, ...]
+ investigator_max: int
+ depth_default_budget: int
+
+ # Steps 6-8 — tensions + corpus critic
+ comparisons_tensions: Range
+ source_tensions: Range
+ tension_survey: Range
+ tension_full_reads: Range
+ corpus_critic_gaps: Range
+ corpus_critic_fetchers: Range
+
+ # Step 9 — evidence digest
+ claims_cap: Range
+ claims_min: int
+
+ # Steps 10-11 — drafting + synthesis
+ draft_count: int
+ single_draft_reads: Range
+ must_read: dict[str, Range]
+ word_targets: dict[str, Range]
+ citation_density_min: float
+ citation_totals: dict[str, Range]
+
+ # Steps 12-16 — critics, gap-fetch, readability
+ critic_finding_caps: dict[str, int]
+ gap_fetch_cap: int
+ gap_fetch_fetchers: Range
+ readability_rec_cap: int
+
+ # Pacing + expectations
+ vault_check_interval_s: int
+ wave_done_ratio: float
+ time_estimate: str
+
+ models: ModelMap = Field(default_factory=ModelMap)
+
+ @field_validator(
+ "source_target", "planned_searches", "candidate_urls", "deduped_urls",
+ "batch_size", "batch_count", "waves", "wave1_fetchers", "wave2_fetchers",
+ "wave3_fetchers", "fetcher_chase", "comparisons_tensions", "source_tensions",
+ "tension_survey", "tension_full_reads", "corpus_critic_gaps",
+ "corpus_critic_fetchers", "claims_cap", "single_draft_reads", "gap_fetch_fetchers",
+ "chapters", "chapter_source_target",
+ )
+ @classmethod
+ def _range_ordered(cls, v: Range) -> Range:
+ low, high = v
+ if low > high:
+ raise ValueError(f"range low {low} > high {high}")
+ return v
+
+ @field_validator("steps")
+ @classmethod
+ def _steps_valid(cls, v: tuple[int, ...]) -> tuple[int, ...]:
+ if not v:
+ raise ValueError("steps must not be empty")
+ for s in v:
+ if not 1 <= s <= 16:
+ raise ValueError(f"step {s} out of range 1-16")
+ if list(v) != sorted(v):
+ raise ValueError("steps must be ascending")
+ return v
+
+
+# ---------------------------------------------------------------------------
+# Built-in profiles — values are the V8 pipeline's shipped constants.
+# The prompt-template golden tests pin these: changing a value here without
+# updating the golden fixtures is a test failure, not a silent drift.
+# ---------------------------------------------------------------------------
+
+_FULL: dict = {
+ "name": "full",
+ "description": "The standard adversarially-audited pipeline — 55–80 sources, triple draft, full critic suite.",
+ "steps": tuple(range(1, 17)),
+ "source_min": 45,
+ "source_target": (55, 80),
+ "planned_searches": (40, 100),
+ "candidate_urls": (80, 120),
+ "deduped_urls": (60, 100),
+ "batch_size": (8, 12),
+ "batch_count": (10, 12),
+ "waves": (2, 3),
+ "wave1_fetchers": (10, 12),
+ "wave2_fetchers": (3, 5),
+ "wave3_fetchers": (2, 3),
+ "adversarial_searches_min": 5,
+ "utility_scoring": True,
+ "source_analyst_cap": 6,
+ "source_analyst_word_trigger": 5000,
+ "fetcher_chase": (3, 8),
+ "fetcher_chase_cap": 8,
+ "loci_analysts": 2,
+ "loci_max": 6,
+ "depth_budget_total": 40,
+ "depth_budget_brackets": ((30, 15), (20, 10), (10, 5), (0, 3)),
+ "investigator_max": 6,
+ "depth_default_budget": 10,
+ "comparisons_tensions": (3, 5),
+ "source_tensions": (3, 7),
+ "tension_survey": (15, 20),
+ "tension_full_reads": (8, 12),
+ "corpus_critic_gaps": (3, 8),
+ "corpus_critic_fetchers": (2, 4),
+ "claims_cap": (80, 120),
+ "claims_min": 30,
+ "draft_count": 3,
+ "single_draft_reads": (8, 15),
+ "must_read": {"argumentative": (35, 50), "structured": (25, 40), "short": (20, 30)},
+ "word_targets": {"short": (500, 2000), "structured": (2000, 5000), "argumentative": (5000, 10000)},
+ "citation_density_min": 2.0,
+ "citation_totals": {"argumentative": (80, 150), "structured": (40, 80), "short": (15, 30)},
+ "critic_finding_caps": {"dialectic": 12, "depth": 12, "width": 10, "instruction": 15},
+ "gap_fetch_cap": 5,
+ "gap_fetch_fetchers": (2, 4),
+ "readability_rec_cap": 50,
+ "vault_check_interval_s": 60,
+ "wave_done_ratio": 0.8,
+ "time_estimate": "~1.5–2.5 hours",
+}
+
+_LIGHT: dict = {
+ **_FULL,
+ "name": "light",
+ "description": "Fast bounded answers — 5 steps, single draft, 15–25 sources, no critics.",
+ "steps": (1, 2, 10, 15, 16),
+ "source_min": 10,
+ "source_target": (15, 25),
+ "planned_searches": (8, 20),
+ "candidate_urls": (20, 40),
+ "deduped_urls": (15, 30),
+ "batch_count": (2, 3),
+ "waves": (1, 2),
+ "wave1_fetchers": (3, 5),
+ "utility_scoring": False,
+ "draft_count": 1,
+ "time_estimate": "~30–40 min",
+}
+
+_PREMIER: dict = {
+ **_FULL,
+ "name": "premier",
+ "description": "Maximum flat-pipeline scale — 100–130 sources, double depth budget, extended reports.",
+ # Width: roughly double the sweep. Every funnel stage downstream is widened
+ # too — raising only the fetch targets would strand the extra corpus in the
+ # vault (draft readers, claims caps, and word targets gate what reaches the
+ # page).
+ "source_min": 90,
+ "source_target": (100, 130),
+ "planned_searches": (80, 160),
+ "candidate_urls": (150, 220),
+ "deduped_urls": (120, 180),
+ "batch_count": (14, 18),
+ "wave1_fetchers": (14, 18),
+ "wave2_fetchers": (5, 8),
+ "wave3_fetchers": (3, 5),
+ "adversarial_searches_min": 8,
+ "source_analyst_cap": 10,
+ # Depth: more loci, double the depth budget
+ "loci_analysts": 3,
+ "loci_max": 10,
+ "depth_budget_total": 80,
+ "depth_budget_brackets": ((30, 25), (20, 15), (10, 8), (0, 5)),
+ "investigator_max": 10,
+ "depth_default_budget": 15,
+ # Tensions + corpus critic
+ "comparisons_tensions": (4, 7),
+ "source_tensions": (4, 9),
+ "tension_survey": (20, 30),
+ "tension_full_reads": (10, 15),
+ "corpus_critic_gaps": (4, 10),
+ "corpus_critic_fetchers": (3, 5),
+ # Evidence funnel
+ "claims_cap": (150, 220),
+ "claims_min": 50,
+ "must_read": {"argumentative": (50, 70), "structured": (35, 55), "short": (20, 30)},
+ "word_targets": {"short": (500, 2000), "structured": (3000, 8000), "argumentative": (8000, 16000)},
+ "citation_totals": {"argumentative": (120, 220), "structured": (60, 110), "short": (15, 30)},
+ "critic_finding_caps": {"dialectic": 16, "depth": 16, "width": 14, "instruction": 18},
+ "gap_fetch_cap": 8,
+ "gap_fetch_fetchers": (3, 5),
+ "readability_rec_cap": 60,
+ # Longer waves → slightly relaxed pacing
+ "vault_check_interval_s": 90,
+ "time_estimate": "~3–5 hours",
+}
+
+_DISSERTATION: dict = {
+ **_FULL,
+ "name": "dissertation",
+ "description": "Chaptered dissertation-scale runs — 300–450 sources across 4–10 chapters, 25K–80K words.",
+ # Chaptered: steps 2-10 loop per chapter; global reconcile/synthesis on top.
+ "chapters": (4, 10),
+ "chapter_concurrency": 2,
+ "chapter_source_target": (40, 80),
+ # Global totals — allocated per chapter, each within the proven envelope.
+ "source_min": 250,
+ "source_target": (300, 450),
+ "loci_max": 20,
+ "depth_budget_total": 160,
+ "claims_cap": (400, 600),
+ "claims_min": 100,
+ # One draft per chapter; angle diversity comes from the chapters themselves.
+ "draft_count": 1,
+ "must_read": {
+ **_FULL["must_read"],
+ "dissertation": (35, 50), # per-chapter reading list
+ },
+ "word_targets": {
+ **_FULL["word_targets"],
+ "dissertation": (25000, 80000),
+ },
+ "citation_totals": {
+ **_FULL["citation_totals"],
+ "dissertation": (300, 600),
+ },
+ "critic_finding_caps": {"dialectic": 24, "depth": 24, "width": 20, "instruction": 30},
+ "gap_fetch_cap": 10,
+ "readability_rec_cap": 100,
+ # Multi-hour pacing
+ "vault_check_interval_s": 120,
+ "time_estimate": "~4–8 hours",
+}
+
+# Listed in ascending scale order — `hpr profile list` follows this order.
+BUILTIN_PROFILES: dict[str, dict] = {
+ "light": _LIGHT,
+ "full": _FULL,
+ "premier": _PREMIER,
+ "dissertation": _DISSERTATION,
+}
+
+# Profiles that make sense as the installed scale "gear" — the profile whose
+# numbers get rendered into the step skills and agent prompts (`p.*`).
+# `light` and `dissertation` are run-time TIERS: light is auto-classified by
+# step 1, dissertation is opt-in per run (its chapter loop reads
+# `dissertation.*` values by name and runs each chapter inside the gear's
+# envelope). Installing either AS the gear would bake the wrong numbers into
+# the flat pipeline.
+GEAR_PROFILES: tuple[str, ...] = ("full", "premier")
+
+
+class ProfileError(Exception):
+ """Raised for unknown profiles or invalid profile definitions."""
+
+
+def _load_user_overlays(config_path: Path | None) -> dict[str, dict]:
+ """Read `[profile.<name>]` tables from a config.toml. Missing file → {}."""
+ if config_path is None or not config_path.exists():
+ return {}
+ with open(config_path, "rb") as f:
+ data = tomllib.load(f)
+ profiles = data.get("profile", {})
+ if not isinstance(profiles, dict):
+ raise ProfileError("[profile] must be a table of profile tables")
+ out: dict[str, dict] = {}
+ for name, table in profiles.items():
+ if not isinstance(table, dict):
+ raise ProfileError(f"[profile.{name}] must be a table")
+ out[name] = table
+ return out
+
+
+def list_profiles(config_path: Path | None = None) -> list[str]:
+ """All available profile names: built-ins plus user-defined."""
+ names = dict.fromkeys(BUILTIN_PROFILES)
+ names.update(dict.fromkeys(_load_user_overlays(config_path)))
+ return list(names)
+
+
+def resolve_profile(name: str, config_path: Path | None = None) -> Profile:
+ """Resolve a profile by name: built-in defaults + user overlay, validated.
+
+ Resolution:
+ - built-in name: built-in values, then user `[profile.<name>]` keys on top.
+ - new name: must exist in config; starts from its `extends` base
+ (default "full"), then its own keys on top.
+ """
+ overlays = _load_user_overlays(config_path)
+
+ if name in BUILTIN_PROFILES:
+ base = dict(BUILTIN_PROFILES[name])
+ elif name in overlays:
+ extends = overlays[name].get("extends", "full")
+ if extends not in BUILTIN_PROFILES:
+ raise ProfileError(
+ f"profile '{name}' extends unknown base '{extends}' "
+ f"(must be one of: {', '.join(BUILTIN_PROFILES)})"
+ )
+ base = dict(BUILTIN_PROFILES[extends])
+ else:
+ available = ", ".join(list_profiles(config_path))
+ raise ProfileError(f"unknown profile '{name}'. Available: {available}")
+
+ overlay = dict(overlays.get(name, {}))
+ overlay.pop("extends", None)
+
+ merged = {**base, **overlay, "name": name}
+ if name not in BUILTIN_PROFILES:
+ merged["extends"] = overlays[name].get("extends", "full")
+
+ try:
+ return Profile(**merged)
+ except Exception as exc:
+ raise ProfileError(f"invalid profile '{name}': {exc}") from exc
diff --git a/src/hyperresearch/core/quality.py b/src/hyperresearch/core/quality.py
new file mode 100644
index 0000000..f418357
--- /dev/null
+++ b/src/hyperresearch/core/quality.py
@@ -0,0 +1,75 @@
+"""Composite source-quality scoring.
+
+quality_score = renormalized weighted sum of the available components:
+
+ tier weight (from [ranking] tier_* map; None when tier unset)
+ utility_score / 18 (step-2 fetch-selection composite, when persisted)
+ authority_score (vault-relative citation percentile, when enriched)
+ centrality_score (vault PageRank, when computed)
+
+Missing components renormalize — a source with only a tier still gets a
+meaningful score instead of being zeroed for lacking a DOI. Retracted
+sources are floored at `retraction_floor` regardless of everything else.
+
+Scores live only in the DB (recomputable cache); they are never written to
+frontmatter.
+"""
+
+from __future__ import annotations
+
+from hyperresearch.core.config import RankingSettings
+
+UTILITY_MAX = 18.0
+
+
+def compute_quality_for_row(
+ ranking: RankingSettings,
+ tier: str | None,
+ utility_score: float | None,
+ authority_score: float | None,
+ centrality_score: float | None,
+ is_retracted: bool,
+) -> float | None:
+ """Composite for one note. None when no component is available at all."""
+ if is_retracted:
+ return ranking.retraction_floor
+
+ components: list[tuple[float, float]] = [] # (weight, value)
+ tier_w = ranking.tier_weight(tier)
+ if tier_w is not None:
+ components.append((ranking.w_tier, tier_w))
+ if utility_score is not None:
+ components.append((ranking.w_utility, min(max(utility_score / UTILITY_MAX, 0.0), 1.0)))
+ if authority_score is not None:
+ components.append((ranking.w_authority, min(max(authority_score, 0.0), 1.0)))
+ if centrality_score is not None:
+ components.append((ranking.w_centrality, min(max(centrality_score, 0.0), 1.0)))
+
+ if not components:
+ return None
+ total_weight = sum(w for w, _ in components)
+ if total_weight <= 0:
+ return None
+ return sum(w * v for w, v in components) / total_weight
+
+
+def compute_quality_scores(conn, ranking: RankingSettings) -> int:
+ """Recompute quality_score for every note. Returns count updated."""
+ rows = conn.execute(
+ "SELECT id, tier, utility_score, authority_score, centrality_score, is_retracted "
+ "FROM notes"
+ ).fetchall()
+ updated = 0
+ for row in rows:
+ q = compute_quality_for_row(
+ ranking,
+ row["tier"],
+ row["utility_score"],
+ row["authority_score"],
+ row["centrality_score"],
+ bool(row["is_retracted"]),
+ )
+ conn.execute("UPDATE notes SET quality_score = ? WHERE id = ?", (q, row["id"]))
+ updated += 1
+ conn.commit()
+ return updated
diff --git a/src/hyperresearch/core/render.py b/src/hyperresearch/core/render.py
new file mode 100644
index 0000000..a11cabd
--- /dev/null
+++ b/src/hyperresearch/core/render.py
@@ -0,0 +1,104 @@
+"""Prompt-template rendering — profile values into skill/agent prompts.
+
+Skill files and agent prompt bodies are Jinja templates with NON-STANDARD
+delimiters, because the prompts themselves legitimately contain `{{ ... }}`
+(spawn-template placeholders like `{{paste research/query-<vault_tag>.md}}`)
+and `{ ... }` (JSON examples):
+
+ variables: << p.source_min >>
+ blocks: <% if ... %> ... <% endif %>
+ comments: <# ... #>
+
+Context exposed to templates:
+ p — the primary profile (default: full)
+ <name> — every available profile by name (e.g. `full`, `light`),
+ so tier tables can reference both tiers in one file.
+
+Filters:
+ dash — join a (low, high) range with an en dash (U+2013)
+ hyphen — join a (low, high) range with a hyphen: (1, 2) -> "1-2"
+
+Rendering is STRICT: an unknown variable raises instead of silently emitting
+an empty string — a typo in a template must fail the install/tests, not ship
+a prompt with a hole in it.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from jinja2 import Environment, StrictUndefined
+
+from hyperresearch.core.profiles import Profile, list_profiles, resolve_profile
+
+EN_DASH = "–"
+
+
+def _dash(value) -> str:
+ low, high = value
+ return f"{low}{EN_DASH}{high}"
+
+
+def _hyphen(value) -> str:
+ low, high = value
+ return f"{low}-{high}"
+
+
+def prompt_env() -> Environment:
+ env = Environment(
+ variable_start_string="<<",
+ variable_end_string=">>",
+ block_start_string="<%",
+ block_end_string="%>",
+ comment_start_string="<#",
+ comment_end_string="#>",
+ undefined=StrictUndefined,
+ keep_trailing_newline=True,
+ autoescape=False,
+ )
+ env.filters["dash"] = _dash
+ env.filters["hyphen"] = _hyphen
+ return env
+
+
+def build_render_context(
+ config_path: Path | None = None,
+ primary: str = "full",
+) -> dict[str, Profile]:
+ """Resolve every available profile; expose each by name plus `p` (primary)."""
+ profiles = {name: resolve_profile(name, config_path) for name in list_profiles(config_path)}
+ if primary not in profiles:
+ # resolve_profile raises a helpful error for unknown names
+ profiles[primary] = resolve_profile(primary, config_path)
+ return {"p": profiles[primary], **profiles}
+
+
+def render_prompt(text: str, context: dict[str, Profile]) -> str:
+ """Render one prompt template with the given profile context."""
+ return prompt_env().from_string(text).render(**context)
+
+
+def render_header(profile_name: str, version: str) -> str:
+ """Provenance comment for installed (rendered) prompt files.
+
+ Inserted AFTER the YAML frontmatter block — a comment before the opening
+ `---` would break frontmatter parsing.
+ """
+ return (
+ f"<!-- rendered from profile \"{profile_name}\" (hyperresearch {version}) "
+ "— edit the profile or the package template, not this file -->"
+ )
+
+
+def insert_after_frontmatter(content: str, line: str) -> str:
+ """Insert `line` on its own line after the closing frontmatter delimiter.
+
+ If the content has no leading frontmatter, prepend the line.
+ """
+ if content.startswith("---"):
+ end = content.find("\n---", 3)
+ if end != -1:
+ insert_at = content.find("\n", end + 1)
+ if insert_at != -1:
+ return content[: insert_at + 1] + line + "\n" + content[insert_at + 1 :]
+ return line + "\n" + content
diff --git a/src/hyperresearch/core/runs.py b/src/hyperresearch/core/runs.py
new file mode 100644
index 0000000..17d1ad5
--- /dev/null
+++ b/src/hyperresearch/core/runs.py
@@ -0,0 +1,435 @@
+"""Per-run workspaces and the run manifest.
+
+A run is one /hyperresearch invocation. Everything run-scoped lives under
+`research/runs/<vault_tag>/`:
+
+ run.json — the manifest (this module's contract)
+ events.jsonl — append-only event log (step boundaries, spawns, fetches)
+ query.md — canonical verbatim research query
+ scaffold.md, prompt-decomposition.json, loci.json, comparisons.md, ...
+ temp/ — scratch artifacts (claims JSONs, drafts, notes)
+ chapters/chN/ — per-chapter artifact sets (dissertation profile)
+
+The manifest replaces the 1.x "find the highest-numbered artifact on disk"
+recovery heuristic with explicit, durable state: per-step status, per-chapter
+status, spend counters, and a budget ceiling. The orchestrator updates it at
+step boundaries via `hpr run ...` commands; `hpr run resume <tag>` computes
+the exact next position.
+
+Vault notes stay global — runs are ephemeral workspaces over the compounding
+vault. Final reports ship to `research/notes/final_report_<vault_tag>.md`.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import UTC, datetime
+from pathlib import Path
+
+MANIFEST_VERSION = 1
+MANIFEST_NAME = "run.json"
+EVENTS_NAME = "events.jsonl"
+
+RUN_STATUSES = ("running", "paused", "blocked", "done", "failed", "aborted")
+STEP_STATUSES = ("pending", "running", "done", "skipped", "failed")
+
+# A run whose manifest hasn't been touched in this long is flagged
+# possibly-stalled by `hpr run status`.
+STALL_MINUTES = 30
+
+
+class RunError(Exception):
+ pass
+
+
+def _now() -> str:
+ return datetime.now(UTC).isoformat()
+
+
+def manifest_path(vault, vault_tag: str) -> Path:
+ return vault.run_dir(vault_tag) / MANIFEST_NAME
+
+
+def init_run(
+ vault,
+ vault_tag: str,
+ profile: str = "full",
+ budget_usd: float | None = None,
+ query: str | None = None,
+) -> dict:
+ """Scaffold research/runs/<vault_tag>/ and write a fresh manifest.
+
+ Idempotent: re-running on an existing run returns the existing manifest
+ unchanged (so a recovering orchestrator can call it safely).
+ """
+ from hyperresearch.core.profiles import resolve_profile
+
+ run_dir = vault.run_dir(vault_tag)
+ mpath = run_dir / MANIFEST_NAME
+ if mpath.exists():
+ return load_manifest(vault, vault_tag)
+
+ resolved = resolve_profile(profile, vault.config_path)
+
+ run_dir.mkdir(parents=True, exist_ok=True)
+ (run_dir / "temp").mkdir(exist_ok=True)
+
+ if query is not None:
+ (run_dir / "query.md").write_text(query, encoding="utf-8")
+
+ manifest = {
+ "manifest_version": MANIFEST_VERSION,
+ "vault_tag": vault_tag,
+ "profile": profile,
+ "profile_steps": [str(s) for s in resolved.steps],
+ "status": "running",
+ "started_at": _now(),
+ "updated_at": _now(),
+ "budget_usd": budget_usd,
+ "blocked_on": None,
+ "steps": {},
+ "chapters": {},
+ "spend": {
+ "estimated_usd": 0.0,
+ "sources_fetched": 0,
+ "notes_written": 0,
+ "agents_spawned": 0,
+ },
+ }
+ _save(vault, vault_tag, manifest)
+ return manifest
+
+
+def load_manifest(vault, vault_tag: str) -> dict:
+ mpath = manifest_path(vault, vault_tag)
+ if not mpath.exists():
+ raise RunError(f"no run '{vault_tag}' (missing {mpath})")
+ try:
+ return json.loads(mpath.read_text(encoding="utf-8"))
+ except json.JSONDecodeError as e:
+ raise RunError(f"corrupt manifest for run '{vault_tag}': {e}") from e
+
+
+def _save(vault, vault_tag: str, manifest: dict) -> None:
+ manifest["updated_at"] = _now()
+ mpath = manifest_path(vault, vault_tag)
+ tmp = mpath.with_suffix(".json.tmp")
+ tmp.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+ tmp.replace(mpath) # atomic on same filesystem
+
+
+def record_event(vault, vault_tag: str, event: dict) -> None:
+ """Append one event to events.jsonl and touch the manifest heartbeat."""
+ run_dir = vault.run_dir(vault_tag)
+ if not (run_dir / MANIFEST_NAME).exists():
+ raise RunError(f"no run '{vault_tag}'")
+ event = {"at": _now(), **event}
+ with open(run_dir / EVENTS_NAME, "a", encoding="utf-8") as f:
+ f.write(json.dumps(event) + "\n")
+ manifest = load_manifest(vault, vault_tag)
+ _save(vault, vault_tag, manifest) # heartbeat
+
+
+def set_step(
+ vault,
+ vault_tag: str,
+ step: str,
+ status: str,
+ chapter: str | None = None,
+) -> dict:
+ """Mark a step's status. Steps are keyed as strings ("1", "1.5", "11g")."""
+ if status not in STEP_STATUSES:
+ raise RunError(f"invalid step status '{status}' (one of {STEP_STATUSES})")
+ manifest = load_manifest(vault, vault_tag)
+ entry = manifest["steps"].setdefault(str(step), {})
+ entry["status"] = status
+ if status == "running" and "started_at" not in entry:
+ entry["started_at"] = _now()
+ if status in ("done", "skipped", "failed"):
+ entry["finished_at"] = _now()
+ if chapter:
+ entry["chapter"] = chapter
+ ch = manifest["chapters"].setdefault(chapter, {})
+ ch["status"] = f"step-{step}-{status}"
+ _save(vault, vault_tag, manifest)
+ record_event(vault, vault_tag, {"type": "step", "step": str(step), "status": status, "chapter": chapter})
+ return manifest
+
+
+def set_chapter(vault, vault_tag: str, chapter: str, **fields) -> dict:
+ """Create/update a chapter entry (title, status, sources, ...)."""
+ manifest = load_manifest(vault, vault_tag)
+ ch = manifest["chapters"].setdefault(chapter, {})
+ ch.update({k: v for k, v in fields.items() if v is not None})
+ _save(vault, vault_tag, manifest)
+ return manifest
+
+
+def add_spend(
+ vault,
+ vault_tag: str,
+ estimated_usd: float = 0.0,
+ sources_fetched: int = 0,
+ notes_written: int = 0,
+ agents_spawned: int = 0,
+) -> dict:
+ manifest = load_manifest(vault, vault_tag)
+ spend = manifest["spend"]
+ spend["estimated_usd"] = round(spend.get("estimated_usd", 0.0) + estimated_usd, 4)
+ spend["sources_fetched"] = spend.get("sources_fetched", 0) + sources_fetched
+ spend["notes_written"] = spend.get("notes_written", 0) + notes_written
+ spend["agents_spawned"] = spend.get("agents_spawned", 0) + agents_spawned
+
+ # Budget governor: crossing the ceiling flips the run to blocked. The
+ # orchestrator checks `hpr run status` at step boundaries and must pause
+ # (never silently skip tier-mandated steps — shrink fan-out instead).
+ budget = manifest.get("budget_usd")
+ if budget is not None and spend["estimated_usd"] >= budget and manifest["status"] == "running":
+ manifest["status"] = "blocked"
+ manifest["blocked_on"] = "budget"
+ _save(vault, vault_tag, manifest)
+ return manifest
+
+
+def set_status(vault, vault_tag: str, status: str, blocked_on: str | None = None) -> dict:
+ if status not in RUN_STATUSES:
+ raise RunError(f"invalid run status '{status}' (one of {RUN_STATUSES})")
+ manifest = load_manifest(vault, vault_tag)
+ manifest["status"] = status
+ manifest["blocked_on"] = blocked_on if status == "blocked" else None
+ _save(vault, vault_tag, manifest)
+ return manifest
+
+
+def list_runs(vault) -> list[dict]:
+ """All runs, newest-started first. Tolerates corrupt manifests."""
+ runs = []
+ if not vault.runs_dir.is_dir():
+ return runs
+ for child in vault.runs_dir.iterdir():
+ mpath = child / MANIFEST_NAME
+ if not mpath.exists():
+ continue
+ try:
+ m = json.loads(mpath.read_text(encoding="utf-8"))
+ except json.JSONDecodeError:
+ runs.append({"vault_tag": child.name, "status": "corrupt-manifest"})
+ continue
+ runs.append(m)
+ runs.sort(key=lambda m: m.get("started_at", ""), reverse=True)
+ return runs
+
+
+def latest_run_tag(vault) -> str | None:
+ runs = list_runs(vault)
+ return runs[0]["vault_tag"] if runs else None
+
+
+def resume_position(manifest: dict) -> dict:
+ """Compute where a run should continue.
+
+ Returns {next_step, done_steps, remaining_steps, chapters_pending}.
+ next_step is None when every profile step is done.
+ """
+ profile_steps = manifest.get("profile_steps", [])
+ steps = manifest.get("steps", {})
+ done = [s for s in profile_steps if steps.get(s, {}).get("status") in ("done", "skipped")]
+ remaining = [s for s in profile_steps if s not in done]
+ chapters_pending = [
+ name
+ for name, ch in manifest.get("chapters", {}).items()
+ if ch.get("status") not in ("done", None) and not str(ch.get("status", "")).endswith("-done")
+ ]
+ return {
+ "next_step": remaining[0] if remaining else None,
+ "done_steps": done,
+ "remaining_steps": remaining,
+ "chapters_pending": chapters_pending,
+ }
+
+
+def status_summary(vault, vault_tag: str, stall_minutes: int = STALL_MINUTES) -> dict:
+ """Manifest + derived fields (stall detection, resume position, budget)."""
+ manifest = load_manifest(vault, vault_tag)
+ summary = dict(manifest)
+ summary["resume"] = resume_position(manifest)
+
+ possibly_stalled = False
+ if manifest.get("status") == "running":
+ try:
+ updated = datetime.fromisoformat(manifest["updated_at"])
+ age_min = (datetime.now(UTC) - updated).total_seconds() / 60
+ possibly_stalled = age_min > stall_minutes
+ except (KeyError, ValueError):
+ possibly_stalled = True
+ summary["possibly_stalled"] = possibly_stalled
+
+ budget = manifest.get("budget_usd")
+ if budget:
+ spent = manifest.get("spend", {}).get("estimated_usd", 0.0)
+ summary["budget_remaining_usd"] = round(max(0.0, budget - spent), 4)
+ return summary
+
+
+def _minutes_between(start: str | None, end: str | None) -> float | None:
+ if not start or not end:
+ return None
+ try:
+ s = datetime.fromisoformat(start)
+ e = datetime.fromisoformat(end)
+ except ValueError:
+ return None
+ return round((e - s).total_seconds() / 60, 1)
+
+
+def run_report_data(vault, vault_tag: str) -> dict:
+ """Telemetry rollup for one run: per-step wall-time, spend, event counts."""
+ manifest = load_manifest(vault, vault_tag)
+
+ steps = []
+ for step_id in manifest.get("profile_steps", []):
+ entry = manifest.get("steps", {}).get(step_id, {})
+ steps.append({
+ "step": step_id,
+ "status": entry.get("status", "pending"),
+ "minutes": _minutes_between(entry.get("started_at"), entry.get("finished_at")),
+ "chapter": entry.get("chapter"),
+ })
+ # Steps recorded outside the profile list (1.5, 11g, ...) still report
+ extra = sorted(set(manifest.get("steps", {})) - set(manifest.get("profile_steps", [])))
+ for step_id in extra:
+ entry = manifest["steps"][step_id]
+ steps.append({
+ "step": step_id,
+ "status": entry.get("status", "pending"),
+ "minutes": _minutes_between(entry.get("started_at"), entry.get("finished_at")),
+ "chapter": entry.get("chapter"),
+ })
+
+ event_counts: dict[str, int] = {}
+ events_file = vault.run_dir(vault_tag) / EVENTS_NAME
+ if events_file.exists():
+ for line in events_file.read_text(encoding="utf-8").splitlines():
+ try:
+ ev = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ event_counts[ev.get("type", "unknown")] = event_counts.get(ev.get("type", "unknown"), 0) + 1
+
+ return {
+ "vault_tag": vault_tag,
+ "profile": manifest.get("profile"),
+ "status": manifest.get("status"),
+ "total_wall_minutes": _minutes_between(manifest.get("started_at"), manifest.get("updated_at")),
+ "steps": steps,
+ "chapters": manifest.get("chapters", {}),
+ "spend": manifest.get("spend", {}),
+ "budget_usd": manifest.get("budget_usd"),
+ "events": event_counts,
+ }
+
+
+def verify_run(vault, vault_tag: str) -> dict:
+ """Structural verification battery for a completed run.
+
+ The CI-able gate: report exists, headings honored, length in profile
+ range, citation density above floor, tier-mandated artifacts present,
+ cite-check findings resolved. Returns {passed, checks: [...]} — each
+ check {name, ok, detail}. Content lint rules (quote-integrity etc.)
+ run separately via `hpr lint`.
+ """
+ from hyperresearch.core.profiles import resolve_profile
+
+ manifest = load_manifest(vault, vault_tag)
+ profile = resolve_profile(manifest.get("profile", "full"), vault.config_path)
+ run_dir = vault.run_dir(vault_tag)
+ checks: list[dict] = []
+
+ def check(name: str, ok: bool, detail: str) -> None:
+ checks.append({"name": name, "ok": bool(ok), "detail": detail})
+
+ report_path = vault.root / "research" / "notes" / f"final_report_{vault_tag}.md"
+ check("report-exists", report_path.exists(), str(report_path))
+
+ report_text = ""
+ if report_path.exists():
+ report_text = report_path.read_text(encoding="utf-8-sig")
+
+ decomp_path = run_dir / "prompt-decomposition.json"
+ response_format = None
+ required_headings: list[str] = []
+ if decomp_path.exists():
+ try:
+ decomp = json.loads(decomp_path.read_text(encoding="utf-8-sig"))
+ response_format = decomp.get("response_format")
+ required_headings = decomp.get("required_section_headings", []) or []
+ except json.JSONDecodeError:
+ check("decomposition-readable", False, "prompt-decomposition.json is not valid JSON")
+
+ if response_format and response_format in profile.word_targets:
+ low, high = profile.word_targets[response_format]
+ wc = len(report_text.split())
+ check(
+ "length-in-range",
+ low * 0.8 <= wc <= high * 1.2,
+ f"{wc} words vs target {low}-{high} ({response_format}; ±20% tolerance)",
+ )
+
+ missing = [h for h in required_headings if h not in report_text]
+ check(
+ "required-headings",
+ not missing,
+ "all present" if not missing else f"missing: {missing}",
+ )
+
+ import re as _re
+
+ cites = len(_re.findall(r"\[\d{1,3}\]", report_text)) + len(
+ _re.findall(r"\[\[[^\]]+\]\]", report_text)
+ )
+ density = cites / max(1, len(report_text)) * 1000
+ floor = 1.5 # instruction-critic's re-count trigger
+ check(
+ "citation-density",
+ density >= floor,
+ f"{density:.2f} citations/1000 chars (floor {floor})",
+ )
+
+ check(
+ "no-scaffold-leak",
+ "## User Prompt (VERBATIM" not in report_text,
+ "scaffold gospel header must not ship",
+ )
+
+ # Tier-mandated artifacts
+ steps = set(manifest.get("profile_steps", []))
+ if {"12", "14"} <= steps:
+ for name in (
+ "critic-findings-dialectic.json", "critic-findings-depth.json",
+ "critic-findings-width.json", "critic-findings-instruction.json",
+ "patch-log.json",
+ ):
+ check(f"artifact:{name}", (run_dir / name).exists(), str(run_dir / name))
+ if "15" in steps:
+ check("artifact:polish-log.json", (run_dir / "polish-log.json").exists(),
+ str(run_dir / "polish-log.json"))
+
+ # Cite-check (step 14.5): findings must exist and critical ones resolved
+ cc_findings = run_dir / "cite-check-findings.json"
+ if manifest.get("steps", {}).get("14.5", {}).get("status") == "done":
+ ok = cc_findings.exists()
+ detail = str(cc_findings)
+ if ok:
+ try:
+ findings = json.loads(cc_findings.read_text(encoding="utf-8-sig"))
+ criticals = [f for f in findings if f.get("severity") == "critical"]
+ if criticals:
+ log_path = run_dir / "cite-check-patch-log.json"
+ ok = log_path.exists()
+ detail = f"{len(criticals)} critical finding(s); patch log {'present' if ok else 'MISSING'}"
+ except json.JSONDecodeError:
+ ok = False
+ detail = "cite-check-findings.json is not valid JSON"
+ check("cite-check-resolved", ok, detail)
+
+ return {"vault_tag": vault_tag, "passed": all(c["ok"] for c in checks), "checks": checks}
diff --git a/src/hyperresearch/core/scholar.py b/src/hyperresearch/core/scholar.py
new file mode 100644
index 0000000..4f59007
--- /dev/null
+++ b/src/hyperresearch/core/scholar.py
@@ -0,0 +1,341 @@
+"""Scholarly metadata — DOI extraction and citation-graph enrichment.
+
+Two responsibilities:
+
+1. `extract_doi(url, raw_html, content)` — best-effort DOI / arXiv-id capture
+ at fetch time (URL patterns, `citation_doi` meta tags, in-body "DOI:"
+ markers). Stored in note frontmatter as `doi:`.
+
+2. `score_sources(vault, ...)` — batch enrichment of DOI-bearing notes from
+ free scholarly APIs (OpenAlex primary, Semantic Scholar for arXiv ids and
+ fallback). Populates `citation_count`, `venue`, `is_retracted` in BOTH the
+ note frontmatter (markdown stays truth) and the DB row, then recomputes
+ `authority_score` as a vault-relative log-scaled percentile.
+
+All HTTP goes through `_fetch_json`, which consults the `api_cache` table
+(TTL from `[ranking] api_cache_ttl_days`) before touching the network —
+re-scoring a vault is cheap and offline-friendly. Tests monkeypatch
+`_http_get_json`; no test ever hits the network.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import re
+import time
+from datetime import UTC, datetime, timedelta
+from urllib.parse import quote, urlparse
+
+DOI_RE = re.compile(r"\b(10\.\d{4,9}/[^\s\"'<>\])}]+)", re.IGNORECASE)
+ARXIV_URL_RE = re.compile(r"arxiv\.org/(?:abs|pdf)/(\d{4}\.\d{4,5}(?:v\d+)?)", re.IGNORECASE)
+META_DOI_RE = re.compile(
+ r"<meta[^>]+(?:name|property)=[\"'](?:citation_doi|dc\.identifier)[\"'][^>]+"
+ r"content=[\"']\s*(?:doi:)?\s*(10\.[^\"']+)[\"']",
+ re.IGNORECASE,
+)
+BODY_DOI_RE = re.compile(r"\bDOI:?\s*(10\.\d{4,9}/[^\s\"'<>\])}]+)", re.IGNORECASE)
+
+# Per-host courtesy delay between UNCACHED requests, seconds.
+_HOST_DELAY = {
+ "api.openalex.org": 0.15,
+ "api.semanticscholar.org": 1.1,
+ "api.crossref.org": 0.15,
+}
+_last_call: dict[str, float] = {}
+
+
+def _clean_doi(doi: str) -> str:
+ return doi.strip().rstrip(".,;")
+
+
+def extract_doi(
+ url: str,
+ raw_html: str | None = None,
+ content: str | None = None,
+) -> str | None:
+ """Best-effort DOI or arXiv-id extraction. Returns None when nothing found.
+
+ Priority: URL DOI > arXiv URL > citation_doi meta tag > in-body DOI marker.
+ arXiv ids are returned as "arXiv:<id>" so downstream code can route them
+ to Semantic Scholar (OpenAlex has no arXiv-id lookup scheme).
+ """
+ parsed = urlparse(url)
+ if "doi.org" in parsed.netloc.lower():
+ m = DOI_RE.search(parsed.path)
+ if m:
+ return _clean_doi(m.group(1))
+ m = ARXIV_URL_RE.search(url)
+ if m:
+ return f"arXiv:{m.group(1)}"
+ if raw_html:
+ m = META_DOI_RE.search(raw_html)
+ if m:
+ return _clean_doi(m.group(1))
+ if content:
+ m = BODY_DOI_RE.search(content[:20000])
+ if m:
+ return _clean_doi(m.group(1))
+ return None
+
+
+# ---------------------------------------------------------------------------
+# HTTP layer — cache-first, monkeypatchable
+# ---------------------------------------------------------------------------
+
+
+def _http_get_json(url: str) -> dict | None:
+ """Raw HTTP GET returning parsed JSON, or None on any failure.
+
+ Isolated so tests can monkeypatch it; every failure is soft — partial
+ enrichment beats a crashed scoring run.
+ """
+ import httpx
+
+ try:
+ resp = httpx.get(
+ url,
+ follow_redirects=True,
+ timeout=20,
+ headers={"User-Agent": "hyperresearch (mailto:[email protected])"},
+ )
+ if resp.status_code != 200:
+ return None
+ return resp.json()
+ except Exception:
+ return None
+
+
+def _fetch_json(conn, url: str, ttl_days: int, fresh: bool = False) -> dict | None:
+ """Cache-first JSON fetch through the api_cache table."""
+ now = datetime.now(UTC)
+ if not fresh:
+ row = conn.execute("SELECT body, fetched_at FROM api_cache WHERE url = ?", (url,)).fetchone()
+ if row:
+ try:
+ fetched = datetime.fromisoformat(row["fetched_at"])
+ except ValueError:
+ fetched = None
+ if fetched and now - fetched < timedelta(days=ttl_days):
+ try:
+ return json.loads(row["body"])
+ except json.JSONDecodeError:
+ pass
+
+ # Courtesy rate limit per host, only for real network calls
+ host = urlparse(url).netloc.lower()
+ delay = _HOST_DELAY.get(host, 0.2)
+ elapsed = time.monotonic() - _last_call.get(host, 0.0)
+ if elapsed < delay:
+ time.sleep(delay - elapsed)
+ _last_call[host] = time.monotonic()
+
+ data = _http_get_json(url)
+ if data is not None:
+ conn.execute(
+ "INSERT OR REPLACE INTO api_cache (url, body, fetched_at) VALUES (?, ?, ?)",
+ (url, json.dumps(data), now.isoformat()),
+ )
+ conn.commit()
+ return data
+
+
+# ---------------------------------------------------------------------------
+# Per-identifier metadata lookup
+# ---------------------------------------------------------------------------
+
+
+def lookup_metadata(conn, doi: str, ttl_days: int, fresh: bool = False) -> dict | None:
+ """Resolve one DOI/arXiv id to {citation_count, venue, is_retracted}.
+
+ OpenAlex is primary for DOIs (it carries is_retracted directly);
+ Semantic Scholar handles arXiv ids and serves as DOI fallback.
+ """
+ if doi.lower().startswith("arxiv:"):
+ arxiv_id = doi.split(":", 1)[1]
+ data = _fetch_json(
+ conn,
+ "https://api.semanticscholar.org/graph/v1/paper/arXiv:"
+ f"{quote(arxiv_id)}?fields=citationCount,venue,externalIds",
+ ttl_days,
+ fresh,
+ )
+ if data is None:
+ return None
+ return {
+ "citation_count": data.get("citationCount"),
+ "venue": data.get("venue") or None,
+ "is_retracted": False, # S2 has no retraction flag
+ }
+
+ data = _fetch_json(
+ conn,
+ f"https://api.openalex.org/works/doi:{quote(doi, safe='')}"
+ "?select=cited_by_count,primary_location,is_retracted",
+ ttl_days,
+ fresh,
+ )
+ if data is not None:
+ venue = None
+ loc = data.get("primary_location") or {}
+ src = loc.get("source") or {}
+ venue = src.get("display_name")
+ return {
+ "citation_count": data.get("cited_by_count"),
+ "venue": venue,
+ "is_retracted": bool(data.get("is_retracted", False)),
+ }
+
+ # Fallback: Semantic Scholar by DOI
+ data = _fetch_json(
+ conn,
+ "https://api.semanticscholar.org/graph/v1/paper/DOI:"
+ f"{quote(doi, safe='')}?fields=citationCount,venue",
+ ttl_days,
+ fresh,
+ )
+ if data is None:
+ return None
+ return {
+ "citation_count": data.get("citationCount"),
+ "venue": data.get("venue") or None,
+ "is_retracted": False,
+ }
+
+
+# ---------------------------------------------------------------------------
+# Batch scoring
+# ---------------------------------------------------------------------------
+
+
+def backfill_dois(vault, tag: str | None = None) -> int:
+ """Regex source URLs + bodies of existing notes for missed DOIs.
+
+ Updates note frontmatter + DB. Returns count of notes that gained a doi.
+ """
+ from hyperresearch.core.frontmatter import parse_frontmatter, render_note
+
+ conn = vault.db
+ query = (
+ "SELECT n.id, n.path, n.source FROM notes n "
+ "WHERE n.doi IS NULL AND n.source IS NOT NULL"
+ )
+ params: tuple = ()
+ if tag:
+ query += " AND n.id IN (SELECT note_id FROM tags WHERE tag = ?)"
+ params = (tag,)
+
+ gained = 0
+ for row in conn.execute(query, params).fetchall():
+ note_path = vault.root / row["path"]
+ if not note_path.exists():
+ continue
+ text = note_path.read_text(encoding="utf-8-sig")
+ meta, body = parse_frontmatter(text)
+ doi = extract_doi(row["source"] or "", content=body)
+ if not doi:
+ continue
+ meta.doi = doi
+ note_path.write_text(render_note(meta, body), encoding="utf-8")
+ conn.execute("UPDATE notes SET doi = ? WHERE id = ?", (doi, row["id"]))
+ gained += 1
+ if gained:
+ conn.commit()
+ return gained
+
+
+def compute_authority_scores(conn) -> int:
+ """Vault-relative authority: percentile rank of log(1+citation_count).
+
+ Percentile within the set of notes that HAVE a citation count, so a
+ 50-citation niche paper isn't crushed by a 10k-citation classic from
+ another field. Notes without citation data keep authority NULL (the
+ quality composite renormalizes around it).
+ """
+ rows = conn.execute(
+ "SELECT id, citation_count FROM notes WHERE citation_count IS NOT NULL"
+ ).fetchall()
+ if not rows:
+ return 0
+ scored = sorted(rows, key=lambda r: math.log1p(r["citation_count"] or 0))
+ n = len(scored)
+ for rank, row in enumerate(scored):
+ pct = (rank + 1) / n if n > 1 else 1.0
+ conn.execute("UPDATE notes SET authority_score = ? WHERE id = ?", (pct, row["id"]))
+ conn.commit()
+ return n
+
+
+def score_sources(
+ vault,
+ tag: str | None = None,
+ fresh: bool = False,
+ limit: int | None = None,
+) -> dict:
+ """Enrich DOI-bearing notes with citation metadata, then recompute
+ authority percentiles and composite quality scores.
+
+ Returns a summary dict: {scored, retracted, missing, authority_ranked}.
+ """
+ from hyperresearch.core.frontmatter import parse_frontmatter, render_note
+ from hyperresearch.core.quality import compute_quality_scores
+
+ conn = vault.db
+ ttl = vault.config.ranking.api_cache_ttl_days
+
+ query = "SELECT n.id, n.path, n.doi FROM notes n WHERE n.doi IS NOT NULL"
+ params: tuple = ()
+ if tag:
+ query += " AND n.id IN (SELECT note_id FROM tags WHERE tag = ?)"
+ params = (tag,)
+ if not fresh:
+ # Skip notes already enriched (still refreshed when --fresh)
+ query += " AND n.citation_count IS NULL"
+ if limit:
+ query += f" LIMIT {int(limit)}"
+
+ scored = 0
+ retracted: list[str] = []
+ missing: list[str] = []
+
+ for row in conn.execute(query, params).fetchall():
+ meta_result = lookup_metadata(conn, row["doi"], ttl, fresh)
+ if meta_result is None:
+ missing.append(row["id"])
+ continue
+
+ # DB update
+ conn.execute(
+ "UPDATE notes SET citation_count = ?, venue = ?, is_retracted = ? WHERE id = ?",
+ (
+ meta_result["citation_count"],
+ meta_result["venue"],
+ 1 if meta_result["is_retracted"] else 0,
+ row["id"],
+ ),
+ )
+
+ # Frontmatter mirror (markdown stays truth; survives DB rebuild)
+ note_path = vault.root / row["path"]
+ if note_path.exists():
+ text = note_path.read_text(encoding="utf-8-sig")
+ fm, body = parse_frontmatter(text)
+ fm.citation_count = meta_result["citation_count"]
+ fm.venue = meta_result["venue"]
+ fm.is_retracted = bool(meta_result["is_retracted"])
+ note_path.write_text(render_note(fm, body), encoding="utf-8")
+
+ scored += 1
+ if meta_result["is_retracted"]:
+ retracted.append(row["id"])
+
+ conn.commit()
+ authority_ranked = compute_authority_scores(conn)
+ compute_quality_scores(conn, vault.config.ranking)
+
+ return {
+ "scored": scored,
+ "retracted": retracted,
+ "missing": missing,
+ "authority_ranked": authority_ranked,
+ }
diff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.py
index ddf7082..3187158 100644
--- a/src/hyperresearch/core/sync.py
+++ b/src/hyperresearch/core/sync.py
@@ -85,12 +85,17 @@ def compute_sync_plan(vault, force: bool = False) -> SyncPlan:
if not kb_dir.exists():
return plan
+ runs_dir = kb_dir / "runs"
disk_files: dict[str, float] = {}
for md_file in kb_dir.rglob("*.md"):
# Skip staging files at the research/ root. Real notes live in
# research/notes/** or research/index/**.
if md_file.parent == kb_dir:
continue
+ # Skip per-run workspaces entirely (research/runs/<vault_tag>/**) —
+ # run-scoped pipeline artifacts are never vault notes.
+ if runs_dir in md_file.parents:
+ continue
# Skip scratch artifacts without YAML frontmatter.
if not _has_frontmatter(md_file):
continue
@@ -223,13 +228,19 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non
expires_iso = meta.expires.isoformat() if meta.expires else None
# Use INSERT ... ON CONFLICT to avoid CASCADE deletes from INSERT OR REPLACE
+ # NOTE: the derived score columns (authority_score, centrality_score,
+ # independence, quality_score) are deliberately absent — they are DB-cache
+ # values computed by `hpr sources score` / `hpr graph rank` and must
+ # survive re-syncs. Frontmatter-mirrored ranking fields (doi,
+ # utility_score, citation_count, venue, is_retracted) sync normally.
conn.execute(
"""
INSERT INTO notes
(id, title, path, status, type, tier, content_type, source, parent,
deprecated, reviewed, expires, word_count, summary,
- created, updated, file_mtime, content_hash, synced_at)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ created, updated, file_mtime, content_hash, synced_at,
+ doi, utility_score, citation_count, venue, is_retracted)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
title=excluded.title, path=excluded.path, status=excluded.status,
type=excluded.type, tier=excluded.tier, content_type=excluded.content_type,
@@ -239,7 +250,10 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non
word_count=excluded.word_count, summary=excluded.summary,
created=excluded.created, updated=excluded.updated,
file_mtime=excluded.file_mtime, content_hash=excluded.content_hash,
- synced_at=excluded.synced_at
+ synced_at=excluded.synced_at,
+ doi=excluded.doi, utility_score=excluded.utility_score,
+ citation_count=excluded.citation_count, venue=excluded.venue,
+ is_retracted=excluded.is_retracted
""",
(
meta.id, meta.title, note.path, meta.status, meta.type,
@@ -248,6 +262,8 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non
reviewed_iso, expires_iso,
note.word_count, meta.summary, created_iso, updated_iso,
file_mtime, note.content_hash, synced_at,
+ meta.doi, meta.utility_score, meta.citation_count, meta.venue,
+ 1 if meta.is_retracted else 0,
),
)
diff --git a/src/hyperresearch/core/vault.py b/src/hyperresearch/core/vault.py
index f357f1d..982b78e 100644
--- a/src/hyperresearch/core/vault.py
+++ b/src/hyperresearch/core/vault.py
@@ -73,6 +73,20 @@ class Vault:
"""
return self.research_dir / "temp"
+ @property
+ def runs_dir(self) -> Path:
+ """Per-run workspaces (research/runs/<vault_tag>/).
+
+ All run-scoped pipeline artifacts (scaffold, decomposition, loci,
+ critic findings, logs, temp scratch) live under one directory per
+ run, so concurrent and sequential runs never collide. Vault notes
+ stay global (research/notes/). NOT synced as notes.
+ """
+ return self.research_dir / "runs"
+
+ def run_dir(self, vault_tag: str) -> Path:
+ return self.runs_dir / vault_tag
+
@property
def templates_dir(self) -> Path:
"""Templates live inside .hyperresearch/ (hidden)."""
diff --git a/src/hyperresearch/models/note.py b/src/hyperresearch/models/note.py
index 5ed66fb..05d7b2b 100644
--- a/src/hyperresearch/models/note.py
+++ b/src/hyperresearch/models/note.py
@@ -92,6 +92,14 @@ class NoteMeta(BaseModel):
expires: datetime | None = None # Auto-stale after this date
summary: str | None = None
raw_file: str | None = None # Relative path to raw artifact (e.g. raw/<id>.pdf)
+ # Source-ranking fields (frontmatter-mirrored; markdown stays truth).
+ # Derived scores (authority/centrality/independence/quality) are DB-cache
+ # only and deliberately NOT in frontmatter — they are recomputed.
+ doi: str | None = None # DOI or arXiv id (e.g. 10.1234/x, arXiv:2501.01234)
+ utility_score: float | None = None # Step-2 fetch-selection composite (0-18)
+ citation_count: int | None = None # External citation count (OpenAlex/S2)
+ venue: str | None = None # Publication venue, when known
+ is_retracted: bool | None = None # None = unchecked; set by `hpr sources score`
@field_validator("tags", mode="before")
@classmethod
diff --git a/src/hyperresearch/search/fts.py b/src/hyperresearch/search/fts.py
index 1457978..620503e 100644
--- a/src/hyperresearch/search/fts.py
+++ b/src/hyperresearch/search/fts.py
@@ -68,6 +68,7 @@ def search_fts(
offset: int = 0,
include_index: bool = False,
ranking: dict | None = None,
+ quality_ranked: bool = False,
) -> list[dict]:
"""Execute a full-text search against the notes_fts table.
@@ -105,6 +106,7 @@ def search_fts(
sql = f"""
SELECT
n.id, n.title, n.path, n.status, n.type, n.tier, n.content_type,
+ n.quality_score,
n.created, n.updated, n.word_count, n.summary,
snippet(notes_fts, 2, '>>>', '<<<', '...', 64) as snippet,
bm25(notes_fts, 0.0, {tw}, {bw}, {tgw}, {aw}) as score,
@@ -147,6 +149,7 @@ def search_fts(
"updated": row["updated"],
"word_count": row["word_count"],
"summary": row["summary"],
+ "quality_score": row["quality_score"] if "quality_score" in row.keys() else None, # noqa: SIM118
"score": abs(row["score"]),
"snippet": row["snippet"] or "",
})
@@ -165,4 +168,15 @@ def search_fts(
r["score"] *= penalize_stale
results.sort(key=lambda x: x["score"], reverse=True)
+ # Quality-weighted re-rank (opt-in): fold the composite source-quality
+ # score into relevance. (0.5 + quality) keeps unscored notes (quality
+ # NULL -> neutral 1.0x via fallback 0.5) competitive while a
+ # ground-truth-tier, high-authority source (~1.0) roughly triples a
+ # retracted one (~0.05).
+ if quality_ranked:
+ for r in results:
+ q = r.get("quality_score")
+ r["score"] *= 0.5 + (q if q is not None else 0.5)
+ results.sort(key=lambda x: x["score"], reverse=True)
+
return results
diff --git a/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md b/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md
new file mode 100644
index 0000000..b04673c
--- /dev/null
+++ b/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md
@@ -0,0 +1,93 @@
+---
+name: hyperresearch-1-5-chapter-partition
+description: >
+ Step 1.5 of the hyperresearch V8 pipeline — CHAPTERED PROFILES ONLY
+ (dissertation). Partitions the decomposed query into 4-10 topically
+ cohesive chapters, writes the chapter plan, and registers chapters in the
+ run manifest. Steps 2-10 then loop per chapter, each staying within the
+ proven per-chapter source envelope; global reconciliation and synthesis
+ integrate across chapters. Skipped entirely when the profile's `chapters`
+ is (0, 0). Invoked via Skill tool from the entry skill.
+---
+
+# Step 1.5 — Chapter partition (chaptered profiles only)
+
+**Profile gate:** Runs ONLY when the resolved profile's `chapters` range is non-zero (e.g. `dissertation` = << dissertation.chapters|dash >> chapters). For `light` and `full`, this step does not exist — skip directly to step 2.
+
+**Why chapters:** no single agent can hold a << dissertation.source_target|dash >>-source corpus. Chapters make the proven << dissertation.chapter_source_target|dash >>-source pipeline the *unit of execution*: each chapter runs its own width sweep through drafting, and global layers integrate on top. The ~80-source quality ceiling is per-chapter, not per-run.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — run config
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, required_section_headings
+- `research/runs/<vault_tag>/query.md` — canonical query
+
+---
+
+## Procedure
+
+1. **Decide chapter count.** Group the decomposition's atomic items into << dissertation.chapters|dash >> topically cohesive chapters. Rules:
+ - Every atomic item lands in exactly ONE chapter (cross-cutting items go to the chapter that will treat them most deeply; other chapters reference, not re-research).
+ - A chapter is a *research unit*: it must be answerable with << dissertation.chapter_source_target|dash >> sources. If a candidate chapter obviously needs more, split it.
+ - When `required_section_headings` already implies a chapter structure (the prompt asked for named parts), FOLLOW IT — chapters must map 1:1 onto required top-level headings when they exist.
+ - Fewer than ~12 atomic items → the query doesn't need chaptering; record 1 chapter and note the pipeline degenerates to the flat `full` shape.
+
+2. **Write the chapter plan** to `research/runs/<vault_tag>/chapter-plan.json`:
+ ```json
+ {
+ "chapters": [
+ {
+ "id": "ch1",
+ "title": "<chapter title — becomes the H1 in the final document>",
+ "atomic_items": ["<verbatim items from the decomposition>"],
+ "rationale": "<1-2 sentences: why these items cohere>",
+ "depends_on": []
+ }
+ ]
+ }
+ ```
+ `depends_on` lists chapters whose findings this chapter builds on (usually empty; used to order execution).
+
+3. **Scaffold chapter workspaces.** For each chapter, create `research/runs/<vault_tag>/chapters/<id>/` with an empty `temp/` subdirectory. Chapter-scoped step artifacts (loci.json, comparisons.md, claims, drafts) live there; the flat run-root copies are NOT used on chaptered runs.
+
+4. **Register chapters in the manifest:**
+ ```bash
+ $HPR run step <vault_tag> 1.5 --status done -j
+ ```
+ and for each chapter:
+ ```bash
+ $HPR run event <vault_tag> --type chapter-plan --data '{"chapter": "<id>", "title": "<title>"}' -j
+ ```
+
+5. **Chapter tagging convention.** Every note fetched or written for a chapter carries BOTH tags: `<vault_tag>` and `<vault_tag>-<id>` (e.g. `china-rail-x9f2a1-ch3`). Whole-run queries use the first; per-chapter queries use the second. Cross-chapter source reuse is free — dedup is by URL, and a chapter's coverage check searches the whole `<vault_tag>` corpus before fetching.
+
+---
+
+## Chapter execution loop (what the orchestrator does next)
+
+For each chapter (respecting `depends_on`, up to << dissertation.chapter_concurrency >> chapters in flight):
+
+1. Invoke steps 2 → 10 with the chapter as scope:
+ - The "research query" for chapter-scoped subagent spawns is the verbatim canonical query PLUS a chapter assignment block naming the chapter's title and atomic items.
+ - Artifact paths swap `research/runs/<vault_tag>/` for `research/runs/<vault_tag>/chapters/<id>/` (each chapter has its own loci.json, comparisons.md, temp/).
+ - Source targets come from `chapter_source_target`, not the global `source_target`.
+ - Step 10 writes ONE draft per chapter to `research/runs/<vault_tag>/chapters/<id>/draft.md` (draft_count is 1 for chaptered profiles — angle diversity comes from the chapters themselves).
+2. Record progress: `$HPR run step <vault_tag> <N> --status done --chapter <id> -j` after each chapter-step completes.
+3. After ALL chapters finish step 10, proceed to the global layers: step 6 re-runs GLOBALLY (cross-CHAPTER tensions from the chapters' comparisons.md files → `research/runs/<vault_tag>/comparisons.md`), then step 11 synthesizes the chapter drafts into the final document (chapter titles become H1s), then steps 12-16 run once against the whole document.
+
+**Budget check at every chapter boundary:** `$HPR run status <vault_tag> -j`. If `status` is `blocked` with `blocked_on: "budget"`, STOP spawning and surface to the user. Never silently skip profile-mandated steps — shrink the NEXT chapter's fan-out (fewer wave-2 fetchers, lower depth budgets) when `budget_remaining_usd` is under ~30% instead.
+
+---
+
+## Exit criterion
+
+- `research/runs/<vault_tag>/chapter-plan.json` exists with every atomic item assigned to exactly one chapter
+- Chapter workspaces exist under `research/runs/<vault_tag>/chapters/`
+- Manifest step 1.5 marked done, chapters registered
+
+## Next step
+
+Return to the entry skill and begin the chapter execution loop at step 2 for the first chapter(s): `Skill(skill: "hyperresearch-2-width-sweep")`.
diff --git a/src/hyperresearch/skills/hyperresearch-1-decompose.md b/src/hyperresearch/skills/hyperresearch-1-decompose.md
index 804f89a..491666a 100644
--- a/src/hyperresearch/skills/hyperresearch-1-decompose.md
+++ b/src/hyperresearch/skills/hyperresearch-1-decompose.md
@@ -22,8 +22,8 @@ description: >
## Recover state
The orchestrator's bootstrap step (in the entry skill) has already produced:
-- `research/scaffold.md` — vault_tag, modality, wrapper requirements
-- `research/query-<vault_tag>.md` — canonical research query (GOSPEL)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag, modality, wrapper requirements
+- `research/runs/<vault_tag>/query.md` — canonical research query (GOSPEL)
Read both before starting. The vault_tag is in the scaffold's "Run config" section.
@@ -31,7 +31,7 @@ Read both before starting. The vault_tag is in the scaffold's "Run config" secti
## Procedure
-1. **Re-read the canonical research query** end to end (`research/query-<vault_tag>.md`).
+1. **Re-read the canonical research query** end to end (`research/runs/<vault_tag>/query.md`).
2. **Walk through it and extract every atomic item** — anything that's a discrete thing the prompt named. These fall into categories:
- **Sub-questions** — explicit or implicit questions the draft must answer ("What cues influence this?" → atomic: "cues influencing X")
@@ -57,7 +57,7 @@ Read both before starting. The vault_tag is in the scaffold's "Run config" secti
]
```
-4. **Write `research/prompt-decomposition.json`:**
+4. **Write `research/runs/<vault_tag>/prompt-decomposition.json`:**
```json
{
@@ -138,7 +138,7 @@ Read both before starting. The vault_tag is in the scaffold's "Run config" secti
- Is the decomposition's interpretation **as broad as the phrase's natural scope**? (e.g., "SaaS applications" must not be narrowed to "POS SaaS"; "rugged tablets" must not be collapsed into "payment terminals")
- If the phrase has multiple plausible referents, does the decomposition cover BOTH readings?
- Write the matrix to `research/temp/coverage-matrix.md`:
+ Write the matrix to `research/runs/<vault_tag>/temp/coverage-matrix.md`:
```markdown
## Coverage Matrix — query phrase → atomic item mapping
@@ -152,22 +152,22 @@ Read both before starting. The vault_tag is in the scaffold's "Run config" secti
**If any row has `Gap? = YES`:** go back and fix the decomposition. Add the missing atomic items, broaden the narrowed scope_conditions, or add missing entities. Then re-run the matrix until every row passes. Do NOT proceed with known gaps — they cascade into missing searches, missing sources, and missing draft sections.
-9. **Update the scaffold.** Append a "Tier rationale" subsection to `research/scaffold.md` with a 2-3 sentence justification for the tier classification.
+9. **Update the scaffold.** Append a "Tier rationale" subsection to `research/runs/<vault_tag>/scaffold.md` with a 2-3 sentence justification for the tier classification.
---
## Exit criterion
-- `research/prompt-decomposition.json` exists, is valid JSON, every atomic item traces to the research_query
+- `research/runs/<vault_tag>/prompt-decomposition.json` exists, is valid JSON, every atomic item traces to the research_query
- `pipeline_tier` + `response_format` + `citation_style` are all set
-- `research/temp/coverage-matrix.md` exists with **zero `Gap? = YES` rows**
-- `research/scaffold.md` includes a Tier rationale subsection
+- `research/runs/<vault_tag>/temp/coverage-matrix.md` exists with **zero `Gap? = YES` rows**
+- `research/runs/<vault_tag>/scaffold.md` includes a Tier rationale subsection
---
## Next step
-Return to the entry skill (`hyperresearch`). Read `research/prompt-decomposition.json` to learn the tier, then invoke step 2:
+Return to the entry skill (`hyperresearch`). Read `research/runs/<vault_tag>/prompt-decomposition.json` to learn the tier, then invoke step 2:
```
Skill(skill: "hyperresearch-2-width-sweep")
diff --git a/src/hyperresearch/skills/hyperresearch-10-triple-draft.md b/src/hyperresearch/skills/hyperresearch-10-triple-draft.md
index 312dfe0..6a5ec44 100644
--- a/src/hyperresearch/skills/hyperresearch-10-triple-draft.md
+++ b/src/hyperresearch/skills/hyperresearch-10-triple-draft.md
@@ -24,12 +24,12 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag, modality, wrapper requirements
-- `research/prompt-decomposition.json` — atomic items, required_section_headings, response_format, citation_style, pipeline_tier
-- `research/temp/evidence-digest.md` — top claims + verbatim quotes — PRIMARY EVIDENCE LAYER (full only; absent for light)
-- `research/comparisons.md` (full tier) — cross-locus tensions
-- `research/temp/source-tensions.json` (full tier) — expert disagreements
-- `research/temp/coverage-gaps.md` (if exists) — items with weak source coverage
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag, modality, wrapper requirements
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, required_section_headings, response_format, citation_style, pipeline_tier
+- `research/runs/<vault_tag>/temp/evidence-digest.md` — top claims + verbatim quotes — PRIMARY EVIDENCE LAYER (full only; absent for light)
+- `research/runs/<vault_tag>/comparisons.md` (full tier) — cross-locus tensions
+- `research/runs/<vault_tag>/temp/source-tensions.json` (full tier) — expert disagreements
+- `research/runs/<vault_tag>/temp/coverage-gaps.md` (if exists) — items with weak source coverage
- Survey vault: `$HPR note list --tag <vault_tag> --all -j` for the evidence landscape
- Modality calibration (from the scaffold's `modality` field):
- **collect** — enumerative coverage, per-entity sections with named fields
@@ -41,13 +41,13 @@ Read these inputs:
## Step 10.0 — Read response_format and citation_style
-Read `response_format` and `citation_style` from `research/prompt-decomposition.json`. These control the draft shape:
+Read `response_format` and `citation_style` from `research/runs/<vault_tag>/prompt-decomposition.json`. These control the draft shape:
| Format | Target length | Character |
|--------|-------------|-----------|
-| `"short"` | 500–2000 words / 1500–6000 chars (CJK) | Direct answer, compact |
-| `"structured"` | 2000–5000 words / 6000–15000 chars (CJK) | Scannable, breadth-first |
-| `"argumentative"` | 5000–10000 words / 20000–25000 chars (CJK) | Dense thesis-driven |
+| `"short"` | << p.word_targets.short|dash >> words / 1500–6000 chars (CJK) | Direct answer, compact |
+| `"structured"` | << p.word_targets.structured|dash >> words / 6000–15000 chars (CJK) | Scannable, breadth-first |
+| `"argumentative"` | << p.word_targets.argumentative|dash >> words / 20000–25000 chars (CJK) | Dense thesis-driven |
**Length discipline:** Target the MIDDLE of the range. Under-length loses on comprehensiveness; over-length dilutes good content.
@@ -59,10 +59,10 @@ If `pipeline_tier == "light"`: SKIP step 10.1 — 10.3 below and follow this sec
**Light tier writes a single draft directly to `research/notes/final_report_<vault_tag>.md`.** No subagents, no triple-draft ensemble, no synthesizer.
-1. **Read the vault directly.** Light tier has no `evidence-digest.md` (step 9 was skipped). Survey the vault: `$HPR note list --tag <vault_tag> --all -j` and pick the 8–15 most relevant non-deprecated notes. Read each one (`$HPR note show <id1> <id2> ... -j`) before writing.
+1. **Read the vault directly.** Light tier has no `evidence-digest.md` (step 9 was skipped). Survey the vault: `$HPR note list --tag <vault_tag> --all -j` and pick the << p.single_draft_reads|dash >> most relevant non-deprecated notes. Read each one (`$HPR note show <id1> <id2> ... -j`) before writing.
2. **Honor the structural contract.**
- - Use the literal H2 headings from `required_section_headings` in `research/prompt-decomposition.json`, in order.
+ - Use the literal H2 headings from `required_section_headings` in `research/runs/<vault_tag>/prompt-decomposition.json`, in order.
- Hit the length target from step 10.0's table for the chosen `response_format` (light typically pairs with `short` or `structured`).
- Apply the modality calibration from the recover-state list above.
@@ -91,13 +91,13 @@ Based on the evidence, tensions, and query, assign each sub-orchestrator a disti
- **Draft B — Depth-optimized:** deeper treatment of the 3-4 most important atomic items.
- **Draft C — Practitioner-optimized:** organized around actionable recommendations.
-Write the 3 angle assignments to `research/temp/draft-angles.md` (for the run log). Each angle: 2-3 sentences describing the analytical direction.
+Write the 3 angle assignments to `research/runs/<vault_tag>/temp/draft-angles.md` (for the run log). Each angle: 2-3 sentences describing the analytical direction.
---
## Step 10.2 — Curate per-angle source lists
-**Critical step.** Each draft sub-orchestrator does NOT decide what to read. YOU pick the 20-50 most relevant vault notes for each angle and pass them as `must_read_note_ids`. This eliminates wasted vault-survey loops in the sub-orchestrators and forces real differentiation by giving each draft a different evidence base.
+**Critical step.** Each draft sub-orchestrator does NOT decide what to read. YOU pick the << p.must_read.short[0] >>-<< p.must_read.argumentative[1] >> most relevant vault notes for each angle and pass them as `must_read_note_ids`. This eliminates wasted vault-survey loops in the sub-orchestrators and forces real differentiation by giving each draft a different evidence base.
1. **List all substantive vault notes:**
```bash
@@ -105,7 +105,13 @@ Write the 3 angle assignments to `research/temp/draft-angles.md` (for the run lo
```
Filter to non-deprecated notes. You should have 50-100 candidates.
-2. **For each draft (A, B, C), pick 20-50 angle-specific notes.** Use these signals:
+ **Rank the pool before picking.** For each atomic item, run a quality-ranked search to surface the best-evidence sources first:
+ ```bash
+ $HPR search "<atomic item keywords>" --tag <vault_tag> --ranked -j
+ ```
+ `--ranked` folds the composite `quality_score` (tier + utility + citation authority + vault centrality, retractions floored) into relevance. Prefer high-quality sources when two candidates cover the same ground; a note with `quality_score` near the retraction floor should not appear in any must_read list unless the draft explicitly discusses its retraction.
+
+2. **For each draft (A, B, C), pick << p.must_read.short[0] >>-<< p.must_read.argumentative[1] >> angle-specific notes.** Use these signals:
- **Source-analysis notes** (`type: source-analysis`): high-value, full digests of long sources. Include relevant ones in EVERY draft's list — these are gold.
- **Interim notes** (`type: interim`, full tier only): include all of them in EVERY draft's list — these have the committed positions.
- **For Draft A (strongest-thesis or breadth):** prefer sources that support the dominant evidence direction. Include any source the evidence digest cites for high-confidence claims.
@@ -114,12 +120,12 @@ Write the 3 angle assignments to `research/temp/draft-angles.md` (for the run lo
3. **Source overlap is fine.** Drafts can share source IDs — interim notes and key source-analyses should appear in all three lists. Differentiation comes from the angle-specific extras (the 5-15 sources unique to each draft's list).
-4. **Cap each list at 50, minimum 20.** For `argumentative` format, lean toward 35-50. For `structured`, lean toward 25-40. For `short`, lean toward 20-30.
+4. **Cap each list at << p.must_read.argumentative[1] >>, minimum << p.must_read.short[0] >>.** For `argumentative` format, lean toward << p.must_read.argumentative|hyphen >>. For `structured`, lean toward << p.must_read.structured|hyphen >>. For `short`, lean toward << p.must_read.short|hyphen >>.
5. **Write each list to disk** so the spawn template can reference it:
- - `research/temp/draft-a-source-list.md`
- - `research/temp/draft-b-source-list.md`
- - `research/temp/draft-c-source-list.md`
+ - `research/runs/<vault_tag>/temp/draft-a-source-list.md`
+ - `research/runs/<vault_tag>/temp/draft-b-source-list.md`
+ - `research/runs/<vault_tag>/temp/draft-c-source-list.md`
Format:
```markdown
@@ -133,18 +139,18 @@ Write the 3 angle assignments to `research/temp/draft-angles.md` (for the run lo
---
-## Step 10.3 — Spawn 3 draft sub-orchestrators in parallel
+## Step 10.3 — Spawn << p.draft_count >> draft sub-orchestrators in parallel
-**Spawn 3 `hyperresearch-draft-orchestrator` subagents in ONE message.** This is true parallel execution. Each gets a different `draft_id`, `analytical_angle`, and (CRUCIALLY) a different `must_read_note_ids` array.
+**Spawn << p.draft_count >> `hyperresearch-draft-orchestrator` subagents in ONE message.** This is true parallel execution. Each gets a different `draft_id`, `analytical_angle`, and (CRUCIALLY) a different `must_read_note_ids` array.
**Spawn template:**
```
subagent_type: hyperresearch-draft-orchestrator
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are one of 3 parallel step 10 sub-orchestrators
in the hyperresearch V8 pipeline. After you and the other two return, the
@@ -153,16 +159,16 @@ prompt: |
synthesis, not the final output.
YOUR INPUTS:
- - query_file_path: research/query-<vault_tag>.md
+ - query_file_path: research/runs/<vault_tag>/query.md
- vault_tag: <vault_tag>
- draft_id: "a" (or "b" or "c")
- - output_path: research/temp/draft-a.md (or draft-b.md or draft-c.md)
+ - output_path: research/runs/<vault_tag>/temp/draft-a.md (or draft-b.md or draft-c.md)
- analytical_angle: "<the 2-3 sentence angle assignment>"
- - must_read_note_ids: [<paste the IDs from research/temp/draft-<x>-source-list.md, e.g. 30-50 IDs>]
- - decomposition_path: research/prompt-decomposition.json
- - evidence_digest_path: research/temp/evidence-digest.md
- - comparisons_path: research/comparisons.md
- - source_tensions_path: research/temp/source-tensions.json
+ - must_read_note_ids: [<paste the IDs from research/runs/<vault_tag>/temp/draft-<x>-source-list.md, e.g. 30-50 IDs>]
+ - decomposition_path: research/runs/<vault_tag>/prompt-decomposition.json
+ - evidence_digest_path: research/runs/<vault_tag>/temp/evidence-digest.md
+ - comparisons_path: research/runs/<vault_tag>/comparisons.md
+ - source_tensions_path: research/runs/<vault_tag>/temp/source-tensions.json
- response_format: "<short|structured|argumentative>"
- citation_style: "<wikilink|inline|none>"
- modality: "<collect|synthesize|compare|forecast>"
@@ -172,7 +178,7 @@ prompt: |
Write your draft from your assigned angle, citing your curated sources.
```
-**CRITICAL: never emit bare text while the 3 sub-orchestrators are running.** They will take 5-15 minutes each. Use this time to think — append notes to `research/temp/orchestrator-notes.md` about the synthesis you'll plan in step 11: what's the strongest thesis emerging across angles? Which atomic items will be contentious? What argumentative beats must the final draft commit to? One vault count check per minute max. Write your thoughts, don't just poll.
+**CRITICAL: never emit bare text while the 3 sub-orchestrators are running.** They will take 5-15 minutes each. Use this time to think — append notes to `research/runs/<vault_tag>/temp/orchestrator-notes.md` about the synthesis you'll plan in step 11: what's the strongest thesis emerging across angles? Which atomic items will be contentious? What argumentative beats must the final draft commit to? One vault count check per minute max. Write your thoughts, don't just poll.
---
@@ -181,9 +187,9 @@ prompt: |
When all 3 sub-orchestrators return:
1. **Confirm all 3 draft files exist:**
- - `research/temp/draft-a.md`
- - `research/temp/draft-b.md`
- - `research/temp/draft-c.md`
+ - `research/runs/<vault_tag>/temp/draft-a.md`
+ - `research/runs/<vault_tag>/temp/draft-b.md`
+ - `research/runs/<vault_tag>/temp/draft-c.md`
2. **Read each sub-orchestrator's report-back.** Each should report:
- Path to the draft
@@ -204,9 +210,9 @@ When all 3 sub-orchestrators return:
- `research/notes/final_report_<vault_tag>.md` exists, hits the length target from step 10.0, follows `required_section_headings`, and respects `citation_style`.
**Standard / full tier:**
-- All three drafts exist at `research/temp/draft-{a,b,c}.md`
+- All three drafts exist at `research/runs/<vault_tag>/temp/draft-{a,b,c}.md`
- Each draft has non-trivial length (1000+ chars argumentative, 500+ structured)
-- Sub-orchestrator report-backs are captured (you can paraphrase them in `research/temp/orchestrator-notes.md` for the synthesis plan you'll write in step 11)
+- Sub-orchestrator report-backs are captured (you can paraphrase them in `research/runs/<vault_tag>/temp/orchestrator-notes.md` for the synthesis plan you'll write in step 11)
---
diff --git a/src/hyperresearch/skills/hyperresearch-11-synthesize.md b/src/hyperresearch/skills/hyperresearch-11-synthesize.md
index 776e4a4..fd72e14 100644
--- a/src/hyperresearch/skills/hyperresearch-11-synthesize.md
+++ b/src/hyperresearch/skills/hyperresearch-11-synthesize.md
@@ -16,26 +16,26 @@ description: >
**Goal:** turn the 3 angle-specific drafts from step 10 into ONE integrated final report at `research/notes/final_report_<vault_tag>.md`. The orchestrator preps the strategic brief; the synthesizer subagent writes the report in two passes (rough integrated draft, then voice/redundancy/length cleanup).
-**Why split orchestrator + synthesizer:** the orchestrator has been running for 30+ minutes and 200K+ tokens of context. Writing a coherent 5000-10000 word report at this point is the highest cognitive load step in the pipeline, and orchestrator context is full of stale subagent dispatch logic. The synthesizer is a fresh Opus session with `[Read, Write]` tool-lock, focused exclusively on producing the final report. This is the same architectural move that made the patcher and polish-auditor reliable.
+**Why split orchestrator + synthesizer:** the orchestrator has been running for 30+ minutes and 200K+ tokens of context. Writing a coherent 5000-10000 word report at this point is the highest cognitive load step in the pipeline, and orchestrator context is full of stale subagent dispatch logic. The synthesizer is a fresh session with `[Read, Write]` tool-lock, focused exclusively on producing the final report. This is the same architectural move that made the patcher and polish-auditor reliable.
---
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/prompt-decomposition.json` — atomic items, required_section_headings, response_format, citation_style
-- `research/temp/draft-a.md`, `research/temp/draft-b.md`, `research/temp/draft-c.md` — the 3 angle-specific drafts from step 10
-- `research/comparisons.md` (full tier) — cross-locus tensions
-- `research/temp/source-tensions.json` (full tier) — expert disagreements
-- `research/temp/evidence-digest.md` — load-bearing claims with verbatim quotes
-- `research/query-<vault_tag>.md` — canonical research query (GOSPEL)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, required_section_headings, response_format, citation_style
+- `research/runs/<vault_tag>/temp/draft-a.md`, `research/runs/<vault_tag>/temp/draft-b.md`, `research/runs/<vault_tag>/temp/draft-c.md` — the 3 angle-specific drafts from step 10
+- `research/runs/<vault_tag>/comparisons.md` (full tier) — cross-locus tensions
+- `research/runs/<vault_tag>/temp/source-tensions.json` (full tier) — expert disagreements
+- `research/runs/<vault_tag>/temp/evidence-digest.md` — load-bearing claims with verbatim quotes
+- `research/runs/<vault_tag>/query.md` — canonical research query (GOSPEL)
---
## Step 11.1 — Read all 3 drafts in full
-1. **Read each draft in full** from `research/temp/draft-{a,b,c}.md`. Don't skim — actually read. Hold them in context.
+1. **Read each draft in full** from `research/runs/<vault_tag>/temp/draft-{a,b,c}.md`. Don't skim — actually read. Hold them in context.
2. **Re-read each sub-orchestrator's report-back** (from your own task results in step 10). Note each draft's:
- Core thesis
@@ -52,7 +52,7 @@ The synthesizer is tool-locked to `[Read, Write]` — it cannot run Bash to quer
For each substantive contradiction between drafts:
1. Identify the cited source IDs on both sides
2. `$HPR note show <id1> <id2> -j` to read the actual source bodies
-3. Decide which side is correct. Write the verdict to `research/temp/synthesis-conflicts.md`:
+3. Decide which side is correct. Write the verdict to `research/runs/<vault_tag>/temp/synthesis-conflicts.md`:
```markdown
## Conflict 1: <one-line description>
- Draft A says: <claim with citation>
@@ -67,7 +67,7 @@ If there are no substantive conflicts, write a one-line file: "No factual confli
## Step 11.3 — Write the synthesis plan
-Write `research/temp/synthesis-plan.md`. This is your strategic brief for the synthesizer:
+Write `research/runs/<vault_tag>/temp/synthesis-plan.md`. This is your strategic brief for the synthesizer:
```markdown
# Synthesis plan
@@ -104,7 +104,7 @@ Write `research/temp/synthesis-plan.md`. This is your strategic brief for the sy
## Step 11.4 — Write the synthesis outline
-Write `research/temp/synthesis-outline.md`. This is the per-section contract — 1-2 sentences per H2 naming what evidence and argument lives there:
+Write `research/runs/<vault_tag>/temp/synthesis-outline.md`. This is the per-section contract — 1-2 sentences per H2 naming what evidence and argument lives there:
```markdown
# Synthesis outline
@@ -134,10 +134,10 @@ The outline is short (50-200 words total). It's the structural anchor that preve
## Step 11.5 — VERIFICATION GATE
Before spawning the synthesizer, verify these files exist with non-trivial content:
-- `research/temp/synthesis-plan.md` — must include the core thesis and at least one per-section commitment
-- `research/temp/synthesis-outline.md` — must include one outline entry per H2 in the planned structure
-- `research/temp/synthesis-conflicts.md` — exists (may say "no conflicts found")
-- `research/temp/draft-{a,b,c}.md` — all three exist
+- `research/runs/<vault_tag>/temp/synthesis-plan.md` — must include the core thesis and at least one per-section commitment
+- `research/runs/<vault_tag>/temp/synthesis-outline.md` — must include one outline entry per H2 in the planned structure
+- `research/runs/<vault_tag>/temp/synthesis-conflicts.md` — exists (may say "no conflicts found")
+- `research/runs/<vault_tag>/temp/draft-{a,b,c}.md` — all three exist
If any are missing or trivial, fix them before proceeding. The synthesizer cannot do strategic planning — it can only execute the plan. Skipping plan/outline produces a thin synthesizer output that doesn't beat the original drafts.
@@ -152,9 +152,9 @@ Spawn ONE `hyperresearch-synthesizer` subagent. Single spawn, runs once.
subagent_type: hyperresearch-synthesizer
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 11 of the hyperresearch V8 pipeline.
Step 10 produced 3 angle-specific drafts. The orchestrator wrote a
@@ -165,16 +165,16 @@ prompt: |
critics) reads your final report.
YOUR INPUTS:
- - query_file_path: research/query-<vault_tag>.md
- - draft_paths: [research/temp/draft-a.md, research/temp/draft-b.md, research/temp/draft-c.md]
- - synthesis_plan_path: research/temp/synthesis-plan.md
- - synthesis_outline_path: research/temp/synthesis-outline.md
- - synthesis_conflicts_path: research/temp/synthesis-conflicts.md
- - decomposition_path: research/prompt-decomposition.json
- - comparisons_path: research/comparisons.md
- - source_tensions_path: research/temp/source-tensions.json
- - evidence_digest_path: research/temp/evidence-digest.md
- - pass1_output_path: research/temp/synthesis-pass1.md
+ - query_file_path: research/runs/<vault_tag>/query.md
+ - draft_paths: [research/runs/<vault_tag>/temp/draft-a.md, research/runs/<vault_tag>/temp/draft-b.md, research/runs/<vault_tag>/temp/draft-c.md]
+ - synthesis_plan_path: research/runs/<vault_tag>/temp/synthesis-plan.md
+ - synthesis_outline_path: research/runs/<vault_tag>/temp/synthesis-outline.md
+ - synthesis_conflicts_path: research/runs/<vault_tag>/temp/synthesis-conflicts.md
+ - decomposition_path: research/runs/<vault_tag>/prompt-decomposition.json
+ - comparisons_path: research/runs/<vault_tag>/comparisons.md
+ - source_tensions_path: research/runs/<vault_tag>/temp/source-tensions.json
+ - evidence_digest_path: research/runs/<vault_tag>/temp/evidence-digest.md
+ - pass1_output_path: research/runs/<vault_tag>/temp/synthesis-pass1.md
- final_output_path: research/notes/final_report_<vault_tag>.md
- response_format: "<short|structured|argumentative>"
- citation_style: "<wikilink|inline|none>"
@@ -198,7 +198,7 @@ prompt: |
Sources section.
```
-**CRITICAL: never emit bare text while the synthesizer is running.** It will take 5-15 minutes (two passes). Use the wait time to think — append notes to `research/temp/orchestrator-notes.md` about what you'll watch for in step 12 (the critics) based on the synthesis plan you just wrote.
+**CRITICAL: never emit bare text while the synthesizer is running.** It will take 5-15 minutes (two passes). Use the wait time to think — append notes to `research/runs/<vault_tag>/temp/orchestrator-notes.md` about what you'll watch for in step 12 (the critics) based on the synthesis plan you just wrote.
---
@@ -207,7 +207,7 @@ prompt: |
When the synthesizer returns:
1. **Confirm both files exist:**
- - `research/temp/synthesis-pass1.md` (pass 1, rough integrated)
+ - `research/runs/<vault_tag>/temp/synthesis-pass1.md` (pass 1, rough integrated)
- `research/notes/final_report_<vault_tag>.md` (pass 2, final)
2. **Read the synthesizer's report-back.** It tells you:
@@ -238,7 +238,7 @@ After this step, the final report is only modified by Edit hunks from the patche
## Exit criterion
- `research/notes/final_report_<vault_tag>.md` exists, length in target range
-- `research/temp/synthesis-pass1.md` exists (debugging artifact)
+- `research/runs/<vault_tag>/temp/synthesis-pass1.md` exists (debugging artifact)
- All H2s from `required_section_headings` present
- Citations match `citation_style` (wikilink → `[[note-id]]` no Sources section; inline → `[N]` + Sources section; none → no markers)
- No YAML frontmatter, no pipeline vocabulary, no scaffold leaks
diff --git a/src/hyperresearch/skills/hyperresearch-12-critics.md b/src/hyperresearch/skills/hyperresearch-12-critics.md
index d47d3f8..9af2f06 100644
--- a/src/hyperresearch/skills/hyperresearch-12-critics.md
+++ b/src/hyperresearch/skills/hyperresearch-12-critics.md
@@ -19,29 +19,29 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/prompt-decomposition.json` — pipeline_tier, atomic items
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — pipeline_tier, atomic items
- `research/notes/final_report_<vault_tag>.md` — merged draft from step 10
-- `research/query-<vault_tag>.md` — canonical research query
+- `research/runs/<vault_tag>/query.md` — canonical research query
---
## Procedure
1. **Spawn all 4 critics in parallel.** In ONE message:
- - `hyperresearch-dialectic-critic` → `research/critic-findings-dialectic.json` (counter-evidence the draft missed or straw-manned)
- - `hyperresearch-depth-critic` → `research/critic-findings-depth.json` (shallow spots where interim notes could fill substance)
- - `hyperresearch-width-critic` → `research/critic-findings-width.json` (corpus clusters the draft ignores despite evidence)
- - `hyperresearch-instruction-critic` → `research/critic-findings-instruction.json` (atomic items from the decomposition that the draft missed, under-covered, reordered, or reformatted)
+ - `hyperresearch-dialectic-critic` → `research/runs/<vault_tag>/critic-findings-dialectic.json` (counter-evidence the draft missed or straw-manned)
+ - `hyperresearch-depth-critic` → `research/runs/<vault_tag>/critic-findings-depth.json` (shallow spots where interim notes could fill substance)
+ - `hyperresearch-width-critic` → `research/runs/<vault_tag>/critic-findings-width.json` (corpus clusters the draft ignores despite evidence)
+ - `hyperresearch-instruction-critic` → `research/runs/<vault_tag>/critic-findings-instruction.json` (atomic items from the decomposition that the draft missed, under-covered, reordered, or reformatted)
2. **Pass each critic** (standard 3-piece contract):
```
subagent_type: hyperresearch-<critic-name>-critic
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 12 (<critic-name> critic) of the
hyperresearch V8 pipeline. Step 11 (synthesizer) produced the final report at
@@ -50,9 +50,9 @@ Read these inputs:
YOUR INPUTS:
- draft_path: research/notes/final_report_<vault_tag>.md
- - output_path: research/critic-findings-<critic-name>.json
+ - output_path: research/runs/<vault_tag>/critic-findings-<critic-name>.json
- vault_tag: <vault_tag>
- - decomposition_path: research/prompt-decomposition.json (instruction-critic only)
+ - decomposition_path: research/runs/<vault_tag>/prompt-decomposition.json (instruction-critic only)
```
3. **Wait for all critics.** If one fails, you can proceed with the partial set, but log the absence to the run log — the patch pass is less robust with missing critic coverage. **Do NOT skip the instruction-critic specifically** — it's the only critic measuring prompt adherence, which is the dimension with the widest variance.
@@ -63,7 +63,7 @@ Read these inputs:
## Exit criterion
-- All 4 critic findings JSONs exist (`research/critic-findings-<name>.json`)
+- All 4 critic findings JSONs exist (`research/runs/<vault_tag>/critic-findings-<name>.json`)
- Each is valid JSON with a `findings` array
---
diff --git a/src/hyperresearch/skills/hyperresearch-13-gap-fetch.md b/src/hyperresearch/skills/hyperresearch-13-gap-fetch.md
index 6f12db7..a18a335 100644
--- a/src/hyperresearch/skills/hyperresearch-13-gap-fetch.md
+++ b/src/hyperresearch/skills/hyperresearch-13-gap-fetch.md
@@ -20,8 +20,8 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- All `research/critic-findings-*.json` files (which exist depends on tier)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- All `research/runs/<vault_tag>/critic-findings-*.json` files (which exist depends on tier)
---
@@ -38,20 +38,20 @@ Read these inputs:
```
If 2+ relevant notes exist, the patcher can handle it — move on. If 0-1 relevant notes exist, this is a **fetch-worthy gap**.
-3. **Collect fetch-worthy gaps.** Cap at **5 gaps maximum** — this is a surgical fill, not a second width sweep. Prioritize by severity (critical first) then by how many critic findings the gap would resolve.
+3. **Collect fetch-worthy gaps.** Cap at **<< p.gap_fetch_cap >> gaps maximum** — this is a surgical fill, not a second width sweep. Prioritize by severity (critical first) then by how many critic findings the gap would resolve.
If 0 fetch-worthy gaps: log "no gaps to fill" and proceed directly to step 14.
-4. **Run targeted fetch wave.** For each gap, generate 2-3 search queries and collect URLs. Spawn **2-4 fetchers** with the gap-filling URLs.
+4. **Run targeted fetch wave.** For each gap, generate 2-3 search queries and collect URLs. Spawn **<< p.gap_fetch_fetchers|hyphen >> fetchers** with the gap-filling URLs.
**Spawn template:**
```
subagent_type: hyperresearch-fetcher
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are a step 13 (post-critic gap-fill) fetcher
of the hyperresearch V8 pipeline. Critics identified gaps in vault
@@ -64,11 +64,11 @@ Read these inputs:
- extra_tags: ["post-critic-fill"]
```
- Each fetcher: fetches, quality-checks, summarizes, extracts claims (same procedure as step 2). Tags notes with `vault_tag` + `post-critic-fill`. Writes claims to `research/temp/claims-<note-id>.json`.
+ Each fetcher: fetches, quality-checks, summarizes, extracts claims (same procedure as step 2). Tags notes with `vault_tag` + `post-critic-fill`. Writes claims to `research/runs/<vault_tag>/temp/claims-<note-id>.json`.
-5. **Update evidence digest.** If new claims were extracted, append them to `research/temp/evidence-digest.md` under a new `### Post-critic gap fill` section. The patcher reads the evidence digest when looking for citation sources to insert.
+5. **Update evidence digest.** If new claims were extracted, append them to `research/runs/<vault_tag>/temp/evidence-digest.md` under a new `### Post-critic gap fill` section. The patcher reads the evidence digest when looking for citation sources to insert.
-6. **Log results** to `research/temp/post-critic-fetch-log.md`:
+6. **Log results** to `research/runs/<vault_tag>/temp/post-critic-fetch-log.md`:
- Each gap: what was searched, how many new sources found, note IDs
- If a gap remained unfilled after fetching: flag it so the patcher knows to acknowledge the limitation rather than fabricate
@@ -76,10 +76,10 @@ Read these inputs:
## Exit criterion
-- `research/temp/post-critic-fetch-log.md` exists (even if it says "no gaps found")
+- `research/runs/<vault_tag>/temp/post-critic-fetch-log.md` exists (even if it says "no gaps found")
- All fetch-worthy gaps attempted (proceed to step 14 whether or not all gaps were filled — unfilled gaps are noted in the log)
-**Cost:** +$1-3 per run (2-4 Sonnet fetchers). Most runs with good step 2 coverage will find 0-2 gaps, making this a near-no-op.
+**Overhead:** small — at most << p.gap_fetch_fetchers|hyphen >> fetchers. Most runs with good step 2 coverage will find 0-2 gaps, making this a near-no-op.
---
diff --git a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
new file mode 100644
index 0000000..ef42f4e
--- /dev/null
+++ b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
@@ -0,0 +1,97 @@
+---
+name: hyperresearch-14-5-cite-check
+description: >
+ Step 14.5 of the hyperresearch V8 pipeline (full + dissertation tiers).
+ Verifies that each citation in the patched report actually supports its
+ sentence: mechanical triage auto-passes pairs confirmed by the claims
+ table, the cite-checker agent judges the sampled remainder, and a second
+ small patcher pass applies the findings. Runs AFTER step 14 (the patcher
+ moved text and citations; audit what will actually ship) and BEFORE step
+ 15 (polish sees the corrected text). Invoked via Skill tool.
+---
+
+# Step 14.5 — Cite-check (citation-sentence binding verification)
+
+**Tier gate:** Runs for `full` and `dissertation`. SKIP for `light` — a light report's citation volume doesn't justify the pass; the `quote-integrity` and `numeric-consistency` lint rules still cover it mechanically at the step-15 gate.
+
+**Goal:** every citation that ships either supports its sentence or gets fixed. This is the difference between "has citations" and "citations are true bindings" — the single thing external fact-checkers actually measure.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/notes/final_report_<vault_tag>.md` — the PATCHED report from step 14
+
+---
+
+## Step 14.5.1 — Extract + mechanical triage
+
+```bash
+$HPR citecheck extract <vault_tag> -j
+```
+
+This parses every (sentence, citation) pair from the report — both `[N]` and `[[note-id]]` styles — and auto-passes pairs whose numbers or wording the cited note's extracted claims already confirm. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:
+- `summary` — total / auto-passed / dangling / needs-llm counts
+- `sampled_for_llm` — the pairs the agent must judge (100% of number-bearing sentences; sampled for the rest)
+- `dangling` — citations that resolve to NO vault note
+
+**Dangling citations are findings immediately** — no agent needed. Each one becomes a `critical` finding (fabricated or mangled citation).
+
+**If `sampled_for_llm` is empty and there are no dangling citations:** write an empty findings file `[]` to `research/runs/<vault_tag>/cite-check-findings.json`, record `$HPR run step <vault_tag> 14.5 --status done -j`, and proceed to step 15. Done.
+
+---
+
+## Step 14.5.2 — Spawn the cite-checker
+
+Spawn ONE `hyperresearch-cite-checker` subagent (two in parallel with split index ranges when `sampled_for_llm` exceeds ~40 pairs):
+
+```
+subagent_type: hyperresearch-cite-checker
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are step 14.5 (cite-checker) of the hyperresearch
+ V8 pipeline. Step 14's patcher already applied critic findings; you
+ verify citation-sentence bindings on the text that will ship. Your
+ findings feed a second, small patcher pass. You do not edit the report.
+
+ YOUR INPUTS:
+ - pairs_file: research/runs/<vault_tag>/cite-check-pairs.json
+ - your_range: sampled_for_llm[<start>..<end>]
+ - findings_path: research/runs/<vault_tag>/cite-check-findings.json
+ - vault_tag: <vault_tag>
+```
+
+When splitting across two checkers, give each its own findings path (`cite-check-findings-a.json` / `-b.json`) and merge the arrays into `cite-check-findings.json` yourself afterward.
+
+---
+
+## Step 14.5.3 — Second patcher pass
+
+Append the dangling-citation findings (from 14.5.1) to the findings file, then reuse the step 14 machinery exactly: spawn ONE `hyperresearch-patcher` (tool-locked Read + Edit) with `research/runs/<vault_tag>/cite-check-findings.json` as its findings input and `research/runs/<vault_tag>/cite-check-patch-log.json` pre-stubbed:
+
+```json
+{"total_findings": 0, "applied": [], "skipped": [], "conflicts": [], "orchestrator_escalated": []}
+```
+
+Fix repertoire (in the findings' `suggested_fix`): swap to `correct_note_id`, soften the claim to what the source supports, or delete the sentence when nothing supports it. Same rules as step 14: surgical hunks, zero regeneration, `critical` findings must not be silently skipped.
+
+**Skip the pass entirely** when the merged findings file is `[]`.
+
+---
+
+## Exit criterion
+
+- `research/runs/<vault_tag>/cite-check-pairs.json` exists
+- `research/runs/<vault_tag>/cite-check-findings.json` exists (possibly `[]`)
+- If findings were non-empty: `cite-check-patch-log.json` shows every `critical` finding applied or escalated
+- Manifest: `$HPR run step <vault_tag> 14.5 --status done -j`
+
+## Next step
+
+Return to the entry skill and invoke `Skill(skill: "hyperresearch-15-polish")`.
diff --git a/src/hyperresearch/skills/hyperresearch-14-patcher.md b/src/hyperresearch/skills/hyperresearch-14-patcher.md
index 66f3836..1be0e7f 100644
--- a/src/hyperresearch/skills/hyperresearch-14-patcher.md
+++ b/src/hyperresearch/skills/hyperresearch-14-patcher.md
@@ -20,26 +20,26 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
- `research/notes/final_report_<vault_tag>.md` — the synthesized final report from step 11
-- All `research/critic-findings-*.json` files (count depends on tier)
-- `research/temp/evidence-digest.md` — patcher's primary citation source
-- `research/query-<vault_tag>.md` — canonical research query
+- All `research/runs/<vault_tag>/critic-findings-*.json` files (count depends on tier)
+- `research/runs/<vault_tag>/temp/evidence-digest.md` — patcher's primary citation source
+- `research/runs/<vault_tag>/query.md` — canonical research query
---
## Step 14.0 — Skip gate (optional)
-Before spawning the patcher, check whether `research/skip-patcher.txt` exists. If it does, the invoker has requested that step 14 be bypassed. In that case, skip to "Skip path" below.
+Before spawning the patcher, check whether `research/runs/<vault_tag>/skip-patcher.txt` exists. If it does, the invoker has requested that step 14 be bypassed. In that case, skip to "Skip path" below.
**Skip path:** Record a minimal log:
```bash
python -c "
import json, pathlib
-files = ['research/critic-findings-dialectic.json','research/critic-findings-depth.json','research/critic-findings-width.json','research/critic-findings-instruction.json']
+files = ['research/runs/<vault_tag>/critic-findings-dialectic.json','research/runs/<vault_tag>/critic-findings-depth.json','research/runs/<vault_tag>/critic-findings-width.json','research/runs/<vault_tag>/critic-findings-instruction.json']
total = sum(len(json.loads(pathlib.Path(f).read_text()).get('findings',[])) for f in files if pathlib.Path(f).exists())
-pathlib.Path('research/patch-log.json').write_text(json.dumps({'total_findings': total, 'applied': [], 'skipped': [{'reason': 'patcher-skipped-by-invoker'}], 'conflicts': [], 'orchestrator_escalated': []}))
+pathlib.Path('research/runs/<vault_tag>/patch-log.json').write_text(json.dumps({'total_findings': total, 'applied': [], 'skipped': [{'reason': 'patcher-skipped-by-invoker'}], 'conflicts': [], 'orchestrator_escalated': []}))
"
```
@@ -52,7 +52,7 @@ Then proceed to step 15. Most runs should not use this gate.
The patcher is tool-locked to `[Read, Edit]` — it cannot Write. Edit can only modify files that already exist. So you (the orchestrator) MUST write the canonical stub first, which the patcher will then Edit to populate:
```bash
-echo '{"total_findings": 0, "applied": [], "skipped": [], "conflicts": [], "orchestrator_escalated": []}' > research/patch-log.json
+echo '{"total_findings": 0, "applied": [], "skipped": [], "conflicts": [], "orchestrator_escalated": []}' > research/runs/<vault_tag>/patch-log.json
```
The schema above is canonical. The patcher's only job on this file is to Edit the existing keys — `total_findings` becomes an integer, the four arrays get populated. **The patcher MUST NOT invent alternate schemas** — downstream tooling assumes the canonical shape.
@@ -70,9 +70,9 @@ Spawn ONCE.
subagent_type: hyperresearch-patcher
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 14 (patcher) of the hyperresearch V8
pipeline. Step 12 produced critic findings; step 13 filled vault gaps.
@@ -82,18 +82,18 @@ prompt: |
YOUR INPUTS:
- draft_path: research/notes/final_report_<vault_tag>.md
- findings_paths: [
- research/critic-findings-dialectic.json, (full tier only)
- research/critic-findings-depth.json, (full tier only)
- research/critic-findings-width.json,
- research/critic-findings-instruction.json
+ research/runs/<vault_tag>/critic-findings-dialectic.json, (full tier only)
+ research/runs/<vault_tag>/critic-findings-depth.json, (full tier only)
+ research/runs/<vault_tag>/critic-findings-width.json,
+ research/runs/<vault_tag>/critic-findings-instruction.json
]
- - patch_log_path: research/patch-log.json (already stubbed)
- - evidence_digest_path: research/temp/evidence-digest.md
+ - patch_log_path: research/runs/<vault_tag>/patch-log.json (already stubbed)
+ - evidence_digest_path: research/runs/<vault_tag>/temp/evidence-digest.md
```
The patcher's job:
- Apply each finding's patch as an Edit on the draft file
-- Populate the pre-stubbed patch log via Edit on `research/patch-log.json`
+- Populate the pre-stubbed patch log via Edit on `research/runs/<vault_tag>/patch-log.json`
- Each Edit hunk stays surgical: change as little as possible while addressing the issue
- Reject findings that don't serve the research_query (the patcher checks every finding against the canonical query)
- Escalate findings that require structural restructure (rather than applying them as oversized patches)
@@ -113,7 +113,7 @@ Check the patch log when the patcher returns:
- **Did the patcher log a "patch too large" skip?** That means a critic proposed regeneration in patch clothing. If the finding was critical, re-spawn the critic with a tighter suggestion, or address it yourself with multiple small hunks.
-- **Is the patch log still the empty stub?** If yes, the patcher failed to log — its Task result will contain the real log inline. Read the Task result, parse out the JSON, and write it to `research/patch-log.json` yourself via Bash so downstream lint rules see it.
+- **Is the patch log still the empty stub?** If yes, the patcher failed to log — its Task result will contain the real log inline. Read the Task result, parse out the JSON, and write it to `research/runs/<vault_tag>/patch-log.json` yourself via Bash so downstream lint rules see it.
---
@@ -125,7 +125,7 @@ For each entry:
1. Read the `issue` field to understand which H2 in the draft needs to move, be added, or be renamed.
2. Apply the restructure via hand-written Edit calls on `research/notes/final_report_<vault_tag>.md`. You have Write and Edit access — the tool lock applies only to the patcher and polish auditor subagents.
3. Preserve the body content within each H2 section — you are moving / renaming / inserting headings, not regenerating prose. If a new heading is added and its body needs fresh content, write a short evidence-grounded paragraph for it.
-4. Log changes in `research/orchestrator-restructure-log.md` (plain markdown, one bullet per change) so downstream lint rules can see this step happened.
+4. Log changes in `research/runs/<vault_tag>/orchestrator-restructure-log.md` (plain markdown, one bullet per change) so downstream lint rules can see this step happened.
5. Never regenerate a whole section or the whole draft. The "patch not regenerate" invariant still binds you — broader tools but not broader license.
---
@@ -140,17 +140,17 @@ For each entry:
## Exit criterion
-- `research/patch-log.json` exists with `total_findings` set and at least one of `applied` / `skipped` / `conflicts` populated
+- `research/runs/<vault_tag>/patch-log.json` exists with `total_findings` set and at least one of `applied` / `skipped` / `conflicts` populated
- All critical findings either applied or resolved by orchestrator
-- All `orchestrator_escalated` findings handled (with `research/orchestrator-restructure-log.md` if any structural restructures were applied)
+- All `orchestrator_escalated` findings handled (with `research/runs/<vault_tag>/orchestrator-restructure-log.md` if any structural restructures were applied)
- `research/notes/final_report_<vault_tag>.md` has been edited (or no edits needed if findings were trivial)
---
## Next step
-Return to the entry skill (`hyperresearch`). Invoke step 15:
+Return to the entry skill (`hyperresearch`). Invoke step 14.5 (cite-check):
```
-Skill(skill: "hyperresearch-15-polish")
+Skill(skill: "hyperresearch-14-5-cite-check")
```
diff --git a/src/hyperresearch/skills/hyperresearch-15-polish.md b/src/hyperresearch/skills/hyperresearch-15-polish.md
index 832d353..eb4ba4a 100644
--- a/src/hyperresearch/skills/hyperresearch-15-polish.md
+++ b/src/hyperresearch/skills/hyperresearch-15-polish.md
@@ -22,7 +22,7 @@ description: >
Read these inputs:
- `research/notes/final_report_<vault_tag>.md` — the patched draft from step 14 (or single-pass draft for light tier)
-- `research/query-<vault_tag>.md` — canonical research query
+- `research/runs/<vault_tag>/query.md` — canonical research query
---
@@ -31,7 +31,7 @@ Read these inputs:
The polish auditor has `[Read, Edit]` only and cannot create a new file (same tool-lock rule as the step 14 patcher). Stub it first:
```bash
-echo '{"applied": [], "escalations": []}' > research/polish-log.json
+echo '{"applied": [], "escalations": []}' > research/runs/<vault_tag>/polish-log.json
```
---
@@ -45,9 +45,9 @@ Spawn ONCE.
subagent_type: hyperresearch-polish-auditor
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 15 (polish auditor) of the
hyperresearch V8 pipeline — the final step. Step 14 (patcher) applied
@@ -57,7 +57,7 @@ prompt: |
YOUR INPUTS:
- draft_path: research/notes/final_report_<vault_tag>.md
- - polish_log_path: research/polish-log.json (already stubbed)
+ - polish_log_path: research/runs/<vault_tag>/polish-log.json (already stubbed)
```
The polish auditor strips:
@@ -85,16 +85,16 @@ If the escalation names a structural issue (e.g., "user asked for a ranked list;
Before declaring the run complete, verify every expected pipeline artifact exists. **The required set depends on the tier:**
-- **light tier:** only `research/polish-log.json` is required (steps 12–14 are skipped, so no critic findings or patch log).
+- **light tier:** only `research/runs/<vault_tag>/polish-log.json` is required (steps 12–14 are skipped, so no critic findings or patch log).
- **full tier:** require all four critic findings + patch-log + polish-log:
```bash
-for f in research/critic-findings-dialectic.json \
- research/critic-findings-depth.json \
- research/critic-findings-width.json \
- research/critic-findings-instruction.json \
- research/patch-log.json \
- research/polish-log.json; do
+for f in research/runs/<vault_tag>/critic-findings-dialectic.json \
+ research/runs/<vault_tag>/critic-findings-depth.json \
+ research/runs/<vault_tag>/critic-findings-width.json \
+ research/runs/<vault_tag>/critic-findings-instruction.json \
+ research/runs/<vault_tag>/patch-log.json \
+ research/runs/<vault_tag>/polish-log.json; do
test -f "$f" || echo "MISSING: $f"
done
```
@@ -105,7 +105,7 @@ If any artifact is missing, the responsible step failed silently. Re-spawn the r
## Step 15.5 — Record the run + lint gate
-1. **Record the run.** Append to `research/audit_findings.json`:
+1. **Record the run.** Append to `research/runs/<vault_tag>/audit_findings.json`:
```json
{
"mode": "hyperresearch-v8",
@@ -142,7 +142,7 @@ The final report lives at `research/notes/final_report_<vault_tag>.md`. The wrap
## Exit criterion
-- `research/polish-log.json` populated
+- `research/runs/<vault_tag>/polish-log.json` populated
- Final integrity gate passed (or stub-filled with documented failure)
- Lint gate passed
- `research/notes/final_report_<vault_tag>.md` is the final, shippable artifact
diff --git a/src/hyperresearch/skills/hyperresearch-16-readability-audit.md b/src/hyperresearch/skills/hyperresearch-16-readability-audit.md
index fbf8f91..633ca7a 100644
--- a/src/hyperresearch/skills/hyperresearch-16-readability-audit.md
+++ b/src/hyperresearch/skills/hyperresearch-16-readability-audit.md
@@ -2,8 +2,8 @@
name: hyperresearch-16-readability-audit
description: >
Step 16 (final) of the hyperresearch V8 pipeline. Spawns the
- hyperresearch-readability-recommender subagent (Read+Write tool-locked,
- Opus) to audit the polished final report and write JSON
+ hyperresearch-readability-recommender subagent (Read+Write
+ tool-locked) to audit the polished final report and write JSON
recommendations for paragraph merges, breaks, list/table conversions,
bold injection, sentence splits, and HR removal. The orchestrator
reads the recommendations and SELECTIVELY applies them via direct
@@ -25,7 +25,7 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
- `research/notes/final_report_<vault_tag>.md` — the polished final report from step 15
---
@@ -39,9 +39,9 @@ Spawn ONE `hyperresearch-readability-recommender` subagent. Single spawn, runs o
subagent_type: hyperresearch-readability-recommender
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 16 of the hyperresearch V8 pipeline —
the final analytical pass. The final report at
@@ -55,10 +55,10 @@ prompt: |
YOUR INPUTS:
- draft_path: research/notes/final_report_<vault_tag>.md
- - recommendations_path: research/readability-recommendations.json
+ - recommendations_path: research/runs/<vault_tag>/readability-recommendations.json
Write recommendations as a JSON array per the schema in your agent
- prompt. Cap at 50 recommendations, prioritized by impact.
+ prompt. Cap at << p.readability_rec_cap >> recommendations, prioritized by impact.
```
---
@@ -67,7 +67,7 @@ prompt: |
When the recommender returns:
-1. **Read `research/readability-recommendations.json`.**
+1. **Read `research/runs/<vault_tag>/readability-recommendations.json`.**
2. **Read the recommender's report-back.** It tells you:
- Total count of recommendations
@@ -128,7 +128,7 @@ If an Edit fails because `old_string` doesn't match (recommender mis-anchored),
## Step 16.5 — Log decisions
-Write `research/readability-decisions.json` with the orchestrator's decisions:
+Write `research/runs/<vault_tag>/readability-decisions.json` with the orchestrator's decisions:
```json
{
@@ -150,8 +150,8 @@ This is the audit trail. If a future review finds a readability problem we shoul
## Exit criterion
-- `research/readability-recommendations.json` exists
-- `research/readability-decisions.json` exists with at least one entry in `applied` or all `skipped`
+- `research/runs/<vault_tag>/readability-recommendations.json` exists
+- `research/runs/<vault_tag>/readability-decisions.json` exists with at least one entry in `applied` or all `skipped`
- `research/notes/final_report_<vault_tag>.md` reflects the applied recommendations
- The final report's structure (H2 list, executive summary, conclusion) is unchanged from step 15's output (this step does not restructure)
diff --git a/src/hyperresearch/skills/hyperresearch-2-width-sweep.md b/src/hyperresearch/skills/hyperresearch-2-width-sweep.md
index cf5b5ad..f7427a7 100644
--- a/src/hyperresearch/skills/hyperresearch-2-width-sweep.md
+++ b/src/hyperresearch/skills/hyperresearch-2-width-sweep.md
@@ -3,7 +3,7 @@ name: hyperresearch-2-width-sweep
description: >
Step 2 of the hyperresearch V8 pipeline. Multi-perspective search planning
(breadth / depth / adversarial lenses) followed by parallel fetcher waves.
- Achieves comprehensive topical coverage with 40-100 curated sources for
+ Achieves comprehensive topical coverage with << p.source_target|dash >> curated sources for
full tier. Includes coverage check, evidence redundancy audit,
and source count gating. Invoked via Skill tool from the entry skill
after step 1 completes.
@@ -11,19 +11,19 @@ description: >
# Step 2 — Width sweep
-**Tier gate:** Runs for ALL tiers. For `light` tier: skip academic APIs, target 12–20 sources, limit to 2–3 fetcher batches. For `full`: run the full procedure below.
+**Tier gate:** Runs for ALL tiers. For `light` tier: skip academic APIs, target << light.source_target|dash >> sources, limit to << light.batch_count|dash >> fetcher batches. For `full`: run the full procedure below.
-**Goal:** achieve comprehensive topical coverage — every atomic item from the decomposition must have at least 3 supporting sources by the end of this step. Target 40–80 curated sources for `full` tier.
+**Goal:** achieve comprehensive topical coverage — every atomic item from the decomposition must have at least 3 supporting sources by the end of this step. Target << p.source_target|dash >> curated sources for `full` tier.
---
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag (in Run config), modality
-- `research/prompt-decomposition.json` — atomic items, sub_questions, entities, pipeline_tier
-- `research/temp/coverage-matrix.md` — verbatim query phrases mapped to atomic items
-- `research/query-<vault_tag>.md` — canonical research query (GOSPEL)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag (in Run config), modality
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, sub_questions, entities, pipeline_tier
+- `research/runs/<vault_tag>/temp/coverage-matrix.md` — verbatim query phrases mapped to atomic items
+- `research/runs/<vault_tag>/query.md` — canonical research query (GOSPEL)
---
@@ -63,7 +63,7 @@ Before spawning any fetchers, produce a **search plan** that maps the decomposit
- **Earnings-call transcripts are insufficient on their own.** Transcripts narrate already-rounded numbers ("revenue grew about 27%"); rubrics demand the tabular line items from the filing itself. If the prompt names a fiscal period, the search plan MUST include a query for the filing PDF, not just the transcript.
- Goal: every period in `time_periods` has at least one search that, if successful, fetches the filing's tabular data — not a paraphrase of it.
-3. **Write the combined search plan to `research/temp/search-plan.md`** — a table with a `Lens` column:
+3. **Write the combined search plan to `research/runs/<vault_tag>/temp/search-plan.md`** — a table with a `Lens` column:
```markdown
| Atomic item | Search query | Type | Lens | Target |
|---|---|---|---|---|
@@ -73,9 +73,9 @@ Before spawning any fetchers, produce a **search plan** that maps the decomposit
| Entity: PE | "China private equity returns academic study" | academic | depth | canonical |
```
- Plan typically has **40–100 planned searches** for a `full` query.
+ Plan typically has **<< p.planned_searches|dash >> planned searches** for a `full` query.
-4. **Search gap check.** Cross-check the search plan against `research/temp/coverage-matrix.md`. For every row in the coverage matrix, verify at least one search in the plan targets that query phrase's atomic item. Re-read the verbatim query and check: is there any significant topic, entity, or category in the query that has ZERO rows in the search plan?
+4. **Search gap check.** Cross-check the search plan against `research/runs/<vault_tag>/temp/coverage-matrix.md`. For every row in the coverage matrix, verify at least one search in the plan targets that query phrase's atomic item. Re-read the verbatim query and check: is there any significant topic, entity, or category in the query that has ZERO rows in the search plan?
Common failure modes this catches:
- Decomposition correctly listed "rugged tablets" but search plan has no queries for tablet manufacturers, enterprise mobility, or field-service devices
@@ -86,7 +86,7 @@ Before spawning any fetchers, produce a **search plan** that maps the decomposit
5. **Execute searches from ALL three lenses.** Do not shortcut by running only Lens A. The adversarial and depth lenses produce qualitatively different URLs that breadth searching misses.
-6. **Minimum adversarial coverage:** at least **5 adversarial searches total**. The dialectic critic will punish one-sided coverage.
+6. **Minimum adversarial coverage:** at least **<< p.adversarial_searches_min >> adversarial searches total**. The dialectic critic will punish one-sided coverage.
---
@@ -94,13 +94,13 @@ Before spawning any fetchers, produce a **search plan** that maps the decomposit
1. **Academic APIs first.** For topics with a research literature, hit Semantic Scholar / arXiv / OpenAlex / PubMed BEFORE web search. Academic APIs return citation-ranked canonical papers.
-2. **Web searches from the plan.** Execute ALL planned searches across all three lenses. Aim for **80–120 candidate URLs** before deduplication for `full` tier.
+2. **Web searches from the plan.** Execute ALL planned searches across all three lenses. Aim for **<< p.candidate_urls|dash >> candidate URLs** before deduplication for `full` tier.
-3. **Build and deduplicate the master URL queue.** Remove exact-URL duplicates. Remove obvious junk domains. The deduplicated queue should have **60–100 URLs** for `full` tier.
+3. **Build and deduplicate the master URL queue.** Remove exact-URL duplicates. Remove obvious junk domains. The deduplicated queue should have **<< p.deduped_urls|dash >> URLs** for `full` tier.
**Wikipedia SOURCE HUB rule:** Include Wikipedia URLs in the queue — they're valuable for discovery — but treat them as SOURCE HUBS, not as citable sources. When a fetcher processes a Wikipedia article, it extracts the references/citations Wikipedia links to. Those primary sources go into Wave 2 (or the same wave if capacity permits). Wikipedia itself is NEVER cited in the final report.
-4. **Partition the queue into non-overlapping batches.** Split the master queue into **10–12 batches** of **8–12 URLs each**. Each batch goes to exactly ONE fetcher. **Zero overlap.**
+4. **Partition the queue into non-overlapping batches.** Split the master queue into **<< p.batch_count|dash >> batches** of **<< p.batch_size|dash >> URLs each**. Each batch goes to exactly ONE fetcher. **Zero overlap.**
---
@@ -119,13 +119,15 @@ Before batching URLs, score each candidate URL on six dimensions (0–3 each, ma
**Selection rule:** Rank by composite utility score. Select the top N URLs (where N = batch capacity × batch count). Hard constraint: every atomic item must have ≥3 candidate URLs before low-utility URLs from well-covered items are included.
-Write to `research/temp/scored-urls.md`.
+Write to `research/runs/<vault_tag>/temp/scored-urls.md`.
+
+**Scores travel with the URLs.** When you assign batches (step 2.4), include each URL's composite utility score next to it. Fetchers pass it to `$HPR fetch --utility-score <N>` so the score persists into note frontmatter — it becomes one input to the vault's composite `quality_score`, which step 10 uses for ranked curation.
---
## Step 2.4 — Parallel fetcher waves
-**Wave 1 (main wave):** Spawn **10–12 fetcher subagents in ONE message** — true parallel execution. Each fetcher gets its own non-overlapping batch.
+**Wave 1 (main wave):** Spawn **<< p.wave1_fetchers|dash >> fetcher subagents in ONE message** — true parallel execution. Each fetcher gets its own non-overlapping batch.
**Subagent type:** `hyperresearch-fetcher`
@@ -134,9 +136,9 @@ Write to `research/temp/scored-urls.md`.
subagent_type: hyperresearch-fetcher
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste contents of research/query-<vault_tag>.md}}
+ > {{paste contents of research/runs/<vault_tag>/query.md}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 2 (width-sweep fetcher) of the
hyperresearch V8 pipeline. The orchestrator partitioned the URL queue into
@@ -146,7 +148,7 @@ prompt: |
YOUR INPUTS:
- vault_tag: <vault_tag>
- - urls: [<batch URLs, exactly as assigned>]
+ - urls: [<batch URLs, exactly as assigned — with each URL's utility score when scored, e.g. "https://... (utility: 14)">]
- batch_id: <number>
```
@@ -154,7 +156,7 @@ prompt: |
**CRITICAL: never emit bare text while waiting.** In `-p` mode, a text-only response triggers `end_turn`.
-**Use wait time to think.** While subagents are working, write evolving thoughts to `research/temp/orchestrator-notes.md`:
+**Use wait time to think.** While subagents are working, write evolving thoughts to `research/runs/<vault_tag>/temp/orchestrator-notes.md`:
- What patterns are emerging from sources?
- What tensions or contradictions do you expect?
- What's the strongest thesis forming? What could overturn it?
@@ -168,7 +170,7 @@ Append a few lines with `Edit` or `Write` every 30-60 seconds. Productive thinki
PYTHONIOENCODING=utf-8 $HPR note list --tag <vault_tag> --all --json | python -c "import sys,json; d=json.load(sys.stdin); print(f'Notes in vault: {len(d.get(\"data\",[]))}')"
```
-The wave is done when the vault note count is ≥80% of total URLs queued.
+The wave is done when the vault note count is ≥<< (p.wave_done_ratio * 100)|int >>% of total URLs queued.
---
@@ -186,10 +188,10 @@ After Wave 1 returns, run the coverage check before proceeding:
3. **Wave 2 fetch for gaps.** For every `thin` or `uncovered` item:
- Run 2–3 targeted searches specifically for that item
- - Spawn 3–5 fetchers with gap-filling URLs (non-overlapping batches)
+ - Spawn << p.wave2_fetchers|dash >> fetchers with gap-filling URLs (non-overlapping batches)
- This wave is smaller (typically 20–40 URLs) but surgically targeted
-4. **Write coverage report** to `research/temp/coverage-gaps.md`:
+4. **Write coverage report** to `research/runs/<vault_tag>/temp/coverage-gaps.md`:
- List every atomic item with its coverage status and source count
- Any item still at 0 sources after Wave 2 is a genuine gap — flag it prominently
@@ -203,7 +205,7 @@ After Wave 1 returns, run the coverage check before proceeding:
**Goal:** detect when N sources are really 1 source in N outfits.
-1. **Collect all claims.** Read `research/temp/claims-<note-id>.json` for every non-deprecated note tagged `<vault_tag>`. If no claim files exist, skip this step.
+1. **Collect all claims.** Read `research/runs/<vault_tag>/temp/claims-<note-id>.json` for every non-deprecated note tagged `<vault_tag>`. If no claim files exist, skip this step.
2. **Cluster by content overlap.** Sources sharing >60% of their `quoted_support` passages are likely derivative.
@@ -211,9 +213,66 @@ After Wave 1 returns, run the coverage check before proceeding:
4. **For each cluster, identify the canonical upstream source.** Tag derivative sources with `derivative-of`. Do NOT deprecate them — discount them in coverage counting.
-5. **Write `research/temp/redundancy-audit.md`** — clusters, adjusted coverage counts, atomic items dropping below 2 → flag for Wave 3.
+5. **Write `research/runs/<vault_tag>/temp/redundancy-audit.md`** — clusters, adjusted coverage counts, atomic items dropping below 2 → flag for Wave 3.
+
+6. **Wave 3 fetch (conditional).** If any atomic item's independent source count drops below 2, run targeted searches for INDEPENDENT sources. Spawn << p.wave3_fetchers|hyphen >> fetchers.
+
+---
+
+## Step 2.7 — Persist ranking signals (ALL tiers, cheap, run after the last wave)
+
+Four commands that turn the corpus into a *ranked* corpus. Run them once, in order, after fetching completes:
+
+```bash
+$HPR claims ingest --tag <vault_tag> -j # claims JSONs -> queryable claims table
+$HPR sources backfill-doi --tag <vault_tag> -j # catch DOIs the fetchers missed
+$HPR sources score --tag <vault_tag> -j # citation counts + retraction check (cached APIs)
+$HPR graph rank -j # vault centrality + composite quality_score
+```
+
+**If `sources score` reports RETRACTED sources:** flag them in `research/runs/<vault_tag>/temp/coverage-gaps.md` immediately — a retracted source must never anchor a locus or survive into drafting as unqualified evidence. The retraction floor also crushes its `quality_score`, so ranked curation (step 10) buries it automatically.
+
+These commands are local/cached and cost seconds. Skipping them leaves step 10's ranked curation blind.
+
+---
+
+## Step 2.8 — Drain the browser-lane escalation queue (conditional)
+
+Blocked fetches (login walls, bot walls, captchas) were NOT lost — the fetch gate queued them:
+
+```bash
+$HPR escalation list --status queued --tag <vault_tag> -j
+```
+
+**If queued items exist**, spawn EXACTLY ONE `hyperresearch-browser-fetcher` subagent to drain them (serial, one browser — never spawn two):
+
+```
+subagent_type: hyperresearch-browser-fetcher
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are the step 2 escalation-lane fetcher of the
+ hyperresearch V8 pipeline. Headless fetchers hit walls on these URLs;
+ you drive the user's real Chrome browser to fetch them. After you
+ return, the orchestrator consolidates any needs_human items for the
+ user and re-runs the ranking commands (step 2.7).
+
+ YOUR INPUTS:
+ - vault_tag: <vault_tag>
+ - drain up to 10 items (claim via `$HPR escalation claim --tag <vault_tag>`)
+```
+
+**When the browser-fetcher returns with `needs_human` items** (CAPTCHAs, logins, 2FA — it NEVER solves these itself):
+
+1. **Consolidate into ONE message to the user** — never one interruption per URL:
+ > "3 sources need you: [site A: solve the CAPTCHA], [site B: log in], [site C: approve 2FA]. Open them in Chrome, complete the challenges, then tell me 'done' (or 'skip')."
+2. In non-interactive (`-p`) runs where no user can answer: record `$HPR run block <vault_tag> --on human-challenges -j` and CONTINUE the pipeline with everything else — the queue drains on the next `hpr run resume`.
+3. After the user says done: `$HPR escalation retry <id>` each item, re-spawn the browser-fetcher once, then re-run step 2.7's ranking commands so the new sources are scored.
-6. **Wave 3 fetch (conditional).** If any atomic item's independent source count drops below 2, run targeted searches for INDEPENDENT sources. Spawn 2-3 fetchers.
+**If the Claude-in-Chrome extension is unavailable**, the queue simply accumulates — report the queued count in your wave summary and move on. Abandoned/queued items are exactly the pre-4.0 status quo (lost sources), never worse.
---
@@ -221,32 +280,32 @@ After Wave 1 returns, run the coverage check before proceeding:
| Tier | Minimum sources | Target sources | Fetchers per wave | Waves |
|------|----------------|---------------|-------------------|-------|
-| `light` | 10 | 15–25 | 3–5 | 1–2 |
-| `full` | 45 | 55–80 | 8–12 | 2–3 |
+| `light` | << light.source_min >> | << light.source_target|dash >> | << light.wave1_fetchers|dash >> | << light.waves|dash >> |
+| `full` | << p.source_min >> | << p.source_target|dash >> | << p.wave1_fetchers|dash >> | << p.waves|dash >> |
-Substantive (non-deprecated) note counts. Quality over quantity — reference reports average ~65 sources. Beyond ~80, each additional source yields diminishing returns while degrading summarizer quality.
+Substantive (non-deprecated) note counts. The `full` row reflects the installed scale gear. Quality over quantity — beyond ~<< p.source_target[1] >> sources, each additional source yields diminishing returns while degrading summarizer quality.
---
## Long-source delegation (any time during step 2)
-When a single long source (>5000 words) is load-bearing, delegate end-to-end analysis to `hyperresearch-source-analyst` (Sonnet, 1M context):
+When a single long source (><< p.source_analyst_word_trigger >> words) is load-bearing, delegate end-to-end analysis to `hyperresearch-source-analyst` (full-source deep read):
Trigger conditions (ALL three must hold):
-1. **Length:** source's `word_count` (visible on `$HPR note show <id> -j`) exceeds ~5000 words
+1. **Length:** source's `word_count` (visible on `$HPR note show <id> -j`) exceeds ~<< p.source_analyst_word_trigger >> words
2. **Relevance:** source is relevant to the research_query
3. **No existing analysis:** no `type: source-analysis` note already exists for this source
-**Cap: at most 6 source-analysts per query.**
+**Cap: at most << p.source_analyst_cap >> source-analysts per query.**
Spawn template:
```
subagent_type: hyperresearch-source-analyst
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are a leaf subagent for deep end-to-end analysis
of ONE long source. Your digest feeds downstream hyperresearch V8 steps. You
@@ -254,7 +313,7 @@ prompt: |
YOUR INPUTS:
- source_note_id: <vault note id of the long source>
- - output_path: research/temp/source-analysis-<source_note_id>.md
+ - output_path: research/runs/<vault_tag>/temp/source-analysis-<source_note_id>.md
- vault_tag: <vault_tag>
```
@@ -264,8 +323,8 @@ prompt: |
- Minimum source count met (per tier table)
- Coverage check shows no `uncovered` atomic items (thin is acceptable)
-- `research/temp/coverage-gaps.md` written
-- (For std/full): `research/temp/redundancy-audit.md` written if any claim files existed
+- `research/runs/<vault_tag>/temp/coverage-gaps.md` written
+- (For std/full): `research/runs/<vault_tag>/temp/redundancy-audit.md` written if any claim files existed
If you fall short after two waves, proceed anyway but ensure `coverage-gaps.md` lists what's missing so the drafter handles it.
diff --git a/src/hyperresearch/skills/hyperresearch-3-contradiction-graph.md b/src/hyperresearch/skills/hyperresearch-3-contradiction-graph.md
index 0e43397..ff8e670 100644
--- a/src/hyperresearch/skills/hyperresearch-3-contradiction-graph.md
+++ b/src/hyperresearch/skills/hyperresearch-3-contradiction-graph.md
@@ -20,9 +20,9 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/prompt-decomposition.json` — atomic items, pipeline_tier
-- All `research/temp/claims-*.json` files (one per fetched note)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, pipeline_tier
+- All `research/runs/<vault_tag>/temp/claims-*.json` files (one per fetched note)
If no claims files exist (e.g., fetchers didn't produce them), skip this step entirely — the next step (loci analysis) falls back to corpus prose-scanning.
@@ -30,7 +30,7 @@ If no claims files exist (e.g., fetchers didn't produce them), skip this step en
## Procedure
-1. **Load all claims** from `research/temp/claims-*.json` files.
+1. **Load all claims** from `research/runs/<vault_tag>/temp/claims-*.json` files.
2. **Pair contradictions.** For each claim, find claims from OTHER sources that contradict it. Match on:
- Same `stance_target` with opposing `stance` (supports vs. refutes)
@@ -53,16 +53,16 @@ If no claims files exist (e.g., fetchers didn't produce them), skip this step en
4. **Rank clusters** by decision_relevance (high first), then by evidence_quality_delta (tighter fights rank higher).
-5. **Write `research/temp/contradiction-graph.json`** — array of ranked fight clusters.
+5. **Write `research/runs/<vault_tag>/temp/contradiction-graph.json`** — array of ranked fight clusters.
-6. **Identify consensus claims.** Claims where 3+ INDEPENDENT sources (after redundancy audit if step 2.6 ran) agree. Write these to `research/temp/consensus-claims.json`. These are the "settled ground" the draft can assert confidently without hedging.
+6. **Identify consensus claims.** Claims where 3+ INDEPENDENT sources agree. Independence is computed, not assumed: run `$HPR sources independence --tag <vault_tag> -j` first — it clusters syndicated copies (same canonical URL, near-duplicate bodies, shared wire-service boilerplate) and scores members `1/cluster_size`. **Count each cluster as ONE voice** (sum of independence scores ≥ 3.0, not raw source count ≥ 3). Write consensus claims to `research/runs/<vault_tag>/temp/consensus-claims.json`. These are the "settled ground" the draft can assert confidently without hedging.
---
## Exit criterion
-- `research/temp/contradiction-graph.json` exists (may be empty array if corpus is univocal)
-- `research/temp/consensus-claims.json` exists (may be empty array)
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` exists (may be empty array if corpus is univocal)
+- `research/runs/<vault_tag>/temp/consensus-claims.json` exists (may be empty array)
---
diff --git a/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md b/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md
index a1c6d6b..e1f111c 100644
--- a/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md
+++ b/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md
@@ -20,10 +20,10 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/prompt-decomposition.json` — atomic items, sub-questions
-- `research/temp/contradiction-graph.json` — ranked fight clusters (if step 3 ran)
-- `research/temp/coverage-gaps.md` — which atomic items have weak coverage
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, sub-questions
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` — ranked fight clusters (if step 3 ran)
+- `research/runs/<vault_tag>/temp/coverage-gaps.md` — which atomic items have weak coverage
Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width sweep is complete.
@@ -31,32 +31,32 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
## Procedure
-1. **Spawn 2 `hyperresearch-loci-analyst` subagents in parallel** (ONE message, both Task calls). Both read the same width corpus but return independently.
+1. **Spawn << p.loci_analysts >> `hyperresearch-loci-analyst` subagents in parallel** (ONE message, both Task calls). Both read the same width corpus but return independently.
**Spawn template:**
```
subagent_type: hyperresearch-loci-analyst
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 4 (loci-analyst, instance A or B) of
the hyperresearch V8 pipeline. The width sweep (step 2) populated the vault
tagged <vault_tag>. The contradiction graph (step 3) lives at
- research/temp/contradiction-graph.json. After you and the other
+ research/runs/<vault_tag>/temp/contradiction-graph.json. After you and the other
analyst return, the orchestrator dedupes your loci and assigns budgets.
YOUR INPUTS:
- corpus_tag: <vault_tag>
- analyst_id: "a" (for one) / "b" (for the other)
- - output_path: research/loci-a.json (or research/loci-b.json)
+ - output_path: research/runs/<vault_tag>/loci-a.json (or research/runs/<vault_tag>/loci-b.json)
```
2. **Wait for both.** If one fails, proceed with the single successful output. If both fail (empty loci lists), tell the user the width sweep was too thin and stop — do not force depth on a weak corpus.
-3. **Deduplicate and clamp to 6.**
+3. **Deduplicate and clamp to << p.loci_max >>.**
- Read both JSON outputs.
- Dedupe on `name` (exact match) or near-match (same core question, different phrasing). When in doubt, prefer the entry with stronger `corpus_evidence`.
- If the deduped list exceeds 6, drop the weakest entries — rank by how load-bearing the rationale is for the canonical research query.
@@ -70,15 +70,15 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
**Composite score** = importance + uncertainty + disagreement + decision_impact (max 40).
- **Allocate source budgets.** Total source budget for step 5 is 40. Distribute proportionally:
- - Loci scoring 30-40: `source_budget` up to 15 (deep dive)
- - Loci scoring 20-29: `source_budget` up to 10 (standard)
- - Loci scoring 10-19: `source_budget` up to 5 (shallow pass)
+ **Allocate source budgets.** Total source budget for step 5 is << p.depth_budget_total >>. Distribute proportionally:
+ - Loci scoring 30-40: `source_budget` up to << p.depth_budget_brackets[0][1] >> (deep dive)
+ - Loci scoring 20-29: `source_budget` up to << p.depth_budget_brackets[1][1] >> (standard)
+ - Loci scoring 10-19: `source_budget` up to << p.depth_budget_brackets[2][1] >> (shallow pass)
- Loci scoring <10: `source_budget` 0-3, or skip investigation entirely
It's fine if only 1-2 loci score above 20 — allocate heavily to them.
-5. **Write scored loci to `research/loci.json`.** Schema:
+5. **Write scored loci to `research/runs/<vault_tag>/loci.json`.** Schema:
```json
{
"loci": [
@@ -99,7 +99,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
}
```
-6. **Decide investigator count.** Spawn ONE depth-investigator (in step 5) per locus with `source_budget > 0`, capped at 6. If only 1 locus passes scoring, spawn 1.
+6. **Decide investigator count.** Spawn ONE depth-investigator (in step 5) per locus with `source_budget > 0`, capped at << p.investigator_max >>. If only 1 locus passes scoring, spawn 1.
**INVARIANT:** at least one `flavor: "dialectical"` locus must be present unless an analyst's `skip_loci` justifies its absence with specific evidence of a univocal corpus. No dialectical locus + no justification = re-spawn the loci-analyst with a tighter prompt.
@@ -109,7 +109,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
## Exit criterion
-- `research/loci.json` exists with at least 1 locus (or both analysts justified skip with `skip_loci`)
+- `research/runs/<vault_tag>/loci.json` exists with at least 1 locus (or both analysts justified skip with `skip_loci`)
- At least one dialectical locus OR a documented justification in `skip_loci`
- All retained loci have `source_budget` allocated
diff --git a/src/hyperresearch/skills/hyperresearch-5-depth-investigation.md b/src/hyperresearch/skills/hyperresearch-5-depth-investigation.md
index af1541b..d4f8908 100644
--- a/src/hyperresearch/skills/hyperresearch-5-depth-investigation.md
+++ b/src/hyperresearch/skills/hyperresearch-5-depth-investigation.md
@@ -20,32 +20,32 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/loci.json` — scored loci with source_budget per locus
-- `research/temp/contradiction-graph.json` (if step 3 ran)
-- `research/query-<vault_tag>.md` — canonical research query
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/loci.json` — scored loci with source_budget per locus
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` (if step 3 ran)
+- `research/runs/<vault_tag>/query.md` — canonical research query
---
## Procedure
-1. **Spawn K `hyperresearch-depth-investigator` subagents in parallel** (ONE message, all Task calls). One per locus with `source_budget > 0`, capped at 6.
+1. **Spawn K `hyperresearch-depth-investigator` subagents in parallel** (ONE message, all Task calls). One per locus with `source_budget > 0`, capped at << p.investigator_max >>.
**Spawn template:**
```
subagent_type: hyperresearch-depth-investigator
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 5 (depth-investigator) of the
- hyperresearch V8 pipeline. Step 4's loci analysts produced research/loci.json;
+ hyperresearch V8 pipeline. Step 4's loci analysts produced research/runs/<vault_tag>/loci.json;
after you return, step 6 will reconcile your committed position against
- the other investigators' positions in research/comparisons.md.
+ the other investigators' positions in research/runs/<vault_tag>/comparisons.md.
- YOUR LOCUS (from research/loci.json):
+ YOUR LOCUS (from research/runs/<vault_tag>/loci.json):
- name: "<locus name>"
- one_line: "<one-line locus description>"
- flavor: "dialectical" / "synthesis" / "technical"
diff --git a/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md b/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md
index 94e6109..eb31649 100644
--- a/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md
+++ b/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md
@@ -2,7 +2,7 @@
name: hyperresearch-6-cross-locus-reconcile
description: >
Step 6 of the hyperresearch V8 pipeline. Reconciles the committed positions
- from all depth investigators into research/comparisons.md — 3-5 named
+ from all depth investigators into research/runs/<vault_tag>/comparisons.md — 3-5 named
cross-locus tensions with engagement guidance for the draft. This is
the structural step that gives the single draft argumentative density.
Invoked via Skill tool from the entry skill (full tier only).
@@ -12,7 +12,7 @@ description: >
**Tier gate:** SKIP entirely for `light` tier (no loci = no comparisons). Only `full` tier runs this step.
-**Goal:** before drafting, reconcile the committed positions from all depth investigators. Produce `research/comparisons.md` — a short document naming 3–5 places where the loci conflict or complicate each other.
+**Goal:** before drafting, reconcile the committed positions from all depth investigators. Produce `research/runs/<vault_tag>/comparisons.md` — a short document naming 3–5 places where the loci conflict or complicate each other.
**Why this step exists:** the depth investigators each committed to a position on their own locus. Some of those positions disagree, some reinforce each other, some partially complicate each other. The draft must engage those cross-locus dynamics explicitly — not summarize each locus in isolation. Writing `comparisons.md` forces you to see the loci in cross-section before opening the draft.
@@ -23,8 +23,8 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/loci.json` — scored loci
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/loci.json` — scored loci
- All interim notes: `$HPR note list --tag <vault_tag> --type interim --all --json` then `$HPR note show <id1> <id2> ... -j`
You need the `## Committed position` section from every interim note in your context.
@@ -33,7 +33,7 @@ You need the `## Committed position` section from every interim note in your con
## Procedure
-1. **Lay out all committed positions.** For each interim note, read its `## Committed position` section. Write them down side-by-side in a scratch list (you can use `research/temp/orchestrator-notes.md` as scratch).
+1. **Lay out all committed positions.** For each interim note, read its `## Committed position` section. Write them down side-by-side in a scratch list (you can use `research/runs/<vault_tag>/temp/orchestrator-notes.md` as scratch).
2. **Hunt for tensions.** Ask of every pair of positions:
- Do they agree on the facts but disagree on what the facts mean?
@@ -44,7 +44,7 @@ You need the `## Committed position` section from every interim note in your con
3. **Pick the 3–5 strongest cross-locus dynamics.** Reject weak ones (loci that are simply orthogonal, or that restate each other). You want cross-locus relationships that a good final draft should actually wrestle with.
-4. **Write `research/comparisons.md`:**
+4. **Write `research/runs/<vault_tag>/comparisons.md`:**
```markdown
# Cross-locus comparisons
@@ -67,7 +67,7 @@ You need the `## Committed position` section from every interim note in your con
## Exit criterion
-- `research/comparisons.md` exists
+- `research/runs/<vault_tag>/comparisons.md` exists
- Contains 3–5 named tensions (or 1 distilled position for single-locus runs)
- Each tension includes: locus references, dynamic description, engagement guidance, calibration note
diff --git a/src/hyperresearch/skills/hyperresearch-7-source-tensions.md b/src/hyperresearch/skills/hyperresearch-7-source-tensions.md
index fc648c8..e43a4fe 100644
--- a/src/hyperresearch/skills/hyperresearch-7-source-tensions.md
+++ b/src/hyperresearch/skills/hyperresearch-7-source-tensions.md
@@ -2,7 +2,7 @@
name: hyperresearch-7-source-tensions
description: >
Step 7 of the hyperresearch V8 pipeline. Extracts explicit expert
- disagreements from the corpus into research/temp/source-tensions.json,
+ disagreements from the corpus into research/runs/<vault_tag>/temp/source-tensions.json,
including orphan tensions that didn't surface as loci. Reads full source
bodies of top 8-12 sources (not summaries) to find tensions that hide
in nuance. The Source Tensions section in step 10's draft is the single
@@ -23,9 +23,9 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/comparisons.md` — cross-locus tensions
-- `research/temp/contradiction-graph.json` (if step 3 ran)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/comparisons.md` — cross-locus tensions
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` (if step 3 ran)
- Survey vault: `$HPR note list --tag <vault_tag> --all -j` for the 15–20 highest-quality non-deprecated sources
---
@@ -42,7 +42,7 @@ Read these inputs:
- Industry claims that contradict independent research
- Historical consensus that recent evidence challenges
-3. **If `research/temp/contradiction-graph.json` exists**, read it. Any high-relevance fight cluster that was NOT promoted to a locus is a prime orphan-tension candidate. It was important enough for the contradiction graph but wasn't investigated in depth — these deserve standalone treatment in the draft.
+3. **If `research/runs/<vault_tag>/temp/contradiction-graph.json` exists**, read it. Any high-relevance fight cluster that was NOT promoted to a locus is a prime orphan-tension candidate. It was important enough for the contradiction graph but wasn't investigated in depth — these deserve standalone treatment in the draft.
4. **Select 3–7 source tensions.** Combine comparisons.md tensions with orphan tensions. Rank by:
- **Decision relevance:** does resolving this tension change the report's recommendation?
@@ -57,7 +57,7 @@ Read these inputs:
- State Side B's strongest case with evidence
- Commit to a reading: which side has the better evidence, or is there a synthesis? Name the load-bearing reason.
-6. **Write `research/temp/source-tensions.json`:**
+6. **Write `research/runs/<vault_tag>/temp/source-tensions.json`:**
```json
{
"tensions": [
@@ -87,7 +87,7 @@ This artifact feeds directly into step 10's mandatory Source Tensions section. E
## Exit criterion
-- `research/temp/source-tensions.json` exists with 3–7 tensions
+- `research/runs/<vault_tag>/temp/source-tensions.json` exists with 3–7 tensions
- Each tension has both sides with proponents, a committed resolution, and decision_relevance
---
diff --git a/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md b/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md
index f77f3b0..9959b09 100644
--- a/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md
+++ b/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md
@@ -20,11 +20,11 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/comparisons.md` — cross-locus tensions
-- `research/loci.json` — scored loci
-- `research/temp/source-tensions.json` — expert disagreements
-- `research/prompt-decomposition.json` — specifically the `time_periods` array
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/comparisons.md` — cross-locus tensions
+- `research/runs/<vault_tag>/loci.json` — scored loci
+- `research/runs/<vault_tag>/temp/source-tensions.json` — expert disagreements
+- `research/runs/<vault_tag>/prompt-decomposition.json` — specifically the `time_periods` array
---
@@ -39,7 +39,7 @@ For each `time_period` entry:
PYTHONIOENCODING=utf-8 {hpr_path} search "<period> <issuer>" --tag <vault_tag> --include-body -j
```
2. Open the candidate notes (`note show <id> -j`) and verify the document's actual reporting period — the filing must cover the SPECIFIC period named in the prompt, not an adjacent one. A Q1 2025 10-Q does NOT satisfy "Q3 2024" — different period, different tabular data.
-3. **If the period-pinned filing is missing, add it to `research/corpus-critic-gaps.json` as a `priority: critical` gap of type `period-pinned-primary` BEFORE spawning the corpus-critic subagent.** Schema:
+3. **If the period-pinned filing is missing, add it to `research/runs/<vault_tag>/corpus-critic-gaps.json` as a `priority: critical` gap of type `period-pinned-primary` BEFORE spawning the corpus-critic subagent.** Schema:
```json
{{
"type": "period-pinned-primary",
@@ -60,40 +60,40 @@ The targeted fetch wave in the next step will pull these filings BEFORE the corp
## Procedure
-1. **Spawn ONE `hyperresearch-corpus-critic` subagent** (Sonnet).
+1. **Spawn ONE `hyperresearch-corpus-critic` subagent**.
**Spawn template:**
```
subagent_type: hyperresearch-corpus-critic
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are step 8 of the hyperresearch V8 pipeline.
- Step 6 produced research/comparisons.md. After you return, the
+ Step 6 produced research/runs/<vault_tag>/comparisons.md. After you return, the
orchestrator runs a targeted fetch wave, then step 10 drafts.
YOUR INPUTS:
- corpus_tag: <vault_tag>
- - comparisons_path: research/comparisons.md
- - loci_path: research/loci.json
- - output_path: research/corpus-critic-gaps.json
+ - comparisons_path: research/runs/<vault_tag>/comparisons.md
+ - loci_path: research/runs/<vault_tag>/loci.json
+ - output_path: research/runs/<vault_tag>/corpus-critic-gaps.json
```
-2. **Read the gaps output** (`research/corpus-critic-gaps.json`). Each gap has a `priority` (critical / high) and a `type` (overturning / strengthening / independent-verification).
+2. **Read the gaps output** (`research/runs/<vault_tag>/corpus-critic-gaps.json`). Each gap has a `priority` (critical / high) and a `type` (overturning / strengthening / independent-verification).
-3. **Targeted fetch wave.** Spawn **2–4 fetcher subagents** (Sonnet) to search for and fetch the sources identified in the gaps.
+3. **Targeted fetch wave.** Spawn **2–4 fetcher subagents** to search for and fetch the sources identified in the gaps.
**Spawn template:**
```
subagent_type: hyperresearch-fetcher
prompt: |
RESEARCH QUERY (verbatim, gospel):
- > {{paste research/query-<vault_tag>.md body}}
+ > {{paste research/runs/<vault_tag>/query.md body}}
- QUERY FILE: research/query-<vault_tag>.md
+ QUERY FILE: research/runs/<vault_tag>/query.md
PIPELINE POSITION: You are a step 8 fetcher (corpus-critic gap-fill)
of the hyperresearch V8 pipeline. The corpus critic identified specific
@@ -108,11 +108,11 @@ The targeted fetch wave in the next step will pull these filings BEFORE the corp
```
4. **Assess results.**
- - **Overturning source found:** re-read the relevant committed position from the interim note. If the new source genuinely undercuts it, update `research/comparisons.md` to note the weakened position — the draft will handle it with appropriate calibration. Do NOT re-run the full depth investigation; adjust the position's confidence level.
+ - **Overturning source found:** re-read the relevant committed position from the interim note. If the new source genuinely undercuts it, update `research/runs/<vault_tag>/comparisons.md` to note the weakened position — the draft will handle it with appropriate calibration. Do NOT re-run the full depth investigation; adjust the position's confidence level.
- **Overturning source NOT found:** the committed position gains confidence. Note this in `comparisons.md` — "adversarial search for counter-evidence to [position] returned no substantive challenges."
- **Strengthening/verification source found:** note the additional support in `comparisons.md`. The draft can assert more confidently.
-5. **Log results** to `research/temp/corpus-critic-results.md`:
+5. **Log results** to `research/runs/<vault_tag>/temp/corpus-critic-results.md`:
- Each gap: what was searched, what was found (or not), how it affects the committed positions
- Updated confidence levels for any positions that changed
@@ -120,10 +120,10 @@ The targeted fetch wave in the next step will pull these filings BEFORE the corp
## Exit criterion
-- `research/corpus-critic-gaps.json` exists
+- `research/runs/<vault_tag>/corpus-critic-gaps.json` exists
- All critical gaps attempted (fetched or documented as unfindable)
-- `research/temp/corpus-critic-results.md` exists
-- `research/comparisons.md` updated with confidence/strengthening/overturning notes
+- `research/runs/<vault_tag>/temp/corpus-critic-results.md` exists
+- `research/runs/<vault_tag>/comparisons.md` updated with confidence/strengthening/overturning notes
---
diff --git a/src/hyperresearch/skills/hyperresearch-9-evidence-digest.md b/src/hyperresearch/skills/hyperresearch-9-evidence-digest.md
index 46f7444..9123896 100644
--- a/src/hyperresearch/skills/hyperresearch-9-evidence-digest.md
+++ b/src/hyperresearch/skills/hyperresearch-9-evidence-digest.md
@@ -3,7 +3,7 @@ name: hyperresearch-9-evidence-digest
description: >
Step 9 of the hyperresearch V8 pipeline. Assembles the top load-bearing
claims and verbatim quotes from the claims JSONs into
- research/temp/evidence-digest.md — a single high-fidelity evidence
+ research/runs/<vault_tag>/temp/evidence-digest.md — a single high-fidelity evidence
index the draft sub-orchestrators read as primary evidence (higher
fidelity than fetcher summaries). Invoked via Skill tool from the
entry skill (full tier).
@@ -20,27 +20,27 @@ description: >
## Recover state
Read these inputs:
-- `research/scaffold.md` — vault_tag
-- `research/prompt-decomposition.json` — atomic items, response_format, pipeline_tier
-- All `research/temp/claims-*.json` files
-- `research/temp/consensus-claims.json` (if step 3 ran)
-- `research/temp/contradiction-graph.json` (if step 3 ran)
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, response_format, pipeline_tier
+- All `research/runs/<vault_tag>/temp/claims-*.json` files
+- `research/runs/<vault_tag>/temp/consensus-claims.json` (if step 3 ran)
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` (if step 3 ran)
---
## Procedure
-1. **Read all claims files** from `research/temp/claims-*.json` for every non-deprecated note tagged with the vault tag. If no claim files exist (e.g., fetchers didn't produce them), skip this step.
+1. **Read all claims files** from `research/runs/<vault_tag>/temp/claims-*.json` for every non-deprecated note tagged with the vault tag. If no claim files exist (e.g., fetchers didn't produce them), skip this step.
-2. **Filter and rank.** Keep claims where `confidence` is `"high"` OR `evidence_type` is `"empirical"` or `"statistical"`. From the remainder, prefer claims with non-empty `numbers` arrays and non-empty `quoted_support`. Cap at **80–120 claims total** for `full` tier.
+2. **Filter and rank.** Keep claims where `confidence` is `"high"` OR `evidence_type` is `"empirical"` or `"statistical"`. From the remainder, prefer claims with non-empty `numbers` arrays and non-empty `quoted_support`. Cap at **<< p.claims_cap|dash >> claims total** for `full` tier.
3. **Group by atomic item.** Match each surviving claim to the atomic item it is most relevant to based on **topic overlap** — do not rely on exact field matching. A claim about "United Health Group regulatory exposure" serves the atomic item "UNH risk factors" even though no field matches exactly. Use the claim's `entities`, `stance_target`, `scope_conditions`, and `claim` text holistically to judge relevance. When uncertain, include the claim under the most relevant item rather than dropping it to Ungrouped. Claims that genuinely don't map to any atomic item go into an "Ungrouped" section at the end.
4. **Include consensus and contested claims.**
- - If `research/temp/consensus-claims.json` exists, include its claims marked as `[consensus]`.
- - If `research/temp/contradiction-graph.json` exists, include the top 3–5 contested claim pairs with both sides' `quoted_support` passages.
+ - If `research/runs/<vault_tag>/temp/consensus-claims.json` exists, include its claims marked as `[consensus]`.
+ - If `research/runs/<vault_tag>/temp/contradiction-graph.json` exists, include the top 3–5 contested claim pairs with both sides' `quoted_support` passages.
-5. **Write `research/temp/evidence-digest.md`.** Format: one H3 per atomic item, bullet list of claims. Each bullet includes:
+5. **Write `research/runs/<vault_tag>/temp/evidence-digest.md`.** Format: one H3 per atomic item, bullet list of claims. Each bullet includes:
- The `claim` text
- The `quoted_support` verbatim passage (block-quoted)
- The `source_note_id`
@@ -64,8 +64,8 @@ Read these inputs:
## Exit criterion
-- `research/temp/evidence-digest.md` exists
-- Contains at least 30 claims for `full` tier
+- `research/runs/<vault_tag>/temp/evidence-digest.md` exists
+- Contains at least << p.claims_min >> claims for `full` tier
- Grouped by atomic item with verbatim quoted_support and source_note_id
If fewer claims exist in total, include all of them.
diff --git a/src/hyperresearch/skills/hyperresearch.md b/src/hyperresearch/skills/hyperresearch.md
index 8e6cc9c..b963808 100644
--- a/src/hyperresearch/skills/hyperresearch.md
+++ b/src/hyperresearch/skills/hyperresearch.md
@@ -2,8 +2,9 @@
name: hyperresearch
description: >
Deep research via the HYPERRESEARCH V8 architecture — a tier-adaptive 16-step
- pipeline (light / full) that scales from a ~30-minute light-tier answer to
- a 1.5–2.5 hour adversarially-audited report. This entry skill is a ROUTER.
+ pipeline (light / full / dissertation) that scales from a ~30-minute light-tier
+ answer to an adversarially-audited report at the installed scale gear
+ (<< p.time_estimate >>). This entry skill is a ROUTER.
It does not contain step procedures — it tells you which Skill to invoke
for each step, in order. Each step's instructions live in its own skill
file (`hyperresearch-1-decompose` through `hyperresearch-16-readability-audit`)
@@ -12,11 +13,11 @@ description: >
# Hyperresearch V8 — multi-skill chain orchestrator
-You are the orchestrator (Opus). Your entire job in this conversation is:
+You are the orchestrator. Your entire job in this conversation is:
1. Read this file once at the start.
2. Bootstrap canonical inputs (research_query, vault_tag, scaffold).
3. Invoke each step skill in sequence via the `Skill` tool.
-4. Between steps, do nothing except mark todos and (optionally) think to `research/temp/orchestrator-notes.md`.
+4. Between steps, do nothing except mark todos and (optionally) think to `research/runs/<vault_tag>/temp/orchestrator-notes.md`.
You do NOT do the work of any step yourself. The step skills do. You just sequence them.
@@ -39,6 +40,7 @@ When you invoke a Skill, that skill's full procedure is loaded into your context
| # | Skill name | What it does | Tiers |
|---|---|---|---|
| 1 | `hyperresearch-1-decompose` | Canonical query → scaffold + decomposition + coverage matrix + tier classification | all |
+| 1.5 | `hyperresearch-1-5-chapter-partition` | Partition atomic items into 4–10 chapters; steps 2–10 then loop per chapter | dissertation |
| 2 | `hyperresearch-2-width-sweep` | Multi-perspective search plan + parallel fetcher waves | all |
| 3 | `hyperresearch-3-contradiction-graph` | Pair contradictions across the corpus into ranked fight clusters | full |
| 4 | `hyperresearch-4-loci-analysis` | 2 loci-analysts → scored loci.json with source budgets | full |
@@ -52,6 +54,7 @@ When you invoke a Skill, that skill's full procedure is loaded into your context
| 12 | `hyperresearch-12-critics` | 4 adversarial critics in parallel → findings JSONs | full |
| 13 | `hyperresearch-13-gap-fetch` | Fetch sources for critic-identified vault gaps | full |
| 14 | `hyperresearch-14-patcher` | Surgical Edit hunks applied to draft | full |
+| 14.5 | `hyperresearch-14-5-cite-check` | Verify citation-sentence bindings; second small patcher pass | full |
| 15 | `hyperresearch-15-polish` | Hygiene + filler pass (Edit-based subagent) | all |
| 16 | `hyperresearch-16-readability-audit` | Readability recommender writes JSON suggestions; orchestrator selectively applies via Edit | all |
@@ -59,15 +62,20 @@ When you invoke a Skill, that skill's full procedure is loaded into your context
## Tier routing
-Step 1 classifies the query into a `pipeline_tier` (`light` / `full`). The tier is written to `research/prompt-decomposition.json`. After step 1, **read that file** to learn the tier, then sequence steps according to:
+Step 1 classifies the query into a `pipeline_tier` (`light` / `full`). The tier is written to `research/runs/<vault_tag>/prompt-decomposition.json`. After step 1, **read that file** to learn the tier, then sequence steps according to:
-| Tier | Steps that run | Typical cost | Typical time |
-|------|---|---|---|
-| `light` | 1 → 2 → 10 (single draft) → 15 → 16 | ~$5–15 | ~30–40 min |
-| `full` | 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10 → 11 → 12 → 13 → 14 → 15 → 16 | ~$60–120 | ~1.5–2.5 hours |
+| Tier | Steps that run | Typical time |
+|------|---|---|
+| `light` | 1 → 2 → 10 (single draft) → 15 → 16 | << light.time_estimate >> |
+| `full` | 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10 → 11 → 12 → 13 → 14 → 14.5 → 15 → 16 | << p.time_estimate >> |
+| `dissertation` | 1 → 1.5 (chapter partition) → [2 → … → 10 per chapter] → 6g/11 (global) → 12 → 13 → 14 → 14.5 → 15 → 16 | << dissertation.time_estimate >> |
+
+`dissertation` is opt-in only — the user must explicitly request it (or the run was initialized with `--profile dissertation`). Step 1 never auto-classifies into it. On dissertation runs, steps 2–10 loop per chapter (see `hyperresearch-1-5-chapter-partition`), with up to << dissertation.chapter_concurrency >> chapters in flight; each chapter stays within the proven << dissertation.chapter_source_target|dash >>-source envelope while the run totals << dissertation.source_target|dash >> sources.
**RESPECT THE TIER GATE.** When step 1 classifies a query as `light`, do NOT run the skipped steps "just to be thorough." The tier classification is a product decision: simple queries should produce fast, right-sized answers. Trust the classification. If you're uncertain, tier up — but never silently upgrade every query to `full`.
+**Scale gear (tier ≠ gear).** The numbers rendered into the step skills — source targets, loci caps, depth budgets, word targets — come from the installed scale profile, the **gear** (currently `<< p.name >>`). The `full` tier row above already reflects it. Two gears ship: `full` (<< full.source_target|dash >> sources, << full.time_estimate >>) and `premier` (<< premier.source_target|dash >> sources, double depth budget, << premier.time_estimate >>). The user switches gears with `hyperresearch profile use <full|premier>` — that re-renders the installed skills, so it takes effect on the NEXT run, never mid-run. `light` and `dissertation` are tiers, not gears: light is auto-classified per query; dissertation is opt-in per run and loops each chapter inside the gear's envelope.
+
---
## Bootstrap (run BEFORE invoking step 1)
@@ -80,7 +88,7 @@ Before you invoke any step skill, do this:
If either command fails because the binary isn't on PATH, tell the user to run `pip install hyperresearch` first. If both files already exist, both commands no-op cheaply — safe to run unconditionally.
-0.5. **Archive any prior run's artifacts.** Run `hyperresearch archive-run --json`. If a previous `/hyperresearch` session left a scaffold, loci.json, comparisons.md, critic-findings, patch-log, polish-log, prompt-decomposition, or any `research/temp/*` scratch, this moves the whole set into `research/runs/archive-<prev-tag>-<UTC-timestamp>/` so the new run starts from a clean slate without losing the prior run's audit trail. Final reports (`research/notes/final_report_<tag>.md`) and canonical query files (`research/query-<tag>.md`) are already namespaced and stay in place. The command no-ops cheaply on a fresh vault — safe to run unconditionally. **Caveat:** this protects sequential runs only. Two `/hyperresearch` invocations that overlap in time still race on the new files they both write; if you need true parallel runs, namespace per-run artifacts under `research/runs/<vault_tag>/` instead.
+0.5. **Archive any pre-3.0 flat artifacts.** Run `hyperresearch archive-run --json`. On vaults that ran pre-3.0 pipelines, flat artifacts (research/scaffold.md, loci.json, etc.) may still sit at the research root; this moves them into `research/runs/archive-<prev-tag>-<UTC-timestamp>/`. On 3.0+ vaults every run already owns `research/runs/<vault_tag>/`, so concurrent and sequential runs never collide and this command cheaply no-ops.
1. **Resolve the canonical research query.** Order of precedence:
- If `research/prompt.txt` exists (legacy harness / wrapped run), read it. Its contents are the canonical research query. GOSPEL.
@@ -88,9 +96,16 @@ Before you invoke any step skill, do this:
- Extract wrapper requirements separately: required save path, citation format, terminal-section shape, wrapper contract. These are binding but NOT part of the query.
- If `research/wrapper_contract.json` exists, read it.
-2. **Mint a unique vault tag.** First produce a short topical slug from the canonical query — 3–5 lowercase hyphen-separated words, e.g. `efield-dft-sac`. Then call `hyperresearch vault-tag <slug> --json` and parse the `vault_tag` field from the response. The CLI appends a random 6-hex-char suffix that's verified unique against every prior run's `research/query-*.md` and `research/notes/final_report_*.md` in this vault. The result — e.g. `efield-dft-sac-a3f9b7` — is the canonical vault_tag for the rest of the pipeline. The suffix guarantees no overwrite of a prior run's final report or query file, even if the user re-runs the exact same query or two different queries slug-collide.
+2. **Mint a unique vault tag.** First produce a short topical slug from the canonical query — 3–5 lowercase hyphen-separated words, e.g. `efield-dft-sac`. Then call `hyperresearch vault-tag <slug> --json` and parse the `vault_tag` field from the response. The CLI appends a random 6-hex-char suffix that's verified unique against every prior run's workspaces, query files, and final reports in this vault. The result — e.g. `efield-dft-sac-a3f9b7` — is the canonical vault_tag for the rest of the pipeline.
+
+2.5. **Initialize the run workspace.** Run:
+ ```bash
+ hyperresearch run init <vault_tag> --profile <full|light|premier|dissertation> --json
+ ```
+ For a standard run, pass the installed gear (`<< p.name >>`) unless the user asked for something else.
+ Pass `--budget <usd>` when the user set a spend ceiling. This scaffolds `research/runs/<vault_tag>/` (with `temp/`) and writes `run.json` — the run manifest. **The manifest is your durable memory**: record every step transition with `hyperresearch run step <vault_tag> <N> --status running|done -j` as you go. The profile here defaults to matching the tier you expect; if step 1 classifies differently, the manifest's profile field is informational — the decomposition's tier rules.
-3. **Persist the query file.** Write the verbatim canonical query to `research/query-<vault_tag>.md`:
+3. **Persist the query file.** Write the verbatim canonical query to `research/runs/<vault_tag>/query.md`:
```markdown
---
vault_tag: <slug>
@@ -108,7 +123,7 @@ Before you invoke any step skill, do this:
- **compare**: proportionate per-entity depth + a committed recommendation
- **forecast**: predictive claims grounded in past + present, explicit time horizon
-5. **Write the scaffold.** Write `research/scaffold.md` (your private planning document — it MUST NOT appear anywhere in the final report). Include in scaffold:
+5. **Write the scaffold.** Write `research/runs/<vault_tag>/scaffold.md` (your private planning document — it MUST NOT appear anywhere in the final report). Include in scaffold:
- User Prompt (VERBATIM — gospel)
- Run config (vault_tag, query_file_path, modality, wrapper requirements)
- Modality classification rationale
@@ -124,13 +139,13 @@ Before you invoke any step skill, do this:
7. **Invoke step 1:** `Skill(skill: "hyperresearch-1-decompose")`.
-After step 1 returns, read `research/prompt-decomposition.json` to learn the tier, then continue invoking step skills per the tier routing table above. After each step's exit criterion is met, mark its todo complete and move to the next.
+After step 1 returns, read `research/runs/<vault_tag>/prompt-decomposition.json` to learn the tier, then continue invoking step skills per the tier routing table above. After each step's exit criterion is met, mark its todo complete and move to the next.
---
## Four canonical rules (ALWAYS in force)
-1. **NEVER EMIT BARE TEXT WHILE TASKS ARE RUNNING.** In non-interactive (`-p`) mode, a text-only response (no tool call) triggers `end_turn` — the process exits and the pipeline dies. Every response while subagent tasks are in flight MUST include a tool call. The best one is appending analytical thoughts to `research/temp/orchestrator-notes.md`. Vault count checks at most once per minute.
+1. **NEVER EMIT BARE TEXT WHILE TASKS ARE RUNNING.** In non-interactive (`-p`) mode, a text-only response (no tool call) triggers `end_turn` — the process exits and the pipeline dies. Every response while subagent tasks are in flight MUST include a tool call. The best one is appending analytical thoughts to `research/runs/<vault_tag>/temp/orchestrator-notes.md`. Vault count checks at most once per minute.
2. **PATCH, NEVER REGENERATE.** After step 11 produces the synthesized final report (or step 10 for light tier), the only modifications are surgical Edit hunks from step 14 (patcher) and step 15 (polish-auditor). Both subagents are tool-locked to `[Read, Edit]`. If a critic's finding would require rewriting a whole section, it escalates to you as a structural issue — not a rewrite. Keep hunks surgical.
@@ -140,13 +155,20 @@ After step 1 returns, read `research/prompt-decomposition.json` to learn the tie
---
+## Browser-lane escalations (all tiers)
+
+Blocked fetches (login walls, bot walls, captchas) are queued, not lost: `$HPR escalation list --status queued --tag <vault_tag> -j`. Step 2.8 drains the queue via ONE `hyperresearch-browser-fetcher` subagent driving the user's real Chrome browser. Two standing rules:
+
+1. **CAPTCHAs / logins / 2FA are ALWAYS the human's.** The browser-fetcher marks them `needs_human`; you consolidate ALL of them into ONE message to the user at a natural pause point (never one interruption per URL). In non-interactive runs, `$HPR run block <vault_tag> --on human-challenges` and continue with everything else.
+2. **One browser-fetcher at a time.** It's the user's actual browser — parallel instances are chaos. Check the queue again after step 13 (gap-fetch) if new fetches got blocked.
+
## Subagent spawn contract (applies to every Task call)
When a step skill instructs you to spawn a subagent, the prompt you pass MUST include three pieces near the top:
-1. **`research_query` — verbatim, block-quoted** from `research/query-<vault_tag>.md`. Do not paraphrase, do not summarize.
+1. **`research_query` — verbatim, block-quoted** from `research/runs/<vault_tag>/query.md`. Do not paraphrase, do not summarize.
-2. **Pipeline position statement.** One sentence naming what step the subagent runs in, what came before, what comes after. Example: *"You are step 5 (depth investigator) of the hyperresearch V8 pipeline. Step 4's loci analysts produced `research/loci.json`; after you return, step 6 will reconcile your committed position against the other investigators'."*
+2. **Pipeline position statement.** One sentence naming what step the subagent runs in, what came before, what comes after. Example: *"You are step 5 (depth investigator) of the hyperresearch V8 pipeline. Step 4's loci analysts produced `research/runs/<vault_tag>/loci.json`; after you return, step 6 will reconcile your committed position against the other investigators'."*
3. **The subagent's specific inputs** (vault_tag, output_path, locus, etc.). Each step skill's spawn template documents the required fields.
@@ -158,24 +180,25 @@ Skipping any of these in a Task prompt is a process violation.
Context compaction may eat parts of this conversation. If you're unsure what step you're on:
+0. **Read the run manifest FIRST.** `hyperresearch run resume <vault_tag> --json` (or with no tag for the newest run) returns the exact next step and the Skill invocation to continue with. This is the primary recovery path — the manifest records every step transition you logged via `hyperresearch run step`. The artifact scan below is the fallback for manifests that are missing or were not kept up to date.
1. **Check the TodoWrite list.** It carries integer step numbers and survives compaction.
-2. **Check disk artifacts.** Each step writes a canonical artifact:
- - Step 1: `research/scaffold.md`, `research/prompt-decomposition.json`, `research/temp/coverage-matrix.md`
+2. **Check disk artifacts (fallback).** Each step writes a canonical artifact:
+ - Step 1: `research/runs/<vault_tag>/scaffold.md`, `research/runs/<vault_tag>/prompt-decomposition.json`, `research/runs/<vault_tag>/temp/coverage-matrix.md`
- Step 2: vault notes tagged with vault_tag (`$HPR note list --tag <vault_tag> --all -j`)
- - Step 3: `research/temp/contradiction-graph.json`, `research/temp/consensus-claims.json`
- - Step 4: `research/loci.json`
+ - Step 3: `research/runs/<vault_tag>/temp/contradiction-graph.json`, `research/runs/<vault_tag>/temp/consensus-claims.json`
+ - Step 4: `research/runs/<vault_tag>/loci.json`
- Step 5: vault notes with `type: interim` (`$HPR note list --tag <vault_tag> --type interim --all -j`)
- - Step 6: `research/comparisons.md`
- - Step 7: `research/temp/source-tensions.json`
- - Step 8: `research/corpus-critic-gaps.json`, `research/temp/corpus-critic-results.md`
- - Step 9: `research/temp/evidence-digest.md`
- - Step 10: `research/temp/draft-{a,b,c}.md` (or `research/notes/final_report_<vault_tag>.md` for light tier single-pass)
- - Step 11: `research/temp/synthesis-plan.md`, `research/temp/synthesis-outline.md`, `research/temp/synthesis-pass1.md`, `research/notes/final_report_<vault_tag>.md`
- - Step 12: `research/critic-findings-{dialectic,depth,width,instruction}.json`
- - Step 13: `research/temp/post-critic-fetch-log.md`
- - Step 14: `research/patch-log.json` (and edited final_report.md)
- - Step 15: `research/polish-log.json` (and edited final_report.md)
- - Step 16: `research/readability-recommendations.json`, `research/readability-decisions.json` (and edited final_report.md)
+ - Step 6: `research/runs/<vault_tag>/comparisons.md`
+ - Step 7: `research/runs/<vault_tag>/temp/source-tensions.json`
+ - Step 8: `research/runs/<vault_tag>/corpus-critic-gaps.json`, `research/runs/<vault_tag>/temp/corpus-critic-results.md`
+ - Step 9: `research/runs/<vault_tag>/temp/evidence-digest.md`
+ - Step 10: `research/runs/<vault_tag>/temp/draft-{a,b,c}.md` (or `research/notes/final_report_<vault_tag>.md` for light tier single-pass)
+ - Step 11: `research/runs/<vault_tag>/temp/synthesis-plan.md`, `research/runs/<vault_tag>/temp/synthesis-outline.md`, `research/runs/<vault_tag>/temp/synthesis-pass1.md`, `research/notes/final_report_<vault_tag>.md`
+ - Step 12: `research/runs/<vault_tag>/critic-findings-{dialectic,depth,width,instruction}.json`
+ - Step 13: `research/runs/<vault_tag>/temp/post-critic-fetch-log.md`
+ - Step 14: `research/runs/<vault_tag>/patch-log.json` (and edited final_report.md)
+ - Step 15: `research/runs/<vault_tag>/polish-log.json` (and edited final_report.md)
+ - Step 16: `research/runs/<vault_tag>/readability-recommendations.json`, `research/runs/<vault_tag>/readability-decisions.json` (and edited final_report.md)
3. **Find the highest-numbered step whose artifact exists.** Resume from the next step.
4. **Re-invoke this entry skill** if you've lost track entirely: `Skill(skill: "hyperresearch")`. It loads fresh.
@@ -188,27 +211,32 @@ If you're ever uncertain what to do next, the answer is: re-read this file and f
Once step 16 returns, run the integrity check:
```bash
-for f in research/critic-findings-dialectic.json \
- research/critic-findings-depth.json \
- research/critic-findings-width.json \
- research/critic-findings-instruction.json \
- research/patch-log.json \
- research/polish-log.json; do
+for f in research/runs/<vault_tag>/critic-findings-dialectic.json \
+ research/runs/<vault_tag>/critic-findings-depth.json \
+ research/runs/<vault_tag>/critic-findings-width.json \
+ research/runs/<vault_tag>/critic-findings-instruction.json \
+ research/runs/<vault_tag>/patch-log.json \
+ research/runs/<vault_tag>/polish-log.json; do
test -f "$f" || echo "MISSING: $f"
done
```
(Light tier skips critics + patcher entirely — the critic-findings and patch-log files won't exist. That's expected; only `polish-log.json` is required for light.)
-Then run lint:
+Then run the verification battery + lint:
```bash
+$HPR run verify <vault_tag> --json # structural: headings, length, citation density, artifacts, cite-check resolution
+$HPR sources retractions --tag <vault_tag> --json # ship-time retraction re-check (bypasses cache)
$HPR lint --rule wrapper-report --json
$HPR lint --rule locus-coverage --json
$HPR lint --rule scaffold-prompt --json
$HPR lint --rule patch-surgery --json
+$HPR lint --rule quote-integrity --json # every quoted span must exist verbatim in a vault note
+$HPR lint --rule retracted-citations --json # retracted sources cited unmarked block the ship
+$HPR lint --rule numeric-consistency --json # warnings: numbers untraceable to evidence — verify or remove
```
-If any rule returns `error` severity issues, address them before declaring complete. Then ship: the final report lives at `research/notes/final_report_<vault_tag>.md`.
+If `run verify` fails or any rule returns `error` severity issues, address them before declaring complete. Then ship: the final report lives at `research/notes/final_report_<vault_tag>.md`.
---
@@ -218,7 +246,7 @@ If any rule returns `error` severity issues, address them before declaring compl
2. **One final report.** Step 11's synthesizer writes the final report ONCE. No re-synthesizing. (Light tier: step 10 writes it once.)
3. **At least one dialectical locus.** Step 4 must surface ≥1 dialectical locus unless skip is justified.
4. **Every interim note commits to a position.** Step 5 investigators end with `## Committed position`.
-5. **`research/comparisons.md` exists when loci count ≥ 1.** Step 6 is mandatory whenever step 4 produced any loci.
+5. **`research/runs/<vault_tag>/comparisons.md` exists when loci count ≥ 1.** Step 6 is mandatory whenever step 4 produced any loci.
6. **Steps are sequential at the outermost level, parallel within.** You cannot start step N+1 before step N completes. Within a step, parallelism is mandatory when there are multiple subagents.
7. **Canonical research query is gospel everywhere.** Every subagent gets the verbatim query.
8. **Hygiene rules apply to the final report only.** Workspace artifacts (scaffold, loci JSONs, interim notes, comparisons.md, patch log) can look however they need to look.
diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.py
index cf95a4f..4e944ee 100644
--- a/src/hyperresearch/web/base.py
+++ b/src/hyperresearch/web/base.py
@@ -6,9 +6,15 @@ from dataclasses import dataclass, field
from datetime import UTC, datetime
from typing import Protocol, runtime_checkable
+from hyperresearch.core.config import FetchSettings, JunkGates
+
# Whitespace that is legitimate in extracted text.
_TEXT_WHITESPACE = "\t\n\r\f\v"
+# Default thresholds — used when no vault config is in play (e.g. direct
+# provider usage in tests/scripts). Matches VaultConfig defaults.
+DEFAULT_GATES = JunkGates()
+
def is_binary_garbage_char(c: str) -> bool:
"""True if `c` indicates binary or mis-decoded content rather than real text.
@@ -33,6 +39,17 @@ def binary_garbage_ratio(text: str) -> float:
return sum(1 for c in text if is_binary_garbage_char(c)) / len(text)
+def is_binary_garbage(sample: str, gates: JunkGates | None = None) -> bool:
+ """Shared threshold check for binary/mis-decoded content.
+
+ Single implementation for both fetch gates (WebResult.looks_like_junk and
+ the crawl4ai post-fetch PDF re-check) so the threshold can't drift apart
+ between them again.
+ """
+ gates = gates or DEFAULT_GATES
+ return binary_garbage_ratio(sample) > gates.binary_garbage_ratio
+
+
@dataclass
class WebResult:
"""A single web fetch or search result."""
@@ -55,21 +72,23 @@ class WebResult:
return urlparse(self.url).netloc
- def looks_like_login_wall(self, original_url: str) -> bool:
+ def looks_like_login_wall(self, original_url: str, gates: JunkGates | None = None) -> bool:
"""Check if the result appears to be a login/signup redirect rather than real content."""
+ gates = gates or DEFAULT_GATES
login_signals = (
"sign in", "sign up", "log in", "login", "create account",
"auth", "register", "sso", "verify your identity",
+ *gates.extra_login_signals,
)
title_lower = (self.title or "").lower()
- content_lower = (self.content or "")[:500].lower()
+ content_lower = (self.content or "")[: gates.login_sample_chars].lower()
# Title contains login language
title_match = any(s in title_lower for s in login_signals)
# Content is mostly login form (very short with login keywords)
content_match = (
- len(self.content or "") < 1000
+ len(self.content or "") < gates.login_wall_max_chars
and any(s in content_lower for s in login_signals)
)
@@ -82,17 +101,18 @@ class WebResult:
return title_match or content_match or url_redirected
- def looks_like_junk(self) -> str | None:
+ def looks_like_junk(self, gates: JunkGates | None = None) -> str | None:
"""Check if the result is junk that shouldn't be saved.
Returns a reason string if junk, None if OK.
"""
+ gates = gates or DEFAULT_GATES
content = self.content or ""
title_lower = (self.title or "").lower()
- content_lower = content[:2000].lower()
+ content_lower = content[: gates.sample_window].lower()
# Empty or near-empty content
- if len(content.strip()) < 300:
+ if len(content.strip()) < gates.min_content_chars:
return "Empty or near-empty content"
# Cloudflare / bot detection pages
@@ -104,7 +124,7 @@ class WebResult:
"enable javascript and cookies", "browser check",
"ddos protection", "attention required",
)
- if any(s in title_lower or s in content_lower for s in cf_signals):
+ if any(s in title_lower or s in content_lower for s in cf_signals + tuple(gates.extra_junk_signals)):
return f"Bot detection page: {self.title}"
# Error pages
@@ -124,15 +144,15 @@ class WebResult:
# Binary garbage from PDFs that weren't properly extracted
pdf_binary_signals = ("endstream", "endobj", "/FlateDecode", "%PDF-")
- sample = content[:2000]
+ sample = content[: gates.sample_window]
if any(m in sample for m in pdf_binary_signals):
return "Binary PDF garbage in content"
- if binary_garbage_ratio(sample) > 0.05:
+ if is_binary_garbage(sample, gates):
return "High ratio of binary/non-printable content"
# Cookie consent / boilerplate pages (short with mostly nav/cookie text)
- if len(content.strip()) < 1500:
+ if len(content.strip()) < gates.cookie_wall_max_chars:
cookie_signals = (
"we use cookies", "cookie policy", "accept cookies", "cookie consent",
"there appears to be a technical issue", "please enable javascript",
@@ -167,6 +187,8 @@ def get_provider(
profile: str | None = None,
magic: bool = False,
headless: bool = True,
+ settings: FetchSettings | None = None,
+ gates: JunkGates | None = None,
) -> WebProvider:
"""Load a web provider by name. Falls back to builtin if none specified."""
if name is None or name == "builtin":
@@ -182,6 +204,8 @@ def get_provider(
profile=profile or None,
magic=magic,
headless=headless,
+ settings=settings,
+ gates=gates,
)
except ImportError:
raise ImportError("crawl4ai provider requires: pip install hyperresearch[crawl4ai]")
diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.py
index 7ba9027..f4bbbb6 100644
--- a/src/hyperresearch/web/crawl4ai_provider.py
+++ b/src/hyperresearch/web/crawl4ai_provider.py
@@ -20,7 +20,8 @@ from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
from crawl4ai.browser_adapter import UndetectedAdapter
from crawl4ai.content_filter_strategy import PruningContentFilter
-from hyperresearch.web.base import WebResult, binary_garbage_ratio
+from hyperresearch.core.config import FetchSettings, JunkGates
+from hyperresearch.web.base import WebResult, is_binary_garbage
# Fix Windows encoding before crawl4ai's managed browser tries to log Unicode
if sys.platform == "win32":
@@ -50,11 +51,12 @@ def _is_pdf_url(url: str) -> bool:
return "arxiv.org" in parsed.netloc and ("/pdf/" in path or "/abs/" in path)
-def _looks_like_binary(text: str) -> bool:
+def _looks_like_binary(text: str, gates: JunkGates | None = None) -> bool:
"""Check if extracted 'content' is actually binary garbage from a PDF."""
if not text:
return False
- sample = text[:2000]
+ gates = gates or JunkGates()
+ sample = text[: gates.sample_window]
# PDF internal structure markers — dead giveaway
pdf_markers = ("endstream", "endobj", "/Filter", "/FlateDecode", "stream\nx", "%PDF-")
if any(m in sample for m in pdf_markers):
@@ -62,7 +64,31 @@ def _looks_like_binary(text: str) -> bool:
# Shared with WebResult.looks_like_junk() \u2014 keep one implementation, since these
# two gates drifting apart is what let `ord(c) > 127` survive in base.py and
# silently discard every non-English page.
- return binary_garbage_ratio(sample) > 0.05
+ return is_binary_garbage(sample, gates)
+
+
+def _smart_wait_js(settings: FetchSettings) -> str:
+ """DOM-stability polling loop shared by the headless and visible paths.
+
+ Waits `wait_initial_ms`, then polls every `poll_interval_ms` until body text
+ length is unchanged for `stable_checks` consecutive polls, giving up after
+ `max_checks` polls.
+ """
+ return (
+ "() => new Promise(r => {"
+ " setTimeout(() => {"
+ " let last = document.body.innerText.length;"
+ " let stable = 0;"
+ " let checks = 0;"
+ " const interval = setInterval(() => {"
+ " const now = document.body.innerText.length;"
+ " if (now === last) { stable++; } else { stable = 0; }"
+ f" if (stable >= {settings.stable_checks} || checks > {settings.max_checks}) {{ clearInterval(interval); r(true); }}"
+ " last = now; checks++;"
+ f" }}, {settings.poll_interval_ms});"
+ f" }}, {settings.wait_initial_ms});"
+ "})"
+ )
_PYMUPDF_MISSING_LOGGED = False
@@ -96,7 +122,7 @@ def _import_pymupdf():
return None
-def _fetch_pdf(url: str) -> WebResult | None:
+def _fetch_pdf(url: str, settings: FetchSettings | None = None) -> WebResult | None:
"""Download a PDF and extract text using pymupdf. Returns None if extraction fails.
Every failure path logs its reason. A silent None here is indistinguishable
@@ -107,6 +133,8 @@ def _fetch_pdf(url: str) -> WebResult | None:
if pymupdf is None:
return None
+ settings = settings or FetchSettings()
+
import httpx
try:
@@ -116,7 +144,8 @@ def _fetch_pdf(url: str) -> WebResult | None:
if not url.endswith(".pdf"):
url += ".pdf"
- resp = httpx.get(url, follow_redirects=True, timeout=30, verify=False, headers={
+ resp = httpx.get(url, follow_redirects=True, timeout=settings.pdf_timeout_s,
+ verify=settings.pdf_verify_tls, headers={
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
@@ -140,7 +169,7 @@ def _fetch_pdf(url: str) -> WebResult | None:
)
return None
- if len(pdf_bytes) < 100:
+ if len(pdf_bytes) < settings.min_pdf_bytes:
_pdf_log().warning("PDF fetch for %s returned only %d bytes", url, len(pdf_bytes))
return None
@@ -198,6 +227,8 @@ class Crawl4AIProvider:
profile: str | None = None,
cookies: list[dict] | None = None,
magic: bool = False,
+ settings: FetchSettings | None = None,
+ gates: JunkGates | None = None,
):
# Resolve profile name to path (crawl4ai stores profiles in ~/.crawl4ai/profiles/)
data_dir = user_data_dir
@@ -207,6 +238,8 @@ class Crawl4AIProvider:
self._data_dir = data_dir
self._headless = headless
self._cookies = cookies
+ self._settings = settings or FetchSettings()
+ self._gates = gates or JunkGates()
browser_kwargs: dict = {"headless": headless}
if data_dir:
@@ -217,22 +250,8 @@ class Crawl4AIProvider:
self._browser_config = BrowserConfig(**browser_kwargs)
- # Smart wait: 2s initial + poll until content stabilizes (10s ceiling)
- self._wait_js = (
- "js:() => new Promise(r => {"
- " setTimeout(() => {"
- " let last = document.body.innerText.length;"
- " let stable = 0;"
- " let checks = 0;"
- " const interval = setInterval(() => {"
- " const now = document.body.innerText.length;"
- " if (now === last) { stable++; } else { stable = 0; }"
- " if (stable >= 2 || checks > 16) { clearInterval(interval); r(true); }"
- " last = now; checks++;"
- " }, 500);"
- " }, 2000);"
- "})"
- )
+ # Smart wait: initial delay + poll until content stabilizes
+ self._wait_js = "js:" + _smart_wait_js(self._settings)
# Use PruningContentFilter to populate fit_markdown (strips nav/footer chrome)
self._md_generator = DefaultMarkdownGenerator(
content_filter=PruningContentFilter(),
@@ -241,7 +260,7 @@ class Crawl4AIProvider:
magic=magic,
simulate_user=True,
screenshot=True,
- page_timeout=30000,
+ page_timeout=self._settings.page_timeout_ms,
wait_for=self._wait_js,
markdown_generator=self._md_generator,
)
@@ -266,7 +285,7 @@ class Crawl4AIProvider:
def fetch(self, url: str) -> WebResult:
# PDF detection: fetch directly with httpx, extract text with pymupdf
if _is_pdf_url(url):
- result = _fetch_pdf(url)
+ result = _fetch_pdf(url, self._settings)
if result is not None:
return result
# Fallback to browser if PDF fetch failed (might be a landing page, not actual PDF)
@@ -279,8 +298,8 @@ class Crawl4AIProvider:
# Post-fetch PDF detection: if the browser got binary garbage (PDF served
# inline without proper content-type handling), re-fetch as a direct PDF download.
- if result.content and _looks_like_binary(result.content):
- pdf_result = _fetch_pdf(url)
+ if result.content and _looks_like_binary(result.content, self._gates):
+ pdf_result = _fetch_pdf(url, self._settings)
if pdf_result is not None:
return pdf_result
@@ -302,22 +321,10 @@ class Crawl4AIProvider:
ignore_https_errors=True,
)
page = context.pages[0] if context.pages else await context.new_page()
- await page.goto(url, wait_until="domcontentloaded", timeout=30000)
-
- # Smart wait — same logic as crawl4ai config
- await page.evaluate("""() => new Promise(r => {
- setTimeout(() => {
- let last = document.body.innerText.length;
- let stable = 0;
- let checks = 0;
- const interval = setInterval(() => {
- const now = document.body.innerText.length;
- if (now === last) { stable++; } else { stable = 0; }
- if (stable >= 2 || checks > 16) { clearInterval(interval); r(true); }
- last = now; checks++;
- }, 500);
- }, 2000);
- })""")
+ await page.goto(url, wait_until="domcontentloaded", timeout=self._settings.page_timeout_ms)
+
+ # Smart wait — same logic (and same builder) as the headless config
+ await page.evaluate(_smart_wait_js(self._settings))
html = await page.content()
title = await page.title()
@@ -413,7 +420,7 @@ class Crawl4AIProvider:
# Fetch PDFs directly (no browser needed)
for url in pdf_urls:
- pdf_result = _fetch_pdf(url)
+ pdf_result = _fetch_pdf(url, self._settings)
if pdf_result is not None:
web_results.append(pdf_result)
@@ -436,8 +443,8 @@ class Crawl4AIProvider:
content = ""
# Post-fetch binary check — browser may have fetched a PDF inline
- if content and _looks_like_binary(content):
- pdf_result = _fetch_pdf(url)
+ if content and _looks_like_binary(content, self._gates):
+ pdf_result = _fetch_pdf(url, self._settings)
if pdf_result is not None:
web_results.append(pdf_result)
continue
diff --git a/tests/fixtures/golden_prompts/agents/browser_fetcher_agent.md b/tests/fixtures/golden_prompts/agents/browser_fetcher_agent.md
new file mode 100644
index 0000000..7f6c57c
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/browser_fetcher_agent.md
@@ -0,0 +1,117 @@
+---
+name: hyperresearch-browser-fetcher
+description: >
+ Escalation-lane fetcher that drives the user's REAL Chrome browser (via
+ Claude-in-Chrome) for sources headless crawling cannot reach: login-gated
+ pages, bot-walled sites, interactive/infinite-scroll pages, viewer-rendered
+ PDFs, and Google Scholar searches. Drains the `hyperresearch escalation`
+ queue serially — one item, one tab, at a time. Spawn EXACTLY ONE at a time;
+ parallel instances fighting over one browser is chaos. HARD BOUNDARY:
+ never attempts to solve CAPTCHAs, 2FA, or logins — those are marked
+ needs_human and consolidated for the user.
+model: sonnet
+tools: Bash, Read, Write, ToolSearch
+color: orange
+---
+
+You are the hyperresearch browser-lane fetcher. You drain the escalation
+queue — URLs that headless crawling could not reach — by driving the user's
+real Chrome browser through the Claude-in-Chrome tools.
+
+## Hard scope boundary (read first)
+
+You NEVER attempt to solve, bypass, or automate CAPTCHAs, 2FA prompts, or
+login forms. The moment a page asks for something only the account owner
+should do, you run:
+
+```bash
+PYTHONIOENCODING=utf-8 {hpr_path} escalation human <id> --detail "<one line: site + what the human must do>" -j
+```
+
+and move to the next item. The orchestrator consolidates all needs_human
+items into ONE prompt for the user at a natural pause point. You read what
+the user's own access can see; you do not evade. Never log out, never change
+account state, never navigate outside the claimed item's domain except for
+redirects.
+
+## Setup (once per session)
+
+Load the Chrome tools in ONE batched ToolSearch call:
+
+ToolSearch query: "select:mcp__claude-in-chrome__tabs_context_mcp,mcp__claude-in-chrome__tabs_create_mcp,mcp__claude-in-chrome__navigate,mcp__claude-in-chrome__get_page_text,mcp__claude-in-chrome__read_page,mcp__claude-in-chrome__find,mcp__claude-in-chrome__computer"
+
+Then call tabs_context_mcp once. ALWAYS open a NEW tab for your work
+(tabs_create_mcp) — never reuse the user's existing tabs. If the extension
+is unavailable or tools error repeatedly, mark the current item back to the
+queue state via `escalation human <id> --detail "Chrome extension unavailable"`
+and stop — report the situation in your final message.
+
+## The drain loop
+
+Repeat up to your assigned batch size (default 10 items):
+
+1. **Claim:**
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation claim --by browser-fetcher --tag <vault_tag> -j
+ ```
+ `queue_empty: true` → you're done; write your summary and return.
+
+2. **Navigate** the claimed URL in your tab. Wait for content. Human-paced:
+ one page at a time, no rapid-fire requests.
+
+3. **Extract.** Prefer `get_page_text` (whole-page text) over DOM surgery.
+ Playbook for hard pages:
+ - **Infinite scroll / "load more":** scroll or click until content
+ stabilizes, hard cap ~10 interactions, then extract once.
+ - **In-page navigation (SPAs, tabs, accordions):** expand sections that
+ contain content relevant to the research query; skip nav chrome.
+ - **PDF in a viewer:** extract the viewer's text layer via get_page_text;
+ if empty, note "PDF viewer without text layer" and mark needs_human
+ with the download suggestion.
+ - **Charts/figures with thin text:** screenshot via the computer tool and
+ transcribe the load-bearing figures/axis values into a
+ `## Extracted figures` section of your writeup.
+ - **CAPTCHA / login / 2FA appears:** STOP. `escalation human` (see
+ boundary above). Next item.
+
+4. **Ingest.** Write the extracted content to a scratch file, then:
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation ingest <id> --title "<page title>" --body-file <scratch-file> --tag <topic-tag> -j
+ ```
+ One command — it writes the vault note (with `fetch_provider: chrome`
+ provenance), records the source row, syncs, and resolves the item. Do
+ NOT use `note new` or `fetch` for escalation items.
+
+5. **Genuinely unreachable** (dead page, geo-block, content gone):
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation abandon <id> --detail "<why>" -j
+ ```
+ Abandoning is fine — the floor is where we started (source lost).
+
+## Scholar items (`reason: scholar_search`)
+
+The item's `url` field is a SEARCH QUERY, not a URL. Google Scholar has no
+API and blocks headless crawlers; you are the lane.
+
+1. Open https://scholar.google.com in your tab, search the query.
+2. Extract the top ~10 results: title, authors, year, venue, citation
+ count, link, and the cited-by link.
+3. Write them as a markdown list and ingest with
+ `--title "Scholar: <query>" --tag scholar-results`.
+4. For the 2-3 highest-citation results directly relevant to the research
+ query, queue their links for a future drain:
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} escalation add "<paper url>" --reason interactive_needed --tag <vault_tag> --suggested-by <scholar-note-id> --detail "high-citation Scholar hit" -j
+ ```
+ Cap: one query at a time, small N, human-paced. This is a courtesy lane,
+ not a scraper.
+
+## Report back
+
+Your final message is data for the orchestrator, not prose for a human:
+- items drained: N fetched / N needs_human / N abandoned
+- note ids created
+- needs_human items with their one-line details (the orchestrator will
+ consolidate these for the user)
+- anything that suggests the whole domain is unreachable (so the
+ orchestrator stops queueing it)
diff --git a/tests/fixtures/golden_prompts/agents/cite_checker_agent.md b/tests/fixtures/golden_prompts/agents/cite_checker_agent.md
new file mode 100644
index 0000000..15dcd11
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/cite_checker_agent.md
@@ -0,0 +1,82 @@
+---
+name: hyperresearch-cite-checker
+description: >
+ Step 14.5 of the hyperresearch V8 pipeline. Verifies that each sampled
+ citation actually supports its sentence by reading the cited note's body.
+ Receives batches of (sentence, note_id) pairs the mechanical triage could
+ not auto-pass; returns per-pair verdicts (supported / partially-supported /
+ unsupported / wrong-source) as findings JSON the patcher consumes.
+ This is reading comprehension at volume, not prose judgment.
+ Never edits the report.
+model: sonnet
+tools: Bash, Read, Write
+color: red
+---
+
+You are the hyperresearch cite-checker. Cited sources make a report
+trustworthy ONLY if they actually say what the sentences citing them claim.
+You verify that binding, pair by pair.
+
+## Pipeline position
+
+You are step 14.5 of the hyperresearch V8 pipeline. The report has been
+synthesized (11), critiqued (12), and patched (14). Mechanical triage
+already auto-passed pairs whose numbers/wording appear in the cited note's
+extracted claims; you get the remainder. Your findings go to a second,
+small patcher pass — you do NOT edit the report yourself.
+
+## Inputs (from your spawn prompt)
+
+- pairs_file: research/runs/<vault_tag>/cite-check-pairs.json (read the
+ `sampled_for_llm` array; your spawn prompt names which index range is yours)
+- findings_path: research/runs/<vault_tag>/cite-check-findings.json
+- vault_tag
+
+## Procedure
+
+For each assigned pair:
+
+1. Read the cited note's body:
+ ```bash
+ PYTHONIOENCODING=utf-8 {hpr_path} note show <note_id> -j
+ ```
+ Batch-read up to 5 ids per call when consecutive pairs cite different notes.
+
+2. Judge: does the note's content support the sentence AS WRITTEN?
+ - **supported** — the note states or directly entails the sentence's claim,
+ including its numbers.
+ - **partially-supported** — the note supports the gist but not the
+ specifics (wrong magnitude, missing qualifier, broader claim than the
+ source makes).
+ - **unsupported** — nothing in the note backs the sentence.
+ - **wrong-source** — the note doesn't back it, but another vault note
+ does. Find it: `PYTHONIOENCODING=utf-8 {hpr_path} claims search "<key phrase>" -j`
+ and name the correct note_id in the finding.
+
+ Judge the SOURCE-SENTENCE binding only. Whether the claim is TRUE is not
+ your question; whether THIS source says it is.
+
+3. Only non-`supported` verdicts become findings. Write ALL your findings in
+ ONE JSON array to your assigned findings path:
+ ```json
+ [
+ {
+ "verdict": "unsupported | partially-supported | wrong-source",
+ "severity": "critical | major",
+ "sentence": "<verbatim from the pairs file>",
+ "cited_note_id": "<id>",
+ "correct_note_id": "<id or null>",
+ "evidence": "<one sentence: what the note actually says / lacks>",
+ "suggested_fix": "<swap citation | soften claim to what the source supports | delete sentence>"
+ }
+ ]
+ ```
+ Severity: `critical` for unsupported number-bearing claims and
+ wrong-source; `major` otherwise. Write `[]` if every pair checked out.
+
+## Rules
+
+- Verdicts default SKEPTICAL: when you cannot find support in the note,
+ the verdict is unsupported — never "probably fine".
+- Do not re-litigate pairs the triage auto-passed.
+- Your final message: counts per verdict + the findings path. Data, not prose.
diff --git a/tests/fixtures/golden_prompts/agents/depth_critic_agent.md b/tests/fixtures/golden_prompts/agents/depth_critic_agent.md
new file mode 100644
index 0000000..9adfbb2
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/depth_critic_agent.md
@@ -0,0 +1,99 @@
+---
+name: hyperresearch-depth-critic
+description: >
+ Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
+ draft and returns a findings list of places where the draft skates
+ over technical substance that the vault's interim notes could
+ actually support. Spawn ONCE per draft, parallel with
+ dialectic-critic and width-critic.
+model: opus
+tools: Bash, Read, Write
+color: red
+---
+
+You are the depth critic. Your only job: find places where the draft
+hand-waves through technical substance that the vault's depth-investigator
+interim notes actually cover in detail. The user spent budget on deep
+investigation; the draft is supposed to reflect that investment.
+
+## Pipeline position
+
+You are **Layer 5** of the 7-phase hyperresearch pipeline. Running in parallel:
+dialectic-critic, width-critic, instruction-critic. You collectively hand
+findings to the patcher (Layer 6, tool-locked `[Read, Edit]`). You do NOT
+patch the draft yourself — you only write findings.
+
+Your specific angle: the vault already contains depth-investigator interim
+notes (Layer 3 output) with rich evidence — quotes, numbers, committed
+positions. Your job is to verify the draft actually USES that evidence
+rather than gesturing at it from a distance.
+
+## Inputs (from the parent agent)
+
+- **research_query**: verbatim user question. GOSPEL. Shallow coverage is
+ only a problem when it matters for answering the research_query; a
+ draft that glosses an irrelevant detail is fine.
+- **query_file_path**: path to the persisted query file (e.g.,
+ `research/runs/<vault_tag>/query.md`). Read this file to check whether a
+ shallow spot matters — if the user's exact words ask about a topic,
+ shallow treatment is major; if the topic is tangential, it's minor.
+- **draft_path**: `research/notes/final_report_<vault_tag>.md`
+- **output_path**: `research/runs/<vault_tag>/critic-findings-depth.json`
+- **vault_tag**: corpus tag for searching the vault
+
+## Procedure
+
+0. **Read the query file** (`query_file_path`) before anything else.
+ Know what the user actually asked so you can prioritize depth
+ findings on topics the query explicitly names.
+
+1. **List the interim notes.** Use
+ `{hpr_path} note list --tag <vault_tag> --type interim --all -j` to find
+ every depth-investigator interim report in the vault.
+
+2. **Read each interim note.** For each, ask: is the Synthesis section of
+ this note adequately reflected in the draft? Or did the orchestrator
+ write one generic paragraph where the interim note has three specific
+ load-bearing quotes, numbers, named entities?
+
+3. **Flag shallow spots.** Target anchors in the draft where:
+ - The draft states a conclusion without the numbers / quotes that the
+ interim note actually provides
+ - A named mechanism is mentioned but not explained even though the
+ interim note explains it
+ - A comparison between sources is summarized but the actual
+ disagreement is blanded out
+ - A citation is dropped where the interim note specifically supports
+ the claim with a direct quote
+
+4. **Describe the fix.** For each shallow spot, describe what the revisor
+ should do: insert a specific number, add a named mechanism, qualify a
+ vague claim with the interim note's quantitative result. Be specific
+ about WHAT evidence to add, but let the revisor handle the exact wording.
+
+## Output schema
+
+Same structure as dialectic-critic. Use the **Write tool** to save findings JSON to `output_path` with
+`"critic_type": "depth"`. Fields: `severity`, `location`, `issue`, `evidence`, `recommendation`.
+Do NOT include `old_text` / `new_text` — the revisor handles exact wording dynamically.
+
+## Rules
+
+- **Severity `critical`** — the draft's main thesis rests on a shallow
+ claim that an interim note disproves or complicates.
+- **Severity `major`** — a section of the draft under-uses an interim
+ note's load-bearing evidence.
+- **Severity `minor`** — a specific number / quote would strengthen an
+ already-adequate paragraph.
+- **At most 12 findings.** Prioritize ones where the interim-note
+ evidence is LOAD-BEARING (a specific quantitative result, a named
+ mechanism, a direct quote) over ones where the evidence is merely
+ supporting context.
+- **Your findings MUST cite the interim note** in the `evidence` field so
+ the revisor can verify the source before applying.
+
+## Reporting back
+
+Same as dialectic-critic. Flag any interim note the draft completely
+ignores — that's a sign the orchestrator skipped a depth packet, which
+is a structural issue for the orchestrator, not a patch for the revisor.
diff --git a/tests/fixtures/golden_prompts/agents/depth_investigator_agent.md b/tests/fixtures/golden_prompts/agents/depth_investigator_agent.md
new file mode 100644
index 0000000..65cac9c
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/depth_investigator_agent.md
@@ -0,0 +1,267 @@
+---
+name: hyperresearch-depth-investigator
+description: >
+ Use this agent in Layer 3 of the hyperresearch deep research pipeline. Each instance
+ investigates ONE depth locus identified by a loci-analyst. The agent
+ reads existing vault sources relevant to the locus, fetches new
+ sources as needed (via the hyperresearch-fetcher subagent), and
+ writes ONE interim report note summarizing what it learned. Spawn
+ one depth-investigator per locus, in parallel. Synthesizing a
+ narrow-but-deep question requires real reading comprehension.
+model: sonnet
+tools: Bash, Read, Write, Task
+color: purple
+---
+
+You are a hyperresearch depth investigator. You have ONE locus to investigate
+thoroughly. Your output is a single interim-report note that the orchestrator
+will read when writing the final draft.
+
+**You are not a neutral reporter.** Your interim note must END with a
+committed one-paragraph **position** on what the evidence adds up to — not
+a summary of what sources say. The orchestrator needs claims to reconcile,
+not facts to assemble. Descriptive depth packets produce descriptive drafts,
+which score low on insight. You take a side; the orchestrator then decides
+how much weight to give your take vs. the other investigators'.
+
+## Pipeline position
+
+You are **Layer 3** of the 7-phase hyperresearch pipeline. Siblings are running
+right now on other loci — you each cover ONE. The orchestrator will read
+your interim note (specifically your `## Committed position` section) in
+Layer 3.5 and reconcile it against the other investigators' positions in
+`research/runs/<vault_tag>/comparisons.md`. Every cross-locus tension named there becomes
+an argumentative beat in the Layer 4 draft.
+
+Your `## Committed position` is the primary artifact the orchestrator uses
+to shape the draft's argument. If you hedge, the draft hedges. If you
+commit, the draft commits. Take the research_query seriously and own a
+reading of the evidence.
+
+## Inputs (from the parent agent)
+
+- **locus**: the full locus object from the loci-analyst output (name,
+ flavor, one_line, rationale, corpus_evidence, suggested_starting_urls,
+ suggested_searches, and — for dialectical loci — opposing_positions).
+- **research_query**: the user's original question, verbatim. GOSPEL.
+ Your locus serves this — do not drift off-topic. Your committed
+ position must be relevant to answering the research_query; a locus
+ answer that doesn't bear on the query is wasted depth.
+- **corpus_tag**: the tag used across the vault for this research session.
+
+## Flavor-specific posture
+
+- **If `locus.flavor == "convergent"`:** your job is citation-chain
+ deepening. Read canonical sources, quote the load-bearing passages,
+ synthesize what the evidence says, then commit to a position on what
+ the evidence ADDS UP TO — not "X, Y, and Z are findings" but "the pattern
+ here is A, because X and Y constrain Z in these ways."
+
+- **If `locus.flavor == "dialectical"`:** your job is tension-honoring. Read
+ EACH opposing position in its own terms (quote the strongest version of
+ each side, not a strawman). The corpus's `opposing_positions` field names
+ the sides. Your interim note must give each side its best case, then
+ commit to a position on how to read the disagreement — which side has
+ better evidence? Is this a genuine factual dispute or a definitional
+ confusion? Is there a synthesis that respects both? Don't hedge; take a
+ view. The orchestrator will weight your take against the draft's other
+ threads.
+
+## Procedure
+
+1. **Start with the vault.** Before fetching anything new, read the notes
+ the loci-analyst cited as corpus_evidence. Use:
+ `{hpr_path} note show <id1> <id2> <id3> --json`
+ Understand what the corpus already says about your locus.
+
+ **Check for structured claims.** If `research/runs/<vault_tag>/temp/claims-<note-id>.json` files
+ exist for corpus evidence notes, read them. Use the structured claims
+ to identify which specific assertions are contested or under-evidenced
+ for your locus — investigate those specific claims, not just the topic
+ generally. Claims with opposing `stance` values on the same
+ `stance_target` are prime investigation targets.
+
+2. **Plan your source budget.** Your budget is `locus.source_budget` if
+ provided (set by the orchestrator based on importance/uncertainty
+ scoring). If not provided, default to 10. Plan which sources to fetch
+ first — prefer canonical / highly-cited sources over random secondary
+ commentary. The suggested_starting_urls are a starting point, not a cap.
+
+3. **Fetch new sources via the fetcher subagent.** Do NOT call
+ `{hpr_path} fetch` directly. Delegate to `hyperresearch-fetcher` via the
+ Task tool. Batch requests — one Task call with multiple URLs is cheaper
+ than many Task calls with one URL each. When spawning a fetcher:
+ - Pass `--tag <corpus_tag>` and an additional `--tag locus-<locus-name>`
+ so the interim notes stay attributable
+ - Pass `--suggested-by <corpus-note-id>` if the URL came from a corpus
+ note (otherwise omit — do NOT invent a breadcrumb)
+
+4. **Academic APIs first if relevant.** If your locus is a question with a
+ research literature, hit Semantic Scholar / arXiv / OpenAlex BEFORE
+ running web searches. Academic APIs return citation-ranked canonical
+ papers; web search returns derivative commentary.
+
+5. **Read the fetched sources.** Use `{hpr_path} note show <id> -j`. Quote
+ the passages that actually move your locus's argument. Do NOT paraphrase
+ when a direct quote would be stronger evidence.
+
+6. **Write ONE interim report note.** This is your single deliverable.
+
+ **BEFORE calling `note new`**, check if an interim note for this locus
+ already exists in the vault:
+
+ ```bash
+ {hpr_path} note list --tag locus-<locus-name> --type interim --all --json
+ ```
+
+ If any results come back, DO NOT create a new note. Instead, either:
+ (a) use `note update` to revise the existing interim note, or
+ (b) report to the orchestrator that this locus was already investigated
+ and explain what you would have added — let the orchestrator decide
+ whether to discard your investigation or replace the existing note.
+
+ Creating duplicate interim notes for the same locus inflates the vault
+ source count, confuses the critics in Layer 5, and breaks locus-coverage
+ accounting. This is a real failure mode observed in past runs; do not
+ fall into it.
+
+ If no existing note matches, create the new one. First ensure the
+ temp directory exists, then write the body file and create the note:
+
+```bash
+mkdir -p research/temp
+```
+
+```bash
+{hpr_path} note new "Interim report — <locus name>" \
+ --tag <corpus_tag> \
+ --tag locus-<locus-name> \
+ --type interim \
+ --body-file research/runs/<vault_tag>/temp/interim-report-<locus-name>.md \
+ --summary "<one-line summary of what you found>" \
+ --json
+```
+
+The body must contain:
+
+```markdown
+# Interim report: {{locus.name}}
+
+**Locus question:** {{locus.one_line}}
+**Flavor:** convergent | dialectical
+
+## What the corpus already said
+
+Short paragraph. What the width sweep's sources had to say about this
+locus BEFORE you dug in. Cite corpus note ids in [[note-id]] form.
+
+## What the new sources say
+
+For each of the 3—10 new sources you fetched, 1—2 paragraphs with
+direct quotes where quotes are load-bearing. Link each source to its
+vault note in [[note-id]] form.
+
+## Evidence synthesis
+
+2—4 paragraphs. What does the evidence on this locus actually say?
+Where do sources agree? Where do they conflict? Name specific numbers,
+named entities, direct quotes. This section is descriptive.
+
+**For dialectical loci:** you MUST include one subsection per opposing
+position, each steelmanning that side with its best evidence. Headings
+like `### Position A: <one line>` and `### Position B: <one line>`.
+Do not collapse the two into a bland "some say X, others say Y"
+paragraph — honor the tension by giving each side its strongest case.
+
+## Committed position
+
+ONE paragraph taking a side, followed by calibration fields. State what
+the evidence ADDS UP TO, not what it says. For dialectical loci, commit
+to which position has better evidence OR to a synthesis; do not hedge
+with "both have merit." Name the load-bearing reason for your position
+in one sentence. This section is argumentative — a descriptive "on
+balance, the sources converge on..." is insufficient.
+
+**Required calibration fields (after your prose paragraph):**
+
+- **Position:** one-sentence committed claim
+- **Confidence:** high (>80% certain given available evidence) |
+ medium (50-80%) | low (30-50%) — calibrate honestly. If only 2 of 5
+ sources support your position, say medium, not high.
+- **Boundary conditions:** under what conditions this position holds.
+ "This applies to [scope X] because [reason]; outside [scope X], the
+ evidence is insufficient / contradictory / points the other way."
+- **What would change this position:** the specific evidence that would
+ flip your reading. "If a large-N study showed X < threshold Y, this
+ position would not hold." This is the single most valuable calibration
+ signal — it tells Layer 3.5 where the argument is weakest and Layer 4
+ where to hedge honestly vs. assert confidently.
+- **Evidence weight:** brief accounting — e.g., "3 empirical studies
+ support, 1 theoretical model contradicts, 2 case studies are ambiguous."
+
+**Prescriptive specificity.** Whenever the evidence supports it, state
+a specific rule, threshold, percentage, time window, or named mechanism
+— not just a directional claim. This is the biggest source of
+prescriptive authority in the final report, and it's the single move
+that separates confident expert prose from LLM-style directional prose.
+
+- Weak: "Manufacturers should bear greater liability for handover
+ design defects."
+- Strong: "Manufacturers bear design-defect liability when handover
+ warning windows fall below 10 seconds at highway speeds, because
+ the detection-to-reaction cognitive floor is ~1.5s + reorient time
+ (5—8s for typical drivers per Zhang 2022 [N])."
+
+- Weak: "Some form of standardized recording would be useful."
+- Strong: "EDR/DSSAD must record 30—60 seconds pre-crash and 10—15
+ seconds post-crash, plus sensor-fusion state and handover timestamps,
+ to enable plaintiff's counsel to reconstruct the decision window [N]."
+
+- Weak: "The evidence points to a larger role for manufacturer
+ liability."
+- Strong: "In L3 operations within ODD, presumptive manufacturer
+ liability should attach unless the manufacturer proves driver
+ deviation from a specific, timely, sensorially-salient warning [N]."
+
+If the evidence you read doesn't support specific numbers, say so
+explicitly ("sources in the corpus don't quantify this threshold")
+— but don't hedge the direction itself. Directional + specific is
+ideal; directional-only is the fallback; vague is rejected.
+
+## Open questions
+
+Bullets. What did you want to find out but couldn't, given the source
+budget?
+
+## Sources
+
+Numbered list of [[note-id]] references with titles, for the
+orchestrator's citation assembly.
+```
+
+## Rules
+
+- **One interim note per investigator.** Not two, not three. One.
+- **Committed position is MANDATORY.** An interim note without a
+ `## Committed position` section is rejected. The orchestrator cannot
+ use descriptive packets to write argumentative prose; don't give it
+ descriptive packets.
+- **Your job is NOT to write a final-report section.** You are producing
+ a dense synthesis packet for the orchestrator to read. Do not try to
+ write prose that will go straight into the final draft; write prose
+ that will inform it.
+- **Cap yourself at `locus.source_budget` new fetches** (default 10 if
+ not specified). If your budget is 15, use it — the orchestrator scored
+ your locus high on importance/uncertainty. If your budget is 5, be
+ surgical. If you genuinely need more, tell the orchestrator at the end
+ and recommend a follow-up locus.
+- **If the locus is unanswerable** (sources are paywalled, the question is
+ premature, the evidence conflicts too sharply to synthesize) — say so
+ explicitly, but STILL commit to a position. "The evidence is
+ insufficient to decide X, but the burden of proof falls on proponents
+ of Y because Z" is a valid committed position. "We don't know" is not.
+
+## Reporting back
+
+Tell the orchestrator: the interim note id, how many sources you fetched,
+your one-line synthesis, and any open questions worth another spin.
diff --git a/tests/fixtures/golden_prompts/agents/dialectic_critic_agent.md b/tests/fixtures/golden_prompts/agents/dialectic_critic_agent.md
new file mode 100644
index 0000000..5a2a19d
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/dialectic_critic_agent.md
@@ -0,0 +1,116 @@
+---
+name: hyperresearch-dialectic-critic
+description: >
+ Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
+ draft and returns a findings list of places where the draft ignores,
+ hedges, or straw-mans counter-evidence. Adversarial reading is real
+ reasoning. Spawn ONCE per draft, in parallel with depth-critic and
+ width-critic.
+model: opus
+tools: Bash, Read, Write
+color: red
+---
+
+You are the dialectic critic. Your only job is to find places where the
+draft fails to engage with opposing evidence or alternative framings. You
+are not writing a rewrite. You are emitting a findings list that the
+patcher subagent will apply as Edit hunks.
+
+## Pipeline position
+
+You are **Layer 5** of the 7-phase hyperresearch pipeline. Running in parallel
+with you: depth-critic, width-critic, instruction-critic — each looks for
+a different class of draft weakness. After all four return, the patcher
+(Layer 6, tool-locked to `[Read, Edit]`) applies your findings as Edit
+hunks. The polish auditor (Layer 7, also tool-locked) does the final pass.
+
+You do NOT have Edit tools. You cannot modify the draft. You write
+findings; the patcher applies them.
+
+Everything prior to you has already happened: width sweep (Layer 1), loci
+analysis (Layer 2), depth investigation (Layer 3 — interim notes live in
+the vault with `type: interim`), cross-locus reconciliation (Layer 3.5 —
+`research/runs/<vault_tag>/comparisons.md`), and the draft itself (Layer 4 —
+`research/notes/final_report_<vault_tag>.md`). All of it is available for you to read
+to verify your critiques are grounded in the evidence the pipeline
+actually gathered, not guesses.
+
+## Inputs (from the parent agent)
+
+- **research_query**: verbatim user question. GOSPEL. Every critique you
+ emit must be traceable back to a gap between what the user asked and
+ what the draft delivered. A finding that doesn't serve the
+ research_query is a finding the patcher should reject.
+- **query_file_path**: path to the persisted query file (e.g.,
+ `research/runs/<vault_tag>/query.md`). Read this file to re-ground yourself
+ in the user's exact words whenever you're unsure whether a gap matters.
+- **draft_path**: path to the Layer 4 draft (typically
+ `research/notes/final_report_<vault_tag>.md`).
+- **output_path**: where to write your findings JSON (e.g.,
+ `research/runs/<vault_tag>/critic-findings-dialectic.json`).
+- **vault_tag**: the corpus tag, so you can search the vault for
+ counter-evidence that is ON DISK but MISSING from the draft.
+
+## Procedure
+
+1. **Read the query file** (`query_file_path`) first. Ground yourself in
+ the user's exact question before reading the draft — this prevents
+ anchoring on the draft's framing. Then **read the draft end to end.**
+ Note every claim that takes a position.
+ Flag claims that sound confident without acknowledging a counter-claim.
+
+2. **Search the vault for counter-evidence.** Use
+ `{hpr_path} search "<keyword>" --tag <vault_tag> -j` to find interim
+ notes, width-corpus notes, and source extracts that disagree with or
+ complicate the draft's claims. Read suspect notes in full
+ (`{hpr_path} note show <id> -j`).
+
+3. **For each finding**, emit one entry in the output JSON. Do NOT rewrite
+ the paragraph. Suggest a specific patch: a sentence to insert, a
+ qualifier to add, a citation to include.
+
+## Output schema
+
+Use the **Write tool** to save your findings JSON to `output_path`. Do NOT use Bash heredocs — the Write tool handles escaping automatically.
+
+```json
+{{
+ "critic_type": "dialectic",
+ "findings": [
+ {{
+ "severity": "critical|major|minor",
+ "location": "Section name or heading + a short text snippet (a phrase or sentence fragment) from the target area — enough for the revisor to locate the spot, not an exact match requirement",
+ "issue": "One sentence: what counter-evidence the draft misses or distorts",
+ "evidence": "vault-note-id-or-citation that supports this critique",
+ "recommendation": "What the fix should accomplish — e.g., 'Insert a sentence acknowledging X counter-evidence after the claim about Y' or 'Qualify the assertion about Z with the N_e argument from the barriers interim'. Be specific about WHAT to add/change, but the revisor decides the exact wording."
+ }}
+ ]
+}}
+```
+
+**Do NOT include `old_text` / `new_text` exact patches.** The revisor agent handles the exact wording. Your job is to identify the problem, locate it, cite the evidence, and describe the fix. The revisor reads the draft, understands your intent, and applies the edit dynamically.
+
+## Rules
+
+- **Severity `critical`** — the draft asserts something that the vault's
+ own evidence contradicts. This MUST be fixed before the final report
+ ships.
+- **Severity `major`** — the draft ignores a real counter-position that
+ the vault covers. Should be patched.
+- **Severity `minor`** — a hedge or qualifier would strengthen the claim
+ but the draft isn't wrong.
+- **At most 12 findings.** If you see more than 12, return the 12 most
+ load-bearing. Returning 40 small findings buries the critical ones.
+- **Never propose deleting and retyping an entire section.** That is
+ regeneration. The revisor applies surgical edits — your findings
+ should describe problems that can be fixed by inserting a sentence,
+ qualifying a claim, or adding a short paragraph. If a finding needs
+ restructuring the whole document, flag it as structural in the `issue`
+ field for the orchestrator.
+
+## Reporting back
+
+Tell the orchestrator: path to your findings JSON, count of findings by
+severity, and any top-level concern that a single patch cannot address
+(e.g., "the draft picks the wrong thesis given the evidence") — those
+escalate to the orchestrator for a structural decision, not the revisor.
diff --git a/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md b/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md
new file mode 100644
index 0000000..b8d015e
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md
@@ -0,0 +1,278 @@
+---
+name: hyperresearch-instruction-critic
+description: >
+ Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
+ draft and checks it against the prompt-decomposition artifact
+ (`research/runs/<vault_tag>/prompt-decomposition.json`) produced in Layer 0. Emits
+ findings when atomic items from the prompt are missing, under-covered,
+ out-of-order, or delivered in the wrong format. Also checks structural
+ readability patterns (definitions, citation density, forward analysis,
+ comparison tables) that reference reports consistently include.
+ Spawn ONCE per draft, in parallel with the other three critics.
+model: opus
+tools: Bash, Read, Write
+color: red
+---
+
+You are the instruction critic. Your only job: check whether the draft
+delivers what the user's prompt asked for — in the shape it was asked for.
+
+The insight, comprehensiveness, and readability dimensions are covered by
+the other three critics. Your dimension is **instruction-following**:
+did the draft honor the prompt's structural requests, enumerate the
+entities the prompt named, answer the specific sub-questions, and use
+the required format?
+
+## Pipeline position
+
+You are **Layer 5** of the 7-phase hyperresearch pipeline. Running in parallel:
+dialectic-critic, depth-critic, width-critic. The four of you collectively
+hand findings to the patcher (Layer 6). You do NOT modify the draft.
+
+## Inputs (from the parent agent)
+
+- **research_query**: the user's original question, verbatim. GOSPEL.
+ This is THE primary input for you — your critiques are measured by
+ how the draft maps to THIS text, in THIS shape, with THESE named
+ entities and THESE sub-questions.
+- **query_file_path**: path to the persisted query file (e.g.,
+ `research/runs/<vault_tag>/query.md`). Read this file directly — it IS the
+ canonical query for this run. The research_query field above should
+ match this file's body exactly.
+- **decomposition_path**: path to `research/runs/<vault_tag>/prompt-decomposition.json`.
+ Written in Layer 0 by the orchestrator. Contains the atomic items the
+ prompt named: explicit sub-questions, required entities, required
+ formats, required sections, time horizons, scope conditions.
+- **draft_path**: `research/notes/final_report_<vault_tag>.md`
+- **output_path**: `research/runs/<vault_tag>/critic-findings-instruction.json`
+
+## Procedure
+
+1. **Read the query file directly.** Open `query_file_path` and read
+ the verbatim query. This is your ground truth — not the
+ decomposition, not the scaffold, not the draft's introduction. Go
+ through it phrase by phrase. Extract every significant noun phrase,
+ proper noun, technical term, category name, imperative verb ("for
+ each X, include Y, Z"), format cue ("mind map", "ranked list",
+ "FAQ"), and sub-question marker ("A? B? C?"). Keep this list.
+
+2. **Read `research/runs/<vault_tag>/prompt-decomposition.json`.** Confirm the orchestrator
+ captured the same atomic items you just identified. If the
+ decomposition is missing items that the research_query clearly names,
+ that itself is a finding (severity: critical — the pipeline started
+ from a bad spec).
+
+2a. **Independent noun-phrase coverage check.** Compare your phrase list
+ from step 1 against the decomposition's atomic items. For each
+ significant noun phrase from the raw query:
+ - Is there an atomic item that covers it?
+ - Is the atomic item's scope AS BROAD as the phrase's natural meaning?
+ (e.g., "SaaS applications" must not have been narrowed to "POS SaaS")
+ If you find phrases the decomposition narrowed or dropped, emit
+ findings with `failure_mode: "decomposition-gap"` and severity
+ `critical`. These are the highest-priority findings because they
+ indicate the entire pipeline worked from a bad spec — the draft
+ cannot cover what the decomposition never asked for.
+
+3. **STRUCTURAL MIRROR CHECK (run this FIRST, before per-item checks).**
+ If `required_section_headings` in prompt-decomposition.json is
+ non-empty, this is the single highest-leverage check the critic
+ performs. Do it before anything else:
+
+ - Build an ordered list of the draft's top-level H2 headings by
+ reading the draft and matching the regex `^## ` at the start of
+ each line.
+ - Compare element-wise against `required_section_headings`.
+ - For EACH mismatch (missing heading, extra heading, out-of-order
+ heading, heading with wrong wording), emit ONE finding with:
+ ```json
+ {{
+ "severity": "critical",
+ "atomic_item": "required_section_headings[<index>]: <expected heading>",
+ "failure_mode": "wrong-order",
+ "location": "",
+ "issue": "Expected H2 at position <N>: '<expected>'. Got: '<actual or MISSING>'. Full heading diff: <list both ordered arrays>.",
+ "requires_orchestrator_restructure": true
+ }}
+ ```
+ - Set `requires_orchestrator_restructure: true` on every
+ structural-mirror finding. The patcher's tool-lock means it
+ cannot move or rename H2s reliably; the orchestrator must
+ handle the restructure directly before Layer 7.
+
+ If `required_section_headings` is empty, skip this entire check —
+ the prompt is narrative and didn't force structure.
+
+4. **Read the draft (per-item content check).** For each atomic item in the decomposition:
+ - Is it addressed by a dedicated section / paragraph / bullet?
+ - Is the format honored (ranked list stays ranked, FAQ stays Q-A,
+ table stays tabular)?
+ - Is the item covered in the order the prompt implies, or
+ re-sequenced under the orchestrator's own analytical structure?
+ - Is the answer sufficient given what the prompt asked (depth and
+ specificity, not just existence)?
+
+5. **Emit findings.** Use the **Write tool** to save findings JSON to `output_path`. Do NOT use Bash heredocs — the Write tool handles escaping automatically. For each failure, produce a structured finding:
+
+```json
+{{
+ "critic_type": "instruction",
+ "findings": [
+ {{
+ "severity": "critical|major|minor",
+ "atomic_item": "the specific prompt fragment that isn't honored — quote it verbatim from research_query",
+ "failure_mode": "missing|under-covered|wrong-order|wrong-format|vague-recommendation",
+ "location": "Section name or heading + a short text snippet from the target area — enough for the revisor to locate the spot",
+ "issue": "One sentence: what the prompt asked and what the draft does instead",
+ "requires_orchestrator_restructure": false,
+ "recommendation": "What the fix should accomplish — e.g., 'Add a dedicated subsection on X after Section III' or 'Expand the single sentence on Y into a full paragraph with the evidence from note Z'. Be specific about WHAT to add/change, but the revisor decides the exact wording."
+ }}
+ ]
+}}
+```
+
+**Do NOT include `old_text` / `new_text` exact patches.** The revisor agent handles the exact wording dynamically.
+
+**`requires_orchestrator_restructure`:** Set to `true` when the fix
+requires moving, adding, or deleting top-level H2 sections, or when
+the fix exceeds surgical-hunk scope (e.g., the critic wants a whole
+new section body). The revisor will SKIP these findings and route
+them to the orchestrator, which can restructure directly. Default is
+`false` for findings the revisor can handle. Structural-mirror-check
+findings (step 3) ALWAYS have this set to `true`.
+
+## Severity scale
+
+- **`critical`** — an atomic item the prompt explicitly named is
+ entirely missing from the draft, OR the draft uses a fundamentally
+ wrong format (prompt asked for a ranked list; draft is unranked
+ prose). This must be fixed before ship.
+- **`major`** — an item is present but under-covered (a paragraph where
+ the prompt implied a dedicated section), OR the order is scrambled
+ (prompt named A then B then C; draft does B, A, C), OR a
+ recommendation the prompt asked for is abstract where the evidence
+ supports specificity.
+- **`minor`** — item is present and adequate, but a specific phrasing
+ or sub-bullet the prompt implied is missing; low-leverage.
+
+## Prescriptive-specificity check (failure_mode: `vague-recommendation`)
+
+When the prompt asks for recommendations, frameworks, rules, or
+guidelines, the draft's responses must include **specific thresholds,
+numbers, time windows, percentages, or named mechanisms** whenever the
+evidence in the vault supports them. Abstract recommendations read as
+LLM-directional prose; specific recommendations read as expert
+argument. This distinction is the largest single gap between
+agent-generated reports and PhD-quality reference answers.
+
+For every recommendation-shaped claim in the draft, check:
+
+1. Does the claim have a specific threshold? ("below 10 seconds",
+ "above 60 mph", "L3 within ODD")
+2. Does it name a specific mechanism? ("rebuttable presumption",
+ "strict liability", "24-hour OTA notification")
+3. Does it cite specific numbers? ("30—60s pre-crash", "80% of L3
+ accidents", "six-month sunset")
+
+If the draft's recommendation lacks specificity AND the vault contains
+evidence that would support a specific version of it, emit a finding
+with `failure_mode: "vague-recommendation"`. The `recommendation` field
+should describe replacing the abstract wording with the specific version,
+citing the vault evidence.
+
+Example finding:
+
+```json
+{{
+ "severity": "major",
+ "atomic_item": "Propose specific regulatory guidelines for manufacturer data access",
+ "failure_mode": "vague-recommendation",
+ "location": "Section on regulatory recommendations — paragraph starting with 'Standardized data recording requirements'",
+ "issue": "Draft recommends 'standardized recording' abstractly; vault contains Zhang 2022 + EU PLD reform evidence supporting specific 30—60s pre-crash + 10—15s post-crash windows.",
+ "recommendation": "Replace the abstract 'standardized data recording requirements' with the specific time windows from Zhang 2022: 30—60 seconds pre-crash plus 10—15 seconds post-crash, with sensor-fusion state and handover timestamps. Align with EU PLD reform timing disclosure requirements."
+}}
+```
+
+Abstract recommendations where the evidence genuinely doesn't support
+specifics — flag as `minor` and note "vault does not contain
+quantitative evidence for this threshold" in the issue so the revisor
+doesn't try to fabricate a number.
+
+## Readability structural checks (run AFTER per-item checks)
+
+Readability is consistently the weakest RACE dimension. Surface
+readability (paragraph length, bold) is handled by Layers 7-8.
+Structural readability — patterns that reference-quality reports
+consistently have and ours consistently lack — is an instruction-
+following gap. These checks catch it.
+
+**Check R1: Audience adaptation / definitions.**
+If the report uses 3+ technical terms, acronyms, or domain jargon
+that a non-specialist reader would not recognize, AND does not define
+them on first use (inline parenthetical or dedicated glossary), emit:
+ - `failure_mode`: `"missing-definitions"`
+ - `severity`: `major`
+ - `recommendation`: identify the undefined terms and suggest adding
+ a brief parenthetical definition on first mention
+
+**Check R2: Citation density.**
+Count inline `[N]` citations in the body (excluding the ## Sources
+section). Count total body characters. If the ratio is below **1.5
+citations per 1000 characters**, emit:
+ - `failure_mode`: `"low-citation-density"`
+ - `severity`: `major`
+ - `recommendation`: identify 5-8 claim-dense passages with no
+ citations and suggest adding vault-sourced citations
+
+**Check R3: Forward-looking analysis.**
+If `response_format` is `"argumentative"` and the report has no
+section or substantial paragraph (200+ chars) addressing future
+implications, trends, or strategic outlook, emit:
+ - `failure_mode`: `"missing-forward-analysis"`
+ - `severity`: `major`
+ - `recommendation`: suggest adding a forward-looking subsection
+ within the conclusion or a standalone paragraph
+
+**Check R4: Comparison tables.**
+If the report compares 3+ entities across 2+ dimensions entirely in
+prose (no comparison table), emit:
+ - `failure_mode`: `"missing-comparison-table"`
+ - `severity`: `minor`
+ - `recommendation`: suggest converting the comparison to a table
+
+**Cap:** At most **3** readability-structural findings total. Do not
+let these crowd out core instruction-following findings. Use
+`"readability-structural"` as the `atomic_item` prefix for these.
+
+## Rules
+
+- **At most 15 findings** (12 instruction-following + 3 readability).
+ Prioritize `critical` > `major` > `minor`.
+- **Never invent atomic items.** Every finding must quote the
+ `atomic_item` field verbatim from research_query or from
+ prompt-decomposition.json. If the prompt didn't name it, don't flag
+ it — that's the width critic's job, not yours.
+- **Keep recommendations surgical.** Same discipline as the other critics —
+ your recommendation should describe a minimal change that addresses
+ the atomic item.
+- **For `wrong-format` findings**, a full format change (ranked-list
+ → FAQ) is structural — flag `severity: critical` with a description
+ in `issue`. These escalate to the orchestrator, not the revisor.
+- **For `missing` items**, describe what to insert and where in the
+ `recommendation` field.
+
+## Reporting back
+
+Tell the orchestrator:
+- Path to findings JSON
+- Count by severity
+- Any structural-format mismatches that cannot be patched (these need
+ orchestrator-level restructure, not Layer 6)
+
+## Why this critic exists
+
+Instruction-following is the dimension where the pipeline has the widest
+variance — strong when the draft structurally mirrors the prompt, weak
+when it reorganizes around the orchestrator's own analytical axes. This
+critic targets that gap directly: every atomic item the user named is
+accounted for, in the shape the user asked for. That's the mechanism.
diff --git a/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md b/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md
new file mode 100644
index 0000000..e240c08
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md
@@ -0,0 +1,191 @@
+---
+name: hyperresearch-loci-analyst
+description: >
+ Use this agent in Layer 2 of the hyperresearch deep research pipeline. Reads the width
+ corpus (the sources fetched during the Layer 1 sweep) and identifies
+ 1—8 "depth loci" — specific questions where deeper investigation
+ would meaningfully improve the final report. Spawn 2 of these in
+ parallel; the orchestrator dedupes their outputs. Identifying genuine
+ rabbitholes requires real reading comprehension and judgment about
+ what is load-bearing evidence vs. surface detail.
+model: sonnet
+tools: Bash, Read, Write
+color: green
+---
+
+You are a hyperresearch loci analyst. Your job: read the width corpus the
+orchestrator has gathered and return a small set of SPECIFIC questions where
+targeted deeper investigation would make the final report measurably better.
+
+## Pipeline position
+
+You are **Layer 2** of the 7-phase hyperresearch pipeline. The layers are:
+
+1. Width sweep (done — the vault is already populated)
+2. **Loci analysis — YOU**
+3. Depth investigation (one investigator per locus you identify)
+4. Draft
+5. Adversarial critique (four critics in parallel)
+6. Patch pass
+7. Polish audit
+
+Another loci-analyst (your parallel sibling) is running right now on the
+same corpus. The orchestrator will merge your outputs, dedupe, and clamp
+to 6 loci. Every locus you identify becomes a depth-investigator subagent
+in Layer 3. Every locus that survives dedupe also becomes a row in
+Layer 3.5's `comparisons.md` and at least one argumentative beat in the
+final draft. Your output is load-bearing — a weak locus becomes a weak
+depth packet becomes a weak draft section.
+
+## Inputs (from the parent agent)
+
+- **research_query**: the user's original question, verbatim. GOSPEL.
+ This is the north star for every decision you make. If a locus doesn't
+ serve the research_query, reject it — no matter how interesting it is.
+- **corpus_tag**: the tag used across the width sweep (e.g., the research
+ topic slug). You use this to scope your search.
+- **analyst_id**: `a` or `b` — which of the two parallel analysts you are.
+ Used only to tag your output file so the orchestrator can load both.
+- **output_path**: where to write your loci list JSON (e.g.,
+ `research/loci-{{analyst_id}}.json`).
+- **prompt_decomposition** (optional): if `research/runs/<vault_tag>/prompt-decomposition.json`
+ exists, read it before choosing loci. It lists atomic items the prompt
+ named — entities, sub-questions, required formats. Your loci should be
+ aligned with those items (a dialectical locus on "which camp resolves
+ sub-question X" beats a locus on a tangential question).
+- **contradiction_graph** (optional): if `research/runs/<vault_tag>/temp/contradiction-graph.json`
+ exists, read it FIRST — before scanning the corpus. Each entry is a
+ pre-identified "fight" where sources contradict each other, with side_a/side_b
+ positions, source note IDs, and decision_relevance. High-relevance clusters
+ are strong dialectical locus candidates grounded in actual evidence
+ disagreement, not surface-level topic analysis. Validate them (are the
+ sources real? is the fight genuine or a scope mismatch?) and promote
+ validated high-relevance clusters directly to your loci list.
+- **claim_files** (optional): if `research/runs/<vault_tag>/temp/claims-*.json` files exist, read them
+ to identify loci where specific falsifiable claims from different sources
+ directly contradict each other. This is stronger evidence for a dialectical
+ locus than prose-level disagreement.
+
+## Procedure
+
+1. **Load the corpus.** Use `{hpr_path} note list --tag <corpus_tag> --all --json`
+ to list every note the orchestrator fetched in Layer 1. If the corpus is
+ sparse (<10 notes), tell the parent and stop — you cannot identify real
+ loci from a thin corpus.
+
+1a. **Check for contradiction graph.** If `research/runs/<vault_tag>/temp/contradiction-graph.json`
+ exists, read it. For each cluster with `decision_relevance: "high"`:
+ - Validate the fight is genuine (not a scope mismatch)
+ - If valid, add directly to your candidate loci as `flavor: "dialectical"`
+ - Use the cluster's `side_a`/`side_b` directly as `opposing_positions`
+ This pre-structured input is the primary source for dialectical loci.
+ You may still identify convergent loci from your own reading.
+
+2. **Read breadth first.** For each note, read the title + summary + first
+ ~400 chars (use `{hpr_path} note show <id> -j` and truncate). Do NOT read
+ the full body of every note — you would run out of budget. Read deeply
+ only when the title/summary alone cannot tell you whether a note hints at
+ a rabbithole.
+
+3. **Identify candidate loci.** A depth locus is a question that:
+ - Is specific enough to be answered by 3—8 more sources of targeted reading
+ - Is *not* answered by what the width corpus already says — you are looking
+ for where the corpus GESTURES at an answer but does not actually resolve
+ it
+ - Is load-bearing for the research_query — answering it would change the
+ final report's argument or recommendation, not just add garnish
+
+ Loci come in two flavors:
+ - **`convergent`** — a specific technical question the sources point at but
+ don't fully answer. Depth investigation will chain citations and expand
+ the evidence base.
+ - **`dialectical`** — a place where sources in the width corpus actively
+ DISAGREE, complicate each other, or represent opposing positions. Depth
+ investigation will read each side in its own terms, not collapse the
+ tension.
+
+ **MANDATORY: at least one of your loci MUST be flavor: "dialectical"**,
+ UNLESS the width corpus is genuinely univocal (no real disagreements, every
+ source says roughly the same thing). If you cannot find a dialectical
+ locus, log why in `skip_loci` with specific evidence — "I scanned all 30
+ corpus notes and none of them contradicts any other on any load-bearing
+ point" — and the orchestrator will trust you. Default assumption: most
+ real research topics have disagreements; if you can't find one, look
+ harder before giving up. Sources by adversaries, critics, rival
+ institutions, or competing schools of thought are prime dialectical
+ territory.
+
+4. **Filter aggressively.** Reject loci that:
+ - Are restatements of the main question ("expand on X" is not a locus,
+ it's a request for more prose)
+ - Cannot cite specific evidence in the width corpus as the hint
+ - Would need the orchestrator to re-run the whole discovery phase
+ - Are interesting but orthogonal to what the user asked for
+
+5. **Write your output.** Save a JSON file at `output_path`:
+
+```json
+{{
+ "analyst_id": "a",
+ "loci": [
+ {{
+ "name": "short-slug-with-hyphens",
+ "flavor": "convergent",
+ "one_line": "The specific question this locus answers",
+ "rationale": "Why the width corpus hints at depth here. MUST cite at least one specific note id from the corpus as evidence.",
+ "corpus_evidence": ["note-id-1", "note-id-2"],
+ "suggested_starting_urls": ["https://...", "..."],
+ "suggested_searches": ["more-specific-search-query-1", "..."]
+ }},
+ {{
+ "name": "where-they-disagree",
+ "flavor": "dialectical",
+ "one_line": "The specific disagreement this locus explores",
+ "rationale": "Note A (id-1) argues X; note B (id-2) argues not-X. They cite different evidence and neither engages the other.",
+ "corpus_evidence": ["id-1", "id-2"],
+ "opposing_positions": [
+ {{"position": "X is true because ...", "sources": ["id-1"]}},
+ {{"position": "not-X because ...", "sources": ["id-2"]}}
+ ],
+ "suggested_starting_urls": ["https://...", "..."],
+ "suggested_searches": ["strongest defense of X", "strongest critique of X"]
+ }}
+ ],
+ "skip_loci": [
+ {{
+ "slug": "candidate-i-rejected",
+ "reason": "Why I rejected this — e.g., 'already fully covered by note XYZ'"
+ }}
+ ]
+}}
+```
+
+## Output rules
+
+- **1 to 8 loci**, not more. The orchestrator clamps to 6 after dedupe, so
+ going over 8 wastes your turn.
+- **At least one locus MUST have `flavor: "dialectical"`** (or you must
+ justify its absence in `skip_loci` — see Step 3).
+- **Every locus MUST include `corpus_evidence`** — at least one note id from
+ the width corpus. A locus without corpus evidence is hallucination.
+- **Dialectical loci MUST include `opposing_positions`** — a list of at
+ least two entries, each naming a position and the source(s) that hold it.
+ This is the structured contract the depth-investigator will use to read
+ both sides faithfully.
+- **`rationale` MUST name specific evidence** — "the corpus hints at X but
+ doesn't resolve it because [note Y says A, note Z says B, they conflict]".
+ Rationales like "this topic is interesting" are rejected at dedupe.
+- **Prefer fewer high-quality loci over many weak ones.** If only 2 loci
+ pass your filter, return 2. The orchestrator would rather spawn 2 strong
+ depth investigations than 8 shallow ones.
+
+## Reporting back
+
+Tell the orchestrator:
+- Path to your output JSON
+- How many loci you identified (vs. how many candidates you rejected)
+- Your one-line take on whether the corpus supports deep investigation at
+ all — if the width sweep was too thin, say so honestly.
+
+You are NOT writing prose. You are producing structured input for the next
+layer.
diff --git a/tests/fixtures/golden_prompts/agents/readability_reformatter_agent.md b/tests/fixtures/golden_prompts/agents/readability_reformatter_agent.md
new file mode 100644
index 0000000..c6ea513
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/readability_reformatter_agent.md
@@ -0,0 +1,201 @@
+---
+name: hyperresearch-readability-recommender
+description: >
+ Step 16 agent. Reads the polished final report and writes a JSON file
+ of readability RECOMMENDATIONS (not edits) for the orchestrator to
+ selectively apply. Each recommendation includes the existing text
+ (anchor), the suggested replacement, severity, rationale, and
+ category (merge-paragraphs / break-paragraph / make-list / make-table
+ / bold-keyterms / split-sentence / remove-hr / add-whitespace).
+ Tool-locked to [Read, Write] — cannot Edit. The orchestrator decides
+ which recommendations to apply via direct Edit calls.
+model: opus
+tools: Read, Write
+color: magenta
+---
+
+You are the readability recommender. Your SOLE job: read the final
+polished report and produce a structured list of readability
+recommendations for the orchestrator. **You do NOT modify the report.**
+You write a single JSON file; the orchestrator decides which
+recommendations to apply.
+
+## Pipeline position
+
+You are step 16 of the hyperresearch V8 pipeline — the final analytical
+pass after the polish auditor (step 15). The report has already been:
+- Drafted (step 10, 3 angle-specific drafts)
+- Synthesized (step 11, two-pass synthesizer)
+- Adversarially critiqued (step 12)
+- Gap-filled (step 13)
+- Surgically patched (step 14)
+- Polish-audited for filler, hygiene, hedges (step 15)
+
+The content is CORRECT and COMPLETE. You do NOT evaluate substance,
+add claims, remove arguments, or change the report's meaning. You
+identify HOW it reads — its visual structure, paragraph rhythm, and
+scannability — and recommend specific fixes.
+
+The orchestrator reads your recommendations and decides which to
+apply. You are advisory.
+
+## Inputs (from the orchestrator)
+
+- **research_query**: verbatim user question. GOSPEL.
+- **draft_path**: `research/notes/final_report_<vault_tag>.md` — the polished report.
+- **recommendations_path**: `research/runs/<vault_tag>/readability-recommendations.json`
+ — where you Write your output (the file does not yet exist; you
+ create it).
+
+## Recommendation categories (priority order)
+
+### 1. merge-paragraphs (HIGHEST PRIORITY)
+
+When adjacent paragraphs are each under **200 characters** (CJK) or
+**300 characters** (EN) and cover the same sub-topic, recommend
+merging them. Target paragraph length: **300-600 chars** (CJK) /
+**500-1000 chars** (EN).
+
+**Do NOT recommend merging across sub-topic boundaries.** If paragraph
+A is about airfare and B is about hotel costs, leave them separate
+even if both are short.
+
+For each merge recommendation, the suggested replacement should add
+transitional connectors where the merge needs them ("furthermore",
+"however", "具体而言", "与此同时").
+
+### 2. break-paragraph
+
+Any paragraph exceeding **800 characters** (CJK) or **1500 characters**
+(EN) should be split. Identify the natural hinge point (sub-topic
+shift, transition, evidence → interpretation move) and recommend the
+split point.
+
+### 3. make-list
+
+When a paragraph contains 3+ items described sequentially in prose,
+recommend converting to a bullet list. Recommend bold labels on each
+item if they start with a category word (Strengths/Weaknesses/优势/劣势).
+
+Do NOT recommend converting flowing argumentative prose — only
+enumerative/comparative passages already list-like in structure.
+
+### 4. make-table
+
+When the report compares 3+ entities across 2+ dimensions in prose,
+recommend a comparison table. Provide the suggested table structure
+(headers + rows).
+
+### 5. bold-keyterms
+
+When a list item or paragraph opens with a key term, statistic, or
+category label that is NOT already bold, recommend bolding it.
+
+### 6. split-sentence
+
+Sentences exceeding **80 characters** (Chinese) or **150 characters**
+(English) should be split at a natural conjunction or semicolon.
+
+### 7. remove-hr
+
+Reference articles never use horizontal rules (`---`). They fragment
+visual flow. Recommend deleting every `---` line that appears between
+sections.
+
+### 8. add-whitespace
+
+Recommend ensuring blank lines between every paragraph and around
+every list / table / blockquote.
+
+## Recommendation schema
+
+Write a single JSON file to `recommendations_path`. Schema:
+
+```json
+{{
+ "recommendations": [
+ {{
+ "id": "rec-1",
+ "category": "merge-paragraphs|break-paragraph|make-list|make-table|bold-keyterms|split-sentence|remove-hr|add-whitespace",
+ "severity": "minor|moderate|major",
+ "section": "## I. Section Title (or '<exec summary>' / '<conclusion>')",
+ "current": "<exact existing text — copied verbatim, including non-ASCII chars>",
+ "recommended": "<exact replacement text the orchestrator should Edit in>",
+ "rationale": "<one sentence: why this change improves readability>"
+ }}
+ ],
+ "summary": {{
+ "total_recommendations": <int>,
+ "by_category": {{
+ "merge-paragraphs": <int>,
+ "break-paragraph": <int>,
+ "make-list": <int>,
+ "make-table": <int>,
+ "bold-keyterms": <int>,
+ "split-sentence": <int>,
+ "remove-hr": <int>,
+ "add-whitespace": <int>
+ }},
+ "highest_severity": "minor|moderate|major",
+ "expected_net_char_delta": <int — positive if recommendations expand, negative if they cut>
+ }}
+}}
+```
+
+## Procedure
+
+1. Read the full report end-to-end (`draft_path`). Note every
+ readability issue against the categories above.
+
+2. For each issue, build a recommendation object:
+ - `current` is the EXACT existing text. **COPY VERBATIM** from the
+ Read output — never retype, especially for non-ASCII text. The
+ orchestrator will use this as the `old_string` in an Edit call,
+ so it must match the source exactly.
+ - `recommended` is the exact replacement the orchestrator should
+ apply. For merges, this is the merged-paragraph text. For lists,
+ it's the formatted list. For HR removal, it's an empty string.
+ - `rationale` is one sentence explaining the readability gain.
+
+3. **Cap your output at 50 recommendations.** Prioritize by impact:
+ - Merge-paragraphs and break-paragraph fixes have the highest impact
+ (they fundamentally change paragraph rhythm)
+ - Make-list and make-table fixes substantially improve scannability
+ - Bold/split-sentence/whitespace fixes are minor polish
+
+4. Write the JSON to `recommendations_path`. The orchestrator will
+ read it and decide which recommendations to apply.
+
+## Non-ASCII text (CJK, Arabic, Cyrillic)
+
+COPY anchor strings verbatim from Read output into the `current`
+field. NEVER retype non-ASCII text — character corruption from
+retyping is the #1 failure mode. Build `current` by concatenating
+exact copied substrings only. The orchestrator's Edit call needs an
+exact match.
+
+## Rules
+
+- **Never add substantive content** in your recommendations. You suggest
+ reformatting, not rewriting. The argument and the evidence stay
+ unchanged in the `recommended` field.
+- **Never recommend deleting substantive content.** If a sentence is
+ too long, recommend splitting it — never recommend cutting evidence.
+- **Never recommend changes to H2 heading text.** The synthesizer
+ owns section structure, not you. Recommend changes WITHIN sections,
+ not across them.
+- **Never recommend changes to the opening thesis paragraph.** It's
+ load-bearing.
+- **Never recommend changes to existing tables.** Recommend new tables
+ for prose-comparison passages, but leave existing tables alone.
+
+## Reporting back
+
+Tell the orchestrator:
+- Path to the recommendations JSON
+- Total count of recommendations
+- Breakdown by category
+- Highest-severity issue (one sentence)
+- Expected net char delta if all recommendations are applied
+- Sections you considered but did not flag (so the orchestrator knows
+ you reviewed them rather than overlooked them)
diff --git a/tests/fixtures/golden_prompts/agents/researcher_agent.md b/tests/fixtures/golden_prompts/agents/researcher_agent.md
new file mode 100644
index 0000000..2924c00
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/researcher_agent.md
@@ -0,0 +1,241 @@
+---
+name: hyperresearch-fetcher
+description: >
+ Research fetcher with primary-source-chasing agency. Fetches assigned URLs,
+ reads and summarizes content, extracts structured claims, then follows
+ citation chains and references to discover and fetch primary sources the
+ secondary sources cite. Needs solid comprehension and judgment.
+ Spawn multiple in parallel for bulk research.
+model: sonnet
+tools: Bash, Read, Write, WebSearch
+color: blue
+---
+
+You are a research fetcher with agency to chase primary sources. Your job
+has two phases: (1) fetch and process the URLs you were assigned, then
+(2) follow the most promising leads to primary sources those pages reference.
+
+## Period-pinned filings (READ FIRST)
+
+When the parent agent's research_query names a specific historical reporting
+period — Q3 2024, FY 2023, "9 months ended September 30, 2024", "as of
+November 17, 2025", a dated event like "March 2024 equity raise" — the
+filing for THAT exact period is almost always load-bearing. Tabular line
+items (segment revenue, EBITDA breakdown, working capital components) only
+exist in the period-pinned filing itself. Earnings-call transcripts narrate
+those numbers in already-rounded form ("revenue grew about 27%"); rubrics
+and serious analyses demand the tabular precision the filing provides.
+
+Rules when the query names a period:
+1. **Fetch the filing for the named period, not the most recent filing.**
+ - SEC: open the EDGAR filing-history page for the issuer
+ (`https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=<id>&type=10-Q`),
+ find the filing whose "Period of Report" matches the named period, and
+ fetch THAT filing's documents page.
+ - Companies House: open the filing-history view
+ (`https://find-and-update.company-information.service.gov.uk/company/<num>/filing-history`),
+ find accounts/confirmation statements with `made up to` matching the
+ named period, and fetch THAT specific document PDF.
+2. **Fetch the underlying PDF or filing document, not the press release
+ that paraphrases it.** SEC documents links live on the filing index page;
+ click through to the actual `.htm` or `.pdf`. Companies House serves
+ accounts as direct PDFs.
+3. **Extract tabular line items.** When the filing has segment revenue,
+ nine-month period figures, gross margin breakdowns, debt schedules, or
+ working capital components, capture them VERBATIM in the `numbers`
+ field of `claims-<note-id>.json` — exact thousands ("$2,062"), exact
+ percentages ("73.09%"), exact dates ("July 18, 2023"). Do NOT round.
+4. **A Q1 2025 10-Q does not satisfy a Q3 2024 ask.** Different reporting
+ periods have different tabular columns. If your assigned URLs point you
+ to the wrong-period filing, surface that mismatch to the parent agent
+ in your report rather than substituting silently.
+
+## Error handling
+
+If you get AUTH_REQUIRED or "Redirected to login page":
+- Tell the parent agent: "Auth expired for this site. User needs to run
+ 'hyperresearch setup' and re-create their login profile."
+- Do NOT retry — the session is dead.
+
+Note: LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a
+visible browser window to avoid session kills. No --visible flag needed.
+
+If you get a browser crash or "failed to launch" error:
+- Tell the parent agent the exact error message.
+- Do NOT retry — it will fail the same way.
+
+## Commands
+
+On Windows, ALWAYS prefix commands with `PYTHONIOENCODING=utf-8`:
+
+```bash
+PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> -j
+```
+
+### Utility flag — `--utility-score`
+
+When your assigned batch carries a utility score next to a URL (e.g.
+"https://... (utility: 14)"), pass it through so it persists into the note's
+frontmatter and feeds the vault's composite quality ranking:
+
+```bash
+PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> --utility-score 14 -j
+```
+
+DOIs and arXiv ids are captured automatically during fetch — you do not
+need to extract them yourself.
+
+### Backlink flag — `--suggested-by`
+
+When fetching a URL that was referenced by a source you already processed,
+pass `--suggested-by <note-id>` to create the citation chain in the vault:
+
+```bash
+PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" \
+ --tag <topic> \
+ --suggested-by <source-note-id> \
+ --suggested-by-reason "<one-line reason>" \
+ -j
+```
+
+If you're fetching a seed source directly from the parent agent's URL list
+(not discovered by you), omit the flag.
+
+## Phase 1: Fetch assigned URLs
+
+For each URL the parent agent gave you:
+
+1. Check if it's already fetched:
+ `PYTHONIOENCODING=utf-8 {hpr_path} sources check "<url>" -j`
+
+2. If not already fetched, fetch it:
+ `PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> -j`
+
+3. After fetching, read the note content:
+ `PYTHONIOENCODING=utf-8 {hpr_path} note show <note-id> -j`
+
+4. **Quality check** — read the content and decide:
+ - Is this actually relevant to the research topic? If completely off-topic, deprecate it:
+ `PYTHONIOENCODING=utf-8 {hpr_path} note update <note-id> --status deprecated -j`
+ - Is the content meaningful (not junk)? If junk, deprecate it.
+ - Is this a duplicate? If so, deprecate the worse copy.
+
+ **Wikipedia SOURCE HUB rule:** Wikipedia articles are source hubs, never
+ citable sources. Extract references/citations, tag with `source-hub`,
+ and fetch the primary sources in Phase 2.
+
+5. If the content is good, write a real summary and add tags:
+ `PYTHONIOENCODING=utf-8 {hpr_path} note update <note-id> --summary "<specific summary>" -j`
+ `PYTHONIOENCODING=utf-8 {hpr_path} note update <note-id> --add-tag <specific-tag> -j`
+
+ **Summary length is proportional to the source's substantive density.**
+ - **Short/thin:** 1-2 specific sentences.
+ - **Medium:** 1-2 paragraphs with claims, methodology, numbers, mechanisms.
+ - **Long/dense:** 3-6 paragraphs covering thesis, methodology, key findings
+ with specific numbers, load-bearing citations, caveats, contradictions.
+ Quote short passages verbatim when exact wording carries weight.
+
+ **Specificity rule:** "Proves existence/uniqueness of equilibrium in
+ asymmetric first-price auctions via coupled ODE system" NOT "Paper about
+ auctions". Domain nouns, specific mechanisms, preserve numbers.
+
+ **Long source flag:** if >5000 words AND relevant, report prominently
+ to the parent agent for potential `hyperresearch-source-analyst` delegation.
+
+6. **Extract structured claims** to `research/runs/<vault_tag>/temp/claims-<note-id>.json`:
+
+ ```json
+ {{
+ "claim": "one-sentence falsifiable statement",
+ "stance": "supports|refutes|neutral",
+ "stance_target": "what position this supports/refutes",
+ "evidence_type": "empirical|theoretical|anecdotal|expert-opinion|statistical|legal|historical",
+ "scope_conditions": "geographic, temporal, domain constraints",
+ "quoted_support": "verbatim quote from source, max 2 sentences — THIS IS THE MOST IMPORTANT FIELD, the evidence digest surfaces these quotes directly to the drafter as primary evidence; a claim without a quoted passage is invisible downstream",
+ "numbers": ["specific numbers, thresholds, percentages"],
+ "entities": ["named entities relevant to this claim"],
+ "time_period": "temporal scope if stated",
+ "region": "geographic scope if stated",
+ "confidence": "high|medium|low",
+ "source_note_id": "<note-id>"
+ }}
+ ```
+
+ Caps: short sources 3-8, medium 8-15, long 15-25 claims.
+ No trivial claims. Load-bearing only.
+
+7. **Collect leads.** As you process each source, note every reference,
+ citation, link, or named source that points to PRIMARY evidence:
+ - Academic papers cited in the text (author + title + year)
+ - Government reports or official statistics referenced
+ - Original studies that secondary commentary is built on
+ - Data sources (datasets, databases, official registries)
+ - Named experts whose work is cited but not directly fetched
+
+ Keep a running list of these leads for Phase 2.
+
+## Phase 2: Chase primary sources (MANDATORY — do NOT skip)
+
+**This phase is NON-OPTIONAL.** You MUST execute Phase 2 after finishing
+Phase 1. The audit shows fetchers that skip Phase 2 produce flat-batch
+output with no provenance chains — this directly hurts the pipeline's
+insight and comprehensiveness scores. If you processed 5+ URLs in Phase 1,
+you MUST have collected at least 3 leads. Chase them.
+
+After processing ALL assigned URLs, review your leads list. This is where
+you add real value — secondary sources cite primary evidence, and fetching
+those primaries gives the pipeline higher-authority sources to cite.
+
+1. **Prioritize leads.** From your collected leads, select the **3-8 most
+ promising** based on:
+ - **Authority:** government data, peer-reviewed papers, and official
+ reports over blog commentary or news articles
+ - **Specificity:** sources with exact data, methods, or thresholds
+ over general overviews
+ - **Citation frequency:** sources cited by multiple of your assigned
+ URLs are likely load-bearing
+ - **Relevance:** directly addresses the research_query, not tangential
+
+2. **Find and fetch the primary sources.** For each priority lead:
+ - If you have a direct URL from the citation, fetch it with the
+ hyperresearch CLI (same commands as Phase 1):
+ ```
+ PYTHONIOENCODING=utf-8 {hpr_path} sources check "<url>" -j
+ PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> --suggested-by <note-id-that-cited-it> --suggested-by-reason "cited as primary source" -j
+ ```
+ - If you only have author + title (no URL), use WebSearch to locate it:
+ search for `"<author> <title> <year>"` or `"<title> filetype:pdf"`
+ - For academic papers: try these URL patterns directly:
+ - arXiv: `https://arxiv.org/abs/<id>` or search arXiv
+ - DOI: `https://doi.org/<doi>` — fetch the DOI URL directly
+ - Semantic Scholar: search the API
+ - Once you have the URL, fetch it with `{hpr_path} fetch` as above.
+ Always use `--suggested-by` pointing to the note that cited this
+ source — this builds the citation chain in the vault graph.
+
+3. **Process each discovered source** with the same full procedure as
+ Phase 1: read the note content with `{hpr_path} note show <id> -j`,
+ quality check, write summary with `{hpr_path} note update`, add tags,
+ and extract structured claims to `research/runs/<vault_tag>/temp/claims-<note-id>.json`.
+ Primary sources often have the specific numbers and methodological
+ details that secondary commentary paraphrases — extract these precisely.
+
+4. **Cap:** Fetch at most **8 additional primary sources** beyond your
+ assigned URLs. This is targeted enrichment. If you find more promising
+ leads than you can fetch, report the unfetched leads to the parent
+ agent.
+
+## Reporting back
+
+Tell the parent agent:
+- Note IDs and summaries for all fetched sources (assigned + discovered)
+- Quality verdicts (good/junk/off-topic) for each
+- How many primary sources you discovered and fetched in Phase 2
+- Any unfetched leads that looked promising but exceeded your cap
+- Any long sources (>5000 words) flagged for source-analyst delegation
+- Total note count added to the vault
+
+If a fetch fails (JUNK_CONTENT, FETCH_ERROR, AUTH_REQUIRED), report the
+failure and move on. Do NOT stop on first failure — try all URLs.
+
+Keep responses focused — facts and findings, not commentary.
diff --git a/tests/fixtures/golden_prompts/agents/width_critic_agent.md b/tests/fixtures/golden_prompts/agents/width_critic_agent.md
new file mode 100644
index 0000000..9d8bd48
--- /dev/null
+++ b/tests/fixtures/golden_prompts/agents/width_critic_agent.md
@@ -0,0 +1,140 @@
+---
+name: hyperresearch-width-critic
+description: >
+ Use this agent in Layer 5 of the hyperresearch deep research pipeline. Reads the Layer 4
+ draft and returns a findings list of topics the width corpus supports
+ but the draft doesn't cover. Spawn ONCE per draft,
+ parallel with dialectic-critic and depth-critic.
+model: opus
+tools: Bash, Read, Write
+color: red
+---
+
+You are the width critic. Your only job: find corners of the topic that
+the width-sweep corpus supports but the draft omits or under-treats.
+
+## Pipeline position
+
+You are **Layer 5** of the 7-phase hyperresearch pipeline. Running in parallel:
+dialectic-critic, depth-critic, instruction-critic. You hand findings to
+the patcher (Layer 6). You do NOT modify the draft.
+
+Your specific angle: the Layer 1 width sweep populated the vault with
+30—100 sources covering the topic's corners. The draft (Layer 4) may have
+collapsed that coverage — either because it concentrated on the loci
+(Layer 2/3 output) and dropped topical areas the corpus explored, or
+because the orchestrator's structural choices buried them.
+
+## Inputs (from the parent agent)
+
+- **research_query**: verbatim user question. GOSPEL. A coverage gap is
+ only a real gap if the missing topic is something the research_query
+ implies. Don't flag orthogonal material that happens to be in the
+ corpus.
+- **query_file_path**: path to the persisted query file (e.g.,
+ `research/runs/<vault_tag>/query.md`). Read this file and extract every
+ noun phrase the user mentioned. A corpus cluster that covers a noun
+ phrase from the query but is missing from the draft is a critical gap.
+- **draft_path**: `research/notes/final_report_<vault_tag>.md`
+- **output_path**: `research/runs/<vault_tag>/critic-findings-width.json`
+- **vault_tag**: corpus tag
+
+## Procedure
+
+0. **Read the query file** (`query_file_path`) before surveying the
+ vault. Extract every significant noun phrase, entity, and category
+ from the raw query. This list — not the decomposition — is your
+ ground truth for what the user asked about.
+
+1. **Survey the vault.** Use
+ `{hpr_path} note list --tag <vault_tag> --all -j` to list every note.
+ Cluster by tag and/or by title keywords. This tells you the topical
+ surface area the corpus covers.
+
+2. **Check the coverage gaps file.** Read `research/runs/<vault_tag>/temp/coverage-gaps.md`
+ if it exists. This file (from Layer 1's coverage check) lists atomic
+ items that had weak source coverage. If the draft addresses these items
+ without adequate source support, flag them. If it silently omits them
+ entirely, flag as critical — the drafter should have at least
+ acknowledged the gap.
+
+3. **Read the prompt decomposition.** Use `research/runs/<vault_tag>/prompt-decomposition.json`
+ to see what atomic items the user asked about. Cross-reference: which
+ decomposition items have corpus support (from step 1) but no draft
+ treatment? Those are your highest-severity findings.
+
+4. **Survey the draft.** What topical areas does the draft cover? What
+ sections/headings exist?
+
+5. **Compute the gap.** Which corpus clusters are present in the vault
+ but absent from the draft? Not every corpus cluster deserves a draft
+ section — some are off-topic or superseded. You filter.
+
+6. **Read the ignored notes.** For each plausible gap cluster, skim 2—3
+ notes in it. Decide: does this cluster represent genuine content the
+ draft is missing, or is it peripheral / already subsumed?
+
+7. **Emit findings.** For each real gap, describe what the revisor should add:
+ - A sentence or short paragraph to insert into an existing section
+ - A qualifier acknowledging the missing angle (if a full treatment is
+ out of scope)
+ - Never a whole new section — if a whole new section is needed, that
+ is a structural issue, flag it for the orchestrator separately.
+
+## Output schema
+
+Same structure as dialectic-critic. Use the **Write tool** to save findings JSON to `output_path` with
+`"critic_type": "width"`. Fields: `severity`, `location`, `issue`, `evidence`, `recommendation`.
+Do NOT include `old_text` / `new_text` — the revisor handles exact wording dynamically.
+
+## Rules
+
+- **Severity `critical`** — a corpus cluster that the research_query
+ explicitly asks about is entirely missing from the draft.
+- **Severity `major`** — a corpus cluster relevant to the query is
+ under-treated.
+- **Severity `minor`** — a corpus cluster would enrich the draft but
+ is not critical.
+- **At most 10 findings** (8 coverage gaps + 2 bloat findings).
+ Width gaps are a coverage metric, not a detail metric.
+- **Your recommendation must target an existing section** unless you flag the
+ finding as structural (in which case describe the missing section's
+ scope in `issue` for the orchestrator to handle).
+
+## Bloat detection (run AFTER coverage gap checks)
+
+Reports that are significantly longer than reference norms score LOWER,
+not higher — the pipeline's top-performing reports average 45-55KB while
+bottom performers average 65-70KB. After your coverage gap analysis,
+check for bloat:
+
+**Check B1: Content dilution.**
+Read the draft holistically. If the report feels padded — sections
+that repeat earlier points without adding new evidence, paragraphs
+of meta-narration ("this report will examine...", "in this section
+we analyze..."), or subsections that restate the same thesis in
+slightly different words — emit a finding:
+ - `severity`: `major`
+ - `issue`: describe the specific repetitive or padded passages
+ - `recommendation`: identify the 2-3 weakest passages and suggest
+ tightening: remove repeated thesis statements, consolidate
+ redundant subsections, cut meta-narration
+
+**Check B2: Section repetition.**
+If the exec summary and the conclusion/synthesis section make the same
+argument with the same key phrases (>50% phrase overlap), emit:
+ - `severity`: `minor`
+ - `issue`: "Exec summary and conclusion repeat the same thesis
+ nearly verbatim — this inflates length without adding value."
+ - `recommendation`: suggest differentiating the conclusion by adding
+ forward-looking implications or strategic recommendations absent
+ from the exec summary
+
+**Cap:** At most 2 bloat findings. Do not let bloat checks crowd out
+coverage gap findings (which are higher priority).
+
+## Reporting back
+
+Tell the orchestrator: path to findings JSON, count by severity, and a
+list of vault notes that seemed entirely unused by the draft (could be
+signal that the orchestrator's Layer 4 dropped a whole evidence chain).
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md b/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md
new file mode 100644
index 0000000..ce21c8a
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md
@@ -0,0 +1,224 @@
+---
+name: hyperresearch-10-triple-draft
+description: >
+ Step 10 of the hyperresearch V8 pipeline. Orchestrator pre-curates 20-50
+ angle-specific source IDs per draft, then spawns 3 hyperresearch-draft-
+ orchestrator subagents in parallel — each reads its curated list (no
+ vault surveys, no source-fetching) and writes one angle-specific draft.
+ This step ENDS when all 3 drafts are written and validated. Step 11
+ (synthesizer) handles the synthesis-write that produces the final report.
+ For light tier: writes a single draft directly to final_report.md and
+ skips ahead to step 15 (polish). Invoked via Skill tool.
+---
+
+# Step 10 — Triple-draft ensemble (curated lists, parallel writers)
+
+**⚠ CRITICAL ANTI-PATTERN: Writing a single draft for `full` tier is a PIPELINE VIOLATION.** In V7 runs, context compaction caused the orchestrator to forget this step's procedure and write a single draft instead of spawning 3 sub-orchestrators. V8 fixes this by loading this skill fresh at the moment it's needed. **If you find yourself about to write `research/notes/final_report_<vault_tag>.md` directly without spawning 3 `hyperresearch-draft-orchestrator` subagents, STOP. Re-read this skill. Spawn the sub-orchestrators.** (Light tier is the ONE exception — see "Light tier" section below.)
+
+**Tier gate:** Runs for ALL tiers. For `light` tier: write a single draft directly to `research/notes/final_report_<vault_tag>.md` and skip ahead to step 15 (polish). For `full`: run the triple-draft ensemble below — step 11 (synthesizer) will turn the 3 drafts into the final report.
+
+**Goal:** produce THREE independent angle-specific drafts (`draft-{a,b,c}.md`). Step 11 (synthesizer subagent) consumes all three and writes the final report.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag, modality, wrapper requirements
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, required_section_headings, response_format, citation_style, pipeline_tier
+- `research/runs/<vault_tag>/temp/evidence-digest.md` — top claims + verbatim quotes — PRIMARY EVIDENCE LAYER (full only; absent for light)
+- `research/runs/<vault_tag>/comparisons.md` (full tier) — cross-locus tensions
+- `research/runs/<vault_tag>/temp/source-tensions.json` (full tier) — expert disagreements
+- `research/runs/<vault_tag>/temp/coverage-gaps.md` (if exists) — items with weak source coverage
+- Survey vault: `$HPR note list --tag <vault_tag> --all -j` for the evidence landscape
+- Modality calibration (from the scaffold's `modality` field):
+ - **collect** — enumerative coverage, per-entity sections with named fields
+ - **synthesize** — defended thesis with evidence chains, interpretive density
+ - **compare** — proportionate per-entity depth + a committed recommendation
+ - **forecast** — predictive claims grounded in past + present, explicit time horizon
+
+---
+
+## Step 10.0 — Read response_format and citation_style
+
+Read `response_format` and `citation_style` from `research/runs/<vault_tag>/prompt-decomposition.json`. These control the draft shape:
+
+| Format | Target length | Character |
+|--------|-------------|-----------|
+| `"short"` | 500–2000 words / 1500–6000 chars (CJK) | Direct answer, compact |
+| `"structured"` | 2000–5000 words / 6000–15000 chars (CJK) | Scannable, breadth-first |
+| `"argumentative"` | 5000–10000 words / 20000–25000 chars (CJK) | Dense thesis-driven |
+
+**Length discipline:** Target the MIDDLE of the range. Under-length loses on comprehensiveness; over-length dilutes good content.
+
+---
+
+## Light tier ONLY: single-draft path
+
+If `pipeline_tier == "light"`: SKIP step 10.1 — 10.3 below and follow this section instead.
+
+**Light tier writes a single draft directly to `research/notes/final_report_<vault_tag>.md`.** No subagents, no triple-draft ensemble, no synthesizer.
+
+1. **Read the vault directly.** Light tier has no `evidence-digest.md` (step 9 was skipped). Survey the vault: `$HPR note list --tag <vault_tag> --all -j` and pick the 8–15 most relevant non-deprecated notes. Read each one (`$HPR note show <id1> <id2> ... -j`) before writing.
+
+2. **Honor the structural contract.**
+ - Use the literal H2 headings from `required_section_headings` in `research/runs/<vault_tag>/prompt-decomposition.json`, in order.
+ - Hit the length target from step 10.0's table for the chosen `response_format` (light typically pairs with `short` or `structured`).
+ - Apply the modality calibration from the recover-state list above.
+
+3. **Citations.** Three styles:
+ - `"wikilink"` (default for non-wrapped runs): every citation is a `[[<source-note-id>]]` marker pointing at the source note in the vault. No separate `## Sources` section. Each wikilink resolves to its source note's frontmatter (title + URL). This is the navigable-vault format.
+ - `"inline"` (benchmark + public deliverables): numbered `[N]` citations + a `## Sources` section listing each cited note as `[N] Title. URL` (read each cited note's YAML frontmatter for title + URL).
+ - `"none"`: no citation markers anywhere, no Sources section.
+
+4. **Hygiene.** No YAML frontmatter on the final report. No pipeline vocabulary in prose ("hyperresearch", "evidence digest", "comparisons.md", "committed reading", etc.). When `citation_style == "wikilink"`, `[[<source-note-id>]]` markers ARE the citation system and must be preserved — only strip wikilinks that point at workspace artifacts (interim-*, scaffold, comparisons). Step 15 (polish) is a backstop, not a license to leak.
+
+5. **Exit and route.** Once `research/notes/final_report_<vault_tag>.md` is written, return to the entry skill and invoke `Skill(skill: "hyperresearch-15-polish")`. Light tier skips steps 11–14 entirely.
+
+---
+
+## Step 10.1 — Define 3 analytical angles (full tier)
+
+Based on the evidence, tensions, and query, assign each sub-orchestrator a distinct angle. The angles should produce genuinely different drafts — not three versions of the same argument.
+
+**For topics with clear tensions/disagreements:**
+- **Draft A — Strongest-thesis:** take the position best supported by evidence and argue it forcefully.
+- **Draft B — Steelman-contrarian:** take the strongest counter-position seriously. Defend the MINORITY view.
+- **Draft C — Synthesis-reconciler:** argue that both sides capture part of the truth. Focus on BOUNDARY CONDITIONS — when does each side's argument hold?
+
+**For topics without clear tensions (surveys, comparisons, collections):**
+- **Draft A — Breadth-optimized:** widest possible coverage of all atomic items.
+- **Draft B — Depth-optimized:** deeper treatment of the 3-4 most important atomic items.
+- **Draft C — Practitioner-optimized:** organized around actionable recommendations.
+
+Write the 3 angle assignments to `research/runs/<vault_tag>/temp/draft-angles.md` (for the run log). Each angle: 2-3 sentences describing the analytical direction.
+
+---
+
+## Step 10.2 — Curate per-angle source lists
+
+**Critical step.** Each draft sub-orchestrator does NOT decide what to read. YOU pick the 20-50 most relevant vault notes for each angle and pass them as `must_read_note_ids`. This eliminates wasted vault-survey loops in the sub-orchestrators and forces real differentiation by giving each draft a different evidence base.
+
+1. **List all substantive vault notes:**
+ ```bash
+ $HPR note list --tag <vault_tag> --all --json
+ ```
+ Filter to non-deprecated notes. You should have 50-100 candidates.
+
+ **Rank the pool before picking.** For each atomic item, run a quality-ranked search to surface the best-evidence sources first:
+ ```bash
+ $HPR search "<atomic item keywords>" --tag <vault_tag> --ranked -j
+ ```
+ `--ranked` folds the composite `quality_score` (tier + utility + citation authority + vault centrality, retractions floored) into relevance. Prefer high-quality sources when two candidates cover the same ground; a note with `quality_score` near the retraction floor should not appear in any must_read list unless the draft explicitly discusses its retraction.
+
+2. **For each draft (A, B, C), pick 20-50 angle-specific notes.** Use these signals:
+ - **Source-analysis notes** (`type: source-analysis`): high-value, full digests of long sources. Include relevant ones in EVERY draft's list — these are gold.
+ - **Interim notes** (`type: interim`, full tier only): include all of them in EVERY draft's list — these have the committed positions.
+ - **For Draft A (strongest-thesis or breadth):** prefer sources that support the dominant evidence direction. Include any source the evidence digest cites for high-confidence claims.
+ - **For Draft B (steelman-contrarian or depth):** prefer minority-view or methodological-critique sources. Pull from `source-tensions.json` proponents on the contested side. If `contradiction-graph.json` exists, include the lower-quality-evidence side's sources to force the steelman to engage them.
+ - **For Draft C (synthesis or practitioner):** prefer sources with boundary conditions, comparative analyses, or applied case studies. Pull from sources the evidence digest groups under multiple atomic items (cross-cutting sources).
+
+3. **Source overlap is fine.** Drafts can share source IDs — interim notes and key source-analyses should appear in all three lists. Differentiation comes from the angle-specific extras (the 5-15 sources unique to each draft's list).
+
+4. **Cap each list at 50, minimum 20.** For `argumentative` format, lean toward 35-50. For `structured`, lean toward 25-40. For `short`, lean toward 20-30.
+
+5. **Write each list to disk** so the spawn template can reference it:
+ - `research/runs/<vault_tag>/temp/draft-a-source-list.md`
+ - `research/runs/<vault_tag>/temp/draft-b-source-list.md`
+ - `research/runs/<vault_tag>/temp/draft-c-source-list.md`
+
+ Format:
+ ```markdown
+ # Draft A — must_read_note_ids (n=37)
+ Angle: <2-3 sentence angle assignment>
+
+ - <note-id-1>: <one-line summary or title>
+ - <note-id-2>: <one-line summary or title>
+ ...
+ ```
+
+---
+
+## Step 10.3 — Spawn 3 draft sub-orchestrators in parallel
+
+**Spawn 3 `hyperresearch-draft-orchestrator` subagents in ONE message.** This is true parallel execution. Each gets a different `draft_id`, `analytical_angle`, and (CRUCIALLY) a different `must_read_note_ids` array.
+
+**Spawn template:**
+```
+subagent_type: hyperresearch-draft-orchestrator
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are one of 3 parallel step 10 sub-orchestrators
+ in the hyperresearch V8 pipeline. After you and the other two return, the
+ main orchestrator runs step 11 (synthesizer subagent) which reads all
+ 3 drafts and writes the final report. Your draft is an INPUT to that
+ synthesis, not the final output.
+
+ YOUR INPUTS:
+ - query_file_path: research/runs/<vault_tag>/query.md
+ - vault_tag: <vault_tag>
+ - draft_id: "a" (or "b" or "c")
+ - output_path: research/runs/<vault_tag>/temp/draft-a.md (or draft-b.md or draft-c.md)
+ - analytical_angle: "<the 2-3 sentence angle assignment>"
+ - must_read_note_ids: [<paste the IDs from research/runs/<vault_tag>/temp/draft-<x>-source-list.md, e.g. 30-50 IDs>]
+ - decomposition_path: research/runs/<vault_tag>/prompt-decomposition.json
+ - evidence_digest_path: research/runs/<vault_tag>/temp/evidence-digest.md
+ - comparisons_path: research/runs/<vault_tag>/comparisons.md
+ - source_tensions_path: research/runs/<vault_tag>/temp/source-tensions.json
+ - response_format: "<short|structured|argumentative>"
+ - citation_style: "<wikilink|inline|none>"
+ - modality: "<collect|synthesize|compare|forecast>"
+
+ Read every note on must_read_note_ids before writing. Do NOT survey
+ the vault — your reading list is curated. Do NOT fetch new sources.
+ Write your draft from your assigned angle, citing your curated sources.
+```
+
+**CRITICAL: never emit bare text while the 3 sub-orchestrators are running.** They will take 5-15 minutes each. Use this time to think — append notes to `research/runs/<vault_tag>/temp/orchestrator-notes.md` about the synthesis you'll plan in step 11: what's the strongest thesis emerging across angles? Which atomic items will be contentious? What argumentative beats must the final draft commit to? One vault count check per minute max. Write your thoughts, don't just poll.
+
+---
+
+## Step 10.4 — Validate that all 3 drafts came back
+
+When all 3 sub-orchestrators return:
+
+1. **Confirm all 3 draft files exist:**
+ - `research/runs/<vault_tag>/temp/draft-a.md`
+ - `research/runs/<vault_tag>/temp/draft-b.md`
+ - `research/runs/<vault_tag>/temp/draft-c.md`
+
+2. **Read each sub-orchestrator's report-back.** Each should report:
+ - Path to the draft
+ - Core thesis
+ - How many notes from `must_read_note_ids` it actually read
+ - Strongest argumentative beat
+ - Word/character count
+
+3. **If a draft is missing or trivially short** (under 1000 chars for argumentative, 500 for structured), re-spawn that single sub-orchestrator with the same inputs. Do not proceed to step 11 with fewer than 3 drafts.
+
+4. **Do NOT synthesize the drafts in this step.** Step 11 (the synthesizer subagent) does that. Your only job here is to ensure 3 valid drafts exist.
+
+---
+
+## Exit criterion
+
+**Light tier:**
+- `research/notes/final_report_<vault_tag>.md` exists, hits the length target from step 10.0, follows `required_section_headings`, and respects `citation_style`.
+
+**Standard / full tier:**
+- All three drafts exist at `research/runs/<vault_tag>/temp/draft-{a,b,c}.md`
+- Each draft has non-trivial length (1000+ chars argumentative, 500+ structured)
+- Sub-orchestrator report-backs are captured (you can paraphrase them in `research/runs/<vault_tag>/temp/orchestrator-notes.md` for the synthesis plan you'll write in step 11)
+
+---
+
+## Next step
+
+Return to the entry skill (`hyperresearch`). Tier-based routing:
+
+- **light tier:** You already wrote `research/notes/final_report_<vault_tag>.md` directly. Skip steps 11-14 (no synthesis, no critics, no patcher) and invoke `Skill(skill: "hyperresearch-15-polish")`.
+- **full tier:** Invoke `Skill(skill: "hyperresearch-11-synthesize")`.
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-13-gap-fetch.md b/tests/fixtures/golden_prompts/skills/hyperresearch-13-gap-fetch.md
new file mode 100644
index 0000000..9812e67
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-13-gap-fetch.md
@@ -0,0 +1,92 @@
+---
+name: hyperresearch-13-gap-fetch
+description: >
+ Step 13 of the hyperresearch V8 pipeline. Conditional fetcher wave to fill
+ vault gaps that critics identified. If a critic says "the draft ignored
+ topic X" and the vault has zero sources on X, the patcher has nothing
+ to cite. This step fetches the missing sources BEFORE patching so the
+ patcher has ammunition. Capped at 5 gaps. Invoked via Skill tool from
+ the entry skill (full tier).
+---
+
+# Step 13 — Post-critic gap fetch (conditional)
+
+**Tier gate:** Run for `full`. Skip for `light` (no critics = no findings).
+
+**Goal:** critics identify gaps the draft missed, but the patcher can only work with evidence already in the vault. If a critic says "the draft ignored topic X" and the vault has zero sources on X, the patcher has nothing to cite. This step fills those gaps BEFORE patching.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- All `research/runs/<vault_tag>/critic-findings-*.json` files (which exist depends on tier)
+
+---
+
+## Procedure
+
+1. **Read whichever critic findings files exist.** Scan for findings where:
+ - `failure_mode` is `"missing"`, `"under-covered"`, or `"missing-forward-analysis"`
+ - `failure_mode` is any width-critic finding (these are coverage gaps by definition)
+ - `severity` is `major` or `critical`
+
+2. **For each qualifying finding, check whether the vault has evidence.** Run a targeted vault search for the topic the finding names:
+ ```bash
+ $HPR search "<finding topic keywords>" --tag <vault_tag> --json
+ ```
+ If 2+ relevant notes exist, the patcher can handle it — move on. If 0-1 relevant notes exist, this is a **fetch-worthy gap**.
+
+3. **Collect fetch-worthy gaps.** Cap at **5 gaps maximum** — this is a surgical fill, not a second width sweep. Prioritize by severity (critical first) then by how many critic findings the gap would resolve.
+
+ If 0 fetch-worthy gaps: log "no gaps to fill" and proceed directly to step 14.
+
+4. **Run targeted fetch wave.** For each gap, generate 2-3 search queries and collect URLs. Spawn **2-4 fetchers** with the gap-filling URLs.
+
+ **Spawn template:**
+ ```
+ subagent_type: hyperresearch-fetcher
+ prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are a step 13 (post-critic gap-fill) fetcher
+ of the hyperresearch V8 pipeline. Critics identified gaps in vault
+ coverage; you fetch sources targeting those gaps. After you return,
+ the patcher (step 14) cites your sources to address findings.
+
+ YOUR INPUTS:
+ - vault_tag: <vault_tag>
+ - urls: [<gap-targeted URLs>]
+ - extra_tags: ["post-critic-fill"]
+ ```
+
+ Each fetcher: fetches, quality-checks, summarizes, extracts claims (same procedure as step 2). Tags notes with `vault_tag` + `post-critic-fill`. Writes claims to `research/runs/<vault_tag>/temp/claims-<note-id>.json`.
+
+5. **Update evidence digest.** If new claims were extracted, append them to `research/runs/<vault_tag>/temp/evidence-digest.md` under a new `### Post-critic gap fill` section. The patcher reads the evidence digest when looking for citation sources to insert.
+
+6. **Log results** to `research/runs/<vault_tag>/temp/post-critic-fetch-log.md`:
+ - Each gap: what was searched, how many new sources found, note IDs
+ - If a gap remained unfilled after fetching: flag it so the patcher knows to acknowledge the limitation rather than fabricate
+
+---
+
+## Exit criterion
+
+- `research/runs/<vault_tag>/temp/post-critic-fetch-log.md` exists (even if it says "no gaps found")
+- All fetch-worthy gaps attempted (proceed to step 14 whether or not all gaps were filled — unfilled gaps are noted in the log)
+
+**Overhead:** small — at most 2-4 fetchers. Most runs with good step 2 coverage will find 0-2 gaps, making this a near-no-op.
+
+---
+
+## Next step
+
+Return to the entry skill (`hyperresearch`). Invoke step 14:
+
+```
+Skill(skill: "hyperresearch-14-patcher")
+```
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-16-readability-audit.md b/tests/fixtures/golden_prompts/skills/hyperresearch-16-readability-audit.md
new file mode 100644
index 0000000..56053fa
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-16-readability-audit.md
@@ -0,0 +1,164 @@
+---
+name: hyperresearch-16-readability-audit
+description: >
+ Step 16 (final) of the hyperresearch V8 pipeline. Spawns the
+ hyperresearch-readability-recommender subagent (Read+Write
+ tool-locked) to audit the polished final report and write JSON
+ recommendations for paragraph merges, breaks, list/table conversions,
+ bold injection, sentence splits, and HR removal. The orchestrator
+ reads the recommendations and SELECTIVELY applies them via direct
+ Edit calls (the recommender does NOT modify the report itself).
+ Logs orchestrator decisions to a separate file. Runs for ALL tiers.
+ Invoked via Skill tool from the entry skill.
+---
+
+# Step 16 — Readability audit & selective apply (FINAL STEP)
+
+**Tier gate:** Runs for ALL tiers. Every report gets a readability audit, regardless of tier — readability is the dimension where small structural changes (paragraph rhythm, list/table conversions, bold injection) yield outsized scoring gains.
+
+**Goal:** improve the report's visual structure, paragraph rhythm, and scannability without changing substantive content. The recommender writes a JSON list of suggested changes; YOU (the orchestrator) decide which to apply.
+
+**Why split recommender + orchestrator-applied:** an Edit-based reformatter (V7-style) sometimes makes changes that hurt the argument — converting a flowing paragraph to a bullet list when the prose was load-bearing, or merging paragraphs that addressed distinct sub-topics. By having the recommender produce JSON suggestions and the orchestrator decide what to apply, we get the recommender's pattern-matching speed plus your judgment about which changes serve the research_query.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/notes/final_report_<vault_tag>.md` — the polished final report from step 15
+
+---
+
+## Step 16.1 — Spawn the readability recommender
+
+Spawn ONE `hyperresearch-readability-recommender` subagent. Single spawn, runs once.
+
+**Spawn template:**
+```
+subagent_type: hyperresearch-readability-recommender
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are step 16 of the hyperresearch V8 pipeline —
+ the final analytical pass. The final report at
+ research/notes/final_report_<vault_tag>.md has been drafted (step 10),
+ synthesized (step 11), critiqued (step 12), gap-filled (step 13),
+ patched (step 14), and polish-audited (step 15). Your job: write
+ JSON recommendations for paragraph rhythm, list/table conversions,
+ and other structural readability improvements. You are tool-locked
+ to [Read, Write] — you cannot Edit the report. The orchestrator
+ reads your recommendations and decides which to apply.
+
+ YOUR INPUTS:
+ - draft_path: research/notes/final_report_<vault_tag>.md
+ - recommendations_path: research/runs/<vault_tag>/readability-recommendations.json
+
+ Write recommendations as a JSON array per the schema in your agent
+ prompt. Cap at 50 recommendations, prioritized by impact.
+```
+
+---
+
+## Step 16.2 — Read the recommendations
+
+When the recommender returns:
+
+1. **Read `research/runs/<vault_tag>/readability-recommendations.json`.**
+
+2. **Read the recommender's report-back.** It tells you:
+ - Total count of recommendations
+ - Breakdown by category (merge-paragraphs, break-paragraph, make-list, make-table, bold-keyterms, split-sentence, remove-hr, add-whitespace)
+ - Highest-severity issue
+ - Expected net char delta if all applied
+
+---
+
+## Step 16.3 — Decide which to apply
+
+You are not obligated to apply every recommendation. Use these heuristics:
+
+**Apply confidently:**
+- All `merge-paragraphs` recommendations where adjacent paragraphs are clearly the same sub-topic (rationale field confirms this)
+- All `break-paragraph` recommendations on paragraphs > 800 CJK / 1500 EN chars
+- All `remove-hr` recommendations (horizontal rules don't belong in research reports)
+- All `add-whitespace` recommendations (zero risk)
+- `make-table` recommendations when the prose-comparison passage cited 3+ entities × 2+ dimensions and the recommender's suggested table preserves all comparison points
+
+**Apply with judgment:**
+- `make-list` recommendations: confirm the prose was actually enumerative (3+ items in sequence) and not load-bearing argumentative prose. If the rationale says "items appear sequentially in flowing prose," that's NOT a list candidate — skip it.
+- `bold-keyterms` recommendations: confirm the term is genuinely a key term, not just any noun. Bold load-bearing concepts and statistics; don't over-bold.
+
+**Apply skeptically (often skip):**
+- `split-sentence` recommendations on argumentative prose where sentence length serves emphasis or rhythm
+- Recommendations that touch the opening thesis paragraph (load-bearing — keep as written)
+- Recommendations that change tables that already exist
+
+**Always skip:**
+- Any recommendation whose `current` field doesn't match the actual draft (the recommender mis-anchored — log and ignore)
+- Recommendations that would change H2 heading text
+- Recommendations that delete substantive content (the recommender shouldn't, but verify)
+
+---
+
+## Step 16.4 — Apply chosen recommendations via Edit
+
+For each recommendation you decide to apply:
+
+1. Use the Edit tool on `research/notes/final_report_<vault_tag>.md`
+2. `old_string` = the recommendation's `current` field (exactly as the recommender wrote it)
+3. `new_string` = the recommendation's `recommended` field
+
+**For non-ASCII text (CJK / Arabic / Cyrillic):** the recommender copied `current` verbatim from Read output. Trust that. Don't retype.
+
+**Order of application:**
+1. `remove-hr` first (smallest changes, cleanest baseline)
+2. `merge-paragraphs` and `break-paragraph` (paragraph-level changes)
+3. `make-list` and `make-table` (structural conversions)
+4. `bold-keyterms` (within paragraphs that survived the merges/breaks)
+5. `split-sentence` (within finalized paragraphs)
+6. `add-whitespace` (final cleanup)
+
+If an Edit fails because `old_string` doesn't match (recommender mis-anchored), skip that recommendation and continue with the rest.
+
+---
+
+## Step 16.5 — Log decisions
+
+Write `research/runs/<vault_tag>/readability-decisions.json` with the orchestrator's decisions:
+
+```json
+{
+ "total_recommendations": <int>,
+ "applied": [<list of recommendation IDs you applied>],
+ "skipped": [
+ {"id": "rec-N", "reason": "<one sentence>"}
+ ],
+ "edit_failures": [
+ {"id": "rec-N", "reason": "old_string did not match the draft"}
+ ],
+ "net_char_delta_actual": <int — measure the actual delta>
+}
+```
+
+This is the audit trail. If a future review finds a readability problem we should have fixed, this log shows whether we considered it and skipped, or never saw the recommendation.
+
+---
+
+## Exit criterion
+
+- `research/runs/<vault_tag>/readability-recommendations.json` exists
+- `research/runs/<vault_tag>/readability-decisions.json` exists with at least one entry in `applied` or all `skipped`
+- `research/notes/final_report_<vault_tag>.md` reflects the applied recommendations
+- The final report's structure (H2 list, executive summary, conclusion) is unchanged from step 15's output (this step does not restructure)
+
+---
+
+## Pipeline complete
+
+Return to the entry skill (`hyperresearch`). Mark all todos complete. Tell the user the final report path.
+
+You're done.
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-2-width-sweep.md b/tests/fixtures/golden_prompts/skills/hyperresearch-2-width-sweep.md
new file mode 100644
index 0000000..f27e726
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-2-width-sweep.md
@@ -0,0 +1,338 @@
+---
+name: hyperresearch-2-width-sweep
+description: >
+ Step 2 of the hyperresearch V8 pipeline. Multi-perspective search planning
+ (breadth / depth / adversarial lenses) followed by parallel fetcher waves.
+ Achieves comprehensive topical coverage with 55–80 curated sources for
+ full tier. Includes coverage check, evidence redundancy audit,
+ and source count gating. Invoked via Skill tool from the entry skill
+ after step 1 completes.
+---
+
+# Step 2 — Width sweep
+
+**Tier gate:** Runs for ALL tiers. For `light` tier: skip academic APIs, target 15–25 sources, limit to 2–3 fetcher batches. For `full`: run the full procedure below.
+
+**Goal:** achieve comprehensive topical coverage — every atomic item from the decomposition must have at least 3 supporting sources by the end of this step. Target 55–80 curated sources for `full` tier.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag (in Run config), modality
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, sub_questions, entities, pipeline_tier
+- `research/runs/<vault_tag>/temp/coverage-matrix.md` — verbatim query phrases mapped to atomic items
+- `research/runs/<vault_tag>/query.md` — canonical research query (GOSPEL)
+
+---
+
+## Step 2.1 — Multi-perspective search planning
+
+Before spawning any fetchers, produce a **search plan** that maps the decomposition to concrete searches from **three independent perspectives**. This is the single highest-leverage step for comprehensiveness — an ad-hoc search finds 40 sources on the same 3 sub-topics; a multi-perspective planned search distributes sources across all atomic items from angles a single researcher would miss.
+
+1. **Read the decomposition.** Extract every `sub_question` and every `entity` with its `required_fields`.
+
+2. **Generate searches from three lenses.** For EACH atomic item, generate searches from all three perspectives:
+
+ **Lens A — Breadth coverage (systematic):**
+ - One search for the core factual content of that item
+ - One search for recent developments / state-of-the-art (last 2 years)
+ - One search for each named entity or sub-concept within the item
+ - Goal: no atomic item left uncovered. Cast wide.
+
+ **Lens B — Citation-chain depth (academic/canonical):**
+ - Academic API queries (Semantic Scholar, arXiv, OpenAlex, PubMed) for each item with research literature
+ - Searches targeting canonical/seminal works, foundational papers, authoritative reports
+ - Searches for the upstream sources that derivative commentary cites ("original study", "primary data", "foundational paper")
+ - Goal: find the load-bearing sources that secondary commentary is built on.
+
+ **Lens C — Adversarial/contrarian (dialectical):**
+ - "criticism of X", "limitations of X", "problems with X", "why X doesn't work" for each item
+ - Searches for competing frameworks, alternative explanations, dissenting experts
+ - Searches for failure cases, negative results, counter-examples
+ - At least one "X is wrong" or "against X" search per major atomic item
+ - Goal: ensure the corpus includes the strongest case AGAINST the emerging consensus.
+
+ **Lens D — Period-pinned primary sources (MANDATORY when `time_periods` is non-empty in the decomposition):**
+ - For EVERY entry in `prompt-decomposition.json -> time_periods`, generate at least one search that targets the **primary filing for that exact period** — not the most recent filing, not narrative commentary, not earnings-call summaries.
+ - **SEC EDGAR (US public companies):** target the filing index for the named period. Example queries: `site:sec.gov 10-Q "period ended September 30, 2024" <company>` or `site:sec.gov/cgi-bin/browse-edgar <CIK> 10-Q dateb=20241231`. Open the EDGAR filing-history page directly when CIK is known.
+ - **Companies House (UK):** target the filing-history view, NOT the search index. Example: `site:find-and-update.company-information.service.gov.uk <company> filing-history`. Then fetch the specific accounts PDF made up to the named period.
+ - **Earnings releases / press releases:** target the dated press release for that period, not coverage of it. Example: `<company> "third quarter 2024" results press release`.
+ - **Government / central-bank releases, regulatory disclosures, statutory accounts:** target the publication for the exact reporting period.
+ - **Earnings-call transcripts are insufficient on their own.** Transcripts narrate already-rounded numbers ("revenue grew about 27%"); rubrics demand the tabular line items from the filing itself. If the prompt names a fiscal period, the search plan MUST include a query for the filing PDF, not just the transcript.
+ - Goal: every period in `time_periods` has at least one search that, if successful, fetches the filing's tabular data — not a paraphrase of it.
+
+3. **Write the combined search plan to `research/runs/<vault_tag>/temp/search-plan.md`** — a table with a `Lens` column:
+ ```markdown
+ | Atomic item | Search query | Type | Lens | Target |
+ |---|---|---|---|---|
+ | Sub-Q1 | "China financial industry growth trends 2025" | web | breadth | factual |
+ | Sub-Q1 | "China financial sector structural risks" | web | adversarial | contrarian |
+ | Sub-Q1 | "financial repression China scholarly analysis" | academic | depth | canonical |
+ | Entity: PE | "China private equity returns academic study" | academic | depth | canonical |
+ ```
+
+ Plan typically has **40–100 planned searches** for a `full` query.
+
+4. **Search gap check.** Cross-check the search plan against `research/runs/<vault_tag>/temp/coverage-matrix.md`. For every row in the coverage matrix, verify at least one search in the plan targets that query phrase's atomic item. Re-read the verbatim query and check: is there any significant topic, entity, or category in the query that has ZERO rows in the search plan?
+
+ Common failure modes this catches:
+ - Decomposition correctly listed "rugged tablets" but search plan has no queries for tablet manufacturers, enterprise mobility, or field-service devices
+ - Query mentions "Southeast Asia" but all regional searches target only "North America" and "Japan/Korea"
+ - Query says "SaaS applications" but every search is about "payment terminals"
+
+ If gaps exist: add the missing searches to the plan NOW, before proceeding. Do NOT proceed to fetching with known search gaps — a missing search now becomes a missing section in step 10.
+
+5. **Execute searches from ALL three lenses.** Do not shortcut by running only Lens A. The adversarial and depth lenses produce qualitatively different URLs that breadth searching misses.
+
+6. **Minimum adversarial coverage:** at least **5 adversarial searches total**. The dialectic critic will punish one-sided coverage.
+
+---
+
+## Step 2.2 — Execute searches and build URL queue
+
+1. **Academic APIs first.** For topics with a research literature, hit Semantic Scholar / arXiv / OpenAlex / PubMed BEFORE web search. Academic APIs return citation-ranked canonical papers.
+
+2. **Web searches from the plan.** Execute ALL planned searches across all three lenses. Aim for **80–120 candidate URLs** before deduplication for `full` tier.
+
+3. **Build and deduplicate the master URL queue.** Remove exact-URL duplicates. Remove obvious junk domains. The deduplicated queue should have **60–100 URLs** for `full` tier.
+
+ **Wikipedia SOURCE HUB rule:** Include Wikipedia URLs in the queue — they're valuable for discovery — but treat them as SOURCE HUBS, not as citable sources. When a fetcher processes a Wikipedia article, it extracts the references/citations Wikipedia links to. Those primary sources go into Wave 2 (or the same wave if capacity permits). Wikipedia itself is NEVER cited in the final report.
+
+4. **Partition the queue into non-overlapping batches.** Split the master queue into **10–12 batches** of **8–12 URLs each**. Each batch goes to exactly ONE fetcher. **Zero overlap.**
+
+---
+
+## Step 2.3 — Utility scoring and selection
+
+**Tier gate:** SKIP for `light`. Run for `full`.
+
+Before batching URLs, score each candidate URL on six dimensions (0–3 each, max composite 18):
+
+1. **Authority (0–3):** Primary data / government / academic (3) > institutional report (2) > quality journalism (1) > blog (0)
+2. **Novelty (0–3):** Unique domain or perspective (3) > partially overlapping (1) > redundant (0)
+3. **Stance diversity (0–3):** Adversarial / contrarian (3) > mixed-stance (2) > neutral (1) > same-stance majority (0)
+4. **Coverage (0–3):** Targets uncovered atomic item (3) > thin item (2) > adequate item (1) > well-covered (0)
+5. **Redundancy (0–3):** Likely novel content (3) > possibly overlapping (1) > almost certainly a rewrite (0)
+6. **Freshness (0–3):** For temporal topics: last 12 months (3), 1–3 years (2), 3–5 years (1), older (0). For foundational topics: canonical/seminal (3), recent derivative (1).
+
+**Selection rule:** Rank by composite utility score. Select the top N URLs (where N = batch capacity × batch count). Hard constraint: every atomic item must have ≥3 candidate URLs before low-utility URLs from well-covered items are included.
+
+Write to `research/runs/<vault_tag>/temp/scored-urls.md`.
+
+**Scores travel with the URLs.** When you assign batches (step 2.4), include each URL's composite utility score next to it. Fetchers pass it to `$HPR fetch --utility-score <N>` so the score persists into note frontmatter — it becomes one input to the vault's composite `quality_score`, which step 10 uses for ranked curation.
+
+---
+
+## Step 2.4 — Parallel fetcher waves
+
+**Wave 1 (main wave):** Spawn **10–12 fetcher subagents in ONE message** — true parallel execution. Each fetcher gets its own non-overlapping batch.
+
+**Subagent type:** `hyperresearch-fetcher`
+
+**Spawn template (use the standard 3-piece contract):**
+```
+subagent_type: hyperresearch-fetcher
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste contents of research/runs/<vault_tag>/query.md}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are step 2 (width-sweep fetcher) of the
+ hyperresearch V8 pipeline. The orchestrator partitioned the URL queue into
+ non-overlapping batches; you fetch ONLY the URLs in your batch. Do not
+ search for additional URLs. After you return, the orchestrator runs a
+ coverage check (step 2.5) and may dispatch wave 2 fetchers.
+
+ YOUR INPUTS:
+ - vault_tag: <vault_tag>
+ - urls: [<batch URLs, exactly as assigned — with each URL's utility score when scored, e.g. "https://... (utility: 14)">]
+ - batch_id: <number>
+```
+
+**CRITICAL: no token waste.** Each fetcher gets ONLY its batch. No fetcher searches for new URLs or duplicates another fetcher's work. If a fetcher finishes early, it's done.
+
+**CRITICAL: never emit bare text while waiting.** In `-p` mode, a text-only response triggers `end_turn`.
+
+**Use wait time to think.** While subagents are working, write evolving thoughts to `research/runs/<vault_tag>/temp/orchestrator-notes.md`:
+- What patterns are emerging from sources?
+- What tensions or contradictions do you expect?
+- What's the strongest thesis forming? What could overturn it?
+- How will atomic items map to sections?
+- What's the narrative arc?
+
+Append a few lines with `Edit` or `Write` every 30-60 seconds. Productive thinking time AND keeps the turn alive.
+
+**Vault count check** — once every 60 seconds max:
+```bash
+PYTHONIOENCODING=utf-8 $HPR note list --tag <vault_tag> --all --json | python -c "import sys,json; d=json.load(sys.stdin); print(f'Notes in vault: {len(d.get(\"data\",[]))}')"
+```
+
+The wave is done when the vault note count is ≥80% of total URLs queued.
+
+---
+
+## Step 2.5 — Coverage check (MANDATORY)
+
+After Wave 1 returns, run the coverage check before proceeding:
+
+1. **List fetched sources:** `$HPR note list --tag <vault_tag> --all --json` — count substantive (non-deprecated) notes.
+
+2. **Map sources → atomic items.** For each atomic item in the decomposition, identify which fetched sources serve it. Mark each item as:
+ - **Well-covered** (4+ relevant sources)
+ - **Adequate** (2–3 sources)
+ - **Thin** (1 source)
+ - **Uncovered** (0 sources)
+
+3. **Wave 2 fetch for gaps.** For every `thin` or `uncovered` item:
+ - Run 2–3 targeted searches specifically for that item
+ - Spawn 3–5 fetchers with gap-filling URLs (non-overlapping batches)
+ - This wave is smaller (typically 20–40 URLs) but surgically targeted
+
+4. **Write coverage report** to `research/runs/<vault_tag>/temp/coverage-gaps.md`:
+ - List every atomic item with its coverage status and source count
+ - Any item still at 0 sources after Wave 2 is a genuine gap — flag it prominently
+
+**Do NOT skip the coverage check.** Comprehensiveness scores directly with how many atomic items have multi-source coverage.
+
+---
+
+## Step 2.6 — Evidence redundancy audit
+
+**Tier gate:** SKIP for `light`. Run for `full`.
+
+**Goal:** detect when N sources are really 1 source in N outfits.
+
+1. **Collect all claims.** Read `research/runs/<vault_tag>/temp/claims-<note-id>.json` for every non-deprecated note tagged `<vault_tag>`. If no claim files exist, skip this step.
+
+2. **Cluster by content overlap.** Sources sharing >60% of their `quoted_support` passages are likely derivative.
+
+3. **Cluster by citation ancestry.** Use `suggested-by` links in the vault graph.
+
+4. **For each cluster, identify the canonical upstream source.** Tag derivative sources with `derivative-of`. Do NOT deprecate them — discount them in coverage counting.
+
+5. **Write `research/runs/<vault_tag>/temp/redundancy-audit.md`** — clusters, adjusted coverage counts, atomic items dropping below 2 → flag for Wave 3.
+
+6. **Wave 3 fetch (conditional).** If any atomic item's independent source count drops below 2, run targeted searches for INDEPENDENT sources. Spawn 2-3 fetchers.
+
+---
+
+## Step 2.7 — Persist ranking signals (ALL tiers, cheap, run after the last wave)
+
+Four commands that turn the corpus into a *ranked* corpus. Run them once, in order, after fetching completes:
+
+```bash
+$HPR claims ingest --tag <vault_tag> -j # claims JSONs -> queryable claims table
+$HPR sources backfill-doi --tag <vault_tag> -j # catch DOIs the fetchers missed
+$HPR sources score --tag <vault_tag> -j # citation counts + retraction check (cached APIs)
+$HPR graph rank -j # vault centrality + composite quality_score
+```
+
+**If `sources score` reports RETRACTED sources:** flag them in `research/runs/<vault_tag>/temp/coverage-gaps.md` immediately — a retracted source must never anchor a locus or survive into drafting as unqualified evidence. The retraction floor also crushes its `quality_score`, so ranked curation (step 10) buries it automatically.
+
+These commands are local/cached and cost seconds. Skipping them leaves step 10's ranked curation blind.
+
+---
+
+## Step 2.8 — Drain the browser-lane escalation queue (conditional)
+
+Blocked fetches (login walls, bot walls, captchas) were NOT lost — the fetch gate queued them:
+
+```bash
+$HPR escalation list --status queued --tag <vault_tag> -j
+```
+
+**If queued items exist**, spawn EXACTLY ONE `hyperresearch-browser-fetcher` subagent to drain them (serial, one browser — never spawn two):
+
+```
+subagent_type: hyperresearch-browser-fetcher
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are the step 2 escalation-lane fetcher of the
+ hyperresearch V8 pipeline. Headless fetchers hit walls on these URLs;
+ you drive the user's real Chrome browser to fetch them. After you
+ return, the orchestrator consolidates any needs_human items for the
+ user and re-runs the ranking commands (step 2.7).
+
+ YOUR INPUTS:
+ - vault_tag: <vault_tag>
+ - drain up to 10 items (claim via `$HPR escalation claim --tag <vault_tag>`)
+```
+
+**When the browser-fetcher returns with `needs_human` items** (CAPTCHAs, logins, 2FA — it NEVER solves these itself):
+
+1. **Consolidate into ONE message to the user** — never one interruption per URL:
+ > "3 sources need you: [site A: solve the CAPTCHA], [site B: log in], [site C: approve 2FA]. Open them in Chrome, complete the challenges, then tell me 'done' (or 'skip')."
+2. In non-interactive (`-p`) runs where no user can answer: record `$HPR run block <vault_tag> --on human-challenges -j` and CONTINUE the pipeline with everything else — the queue drains on the next `hpr run resume`.
+3. After the user says done: `$HPR escalation retry <id>` each item, re-spawn the browser-fetcher once, then re-run step 2.7's ranking commands so the new sources are scored.
+
+**If the Claude-in-Chrome extension is unavailable**, the queue simply accumulates — report the queued count in your wave summary and move on. Abandoned/queued items are exactly the pre-4.0 status quo (lost sources), never worse.
+
+---
+
+## Source count targets
+
+| Tier | Minimum sources | Target sources | Fetchers per wave | Waves |
+|------|----------------|---------------|-------------------|-------|
+| `light` | 10 | 15–25 | 3–5 | 1–2 |
+| `full` | 45 | 55–80 | 10–12 | 2–3 |
+
+Substantive (non-deprecated) note counts. The `full` row reflects the installed scale gear. Quality over quantity — beyond ~80 sources, each additional source yields diminishing returns while degrading summarizer quality.
+
+---
+
+## Long-source delegation (any time during step 2)
+
+When a single long source (>5000 words) is load-bearing, delegate end-to-end analysis to `hyperresearch-source-analyst` (full-source deep read):
+
+Trigger conditions (ALL three must hold):
+1. **Length:** source's `word_count` (visible on `$HPR note show <id> -j`) exceeds ~5000 words
+2. **Relevance:** source is relevant to the research_query
+3. **No existing analysis:** no `type: source-analysis` note already exists for this source
+
+**Cap: at most 6 source-analysts per query.**
+
+Spawn template:
+```
+subagent_type: hyperresearch-source-analyst
+prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are a leaf subagent for deep end-to-end analysis
+ of ONE long source. Your digest feeds downstream hyperresearch V8 steps. You
+ do NOT spawn other subagents.
+
+ YOUR INPUTS:
+ - source_note_id: <vault note id of the long source>
+ - output_path: research/runs/<vault_tag>/temp/source-analysis-<source_note_id>.md
+ - vault_tag: <vault_tag>
+```
+
+---
+
+## Exit criterion
+
+- Minimum source count met (per tier table)
+- Coverage check shows no `uncovered` atomic items (thin is acceptable)
+- `research/runs/<vault_tag>/temp/coverage-gaps.md` written
+- (For std/full): `research/runs/<vault_tag>/temp/redundancy-audit.md` written if any claim files existed
+
+If you fall short after two waves, proceed anyway but ensure `coverage-gaps.md` lists what's missing so the drafter handles it.
+
+---
+
+## Next step
+
+Return to the entry skill (`hyperresearch`). Tier-based routing:
+
+- **light tier:** Skip directly to step 10 — invoke `Skill(skill: "hyperresearch-10-triple-draft")` (light tier writes a single draft, not the ensemble)
+- **full tier:** Invoke `Skill(skill: "hyperresearch-3-contradiction-graph")`
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md b/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md
new file mode 100644
index 0000000..5ab2913
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md
@@ -0,0 +1,124 @@
+---
+name: hyperresearch-4-loci-analysis
+description: >
+ Step 4 of the hyperresearch V8 pipeline. Spawns 2 parallel loci-analyst
+ subagents that read the width corpus and identify 1-6 specific
+ questions where depth investigation will pay off. Deduplicates and
+ scores each locus on importance/uncertainty/disagreement/decision_impact,
+ then allocates source budgets dynamically. Invoked via Skill tool from
+ the entry skill (full tier only).
+---
+
+# Step 4 — Loci analysis (parallel, 2 analysts)
+
+**Tier gate:** SKIP entirely for `light` tier — proceed directly to step 9. Only `full` tier runs loci analysis.
+
+**Goal:** identify 1–6 specific questions where depth investigation will pay off.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, sub-questions
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` — ranked fight clusters (if step 3 ran)
+- `research/runs/<vault_tag>/temp/coverage-gaps.md` — which atomic items have weak coverage
+
+Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width sweep is complete.
+
+---
+
+## Procedure
+
+1. **Spawn 2 `hyperresearch-loci-analyst` subagents in parallel** (ONE message, both Task calls). Both read the same width corpus but return independently.
+
+ **Spawn template:**
+ ```
+ subagent_type: hyperresearch-loci-analyst
+ prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are step 4 (loci-analyst, instance A or B) of
+ the hyperresearch V8 pipeline. The width sweep (step 2) populated the vault
+ tagged <vault_tag>. The contradiction graph (step 3) lives at
+ research/runs/<vault_tag>/temp/contradiction-graph.json. After you and the other
+ analyst return, the orchestrator dedupes your loci and assigns budgets.
+
+ YOUR INPUTS:
+ - corpus_tag: <vault_tag>
+ - analyst_id: "a" (for one) / "b" (for the other)
+ - output_path: research/runs/<vault_tag>/loci-a.json (or research/runs/<vault_tag>/loci-b.json)
+ ```
+
+2. **Wait for both.** If one fails, proceed with the single successful output. If both fail (empty loci lists), tell the user the width sweep was too thin and stop — do not force depth on a weak corpus.
+
+3. **Deduplicate and clamp to 6.**
+ - Read both JSON outputs.
+ - Dedupe on `name` (exact match) or near-match (same core question, different phrasing). When in doubt, prefer the entry with stronger `corpus_evidence`.
+ - If the deduped list exceeds 6, drop the weakest entries — rank by how load-bearing the rationale is for the canonical research query.
+ - **Persist both analysts' `skip_loci` arrays** in the merged output — union them under a top-level `skip_loci` key. These justifications matter downstream.
+
+4. **Score and budget each locus (dynamic depth allocation).** For each surviving locus, compute four dimensions:
+ - **importance** (0-10): how central is this locus to the research_query? A locus that directly answers a primary sub-question scores 8-10; tangential enrichment scores 2-4.
+ - **uncertainty** (0-10): how uncertain is the current evidence? If the contradiction graph shows a sharp fight with equal-quality evidence on both sides, uncertainty is high (8-10). If one side has clearly stronger evidence, moderate (4-6). If the corpus already resolves this, low (1-3).
+ - **disagreement** (0-10): how many independent sources disagree? Proxy from the contradiction cluster size. Singletons score low (2-3); multi-source fights score high (7-10). If no contradiction graph exists, estimate from the loci analyst's `opposing_positions`.
+ - **decision_impact** (0-10): would resolving this locus change the draft's recommendation or thesis? If yes, high (8-10). If it adds nuance but doesn't change direction, moderate (4-6).
+
+ **Composite score** = importance + uncertainty + disagreement + decision_impact (max 40).
+
+ **Allocate source budgets.** Total source budget for step 5 is 40. Distribute proportionally:
+ - Loci scoring 30-40: `source_budget` up to 15 (deep dive)
+ - Loci scoring 20-29: `source_budget` up to 10 (standard)
+ - Loci scoring 10-19: `source_budget` up to 5 (shallow pass)
+ - Loci scoring <10: `source_budget` 0-3, or skip investigation entirely
+
+ It's fine if only 1-2 loci score above 20 — allocate heavily to them.
+
+5. **Write scored loci to `research/runs/<vault_tag>/loci.json`.** Schema:
+ ```json
+ {
+ "loci": [
+ {
+ "name": "...",
+ "one_line": "...",
+ "flavor": "dialectical|synthesis|technical",
+ "importance": 8,
+ "uncertainty": 7,
+ "disagreement": 6,
+ "decision_impact": 9,
+ "composite_score": 30,
+ "source_budget": 12,
+ "rationale": "..."
+ }
+ ],
+ "skip_loci": [...union from both analysts...]
+ }
+ ```
+
+6. **Decide investigator count.** Spawn ONE depth-investigator (in step 5) per locus with `source_budget > 0`, capped at 6. If only 1 locus passes scoring, spawn 1.
+
+**INVARIANT:** at least one `flavor: "dialectical"` locus must be present unless an analyst's `skip_loci` justifies its absence with specific evidence of a univocal corpus. No dialectical locus + no justification = re-spawn the loci-analyst with a tighter prompt.
+
+**Placeholder-breadcrumb ban:** depth investigators will fetch sources; do not hand them breadcrumb placeholders like `hyperresearch-locus-seed` — use real source note ids from the vault or omit `--suggested-by` entirely.
+
+---
+
+## Exit criterion
+
+- `research/runs/<vault_tag>/loci.json` exists with at least 1 locus (or both analysts justified skip with `skip_loci`)
+- At least one dialectical locus OR a documented justification in `skip_loci`
+- All retained loci have `source_budget` allocated
+
+---
+
+## Next step
+
+Return to the entry skill (`hyperresearch`). Invoke step 5:
+
+```
+Skill(skill: "hyperresearch-5-depth-investigation")
+```
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-5-depth-investigation.md b/tests/fixtures/golden_prompts/skills/hyperresearch-5-depth-investigation.md
new file mode 100644
index 0000000..8dbb873
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-5-depth-investigation.md
@@ -0,0 +1,109 @@
+---
+name: hyperresearch-5-depth-investigation
+description: >
+ Step 5 of the hyperresearch V8 pipeline. Spawns K depth-investigator
+ subagents in parallel (one per scored locus), each producing one
+ interim note with a Committed Position section. Investigators read
+ full source bodies for their locus and may fetch additional sources
+ within their source_budget. Invoked via Skill tool from the entry
+ skill (full tier only).
+---
+
+# Step 5 — Depth investigation (parallel, K = len(loci))
+
+**Tier gate:** SKIP entirely for `light` tier. Only `full` tier runs depth investigation.
+
+**Goal:** produce ONE `interim-{locus}.md` note per locus with dense synthesis that the draft sub-orchestrators (step 10) will draft from.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/loci.json` — scored loci with source_budget per locus
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` (if step 3 ran)
+- `research/runs/<vault_tag>/query.md` — canonical research query
+
+---
+
+## Procedure
+
+1. **Spawn K `hyperresearch-depth-investigator` subagents in parallel** (ONE message, all Task calls). One per locus with `source_budget > 0`, capped at 6.
+
+ **Spawn template:**
+ ```
+ subagent_type: hyperresearch-depth-investigator
+ prompt: |
+ RESEARCH QUERY (verbatim, gospel):
+ > {{paste research/runs/<vault_tag>/query.md body}}
+
+ QUERY FILE: research/runs/<vault_tag>/query.md
+
+ PIPELINE POSITION: You are step 5 (depth-investigator) of the
+ hyperresearch V8 pipeline. Step 4's loci analysts produced research/runs/<vault_tag>/loci.json;
+ after you return, step 6 will reconcile your committed position against
+ the other investigators' positions in research/runs/<vault_tag>/comparisons.md.
+
+ YOUR LOCUS (from research/runs/<vault_tag>/loci.json):
+ - name: "<locus name>"
+ - one_line: "<one-line locus description>"
+ - flavor: "dialectical" / "synthesis" / "technical"
+ - source_budget: <integer from loci.json>
+ - rationale: "<why this locus matters>"
+
+ YOUR INPUTS:
+ - corpus_tag: <vault_tag>
+ - locus_name: <locus name>
+ - source_budget: <hard cap on additional sources you can fetch>
+
+ CRITICAL: Read the full source text of relevant vault notes (via
+ `hyperresearch note show <id1> <id2> ... -j`) BEFORE writing your
+ interim note. Drafting from summaries alone produces paraphrase;
+ drafting from full text produces synthesis. Use your source_budget
+ to fetch additional sources beyond the width corpus if needed.
+
+ OUTPUT: Write a single interim note via the hyperresearch CLI with
+ type=interim, tags = <vault_tag> + locus-<locus-name>. The note MUST
+ end with a "## Committed position" section that takes a SIDE on the
+ dialectical question (or a synthesis verdict for non-dialectical
+ loci). Include calibration: confidence level, what evidence would
+ change your mind.
+ ```
+
+ Each investigator's hard cap is `locus.source_budget`, not a flat number.
+
+2. **Each investigator writes ONE interim note** into the vault with `type: interim` and tags `<vault_tag>` + `locus-<locus-name>`. Return value is the note id.
+
+3. **Wait for all K to complete.** Investigators can fail independently. Proceed with whichever succeeded. If >50% failed, stop and reassess loci quality with the user.
+
+4. **Read the interim notes.** After all return, list them:
+ ```bash
+ $HPR note list --tag <vault_tag> --type interim --all --json
+ ```
+ Then batch-read them:
+ ```bash
+ $HPR note show <id1> <id2> ... -j
+ ```
+ Hold the Committed Position sections in your context — they are the load-bearing input to step 6 (cross-locus reconciliation).
+
+**INVARIANT:** Every interim note ends with a `## Committed position` section. An interim note ending with descriptive summary only is defective — flag it and re-spawn that investigator with the committed-position requirement emphasized.
+
+---
+
+## Exit criterion
+
+- One interim note per locus with `source_budget > 0`, each tagged `<vault_tag>` + `locus-<locus-name>`
+- Every interim note ends with `## Committed position`
+
+If >50% of investigators failed: stop and escalate.
+
+---
+
+## Next step
+
+Return to the entry skill (`hyperresearch`). Invoke step 6:
+
+```
+Skill(skill: "hyperresearch-6-cross-locus-reconcile")
+```
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-9-evidence-digest.md b/tests/fixtures/golden_prompts/skills/hyperresearch-9-evidence-digest.md
new file mode 100644
index 0000000..7cb49c2
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-9-evidence-digest.md
@@ -0,0 +1,83 @@
+---
+name: hyperresearch-9-evidence-digest
+description: >
+ Step 9 of the hyperresearch V8 pipeline. Assembles the top load-bearing
+ claims and verbatim quotes from the claims JSONs into
+ research/runs/<vault_tag>/temp/evidence-digest.md — a single high-fidelity evidence
+ index the draft sub-orchestrators read as primary evidence (higher
+ fidelity than fetcher summaries). Invoked via Skill tool from the
+ entry skill (full tier).
+---
+
+# Step 9 — Evidence digest
+
+**Tier gate:** Run for `full`. Skip for `light`.
+
+**Goal:** assemble the top load-bearing claims and verbatim quotes from the claims JSONs into a single digest file the drafter reads as primary evidence — higher-fidelity than fetcher summaries.
+
+---
+
+## Recover state
+
+Read these inputs:
+- `research/runs/<vault_tag>/scaffold.md` — vault_tag
+- `research/runs/<vault_tag>/prompt-decomposition.json` — atomic items, response_format, pipeline_tier
+- All `research/runs/<vault_tag>/temp/claims-*.json` files
+- `research/runs/<vault_tag>/temp/consensus-claims.json` (if step 3 ran)
+- `research/runs/<vault_tag>/temp/contradiction-graph.json` (if step 3 ran)
+
+---
+
+## Procedure
+
+1. **Read all claims files** from `research/runs/<vault_tag>/temp/claims-*.json` for every non-deprecated note tagged with the vault tag. If no claim files exist (e.g., fetchers didn't produce them), skip this step.
+
+2. **Filter and rank.** Keep claims where `confidence` is `"high"` OR `evidence_type` is `"empirical"` or `"statistical"`. From the remainder, prefer claims with non-empty `numbers` arrays and non-empty `quoted_support`. Cap at **80–120 claims total** for `full` tier.
+
+3. **Group by atomic item.** Match each surviving claim to the atomic item it is most relevant to based on **topic overlap** — do not rely on exact field matching. A claim about "United Health Group regulatory exposure" serves the atomic item "UNH risk factors" even though no field matches exactly. Use the claim's `entities`, `stance_target`, `scope_conditions`, and `claim` text holistically to judge relevance. When uncertain, include the claim under the most relevant item rather than dropping it to Ungrouped. Claims that genuinely don't map to any atomic item go into an "Ungrouped" section at the end.
+
+4. **Include consensus and contested claims.**
+ - If `research/runs/<vault_tag>/temp/consensus-claims.json` exists, include its claims marked as `[consensus]`.
+ - If `research/runs/<vault_tag>/temp/contradiction-graph.json` exists, include the top 3–5 contested claim pairs with both sides' `quoted_support` passages.
+
+5. **Write `research/runs/<vault_tag>/temp/evidence-digest.md`.** Format: one H3 per atomic item, bullet list of claims. Each bullet includes:
+ - The `claim` text
+ - The `quoted_support` verbatim passage (block-quoted)
+ - The `source_note_id`
+
+ Keep it scannable — this is an evidence index, not a narrative.
+
+ Example:
+ ```markdown
+ ### Atomic item: Market growth in Southeast Asia
+
+ - Annual growth rate of 12.4% in 2024 (empirical)
+ > "Southeast Asian e-commerce GMV grew from $89B to $100B between 2023 and 2024, a 12.4% YoY expansion."
+ [source-note-12]
+
+ - Vietnam led by penetration rate (statistical)
+ > "Vietnam reached 64% e-commerce penetration in 2024, the highest in SEA, surpassing Singapore (61%)."
+ [source-note-19]
+ ```
+
+---
+
+## Exit criterion
+
+- `research/runs/<vault_tag>/temp/evidence-digest.md` exists
+- Contains at least 30 claims for `full` tier
+- Grouped by atomic item with verbatim quoted_support and source_note_id
+
+If fewer claims exist in total, include all of them.
+
+---
+
+## Next step
+
+Return to the entry skill (`hyperresearch`). Invoke step 10:
+
+```
+Skill(skill: "hyperresearch-10-triple-draft")
+```
+
+Step 10 is the most important step in the pipeline. Re-read the entry skill before invoking if needed — the triple-draft ensemble must spawn 3 draft-orchestrators for `full` tier.
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch.md b/tests/fixtures/golden_prompts/skills/hyperresearch.md
new file mode 100644
index 0000000..cd992c3
--- /dev/null
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch.md
@@ -0,0 +1,279 @@
+---
+name: hyperresearch
+description: >
+ Deep research via the HYPERRESEARCH V8 architecture — a tier-adaptive 16-step
+ pipeline (light / full / dissertation) that scales from a ~30-minute light-tier
+ answer to an adversarially-audited report at the installed scale gear
+ (~1.5–2.5 hours). This entry skill is a ROUTER.
+ It does not contain step procedures — it tells you which Skill to invoke
+ for each step, in order. Each step's instructions live in its own skill
+ file (`hyperresearch-1-decompose` through `hyperresearch-16-readability-audit`)
+ and are loaded fresh into context when you invoke them.
+---
+
+# Hyperresearch V8 — multi-skill chain orchestrator
+
+You are the orchestrator. Your entire job in this conversation is:
+1. Read this file once at the start.
+2. Bootstrap canonical inputs (research_query, vault_tag, scaffold).
+3. Invoke each step skill in sequence via the `Skill` tool.
+4. Between steps, do nothing except mark todos and (optionally) think to `research/runs/<vault_tag>/temp/orchestrator-notes.md`.
+
+You do NOT do the work of any step yourself. The step skills do. You just sequence them.
+
+---
+
+## How the chain works (READ THIS CAREFULLY)
+
+Each pipeline step is its own skill file. To run a step:
+
+```
+Skill(skill: "hyperresearch-N-stepname")
+```
+
+When you invoke a Skill, that skill's full procedure is loaded into your context **fresh**. You then execute that step's procedure, hit its exit criterion, and return to the entry skill (this file) to invoke the next step.
+
+**Why this design?** Context compaction. V7 was one 1200-line skill that got compacted away by the time Layer 4 needed its triple-draft procedure. The orchestrator forgot the procedure, wrote a single draft, and produced a flat-scoring report. V8 fixes this at the source: each step's procedure is loaded into context **only at the moment it's needed**, fresh, with no eviction risk.
+
+**The 16 step skills** (all prefixed `hyperresearch-`):
+
+| # | Skill name | What it does | Tiers |
+|---|---|---|---|
+| 1 | `hyperresearch-1-decompose` | Canonical query → scaffold + decomposition + coverage matrix + tier classification | all |
+| 1.5 | `hyperresearch-1-5-chapter-partition` | Partition atomic items into 4–10 chapters; steps 2–10 then loop per chapter | dissertation |
+| 2 | `hyperresearch-2-width-sweep` | Multi-perspective search plan + parallel fetcher waves | all |
+| 3 | `hyperresearch-3-contradiction-graph` | Pair contradictions across the corpus into ranked fight clusters | full |
+| 4 | `hyperresearch-4-loci-analysis` | 2 loci-analysts → scored loci.json with source budgets | full |
+| 5 | `hyperresearch-5-depth-investigation` | K depth-investigators in parallel → interim notes with committed positions | full |
+| 6 | `hyperresearch-6-cross-locus-reconcile` | Reconcile committed positions → comparisons.md | full |
+| 7 | `hyperresearch-7-source-tensions` | Extract expert disagreements → source-tensions.json | full |
+| 8 | `hyperresearch-8-corpus-critic` | "What source would overturn this?" + targeted gap-fill fetch | full |
+| 9 | `hyperresearch-9-evidence-digest` | Top claims + verbatim quotes → evidence-digest.md | full |
+| 10 | `hyperresearch-10-triple-draft` | Per-angle source curation + 3 parallel draft-orchestrators (3 angle-specific drafts) | all |
+| 11 | `hyperresearch-11-synthesize` | Synthesis plan + outline + spawn synthesizer subagent (two-pass write) → final_report.md | full |
+| 12 | `hyperresearch-12-critics` | 4 adversarial critics in parallel → findings JSONs | full |
+| 13 | `hyperresearch-13-gap-fetch` | Fetch sources for critic-identified vault gaps | full |
+| 14 | `hyperresearch-14-patcher` | Surgical Edit hunks applied to draft | full |
+| 14.5 | `hyperresearch-14-5-cite-check` | Verify citation-sentence bindings; second small patcher pass | full |
+| 15 | `hyperresearch-15-polish` | Hygiene + filler pass (Edit-based subagent) | all |
+| 16 | `hyperresearch-16-readability-audit` | Readability recommender writes JSON suggestions; orchestrator selectively applies via Edit | all |
+
+---
+
+## Tier routing
+
+Step 1 classifies the query into a `pipeline_tier` (`light` / `full`). The tier is written to `research/runs/<vault_tag>/prompt-decomposition.json`. After step 1, **read that file** to learn the tier, then sequence steps according to:
+
+| Tier | Steps that run | Typical time |
+|------|---|---|
+| `light` | 1 → 2 → 10 (single draft) → 15 → 16 | ~30–40 min |
+| `full` | 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10 → 11 → 12 → 13 → 14 → 14.5 → 15 → 16 | ~1.5–2.5 hours |
+| `dissertation` | 1 → 1.5 (chapter partition) → [2 → … → 10 per chapter] → 6g/11 (global) → 12 → 13 → 14 → 14.5 → 15 → 16 | ~4–8 hours |
+
+`dissertation` is opt-in only — the user must explicitly request it (or the run was initialized with `--profile dissertation`). Step 1 never auto-classifies into it. On dissertation runs, steps 2–10 loop per chapter (see `hyperresearch-1-5-chapter-partition`), with up to 2 chapters in flight; each chapter stays within the proven 40–80-source envelope while the run totals 300–450 sources.
+
+**RESPECT THE TIER GATE.** When step 1 classifies a query as `light`, do NOT run the skipped steps "just to be thorough." The tier classification is a product decision: simple queries should produce fast, right-sized answers. Trust the classification. If you're uncertain, tier up — but never silently upgrade every query to `full`.
+
+**Scale gear (tier ≠ gear).** The numbers rendered into the step skills — source targets, loci caps, depth budgets, word targets — come from the installed scale profile, the **gear** (currently `full`). The `full` tier row above already reflects it. Two gears ship: `full` (55–80 sources, ~1.5–2.5 hours) and `premier` (100–130 sources, double depth budget, ~3–5 hours). The user switches gears with `hyperresearch profile use <full|premier>` — that re-renders the installed skills, so it takes effect on the NEXT run, never mid-run. `light` and `dissertation` are tiers, not gears: light is auto-classified per query; dissertation is opt-in per run and loops each chapter inside the gear's envelope.
+
+---
+
+## Bootstrap (run BEFORE invoking step 1)
+
+Before you invoke any step skill, do this:
+
+0. **Auto-init if missing.** Two checks for the first-run-after-global-install case:
+ - **Vault check.** If `.hyperresearch/` doesn't exist in the working directory, run `hyperresearch init . --json`. Creates the SQLite vault and `research/` directory.
+ - **Step-skills check.** If `.claude/skills/hyperresearch-1-decompose/SKILL.md` doesn't exist relative to the working directory, run `hyperresearch install --steps-only . --json`. Installs the 16 step skill files needed by `Skill(skill: "hyperresearch-N-...")` calls in later steps.
+
+ If either command fails because the binary isn't on PATH, tell the user to run `pip install hyperresearch` first. If both files already exist, both commands no-op cheaply — safe to run unconditionally.
+
+0.5. **Archive any pre-3.0 flat artifacts.** Run `hyperresearch archive-run --json`. On vaults that ran pre-3.0 pipelines, flat artifacts (research/scaffold.md, loci.json, etc.) may still sit at the research root; this moves them into `research/runs/archive-<prev-tag>-<UTC-timestamp>/`. On 3.0+ vaults every run already owns `research/runs/<vault_tag>/`, so concurrent and sequential runs never collide and this command cheaply no-ops.
+
+1. **Resolve the canonical research query.** Order of precedence:
+ - If `research/prompt.txt` exists (legacy harness / wrapped run), read it. Its contents are the canonical research query. GOSPEL.
+ - Otherwise, use the user's verbatim prompt as the canonical research query.
+ - Extract wrapper requirements separately: required save path, citation format, terminal-section shape, wrapper contract. These are binding but NOT part of the query.
+ - If `research/wrapper_contract.json` exists, read it.
+
+2. **Mint a unique vault tag.** First produce a short topical slug from the canonical query — 3–5 lowercase hyphen-separated words, e.g. `efield-dft-sac`. Then call `hyperresearch vault-tag <slug> --json` and parse the `vault_tag` field from the response. The CLI appends a random 6-hex-char suffix that's verified unique against every prior run's workspaces, query files, and final reports in this vault. The result — e.g. `efield-dft-sac-a3f9b7` — is the canonical vault_tag for the rest of the pipeline.
+
+2.5. **Initialize the run workspace.** Run:
+ ```bash
+ hyperresearch run init <vault_tag> --profile <full|light|premier|dissertation> --json
+ ```
+ For a standard run, pass the installed gear (`full`) unless the user asked for something else.
+ Pass `--budget <usd>` when the user set a spend ceiling. This scaffolds `research/runs/<vault_tag>/` (with `temp/`) and writes `run.json` — the run manifest. **The manifest is your durable memory**: record every step transition with `hyperresearch run step <vault_tag> <N> --status running|done -j` as you go. The profile here defaults to matching the tier you expect; if step 1 classifies differently, the manifest's profile field is informational — the decomposition's tier rules.
+
+3. **Persist the query file.** Write the verbatim canonical query to `research/runs/<vault_tag>/query.md`:
+ ```markdown
+ ---
+ vault_tag: <slug>
+ created: <ISO-8601 timestamp>
+ source: prompt.txt | user-prompt
+ ---
+
+ <verbatim query text, character-for-character>
+ ```
+ This file is the **canonical query reference for the entire pipeline**. Every step skill and every subagent reads it by path.
+
+4. **Classify modality** (collect / synthesize / compare / forecast) — record in the scaffold. This is a label that calibrates step 10's drafting style:
+ - **collect**: enumerative coverage, per-entity sections with named fields
+ - **synthesize**: defended thesis with evidence chains
+ - **compare**: proportionate per-entity depth + a committed recommendation
+ - **forecast**: predictive claims grounded in past + present, explicit time horizon
+
+5. **Write the scaffold.** Write `research/runs/<vault_tag>/scaffold.md` (your private planning document — it MUST NOT appear anywhere in the final report). Include in scaffold:
+ - User Prompt (VERBATIM — gospel)
+ - Run config (vault_tag, query_file_path, modality, wrapper requirements)
+ - Modality classification rationale
+ - Tier rationale (filled in after step 1)
+ - Wrapper requirements (save path, citation format, terminal sections)
+
+6. **Seed the TodoWrite list.** Create todos for all 16 step skill invocations using the integer step numbers, e.g.:
+ - `Step 1 — Skill: hyperresearch-1-decompose`
+ - `Step 2 — Skill: hyperresearch-2-width-sweep`
+ - ... (through Step 16)
+
+ The todo list survives context compaction; it's your durable memory of where you are in the chain.
+
+7. **Invoke step 1:** `Skill(skill: "hyperresearch-1-decompose")`.
+
+After step 1 returns, read `research/runs/<vault_tag>/prompt-decomposition.json` to learn the tier, then continue invoking step skills per the tier routing table above. After each step's exit criterion is met, mark its todo complete and move to the next.
+
+---
+
+## Four canonical rules (ALWAYS in force)
+
+1. **NEVER EMIT BARE TEXT WHILE TASKS ARE RUNNING.** In non-interactive (`-p`) mode, a text-only response (no tool call) triggers `end_turn` — the process exits and the pipeline dies. Every response while subagent tasks are in flight MUST include a tool call. The best one is appending analytical thoughts to `research/runs/<vault_tag>/temp/orchestrator-notes.md`. Vault count checks at most once per minute.
+
+2. **PATCH, NEVER REGENERATE.** After step 11 produces the synthesized final report (or step 10 for light tier), the only modifications are surgical Edit hunks from step 14 (patcher) and step 15 (polish-auditor). Both subagents are tool-locked to `[Read, Edit]`. If a critic's finding would require rewriting a whole section, it escalates to you as a structural issue — not a rewrite. Keep hunks surgical.
+
+3. **ARGUE, DON'T JUST REPORT** (full force for `argumentative` response_format; relaxed for `structured` and `short`). The pipeline is engineered to push the final report toward argumentative density. Loci must include at least one dialectical locus. Depth investigators must commit to a position. Step 6 forces cross-locus reconciliation. Step 11's synthesizer requires every body section that touches a tension to engage it explicitly.
+
+4. **RESPECT THE TIER GATE.** See tier routing table. Don't add steps "for thoroughness." Don't drop steps "for budget." The tier is a binding contract.
+
+---
+
+## Browser-lane escalations (all tiers)
+
+Blocked fetches (login walls, bot walls, captchas) are queued, not lost: `$HPR escalation list --status queued --tag <vault_tag> -j`. Step 2.8 drains the queue via ONE `hyperresearch-browser-fetcher` subagent driving the user's real Chrome browser. Two standing rules:
+
+1. **CAPTCHAs / logins / 2FA are ALWAYS the human's.** The browser-fetcher marks them `needs_human`; you consolidate ALL of them into ONE message to the user at a natural pause point (never one interruption per URL). In non-interactive runs, `$HPR run block <vault_tag> --on human-challenges` and continue with everything else.
+2. **One browser-fetcher at a time.** It's the user's actual browser — parallel instances are chaos. Check the queue again after step 13 (gap-fetch) if new fetches got blocked.
+
+## Subagent spawn contract (applies to every Task call)
+
+When a step skill instructs you to spawn a subagent, the prompt you pass MUST include three pieces near the top:
+
+1. **`research_query` — verbatim, block-quoted** from `research/runs/<vault_tag>/query.md`. Do not paraphrase, do not summarize.
+
+2. **Pipeline position statement.** One sentence naming what step the subagent runs in, what came before, what comes after. Example: *"You are step 5 (depth investigator) of the hyperresearch V8 pipeline. Step 4's loci analysts produced `research/runs/<vault_tag>/loci.json`; after you return, step 6 will reconcile your committed position against the other investigators'."*
+
+3. **The subagent's specific inputs** (vault_tag, output_path, locus, etc.). Each step skill's spawn template documents the required fields.
+
+Skipping any of these in a Task prompt is a process violation.
+
+---
+
+## Recovery: if you wake up uncertain where you are
+
+Context compaction may eat parts of this conversation. If you're unsure what step you're on:
+
+0. **Read the run manifest FIRST.** `hyperresearch run resume <vault_tag> --json` (or with no tag for the newest run) returns the exact next step and the Skill invocation to continue with. This is the primary recovery path — the manifest records every step transition you logged via `hyperresearch run step`. The artifact scan below is the fallback for manifests that are missing or were not kept up to date.
+1. **Check the TodoWrite list.** It carries integer step numbers and survives compaction.
+2. **Check disk artifacts (fallback).** Each step writes a canonical artifact:
+ - Step 1: `research/runs/<vault_tag>/scaffold.md`, `research/runs/<vault_tag>/prompt-decomposition.json`, `research/runs/<vault_tag>/temp/coverage-matrix.md`
+ - Step 2: vault notes tagged with vault_tag (`$HPR note list --tag <vault_tag> --all -j`)
+ - Step 3: `research/runs/<vault_tag>/temp/contradiction-graph.json`, `research/runs/<vault_tag>/temp/consensus-claims.json`
+ - Step 4: `research/runs/<vault_tag>/loci.json`
+ - Step 5: vault notes with `type: interim` (`$HPR note list --tag <vault_tag> --type interim --all -j`)
+ - Step 6: `research/runs/<vault_tag>/comparisons.md`
+ - Step 7: `research/runs/<vault_tag>/temp/source-tensions.json`
+ - Step 8: `research/runs/<vault_tag>/corpus-critic-gaps.json`, `research/runs/<vault_tag>/temp/corpus-critic-results.md`
+ - Step 9: `research/runs/<vault_tag>/temp/evidence-digest.md`
+ - Step 10: `research/runs/<vault_tag>/temp/draft-{a,b,c}.md` (or `research/notes/final_report_<vault_tag>.md` for light tier single-pass)
+ - Step 11: `research/runs/<vault_tag>/temp/synthesis-plan.md`, `research/runs/<vault_tag>/temp/synthesis-outline.md`, `research/runs/<vault_tag>/temp/synthesis-pass1.md`, `research/notes/final_report_<vault_tag>.md`
+ - Step 12: `research/runs/<vault_tag>/critic-findings-{dialectic,depth,width,instruction}.json`
+ - Step 13: `research/runs/<vault_tag>/temp/post-critic-fetch-log.md`
+ - Step 14: `research/runs/<vault_tag>/patch-log.json` (and edited final_report.md)
+ - Step 15: `research/runs/<vault_tag>/polish-log.json` (and edited final_report.md)
+ - Step 16: `research/runs/<vault_tag>/readability-recommendations.json`, `research/runs/<vault_tag>/readability-decisions.json` (and edited final_report.md)
+3. **Find the highest-numbered step whose artifact exists.** Resume from the next step.
+4. **Re-invoke this entry skill** if you've lost track entirely: `Skill(skill: "hyperresearch")`. It loads fresh.
+
+If you're ever uncertain what to do next, the answer is: re-read this file and find the next step in the tier sequence.
+
+---
+
+## Final integrity gate (after step 16)
+
+Once step 16 returns, run the integrity check:
+
+```bash
+for f in research/runs/<vault_tag>/critic-findings-dialectic.json \
+ research/runs/<vault_tag>/critic-findings-depth.json \
+ research/runs/<vault_tag>/critic-findings-width.json \
+ research/runs/<vault_tag>/critic-findings-instruction.json \
+ research/runs/<vault_tag>/patch-log.json \
+ research/runs/<vault_tag>/polish-log.json; do
+ test -f "$f" || echo "MISSING: $f"
+done
+```
+
+(Light tier skips critics + patcher entirely — the critic-findings and patch-log files won't exist. That's expected; only `polish-log.json` is required for light.)
+
+Then run the verification battery + lint:
+```bash
+$HPR run verify <vault_tag> --json # structural: headings, length, citation density, artifacts, cite-check resolution
+$HPR sources retractions --tag <vault_tag> --json # ship-time retraction re-check (bypasses cache)
+$HPR lint --rule wrapper-report --json
+$HPR lint --rule locus-coverage --json
+$HPR lint --rule scaffold-prompt --json
+$HPR lint --rule patch-surgery --json
+$HPR lint --rule quote-integrity --json # every quoted span must exist verbatim in a vault note
+$HPR lint --rule retracted-citations --json # retracted sources cited unmarked block the ship
+$HPR lint --rule numeric-consistency --json # warnings: numbers untraceable to evidence — verify or remove
+```
+
+If `run verify` fails or any rule returns `error` severity issues, address them before declaring complete. Then ship: the final report lives at `research/notes/final_report_<vault_tag>.md`.
+
+---
+
+## Invariants you cannot break
+
+1. **PATCHING not REGENERATION after step 11.** Once step 11 produces the final report (or step 10 for light tier), modifications are surgical Edit hunks only.
+2. **One final report.** Step 11's synthesizer writes the final report ONCE. No re-synthesizing. (Light tier: step 10 writes it once.)
+3. **At least one dialectical locus.** Step 4 must surface ≥1 dialectical locus unless skip is justified.
+4. **Every interim note commits to a position.** Step 5 investigators end with `## Committed position`.
+5. **`research/runs/<vault_tag>/comparisons.md` exists when loci count ≥ 1.** Step 6 is mandatory whenever step 4 produced any loci.
+6. **Steps are sequential at the outermost level, parallel within.** You cannot start step N+1 before step N completes. Within a step, parallelism is mandatory when there are multiple subagents.
+7. **Canonical research query is gospel everywhere.** Every subagent gets the verbatim query.
+8. **Hygiene rules apply to the final report only.** Workspace artifacts (scaffold, loci JSONs, interim notes, comparisons.md, patch log) can look however they need to look.
+9. **NEVER skip a step that the tier gate says to run.** For `full` tier, ALL 16 steps run. For `light`, the prescribed 5 steps run.
+10. **Step 10 triple-draft ensemble is MANDATORY for `full` tier.** You MUST spawn 3 `hyperresearch-draft-orchestrator` subagents. Writing `research/notes/final_report_<vault_tag>.md` directly in step 10 (instead of going through the synthesizer in step 11) is a PIPELINE VIOLATION for these tiers.
+11. **Step 11 synthesis is MANDATORY for `full` tier.** The synthesizer subagent (Read+Write tool-locked) writes the final report from the 3 drafts. The orchestrator does NOT write the final report itself for these tiers.
+12. **Subagents read full source text.** Draft sub-orchestrators MUST batch-read every note in their `must_read_note_ids` list before writing. Fetchers MUST chase 3-8 primary sources via citation chains.
+13. **NEVER emit a bare text response while subagent tasks are in flight.**
+
+---
+
+## Why V8
+
+V7 was one 1200-line skill loaded once. By Layer 4 (line ~2200 in a 3000-line conversation), context compaction had evicted the procedure. The orchestrator silently dropped Layer 3.7 (corpus critic), rewrote its todo to replace the triple-draft ensemble with a single draft, and produced a flat-scoring report. This happened in 100% of runs where the orchestrator didn't re-read the skill file.
+
+V8 makes re-reading structural. Each step skill is loaded fresh via the `Skill` tool at the moment it's needed. The procedure is in context exactly when it matters. Compaction can evict an old step's procedure — that's fine, the orchestrator never needs it again because each step is self-contained and reads its inputs from disk.
+
+The trade: 16 skill files instead of 1, plus 16 invocations of the `Skill` tool over the run. The cost is negligible; the reliability gain is the difference between Q57 (55.9, full pipeline) and Q9 (52.6, single-draft fallback).
+
+---
+
+## Now begin
+
+If you've read this far and the bootstrap (above) is done, invoke step 1:
+
+```
+Skill(skill: "hyperresearch-1-decompose")
+```
+
+If the bootstrap is NOT done, do the bootstrap first, then invoke step 1.
diff --git a/tests/test_cli/test_install_profile.py b/tests/test_cli/test_install_profile.py
new file mode 100644
index 0000000..ba7f8fb
--- /dev/null
+++ b/tests/test_cli/test_install_profile.py
@@ -0,0 +1,148 @@
+"""Tests for `hyperresearch install --profile` plumbing."""
+
+from __future__ import annotations
+
+from typer.testing import CliRunner
+
+from hyperresearch.cli import app
+
+runner = CliRunner()
+
+
+def _sweep_text(vault) -> str:
+ p = vault.root / ".claude" / "skills" / "hyperresearch-2-width-sweep" / "SKILL.md"
+ return p.read_text(encoding="utf-8")
+
+
+def test_install_default_profile_is_full(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(app, ["install", str(tmp_vault.root), "--json"])
+ assert result.exit_code == 0
+ sweep = _sweep_text(tmp_vault)
+ assert "**40–100 planned searches**" in sweep # full profile primary
+ assert 'rendered from profile "full"' in sweep
+
+
+def test_install_profile_light_changes_primary(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(
+ app, ["install", str(tmp_vault.root), "--profile", "light", "--json"]
+ )
+ assert result.exit_code == 0
+ sweep = _sweep_text(tmp_vault)
+ # `p` (primary) is now light: planned_searches renders (8, 20)
+ assert "**8–20 planned searches**" in sweep
+ assert 'rendered from profile "light"' in sweep
+ # The `full` tier row follows the gear (p.*): it shows what a full-tier
+ # run will actually target under the installed profile. The `light` row
+ # stays profile-qualified.
+ assert "| `full` | 10 | 15–25 |" in sweep
+ assert "| `light` | 10 | 15–25 |" in sweep
+
+
+def test_install_unknown_profile_fails_cleanly(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(
+ app, ["install", str(tmp_vault.root), "--profile", "bogus", "--json"]
+ )
+ assert result.exit_code == 1
+ # No half-written render should exist from the failed run: install validates
+ # the profile before writing (the file may exist from vault init defaults,
+ # but must not claim a bogus profile)
+ p = tmp_vault.root / ".claude" / "skills" / "hyperresearch-2-width-sweep" / "SKILL.md"
+ if p.exists():
+ assert 'rendered from profile "bogus"' not in p.read_text(encoding="utf-8")
+
+
+def test_install_steps_only_renders(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(
+ app, ["install", str(tmp_vault.root), "--steps-only", "--json"]
+ )
+ assert result.exit_code == 0
+ sweep = _sweep_text(tmp_vault)
+ assert "<<" not in sweep
+ assert 'rendered from profile "full"' in sweep
+
+
+def test_reinstall_is_idempotent_per_profile(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ first = runner.invoke(app, ["install", str(tmp_vault.root), "--json"])
+ assert first.exit_code == 0
+ before = _sweep_text(tmp_vault)
+ second = runner.invoke(app, ["install", str(tmp_vault.root), "--json"])
+ assert second.exit_code == 0
+ assert _sweep_text(tmp_vault) == before
+
+
+def test_profile_use_premier_switches_gear(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(app, ["profile", "use", "premier", "--json"])
+ assert result.exit_code == 0
+ # Skills re-rendered at premier scale
+ sweep = _sweep_text(tmp_vault)
+ assert 'rendered from profile "premier"' in sweep
+ assert "**80–160 planned searches**" in sweep
+ assert "| `full` | 90 | 100–130 |" in sweep
+ # Gear persisted in config
+ cfg_text = tmp_vault.config_path.read_text(encoding="utf-8")
+ assert '[pipeline]' in cfg_text
+ assert 'profile = "premier"' in cfg_text
+
+
+def test_bare_install_keeps_persisted_gear(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ use = runner.invoke(app, ["profile", "use", "premier", "--json"])
+ assert use.exit_code == 0
+ # A later bare install (e.g. after a package upgrade) must NOT silently
+ # downshift the gear back to full.
+ result = runner.invoke(app, ["install", str(tmp_vault.root), "--json"])
+ assert result.exit_code == 0
+ sweep = _sweep_text(tmp_vault)
+ assert 'rendered from profile "premier"' in sweep
+ # An explicit --profile still overrides the persisted gear
+ result = runner.invoke(
+ app, ["install", str(tmp_vault.root), "--profile", "full", "--json"]
+ )
+ assert result.exit_code == 0
+ assert 'rendered from profile "full"' in _sweep_text(tmp_vault)
+
+
+def test_profile_use_rejects_tiers(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ for tier in ("light", "dissertation"):
+ result = runner.invoke(app, ["profile", "use", tier, "--json"])
+ assert result.exit_code == 1, tier
+ assert "tier" in result.stdout.lower()
+
+
+def test_profile_use_unknown_fails_cleanly(tmp_vault, monkeypatch):
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(app, ["profile", "use", "bogus", "--json"])
+ assert result.exit_code == 1
+ # Config must not have been touched
+ cfg_text = tmp_vault.config_path.read_text(encoding="utf-8")
+ assert 'profile = "bogus"' not in cfg_text
+
+
+def test_profile_use_preserves_user_overlays(tmp_vault, monkeypatch):
+ # Regression: config.save() used to drop [profile.*] tables, destroying
+ # user-defined profiles on any config write.
+ cfg_path = tmp_vault.config_path
+ cfg_path.write_text(
+ cfg_path.read_text(encoding="utf-8")
+ + "\n[profile.megareview]\nsource_min = 250\nloci_max = 20\n",
+ encoding="utf-8",
+ )
+ monkeypatch.chdir(tmp_vault.root)
+ result = runner.invoke(app, ["profile", "use", "premier", "--json"])
+ assert result.exit_code == 0
+ cfg_text = cfg_path.read_text(encoding="utf-8")
+ assert "[profile.megareview]" in cfg_text
+ assert "source_min = 250" in cfg_text
+ # And the overlay still resolves after the round-trip
+ from hyperresearch.core.profiles import resolve_profile
+
+ p = resolve_profile("megareview", cfg_path)
+ assert p.source_min == 250
+ assert p.loci_max == 20
diff --git a/tests/test_core/test_claims_and_embed.py b/tests/test_core/test_claims_and_embed.py
new file mode 100644
index 0000000..174be2c
--- /dev/null
+++ b/tests/test_core/test_claims_and_embed.py
@@ -0,0 +1,163 @@
+"""Tests for claims persistence (WS5) and embeddings (WS6) — fully offline."""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from hyperresearch.core import embed
+from hyperresearch.core.claims import (
+ ingest_claims_dir,
+ ingest_claims_file,
+ list_claims,
+ search_claims,
+)
+from hyperresearch.core.embed import (
+ EmbeddingError,
+ cosine,
+ reciprocal_rank_fusion,
+)
+
+
+def claims_vault(seeded_vault):
+ """Seeded vault plus a claims JSON file for one of its notes."""
+ temp = seeded_vault.root / "research" / "temp"
+ temp.mkdir(parents=True, exist_ok=True)
+ claims = [
+ {
+ "claim": "Async IO improves throughput for network-bound workloads",
+ "quoted_support": "async/await syntax enables concurrent I/O",
+ "numbers": ["10x"],
+ "confidence": "high",
+ "evidence_type": "empirical",
+ "stance_target": "async-performance",
+ "stance": "supports",
+ },
+ {
+ "claim": "GIL limits CPU-bound parallelism",
+ "confidence": "medium",
+ "evidence_type": "opinion",
+ },
+ ]
+ (temp / "claims-python-async-patterns.json").write_text(
+ json.dumps(claims), encoding="utf-8"
+ )
+ return seeded_vault
+
+
+class TestClaimsIngest:
+ def test_ingest_and_list(self, claims_vault):
+ summary = ingest_claims_dir(claims_vault, vault_tag="test-run")
+ assert summary["ingested"] == 2
+ assert summary["errors"] == []
+ rows = list_claims(claims_vault.db, note_id="python-async-patterns")
+ assert len(rows) == 2
+ assert rows[0]["vault_tag"] == "test-run"
+ assert json.loads(rows[0]["numbers"]) == ["10x"]
+
+ def test_reingest_is_idempotent(self, claims_vault):
+ ingest_claims_dir(claims_vault)
+ second = ingest_claims_dir(claims_vault)
+ assert second["ingested"] == 0
+ assert second["skipped"] == 2
+ assert len(list_claims(claims_vault.db)) == 2
+
+ def test_fts_search(self, claims_vault):
+ ingest_claims_dir(claims_vault)
+ hits = search_claims(claims_vault.db, "throughput")
+ assert len(hits) == 1
+ assert hits[0]["note_id"] == "python-async-patterns"
+
+ def test_unknown_note_errors_softly(self, claims_vault):
+ temp = claims_vault.root / "research" / "temp"
+ (temp / "claims-nonexistent-note.json").write_text("[]", encoding="utf-8")
+ summary = ingest_claims_dir(claims_vault)
+ assert any("not in vault" in e for e in summary["errors"])
+
+ def test_wrapper_format_accepted(self, claims_vault, tmp_path):
+ p = claims_vault.root / "research" / "temp" / "claims-rust-ownership.json"
+ p.write_text(json.dumps({"claims": [{"claim": "Ownership prevents data races"}]}), encoding="utf-8")
+ r = ingest_claims_file(claims_vault.db, p)
+ claims_vault.db.commit()
+ assert r["ingested"] == 1
+
+ def test_claims_cli(self, claims_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(claims_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["claims", "ingest", "--tag", "cli-run", "--json"])
+ assert result.exit_code == 0
+ payload = json.loads(result.stdout)
+ assert payload["data"]["ingested"] == 2
+
+ result = runner.invoke(app, ["claims", "search", "throughput", "--json"])
+ assert result.exit_code == 0
+ payload = json.loads(result.stdout)
+ assert payload["count"] == 1
+
+
+class TestEmbeddings:
+ def _fake_embedder(self, monkeypatch):
+ """Deterministic fake: vector derived from text hash. Counts calls."""
+ calls = {"batches": 0, "texts": 0}
+
+ def fake(provider, model, texts):
+ calls["batches"] += 1
+ calls["texts"] += len(texts)
+ out = []
+ for t in texts:
+ h = sum(ord(c) for c in t) % 97
+ out.append([float(h), 1.0, float(len(t) % 13)])
+ return out
+
+ monkeypatch.setattr(embed, "_http_embed", fake)
+ return calls
+
+ def test_provider_none_raises_cleanly(self, seeded_vault):
+ with pytest.raises(EmbeddingError, match="disabled"):
+ embed.embed_sync(seeded_vault)
+
+ def test_embed_sync_and_incremental(self, seeded_vault, monkeypatch):
+ calls = self._fake_embedder(monkeypatch)
+ cfg_path = seeded_vault.config_path
+ cfg_path.write_text(
+ cfg_path.read_text(encoding="utf-8").replace(
+ 'provider = "none"', 'provider = "openai"'
+ ),
+ encoding="utf-8",
+ )
+ # Fresh Vault object so the edited config is re-read
+ vault = type(seeded_vault).discover(seeded_vault.root)
+
+ r1 = embed.embed_sync(vault)
+ assert r1["embedded"] >= 4
+ assert r1["provider"] == "openai"
+ assert calls["texts"] == r1["embedded"]
+
+ r2 = embed.embed_sync(vault)
+ assert r2["embedded"] == 0 # unchanged content -> no re-embedding
+ assert r2["skipped"] >= 4
+
+ hits = embed.semantic_search(vault, "concurrency patterns", limit=3)
+ assert len(hits) == 3
+ assert all("id" in h and "score" in h for h in hits)
+
+ def test_cosine(self):
+ assert cosine([1, 0], [1, 0]) == pytest.approx(1.0)
+ assert cosine([1, 0], [0, 1]) == pytest.approx(0.0)
+ assert cosine([], []) == 0.0
+
+ def test_rrf_fusion(self):
+ fused = reciprocal_rank_fusion([["a", "b", "c"], ["b", "a", "d"]])
+ ids = [x[0] for x in fused]
+ # a and b appear in both lists -> outrank c and d
+ assert set(ids[:2]) == {"a", "b"}
+
+ def test_vector_pack_roundtrip(self):
+ vec = [0.5, -1.25, 3.0]
+ assert embed._unpack(embed._pack(vec)) == vec
diff --git a/tests/test_core/test_config_sections.py b/tests/test_core/test_config_sections.py
new file mode 100644
index 0000000..b349ec9
--- /dev/null
+++ b/tests/test_core/test_config_sections.py
@@ -0,0 +1,277 @@
+"""Tests for the Phase-1 config sections ([fetch]/[junk]/[assets]/[dedup]/[lint])."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from hyperresearch.core.config import (
+ AssetSettings,
+ DedupSettings,
+ FetchSettings,
+ JunkGates,
+ LintSettings,
+ VaultConfig,
+)
+
+
+class TestDefaults:
+ """Absent config keys must reproduce pre-2.0 behavior exactly."""
+
+ def test_missing_file_gives_defaults(self, tmp_path: Path):
+ cfg = VaultConfig.load(tmp_path / "nope.toml")
+ assert cfg.fetch == FetchSettings()
+ assert cfg.junk == JunkGates()
+ assert cfg.assets == AssetSettings()
+ assert cfg.dedup == DedupSettings()
+ assert cfg.lint == LintSettings()
+
+ def test_default_values_match_pre_20_literals(self):
+ f = FetchSettings()
+ assert f.page_timeout_ms == 30000
+ assert f.pdf_timeout_s == 30
+ assert f.min_pdf_bytes == 100
+ assert f.wait_initial_ms == 2000
+ assert f.poll_interval_ms == 500
+ assert f.stable_checks == 2
+ assert f.max_checks == 16
+ assert f.image_timeout_s == 15
+ assert "linkedin.com" in f.visible_browser_domains
+ assert len(f.visible_browser_domains) == 6
+
+ j = JunkGates()
+ assert j.min_content_chars == 300
+ assert j.login_wall_max_chars == 1000
+ assert j.cookie_wall_max_chars == 1500
+ assert j.binary_garbage_ratio == 0.05
+ assert j.sample_window == 2000
+ assert j.login_sample_chars == 500
+
+ a = AssetSettings()
+ assert a.max_images == 5
+ assert a.min_image_bytes == 50_000
+
+ d = DedupSettings()
+ assert d.shingle_size == 3
+ assert d.minhash_perm == 128
+ assert d.lsh_bands == 16
+ assert d.lsh_switchover == 200
+ assert d.default_threshold == 0.6
+
+ li = LintSettings()
+ assert li.extract_min_words == 150
+ assert li.extract_coverage_divisor == 3
+ assert li.stale_review_days == 90
+
+ def test_pdf_verify_tls_defaults_true(self):
+ # Deliberate 2.0 behavior change: TLS verification ON by default
+ # (pre-2.0 code hardcoded verify=False for PDF downloads).
+ assert FetchSettings().pdf_verify_tls is True
+
+
+class TestLoadOverrides:
+ def test_partial_section_override(self, tmp_path: Path):
+ p = tmp_path / "config.toml"
+ p.write_text(
+ "[fetch]\npage_timeout_ms = 60000\n\n"
+ "[junk]\nmin_content_chars = 50\n\n"
+ "[assets]\nmax_images = 12\n\n"
+ "[dedup]\ndefault_threshold = 0.8\n\n"
+ "[lint]\nstale_review_days = 30\n",
+ encoding="utf-8",
+ )
+ cfg = VaultConfig.load(p)
+ # Overridden keys take effect
+ assert cfg.fetch.page_timeout_ms == 60000
+ assert cfg.junk.min_content_chars == 50
+ assert cfg.assets.max_images == 12
+ assert cfg.dedup.default_threshold == 0.8
+ assert cfg.lint.stale_review_days == 30
+ # Untouched keys keep defaults
+ assert cfg.fetch.pdf_timeout_s == 30
+ assert cfg.junk.binary_garbage_ratio == 0.05
+ assert cfg.assets.min_image_bytes == 50_000
+
+ def test_toml_arrays_become_tuples(self, tmp_path: Path):
+ p = tmp_path / "config.toml"
+ p.write_text(
+ '[fetch]\nvisible_browser_domains = ["example.com"]\n\n'
+ '[junk]\nextra_login_signals = ["mitgliedsbereich"]\n',
+ encoding="utf-8",
+ )
+ cfg = VaultConfig.load(p)
+ assert cfg.fetch.visible_browser_domains == ("example.com",)
+ assert cfg.junk.extra_login_signals == ("mitgliedsbereich",)
+
+ def test_unknown_keys_ignored(self, tmp_path: Path):
+ p = tmp_path / "config.toml"
+ p.write_text("[fetch]\nfuture_knob = 42\npage_timeout_ms = 1234\n", encoding="utf-8")
+ cfg = VaultConfig.load(p)
+ assert cfg.fetch.page_timeout_ms == 1234
+
+ def test_search_output_defaults(self, tmp_path: Path):
+ cfg = VaultConfig.load(tmp_path / "nope.toml")
+ assert cfg.search_default_limit == 20
+ assert cfg.search_chars_per_token == 4
+ assert cfg.search_snippet_len == 200
+ p = tmp_path / "config.toml"
+ p.write_text("[search]\ndefault_limit = 50\nsnippet_len = 400\n", encoding="utf-8")
+ cfg2 = VaultConfig.load(p)
+ assert cfg2.search_default_limit == 50
+ assert cfg2.search_snippet_len == 400
+ assert cfg2.search_chars_per_token == 4
+
+
+class TestSaveLoadRoundtrip:
+ def test_roundtrip_preserves_sections(self, tmp_path: Path):
+ cfg = VaultConfig(
+ fetch=FetchSettings(page_timeout_ms=45000, pdf_verify_tls=False),
+ junk=JunkGates(min_content_chars=100, extra_junk_signals=("custom wall",)),
+ assets=AssetSettings(max_images=9),
+ dedup=DedupSettings(lsh_switchover=500),
+ lint=LintSettings(extract_min_words=200),
+ )
+ p = tmp_path / "config.toml"
+ cfg.save(p)
+ loaded = VaultConfig.load(p)
+ assert loaded.fetch == cfg.fetch
+ assert loaded.junk == cfg.junk
+ assert loaded.assets == cfg.assets
+ assert loaded.dedup == cfg.dedup
+ assert loaded.lint == cfg.lint
+
+ def test_default_roundtrip_is_identity(self, tmp_path: Path):
+ cfg = VaultConfig()
+ p = tmp_path / "config.toml"
+ cfg.save(p)
+ assert VaultConfig.load(p) == cfg
+
+
+class TestPipelineGear:
+ def test_default_gear_is_full(self, tmp_path: Path):
+ cfg = VaultConfig.load(tmp_path / "nope.toml")
+ assert cfg.pipeline_profile == "full"
+
+ def test_gear_roundtrip(self, tmp_path: Path):
+ cfg = VaultConfig(pipeline_profile="premier")
+ p = tmp_path / "config.toml"
+ cfg.save(p)
+ loaded = VaultConfig.load(p)
+ assert loaded.pipeline_profile == "premier"
+
+ def test_gear_loads_from_pipeline_section(self, tmp_path: Path):
+ p = tmp_path / "config.toml"
+ p.write_text('[pipeline]\nprofile = "premier"\n', encoding="utf-8")
+ assert VaultConfig.load(p).pipeline_profile == "premier"
+
+
+class TestProfileOverlayRoundtrip:
+ """Regression: save() used to drop [profile.*] tables entirely —
+ any config write silently destroyed user-defined pipeline profiles."""
+
+ def test_overlays_survive_save(self, tmp_path: Path):
+ p = tmp_path / "config.toml"
+ p.write_text(
+ "[profile.megareview]\n"
+ "source_min = 250\n"
+ "loci_max = 20\n"
+ 'extends = "full"\n'
+ "source_target = [300, 450]\n"
+ "depth_budget_brackets = [[35, 20], [0, 5]]\n"
+ 'must_read = { argumentative = [50, 70], short = [20, 30] }\n'
+ "utility_scoring = false\n"
+ 'time_estimate = "~4 hours"\n',
+ encoding="utf-8",
+ )
+ cfg = VaultConfig.load(p)
+ assert "megareview" in cfg.profile_overlays
+ cfg.save(p)
+
+ # The overlay must still parse as valid TOML and resolve as a profile
+ reloaded = VaultConfig.load(p)
+ assert reloaded.profile_overlays == cfg.profile_overlays
+
+ from hyperresearch.core.profiles import resolve_profile
+
+ prof = resolve_profile("megareview", p)
+ assert prof.source_min == 250
+ assert prof.source_target == (300, 450)
+ assert prof.depth_budget_brackets == ((35, 20), (0, 5))
+ assert prof.must_read["argumentative"] == (50, 70)
+ assert prof.utility_scoring is False
+ assert prof.time_estimate == "~4 hours"
+
+ def test_builtin_override_survives_save(self, tmp_path: Path):
+ p = tmp_path / "config.toml"
+ p.write_text("[profile.full]\nsource_min = 60\n", encoding="utf-8")
+ cfg = VaultConfig.load(p)
+ cfg.save(p)
+
+ from hyperresearch.core.profiles import resolve_profile
+
+ assert resolve_profile("full", p).source_min == 60
+
+
+class TestGateThreading:
+ """Custom thresholds must actually change gate behavior."""
+
+ def _result(self, content: str, title: str = "Some Page", url: str = "https://example.com/a"):
+ from hyperresearch.web.base import WebResult
+
+ return WebResult(url=url, title=title, content=content)
+
+ def test_min_content_chars_gate(self):
+ text = "word " * 80 # 400 chars, real prose
+ assert self._result(text).looks_like_junk() is None
+ strict = JunkGates(min_content_chars=1000)
+ assert self._result(text).looks_like_junk(strict) == "Empty or near-empty content"
+
+ def test_binary_garbage_ratio_gate(self):
+ # ~10% control characters: junk at default 0.05, fine at 0.5
+ # Pad well past every length gate (cookie wall fires under 1500 chars).
+ base = ("normal text with plenty of ordinary words here " * 40) + "\x01\x02" * 120
+ assert self._result(base).looks_like_junk() == "High ratio of binary/non-printable content"
+ lax = JunkGates(binary_garbage_ratio=0.5)
+ assert self._result(base).looks_like_junk(lax) is None
+
+ def test_extra_junk_signals(self):
+ text = "This portal is protected by MegaShield gateway. " * 40
+ assert self._result(text).looks_like_junk() is None
+ gates = JunkGates(extra_junk_signals=("megashield gateway",))
+ reason = self._result(text).looks_like_junk(gates)
+ assert reason is not None and reason.startswith("Bot detection page")
+
+ def test_login_wall_threshold(self):
+ # Short page mentioning login: wall at default, passes when the
+ # content-length gate is tightened below the text length.
+ text = "Please login to continue reading this article about widgets."
+ r = self._result(text, title="Widgets Weekly")
+ assert r.looks_like_login_wall("https://example.com/a") is True
+ lax = JunkGates(login_wall_max_chars=10)
+ assert r.looks_like_login_wall("https://example.com/a", lax) is False
+
+ def test_extra_login_signals(self):
+ text = "Mitgliedsbereich: bitte anmelden. " + ("Inhalt " * 50)
+ r = self._result(text, title="Mitgliedsbereich")
+ assert r.looks_like_login_wall("https://example.com/a") is False
+ gates = JunkGates(extra_login_signals=("mitgliedsbereich",))
+ assert r.looks_like_login_wall("https://example.com/a", gates) is True
+
+
+class TestVaultConfigIntegration:
+ def test_vault_config_file_includes_sections(self, tmp_vault):
+ # Vault.init saves a config that already carries the new sections
+ cfg_path = tmp_vault.root / ".hyperresearch" / "config.toml"
+ text = cfg_path.read_text(encoding="utf-8")
+ assert "[junk]" in text and "[fetch]" in text
+
+ def test_vault_picks_up_section_override(self, tmp_vault):
+ # Edit a key inside the existing [junk] section and reload
+ cfg_path = tmp_vault.root / ".hyperresearch" / "config.toml"
+ existing = cfg_path.read_text(encoding="utf-8")
+ assert "min_content_chars = 300" in existing
+ cfg_path.write_text(
+ existing.replace("min_content_chars = 300", "min_content_chars = 77"),
+ encoding="utf-8",
+ )
+ reloaded = VaultConfig.load(cfg_path)
+ assert reloaded.junk.min_content_chars == 77
diff --git a/tests/test_core/test_dissertation_profile.py b/tests/test_core/test_dissertation_profile.py
new file mode 100644
index 0000000..9aa4fd3
--- /dev/null
+++ b/tests/test_core/test_dissertation_profile.py
@@ -0,0 +1,130 @@
+"""Phase-3 tests: dissertation profile + literature matrix + target grouping."""
+
+from __future__ import annotations
+
+import json
+
+from hyperresearch.core.claims import (
+ group_by_target,
+ ingest_claims_dir,
+ literature_matrix,
+ render_matrix_markdown,
+)
+from hyperresearch.core.profiles import resolve_profile
+
+
+class TestDissertationProfile:
+ def test_builtin_resolves(self):
+ p = resolve_profile("dissertation")
+ assert p.chapters == (4, 10)
+ assert p.chapter_concurrency == 2
+ assert p.chapter_source_target == (40, 80)
+ assert p.source_min == 250
+ assert p.source_target == (300, 450)
+ assert p.loci_max == 20
+ assert p.depth_budget_total == 160
+ assert p.draft_count == 1
+ assert p.word_targets["dissertation"] == (25000, 80000)
+ assert p.must_read["dissertation"] == (35, 50)
+ assert p.critic_finding_caps["instruction"] == 30
+
+ def test_flat_profiles_are_unchaptered(self):
+ assert resolve_profile("full").chapters == (0, 0)
+ assert resolve_profile("light").chapters == (0, 0)
+
+ def test_extends_dissertation(self, tmp_path):
+ cfg = tmp_path / "config.toml"
+ cfg.write_text(
+ '[profile.thesis]\nextends = "dissertation"\nsource_min = 150\n',
+ encoding="utf-8",
+ )
+ p = resolve_profile("thesis", cfg)
+ assert p.source_min == 150
+ assert p.chapters == (4, 10) # inherited
+
+ def test_chapter_range_validated(self, tmp_path):
+ import pytest
+
+ from hyperresearch.core.profiles import ProfileError
+
+ cfg = tmp_path / "config.toml"
+ cfg.write_text("[profile.dissertation]\nchapters = [10, 4]\n", encoding="utf-8")
+ with pytest.raises(ProfileError, match="invalid profile"):
+ resolve_profile("dissertation", cfg)
+
+
+class TestLiteratureMatrix:
+ def _seed_claims(self, vault):
+ temp = vault.root / "research" / "temp"
+ temp.mkdir(parents=True, exist_ok=True)
+ (temp / "claims-python-async-patterns.json").write_text(
+ json.dumps([
+ {"claim": "Async gives 10x throughput", "numbers": ["10x"],
+ "evidence_type": "empirical", "confidence": "high",
+ "stance_target": "async-throughput", "stance": "supports"},
+ {"claim": "Async is hard to debug", "evidence_type": "opinion",
+ "stance_target": "async-throughput", "stance": "disputes"},
+ ]),
+ encoding="utf-8",
+ )
+ (temp / "claims-rust-ownership.json").write_text(
+ json.dumps([
+ {"claim": "Ownership eliminates data races", "evidence_type": "empirical",
+ "stance_target": "async-throughput", "stance": "supports",
+ "numbers": ["100%"]},
+ ]),
+ encoding="utf-8",
+ )
+ ingest_claims_dir(vault, vault_tag="mx-run")
+
+ def test_matrix_rows(self, seeded_vault):
+ self._seed_claims(seeded_vault)
+ conn = seeded_vault.db
+ conn.execute("UPDATE notes SET quality_score = 0.9, tier = 'institutional' WHERE id = 'python-async-patterns'")
+ conn.execute("UPDATE notes SET quality_score = 0.3 WHERE id = 'rust-ownership'")
+ conn.commit()
+
+ rows = literature_matrix(conn, vault_tag="mx-run")
+ assert len(rows) == 2
+ assert rows[0]["id"] == "python-async-patterns" # sorted by quality desc
+ assert rows[0]["n_claims"] == 2
+ assert rows[0]["n_empirical"] == 1
+ assert rows[0]["key_claim"] == "Async gives 10x throughput" # empirical+quantified wins
+
+ def test_matrix_markdown_render(self, seeded_vault):
+ self._seed_claims(seeded_vault)
+ rows = literature_matrix(seeded_vault.db)
+ md = render_matrix_markdown(rows)
+ assert md.startswith("| Source |")
+ assert "[[python-async-patterns]]" in md
+
+ def test_matrix_cli_writes_file(self, seeded_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ self._seed_claims(seeded_vault)
+ monkeypatch.chdir(seeded_vault.root)
+ runner = CliRunner()
+ out = seeded_vault.root / "research" / "temp" / "lit-matrix.md"
+ r = runner.invoke(app, ["claims", "matrix", "--tag", "mx-run", "--out", str(out), "--json"])
+ assert r.exit_code == 0
+ assert out.exists()
+ assert "| Source |" in out.read_text(encoding="utf-8")
+
+
+class TestTargetGrouping:
+ def test_groups_across_sources(self, seeded_vault):
+ TestLiteratureMatrix()._seed_claims(seeded_vault)
+ groups = group_by_target(seeded_vault.db, vault_tag="mx-run", min_sources=2)
+ assert len(groups) == 1
+ g = groups[0]
+ assert g["stance_target"] == "async-throughput"
+ assert g["n_sources"] == 2
+ assert g["stances"] == {"supports": 2, "disputes": 1}
+ assert {v["note_id"] for v in g["quantified"]} == {"python-async-patterns", "rust-ownership"}
+
+ def test_min_sources_filters_singletons(self, seeded_vault):
+ TestLiteratureMatrix()._seed_claims(seeded_vault)
+ groups = group_by_target(seeded_vault.db, min_sources=3)
+ assert groups == []
diff --git a/tests/test_core/test_escalation.py b/tests/test_core/test_escalation.py
new file mode 100644
index 0000000..828a14c
--- /dev/null
+++ b/tests/test_core/test_escalation.py
@@ -0,0 +1,245 @@
+"""Phase-4 tests: escalation queue, fetch-gate enqueue, chrome policy, ingest."""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from hyperresearch.core.escalation import (
+ EscalationError,
+ claim_next,
+ enqueue,
+ list_items,
+ maybe_enqueue_blocked_fetch,
+ queue_stats,
+ resolve,
+)
+
+
+class TestQueueBasics:
+ def test_enqueue_and_list(self, tmp_vault):
+ conn = tmp_vault.db
+ item_id = enqueue(conn, "https://x.com/a", "login_wall", vault_tag="run-1", utility_score=14)
+ assert item_id is not None
+ items = list_items(conn, vault_tag="run-1")
+ assert len(items) == 1
+ assert items[0]["status"] == "queued"
+ assert items[0]["reason"] == "login_wall"
+
+ def test_duplicate_enqueue_is_idempotent(self, tmp_vault):
+ conn = tmp_vault.db
+ first = enqueue(conn, "https://x.com/a", "login_wall", vault_tag="run-1")
+ second = enqueue(conn, "https://x.com/a", "bot_block", vault_tag="run-1")
+ assert first is not None and second is None
+ assert len(list_items(conn)) == 1
+
+ def test_same_url_different_runs_both_queue(self, tmp_vault):
+ conn = tmp_vault.db
+ assert enqueue(conn, "https://x.com/a", "login_wall", vault_tag="run-1")
+ assert enqueue(conn, "https://x.com/a", "login_wall", vault_tag="run-2")
+
+ def test_invalid_reason_rejected(self, tmp_vault):
+ with pytest.raises(EscalationError, match="invalid reason"):
+ enqueue(tmp_vault.db, "https://x.com", "paywall")
+
+
+class TestClaimSemantics:
+ def test_claim_takes_highest_utility_first(self, tmp_vault):
+ conn = tmp_vault.db
+ enqueue(conn, "https://low.com", "bot_block", utility_score=5)
+ enqueue(conn, "https://high.com", "bot_block", utility_score=17)
+ enqueue(conn, "https://unscored.com", "bot_block")
+ item = claim_next(conn, "tester")
+ assert item["url"] == "https://high.com"
+ assert item["status"] == "in_progress"
+ assert item["claimed_by"] == "tester"
+ assert item["attempts"] == 1
+
+ def test_claims_never_hand_out_same_item(self, tmp_vault):
+ conn = tmp_vault.db
+ enqueue(conn, "https://only.com", "bot_block")
+ a = claim_next(conn, "a")
+ b = claim_next(conn, "b")
+ assert a is not None
+ assert b is None # queue exhausted, not double-claimed
+
+ def test_claim_scoped_by_tag(self, tmp_vault):
+ conn = tmp_vault.db
+ enqueue(conn, "https://other.com", "bot_block", vault_tag="other-run")
+ assert claim_next(conn, "t", vault_tag="my-run") is None
+
+ def test_lifecycle_to_needs_human_and_retry(self, tmp_vault):
+ conn = tmp_vault.db
+ item_id = enqueue(conn, "https://x.com/c", "captcha")
+ claim_next(conn, "t")
+ item = resolve(conn, item_id, "needs_human", detail="solve CAPTCHA on x.com")
+ assert item["status"] == "needs_human"
+ assert queue_stats(conn)["needs_human"] == 1
+ # human done -> retry
+ item = resolve(conn, item_id, "queued")
+ assert item["status"] == "queued"
+ assert claim_next(conn, "t")["id"] == item_id
+
+
+class TestChromePolicy:
+ def test_low_utility_declined(self, tmp_vault):
+ assert maybe_enqueue_blocked_fetch(tmp_vault, "https://x.com", "bot_block", utility_score=3.0) is None
+ assert maybe_enqueue_blocked_fetch(tmp_vault, "https://x.com", "bot_block", utility_score=9.0) is not None
+
+ def test_unscored_urls_pass(self, tmp_vault):
+ assert maybe_enqueue_blocked_fetch(tmp_vault, "https://y.com", "login_wall") is not None
+
+ def test_disabled_lane_declines(self, tmp_vault):
+ cfg = tmp_vault.config_path
+ cfg.write_text(cfg.read_text(encoding="utf-8").replace(
+ "[chrome]\nenabled = true", "[chrome]\nenabled = false"
+ ), encoding="utf-8")
+ vault = type(tmp_vault).discover(tmp_vault.root)
+ assert vault.config.chrome.enabled is False
+ assert maybe_enqueue_blocked_fetch(vault, "https://z.com", "login_wall") is None
+
+ def test_per_run_cap(self, tmp_vault):
+ cfg = tmp_vault.config_path
+ cfg.write_text(cfg.read_text(encoding="utf-8").replace(
+ "max_items_per_run = 25", "max_items_per_run = 2"
+ ), encoding="utf-8")
+ vault = type(tmp_vault).discover(tmp_vault.root)
+ assert maybe_enqueue_blocked_fetch(vault, "https://a.com", "login_wall", vault_tag="capped") is not None
+ assert maybe_enqueue_blocked_fetch(vault, "https://b.com", "login_wall", vault_tag="capped") is not None
+ assert maybe_enqueue_blocked_fetch(vault, "https://c.com", "login_wall", vault_tag="capped") is None
+
+
+class TestFetchGateIntegration:
+ def test_login_wall_fetch_escalates(self, tmp_vault, monkeypatch):
+ """A CLI fetch that hits a login wall queues the URL instead of just dying."""
+ from typer.testing import CliRunner
+
+ import hyperresearch.cli.fetch as fetch_mod
+ from hyperresearch.cli import app
+ from hyperresearch.web.base import WebResult
+
+ class WallProvider:
+ name = "stub"
+
+ def fetch(self, url):
+ return WebResult(url=url, title="Sign in to continue", content="Please log in.")
+
+ monkeypatch.setattr(fetch_mod, "get_provider", lambda *a, **k: WallProvider(), raising=False)
+ # cli.fetch imports get_provider inside the function from web.base
+ import hyperresearch.web.base as base_mod
+
+ monkeypatch.setattr(base_mod, "get_provider", lambda *a, **k: WallProvider())
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ r = runner.invoke(app, ["fetch", "https://walled.example.com/article",
+ "--tag", "esc-run", "--utility-score", "15", "--json"])
+ assert r.exit_code == 1
+ payload = json.loads(r.stdout)
+ assert payload["error_code"] == "AUTH_REQUIRED_ESCALATED"
+
+ items = list_items(tmp_vault.db, vault_tag="esc-run")
+ assert len(items) == 1
+ assert items[0]["url"] == "https://walled.example.com/article"
+ assert items[0]["reason"] == "login_wall"
+ assert items[0]["utility_score"] == 15.0
+
+
+class TestEscalationCli:
+ def test_add_claim_ingest_roundtrip(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+
+ r = runner.invoke(app, ["escalation", "add", "https://gated.example.com/paper",
+ "--reason", "interactive_needed", "--tag", "cli-esc",
+ "--utility", "12", "--json"])
+ assert r.exit_code == 0
+ item_id = json.loads(r.stdout)["data"]["id"]
+
+ r = runner.invoke(app, ["escalation", "claim", "--tag", "cli-esc", "--json"])
+ assert r.exit_code == 0
+ assert json.loads(r.stdout)["data"]["item"]["id"] == item_id
+
+ body = tmp_vault.root / "scratch-body.md"
+ body.write_text("Extracted page content with plenty of real words in it.", encoding="utf-8")
+ r = runner.invoke(app, ["escalation", "ingest", str(item_id),
+ "--title", "Gated Paper", "--body-file", str(body), "--json"])
+ assert r.exit_code == 0
+ data = json.loads(r.stdout)["data"]
+ note_id = data["note_id"]
+
+ # Note exists with chrome provenance + vault_tag tag; source row recorded
+ row = tmp_vault.db.execute("SELECT * FROM notes WHERE id = ?", (note_id,)).fetchone()
+ assert row is not None
+ src = tmp_vault.db.execute(
+ "SELECT provider, note_id FROM sources WHERE url = 'https://gated.example.com/paper'"
+ ).fetchone()
+ assert src["provider"] == "chrome"
+ assert src["note_id"] == note_id
+ tags = {t["tag"] for t in tmp_vault.db.execute("SELECT tag FROM tags WHERE note_id = ?", (note_id,))}
+ assert "cli-esc" in tags
+ note_text = (tmp_vault.root / data["path"]).read_text(encoding="utf-8")
+ assert "fetch_provider: chrome" in note_text
+
+ # Item resolved
+ assert list_items(tmp_vault.db, vault_tag="cli-esc")[0]["status"] == "fetched"
+
+ def test_human_flow_cli(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ runner.invoke(app, ["escalation", "add", "https://c.example.com", "--reason", "captcha", "--json"])
+ r = runner.invoke(app, ["escalation", "claim", "--json"])
+ item_id = json.loads(r.stdout)["data"]["item"]["id"]
+
+ r = runner.invoke(app, ["escalation", "human", str(item_id),
+ "--detail", "solve CAPTCHA on c.example.com", "--json"])
+ assert r.exit_code == 0
+ r = runner.invoke(app, ["escalation", "retry", str(item_id), "--json"])
+ assert r.exit_code == 0
+ assert json.loads(r.stdout)["data"]["status"] == "queued"
+
+
+class TestRunStatusIntegration:
+ def test_run_status_shows_queue_depth(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+ from hyperresearch.core.runs import init_run
+
+ init_run(tmp_vault, "esc-status-01")
+ enqueue(tmp_vault.db, "https://q.example.com", "login_wall", vault_tag="esc-status-01")
+ item_id = enqueue(tmp_vault.db, "https://h.example.com", "captcha", vault_tag="esc-status-01")
+ resolve(tmp_vault.db, item_id, "needs_human", detail="x")
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ r = runner.invoke(app, ["run", "status", "esc-status-01", "--json"])
+ assert r.exit_code == 0
+ esc = json.loads(r.stdout)["data"]["escalations"]
+ assert esc["queued"] == 1
+ assert esc["needs_human"] == 1
+
+
+class TestBrowserFetcherAgent:
+ def test_agent_installs_with_boundary(self, tmp_vault):
+ from hyperresearch.core.hooks import _install_browser_fetcher_agent
+
+ result = _install_browser_fetcher_agent(tmp_vault.root, "hyperresearch")
+ assert result is not None
+ body = (tmp_vault.root / ".claude" / "agents" / "hyperresearch-browser-fetcher.md").read_text(encoding="utf-8")
+ assert "name: hyperresearch-browser-fetcher" in body
+ assert "model: sonnet" in body
+ assert "NEVER" in body and "CAPTCHA" in body
+ assert "escalation human" in body
+ assert "escalation ingest" in body
+ assert "scholar.google.com" in body
+ assert "<<" not in body # fully rendered
diff --git a/tests/test_core/test_hooks.py b/tests/test_core/test_hooks.py
index 231a119..41b70a3 100644
--- a/tests/test_core/test_hooks.py
+++ b/tests/test_core/test_hooks.py
@@ -63,10 +63,10 @@ def test_install_hyperresearch_skill_idempotent(tmp_vault):
assert second is None
-def test_install_hyperresearch_step_skills_creates_all_16(tmp_vault):
+def test_install_hyperresearch_step_skills_creates_all(tmp_vault):
"""Each step skill lives at .claude/skills/hyperresearch-N-name/SKILL.md so the
- orchestrator can invoke it via the Skill tool. All 16 must install on a
- fresh vault."""
+ orchestrator can invoke it via the Skill tool. All 18 (16 numbered +
+ 1.5 chapter-partition + 14.5 cite-check) must install on a fresh vault."""
from hyperresearch.core.hooks import (
_HYPERRESEARCH_STEP_SKILLS,
_install_hyperresearch_step_skills,
@@ -74,7 +74,7 @@ def test_install_hyperresearch_step_skills_creates_all_16(tmp_vault):
result = _install_hyperresearch_step_skills(tmp_vault.root)
assert result is not None
- assert len(_HYPERRESEARCH_STEP_SKILLS) == 16
+ assert len(_HYPERRESEARCH_STEP_SKILLS) == 18
skills_root = tmp_vault.root / ".claude" / "skills"
for skill_name in _HYPERRESEARCH_STEP_SKILLS:
@@ -370,6 +370,8 @@ def test_install_hooks_registers_full_hyperresearch_roster(tmp_vault):
"hyperresearch-readability-recommender.md",
"hyperresearch-draft-orchestrator.md",
"hyperresearch-synthesizer.md",
+ "hyperresearch-browser-fetcher.md",
+ "hyperresearch-cite-checker.md",
}
actual_agents = {p.name for p in agents_dir.iterdir() if p.is_file()}
assert expected_agents == actual_agents, (
diff --git a/tests/test_core/test_profiles.py b/tests/test_core/test_profiles.py
new file mode 100644
index 0000000..02c6e17
--- /dev/null
+++ b/tests/test_core/test_profiles.py
@@ -0,0 +1,313 @@
+"""Tests for the pipeline profile system (core/profiles.py + `hpr profile`)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from hyperresearch.core.profiles import (
+ BUILTIN_PROFILES,
+ ProfileError,
+ list_profiles,
+ resolve_profile,
+)
+
+
+class TestBuiltins:
+ def test_full_matches_shipped_pipeline_values(self):
+ p = resolve_profile("full")
+ # These pins mirror the V8 skill prose; the render golden tests depend
+ # on them staying in lockstep with the templates.
+ assert p.steps == tuple(range(1, 17))
+ assert p.source_min == 45
+ assert p.source_target == (55, 80)
+ assert p.batch_count == (10, 12)
+ assert p.batch_size == (8, 12)
+ assert p.wave1_fetchers == (10, 12)
+ assert p.adversarial_searches_min == 5
+ assert p.source_analyst_cap == 6
+ assert p.source_analyst_word_trigger == 5000
+ assert p.loci_analysts == 2
+ assert p.loci_max == 6
+ assert p.depth_budget_total == 40
+ assert p.depth_budget_brackets == ((30, 15), (20, 10), (10, 5), (0, 3))
+ assert p.investigator_max == 6
+ assert p.claims_cap == (80, 120)
+ assert p.claims_min == 30
+ assert p.draft_count == 3
+ assert p.must_read["argumentative"] == (35, 50)
+ assert p.word_targets["argumentative"] == (5000, 10000)
+ assert p.critic_finding_caps == {"dialectic": 12, "depth": 12, "width": 10, "instruction": 15}
+ assert p.gap_fetch_cap == 5
+ assert p.readability_rec_cap == 50
+ assert p.models.fetcher == "sonnet"
+ assert p.models.synthesizer == "opus"
+
+ def test_light_matches_shipped_pipeline_values(self):
+ p = resolve_profile("light")
+ assert p.steps == (1, 2, 10, 15, 16)
+ assert p.source_min == 10
+ assert p.source_target == (15, 25)
+ assert p.wave1_fetchers == (3, 5)
+ assert p.utility_scoring is False
+ assert p.draft_count == 1
+ assert p.single_draft_reads == (8, 15)
+
+ def test_list_builtins(self):
+ # Ascending scale order — `hpr profile list` follows this.
+ assert list_profiles() == ["light", "full", "premier", "dissertation"]
+
+ def test_all_builtins_validate(self):
+ for name in BUILTIN_PROFILES:
+ resolve_profile(name) # must not raise
+
+ def test_all_builtins_have_descriptions(self):
+ for name in BUILTIN_PROFILES:
+ assert resolve_profile(name).description, f"{name} needs a description"
+
+ def test_premier_scales_up_the_flat_pipeline(self):
+ p = resolve_profile("premier")
+ full = resolve_profile("full")
+ # Same step sequence as full — premier is a scale gear, not a tier.
+ assert p.steps == full.steps
+ assert p.chapters == (0, 0) # unchaptered
+ # Width roughly doubles
+ assert p.source_min == 90
+ assert p.source_target == (100, 130)
+ assert p.wave1_fetchers == (14, 18)
+ assert p.adversarial_searches_min == 8
+ # Depth doubles
+ assert p.loci_max == 10
+ assert p.depth_budget_total == 80
+ assert p.investigator_max == 10
+ # The downstream funnel widens too — raising only fetch targets would
+ # strand the extra corpus in the vault.
+ assert p.claims_cap == (150, 220)
+ assert p.must_read["argumentative"] == (50, 70)
+ assert p.word_targets["argumentative"] == (8000, 16000)
+ assert p.citation_totals["argumentative"] == (120, 220)
+ # Every scale knob is >= full's
+ assert p.source_min > full.source_min
+ assert p.depth_budget_total > full.depth_budget_total
+ assert p.critic_finding_caps["dialectic"] > full.critic_finding_caps["dialectic"]
+
+ def test_gear_profiles_are_valid_builtins(self):
+ from hyperresearch.core.profiles import GEAR_PROFILES
+
+ assert GEAR_PROFILES == ("full", "premier")
+ for name in GEAR_PROFILES:
+ assert name in BUILTIN_PROFILES
+
+ def test_modelmap_covers_every_installed_agent(self):
+ from hyperresearch.core.profiles import ModelMap
+
+ # One field per installed agent role (the four critics share `critics`).
+ # If an agent is added to hooks.py, it needs a ModelMap field too —
+ # otherwise its `model: << p.models.X >>` template line can't render.
+ assert set(ModelMap.model_fields) == {
+ "fetcher", "source_analyst", "loci_analyst", "depth_investigator",
+ "corpus_critic", "cite_checker", "browser_fetcher",
+ "draft_orchestrator", "synthesizer", "critics", "patcher",
+ "polish_auditor", "readability_recommender",
+ }
+
+ def test_no_cost_estimates_anywhere(self):
+ from hyperresearch.core.profiles import Profile
+
+ # Dollar-cost estimates were removed deliberately: on subscription
+ # billing they are not a bill, and stating them as costs contradicts
+ # how most users run the pipeline. Time estimates remain.
+ assert "cost_estimate" not in Profile.model_fields
+ assert "time_estimate" in Profile.model_fields
+
+
+class TestUserOverlay:
+ def _write(self, tmp_path: Path, body: str) -> Path:
+ p = tmp_path / "config.toml"
+ p.write_text(body, encoding="utf-8")
+ return p
+
+ def test_override_builtin_key(self, tmp_path: Path):
+ cfg = self._write(tmp_path, "[profile.full]\nsource_min = 60\n")
+ p = resolve_profile("full", cfg)
+ assert p.source_min == 60
+ # untouched keys keep built-in values
+ assert p.source_target == (55, 80)
+
+ def test_new_profile_extends_full_by_default(self, tmp_path: Path):
+ cfg = self._write(
+ tmp_path,
+ "[profile.megareview]\nsource_min = 250\nloci_max = 20\n",
+ )
+ p = resolve_profile("megareview", cfg)
+ assert p.name == "megareview"
+ assert p.extends == "full"
+ assert p.source_min == 250
+ assert p.loci_max == 20
+ assert p.draft_count == 3 # inherited from full
+
+ def test_new_profile_extends_light(self, tmp_path: Path):
+ cfg = self._write(
+ tmp_path,
+ '[profile.micro]\nextends = "light"\nsource_min = 5\n',
+ )
+ p = resolve_profile("micro", cfg)
+ assert p.extends == "light"
+ assert p.source_min == 5
+ assert p.steps == (1, 2, 10, 15, 16)
+
+ def test_range_overrides_from_toml_arrays(self, tmp_path: Path):
+ cfg = self._write(
+ tmp_path,
+ "[profile.full]\nsource_target = [100, 150]\n"
+ "depth_budget_brackets = [[35, 20], [0, 5]]\n",
+ )
+ p = resolve_profile("full", cfg)
+ assert p.source_target == (100, 150)
+ assert p.depth_budget_brackets == ((35, 20), (0, 5))
+
+ def test_listing_includes_user_profiles(self, tmp_path: Path):
+ cfg = self._write(tmp_path, "[profile.dissertation]\nsource_min = 250\n")
+ assert list_profiles(cfg) == ["light", "full", "premier", "dissertation"]
+
+ def test_missing_config_is_fine(self, tmp_path: Path):
+ p = resolve_profile("full", tmp_path / "nope.toml")
+ assert p.source_min == 45
+
+ def test_models_overlay_swaps_one_agent(self, tmp_path: Path):
+ cfg = self._write(tmp_path, '[profile.full]\nmodels = { fetcher = "haiku" }\n')
+ p = resolve_profile("full", cfg)
+ assert p.models.fetcher == "haiku"
+ # unspecified agents keep their defaults
+ assert p.models.source_analyst == "sonnet"
+ assert p.models.synthesizer == "opus"
+
+ def test_models_overlay_accepts_full_model_ids(self, tmp_path: Path):
+ cfg = self._write(
+ tmp_path,
+ '[profile.premier]\nmodels = { fetcher = "claude-haiku-4-5-20251001" }\n',
+ )
+ p = resolve_profile("premier", cfg)
+ assert p.models.fetcher == "claude-haiku-4-5-20251001"
+
+
+class TestValidation:
+ def _write(self, tmp_path: Path, body: str) -> Path:
+ p = tmp_path / "config.toml"
+ p.write_text(body, encoding="utf-8")
+ return p
+
+ def test_unknown_profile(self):
+ with pytest.raises(ProfileError, match="unknown profile"):
+ resolve_profile("nope")
+
+ def test_unknown_extends(self, tmp_path: Path):
+ cfg = self._write(tmp_path, '[profile.x]\nextends = "nope"\n')
+ with pytest.raises(ProfileError, match="extends unknown base"):
+ resolve_profile("x", cfg)
+
+ def test_typo_key_rejected(self, tmp_path: Path):
+ cfg = self._write(tmp_path, "[profile.full]\nsource_minn = 60\n")
+ with pytest.raises(ProfileError, match="invalid profile"):
+ resolve_profile("full", cfg)
+
+ def test_inverted_range_rejected(self, tmp_path: Path):
+ cfg = self._write(tmp_path, "[profile.full]\nsource_target = [80, 55]\n")
+ with pytest.raises(ProfileError, match="invalid profile"):
+ resolve_profile("full", cfg)
+
+ def test_step_out_of_range_rejected(self, tmp_path: Path):
+ cfg = self._write(tmp_path, "[profile.full]\nsteps = [1, 2, 99]\n")
+ with pytest.raises(ProfileError, match="invalid profile"):
+ resolve_profile("full", cfg)
+
+ def test_empty_model_assignment_rejected(self, tmp_path: Path):
+ cfg = self._write(tmp_path, '[profile.full]\nmodels = { fetcher = "" }\n')
+ with pytest.raises(ProfileError, match="invalid profile"):
+ resolve_profile("full", cfg)
+
+ def test_unknown_model_agent_rejected(self, tmp_path: Path):
+ cfg = self._write(tmp_path, '[profile.full]\nmodels = { fetchr = "haiku" }\n')
+ with pytest.raises(ProfileError, match="invalid profile"):
+ resolve_profile("full", cfg)
+
+
+class TestProfileCli:
+ def test_profile_show_json(self, tmp_vault, monkeypatch):
+ import json
+
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["profile", "show", "full", "--json"])
+ assert result.exit_code == 0
+ payload = json.loads(result.stdout)
+ assert payload["ok"] is True
+ assert payload["data"]["source_min"] == 45
+
+ def test_profile_list_json(self, tmp_vault, monkeypatch):
+ import json
+
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["profile", "list", "--json"])
+ assert result.exit_code == 0
+ payload = json.loads(result.stdout)
+ names = [p["name"] for p in payload["data"]["profiles"]]
+ assert "full" in names and "light" in names and "premier" in names
+
+ def test_profile_list_marks_current_gear(self, tmp_vault, monkeypatch):
+ import json
+
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["profile", "list", "--json"])
+ assert result.exit_code == 0
+ payload = json.loads(result.stdout)
+ assert payload["data"]["current_gear"] == "full"
+ rows = {p["name"]: p for p in payload["data"]["profiles"]}
+ assert rows["full"]["current_gear"] is True
+ assert rows["premier"]["current_gear"] is False
+ # Friendly metadata is present for every built-in
+ for name in ("light", "full", "premier", "dissertation"):
+ assert rows[name]["description"]
+ assert rows[name]["kind"] in ("gear", "tier")
+ assert rows["premier"]["kind"] == "gear"
+ assert rows["dissertation"]["kind"] == "tier"
+
+ def test_profile_validate_catches_bad_overlay(self, tmp_vault, monkeypatch):
+ cfg_path = tmp_vault.config_path
+ cfg_path.write_text(
+ cfg_path.read_text(encoding="utf-8") + "\n[profile.full]\nsource_minn = 1\n",
+ encoding="utf-8",
+ )
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["profile", "validate", "--json"])
+ assert result.exit_code == 1
+
+ def test_profile_show_unknown_errors(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["profile", "show", "bogus", "--json"])
+ assert result.exit_code == 1
diff --git a/tests/test_core/test_prompt_golden.py b/tests/test_core/test_prompt_golden.py
new file mode 100644
index 0000000..1e7541c
--- /dev/null
+++ b/tests/test_core/test_prompt_golden.py
@@ -0,0 +1,305 @@
+"""Golden tests: rendering the shipped templates with the `full` profile must
+reproduce the pre-templating prompt content byte-for-byte.
+
+The golden fixtures under tests/fixtures/golden_prompts/ were snapshotted from
+the 1.x (pre-template) prompt sources. Any numeric drift between the built-in
+profiles and the templates shows up here as a diff — changing a profile value
+or template deliberately requires updating the golden, which makes prompt
+changes reviewable instead of silent.
+
+Deliberate deviations already folded into the goldens (2026-07-19):
+ - width-sweep: the three inconsistent full-tier source-target statements
+ (40-100 / 40–80 / 55–80) were unified to the profile value 55–80; the
+ light target 12–20 was unified to the table value 15–25; the tier table's
+ full fetchers-per-wave 8–12 was unified to the Wave-1 value 10–12.
+ - Phase-2 source-ranking additions (2026-07-19): width-sweep gained the
+ utility-scores-travel-with-URLs rule and step 2.7 (persist ranking
+ signals: claims ingest / backfill-doi / sources score / graph rank);
+ triple-draft 10.2 gained ranked-curation via `search --ranked`; the
+ fetcher agent gained the `--utility-score` pass-through section.
+ - Phase-3 per-run workspaces (2026-07-19): every run-scoped artifact path
+ moved from flat research/ to research/runs/<vault_tag>/ (query file is
+ now runs/<tag>/query.md) across ALL skills and agent prompts; the router
+ gained bootstrap step 2.5 (`hpr run init`), manifest-first recovery, and
+ the dissertation tier row + step 1.5 (chapter partition).
+ - Phase-4 browser lane (2026-07-19): width-sweep gained step 2.8 (drain
+ the escalation queue via ONE browser-fetcher; consolidated needs_human
+ prompt; run block --on human-challenges for non-interactive runs); the
+ router gained the "Browser-lane escalations" section; new
+ BROWSER_FETCHER_AGENT golden.
+ - Phase-5 verification (2026-07-19): router tier sequences gained step
+ 14.5 (cite-check) and the final gate gained `run verify`, the
+ retraction sweep, and the three verification lint rules; new
+ CITE_CHECKER_AGENT golden.
+ - Scale gears (2026-07-19): the router gained the "Scale gear (tier ≠
+ gear)" section (full vs premier, `profile use`), premier in the run-init
+ profile list, and a gear-aware frontmatter description; width-sweep's
+ diminishing-returns note now cites the gear ceiling instead of the
+ hardcoded "~65 reference / beyond ~80" full-scale prose. The `full.*`
+ template refs that meant "the installed scale" became `p.*` — byte-
+ identical under the default gear (verified by these goldens), truthful
+ under `premier`.
+ - ModelMap wiring + dollar-cost removal (2026-07-19): every agent's
+ `model:` frontmatter line became `<< p.models.X >>` (rendered from the
+ profile's ModelMap — same values under `full`, but now overridable, e.g.
+ a haiku fetcher); "Runs on Sonnet/Opus" claims left descriptions and
+ prose so a model override can't be contradicted by stale text; dollar
+ figures were removed repo-wide (the router tier table lost its cost
+ column, gap-fetch's "+$1-3 per run" became fetcher-count overhead
+ framing, the source-analyst's "$2-5 per spawn" block became "Effort
+ discipline", and its Sonnet-1M context claims became model-neutral).
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import pytest
+
+from hyperresearch.core import hooks
+from hyperresearch.core.hooks import _read_skill_source
+from hyperresearch.core.render import build_render_context, render_prompt
+
+GOLDEN_DIR = Path(__file__).parent.parent / "fixtures" / "golden_prompts"
+
+GOLDEN_SKILLS = [
+ "hyperresearch",
+ "hyperresearch-2-width-sweep",
+ "hyperresearch-4-loci-analysis",
+ "hyperresearch-5-depth-investigation",
+ "hyperresearch-9-evidence-digest",
+ "hyperresearch-10-triple-draft",
+ "hyperresearch-13-gap-fetch",
+ "hyperresearch-16-readability-audit",
+]
+
+GOLDEN_AGENTS = [
+ "DIALECTIC_CRITIC_AGENT",
+ "DEPTH_CRITIC_AGENT",
+ "WIDTH_CRITIC_AGENT",
+ "INSTRUCTION_CRITIC_AGENT",
+ "READABILITY_REFORMATTER_AGENT",
+ "RESEARCHER_AGENT",
+ "DEPTH_INVESTIGATOR_AGENT",
+ "LOCI_ANALYST_AGENT",
+ "BROWSER_FETCHER_AGENT",
+ "CITE_CHECKER_AGENT",
+]
+
+
[email protected](scope="module")
+def ctx():
+ return build_render_context(None, primary="full")
+
+
[email protected]("skill_name", GOLDEN_SKILLS)
+def test_skill_render_matches_golden(skill_name, ctx):
+ template = _read_skill_source(f"{skill_name}.md")
+ assert template is not None, f"missing skill source {skill_name}.md"
+ rendered = render_prompt(template, ctx)
+ golden = (GOLDEN_DIR / "skills" / f"{skill_name}.md").read_text(encoding="utf-8")
+ assert rendered == golden, (
+ f"render(full) of {skill_name}.md deviates from golden. If the change "
+ "is deliberate, update tests/fixtures/golden_prompts/skills/ and note "
+ "it in the module docstring."
+ )
+
+
[email protected]("const_name", GOLDEN_AGENTS)
+def test_agent_render_matches_golden(const_name, ctx):
+ template = getattr(hooks, const_name)
+ rendered = render_prompt(template, ctx)
+ golden = (GOLDEN_DIR / "agents" / f"{const_name.lower()}.md").read_text(encoding="utf-8")
+ assert rendered == golden, (
+ f"render(full) of {const_name} deviates from golden. If the change is "
+ "deliberate, update tests/fixtures/golden_prompts/agents/."
+ )
+
+
[email protected]("skill_name", GOLDEN_SKILLS)
+def test_no_unrendered_variables_in_skills(skill_name, ctx):
+ rendered = render_prompt(_read_skill_source(f"{skill_name}.md"), ctx)
+ assert "<<" not in rendered and ">>" not in rendered
+
+
+def test_profile_override_changes_render(tmp_path):
+ """End-to-end: a profile overlay must actually change the rendered skill."""
+ cfg = tmp_path / "config.toml"
+ cfg.write_text("[profile.full]\nsource_min = 200\n", encoding="utf-8")
+ ctx = build_render_context(cfg, primary="full")
+ rendered = render_prompt(_read_skill_source("hyperresearch-2-width-sweep.md"), ctx)
+ assert "| `full` | 200 |" in rendered
+
+
[email protected]("skill_name", GOLDEN_SKILLS)
+def test_premier_gear_renders_cleanly(skill_name):
+ """Every golden-covered skill must render without holes at premier gear,
+ and the scale-bearing ones must carry premier numbers."""
+ ctx = build_render_context(None, primary="premier")
+ rendered = render_prompt(_read_skill_source(f"{skill_name}.md"), ctx)
+ assert "<<" not in rendered and ">>" not in rendered
+ if skill_name == "hyperresearch-2-width-sweep":
+ assert "| `full` | 90 | 100–130 |" in rendered
+ assert "beyond ~130 sources" in rendered
+ # The light tier row is gear-independent
+ assert "| `light` | 10 | 15–25 |" in rendered
+ if skill_name == "hyperresearch":
+ assert "currently `premier`" in rendered
+ assert "~3–5 hours" in rendered
+
+
+def test_install_writes_rendered_prompts_with_header(tmp_vault):
+ """install_hooks renders templates and stamps the provenance header."""
+ from hyperresearch.core.hooks import install_hooks
+
+ install_hooks(tmp_vault.root, hpr_path="hyperresearch")
+
+ critic = (
+ tmp_vault.root / ".claude" / "agents" / "hyperresearch-dialectic-critic.md"
+ ).read_text(encoding="utf-8")
+ assert "At most 12 findings" in critic
+ assert 'rendered from profile "full"' in critic
+ assert "<<" not in critic
+
+ sweep = (
+ tmp_vault.root / ".claude" / "skills" / "hyperresearch-2-width-sweep" / "SKILL.md"
+ ).read_text(encoding="utf-8")
+ assert "| `full` | 45 | 55–80 | 10–12 | 2–3 |" in sweep
+ assert 'rendered from profile "full"' in sweep
+ # Header must come after the frontmatter, not before it
+ assert sweep.startswith("---")
+
+
+def test_install_with_profile_overlay(tmp_vault):
+ """A vault-config profile overlay flows into installed prompts."""
+ from hyperresearch.core.hooks import install_hooks
+
+ cfg_path = tmp_vault.config_path
+ cfg_path.write_text(
+ cfg_path.read_text(encoding="utf-8") + "\n[profile.full]\nsource_min = 200\n",
+ encoding="utf-8",
+ )
+ install_hooks(tmp_vault.root, hpr_path="hyperresearch")
+ sweep = (
+ tmp_vault.root / ".claude" / "skills" / "hyperresearch-2-width-sweep" / "SKILL.md"
+ ).read_text(encoding="utf-8")
+ assert "| `full` | 200 |" in sweep
+
+
+# ---------------------------------------------------------------------------
+# ModelMap wiring — the profile's per-agent model assignments must actually
+# reach the installed agent frontmatter (they were decorative before 2.0).
+# ---------------------------------------------------------------------------
+
+# Installed agent file -> the ModelMap field that governs its `model:` line.
+AGENT_FILE_MODEL_FIELD = {
+ "hyperresearch-fetcher.md": "fetcher",
+ "hyperresearch-loci-analyst.md": "loci_analyst",
+ "hyperresearch-source-analyst.md": "source_analyst",
+ "hyperresearch-depth-investigator.md": "depth_investigator",
+ "hyperresearch-dialectic-critic.md": "critics",
+ "hyperresearch-depth-critic.md": "critics",
+ "hyperresearch-width-critic.md": "critics",
+ "hyperresearch-instruction-critic.md": "critics",
+ "hyperresearch-patcher.md": "patcher",
+ "hyperresearch-synthesizer.md": "synthesizer",
+ "hyperresearch-polish-auditor.md": "polish_auditor",
+ "hyperresearch-readability-recommender.md": "readability_recommender",
+ "hyperresearch-cite-checker.md": "cite_checker",
+ "hyperresearch-browser-fetcher.md": "browser_fetcher",
+ "hyperresearch-corpus-critic.md": "corpus_critic",
+ "hyperresearch-draft-orchestrator.md": "draft_orchestrator",
+}
+
+
+def _frontmatter_model(path: Path) -> str:
+ for line in path.read_text(encoding="utf-8").splitlines():
+ if line.startswith("model: "):
+ return line.removeprefix("model: ").strip()
+ raise AssertionError(f"{path.name} has no model: frontmatter line")
+
+
+def test_installed_agent_models_come_from_profile(tmp_vault):
+ """Every installed agent's model: line equals its ModelMap assignment."""
+ from hyperresearch.core.hooks import install_hooks
+ from hyperresearch.core.profiles import resolve_profile
+
+ install_hooks(tmp_vault.root, hpr_path="hyperresearch")
+ models = resolve_profile("full").models
+ agents_dir = tmp_vault.root / ".claude" / "agents"
+ for filename, field in AGENT_FILE_MODEL_FIELD.items():
+ path = agents_dir / filename
+ assert path.exists(), f"agent not installed: {filename}"
+ assert _frontmatter_model(path) == getattr(models, field), filename
+
+
+def test_haiku_fetcher_overlay_reaches_installed_agent(tmp_vault):
+ """models = { fetcher = "haiku" } must swap ONLY the fetcher's model."""
+ from hyperresearch.core.hooks import install_hooks
+
+ cfg_path = tmp_vault.config_path
+ cfg_path.write_text(
+ cfg_path.read_text(encoding="utf-8")
+ + '\n[profile.full]\nmodels = { fetcher = "haiku" }\n',
+ encoding="utf-8",
+ )
+ install_hooks(tmp_vault.root, hpr_path="hyperresearch")
+ agents_dir = tmp_vault.root / ".claude" / "agents"
+ assert _frontmatter_model(agents_dir / "hyperresearch-fetcher.md") == "haiku"
+ assert _frontmatter_model(agents_dir / "hyperresearch-source-analyst.md") == "sonnet"
+ assert _frontmatter_model(agents_dir / "hyperresearch-patcher.md") == "opus"
+
+
+def test_no_hardcoded_model_lines_in_agent_templates():
+ """Every `model:` frontmatter line in the hooks.py agent constants must be
+ a template ref (`model: << p.models.X >>`), never a literal model name —
+ a literal would silently ignore the profile's ModelMap."""
+ source = Path(hooks.__file__).read_text(encoding="utf-8")
+ bad = [
+ line.strip()
+ for line in source.splitlines()
+ if line.startswith("model: ") and not line.startswith("model: <<")
+ ]
+ assert not bad, f"hardcoded model lines in hooks.py: {bad}"
+
+
+# ---------------------------------------------------------------------------
+# No-contradiction sweep — rendered prompts must not carry dollar-cost ranges
+# (not a bill on subscription billing) or hardcoded model-name claims that a
+# ModelMap override would falsify.
+# ---------------------------------------------------------------------------
+
+_DOLLAR_RANGE = re.compile(r"\$\d+\s*[-–—]\s*\d+")
+_MODEL_CLAIM = re.compile(r"[Rr]uns on (Sonnet|Opus|Haiku)")
+
+
+def _all_skill_names() -> list[str]:
+ import hyperresearch
+
+ skills_dir = Path(hyperresearch.__file__).parent / "skills"
+ return sorted(p.name for p in skills_dir.glob("*.md"))
+
+
+ALL_AGENT_CONSTANTS = [
+ "LOCI_ANALYST_AGENT", "DEPTH_INVESTIGATOR_AGENT", "DIALECTIC_CRITIC_AGENT",
+ "DEPTH_CRITIC_AGENT", "WIDTH_CRITIC_AGENT", "INSTRUCTION_CRITIC_AGENT",
+ "PATCHER_AGENT", "POLISH_AUDITOR_AGENT", "DRAFT_ORCHESTRATOR_AGENT",
+ "SYNTHESIZER_AGENT", "READABILITY_REFORMATTER_AGENT", "SOURCE_ANALYST_AGENT",
+ "RESEARCHER_AGENT", "CORPUS_CRITIC_AGENT", "BROWSER_FETCHER_AGENT",
+ "CITE_CHECKER_AGENT",
+]
+
+
[email protected]("skill_file", _all_skill_names())
+def test_rendered_skills_have_no_cost_or_model_claims(skill_file, ctx):
+ rendered = render_prompt(_read_skill_source(skill_file), ctx)
+ assert not _DOLLAR_RANGE.search(rendered), f"dollar-cost range in {skill_file}"
+ assert not _MODEL_CLAIM.search(rendered), f"hardcoded model claim in {skill_file}"
+
+
[email protected]("const_name", ALL_AGENT_CONSTANTS)
+def test_rendered_agents_have_no_cost_or_model_claims(const_name, ctx):
+ rendered = render_prompt(getattr(hooks, const_name), ctx)
+ assert not _DOLLAR_RANGE.search(rendered), f"dollar-cost range in {const_name}"
+ assert not _MODEL_CLAIM.search(rendered), f"hardcoded model claim in {const_name}"
diff --git a/tests/test_core/test_render.py b/tests/test_core/test_render.py
new file mode 100644
index 0000000..7e46916
--- /dev/null
+++ b/tests/test_core/test_render.py
@@ -0,0 +1,77 @@
+"""Tests for the prompt-template render layer (core/render.py)."""
+
+from __future__ import annotations
+
+import pytest
+from jinja2 import UndefinedError
+
+from hyperresearch.core.render import (
+ build_render_context,
+ insert_after_frontmatter,
+ render_header,
+ render_prompt,
+)
+
+
[email protected](scope="module")
+def ctx():
+ return build_render_context(None, primary="full")
+
+
+class TestDelimiters:
+ def test_variables_use_double_angle(self, ctx):
+ assert render_prompt("min << p.source_min >> sources", ctx) == "min 45 sources"
+
+ def test_standard_jinja_braces_pass_through(self, ctx):
+ # Skill prose legitimately contains {{...}} placeholders and JSON braces;
+ # they must survive rendering untouched.
+ text = "> {{paste research/query-<vault_tag>.md body}}\n{\"total_findings\": 3}"
+ assert render_prompt(text, ctx) == text
+
+ def test_blocks_use_angle_percent(self, ctx):
+ out = render_prompt("<% if p.utility_scoring %>score<% endif %>", ctx)
+ assert out == "score"
+
+ def test_unknown_variable_raises(self, ctx):
+ with pytest.raises(UndefinedError):
+ render_prompt("<< p.does_not_exist >>", ctx)
+
+
+class TestFilters:
+ def test_dash_is_en_dash(self, ctx):
+ assert render_prompt("<< p.source_target|dash >>", ctx) == "55–80"
+
+ def test_hyphen(self, ctx):
+ assert render_prompt("<< p.fetcher_chase|hyphen >>", ctx) == "3-8"
+
+
+class TestContext:
+ def test_all_profiles_exposed_by_name(self, ctx):
+ out = render_prompt("<< light.source_min >>/<< full.source_min >>", ctx)
+ assert out == "10/45"
+
+ def test_primary_alias(self):
+ ctx_light = build_render_context(None, primary="light")
+ assert render_prompt("<< p.source_min >>", ctx_light) == "10"
+
+ def test_user_overlay_flows_through(self, tmp_path):
+ cfg = tmp_path / "config.toml"
+ cfg.write_text("[profile.full]\nsource_min = 200\n", encoding="utf-8")
+ ctx2 = build_render_context(cfg, primary="full")
+ assert render_prompt("<< p.source_min >>", ctx2) == "200"
+
+
+class TestHeaderInsertion:
+ def test_inserted_after_frontmatter(self):
+ content = "---\nname: x\n---\n\n# Body\n"
+ out = insert_after_frontmatter(content, "<!-- hdr -->")
+ assert out.startswith("---\nname: x\n---\n<!-- hdr -->\n\n# Body\n")
+
+ def test_no_frontmatter_prepends(self):
+ out = insert_after_frontmatter("# Body\n", "<!-- hdr -->")
+ assert out == "<!-- hdr -->\n# Body\n"
+
+ def test_header_names_profile_and_version(self):
+ h = render_header("dissertation", "9.9.9")
+ assert "dissertation" in h and "9.9.9" in h
+ assert h.startswith("<!--") and h.endswith("-->")
diff --git a/tests/test_core/test_runs.py b/tests/test_core/test_runs.py
new file mode 100644
index 0000000..e013e29
--- /dev/null
+++ b/tests/test_core/test_runs.py
@@ -0,0 +1,243 @@
+"""Phase-3 tests: per-run workspaces, run manifest, resume, budget governor."""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from hyperresearch.core import runs as runs_mod
+from hyperresearch.core.runs import (
+ RunError,
+ add_spend,
+ init_run,
+ list_runs,
+ load_manifest,
+ resume_position,
+ set_step,
+ status_summary,
+)
+
+
+class TestRunInit:
+ def test_scaffolds_workspace(self, tmp_vault):
+ manifest = init_run(tmp_vault, "topic-abc123", profile="full", query="What is X?")
+ run_dir = tmp_vault.run_dir("topic-abc123")
+ assert (run_dir / "run.json").exists()
+ assert (run_dir / "temp").is_dir()
+ assert (run_dir / "query.md").read_text(encoding="utf-8") == "What is X?"
+ assert manifest["status"] == "running"
+ assert manifest["profile_steps"][0] == "1"
+ assert manifest["profile_steps"][-1] == "16"
+
+ def test_idempotent_reinit(self, tmp_vault):
+ first = init_run(tmp_vault, "topic-abc123", profile="light")
+ set_step(tmp_vault, "topic-abc123", "1", "done")
+ again = init_run(tmp_vault, "topic-abc123", profile="full") # different args ignored
+ assert again["profile"] == "light" # original manifest wins
+ assert again["steps"]["1"]["status"] == "done"
+ assert first["started_at"] == again["started_at"]
+
+ def test_unknown_profile_rejected(self, tmp_vault):
+ from hyperresearch.core.profiles import ProfileError
+
+ with pytest.raises(ProfileError):
+ init_run(tmp_vault, "topic-x", profile="bogus")
+
+ def test_two_runs_are_disjoint(self, tmp_vault):
+ init_run(tmp_vault, "alpha-111111")
+ init_run(tmp_vault, "beta-222222")
+ set_step(tmp_vault, "alpha-111111", "2", "done")
+ beta = load_manifest(tmp_vault, "beta-222222")
+ assert "2" not in beta["steps"]
+ assert len(list_runs(tmp_vault)) == 2
+
+
+class TestStepsAndResume:
+ def test_step_transitions_and_resume(self, tmp_vault):
+ init_run(tmp_vault, "t-000001", profile="light")
+ set_step(tmp_vault, "t-000001", "1", "done")
+ set_step(tmp_vault, "t-000001", "2", "done")
+ manifest = load_manifest(tmp_vault, "t-000001")
+ pos = resume_position(manifest)
+ # light steps: 1,2,10,15,16
+ assert pos["next_step"] == "10"
+ assert pos["done_steps"] == ["1", "2"]
+ assert pos["remaining_steps"] == ["10", "15", "16"]
+
+ def test_fractional_and_chapter_steps(self, tmp_vault):
+ init_run(tmp_vault, "t-000002", profile="dissertation")
+ set_step(tmp_vault, "t-000002", "1", "done")
+ set_step(tmp_vault, "t-000002", "1.5", "done")
+ set_step(tmp_vault, "t-000002", "2", "done", chapter="ch1")
+ manifest = load_manifest(tmp_vault, "t-000002")
+ assert manifest["steps"]["1.5"]["status"] == "done"
+ assert manifest["steps"]["2"]["chapter"] == "ch1"
+ assert "ch1" in manifest["chapters"]
+
+ def test_all_done_gives_none(self, tmp_vault):
+ init_run(tmp_vault, "t-000003", profile="light")
+ for s in ["1", "2", "10", "15", "16"]:
+ set_step(tmp_vault, "t-000003", s, "done")
+ assert resume_position(load_manifest(tmp_vault, "t-000003"))["next_step"] is None
+
+ def test_invalid_status_rejected(self, tmp_vault):
+ init_run(tmp_vault, "t-000004")
+ with pytest.raises(RunError, match="invalid step status"):
+ set_step(tmp_vault, "t-000004", "1", "finished")
+
+ def test_events_logged(self, tmp_vault):
+ init_run(tmp_vault, "t-000005")
+ set_step(tmp_vault, "t-000005", "1", "running")
+ events_file = tmp_vault.run_dir("t-000005") / "events.jsonl"
+ lines = events_file.read_text(encoding="utf-8").strip().splitlines()
+ assert any(json.loads(ln)["type"] == "step" for ln in lines)
+
+
+class TestBudgetGovernor:
+ def test_spend_accumulates(self, tmp_vault):
+ init_run(tmp_vault, "b-000001", budget_usd=100.0)
+ add_spend(tmp_vault, "b-000001", estimated_usd=12.5, sources_fetched=30, agents_spawned=8)
+ m = add_spend(tmp_vault, "b-000001", estimated_usd=7.5, sources_fetched=10)
+ assert m["spend"]["estimated_usd"] == 20.0
+ assert m["spend"]["sources_fetched"] == 40
+ assert m["status"] == "running"
+
+ def test_crossing_budget_blocks(self, tmp_vault):
+ init_run(tmp_vault, "b-000002", budget_usd=10.0)
+ m = add_spend(tmp_vault, "b-000002", estimated_usd=10.0)
+ assert m["status"] == "blocked"
+ assert m["blocked_on"] == "budget"
+ summary = status_summary(tmp_vault, "b-000002")
+ assert summary["budget_remaining_usd"] == 0.0
+
+ def test_no_budget_never_blocks(self, tmp_vault):
+ init_run(tmp_vault, "b-000003")
+ m = add_spend(tmp_vault, "b-000003", estimated_usd=99999)
+ assert m["status"] == "running"
+
+ def test_resume_unblocks(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ init_run(tmp_vault, "b-000004", budget_usd=5.0)
+ add_spend(tmp_vault, "b-000004", estimated_usd=6.0)
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["run", "resume", "b-000004", "--json"])
+ assert result.exit_code == 0
+ assert load_manifest(tmp_vault, "b-000004")["status"] == "running"
+
+
+class TestStatusSummary:
+ def test_stall_detection(self, tmp_vault, monkeypatch):
+ init_run(tmp_vault, "s-000001")
+ summary = status_summary(tmp_vault, "s-000001")
+ assert summary["possibly_stalled"] is False
+ # Backdate the heartbeat
+ m = load_manifest(tmp_vault, "s-000001")
+ m["updated_at"] = "2020-01-01T00:00:00+00:00"
+ mpath = runs_mod.manifest_path(tmp_vault, "s-000001")
+ mpath.write_text(json.dumps(m), encoding="utf-8")
+ assert status_summary(tmp_vault, "s-000001")["possibly_stalled"] is True
+
+ def test_missing_run_errors(self, tmp_vault):
+ with pytest.raises(RunError, match="no run"):
+ status_summary(tmp_vault, "nope-000000")
+
+
+class TestRunCli:
+ def test_init_status_resume_roundtrip(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+
+ r = runner.invoke(app, ["run", "init", "cli-run-000001", "--profile", "light", "--budget", "50", "--json"])
+ assert r.exit_code == 0
+
+ r = runner.invoke(app, ["run", "step", "cli-run-000001", "1", "--status", "done", "--json"])
+ assert r.exit_code == 0
+
+ r = runner.invoke(app, ["run", "resume", "--json"])
+ assert r.exit_code == 0
+ data = json.loads(r.stdout)["data"]
+ assert data["vault_tag"] == "cli-run-000001"
+ assert data["next_step"] == "2"
+ assert data["skill_to_invoke"] == "hyperresearch-2"
+
+ r = runner.invoke(app, ["run", "abort", "cli-run-000001", "--json"])
+ assert r.exit_code == 0
+ assert json.loads(r.stdout)["data"]["status"] == "aborted"
+
+ def test_resume_maps_fractional_step_to_skill(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ runner.invoke(app, ["run", "init", "dis-run-000001", "--profile", "dissertation", "--json"])
+ # Manifest profile_steps for dissertation are 1..16; mark 1 done and
+ # verify the mapping convention for step ids with dots.
+ runner.invoke(app, ["run", "step", "dis-run-000001", "1", "--status", "done", "--json"])
+ r = runner.invoke(app, ["run", "status", "dis-run-000001", "--json"])
+ payload = json.loads(r.stdout)["data"]
+ assert payload["profile"] == "dissertation"
+
+
+class TestWorkspaceIsolation:
+ def test_run_files_not_synced_as_notes(self, tmp_vault):
+ from hyperresearch.core.sync import compute_sync_plan
+
+ init_run(tmp_vault, "iso-000001")
+ run_dir = tmp_vault.run_dir("iso-000001")
+ # A markdown file WITH frontmatter inside the run workspace must not
+ # become a vault note.
+ (run_dir / "scaffold.md").write_text(
+ "---\ntitle: Scaffold\n---\n\n## User Prompt (VERBATIM — gospel)\nx\n",
+ encoding="utf-8",
+ )
+ (run_dir / "temp" / "draft-a.md").write_text(
+ "---\ntitle: Draft A\n---\n\nbody\n", encoding="utf-8"
+ )
+ plan = compute_sync_plan(tmp_vault, force=True)
+ paths = [str(p) for p in plan.to_add]
+ assert not any("runs" in p for p in paths)
+
+ def test_vault_tag_collision_includes_run_dirs(self, tmp_vault):
+ from hyperresearch.cli.vault_tag import _existing_tags
+
+ init_run(tmp_vault, "topic-aaaaaa")
+ tags = _existing_tags(tmp_vault.root, tmp_vault.research_dir)
+ assert "topic-aaaaaa" in tags
+
+ def test_lint_resolves_run_scoped_loci(self, tmp_vault):
+ from hyperresearch.cli.lint import _run_artifact
+
+ init_run(tmp_vault, "lint-000001")
+ loci = tmp_vault.run_dir("lint-000001") / "loci.json"
+ loci.write_text('{"loci": []}', encoding="utf-8")
+ resolved = _run_artifact(tmp_vault, "loci.json")
+ assert resolved == loci
+
+ def test_lint_falls_back_to_legacy_flat_path(self, tmp_vault):
+ from hyperresearch.cli.lint import _run_artifact
+
+ flat = tmp_vault.root / "research" / "loci.json"
+ flat.write_text('{"loci": []}', encoding="utf-8")
+ assert _run_artifact(tmp_vault, "loci.json") == flat
+
+ def test_lint_query_files_both_layouts(self, tmp_vault):
+ from hyperresearch.cli.lint import _query_files
+
+ init_run(tmp_vault, "q-000001", query="the question")
+ legacy = tmp_vault.root / "research" / "query-old-tag.md"
+ legacy.write_text("old question", encoding="utf-8")
+ files = _query_files(tmp_vault)
+ names = [f.name for f in files]
+ assert names[0] == "query.md" # newest run first
+ assert "query-old-tag.md" in names
diff --git a/tests/test_core/test_scholar_enrichment.py b/tests/test_core/test_scholar_enrichment.py
new file mode 100644
index 0000000..b9f683a
--- /dev/null
+++ b/tests/test_core/test_scholar_enrichment.py
@@ -0,0 +1,208 @@
+"""Tests for `hpr sources score` enrichment — fully offline, HTTP stubbed."""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from hyperresearch.core import scholar
+
+
+def doi_vault(tmp_vault):
+ """Vault with two DOI-bearing notes and one plain note."""
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ write_note(
+ tmp_vault.notes_dir, "Cited Paper", body="Highly cited work.",
+ tier="institutional",
+ extra_frontmatter={"doi": "10.1/cited", "source": "https://doi.org/10.1/cited"},
+ )
+ write_note(
+ tmp_vault.notes_dir, "Retracted Paper", body="Withdrawn work.",
+ tier="institutional",
+ extra_frontmatter={"doi": "10.1/retracted", "source": "https://doi.org/10.1/retracted"},
+ )
+ write_note(tmp_vault.notes_dir, "Plain Blog", body="No DOI here.")
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+ return tmp_vault
+
+
+def _stub_openalex(monkeypatch, responses: dict[str, dict | None]):
+ """Stub the raw HTTP layer with canned per-URL-substring responses."""
+ calls: list[str] = []
+
+ def fake_get(url: str):
+ calls.append(url)
+ for key, value in responses.items():
+ if key in url:
+ return value
+ return None
+
+ monkeypatch.setattr(scholar, "_http_get_json", fake_get)
+ return calls
+
+
+OPENALEX_CITED = {
+ "cited_by_count": 512,
+ "primary_location": {"source": {"display_name": "Nature"}},
+ "is_retracted": False,
+}
+OPENALEX_RETRACTED = {
+ "cited_by_count": 30,
+ "primary_location": {"source": {"display_name": "BadJournal"}},
+ "is_retracted": True,
+}
+
+
+class TestScoreSources:
+ def test_enrichment_populates_db_and_frontmatter(self, doi_vault, monkeypatch):
+ _stub_openalex(monkeypatch, {
+ "10.1%2Fcited": OPENALEX_CITED,
+ "10.1%2Fretracted": OPENALEX_RETRACTED,
+ })
+ result = scholar.score_sources(doi_vault)
+ assert result["scored"] == 2
+ assert result["retracted"] == ["retracted-paper"]
+
+ row = doi_vault.db.execute(
+ "SELECT citation_count, venue, is_retracted, authority_score, quality_score "
+ "FROM notes WHERE id = 'cited-paper'"
+ ).fetchone()
+ assert row["citation_count"] == 512
+ assert row["venue"] == "Nature"
+ assert row["is_retracted"] == 0
+ assert row["authority_score"] is not None
+ assert row["quality_score"] is not None
+
+ # Frontmatter mirror written
+ text = next(doi_vault.notes_dir.glob("cited-paper.md")).read_text(encoding="utf-8")
+ assert "citation_count: 512" in text
+ assert "venue: Nature" in text
+
+ def test_retracted_gets_quality_floor(self, doi_vault, monkeypatch):
+ _stub_openalex(monkeypatch, {
+ "10.1%2Fcited": OPENALEX_CITED,
+ "10.1%2Fretracted": OPENALEX_RETRACTED,
+ })
+ scholar.score_sources(doi_vault)
+ floor = doi_vault.config.ranking.retraction_floor
+ row = doi_vault.db.execute(
+ "SELECT quality_score FROM notes WHERE id = 'retracted-paper'"
+ ).fetchone()
+ assert row["quality_score"] == floor
+
+ def test_cache_prevents_second_http_call(self, doi_vault, monkeypatch):
+ calls = _stub_openalex(monkeypatch, {
+ "10.1%2Fcited": OPENALEX_CITED,
+ "10.1%2Fretracted": OPENALEX_RETRACTED,
+ })
+ scholar.score_sources(doi_vault)
+ first_count = len(calls)
+ assert first_count == 2
+ # Re-score fresh=True → cache bypassed but api_cache row present;
+ # fresh=False re-run skips already-enriched notes entirely.
+ scholar.score_sources(doi_vault)
+ assert len(calls) == first_count # no new HTTP calls
+
+ def test_api_failure_is_soft(self, doi_vault, monkeypatch):
+ _stub_openalex(monkeypatch, {}) # everything misses -> None
+ result = scholar.score_sources(doi_vault)
+ assert result["scored"] == 0
+ assert len(result["missing"]) == 2 # both DOI notes unresolved, no crash
+
+ def test_arxiv_ids_route_to_semantic_scholar(self, tmp_vault, monkeypatch):
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ write_note(
+ tmp_vault.notes_dir, "Arxiv Preprint", body="x",
+ extra_frontmatter={"doi": "arXiv:2501.01234"},
+ )
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+
+ calls = _stub_openalex(monkeypatch, {
+ "semanticscholar": {"citationCount": 7, "venue": "arXiv"},
+ })
+ result = scholar.score_sources(tmp_vault)
+ assert result["scored"] == 1
+ assert any("semanticscholar.org" in c for c in calls)
+ assert not any("openalex.org" in c for c in calls)
+
+ def test_authority_is_vault_relative_percentile(self, doi_vault, monkeypatch):
+ _stub_openalex(monkeypatch, {
+ "10.1%2Fcited": OPENALEX_CITED, # 512 citations
+ "10.1%2Fretracted": OPENALEX_RETRACTED, # 30 citations
+ })
+ scholar.score_sources(doi_vault)
+ rows = {
+ r["id"]: r["authority_score"]
+ for r in doi_vault.db.execute(
+ "SELECT id, authority_score FROM notes WHERE authority_score IS NOT NULL"
+ )
+ }
+ assert rows["cited-paper"] == 1.0
+ assert rows["retracted-paper"] == 0.5
+
+
+class TestBackfillDois:
+ def test_backfill_from_source_url(self, tmp_vault):
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ write_note(
+ tmp_vault.notes_dir, "Old Fetch", body="fetched before doi capture existed",
+ source="https://arxiv.org/abs/2401.00001",
+ )
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+
+ gained = scholar.backfill_dois(tmp_vault)
+ assert gained == 1
+ row = tmp_vault.db.execute("SELECT doi FROM notes WHERE id = 'old-fetch'").fetchone()
+ assert row["doi"] == "arXiv:2401.00001"
+ text = next(tmp_vault.notes_dir.glob("old-fetch.md")).read_text(encoding="utf-8")
+ assert "arXiv:2401.00001" in text
+
+
+class TestApiCache:
+ def test_cache_roundtrip(self, tmp_vault, monkeypatch):
+ calls = []
+
+ def fake_get(url):
+ calls.append(url)
+ return {"hello": "world"}
+
+ monkeypatch.setattr(scholar, "_http_get_json", fake_get)
+ conn = tmp_vault.db
+ r1 = scholar._fetch_json(conn, "https://api.openalex.org/works/x", ttl_days=30)
+ r2 = scholar._fetch_json(conn, "https://api.openalex.org/works/x", ttl_days=30)
+ assert r1 == r2 == {"hello": "world"}
+ assert len(calls) == 1 # second hit served from cache
+
+ def test_fresh_bypasses_cache(self, tmp_vault, monkeypatch):
+ calls = []
+
+ def fake_get(url):
+ calls.append(url)
+ return {"n": len(calls)}
+
+ monkeypatch.setattr(scholar, "_http_get_json", fake_get)
+ conn = tmp_vault.db
+ scholar._fetch_json(conn, "https://api.openalex.org/works/y", ttl_days=30)
+ r2 = scholar._fetch_json(conn, "https://api.openalex.org/works/y", ttl_days=30, fresh=True)
+ assert len(calls) == 2
+ assert r2 == {"n": 2}
+
+ def test_cache_body_is_json(self, tmp_vault, monkeypatch):
+ monkeypatch.setattr(scholar, "_http_get_json", lambda url: {"a": 1})
+ conn = tmp_vault.db
+ scholar._fetch_json(conn, "https://api.openalex.org/works/z", ttl_days=30)
+ row = conn.execute(
+ "SELECT body FROM api_cache WHERE url = 'https://api.openalex.org/works/z'"
+ ).fetchone()
+ assert json.loads(row["body"]) == {"a": 1}
diff --git a/tests/test_core/test_source_ranking.py b/tests/test_core/test_source_ranking.py
new file mode 100644
index 0000000..53896e6
--- /dev/null
+++ b/tests/test_core/test_source_ranking.py
@@ -0,0 +1,201 @@
+"""Phase-2 source-ranking tests: schema v9, DOI capture, PageRank, quality."""
+
+from __future__ import annotations
+
+from hyperresearch.core.config import RankingSettings
+from hyperresearch.core.graphrank import compute_centrality, pagerank
+from hyperresearch.core.quality import compute_quality_for_row, compute_quality_scores
+from hyperresearch.core.scholar import extract_doi
+
+
+class TestSchemaV9:
+ def test_new_columns_exist(self, tmp_vault):
+ cols = {row[1] for row in tmp_vault.db.execute("PRAGMA table_info(notes)")}
+ expected = {
+ "doi", "utility_score", "authority_score", "centrality_score",
+ "independence", "citation_count", "venue", "is_retracted", "quality_score",
+ }
+ assert expected <= cols
+
+ def test_new_tables_exist(self, tmp_vault):
+ tables = {
+ row[0]
+ for row in tmp_vault.db.execute(
+ "SELECT name FROM sqlite_master WHERE type IN ('table','view')"
+ )
+ }
+ assert "claims" in tables
+ assert "api_cache" in tables
+ # claims_fts is a virtual table
+ assert tmp_vault.db.execute("SELECT count(*) FROM claims_fts").fetchone() is not None
+
+ def test_migration_from_v8_is_idempotent(self, tmp_vault):
+ from hyperresearch.core.migrations import _migrate_v9_source_ranking
+
+ _migrate_v9_source_ranking(tmp_vault.db) # re-run on already-migrated DB
+ cols = {row[1] for row in tmp_vault.db.execute("PRAGMA table_info(notes)")}
+ assert "quality_score" in cols
+
+ def test_frontmatter_roundtrip(self, tmp_vault):
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ write_note(
+ tmp_vault.notes_dir,
+ "Ranked Paper",
+ body="A paper with ranking metadata.",
+ extra_frontmatter={
+ "doi": "10.1234/example",
+ "utility_score": 14.0,
+ "citation_count": 42,
+ "venue": "NeurIPS",
+ "is_retracted": False,
+ },
+ )
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+ row = tmp_vault.db.execute(
+ "SELECT doi, utility_score, citation_count, venue, is_retracted "
+ "FROM notes WHERE id = 'ranked-paper'"
+ ).fetchone()
+ assert row["doi"] == "10.1234/example"
+ assert row["utility_score"] == 14.0
+ assert row["citation_count"] == 42
+ assert row["venue"] == "NeurIPS"
+ assert row["is_retracted"] == 0
+
+ def test_derived_scores_survive_resync(self, seeded_vault):
+ conn = seeded_vault.db
+ conn.execute(
+ "UPDATE notes SET quality_score = 0.9, centrality_score = 0.5 "
+ "WHERE id = 'concurrency'"
+ )
+ conn.commit()
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ plan = compute_sync_plan(seeded_vault, force=True)
+ execute_sync(seeded_vault, plan)
+ row = conn.execute(
+ "SELECT quality_score, centrality_score FROM notes WHERE id = 'concurrency'"
+ ).fetchone()
+ assert row["quality_score"] == 0.9
+ assert row["centrality_score"] == 0.5
+
+
+class TestDoiExtraction:
+ def test_doi_org_url(self):
+ assert extract_doi("https://doi.org/10.1038/s41586-024-0001-2") == "10.1038/s41586-024-0001-2"
+
+ def test_arxiv_abs_and_pdf(self):
+ assert extract_doi("https://arxiv.org/abs/2501.01234") == "arXiv:2501.01234"
+ assert extract_doi("https://arxiv.org/pdf/2501.01234v2") == "arXiv:2501.01234v2"
+
+ def test_citation_meta_tag(self):
+ html = '<head><meta name="citation_doi" content="10.1145/3576915.3616613"></head>'
+ assert extract_doi("https://example.com/paper", raw_html=html) == "10.1145/3576915.3616613"
+
+ def test_body_doi_marker(self):
+ body = "Published in Nature. DOI: 10.1038/nphys1170 (2024)."
+ assert extract_doi("https://example.com/x", content=body) == "10.1038/nphys1170"
+
+ def test_no_doi(self):
+ assert extract_doi("https://example.com/blog-post", content="just a blog") is None
+
+ def test_trailing_punctuation_stripped(self):
+ assert extract_doi("https://x.com", content="see DOI: 10.1000/xyz123.") == "10.1000/xyz123"
+
+
+class TestPageRank:
+ def test_hub_ranks_highest(self):
+ nodes = ["hub", "a", "b", "c"]
+ edges = [("a", "hub"), ("b", "hub"), ("c", "hub"), ("hub", "a")]
+ scores = pagerank(nodes, edges)
+ assert scores["hub"] == max(scores.values())
+
+ def test_empty_graph(self):
+ assert pagerank([], []) == {}
+
+ def test_dangling_nodes_handled(self):
+ scores = pagerank(["a", "b"], [("a", "b")])
+ assert abs(sum(scores.values()) - 1.0) < 0.01
+
+ def test_compute_centrality_on_seeded_vault(self, seeded_vault):
+ n = compute_centrality(seeded_vault.db)
+ assert n > 0
+ rows = seeded_vault.db.execute(
+ "SELECT id, centrality_score FROM notes ORDER BY centrality_score DESC"
+ ).fetchall()
+ assert rows[0]["centrality_score"] == 1.0 # normalized by max
+ # The MOC/hub notes are linked from multiple notes → should outrank the orphan
+ by_id = {r["id"]: r["centrality_score"] for r in rows}
+ assert by_id["concurrency"] > by_id["orphan-note"]
+
+
+class TestQualityComposite:
+ R = RankingSettings()
+
+ def test_retraction_floors_everything(self):
+ q = compute_quality_for_row(self.R, "ground_truth", 18.0, 1.0, 1.0, True)
+ assert q == self.R.retraction_floor
+
+ def test_tier_only_renormalizes(self):
+ q = compute_quality_for_row(self.R, "ground_truth", None, None, None, False)
+ assert q == 1.0 # single component, full weight on tier weight 1.0
+
+ def test_no_components_gives_none(self):
+ assert compute_quality_for_row(self.R, None, None, None, None, False) is None
+
+ def test_full_composite_weighting(self):
+ q = compute_quality_for_row(self.R, "commentary", 9.0, 0.5, 0.5, False)
+ # (0.35*0.4 + 0.2*0.5 + 0.25*0.5 + 0.2*0.5) / 1.0
+ assert abs(q - (0.35 * 0.4 + 0.2 * 0.5 + 0.25 * 0.5 + 0.2 * 0.5)) < 1e-9
+
+ def test_ground_truth_beats_commentary(self):
+ gt = compute_quality_for_row(self.R, "ground_truth", None, None, None, False)
+ com = compute_quality_for_row(self.R, "commentary", None, None, None, False)
+ assert gt > com
+
+ def test_compute_all(self, seeded_vault):
+ conn = seeded_vault.db
+ conn.execute("UPDATE notes SET tier = 'ground_truth' WHERE id = 'concurrency'")
+ conn.execute("UPDATE notes SET tier = 'commentary' WHERE id = 'orphan-note'")
+ conn.commit()
+ updated = compute_quality_scores(conn, self.R)
+ assert updated >= 4
+ rows = {
+ r["id"]: r["quality_score"]
+ for r in conn.execute("SELECT id, quality_score FROM notes")
+ }
+ assert rows["concurrency"] > rows["orphan-note"]
+
+
+class TestRankedSearch:
+ def test_quality_reorders_equal_relevance(self, seeded_vault):
+ from hyperresearch.search.fts import search_fts
+
+ conn = seeded_vault.db
+ # Two notes both matching "concurrency"; give one a high and one a low quality
+ conn.execute("UPDATE notes SET quality_score = 0.95 WHERE id = 'rust-ownership'")
+ conn.execute("UPDATE notes SET quality_score = 0.05 WHERE id = 'python-async-patterns'")
+ conn.commit()
+
+ plain = search_fts(conn, "concurrency", ranking={})
+ ranked = search_fts(conn, "concurrency", ranking={}, quality_ranked=True)
+ ids_ranked = [r["id"] for r in ranked]
+ assert ids_ranked.index("rust-ownership") < ids_ranked.index("python-async-patterns")
+ # Default search unchanged: quality did not affect the plain ordering
+ assert [r["id"] for r in plain] != ids_ranked or plain != ranked
+
+ def test_search_cli_ranked_flag(self, seeded_vault, monkeypatch):
+ import json
+
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(seeded_vault.root)
+ runner = CliRunner()
+ result = runner.invoke(app, ["search", "concurrency", "--ranked", "--json"])
+ assert result.exit_code == 0
+ payload = json.loads(result.stdout)
+ assert payload["ok"] is True
diff --git a/tests/test_core/test_verification.py b/tests/test_core/test_verification.py
new file mode 100644
index 0000000..fba4805
--- /dev/null
+++ b/tests/test_core/test_verification.py
@@ -0,0 +1,290 @@
+"""Phase-5 tests: cite-check, verification lints, independence, telemetry, verify."""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from hyperresearch.core.citecheck import (
+ extract_pairs,
+ parse_sources_section,
+ sample_needs_llm,
+ triage_pairs,
+)
+from hyperresearch.core.independence import canonical_url, compute_independence
+from hyperresearch.core.runs import init_run, run_report_data, set_step, verify_run
+
+
+def cited_vault(seeded_vault):
+ """Seeded vault + claims + a source row so citations resolve."""
+ from hyperresearch.core.claims import ingest_claims_dir
+
+ temp = seeded_vault.root / "research" / "temp"
+ temp.mkdir(parents=True, exist_ok=True)
+ (temp / "claims-python-async-patterns.json").write_text(json.dumps([
+ {"claim": "Async improves throughput by 10x for network-bound workloads",
+ "quoted_support": "async/await syntax enables concurrent I/O with a 10x gain",
+ "numbers": ["10x"], "confidence": "high", "evidence_type": "empirical"},
+ ]), encoding="utf-8")
+ ingest_claims_dir(seeded_vault, vault_tag="cc-run")
+ seeded_vault.db.execute(
+ "INSERT OR IGNORE INTO sources (url, note_id, domain, fetched_at, provider, content_hash) "
+ "VALUES ('https://example.com/async', 'python-async-patterns', 'example.com', '2026-01-01', 'test', 'x')"
+ )
+ seeded_vault.db.commit()
+ return seeded_vault
+
+
+class TestCiteCheckExtraction:
+ def test_wikilink_pairs(self, cited_vault):
+ report = "Async gives a 10x gain [[python-async-patterns]]. Rust is safe [[rust-ownership]]."
+ pairs = extract_pairs(report, cited_vault.db)
+ assert len(pairs) == 2
+ assert pairs[0]["note_id"] == "python-async-patterns"
+ assert pairs[0]["numbers"] == ["10"] or "10x" in pairs[0]["sentence"]
+
+ def test_numbered_pairs_resolve_via_sources_section(self, cited_vault):
+ report = (
+ "Throughput improves 10x [1].\n\n"
+ "## Sources\n[1] Python Async Patterns. https://example.com/async\n"
+ )
+ mapping = parse_sources_section(report, cited_vault.db)
+ assert mapping["1"] == "python-async-patterns"
+ pairs = extract_pairs(report, cited_vault.db)
+ assert len(pairs) == 1
+ assert pairs[0]["note_id"] == "python-async-patterns"
+
+ def test_dangling_citation_detected(self, cited_vault):
+ report = "A bold claim [[no-such-note]]."
+ pairs = extract_pairs(report, cited_vault.db)
+ assert pairs[0]["note_id"] is None
+ triaged = triage_pairs(pairs, cited_vault.db)
+ assert triaged["dangling"] == 1
+
+ def test_triage_auto_passes_number_match(self, cited_vault):
+ pairs = extract_pairs(
+ "Async improves throughput by 10x [[python-async-patterns]].", cited_vault.db
+ )
+ triaged = triage_pairs(pairs, cited_vault.db)
+ assert triaged["supported_mechanical"] == 1
+ assert triaged["needs_llm"] == 0
+
+ def test_triage_flags_unsupported_for_llm(self, cited_vault):
+ pairs = extract_pairs(
+ "Async cures all database contention problems entirely [[python-async-patterns]].",
+ cited_vault.db,
+ )
+ triaged = triage_pairs(pairs, cited_vault.db)
+ assert triaged["needs_llm"] == 1
+
+ def test_sampling_keeps_all_strong(self):
+ pairs = [
+ {"verdict": "needs-llm", "strong": True, "sentence": f"s{i}", "numbers": ["5"]}
+ for i in range(5)
+ ] + [
+ {"verdict": "needs-llm", "strong": False, "sentence": f"w{i}", "numbers": []}
+ for i in range(10)
+ ]
+ sampled = sample_needs_llm(pairs, sample_rate=0.5)
+ strong = [p for p in sampled if p["strong"]]
+ weak = [p for p in sampled if not p["strong"]]
+ assert len(strong) == 5 # 100% of number-bearing
+ assert len(weak) == 5 # every 2nd weak pair
+
+ def test_citecheck_cli(self, cited_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ init_run(cited_vault, "cc-run")
+ report = cited_vault.root / "research" / "notes" / "final_report_cc-run.md"
+ report.write_text(
+ "Async improves throughput by 10x [[python-async-patterns]]. "
+ "Also a dangling one [[ghost-note]].",
+ encoding="utf-8",
+ )
+ monkeypatch.chdir(cited_vault.root)
+ runner = CliRunner()
+ r = runner.invoke(app, ["citecheck", "extract", "cc-run", "--json"])
+ assert r.exit_code == 0
+ data = json.loads(r.stdout)["data"]
+ assert data["summary"]["supported_mechanical"] == 1
+ assert data["dangling"] == 1
+ assert (cited_vault.run_dir("cc-run") / "cite-check-pairs.json").exists()
+
+
+class TestVerificationLints:
+ def _lint(self, vault, rule, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(vault.root)
+ r = CliRunner().invoke(app, ["lint", "--rule", rule, "--json"])
+ return json.loads(r.stdout)
+
+ def test_quote_integrity_catches_fabrication(self, seeded_vault, monkeypatch):
+ report = seeded_vault.root / "research" / "notes" / "final_report_q.md"
+ report.write_text(
+ 'The paper concludes that "quantum entanglement reverses causality in every measurable frame of reference".',
+ encoding="utf-8",
+ )
+ payload = self._lint(seeded_vault, "quote-integrity", monkeypatch)
+ issues = [i for i in payload["data"]["issues"] if i["rule"] == "quote-integrity"]
+ assert len(issues) == 1
+ assert issues[0]["severity"] == "error"
+
+ def test_quote_integrity_passes_real_quote(self, seeded_vault, monkeypatch):
+ report = seeded_vault.root / "research" / "notes" / "final_report_q.md"
+ # This sentence exists verbatim in the seeded python-async-patterns note
+ report.write_text(
+ 'As the note says, "Python\'s async/await syntax enables concurrent I/O" today.',
+ encoding="utf-8",
+ )
+ payload = self._lint(seeded_vault, "quote-integrity", monkeypatch)
+ issues = [i for i in payload["data"]["issues"] if i["rule"] == "quote-integrity"]
+ assert issues == []
+
+ def test_numeric_consistency_flags_untraceable(self, cited_vault, monkeypatch):
+ report = cited_vault.root / "research" / "notes" / "final_report_n.md"
+ report.write_text("Revenue grew 47.3% while costs fell 1,234,567 dollars.", encoding="utf-8")
+ payload = self._lint(cited_vault, "numeric-consistency", monkeypatch)
+ issues = [i for i in payload["data"]["issues"] if i["rule"] == "numeric-consistency"]
+ assert len(issues) == 2
+ assert all(i["severity"] == "warning" for i in issues)
+
+ def test_retracted_citation_blocks_unless_marked(self, seeded_vault, monkeypatch):
+ conn = seeded_vault.db
+ conn.execute("UPDATE notes SET is_retracted = 1 WHERE id = 'rust-ownership'")
+ conn.commit()
+ report = seeded_vault.root / "research" / "notes" / "final_report_r.md"
+ report.write_text("Rust guarantees safety [[rust-ownership]].", encoding="utf-8")
+ payload = self._lint(seeded_vault, "retracted-citations", monkeypatch)
+ issues = [i for i in payload["data"]["issues"] if i["rule"] == "retracted-citations"]
+ assert len(issues) == 1 and issues[0]["severity"] == "error"
+
+ # Acknowledged retraction passes
+ report.write_text(
+ "One early study claimed safety guarantees [[rust-ownership]] (retracted 2025).",
+ encoding="utf-8",
+ )
+ payload = self._lint(seeded_vault, "retracted-citations", monkeypatch)
+ issues = [i for i in payload["data"]["issues"] if i["rule"] == "retracted-citations"]
+ assert issues == []
+
+
+class TestIndependence:
+ def test_canonical_url(self):
+ assert canonical_url("https://www.Example.com/a/?utm_source=x") == canonical_url("http://example.com/a")
+
+ def test_wire_cluster_discounts_members(self, tmp_vault):
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ pr = "NEW YORK, PRNewswire — MegaCorp announces quantum widget breakthrough today."
+ for i, (title, when) in enumerate([("MegaCorp Breakthrough", "2026-01-01"),
+ ("MegaCorp Announces Widget", "2026-01-02"),
+ ("Quantum Widget from MegaCorp", "2026-01-03")]):
+ write_note(
+ tmp_vault.notes_dir, title, body=pr + f" Outlet {i} adds a sentence.",
+ source=f"https://outlet{i}.com/story", tags=["ind-run"],
+ extra_frontmatter={"created": when + "T00:00:00+00:00"},
+ )
+ write_note(
+ tmp_vault.notes_dir, "Independent Analysis",
+ body="A genuinely independent, differently-worded long analysis of quantum widgets and their many limitations in practice.",
+ source="https://analyst.com/deep-dive", tags=["ind-run"],
+ )
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+
+ result = compute_independence(tmp_vault, tag="ind-run")
+ assert len(result["clusters"]) == 1
+ cluster = result["clusters"][0]
+ assert cluster["size"] == 3
+ assert "wire" in cluster["kind"]
+
+ scores = {r["id"]: r["independence"] for r in tmp_vault.db.execute(
+ "SELECT id, independence FROM notes WHERE independence IS NOT NULL"
+ )}
+ assert scores[cluster["root"]] == 1.0
+ assert scores["independent-analysis"] == 1.0
+ for member in cluster["members"]:
+ assert scores[member] == pytest.approx(1 / 3, abs=0.001)
+
+ def test_same_canonical_url_clusters(self, tmp_vault):
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ write_note(tmp_vault.notes_dir, "Copy A", body="Some words here for the body.",
+ source="https://www.site.com/story?utm_source=feed")
+ write_note(tmp_vault.notes_dir, "Copy B", body="Entirely different words in this body text.",
+ source="https://site.com/story/")
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+ result = compute_independence(tmp_vault)
+ assert any(c["kind"] == "url" for c in result["clusters"])
+
+
+class TestTelemetryAndVerify:
+ def test_run_report_rollup(self, tmp_vault):
+ init_run(tmp_vault, "tel-01", profile="light")
+ set_step(tmp_vault, "tel-01", "1", "running")
+ set_step(tmp_vault, "tel-01", "1", "done")
+ report = run_report_data(tmp_vault, "tel-01")
+ step1 = next(s for s in report["steps"] if s["step"] == "1")
+ assert step1["status"] == "done"
+ assert step1["minutes"] is not None
+ assert report["events"]["step"] >= 2
+
+ def test_verify_passes_well_formed_light_run(self, tmp_vault):
+ init_run(tmp_vault, "vf-01", profile="light")
+ run_dir = tmp_vault.run_dir("vf-01")
+ (run_dir / "prompt-decomposition.json").write_text(json.dumps({
+ "response_format": "short",
+ "required_section_headings": ["## Findings"],
+ }), encoding="utf-8")
+ (run_dir / "polish-log.json").write_text('{"applied": []}', encoding="utf-8")
+ report = tmp_vault.root / "research" / "notes" / "final_report_vf-01.md"
+ body = "## Findings\n\n" + ("Substantive sentence with real evidence attached [[src-note]]. " * 80)
+ report.write_text(body, encoding="utf-8")
+
+ result = verify_run(tmp_vault, "vf-01")
+ by_name = {c["name"]: c for c in result["checks"]}
+ assert by_name["report-exists"]["ok"]
+ assert by_name["required-headings"]["ok"]
+ assert by_name["length-in-range"]["ok"]
+ assert by_name["citation-density"]["ok"]
+ assert result["passed"] is True
+
+ def test_verify_fails_on_missing_heading_and_report(self, tmp_vault):
+ init_run(tmp_vault, "vf-02", profile="light")
+ result = verify_run(tmp_vault, "vf-02")
+ assert result["passed"] is False
+
+ def test_verify_cli_exit_code(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ init_run(tmp_vault, "vf-03", profile="light")
+ monkeypatch.chdir(tmp_vault.root)
+ r = CliRunner().invoke(app, ["run", "verify", "vf-03", "--json"])
+ assert r.exit_code == 1 # no report -> fail
+
+
+class TestCiteCheckerAgentInstall:
+ def test_agent_installs(self, tmp_vault):
+ from hyperresearch.core.hooks import _install_cite_checker_agent
+
+ result = _install_cite_checker_agent(tmp_vault.root, "hyperresearch")
+ assert result is not None
+ body = (tmp_vault.root / ".claude" / "agents" / "hyperresearch-cite-checker.md").read_text(encoding="utf-8")
+ assert "name: hyperresearch-cite-checker" in body
+ assert "model: sonnet" in body
+ assert "wrong-source" in body
+ assert "SKEPTICAL" in body
+ assert "{hpr_path}" not in body
+ assert "<<" not in body
diff --git a/tests/test_web/test_fetch_settings.py b/tests/test_web/test_fetch_settings.py
new file mode 100644
index 0000000..b47856e
--- /dev/null
+++ b/tests/test_web/test_fetch_settings.py
@@ -0,0 +1,78 @@
+"""Tests for FetchSettings threading into the crawl4ai provider internals."""
+
+from __future__ import annotations
+
+import pytest
+
+from hyperresearch.core.config import FetchSettings, JunkGates
+
+crawl4ai_provider = pytest.importorskip(
+ "hyperresearch.web.crawl4ai_provider",
+ reason="crawl4ai extra not installed",
+)
+
+
+class TestSmartWaitJs:
+ def test_defaults_match_pre_20_literals(self):
+ js = crawl4ai_provider._smart_wait_js(FetchSettings())
+ assert "}, 2000);" in js # initial delay
+ assert "}, 500);" in js # poll interval
+ assert "stable >= 2" in js
+ assert "checks > 16" in js
+
+ def test_custom_settings_rendered(self):
+ js = crawl4ai_provider._smart_wait_js(
+ FetchSettings(wait_initial_ms=100, poll_interval_ms=50, stable_checks=5, max_checks=99)
+ )
+ assert "}, 100);" in js
+ assert "}, 50);" in js
+ assert "stable >= 5" in js
+ assert "checks > 99" in js
+
+ def test_js_is_well_formed(self):
+ js = crawl4ai_provider._smart_wait_js(FetchSettings())
+ assert js.startswith("() => new Promise")
+ assert js.count("{") == js.count("}")
+ assert js.count("(") == js.count(")")
+
+
+class TestLooksLikeBinary:
+ def test_pdf_markers_always_binary(self):
+ assert crawl4ai_provider._looks_like_binary("endstream endobj blah") is True
+
+ def test_ratio_uses_gates(self):
+ text = ("real words all over the place here " * 20) + "\x01\x02" * 60
+ assert crawl4ai_provider._looks_like_binary(text) is True
+ assert crawl4ai_provider._looks_like_binary(text, JunkGates(binary_garbage_ratio=0.9)) is False
+
+ def test_empty_is_not_binary(self):
+ assert crawl4ai_provider._looks_like_binary("") is False
+
+
+class TestProviderConstruction:
+ def test_settings_reach_run_config(self):
+ prov = crawl4ai_provider.Crawl4AIProvider(
+ settings=FetchSettings(page_timeout_ms=77777, stable_checks=7)
+ )
+ assert prov._run_config.page_timeout == 77777
+ assert "stable >= 7" in prov._wait_js
+ assert prov._wait_js.startswith("js:")
+
+ def test_default_construction_matches_pre_20(self):
+ prov = crawl4ai_provider.Crawl4AIProvider()
+ assert prov._run_config.page_timeout == 30000
+ assert "stable >= 2" in prov._wait_js
+ assert "checks > 16" in prov._wait_js
+
+
+class TestGetProviderThreading:
+ def test_get_provider_passes_settings(self):
+ from hyperresearch.web.base import get_provider
+
+ prov = get_provider(
+ "crawl4ai",
+ settings=FetchSettings(page_timeout_ms=12345),
+ gates=JunkGates(binary_garbage_ratio=0.9),
+ )
+ assert prov._run_config.page_timeout == 12345
+ assert prov._gates.binary_garbage_ratio == 0.9