Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

v0.4.0: Request-type protocols, framework-first scaffold, note triage

Jordane0eba940238a
diff --git a/.gitignore b/.gitignore
index a86442c..8ecd429 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,3 +13,4 @@ build/
*.egg
.env
bench/
+codex_bench/
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2368487..dbfd2cf 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,25 @@
# Changelog
+## [0.4.0] - 2026-04-13
+
+### New
+
+- **Request-type classification (Step 0)** — The research workflow now starts by classifying the user's request into one of 7 types (Canonical Knowledge Retrieval, Market / Landscape Mapping, Engineering / Technical How-To, Interpretive / Humanities Analysis, Comparative Evaluation, Emerging / Cutting-Edge Research, Forecast / Strategy / Recommendation) plus a General fallback. Classification happens before any searching and governs the rest of the workflow.
+- **Type-specific parameter blocks** — Each of the 7 types specifies its own source strategy (count + primary/secondary mix), target length, opening-section shape, H2 heading count, analytical mode, and special rules. A humanities analysis wants 6–10 long thematic sections; a market landscape wants 8–14 vendor-cluster sections with a mandatory comparison matrix; a cutting-edge research request wants primary-heavy preprint reading with a "What we don't know yet" section. One workflow, seven parameterizations.
+- **Primary-heavy vs. secondary-heavy source policy** — New explicit axis: Types 1/4/5/6 are primary-heavy (cite originals, engage deeply, prune irrelevant secondary coverage), Types 2/7 are secondary-heavy (triangulate across many descriptions), Type 3 is balanced. Source count is now a function of request type, not topic complexity.
+- **Conceptual scaffold step (before writing)** — Agent must answer four questions in a scratch file before drafting: the hard question, the naive answer, the structural tension, and a dependency-ordered heading sketch. The final report's opening section must be a framework section, not a definition.
+- **Cross-source comparison step** — Before writing the body, agent finds 3–5 places where sources actually disagree and captures short comparison blocks. Sources earn citations by being compared, not listed. These become the backbone of body sections.
+- **Writing-draft hard constraints** — Target 400–600 words per H2, 12–20 H2s on a 10K-word report, never one-section-per-source, every section ends with an analytical beat, comparison tables not fact tables. Type-specific blocks override these (Type 4 Humanities targets 800–1500 words per section across 6–10 sections).
+- **Frontmatter-first note triage (Step 4.5)** — Six-level protocol for reading notes efficiently. Always start with `note list -j` for summaries, use `note show --meta -j` for frontmatter-only reads, `search --include-body --max-tokens 6000 -j` for token-capped multi-note pulls, and **delegate notes with `word_count > 6000` to a fresh Sonnet subagent** with a pointed extraction prompt (~40× context savings per large note). Rely on the summary field first; read the body only when it earns its place.
+- **Type-aware adversarial audit** — The structure-auditor subagent now checks whether the draft honors its declared type's parameter block: thematic sections for Humanities, mandatory comparison matrix for Comparative, "What we don't know yet" for Emerging, a position on winners for Market. Flags every type violation.
+
+### Changed
+
+- **`fit_markdown` via PruningContentFilter** — crawl4ai provider now uses `DefaultMarkdownGenerator` with `PruningContentFilter` so fetched notes contain just the main content, stripping navigation, footers, and sidebar chrome. Both `AsyncWebCrawler.arun()` and the Playwright visible-browser path use the same generator for consistent output. Applied to single fetch, batch fetch, and visible browser paths.
+- **Skip numeric wiki-links in note parser** — `[[100]]`-style citation markers in bibliographies and academic papers are no longer extracted as note references. Avoids thousands of spurious broken-link warnings on papers that use numbered references.
+- **"Over-collect, then prune" reframed as "over-collect, then engage deeply"** — A report built from 30 sources that disagree and force you to take positions is worth more than a report built from 80 sources that each contribute one bullet of description. Collection is a means to an argument, not the goal.
+- **Scaffold and comparison artifacts are ephemeral, NOT hyperresearch notes** — Both the conceptual scaffold and the cross-source comparison blocks live in `/tmp/scaffold.md` or working memory, explicitly not as notes. Protects the research base from pre-writing scratch work.
+
## [0.3.0] - 2026-04-11
### New
@@ -98,6 +118,7 @@ Initial release. Forked from [llm-kasten](https://github.com/jordan-gibbs/llm-ka
- Web viewer with force-directed knowledge graph
- 70 tests
+[0.4.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.4.0
[0.3.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.3.0
[0.2.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.2.0
[0.1.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.1.0
diff --git a/pyproject.toml b/pyproject.toml
index 5447b6a..bd397a6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "hyperresearch"
-version = "0.3.0"
+version = "0.4.0"
description = "Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki."
readme = "README.md"
license = "MIT"
diff --git a/src/hyperresearch/__init__.py b/src/hyperresearch/__init__.py
index bb3f10c..3a81bf5 100644
--- a/src/hyperresearch/__init__.py
+++ b/src/hyperresearch/__init__.py
@@ -1,3 +1,3 @@
"""Hyperresearch — agent-driven research knowledge base."""
-__version__ = "0.2.0"
+__version__ = "0.4.0"
diff --git a/src/hyperresearch/core/agent_docs.py b/src/hyperresearch/core/agent_docs.py
index f760f7d..d49f8d7 100644
--- a/src/hyperresearch/core/agent_docs.py
+++ b/src/hyperresearch/core/agent_docs.py
@@ -20,13 +20,17 @@ HYPERRESEARCH_BLURB = """\
**CLI path: `{hpr}`** — use this exact path for all hyperresearch commands below. It may not be on your system PATH.
+**IMPORTANT — paths:** The CLI path above is the location of the `hyperresearch` binary, **nothing else**. It is NOT your working directory. All file and directory paths in this document (`research/notes/`, `research/raw/`, `.hyperresearch/config.toml`, etc.) are **relative to your current working directory**. When you save files with the Write tool, use relative paths like `research/notes/final_report.md` — do NOT prefix them with the directory containing the hyperresearch binary. The `research/` folder lives wherever you are running from.
+
This project uses hyperresearch as an agent-driven research knowledge base. The `research/` directory contains markdown notes collected from web sources and original research. Append `--json` to any command for structured output.
### MANDATORY: How to do research
When the user asks you to research a topic, **you MUST follow this workflow**. Do NOT use WebFetch for source pages — use `{hpr} fetch` instead. It runs a real headless browser, handles JavaScript, bypasses bot detection, saves full content with screenshots and images, and indexes everything for future sessions.
-**This is a deep research tool.** Your job is to go down rabbit holes, cast a wide net, and collect as much raw source material as possible. **Over-collect, then prune.** It is always better to have too many sources than too few — you can deprecate notes later, but you can't un-skip a source you never fetched.
+**This is a deep research tool.** Your job is to go down rabbit holes, cast a wide net, and collect enough raw source material to say something meaningful across it. **Over-collect, then prune.** It is better to have too many sources than too few — you can deprecate notes later, but you can't un-skip a source you never fetched.
+
+**But collection is not the goal.** Collection is a means to an argument. A report built from 30 sources that disagree, contextualize each other, and force you to take positions is worth more than a report built from 80 sources that each contribute one paragraph of description. Optimise for **depth of engagement** (each source read carefully, compared with 2–3 others, cited multiple times across the report) rather than **count of citations**. If you find yourself translating one source into one paragraph or one section, you are building a catalog — stop and restructure.
**There is no time limit.** Deep research can run for hours if the topic demands it. Do multiple rounds of research — as new subtopics, angles, and questions reveal themselves, launch new rounds of fetcher subagents in parallel to chase them down. The first round uncovers the landscape; the second round digs into what you found; the third round fills gaps. Keep going.
@@ -42,12 +46,56 @@ Don't collect sources just to hit a number. Instead, review what you have and as
If the answer to any of these is "yes, there's more to get" — go back and get it. Spawn more fetcher agents, search new angles, follow links from existing notes. Keep going until you genuinely can't find anything new worth adding.
-Quality over quantity — but on complex topics, true comprehensiveness requires quantity. If you've only collected 15-25 sources on a PhD-level question, you've barely scratched the surface. Expect to need:
-- Simple factual question: 20-30 sources
-- Complex or technical topic: 50-80 sources
-- PhD-level research question: 80-100+ sources
+**Step 0: Classify the request and pick a protocol (MANDATORY)** — Different research requests need fundamentally different strategies. Source counts, primary/secondary mix, section structure, target length, analytical mode — all change based on request shape. A single universal protocol is wrong. **Before Step 1, classify the request and state your classification in writing** (in working memory or `/tmp/scaffold.md` — NOT a hyperresearch note).
+
+The 7 request types:
+
+| Type | Name | Use when |
+|---|---|---|
+| 1 | **Canonical Knowledge Retrieval** | Mature field with textbook chapters or canonical surveys; an expert would start by citing a classic source. |
+| 2 | **Market / Landscape Mapping** | Enumerable competitive analysis — list of companies, products, vendors with attributes. |
+| 3 | **Engineering / Technical How-To** | User is trying to DO something. Ideal answer is a procedure or method selection. |
+| 4 | **Interpretive / Humanities Analysis** | Subject is a text, work, tradition, or cultural phenomenon. Good answer takes a position and defends it. |
+| 5 | **Comparative Evaluation** | Prompt names multiple alternatives by name and expects a matrix + pick. |
+| 6 | **Emerging / Cutting-Edge Research** | Most useful material is <2 years old; best sources are arXiv/bioRxiv/conference papers. |
+| 7 | **Forecast / Strategy / Recommendation** | Asks what will happen or should be done; ideal answer commits to a prediction or prescription. |
+| 0 | **General Research (fallback)** | Doesn't cleanly fit one type, or genuinely spans multiple. |
+
+Overlap is expected. Classify a **primary** type and, if applicable, a **secondary** type. Apply the primary's parameter block; borrow special rules from the secondary.
+
+**State your classification** before Step 1:
+1. Primary type: #N — one-sentence justification tied to the prompt.
+2. Secondary type: #N or "none".
+3. Applied parameters (5 lines): source strategy, target length, opening-section shape, target H2 count, analytical mode.
+
+**Request-type parameter blocks** — apply the block for your primary type. These overrides take precedence over the generic defaults referenced in Step 3 and Step 7.
+
+**Type 1 — Canonical Knowledge Retrieval.** Sources: 5–15 primary (seminal papers, textbooks, canonical surveys) + 5–15 secondary. Primary-heavy. Target 6,000–10,000 words. Opening: historical/conceptual progression — how the field arrived at consensus. H2 count: 10–15, sequenced by dependency (concept → result → consequence). Analytical mode: explain WHY canonical results hold; compare competing formulations; signal which textbook to start with. Special rule: sequence by dependency, not by source.
+
+**Type 2 — Market / Landscape Mapping.** Sources: 20–50 secondary (reports, press, official docs) + 5–10 primary (filings, specs, API docs). Secondary-heavy. Target 5,000–9,000 words. Opening: market definition + segmentation + the framework used to score vendors. H2 count: 8–14, one per vendor *cluster* or category — NEVER one per individual vendor. Analytical mode: take a position on who is winning and why. Special rule: at least one comparison matrix in the body is MANDATORY. Never list vendors in isolation. If the user asked for rankings, deliver with confidence.
+
+**Type 3 — Engineering / Technical How-To.** Sources: 5–15 primary (papers, docs, GitHub, RFCs, standards) + 5–15 secondary (blog posts, tutorials, SO). Balanced. Target 4,000–8,000 words. Opening: problem statement + constraint landscape — what makes this hard. H2 count: 8–14. Analytical mode: pick a recommended approach; explain when to use alternatives. Special rule: if a method has a reference implementation, link and characterize it. Include a decision tree or selection matrix.
+
+**Type 4 — Interpretive / Humanities Analysis.** Sources: 3–10 primary (the work itself, key scholarly monographs, canonical commentary) + 10–25 secondary (critical essays, interviews, reviews). Primary-heavy for depth. Target 7,000–11,000 words. Opening: thesis + the interpretive tradition this work sits in. H2 count: **6–10** (fewer, LONGER sections — 800–1500 words each). Sections are THEMATIC, not entity-cataloged. Analytical mode: take and defend an interpretation; engage dissenting readings. Special rule: NEVER one section per character / chapter / author. Special rule: at least 2 body sections must put sources in tension with each other. Special rule: this is the type where catalog-style reports fail hardest — double-check that no section is <400 words or entity-named.
+
+**Type 5 — Comparative Evaluation.** Sources: 3–5 primary per comparand + 2–5 secondary per comparand (so N×5 to N×10 total). Depth scales with N. Target 5,000–10,000 words. Opening: the comparison framework — dimensions being scored and why they matter. H2 count: typically 1 per comparand + 2–4 synthesis sections (~8–12 total). Analytical mode: explicit scoring on each dimension + recommended pick with tradeoffs. Special rule: a comparison matrix in the body is MANDATORY. Special rule: equal depth across comparands — rebalance if uneven.
+
+**Type 6 — Emerging / Cutting-Edge Research.** Sources: 5–15 primary (preprints, conference papers, authoritative reviews if any) + 3–8 secondary (news, blog posts from authors). Primary-dominant. Target 5,000–9,000 words. Opening: the open problem + why prior approaches fell short. H2 count: 8–12. Analytical mode: honest about disagreement; forecast 2–5 years out with caveats. Special rule: if a claim is from a 2024–2026 preprint, say so — don't assume consensus. Include an explicit "What we don't know yet" section. Do NOT pad with mature-field material to hit a source count.
+
+**Type 7 — Forecast / Strategy / Recommendation.** Sources: 15–30 secondary (analyst reports, think-tanks, news) + 5–10 primary (official announcements, filings, datasets). Secondary-heavy. Target 6,000–10,000 words. Opening: the decision/question being answered + the forces shaping it. H2 count: 8–12. Analytical mode: confident prescription with explicit reasoning chain; probability language over hedging. Special rule: at least one body section must commit to a direction — not everything can be deferred to the Opinionated Synthesis. Acknowledge the time horizon explicitly.
+
+**Type 0 — General Research (fallback).** Use universal defaults: 15–25 / 30–50 / 50–80 source floors (see below), framework opening, 12–20 H2 headings, 400–600 words per H2 (see Step 7).
-If you've exhausted every angle and there's genuinely nothing left to find, that's fine — move on. But be honest with yourself about whether you've actually exhausted it or just stopped early.
+**Source-count floors for Type 0 (fallback only):**
+- Simple factual question: 15–25 sources
+- Complex or technical topic: 30–50 sources
+- PhD-level research question: 50–80 sources
+
+**Source count is a function of request type, not topic complexity.** If you classified in Step 0, use that type's parameter block as primary guidance — fall back to the floors above only for Type 0.
+
+**Primary-heavy vs. secondary-heavy is the axis that matters most — getting this wrong is a worse failure than getting the count wrong.** Types 1, 4, 5, 6 are primary-heavy — cite originals and engage with them deeply. Types 2, 7 are secondary-heavy — triangulate across many descriptions. Type 3 is balanced. Most sources should be cited multiple times across multiple sections. If a source earns only one citation, it's probably not pulling its weight — either deepen engagement or cut it.
+
+Stop fetching when you can't find anything new worth adding, not when you hit a number. And be honest about the opposite: if you're still fetching past the type's range without a clear gap in mind, you're procrastinating on the writing.
**Step 1: Check what's already known**
```bash
@@ -96,14 +144,89 @@ Look for links in the content that point to **primary sources, references, relat
**Do NOT stop collecting until you are certain you have enough breadth.** After each round of fetching, ask yourself: "Are there major angles, subtopics, or perspectives I haven't covered yet?" If yes, search and fetch more. If a topic has 5 facets, you need sources on all 5 — not just the first 2 you found. Keep going until you can confidently say the collection covers the full scope of the topic. A half-covered topic is worse than no coverage at all.
-**Step 5: Synthesize** — Once you have the raw sources indexed, search and read them to build your synthesis:
+**Step 4.5: Read notes efficiently — triage by frontmatter (MANDATORY)** — When inspecting notes, NEVER blindly `{hpr} note show` every note. Bodies can be 15,000+ words; frontmatter summaries are usually under 40. **Triage first, body-read only when earned.** This protocol applies throughout Steps 4, 6, and 7.
+
+**Level 1 — Frontmatter sweep (cheap, always).** Start with:
+```bash
+{hpr} note list -j
+```
+Returns per note: `id`, `title`, `tags`, `summary`, `word_count`, `status` — **no bodies**. Scan this to build a relevance shortlist. For many claims the summary alone is enough — cite the source URL directly without re-reading the body.
+
+**Level 2 — Frontmatter-only show.** For a single note whose summary hints at deeper value, pull full frontmatter without body:
+```bash
+{hpr} note show <id> --meta -j
+```
+Adds `source`, `parent`, `created`, `updated`, `raw_file` (if the note has a PDF). Still cheap.
+
+**Level 3 — Inline body read (small notes only).** For `word_count < 2000` (~2700 tokens), read directly:
+```bash
+{hpr} note show <id> -j
+```
+Batch siblings: `{hpr} note show <id1> <id2> <id3> -j`.
+
+**Level 4 — Targeted search with body, token-capped.** When pulling content from multiple notes at once:
+```bash
+{hpr} search "<precise question>" --include-body --max-tokens 6000 -j
+```
+`--max-tokens` truncates intelligently and flags `"truncated": true` on the last fitting result. Use this instead of N separate `note show` calls.
+
+**Level 5 — Delegate heavy notes to a Sonnet subagent (MANDATORY for large notes).** For any note with `word_count > 6000` (~8000 tokens, often PDFs), do NOT dump it into your own context. Spawn a Sonnet subagent with a pointed extraction prompt:
+
+> Fresh Sonnet subagent prompt:
+> "Read `research/notes/<id>.md` (and its raw file at `research/raw/<id>.pdf` if the frontmatter has `raw_file` set). Extract ONLY what answers this specific question: '<your precise question>'. Return under 500 words, with direct quotes for any non-trivial claim and the source URL. If the note doesn't actually answer the question, say so and stop."
+
+The subagent reads the full 20K+ tokens internally but returns only ~500 tokens to you — roughly a 40× context savings per large note, which compounds across a session.
+
+**Level 6 — Raw PDF re-extraction (rare).** If a note has `raw_file: raw/<id>.pdf` AND the extracted markdown looks garbled or is missing content you need (specific figure, table, equation), read the raw PDF directly with the Read tool. Most of the time the extracted markdown is sufficient — only fall back here when it's clearly broken.
+
+**Triage thresholds (guidance, not hard rules):**
+- `word_count < 2000` → **Trivial**: read inline (Level 3), batch with siblings
+- `word_count 2000–6000` → **Medium**: Level 2 frontmatter first; body only if summary suggests high value
+- `word_count > 6000` → **Heavy**: Level 5 Sonnet delegation. Don't read inline.
+
+Override with reason (e.g. a single 8000-word paper is your most important primary source — read it once, directly). But defaulting to "just `note show` everything" is how sessions run out of context and reports turn shallow.
+
+**Step 5: Build the conceptual scaffold (MANDATORY — do this BEFORE writing)** — This is the single biggest lever for report quality. Strong reports are built from a scaffold, not assembled by translating notes into sections. Before you write a single paragraph of prose, answer these four questions.
+
+**Where to put the scaffold:** keep it in your working memory, or dump it to a temp file like `/tmp/scaffold.md`. **DO NOT save it as a hyperresearch note.** The research base is for durable source material, not ephemeral pre-writing analysis. Notes are forever; scaffolds are for the next hour of writing.
+
+1. **What is the hard question?** One sentence — not the surface query, the underlying difficulty. ("Why is eukaryotic HGT rare despite being evolutionarily important?" not "What is HGT?")
+2. **What is the naive answer?** What a reader would guess before reading the sources. This is your foil — the thing the report has to explain away.
+3. **What is the structural tension?** What makes this topic worth 10,000 words instead of 500? The constraint, the paradox, the thing sources disagree on, the reason the field hasn't converged.
+4. **What is the progression?** Sketch 6–12 H2 section headings in **dependency order**. Each section must BUILD on the one before, not sit parallel to it. A reader should only be able to understand section 3 because they read sections 1 and 2. If your headings could be read in any order, that's a catalog, not a report.
+
+The opening section of the final report MUST be a framework section — not "What is X." It should establish the hard question, the naive answer, and why the naive answer is insufficient. This is what separates analytical reports from catalog-style ones.
+
+**Step 6: Compare sources across a common axis (MANDATORY)** — Before writing the body, find 3–5 places where your sources actually disagree, emphasize different things, or frame the same problem differently. Use search to locate them:
```bash
{hpr} search "topic" --include-body -j
```
-Write a draft report with inline URL citations.
+For each disagreement, capture a short comparison block (in your working memory or in the same scratch file as the scaffold from Step 5 — **NOT as a hyperresearch note**; these are ephemeral analysis artifacts, not durable sources):
+- what each source says, with URLs
+- why they differ (different data? methodology? assumptions? era? scope?)
+- your position on which is more credible, with reasoning
+
+These comparisons become the backbone of body sections — they will end up as prose *inside* the final report, not as separate KB entries. Sources earn their citations by being **compared**, not by being listed. If every source is corroborating every other source, you didn't need many sources — go back and search for contrarian views, criticism, or alternative frameworks. Reference-quality reports always surface disagreement; catalog-style reports never do.
+
+**Step 7: Write the draft** — First, pull up your Step 0 classification. The parameter block for your primary type sets target length, H2 count, opening shape, and analytical mode. The hard constraints below are Type 0 defaults — they apply EXCEPT where your type's block overrides them. Notable overrides:
+- **Type 4 Humanities** targets 6–10 longer H2s (800–1500 words each), not 12–20.
+- **Type 2 Market** wants one section per vendor *cluster*, not per individual vendor.
+- **Type 5 Comparative** requires a mandatory comparison matrix in the body.
+- **Type 6 Emerging** requires a "What we don't know yet" section.
-**Step 6: Gap analysis (MANDATORY)** — After writing your draft, stop and compare it against the original query. Re-read the user's request word by word. Ask yourself:
+Now, and only now, write the report. Follow these rules as hard constraints (under Type 0; type-specific overrides apply where listed above):
+
+- **Target 400–600 words per H2 section.** If a section is under 200 words, merge it into a sibling. If over 800, split it — but only if the split sections stay above 300. (Type 4 Humanities: 800–1500 per section.)
+- **Aim for 12–20 H2 headings on a 10K-word report; 8–15 on a 5K-word report.** If you're above 25, you are fragmenting. Merge. (Type 4 Humanities: 6–10.)
+- **Never write one section per source or one section per named entity.** Sources serve sections, not the other way around. A single H2 section should weave 3–8 sources together. A character, vendor, or paper does not earn its own section unless it has >400 words of analysis attached.
+- **Every body section ends with an analytical beat** — a 1–2 sentence "so what does this tell us" that integrates across the sources just cited. Not a summary; a step forward in the argument. The Opinionated Synthesis at the end is NOT supposed to carry the synthesis load alone.
+- **Open with the framework, not a definition.** Section 1 = the hard question, the naive answer, the structural tension. Not "X is a Y that does Z." Definitions can go into a sub-section or a sidebar.
+- **Cite inline with parenthetical URLs.** Every non-trivial claim needs one. Prefer primary sources over secondary summaries.
+- **Use comparison tables, not fact tables.** A good table contrasts cases on a common axis (e.g., Recipient → Donor → Gene → Function). A bad table dumps attributes of a single entity (Name | Star | Technique | Fate). If your table is dumping facts about one thing, it probably belongs in prose.
+- **Word count targets by topic difficulty.** Simple factual: 3,000–5,000. Complex or technical: 6,000–9,000. PhD-level: 9,000–12,000. Hitting these is easier when sections are 400–600 words each, not 80–150. Missing these is usually a signal that the scaffold from Step 5 is off — revisit it.
+
+**Step 8: Gap analysis (MANDATORY)** — After writing your draft, stop and compare it against the original query. Re-read the user's request word by word. Ask yourself:
- What specific questions did they ask that I haven't fully answered?
- What subtopics or angles did I miss entirely?
- Where did I make claims without strong enough evidence?
@@ -111,19 +234,20 @@ Write a draft report with inline URL citations.
Then launch a new round of research to fill every gap — web search, fetch, follow links, the full workflow. Spawn fetcher subagents in parallel for the new URLs. Add the new material to your draft.
-**Step 7: Adversarial audit (MANDATORY — do this twice)** — After the gap-filling round, launch two subagents in parallel to audit the revised draft. This runs up to 2 loops.
+**Step 9: Adversarial audit (MANDATORY — do this twice)** — After the gap-filling round, launch two subagents in parallel to audit the revised draft. This runs up to 2 loops.
Spawn two agents in parallel:
- **Agent 1 — Comprehensiveness auditor:** "Read this report and search the research base (`{hpr} search` and `{hpr} note show`) to find gaps. What subtopics, angles, data points, or counterarguments are missing? What claims lack citations? What sections are shallow? Compare against the original query and be ruthless — list every gap."
-- **Agent 2 — Logic and structure auditor:** "Read this report as a domain expert. Does the argument flow logically? Are conclusions supported by the evidence presented? Are there logical leaps, unsupported claims, or contradictions? Is the structure optimal for the topic? Be specific — list every weakness."
+- **Agent 2 — Logic and structure auditor:** "Read this report as a domain expert. The report should declare its Step 0 request-type classification and follow that type's parameter block. Check specifically: (1) Does section 1 establish a framework (hard question, naive answer, structural tension) or just define terms? (2) Are sections in dependency order — does each build on the previous, or could they be shuffled? (3) Short sections: are there any under 200 words that should be merged into siblings? (Type 4 Humanities floor is 400, not 200.) (4) Count H2 headings. Fragmentation threshold varies by type: Type 0 General >25 on a <12K-word report; Type 4 Humanities >12; Type 2 Market >15. (5) Does each section end with an analytical beat, or does it just stop? (6) Are sources compared against each other anywhere, or only listed in parallel? (7) Does the argument flow logically — leaps, unsupported claims, contradictions? (8) Does the report match its declared request type? If Type 4 Humanities, are sections thematic rather than entity-catalog? If Type 5 Comparative, is there a mandatory comparison matrix? If Type 6 Emerging, is there a 'What we don't know yet' section? If Type 2 Market, is there a position on who is winning? Flag every type violation. List every weakness with specific section names and quotes."
Pass your full draft report text to both agents. They search the research base independently and report back.
**After each audit round:**
1. Read both agents' feedback
2. If they identified missing topics or weak arguments — do another round of web search and fetching to fill the gaps
-3. Rewrite the report incorporating new sources and fixing structural issues
-4. Run the audit again (second loop) on the revised report
+3. If the structure auditor flagged fragmentation, over-short sections, or a missing framework — **consolidate headings and rewrite, don't just patch.** Merging five 100-word sections into one 500-word section is a rewrite, not an edit. Do not skip this; it's where catalog-style drafts become argument-style drafts.
+4. Rewrite the report incorporating new sources and fixing structural issues
+5. Run the audit again (second loop) on the revised report
**You MUST do at least one audit loop. Do two if the first audit found significant gaps.** Only after the auditors have no major complaints should you finalize the report.
@@ -134,7 +258,40 @@ Pass your full draft report text to both agents. They search the research base i
- Run `{hpr} lint -j` and fix any issues it finds
- Read notes you haven't read yet and look for links worth following in future rounds
-**Step 8: Prune** — After the final report is done, deprecate notes that turned out to be irrelevant:
+**Step 10: Opinionated Synthesis (MANDATORY — end of report)** — A catalog of facts is not a report. You **MUST** append a section titled `## Opinionated Synthesis` at the very end of the report. This is where you stop reciting what sources said and start reasoning across them. **This section complements — it does not replace — the per-section analytical beats from Step 7.** If the body sections are purely descriptive and all the analysis is dumped in this section, the report is still a catalog with a synthesis bolted onto the end.
+
+The synthesis section must include:
+
+1. **Comparative analysis across categories** — don't just describe each concept/group/topic in isolation. Compare them. Use tables where useful.
+2. **Thematic threads and progression** — what patterns cut across the topic? What's the narrative arc?
+3. **Your reasoned position** — take a defensible stance, clearly labeled as your own analysis grounded in the sources.
+4. **Open questions and limitations** — what did the sources NOT answer?
+5. **Concluding thoughts** — one tight paragraph answering the original question.
+
+Use these exact subheadings at the end of your report:
+
+```markdown
+## Opinionated Synthesis
+
+### Comparative Analysis
+<cross-cutting comparisons — tables where useful>
+
+### Thematic Threads
+<patterns, progressions, throughlines>
+
+### My Reasoned Position
+<your defensible stance, clearly labeled as analysis>
+
+### Open Questions
+<gaps in the research, what the sources didn't answer>
+
+### Concluding Thoughts
+<one tight paragraph, answers the original question>
+```
+
+**Do NOT skip this section.** Even a 9,000-word body of cataloged facts is incomplete without it. The synthesis is what separates a research report from a Wikipedia dump.
+
+**Step 11: Prune** — After the final report is done, deprecate notes that turned out to be irrelevant:
```bash
{hpr} note update <id> --status deprecated -j
```
diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.py
index b9abd32..9ea8c37 100644
--- a/src/hyperresearch/core/note.py
+++ b/src/hyperresearch/core/note.py
@@ -32,6 +32,7 @@ def read_note(file_path: Path, vault_root: Path) -> Note:
outgoing = list(dict.fromkeys(
ref for ref in raw_links
if ref and not ref.startswith(("http://", "https://", "ftp://", "mailto:"))
+ and not ref.isdigit() # Skip [[100]]-style citation numbers
))
# Word count (simple split)
diff --git a/src/hyperresearch/skills/research.md b/src/hyperresearch/skills/research.md
index 72ada47..22129d4 100644
--- a/src/hyperresearch/skills/research.md
+++ b/src/hyperresearch/skills/research.md
@@ -19,6 +19,116 @@ Before doing ANY research, evaluate the user's request. If it is vague, ambiguou
Ask 1-3 focused questions. Once you understand the scope, proceed. If the request is already specific and clear, skip this step and start researching immediately.
+## Step 0.5: Classify the request and pick a protocol (MANDATORY)
+
+Different research requests need fundamentally different strategies. Source counts, primary/secondary mix, section structure, target length, analytical mode — all change based on the request shape. A single universal protocol is wrong. **Before Step 1, classify the request into one of the 7 types below and state your classification in writing** (in working memory or `/tmp/scaffold.md` — NOT a hyperresearch note).
+
+### The 7 request types
+
+| Type | Name | Use when |
+|---|---|---|
+| 1 | **Canonical Knowledge Retrieval** | Mature field with textbook chapters or canonical survey papers; a domain expert would start by citing a classic source. |
+| 2 | **Market / Landscape Mapping** | Enumerable competitive analysis — list of companies, products, vendors with attributes. |
+| 3 | **Engineering / Technical How-To** | User is trying to DO something. Ideal answer is a procedure or method selection. |
+| 4 | **Interpretive / Humanities Analysis** | Subject is a text, work, tradition, or cultural phenomenon. Good answer takes a position and defends it. |
+| 5 | **Comparative Evaluation** | Prompt names multiple alternatives by name and expects a matrix + pick. |
+| 6 | **Emerging / Cutting-Edge Research** | Most useful material is <2 years old; best sources are arXiv/bioRxiv/conference papers. |
+| 7 | **Forecast / Strategy / Recommendation** | Asks what will happen or what should be done; ideal answer commits to a prediction or prescription. |
+| 0 | **General Research (fallback)** | Doesn't cleanly fit one type, or genuinely spans multiple. |
+
+Overlap is expected. Classify a **primary** type and, if applicable, a **secondary** type. Apply the parameter block for the primary; borrow special rules from the secondary.
+
+### State your classification
+
+Before Step 1, write this down:
+
+1. **Primary type**: #N — one-sentence justification tied to the prompt.
+2. **Secondary type**: #N or "none".
+3. **Applied parameters** (5 lines): source strategy, target length, opening-section shape, target H2 count, analytical mode.
+
+Refer back to this throughout the workflow. It's how you avoid drifting into one-size-fits-all defaults.
+
+---
+
+## Request-type parameter blocks
+
+Pick the block for your primary type and apply it. These **override** the generic defaults in Step 3 (source count) and Step 9 (writing constraints).
+
+### Type 1 — Canonical Knowledge Retrieval
+
+- **Source strategy:** 5–15 primary (seminal papers, textbooks, canonical surveys) + 5–15 secondary (recent reviews, textbook chapters). Primary-heavy. Read primary DEEP.
+- **Target length:** 6,000–10,000 words
+- **Opening:** historical / conceptual progression — how the field arrived at current consensus
+- **H2 count:** 10–15, sequenced by dependency (concept → result → consequence)
+- **Analytical mode:** explain WHY canonical results hold; compare competing formulations; signal which textbook to start with
+- **Special rule:** sequence content by dependency, not by source. A reader should need sections 1 and 2 to understand section 3.
+
+### Type 2 — Market / Landscape Mapping
+
+- **Source strategy:** 20–50 secondary (analyst reports, press, official docs) + 5–10 primary (filings, product specs, API docs). Secondary-heavy — triangulation across many descriptions is correct here.
+- **Target length:** 5,000–9,000 words
+- **Opening:** market definition + segmentation + the framework used to score vendors
+- **H2 count:** 8–14, one per vendor *cluster* or category — NEVER one per individual vendor
+- **Analytical mode:** take a position on who is winning and why
+- **Special rule:** at least one comparison matrix in the body, with common axes, is MANDATORY. Never list vendors in isolation.
+- **Special rule:** if the user asked for rankings, deliver them with confidence. Rankings without commitment are worthless.
+
+### Type 3 — Engineering / Technical How-To
+
+- **Source strategy:** 5–15 primary (papers, docs, GitHub repos, RFCs, standards) + 5–15 secondary (blog posts, tutorials, StackOverflow). Roughly balanced.
+- **Target length:** 4,000–8,000 words
+- **Opening:** the problem statement + the constraint landscape — what makes this hard, why naive approaches fail
+- **H2 count:** 8–14
+- **Analytical mode:** pick a recommended approach; explain when to use alternatives
+- **Special rule:** if a method has a reference implementation, link it and characterize it
+- **Special rule:** include a decision tree or selection matrix ("use X when..., use Y when...")
+
+### Type 4 — Interpretive / Humanities Analysis
+
+- **Source strategy:** 3–10 primary (the work itself, key scholarly monographs, canonical commentary) + 10–25 secondary (critical essays, interviews, reviews). Primary-heavy for depth.
+- **Target length:** 7,000–11,000 words
+- **Opening:** thesis + the interpretive tradition this work sits in
+- **H2 count:** **6–10** (fewer, LONGER sections — 800–1500 words each). Sections are THEMATIC, not entity-cataloged.
+- **Analytical mode:** take and defend an interpretation; engage dissenting readings
+- **Special rule:** NEVER one section per character / chapter / author. Sections must be thematic (e.g. "Sacrifice as cosmology" — not "Pegasus Seiya").
+- **Special rule:** at least 2 body sections must put sources in tension with each other — find where scholars disagree and walk through it.
+- **Special rule:** this is the type where catalog-style reports fail hardest. Before submitting, double-check that no section is <400 words or entity-named.
+
+### Type 5 — Comparative Evaluation
+
+- **Source strategy:** 3–5 primary per comparand + 2–5 secondary per comparand (so N×5 to N×10 total). Depth scales with N.
+- **Target length:** 5,000–10,000 words
+- **Opening:** the comparison framework itself — the dimensions being scored and why they matter
+- **H2 count:** typically 1 per comparand + 2–4 synthesis sections (~8–12 total)
+- **Analytical mode:** explicit scoring on each dimension + a recommended pick with tradeoffs
+- **Special rule:** one comparison matrix in the body is MANDATORY (not optional)
+- **Special rule:** equal depth across comparands — if one section is 400 words and another is 1200, rebalance
+
+### Type 6 — Emerging / Cutting-Edge Research
+
+- **Source strategy:** 5–15 primary (preprints, conference papers, authoritative reviews if any) + 3–8 secondary (news, blog posts from authors). Primary-dominant.
+- **Target length:** 5,000–9,000 words
+- **Opening:** the open problem + why prior approaches fell short
+- **H2 count:** 8–12
+- **Analytical mode:** honest about disagreement; forecast 2–5 years out with caveats
+- **Special rule:** if a claim is from a 2024–2026 preprint, say so. Don't assume consensus in fast-moving fields.
+- **Special rule:** include an explicit "What we don't know yet" section
+- **Special rule:** do NOT pad with mature-field material to hit a source count. Fewer primary sources deeply engaged beats more sources shallowly skimmed.
+
+### Type 7 — Forecast / Strategy / Recommendation
+
+- **Source strategy:** 15–30 secondary (analyst reports, think-tanks, news) + 5–10 primary (official announcements, filings, datasets). Secondary-heavy.
+- **Target length:** 6,000–10,000 words
+- **Opening:** the decision / question being answered + the forces shaping it
+- **H2 count:** 8–12
+- **Analytical mode:** confident prescription with an explicit reasoning chain. Probability language over hedging.
+- **Special rule:** at least one body section must commit to a direction — not everything can be deferred to the Opinionated Synthesis at the end.
+- **Special rule:** acknowledge the time horizon explicitly — what's true in 6 months vs. 5 years.
+
+### Type 0 — General Research (fallback)
+
+Use the generic defaults: 15–25 / 30–50 / 50–80 source floors (see Step 3), framework opening, 12–20 H2 headings, 400–600 words per H2 (see Step 9).
+
---
When invoked, follow these steps **in order**. Do not skip steps. The CLI path will be provided by the CLAUDE.md — look for the "CLI path:" line in the Research Base section.
@@ -65,7 +175,7 @@ Spawn hyperresearch-fetcher agents for each URL batch:
Each fetcher will report back with note IDs and interesting links found in the content.
-**Over-collect.** It is always better to have too many sources than too few. You can deprecate notes later.
+**Over-collect, then engage deeply.** It's better to have too many sources than too few — but collection is a means to an argument, not the goal. A report built from 30 sources that disagree and force you to take positions beats a report built from 80 sources that each contribute one bullet of description. Read each source carefully. If you find yourself translating one source into one paragraph or one section, stop — you are building a catalog.
**There is no time limit.** Do multiple rounds of fetching — as new subtopics and questions reveal themselves, launch new rounds of fetcher subagents in parallel to chase them down. The first round uncovers the landscape, the second digs into what you found, the third fills gaps. Keep going until you've exhausted the topic. Do not rush.
@@ -79,7 +189,14 @@ Review what you have. Don't chase a number — chase completeness:
- Are there counterarguments or alternative viewpoints missing?
- Do you have primary sources or just secondhand summaries?
-If there's more to get, go get it. If you've genuinely exhausted the topic, move on. But be honest — on complex topics, expect 50-80+ sources before you've truly covered it.
+If there's more to get, go get it. **Source count is a function of request type, not topic complexity** — use the parameter block from Step 0.5 (your classified primary type) as your primary guidance. Fall back to the universal floors below ONLY if you classified as Type 0 General Research:
+- Simple factual question: ~15–25 sources
+- Complex or technical topic: ~30–50 sources
+- PhD-level research question: ~50–80 sources
+
+**Primary-heavy vs. secondary-heavy is the axis that matters most — getting this wrong is a worse failure than getting the count wrong.** "Primary" means the claim comes from first-hand: an original paper, an official document, a dataset, a product spec. "Secondary" means any description or discussion of a primary source. Types 1, 4, 5, 6 are primary-heavy — cite originals, engage with them deeply, prune irrelevant secondary coverage. Types 2, 7 are secondary-heavy — triangulate across many descriptions because no single secondary source is authoritative. Type 3 is balanced.
+
+Stop fetching when you can't find anything new worth adding, not when you hit a number. And be honest about the opposite failure: if you're still fetching past the type's range without a clear gap in mind, you're procrastinating on the writing.
## Step 4: Go down rabbit holes
@@ -115,6 +232,69 @@ Spawn more fetcher agents for these secondary sources.
**Do NOT stop collecting until you are certain you have enough breadth.** After each round of fetching, ask yourself: "Are there major angles, subtopics, or perspectives I haven't covered yet?" If yes, search and fetch more. If a topic has 5 facets, you need sources on all 5 — not just the first 2 you found. Keep going until you can confidently say the collection covers the full scope of the topic.
+## Step 4.5: Read notes efficiently — triage by frontmatter (MANDATORY)
+
+When inspecting notes in the research base, NEVER blindly `note show` every note. Bodies can be 15,000+ words; frontmatter summaries are usually under 40. **Triage first, body-read only when earned.** This protocol applies throughout the rest of the workflow — use it in Step 4 (rabbit holes), Step 8 (cross-source comparison), and Step 9 (draft writing).
+
+### Level 1 — Frontmatter sweep (cheap, always start here)
+
+```bash
+$HPR note list -j
+```
+
+Returns one object per note with `id`, `title`, `tags`, `summary`, `word_count`, `status` — **no bodies**. Scan this to build a relevance shortlist. For many claims the summary field is enough — a well-curated note's summary already tells you what the source claims, and you cite the source URL directly without re-reading the body.
+
+### Level 2 — Frontmatter-only show (targeted)
+
+For a single note whose summary hints at deeper value, pull its full frontmatter without the body:
+
+```bash
+$HPR note show <id> --meta -j
+```
+
+Adds `source`, `parent`, `created`, `updated`, `raw_file` (if the note has a PDF). Still cheap. Decide from here whether the body is worth pulling.
+
+### Level 3 — Inline body read (only for small notes)
+
+```bash
+$HPR note show <id> -j
+```
+
+Threshold: `word_count < 2000` (~2700 tokens). Small enough to read directly. Batch several at once: `note show <id1> <id2> <id3> -j`.
+
+### Level 4 — Targeted search with body, token-capped
+
+When you need content from multiple notes at once:
+
+```bash
+$HPR search "<precise question>" --include-body --max-tokens 6000 -j
+```
+
+`--max-tokens` truncates intelligently and flags `"truncated": true` on the last fitting result. Prefer this over N separate `note show` calls.
+
+### Level 5 — Delegate heavy notes to a Sonnet subagent (MANDATORY for large notes)
+
+For any note with `word_count > 6000` (~8000 tokens, often PDFs), do NOT dump it into your own context. Spawn a Sonnet subagent with a pointed extraction prompt:
+
+> Use a fresh Sonnet subagent:
+> "Read `research/notes/<id>.md` (and its raw file at `research/raw/<id>.pdf` if the frontmatter has `raw_file` set). Extract ONLY what answers this specific question: '<your precise question>'. Return under 500 words, with direct quotes for any non-trivial claim and the source URL. If the note doesn't actually answer the question, say so and stop."
+
+The subagent reads the full 20K+ tokens internally but returns only ~500 tokens to you — roughly a 40× context savings per large note, which compounds across a session.
+
+### Level 6 — Raw PDF re-extraction (rare)
+
+If a note has `raw_file: raw/<id>.pdf` in its frontmatter AND the extracted markdown looks garbled or is missing content you need (specific figure, table, equation), read the raw PDF directly with the Read tool. Most of the time the extracted markdown is sufficient — only fall back here when it's clearly broken.
+
+### Triage thresholds (guidance, not hard rules)
+
+| word_count | Tier | Action |
+|---|---|---|
+| `< 2000` | Trivial | Read inline via Level 3. Batch with siblings. |
+| `2000 – 6000` | Medium | Level 2 frontmatter first. Body only if the summary suggests high value. |
+| `> 6000` | Heavy | Level 5 Sonnet delegation. Don't read inline. |
+
+Override with reason (e.g. a single 8000-word paper is your most important primary source — read it directly, once). But defaulting to "just `note show` everything" is how sessions run out of context and reports turn shallow.
+
## Step 5: Auto-link and curate
Run the auto-linker to connect related notes:
@@ -145,17 +325,54 @@ $HPR graph hubs -j
$HPR graph backlinks <most-linked-note-id> -j
```
-## Step 7: Synthesize draft report
+## Step 7: Build the conceptual scaffold (MANDATORY — before writing)
-Search the full research base with bodies:
+This is the single biggest lever for report quality. Reports that look like references are built from a scaffold; reports that look like Wikipedia dumps are assembled by translating notes into sections. Before a single paragraph of prose, answer these four questions.
+
+**Where to put the scaffold:** keep it in your working memory, or dump it to a temp file like `/tmp/scaffold.md`. **DO NOT save it as a hyperresearch note.** The research base is for durable source material, not ephemeral pre-writing analysis. Notes are forever; scaffolds are for the next hour of writing.
+
+1. **What is the hard question?** One sentence — the underlying difficulty, not the surface query.
+2. **What is the naive answer?** What a reader would guess before reading any sources. This is your foil — the thing the report has to explain away.
+3. **What is the structural tension?** What makes this topic worth 10,000 words? The paradox, the constraint, the thing sources disagree on, the unsettled question.
+4. **What is the progression?** Sketch 6–12 H2 section headings in **dependency order**. Each section must build on the previous one. If the headings could be shuffled, you have a catalog, not a report.
+
+The final report's **opening section MUST be a framework section**, not a definition section. Section 1 establishes the hard question and the structural tension. Not "X is a Y that does Z."
+
+## Step 8: Cross-source comparison (MANDATORY)
+
+Sources earn their citations by being compared against each other, not by being listed in parallel. Find 3–5 places where your sources actually disagree, emphasize different things, or frame the same problem differently.
```bash
$HPR search "<topic>" --include-body -j
```
-Read the top notes and write a comprehensive draft report with inline URL citations.
+For each disagreement, capture a short comparison block in the same scratch file as the scaffold from Step 7 — **NOT as a hyperresearch note**. These are ephemeral analysis artifacts that will end up as prose inside the final report, not durable KB entries:
+- what each source says, with URLs
+- why they differ (data, methodology, assumptions, era, scope)
+- your position on which is more credible, with reasoning
+
+These comparisons become the backbone of body sections. If every source is corroborating every other source, you didn't need many sources — search again for contrarian views, criticism, or competing frameworks.
+
+## Step 9: Write the draft
+
+**First, pull up your classification from Step 0.5.** The parameter block for your primary type sets target length, H2 count, opening shape, and analytical mode. The hard constraints below are Type 0 defaults — they apply EXCEPT where your type's block overrides them. Notable overrides:
+- **Type 4 Humanities** targets 6–10 longer H2s (800–1500 words each), not 12–20.
+- **Type 2 Market** wants one section per vendor *cluster*, not per individual vendor.
+- **Type 5 Comparative** requires a mandatory comparison matrix in the body.
+- **Type 6 Emerging** requires a "What we don't know yet" section.
+
+Now write the report. These are hard constraints (under Type 0; type-specific overrides apply where listed above):
-## Step 8: Gap analysis (MANDATORY)
+- **Target 400–600 words per H2 section.** Sections under 200 words get merged. Sections over 800 get split — only if both halves stay above 300. (Type 4 Humanities: 800–1500.)
+- **Aim for 12–20 H2 headings on a 10K-word report; 8–15 on a 5K-word report.** Above 25 is fragmentation — merge. (Type 4 Humanities: 6–10.)
+- **Never write one section per source or one section per named entity.** A single H2 section weaves 3–8 sources together. A character, vendor, or paper does not earn its own section unless it has >400 words of real analysis attached.
+- **Every body section ends with an analytical beat** — a 1–2 sentence "so what does this tell us" that integrates across the sources just cited. The Opinionated Synthesis at the end is NOT supposed to carry the synthesis load alone.
+- **Open with the framework from Step 7, not a definition.** Section 1 = hard question + naive answer + structural tension.
+- **Cite inline with parenthetical URLs.** Every non-trivial claim needs one. Prefer primary sources.
+- **Comparison tables over fact tables.** A good table contrasts cases on a common axis. A bad table dumps attributes of one entity.
+- **Word count targets.** Simple factual: 3,000–5,000. Complex/technical: 6,000–9,000. PhD-level: 9,000–12,000. Missing these is usually a signal that the scaffold from Step 7 is wrong — go back and fix the scaffold before writing more prose.
+
+## Step 10: Gap analysis (MANDATORY)
After writing your draft, stop and compare it against the original query. Re-read the user's request word by word:
- What specific questions did they ask that I haven't fully answered?
@@ -165,7 +382,7 @@ After writing your draft, stop and compare it against the original query. Re-rea
Launch a new round of research to fill every gap — web search, fetch, follow links, spawn fetcher subagents in parallel. Add the new material to your draft.
-## Step 9: Adversarial audit (MANDATORY — do this twice)
+## Step 11: Adversarial audit (MANDATORY — do this twice)
After the gap-filling round, launch two subagents in parallel to audit the revised draft:
@@ -179,9 +396,24 @@ list every gap you find.
**Agent 2 — Logic and structure auditor:**
```
-Read this report as a domain expert. Does the argument flow logically? Are conclusions
-supported by the evidence presented? Are there logical leaps, unsupported claims, or
-contradictions? Is the structure optimal for the topic? Be specific — list every weakness.
+Read this report as a domain expert. The report should declare its request type
+(Step 0.5 classification) and follow the matching parameter block. Check specifically:
+ (1) Does section 1 establish a framework — hard question, naive answer, structural
+ tension — or does it just define terms?
+ (2) Are sections in dependency order? Does each section build on the previous one, or
+ could they be shuffled without loss?
+ (3) Are there sections under 200 words that should be merged into siblings?
+ (Type 4 Humanities: <400 is the floor, not 200.)
+ (4) Count H2 headings. Fragmentation thresholds vary by type: Type 0 General is >25 on
+ a <12K-word report; Type 4 Humanities is >12; Type 2 Market is >15.
+ (5) Does each body section end with an analytical beat, or does it just stop?
+ (6) Are sources compared against each other anywhere, or only listed in parallel?
+ (7) Does the argument flow logically — leaps, unsupported claims, contradictions?
+ (8) Does the report match its declared request type? If Type 4 Humanities, are sections
+ thematic rather than entity-catalog? If Type 5 Comparative, is there a mandatory
+ comparison matrix? If Type 6 Emerging, is there a "What we don't know yet" section?
+ If Type 2 Market, is there a position on who is winning? Flag every type violation.
+List every weakness with specific section names and direct quotes.
```
Pass your full draft report text to both agents.
@@ -189,8 +421,9 @@ Pass your full draft report text to both agents.
**After each audit round:**
1. Read both agents' feedback
2. If they found missing topics or weak arguments — do another round of web search + fetching
-3. Rewrite the report with new sources and structural fixes
-4. Run the audit again (second loop) on the revised report
+3. If the structure auditor flagged fragmentation, short sections, or a missing framework — **consolidate and rewrite, don't just patch.** Merging five 100-word sections into one 500-word section is a rewrite, not an edit. This is where catalog-style drafts become argument-style drafts.
+4. Rewrite the report with new sources and structural fixes
+5. Run the audit again (second loop) on the revised report
**Do at least one audit loop. Do two if the first found significant gaps.** Only finalize after auditors have no major complaints.
@@ -201,7 +434,48 @@ Pass your full draft report text to both agents.
- Run `$HPR lint -j` and fix issues
- Read notes you haven't reviewed yet for links worth following
-## Step 10: Save the final report
+## Step 12: Opinionated synthesis (MANDATORY — end of report)
+
+A comprehensive catalog of facts is not a report. Before finalizing, you **MUST** append a section titled **`## Opinionated Synthesis`** at the very end of the report. This is where you stop reciting what sources said and start **reasoning across them**.
+
+This section **complements — it does not replace** — the per-section analytical beats from Step 9. If all the analysis is dumped in this appendix and the body sections are purely descriptive, the report is still a catalog with a synthesis bolted onto the end. Both are required.
+
+**Your Opinionated Synthesis section MUST include:**
+
+1. **Comparative analysis across categories.** Don't just describe each group or concept in isolation — compare them. What do the groups have in common? Where do they differ? What patterns emerge when you look at them side by side? Build at least one comparison table if the topic supports it.
+
+2. **Thematic threads and progression.** What themes cut across the whole topic? What's the narrative arc? How do things evolve over time, across versions, across authors, across schools of thought? Identify the throughlines.
+
+3. **Your reasoned position.** Take a defensible stance. What's the most important finding? What's over-hyped? What's underrated? What do you expect to change in the next 2–5 years? Label it clearly as your own analysis, grounded in the sources you fetched.
+
+4. **Open questions and limitations.** What did the sources NOT answer? What would an expert say is still unclear? Where is the research thin? Name the gaps explicitly.
+
+5. **Concluding thoughts.** One tight paragraph that ties everything back to the user's original question. What's the takeaway they should remember?
+
+**Structure of the synthesis section (use these subheadings exactly):**
+
+```markdown
+## Opinionated Synthesis
+
+### Comparative Analysis
+<cross-cutting comparisons — tables where useful>
+
+### Thematic Threads
+<patterns, progressions, throughlines across the topic>
+
+### My Reasoned Position
+<your defensible stance, clearly labeled as analysis>
+
+### Open Questions
+<gaps in the research, what the sources didn't answer>
+
+### Concluding Thoughts
+<one tight paragraph, answers the original question>
+```
+
+**Do NOT skip this section.** A report without an Opinionated Synthesis is incomplete. Even 9,000 words of carefully cataloged facts is just reference material without it. The synthesis is what separates a research report from a Wikipedia dump.
+
+## Step 13: Save the final report
Save the final reviewed report as a synthesis note:
@@ -222,7 +496,7 @@ SYNTHESIS
$HPR note new "Synthesis: <topic>" --tag <topic> --tag synthesis --type moc --status review --body-file /tmp/synthesis.md --summary "<one-line summary of findings>" -j
```
-## Step 11: Present to user
+## Step 14: Present to user
Present your findings with:
diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.py
index 75740b1..af1345d 100644
--- a/src/hyperresearch/web/crawl4ai_provider.py
+++ b/src/hyperresearch/web/crawl4ai_provider.py
@@ -14,7 +14,8 @@ import pathlib
import sys
from datetime import UTC, datetime
-from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, DefaultMarkdownGenerator
+from crawl4ai.content_filter_strategy import PruningContentFilter
from hyperresearch.web.base import WebResult
@@ -177,12 +178,17 @@ class Crawl4AIProvider:
" }, 2000);"
"})"
)
+ # Use PruningContentFilter to populate fit_markdown (strips nav/footer chrome)
+ self._md_generator = DefaultMarkdownGenerator(
+ content_filter=PruningContentFilter(),
+ )
self._run_config = CrawlerRunConfig(
magic=magic,
simulate_user=True,
screenshot=True,
page_timeout=30000,
wait_for=self._wait_js,
+ markdown_generator=self._md_generator,
)
def fetch(self, url: str) -> WebResult:
@@ -249,12 +255,12 @@ class Crawl4AIProvider:
await context.close()
# Convert HTML to markdown using crawl4ai's markdown generator
- from crawl4ai import DefaultMarkdownGenerator
-
- md_gen = DefaultMarkdownGenerator()
- md_result = md_gen.generate_markdown(html, base_url=final_url)
+ # Prefer fit_markdown (main content, no nav/footer chrome) over raw_markdown.
+ md_result = self._md_generator.generate_markdown(html, base_url=final_url)
content = ""
- if md_result and hasattr(md_result, "raw_markdown"):
+ if md_result and hasattr(md_result, "fit_markdown"):
+ content = md_result.fit_markdown or md_result.raw_markdown or ""
+ elif md_result and hasattr(md_result, "raw_markdown"):
content = md_result.raw_markdown or ""
elif isinstance(md_result, str):
content = md_result
@@ -276,8 +282,11 @@ class Crawl4AIProvider:
# result.markdown is a MarkdownGenerationResult with .raw_markdown,
# .fit_markdown, .markdown_with_citations, etc.
+ # Prefer fit_markdown (main content, no nav/footer chrome) over raw_markdown.
md = result.markdown
- if md and hasattr(md, "raw_markdown"):
+ if md and hasattr(md, "fit_markdown"):
+ content = md.fit_markdown or md.raw_markdown or ""
+ elif md and hasattr(md, "raw_markdown"):
content = md.raw_markdown or ""
elif isinstance(md, str):
content = md
@@ -345,7 +354,9 @@ class Crawl4AIProvider:
continue
metadata = cr.metadata or {}
md = cr.markdown
- if md and hasattr(md, "raw_markdown"):
+ if md and hasattr(md, "fit_markdown"):
+ content = md.fit_markdown or md.raw_markdown or ""
+ elif md and hasattr(md, "raw_markdown"):
content = md.raw_markdown or ""
elif isinstance(md, str):
content = md