Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

docs: README v0.5.0 — architecture diagrams + subagent triad

Jordan34752c7ac86b
diff --git a/README.md b/README.md
index 7f60fd4..9fbdd71 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
<img src="assets/banner.png" alt="HYPERRESEARCH" width="700">
</p>
-<h3 align="center">Ultra deep, persistent web research for AI coding agents</h3>
+<h3 align="center">Deep, structured, adversarially-audited research for AI coding agents</h3>
<p align="center">
<a href="https://pypi.org/project/hyperresearch/"><img src="https://img.shields.io/pypi/v/hyperresearch" alt="PyPI version"></a>
@@ -13,50 +13,206 @@
---
-Your AI agent searches the web, finds great sources, synthesizes an answer — then the session ends and everything is gone. Next time, it starts from zero.
+Your AI agent searches the web, skims sources, writes an answer that sounds good, and ships it. There's no scaffold, no audit, no source-vs-source comparison, no provenance, no way to know if it actually engaged with the strongest counter-position. Next session, the work is gone.
-**Hyperresearch makes research persist.** Every source your agent finds is fetched with a real headless browser, saved as searchable markdown, and indexed into a knowledge base that compounds across sessions. Hyperresearch can follow maps of links to digest paths of information that current agent web searching isn't capable of.
+**Hyperresearch makes research disciplined and persistent.** Every fetched source becomes a markdown note in a Zettelkasten knowledge base. Every research session follows a 14-step protocol with four hard checkpoints. Two adversarial auditor subagents tear the draft apart against the user's verbatim prompt before save is allowed. The user's prompt is gospel — pasted into the scaffold's first section, re-read at every step, machine-checked at the gate.
```bash
pip install hyperresearch
hyperresearch install
```
-## What people use it for
+That's it. In Claude Code: type `/research <topic>`.
+
+---
+
+## The architecture
+
+### One dispatcher, four cognitive activities
+
+Hyperresearch routes every research request by what the output needs to **do**, not what the subject **is**. A query about a fictional franchise can be `collect` (per-character enumeration), `synthesize` (a thesis about meaning), `compare` (this work vs another), or `forecast` (will this sequel succeed). Subject doesn't decide; activity does.
+
+```mermaid
+graph LR
+ P[User Prompt] --> D{Dispatcher}
+ D -->|"what exists"| C[collect<br/>enumerative coverage]
+ D -->|"what it means"| S[synthesize<br/>defended thesis]
+ D -->|"which is best"| CMP[compare<br/>committed pick]
+ D -->|"what will happen"| F[forecast<br/>committed prediction]
+
+ C --> R[Final Report<br/>+ Audit Trail]
+ S --> R
+ CMP --> R
+ F --> R
+
+ style D fill:#1e293b,stroke:#475569,color:#fff
+ style C fill:#0f4c5c,stroke:#3b82f6,color:#fff
+ style S fill:#0f4c5c,stroke:#3b82f6,color:#fff
+ style CMP fill:#0f4c5c,stroke:#3b82f6,color:#fff
+ style F fill:#0f4c5c,stroke:#3b82f6,color:#fff
+ style R fill:#166534,stroke:#22c55e,color:#fff
+```
+
+Each modality file encodes only its substance differences (source strategy, writing rules, conformance checks). The shared 14-step protocol — discovery, fetch, curation, scaffold, comparisons, draft, audit, synthesis — lives in one dispatcher. Tight, factored, no duplication.
+
+### The context flow — verbatim prompt is gospel
+
+Every step re-encounters the user's original prompt. It gets pasted verbatim into the scaffold's first section, the auditor reads it at every audit, the save gate machine-checks it. No paraphrasing, no drift.
+
+```mermaid
+flowchart TB
+ P[📥 User Prompt] -->|verbatim, never paraphrased| CLS[1. Classify activity<br/>+ extract deliverables]
+ CLS --> SRC[2. Source discovery<br/>+ adversarial round]
+ SRC --> FET[3. Seed fetch<br/>3-5 high-signal URLs]
+ FET --> LOOP{4. Guided reading loop}
+ LOOP -->|analyst proposes targets| FET2[Fetch with --suggested-by<br/>builds rooted provenance graph]
+ FET2 --> LOOP
+ LOOP -->|coverage complete| CK1[✓ Checkpoint 1<br/>voice diversity + dissent]
+
+ CK1 --> CUR[5-6. Curate every source<br/>tier + content_type + analyst extract]
+ CUR --> CK2[✓ Checkpoint 2<br/>provenance rooted-tree<br/>analyst coverage]
+
+ CK2 --> SCF[7. Build scaffold<br/>📌 verbatim prompt as §1]
+ SCF --> CMP[8. Comparisons note<br/>3-5 source-vs-source disagreements]
+ CMP --> CK3[✓ Checkpoint 3<br/>scaffold-prompt lint<br/>artifacts present]
+
+ CK3 --> DRF[9. Write draft<br/>against the scaffold]
+ DRF --> GAP[10. Gap analysis<br/>+ adversarial search]
+ GAP --> AUD[11. Adversarial audit<br/>2 modes, persisted to JSON]
+ AUD --> CK4[✓ Checkpoint 4<br/>audit-gate self-cert detection]
+
+ CK4 --> SYN[12. Opinionated synthesis]
+ SYN --> SAVE[13-14. Save synthesis note<br/>blocked until audit-gate passes]
+
+ style P fill:#7c2d12,stroke:#ea580c,color:#fff
+ style SCF fill:#7c2d12,stroke:#ea580c,color:#fff
+ style AUD fill:#581c87,stroke:#a855f7,color:#fff
+ style CK1 fill:#166534,stroke:#22c55e,color:#fff
+ style CK2 fill:#166534,stroke:#22c55e,color:#fff
+ style CK3 fill:#166534,stroke:#22c55e,color:#fff
+ style CK4 fill:#166534,stroke:#22c55e,color:#fff
+ style SAVE fill:#166534,stroke:#22c55e,color:#fff
+```
+
+The four checkpoints are hard gates. **Checkpoint 3** runs `scaffold-prompt` lint that machine-checks the verbatim prompt is in the scaffold's first section. **Checkpoint 4** runs `audit-gate` lint that re-runs the underlying lint rules referenced by every CRITICAL audit finding — if a finding was marked "fixed" but the underlying rule still fails, the gate emits a `SELF-CERTIFICATION VIOLATION` and blocks save. The agent cannot ship a draft that grades its own homework.
+
+### The subagent triad
+
+Hyperresearch ships three registered Claude Code subagents, each picked for its job:
-- **In-depth topic research** — "Research the latest advances in state space models" → agent fetches 20+ papers, docs, and blog posts, follows citations to primary sources, builds a linked knowledge graph
-- **News tracking over time** — run research sessions weekly on a topic, each session builds on what's already collected, nothing gets re-fetched
-- **State-of-the-art surveys** — "What's the current SOTA for speech recognition?" → agent goes down rabbit holes, collects benchmarks, papers, and implementations
-- **Competitive analysis** — scrape company pages, LinkedIn profiles, product docs, news articles into a persistent, searchable corpus
-- **Due diligence** — aggregate everything about a person, company, or technology from across the web with authenticated crawling (LinkedIn, Twitter, paywalled sites)
+```mermaid
+graph TB
+ M[🤖 Main Agent<br/>your Claude Code session]
-## How it works
+ M -.spawns.-> F[🌐 hyperresearch-fetcher<br/><b>Haiku</b><br/>cheap URL fetching]
+ M -.spawns.-> A[📖 hyperresearch-analyst<br/><b>Sonnet</b><br/>guided reading + extraction]
+ M -.spawns.-> AU[🔍 hyperresearch-auditor<br/><b>Opus</b><br/>adversarial review]
-1. **You ask your agent to research something**
-2. **Agent searches the web** — multiple queries, different angles
-3. **Fetches every source** with a real headless browser (crawl4ai) — JS rendering, bot detection bypass, login-gated content
-4. **Saves each page** as a searchable markdown note with tags, summary, and source tracking
-5. **Follows links** to primary sources — the paper, not the blog post about the paper
-6. **Auto-links related notes** with `[[wiki-links]]` across the knowledge graph
-7. **Synthesizes findings** into a summary note linking all sources
-8. **Next session** — agent checks the KB before searching the web. Knowledge compounds.
+ F -->|fetched note| V[(Vault<br/>SQLite + markdown)]
+ A -->|extract note + next_targets| V
+ A -.proposes URLs.-> M
+ AU -->|findings JSON| V
+ AU -->|pass/fail| M
+ style M fill:#1e293b,stroke:#475569,color:#fff
+ style F fill:#0c4a6e,stroke:#0ea5e9,color:#fff
+ style A fill:#0e7490,stroke:#06b6d4,color:#fff
+ style AU fill:#581c87,stroke:#a855f7,color:#fff
+ style V fill:#166534,stroke:#22c55e,color:#fff
```
-your-repo/
- .hyperresearch/ # Config + SQLite FTS5 index (rebuildable)
- research/
- notes/ # Markdown notes — the source of truth
- index/ # Auto-generated wiki pages
- CLAUDE.md # Agent docs (auto-injected)
+
+| Agent | Model | Job | Why this model |
+|---|---|---|---|
+| **hyperresearch-fetcher** | Haiku | Fetches a URL via crawl4ai, persists the note. Mechanical, fast, parallel. | URL fetching is mechanical. Haiku is 10× cheaper. |
+| **hyperresearch-analyst** | Sonnet | Reads one source with the research goal in hand, writes a focused extract, proposes 2-5 next targets for the guided loop. | Reasoning about what's relevant + what to read next is real work. Sonnet hits the cost/quality sweet spot for hundreds of source reads per session. |
+| **hyperresearch-auditor** | Opus | Runs in two modes (`comprehensiveness` + `conformance`) against the verbatim prompt. Independently re-extracts entities, runs lint rules, persists findings to JSON. | Adversarial review over a long document against multiple criteria is critical reasoning. Worth Opus's ceiling. |
+
+The **guided reading loop** is the engine of deep research: seed-fetch 3-5 URLs → spawn an analyst on each → analysts propose `next_targets` → main agent fetches those WITH `--suggested-by` provenance → loop. Every fetched source carries a wiki-link breadcrumb back to the source that recommended it, so the research graph is a rooted, traceable tree. The provenance lint rule enforces this structurally.
+
+### The audit-gate self-certification detector
+
+The most important quality guarantee in v0.5.0. Two modes of the auditor run sequentially, each persisting findings to `research/audit_findings.json`. The agent applies fixes and marks each CRITICAL `fixed_at: <timestamp>`. The save gate then doesn't trust those marks blindly:
+
+```mermaid
+flowchart LR
+ A[Auditor finds<br/>CRITICAL: provenance broken] --> B[Agent applies fix]
+ B --> C[Agent marks<br/>fixed_at: timestamp]
+ C --> D{audit-gate lint}
+ D -->|extracts keyword<br/>'provenance'| E[Re-run<br/>provenance lint rule]
+ E -->|still has errors| F[❌ SELF-CERTIFICATION VIOLATION<br/>save blocked]
+ E -->|now clean| G[✓ Save proceeds]
+
+ style F fill:#7f1d1d,stroke:#ef4444,color:#fff
+ style G fill:#166534,stroke:#22c55e,color:#fff
+ style D fill:#581c87,stroke:#a855f7,color:#fff
+```
+
+Every CRITICAL with a `fixed_at` timestamp gets its underlying lint rule re-run in the same lint pass. If the rule still returns errors, the agent's "fix" was bookkeeping not substance — and the synthesis save is blocked until the actual vault state matches the claim. The 60+ keyword map covers `provenance`, `analyst-coverage`, `scaffold-prompt`, `workflow`, `uncurated` and natural-language variants. Findings that don't map to any rule emit an advisory warning ("not machine-verified") so the human reviewer sees which findings rest on agent self-report.
+
+---
+
+## What people use it for
+
+- **In-depth topic research** — "What does the literature say about ion-trap quantum scaling?" → 25+ academic sources, 8+ analyst extracts, dissenting voices represented, full provenance graph
+- **Comparative evaluation** — "TigerBeetle vs Postgres vs FoundationDB for write-heavy workloads" → per-entity coverage, comparison matrix, committed recommendation, mandatory critical voice on the market leader
+- **Forecasting + strategy** — "Will US inflation stay above 3% through 2026?" → ground-truth statistics + institutional analysis + named contrarians, probability language not hedge language, explicit time horizon
+- **Interpretive analysis** — "What does Blood Meridian's violence mean?" → primary text + critical tradition + dissenting reading, every paragraph fuses fact with interpretive claim
+- **Enumerative coverage** — "For each Napoleonic marshal, cover key campaigns and fate" → per-entity treatment with the 4 fields, no silent downgrades from "each" to "representative examples"
+- **Persistent knowledge base** — every source ever fetched stays searchable. Future sessions check the vault before searching the web. Knowledge compounds.
+
+---
+
+## What you get out of the box
+
+```bash
+pip install hyperresearch
+hyperresearch install
```
-## Works with every major agent
+That single sequence:
+
+1. Initializes a v6 SQLite vault with FTS5 search at `.hyperresearch/`
+2. Auto-installs Chromium for crawl4ai if missing
+3. Writes `CLAUDE.md` with the protocol overview
+4. Installs the `/research` skill (dispatcher + 4 modality files) to `.claude/skills/hyperresearch/`
+5. Registers all 3 subagents (fetcher, analyst, auditor) to `.claude/agents/`
+6. Wires PreToolUse hooks into `.claude/settings.json`
+7. Sets crawl4ai as the default web provider
+
+You're ready. Type `/research <anything>` in Claude Code.
+
+### What's enforced
+
+- **Verbatim prompt as gospel** — machine-checked via `scaffold-prompt` lint rule
+- **Rooted-tree provenance graph** — `--suggested-by` chain must form an actual tree from seeds; isolated breadcrumbs are flagged
+- **Analyst coverage** — at least 1 extract per 3 sources (no silent skipping)
+- **Adversarial dissent** — Checkpoint 1 requires at least one source explicitly dissents from the dominant view
+- **Audit gate self-cert detection** — CRITICAL findings marked fixed must actually be fixed; bookkeeping fixes get caught
+- **Save blocked** until audit-gate passes — synthesis cannot ship if any CRITICAL is unresolved
+- **Schema integrity** — `tier` and `content_type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index
+- **PDF + raw artifact persistence** — fetched PDFs land in `research/raw/` and the `raw_file` frontmatter field survives all re-serializations
+
+### Key features
+
+- **Real headless browser** — crawl4ai runs local Chromium. JS rendering, bot detection bypass, SPA support. Not an HTTP fetch.
+- **Native PDF extraction** — fetched PDFs go through pymupdf, full text extracted, raw file persisted
+- **Authenticated crawling** — log into LinkedIn / Twitter / paywalled news once via `hyperresearch setup`; the profile auto-applies on every fetch
+- **Junk detection** — captchas, cookie walls, login redirects, binary garbage, error pages all caught and rejected
+- **Fetch resilience** — when a high-priority source fails, the agent tries alternative URLs (preprint server, author page, institutional repo), then `--visible` browser, then a summary as fallback. Failures get documented; nothing silently disappears.
+- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP client
+- **Note lifecycle** — `draft → review → evergreen → stale → deprecated → archive`
+- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking
+- **FTS5 search** — instant full-text search with BM25 ranking across thousands of notes
+
+---
+
+## Hooks for every major agent
-`hyperresearch install` hooks into your agent in one step:
+`hyperresearch install` wires into your agent in one step:
| Platform | Hook | Trigger |
|----------|------|---------|
-| **Claude Code** | `.claude/settings.json` + `/research` skill | Before WebSearch, WebFetch |
+| **Claude Code** | `.claude/settings.json` + `/research` skill + 3 subagents | Before WebSearch, WebFetch |
| **Codex** | `.codex/hooks.json` | Before Bash |
| **Cursor** | `.cursor/rules/hyperresearch.mdc` | Always-apply rule |
| **Gemini CLI** | `.gemini/settings.json` | Before tool calls |
@@ -65,50 +221,42 @@ your-repo/
hyperresearch install --platform all # Hook every platform at once
```
-## Key features
-
-- **Real headless browser** — crawl4ai runs local Chromium. Handles JavaScript, bypasses bot detection, renders SPAs. Not a simple HTTP fetch.
-- **Native PDF extraction** — fetches PDFs directly, extracts full text with pymupdf, saves raw files to `research/raw/`. Academic papers, whitepapers, reports — all first-class citizens.
-- **Authenticated crawling** — log into LinkedIn, Twitter, paywalled news. Your sessions persist across fetches.
-- **Agent-driven curation** — fetcher subagents read each source, write real summaries, add meaningful tags, and flag junk. No keyword-matching shortcuts.
-- **Junk detection** — Cloudflare captchas, error pages, cookie walls, binary garbage, login redirects all caught and rejected before saving.
-- **Multi-round deep research** — the agent does multiple rounds of search → fetch → follow links, with a source checkpoint that forces breadth before synthesis.
-- **Gap analysis + adversarial audit** — after drafting, the agent re-reads the original query to find gaps, then spawns two auditor subagents (comprehensiveness + logic) to tear the draft apart. Runs twice.
-- **Cheap parallel fetching** — ships a Haiku-powered subagent that fetches, summarizes, and quality-checks URLs in parallel for pennies. Spawn 10-20 per round.
-- **Scholarly API guidance** — agent docs encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic topics.
-- **`/research` skill** — scripted deep research workflow. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, audits, synthesizes.
-- **Smart SPA wait** — polls DOM stability instead of fixed delays. Fast pages finish instantly, SPAs get up to 10 seconds.
-- **FTS5 search** — instant full-text search across thousands of notes with BM25 ranking
-- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking
-- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP client
-- **Note lifecycle** — draft → review → evergreen → stale → deprecated → archive
+---
## Commands
```bash
-# Research
-hyperresearch fetch <url> --tag t -j # Fetch a URL into the KB
-hyperresearch fetch-batch <urls...> -j # Fetch many URLs at once
-hyperresearch research "topic" --max 5 -j # Full pipeline: search → fetch → link → synthesize
-
-# Search
-hyperresearch search "query" -j # Full-text search
-hyperresearch search "query" --max-tokens 8000 # Stay within context budget
-hyperresearch note show <id> -j # Read a note
+# Research workflow
+hyperresearch fetch <url> --tag t -j # Fetch a URL into the KB
+hyperresearch fetch <url> --suggested-by <id> -j # Track provenance during the guided loop
+hyperresearch fetch <url> --visible -j # Bypass bot detection with a visible browser
+
+# Search + read
+hyperresearch search "query" -j # Full-text search
+hyperresearch search "query" --tier ground_truth -j # Filter by epistemic tier
+hyperresearch search "query" --content-type paper -j # Filter by artifact kind
+hyperresearch note show <id> -j # Read a note
+hyperresearch note show <id> --meta -j # Read frontmatter only (cheap triage)
+hyperresearch note list --status draft -j # List notes with summaries
# Knowledge graph
-hyperresearch link --auto -j # Auto-link related notes
-hyperresearch graph hubs -j # Most-connected notes
-hyperresearch graph backlinks <id> -j # What links to this note
-
-# Manage
-hyperresearch sources list -j # Every URL ever fetched
-hyperresearch lint -j # Health check
-hyperresearch repair -j # Fix links, rebuild indexes
+hyperresearch link --auto -j # Auto-link related notes
+hyperresearch graph hubs -j # Most-connected notes
+hyperresearch graph backlinks <id> -j # What links to this note
+
+# Health checks
+hyperresearch lint -j # Run all rules
+hyperresearch lint --rule scaffold-prompt -j # Verbatim prompt gospel rule
+hyperresearch lint --rule provenance -j # Rooted-tree breadcrumb chain
+hyperresearch lint --rule audit-gate -j # Self-certification detector
+hyperresearch lint --rule analyst-coverage -j # Extract:source ratio
+hyperresearch repair -j # Fix links, rebuild indexes
```
Every command returns `{"ok": true, "data": {...}}` with `-j`.
+---
+
## Authenticated crawling
Fetch from LinkedIn, Twitter, paywalled sites — anything you can log into:
@@ -126,23 +274,34 @@ profile = "research"
LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills.
+---
+
## Philosophy
-- **No LLM calls.** Hyperresearch stores, indexes, and searches. Your agent is the LLM.
-- **Markdown is truth.** Notes are plain files. SQLite is a rebuildable cache.
-- **Over-collect, then prune.** Fetch aggressively. Deprecate what you don't need.
-- **Check before you fetch.** Hooks kill redundant searches across sessions.
-- **Raw content is king.** Save the original with formatting, not a rewritten summary.
+- **Process is load-bearing.** A draft without a scaffold, comparisons, audit findings, and a clean provenance graph is unfinished — regardless of how good the prose reads.
+- **The user's prompt is the only authority.** Activity classification, source strategy, writing constraints all serve the prompt. Substance rules never override what the user actually asked for.
+- **No LLM calls inside the tool.** Hyperresearch stores, indexes, lints, and orchestrates. Your agent is the LLM.
+- **Markdown is truth, SQLite is cache.** Notes are plain files. Delete the DB; `hyperresearch sync` rebuilds it.
+- **Audit findings are artifacts, not just outputs.** They persist to JSON, get verified by lint rules, and gate the save. Self-certification is structurally prevented.
+- **Deeper-not-broader.** v0.5.0 prefers 15 well-extracted sources over 50 skim-fetched ones. The protocol enforces analyst coverage, not source count.
+
+---
## Requirements
- Python 3.11+
+- Claude Code (or Codex / Cursor / Gemini CLI) with API access
+- Anthropic API key with access to Opus, Sonnet, and Haiku (for the subagent triad)
- Windows, macOS, Linux
+---
+
## License
[MIT](LICENSE)
+---
+
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=jordan-gibbs/hyperresearch&type=Date)](https://star-history.com/#jordan-gibbs/hyperresearch&Date)