Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
docs: README v0.5.0 — architecture diagrams + subagent triad
Jordan34752c7ac86b
diff --git a/README.md b/README.mdindex 7f60fd4..9fbdd71 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@<img src="assets/banner.png" alt="HYPERRESEARCH" width="700"></p>-<h3 align="center">Ultra deep, persistent web research for AI coding agents</h3>+<h3 align="center">Deep, structured, adversarially-audited research for AI coding agents</h3><p align="center"><a href="https://pypi.org/project/hyperresearch/"><img src="https://img.shields.io/pypi/v/hyperresearch" alt="PyPI version"></a>@@ -13,50 +13,206 @@----Your AI agent searches the web, finds great sources, synthesizes an answer — then the session ends and everything is gone. Next time, it starts from zero.+Your AI agent searches the web, skims sources, writes an answer that sounds good, and ships it. There's no scaffold, no audit, no source-vs-source comparison, no provenance, no way to know if it actually engaged with the strongest counter-position. Next session, the work is gone.-**Hyperresearch makes research persist.** Every source your agent finds is fetched with a real headless browser, saved as searchable markdown, and indexed into a knowledge base that compounds across sessions. Hyperresearch can follow maps of links to digest paths of information that current agent web searching isn't capable of.+**Hyperresearch makes research disciplined and persistent.** Every fetched source becomes a markdown note in a Zettelkasten knowledge base. Every research session follows a 14-step protocol with four hard checkpoints. Two adversarial auditor subagents tear the draft apart against the user's verbatim prompt before save is allowed. The user's prompt is gospel — pasted into the scaffold's first section, re-read at every step, machine-checked at the gate.```bashpip install hyperresearchhyperresearch install```-## What people use it for+That's it. In Claude Code: type `/research <topic>`.++---++## The architecture++### One dispatcher, four cognitive activities++Hyperresearch routes every research request by what the output needs to **do**, not what the subject **is**. A query about a fictional franchise can be `collect` (per-character enumeration), `synthesize` (a thesis about meaning), `compare` (this work vs another), or `forecast` (will this sequel succeed). Subject doesn't decide; activity does.++```mermaid+graph LR+ P[User Prompt] --> D{Dispatcher}+ D -->|"what exists"| C[collect<br/>enumerative coverage]+ D -->|"what it means"| S[synthesize<br/>defended thesis]+ D -->|"which is best"| CMP[compare<br/>committed pick]+ D -->|"what will happen"| F[forecast<br/>committed prediction]++ C --> R[Final Report<br/>+ Audit Trail]+ S --> R+ CMP --> R+ F --> R++ style D fill:#1e293b,stroke:#475569,color:#fff+ style C fill:#0f4c5c,stroke:#3b82f6,color:#fff+ style S fill:#0f4c5c,stroke:#3b82f6,color:#fff+ style CMP fill:#0f4c5c,stroke:#3b82f6,color:#fff+ style F fill:#0f4c5c,stroke:#3b82f6,color:#fff+ style R fill:#166534,stroke:#22c55e,color:#fff+```++Each modality file encodes only its substance differences (source strategy, writing rules, conformance checks). The shared 14-step protocol — discovery, fetch, curation, scaffold, comparisons, draft, audit, synthesis — lives in one dispatcher. Tight, factored, no duplication.++### The context flow — verbatim prompt is gospel++Every step re-encounters the user's original prompt. It gets pasted verbatim into the scaffold's first section, the auditor reads it at every audit, the save gate machine-checks it. No paraphrasing, no drift.++```mermaid+flowchart TB+ P[📥 User Prompt] -->|verbatim, never paraphrased| CLS[1. Classify activity<br/>+ extract deliverables]+ CLS --> SRC[2. Source discovery<br/>+ adversarial round]+ SRC --> FET[3. Seed fetch<br/>3-5 high-signal URLs]+ FET --> LOOP{4. Guided reading loop}+ LOOP -->|analyst proposes targets| FET2[Fetch with --suggested-by<br/>builds rooted provenance graph]+ FET2 --> LOOP+ LOOP -->|coverage complete| CK1[✓ Checkpoint 1<br/>voice diversity + dissent]++ CK1 --> CUR[5-6. Curate every source<br/>tier + content_type + analyst extract]+ CUR --> CK2[✓ Checkpoint 2<br/>provenance rooted-tree<br/>analyst coverage]++ CK2 --> SCF[7. Build scaffold<br/>📌 verbatim prompt as §1]+ SCF --> CMP[8. Comparisons note<br/>3-5 source-vs-source disagreements]+ CMP --> CK3[✓ Checkpoint 3<br/>scaffold-prompt lint<br/>artifacts present]++ CK3 --> DRF[9. Write draft<br/>against the scaffold]+ DRF --> GAP[10. Gap analysis<br/>+ adversarial search]+ GAP --> AUD[11. Adversarial audit<br/>2 modes, persisted to JSON]+ AUD --> CK4[✓ Checkpoint 4<br/>audit-gate self-cert detection]++ CK4 --> SYN[12. Opinionated synthesis]+ SYN --> SAVE[13-14. Save synthesis note<br/>blocked until audit-gate passes]++ style P fill:#7c2d12,stroke:#ea580c,color:#fff+ style SCF fill:#7c2d12,stroke:#ea580c,color:#fff+ style AUD fill:#581c87,stroke:#a855f7,color:#fff+ style CK1 fill:#166534,stroke:#22c55e,color:#fff+ style CK2 fill:#166534,stroke:#22c55e,color:#fff+ style CK3 fill:#166534,stroke:#22c55e,color:#fff+ style CK4 fill:#166534,stroke:#22c55e,color:#fff+ style SAVE fill:#166534,stroke:#22c55e,color:#fff+```++The four checkpoints are hard gates. **Checkpoint 3** runs `scaffold-prompt` lint that machine-checks the verbatim prompt is in the scaffold's first section. **Checkpoint 4** runs `audit-gate` lint that re-runs the underlying lint rules referenced by every CRITICAL audit finding — if a finding was marked "fixed" but the underlying rule still fails, the gate emits a `SELF-CERTIFICATION VIOLATION` and blocks save. The agent cannot ship a draft that grades its own homework.++### The subagent triad++Hyperresearch ships three registered Claude Code subagents, each picked for its job:-- **In-depth topic research** — "Research the latest advances in state space models" → agent fetches 20+ papers, docs, and blog posts, follows citations to primary sources, builds a linked knowledge graph-- **News tracking over time** — run research sessions weekly on a topic, each session builds on what's already collected, nothing gets re-fetched-- **State-of-the-art surveys** — "What's the current SOTA for speech recognition?" → agent goes down rabbit holes, collects benchmarks, papers, and implementations-- **Competitive analysis** — scrape company pages, LinkedIn profiles, product docs, news articles into a persistent, searchable corpus-- **Due diligence** — aggregate everything about a person, company, or technology from across the web with authenticated crawling (LinkedIn, Twitter, paywalled sites)+```mermaid+graph TB+ M[🤖 Main Agent<br/>your Claude Code session]-## How it works+ M -.spawns.-> F[🌐 hyperresearch-fetcher<br/><b>Haiku</b><br/>cheap URL fetching]+ M -.spawns.-> A[📖 hyperresearch-analyst<br/><b>Sonnet</b><br/>guided reading + extraction]+ M -.spawns.-> AU[🔍 hyperresearch-auditor<br/><b>Opus</b><br/>adversarial review]-1. **You ask your agent to research something**-2. **Agent searches the web** — multiple queries, different angles-3. **Fetches every source** with a real headless browser (crawl4ai) — JS rendering, bot detection bypass, login-gated content-4. **Saves each page** as a searchable markdown note with tags, summary, and source tracking-5. **Follows links** to primary sources — the paper, not the blog post about the paper-6. **Auto-links related notes** with `[[wiki-links]]` across the knowledge graph-7. **Synthesizes findings** into a summary note linking all sources-8. **Next session** — agent checks the KB before searching the web. Knowledge compounds.+ F -->|fetched note| V[(Vault<br/>SQLite + markdown)]+ A -->|extract note + next_targets| V+ A -.proposes URLs.-> M+ AU -->|findings JSON| V+ AU -->|pass/fail| M+ style M fill:#1e293b,stroke:#475569,color:#fff+ style F fill:#0c4a6e,stroke:#0ea5e9,color:#fff+ style A fill:#0e7490,stroke:#06b6d4,color:#fff+ style AU fill:#581c87,stroke:#a855f7,color:#fff+ style V fill:#166534,stroke:#22c55e,color:#fff```-your-repo/- .hyperresearch/ # Config + SQLite FTS5 index (rebuildable)- research/- notes/ # Markdown notes — the source of truth- index/ # Auto-generated wiki pages- CLAUDE.md # Agent docs (auto-injected)++| Agent | Model | Job | Why this model |+|---|---|---|---|+| **hyperresearch-fetcher** | Haiku | Fetches a URL via crawl4ai, persists the note. Mechanical, fast, parallel. | URL fetching is mechanical. Haiku is 10× cheaper. |+| **hyperresearch-analyst** | Sonnet | Reads one source with the research goal in hand, writes a focused extract, proposes 2-5 next targets for the guided loop. | Reasoning about what's relevant + what to read next is real work. Sonnet hits the cost/quality sweet spot for hundreds of source reads per session. |+| **hyperresearch-auditor** | Opus | Runs in two modes (`comprehensiveness` + `conformance`) against the verbatim prompt. Independently re-extracts entities, runs lint rules, persists findings to JSON. | Adversarial review over a long document against multiple criteria is critical reasoning. Worth Opus's ceiling. |++The **guided reading loop** is the engine of deep research: seed-fetch 3-5 URLs → spawn an analyst on each → analysts propose `next_targets` → main agent fetches those WITH `--suggested-by` provenance → loop. Every fetched source carries a wiki-link breadcrumb back to the source that recommended it, so the research graph is a rooted, traceable tree. The provenance lint rule enforces this structurally.++### The audit-gate self-certification detector++The most important quality guarantee in v0.5.0. Two modes of the auditor run sequentially, each persisting findings to `research/audit_findings.json`. The agent applies fixes and marks each CRITICAL `fixed_at: <timestamp>`. The save gate then doesn't trust those marks blindly:++```mermaid+flowchart LR+ A[Auditor finds<br/>CRITICAL: provenance broken] --> B[Agent applies fix]+ B --> C[Agent marks<br/>fixed_at: timestamp]+ C --> D{audit-gate lint}+ D -->|extracts keyword<br/>'provenance'| E[Re-run<br/>provenance lint rule]+ E -->|still has errors| F[❌ SELF-CERTIFICATION VIOLATION<br/>save blocked]+ E -->|now clean| G[✓ Save proceeds]++ style F fill:#7f1d1d,stroke:#ef4444,color:#fff+ style G fill:#166534,stroke:#22c55e,color:#fff+ style D fill:#581c87,stroke:#a855f7,color:#fff+```++Every CRITICAL with a `fixed_at` timestamp gets its underlying lint rule re-run in the same lint pass. If the rule still returns errors, the agent's "fix" was bookkeeping not substance — and the synthesis save is blocked until the actual vault state matches the claim. The 60+ keyword map covers `provenance`, `analyst-coverage`, `scaffold-prompt`, `workflow`, `uncurated` and natural-language variants. Findings that don't map to any rule emit an advisory warning ("not machine-verified") so the human reviewer sees which findings rest on agent self-report.++---++## What people use it for++- **In-depth topic research** — "What does the literature say about ion-trap quantum scaling?" → 25+ academic sources, 8+ analyst extracts, dissenting voices represented, full provenance graph+- **Comparative evaluation** — "TigerBeetle vs Postgres vs FoundationDB for write-heavy workloads" → per-entity coverage, comparison matrix, committed recommendation, mandatory critical voice on the market leader+- **Forecasting + strategy** — "Will US inflation stay above 3% through 2026?" → ground-truth statistics + institutional analysis + named contrarians, probability language not hedge language, explicit time horizon+- **Interpretive analysis** — "What does Blood Meridian's violence mean?" → primary text + critical tradition + dissenting reading, every paragraph fuses fact with interpretive claim+- **Enumerative coverage** — "For each Napoleonic marshal, cover key campaigns and fate" → per-entity treatment with the 4 fields, no silent downgrades from "each" to "representative examples"+- **Persistent knowledge base** — every source ever fetched stays searchable. Future sessions check the vault before searching the web. Knowledge compounds.++---++## What you get out of the box++```bash+pip install hyperresearch+hyperresearch install```-## Works with every major agent+That single sequence:++1. Initializes a v6 SQLite vault with FTS5 search at `.hyperresearch/`+2. Auto-installs Chromium for crawl4ai if missing+3. Writes `CLAUDE.md` with the protocol overview+4. Installs the `/research` skill (dispatcher + 4 modality files) to `.claude/skills/hyperresearch/`+5. Registers all 3 subagents (fetcher, analyst, auditor) to `.claude/agents/`+6. Wires PreToolUse hooks into `.claude/settings.json`+7. Sets crawl4ai as the default web provider++You're ready. Type `/research <anything>` in Claude Code.++### What's enforced++- **Verbatim prompt as gospel** — machine-checked via `scaffold-prompt` lint rule+- **Rooted-tree provenance graph** — `--suggested-by` chain must form an actual tree from seeds; isolated breadcrumbs are flagged+- **Analyst coverage** — at least 1 extract per 3 sources (no silent skipping)+- **Adversarial dissent** — Checkpoint 1 requires at least one source explicitly dissents from the dominant view+- **Audit gate self-cert detection** — CRITICAL findings marked fixed must actually be fixed; bookkeeping fixes get caught+- **Save blocked** until audit-gate passes — synthesis cannot ship if any CRITICAL is unresolved+- **Schema integrity** — `tier` and `content_type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index+- **PDF + raw artifact persistence** — fetched PDFs land in `research/raw/` and the `raw_file` frontmatter field survives all re-serializations++### Key features++- **Real headless browser** — crawl4ai runs local Chromium. JS rendering, bot detection bypass, SPA support. Not an HTTP fetch.+- **Native PDF extraction** — fetched PDFs go through pymupdf, full text extracted, raw file persisted+- **Authenticated crawling** — log into LinkedIn / Twitter / paywalled news once via `hyperresearch setup`; the profile auto-applies on every fetch+- **Junk detection** — captchas, cookie walls, login redirects, binary garbage, error pages all caught and rejected+- **Fetch resilience** — when a high-priority source fails, the agent tries alternative URLs (preprint server, author page, institutional repo), then `--visible` browser, then a summary as fallback. Failures get documented; nothing silently disappears.+- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP client+- **Note lifecycle** — `draft → review → evergreen → stale → deprecated → archive`+- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking+- **FTS5 search** — instant full-text search with BM25 ranking across thousands of notes++---++## Hooks for every major agent-`hyperresearch install` hooks into your agent in one step:+`hyperresearch install` wires into your agent in one step:| Platform | Hook | Trigger ||----------|------|---------|-| **Claude Code** | `.claude/settings.json` + `/research` skill | Before WebSearch, WebFetch |+| **Claude Code** | `.claude/settings.json` + `/research` skill + 3 subagents | Before WebSearch, WebFetch || **Codex** | `.codex/hooks.json` | Before Bash || **Cursor** | `.cursor/rules/hyperresearch.mdc` | Always-apply rule || **Gemini CLI** | `.gemini/settings.json` | Before tool calls |@@ -65,50 +221,42 @@ your-repo/hyperresearch install --platform all # Hook every platform at once```-## Key features--- **Real headless browser** — crawl4ai runs local Chromium. Handles JavaScript, bypasses bot detection, renders SPAs. Not a simple HTTP fetch.-- **Native PDF extraction** — fetches PDFs directly, extracts full text with pymupdf, saves raw files to `research/raw/`. Academic papers, whitepapers, reports — all first-class citizens.-- **Authenticated crawling** — log into LinkedIn, Twitter, paywalled news. Your sessions persist across fetches.-- **Agent-driven curation** — fetcher subagents read each source, write real summaries, add meaningful tags, and flag junk. No keyword-matching shortcuts.-- **Junk detection** — Cloudflare captchas, error pages, cookie walls, binary garbage, login redirects all caught and rejected before saving.-- **Multi-round deep research** — the agent does multiple rounds of search → fetch → follow links, with a source checkpoint that forces breadth before synthesis.-- **Gap analysis + adversarial audit** — after drafting, the agent re-reads the original query to find gaps, then spawns two auditor subagents (comprehensiveness + logic) to tear the draft apart. Runs twice.-- **Cheap parallel fetching** — ships a Haiku-powered subagent that fetches, summarizes, and quality-checks URLs in parallel for pennies. Spawn 10-20 per round.-- **Scholarly API guidance** — agent docs encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic topics.-- **`/research` skill** — scripted deep research workflow. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, audits, synthesizes.-- **Smart SPA wait** — polls DOM stability instead of fixed delays. Fast pages finish instantly, SPAs get up to 10 seconds.-- **FTS5 search** — instant full-text search across thousands of notes with BM25 ranking-- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking-- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP client-- **Note lifecycle** — draft → review → evergreen → stale → deprecated → archive+---## Commands```bash-# Research-hyperresearch fetch <url> --tag t -j # Fetch a URL into the KB-hyperresearch fetch-batch <urls...> -j # Fetch many URLs at once-hyperresearch research "topic" --max 5 -j # Full pipeline: search → fetch → link → synthesize--# Search-hyperresearch search "query" -j # Full-text search-hyperresearch search "query" --max-tokens 8000 # Stay within context budget-hyperresearch note show <id> -j # Read a note+# Research workflow+hyperresearch fetch <url> --tag t -j # Fetch a URL into the KB+hyperresearch fetch <url> --suggested-by <id> -j # Track provenance during the guided loop+hyperresearch fetch <url> --visible -j # Bypass bot detection with a visible browser++# Search + read+hyperresearch search "query" -j # Full-text search+hyperresearch search "query" --tier ground_truth -j # Filter by epistemic tier+hyperresearch search "query" --content-type paper -j # Filter by artifact kind+hyperresearch note show <id> -j # Read a note+hyperresearch note show <id> --meta -j # Read frontmatter only (cheap triage)+hyperresearch note list --status draft -j # List notes with summaries# Knowledge graph-hyperresearch link --auto -j # Auto-link related notes-hyperresearch graph hubs -j # Most-connected notes-hyperresearch graph backlinks <id> -j # What links to this note--# Manage-hyperresearch sources list -j # Every URL ever fetched-hyperresearch lint -j # Health check-hyperresearch repair -j # Fix links, rebuild indexes+hyperresearch link --auto -j # Auto-link related notes+hyperresearch graph hubs -j # Most-connected notes+hyperresearch graph backlinks <id> -j # What links to this note++# Health checks+hyperresearch lint -j # Run all rules+hyperresearch lint --rule scaffold-prompt -j # Verbatim prompt gospel rule+hyperresearch lint --rule provenance -j # Rooted-tree breadcrumb chain+hyperresearch lint --rule audit-gate -j # Self-certification detector+hyperresearch lint --rule analyst-coverage -j # Extract:source ratio+hyperresearch repair -j # Fix links, rebuild indexes```Every command returns `{"ok": true, "data": {...}}` with `-j`.+---+## Authenticated crawlingFetch from LinkedIn, Twitter, paywalled sites — anything you can log into:@@ -126,23 +274,34 @@ profile = "research"LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills.+---+## Philosophy-- **No LLM calls.** Hyperresearch stores, indexes, and searches. Your agent is the LLM.-- **Markdown is truth.** Notes are plain files. SQLite is a rebuildable cache.-- **Over-collect, then prune.** Fetch aggressively. Deprecate what you don't need.-- **Check before you fetch.** Hooks kill redundant searches across sessions.-- **Raw content is king.** Save the original with formatting, not a rewritten summary.+- **Process is load-bearing.** A draft without a scaffold, comparisons, audit findings, and a clean provenance graph is unfinished — regardless of how good the prose reads.+- **The user's prompt is the only authority.** Activity classification, source strategy, writing constraints all serve the prompt. Substance rules never override what the user actually asked for.+- **No LLM calls inside the tool.** Hyperresearch stores, indexes, lints, and orchestrates. Your agent is the LLM.+- **Markdown is truth, SQLite is cache.** Notes are plain files. Delete the DB; `hyperresearch sync` rebuilds it.+- **Audit findings are artifacts, not just outputs.** They persist to JSON, get verified by lint rules, and gate the save. Self-certification is structurally prevented.+- **Deeper-not-broader.** v0.5.0 prefers 15 well-extracted sources over 50 skim-fetched ones. The protocol enforces analyst coverage, not source count.++---## Requirements- Python 3.11++- Claude Code (or Codex / Cursor / Gemini CLI) with API access+- Anthropic API key with access to Opus, Sonnet, and Haiku (for the subagent triad)- Windows, macOS, Linux+---+## License[MIT](LICENSE)+---+## Star History[](https://star-history.com/#jordan-gibbs/hyperresearch&Date)