Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
v0.3.0: PDF extraction, adversarial audit, agent-driven curation, junk detection
Jordanb0bcd0a9a0b3
diff --git a/.gitignore b/.gitignoreindex 9b8b149..a86442c 100644--- a/.gitignore+++ b/.gitignore@@ -11,3 +11,5 @@ build/.mypy_cache/.ruff_cache/*.egg+.env+bench/diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 64ed68f..2368487 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,35 @@# Changelog+## [0.3.0] - 2026-04-11++### New++- **Native PDF extraction** — PDFs detected by URL pattern, downloaded directly with httpx, text extracted with pymupdf. No browser needed. arXiv `/abs/` links auto-convert to `/pdf/`.+- **Raw file storage** — PDF bytes saved to `research/raw/<note-id>.pdf`, linked from note frontmatter via `raw_file:` field. Agent can read the raw PDF directly.+- **Junk page detection** — `WebResult.looks_like_junk()` catches Cloudflare captchas, error pages, cookie walls, binary garbage, reCAPTCHA, and empty content before saving. Returns `JUNK_CONTENT` error instead of creating useless notes.+- **Gap analysis step** — after drafting the report, agent re-reads the original query word by word, identifies gaps, and does another full round of research to fill them.+- **Adversarial audit** — two subagents (comprehensiveness auditor + logic/structure auditor) review the draft in parallel. Runs up to 2 loops. Agent uses wait time productively to improve summaries and tags.+- **Source checkpoint** — agent must review collected sources before writing any draft. Checks coverage breadth, missing angles, uncited references. Expects 50-100+ sources on complex topics.+- **Scholarly API guidance** — CLAUDE.md and `/research` skill now encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic research.+- **Date injection** — today's date injected programmatically into CLAUDE.md at install time.+- **Multi-round research emphasis** — agent docs stress multiple rounds of search → fetch → follow links, spawning 10-20 fetcher agents per round.++### Changed++- **Agent-driven curation replaces auto-enrich** — removed the keyword-matching `enrich_note_file()` from the fetch pipeline. Fetcher subagents now read content, write real summaries, add meaningful tags, and quality-check each source (deprecating junk/off-topic notes).+- **Fetcher subagent quality gate** — subagent now checks relevance, content quality, and duplicates. Deprecates bad notes instead of leaving them as drafts.+- **`_resolve_executable()` prioritizes venv** — checks venv `Scripts/` dir before PATH, preventing system-wide installs from overriding the project's venv.+- **PDF binary detection improved** — checks for `endstream`, `endobj`, `/FlateDecode`, `%PDF-` markers and non-printable character ratios. Catches binary garbage in both single-fetch and batch-fetch paths.+- **Junk detection thresholds raised** — empty content threshold: 100→300 chars, cookie page threshold: 500→1500 chars. Added `recaptcha`, `checking your browser`, `verify you are human` to bot detection signals.+- **SSL verification disabled for PDF downloads** — academic sites often have self-signed certs. `httpx.get(verify=False)` for PDF fetches only.+- **PDF fetch logging** — `_fetch_pdf` failures now logged via `logging.getLogger("hyperresearch.pdf")` instead of silently returning None.+- **Fetcher subagent continues on failure** — no longer stops on first fetch error, tries all URLs and reports failures individually.++### Added dependencies++- `pymupdf>=1.24` — PDF text extraction+- `httpx>=0.27` — direct HTTP downloads for PDFs (bypasses browser)+## [0.2.0] - 2026-04-10### New@@ -68,5 +98,6 @@ Initial release. Forked from [llm-kasten](https://github.com/jordan-gibbs/llm-ka- Web viewer with force-directed knowledge graph- 70 tests+[0.3.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.3.0[0.2.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.2.0[0.1.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.1.0diff --git a/PLAN.md b/PLAN.mddeleted file mode 100644index 2e2e046..0000000--- a/PLAN.md+++ /dev/null@@ -1,330 +0,0 @@-# Hyperresearch: Agent-Driven Research Knowledge Base--## Context--Kasten is a write-side knowledge base tool — agents/humans manually create notes. The gap in the ecosystem is a tool that lets an agent **actively collect, browse, and synthesize research** into a persistent, searchable knowledge base. Think: "kasten backbone + web browsing + slash-command UX like Graphify."--The user wants a new repo, new CLI name (`hyperresearch`), same core engine (markdown + YAML frontmatter + SQLite FTS5 + wiki-links + JSON output), but repositioned for **research workflows** where agents crawl sources, extract findings, and build a knowledge graph over time.--## What stays from kasten (the backbone)--These modules port over nearly verbatim:--| Module | What it does | Changes needed |-|--------|-------------|----------------|-| `core/vault.py` | Vault init, discovery, path properties | Rename dirs (`research/` instead of `knowledge/`?), rebrand |-| `core/sync.py` | mtime + SHA-256 sync, link resolution | None — same engine |-| `core/db.py` | SQLite schema, FTS5, WAL mode | Add `sources` table for URL tracking |-| `core/note.py` | read/write markdown with frontmatter | Add source-aware fields |-| `core/frontmatter.py` | YAML parse/serialize | None |-| `core/config.py` | TOML config | New defaults, new sections for web providers |-| `core/patterns.py` | Regex (wiki-links, code blocks) | None |-| `core/agent_docs.py` | Inject into CLAUDE.md etc | Rewrite blurb for research workflow |-| `models/` | Note, Envelope, SearchResult | Extend with Source model |-| `search/` | FTS5 + filters | Add source/URL filters |-| `serve/` | Web viewer | Rebrand theme |-| `mcp/` | MCP server | Add research-specific tools |--## What's new--### 1. Web provider plugin system--**Architecture:** Abstract `WebProvider` protocol, concrete implementations as optional deps.--```python-# src/hyperresearch/web/base.py-class WebProvider(Protocol):- def fetch(self, url: str) -> WebResult: ...- def search(self, query: str, max_results: int = 5) -> list[WebResult]: ...--@dataclass-class WebResult:- url: str- title: str- content: str # clean markdown- raw_html: str | None- fetched_at: datetime- metadata: dict # author, date, domain, etc.-```--**Provider implementations (each an optional dep group):**--| Provider | Install | Cost | Best for |-|----------|---------|------|----------|-| `crawl4ai` | `pip install hyperresearch[crawl4ai]` | Free, open-source | Default. 60k stars, agentic crawling, JS rendering |-| `tavily` | `pip install hyperresearch[tavily]` | $0.008/credit, 1k free/mo | Search-first workflows (API key required) |-| `firecrawl` | `pip install hyperresearch[firecrawl]` | Free self-hosted or paid API | Best markdown output, site-wide crawling |-| `trafilatura` | `pip install hyperresearch[trafilatura]` | Free | Lightweight text extraction, no JS |-| builtin | (included) | Free | `httpx` + `beautifulsoup4` fallback, no JS rendering |--**Config in `.hyperresearch/config.toml`:**-```toml-[web]-provider = "crawl4ai" # or "tavily", "firecrawl", "trafilatura", "builtin"-tavily_api_key = "" # or env var TAVILY_API_KEY-firecrawl_api_key = "" # or env var FIRECRAWL_API_KEY-```--### 2. New CLI commands (research-focused)--```bash-# Fetch a URL, extract content, save as a note-hyperresearch fetch "https://arxiv.org/abs/2401.12345" --tag transformers --tag attention--# Search the web, save top results as notes-hyperresearch websearch "state space models 2026" --max 5 --tag ssm--# Deep research: search → fetch → follow links → extract → save-hyperresearch research "mamba architecture" --depth 2 --max-pages 10--# All the kasten stuff still works-hyperresearch search "attention" -j-hyperresearch note show attention-is-all-you-need -j-hyperresearch repair -j-hyperresearch sync-```--**Command details:**--- **`fetch <url>`** — Single URL → note. Extracts clean markdown via provider, creates note with `source: <url>`, auto-tags if possible, records fetch metadata. Deduplicates by URL (warns if already fetched).--- **`websearch <query>`** — Uses provider's search API → creates one note per result. Each note links back to source URL. Returns JSON list of created note IDs.--- **`research <topic>`** — The flagship command. Agent-driven deep research:- 1. Web search for topic- 2. Fetch top N results- 3. Follow outbound links up to `--depth` levels- 4. Extract and save as notes with wiki-links between related findings- 5. Generate a synthesis note (map-of-content) linking everything found-- This is the command that makes it "fancy" — it's what agents invoke to build a knowledge base from scratch on a topic.--### 3. Slash-command / hook integration (like Graphify)--**This is the key adoption play.** Install hyperresearch, run one command, and every agent in the repo knows about it.--```bash-pip install hyperresearch-hyperresearch install # sets up hooks + agent docs-```--**`hyperresearch install` does:**--1. Creates `.hyperresearch/` directory (vault init)-2. Injects research-focused blurb into agent config files-3. **Installs PreToolUse hooks** for supported platforms:--| Platform | Hook location | Trigger | What it does |-|----------|--------------|---------|--------------|-| Claude Code | `.claude/settings.json` | Before `Glob`, `Grep`, `WebSearch`, `WebFetch` | "Check hyperresearch KB first. Run `hyperresearch search <topic> -j` before raw web searches." |-| Codex | `.codex/hooks.json` | Before `Bash` | Same reminder |-| Cursor | `.cursor/rules/hyperresearch.mdc` | `alwaysApply: true` | System rule with research workflow |-| Gemini CLI | `.gemini/settings.json` | `BeforeTool` | Same pattern |--The hook script (a small JS or Python file at `.hyperresearch/hook.js`):-- Checks if a research KB exists-- If yes, reminds the agent to `hyperresearch search` before going to the web-- Prevents redundant web fetches for topics already researched--### 4. Source tracking (new DB table)--```sql-CREATE TABLE sources (- url TEXT PRIMARY KEY,- note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,- domain TEXT,- fetched_at TEXT,- provider TEXT, -- which web provider fetched it- content_hash TEXT, -- detect if page changed- status TEXT DEFAULT 'active', -- active, dead, redirected- CHECK (status IN ('active', 'dead', 'redirected'))-);-CREATE INDEX idx_sources_domain ON sources(domain);-CREATE INDEX idx_sources_note ON sources(note_id);-```--This lets agents ask: "have we already fetched this URL?" and "what sources did we use for this topic?"--### 5. Extended frontmatter for research notes--```yaml-----title: "Mamba: Linear-Time Sequence Modeling"-id: mamba-linear-time-sequence-modeling-tags: [ssm, mamba, sequence-modeling]-status: review-source: https://arxiv.org/abs/2312.00752-source_domain: arxiv.org-fetched_at: 2026-04-09T12:00:00Z-fetch_provider: crawl4ai-confidence: 0.85-summary: "Mamba introduces selective state spaces that achieve transformer-quality with linear scaling."-----```--The `source`, `source_domain`, `fetched_at`, `fetch_provider` fields are new. The rest already exists in kasten's NoteMeta.--## Project structure--```-hyperresearch/-├── pyproject.toml-├── src/hyperresearch/-│ ├── __init__.py-│ ├── __main__.py-│ ├── cli/ # [from kasten] + new commands-│ │ ├── __init__.py # App assembly-│ │ ├── main.py # init, status, sync, install-│ │ ├── search.py # [from kasten]-│ │ ├── note.py # [from kasten]-│ │ ├── repair.py # [from kasten]-│ │ ├── fetch.py # NEW: fetch, websearch commands-│ │ ├── research.py # NEW: deep research command-│ │ ├── sources.py # NEW: source management-│ │ ├── graph.py # [from kasten]-│ │ ├── lint.py # [from kasten]-│ │ ├── tag.py # [from kasten]-│ │ └── ...-│ ├── core/ # [from kasten, nearly verbatim]-│ │ ├── vault.py-│ │ ├── sync.py-│ │ ├── db.py # + sources table-│ │ ├── note.py-│ │ ├── config.py # + web provider config-│ │ ├── agent_docs.py # rewritten blurb-│ │ ├── hooks.py # NEW: PreToolUse hook installer-│ │ └── ...-│ ├── web/ # NEW: web provider system-│ │ ├── __init__.py-│ │ ├── base.py # WebProvider protocol + WebResult-│ │ ├── builtin.py # httpx + bs4 fallback-│ │ ├── crawl4ai_provider.py-│ │ ├── tavily_provider.py-│ │ ├── firecrawl_provider.py-│ │ └── trafilatura_provider.py-│ ├── models/ # [from kasten] + Source model-│ ├── search/ # [from kasten]-│ ├── serve/ # [from kasten, rebranded]-│ ├── mcp/ # [from kasten] + research tools-│ └── indexgen/ # [from kasten]-├── tests/-├── README.md-├── CLAUDE.md-└── .github/workflows/-```--## Implementation phases--### Phase 1: Fork and rebrand — NOW-**Target:** `C:\Users\Jordan\PycharmProjects\hyperresearch` (PyCharm project already created)--1. Copy kasten's `src/kasten/` → `src/hyperresearch/` (all modules)-2. Copy `tests/`, `pyproject.toml`, `LICENSE`, `CONTRIBUTING.md`-3. Global rename in all files: `kasten` → `hyperresearch`, `kas` → `hpr` (alias)-4. Rename `knowledge/` → `research/` in vault defaults-5. Rename `.kasten/` → `.hyperresearch/` in vault discovery-6. Update pyproject.toml: name, entry points, description, add httpx + bs4 to core deps-7. Verify all tests pass with new names-8. `pip install -e ".[dev]"` works-9. `git init` + initial commit--### Phase 2: Web provider system (day 2-3)-1. Define `WebProvider` protocol in `web/base.py`-2. Implement `builtin` provider (httpx + bs4, always available)-3. Implement `crawl4ai` provider (optional dep)-4. Add `[web]` section to config.toml-5. Add `sources` table to DB schema-6. Wire up provider selection in config--### Phase 3: fetch + websearch commands (day 3-4)-1. `hyperresearch fetch <url>` — single URL to note-2. `hyperresearch websearch <query>` — search and save results-3. URL deduplication via sources table-4. Source metadata in frontmatter-5. Tests for both commands--### Phase 4: Hook system + install command (day 4-5)-1. `hyperresearch install` command-2. Claude Code PreToolUse hook (`.claude/settings.json`)-3. Cursor rule (`.cursor/rules/hyperresearch.mdc`)-4. Codex hook (`.codex/hooks.json`)-5. Gemini hook (`.gemini/settings.json`)-6. Rewrite agent docs blurb for research workflow-7. Hook reminder script (JS or Python)--### Phase 5: research command (day 5-7)-1. Orchestrates websearch → fetch → follow links → save-2. Depth-limited crawling-3. Auto wiki-linking between fetched notes-4. Synthesis note generation (map-of-content)-5. This is the flagship "demo" command--### Phase 6: Additional providers + polish (day 7-8)-1. Tavily provider-2. Firecrawl provider-3. Trafilatura provider-4. MCP server with research-specific tools (fetch, websearch, sources)-5. README, demo GIF, PyPI publish--## pyproject.toml sketch--```toml-[project]-name = "hyperresearch"-version = "0.1.0"-description = "Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki."-requires-python = ">=3.11"-dependencies = [- "typer>=0.9.0",- "rich>=13.0",- "pyyaml>=6.0",- "pydantic>=2.0",- "jinja2>=3.1",- "platformdirs>=4.0",-]--# httpx + bs4 NOT core — agents already have WebSearch/WebFetch--[project.optional-dependencies]-crawl4ai = ["Crawl4AI>=0.4"]-tavily = ["tavily-python>=0.5"]-firecrawl = ["firecrawl>=1.0"]-trafilatura = ["trafilatura>=2.0"]-mcp = ["mcp>=1.6"]-all = ["hyperresearch[crawl4ai,tavily,firecrawl,trafilatura,mcp]"]-dev = ["pytest", "pytest-cov", "ruff", "mypy"]--[project.scripts]-hyperresearch = "hyperresearch.cli:app"-hpr = "hyperresearch.cli:app"-```--## Key design decisions--1. **httpx + bs4 are core deps, not optional** — the builtin provider should always work without extras. This is the "it just works" baseline.--2. **crawl4ai is the recommended default** — free, open-source, 60k stars, handles JS rendering. But it's optional because it pulls in heavy deps (playwright/chromium).--3. **`install` not `init`** — Graphify uses `install` for the hook setup. We use `install` to do vault init + hook installation in one step. `init` still works as an alias for just the vault.--4. **Hooks remind, not enforce** — the PreToolUse hook says "check hyperresearch first" but doesn't block the agent. Blocking would be annoying.--5. **Notes are still files** — same as kasten philosophy. The web provider fetches content, but it gets saved as a .md file with frontmatter. Files are truth, DB is cache.--6. **No LLM calls in the tool itself** — hyperresearch doesn't call Claude/GPT to summarize or synthesize. The AGENT using hyperresearch does that. hyperresearch is still a "dumb tool" — it stores, indexes, and searches. The agent decides what to do with it.--7. **Agents already have web access** — Claude Code has WebSearch/WebFetch, Codex has browsing, etc. So the DEFAULT flow is: agent fetches content with its own tools → pipes it into hyperresearch via `--body-file` or `--body-stdin`. The optional web providers (crawl4ai, tavily, firecrawl) are extras for standalone/headless use, NOT required.--## Verification--1. `pip install -e ".[dev]"` — core install works-2. `hyperresearch install` — creates vault + hooks-3. `hyperresearch fetch "https://en.wikipedia.org/wiki/Zettelkasten"` — fetches and saves note-4. `hyperresearch search "zettelkasten" -j` — finds the fetched note-5. `hyperresearch websearch "state space models" --max 3` — searches and saves 3 notes-6. `python -m pytest tests/` — all tests pass-7. Open a Claude Code session in the repo — agent sees the hook reminder, uses `hyperresearch search` before web searches--## Name availability check needed-- PyPI: check if `hyperresearch` is taken-- GitHub: check if `hyperresearch` repo name is availablediff --git a/README.md b/README.mdindex 9dd41ec..7f60fd4 100644--- a/README.md+++ b/README.md@@ -68,12 +68,16 @@ hyperresearch install --platform all # Hook every platform at once## Key features- **Real headless browser** — crawl4ai runs local Chromium. Handles JavaScript, bypasses bot detection, renders SPAs. Not a simple HTTP fetch.+- **Native PDF extraction** — fetches PDFs directly, extracts full text with pymupdf, saves raw files to `research/raw/`. Academic papers, whitepapers, reports — all first-class citizens.- **Authenticated crawling** — log into LinkedIn, Twitter, paywalled news. Your sessions persist across fetches.-- **Auto-curation** — every fetched note gets auto-tagged, auto-summarized, and auto-linked to related notes+- **Agent-driven curation** — fetcher subagents read each source, write real summaries, add meaningful tags, and flag junk. No keyword-matching shortcuts.+- **Junk detection** — Cloudflare captchas, error pages, cookie walls, binary garbage, login redirects all caught and rejected before saving.+- **Multi-round deep research** — the agent does multiple rounds of search → fetch → follow links, with a source checkpoint that forces breadth before synthesis.+- **Gap analysis + adversarial audit** — after drafting, the agent re-reads the original query to find gaps, then spawns two auditor subagents (comprehensiveness + logic) to tear the draft apart. Runs twice.+- **Cheap parallel fetching** — ships a Haiku-powered subagent that fetches, summarizes, and quality-checks URLs in parallel for pennies. Spawn 10-20 per round.+- **Scholarly API guidance** — agent docs encourage use of arXiv, Semantic Scholar, CrossRef, and PubMed APIs for academic topics.+- **`/research` skill** — scripted deep research workflow. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, audits, synthesizes.- **Smart SPA wait** — polls DOM stability instead of fixed delays. Fast pages finish instantly, SPAs get up to 10 seconds.-- **Cheap parallel fetching** — ships a Haiku-powered subagent that fetches URLs in parallel for pennies-- **`/research` skill** — scripted deep research workflow. Clarifies ambiguous requests, searches broadly, fetches aggressively, follows rabbit holes, curates, synthesizes.-- **Login wall detection** — detects auth redirects and tells you to set up a profile instead of saving junk- **FTS5 search** — instant full-text search across thousands of notes with BM25 ranking- **Knowledge graph** — `[[wiki-links]]`, backlinks, hub detection, auto-linking- **MCP server** — 13 tools (read + write) for Claude Desktop, Cursor, or any MCP clientdiff --git a/pyproject.toml b/pyproject.tomlindex 2563928..5447b6a 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "hatchling.build"[project]name = "hyperresearch"-version = "0.2.0"+version = "0.3.0"description = "Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki."readme = "README.md"license = "MIT"@@ -29,6 +29,8 @@ dependencies = ["jinja2>=3.1","platformdirs>=4.0","Crawl4AI>=0.4",+ "pymupdf>=1.24",+ "httpx>=0.27",][project.optional-dependencies]diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.pyindex 54f427e..71f2821 100644--- a/src/hyperresearch/cli/fetch.py+++ b/src/hyperresearch/cli/fetch.py@@ -116,6 +116,16 @@ def fetch(console.print(f"[red]Auth required:[/] {msg}")raise typer.Exit(1)+ # Detect junk pages — captcha, error pages, binary garbage, empty content+ junk_reason = result.looks_like_junk()+ if junk_reason:+ msg = f"Skipped junk content from {url}: {junk_reason}"+ if json_output:+ output(error(msg, "JUNK_CONTENT"), json_mode=True)+ else:+ console.print(f"[yellow]Skipped:[/] {msg}")+ raise typer.Exit(1)+# Write notenote_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"domain = result.domain@@ -140,10 +150,40 @@ def fetch(extra_frontmatter=extra_meta,)- # Auto-enrich: add suggested tags and summary before sync- from hyperresearch.core.enrich import enrich_note_file-- enrich_note_file(note_path, conn, tags)+ # Save raw file (PDF, etc.) if present+ raw_file_path = None+ if result.raw_bytes and result.raw_content_type:+ ext_map = {+ "application/pdf": ".pdf",+ "image/png": ".png",+ "image/jpeg": ".jpg",+ "image/gif": ".gif",+ "image/webp": ".webp",+ }+ ext = ext_map.get(result.raw_content_type, "")+ if ext:+ raw_dir = vault.root / "research" / "raw"+ raw_dir.mkdir(parents=True, exist_ok=True)+ raw_filename = note_path.stem + ext+ raw_file = raw_dir / raw_filename+ raw_file.write_bytes(result.raw_bytes)+ raw_file_path = f"raw/{raw_filename}"++ # Note: tagging and summarization is the agent's job, not an automatic process.+ # The agent reads the fetched content and writes meaningful summaries and tags.++ # Add raw_file reference to frontmatter AFTER enrich (enrich rewrites frontmatter)+ if raw_file_path:+ note_text = note_path.read_text(encoding="utf-8")+ if note_text.startswith("---") and "raw_file:" not in note_text:+ end_idx = note_text.find("---", 3)+ if end_idx != -1:+ note_text = (+ note_text[:end_idx]+ + f"raw_file: {raw_file_path}\n"+ + note_text[end_idx:]+ )+ note_path.write_text(note_text, encoding="utf-8")# Sync first so the note exists in the notes table (needed for FK on sources/assets)note_id = note_path.stem@@ -175,6 +215,7 @@ def fetch("path": str(note_path.relative_to(vault.root)),"word_count": len(result.content.split()),"assets": saved_assets,+ "raw_file": raw_file_path,}if json_output:diff --git a/src/hyperresearch/core/agent_docs.py b/src/hyperresearch/core/agent_docs.pyindex b07d96d..f760f7d 100644--- a/src/hyperresearch/core/agent_docs.py+++ b/src/hyperresearch/core/agent_docs.py@@ -16,7 +16,7 @@ HYPERRESEARCH_SECTION_END = "<!-- hyperresearch:end -->"HYPERRESEARCH_BLURB = """\{marker}-## Research Base (hyperresearch)+## Research Base (hyperresearch) — Today is {today}**CLI path: `{hpr}`** — use this exact path for all hyperresearch commands below. It may not be on your system PATH.@@ -28,6 +28,27 @@ When the user asks you to research a topic, **you MUST follow this workflow**. D**This is a deep research tool.** Your job is to go down rabbit holes, cast a wide net, and collect as much raw source material as possible. **Over-collect, then prune.** It is always better to have too many sources than too few — you can deprecate notes later, but you can't un-skip a source you never fetched.+**There is no time limit.** Deep research can run for hours if the topic demands it. Do multiple rounds of research — as new subtopics, angles, and questions reveal themselves, launch new rounds of fetcher subagents in parallel to chase them down. The first round uncovers the landscape; the second round digs into what you found; the third round fills gaps. Keep going.++**CHECKPOINT: Before you start writing ANY draft, stop and take stock.** Run:+```bash+{hpr} note list -j+```+Don't collect sources just to hit a number. Instead, review what you have and ask:+- What angles, subtopics, or perspectives are NOT yet covered?+- Which existing notes reference papers, data, or sources I haven't fetched yet?+- Are there counterarguments, alternative viewpoints, or competing theories I'm missing?+- Do I have primary sources, or just secondhand summaries?++If the answer to any of these is "yes, there's more to get" — go back and get it. Spawn more fetcher agents, search new angles, follow links from existing notes. Keep going until you genuinely can't find anything new worth adding.++Quality over quantity — but on complex topics, true comprehensiveness requires quantity. If you've only collected 15-25 sources on a PhD-level question, you've barely scratched the surface. Expect to need:+- Simple factual question: 20-30 sources+- Complex or technical topic: 50-80 sources+- PhD-level research question: 80-100+ sources++If you've exhausted every angle and there's genuinely nothing left to find, that's fine — move on. But be honest with yourself about whether you've actually exhausted it or just stopped early.+**Step 1: Check what's already known**```bash{hpr} search "topic" --include-body -j@@ -45,7 +66,7 @@ Use the hyperresearch-fetcher agent to fetch these URLs with tag "<topic>":- https://example.com/article2```-**Spawn multiple fetcher agents in parallel** — one per URL or small batch. They run on Haiku, so this is fast and cheap. Err on the side of fetching too many pages.+**Spawn 10-20 fetcher agents in parallel per round** — give each agent 2-3 URLs max. They run on Haiku and cost fractions of a cent each. After each web search, immediately spawn fetchers for ALL promising URLs from the results — don't cherry-pick, fetch everything. Then do another round of searches and another round of fetchers. Repeat until you hit the source checkpoint.If the `hyperresearch-fetcher` agent is not available (other platforms), fall back to running the command directly:```bash@@ -64,15 +85,56 @@ Look for links in the content that point to **primary sources, references, relat- A paper cites 3 key references? Fetch all 3.- A profile mentions a company? Fetch the company page.- Found a related topic that might be relevant? Fetch it. You can always deprecate it later.+- A page links to raw data, CSV files, or datasets? Fetch those and analyze them.+- A source references specific statistics without citing them? Search for and fetch the primary data.++**Follow links deeply.** Don't stop at the first page. When a fetched page links to more detailed sources, primary research, raw data, or technical documentation — follow those links and fetch them too. The best insights come from primary sources, not secondhand summaries.++**Run analysis when needed.** If you find raw data (tables, CSVs, statistics), don't just quote numbers — write and run code to compute real figures, verify claims, calculate trends, and produce original analysis. Concrete numbers from actual data are worth more than vague summaries.**The goal is exhaustive collection.** You are building a knowledge base that future agents will rely on. Missing a source is worse than having one extra note. Prune later during curation.+**Do NOT stop collecting until you are certain you have enough breadth.** After each round of fetching, ask yourself: "Are there major angles, subtopics, or perspectives I haven't covered yet?" If yes, search and fetch more. If a topic has 5 facets, you need sources on all 5 — not just the first 2 you found. Keep going until you can confidently say the collection covers the full scope of the topic. A half-covered topic is worse than no coverage at all.+**Step 5: Synthesize** — Once you have the raw sources indexed, search and read them to build your synthesis:```bash{hpr} search "topic" --include-body -j```-**Step 6: Prune** — After synthesis, deprecate notes that turned out to be irrelevant:+Write a draft report with inline URL citations.++**Step 6: Gap analysis (MANDATORY)** — After writing your draft, stop and compare it against the original query. Re-read the user's request word by word. Ask yourself:+- What specific questions did they ask that I haven't fully answered?+- What subtopics or angles did I miss entirely?+- Where did I make claims without strong enough evidence?+- What would an expert in this field say is missing?++Then launch a new round of research to fill every gap — web search, fetch, follow links, the full workflow. Spawn fetcher subagents in parallel for the new URLs. Add the new material to your draft.++**Step 7: Adversarial audit (MANDATORY — do this twice)** — After the gap-filling round, launch two subagents in parallel to audit the revised draft. This runs up to 2 loops.++Spawn two agents in parallel:+- **Agent 1 — Comprehensiveness auditor:** "Read this report and search the research base (`{hpr} search` and `{hpr} note show`) to find gaps. What subtopics, angles, data points, or counterarguments are missing? What claims lack citations? What sections are shallow? Compare against the original query and be ruthless — list every gap."+- **Agent 2 — Logic and structure auditor:** "Read this report as a domain expert. Does the argument flow logically? Are conclusions supported by the evidence presented? Are there logical leaps, unsupported claims, or contradictions? Is the structure optimal for the topic? Be specific — list every weakness."++Pass your full draft report text to both agents. They search the research base independently and report back.++**After each audit round:**+1. Read both agents' feedback+2. If they identified missing topics or weak arguments — do another round of web search and fetching to fill the gaps+3. Rewrite the report incorporating new sources and fixing structural issues+4. Run the audit again (second loop) on the revised report++**You MUST do at least one audit loop. Do two if the first audit found significant gaps.** Only after the auditors have no major complaints should you finalize the report.++**While audit agents are running, don't idle.** Use the wait time productively:+- Improve summaries on notes that have weak or missing summaries+- Add more specific tags to notes that only have one generic tag+- Add `[[wiki-links]]` between related notes to build the knowledge graph+- Run `{hpr} lint -j` and fix any issues it finds+- Read notes you haven't read yet and look for links worth following in future rounds++**Step 8: Prune** — After the final report is done, deprecate notes that turned out to be irrelevant:```bash{hpr} note update <id> --status deprecated -j```@@ -82,6 +144,27 @@ This keeps the KB clean without losing anything permanently.`{hpr} fetch` runs a real headless Chromium browser — it bypasses bot detection, saves full content with formatting, persists across sessions, and tracks URLs to prevent re-fetching. **Use WebFetch only for quick one-off lookups you don't need to save.**+### PDFs are fully supported — fetch them directly++`{hpr} fetch` automatically detects PDF URLs and extracts full text using pymupdf — no browser needed. **Fetch PDFs aggressively:**+- arXiv papers (both `/abs/` and `/pdf/` links work — auto-converted)+- NBER working papers, SSRN papers, direct `.pdf` links+- Conference proceedings, technical reports, whitepapers++PDF extraction is fast and produces clean text from all pages. **Do not skip a source just because it's a PDF.** If a paper is behind a paywall, look for preprint versions on arXiv, SSRN, ResearchGate, or the author's personal page.++Raw files are automatically saved to `research/raw/<note-id>.pdf` and linked from the note's frontmatter (`raw_file: raw/<note-id>.pdf`). You can read the raw PDF directly if you need to verify content or extract figures.++### Use scholarly APIs when available++For academic research, use APIs to find and access papers programmatically:+- **arXiv API** — `https://export.arxiv.org/api/query?search_query=...` returns XML with abstracts, authors, PDF links+- **Semantic Scholar API** — `https://api.semanticscholar.org/graph/v1/paper/search?query=...` returns structured paper data with citations+- **CrossRef API** — `https://api.crossref.org/works?query=...` for DOI lookups and metadata+- **PubMed/NCBI API** — for biomedical research++Write code to call these APIs when the topic is academic. The structured data (citation counts, related papers, abstracts) is often more useful than scraping a webpage. Fetch the actual papers via their PDF links after finding them through the API.+### Raw content is king**Save raw source material with original formatting** — not rewritten summaries. Future agents need the full context to draw their own conclusions. Preserve headings, tables, code blocks, and technical details. Follow links to primary sources (the paper, not the blog post about the paper). Connect related notes with `[[note-id]]` wiki-links.@@ -149,16 +232,18 @@ to recreate the profile, or `playwright install chromium` to reinstall the browsThe research base is a **long-lived knowledge investment**, not a scratchpad. Every future agent session benefits from well-organized notes. **After fetching sources, you MUST do a curation pass.**-**Step 1: Tag and summarize every new note**-Fetched notes arrive as drafts with no summary. Fix each one:+**Step 1: Read and summarize every new note**+Fetched notes arrive as drafts with no summary. **You must read each note and write a real summary** — not a generic label but a specific description of what the source contains:```bash{hpr} note list --status draft -j # Find unprocessed notes-{hpr} note update <id> --summary "One-line summary" -j # Add summary-{hpr} note update <id> --add-tag <topic> -j # Add tags+{hpr} note show <id> -j # Read the note content+{hpr} note update <id> --summary "One-line summary" -j # Add YOUR summary after reading it+{hpr} note update <id> --add-tag <topic> -j # Add meaningful tags```-- REUSE existing tags: `{hpr} tags -j` — do not invent new ones unless truly novel+- **Read the content first**, then write a summary that captures what's actually in it. "Maskin & Riley prove existence and uniqueness of equilibrium in asymmetric first-price auctions via ODE system" — not "Paper about auctions"+- REUSE existing tags: `{hpr} tags -j` — do not invent new ones unless truly novel, but add multiple relevant tags per note- Every note MUST have: at least one tag, a summary (under 120 chars), a source URL-- Summaries should be specific and searchable, not generic ("Overview of X" is useless; "X achieves Y by doing Z" is useful)+- For PDF notes, check the `raw_file` field in the frontmatter — it points to the actual PDF in `research/raw/`. You can read it directly if the extracted text is incomplete**Step 2: Connect related notes with wiki-links**Read through new notes and add `[[other-note-id]]` links to connect related material:@@ -224,16 +309,15 @@ AGENT_FILES = {def _resolve_executable() -> str:- """Find the absolute path to the hyperresearch executable."""+ """Find the absolute path to the hyperresearch executable.++ Priority: venv sibling of current python > PATH > bare name.+ """import shutilimport sys- # Check if it's on PATH already- which = shutil.which("hyperresearch")- if which:- return which-- # Find it relative to the current Python interpreter (works for venv installs)+ # First: find it relative to the current Python interpreter (venv installs).+ # This takes priority over PATH to avoid picking up a system-wide install.python_dir = Path(sys.executable).parentfor name in ("hyperresearch", "hyperresearch.exe"):candidate = python_dir / name@@ -245,6 +329,11 @@ def _resolve_executable() -> str:if candidate.exists():return str(candidate)+ # Second: check PATH+ which = shutil.which("hyperresearch")+ if which:+ return which+# Fallback — bare name, hope it's on PATHreturn "hyperresearch"@@ -267,10 +356,12 @@ def inject_agent_docs(hpr_path = _resolve_executable()# Use forward slashes — bash on Windows eats backslasheshpr_path = hpr_path.replace("\\", "/")+ from datetime import dateblurb = HYPERRESEARCH_BLURB.format(marker=HYPERRESEARCH_SECTION_MARKER,end_marker=HYPERRESEARCH_SECTION_END,hpr=hpr_path,+ today=date.today().isoformat(),)modified = []diff --git a/src/hyperresearch/core/fetcher.py b/src/hyperresearch/core/fetcher.pyindex 4b7994a..725bbdc 100644--- a/src/hyperresearch/core/fetcher.py+++ b/src/hyperresearch/core/fetcher.py@@ -22,7 +22,6 @@ def fetch_and_save(ValueError: If URL is already fetched.RuntimeError: If fetch fails."""- from hyperresearch.core.enrich import enrich_note_filefrom hyperresearch.core.note import write_notefrom hyperresearch.core.sync import compute_sync_plan, execute_syncfrom hyperresearch.web.base import get_provider@@ -65,6 +64,11 @@ def fetch_and_save("Run 'hyperresearch setup' and create a new login profile.")+ # Detect junk pages — captcha, error pages, binary garbage, empty content+ junk_reason = result.looks_like_junk()+ if junk_reason:+ raise RuntimeError(f"Skipped junk content: {junk_reason}")+# Write notenote_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"domain = result.domain@@ -89,8 +93,39 @@ def fetch_and_save(extra_frontmatter=extra_meta,)- # Auto-enrich- enrich_note_file(note_path, conn, tags)+ # Save raw file (PDF, etc.) if present+ raw_file_path = None+ if result.raw_bytes and result.raw_content_type:+ ext_map = {+ "application/pdf": ".pdf",+ "image/png": ".png",+ "image/jpeg": ".jpg",+ "image/gif": ".gif",+ "image/webp": ".webp",+ }+ ext = ext_map.get(result.raw_content_type, "")+ if ext:+ raw_dir = vault.root / "research" / "raw"+ raw_dir.mkdir(parents=True, exist_ok=True)+ raw_filename = note_path.stem + ext+ raw_file = raw_dir / raw_filename+ raw_file.write_bytes(result.raw_bytes)+ raw_file_path = f"raw/{raw_filename}"++ # Note: tagging and summarization is the agent's job, not an automatic process.++ # Add raw_file reference to frontmatter AFTER enrich (enrich rewrites frontmatter)+ if raw_file_path:+ note_text = note_path.read_text(encoding="utf-8")+ if note_text.startswith("---") and "raw_file:" not in note_text:+ end = note_text.find("---", 3)+ if end != -1:+ note_text = (+ note_text[:end]+ + f"raw_file: {raw_file_path}\n"+ + note_text[end:]+ )+ note_path.write_text(note_text, encoding="utf-8")# Syncnote_id = note_path.stem@@ -124,4 +159,5 @@ def fetch_and_save("path": str(note_path.relative_to(vault.root)),"word_count": len(result.content.split()),"assets": saved_assets,+ "raw_file": raw_file_path,}diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.pyindex 240de2c..2927244 100644--- a/src/hyperresearch/core/hooks.py+++ b/src/hyperresearch/core/hooks.py@@ -57,16 +57,25 @@ For each URL you are given:2. If not already fetched, fetch it:PYTHONIOENCODING=utf-8 {hpr_path} fetch "<url>" --tag <topic> -j-3. After fetching, read the note and look for links worth following:+3. After fetching, read the note content:PYTHONIOENCODING=utf-8 {hpr_path} note show <note-id> -j-4. Report back: the note ID, title, word count, AND a list of links found- in the content that look like they lead to primary sources, references,- related material, or deeper content. Be aggressive — list anything that- might be worth fetching. The parent agent will decide what to pursue.+4. **Quality check** — read the content and decide:+ - Is this actually relevant to the research topic? If it's completely off-topic, deprecate it:+ PYTHONIOENCODING=utf-8 {hpr_path} note update <note-id> --status deprecated -j+ - Is the content meaningful (not binary garbage, not a cookie page, not just nav elements)?+ If it's junk, deprecate it and report "junk content" to the parent agent.+ - Is this a duplicate of another source? If so, deprecate the worse copy.-If the first fetch fails with a browser error, stop and report the error.-Do not attempt remaining URLs.+5. If the content is good, write a real summary and add tags:+ PYTHONIOENCODING=utf-8 {hpr_path} note update <note-id> --summary "<specific summary>" -j+ PYTHONIOENCODING=utf-8 {hpr_path} note update <note-id> --add-tag <specific-tag> -j++ The summary must be specific — "Proves existence/uniqueness of equilibrium in asymmetric first-price auctions via coupled ODE system" NOT "Paper about auctions". Add multiple relevant tags based on the actual content.++6. Report back: the note ID, title, word count, your summary, quality verdict (good/junk/off-topic), AND a list of links found in the content that look like they lead to primary sources, references, related material, or deeper content. The parent agent will decide what to pursue.++If a fetch fails (JUNK_CONTENT, FETCH_ERROR, AUTH_REQUIRED), report the failure and move on to the next URL. Do NOT stop on first failure — try all URLs.If given multiple URLs and fetching works, process them sequentially. Report results for each.diff --git a/src/hyperresearch/skills/research.md b/src/hyperresearch/skills/research.mdindex 4dfa6e0..72ada47 100644--- a/src/hyperresearch/skills/research.md+++ b/src/hyperresearch/skills/research.md@@ -67,6 +67,20 @@ Each fetcher will report back with note IDs and interesting links found in the c**Over-collect.** It is always better to have too many sources than too few. You can deprecate notes later.+**There is no time limit.** Do multiple rounds of fetching — as new subtopics and questions reveal themselves, launch new rounds of fetcher subagents in parallel to chase them down. The first round uncovers the landscape, the second digs into what you found, the third fills gaps. Keep going until you've exhausted the topic. Do not rush.++**CHECKPOINT: Before writing any draft, stop and take stock:**+```bash+$HPR note list -j+```+Review what you have. Don't chase a number — chase completeness:+- What angles or subtopics are NOT yet covered?+- Which notes reference sources you haven't fetched?+- Are there counterarguments or alternative viewpoints missing?+- Do you have primary sources or just secondhand summaries?++If there's more to get, go get it. If you've genuinely exhausted the topic, move on. But be honest — on complex topics, expect 50-80+ sources before you've truly covered it.+## Step 4: Go down rabbit holesRead the fetched notes and follow links to deeper material:@@ -76,13 +90,30 @@ $HPR note show <note-id> -j```For each note, look for:-- **Papers cited** → fetch them+- **Papers cited** → fetch them (PDFs are fully supported — fetch directly)- **Documentation linked** → fetch it- **GitHub repos mentioned** → fetch the README- **Related topics referenced** → search and fetch- **People/companies mentioned** → fetch their pages+- **Raw data, CSVs, datasets linked** → fetch and analyze them+- **Statistics without primary sources** → search for and fetch the original data++**PDFs work natively.** arXiv papers, NBER working papers, conference proceedings — fetch them directly. The tool auto-detects PDFs and extracts full text. Don't skip sources because they're PDFs.++**Use scholarly APIs for academic topics.** When the research is academic, use APIs to find papers:+- arXiv API: `https://export.arxiv.org/api/query?search_query=...`+- Semantic Scholar API: `https://api.semanticscholar.org/graph/v1/paper/search?query=...`+- CrossRef API: `https://api.crossref.org/works?query=...`-Spawn more fetcher agents for these secondary sources. Keep going until you've exhausted the link trail or have 20+ sources.+Write code to call these, get structured paper data (citations, related work, abstracts), then fetch the actual PDFs.++Follow links deeply — don't stop at the first page. When a source links to more detailed material, primary research, or raw data, follow those links and fetch them too.++**Run analysis when needed.** If you find raw data (tables, CSVs, statistics), write and run code to compute real figures, verify claims, calculate trends, and produce original analysis. Concrete numbers from actual data beat vague summaries.++Spawn more fetcher agents for these secondary sources.++**Do NOT stop collecting until you are certain you have enough breadth.** After each round of fetching, ask yourself: "Are there major angles, subtopics, or perspectives I haven't covered yet?" If yes, search and fetch more. If a topic has 5 facets, you need sources on all 5 — not just the first 2 you found. Keep going until you can confidently say the collection covers the full scope of the topic.## Step 5: Auto-link and curate@@ -114,7 +145,7 @@ $HPR graph hubs -j$HPR graph backlinks <most-linked-note-id> -j```-## Step 7: Synthesize and report+## Step 7: Synthesize draft reportSearch the full research base with bodies:@@ -122,9 +153,58 @@ Search the full research base with bodies:$HPR search "<topic>" --include-body -j```-Read the top notes and build a comprehensive synthesis. Then **save the synthesis as a note**:+Read the top notes and write a comprehensive draft report with inline URL citations.++## Step 8: Gap analysis (MANDATORY)++After writing your draft, stop and compare it against the original query. Re-read the user's request word by word:+- What specific questions did they ask that I haven't fully answered?+- What subtopics or angles did I miss entirely?+- Where did I make claims without strong enough evidence?+- What would an expert in this field say is missing?++Launch a new round of research to fill every gap — web search, fetch, follow links, spawn fetcher subagents in parallel. Add the new material to your draft.++## Step 9: Adversarial audit (MANDATORY — do this twice)++After the gap-filling round, launch two subagents in parallel to audit the revised draft:++**Agent 1 — Comprehensiveness auditor:**+```+Read this report and search the research base ($HPR search, $HPR note show) to find gaps.+What subtopics, angles, data points, or counterarguments are missing? What claims lack+citations? What sections are shallow? Compare against the original query and be ruthless —+list every gap you find.+```++**Agent 2 — Logic and structure auditor:**+```+Read this report as a domain expert. Does the argument flow logically? Are conclusions+supported by the evidence presented? Are there logical leaps, unsupported claims, or+contradictions? Is the structure optimal for the topic? Be specific — list every weakness.+```++Pass your full draft report text to both agents.++**After each audit round:**+1. Read both agents' feedback+2. If they found missing topics or weak arguments — do another round of web search + fetching+3. Rewrite the report with new sources and structural fixes+4. Run the audit again (second loop) on the revised report++**Do at least one audit loop. Do two if the first found significant gaps.** Only finalize after auditors have no major complaints.++**While audit agents are running, don't idle.** Use the wait time to:+- Improve weak or missing summaries on notes (`$HPR note update <id> --summary "..." -j`)+- Add more specific tags to under-tagged notes (`$HPR note update <id> --add-tag <tag> -j`)+- Add `[[wiki-links]]` between related notes+- Run `$HPR lint -j` and fix issues+- Read notes you haven't reviewed yet for links worth following++## Step 10: Save the final report++Save the final reviewed report as a synthesis note:-Write your synthesis to a temp file and save it:```bashcat > /tmp/synthesis.md << 'SYNTHESIS'# <Topic> — Research Synthesis@@ -142,7 +222,7 @@ SYNTHESIS$HPR note new "Synthesis: <topic>" --tag <topic> --tag synthesis --type moc --status review --body-file /tmp/synthesis.md --summary "<one-line summary of findings>" -j```-## Step 8: Present to user+## Step 11: Present to userPresent your findings with:diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.pyindex ee2afbe..32d7c84 100644--- a/src/hyperresearch/web/base.py+++ b/src/hyperresearch/web/base.py@@ -20,6 +20,8 @@ class WebResult:media: list[dict] = field(default_factory=list) # images: {src, alt, score, ...}links: list[dict] = field(default_factory=list) # {href, text, type}screenshot: bytes | None = None # PNG screenshot of the rendered page+ raw_bytes: bytes | None = None # Raw file bytes (PDF, etc.)+ raw_content_type: str | None = None # MIME type of raw file (application/pdf, etc.)@propertydef domain(self) -> str:@@ -54,6 +56,67 @@ class WebResult:return title_match or content_match or url_redirected+ def looks_like_junk(self) -> str | None:+ """Check if the result is junk that shouldn't be saved.++ Returns a reason string if junk, None if OK.+ """+ content = self.content or ""+ title_lower = (self.title or "").lower()+ content_lower = content[:2000].lower()++ # Empty or near-empty content+ if len(content.strip()) < 300:+ return "Empty or near-empty content"++ # Cloudflare / bot detection pages+ cf_signals = (+ "just a moment", "checking your browser", "ray id", "cloudflare",+ "please wait while we verify", "unusual activity", "captcha",+ "recaptcha", "verify you are human", "verify you are not a robot",+ "please complete the security check", "access denied",+ "enable javascript and cookies", "browser check",+ "ddos protection", "attention required",+ )+ if any(s in title_lower or s in content_lower for s in cf_signals):+ return f"Bot detection page: {self.title}"++ # Error pages+ error_signals = (+ "404 not found", "page not found", "403 forbidden",+ "500 internal server error", "502 bad gateway",+ "an error occurred", "this page isn't available",+ "the page you requested", "sorry, we couldn't find",+ )+ if any(s in title_lower or s in content_lower for s in error_signals):+ return f"Error page: {self.title}"++ # Search result / index pages (not actual content)+ search_signals = ("search results for", "results for query")+ if any(s in title_lower for s in search_signals):+ return f"Search results page: {self.title}"++ # Binary garbage from PDFs that weren't properly extracted+ pdf_binary_signals = ("endstream", "endobj", "/FlateDecode", "%PDF-")+ sample = content[:2000]+ if any(m in sample for m in pdf_binary_signals):+ return "Binary PDF garbage in content"++ non_printable = sum(1 for c in sample if ord(c) > 127 or c in '\x00\x01\x02\x03\x04\x05')+ if non_printable > len(sample) * 0.15:+ return "High ratio of binary/non-printable content"++ # Cookie consent / boilerplate pages (short with mostly nav/cookie text)+ if len(content.strip()) < 1500:+ cookie_signals = (+ "we use cookies", "cookie policy", "accept cookies", "cookie consent",+ "there appears to be a technical issue", "please enable javascript",+ )+ if any(s in content_lower for s in cookie_signals):+ return "Cookie/boilerplate page"++ return None+@runtime_checkableclass WebProvider(Protocol):diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.pyindex a6353bf..75740b1 100644--- a/src/hyperresearch/web/crawl4ai_provider.py+++ b/src/hyperresearch/web/crawl4ai_provider.py@@ -30,6 +30,108 @@ if sys.platform == "win32":pass+def _is_pdf_url(url: str) -> bool:+ """Check if URL likely points to a PDF."""+ from urllib.parse import urlparse, parse_qs++ parsed = urlparse(url.lower())+ path = parsed.path+ # Direct .pdf links+ if path.endswith(".pdf"):+ return True+ # Common academic PDF patterns+ if "/pdf/" in path or "/pdfs/" in path:+ return True+ # arXiv PDF links+ if "arxiv.org" in parsed.netloc and ("/pdf/" in path or "/abs/" in path):+ return True+ return False+++def _looks_like_binary(text: str) -> bool:+ """Check if extracted 'content' is actually binary garbage from a PDF."""+ if not text:+ return False+ sample = text[:2000]+ # PDF internal structure markers — dead giveaway+ pdf_markers = ("endstream", "endobj", "/Filter", "/FlateDecode", "stream\nx", "%PDF-")+ if any(m in sample for m in pdf_markers):+ return True+ # High ratio of non-printable or replacement chars+ non_printable = sum(1 for c in sample if ord(c) > 127 or c in '\x00\x01\x02\x03\x04\x05')+ return non_printable > len(sample) * 0.15+++def _fetch_pdf(url: str) -> WebResult | None:+ """Download a PDF and extract text using pymupdf. Returns None if extraction fails."""+ try:+ import pymupdf+ except ImportError:+ return None++ import httpx++ try:+ # Convert arXiv abs links to PDF links+ if "arxiv.org/abs/" in url:+ url = url.replace("/abs/", "/pdf/")+ if not url.endswith(".pdf"):+ url += ".pdf"++ resp = httpx.get(url, follow_redirects=True, timeout=30, verify=False, headers={+ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"+ })+ if resp.status_code != 200:+ return None++ content_type = resp.headers.get("content-type", "")+ if "pdf" not in content_type and not url.lower().endswith(".pdf"):+ return None # Not actually a PDF++ pdf_bytes = resp.content+ if len(pdf_bytes) < 100:+ return None++ doc = pymupdf.open(stream=pdf_bytes, filetype="pdf")++ # Extract text from all pages+ pages = []+ for page in doc:+ text = page.get_text("text")+ if text.strip():+ pages.append(text)++ doc.close()++ if not pages:+ return None++ # Build markdown from extracted text+ full_text = "\n\n---\n\n".join(pages)+ title = ""+ # Try to get title from first page (first non-empty line)+ for line in pages[0].split("\n"):+ line = line.strip()+ if len(line) > 10:+ title = line+ break++ return WebResult(+ url=url,+ title=title or f"PDF: {url.split('/')[-1]}",+ content=full_text,+ fetched_at=datetime.now(UTC),+ metadata={"content_type": "application/pdf", "pages": len(pages)},+ raw_bytes=pdf_bytes,+ raw_content_type="application/pdf",+ )++ except Exception as e:+ import logging+ logging.getLogger("hyperresearch.pdf").warning(f"PDF extraction failed for {url}: {e}")+ return None++class Crawl4AIProvider:name = "crawl4ai"@@ -84,10 +186,27 @@ class Crawl4AIProvider:)def fetch(self, url: str) -> WebResult:+ # PDF detection: fetch directly with httpx, extract text with pymupdf+ if _is_pdf_url(url):+ result = _fetch_pdf(url)+ if result is not None:+ return result+ # Fallback to browser if PDF fetch failed (might be a landing page, not actual PDF)+# When visible + profile: use Playwright directly (crawl4ai managed browser ignores headless=False)if not self._headless and self._data_dir:return asyncio.run(self._fetch_visible(url))- return asyncio.run(self._fetch_async(url))++ result = asyncio.run(self._fetch_async(url))++ # Post-fetch PDF detection: if the browser got binary garbage (PDF served+ # inline without proper content-type handling), re-fetch as a direct PDF download.+ if result.content and _looks_like_binary(result.content):+ pdf_result = _fetch_pdf(url)+ if pdf_result is not None:+ return pdf_result++ return resultasync def _fetch_visible(self, url: str) -> WebResult:"""Fetch using Playwright directly with a visible browser window.@@ -205,44 +324,64 @@ class Crawl4AIProvider:return asyncio.run(self._fetch_many_async(urls))async def _fetch_many_async(self, urls: list[str]) -> list[WebResult]:- async with AsyncWebCrawler(config=self._browser_config) as crawler:- results = await crawler.arun_many(urls=urls, config=self._run_config)- web_results = []- for cr, url in zip(results, urls, strict=False):- if not cr.success:- continue- metadata = cr.metadata or {}- md = cr.markdown- if md and hasattr(md, "raw_markdown"):- content = md.raw_markdown or ""- elif isinstance(md, str):- content = md- else:- content = ""-- media_raw = cr.media or {}- media = media_raw.get("images", []) if isinstance(media_raw, dict) else []-- screenshot_bytes = None- if cr.screenshot:- import base64-- try:- screenshot_bytes = base64.b64decode(cr.screenshot)- except Exception:- pass-- web_results.append(WebResult(- url=cr.url or url,- title=metadata.get("title", ""),- content=content,- raw_html=cr.html,- fetched_at=datetime.now(UTC),- metadata=metadata,- media=media,- screenshot=screenshot_bytes,- ))- return web_results+ # Split: PDFs go direct, rest go through browser+ pdf_urls = [u for u in urls if _is_pdf_url(u)]+ html_urls = [u for u in urls if not _is_pdf_url(u)]++ web_results = []++ # Fetch PDFs directly (no browser needed)+ for url in pdf_urls:+ pdf_result = _fetch_pdf(url)+ if pdf_result is not None:+ web_results.append(pdf_result)++ # Fetch HTML pages with browser+ if html_urls:+ async with AsyncWebCrawler(config=self._browser_config) as crawler:+ results = await crawler.arun_many(urls=html_urls, config=self._run_config)+ for cr, url in zip(results, html_urls, strict=False):+ if not cr.success:+ continue+ metadata = cr.metadata or {}+ md = cr.markdown+ if md and hasattr(md, "raw_markdown"):+ content = md.raw_markdown or ""+ elif isinstance(md, str):+ content = md+ else:+ content = ""++ # Post-fetch binary check — browser may have fetched a PDF inline+ if content and _looks_like_binary(content):+ pdf_result = _fetch_pdf(url)+ if pdf_result is not None:+ web_results.append(pdf_result)+ continue++ media_raw = cr.media or {}+ media = media_raw.get("images", []) if isinstance(media_raw, dict) else []++ screenshot_bytes = None+ if cr.screenshot:+ import base64++ try:+ screenshot_bytes = base64.b64decode(cr.screenshot)+ except Exception:+ pass++ web_results.append(WebResult(+ url=cr.url or url,+ title=metadata.get("title", ""),+ content=content,+ raw_html=cr.html,+ fetched_at=datetime.now(UTC),+ metadata=metadata,+ media=media,+ screenshot=screenshot_bytes,+ ))+ return web_resultsdef search(self, query: str, max_results: int = 5) -> list[WebResult]:raise NotImplementedError(