Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
Add web providers, fetch command, agent hooks, sources table
Jordan3cb6459ff21f
diff --git a/PLAN.md b/PLAN.mdnew file mode 100644index 0000000..2e2e046--- /dev/null+++ b/PLAN.md@@ -0,0 +1,330 @@+# Hyperresearch: Agent-Driven Research Knowledge Base++## Context++Kasten is a write-side knowledge base tool — agents/humans manually create notes. The gap in the ecosystem is a tool that lets an agent **actively collect, browse, and synthesize research** into a persistent, searchable knowledge base. Think: "kasten backbone + web browsing + slash-command UX like Graphify."++The user wants a new repo, new CLI name (`hyperresearch`), same core engine (markdown + YAML frontmatter + SQLite FTS5 + wiki-links + JSON output), but repositioned for **research workflows** where agents crawl sources, extract findings, and build a knowledge graph over time.++## What stays from kasten (the backbone)++These modules port over nearly verbatim:++| Module | What it does | Changes needed |+|--------|-------------|----------------|+| `core/vault.py` | Vault init, discovery, path properties | Rename dirs (`research/` instead of `knowledge/`?), rebrand |+| `core/sync.py` | mtime + SHA-256 sync, link resolution | None — same engine |+| `core/db.py` | SQLite schema, FTS5, WAL mode | Add `sources` table for URL tracking |+| `core/note.py` | read/write markdown with frontmatter | Add source-aware fields |+| `core/frontmatter.py` | YAML parse/serialize | None |+| `core/config.py` | TOML config | New defaults, new sections for web providers |+| `core/patterns.py` | Regex (wiki-links, code blocks) | None |+| `core/agent_docs.py` | Inject into CLAUDE.md etc | Rewrite blurb for research workflow |+| `models/` | Note, Envelope, SearchResult | Extend with Source model |+| `search/` | FTS5 + filters | Add source/URL filters |+| `serve/` | Web viewer | Rebrand theme |+| `mcp/` | MCP server | Add research-specific tools |++## What's new++### 1. Web provider plugin system++**Architecture:** Abstract `WebProvider` protocol, concrete implementations as optional deps.++```python+# src/hyperresearch/web/base.py+class WebProvider(Protocol):+ def fetch(self, url: str) -> WebResult: ...+ def search(self, query: str, max_results: int = 5) -> list[WebResult]: ...++@dataclass+class WebResult:+ url: str+ title: str+ content: str # clean markdown+ raw_html: str | None+ fetched_at: datetime+ metadata: dict # author, date, domain, etc.+```++**Provider implementations (each an optional dep group):**++| Provider | Install | Cost | Best for |+|----------|---------|------|----------|+| `crawl4ai` | `pip install hyperresearch[crawl4ai]` | Free, open-source | Default. 60k stars, agentic crawling, JS rendering |+| `tavily` | `pip install hyperresearch[tavily]` | $0.008/credit, 1k free/mo | Search-first workflows (API key required) |+| `firecrawl` | `pip install hyperresearch[firecrawl]` | Free self-hosted or paid API | Best markdown output, site-wide crawling |+| `trafilatura` | `pip install hyperresearch[trafilatura]` | Free | Lightweight text extraction, no JS |+| builtin | (included) | Free | `httpx` + `beautifulsoup4` fallback, no JS rendering |++**Config in `.hyperresearch/config.toml`:**+```toml+[web]+provider = "crawl4ai" # or "tavily", "firecrawl", "trafilatura", "builtin"+tavily_api_key = "" # or env var TAVILY_API_KEY+firecrawl_api_key = "" # or env var FIRECRAWL_API_KEY+```++### 2. New CLI commands (research-focused)++```bash+# Fetch a URL, extract content, save as a note+hyperresearch fetch "https://arxiv.org/abs/2401.12345" --tag transformers --tag attention++# Search the web, save top results as notes+hyperresearch websearch "state space models 2026" --max 5 --tag ssm++# Deep research: search → fetch → follow links → extract → save+hyperresearch research "mamba architecture" --depth 2 --max-pages 10++# All the kasten stuff still works+hyperresearch search "attention" -j+hyperresearch note show attention-is-all-you-need -j+hyperresearch repair -j+hyperresearch sync+```++**Command details:**++- **`fetch <url>`** — Single URL → note. Extracts clean markdown via provider, creates note with `source: <url>`, auto-tags if possible, records fetch metadata. Deduplicates by URL (warns if already fetched).++- **`websearch <query>`** — Uses provider's search API → creates one note per result. Each note links back to source URL. Returns JSON list of created note IDs.++- **`research <topic>`** — The flagship command. Agent-driven deep research:+ 1. Web search for topic+ 2. Fetch top N results+ 3. Follow outbound links up to `--depth` levels+ 4. Extract and save as notes with wiki-links between related findings+ 5. Generate a synthesis note (map-of-content) linking everything found++ This is the command that makes it "fancy" — it's what agents invoke to build a knowledge base from scratch on a topic.++### 3. Slash-command / hook integration (like Graphify)++**This is the key adoption play.** Install hyperresearch, run one command, and every agent in the repo knows about it.++```bash+pip install hyperresearch+hyperresearch install # sets up hooks + agent docs+```++**`hyperresearch install` does:**++1. Creates `.hyperresearch/` directory (vault init)+2. Injects research-focused blurb into agent config files+3. **Installs PreToolUse hooks** for supported platforms:++| Platform | Hook location | Trigger | What it does |+|----------|--------------|---------|--------------|+| Claude Code | `.claude/settings.json` | Before `Glob`, `Grep`, `WebSearch`, `WebFetch` | "Check hyperresearch KB first. Run `hyperresearch search <topic> -j` before raw web searches." |+| Codex | `.codex/hooks.json` | Before `Bash` | Same reminder |+| Cursor | `.cursor/rules/hyperresearch.mdc` | `alwaysApply: true` | System rule with research workflow |+| Gemini CLI | `.gemini/settings.json` | `BeforeTool` | Same pattern |++The hook script (a small JS or Python file at `.hyperresearch/hook.js`):+- Checks if a research KB exists+- If yes, reminds the agent to `hyperresearch search` before going to the web+- Prevents redundant web fetches for topics already researched++### 4. Source tracking (new DB table)++```sql+CREATE TABLE sources (+ url TEXT PRIMARY KEY,+ note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,+ domain TEXT,+ fetched_at TEXT,+ provider TEXT, -- which web provider fetched it+ content_hash TEXT, -- detect if page changed+ status TEXT DEFAULT 'active', -- active, dead, redirected+ CHECK (status IN ('active', 'dead', 'redirected'))+);+CREATE INDEX idx_sources_domain ON sources(domain);+CREATE INDEX idx_sources_note ON sources(note_id);+```++This lets agents ask: "have we already fetched this URL?" and "what sources did we use for this topic?"++### 5. Extended frontmatter for research notes++```yaml+---+title: "Mamba: Linear-Time Sequence Modeling"+id: mamba-linear-time-sequence-modeling+tags: [ssm, mamba, sequence-modeling]+status: review+source: https://arxiv.org/abs/2312.00752+source_domain: arxiv.org+fetched_at: 2026-04-09T12:00:00Z+fetch_provider: crawl4ai+confidence: 0.85+summary: "Mamba introduces selective state spaces that achieve transformer-quality with linear scaling."+---+```++The `source`, `source_domain`, `fetched_at`, `fetch_provider` fields are new. The rest already exists in kasten's NoteMeta.++## Project structure++```+hyperresearch/+├── pyproject.toml+├── src/hyperresearch/+│ ├── __init__.py+│ ├── __main__.py+│ ├── cli/ # [from kasten] + new commands+│ │ ├── __init__.py # App assembly+│ │ ├── main.py # init, status, sync, install+│ │ ├── search.py # [from kasten]+│ │ ├── note.py # [from kasten]+│ │ ├── repair.py # [from kasten]+│ │ ├── fetch.py # NEW: fetch, websearch commands+│ │ ├── research.py # NEW: deep research command+│ │ ├── sources.py # NEW: source management+│ │ ├── graph.py # [from kasten]+│ │ ├── lint.py # [from kasten]+│ │ ├── tag.py # [from kasten]+│ │ └── ...+│ ├── core/ # [from kasten, nearly verbatim]+│ │ ├── vault.py+│ │ ├── sync.py+│ │ ├── db.py # + sources table+│ │ ├── note.py+│ │ ├── config.py # + web provider config+│ │ ├── agent_docs.py # rewritten blurb+│ │ ├── hooks.py # NEW: PreToolUse hook installer+│ │ └── ...+│ ├── web/ # NEW: web provider system+│ │ ├── __init__.py+│ │ ├── base.py # WebProvider protocol + WebResult+│ │ ├── builtin.py # httpx + bs4 fallback+│ │ ├── crawl4ai_provider.py+│ │ ├── tavily_provider.py+│ │ ├── firecrawl_provider.py+│ │ └── trafilatura_provider.py+│ ├── models/ # [from kasten] + Source model+│ ├── search/ # [from kasten]+│ ├── serve/ # [from kasten, rebranded]+│ ├── mcp/ # [from kasten] + research tools+│ └── indexgen/ # [from kasten]+├── tests/+├── README.md+├── CLAUDE.md+└── .github/workflows/+```++## Implementation phases++### Phase 1: Fork and rebrand — NOW+**Target:** `C:\Users\Jordan\PycharmProjects\hyperresearch` (PyCharm project already created)++1. Copy kasten's `src/kasten/` → `src/hyperresearch/` (all modules)+2. Copy `tests/`, `pyproject.toml`, `LICENSE`, `CONTRIBUTING.md`+3. Global rename in all files: `kasten` → `hyperresearch`, `kas` → `hpr` (alias)+4. Rename `knowledge/` → `research/` in vault defaults+5. Rename `.kasten/` → `.hyperresearch/` in vault discovery+6. Update pyproject.toml: name, entry points, description, add httpx + bs4 to core deps+7. Verify all tests pass with new names+8. `pip install -e ".[dev]"` works+9. `git init` + initial commit++### Phase 2: Web provider system (day 2-3)+1. Define `WebProvider` protocol in `web/base.py`+2. Implement `builtin` provider (httpx + bs4, always available)+3. Implement `crawl4ai` provider (optional dep)+4. Add `[web]` section to config.toml+5. Add `sources` table to DB schema+6. Wire up provider selection in config++### Phase 3: fetch + websearch commands (day 3-4)+1. `hyperresearch fetch <url>` — single URL to note+2. `hyperresearch websearch <query>` — search and save results+3. URL deduplication via sources table+4. Source metadata in frontmatter+5. Tests for both commands++### Phase 4: Hook system + install command (day 4-5)+1. `hyperresearch install` command+2. Claude Code PreToolUse hook (`.claude/settings.json`)+3. Cursor rule (`.cursor/rules/hyperresearch.mdc`)+4. Codex hook (`.codex/hooks.json`)+5. Gemini hook (`.gemini/settings.json`)+6. Rewrite agent docs blurb for research workflow+7. Hook reminder script (JS or Python)++### Phase 5: research command (day 5-7)+1. Orchestrates websearch → fetch → follow links → save+2. Depth-limited crawling+3. Auto wiki-linking between fetched notes+4. Synthesis note generation (map-of-content)+5. This is the flagship "demo" command++### Phase 6: Additional providers + polish (day 7-8)+1. Tavily provider+2. Firecrawl provider+3. Trafilatura provider+4. MCP server with research-specific tools (fetch, websearch, sources)+5. README, demo GIF, PyPI publish++## pyproject.toml sketch++```toml+[project]+name = "hyperresearch"+version = "0.1.0"+description = "Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki."+requires-python = ">=3.11"+dependencies = [+ "typer>=0.9.0",+ "rich>=13.0",+ "pyyaml>=6.0",+ "pydantic>=2.0",+ "jinja2>=3.1",+ "platformdirs>=4.0",+]++# httpx + bs4 NOT core — agents already have WebSearch/WebFetch++[project.optional-dependencies]+crawl4ai = ["Crawl4AI>=0.4"]+tavily = ["tavily-python>=0.5"]+firecrawl = ["firecrawl>=1.0"]+trafilatura = ["trafilatura>=2.0"]+mcp = ["mcp>=1.6"]+all = ["hyperresearch[crawl4ai,tavily,firecrawl,trafilatura,mcp]"]+dev = ["pytest", "pytest-cov", "ruff", "mypy"]++[project.scripts]+hyperresearch = "hyperresearch.cli:app"+hpr = "hyperresearch.cli:app"+```++## Key design decisions++1. **httpx + bs4 are core deps, not optional** — the builtin provider should always work without extras. This is the "it just works" baseline.++2. **crawl4ai is the recommended default** — free, open-source, 60k stars, handles JS rendering. But it's optional because it pulls in heavy deps (playwright/chromium).++3. **`install` not `init`** — Graphify uses `install` for the hook setup. We use `install` to do vault init + hook installation in one step. `init` still works as an alias for just the vault.++4. **Hooks remind, not enforce** — the PreToolUse hook says "check hyperresearch first" but doesn't block the agent. Blocking would be annoying.++5. **Notes are still files** — same as kasten philosophy. The web provider fetches content, but it gets saved as a .md file with frontmatter. Files are truth, DB is cache.++6. **No LLM calls in the tool itself** — hyperresearch doesn't call Claude/GPT to summarize or synthesize. The AGENT using hyperresearch does that. hyperresearch is still a "dumb tool" — it stores, indexes, and searches. The agent decides what to do with it.++7. **Agents already have web access** — Claude Code has WebSearch/WebFetch, Codex has browsing, etc. So the DEFAULT flow is: agent fetches content with its own tools → pipes it into hyperresearch via `--body-file` or `--body-stdin`. The optional web providers (crawl4ai, tavily, firecrawl) are extras for standalone/headless use, NOT required.++## Verification++1. `pip install -e ".[dev]"` — core install works+2. `hyperresearch install` — creates vault + hooks+3. `hyperresearch fetch "https://en.wikipedia.org/wiki/Zettelkasten"` — fetches and saves note+4. `hyperresearch search "zettelkasten" -j` — finds the fetched note+5. `hyperresearch websearch "state space models" --max 3` — searches and saves 3 notes+6. `python -m pytest tests/` — all tests pass+7. Open a Claude Code session in the repo — agent sees the hook reminder, uses `hyperresearch search` before web searches++## Name availability check needed+- PyPI: check if `hyperresearch` is taken+- GitHub: check if `hyperresearch` repo name is availablediff --git a/src/hyperresearch/cli/__init__.py b/src/hyperresearch/cli/__init__.pyindex e3aaa06..2069b7b 100644--- a/src/hyperresearch/cli/__init__.py+++ b/src/hyperresearch/cli/__init__.py@@ -38,12 +38,16 @@ from hyperresearch.cli.repair import repair as _repairfrom hyperresearch.cli.search import search as _searchfrom hyperresearch.cli.serve import serve as _servefrom hyperresearch.cli.tag import tag_list as _tags+from hyperresearch.cli.fetch import fetch as _fetch+from hyperresearch.cli.install import install as _installfrom hyperresearch.cli.watch import watch as _watch+app.command("install")(_install)app.command("init")(_init)app.command("status")(_status)app.command("sync")(_sync)app.command("search")(_search)+app.command("fetch")(_fetch)app.command("tags")(_tags)app.command("show", hidden=True)(_show)app.command("dedup")(_dedup)diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.pynew file mode 100644index 0000000..5c5636e--- /dev/null+++ b/src/hyperresearch/cli/fetch.py@@ -0,0 +1,128 @@+"""Fetch and websearch commands — save web content as research notes."""++from __future__ import annotations++import hashlib+import sys+from datetime import datetime, timezone+from pathlib import Path+from urllib.parse import urlparse++import typer++from hyperresearch.cli._output import console, output+from hyperresearch.models.output import error, success++app = typer.Typer()++[email protected]("fetch")+def fetch(+ url: str = typer.Argument(..., help="URL to fetch and save as a note"),+ tags: list[str] = typer.Option([], "--tag", "-t", help="Tags (repeatable)"),+ title: str | None = typer.Option(None, "--title", help="Override title"),+ parent: str | None = typer.Option(None, "--parent", "-p", help="Parent topic"),+ provider_name: str | None = typer.Option(None, "--provider", help="Web provider override"),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """Fetch a URL and save its content as a research note."""+ from hyperresearch.core.note import write_note+ from hyperresearch.core.sync import compute_sync_plan, execute_sync+ from hyperresearch.core.vault import Vault, VaultError+ from hyperresearch.web.base import get_provider++ try:+ vault = Vault.discover()+ except VaultError as e:+ if json_output:+ output(error(str(e), "NO_VAULT"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {e}")+ raise typer.Exit(1)++ vault.auto_sync()+ conn = vault.db++ # Check if URL already fetched+ existing = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()+ if existing:+ note_id = existing["note_id"]+ if json_output:+ output(+ error(f"URL already fetched as note '{note_id}'", "DUPLICATE_URL"),+ json_mode=True,+ )+ else:+ console.print(f"[yellow]Already fetched:[/] {url} → note '{note_id}'")+ raise typer.Exit(1)++ # Fetch content+ prov = get_provider(provider_name or vault.config.web_provider)+ if not json_output:+ console.print(f"[dim]Fetching with {prov.name}...[/]")++ try:+ result = prov.fetch(url)+ except Exception as e:+ if json_output:+ output(error(str(e), "FETCH_ERROR"), json_mode=True)+ else:+ console.print(f"[red]Fetch failed:[/] {e}")+ raise typer.Exit(1)++ # Write note+ note_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"+ domain = result.domain++ extra_meta = {+ "source": url,+ "source_domain": domain,+ "fetched_at": result.fetched_at.isoformat(),+ "fetch_provider": prov.name,+ }+ if result.metadata.get("author"):+ extra_meta["author"] = result.metadata["author"]++ note_path = write_note(+ vault.notes_dir,+ title=note_title,+ body=result.content,+ tags=tags,+ status="draft",+ source=url,+ parent=parent,+ extra_frontmatter=extra_meta,+ )++ # Record source in DB+ content_hash = hashlib.sha256(result.content.encode("utf-8")).hexdigest()[:16]+ note_id = note_path.stem+ conn.execute(+ """INSERT INTO sources (url, note_id, domain, fetched_at, provider, content_hash)+ VALUES (?, ?, ?, ?, ?, ?)""",+ (url, note_id, domain, result.fetched_at.isoformat(), prov.name, content_hash),+ )+ conn.commit()++ # Sync to index the new note+ plan = compute_sync_plan(vault)+ if plan.to_add or plan.to_update:+ execute_sync(vault, plan)++ data = {+ "note_id": note_id,+ "title": note_title,+ "url": url,+ "domain": domain,+ "provider": prov.name,+ "path": str(note_path.relative_to(vault.root)),+ "word_count": len(result.content.split()),+ }++ if json_output:+ output(success(data, vault=str(vault.root)), json_mode=True)+ else:+ console.print(f"[green]Saved:[/] {note_title}")+ console.print(f" ID: {note_id}")+ console.print(f" Source: {url}")+ console.print(f" Words: {data['word_count']}")diff --git a/src/hyperresearch/cli/install.py b/src/hyperresearch/cli/install.pynew file mode 100644index 0000000..0e90bce--- /dev/null+++ b/src/hyperresearch/cli/install.py@@ -0,0 +1,77 @@+"""Install command — one-step setup: vault init + agent hooks + docs injection."""++from __future__ import annotations++from pathlib import Path++import typer++from hyperresearch.cli._output import console, output+from hyperresearch.models.output import error, success+++def install(+ path: str = typer.Argument(".", help="Path to install in"),+ name: str = typer.Option("Research Base", "--name", "-n", help="Vault name"),+ platforms: list[str] = typer.Option(+ ["claude"],+ "--platform",+ "-p",+ help="Agent platforms to hook: claude|codex|cursor|gemini|all",+ ),+ agents: list[str] = typer.Option(+ ["claude"],+ "--agents",+ "-a",+ help="Agent doc files: claude|agents|gemini|copilot",+ ),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """Install hyperresearch: init vault + inject agent docs + install hooks."""+ from hyperresearch.core.hooks import install_hooks+ from hyperresearch.core.vault import Vault, VaultError++ root = Path(path).resolve()++ # Step 1: Init vault (skip if already exists)+ try:+ vault = Vault.discover(root)+ vault_action = "existing"+ except VaultError:+ try:+ vault = Vault.init(root, name=name, agents=agents)+ vault_action = "created"+ except VaultError as e:+ if json_output:+ output(error(str(e), "INIT_ERROR"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {e}")+ raise typer.Exit(1)++ # Step 2: Install hooks+ hook_actions = install_hooks(root, platforms=platforms)++ # Step 3: Report+ data = {+ "vault_path": str(vault.root),+ "vault": vault_action,+ "hooks_installed": hook_actions,+ "platforms": platforms,+ }++ if json_output:+ output(success(data, vault=str(vault.root)), json_mode=True)+ else:+ if vault_action == "created":+ console.print(f"[green]Vault created:[/] {vault.root}")+ else:+ console.print(f"[dim]Vault exists:[/] {vault.root}")++ if hook_actions:+ console.print("[green]Hooks installed:[/]")+ for action in hook_actions:+ console.print(f" {action}")+ else:+ console.print("[dim]All hooks already installed.[/]")++ console.print("\n[bold]Ready.[/] Agents will now check the research base before web searches.")diff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.pyindex cdcaf14..8d530f1 100644--- a/src/hyperresearch/core/config.py+++ b/src/hyperresearch/core/config.py@@ -31,6 +31,9 @@ class VaultConfig:])+ # Web provider+ web_provider: str = "builtin"+# Indexauto_build_index: bool = Trueindex_pages: list[str] = field(@@ -48,6 +51,7 @@ class VaultConfig:sync = data.get("sync", {})index = data.get("index", {})search = data.get("search", {})+ web = data.get("web", {})return cls(name=vault.get("name", cls.name),@@ -60,6 +64,7 @@ class VaultConfig:search_boost_evergreen=search.get("boost_evergreen", cls.search_boost_evergreen),search_penalize_deprecated=search.get("penalize_deprecated", cls.search_penalize_deprecated),search_penalize_stale=search.get("penalize_stale", cls.search_penalize_stale),+ web_provider=web.get("provider", cls.web_provider),auto_sync=sync.get("auto_sync", cls.auto_sync),exclude_patterns=sync.get("exclude_patterns", cls().exclude_patterns),auto_build_index=index.get("auto_build", cls.auto_build_index),@@ -87,6 +92,9 @@ class VaultConfig:f"penalize_deprecated = {self.search_penalize_deprecated}",f"penalize_stale = {self.search_penalize_stale}","",+ "[web]",+ f'provider = "{self.web_provider}"',+ "","[sync]",f"auto_sync = {'true' if self.auto_sync else 'false'}",f"exclude_patterns = {self._toml_array(self.exclude_patterns)}",diff --git a/src/hyperresearch/core/db.py b/src/hyperresearch/core/db.pyindex e122dd4..ff4c0a9 100644--- a/src/hyperresearch/core/db.py+++ b/src/hyperresearch/core/db.py@@ -5,7 +5,7 @@ from __future__ import annotationsimport sqlite3from pathlib import Path-SCHEMA_VERSION = 2+SCHEMA_VERSION = 3SCHEMA_SQL = """PRAGMA journal_mode=WAL;@@ -99,6 +99,20 @@ CREATE TABLE IF NOT EXISTS tag_aliases (canonical TEXT NOT NULL);+CREATE TABLE IF NOT EXISTS sources (+ url TEXT PRIMARY KEY,+ note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,+ domain TEXT,+ fetched_at TEXT,+ provider TEXT,+ content_hash TEXT,+ status TEXT NOT NULL DEFAULT 'active'+ CHECK (status IN ('active', 'dead', 'redirected'))+);++CREATE INDEX IF NOT EXISTS idx_sources_domain ON sources(domain);+CREATE INDEX IF NOT EXISTS idx_sources_note ON sources(note_id);+"""FTS_SQL = """diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.pynew file mode 100644index 0000000..b6a8016--- /dev/null+++ b/src/hyperresearch/core/hooks.py@@ -0,0 +1,220 @@+"""Agent hook installer — injects PreToolUse hooks for Claude Code, Codex, Cursor, Gemini CLI.++These hooks remind agents to check the research base before doing raw web searches.+"""++from __future__ import annotations++import json+from pathlib import Path++# The hook script that gets installed+HOOK_SCRIPT = """\+#!/usr/bin/env node+/**+ * hyperresearch PreToolUse hook — reminds agent to check research base first.+ * Installed by: hyperresearch install+ */+const fs = require('fs');+const path = require('path');++// Check if a .hyperresearch directory exists (vault is initialized)+function findVault() {+ let dir = process.env.CLAUDE_PROJECT_DIR || process.cwd();+ while (true) {+ if (fs.existsSync(path.join(dir, '.hyperresearch'))) return dir;+ const parent = path.dirname(dir);+ if (parent === dir) return null;+ dir = parent;+ }+}++const vault = findVault();+if (vault) {+ const msg = [+ 'HYPERRESEARCH: A research knowledge base exists in this project.',+ 'Before searching the web, check existing research:',+ ' hyperresearch search "<your query>" -j',+ 'Save useful findings back:',+ ' hyperresearch note new "Title" --tag t --body-file /tmp/content.md --source "url" -j',+ ].join('\\n');+ process.stderr.write(msg + '\\n');+}+"""++# Cursor rule file content+CURSOR_RULE = """\+---+description: Hyperresearch research base integration+alwaysApply: true+---++# Research Base (hyperresearch)++This project has a hyperresearch research base. Before searching the web or browsing URLs:++1. **Check existing research first**: `hyperresearch search "<query>" -j`+2. **Save useful findings**: `hyperresearch note new "Title" --tag t --body-file /tmp/content.md --source "url" -j`+3. **Browse the knowledge graph**: `hyperresearch graph hubs -j` for key topics++The research base persists across sessions. Don't duplicate work — check before fetching.+"""+++def install_hooks(vault_root: Path, platforms: list[str] | None = None) -> list[str]:+ """Install agent hooks for specified platforms. Returns list of actions taken."""+ if platforms is None:+ platforms = ["claude"]++ actions = []++ if "claude" in platforms or "all" in platforms:+ result = _install_claude_hook(vault_root)+ if result:+ actions.append(result)++ if "codex" in platforms or "all" in platforms:+ result = _install_codex_hook(vault_root)+ if result:+ actions.append(result)++ if "cursor" in platforms or "all" in platforms:+ result = _install_cursor_rule(vault_root)+ if result:+ actions.append(result)++ if "gemini" in platforms or "all" in platforms:+ result = _install_gemini_hook(vault_root)+ if result:+ actions.append(result)++ return actions+++def _write_hook_script(vault_root: Path) -> Path:+ """Write the hook JS script to .hyperresearch/hook.js."""+ hook_dir = vault_root / ".hyperresearch"+ hook_dir.mkdir(parents=True, exist_ok=True)+ hook_path = hook_dir / "hook.js"+ hook_path.write_text(HOOK_SCRIPT, encoding="utf-8")+ return hook_path+++def _install_claude_hook(vault_root: Path) -> str | None:+ """Install PreToolUse hook into .claude/settings.json."""+ hook_path = _write_hook_script(vault_root)++ settings_dir = vault_root / ".claude"+ settings_dir.mkdir(exist_ok=True)+ settings_path = settings_dir / "settings.json"++ settings = {}+ if settings_path.exists():+ try:+ settings = json.loads(settings_path.read_text(encoding="utf-8"))+ except (json.JSONDecodeError, OSError):+ pass++ hooks = settings.setdefault("hooks", {})+ pre_tool = hooks.setdefault("PreToolUse", [])++ # Check if already installed+ for entry in pre_tool:+ if isinstance(entry, dict):+ for h in entry.get("hooks", []):+ if "hyperresearch" in h.get("command", ""):+ return None # Already installed++ # Add hook that fires before web-related tools+ pre_tool.append({+ "matcher": "Glob|Grep|WebSearch|WebFetch",+ "hooks": [{+ "type": "command",+ "command": f"node {hook_path.as_posix()}",+ }],+ })++ settings_path.write_text(json.dumps(settings, indent=2) + "\n", encoding="utf-8")+ return "Claude Code: .claude/settings.json (PreToolUse hook)"+++def _install_codex_hook(vault_root: Path) -> str | None:+ """Install hook into .codex/hooks.json."""+ hook_path = _write_hook_script(vault_root)++ codex_dir = vault_root / ".codex"+ codex_dir.mkdir(exist_ok=True)+ hooks_path = codex_dir / "hooks.json"++ hooks = {}+ if hooks_path.exists():+ try:+ hooks = json.loads(hooks_path.read_text(encoding="utf-8"))+ except (json.JSONDecodeError, OSError):+ pass++ pre_tool = hooks.setdefault("PreToolUse", [])+ for entry in pre_tool:+ if isinstance(entry, dict):+ for h in entry.get("hooks", []):+ if "hyperresearch" in h.get("command", ""):+ return None++ pre_tool.append({+ "matcher": "Bash",+ "hooks": [{+ "type": "command",+ "command": f"node {hook_path.as_posix()}",+ }],+ })++ hooks_path.write_text(json.dumps(hooks, indent=2) + "\n", encoding="utf-8")+ return "Codex: .codex/hooks.json (PreToolUse hook)"+++def _install_cursor_rule(vault_root: Path) -> str | None:+ """Install always-apply rule into .cursor/rules/hyperresearch.mdc."""+ rules_dir = vault_root / ".cursor" / "rules"+ rules_dir.mkdir(parents=True, exist_ok=True)+ rule_path = rules_dir / "hyperresearch.mdc"++ if rule_path.exists():+ return None # Already exists++ rule_path.write_text(CURSOR_RULE, encoding="utf-8")+ return "Cursor: .cursor/rules/hyperresearch.mdc (alwaysApply rule)"+++def _install_gemini_hook(vault_root: Path) -> str | None:+ """Install BeforeTool hook into .gemini/settings.json."""+ hook_path = _write_hook_script(vault_root)++ gemini_dir = vault_root / ".gemini"+ gemini_dir.mkdir(exist_ok=True)+ settings_path = gemini_dir / "settings.json"++ settings = {}+ if settings_path.exists():+ try:+ settings = json.loads(settings_path.read_text(encoding="utf-8"))+ except (json.JSONDecodeError, OSError):+ pass++ hooks = settings.setdefault("hooks", {})+ before_tool = hooks.setdefault("BeforeTool", [])++ for entry in before_tool:+ if isinstance(entry, dict):+ for h in entry.get("hooks", []):+ if "hyperresearch" in h.get("command", ""):+ return None++ before_tool.append({+ "hooks": [{+ "type": "command",+ "command": f"node {hook_path.as_posix()}",+ }],+ })++ settings_path.write_text(json.dumps(settings, indent=2) + "\n", encoding="utf-8")+ return "Gemini CLI: .gemini/settings.json (BeforeTool hook)"diff --git a/src/hyperresearch/core/migrations.py b/src/hyperresearch/core/migrations.pyindex 1748f25..8a9156d 100644--- a/src/hyperresearch/core/migrations.py+++ b/src/hyperresearch/core/migrations.py@@ -12,6 +12,20 @@ CREATE TABLE IF NOT EXISTS tag_aliases (alias TEXT PRIMARY KEY,canonical TEXT NOT NULL);+""",+ 3: """+CREATE TABLE IF NOT EXISTS sources (+ url TEXT PRIMARY KEY,+ note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,+ domain TEXT,+ fetched_at TEXT,+ provider TEXT,+ content_hash TEXT,+ status TEXT NOT NULL DEFAULT 'active'+ CHECK (status IN ('active', 'dead', 'redirected'))+);+CREATE INDEX IF NOT EXISTS idx_sources_domain ON sources(domain);+CREATE INDEX IF NOT EXISTS idx_sources_note ON sources(note_id);""",}diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.pyindex e3fc684..b9abd32 100644--- a/src/hyperresearch/core/note.py+++ b/src/hyperresearch/core/note.py@@ -59,14 +59,16 @@ def write_note(parent: str | None = None,source: str | None = None,summary: str | None = None,+ extra_frontmatter: dict | None = None,) -> Path:"""Create a new note file on disk. Returns the file path.Args:notes_dir: The directory to write into (e.g. vault.notes_dir).+ extra_frontmatter: Additional fields to set on NoteMeta (e.g. source_domain, fetched_at)."""nid = note_id or slugify(title)- meta = NoteMeta(+ kwargs: dict = dict(title=title,id=nid,tags=tags or [],@@ -77,6 +79,9 @@ def write_note(summary=summary,created=datetime.now(UTC),)+ if extra_frontmatter:+ kwargs.update(extra_frontmatter)+ meta = NoteMeta(**kwargs)# Determine output path, avoid collisionstarget_dir = notes_dirdiff --git a/src/hyperresearch/models/note.py b/src/hyperresearch/models/note.pyindex 336fa8c..b293eb7 100644--- a/src/hyperresearch/models/note.py+++ b/src/hyperresearch/models/note.py@@ -48,6 +48,9 @@ class NoteMeta(BaseModel):created: datetime = Field(default_factory=lambda: datetime.now(UTC))updated: datetime | None = Nonesource: str | None = None+ source_domain: str | None = None+ fetched_at: datetime | None = None+ fetch_provider: str | None = Nonestatus: NoteStatus = NoteStatus.DRAFTtype: NoteType = NoteType.NOTEaliases: list[str] = Field(default_factory=list)diff --git a/src/hyperresearch/web/__init__.py b/src/hyperresearch/web/__init__.pynew file mode 100644index 0000000..23ae07f--- /dev/null+++ b/src/hyperresearch/web/__init__.py@@ -0,0 +1 @@+"""Web provider system — pluggable backends for fetching and searching the web."""diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.pynew file mode 100644index 0000000..c5ffd0a--- /dev/null+++ b/src/hyperresearch/web/base.py@@ -0,0 +1,88 @@+"""Base protocol and data types for web providers."""++from __future__ import annotations++from dataclasses import dataclass, field+from datetime import datetime, timezone+from typing import Protocol, runtime_checkable+++@dataclass+class WebResult:+ """A single web fetch or search result."""++ url: str+ title: str+ content: str # clean markdown or plain text+ fetched_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))+ raw_html: str | None = None+ metadata: dict = field(default_factory=dict) # author, date, domain, etc.++ @property+ def domain(self) -> str:+ from urllib.parse import urlparse++ return urlparse(self.url).netloc+++@runtime_checkable+class WebProvider(Protocol):+ """Protocol for web content providers.++ Implementations must support at least fetch(). search() is optional —+ providers that don't support search raise NotImplementedError.+ """++ name: str++ def fetch(self, url: str) -> WebResult:+ """Fetch a single URL and return clean content."""+ ...++ def search(self, query: str, max_results: int = 5) -> list[WebResult]:+ """Search the web and return results with content."""+ ...+++def get_provider(name: str | None = None) -> WebProvider:+ """Load a web provider by name. Falls back to builtin if none specified."""+ if name is None or name == "builtin":+ from hyperresearch.web.builtin import BuiltinProvider++ return BuiltinProvider()++ if name == "crawl4ai":+ try:+ from hyperresearch.web.crawl4ai_provider import Crawl4AIProvider++ return Crawl4AIProvider()+ except ImportError:+ raise ImportError("crawl4ai provider requires: pip install hyperresearch[crawl4ai]")++ if name == "tavily":+ try:+ from hyperresearch.web.tavily_provider import TavilyProvider++ return TavilyProvider()+ except ImportError:+ raise ImportError("tavily provider requires: pip install hyperresearch[tavily]")++ if name == "firecrawl":+ try:+ from hyperresearch.web.firecrawl_provider import FirecrawlProvider++ return FirecrawlProvider()+ except ImportError:+ raise ImportError("firecrawl provider requires: pip install hyperresearch[firecrawl]")++ if name == "trafilatura":+ try:+ from hyperresearch.web.trafilatura_provider import TrafilaturaProvider++ return TrafilaturaProvider()+ except ImportError:+ raise ImportError(+ "trafilatura provider requires: pip install hyperresearch[trafilatura]"+ )++ raise ValueError(f"Unknown web provider: {name!r}")diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.pynew file mode 100644index 0000000..b1d739f--- /dev/null+++ b/src/hyperresearch/web/builtin.py@@ -0,0 +1,112 @@+"""Builtin web provider — uses httpx + beautifulsoup4 if available, else bare urllib."""++from __future__ import annotations++import re+from datetime import datetime, timezone+from html.parser import HTMLParser++from hyperresearch.web.base import WebResult+++class _TextExtractor(HTMLParser):+ """Minimal HTML-to-text extractor (no external deps)."""++ def __init__(self):+ super().__init__()+ self._pieces: list[str] = []+ self._skip = False+ self._skip_tags = {"script", "style", "nav", "footer", "header"}+ self._title = ""+ self._in_title = False++ def handle_starttag(self, tag, attrs):+ if tag in self._skip_tags:+ self._skip = True+ if tag == "title":+ self._in_title = True+ if tag in ("p", "br", "div", "h1", "h2", "h3", "h4", "h5", "h6", "li"):+ self._pieces.append("\n")++ def handle_endtag(self, tag):+ if tag in self._skip_tags:+ self._skip = False+ if tag == "title":+ self._in_title = False++ def handle_data(self, data):+ if self._in_title:+ self._title = data.strip()+ if not self._skip:+ self._pieces.append(data)++ def get_text(self) -> str:+ raw = "".join(self._pieces)+ # Collapse whitespace+ lines = [line.strip() for line in raw.splitlines()]+ return re.sub(r"\n{3,}", "\n\n", "\n".join(lines)).strip()+++class BuiltinProvider:+ """Minimal web fetcher using stdlib + optional httpx/bs4."""++ name = "builtin"++ def fetch(self, url: str) -> WebResult:+ html, final_url = self._download(url)+ title, content = self._extract(html)+ return WebResult(+ url=final_url,+ title=title,+ content=content,+ raw_html=html,+ fetched_at=datetime.now(timezone.utc),+ )++ def search(self, query: str, max_results: int = 5) -> list[WebResult]:+ raise NotImplementedError(+ "Builtin provider does not support web search. "+ "Install a search-capable provider: pip install hyperresearch[tavily] "+ "or pip install hyperresearch[crawl4ai]"+ )++ def _download(self, url: str) -> tuple[str, str]:+ """Download URL, return (html, final_url). Tries httpx first, falls back to urllib."""+ try:+ import httpx++ with httpx.Client(follow_redirects=True, timeout=30) as client:+ resp = client.get(url, headers={"User-Agent": "hyperresearch/0.1"})+ resp.raise_for_status()+ return resp.text, str(resp.url)+ except ImportError:+ pass++ import urllib.request++ req = urllib.request.Request(url, headers={"User-Agent": "hyperresearch/0.1"})+ with urllib.request.urlopen(req, timeout=30) as resp:+ html = resp.read().decode("utf-8", errors="replace")+ return html, resp.url or url++ def _extract(self, html: str) -> tuple[str, str]:+ """Extract title and clean text from HTML. Tries bs4 first, falls back to stdlib."""+ try:+ from bs4 import BeautifulSoup++ soup = BeautifulSoup(html, "html.parser")+ # Remove noise+ for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):+ tag.decompose()+ title = soup.title.string.strip() if soup.title and soup.title.string else ""+ text = soup.get_text(separator="\n", strip=True)+ # Collapse blank lines+ text = re.sub(r"\n{3,}", "\n\n", text)+ return title, text+ except ImportError:+ pass++ # Fallback: stdlib HTML parser+ parser = _TextExtractor()+ parser.feed(html)+ return parser._title, parser.get_text()diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.pynew file mode 100644index 0000000..4779931--- /dev/null+++ b/src/hyperresearch/web/crawl4ai_provider.py@@ -0,0 +1,34 @@+"""Crawl4AI web provider — free, open-source, handles JS rendering."""++from __future__ import annotations++from datetime import datetime, timezone++from crawl4ai import CrawlerStrategy, WebCrawler++from hyperresearch.web.base import WebResult+++class Crawl4AIProvider:+ name = "crawl4ai"++ def __init__(self):+ self._crawler = WebCrawler()+ self._crawler.warmup()++ def fetch(self, url: str) -> WebResult:+ result = self._crawler.run(url=url)+ return WebResult(+ url=url,+ title=result.metadata.get("title", "") if result.metadata else "",+ content=result.markdown or result.extracted_content or "",+ raw_html=result.html,+ fetched_at=datetime.now(timezone.utc),+ metadata=result.metadata or {},+ )++ def search(self, query: str, max_results: int = 5) -> list[WebResult]:+ raise NotImplementedError(+ "crawl4ai does not support web search directly. "+ "Use tavily for search: pip install hyperresearch[tavily]"+ )diff --git a/src/hyperresearch/web/firecrawl_provider.py b/src/hyperresearch/web/firecrawl_provider.pynew file mode 100644index 0000000..a5d9188--- /dev/null+++ b/src/hyperresearch/web/firecrawl_provider.py@@ -0,0 +1,41 @@+"""Firecrawl web provider — AI-optimized scraping, returns clean markdown."""++from __future__ import annotations++import os+from datetime import datetime, timezone++from firecrawl import FirecrawlApp++from hyperresearch.web.base import WebResult+++class FirecrawlProvider:+ name = "firecrawl"++ def __init__(self, api_key: str | None = None, api_url: str | None = None):+ key = api_key or os.environ.get("FIRECRAWL_API_KEY", "")+ url = api_url or os.environ.get("FIRECRAWL_API_URL")+ kwargs = {}+ if key:+ kwargs["api_key"] = key+ if url:+ kwargs["api_url"] = url+ self._app = FirecrawlApp(**kwargs)++ def fetch(self, url: str) -> WebResult:+ result = self._app.scrape_url(url, params={"formats": ["markdown"]})+ return WebResult(+ url=url,+ title=result.get("metadata", {}).get("title", ""),+ content=result.get("markdown", ""),+ raw_html=result.get("html"),+ fetched_at=datetime.now(timezone.utc),+ metadata=result.get("metadata", {}),+ )++ def search(self, query: str, max_results: int = 5) -> list[WebResult]:+ raise NotImplementedError(+ "Firecrawl does not support web search directly. "+ "Use tavily for search: pip install hyperresearch[tavily]"+ )diff --git a/src/hyperresearch/web/tavily_provider.py b/src/hyperresearch/web/tavily_provider.pynew file mode 100644index 0000000..56dea05--- /dev/null+++ b/src/hyperresearch/web/tavily_provider.py@@ -0,0 +1,55 @@+"""Tavily web provider — search + extract API for LLM agents."""++from __future__ import annotations++import os+from datetime import datetime, timezone++from tavily import TavilyClient++from hyperresearch.web.base import WebResult+++class TavilyProvider:+ name = "tavily"++ def __init__(self, api_key: str | None = None):+ key = api_key or os.environ.get("TAVILY_API_KEY", "")+ if not key:+ raise ValueError(+ "Tavily API key required. Set TAVILY_API_KEY env var "+ "or configure in .hyperresearch/config.toml"+ )+ self._client = TavilyClient(api_key=key)++ def fetch(self, url: str) -> WebResult:+ result = self._client.extract(urls=[url])+ if result and result.get("results"):+ r = result["results"][0]+ return WebResult(+ url=url,+ title=r.get("title", ""),+ content=r.get("raw_content", r.get("text", "")),+ fetched_at=datetime.now(timezone.utc),+ metadata={"source": "tavily-extract"},+ )+ raise RuntimeError(f"Tavily extract returned no results for {url}")++ def search(self, query: str, max_results: int = 5) -> list[WebResult]:+ response = self._client.search(+ query=query,+ max_results=max_results,+ include_raw_content=True,+ )+ results = []+ for r in response.get("results", []):+ results.append(+ WebResult(+ url=r["url"],+ title=r.get("title", ""),+ content=r.get("raw_content", r.get("content", "")),+ fetched_at=datetime.now(timezone.utc),+ metadata={"score": r.get("score", 0), "source": "tavily-search"},+ )+ )+ return resultsdiff --git a/src/hyperresearch/web/trafilatura_provider.py b/src/hyperresearch/web/trafilatura_provider.pynew file mode 100644index 0000000..74fe277--- /dev/null+++ b/src/hyperresearch/web/trafilatura_provider.py@@ -0,0 +1,49 @@+"""Trafilatura web provider — lightweight text extraction, no JS rendering."""++from __future__ import annotations++from datetime import datetime, timezone++import trafilatura++from hyperresearch.web.base import WebResult+++class TrafilaturaProvider:+ name = "trafilatura"++ def fetch(self, url: str) -> WebResult:+ downloaded = trafilatura.fetch_url(url)+ if downloaded is None:+ raise RuntimeError(f"Failed to download {url}")++ content = trafilatura.extract(+ downloaded,+ output_format="txt",+ include_links=True,+ include_tables=True,+ ) or ""++ metadata = trafilatura.extract_metadata(downloaded)+ title = metadata.title if metadata and metadata.title else ""+ meta_dict = {}+ if metadata:+ if metadata.author:+ meta_dict["author"] = metadata.author+ if metadata.date:+ meta_dict["date"] = metadata.date++ return WebResult(+ url=url,+ title=title,+ content=content,+ raw_html=downloaded,+ fetched_at=datetime.now(timezone.utc),+ metadata=meta_dict,+ )++ def search(self, query: str, max_results: int = 5) -> list[WebResult]:+ raise NotImplementedError(+ "Trafilatura does not support web search. "+ "Use tavily for search: pip install hyperresearch[tavily]"+ )