Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

Add web providers, fetch command, agent hooks, sources table

Jordan3cb6459ff21f
diff --git a/PLAN.md b/PLAN.md
new file mode 100644
index 0000000..2e2e046
--- /dev/null
+++ b/PLAN.md
@@ -0,0 +1,330 @@
+# Hyperresearch: Agent-Driven Research Knowledge Base
+
+## Context
+
+Kasten is a write-side knowledge base tool — agents/humans manually create notes. The gap in the ecosystem is a tool that lets an agent **actively collect, browse, and synthesize research** into a persistent, searchable knowledge base. Think: "kasten backbone + web browsing + slash-command UX like Graphify."
+
+The user wants a new repo, new CLI name (`hyperresearch`), same core engine (markdown + YAML frontmatter + SQLite FTS5 + wiki-links + JSON output), but repositioned for **research workflows** where agents crawl sources, extract findings, and build a knowledge graph over time.
+
+## What stays from kasten (the backbone)
+
+These modules port over nearly verbatim:
+
+| Module | What it does | Changes needed |
+|--------|-------------|----------------|
+| `core/vault.py` | Vault init, discovery, path properties | Rename dirs (`research/` instead of `knowledge/`?), rebrand |
+| `core/sync.py` | mtime + SHA-256 sync, link resolution | None — same engine |
+| `core/db.py` | SQLite schema, FTS5, WAL mode | Add `sources` table for URL tracking |
+| `core/note.py` | read/write markdown with frontmatter | Add source-aware fields |
+| `core/frontmatter.py` | YAML parse/serialize | None |
+| `core/config.py` | TOML config | New defaults, new sections for web providers |
+| `core/patterns.py` | Regex (wiki-links, code blocks) | None |
+| `core/agent_docs.py` | Inject into CLAUDE.md etc | Rewrite blurb for research workflow |
+| `models/` | Note, Envelope, SearchResult | Extend with Source model |
+| `search/` | FTS5 + filters | Add source/URL filters |
+| `serve/` | Web viewer | Rebrand theme |
+| `mcp/` | MCP server | Add research-specific tools |
+
+## What's new
+
+### 1. Web provider plugin system
+
+**Architecture:** Abstract `WebProvider` protocol, concrete implementations as optional deps.
+
+```python
+# src/hyperresearch/web/base.py
+class WebProvider(Protocol):
+ def fetch(self, url: str) -> WebResult: ...
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]: ...
+
+@dataclass
+class WebResult:
+ url: str
+ title: str
+ content: str # clean markdown
+ raw_html: str | None
+ fetched_at: datetime
+ metadata: dict # author, date, domain, etc.
+```
+
+**Provider implementations (each an optional dep group):**
+
+| Provider | Install | Cost | Best for |
+|----------|---------|------|----------|
+| `crawl4ai` | `pip install hyperresearch[crawl4ai]` | Free, open-source | Default. 60k stars, agentic crawling, JS rendering |
+| `tavily` | `pip install hyperresearch[tavily]` | $0.008/credit, 1k free/mo | Search-first workflows (API key required) |
+| `firecrawl` | `pip install hyperresearch[firecrawl]` | Free self-hosted or paid API | Best markdown output, site-wide crawling |
+| `trafilatura` | `pip install hyperresearch[trafilatura]` | Free | Lightweight text extraction, no JS |
+| builtin | (included) | Free | `httpx` + `beautifulsoup4` fallback, no JS rendering |
+
+**Config in `.hyperresearch/config.toml`:**
+```toml
+[web]
+provider = "crawl4ai" # or "tavily", "firecrawl", "trafilatura", "builtin"
+tavily_api_key = "" # or env var TAVILY_API_KEY
+firecrawl_api_key = "" # or env var FIRECRAWL_API_KEY
+```
+
+### 2. New CLI commands (research-focused)
+
+```bash
+# Fetch a URL, extract content, save as a note
+hyperresearch fetch "https://arxiv.org/abs/2401.12345" --tag transformers --tag attention
+
+# Search the web, save top results as notes
+hyperresearch websearch "state space models 2026" --max 5 --tag ssm
+
+# Deep research: search → fetch → follow links → extract → save
+hyperresearch research "mamba architecture" --depth 2 --max-pages 10
+
+# All the kasten stuff still works
+hyperresearch search "attention" -j
+hyperresearch note show attention-is-all-you-need -j
+hyperresearch repair -j
+hyperresearch sync
+```
+
+**Command details:**
+
+- **`fetch <url>`** — Single URL → note. Extracts clean markdown via provider, creates note with `source: <url>`, auto-tags if possible, records fetch metadata. Deduplicates by URL (warns if already fetched).
+
+- **`websearch <query>`** — Uses provider's search API → creates one note per result. Each note links back to source URL. Returns JSON list of created note IDs.
+
+- **`research <topic>`** — The flagship command. Agent-driven deep research:
+ 1. Web search for topic
+ 2. Fetch top N results
+ 3. Follow outbound links up to `--depth` levels
+ 4. Extract and save as notes with wiki-links between related findings
+ 5. Generate a synthesis note (map-of-content) linking everything found
+
+ This is the command that makes it "fancy" — it's what agents invoke to build a knowledge base from scratch on a topic.
+
+### 3. Slash-command / hook integration (like Graphify)
+
+**This is the key adoption play.** Install hyperresearch, run one command, and every agent in the repo knows about it.
+
+```bash
+pip install hyperresearch
+hyperresearch install # sets up hooks + agent docs
+```
+
+**`hyperresearch install` does:**
+
+1. Creates `.hyperresearch/` directory (vault init)
+2. Injects research-focused blurb into agent config files
+3. **Installs PreToolUse hooks** for supported platforms:
+
+| Platform | Hook location | Trigger | What it does |
+|----------|--------------|---------|--------------|
+| Claude Code | `.claude/settings.json` | Before `Glob`, `Grep`, `WebSearch`, `WebFetch` | "Check hyperresearch KB first. Run `hyperresearch search <topic> -j` before raw web searches." |
+| Codex | `.codex/hooks.json` | Before `Bash` | Same reminder |
+| Cursor | `.cursor/rules/hyperresearch.mdc` | `alwaysApply: true` | System rule with research workflow |
+| Gemini CLI | `.gemini/settings.json` | `BeforeTool` | Same pattern |
+
+The hook script (a small JS or Python file at `.hyperresearch/hook.js`):
+- Checks if a research KB exists
+- If yes, reminds the agent to `hyperresearch search` before going to the web
+- Prevents redundant web fetches for topics already researched
+
+### 4. Source tracking (new DB table)
+
+```sql
+CREATE TABLE sources (
+ url TEXT PRIMARY KEY,
+ note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,
+ domain TEXT,
+ fetched_at TEXT,
+ provider TEXT, -- which web provider fetched it
+ content_hash TEXT, -- detect if page changed
+ status TEXT DEFAULT 'active', -- active, dead, redirected
+ CHECK (status IN ('active', 'dead', 'redirected'))
+);
+CREATE INDEX idx_sources_domain ON sources(domain);
+CREATE INDEX idx_sources_note ON sources(note_id);
+```
+
+This lets agents ask: "have we already fetched this URL?" and "what sources did we use for this topic?"
+
+### 5. Extended frontmatter for research notes
+
+```yaml
+---
+title: "Mamba: Linear-Time Sequence Modeling"
+id: mamba-linear-time-sequence-modeling
+tags: [ssm, mamba, sequence-modeling]
+status: review
+source: https://arxiv.org/abs/2312.00752
+source_domain: arxiv.org
+fetched_at: 2026-04-09T12:00:00Z
+fetch_provider: crawl4ai
+confidence: 0.85
+summary: "Mamba introduces selective state spaces that achieve transformer-quality with linear scaling."
+---
+```
+
+The `source`, `source_domain`, `fetched_at`, `fetch_provider` fields are new. The rest already exists in kasten's NoteMeta.
+
+## Project structure
+
+```
+hyperresearch/
+├── pyproject.toml
+├── src/hyperresearch/
+│ ├── __init__.py
+│ ├── __main__.py
+│ ├── cli/ # [from kasten] + new commands
+│ │ ├── __init__.py # App assembly
+│ │ ├── main.py # init, status, sync, install
+│ │ ├── search.py # [from kasten]
+│ │ ├── note.py # [from kasten]
+│ │ ├── repair.py # [from kasten]
+│ │ ├── fetch.py # NEW: fetch, websearch commands
+│ │ ├── research.py # NEW: deep research command
+│ │ ├── sources.py # NEW: source management
+│ │ ├── graph.py # [from kasten]
+│ │ ├── lint.py # [from kasten]
+│ │ ├── tag.py # [from kasten]
+│ │ └── ...
+│ ├── core/ # [from kasten, nearly verbatim]
+│ │ ├── vault.py
+│ │ ├── sync.py
+│ │ ├── db.py # + sources table
+│ │ ├── note.py
+│ │ ├── config.py # + web provider config
+│ │ ├── agent_docs.py # rewritten blurb
+│ │ ├── hooks.py # NEW: PreToolUse hook installer
+│ │ └── ...
+│ ├── web/ # NEW: web provider system
+│ │ ├── __init__.py
+│ │ ├── base.py # WebProvider protocol + WebResult
+│ │ ├── builtin.py # httpx + bs4 fallback
+│ │ ├── crawl4ai_provider.py
+│ │ ├── tavily_provider.py
+│ │ ├── firecrawl_provider.py
+│ │ └── trafilatura_provider.py
+│ ├── models/ # [from kasten] + Source model
+│ ├── search/ # [from kasten]
+│ ├── serve/ # [from kasten, rebranded]
+│ ├── mcp/ # [from kasten] + research tools
+│ └── indexgen/ # [from kasten]
+├── tests/
+├── README.md
+├── CLAUDE.md
+└── .github/workflows/
+```
+
+## Implementation phases
+
+### Phase 1: Fork and rebrand — NOW
+**Target:** `C:\Users\Jordan\PycharmProjects\hyperresearch` (PyCharm project already created)
+
+1. Copy kasten's `src/kasten/` → `src/hyperresearch/` (all modules)
+2. Copy `tests/`, `pyproject.toml`, `LICENSE`, `CONTRIBUTING.md`
+3. Global rename in all files: `kasten` → `hyperresearch`, `kas` → `hpr` (alias)
+4. Rename `knowledge/` → `research/` in vault defaults
+5. Rename `.kasten/` → `.hyperresearch/` in vault discovery
+6. Update pyproject.toml: name, entry points, description, add httpx + bs4 to core deps
+7. Verify all tests pass with new names
+8. `pip install -e ".[dev]"` works
+9. `git init` + initial commit
+
+### Phase 2: Web provider system (day 2-3)
+1. Define `WebProvider` protocol in `web/base.py`
+2. Implement `builtin` provider (httpx + bs4, always available)
+3. Implement `crawl4ai` provider (optional dep)
+4. Add `[web]` section to config.toml
+5. Add `sources` table to DB schema
+6. Wire up provider selection in config
+
+### Phase 3: fetch + websearch commands (day 3-4)
+1. `hyperresearch fetch <url>` — single URL to note
+2. `hyperresearch websearch <query>` — search and save results
+3. URL deduplication via sources table
+4. Source metadata in frontmatter
+5. Tests for both commands
+
+### Phase 4: Hook system + install command (day 4-5)
+1. `hyperresearch install` command
+2. Claude Code PreToolUse hook (`.claude/settings.json`)
+3. Cursor rule (`.cursor/rules/hyperresearch.mdc`)
+4. Codex hook (`.codex/hooks.json`)
+5. Gemini hook (`.gemini/settings.json`)
+6. Rewrite agent docs blurb for research workflow
+7. Hook reminder script (JS or Python)
+
+### Phase 5: research command (day 5-7)
+1. Orchestrates websearch → fetch → follow links → save
+2. Depth-limited crawling
+3. Auto wiki-linking between fetched notes
+4. Synthesis note generation (map-of-content)
+5. This is the flagship "demo" command
+
+### Phase 6: Additional providers + polish (day 7-8)
+1. Tavily provider
+2. Firecrawl provider
+3. Trafilatura provider
+4. MCP server with research-specific tools (fetch, websearch, sources)
+5. README, demo GIF, PyPI publish
+
+## pyproject.toml sketch
+
+```toml
+[project]
+name = "hyperresearch"
+version = "0.1.0"
+description = "Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki."
+requires-python = ">=3.11"
+dependencies = [
+ "typer>=0.9.0",
+ "rich>=13.0",
+ "pyyaml>=6.0",
+ "pydantic>=2.0",
+ "jinja2>=3.1",
+ "platformdirs>=4.0",
+]
+
+# httpx + bs4 NOT core — agents already have WebSearch/WebFetch
+
+[project.optional-dependencies]
+crawl4ai = ["Crawl4AI>=0.4"]
+tavily = ["tavily-python>=0.5"]
+firecrawl = ["firecrawl>=1.0"]
+trafilatura = ["trafilatura>=2.0"]
+mcp = ["mcp>=1.6"]
+all = ["hyperresearch[crawl4ai,tavily,firecrawl,trafilatura,mcp]"]
+dev = ["pytest", "pytest-cov", "ruff", "mypy"]
+
+[project.scripts]
+hyperresearch = "hyperresearch.cli:app"
+hpr = "hyperresearch.cli:app"
+```
+
+## Key design decisions
+
+1. **httpx + bs4 are core deps, not optional** — the builtin provider should always work without extras. This is the "it just works" baseline.
+
+2. **crawl4ai is the recommended default** — free, open-source, 60k stars, handles JS rendering. But it's optional because it pulls in heavy deps (playwright/chromium).
+
+3. **`install` not `init`** — Graphify uses `install` for the hook setup. We use `install` to do vault init + hook installation in one step. `init` still works as an alias for just the vault.
+
+4. **Hooks remind, not enforce** — the PreToolUse hook says "check hyperresearch first" but doesn't block the agent. Blocking would be annoying.
+
+5. **Notes are still files** — same as kasten philosophy. The web provider fetches content, but it gets saved as a .md file with frontmatter. Files are truth, DB is cache.
+
+6. **No LLM calls in the tool itself** — hyperresearch doesn't call Claude/GPT to summarize or synthesize. The AGENT using hyperresearch does that. hyperresearch is still a "dumb tool" — it stores, indexes, and searches. The agent decides what to do with it.
+
+7. **Agents already have web access** — Claude Code has WebSearch/WebFetch, Codex has browsing, etc. So the DEFAULT flow is: agent fetches content with its own tools → pipes it into hyperresearch via `--body-file` or `--body-stdin`. The optional web providers (crawl4ai, tavily, firecrawl) are extras for standalone/headless use, NOT required.
+
+## Verification
+
+1. `pip install -e ".[dev]"` — core install works
+2. `hyperresearch install` — creates vault + hooks
+3. `hyperresearch fetch "https://en.wikipedia.org/wiki/Zettelkasten"` — fetches and saves note
+4. `hyperresearch search "zettelkasten" -j` — finds the fetched note
+5. `hyperresearch websearch "state space models" --max 3` — searches and saves 3 notes
+6. `python -m pytest tests/` — all tests pass
+7. Open a Claude Code session in the repo — agent sees the hook reminder, uses `hyperresearch search` before web searches
+
+## Name availability check needed
+- PyPI: check if `hyperresearch` is taken
+- GitHub: check if `hyperresearch` repo name is available
diff --git a/src/hyperresearch/cli/__init__.py b/src/hyperresearch/cli/__init__.py
index e3aaa06..2069b7b 100644
--- a/src/hyperresearch/cli/__init__.py
+++ b/src/hyperresearch/cli/__init__.py
@@ -38,12 +38,16 @@ from hyperresearch.cli.repair import repair as _repair
from hyperresearch.cli.search import search as _search
from hyperresearch.cli.serve import serve as _serve
from hyperresearch.cli.tag import tag_list as _tags
+from hyperresearch.cli.fetch import fetch as _fetch
+from hyperresearch.cli.install import install as _install
from hyperresearch.cli.watch import watch as _watch
+app.command("install")(_install)
app.command("init")(_init)
app.command("status")(_status)
app.command("sync")(_sync)
app.command("search")(_search)
+app.command("fetch")(_fetch)
app.command("tags")(_tags)
app.command("show", hidden=True)(_show)
app.command("dedup")(_dedup)
diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.py
new file mode 100644
index 0000000..5c5636e
--- /dev/null
+++ b/src/hyperresearch/cli/fetch.py
@@ -0,0 +1,128 @@
+"""Fetch and websearch commands — save web content as research notes."""
+
+from __future__ import annotations
+
+import hashlib
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+from urllib.parse import urlparse
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def fetch(
+ url: str = typer.Argument(..., help="URL to fetch and save as a note"),
+ tags: list[str] = typer.Option([], "--tag", "-t", help="Tags (repeatable)"),
+ title: str | None = typer.Option(None, "--title", help="Override title"),
+ parent: str | None = typer.Option(None, "--parent", "-p", help="Parent topic"),
+ provider_name: str | None = typer.Option(None, "--provider", help="Web provider override"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Fetch a URL and save its content as a research note."""
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+ from hyperresearch.core.vault import Vault, VaultError
+ from hyperresearch.web.base import get_provider
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ conn = vault.db
+
+ # Check if URL already fetched
+ existing = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()
+ if existing:
+ note_id = existing["note_id"]
+ if json_output:
+ output(
+ error(f"URL already fetched as note '{note_id}'", "DUPLICATE_URL"),
+ json_mode=True,
+ )
+ else:
+ console.print(f"[yellow]Already fetched:[/] {url} → note '{note_id}'")
+ raise typer.Exit(1)
+
+ # Fetch content
+ prov = get_provider(provider_name or vault.config.web_provider)
+ if not json_output:
+ console.print(f"[dim]Fetching with {prov.name}...[/]")
+
+ try:
+ result = prov.fetch(url)
+ except Exception as e:
+ if json_output:
+ output(error(str(e), "FETCH_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Fetch failed:[/] {e}")
+ raise typer.Exit(1)
+
+ # Write note
+ note_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"
+ domain = result.domain
+
+ extra_meta = {
+ "source": url,
+ "source_domain": domain,
+ "fetched_at": result.fetched_at.isoformat(),
+ "fetch_provider": prov.name,
+ }
+ if result.metadata.get("author"):
+ extra_meta["author"] = result.metadata["author"]
+
+ note_path = write_note(
+ vault.notes_dir,
+ title=note_title,
+ body=result.content,
+ tags=tags,
+ status="draft",
+ source=url,
+ parent=parent,
+ extra_frontmatter=extra_meta,
+ )
+
+ # Record source in DB
+ content_hash = hashlib.sha256(result.content.encode("utf-8")).hexdigest()[:16]
+ note_id = note_path.stem
+ conn.execute(
+ """INSERT INTO sources (url, note_id, domain, fetched_at, provider, content_hash)
+ VALUES (?, ?, ?, ?, ?, ?)""",
+ (url, note_id, domain, result.fetched_at.isoformat(), prov.name, content_hash),
+ )
+ conn.commit()
+
+ # Sync to index the new note
+ plan = compute_sync_plan(vault)
+ if plan.to_add or plan.to_update:
+ execute_sync(vault, plan)
+
+ data = {
+ "note_id": note_id,
+ "title": note_title,
+ "url": url,
+ "domain": domain,
+ "provider": prov.name,
+ "path": str(note_path.relative_to(vault.root)),
+ "word_count": len(result.content.split()),
+ }
+
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]Saved:[/] {note_title}")
+ console.print(f" ID: {note_id}")
+ console.print(f" Source: {url}")
+ console.print(f" Words: {data['word_count']}")
diff --git a/src/hyperresearch/cli/install.py b/src/hyperresearch/cli/install.py
new file mode 100644
index 0000000..0e90bce
--- /dev/null
+++ b/src/hyperresearch/cli/install.py
@@ -0,0 +1,77 @@
+"""Install command — one-step setup: vault init + agent hooks + docs injection."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+
+def install(
+ path: str = typer.Argument(".", help="Path to install in"),
+ name: str = typer.Option("Research Base", "--name", "-n", help="Vault name"),
+ platforms: list[str] = typer.Option(
+ ["claude"],
+ "--platform",
+ "-p",
+ help="Agent platforms to hook: claude|codex|cursor|gemini|all",
+ ),
+ agents: list[str] = typer.Option(
+ ["claude"],
+ "--agents",
+ "-a",
+ help="Agent doc files: claude|agents|gemini|copilot",
+ ),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Install hyperresearch: init vault + inject agent docs + install hooks."""
+ from hyperresearch.core.hooks import install_hooks
+ from hyperresearch.core.vault import Vault, VaultError
+
+ root = Path(path).resolve()
+
+ # Step 1: Init vault (skip if already exists)
+ try:
+ vault = Vault.discover(root)
+ vault_action = "existing"
+ except VaultError:
+ try:
+ vault = Vault.init(root, name=name, agents=agents)
+ vault_action = "created"
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "INIT_ERROR"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ # Step 2: Install hooks
+ hook_actions = install_hooks(root, platforms=platforms)
+
+ # Step 3: Report
+ data = {
+ "vault_path": str(vault.root),
+ "vault": vault_action,
+ "hooks_installed": hook_actions,
+ "platforms": platforms,
+ }
+
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ if vault_action == "created":
+ console.print(f"[green]Vault created:[/] {vault.root}")
+ else:
+ console.print(f"[dim]Vault exists:[/] {vault.root}")
+
+ if hook_actions:
+ console.print("[green]Hooks installed:[/]")
+ for action in hook_actions:
+ console.print(f" {action}")
+ else:
+ console.print("[dim]All hooks already installed.[/]")
+
+ console.print("\n[bold]Ready.[/] Agents will now check the research base before web searches.")
diff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.py
index cdcaf14..8d530f1 100644
--- a/src/hyperresearch/core/config.py
+++ b/src/hyperresearch/core/config.py
@@ -31,6 +31,9 @@ class VaultConfig:
]
)
+ # Web provider
+ web_provider: str = "builtin"
+
# Index
auto_build_index: bool = True
index_pages: list[str] = field(
@@ -48,6 +51,7 @@ class VaultConfig:
sync = data.get("sync", {})
index = data.get("index", {})
search = data.get("search", {})
+ web = data.get("web", {})
return cls(
name=vault.get("name", cls.name),
@@ -60,6 +64,7 @@ class VaultConfig:
search_boost_evergreen=search.get("boost_evergreen", cls.search_boost_evergreen),
search_penalize_deprecated=search.get("penalize_deprecated", cls.search_penalize_deprecated),
search_penalize_stale=search.get("penalize_stale", cls.search_penalize_stale),
+ web_provider=web.get("provider", cls.web_provider),
auto_sync=sync.get("auto_sync", cls.auto_sync),
exclude_patterns=sync.get("exclude_patterns", cls().exclude_patterns),
auto_build_index=index.get("auto_build", cls.auto_build_index),
@@ -87,6 +92,9 @@ class VaultConfig:
f"penalize_deprecated = {self.search_penalize_deprecated}",
f"penalize_stale = {self.search_penalize_stale}",
"",
+ "[web]",
+ f'provider = "{self.web_provider}"',
+ "",
"[sync]",
f"auto_sync = {'true' if self.auto_sync else 'false'}",
f"exclude_patterns = {self._toml_array(self.exclude_patterns)}",
diff --git a/src/hyperresearch/core/db.py b/src/hyperresearch/core/db.py
index e122dd4..ff4c0a9 100644
--- a/src/hyperresearch/core/db.py
+++ b/src/hyperresearch/core/db.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import sqlite3
from pathlib import Path
-SCHEMA_VERSION = 2
+SCHEMA_VERSION = 3
SCHEMA_SQL = """
PRAGMA journal_mode=WAL;
@@ -99,6 +99,20 @@ CREATE TABLE IF NOT EXISTS tag_aliases (
canonical TEXT NOT NULL
);
+CREATE TABLE IF NOT EXISTS sources (
+ url TEXT PRIMARY KEY,
+ note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,
+ domain TEXT,
+ fetched_at TEXT,
+ provider TEXT,
+ content_hash TEXT,
+ status TEXT NOT NULL DEFAULT 'active'
+ CHECK (status IN ('active', 'dead', 'redirected'))
+);
+
+CREATE INDEX IF NOT EXISTS idx_sources_domain ON sources(domain);
+CREATE INDEX IF NOT EXISTS idx_sources_note ON sources(note_id);
+
"""
FTS_SQL = """
diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.py
new file mode 100644
index 0000000..b6a8016
--- /dev/null
+++ b/src/hyperresearch/core/hooks.py
@@ -0,0 +1,220 @@
+"""Agent hook installer — injects PreToolUse hooks for Claude Code, Codex, Cursor, Gemini CLI.
+
+These hooks remind agents to check the research base before doing raw web searches.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+# The hook script that gets installed
+HOOK_SCRIPT = """\
+#!/usr/bin/env node
+/**
+ * hyperresearch PreToolUse hook — reminds agent to check research base first.
+ * Installed by: hyperresearch install
+ */
+const fs = require('fs');
+const path = require('path');
+
+// Check if a .hyperresearch directory exists (vault is initialized)
+function findVault() {
+ let dir = process.env.CLAUDE_PROJECT_DIR || process.cwd();
+ while (true) {
+ if (fs.existsSync(path.join(dir, '.hyperresearch'))) return dir;
+ const parent = path.dirname(dir);
+ if (parent === dir) return null;
+ dir = parent;
+ }
+}
+
+const vault = findVault();
+if (vault) {
+ const msg = [
+ 'HYPERRESEARCH: A research knowledge base exists in this project.',
+ 'Before searching the web, check existing research:',
+ ' hyperresearch search "<your query>" -j',
+ 'Save useful findings back:',
+ ' hyperresearch note new "Title" --tag t --body-file /tmp/content.md --source "url" -j',
+ ].join('\\n');
+ process.stderr.write(msg + '\\n');
+}
+"""
+
+# Cursor rule file content
+CURSOR_RULE = """\
+---
+description: Hyperresearch research base integration
+alwaysApply: true
+---
+
+# Research Base (hyperresearch)
+
+This project has a hyperresearch research base. Before searching the web or browsing URLs:
+
+1. **Check existing research first**: `hyperresearch search "<query>" -j`
+2. **Save useful findings**: `hyperresearch note new "Title" --tag t --body-file /tmp/content.md --source "url" -j`
+3. **Browse the knowledge graph**: `hyperresearch graph hubs -j` for key topics
+
+The research base persists across sessions. Don't duplicate work — check before fetching.
+"""
+
+
+def install_hooks(vault_root: Path, platforms: list[str] | None = None) -> list[str]:
+ """Install agent hooks for specified platforms. Returns list of actions taken."""
+ if platforms is None:
+ platforms = ["claude"]
+
+ actions = []
+
+ if "claude" in platforms or "all" in platforms:
+ result = _install_claude_hook(vault_root)
+ if result:
+ actions.append(result)
+
+ if "codex" in platforms or "all" in platforms:
+ result = _install_codex_hook(vault_root)
+ if result:
+ actions.append(result)
+
+ if "cursor" in platforms or "all" in platforms:
+ result = _install_cursor_rule(vault_root)
+ if result:
+ actions.append(result)
+
+ if "gemini" in platforms or "all" in platforms:
+ result = _install_gemini_hook(vault_root)
+ if result:
+ actions.append(result)
+
+ return actions
+
+
+def _write_hook_script(vault_root: Path) -> Path:
+ """Write the hook JS script to .hyperresearch/hook.js."""
+ hook_dir = vault_root / ".hyperresearch"
+ hook_dir.mkdir(parents=True, exist_ok=True)
+ hook_path = hook_dir / "hook.js"
+ hook_path.write_text(HOOK_SCRIPT, encoding="utf-8")
+ return hook_path
+
+
+def _install_claude_hook(vault_root: Path) -> str | None:
+ """Install PreToolUse hook into .claude/settings.json."""
+ hook_path = _write_hook_script(vault_root)
+
+ settings_dir = vault_root / ".claude"
+ settings_dir.mkdir(exist_ok=True)
+ settings_path = settings_dir / "settings.json"
+
+ settings = {}
+ if settings_path.exists():
+ try:
+ settings = json.loads(settings_path.read_text(encoding="utf-8"))
+ except (json.JSONDecodeError, OSError):
+ pass
+
+ hooks = settings.setdefault("hooks", {})
+ pre_tool = hooks.setdefault("PreToolUse", [])
+
+ # Check if already installed
+ for entry in pre_tool:
+ if isinstance(entry, dict):
+ for h in entry.get("hooks", []):
+ if "hyperresearch" in h.get("command", ""):
+ return None # Already installed
+
+ # Add hook that fires before web-related tools
+ pre_tool.append({
+ "matcher": "Glob|Grep|WebSearch|WebFetch",
+ "hooks": [{
+ "type": "command",
+ "command": f"node {hook_path.as_posix()}",
+ }],
+ })
+
+ settings_path.write_text(json.dumps(settings, indent=2) + "\n", encoding="utf-8")
+ return "Claude Code: .claude/settings.json (PreToolUse hook)"
+
+
+def _install_codex_hook(vault_root: Path) -> str | None:
+ """Install hook into .codex/hooks.json."""
+ hook_path = _write_hook_script(vault_root)
+
+ codex_dir = vault_root / ".codex"
+ codex_dir.mkdir(exist_ok=True)
+ hooks_path = codex_dir / "hooks.json"
+
+ hooks = {}
+ if hooks_path.exists():
+ try:
+ hooks = json.loads(hooks_path.read_text(encoding="utf-8"))
+ except (json.JSONDecodeError, OSError):
+ pass
+
+ pre_tool = hooks.setdefault("PreToolUse", [])
+ for entry in pre_tool:
+ if isinstance(entry, dict):
+ for h in entry.get("hooks", []):
+ if "hyperresearch" in h.get("command", ""):
+ return None
+
+ pre_tool.append({
+ "matcher": "Bash",
+ "hooks": [{
+ "type": "command",
+ "command": f"node {hook_path.as_posix()}",
+ }],
+ })
+
+ hooks_path.write_text(json.dumps(hooks, indent=2) + "\n", encoding="utf-8")
+ return "Codex: .codex/hooks.json (PreToolUse hook)"
+
+
+def _install_cursor_rule(vault_root: Path) -> str | None:
+ """Install always-apply rule into .cursor/rules/hyperresearch.mdc."""
+ rules_dir = vault_root / ".cursor" / "rules"
+ rules_dir.mkdir(parents=True, exist_ok=True)
+ rule_path = rules_dir / "hyperresearch.mdc"
+
+ if rule_path.exists():
+ return None # Already exists
+
+ rule_path.write_text(CURSOR_RULE, encoding="utf-8")
+ return "Cursor: .cursor/rules/hyperresearch.mdc (alwaysApply rule)"
+
+
+def _install_gemini_hook(vault_root: Path) -> str | None:
+ """Install BeforeTool hook into .gemini/settings.json."""
+ hook_path = _write_hook_script(vault_root)
+
+ gemini_dir = vault_root / ".gemini"
+ gemini_dir.mkdir(exist_ok=True)
+ settings_path = gemini_dir / "settings.json"
+
+ settings = {}
+ if settings_path.exists():
+ try:
+ settings = json.loads(settings_path.read_text(encoding="utf-8"))
+ except (json.JSONDecodeError, OSError):
+ pass
+
+ hooks = settings.setdefault("hooks", {})
+ before_tool = hooks.setdefault("BeforeTool", [])
+
+ for entry in before_tool:
+ if isinstance(entry, dict):
+ for h in entry.get("hooks", []):
+ if "hyperresearch" in h.get("command", ""):
+ return None
+
+ before_tool.append({
+ "hooks": [{
+ "type": "command",
+ "command": f"node {hook_path.as_posix()}",
+ }],
+ })
+
+ settings_path.write_text(json.dumps(settings, indent=2) + "\n", encoding="utf-8")
+ return "Gemini CLI: .gemini/settings.json (BeforeTool hook)"
diff --git a/src/hyperresearch/core/migrations.py b/src/hyperresearch/core/migrations.py
index 1748f25..8a9156d 100644
--- a/src/hyperresearch/core/migrations.py
+++ b/src/hyperresearch/core/migrations.py
@@ -12,6 +12,20 @@ CREATE TABLE IF NOT EXISTS tag_aliases (
alias TEXT PRIMARY KEY,
canonical TEXT NOT NULL
);
+""",
+ 3: """
+CREATE TABLE IF NOT EXISTS sources (
+ url TEXT PRIMARY KEY,
+ note_id TEXT REFERENCES notes(id) ON DELETE SET NULL,
+ domain TEXT,
+ fetched_at TEXT,
+ provider TEXT,
+ content_hash TEXT,
+ status TEXT NOT NULL DEFAULT 'active'
+ CHECK (status IN ('active', 'dead', 'redirected'))
+);
+CREATE INDEX IF NOT EXISTS idx_sources_domain ON sources(domain);
+CREATE INDEX IF NOT EXISTS idx_sources_note ON sources(note_id);
""",
}
diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.py
index e3fc684..b9abd32 100644
--- a/src/hyperresearch/core/note.py
+++ b/src/hyperresearch/core/note.py
@@ -59,14 +59,16 @@ def write_note(
parent: str | None = None,
source: str | None = None,
summary: str | None = None,
+ extra_frontmatter: dict | None = None,
) -> Path:
"""Create a new note file on disk. Returns the file path.
Args:
notes_dir: The directory to write into (e.g. vault.notes_dir).
+ extra_frontmatter: Additional fields to set on NoteMeta (e.g. source_domain, fetched_at).
"""
nid = note_id or slugify(title)
- meta = NoteMeta(
+ kwargs: dict = dict(
title=title,
id=nid,
tags=tags or [],
@@ -77,6 +79,9 @@ def write_note(
summary=summary,
created=datetime.now(UTC),
)
+ if extra_frontmatter:
+ kwargs.update(extra_frontmatter)
+ meta = NoteMeta(**kwargs)
# Determine output path, avoid collisions
target_dir = notes_dir
diff --git a/src/hyperresearch/models/note.py b/src/hyperresearch/models/note.py
index 336fa8c..b293eb7 100644
--- a/src/hyperresearch/models/note.py
+++ b/src/hyperresearch/models/note.py
@@ -48,6 +48,9 @@ class NoteMeta(BaseModel):
created: datetime = Field(default_factory=lambda: datetime.now(UTC))
updated: datetime | None = None
source: str | None = None
+ source_domain: str | None = None
+ fetched_at: datetime | None = None
+ fetch_provider: str | None = None
status: NoteStatus = NoteStatus.DRAFT
type: NoteType = NoteType.NOTE
aliases: list[str] = Field(default_factory=list)
diff --git a/src/hyperresearch/web/__init__.py b/src/hyperresearch/web/__init__.py
new file mode 100644
index 0000000..23ae07f
--- /dev/null
+++ b/src/hyperresearch/web/__init__.py
@@ -0,0 +1 @@
+"""Web provider system — pluggable backends for fetching and searching the web."""
diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.py
new file mode 100644
index 0000000..c5ffd0a
--- /dev/null
+++ b/src/hyperresearch/web/base.py
@@ -0,0 +1,88 @@
+"""Base protocol and data types for web providers."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Protocol, runtime_checkable
+
+
+@dataclass
+class WebResult:
+ """A single web fetch or search result."""
+
+ url: str
+ title: str
+ content: str # clean markdown or plain text
+ fetched_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+ raw_html: str | None = None
+ metadata: dict = field(default_factory=dict) # author, date, domain, etc.
+
+ @property
+ def domain(self) -> str:
+ from urllib.parse import urlparse
+
+ return urlparse(self.url).netloc
+
+
+@runtime_checkable
+class WebProvider(Protocol):
+ """Protocol for web content providers.
+
+ Implementations must support at least fetch(). search() is optional —
+ providers that don't support search raise NotImplementedError.
+ """
+
+ name: str
+
+ def fetch(self, url: str) -> WebResult:
+ """Fetch a single URL and return clean content."""
+ ...
+
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]:
+ """Search the web and return results with content."""
+ ...
+
+
+def get_provider(name: str | None = None) -> WebProvider:
+ """Load a web provider by name. Falls back to builtin if none specified."""
+ if name is None or name == "builtin":
+ from hyperresearch.web.builtin import BuiltinProvider
+
+ return BuiltinProvider()
+
+ if name == "crawl4ai":
+ try:
+ from hyperresearch.web.crawl4ai_provider import Crawl4AIProvider
+
+ return Crawl4AIProvider()
+ except ImportError:
+ raise ImportError("crawl4ai provider requires: pip install hyperresearch[crawl4ai]")
+
+ if name == "tavily":
+ try:
+ from hyperresearch.web.tavily_provider import TavilyProvider
+
+ return TavilyProvider()
+ except ImportError:
+ raise ImportError("tavily provider requires: pip install hyperresearch[tavily]")
+
+ if name == "firecrawl":
+ try:
+ from hyperresearch.web.firecrawl_provider import FirecrawlProvider
+
+ return FirecrawlProvider()
+ except ImportError:
+ raise ImportError("firecrawl provider requires: pip install hyperresearch[firecrawl]")
+
+ if name == "trafilatura":
+ try:
+ from hyperresearch.web.trafilatura_provider import TrafilaturaProvider
+
+ return TrafilaturaProvider()
+ except ImportError:
+ raise ImportError(
+ "trafilatura provider requires: pip install hyperresearch[trafilatura]"
+ )
+
+ raise ValueError(f"Unknown web provider: {name!r}")
diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.py
new file mode 100644
index 0000000..b1d739f
--- /dev/null
+++ b/src/hyperresearch/web/builtin.py
@@ -0,0 +1,112 @@
+"""Builtin web provider — uses httpx + beautifulsoup4 if available, else bare urllib."""
+
+from __future__ import annotations
+
+import re
+from datetime import datetime, timezone
+from html.parser import HTMLParser
+
+from hyperresearch.web.base import WebResult
+
+
+class _TextExtractor(HTMLParser):
+ """Minimal HTML-to-text extractor (no external deps)."""
+
+ def __init__(self):
+ super().__init__()
+ self._pieces: list[str] = []
+ self._skip = False
+ self._skip_tags = {"script", "style", "nav", "footer", "header"}
+ self._title = ""
+ self._in_title = False
+
+ def handle_starttag(self, tag, attrs):
+ if tag in self._skip_tags:
+ self._skip = True
+ if tag == "title":
+ self._in_title = True
+ if tag in ("p", "br", "div", "h1", "h2", "h3", "h4", "h5", "h6", "li"):
+ self._pieces.append("\n")
+
+ def handle_endtag(self, tag):
+ if tag in self._skip_tags:
+ self._skip = False
+ if tag == "title":
+ self._in_title = False
+
+ def handle_data(self, data):
+ if self._in_title:
+ self._title = data.strip()
+ if not self._skip:
+ self._pieces.append(data)
+
+ def get_text(self) -> str:
+ raw = "".join(self._pieces)
+ # Collapse whitespace
+ lines = [line.strip() for line in raw.splitlines()]
+ return re.sub(r"\n{3,}", "\n\n", "\n".join(lines)).strip()
+
+
+class BuiltinProvider:
+ """Minimal web fetcher using stdlib + optional httpx/bs4."""
+
+ name = "builtin"
+
+ def fetch(self, url: str) -> WebResult:
+ html, final_url = self._download(url)
+ title, content = self._extract(html)
+ return WebResult(
+ url=final_url,
+ title=title,
+ content=content,
+ raw_html=html,
+ fetched_at=datetime.now(timezone.utc),
+ )
+
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]:
+ raise NotImplementedError(
+ "Builtin provider does not support web search. "
+ "Install a search-capable provider: pip install hyperresearch[tavily] "
+ "or pip install hyperresearch[crawl4ai]"
+ )
+
+ def _download(self, url: str) -> tuple[str, str]:
+ """Download URL, return (html, final_url). Tries httpx first, falls back to urllib."""
+ try:
+ import httpx
+
+ with httpx.Client(follow_redirects=True, timeout=30) as client:
+ resp = client.get(url, headers={"User-Agent": "hyperresearch/0.1"})
+ resp.raise_for_status()
+ return resp.text, str(resp.url)
+ except ImportError:
+ pass
+
+ import urllib.request
+
+ req = urllib.request.Request(url, headers={"User-Agent": "hyperresearch/0.1"})
+ with urllib.request.urlopen(req, timeout=30) as resp:
+ html = resp.read().decode("utf-8", errors="replace")
+ return html, resp.url or url
+
+ def _extract(self, html: str) -> tuple[str, str]:
+ """Extract title and clean text from HTML. Tries bs4 first, falls back to stdlib."""
+ try:
+ from bs4 import BeautifulSoup
+
+ soup = BeautifulSoup(html, "html.parser")
+ # Remove noise
+ for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
+ tag.decompose()
+ title = soup.title.string.strip() if soup.title and soup.title.string else ""
+ text = soup.get_text(separator="\n", strip=True)
+ # Collapse blank lines
+ text = re.sub(r"\n{3,}", "\n\n", text)
+ return title, text
+ except ImportError:
+ pass
+
+ # Fallback: stdlib HTML parser
+ parser = _TextExtractor()
+ parser.feed(html)
+ return parser._title, parser.get_text()
diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.py
new file mode 100644
index 0000000..4779931
--- /dev/null
+++ b/src/hyperresearch/web/crawl4ai_provider.py
@@ -0,0 +1,34 @@
+"""Crawl4AI web provider — free, open-source, handles JS rendering."""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+
+from crawl4ai import CrawlerStrategy, WebCrawler
+
+from hyperresearch.web.base import WebResult
+
+
+class Crawl4AIProvider:
+ name = "crawl4ai"
+
+ def __init__(self):
+ self._crawler = WebCrawler()
+ self._crawler.warmup()
+
+ def fetch(self, url: str) -> WebResult:
+ result = self._crawler.run(url=url)
+ return WebResult(
+ url=url,
+ title=result.metadata.get("title", "") if result.metadata else "",
+ content=result.markdown or result.extracted_content or "",
+ raw_html=result.html,
+ fetched_at=datetime.now(timezone.utc),
+ metadata=result.metadata or {},
+ )
+
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]:
+ raise NotImplementedError(
+ "crawl4ai does not support web search directly. "
+ "Use tavily for search: pip install hyperresearch[tavily]"
+ )
diff --git a/src/hyperresearch/web/firecrawl_provider.py b/src/hyperresearch/web/firecrawl_provider.py
new file mode 100644
index 0000000..a5d9188
--- /dev/null
+++ b/src/hyperresearch/web/firecrawl_provider.py
@@ -0,0 +1,41 @@
+"""Firecrawl web provider — AI-optimized scraping, returns clean markdown."""
+
+from __future__ import annotations
+
+import os
+from datetime import datetime, timezone
+
+from firecrawl import FirecrawlApp
+
+from hyperresearch.web.base import WebResult
+
+
+class FirecrawlProvider:
+ name = "firecrawl"
+
+ def __init__(self, api_key: str | None = None, api_url: str | None = None):
+ key = api_key or os.environ.get("FIRECRAWL_API_KEY", "")
+ url = api_url or os.environ.get("FIRECRAWL_API_URL")
+ kwargs = {}
+ if key:
+ kwargs["api_key"] = key
+ if url:
+ kwargs["api_url"] = url
+ self._app = FirecrawlApp(**kwargs)
+
+ def fetch(self, url: str) -> WebResult:
+ result = self._app.scrape_url(url, params={"formats": ["markdown"]})
+ return WebResult(
+ url=url,
+ title=result.get("metadata", {}).get("title", ""),
+ content=result.get("markdown", ""),
+ raw_html=result.get("html"),
+ fetched_at=datetime.now(timezone.utc),
+ metadata=result.get("metadata", {}),
+ )
+
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]:
+ raise NotImplementedError(
+ "Firecrawl does not support web search directly. "
+ "Use tavily for search: pip install hyperresearch[tavily]"
+ )
diff --git a/src/hyperresearch/web/tavily_provider.py b/src/hyperresearch/web/tavily_provider.py
new file mode 100644
index 0000000..56dea05
--- /dev/null
+++ b/src/hyperresearch/web/tavily_provider.py
@@ -0,0 +1,55 @@
+"""Tavily web provider — search + extract API for LLM agents."""
+
+from __future__ import annotations
+
+import os
+from datetime import datetime, timezone
+
+from tavily import TavilyClient
+
+from hyperresearch.web.base import WebResult
+
+
+class TavilyProvider:
+ name = "tavily"
+
+ def __init__(self, api_key: str | None = None):
+ key = api_key or os.environ.get("TAVILY_API_KEY", "")
+ if not key:
+ raise ValueError(
+ "Tavily API key required. Set TAVILY_API_KEY env var "
+ "or configure in .hyperresearch/config.toml"
+ )
+ self._client = TavilyClient(api_key=key)
+
+ def fetch(self, url: str) -> WebResult:
+ result = self._client.extract(urls=[url])
+ if result and result.get("results"):
+ r = result["results"][0]
+ return WebResult(
+ url=url,
+ title=r.get("title", ""),
+ content=r.get("raw_content", r.get("text", "")),
+ fetched_at=datetime.now(timezone.utc),
+ metadata={"source": "tavily-extract"},
+ )
+ raise RuntimeError(f"Tavily extract returned no results for {url}")
+
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]:
+ response = self._client.search(
+ query=query,
+ max_results=max_results,
+ include_raw_content=True,
+ )
+ results = []
+ for r in response.get("results", []):
+ results.append(
+ WebResult(
+ url=r["url"],
+ title=r.get("title", ""),
+ content=r.get("raw_content", r.get("content", "")),
+ fetched_at=datetime.now(timezone.utc),
+ metadata={"score": r.get("score", 0), "source": "tavily-search"},
+ )
+ )
+ return results
diff --git a/src/hyperresearch/web/trafilatura_provider.py b/src/hyperresearch/web/trafilatura_provider.py
new file mode 100644
index 0000000..74fe277
--- /dev/null
+++ b/src/hyperresearch/web/trafilatura_provider.py
@@ -0,0 +1,49 @@
+"""Trafilatura web provider — lightweight text extraction, no JS rendering."""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+
+import trafilatura
+
+from hyperresearch.web.base import WebResult
+
+
+class TrafilaturaProvider:
+ name = "trafilatura"
+
+ def fetch(self, url: str) -> WebResult:
+ downloaded = trafilatura.fetch_url(url)
+ if downloaded is None:
+ raise RuntimeError(f"Failed to download {url}")
+
+ content = trafilatura.extract(
+ downloaded,
+ output_format="txt",
+ include_links=True,
+ include_tables=True,
+ ) or ""
+
+ metadata = trafilatura.extract_metadata(downloaded)
+ title = metadata.title if metadata and metadata.title else ""
+ meta_dict = {}
+ if metadata:
+ if metadata.author:
+ meta_dict["author"] = metadata.author
+ if metadata.date:
+ meta_dict["date"] = metadata.date
+
+ return WebResult(
+ url=url,
+ title=title,
+ content=content,
+ raw_html=downloaded,
+ fetched_at=datetime.now(timezone.utc),
+ metadata=meta_dict,
+ )
+
+ def search(self, query: str, max_results: int = 5) -> list[WebResult]:
+ raise NotImplementedError(
+ "Trafilatura does not support web search. "
+ "Use tavily for search: pip install hyperresearch[tavily]"
+ )