Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

Add research command, sources CLI, CI, README, lint fixes

Jordancbef1a8d9bc8
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000..6a8c787
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,23 @@
+name: CI
+
+on:
+ push:
+ branches: [main]
+ pull_request:
+ branches: [main]
+
+jobs:
+ test:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ python-version: ["3.11", "3.12", "3.13"]
+
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: ${{ matrix.python-version }}
+ - run: pip install -e ".[dev]"
+ - run: ruff check src/ tests/
+ - run: pytest tests/ -v
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7a28c2e..4f14f05 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,75 +1,31 @@
# Changelog
-All notable changes to this project will be documented in this file.
-
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-
-## [0.2.0] - 2026-04-07
-
-### Added
-
-- MCP server with 8 read-only tools (`pip install llm-kasten[mcp]`)
-- `kasten mcp` command (stdio transport for Claude Desktop, Cursor)
-- `kasten tags --min N` to filter out low-count tags
-- `kasten tag alias` and `kasten tag suggest` for tag management
-- `kasten note update` for single-note metadata changes
-- `kasten note new --body-file` to avoid shell escaping
-- `kasten note new` returns `existing_tags` in JSON to reduce tag noise
-- `kasten search` auto-includes note bodies in JSON mode
-- `kasten search --linked-from`, `--linked-to`, `--min-inbound` graph filters
-- `kasten repair --enrich` auto-fills missing tags and summaries
-- `kasten graph stub` creates stub notes for all broken links
-- `kasten serve` with interactive force-directed knowledge graph
-- `kasten dedup` for content similarity detection
-- Auto-inject docs into CLAUDE.md, AGENTS.md, GEMINI.md, copilot-instructions.md
-- Temporal index pages: by-month, stale notes, most-linked
-- `-j` short flag for `--json` on every command
-- Alphanumeric search tokenization (mamba3 matches mamba 3)
-- Plural tag normalization during sync
-- Note lifecycle: draft, review, evergreen, stale, deprecated, archive
-- `missing-summary` and `singleton-tags` lint rules
-- Built-in note templates: concept, reference, guide, comparison, moc
-- Vintage cream/brown theme for web UI
-
-### Removed
-
-- External LLM dependencies (OpenAI, Anthropic, Ollama providers)
-- `kasten compile` and `kasten ask` commands (agents drive everything)
-- `kasten ingest` (file, web, pdf) -- agents write files directly
-- Semantic search and ranking modules (dead code)
-- `kasten promote` and `kasten diff` (redundant with repair and sync --dry-run)
-
-### Fixed
-
-- Wiki-link parser strips trailing backslash from shell escaping artifacts
-- URLs in `[[https://...]]` no longer treated as broken note references
-- Index generator deletes stale pages before regenerating
-- XSS protection in web UI
-- Ctrl+C works on Windows for `kasten serve`
-- FTS5 BM25 weights now read from config (were hardcoded)
-- Import writes to knowledge directory, not vault root
-- Batch operations validate status values and handle per-file errors
-
-## [0.1.0] - 2026-04-06
-
-Initial release.
-
-### Core
-
-- Markdown files with YAML frontmatter as source of truth
-- SQLite FTS5 full-text search with BM25 ranking and porter stemming
-- `[[wiki-link]]` parsing with backlink tracking and broken link detection
-- mtime + SHA-256 change detection with atomic transactions
-
-### CLI
-
-- `kasten init`, `status`, `sync`, `search`, `note` CRUD, `graph`, `lint`, `repair`
-- `kasten index build` auto-generated wiki pages
-- `kasten batch` bulk operations
-- `kasten export json` and `export vault`
-- `kasten import`, `watch`, `serve`, `git log`
-- `--json` on every command with consistent envelope format
-
-[0.2.0]: https://github.com/jordan-gibbs/llm-kasten/releases/tag/v0.2.0
-[0.1.0]: https://github.com/jordan-gibbs/llm-kasten/releases/tag/v0.1.0
+## [0.1.0] - 2026-04-09
+
+Initial release. Forked from [llm-kasten](https://github.com/jordan-gibbs/llm-kasten) and repositioned for agent-driven research workflows.
+
+### New
+
+- **`hyperresearch install`** — One-step setup: init vault + inject agent docs + install PreToolUse hooks for Claude Code, Codex, Cursor, Gemini CLI
+- **`hyperresearch fetch <url>`** — Fetch a URL, extract content, save as a research note with source tracking
+- **`hyperresearch research <topic>`** — Deep research: web search, fetch results, follow links, save as linked notes, generate synthesis MOC
+- **`hyperresearch sources list/check`** — List and query fetched web sources
+- **Web provider plugin system** — Pluggable backends: builtin (stdlib), crawl4ai, tavily, firecrawl, trafilatura
+- **Agent hook system** — PreToolUse hooks that remind agents to check the research base before web searches
+- **Sources table** — URL deduplication, domain tracking, fetch metadata
+- **Extended frontmatter** — `source_domain`, `fetched_at`, `fetch_provider` fields
+- **MCP server** with 10 tools including `check_source` and `list_sources`
+
+### From kasten (the backbone)
+
+- SQLite FTS5 full-text search with BM25 ranking
+- Markdown notes with YAML frontmatter as source of truth
+- `[[wiki-link]]` tracking with backlinks
+- `--json` / `-j` structured output on every command
+- Note lifecycle: draft → review → evergreen → stale → deprecated → archive
+- Auto-sync (mtime + SHA-256 change detection)
+- Agent doc injection (CLAUDE.md, AGENTS.md, GEMINI.md, copilot-instructions.md)
+- Web viewer with force-directed knowledge graph
+- 70 tests
+
+[0.1.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.1.0
diff --git a/README.md b/README.md
index f32a337..766adc6 100644
--- a/README.md
+++ b/README.md
@@ -1,9 +1,116 @@
# Hyperresearch
-Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki with `[[wiki-links]]`, full-text search, and structured JSON output.
+Agent-driven research knowledge base. Install it, and your AI coding agent can collect, search, and synthesize web research into a persistent, searchable wiki — across sessions.
```bash
pip install hyperresearch
-hyperresearch init .
-hyperresearch search "query" -j
+hyperresearch install # init vault + hook your agent
```
+
+That's it. Your agent now checks the research base before searching the web, saves useful findings automatically, and builds a knowledge graph over time.
+
+## How it works
+
+1. **Agent finds something useful** (via its own web search, browsing, or your input)
+2. **Agent saves it**: `hyperresearch fetch "https://..." --tag ml -j` or `hyperresearch note new "Title" --body-file content.md -j`
+3. **Next time it needs info**, the PreToolUse hook reminds it: *"check hyperresearch first"*
+4. **Agent searches the KB**: `hyperresearch search "attention mechanisms" -j`
+5. **Knowledge compounds** across sessions — no redundant fetches, no lost context
+
+```
+your-repo/
+ .hyperresearch/ # Hidden: config, SQLite index, hook script
+ research/
+ notes/ # Markdown notes (source of truth)
+ index/ # Auto-generated wiki pages
+ CLAUDE.md # Agent docs (auto-injected)
+```
+
+## Commands
+
+```bash
+# Setup
+hyperresearch install # Init + hooks (Claude Code, Cursor, Codex, Gemini)
+hyperresearch install --platform all # Hook all supported platforms
+
+# Collect
+hyperresearch fetch <url> --tag t -j # Save a URL as a note
+hyperresearch research "topic" --max 5 -j # Search → fetch → link → synthesize (needs tavily)
+
+# Search & read
+hyperresearch search "query" -j # Full-text search
+hyperresearch note show <id> -j # Read a note
+hyperresearch note list --tag ml -j # List notes by tag
+
+# Manage
+hyperresearch sources list -j # What URLs have been fetched
+hyperresearch sources check <url> -j # Has this URL been fetched?
+hyperresearch repair -j # Fix links, promote notes, rebuild indexes
+hyperresearch status -j # Vault health overview
+```
+
+Every command returns `{"ok": true, "data": {...}}` with `-j`.
+
+## Agent integration
+
+`hyperresearch install` does three things:
+
+1. **Creates the vault** (`.hyperresearch/` + `research/`)
+2. **Injects usage docs** into CLAUDE.md (or AGENTS.md, GEMINI.md, copilot-instructions.md)
+3. **Installs PreToolUse hooks** that fire before web searches:
+
+| Platform | Hook | Trigger |
+|----------|------|---------|
+| Claude Code | `.claude/settings.json` | Before Glob, Grep, WebSearch, WebFetch |
+| Codex | `.codex/hooks.json` | Before Bash |
+| Cursor | `.cursor/rules/hyperresearch.mdc` | Always-apply rule |
+| Gemini CLI | `.gemini/settings.json` | Before tool calls |
+
+The hook doesn't block — it reminds the agent to check the research base first.
+
+## Web providers
+
+By default, agents use their own web tools (WebSearch, WebFetch) and pipe content into hyperresearch. For standalone/headless use, optional providers add direct fetching:
+
+```bash
+pip install hyperresearch[tavily] # Search + extract API ($0.008/credit, 1k free/mo)
+pip install hyperresearch[crawl4ai] # Free, open-source, JS rendering
+pip install hyperresearch[firecrawl] # AI-optimized markdown extraction
+pip install hyperresearch[trafilatura] # Lightweight text extraction
+```
+
+Configure in `.hyperresearch/config.toml`:
+```toml
+[web]
+provider = "tavily" # or "crawl4ai", "firecrawl", "trafilatura", "builtin"
+```
+
+## MCP server
+
+For Claude Desktop, Cursor inline, or any MCP-compatible agent:
+
+```bash
+pip install hyperresearch[mcp]
+```
+
+```json
+{"mcpServers": {"hyperresearch": {"command": "hyperresearch", "args": ["mcp"]}}}
+```
+
+10 tools: `search_notes`, `read_note`, `read_many`, `list_notes`, `get_backlinks`, `get_hubs`, `vault_status`, `lint_vault`, `check_source`, `list_sources`.
+
+## Philosophy
+
+- **The agent IS the LLM** — hyperresearch is a dumb tool that stores, indexes, and searches. It never calls an LLM.
+- **Files are truth** — markdown notes survive the tool dying. SQLite is a rebuildable cache.
+- **Agents already have web access** — hyperresearch is where they *store* what they find, not how they find it.
+- **Check before you fetch** — the hook system prevents redundant web searches across sessions.
+
+## Requirements
+
+- Python 3.11+
+- Works on Windows, macOS, Linux
+
+## License
+
+[MIT](LICENSE)
diff --git a/pyproject.toml b/pyproject.toml
index 6ba7674..ab4aa9c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -69,6 +69,7 @@ ignore = [
"B904",
"E402",
"N817",
+ "SIM105",
]
[tool.mypy]
diff --git a/src/hyperresearch/cli/__init__.py b/src/hyperresearch/cli/__init__.py
index 2069b7b..cd80dca 100644
--- a/src/hyperresearch/cli/__init__.py
+++ b/src/hyperresearch/cli/__init__.py
@@ -28,18 +28,19 @@ def main(
# Root-level commands
from hyperresearch.cli.dedup import dedup as _dedup
+from hyperresearch.cli.fetch import fetch as _fetch
from hyperresearch.cli.import_cmd import import_vault as _import
+from hyperresearch.cli.install import install as _install
from hyperresearch.cli.main import init as _init
from hyperresearch.cli.main import status as _status
from hyperresearch.cli.main import sync as _sync
from hyperresearch.cli.mcp_cmd import mcp as _mcp
from hyperresearch.cli.note import note_show as _show
from hyperresearch.cli.repair import repair as _repair
+from hyperresearch.cli.research import research as _research
from hyperresearch.cli.search import search as _search
from hyperresearch.cli.serve import serve as _serve
from hyperresearch.cli.tag import tag_list as _tags
-from hyperresearch.cli.fetch import fetch as _fetch
-from hyperresearch.cli.install import install as _install
from hyperresearch.cli.watch import watch as _watch
app.command("install")(_install)
@@ -48,6 +49,7 @@ app.command("status")(_status)
app.command("sync")(_sync)
app.command("search")(_search)
app.command("fetch")(_fetch)
+app.command("research")(_research)
app.command("tags")(_tags)
app.command("show", hidden=True)(_show)
app.command("dedup")(_dedup)
@@ -81,3 +83,7 @@ app.add_typer(batch_app, name="batch", help="Bulk operations.")
app.add_typer(template_app, name="template", help="Note templates.")
app.add_typer(git_app, name="git", help="Git integration.")
app.add_typer(tag_app, name="tag", help="Tag management.")
+
+from hyperresearch.cli.sources import app as sources_app
+
+app.add_typer(sources_app, name="sources", help="Fetched web sources.")
diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.py
index 5c5636e..b56fac7 100644
--- a/src/hyperresearch/cli/fetch.py
+++ b/src/hyperresearch/cli/fetch.py
@@ -3,9 +3,6 @@
from __future__ import annotations
import hashlib
-import sys
-from datetime import datetime, timezone
-from pathlib import Path
from urllib.parse import urlparse
import typer
diff --git a/src/hyperresearch/cli/research.py b/src/hyperresearch/cli/research.py
new file mode 100644
index 0000000..930a5df
--- /dev/null
+++ b/src/hyperresearch/cli/research.py
@@ -0,0 +1,262 @@
+"""Research command — deep research: search → fetch → follow links → save → synthesize."""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from datetime import UTC, datetime
+from urllib.parse import urlparse
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+
+def research(
+ topic: str = typer.Argument(..., help="Research topic or question"),
+ max_results: int = typer.Option(5, "--max", "-m", help="Max search results to fetch"),
+ depth: int = typer.Option(0, "--depth", "-d", help="Link-follow depth (0 = no following)"),
+ max_pages: int = typer.Option(10, "--max-pages", help="Max total pages to fetch"),
+ tags: list[str] = typer.Option([], "--tag", "-t", help="Tags for all created notes"),
+ parent: str | None = typer.Option(None, "--parent", "-p", help="Parent topic"),
+ provider_name: str | None = typer.Option(None, "--provider", help="Web provider override"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Deep research: search the web, fetch results, save as linked notes, generate synthesis."""
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+ from hyperresearch.core.vault import Vault, VaultError
+ from hyperresearch.models.note import slugify
+ from hyperresearch.web.base import get_provider
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ vault.auto_sync()
+ conn = vault.db
+ prov = get_provider(provider_name or vault.config.web_provider)
+
+ # Step 1: Search
+ if not json_output:
+ console.print(f"[bold]Researching:[/] {topic}")
+ console.print(f"[dim]Provider: {prov.name}, max results: {max_results}, depth: {depth}[/]")
+
+ try:
+ search_results = prov.search(topic, max_results=max_results)
+ except NotImplementedError:
+ if json_output:
+ output(
+ error(
+ f"Provider '{prov.name}' does not support web search. "
+ "Use --provider tavily or let your agent search and use 'hyperresearch fetch' instead.",
+ "NO_SEARCH",
+ ),
+ json_mode=True,
+ )
+ else:
+ console.print(
+ f"[red]Provider '{prov.name}' cannot search.[/] "
+ "Use --provider tavily, or have your agent search and 'hyperresearch fetch' each URL."
+ )
+ raise typer.Exit(1)
+
+ if not search_results:
+ if json_output:
+ output(success({"notes_created": [], "topic": topic}, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print("[yellow]No results found.[/]")
+ return
+
+ if not json_output:
+ console.print(f"[green]Found {len(search_results)} results[/]")
+
+ # Step 2: Fetch and save each result
+ created_notes: list[dict] = []
+ fetched_urls: set[str] = set()
+ pages_fetched = 0
+
+ for result in search_results:
+ if pages_fetched >= max_pages:
+ break
+ note_data = _save_result(vault, conn, prov, result, tags, parent)
+ if note_data:
+ created_notes.append(note_data)
+ fetched_urls.add(result.url)
+ pages_fetched += 1
+ if not json_output:
+ console.print(f" [green]+[/] {note_data['title']}")
+
+ # Step 3: Follow links (depth > 0)
+ if depth > 0 and pages_fetched < max_pages:
+ links_to_follow = _extract_links_from_results(search_results)
+ for d in range(depth):
+ if pages_fetched >= max_pages:
+ break
+ next_links: list[str] = []
+ for link_url in links_to_follow:
+ if pages_fetched >= max_pages:
+ break
+ if link_url in fetched_urls:
+ continue
+ # Check if already in DB
+ existing = conn.execute(
+ "SELECT note_id FROM sources WHERE url = ?", (link_url,)
+ ).fetchone()
+ if existing:
+ fetched_urls.add(link_url)
+ continue
+
+ try:
+ result = prov.fetch(link_url)
+ note_data = _save_result(vault, conn, prov, result, tags, parent)
+ if note_data:
+ created_notes.append(note_data)
+ fetched_urls.add(link_url)
+ pages_fetched += 1
+ if not json_output:
+ console.print(f" [dim]+[/] {note_data['title']} [dim](depth {d + 1})[/]")
+ # Collect links from this page for next depth
+ next_links.extend(_extract_links_from_results([result]))
+ except Exception:
+ continue
+ links_to_follow = [u for u in next_links if u not in fetched_urls]
+
+ # Step 4: Sync to index everything
+ plan = compute_sync_plan(vault)
+ if plan.to_add or plan.to_update:
+ execute_sync(vault, plan)
+
+ # Step 5: Generate synthesis note (map-of-content)
+ if created_notes:
+ moc_id = slugify(f"research-{topic}")
+ wiki_links = "\n".join(
+ f"- [[{n['note_id']}]] — {n.get('summary', n['title'])}"
+ for n in created_notes
+ )
+ moc_body = (
+ f"# Research: {topic}\n\n"
+ f"**{len(created_notes)} sources collected** "
+ f"on {datetime.now(UTC).strftime('%Y-%m-%d')}.\n\n"
+ f"## Sources\n\n{wiki_links}\n"
+ )
+
+ moc_tags = list(set([*tags, "research", "moc"]))
+ moc_path = write_note(
+ vault.notes_dir,
+ title=f"Research: {topic}",
+ body=moc_body,
+ note_id=moc_id,
+ tags=moc_tags,
+ status="review",
+ note_type="moc",
+ parent=parent,
+ summary=f"Research synthesis: {topic} ({len(created_notes)} sources)",
+ )
+
+ # Final sync to pick up the MOC
+ plan = compute_sync_plan(vault)
+ if plan.to_add or plan.to_update:
+ execute_sync(vault, plan)
+
+ created_notes.append({
+ "note_id": moc_id,
+ "title": f"Research: {topic}",
+ "type": "moc",
+ "path": str(moc_path.relative_to(vault.root)),
+ })
+
+ if not json_output:
+ console.print(f"\n[bold green]Synthesis:[/] [[{moc_id}]]")
+
+ # Output
+ data = {
+ "topic": topic,
+ "provider": prov.name,
+ "notes_created": created_notes,
+ "total_fetched": pages_fetched,
+ "depth": depth,
+ }
+
+ if json_output:
+ output(success(data, count=len(created_notes), vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(
+ f"\n[bold]Done:[/] {len(created_notes)} notes created. "
+ f"Start with: hyperresearch note show {created_notes[-1]['note_id']} -j"
+ )
+
+
+def _save_result(vault, conn, prov, result, tags, parent) -> dict | None:
+ """Save a single WebResult as a note. Returns note data dict or None if skipped."""
+ from hyperresearch.core.note import write_note
+
+ url = result.url
+
+ # Skip if already fetched
+ existing = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()
+ if existing:
+ return None
+
+ title = result.title or urlparse(url).path.split("/")[-1] or "Untitled"
+ domain = result.domain
+ content_hash = hashlib.sha256(result.content.encode("utf-8")).hexdigest()[:16]
+
+ extra_meta = {
+ "source": url,
+ "source_domain": domain,
+ "fetched_at": result.fetched_at.isoformat(),
+ "fetch_provider": prov.name,
+ }
+
+ note_path = write_note(
+ vault.notes_dir,
+ title=title,
+ body=result.content,
+ tags=tags,
+ status="draft",
+ source=url,
+ parent=parent,
+ extra_frontmatter=extra_meta,
+ )
+
+ note_id = note_path.stem
+ conn.execute(
+ """INSERT OR IGNORE INTO sources (url, note_id, domain, fetched_at, provider, content_hash)
+ VALUES (?, ?, ?, ?, ?, ?)""",
+ (url, note_id, domain, result.fetched_at.isoformat(), prov.name, content_hash),
+ )
+ conn.commit()
+
+ return {
+ "note_id": note_id,
+ "title": title,
+ "url": url,
+ "domain": domain,
+ "path": str(note_path.relative_to(vault.root)),
+ "word_count": len(result.content.split()),
+ "summary": result.content[:120].replace("\n", " ").strip(),
+ }
+
+
+def _extract_links_from_results(results) -> list[str]:
+ """Extract outbound HTTP links from result content."""
+ urls = []
+ url_re = re.compile(r'https?://[^\s<>\[\]"\']+')
+ for r in results:
+ for match in url_re.finditer(r.content):
+ url = match.group(0).rstrip(".,;:)")
+ # Skip common noise
+ parsed = urlparse(url)
+ if parsed.netloc and not any(
+ skip in parsed.netloc
+ for skip in ["google.com", "facebook.com", "twitter.com", "youtube.com"]
+ ):
+ urls.append(url)
+ return list(dict.fromkeys(urls)) # Dedupe, preserve order
diff --git a/src/hyperresearch/cli/sources.py b/src/hyperresearch/cli/sources.py
new file mode 100644
index 0000000..21a7f6e
--- /dev/null
+++ b/src/hyperresearch/cli/sources.py
@@ -0,0 +1,97 @@
+"""Sources command — list and manage fetched web sources."""
+
+from __future__ import annotations
+
+import typer
+
+from hyperresearch.cli._output import console, output
+from hyperresearch.models.output import error, success
+
+app = typer.Typer()
+
+
+def source_list(
+ domain: str | None = typer.Option(None, "--domain", help="Filter by domain"),
+ limit: int = typer.Option(50, "--limit", "-n", help="Max results"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """List all fetched web sources."""
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ conn = vault.db
+
+ if domain:
+ rows = conn.execute(
+ "SELECT url, note_id, domain, fetched_at, provider, status "
+ "FROM sources WHERE domain = ? ORDER BY fetched_at DESC LIMIT ?",
+ (domain, limit),
+ ).fetchall()
+ else:
+ rows = conn.execute(
+ "SELECT url, note_id, domain, fetched_at, provider, status "
+ "FROM sources ORDER BY fetched_at DESC LIMIT ?",
+ (limit,),
+ ).fetchall()
+
+ sources = [dict(row) for row in rows]
+
+ if json_output:
+ output(success(sources, count=len(sources), vault=str(vault.root)), json_mode=True)
+ else:
+ if not sources:
+ console.print("[dim]No sources fetched yet. Use 'hyperresearch fetch <url>' to start.[/]")
+ return
+ for s in sources:
+ status_color = "green" if s["status"] == "active" else "red"
+ console.print(
+ f"[{status_color}]{s['status']}[/] {s['url']}"
+ f" → [cyan]{s['note_id'] or '(deleted)'}[/]"
+ f" [dim]({s['provider']}, {s['fetched_at']})[/]"
+ )
+ console.print(f"\n[dim]{len(sources)} sources[/]")
+
+
+def source_check(
+ url: str = typer.Argument(..., help="URL to check"),
+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
+) -> None:
+ """Check if a URL has already been fetched."""
+ from hyperresearch.core.vault import Vault, VaultError
+
+ try:
+ vault = Vault.discover()
+ except VaultError as e:
+ if json_output:
+ output(error(str(e), "NO_VAULT"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1)
+
+ row = vault.db.execute(
+ "SELECT url, note_id, domain, fetched_at, provider FROM sources WHERE url = ?",
+ (url,),
+ ).fetchone()
+
+ if row:
+ data = {"exists": True, **dict(row)}
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[green]Found:[/] {url} → note '{row['note_id']}'")
+ else:
+ data = {"exists": False, "url": url}
+ if json_output:
+ output(success(data, vault=str(vault.root)), json_mode=True)
+ else:
+ console.print(f"[dim]Not fetched:[/] {url}")
diff --git a/src/hyperresearch/mcp/server.py b/src/hyperresearch/mcp/server.py
index 3c12804..b4fa87d 100644
--- a/src/hyperresearch/mcp/server.py
+++ b/src/hyperresearch/mcp/server.py
@@ -233,3 +233,44 @@ def lint_vault(rule: str = "") -> str:
for r in conn.execute("SELECT id FROM notes WHERE type NOT IN ('index','raw') AND id NOT IN (SELECT DISTINCT target_id FROM links WHERE target_id IS NOT NULL) AND id NOT IN (SELECT DISTINCT source_id FROM links)"):
issues.append({"rule": "orphaned-notes", "severity": "info", "note_id": r["id"], "message": "No links"})
return json.dumps({"issues": issues, "total": len(issues), "warnings": sum(1 for i in issues if i["severity"] == "warning")})
+
+
+def check_source(url: str) -> str:
+ """Check if a URL has already been fetched into the research base.
+
+ Args:
+ url: The URL to check
+ """
+ vault = _get_vault()
+ row = vault.db.execute(
+ "SELECT url, note_id, domain, fetched_at, provider FROM sources WHERE url = ?",
+ (url,),
+ ).fetchone()
+ if row:
+ return json.dumps({"exists": True, **dict(row)})
+ return json.dumps({"exists": False, "url": url})
+
+
+def list_sources(domain: str = "", limit: int = 50) -> str:
+ """List fetched web sources, optionally filtered by domain.
+
+ Args:
+ domain: Filter by domain (e.g. "arxiv.org"). Leave empty for all.
+ limit: Max results (default 50)
+ """
+ vault = _get_vault()
+ if domain:
+ rows = vault.db.execute(
+ "SELECT url, note_id, domain, fetched_at, provider, status "
+ "FROM sources WHERE domain = ? ORDER BY fetched_at DESC LIMIT ?",
+ (domain, limit),
+ ).fetchall()
+ else:
+ rows = vault.db.execute(
+ "SELECT url, note_id, domain, fetched_at, provider, status "
+ "FROM sources ORDER BY fetched_at DESC LIMIT ?",
+ (limit,),
+ ).fetchall()
+ return json.dumps([dict(r) for r in rows])
diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.py
index c5ffd0a..84b707c 100644
--- a/src/hyperresearch/web/base.py
+++ b/src/hyperresearch/web/base.py
@@ -3,7 +3,7 @@
from __future__ import annotations
from dataclasses import dataclass, field
-from datetime import datetime, timezone
+from datetime import UTC, datetime
from typing import Protocol, runtime_checkable
@@ -14,7 +14,7 @@ class WebResult:
url: str
title: str
content: str # clean markdown or plain text
- fetched_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+ fetched_at: datetime = field(default_factory=lambda: datetime.now(UTC))
raw_html: str | None = None
metadata: dict = field(default_factory=dict) # author, date, domain, etc.
diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.py
index b1d739f..de7b24b 100644
--- a/src/hyperresearch/web/builtin.py
+++ b/src/hyperresearch/web/builtin.py
@@ -3,7 +3,7 @@
from __future__ import annotations
import re
-from datetime import datetime, timezone
+from datetime import UTC, datetime
from html.parser import HTMLParser
from hyperresearch.web.base import WebResult
@@ -60,7 +60,7 @@ class BuiltinProvider:
title=title,
content=content,
raw_html=html,
- fetched_at=datetime.now(timezone.utc),
+ fetched_at=datetime.now(UTC),
)
def search(self, query: str, max_results: int = 5) -> list[WebResult]:
diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.py
index 4779931..0c796ca 100644
--- a/src/hyperresearch/web/crawl4ai_provider.py
+++ b/src/hyperresearch/web/crawl4ai_provider.py
@@ -2,9 +2,9 @@
from __future__ import annotations
-from datetime import datetime, timezone
+from datetime import UTC, datetime
-from crawl4ai import CrawlerStrategy, WebCrawler
+from crawl4ai import WebCrawler
from hyperresearch.web.base import WebResult
@@ -23,7 +23,7 @@ class Crawl4AIProvider:
title=result.metadata.get("title", "") if result.metadata else "",
content=result.markdown or result.extracted_content or "",
raw_html=result.html,
- fetched_at=datetime.now(timezone.utc),
+ fetched_at=datetime.now(UTC),
metadata=result.metadata or {},
)
diff --git a/src/hyperresearch/web/firecrawl_provider.py b/src/hyperresearch/web/firecrawl_provider.py
index a5d9188..82cba9e 100644
--- a/src/hyperresearch/web/firecrawl_provider.py
+++ b/src/hyperresearch/web/firecrawl_provider.py
@@ -3,7 +3,7 @@
from __future__ import annotations
import os
-from datetime import datetime, timezone
+from datetime import UTC, datetime
from firecrawl import FirecrawlApp
@@ -30,7 +30,7 @@ class FirecrawlProvider:
title=result.get("metadata", {}).get("title", ""),
content=result.get("markdown", ""),
raw_html=result.get("html"),
- fetched_at=datetime.now(timezone.utc),
+ fetched_at=datetime.now(UTC),
metadata=result.get("metadata", {}),
)
diff --git a/src/hyperresearch/web/tavily_provider.py b/src/hyperresearch/web/tavily_provider.py
index 56dea05..01a75ee 100644
--- a/src/hyperresearch/web/tavily_provider.py
+++ b/src/hyperresearch/web/tavily_provider.py
@@ -3,7 +3,7 @@
from __future__ import annotations
import os
-from datetime import datetime, timezone
+from datetime import UTC, datetime
from tavily import TavilyClient
@@ -30,7 +30,7 @@ class TavilyProvider:
url=url,
title=r.get("title", ""),
content=r.get("raw_content", r.get("text", "")),
- fetched_at=datetime.now(timezone.utc),
+ fetched_at=datetime.now(UTC),
metadata={"source": "tavily-extract"},
)
raise RuntimeError(f"Tavily extract returned no results for {url}")
@@ -48,7 +48,7 @@ class TavilyProvider:
url=r["url"],
title=r.get("title", ""),
content=r.get("raw_content", r.get("content", "")),
- fetched_at=datetime.now(timezone.utc),
+ fetched_at=datetime.now(UTC),
metadata={"score": r.get("score", 0), "source": "tavily-search"},
)
)
diff --git a/src/hyperresearch/web/trafilatura_provider.py b/src/hyperresearch/web/trafilatura_provider.py
index 74fe277..a9386ea 100644
--- a/src/hyperresearch/web/trafilatura_provider.py
+++ b/src/hyperresearch/web/trafilatura_provider.py
@@ -2,7 +2,7 @@
from __future__ import annotations
-from datetime import datetime, timezone
+from datetime import UTC, datetime
import trafilatura
@@ -38,7 +38,7 @@ class TrafilaturaProvider:
title=title,
content=content,
raw_html=downloaded,
- fetched_at=datetime.now(timezone.utc),
+ fetched_at=datetime.now(UTC),
metadata=meta_dict,
)
diff --git a/tests/test_core/test_frontmatter.py b/tests/test_core/test_frontmatter.py
index 9fd3510..1e352b8 100644
--- a/tests/test_core/test_frontmatter.py
+++ b/tests/test_core/test_frontmatter.py
@@ -1,6 +1,6 @@
"""Tests for frontmatter parsing and serialization."""
-from hyperresearch.core.frontmatter import parse_frontmatter, serialize_frontmatter, render_note
+from hyperresearch.core.frontmatter import parse_frontmatter, render_note, serialize_frontmatter
from hyperresearch.models.note import NoteMeta
@@ -22,7 +22,7 @@ def test_parse_missing_frontmatter():
def test_parse_empty_frontmatter():
content = "---\n---\n\nBody here."
- meta, body = parse_frontmatter(content)
+ meta, _body = parse_frontmatter(content)
assert meta.title == "Untitled"
diff --git a/tests/test_core/test_note.py b/tests/test_core/test_note.py
index 7cf1c0a..e661377 100644
--- a/tests/test_core/test_note.py
+++ b/tests/test_core/test_note.py
@@ -1,8 +1,7 @@
"""Tests for note read/write operations."""
-from pathlib import Path
-from hyperresearch.core.note import read_note, write_note, strip_markdown
+from hyperresearch.core.note import read_note, strip_markdown, write_note
from hyperresearch.models.note import slugify
diff --git a/tests/test_graph/test_links.py b/tests/test_graph/test_links.py
index dc67713..942aaa4 100644
--- a/tests/test_graph/test_links.py
+++ b/tests/test_graph/test_links.py
@@ -1,6 +1,6 @@
"""Tests for link parsing and graph operations."""
-from hyperresearch.core.note import WIKI_LINK_RE, CODE_BLOCK_RE, INLINE_CODE_RE
+from hyperresearch.core.note import CODE_BLOCK_RE, INLINE_CODE_RE, WIKI_LINK_RE
def test_wiki_link_regex_simple():
diff --git a/tests/test_search/test_fts.py b/tests/test_search/test_fts.py
index 68869fb..dd1afc5 100644
--- a/tests/test_search/test_fts.py
+++ b/tests/test_search/test_fts.py
@@ -1,7 +1,7 @@
"""Tests for full-text search."""
-from hyperresearch.search.fts import search_fts, preprocess_query
from hyperresearch.search.filters import SearchFilters
+from hyperresearch.search.fts import preprocess_query, search_fts
def test_preprocess_simple_query():