Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
Add research command, sources CLI, CI, README, lint fixes
Jordancbef1a8d9bc8
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlnew file mode 100644index 0000000..6a8c787--- /dev/null+++ b/.github/workflows/ci.yml@@ -0,0 +1,23 @@+name: CI++on:+ push:+ branches: [main]+ pull_request:+ branches: [main]++jobs:+ test:+ runs-on: ubuntu-latest+ strategy:+ matrix:+ python-version: ["3.11", "3.12", "3.13"]++ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: ${{ matrix.python-version }}+ - run: pip install -e ".[dev]"+ - run: ruff check src/ tests/+ - run: pytest tests/ -vdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7a28c2e..4f14f05 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,75 +1,31 @@# Changelog-All notable changes to this project will be documented in this file.--The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).--## [0.2.0] - 2026-04-07--### Added--- MCP server with 8 read-only tools (`pip install llm-kasten[mcp]`)-- `kasten mcp` command (stdio transport for Claude Desktop, Cursor)-- `kasten tags --min N` to filter out low-count tags-- `kasten tag alias` and `kasten tag suggest` for tag management-- `kasten note update` for single-note metadata changes-- `kasten note new --body-file` to avoid shell escaping-- `kasten note new` returns `existing_tags` in JSON to reduce tag noise-- `kasten search` auto-includes note bodies in JSON mode-- `kasten search --linked-from`, `--linked-to`, `--min-inbound` graph filters-- `kasten repair --enrich` auto-fills missing tags and summaries-- `kasten graph stub` creates stub notes for all broken links-- `kasten serve` with interactive force-directed knowledge graph-- `kasten dedup` for content similarity detection-- Auto-inject docs into CLAUDE.md, AGENTS.md, GEMINI.md, copilot-instructions.md-- Temporal index pages: by-month, stale notes, most-linked-- `-j` short flag for `--json` on every command-- Alphanumeric search tokenization (mamba3 matches mamba 3)-- Plural tag normalization during sync-- Note lifecycle: draft, review, evergreen, stale, deprecated, archive-- `missing-summary` and `singleton-tags` lint rules-- Built-in note templates: concept, reference, guide, comparison, moc-- Vintage cream/brown theme for web UI--### Removed--- External LLM dependencies (OpenAI, Anthropic, Ollama providers)-- `kasten compile` and `kasten ask` commands (agents drive everything)-- `kasten ingest` (file, web, pdf) -- agents write files directly-- Semantic search and ranking modules (dead code)-- `kasten promote` and `kasten diff` (redundant with repair and sync --dry-run)--### Fixed--- Wiki-link parser strips trailing backslash from shell escaping artifacts-- URLs in `[[https://...]]` no longer treated as broken note references-- Index generator deletes stale pages before regenerating-- XSS protection in web UI-- Ctrl+C works on Windows for `kasten serve`-- FTS5 BM25 weights now read from config (were hardcoded)-- Import writes to knowledge directory, not vault root-- Batch operations validate status values and handle per-file errors--## [0.1.0] - 2026-04-06--Initial release.--### Core--- Markdown files with YAML frontmatter as source of truth-- SQLite FTS5 full-text search with BM25 ranking and porter stemming-- `[[wiki-link]]` parsing with backlink tracking and broken link detection-- mtime + SHA-256 change detection with atomic transactions--### CLI--- `kasten init`, `status`, `sync`, `search`, `note` CRUD, `graph`, `lint`, `repair`-- `kasten index build` auto-generated wiki pages-- `kasten batch` bulk operations-- `kasten export json` and `export vault`-- `kasten import`, `watch`, `serve`, `git log`-- `--json` on every command with consistent envelope format--[0.2.0]: https://github.com/jordan-gibbs/llm-kasten/releases/tag/v0.2.0-[0.1.0]: https://github.com/jordan-gibbs/llm-kasten/releases/tag/v0.1.0+## [0.1.0] - 2026-04-09++Initial release. Forked from [llm-kasten](https://github.com/jordan-gibbs/llm-kasten) and repositioned for agent-driven research workflows.++### New++- **`hyperresearch install`** — One-step setup: init vault + inject agent docs + install PreToolUse hooks for Claude Code, Codex, Cursor, Gemini CLI+- **`hyperresearch fetch <url>`** — Fetch a URL, extract content, save as a research note with source tracking+- **`hyperresearch research <topic>`** — Deep research: web search, fetch results, follow links, save as linked notes, generate synthesis MOC+- **`hyperresearch sources list/check`** — List and query fetched web sources+- **Web provider plugin system** — Pluggable backends: builtin (stdlib), crawl4ai, tavily, firecrawl, trafilatura+- **Agent hook system** — PreToolUse hooks that remind agents to check the research base before web searches+- **Sources table** — URL deduplication, domain tracking, fetch metadata+- **Extended frontmatter** — `source_domain`, `fetched_at`, `fetch_provider` fields+- **MCP server** with 10 tools including `check_source` and `list_sources`++### From kasten (the backbone)++- SQLite FTS5 full-text search with BM25 ranking+- Markdown notes with YAML frontmatter as source of truth+- `[[wiki-link]]` tracking with backlinks+- `--json` / `-j` structured output on every command+- Note lifecycle: draft → review → evergreen → stale → deprecated → archive+- Auto-sync (mtime + SHA-256 change detection)+- Agent doc injection (CLAUDE.md, AGENTS.md, GEMINI.md, copilot-instructions.md)+- Web viewer with force-directed knowledge graph+- 70 tests++[0.1.0]: https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.1.0diff --git a/README.md b/README.mdindex f32a337..766adc6 100644--- a/README.md+++ b/README.md@@ -1,9 +1,116 @@# Hyperresearch-Agent-driven research knowledge base. Browse, collect, and synthesize web sources into a searchable wiki with `[[wiki-links]]`, full-text search, and structured JSON output.+Agent-driven research knowledge base. Install it, and your AI coding agent can collect, search, and synthesize web research into a persistent, searchable wiki — across sessions.```bashpip install hyperresearch-hyperresearch init .-hyperresearch search "query" -j+hyperresearch install # init vault + hook your agent```++That's it. Your agent now checks the research base before searching the web, saves useful findings automatically, and builds a knowledge graph over time.++## How it works++1. **Agent finds something useful** (via its own web search, browsing, or your input)+2. **Agent saves it**: `hyperresearch fetch "https://..." --tag ml -j` or `hyperresearch note new "Title" --body-file content.md -j`+3. **Next time it needs info**, the PreToolUse hook reminds it: *"check hyperresearch first"*+4. **Agent searches the KB**: `hyperresearch search "attention mechanisms" -j`+5. **Knowledge compounds** across sessions — no redundant fetches, no lost context++```+your-repo/+ .hyperresearch/ # Hidden: config, SQLite index, hook script+ research/+ notes/ # Markdown notes (source of truth)+ index/ # Auto-generated wiki pages+ CLAUDE.md # Agent docs (auto-injected)+```++## Commands++```bash+# Setup+hyperresearch install # Init + hooks (Claude Code, Cursor, Codex, Gemini)+hyperresearch install --platform all # Hook all supported platforms++# Collect+hyperresearch fetch <url> --tag t -j # Save a URL as a note+hyperresearch research "topic" --max 5 -j # Search → fetch → link → synthesize (needs tavily)++# Search & read+hyperresearch search "query" -j # Full-text search+hyperresearch note show <id> -j # Read a note+hyperresearch note list --tag ml -j # List notes by tag++# Manage+hyperresearch sources list -j # What URLs have been fetched+hyperresearch sources check <url> -j # Has this URL been fetched?+hyperresearch repair -j # Fix links, promote notes, rebuild indexes+hyperresearch status -j # Vault health overview+```++Every command returns `{"ok": true, "data": {...}}` with `-j`.++## Agent integration++`hyperresearch install` does three things:++1. **Creates the vault** (`.hyperresearch/` + `research/`)+2. **Injects usage docs** into CLAUDE.md (or AGENTS.md, GEMINI.md, copilot-instructions.md)+3. **Installs PreToolUse hooks** that fire before web searches:++| Platform | Hook | Trigger |+|----------|------|---------|+| Claude Code | `.claude/settings.json` | Before Glob, Grep, WebSearch, WebFetch |+| Codex | `.codex/hooks.json` | Before Bash |+| Cursor | `.cursor/rules/hyperresearch.mdc` | Always-apply rule |+| Gemini CLI | `.gemini/settings.json` | Before tool calls |++The hook doesn't block — it reminds the agent to check the research base first.++## Web providers++By default, agents use their own web tools (WebSearch, WebFetch) and pipe content into hyperresearch. For standalone/headless use, optional providers add direct fetching:++```bash+pip install hyperresearch[tavily] # Search + extract API ($0.008/credit, 1k free/mo)+pip install hyperresearch[crawl4ai] # Free, open-source, JS rendering+pip install hyperresearch[firecrawl] # AI-optimized markdown extraction+pip install hyperresearch[trafilatura] # Lightweight text extraction+```++Configure in `.hyperresearch/config.toml`:+```toml+[web]+provider = "tavily" # or "crawl4ai", "firecrawl", "trafilatura", "builtin"+```++## MCP server++For Claude Desktop, Cursor inline, or any MCP-compatible agent:++```bash+pip install hyperresearch[mcp]+```++```json+{"mcpServers": {"hyperresearch": {"command": "hyperresearch", "args": ["mcp"]}}}+```++10 tools: `search_notes`, `read_note`, `read_many`, `list_notes`, `get_backlinks`, `get_hubs`, `vault_status`, `lint_vault`, `check_source`, `list_sources`.++## Philosophy++- **The agent IS the LLM** — hyperresearch is a dumb tool that stores, indexes, and searches. It never calls an LLM.+- **Files are truth** — markdown notes survive the tool dying. SQLite is a rebuildable cache.+- **Agents already have web access** — hyperresearch is where they *store* what they find, not how they find it.+- **Check before you fetch** — the hook system prevents redundant web searches across sessions.++## Requirements++- Python 3.11++- Works on Windows, macOS, Linux++## License++[MIT](LICENSE)diff --git a/pyproject.toml b/pyproject.tomlindex 6ba7674..ab4aa9c 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -69,6 +69,7 @@ ignore = ["B904","E402","N817",+ "SIM105",][tool.mypy]diff --git a/src/hyperresearch/cli/__init__.py b/src/hyperresearch/cli/__init__.pyindex 2069b7b..cd80dca 100644--- a/src/hyperresearch/cli/__init__.py+++ b/src/hyperresearch/cli/__init__.py@@ -28,18 +28,19 @@ def main(# Root-level commandsfrom hyperresearch.cli.dedup import dedup as _dedup+from hyperresearch.cli.fetch import fetch as _fetchfrom hyperresearch.cli.import_cmd import import_vault as _import+from hyperresearch.cli.install import install as _installfrom hyperresearch.cli.main import init as _initfrom hyperresearch.cli.main import status as _statusfrom hyperresearch.cli.main import sync as _syncfrom hyperresearch.cli.mcp_cmd import mcp as _mcpfrom hyperresearch.cli.note import note_show as _showfrom hyperresearch.cli.repair import repair as _repair+from hyperresearch.cli.research import research as _researchfrom hyperresearch.cli.search import search as _searchfrom hyperresearch.cli.serve import serve as _servefrom hyperresearch.cli.tag import tag_list as _tags-from hyperresearch.cli.fetch import fetch as _fetch-from hyperresearch.cli.install import install as _installfrom hyperresearch.cli.watch import watch as _watchapp.command("install")(_install)@@ -48,6 +49,7 @@ app.command("status")(_status)app.command("sync")(_sync)app.command("search")(_search)app.command("fetch")(_fetch)+app.command("research")(_research)app.command("tags")(_tags)app.command("show", hidden=True)(_show)app.command("dedup")(_dedup)@@ -81,3 +83,7 @@ app.add_typer(batch_app, name="batch", help="Bulk operations.")app.add_typer(template_app, name="template", help="Note templates.")app.add_typer(git_app, name="git", help="Git integration.")app.add_typer(tag_app, name="tag", help="Tag management.")++from hyperresearch.cli.sources import app as sources_app++app.add_typer(sources_app, name="sources", help="Fetched web sources.")diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.pyindex 5c5636e..b56fac7 100644--- a/src/hyperresearch/cli/fetch.py+++ b/src/hyperresearch/cli/fetch.py@@ -3,9 +3,6 @@from __future__ import annotationsimport hashlib-import sys-from datetime import datetime, timezone-from pathlib import Pathfrom urllib.parse import urlparseimport typerdiff --git a/src/hyperresearch/cli/research.py b/src/hyperresearch/cli/research.pynew file mode 100644index 0000000..930a5df--- /dev/null+++ b/src/hyperresearch/cli/research.py@@ -0,0 +1,262 @@+"""Research command — deep research: search → fetch → follow links → save → synthesize."""++from __future__ import annotations++import hashlib+import re+from datetime import UTC, datetime+from urllib.parse import urlparse++import typer++from hyperresearch.cli._output import console, output+from hyperresearch.models.output import error, success+++def research(+ topic: str = typer.Argument(..., help="Research topic or question"),+ max_results: int = typer.Option(5, "--max", "-m", help="Max search results to fetch"),+ depth: int = typer.Option(0, "--depth", "-d", help="Link-follow depth (0 = no following)"),+ max_pages: int = typer.Option(10, "--max-pages", help="Max total pages to fetch"),+ tags: list[str] = typer.Option([], "--tag", "-t", help="Tags for all created notes"),+ parent: str | None = typer.Option(None, "--parent", "-p", help="Parent topic"),+ provider_name: str | None = typer.Option(None, "--provider", help="Web provider override"),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """Deep research: search the web, fetch results, save as linked notes, generate synthesis."""+ from hyperresearch.core.note import write_note+ from hyperresearch.core.sync import compute_sync_plan, execute_sync+ from hyperresearch.core.vault import Vault, VaultError+ from hyperresearch.models.note import slugify+ from hyperresearch.web.base import get_provider++ try:+ vault = Vault.discover()+ except VaultError as e:+ if json_output:+ output(error(str(e), "NO_VAULT"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {e}")+ raise typer.Exit(1)++ vault.auto_sync()+ conn = vault.db+ prov = get_provider(provider_name or vault.config.web_provider)++ # Step 1: Search+ if not json_output:+ console.print(f"[bold]Researching:[/] {topic}")+ console.print(f"[dim]Provider: {prov.name}, max results: {max_results}, depth: {depth}[/]")++ try:+ search_results = prov.search(topic, max_results=max_results)+ except NotImplementedError:+ if json_output:+ output(+ error(+ f"Provider '{prov.name}' does not support web search. "+ "Use --provider tavily or let your agent search and use 'hyperresearch fetch' instead.",+ "NO_SEARCH",+ ),+ json_mode=True,+ )+ else:+ console.print(+ f"[red]Provider '{prov.name}' cannot search.[/] "+ "Use --provider tavily, or have your agent search and 'hyperresearch fetch' each URL."+ )+ raise typer.Exit(1)++ if not search_results:+ if json_output:+ output(success({"notes_created": [], "topic": topic}, vault=str(vault.root)), json_mode=True)+ else:+ console.print("[yellow]No results found.[/]")+ return++ if not json_output:+ console.print(f"[green]Found {len(search_results)} results[/]")++ # Step 2: Fetch and save each result+ created_notes: list[dict] = []+ fetched_urls: set[str] = set()+ pages_fetched = 0++ for result in search_results:+ if pages_fetched >= max_pages:+ break+ note_data = _save_result(vault, conn, prov, result, tags, parent)+ if note_data:+ created_notes.append(note_data)+ fetched_urls.add(result.url)+ pages_fetched += 1+ if not json_output:+ console.print(f" [green]+[/] {note_data['title']}")++ # Step 3: Follow links (depth > 0)+ if depth > 0 and pages_fetched < max_pages:+ links_to_follow = _extract_links_from_results(search_results)+ for d in range(depth):+ if pages_fetched >= max_pages:+ break+ next_links: list[str] = []+ for link_url in links_to_follow:+ if pages_fetched >= max_pages:+ break+ if link_url in fetched_urls:+ continue+ # Check if already in DB+ existing = conn.execute(+ "SELECT note_id FROM sources WHERE url = ?", (link_url,)+ ).fetchone()+ if existing:+ fetched_urls.add(link_url)+ continue++ try:+ result = prov.fetch(link_url)+ note_data = _save_result(vault, conn, prov, result, tags, parent)+ if note_data:+ created_notes.append(note_data)+ fetched_urls.add(link_url)+ pages_fetched += 1+ if not json_output:+ console.print(f" [dim]+[/] {note_data['title']} [dim](depth {d + 1})[/]")+ # Collect links from this page for next depth+ next_links.extend(_extract_links_from_results([result]))+ except Exception:+ continue+ links_to_follow = [u for u in next_links if u not in fetched_urls]++ # Step 4: Sync to index everything+ plan = compute_sync_plan(vault)+ if plan.to_add or plan.to_update:+ execute_sync(vault, plan)++ # Step 5: Generate synthesis note (map-of-content)+ if created_notes:+ moc_id = slugify(f"research-{topic}")+ wiki_links = "\n".join(+ f"- [[{n['note_id']}]] — {n.get('summary', n['title'])}"+ for n in created_notes+ )+ moc_body = (+ f"# Research: {topic}\n\n"+ f"**{len(created_notes)} sources collected** "+ f"on {datetime.now(UTC).strftime('%Y-%m-%d')}.\n\n"+ f"## Sources\n\n{wiki_links}\n"+ )++ moc_tags = list(set([*tags, "research", "moc"]))+ moc_path = write_note(+ vault.notes_dir,+ title=f"Research: {topic}",+ body=moc_body,+ note_id=moc_id,+ tags=moc_tags,+ status="review",+ note_type="moc",+ parent=parent,+ summary=f"Research synthesis: {topic} ({len(created_notes)} sources)",+ )++ # Final sync to pick up the MOC+ plan = compute_sync_plan(vault)+ if plan.to_add or plan.to_update:+ execute_sync(vault, plan)++ created_notes.append({+ "note_id": moc_id,+ "title": f"Research: {topic}",+ "type": "moc",+ "path": str(moc_path.relative_to(vault.root)),+ })++ if not json_output:+ console.print(f"\n[bold green]Synthesis:[/] [[{moc_id}]]")++ # Output+ data = {+ "topic": topic,+ "provider": prov.name,+ "notes_created": created_notes,+ "total_fetched": pages_fetched,+ "depth": depth,+ }++ if json_output:+ output(success(data, count=len(created_notes), vault=str(vault.root)), json_mode=True)+ else:+ console.print(+ f"\n[bold]Done:[/] {len(created_notes)} notes created. "+ f"Start with: hyperresearch note show {created_notes[-1]['note_id']} -j"+ )+++def _save_result(vault, conn, prov, result, tags, parent) -> dict | None:+ """Save a single WebResult as a note. Returns note data dict or None if skipped."""+ from hyperresearch.core.note import write_note++ url = result.url++ # Skip if already fetched+ existing = conn.execute("SELECT note_id FROM sources WHERE url = ?", (url,)).fetchone()+ if existing:+ return None++ title = result.title or urlparse(url).path.split("/")[-1] or "Untitled"+ domain = result.domain+ content_hash = hashlib.sha256(result.content.encode("utf-8")).hexdigest()[:16]++ extra_meta = {+ "source": url,+ "source_domain": domain,+ "fetched_at": result.fetched_at.isoformat(),+ "fetch_provider": prov.name,+ }++ note_path = write_note(+ vault.notes_dir,+ title=title,+ body=result.content,+ tags=tags,+ status="draft",+ source=url,+ parent=parent,+ extra_frontmatter=extra_meta,+ )++ note_id = note_path.stem+ conn.execute(+ """INSERT OR IGNORE INTO sources (url, note_id, domain, fetched_at, provider, content_hash)+ VALUES (?, ?, ?, ?, ?, ?)""",+ (url, note_id, domain, result.fetched_at.isoformat(), prov.name, content_hash),+ )+ conn.commit()++ return {+ "note_id": note_id,+ "title": title,+ "url": url,+ "domain": domain,+ "path": str(note_path.relative_to(vault.root)),+ "word_count": len(result.content.split()),+ "summary": result.content[:120].replace("\n", " ").strip(),+ }+++def _extract_links_from_results(results) -> list[str]:+ """Extract outbound HTTP links from result content."""+ urls = []+ url_re = re.compile(r'https?://[^\s<>\[\]"\']+')+ for r in results:+ for match in url_re.finditer(r.content):+ url = match.group(0).rstrip(".,;:)")+ # Skip common noise+ parsed = urlparse(url)+ if parsed.netloc and not any(+ skip in parsed.netloc+ for skip in ["google.com", "facebook.com", "twitter.com", "youtube.com"]+ ):+ urls.append(url)+ return list(dict.fromkeys(urls)) # Dedupe, preserve orderdiff --git a/src/hyperresearch/cli/sources.py b/src/hyperresearch/cli/sources.pynew file mode 100644index 0000000..21a7f6e--- /dev/null+++ b/src/hyperresearch/cli/sources.py@@ -0,0 +1,97 @@+"""Sources command — list and manage fetched web sources."""++from __future__ import annotations++import typer++from hyperresearch.cli._output import console, output+from hyperresearch.models.output import error, success++app = typer.Typer()++[email protected]("list")+def source_list(+ domain: str | None = typer.Option(None, "--domain", help="Filter by domain"),+ limit: int = typer.Option(50, "--limit", "-n", help="Max results"),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """List all fetched web sources."""+ from hyperresearch.core.vault import Vault, VaultError++ try:+ vault = Vault.discover()+ except VaultError as e:+ if json_output:+ output(error(str(e), "NO_VAULT"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {e}")+ raise typer.Exit(1)++ conn = vault.db++ if domain:+ rows = conn.execute(+ "SELECT url, note_id, domain, fetched_at, provider, status "+ "FROM sources WHERE domain = ? ORDER BY fetched_at DESC LIMIT ?",+ (domain, limit),+ ).fetchall()+ else:+ rows = conn.execute(+ "SELECT url, note_id, domain, fetched_at, provider, status "+ "FROM sources ORDER BY fetched_at DESC LIMIT ?",+ (limit,),+ ).fetchall()++ sources = [dict(row) for row in rows]++ if json_output:+ output(success(sources, count=len(sources), vault=str(vault.root)), json_mode=True)+ else:+ if not sources:+ console.print("[dim]No sources fetched yet. Use 'hyperresearch fetch <url>' to start.[/]")+ return+ for s in sources:+ status_color = "green" if s["status"] == "active" else "red"+ console.print(+ f"[{status_color}]{s['status']}[/] {s['url']}"+ f" → [cyan]{s['note_id'] or '(deleted)'}[/]"+ f" [dim]({s['provider']}, {s['fetched_at']})[/]"+ )+ console.print(f"\n[dim]{len(sources)} sources[/]")++[email protected]("check")+def source_check(+ url: str = typer.Argument(..., help="URL to check"),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """Check if a URL has already been fetched."""+ from hyperresearch.core.vault import Vault, VaultError++ try:+ vault = Vault.discover()+ except VaultError as e:+ if json_output:+ output(error(str(e), "NO_VAULT"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {e}")+ raise typer.Exit(1)++ row = vault.db.execute(+ "SELECT url, note_id, domain, fetched_at, provider FROM sources WHERE url = ?",+ (url,),+ ).fetchone()++ if row:+ data = {"exists": True, **dict(row)}+ if json_output:+ output(success(data, vault=str(vault.root)), json_mode=True)+ else:+ console.print(f"[green]Found:[/] {url} → note '{row['note_id']}'")+ else:+ data = {"exists": False, "url": url}+ if json_output:+ output(success(data, vault=str(vault.root)), json_mode=True)+ else:+ console.print(f"[dim]Not fetched:[/] {url}")diff --git a/src/hyperresearch/mcp/server.py b/src/hyperresearch/mcp/server.pyindex 3c12804..b4fa87d 100644--- a/src/hyperresearch/mcp/server.py+++ b/src/hyperresearch/mcp/server.py@@ -233,3 +233,44 @@ def lint_vault(rule: str = "") -> str:for r in conn.execute("SELECT id FROM notes WHERE type NOT IN ('index','raw') AND id NOT IN (SELECT DISTINCT target_id FROM links WHERE target_id IS NOT NULL) AND id NOT IN (SELECT DISTINCT source_id FROM links)"):issues.append({"rule": "orphaned-notes", "severity": "info", "note_id": r["id"], "message": "No links"})return json.dumps({"issues": issues, "total": len(issues), "warnings": sum(1 for i in issues if i["severity"] == "warning")})+++def check_source(url: str) -> str:+ """Check if a URL has already been fetched into the research base.++ Args:+ url: The URL to check+ """+ vault = _get_vault()+ row = vault.db.execute(+ "SELECT url, note_id, domain, fetched_at, provider FROM sources WHERE url = ?",+ (url,),+ ).fetchone()+ if row:+ return json.dumps({"exists": True, **dict(row)})+ return json.dumps({"exists": False, "url": url})+++def list_sources(domain: str = "", limit: int = 50) -> str:+ """List fetched web sources, optionally filtered by domain.++ Args:+ domain: Filter by domain (e.g. "arxiv.org"). Leave empty for all.+ limit: Max results (default 50)+ """+ vault = _get_vault()+ if domain:+ rows = vault.db.execute(+ "SELECT url, note_id, domain, fetched_at, provider, status "+ "FROM sources WHERE domain = ? ORDER BY fetched_at DESC LIMIT ?",+ (domain, limit),+ ).fetchall()+ else:+ rows = vault.db.execute(+ "SELECT url, note_id, domain, fetched_at, provider, status "+ "FROM sources ORDER BY fetched_at DESC LIMIT ?",+ (limit,),+ ).fetchall()+ return json.dumps([dict(r) for r in rows])diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.pyindex c5ffd0a..84b707c 100644--- a/src/hyperresearch/web/base.py+++ b/src/hyperresearch/web/base.py@@ -3,7 +3,7 @@from __future__ import annotationsfrom dataclasses import dataclass, field-from datetime import datetime, timezone+from datetime import UTC, datetimefrom typing import Protocol, runtime_checkable@@ -14,7 +14,7 @@ class WebResult:url: strtitle: strcontent: str # clean markdown or plain text- fetched_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))+ fetched_at: datetime = field(default_factory=lambda: datetime.now(UTC))raw_html: str | None = Nonemetadata: dict = field(default_factory=dict) # author, date, domain, etc.diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.pyindex b1d739f..de7b24b 100644--- a/src/hyperresearch/web/builtin.py+++ b/src/hyperresearch/web/builtin.py@@ -3,7 +3,7 @@from __future__ import annotationsimport re-from datetime import datetime, timezone+from datetime import UTC, datetimefrom html.parser import HTMLParserfrom hyperresearch.web.base import WebResult@@ -60,7 +60,7 @@ class BuiltinProvider:title=title,content=content,raw_html=html,- fetched_at=datetime.now(timezone.utc),+ fetched_at=datetime.now(UTC),)def search(self, query: str, max_results: int = 5) -> list[WebResult]:diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.pyindex 4779931..0c796ca 100644--- a/src/hyperresearch/web/crawl4ai_provider.py+++ b/src/hyperresearch/web/crawl4ai_provider.py@@ -2,9 +2,9 @@from __future__ import annotations-from datetime import datetime, timezone+from datetime import UTC, datetime-from crawl4ai import CrawlerStrategy, WebCrawler+from crawl4ai import WebCrawlerfrom hyperresearch.web.base import WebResult@@ -23,7 +23,7 @@ class Crawl4AIProvider:title=result.metadata.get("title", "") if result.metadata else "",content=result.markdown or result.extracted_content or "",raw_html=result.html,- fetched_at=datetime.now(timezone.utc),+ fetched_at=datetime.now(UTC),metadata=result.metadata or {},)diff --git a/src/hyperresearch/web/firecrawl_provider.py b/src/hyperresearch/web/firecrawl_provider.pyindex a5d9188..82cba9e 100644--- a/src/hyperresearch/web/firecrawl_provider.py+++ b/src/hyperresearch/web/firecrawl_provider.py@@ -3,7 +3,7 @@from __future__ import annotationsimport os-from datetime import datetime, timezone+from datetime import UTC, datetimefrom firecrawl import FirecrawlApp@@ -30,7 +30,7 @@ class FirecrawlProvider:title=result.get("metadata", {}).get("title", ""),content=result.get("markdown", ""),raw_html=result.get("html"),- fetched_at=datetime.now(timezone.utc),+ fetched_at=datetime.now(UTC),metadata=result.get("metadata", {}),)diff --git a/src/hyperresearch/web/tavily_provider.py b/src/hyperresearch/web/tavily_provider.pyindex 56dea05..01a75ee 100644--- a/src/hyperresearch/web/tavily_provider.py+++ b/src/hyperresearch/web/tavily_provider.py@@ -3,7 +3,7 @@from __future__ import annotationsimport os-from datetime import datetime, timezone+from datetime import UTC, datetimefrom tavily import TavilyClient@@ -30,7 +30,7 @@ class TavilyProvider:url=url,title=r.get("title", ""),content=r.get("raw_content", r.get("text", "")),- fetched_at=datetime.now(timezone.utc),+ fetched_at=datetime.now(UTC),metadata={"source": "tavily-extract"},)raise RuntimeError(f"Tavily extract returned no results for {url}")@@ -48,7 +48,7 @@ class TavilyProvider:url=r["url"],title=r.get("title", ""),content=r.get("raw_content", r.get("content", "")),- fetched_at=datetime.now(timezone.utc),+ fetched_at=datetime.now(UTC),metadata={"score": r.get("score", 0), "source": "tavily-search"},))diff --git a/src/hyperresearch/web/trafilatura_provider.py b/src/hyperresearch/web/trafilatura_provider.pyindex 74fe277..a9386ea 100644--- a/src/hyperresearch/web/trafilatura_provider.py+++ b/src/hyperresearch/web/trafilatura_provider.py@@ -2,7 +2,7 @@from __future__ import annotations-from datetime import datetime, timezone+from datetime import UTC, datetimeimport trafilatura@@ -38,7 +38,7 @@ class TrafilaturaProvider:title=title,content=content,raw_html=downloaded,- fetched_at=datetime.now(timezone.utc),+ fetched_at=datetime.now(UTC),metadata=meta_dict,)diff --git a/tests/test_core/test_frontmatter.py b/tests/test_core/test_frontmatter.pyindex 9fd3510..1e352b8 100644--- a/tests/test_core/test_frontmatter.py+++ b/tests/test_core/test_frontmatter.py@@ -1,6 +1,6 @@"""Tests for frontmatter parsing and serialization."""-from hyperresearch.core.frontmatter import parse_frontmatter, serialize_frontmatter, render_note+from hyperresearch.core.frontmatter import parse_frontmatter, render_note, serialize_frontmatterfrom hyperresearch.models.note import NoteMeta@@ -22,7 +22,7 @@ def test_parse_missing_frontmatter():def test_parse_empty_frontmatter():content = "---\n---\n\nBody here."- meta, body = parse_frontmatter(content)+ meta, _body = parse_frontmatter(content)assert meta.title == "Untitled"diff --git a/tests/test_core/test_note.py b/tests/test_core/test_note.pyindex 7cf1c0a..e661377 100644--- a/tests/test_core/test_note.py+++ b/tests/test_core/test_note.py@@ -1,8 +1,7 @@"""Tests for note read/write operations."""-from pathlib import Path-from hyperresearch.core.note import read_note, write_note, strip_markdown+from hyperresearch.core.note import read_note, strip_markdown, write_notefrom hyperresearch.models.note import slugifydiff --git a/tests/test_graph/test_links.py b/tests/test_graph/test_links.pyindex dc67713..942aaa4 100644--- a/tests/test_graph/test_links.py+++ b/tests/test_graph/test_links.py@@ -1,6 +1,6 @@"""Tests for link parsing and graph operations."""-from hyperresearch.core.note import WIKI_LINK_RE, CODE_BLOCK_RE, INLINE_CODE_RE+from hyperresearch.core.note import CODE_BLOCK_RE, INLINE_CODE_RE, WIKI_LINK_REdef test_wiki_link_regex_simple():diff --git a/tests/test_search/test_fts.py b/tests/test_search/test_fts.pyindex 68869fb..dd1afc5 100644--- a/tests/test_search/test_fts.py+++ b/tests/test_search/test_fts.py@@ -1,7 +1,7 @@"""Tests for full-text search."""-from hyperresearch.search.fts import search_fts, preprocess_queryfrom hyperresearch.search.filters import SearchFilters+from hyperresearch.search.fts import preprocess_query, search_ftsdef test_preprocess_simple_query():