Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix(mcp): run fetch_url's network fetch off the event loop (#118) (#181)
Jordan Gibbs36c59eb5bd62
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ce3ed7a..cda7a1a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,7 @@- **`run resume` follows the tier step 1 declared.** A run is initialized with the installed gear's step list, and step 1 can reclassify it (usually to `light`) in `prompt-decomposition.json`. The ship gate already honoured that tier; `resume_position` did not, so a light-tier run that finished step 2 was told to resume at step 3, a step its tier skips. `run resume`, `run status` and the Codex Stop gate now use the declared tier's steps. Found during the first live Codex run.- **Codex bench harness rebuilt (`codex_bench/`).** Each query runs in its own git-initialized project set up by `install --target codex`, with the prompt in `research/prompt.txt` and the inline-citation wrapper contract. The session is one `codex exec` under the workspace-write-plus-network sandbox, with JSONL events captured. `result.json` records whether the pipeline ran: vault note count, manifest steps done and not done, tier, subagent spawns, sandbox rejections, and `run verify`. An inline answer that skipped the vault is reported as `PIPELINE_SKIPPED`. `--dry-run` sets up the run dir and prints the exact `codex exec` command without running it. There are no dollar fields.- **Run spend counters and count ceilings (#92).** `hpr run step --status done` and `hpr run reconcile` count tagged notes and distinct fetched URLs linked to those notes in the vault DB, including HTML and PDFs. Optional `hpr run init --max-sources` / `--max-notes` ceilings block at step boundaries. `run resume` or `run step --status done --force` can clear a count block up to three times; the same count stays clear until it increases, and a fourth clear rejects the step. Agent-reported `estimated_usd` stays unchanged; `--budget` stays inert unless something calls `hpr run spend`. Real runs get count ceilings only when the entry skill passes them to `run init`.+- **The MCP `fetch_url` tool no longer blocks the server during a fetch (#118, reported by @AmirF194).** It was a sync tool, so FastMCP ran it on the event loop's thread, and the crawl4ai provider joined a helper thread there for the whole fetch; the server answered nothing else until it returned. `fetch_url` is now async and runs only the provider's `fetch` in a worker thread. The duplicate check and the note and `sources` writes stay on the loop thread, because the vault's sqlite connection only works on the thread that opened it. `fetch_and_save` is unchanged for its callers; it is now `prepare_fetch` -> `prov.fetch` -> `save_fetched`, which the tool calls in pieces. `fetch_many` is not an MCP tool and only the CLI batch path uses it.### Fixeddiff --git a/src/hyperresearch/core/fetcher.py b/src/hyperresearch/core/fetcher.pyindex 69222cc..03c163a 100644--- a/src/hyperresearch/core/fetcher.py+++ b/src/hyperresearch/core/fetcher.py@@ -99,11 +99,32 @@ def fetch_and_save(ValueError: If URL is already fetched.RuntimeError: If fetch fails, or sync does not index the new note."""- from hyperresearch.core.note import indexed_note_ids, write_note- from hyperresearch.core.sync import compute_sync_plan, execute_sync+ prov = prepare_fetch(vault, url, provider_name=provider_name, visible=visible)+ result = prov.fetch(url)+ return save_fetched(+ vault, url, prov, result,+ tags=tags, title=title, parent=parent, save_assets=save_assets,+ )+++def prepare_fetch(+ vault,+ url: str,+ provider_name: str | None = None,+ visible: bool = False,+):+ """Refuse a duplicate URL and build the provider that will fetch it.++ ``fetch_and_save`` is prepare -> ``prov.fetch(url)`` -> ``save_fetched``.+ The split lets the MCP server run only the network fetch in a worker+ thread while the DB work stays on its loop thread, where the sqlite+ connection was opened (#118).++ Raises:+ ValueError: If URL is already fetched.+ """from hyperresearch.web.base import get_provider- tags = tags or []conn = vault.db# Check if URL already fetched (an orphaned row — note deleted — is not a duplicate)@@ -128,8 +149,29 @@ def fetch_and_save(settings=vault.config.fetch,gates=vault.config.junk,)+ return prov- result = prov.fetch(url)++def save_fetched(+ vault,+ url: str,+ prov,+ result,+ tags: list[str] | None = None,+ title: str | None = None,+ parent: str | None = None,+ save_assets: bool = False,+) -> dict:+ """Gate a fetched ``result`` and save it as a research note. Returns result dict.++ Raises:+ RuntimeError: If the result is a login wall or junk.+ """+ from hyperresearch.core.note import indexed_note_ids, write_note+ from hyperresearch.core.sync import compute_sync_plan, execute_sync++ tags = tags or []+ conn = vault.db# Detect login redirects — abort, but escalate to the browser laneif result.looks_like_login_wall(url, vault.config.junk):diff --git a/src/hyperresearch/mcp/server.py b/src/hyperresearch/mcp/server.pyindex 7f13ea3..633d875 100644--- a/src/hyperresearch/mcp/server.py+++ b/src/hyperresearch/mcp/server.py@@ -296,7 +296,7 @@ def list_sources(domain: str = "", limit: int = 50) -> str:@server.tool()-def fetch_url(url: str, tags: str = "", provider: str = "") -> str:+async def fetch_url(url: str, tags: str = "", provider: str = "") -> str:"""Fetch a URL and save it as a research note.Args:@@ -304,17 +304,21 @@ def fetch_url(url: str, tags: str = "", provider: str = "") -> str:tags: Comma-separated tags (e.g. "ml,transformers")provider: Web provider override (leave empty for default)"""- from hyperresearch.core.fetcher import fetch_and_save+ import anyio++ from hyperresearch.core.fetcher import prepare_fetch, save_fetchedvault = _get_vault()vault.auto_sync()tag_list = [t.strip() for t in tags.split(",") if t.strip()]try:- result = fetch_and_save(- vault, url, tags=tag_list,- provider_name=provider or None,- )+ # Only the network fetch leaves the loop thread (#118): a sync fetch here+ # stalls every other request, and the vault's sqlite connection only+ # works on the thread that opened it, so prepare/save stay on this one.+ prov = prepare_fetch(vault, url, provider_name=provider or None)+ fetched = await anyio.to_thread.run_sync(prov.fetch, url)+ result = save_fetched(vault, url, prov, fetched, tags=tag_list)return json.dumps({"ok": True, "data": result})except ValueError as e:return json.dumps({"ok": False, "error": str(e), "error_code": "DUPLICATE_URL"})diff --git a/tests/test_mcp/test_fetch_url.py b/tests/test_mcp/test_fetch_url.pynew file mode 100644index 0000000..96cb6b6--- /dev/null+++ b/tests/test_mcp/test_fetch_url.py@@ -0,0 +1,123 @@+"""The MCP fetch_url tool must not block the server's event loop (#118).++The fake provider's fetch blocks until another coroutine on the same loop+releases it. If fetch_url ran the fetch on the loop thread, that coroutine+could never run and the fetch would time out.+"""++from __future__ import annotations++import asyncio+import json+import os+import threading+from pathlib import Path++import pytest+from typer.testing import CliRunner++from hyperresearch.cli import app+from hyperresearch.web.base import WebResult++mcp = pytest.importorskip("hyperresearch.mcp.server")++runner = CliRunner()+++def mcp_vault(tmp_path: Path) -> Path:+ vault_dir = tmp_path / "kb"+ runner.invoke(app, ["init", str(vault_dir)])+ os.chdir(vault_dir)+ mcp._vault = None # module-global cache; force rediscovery under tmp_path+ yield vault_dir+ mcp._vault = None+++class _BlockingProvider:+ name = "fake"++ def __init__(self) -> None:+ self.started = threading.Event()+ self.release = threading.Event()+ self.fetch_thread: int | None = None++ def fetch(self, url: str) -> WebResult:+ self.fetch_thread = threading.get_ident()+ self.started.set()+ if not self.release.wait(timeout=5):+ raise RuntimeError("fetch was never released: the event loop was blocked")+ return WebResult(url=url, title="Fake Title", content="enough content to clear the junk gate. " * 10)+++def _run_on_a_fresh_thread(coro):+ """Run ``coro`` on a new thread with its own loop.++ See tests/test_web/test_fetch_many_fallback.py's ``_run``: an earlier test+ can leave the main thread's asyncio state marked running. The vault is+ opened inside ``coro`` too, so its sqlite connection belongs to that loop's+ thread, as it does in the server.+ """+ box: dict = {}++ def target() -> None:+ try:+ box["result"] = asyncio.run(coro)+ except BaseException as exc:+ box["error"] = exc++ thread = threading.Thread(target=target)+ thread.start()+ thread.join()+ if "error" in box:+ raise box["error"]+ return box["result"]+++def test_fetch_url_keeps_the_event_loop_responsive(mcp_vault, monkeypatch):+ prov = _BlockingProvider()+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: prov)+ url = "https://example.com/slow-page"++ async def scenario():+ loop_thread = threading.get_ident()++ async def release_once_started() -> None:+ # Polls on the loop; only runs at all if fetch_url yielded it.+ for _ in range(500):+ if prov.started.is_set():+ prov.release.set()+ return+ await asyncio.sleep(0.01)+ raise AssertionError("fetch never started")++ out, _ = await asyncio.gather(mcp.fetch_url(url), release_once_started())+ return loop_thread, out++ loop_thread, out = _run_on_a_fresh_thread(scenario())+ out = json.loads(out)++ assert out["ok"] is True, out+ assert prov.fetch_thread != loop_thread+ assert (mcp_vault / out["data"]["path"]).exists()+++def test_fetch_url_saves_the_source_row_on_the_loop_thread(mcp_vault, monkeypatch):+ """The sources INSERT uses the loop thread's sqlite connection; a second+ fetch of the same URL sees it and is refused before any network call."""+ prov = _BlockingProvider()+ prov.release.set()+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: prov)+ url = "https://example.com/once"++ async def fetch_twice():+ first = json.loads(await mcp.fetch_url(url))+ prov.started.clear()+ second = json.loads(await mcp.fetch_url(url))+ return first, second++ first, second = _run_on_a_fresh_thread(fetch_twice())+ assert first["ok"] is True, first+ assert second["ok"] is False+ assert second["error_code"] == "DUPLICATE_URL"+ assert not prov.started.is_set()