Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

fix(mcp): run fetch_url's network fetch off the event loop (#118) (#181)

Jordan Gibbs36c59eb5bd62
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ce3ed7a..cda7a1a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,7 @@
- **`run resume` follows the tier step 1 declared.** A run is initialized with the installed gear's step list, and step 1 can reclassify it (usually to `light`) in `prompt-decomposition.json`. The ship gate already honoured that tier; `resume_position` did not, so a light-tier run that finished step 2 was told to resume at step 3, a step its tier skips. `run resume`, `run status` and the Codex Stop gate now use the declared tier's steps. Found during the first live Codex run.
- **Codex bench harness rebuilt (`codex_bench/`).** Each query runs in its own git-initialized project set up by `install --target codex`, with the prompt in `research/prompt.txt` and the inline-citation wrapper contract. The session is one `codex exec` under the workspace-write-plus-network sandbox, with JSONL events captured. `result.json` records whether the pipeline ran: vault note count, manifest steps done and not done, tier, subagent spawns, sandbox rejections, and `run verify`. An inline answer that skipped the vault is reported as `PIPELINE_SKIPPED`. `--dry-run` sets up the run dir and prints the exact `codex exec` command without running it. There are no dollar fields.
- **Run spend counters and count ceilings (#92).** `hpr run step --status done` and `hpr run reconcile` count tagged notes and distinct fetched URLs linked to those notes in the vault DB, including HTML and PDFs. Optional `hpr run init --max-sources` / `--max-notes` ceilings block at step boundaries. `run resume` or `run step --status done --force` can clear a count block up to three times; the same count stays clear until it increases, and a fourth clear rejects the step. Agent-reported `estimated_usd` stays unchanged; `--budget` stays inert unless something calls `hpr run spend`. Real runs get count ceilings only when the entry skill passes them to `run init`.
+- **The MCP `fetch_url` tool no longer blocks the server during a fetch (#118, reported by @AmirF194).** It was a sync tool, so FastMCP ran it on the event loop's thread, and the crawl4ai provider joined a helper thread there for the whole fetch; the server answered nothing else until it returned. `fetch_url` is now async and runs only the provider's `fetch` in a worker thread. The duplicate check and the note and `sources` writes stay on the loop thread, because the vault's sqlite connection only works on the thread that opened it. `fetch_and_save` is unchanged for its callers; it is now `prepare_fetch` -> `prov.fetch` -> `save_fetched`, which the tool calls in pieces. `fetch_many` is not an MCP tool and only the CLI batch path uses it.
### Fixed
diff --git a/src/hyperresearch/core/fetcher.py b/src/hyperresearch/core/fetcher.py
index 69222cc..03c163a 100644
--- a/src/hyperresearch/core/fetcher.py
+++ b/src/hyperresearch/core/fetcher.py
@@ -99,11 +99,32 @@ def fetch_and_save(
ValueError: If URL is already fetched.
RuntimeError: If fetch fails, or sync does not index the new note.
"""
- from hyperresearch.core.note import indexed_note_ids, write_note
- from hyperresearch.core.sync import compute_sync_plan, execute_sync
+ prov = prepare_fetch(vault, url, provider_name=provider_name, visible=visible)
+ result = prov.fetch(url)
+ return save_fetched(
+ vault, url, prov, result,
+ tags=tags, title=title, parent=parent, save_assets=save_assets,
+ )
+
+
+def prepare_fetch(
+ vault,
+ url: str,
+ provider_name: str | None = None,
+ visible: bool = False,
+):
+ """Refuse a duplicate URL and build the provider that will fetch it.
+
+ ``fetch_and_save`` is prepare -> ``prov.fetch(url)`` -> ``save_fetched``.
+ The split lets the MCP server run only the network fetch in a worker
+ thread while the DB work stays on its loop thread, where the sqlite
+ connection was opened (#118).
+
+ Raises:
+ ValueError: If URL is already fetched.
+ """
from hyperresearch.web.base import get_provider
- tags = tags or []
conn = vault.db
# Check if URL already fetched (an orphaned row — note deleted — is not a duplicate)
@@ -128,8 +149,29 @@ def fetch_and_save(
settings=vault.config.fetch,
gates=vault.config.junk,
)
+ return prov
- result = prov.fetch(url)
+
+def save_fetched(
+ vault,
+ url: str,
+ prov,
+ result,
+ tags: list[str] | None = None,
+ title: str | None = None,
+ parent: str | None = None,
+ save_assets: bool = False,
+) -> dict:
+ """Gate a fetched ``result`` and save it as a research note. Returns result dict.
+
+ Raises:
+ RuntimeError: If the result is a login wall or junk.
+ """
+ from hyperresearch.core.note import indexed_note_ids, write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ tags = tags or []
+ conn = vault.db
# Detect login redirects — abort, but escalate to the browser lane
if result.looks_like_login_wall(url, vault.config.junk):
diff --git a/src/hyperresearch/mcp/server.py b/src/hyperresearch/mcp/server.py
index 7f13ea3..633d875 100644
--- a/src/hyperresearch/mcp/server.py
+++ b/src/hyperresearch/mcp/server.py
@@ -296,7 +296,7 @@ def list_sources(domain: str = "", limit: int = 50) -> str:
@server.tool()
-def fetch_url(url: str, tags: str = "", provider: str = "") -> str:
+async def fetch_url(url: str, tags: str = "", provider: str = "") -> str:
"""Fetch a URL and save it as a research note.
Args:
@@ -304,17 +304,21 @@ def fetch_url(url: str, tags: str = "", provider: str = "") -> str:
tags: Comma-separated tags (e.g. "ml,transformers")
provider: Web provider override (leave empty for default)
"""
- from hyperresearch.core.fetcher import fetch_and_save
+ import anyio
+
+ from hyperresearch.core.fetcher import prepare_fetch, save_fetched
vault = _get_vault()
vault.auto_sync()
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
try:
- result = fetch_and_save(
- vault, url, tags=tag_list,
- provider_name=provider or None,
- )
+ # Only the network fetch leaves the loop thread (#118): a sync fetch here
+ # stalls every other request, and the vault's sqlite connection only
+ # works on the thread that opened it, so prepare/save stay on this one.
+ prov = prepare_fetch(vault, url, provider_name=provider or None)
+ fetched = await anyio.to_thread.run_sync(prov.fetch, url)
+ result = save_fetched(vault, url, prov, fetched, tags=tag_list)
return json.dumps({"ok": True, "data": result})
except ValueError as e:
return json.dumps({"ok": False, "error": str(e), "error_code": "DUPLICATE_URL"})
diff --git a/tests/test_mcp/test_fetch_url.py b/tests/test_mcp/test_fetch_url.py
new file mode 100644
index 0000000..96cb6b6
--- /dev/null
+++ b/tests/test_mcp/test_fetch_url.py
@@ -0,0 +1,123 @@
+"""The MCP fetch_url tool must not block the server's event loop (#118).
+
+The fake provider's fetch blocks until another coroutine on the same loop
+releases it. If fetch_url ran the fetch on the loop thread, that coroutine
+could never run and the fetch would time out.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import threading
+from pathlib import Path
+
+import pytest
+from typer.testing import CliRunner
+
+from hyperresearch.cli import app
+from hyperresearch.web.base import WebResult
+
+mcp = pytest.importorskip("hyperresearch.mcp.server")
+
+runner = CliRunner()
+
+
+def mcp_vault(tmp_path: Path) -> Path:
+ vault_dir = tmp_path / "kb"
+ runner.invoke(app, ["init", str(vault_dir)])
+ os.chdir(vault_dir)
+ mcp._vault = None # module-global cache; force rediscovery under tmp_path
+ yield vault_dir
+ mcp._vault = None
+
+
+class _BlockingProvider:
+ name = "fake"
+
+ def __init__(self) -> None:
+ self.started = threading.Event()
+ self.release = threading.Event()
+ self.fetch_thread: int | None = None
+
+ def fetch(self, url: str) -> WebResult:
+ self.fetch_thread = threading.get_ident()
+ self.started.set()
+ if not self.release.wait(timeout=5):
+ raise RuntimeError("fetch was never released: the event loop was blocked")
+ return WebResult(url=url, title="Fake Title", content="enough content to clear the junk gate. " * 10)
+
+
+def _run_on_a_fresh_thread(coro):
+ """Run ``coro`` on a new thread with its own loop.
+
+ See tests/test_web/test_fetch_many_fallback.py's ``_run``: an earlier test
+ can leave the main thread's asyncio state marked running. The vault is
+ opened inside ``coro`` too, so its sqlite connection belongs to that loop's
+ thread, as it does in the server.
+ """
+ box: dict = {}
+
+ def target() -> None:
+ try:
+ box["result"] = asyncio.run(coro)
+ except BaseException as exc:
+ box["error"] = exc
+
+ thread = threading.Thread(target=target)
+ thread.start()
+ thread.join()
+ if "error" in box:
+ raise box["error"]
+ return box["result"]
+
+
+def test_fetch_url_keeps_the_event_loop_responsive(mcp_vault, monkeypatch):
+ prov = _BlockingProvider()
+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: prov)
+ url = "https://example.com/slow-page"
+
+ async def scenario():
+ loop_thread = threading.get_ident()
+
+ async def release_once_started() -> None:
+ # Polls on the loop; only runs at all if fetch_url yielded it.
+ for _ in range(500):
+ if prov.started.is_set():
+ prov.release.set()
+ return
+ await asyncio.sleep(0.01)
+ raise AssertionError("fetch never started")
+
+ out, _ = await asyncio.gather(mcp.fetch_url(url), release_once_started())
+ return loop_thread, out
+
+ loop_thread, out = _run_on_a_fresh_thread(scenario())
+ out = json.loads(out)
+
+ assert out["ok"] is True, out
+ assert prov.fetch_thread != loop_thread
+ assert (mcp_vault / out["data"]["path"]).exists()
+
+
+def test_fetch_url_saves_the_source_row_on_the_loop_thread(mcp_vault, monkeypatch):
+ """The sources INSERT uses the loop thread's sqlite connection; a second
+ fetch of the same URL sees it and is refused before any network call."""
+ prov = _BlockingProvider()
+ prov.release.set()
+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: prov)
+ url = "https://example.com/once"
+
+ async def fetch_twice():
+ first = json.loads(await mcp.fetch_url(url))
+ prov.started.clear()
+ second = json.loads(await mcp.fetch_url(url))
+ return first, second
+
+ first, second = _run_on_a_fresh_thread(fetch_twice())
+ assert first["ok"] is True, first
+ assert second["ok"] is False
+ assert second["error_code"] == "DUPLICATE_URL"
+ assert not prov.started.is_set()