Skip to content

160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.

Runtime: build the shared set once under a lock and publish it whole

Phoenix Agent (mac-studio)7040ec4da993
diff --git a/CLAUDE.md b/CLAUDE.md
index 30e7b7e..dbf5f5f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -47,7 +47,7 @@ collector.py (daemon) ──┘
- `CircuitBreaker`: Per-source tracking. 3 consecutive failures trips the breaker for 5 minutes. Each RSS feed gets its own breaker (`rss:bbc_world`).
- `Cache`: SQLite WAL-mode TTL cache. `get()` returns live data, `get_stale()` returns expired data for fallback.
-**Tool registry** (`tools/*.py` + `runtime.py`, Phase 26): server.py is a thin shell; the 139 tools live in 12 domain modules under `tools/`, each exporting `TOOLS` (list of Tool definitions) and `HANDLERS` (tool name -> `async def h(arguments)`). Shared infrastructure (cache, circuit breaker, AOI store, vector store, fetcher) is constructed once in `runtime.py`; handlers reach it as `runtime.fetcher` etc. `tools/aggregate()` enforces TOOLS/HANDLERS parity and cross-module name uniqueness at import time — a drifted registry refuses to start.
+**Tool registry** (`tools/*.py` + `runtime.py`, Phase 26): server.py is a thin shell; the 139 tools live in 12 domain modules under `tools/`, each exporting `TOOLS` (list of Tool definitions) and `HANDLERS` (tool name -> `async def h(arguments)`). Shared infrastructure (cache, circuit breaker, AOI store, vector store, fetcher) lives in `runtime.py` and is built lazily: importing any module opens no database, and the first access to any of those names builds the whole set once, under a module lock, and publishes it in one step (a failed build binds nothing and closes what it opened). Handlers reach it as `runtime.fetcher` etc. at call time; binding one at module level would open the databases at import. The cache's and AOI store's SQLite connections belong to the thread that made the first access, so make it on the thread that will use them (the MCP server does, in `_run()` before serving), not from `asyncio.to_thread` or an executor. `tools/aggregate()` enforces TOOLS/HANDLERS parity and cross-module name uniqueness at import time — a drifted registry refuses to start.
**Source modules** (`sources/*.py`): Each module exports `async def fetch_*(fetcher: Fetcher, **kwargs) -> dict`. Pure data fetching — no MCP awareness. 30+ modules covering markets, seismology, military, cyber, health, tech, environmental, etc.
diff --git a/src/world_intel_mcp/runtime.py b/src/world_intel_mcp/runtime.py
index b829065..ab3937d 100644
--- a/src/world_intel_mcp/runtime.py
+++ b/src/world_intel_mcp/runtime.py
@@ -3,11 +3,19 @@
Extracted from server.py in the Phase 26 modularization so that the
domain tool modules under ``tools/`` and the server shell can share one
cache, one circuit breaker, one AOI store, and one fetcher without a
-circular import. Infrastructure is initialized on first access so importing
-modules does not open databases.
+circular import.
+
+Nothing is built at import. The first access to any shared name builds
+the whole set under a module lock and publishes it in one step, so
+concurrent first accesses all get the same objects, and a build that
+fails binds nothing and closes whatever it had opened. The SQLite
+connections behind the cache and the AOI store belong to the thread
+that made that first access; the MCP server makes it on its event-loop
+thread in ``server._run()`` before serving.
"""
import logging
+import threading
from .analysis import aoi
from .cache import Cache
@@ -16,40 +24,79 @@ from .fetcher import Fetcher
logger = logging.getLogger("world-intel-mcp")
+_SHARED = frozenset({
+ "cache", "breaker", "aoi_store", "_aoi_store",
+ "vector_store", "_vector_store", "fetcher",
+})
+_init_lock = threading.Lock()
+
def __getattr__(name: str):
- if name not in {"cache", "breaker", "aoi_store", "_aoi_store",
- "vector_store", "_vector_store", "fetcher"}:
+ if name not in _SHARED:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
- _initialize()
+ with _init_lock:
+ # Another thread may have published the set while this one waited.
+ if not _SHARED.issubset(globals()):
+ globals().update(_build())
return globals()[name]
def _initialize() -> None:
- global cache, breaker, aoi_store, _aoi_store, vector_store, _vector_store, fetcher
+ """Build a new set and publish it in place of any set already bound."""
+ with _init_lock:
+ globals().update(_build())
+
+
+def _build() -> dict:
+ """Construct every shared object as a local; bind nothing.
+
+ If a step fails, close what the earlier steps opened and re-raise, so
+ the module keeps whatever it had and the next access starts over.
+ """
+ opened = []
+ try:
+ cache = Cache()
+ opened.append(cache)
+ breaker = CircuitBreaker(failure_threshold=3, cooldown_seconds=300)
+ # Same physical SQLite file the Cache above resolved to (fallback path
+ # included); see analysis/aoi.py's module docstring for why AOIs live in
+ # a dedicated table there rather than as Cache entries.
+ aoi_store = aoi.AOIStore(cache.db_path)
+ opened.append(aoi_store)
+ vector_store = _optional_vector_store()
+ fetcher = Fetcher(cache=cache, breaker=breaker, vector_store=vector_store)
+ except BaseException:
+ for resource in reversed(opened):
+ try:
+ resource.close()
+ except Exception:
+ logger.warning(
+ "Could not close %r after a failed runtime build",
+ resource,
+ exc_info=True,
+ )
+ raise
+ return {
+ "cache": cache,
+ "breaker": breaker,
+ "aoi_store": aoi_store,
+ "_aoi_store": aoi_store,
+ "vector_store": vector_store,
+ "_vector_store": vector_store,
+ "fetcher": fetcher,
+ }
- cache = Cache()
- breaker = CircuitBreaker(failure_threshold=3, cooldown_seconds=300)
- # Same physical SQLite file the Cache above resolved to (fallback path
- # included); see analysis/aoi.py's module docstring for why AOIs live in
- # a dedicated table there rather than as Cache entries.
- _aoi_store = aoi.AOIStore(cache.db_path)
- aoi_store = _aoi_store
+def _optional_vector_store():
# Vector store — optional, degrades gracefully if Qdrant unavailable.
- _vector_store = None
try:
from .vector_store import VectorStore, vector_dependencies_available
if vector_dependencies_available():
- _vector_store = VectorStore(enabled=True)
- else:
- logger.info(
- "Vector store unavailable (qdrant_client / fastembed not installed)"
- )
+ return VectorStore(enabled=True)
+ logger.info(
+ "Vector store unavailable (qdrant_client / fastembed not installed)"
+ )
except Exception as exc:
logger.info("Vector store unavailable: %s", exc)
-
- vector_store = _vector_store
-
- fetcher = Fetcher(cache=cache, breaker=breaker, vector_store=_vector_store)
+ return None
diff --git a/src/world_intel_mcp/tests/test_runtime_init.py b/src/world_intel_mcp/tests/test_runtime_init.py
new file mode 100644
index 0000000..f62d655
--- /dev/null
+++ b/src/world_intel_mcp/tests/test_runtime_init.py
@@ -0,0 +1,260 @@
+"""The shared runtime is built once, on first access, as one consistent set.
+
+Each test starts from an unbound runtime (as a fresh process sees it) with
+the cache pointed at a temp file, and puts the session's objects back
+afterwards. The thread test swaps in recording stand-ins for the cache
+and AOI store: SQLite connections belong to the thread that opened them,
+and what is under test is the build protocol, not SQLite.
+"""
+
+import contextlib
+import socket
+import sqlite3
+import threading
+import time
+from pathlib import Path
+
+import pytest
+
+from world_intel_mcp import runtime
+from world_intel_mcp import vector_store as vector_store_mod
+from world_intel_mcp.analysis import aoi
+from world_intel_mcp.analysis.aoi import AOIStore
+from world_intel_mcp.cache import Cache
+from world_intel_mcp.circuit_breaker import CircuitBreaker
+from world_intel_mcp.fetcher import Fetcher
+
+# The names tools and the server read from runtime, spelled out here rather
+# than taken from runtime so the tests pin the contract instead of
+# mirroring it.
+_SHARED_NAMES = (
+ "cache", "breaker", "aoi_store", "_aoi_store",
+ "vector_store", "_vector_store", "fetcher",
+)
+
+
+def unbound_runtime(monkeypatch, tmp_path: Path):
+ """Nothing bound and the cache on a temp file; yields that path."""
+ cache_db = tmp_path / "cache.db"
+ monkeypatch.setenv("WORLD_INTEL_CACHE_DB", str(cache_db))
+ namespace = vars(runtime)
+ saved = {name: namespace.pop(name) for name in _SHARED_NAMES if name in namespace}
+ try:
+ yield cache_db
+ finally:
+ built = {name: namespace.pop(name, None) for name in _SHARED_NAMES}
+ namespace.update(saved)
+ for name in ("aoi_store", "cache"):
+ if isinstance(built[name], (AOIStore, Cache)):
+ built[name].close()
+
+
+def fake_vector_store(monkeypatch):
+ """Vector dependencies reported present; the store records its lifecycle."""
+ events: list[str] = []
+
+ class FakeVectorStore:
+ def __init__(self, enabled: bool = True):
+ events.append("vector store built")
+
+ async def start(self) -> None:
+ events.append("vector store started")
+
+ async def stop(self) -> None:
+ events.append("vector store stopped")
+
+ monkeypatch.setattr(vector_store_mod, "vector_dependencies_available", lambda: True)
+ monkeypatch.setattr(vector_store_mod, "VectorStore", FakeVectorStore)
+ return events
+
+
+def no_network(monkeypatch):
+ """Refuse every outbound connection and name lookup; yields the attempts."""
+ attempts: list[tuple] = []
+
+ def refuse(*args, **kwargs):
+ attempts.append(args)
+ raise OSError("network access is blocked in this test")
+
+ monkeypatch.setattr(socket.socket, "connect", refuse)
+ monkeypatch.setattr(socket.socket, "connect_ex", refuse)
+ monkeypatch.setattr(socket, "create_connection", refuse)
+ monkeypatch.setattr(socket, "getaddrinfo", refuse)
+ return attempts
+
+
+def test_first_access_publishes_the_whole_set_wired_together(unbound_runtime) -> None:
+ fetcher = runtime.fetcher
+
+ namespace = vars(runtime)
+ assert [name for name in _SHARED_NAMES if name not in namespace] == []
+ assert isinstance(fetcher, Fetcher)
+ assert namespace["fetcher"] is fetcher
+ assert runtime.fetcher is fetcher # bound once; later reads do not rebuild
+ assert fetcher.cache is runtime.cache
+ assert fetcher.breaker is runtime.breaker
+ assert fetcher.vector_store is runtime.vector_store
+ assert runtime._aoi_store is runtime.aoi_store
+ assert runtime._vector_store is runtime.vector_store
+ assert runtime.cache.db_path == unbound_runtime
+ assert runtime.aoi_store.db_path == unbound_runtime
+
+
+def test_concurrent_first_access_builds_one_consistent_set(
+ unbound_runtime, monkeypatch,
+) -> None:
+ caches, stores, breakers, fetchers = [], [], [], []
+
+ class SlowCache:
+ def __init__(self):
+ caches.append(self)
+ # Long enough for every thread to reach runtime before this
+ # returns, so an unguarded first access builds once per thread.
+ time.sleep(0.3)
+ self.db_path = unbound_runtime
+
+ def close(self) -> None:
+ pass
+
+ class RecordingAOIStore:
+ def __init__(self, db_path):
+ stores.append(self)
+ self.db_path = db_path
+
+ def close(self) -> None:
+ pass
+
+ class RecordingBreaker(CircuitBreaker):
+ def __init__(self, *args, **kwargs):
+ breakers.append(self)
+ super().__init__(*args, **kwargs)
+
+ class RecordingFetcher(Fetcher):
+ def __init__(self, *args, **kwargs):
+ fetchers.append(self)
+ super().__init__(*args, **kwargs)
+
+ monkeypatch.setattr(runtime, "Cache", SlowCache)
+ monkeypatch.setattr(runtime, "CircuitBreaker", RecordingBreaker)
+ monkeypatch.setattr(runtime, "Fetcher", RecordingFetcher)
+ monkeypatch.setattr(aoi, "AOIStore", RecordingAOIStore)
+ monkeypatch.setattr(vector_store_mod, "vector_dependencies_available", lambda: False)
+
+ names = ("fetcher", "breaker", "cache", "aoi_store", "vector_store", "fetcher")
+ barrier = threading.Barrier(len(names))
+ seen: list[object] = [None] * len(names)
+ errors: list[BaseException] = []
+
+ def first_access(index: int, name: str) -> None:
+ try:
+ barrier.wait(timeout=10)
+ seen[index] = getattr(runtime, name)
+ except BaseException as exc: # re-raised by the assertion below
+ errors.append(exc)
+
+ threads = [
+ threading.Thread(target=first_access, args=(index, name))
+ for index, name in enumerate(names)
+ ]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join(timeout=30)
+
+ assert [t.name for t in threads if t.is_alive()] == [], "a first access never returned"
+ assert errors == []
+ assert (len(caches), len(stores), len(breakers), len(fetchers)) == (1, 1, 1, 1)
+ fetcher, breaker, cache, aoi_store, vector_store, fetcher_again = seen
+ assert fetcher_again is fetcher
+ assert fetcher is runtime.fetcher
+ assert breaker is runtime.breaker
+ assert cache is runtime.cache
+ assert aoi_store is runtime.aoi_store
+ assert fetcher.cache is cache
+ assert fetcher.breaker is breaker
+ assert aoi_store.db_path == cache.db_path
+ assert vector_store is None and runtime.vector_store is None
+
+
+def test_failed_first_access_binds_nothing_and_closes_what_it_opened(
+ unbound_runtime, monkeypatch,
+) -> None:
+ caches: list[Cache] = []
+ store_attempts: list[Path] = []
+
+ class RecordingCache(Cache):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ caches.append(self)
+
+ class FlakyAOIStore(AOIStore):
+ def __init__(self, db_path):
+ store_attempts.append(db_path)
+ if len(store_attempts) == 1:
+ raise sqlite3.OperationalError("database is locked")
+ super().__init__(db_path)
+
+ monkeypatch.setattr(runtime, "Cache", RecordingCache)
+ monkeypatch.setattr(aoi, "AOIStore", FlakyAOIStore)
+
+ with pytest.raises(sqlite3.OperationalError, match="database is locked"):
+ runtime.fetcher
+ assert [name for name in _SHARED_NAMES if name in vars(runtime)] == []
+ assert len(caches) == 1
+ assert caches[0]._conn is None # the cache that build opened was closed
+
+ fetcher = runtime.fetcher # the next access starts over
+ assert len(caches) == 2
+ assert fetcher.cache is runtime.cache is caches[1]
+ assert fetcher.breaker is runtime.breaker
+ assert runtime.aoi_store.db_path == runtime.cache.db_path == unbound_runtime
+
+
+def test_server_aliases_forward_to_the_shared_runtime(
+ unbound_runtime, fake_vector_store,
+) -> None:
+ from world_intel_mcp import server
+
+ assert server.cache is runtime.cache
+ assert server.breaker is runtime.breaker
+ assert server.fetcher is runtime.fetcher
+ assert server._aoi_store is runtime.aoi_store
+ assert server._vector_store is runtime.vector_store
+ assert runtime.vector_store is not None
+ assert fake_vector_store == ["vector store built"]
+
+
+async def test_run_starts_the_vector_store_before_serving_and_stops_it_after(
+ unbound_runtime, fake_vector_store, no_network, monkeypatch,
+) -> None:
+ from world_intel_mcp import server
+
+ events = fake_vector_store
+
+ @contextlib.asynccontextmanager
+ async def fake_stdio_server():
+ events.append("stdio opened")
+ yield object(), object()
+ events.append("stdio closed")
+
+ async def fake_serve(read_stream, write_stream, initialization_options):
+ bound = all(name in vars(runtime) for name in _SHARED_NAMES)
+ events.append(f"serving, runtime bound: {bound}")
+
+ monkeypatch.setattr(server, "stdio_server", fake_stdio_server)
+ monkeypatch.setattr(server.server, "run", fake_serve)
+
+ await server._run()
+
+ assert events == [
+ "vector store built",
+ "vector store started",
+ "stdio opened",
+ "serving, runtime bound: True",
+ "stdio closed",
+ "vector store stopped",
+ ]
+ assert no_network == []
diff --git a/src/world_intel_mcp/tests/test_server_registry.py b/src/world_intel_mcp/tests/test_server_registry.py
index 75cf85e..fa7d80a 100644
--- a/src/world_intel_mcp/tests/test_server_registry.py
+++ b/src/world_intel_mcp/tests/test_server_registry.py
@@ -1,13 +1,15 @@
"""Import-based tests for server.py's tool registry and dispatcher.
-Every other test file reads server.py as *text* because its shared
-``Cache()``/``AOIStore()`` open at the default on-disk path. Here we
-import it for real, under a temp cache path (``WORLD_INTEL_CACHE_DB`` is
-honored by ``Cache``; the fixture binds the shared infrastructure while
-the override is in force), which is what lets these tests verify the
-registry structurally instead of by regex: the TOOLS/_dispatch parity
-invariant, schema well-formedness, and the actual dispatch path for the
-pure (no-network) tools.
+Importing server.py opens no database: the shared ``Cache``/``AOIStore``
+in runtime.py are built on first access, not at import. The fixture
+imports the module for real with ``WORLD_INTEL_CACHE_DB`` pointed at a
+temp file and then rebuilds the shared runtime under that override
+(``runtime._initialize()``), because an earlier test module may already
+have built it at another path; the first test guards that premise.
+Importing it is what lets these tests verify the registry structurally
+instead of by regex: the TOOLS/_dispatch parity invariant, schema
+well-formedness, and the actual dispatch path for the pure (no-network)
+tools.
"""
import importlib