160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
What This Is
World Intelligence MCP Server: 160 tools across 30+ domains providing real-time global intelligence from free public APIs. Serves four interfaces: MCP stdio (for Claude Code/Cursor), a live Starlette dashboard with SSE, a Click CLI with Rich output, and a collector daemon for 24/7 vector store population. Python 3.11+, built with hatchling.
Commands
# Install
pip install -e ".[dev,dashboard]"
# Run MCP server (stdio mode)
world-intel-mcp
# Run tests (pytest-asyncio, auto mode)
pytest
pytest --cov=world_intel_mcp
pytest src/world_intel_mcp/tests/test_sources.py::test_fetch_market_quotes -v # single test
# CLI
intel markets # stock indices
intel earthquakes --min-mag 5.0 # USGS quakes
intel status # cache + circuit breaker health
# Dashboard (requires [dashboard] extra)
intel-dashboard --port 8501
# Or with .env auto-loaded:
./run-dashboard.sh
Architecture
Three consumers share the same source modules and infrastructure stack:
server.py (MCP stdio) ─┐ ┌─ VectorStore (Qdrant)
cli.py (Click CLI) ├─> sources/*.py ─> Fetcher ─> CircuitBreaker ─┤
dashboard/app.py (SSE) ─┘ analysis/*.py └─ Cache (SQLite)
collector.py (daemon) ──┘
Infrastructure layer (fetcher.py, cache.py, circuit_breaker.py):
Fetcher: Centralized async HTTP client (httpx). All external calls go throughget_json(),get_text(),get_xml()orget_bytes()(binary bodies, streamed under a size cap, never cached). Handles retries (2 max), per-source rate limiting, and stale-data fallback (never returns blank if old data exists). Failure logs and returned error strings pass throughredact(), which masks credential query parameters and the values of*_KEY/*_TOKEN/*_SECRET/*_PASSWORDenvironment variables.CircuitBreaker: Per-source tracking. 3 consecutive failures trips the breaker for 5 minutes. Each RSS feed gets its own breaker (rss:bbc_world).Cache: SQLite WAL-mode TTL cache.get()returns live data,get_stale()returns expired data for fallback.
Tool registry (tools/*.py + runtime.py, Phase 26): server.py is a thin shell; the 160 tools live in 20 domain modules under tools/, each exporting TOOLS (list of Tool definitions) and HANDLERS (tool name -> async def h(arguments)). Shared infrastructure (cache, circuit breaker, AOI store, vector store, fetcher) lives in runtime.py and is built lazily: importing any module opens no database, and the first access to any of those names builds the whole set once, under a module lock, and publishes it in one step (a failed build binds nothing and closes what it opened). Handlers reach it as runtime.fetcher etc. at call time; binding one at module level would open the databases at import. The cache's and AOI store's SQLite connections belong to the thread that made the first access, so make it on the thread that will use them (the MCP server does, in _run() before serving), not from asyncio.to_thread or an executor. tools/aggregate() enforces TOOLS/HANDLERS parity and cross-module name uniqueness at import time; a drifted registry refuses to start.
Source modules (sources/*.py): Each module exports async def fetch_*(fetcher: Fetcher, **kwargs) -> dict. Pure data fetching — no MCP awareness. 30+ modules covering markets, seismology, military, cyber, health, tech, environmental, etc.
Analysis modules (analysis/*.py): Cross-domain intelligence that consumes outputs from multiple sources. Includes signal aggregation, instability indexing, NLP (entity extraction, classification, clustering, spike detection via Welford's algorithm), and strategic synthesis.
Static config (config/*.py): Curated datasets — 22 intel hotspots, 70+ military bases, 40 ports, 24 pipelines, 24 nuclear facilities, 34 undersea cables, 48 AI datacenters, 27 spaceports, 27 mineral deposits, 82 stock exchanges, 105 major cities, 28 world leaders, 36 APT groups.
Vector store (vector_store.py): Optional Qdrant + FastEmbed (BAAI/bge-small-en-v1.5, 384-dim ONNX). Async background worker queue (asyncio.Queue + asyncio.to_thread()) for non-blocking storage. Fetcher auto-stores on fresh HTTP fetches. 65+ source name → category mappings for filtered search.
Collector (collector.py): Standalone daemon for 24/7 vector store population. 61 sources organized by domain, with --daemon, --interval, --sources CLI args. Dynamic import via _import_fetch_fn(). Per-source timeout overrides in SOURCE_TIMEOUTS (the aoi_digest sweep gets 240 s; everything else the flat 45 s). Entry point: intel-collector.
Observation history (history.py, history_extractors.py, history_backfill.py, history_timeline.py): every collector capture is written to the same SQLite database as the cache: a run log per source (status, reason, stale replays), deduplicated payload bodies, and observations with three dates (observed_at the period described, published_at the release when known, collected_at when saved). Feeds intel_collection_health, intel_series_history, intel_capture_history, intel_revisions, intel_lagged_connections, intel_day_timeline and intel_history_backfill (FRED and ECB). Retention and caps are set by the WORLD_INTEL_HISTORY_* variables (README). Nothing exists before the collector's first recorded cycle except backfilled series.
Research guide (guide.py, guide_prompts.py): the MCP prompts (29 worked research questions) and resources (intel://guide, generated from the live registry; intel://guide/measures; intel://guide/source-health).
Reports (reports.py): PDF/HTML intelligence report generator. Collects 18 domains in parallel, renders styled HTML, converts to PDF via WeasyPrint. Optional dependency: pip install -e ".[pdf]".
Dashboard (dashboard/): Self-contained Starlette app with a single index.html template (no frontend build step). SSE endpoint streams all domains in parallel via asyncio.gather(), refreshes every 30 seconds. Loads .env from project root on startup. /api/aois feeds the AOI geofence layer from the shared AOI store: it opens a fresh AOIStore per request (_open_aoi_store) because SQLite connections are thread-bound and the handler may run on a different thread than a cached store's creator; it reads stored sweep snapshots and never gathers live. Endpoint tests use Starlette's TestClient without the context manager so lifespan (default-path Cache) never runs.
Adding a New Tool
- Create
sources/your_source.pywithasync def fetch_your_data(fetcher: Fetcher, **kwargs) -> dict - Use
fetcher.get_json(url, source="your-source", cache_key=..., cache_ttl=300)— this gives you caching, retries, circuit breaking, and rate limiting automatically - In the matching
tools/<domain>.py: append theTool(...)to that module'sTOOLSand add a handler to itsHANDLERS(async def h(arguments)calling your fetch viaruntime.fetcher);tools/aggregate()fails the import if the two disagree - Optionally add to
dashboard/app.py(SSE endpoint) andcli.py(Click command) - Add tests using
respxto mock HTTP (seetests/test_sources.pyfor pattern)
Key Patterns
- Source name string: The
sourceparameter infetcher.get_json()identifies the API for circuit breaking and rate limiting. Must match entries in_SOURCE_RATE_LIMITSif rate-limited (e.g.,"yahoo-finance","coingecko","adsblol","nominatim"). - GDELT needs its own timeout:
api.gdeltproject.orgTCP-connects instantly but its TLS handshake runs 17-20 s and sometimes longer (measured 2026-09-02), and the Fetcher's 15 s default counts TLS inside connect. Every GDELTget_jsoncall must passtimeout=GDELT_TIMEOUT(fromsources/news.py, 45 s) or it silently fails and serves stale cache. - AOI news is geographic (
sources/geocode.py): AOI briefs reverse-geocode the area centre with OSM Nominatim and search GDELT for the settlement/county, never the AOI's name. Nominatim policy: 1 req/s (_SOURCE_RATE_LIMITS), identifying User-Agent with contact URL, results cached 30 days. Tests must stubgeocode.fetch_place_context(the shared_patch_all_domainshelper intest_aoi.pydoes) — a suite that hits Nominatim live gets rate-limited within seconds. - Heatmap tools (
sources/jet_cohort.py,analysis/jet_activity.py): adsb.lol heatmaps are binary (~14 MB); fetch them withfetcher.get_bytes(sourceadsblol-history, 2 s floor) and the on-diskHeatmapDiskCache, never the SQLite TTL cache. Tests build synthetic heatmaps with the encoder intests/test_jet_activity.pyand zero the rate floor via monkeypatch. - Tool dispatch:
server.pylooks the name up intools.ALL_HANDLERS(built bytools.aggregate()from each module'sHANDLERS); there is nomatch/case. Tool names follow theintel_*convention, andtest_server_registry.pyfails if README.md's per-tool reference or its counts drift from the registry. - UNHCR codes are not ISO (
sources/migration.py,sources/displacement.py): UNHCR's owncoo/coacodes differ from ISO-3 (its "AUS" is Austria). Sendcf_type=ISOand readcoo_iso/coa_iso. Withoutcoo_all=truethe population endpoint returns one global row (coo_name"-"). Its country list also carries two non-countries, UNK "Unknown" and XXA "Stateless", which surge rankings must exclude. - All source functions take
fetcheras first arg — never construct your own httpx client. - Tests strip proxy env vars automatically via
conftest.pyfixture (prevents SOCKS proxy interference). The conftest also resets global fetcher rate-limit locks between tests to avoid cross-event-loop binding.
Environment Variables
Only these unlock additional data sources (everything else works unauthenticated):
ACLED_EMAIL/ACLED_PASSWORD: conflict events, unrest and risk scoresUCDP_ACCESS_TOKEN: UCDP organized violence events (the API refuses requests without it)TOMTOM_API_KEY: traffic flow and incidentsWINDY_API_KEY: webcamsOPENSKY_USERNAME/OPENSKY_PASSWORD: OpenSky account for the aviation toolsNASA_FIRMS_API_KEY— satellite wildfire dataEIA_API_KEY— energy pricesCLOUDFLARE_API_TOKEN— internet outage dataFRED_API_KEY— macro economic dataOPENSKY_CLIENT_ID/OPENSKY_CLIENT_SECRET— military flight fallbackOLLAMA_API_URL/OLLAMA_MODEL: Ollama server and model for AI-generated briefs (defaulthttp://localhost:11434/llama3.2)WORLD_INTEL_AOI_WEBHOOK/WORLD_INTEL_AOI_WEBHOOK_FORMAT— AOI sweep change notifications (collector daemon): POST on non-quiet sweeps; formatjson(default) ortext(raw markdown +Titleheader for ntfy-style sinks). Delivery record in the digest'snotificationfield; a dead sink never fails the sweep.
Testing
Tests use respx to mock httpx responses. Fixtures in conftest.py provide cache (tmp_path SQLite) and fetcher (with clean breaker). Tests are async (pytest-asyncio in auto mode). Proxy env vars are stripped automatically.