160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.
World Intel MCP — Feature Parity Roadmap
Benchmark: koala73/worldmonitor Updated: 2026-10-18 (Unreleased, merged to main 2026-10-18: Phase 29 research-grade history and evidence, and US and UK border-crossing flows; 0.11.0: aircraft-cohort early warning, cross-border migration) Current tools: 160 (159 intel + 1 status)
Legend
| Icon | Meaning |
|---|---|
| :white_check_mark: | We have this |
| :yellow_circle: | Partial — we have the data source but lack the analysis layer |
| :red_circle: | Missing entirely |
0. Assessment Snapshot (2026-09-01; counts as of that date)
| Area | Finding | Status | Action |
|---|---|---|---|
| MCP tool parity | 128 tools declared in TOOLS; 128 routed in _dispatch() |
:white_check_mark: | Keep as an invariant (machine-checked by import-based tests) |
| Optional vector runtime | Missing qdrant-client / fastembed previously surfaced as runtime failures |
:white_check_mark: Fixed | Vector features now degrade cleanly and report availability |
| Base-environment test run | pytest -q fails collection without dev extras because respx is not installed |
:yellow_circle: | Run pip install -e ".[dev]" before full-suite validation |
| Test coverage truth | Was 59% overall on 2026-09-01 morning (server.py/cli.py/collector.py at 0%, analysis/ NLP modules 0-14%, sources/intelligence.py 22%). After the same-day test waves: 81% overall, 597 tests (analysis modules 96-100%, sources/intelligence.py 93%, server.py 47% via import-based registry tests). Remaining zeros: cli.py (1,076 stmts); collector.py at 20% (map verified, run loop untested) |
:yellow_circle: Improved | CI coverage gate that ratchets (Phase 24); cli.py smoke tests |
| Geofence correctness | Antimeridian AOIs lost the far side of the dateline (bbox clamp); pipelines/cables matched on endpoints only | :white_check_mark: Fixed in Phase 22 | Segment distance + split bboxes shipped with tests |
| Security posture | Issue #21 (external report) assessed; no shell/eval/exec, SQL parameterized throughout, report paths server-generated | :white_check_mark: | SECURITY.md added with explicit threat model; cache db now 0600 |
| Documentation drift | Prior roadmap documented 89/110 tools while the codebase now exposes 113 | :white_check_mark: Updated below | Keep roadmap synced with phase increments |
| Maintainability | server.py split into 12 domain modules + runtime.py (158-line shell); parity enforced at import time | :white_check_mark: Shipped 2026-09-01 (Phase 26) | Largest module 444 lines |
| CLI/dashboard parity for AOIs | Was MCP-only. intel aoi CLI group (9 subcommands) shipped in 0.8.0; dashboard AOI layer with last-sweep counts shipped in 0.10.0 |
:white_check_mark: Shipped | Phase 23 complete |
Implemented Addendum Missing From Prior Roadmap
Financial Intelligence Extensions (12 tools)
| Tool | Purpose | Status |
|---|---|---|
intel_forex_rates |
Latest FX rates by base + symbol filters | :white_check_mark: |
intel_forex_timeseries |
Historical FX series with configurable lookback | :white_check_mark: |
intel_major_crosses |
Major crosses + DXY proxy snapshot | :white_check_mark: |
intel_yield_curve |
Treasury curve + inversion analysis | :white_check_mark: |
intel_bond_indices |
Bond ETF summary (AGG, TLT, HYG, LQD, TIP) | :white_check_mark: |
intel_earnings_calendar |
Upcoming earnings calendar | :white_check_mark: |
intel_earnings_surprise |
Historical earnings surprise analysis | :white_check_mark: |
intel_sec_filings |
Full-text SEC EDGAR filing search | :white_check_mark: |
intel_company_filings |
Company-specific 10-K / 10-Q / 8-K retrieval | :white_check_mark: |
intel_recent_8k |
Recent material-event 8-K stream | :white_check_mark: |
intel_company_profile |
Composite company enrichment profile | :white_check_mark: |
intel_macro_composite |
Weighted market regime / macro composite score | :white_check_mark: |
Vector & Cross-Domain Analytics (8 tools)
| Tool | Purpose | Status |
|---|---|---|
intel_semantic_search |
Natural-language search across accumulated intelligence | :white_check_mark: |
intel_similar_events |
Similarity search against historical events | :white_check_mark: |
intel_timeline |
Chronological timeline from vector store history | :white_check_mark: |
intel_vector_stats |
Qdrant collection statistics | :white_check_mark: |
intel_collect |
On-demand collection cycle for vector population | :white_check_mark: |
intel_cross_correlate |
Cross-category correlation for a topic/query | :white_check_mark: |
intel_domain_summary |
Per-category summary of stored intelligence | :white_check_mark: |
intel_trend_detection |
Recent-vs-baseline activity surge/drop detection | :white_check_mark: |
Intelligence Reports (1 tool)
| Tool | Purpose | Status |
|---|---|---|
intel_generate_report |
Generate PDF/HTML multi-domain intelligence reports with HTML fallback when .[pdf] is unavailable |
:white_check_mark: |
1. Data Sources — Complete Inventory
The per-section catalog below covers 119 of the 160 registered tools; most later additions are named only in their phase entries, and README.md lists every registered tool.
Markets & Economics (16 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_market_quotes |
list-market-quotes |
:white_check_mark: |
intel_crypto_quotes |
list-crypto-quotes |
:white_check_mark: |
intel_stablecoin_status |
list-stablecoin-markets |
:white_check_mark: |
intel_etf_flows |
list-etf-flows |
:white_check_mark: |
intel_sector_heatmap |
get-sector-summary |
:white_check_mark: |
intel_macro_signals |
get-macro-signals |
:white_check_mark: |
intel_commodity_quotes |
list-commodity-quotes |
:white_check_mark: |
intel_gas_prices |
AAA retail fuel prices | :white_check_mark: |
intel_residential_natgas |
EIA residential natural gas prices | :white_check_mark: |
intel_electricity_rates |
EIA electricity rates by sector/state | :white_check_mark: |
intel_energy_prices |
get-energy-prices |
:white_check_mark: |
intel_fred_series |
get-fred-series |
:white_check_mark: |
intel_world_bank_indicators |
list-world-bank-indicators |
:white_check_mark: |
intel_country_stocks |
Country main index ticker | :white_check_mark: |
intel_btc_technicals |
BTC SMA-50/200, Mayer Multiple, cross signals | :white_check_mark: |
intel_central_bank_rates |
15 central bank policy rates (FRED + curated) | :white_check_mark: |
Natural Disasters & Climate (5 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_earthquakes |
list-earthquakes |
:white_check_mark: |
intel_wildfires |
list-fire-detections |
:white_check_mark: |
intel_climate_anomalies |
list-climate-anomalies |
:white_check_mark: |
intel_environmental_events |
NASA EONET events | :white_check_mark: |
intel_disaster_alerts |
GDACS global disaster alerts | :white_check_mark: |
Conflict & Security (4 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_acled_events |
list-acled-events |
:white_check_mark: |
intel_ucdp_events |
list-ucdp-events |
:white_check_mark: |
intel_unrest_events |
ACLED protests + GDELT dedup | :white_check_mark: |
intel_cyber_threats |
list-cyber-threats |
:white_check_mark: |
Military & Defense (8 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_military_flights |
list-military-flights |
:white_check_mark: |
intel_theater_posture |
get-theater-posture |
:white_check_mark: |
intel_aircraft_details |
get-aircraft-details |
:white_check_mark: |
intel_aircraft_batch |
Batch ICAO24 lookup | :white_check_mark: |
intel_vessel_snapshot |
get-vessel-snapshot |
:white_check_mark: |
intel_military_surge |
military-surge.ts |
:white_check_mark: |
intel_military_bases |
Static dataset (70 bases) | :white_check_mark: |
intel_usni_fleet |
USNI Fleet Tracker weekly disposition | :white_check_mark: |
Infrastructure & Maritime (6 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_internet_outages |
list-internet-outages |
:white_check_mark: |
intel_cable_health |
get-cable-health |
:white_check_mark: |
intel_nav_warnings |
list-navigational-warnings |
:white_check_mark: |
intel_cascade_analysis |
infrastructure-cascade.ts |
:white_check_mark: |
intel_strategic_ports |
Static dataset (40 ports) | :white_check_mark: |
intel_pipelines |
Static dataset (24 pipelines) | :white_check_mark: |
Humanitarian & Social (10 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_humanitarian_summary |
get-humanitarian-summary |
:white_check_mark: |
intel_displacement_summary |
get-displacement-summary |
:white_check_mark: (per-origin rows since 2026-10-11; before that one global row) |
intel_migration_flows |
none (beyond parity) | :white_check_mark: UNHCR origin -> destination corridors |
intel_migration_surges |
none (beyond parity) | :white_check_mark: year-over-year mass-movement ranking |
intel_migration_history |
none (beyond parity) | :white_check_mark: per-country series since 1951 |
intel_net_migration |
none (beyond parity) | :white_check_mark: World Bank net migration + migrant stock |
intel_us_border_encounters |
none (beyond parity) | :white_check_mark: CBP monthly encounters (a flow) |
intel_uk_small_boat_arrivals |
none (beyond parity) | :white_check_mark: Home Office small-boat arrivals (a flow) |
intel_social_signals |
Reddit public intelligence | :white_check_mark: |
intel_disease_outbreaks |
WHO/ProMED/CIDRAP RSS | :white_check_mark: |
News & Information (4 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_news_feed |
100 RSS feeds, 4-tier sources | :white_check_mark: |
intel_trending_keywords |
trending-keywords service | :white_check_mark: |
intel_gdelt_search |
search-gdelt-documents |
:white_check_mark: |
intel_ai_releases |
AI model/paper tracker | :white_check_mark: |
Transport & Traffic (6 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_prediction_markets |
list-prediction-markets |
:white_check_mark: |
intel_airport_delays |
list-airport-delays |
:white_check_mark: |
intel_shipping_index |
Yahoo Finance shipping ETFs | :white_check_mark: |
intel_traffic_flow |
Real-time congestion in 20 cities (TomTom) | :white_check_mark: |
intel_traffic_incidents |
Major incidents in 5 strategic regions (TomTom) | :white_check_mark: |
intel_aviation_domestic |
Global air traffic snapshot (OpenSky) | :white_check_mark: |
Webcams (1 tool)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_webcams |
Public webcam locations worldwide (Windy) | :white_check_mark: |
Analysis & Intelligence (9 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_risk_scores |
get-risk-scores |
:white_check_mark: |
intel_instability_index |
CII v2 (multi-signal blend) | :white_check_mark: |
intel_signal_convergence |
geo-convergence.ts |
:white_check_mark: |
intel_focal_points |
focal-point-detector.ts |
:white_check_mark: |
intel_signal_summary |
signal-aggregator.ts |
:white_check_mark: |
intel_temporal_anomalies |
temporal-baseline.ts |
:white_check_mark: |
intel_hotspot_escalation |
hotspot-escalation.ts |
:white_check_mark: |
intel_alert_digest |
Cross-domain alert synthesis | :white_check_mark: |
intel_weekly_trends |
Temporal trend analysis | :white_check_mark: |
Country & Geopolitical (4 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_country_brief |
get-country-intel-brief |
:white_check_mark: |
intel_country_dossier |
Comprehensive 6-source country analysis | :white_check_mark: |
intel_election_calendar |
Election proximity risk | :white_check_mark: |
intel_sanctions_search |
OFAC SDN search | :white_check_mark: |
Strategic Synthesis (4 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_strategic_posture |
Composite 9-domain risk assessment | :white_check_mark: |
intel_world_brief |
Structured daily intelligence summary | :white_check_mark: |
intel_fleet_report |
Naval fleet activity report | :white_check_mark: |
intel_population_exposure |
Population near active events | :white_check_mark: |
Tech & Science (3 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_hacker_news |
Top HN stories (Firebase API) | :white_check_mark: |
intel_trending_repos |
GitHub trending repos | :white_check_mark: |
intel_arxiv_papers |
Recent AI/ML papers | :white_check_mark: |
Government (1 tool)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_usa_spending |
USAspending.gov federal data | :white_check_mark: |
Specialist (3 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_space_weather |
NOAA/SWPC feeds | :white_check_mark: |
intel_nuclear_monitor |
USGS seismic near test sites | :white_check_mark: |
intel_service_status |
Cloudflare/AWS/Azure/GCP | :white_check_mark: |
NLP Intelligence (4 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_extract_entities |
Regex NER (28 leaders, 41 orgs, 36 APTs) | :white_check_mark: |
intel_classify_event |
Keyword threat classification (14 categories) | :white_check_mark: |
intel_news_clusters |
Jaccard similarity clustering | :white_check_mark: |
intel_keyword_spikes |
Welford's algorithm spike detection | :white_check_mark: |
Extended Geospatial (9 tools)
| Tool | WM Equivalent | Status |
|---|---|---|
intel_undersea_cables |
34 cables with landing points | :white_check_mark: |
intel_ai_datacenters |
48 global AI/cloud clusters | :white_check_mark: |
intel_spaceports |
27 launch facilities | :white_check_mark: |
intel_critical_minerals |
27 deposit types | :white_check_mark: |
intel_stock_exchanges |
82 global exchanges | :white_check_mark: |
intel_trade_routes |
19 maritime chokepoints/routes | :white_check_mark: |
intel_cloud_regions |
28 AWS/Azure/GCP regions | :white_check_mark: |
intel_financial_centers |
20 GFCI-ranked cities | :white_check_mark: |
intel_nuclear_facilities |
24 power/enrichment/research | :white_check_mark: |
Financial Intelligence (12 tools)
| Tool | Purpose | Status |
|---|---|---|
intel_forex_rates |
Latest FX rates by base + symbol filters | :white_check_mark: |
intel_forex_timeseries |
Historical FX series with configurable lookback | :white_check_mark: |
intel_major_crosses |
Major crosses + DXY proxy snapshot | :white_check_mark: |
intel_yield_curve |
Treasury curve + inversion analysis | :white_check_mark: |
intel_bond_indices |
Bond ETF summary (AGG, TLT, HYG, LQD, TIP) | :white_check_mark: |
intel_earnings_calendar |
Upcoming earnings calendar | :white_check_mark: |
intel_earnings_surprise |
Historical earnings surprise analysis | :white_check_mark: |
intel_sec_filings |
Full-text SEC EDGAR filing search | :white_check_mark: |
intel_company_filings |
Company-specific 10-K / 10-Q / 8-K retrieval | :white_check_mark: |
intel_recent_8k |
Recent material-event 8-K stream | :white_check_mark: |
intel_company_profile |
Composite company enrichment profile | :white_check_mark: |
intel_macro_composite |
Weighted market regime / macro composite score | :white_check_mark: |
Vector Intelligence (5 tools)
| Tool | Purpose | Status |
|---|---|---|
intel_semantic_search |
Natural-language search across accumulated intelligence | :white_check_mark: |
intel_similar_events |
Similarity search against historical events | :white_check_mark: |
intel_timeline |
Chronological timeline from vector store history | :white_check_mark: |
intel_vector_stats |
Qdrant collection statistics | :white_check_mark: |
intel_collect |
On-demand collection cycle for vector population | :white_check_mark: |
Cross-Domain Analytics (3 tools)
| Tool | Purpose | Status |
|---|---|---|
intel_cross_correlate |
Cross-category correlation for a topic/query | :white_check_mark: |
intel_domain_summary |
Per-category summary of stored intelligence | :white_check_mark: |
intel_trend_detection |
Recent-vs-baseline activity surge/drop detection | :white_check_mark: |
Reports (1 tool)
| Tool | Purpose | Status |
|---|---|---|
intel_generate_report |
PDF/HTML multi-domain intelligence reports | :white_check_mark: |
System (1 tool)
| Tool | Purpose | Status |
|---|---|---|
intel_status |
Source health, cache stats, tool count | :white_check_mark: |
2. Static Geospatial Datasets
| Dataset | Records | Query Tool | Status |
|---|---|---|---|
| Military bases | 70 bases, 9 operators | intel_military_bases |
:white_check_mark: |
| Strategic ports | 40 ports, 6 types | intel_strategic_ports |
:white_check_mark: |
| Pipelines | 24 oil/gas/hydrogen | intel_pipelines |
:white_check_mark: |
| Nuclear facilities | 24 power/enrichment/research | intel_nuclear_facilities |
:white_check_mark: |
| Intel hotspots | 22 geopolitical hotspots | config/countries.py | :white_check_mark: |
| Conflict zones | Active conflict centers | config/countries.py | :white_check_mark: |
| Strategic waterways | 8 chokepoints | config/countries.py | :white_check_mark: |
| Nuclear test sites | 5 sites with monitoring | config/countries.py | :white_check_mark: |
| Countries config | 22 nations with risk baselines | config/countries.py | :white_check_mark: |
| Major cities | 105 cities (pop > 2M, 1B coverage) | config/population.py | :white_check_mark: |
| Undersea cables | 34 cables with landing points | intel_undersea_cables |
:white_check_mark: |
| AI datacenters | 48 global clusters | intel_ai_datacenters |
:white_check_mark: |
| Spaceports | 27 launch facilities | intel_spaceports |
:white_check_mark: |
| Critical minerals | 27 deposit types | intel_critical_minerals |
:white_check_mark: |
| Stock exchanges | 82 global exchanges | intel_stock_exchanges |
:white_check_mark: |
| Trade routes | 19 maritime chokepoints/routes | intel_trade_routes |
:white_check_mark: |
| Cloud regions | 28 AWS/Azure/GCP regions | intel_cloud_regions |
:white_check_mark: |
| Financial centers | 20 GFCI-ranked cities | intel_financial_centers |
:white_check_mark: |
3. RSS Feed Coverage
Expanded to 119 feeds across 24 categories with 4-tier source ranking (wire/major/specialty/aggregator) and propaganda risk labels.
| Category | Count | Status |
|---|---|---|
| Geopolitics | 9 | :white_check_mark: |
| Security/Cyber | 8 | :white_check_mark: |
| Technology | 6 | :white_check_mark: |
| Finance | 6 | :white_check_mark: |
| Defense/Military | 6 | :white_check_mark: |
| Science | 6 | :white_check_mark: |
| Think Tanks | 7 | :white_check_mark: |
| Middle East | 4 | :white_check_mark: |
| Asia-Pacific | 5 | :white_check_mark: |
| Africa | 5 | :white_check_mark: |
| Latin America | 8 | :white_check_mark: |
| Multilingual (ES/FR/DE) | 7 | :white_check_mark: |
| Energy | 5 | :white_check_mark: |
| Government | 4 | :white_check_mark: |
| Crisis/Intl Orgs | 4 | :white_check_mark: |
| Europe | 4 | :white_check_mark: |
| South Asia | 3 | :white_check_mark: |
| Health | 4 | :white_check_mark: |
| Central Asia | 3 | :white_check_mark: |
| Arctic | 3 | :white_check_mark: |
| Maritime | 3 | :white_check_mark: |
| Space | 3 | :white_check_mark: |
| Nuclear | 3 | :white_check_mark: |
| Climate | 3 | :white_check_mark: |
4. Dashboard
Live Starlette app with SSE streaming at intel-dashboard --port 8501.
| Feature | Status |
|---|---|
| SSE streaming (30s refresh) | :white_check_mark: 39 data streams |
| Leaflet map (14 layers + 6 static + trade routes) | :white_check_mark: |
| 14 expandable drawer sections | :white_check_mark: |
| USNI Fleet Tracker drawer | :white_check_mark: |
| BTC Technicals drawer (SMA, Mayer, cross signals) | :white_check_mark: |
| Central Bank Rates drawer (15 banks) | :white_check_mark: |
| Data freshness monitoring drawer | :white_check_mark: |
| Per-source circuit breaker health | :white_check_mark: |
| AI situation brief (Ollama-powered) | :white_check_mark: |
| Static HTML reports | Removed (dashboard replaces) |
5. System Architecture
| Feature | Status | Notes |
|---|---|---|
| SQLite WAL-mode cache | :white_check_mark: | Persistent TTL, stale fallback |
| Per-source circuit breaker | :white_check_mark: | Configurable thresholds |
| Data freshness monitoring | :white_check_mark: | Per-source staleness in dashboard |
| Per-coro timeout (45s) | :white_check_mark: | No single source blocks dashboard |
Completed Phases
Phase 1-4: Foundation (0 -> 36 tools)
Core data sources: markets, crypto, macro, earthquakes, wildfires, ACLED, UCDP, humanitarian, military flights, theater posture, aircraft, internet outages, cable health, nav warnings, climate, prediction markets, displacement, airport delays, cyber threats, news feeds, GDELT, trending keywords, country briefs, risk scores, instability index, signal convergence.
Phase 5: Core Analysis Engine (+3 = 39 tools)
intel_focal_points, intel_signal_summary, intel_temporal_anomalies
Countries config with 22 nations, intel hotspots, conflict zones, strategic waterways. CII v2 upgraded with multi-signal weighted blend. Welford's online algorithm for temporal baseline anomaly detection.
Phase 6: Military & Infrastructure Intelligence (+6 = 45 tools)
intel_vessel_snapshot, intel_military_surge, intel_cascade_analysis, intel_hotspot_escalation, intel_commodity_quotes, intel_unrest_events
AIS vessel tracking, military surge detection in 17 sensitive regions, infrastructure cascade analysis, hotspot escalation scoring for 22 locations.
Phase 7: Domain Expansion (+10 = 55 tools)
intel_space_weather, intel_ai_releases, intel_disease_outbreaks, intel_sanctions_search, intel_election_calendar, intel_shipping_index, intel_social_signals, intel_nuclear_monitor, intel_alert_digest, intel_weekly_trends
80+ RSS feeds with 4-tier source ranking. WHO/ProMED/CIDRAP health monitoring. OFAC sanctions search. Election proximity risk scoring. Reddit social signals. Nuclear test site seismic monitoring. Cross-domain alert digest. Temporal weekly trend analysis.
Phase 8: Service Status & Geospatial (+5 = 60 tools)
intel_service_status, intel_military_bases, intel_strategic_ports, intel_pipelines, intel_nuclear_facilities
Cloud service status monitoring (Cloudflare/AWS/Azure/GCP). Static geospatial datasets: 70 military bases from 9 operators, 40 strategic ports across 6 types, 24 oil/gas/hydrogen pipelines, 24 nuclear facilities. All queryable with filters. Dashboard infrastructure map layer.
Phase 9: NLP Intelligence (+4 = 64 tools)
intel_extract_entities, intel_classify_event, intel_news_clusters, intel_keyword_spikes
Regex-based NER (28 leaders, 41 orgs, 25 companies, 36 APT groups, CVE extraction). Keyword-based threat classification into 14 categories with severity scoring. Jaccard similarity news clustering with keyword extraction. Welford's algorithm keyword spike detection against rolling baselines. Entity reference database in config/entities.py. No ML dependencies.
Phase 10: Strategic Synthesis (+4 = 68 tools)
intel_strategic_posture, intel_world_brief, intel_fleet_report, intel_population_exposure
Composite strategic posture assessment from 9 weighted domains (military, political, conflict, infrastructure, economic, cyber, health, climate, space). Structured world intelligence brief aggregating posture, focal points, news clusters, temporal anomalies, and keyword spikes. Naval fleet activity report combining theater posture, vessel snapshots, and surge detections. Population exposure analysis near active events using 105-city dataset (1B pop coverage).
Phase 11: Extended Data & Geospatial (+12 = 80 tools)
intel_country_stocks, intel_aircraft_batch, intel_hacker_news, intel_trending_repos, intel_arxiv_papers, intel_usa_spending, intel_environmental_events, intel_disaster_alerts, intel_undersea_cables, intel_ai_datacenters, intel_spaceports, intel_critical_minerals, intel_stock_exchanges, intel_usni_fleet
Static datasets completed: 34 undersea cables, 48 AI datacenters, 27 spaceports, 27 critical mineral deposits, 82 stock exchanges. USNI Fleet Tracker for Navy disposition. Data freshness monitoring added to dashboard. Static HTML report generation removed (live dashboard replaces).
Phase 12: Financial Intelligence & Geospatial Expansion (+5 = 84 tools)
intel_btc_technicals, intel_central_bank_rates, intel_trade_routes, intel_cloud_regions, intel_financial_centers
BTC technical analysis with SMA-50/200, Mayer Multiple, golden/death cross signals, and ATH distance via CoinGecko historical data. Central bank policy rates for 15 major banks (Fed, ECB, BoE, BoJ, PBoC, RBI, RBA, BoC, SNB, BCB, BoK, CBRT, SARB, Banxico, BI) — live FRED data when API key set, curated fallback otherwise. Static geospatial datasets: 19 maritime trade routes/chokepoints with oil flow and vessel transit data, 28 cloud provider regions (AWS/Azure/GCP), 20 GFCI-ranked financial centers. Trade route markers added to dashboard infrastructure map layer. BTC technicals and central bank rates added to dashboard drawer.
Phase 13: Country Dossier, Full Tool Exposure & Feed Expansion (+5 = 89 tools)
intel_country_dossier, intel_traffic_flow, intel_traffic_incidents, intel_aviation_domestic, intel_webcams
Comprehensive country intelligence dossier aggregating 6 sources in parallel (economy, markets, elections, sanctions, news, security). Exposed 4 previously hidden source functions: TomTom traffic flow (20 cities) and incidents (5 regions), OpenSky global air traffic snapshot, Windy public webcams. RSS feeds expanded from 100 to 119 across 24 categories (+6 new categories: central_asia, arctic, maritime, space, nuclear, climate). 49 CLI commands; the repo test suite has since grown well beyond this phase snapshot.
Phase 14: Financial Intelligence Extensions (+12 = 101 tools)
intel_forex_rates, intel_forex_timeseries, intel_major_crosses, intel_yield_curve, intel_bond_indices, intel_earnings_calendar, intel_earnings_surprise, intel_sec_filings, intel_company_filings, intel_recent_8k, intel_company_profile, intel_macro_composite
Added business / market-intelligence depth: FX rates and timeseries, bond curves and ETF indices, earnings calendar and surprise analysis, SEC EDGAR search and company filings, composite company enrichment, and a weighted macro-composite market regime layer.
Phase 15: Vector Intelligence (+5 = 106 tools)
intel_semantic_search, intel_similar_events, intel_timeline, intel_vector_stats, intel_collect
Qdrant-backed semantic retrieval added across all fetched intelligence, plus timeline reconstruction, store statistics, and on-demand collection. Optional dependencies remain behind .[vector] and now degrade cleanly when unavailable.
Phase 16: Cross-Domain Analytics (+3 = 109 tools)
intel_cross_correlate, intel_domain_summary, intel_trend_detection
Added historical cross-category correlation, stored-data summarization, and recent-vs-baseline trend detection on top of the vector archive for early-warning and activity-shift analysis.
Phase 17: Intelligence Reports (+1 = 110 tools)
intel_generate_report
Added PDF/HTML intelligence report generation over the existing multi-domain data collection stack. PDF output remains optional behind .[pdf], with HTML fallback available when WeasyPrint is not installed.
Phase 18: Consumer Energy Signals (+3 = 113 tools)
intel_gas_prices, intel_residential_natgas, intel_electricity_rates
Added retail fuel, residential natural gas, and electricity-rate tools to round out consumer energy monitoring alongside existing EIA crude, gas, FRED, and World Bank economic signals.
Phase 19: Cited Situation Briefs + Daily Digest (+1 = 114 tools)
intel_daily_digest
The situation brief (used by the dashboard, not itself a separate tool) now
builds a numbered sources list while walking the overview data, and
carries an honest cited flag: true only when the returned text actually
references a real source number, so a brief where the model ignored the
citation instruction is distinguishable from one that didn't. The new
intel_daily_digest tool composes a cited markdown morning brief from
current events across several domains plus, when the optional vector
store is installed, recent trend detection and a timeline, degrading via
data_gaps rather than an empty section when the vector store isn't
available.
Phase 20: AOI Geofences (+5 = 119 tools)
intel_aoi_define, intel_aoi_list, intel_aoi_delete, intel_aoi_brief, intel_aoi_escalation
User-defined areas of interest (AOIs), the intel-community term for what
consumer software calls a geofence. Before this phase, only
intel_signal_convergence accepted a real point-plus-radius, only
intel_military_flights took a bbox, and hotspot escalation scoring was
restricted to the 22 hardcoded INTEL_HOTSPOTS. intel_aoi_define
persists a named point-radius area in a dedicated table inside the
existing SQLite cache database; intel_aoi_brief composes a cited view
of earthquakes, military flights (bbox-derived), wildfires
(region-mapped), ACLED conflict events, sampled aviation, nearby static
infrastructure with distances in km, and news headline mentions, all
filtered to the AOI's radius, with data_gaps for domains that can't be
geographically scoped. intel_aoi_escalation reuses the existing
score_hotspot scoring engine unmodified for a user AOI (#16).
Phase 21: Situation Brief over MCP (+1 = 120 tools)
intel_situation_brief
The cited situation brief (#15) was previously reachable only through the
dashboard's SSE overview; MCP clients, the primary consumer this server
exists for, couldn't call it. intel_situation_brief gathers a bounded
server-side overview (earthquakes, military flights, ACLED conflict
events, wildfires, cyber threats, disease outbreaks, news, space weather,
strategic posture, alert digest, not the dashboard's full 47-source
fan-out) and delegates to the existing, unmodified fetch_situation_brief
for the AI-generated brief or its mechanically-cited fallback when Ollama
is unreachable (#18).
Phase 22: Geofence Hardening + Change Detection (+2 = 122 tools)
intel_aoi_update, intel_aoi_changes
Geofence correctness and the alerting primitive:
- Antimeridian AOIs.
bboxes_from_radius_kmreplaces the old single-box clamp: an AOI whose circle crosses the dateline (Bering Strait, Fiji, Chukotka) now produces two bounding boxes instead of silently losing everything on the far side of lon ±180. Military flight fetches run per box and merge with icao24 dedup; a one-box failure surfaces aspartial coverageindata_gapsinstead of passing half-coverage off as full. Wildfire region mapping uses the same wrap-aware boxes, so an AOI just east of the dateline maps into the oceania FIRMS box instead of reporting a false coverage gap. - Lines are lines now.
segment_distance_km(great-circle cross-track distance with endpoint clamping) replaces endpoint-only proximity for pipelines and landing-point-only proximity for undersea cables: a pipeline or cable whose midspan passes through the AOI is detected even when its endpoints are far away. Still an approximation of surveyed routes; the docstrings say exactly how. intel_aoi_update. Rename and/or re-center/resize an AOI in place with define-grade validation and collision checks. A rename keeps change-detection history; a geometry change drops it (the old baseline described a different piece of the planet).intel_aoi_changes. The geofence alerting primitive: what entered or left since the last sweep, per domain (earthquakes, military flights, ACLED events, wildfire clusters, news mentions), built on the same scoped gather asintel_aoi_briefso the two tools can never disagree about what is inside the fence. First sweep is an explicit baseline; a failed domain fetch goes todata_gapsand is excluded from the diff (a failed fetch never reads as "everything left the area"), keeping its last real observation for the next successful sweep. Sampled aviation is excluded by design (1-in-10 sample churn is not signal).
Also in this phase: SECURITY.md with an explicit threat model
(prompted by issue #21), cache database created 0600, and the brief's
gather refactored into a single shared scoping path
(_gather_scoped_domains).
Phase 27: Aircraft-Cohort Early Warning (+3 = 135 tools)
intel_jet_cohort_snapshot, intel_jet_activity_anomaly, intel_jet_excess_origins
Approach inspired by kylemcdonald/ews (independent implementation, no
ews code). Business-jet (69 ICAO types) and military cohorts from
adsb.lol, learned 24/7 by the collector (bizjet_cohort,
military_cohort, collector sources 51 -> 53). The anomaly tool compares
a UTC half-hour's peak airborne cohort count against the same slot on
prior same-weekday dates (US holidays swapped out) from globe_history
heatmaps; the excess-origins tool infers takeoffs/landings and clusters
them. Only aircraft first seen 24 h before the window count, so a fresh
install answers warming_up for a day.
Phase 28: Cross-Border Migration (+4 = 139 tools in 0.11.0; +2 border-crossing tools Unreleased)
intel_migration_flows, intel_migration_surges, intel_migration_history, intel_net_migration
UNHCR origin -> destination corridors (about 6,000 for 2025) with
prior-year change and country/region filters; a surge ranking by
origin, corridor and internal displacement sized against population
(World Bank SP.POP.TOTL); per-country series since 1951 as origin or
host; World Bank net migration and migrant stock for migration of
every kind. Fixed in the same change: intel_displacement_summary had
never sent coo_all=true and returned one global row. All figures are
year-end stocks (stated in each response), not crossings.
Shipped in 0.11.0 with the dashboard corridor map, CLI commands and collector sources. Open: near-real-time displacement (IOM DTM, HDX HAPI: both need keys).
Border crossings (Unreleased, sources/border_crossings.py; intel_us_border_encounters, intel_uk_small_boat_arrivals): FLOWS, events
per day, week or month, kept apart from the stocks above and never added to
or compared with them. Publisher status as measured on 2026-10-17 with an
identifying User-Agent:
| Publisher | Measured | Status |
|---|---|---|
| U.S. CBP | Dataset page cbp.gov/document/stats/nationwide-encounters 200; newest file a 6.7 MB CSV under /sites/default/files/2026-09/ (200); robots.txt 200, disallows /sites/default/files/assets/documents/ (the FY20-FY23 files and data dictionary, never requested); public domain, cite CBP |
:white_check_mark: intel_us_border_encounters, monthly from October 2022 |
| UK Home Office | GOV.UK Content API 200 (keyless, 10 requests/s); time series published only as ODS on assets.publishing.service.gov.uk (200, robots.txt 404), read with zipfile and the expat streaming parser (DTDs refused; only the three sheets used are read, under one row, cell and text budget), no new dependency; Open Government Licence v3.0 |
:white_check_mark: intel_uk_small_boat_arrivals, daily from 2018, weekly preventions from May 2024 |
| Frontex | Migratory map page 200; its only data file is an XLSX under /assets/, which robots.txt (200) disallows for every user agent; not requested, no workaround |
:red_circle: blocked by robots.txt (and XLSX only) |
Not exposed yet: CBP sector and field-office (area of responsibility) and
by-state detail, and CBP years before FY2023 (robots-disallowed path).
Both tools request with Fetcher.get_bytes(..., follow_redirects=False)
and a per-request cap checked as the body streams in; a redirect or a body
past its cap reads as a failed refresh that names the Location or the cap.
Planned Phases
The roadmap above is history; this section is the actual road ahead. Ordered by value; numbers are proposals, not commitments.
Phase 29 (merged to main 2026-10-18, unreleased): Research-Grade History and Evidence
Clean-room design: ideas only, no third-party code or text is used. The pattern this phase adopts is that every answer is built from saved records with three separate dates: the period a value describes, when its publisher released it, and when this server collected it. The server could not do that: the cache keeps one latest value per key, so there was no history to correlate, revise or replay.
Wave 1 runs in parallel; wave 2 builds on the history store; wave 3 is the dashboard.
| Wave | Feature | What it means here | Status |
|---|---|---|---|
| 1 | Observation history store | Every collector capture saved with observed_at (period the data describe), published_at (publisher release, when known) and collected_at (when we saved it); unchanged payloads stored once; bounded retention; backfill for sources that serve history (FRED CSV) |
:white_check_mark: Unreleased: store, three dates, dedupe, retention, body and replay size caps, and register_extractor() done (history.py; extractors for Kp, USGS magnitudes, FRED central bank rates, Yahoo quotes, and in history_extractors.py the liquidity board, net liquidity and World Bank net migration); points from stale-cache replays are not stored. Backfill built 2026-10-17 (history_backfill.py, intel_history_backfill, `intel history backfill fred |
| 1 | Capture history and revisions | Earlier saved copies of a source, and values a later copy revised | :white_check_mark: Unreleased (intel_series_history for one series' observations with all three dates or the list of available series, intel_capture_history with payload replay, intel_revisions; CLI `intel history series |
| 1 | Collection health | Per-source expected cadence; overdue / failing / never-run per source, measured from the run log, not inferred from cache freshness. Says what it cannot prove | :white_check_mark: Unreleased (intel_collection_health, intel history health): a success counts only from a live capture, so a source the fetcher answers from expired cache reads stale and goes overdue on schedule; lost history writes are recorded and reported. Stale data that bypass the fetcher's expired-cache fallback are not seen, and health says so |
| 1 | Research guide over MCP | MCP resources (guide generated from the live registry and source health, measures glossary, "what this server will not do") and MCP prompts (worked research questions, each naming the tools it needs and the limit to respect). Server exposed no prompts or resources before | :white_check_mark: Built 2026-10-17, unreleased: 3 resources (intel://guide, intel://guide/source-health, intel://guide/measures), 29 prompts, intel guide; verified over a live stdio session |
| 1 | Hostility timeline | Per country pair or country, daily counts of threat / sanction / force / criticism coverage plus tone (intel_hostility_timeline); situations between and inside countries, sortable by most covered, most hostile, biggest change (intel_situations) |
:white_check_mark: Built 2026-10-17, unreleased: intel_hostility_timeline (GDELT DOC 2.0 timelinevolraw + timelinetone, one call per category; failed calls, expired cached copies and missing days null; column_status per call; partial days judged per column) and intel_situations (RSS headlines, ranked by covered / hostile / change, change defined over feeds whose history spans both windows, null with change_measured: false when no feed does). Review fixes 2026-10-17: expired GDELT copies were shown as current counts; a day with no buckets in the judging series read as complete; "no change" was reported where nothing was compared; a GDELT 429 was retried after 1 s and then 2 s; unknown codes such as "GBR" were searched as text; a partial RSS outage was unmarked; limits below 1 misbehaved. All fixed with failing-first tests. Measured 2026-10-17: timeline modes answer (HTTP 200, date_resolution hour for timespan=7d and 15m for 3d, explicit zero buckets, norm = all monitored articles), but a timespan=7d request returned 54 hours (2026-10-15T10Z onward), and about half of calls drew 429 even 30 s apart (later probes, 16:51 to 16:53Z, 4 of 4 drew 429, so the resolution for 14 days and coverage before 2026-10-15T10Z are unmeasured; 30 days was not probed; a live MCP run at 17:00Z got 4 of 6 columns after the retry pass, one at 17:20Z got 1 of 6, consistent with the server's breaker opening after 3 failed calls, which also skips the retry pass). The fetcher now spaces every live GDELT call at least 5 s apart across the processes that share one cache DB (shared_floor.py), and retries a 429 no sooner than that; the timeline's retry pass stops at the first retry that fails again. Second review 2026-10-17: a shared slot more than 40 s ahead (eight processes at 5 s; it was 600 s) is a clock jump, overwritten and logged, as is an unreadable row; a process keeps its own queued slot after a cancelled wait; the slot time is taken once the write lock is held; an empty or one-letter country is refused. The AOI sweep makes one GDELT call per AOI, so the floor bounds it: against a mocked GDELT, 45 AOIs took 220 s of the collector's 240 s aoi_digest budget and 50 timed out after 48 calls (every other AOI domain stubbed; analysis/aoi.py not changed here). Artlist was not used for daily counts: 250 records sorted by date covered 15 minutes. Situations see only the 22-country entity table, whose matcher still counts "South Sudan" as Sudan and "Indian Ocean" as India (analysis/entities.py, not changed here) |
| 1 | Story breadth | News clusters carry a distinct-outlet count; a top-stories view ranks by how many outlets carry a story | :white_check_mark: Built 2026-10-17, unreleased: clusters gain outlets / outlet_count (sister feeds of one publisher count once); intel_top_stories ranks by outlet count with an attention list (countries and organizations by mention growth). Found on the way: cluster sources was empty in every live cluster (key mismatch, feed_name vs source_name) and RSS times were shifted by the host UTC offset; both fixed. Concurrent RSS reads now share one request per feed (a live collector news-group cycle: 119 requests for 119 feeds). Review fixes 2026-10-17: clustering runs off the event loop; attention is empty with attention_change_measured: false when no feed spans both windows, never "nothing grew" |
| 1 | Liquidity and flows | Fed balance sheet, Treasury General Account, reverse repo and net liquidity (FRED keyless CSV), money supply (US M2, euro area M1/M2/M3, never summed), ECB, ICI fund flows and ETF net issuance, FINRA margin statistics, BLS series; amounts kept separate from flows | :yellow_circle: Built, unreleased: 5 tools (intel_liquidity_board, intel_net_liquidity, intel_money_supply, intel_fed_liquidity_series, intel_ecb_series), 5 CLI commands, 2 collector sources. FRED graph CSV and ECB Data Portal live-verified 2026-10-17. The TGA term is WDTGAL (Wednesday level, USD millions): WTREGEN is a week average; the net liquidity method block states each term's units, conversion and how RRPONTSYD (daily, USD billions) is aligned to the Wednesday. Blocked, measured 2026-10-17, and listed in the board's blocked field; no proxy is substituted, so fund flows and ETF net issuance have no source: ICI (money market fund assets at /research/stats/mmf, ETF net issuance at /research/stats/etf) answers 403 Akamai "Access Denied" on every path, robots.txt included; FINRA margin statistics XLSX answers 200, but the finra.org Terms of Use (Restrictions d and e) forbid building a database from the site and data mining, scraping or harvesting tools, robots included, and the page says data feeds are not available; BLS public API v1 answers 200 without a key (CPI-U, payrolls), but api.bls.gov robots.txt is Disallow: / and the terms page on www.bls.gov answers 403 to this server's User-Agent, so it is not used |
| 1 | Kalshi + cross-venue markets | Kalshi public market data next to Polymarket; same-event comparison with an explicit match confidence; settled markets scored against pre-settlement prices (Brier) | :white_check_mark: Unreleased: intel_kalshi_markets, intel_prediction_compare, intel_prediction_settled (sources/kalshi.py, sources/prediction_cross.py, analysis/prediction_matching.py), CLI and 2 collector sources. Kalshi endpoints measured 2026-10-17, all HTTP 200 without an account: /events (nested markets), /series (include_volume), /markets?status=settled, /markets/candlesticks (batch, capped at 10,000 candles per call), /historical/cutoff. The /events category parameter is ignored by the API, so categories resolve through /series?category=, whose filter is loose too (7 of the 869 series it returned for Economics were other categories), so series are matched exactly; an unknown category answers {"series": null} and is reported as a gap, not an outage. The exchange-wide settled listing is intraday sports (1,000 rows covered about 40 minutes; none open 24h), so settled samples default to the Economics category's top series. Polymarket Gamma caps a page at 100 rows. Matching is deliberately conservative: live 2026-10-17 it matched the Fed 25 bps hike/cut pairs for October and December and rejected 25 vs 50+ bps, 2026 vs 2027, and the Fed-hold pair ("no change" vs "Hike 0bps": a known false negative). Team and person aliases are not resolved, so most sports pairs stay unmatched. Kalshi requests are spaced 0.2 s apart in-process: the two collector sources scanning the same series at once drew HTTP 429, and at 0.1 s a 26-request category scan still drew one (none at 0.2 s in one live run). Settled samples are the most recently settled markets and cluster in time (one football match supplied 11 of 25 Polymarket markets in a live run), so a Brier score describes its sample, not a venue's skill. Every number a caller passes has a range stated in the input schema (at most 25 Kalshi series or pages per scan), so one call cannot fan out into unbounded upstream requests. Review round (2026-10-17): expired-cache serves are marked on every Kalshi read and on Polymarket's settled listing and histories, and a stale history is not scored; matching adds rate-action and direction gates, keeps signs and ordinals, reads a $ amount as an amount, and its candidate blocking is complete against a brute-force scan in tests; compare carries each side's price basis, spread and Kalshi last-trade time (public /markets/trades, HTTP 200 without an account, measured 2026-10-17) and ranks flagged pairs after comparable ones; Polymarket is scored against the earlier of endDate and closedTime (on 20 closed markets measured 2026-10-17, closedTime came 2.7 to 5.5 h after endDate for football and 164 h before it for one early resolution, and umaEndDate equalled closedTime in all 20, so no outcome-known time is published); Kalshi pacing applies only on a cache miss. The shared polymarket circuit breaker (three failed history calls pause all Polymarket reads for 300 s) is documented, not changed. Re-review round (2026-10-17): matching also rejects different days of the same month, disjoint months, different clock times, quantity unit or kind mismatches, reordered quantities, opposite wording and a name on one venue only (adversarial set: 42 of 46 cases right, the 4 misses are same-event pairs rejected); a side priced from an expired copy is flagged and not comparable; Kalshi last-trade lookups read the trade price, skip expired listings, stop after 10 lookups or 10 s and name each null time; matching runs in a worker thread (2,127 Kalshi x 2,000 Polymarket in 3.4 s on mac-studio, scratch benchmark) |
| 1 | News vs market reaction | Companies and countries named in story clusters mapped to tickers / country indices; same-day move against a benchmark | :yellow_circle: Built 2026-10-17, merged to main 2026-10-18, unreleased: intel_news_market_reaction and intel news-reaction. Company rows: stock, benchmark (SPY, or the exchange country's index) and excess return on the story's session; country rows: index return with a trailing z-score. Session rule in the exchange's timezone (after close = next session; weekends and holidays read from the listing's own bars); a last bar Yahoo has not finalized is pending. Open: ^TASI and ^TA125 (Saudi and Israel in the country-index table) return 404 on Yahoo, so those benchmarks are data gaps |
| 2 | Connections with timing offsets | Correlation between retained series over a shared sample, tested at leads and lags, with minimum sample size and false-discovery control; states that it does not prove causation | :white_check_mark: Built 2026-10-17, unreleased: intel_lagged_connections and intel history connections. Pearson r on a UTC day, week or month grid at every lag from -max_lag to +max_lag (default 5, cap 30, at most 1,000 tests a call), period-over-period changes by default, p from Student's t (regularized incomplete beta, standard library), Benjamini-Hochberg adjustment across the call (default 0.05) with significance decided on the AR(1)-adjusted q_ar1 (independent random walks in levels: 91 of 100 seeds flagged by the unadjusted q, none by q_ar1), "no evidence" below an effective sample of 3, a 30-point minimum with every untested pair and lag listed with its n, latest vintage, rows without observed_at refused with the reason, every clamp reported in the MCP response and the CLI, and a cannot_prove list. p-values match scipy to 3.3e-13 relative error. Verified 2026-10-17 over a live MCP stdio session and the CLI on four real FRED series loaded into an isolated store. Not done: rank correlation, prewhitening |
| 2 | Day timeline | What every retained source recorded for a given day, by observation date or collection date | :white_check_mark: Built 2026-10-17, unreleased: intel_day_timeline and intel history day (history_timeline.py; read-only mode=ro connection, never fetches). by=collected: per-source run-log state (changed, unchanged, answered, degraded, failed, not_attempted, collector_did_not_run, before_run_log, never_run, unknown_past_retention; collector ran, did_not_run, unknown_past_retention or not_recorded) with attempts, status counts, reasons, extractor failures, a partial_failure flag and payload changes; by=observed: date and UTC-instant values on the day, month and year values listed apart as period values, undated observations counted and excluded. Retention is stated per record kind for the day. Measured 2026-10-17: two live collector cycles of 6 sources four minutes apart (12 captures: ok 8, degraded 2, error 2; every payload in the second cycle matched the first) read back over a live MCP stdio session and the CLI; 2026-10-16 in the same store reads collector not_recorded (context before_first_capture) with sources before_run_log or never_run (re-measured 2026-10-18T02:16Z after the review fixes, MCP stdio and CLI). On a synthetic store (475,200 captures, 1.75 million observations, 10,982 series) one call took 24 to 46 ms (measured before the review fixes, not re-measured), and no statement full-scans a history table under SQLite 3.45.3, 3.53.4 and 3.54.0. Not measured: a live store with weeks of history |
| 3 | Dashboard views | Liquidity board, situations, top stories, collection health panels | :white_check_mark: Built 2026-10-17, unreleased: four tabs behind RESEARCH (key p), each loaded from its own route that calls the tool's function (/api/liquidity-board, /api/situations, /api/top-stories, and /api/collection-health on the dashboard cache's history store); a failed call renders as UNAVAILABLE or REFRESH FAILED with the function's reason, never as an empty list. Checked 2026-10-17 in headless Chrome with real upstreams (9 liquidity rows; 20 situations and 9 stories over 95 of 119 feeds; collection health after one collector cycle: 38 ok, 5 degraded, 16 failing), with FRED and ECB, every RSS feed and the history store made unreachable (each tab showed the outage and its reason), and with markup injected into a headline (shown as text). Situations and top stories take 5 to 15 s per call |
Considered and not planned:
| Feature | Reason |
|---|---|
| Sportsbook odds | Odds APIs are paid or key-gated; Kalshi and Polymarket sports markets are covered by the cross-venue work |
| NFL analytics (rosters, depth charts, game model) | Outside a world-intelligence server's scope |
| Bulk paper corpus (300,000+ papers) | intel_arxiv_papers queries live; a local corpus is a separate service |
| Game-theory actor modelling | No measurable spec yet |
| "Intelligent table" encoders and 88% "kernel" compression | Claimed without a published method, code or evaluation; nothing to reproduce |
Phase 23 (planned): Geofence Depth
| Feature | Why | Status |
|---|---|---|
| Polygon AOIs | 3-64 vertex shapes, dateline-aware, exact membership across brief/changes/escalation; DB migrates in place | :white_check_mark: Shipped in 0.5.0 (intel_aoi_define_polygon) |
| Corridor AOIs | Waypoint route + width; distances measured to the route | :white_check_mark: Shipped in 0.5.0 (intel_aoi_define_corridor) |
| AOI groups / watchlists | intel_aoi_digest sweeps all (or named) AOIs in one call, advancing snapshots |
:white_check_mark: Shipped in 0.8.0 |
| Escalation news + convergence components | intel_aoi_escalation reported null for news/convergence. News is now the change in the geocoded place's share of all GDELT coverage, the 7 days ending with GDELT's current (still filling) daily step against the 7 before (timelinevolraw for the brief's settlement + county query), not name-mention counts: GDELT's own intake has multi-day outages (measured 2026-10-17 for Pittsburgh: no data for 16 of 85 days), which raw counts would read as a surge on recovery, so windows with uneven intake are refused as a data gap. Convergence runs analysis/convergence.py over the earthquakes, military flights, ACLED events and wildfires inside the AOI's exact shape, on four half-degree-offset grids. Outages and expired cached copies count as missing and capped global samples as partial; a failed news or convergence input makes that component null plus a data_gaps entry, never a 0; computed_from names the components behind the score |
:white_check_mark: Shipped (Unreleased) |
| Geo-scoped news | AOI centre reverse-geocoded via OSM Nominatim (sources/geocode.py); GDELT searched for the settlement + county, e.g. ("Pittsburgh" OR "Allegheny County"). Brief reports news_scoping; name-mention fallback is disclosed as a data gap. GDELT GEO 2.0 API was tried first and is 404 (both path variants, measured), so per-article coordinates are not available; the curated 105-city set could not do this either (nearest to Pittsburgh: Toronto, ~360 km) |
:white_check_mark: Shipped in 0.10.0 |
| CLI parity | intel aoi group with all 9 subcommands, shared store semantics, live-smoked |
:white_check_mark: Shipped in 0.8.0 |
| Dashboard AOI layer | /api/aois + a toggleable "AOI Geofences" Leaflet layer: circles, polygons, and corridors drawn from the store, tooltip/detail with what the last collector sweep counted inside each. Reads the stored snapshot, never gathers live (a never-swept AOI says so instead of showing zeros; a broken store is a 503, not an empty map). Verified in headless Chromium: 2 shapes rendered, 50 km circle at 50,000 m, no console errors |
:white_check_mark: Shipped in 0.10.0 |
| Scheduled AOI sweeps | aoi_digest is a collector source (fetch_aoi_sweep, 240 s budget): every daemon cycle advances all AOI snapshots, so --daemon + the launchd wrapper IS the schedule. Change digests land in the collector log and vector store |
:white_check_mark: Shipped in 0.9.0 |
| AOI change notifications | WORLD_INTEL_AOI_WEBHOOK (+ _FORMAT=json|text): the sweep POSTs non-quiet digests; quiet sweeps and dead sinks are honest notification records, never silent or fatal. Live-verified against a local sink. Email is out of scope (bring a webhook bridge) |
:white_check_mark: Shipped in 0.9.0 |
Phase 24 (planned): Test Coverage Gate
Measured 2026-09-01 (before the current test push): 59% statement
coverage overall; server.py, cli.py, collector.py at 0%;
analysis/ NLP modules 0-14%; sources/intelligence.py 22%.
| Item | Why | Status |
|---|---|---|
| Analysis-layer tests (classifier, entities, convergence, spikes, clustering, signals, focal points, surge, cascade, exposure, posture, alerts, instability, world_brief) | The NLP/analysis layer had no executable verification at all | :white_check_mark: Shipped 2026-09-01: 131 tests, all 14 modules at 96-100% |
| Source-layer tests (intelligence, cyber, climate, displacement, fleet, prediction, service_status, maritime, military helpers, news) | Same class of gap on the fetch/parse layer | :white_check_mark: Shipped 2026-09-01: 89 tests, modules at 79-100% (intelligence.py 93%) |
Import-based server.py registry tests |
The TOOLS/_dispatch parity invariant was checked by reading server.py as text; now imported under a temp cache path and verified structurally, including a real dispatch round-trip |
:white_check_mark: Shipped 2026-09-01 (server.py 0% -> 47%) |
collector.py source-map test |
Map verified earlier; the collect/daemon run loop, filters, and cycle accounting now tested too (97% module coverage) | :white_check_mark: Shipped 2026-09-01 |
cli.py smoke tests (CliRunner) |
1,076 statements, zero executed by tests; now 76 tests driving all 52 reachable commands, 92% coverage | :white_check_mark: Shipped 2026-09-01 |
| CI coverage gate with ratchet | --cov-fail-under=89 live in ci.yml (90% measured 2026-09-01 evening; floor one point under for platform variance). Raise as waves land; never lower |
:white_check_mark: Shipped 2026-09-01, ratcheted same day |
Phase 24.5: Data-Honesty Backlog (top three fixed 2026-09-01)
Verified bugs found by the 2026-09-01 test waves. The two cross-module
silent-zero key mismatches found at the same time (world_brief
article_count vs size, fleet warning_count vs
naval_warnings) were fixed in 0.4.0 with regression tests; the three
fail-quietly bugs below were fixed in the same-day follow-up
(issues #22-#24, each with a flipped regression test).
| Bug | Where | Class | Status |
|---|---|---|---|
| UNHCR outage returned all-zero global totals with no error/degraded key ("zero refugees worldwide") | sources/displacement.py |
fail-reads-as-success | :white_check_mark: Fixed (#22) |
A climate zone whose fetch failed was silently omitted; full outage yielded {"zones": {}} with no marker |
sources/climate.py |
silent degradation | :white_check_mark: Fixed (#23) |
| "Resolved: Major outage" classified as an active critical incident; a dead provider feed still listed as checked | sources/service_status.py |
misclassification | :white_check_mark: Fixed (#24) |
Naval warnings apply to all 9 waterways identically (no proximity/NAVAREA filter); total_nearby computed but never emitted |
sources/intelligence.py vessel snapshot |
precision | :white_check_mark: Fixed (Unreleased): a warning counts only at waterways within 300 km of a position its full text states; one that states no readable position counts nowhere (crediting positionless NAVAREA IV and XII warnings to Panama was measured wrong 5 of 5 times on the live feed and rejected) and, if naval, is listed in unplaced_naval_warnings with a reason; the rule is in matching_rule; total_nearby emitted; an NGA outage reads unknown and a frozen feed stale, not clear or a current threat level; a feed whose currency cannot be checked (no parseable issue date, or no active warnings at all) reads unknown and the fleet leaves it out of its score; a position match cannot start right after a letter ('20-O3-44N' used to read 3.73N), which changed no position in 3,643 real warnings. Open: an unplaced naval warning leaves every waterway's status unchanged; it is named only in unplaced_naval_warnings and data_gaps |
| Country/company/leader entity matching was substring-based ("usa" inside "thousand", "hamas" inside "Bahamas") | analysis/entities.py |
false positives | :white_check_mark: Fixed in 0.5.0 (word-boundary alternations, measured faster) |
| Category keywords matched substrings ("airstrike" bumped severity via "strike"; "denied" fired "ied") | analysis/classifier.py |
false positives | :white_check_mark: Fixed in 0.5.0 (boundary-anchored stems, per-keyword overrides) |
| Silent-empty degradation without a marker on RSS/API failure | sources/prediction.py (documented as intended), sources/maritime.py, sources/news.py RSS path, sources/fleet.py _safe wrapper, sources/military.py military flights and theater posture |
silent degradation | :yellow_circle: sources/news.py RSS path fixed 2026-10-17 (every call reports feeds_attempted / feeds_ok / feeds_failed / feeds_stale; no feed answering live is error + degraded + reason: all_feeds_failed, including the warm-cache case where failed feeds came back as expired copies and counted as healthy; failing-first regression tests). Since the 2026-10-17 review: fewer than half of the feeds answering live is degraded + reason: partial_feeds_failed with no error (measured that day: 95 of 119 answered, the rest permanent 404 / 403 / 410 plus one timeout), passed through by trending keywords, situations, top stories and the dossier; links are kept only when http(s) and titles are capped at 500 characters. maritime.py, fleet.py, military flights and theater posture fixed (Unreleased): outages carry error/degraded/reason and data_gaps; failed or stale fleet components are named in unavailable_components, and fleet readiness reads UNKNOWN with none available and is marked a lower bound with some missing; military flights with no live source carries error/degraded/reason and data_gaps beside source: "none", including the warm-cache case, where an expired adsb.lol or OpenSky copy used to come back as current flights (it now goes under stale_copy with its age, after OpenSky is tried live); posture names the theaters no flight source answered for (stale_theaters for those only a copy answered) and does not cache them; a malformed bbox is invalid_bbox instead of the whole world; intel posture prints those theaters as unknown with the gaps; an empty NGA answer is degraded, not a quiet sea; nav warnings flag (feed_stale) a feed whose newest warning is over 14 days old (NGA's API holds nothing issued after 10 May 2024, measured 2026-10-17). sources/prediction.py fixed, unreleased: an outage or unexpected response adds error + degraded, an expired-cache serve adds stale; keys unchanged; regression tests written failing-first (3 failed on the old module). The fetcher's stale fallback marks only dict payloads, so other text and XML sources can serve expired copies unmarked Open: the dashboard and collector do not yet read the new markers; strategic posture and temporal anomalies still read a posture outage as zero activity |
Country dossier news was always empty with no marker, for every country: it asked the feed for category="all" (not a category, so fetch_news_feed answered with an error dict), read articles / source while the feed emits items / feed_name, and checked only its own _error, never the feed's error. Also, 28 of the 50 accepted codes have no headline keywords, so their search could never match |
analysis/dossier.py news section |
fail-reads-as-success | :white_check_mark: Fixed 2026-10-17, unreleased (reads every feed; a feed error or outage passes through as error / degraded / reason with mention_count: null and a data gap; failed and stale feeds named; keywordless countries report "mentions not searched"; intel dossier prints News: not measured; failing-first tests drive the real fetch_news_feed output). Since the 2026-10-17 review: headlines match on whole words (the substring test counted "Tirana" as Iran and "Indiana" as India), four compound names are removed first, and the section reports matching_items, mention_count_capped, items_searched and searched_span. Second review 2026-10-17: demonyms such as "North Korean(s)" and "Gazan(s)" match again (an -n / -ns / -s ending on each keyword); searched_span spans parsed UTC times only and items_without_parsed_time counts the rest |
| 33 CLI commands rendered upstream errors as healthy empty states | cli.py |
fail-reads-as-success | :white_check_mark: Fixed in 0.6.0 (shared bail-on-error path; 43 failing-first tests) |
| Rich markup swallowed lowercase bracketed values and let remote titles inject markup | cli.py |
output corruption | :white_check_mark: Fixed in 0.6.0 (per-site escaping; injection test) |
| Remote data inside Rich TABLE cells was not markup-escaped | cli.py |
output corruption | :white_check_mark: Fixed in 0.6.0 (_cell Text-wrapping convention, 11 injection tests; error styles unified across all 53 commands) |
Phase 25 (planned): Missing Domains
Verified absent from sources/ on 2026-09-01 (grep, not memory):
| Feature | Source candidate | Status |
|---|---|---|
| Severe weather alerts | NWS (US, 0.5.0) + Meteoalarm (39 EU countries, 0.8.0; per-country feeds only - no Europe-wide feed exists) | :white_check_mark: |
| Tropical cyclone tracking | NHC (0.5.0) + JTWC (0.8.0: NW Pacific/N Indian/S Hemisphere). Forecast tracks (Unreleased): opt-in include_forecast on both tools. NHC points come from the linked forecast/advisory text and the cone from the linked trackCone CONE.kmz (simplified); JTWC points come from each linked warning text, and JTWC links no cone. Not upstream-blocked, measured 2026-10-17: keyless HTTP 200 for NHC CurrentStorms.json, text/MIATCMEP1.shtml, EP212026_017adv_CONE.kmz and JTWC jtwc.rss, wp2926web.txt, sh0127web.txt; nhc.noaa.gov robots.txt disallows nothing and the NWS disclaimer is public domain; JTWC jtwc.html and robots.txt answered 403 (CloudFront), so JTWC's terms are unread. Wind radii not parsed |
:white_check_mark: warnings + forecast tracks |
| Dedicated volcano monitoring | GVP weekly report shipped in 0.5.0 (intel_volcano_activity) |
:white_check_mark: |
| NOTAMs | BLOCKED upstream: official FAA API is key-required (401 verified live 2026-09-01); unofficial backend POST-only/undocumented. Revisit if FAA opens access | :red_circle: blocked |
| Launch schedules | Launch Library 2 shipped in 0.5.0 (intel_launch_schedule) |
:white_check_mark: |
| BGP routing status | intel_bgp_status (0.8.0): per-resource RIS visibility + RPKI via RIPEstat. Global incident/hijack feeds remain open (no key-free target-free feed found) |
:yellow_circle: per-resource done |
Note: resolved in 0.6.0 - the four domains are in the collector's roster (50 sources), with the invariant test and docs updated together.
Phase 26: server.py Modularization (shipped 2026-09-01)
The 2,890-line monolith is a 158-line shell over 12 domain modules in
tools/ plus shared infrastructure in runtime.py. The
TOOLS/_dispatch parity invariant is now enforced at import time by
tools.aggregate() (drift or collision refuses to start), verified by
an AST byte-parity gate against the pre-split registry, the full
suite, and a live MCP stdio session.
Summary
| Category | Current | Notes |
|---|---|---|
| Total MCP tools | 160 | 159 intelligence tools + intel_status |
| Tool parity | 160 / 160 | TOOLS and handlers are aligned at import time; README's per-tool reference is pinned to the registry by test_readme_documents_every_registered_tool |
| Static datasets | 18 | Bases, ports, pipelines, nuclear, cables, datacenters, spaceports, minerals, exchanges, trade routes, cloud regions, financial centers |
| RSS feeds | 119 | 24 categories |
| Tests in repo | 2,801 | 2,781 non-smoke tests + 20 live smoke tests (measured 2026-10-18: pytest -m 'not smoke' -> 2,781 passed, 20 deselected); full suite requires .[dev] |
| Statement coverage | 96% | Measured 2026-10-18 on the merged main (a7374b7), pytest -m 'not smoke' --cov: 96.46% on fedora (Python 3.13) and ai-lab (3.12), 96.45% on fedora (3.11); 92.02% on 2026-10-12; 91.65% on 2026-10-11; 91.18% on 2026-09-01, 59% the morning before. CI ratchet at 89 |
| MCP prompts and resources | 29 / 3 | Research guide over MCP (Phase 29): worked research prompts; guide, measures glossary and source health resources |
| Collector sources | 61 | collector.SOURCES; every capture is written to the observation history |
| Primary remaining gaps | Blocked publishers (ICI fund flows and ETF issuance, FINRA margin statistics, BLS series); Saudi and Israel benchmarks for news vs market reaction (Yahoo 404); rank correlation and prewhitening for lagged connections | See Phase 29 |
Bottom line (2026-10-18): 160 tools across 30+ domains, 29 MCP prompts and 3 resources, 61 collector sources. Phase 29 (dated observation history with revisions and backfill, collection health, lagged connections, day timeline, liquidity and money, Kalshi and cross-venue prediction markets, news signals and market reaction, the research guide over MCP and the dashboard research panels), tropical cyclone forecast tracks and US and UK border-crossing flows are merged to main, unreleased. Earlier (2026-10-11): 139 tools across 30+ domains; Phases 27-28 added aircraft-cohort early warning and cross-border migration. Earlier (2026-09-02): 122 tools across 30+ domains, with the roadmap aligned to the live MCP registry and, for the first time, a forward-looking plan (Phases 23-26): geofence depth, the coverage gate, missing domains, and server.py modularization.