160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.
Python ai-toolsai-watchanthropicclaudecybersecuritydashboardgeofencinggeopoliticalgeospatialintelligencemarketsmcpmigrationmilitarymodel-context-protocolnuclearosintreal-timespace-weatherthreat-intelligence
Merge feat/history-store into feat/research-grade
Phoenix Agent (mac-studio)39cb303997bf
diff --cc README.mdindex 4670ac7,0ac37f6..74d34a2--- a/README.md+++ b/README.md@@@ -630,24 -512,15 +630,24 @@@ different, openly licensed data source|------|-------------|| `intel_situation_brief` | Cited situational awareness brief, generated on demand over MCP: a bounded server-side overview (earthquakes, military flights, ACLED conflict events, wildfires, cyber threats, disease outbreaks, news, space weather, strategic posture, alert digest), synthesized via local Ollama or a mechanically-cited fallback when Ollama is unreachable |-### Collection History (4)-History accrues only while the collector (`intel-collector`) runs; nothing before its first recorded cycle exists. Every observation keeps three dates: `observed_at` (the period the value describes, from the payload, or null when the payload has none), `published_at` (publisher release, when known) and `collected_at` (when this server saved it). A run answered from an expired cached copy is recorded as a stale-cache replay: it never counts as a success, and its observations are not stored, because a replay is not a new observation.+### Collection History (6)- History accrues only while the collector (`intel-collector`) runs; nothing before its first recorded cycle exists, except what a backfill loads (`intel_history_backfill`, `intel history backfill`): FRED graph CSV and ECB Data Portal series, stored under `backfill:fred` and `backfill:ecb` with the publisher's current vintage, so revisions made before the first backfill are invisible. Every observation keeps three dates: `observed_at` (the period the value describes, from the payload, or null when the payload has none), `published_at` (publisher release, when known) and `collected_at` (when this server saved it).++History accrues only while the collector (`intel-collector`) runs; nothing before its first recorded cycle exists, except what a backfill loads (`intel_history_backfill`, `intel history backfill`): FRED graph CSV and ECB Data Portal series, stored under `backfill:fred` and `backfill:ecb` with the publisher's current vintage, so revisions made before the first backfill are invisible. Every observation keeps three dates: `observed_at` (the period the value describes, from the payload, or null when the payload has none), `published_at` (publisher release, when known) and `collected_at` (when this server saved it). A run answered from an expired cached copy is recorded as a stale-cache replay: it never counts as a success, and its observations are not stored, because a replay is not a new observation.| Tool | Description ||------|-------------|- | `intel_collection_health` | Per-source health from the collector's run log: last attempt, last success, consecutive failures, overdue against the expected cadence, never-run sources, history storage size; states what it cannot prove (a fresh capture does not mean upstream observations are current) |- | `intel_capture_history` | A source's earlier captures (ok / degraded / error with reason, duration, record count) and whether each changed from the previous one; `capture_id` replays one stored payload |+ | `intel_collection_health` | Per-source health from the collector's run log: last attempt, last live success (a source served from stale cache reads `stale` and goes overdue), last fully ok capture, consecutive failures, overdue against the expected cadence, never-run sources, lost history writes, history storage size; states its data gaps and what it cannot prove (a fresh capture does not mean upstream observations are current) |+ | `intel_capture_history` | A source's earlier captures (ok / degraded / error with reason, such as `served stale cache`, duration, record count) and whether each changed from the previous one; `capture_id` replays one stored payload (an outline of its keys past the replay cap) || `intel_series_history` | One series' observations with all three dates, or the list of available series |- | `intel_revisions` | Values for the same `observed_at` that a later capture changed (USGS magnitude revisions, NOAA Kp updates) |+ | `intel_revisions` | Values for the same `observed_at` that a later capture changed (USGS magnitude revisions, NOAA Kp updates), with how many observations and dated series were checked |+| `intel_day_timeline` | What the history store holds for one UTC day (`YYYY-MM-DD`; a future day is an error). `by=collected` (default): per source, that day's captures (attempts, ok / degraded / error with reasons, changed vs unchanged payloads, first and last capture, observations stored), and extractor failures and partial failures, and for a source with no capture whether the collector did not run that day, ran without it, never ran it, the day is before the run log starts, or it is past run-log retention. `by=observed`: values whose `observed_at` falls on the day; month and year values containing the day are listed apart as period values, and undated observations are counted and excluded. Every item carries `observed_at`, `published_at` and `collected_at`; pruned records are named, never read as a quiet day; per-source and total caps (shared by both by=observed lists) with truncation disclosed. Read-only, one snapshot per call, never fetches |+| `intel_history_backfill` | Load the history FRED (graph CSV) or the ECB Data Portal already serve into the history store: one capture per request (a FRED id, or an ECB key that may match several series) under `backfill:fred` / `backfill:ecb`, `observed_at` from the publisher's dates, `published_at` null, `collected_at` the backfill time; re-runs store only new or changed values (a change is a revision); FRED missing values skipped and counted, ECB missing values not countable (null); writes and fetches, so capped at 5 FRED series or 2 ECB keys per call |++### Connections Between Series (1)+Reads the history store only, so the sample is what the collector saved, not each series' full history. A connection is a correlation in that sample: it does not show causation, and a lead does not show causal direction.++| Tool | Description |+|------|-------------|+| `intel_lagged_connections` | Correlation between 2 to 8 stored series (keys as `intel_series_history` lists them) on a UTC day, week or month grid, tested at every lag from `-max_lag` to `+max_lag` (default 5, at most 30, at most 1,000 tests a call). Default transform is period-over-period changes, because levels of trending series correlate spuriously; `levels` on request. Per pair: the lag with the smallest p, with Pearson r, overlap n, two-sided p from Student's t (periods treated as independent), `p_ar1` recomputed with an AR(1) effective sample size from both series' lag-1 autocorrelations, and the Benjamini-Hochberg `q` and `q_ar1` across every test in the call. `significant` means `q_ar1` <= 0.05 by default (`significant_if_independent` applies the threshold to `q`); an effective sample under 3 is reported as no evidence, and `data_gaps` names autocorrelated series and pairs where serial correlation dominates. Lag +k compares A at period t with B at t+k, so A leads B by k. Pair/lag overlaps under 30 points are listed as not tested with their n; revised points use the latest vintage; rows without `observed_at` (Yahoo quotes) are not used; every clamped input is reported, in the CLI too |---diff --cc ROADMAP.mdindex 2ec7eb9,4c79a55..0d26fa1--- a/ROADMAP.md+++ b/ROADMAP.md@@@ -590,18 -571,18 +590,18 @@@ dashboard| Wave | Feature | What it means here | Status ||------|---------|--------------------|--------|- | 1 | Observation history store | Every collector capture saved with `observed_at` (period the data describe), `published_at` (publisher release, when known) and `collected_at` (when we saved it); unchanged payloads stored once; bounded retention; backfill for sources that serve history (FRED CSV) | :white_check_mark: Unreleased: store, three dates, dedupe, retention and `register_extractor()` done (`history.py`; extractors for Kp, USGS magnitudes, FRED central bank rates, Yahoo quotes, and in `history_extractors.py` the liquidity board, net liquidity and World Bank net migration). Backfill built 2026-10-17 (`history_backfill.py`, `intel_history_backfill`, `intel history backfill fred|ecb`) for FRED graph CSV and ECB Data Portal (BSI, ILM) series, under its own sources `backfill:fred` and `backfill:ecb`; `published_at` is NULL (neither response names a release date) and values are the vintage of the backfill day. Measured live 2026-10-17: WALCL 1,244 values (2002-12-18 to 2026-10-14), RRPONTSYD 3,348 (2,833 blank cells skipped), DGS10 16,182 (721 skipped), euro area M3 560 months (1980-01 to 2026-08); a second run and a refetch stored 0 |- | 1 | Capture history and revisions | Earlier saved copies of a source, and values a later copy revised | :white_check_mark: Unreleased (`intel_capture_history` with payload replay, `intel_revisions`) |- | 1 | Collection health | Per-source expected cadence; overdue / failing / never-run per source, measured from the run log, not inferred from cache freshness. Says what it cannot prove | :white_check_mark: Unreleased (`intel_collection_health`, `intel history health`) |-| 1 | Observation history store | Every collector capture saved with `observed_at` (period the data describe), `published_at` (publisher release, when known) and `collected_at` (when we saved it); unchanged payloads stored once; bounded retention; backfill for sources that serve history (FRED CSV) | :yellow_circle: Unreleased: store, three dates, dedupe, retention, body and replay size caps, and `register_extractor()` done (`history.py`; extractors for Kp, USGS magnitudes, FRED central bank rates, Yahoo quotes); points from stale-cache replays are not stored; backfill not done |++| 1 | Observation history store | Every collector capture saved with `observed_at` (period the data describe), `published_at` (publisher release, when known) and `collected_at` (when we saved it); unchanged payloads stored once; bounded retention; backfill for sources that serve history (FRED CSV) | :white_check_mark: Unreleased: store, three dates, dedupe, retention, body and replay size caps, and `register_extractor()` done (`history.py`; extractors for Kp, USGS magnitudes, FRED central bank rates, Yahoo quotes, and in `history_extractors.py` the liquidity board, net liquidity and World Bank net migration); points from stale-cache replays are not stored. Backfill built 2026-10-17 (`history_backfill.py`, `intel_history_backfill`, `intel history backfill fred|ecb`) for FRED graph CSV and ECB Data Portal (BSI, ILM) series, under its own sources `backfill:fred` and `backfill:ecb`; `published_at` is NULL (neither response names a release date) and values are the vintage of the backfill day. Measured live 2026-10-17: WALCL 1,244 values (2002-12-18 to 2026-10-14), RRPONTSYD 3,348 (2,833 blank cells skipped), DGS10 16,182 (721 skipped), euro area M3 560 months (1980-01 to 2026-08); a second run and a refetch stored 0 |+ | 1 | Capture history and revisions | Earlier saved copies of a source, and values a later copy revised | :white_check_mark: Unreleased (`intel_capture_history` with payload replay, `intel_revisions`); a replay names the capture that stored its body, revisions say what they checked |+ | 1 | Collection health | Per-source expected cadence; overdue / failing / never-run per source, measured from the run log, not inferred from cache freshness. Says what it cannot prove | :white_check_mark: Unreleased (`intel_collection_health`, `intel history health`): a success counts only from a live capture, so a source the fetcher answers from expired cache reads `stale` and goes overdue on schedule; lost history writes are recorded and reported. Stale data that bypass the fetcher's expired-cache fallback are not seen, and health says so |-| 1 | Research guide over MCP | MCP resources (guide generated from the live registry and source health, measures glossary, "what this server will not do") and MCP prompts (worked research questions, each naming the tools it needs and the limit to respect). Server exposed no prompts or resources before | :red_circle: |-| 1 | Hostility timeline | Per country pair or country, daily counts of threat / sanction / force / criticism coverage plus tone; sortable by most covered, most hostile, biggest change | :red_circle: |-| 1 | Story breadth | News clusters carry a distinct-outlet count; a top-stories view ranks by how many outlets carry a story | :red_circle: |-| 1 | Liquidity and flows | Fed balance sheet, Treasury General Account, reverse repo and net liquidity (FRED keyless CSV), money supply (US M2, euro area M1/M2/M3, never summed), ECB, ICI fund flows and ETF net issuance, FINRA margin statistics, BLS series; amounts kept separate from flows | :red_circle: |-| 1 | Kalshi + cross-venue markets | Kalshi public market data next to Polymarket; same-event comparison with an explicit match confidence; settled markets scored against pre-settlement prices (Brier) | :red_circle: |-| 1 | News vs market reaction | Companies and countries named in story clusters mapped to tickers / country indices; same-day move against a benchmark | :red_circle: |-| 2 | Connections with timing offsets | Correlation between retained series over a shared sample, tested at leads and lags, with minimum sample size and false-discovery control; states that it does not prove causation | :red_circle: |-| 2 | Day timeline | What every retained source recorded for a given day, by observation date or collection date | :red_circle: |-| 3 | Dashboard views | Liquidity board, situations, top stories, collection health panels | :red_circle: |+| 1 | Research guide over MCP | MCP resources (guide generated from the live registry and source health, measures glossary, "what this server will not do") and MCP prompts (worked research questions, each naming the tools it needs and the limit to respect). Server exposed no prompts or resources before | :white_check_mark: Built 2026-10-17, unreleased: 3 resources (`intel://guide`, `intel://guide/source-health`, `intel://guide/measures`), 29 prompts, `intel guide`; verified over a live stdio session |+| 1 | Hostility timeline | Per country pair or country, daily counts of threat / sanction / force / criticism coverage plus tone (`intel_hostility_timeline`); situations between and inside countries, sortable by most covered, most hostile, biggest change (`intel_situations`) | :white_check_mark: Built 2026-10-17, unreleased: `intel_hostility_timeline` (GDELT DOC 2.0 `timelinevolraw` + `timelinetone`, one call per category; failed calls, expired cached copies and missing days null; `column_status` per call; partial days judged per column) and `intel_situations` (RSS headlines, ranked by covered / hostile / change, change defined over feeds whose history spans both windows, null with `change_measured: false` when no feed does). Review fixes 2026-10-17: expired GDELT copies were shown as current counts; a day with no buckets in the judging series read as complete; "no change" was reported where nothing was compared; a GDELT 429 was retried after 1 s and then 2 s; unknown codes such as "GBR" were searched as text; a partial RSS outage was unmarked; limits below 1 misbehaved. All fixed with failing-first tests. Measured 2026-10-17: timeline modes answer (HTTP 200, `date_resolution` `hour` for `timespan=7d` and `15m` for `3d`, explicit zero buckets, `norm` = all monitored articles), but a `timespan=7d` request returned 54 hours (2026-10-15T10Z onward), and about half of calls drew 429 even 30 s apart (later probes, 16:51 to 16:53Z, 4 of 4 drew 429, so the resolution for 14 days and coverage before 2026-10-15T10Z are unmeasured; 30 days was not probed; a live MCP run at 17:00Z got 4 of 6 columns after the retry pass, one at 17:20Z got 1 of 6, consistent with the server's breaker opening after 3 failed calls, which also skips the retry pass). The fetcher now spaces every live GDELT call at least 5 s apart across the processes that share one cache DB (`shared_floor.py`), and retries a 429 no sooner than that; the timeline's retry pass stops at the first retry that fails again. Second review 2026-10-17: a shared slot more than 40 s ahead (eight processes at 5 s; it was 600 s) is a clock jump, overwritten and logged, as is an unreadable row; a process keeps its own queued slot after a cancelled wait; the slot time is taken once the write lock is held; an empty or one-letter country is refused. The AOI sweep makes one GDELT call per AOI, so the floor bounds it: against a mocked GDELT, 45 AOIs took 220 s of the collector's 240 s `aoi_digest` budget and 50 timed out after 48 calls (every other AOI domain stubbed; `analysis/aoi.py` not changed here). Artlist was not used for daily counts: 250 records sorted by date covered 15 minutes. Situations see only the 22-country entity table, whose matcher still counts "South Sudan" as Sudan and "Indian Ocean" as India (`analysis/entities.py`, not changed here) |+| 1 | Story breadth | News clusters carry a distinct-outlet count; a top-stories view ranks by how many outlets carry a story | :white_check_mark: Built 2026-10-17, unreleased: clusters gain `outlets` / `outlet_count` (sister feeds of one publisher count once); `intel_top_stories` ranks by outlet count with an attention list (countries and organizations by mention growth). Found on the way: cluster `sources` was empty in every live cluster (key mismatch, `feed_name` vs `source_name`) and RSS times were shifted by the host UTC offset; both fixed. Concurrent RSS reads now share one request per feed (a live collector news-group cycle: 119 requests for 119 feeds). Review fixes 2026-10-17: clustering runs off the event loop; attention is empty with `attention_change_measured: false` when no feed spans both windows, never "nothing grew" |+| 1 | Liquidity and flows | Fed balance sheet, Treasury General Account, reverse repo and net liquidity (FRED keyless CSV), money supply (US M2, euro area M1/M2/M3, never summed), ECB, ICI fund flows and ETF net issuance, FINRA margin statistics, BLS series; amounts kept separate from flows | :yellow_circle: Built, unreleased: 5 tools (`intel_liquidity_board`, `intel_net_liquidity`, `intel_money_supply`, `intel_fed_liquidity_series`, `intel_ecb_series`), 5 CLI commands, 2 collector sources. FRED graph CSV and ECB Data Portal live-verified 2026-10-17. The TGA term is WDTGAL (Wednesday level, USD millions): WTREGEN is a week average; the net liquidity `method` block states each term's units, conversion and how RRPONTSYD (daily, USD billions) is aligned to the Wednesday. Blocked, measured 2026-10-17, and listed in the board's `blocked` field; no proxy is substituted, so fund flows and ETF net issuance have no source: ICI (money market fund assets at /research/stats/mmf, ETF net issuance at /research/stats/etf) answers 403 Akamai "Access Denied" on every path, robots.txt included; FINRA margin statistics XLSX answers 200, but the finra.org Terms of Use (Restrictions d and e) forbid building a database from the site and data mining, scraping or harvesting tools, robots included, and the page says data feeds are not available; BLS public API v1 answers 200 without a key (CPI-U, payrolls), but api.bls.gov robots.txt is `Disallow: /` and the terms page on www.bls.gov answers 403 to this server's User-Agent, so it is not used |+| 1 | Kalshi + cross-venue markets | Kalshi public market data next to Polymarket; same-event comparison with an explicit match confidence; settled markets scored against pre-settlement prices (Brier) | :white_check_mark: Unreleased: `intel_kalshi_markets`, `intel_prediction_compare`, `intel_prediction_settled` (`sources/kalshi.py`, `sources/prediction_cross.py`, `analysis/prediction_matching.py`), CLI and 2 collector sources. Kalshi endpoints measured 2026-10-17, all HTTP 200 without an account: `/events` (nested markets), `/series` (`include_volume`), `/markets?status=settled`, `/markets/candlesticks` (batch, capped at 10,000 candles per call), `/historical/cutoff`. The `/events` `category` parameter is ignored by the API, so categories resolve through `/series?category=`, whose filter is loose too (7 of the 869 series it returned for Economics were other categories), so series are matched exactly; an unknown category answers `{"series": null}` and is reported as a gap, not an outage. The exchange-wide settled listing is intraday sports (1,000 rows covered about 40 minutes; none open 24h), so settled samples default to the Economics category's top series. Polymarket Gamma caps a page at 100 rows. Matching is deliberately conservative: live 2026-10-17 it matched the Fed 25 bps hike/cut pairs for October and December and rejected 25 vs 50+ bps, 2026 vs 2027, and the Fed-hold pair ("no change" vs "Hike 0bps": a known false negative). Team and person aliases are not resolved, so most sports pairs stay unmatched. Kalshi requests are spaced 0.2 s apart in-process: the two collector sources scanning the same series at once drew HTTP 429, and at 0.1 s a 26-request category scan still drew one (none at 0.2 s in one live run). Settled samples are the most recently settled markets and cluster in time (one football match supplied 11 of 25 Polymarket markets in a live run), so a Brier score describes its sample, not a venue's skill. Every number a caller passes has a range stated in the input schema (at most 25 Kalshi series or pages per scan), so one call cannot fan out into unbounded upstream requests. Review round (2026-10-17): expired-cache serves are marked on every Kalshi read and on Polymarket's settled listing and histories, and a stale history is not scored; matching adds rate-action and direction gates, keeps signs and ordinals, reads a $ amount as an amount, and its candidate blocking is complete against a brute-force scan in tests; compare carries each side's price basis, spread and Kalshi last-trade time (public `/markets/trades`, HTTP 200 without an account, measured 2026-10-17) and ranks flagged pairs after comparable ones; Polymarket is scored against the earlier of endDate and closedTime (on 20 closed markets measured 2026-10-17, closedTime came 2.7 to 5.5 h after endDate for football and 164 h before it for one early resolution, and umaEndDate equalled closedTime in all 20, so no outcome-known time is published); Kalshi pacing applies only on a cache miss. The shared `polymarket` circuit breaker (three failed history calls pause all Polymarket reads for 300 s) is documented, not changed. Re-review round (2026-10-17): matching also rejects different days of the same month, disjoint months, different clock times, quantity unit or kind mismatches, reordered quantities, opposite wording and a name on one venue only (adversarial set: 42 of 46 cases right, the 4 misses are same-event pairs rejected); a side priced from an expired copy is flagged and not comparable; Kalshi last-trade lookups read the trade price, skip expired listings, stop after 10 lookups or 10 s and name each null time; matching runs in a worker thread (2,127 Kalshi x 2,000 Polymarket in 3.4 s on mac-studio, scratch benchmark) |+| 1 | News vs market reaction | Companies and countries named in story clusters mapped to tickers / country indices; same-day move against a benchmark | :yellow_circle: Built 2026-10-17, unreleased; 7 test expectations await correction at merge: `intel_news_market_reaction` and `intel news-reaction`. Company rows: stock, benchmark (SPY, or the exchange country's index) and excess return on the story's session; country rows: index return with a trailing z-score. Session rule in the exchange's timezone (after close = next session; weekends and holidays read from the listing's own bars); a last bar Yahoo has not finalized is pending. Open: `^TASI` and `^TA125` (Saudi and Israel in the country-index table) return 404 on Yahoo, so those benchmarks are data gaps; story times inherit `sources/news.py`'s RSS timestamp offset until that is fixed |+| 2 | Connections with timing offsets | Correlation between retained series over a shared sample, tested at leads and lags, with minimum sample size and false-discovery control; states that it does not prove causation | :white_check_mark: Built 2026-10-17, unreleased: `intel_lagged_connections` and `intel history connections`. Pearson r on a UTC day, week or month grid at every lag from -max_lag to +max_lag (default 5, cap 30, at most 1,000 tests a call), period-over-period changes by default, p from Student's t (regularized incomplete beta, standard library), Benjamini-Hochberg adjustment across the call (default 0.05) with significance decided on the AR(1)-adjusted `q_ar1` (independent random walks in levels: 91 of 100 seeds flagged by the unadjusted q, none by `q_ar1`), "no evidence" below an effective sample of 3, a 30-point minimum with every untested pair and lag listed with its n, latest vintage, rows without `observed_at` refused with the reason, every clamp reported in the MCP response and the CLI, and a `cannot_prove` list. p-values match scipy to 3.3e-13 relative error. Verified 2026-10-17 over a live MCP stdio session and the CLI on four real FRED series loaded into an isolated store. Not done: rank correlation, prewhitening |+| 2 | Day timeline | What every retained source recorded for a given day, by observation date or collection date | :white_check_mark: Built 2026-10-17, unreleased: `intel_day_timeline` and `intel history day` (`history_timeline.py`; read-only `mode=ro` connection, never fetches). by=collected: per-source run-log state (changed, unchanged, answered, degraded, failed, not_attempted, collector_did_not_run, before_run_log, never_run, unknown_past_retention; collector ran, did_not_run, unknown_past_retention or not_recorded) with attempts, status counts, reasons, extractor failures, a partial_failure flag and payload changes; by=observed: date and UTC-instant values on the day, month and year values listed apart as period values, undated observations counted and excluded. Retention is stated per record kind for the day. Measured 2026-10-17: two live collector cycles of 6 sources four minutes apart (12 captures: ok 8, degraded 2, error 2; every payload in the second cycle matched the first) read back over a live MCP stdio session and the CLI; 2026-10-16 in the same store reads collector `not_recorded` (context `before_first_capture`) with sources `before_run_log` or `never_run` (re-measured 2026-10-18T02:16Z after the review fixes, MCP stdio and CLI). On a synthetic store (475,200 captures, 1.75 million observations, 10,982 series) one call took 24 to 46 ms (measured before the review fixes, not re-measured), and no statement full-scans a history table under SQLite 3.45.3, 3.53.4 and 3.54.0. Not measured: a live store with weeks of history |+| 3 | Dashboard views | Liquidity board, situations, top stories, collection health panels | :white_check_mark: Built 2026-10-17, unreleased: four tabs behind RESEARCH (key `p`), each loaded from its own route that calls the tool's function (`/api/liquidity-board`, `/api/situations`, `/api/top-stories`, and `/api/collection-health` on the dashboard cache's history store); a failed call renders as UNAVAILABLE or REFRESH FAILED with the function's reason, never as an empty list. Checked 2026-10-17 in headless Chrome with real upstreams (9 liquidity rows; 20 situations and 9 stories over 95 of 119 feeds; collection health after one collector cycle: 38 ok, 5 degraded, 16 failing), with FRED and ECB, every RSS feed and the history store made unreachable (each tab showed the outage and its reason), and with markup injected into a headline (shown as text). Situations and top stories take 5 to 15 s per call |Considered and not planned:diff --cc src/world_intel_mcp/cli.pyindex 60f0ab1,9c26b01..9ad4d8d--- a/src/world_intel_mcp/cli.py+++ b/src/world_intel_mcp/cli.py@@@ -3554,28 -2708,14 +3576,30 @@@ def history_health(ctx: click.Context,Text(label, style=styles.get(state, "red")),_cell(row.get("last_attempt_at") or "-"),_cell(row.get("last_success_at") or "-"),+ _cell(row.get("last_ok_at") or "-"),str(row.get("consecutive_failures", "-")),_cell(row.get("latest_observed_at") or "-"),- _cell(row.get("last_reason") or ""),+ _cell(_reason_text(row.get("last_reason"), row.get("last_stale_age_s"),+ row.get("last_stale_fetches"))),)- console.print(table)- counts = ", ".join(f"{k} {v}" for k, v in sorted(data["state_counts"].items()))+ if data["sources"]:+ console.print(table)+ for row in data.get("backfills", []):+ console.print(+ f"Backfill [bold]{escape(row['source'])}[/bold]: {row['captures']} capture(s), "+ f"last {row['last_run_at']} ({row['last_status']}"+ + (+ f", {row['last_run_failed']} of {row['last_run_requests']} request(s) failed"+ if row.get("last_run_failed") else ""+ )+ + f"), {row['observations']} "+ f"observations in {row['series']} series, observed "+ f"{escape(str(row['first_observed_at'] or '-'))} to "+ f"{escape(str(row['latest_observed_at'] or '-'))}."+ )+ if row.get("last_run_failed"):+ console.print(f" Last run failed: {escape(str(row['last_reason']))}")+ counts = ", ".join(f"{k} {v}" for k, v in sorted(data["state_counts"].items())) or "none"storage = data["storage"]console.print(f"States: {counts}. Stored: {storage['captures']} captures, "diff --cc src/world_intel_mcp/fetcher.pyindex fa454e3,a3f7f52..aa47224--- a/src/world_intel_mcp/fetcher.py+++ b/src/world_intel_mcp/fetcher.py@@@ -7,12 -10,11 +10,13 @@@ log, which the collector reads per sourimport asyncioimport contextlib+ import contextvarsimport logging+import os+import sqlite3import timefrom dataclasses import dataclass- from typing import Any, Callable-from typing import Any, Iterator++from typing import Any, Callable, Iteratorimport httpxdiff --cc src/world_intel_mcp/history.pyindex 5cbe46d,509158c..fb7862f--- a/src/world_intel_mcp/history.py+++ b/src/world_intel_mcp/history.py@@@ -24,14 -38,21 +38,25 @@@ settled on, fallback path included)one, else NULL. ``collected_at`` is when this server saved the capture.A point is stored when it is new or its value differs from the laststored value for the same series and ``observed_at``, so successive- rows for one ``(series_key, observed_at)`` are revisions.+ rows for one ``(series_key, observed_at)`` are revisions. Points are+ stored from live captures only: a stale-cache replay is not a new+ observation, so its points are extracted and counted but not stored. A+ live capture that is degraded for another reason (empty, partial,+ marked degraded) still stores the points it has.-History accrues only while the collector runs: nothing is backfilled, and-a source the collector does not run reads ``never_run``. Health counts a-success only from a live capture, so a source served from stale cache-reads ``stale`` and goes overdue on schedule. When a cycle's write fails-twice (a locked or unwritable DB), the collector appends a record of the-lost cycle to ``<db>-history-write-failures.jsonl`` beside the DB (newest-500 kept), which ``collection_health`` reports. That file is rewritten-through an exclusive temp file and never read or written through a-symlink.+History accrues while the collector runs, plus what a backfill+(``history_backfill.py``) loads on demand from FRED and the ECB Data+Portal, which serve their whole history; a source the collector does not+run reads ``never_run``. Backfills are captures under ``backfill:*``+sources, so their rows never read as collector captures, and+``collection_health`` lists them apart from the collector's roster.++Health counts a success only from a live capture, so a source served from++stale cache reads ``stale`` and goes overdue on schedule. When a cycle's++write fails twice (a locked or unwritable DB), the collector appends a++record of the lost cycle to ``<db>-history-write-failures.jsonl`` beside++the DB (newest 500 kept), which ``collection_health`` reports. That file++is rewritten through an exclusive temp file and never read or written++through a symlink.Registering an extractor (the public API)::@@@ -74,12 -98,15 +102,17 @@@ Gaps / not covered- Growth was measured on a Saturday with markets closed: the 15 Yahoo quoteseries, which have no observation date, gain a row per price change whilemarkets trade, and that rate is not measured.-- No backfill: sources that serve history (FRED CSV) are not replayed into- the store; only what the collector sees from the first recorded cycle on.+- Backfill covers FRED graph CSV and ECB Data Portal series only, and+ stores the publisher's current vintage: revisions made before a+ backfill are invisible (see ``history_backfill.py``). Every other source+ has only what the collector saw from its first recorded cycle on.- ``published_at`` is NULL for every built-in extractor: none of theirpayloads names a release time.+ - A built-in extractor skips a row whose date it cannot read (not ASCII+ ISO 8601, or no such calendar date) without counting it; that shows+ only when every row is skipped, which reads ``degraded``. A point with+ such a date that an extractor does return is counted in+ ``extract_error``.- ``record_count`` is a size hint (the payload's longest top-level list,else its integer ``count``), not a semantic count; payloads shaped asnested dicts read NULL.@@@ -124,21 -183,19 +189,29 @@@ OVERDUE_MULTIPLE = 3._DEFAULT_PAYLOAD_DAYS = 7.0_DEFAULT_CAPTURE_DAYS = 30.0_DEFAULT_HISTORY_DAYS = 365.0+ # No body in the 2026-10-17 three-cycle measurement can have exceeded+ # 1 MiB of JSON (the largest cycle added 958,424 bytes across 10 bodies),+ # so the cap stops a runaway payload, not a real one.+ _DEFAULT_MAX_BODY_BYTES = 4 * 1024 * 1024+ _DEFAULT_REPLAY_MAX_BYTES = 256 * 1024_MAX_LIMIT = 1000+ _WRITE_FAILURES_SUFFIX = "-history-write-failures.jsonl"+ _WRITE_FAILURES_KEPT = 500+HISTORY_ACCRUAL_NOTE = ("History accrues only while the collector (intel-collector) runs; "- "nothing before its first recorded cycle exists here."+ "nothing before its first recorded cycle exists here, except series under "+ "backfill:* sources, which a backfill loaded from the publisher's current data."+)++# Source-name prefix of backfill captures (history_backfill.py). They run on+# demand, so collection_health reports them apart from the cadence roster.+BACKFILL_PREFIX = "backfill:"+BACKFILL_VINTAGE_NOTE = (+ "Backfilled: each value is the publisher's vintage on the day it was "+ "collected (collected_at is the backfill time), so revisions made before the "+ "first backfill are invisible; later backfills store changed values as revisions.")@@@ -709,17 -1162,17 +1178,19 @@@ class HistoryStoreknown = set(roster)conn = self._connlogged = [r[0] for r in conn.execute("SELECT DISTINCT source FROM history_captures")]- names = roster + sorted(set(logged) - known)+ backfilled = sorted(s for s in logged if s.startswith(BACKFILL_PREFIX))+ names = roster + sorted(set(logged) - known - set(backfilled))if source is not None:- if source not in names:+ if source not in names and source not in backfilled:return {"error": f"unknown source {source!r}: not a collector source and never captured",- "known_sources": sorted(names),+ "known_sources": sorted(names + backfilled),}- names = [source]+ names = [source] if source in names else []+ backfilled = [source] if source in backfilled else []+ failures, unreadable_failures = read_write_failures(self.db_path, now)+ live_args = (len(STALE_REASON), STALE_REASON)rows = []for name in names:cadence = overrides.get(name, default_cadence)@@@ -828,56 -1348,38 +1368,76 @@@"not healthy.",],"states_explained": {- "ok": "last capture delivered records with no degradation marker",- "degraded": "last capture delivered a payload marked stale/degraded, empty, "- "or without the measures its extractor expects",- "failing": "last capture was an error (consecutive_failures counts the run; "- f"overdue is true when no payload arrived within {OVERDUE_MULTIPLE:g} "- "cadences)",+ "ok": "last capture was live and complete: records, every upstream fetch "+ "answered, no degradation marker",+ "degraded": "last capture was live but degraded: marked degraded, empty, "+ "partial (fewer series than expected), or its extractor found nothing or "+ "raised",+ "stale": "last capture replayed an expired cached copy because an upstream "+ "fetch failed; that is not a success, so overdue counts from the last live "+ "capture",+ "failing": "last capture was an error (overdue is true when no live payload "+ f"arrived within {OVERDUE_MULTIPLE:g} cadences)","not_attempted_recently": f"the collector has not attempted it within "- f"{OVERDUE_MULTIPLE:g} cadences (overdue is then always true)",+ f"{OVERDUE_MULTIPLE:g} cadences, or its attempts were not recorded (see "+ "unrecorded_attempts); overdue is then always true","never_run": "no capture recorded",},+ "fields_explained": {+ "last_success_at": "last live capture: ok, or degraded for any reason but a "+ "stale-cache replay; overdue and last_success_age_s count from it",+ "last_ok_at": "last ok capture: live, complete and unmarked",+ "consecutive_failures": "errors and stale-cache replays recorded since the "+ "last live capture",+ "unrecorded_attempts": "cycles after this source's last recorded capture "+ "whose history write failed",+ "last_stale_age_s": "present when the last capture was a stale-cache "+ "replay: age in seconds of the oldest expired copy it served (null when "+ "none was reported)",+ "last_stale_fetches": "present when the last capture was a stale-cache "+ "replay: how many upstream fetches the fetcher answered from expired "+ "cache (null when only the payload's _stale marker showed the replay)",+ },}+ def _backfill_row(self, name: str) -> dict:+ """One backfill source: its captures, its latest run and the series+ it stored. Every capture of one backfill call shares one+ collected_at, so the latest run is all captures at the latest time:+ error if any request in it failed (with the first failure's reason),+ whatever the order the requests were named in."""+ conn = self._conn+ captures, last_at = conn.execute(+ "SELECT COUNT(*), MAX(collected_at) FROM history_captures WHERE source = ?", (name,)+ ).fetchone()+ run = conn.execute(+ "SELECT status, error FROM history_captures WHERE source = ? AND collected_at = ? "+ "ORDER BY id",+ (name, last_at),+ ).fetchall()+ failed = [r for r in run if r[0] == "error"]+ last_status = "error" if failed else (run[-1][0] if run else None)+ last_reason = failed[0][1] if failed else None+ series, points, first_obs, last_obs = conn.execute(+ "SELECT COUNT(DISTINCT series_key), COUNT(*), MIN(observed_at), MAX(observed_at) "+ "FROM history_observations WHERE source = ?",+ (name,),+ ).fetchone()+ return {+ "source": name,+ "captures": captures,+ "last_run_at": _iso(last_at),+ "last_status": last_status,+ "last_reason": last_reason,+ "last_run_requests": len(run),+ "last_run_failed": len(failed),+ "series": series,+ "observations": points,+ "first_observed_at": first_obs,+ "latest_observed_at": last_obs,+ "note": BACKFILL_VINTAGE_NOTE,+ }+def capture_history(self, source: str, limit: Any = 20, capture_id: Any = None) -> dict:diff --cc src/world_intel_mcp/tools/history.pyindex 3f52de2,5f0b66f..b4195c8--- a/src/world_intel_mcp/tools/history.py+++ b/src/world_intel_mcp/tools/history.py@@@ -25,11 -23,12 +25,13 @@@ TOOLS: list[Tool] =name="intel_collection_health",description=("Per-source collection health from the collector's run log: last attempt, "- "last success, consecutive failures, overdue against the expected cadence "- "(default 300 s, overdue past 3 cadences), never-run sources, backfill runs "- "listed apart (backfills), and history storage size. States what it cannot "- "prove: a fresh capture does not mean upstream observations are current (see "- "latest_observed_at). " + _ACCRUAL+ "last live success (a stale-cache replay never counts; such a source reads "+ "stale), last fully ok capture, consecutive failures, overdue against the "+ "expected cadence (default 300 s, overdue past 3 cadences), never-run "- "sources, cycles whose history write failed, and history storage size. "- "States its data gaps and what it cannot prove: a fresh capture does not "- "mean upstream observations are current (see latest_observed_at). " + _ACCRUAL++ "sources, backfill runs listed apart (backfills), cycles whose history "++ "write failed, and history storage size. States its data gaps and what it "++ "cannot prove: a fresh capture does not mean upstream observations are "++ "current (see latest_observed_at). " + _ACCRUAL),inputSchema={"type": "object",