Skip to content

160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.

Merge feat/history-backfill: FRED and ECB history into the store, three history extractors (Phase 29)

Phoenix Agent (mac-studio)fef80e5b5e1a
diff --cc CHANGELOG.md
index b25de08,4ea431c..2b01ab6
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -349,56 -229,60 +349,110 @@@ source here, and no proxy is substitute
history capture is an error.
- Tools 139 -> 143.
+### Added: connections with timing offsets (Phase 29)
+- `intel_lagged_connections` (CLI: `intel history connections`): Pearson
+ correlation between 2 to 8 series in the history store (keys as
+ `intel_series_history` lists them), aligned on a UTC day, week (Monday to
+ Sunday) or month grid and tested at every lag from `-max_lag` to
+ `+max_lag` (default 5, at most 30, reduced so one call runs at most 1,000
+ tests). Lag +k compares A at period t with B at t+k, so A leads B by k.
+- Default transform: period-over-period changes, never taken across a gap;
+ `levels` on request. Responses say which ran and why changes are the
+ default (levels of trending series correlate spuriously).
+- Per pair: the lag with the smallest p, with r, n (overlapping periods
+ after the transform and shift), a two-sided p from Student's t (computed
+ from the regularized incomplete beta function, standard library only) and
+ the Benjamini-Hochberg q across every test in the call, with the
+ Benjamini-Yekutieli `q_by` alongside. Significant at q <= 0.05 by default
+ (allowed 0.001 to 0.25); the other significant lags are listed too.
+- A pair and lag with fewer than 30 overlapping periods (callers may go down
+ to 20), or a constant side, is listed as not tested with its n, never
+ dropped. 30 points give about 80% power for a true |r| of 0.5 at alpha
+ 0.05 (module docstring).
+- One vintage, the latest collected value for each `observed_at`. Rows
+ without `observed_at` (Yahoo quotes) are not used, with the reason: rows
+ are stored only on change, so a missing day cannot be told from an
+ unchanged one. A period takes its last observation; nothing is averaged.
+- Serial correlation: each series' lag-1 autocorrelation after the
+ transform, and per strongest result an AR(1) effective sample size and
+ the p recomputed with it, as a diagnostic outside q. Nothing is
+ prewhitened.
+- `cannot_prove` in every response: correlation is not causation, a lead is
+ not a causal direction, serial correlation inflates significance, a
+ shared driver produces correlation, the store holds only what the
+ collector recorded, BH under dependence is approximate and covers one
+ call, and a small q is not a large effect.
+- An unknown grid or transform is an error naming the allowed values;
+ numeric inputs are clamped and each clamp is reported.
+- Measured 2026-10-17 over a live MCP stdio session and the CLI, on four
+ real FRED series (WALCL, WDTGAL, WTREGEN, RRPONTSYD) loaded into an
+ isolated store as one capture; illustrative, not findings. Week grid,
+ changes: 26 of 66 tests significant; the Wednesday TGA level against its
+ week average gives r 0.61 at lag 0 and 0.59 at lag +1, the averaging
+ window rather than a cause. Levels: Fed total assets against either TGA
+ series gives r 0.74 at lag 0 (n 1,244) with an AR(1) effective sample
+ size of 5.3 and 3.4. Day grid: the weekly series have no adjacent-day
+ changes, so nothing is tested and the response says why.
+- With the default 365-day observation retention
+ (`WORLD_INTEL_HISTORY_DAYS`), a store filled only by the collector spans
+ about a year, so the month grid cannot reach the 20-point floor without a
+ longer retention or backfilled history.
+- Tools 152 -> 153.
+
+ ### Added: history backfill and more history extractors (Phase 29)
+ - `intel history backfill fred SERIES... [--start YYYY-MM-DD]`,
+ `intel history backfill ecb DATASET.KEY... [--start YYYY-MM-DD]` and the
+ MCP tool `intel_history_backfill` load the history that FRED's keyless graph CSV
+ and the ECB Data Portal (BSI and ILM datasets) already serve into the
+ history store. Requests reuse the liquidity tools' fetch and parse code,
+ so they go through the Fetcher (FRED rate floor, circuit breaker, cache):
+ a copy cached by an earlier call (FRED 1 h, ECB 3 h) is reused without a
+ request.
+ - Backfilled rows have their own sources, `backfill:fred` and
+ `backfill:ecb` (series `backfill:fred:<ID>` and
+ `backfill:ecb:<DATASET>.<KEY>`), so they never read as collector
+ captures; `intel_collection_health` and `intel history health` list them
+ apart (`backfills`). Every run is logged in a new table,
+ `history_backfills`, created in place with `CREATE TABLE IF NOT EXISTS`
+ (existing rows untouched).
+ - Dates: `observed_at` is the publisher's period (FRED's date; ECB months
+ as written, ECB weeks as the Sunday that ends the ISO week);
+ `published_at` is NULL, because neither response names a release or
+ vintage date; `collected_at` is the backfill time, stamped after the
+ call's last fetch. Values are the publisher's vintage on the backfill
+ day, already revised, so revisions made before the first backfill are
+ invisible.
+ - Idempotent: a re-run stores only new or changed values, and a changed
+ value is a revision in `intel_revisions`. FRED's missing cells (`.` and
+ blank) are skipped and counted, never stored as zero. ECB missing values
+ cannot be counted (the SDMX-JSON parser drops them) and read `unknown`.
+ - Failures are errors, never an empty success: an HTTP error, an empty or
+ malformed body, a window without values, and a response available only
+ as a stale cache copy each record an error capture and store nothing.
+ The CLI exits 1 when any request failed.
+ - Caps per call: 5 FRED series or 2 ECB keys (at most 20 series per key),
+ the newest 20,000 points per series and 100,000 per call; `--start`
+ narrows the window.
+ - Retention: observations are pruned by `collected_at`, not `observed_at`,
+ so a 1962 value is kept as long as any value stored by the same backfill
+ (`WORLD_INTEL_HISTORY_DAYS`, default 365). A backfill prunes before it
+ stores, so a pruned value is stored again as new.
+ - New history extractors (`history_extractors.py`): `liquidity_board`
+ (latest and previous value per indicator; the derived net liquidity row
+ is left to its own source), `net_liquidity` (net liquidity and its three
+ terms per Wednesday, reverse repo dated by the day it was taken from) and
+ `net_migration` (World Bank SM.POP.NETM per country and year). Each
+ dates a value only from the payload.
+ - Measured 2026-10-17, live, isolated HOME: WALCL 1,244 values
+ (2002-12-18 to 2026-10-14), RRPONTSYD 3,348 (2,833 blank cells skipped),
+ DGS10 16,182 (721 skipped; units unknown, it is not in this server's
+ FRED catalog), euro area M3 560 months (1980-01 to 2026-08, 1
+ provisional). A second run from the cached copies and a refetch of WALCL
+ and M3 stored 0.
+ - FRED's terms of use speak of personal, non-commercial use of FRED data;
+ whether a deployment fits them is the operator's call.
+ - Tools 152 -> 153; the `intel history` group gains `backfill fred|ecb`.
+
### Added: news signals (ROADMAP Phase 29)
- `intel_hostility_timeline`: per UTC day, GDELT DOC 2.0 coverage of a
country or a pair of countries (`timelinevolraw`): matching articles,
diff --cc README.md
index a3b943a,f6fc7fc..a5570d2
--- a/README.md
+++ b/README.md
@@@ -6,7 -6,7 +6,7 @@@
[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-green)](https://python.org)
[![License](https://img.shields.io/badge/License-MIT-yellow)](LICENSE)
- Real-time global intelligence across **30+ domains** with **159 MCP tools**, a live ops-center dashboard, a CLI, and a **Qdrant vector store** for enterprise-grade semantic search across accumulated intelligence. All data comes from free, public APIs: no paid subscriptions required.
-Real-time global intelligence across **30+ domains** with **153 MCP tools**, a live ops-center dashboard, a CLI, and a **Qdrant vector store** for enterprise-grade semantic search across accumulated intelligence. All data comes from free, public APIs: no paid subscriptions required.
++Real-time global intelligence across **30+ domains** with **160 MCP tools**, a live ops-center dashboard, a CLI, and a **Qdrant vector store** for enterprise-grade semantic search across accumulated intelligence. All data comes from free, public APIs: no paid subscriptions required.
Built for AI agents that need world awareness: market conditions, geopolitical risk, military posture, aircraft-cohort early warning, refugee and migration flows, supply chain disruptions, cyber threats, and more — all queryable via the Model Context Protocol. The vector store enables natural language queries like *"military activity near Taiwan"* or *"cyber threats targeting healthcare"* across all historical data.
@@@ -66,10 -66,9 +66,10 @@@
| **Volcanoes** | 1 | Smithsonian GVP weekly activity report |
| **Tropical Cyclones** | 2 | NHC (Atlantic, E/C Pacific) + JTWC (NW Pacific, N Indian, S Hemisphere), with opt-in forecast tracks: positions, UTC valid times and intensities, plus NHC's cone of uncertainty |
| **Situation Brief** | 1 | Cited situational awareness brief over MCP: bounded server-side overview synthesized via local Ollama, with a mechanically-cited fallback |
- | **Collection History** | 5 | Collector run log (per-source health, overdue and failing sources), saved captures with change detection and replay, dated observation series, revisions, and one UTC day's timeline by collection or observation date |
-| **Collection History** | 5 | Collector run log (per-source health, overdue and failing sources), saved captures with change detection and replay, dated observation series, revisions, and on-demand backfill of the history FRED and the ECB Data Portal already serve |
++| **Collection History** | 6 | Collector run log (per-source health, overdue and failing sources), saved captures with change detection and replay, dated observation series, revisions, one UTC day's timeline by collection or observation date, and on-demand backfill of the history FRED and the ECB Data Portal already serve |
+| **Connections Between Series** | 1 | Correlation between stored series at leads and lags, with a minimum sample, false-discovery control and what it cannot prove |
- **Total: 159 tools** across 30+ intelligence domains.
-**Total: 153 tools** across 30+ intelligence domains.
++**Total: 160 tools** across 30+ intelligence domains.
---
@@@ -146,9 -142,8 +146,11 @@@ intel prediction-settled # Brier scor
intel status # cache + circuit breaker health
intel guide # the research guide (also served as intel://guide)
intel history health # per-source collection health from the collector's run log
+intel history connections central_bank_rates:federal_funds_rate central_bank_rates:boe_bank_rate_sonia
+ # correlation at leads and lags between stored series (day grid by default)
+intel history day 2026-10-17 --by observed # values observed on one UTC day (default: captures that day)
+ intel history backfill fred WALCL RRPONTSYD --start 2020-01-01 # load FRED history into the history store
+ intel history backfill ecb BSI.M.U2.Y.V.M30.X.1.U2.2300.Z01.E # load ECB Data Portal history (euro area M3)
```
---
@@@ -620,8 -598,8 +622,8 @@@ different, openly licensed data source
|------|-------------|
| `intel_situation_brief` | Cited situational awareness brief, generated on demand over MCP: a bounded server-side overview (earthquakes, military flights, ACLED conflict events, wildfires, cyber threats, disease outbreaks, news, space weather, strategic posture, alert digest), synthesized via local Ollama or a mechanically-cited fallback when Ollama is unreachable |
--### Collection History (5)
- History accrues only while the collector (`intel-collector`) runs; nothing before its first recorded cycle exists. Every observation keeps three dates: `observed_at` (the period the value describes, from the payload, or null when the payload has none), `published_at` (publisher release, when known) and `collected_at` (when this server saved it).
++### Collection History (6)
+ History accrues only while the collector (`intel-collector`) runs; nothing before its first recorded cycle exists, except what a backfill loads (`intel_history_backfill`, `intel history backfill`): FRED graph CSV and ECB Data Portal series, stored under `backfill:fred` and `backfill:ecb` with the publisher's current vintage, so revisions made before the first backfill are invisible. Every observation keeps three dates: `observed_at` (the period the value describes, from the payload, or null when the payload has none), `published_at` (publisher release, when known) and `collected_at` (when this server saved it).
| Tool | Description |
|------|-------------|
@@@ -629,15 -607,8 +631,16 @@@
| `intel_capture_history` | A source's earlier captures (ok / degraded / error with reason, duration, record count) and whether each changed from the previous one; `capture_id` replays one stored payload |
| `intel_series_history` | One series' observations with all three dates, or the list of available series |
| `intel_revisions` | Values for the same `observed_at` that a later capture changed (USGS magnitude revisions, NOAA Kp updates) |
+| `intel_day_timeline` | What the history store holds for one UTC day (`YYYY-MM-DD`; a future day is an error). `by=collected` (default): per source, that day's captures (attempts, ok / degraded / error with reasons, changed vs unchanged payloads, first and last capture, observations stored), and for a source with no capture whether the collector did not run that day, ran without it, never ran it, or the day is past run-log retention. `by=observed`: values whose `observed_at` falls on the day; month and year values containing the day are listed apart as period values, and undated observations are counted and excluded. Every item carries `observed_at`, `published_at` and `collected_at`; pruned records are named, never read as a quiet day; per-source and total caps with truncation disclosed. Read-only, never fetches |
+ | `intel_history_backfill` | Load the history FRED (graph CSV) or the ECB Data Portal already serve into the history store: one capture per series under `backfill:fred` / `backfill:ecb`, `observed_at` from the publisher's dates, `published_at` null, `collected_at` the backfill time; re-runs store only new or changed values (a change is a revision); missing values skipped and counted; writes and fetches, so capped at 5 FRED series or 2 ECB keys per call |
+### Connections Between Series (1)
+Reads the history store only, so the sample is what the collector saved, not each series' full history. A connection is a correlation in that sample: it does not show causation, and a lead does not show causal direction.
+
+| Tool | Description |
+|------|-------------|
+| `intel_lagged_connections` | Correlation between 2 to 8 stored series (keys as `intel_series_history` lists them) on a UTC day, week or month grid, tested at every lag from `-max_lag` to `+max_lag` (default 5, at most 30, at most 1,000 tests a call). Default transform is period-over-period changes, because levels of trending series correlate spuriously; `levels` on request. Per pair: the lag with the smallest p, with Pearson r, overlap n, two-sided p from Student's t and the Benjamini-Hochberg q across every test in the call (significant at q <= 0.05 by default). Lag +k compares A at period t with B at t+k, so A leads B by k. Pair/lag overlaps under 30 points are listed as not tested with their n; revised points use the latest vintage; rows without `observed_at` (Yahoo quotes) are not used. Each strongest result also carries an AR(1) effective sample size as a serial-correlation diagnostic |
+
---
## Watching your own area (geofences/AOIs)
diff --cc src/world_intel_mcp/cli.py
index 9074957,7005d22..45764bf
--- a/src/world_intel_mcp/cli.py
+++ b/src/world_intel_mcp/cli.py
@@@ -3581,300 -3328,106 +3591,400 @@@ def history_revisions
console.print(f"[yellow]Cannot prove:[/yellow] {escape(line)}")
+from .analysis import lagged_connections as _lagged # noqa: E402
+
+
+def _p_text(value: float) -> str:
+ return "<1e-300" if value == 0 else f"{value:.2g}"
+
+
+@history_group.command(name="connections")
[email protected]("series_keys", nargs=-1, required=True)
[email protected]("--grid", type=click.Choice(_lagged.GRIDS), default=_lagged.DEFAULT_GRID,
+ show_default=True, help="UTC calendar grid (weeks run Monday to Sunday)")
[email protected]("--transform", type=click.Choice(_lagged.TRANSFORMS),
+ default=_lagged.DEFAULT_TRANSFORM, show_default=True,
+ help="changes (period-over-period) or levels")
[email protected]("--max-lag", type=int, default=_lagged.DEFAULT_MAX_LAG, show_default=True,
+ help=f"Largest lead or lag in grid periods (0 to {_lagged.MAX_LAG_CAP})")
[email protected]("--min-overlap", type=int, default=_lagged.DEFAULT_MIN_OVERLAP, show_default=True,
+ help=f"Fewest overlapping periods a test needs (at least {_lagged.MIN_OVERLAP_FLOOR})")
[email protected]("--q", "q", type=float, default=_lagged.DEFAULT_Q, show_default=True,
+ help="False discovery rate threshold (Benjamini-Hochberg)")
[email protected]("--start", default=None, help="First date YYYY-MM-DD")
[email protected]("--end", default=None, help="Last date YYYY-MM-DD")
+def history_connections(
+ ctx: click.Context, series_keys: tuple[str, ...], grid: str, transform: str,
+ max_lag: int, min_overlap: int, q: float, start: str | None, end: str | None,
+) -> None:
+ """Correlation between stored series at leads and lags.
+
+ Lag +k compares the first series of a pair at period t with the second
+ at t+k, so the first leads by k. A connection is not a cause.
+ """
+ data = _with_history_store(
+ lambda s: _lagged.lagged_connections(
+ s, list(series_keys), grid=grid, transform=transform, max_lag=max_lag,
+ min_overlap=min_overlap, q=q, start=start, end=end,
+ )
+ )
+ if _bail_on_error(ctx, data):
+ return
+
+ method = data["method"]
+ lo, hi = method["lags"]
+ table = Table(
+ title=f"Connections ({method['grid']} grid, {method['transform']}, lags {lo:+d} to "
+ f"{hi:+d}, {data['tests_run']} tests, BH q <= {method['q_threshold']:g})",
+ box=box.SIMPLE_HEAVY,
+ )
+ for name in ("A", "B", "Lag", "Timing", "r", "n", "n_eff", "p", "q", "Significant"):
+ justify = "right" if name in ("Lag", "r", "n", "n_eff", "p", "q") else "left"
+ table.add_column(name, justify=justify)
+ for pair in data["pairs"]:
+ if not pair["strongest"]:
+ table.add_row(_cell(pair["a"]), _cell(pair["b"]), "-",
+ Text("not tested", style="dim"), "-", "-", "-", "-", "-", "-")
+ for entry in pair["strongest"]:
+ n_eff = entry.get("n_effective_ar1")
+ table.add_row(
+ _cell(pair["a"]), _cell(pair["b"]), f"{entry['lag']:+d}", entry["timing"],
+ f"{entry['r']:+.3f}", str(entry["n"]), "-" if n_eff is None else f"{n_eff:g}",
+ _p_text(entry["p"]), _p_text(entry["q"]),
+ Text("yes", style="green") if entry["significant"] else Text("no"),
+ )
+ console.print(table)
+ console.print(escape(data["summary"]))
+ console.print(
+ "n_eff: AR(1) effective sample size from the two series' lag-1 autocorrelations, "
+ "a diagnostic only: p and q use n."
+ )
+ console.print(f"Lag convention: {escape(method['lag_convention'])}")
+ console.print(
+ f"Transform: {escape(method['transform_rule'])} {escape(method['why_changes_default'])}"
+ )
+ for pair in data["pairs"]:
+ best = {e["lag"] for e in pair["strongest"]}
+ others = [e for e in pair["significant"] if e["lag"] not in best]
+ if others:
+ listed = ", ".join(
+ f"lag {e['lag']:+d} (r {e['r']:+.3f}, q {_p_text(e['q'])})" for e in others
+ )
+ console.print(f"Also significant: {escape(pair['a'])} / {escape(pair['b'])}: {listed}")
+ if pair["not_tested"]:
+ listed = ", ".join(
+ f"lag {nt['lag']:+d} (n {nt['n']}"
+ + (", no variation)" if nt["reason"] == "no_variation" else ")")
+ for nt in pair["not_tested"]
+ )
+ console.print(
+ f"[yellow]Not tested[/yellow] (minimum {method['min_overlap']} overlapping "
+ f"periods): {escape(pair['a'])} / {escape(pair['b'])}: {listed}"
+ )
+ for item in data["excluded_series"]:
+ console.print(
+ f"[yellow]Excluded:[/yellow] {escape(item['series_key'])}: {escape(item['reason'])}"
+ )
+ _print_data_gaps(data)
+ for line in data["cannot_prove"]:
+ console.print(f"[yellow]Cannot prove:[/yellow] {escape(line)}")
+
+
+@history_group.command(name="day")
+ "--by",
+ type=click.Choice(["collected", "observed"]),
+ default="collected",
+ show_default=True,
+ help="collected: what was captured that day; observed: values whose observed_at is that day",
+)
[email protected]("--source", default=None, help="One collector source name")
[email protected]("--limit", default=None, type=int, help="Most items listed in all")
[email protected]("--per-source", "per_source", default=None, type=int, help="Most items listed per source")
+def history_day(
+ ctx: click.Context,
+ day: str,
+ by: str,
+ source: str | None,
+ limit: int | None,
+ per_source: int | None,
+) -> None:
+ """What the history store recorded for one UTC day (YYYY-MM-DD)."""
+ import time
+
+ from .collector import SOURCES
+ from .history_timeline import (
+ DEFAULT_LIMIT,
+ DEFAULT_PER_SOURCE_LIMIT,
+ day_timeline,
+ parse_day,
+ )
+
+ now = time.time()
+ _, problem = parse_day(day, now)
+ if problem:
+ raise click.BadParameter(problem, param_hint="DAY")
+ data = day_timeline(
+ _get_fetcher().cache.db_path,
+ day,
+ by=by,
+ source=source,
+ roster=[name for name, _, _, _ in SOURCES],
+ limit=DEFAULT_LIMIT if limit is None else limit,
+ per_source_limit=DEFAULT_PER_SOURCE_LIMIT if per_source is None else per_source,
+ now=now,
+ )
+ if _bail_on_error(ctx, data):
+ return
+
+ window = data["window"]
+ span = f"{window['start']} to {window['end']} (end exclusive)"
+ if not window["complete"]:
+ span += f", in progress through {window['through']}"
+ console.print(f"[bold]{data['day']}[/bold] (UTC) by {data['by']} date: {span}")
+ if by == "collected":
+ _print_day_collected(data)
+ else:
+ _print_day_observed(data)
+ _print_data_gaps(data)
+ for line in data["cannot_prove"]:
+ console.print(f"[yellow]Cannot prove:[/yellow] {escape(line)}")
+
+
+def _print_day_collected(data: dict) -> None:
+ c = data["collector"]
+ if c["state"] == "ran":
+ console.print(
+ f"Collector: ran, {c['captures']:,} captures of {c['sources_captured']} "
+ f"source(s), {c['first_capture_at']} to {c['last_capture_at']}; longest gap "
+ f"{c['longest_gap_s']:,.0f} s ({c['longest_gap_from']} to {c['longest_gap_to']})"
+ )
+ else:
+ console.print(f"Collector: [yellow]{c['state']}[/yellow]. {escape(c['explanation'])}")
+
+ styles = {"changed": "green", "unchanged": "green", "answered": "green",
+ "failed": "red", "never_run": "dim"}
+ table = Table(title=f"Sources on {data['day']}", box=box.SIMPLE_HEAVY)
+ for name in ("Source", "State", "Tries", "OK", "Degr", "Err", "Changed", "Same",
+ "First", "Last", "Obs", "Reason / context"):
+ table.add_column(name, justify="right" if name in {
+ "Tries", "OK", "Degr", "Err", "Changed", "Same", "Obs"} else "left")
+ for row in data["sources"]:
+ counts = row.get("status_counts") or {}
+ changes = row.get("payload_changes") or {}
+ if row["attempts"]:
+ top = (row.get("reasons") or [None])[0]
+ context = f"{top['reason']} (x{top['count']})" if top else ""
+ if row.get("more_reasons"):
+ context += f" +{row['more_reasons']} more"
+ stored = row.get("observations_stored")
+ obs = "-" if stored is None else str(stored)
+ else:
+ parts = []
+ if row.get("last_capture_before"):
+ parts.append(f"last before {row['last_capture_before']}")
+ if row.get("next_capture_after"):
+ parts.append(f"next after {row['next_capture_after']}")
+ context = ", ".join(parts)
+ obs = str(row["observations_retained"]) if row["observations_retained"] else "-"
+ table.add_row(
+ _cell(row["source"]),
+ Text(row["day_state"], style=styles.get(row["day_state"], "yellow")),
+ str(row["attempts"]),
+ str(counts.get("ok", "-")),
+ str(counts.get("degraded", "-")),
+ str(counts.get("error", "-")),
+ (f"{changes['changed']}" + (f" +{changes['first']} first" if changes["first"] else ""))
+ if changes else "-",
+ str(changes.get("unchanged", "-")),
+ _cell(row.get("first_capture_at") or "-"),
+ _cell(row.get("last_capture_at") or "-"),
+ obs,
+ _cell(context),
+ )
+ console.print(table)
+ states = ", ".join(f"{k} {v}" for k, v in sorted(data["state_counts"].items()))
+ console.print(f"States: {states}.")
+
+ items = data["observations"]
+ trunc = data["truncation"]
+ if items:
+ obs_table = Table(
+ title=f"Observations saved that day ({trunc['returned']:,} of {trunc['matched']:,})",
+ box=box.SIMPLE_HEAVY,
+ )
+ for name in ("Collected", "Source", "Series", "Observed", "Published", "Value", "Unit"):
+ obs_table.add_column(name, justify="right" if name == "Value" else "left")
+ for item in items:
+ obs_table.add_row(
+ item["collected_at"],
+ _cell(item["source"]),
+ _cell(item["series_key"]),
+ _cell(item["observed_at"] or "unknown"),
+ _cell(item["published_at"] or "unknown"),
+ f"{item['value']:g}",
+ _cell(item.get("unit") or ""),
+ )
+ console.print(obs_table)
+ else:
+ console.print("[dim]No observations saved that day.[/dim]")
+
+
+def _print_day_observed(data: dict) -> None:
+ def _points_table(title: str, points: list[dict]) -> Table:
+ table = Table(title=title, box=box.SIMPLE_HEAVY)
+ for name in ("Observed", "Precision", "Series", "Value", "Unit", "Published",
+ "Collected", "Stored"):
+ table.add_column(name, justify="right" if name in {"Value", "Stored"} else "left")
+ for point in points:
+ stored = str(point["stored_values"])
+ if point["stored_values"] > 1:
+ stored += f" (first {point['first_value']:g} at {point['first_collected_at']})"
+ table.add_row(
+ _cell(point["observed_at"]),
+ point["precision"],
+ _cell(point["series_key"]),
+ f"{point['value']:g}",
+ _cell(point.get("unit") or ""),
+ _cell(point["published_at"] or "unknown"),
+ point["collected_at"],
+ stored,
+ )
+ return table
+
+ day_trunc = data["truncation"]["day_values"]
+ if data["day_values"]:
+ console.print(_points_table(
+ f"Values observed on {data['day']} ({day_trunc['returned']:,} of "
+ f"{day_trunc['matched']:,})",
+ data["day_values"],
+ ))
+ else:
+ console.print(f"[yellow]No stored value is observed on {data['day']}.[/yellow]")
+ if data["period_values"]:
+ period_trunc = data["truncation"]["period_values"]
+ console.print(_points_table(
+ f"Period values containing the day, not values for it "
+ f"({period_trunc['returned']:,} of {period_trunc['matched']:,})",
+ data["period_values"],
+ ))
+ console.print(f"[dim]{escape(data['period_values_note'])}[/dim]")
+ if data["sources"]:
+ table = Table(title="By source", box=box.SIMPLE_HEAVY)
+ for name in ("Source", "Day values", "Revised", "Period values", "No observed_at"):
+ table.add_column(name, justify="left" if name == "Source" else "right")
+ for row in data["sources"]:
+ table.add_row(
+ _cell(row["source"]),
+ str(row["day_values"]),
+ str(row["day_values_revised"]),
+ str(row["period_values"]),
+ str(row["without_observed_at_collected_that_day"]),
+ )
+ console.print(table)
+ # ---------------------------------------------------------------------------
+ # Backfill (history FRED and the ECB already serve, into the history store)
+ # ---------------------------------------------------------------------------
+
+
+ @history_group.group(name="backfill")
+ def backfill_group() -> None:
+ """Load history FRED or the ECB Data Portal already serve into the history store.
+
+ Values are the publisher's current vintage, so revisions made before the
+ first backfill are invisible. Read them back with intel history series.
+ """
+
+
+ def _run_backfill(
+ ctx: click.Context, provider: str, requests: tuple[str, ...], start: str | None
+ ) -> None:
+ from . import history_backfill
+ from .history import HistoryStore
+
+ fetcher = _get_fetcher()
+ run = history_backfill.backfill_fred if provider == "fred" else history_backfill.backfill_ecb
+ store = HistoryStore(fetcher.cache.db_path)
+ try:
+ data = _run(run(fetcher, store, list(requests), start))
+ finally:
+ store.close()
+ if ctx.obj.get("json"):
+ _print_json(data)
+ else:
+ _print_backfill(data)
+ if "error" in data or data["totals"]["failed"]:
+ ctx.exit(1)
+
+
+ def _print_backfill(data: dict) -> None:
+ if not data.get("requests"):
+ console.print(f"[red]Error:[/red] {escape(str(data['error']))}")
+ return
+ title = f"Backfill into {data['source']} at {data['collected_at']}"
+ if data.get("start"):
+ title += f" from {data['start']}"
+ table = Table(title=escape(title), box=box.SIMPLE_HEAVY)
+ table.add_column("Series", style="bold")
+ table.add_column("Status")
+ table.add_column("Loaded", justify="right")
+ table.add_column("New", justify="right")
+ table.add_column("Revised", justify="right")
+ table.add_column("Unchanged", justify="right")
+ table.add_column("Missing", justify="right")
+ table.add_column("Observed from")
+ table.add_column("Observed to")
+ for req in data["requests"]:
+ if req["status"] == "error":
+ table.add_row(_cell(req["request"]), Text("error", style="red"), *["-"] * 7)
+ continue
+ for s in req["series"]:
+ table.add_row(
+ _cell(s["series_key"]),
+ Text("ok", style="green"),
+ str(s["loaded"]),
+ str(s["stored_new"]),
+ str(s["stored_revised"]),
+ str(s["unchanged"]),
+ "unknown" if s["missing_skipped"] is None else str(s["missing_skipped"]),
+ _cell(s["first_observed_at"] or "-"),
+ _cell(s["last_observed_at"] or "-"),
+ )
+ console.print(table)
+ totals = data["totals"]
+ console.print(
+ f"Stored {totals['stored_new']} new and {totals['stored_revised']} revised values, "
+ f"{totals['unchanged']} unchanged, in {escape(data['db_path'])}."
+ )
+ _print_data_gaps(data)
+
+
+ @backfill_group.command(name="fred")
+ @click.argument("series_ids", nargs=-1, required=True)
+ @click.option("--start", default=None, help="Only observations on or after YYYY-MM-DD")
+ @click.pass_context
+ def backfill_fred_cmd(ctx: click.Context, series_ids: tuple[str, ...], start: str | None) -> None:
+ """Load FRED graph CSV series (up to 5).
+
+ For example: intel history backfill fred WALCL RRPONTSYD DGS10 --start 2020-01-01
+ """
+ _run_backfill(ctx, "fred", series_ids, start)
+
+
+ @backfill_group.command(name="ecb")
+ @click.argument("keys", nargs=-1, required=True)
+ @click.option("--start", default=None, help="Only observations on or after YYYY-MM-DD")
+ @click.pass_context
+ def backfill_ecb_cmd(ctx: click.Context, keys: tuple[str, ...], start: str | None) -> None:
+ """Load ECB Data Portal series (up to 2 keys).
+
+ Each key is DATASET.KEY, for example BSI.M.U2.Y.V.M30.X.1.U2.2300.Z01.E
+ (euro area M3); a key with + alternatives loads every series it matches.
+ """
+ _run_backfill(ctx, "ecb", keys, start)
if __name__ == "__main__":
diff --cc src/world_intel_mcp/tools/__init__.py
index 82119e2,fb02ea6..9b2f84d
--- a/src/world_intel_mcp/tools/__init__.py
+++ b/src/world_intel_mcp/tools/__init__.py
@@@ -30,9 -29,9 +30,10 @@@ from . import ( # noqa: E40
geospatial,
hazards,
history,
+ history_backfill,
infrastructure,
intelligence,
+ lagged_connections,
liquidity,
markets,
news_market,
@@@ -63,8 -60,7 +64,9 @@@ _MODULES =
vector,
aoi,
history,
+ lagged_connections,
+ day_timeline,
+ history_backfill,
system,
]