160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.
Python ai-toolsai-watchanthropicclaudecybersecuritydashboardgeofencinggeopoliticalgeospatialintelligencemarketsmcpmigrationmilitarymodel-context-protocolnuclearosintreal-timespace-weatherthreat-intelligence
Add lagged connections between stored series (Phase 29 wave 2)
Phoenix Agent (mac-studio)4bc1d0efa7a2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c12802f..76b3fe5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -229,6 +229,56 @@ source here, and no proxy is substituted for any of these measures.history capture is an error.- Tools 139 -> 143.+### Added: connections with timing offsets (Phase 29)+- `intel_lagged_connections` (CLI: `intel history connections`): Pearson+ correlation between 2 to 8 series in the history store (keys as+ `intel_series_history` lists them), aligned on a UTC day, week (Monday to+ Sunday) or month grid and tested at every lag from `-max_lag` to+ `+max_lag` (default 5, at most 30, reduced so one call runs at most 1,000+ tests). Lag +k compares A at period t with B at t+k, so A leads B by k.+- Default transform: period-over-period changes, never taken across a gap;+ `levels` on request. Responses say which ran and why changes are the+ default (levels of trending series correlate spuriously).+- Per pair: the lag with the smallest p, with r, n (overlapping periods+ after the transform and shift), a two-sided p from Student's t (computed+ from the regularized incomplete beta function, standard library only) and+ the Benjamini-Hochberg q across every test in the call, with the+ Benjamini-Yekutieli `q_by` alongside. Significant at q <= 0.05 by default+ (allowed 0.001 to 0.25); the other significant lags are listed too.+- A pair and lag with fewer than 30 overlapping periods (callers may go down+ to 20), or a constant side, is listed as not tested with its n, never+ dropped. 30 points give about 80% power for a true |r| of 0.5 at alpha+ 0.05 (module docstring).+- One vintage, the latest collected value for each `observed_at`. Rows+ without `observed_at` (Yahoo quotes) are not used, with the reason: rows+ are stored only on change, so a missing day cannot be told from an+ unchanged one. A period takes its last observation; nothing is averaged.+- Serial correlation: each series' lag-1 autocorrelation after the+ transform, and per strongest result an AR(1) effective sample size and+ the p recomputed with it, as a diagnostic outside q. Nothing is+ prewhitened.+- `cannot_prove` in every response: correlation is not causation, a lead is+ not a causal direction, serial correlation inflates significance, a+ shared driver produces correlation, the store holds only what the+ collector recorded, BH under dependence is approximate and covers one+ call, and a small q is not a large effect.+- An unknown grid or transform is an error naming the allowed values;+ numeric inputs are clamped and each clamp is reported.+- Measured 2026-10-17 over a live MCP stdio session and the CLI, on four+ real FRED series (WALCL, WDTGAL, WTREGEN, RRPONTSYD) loaded into an+ isolated store as one capture; illustrative, not findings. Week grid,+ changes: 26 of 66 tests significant; the Wednesday TGA level against its+ week average gives r 0.61 at lag 0 and 0.59 at lag +1, the averaging+ window rather than a cause. Levels: Fed total assets against either TGA+ series gives r 0.74 at lag 0 (n 1,244) with an AR(1) effective sample+ size of 5.3 and 3.4. Day grid: the weekly series have no adjacent-day+ changes, so nothing is tested and the response says why.+- With the default 365-day observation retention+ (`WORLD_INTEL_HISTORY_DAYS`), a store filled only by the collector spans+ about a year, so the month grid cannot reach the 20-point floor without a+ longer retention or backfilled history.+- Tools 152 -> 153.+### Added: news signals (ROADMAP Phase 29)- `intel_hostility_timeline`: per UTC day, GDELT DOC 2.0 coverage of acountry or a pair of countries (`timelinevolraw`): matching articles,diff --git a/README.md b/README.mdindex 0638fe3..1b291d5 100644--- a/README.md+++ b/README.md@@ -6,7 +6,7 @@[](https://python.org)[](LICENSE)-Real-time global intelligence across **30+ domains** with **152 MCP tools**, a live ops-center dashboard, a CLI, and a **Qdrant vector store** for enterprise-grade semantic search across accumulated intelligence. All data comes from free, public APIs: no paid subscriptions required.+Real-time global intelligence across **30+ domains** with **153 MCP tools**, a live ops-center dashboard, a CLI, and a **Qdrant vector store** for enterprise-grade semantic search across accumulated intelligence. All data comes from free, public APIs: no paid subscriptions required.Built for AI agents that need world awareness: market conditions, geopolitical risk, military posture, aircraft-cohort early warning, refugee and migration flows, supply chain disruptions, cyber threats, and more — all queryable via the Model Context Protocol. The vector store enables natural language queries like *"military activity near Taiwan"* or *"cyber threats targeting healthcare"* across all historical data.@@ -67,8 +67,9 @@ Built for AI agents that need world awareness: market conditions, geopolitical r| **Tropical Cyclones** | 2 | NHC (Atlantic, E/C Pacific) + JTWC (NW Pacific, N Indian, S Hemisphere), with opt-in forecast tracks: positions, UTC valid times and intensities, plus NHC's cone of uncertainty || **Situation Brief** | 1 | Cited situational awareness brief over MCP: bounded server-side overview synthesized via local Ollama, with a mechanically-cited fallback || **Collection History** | 4 | Collector run log (per-source health, overdue and failing sources), saved captures with change detection and replay, dated observation series, revisions |+| **Connections Between Series** | 1 | Correlation between stored series at leads and lags, with a minimum sample, false-discovery control and what it cannot prove |-**Total: 152 tools** across 30+ intelligence domains.+**Total: 153 tools** across 30+ intelligence domains.---@@ -142,6 +143,8 @@ intel top-stories -m 4 # stories carried by 4+ outletsintel status # cache + circuit breaker healthintel guide # the research guide (also served as intel://guide)intel history health # per-source collection health from the collector's run log+intel history connections central_bank_rates:federal_funds_rate central_bank_rates:boe_bank_rate_sonia+ # correlation at leads and lags between stored series (day grid by default)```---@@ -606,6 +609,13 @@ History accrues only while the collector (`intel-collector`) runs; nothing befor| `intel_series_history` | One series' observations with all three dates, or the list of available series || `intel_revisions` | Values for the same `observed_at` that a later capture changed (USGS magnitude revisions, NOAA Kp updates) |+### Connections Between Series (1)+Reads the history store only, so the sample is what the collector saved, not each series' full history. A connection is a correlation in that sample: it does not show causation, and a lead does not show causal direction.++| Tool | Description |+|------|-------------|+| `intel_lagged_connections` | Correlation between 2 to 8 stored series (keys as `intel_series_history` lists them) on a UTC day, week or month grid, tested at every lag from `-max_lag` to `+max_lag` (default 5, at most 30, at most 1,000 tests a call). Default transform is period-over-period changes, because levels of trending series correlate spuriously; `levels` on request. Per pair: the lag with the smallest p, with Pearson r, overlap n, two-sided p from Student's t and the Benjamini-Hochberg q across every test in the call (significant at q <= 0.05 by default). Lag +k compares A at period t with B at t+k, so A leads B by k. Pair/lag overlaps under 30 points are listed as not tested with their n; revised points use the latest vintage; rows without `observed_at` (Yahoo quotes) are not used. Each strongest result also carries an AR(1) effective sample size as a serial-correlation diagnostic |+---## Watching your own area (geofences/AOIs)diff --git a/ROADMAP.md b/ROADMAP.mdindex 259a553..29f1c5b 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -580,7 +580,7 @@ dashboard.| 1 | Liquidity and flows | Fed balance sheet, Treasury General Account, reverse repo and net liquidity (FRED keyless CSV), money supply (US M2, euro area M1/M2/M3, never summed), ECB, ICI fund flows and ETF net issuance, FINRA margin statistics, BLS series; amounts kept separate from flows | :yellow_circle: Built, unreleased: 5 tools (`intel_liquidity_board`, `intel_net_liquidity`, `intel_money_supply`, `intel_fed_liquidity_series`, `intel_ecb_series`), 5 CLI commands, 2 collector sources. FRED graph CSV and ECB Data Portal live-verified 2026-10-17. The TGA term is WDTGAL (Wednesday level, USD millions): WTREGEN is a week average; the net liquidity `method` block states each term's units, conversion and how RRPONTSYD (daily, USD billions) is aligned to the Wednesday. Blocked, measured 2026-10-17, and listed in the board's `blocked` field; no proxy is substituted, so fund flows and ETF net issuance have no source: ICI (money market fund assets at /research/stats/mmf, ETF net issuance at /research/stats/etf) answers 403 Akamai "Access Denied" on every path, robots.txt included; FINRA margin statistics XLSX answers 200, but the finra.org Terms of Use (Restrictions d and e) forbid building a database from the site and data mining, scraping or harvesting tools, robots included, and the page says data feeds are not available; BLS public API v1 answers 200 without a key (CPI-U, payrolls), but api.bls.gov robots.txt is `Disallow: /` and the terms page on www.bls.gov answers 403 to this server's User-Agent, so it is not used || 1 | Kalshi + cross-venue markets | Kalshi public market data next to Polymarket; same-event comparison with an explicit match confidence; settled markets scored against pre-settlement prices (Brier) | :red_circle: || 1 | News vs market reaction | Companies and countries named in story clusters mapped to tickers / country indices; same-day move against a benchmark | :white_check_mark: Built 2026-10-17, unreleased: `intel_news_market_reaction` and `intel news-reaction`. Company rows: stock, benchmark (SPY, or the exchange country's index) and excess return on the story's session; country rows: index return with a trailing z-score. Session rule in the exchange's timezone (after close = next session; weekends and holidays read from the listing's own bars). Open: `^TASI` and `^TA125` (Saudi and Israel in the country-index table) return 404 on Yahoo, so those benchmarks are data gaps; story times inherit `sources/news.py`'s RSS timestamp offset until that is fixed |-| 2 | Connections with timing offsets | Correlation between retained series over a shared sample, tested at leads and lags, with minimum sample size and false-discovery control; states that it does not prove causation | :red_circle: |+| 2 | Connections with timing offsets | Correlation between retained series over a shared sample, tested at leads and lags, with minimum sample size and false-discovery control; states that it does not prove causation | :white_check_mark: Built 2026-10-17, unreleased: `intel_lagged_connections` and `intel history connections`. Pearson r on a UTC day, week or month grid at every lag from -max_lag to +max_lag (default 5, cap 30, at most 1,000 tests a call), period-over-period changes by default, p from Student's t (regularized incomplete beta, standard library), Benjamini-Hochberg q across the call (default 0.05), a 30-point minimum with every untested pair and lag listed with its n, latest vintage, rows without `observed_at` refused with the reason, an AR(1) effective sample size as a diagnostic, and a `cannot_prove` list. p-values match scipy to 3.3e-13 relative error. Verified 2026-10-17 over a live MCP stdio session and the CLI on four real FRED series loaded into an isolated store. Not done: rank correlation, prewhitening || 2 | Day timeline | What every retained source recorded for a given day, by observation date or collection date | :red_circle: || 3 | Dashboard views | Liquidity board, situations, top stories, collection health panels | :red_circle: |diff --git a/src/world_intel_mcp/analysis/lagged_connections.py b/src/world_intel_mcp/analysis/lagged_connections.pynew file mode 100644index 0000000..94c2a35--- /dev/null+++ b/src/world_intel_mcp/analysis/lagged_connections.py@@ -0,0 +1,845 @@+"""Connections with timing offsets between series in the history store.++For 2 to ``MAX_SERIES`` series the collector saved (``history.py``; keys as+``intel_series_history`` lists them), the series are aligned on a UTC+calendar grid, transformed, and their Pearson correlation is tested at every+lag from ``-max_lag`` to ``+max_lag``, with a Benjamini-Hochberg+false-discovery adjustment across every test run in the call. Standard+library only.++Alignment:+- Grid periods are UTC calendar days, ISO weeks (Monday to Sunday, labelled+ by the Monday's date) or calendar months. ``observed_at`` decides the+ period: a ``YYYY-MM-DD`` date or an instant (stored in UTC) falls in the+ day, week and month containing it; a ``YYYY-MM`` month fits only the month+ grid and a ``YYYY`` year fits none. Rows coarser than the grid are not+ used, and are counted.+- Vintage: when a later capture revised the value for one ``observed_at``,+ the latest collected value is used (ties broken by insertion order).+ Earlier vintages are ignored; the number of revised observation dates is+ reported per series.+- Sampling: a period takes the value of its last observation by+ ``observed_at``, so a daily series on the week grid is its last day of the+ week. Values inside a period are not averaged: averaging a random walk+ makes its period-to-period changes serially correlated.+- Rows without ``observed_at`` (Yahoo quotes) are not used. Their only date+ is ``collected_at``, when this server saw a changed value, and rows are+ stored only when the value changes, so a day with no row cannot be told+ from a day with an unchanged value. A series with no dated rows is+ excluded with that reason.+- ``start`` / ``end`` select whole grid periods (the ones containing them and+ every period between). Without them the window is the series' combined+ span, cut to the most recent ``MAX_PERIODS`` periods.++Transform:+- ``changes`` (default): a period's value minus the immediately preceding+ period's, only when both exist; a change is never taken across a gap.+ Levels of trending series correlate spuriously (two independent random+ walks routinely reach |r| above 0.5), which is why changes are the default.+- ``levels``: the sampled values.++Tests:+- Pairs are (A, B) in the order the series were given. Lag +k compares A at+ period t with B at period t+k: a connection at +k means A's moves come k+ periods before B's (A leads B by k); lag -k means B leads A by k.+- n counts the periods where both transformed values exist after the shift+ (pairwise complete). A (pair, lag) with n below the minimum, or with no+ variation on either side, is not tested and is listed with its n.+- p is two-sided, from Student's t with n - 2 degrees of freedom. With+ t^2 = r^2 (n - 2) / (1 - r^2), df / (df + t^2) = 1 - r^2, so+ P(|T| >= |t|) = I_{1 - r^2}((n - 2) / 2, 1 / 2), the regularized incomplete+ beta function, evaluated by its continued fraction (DLMF 8.17.22) with the+ modified Lentz method. No permutation test is run.+- q is the Benjamini-Hochberg adjusted p over every test run in the call;+ ``q_by`` is the Benjamini-Yekutieli version, valid under any dependence+ between tests. A test is significant when q <= the threshold (default+ ``DEFAULT_Q`` = 0.05).++Minimum sample, ``DEFAULT_MIN_OVERLAP`` = 30 overlapping points (callers may+raise it, or lower it to ``MIN_OVERLAP_FLOOR`` = 20): at n = 30 a two-sided+test at alpha 0.05 detects a true |rho| of 0.5 with about 80% power before+any multiple-testing correction (Fisher z approximation: n = ((1.960 ++0.842) / atanh 0.5)^2 + 3 = 29.0; simulated with bivariate normal draws:+0.826 at n = 30, 0.643 at n = 20). Below that, a moderate real correlation is+more often missed than found, and one that passes rests on few points. The+floor is 20 because below it a single outlying pair can dominate r and the t+reference distribution leans on near-normal values.++Serial correlation: p and q treat periods as independent. Each series'+lag-1 autocorrelation after the transform is reported, and each strongest+result carries ``n_effective_ar1`` and ``p_ar1``: n scaled by+(1 - phi_a phi_b) / (1 + phi_a phi_b), the large-sample variance inflation+of r for two AR(1) series (Bartlett), capped at n. That is a diagnostic only:+q uses p, and nothing is prewhitened.++Gaps / not covered:+- Pearson only: no rank correlation, so a few extreme changes can drive r.+- No prewhitening and no correction for autocorrelation beyond lag 1 or for+ volatility clustering; ``p_ar1`` is an AR(1) approximation outside q.+- Benjamini-Hochberg assumes independent or positively dependent tests;+ neighbouring lags of one pair share most of their data, so q is+ approximate (``q_by`` is the any-dependence bound).+- The adjustment covers one call. Other calls with other series, grids or+ windows are further tests it cannot see.+- Business-day series lose their Monday changes on the day grid (Friday to+ Monday is not adjacent). Within one week or month, two series' last+ observations can fall on different days (Wednesday levels against a+ Friday close), so "same period" is not "same day".+- One vintage only (latest seen); there is no first-release view.+- Reference p-values in the tests come from scipy in a throwaway venv, not+ from an independent derivation at run time.+"""++from __future__ import annotations++import math+import re+from datetime import date+from typing import Any, Iterable, Sequence++from ..history import HISTORY_ACCRUAL_NOTE, HistoryStore++GRIDS = ("day", "week", "month")+TRANSFORMS = ("changes", "levels")+DEFAULT_GRID = "day"+DEFAULT_TRANSFORM = "changes"+DEFAULT_MAX_LAG = 5+MAX_LAG_CAP = 30+DEFAULT_MIN_OVERLAP = 30+MIN_OVERLAP_FLOOR = 20+DEFAULT_Q = 0.05+Q_FLOOR = 0.001+Q_CAP = 0.25+MAX_SERIES = 8+MAX_TESTS = 1000+MAX_PERIODS = 3000+# Fewer adjacent pairs than this and a lag-1 autocorrelation is not reported.+_AUTOCORR_MIN_PAIRS = 10++_UNIT = {"day": "day", "week": "week", "month": "month"}++GRID_RULES = {+ "day": "UTC calendar days",+ "week": "ISO weeks in UTC, Monday to Sunday, labelled by the Monday's date",+ "month": "UTC calendar months, labelled YYYY-MM",+}+SAMPLING_RULE = (+ "Each period takes the value of its last observation by observed_at (a daily "+ "series on the week grid is its last day of the week). Values inside a period "+ "are not averaged."+)+VINTAGE_RULE = (+ "Latest vintage: where later captures revised the value for an observed_at, the "+ "latest collected value is used and earlier values are ignored."+)+UNDATED_RULE = (+ "Rows without observed_at (for example Yahoo quotes, saved with collected_at "+ "only) are not used: collected_at is when this server saw a changed value, not "+ "when it was observed, and rows are stored only on change, so a day with no row "+ "cannot be told from a day with an unchanged value."+)+TRANSFORM_RULES = {+ "changes": (+ "Period-over-period changes: a period's value minus the immediately "+ "preceding period's, only when both exist; never across a gap."+ ),+ "levels": "Levels: the sampled values, untransformed.",+}+WHY_CHANGES = (+ "Changes are the default because levels of trending series correlate "+ "spuriously: two independent random walks routinely reach |r| above 0.5 in "+ "levels and near 0 in changes."+)+LAG_CONVENTION = (+ "Pairs are (A, B) in the order the series were given. Lag +k: A's value at "+ "period t is compared with B's value at period t+k, so a connection at lag +k "+ "means A's moves come k periods before B's (A leads B by k). Lag -k: B leads A "+ "by k. Lag 0: the same period."+)+P_RULE = (+ "Two-sided p from Student's t with n - 2 degrees of freedom, computed exactly "+ "as I_(1 - r^2)((n - 2)/2, 1/2), the regularized incomplete beta function."+)+MIN_OVERLAP_REASON = (+ "30 overlapping points give about 80% power to detect a true |r| of 0.5 at "+ "alpha 0.05 (two-sided, before multiple-testing correction); fewer miss "+ "moderate real correlations more often than they find them. The floor is 20: "+ "below it one outlying pair can dominate r."+)+AR1_RULE = (+ "p and q treat periods as independent. Done: each series' lag-1 "+ "autocorrelation after the transform is reported, and each strongest result "+ "carries n_effective_ar1 (n scaled by (1 - phi_a phi_b) / (1 + phi_a phi_b), "+ "capped at n) and p_ar1 (p recomputed with it). Not done: prewhitening, "+ "autocorrelation beyond lag 1, volatility clustering; p_ar1 does not enter q."+)+NOT_TESTED_REASONS = {+ "below_min_overlap": "fewer overlapping periods than min_overlap",+ "no_variation": "one side is constant over the overlap, so r is undefined",+}+CANNOT_PROVE = [+ "Correlation is not causation: a strong r at any lag shows the two series moved "+ "together in this sample, not that one moved the other.",+ "A lead is not evidence of causal direction: when A's moves come k periods "+ "before B's, B can still drive A (through expectations or release timing), or "+ "a third factor can reach A first.",+ "Serial correlation inflates significance even after differencing: p and q "+ "treat periods as independent. What was done: lag-1 autocorrelations are "+ "reported and p_ar1 recomputes the strongest results with an AR(1) effective "+ "sample size. What was not: prewhitening, autocorrelation beyond lag 1, "+ "volatility clustering. p_ar1 does not enter q.",+ "A shared common driver produces correlation: two series that both respond to "+ "a third (the business cycle, a policy date, a holiday calendar) correlate "+ "with no link between them.",+ "The store holds only what the collector recorded. " + HISTORY_ACCRUAL_NOTE+ + " The sample is the collection period, not the series' full history, and "+ "periods the collector missed are absent, not zero.",+ "Benjamini-Hochberg controls the false discovery rate for independent or "+ "positively dependent tests; neighbouring lags of one pair share most of "+ "their data, so q is approximate (q_by holds under any dependence). The "+ "adjustment covers this call only, not other calls with other series, grids "+ "or windows.",+]+++# ---------------------------------------------------------------------------+# Statistics (standard library only)+# ---------------------------------------------------------------------------++_CF_EPS = 1e-15+_CF_TINY = 1e-300+_CF_MAX_ITER = 10_000+++def _beta_continued_fraction(a: float, b: float, x: float) -> float:+ """Denominator f of I_x(a, b) = x^a (1-x)^b / (a B(a, b)) / f, where+ f = 1 + d1 / (1 + d2 / (1 + ...)) with DLMF 8.17.22's coefficients+ d_(2m+1) = -(a+m)(a+b+m)x / ((a+2m)(a+2m+1)) and+ d_(2m) = m(b-m)x / ((a+2m-1)(a+2m)), by the modified Lentz method."""+ f = c = 1.0+ d = 0.0+ for j in range(1, _CF_MAX_ITER):+ m = j // 2+ if j % 2:+ coef = -(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1))+ else:+ coef = m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m))+ d = 1.0 + coef * d+ if abs(d) < _CF_TINY:+ d = _CF_TINY+ d = 1.0 / d+ c = 1.0 + coef / c+ if abs(c) < _CF_TINY:+ c = _CF_TINY+ delta = c * d+ f *= delta+ if abs(delta - 1.0) < _CF_EPS:+ return f+ raise ArithmeticError(f"incomplete beta did not converge (a={a}, b={b}, x={x})")+++def _ibeta(a: float, b: float, x: float, y: float) -> float:+ """I_x(a, b), with y = 1 - x supplied by the caller so that whichever of+ the two is small keeps its precision."""+ if x <= 0.0:+ return 0.0+ if y <= 0.0:+ return 1.0+ if x > (a + 1.0) / (a + b + 2.0):+ # The fraction converges fast only below this point; use+ # I_x(a, b) = 1 - I_(1-x)(b, a).+ return 1.0 - _ibeta(b, a, y, x)+ log_front = (+ a * math.log(x) + b * math.log(y) - math.log(a)+ - (math.lgamma(a) + math.lgamma(b) - math.lgamma(a + b))+ )+ return math.exp(log_front) / _beta_continued_fraction(a, b, x)+++def regularized_incomplete_beta(a: float, b: float, x: float) -> float:+ """I_x(a, b) for a, b > 0 and 0 <= x <= 1."""+ if not (a > 0 and b > 0) or not 0.0 <= x <= 1.0:+ raise ValueError(f"need a, b > 0 and 0 <= x <= 1, got a={a}, b={b}, x={x}")+ return _ibeta(a, b, x, 1.0 - x)+++def correlation_p_value(r: float, n: float) -> float:+ """Two-sided p of H0 rho = 0 from Student's t with n - 2 degrees of+ freedom. ``n`` may be fractional (an effective sample size)."""+ df = n - 2.0+ if not df > 0:+ raise ValueError(f"need n > 2, got {n}")+ r = abs(r)+ if r >= 1.0:+ return 0.0+ # t^2 = r^2 df / (1 - r^2), so df / (df + t^2) = 1 - r^2 exactly.+ return min(1.0, _ibeta(df / 2.0, 0.5, (1.0 - r) * (1.0 + r), r * r))+++# Quotes a p from correlation_p_value, so it joins the list once that exists.+CANNOT_PROVE.append(+ "A small q is not a large effect: q says r is unlikely to be zero in this "+ "sample, not that it is big. With 1,000 overlapping periods an |r| of 0.1 "+ f"(1% of the variance shared) already gives p = {correlation_p_value(0.1, 1000):.2g}; "+ "read r and n alongside q."+)+++def pearson_r(xs: Sequence[float], ys: Sequence[float]) -> float | None:+ """Pearson correlation, or None when either side does not vary."""+ n = len(xs)+ if n != len(ys):+ raise ValueError("xs and ys differ in length")+ if n < 2 or min(xs) == max(xs) or min(ys) == max(ys):+ return None+ mx = math.fsum(xs) / n+ my = math.fsum(ys) / n+ dx = [x - mx for x in xs]+ dy = [y - my for y in ys]+ sxx = math.fsum(d * d for d in dx)+ syy = math.fsum(d * d for d in dy)+ if sxx <= 0.0 or syy <= 0.0:+ return None+ r = math.fsum(u * v for u, v in zip(dx, dy)) / (math.sqrt(sxx) * math.sqrt(syy))+ return max(-1.0, min(1.0, r))+++def benjamini_hochberg(pvalues: Sequence[float]) -> list[float]:+ """BH adjusted p-values (q), in the input order: q at rank i is the+ minimum over ranks j >= i of m p_(j) / j, capped at 1."""+ m = len(pvalues)+ order = sorted(range(m), key=lambda i: pvalues[i])+ adjusted = [1.0] * m+ running = 1.0+ for rank in range(m, 0, -1):+ i = order[rank - 1]+ running = min(running, pvalues[i] * m / rank)+ adjusted[i] = running+ return adjusted+++def benjamini_yekutieli(pvalues: Sequence[float]) -> list[float]:+ """BY adjusted p-values: BH scaled by the harmonic sum 1 + 1/2 + ... + 1/m,+ which keeps the false discovery rate under any dependence."""+ harmonic = math.fsum(1.0 / i for i in range(1, len(pvalues) + 1))+ return [min(1.0, q * harmonic) for q in benjamini_hochberg(pvalues)]+++# ---------------------------------------------------------------------------+# Grid+# ---------------------------------------------------------------------------++_DAY = re.compile(r"(\d{4})-(\d{2})-(\d{2})(?:T\d{2}:\d{2}:\d{2}Z)?")+_MONTH = re.compile(r"(\d{4})-(\d{2})")+_YEAR = re.compile(r"\d{4}")+_ISO_DATE = re.compile(r"\d{4}-\d{2}-\d{2}")+++def period_of(observed_at: str, grid: str) -> tuple[int | None, str | None]:+ """Integer period index of a stored ``observed_at`` on ``grid`` (adjacent+ periods differ by 1), or ``(None, "coarser" | "unreadable")``."""+ text = str(observed_at)+ m = _DAY.fullmatch(text)+ if m:+ try:+ day = date(int(m[1]), int(m[2]), int(m[3]))+ except ValueError:+ return None, "unreadable"+ if grid == "day":+ return day.toordinal(), None+ if grid == "week":+ # Ordinal 1 (0001-01-01) is a Monday.+ return (day.toordinal() - 1) // 7, None+ return day.year * 12 + day.month - 1, None+ m = _MONTH.fullmatch(text)+ if m:+ if not 1 <= int(m[2]) <= 12:+ return None, "unreadable"+ if grid == "month":+ return int(m[1]) * 12 + int(m[2]) - 1, None+ return None, "coarser"+ if _YEAR.fullmatch(text):+ return None, "coarser"+ return None, "unreadable"+++def period_label(index: int, grid: str) -> str:+ if grid == "day":+ return date.fromordinal(index).isoformat()+ if grid == "week":+ return date.fromordinal(7 * index + 1).isoformat()+ return f"{index // 12:04d}-{index % 12 + 1:02d}"+++def apply_transform(levels: dict[int, float], transform: str) -> dict[int, float]:+ if transform == "levels":+ return dict(levels)+ return {p: v - levels[p - 1] for p, v in levels.items() if p - 1 in levels}+++def lag1_autocorrelation(values: dict[int, float]) -> tuple[float | None, int]:+ """Pearson r between each value and the next period's, with the number of+ adjacent pairs; None below ``_AUTOCORR_MIN_PAIRS`` pairs or without+ variation."""+ xs = [values[p] for p in sorted(values) if p + 1 in values]+ ys = [values[p + 1] for p in sorted(values) if p + 1 in values]+ if len(xs) < _AUTOCORR_MIN_PAIRS:+ return None, len(xs)+ return pearson_r(xs, ys), len(xs)+++def overlap_at_lag(+ a: dict[int, float], b: dict[int, float], lag: int+) -> tuple[list[float], list[float]]:+ """A at t paired with B at t + lag, over every t where both exist."""+ xs: list[float] = []+ ys: list[float] = []+ for p in sorted(a):+ other = b.get(p + lag)+ if other is not None:+ xs.append(a[p])+ ys.append(other)+ return xs, ys+++# ---------------------------------------------------------------------------+# Inputs+# ---------------------------------------------------------------------------+++class _InputError(ValueError):+ pass+++def _integer(name: str, value: Any, default: int) -> int:+ if value is None:+ return default+ if isinstance(value, bool):+ raise _InputError(f"{name} must be an integer, got {value!r}")+ if isinstance(value, int):+ return value+ if isinstance(value, float) and value.is_integer():+ return int(value)+ raise _InputError(f"{name} must be an integer, got {value!r}")+++def _clamp(name: str, value: Any, lo: Any, hi: Any, clamps: list[dict]) -> Any:+ used = max(lo, min(hi, value))+ if used != value:+ clamps.append({"input": name, "requested": value, "used": used, "allowed": [lo, hi]})+ return used+++def _iso_date(name: str, value: Any) -> date | None:+ if value is None or value == "":+ return None+ if not isinstance(value, str) or not _ISO_DATE.fullmatch(value.strip()):+ raise _InputError(f"{name} must be a date YYYY-MM-DD, got {value!r}")+ try:+ return date.fromisoformat(value.strip())+ except ValueError as exc:+ raise _InputError(f"{name} must be a date YYYY-MM-DD, got {value!r}") from exc+++def _series_keys(value: Any) -> list[str]:+ if isinstance(value, str) or not isinstance(value, (list, tuple)):+ raise _InputError(+ "series_keys must be a list of 2 to "+ f"{MAX_SERIES} series keys (as intel_series_history lists them)"+ )+ keys = []+ for item in value:+ if not isinstance(item, str) or not item.strip():+ raise _InputError(f"series_keys entries must be non-empty strings, got {item!r}")+ keys.append(item.strip())+ if len(keys) < 2:+ raise _InputError(f"series_keys needs at least 2 series, got {len(keys)}")+ if len(keys) > MAX_SERIES:+ raise _InputError(f"series_keys takes at most {MAX_SERIES} series per call, got {len(keys)}")+ duplicates = sorted({k for k in keys if keys.count(k) > 1})+ if duplicates:+ raise _InputError(f"series_keys repeats {duplicates}")+ return keys+++# ---------------------------------------------------------------------------+# Store reads and per-series preparation+# ---------------------------------------------------------------------------+++def _read_rows(store: HistoryStore, series_key: str) -> list[tuple]:+ # HistoryStore has no bulk read for one series' every vintage, so this+ # reads its observations table directly (parameterized, read-only).+ return store._conn.execute(+ "SELECT observed_at, value, source, unit, label FROM history_observations "+ "WHERE series_key = ? ORDER BY observed_at, collected_at, id",+ (series_key,),+ ).fetchall()+++def _sample_on_grid(key: str, rows: list[tuple], grid: str) -> dict[str, Any]:+ latest: dict[str, float] = {}+ distinct: dict[str, set[float]] = {}+ undated = 0+ for observed_at, value, _source, _unit, _label in rows:+ if observed_at is None:+ undated += 1+ continue+ latest[observed_at] = float(value) # rows run oldest vintage first+ distinct.setdefault(observed_at, set()).add(float(value))+ coarser = unreadable = 0+ coarser_kinds: set[str] = set()+ levels: dict[int, float] = {}+ for observed_at in sorted(latest):+ period, why = period_of(observed_at, grid)+ if period is None:+ if why == "coarser":+ coarser += 1+ coarser_kinds.add("month" if len(observed_at) == 7 else "year")+ else:+ unreadable += 1+ continue+ levels[period] = latest[observed_at] # sorted, so the last one wins+ last_row = rows[-1] if rows else (None, None, None, None, None)+ return {+ "series_key": key,+ "source": last_row[2],+ "unit": next((r[3] for r in reversed(rows) if r[3]), None),+ "label": next((r[4] for r in reversed(rows) if r[4]), None),+ "rows": len(rows),+ "rows_without_observed_at": undated,+ "rows_coarser_than_grid": coarser,+ "rows_unreadable_observed_at": unreadable,+ "revised_observed_at": sum(1 for values in distinct.values() if len(values) > 1),+ "dated_observations": len(latest),+ "_levels": levels,+ "_coarser_kinds": coarser_kinds,+ }+++def _exclusion_reason(info: dict, grid: str) -> str | None:+ if info["dated_observations"] == 0:+ return (+ f"no row has observed_at ({info['rows_without_observed_at']} rows carry "+ "collected_at only). " + UNDATED_RULE+ )+ if not info["_levels"]:+ if info["rows_coarser_than_grid"]:+ kinds = " and ".join(sorted(info["_coarser_kinds"]))+ hint = " Use grid=month." if info["_coarser_kinds"] == {"month"} else ""+ return (+ f"every observation describes a {kinds}, coarser than the {grid} grid."+ + hint+ )+ return "no observed_at could be read as a date"+ return None+++def _median_gap(periods: Iterable[int]) -> float | None:+ ordered = sorted(periods)+ gaps = sorted(b - a for a, b in zip(ordered, ordered[1:]))+ if not gaps:+ return None+ mid = len(gaps) // 2+ return float(gaps[mid]) if len(gaps) % 2 else (gaps[mid - 1] + gaps[mid]) / 2.0+++def _sig(value: float) -> float:+ return float(f"{value:.3g}")+++# ---------------------------------------------------------------------------+# The analysis+# ---------------------------------------------------------------------------+++def lagged_connections(+ store: HistoryStore,+ series_keys: Any,+ grid: Any = None,+ transform: Any = None,+ max_lag: Any = None,+ min_overlap: Any = None,+ q: Any = None,+ start: Any = None,+ end: Any = None,+) -> dict:+ """Correlation between stored series at leads and lags (module docstring)."""+ clamps: list[dict] = []+ try:+ keys = _series_keys(series_keys)+ grid = DEFAULT_GRID if grid is None else grid+ if grid not in GRIDS:+ raise _InputError(f"unknown grid {grid!r}; allowed: {', '.join(GRIDS)}")+ transform = DEFAULT_TRANSFORM if transform is None else transform+ if transform not in TRANSFORMS:+ raise _InputError(+ f"unknown transform {transform!r}; allowed: {', '.join(TRANSFORMS)}"+ )+ requested_lag = _clamp(+ "max_lag", _integer("max_lag", max_lag, DEFAULT_MAX_LAG), 0, MAX_LAG_CAP, clamps+ )+ min_n = _clamp(+ "min_overlap",+ _integer("min_overlap", min_overlap, DEFAULT_MIN_OVERLAP),+ MIN_OVERLAP_FLOOR, MAX_PERIODS, clamps,+ )+ if q is None:+ q = DEFAULT_Q+ if isinstance(q, bool) or not isinstance(q, (int, float)) or not math.isfinite(q):+ raise _InputError(f"q must be a number, got {q!r}")+ q_threshold = _clamp("q", float(q), Q_FLOOR, Q_CAP, clamps)+ start_d = _iso_date("start", start)+ end_d = _iso_date("end", end)+ if start_d and end_d and start_d > end_d:+ raise _InputError(f"start {start_d} is after end {end_d}")+ except _InputError as exc:+ return {"error": str(exc)}++ rows = {key: _read_rows(store, key) for key in keys}+ unknown = [key for key in keys if not rows[key]]+ if unknown:+ return {+ "error": f"no observations stored for {unknown}; call intel_series_history "+ "without series_key to list available series",+ "unknown_series": unknown,+ }++ infos = [_sample_on_grid(key, rows[key], grid) for key in keys]+ excluded = []+ usable = []+ for info in infos:+ reason = _exclusion_reason(info, grid)+ if reason:+ excluded.append({"series_key": info["series_key"], "reason": reason})+ else:+ usable.append(info)++ # One shared window of whole grid periods.+ window: dict[str, Any] = {"start": None, "end": None, "limited_to_recent": False}+ lo = period_of(start_d.isoformat(), grid)[0] if start_d else None+ hi = period_of(end_d.isoformat(), grid)[0] if end_d else None+ if usable:+ first = min(min(i["_levels"]) for i in usable)+ last = max(max(i["_levels"]) for i in usable)+ lo = first if lo is None else lo+ hi = last if hi is None else hi+ if hi - lo + 1 > MAX_PERIODS:+ lo = hi - MAX_PERIODS + 1+ window["limited_to_recent"] = True+ window.update(start=period_label(lo, grid), end=period_label(hi, grid),+ periods=max(0, hi - lo + 1))++ still_usable = []+ for info in usable:+ inside = {p: v for p, v in info["_levels"].items() if lo <= p <= hi}+ info["periods_outside_window"] = len(info["_levels"]) - len(inside)+ info["_levels"] = inside+ if not inside:+ excluded.append({"series_key": info["series_key"],+ "reason": "no observation inside the window"})+ continue+ info["_values"] = apply_transform(inside, transform)+ still_usable.append(info)+ usable = still_usable++ pair_index = [(i, j) for i in range(len(usable)) for j in range(i + 1, len(usable))]+ lag = requested_lag+ if pair_index and len(pair_index) * (2 * lag + 1) > MAX_TESTS:+ lag = max(0, (MAX_TESTS // len(pair_index) - 1) // 2)+ clamps.append({+ "input": "max_lag", "requested": requested_lag, "used": lag,+ "reason": f"{len(pair_index)} pairs x {2 * lag + 1} lags keeps the call "+ f"within {MAX_TESTS} tests",+ })++ tests: list[dict] = []+ pairs: list[dict] = []+ for i, j in pair_index:+ a, b = usable[i], usable[j]+ pair = {"a": a["series_key"], "b": b["series_key"], "not_tested": [], "_tests": []}+ for k in range(-lag, lag + 1):+ xs, ys = overlap_at_lag(a["_values"], b["_values"], k)+ n = len(xs)+ if n < min_n:+ pair["not_tested"].append({"lag": k, "n": n, "reason": "below_min_overlap"})+ continue+ r = pearson_r(xs, ys)+ if r is None:+ pair["not_tested"].append({"lag": k, "n": n, "reason": "no_variation"})+ continue+ t_abs = math.inf if abs(r) >= 1.0 else abs(r) * math.sqrt((n - 2) / (1 - r * r))+ test = {"lag": k, "r": r, "n": n, "p": correlation_p_value(r, n), "_t": t_abs}+ tests.append(test)+ pair["_tests"].append(test)+ pairs.append(pair)++ pvalues = [t["p"] for t in tests]+ for test, q_bh, q_by in zip(tests, benjamini_hochberg(pvalues), benjamini_yekutieli(pvalues)):+ test["q"], test["q_by"] = q_bh, q_by+ test["significant"] = q_bh <= q_threshold++ for info in usable:+ info["_phi"], info["_phi_pairs"] = lag1_autocorrelation(info["_values"])++ unit = _UNIT[grid]+ gaps: list[str] = []++ def _entry(test: dict, a: dict, b: dict, full: bool) -> dict:+ k = test["lag"]+ plural = "" if abs(k) == 1 else "s"+ timing = (+ f"A leads B by {k} {unit}{plural}" if k > 0+ else f"B leads A by {-k} {unit}{plural}" if k < 0+ else f"same {unit}"+ )+ out = {+ "lag": k,+ "timing": timing,+ "r": round(test["r"], 4),+ "direction": "same" if test["r"] > 0 else "opposite" if test["r"] < 0 else "none",+ "n": test["n"],+ "p": _sig(test["p"]),+ "q": _sig(test["q"]),+ "q_by": _sig(test["q_by"]),+ "significant": test["significant"],+ }+ if full:+ phi_a, phi_b = a["_phi"], b["_phi"]+ if phi_a is None or phi_b is None:+ out["n_effective_ar1"] = out["p_ar1"] = None+ else:+ prod = phi_a * phi_b+ if prod >= 1.0:+ n_eff = 0.0+ elif prod <= -1.0:+ n_eff = float(test["n"])+ else:+ n_eff = min(float(test["n"]), test["n"] * (1.0 - prod) / (1.0 + prod))+ out["n_effective_ar1"] = round(n_eff, 1)+ out["p_ar1"] = _sig(correlation_p_value(test["r"], n_eff)) if n_eff >= 3 else None+ return out++ pair_out = []+ by_key = {info["series_key"]: info for info in usable}+ for pair in pairs:+ a, b = by_key[pair["a"]], by_key[pair["b"]]+ ranked = sorted(pair["_tests"], key=lambda t: (t["p"], -t["_t"]))+ strongest = [t for t in ranked if (t["p"], t["_t"]) == (ranked[0]["p"], ranked[0]["_t"])] if ranked else []+ pair_out.append({+ "a": pair["a"],+ "b": pair["b"],+ "lags_tested": len(pair["_tests"]),+ "lags_not_tested": len(pair["not_tested"]),+ "strongest": [_entry(t, a, b, full=True) for t in strongest],+ "significant": [_entry(t, a, b, full=False) for t in ranked if t["significant"]],+ "not_tested": pair["not_tested"],+ })+ if not ranked:+ biggest = max((nt["n"] for nt in pair["not_tested"]), default=0)+ gaps.append(+ f"{pair['a']} / {pair['b']}: no lag could be tested (largest overlap "+ f"{biggest} periods; minimum {min_n})"+ )++ series_out = []+ for info in usable:+ values = info["_values"]+ series_out.append({+ "series_key": info["series_key"],+ "source": info["source"],+ "label": info["label"],+ "unit": info["unit"],+ "rows": info["rows"],+ "rows_without_observed_at": info["rows_without_observed_at"],+ "rows_coarser_than_grid": info["rows_coarser_than_grid"],+ "rows_unreadable_observed_at": info["rows_unreadable_observed_at"],+ "revised_observed_at": info["revised_observed_at"],+ "dated_observations": info["dated_observations"],+ "periods": len(info["_levels"]),+ "periods_outside_window": info["periods_outside_window"],+ "first_period": period_label(min(info["_levels"]), grid),+ "last_period": period_label(max(info["_levels"]), grid),+ "median_period_gap": _median_gap(info["_levels"]),+ "points_after_transform": len(values),+ "lag1_autocorrelation": None if info["_phi"] is None else round(info["_phi"], 4),+ "lag1_pairs": info["_phi_pairs"],+ })+ key = info["series_key"]+ if info["rows_without_observed_at"]:+ gaps.append(f"{key}: {info['rows_without_observed_at']} rows without observed_at not used")+ if info["rows_coarser_than_grid"]:+ gaps.append(f"{key}: {info['rows_coarser_than_grid']} rows coarser than the {grid} grid not used")+ if info["rows_unreadable_observed_at"]:+ gaps.append(f"{key}: {info['rows_unreadable_observed_at']} rows with an unreadable observed_at not used")+ if info["revised_observed_at"]:+ gaps.append(+ f"{key}: {info['revised_observed_at']} observation dates were revised between "+ "captures; the latest collected value is used"+ )+ if len(values) < min_n:+ what = "period-over-period changes" if transform == "changes" else "periods"+ gap = (f"{key}: {len(values)} {what} on the {grid} grid, fewer than the "+ f"{min_n}-point minimum, so no test involving it can run")+ spacing = _median_gap(info["_levels"])+ if transform == "changes" and spacing is not None and spacing > 1:+ gap += (f"; its observations are a median {spacing:g} {unit}s apart, "+ "and changes need adjacent periods, so a coarser grid may fit")+ gaps.append(gap)+ if window["limited_to_recent"]:+ gaps.append(f"window limited to the most recent {MAX_PERIODS} {unit}s")+ if len(usable) < 2:+ gaps.append("fewer than two series could be used, so nothing was tested")++ significant = sum(1 for t in tests if t["significant"])+ with_signal = sum(1 for p in pair_out if p["significant"])+ not_run = sum(p["lags_not_tested"] for p in pair_out)+ return {+ "summary": (+ f"{with_signal} of {len(pair_out)} pairs have a lag with q <= {q_threshold:g} "+ f"({significant} of {len(tests)} tests; {not_run} pair/lag combinations not tested)"+ ),+ "series": series_out,+ "excluded_series": excluded,+ "pairs": pair_out,+ "tests_run": len(tests),+ "tests_not_run": not_run,+ "significant_tests": significant,+ "method": {+ "grid": grid,+ "grid_rule": GRID_RULES[grid],+ "window": window,+ "sampling": SAMPLING_RULE,+ "vintage": VINTAGE_RULE,+ "undated_rows": UNDATED_RULE,+ "transform": transform,+ "transform_rule": TRANSFORM_RULES[transform],+ "why_changes_default": WHY_CHANGES,+ "lags": [-lag, lag],+ "lag_convention": LAG_CONVENTION,+ "overlap": "Pairwise: each (pair, lag) test uses every period where both "+ "transformed values exist; n is reported per test.",+ "correlation": "Pearson r",+ "p_value": P_RULE,+ "multiple_testing": f"Benjamini-Hochberg q across all {len(tests)} tests run in "+ "this call; q_by is the Benjamini-Yekutieli version, valid under any dependence.",+ "q_threshold": q_threshold,+ "q_threshold_default": DEFAULT_Q,+ "strongest": "Per pair, the tested lag(s) with the smallest p (ties broken by "+ "the larger |t|).",+ "min_overlap": min_n,+ "min_overlap_default": DEFAULT_MIN_OVERLAP,+ "min_overlap_reason": MIN_OVERLAP_REASON,+ "not_tested_reasons": dict(NOT_TESTED_REASONS),+ "serial_correlation": AR1_RULE,+ "limits": {+ "max_series": MAX_SERIES, "max_lag": MAX_LAG_CAP,+ "max_tests": MAX_TESTS, "max_periods": MAX_PERIODS,+ "min_overlap_floor": MIN_OVERLAP_FLOOR, "q_range": [Q_FLOOR, Q_CAP],+ },+ },+ "inputs_clamped": clamps,+ "cannot_prove": list(CANNOT_PROVE),+ "data_gaps": gaps,+ "note": HISTORY_ACCRUAL_NOTE,+ }diff --git a/src/world_intel_mcp/cli.py b/src/world_intel_mcp/cli.pyindex 23e13a5..6af7f9b 100644--- a/src/world_intel_mcp/cli.py+++ b/src/world_intel_mcp/cli.py@@ -3318,5 +3318,105 @@ def history_revisions(console.print(f"[yellow]Cannot prove:[/yellow] {escape(line)}")+from .analysis import lagged_connections as _lagged # noqa: E402+++def _p_text(value: float) -> str:+ return "<1e-300" if value == 0 else f"{value:.2g}"+++@history_group.command(name="connections")[email protected]("series_keys", nargs=-1, required=True)[email protected]("--grid", type=click.Choice(_lagged.GRIDS), default=_lagged.DEFAULT_GRID,+ show_default=True, help="UTC calendar grid (weeks run Monday to Sunday)")[email protected]("--transform", type=click.Choice(_lagged.TRANSFORMS),+ default=_lagged.DEFAULT_TRANSFORM, show_default=True,+ help="changes (period-over-period) or levels")[email protected]("--max-lag", type=int, default=_lagged.DEFAULT_MAX_LAG, show_default=True,+ help=f"Largest lead or lag in grid periods (0 to {_lagged.MAX_LAG_CAP})")[email protected]("--min-overlap", type=int, default=_lagged.DEFAULT_MIN_OVERLAP, show_default=True,+ help=f"Fewest overlapping periods a test needs (at least {_lagged.MIN_OVERLAP_FLOOR})")[email protected]("--q", "q", type=float, default=_lagged.DEFAULT_Q, show_default=True,+ help="False discovery rate threshold (Benjamini-Hochberg)")[email protected]("--start", default=None, help="First date YYYY-MM-DD")[email protected]("--end", default=None, help="Last date YYYY-MM-DD")[email protected]_context+def history_connections(+ ctx: click.Context, series_keys: tuple[str, ...], grid: str, transform: str,+ max_lag: int, min_overlap: int, q: float, start: str | None, end: str | None,+) -> None:+ """Correlation between stored series at leads and lags.++ Lag +k compares the first series of a pair at period t with the second+ at t+k, so the first leads by k. A connection is not a cause.+ """+ data = _with_history_store(+ lambda s: _lagged.lagged_connections(+ s, list(series_keys), grid=grid, transform=transform, max_lag=max_lag,+ min_overlap=min_overlap, q=q, start=start, end=end,+ )+ )+ if _bail_on_error(ctx, data):+ return++ method = data["method"]+ lo, hi = method["lags"]+ table = Table(+ title=f"Connections ({method['grid']} grid, {method['transform']}, lags {lo:+d} to "+ f"{hi:+d}, {data['tests_run']} tests, BH q <= {method['q_threshold']:g})",+ box=box.SIMPLE_HEAVY,+ )+ for name in ("A", "B", "Lag", "Timing", "r", "n", "n_eff", "p", "q", "Significant"):+ justify = "right" if name in ("Lag", "r", "n", "n_eff", "p", "q") else "left"+ table.add_column(name, justify=justify)+ for pair in data["pairs"]:+ if not pair["strongest"]:+ table.add_row(_cell(pair["a"]), _cell(pair["b"]), "-",+ Text("not tested", style="dim"), "-", "-", "-", "-", "-", "-")+ for entry in pair["strongest"]:+ n_eff = entry.get("n_effective_ar1")+ table.add_row(+ _cell(pair["a"]), _cell(pair["b"]), f"{entry['lag']:+d}", entry["timing"],+ f"{entry['r']:+.3f}", str(entry["n"]), "-" if n_eff is None else f"{n_eff:g}",+ _p_text(entry["p"]), _p_text(entry["q"]),+ Text("yes", style="green") if entry["significant"] else Text("no"),+ )+ console.print(table)+ console.print(escape(data["summary"]))+ console.print(+ "n_eff: AR(1) effective sample size from the two series' lag-1 autocorrelations, "+ "a diagnostic only: p and q use n."+ )+ console.print(f"Lag convention: {escape(method['lag_convention'])}")+ console.print(+ f"Transform: {escape(method['transform_rule'])} {escape(method['why_changes_default'])}"+ )+ for pair in data["pairs"]:+ best = {e["lag"] for e in pair["strongest"]}+ others = [e for e in pair["significant"] if e["lag"] not in best]+ if others:+ listed = ", ".join(+ f"lag {e['lag']:+d} (r {e['r']:+.3f}, q {_p_text(e['q'])})" for e in others+ )+ console.print(f"Also significant: {escape(pair['a'])} / {escape(pair['b'])}: {listed}")+ if pair["not_tested"]:+ listed = ", ".join(+ f"lag {nt['lag']:+d} (n {nt['n']}"+ + (", no variation)" if nt["reason"] == "no_variation" else ")")+ for nt in pair["not_tested"]+ )+ console.print(+ f"[yellow]Not tested[/yellow] (minimum {method['min_overlap']} overlapping "+ f"periods): {escape(pair['a'])} / {escape(pair['b'])}: {listed}"+ )+ for item in data["excluded_series"]:+ console.print(+ f"[yellow]Excluded:[/yellow] {escape(item['series_key'])}: {escape(item['reason'])}"+ )+ _print_data_gaps(data)+ for line in data["cannot_prove"]:+ console.print(f"[yellow]Cannot prove:[/yellow] {escape(line)}")++if __name__ == "__main__":main()diff --git a/src/world_intel_mcp/tests/test_lagged_connections.py b/src/world_intel_mcp/tests/test_lagged_connections.pynew file mode 100644index 0000000..e203f41--- /dev/null+++ b/src/world_intel_mcp/tests/test_lagged_connections.py@@ -0,0 +1,714 @@+"""Tests for analysis/lagged_connections.py, the intel_lagged_connections+tool and ``intel history connections``.++Reference values (P_CASES, BETA_CASES, PEARSON_*) were computed once with+scipy 1.18.1 and numpy 2.5.4 in a throwaway venv outside the repo and are+hardcoded here; nothing in the repo imports scipy:++- P_CASES: ``2 * scipy.stats.t.sf(abs(r) * sqrt(df / (1 - r**2)), df)`` with+ ``df = n - 2`` (fractional n stands in for an effective sample size).+- BETA_CASES: ``scipy.special.betainc(a, b, x)``.+- PEARSON_R, PEARSON_P: ``scipy.stats.pearsonr(PEARSON_X, PEARSON_Y)``.++The Benjamini-Hochberg examples are worked by hand in their test's+docstring. Synthetic series come from ``random.Random(seed).random()``+through Box-Muller, so they are identical on every supported Python (the+module promises ``random()`` stays stable across versions; ``gauss`` does+not). Series are written through ``HistoryStore.record_cycle`` with a+test-only extractor, so the analysis reads what the production write path+stored.+"""++import importlib+import json+import math+import os+import random+import statistics+import types+from datetime import date, timedelta++import pytest+from click.testing import CliRunner++from world_intel_mcp import cli, history+from world_intel_mcp.analysis import lagged_connections as lc+from world_intel_mcp.history import HistoryStore, Observation, SourceOutcome++from .test_server_registry import _TMP_CACHE++T0 = 1_760_000_000.0 # 2025-10-09T08:53:20Z+D0 = date(2025, 1, 6) # a Monday++P_CASES = [+ (0.0, 30, 1.0),+ (0.05, 30, 0.7930219578171515),+ (0.3, 30, 0.10724594805795433),+ (0.5, 30, 0.004899933667068084),+ (0.9, 30, 1.316606070026386e-11),+ (-0.5, 30, 0.004899933667068084),+ (0.2, 100, 0.04603628646005413),+ (0.1, 1000, 0.0015441161074010938),+ (0.999, 50, 1.9009987456813015e-66),+ (0.7, 3, 0.5063666222132699),+ (0.5, 4, 0.5),+ (0.3, 3000, 1.9407172086957736e-63),+ (0.08, 3000, 1.1493171414190742e-05),+ (0.5, 21.5, 0.019331374409111864),+ (0.35, 7.25, 0.42927655756801303),+ (0.62, 12, 0.031508181347306076),+]+BETA_CASES = [+ (0.5, 0.5, 0.3, 0.36901011956554536),+ (2.5, 0.5, 0.9, 0.4895897445644278),+ (10.0, 3.0, 0.7, 0.25281534785499993),+ (100.0, 0.5, 0.99, 0.15677586542444089),+ (0.5, 50.0, 0.01, 0.6826956021258024),+ (14.0, 0.5, 0.75, 0.004899933667068084),+ (3.0, 7.0, 0.2, 0.26180249600000016),+ (1.0, 1.0, 0.42, 0.42),+]+PEARSON_X = [1.2, 2.3, 2.9, 4.1, 5.0, 6.2, 6.8, 8.1, 9.0, 9.9, 11.3, 12.0]+PEARSON_Y = [2.0, 1.1, 3.9, 3.0, 6.1, 4.2, 7.9, 6.0, 8.1, 11.9, 9.0, 10.2]+PEARSON_R = 0.9036537837015148+PEARSON_P = 5.551095396255053e-05+++def _normals(seed: int, n: int) -> list[float]:+ rng = random.Random(seed)+ out: list[float] = []+ while len(out) < n:+ u1, u2 = rng.random(), rng.random()+ radius = math.sqrt(-2.0 * math.log(1.0 - u1))+ out += [radius * math.cos(2 * math.pi * u2), radius * math.sin(2 * math.pi * u2)]+ return out[:n]+++def _walk(seed: int, n: int) -> list[float]:+ level, out = 0.0, []+ for step in _normals(seed, n):+ level += step+ out.append(level)+ return out+++def _days(values, offset: int = 0) -> list[tuple[str, float]]:+ return [((D0 + timedelta(days=i + offset)).isoformat(), v) for i, v in enumerate(values)]+++def store(tmp_path, monkeypatch):+ monkeypatch.setattr(history, "_EXTRACTORS", dict(history._EXTRACTORS))+ history.register_extractor(+ "lagtest",+ lambda payload: [+ Observation(p["key"], p["value"], observed_at=p["observed_at"])+ for p in payload["points"]+ ],+ )+ s = HistoryStore(tmp_path / "history.db")+ yield s+ s.close()+++def _write(store, series: dict, at: float = T0) -> None:+ """One collector capture of source 'lagtest' holding every given point."""+ points = [+ {"key": key, "observed_at": observed_at, "value": value}+ for key, rows in series.items()+ for observed_at, value in rows+ ]+ store.record_cycle([SourceOutcome("lagtest", at, 1.0, {"points": points})], now=at)+++def _run(store, keys, **kw):+ return lc.lagged_connections(store, [f"lagtest:{k}" if ":" not in k else k for k in keys], **kw)+++def _planted(store, at: float = T0) -> None:+ """x is a random walk; y_t = x_(t-3) + noise, 200 days."""+ x = _walk(11, 200)+ noise = _normals(12, 200)+ y = [x[t - 3] + 0.3 * noise[t] for t in range(3, 200)]+ _write(store, {"x": _days(x), "y": _days(y, offset=3)}, at)+++# ---------------------------------------------------------------------------+# Statistics against independent reference values+# ---------------------------------------------------------------------------++[email protected]("r,n,expected", P_CASES)+def test_p_value_matches_scipy_reference(r, n, expected) -> None:+ assert lc.correlation_p_value(r, n) == pytest.approx(expected, rel=1e-9, abs=1e-300)++[email protected]("a,b,x,expected", BETA_CASES)+def test_incomplete_beta_matches_scipy_reference(a, b, x, expected) -> None:+ assert lc.regularized_incomplete_beta(a, b, x) == pytest.approx(expected, rel=1e-12)+++def test_incomplete_beta_edges_and_bad_arguments() -> None:+ assert lc.regularized_incomplete_beta(2.0, 3.0, 0.0) == 0.0+ assert lc.regularized_incomplete_beta(2.0, 3.0, 1.0) == 1.0+ for bad in [(0.0, 1.0, 0.5), (1.0, -1.0, 0.5), (1.0, 1.0, 1.5)]:+ with pytest.raises(ValueError):+ lc.regularized_incomplete_beta(*bad)+ with pytest.raises(ValueError):+ lc.correlation_p_value(0.5, 2)+ assert lc.correlation_p_value(1.0, 10) == 0.0+ assert lc.correlation_p_value(-1.0, 10) == 0.0+++def test_pearson_matches_scipy_reference() -> None:+ r = lc.pearson_r(PEARSON_X, PEARSON_Y)+ assert r == pytest.approx(PEARSON_R, rel=1e-12)+ assert lc.correlation_p_value(r, len(PEARSON_X)) == pytest.approx(PEARSON_P, rel=1e-9)+ # statistics.correlation is a second, independent Pearson implementation.+ assert r == pytest.approx(statistics.correlation(PEARSON_X, PEARSON_Y), rel=1e-12)+ assert lc.pearson_r([1.0, 1.0, 1.0], [1.0, 2.0, 3.0]) is None+ assert lc.pearson_r([0.1, 0.1, 0.1, 0.1], [1.0, 2.0, 3.0, 4.0]) is None+ with pytest.raises(ValueError):+ lc.pearson_r([1.0, 2.0], [1.0])+++def test_benjamini_hochberg_hand_worked() -> None:+ """m = 4, p = (0.02, 0.03, 0.035, 0.5). Sorted ranks give m p / rank of+ 0.08, 0.06, 0.04667 and 0.5; stepping down from the largest rank keeps+ the running minimum, so q = (0.04667, 0.04667, 0.04667, 0.5). The first+ p alone (0.08 > 0.05) would not pass; the step-up rescues it.++ m = 5, p = (0.01, 0.04, 0.03, 0.005, 0.2): m p / rank in rank order is+ 0.025, 0.025, 0.05, 0.05, 0.2, already non-decreasing, so q in input+ order is (0.025, 0.05, 0.05, 0.025, 0.2).++ BY multiplies BH by 1 + 1/2 + 1/3 + 1/4 = 25/12 for m = 4: 0.04667 x+ 25/12 = 0.09722, and 0.5 x 25/12 caps at 1."""+ assert lc.benjamini_hochberg([0.02, 0.03, 0.035, 0.5]) == pytest.approx(+ [0.14 / 3, 0.14 / 3, 0.14 / 3, 0.5]+ )+ assert lc.benjamini_hochberg([0.01, 0.04, 0.03, 0.005, 0.2]) == pytest.approx(+ [0.025, 0.05, 0.05, 0.025, 0.2]+ )+ assert lc.benjamini_yekutieli([0.02, 0.03, 0.035, 0.5]) == pytest.approx(+ [0.14 / 3 * 25 / 12] * 3 + [1.0]+ )+ assert lc.benjamini_hochberg([0.9, 0.95]) == pytest.approx([0.95, 0.95])+ assert lc.benjamini_hochberg([]) == []+++# ---------------------------------------------------------------------------+# Grid, sampling, transform+# ---------------------------------------------------------------------------+++def test_period_rules_for_each_grid() -> None:+ day = date(2026, 10, 12).toordinal()+ assert lc.period_of("2026-10-12", "day") == (day, None)+ assert lc.period_of("2026-10-12T23:59:59Z", "day") == (day, None)+ assert lc.period_of("2026-10-13", "day") == (day + 1, None)+ # Weeks run Monday (2026-10-12) to Sunday (2026-10-18).+ monday = lc.period_of("2026-10-12", "week")[0]+ assert lc.period_of("2026-10-18T22:00:00Z", "week")[0] == monday+ assert lc.period_of("2026-10-19", "week")[0] == monday + 1+ assert lc.period_label(lc.period_of("2026-10-15", "week")[0], "week") == "2026-10-12"+ december = lc.period_of("2025-12-31", "month")[0]+ assert lc.period_of("2026-01-01", "month")[0] == december + 1+ assert lc.period_of("2026-01", "month")[0] == december + 1+ assert lc.period_label(december + 1, "month") == "2026-01"+ assert lc.period_label(day, "day") == "2026-10-12"+ assert lc.period_of("2026-01", "day") == (None, "coarser")+ assert lc.period_of("2026-01", "week") == (None, "coarser")+ assert lc.period_of("2026", "month") == (None, "coarser")+ for bad in ("2026-13", "2026-02-30", "yesterday", "2026-10-12T03:00:00+02:00"):+ assert lc.period_of(bad, "day")[1] == "unreadable"+++def test_changes_are_never_taken_across_a_gap() -> None:+ levels = {1: 10.0, 2: 12.0, 3: 11.0, 5: 20.0, 6: 26.0}+ assert lc.apply_transform(levels, "changes") == {2: 2.0, 3: -1.0, 6: 6.0}+ assert lc.apply_transform(levels, "levels") == levels+++def test_lag1_autocorrelation_uses_adjacent_periods_only() -> None:+ values = {p: v for p, v in enumerate([3.0, 1.0, 4.0, 1.0, 5.0, 9.0, 2.0, 6.0, 5.0, 3.0, 5.0, 8.0])}+ values[40] = 100.0 # isolated: no neighbour, so it joins no pair+ phi, pairs = lc.lag1_autocorrelation(values)+ seq = [values[p] for p in range(12)]+ assert pairs == 11+ assert phi == pytest.approx(statistics.correlation(seq[:-1], seq[1:]), rel=1e-12)+ assert lc.lag1_autocorrelation({p: float(p % 3) for p in range(5)}) == (None, 4)+++def test_week_grid_takes_each_weeks_last_observation_and_latest_vintage(store) -> None:+ # Business days 2025-01-06 (Mon) to 2025-01-16 (Thu), valued 0..8.+ weekdays = [D0 + timedelta(days=i) for i in range(11) if (D0 + timedelta(days=i)).weekday() < 5]+ _write(store, {"w": [(d.isoformat(), float(i)) for i, d in enumerate(weekdays)]})+ friday = weekdays[4].isoformat() # 2025-01-10+ _write(store, {"w": [(friday, 99.0)]}, at=T0 + 600) # a later capture revises Friday+ info = lc._sample_on_grid("lagtest:w", lc._read_rows(store, "lagtest:w"), "week")+ first_week = lc.period_of(D0.isoformat(), "week")[0]+ # Week 1 ends on Friday, read at its revised value; week 2 ends on Thursday (8).+ assert info["_levels"] == {first_week: 99.0, first_week + 1: 8.0}+ assert info["revised_observed_at"] == 1+ assert (info["rows"], info["dated_observations"]) == (10, 9)+++# ---------------------------------------------------------------------------+# What the tests find+# ---------------------------------------------------------------------------+++def test_planted_lag_is_found_at_the_predicted_lag_and_survives_fdr(store) -> None:+ _planted(store)+ result = _run(store, ["x", "y"])+ (pair,) = result["pairs"]+ (best,) = pair["strongest"]+ # y_t = x_(t-3): x's value at t is compared with y's at t+3, so +3.+ assert best["lag"] == 3+ assert best["timing"] == "A leads B by 3 days"+ assert best["direction"] == "same"+ assert best["r"] > 0.9+ # 199 changes of x (days 1-199), 196 of y (days 4-199); at +3 they meet on 196.+ assert best["n"] == 196+ assert best["q"] < 1e-50 and best["significant"] is True+ assert [e["lag"] for e in pair["significant"]] == [3]+ assert result["tests_run"] == 11 and result["significant_tests"] == 1+ assert result["method"]["q_threshold"] == 0.05+ assert "A leads B by k" in result["method"]["lag_convention"]+ assert result["method"]["transform"] == "changes"+++def test_lag_sign_follows_the_order_the_series_are_given(store) -> None:+ """B_t = A_(t-2) exactly: given as (A, B) the connection sits at +2 with+ r = 1; given as (B, A) it sits at -2, read as B leads A."""+ a = _walk(5, 60)+ _write(store, {"a": _days(a), "b": _days(a[:-2], offset=2)})+ forward = _run(store, ["a", "b"])["pairs"][0]["strongest"][0]+ assert (forward["lag"], forward["timing"]) == (2, "A leads B by 2 days")+ assert forward["r"] == pytest.approx(1.0) and forward["p"] == 0.0+ backward = _run(store, ["b", "a"])["pairs"][0]["strongest"][0]+ assert (backward["lag"], backward["timing"]) == (-2, "B leads A by 2 days")+++def test_independent_series_give_no_fdr_significant_result(store) -> None:+ """Four independent random walks (seeds 101-104), 250 days, default+ settings: 6 pairs x 11 lags = 66 tests, none at q <= 0.05 (the smallest+ q measured is 0.699)."""+ _write(store, {f"ind{i}": _days(_walk(100 + i, 250)) for i in range(1, 5)})+ result = _run(store, [f"ind{i}" for i in range(1, 5)])+ assert result["tests_run"] == 66+ assert result["significant_tests"] == 0+ assert all(not p["significant"] for p in result["pairs"])+ assert min(e["q"] for p in result["pairs"] for e in p["strongest"]) > 0.05+ assert result["summary"].startswith("0 of 6 pairs")+++def test_random_walks_correlate_in_levels_but_not_in_changes(store) -> None:+ """Independent random walks are the textbook spurious correlation. Seeds+ 1000/1001: |r| = 0.87 in levels (q ~ 1e-59), nothing in changes. Over 20+ seed pairs (measured: 14 with an FDR-significant level lag, 1 with an+ FDR-significant change lag, the nominal 5% rate) the default protects.+ The AR(1) diagnostic flags the level result: its effective n is ~1."""+ _write(store, {"rwa": _days(_walk(1000, 200)), "rwb": _days(_walk(1001, 200))})+ levels = _run(store, ["rwa", "rwb"], transform="levels")+ best = levels["pairs"][0]["strongest"][0]+ assert abs(best["r"]) > 0.8 and best["significant"] is True+ assert best["n_effective_ar1"] < 0.1 * best["n"]+ assert best["p_ar1"] is None or best["p_ar1"] > 100 * best["p"]+ assert all(s["lag1_autocorrelation"] > 0.9 for s in levels["series"])+ changes = _run(store, ["rwa", "rwb"])+ assert changes["significant_tests"] == 0+ assert abs(changes["pairs"][0]["strongest"][0]["r"]) < 0.2++ level_hits = change_hits = 0+ for s in range(1, 20):+ _write(store, {f"a{s}": _days(_walk(1000 + 2 * s, 200)),+ f"b{s}": _days(_walk(1001 + 2 * s, 200))}, at=T0 + s)+ level_hits += _run(store, [f"a{s}", f"b{s}"], transform="levels")["significant_tests"] > 0+ change_hits += _run(store, [f"a{s}", f"b{s}"])["significant_tests"] > 0+ assert level_hits >= 12+ assert change_hits <= 2+++def test_weekly_grid_finds_a_two_week_lead_in_daily_data(store) -> None:+ """y(d) = x(d - 14) daily; on the week grid each week takes its Sunday,+ so Y_w = X_(w-2) and the weekly changes match exactly at +2 weeks."""+ x = _walk(21, 7 * 60)+ _write(store, {"x": _days(x), "y": _days(x[:-14], offset=14)})+ result = _run(store, ["x", "y"], grid="week", max_lag=3)+ best = result["pairs"][0]["strongest"][0]+ assert (best["lag"], best["timing"]) == (2, "A leads B by 2 weeks")+ assert best["r"] == pytest.approx(1.0)+ assert result["method"]["grid_rule"].startswith("ISO weeks in UTC, Monday to Sunday")+ assert {s["series_key"]: s["first_period"] for s in result["series"]} == {+ "lagtest:x": D0.isoformat(), "lagtest:y": (D0 + timedelta(days=14)).isoformat()+ }+++def test_month_periods_fit_the_month_grid_and_are_coarser_than_days(store) -> None:+ a = _walk(31, 41)+ months = [f"{2020 + m // 12}-{m % 12 + 1:02d}" for m in range(41)]+ _write(store, {"ma": list(zip(months, a)), "mb": list(zip(months[1:], a[:-1]))})+ monthly = _run(store, ["ma", "mb"], grid="month", max_lag=2)+ best = monthly["pairs"][0]["strongest"][0]+ assert (best["lag"], best["timing"]) == (1, "A leads B by 1 month")+ daily = _run(store, ["ma", "mb"])+ assert daily["pairs"] == []+ reasons = {e["series_key"]: e["reason"] for e in daily["excluded_series"]}+ assert "coarser than the day grid" in reasons["lagtest:ma"]+ assert "Use grid=month" in reasons["lagtest:ma"]+ assert "fewer than two series could be used, so nothing was tested" in daily["data_gaps"]+++# ---------------------------------------------------------------------------+# Minimum sample, revisions, undated rows+# ---------------------------------------------------------------------------+++def test_minimum_overlap_excludes_and_reports_each_lag_with_its_n(store) -> None:+ """33 days each, so 32 changes on the same days: lag k overlaps 32 - |k|.+ At the default minimum of 30, lags -2..+2 run and the rest are listed."""+ _write(store, {"s1": _days(_walk(41, 33)), "s2": _days(_walk(42, 33))})+ result = _run(store, ["s1", "s2"])+ (pair,) = result["pairs"]+ assert result["method"]["min_overlap"] == lc.DEFAULT_MIN_OVERLAP == 30+ assert "80% power" in result["method"]["min_overlap_reason"]+ assert pair["not_tested"] == [+ {"lag": -5, "n": 27, "reason": "below_min_overlap"},+ {"lag": -4, "n": 28, "reason": "below_min_overlap"},+ {"lag": -3, "n": 29, "reason": "below_min_overlap"},+ {"lag": 3, "n": 29, "reason": "below_min_overlap"},+ {"lag": 4, "n": 28, "reason": "below_min_overlap"},+ {"lag": 5, "n": 27, "reason": "below_min_overlap"},+ ]+ assert (pair["lags_tested"], pair["lags_not_tested"]) == (5, 6)+ assert (result["tests_run"], result["tests_not_run"]) == (5, 6)+ assert abs(pair["strongest"][0]["lag"]) <= 2+ lowered = _run(store, ["s1", "s2"], min_overlap=20)+ assert (lowered["tests_run"], lowered["tests_not_run"]) == (11, 0)+++def test_pairs_without_variation_and_short_series_are_reported(store) -> None:+ _write(store, {+ "flat": _days([4.33] * 40),+ "moving": _days(_walk(51, 40)),+ "short": _days(_walk(52, 10)),+ })+ result = _run(store, ["flat", "moving", "short"], max_lag=1)+ by_pair = {(p["a"], p["b"]): p for p in result["pairs"]}+ flat = by_pair[("lagtest:flat", "lagtest:moving")]+ assert [nt["reason"] for nt in flat["not_tested"]] == ["no_variation"] * 3+ assert [nt["n"] for nt in flat["not_tested"]] == [38, 39, 38]+ assert flat["strongest"] == []+ gaps = "\n".join(result["data_gaps"])+ assert "lagtest:short: 9 period-over-period changes on the day grid" in gaps+ assert "lagtest:flat / lagtest:moving: no lag could be tested" in gaps+++def test_revisions_use_the_latest_vintage(store) -> None:+ """First capture: a is unrelated noise. A later capture revises every one+ of a's 60 points to b two days later. The tool must read the revision."""+ b = _walk(61, 62)+ _write(store, {"a": _days(_normals(62, 60)), "b": _days(b)})+ before = _run(store, ["a", "b"], max_lag=3)+ assert before["significant_tests"] == 0+ _write(store, {"a": _days(b[2:])}, at=T0 + 3600)+ after = _run(store, ["a", "b"], max_lag=3)+ best = after["pairs"][0]["strongest"][0]+ assert (best["lag"], best["r"]) == (2, pytest.approx(1.0))+ a_info = next(s for s in after["series"] if s["series_key"] == "lagtest:a")+ assert a_info["revised_observed_at"] == 60+ assert a_info["rows"] == 120 and a_info["dated_observations"] == 60+ assert "latest collected value" in after["method"]["vintage"]+ assert any("60 observation dates were revised" in g for g in after["data_gaps"])+++def test_series_without_observed_at_are_excluded_with_the_stated_rule(store) -> None:+ for i, price in enumerate(_walk(71, 40)):+ store.record_cycle(+ [SourceOutcome("market_quotes", T0 + 300 * i, 1.0,+ {"quotes": [{"symbol": "^GSPC", "price": 5000 + price}]})],+ now=T0 + 300 * i,+ )+ _planted(store)+ result = _run(store, ["market_quotes:^GSPC:price", "x", "y"])+ (excluded,) = result["excluded_series"]+ assert excluded["series_key"] == "market_quotes:^GSPC:price"+ assert "no row has observed_at (40 rows carry collected_at only)" in excluded["reason"]+ assert "stored only on change" in excluded["reason"]+ assert [(p["a"], p["b"]) for p in result["pairs"]] == [("lagtest:x", "lagtest:y")]+ assert "collected_at" in result["method"]["undated_rows"]+ alone = _run(store, ["market_quotes:^GSPC:price", "x"])+ assert alone["pairs"] == [] and alone["tests_run"] == 0+ assert "fewer than two series could be used, so nothing was tested" in alone["data_gaps"]+++def test_window_selects_whole_periods(store) -> None:+ _planted(store)+ result = _run(store, ["x", "y"], start="2025-03-01", end="2025-05-31")+ window = result["method"]["window"]+ assert (window["start"], window["end"], window["periods"]) == ("2025-03-01", "2025-05-31", 92)+ x_info = result["series"][0]+ assert x_info["periods"] == 92 and x_info["periods_outside_window"] == 108+ assert result["pairs"][0]["strongest"][0]["lag"] == 3+ weekly = _run(store, ["x", "y"], grid="week", start="2025-03-05", end="2025-03-05", min_overlap=20)+ assert weekly["method"]["window"]["start"] == "2025-03-03"+ empty = _run(store, ["x", "y"], start="2030-01-01")+ assert empty["pairs"] == []+ assert {e["reason"] for e in empty["excluded_series"]} == {"no observation inside the window"}+++def test_window_is_capped_to_the_most_recent_periods(store, monkeypatch) -> None:+ monkeypatch.setattr(lc, "MAX_PERIODS", 120)+ _planted(store)+ result = _run(store, ["x", "y"])+ window = result["method"]["window"]+ assert window["limited_to_recent"] is True and window["periods"] == 120+ assert window["end"] == (D0 + timedelta(days=199)).isoformat()+ assert "window limited to the most recent 120 days" in result["data_gaps"]+++# ---------------------------------------------------------------------------+# Serial-correlation diagnostic+# ---------------------------------------------------------------------------+++def test_ar1_effective_sample_size_follows_the_reported_autocorrelations(store) -> None:+ _planted(store)+ result = _run(store, ["x", "y"], transform="levels")+ best = result["pairs"][0]["strongest"][0]+ phi_a, phi_b = (s["lag1_autocorrelation"] for s in result["series"])+ expected = min(best["n"], best["n"] * (1 - phi_a * phi_b) / (1 + phi_a * phi_b))+ assert best["n_effective_ar1"] == pytest.approx(expected, abs=0.5)+ assert best["n_effective_ar1"] < best["n"]+ assert "does not enter q" in result["method"]["serial_correlation"]+ assert any("Serial correlation inflates significance" in c for c in result["cannot_prove"])+++def test_cannot_prove_says_a_small_q_is_not_a_large_effect(store) -> None:+ """The effect-size line quotes the module's own p for |r| 0.1 at n 1,000+ (P_CASES pins that p to scipy's 0.00154). The list a call returns is a+ copy: clearing it leaves the module's list whole."""+ _planted(store)+ result = _run(store, ["x", "y"])+ line = next(c for c in result["cannot_prove"] if c.startswith("A small q is not a large effect"))+ assert "an |r| of 0.1 (1% of the variance shared) already gives p = 0.0015;" in line+ result["cannot_prove"].clear()+ result["method"]["not_tested_reasons"].clear()+ assert len(lc.CANNOT_PROVE) == 7+ assert set(lc.NOT_TESTED_REASONS) == {"below_min_overlap", "no_variation"}+++# ---------------------------------------------------------------------------+# Inputs: clamps and errors+# ---------------------------------------------------------------------------+++def test_numeric_inputs_are_clamped_and_reported(store) -> None:+ _planted(store)+ result = _run(store, ["x", "y"], max_lag=500, min_overlap=5, q=0.9)+ assert result["method"]["lags"] == [-30, 30]+ assert result["method"]["min_overlap"] == 20+ assert result["method"]["q_threshold"] == 0.25+ assert {c["input"]: (c["requested"], c["used"]) for c in result["inputs_clamped"]} == {+ "max_lag": (500, 30), "min_overlap": (5, 20), "q": (0.9, 0.25)+ }+ low = _run(store, ["x", "y"], max_lag=-4, q=0)+ assert low["method"]["lags"] == [0, 0] and low["tests_run"] == 1+ assert low["method"]["q_threshold"] == 0.001+ assert _run(store, ["x", "y"], max_lag=4.0)["method"]["lags"] == [-4, 4]+++def test_total_tests_are_capped_by_reducing_max_lag(store) -> None:+ _write(store, {f"c{i}": _days(_walk(80 + i, 40)) for i in range(8)})+ result = _run(store, [f"c{i}" for i in range(8)], max_lag=30, min_overlap=20)+ assert result["method"]["lags"] == [-17, 17]+ assert len(result["pairs"]) == 28+ assert result["tests_run"] + result["tests_not_run"] == 28 * 35 <= lc.MAX_TESTS+ reduced = [c for c in result["inputs_clamped"] if c.get("reason")]+ assert reduced == [{+ "input": "max_lag", "requested": 30, "used": 17,+ "reason": "28 pairs x 35 lags keeps the call within 1000 tests",+ }]+++ "kwargs,message",+ [+ ({"series_keys": ["lagtest:x"]}, "at least 2 series"),+ ({"series_keys": [f"lagtest:k{i}" for i in range(9)]}, "at most 8 series"),+ ({"series_keys": ["lagtest:x", "lagtest:x"]}, "repeats ['lagtest:x']"),+ ({"series_keys": "lagtest:x"}, "must be a list"),+ ({"series_keys": None}, "must be a list"),+ ({"series_keys": ["lagtest:x", ""]}, "non-empty strings"),+ ({"grid": "hourly"}, "unknown grid 'hourly'; allowed: day, week, month"),+ ({"transform": "logs"}, "unknown transform 'logs'; allowed: changes, levels"),+ ({"max_lag": "5"}, "max_lag must be an integer"),+ ({"max_lag": True}, "max_lag must be an integer"),+ ({"min_overlap": 25.5}, "min_overlap must be an integer"),+ ({"q": "0.05"}, "q must be a number"),+ ({"q": float("nan")}, "q must be a number"),+ ({"start": "2025/01/01"}, "start must be a date YYYY-MM-DD"),+ ({"end": "2025-02-30"}, "end must be a date YYYY-MM-DD"),+ ({"start": "2025-06-01", "end": "2025-05-01"}, "is after end"),+ ],+)+def test_input_errors_name_what_is_allowed(store, kwargs, message) -> None:+ _planted(store)+ args = {"series_keys": ["lagtest:x", "lagtest:y"], **kwargs}+ keys = args.pop("series_keys")+ result = lc.lagged_connections(store, keys, **args)+ assert set(result) == {"error"}+ assert message in result["error"]+++def test_unknown_series_is_an_error_with_a_listing_hint(store) -> None:+ _planted(store)+ result = _run(store, ["x", "nope"])+ assert result["unknown_series"] == ["lagtest:nope"]+ assert "intel_series_history" in result["error"]+++def test_user_facing_text_has_no_em_dashes() -> None:+ from world_intel_mcp.tools import lagged_connections as tool_mod++ texts = [tool_mod.TOOLS[0].description, json.dumps(tool_mod.TOOLS[0].inputSchema),+ cli.history_connections.help or ""]+ texts += [v for k, v in vars(lc).items() if k.isupper() and isinstance(v, str)]+ texts += lc.CANNOT_PROVE + list(lc.GRID_RULES.values()) + list(lc.TRANSFORM_RULES.values())+ texts += [p.help or "" for p in cli.history_connections.params if hasattr(p, "help")]+ assert not [t for t in texts if "—" in t]+++# ---------------------------------------------------------------------------+# MCP tool through the real dispatcher+# ---------------------------------------------------------------------------++[email protected](scope="module")+def server():+ """server.py imported under test_server_registry's temp cache path, the+ shared runtime rebound while the override is in force (that file's+ fixture pattern)."""+ prior = os.environ.get("WORLD_INTEL_CACHE_DB")+ os.environ["WORLD_INTEL_CACHE_DB"] = str(_TMP_CACHE)+ try:+ module = importlib.import_module("world_intel_mcp.server")+ importlib.import_module("world_intel_mcp.runtime")._initialize()+ yield module+ finally:+ if prior is None:+ os.environ.pop("WORLD_INTEL_CACHE_DB", None)+ else:+ os.environ["WORLD_INTEL_CACHE_DB"] = prior+++def tool_db(server, store, monkeypatch):+ _planted(store)+ runtime = importlib.import_module("world_intel_mcp.runtime")+ monkeypatch.setattr(runtime, "cache", types.SimpleNamespace(db_path=store.db_path))+ return store.db_path+++def test_tool_is_registered_with_its_schema(server) -> None:+ (tool,) = [t for t in server.TOOLS if t.name == "intel_lagged_connections"]+ props = tool.inputSchema["properties"]+ assert tool.inputSchema["required"] == ["series_keys"]+ assert props["grid"]["enum"] == ["day", "week", "month"]+ assert props["transform"]["enum"] == ["changes", "levels"]+ assert props["transform"]["default"] == "changes"+ assert "accrues only while the collector" in tool.description+ assert "correlation is not causation" in tool.description+ assert "A leads B by k" in tool.description+++async def test_tool_through_the_real_dispatcher(server, tool_db) -> None:+ args = {"series_keys": ["lagtest:x", "lagtest:y"], "max_lag": 4}+ result = await server._dispatch("intel_lagged_connections", args)+ best = result["pairs"][0]["strongest"][0]+ assert (best["lag"], best["n"], best["significant"]) == (3, 196, True)+ assert result["method"]["lags"] == [-4, 4] and result["tests_run"] == 9++ (content,) = await server.call_tool("intel_lagged_connections", args)+ assert json.loads(content.text)["pairs"][0]["strongest"][0]["lag"] == 3++ bad = await server._dispatch(+ "intel_lagged_connections", {"series_keys": ["lagtest:x", "lagtest:y"], "grid": "hourly"}+ )+ assert bad == {"error": "unknown grid 'hourly'; allowed: day, week, month"}+ missing = await server._dispatch("intel_lagged_connections", {})+ assert "must be a list" in missing["error"]+++# ---------------------------------------------------------------------------+# CLI+# ---------------------------------------------------------------------------+++def cli_db(tmp_path, monkeypatch):+ monkeypatch.setattr(history, "_EXTRACTORS", dict(history._EXTRACTORS))+ history.register_extractor(+ "lagtest",+ lambda payload: [+ Observation(p["key"], p["value"], observed_at=p["observed_at"])+ for p in payload["points"]+ ],+ )+ db = tmp_path / "cli-cache.db"+ monkeypatch.setenv("WORLD_INTEL_CACHE_DB", str(db))+ monkeypatch.setenv("COLUMNS", "240")+ monkeypatch.setattr(cli, "_fetcher", None)+ s = HistoryStore(db)+ _planted(s)+ _write(s, {"[red]odd[/red]": _days(_walk(91, 33))})+ s.close()+ yield db+ # The CLI's shared fetcher opened a Cache on this db; close it before+ # monkeypatch restores the previous fetcher.+ if cli._fetcher is not None:+ cli._fetcher.cache.close()+++def _invoke(*args):+ return CliRunner().invoke(cli.main, list(args), catch_exceptions=False)+++def test_cli_connections_table_states_convention_and_limits(cli_db) -> None:+ out = _invoke("history", "connections", "lagtest:x", "lagtest:y").output+ assert "A leads B by 3 days" in out and "+3" in out+ assert "Lag convention: Pairs are (A, B)" in out+ assert "Transform: Period-over-period changes" in out+ assert "Cannot prove: Correlation is not causation" in out+ assert "1 of 1 pairs have a lag with q <= 0.05" in out+++def test_cli_connections_lists_untested_lags_and_renders_keys_literally(cli_db) -> None:+ out = _invoke("history", "connections", "lagtest:x", "lagtest:[red]odd[/red]").output+ assert "lagtest:[red]odd[/red]" in out+ assert "Not tested (minimum 30 overlapping periods)" in out+ assert "lag +3 (n 29)" in out+++def test_cli_connections_shows_the_ar1_effective_sample_size(cli_db) -> None:+ args = ("history", "connections", "lagtest:x", "lagtest:y", "--transform", "levels")+ best = json.loads(_invoke("--json-output", *args).output)["pairs"][0]["strongest"][0]+ assert best["n_effective_ar1"] < best["n"] / 2+ out = _invoke(*args).output+ assert f" {best['n_effective_ar1']:g} " in out and f" {best['n']} " in out+ assert "n_eff: AR(1) effective sample size" in out and "p and q use n" in out+++def test_cli_connections_json_and_errors(cli_db) -> None:+ data = json.loads(+ _invoke("--json-output", "history", "connections", "lagtest:x", "lagtest:y",+ "--grid", "week", "--min-overlap", "20").output+ )+ assert data["method"]["grid"] == "week"+ assert data["method"]["min_overlap"] == 20+ out = _invoke("history", "connections", "lagtest:x", "lagtest:nope").output+ assert out.startswith("Error:") and "lagtest:nope" in out+ bad = CliRunner().invoke(cli.main, ["history", "connections", "a", "b", "--grid", "hour"])+ assert bad.exit_code == 2 and "day" in bad.output and "month" in bad.outputdiff --git a/src/world_intel_mcp/tools/__init__.py b/src/world_intel_mcp/tools/__init__.pyindex fc0bb4d..8a59c12 100644--- a/src/world_intel_mcp/tools/__init__.py+++ b/src/world_intel_mcp/tools/__init__.py@@ -31,6 +31,7 @@ from . import ( # noqa: E402history,infrastructure,intelligence,+ lagged_connections,liquidity,markets,news_market,@@ -59,6 +60,7 @@ _MODULES = [vector,aoi,history,+ lagged_connections,system,]diff --git a/src/world_intel_mcp/tools/lagged_connections.py b/src/world_intel_mcp/tools/lagged_connections.pynew file mode 100644index 0000000..44c5298--- /dev/null+++ b/src/world_intel_mcp/tools/lagged_connections.py@@ -0,0 +1,131 @@+"""Lagged connections: correlation between stored series at leads and lags.++Reads the history store (``world_intel_mcp.history``) on the same+``db_path`` the runtime cache resolved and runs+``analysis/lagged_connections.py`` over it; nothing here fetches upstream+data or writes history.+"""++import asyncio+from typing import Any++from mcp.types import Tool++from .. import runtime+from ..analysis import lagged_connections as lc+from ..history import HistoryStore++TOOLS: list[Tool] = [+ Tool(+ name="intel_lagged_connections",+ description=(+ f"Correlation between 2 to {lc.MAX_SERIES} series in the history store (keys as "+ "intel_series_history lists them), aligned on a UTC day, week or month grid and "+ "tested at every lag from -max_lag to +max_lag. Default transform is period-over-"+ "period changes, because levels of trending series correlate spuriously; 'levels' "+ "on request. Per pair: the lag with the strongest evidence (smallest p) with "+ "Pearson r, overlap n, two-sided p from Student's t, and the Benjamini-Hochberg q "+ f"across every test in the call (significant at q <= {lc.DEFAULT_Q:g} by default). "+ "Lag +k compares A at period t with B at t+k, so A leads B by k. Pair/lag "+ f"overlaps under {lc.DEFAULT_MIN_OVERLAP} points are listed as not tested. Revised "+ "points use the latest vintage; rows without observed_at (Yahoo quotes) are not "+ "used. States what it cannot prove: correlation is not causation and a lead is not "+ "a causal direction. History accrues only while the collector (intel-collector, "+ "or intel_collect) runs; nothing before its first recorded cycle exists."+ ),+ inputSchema={+ "type": "object",+ "properties": {+ "series_keys": {+ "type": "array",+ "items": {"type": "string"},+ "description": (+ f"2 to {lc.MAX_SERIES} series keys, e.g. ['space_weather:kp', "+ "'central_bank_rates:federal_funds_rate']; pairs are formed in this order"+ ),+ },+ "grid": {+ "type": "string",+ "enum": list(lc.GRIDS),+ "description": "UTC calendar grid (default day; weeks run Monday to Sunday)",+ "default": lc.DEFAULT_GRID,+ },+ "transform": {+ "type": "string",+ "enum": list(lc.TRANSFORMS),+ "description": "changes (default, period-over-period differences) or levels",+ "default": lc.DEFAULT_TRANSFORM,+ },+ "max_lag": {+ "type": "integer",+ "description": (+ f"Largest lead or lag in grid periods (default {lc.DEFAULT_MAX_LAG}, "+ f"clamped to 0..{lc.MAX_LAG_CAP}, reduced to keep a call within "+ f"{lc.MAX_TESTS:,} tests)"+ ),+ "default": lc.DEFAULT_MAX_LAG,+ },+ "min_overlap": {+ "type": "integer",+ "description": (+ f"Fewest overlapping periods a pair/lag needs to be tested (default "+ f"{lc.DEFAULT_MIN_OVERLAP}, clamped to {lc.MIN_OVERLAP_FLOOR}.."+ f"{lc.MAX_PERIODS})"+ ),+ "default": lc.DEFAULT_MIN_OVERLAP,+ },+ "q": {+ "type": "number",+ "description": (+ f"False discovery rate threshold (default {lc.DEFAULT_Q:g}, clamped to "+ f"{lc.Q_FLOOR:g}..{lc.Q_CAP:g})"+ ),+ "default": lc.DEFAULT_Q,+ },+ "start": {+ "type": "string",+ "description": "First date YYYY-MM-DD; its grid period and later are used",+ },+ "end": {+ "type": "string",+ "description": "Last date YYYY-MM-DD; its grid period and earlier are used",+ },+ },+ "required": ["series_keys"],+ },+ ),+]+++def _run(db_path: Any, arguments: dict[str, Any]) -> dict:+ # Opened and closed in the worker thread: sqlite3 connections stay on+ # the thread that created them.+ store = HistoryStore(db_path)+ try:+ return lc.lagged_connections(+ store,+ arguments.get("series_keys"),+ grid=arguments.get("grid"),+ transform=arguments.get("transform"),+ max_lag=arguments.get("max_lag"),+ min_overlap=arguments.get("min_overlap"),+ q=arguments.get("q"),+ start=arguments.get("start"),+ end=arguments.get("end"),+ )+ finally:+ store.close()+++async def _lagged_connections(arguments: dict[str, Any]) -> Any:+ # Up to 1,000 tests over up to 3,000 periods is CPU-bound for a moment,+ # so it runs off the event loop. The path is resolved here, on the loop+ # thread: a first touch of runtime.cache opens the shared Cache, whose+ # sqlite connection must not be born in a worker thread.+ db_path = runtime.cache.db_path+ return await asyncio.to_thread(_run, db_path, arguments)+++HANDLERS = {+ "intel_lagged_connections": _lagged_connections,+}