Skip to content

160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.

Dashboard research panels: liquidity board, situations, top stories, collection health

Phoenix Agent (mac-studio)2e951fe2ee34
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 37b196a..7cc85e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,37 @@
## Unreleased
+### Added: dashboard research panels (Phase 29)
+- RESEARCH in the dashboard top bar (or the `p` key) opens four tabs:
+ liquidity board, situations, top stories and collection health. Each tab
+ loads from its own route when opened and refreshes every 60 s while the
+ panel is open: `/api/liquidity-board`, `/api/situations`,
+ `/api/top-stories`, `/api/collection-health`. Each route calls the
+ function the MCP tool calls (`fetch_liquidity_board`, `fetch_situations`,
+ `fetch_top_stories`, and `HistoryStore.collection_health` on the
+ dashboard cache's database) and passes its payload through; the
+ liquidity route adds the net liquidity `method` block.
+- Failures read as failures: a payload with `error` is a 503 with the
+ payload intact, an exception a 503 with the reason, the 90 s budget a
+ 504, a bad parameter a 400 before anything is fetched. A panel never
+ shows figures it has not received: before the first answer it says so, a
+ failed first load shows UNAVAILABLE with the reason and the payload's
+ diagnostics (unavailable indicators, blocked sources, data gaps), and a
+ failed refresh keeps the earlier copy under a REFRESH FAILED banner with
+ the time it loaded. An answer to a query the controls have since changed
+ is dropped.
+- Liquidity rows keep the units each row states and are grouped by
+ measure, amounts outstanding apart from flows. Collection health lists
+ failing, degraded, not-attempted-recently and never-run sources and shows
+ `cannot_prove` as given; a database with no capture says every source
+ reads never_run and why.
+- Remote text is escaped by the renderers and sanitized again on
+ assignment; only http and https links are kept.
+- Reading the collector roster imports the collector module, whose
+ `logging.basicConfig` would attach a root handler in the dashboard
+ process; the root logger is put back as it was, so the first
+ collection-health request does not start printing every upstream request.
+
### Fixed: development runs no longer touch user databases
- Temporal and keyword spike databases now follow `XDG_CACHE_HOME` like
the cache: `$XDG_CACHE_HOME/world-intel-mcp/temporal.db` and
diff --git a/README.md b/README.md
index dce211e..1a3c4f0 100644
--- a/README.md
+++ b/README.md
@@ -114,6 +114,14 @@ intel-dashboard # http://localhost:8501
intel-dashboard --port 9000 # custom port
```
+**Research panels** (RESEARCH in the top bar, or the `p` key): four tabs over the same functions the MCP tools call, each loaded from its own JSON route when the tab opens and refreshed every 60 s while the panel is open.
+
+- **Liquidity board** (`/api/liquidity-board?sort=biggest_move|newest`, `fetch_liquidity_board`): latest value, units as each row states them (USD millions, USD billions, EUR millions; nothing converted), change and percent change against the previous observation, freshness and stale-cache badges, the ranking rule, unavailable indicators with their reasons, the blocked upstreams (ICI, FINRA, BLS) and the net liquidity method. Amounts outstanding are grouped apart from flows; a change between two levels is labelled as not a flow.
+- **Situations** (`/api/situations?hours=&sort=&kind=`, `fetch_situations`) and **Top stories** (`/api/top-stories?hours=&min_outlets=`, `fetch_top_stories`): ranked situations or multi-outlet stories with sample headlines, the attention list, feed coverage (failed feeds named) and the functions' own definitions, method and data gaps.
+- **Collection health** (`/api/collection-health`, `HistoryStore.collection_health`): state counts, then failing, degraded, not-attempted-recently and never-run sources with last attempt, last success, failures in a row, overdue and reason, storage size, and the report's `cannot_prove` text as given.
+
+A panel that has not loaded says so; a failed call shows UNAVAILABLE with the reason (HTTP 503, or 504 after a 90 s budget) and never an empty list; a failed refresh keeps the earlier copy under a REFRESH FAILED banner with the time it loaded. Limits: the situations and top stories panels read every RSS feed: 5 to 15 s per call in the 2026-10-17 runs, warm cache included, because a feed that fails is not cached and every call tries it again, with retries, for up to 12 s. Collection health reads the history store in the dashboard's own cache database (`WORLD_INTEL_CACHE_DB`, or the default path), so it shows only cycles a collector ran against that same database; one writing elsewhere leaves every source `never_run`. Like the other dashboard routes, these answer with `Access-Control-Allow-Origin: *`, so a page open in the same browser can read them, the cache database path included; the server binds 127.0.0.1 unless `--host` says otherwise.
+
### PDF/HTML Reports
```bash
diff --git a/ROADMAP.md b/ROADMAP.md
index 5c992a8..b31d680 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -598,7 +598,7 @@ dashboard.
| 1 | News vs market reaction | Companies and countries named in story clusters mapped to tickers / country indices; same-day move against a benchmark | :white_check_mark: Built 2026-10-17, unreleased: `intel_news_market_reaction` and `intel news-reaction`. Company rows: stock, benchmark (SPY, or the exchange country's index) and excess return on the story's session; country rows: index return with a trailing z-score. Session rule in the exchange's timezone (after close = next session; weekends and holidays read from the listing's own bars). Open: `^TASI` and `^TA125` (Saudi and Israel in the country-index table) return 404 on Yahoo, so those benchmarks are data gaps; story times inherit `sources/news.py`'s RSS timestamp offset until that is fixed |
| 2 | Connections with timing offsets | Correlation between retained series over a shared sample, tested at leads and lags, with minimum sample size and false-discovery control; states that it does not prove causation | :red_circle: |
| 2 | Day timeline | What every retained source recorded for a given day, by observation date or collection date | :red_circle: |
-| 3 | Dashboard views | Liquidity board, situations, top stories, collection health panels | :red_circle: |
+| 3 | Dashboard views | Liquidity board, situations, top stories, collection health panels | :white_check_mark: Built 2026-10-17, unreleased: four tabs behind RESEARCH (key `p`), each loaded from its own route that calls the tool's function (`/api/liquidity-board`, `/api/situations`, `/api/top-stories`, and `/api/collection-health` on the dashboard cache's history store); a failed call renders as UNAVAILABLE or REFRESH FAILED with the function's reason, never as an empty list. Checked 2026-10-17 in headless Chrome with real upstreams (9 liquidity rows; 20 situations and 9 stories over 95 of 119 feeds; collection health after one collector cycle: 38 ok, 5 degraded, 16 failing), with FRED and ECB, every RSS feed and the history store made unreachable (each tab showed the outage and its reason), and with markup injected into a headline (shown as text). Situations and top stories take 5 to 15 s per call |
Considered and not planned:
diff --git a/src/world_intel_mcp/dashboard/app.py b/src/world_intel_mcp/dashboard/app.py
index 3f3e7c5..67ac6f0 100644
--- a/src/world_intel_mcp/dashboard/app.py
+++ b/src/world_intel_mcp/dashboard/app.py
@@ -5,6 +5,7 @@ All data pulled from the same source modules used by the MCP server.
"""
import asyncio
+import copy
import json
import logging
from contextlib import asynccontextmanager
@@ -50,7 +51,10 @@ from world_intel_mcp.sources import (
traffic,
webcams,
migration,
+ liquidity,
)
+from world_intel_mcp.analysis import news_signals
+from world_intel_mcp.history import HistoryStore
from world_intel_mcp.analysis.alerts import fetch_alert_digest, fetch_weekly_trends
from world_intel_mcp.analysis.posture import fetch_strategic_posture
from world_intel_mcp.analysis.exposure import fetch_population_exposure
@@ -732,6 +736,199 @@ async def api_migration(request):
return JSONResponse(payload, headers=cors)
+# ---------------------------------------------------------------------------
+# Research panels (ROADMAP Phase 29): liquidity board, situations, top
+# stories, collection health. Each route calls the function the matching MCP
+# tool calls and passes its payload through unchanged, so the partial-failure
+# markers it carries (unavailable rows, data_gaps, stale and degraded flags)
+# reach the page. Only whole-call failures become HTTP errors: a payload with
+# ``error`` is a 503 with the payload intact, a timeout a 504, an exception a
+# 503 with the reason. None of them answers with an empty list that could
+# read as "nothing happened".
+# ---------------------------------------------------------------------------
+
+_CORS = {"Access-Control-Allow-Origin": "*"}
+
+# The budget /api/stream gives its slow sources. The RSS panels read every
+# feed: 5 to 15 s per call measured 2026-10-17, warm cache included, as a
+# failed feed is not cached and each call retries it for up to 12 s.
+_RESEARCH_TIMEOUT_S = 90.0
+
+# Accepted values, as each MCP tool's input schema declares them.
+_LIQUIDITY_SORTS = ("biggest_move", "newest")
+_SITUATION_SORTS = ("covered", "hostile", "change")
+_SITUATION_KINDS = ("all", "between", "inside")
+
+
+def _query_number(params, name: str, default, kind, low, high):
+ """A numeric query parameter, or ``default`` when absent. Raises
+ ValueError (answered as a 400) when it does not parse or is outside
+ ``[low, high]``, so a request is never silently answered for other
+ values than it asked for."""
+ raw = params.get(name)
+ if raw is None:
+ return default
+ try:
+ value = kind(raw)
+ except ValueError:
+ raise ValueError(f"{name} must be a number between {low} and {high}, got {raw!r}") from None
+ if not low <= value <= high: # also refuses nan and inf
+ raise ValueError(f"{name} must be between {low} and {high}, got {raw!r}")
+ return value
+
+
+def _query_choice(params, name: str, choices: tuple[str, ...]) -> str:
+ value = params.get(name, choices[0])
+ if value not in choices:
+ raise ValueError(f"{name} must be one of {', '.join(choices)}, got {value!r}")
+ return value
+
+
+async def _research_payload(name: str, coro) -> tuple[dict, int]:
+ """Await one research function: (payload, HTTP status). Only this
+ budget expiring is reported as the timeout; a TimeoutError raised
+ inside the function is a failure like any other exception."""
+ budget = asyncio.timeout(_RESEARCH_TIMEOUT_S)
+ try:
+ async with budget:
+ data = await coro
+ except Exception as exc:
+ if isinstance(exc, TimeoutError) and budget.expired():
+ logger.warning("Dashboard %s timed out after %gs", name, _RESEARCH_TIMEOUT_S)
+ return {"error": f"{name} timed out after {_RESEARCH_TIMEOUT_S:g} s", "timeout": True}, 504
+ logger.warning("Dashboard %s failed: %s", name, exc)
+ return {"error": f"{name} failed: {type(exc).__name__}: {str(exc)[:200]}"}, 503
+ return data, 503 if data.get("error") else 200
+
+
+def _bad_request(exc: ValueError) -> JSONResponse:
+ return JSONResponse({"error": str(exc)}, status_code=400, headers=_CORS)
+
+
+async def api_liquidity_board(request):
+ """``sources.liquidity.fetch_liquidity_board`` (``?sort=biggest_move|newest``).
+
+ Adds ``net_liquidity_method``: the liquidity module's own statement of
+ how the derived net liquidity row is computed (each term's units and
+ conversion, the RRP alignment, why the TGA term is the Wednesday level),
+ the block ``intel_net_liquidity`` returns as ``method``. It is attached
+ to failures too, so a board that could not compute still says what it
+ would have computed.
+ """
+ try:
+ sort = _query_choice(request.query_params, "sort", _LIQUIDITY_SORTS)
+ except ValueError as exc:
+ return _bad_request(exc)
+ payload, status = await _research_payload(
+ "liquidity board", liquidity.fetch_liquidity_board(_ensure_fetcher(), sort=sort)
+ )
+ payload.setdefault("net_liquidity_method", copy.deepcopy(liquidity.NET_LIQUIDITY_METHOD))
+ return JSONResponse(payload, status_code=status, headers=_CORS)
+
+
+async def api_situations(request):
+ """``analysis.news_signals.fetch_situations`` over the RSS feeds
+ (``?hours=1-168&sort=covered|hostile|change&kind=all|between|inside&limit=&min_articles=``).
+ Every RSS feed failing is the function's own error answer, a 503."""
+ params = request.query_params
+ try:
+ kwargs = {
+ "hours": _query_number(params, "hours", 24.0, float, 1, 168),
+ "sort": _query_choice(params, "sort", _SITUATION_SORTS),
+ "kind": _query_choice(params, "kind", _SITUATION_KINDS),
+ "limit": _query_number(params, "limit", 20, int, 1, 100),
+ "min_articles": _query_number(params, "min_articles", 2, int, 1, 1000),
+ }
+ except ValueError as exc:
+ return _bad_request(exc)
+ payload, status = await _research_payload(
+ "situations", news_signals.fetch_situations(_ensure_fetcher(), **kwargs)
+ )
+ return JSONResponse(payload, status_code=status, headers=_CORS)
+
+
+async def api_top_stories(request):
+ """``analysis.news_signals.fetch_top_stories``
+ (``?hours=1-168&min_outlets=&limit=&attention_limit=``)."""
+ params = request.query_params
+ try:
+ kwargs = {
+ "hours": _query_number(params, "hours", 24.0, float, 1, 168),
+ "min_outlets": _query_number(params, "min_outlets", 3, int, 1, 50),
+ "limit": _query_number(params, "limit", 10, int, 1, 50),
+ "attention_limit": _query_number(params, "attention_limit", 10, int, 1, 50),
+ }
+ except ValueError as exc:
+ return _bad_request(exc)
+ payload, status = await _research_payload(
+ "top stories", news_signals.fetch_top_stories(_ensure_fetcher(), **kwargs)
+ )
+ return JSONResponse(payload, status_code=status, headers=_CORS)
+
+
+def _history_db_path() -> Path:
+ """The cache database the dashboard's Fetcher opened. The collector
+ writes its run log into the cache database of the environment it runs
+ in, so this is the store ``intel_collection_health`` reads in an MCP
+ server configured the same way."""
+ return _ensure_fetcher().cache.db_path
+
+
+def _collector_roster() -> list[str]:
+ """The collector's source names, imported lazily as tools/history.py
+ does. Importing the collector module runs its ``logging.basicConfig``
+ (level INFO); in this process, where nothing configures the root
+ logger, that would attach a stderr handler and start printing a line
+ per upstream request. The root logger's handlers and level are put
+ back as they were. Called on the event loop thread, so no other
+ request reconfigures logging in between."""
+ root = logging.getLogger()
+ handlers, level = list(root.handlers), root.level
+ from world_intel_mcp.collector import SOURCES
+
+ for handler in [h for h in root.handlers if h not in handlers]:
+ root.removeHandler(handler)
+ root.setLevel(level)
+ return [name for name, _, _, _ in SOURCES]
+
+
+def _read_collection_health(db_path: Path, roster: list[str], source: str | None) -> dict:
+ """Open the history store, read collection health, close. Runs in a
+ worker thread, so the store's SQLite connection is created and used on
+ that one thread and the read (which sums payload sizes) never stalls
+ the event loop that serves the SSE stream."""
+ store = HistoryStore(db_path)
+ try:
+ return store.collection_health(roster, source=source)
+ finally:
+ store.close()
+
+
+async def api_collection_health(request):
+ """``HistoryStore.collection_health`` on the dashboard cache's database
+ (``?source=`` for one source). A store that cannot be opened or read is
+ a 503 with the reason and the database path, never a list of zeros; a
+ database the collector never wrote reports every source ``never_run``;
+ an unknown source is a 404 naming the known ones."""
+ source = request.query_params.get("source") or None
+ db_path = None
+ try:
+ db_path = _history_db_path()
+ roster = _collector_roster()
+ data = await asyncio.to_thread(_read_collection_health, db_path, roster, source)
+ except Exception as exc:
+ logger.warning("Collection health unavailable: %s", exc)
+ return JSONResponse(
+ {
+ "error": f"history store unavailable: {type(exc).__name__}: {exc}",
+ "db_path": None if db_path is None else str(db_path),
+ },
+ status_code=503,
+ headers=_CORS,
+ )
+ return JSONResponse(data, status_code=404 if "error" in data else 200, headers=_CORS)
+
+
async def api_health(request):
"""Health check."""
return JSONResponse({"status": "ok"})
@@ -887,6 +1084,10 @@ app = Starlette(
Route("/api/static", api_static),
Route("/api/aois", api_aois),
Route("/api/migration", api_migration),
+ Route("/api/liquidity-board", api_liquidity_board),
+ Route("/api/situations", api_situations),
+ Route("/api/top-stories", api_top_stories),
+ Route("/api/collection-health", api_collection_health),
Route("/api/health", api_health),
Route("/api/report/pdf", api_report_pdf),
Route("/sse/vector-analytics", sse_vector_analytics),
diff --git a/src/world_intel_mcp/dashboard/index.html b/src/world_intel_mcp/dashboard/index.html
index 02dfa72..15b92f7 100644
--- a/src/world_intel_mcp/dashboard/index.html
+++ b/src/world_intel_mcp/dashboard/index.html
@@ -623,6 +623,39 @@ a { color: var(--accent); text-decoration: none; }
.cluster-conflict div { background: rgba(245,158,11,0.3) !important; color: var(--amber) !important; }
.cluster-fire div { background: rgba(240,184,64,0.3) !important; color: var(--gold) !important; }
.cluster-military div { background: rgba(96,165,250,0.3) !important; color: var(--blue) !important; }
+/* ═══════════════ RESEARCH PANELS ═══════════════ */
+#researchPanel {
+ /* right: clears the open drawer and its toggle tab */
+ position: fixed; top: var(--layers-top, 66px); left: 220px; right: 396px; bottom: 46px; z-index: 48;
+ display: flex; flex-direction: column; padding: 10px 14px;
+}
+#researchPanel[hidden] { display: none; }
+.rs-head { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 6px; padding-bottom: 8px; border-bottom: 1px solid var(--border); }
+.rs-tabs, .rs-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 4px; }
+.rs-ctl { display: none; align-items: center; gap: 4px; font-family: var(--mono); font-size: 0.6rem; color: var(--dim); }
+.rs-ctl.on { display: inline-flex; }
+.rs-ctl select {
+ background: rgba(0,0,0,0.35); color: var(--bright); border: 1px solid var(--border); border-radius: 4px;
+ font-family: var(--mono); font-size: 0.6rem; padding: 2px 4px;
+}
+#rsClose { position: static; float: none; padding: 0 4px 0 8px; font-size: 1.2rem; line-height: 1; }
+.rs-body { flex: 1; overflow-y: auto; padding-top: 8px; font-size: 0.7rem; }
+.rs-body .sub { margin-top: 12px; }
+.rs-banner { padding: 6px 10px; margin: 4px 0 8px; border-radius: 6px; border: 1px solid var(--border); font-family: var(--mono); font-size: 0.64rem; line-height: 1.45; color: var(--bright); }
+.rs-banner.err { background: rgba(255,59,59,0.08); border-color: rgba(255,59,59,0.28); }
+.rs-banner.warn { background: rgba(255,159,28,0.07); border-color: rgba(255,159,28,0.25); }
+.rs-banner.info { background: rgba(0,229,255,0.05); border-color: rgba(0,229,255,0.15); }
+.rs-badge { display: inline-block; padding: 0 5px; margin-right: 2px; border-radius: 3px; border: 1px solid var(--border); font-family: var(--mono); font-size: 0.55rem; letter-spacing: 0.5px; color: var(--text); }
+.rs-badge.err { color: var(--red); border-color: rgba(255,59,59,0.35); }
+.rs-badge.warn { color: var(--amber); border-color: rgba(255,159,28,0.35); }
+.rs-note { margin: 2px 0 6px; color: var(--dim); font-size: 0.62rem; line-height: 1.45; }
+.rs-list > div { padding: 2px 0; border-top: 1px solid rgba(255,255,255,0.03); line-height: 1.4; }
+.rs-story { padding: 6px 0; border-top: 1px solid rgba(255,255,255,0.05); line-height: 1.45; }
+.rs-body .dtable td { vertical-align: top; }
+.rs-num { text-align: right; white-space: nowrap; }
+@media (max-width: 900px) {
+ #researchPanel { left: 10px; right: 10px; }
+}
</style>
</head>
<body>
@@ -646,6 +679,7 @@ a { color: var(--accent); text-decoration: none; }
<div class="search-box">
<input class="search-input" id="searchInput" type="text" placeholder="/ Search..." spellcheck="false" autocomplete="off" />
</div>
+ <button class="rgn-btn" id="researchToggle" title="Research panels: liquidity board, situations, top stories, collection health (key p)">RESEARCH</button>
<div class="hud-stats" id="hudStats"></div>
<div class="hud-status">
<span class="conn-dot" id="connDot"></span>
@@ -823,6 +857,41 @@ a { color: var(--accent); text-decoration: none; }
</div>
</div>
+<!-- RESEARCH PANELS -->
+<section id="researchPanel" class="glass" hidden aria-label="Research panels">
+ <div class="rs-head">
+ <div class="rs-tabs">
+ <button class="rgn-btn rs-tab" data-rs-tab="liquidity">Liquidity board</button>
+ <button class="rgn-btn rs-tab" data-rs-tab="situations">Situations</button>
+ <button class="rgn-btn rs-tab" data-rs-tab="stories">Top stories</button>
+ <button class="rgn-btn rs-tab" data-rs-tab="health">Collection health</button>
+ </div>
+ <div class="rs-controls">
+ <label class="rs-ctl" data-rs-for="liquidity">sort
+ <select id="rsLiqSort"><option value="biggest_move">biggest move</option><option value="newest">newest</option></select>
+ </label>
+ <label class="rs-ctl" data-rs-for="situations">window
+ <select id="rsSitHours"><option value="6">6 h</option><option value="24" selected>24 h</option><option value="72">72 h</option><option value="168">7 d</option></select>
+ </label>
+ <label class="rs-ctl" data-rs-for="situations">sort
+ <select id="rsSitSort"><option value="covered">most covered</option><option value="hostile">most hostile</option><option value="change">biggest change</option></select>
+ </label>
+ <label class="rs-ctl" data-rs-for="situations">kind
+ <select id="rsSitKind"><option value="all">all</option><option value="between">between countries</option><option value="inside">inside one country</option></select>
+ </label>
+ <label class="rs-ctl" data-rs-for="stories">window
+ <select id="rsTopHours"><option value="6">6 h</option><option value="24" selected>24 h</option><option value="72">72 h</option><option value="168">7 d</option></select>
+ </label>
+ <label class="rs-ctl" data-rs-for="stories">min outlets
+ <select id="rsTopOutlets"><option value="2">2</option><option value="3" selected>3</option><option value="4">4</option><option value="5">5</option></select>
+ </label>
+ <button class="rgn-btn" id="rsRefresh" title="Reload this panel now">Refresh</button>
+ <button class="detail-close" id="rsClose" title="Close (Esc)">&times;</button>
+ </div>
+ </div>
+ <div class="rs-body" id="rsBody"></div>
+</section>
+
<script>
// ════════════════════════════════════════════════════════════════════════════
// Phoenix Intelligence Dashboard v3 — Map-first global intelligence
@@ -1561,6 +1630,7 @@ function _buildHelpOverlay() {
['__section', 'Actions'],
['a', 'Toggle all layers on/off'],
['f', 'Toggle fullscreen'],
+ ['p', 'Research panels (liquidity, situations, stories, collection health)'],
];
rows.forEach(function(r) {
var tr = document.createElement('tr');
@@ -3548,6 +3618,458 @@ function connectVectorSSE() {
source.onopen = function() { vecReconnectDelay = 1000; };
}
connectVectorSSE();
+
+// ════════════ RESEARCH PANELS ════════════
+// research-render:begin
+// Pure renderers: (panel request state, payload) -> HTML string. They touch
+// no DOM and no page globals, so the test suite runs this block under a
+// JavaScript runtime and checks the markup. Every payload string passes
+// through rEsc and every link through rHref (http and https only); the
+// controller below still assigns the result through safe().
+var RESEARCH_REFRESH_MS = 60000;
+
+function rEsc(s) {
+ return String(s == null ? '' : s).replace(/[&<>"']/g, function(c) {
+ return { '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;' }[c];
+ });
+}
+function rHref(u) {
+ var s = String(u == null ? '' : u).trim();
+ return /^https?:\/\/[^\s"'<>]+$/i.test(s) ? s : null;
+}
+function rLink(text, url) {
+ var h = rHref(url);
+ return h ? '<a href="' + rEsc(h) + '" data-external="1">' + rEsc(text) + '</a>' : rEsc(text);
+}
+function rNum(v, maxDigits) {
+ if (typeof v !== 'number' || !isFinite(v)) return 'n/a';
+ return v.toLocaleString('en-US', { maximumFractionDigits: maxDigits == null ? 2 : maxDigits });
+}
+function rSigned(v, maxDigits) {
+ if (typeof v !== 'number' || !isFinite(v)) return 'n/a';
+ return (v > 0 ? '+' : '') + rNum(v, maxDigits);
+}
+function rClock(ms) { return new Date(ms).toISOString().slice(11, 19) + 'Z'; }
+function rAge(seconds) {
+ if (typeof seconds !== 'number' || !isFinite(seconds)) return 'n/a';
+ var s = Math.max(0, Math.round(seconds));
+ if (s < 120) return s + ' s';
+ if (s < 7200) return Math.round(s / 60) + ' min';
+ if (s < 172800) return Math.round(s / 3600) + ' h';
+ return Math.round(s / 86400) + ' d';
+}
+function rPlural(n, word) { return rNum(n, 0) + ' ' + word + (n === 1 ? '' : 's'); }
+function rArr(v) { return Array.isArray(v) ? v : []; }
+function rList(items) {
+ return '<div class="rs-list">' + items.map(function(t) { return '<div>' + rEsc(t) + '</div>'; }).join('') + '</div>';
+}
+function rGaps(d) {
+ var gaps = rArr(d.data_gaps);
+ return gaps.length ? '<div class="sub">Data gaps</div>' + rList(gaps) : '';
+}
+function rNotes(d) {
+ var h = '';
+ var defs = d.definitions && typeof d.definitions === 'object' ? d.definitions : {};
+ var keys = Object.keys(defs);
+ if (keys.length) h += '<div class="sub">Definitions</div>' + rList(keys.map(function(k) { return k + ': ' + defs[k]; }));
+ if (d.method) h += '<div class="sub">Method</div><div class="rs-note">' + rEsc(d.method) + '</div>';
+ return h + rGaps(d);
+}
+
+// Null when the response is a usable payload; otherwise the reason, in the
+// server's words when it gave any. The HTTP status is shown separately.
+function researchAttemptError(http, data) {
+ if (data === null || typeof data !== 'object' || Array.isArray(data)) return http ? 'response was not JSON' : 'no response';
+ if (data.error) return typeof data.error === 'string' ? data.error : JSON.stringify(data.error);
+ if (!(http >= 200 && http < 300)) return 'no error message in the response';
+ return null;
+}
+
+// Liquidity board. Rows are grouped by what they measure and never mixed:
+// an amount outstanding is a level, a flow is a transaction total, and a
+// change between two levels is not a flow. Values keep the units the
+// payload states; nothing is converted.
+var R_MEASURES = [
+ ['amount_outstanding', 'Amounts outstanding (stocks)',
+ 'Level at the end of each period, in the units each row states; nothing is converted between units or currencies. A change is the difference between two levels, in the row’s units: it is not a flow.'],
+ ['flow', 'Flows (transactions within the period)',
+ 'Transactions during the period. A flow is not a level: it is never added to or netted against the amounts outstanding.'],
+ ['growth_rate', 'Growth rates (percent)', 'Annual growth rates; a change in a rate is in percentage points.'],
+ ['index', 'Indices', 'Index levels, not amounts.']
+];
+function rLiqRow(r) {
+ var f = r.freshness && typeof r.freshness === 'object' ? r.freshness : {};
+ var fresh = (typeof f.latest_age_days === 'number' ? rNum(f.latest_age_days, 0) + ' d old' : 'age unknown') +
+ (typeof f.expected_max_age_days === 'number' ? ' (expected within ' + rNum(f.expected_max_age_days, 0) + ' d)' : '');
+ if (f.stale) fresh += ' <span class="rs-badge warn">STALE</span>';
+ if (r.served_from_stale_cache || f.served_from_stale_cache) {
+ var age = typeof r.cache_age_seconds === 'number' ? r.cache_age_seconds : f.cache_age_seconds;
+ fresh += ' <span class="rs-badge warn">STALE CACHE' + (typeof age === 'number' ? ', ' + rAge(age) + ' old' : '') + '</span>';
+ }
+ return '<tr><td><span class="bright">' + rEsc(r.label || r.indicator) + '</span>' +
+ (r.status ? ' <span class="rs-badge">' + rEsc(r.status) + '</span>' : '') +
+ '<br><span class="dim">' + rEsc([r.dataset, r.source, r.frequency].filter(Boolean).join(' · ')) + '</span></td>' +
+ '<td class="rs-num">' + rNum(r.latest_value) + '</td>' +
+ '<td>' + rEsc(r.units || 'units not stated') + '</td>' +
+ '<td>' + rEsc(r.observed_at || 'n/a') + (r.period_end ? '<br><span class="dim">period end ' + rEsc(r.period_end) + '</span>' : '') + '</td>' +
+ '<td class="rs-num">' + rSigned(r.change) + (typeof r.change_pct === 'number' ? '<br><span class="dim">' + rSigned(r.change_pct) + '%</span>' : '') + '</td>' +
+ '<td class="rs-num">' + rNum(r.previous_value) + (r.previous_observed_at ? '<br><span class="dim">' + rEsc(r.previous_observed_at) + '</span>' : '') + '</td>' +
+ '<td>' + fresh + '</td></tr>';
+}
+function rLiqGroup(rows, title, note) {
+ if (!rows.length) return '';
+ var bases = [];
+ rows.forEach(function(r) { if (r.change_basis && bases.indexOf(r.change_basis) < 0) bases.push(r.change_basis); });
+ return '<div class="sub">' + rEsc(title) + '</div><div class="rs-note">' + rEsc(note) +
+ (bases.length ? ' Change: ' + rEsc(bases.join('; ')) + '.' : '') + '</div>' +
+ '<table class="dtable"><thead><tr><th>Indicator</th><th>Latest</th><th>Units</th><th>Observed</th><th>Change</th><th>Previous</th><th>Freshness</th></tr></thead><tbody>' +
+ rows.map(rLiqRow).join('') + '</tbody></table>';
+}
+function rLiqMethod(d) {
+ var m = d.net_liquidity_method && typeof d.net_liquidity_method === 'object' ? d.net_liquidity_method : {};
+ var formula = d.net_liquidity_formula || m.formula;
+ var comps = m.components && typeof m.components === 'object' ? m.components : {};
+ var keys = Object.keys(comps);
+ if (!formula && !keys.length && !m.alignment && !m.tga_choice) return '';
+ var h = '<div class="sub">How net liquidity is computed</div>';
+ if (formula) h += '<div class="rs-note">' + rEsc(formula) + '</div>';
+ if (keys.length) {
+ h += '<table class="dtable"><thead><tr><th>Series</th><th>Term</th><th>Observed</th><th>Units</th><th>Conversion</th></tr></thead><tbody>';
+ keys.forEach(function(k) {
+ var c = comps[k] && typeof comps[k] === 'object' ? comps[k] : {};
+ h += '<tr><td>' + rEsc(k) + '</td><td>' + rEsc(c.term) + '</td><td>' + rEsc(c.observed) +
+ (c.alignment ? '<br><span class="dim">' + rEsc(c.alignment) + '</span>' : '') + '</td><td>' + rEsc(c.units) + '</td><td>' + rEsc(c.conversion) + '</td></tr>';
+ });
+ h += '</tbody></table>';
+ }
+ if (m.alignment) h += '<div class="rs-note">Alignment: ' + rEsc(m.alignment) + '</div>';
+ if (m.tga_choice) h += '<div class="rs-note">TGA series: ' + rEsc(m.tga_choice) + '</div>';
+ return h;
+}
+function renderLiquidity(d, failed) {
+ var h = '';
+ if (!failed) {
+ var rows = rArr(d.indicators);
+ if (!rows.length) h += '<div class="rs-banner warn">The board answered with no indicator rows and no error.</div>';
+ var typed = {};
+ R_MEASURES.forEach(function(g) {
+ typed[g[0]] = true;
+ h += rLiqGroup(rows.filter(function(r) { return r.measure === g[0]; }), g[1], g[2]);
+ });
+ h += rLiqGroup(rows.filter(function(r) { return !typed[r.measure]; }), 'Measure not typed',
+ 'The payload did not type these rows as a stock, flow, growth rate or index; read each on its own.');
+ if (d.ranking_rule) h += '<div class="rs-note">Sorted by ' + rEsc(d.sort || 'the order returned') + '. ' + rEsc(d.ranking_rule) + '</div>';
+ if (d.note) h += '<div class="rs-note">' + rEsc(d.note) + '</div>';
+ }
+ var un = rArr(d.unavailable);
+ if (un.length) {
+ h += '<div class="rs-banner warn">' + rPlural(un.length, 'indicator') + ' unavailable: shown as missing, never as zero.</div>' +
+ '<table class="dtable"><thead><tr><th>Indicator</th><th>Dataset</th><th>Reason</th></tr></thead><tbody>' +
+ un.map(function(u) { u = u || {}; return '<tr><td>' + rEsc(u.indicator) + '</td><td>' + rEsc(u.dataset || 'n/a') + '</td><td>' + rEsc(u.error) + '</td></tr>'; }).join('') +
+ '</tbody></table>';
+ }
+ h += rLiqMethod(d);
+ var blocked = rArr(d.blocked);
+ if (blocked.length) {
+ h += '<div class="sub">Not served: blocked upstreams</div>' +
+ '<table class="dtable"><thead><tr><th>Measure</th><th>Publisher</th><th>Status</th><th>Measured</th></tr></thead><tbody>' +
+ blocked.map(function(b) { b = b || {}; return '<tr><td>' + rEsc(b.measure) + '</td><td>' + rEsc(b.publisher) + '</td><td><span class="rs-badge warn">' + rEsc(b.status || 'blocked') + '</span></td><td>' + rEsc(b.measured) + '</td></tr>'; }).join('') +
+ '</tbody></table>';
+ }
+ return h + rGaps(d);
+}
+
+// Situations and top stories read the same RSS feeds; coverage says how
+// much of them could be read, so an empty list is a stated result.
+function rCoverage(d) {
+ var h = '';
+ var w = d.window && typeof d.window === 'object' ? d.window : null;
+ if (w && w.current) {
+ h += '<div class="rs-note">Window: last ' + rNum(w.hours, 1) + ' h, ' + rEsc(w.current.start) + ' to ' + rEsc(w.current.end) +
+ (w.prior ? '; compared with ' + rEsc(w.prior.start) + ' to ' + rEsc(w.prior.end) : '') + '.</div>';
+ }
+ var c = d.coverage && typeof d.coverage === 'object' ? d.coverage : null;
+ if (c) {
+ h += '<div class="rs-note">Read ' + rNum(c.items_read, 0) + ' items from ' + rPlural(c.feeds_ok, 'feed') + '; ' +
+ rNum(c.items_current_window, 0) + ' in the current window, ' + rNum(c.items_current_window_comparable, 0) + ' of them from the ' +
+ rNum(c.feeds_comparable, 0) + ' feeds comparable with the prior window, which had ' + rNum(c.items_prior_window, 0) + '.</div>';
+ var failedFeeds = rArr(c.feeds_failed), staleFeeds = rArr(c.feeds_stale);
+ if (failedFeeds.length) {
+ h += '<div class="rs-banner warn">Partial coverage: ' + rPlural(failedFeeds.length, 'feed') + ' failed or timed out and ' +
+ (failedFeeds.length === 1 ? 'is' : 'are') + ' absent: ' + rEsc(failedFeeds.join(', ')) + '.</div>';
+ }
+ if (staleFeeds.length) {
+ h += '<div class="rs-banner warn">' + rPlural(staleFeeds.length, 'feed') + ' failed live and came from an earlier copy: ' + rEsc(staleFeeds.join(', ')) + '.</div>';
+ }
+ }
+ return h;
+}
+function rFailedNews(d) {
+ return (d.reason ? '<div class="rs-note">Reason: ' + rEsc(d.reason) + '</div>' : '') + rCoverage(d) + rGaps(d);
+}
+function rHeadline(item) {
+ item = item || {};
+ var meta = [item.outlet, item.published].concat(rArr(item.hostility).length ? ['hostility: ' + rArr(item.hostility).join(', ')] : []);
+ return '<div>' + rLink(item.title || '(untitled)', item.link) + ' <span class="dim">' + rEsc(meta.filter(Boolean).join(', ')) + '</span></div>';
+}
+function renderSituations(d, failed) {
+ if (failed) return rFailedNews(d);
+ var list = rArr(d.situations);
+ var h = rCoverage(d);
+ if (!list.length) {
+ h += '<div class="rs-banner info">No situation' + (d.kind && d.kind !== 'all' ? ' (' + rEsc(d.kind) + ')' : '') + ' reached ' +
+ rNum(d.min_articles, 0) + ' articles in this window, over the coverage stated above.</div>';
+ }
+ list.forEach(function(s) {
+ s = s || {};
+ var names = rArr(s.countries).map(function(c) { return c && (c.name || c.iso3); }).filter(Boolean);
+ var cats = s.by_category && typeof s.by_category === 'object' ? s.by_category : {};
+ var catText = Object.keys(cats).filter(function(k) { return cats[k]; }).map(function(k) { return k + ' ' + cats[k]; }).join(', ');
+ h += '<div class="rs-story"><div><span class="bright">' + rEsc(names.join(s.kind === 'between' ? ' ↔ ' : ', ') || s.key) + '</span> ' +
+ '<span class="rs-badge">' + rEsc(s.kind) + '</span></div>' +
+ '<div class="dim">' + rPlural(s.articles, 'article') + ' from ' + rPlural(s.outlet_count, 'outlet') + '; hostile ' + rNum(s.hostile_articles, 0) +
+ (typeof s.hostile_share === 'number' ? ' (' + rNum(s.hostile_share * 100, 0) + '%)' : '') + (catText ? ': ' + rEsc(catText) : '') +
+ '; comparable feeds ' + rNum(s.current_comparable, 0) + ' now vs ' + rNum(s.prior, 0) + ' prior (change ' + rSigned(s.change, 0) +
+ ', ratio ' + rNum(s.change_ratio, 3) + ')</div>' +
+ '<div class="rs-list">' + rArr(s.sample_headlines).map(rHeadline).join('') + '</div></div>';
+ });
+ return h + rNotes(d);
+}
+function renderTopStories(d, failed) {
+ if (failed) return rFailedNews(d);
+ var stories = rArr(d.stories), att = rArr(d.attention);
+ var h = rCoverage(d) + '<div class="sub">Stories carried by at least ' + rPlural(d.min_outlets, 'outlet') + '</div>';
+ if (!stories.length) {
+ h += '<div class="rs-banner info">No story was carried by ' + rNum(d.min_outlets, 0) + ' or more distinct outlets in this window, over the coverage stated above.</div>';
+ }
+ stories.forEach(function(s) {
+ s = s || {};
+ h += '<div class="rs-story"><div class="bright">' + rEsc(s.headline) + '</div>' +
+ '<div class="dim">' + rPlural(s.outlet_count, 'outlet') + ', ' + rPlural(s.articles, 'article') + '; first seen ' + rEsc(s.first_seen) + ', last seen ' + rEsc(s.last_seen) + '</div>' +
+ '<div class="dim">Outlets: ' + rEsc(rArr(s.outlets).join(', ')) + '</div>' +
+ (rArr(s.keywords).length ? '<div class="tags">' + rArr(s.keywords).map(function(k) { return '<span class="tag">' + rEsc(k) + '</span>'; }).join('') + '</div>' : '') +
+ '<div class="rs-list">' + rArr(s.items).map(rHeadline).join('') + '</div></div>';
+ });
+ h += '<div class="sub">Attention: named entities that grew against the prior window</div>';
+ if (!att.length) {
+ h += '<div class="rs-banner info">No named country or organization grew against the prior window; only growth is listed.</div>';
+ } else {
+ h += '<table class="dtable"><thead><tr><th>Entity</th><th>Type</th><th>Now</th><th>Prior</th><th>Change</th><th>Ratio</th></tr></thead><tbody>' +
+ att.map(function(a) {
+ a = a || {};
+ return '<tr><td>' + rEsc(a.name) + (a.iso3 ? ' <span class="dim">' + rEsc(a.iso3) + '</span>' : '') + '</td><td>' + rEsc(a.type) + '</td>' +
+ '<td class="rs-num">' + rNum(a.current_comparable, 0) + '</td><td class="rs-num">' + rNum(a.prior, 0) + '</td>' +
+ '<td class="rs-num">' + rSigned(a.change, 0) + '</td><td class="rs-num">' + rNum(a.change_ratio, 3) + '</td></tr>';
+ }).join('') + '</tbody></table>';
+ }
+ return h + rNotes(d);
+}
+
+// Collection health: problems first; never_run is listed, not hidden, and
+// the report's own cannot_prove text is shown as given.
+var R_HEALTH_STATES = ['failing', 'degraded', 'not_attempted_recently', 'never_run', 'ok'];
+function rHealthRow(s) {
+ var reason = [s.last_reason, s.last_extract_error ? 'extraction: ' + s.last_extract_error : null].filter(Boolean).join('; ');
+ var overdue = s.overdue === true ? '<span class="rs-badge err">OVERDUE</span>' : (s.overdue === false ? 'no' : 'n/a');
+ return '<tr><td>' + rEsc(s.source) + (s.in_collector_roster === false ? ' <span class="rs-badge warn">not in roster</span>' : '') + '</td>' +
+ '<td>' + rEsc(s.last_attempt_at || 'n/a') + (typeof s.last_attempt_age_s === 'number' ? '<br><span class="dim">' + rAge(s.last_attempt_age_s) + ' before check</span>' : '') + '</td>' +
+ '<td>' + rEsc(s.last_success_at || 'never') + (typeof s.last_success_age_s === 'number' ? '<br><span class="dim">' + rAge(s.last_success_age_s) + ' before check</span>' : '') + '</td>' +
+ '<td class="rs-num">' + rNum(s.consecutive_failures, 0) + '</td>' +
+ '<td>' + overdue + (typeof s.overdue_after_s === 'number' ? '<br><span class="dim">after ' + rAge(s.overdue_after_s) + '</span>' : '') + '</td>' +
+ '<td>' + rEsc(reason || 'n/a') + '</td>' +
+ '<td>' + rEsc(s.latest_observed_at || 'n/a') + '</td></tr>';
+}
+function renderCollectionHealth(d, failed) {
+ if (failed) {
+ return (d.db_path ? '<div class="rs-note">Database: ' + rEsc(d.db_path) + '</div>' : '') +
+ (rArr(d.known_sources).length ? '<div class="rs-note">Known sources: ' + rEsc(rArr(d.known_sources).join(', ')) + '</div>' : '');
+ }
+ var sources = rArr(d.sources);
+ var counts = d.state_counts && typeof d.state_counts === 'object' ? d.state_counts : {};
+ var st = d.storage && typeof d.storage === 'object' ? d.storage : {};
+ var states = R_HEALTH_STATES.slice();
+ sources.forEach(function(s) { if (s && states.indexOf(s.state) < 0) states.push(s.state); });
+ var h = '<div class="rs-note">Checked ' + rEsc(d.checked_at) + ' in ' + rEsc(st.db_path || 'the dashboard cache database') +
+ '. Expected cadence ' + rNum(d.default_cadence_s, 0) + ' s unless a source overrides it; overdue past ' + rNum(d.overdue_multiple, 1) + ' cadences.</div>';
+ h += '<div class="mini-row">' + states.map(function(k) {
+ return '<div class="mini-box"><div class="v">' + rNum(typeof counts[k] === 'number' ? counts[k] : 0, 0) + '</div><div class="l">' + rEsc(String(k).replace(/_/g, ' ')) + '</div></div>';
+ }).join('') + '</div>';
+ if (!sources.length) {
+ h += '<div class="rs-banner warn">The report lists no sources.</div>';
+ } else if (sources.every(function(s) { return s && s.state === 'never_run'; })) {
+ h += '<div class="rs-banner warn">No collector capture is recorded in this database: every source reads never_run, which is not healthy. ' +
+ 'History accrues only while intel-collector runs against this cache database (WORLD_INTEL_CACHE_DB); a collector writing to another database does not show here.</div>';
+ }
+ states.forEach(function(state) {
+ var rows = sources.filter(function(s) { return s && s.state === state; });
+ if (!rows.length) return;
+ h += '<div class="sub">' + rEsc(String(state).replace(/_/g, ' ')) + ' (' + rNum(rows.length, 0) + ')</div>';
+ if (state === 'never_run') {
+ h += '<div class="tags">' + rows.map(function(s) {
+ return '<span class="tag">' + rEsc(s.source) + (s.in_collector_roster === false ? ' (not in roster)' : '') + '</span>';
+ }).join('') + '</div>';
+ return;
+ }
+ h += '<table class="dtable"><thead><tr><th>Source</th><th>Last attempt</th><th>Last success</th><th>Failures in a row</th><th>Overdue</th><th>Reason</th><th>Latest observed</th></tr></thead><tbody>' +
+ rows.map(rHealthRow).join('') + '</tbody></table>';
+ });
+ var ret = st.retention && typeof st.retention === 'object' ? st.retention : {};
+ h += '<div class="sub">Storage</div><div class="rs-note">' + rNum(st.captures, 0) + ' captures since ' + rEsc(st.first_capture_at || 'never') + '; ' +
+ rNum(st.payload_bodies, 0) + ' payload bodies (' + rNum(st.payload_bytes_compressed, 0) + ' bytes compressed, ' + rNum(st.payload_bytes_raw, 0) + ' raw); ' +
+ rNum(st.observations, 0) + ' observations in ' + rNum(st.series, 0) + ' series. Retention in days (0 keeps forever): ' +
+ rEsc(Object.keys(ret).map(function(k) { return k + ' ' + ret[k]; }).join(', ') || 'not stated') + '.</div>';
+ var cannot = rArr(d.cannot_prove);
+ if (cannot.length) h += '<div class="sub">What this report cannot prove</div>' + rList(cannot);
+ var ex = d.states_explained && typeof d.states_explained === 'object' ? d.states_explained : {};
+ var exKeys = Object.keys(ex);
+ if (exKeys.length) h += '<div class="sub">States</div>' + rList(exKeys.map(function(k) { return k + ': ' + ex[k]; }));
+ return h;
+}
+
+var RESEARCH_RENDERERS = { liquidity: renderLiquidity, situations: renderSituations, stories: renderTopStories, health: renderCollectionHealth };
+
+function rMarked(d) {
+ var h = '';
+ if (d.degraded === true) h += '<div class="rs-banner warn"><span class="rs-badge warn">DEGRADED</span> ' + rEsc(d.reason || 'the source marked this result degraded') + '</div>';
+ if (d._stale) h += '<div class="rs-banner warn"><span class="rs-badge warn">STALE COPY</span> served from cache' + (typeof d._stale_age_seconds === 'number' ? ', ' + rAge(d._stale_age_seconds) + ' old' : '') + '</div>';
+ return h;
+}
+
+// st: {inFlight, requestedAt, lastAttempt: {at, http, data, error}, lastGood}.
+// A panel never shows figures it has not received: before the first answer
+// it says so, a failure with no earlier copy shows the reason and the
+// payload's diagnostics only, and a failed refresh keeps the earlier copy
+// under a banner naming the failure and when that copy was loaded.
+function renderResearch(name, st) {
+ var render = RESEARCH_RENDERERS[name];
+ if (!render) return '<div class="rs-banner err">Unknown panel: ' + rEsc(name) + '</div>';
+ if (!st || (!st.lastAttempt && !st.inFlight)) {
+ return '<div class="rs-banner info">Not loaded yet: no request has been made for this panel, so nothing is shown.</div>';
+ }
+ if (!st.lastAttempt) {
+ return '<div class="rs-banner info">Loading: first request sent ' + rClock(st.requestedAt) + '. Nothing has loaded yet, so no figures are shown.</div>';
+ }
+ var a = st.lastAttempt, g = st.lastGood;
+ var status = a.http ? 'HTTP ' + a.http + ': ' : '';
+ if (a.error && !g) {
+ var diag = a.data && typeof a.data === 'object' && !Array.isArray(a.data) ? a.data : {};
+ return '<div class="rs-banner err"><span class="rs-badge err">UNAVAILABLE</span> ' + rEsc(status + a.error) +
+ '<br><span class="dim">Attempted ' + rClock(a.at) + '. Nothing has loaded for this query, so no figures are shown.</span></div>' +
+ (st.inFlight ? '<div class="rs-note">Retrying: request sent ' + rClock(st.requestedAt) + '.</div>' : '') + render(diag, true);
+ }
+ var head = a.error
+ ? '<div class="rs-banner warn"><span class="rs-badge warn">REFRESH FAILED</span> ' + rEsc(status + a.error) + ' (' + rClock(a.at) + ').<br>' +
+ 'Showing the copy loaded ' + rClock(g.at) + ', ' + rAge((a.at - g.at) / 1000) + ' before that attempt.</div>'
+ : '<div class="rs-note">Loaded ' + rClock(g.at) + '; refreshes every ' + RESEARCH_REFRESH_MS / 1000 + ' s while this panel is open.</div>';
+ if (st.inFlight) head += '<div class="rs-note">Refreshing: request sent ' + rClock(st.requestedAt) + '.</div>';
+ return head + rMarked(g.data) + render(g.data, false);
+}
+// research-render:end
+
+// research-panel:begin
+// Controller: opens the panel, loads the active tab from its route when it
+// is opened, refreshes it every RESEARCH_REFRESH_MS while open and visible,
+// and drops answers to a query the controls have since changed.
+var RESEARCH_URLS = {
+ liquidity: function() { return '/api/liquidity-board?sort=' + encodeURIComponent($('#rsLiqSort').value); },
+ situations: function() {
+ return '/api/situations?hours=' + encodeURIComponent($('#rsSitHours').value) + '&sort=' + encodeURIComponent($('#rsSitSort').value) +
+ '&kind=' + encodeURIComponent($('#rsSitKind').value);
+ },
+ stories: function() { return '/api/top-stories?hours=' + encodeURIComponent($('#rsTopHours').value) + '&min_outlets=' + encodeURIComponent($('#rsTopOutlets').value); },
+ health: function() { return '/api/collection-health'; }
+};
+var researchState = {};
+var researchActive = 'liquidity';
+
+function researchIsOpen() { return !$('#researchPanel').hidden; }
+
+function researchRender() {
+ var name = researchActive, html;
+ $$('.rs-tab').forEach(function(b) { b.classList.toggle('active', b.getAttribute('data-rs-tab') === name); });
+ $$('.rs-ctl').forEach(function(c) { c.classList.toggle('on', c.getAttribute('data-rs-for') === name); });
+ try { html = renderResearch(name, researchState[name]); }
+ catch (e) {
+ console.error('research panel render failed:', e);
+ html = '<div class="rs-banner err">This panel failed to render: ' + rEsc(String(e)) + '</div>';
+ }
+ $('#rsBody').innerHTML = safe(html);
+}
+
+function researchLoad(name) {
+ var url = RESEARCH_URLS[name]();
+ var st = researchState[name];
+ if (!st || st.url !== url) {
+ // A different question: an answer to the previous one is not shown for it.
+ st = researchState[name] = { seq: st ? st.seq : 0, url: url };
+ }
+ if (st.inFlight) return;
+ var seq = ++st.seq;
+ st.inFlight = true;
+ st.requestedAt = Date.now();
+ if (name === researchActive && researchIsOpen()) researchRender();
+ fetch(url, { cache: 'no-store' })
+ .then(function(r) {
+ return r.text().then(function(text) {
+ var data = null;
+ try { data = JSON.parse(text); } catch (e) { data = null; }
+ return { http: r.status, data: data, error: researchAttemptError(r.status, data) };
+ });
+ })
+ .catch(function(e) { return { http: null, data: null, error: 'request failed: ' + e }; })
+ .then(function(res) {
+ if (researchState[name] !== st || st.seq !== seq) return;
+ st.inFlight = false;
+ st.lastAttempt = { at: Date.now(), http: res.http, data: res.data, error: res.error };
+ if (!res.error) st.lastGood = st.lastAttempt;
+ if (name === researchActive && researchIsOpen()) researchRender();
+ });
+}
+
+function researchDue(name) {
+ var st = researchState[name];
+ if (!st || st.url !== RESEARCH_URLS[name]()) return true;
+ if (st.inFlight) return false;
+ return !st.lastAttempt || Date.now() - st.lastAttempt.at >= RESEARCH_REFRESH_MS;
+}
+
+function researchShow(name) {
+ researchActive = name;
+ researchRender();
+ if (researchDue(name)) researchLoad(name);
+}
+
+function researchToggle(open) {
+ var panel = $('#researchPanel');
+ panel.hidden = open === undefined ? !panel.hidden : !open;
+ $('#researchToggle').classList.toggle('active', !panel.hidden);
+ if (!panel.hidden) researchShow(researchActive);
+}
+
+$('#researchToggle').addEventListener('click', function() { researchToggle(); });
+$('#rsClose').addEventListener('click', function() { researchToggle(false); });
+$('#rsRefresh').addEventListener('click', function() { researchLoad(researchActive); });
+$$('.rs-tab').forEach(function(b) { b.addEventListener('click', function() { researchShow(b.getAttribute('data-rs-tab')); }); });
+$$('.rs-ctl select').forEach(function(s) {
+ s.addEventListener('change', function() { researchLoad(s.closest('.rs-ctl').getAttribute('data-rs-for')); });
+});
+// Capture phase: runs before the map shortcuts, so Esc closes this panel
+// too while the help overlay, when open, keeps Esc for itself.
+document.addEventListener('keydown', function(e) {
+ if (e.metaKey || e.ctrlKey || e.altKey) return;
+ if (e.key === 'Escape') { if (!_helpOverlay && researchIsOpen()) researchToggle(false); return; }
+ var tag = e.target.tagName;
+ if (tag === 'INPUT' || tag === 'TEXTAREA' || tag === 'SELECT') return;
+ if (e.key === 'p') researchToggle();
+}, true);
+// Keys typed into a panel control pick an option; they must not reach the map shortcuts.
+$('#researchPanel').addEventListener('keydown', function(e) {
+ if (e.target.tagName === 'SELECT' && e.key !== 'Escape') e.stopPropagation();
+});
+setInterval(function() {
+ if (researchIsOpen() && !document.hidden && researchDue(researchActive)) researchLoad(researchActive);
+}, 5000);
+// research-panel:end
</script>
</body>
</html>
diff --git a/src/world_intel_mcp/tests/test_dashboard_research.py b/src/world_intel_mcp/tests/test_dashboard_research.py
new file mode 100644
index 0000000..f85449b
--- /dev/null
+++ b/src/world_intel_mcp/tests/test_dashboard_research.py
@@ -0,0 +1,871 @@
+"""Tests for the dashboard research panels (ROADMAP Phase 29, wave 3).
+
+Four routes back four panels: /api/liquidity-board, /api/situations,
+/api/top-stories and /api/collection-health. Each calls the function the
+matching MCP tool calls. The route tests stub that function (collection
+health reads a real temporary history store written with ``record_cycle``)
+and pin the JSON each route answers: the payload passed through unchanged
+on success, partial failures left inside it, a whole-call failure as a 503
+with the payload or reason intact, a timeout as a 504, bad parameters as a
+400 before anything is fetched.
+
+The page's renderers are pure functions between the ``research-render``
+markers in index.html. The JS tests run that block under node, deno or bun
+and parse the markup they return: remote strings escaped, links limited to
+http and https, a never-loaded or failed panel showing no figures, the
+liquidity board keeping units and keeping amounts apart from flows, and the
+collection health panel listing problems and its cannot_prove text.
+
+Uses Starlette's TestClient without the context manager so lifespan
+(default-path Cache, vector store) never runs.
+
+Gaps / not covered: the controller half of the page (fetch, tabs, refresh
+timer, Esc) runs only in a browser and is checked by driving the dashboard
+live, not here; the JS tests skip when no JavaScript runtime is on PATH; no
+route is load tested.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import copy
+import json
+import logging
+import re
+import shutil
+import subprocess
+import sys
+import types
+from datetime import datetime, timezone
+from html.parser import HTMLParser
+from pathlib import Path
+
+import pytest
+from starlette.testclient import TestClient
+
+from world_intel_mcp import history
+from world_intel_mcp.analysis import news_signals
+from world_intel_mcp.dashboard import app as dashboard_app
+from world_intel_mcp.history import HistoryStore, SourceOutcome
+from world_intel_mcp.sources import liquidity, news
+
+_INDEX = Path(dashboard_app.__file__).with_name("index.html")
+_METHOD = liquidity.NET_LIQUIDITY_METHOD
+
+
+def client() -> TestClient:
+ return TestClient(dashboard_app.app)
+
+
+def fetcher(monkeypatch: pytest.MonkeyPatch) -> object:
+ sentinel = object()
+ monkeypatch.setattr(dashboard_app, "_ensure_fetcher", lambda: sentinel)
+ return sentinel
+
+
+def _stub(monkeypatch, module, name, result=None, exc=None, delay=None) -> list:
+ """Replace ``module.name`` with a coroutine that records its call and
+ returns a deep copy of ``result`` (or raises ``exc``)."""
+ calls: list = []
+
+ async def fake(fetcher, **kwargs):
+ calls.append((fetcher, kwargs))
+ if delay:
+ await asyncio.sleep(delay)
+ if exc is not None:
+ raise exc
+ return copy.deepcopy(result)
+
+ monkeypatch.setattr(module, name, fake)
+ return calls
+
+
+# ---------------------------------------------------------------------------
+# Payloads in the shapes the functions return (measured live 2026-10-17)
+# ---------------------------------------------------------------------------
+
+
+def _row(indicator, label, units, value, previous, change, pct, measure="amount_outstanding", **extra):
+ return {
+ "indicator": indicator, "label": label, "dataset": f"FRED {indicator.upper()}",
+ "source": "fred-graph", "measure": measure, "frequency": "weekly", "units": units,
+ "latest_value": value, "status": None, "observed_at": "2026-10-08",
+ "period_end": "2026-10-08", "previous_value": previous,
+ "previous_observed_at": "2026-10-01", "change": change, "change_pct": pct,
+ "change_basis": "vs previous observation",
+ "freshness": {"latest_age_days": 9, "expected_max_age_days": 14, "stale": False},
+ **extra,
+ }
+
+
+_BOARD = {
+ "sort": "biggest_move",
+ "ranking_rule": "biggest_move ranks absolute changes in billions of each row currency",
+ "count": 2,
+ "indicators": [
+ _row("fed_total_assets", "Fed total assets", "USD millions", 6_600_000.0, 6_612_345.0, -12_345.0, -0.1867),
+ _row(
+ "reverse_repo", "Overnight reverse repo", "USD billions", 210.5, 205.0, 5.5, 2.6829,
+ served_from_stale_cache=True, cache_age_seconds=4000,
+ ),
+ ],
+ "unavailable": [{"indicator": "euro_area_m1", "dataset": "ECB BSI", "error": "ECB HTTP 503"}],
+ "blocked": copy.deepcopy(liquidity.BLOCKED_SOURCES),
+ "net_liquidity_formula": liquidity.NET_LIQUIDITY_FORMULA,
+ "note": "M1 is contained in M2 and M2 in M3.",
+ "sources": ["fred-graph", "ecb-data-portal"],
+ "data_gaps": ["euro_area_m1: ECB HTTP 503"],
+ "timestamp": "2026-10-17T19:04:08+00:00",
+}
+
+_BOARD_OUTAGE = {
+ **_BOARD,
+ "count": 0,
+ "indicators": [],
+ "unavailable": [
+ {"indicator": "fed_total_assets", "dataset": "FRED WALCL", "error": "FRED WALCL: HTTP 503"},
+ {"indicator": "net_liquidity", "error": "a component failed; never computed with a zero term"},
+ ],
+ "data_gaps": ["FRED WALCL: HTTP 503", "net liquidity not computed: a component series failed"],
+ "error": "liquidity board unavailable: every source failed",
+}
+
+_WINDOW = {
+ "hours": 24.0,
+ "current": {"start": "2026-10-16T19:04:08Z", "end": "2026-10-17T19:04:08Z"},
+ "prior": {"start": "2026-10-15T19:04:08Z", "end": "2026-10-16T19:04:08Z"},
+}
+_COVERAGE = {
+ "items_read": 2632, "items_current_window": 632, "items_current_window_comparable": 185,
+ "items_prior_window": 487, "undated_items": 50, "future_dated_items": 0, "feeds_ok": 94,
+ "feeds_failed": ["CFR", "HRW"], "feeds_comparable": 55, "feeds_not_comparable": ["SCMP"],
+ "feeds_stale": [],
+}
+
+_SITUATIONS = {
+ "situations": [
+ {
+ "kind": "between", "key": "IRN|ISR",
+ "countries": [{"iso3": "IRN", "name": "Iran"}, {"iso3": "ISR", "name": "Israel"}],
+ "articles": 12, "hostile_articles": 7, "hostile_share": 0.583,
+ "by_category": {"threat": 4, "sanction": 0, "force": 3, "criticism": 0},
+ "outlet_count": 8, "current_comparable": 9, "prior": 4, "change": 5, "change_ratio": 2.0,
+ "sample_headlines": [
+ {"title": "Israel and Iran trade warnings", "outlet": "BBC",
+ "link": "https://www.bbc.co.uk/news/1", "published": "2026-10-17T10:00:00Z",
+ "hostility": ["threat"]},
+ ],
+ }
+ ],
+ "count": 1, "sort": "covered", "kind": "all", "min_articles": 2,
+ "source": "rss-aggregator+news-signals", "timestamp": "2026-10-17T19:04:08Z",
+ "window": _WINDOW, "coverage": _COVERAGE,
+ "definitions": {"between": "a headline naming two or more countries"},
+ "method": "RSS headlines from every configured feed.",
+ "data_gaps": ["2 RSS feeds failed or timed out and are absent: CFR, HRW"],
+}
+
+# What fetch_situations answers when every RSS feed failed (its own shape).
+_SITUATIONS_DEGRADED = {
+ "situations": [], "count": 0, "sort": "covered", "kind": "all",
+ "source": "rss-aggregator+news-signals", "timestamp": "2026-10-17T19:04:08Z",
+ "error": "RSS feed read failed", "degraded": True, "reason": "rss_unavailable",
+ "data_gaps": ["every RSS feed failed: no coverage could be read, which is not the same as no coverage"],
+}
+
+_TOP = {
+ "stories": [
+ {
+ "headline": "Pope returns to Vatican after surgery", "outlet_count": 4,
+ "outlets": ["BBC", "NDTV", "SCMP", "Al Jazeera"], "articles": 5,
+ "keywords": ["pope", "surgery"], "first_seen": "2026-10-17T10:26:27Z",
+ "last_seen": "2026-10-17T17:37:52Z",
+ "items": [{"title": "Pope returns to Vatican", "outlet": "BBC",
+ "link": "https://www.bbc.co.uk/news/2", "published": "2026-10-17T17:37:52Z"}],
+ }
+ ],
+ "count": 1,
+ "attention": [{"type": "country", "name": "Yemen", "iso3": "YEM", "current_comparable": 4,
+ "prior": 0, "change": 4, "change_ratio": 5.0}],
+ "min_outlets": 3, "source": "rss-aggregator+news-signals", "timestamp": "2026-10-17T19:04:08Z",
+ "window": _WINDOW, "coverage": _COVERAGE,
+ "definitions": {"outlet": "a publisher"}, "method": "Clustering is lexical.",
+ "data_gaps": ["2 RSS feeds failed or timed out and are absent: CFR, HRW"],
+}
+
+
+# ---------------------------------------------------------------------------
+# /api/liquidity-board
+# ---------------------------------------------------------------------------
+
+
+def test_liquidity_board_passes_payload_through_with_the_method_block(client, fetcher, monkeypatch) -> None:
+ calls = _stub(monkeypatch, liquidity, "fetch_liquidity_board", _BOARD)
+ r = client.get("/api/liquidity-board")
+ assert r.status_code == 200
+ assert r.headers.get("access-control-allow-origin") == "*"
+ # Partial outage (unavailable), stale-cache row and blocked list all stay in the payload.
+ assert r.json() == {**_BOARD, "net_liquidity_method": _METHOD}
+ assert calls == [(fetcher, {"sort": "biggest_move"})]
+
+
+def test_liquidity_board_sort_is_passed_and_validated(client, fetcher, monkeypatch) -> None:
+ calls = _stub(monkeypatch, liquidity, "fetch_liquidity_board", _BOARD)
+ assert client.get("/api/liquidity-board?sort=newest").status_code == 200
+ assert calls == [(fetcher, {"sort": "newest"})]
+ r = client.get("/api/liquidity-board?sort=pct")
+ assert r.status_code == 400
+ assert r.json() == {"error": "sort must be one of biggest_move, newest, got 'pct'"}
+ assert len(calls) == 1 # nothing fetched for a refused request
+
+
+def test_liquidity_board_total_outage_is_503_with_reasons_intact(client, fetcher, monkeypatch) -> None:
+ _stub(monkeypatch, liquidity, "fetch_liquidity_board", _BOARD_OUTAGE)
+ r = client.get("/api/liquidity-board")
+ assert r.status_code == 503
+ assert r.json() == {**_BOARD_OUTAGE, "net_liquidity_method": _METHOD}
+
+
+def test_liquidity_board_exception_is_503_with_reason(client, fetcher, monkeypatch) -> None:
+ _stub(monkeypatch, liquidity, "fetch_liquidity_board", exc=RuntimeError("socket closed"))
+ r = client.get("/api/liquidity-board")
+ assert r.status_code == 503
+ assert r.json() == {
+ "error": "liquidity board failed: RuntimeError: socket closed",
+ "net_liquidity_method": _METHOD,
+ }
+
+
+def test_liquidity_board_timeout_is_504(client, fetcher, monkeypatch) -> None:
+ monkeypatch.setattr(dashboard_app, "_RESEARCH_TIMEOUT_S", 0.05)
+ _stub(monkeypatch, liquidity, "fetch_liquidity_board", _BOARD, delay=5)
+ r = client.get("/api/liquidity-board")
+ assert r.status_code == 504
+ assert r.json() == {
+ "error": "liquidity board timed out after 0.05 s",
+ "timeout": True,
+ "net_liquidity_method": _METHOD,
+ }
+
+
+def test_timeout_raised_inside_the_function_is_a_failure_not_the_budget(client, fetcher, monkeypatch) -> None:
+ _stub(monkeypatch, liquidity, "fetch_liquidity_board", exc=TimeoutError("ECB read timed out"))
+ r = client.get("/api/liquidity-board")
+ assert r.status_code == 503
+ assert r.json() == {
+ "error": "liquidity board failed: TimeoutError: ECB read timed out",
+ "net_liquidity_method": _METHOD,
+ }
+
+
+# ---------------------------------------------------------------------------
+# /api/situations and /api/top-stories
+# ---------------------------------------------------------------------------
+
+
+def test_situations_pass_through_with_default_arguments(client, fetcher, monkeypatch) -> None:
+ calls = _stub(monkeypatch, news_signals, "fetch_situations", _SITUATIONS)
+ r = client.get("/api/situations")
+ assert r.status_code == 200
+ assert r.headers.get("access-control-allow-origin") == "*"
+ assert r.json() == _SITUATIONS
+ assert calls == [(fetcher, {"hours": 24.0, "sort": "covered", "kind": "all", "limit": 20, "min_articles": 2})]
+
+
+def test_situations_arguments_are_parsed(client, fetcher, monkeypatch) -> None:
+ calls = _stub(monkeypatch, news_signals, "fetch_situations", _SITUATIONS)
+ r = client.get("/api/situations?hours=6&sort=hostile&kind=between&limit=5&min_articles=3")
+ assert r.status_code == 200
+ assert calls == [(fetcher, {"hours": 6.0, "sort": "hostile", "kind": "between", "limit": 5, "min_articles": 3})]
+
+
+ ("query", "error"),
+ [
+ ("hours=0", "hours must be between 1 and 168, got '0'"),
+ ("hours=169", "hours must be between 1 and 168, got '169'"),
+ ("hours=nan", "hours must be between 1 and 168, got 'nan'"),
+ ("hours=abc", "hours must be a number between 1 and 168, got 'abc'"),
+ ("sort=loudest", "sort must be one of covered, hostile, change, got 'loudest'"),
+ ("kind=near", "kind must be one of all, between, inside, got 'near'"),
+ ("limit=0", "limit must be between 1 and 100, got '0'"),
+ ("limit=2.5", "limit must be a number between 1 and 100, got '2.5'"),
+ ("min_articles=0", "min_articles must be between 1 and 1000, got '0'"),
+ ],
+)
+def test_situations_bad_arguments_are_400_before_any_fetch(client, fetcher, monkeypatch, query, error) -> None:
+ calls = _stub(monkeypatch, news_signals, "fetch_situations", _SITUATIONS)
+ r = client.get(f"/api/situations?{query}")
+ assert r.status_code == 400
+ assert r.json() == {"error": error}
+ assert calls == []
+
+
+def test_situations_every_feed_failed_is_503_not_an_empty_list(client, fetcher, monkeypatch) -> None:
+ _stub(monkeypatch, news_signals, "fetch_situations", _SITUATIONS_DEGRADED)
+ r = client.get("/api/situations")
+ assert r.status_code == 503
+ assert r.json() == _SITUATIONS_DEGRADED
+
+
+def test_situations_quiet_window_is_200_with_its_coverage(client, fetcher, monkeypatch) -> None:
+ empty = {**_SITUATIONS, "situations": [], "count": 0}
+ _stub(monkeypatch, news_signals, "fetch_situations", empty)
+ r = client.get("/api/situations")
+ assert r.status_code == 200
+ assert r.json() == empty
+
+
+def test_situations_route_surfaces_the_real_function_outage(client, fetcher, monkeypatch) -> None:
+ """Not a stub of the analysis: the real fetch_situations runs over a
+ feed reader that failed, and its own degraded answer reaches the page."""
+
+ async def dead_feed(fetcher, **kwargs):
+ return {"error": "all 119 RSS feeds failed", "degraded": True, "reason": "rss_unavailable"}
+
+ monkeypatch.setattr(news, "fetch_news_feed", dead_feed)
+ r = client.get("/api/situations")
+ assert r.status_code == 503
+ body = r.json()
+ assert body["error"] == "all 119 RSS feeds failed"
+ assert body["degraded"] is True and body["reason"] == "rss_unavailable"
+
+
+def test_top_stories_pass_through_with_default_arguments(client, fetcher, monkeypatch) -> None:
+ calls = _stub(monkeypatch, news_signals, "fetch_top_stories", _TOP)
+ r = client.get("/api/top-stories")
+ assert r.status_code == 200
+ assert r.json() == _TOP
+ assert calls == [(fetcher, {"hours": 24.0, "min_outlets": 3, "limit": 10, "attention_limit": 10})]
+
+
+def test_top_stories_arguments_are_parsed_and_validated(client, fetcher, monkeypatch) -> None:
+ calls = _stub(monkeypatch, news_signals, "fetch_top_stories", _TOP)
+ assert client.get("/api/top-stories?hours=72&min_outlets=5&limit=3&attention_limit=4").status_code == 200
+ assert calls == [(fetcher, {"hours": 72.0, "min_outlets": 5, "limit": 3, "attention_limit": 4})]
+ for query, error in (
+ ("min_outlets=0", "min_outlets must be between 1 and 50, got '0'"),
+ ("attention_limit=51", "attention_limit must be between 1 and 50, got '51'"),
+ ("hours=x", "hours must be a number between 1 and 168, got 'x'"),
+ ):
+ r = client.get(f"/api/top-stories?{query}")
+ assert r.status_code == 400, query
+ assert r.json() == {"error": error}
+ assert len(calls) == 1
+
+
+def test_top_stories_outage_exception_and_timeout(client, fetcher, monkeypatch) -> None:
+ degraded = {"stories": [], "count": 0, "attention": [], "min_outlets": 3,
+ "error": "RSS feed read failed", "degraded": True, "reason": "rss_unavailable",
+ "data_gaps": ["every RSS feed failed"]}
+ _stub(monkeypatch, news_signals, "fetch_top_stories", degraded)
+ r = client.get("/api/top-stories")
+ assert (r.status_code, r.json()) == (503, degraded)
+
+ _stub(monkeypatch, news_signals, "fetch_top_stories", exc=ValueError("bad cluster"))
+ r = client.get("/api/top-stories")
+ assert (r.status_code, r.json()) == (503, {"error": "top stories failed: ValueError: bad cluster"})
+
+ monkeypatch.setattr(dashboard_app, "_RESEARCH_TIMEOUT_S", 0.05)
+ _stub(monkeypatch, news_signals, "fetch_top_stories", _TOP, delay=5)
+ r = client.get("/api/top-stories")
+ assert (r.status_code, r.json()) == (504, {"error": "top stories timed out after 0.05 s", "timeout": True})
+
+
+def test_top_stories_empty_result_is_200(client, fetcher, monkeypatch) -> None:
+ empty = {**_TOP, "stories": [], "count": 0, "attention": []}
+ _stub(monkeypatch, news_signals, "fetch_top_stories", empty)
+ r = client.get("/api/top-stories")
+ assert (r.status_code, r.json()) == (200, empty)
+
+
+# ---------------------------------------------------------------------------
+# /api/collection-health
+# ---------------------------------------------------------------------------
+
+_ROSTER = ["quakes", "fires", "storms", "ports", "tides"]
+_T0 = 1_760_700_000.0 # last cycle
+_CHECK = _T0 + 120.0 # when the route reads the store
+
+
+def _populate(db_path: Path) -> None:
+ store = HistoryStore(db_path)
+ try:
+ store.record_cycle(
+ [SourceOutcome("ports", _T0 - 3600, 0.4, payload={"ports": [{"name": "Rotterdam"}]})],
+ now=_T0 - 3600,
+ )
+ store.record_cycle(
+ [SourceOutcome("fires", _T0 - 300, 2.0, payload={"fires": [{"lat": 1.0}]}),
+ SourceOutcome("legacy_feed", _T0 - 300, 0.1, payload={"items": [{"id": 1}]})],
+ now=_T0 - 300,
+ )
+ store.record_cycle(
+ [SourceOutcome("quakes", _T0, 0.5, payload={"earthquakes": [{"magnitude": 5.1}]}),
+ SourceOutcome("fires", _T0, 12.0, error="HTTP 503 from FIRMS"),
+ SourceOutcome("storms", _T0, 0.3, payload={"storms": []})],
+ now=_T0,
+ )
+ finally:
+ store.close()
+
+
+def _expected(db_path: Path, source: str | None = None) -> dict:
+ store = HistoryStore(db_path)
+ try:
+ return store.collection_health(_ROSTER, source=source, now=_CHECK)
+ finally:
+ store.close()
+
+
+def pinned_history(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Default cadence and retention, and the clock collection_health reads
+ (time.time() when the route calls it) pinned to _CHECK."""
+ for var in ("WORLD_INTEL_HISTORY_CADENCE_S", "WORLD_INTEL_HISTORY_CADENCES",
+ "WORLD_INTEL_HISTORY_PAYLOAD_DAYS", "WORLD_INTEL_HISTORY_CAPTURE_DAYS",
+ "WORLD_INTEL_HISTORY_DAYS"):
+ monkeypatch.delenv(var, raising=False)
+ monkeypatch.setattr(history, "time", types.SimpleNamespace(time=lambda: _CHECK))
+ monkeypatch.setattr(dashboard_app, "_collector_roster", lambda: list(_ROSTER))
+
+
+def health_db(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, pinned_history) -> Path:
+ db_path = tmp_path / "cache.db"
+ monkeypatch.setattr(dashboard_app, "_history_db_path", lambda: db_path)
+ return db_path
+
+
+def test_collection_health_reads_the_store_and_returns_the_report_unchanged(client, health_db) -> None:
+ _populate(health_db)
+ r = client.get("/api/collection-health")
+ assert r.status_code == 200
+ assert r.headers.get("access-control-allow-origin") == "*"
+ body = r.json()
+ assert body == _expected(health_db)
+ states = {s["source"]: s["state"] for s in body["sources"]}
+ assert states == {
+ "quakes": "ok", "fires": "failing", "storms": "degraded",
+ "ports": "not_attempted_recently", "tides": "never_run", "legacy_feed": "ok",
+ }
+ by = {s["source"]: s for s in body["sources"]}
+ assert by["fires"]["consecutive_failures"] == 1
+ assert by["fires"]["last_reason"] == "HTTP 503 from FIRMS"
+ assert by["ports"]["overdue"] is True
+ assert by["legacy_feed"]["in_collector_roster"] is False
+ assert by["tides"]["attempts"] == 0
+ assert body["cannot_prove"] and body["storage"]["db_path"] == str(health_db)
+
+
+def test_collection_health_one_source_and_unknown_source(client, health_db) -> None:
+ _populate(health_db)
+ r = client.get("/api/collection-health?source=fires")
+ assert r.status_code == 200
+ assert r.json() == _expected(health_db, source="fires")
+ assert [s["source"] for s in r.json()["sources"]] == ["fires"]
+ r = client.get("/api/collection-health?source=nope")
+ assert r.status_code == 404
+ assert r.json() == _expected(health_db, source="nope")
+ assert "known_sources" in r.json()
+
+
+def test_collection_health_store_never_written_reads_never_run(client, health_db) -> None:
+ assert not health_db.exists()
+ r = client.get("/api/collection-health")
+ assert r.status_code == 200
+ body = r.json()
+ assert body == _expected(health_db)
+ assert {s["state"] for s in body["sources"]} == {"never_run"}
+ assert body["state_counts"] == {"never_run": len(_ROSTER)}
+ assert body["storage"]["captures"] == 0
+
+
+def test_collection_health_corrupt_store_is_503(client, health_db) -> None:
+ health_db.write_bytes(b"this is not a sqlite database " * 64)
+ r = client.get("/api/collection-health")
+ assert r.status_code == 503
+ body = r.json()
+ assert set(body) == {"error", "db_path"}
+ assert body["error"].startswith("history store unavailable: DatabaseError: ")
+ assert body["db_path"] == str(health_db)
+
+
+def test_collection_health_store_that_cannot_be_created_is_503(client, tmp_path, monkeypatch, pinned_history) -> None:
+ blocker = tmp_path / "not-a-dir"
+ blocker.write_text("a file where the cache directory should be")
+ db_path = blocker / "cache.db"
+ monkeypatch.setattr(dashboard_app, "_history_db_path", lambda: db_path)
+ r = client.get("/api/collection-health")
+ assert r.status_code == 503
+ body = r.json()
+ assert set(body) == {"error", "db_path"}
+ assert body["error"].startswith("history store unavailable: ")
+ assert body["db_path"] == str(db_path)
+
+
+def test_collection_health_opens_the_dashboard_cache_database(client, tmp_path, monkeypatch, pinned_history) -> None:
+ """Unpatched _history_db_path: the store is the one the dashboard's own
+ Fetcher cache uses, the database the collector writes in the same
+ environment."""
+ db_path = tmp_path / "dash-cache.db"
+ _populate(db_path)
+ monkeypatch.setattr(
+ dashboard_app, "_ensure_fetcher",
+ lambda: types.SimpleNamespace(cache=types.SimpleNamespace(db_path=db_path)),
+ )
+ assert dashboard_app._history_db_path() == db_path
+ r = client.get("/api/collection-health")
+ assert r.status_code == 200
+ assert r.json() == _expected(db_path)
+ assert {s["source"]: s["state"] for s in r.json()["sources"]}["fires"] == "failing"
+
+
+def test_roster_is_the_collector_sources() -> None:
+ from world_intel_mcp.collector import SOURCES
+
+ assert dashboard_app._collector_roster() == [name for name, _, _, _ in SOURCES]
+
+
+def test_reading_the_roster_leaves_root_logging_as_it_was(monkeypatch) -> None:
+ """Importing the collector runs logging.basicConfig at INFO. In the
+ dashboard process nothing configures the root logger, so without the
+ guard the first collection-health request would start printing every
+ upstream request line. Force a fresh import against an unconfigured
+ root logger and check nothing was left on it."""
+ import world_intel_mcp
+ from world_intel_mcp import collector
+
+ root = logging.getLogger()
+ monkeypatch.setattr(root, "handlers", [])
+ monkeypatch.setattr(root, "level", logging.WARNING)
+ monkeypatch.delitem(sys.modules, "world_intel_mcp.collector")
+ monkeypatch.setattr(world_intel_mcp, "collector", collector)
+ roster = dashboard_app._collector_roster()
+ assert root.handlers == []
+ assert root.level == logging.WARNING
+ assert roster == [name for name, _, _, _ in collector.SOURCES]
+
+
+# ---------------------------------------------------------------------------
+# Page: static checks (no JavaScript runtime needed)
+# ---------------------------------------------------------------------------
+
+
+def _section(src: str, marker: str) -> str:
+ return src[src.index(f"// {marker}:begin"):src.index(f"// {marker}:end")]
+
+
+def test_research_markup_is_assigned_through_the_sanitizer() -> None:
+ src = _INDEX.read_text()
+ controller = _section(src, "research-panel")
+ assignments = re.findall(r"innerHTML\s*=\s*([^;]+);", controller)
+ assert assignments, "no innerHTML assignment found; the extraction is stale"
+ assert all(a.startswith("safe(") for a in assignments), assignments
+
+
+def test_renderers_touch_no_dom_or_network() -> None:
+ block = _section(_INDEX.read_text(), "research-render")
+ for banned in ("document.", "window.", "innerHTML", "fetch(", "$(", "localStorage", "eval("):
+ assert banned not in block, banned
+
+
+def test_page_calls_the_four_research_routes() -> None:
+ controller = _section(_INDEX.read_text(), "research-panel")
+ paths = {r.path for r in dashboard_app.app.routes}
+ for path in ("/api/liquidity-board", "/api/situations", "/api/top-stories", "/api/collection-health"):
+ assert f"'{path}" in controller, path
+ assert path in paths, path
+
+
+# ---------------------------------------------------------------------------
+# Page: the renderers, run under a JavaScript runtime
+# ---------------------------------------------------------------------------
+
+_T_MS = 1_760_727_848_000
+_CLOCK = datetime.fromtimestamp(_T_MS / 1000, tz=timezone.utc).strftime("%H:%M:%SZ")
+_PAGE_TAGS = {"div", "span", "table", "thead", "tbody", "tr", "th", "td", "a", "br"}
+_PAGE_ATTRS = {"class", "href", "data-external"}
+_HOSTILE = '<img src=x onerror="alert(1)">'
+_SCRIPT = "<script>alert(2)</script>"
+
+
+def _js_command() -> list[str]:
+ for name, args in (("node", []), ("deno", ["run", "--quiet", "--no-prompt"]), ("bun", ["run"])):
+ path = shutil.which(name)
+ if path:
+ return [path, *args]
+ pytest.skip("no JavaScript runtime (node, deno or bun) on PATH")
+
+
+def _js(tmp_path: Path, calls: list) -> list:
+ """Run the page's render block; each call is [function name, *args]."""
+ script = tmp_path / "research_render.mjs"
+ script.write_text(
+ _section(_INDEX.read_text(), "research-render")
+ + "\nvar FNS = { rEsc: rEsc, rHref: rHref, renderResearch: renderResearch,"
+ " researchAttemptError: researchAttemptError };\n"
+ + "var CALLS = " + json.dumps(calls) + ";\n"
+ + "console.log(JSON.stringify(CALLS.map(function(c) { return FNS[c[0]].apply(null, c.slice(1)); })));\n"
+ )
+ proc = subprocess.run(
+ [*_js_command(), str(script)], capture_output=True, text=True, timeout=60, check=False
+ )
+ assert proc.returncode == 0, proc.stderr
+ return json.loads(proc.stdout)
+
+
+def _norm(text: str) -> str:
+ return re.sub(r"\s+", " ", text).strip()
+
+
+class _Markup(HTMLParser):
+ """Tags, attributes, hrefs and the visible text (entities decoded,
+ whitespace collapsed) of a rendered panel."""
+
+ def __init__(self, html: str) -> None:
+ super().__init__(convert_charrefs=True)
+ self.tags: set[str] = set()
+ self.attrs: set[str] = set()
+ self.hrefs: list[str] = []
+ self._text: list[str] = []
+ self.feed(html)
+ self.close()
+ self.text = _norm(" ".join(self._text))
+
+ def handle_starttag(self, tag, attrs) -> None:
+ self.tags.add(tag)
+ for name, value in attrs:
+ self.attrs.add(name)
+ if name == "href":
+ self.hrefs.append(value)
+
+ def handle_data(self, data) -> None:
+ self._text.append(data)
+
+
+def _assert_inert(html: str) -> _Markup:
+ m = _Markup(html)
+ assert m.tags <= _PAGE_TAGS, m.tags - _PAGE_TAGS
+ assert m.attrs <= _PAGE_ATTRS, m.attrs - _PAGE_ATTRS
+ assert all(h.startswith(("http://", "https://")) for h in m.hrefs), m.hrefs
+ return m
+
+
+def _loaded(data: dict, http: int = 200) -> dict:
+ attempt = {"at": _T_MS, "http": http, "data": data, "error": None}
+ return {"lastAttempt": attempt, "lastGood": attempt}
+
+
+def _failed(data, error: str, http: int | None = 503) -> dict:
+ return {"lastAttempt": {"at": _T_MS, "http": http, "data": data, "error": error}}
+
+
+def _poison(value, mark: str):
+ """Every string in a payload, prefixed with markup."""
+ if isinstance(value, str):
+ return mark + value
+ if isinstance(value, list):
+ return [_poison(v, mark) for v in value]
+ if isinstance(value, dict):
+ return {k: _poison(v, mark) for k, v in value.items()}
+ return value
+
+
+def test_escape_helper_and_link_filter(tmp_path) -> None:
+ out = _js(tmp_path, [
+ ["rEsc", "<a href=\"x\">'&"],
+ ["rEsc", None],
+ ["rHref", "https://example.org/a?b=1&c=2"],
+ ["rHref", "javascript:alert(1)"],
+ ["rHref", " JAVASCRIPT:alert(1)"],
+ ["rHref", "data:text/html,<script>alert(1)</script>"],
+ ["rHref", "//evil.example/x"],
+ ["rHref", "https://example.org/\" onmouseover=\"alert(1)"],
+ ])
+ assert out == [
+ "&lt;a href=&quot;x&quot;&gt;&#39;&amp;", "",
+ "https://example.org/a?b=1&c=2", None, None, None, None, None,
+ ]
+
+
+def test_headline_markup_renders_as_text_not_markup(tmp_path) -> None:
+ top = copy.deepcopy(_TOP)
+ story = top["stories"][0]
+ story["headline"] = _HOSTILE
+ story["outlets"] = ["<b>Evil</b>"]
+ story["keywords"] = [_SCRIPT]
+ story["items"] = [{"title": _HOSTILE, "outlet": _SCRIPT, "link": "javascript:alert(3)", "published": "x"}]
+ top["attention"][0]["name"] = _SCRIPT
+ sit = copy.deepcopy(_SITUATIONS)
+ sit["situations"][0]["sample_headlines"][0].update(title=_HOSTILE, link="javascript:alert(4)")
+ sit["situations"][0]["countries"][0]["name"] = _SCRIPT
+ html_top, html_sit = _js(tmp_path, [
+ ["renderResearch", "stories", _loaded(top)],
+ ["renderResearch", "situations", _loaded(sit)],
+ ])
+ for html in (html_top, html_sit):
+ m = _assert_inert(html)
+ assert _HOSTILE in m.text and _SCRIPT in m.text
+ assert "javascript:" not in html
+ assert m.hrefs == [] # the javascript: links became plain text
+ assert "&lt;img src=x onerror=&quot;alert(1)&quot;&gt;" in html_top
+ # An http(s) link stays a link.
+ assert _Markup(_js(tmp_path, [["renderResearch", "stories", _loaded(_TOP)]])[0]).hrefs == [
+ "https://www.bbc.co.uk/news/2"
+ ]
+
+
[email protected]("panel", ["liquidity", "situations", "stories", "health"])
+def test_every_payload_string_is_escaped(tmp_path, panel) -> None:
+ payload = {
+ "liquidity": {**_BOARD, "net_liquidity_method": _METHOD},
+ "situations": _SITUATIONS,
+ "stories": _TOP,
+ "health": None,
+ }[panel]
+ if payload is None:
+ db = tmp_path / "h.db"
+ _populate(db)
+ payload = _expected(db)
+ mark = '"\'><i data-x="1">'
+ html_ok, html_failed = _js(tmp_path, [
+ ["renderResearch", panel, _loaded(_poison(payload, mark))],
+ ["renderResearch", panel, _failed(_poison({**payload, "error": "down"}, mark), mark + "down")],
+ ])
+ for html in (html_ok, html_failed):
+ m = _assert_inert(html)
+ assert mark in m.text # shown, as text
+
+
+def test_panel_that_never_loaded_says_so(tmp_path) -> None:
+ never, never_state, loading = _js(tmp_path, [
+ ["renderResearch", "liquidity", None],
+ ["renderResearch", "health", {}],
+ ["renderResearch", "situations", {"inFlight": True, "requestedAt": _T_MS}],
+ ])
+ for html in (never, never_state):
+ assert _Markup(html).text == "Not loaded yet: no request has been made for this panel, so nothing is shown."
+ assert _Markup(loading).text == (
+ f"Loading: first request sent {_CLOCK}. Nothing has loaded yet, so no figures are shown."
+ )
+
+
+def test_failed_call_never_reads_as_no_data(tmp_path) -> None:
+ html, not_json, offline = _js(tmp_path, [
+ ["renderResearch", "situations", _failed(_SITUATIONS_DEGRADED, "RSS feed read failed")],
+ ["renderResearch", "stories", _failed(None, "response was not JSON", http=502)],
+ ["renderResearch", "health", _failed(None, "request failed: TypeError: Failed to fetch", http=None)],
+ ])
+ m = _assert_inert(html)
+ assert "UNAVAILABLE HTTP 503: RSS feed read failed" in m.text
+ assert "Reason: rss_unavailable" in m.text
+ assert _SITUATIONS_DEGRADED["data_gaps"][0] in m.text
+ assert "No situation" not in m.text and "0 articles" not in m.text and "rs-story" not in html
+ assert "UNAVAILABLE HTTP 502: response was not JSON" in _Markup(not_json).text
+ assert "No story" not in not_json
+ assert "UNAVAILABLE request failed: TypeError: Failed to fetch" in _Markup(offline).text
+ assert "mini-box" not in offline # no zero counts for a store that was never read
+
+
+def test_failed_refresh_keeps_the_earlier_copy_and_says_how_old(tmp_path) -> None:
+ st = {
+ "lastAttempt": {"at": _T_MS + 60_000, "http": 504, "data": {"error": "situations timed out after 90 s", "timeout": True},
+ "error": "situations timed out after 90 s"},
+ "lastGood": {"at": _T_MS, "http": 200, "data": _SITUATIONS, "error": None},
+ }
+ (html,) = _js(tmp_path, [["renderResearch", "situations", st]])
+ m = _assert_inert(html)
+ assert "REFRESH FAILED HTTP 504: situations timed out after 90 s" in m.text
+ assert f"Showing the copy loaded {_CLOCK}, 60 s before that attempt." in m.text
+ assert "Israel and Iran trade warnings" in m.text
+
+
+def test_empty_result_states_its_threshold_and_coverage(tmp_path) -> None:
+ sit, top = _js(tmp_path, [
+ ["renderResearch", "situations", _loaded({**_SITUATIONS, "situations": [], "count": 0})],
+ ["renderResearch", "stories", _loaded({**_TOP, "stories": [], "count": 0, "attention": []})],
+ ])
+ s = _Markup(sit).text
+ assert "No situation reached 2 articles in this window, over the coverage stated above." in s
+ assert "Read 2,632 items from 94 feeds" in s
+ assert "Partial coverage: 2 feeds failed or timed out and are absent: CFR, HRW." in s
+ t = _Markup(top).text
+ assert "No story was carried by 3 or more distinct outlets in this window" in t
+ assert "No named country or organization grew against the prior window" in t
+
+
+def test_liquidity_keeps_units_and_keeps_amounts_apart_from_flows(tmp_path) -> None:
+ board = copy.deepcopy(_BOARD)
+ board["indicators"].append(
+ _row("euro_area_m3_flow", "Euro area M3 (flow)", "EUR millions", 1234.5, None, None, None, measure="flow")
+ )
+ board["indicators"][0]["freshness"]["stale"] = True
+ (html,) = _js(tmp_path, [["renderResearch", "liquidity", _loaded({**board, "net_liquidity_method": _METHOD})]])
+ m = _assert_inert(html)
+ text = m.text
+ amounts, flows = text.index("Amounts outstanding (stocks)"), text.index("Flows (transactions within the period)")
+ assert amounts < text.index("Fed total assets") < text.index("Overnight reverse repo") < flows
+ assert flows < text.index("Euro area M3 (flow)")
+ assert "it is not a flow" in text
+ # Values as the payload states them: USD millions stay millions, no conversion.
+ assert '<td class="rs-num">6,600,000</td><td>USD millions</td>' in html
+ assert '<td class="rs-num">210.5</td><td>USD billions</td>' in html
+ assert '<td class="rs-num">1,234.5</td><td>EUR millions</td>' in html
+ assert "-12,345" in text and "-0.19%" in text
+ assert "STALE" in text and "STALE CACHE, 67 min old" in text
+ assert "1 indicator unavailable: shown as missing, never as zero." in text and "ECB HTTP 503" in text
+ for blocked in liquidity.BLOCKED_SOURCES:
+ assert _norm(blocked["measured"]) in text
+ for series, comp in _METHOD["components"].items():
+ assert series in text and comp["units"] in text and _norm(comp["conversion"]) in text
+ assert _norm(_METHOD["alignment"]) in text and _norm(_METHOD["tga_choice"]) in text
+ assert board["ranking_rule"] in text
+
+
+def test_liquidity_total_outage_shows_reasons_and_no_board(tmp_path) -> None:
+ (html,) = _js(tmp_path, [[
+ "renderResearch", "liquidity",
+ _failed({**_BOARD_OUTAGE, "net_liquidity_method": _METHOD}, _BOARD_OUTAGE["error"]),
+ ]])
+ m = _assert_inert(html)
+ assert "UNAVAILABLE HTTP 503: liquidity board unavailable: every source failed" in m.text
+ assert "FRED WALCL: HTTP 503" in m.text
+ assert "a component failed; never computed with a zero term" in m.text
+ assert "Amounts outstanding" not in m.text and "no indicator rows" not in m.text
+
+
+def test_collection_health_lists_problems_and_cannot_prove(tmp_path, monkeypatch) -> None:
+ for var in ("WORLD_INTEL_HISTORY_CADENCE_S", "WORLD_INTEL_HISTORY_CADENCES"):
+ monkeypatch.delenv(var, raising=False)
+ db = tmp_path / "h.db"
+ _populate(db)
+ report = _expected(db)
+ fresh = tmp_path / "fresh.db"
+ never = _expected(fresh)
+ html, html_never, html_down = _js(tmp_path, [
+ ["renderResearch", "health", _loaded(report)],
+ ["renderResearch", "health", _loaded(never)],
+ ["renderResearch", "health", _failed(
+ {"error": "history store unavailable: DatabaseError: file is not a database", "db_path": "/x/cache.db"},
+ "history store unavailable: DatabaseError: file is not a database",
+ )],
+ ])
+ text = _assert_inert(html).text
+ for line in report["cannot_prove"]:
+ assert _norm(line) in text
+ for state, meaning in report["states_explained"].items():
+ assert _norm(f"{state}: {meaning}") in text
+ assert text.index("failing (1)") < text.index("HTTP 503 from FIRMS") < text.index("ok (2)")
+ assert "OVERDUE" in text and "not attempted recently (1)" in text
+ assert "never run (1)" in text and "tides" in text
+ assert "not in roster" in text # legacy_feed: logged, but no longer collected
+ assert "No collector capture is recorded" not in text
+ never_text = _Markup(html_never).text
+ assert "No collector capture is recorded in this database: every source reads never_run, which is not healthy." in never_text
+ down = _Markup(html_down).text
+ assert "UNAVAILABLE HTTP 503: history store unavailable: DatabaseError: file is not a database" in down
+ assert "Database: /x/cache.db" in down
+ assert "mini-box" not in html_down