Skip to content

160-tool MCP server for real-time global intelligence from free public APIs: markets, liquidity, prediction markets, SEC filings, conflict, military, aircraft-cohort early warning, migration and border flows, cyber, climate, news and more. Dated observation history, 29 research prompts, cited briefs, geofences, SSE dashboard. MIT, no paid API keys.

News signals: fix the second 2026-10-17 review findings (NS-1 to NS-9, F1 to F12)

Phoenix Agent (mac-studio)52ca3f52b2f7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b1d7b9b..7e0c597 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -145,6 +145,27 @@
divided by zero; feed links of any scheme (`javascript:`, `data:`)
were passed on and titles had no length cap (now http(s) only, titles
capped at 500 characters).
+- Second review fixes for the news signals (2026-10-17):
+ - The cross-process GDELT floor honoured a reserved slot up to 600 s
+ ahead; the bound is now 40 s (eight processes at 5 s), and a row
+ further ahead is treated as a clock jump: overwritten with the current
+ time and logged. A row that cannot be read as numbers raised on every
+ call; it is now overwritten the same way.
+ - A cancelled floor wait left the process's own future slot, and its
+ next call ignored it and went first; a process now keeps its own
+ queued slot. The slot time is taken once the write lock is held, so a
+ wait for the lock never shortens the gap.
+ - Dossier headline matching stopped counting demonyms such as "North
+ Korean(s)" and "Gazan(s)"; each keyword now also matches with an
+ `-n`, `-ns` or `-s` ending, the compound exclusions unchanged.
+ `searched_span` was sorted as text, so an unparsed date ("Okt 17,
+ 2026") read as the newest item; it now spans parsed UTC times only,
+ and `items_without_parsed_time` counts the rest.
+ - `intel_hostility_timeline` refuses an empty, blank or one-letter
+ country (it went out as nine GDELT requests).
+ - `intel situations` and `intel top-stories` never printed the
+ partial-outage line: the results now carry `feeds_attempted` and
+ `feeds_ok` at the top level.
## 0.11.0 - 2026-10-12
diff --git a/ROADMAP.md b/ROADMAP.md
index 2955cb0..f9537db 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -575,7 +575,7 @@ dashboard.
| 1 | Capture history and revisions | Earlier saved copies of a source, and values a later copy revised | :red_circle: |
| 1 | Collection health | Per-source expected cadence; overdue / failing / never-run per source, measured from the run log, not inferred from cache freshness. Says what it cannot prove | :red_circle: |
| 1 | Research guide over MCP | MCP resources (guide generated from the live registry and source health, measures glossary, "what this server will not do") and MCP prompts (worked research questions, each naming the tools it needs and the limit to respect). Server exposed no prompts or resources before | :red_circle: |
-| 1 | Hostility timeline | Per country pair or country, daily counts of threat / sanction / force / criticism coverage plus tone (`intel_hostility_timeline`); situations between and inside countries, sortable by most covered, most hostile, biggest change (`intel_situations`) | :white_check_mark: Built 2026-10-17, unreleased: `intel_hostility_timeline` (GDELT DOC 2.0 `timelinevolraw` + `timelinetone`, one call per category; failed calls, expired cached copies and missing days null; `column_status` per call; partial days judged per column) and `intel_situations` (RSS headlines, ranked by covered / hostile / change, change defined over feeds whose history spans both windows, null with `change_measured: false` when no feed does). Review fixes 2026-10-17: expired GDELT copies were shown as current counts; a day with no buckets in the judging series read as complete; "no change" was reported where nothing was compared; a GDELT 429 was retried after 1 s and then 2 s; unknown codes such as "GBR" were searched as text; a partial RSS outage was unmarked; limits below 1 misbehaved. All fixed with failing-first tests. Measured 2026-10-17: timeline modes answer (HTTP 200, `date_resolution` `hour` for `timespan=7d` and `15m` for `3d`, explicit zero buckets, `norm` = all monitored articles), but a `timespan=7d` request returned 54 hours (2026-10-15T10Z onward), and about half of calls drew 429 even 30 s apart (later probes, 16:51 to 16:53Z, 4 of 4 drew 429, so the resolution for 14 days and coverage before 2026-10-15T10Z are unmeasured; 30 days was not probed; a live MCP run at 17:00Z got 4 of 6 columns after the retry pass, one at 17:20Z got 1 of 6, consistent with the server's breaker opening after 3 failed calls, which also skips the retry pass). The fetcher now spaces every live GDELT call at least 5 s apart across the processes that share one cache DB (`shared_floor.py`), and retries a 429 no sooner than that; the timeline's retry pass stops at the first retry that fails again. The AOI sweep makes one GDELT call per AOI, so the floor bounds it: against a mocked GDELT, 45 AOIs took 220 s of the collector's 240 s `aoi_digest` budget and 50 timed out after 48 calls (every other AOI domain stubbed; `analysis/aoi.py` not changed here). Artlist was not used for daily counts: 250 records sorted by date covered 15 minutes. Situations see only the 22-country entity table, whose matcher still counts "South Sudan" as Sudan and "Indian Ocean" as India (`analysis/entities.py`, not changed here) |
+| 1 | Hostility timeline | Per country pair or country, daily counts of threat / sanction / force / criticism coverage plus tone (`intel_hostility_timeline`); situations between and inside countries, sortable by most covered, most hostile, biggest change (`intel_situations`) | :white_check_mark: Built 2026-10-17, unreleased: `intel_hostility_timeline` (GDELT DOC 2.0 `timelinevolraw` + `timelinetone`, one call per category; failed calls, expired cached copies and missing days null; `column_status` per call; partial days judged per column) and `intel_situations` (RSS headlines, ranked by covered / hostile / change, change defined over feeds whose history spans both windows, null with `change_measured: false` when no feed does). Review fixes 2026-10-17: expired GDELT copies were shown as current counts; a day with no buckets in the judging series read as complete; "no change" was reported where nothing was compared; a GDELT 429 was retried after 1 s and then 2 s; unknown codes such as "GBR" were searched as text; a partial RSS outage was unmarked; limits below 1 misbehaved. All fixed with failing-first tests. Measured 2026-10-17: timeline modes answer (HTTP 200, `date_resolution` `hour` for `timespan=7d` and `15m` for `3d`, explicit zero buckets, `norm` = all monitored articles), but a `timespan=7d` request returned 54 hours (2026-10-15T10Z onward), and about half of calls drew 429 even 30 s apart (later probes, 16:51 to 16:53Z, 4 of 4 drew 429, so the resolution for 14 days and coverage before 2026-10-15T10Z are unmeasured; 30 days was not probed; a live MCP run at 17:00Z got 4 of 6 columns after the retry pass, one at 17:20Z got 1 of 6, consistent with the server's breaker opening after 3 failed calls, which also skips the retry pass). The fetcher now spaces every live GDELT call at least 5 s apart across the processes that share one cache DB (`shared_floor.py`), and retries a 429 no sooner than that; the timeline's retry pass stops at the first retry that fails again. Second review 2026-10-17: a shared slot more than 40 s ahead (eight processes at 5 s; it was 600 s) is a clock jump, overwritten and logged, as is an unreadable row; a process keeps its own queued slot after a cancelled wait; the slot time is taken once the write lock is held; an empty or one-letter country is refused. The AOI sweep makes one GDELT call per AOI, so the floor bounds it: against a mocked GDELT, 45 AOIs took 220 s of the collector's 240 s `aoi_digest` budget and 50 timed out after 48 calls (every other AOI domain stubbed; `analysis/aoi.py` not changed here). Artlist was not used for daily counts: 250 records sorted by date covered 15 minutes. Situations see only the 22-country entity table, whose matcher still counts "South Sudan" as Sudan and "Indian Ocean" as India (`analysis/entities.py`, not changed here) |
| 1 | Story breadth | News clusters carry a distinct-outlet count; a top-stories view ranks by how many outlets carry a story | :white_check_mark: Built 2026-10-17, unreleased: clusters gain `outlets` / `outlet_count` (sister feeds of one publisher count once); `intel_top_stories` ranks by outlet count with an attention list (countries and organizations by mention growth). Found on the way: cluster `sources` was empty in every live cluster (key mismatch, `feed_name` vs `source_name`) and RSS times were shifted by the host UTC offset; both fixed. Concurrent RSS reads now share one request per feed (a live collector news-group cycle: 119 requests for 119 feeds). Review fixes 2026-10-17: clustering runs off the event loop; attention is empty with `attention_change_measured: false` when no feed spans both windows, never "nothing grew" |
| 1 | Liquidity and flows | Fed balance sheet, Treasury General Account, reverse repo and net liquidity (FRED keyless CSV), money supply (US M2, euro area M1/M2/M3, never summed), ECB, ICI fund flows and ETF net issuance, FINRA margin statistics, BLS series; amounts kept separate from flows | :red_circle: |
| 1 | Kalshi + cross-venue markets | Kalshi public market data next to Polymarket; same-event comparison with an explicit match confidence; settled markets scored against pre-settlement prices (Brier) | :red_circle: |
@@ -638,7 +638,7 @@ fail-quietly bugs below were fixed in the same-day follow-up
| Country/company/leader entity matching was substring-based ("usa" inside "thousand", "hamas" inside "Bahamas") | `analysis/entities.py` | false positives | :white_check_mark: Fixed in 0.5.0 (word-boundary alternations, measured faster) |
| Category keywords matched substrings ("airstrike" bumped severity via "strike"; "denied" fired "ied") | `analysis/classifier.py` | false positives | :white_check_mark: Fixed in 0.5.0 (boundary-anchored stems, per-keyword overrides) |
| Silent-empty degradation without a marker on RSS/API failure | `sources/prediction.py` (documented as intended), `sources/maritime.py`, `sources/news.py` RSS path, `sources/fleet.py` `_safe` wrapper | silent degradation | :yellow_circle: `sources/news.py` RSS path fixed 2026-10-17 (every call reports `feeds_attempted` / `feeds_ok` / `feeds_failed` / `feeds_stale`; no feed answering live is `error` + `degraded` + `reason: all_feeds_failed`, including the warm-cache case where failed feeds came back as expired copies and counted as healthy; failing-first regression tests). Since the 2026-10-17 review: fewer than half of the feeds answering live is `degraded` + `reason: partial_feeds_failed` with no error (measured that day: 95 of 119 answered, the rest permanent 404 / 403 / 410 plus one timeout), passed through by trending keywords, situations, top stories and the dossier; links are kept only when http(s) and titles are capped at 500 characters. prediction, maritime and fleet still open. The fetcher's stale fallback marks only dict payloads, so other text and XML sources can serve expired copies unmarked |
-| Country dossier news was always empty with no marker, for every country: it asked the feed for `category="all"` (not a category, so `fetch_news_feed` answered with an error dict), read `articles` / `source` while the feed emits `items` / `feed_name`, and checked only its own `_error`, never the feed's `error`. Also, 28 of the 50 accepted codes have no headline keywords, so their search could never match | `analysis/dossier.py` news section | fail-reads-as-success | :white_check_mark: Fixed 2026-10-17, unreleased (reads every feed; a feed error or outage passes through as `error` / `degraded` / `reason` with `mention_count: null` and a data gap; failed and stale feeds named; keywordless countries report "mentions not searched"; `intel dossier` prints `News: not measured`; failing-first tests drive the real `fetch_news_feed` output). Since the 2026-10-17 review: headlines match on whole words (the substring test counted "Tirana" as Iran and "Indiana" as India), four compound names are removed first, and the section reports `matching_items`, `mention_count_capped`, `items_searched` and `searched_span` |
+| Country dossier news was always empty with no marker, for every country: it asked the feed for `category="all"` (not a category, so `fetch_news_feed` answered with an error dict), read `articles` / `source` while the feed emits `items` / `feed_name`, and checked only its own `_error`, never the feed's `error`. Also, 28 of the 50 accepted codes have no headline keywords, so their search could never match | `analysis/dossier.py` news section | fail-reads-as-success | :white_check_mark: Fixed 2026-10-17, unreleased (reads every feed; a feed error or outage passes through as `error` / `degraded` / `reason` with `mention_count: null` and a data gap; failed and stale feeds named; keywordless countries report "mentions not searched"; `intel dossier` prints `News: not measured`; failing-first tests drive the real `fetch_news_feed` output). Since the 2026-10-17 review: headlines match on whole words (the substring test counted "Tirana" as Iran and "Indiana" as India), four compound names are removed first, and the section reports `matching_items`, `mention_count_capped`, `items_searched` and `searched_span`. Second review 2026-10-17: demonyms such as "North Korean(s)" and "Gazan(s)" match again (an `-n` / `-ns` / `-s` ending on each keyword); `searched_span` spans parsed UTC times only and `items_without_parsed_time` counts the rest |
| 33 CLI commands rendered upstream errors as healthy empty states | `cli.py` | fail-reads-as-success | :white_check_mark: Fixed in 0.6.0 (shared bail-on-error path; 43 failing-first tests) |
| Rich markup swallowed lowercase bracketed values and let remote titles inject markup | `cli.py` | output corruption | :white_check_mark: Fixed in 0.6.0 (per-site escaping; injection test) |
| Remote data inside Rich TABLE cells was not markup-escaped | `cli.py` | output corruption | :white_check_mark: Fixed in 0.6.0 (`_cell` Text-wrapping convention, 11 injection tests; error styles unified across all 53 commands) |
diff --git a/src/world_intel_mcp/analysis/dossier.py b/src/world_intel_mcp/analysis/dossier.py
index ef2b584..fe3cb7a 100644
--- a/src/world_intel_mcp/analysis/dossier.py
+++ b/src/world_intel_mcp/analysis/dossier.py
@@ -32,8 +32,9 @@ _COMPOUND_EXCLUSIONS: dict[str, re.Pattern[str]] = {
_MATCH_METHOD = (
"whole-word match of each headline against the country's tier-1 keywords "
- "(name, demonym, capital; plurals of words of five letters or more), so "
- "'Tirana' is not Iran and 'Indiana' is not India; South Sudan, the Indian "
+ "(name, demonym, capital), each allowed an -n, -ns or -s ending on its last "
+ "word ('North Koreans', 'Gazan', 'Afghans'), so 'Tirana' is not Iran and "
+ "'Indiana' is not India; South Sudan, the Indian "
"Ocean, Latin/South/Central/North American and New Mexico are removed "
"first. Other compound names still count (Lagos in Portugal, the Republic "
"of the Congo). Searched: the newest items across every RSS feed."
@@ -47,15 +48,29 @@ def _keyword_pattern(iso3: str) -> re.Pattern[str] | None:
keywords = (TIER1_COUNTRIES.get(iso3) or {}).get("keywords") or []
if not keywords:
return None
- parts = []
- for kw in sorted(keywords, key=len, reverse=True):
- body = re.escape(kw.lower())
- if kw.isalpha() and len(kw) >= 5:
- body += "s?"
- parts.append(body)
+ # The keyword table lists names and some demonyms, not every form:
+ # "north korea" must also find "North Korean(s)", "gaza" "Gazan(s)".
+ parts = [
+ re.escape(kw.lower()) + "(?:n|ns|s)?"
+ for kw in sorted(keywords, key=len, reverse=True)
+ ]
return re.compile(r"\b(?:" + "|".join(parts) + r")\b")
+def _utc_stamp(value: object) -> datetime | None:
+ """A feed item's published time if it is the feed's ISO UTC form."""
+ if not isinstance(value, str):
+ return None
+ try:
+ return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
+ except ValueError:
+ return None
+
+
+def _iso_z(when: datetime) -> str:
+ return when.strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
def _names_country(title: str, iso3: str) -> bool:
"""Whether a headline names the tier-1 country ``iso3`` (whole words)."""
pattern = _keyword_pattern(iso3)
@@ -271,14 +286,18 @@ async def fetch_country_dossier(
"link": article.get("link"),
"category": article.get("category"),
})
- stamps = sorted(
- a["published"] for a in items if isinstance(a.get("published"), str) and a["published"]
- )
+ # The span comes from parsed UTC times only: the feed passes a date
+ # it cannot parse through raw ("Okt 17, 2026"), which would sort
+ # anywhere as text.
+ stamps = sorted(t for t in (_utc_stamp(a.get("published")) for a in items) if t)
news_reach = {
"matching_items": matching,
"mention_count_capped": matching > len(news_mentions),
"items_searched": len(items),
- "searched_span": {"oldest": stamps[0], "newest": stamps[-1]} if stamps else None,
+ "searched_span": (
+ {"oldest": _iso_z(stamps[0]), "newest": _iso_z(stamps[-1])} if stamps else None
+ ),
+ "items_without_parsed_time": len(items) - len(stamps),
"match_method": _MATCH_METHOD,
}
if news_data.get("degraded"):
diff --git a/src/world_intel_mcp/analysis/news_signals.py b/src/world_intel_mcp/analysis/news_signals.py
index bc9f7da..4d58627 100644
--- a/src/world_intel_mcp/analysis/news_signals.py
+++ b/src/world_intel_mcp/analysis/news_signals.py
@@ -288,8 +288,12 @@ _UNMEASURED_CHANGE = {
def _feed_marker(feed: dict) -> dict:
- """A partial RSS outage's degraded/reason, passed on with the ranking."""
- return {k: feed[k] for k in ("degraded", "reason") if k in feed}
+ """A partial RSS outage's degraded/reason, passed on with the ranking,
+ plus the feed counts it rests on (the CLI's partial-outage line reads
+ them)."""
+ return {
+ k: feed[k] for k in ("degraded", "reason", "feeds_attempted", "feeds_ok") if k in feed
+ }
_CHANGE_DEFINITION = (
@@ -482,21 +486,35 @@ async def fetch_hostility_timeline(
name, iso3 = _resolve_country(value)
parties.append({"input": value, "name": name, "iso3": iso3})
- codes = [p["input"] for p in parties if p["iso3"] is None and _looks_like_code(p["input"])]
- if codes:
+ def _refused(error: str) -> dict:
return {
"parties": parties,
"days_requested": days,
"days": [],
- "error": "; ".join(
+ "error": error,
+ "source": "gdelt-doc-2.0",
+ "timestamp": _iso(now),
+ }
+
+ # An empty or one-letter input would send GDELT empty or one-letter
+ # queries (nine requests with the retry pass) and could open its breaker.
+ short = [p["input"] for p in parties if len("".join(str(p["input"] or "").split())) < 2]
+ if short:
+ return _refused(
+ "; ".join(f"country {c!r} is too short to search" for c in short)
+ + "; pass a country name, or an ISO3 or ISO2 code from the "
+ f"{len(TIER1_COUNTRIES)}-country table"
+ )
+ codes = [p["input"] for p in parties if p["iso3"] is None and _looks_like_code(p["input"])]
+ if codes:
+ return _refused(
+ "; ".join(
f"'{c}' looks like a country code but is not in the "
f"{len(TIER1_COUNTRIES)}-country table"
for c in codes
)
- + "; pass the country's name instead (names are searched verbatim)",
- "source": "gdelt-doc-2.0",
- "timestamp": _iso(now),
- }
+ + "; pass the country's name instead (names are searched verbatim)"
+ )
base_query = " ".join(_gdelt_term(p["name"]) for p in parties)
queries = {"articles": base_query}
diff --git a/src/world_intel_mcp/shared_floor.py b/src/world_intel_mcp/shared_floor.py
index 12b85c3..f73992b 100644
--- a/src/world_intel_mcp/shared_floor.py
+++ b/src/world_intel_mcp/shared_floor.py
@@ -9,35 +9,45 @@ collector daemon, one MCP stdio server per client, the dashboard, and
every ``intel`` CLI run. For the sources in ``SHARED_SOURCES`` each live
request therefore also reserves a slot in a one-row-per-source table of
that cache DB. The reservation runs under ``BEGIN IMMEDIATE``, so two
-processes can never read the same last slot.
+processes can never read the same last slot, and the slot is stamped only
+once the write lock is held, so a wait for the lock never shortens the gap.
-A process's own earlier reservation is left to its in-process floor:
-only another process's slot pushes this one back. A slot more than
-``MAX_AHEAD_S`` ahead of the clock is treated as a clock jump or a corrupt
-row, not a queue, and ignored.
+Another process's slot pushes this one to that slot plus the interval. A
+process's own earlier slot is left to its in-process floor, except that
+a slot of its own still ahead of the clock (a reservation whose wait was
+cancelled) is kept, so the next call cannot jump the queue it joined. A
+slot more than ``MAX_AHEAD_S`` ahead of the clock is not a queue: it is
+treated as a clock jump, overwritten with the current time and logged. A
+row that cannot be read as numbers is overwritten the same way.
Gaps / not covered: processes that resolve different cache DB paths
(``WORLD_INTEL_CACHE_DB`` / ``XDG_CACHE_HOME`` decide it) do not see each
other's slots; other hosts are separate IP addresses with their own DBs;
-a reservation that fails (database locked past the timeout, read-only
-file) is logged by the fetcher and the call goes ahead on the in-process
-floor alone; sources outside ``SHARED_SOURCES`` keep a per-process floor.
+a reservation that fails (database locked past the timeout, a file that
+is not a database, read-only file) is logged by the fetcher and the call
+goes ahead on the in-process floor alone; sources outside
+``SHARED_SOURCES`` keep a per-process floor; a backward clock step smaller
+than ``MAX_AHEAD_S`` is still waited out.
"""
from __future__ import annotations
+import logging
import os
import sqlite3
import time
from pathlib import Path
+logger = logging.getLogger("world-intel-mcp.shared_floor")
+
# Sources whose floor holds across processes. Others keep the in-process
# floor only.
SHARED_SOURCES = frozenset({"gdelt"})
-# Ten minutes of queued 5 s slots is far beyond any real queue (one waiter
-# per process at a time); a slot further ahead than this is not one.
-MAX_AHEAD_S = 600.0
+# A real queue holds at most one waiter per process: eight processes on one
+# host (collector, a few MCP servers, dashboard, CLI runs) at GDELT's 5 s
+# is 40 s. A slot further ahead than this is a clock jump, not a queue.
+MAX_AHEAD_S = 40.0
_LOCK_TIMEOUT_S = 5.0
@@ -47,16 +57,17 @@ def reserve(
) -> float:
"""Reserve the next request slot for ``source``; return its Unix time.
- The slot is ``now`` unless another process reserved one less than
- ``interval`` before ``now`` (or one still ahead of it), in which case it
- is that slot plus ``interval``. Raises ``sqlite3.Error`` when the DB
- cannot be locked or written.
+ The slot is the current time (``now`` when given, else the clock once
+ the write lock is held), pushed to another process's slot plus
+ ``interval`` or kept at this process's own slot when either is still
+ ahead. Raises ``sqlite3.Error`` when the DB cannot be locked or written.
"""
- now = time.time() if now is None else now
pid = os.getpid()
conn = sqlite3.connect(str(db_path), timeout=_LOCK_TIMEOUT_S, isolation_level=None)
try:
conn.execute("BEGIN IMMEDIATE")
+ # Sampled with the lock held: a slot never predates its reservation.
+ now = time.time() if now is None else now
conn.execute(
"CREATE TABLE IF NOT EXISTS source_floor ("
"source TEXT PRIMARY KEY, slot REAL NOT NULL, pid INTEGER NOT NULL)"
@@ -66,9 +77,26 @@ def reserve(
).fetchone()
slot = now
if row is not None:
- last, last_pid = float(row[0]), int(row[1])
- if last_pid != pid and last - now <= MAX_AHEAD_S:
- slot = max(now, last + interval)
+ try:
+ last, last_pid = float(row[0]), int(row[1])
+ except (TypeError, ValueError):
+ logger.warning(
+ "Unreadable %s floor row %r; overwriting it with the current time",
+ source,
+ tuple(row),
+ )
+ else:
+ if last - now > MAX_AHEAD_S:
+ logger.warning(
+ "%s floor slot is %.0f s ahead of the clock (a clock jump, not a "
+ "queue); overwriting it with the current time",
+ source,
+ last - now,
+ )
+ elif last_pid != pid:
+ slot = max(now, last + interval)
+ else:
+ slot = max(now, last)
conn.execute(
"INSERT OR REPLACE INTO source_floor (source, slot, pid) VALUES (?, ?, ?)",
(source, slot, pid),
diff --git a/src/world_intel_mcp/tests/test_gdelt_floor_guarantees.py b/src/world_intel_mcp/tests/test_gdelt_floor_guarantees.py
new file mode 100644
index 0000000..6582d67
--- /dev/null
+++ b/src/world_intel_mcp/tests/test_gdelt_floor_guarantees.py
@@ -0,0 +1,475 @@
+"""What the cross-process GDELT floor guarantees, pinned one guarantee at a time.
+
+Second review of the news signals (2026-10-17): the floor tests left four
+mutants alive, and the floor itself let a slot predate its write lock,
+let a cancelled wait jump its own queue, honoured slots up to 600 s
+ahead and crashed on a row it could not read. Each test below names the
+finding it covers. Clock-driven tests advance a fake clock only through
+the patched ``asyncio.sleep`` and record the clock at each request, so
+they measure request spacing, not requested sleeps.
+
+Gaps / not covered: retries after a multi-attempt failure in one call are
+spaced by the per-attempt throttle of the integration branch (its
+test_fetcher_retry_spacing.py), not on this branch; real GDELT behaviour
+(no network); processes on different cache DB paths (they do not
+coordinate, by design).
+"""
+
+import asyncio
+import logging
+import os
+import sqlite3
+import subprocess
+import sys
+import textwrap
+import threading
+import time
+
+import httpx
+import pytest
+import respx
+
+from world_intel_mcp import fetcher as fetcher_mod
+from world_intel_mcp import shared_floor
+from world_intel_mcp.analysis import news_signals as ns
+from world_intel_mcp.circuit_breaker import CircuitBreaker
+from world_intel_mcp.fetcher import Fetcher
+from world_intel_mcp.sources import news
+from world_intel_mcp.tests.test_news_signals import _NOW_GDELT, _gdelt_router
+
+_GDELT_URL = r".*api\.gdeltproject\.org.*"
+_OTHER_PID = os.getpid() + 1
+_T = 1_000_000.0
+_FLOOR = fetcher_mod._SOURCE_RATE_LIMITS["gdelt"]
+_REAL_SLEEP = asyncio.sleep
+
+
+def _seed(db, source: str, slot, pid) -> None:
+ conn = sqlite3.connect(str(db))
+ try:
+ conn.execute(
+ "CREATE TABLE IF NOT EXISTS source_floor ("
+ "source TEXT PRIMARY KEY, slot REAL NOT NULL, pid INTEGER NOT NULL)"
+ )
+ conn.execute(
+ "INSERT OR REPLACE INTO source_floor (source, slot, pid) VALUES (?, ?, ?)",
+ (source, slot, pid),
+ )
+ conn.commit()
+ finally:
+ conn.close()
+
+
+def _row(db, source: str = "gdelt"):
+ conn = sqlite3.connect(str(db))
+ try:
+ return conn.execute(
+ "SELECT slot, pid FROM source_floor WHERE source = ?", (source,)
+ ).fetchone()
+ finally:
+ conn.close()
+
+
+class _Clock:
+ """Wall clock that moves only when someone sleeps."""
+
+ def __init__(self, start: float) -> None:
+ self.now = start
+ self.sleeps: list[float] = []
+
+ def time(self) -> float:
+ return self.now
+
+ async def sleep(self, delay, *args, **kwargs) -> None:
+ self.sleeps.append(delay)
+ self.now += max(0.0, delay)
+ await _REAL_SLEEP(0)
+
+
+def clock(monkeypatch: pytest.MonkeyPatch) -> _Clock:
+ c = _Clock(time.time())
+ monkeypatch.setattr(time, "time", c.time)
+ monkeypatch.setattr(asyncio, "sleep", c.sleep)
+ return c
+
+
+def _gaps(stamps: list[float]) -> list[float]:
+ return [b - a for a, b in zip(stamps, stamps[1:])]
+
+
+# ---------------------------------------------------------------------------
+# reserve(): the row
+# ---------------------------------------------------------------------------
+
+
+def test_reserve_overwrites_a_slot_far_ahead_as_a_clock_jump(
+ tmp_path, caplog: pytest.LogCaptureFixture
+) -> None:
+ # NS-2 / F9: a row 120 s ahead (a backward clock step) was honoured, so
+ # every other process slept two minutes before its first GDELT call.
+ db = tmp_path / "floor.db"
+ _seed(db, "gdelt", slot=_T + 120, pid=_OTHER_PID)
+ with caplog.at_level(logging.WARNING, logger="world-intel-mcp.shared_floor"):
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T) == _T
+ assert _row(db) == (_T, os.getpid())
+ assert any("clock jump" in r.message for r in caplog.records)
+ # A real queue stays inside the bound and is still honoured.
+ _seed(db, "gdelt", slot=_T + 30, pid=_OTHER_PID)
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T) == _T + 35
+
+
+def test_reserve_after_an_hour_old_row_goes_now(tmp_path) -> None:
+ # NS-5 (S2): without max(now, ...) an old row from another process,
+ # the steady state of any long-lived cache DB, put the slot in the past.
+ db = tmp_path / "floor.db"
+ _seed(db, "gdelt", slot=_T - 3600, pid=_OTHER_PID)
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T) == _T
+
+
+def test_reserve_keeps_its_own_queued_slot(tmp_path) -> None:
+ # NS-3 / F2: a process queued behind another one, its wait cancelled,
+ # used to go at once on its next call and rewind the shared row.
+ db = tmp_path / "floor.db"
+ _seed(db, "gdelt", slot=_T + 2, pid=_OTHER_PID)
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T) == _T + 7
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T + 1) == _T + 7
+ assert _row(db) == (_T + 7, os.getpid())
+ # Once that slot has passed, its own floor governs again.
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T + 9) == _T + 9
+
+
[email protected]("bad_slot", ["2026-10-17T22:00:00Z", b"\x00"])
+def test_reserve_overwrites_an_unreadable_row(
+ tmp_path, caplog: pytest.LogCaptureFixture, bad_slot
+) -> None:
+ # NS-6: a text or blob slot raised ValueError on every GDELT call,
+ # forever, since nothing rewrote the row.
+ db = tmp_path / "floor.db"
+ _seed(db, "gdelt", slot=bad_slot, pid=_OTHER_PID)
+ with caplog.at_level(logging.WARNING, logger="world-intel-mcp.shared_floor"):
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T) == _T
+ assert any("Unreadable gdelt floor row" in r.message for r in caplog.records)
+ assert _row(db) == (_T, os.getpid())
+ assert shared_floor.reserve(db, "gdelt", 5.0, now=_T + 6) == _T + 6
+
+
+def test_reserve_stamps_the_slot_once_it_holds_the_write_lock(tmp_path) -> None:
+ # NS-4 / F1: the slot was the time the caller started waiting for the
+ # lock, so the next process was spaced from a moment that had passed.
+ db = tmp_path / "floor.db"
+ shared_floor.reserve(db, "gdelt", 5.0, now=_T)
+ held = threading.Event()
+ released: list[float] = []
+
+ def _hold() -> None:
+ conn = sqlite3.connect(str(db), isolation_level=None)
+ try:
+ conn.execute("BEGIN IMMEDIATE")
+ held.set()
+ time.sleep(1.0)
+ released.append(time.time())
+ conn.execute("COMMIT")
+ finally:
+ conn.close()
+
+ holder = threading.Thread(target=_hold)
+ holder.start()
+ try:
+ assert held.wait(5.0)
+ asked = time.time()
+ slot = shared_floor.reserve(db, "gdelt", 5.0)
+ finally:
+ holder.join()
+ assert slot >= released[0]
+ assert slot - asked >= 0.9
+
+
+_PAUSING_CHILD = textwrap.dedent(
+ """
+ import sys, time
+ from pathlib import Path
+ from types import SimpleNamespace
+ import sqlite3
+ from world_intel_mcp import shared_floor
+
+ db, role, marker, interval = sys.argv[1], sys.argv[2], Path(sys.argv[3]), float(sys.argv[4])
+ if role == "slow":
+ real = sqlite3.connect
+
+ class _One:
+ def __init__(self, row):
+ self.row = row
+
+ def fetchone(self):
+ return self.row
+
+ class _Pausing:
+ def __init__(self, conn):
+ self.conn = conn
+
+ def execute(self, sql, *args):
+ cur = self.conn.execute(sql, *args)
+ if sql.startswith("SELECT"):
+ row = cur.fetchone()
+ marker.touch()
+ time.sleep(1.0)
+ return _One(row)
+ return cur
+
+ def close(self):
+ self.conn.close()
+
+ shared_floor.sqlite3 = SimpleNamespace(connect=lambda *a, **k: _Pausing(real(*a, **k)))
+ else:
+ deadline = time.time() + 30
+ while not marker.exists():
+ if time.time() > deadline:
+ sys.exit("slow child never paused")
+ time.sleep(0.01)
+ print(repr(shared_floor.reserve(db, "gdelt", interval)))
+ """
+)
+
+
+def test_begin_immediate_keeps_two_overlapping_reservations_apart(tmp_path) -> None:
+ # F3 / NS-5 (S4): with a deferred BEGIN, a reservation made while
+ # another sits between its read and its write either reads the same
+ # old row or fails with "database is locked"; nothing caught it.
+ db = tmp_path / "floor.db"
+ marker = tmp_path / "paused"
+ interval = 0.2
+ shared_floor.reserve(db, "gdelt", interval, now=time.time() - 100)
+ procs = []
+ try:
+ for role in ("slow", "fast"):
+ procs.append(subprocess.Popen(
+ [sys.executable, "-c", _PAUSING_CHILD, str(db), role, str(marker), str(interval)],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
+ ))
+ outs = [p.communicate(timeout=60) for p in procs]
+ finally:
+ for p in procs:
+ if p.poll() is None:
+ p.kill()
+ for p, (_, err) in zip(procs, outs):
+ assert p.returncode == 0, err
+ slow, fast = (float(out.strip()) for out, _ in outs)
+ # The fast child had to wait for the slow one's write lock.
+ assert fast - slow >= 0.9
+
+
+_THROTTLE_CHILD = textwrap.dedent(
+ """
+ import asyncio, json, sys, time
+ from pathlib import Path
+ from world_intel_mcp import fetcher as fetcher_mod
+ from world_intel_mcp import shared_floor
+ from world_intel_mcp.cache import Cache
+ from world_intel_mcp.circuit_breaker import CircuitBreaker
+
+ db, start, interval = sys.argv[1], float(sys.argv[2]), float(sys.argv[3])
+ fetcher_mod._SOURCE_RATE_LIMITS["gdelt"] = interval
+ slots = []
+ real_reserve = shared_floor.reserve
+
+ def _recording(*args, **kwargs):
+ slots.append(real_reserve(*args, **kwargs))
+ return slots[-1]
+
+ shared_floor.reserve = _recording
+ f = fetcher_mod.Fetcher(cache=Cache(db_path=Path(db)), breaker=CircuitBreaker())
+
+ async def main():
+ ready = time.time()
+ while time.time() < start:
+ await asyncio.sleep(0.005)
+ await f._source_throttle("gdelt")
+ print(json.dumps({"ready": ready, "slot": slots[0], "went": time.time()}))
+
+ asyncio.run(main())
+ """
+)
+
+
+def test_floor_holds_across_processes_while_another_writer_holds_the_lock(tmp_path) -> None:
+ # NS-5 / F12: four processes ask at one instant while a cache write
+ # holds the DB lock. Judged from the reserved slots, not wall gaps: each
+ # slot is at least one interval after the one before, none predates the
+ # lock's release, and no process fell back to its local floor.
+ import json
+
+ db = tmp_path / "shared.db"
+ interval = 0.5
+ # Switch the DB to WAL here, as a running deployment already is: four
+ # children making the switch at once is a startup race (measured: one
+ # child in four runs died "database is locked" in PRAGMA journal_mode).
+ from world_intel_mcp.cache import Cache
+
+ Cache(db_path=db).close()
+ shared_floor.reserve(db, "gdelt", interval, now=time.time() - 100)
+ start = time.time() + 4.0
+ released: list[float] = []
+ held = threading.Event()
+
+ def _hold() -> None:
+ conn = sqlite3.connect(str(db), isolation_level=None)
+ try:
+ while time.time() < start - 0.2:
+ time.sleep(0.01)
+ conn.execute("BEGIN IMMEDIATE")
+ held.set()
+ time.sleep(1.2)
+ released.append(time.time())
+ conn.execute("COMMIT")
+ finally:
+ conn.close()
+
+ holder = threading.Thread(target=_hold)
+ holder.start()
+ procs = []
+ try:
+ for _ in range(4):
+ procs.append(subprocess.Popen(
+ [sys.executable, "-c", _THROTTLE_CHILD, str(db), repr(start), str(interval)],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
+ ))
+ outs = [p.communicate(timeout=60) for p in procs]
+ finally:
+ for p in procs:
+ if p.poll() is None:
+ p.kill()
+ holder.join()
+ assert held.is_set()
+ for p, (_, err) in zip(procs, outs):
+ assert p.returncode == 0, err
+ assert "floor unavailable" not in err, err
+ runs = sorted((json.loads(out) for out, _ in outs), key=lambda r: r["slot"])
+ # Every child was waiting before the first slot, so the race was real.
+ assert all(r["ready"] < start for r in runs), runs
+ assert all(r["slot"] >= released[0] for r in runs), (runs, released)
+ assert all(g >= interval - 1e-9 for g in _gaps([r["slot"] for r in runs])), runs
+ # asyncio.sleep runs on the monotonic clock: allow 10 ms of drift.
+ assert all(r["went"] >= r["slot"] - 0.01 for r in runs), runs
+
+
+# ---------------------------------------------------------------------------
+# The fetcher's use of the floor
+# ---------------------------------------------------------------------------
+
+
+async def test_a_far_ahead_row_costs_at_most_one_interval(fetcher, clock: _Clock) -> None:
+ # NS-2: the wait the caller sees, end to end through the fetcher.
+ shared_floor.reserve(fetcher.cache.db_path, "gdelt", _FLOOR)
+ _seed(fetcher.cache.db_path, "gdelt", slot=clock.now + 120, pid=_OTHER_PID)
+ await fetcher._source_throttle("gdelt")
+ assert sum(clock.sleeps) <= _FLOOR
+
+
+async def test_local_floor_counts_from_when_the_call_went(fetcher, clock: _Clock) -> None:
+ # F4 / NS-5 (S8): stamping the in-process floor before the shared wait
+ # spaced this process's next call from when the first one started
+ # waiting, so the two went 0 s apart.
+ _seed(fetcher.cache.db_path, "gdelt", slot=clock.now + 2, pid=_OTHER_PID)
+ went = []
+ for _ in range(2):
+ await fetcher._source_throttle("gdelt")
+ went.append(clock.now)
+ assert went[1] - went[0] >= _FLOOR
+
+
+async def test_a_cancelled_wait_still_holds_its_place_in_the_queue(
+ fetcher, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # NS-3 / F2 end to end, on the real clock: a caller's timeout cancels
+ # the wait behind another process; the next call waits for the slot
+ # this process had queued for, not 0 s.
+ interval = 0.5
+ monkeypatch.setitem(fetcher_mod._SOURCE_RATE_LIMITS, "gdelt", interval)
+ other_slot = time.time() + 1.0
+ _seed(fetcher.cache.db_path, "gdelt", slot=other_slot, pid=_OTHER_PID)
+ with pytest.raises(asyncio.TimeoutError):
+ await asyncio.wait_for(fetcher._source_throttle("gdelt"), 0.2)
+ await fetcher._source_throttle("gdelt")
+ assert time.time() >= other_slot + interval
+
+
+async def test_source_throttle_falls_back_when_the_cache_file_is_not_a_database(
+ fetcher, monkeypatch: pytest.MonkeyPatch, tmp_path, caplog: pytest.LogCaptureFixture
+) -> None:
+ # NS-5 (S6): narrowing the fetcher's except to OperationalError made a
+ # corrupt cache file raise DatabaseError on every GDELT call.
+ junk = tmp_path / "junk.db"
+ junk.write_bytes(b"this is not an sqlite database " * 200)
+ monkeypatch.setattr(fetcher.cache, "db_path", junk)
+ with caplog.at_level(logging.WARNING, logger="world-intel-mcp.fetcher"):
+ await fetcher._source_throttle("gdelt")
+ assert any(
+ "Cross-process gdelt floor unavailable" in r.message and "not a database" in r.message
+ for r in caplog.records
+ )
+ assert fetcher_mod._source_last_call["gdelt"] > 0
+
+
+async def test_429_retries_are_spaced_by_request_time(fetcher, clock: _Clock) -> None:
+ # F10: the clock at each request, not the sleeps asked for.
+ sent: list[float] = []
+ answers = iter([429, 429, 200])
+
+ def _answer(request: httpx.Request) -> httpx.Response:
+ sent.append(clock.now)
+ status = next(answers)
+ if status == 429:
+ return httpx.Response(429, text="Please limit requests to one every 5 seconds")
+ return httpx.Response(200, json={"articles": []})
+
+ respx.get(url__regex=_GDELT_URL).mock(side_effect=_answer)
+ await news.fetch_gdelt_search(fetcher, "Iran")
+ assert len(sent) == 3
+ assert min(_gaps(sent)) >= _FLOOR, sent
+
+
+async def test_timeline_requests_are_spaced_and_the_retry_waits_its_pause(
+ fetcher, clock: _Clock
+) -> None:
+ # F10 and F5 (M08): every timeline request follows the one before by at
+ # least the floor, and the retry pass's request by its full pause.
+ # One attempt per call, so each request is its own fetcher call.
+ single = Fetcher(cache=fetcher.cache, breaker=CircuitBreaker(), max_retries=0)
+ route, calls = _gdelt_router(fail_times={"sanction": 1})
+ sent: list[tuple[str, float]] = []
+
+ def _answer(request: httpx.Request) -> httpx.Response:
+ response = route(request)
+ sent.append((calls[-1][0], clock.now))
+ return response
+
+ respx.get(url__regex=_GDELT_URL).mock(side_effect=_answer)
+ r = await ns.fetch_hostility_timeline(single, "Iran", "Israel", days=3, now=_NOW_GDELT)
+ assert r["column_status"]["sanction"] == "ok"
+ assert [c for c, _ in sent] == [
+ "articles", "threat", "sanction", "force", "criticism", "tone", "sanction"
+ ]
+ stamps = [t for _, t in sent]
+ assert min(_gaps(stamps)) >= _FLOOR, stamps
+ assert stamps[-1] - stamps[-2] >= ns._GDELT_RETRY_WAIT_S, stamps
+
+
+async def test_timeline_retry_pass_is_skipped_while_the_breaker_is_open(
+ fetcher, clock: _Clock
+) -> None:
+ # F5 (M10): three failed columns open the GDELT breaker; the retry pass
+ # must not pause and ask again, and says why it did not.
+ route, calls = _gdelt_router(fail={"threat", "sanction", "force"})
+ respx.get(url__regex=_GDELT_URL).mock(side_effect=route)
+ r = await ns.fetch_hostility_timeline(fetcher, "Iran", "Israel", days=3, now=_NOW_GDELT)
+ assert any(g.startswith("retry pass skipped: the GDELT circuit breaker is open")
+ for g in r["data_gaps"]), r["data_gaps"]
+ assert ns._GDELT_RETRY_WAIT_S not in clock.sleeps
+ # articles once, three attempts for each failing column, nothing after.
+ assert len(calls) == 1 + 3 * 3
+ assert r["column_status"]["criticism"] == "failed"
+ assert r["column_status"]["tone"] == "failed"
diff --git a/src/world_intel_mcp/tests/test_news_signals_guarantees.py b/src/world_intel_mcp/tests/test_news_signals_guarantees.py
new file mode 100644
index 0000000..65a3c9d
--- /dev/null
+++ b/src/world_intel_mcp/tests/test_news_signals_guarantees.py
@@ -0,0 +1,308 @@
+"""News signals: the second review's findings outside the GDELT floor.
+
+Second review of 2026-10-17. Each test names its finding: demonyms the
+whole-word dossier matcher stopped counting (NS-1), the CLI partial-outage
+line that never printed for situations and top stories (NS-7 / F11), a
+searched span sorted as text (NS-8), empty and one-letter countries sent
+to GDELT (NS-9), and the boundaries no test pinned (F6, F7, F8), plus an
+event-loop test that does not count ticks (F12).
+
+Gaps / not covered: real feeds and real GDELT (no network); the demonym
+rule is checked on the forms below, not on every headline form.
+"""
+
+import asyncio
+import threading
+
+import httpx
+import pytest
+import respx
+from click.testing import CliRunner
+
+from world_intel_mcp import cli
+from world_intel_mcp.analysis import clustering, dossier
+from world_intel_mcp.analysis import news_signals as ns
+from world_intel_mcp.sources import elections, intelligence, markets, news, sanctions
+from world_intel_mcp.sources.news import _RSS_FEEDS, fetch_news_feed
+from world_intel_mcp.tests.test_news_signals import (
+ _DERIVED,
+ _NOW,
+ _NOW_GDELT,
+ _SITUATION_ITEMS,
+ _column,
+ _feed,
+ _gdelt_router,
+ _patch_feed,
+ _top_items,
+ _volraw_dropping,
+)
+
+_GDELT_URL = r".*api\.gdeltproject\.org.*"
+
+
+def _no_sleep(monkeypatch: pytest.MonkeyPatch) -> None:
+ async def _skip(*args, **kwargs) -> None:
+ return None
+
+ monkeypatch.setattr(asyncio, "sleep", _skip)
+
+
+def _other_sections(monkeypatch: pytest.MonkeyPatch) -> None:
+ """The dossier's four non-news sources, answered locally."""
+
+ async def _brief(*args, **kwargs) -> dict:
+ return {"supporting_data": {}, "brief": ""}
+
+ async def _stocks(*args, **kwargs) -> dict:
+ return {"error": "not under test"}
+
+ async def _elections(*args, **kwargs) -> dict:
+ return {"elections": []}
+
+ async def _sanctions(*args, **kwargs) -> dict:
+ return {"results": [], "count": 0}
+
+ monkeypatch.setattr(intelligence, "fetch_country_brief", _brief)
+ monkeypatch.setattr(markets, "fetch_country_stocks", _stocks)
+ monkeypatch.setattr(elections, "fetch_election_calendar", _elections)
+ monkeypatch.setattr(sanctions, "fetch_sanctions_search", _sanctions)
+
+
+def _news_item(title: str, published) -> dict:
+ return {
+ "title": title,
+ "link": "https://news.example/x",
+ "published": published,
+ "summary": "",
+ "feed_name": "BBC World",
+ "category": "geopolitics",
+ }
+
+
+# ---------------------------------------------------------------------------
+# Dossier
+# ---------------------------------------------------------------------------
+
+
+ "iso3,title,expected",
+ [
+ # NS-1: counted by the substring matcher before a5ced4a, lost by the
+ # whole-word one.
+ ("PRK", "North Korean troops cross into Russia", True),
+ ("PRK", "North Koreans fighting for Russia captured", True),
+ ("PSE", "Gazans flee south as strikes resume", True),
+ ("PSE", "Gazan hospital runs out of fuel", True),
+ ("AFG", "Afghans queue at the border", True),
+ # The new endings must not reopen the traps the matcher closed.
+ ("IRN", "Tirana mayor resigns", False),
+ ("IND", "Indiana governor signs bill", False),
+ ("IND", "Storm builds over the Indian Ocean", False),
+ ("PRK", "South Korea holds drills", False),
+ ],
+)
+def test_dossier_matcher_counts_demonym_endings(iso3: str, title: str, expected: bool) -> None:
+ assert dossier._names_country(title, iso3) is expected
+
+
+async def test_dossier_mentions_not_capped_when_every_match_is_listed(
+ fetcher, _other_sections, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # F8 (M20): with `>=`, three matches all listed read as capped.
+ items = [_news_item(f"Iran talks round {i}", f"2026-10-17T1{i}:00:00Z") for i in range(3)]
+ _patch_feed(monkeypatch, _feed(items))
+ section = (await dossier.fetch_country_dossier(fetcher, country="IR"))["news"]
+ assert section["matching_items"] == 3
+ assert section["mention_count"] == 3
+ assert section["mention_count_capped"] is False
+
+
+async def test_dossier_searched_span_uses_parsed_times_only(
+ fetcher, _other_sections, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # NS-8: the span sorted raw strings, so a date the feed could not parse
+ # ("Okt 17, 2026 (Ortszeit)") was reported as the newest item.
+ items = [
+ _news_item("Iran talks resume", "2026-10-16T09:00:00Z"),
+ _news_item("Iran envoy arrives", "Okt 17, 2026 (Ortszeit)"),
+ _news_item("Iran deal signed", "2026-10-17T10:00:00Z"),
+ _news_item("Pope returns", None),
+ ]
+ _patch_feed(monkeypatch, _feed(items))
+ section = (await dossier.fetch_country_dossier(fetcher, country="IR"))["news"]
+ assert section["searched_span"] == {
+ "oldest": "2026-10-16T09:00:00Z",
+ "newest": "2026-10-17T10:00:00Z",
+ }
+ assert section["items_without_parsed_time"] == 2
+ assert section["items_searched"] == 4
+
+
+# ---------------------------------------------------------------------------
+# RSS outage threshold
+# ---------------------------------------------------------------------------
+
+
+_SECURITY = _RSS_FEEDS["security"]
+_RSS = """<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0"><channel><title>Feed</title>
+<item>
+ <title>Sanctions talks resume in Geneva</title>
+ <link>https://news.example/1</link>
+ <pubDate>Sat, 17 Oct 2026 14:00:00 GMT</pubDate>
+</item>
+</channel></rss>"""
+
+
+async def test_fetch_news_feed_three_of_eight_feeds_live_is_degraded(fetcher, _no_sleep) -> None:
+ # F7 (M19): only 4 of 8 (not degraded) was pinned, which holds for any
+ # threshold up to one half; 3 of 8 must be marked.
+ assert len(_SECURITY) == 8
+ for _, url in _SECURITY[:5]:
+ respx.get(url).mock(return_value=httpx.Response(503))
+ respx.route().mock(return_value=httpx.Response(200, text=_RSS))
+ result = await fetch_news_feed(fetcher, category="security")
+ assert (result["feeds_ok"], result["feeds_attempted"]) == (3, 8)
+ assert result["degraded"] is True
+ assert result["reason"] == "partial_feeds_failed"
+ assert "error" not in result
+
+
+# ---------------------------------------------------------------------------
+# Hostility timeline
+# ---------------------------------------------------------------------------
+
+
+async def test_hostility_timeline_hours_missing_is_the_union_of_partial_columns(
+ fetcher, _no_sleep, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # F6 (M15): two columns partial on disjoint hours; the last column's
+ # hours alone would tell a reader 00-05 were complete.
+ monkeypatch.setattr(ns, "_GDELT_RETRY_WAIT_S", 0.0)
+ route, _ = _gdelt_router()
+ drops = {
+ "threat": lambda d: d.startswith("20261016") and int(d[9:11]) < 6,
+ "force": lambda d: d.startswith("20261016") and int(d[9:11]) >= 12,
+ }
+
+ def _answer(request: httpx.Request) -> httpx.Response:
+ col = _column(request)
+ if col in drops:
+ return httpx.Response(200, json=_volraw_dropping(
+ request.url.params["query"], _DERIVED[col], drops[col],
+ ))
+ return route(request)
+
+ respx.get(url__regex=_GDELT_URL).mock(side_effect=_answer)
+ r = await ns.fetch_hostility_timeline(fetcher, "Iran", "Israel", days=3, now=_NOW_GDELT)
+ d16 = next(d for d in r["days"] if d["date"] == "2026-10-16")
+ assert d16["partial_columns"] == ["threat", "force"]
+ assert d16["hours_missing"] == [f"{h:02d}" for h in (*range(0, 6), *range(12, 24))]
+
+
+async def test_hostility_timeline_refuses_empty_and_one_letter_countries(
+ fetcher, _no_sleep
+) -> None:
+ # NS-9: '' or 'X' went out as nine GDELT requests and could open the
+ # breaker every GDELT caller shares.
+ route, calls = _gdelt_router()
+ respx.get(url__regex=_GDELT_URL).mock(side_effect=route)
+ for country, other in (("", None), (" ", None), ("X", None), ("Iran", " ")):
+ r = await ns.fetch_hostility_timeline(fetcher, country, other, now=_NOW_GDELT)
+ assert "too short to search" in r["error"], r
+ assert r["days"] == []
+ assert calls == []
+
+
+# ---------------------------------------------------------------------------
+# CLI partial-outage line
+# ---------------------------------------------------------------------------
+
+
+def _cli_env(tmp_path, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("WORLD_INTEL_CACHE_DB", str(tmp_path / "cli-cache.db"))
+ monkeypatch.setenv("COLUMNS", "200")
+ monkeypatch.setattr(cli, "_fetcher", None)
+
+
[email protected]("command,items", [("situations", _SITUATION_ITEMS), ("top-stories", None)])
+def test_cli_situations_and_top_stories_print_the_partial_outage_line(
+ _cli_env, monkeypatch: pytest.MonkeyPatch, command: str, items
+) -> None:
+ # NS-7 / F11: the line read a top-level feeds_attempted these two
+ # results did not carry, so only `intel news` and `intel trending`
+ # ever printed it.
+ marker = {"degraded": True, "reason": "partial_feeds_failed", "feeds_ok": 3,
+ "feeds_attempted": 119, "feeds_failed": ["CFR"]}
+ _patch_feed(monkeypatch, _feed(items if items is not None else _top_items(), **marker))
+ result = CliRunner().invoke(cli.main, [command], catch_exceptions=False)
+ assert result.exit_code == 0
+ assert "Partial outage: only 3 of 119 RSS feeds answered live" in result.output
+
+
+async def test_situations_and_top_stories_carry_the_feed_counts(
+ fetcher, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ marker = {"degraded": True, "reason": "partial_feeds_failed", "feeds_ok": 1,
+ "feeds_attempted": 3}
+ _patch_feed(monkeypatch, _feed(_SITUATION_ITEMS, **marker))
+ r = await ns.fetch_situations(fetcher, min_articles=1, now=_NOW)
+ assert (r["feeds_ok"], r["feeds_attempted"]) == (1, 3)
+ _patch_feed(monkeypatch, _feed(_top_items(), **marker))
+ t = await ns.fetch_top_stories(fetcher, now=_NOW)
+ assert (t["feeds_ok"], t["feeds_attempted"]) == (1, 3)
+
+
+# ---------------------------------------------------------------------------
+# Clustering off the event loop, without counting ticks
+# ---------------------------------------------------------------------------
+
+
+def _gated_clusterer(monkeypatch: pytest.MonkeyPatch, module, gate: threading.Event,
+ seen: list[bool]) -> None:
+ real = module.cluster_articles
+
+ def _waits_for_the_loop(*args, **kwargs):
+ # Released by a coroutine on the event loop: if clustering ran on
+ # the loop, that coroutine could not run and the wait times out.
+ seen.append(gate.wait(timeout=5.0))
+ return real(*args, **kwargs)
+
+ monkeypatch.setattr(module, "cluster_articles", _waits_for_the_loop)
+
+
+async def _with_loop_release(gate: threading.Event, coro):
+ async def _release() -> None:
+ await asyncio.sleep(0.05)
+ gate.set()
+
+ releaser = asyncio.create_task(_release())
+ try:
+ return await coro
+ finally:
+ await releaser
+
+
+async def test_clustering_lets_the_event_loop_run_meanwhile(
+ fetcher, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # F12: the earlier test counted 10 ms ticks and failed under machine
+ # load (8 ticks counted); this one asks only whether the loop ran.
+ _patch_feed(monkeypatch, _feed(_top_items()))
+ gate, seen = threading.Event(), []
+ _gated_clusterer(monkeypatch, ns, gate, seen)
+ r = await _with_loop_release(gate, ns.fetch_top_stories(fetcher, now=_NOW))
+ assert r["count"] == 1
+ assert seen == [True]
+
+ gate, seen = threading.Event(), []
+ _gated_clusterer(monkeypatch, clustering, gate, seen)
+ c = await _with_loop_release(gate, clustering.fetch_news_clusters(fetcher))
+ assert c["cluster_count"] >= 1
+ assert seen == [True]