Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(monitoring): harden the #90 vendor_status gates after adversarial review

Rufus95de098d65c5
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index 868334a1..86566faa 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -1334,7 +1334,7 @@ external_monitoring:
`Source` instance that handles it. New adapters register here;
add a top-level fact in this YAML when they ship.
shipped:
- - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; fallbacks are a label leading the body text and an AWS-style [RESOLVED] title marker) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."
+ - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; only the FIRST such label is read — it is the newest update, and falling through to an older one would report a resolved incident as open; fallbacks, used only when a body carries no such markup, are a known label anywhere in the body text and an AWS-style [RESOLVED] title marker, which can only ever say CLOSED, so an open incident is not detectable on AWS-shaped feeds and is reported on the vendor's next update item instead) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."
- "rss — generic Atom/RSS feed adapter via feedparser; handles competitor changelogs, layoffs.fyi, USPTO patent feeds, regulatory filings, blog RSS. Severity hint LOW by default (most RSS is noisy). Optional `trigger_json.keywords` filters by title + summary. Each entry's <pubDate>/<updated> is parsed into Signal.published_at; declares `seed_on_first_poll` so a new watch baselines the feed's existing entries instead of firing them, and entries older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are recorded as `suppressed_stale` (issue #80)."
- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."
- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."
diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
index ed5e6b31..d2032937 100644
--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
@@ -1,7 +1,7 @@
{
"section_id": "external_monitoring",
"title": "External Monitoring",
- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / …) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",
+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — only that first label is read, since falling through to an older one would report a resolved incident as open) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and skips the stale gate, since an incident the adapter calls still-happening may not have been updated for days. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",
"mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",
- "generated_at": "2026-09-07T18:35:50+00:00"
+ "generated_at": "2026-09-07T18:51:09+00:00"
}
diff --git a/packages/core/openexecutive/monitoring/pipeline.py b/packages/core/openexecutive/monitoring/pipeline.py
index a3814e99..d9bc7cbf 100644
--- a/packages/core/openexecutive/monitoring/pipeline.py
+++ b/packages/core/openexecutive/monitoring/pipeline.py
@@ -28,6 +28,7 @@ Design notes:
from __future__ import annotations
import logging
+from collections.abc import Callable
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -451,8 +452,20 @@ class _LostRace:
"""
cutoff: str
+ # Entries our own adapter carved out of the baseline (see
+ # ``_split_baseline_exempt``). The winner didn't baseline these either
+ # — it promotes them after its transaction — so treating one as
+ # "already recorded" here is exactly how a live outage goes missing:
+ # we unblock the moment the winner commits, while it still has up to
+ # 200 audit inserts to get through, so the loser reaches the open
+ # incident FIRST and would burn its dedup key as baseline. Never
+ # covered, whatever its date; the UNIQUE(dedup_key) constraint keeps
+ # the two scans from both promoting it.
+ exempt: frozenset[str] = frozenset()
def covers(self, signal: Signal) -> bool:
+ if signal.dedup_key in self.exempt:
+ return False
published = _published(signal)
if published is None:
# Undated, or a bad date. Fail closed — it must not become an
@@ -468,7 +481,11 @@ class _LostRace:
def _lost_race(
- item: WatchlistItem, fetched_at: datetime, *, db_path: Path | None = None,
+ item: WatchlistItem,
+ fetched_at: datetime,
+ exempt: frozenset[str] = frozenset(),
+ *,
+ db_path: Path | None = None,
) -> _LostRace:
"""Build the lost-race policy from the winner's stamp."""
assert item.id is not None
@@ -485,11 +502,14 @@ def _lost_race(
"visible — treating this tick's entries as baseline", item.slug,
)
cutoff = (fetched_at + _RACE_CLOCK_TOLERANCE).isoformat()
- return _LostRace(cutoff)
+ return _LostRace(cutoff, exempt)
def _split_baseline_exempt(
- src: Source, item: WatchlistItem, entries: list[Signal],
+ src: Source,
+ item: WatchlistItem,
+ entries: list[Signal],
+ matches: Callable[[Signal], bool],
) -> tuple[list[Signal], list[Signal]]:
"""Split a seeding row's first-poll entries into (archive, exempt).
@@ -499,6 +519,13 @@ def _split_baseline_exempt(
baselined exactly as before. An adapter without the hook exempts
nothing, which is the historical behaviour for ``rss`` / ``edgar``.
+ An entry that misses the row's trigger filter stays in ``archive``
+ however live it is. The baseline's guarantee is that EVERYTHING in the
+ feed is recorded, trigger misses included, so that widening a trigger
+ later cannot resurface what was already there — and an exempt entry
+ that missed the trigger would be dropped unrecorded by the cascade,
+ punching a hole in exactly that guarantee.
+
A hook that raises is treated as "not exempt" — the same fail-quiet
stance ``matches_trigger`` takes, and the safe one: a crashing hook
must not turn a first poll into a back-catalogue replay.
@@ -510,7 +537,7 @@ def _split_baseline_exempt(
exempt: list[Signal] = []
for signal in entries:
try:
- is_exempt = bool(hook(signal, item))
+ is_exempt = bool(hook(signal, item)) and matches(signal)
except Exception:
logger.exception(
"monitoring.pipeline: promote_on_baseline crashed on %r — "
@@ -529,8 +556,10 @@ def _record_baseline(
db_path: Path | None = None,
) -> int | None:
"""Try to baseline a seeding row: claim the stamp and record every
- fetched entry — trigger misses included — as ``suppressed_baseline``
- in ONE transaction.
+ entry it is handed — the row's back-catalogue, trigger misses
+ included, adapter-exempted entries excluded (see
+ ``_split_baseline_exempt``) — as ``suppressed_baseline`` in ONE
+ transaction.
Returns how many rows were recorded, or ``None`` when another scan won
the claim first (nothing written). The claim is a compare-and-swap on
@@ -694,13 +723,36 @@ async def _poll_one_watchlist_item(
# watch is added surfaces on that first poll while the resolved
# archive is still swallowed. Exempt entries skip the baseline write
# and fall through to the ordinary cascade below.
+ # Apply the per-item trigger filter (default: True). Entries that miss
+ # it are dropped — unless this scan lost the baseline race, in which
+ # case a miss the winner didn't hold is still recorded as baseline so a
+ # later, wider trigger can't resurface it (same guarantee the winner
+ # gives by recording misses).
+ def _matches(signal: Signal) -> bool:
+ try:
+ return bool(src.matches_trigger(signal, item))
+ except Exception:
+ logger.exception(
+ "monitoring.pipeline: matches_trigger crashed on %r — "
+ "treating as no-match", item.slug
+ )
+ return False
+
lost_race: _LostRace | None = None
+ # Entries the adapter carved out of its own baseline. They bypass the
+ # age gate below: the adapter has declared them still happening, and
+ # judging them by an <updated> stamp that can be days old (a
+ # long-running incident updates rarely) would record one as stale,
+ # burning its dedup key and muting it until the vendor next touches it
+ # — the regression the exemption exists to prevent.
+ exempt_keys: set[str] = set()
if (
getattr(src, "seed_on_first_poll", False)
and item.id is not None
and item.baselined_at is None
):
- archive, exempt = _split_baseline_exempt(src, item, capped)
+ archive, exempt = _split_baseline_exempt(src, item, capped, _matches)
+ exempt_keys = {signal.dedup_key for signal in exempt}
recorded = _record_baseline(item, archive, fetched_at, db_path=db_path)
if recorded is not None:
tally.written += recorded
@@ -708,28 +760,22 @@ async def _poll_one_watchlist_item(
return
# The stamp is claimed and the archive is recorded, so this row
# is no longer seeding: the exempt entries run the ordinary
- # cascade. Crash-safe in the direction that matters — dying
- # here leaves them unrecorded, and the next tick (no longer a
- # first poll) promotes them normally instead of replaying the
- # archive.
+ # cascade. Dying here leaves them unrecorded, and the next tick
+ # (no longer a first poll) promotes them normally instead of
+ # replaying the archive. The one gap is a concurrent scan: if
+ # it lost the CAS while we were fetching, our stamp covers the
+ # exempt entries it also holds, so it may record one as
+ # baseline before we promote it — muted until its next
+ # <updated>, which for a live incident is minutes away.
capped = exempt
else:
- lost_race = _lost_race(item, fetched_at, db_path=db_path)
-
- # Apply the per-item trigger filter (default: True). Entries that miss
- # it are dropped — unless this scan lost the baseline race, in which
- # case a miss the winner didn't hold is still recorded as baseline so a
- # later, wider trigger can't resurface it (same guarantee the winner
- # gives by recording misses).
- def _matches(signal: Signal) -> bool:
- try:
- return bool(src.matches_trigger(signal, item))
- except Exception:
- logger.exception(
- "monitoring.pipeline: matches_trigger crashed on %r — "
- "treating as no-match", item.slug
+ # Lost the claim: keep the whole feed (the winner's stamp
+ # decides what counts as baseline) but carry the exempt keys
+ # into that policy, so an open incident is judged as news here
+ # too rather than swallowed by the winner's cutoff.
+ lost_race = _lost_race(
+ item, fetched_at, frozenset(exempt_keys), db_path=db_path,
)
- return False
# On a race tick the deferral bound is the tight drift tolerance, not
# the lenient configured skew: nothing we fetched can honestly be dated
@@ -788,7 +834,7 @@ async def _poll_one_watchlist_item(
)
continue
- if _is_stale(signal, now, max_age_days):
+ if signal.dedup_key not in exempt_keys and _is_stale(signal, now, max_age_days):
_suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_STALE, db_path=db_path)
continue
diff --git a/packages/core/openexecutive/monitoring/sources/vendor_status.py b/packages/core/openexecutive/monitoring/sources/vendor_status.py
index c528673b..3142a453 100644
--- a/packages/core/openexecutive/monitoring/sources/vendor_status.py
+++ b/packages/core/openexecutive/monitoring/sources/vendor_status.py
@@ -34,7 +34,9 @@ incident, both #80 freshness gates become safe here:
- the row is seeded on its first poll (``seed_on_first_poll``), with
the incidents that are still OPEN exempted from the baseline by
``promote_on_baseline`` so a live outage surfaces immediately while
- the resolved archive is recorded and never promoted;
+ the resolved archive is recorded and never promoted (openness is read
+ from the entry body, so it is only detectable on feeds that publish
+ per-update status labels — see ``promote_on_baseline``);
- ``published_at`` is parsed from ``<updated>`` / ``<pubDate>``, so the
age gate and the future-date deferral judge vendor incidents the same
way they judge ``rss`` and ``edgar`` entries.
@@ -103,15 +105,28 @@ _OPEN_STATUSES = frozenset({
})
_CLOSED_STATUSES = frozenset({"resolved", "completed", "postmortem"})
-# Newest-update label in a Statuspage entry body. Bounded repetition, and
-# only over the head of the body (see _BODY_SCAN_CHARS), so a hostile feed
-# can't make this scan expensive.
-_STRONG_LABEL_RE = re.compile(r"<strong>\s*([^<>]{1,40}?)\s*</strong>", re.IGNORECASE)
+# Newest-update label in a Statuspage entry body. NO \s* padding around
+# the capture: \s is a subset of [^<>], so the two overlap and the engine
+# backtracks quadratically over a run of whitespace — a feed of entries
+# whose body is "<strong>" plus 8k spaces (well inside the 2MB fetch cap)
+# cost 3.4s PER ENTRY, synchronously, on the event loop the API serves
+# from. Whitespace is folded by _known_status anyway, so dropping the
+# padding is free: the same body now scans in ~0ms. Keep any future edit
+# to this pattern free of overlapping quantifiers.
+_STRONG_LABEL_RE = re.compile(r"<strong>([^<>]{1,40})</strong>", re.IGNORECASE)
# Same label, in a body that carries no markup for us to key on — an
-# xhtml-typed Atom <content> (real child elements, so the tags never reach
-# us as text) or a plain-text update. Statuspage's own wording puts the
-# label first: "Resolved - The issue has been fixed."
-_TEXT_LABEL_RE = re.compile(r"^\s*([A-Za-z][A-Za-z ]{1,18}?)\s*[-–—:]\s", re.IGNORECASE)
+# xhtml-typed Atom <content> (real child elements, so the tags are consumed
+# by the parser and never reach us as text) or a plain-text update. NOT
+# anchored: itertext() glues the update's timestamp onto the label
+# ("Sep 7, 19:07 UTCInvestigating - looking into it."), so an anchored
+# pattern would never fire on a real body. Safety comes from the
+# vocabulary, not the position — only a label we already know can match,
+# and only when followed by Statuspage's " - " / ": " separator.
+_TEXT_LABEL_RE = re.compile(
+ r"(" + "|".join(re.escape(s) for s in sorted(_OPEN_STATUSES | _CLOSED_STATUSES))
+ + r")\s*[-–—:]\s",
+ re.IGNORECASE,
+)
# AWS's rss/all.rss carries no per-update markup; it stamps the resolution
# into the title instead ("Service is operating normally: [RESOLVED] …").
_TITLE_MARKER_RE = re.compile(r"\[\s*(resolved|completed)\s*\]", re.IGNORECASE)
@@ -183,41 +198,47 @@ class VendorStatusSource:
item.config_json.get("vendor_label")
or _host_label(item.target)
)
- signals: list[Signal] = []
- for entry in entries[:_MAX_ENTRIES_PER_FEED]:
- entry_id = entry.get("id") or entry.get("link") or ""
- if not entry_id:
- # Without a stable upstream id we cannot dedup — skip
- # rather than emit a noisy hash-of-title signal that
- # would re-fire on every minor edit.
- continue
- title = collapse_whitespace(entry.get("title") or "") or "(untitled incident)"
- summary = f"[{vendor_label}] {title}"
- updated = entry.get("updated", "")
- signals.append(Signal(
- watchlist_id=item.id or 0,
- source_kind=self.kind,
- # The incident, not the state — two states of one incident
- # share it, which is what makes an operator's "show me this
- # incident" query work. Uniqueness lives on dedup_key.
- source_external_id=entry_id[:500],
- captured_at=datetime.now(UTC).isoformat(),
- published_at=feed_text_published_at(updated),
- normalized_summary=summary[:500],
- raw_payload={
- "vendor_label": vendor_label,
- "target_url": item.target,
- "entry_id": entry_id,
- "title": title,
- "link": entry.get("link", ""),
- "updated": updated,
- "status": _latest_status(entry.get("body", ""), title),
- },
- provenance_url=entry.get("link") or item.target,
- severity_hint=AlertSeverity.HIGH,
- dedup_key=_make_dedup_key(item.slug, entry_id, updated),
- ))
- return signals
+ built = (
+ self._build_signal(entry, item, vendor_label)
+ for entry in entries[:_MAX_ENTRIES_PER_FEED]
+ )
+ return [signal for signal in built if signal is not None]
+
+ def _build_signal(
+ self, entry: dict[str, str], item: WatchlistItem, vendor_label: str,
+ ) -> Signal | None:
+ """One parsed feed entry → one Signal, or None when it can't dedup."""
+ entry_id = entry.get("id") or entry.get("link") or ""
+ if not entry_id:
+ # Without a stable upstream id we cannot dedup — skip rather
+ # than emit a noisy hash-of-title signal that would re-fire on
+ # every minor edit.
+ return None
+ title = collapse_whitespace(entry.get("title") or "") or "(untitled incident)"
+ updated = entry.get("updated", "")
+ return Signal(
+ watchlist_id=item.id or 0,
+ source_kind=self.kind,
+ # The incident, not the state — two states of one incident
+ # share it, which is what makes an operator's "show me this
+ # incident" query work. Uniqueness lives on dedup_key.
+ source_external_id=entry_id[:500],
+ captured_at=datetime.now(UTC).isoformat(),
+ published_at=feed_text_published_at(updated),
+ normalized_summary=f"[{vendor_label}] {title}"[:500],
+ raw_payload={
+ "vendor_label": vendor_label,
+ "target_url": item.target,
+ "entry_id": entry_id,
+ "title": title,
+ "link": entry.get("link", ""),
+ "updated": updated,
+ "status": _latest_status(entry.get("body", ""), title),
+ },
+ provenance_url=entry.get("link") or item.target,
+ severity_hint=AlertSeverity.HIGH,
+ dedup_key=_make_dedup_key(item.slug, entry_id, updated),
+ )
def matches_trigger(self, signal: Signal, item: WatchlistItem) -> bool:
"""Optional keyword filter — see module docstring."""
@@ -241,6 +262,15 @@ class VendorStatusSource:
archive at HIGH. Nothing is lost for good — that entry is recorded
as baseline, and the next ``<updated>`` bump mints a new dedup key
and surfaces normally.
+
+ That fail-closed default is the whole behaviour for a feed with no
+ per-update markup, AWS's ``rss/all.rss`` among them: its resolution
+ marker lives in the title, and an OPEN state there is only
+ inferable from the ABSENCE of one, which is exactly the guess this
+ refuses to make. So a watch on such a feed added mid-incident
+ baselines that incident and reports the vendor's next update
+ instead — AWS publishes one item per update, each with its own
+ guid, so that update is a new entry the baseline no longer covers.
"""
return _is_open(str(signal.raw_payload.get("status") or ""))
@@ -338,15 +368,22 @@ def _latest_status(body: str, title: str) -> str:
closed.
"""
head = body[:_BODY_SCAN_CHARS]
- for match in _STRONG_LABEL_RE.finditer(head):
- label = _known_status(match.group(1))
- if label:
- return label
- leading = _TEXT_LABEL_RE.match(head)
- if leading:
- label = _known_status(leading.group(1))
- if label:
- return label
+ newest = _STRONG_LABEL_RE.search(head)
+ if newest is not None:
+ # This vendor uses Statuspage's per-update markup, so the FIRST
+ # bold label is the newest update's — the incident's state right
+ # now — and it is the only one worth reading. An unrecognised
+ # label means we do not understand this feed: return "" and let
+ # the caller fail closed. Scanning past it to a later <strong>
+ # would report an OLDER state, and the older labels on an incident
+ # are nearly always open ones, so a resolved incident would come
+ # back "investigating" and promote itself out of the baseline.
+ return _known_status(newest.group(1))
+ # No per-update markup anywhere in the head: fall back to a known
+ # label in the body text, then to a title marker.
+ leading = _TEXT_LABEL_RE.search(head)
+ if leading is not None:
+ return _known_status(leading.group(1))
marker = _TITLE_MARKER_RE.search(title)
return marker.group(1).lower() if marker else ""
@@ -393,11 +430,20 @@ def _make_dedup_key(slug: str, entry_id: str, updated: str) -> str:
unchanged feed still dedups (same id, same stamp), while an incident
moving Investigating → Resolved mints a new key and surfaces.
+ The stamp is NORMALISED before hashing (the same parse that fills
+ ``published_at``), so the same instant spelled two ways — ``Z`` vs
+ ``+00:00``, ``GMT`` vs ``+0000`` — is one key. A vendor changing how
+ it serialises its dates must not rekey, and thereby re-alert, every
+ open incident it has. An unparseable stamp is hashed as-is.
+
A feed that omits ``<updated>`` degrades to a stable per-incident key
(the pre-#90 behaviour) rather than to one that changes every poll —
an entry with no stamp has nothing to change, and a per-poll key
- would re-alert forever.
+ would re-alert forever. A feed that regenerates a real stamp on every
+ request is the one shape this cannot defend against; nothing in the
+ promotion path rate-limits a row, so such a feed re-alerts each tick.
"""
- payload = f"{slug}\x00{entry_id}\x00{updated}".encode()
+ stamp = feed_text_published_at(updated) or (updated or "").strip()
+ payload = f"{slug}\x00{entry_id}\x00{stamp}".encode()
digest = hashlib.sha256(payload).hexdigest()[:32]
return f"vendor_status:{digest}"
diff --git a/packages/core/tests/conftest.py b/packages/core/tests/conftest.py
index 65b8aeb1..bd0720c5 100644
--- a/packages/core/tests/conftest.py
+++ b/packages/core/tests/conftest.py
@@ -17,3 +17,37 @@ def reset_active_gateway():
set_active_gateway(None)
yield
set_active_gateway(None)
+
+
+def install_source_feed(monkeypatch: pytest.MonkeyPatch):
+ """Point one monitoring source adapter's bounded fetch at a canned body.
+
+ Every feed adapter (``vendor_status``, ``rss``, ``edgar``, …) reaches
+ the network through its own module-level ``fetch_bounded`` +
+ ``validate_target_url`` pair, so each test module used to carry its own
+ near-identical monkeypatch helper. Yields a setter:
+
+ captured = install_source_feed("edgar", b"<feed>…</feed>")
+ ...
+ assert captured["user_agent"] == ...
+
+ The returned dict records the arguments of the LAST fetch — ``url``,
+ ``max_bytes``, and any keyword the adapter passes (edgar sends
+ ``user_agent``; keys an adapter doesn't send are absent).
+ """
+ def _install(module: str, body: bytes | str) -> dict:
+ captured: dict = {}
+ payload = body.encode() if isinstance(body, str) else body
+
+ async def fake_fetch(url: str, max_bytes: int, **kwargs) -> bytes:
+ captured.clear()
+ captured.update({"url": url, "max_bytes": max_bytes, **kwargs})
+ return payload
+
+ base = f"openexecutive.monitoring.sources.{module}"
+ monkeypatch.setattr(f"{base}.fetch_bounded", fake_fetch)
+ monkeypatch.setattr(f"{base}.validate_target_url", lambda u: (True, ""))
+ return captured
+
+ return _install
diff --git a/packages/core/tests/unit/test_monitoring_edgar.py b/packages/core/tests/unit/test_monitoring_edgar.py
index be833cea..7b4d3d54 100644
--- a/packages/core/tests/unit/test_monitoring_edgar.py
+++ b/packages/core/tests/unit/test_monitoring_edgar.py
@@ -67,32 +67,14 @@ def _make_item(
)
-def _install_feed(monkeypatch: pytest.MonkeyPatch, body: bytes) -> dict[str, Any]:
- captured: dict[str, Any] = {}
-
- async def fake_fetch(url: str, max_bytes: int, *, user_agent: str | None = None) -> bytes:
- captured["url"] = url
- captured["user_agent"] = user_agent
- return body
-
- monkeypatch.setattr("openexecutive.monitoring.sources.edgar.fetch_bounded", fake_fetch)
- monkeypatch.setattr(
- "openexecutive.monitoring.sources.edgar.validate_target_url",
- lambda u: (True, ""),
- )
- return captured
-
-
# --------------------------------------------------------------------- #
# poll()
# --------------------------------------------------------------------- #
@pytest.mark.asyncio
-async def test_edgar_emits_signal_per_matching_filing(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- captured = _install_feed(monkeypatch, _SAMPLE_ATOM)
+async def test_edgar_emits_signal_per_matching_filing(install_source_feed) -> None:
+ captured = install_source_feed("edgar", _SAMPLE_ATOM)
src = EdgarSource()
signals = await src.poll(_make_item(config={"label": "Apple"}))
@@ -121,8 +103,8 @@ async def test_edgar_emits_signal_per_matching_filing(
@pytest.mark.asyncio
-async def test_edgar_dedup_stable_across_polls(monkeypatch: pytest.MonkeyPatch) -> None:
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+async def test_edgar_dedup_stable_across_polls(install_source_feed) -> None:
+ install_source_feed("edgar", _SAMPLE_ATOM)
src = EdgarSource()
item = _make_item()
first = await src.poll(item)
@@ -131,8 +113,8 @@ async def test_edgar_dedup_stable_across_polls(monkeypatch: pytest.MonkeyPatch)
@pytest.mark.asyncio
-async def test_edgar_forms_filter_override(monkeypatch: pytest.MonkeyPatch) -> None:
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+async def test_edgar_forms_filter_override(install_source_feed) -> None:
+ install_source_feed("edgar", _SAMPLE_ATOM)
src = EdgarSource()
# Opt into insider filings only.
signals = await src.poll(_make_item(trigger={"forms": ["4"]}))
@@ -142,11 +124,11 @@ async def test_edgar_forms_filter_override(monkeypatch: pytest.MonkeyPatch) -> N
@pytest.mark.asyncio
-async def test_edgar_amendment_matches_base_form(monkeypatch: pytest.MonkeyPatch) -> None:
+async def test_edgar_amendment_matches_base_form(install_source_feed) -> None:
amended = _SAMPLE_ATOM.replace(b'term="8-K"', b'term="8-K/A"').replace(
b"<title>8-K -", b"<title>8-K/A -"
)
- _install_feed(monkeypatch, amended)
+ install_source_feed("edgar", amended)
src = EdgarSource()
# Default filter contains "8-K"; the "8-K/A" amendment should match it.
signals = await src.poll(_make_item())
@@ -155,9 +137,7 @@ async def test_edgar_amendment_matches_base_form(monkeypatch: pytest.MonkeyPatch
@pytest.mark.asyncio
-async def test_edgar_filter_then_cap_does_not_starve(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_edgar_filter_then_cap_does_not_starve(install_source_feed) -> None:
"""Excluded forms at the front of the feed must not consume the output cap."""
# 5 Form-4 entries (excluded by default) followed by one 8-K.
entries = "".join(
@@ -183,7 +163,7 @@ async def test_edgar_filter_then_cap_does_not_starve(
<updated>2026-05-29T09:00:00-04:00</updated>
</entry></feed>"""
)
- _install_feed(monkeypatch, feed)
+ install_source_feed("edgar", feed)
src = EdgarSource()
signals = await src.poll(_make_item()) # default forms exclude Form 4
# The 8-K behind the Form-4 run still surfaces.
@@ -193,8 +173,8 @@ async def test_edgar_filter_then_cap_does_not_starve(
@pytest.mark.asyncio
-async def test_edgar_bad_target_skipped(monkeypatch: pytest.MonkeyPatch) -> None:
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+async def test_edgar_bad_target_skipped(install_source_feed) -> None:
+ install_source_feed("edgar", _SAMPLE_ATOM)
src = EdgarSource()
assert await src.poll(_make_item(target="")) == []
assert await src.poll(_make_item(target="../etc/passwd")) == []
diff --git a/packages/core/tests/unit/test_monitoring_pipeline.py b/packages/core/tests/unit/test_monitoring_pipeline.py
index f61bbb56..f157c3e7 100644
--- a/packages/core/tests/unit/test_monitoring_pipeline.py
+++ b/packages/core/tests/unit/test_monitoring_pipeline.py
@@ -11,6 +11,7 @@ from __future__ import annotations
import asyncio
import sqlite3
+import time
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -585,22 +586,9 @@ _VENDOR_ATOM = """<?xml version="1.0" encoding="UTF-8"?>
"""
-def _install_vendor_feed(monkeypatch: pytest.MonkeyPatch, body: str) -> None:
- """Point the REAL vendor_status adapter at a canned feed."""
- async def fake_fetch(url: str, max_bytes: int) -> bytes:
- return body.encode()
-
- monkeypatch.setattr(
- "openexecutive.monitoring.sources.vendor_status.fetch_bounded", fake_fetch
- )
- monkeypatch.setattr(
- "openexecutive.monitoring.sources.vendor_status.validate_target_url",
- lambda u: (True, ""),
- )
-
-
def test_vendor_status_first_poll_alerts_the_outage_not_the_archive(
db: Path,
+ install_source_feed,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""End to end on the REAL adapter (issue #90's acceptance criteria):
@@ -608,7 +596,7 @@ def test_vendor_status_first_poll_alerts_the_outage_not_the_archive(
resolved history, and the incident re-fires when its status changes."""
promoted = _install_promotion_recorder(monkeypatch)
_insert_feed(db, "vendor-stripe", target="https://status.stripe.com/history.atom")
- _install_vendor_feed(monkeypatch, _VENDOR_ATOM.format(
+ install_source_feed("vendor_status", _VENDOR_ATOM.format(
open_updated=_iso_days_ago(0.02),
open_status="Investigating",
open_note="We are looking into elevated error rates.",
@@ -635,7 +623,7 @@ def test_vendor_status_first_poll_alerts_the_outage_not_the_archive(
# The incident resolves — a new <updated>, so a new dedup key, so the
# status change reaches the principal instead of being muted for good.
- _install_vendor_feed(monkeypatch, _VENDOR_ATOM.format(
+ install_source_feed("vendor_status", _VENDOR_ATOM.format(
open_updated=_iso_days_ago(0.01),
open_status="Resolved",
open_note="Error rates are back to normal.",
@@ -645,6 +633,67 @@ def test_vendor_status_first_poll_alerts_the_outage_not_the_archive(
assert len(promoted) == 2
+def test_baseline_exempt_entry_bypasses_the_age_gate(
+ db: Path,
+ install_fake_source,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """An adapter that exempts an entry has declared it still happening, so
+ the age gate must not judge it by its <updated> stamp: a long-running
+ incident whose last update is 8 days old would be recorded as stale,
+ burning its dedup key and muting it until the vendor next touches it."""
+ promoted = _install_promotion_recorder(monkeypatch)
+ _insert_feed(db, "vendor-stripe")
+ install_fake_source(
+ [
+ _make_signal(dedup_key="vendor_status:quiet", published_at=_iso_days_ago(8)),
+ _make_signal(dedup_key="vendor_status:old", published_at=_iso_days_ago(90)),
+ ],
+ seed=True,
+ promote=lambda signal, item: signal.dedup_key.endswith(":quiet"),
+ )
+
+ asyncio.run(mp.run_external_monitor_scan(db_path=db))
+
+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}
+ assert outcomes["vendor_status:quiet"] == OUTCOME_ALERTED
+ assert outcomes["vendor_status:old"] == OUTCOME_SUPPRESSED_BASELINE
+ assert [e.external_id for e in promoted] == ["vendor_status:quiet"]
+
+
+def test_baseline_exempt_trigger_miss_is_still_recorded(
+ db: Path,
+ install_fake_source,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The baseline records trigger misses so that widening a trigger later
+ cannot resurface what was already in the feed. An exempt entry that
+ misses the trigger must therefore be baselined, not dropped — dropping
+ it unrecorded would let it fire as news after a trigger change."""
+ promoted = _install_promotion_recorder(monkeypatch)
+ _insert_feed(db, "vendor-stripe")
+ fake = install_fake_source(
+ [_make_signal(dedup_key="vendor_status:live", summary="API latency")],
+ seed=True,
+ promote=lambda signal, item: True,
+ )
+ fake.match = lambda signal: "payments" in signal.normalized_summary
+
+ written = asyncio.run(mp.run_external_monitor_scan(db_path=db))
+
+ assert written == 1 # recorded despite missing the trigger
+ assert promoted == []
+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}
+ assert outcomes == {"vendor_status:live": OUTCOME_SUPPRESSED_BASELINE}
+
+ # Operator widens the trigger: the entry must NOT resurface as news.
+ fake.match = None
+ item = ms.list_watchlist(db_path=db)[0]
+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)
+ asyncio.run(mp.run_external_monitor_scan(db_path=db))
+ assert promoted == []
+
+
def test_every_entry_exempt_still_stamps_the_baseline(
db: Path,
install_fake_source,
@@ -877,6 +926,62 @@ def _another_scan_wins_while_fetching(
fake.poll = poll # type: ignore[method-assign]
+def test_loser_of_the_baseline_race_still_promotes_an_open_incident(
+ db: Path,
+ install_fake_source,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The winner of the CAS does not baseline an adapter-exempted entry —
+ it promotes it AFTER committing, behind a queue of audit writes — so a
+ scan that loses the race must not swallow that entry as back-catalogue
+ on the strength of the winner's stamp. It unblocks the instant the
+ winner commits and would otherwise reach the open incident first and
+ burn its dedup key, muting the outage the exemption exists to surface."""
+ promoted = _install_promotion_recorder(monkeypatch)
+ wl_id = _insert_feed(db, "vendor-stripe")
+ stamp = datetime.now(UTC) - _WINNER_STAMP_AGE
+ fake = install_fake_source(
+ [
+ # Dated BEFORE the winner's stamp, so the cutoff alone would
+ # class it as back-catalogue.
+ _make_signal(
+ dedup_key="vendor_status:open",
+ published_at=(stamp - timedelta(minutes=30)).isoformat(),
+ ),
+ _make_signal(
+ dedup_key="vendor_status:archived",
+ published_at=(stamp - timedelta(days=200)).isoformat(),
+ ),
+ ],
+ seed=True,
+ promote=lambda signal, item: signal.dedup_key.endswith(":open"),
+ )
+ _another_scan_wins_while_fetching(fake, wl_id, db, stamp=stamp)
+
+ asyncio.run(mp.run_external_monitor_scan(db_path=db))
+
+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}
+ assert outcomes["vendor_status:open"] == OUTCOME_ALERTED
+ assert outcomes["vendor_status:archived"] == OUTCOME_SUPPRESSED_BASELINE
+ assert [e.external_id for e in promoted] == ["vendor_status:open"]
+
+
+def test_status_scan_is_linear_on_a_hostile_body() -> None:
+ """The <strong>-label pattern must not backtrack: \\s overlapping
+ [^<>] made a body of "<strong>" plus 8k spaces — comfortably inside the
+ 2MB fetch cap — cost seconds PER ENTRY, synchronously, on the event
+ loop the API serves from. 100 such entries per poll, every 5 minutes,
+ is a full-process stall from anyone who controls a watched URL."""
+ from openexecutive.monitoring.sources.vendor_status import _latest_status
+
+ hostile = "<strong>" + " " * 7_991 + "."
+ started = time.perf_counter()
+ assert _latest_status(hostile, "") == ""
+ # The fixed pattern scans this in ~0ms and the vulnerable one took
+ # >3s, so the bound is three orders of magnitude clear of both.
+ assert time.perf_counter() - started < 2.0
+
+
def test_scan_that_loses_baseline_race_does_not_replay_back_catalogue(
db: Path,
install_fake_source,
diff --git a/packages/core/tests/unit/test_monitoring_vendor_status.py b/packages/core/tests/unit/test_monitoring_vendor_status.py
index fecc98d0..0ea419d1 100644
--- a/packages/core/tests/unit/test_monitoring_vendor_status.py
+++ b/packages/core/tests/unit/test_monitoring_vendor_status.py
@@ -44,19 +44,6 @@ def _make_item(
)
-def _install_feed(monkeypatch: pytest.MonkeyPatch, body: bytes) -> None:
- async def fake_fetch(url: str, max_bytes: int) -> bytes:
- return body
-
- monkeypatch.setattr(
- "openexecutive.monitoring.sources.vendor_status.fetch_bounded", fake_fetch
- )
- monkeypatch.setattr(
- "openexecutive.monitoring.sources.vendor_status.validate_target_url",
- lambda u: (True, ""),
- )
-
-
def _atom_entry(
*, incident: str, title: str, updated: str, body: str,
) -> str:
@@ -168,12 +155,17 @@ _RESOLVED_BODY_TEXT = (
("<p><strong>Postmortem</strong> - write-up.</p>", "x", "postmortem"),
# Case and whitespace are the vendor's business, not ours.
("<p><STRONG> resolved </STRONG> - done.</p>", "x", "resolved"),
- # No markup at all: a label leading the body text still counts
+ # An unrecognised NEWEST label must NOT fall through to an older
+ # one: the older labels on an incident are nearly always open, so
+ # scanning on would report a resolved incident as investigating.
+ ("<p><strong>Fixed</strong> - all good.</p>"
+ "<p><strong>Investigating</strong> - looking into it.</p>", "x", ""),
+ # No markup at all: a known label in the body text still counts
# (xhtml <content>, or a vendor publishing plain-text updates).
- ("Investigating - we are looking into it.", "x", "investigating"),
+ # NOT anchored — itertext() glues the timestamp onto the label,
+ # which is what every real Statuspage body looks like here.
+ ("Sep 7, 19:07 UTCInvestigating - looking into it.", "x", "investigating"),
("Completed: the maintenance window is over.", "x", "completed"),
- # …but only at the start; mid-body prose must not be mined for one.
- ("We were monitoring - then it recovered.", "x", ""),
# No per-update markup: fall back to the title marker (AWS).
("plain text body", "[RESOLVED] Increased error rates", "resolved"),
# A <strong> that isn't a status label must not be mistaken for one.
@@ -254,10 +246,8 @@ def test_dedup_key_without_updated_is_stable_per_incident() -> None:
@pytest.mark.asyncio
-async def test_poll_emits_status_published_at_and_keys(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+async def test_poll_emits_status_published_at_and_keys(install_source_feed) -> None:
+ install_source_feed("vendor_status", _SAMPLE_ATOM)
src = VendorStatusSource()
item = _make_item(config={"vendor_label": "Stripe"})
@@ -280,12 +270,10 @@ async def test_poll_emits_status_published_at_and_keys(
@pytest.mark.asyncio
-async def test_identical_poll_produces_identical_dedup_keys(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_identical_poll_produces_identical_dedup_keys(install_source_feed) -> None:
"""Re-polling an unchanged feed must dedup — the status page returns
the same entries every five minutes."""
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+ install_source_feed("vendor_status", _SAMPLE_ATOM)
src = VendorStatusSource()
item = _make_item()
@@ -296,21 +284,19 @@ async def test_identical_poll_produces_identical_dedup_keys(
@pytest.mark.asyncio
-async def test_incident_update_mints_a_new_key(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_incident_update_mints_a_new_key(install_source_feed) -> None:
"""Same incident, one status update later: a new key, so it re-fires."""
src = VendorStatusSource()
item = _make_item()
- _install_feed(monkeypatch, _atom_feed(_OPEN_ENTRY))
+ install_source_feed("vendor_status", _atom_feed(_OPEN_ENTRY))
before = (await src.poll(item))[0]
resolved_now = _atom_entry(
incident="2001", title="Elevated API error rates",
updated="2026-09-07T13:05:00Z", body=_RESOLVED_BODY,
)
- _install_feed(monkeypatch, _atom_feed(resolved_now))
+ install_source_feed("vendor_status", _atom_feed(resolved_now))
after = (await src.poll(item))[0]
assert before.source_external_id == after.source_external_id
@@ -320,12 +306,10 @@ async def test_incident_update_mints_a_new_key(
@pytest.mark.asyncio
-async def test_promote_on_baseline_exempts_only_open_incidents(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_promote_on_baseline_exempts_only_open_incidents(install_source_feed) -> None:
"""The first-poll contract: an open incident is live news, the resolved
archive is not."""
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+ install_source_feed("vendor_status", _SAMPLE_ATOM)
src = VendorStatusSource()
item = _make_item()
@@ -337,16 +321,14 @@ async def test_promote_on_baseline_exempts_only_open_incidents(
@pytest.mark.asyncio
-async def test_unknown_status_is_baselined_on_first_poll(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_unknown_status_is_baselined_on_first_poll(install_source_feed) -> None:
"""A feed whose format we don't recognise must not promote its archive.
Nothing is lost: the next <updated> bump mints a new key."""
unknown = _atom_entry(
incident="9", title="Something happened",
updated="2026-09-07T11:30:00Z", body="plain text, no markup",
)
- _install_feed(monkeypatch, _atom_feed(unknown))
+ install_source_feed("vendor_status", _atom_feed(unknown))
src = VendorStatusSource()
item = _make_item()
@@ -357,9 +339,9 @@ async def test_unknown_status_is_baselined_on_first_poll(
@pytest.mark.asyncio
-async def test_rss_status_and_pubdate(monkeypatch: pytest.MonkeyPatch) -> None:
+async def test_rss_status_and_pubdate(install_source_feed) -> None:
"""Statuspage's RSS variant carries the same markup in <description>."""
- _install_feed(monkeypatch, _SAMPLE_RSS)
+ install_source_feed("vendor_status", _SAMPLE_RSS)
src = VendorStatusSource()
item = _make_item(target="https://status.twilio.com/history.rss")
@@ -373,8 +355,8 @@ async def test_rss_status_and_pubdate(monkeypatch: pytest.MonkeyPatch) -> None:
@pytest.mark.asyncio
-async def test_aws_rss_title_marker(monkeypatch: pytest.MonkeyPatch) -> None:
- _install_feed(monkeypatch, _AWS_RSS)
+async def test_aws_rss_title_marker(install_source_feed) -> None:
+ install_source_feed("vendor_status", _AWS_RSS)
src = VendorStatusSource()
item = _make_item(target="https://status.aws.amazon.com/rss/all.rss")
@@ -386,12 +368,11 @@ async def test_aws_rss_title_marker(monkeypatch: pytest.MonkeyPatch) -> None:
@pytest.mark.asyncio
-async def test_xhtml_content_still_yields_a_status(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_xhtml_content_still_yields_a_status(install_source_feed) -> None:
"""An Atom entry whose <content> holds real child elements (type=xhtml)
- rather than escaped markup — findtext would return only the leading
- text and lose the label."""
+ rather than escaped markup: findtext would return only the leading text
+ and lose the label, and itertext glues the update's timestamp onto the
+ label — so the text fallback must not be anchored at the start."""
feed = b"""<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<entry>
@@ -399,11 +380,11 @@ async def test_xhtml_content_still_yields_a_status(
<updated>2026-09-07T11:30:00Z</updated>
<link rel="alternate" href="https://status.example.com/incidents/5"/>
<title>Partial outage</title>
- <content type="xhtml"><p><strong>Investigating</strong> - looking.</p></content>
+ <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><small>Sep 7, 19:07 UTC</small><br/><strong>Investigating</strong> - looking.</p></div></content>
</entry>
</feed>
"""
- _install_feed(monkeypatch, feed)
+ install_source_feed("vendor_status", feed)
src = VendorStatusSource()
signal = (await src.poll(_make_item()))[0]
@@ -412,9 +393,7 @@ async def test_xhtml_content_still_yields_a_status(
@pytest.mark.asyncio
-async def test_entry_without_id_or_link_is_skipped(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
+async def test_entry_without_id_or_link_is_skipped(install_source_feed) -> None:
"""Unchanged behaviour: no stable upstream id → no reliable dedup."""
feed = b"""<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
@@ -424,15 +403,13 @@ async def test_entry_without_id_or_link_is_skipped(
</entry>
</feed>
"""
- _install_feed(monkeypatch, feed)
+ install_source_feed("vendor_status", feed)
assert await VendorStatusSource().poll(_make_item()) == []
@pytest.mark.asyncio
-async def test_keyword_trigger_still_filters_on_title(
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- _install_feed(monkeypatch, _SAMPLE_ATOM)
+async def test_keyword_trigger_still_filters_on_title(install_source_feed) -> None:
+ install_source_feed("vendor_status", _SAMPLE_ATOM)
src = VendorStatusSource()
item = _make_item(trigger={"keywords": ["api"]})
@@ -440,3 +417,25 @@ async def test_keyword_trigger_still_filters_on_title(
assert src.matches_trigger(live, item) is True
assert src.matches_trigger(archived, item) is False
+
+
+def test_dedup_key_is_stable_across_date_spellings() -> None:
+ """The same instant spelled two ways is ONE key. A vendor changing how
+ it serialises dates must not rekey — and so re-alert — every open
+ incident it has."""
+ spellings = [
+ "2026-09-07T19:07:00Z",
+ "2026-09-07T19:07:00+00:00",
+ "2026-09-07T12:07:00-07:00",
+ ]
+ keys = {_make_dedup_key("vendor-x", "Incident/7", raw) for raw in spellings}
+ assert len(keys) == 1
+
+ rfc822 = {
+ _make_dedup_key("vendor-x", "Incident/7", raw)
+ for raw in ("Mon, 07 Sep 2026 19:07:00 GMT", "Mon, 07 Sep 2026 19:07:00 +0000")
+ }
+ assert rfc822 == keys # …and the two dialects agree with each other
+
+ # An unparseable stamp is still hashed, so such a feed keeps working.
+ assert _make_dedup_key("vendor-x", "Incident/7", "whenever") != keys.pop()