Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(monitoring): parse the incident body instead of scanning it (#90)

Rufus022bce8d0e1b
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index 8fa64147..3bee851a 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -1334,7 +1334,7 @@ external_monitoring:
`Source` instance that handles it. New adapters register here;
add a top-level fact in this YAML when they ship.
shipped:
- - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; only the FIRST such label is read — it is the newest update, and falling through to an older one would report a resolved incident as open, so a first label that is unreadable for any reason (unrecognised, unclosed, longer than a label) scores unknown rather than letting the scan look past it. Status is NEVER inferred from prose: an earlier cut searched the body for any known status word plus a separator, and 'Update:' in an ordinary sentence then classified a 400-day-old RESOLVED incident as open — which, since open entries skip both the baseline and the age gate, promoted it at HIGH, i.e. #90 reintroduced. Where the markup never reaches us as text (an xhtml-typed <content> is parsed into elements) the label is taken from the element tree instead; the last resort is an AWS-style [RESOLVED] title marker, which can only ever say CLOSED, so an open incident is not detectable on AWS-shaped feeds and is reported on the vendor's next update item instead) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."
+ - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; only the FIRST label is read, and only if it belongs to the newest update. That is the whole safety property: an older label is nearly always an open status, so mistaking one for the newest reports a long-resolved incident as live — and open entries skip both the baseline and the age gate, so it promotes at HIGH. Three review rounds each found a different way to make that mistake (prose mining, a <strong>-only scan that a <b> or a comment walked past, and a tree walk that skipped an unlabelled newest update), so the body is parsed by html.parser.HTMLParser rather than scanned: a label in a comment or an attribute value is not a label, <b> counts as much as <strong>, nested emphasis and a spaced close tag are handled, and the scan stops at the first block element to CLOSE — one update is one block, so a newest update with no label yields unknown instead of the previous update's status. Body and label always come from ONE element (_entry_body), which re-serialises an xhtml-typed <content> so both feed shapes reach the parser identically. Status is NEVER inferred from prose. Last resort is an AWS-style [RESOLVED] title marker, which can only ever say CLOSED, so an open incident is not detectable on AWS-shaped feeds and is reported on the vendor's next update item instead) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."
- "rss — generic Atom/RSS feed adapter via feedparser; handles competitor changelogs, layoffs.fyi, USPTO patent feeds, regulatory filings, blog RSS. Severity hint LOW by default (most RSS is noisy). Optional `trigger_json.keywords` filters by title + summary. Each entry's <pubDate>/<updated> is parsed into Signal.published_at; declares `seed_on_first_poll` so a new watch baselines the feed's existing entries instead of firing them, and entries older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are recorded as `suppressed_stale` (issue #80)."
- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."
- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."
diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
index 2e3f36da..c810f5b1 100644
--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
@@ -1,7 +1,7 @@
{
"section_id": "external_monitoring",
"title": "External Monitoring",
- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — only that first label is read, since falling through to an older one would report a resolved incident as open, and a status is never inferred from prose, since a stray \"Update:\" in an old entry would otherwise promote a resolved incident at HIGH) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and skips the stale gate, since an incident the adapter calls still-happening may not have been updated for days. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",
+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser and only the newest update's label is read, since falling through to an older one reports a resolved incident as open, and a status is never inferred from prose, since a stray \"Update:\" in an old entry would otherwise promote a resolved incident at HIGH) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and skips the stale gate, since an incident the adapter calls still-happening may not have been updated for days. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",
"mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",
- "generated_at": "2026-09-07T19:09:17+00:00"
+ "generated_at": "2026-09-07T19:32:07+00:00"
}
diff --git a/packages/core/openexecutive/monitoring/sources/vendor_status.py b/packages/core/openexecutive/monitoring/sources/vendor_status.py
index 890cdae5..5d9b99cb 100644
--- a/packages/core/openexecutive/monitoring/sources/vendor_status.py
+++ b/packages/core/openexecutive/monitoring/sources/vendor_status.py
@@ -48,6 +48,7 @@ import hashlib
import logging
import re
from datetime import UTC, datetime
+from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import urlparse
from xml.etree.ElementTree import Element, ParseError
@@ -106,28 +107,22 @@ _OPEN_STATUSES = frozenset({
})
_CLOSED_STATUSES = frozenset({"resolved", "completed", "postmortem"})
-# Opening tag of the newest update's label. Attributes allowed (some
-# Statuspage themes emit <strong class="…">); the label TEXT is then taken
-# by plain string search, not by regex, so no pattern ever runs over
-# body-length input. That is deliberate: the previous pattern padded the
-# capture with \s*, which overlaps [^<>] and backtracked quadratically —
-# a feed of entries whose body was "<strong>" plus 8k spaces (well inside
-# the 2MB fetch cap) cost 3.4s PER ENTRY, synchronously, on the event loop
-# the API serves from. Keep this path free of unbounded regex.
-_STRONG_OPEN_RE = re.compile(r"<strong\b[^>]*>", re.IGNORECASE)
-_STRONG_CLOSE = "</strong>"
-# Inline tags INSIDE a label (<strong><em>Resolved</em></strong>) — stripped
-# before the vocabulary check so the emphasis doesn't hide the word.
-_TAG_RE = re.compile(r"<[^>]*>")
# AWS's rss/all.rss carries no per-update markup; it stamps the resolution
# into the title instead ("Service is operating normally: [RESOLVED] …").
_TITLE_MARKER_RE = re.compile(r"\[\s*(resolved|completed)\s*\]", re.IGNORECASE)
# The newest update sits at the top of the body; a few KB is far more than
# enough to find it and bounds the work regardless of body size.
_BODY_SCAN_CHARS = 8_000
-# A status label is a word or two. Anything longer between the tags is not
-# a label, and reading on would only find an OLDER update's.
+# A status label is a word or two. Anything longer is not a label.
_MAX_LABEL_CHARS = 200
+# Tags a vendor marks an update's status label with.
+_LABEL_TAGS = frozenset({"strong", "b"})
+# Block-level tags. One update = one block, so the first block to CLOSE
+# ends the newest update: a label found after that belongs to an older one.
+_BLOCK_TAGS = frozenset({
+ "p", "div", "li", "ul", "ol", "tr", "td", "th", "table",
+ "section", "article", "blockquote", "dd", "dt", "dl",
+})
class VendorStatusSource:
@@ -228,9 +223,7 @@ class VendorStatusSource:
"title": title,
"link": entry.get("link", ""),
"updated": updated,
- "status": _latest_status(
- entry.get("body", ""), title, entry.get("label", ""),
- ),
+ "status": _latest_status(entry.get("body", ""), title),
},
provenance_url=entry.get("link") or item.target,
severity_hint=AlertSeverity.HIGH,
@@ -306,11 +299,9 @@ def _parse_atom(feed: Element) -> list[dict[str, str]]:
if link_el is not None:
link = (link_el.get("href") or "").strip()
link = strip_url_query(link) if link else ""
- body_tags = (f"{_ATOM_NS}content", f"{_ATOM_NS}summary")
out.append({
"id": entry_id, "title": title, "link": link, "updated": updated,
- "body": _element_text(entry, body_tags),
- "label": _element_label(entry, body_tags),
+ "body": _entry_body(entry, (f"{_ATOM_NS}content", f"{_ATOM_NS}summary")),
})
return out
@@ -326,109 +317,173 @@ def _parse_rss(channel: Element) -> list[dict[str, str]]:
# Prefer guid for id (stable upstream identifier); fall back to
# the cleaned link so dedup remains stable.
entry_id = guid or link
- body_tags = ("description", _CONTENT_ENCODED)
out.append({
"id": entry_id, "title": title, "link": link, "updated": pub,
- "body": _element_text(item, body_tags),
- "label": _element_label(item, body_tags),
+ "body": _entry_body(item, ("description", _CONTENT_ENCODED)),
})
return out
-def _element_label(parent: Element, tags: tuple[str, ...]) -> str:
- """Text of the first ``<strong>`` / ``<b>`` descendant of the body.
+def _entry_body(parent: Element, tags: tuple[str, ...]) -> str:
+ """The entry's update log, as MARKUP, from ONE element.
+
+ Body and status label must come from the same place: reading the text
+ of ``<content>`` while taking the label from ``<summary>`` let an
+ entry whose body says "Resolved" report the status of a different
+ element entirely.
- For an xhtml-typed Atom ``<content>`` the markup is real elements, so
- the parser consumes the tags and ``_element_text`` returns only their
- text — with the update's timestamp glued to the label ("Sep 7, 19:07
- UTCInvestigating - …"). Reading the element tree recovers the label
- exactly, with no guessing about where it starts.
+ Feeds carry that markup two ways. Usually ``<content type="html">``
+ holds escaped HTML, so the XML parser hands us the tags as text and
+ they are already markup. An xhtml-typed ``<content>`` instead holds
+ real child elements, so the tags are consumed by the XML parser and
+ ``itertext`` would glue the update's timestamp onto its label; those
+ are re-serialised (local names only) so both shapes reach the HTML
+ parser in the same form.
"""
for tag in tags:
el = parent.find(tag)
if el is None:
continue
- for node in el.iter():
- # Namespace-agnostic: xhtml children arrive as
- # "{http://www.w3.org/1999/xhtml}strong".
- local = node.tag.rsplit("}", 1)[-1].lower() if isinstance(node.tag, str) else ""
- if local in {"strong", "b"}:
- text = "".join(node.itertext()).strip()
- if text:
- return text[:_MAX_LABEL_CHARS]
+ markup = _element_markup(el) if len(el) else (el.text or "")
+ if markup.strip():
+ return markup
return ""
-def _element_text(parent: Element, tags: tuple[str, ...]) -> str:
- """Text of the first present child in ``tags``, or "".
+def _element_markup(el: Element) -> str:
+ """Re-serialise an element's children as simple HTML, local names only.
- ``itertext`` rather than ``findtext`` so an XHTML-typed Atom
- ``<content>`` (child elements rather than escaped markup) still yields
- its text instead of an empty string.
+ Bounded by ``_BODY_SCAN_CHARS``: the newest update is at the top, and
+ an unbounded walk would let a label buried hundreds of updates deep in
+ an archive decide that the incident is open.
"""
- for tag in tags:
- el = parent.find(tag)
- if el is None:
- continue
- text = "".join(el.itertext()).strip()
- if text:
- return text
- return ""
-
-
-def _latest_status(body: str, title: str, label: str = "") -> str:
+ out: list[str] = []
+ size = 0
+
+ def emit(text: str) -> bool:
+ nonlocal size
+ out.append(text)
+ size += len(text)
+ return size < _BODY_SCAN_CHARS
+
+ def walk(node: Element) -> bool:
+ for child in node:
+ local = (
+ child.tag.rsplit("}", 1)[-1].lower()
+ if isinstance(child.tag, str) else ""
+ )
+ if not emit(f"<{local}>"):
+ return False
+ if child.text and not emit(child.text):
+ return False
+ if not walk(child):
+ return False
+ if not emit(f"</{local}>"):
+ return False
+ if child.tail and not emit(child.tail):
+ return False
+ return True
+
+ if el.text:
+ emit(el.text)
+ walk(el)
+ return "".join(out)
+
+
+def _latest_status(body: str, title: str) -> str:
"""The incident's CURRENT status, lowercased, or "" when unknown.
- Statuspage puts the newest update first in the entry body and labels
- it ``<strong>Resolved</strong>`` / ``Investigating`` / …, so the state
- right now is whatever the FIRST such label says — and only that one.
- An unrecognised first label means we do not understand this feed:
- return "" and let the caller fail closed. Reading on to a later label
- would report an OLDER state, and the older labels on an incident are
- nearly always open ones, so a resolved incident would come back
- "investigating" and promote itself out of the first-poll baseline.
-
- ``label`` is the same thing extracted structurally by the parser for a
- body whose markup never reaches us as text (an xhtml-typed Atom
- ``<content>`` holds real child elements) — see ``_element_label``.
- Last resort is a title marker, which only ever says CLOSED.
-
- Deliberately NOT inferred from prose. An earlier cut searched the body
- text for any known status word followed by a separator; "Update:" in
- an ordinary sentence then classified a long-resolved incident as open,
- which — since open entries skip both the baseline and the age gate —
- promoted a 400-day-old resolved incident at HIGH on a new watch, i.e.
- issue #90 reintroduced. A status only ever comes from a label the
- vendor marked up as one.
+ Statuspage puts the newest update first and marks its status up as a
+ label (``<strong>Resolved</strong>``), so the state right now is what
+ that ONE label says. Everything here exists to avoid reading any other
+ one: an older label is nearly always an open status, so mistaking one
+ for the newest reports a long-resolved incident as live — and since an
+ open incident skips both the first-poll baseline and the age gate,
+ that promotes a years-old resolved incident at HIGH. Three review
+ rounds each found a different way to make that mistake.
+
+ So the body is parsed by a real HTML parser (``_FirstLabel``) rather
+ than scanned, and the answer is whatever the first label element
+ contains — even if that is nothing recognisable, in which case this
+ returns "" and ``_is_open`` fails closed. A body with no label at all
+ falls through to the title marker, which can only ever say CLOSED.
+
+ Status is never inferred from prose: "Update:" in an ordinary
+ sentence is not a status label, and treating it as one is how a
+ 400-day-old resolved incident promoted itself at HIGH.
"""
- inline = _newest_label(body[:_BODY_SCAN_CHARS])
- if inline is not None:
- return _known_status(inline)
- if label:
+ label = _first_label(body[:_BODY_SCAN_CHARS])
+ if label is not None:
return _known_status(label)
marker = _TITLE_MARKER_RE.search(title)
return marker.group(1).lower() if marker else ""
-def _newest_label(head: str) -> str | None:
- """Text of the FIRST ``<strong>`` element in ``head``.
+class _FirstLabel(HTMLParser):
+ """Extracts the text of the FIRST status label in an update body.
- None when the body carries no such markup at all (the caller then
- tries its other sources); "" when markup IS present but unusable —
- unclosed, or too long to be a label — which fails closed rather than
- letting the caller look past it at an older update.
+ A real parser, so a ``<strong>`` inside a comment or an attribute
+ value is not a label, ``<b>`` counts as much as ``<strong>``, nested
+ emphasis inside a label is kept, and ``</strong >`` closes normally —
+ each of which a hand-rolled scan got wrong in a way that reported a
+ resolved incident as open.
- The label text is bounded by ``str.find`` rather than a regex, so no
- pattern ever runs over attacker-controlled body-length input.
+ ``label`` is None when the body holds no label at all (the caller
+ falls through), and a string — possibly empty — once one is found or
+ ruled out. It stops at the first block element to CLOSE: one update is
+ one block, so a label after that point belongs to an OLDER update and
+ must not be read. An unclosed label yields "" for the same reason.
"""
- opening = _STRONG_OPEN_RE.search(head)
- if opening is None:
- return None
- rest = head[opening.end():opening.end() + _MAX_LABEL_CHARS]
- close = rest.lower().find(_STRONG_CLOSE)
- if close == -1:
- return ""
- return _TAG_RE.sub("", rest[:close])
+
+ def __init__(self) -> None:
+ super().__init__(convert_charrefs=True)
+ self._parts: list[str] = []
+ self._tag: str | None = None
+ self._depth = 0
+ self._done = False
+ self._found = False
+
+ @property
+ def label(self) -> str | None:
+ if not self._found:
+ return None
+ return "".join(self._parts).strip()[:_MAX_LABEL_CHARS] if self._done else ""
+
+ def handle_starttag(self, tag: str, attrs: object) -> None:
+ if self._done:
+ return
+ if not self._found and tag in _LABEL_TAGS:
+ self._found, self._tag, self._depth = True, tag, 1
+ elif self._found and tag == self._tag:
+ self._depth += 1
+
+ def handle_endtag(self, tag: str) -> None:
+ if self._done:
+ return
+ if self._found and tag == self._tag:
+ self._depth -= 1
+ if self._depth == 0:
+ self._done = True
+ elif not self._found and tag in _BLOCK_TAGS:
+ # The newest update ended without a label. Stop rather than
+ # walk on into the previous update's.
+ self._found, self._done = True, True
+
+ def handle_data(self, data: str) -> None:
+ if self._found and not self._done:
+ self._parts.append(data)
+
+
+def _first_label(body: str) -> str | None:
+ """``_FirstLabel`` applied to one body; None when it holds no label."""
+ parser = _FirstLabel()
+ try:
+ parser.feed(body)
+ parser.close()
+ except Exception: # pragma: no cover - HTMLParser is lenient by design
+ logger.debug("vendor_status: label parse failed", exc_info=True)
+ return parser.label
+ return parser.label
def _known_status(raw: str) -> str:
diff --git a/packages/core/tests/unit/test_monitoring_pipeline.py b/packages/core/tests/unit/test_monitoring_pipeline.py
index fe8343c0..59bb8ade 100644
--- a/packages/core/tests/unit/test_monitoring_pipeline.py
+++ b/packages/core/tests/unit/test_monitoring_pipeline.py
@@ -1026,15 +1026,18 @@ def test_status_scan_is_linear_on_a_hostile_body() -> None:
is a full-process stall from anyone who controls a watched URL."""
from openexecutive.monitoring.sources.vendor_status import _latest_status
- hostile = "<strong>" + " " * 7_991 + "."
- started = time.perf_counter()
- assert _latest_status(hostile, "") == ""
- elapsed = time.perf_counter() - started
- # The label text is now bounded by str.find rather than a regex, so
- # this runs in microseconds; the vulnerable pattern took ~3s. A 0.5s
- # budget is ~1000x the real cost and still catches a regression on a
- # runner several times faster than this one — 2.0s would not have.
- assert elapsed < 0.5, f"status scan took {elapsed:.2f}s — check for backtracking"
+ # Both shapes: a long whitespace run inside a label (the original
+ # 3.4s payload) and a body that is nothing but unterminated opening
+ # tags, which a later `<strong\b[^>]*>` pattern scanned quadratically
+ # at 3.7ms/entry — 370ms per 100-entry feed on the API's event loop.
+ for hostile in ("<strong>" + " " * 7_991 + ".", ("<strong " * 1_000)[:8_000]):
+ started = time.perf_counter()
+ assert _latest_status(hostile, "") == ""
+ elapsed = time.perf_counter() - started
+ # An HTML parser walks these in well under a millisecond. The
+ # budget is ~1000x that and still an order of magnitude under the
+ # slowest regression above, on a runner far faster than this one.
+ assert elapsed < 0.1, f"status scan took {elapsed:.3f}s — not linear"
def test_scan_that_loses_baseline_race_does_not_replay_back_catalogue(
diff --git a/packages/core/tests/unit/test_monitoring_vendor_status.py b/packages/core/tests/unit/test_monitoring_vendor_status.py
index 5533de64..2a7a845f 100644
--- a/packages/core/tests/unit/test_monitoring_vendor_status.py
+++ b/packages/core/tests/unit/test_monitoring_vendor_status.py
@@ -17,6 +17,7 @@ from openexecutive.monitoring.models import WatchlistItem
from openexecutive.monitoring.sources.base import feed_text_published_at
from openexecutive.monitoring.sources.vendor_status import (
VendorStatusSource,
+ _parse_feed,
_is_open,
_latest_status,
_make_dedup_key,
@@ -202,14 +203,87 @@ def test_a_label_past_the_scan_head_fails_closed() -> None:
assert _latest_status(body, "") == ""
-def test_structural_label_is_used_when_markup_never_reaches_the_text() -> None:
- """An xhtml <content> is parsed into elements, so the tags are gone by
- the time the body is text — the label comes from the element tree."""
- assert _latest_status("Sep 7, 19:07 UTCInvestigating - looking.", "x",
- "Investigating") == "investigating"
- # …and it is still only consulted when the text carries no markup.
- assert _latest_status("<p><strong>Resolved</strong> - done.</p>", "x",
- "Investigating") == "resolved"
+ ("body", "expected"),
+ [
+ # A real parser, so none of these read a LATER (older, and so
+ # nearly always open) label by mistake. Each line was a way a
+ # resolved incident reported itself live.
+ ("<p><b>Resolved</b> - fixed.</p>"
+ "<p><strong>Investigating</strong> - looking.</p>", "resolved"),
+ ("<!-- <strong>Investigating</strong> -->"
+ "<p><strong>Resolved</strong> - fixed.</p>", "resolved"),
+ ('<p><a href="/x?q=<strong>Investigating</strong>">Resolved</a>'
+ " - fixed.</p>", ""),
+ ("<p><strong><em>Resolved</em></strong> - fixed.</p>"
+ "<p><strong>Investigating</strong> - looking.</p>", "resolved"),
+ ('<p><strong class="hl">Resolved</strong> - fixed.</p>'
+ "<p><strong>Investigating</strong> - looking.</p>", "resolved"),
+ # The newest update carries no label, or an empty one: stop at the
+ # end of its block rather than reading the previous update's.
+ ("<p>Resolved - this incident has been resolved.</p>"
+ "<p><strong>Investigating</strong> - looking.</p>", ""),
+ ("<p><strong></strong> - x.</p>"
+ "<p><strong>Investigating</strong> - looking.</p>", ""),
+ # A close tag with a space is valid HTML and must not mute a live
+ # outage.
+ ("<p><strong>Investigating</strong > - live.</p>", "investigating"),
+ ],
+)
+def test_only_the_newest_label_decides(body: str, expected: str) -> None:
+ assert _latest_status(body, "") == expected
+
+
+def test_xhtml_body_reads_only_its_newest_update() -> None:
+ """The shape `_entry_body` re-serialises: an xhtml <content> holds real
+ elements, so the tags never reach us as text. A multi-update body is
+ the case that matters — with one update, "read only the newest" is
+ vacuous."""
+ def _feed(newest: str) -> bytes:
+ return f"""<?xml version="1.0" encoding="UTF-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+ <entry>
+ <id>tag:status.example.com,2005:Incident/5</id>
+ <updated>2026-09-07T11:30:00Z</updated>
+ <link rel="alternate" href="https://status.example.com/incidents/5"/>
+ <title>Elevated errors</title>
+ <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml">
+ <p><small>12:00 UTC</small><br/>{newest}</p>
+ <p><small>09:00 UTC</small><br/><strong>Investigating</strong> - looking.</p>
+ </div></content>
+ </entry>
+</feed>""".encode()
+
+ labelled = _parse_feed(_feed("<strong>Resolved</strong> - fixed."))[0]
+ assert _latest_status(labelled["body"], labelled["title"]) == "resolved"
+
+ # Newest update unlabelled: must NOT fall through to the older
+ # "Investigating", which would promote a resolved incident at HIGH.
+ unlabelled = _parse_feed(_feed("Resolved - this incident has been resolved."))[0]
+ assert _latest_status(unlabelled["body"], unlabelled["title"]) == ""
+
+
+def test_label_never_comes_from_a_different_element_than_the_body() -> None:
+ """Body and label must be one element's: reading text from <content>
+ while taking the label from <summary> reported the status of an
+ element whose text was never used."""
+ feed = b"""<?xml version="1.0" encoding="UTF-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+ <entry>
+ <id>tag:status.example.com,2005:Incident/6</id>
+ <updated>2026-09-07T11:30:00Z</updated>
+ <link rel="alternate" href="https://status.example.com/incidents/6"/>
+ <title>Elevated errors</title>
+ <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml">
+ <p>Jun 1 12:00 UTC Resolved - all clear.</p>
+ </div></content>
+ <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml">
+ <p><b>Monitoring</b> - watching.</p>
+ </div></summary>
+ </entry>
+</feed>"""
+ entry = _parse_feed(feed)[0]
+ assert _latest_status(entry["body"], entry["title"]) == ""
def test_is_open_fails_closed_on_unknown_status() -> None: