Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(monitoring): never infer a vendor incident's status from prose (#90)

Rufusd239449195cc
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index 86566faa..8fa64147 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -1334,7 +1334,7 @@ external_monitoring:
`Source` instance that handles it. New adapters register here;
add a top-level fact in this YAML when they ship.
shipped:
- - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; only the FIRST such label is read — it is the newest update, and falling through to an older one would report a resolved incident as open; fallbacks, used only when a body carries no such markup, are a known label anywhere in the body text and an AWS-style [RESOLVED] title marker, which can only ever say CLOSED, so an open incident is not detectable on AWS-shaped feeds and is reported on the vendor's next update item instead) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."
+ - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; only the FIRST such label is read — it is the newest update, and falling through to an older one would report a resolved incident as open, so a first label that is unreadable for any reason (unrecognised, unclosed, longer than a label) scores unknown rather than letting the scan look past it. Status is NEVER inferred from prose: an earlier cut searched the body for any known status word plus a separator, and 'Update:' in an ordinary sentence then classified a 400-day-old RESOLVED incident as open — which, since open entries skip both the baseline and the age gate, promoted it at HIGH, i.e. #90 reintroduced. Where the markup never reaches us as text (an xhtml-typed <content> is parsed into elements) the label is taken from the element tree instead; the last resort is an AWS-style [RESOLVED] title marker, which can only ever say CLOSED, so an open incident is not detectable on AWS-shaped feeds and is reported on the vendor's next update item instead) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."
- "rss — generic Atom/RSS feed adapter via feedparser; handles competitor changelogs, layoffs.fyi, USPTO patent feeds, regulatory filings, blog RSS. Severity hint LOW by default (most RSS is noisy). Optional `trigger_json.keywords` filters by title + summary. Each entry's <pubDate>/<updated> is parsed into Signal.published_at; declares `seed_on_first_poll` so a new watch baselines the feed's existing entries instead of firing them, and entries older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are recorded as `suppressed_stale` (issue #80)."
- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."
- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."
diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
index d2032937..2e3f36da 100644
--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
@@ -1,7 +1,7 @@
{
"section_id": "external_monitoring",
"title": "External Monitoring",
- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — only that first label is read, since falling through to an older one would report a resolved incident as open) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and skips the stale gate, since an incident the adapter calls still-happening may not have been updated for days. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",
+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — only that first label is read, since falling through to an older one would report a resolved incident as open, and a status is never inferred from prose, since a stray \"Update:\" in an old entry would otherwise promote a resolved incident at HIGH) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and skips the stale gate, since an incident the adapter calls still-happening may not have been updated for days. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",
"mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",
- "generated_at": "2026-09-07T18:51:09+00:00"
+ "generated_at": "2026-09-07T19:09:17+00:00"
}
diff --git a/packages/core/openexecutive/monitoring/sources/vendor_status.py b/packages/core/openexecutive/monitoring/sources/vendor_status.py
index 3142a453..890cdae5 100644
--- a/packages/core/openexecutive/monitoring/sources/vendor_status.py
+++ b/packages/core/openexecutive/monitoring/sources/vendor_status.py
@@ -34,9 +34,10 @@ incident, both #80 freshness gates become safe here:
- the row is seeded on its first poll (``seed_on_first_poll``), with
the incidents that are still OPEN exempted from the baseline by
``promote_on_baseline`` so a live outage surfaces immediately while
- the resolved archive is recorded and never promoted (openness is read
- from the entry body, so it is only detectable on feeds that publish
- per-update status labels — see ``promote_on_baseline``);
+ the resolved archive is recorded and never promoted. Openness is read
+ from the label the vendor marked up on its newest update, never
+ inferred from prose, so it is only detectable on feeds that publish
+ per-update status labels — see ``promote_on_baseline``;
- ``published_at`` is parsed from ``<updated>`` / ``<pubDate>``, so the
age gate and the future-date deferral judge vendor incidents the same
way they judge ``rss`` and ``edgar`` entries.
@@ -105,34 +106,28 @@ _OPEN_STATUSES = frozenset({
})
_CLOSED_STATUSES = frozenset({"resolved", "completed", "postmortem"})
-# Newest-update label in a Statuspage entry body. NO \s* padding around
-# the capture: \s is a subset of [^<>], so the two overlap and the engine
-# backtracks quadratically over a run of whitespace — a feed of entries
-# whose body is "<strong>" plus 8k spaces (well inside the 2MB fetch cap)
-# cost 3.4s PER ENTRY, synchronously, on the event loop the API serves
-# from. Whitespace is folded by _known_status anyway, so dropping the
-# padding is free: the same body now scans in ~0ms. Keep any future edit
-# to this pattern free of overlapping quantifiers.
-_STRONG_LABEL_RE = re.compile(r"<strong>([^<>]{1,40})</strong>", re.IGNORECASE)
-# Same label, in a body that carries no markup for us to key on — an
-# xhtml-typed Atom <content> (real child elements, so the tags are consumed
-# by the parser and never reach us as text) or a plain-text update. NOT
-# anchored: itertext() glues the update's timestamp onto the label
-# ("Sep 7, 19:07 UTCInvestigating - looking into it."), so an anchored
-# pattern would never fire on a real body. Safety comes from the
-# vocabulary, not the position — only a label we already know can match,
-# and only when followed by Statuspage's " - " / ": " separator.
-_TEXT_LABEL_RE = re.compile(
- r"(" + "|".join(re.escape(s) for s in sorted(_OPEN_STATUSES | _CLOSED_STATUSES))
- + r")\s*[-–—:]\s",
- re.IGNORECASE,
-)
+# Opening tag of the newest update's label. Attributes allowed (some
+# Statuspage themes emit <strong class="…">); the label TEXT is then taken
+# by plain string search, not by regex, so no pattern ever runs over
+# body-length input. That is deliberate: the previous pattern padded the
+# capture with \s*, which overlaps [^<>] and backtracked quadratically —
+# a feed of entries whose body was "<strong>" plus 8k spaces (well inside
+# the 2MB fetch cap) cost 3.4s PER ENTRY, synchronously, on the event loop
+# the API serves from. Keep this path free of unbounded regex.
+_STRONG_OPEN_RE = re.compile(r"<strong\b[^>]*>", re.IGNORECASE)
+_STRONG_CLOSE = "</strong>"
+# Inline tags INSIDE a label (<strong><em>Resolved</em></strong>) — stripped
+# before the vocabulary check so the emphasis doesn't hide the word.
+_TAG_RE = re.compile(r"<[^>]*>")
# AWS's rss/all.rss carries no per-update markup; it stamps the resolution
# into the title instead ("Service is operating normally: [RESOLVED] …").
_TITLE_MARKER_RE = re.compile(r"\[\s*(resolved|completed)\s*\]", re.IGNORECASE)
# The newest update sits at the top of the body; a few KB is far more than
-# enough to find it and bounds the regex work regardless of body size.
+# enough to find it and bounds the work regardless of body size.
_BODY_SCAN_CHARS = 8_000
+# A status label is a word or two. Anything longer between the tags is not
+# a label, and reading on would only find an OLDER update's.
+_MAX_LABEL_CHARS = 200
class VendorStatusSource:
@@ -233,7 +228,9 @@ class VendorStatusSource:
"title": title,
"link": entry.get("link", ""),
"updated": updated,
- "status": _latest_status(entry.get("body", ""), title),
+ "status": _latest_status(
+ entry.get("body", ""), title, entry.get("label", ""),
+ ),
},
provenance_url=entry.get("link") or item.target,
severity_hint=AlertSeverity.HIGH,
@@ -309,9 +306,11 @@ def _parse_atom(feed: Element) -> list[dict[str, str]]:
if link_el is not None:
link = (link_el.get("href") or "").strip()
link = strip_url_query(link) if link else ""
+ body_tags = (f"{_ATOM_NS}content", f"{_ATOM_NS}summary")
out.append({
"id": entry_id, "title": title, "link": link, "updated": updated,
- "body": _element_text(entry, (f"{_ATOM_NS}content", f"{_ATOM_NS}summary")),
+ "body": _element_text(entry, body_tags),
+ "label": _element_label(entry, body_tags),
})
return out
@@ -327,13 +326,39 @@ def _parse_rss(channel: Element) -> list[dict[str, str]]:
# Prefer guid for id (stable upstream identifier); fall back to
# the cleaned link so dedup remains stable.
entry_id = guid or link
+ body_tags = ("description", _CONTENT_ENCODED)
out.append({
"id": entry_id, "title": title, "link": link, "updated": pub,
- "body": _element_text(item, ("description", _CONTENT_ENCODED)),
+ "body": _element_text(item, body_tags),
+ "label": _element_label(item, body_tags),
})
return out
+def _element_label(parent: Element, tags: tuple[str, ...]) -> str:
+ """Text of the first ``<strong>`` / ``<b>`` descendant of the body.
+
+ For an xhtml-typed Atom ``<content>`` the markup is real elements, so
+ the parser consumes the tags and ``_element_text`` returns only their
+ text — with the update's timestamp glued to the label ("Sep 7, 19:07
+ UTCInvestigating - …"). Reading the element tree recovers the label
+ exactly, with no guessing about where it starts.
+ """
+ for tag in tags:
+ el = parent.find(tag)
+ if el is None:
+ continue
+ for node in el.iter():
+ # Namespace-agnostic: xhtml children arrive as
+ # "{http://www.w3.org/1999/xhtml}strong".
+ local = node.tag.rsplit("}", 1)[-1].lower() if isinstance(node.tag, str) else ""
+ if local in {"strong", "b"}:
+ text = "".join(node.itertext()).strip()
+ if text:
+ return text[:_MAX_LABEL_CHARS]
+ return ""
+
+
def _element_text(parent: Element, tags: tuple[str, ...]) -> str:
"""Text of the first present child in ``tags``, or "".
@@ -351,43 +376,61 @@ def _element_text(parent: Element, tags: tuple[str, ...]) -> str:
return ""
-def _latest_status(body: str, title: str) -> str:
+def _latest_status(body: str, title: str, label: str = "") -> str:
"""The incident's CURRENT status, lowercased, or "" when unknown.
Statuspage puts the newest update first in the entry body and labels
- it ``<strong>Resolved</strong>`` / ``Investigating`` / …, so the first
- recognised label in the body head is the incident's state right now.
- Two fallbacks for feeds shaped differently: a label leading the body
- text (an xhtml-typed ``<content>`` reaches us as text with its tags
- already consumed by the parser, and some vendors publish plain-text
- updates), then a title marker (AWS stamps ``[RESOLVED]`` there and
- publishes no per-update markup at all).
-
- Returns "" for anything unrecognised rather than guessing — callers
- decide what to do with an unknown status, and ``_is_open`` fails
- closed.
+ it ``<strong>Resolved</strong>`` / ``Investigating`` / …, so the state
+ right now is whatever the FIRST such label says — and only that one.
+ An unrecognised first label means we do not understand this feed:
+ return "" and let the caller fail closed. Reading on to a later label
+ would report an OLDER state, and the older labels on an incident are
+ nearly always open ones, so a resolved incident would come back
+ "investigating" and promote itself out of the first-poll baseline.
+
+ ``label`` is the same thing extracted structurally by the parser for a
+ body whose markup never reaches us as text (an xhtml-typed Atom
+ ``<content>`` holds real child elements) — see ``_element_label``.
+ Last resort is a title marker, which only ever says CLOSED.
+
+ Deliberately NOT inferred from prose. An earlier cut searched the body
+ text for any known status word followed by a separator; "Update:" in
+ an ordinary sentence then classified a long-resolved incident as open,
+ which — since open entries skip both the baseline and the age gate —
+ promoted a 400-day-old resolved incident at HIGH on a new watch, i.e.
+ issue #90 reintroduced. A status only ever comes from a label the
+ vendor marked up as one.
"""
- head = body[:_BODY_SCAN_CHARS]
- newest = _STRONG_LABEL_RE.search(head)
- if newest is not None:
- # This vendor uses Statuspage's per-update markup, so the FIRST
- # bold label is the newest update's — the incident's state right
- # now — and it is the only one worth reading. An unrecognised
- # label means we do not understand this feed: return "" and let
- # the caller fail closed. Scanning past it to a later <strong>
- # would report an OLDER state, and the older labels on an incident
- # are nearly always open ones, so a resolved incident would come
- # back "investigating" and promote itself out of the baseline.
- return _known_status(newest.group(1))
- # No per-update markup anywhere in the head: fall back to a known
- # label in the body text, then to a title marker.
- leading = _TEXT_LABEL_RE.search(head)
- if leading is not None:
- return _known_status(leading.group(1))
+ inline = _newest_label(body[:_BODY_SCAN_CHARS])
+ if inline is not None:
+ return _known_status(inline)
+ if label:
+ return _known_status(label)
marker = _TITLE_MARKER_RE.search(title)
return marker.group(1).lower() if marker else ""
+def _newest_label(head: str) -> str | None:
+ """Text of the FIRST ``<strong>`` element in ``head``.
+
+ None when the body carries no such markup at all (the caller then
+ tries its other sources); "" when markup IS present but unusable —
+ unclosed, or too long to be a label — which fails closed rather than
+ letting the caller look past it at an older update.
+
+ The label text is bounded by ``str.find`` rather than a regex, so no
+ pattern ever runs over attacker-controlled body-length input.
+ """
+ opening = _STRONG_OPEN_RE.search(head)
+ if opening is None:
+ return None
+ rest = head[opening.end():opening.end() + _MAX_LABEL_CHARS]
+ close = rest.lower().find(_STRONG_CLOSE)
+ if close == -1:
+ return ""
+ return _TAG_RE.sub("", rest[:close])
+
+
def _known_status(raw: str) -> str:
"""``raw`` normalised to a status we know, or "" — never a guess."""
label = collapse_whitespace(raw).lower()
diff --git a/packages/core/tests/unit/test_monitoring_pipeline.py b/packages/core/tests/unit/test_monitoring_pipeline.py
index f157c3e7..fe8343c0 100644
--- a/packages/core/tests/unit/test_monitoring_pipeline.py
+++ b/packages/core/tests/unit/test_monitoring_pipeline.py
@@ -694,6 +694,58 @@ def test_baseline_exempt_trigger_miss_is_still_recorded(
assert promoted == []
+_AWS_SHAPED_RSS = """<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0">
+ <channel>
+ <title>Vendor Status</title>
+ <item>
+ <guid>https://status.example.com/#svc_1757251800</guid>
+ <title>Service is operating normally: [RESOLVED] Increased error rates</title>
+ <link>https://status.example.com/</link>
+ <pubDate>{pub}</pubDate>
+ <description>{body}</description>
+ </item>
+ </channel>
+</rss>
+"""
+
+
+ "body",
+ [
+ # Prose that names a status. Mining this was how a resolved
+ # archive entry classified itself OPEN — and an open entry skips
+ # the baseline AND the age gate, so it promoted at HIGH.
+ "Update: Between 10:19 and 11:30 the issue was resolved.",
+ "We were monitoring - then it recovered.",
+ "Between 9:00 and 11:30 PDT we experienced elevated errors.",
+ ],
+)
+def test_old_resolved_entry_is_never_promoted_on_a_first_poll(
+ db: Path,
+ install_source_feed,
+ monkeypatch: pytest.MonkeyPatch,
+ body: str,
+) -> None:
+ """Issue #90's acceptance criterion, end to end on the real adapter: a
+ watch added today must not alert on a vendor's resolved archive — not
+ for an entry whose title says [RESOLVED] and whose body merely talks
+ about statuses, and not at 400 days old."""
+ promoted = _install_promotion_recorder(monkeypatch)
+ _insert_feed(db, "vendor-aws", target="https://status.example.com/rss/all.rss")
+ install_source_feed("vendor_status", _AWS_SHAPED_RSS.format(
+ pub=(datetime.now(UTC) - timedelta(days=400)).strftime(
+ "%a, %d %b %Y %H:%M:%S +0000"),
+ body=body,
+ ))
+
+ asyncio.run(mp.run_external_monitor_scan(db_path=db))
+
+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}
+ assert list(outcomes.values()) == [OUTCOME_SUPPRESSED_BASELINE]
+ assert promoted == []
+
+
def test_every_entry_exempt_still_stamps_the_baseline(
db: Path,
install_fake_source,
@@ -977,9 +1029,12 @@ def test_status_scan_is_linear_on_a_hostile_body() -> None:
hostile = "<strong>" + " " * 7_991 + "."
started = time.perf_counter()
assert _latest_status(hostile, "") == ""
- # The fixed pattern scans this in ~0ms and the vulnerable one took
- # >3s, so the bound is three orders of magnitude clear of both.
- assert time.perf_counter() - started < 2.0
+ elapsed = time.perf_counter() - started
+ # The label text is now bounded by str.find rather than a regex, so
+ # this runs in microseconds; the vulnerable pattern took ~3s. A 0.5s
+ # budget is ~1000x the real cost and still catches a regression on a
+ # runner several times faster than this one — 2.0s would not have.
+ assert elapsed < 0.5, f"status scan took {elapsed:.2f}s — check for backtracking"
def test_scan_that_loses_baseline_race_does_not_replay_back_catalogue(
diff --git a/packages/core/tests/unit/test_monitoring_vendor_status.py b/packages/core/tests/unit/test_monitoring_vendor_status.py
index 0ea419d1..5533de64 100644
--- a/packages/core/tests/unit/test_monitoring_vendor_status.py
+++ b/packages/core/tests/unit/test_monitoring_vendor_status.py
@@ -155,19 +155,35 @@ _RESOLVED_BODY_TEXT = (
("<p><strong>Postmortem</strong> - write-up.</p>", "x", "postmortem"),
# Case and whitespace are the vendor's business, not ours.
("<p><STRONG> resolved </STRONG> - done.</p>", "x", "resolved"),
- # An unrecognised NEWEST label must NOT fall through to an older
- # one: the older labels on an incident are nearly always open, so
- # scanning on would report a resolved incident as investigating.
+ # The NEWEST label decides, and only it. Falling through to an
+ # older one would report a resolved incident as investigating,
+ # since the older labels on an incident are nearly always open.
("<p><strong>Fixed</strong> - all good.</p>"
"<p><strong>Investigating</strong> - looking into it.</p>", "x", ""),
- # No markup at all: a known label in the body text still counts
- # (xhtml <content>, or a vendor publishing plain-text updates).
- # NOT anchored — itertext() glues the timestamp onto the label,
- # which is what every real Statuspage body looks like here.
- ("Sep 7, 19:07 UTCInvestigating - looking into it.", "x", "investigating"),
- ("Completed: the maintenance window is over.", "x", "completed"),
- # No per-update markup: fall back to the title marker (AWS).
+ # …including when the first label is wrapped, attributed, or too
+ # long to be a label — each of which a pattern match would skip,
+ # landing on the older one.
+ ("<p><strong><em>Resolved</em></strong> - fixed.</p>"
+ "<p><strong>Investigating</strong> - looking.</p>", "x", "resolved"),
+ ('<p><strong class="hl">Resolved</strong> - fixed.</p>'
+ "<p><strong>Investigating</strong> - looking.</p>", "x", "resolved"),
+ ("<p><strong>" + "x" * 300 + "</strong></p>"
+ "<p><strong>Investigating</strong> - looking.</p>", "x", ""),
+ ("<p><strong>Investigating - never closed", "x", ""),
+ # Status is NEVER inferred from prose. "Update:" in an ordinary
+ # sentence used to classify a long-resolved incident as open —
+ # and open entries skip both the baseline and the age gate, so a
+ # 400-day-old resolved incident promoted itself at HIGH.
+ ("This incident has been fully resolved. Update: no further action.",
+ "x", ""),
+ ("We were monitoring - then it recovered.", "x", ""),
+ ("Sep 7, 19:07 UTCInvestigating - looking into it.", "x", ""),
+ # No markup anywhere: the title marker is the last resort, and
+ # body prose can no longer override it.
("plain text body", "[RESOLVED] Increased error rates", "resolved"),
+ ("9:16 AM PDT We are investigating: increased error rates.",
+ "Service is operating normally: [RESOLVED] Increased error rates",
+ "resolved"),
# A <strong> that isn't a status label must not be mistaken for one.
("<p><strong>Note</strong> - unrelated bold text.</p>", "x", ""),
("", "", ""),
@@ -179,6 +195,23 @@ def test_latest_status_reads_the_newest_update(
assert _latest_status(body, title) == expected
+def test_a_label_past_the_scan_head_fails_closed() -> None:
+ """A body whose first <strong> sits past the scanned head must score
+ unknown, not fall back to something the prose happens to contain."""
+ body = "Update: we are on it. " + ("filler " * 1_200) + "<strong>Resolved</strong>"
+ assert _latest_status(body, "") == ""
+
+
+def test_structural_label_is_used_when_markup_never_reaches_the_text() -> None:
+ """An xhtml <content> is parsed into elements, so the tags are gone by
+ the time the body is text — the label comes from the element tree."""
+ assert _latest_status("Sep 7, 19:07 UTCInvestigating - looking.", "x",
+ "Investigating") == "investigating"
+ # …and it is still only consulted when the text carries no markup.
+ assert _latest_status("<p><strong>Resolved</strong> - done.</p>", "x",
+ "Investigating") == "resolved"
+
+
def test_is_open_fails_closed_on_unknown_status() -> None:
"""An unrecognised status must NOT count as open: a vendor changing its
feed format must not be able to promote its whole archive at HIGH."""