AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
fix(monitoring): decide vendor status by evidence, not by position (#90)
Rufus461274ac5590
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yamlindex 3bee851a..ed0671c7 100644--- a/packages/core/openexecutive/architecture/architecture-facts.yaml+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml@@ -1334,7 +1334,7 @@ external_monitoring:`Source` instance that handles it. New adapters register here;add a top-level fact in this YAML when they ship.shipped:- - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; only the FIRST label is read, and only if it belongs to the newest update. That is the whole safety property: an older label is nearly always an open status, so mistaking one for the newest reports a long-resolved incident as live — and open entries skip both the baseline and the age gate, so it promotes at HIGH. Three review rounds each found a different way to make that mistake (prose mining, a <strong>-only scan that a <b> or a comment walked past, and a tree walk that skipped an unlabelled newest update), so the body is parsed by html.parser.HTMLParser rather than scanned: a label in a comment or an attribute value is not a label, <b> counts as much as <strong>, nested emphasis and a spaced close tag are handled, and the scan stops at the first block element to CLOSE — one update is one block, so a newest update with no label yields unknown instead of the previous update's status. Body and label always come from ONE element (_entry_body), which re-serialises an xhtml-typed <content> so both feed shapes reach the parser identically. Status is NEVER inferred from prose. Last resort is an AWS-style [RESOLVED] title marker, which can only ever say CLOSED, so an open incident is not detectable on AWS-shaped feeds and is reported on the vendor's next update item instead) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."+ - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; the classifier is deliberately ASYMMETRIC and position-independent. Evidence that an incident is OVER counts from anywhere — an AWS-style [RESOLVED] title marker, a status label, or an update's prose; evidence that it is still OPEN counts only from a label the vendor marked up (<strong>Investigating</strong>), never from prose. Nothing tries to work out which update is newest: four review rounds each found a different way to get that wrong (scanning past an unrecognised label, mining prose for status words, skipping an unlabelled update in the element tree, and reading past a block whose end tag HTML lets you omit), and each reported a long-resolved incident as live, promoting an archive at HIGH. Position is not a property tag soup can express, so the rule does not depend on it and is monotone: more evidence only ever makes the answer safer. Bodies are parsed by html.parser.HTMLParser (a label in a comment, a script or an attribute value is not a label; <b> counts as <strong>), off the event loop via asyncio.to_thread, from ONE element chosen by its text (_entry_body), with an xhtml-typed <content> re-serialised — escaped — so both feed shapes reach the parser identically. Known residual: an unrecognised or empty label on the newest update, with an older open label and no sign of closure anywhere, reads as open — which is why an exempt entry is NOT excused the age gate, bounding that to an entry the feed dates inside the age window) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). Exempt entries must also pass the row's trigger filter — one that misses stays in the baseline, or widening the trigger later would resurface it as news — and they bypass the age gate, since an incident the adapter calls still-happening can have an <updated> stamp days old and recording it as stale would burn its key. The dedup key hashes the NORMALISED stamp (the same parse that fills published_at), so Z vs +00:00 is one key and a serialisation change does not rekey a vendor's open incidents. One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."- "rss — generic Atom/RSS feed adapter via feedparser; handles competitor changelogs, layoffs.fyi, USPTO patent feeds, regulatory filings, blog RSS. Severity hint LOW by default (most RSS is noisy). Optional `trigger_json.keywords` filters by title + summary. Each entry's <pubDate>/<updated> is parsed into Signal.published_at; declares `seed_on_first_poll` so a new watch baselines the feed's existing entries instead of firing them, and entries older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are recorded as `suppressed_stale` (issue #80)."- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.jsonindex c810f5b1..306439dc 100644--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json@@ -1,7 +1,7 @@{"section_id": "external_monitoring","title": "External Monitoring",- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser and only the newest update's label is read, since falling through to an older one reports a resolved incident as open, and a status is never inferred from prose, since a stray \"Update:\" in an old entry would otherwise promote a resolved incident at HIGH) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and skips the stale gate, since an incident the adapter calls still-happening may not have been updated for days. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.","mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",- "generated_at": "2026-09-07T19:32:07+00:00"+ "generated_at": "2026-09-07T19:52:01+00:00"}diff --git a/packages/core/openexecutive/monitoring/pipeline.py b/packages/core/openexecutive/monitoring/pipeline.pyindex d9bc7cbf..fdec241a 100644--- a/packages/core/openexecutive/monitoring/pipeline.py+++ b/packages/core/openexecutive/monitoring/pipeline.py@@ -739,12 +739,19 @@ async def _poll_one_watchlist_item(return Falselost_race: _LostRace | None = None- # Entries the adapter carved out of its own baseline. They bypass the- # age gate below: the adapter has declared them still happening, and- # judging them by an <updated> stamp that can be days old (a- # long-running incident updates rarely) would record one as stale,- # burning its dedup key and muting it until the vendor next touches it- # — the regression the exemption exists to prevent.+ # Entries the adapter carved out of its own baseline. They still face+ # the age gate below. That gate was briefly lifted for them, on the+ # reasoning that a long-running incident updates rarely and would be+ # recorded as stale; four review rounds then showed what it was+ # holding back. An adapter's "this is live" is a reading of markup the+ # VENDOR controls, and every way that reading went wrong reported a+ # years-old resolved incident as live — with the gate lifted, nothing+ # stood between an archive and a HIGH alert. So the exemption buys an+ # entry past the baseline, never past its own timestamp: whatever the+ # adapter believes, an entry the feed dates outside the age window is+ # not news. The cost is the case that lifted it — an open incident+ # silent for longer than the window is reported on its next update+ # rather than immediately — and that is the direction to be wrong in.exempt_keys: set[str] = set()if (getattr(src, "seed_on_first_poll", False)@@ -834,7 +841,7 @@ async def _poll_one_watchlist_item()continue- if signal.dedup_key not in exempt_keys and _is_stale(signal, now, max_age_days):+ if _is_stale(signal, now, max_age_days):_suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_STALE, db_path=db_path)continuediff --git a/packages/core/openexecutive/monitoring/sources/vendor_status.py b/packages/core/openexecutive/monitoring/sources/vendor_status.pyindex 5d9b99cb..f893b240 100644--- a/packages/core/openexecutive/monitoring/sources/vendor_status.py+++ b/packages/core/openexecutive/monitoring/sources/vendor_status.py@@ -44,10 +44,12 @@ incident, both #80 freshness gates become safe here:"""from __future__ import annotations+import asyncioimport hashlibimport loggingimport refrom datetime import UTC, datetime+from html import escapefrom html.parser import HTMLParserfrom pathlib import Pathfrom urllib.parse import urlparse@@ -91,7 +93,8 @@ _CONTENT_ENCODED = "{http://purl.org/rss/1.0/modules/content/}encoded"# The documented Statuspage history feed is bounded (~25 entries) — a# runaway count here means the feed is malformed and we should stop-# reading rather than process megabytes of garbage.+# reading rather than process megabytes of garbage. Applied in the+# parsers, before each entry's body is serialised and parsed._MAX_ENTRIES_PER_FEED = 100# Incident status vocabulary. Statuspage renders each update in the entry@@ -117,12 +120,18 @@ _BODY_SCAN_CHARS = 8_000_MAX_LABEL_CHARS = 200# Tags a vendor marks an update's status label with._LABEL_TAGS = frozenset({"strong", "b"})-# Block-level tags. One update = one block, so the first block to CLOSE-# ends the newest update: a label found after that belongs to an older one.-_BLOCK_TAGS = frozenset({- "p", "div", "li", "ul", "ol", "tr", "td", "th", "table",- "section", "article", "blockquote", "dd", "dt", "dl",-})+# Evidence that the incident is over, taken from ANYWHERE in the body —+# label or prose. See _latest_status for why this direction may read prose+# and the OPEN direction may not.+_CLOSED_WORD_RE = re.compile(+ r"\b(" + "|".join(sorted(_CLOSED_STATUSES)) + r")\b", re.IGNORECASE+)+# Labels to read from one body. The body is already capped; this bounds+# the list a pathological body can build.+_MAX_LABELS = 50+# Nesting depth _element_markup will serialise. Deeper than any real feed,+# and shallow enough that the walk cannot exhaust the stack.+_MAX_ELEMENT_DEPTH = 100class VendorStatusSource:@@ -174,7 +183,12 @@ class VendorStatusSource:return []try:- entries = _parse_feed(body)+ # Off the event loop: parsing is CPU-bound over bytes a+ # watched URL controls, and the scan runs inside the API+ # process. A body of 8k "<" costs ~5ms per entry to tokenize —+ # half a second for a 100-entry feed — which is not a bug to+ # fix in the parser but latency no HTTP handler should eat.+ entries = await asyncio.to_thread(_parse_feed, body)except ParseError as exc:logger.warning("vendor_status: feed %s failed to parse: %s", item.target, exc@@ -190,7 +204,7 @@ class VendorStatusSource:)built = (self._build_signal(entry, item, vendor_label)- for entry in entries[:_MAX_ENTRIES_PER_FEED]+ for entry in entries)return [signal for signal in built if signal is not None]@@ -290,7 +304,10 @@ def _parse_feed(body: bytes) -> list[dict[str, str]]:def _parse_atom(feed: Element) -> list[dict[str, str]]:out: list[dict[str, str]] = []- for entry in feed.findall(f"{_ATOM_NS}entry"):+ # Capped HERE, not after parsing: each entry's body is re-serialised+ # and parsed, so a 5000-entry feed would pay that cost 5000 times+ # before the cap dropped all but the first 100.+ for entry in feed.findall(f"{_ATOM_NS}entry")[:_MAX_ENTRIES_PER_FEED]:entry_id = (entry.findtext(f"{_ATOM_NS}id") or "").strip()title = (entry.findtext(f"{_ATOM_NS}title") or "").strip()updated = (entry.findtext(f"{_ATOM_NS}updated") or "").strip()@@ -308,7 +325,7 @@ def _parse_atom(feed: Element) -> list[dict[str, str]]:def _parse_rss(channel: Element) -> list[dict[str, str]]:out: list[dict[str, str]] = []- for item in channel.findall("item"):+ for item in channel.findall("item")[:_MAX_ENTRIES_PER_FEED]:guid = (item.findtext("guid") or "").strip()link = (item.findtext("link") or "").strip()title = (item.findtext("title") or "").strip()@@ -344,146 +361,182 @@ def _entry_body(parent: Element, tags: tuple[str, ...]) -> str:el = parent.find(tag)if el is None:continue- markup = _element_markup(el) if len(el) else (el.text or "")- if markup.strip():- return markup+ # Chosen on the element's TEXT: an empty <content><div/></content>+ # serialises to "<div></div>", which is truthy markup carrying no+ # content, and would shadow a <summary> holding the real log.+ if not "".join(el.itertext()).strip():+ continue+ return _element_markup(el) if len(el) else (el.text or "")return ""def _element_markup(el: Element) -> str:"""Re-serialise an element's children as simple HTML, local names only.- Bounded by ``_BODY_SCAN_CHARS``: the newest update is at the top, and- an unbounded walk would let a label buried hundreds of updates deep in- an archive decide that the incident is open.+ Text is HTML-escaped on the way out. The XML parser has already turned+ ``<strong>`` in an update's prose back into literal ``<strong>``,+ and emitting that raw would hand the HTML parser a status label the+ vendor never marked up — letting an entry that merely *mentions* a+ label forge one.++ Iterative with an explicit depth cap: a body nested a few thousand+ elements deep would otherwise raise RecursionError out of the parse,+ which the pipeline catches as an adapter crash — the row would then+ fail every tick, never baseline, and emit nothing, silencing its own+ watch. Bounded by ``_BODY_SCAN_CHARS`` as well: the newest update is at+ the top, and serialising an entire archive is work no answer needs."""out: list[str] = []size = 0+ # (element, depth) to open, or a literal string to emit as-is.+ stack: list[tuple[Element, int] | str] = []- def emit(text: str) -> bool:+ def push(text: str) -> None:nonlocal sizeout.append(text)size += len(text)- return size < _BODY_SCAN_CHARS-- def walk(node: Element) -> bool:- for child in node:- local = (- child.tag.rsplit("}", 1)[-1].lower()- if isinstance(child.tag, str) else ""- )- if not emit(f"<{local}>"):- return False- if child.text and not emit(child.text):- return False- if not walk(child):- return False- if not emit(f"</{local}>"):- return False- if child.tail and not emit(child.tail):- return False- return Trueif el.text:- emit(el.text)- walk(el)+ push(escape(el.text))+ stack.extend(reversed([(child, 1) for child in el]))+ while stack and size < _BODY_SCAN_CHARS:+ item = stack.pop()+ if isinstance(item, str):+ push(item)+ continue+ node, depth = item+ local = (+ node.tag.rsplit("}", 1)[-1].lower() if isinstance(node.tag, str) else ""+ )+ push(f"<{local}>")+ if node.text:+ push(escape(node.text))+ tail = escape(node.tail) if node.tail else ""+ # Children first, then this element's close tag, then its tail.+ pending: list[tuple[Element, int] | str] = [f"</{local}>"]+ if tail:+ pending.append(tail)+ if depth < _MAX_ELEMENT_DEPTH:+ children: list[tuple[Element, int] | str] = [+ (child, depth + 1) for child in node+ ]+ pending = children + pending+ stack.extend(reversed(pending))return "".join(out)def _latest_status(body: str, title: str) -> str:- """The incident's CURRENT status, lowercased, or "" when unknown.-- Statuspage puts the newest update first and marks its status up as a- label (``<strong>Resolved</strong>``), so the state right now is what- that ONE label says. Everything here exists to avoid reading any other- one: an older label is nearly always an open status, so mistaking one- for the newest reports a long-resolved incident as live — and since an- open incident skips both the first-poll baseline and the age gate,- that promotes a years-old resolved incident at HIGH. Three review- rounds each found a different way to make that mistake.-- So the body is parsed by a real HTML parser (``_FirstLabel``) rather- than scanned, and the answer is whatever the first label element- contains — even if that is nothing recognisable, in which case this- returns "" and ``_is_open`` fails closed. A body with no label at all- falls through to the title marker, which can only ever say CLOSED.-- Status is never inferred from prose: "Update:" in an ordinary- sentence is not a status label, and treating it as one is how a- 400-day-old resolved incident promoted itself at HIGH.+ """Whether this incident is still happening, lowercased, or "" .++ Read the asymmetry here first, because it is the safety property:++ * evidence that the incident is OVER counts from anywhere — the+ title's ``[RESOLVED]`` marker, a status label, or the prose of an+ update;+ * evidence that it is still OPEN counts only from a status label+ the vendor marked up as one (``<strong>Investigating</strong>``).++ Nothing here tries to work out which update is the newest, and that is+ deliberate. Four review rounds each found a different way to get that+ wrong — scanning past an unrecognised label, mining prose for status+ words, skipping an unlabelled update in the element tree, and reading+ past a block whose end tag HTML lets you omit — and every one of them+ reported a long-resolved incident as live. An "open" verdict promotes+ at HIGH and skips the first-poll baseline, so reading an older, open+ label replays exactly the archive this adapter exists to stop+ replaying. Position in the body is not a property tag soup can be+ trusted to express, so the rule no longer depends on it: ANY sign of+ closure closes, wherever it sits, and that is monotone — more evidence+ can only ever make the answer safer.++ The cost is that a live incident whose prose happens to say "resolved"+ is read as closed, so a watch added mid-outage reports it on its next+ update rather than immediately. That is the direction to be wrong in."""- label = _first_label(body[:_BODY_SCAN_CHARS])- if label is not None:- return _known_status(label)marker = _TITLE_MARKER_RE.search(title)- return marker.group(1).lower() if marker else ""+ if marker is not None:+ return marker.group(1).lower()+ reader = _read_body(body[:_BODY_SCAN_CHARS])+ statuses = [_known_status(raw) for raw in reader.labels]+ # Labels first, exactly: a label whose own text is split across inline+ # tags survives here even though the text probe below would see a+ # space through the middle of the word.+ for status in statuses:+ if status in _CLOSED_STATUSES:+ return status+ closed = _CLOSED_WORD_RE.search(reader.text)+ if closed is not None:+ return closed.group(1).lower()+ for status in statuses:+ if status in _OPEN_STATUSES:+ return status+ return ""-class _FirstLabel(HTMLParser):- """Extracts the text of the FIRST status label in an update body.+class _BodyReader(HTMLParser):+ """Splits an update log into its plain text and its status labels.- A real parser, so a ``<strong>`` inside a comment or an attribute+ A real parser, so a label inside a comment, a script, or an attributevalue is not a label, ``<b>`` counts as much as ``<strong>``, nested- emphasis inside a label is kept, and ``</strong >`` closes normally —- each of which a hand-rolled scan got wrong in a way that reported a- resolved incident as open.-- ``label`` is None when the body holds no label at all (the caller- falls through), and a string — possibly empty — once one is found or- ruled out. It stops at the first block element to CLOSE: one update is- one block, so a label after that point belongs to an OLDER update and- must not be read. An unclosed label yields "" for the same reason.+ emphasis inside a label is kept, and ``</strong >`` closes normally.+ No notion of where one update ends and the next begins — see+ ``_latest_status``."""def __init__(self) -> None:super().__init__(convert_charrefs=True)- self._parts: list[str] = []+ self.text: str = ""+ self.labels: list[str] = []+ self._chunks: list[str] = []+ self._label: list[str] | None = Noneself._tag: str | None = Noneself._depth = 0- self._done = False- self._found = False-- @property- def label(self) -> str | None:- if not self._found:- return None- return "".join(self._parts).strip()[:_MAX_LABEL_CHARS] if self._done else ""def handle_starttag(self, tag: str, attrs: object) -> None:- if self._done:- return- if not self._found and tag in _LABEL_TAGS:- self._found, self._tag, self._depth = True, tag, 1- elif self._found and tag == self._tag:+ if self._label is None:+ if tag in _LABEL_TAGS and len(self.labels) < _MAX_LABELS:+ self._label, self._tag, self._depth = [], tag, 1+ elif tag == self._tag:self._depth += 1def handle_endtag(self, tag: str) -> None:- if self._done:- return- if self._found and tag == self._tag:+ if self._label is not None and tag == self._tag:self._depth -= 1if self._depth == 0:- self._done = True- elif not self._found and tag in _BLOCK_TAGS:- # The newest update ended without a label. Stop rather than- # walk on into the previous update's.- self._found, self._done = True, True+ self.labels.append("".join(self._label).strip())+ self._label, self._tag = None, Nonedef handle_data(self, data: str) -> None:- if self._found and not self._done:- self._parts.append(data)---def _first_label(body: str) -> str | None:- """``_FirstLabel`` applied to one body; None when it holds no label."""- parser = _FirstLabel()+ self._chunks.append(data)+ if self._label is not None:+ self._label.append(data)++ def handle_startendtag(self, tag: str, attrs: object) -> None:+ # <strong/> and friends: a self-closing label opens and closes at+ # once, so it must not swallow the rest of the body as its text.+ return++ def close(self) -> None:+ super().close()+ # Joined with a SPACE, not "": a tag boundary is a token boundary,+ # and "".join glues the words either side of one+ # ("<small>12:00 UTC</small><strong>Resolved</strong>" → "UTCResolved"),+ # which hides the word from a \b-anchored search. Adding+ # boundaries can only reveal evidence of closure, never conceal it.+ self.text = " ".join(self._chunks)+++def _read_body(body: str) -> _BodyReader:+ """``_BodyReader`` over one body. Lenient: a parse that gives up keeps+ whatever it read, which can only mean less evidence, never more."""+ reader = _BodyReader()try:- parser.feed(body)- parser.close()+ reader.feed(body)+ reader.close()except Exception: # pragma: no cover - HTMLParser is lenient by design- logger.debug("vendor_status: label parse failed", exc_info=True)- return parser.label- return parser.label+ logger.debug("vendor_status: body parse failed", exc_info=True)+ reader.text = "".join(reader._chunks)+ return readerdef _known_status(raw: str) -> str:diff --git a/packages/core/tests/unit/test_monitoring_pipeline.py b/packages/core/tests/unit/test_monitoring_pipeline.pyindex 59bb8ade..62a61a2f 100644--- a/packages/core/tests/unit/test_monitoring_pipeline.py+++ b/packages/core/tests/unit/test_monitoring_pipeline.py@@ -633,32 +633,37 @@ def test_vendor_status_first_poll_alerts_the_outage_not_the_archive(assert len(promoted) == 2-def test_baseline_exempt_entry_bypasses_the_age_gate(+def test_baseline_exempt_entry_still_faces_the_age_gate(db: Path,install_fake_source,monkeypatch: pytest.MonkeyPatch,) -> None:- """An adapter that exempts an entry has declared it still happening, so- the age gate must not judge it by its <updated> stamp: a long-running- incident whose last update is 8 days old would be recorded as stale,- burning its dedup key and muting it until the vendor next touches it."""+ """The exemption buys an entry past the BASELINE, never past its own+ timestamp. An adapter's "this is live" is a reading of vendor-+ controlled markup, and four review rounds each found a way to make+ that reading say "open" about a years-old resolved incident — with the+ age gate lifted, nothing stood between an archive and a HIGH alert."""promoted = _install_promotion_recorder(monkeypatch)_insert_feed(db, "vendor-stripe")install_fake_source([- _make_signal(dedup_key="vendor_status:quiet", published_at=_iso_days_ago(8)),+ _make_signal(dedup_key="vendor_status:live", published_at=_iso_days_ago(0.1)),+ _make_signal(dedup_key="vendor_status:ancient", published_at=_iso_days_ago(400)),_make_signal(dedup_key="vendor_status:old", published_at=_iso_days_ago(90)),],seed=True,- promote=lambda signal, item: signal.dedup_key.endswith(":quiet"),+ # The adapter claims BOTH the fresh and the 400-day-old entry are+ # open — the misclassification every review round produced.+ promote=lambda signal, item: not signal.dedup_key.endswith(":old"),)asyncio.run(mp.run_external_monitor_scan(db_path=db))outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}- assert outcomes["vendor_status:quiet"] == OUTCOME_ALERTED+ assert outcomes["vendor_status:live"] == OUTCOME_ALERTED+ assert outcomes["vendor_status:ancient"] == OUTCOME_SUPPRESSED_STALEassert outcomes["vendor_status:old"] == OUTCOME_SUPPRESSED_BASELINE- assert [e.external_id for e in promoted] == ["vendor_status:quiet"]+ assert [e.external_id for e in promoted] == ["vendor_status:live"]def test_baseline_exempt_trigger_miss_is_still_recorded(@@ -999,7 +1004,7 @@ def test_loser_of_the_baseline_race_still_promotes_an_open_incident(_make_signal(dedup_key="vendor_status:open",published_at=(stamp - timedelta(minutes=30)).isoformat(),- ),+ ), # before the cutoff, but well inside the age window_make_signal(dedup_key="vendor_status:archived",published_at=(stamp - timedelta(days=200)).isoformat(),@@ -1030,7 +1035,11 @@ def test_status_scan_is_linear_on_a_hostile_body() -> None:# 3.4s payload) and a body that is nothing but unterminated opening# tags, which a later `<strong\b[^>]*>` pattern scanned quadratically# at 3.7ms/entry — 370ms per 100-entry feed on the API's event loop.- for hostile in ("<strong>" + " " * 7_991 + ".", ("<strong " * 1_000)[:8_000]):+ for hostile in (+ "<strong>" + " " * 7_991 + ".",+ ("<strong " * 1_000)[:8_000],+ "<" * 8_000, # the slowest shape measured: pure opening delimiters+ ):started = time.perf_counter()assert _latest_status(hostile, "") == ""elapsed = time.perf_counter() - starteddiff --git a/packages/core/tests/unit/test_monitoring_vendor_status.py b/packages/core/tests/unit/test_monitoring_vendor_status.pyindex 2a7a845f..0363ba23 100644--- a/packages/core/tests/unit/test_monitoring_vendor_status.py+++ b/packages/core/tests/unit/test_monitoring_vendor_status.py@@ -156,81 +156,101 @@ _RESOLVED_BODY_TEXT = (("<p><strong>Postmortem</strong> - write-up.</p>", "x", "postmortem"),# Case and whitespace are the vendor's business, not ours.("<p><STRONG> resolved </STRONG> - done.</p>", "x", "resolved"),- # The NEWEST label decides, and only it. Falling through to an- # older one would report a resolved incident as investigating,- # since the older labels on an incident are nearly always open.- ("<p><strong>Fixed</strong> - all good.</p>"- "<p><strong>Investigating</strong> - looking into it.</p>", "x", ""),- # …including when the first label is wrapped, attributed, or too- # long to be a label — each of which a pattern match would skip,- # landing on the older one.- ("<p><strong><em>Resolved</em></strong> - fixed.</p>"- "<p><strong>Investigating</strong> - looking.</p>", "x", "resolved"),- ('<p><strong class="hl">Resolved</strong> - fixed.</p>'- "<p><strong>Investigating</strong> - looking.</p>", "x", "resolved"),- ("<p><strong>" + "x" * 300 + "</strong></p>"- "<p><strong>Investigating</strong> - looking.</p>", "x", ""),- ("<p><strong>Investigating - never closed", "x", ""),- # Status is NEVER inferred from prose. "Update:" in an ordinary- # sentence used to classify a long-resolved incident as open —- # and open entries skip both the baseline and the age gate, so a- # 400-day-old resolved incident promoted itself at HIGH.+ # Closure counts from prose as well as from a label — an update+ # that says it is resolved closes the incident even though no+ # label says so.("This incident has been fully resolved. Update: no further action.",- "x", ""),+ "x", "resolved"),+ ("<p>Resolved - this incident has been resolved.</p>"+ "<p><strong>Investigating</strong> - looking.</p>", "x", "resolved"),+ # …but an OPEN reading needs a marked-up label. Prose alone never+ # opens an incident: "Update:" in a sentence is not a status.("We were monitoring - then it recovered.", "x", ""),- ("Sep 7, 19:07 UTCInvestigating - looking into it.", "x", ""),- # No markup anywhere: the title marker is the last resort, and- # body prose can no longer override it.+ ("Sep 7, 19:07 UTC Investigating - looking into it.", "x", ""),+ # No markup anywhere: the title marker decides, and it can only+ # ever close.("plain text body", "[RESOLVED] Increased error rates", "resolved"),("9:16 AM PDT We are investigating: increased error rates.","Service is operating normally: [RESOLVED] Increased error rates","resolved"),- # A <strong> that isn't a status label must not be mistaken for one.+ # A <strong> that isn't a status label is not a status.("<p><strong>Note</strong> - unrelated bold text.</p>", "x", ""),("", "", ""),],)-def test_latest_status_reads_the_newest_update(+def test_status_reads_closure_from_anywhere_and_openness_only_from_a_label(body: str, title: str, expected: str,) -> None:assert _latest_status(body, title) == expected-def test_a_label_past_the_scan_head_fails_closed() -> None:- """A body whose first <strong> sits past the scanned head must score- unknown, not fall back to something the prose happens to contain."""- body = "Update: we are on it. " + ("filler " * 1_200) + "<strong>Resolved</strong>"+ "body",+ [+ # An unrecognised or empty label on the newest update, with an+ # older open label and NO evidence of closure anywhere. This reads+ # as open, and that is the rule's known residual: it is why the+ # exemption does not also buy an entry past the age gate, so the+ # blast radius is an entry the feed dates inside the age window+ # rather than a years-old archive.+ "<p><strong>Fixed</strong> - all good.</p>"+ "<p><strong>Investigating</strong> - looking into it.</p>",+ "<p><strong></strong> - x.</p>"+ "<p><strong>Investigating</strong> - looking.</p>",+ ],+)+def test_known_residual_an_unreadable_newest_label_reads_as_open(body: str) -> None:+ assert _latest_status(body, "") == "investigating"+++def test_a_label_past_the_scan_head_is_not_read() -> None:+ """The body is capped, so a label buried past the head cannot decide+ anything — in either direction."""+ body = "Update: we are on it. " + ("filler " * 1_200) + "<strong>Investigating</strong>"assert _latest_status(body, "") == ""@pytest.mark.parametrize(("body", "expected"),[- # A real parser, so none of these read a LATER (older, and so- # nearly always open) label by mistake. Each line was a way a- # resolved incident reported itself live.+ # Position is not a property tag soup can express, so the rule no+ # longer depends on it. Each of these was a shape that made a+ # position-based rule read an OLDER, open label and report a+ # resolved incident as live — implicit <p> close, one block with+ # <br> separators, no wrapper, a block tag outside any list we+ # could keep, and a close tag swallowed by a script or a comment.+ ("<p>Resolved - this incident has been resolved."+ "<p><strong>Investigating</strong> - looking.</p>", "resolved"),+ ("<div>Sep 7 12:00 UTC - Resolved - all clear.<br/><br/>"+ "Sep 7 09:00 UTC - <strong>Investigating</strong> - looking.</div>",+ "resolved"),+ ("Sep 7 12:00 UTC Resolved - all clear.<br/>"+ "Sep 7 09:00 UTC <strong>Investigating</strong> - looking.", "resolved"),+ ("<h3>Sep 7 - Resolved - all clear.</h3>"+ "<h3><strong>Investigating</strong></h3>", "resolved"),+ ("<header>Resolved - all clear.</header>"+ "<p><strong>Investigating</strong></p>", "resolved"),+ ('<p>Resolved - all clear.<script>var s = "</p>";</script>'+ "<p><strong>Investigating</strong></p>", "resolved"),+ ("<p>Resolved - all clear.<!-- </p> -->"+ "<p><strong>Investigating</strong></p>", "resolved"),+ ("<p><em>Resolved</em> - fixed.<p><strong>Investigating</strong></p>",+ "resolved"),+ # A real parser, so a label in a comment or an attribute value is+ # not a label, <b> counts as much as <strong>, nested emphasis is+ # kept, and a spaced close tag closes.("<p><b>Resolved</b> - fixed.</p>""<p><strong>Investigating</strong> - looking.</p>", "resolved"),- ("<!-- <strong>Investigating</strong> -->"- "<p><strong>Resolved</strong> - fixed.</p>", "resolved"),('<p><a href="/x?q=<strong>Investigating</strong>">Resolved</a>'- " - fixed.</p>", ""),- ("<p><strong><em>Resolved</em></strong> - fixed.</p>"- "<p><strong>Investigating</strong> - looking.</p>", "resolved"),- ('<p><strong class="hl">Resolved</strong> - fixed.</p>'- "<p><strong>Investigating</strong> - looking.</p>", "resolved"),- # The newest update carries no label, or an empty one: stop at the- # end of its block rather than reading the previous update's.- ("<p>Resolved - this incident has been resolved.</p>"- "<p><strong>Investigating</strong> - looking.</p>", ""),- ("<p><strong></strong> - x.</p>"- "<p><strong>Investigating</strong> - looking.</p>", ""),- # A close tag with a space is valid HTML and must not mute a live- # outage.+ " - fixed.</p>", "resolved"),("<p><strong>Investigating</strong > - live.</p>", "investigating"),+ ("<p><strong><em>Monitoring</em></strong> - a fix is applied.</p>",+ "monitoring"),],)-def test_only_the_newest_label_decides(body: str, expected: str) -> None:+def test_closure_is_read_however_the_body_is_shaped(+ body: str, expected: str,+) -> None:assert _latest_status(body, "") == expected@@ -257,10 +277,10 @@ def test_xhtml_body_reads_only_its_newest_update() -> None:labelled = _parse_feed(_feed("<strong>Resolved</strong> - fixed."))[0]assert _latest_status(labelled["body"], labelled["title"]) == "resolved"- # Newest update unlabelled: must NOT fall through to the older- # "Investigating", which would promote a resolved incident at HIGH.+ # Newest update unlabelled: its prose still closes the incident, so it+ # cannot fall through to the older "Investigating" and promote at HIGH.unlabelled = _parse_feed(_feed("Resolved - this incident has been resolved."))[0]- assert _latest_status(unlabelled["body"], unlabelled["title"]) == ""+ assert _latest_status(unlabelled["body"], unlabelled["title"]) == "resolved"def test_label_never_comes_from_a_different_element_than_the_body() -> None:@@ -283,7 +303,7 @@ def test_label_never_comes_from_a_different_element_than_the_body() -> None:</entry></feed>"""entry = _parse_feed(feed)[0]- assert _latest_status(entry["body"], entry["title"]) == ""+ assert _latest_status(entry["body"], entry["title"]) == "resolved"def test_is_open_fails_closed_on_unknown_status() -> None: