AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
fix(monitoring): stop vendor_status replaying resolved incidents (#90)
Rufusc44e6e3cb3ea
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yamlindex eb737e13..868334a1 100644--- a/packages/core/openexecutive/architecture/architecture-facts.yaml+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml@@ -1334,7 +1334,7 @@ external_monitoring:`Source` instance that handles it. New adapters register here;add a top-level fact in this YAML when they ship.shipped:- - "vendor_status — polls Statuspage Atom history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Deliberately outside BOTH #80 freshness gates (no published_at, not seeded): an incident's entry id never changes across updates and a suppressed insert burns the dedup key, so either gate would permanently mute an incident that is open (or quiet for a week) when the watch is added. New watches therefore still replay the history feed's resolved incidents — follow-up: dedup on id+<updated> or an open/resolved filter."+ - "vendor_status — polls Statuspage Atom/RSS history feeds (AWS, Stripe, Twilio, GitHub, Cloudflare, …) and emits one Signal per incident entry. Severity hint HIGH by default. Optional `trigger_json.keywords` narrows to entries whose title contains a keyword. Dedup key is sha256(slug + NUL + entry id + NUL + <updated>) — the <updated> component is load-bearing (issue #90): an incident's entry id never changes across its updates, so keying on the id alone made every suppression permanent, which is why this adapter used to sit outside BOTH #80 freshness gates and replayed a vendor's resolved archive as HIGH alerts on the first poll. With the state in the key a suppressed insert burns one state of an incident rather than the incident, so both gates now apply: published_at is parsed from <updated>/<pubDate> (age gate + future deferral), and the row is seeded on its first poll. The adapter reads each incident's CURRENT status out of the entry body (Statuspage renders the newest update first as <strong>Resolved</strong> / Investigating / Monitoring / …; fallbacks are a label leading the body text and an AWS-style [RESOLVED] title marker) and implements the optional `promote_on_baseline` hook, which exempts incidents that are still OPEN from its own baseline — so a new watch added during an outage surfaces that outage immediately while the resolved archive is recorded as suppressed_baseline. Unrecognised status counts as CLOSED (fail closed: a vendor changing feed format must not be able to promote its archive; the next <updated> bump mints a new key and surfaces normally). One-time migration `vendor_status_dedup_rekey_90` (marker row in `monitoring_migrations`) clears `baselined_at` on existing vendor_status rows so the upgrade tick re-seeds under the new key scheme instead of replaying."- "rss — generic Atom/RSS feed adapter via feedparser; handles competitor changelogs, layoffs.fyi, USPTO patent feeds, regulatory filings, blog RSS. Severity hint LOW by default (most RSS is noisy). Optional `trigger_json.keywords` filters by title + summary. Each entry's <pubDate>/<updated> is parsed into Signal.published_at; declares `seed_on_first_poll` so a new watch baselines the feed's existing entries instead of firing them, and entries older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are recorded as `suppressed_stale` (issue #80)."- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.jsonindex ae1197cc..ed5e6b31 100644--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json@@ -1,7 +1,7 @@{"section_id": "external_monitoring","title": "External Monitoring",- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom incident feeds (AWS, Stripe, GitHub, …).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` is deliberately outside both gates: an incident's id never changes across updates, so either gate would permanently mute an incident that is open when the watch is added. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / …) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.","mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",- "generated_at": "2026-09-07T16:18:57+00:00"+ "generated_at": "2026-09-07T18:35:50+00:00"}diff --git a/packages/core/openexecutive/monitoring/pipeline.py b/packages/core/openexecutive/monitoring/pipeline.pyindex 3204e14c..a3814e99 100644--- a/packages/core/openexecutive/monitoring/pipeline.py+++ b/packages/core/openexecutive/monitoring/pipeline.py@@ -52,7 +52,7 @@ from openexecutive.monitoring.models import (Signal,WatchlistItem,)-from openexecutive.monitoring.sources import get_source_for_kind+from openexecutive.monitoring.sources import Source, get_source_for_kindfrom openexecutive.monitoring.sources._http import strip_url_queryfrom openexecutive.monitoring.sources.base import (collapse_whitespace,@@ -488,6 +488,39 @@ def _lost_race(return _LostRace(cutoff)+def _split_baseline_exempt(+ src: Source, item: WatchlistItem, entries: list[Signal],+) -> tuple[list[Signal], list[Signal]]:+ """Split a seeding row's first-poll entries into (archive, exempt).++ ``exempt`` is what the adapter's optional ``promote_on_baseline`` hook+ claims is live news rather than back-catalogue (see+ ``sources.base.Source``); ``archive`` is everything else and is+ baselined exactly as before. An adapter without the hook exempts+ nothing, which is the historical behaviour for ``rss`` / ``edgar``.++ A hook that raises is treated as "not exempt" — the same fail-quiet+ stance ``matches_trigger`` takes, and the safe one: a crashing hook+ must not turn a first poll into a back-catalogue replay.+ """+ hook = getattr(src, "promote_on_baseline", None)+ if hook is None:+ return entries, []+ archive: list[Signal] = []+ exempt: list[Signal] = []+ for signal in entries:+ try:+ is_exempt = bool(hook(signal, item))+ except Exception:+ logger.exception(+ "monitoring.pipeline: promote_on_baseline crashed on %r — "+ "treating as baseline", item.slug,+ )+ is_exempt = False+ (exempt if is_exempt else archive).append(signal)+ return archive, exempt++def _record_baseline(item: WatchlistItem,entries: list[Signal],@@ -654,17 +687,34 @@ async def _poll_one_watchlist_item(# baseline. ``item`` is the pre-poll snapshot, so the in-memory value# is unaffected by ``mark_polled``. Baseline rows never reach the# alert pipeline, so they are never charged to the scan budget.+ #+ # An adapter may exempt individual entries from its own baseline via+ # the optional ``promote_on_baseline`` hook (see ``sources.base``):+ # ``vendor_status`` uses it so an incident that is still OPEN when the+ # watch is added surfaces on that first poll while the resolved+ # archive is still swallowed. Exempt entries skip the baseline write+ # and fall through to the ordinary cascade below.lost_race: _LostRace | None = Noneif (getattr(src, "seed_on_first_poll", False)and item.id is not Noneand item.baselined_at is None):- recorded = _record_baseline(item, capped, fetched_at, db_path=db_path)+ archive, exempt = _split_baseline_exempt(src, item, capped)+ recorded = _record_baseline(item, archive, fetched_at, db_path=db_path)if recorded is not None:- tally.written = recorded- return- lost_race = _lost_race(item, fetched_at, db_path=db_path)+ tally.written += recorded+ if not exempt:+ return+ # The stamp is claimed and the archive is recorded, so this row+ # is no longer seeding: the exempt entries run the ordinary+ # cascade. Crash-safe in the direction that matters — dying+ # here leaves them unrecorded, and the next tick (no longer a+ # first poll) promotes them normally instead of replaying the+ # archive.+ capped = exempt+ else:+ lost_race = _lost_race(item, fetched_at, db_path=db_path)# Apply the per-item trigger filter (default: True). Entries that miss# it are dropped — unless this scan lost the baseline race, in whichdiff --git a/packages/core/openexecutive/monitoring/sources/base.py b/packages/core/openexecutive/monitoring/sources/base.pyindex 5f5101c3..04418235 100644--- a/packages/core/openexecutive/monitoring/sources/base.py+++ b/packages/core/openexecutive/monitoring/sources/base.py@@ -27,6 +27,7 @@ import loggingimport mathfrom collections.abc import Mappingfrom datetime import UTC, datetime, timedelta+from email.utils import parsedate_to_datetimefrom pathlib import Pathfrom typing import Any, Protocol@@ -76,6 +77,24 @@ class Source(Protocol):"""...+ # OPTIONAL hook, deliberately NOT a Protocol member: adding a required+ # one here would break structural conformance for the five adapters+ # that don't need it. The pipeline looks it up with ``getattr`` and+ # treats "absent" as "everything is baseline".+ #+ # def promote_on_baseline(self, signal: Signal, item: WatchlistItem) -> bool+ #+ # Answers "on the FIRST poll of this row, is this entry live news rather+ # than back-catalogue?". Only ``seed_on_first_poll`` adapters are asked.+ # Entries it returns True for skip the baseline and run the normal+ # cascade; everything else is recorded as ``suppressed_baseline`` exactly+ # as before. ``vendor_status`` implements it (an incident still OPEN when+ # the watch is added is news; the resolved archive is not) — see+ # ``sources/vendor_status.py``. An adapter that defines it MUST also key+ # its ``dedup_key`` on the entry's mutable state, not just its id:+ # a baselined entry's key is burned, so a fixed-id key would mute the+ # entry for good.+# Re-export Signal so adapter code can ``from .base import Signal``.__all__ = ["Signal", "Source"]@@ -144,6 +163,48 @@ def feed_entry_published_at(return fallback+def feed_text_published_at(raw: str | None) -> str | None:+ """ISO 8601 UTC publish timestamp from a RAW feed date string, or None.++ The sibling of ``feed_entry_published_at`` for adapters that parse XML+ themselves instead of going through feedparser (``vendor_status``), so+ they get the same ``Signal.published_at`` contract: an ISO 8601 string+ in UTC, or None when the feed gives us nothing parseable.++ Accepts both dialects the status-page feeds use — Atom ``<updated>``+ (RFC 3339, ``2026-09-07T15:04:05Z``) and RSS ``<pubDate>``+ (RFC 822, ``Sun, 07 Sep 2026 15:04:05 +0000``) — and normalises a+ naive value to UTC, which is what every other date path here assumes.++ Unlike ``feed_entry_published_at`` this takes no ``skew``: that+ parameter only picks BETWEEN an entry's two date keys, and there is+ one key here. An implausibly future value is returned as-is so the+ pipeline can defer the entry (``pipeline._is_future``) rather than+ record it — recording burns the dedup key.+ """+ text = (raw or "").strip()+ if not text:+ return None+ dt: datetime | None = None+ try:+ # 3.11's fromisoformat takes the trailing "Z"; it is stricter than+ # RFC 3339 about fractional-second digits, hence the fallback.+ dt = datetime.fromisoformat(text)+ except ValueError:+ try:+ dt = parsedate_to_datetime(text)+ except (TypeError, ValueError):+ return None+ if dt is None:+ return None+ if dt.tzinfo is None:+ dt = dt.replace(tzinfo=UTC)+ try:+ return dt.astimezone(UTC).isoformat()+ except (OverflowError, OSError, ValueError):+ return None++def _valid_skew_hours(raw: Any) -> float | None:"""``raw`` as hours if it is a finite number in 0..MAX_FUTURE_SKEW_HOURS,else None. Booleans are rejected (``False`` is not "off"; JSON ``true``diff --git a/packages/core/openexecutive/monitoring/sources/vendor_status.py b/packages/core/openexecutive/monitoring/sources/vendor_status.pyindex d5753f74..c528673b 100644--- a/packages/core/openexecutive/monitoring/sources/vendor_status.py+++ b/packages/core/openexecutive/monitoring/sources/vendor_status.py@@ -21,11 +21,29 @@ Severity hint: defaults to ``HIGH`` for any new incident. The triagepipeline downstream still has final say — a HIGH hint with a low-trustscore from the watchlist (configured in PR-B's tuning loop) gets dampedbefore the principal sees it.++Freshness (issue #90): a history feed replays its whole archive on the+first poll, so a new watch used to promote up to 100 resolved incidents+at HIGH. The fix is one change with three consequences: the ``dedup_key``+is keyed on ``(entry id, <updated>)`` rather than the entry id alone.+Statuspage bumps ``<updated>`` on every incident update, so a state+change mints a new key and re-fires while an identical poll still dedups+— and because a suppressed insert now burns only ONE state of an+incident, both #80 freshness gates become safe here:++ - the row is seeded on its first poll (``seed_on_first_poll``), with+ the incidents that are still OPEN exempted from the baseline by+ ``promote_on_baseline`` so a live outage surfaces immediately while+ the resolved archive is recorded and never promoted;+ - ``published_at`` is parsed from ``<updated>`` / ``<pubDate>``, so the+ age gate and the future-date deferral judge vendor incidents the same+ way they judge ``rss`` and ``edgar`` entries."""from __future__ import annotationsimport hashlibimport logging+import refrom datetime import UTC, datetimefrom pathlib import Pathfrom urllib.parse import urlparse@@ -52,7 +70,10 @@ from openexecutive.monitoring.sources._http import (strip_url_query,validate_target_url,)-from openexecutive.monitoring.sources.base import collapse_whitespace+from openexecutive.monitoring.sources.base import (+ collapse_whitespace,+ feed_text_published_at,+)logger = logging.getLogger(__name__)@@ -60,25 +81,56 @@ logger = logging.getLogger(__name__)# at /history.atom; a few publish RSS 2.0 at /rss/all.rss. We support# both: see _parse_feed._ATOM_NS = "{http://www.w3.org/2005/Atom}"+# RSS feeds that carry HTML bodies use content:encoded; ElementTree needs+# the expanded name (a bare "content:encoded" raises on the unknown prefix).+_CONTENT_ENCODED = "{http://purl.org/rss/1.0/modules/content/}encoded"# The documented Statuspage history feed is bounded (~25 entries) — a# runaway count here means the feed is malformed and we should stop# reading rather than process megabytes of garbage._MAX_ENTRIES_PER_FEED = 100+# Incident status vocabulary. Statuspage renders each update in the entry+# body as "<strong>Resolved</strong> - ...", newest update first, and uses+# one of these labels: incidents run Investigating → Identified →+# Monitoring → Resolved (with "Update" for interim notes and "Postmortem"+# after the fact), scheduled maintenances run Scheduled → In progress →+# Verifying → Completed. OPEN means "still happening" — the operator wants+# to hear about it the moment the watch is added.+_OPEN_STATUSES = frozenset({+ "investigating", "identified", "monitoring", "update",+ "scheduled", "in progress", "verifying",+})+_CLOSED_STATUSES = frozenset({"resolved", "completed", "postmortem"})++# Newest-update label in a Statuspage entry body. Bounded repetition, and+# only over the head of the body (see _BODY_SCAN_CHARS), so a hostile feed+# can't make this scan expensive.+_STRONG_LABEL_RE = re.compile(r"<strong>\s*([^<>]{1,40}?)\s*</strong>", re.IGNORECASE)+# Same label, in a body that carries no markup for us to key on — an+# xhtml-typed Atom <content> (real child elements, so the tags never reach+# us as text) or a plain-text update. Statuspage's own wording puts the+# label first: "Resolved - The issue has been fixed."+_TEXT_LABEL_RE = re.compile(r"^\s*([A-Za-z][A-Za-z ]{1,18}?)\s*[-–—:]\s", re.IGNORECASE)+# AWS's rss/all.rss carries no per-update markup; it stamps the resolution+# into the title instead ("Service is operating normally: [RESOLVED] …").+_TITLE_MARKER_RE = re.compile(r"\[\s*(resolved|completed)\s*\]", re.IGNORECASE)+# The newest update sits at the top of the body; a few KB is far more than+# enough to find it and bounds the regex work regardless of body size.+_BODY_SCAN_CHARS = 8_000+class VendorStatusSource:kind: str = SOURCE_KIND_VENDOR_STATUSdefault_poll_interval_minutes: int = 5- # Deliberately outside BOTH freshness gates. An incident's entry id- # never changes across its updates, and a suppressed insert burns the- # dedup key, so either a first-poll baseline or an age gate on- # <updated> would permanently mute an incident that is open (or goes- # quiet for a week) when the watch is added. The history feed does- # replay resolved incidents on a new watch; fixing that needs a dedup- # key that includes <updated> or an open/resolved filter — tracked as- # a follow-up, not folded into the #80 gates.- seed_on_first_poll: bool = False+ # A history feed returns its whole archive on every poll, so the first+ # poll is a baseline — otherwise every incident the vendor ever had+ # fires as news when the watch is added (issue #90). Safe here ONLY+ # because the dedup key carries <updated> (see _make_dedup_key): a+ # baselined entry burns one STATE of an incident, not the incident, so+ # the next update surfaces. Incidents that are still open are exempted+ # from the baseline entirely — see promote_on_baseline.+ seed_on_first_poll: bool = Trueasync def poll(self, item: WatchlistItem, *, db_path: Path | None = None@@ -141,12 +193,16 @@ class VendorStatusSource:continuetitle = collapse_whitespace(entry.get("title") or "") or "(untitled incident)"summary = f"[{vendor_label}] {title}"- dedup_key = _make_dedup_key(item.slug, entry_id)+ updated = entry.get("updated", "")signals.append(Signal(watchlist_id=item.id or 0,source_kind=self.kind,+ # The incident, not the state — two states of one incident+ # share it, which is what makes an operator's "show me this+ # incident" query work. Uniqueness lives on dedup_key.source_external_id=entry_id[:500],captured_at=datetime.now(UTC).isoformat(),+ published_at=feed_text_published_at(updated),normalized_summary=summary[:500],raw_payload={"vendor_label": vendor_label,@@ -154,11 +210,12 @@ class VendorStatusSource:"entry_id": entry_id,"title": title,"link": entry.get("link", ""),- "updated": entry.get("updated", ""),+ "updated": updated,+ "status": _latest_status(entry.get("body", ""), title),},provenance_url=entry.get("link") or item.target,severity_hint=AlertSeverity.HIGH,- dedup_key=dedup_key,+ dedup_key=_make_dedup_key(item.slug, entry_id, updated),))return signals@@ -170,9 +227,26 @@ class VendorStatusSource:haystack = (signal.normalized_summary or "").lower()return any(kw.lower() in haystack for kw in keywords)+ def promote_on_baseline(self, signal: Signal, item: WatchlistItem) -> bool:+ """First poll only: True for an incident that is still OPEN.++ The optional hook documented on ``sources.base.Source``. A history+ feed's archive is what the baseline exists to swallow, but an+ incident that is open the moment the watch is added is live news —+ the operator adding a Stripe watch during a Stripe outage must hear+ about it now, not at the vendor's next update.++ Unrecognised status counts as CLOSED, deliberately: a vendor that+ changes its feed format must not be able to promote its whole+ archive at HIGH. Nothing is lost for good — that entry is recorded+ as baseline, and the next ``<updated>`` bump mints a new dedup key+ and surfaces normally.+ """+ return _is_open(str(signal.raw_payload.get("status") or ""))+def _parse_feed(body: bytes) -> list[dict[str, str]]:- """Parse an Atom or RSS feed into a list of {id, title, link, updated} dicts.+ """Parse a feed into {id, title, link, updated, body} dicts.Uses ``defusedxml.ElementTree`` to block entity-expansion attacks(billion-laughs, quadratic-blowup) — see the module-level import@@ -205,7 +279,10 @@ def _parse_atom(feed: Element) -> list[dict[str, str]]:if link_el is not None:link = (link_el.get("href") or "").strip()link = strip_url_query(link) if link else ""- out.append({"id": entry_id, "title": title, "link": link, "updated": updated})+ out.append({+ "id": entry_id, "title": title, "link": link, "updated": updated,+ "body": _element_text(entry, (f"{_ATOM_NS}content", f"{_ATOM_NS}summary")),+ })return out@@ -220,10 +297,74 @@ def _parse_rss(channel: Element) -> list[dict[str, str]]:# Prefer guid for id (stable upstream identifier); fall back to# the cleaned link so dedup remains stable.entry_id = guid or link- out.append({"id": entry_id, "title": title, "link": link, "updated": pub})+ out.append({+ "id": entry_id, "title": title, "link": link, "updated": pub,+ "body": _element_text(item, ("description", _CONTENT_ENCODED)),+ })return out+def _element_text(parent: Element, tags: tuple[str, ...]) -> str:+ """Text of the first present child in ``tags``, or "".++ ``itertext`` rather than ``findtext`` so an XHTML-typed Atom+ ``<content>`` (child elements rather than escaped markup) still yields+ its text instead of an empty string.+ """+ for tag in tags:+ el = parent.find(tag)+ if el is None:+ continue+ text = "".join(el.itertext()).strip()+ if text:+ return text+ return ""+++def _latest_status(body: str, title: str) -> str:+ """The incident's CURRENT status, lowercased, or "" when unknown.++ Statuspage puts the newest update first in the entry body and labels+ it ``<strong>Resolved</strong>`` / ``Investigating`` / …, so the first+ recognised label in the body head is the incident's state right now.+ Two fallbacks for feeds shaped differently: a label leading the body+ text (an xhtml-typed ``<content>`` reaches us as text with its tags+ already consumed by the parser, and some vendors publish plain-text+ updates), then a title marker (AWS stamps ``[RESOLVED]`` there and+ publishes no per-update markup at all).++ Returns "" for anything unrecognised rather than guessing — callers+ decide what to do with an unknown status, and ``_is_open`` fails+ closed.+ """+ head = body[:_BODY_SCAN_CHARS]+ for match in _STRONG_LABEL_RE.finditer(head):+ label = _known_status(match.group(1))+ if label:+ return label+ leading = _TEXT_LABEL_RE.match(head)+ if leading:+ label = _known_status(leading.group(1))+ if label:+ return label+ marker = _TITLE_MARKER_RE.search(title)+ return marker.group(1).lower() if marker else ""+++def _known_status(raw: str) -> str:+ """``raw`` normalised to a status we know, or "" — never a guess."""+ label = collapse_whitespace(raw).lower()+ return label if label in _OPEN_STATUSES or label in _CLOSED_STATUSES else ""+++def _is_open(status: str) -> bool:+ """True only for a status we recognise AND that means "still happening".++ Fails closed on "" (unknown): see ``VendorStatusSource.promote_on_baseline``.+ """+ return status in _OPEN_STATUSES++def _host_label(url: str) -> str:"""Best-effort vendor name from a URL host when config doesn't set one."""try:@@ -235,14 +376,28 @@ def _host_label(url: str) -> str:return parts[0] if parts else host or "vendor"-def _make_dedup_key(slug: str, entry_id: str) -> str:- """Deterministic dedup_key over (watchlist slug, upstream entry id).+def _make_dedup_key(slug: str, entry_id: str, updated: str) -> str:+ """Deterministic dedup_key over (watchlist slug, entry id, ``<updated>``).Hashed because Atom ids can be 200+ char tag URIs; the watchlistslug prefix ensures two different watchlist rows pointing at thesame vendor still produce distinct keys (so each row's owntrigger / routing fires independently).++ ``<updated>`` is in the key on purpose (issue #90). An incident's id+ is stable across its whole lifetime, so keying on the id alone made+ every suppression permanent — one baselined or stale insert and that+ incident could never surface again, which is why this adapter used to+ sit outside both freshness gates. Keyed on the id AND the update+ stamp, a suppressed insert burns a single state: re-polling an+ unchanged feed still dedups (same id, same stamp), while an incident+ moving Investigating → Resolved mints a new key and surfaces.++ A feed that omits ``<updated>`` degrades to a stable per-incident key+ (the pre-#90 behaviour) rather than to one that changes every poll —+ an entry with no stamp has nothing to change, and a per-poll key+ would re-alert forever."""- payload = f"{slug}\x00{entry_id}".encode()+ payload = f"{slug}\x00{entry_id}\x00{updated}".encode()digest = hashlib.sha256(payload).hexdigest()[:32]return f"vendor_status:{digest}"diff --git a/packages/core/openexecutive/monitoring/store.py b/packages/core/openexecutive/monitoring/store.pyindex a67c0403..e649b260 100644--- a/packages/core/openexecutive/monitoring/store.py+++ b/packages/core/openexecutive/monitoring/store.py@@ -123,6 +123,16 @@ def initialize_db(db_path: Path | None = None) -> None:text_snapshot TEXT NOT NULL DEFAULT '',updated_at TEXT NOT NULL);++ -- One row per one-time data migration that has been applied.+ -- There is no migration framework in this repo, and column+ -- guards (_ensure_column) only work for migrations that add a+ -- column; a pure data fix-up needs its own marker to run+ -- exactly once. See _apply_once.+ CREATE TABLE IF NOT EXISTS monitoring_migrations (+ name TEXT PRIMARY KEY,+ applied_at TEXT NOT NULL+ );""")# Additive migration for DBs created before enrichment landed.# ``CREATE TABLE IF NOT EXISTS`` above is a no-op on an existing@@ -150,6 +160,7 @@ def initialize_db(db_path: Path | None = None) -> None:if not _has_column(conn, "watchlist", "baselined_at"):_migrate_baselined_at(conn)# Add future migrations BELOW, not inside the guard above.+ _migrate_vendor_status_rekey(conn)def _migrate_baselined_at(conn: sqlite3.Connection) -> None:@@ -170,6 +181,66 @@ def _migrate_baselined_at(conn: sqlite3.Connection) -> None:raise+_MIGRATION_VENDOR_STATUS_REKEY = "vendor_status_dedup_rekey_90"+++def _apply_once(conn: sqlite3.Connection, name: str) -> bool:+ """Claim a one-time data migration. True only for the caller that won.++ Must be called INSIDE the caller's transaction so the claim and the+ migration's own writes commit or roll back together — a marker without+ its data change would skip the migration for good.+ """+ claimed = conn.execute(+ "INSERT OR IGNORE INTO monitoring_migrations (name, applied_at) "+ "VALUES (?, ?)",+ (name, datetime.now(UTC).isoformat()),+ )+ return claimed.rowcount > 0+++def _migrate_vendor_status_rekey(conn: sqlite3.Connection) -> None:+ """One-time: let ``vendor_status`` rows re-seed after the #90 rekey.++ ``vendor_status`` dedup keys used to be ``sha256(slug, entry id)`` and+ are now ``sha256(slug, entry id, <updated>)``, so on the upgrade tick+ every entry in a vendor's history feed hashes to a key the DB has+ never seen. For rows already stamped ``baselined_at`` (the #80+ backfill stamped every previously-polled row) that would replay the+ archive as HIGH alerts — the very bug #90 is about, just once.++ Clearing the stamp makes that tick a first poll under the new scheme:+ the resolved archive is recorded as ``suppressed_baseline`` and only+ incidents that are still open are promoted. Signal rows are left+ alone — the old keys are inert, and deleting them would lose the+ audit trail of what the row surfaced before the upgrade.+ """+ already = conn.execute(+ "SELECT 1 FROM monitoring_migrations WHERE name = ?",+ (_MIGRATION_VENDOR_STATUS_REKEY,),+ ).fetchone()+ if already is not None:+ # Steady-state boot: no write transaction, so startup never+ # contends for the write lock on the shared episodic_memory.db.+ return+ conn.execute("BEGIN IMMEDIATE")+ try:+ if _apply_once(conn, _MIGRATION_VENDOR_STATUS_REKEY):+ reset = conn.execute(+ "UPDATE watchlist SET baselined_at = NULL "+ "WHERE signal_type = 'vendor_status' AND baselined_at IS NOT NULL"+ )+ if reset.rowcount:+ logger.info(+ "monitoring.store: re-seeding %d vendor_status row(s) after "+ "the #90 dedup rekey", reset.rowcount,+ )+ conn.execute("COMMIT")+ except Exception:+ conn.execute("ROLLBACK")+ raise++def _ensure_column(conn: sqlite3.Connection, table: str, column: str, decl: str) -> bool:diff --git a/packages/core/tests/unit/test_monitoring_pipeline.py b/packages/core/tests/unit/test_monitoring_pipeline.pyindex fec1b0bb..f61bbb56 100644--- a/packages/core/tests/unit/test_monitoring_pipeline.py+++ b/packages/core/tests/unit/test_monitoring_pipeline.py@@ -10,6 +10,7 @@ pipeline orchestration code instead.from __future__ import annotationsimport asyncio+import sqlite3from collections.abc import Callablefrom datetime import UTC, datetime, timedeltafrom pathlib import Path@@ -62,12 +63,24 @@ class _FakeSource:default_poll_interval_minutes: int = 5seed_on_first_poll: bool = False- def __init__(self, signals: list[Signal], *, seed: bool = False) -> None:+ def __init__(+ self,+ signals: list[Signal],+ *,+ seed: bool = False,+ promote: Any = None,+ ) -> None:self._signals = signalsself.calls = 0self.seed_on_first_poll = seed# Optional per-signal trigger predicate (default: everything matches).self.match: Any = None+ # The optional ``promote_on_baseline`` hook (sources.base.Source) is+ # bound as an INSTANCE attribute only when a test asks for one, so+ # the default fake has no such attribute at all — that is the shape+ # rss / edgar present to the pipeline's getattr lookup.+ if promote is not None:+ self.promote_on_baseline = promote # type: ignore[attr-defined]async def poll(self, item: WatchlistItem, *, db_path: Path | None = None@@ -91,8 +104,10 @@ def install_fake_source(monkeypatch: pytest.MonkeyPatch):original = SOURCE_REGISTRY.get("vendor_status")installed: list[_FakeSource] = []- def _set(signals: list[Signal], *, seed: bool = False) -> _FakeSource:- fake = _FakeSource(signals, seed=seed)+ def _set(+ signals: list[Signal], *, seed: bool = False, promote: Any = None,+ ) -> _FakeSource:+ fake = _FakeSource(signals, seed=seed, promote=promote)SOURCE_REGISTRY["vendor_status"] = fake # type: ignore[assignment]installed.append(fake)return fake@@ -511,6 +526,174 @@ def test_first_poll_baselines_seeding_source(assert outcomes["vendor_status:e1"] == OUTCOME_SUPPRESSED_BASELINE # untouched+def test_baseline_exempt_entries_are_promoted_on_the_first_poll(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """``promote_on_baseline`` (issue #90): an adapter may carve live news+ out of its own first-poll baseline. The archive is still recorded and+ never promoted; the exempt entry runs the ordinary cascade."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "vendor-stripe")+ install_fake_source(+ [+ _make_signal(dedup_key="vendor_status:open", published_at=_iso_days_ago(0.1)),+ _make_signal(dedup_key="vendor_status:old1", published_at=_iso_days_ago(120)),+ _make_signal(dedup_key="vendor_status:old2", published_at=_iso_days_ago(200)),+ ],+ seed=True,+ promote=lambda signal, item: signal.dedup_key.endswith(":open"),+ )++ written = asyncio.run(mp.run_external_monitor_scan(db_path=db))++ assert written == 3 # archive AND the exempt entry are recorded+ assert [e.external_id for e in promoted] == ["vendor_status:open"]+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert outcomes == {+ "vendor_status:open": OUTCOME_ALERTED,+ "vendor_status:old1": OUTCOME_SUPPRESSED_BASELINE,+ "vendor_status:old2": OUTCOME_SUPPRESSED_BASELINE,+ }+ item = ms.list_watchlist(db_path=db)[0]+ assert item.baselined_at is not None # the row is seeded, so tick 2 is normal++ # Tick 2: the archive stays put, and the same entries don't re-fire.+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert [e.external_id for e in promoted] == ["vendor_status:open"]+++_VENDOR_ATOM = """<?xml version="1.0" encoding="UTF-8"?>+<feed xmlns="http://www.w3.org/2005/Atom">+ <entry>+ <id>tag:status.stripe.com,2005:Incident/2001</id>+ <updated>{open_updated}</updated>+ <link rel="alternate" href="https://status.stripe.com/incidents/2001"/>+ <title>Elevated API error rates</title>+ <content type="html"><p><strong>{open_status}</strong> - {open_note}</p></content>+ </entry>+ <entry>+ <id>tag:status.stripe.com,2005:Incident/1004</id>+ <updated>2026-03-02T18:00:00+00:00</updated>+ <link rel="alternate" href="https://status.stripe.com/incidents/1004"/>+ <title>Dashboard latency</title>+ <content type="html"><p><strong>Resolved</strong> - Fixed months ago.</p></content>+ </entry>+</feed>+"""+++def _install_vendor_feed(monkeypatch: pytest.MonkeyPatch, body: str) -> None:+ """Point the REAL vendor_status adapter at a canned feed."""+ async def fake_fetch(url: str, max_bytes: int) -> bytes:+ return body.encode()++ monkeypatch.setattr(+ "openexecutive.monitoring.sources.vendor_status.fetch_bounded", fake_fetch+ )+ monkeypatch.setattr(+ "openexecutive.monitoring.sources.vendor_status.validate_target_url",+ lambda u: (True, ""),+ )+++def test_vendor_status_first_poll_alerts_the_outage_not_the_archive(+ db: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """End to end on the REAL adapter (issue #90's acceptance criteria):+ a watch added mid-outage reports the outage and swallows the vendor's+ resolved history, and the incident re-fires when its status changes."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "vendor-stripe", target="https://status.stripe.com/history.atom")+ _install_vendor_feed(monkeypatch, _VENDOR_ATOM.format(+ open_updated=_iso_days_ago(0.02),+ open_status="Investigating",+ open_note="We are looking into elevated error rates.",+ ))++ asyncio.run(mp.run_external_monitor_scan(db_path=db))++ outcomes = {+ r["normalized_summary"]: r["processed_outcome"]+ for r in ms.list_recent_signals(db_path=db)+ }+ assert outcomes == {+ "[stripe] Elevated API error rates": OUTCOME_ALERTED,+ "[stripe] Dashboard latency": OUTCOME_SUPPRESSED_BASELINE,+ }+ assert len(promoted) == 1++ item = ms.list_watchlist(db_path=db)[0]++ # Same feed again: nothing new, nothing re-fires.+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert len(promoted) == 1++ # The incident resolves — a new <updated>, so a new dedup key, so the+ # status change reaches the principal instead of being muted for good.+ _install_vendor_feed(monkeypatch, _VENDOR_ATOM.format(+ open_updated=_iso_days_ago(0.01),+ open_status="Resolved",+ open_note="Error rates are back to normal.",+ ))+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert len(promoted) == 2+++def test_every_entry_exempt_still_stamps_the_baseline(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A first poll where the adapter exempts everything (a vendor whose+ every listed incident is open) must still claim the stamp — otherwise+ the next tick would baseline the feed and swallow real news."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "vendor-aws")+ install_fake_source(+ [_make_signal(dedup_key="vendor_status:live", published_at=_iso_days_ago(0.1))],+ seed=True,+ promote=lambda signal, item: True,+ )++ asyncio.run(mp.run_external_monitor_scan(db_path=db))++ assert [e.external_id for e in promoted] == ["vendor_status:live"]+ assert ms.list_watchlist(db_path=db)[0].baselined_at is not None+++def test_crashing_promote_on_baseline_falls_back_to_baseline(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A hook that raises must not turn a first poll into a back-catalogue+ replay — the same fail-quiet stance matches_trigger takes."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "vendor-broken")++ def _boom(signal: Signal, item: WatchlistItem) -> bool:+ raise RuntimeError("hook is broken")++ install_fake_source(+ [_make_signal(dedup_key="vendor_status:x", published_at=_iso_days_ago(0.1))],+ seed=True,+ promote=_boom,+ )++ written = asyncio.run(mp.run_external_monitor_scan(db_path=db))++ assert written == 1+ assert promoted == []+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert outcomes == {"vendor_status:x": OUTCOME_SUPPRESSED_BASELINE}++def test_failed_first_fetch_does_not_forfeit_baseline(db: Path,install_fake_source,@@ -621,6 +804,35 @@ def test_baselined_at_backfilled_for_previously_polled_rows(tmp_path: Path) -> Nassert by_slug["never-polled"].baselined_at is None+def test_vendor_status_rows_reseed_once_after_the_rekey(db: Path) -> None:+ """Issue #90: vendor_status dedup keys now carry <updated>, so on the+ upgrade tick every archive entry hashes to a key the DB has never seen.+ Clearing ``baselined_at`` on exactly those rows makes that tick a first+ poll (archive suppressed, open incidents promoted) instead of a replay+ — and it must happen once, not on every boot."""+ stamp = "2026-09-01T00:00:00+00:00"+ vendor_id = _insert_feed(db, "vendor-stripe")+ rss_id = _insert_feed(db, "acme-changelog", signal_type="rss")+ for wl_id in (vendor_id, rss_id):+ ms.record_baseline(wl_id, datetime.fromisoformat(stamp), [], db_path=db)++ # The db fixture already ran initialize_db, so the marker is set and+ # this row's stamp survives — simulate the pre-upgrade state by+ # dropping the marker, then re-running the migration.+ with sqlite3.connect(db) as conn:+ conn.execute("DELETE FROM monitoring_migrations")+ ms.initialize_db(db)++ by_slug = {i.slug: i for i in ms.list_watchlist(db_path=db)}+ assert by_slug["vendor-stripe"].baselined_at is None # re-seeds+ assert by_slug["acme-changelog"].baselined_at == stamp # rss untouched++ # Idempotent: a row baselined AFTER the migration is never reset again.+ ms.record_baseline(vendor_id, datetime.fromisoformat(stamp), [], db_path=db)+ ms.initialize_db(db)+ assert ms.get_watchlist_item(vendor_id, db_path=db).baselined_at == stamp++def test_record_baseline_is_a_compare_and_swap(db: Path) -> None:"""Exactly one caller records the baseline for a row; a second attempt(or one on an already-baselined row) returns None and writes nothing."""diff --git a/packages/core/tests/unit/test_monitoring_vendor_status.py b/packages/core/tests/unit/test_monitoring_vendor_status.pynew file mode 100644index 00000000..fecc98d0--- /dev/null+++ b/packages/core/tests/unit/test_monitoring_vendor_status.py@@ -0,0 +1,442 @@+"""Unit tests for the vendor_status adapter (issue #90).++The adapter used to key its ``dedup_key`` on the incident id alone, which+made every suppression permanent and kept it outside both #80 freshness+gates — so a new watch replayed the vendor's whole resolved archive as+HIGH alerts. These tests pin the three moving parts of the fix: the key+now carries ``<updated>``, ``published_at`` is parsed from the feed, and+``promote_on_baseline`` exempts incidents that are still open from the+first-poll baseline.+"""+from __future__ import annotations++import pytest++from openexecutive.alerts.models import AlertSeverity+from openexecutive.monitoring.models import WatchlistItem+from openexecutive.monitoring.sources.base import feed_text_published_at+from openexecutive.monitoring.sources.vendor_status import (+ VendorStatusSource,+ _is_open,+ _latest_status,+ _make_dedup_key,+)++# --------------------------------------------------------------------- #+# Helpers / fixtures+# --------------------------------------------------------------------- #+++def _make_item(+ *,+ slug: str = "vendor-stripe",+ target: str = "https://status.stripe.com/history.atom",+ config: dict | None = None,+ trigger: dict | None = None,+) -> WatchlistItem:+ return WatchlistItem(+ id=1,+ slug=slug,+ signal_type="vendor_status",+ target=target,+ config_json=config or {},+ trigger_json=trigger or {},+ )+++def _install_feed(monkeypatch: pytest.MonkeyPatch, body: bytes) -> None:+ async def fake_fetch(url: str, max_bytes: int) -> bytes:+ return body++ monkeypatch.setattr(+ "openexecutive.monitoring.sources.vendor_status.fetch_bounded", fake_fetch+ )+ monkeypatch.setattr(+ "openexecutive.monitoring.sources.vendor_status.validate_target_url",+ lambda u: (True, ""),+ )+++def _atom_entry(+ *, incident: str, title: str, updated: str, body: str,+) -> str:+ return f"""+ <entry>+ <id>tag:status.stripe.com,2005:Incident/{incident}</id>+ <published>2026-09-05T09:00:00Z</published>+ <updated>{updated}</updated>+ <link rel="alternate" type="text/html"+ href="https://status.stripe.com/incidents/{incident}?utm_source=feed"/>+ <title>{title}</title>+ <content type="html">{body}</content>+ </entry>"""+++def _atom_feed(*entries: str) -> bytes:+ joined = "".join(entries)+ return f"""<?xml version="1.0" encoding="UTF-8"?>+<feed xmlns="http://www.w3.org/2005/Atom">+ <title>Stripe Status</title>+ <updated>2026-09-07T12:00:00Z</updated>{joined}+</feed>+""".encode()+++# An incident still in progress: Statuspage lists the newest update first,+# so "Monitoring" is its current state even though "Investigating" follows.+_OPEN_BODY = (+ "<p><small>Sep 7, 11:30 UTC</small><br>"+ "<strong>Monitoring</strong> - A fix has been applied.</p>"+ "<p><strong>Investigating</strong> - Looking into it.</p>"+)+_RESOLVED_BODY = (+ "<p><strong>Resolved</strong> - This incident is resolved.</p>"+ "<p><strong>Investigating</strong> - Looking into it.</p>"+)++_OPEN_ENTRY = _atom_entry(+ incident="2001", title="Elevated API error rates",+ updated="2026-09-07T11:30:00Z", body=_OPEN_BODY,+)+_RESOLVED_ENTRY = _atom_entry(+ incident="1004", title="Dashboard latency",+ updated="2026-03-02T18:00:00Z", body=_RESOLVED_BODY,+)++_SAMPLE_ATOM = _atom_feed(_OPEN_ENTRY, _RESOLVED_ENTRY)++_SAMPLE_RSS = b"""<?xml version="1.0" encoding="UTF-8"?>+<rss version="2.0">+ <channel>+ <title>Twilio Status</title>+ <item>+ <guid>https://status.twilio.com/incidents/xyz</guid>+ <title>SMS delivery delays</title>+ <link>https://status.twilio.com/incidents/xyz?utm_source=feed</link>+ <pubDate>Mon, 07 Sep 2026 11:30:00 +0000</pubDate>+ <description><p><strong>Identified</strong> - Root cause found.</p></description>+ </item>+ </channel>+</rss>+"""++# AWS publishes one item per update and carries the resolution in the+# title instead of Statuspage's per-update markup.+_AWS_RSS = b"""<?xml version="1.0" encoding="UTF-8"?>+<rss version="2.0">+ <channel>+ <title>Amazon Web Services Service Status</title>+ <item>+ <guid>http://status.aws.amazon.com/#ec2-us-east-1_1757251800</guid>+ <title>Service is operating normally: [RESOLVED] Increased error rates</title>+ <link>http://status.aws.amazon.com/</link>+ <pubDate>Mon, 07 Sep 2026 12:10:00 PDT</pubDate>+ <description>Between 9:00 AM and 11:30 AM PDT we experienced errors.</description>+ </item>+ </channel>+</rss>+"""+++# --------------------------------------------------------------------- #+# Status parsing+# --------------------------------------------------------------------- #+++# _latest_status sees the body AFTER the XML parser has unescaped it, so+# these are the literal-markup forms of _OPEN_BODY / _RESOLVED_BODY above.+_OPEN_BODY_TEXT = (+ "<p><small>Sep 7, 11:30 UTC</small><br>"+ "<strong>Monitoring</strong> - A fix has been applied.</p>"+ "<p><strong>Investigating</strong> - Looking into it.</p>"+)+_RESOLVED_BODY_TEXT = (+ "<p><strong>Resolved</strong> - This incident is resolved.</p>"+ "<p><strong>Investigating</strong> - Looking into it.</p>"+)+++ ("body", "title", "expected"),+ [+ (_OPEN_BODY_TEXT, "Elevated API error rates", "monitoring"),+ (_RESOLVED_BODY_TEXT, "Dashboard latency", "resolved"),+ ("<p><strong>Investigating</strong> - digging in.</p>", "x", "investigating"),+ ("<p><strong>Identified</strong> - found it.</p>", "x", "identified"),+ ("<p><strong>Scheduled</strong> - maintenance window.</p>", "x", "scheduled"),+ ("<p><strong>Completed</strong> - maintenance done.</p>", "x", "completed"),+ ("<p><strong>Postmortem</strong> - write-up.</p>", "x", "postmortem"),+ # Case and whitespace are the vendor's business, not ours.+ ("<p><STRONG> resolved </STRONG> - done.</p>", "x", "resolved"),+ # No markup at all: a label leading the body text still counts+ # (xhtml <content>, or a vendor publishing plain-text updates).+ ("Investigating - we are looking into it.", "x", "investigating"),+ ("Completed: the maintenance window is over.", "x", "completed"),+ # …but only at the start; mid-body prose must not be mined for one.+ ("We were monitoring - then it recovered.", "x", ""),+ # No per-update markup: fall back to the title marker (AWS).+ ("plain text body", "[RESOLVED] Increased error rates", "resolved"),+ # A <strong> that isn't a status label must not be mistaken for one.+ ("<p><strong>Note</strong> - unrelated bold text.</p>", "x", ""),+ ("", "", ""),+ ],+)+def test_latest_status_reads_the_newest_update(+ body: str, title: str, expected: str,+) -> None:+ assert _latest_status(body, title) == expected+++def test_is_open_fails_closed_on_unknown_status() -> None:+ """An unrecognised status must NOT count as open: a vendor changing its+ feed format must not be able to promote its whole archive at HIGH."""+ assert _is_open("investigating") is True+ assert _is_open("monitoring") is True+ assert _is_open("resolved") is False+ assert _is_open("postmortem") is False+ assert _is_open("") is False+ assert _is_open("nonsense") is False+++# --------------------------------------------------------------------- #+# published_at parsing (sources.base.feed_text_published_at)+# --------------------------------------------------------------------- #+++ ("raw", "expected"),+ [+ ("2026-09-07T11:30:00Z", "2026-09-07T11:30:00+00:00"),+ ("2026-09-07T04:30:00-07:00", "2026-09-07T11:30:00+00:00"),+ ("Mon, 07 Sep 2026 11:30:00 +0000", "2026-09-07T11:30:00+00:00"),+ ("Mon, 07 Sep 2026 11:30:00 GMT", "2026-09-07T11:30:00+00:00"),+ # Naive values are read as UTC, like every other date path here.+ ("2026-09-07T11:30:00", "2026-09-07T11:30:00+00:00"),+ ("", None),+ (None, None),+ ("not a date", None),+ ],+)+def test_feed_text_published_at(raw: str | None, expected: str | None) -> None:+ assert feed_text_published_at(raw) == expected+++# --------------------------------------------------------------------- #+# Dedup key+# --------------------------------------------------------------------- #+++def test_dedup_key_changes_with_updated() -> None:+ """The whole fix: an incident id is stable for the incident's life, so+ the key must carry its update stamp or a suppression is permanent."""+ first = _make_dedup_key("vendor-stripe", "Incident/2001", "2026-09-07T09:00:00Z")+ same = _make_dedup_key("vendor-stripe", "Incident/2001", "2026-09-07T09:00:00Z")+ later = _make_dedup_key("vendor-stripe", "Incident/2001", "2026-09-07T11:30:00Z")+ other_row = _make_dedup_key("vendor-aws", "Incident/2001", "2026-09-07T09:00:00Z")++ assert first == same # an unchanged feed still dedups+ assert first != later # a status change surfaces again+ assert first != other_row # rows stay independent+ assert later.startswith("vendor_status:")+++def test_dedup_key_without_updated_is_stable_per_incident() -> None:+ """A feed with no <updated> degrades to a per-incident key rather than+ one that changes every poll — which would re-alert forever."""+ a = _make_dedup_key("vendor-x", "Incident/7", "")+ b = _make_dedup_key("vendor-x", "Incident/7", "")+ assert a == b+++# --------------------------------------------------------------------- #+# poll()+# --------------------------------------------------------------------- #+++async def test_poll_emits_status_published_at_and_keys(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ _install_feed(monkeypatch, _SAMPLE_ATOM)+ src = VendorStatusSource()+ item = _make_item(config={"vendor_label": "Stripe"})++ signals = await src.poll(item)++ assert len(signals) == 2+ live, archived = signals+ assert live.normalized_summary == "[Stripe] Elevated API error rates"+ assert live.raw_payload["status"] == "monitoring"+ assert live.published_at == "2026-09-07T11:30:00+00:00"+ assert live.severity_hint is AlertSeverity.HIGH+ # Tracking query stripped from the click-through URL.+ assert live.provenance_url == "https://status.stripe.com/incidents/2001"+ assert archived.raw_payload["status"] == "resolved"+ assert archived.published_at == "2026-03-02T18:00:00+00:00"+ # source_external_id stays the incident, so both states of one incident+ # remain findable together; only dedup_key distinguishes them.+ assert live.source_external_id.endswith("Incident/2001")+ assert live.dedup_key != archived.dedup_key+++async def test_identical_poll_produces_identical_dedup_keys(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Re-polling an unchanged feed must dedup — the status page returns+ the same entries every five minutes."""+ _install_feed(monkeypatch, _SAMPLE_ATOM)+ src = VendorStatusSource()+ item = _make_item()++ first = await src.poll(item)+ second = await src.poll(item)++ assert [s.dedup_key for s in first] == [s.dedup_key for s in second]+++async def test_incident_update_mints_a_new_key(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Same incident, one status update later: a new key, so it re-fires."""+ src = VendorStatusSource()+ item = _make_item()++ _install_feed(monkeypatch, _atom_feed(_OPEN_ENTRY))+ before = (await src.poll(item))[0]++ resolved_now = _atom_entry(+ incident="2001", title="Elevated API error rates",+ updated="2026-09-07T13:05:00Z", body=_RESOLVED_BODY,+ )+ _install_feed(monkeypatch, _atom_feed(resolved_now))+ after = (await src.poll(item))[0]++ assert before.source_external_id == after.source_external_id+ assert before.dedup_key != after.dedup_key+ assert before.raw_payload["status"] == "monitoring"+ assert after.raw_payload["status"] == "resolved"+++async def test_promote_on_baseline_exempts_only_open_incidents(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """The first-poll contract: an open incident is live news, the resolved+ archive is not."""+ _install_feed(monkeypatch, _SAMPLE_ATOM)+ src = VendorStatusSource()+ item = _make_item()++ live, archived = await src.poll(item)++ assert src.seed_on_first_poll is True+ assert src.promote_on_baseline(live, item) is True+ assert src.promote_on_baseline(archived, item) is False+++async def test_unknown_status_is_baselined_on_first_poll(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A feed whose format we don't recognise must not promote its archive.+ Nothing is lost: the next <updated> bump mints a new key."""+ unknown = _atom_entry(+ incident="9", title="Something happened",+ updated="2026-09-07T11:30:00Z", body="plain text, no markup",+ )+ _install_feed(monkeypatch, _atom_feed(unknown))+ src = VendorStatusSource()+ item = _make_item()++ signal = (await src.poll(item))[0]++ assert signal.raw_payload["status"] == ""+ assert src.promote_on_baseline(signal, item) is False+++async def test_rss_status_and_pubdate(monkeypatch: pytest.MonkeyPatch) -> None:+ """Statuspage's RSS variant carries the same markup in <description>."""+ _install_feed(monkeypatch, _SAMPLE_RSS)+ src = VendorStatusSource()+ item = _make_item(target="https://status.twilio.com/history.rss")++ signal = (await src.poll(item))[0]++ assert signal.raw_payload["status"] == "identified"+ assert signal.published_at == "2026-09-07T11:30:00+00:00"+ assert src.promote_on_baseline(signal, item) is True+ # No vendor_label configured → derived from the host.+ assert signal.normalized_summary.startswith("[twilio] ")+++async def test_aws_rss_title_marker(monkeypatch: pytest.MonkeyPatch) -> None:+ _install_feed(monkeypatch, _AWS_RSS)+ src = VendorStatusSource()+ item = _make_item(target="https://status.aws.amazon.com/rss/all.rss")++ signal = (await src.poll(item))[0]++ assert signal.raw_payload["status"] == "resolved"+ assert src.promote_on_baseline(signal, item) is False+ assert signal.published_at is not None+++async def test_xhtml_content_still_yields_a_status(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """An Atom entry whose <content> holds real child elements (type=xhtml)+ rather than escaped markup — findtext would return only the leading+ text and lose the label."""+ feed = b"""<?xml version="1.0" encoding="UTF-8"?>+<feed xmlns="http://www.w3.org/2005/Atom">+ <entry>+ <id>tag:status.example.com,2005:Incident/5</id>+ <updated>2026-09-07T11:30:00Z</updated>+ <link rel="alternate" href="https://status.example.com/incidents/5"/>+ <title>Partial outage</title>+ <content type="xhtml"><p><strong>Investigating</strong> - looking.</p></content>+ </entry>+</feed>+"""+ _install_feed(monkeypatch, feed)+ src = VendorStatusSource()++ signal = (await src.poll(_make_item()))[0]++ assert signal.raw_payload["status"] == "investigating"+++async def test_entry_without_id_or_link_is_skipped(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Unchanged behaviour: no stable upstream id → no reliable dedup."""+ feed = b"""<?xml version="1.0" encoding="UTF-8"?>+<feed xmlns="http://www.w3.org/2005/Atom">+ <entry>+ <title>Anonymous incident</title>+ <updated>2026-09-07T11:30:00Z</updated>+ </entry>+</feed>+"""+ _install_feed(monkeypatch, feed)+ assert await VendorStatusSource().poll(_make_item()) == []+++async def test_keyword_trigger_still_filters_on_title(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ _install_feed(monkeypatch, _SAMPLE_ATOM)+ src = VendorStatusSource()+ item = _make_item(trigger={"keywords": ["api"]})++ live, archived = await src.poll(item)++ assert src.matches_trigger(live, item) is True+ assert src.matches_trigger(archived, item) is False