AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
fix(monitoring): harden the #80 freshness gates after adversarial review
johnrufusone07de6765fcfa
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yamlindex cec9e821..eb737e13 100644--- a/packages/core/openexecutive/architecture/architecture-facts.yaml+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml@@ -1184,8 +1184,9 @@ audit:- committee_review — one row per committee-reviewed turn; carries draft length, final length, per-reviewer critiques and timings (previously emitted but newly declared as a recognised event_type)- peer_memory — one row per `openexecutive.memory.honcho_client` call (prefetch / sync_turn / directional_chat / prefetch_department / sync_department_turn / append_department_note); carries `op`, `person_id`, optional `department_slug`, `outcome` (ok / timeout / error / disabled / no_person / no_department / no_loop / empty), `duration_ms`, plus op-specific details (query preview, reasoning_level, peer count, co_present_person_ids, co_present_department_slugs, originating_person_id, target_person_id, session_id, kind) so the flow-chart view renders the peer_memory path alongside `<retrieved_context>` and `<past_decisions>`. Disabled / no-Person / no-Department turns still emit (outcome marker) so audit absence equals "code path never ran" instead of "the provider was down."- external_signal_received — one row per signal an external-monitor adapter wrote to `external_signals`. Carries `signal_id`, `watchlist_id`, `watchlist_slug`, `source_kind`, `severity_hint`, `dedup_key`. The audit trail of what the monitor noticed, separate from what it surfaced.+ - external_signal_deferred — one row per poll that skipped entries whose published_at is more than the row's future-skew tolerance ahead (aggregated — `count`, `future_skew_hours`, `race_tick`, and up to 10 `entries` each with `dedup_key`, `published_at`, `summary` and `provenance_url`, so an operator can tell WHAT was held back). Emitted from the scan loop's finally, so it survives a poll that raises later in the same row. A deferred entry is by design NOT written to external_signals (that would burn its dedup key and mute it for good), so this row is its only trace until its date passes and it is judged normally.- external_signal_promoted — one row per signal promoted into the alerts pipeline (post-suppression cascade). Carries `signal_id`, `watchlist_id`, `watchlist_slug`, `source_kind`, `severity_hint`, `dedup_key`, `provenance_url`. Pair with `external_signal_received` to compute promotion rate per watchlist row.- - external_monitor_poll — one row per watchlist row polled in a scan tick. Carries `watchlist_id`, `watchlist_slug`, `signal_type`, `signals_emitted`, `duration_ms`. Enables per-source health drill-down (`GET /audit?event_type=external_monitor_poll`) without needing a dedicated debug endpoint.+ - external_monitor_poll — one row per watchlist row polled in a scan tick, emitted even when the row's poll raised or was cancelled (`failed` set to true, with `signals_emitted` still counting rows that landed before the failure, so the audit log stands on its own); `failed` is also true when the adapter crashed or no adapter is registered. A row with no registered adapter is still marked polled so it honours the cadence rather than re-firing every heartbeat. Carries `watchlist_id`, `watchlist_slug`, `signal_type`, `signals_emitted`, `duration_ms`, `failed`. Enables per-source health drill-down (`GET /audit?event_type=external_monitor_poll`) without needing a dedicated debug endpoint.- watchlist_research_periodic_ran — emitted by the periodic cron whenever the workflow actually fires. Carries `state_hash` (the input fingerprint at run time), `trigger` (`cold_start`, `state_changed`, or `staleness_floor`), `findings` count, `tool_calls` count, `ok_tool_calls`, `run_id`. The next tick reads the most recent row to decide skip-or-run (its `state_hash` for the fingerprint compare, its `ts` for the staleness floor). These three research run-history rows are the ONLY audit_log rows a fixture load/unload clears (via `monitoring.research.scheduler.clear_research_run_history`, called from `cli.fixture_loader._apply_state_from_source`): load wipes the profile/initiatives/watchlist that feed the fingerprint but otherwise preserves audit_log, so without this reset a same-fixture reload within the staleness floor would make the freshly-seeded scan skip itself and surface zero findings. A full reset still wipes all of audit_log.- watchlist_research_periodic_skipped — emitted when the state_hash matches the last `_ran` row. Carries `state_hash`. Visible cost-saving signal in the audit log.- watchlist_research_periodic_failed — emitted on workflow timeout / crash. Carries `state_hash` + `error` or `reason`.@@ -1254,8 +1255,25 @@ external_monitoring:never spend an LLM call: (1) `suppressed_baseline` — on the firstpoll that RETURNS ENTRIES for a row whose adapter declares`seed_on_first_poll` (rss, edgar) and whose `baselined_at` is NULL,- every entry in the feed is recorded as seen, nothing is promoted,- and `baselined_at` is set — so a new watch reports what changes+ every entry in the feed is recorded as seen and nothing is promoted.+ The claim and the recording are ONE transaction+ (store.record_baseline, BEGIN IMMEDIATE): a compare-and-swap on+ `baselined_at IS NULL` stamps the row, then EVERY fetched entry —+ trigger misses included, so the first entry that ever matches a+ keyword is news rather than "back-catalogue", and widening a+ trigger later can't resurface what was already in the feed — is+ inserted and marked suppressed_baseline; a failure part-way rolls+ the stamp back with the rows, so the next tick baselines the whole+ feed again rather than replaying an unrecorded tail. One row's+ store failure is logged and skipped so the rest of the scan runs.+ A scan that loses the race (overlapping heartbeat + client+ rotation, or a slow tick) re-reads the winner's stamp and treats+ entries published at or before it — and undated ones, since the+ feed controls published_at — as baseline, promoting entries dated+ after the stamp (and deferring, not recording, any dated more than+ a few minutes past its own clock), so neither the back-catalogue replays as news+ nor real news gets buried; later ticks process normally. The+ net effect is that a new watch reports what changesfrom now on rather than replaying the feed's back-catalogue(mirrors page_watch's first-observation baseline). Keyed onbaselined_at, not last_polled_at, so a failed/empty first fetch@@ -1264,12 +1282,27 @@ external_monitoring:feed can't be half-baselined and replay its tail next tick;(2) `suppressed_stale` — a signal whose published_at is older thanEXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS (default 7; 0 disables) at- capture time is recorded but never promoted. The alert body handed+ capture time is recorded but never promoted. An entry dated more+ than EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS (default 24; 0 disables;+ overridable per row via config_json.max_future_skew_hours, 0..8760,+ for a calendar or maintenance feed that legitimately dates ahead; an+ out-of-range or unparseable override is ignored with a warning, never+ coerced — the adapters use the same value when choosing between an+ entry's date keys) in the FUTURE is instead+ deferred — skipped for the tick and NOT recorded, since recording+ burns the dedup key and would let a feed mute an announcement for+ good by post-dating it; the poll emits one aggregated+ `external_signal_deferred` audit row so the skip is visible, and+ once the date passes the entry is judged like any other. On a+ lost-race tick the bound is a tight 5-minute drift tolerance+ instead (nothing the scan fetched can honestly be dated later). The alert body handedto triage carries `Published:` (when known) and `Discovered:`lines so anything inside the window is still judged on when it- happened, not when it was found. Adapters drop a published_at- more than a day in the future (a lying feed would otherwise never- age out) and collapse whitespace in feed titles so an entry can't+ happened, not when it was found. Adapters prefer a plausible+ date key over one more than a day in the future, and keep the+ future value when every key is implausible so the pipeline can+ defer it rather than wave an undated entry through; they also+ collapse whitespace in feed titles so an entry can'tforge its own labeled body line.enrichment_json holds the capture-time relevance read(relevance_score 0..1, why_it_matters, matched_initiative) written bydiff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.jsonindex 0e642e8b..ae1197cc 100644--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json@@ -1,7 +1,7 @@{"section_id": "external_monitoring","title": "External Monitoring",- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom incident feeds (AWS, Stripe, GitHub, …).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it, and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` is deliberately outside both gates: an incident's id never changes across updates, so either gate would permanently mute an incident that is open when the watch is added. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters drop a `published_at` more than a day in the future (a lying feed would never age out) and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",- "mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",- "generated_at": "2026-09-07T12:49:16+00:00"+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom incident feeds (AWS, Stripe, GitHub, …).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` is deliberately outside both gates: an incident's id never changes across updates, so either gate would permanently mute an incident that is open when the watch is added. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch. Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.",+ "mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",+ "generated_at": "2026-09-07T16:18:57+00:00"}diff --git a/packages/core/openexecutive/config.py b/packages/core/openexecutive/config.pyindex 184a267a..209b859b 100644--- a/packages/core/openexecutive/config.py+++ b/packages/core/openexecutive/config.py@@ -63,6 +63,14 @@ def _parse_csv_list(v: Any) -> list[str]:return [x.strip() for x in items if x.strip()]++# Bounds for the external-monitor freshness settings, shared with+# monitoring.sources.base so the per-row override is validated against the+# same range as the env var. One year of future skew / a century of age is+# far past any sane value and well inside datetime arithmetic limits.+MAX_FUTURE_SKEW_HOURS = 24 * 365+MAX_SIGNAL_AGE_DAYS = 365 * 100+class Settings(BaseSettings):model_config = SettingsConfigDict(env_file=str(_ENV_FILE),@@ -629,10 +637,23 @@ class Settings(BaseSettings):# older than this many days at capture time is recorded but never# promoted (outcome ``suppressed_stale``) — a feed that resurfaces a# January article in September must not become September news (issue- # #80). 0 disables the gate. Sources without an upstream timestamp- # (stock, page_watch, query) are unaffected.+ # #80). 0 disables the gate. A timestamp more than a day in the FUTURE+ # is deferred instead (skipped, not recorded, re-judged once the date+ # passes) so a feed can't mute an announcement by post-dating it.+ # Sources without an upstream timestamp (stock, page_watch, query) are+ # unaffected.external_monitor_max_signal_age_days: int = Field(- 7, alias="EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS"+ 7, ge=0, le=MAX_SIGNAL_AGE_DAYS, alias="EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS"+ )+ # How far ahead of our clock a published_at may be before the entry is+ # deferred (skipped for the tick, re-judged once the date passes; one+ # ``external_signal_deferred`` audit row per poll). 0 disables the+ # deferral — use it for feeds that legitimately date entries ahead+ # (scheduled-maintenance or event calendars).+ # Per-row override: ``config_json["max_future_skew_hours"]`` (0 = off+ # for that row only) — the global switch affects every feed.+ external_monitor_max_future_skew_hours: int = Field(+ 24, ge=0, le=MAX_FUTURE_SKEW_HOURS, alias="EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS")# Adapter-fetch ceiling (bytes). Caps the body we read from any single# external feed — defence against runaway sources (e.g. malformed RSSdiff --git a/packages/core/openexecutive/monitoring/pipeline.py b/packages/core/openexecutive/monitoring/pipeline.pyindex a618c559..3204e14c 100644--- a/packages/core/openexecutive/monitoring/pipeline.py+++ b/packages/core/openexecutive/monitoring/pipeline.py@@ -28,6 +28,7 @@ Design notes:from __future__ import annotationsimport logging+from dataclasses import dataclass, fieldfrom datetime import UTC, datetime, timedeltafrom pathlib import Path@@ -53,7 +54,11 @@ from openexecutive.monitoring.models import ()from openexecutive.monitoring.sources import get_source_for_kindfrom openexecutive.monitoring.sources._http import strip_url_query-from openexecutive.monitoring.sources.base import collapse_whitespace+from openexecutive.monitoring.sources.base import (+ collapse_whitespace,+ future_skew_for,+ is_implausibly_future,+)logger = logging.getLogger(__name__)@@ -79,8 +84,8 @@ def _poll_floor_minutes_for(signal_type: str) -> int:Read directly off the registered adapter's``default_poll_interval_minutes`` — no central switch statement tokeep in sync as new adapters land. Falls back to the heartbeat- interval for unknown kinds (defensive: adapter-less rows are- filtered out upstream, but a stale watchlist row shouldn't crash).+ interval for unknown kinds (a row whose adapter was removed is still+ marked polled at that cadence, see _poll_one_watchlist_item)."""src = get_source_for_kind(signal_type)if src is None:@@ -106,30 +111,58 @@ def _due_for_poll(item: WatchlistItem, now: datetime) -> bool:# --------------------------------------------------------------------- #-def _is_stale(signal: Signal, now: datetime, max_age_days: int) -> bool:- """True when the upstream publish time is older than the age gate.+def _published(signal: Signal) -> datetime | None:+ """Parsed, tz-aware ``published_at``; None when absent or unparseable.- Only signals that carry ``published_at`` can be judged; sources without- an upstream timestamp (stock, page_watch, query) always pass. An- unparseable timestamp also passes — the adapter writes ISO 8601, so a- bad value is a bug to surface downstream, not a reason to drop the- event silently. ``max_age_days <= 0`` disables the gate.+ An unparseable value is logged and treated as absent — the adapter+ writes ISO 8601, so a bad value is a bug to surface downstream, not a+ reason to drop the event silently."""- if max_age_days <= 0 or not signal.published_at:- return False+ if not signal.published_at:+ return Nonetry:published = datetime.fromisoformat(signal.published_at)except ValueError:logger.warning("monitoring.pipeline: unparseable published_at %r on %s — "- "treating as fresh", signal.published_at, signal.dedup_key,+ "treating as undated", signal.published_at, signal.dedup_key,)- return False+ return Noneif published.tzinfo is None:published = published.replace(tzinfo=UTC)+ return published+++def _is_stale(signal: Signal, now: datetime, max_age_days: int) -> bool:+ """True when the upstream publish time is older than the age gate.++ Only signals that carry ``published_at`` can be judged; sources without+ an upstream timestamp (stock, page_watch, query) always pass.+ ``max_age_days <= 0`` disables the gate.+ """+ published = _published(signal)+ if published is None or max_age_days <= 0:+ return Falsereturn now - published > timedelta(days=max_age_days)+def _is_future(signal: Signal, now: datetime, skew: timedelta) -> bool:+ """True when ``published_at`` is more than ``skew`` ahead of ``now``.++ The feed controls the date, and a future one is a clock bug or a+ forgery. Unlike a stale entry this one can become valid — the date+ will pass — so the caller must NOT record it (recording burns the+ dedup key and would mute the entry forever, handing a feed a way to+ hide an announcement by dating it a day ahead). It is skipped for the+ tick, audited as deferred, and re-judged next time. A non-positive+ ``skew`` disables the check.+ """+ if skew <= timedelta(0):+ return False+ published = _published(signal)+ return published is not None and is_implausibly_future(published, now, skew=skew)++# --------------------------------------------------------------------- ## Signal → alert promotion# --------------------------------------------------------------------- #@@ -213,7 +246,8 @@ def _signal_to_alert_event(# lines; collapse whitespace here (whatever the adapter did) so no# source — feed title, model-emitted query hit — can forge its own# ``Severity hint:`` / ``Published:`` line.- parts: list[str] = [collapse_whitespace(signal.normalized_summary)]+ summary = collapse_whitespace(signal.normalized_summary)+ parts: list[str] = [summary]if enrichment:why = collapse_whitespace(str(enrichment.get("why_it_matters") or ""))if why:@@ -237,7 +271,7 @@ def _signal_to_alert_event(return AlertEvent(source=signal.source_kind,external_id=signal.dedup_key,- subject=signal.normalized_summary[:200],+ subject=summary[:200],body="\n".join(parts)[:4000],)@@ -308,6 +342,65 @@ async def promote_signal_to_alert(# --------------------------------------------------------------------- #+def _audit_received(signal_id: int, item: WatchlistItem, signal: Signal) -> None:+ """One ``external_signal_received`` row per signal landed in the DB."""+ audit_log(+ "external_signal_received",+ f"Signal received: {collapse_whitespace(signal.normalized_summary)[:120]}",+ actor="external_monitor",+ details={+ "signal_id": signal_id,+ "watchlist_id": item.id,+ "watchlist_slug": item.slug,+ "source_kind": signal.source_kind,+ "severity_hint": signal.severity_hint.value,+ "dedup_key": signal.dedup_key,+ },+ )+++def _entries(n: int) -> str:+ return f"{n} entr{'y' if n == 1 else 'ies'}"+++def _audit_deferred(item: WatchlistItem, tally: _PollTally) -> None:+ """One ``external_signal_deferred`` row per poll that skipped entries+ dated implausibly far in the future — aggregated, since a deferred entry+ is by design not recorded and would otherwise re-log on every tick.+ Carries what the row would have shown (summary + link), so an operator+ can tell WHAT was held back without re-fetching the feed."""+ deferred = tally.deferred+ skew = tally.deferred_skew+ logger.debug(+ "monitoring.pipeline: %r deferred %s dated in the future",+ item.slug, _entries(len(deferred)),+ )+ audit_log(+ "external_signal_deferred",+ f"Deferred {_entries(len(deferred))} dated in the future on {item.slug}",+ actor="external_monitor",+ details={+ "watchlist_id": item.id,+ "watchlist_slug": item.slug,+ "source_kind": deferred[0].source_kind,+ "count": len(deferred),+ "future_skew_hours": skew.total_seconds() / 3600,+ "race_tick": tally.race_tick,+ # Enough to find and read them; the same fields the received /+ # promoted rows already carry.+ "entries": [+ {+ "dedup_key": s.dedup_key,+ "published_at": s.published_at,+ "summary": collapse_whitespace(s.normalized_summary)[:120],+ "provenance_url": _safe_url(s.provenance_url),+ }+ for s in deferred[:10]+ ],+ },+ )++def _audit_suppressed(signal_id: int,item: WatchlistItem,@@ -320,7 +413,7 @@ def _audit_suppressed(audit_log("external_signal_suppressed",f"Signal {signal_id} suppressed ({outcome}): "- f"{signal.normalized_summary[:120]}",+ f"{collapse_whitespace(signal.normalized_summary)[:120]}",actor="external_monitor",details={"signal_id": signal_id,@@ -334,23 +427,176 @@ def _audit_suppressed()+# During a lost race, how far past our own clock an entry may be dated and+# still be judged now (ordinary publisher drift); anything later is deferred+# rather than recorded, since nothing we fetched can honestly be dated later.+# Independent of MAX_FUTURE_SKEW / EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS,+# which bound the normal (non-race) path far more leniently.+_RACE_CLOCK_TOLERANCE = timedelta(minutes=5)+++@dataclass(frozen=True)+class _LostRace:+ """How a scan that lost the baseline claim treats the row's entries.++ Another scan baselined the row (stamp ``cutoff``) while our fetch was+ in flight, so we hold (mostly) the same back-catalogue — and every+ entry the winner held is already recorded, so it dedups. Entries+ dated at or before the cutoff are baseline; undated or unparseable+ ones too — the feed controls ``published_at`` and we can't tell old+ from new without it. Entries dated after the cutoff are news, except+ that the caller defers (never records) anything dated more than+ ``_RACE_CLOCK_TOLERANCE`` past its own clock, so a post-dated entry+ that only our fetch holds is re-judged later instead of burned.+ """++ cutoff: str++ def covers(self, signal: Signal) -> bool:+ published = _published(signal)+ if published is None:+ # Undated, or a bad date. Fail closed — it must not become an+ # alert on a race tick.+ return True+ try:+ cutoff = datetime.fromisoformat(self.cutoff)+ except ValueError:+ return True+ if cutoff.tzinfo is None:+ cutoff = cutoff.replace(tzinfo=UTC)+ return published <= cutoff+++def _lost_race(+ item: WatchlistItem, fetched_at: datetime, *, db_path: Path | None = None,+) -> _LostRace:+ """Build the lost-race policy from the winner's stamp."""+ assert item.id is not None+ fresh = store.get_watchlist_item(item.id, db_path=db_path)+ cutoff = fresh.baselined_at if fresh is not None and fresh.baselined_at else None+ if cutoff is None:+ # Lost the claim yet no stamp is visible: the row was deleted, or a+ # concurrent initialize_db backfill stamped and rolled back. Fall+ # back to a cutoff past our own fetch (plus the drift tolerance),+ # so everything we fetched is covered: this tick is recorded as+ # baseline, nothing promoted — say so.+ logger.warning(+ "monitoring.pipeline: %r lost the baseline claim but no stamp is "+ "visible — treating this tick's entries as baseline", item.slug,+ )+ cutoff = (fetched_at + _RACE_CLOCK_TOLERANCE).isoformat()+ return _LostRace(cutoff)+++def _record_baseline(+ item: WatchlistItem,+ entries: list[Signal],+ fetched_at: datetime,+ *,+ db_path: Path | None = None,+) -> int | None:+ """Try to baseline a seeding row: claim the stamp and record every+ fetched entry — trigger misses included — as ``suppressed_baseline``+ in ONE transaction.++ Returns how many rows were recorded, or ``None`` when another scan won+ the claim first (nothing written). The claim is a compare-and-swap on+ ``baselined_at IS NULL``: two overlapping scans (heartbeat + client+ rotation, or a slow tick) can both load the row un-baselined, and+ exactly one of them may record the back-catalogue. The write is+ all-or-nothing, so a failure part-way leaves the stamp NULL and the+ next tick baselines the whole feed again rather than replaying an+ unrecorded tail as news.+ """+ assert item.id is not None+ recorded = store.record_baseline(item.id, fetched_at, entries, db_path=db_path)+ if recorded is None:+ return None+ for signal_id, signal in recorded:+ _audit_received(signal_id, item, signal)+ _audit_suppressed(signal_id, item, signal, OUTCOME_SUPPRESSED_BASELINE)+ logger.info(+ "monitoring.pipeline: baselined %r (%s) — %s recorded, none promoted",+ item.slug, item.signal_type, _entries(len(recorded)),+ )+ return len(recorded)++@dataclass+class _ScanBudget:+ """The scan-wide ``max_signals_per_scan`` allowance.++ Mutable and decremented in place by ``_poll_one_watchlist_item`` as it+ admits signals to the alert pipeline, so the accounting survives a row+ that raises part-way (after promoting) — a returned count would be+ lost with the exception and the cap would fail open.+ """++ remaining: int++ @property+ def exhausted(self) -> bool:+ return self.remaining <= 0++ def admit(self) -> None:+ self.remaining -= 1+++@dataclass+class _PollTally:+ """Per-row outcome of ``_poll_one_watchlist_item``, filled in place.++ ``written`` is the number of rows landed in ``external_signals`` this+ tick (NOT the number alerted — some get suppressed by floor / dry_run+ / dup). Mutable so the caller still sees a true count when the poll+ raises part-way (the audit row must not claim "0 signals" for a row+ that inserted four and then died). ``failed`` is set for a poll that+ could not run at all (no adapter, adapter crash); an exception that+ escapes the poll is the other failure signal and is handled by the+ caller.+ """++ written: int = 0+ failed: bool = False+ # Entries skipped this tick for being dated implausibly far ahead. Kept+ # here (not as a loop local) so the caller can audit them even when+ # the poll raises part-way — the audit row is their only trace.+ deferred: list[Signal] = field(default_factory=list)+ deferred_skew: timedelta = timedelta(0)+ race_tick: bool = False+++def _suppress(+ signal_id: int,+ item: WatchlistItem,+ signal: Signal,+ outcome: str,+ *,+ db_path: Path | None = None,+) -> None:+ """Record a suppression outcome on the row and emit its paired audit+ event. Every branch of the cascade goes through here so the two stay+ in lockstep."""+ store.mark_signal_processed(signal_id, outcome, db_path=db_path)+ _audit_suppressed(signal_id, item, signal, outcome)++async def _poll_one_watchlist_item(item: WatchlistItem,now: datetime,*,- cap_remaining: int,+ budget: _ScanBudget,+ tally: _PollTally,company_ctx: str = "",db_path: Path | None = None,-) -> tuple[int, int]:+) -> None:"""Poll one watchlist row, persist signals, promote where appropriate.- Returns ``(written, admitted)``: rows landed in ``external_signals``- on this tick (NOT the number alerted — some get suppressed by floor /- dry_run / dup), and how many of those count against the scan-wide- ``max_signals_per_scan`` cap. Baseline rows are written but not- admitted: they never reach the alert pipeline the cap protects, and- counting them would truncate the baseline mid-feed and let the tail- replay as news on the next tick.+ Rows landed in ``external_signals`` are counted on ``tally`` and each+ signal admitted to the cascade is charged to ``budget`` — both in+ place, so the accounting survives a poll that raises part-way.+ Baseline rows are written but never charged: they don't reach the+ alert pipeline the cap protects, and counting them would truncate the+ baseline mid-feed and let the tail replay as news on the next tick."""src = get_source_for_kind(item.signal_type)if src is None:@@ -359,18 +605,28 @@ async def _poll_one_watchlist_item("(watchlist slug=%r) — skipping",item.signal_type, item.slug,)- return 0, 0+ tally.failed = True+ # Still consume the row's cadence slot, or it would re-fire (and+ # emit a failed audit row) on every heartbeat until fixed.+ if item.id is not None:+ store.mark_polled(item.id, now, db_path=db_path)+ returnmax_age_days = get_settings().external_monitor_max_signal_age_daystry:emitted = await src.poll(item, db_path=db_path)+ fetched_at = datetime.now(UTC)+ # Inside the guard so a settings/config surprise here is handled+ # like an adapter crash (logged, row still marked polled).+ future_skew = future_skew_for(item)except Exception:logger.exception("monitoring.pipeline: adapter %s crashed for watchlist %r",item.signal_type, item.slug,)- return 0, 0+ tally.failed = True+ returnfinally:# Always mark polled even on adapter crash so a broken source# doesn't get hammered on every tick.@@ -380,42 +636,80 @@ async def _poll_one_watchlist_item(if not emitted:# Failed or empty fetch: nothing to baseline, and ``baselined_at``# stays NULL so the next successful poll IS the baseline.- return 0, 0+ return- # Feed-listing sources (Source.seed_on_first_poll) are baselined on- # the first poll that returns entries: everything already in the feed- # is recorded as seen, nothing is promoted. Keyed on ``baselined_at``- # rather than ``last_polled_at`` so a transient fetch failure on the- # first tick can't forfeit the baseline. ``item`` is the pre-poll- # snapshot, so the in-memory value is unaffected by ``mark_polled``.- is_baseline_poll = (- getattr(src, "seed_on_first_poll", False) and item.baselined_at is None- )+ capped = [+ raw.model_copy(update={"severity_hint": _cap_to_ceiling(raw.severity_hint, item)})+ for raw in emitted+ ]- written = 0- admitted = 0- for raw_signal in emitted:- if not is_baseline_poll and cap_remaining <= 0:- logger.info(- "monitoring.pipeline: max_signals_per_scan cap hit at "- "watchlist %r (signal_type=%s) — dropping remaining %d",- item.slug, item.signal_type, len(emitted) - written,- )- break-- # Apply per-item trigger filter (default: True).+ # Feed-listing sources (Source.seed_on_first_poll) are baselined on+ # the first poll that returns entries: EVERYTHING in the feed —+ # including entries that miss the trigger filter — is recorded as+ # seen, nothing is promoted. Recording the misses too means the first+ # entry that ever matches a keyword is news, not "back-catalogue",+ # and widening a trigger later cannot resurface what was already in+ # the feed. Keyed on ``baselined_at`` rather than ``last_polled_at``+ # so a transient fetch failure on the first tick can't forfeit the+ # baseline. ``item`` is the pre-poll snapshot, so the in-memory value+ # is unaffected by ``mark_polled``. Baseline rows never reach the+ # alert pipeline, so they are never charged to the scan budget.+ lost_race: _LostRace | None = None+ if (+ getattr(src, "seed_on_first_poll", False)+ and item.id is not None+ and item.baselined_at is None+ ):+ recorded = _record_baseline(item, capped, fetched_at, db_path=db_path)+ if recorded is not None:+ tally.written = recorded+ return+ lost_race = _lost_race(item, fetched_at, db_path=db_path)++ # Apply the per-item trigger filter (default: True). Entries that miss+ # it are dropped — unless this scan lost the baseline race, in which+ # case a miss the winner didn't hold is still recorded as baseline so a+ # later, wider trigger can't resurface it (same guarantee the winner+ # gives by recording misses).+ def _matches(signal: Signal) -> bool:try:- if not src.matches_trigger(raw_signal, item):- continue+ return bool(src.matches_trigger(signal, item))except Exception:logger.exception("monitoring.pipeline: matches_trigger crashed on %r — ""treating as no-match", item.slug)+ return False++ # On a race tick the deferral bound is the tight drift tolerance, not+ # the lenient configured skew: nothing we fetched can honestly be dated+ # much later than our own clock, and deferring (rather than recording)+ # is what keeps a post-dated entry from being burned for good.+ skew = _RACE_CLOCK_TOLERANCE if lost_race is not None else future_skew+ tally.deferred_skew = skew+ tally.race_tick = lost_race is not None+ cap_logged = False+ for signal in capped:+ baseline_entry = lost_race is not None and lost_race.covers(signal)+ # The three guards below apply only to entries that are NOT baseline;+ # a baseline entry is always recorded (and never charged).+ if not baseline_entry and not _matches(signal):+ continue+ if not baseline_entry and _is_future(signal, fetched_at, skew):+ tally.deferred.append(signal) # not recorded: see _is_future+ continue+ if not baseline_entry and budget.exhausted:+ # Skip (don't break): baseline-covered entries later in the+ # list must still be recorded, or they'd replay next tick.+ if not cap_logged:+ logger.info(+ "monitoring.pipeline: max_signals_per_scan cap hit at "+ "watchlist %r (signal_type=%s) — dropping the remaining "+ "non-baseline entries this tick",+ item.slug, item.signal_type,+ )+ cap_logged = Truecontinue-- capped = _cap_to_ceiling(raw_signal.severity_hint, item)- signal = raw_signal.model_copy(update={"severity_hint": capped})signal_id = store.insert_signal(signal, db_path=db_path)if signal_id is None:@@ -424,23 +718,10 @@ async def _poll_one_watchlist_item(# vendor status page returns the same 25 entries every poll.continue- written += 1- if not is_baseline_poll:- cap_remaining -= 1- admitted += 1- audit_log(- "external_signal_received",- f"Signal received: {signal.normalized_summary[:120]}",- actor="external_monitor",- details={- "signal_id": signal_id,- "watchlist_id": item.id,- "watchlist_slug": item.slug,- "source_kind": signal.source_kind,- "severity_hint": signal.severity_hint.value,- "dedup_key": signal.dedup_key,- },- )+ tally.written += 1+ if not baseline_entry:+ budget.admit()+ _audit_received(signal_id, item, signal)# Suppression cascade — order matters. Each branch emits a# paired external_signal_suppressed audit event so a reader can@@ -450,31 +731,19 @@ async def _poll_one_watchlist_item(# Freshness gates come first: a baseline entry or a stale one is# "already happened" news regardless of mode / floor, and neither# should spend an enrichment call.- if is_baseline_poll:- store.mark_signal_processed(- signal_id, OUTCOME_SUPPRESSED_BASELINE, db_path=db_path- )- _audit_suppressed(+ if baseline_entry:+ _suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_BASELINE,+ db_path=db_path,)continueif _is_stale(signal, now, max_age_days):- store.mark_signal_processed(- signal_id, OUTCOME_SUPPRESSED_STALE, db_path=db_path- )- _audit_suppressed(- signal_id, item, signal, OUTCOME_SUPPRESSED_STALE,- )+ _suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_STALE, db_path=db_path)continueif item.mode == MODE_DRY_RUN:- store.mark_signal_processed(- signal_id, OUTCOME_SUPPRESSED_DRY_RUN, db_path=db_path- )- _audit_suppressed(- signal_id, item, signal, OUTCOME_SUPPRESSED_DRY_RUN,- )+ _suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_DRY_RUN, db_path=db_path)continue# Standing-query signals arrive flat-LOW: the `query` adapter can't@@ -507,12 +776,7 @@ async def _poll_one_watchlist_item(sev_rank = SEVERITY_RANK[signal.severity_hint.value]floor_rank = SEVERITY_RANK[item.severity_floor.value]if sev_rank < floor_rank:- store.mark_signal_processed(- signal_id, OUTCOME_SUPPRESSED_BELOW_FLOOR, db_path=db_path- )- _audit_suppressed(- signal_id, item, signal, OUTCOME_SUPPRESSED_BELOW_FLOOR,- )+ _suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_BELOW_FLOOR, db_path=db_path)continue# Capture-time enrichment for pre-graded sources runs here — AFTER the@@ -534,15 +798,6 @@ async def _poll_one_watchlist_item(enrichment=enrichment_payload,)- if is_baseline_poll and item.id is not None:- store.mark_baselined(item.id, now, db_path=db_path)- logger.info(- "monitoring.pipeline: baselined %r (%s) — %d existing entr%s "- "recorded, none promoted",- item.slug, item.signal_type, written, "y" if written == 1 else "ies",- )-- return written, admittedasync def _maybe_enrich(@@ -585,12 +840,7 @@ async def _maybe_enrich()if result.relevance_score < settings.external_monitor_enrichment_min_relevance:- store.mark_signal_processed(- signal_id, OUTCOME_SUPPRESSED_LOW_RELEVANCE, db_path=db_path- )- _audit_suppressed(- signal_id, item, signal, OUTCOME_SUPPRESSED_LOW_RELEVANCE,- )+ _suppress(signal_id, item, signal, OUTCOME_SUPPRESSED_LOW_RELEVANCE, db_path=db_path)return False, payloadreturn True, payload@@ -627,10 +877,10 @@ async def run_external_monitor_scan()total_written = 0- cap_remaining = cap_total+ budget = _ScanBudget(remaining=cap_total)poll_start = datetime.now(UTC)for item in items:- if cap_remaining <= 0:+ if budget.exhausted:break# Billed standing-query adapter has its own kill switch — skip those# rows entirely (no poll, no mark_polled) when disabled.@@ -642,35 +892,62 @@ async def run_external_monitor_scan(if not _due_for_poll(item, now):continueper_source_start = datetime.now(UTC)- written, admitted = await _poll_one_watchlist_item(- item, now, cap_remaining=cap_remaining,- company_ctx=company_ctx, db_path=db_path,- )- cap_remaining -= admitted- total_written += written- duration_ms = int(- (datetime.now(UTC) - per_source_start).total_seconds() * 1000- )- audit_log(- "external_monitor_poll",- f"Polled {item.slug} ({item.signal_type}) — {written} signal(s)",- actor="external_monitor",- details={- "watchlist_id": item.id,- "watchlist_slug": item.slug,- "signal_type": item.signal_type,- "signals_emitted": written,- "duration_ms": duration_ms,- },- )+ tally = _PollTally()+ failed = False+ try:+ await _poll_one_watchlist_item(+ item, now, budget=budget, tally=tally,+ company_ctx=company_ctx, db_path=db_path,+ )+ except Exception:+ # A store failure on one row (e.g. the shared DB locked during+ # its baseline transaction) must not skip every row after it;+ # the poll is retried on the next tick. Anything it admitted+ # before failing is already charged to the budget.+ failed = True+ logger.exception(+ "monitoring.pipeline: polling %r (%s) failed — continuing "+ "with the remaining watchlist rows", item.slug, item.signal_type,+ )+ except BaseException:+ # Cancellation (shutdown mid-scan) must propagate, but the+ # audit row below must not read as a clean, empty poll.+ failed = True+ raise+ finally:+ # The poll audit row is emitted even for a failed row, so the+ # audit log stands on its own as the record of what the+ # monitor did this tick — including how many rows landed+ # before a failure.+ failed = failed or tally.failed+ total_written += tally.written+ if tally.deferred:+ _audit_deferred(item, tally)+ duration_ms = int(+ (datetime.now(UTC) - per_source_start).total_seconds() * 1000+ )+ status = "FAILED" if failed else f"{tally.written} signal(s)"+ audit_log(+ "external_monitor_poll",+ f"Polled {item.slug} ({item.signal_type}) — {status}",+ actor="external_monitor",+ details={+ "watchlist_id": item.id,+ "watchlist_slug": item.slug,+ "signal_type": item.signal_type,+ "signals_emitted": tally.written,+ "duration_ms": duration_ms,+ "failed": failed,+ },+ )scan_duration_ms = int((datetime.now(UTC) - poll_start).total_seconds() * 1000)logger.info("monitoring.pipeline: scan complete — %d signal(s) across %d watchlist "- "row(s) in %dms (cap_total=%d, cap_remaining=%d)",- total_written, len(items), scan_duration_ms, cap_total, cap_remaining,+ "row(s) in %dms (cap_total=%d, budget_remaining=%d)",+ total_written, len(items), scan_duration_ms, cap_total, budget.remaining,)return total_writtendiff --git a/packages/core/openexecutive/monitoring/sources/base.py b/packages/core/openexecutive/monitoring/sources/base.pyindex cfae9081..5f5101c3 100644--- a/packages/core/openexecutive/monitoring/sources/base.py+++ b/packages/core/openexecutive/monitoring/sources/base.py@@ -23,14 +23,18 @@ Invariants every adapter MUST uphold:from __future__ import annotationsimport calendar+import logging+import mathfrom collections.abc import Mappingfrom datetime import UTC, datetime, timedelta-from email.utils import parsedate_to_datetimefrom pathlib import Pathfrom typing import Any, Protocol+from openexecutive.config import MAX_FUTURE_SKEW_HOURS, get_settingsfrom openexecutive.monitoring.models import Signal, WatchlistItem+logger = logging.getLogger(__name__)+class Source(Protocol):"""Adapter contract — see module docstring for invariants."""@@ -78,34 +82,54 @@ __all__ = ["Signal", "Source"]# A feed claiming an item was published more than this far in the future is-# lying (or has a broken clock). Such a value would never age out and would-# render as a huge "ago", so it's treated as no timestamp at all.-_MAX_FUTURE_SKEW = timedelta(days=1)---def _as_published_at(dt: datetime, now: datetime | None) -> str | None:+# lying (or has a broken clock). The parser prefers another date key when+# one is plausible; when every key is implausible the value is still kept+# — the pipeline then defers the entry (see ``monitoring.pipeline._is_future``)+# rather than letting an undated entry bypass the age gate.+MAX_FUTURE_SKEW = timedelta(days=1)+++def is_implausibly_future(+ dt: datetime, now: datetime | None = None, *, skew: timedelta = MAX_FUTURE_SKEW,+) -> bool:+ """True when ``dt`` is more than ``skew`` ahead of ``now``.++ Both the adapters (choosing between an entry's date keys) and the+ pipeline (deciding to defer) pass the configured+ ``EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS``; the constant is only the+ fallback default.+ """if dt.tzinfo is None:dt = dt.replace(tzinfo=UTC)- if dt > (now or datetime.now(UTC)) + _MAX_FUTURE_SKEW:- return None- return dt.isoformat()+ return dt > (now or datetime.now(UTC)) + skewdef feed_entry_published_at(- entry: Mapping[str, Any], *, now: datetime | None = None+ entry: Mapping[str, Any],+ *,+ now: datetime | None = None,+ skew: timedelta = MAX_FUTURE_SKEW,) -> str | None:"""ISO 8601 UTC publish timestamp of a feedparser entry, or None.feedparser normalises ``<pubDate>`` / ``<published>`` / ``<updated>``into ``*_parsed`` ``time.struct_time`` values already converted toUTC; we prefer ``published`` (when the item first appeared) over- ``updated`` (last edit). Entries with no parseable date — or a date- past ``_MAX_FUTURE_SKEW`` — return None: the pipeline's age gate then- can't judge them, and only the first-poll baseline protects against- replaying them as new.+ ``updated`` (last edit), and a plausible key over one dated past+ ``skew`` (a non-positive ``skew`` turns that preference off, for feeds+ that legitimately date entries ahead). Entries with no parseable date+ return None: the+ pipeline's age gate then can't judge them, and only the first-poll+ baseline protects against replaying them as new. An entry whose every+ key is implausibly future keeps that value, so the pipeline can defer+ it instead of waving an undated entry through."""+ fallback: str | None = None+ # Only read keys the entry actually has: feedparser aliases a missing+ # ``updated_parsed`` to ``published_parsed`` with a DeprecationWarning.+ present = set(entry.keys())for key in ("published_parsed", "updated_parsed"):- st = entry.get(key)+ st = entry.get(key) if key in present else Noneif not st:continuetry:@@ -114,30 +138,57 @@ def feed_entry_published_at(dt = datetime.fromtimestamp(calendar.timegm(st), tz=UTC)except (TypeError, ValueError, OverflowError, OSError):continue- return _as_published_at(dt, now)- return None+ if skew <= timedelta(0) or not is_implausibly_future(dt, now, skew=skew):+ return dt.isoformat()+ fallback = fallback or dt.isoformat()+ return fallback+++def _valid_skew_hours(raw: Any) -> float | None:+ """``raw`` as hours if it is a finite number in 0..MAX_FUTURE_SKEW_HOURS,+ else None. Booleans are rejected (``False`` is not "off"; JSON ``true``+ is not one hour), as is anything unparseable or non-finite — JSON can+ carry ``1e999`` → inf."""+ if isinstance(raw, bool):+ return None+ try:+ hours = float(raw)+ except (TypeError, ValueError, OverflowError):+ return None+ if not math.isfinite(hours) or not 0 <= hours <= MAX_FUTURE_SKEW_HOURS:+ return None+ return hours-def iso_published_at(value: str, *, now: datetime | None = None) -> str | None:- """Parse a raw ``<updated>`` (ISO 8601) or ``<pubDate>`` (RFC 822) string.+def future_skew_for(item: WatchlistItem) -> timedelta:+ """The row's future-date tolerance: a valid ``config_json``+ ``["max_future_skew_hours"]`` (0 .. MAX_FUTURE_SKEW_HOURS; 0 = off for+ this row), else ``EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS``.- For adapters that parse XML by hand (vendor_status) rather than via- feedparser. Same contract as ``feed_entry_published_at``: ISO 8601 UTC- or None when the value is empty, unparseable, or implausibly future.+ An out-of-range or malformed override is ignored with a warning rather+ than coerced: a negative typo must not silently switch the gate off,+ and a huge one must not overflow the date arithmetic and wedge the row.+ The global value goes through the same check (the env var is+ bounds-checked, but a Settings copy is not); if it is invalid the+ ceiling is used."""- value = (value or "").strip()- if not value:- return None- try:- dt = datetime.fromisoformat(value)- except ValueError:- try:- dt = parsedate_to_datetime(value)- except (TypeError, ValueError, IndexError, OverflowError, OSError):- # OverflowError: absurd numeric zone offsets; a poisoned entry- # must not take the whole feed's poll down with it.- return None- return _as_published_at(dt, now)+ global_hours = _valid_skew_hours(+ get_settings().external_monitor_max_future_skew_hours+ )+ if global_hours is None:+ global_hours = float(MAX_FUTURE_SKEW_HOURS)+ raw = item.config_json.get("max_future_skew_hours")+ if raw is None:+ return timedelta(hours=global_hours)+ hours = _valid_skew_hours(raw)+ if hours is None:+ logger.warning(+ "monitoring: watchlist %r has an invalid max_future_skew_hours %s "+ "(want 0..%d) — using the global setting",+ item.slug, repr(raw)[:64], MAX_FUTURE_SKEW_HOURS,+ )+ return timedelta(hours=global_hours)+ return timedelta(hours=hours)def collapse_whitespace(text: str) -> str:diff --git a/packages/core/openexecutive/monitoring/sources/edgar.py b/packages/core/openexecutive/monitoring/sources/edgar.pyindex 9e87f0fd..08f0241d 100644--- a/packages/core/openexecutive/monitoring/sources/edgar.py+++ b/packages/core/openexecutive/monitoring/sources/edgar.py@@ -36,7 +36,7 @@ from __future__ import annotationsimport hashlibimport loggingimport re-from datetime import UTC, datetime+from datetime import UTC, datetime, timedeltafrom pathlib import Pathfrom urllib.parse import quote@@ -51,7 +51,10 @@ from openexecutive.monitoring.sources._http import (fetch_bounded,validate_target_url,)-from openexecutive.monitoring.sources.base import feed_entry_published_at+from openexecutive.monitoring.sources.base import (+ feed_entry_published_at,+ future_skew_for,+)logger = logging.getLogger(__name__)@@ -135,6 +138,7 @@ class EdgarSource:allowed = _allowed_forms(item.trigger_json)label = item.config_json.get("label") or ident.upper()+ skew = future_skew_for(item)signals: list[Signal] = []# Filter first, then cap the OUTPUT — capping the raw entry list before# filtering would let a run of excluded forms (e.g. many Form 4s) at the@@ -142,7 +146,7 @@ class EdgarSource:for entry in parsed.entries:if len(signals) >= _MAX_FILINGS_PER_POLL:break- signal = _entry_to_signal(entry, item, label, allowed)+ signal = _entry_to_signal(entry, item, label, allowed, skew)if signal is not None:signals.append(signal)return signals@@ -204,7 +208,8 @@ def _accession(entry: dict) -> str:def _entry_to_signal(- entry: dict, item: WatchlistItem, label: str, allowed: frozenset[str]+ entry: dict, item: WatchlistItem, label: str, allowed: frozenset[str],+ skew: timedelta,) -> Signal | None:form = _entry_form(entry)if not _form_matches(form, allowed):@@ -232,7 +237,7 @@ def _entry_to_signal(source_kind=SOURCE_KIND_EDGAR,source_external_id=accession[:500],captured_at=datetime.now(UTC).isoformat(),- published_at=feed_entry_published_at(entry),+ published_at=feed_entry_published_at(entry, skew=skew),normalized_summary=summary[:500],raw_payload={"label": label,diff --git a/packages/core/openexecutive/monitoring/sources/rss.py b/packages/core/openexecutive/monitoring/sources/rss.pyindex 766de3c8..88bcf5e4 100644--- a/packages/core/openexecutive/monitoring/sources/rss.py+++ b/packages/core/openexecutive/monitoring/sources/rss.py@@ -10,7 +10,10 @@ Watchlist row shape:- ``target``: the full feed URL- ``config_json``: optional ``{"feed_label": "Acme Changelog"}`` — usedas a prefix in the normalized summary. Falls back to a host-derived- label when missing.+ label when missing. Optional ``"max_future_skew_hours"`` (0 .. 8760)+ overrides ``EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS`` for this feed (0 =+ off, for calendars / maintenance windows that legitimately date entries+ ahead); an out-of-range value is ignored with a warning.- ``trigger_json``: optional ``{"keywords": ["pricing", "launch"]}``— when present, only entries whose title OR summary contain at leastone keyword surface. Otherwise every new entry surfaces.@@ -21,7 +24,9 @@ from *when we first saw it* (``captured_at``). Two guards follow fromthat (issue #80 — a January article was surfacing as September news):the first poll of a row records the feed's existing entries as abaseline without promoting them, and entries older than-``EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS`` are recorded but never promoted.+``EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS`` are recorded but never promoted+(entries dated implausibly far in the future are deferred until the date+passes).Severity hint defaults to ``LOW`` — RSS is by far the noisiest source,so a low default lets the watchlist's ``severity_floor`` filter@@ -59,6 +64,7 @@ from openexecutive.monitoring.sources._http import (from openexecutive.monitoring.sources.base import (collapse_whitespace,feed_entry_published_at,+ future_skew_for,)logger = logging.getLogger(__name__)@@ -126,6 +132,7 @@ class RssSource:item.config_json.get("feed_label")or _host_label(item.target))+ skew = future_skew_for(item)signals: list[Signal] = []for entry in parsed.entries[:_MAX_ENTRIES_PER_FEED]:entry_id = _entry_id(entry)@@ -140,7 +147,7 @@ class RssSource:source_kind=self.kind,source_external_id=entry_id[:500],captured_at=datetime.now(UTC).isoformat(),- published_at=feed_entry_published_at(entry),+ published_at=feed_entry_published_at(entry, skew=skew),normalized_summary=summary[:500],raw_payload={"feed_label": feed_label,diff --git a/packages/core/openexecutive/monitoring/store.py b/packages/core/openexecutive/monitoring/store.pyindex d0add6e4..a67c0403 100644--- a/packages/core/openexecutive/monitoring/store.py+++ b/packages/core/openexecutive/monitoring/store.py@@ -33,6 +33,7 @@ from openexecutive.alerts.models import AlertSeverityfrom openexecutive.memory.episodic import DB_PATH, _resolve_db_pathfrom openexecutive.monitoring.models import (MODE_ACTIVE,+ OUTCOME_SUPPRESSED_BASELINE,Signal,WatchlistItem,is_valid_mode,@@ -140,21 +141,33 @@ def initialize_db(db_path: Path | None = None) -> None:# crash between the two would leave the column present (so the# backfill never re-runs) but every already-polled row un-# baselined — and the next tick would swallow their new entries.- conn.execute("BEGIN")- try:- if _ensure_column(conn, "watchlist", "baselined_at", "TEXT"):- # Rows polled before this column existed have already- # surfaced whatever their feed held; treat them as- # baselined so the upgrade tick doesn't swallow genuinely- # new entries.- conn.execute(- "UPDATE watchlist SET baselined_at = last_polled_at "- "WHERE baselined_at IS NULL AND last_polled_at IS NOT NULL"- )- conn.execute("COMMIT")- except Exception:- conn.execute("ROLLBACK")- raise+ # Only open a write transaction when there is actually something to+ # migrate: the steady-state boot must not contend for the write lock+ # on the shared episodic_memory.db (a 5s busy timeout would abort+ # startup). IMMEDIATE takes the lock up front, so the PRAGMA re-check+ # inside _ensure_column sees a settled schema even if another+ # initializer raced us to the ALTER.+ if not _has_column(conn, "watchlist", "baselined_at"):+ _migrate_baselined_at(conn)+ # Add future migrations BELOW, not inside the guard above.+++def _migrate_baselined_at(conn: sqlite3.Connection) -> None:+ """Add ``watchlist.baselined_at`` and backfill it, atomically."""+ conn.execute("BEGIN IMMEDIATE")+ try:+ if _ensure_column(conn, "watchlist", "baselined_at", "TEXT"):+ # Rows polled before this column existed have already surfaced+ # whatever their feed held; treat them as baselined so the+ # upgrade tick doesn't swallow genuinely new entries.+ conn.execute(+ "UPDATE watchlist SET baselined_at = last_polled_at "+ "WHERE baselined_at IS NULL AND last_polled_at IS NOT NULL"+ )+ conn.execute("COMMIT")+ except Exception:+ conn.execute("ROLLBACK")+ raisedef _ensure_column(@@ -168,13 +181,25 @@ def _ensure_column(Table + column names are internal constants (never user input), so thef-string interpolation here carries no injection surface."""- cols = {row["name"] for row in conn.execute(f"PRAGMA table_info({table})")}- if column in cols:+ if _has_column(conn, table, column):return False- conn.execute(f"ALTER TABLE {table} ADD COLUMN {column} {decl}")+ try:+ conn.execute(f"ALTER TABLE {table} ADD COLUMN {column} {decl}")+ except sqlite3.OperationalError as exc:+ # Another initializer (a CLI run, an overlapping deploy) added the+ # column between our PRAGMA check and the ALTER. That's the same+ # end state as "already present" — and it must not abort boot.+ if "duplicate column" in str(exc).lower():+ return False+ raisereturn True+def _has_column(conn: sqlite3.Connection, table: str, column: str) -> bool:+ cols = {row["name"] for row in conn.execute(f"PRAGMA table_info({table})")}+ return column in cols++# --------------------------------------------------------------------- ## Watchlist CRUD# --------------------------------------------------------------------- #@@ -310,23 +335,6 @@ def mark_polled()-def mark_baselined(- item_id: int,- at: datetime,- db_path: Path | None = None,-) -> None:- """Record that a seeding source's back-catalogue has been captured.-- Called once, after the first poll that returned entries; from then on- the pipeline promotes new entries instead of baselining them.- """- with _get_conn(db_path) as conn:- conn.execute(- "UPDATE watchlist SET baselined_at = ? WHERE id = ?",- (at.isoformat(), item_id),- )--def mark_fired(item_id: int,at: datetime,@@ -437,28 +445,80 @@ def insert_signal(signal: Signal, db_path: Path | None = None) -> int | None:(see ``alerts/store.py:insert_alert``)."""with _get_conn(db_path) as conn:- cursor = conn.execute(- "INSERT OR IGNORE INTO external_signals "- "(watchlist_id, source_kind, source_external_id, captured_at, "- "published_at, normalized_summary, raw_payload_json, "- "provenance_url, severity_hint, dedup_key) "- "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",- (- signal.watchlist_id,- signal.source_kind,- signal.source_external_id,- signal.captured_at,- signal.published_at,- signal.normalized_summary,- json.dumps(signal.raw_payload),- signal.provenance_url,- signal.severity_hint.value,- signal.dedup_key,- ),- )- if cursor.rowcount == 0:- return None- return int(cursor.lastrowid or 0)+ return _insert_signal_row(conn, signal)+++def _insert_signal_row(conn: sqlite3.Connection, signal: Signal) -> int | None:+ cursor = conn.execute(+ "INSERT OR IGNORE INTO external_signals "+ "(watchlist_id, source_kind, source_external_id, captured_at, "+ "published_at, normalized_summary, raw_payload_json, "+ "provenance_url, severity_hint, dedup_key) "+ "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",+ (+ signal.watchlist_id,+ signal.source_kind,+ signal.source_external_id,+ signal.captured_at,+ signal.published_at,+ signal.normalized_summary,+ json.dumps(signal.raw_payload),+ signal.provenance_url,+ signal.severity_hint.value,+ signal.dedup_key,+ ),+ )+ if cursor.rowcount == 0:+ return None+ return int(cursor.lastrowid or 0)+++def record_baseline(+ item_id: int,+ at: datetime,+ signals: list[Signal],+ db_path: Path | None = None,+) -> list[tuple[int, Signal]] | None:+ """Atomically claim a seeding row's baseline and record its back-catalogue.++ One ``BEGIN IMMEDIATE`` transaction: a compare-and-swap on+ ``baselined_at IS NULL`` stamps the row, then every signal is inserted+ (``INSERT OR IGNORE`` on ``dedup_key``) and marked+ ``suppressed_baseline``. Returns the ``(signal_id, signal)`` pairs that+ were newly recorded, or ``None`` when another scan already holds the+ stamp — in which case nothing was written and the caller takes the+ lost-race path. All-or-nothing: a failure part-way rolls the stamp+ back with the rows, so the next tick baselines the whole feed again+ instead of replaying an unrecorded tail as news.+ """+ with _get_conn(db_path) as conn:+ conn.execute("BEGIN IMMEDIATE")+ try:+ claimed = conn.execute(+ "UPDATE watchlist SET baselined_at = ? "+ "WHERE id = ? AND baselined_at IS NULL",+ (at.isoformat(), item_id),+ )+ if claimed.rowcount == 0:+ conn.execute("ROLLBACK")+ return None+ recorded: list[tuple[int, Signal]] = []+ processed_at = datetime.now(UTC).isoformat()+ for signal in signals:+ signal_id = _insert_signal_row(conn, signal)+ if signal_id is None:+ continue+ conn.execute(+ "UPDATE external_signals "+ "SET processed_at = ?, processed_outcome = ? WHERE id = ?",+ (processed_at, OUTCOME_SUPPRESSED_BASELINE, signal_id),+ )+ recorded.append((signal_id, signal))+ conn.execute("COMMIT")+ return recorded+ except Exception:+ conn.execute("ROLLBACK")+ raisedef mark_signal_processed(diff --git a/packages/core/tests/unit/test_monitoring_pipeline.py b/packages/core/tests/unit/test_monitoring_pipeline.pyindex af0669b8..fec1b0bb 100644--- a/packages/core/tests/unit/test_monitoring_pipeline.py+++ b/packages/core/tests/unit/test_monitoring_pipeline.py@@ -10,6 +10,7 @@ pipeline orchestration code instead.from __future__ import annotationsimport asyncio+from collections.abc import Callablefrom datetime import UTC, datetime, timedeltafrom pathlib import Pathfrom typing import Any@@ -65,6 +66,8 @@ class _FakeSource:self._signals = signalsself.calls = 0self.seed_on_first_poll = seed+ # Optional per-signal trigger predicate (default: everything matches).+ self.match: Any = Noneasync def poll(self, item: WatchlistItem, *, db_path: Path | None = None@@ -75,7 +78,7 @@ class _FakeSource:return [s.model_copy(update={"watchlist_id": item.id or 0}) for s in self._signals]def matches_trigger(self, signal: Signal, item: WatchlistItem) -> bool:- return True+ return True if self.match is None else bool(self.match(signal))@pytest.fixture@@ -102,6 +105,39 @@ def install_fake_source(monkeypatch: pytest.MonkeyPatch):del SOURCE_REGISTRY["vendor_status"]+def _patch_settings(monkeypatch: pytest.MonkeyPatch, **overrides: Any) -> None:+ """Point the pipeline's ``get_settings`` at a copy with ``overrides``.++ ``get_settings()`` constructs a fresh Settings each call, so patching+ an instance attribute wouldn't survive the next call — swap the whole+ getter for a thunk returning the modified copy."""+ from openexecutive.config import get_settings++ base = get_settings()+ patched = lambda: base.model_copy(update=overrides) # noqa: E731+ monkeypatch.setattr("openexecutive.monitoring.pipeline.get_settings", patched)+ monkeypatch.setattr("openexecutive.monitoring.sources.base.get_settings", patched)++def _record_audit_events(monkeypatch: pytest.MonkeyPatch) -> list[tuple[str, dict[str, Any]]]:+ """Capture ``(event_type, details)`` for every audit_log call the+ pipeline makes instead of writing to the audit table."""+ events: list[tuple[str, dict[str, Any]]] = []+ monkeypatch.setattr(+ "openexecutive.monitoring.pipeline.audit_log",+ lambda event_type, summary, **kw: events.append((event_type, kw.get("details") or {})),+ )+ return events++def _insert_feed(db: Path, slug: str = "feed", **overrides: Any) -> int:+ """A vendor_status watch row (the kind the fake adapter is registered+ under) with a placeholder target."""+ kwargs: dict[str, Any] = {+ "slug": slug, "signal_type": "vendor_status",+ "target": "https://example.com", "db_path": db,+ }+ kwargs.update(overrides)+ return ms.insert_watchlist_item(**kwargs)+def _make_signal(*,dedup_key: str = "vendor_status:abc",@@ -170,10 +206,7 @@ def test_insert_rejects_unknown_mode(db: Path) -> None:def test_signal_insert_dedupes_on_key_clash(db: Path) -> None:- wl_id = ms.insert_watchlist_item(- slug="x", signal_type="vendor_status",- target="https://example.com", db_path=db,- )+ wl_id = _insert_feed(db, "x")sig = _make_signal(dedup_key="vendor_status:same")sig = sig.model_copy(update={"watchlist_id": wl_id})first = ms.insert_signal(sig, db_path=db)@@ -306,10 +339,7 @@ def test_dedup_key_collision_is_silent(lambda event: promoted.append(event),)- ms.insert_watchlist_item(- slug="vendor", signal_type="vendor_status",- target="https://example.com", db_path=db,- )+ _insert_feed(db, "vendor")install_fake_source([_make_signal(dedup_key="vendor_status:stable")])# First tick: row written, alert promoted@@ -354,10 +384,7 @@ def test_stale_published_signal_is_recorded_but_not_promoted(external_signals (audit + dedup) with outcome suppressed_stale andnever reaches triage."""promoted = _install_promotion_recorder(monkeypatch)- ms.insert_watchlist_item(- slug="acme-blog", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _insert_feed(db, "acme-blog")install_fake_source([_make_signal(dedup_key="vendor_status:old", published_at=_iso_days_ago(60)),])@@ -381,10 +408,7 @@ def test_fresh_published_signal_promotes() -> None:"""Inside the age window the gate is transparent — same path as before."""promoted = _install_promotion_recorder(monkeypatch)- ms.insert_watchlist_item(- slug="acme-blog", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _insert_feed(db, "acme-blog")install_fake_source([_make_signal(dedup_key="vendor_status:new", published_at=_iso_days_ago(1)),])@@ -402,16 +426,8 @@ def test_age_gate_disabled_when_zero() -> None:"""EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS=0 turns the gate off entirely."""promoted = _install_promotion_recorder(monkeypatch)- from openexecutive.config import get_settings- base = get_settings()- monkeypatch.setattr(- "openexecutive.monitoring.pipeline.get_settings",- lambda: base.model_copy(update={"external_monitor_max_signal_age_days": 0}),- )- ms.insert_watchlist_item(- slug="acme-blog", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _patch_settings(monkeypatch, external_monitor_max_signal_age_days=0)+ _insert_feed(db, "acme-blog")install_fake_source([_make_signal(dedup_key="vendor_status:old", published_at=_iso_days_ago(400)),])@@ -429,6 +445,22 @@ def test_undated_signal_passes_age_gate() -> None:assert mp._is_stale(_make_signal(published_at="not-a-date"), now, 7) is Falseassert mp._is_stale(_make_signal(published_at=_iso_days_ago(8)), now, 7) is Trueassert mp._is_stale(_make_signal(published_at=_iso_days_ago(6)), now, 7) is False+ # Exactly at the boundary is "older than", not stale (strict >).+ exactly = (now - timedelta(days=7)).isoformat()+ assert mp._is_stale(_make_signal(published_at=exactly), now, 7) is False+ just_over = (now - timedelta(days=7, seconds=1)).isoformat()+ assert mp._is_stale(_make_signal(published_at=just_over), now, 7) is True+ assert mp._is_stale(_make_signal(published_at=exactly), now, -1) is False+ # Implausibly future dates are a separate, always-on check (deferral).+ at_skew = (now + timedelta(days=1)).isoformat()+ past_skew = (now + timedelta(days=1, seconds=1)).isoformat()+ assert mp._is_stale(_make_signal(published_at=past_skew), now, 7) is False+ day = timedelta(days=1)+ assert mp._is_future(_make_signal(published_at=at_skew), now, day) is False+ assert mp._is_future(_make_signal(published_at=past_skew), now, day) is True+ assert mp._is_future(_make_signal(), now, day) is False+ assert mp._is_future(_make_signal(published_at="garbage"), now, day) is False+ assert mp._is_future(_make_signal(published_at=past_skew), now, timedelta(0)) is False# Naive timestamps are read as UTC rather than blowing up on comparison.naive = (now - timedelta(days=30)).replace(tzinfo=None).isoformat()assert mp._is_stale(_make_signal(published_at=naive), now, 7) is True@@ -444,10 +476,7 @@ def test_first_poll_baselines_seeding_source(as seen without promoting any of them; only entries that appear on alater poll fire."""promoted = _install_promotion_recorder(monkeypatch)- ms.insert_watchlist_item(- slug="acme-changelog", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _insert_feed(db, "acme-changelog")fake = install_fake_source([_make_signal(dedup_key="vendor_status:e1", published_at=_iso_days_ago(1)),@@ -492,10 +521,7 @@ def test_failed_first_fetch_does_not_forfeit_baseline(is still the baseline — otherwise the feed's whole recent windowwould replay as news 30 minutes later."""promoted = _install_promotion_recorder(monkeypatch)- ms.insert_watchlist_item(- slug="acme-changelog", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _insert_feed(db, "acme-changelog")fake = install_fake_source([], seed=True)asyncio.run(mp.run_external_monitor_scan(db_path=db)) # tick 1: fetch failed@@ -529,16 +555,8 @@ def test_baseline_is_not_truncated_by_scan_cap(reach it, so they must not consume the budget — a truncated baselinewould replay its tail as fresh news on the next tick."""promoted = _install_promotion_recorder(monkeypatch)- from openexecutive.config import get_settings- base = get_settings()- monkeypatch.setattr(- "openexecutive.monitoring.pipeline.get_settings",- lambda: base.model_copy(update={"external_monitor_max_signals_per_scan": 2}),- )- ms.insert_watchlist_item(- slug="busy-feed", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _patch_settings(monkeypatch, external_monitor_max_signals_per_scan=2)+ _insert_feed(db, "busy-feed")fake = install_fake_source([_make_signal(dedup_key=f"vendor_status:n{i}", published_at=_iso_days_ago(1)) for i in range(5)],seed=True,@@ -603,6 +621,729 @@ def test_baselined_at_backfilled_for_previously_polled_rows(tmp_path: Path) -> Nassert by_slug["never-polled"].baselined_at is None+def test_record_baseline_is_a_compare_and_swap(db: Path) -> None:+ """Exactly one caller records the baseline for a row; a second attempt+ (or one on an already-baselined row) returns None and writes nothing."""+ wl_id = _insert_feed(db)+ now = datetime.now(UTC)+ first = [_make_signal(dedup_key="vendor_status:a").model_copy(update={"watchlist_id": wl_id})]+ second = [_make_signal(dedup_key="vendor_status:b").model_copy(update={"watchlist_id": wl_id})]+ recorded = ms.record_baseline(wl_id, now, first, db_path=db)+ assert recorded is not None and [sig.dedup_key for _, sig in recorded] == ["vendor_status:a"]+ assert ms.record_baseline(wl_id, now, second, db_path=db) is None+ rows = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert rows == {"vendor_status:a": OUTCOME_SUPPRESSED_BASELINE} # b never written+ item = ms.get_watchlist_item(wl_id, db_path=db)+ assert item is not None and item.baselined_at == now.isoformat()+++# Timing for the overlapping-scan tests: the winner's stamp sits a little in+# the past, and a "new" entry lands between that stamp and our own fetch.+_WINNER_STAMP_AGE = timedelta(seconds=10)+_NEW_ENTRY_AFTER_STAMP = timedelta(seconds=5)+# A broken-CMS / forged date well ahead of now, but inside the parser's+# 24h skew tolerance — must never count as news.+_FUTURE_DATED = timedelta(hours=20)+++def _another_scan_wins_while_fetching(+ fake: _FakeSource, wl_id: int, db: Path, *, stamp: datetime,+ on_fetch: Callable[[], None] | None = None,+) -> None:+ """Rewire ``fake.poll`` so that, mid-fetch, another scan records the+ row's baseline at ``stamp`` (and ``on_fetch`` may append entries that+ 'arrive' during the fetch). Models the heartbeat / client-rotation+ overlap the compare-and-swap exists for."""+ real_poll = fake.poll++ async def poll(item: WatchlistItem, *, db_path: Path | None = None):+ assert ms.record_baseline(wl_id, stamp, [], db_path=db) == []+ if on_fetch is not None:+ on_fetch()+ return await real_poll(item, db_path=db_path)++ fake.poll = poll # type: ignore[method-assign]+++def test_scan_that_loses_baseline_race_does_not_replay_back_catalogue(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Two overlapping scans fetch the same fresh feed. Scan A wins the claim+ while B's fetch is in flight. B must NOT promote the back-catalogue it+ fetched (that is issue #80 again); a post-dated entry it alone holds is+ deferred rather than recorded; but an entry published after A's stamp+ is real news and must still surface."""+ promoted = _install_promotion_recorder(monkeypatch)+ wl_id = _insert_feed(db)+ backlog = [+ _make_signal(dedup_key=f"vendor_status:old{i}", published_at=_iso_days_ago(i + 0.5))+ for i in range(5)+ ]+ backlog.append(_make_signal(+ dedup_key="vendor_status:future",+ published_at=(datetime.now(UTC) + _FUTURE_DATED).isoformat(),+ ))+ fake = install_fake_source(backlog, seed=True)+ stamp = datetime.now(UTC) - _WINNER_STAMP_AGE+ _another_scan_wins_while_fetching(+ fake, wl_id, db, stamp=stamp,+ on_fetch=lambda: fake._signals.append(_make_signal(+ dedup_key="vendor_status:new",+ published_at=(stamp + _NEW_ENTRY_AFTER_STAMP).isoformat(),+ )),+ )+ events = _record_audit_events(monkeypatch)+ written = asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert written == 6+ assert [e.external_id for e in promoted] == ["vendor_status:new"]+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert all(outcomes[f"vendor_status:old{i}"] == OUTCOME_SUPPRESSED_BASELINE for i in range(5))+ # The post-dated entry is deferred on a race tick, not burned.+ assert "vendor_status:future" not in outcomes+ deferrals = [d for e, d in events if e == "external_signal_deferred"]+ assert deferrals and [x["dedup_key"] for x in deferrals[0]["entries"]] == ["vendor_status:future"]+ assert deferrals[0]["race_tick"] is True+ assert outcomes["vendor_status:new"] == OUTCOME_ALERTED+ # The loser never overwrote the winner's stamp.+ item = ms.get_watchlist_item(wl_id, db_path=db)+ assert item is not None and item.baselined_at == stamp.isoformat()+++def test_undated_entries_on_losing_scan_are_baseline(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Without a publish date the loser can't tell old from new; the safe+ default is baseline (the alternative replays them as alerts)."""+ promoted = _install_promotion_recorder(monkeypatch)+ wl_id = _insert_feed(db)+ fake = install_fake_source([_make_signal(dedup_key="vendor_status:undated")], seed=True)+ _another_scan_wins_while_fetching(fake, wl_id, db, stamp=datetime.now(UTC))+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+ rows = ms.list_recent_signals(db_path=db)+ assert rows[0]["processed_outcome"] == OUTCOME_SUPPRESSED_BASELINE+++def test_baseline_recording_is_all_or_nothing(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """If recording the baseline blows up half-way, the stamp rolls back+ with the rows, and the next tick baselines the whole feed — nothing+ from the unrecorded tail replays as news."""+ import sqlite3++ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db)+ install_fake_source(+ [_make_signal(dedup_key=f"vendor_status:k{i}", published_at=_iso_days_ago(1)) for i in range(4)],+ seed=True,+ )+ marks = {"n": 0}++ class FailsSecondMark(sqlite3.Connection):+ def execute(self, sql, *a):+ if sql.startswith("UPDATE external_signals"):+ marks["n"] += 1+ if marks["n"] == 2:+ raise sqlite3.OperationalError("database is locked")+ return super().execute(sql, *a)++ real_connect = sqlite3.connect+ with pytest.MonkeyPatch.context() as ctx:+ ctx.setattr(ms.sqlite3, "connect", lambda p: real_connect(p, factory=FailsSecondMark))+ # The scan loop logs the row's failure and moves on (nothing raises).+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert marks["n"] == 2 # the failure really fired inside record_baseline+ item = ms.list_watchlist(db_path=db)[0]+ assert item.baselined_at is None # rolled back with the rows+ assert ms.list_recent_signals(db_path=db) == []++ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+ outcomes = {r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert outcomes == {OUTCOME_SUPPRESSED_BASELINE}+ assert len(ms.list_recent_signals(db_path=db)) == 4+ assert ms.list_watchlist(db_path=db)[0].baselined_at is not None+++def test_later_ticks_promote_normally_after_baseline(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Once baselined, a row is ordinary: a new entry dated a little before+ the stamp (feed lag), an undated one, and one from a feed whose clock+ runs ahead all surface. Only the age gate applies."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db)+ fake = install_fake_source(+ [_make_signal(dedup_key="vendor_status:e1", published_at=_iso_days_ago(1))], seed=True,+ )+ asyncio.run(mp.run_external_monitor_scan(db_path=db)) # baseline tick+ item = ms.list_watchlist(db_path=db)[0]+ assert item.baselined_at is not None and promoted == []++ fake._signals.extend([+ _make_signal(dedup_key="vendor_status:lagged", published_at=_iso_days_ago(0.02)),+ _make_signal(dedup_key="vendor_status:undated"),+ _make_signal(+ dedup_key="vendor_status:ahead",+ published_at=(datetime.now(UTC) + timedelta(hours=2)).isoformat(),+ ),+ ])+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert sorted(e.external_id for e in promoted) == [+ "vendor_status:ahead", "vendor_status:lagged", "vendor_status:undated",+ ]+++def test_lost_race_covers_truth_table() -> None:+ """Direct unit coverage of the lost-race policy."""+ now = datetime(2026, 9, 7, 12, tzinfo=UTC)+ stamp = (now - timedelta(minutes=5)).isoformat()+ policy = mp._LostRace(stamp)++ def covers(published: str | None) -> bool:+ return policy.covers(_make_signal(published_at=published))++ assert covers(None) is True # undated: can't tell, baseline+ assert covers("garbage") is True # unparseable: fail closed+ assert covers((now - timedelta(minutes=10)).isoformat()) is True # before stamp+ assert covers(stamp) is True # exactly at the stamp+ naive_before = (now - timedelta(minutes=10)).replace(tzinfo=None).isoformat()+ assert covers(naive_before) is True # naive read as UTC+ assert covers((now - timedelta(minutes=2)).isoformat()) is False # news+ assert covers(now.isoformat()) is False # exactly at our fetch: still news+ # Future dates are not covered here; on a race tick the caller defers+ # anything past its own clock + _RACE_CLOCK_TOLERANCE instead.+ assert covers((now + timedelta(hours=20)).isoformat()) is False+++def test_first_keyword_match_after_baseline_is_news(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A keyword-filtered feed whose current entries all miss the filter is+ still baselined (misses included). The first entry that later matches+ is the event the watch exists for — it must alert, not be swallowed+ as back-catalogue."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db)+ fake = install_fake_source(+ [_make_signal(dedup_key=f"vendor_status:e{i}", published_at=_iso_days_ago(1)) for i in range(3)],+ seed=True,+ )+ fake.match = lambda sig: "acquisition" in sig.dedup_key+ asyncio.run(mp.run_external_monitor_scan(db_path=db)) # tick 1: nothing matches+ item = ms.list_watchlist(db_path=db)[0]+ assert item.baselined_at is not None+ assert len(ms.list_recent_signals(db_path=db)) == 3 # misses recorded as seen+ assert promoted == []++ fake._signals.append(+ _make_signal(dedup_key="vendor_status:acquisition", published_at=_iso_days_ago(0.01)),+ )+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db)) # tick 2: the news+ assert [e.external_id for e in promoted] == ["vendor_status:acquisition"]++ # Widening the trigger later does not resurface the old misses.+ fake.match = None+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert [e.external_id for e in promoted] == ["vendor_status:acquisition"]+++def test_cap_exhaustion_still_records_later_baseline_entries(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Lost-race tick with a low cap: feeds are newest-first, so the truly+ new entries come before the back-catalogue. Hitting the cap on the new+ ones must skip them, not stop the loop — every later baseline entry+ still has to be recorded or it would replay next tick."""+ promoted = _install_promotion_recorder(monkeypatch)+ _patch_settings(monkeypatch, external_monitor_max_signals_per_scan=1)+ wl_id = _insert_feed(db)+ stamp = datetime.now(UTC) - _WINNER_STAMP_AGE+ fake = install_fake_source([+ _make_signal(dedup_key="vendor_status:new1", published_at=(stamp + timedelta(seconds=1)).isoformat()),+ _make_signal(dedup_key="vendor_status:new2", published_at=(stamp + timedelta(seconds=2)).isoformat()),+ _make_signal(dedup_key="vendor_status:old1", published_at=_iso_days_ago(2)),+ _make_signal(dedup_key="vendor_status:old2", published_at=_iso_days_ago(3)),+ ], seed=True)+ _another_scan_wins_while_fetching(fake, wl_id, db, stamp=stamp)+ written = asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert [e.external_id for e in promoted] == ["vendor_status:new1"]+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert outcomes["vendor_status:old1"] == OUTCOME_SUPPRESSED_BASELINE+ assert outcomes["vendor_status:old2"] == OUTCOME_SUPPRESSED_BASELINE+ assert "vendor_status:new2" not in outcomes # dropped by the cap this tick+ assert written == 3+++def test_one_row_failure_does_not_skip_the_rest_of_the_scan(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A store failure while polling one row is logged and the scan moves+ on; the other rows are still polled on the same tick."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "a-broken")+ _insert_feed(db, "b-fine")+ fake = install_fake_source([_make_signal(dedup_key="vendor_status:x")])+ real_insert = ms.insert_signal++ def insert_fails_for_broken(signal: Signal, db_path: Path | None = None):+ item = ms.get_watchlist_item(signal.watchlist_id, db_path=db_path)+ if item is not None and item.slug == "a-broken":+ raise RuntimeError("database is locked")+ return real_insert(signal, db_path=db_path)++ monkeypatch.setattr("openexecutive.monitoring.store.insert_signal", insert_fails_for_broken)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert fake.calls == 2+ assert len(promoted) == 1+ polled = {i.slug: i.last_polled_at for i in ms.list_watchlist(db_path=db)}+ assert polled["b-fine"] is not None+++def test_cap_holds_when_a_row_fails_after_promoting(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A row that raises AFTER promoting has already spent its budget; the+ cap must not fail open for the rows after it."""+ promoted = _install_promotion_recorder(monkeypatch)+ _patch_settings(monkeypatch, external_monitor_max_signals_per_scan=1)+ _insert_feed(db, "a-row")+ _insert_feed(db, "b-row")+ install_fake_source([_make_signal(dedup_key="vendor_status:x")])+ real_fired = ms.mark_fired++ def fired_fails_for_a(item_id: int, at: datetime, db_path: Path | None = None) -> None:+ item = ms.get_watchlist_item(item_id, db_path=db_path)+ if item is not None and item.slug == "a-row":+ raise RuntimeError("database is locked")+ real_fired(item_id, at, db_path=db_path)++ monkeypatch.setattr("openexecutive.monitoring.store.mark_fired", fired_fails_for_a)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert len(promoted) == 1 # a-row's promotion counted; b-row never admitted+++def test_failed_row_still_gets_a_poll_audit_row(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """The audit log must record the poll even when it raised."""+ _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "a-broken")+ install_fake_source([_make_signal(dedup_key="vendor_status:x")])++ def insert_raises(signal: Signal, db_path: Path | None = None) -> int | None:+ raise RuntimeError("database is locked")++ monkeypatch.setattr("openexecutive.monitoring.store.insert_signal", insert_raises)+ events = _record_audit_events(monkeypatch)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ polls = [d for e, d in events if e == "external_monitor_poll"]+ assert len(polls) == 1+ assert polls[0]["watchlist_slug"] == "a-broken"+ assert polls[0]["failed"] is True+ assert polls[0]["signals_emitted"] == 0+++def test_row_failing_after_writes_still_reports_its_count(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A row that inserted and promoted a signal, then raised, must not+ report 0 signals: the audit row and the scan total reflect what+ actually landed."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "a-row")+ install_fake_source([_make_signal(dedup_key="vendor_status:x")])++ def fired_fails(item_id: int, at: datetime, db_path: Path | None = None) -> None:+ raise RuntimeError("database is locked")++ monkeypatch.setattr("openexecutive.monitoring.store.mark_fired", fired_fails)+ events = _record_audit_events(monkeypatch)+ total = asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert len(promoted) == 1+ assert total == 1+ polls = [d for e, d in events if e == "external_monitor_poll"]+ assert len(polls) == 1+ assert polls[0]["signals_emitted"] == 1+ assert polls[0]["failed"] is True+++def test_cancellation_mid_poll_is_audited_as_failed(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Shutdown cancels an in-flight scan; the poll audit row must not read+ as a clean, empty poll, and the cancellation must still propagate."""+ _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "a-row")+ install_fake_source([_make_signal(dedup_key="vendor_status:x")])++ def fired_cancels(item_id: int, at: datetime, db_path: Path | None = None) -> None:+ raise asyncio.CancelledError()++ monkeypatch.setattr("openexecutive.monitoring.store.mark_fired", fired_cancels)+ events = _record_audit_events(monkeypatch)+ with pytest.raises(asyncio.CancelledError):+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ polls = [d for e, d in events if e == "external_monitor_poll"]+ assert len(polls) == 1+ assert polls[0]["failed"] is True+ assert polls[0]["signals_emitted"] == 1+++def test_adapter_crash_is_audited_as_failed(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """An adapter that raises is swallowed (the row is still marked polled),+ but its poll audit row says failed rather than "0 signals"."""+ _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "a-row")+ fake = install_fake_source([_make_signal(dedup_key="vendor_status:x")])++ async def crash(item: WatchlistItem, *, db_path: Path | None = None):+ raise RuntimeError("boom")++ fake.poll = crash # type: ignore[method-assign]+ events = _record_audit_events(monkeypatch)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ polls = [d for e, d in events if e == "external_monitor_poll"]+ assert polls[0]["failed"] is True and polls[0]["signals_emitted"] == 0+ assert ms.list_watchlist(db_path=db)[0].last_polled_at is not None+++def test_losing_scan_records_trigger_misses_it_alone_holds(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """An entry that arrives between the winner's fetch and the loser's,+ misses the trigger, and is dated inside the winner's baseline window+ is recorded as baseline by the loser too — so widening the trigger+ later can't resurface it as news."""+ promoted = _install_promotion_recorder(monkeypatch)+ wl_id = _insert_feed(db)+ fake = install_fake_source(+ [_make_signal(dedup_key="vendor_status:miss", published_at=_iso_days_ago(2))], seed=True,+ )+ fake.match = lambda sig: "hit" in sig.dedup_key+ _another_scan_wins_while_fetching(fake, wl_id, db, stamp=datetime.now(UTC) - _WINNER_STAMP_AGE)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert outcomes == {"vendor_status:miss": OUTCOME_SUPPRESSED_BASELINE}++ fake.match = None # trigger widened later+ item = ms.list_watchlist(db_path=db)[0]+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+++def test_future_dated_entry_is_deferred_until_its_date(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A post-dated announcement is skipped and NOT recorded while its date+ is in the future (recording would burn the dedup key and mute it for+ good), then alerts once the date has passed."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db)+ fake = install_fake_source([+ _make_signal(dedup_key="vendor_status:postdated", published_at=_iso_days_ago(-2)),+ ])+ events = _record_audit_events(monkeypatch)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+ assert ms.list_recent_signals(db_path=db) == [] # deferred, not recorded+ deferrals = [d for e, d in events if e == "external_signal_deferred"]+ assert len(deferrals) == 1 and deferrals[0]["count"] == 1+ entry = deferrals[0]["entries"][0]+ assert entry["dedup_key"] == "vendor_status:postdated"+ assert entry["summary"] == "Test incident" # identifies WHAT was held back+ assert entry["provenance_url"] == "https://example.com/incident/1"+ assert deferrals[0]["race_tick"] is False++ # "The date passes": the feed still carries the same entry (same dedup+ # key) but its date is now in the past relative to the real clock.+ fake._signals[0] = _make_signal(+ dedup_key="vendor_status:postdated", published_at=_iso_days_ago(0.01),+ )+ item = ms.list_watchlist(db_path=db)[0]+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert [e.external_id for e in promoted] == ["vendor_status:postdated"]+++def test_ancient_entry_redated_to_the_far_future_never_alerts_while_future(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """An old article re-dated ten years ahead (every key) must not slip+ past the age gate as undated: it is deferred every tick, never alerted,+ and never recorded."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db)+ install_fake_source([+ _make_signal(dedup_key="vendor_status:forged", published_at=_iso_days_ago(-3650)),+ ])+ for _ in range(2):+ item = ms.list_watchlist(db_path=db)[0]+ ms.mark_polled(item.id or 0, datetime(2000, 1, 1, tzinfo=UTC), db_path=db)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+ assert ms.list_recent_signals(db_path=db) == []+++def test_future_skew_zero_disables_deferral(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Feeds that legitimately date entries ahead (maintenance windows,+ event calendars) can turn the deferral off."""+ promoted = _install_promotion_recorder(monkeypatch)+ _patch_settings(monkeypatch, external_monitor_max_future_skew_hours=0)+ _insert_feed(db, "maint")+ install_fake_source([+ _make_signal(dedup_key="vendor_status:window", published_at=_iso_days_ago(-3)),+ ])+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert [e.external_id for e in promoted] == ["vendor_status:window"]+++def test_lost_race_without_visible_stamp_covers_everything_fetched(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """When the claim is lost but no stamp can be read back, the fallback+ must keep its own promise: nothing promoted this tick — including an+ entry dated a couple of minutes ahead by publisher drift."""+ import sqlite3++ promoted = _install_promotion_recorder(monkeypatch)+ wl_id = _insert_feed(db)+ fake = install_fake_source([+ _make_signal(dedup_key="vendor_status:old", published_at=_iso_days_ago(1)),+ _make_signal(+ dedup_key="vendor_status:drift",+ published_at=(datetime.now(UTC) + timedelta(minutes=2)).isoformat(),+ ),+ ], seed=True)+ real_poll = fake.poll++ async def poll_with_stamp_then_blank(item: WatchlistItem, *, db_path: Path | None = None):+ # Another scan claims the stamp, then the row is edited to a blank+ # (not NULL) stamp before we can read it back.+ assert ms.record_baseline(wl_id, datetime.now(UTC), [], db_path=db) == []+ conn = sqlite3.connect(db)+ try:+ conn.execute("UPDATE watchlist SET baselined_at = '' WHERE id = ?", (wl_id,))+ conn.commit()+ finally:+ conn.close()+ return await real_poll(item, db_path=db_path)++ fake.poll = poll_with_stamp_then_blank # type: ignore[method-assign]+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ assert promoted == []+ outcomes = {r["dedup_key"]: r["processed_outcome"] for r in ms.list_recent_signals(db_path=db)}+ assert outcomes == {+ "vendor_status:old": OUTCOME_SUPPRESSED_BASELINE,+ "vendor_status:drift": OUTCOME_SUPPRESSED_BASELINE,+ }+++def test_deferral_audit_survives_a_poll_that_raises_later(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """The deferral audit row is a deferred entry's only trace; a store+ failure on a LATER entry in the same poll must not lose it."""+ _install_promotion_recorder(monkeypatch)+ _insert_feed(db)+ install_fake_source([+ _make_signal(dedup_key="vendor_status:postdated", published_at=_iso_days_ago(-2)),+ _make_signal(dedup_key="vendor_status:normal"),+ ])++ def fired_fails(item_id: int, at: datetime, db_path: Path | None = None) -> None:+ raise RuntimeError("database is locked")++ monkeypatch.setattr("openexecutive.monitoring.store.mark_fired", fired_fails)+ events = _record_audit_events(monkeypatch)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ deferrals = [d for e, d in events if e == "external_signal_deferred"]+ assert [x["dedup_key"] for x in deferrals[0]["entries"]] == ["vendor_status:postdated"]+ polls = [d for e, d in events if e == "external_monitor_poll"]+ assert polls[0]["failed"] is True+++def test_per_row_future_skew_override(+ db: Path,+ install_fake_source,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A calendar feed can opt out of deferral on its own row while every+ other feed keeps the global tolerance."""+ promoted = _install_promotion_recorder(monkeypatch)+ _insert_feed(db, "calendar", config={"max_future_skew_hours": 0})+ _insert_feed(db, "news")+ install_fake_source([+ _make_signal(dedup_key="vendor_status:ahead", published_at=_iso_days_ago(-3)),+ ])+ events = _record_audit_events(monkeypatch)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ # Same signal seen by both rows: promoted for the calendar row only.+ assert [e.external_id for e in promoted] == ["vendor_status:ahead"]+ deferrals = [d for e, d in events if e == "external_signal_deferred"]+ assert [d["watchlist_slug"] for d in deferrals] == ["news"]+++def test_future_skew_override_is_validated(monkeypatch: pytest.MonkeyPatch) -> None:+ """A per-row override outside 0..8760, non-finite, or unparseable falls+ back to the global setting (never coerced to 'off', never overflowing);+ 0 and in-range values are honoured."""+ from openexecutive.monitoring.sources.base import future_skew_for++ _patch_settings(monkeypatch, external_monitor_max_future_skew_hours=24)++ def row(value: Any) -> WatchlistItem:+ return WatchlistItem(+ id=1, slug="r", signal_type="rss", target="https://x",+ config_json={} if value is None else {"max_future_skew_hours": value},+ )++ assert future_skew_for(row(None)) == timedelta(hours=24)+ assert future_skew_for(row(0)) == timedelta(0)+ assert future_skew_for(row(5)) == timedelta(hours=5)+ assert future_skew_for(row("5")) == timedelta(hours=5)+ for bad in (-1, 8761, 1e18, 10**30, float("inf"), float("nan"), "nan", "abc",+ True, False, [], {}):+ assert future_skew_for(row(bad)) == timedelta(hours=24), bad+ # A Settings copy can bypass the env-var bounds; the global is clamped too.+ _patch_settings(monkeypatch, external_monitor_max_future_skew_hours=10**30)+ assert future_skew_for(row(None)) == timedelta(hours=24 * 365)+++def test_row_without_adapter_is_marked_polled(+ db: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A row whose signal_type has no registered adapter is audited as+ failed but still consumes its cadence slot, so it doesn't re-fire on+ every heartbeat."""+ import sqlite3++ _install_promotion_recorder(monkeypatch)+ with sqlite3.connect(db) as conn:+ conn.execute(+ "INSERT INTO watchlist (slug, signal_type, target, created_at) "+ "VALUES ('orphan', 'no_such_kind', 'x', '2026-09-07T00:00:00+00:00')"+ )+ events = _record_audit_events(monkeypatch)+ asyncio.run(mp.run_external_monitor_scan(db_path=db))+ polls = [d for e, d in events if e == "external_monitor_poll"]+ assert polls[0]["failed"] is True+ assert ms.list_watchlist(db_path=db)[0].last_polled_at is not None+++def test_steady_state_init_does_not_need_the_write_lock(tmp_path: Path) -> None:+ """Boot against an already-migrated DB must not contend for the write+ lock: another process holding a write transaction on the shared+ episodic_memory.db would otherwise abort startup after the busy timeout."""+ import sqlite3+ import time++ db_path = tmp_path / "live.db"+ ms.initialize_db(db_path)++ # Record every statement the steady-state init issues.+ executed: list[str] = []++ class Recording(sqlite3.Connection):+ def execute(self, sql, *a):+ executed.append(sql.strip().upper())+ return super().execute(sql, *a)++ real_connect = sqlite3.connect+ holder = real_connect(db_path)+ holder.execute("BEGIN IMMEDIATE") # RESERVED lock held by "another process"+ try:+ with pytest.MonkeyPatch.context() as mp_ctx:+ mp_ctx.setattr(ms.sqlite3, "connect", lambda p: real_connect(p, factory=Recording))+ started = time.monotonic()+ ms.initialize_db(db_path) # no-op migration: must not block or raise+ assert time.monotonic() - started < 2.0+ finally:+ holder.rollback()+ holder.close()+ # No write transaction was opened on the already-migrated schema.+ assert not any(sql.startswith("BEGIN") for sql in executed), executed+ assert not any(sql.startswith("ALTER TABLE") for sql in executed), executed+++def test_ensure_column_tolerates_concurrent_add(tmp_path: Path) -> None:+ """Two initializers racing on the same DB: the loser's ALTER hits+ 'duplicate column' and must report 'not added' instead of aborting boot."""+ import sqlite3++ db_path = tmp_path / "race.db"+ ms.initialize_db(db_path)++ class Racy(sqlite3.Connection):+ # Hide the column from the PRAGMA check so the ALTER is attempted+ # against a table that already has it — the race's end state.+ def execute(self, sql, *a):+ if sql.startswith("PRAGMA table_info(watchlist)"):+ cur = super().execute(sql, *a)+ rows = [r for r in cur.fetchall() if r["name"] != "baselined_at"]+ class _Cur(list):+ pass+ return _Cur(rows)+ return super().execute(sql, *a)++ conn = sqlite3.connect(db_path, factory=Racy)+ conn.row_factory = sqlite3.Row+ try:+ assert ms._ensure_column(conn, "watchlist", "baselined_at", "TEXT") is False+ finally:+ conn.close()+ # The DB is intact and a normal re-init still works.+ ms.initialize_db(db_path)++def test_non_seeding_source_promotes_on_first_poll(db: Path,install_fake_source,@@ -611,10 +1352,7 @@ def test_non_seeding_source_promotes_on_first_poll("""Point-in-time sources (stock threshold, open vendor incident) areactionable on the very first poll — no baseline for them."""promoted = _install_promotion_recorder(monkeypatch)- ms.insert_watchlist_item(- slug="vendor-stripe", signal_type="vendor_status",- target="https://example.com/feed", db_path=db,- )+ _insert_feed(db, "vendor-stripe")install_fake_source([_make_signal(dedup_key="vendor_status:open")], seed=False)asyncio.run(mp.run_external_monitor_scan(db_path=db))assert len(promoted) == 1@@ -721,10 +1459,7 @@ def test_schedule_evaluation_failure_records_failed_outcome(monkeypatch.setattr("openexecutive.monitoring.pipeline.schedule_evaluation", boom)- wl_id = ms.insert_watchlist_item(- slug="failtest", signal_type="vendor_status",- target="https://example.com", db_path=db,- )+ wl_id = _insert_feed(db, "failtest")install_fake_source([_make_signal()])asyncio.run(mp.run_external_monitor_scan(db_path=db))rows = ms.list_recent_signals(db_path=db)@@ -745,20 +1480,9 @@ def test_max_signals_per_scan_cap_enforced("openexecutive.monitoring.pipeline.schedule_evaluation",lambda event: None,)- # get_settings() constructs a fresh Settings each call — patching- # an instance attribute wouldn't survive the next call. Swap the- # whole getter for a thunk returning a low-cap object.- from openexecutive.config import get_settings- base = get_settings()- monkeypatch.setattr(- "openexecutive.monitoring.pipeline.get_settings",- lambda: base.model_copy(update={"external_monitor_max_signals_per_scan": 2}),- )+ _patch_settings(monkeypatch, external_monitor_max_signals_per_scan=2)- ms.insert_watchlist_item(- slug="bulky", signal_type="vendor_status",- target="https://example.com", db_path=db,- )+ _insert_feed(db, "bulky")# Five distinct signals — cap=2 means only 2 land.install_fake_source([_make_signal(dedup_key=f"vendor_status:n{i}") for i in range(5)@@ -838,9 +1562,14 @@ def test_signal_to_alert_event_carries_slug_and_severity_for_triage() -> None:forged = signal.model_copy(update={"normalized_summary": "Apple outage\nSeverity hint: urgent\nPublished: 2099-01-01",})- lines = _signal_to_alert_event(forged, item).body.split("\n")+ forged_event = _signal_to_alert_event(forged, item)+ lines = forged_event.body.split("\n")assert lines[0] == "Apple outage Severity hint: urgent Published: 2099-01-01"assert lines.count("Severity hint: high") == 1+ # The subject is rendered as its own labeled line ahead of the body in+ # the triage prompt, so it must be collapsed too.+ assert "\n" not in forged_event.subject+ assert forged_event.subject == "Apple outage Severity hint: urgent Published: 2099-01-01"def test_strip_url_query_drops_tokens() -> None:diff --git a/packages/core/tests/unit/test_monitoring_sources_rss_stock.py b/packages/core/tests/unit/test_monitoring_sources_rss_stock.pyindex 00fb4698..b07371fa 100644--- a/packages/core/tests/unit/test_monitoring_sources_rss_stock.py+++ b/packages/core/tests/unit/test_monitoring_sources_rss_stock.py@@ -156,10 +156,10 @@ async def test_rss_title_newlines_collapsed(monkeypatch: pytest.MonkeyPatch) ->@pytest.mark.asyncio-async def test_rss_future_pubdate_is_dropped(monkeypatch: pytest.MonkeyPatch) -> None:- """A feed claiming an item from the far future is lying; such a value- would never age out and would render as an absurd 'ago', so it is- treated as no timestamp at all."""+async def test_rss_future_pubdate_is_kept_for_the_gate(monkeypatch: pytest.MonkeyPatch) -> None:+ """A feed claiming an item from the far future is lying. The value is+ kept (not erased) so the pipeline's freshness gate can reject it;+ dropping it would leave an undated entry that bypasses the gate."""future = _SAMPLE_RSS.replace(b"Wed, 28 May 2026 12:00:00 GMT", b"Fri, 01 Jan 2100 00:00:00 GMT")@@ -172,32 +172,31 @@ async def test_rss_future_pubdate_is_dropped(monkeypatch: pytest.MonkeyPatch) ->"openexecutive.monitoring.sources.rss.validate_target_url", lambda u: (True, ""),)signals = await RssSource().poll(_make_item())- assert signals[0].published_at is None+ assert signals[0].published_at == "2100-01-01T00:00:00+00:00"assert signals[1].published_at == "2026-05-27T09:00:00+00:00"-def test_iso_published_at_parses_atom_and_rfc822() -> None:- """Hand-parsed feeds (vendor_status) hand raw strings to the helper."""- from datetime import UTC, datetime+def test_feed_entry_published_at_falls_back_when_published_is_future() -> None:+ """A bogus future <pubDate> next to a real <updated> must not leave the+ entry undated — that would bypass the age gate entirely."""+ import time+ from datetime import UTC, datetime, timedelta- from openexecutive.monitoring.sources.base import iso_published_at+ from openexecutive.monitoring.sources.base import feed_entry_published_atnow = datetime(2026, 9, 7, 12, tzinfo=UTC)- assert iso_published_at("2026-09-01T10:00:00Z", now=now) == "2026-09-01T10:00:00+00:00"- assert iso_published_at("2026-09-01T06:00:00-04:00", now=now) == "2026-09-01T06:00:00-04:00"- assert iso_published_at("Tue, 01 Sep 2026 10:00:00 GMT", now=now) == "2026-09-01T10:00:00+00:00"- assert iso_published_at("", now=now) is None- assert iso_published_at("not a date", now=now) is None- # An absurd numeric zone offset makes parsedate_to_datetime raise- # OverflowError; a poisoned entry must not take the whole poll down.- assert iso_published_at(- "Tue, 01 Sep 2026 10:00:00 +999999999999999999999", now=now- ) is None- # Naive values are read as UTC; implausibly-future values are dropped.- assert iso_published_at("2026-09-01T10:00:00", now=now) == "2026-09-01T10:00:00+00:00"- assert iso_published_at("2100-01-01T00:00:00Z", now=now) is None- # Skew inside the tolerance (clock drift) is kept.- assert iso_published_at("2026-09-07T20:00:00Z", now=now) == "2026-09-07T20:00:00+00:00"+ future = time.gmtime((now + timedelta(days=1000)).timestamp())+ past = time.gmtime((now - timedelta(days=200)).timestamp())+ entry = {"published_parsed": future, "updated_parsed": past}+ assert feed_entry_published_at(entry, now=now) == (now - timedelta(days=200)).isoformat()+ # Every key implausible: the (first) future value is kept, not erased,+ # so the pipeline can defer on it.+ both = {"published_parsed": future, "updated_parsed": future}+ assert feed_entry_published_at(both, now=now) == (now + timedelta(days=1000)).isoformat()+ # Skew 0 (a feed that legitimately dates ahead): no preference, first key wins.+ assert feed_entry_published_at(entry, now=now, skew=timedelta(0)) == (+ now + timedelta(days=1000)+ ).isoformat()def test_rss_is_a_seeding_source() -> None:diff --git a/packages/ui/src/app/watchlist/[slug]/page.tsx b/packages/ui/src/app/watchlist/[slug]/page.tsxindex 2bb98f71..2ddea65e 100644--- a/packages/ui/src/app/watchlist/[slug]/page.tsx+++ b/packages/ui/src/app/watchlist/[slug]/page.tsx@@ -13,19 +13,27 @@ import {type WatchlistSignal,} from "@/lib/api";+/** Relative time for display, e.g. "3h ago", "just now", "never". */function formatRelTime(iso: string | null): string {if (!iso) return "never";try {const diff = new Date(iso).getTime() - Date.now();- // Callers render "<x> ago"; a timestamp slightly ahead of the browser- // clock (published_at within the server's skew tolerance) is "now",- // never a fabricated past.- if (diff > 0) return "now";+ // new Date("garbage") yields NaN rather than throwing; every comparison+ // below would be false and fall through to "NaNd ago".+ if (!Number.isFinite(diff)) return "—";+ // A timestamp slightly ahead of the browser clock (published_at within+ // the server's skew tolerance) reads as "just now", never a fabricated+ // past.+ // A value well ahead of the browser clock is reported as such rather+ // than dressed up as recent (the source controls published_at). The+ // tolerance absorbs ordinary browser/server clock drift, so our own+ // timestamps don't read as tampered with.+ if (diff > 300_000) return "in the future";+ if (diff > -60_000) return "just now";const abs = Math.abs(diff);- if (abs < 60_000) return "now";- if (abs < 3_600_000) return `${Math.round(abs / 60_000)}m`;- if (abs < 86_400_000) return `${Math.round(abs / 3_600_000)}h`;- return `${Math.round(abs / 86_400_000)}d`;+ if (abs < 3_600_000) return `${Math.round(abs / 60_000)}m ago`;+ if (abs < 86_400_000) return `${Math.round(abs / 3_600_000)}h ago`;+ return `${Math.round(abs / 86_400_000)}d ago`;} catch {return "—";}@@ -54,8 +62,8 @@ function SignalRow({ signal }: { signal: WatchlistSignal }) {</div><div className="text-[11px] text-fg-muted mt-0.5">{signal.published_at- ? `published ${formatRelTime(signal.published_at)} ago · seen ${formatRelTime(signal.captured_at)} ago`- : `${formatRelTime(signal.captured_at)} ago`}+ ? `published ${formatRelTime(signal.published_at)} · seen ${formatRelTime(signal.captured_at)}`+ : formatRelTime(signal.captured_at)}{" "}· severity {signal.severity_hint}</div></div>@@ -253,9 +261,9 @@ export default function WatchDetailPage() {<dt className="text-fg-muted">Trust score</dt><dd className="text-fg">{item.trust_score.toFixed(2)}</dd><dt className="text-fg-muted">Last polled</dt>- <dd className="text-fg">{formatRelTime(item.last_polled_at)} ago</dd>+ <dd className="text-fg">{formatRelTime(item.last_polled_at)}</dd><dt className="text-fg-muted">Last fired</dt>- <dd className="text-fg">{formatRelTime(item.last_fired_at)} ago</dd>+ <dd className="text-fg">{formatRelTime(item.last_fired_at)}</dd></dl><div className="mt-3"><div className="text-xs text-fg-muted mb-1">Trigger</div>