AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
fix(research): harden the run after adversarial review
Rufusa744c877ca65
diff --git a/packages/core/openexecutive/api/routes/audit.py b/packages/core/openexecutive/api/routes/audit.pyindex 862fbcb6..a13469a1 100644--- a/packages/core/openexecutive/api/routes/audit.py+++ b/packages/core/openexecutive/api/routes/audit.py@@ -418,6 +418,7 @@ def _compute_cost_summary(events: list[Any]) -> CostSummary | None:"cache_read_input_tokens","cache_creation_input_tokens","output_tokens",+ "web_search_requests",)def _as_int(value: Any) -> int:diff --git a/packages/core/openexecutive/api/routes/watchlist.py b/packages/core/openexecutive/api/routes/watchlist.pyindex 0bbc5ca9..63dea4c5 100644--- a/packages/core/openexecutive/api/routes/watchlist.py+++ b/packages/core/openexecutive/api/routes/watchlist.py@@ -230,7 +230,7 @@ async def create_watchlist_entry(body: WatchlistCreate) -> WatchlistItem:config.setdefault("display_name", body.display_label)# Insert-time validation: a non-feed rss target is converted to a- # scrape-backed page_watch (when scrapeable) or rejected, so a dead+ # page_watch (when the page has readable text) or rejected, so a dead# row never reaches the scan loop. Non-rss types pass through untouched.try:signal_type, target, config = await validate_and_normalize_target(diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.jsonindex d4b4a359..4b6063df 100644--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json@@ -1,7 +1,7 @@{"section_id": "external_monitoring","title": "External Monitoring",- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page, fetched with the keyless httpx fetcher. A row written before the external scrape path was removed still carries `config.fetch = \"xcrawl\"`; the adapter treats that as a stale baseline, re-captures it from the plain fetch without emitting a signal, and clears the marker.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a `page_watch` when the body the feed check already fetched reduces to readable text, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never fetched. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (each list entry is parsed to its name: \"Tesla (TSLA) — Model Y is the benchmark\" grounds as `Tesla` plus the ticker `TSLA`, \"GM / Chevrolet (…)\" as both names, and \"Brex (brex.com)\" pins Brex's own domains so only those sites count as its own source; the description after the dash never becomes a term, short names such as BYD or GM match as whole words, and a ticker grounds only as itself) (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's leading name or all its words run together, never a generic trailing word; or a primary-source host such as sec.gov), +1 finding confidence high, +1 cross-specialist consensus (the finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. It also always needs a linked finding that cites the source — company data plus a decision mention can reach the score bar alone, but nothing goes live on hearsay. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Linked finding.** `finding_index` is required on `propose_watch` and the pass is told so; when the model omits it anyway, or cites a finding that does not concern the source, the policy links the finding that best does (a cited URL on the target's site or the ticker named in its text outranks a mere mention of the entity; then the entity's own site, confidence) and says so in the first reason. A proposal that no finding cites is rejected as having no evidence.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `department=<slug>` — a tag hint, deliberately not the `channel` field triage reads as a team room), so the alerts it raises queue on the head's `/today` (as action items, since routed alerts are never hidden in Monitoring) and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Every fetch is the on-box httpx fetcher: SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`. No path delegates a fetch to an external service.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page, fetched with the keyless httpx fetcher. A row written before the external scrape path was removed still carries `config.fetch = \"xcrawl\"`; the adapter treats that as a stale baseline, re-captures it from the plain fetch without emitting a signal, and clears the marker.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a `page_watch` when the body the feed check already fetched reduces to readable text, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never fetched. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nThe run itself: seven specialists (or the subset named by `RESEARCH_SPECIALISTS`) each make one call with the basic `web_search` server tool, capped at `RESEARCH_WEB_SEARCH_MAX_USES` searches (default 3, separate from the chat knob), and emit at most three findings — the cap sits in the tool schema and is applied again after parsing. Every specialist sees what department heads asked to have watched (a `DEPARTMENT WATCH INTERESTS` block) and the ten most recent decisions, and may ground a finding in either. Findings are deduplicated, low-confidence ones dropped, then the Executive routes the rest; the routing pass cannot start a workflow (`run_workflow` is withheld, `suggest_workflow` is not). There is no external fetch or verify service: every page read is the on-box fetcher. The periodic run is considered every `WATCHLIST_RESEARCH_INTERVAL_MINUTES` (default 360) and skipped when nothing changed. Every model call the run makes is recorded as a `cache_event` audit row tagged with the run id, and the run's result and its `watchlist_research_periodic_ran` row carry a `usage` block (calls, tokens, searches, per source), so what a run did is readable from the audit log.\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (each list entry is parsed to its name: \"Tesla (TSLA) — Model Y is the benchmark\" grounds as `Tesla` plus the ticker `TSLA`, \"GM / Chevrolet (…)\" as both names, and \"Brex (brex.com)\" pins Brex's own domains so only those sites count as its own source; the description after the dash never becomes a term, short names such as BYD or GM match as whole words, and a ticker grounds only as itself) (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's leading name or all its words run together, never a generic trailing word; or a primary-source host such as sec.gov), +1 finding confidence high, +1 cross-specialist consensus (the finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. It also always needs a linked finding that cites the source — company data plus a decision mention can reach the score bar alone, but nothing goes live on hearsay. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Linked finding.** `finding_index` is required on `propose_watch` and the pass is told so; when the model omits it anyway, or cites a finding that does not concern the source, the policy links the finding that best does (a cited URL on the target's site or the ticker named in its text outranks a mere mention of the entity; then the entity's own site, confidence) and says so in the first reason. A proposal that no finding cites is rejected as having no evidence.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `department=<slug>` — a tag hint, deliberately not the `channel` field triage reads as a team room), so the alerts it raises queue on the head's `/today` (as action items, since routed alerts are never hidden in Monitoring) and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Every fetch is the on-box httpx fetcher: SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`. No path delegates a fetch to an external service.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.","mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / readable page → page_watch / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute","generated_at": "2026-09-12T17:10:00+00:00"}diff --git a/packages/core/openexecutive/audit/usage.py b/packages/core/openexecutive/audit/usage.pyindex a980c4fd..2b865436 100644--- a/packages/core/openexecutive/audit/usage.py+++ b/packages/core/openexecutive/audit/usage.py@@ -40,7 +40,13 @@ def bind_research_run(run_id: str | None) -> Iterator[UsageRollup]:"""Tag every usage row logged inside the block with ``run_id`` and sumit into the yielded :class:`UsageRollup`. Tasks spawned inside theblock (the specialist fan-out) inherit the binding and share the same- rollup object, so their calls count too."""+ rollup object, so their calls count too.++ Bind in the task that runs the calls, never inside an async generator+ that a consumer may abandon: a ContextVar set inside a generator lands+ in the consumer's context, and an abandoned generator's ``finally``+ runs elsewhere (``executive_research.run`` drives its body in a child+ task for this reason)."""rollup = UsageRollup()token_id = _research_run_id.set(run_id)token_rollup = _research_rollup.set(rollup)@@ -59,6 +65,13 @@ def usage_counts(message: Any) -> dict[str, Any] | None:usage = getattr(message, "usage", None)if usage is None:return None+ # A real usage block carries integer token counts; anything else (a+ # mock, a provider quirk) is not usage and must not become a row.+ if not any(+ isinstance(getattr(usage, field, None), int)+ for field in ("input_tokens", "output_tokens")+ ):+ return Noneraw_cost = getattr(usage, "cost", None)try:cost_usd = float(raw_cost) if raw_cost is not None else None@@ -90,16 +103,21 @@ def log_model_usage(Never raises and never touches the request: it reads ``message.usage``after the call, so prompt caching is unaffected. Returns the counters it- recorded (``None`` when the response had no usage block). ``session_id``- / ``turn_id`` fall back to the audit ContextVars when omitted, as every- other audit row does.+ recorded (``None`` when the response had no usage block — the call still+ counts toward a bound research run's rollup, so the run's call count is+ honest even when a provider omits usage). ``session_id`` / ``turn_id``+ fall back to the audit ContextVars when omitted, as every other audit+ row does."""- counts = usage_counts(message)+ try:+ counts = usage_counts(message)+ rollup = _research_rollup.get()+ if rollup is not None:+ rollup.add(actor, counts)+ except Exception: # noqa: BLE001 — a malformed usage block is not the caller's problem+ return Noneif counts is None:return None- rollup = _research_rollup.get()- if rollup is not None:- rollup.add(actor, counts)stop_reason = getattr(message, "stop_reason", None)details: dict[str, Any] = {"model": model,@@ -149,13 +167,13 @@ class UsageRollup:self.by_actor: dict[str, dict[str, int]] = {}def add(self, actor: str, counts: dict[str, Any] | None) -> None:- if counts is None:- return+ """Count one call; ``counts`` may be ``None`` for a response that+ carried no usage block (the call happened, its tokens are unknown)."""self.calls += 1bucket = self.by_actor.setdefault(actor, {"calls": 0, **dict.fromkeys(self._FIELDS, 0)})bucket["calls"] += 1for field in self._FIELDS:- value = _as_int(counts.get(field, 0))+ value = _as_int((counts or {}).get(field, 0))self.totals[field] += valuebucket[field] += value@@ -170,7 +188,7 @@ class UsageRollup:def _as_int(value: Any) -> int:try:return int(value or 0)- except (TypeError, ValueError):+ except (TypeError, ValueError, OverflowError):return 0diff --git a/packages/core/openexecutive/config.py b/packages/core/openexecutive/config.pyindex 68adbcb5..4755efe4 100644--- a/packages/core/openexecutive/config.py+++ b/packages/core/openexecutive/config.py@@ -460,9 +460,9 @@ class Settings(BaseSettings):# WEB_SEARCH_MAX_USES for deeper digs.web_search_max_uses: int = Field(2, alias="WEB_SEARCH_MAX_USES")# Searches per specialist in the executive_research fan-out. Separate- # from the chat knob above because the fan-out multiplies it by the- # number of specialists (seven by default), and every search adds- # results that each later pass re-reads.+ # from the chat knob above because every specialist in the fan-out+ # (seven by default) gets this many, and every search adds results that+ # each later pass re-reads.research_web_search_max_uses: int = Field(3, alias="RESEARCH_WEB_SEARCH_MAX_USES")diff --git a/packages/core/openexecutive/monitoring/research/scheduler.py b/packages/core/openexecutive/monitoring/research/scheduler.pyindex 7ef5b8b3..f9844b4c 100644--- a/packages/core/openexecutive/monitoring/research/scheduler.py+++ b/packages/core/openexecutive/monitoring/research/scheduler.py@@ -459,11 +459,14 @@ async def run_watchlist_research_scan(workflow = WORKFLOW_REGISTRY["executive_research"]input_cls = workflow.input_model()+ run_id = str(uuid.uuid4())+ # The workflow tags every model-usage row it writes with this id, so+ # a run that times out or crashes is still traceable to its calls.wf_inputs = input_cls(note=f"periodic research tick at {now.isoformat()}",+ run_id=run_id,)- run_id = str(uuid.uuid4())try:create_run(run_id,@@ -574,7 +577,7 @@ async def run_watchlist_research_scan(EVENT_FAILED,"Watchlist research timed out",actor="scheduler",- details={"state_hash": state_hash, "reason": "timeout"},+ details={"state_hash": state_hash, "reason": "timeout", "run_id": run_id},)return 0except Exception as exc:@@ -586,7 +589,7 @@ async def run_watchlist_research_scan(EVENT_FAILED,f"Watchlist research crashed: {exc}",actor="scheduler",- details={"state_hash": state_hash, "error": str(exc)[:300]},+ details={"state_hash": state_hash, "error": str(exc)[:300], "run_id": run_id},)return 0diff --git a/packages/core/openexecutive/monitoring/research/specialist_research.py b/packages/core/openexecutive/monitoring/research/specialist_research.pyindex 55872cef..051aaeba 100644--- a/packages/core/openexecutive/monitoring/research/specialist_research.py+++ b/packages/core/openexecutive/monitoring/research/specialist_research.py@@ -47,8 +47,8 @@ async def research_one_specialist()tools: list[dict[str, Any]] = [EMIT_RESEARCH_FINDINGS_TOOL]- # The research fan-out has its own search cap: the shared chat knob is- # multiplied by the number of specialists here.+ # The research fan-out has its own search cap (each specialist gets+ # this many searches; the chat knob stays with chat and standing queries).web_search = build_web_search_tool(max_uses=get_settings().research_web_search_max_uses,)@@ -111,15 +111,7 @@ def _extract_findings(block_input = getattr(block, "input", None) or {}items = block_input.get("findings")if isinstance(items, list):- if len(items) > PER_SPECIALIST_FINDING_CAP:- # The schema's maxItems already refuses this on Anthropic;- # a translated provider may not enforce it, so keep the- # first N — the model was told to lead with the material ones.- logger.warning(- "research: specialist=%s emitted %d findings — keeping the first %d",- specialist_slug, len(items), PER_SPECIALIST_FINDING_CAP,- )- raw_findings.extend(items[:PER_SPECIALIST_FINDING_CAP])+ raw_findings.extend(items)if not raw_findings:# Empty research is a valid answer ("nothing in my domain@@ -143,6 +135,15 @@ def _extract_findings()continueparsed.append(finding)+ if len(parsed) > PER_SPECIALIST_FINDING_CAP:+ # The schema states the cap; a provider may not enforce it. Applied+ # after parsing so malformed items never crowd out valid ones — the+ # model was told to lead with the material findings.+ logger.warning(+ "research: specialist=%s emitted %d findings — keeping the first %d",+ specialist_slug, len(parsed), PER_SPECIALIST_FINDING_CAP,+ )+ parsed = parsed[:PER_SPECIALIST_FINDING_CAP]return parseddiff --git a/packages/core/openexecutive/monitoring/research/tools.py b/packages/core/openexecutive/monitoring/research/tools.pyindex 91f7cde6..7afeeace 100644--- a/packages/core/openexecutive/monitoring/research/tools.py+++ b/packages/core/openexecutive/monitoring/research/tools.py@@ -28,9 +28,9 @@ EMIT_RESEARCH_FINDINGS_TOOL: dict[str, Any] = {"properties": {"findings": {"type": "array",- # The cap is part of the schema, not only the prompt: a- # longer list is refused at the API, and the parser slices- # to the same number for providers that do not enforce it.+ # The cap is part of the schema, not only the prompt; the+ # parser applies the same number after parsing, so it holds+ # whether or not the provider honours maxItems."maxItems": PER_SPECIALIST_FINDING_CAP,"description": (f"At most {PER_SPECIALIST_FINDING_CAP} findings. Prefer "diff --git a/packages/core/openexecutive/monitoring/sources/page_watch.py b/packages/core/openexecutive/monitoring/sources/page_watch.pyindex 9b177359..bca88adb 100644--- a/packages/core/openexecutive/monitoring/sources/page_watch.py+++ b/packages/core/openexecutive/monitoring/sources/page_watch.py@@ -79,9 +79,11 @@ _SUMMARY_SNIPPET_CHARS = 160_ADDED_TEXT_CHARS = 1_000# <script>/<style>/<noscript> blocks: drop content, not just tags, so inline-# JS/CSS never counts as "visible text".-_DROP_BLOCKS_RE = re.compile(r"(?is)<(script|style|noscript)\b.*?</\1>")-_COMMENT_RE = re.compile(r"(?s)<!--.*?-->")+# JS/CSS never counts as "visible text". Removed by a linear scan+# (_strip_blocks), not a lazy `.*?` regex: on a page full of unterminated+# openers such a regex is quadratic, and a page the server sends is the+# attacker's to shape.+_DROP_BLOCK_TAGS = ("script", "style", "noscript")_TAG_RE = re.compile(r"(?s)<[^>]+>")_WS_RE = re.compile(r"\s+")@@ -245,8 +247,7 @@ def html_to_text(body: bytes) -> str:markup = body.decode("utf-8", errors="replace")except Exception:return ""- markup = _DROP_BLOCKS_RE.sub(" ", markup)- markup = _COMMENT_RE.sub(" ", markup)+ markup = _strip_blocks(markup)markup = _TAG_RE.sub(" ", markup)text = html.unescape(markup)# Full normalized text — NOT truncated. The fetch is already byte-capped@@ -255,6 +256,48 @@ def html_to_text(body: bytes) -> str:return _WS_RE.sub(" ", text).strip()+def _strip_blocks(markup: str) -> str:+ """Drop HTML comments and <script>/<style>/<noscript> blocks (content+ included) in one pass that is linear in the page length. An unterminated+ comment or block runs to the end of the page: a truncated script is not+ visible text either."""+ lower = markup.lower()+ n = len(markup)+ out: list[str] = []+ i = 0+ while i < n:+ j = lower.find("<", i)+ if j == -1:+ out.append(markup[i:])+ break+ out.append(markup[i:j])+ if lower.startswith("<!--", j):+ end = lower.find("-->", j + 4)+ i = n if end == -1 else end + 3+ out.append(" ")+ continue+ tag = next(+ (+ t for t in _DROP_BLOCK_TAGS+ if lower.startswith("<" + t, j)+ and (j + 1 + len(t) >= n or not (lower[j + 1 + len(t)].isalnum() or lower[j + 1 + len(t)] == "_"))+ ),+ None,+ )+ if tag is None:+ out.append("<")+ i = j + 1+ continue+ close = lower.find("</" + tag, j)+ if close == -1:+ i = n+ else:+ gt = lower.find(">", close)+ i = n if gt == -1 else gt + 1+ out.append(" ")+ return "".join(out)++def _diff(old: str, new: str) -> tuple[int, str]:"""Return (percent-changed, added-text) in a single diff pass.diff --git a/packages/core/openexecutive/monitoring/target_validation.py b/packages/core/openexecutive/monitoring/target_validation.pyindex 25ca7b0b..ed263e7b 100644--- a/packages/core/openexecutive/monitoring/target_validation.py+++ b/packages/core/openexecutive/monitoring/target_validation.py@@ -46,6 +46,10 @@ logger = logging.getLogger(__name__)# happen to decode to a few characters; a real article, pricing or news page# clears it easily._MIN_PAGE_TEXT_CHARS = 200+# Only the head of the body is reduced for that check: 200 visible+# characters never need more, and the insert path runs on the request+# thread, so the work is bounded whatever the server sends.+_READABILITY_SCAN_BYTES = 262_144class WatchlistTargetError(Exception):@@ -116,7 +120,7 @@ def _is_readable_page(body: bytes) -> bool:the text threshold rejects shells and one-line error pages."""if b"\x00" in body[:4096]:return False- return len(html_to_text(body)) >= _MIN_PAGE_TEXT_CHARS+ return len(html_to_text(body[:_READABILITY_SCAN_BYTES])) >= _MIN_PAGE_TEXT_CHARSasync def _feed_check(diff --git a/packages/core/openexecutive/orchestrator/watchlist_tools.py b/packages/core/openexecutive/orchestrator/watchlist_tools.pyindex d3911b10..df908449 100644--- a/packages/core/openexecutive/orchestrator/watchlist_tools.py+++ b/packages/core/openexecutive/orchestrator/watchlist_tools.py@@ -452,7 +452,7 @@ async def _validated_target(config = _display_label_config(signal_type, tool_input.get("display_label"))# Validate the target before it lands: a non-feed rss URL is converted- # to a scrape-backed page_watch (when scrapeable) or rejected, so the+ # to a page_watch (when the page has readable text) or rejected, so the# research workflow can't seed the watchlist with dead feed rows.try:signal_type, target, config = await validate_and_normalize_target(diff --git a/packages/core/openexecutive/workflows/executive_research.py b/packages/core/openexecutive/workflows/executive_research.pyindex 22590420..26238652 100644--- a/packages/core/openexecutive/workflows/executive_research.py+++ b/packages/core/openexecutive/workflows/executive_research.py@@ -309,6 +309,14 @@ class ExecutiveResearchInput(BaseModel):"competitor signals')."),)+ run_id: str | None = Field(+ default=None,+ description=(+ "Optional id the caller already recorded for this run (the "+ "periodic scheduler's workflow_runs id). Every model-usage row "+ "the run writes carries it; generated when omitted."+ ),+ )class ExecutiveResearchWorkflow(Workflow):@@ -383,18 +391,41 @@ class ExecutiveResearchWorkflow(Workflow):store: ChromaDBStore,) -> AsyncIterator[WorkflowEvent]:assert isinstance(inputs, ExecutiveResearchInput)+ run_id = inputs.run_id or f"research-{uuid.uuid4().hex[:12]}"# Every model call the run makes (specialists, synthesis, watchlist- # pass) records a cache_event row tagged with this id and summed into- # the rollup, which the result event reports as `usage`.- with bind_research_run(f"research-{uuid.uuid4().hex[:12]}") as rollup:- async for event in self._run_events(inputs, store, rollup):+ # pass) records a cache_event row tagged with the run id and summed+ # into a rollup the result event reports as `usage`. The body runs in+ # its own task so the binding lives in that task's context: set+ # inside this generator it would land in the consumer's context and+ # outlive an abandoned run.+ queue: asyncio.Queue[WorkflowEvent | None] = asyncio.Queue()++ async def _produce() -> None:+ try:+ with bind_research_run(run_id) as rollup:+ async for event in self._run_events(inputs, store, rollup, run_id):+ queue.put_nowait(event)+ finally:+ queue.put_nowait(None)++ producer = asyncio.create_task(_produce())+ try:+ while True:+ event = await queue.get()+ if event is None:+ breakyield event+ await producer # surface a crash inside the run to the consumer+ finally:+ if not producer.done():+ producer.cancel()async def _run_events(self,inputs: ExecutiveResearchInput,store: ChromaDBStore,rollup: UsageRollup,+ run_id: str,) -> AsyncIterator[WorkflowEvent]:# ------------------------------------------------------------------# Step 1: gather_context@@ -569,7 +600,7 @@ class ExecutiveResearchWorkflow(Workflow):type="result",data={"findings": [], "tool_calls": [], "narrative": "",- "usage": rollup.as_dict(),+ "usage": {"run_id": run_id, **rollup.as_dict()},},)yield WorkflowEvent(@@ -695,7 +726,7 @@ class ExecutiveResearchWorkflow(Workflow):],"tool_calls": tool_calls,"narrative": narrative,- "usage": rollup.as_dict(),+ "usage": {"run_id": run_id, **rollup.as_dict()},},)yield WorkflowEvent(type="artifact", content=artifact)@@ -1222,11 +1253,17 @@ def _render_research_context(return "\n".join(parts)+# Bounds on the DEPARTMENT WATCH INTERESTS block: departments listed and+# entities named per department. Generous for any real org chart; they only+# stop a runaway list from crowding the specialist turn.+_MAX_INTEREST_DEPARTMENTS = 20+_MAX_INTEREST_ENTITIES = 20++def _render_department_interests(departments: list[Any] | None) -> list[str]:"""One ``- <slug>: <entities>`` line per department with watched- entities (at most 20 departments, 20 entities each). Shared by the- specialist context and the watchlist turn so both name the same- interests."""+ entities. Shared by the specialist context and the watchlist turn so+ both name the same interests."""lines: list[str] = []for state in departments or []:config = getattr(state, "config", None)@@ -1237,8 +1274,8 @@ def _render_department_interests(departments: list[Any] | None) -> list[str]:if str(e).strip()]if slug and entities:- lines.append(f"- {slug}: {', '.join(entities[:20])}")- return lines[:20]+ lines.append(f"- {slug}: {', '.join(entities[:_MAX_INTEREST_ENTITIES])}")+ return lines[:_MAX_INTEREST_DEPARTMENTS]def _render_team_roster(people: list[Any]) -> str:diff --git a/packages/core/tests/unit/test_audit_usage.py b/packages/core/tests/unit/test_audit_usage.pyindex abc66623..20aa5914 100644--- a/packages/core/tests/unit/test_audit_usage.py+++ b/packages/core/tests/unit/test_audit_usage.py@@ -38,6 +38,10 @@ def test_usage_counts_reads_tokens_cost_and_server_searches() -> None:def test_usage_counts_tolerates_missing_fields_and_no_usage() -> None:assert au.usage_counts(SimpleNamespace(content=[])) is None+ # A mock (or a usage object without integer counts) is not usage.+ from unittest.mock import MagicMock+ assert au.usage_counts(MagicMock()) is None+ assert au.usage_counts(SimpleNamespace(usage=SimpleNamespace(cost="0.1"))) is Nonecounts = au.usage_counts(_message(input_tokens=5, cost="not-a-number"))assert counts is not Noneassert counts["output_tokens"] == 0 and counts["web_search_requests"] == 0@@ -94,6 +98,13 @@ async def test_rollup_collects_calls_made_in_gathered_tasks(audit: AuditLogger)assert rollup.as_dict()["calls"] == 3 and rollup.as_dict()["input_tokens"] == 6+def test_log_model_usage_never_raises_on_a_malformed_usage_block(audit: AuditLogger) -> None:+ msg = _message(input_tokens=float("inf"), output_tokens=3)+ assert au.log_model_usage(msg, model="m", actor="triage") is not None+ row = audit.query(event_type="cache_event")[0]+ assert row.details["input_tokens"] == 0 and row.details["output_tokens"] == 3++def test_log_model_usage_returns_none_without_usage(audit: AuditLogger) -> None:assert au.log_model_usage(SimpleNamespace(content=[]), model="m", actor="triage") is Noneassert audit.query(event_type="cache_event") == []@@ -104,13 +115,21 @@ def test_usage_rollup_sums_per_source() -> None:roll.add("specialist_research", {"input_tokens": 10, "output_tokens": 2, "web_search_requests": 3})roll.add("specialist_research", {"input_tokens": 5, "output_tokens": 1, "web_search_requests": 1})roll.add("research_synthesis", {"input_tokens": 7, "cache_read_input_tokens": 100})- roll.add("research_watchlist", None) # a call without usage is not counted+ roll.add("research_watchlist", None) # a call without usage still counts as a callout = roll.as_dict()- assert out["calls"] == 3+ assert out["calls"] == 4+ assert out["by_source"]["research_watchlist"]["calls"] == 1assert out["input_tokens"] == 22 and out["web_search_requests"] == 4assert out["cache_read_input_tokens"] == 100assert out["by_source"]["specialist_research"] == {"calls": 2, "input_tokens": 15, "cache_read_input_tokens": 0,"cache_creation_input_tokens": 0, "output_tokens": 3, "web_search_requests": 4,}- assert list(out["by_source"]) == ["research_synthesis", "specialist_research"]+ assert list(out["by_source"]) == ["research_synthesis", "research_watchlist", "specialist_research"]+++def test_log_model_usage_counts_a_call_without_usage_in_the_rollup(audit: AuditLogger) -> None:+ with au.bind_research_run("run-x") as rollup:+ assert au.log_model_usage(SimpleNamespace(content=[]), model="m", actor="triage") is None+ assert rollup.as_dict()["calls"] == 1+ assert audit.query(event_type="cache_event") == [] # no row without countsdiff --git a/packages/core/tests/unit/test_executive_research_scheduler.py b/packages/core/tests/unit/test_executive_research_scheduler.pyindex 8fd7c637..58678753 100644--- a/packages/core/tests/unit/test_executive_research_scheduler.py+++ b/packages/core/tests/unit/test_executive_research_scheduler.py@@ -139,8 +139,9 @@ def test_bootstrap_is_idempotent(db: Path) -> None:class _MinimalInput:- def __init__(self, note: str = "", **_: Any) -> None:+ def __init__(self, note: str = "", run_id: str | None = None, **_: Any) -> None:self.note = note+ self.run_id = run_iddef model_dump(self) -> dict[str, Any]:return {"note": self.note}@@ -192,8 +193,11 @@ async def test_scan_audit_row_carries_the_runs_usage("by_source": {"specialist_research": {"calls": 7, "web_search_requests": 14}},}+ seen_inputs: list[Any] = []+async def fake_run(*, inputs, store):from openexecutive.workflows.base import WorkflowEvent+ seen_inputs.append(inputs)yield WorkflowEvent(type="result", data={"findings": [], "tool_calls": [], "usage": usage},)@@ -214,6 +218,8 @@ async def test_scan_audit_row_carries_the_runs_usage(ran = get_audit_logger().query(event_type=research_scheduler.EVENT_RAN, limit=1)assert (ran[0].details or {}).get("usage") == usageassert "9 model call(s), 14 search(es)" in ran[0].summary+ # The scheduler hands its run id to the workflow so usage rows link to it.+ assert seen_inputs[0].run_id == (ran[0].details or {}).get("run_id")@pytest.mark.asynciodiff --git a/packages/core/tests/unit/test_executive_research_workflow.py b/packages/core/tests/unit/test_executive_research_workflow.pyindex c3fbc18c..7659dfd9 100644--- a/packages/core/tests/unit/test_executive_research_workflow.py+++ b/packages/core/tests/unit/test_executive_research_workflow.py@@ -309,6 +309,46 @@ async def test_workflow_runs_end_to_end_with_stubbed_specialists(# The run reports what it did; the stubs made no model calls.assert result_data["usage"]["calls"] == 0assert result_data["usage"]["by_source"] == {}+ assert result_data["usage"]["run_id"].startswith("research-")+++async def test_abandoned_run_leaves_no_usage_binding_behind(monkeypatch: pytest.MonkeyPatch) -> None:+ """A consumer that stops iterating mid-run must not keep the run's id+ bound in its own context (later usage rows would be mis-tagged), and+ the run's own cleanup must not raise."""+ import asyncio++ from openexecutive.audit import usage as au++ async def slow_research_one(slug, agent, ctx):+ await asyncio.sleep(0.01)+ return []++ monkeypatch.setattr(+ "openexecutive.workflows.executive_research.research_one_specialist", slow_research_one,+ )+ workflow = ExecutiveResearchWorkflow()+ gen = workflow.run(inputs=ExecutiveResearchInput(note="t", run_id="run-abandon"), store=MagicMock())+ first = await gen.__anext__()+ assert first.type == "step_start"+ assert au.get_research_run_id() is None # the binding lives in the run's task+ await gen.aclose()+ await asyncio.sleep(0.05)+ assert au.get_research_run_id() is None+++def test_finding_cap_applies_after_parsing() -> None:+ """Malformed items ahead of valid ones must not consume the cap."""+ from openexecutive.monitoring.research.prompts import PER_SPECIALIST_FINDING_CAP++ bad = {"title": "no summary"}+ good = {"title": "ok", "summary": "s", "severity_hint": "low",+ "suggested_audience": "noone", "confidence": "medium"}+ msg = _make_msg_with_tool_use(+ "emit_research_findings", {"findings": [bad] * PER_SPECIALIST_FINDING_CAP + [good, good]},+ )+ assert len(_extract_findings(msg, "cso")) == 2@pytest.mark.asynciodiff --git a/packages/core/tests/unit/test_monitoring_page_watch.py b/packages/core/tests/unit/test_monitoring_page_watch.pyindex 8d742b1f..711e9cae 100644--- a/packages/core/tests/unit/test_monitoring_page_watch.py+++ b/packages/core/tests/unit/test_monitoring_page_watch.py@@ -264,6 +264,24 @@ def test_page_watch_state_roundtrip(db: Path) -> None:assert st2["text_snapshot"] == "text B"+def test_html_to_text_is_linear_on_unterminated_openers() -> None:+ """A page full of unterminated comment / script openers must reduce in+ linear time: the insert-time conversion runs this on whatever a server+ sends, on the request thread."""+ import time++ for body in (b"<!--" * 100_000, b"<script>" * 60_000, b"<style" * 80_000):+ t0 = time.monotonic()+ assert html_to_text(body) == ""+ assert time.monotonic() - t0 < 1.0+ # Block stripping still behaves: content of script/style/noscript and+ # comments is gone, an unrelated tag such as <scripts> is only a tag.+ assert html_to_text(+ b"<p>a</p><script type=x>js()</script>b<STYLE>c</STYLE><!-- x -->d"+ b"<noscript>n</noscript><scripts>e</scripts>"+ ) == "a b d e"++def test_page_watch_registered() -> None:assert "page_watch" in list_registered_kinds()diff --git a/packages/core/tests/unit/test_target_validation.py b/packages/core/tests/unit/test_target_validation.pyindex d449d555..ca929032 100644--- a/packages/core/tests/unit/test_target_validation.py+++ b/packages/core/tests/unit/test_target_validation.py@@ -132,6 +132,22 @@ async def test_empty_body_is_not_feed_no_crash(@pytest.mark.asyncio+async def test_readability_check_is_bounded(monkeypatch: pytest.MonkeyPatch) -> None:+ """Only the head of a huge body is reduced for the readability check."""+ calls: list[int] = []++ def fake_html_to_text(body: bytes) -> str:+ calls.append(len(body))+ return "x" * 500++ monkeypatch.setattr(tv, "html_to_text", fake_html_to_text)+ _install(monkeypatch, fetch=_fetch_returning(b"<p>" * 1_000_000))+ st, _target, _config = await tv.validate_and_normalize_target("rss", "https://big.example/x", {})+ assert st == "page_watch"+ assert calls == [tv._READABILITY_SCAN_BYTES]++@pytest.mark.parametrize("body", [_EMPTY_SHELL, _THIN_PAGE, _BINARY], ids=["shell", "thin", "binary"])async def test_non_feed_without_readable_text_rejected(monkeypatch: pytest.MonkeyPatch, body: bytes,diff --git a/packages/ui/src/app/audit/usage/page.tsx b/packages/ui/src/app/audit/usage/page.tsxindex 17562130..0adb6bce 100644--- a/packages/ui/src/app/audit/usage/page.tsx+++ b/packages/ui/src/app/audit/usage/page.tsx@@ -58,13 +58,14 @@ function UsageRowCells({ u }: { u: UsageTotals }) {<td className="px-3 py-1.5 text-right tabular-nums">{fmtInt(u.cache_read_input_tokens)}</td><td className="px-3 py-1.5 text-right tabular-nums">{fmtInt(u.cache_creation_input_tokens)}</td><td className="px-3 py-1.5 text-right tabular-nums">{fmtInt(u.output_tokens)}</td>+ <td className="px-3 py-1.5 text-right tabular-nums">{fmtInt(u.web_search_requests ?? 0)}</td><td className="px-3 py-1.5 text-right tabular-nums">{cacheHitPct(u)}%</td><td className="px-3 py-1.5 text-right tabular-nums">{fmtCost(u.cost_usd)}</td></>);}-const COL_HEADERS = ["Calls", "Input", "Cache read", "Cache write", "Output", "Cached", "Cost"];+const COL_HEADERS = ["Calls", "Input", "Cache read", "Cache write", "Output", "Searches", "Cached", "Cost"];// Debounce window for refetching as the date-range filter changes (matches// the /audit list page).@@ -186,6 +187,7 @@ export default function TokenUsagePage() {<StatCard label="Fresh input" value={fmtInt(totals.input_tokens)} /><StatCard label="Cache read" value={fmtInt(totals.cache_read_input_tokens)} /><StatCard label="Cache write" value={fmtInt(totals.cache_creation_input_tokens)} />+ <StatCard label="Searches" value={fmtInt(totals.web_search_requests ?? 0)} hint="server-side web searches" /></div>) : null}@@ -201,7 +203,6 @@ export default function TokenUsagePage() {<thead><tr className="bg-surface-elevated/60 text-fg-muted text-xs"><th className="px-3 py-2 text-left font-medium">Source</th>- <th className="px-3 py-2 text-right font-medium">Searches</th>{COL_HEADERS.map((h) => (<th key={h} className="px-3 py-2 text-right font-medium">{h}</th>))}@@ -211,7 +212,6 @@ export default function TokenUsagePage() {{data.by_source.map((s: UsageBySource) => (<tr key={s.source} className="border-t border-line/60"><td className="px-3 py-1.5 font-mono text-xs text-fg">{s.source}</td>- <td className="px-3 py-1.5 text-right tabular-nums">{fmtInt(s.web_search_requests ?? 0)}</td><UsageRowCells u={s} /></tr>))}diff --git a/packages/ui/src/lib/api.ts b/packages/ui/src/lib/api.tsindex 37db0cf9..d280919f 100644--- a/packages/ui/src/lib/api.ts+++ b/packages/ui/src/lib/api.ts@@ -1617,7 +1617,8 @@ export interface UsageTotals {cache_read_input_tokens: number;cache_creation_input_tokens: number;output_tokens: number;- web_search_requests: number;+ // Absent on rows written before searches were recorded.+ web_search_requests?: number;cost_usd: number;}@@ -1641,7 +1642,8 @@ export interface UsageSummary {totals: UsageTotals;by_day: UsageByDay[];by_model: UsageByModel[];- by_source: UsageBySource[];+ // Absent on a backend older than the by-source breakdown.+ by_source?: UsageBySource[];}export async function getAuditUsage(