Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(watchlist): tighten department/decision grounding after adversarial review

Rufus7eaeff93e462
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index 53c29d39..f26becd2 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -1388,17 +1388,25 @@ external_monitoring:
match a company-data term for a direct add. Every vocabulary value is
a VocabEntry(kind, department): terms that came from a department carry
its slug, and a profile term a department also lists keeps the profile
- kind but gains the department (stronger kind wins, department is kept).
+ kind but gains the department (stronger kind wins, department is kept;
+ between two departments the one supplying the winning kind owns it).
- vocabulary precedence when a term appears in several sources:
- company > competitor > department_entity > vendor > ticker > initiative
- > priority > department_scope > watch.
+ company > competitor > vendor > ticker > department_entity > initiative
+ > priority > department_scope > watch. Profile kinds outrank department
+ kinds so the policy's (signal_type, kind) history keeps applying.
Partial (token) matching ignores stopwords and tokens < 4 chars.
- episodic decisions: the run loads the 10 most recent `decisions` rows
- from the last 90 days (watch_policy.recent_decisions); a proposal whose
- matched entity is named in one scores +1 ("named in a recent
- decision"), and when the term has no department but the decision was
- recorded for one, that department becomes the routing hint. Decisions
- are rendered to every specialist as RECENT DECISIONS (user turn).
+ from the last 90 days (watch_policy.recent_decisions; an unreadable
+ timestamp means not recent); a proposal whose matched COMPANY-DATA
+ entity (never a bare watch label) is named in one scores +1 ("named in
+ a recent decision"), and when the term has no department but the
+ decision was recorded for one, that department becomes the routing
+ hint. "Named" = the whole term as a phrase, or EVERY distinctive token
+ of the term as a whole word — a common word shared with the term is
+ not a mention. Decisions are rendered to every specialist as RECENT
+ DECISIONS (user turn) whether or not they match; only matching ones
+ move the research fingerprint (a decision that names nothing the
+ policy knows cannot change what it would do).
- score ledger: +2 entity in company data (mandatory), +1 entity already
watched under another source, +1 target is the entity's own source (for
stock/edgar the grounding entity IS the ticker / a URL whose SITE LABEL —
@@ -1452,9 +1460,13 @@ external_monitoring:
watchlist). propose_via_alert's new `external_id_suffix` (the ISO week)
makes the card recurring: an open card is refreshed in place
(coalesce_alert), an acknowledged one is not re-minted until the next
- week. The principal's pile-up nudge counts only suggestions with no
- route_to_department. /watchlist shows "for: <department>" on routed
- suggestion and research cards. Approve/decline stay identity-free.
+ week; an open card is refreshed whatever the week, so one department
+ never has two open cards. The principal's pile-up nudge counts only
+ suggestions that reached no head (route_to_person_id NULL — the
+ principal's own, and rows for a department without a head), so a
+ suggestion can never become invisible. /watchlist shows
+ "for: <department>" on routed suggestion and research cards.
+ Approve/decline stay identity-free.
- declines (`watchlist_declines`, keyed by normalized target — https
forced, www. dropped, host/path lower-cased, query/fragment/trailing
slash removed): explicit (Decline button, "Stop watching…",
@@ -1494,9 +1506,11 @@ external_monitoring:
approving / declining a suggestion never triggers a fresh research run.
It also hashes each department's watched_entities / charter scope /
goal key results, and the recent decisions whose summary names a
- vocabulary term (only those: the chat extractor writes decisions
- freely, and one naming nothing the policy knows cannot change what
- the policy would do).
+ vocabulary term — the same vocabulary the policy grounds with
+ (profile + initiatives + departments; watch labels never ground) —
+ only those: the chat extractor writes decisions freely, and one
+ naming nothing the policy knows cannot change what the policy would
+ do. A matching decision ageing past 90 days moves the hash once.
- surfaces: /watchlist "Suggested by the Executive" (Approve / Decline…
with reason), "Added by the Executive · rationale" on research cards
with "Stop watching…", the morning brief's HANDLED block (watching /
diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
index cab9c230..e278695c 100644
--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
@@ -1,7 +1,7 @@
{
"section_id": "external_monitoring",
"title": "External Monitoring",
- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's; or a primary-source host such as sec.gov), +1 finding confidence high, +1 verification confirmed, +1 cross-specialist consensus (the three finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity is named in one of the ten most recent episodic decisions from the last 90 days, ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > department entity > vendor > ticker > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile competitor that Finance also lists still routes to Finance.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `channel=\"department:<slug>\"`), so the alerts it raises queue on the head's `/today` and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions with no department. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",
+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's; or a primary-source host such as sec.gov), +1 finding confidence high, +1 verification confirmed, +1 cross-specialist consensus (the three finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `channel=\"department:<slug>\"`), so the alerts it raises queue on the head's `/today` and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",
"mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",
"generated_at": "2026-09-12T12:03:38.574978+00:00"
}
diff --git a/packages/core/openexecutive/departments/authority.py b/packages/core/openexecutive/departments/authority.py
index 06633c19..3725d676 100644
--- a/packages/core/openexecutive/departments/authority.py
+++ b/packages/core/openexecutive/departments/authority.py
@@ -185,10 +185,12 @@ def propose_via_alert(
``external_id_suffix`` turns the card into a recurring one: an open
(unread) card with the same dedup key is refreshed in place with the new
- body instead of a second row being inserted, and once the person has
- acted on it a fresh card is minted only when the suffix changes (callers
- pass e.g. the ISO week). Without a suffix the proposal is one-shot and a
- repeat with the same summary is suppressed for good.
+ body — whatever its suffix, so one situation never has two open cards —
+ and once the person has acted on it a fresh card is minted only when
+ the suffix changes (callers pass e.g. the ISO week). Without a suffix
+ the proposal is one-shot and a repeat with the same summary is
+ suppressed for good. Coalescing keeps the open card's routing: a head
+ change re-routes it only once the current card has been handled.
"""
from openexecutive.alerts.store import coalesce_alert, insert_alert
diff --git a/packages/core/openexecutive/monitoring/research/scheduler.py b/packages/core/openexecutive/monitoring/research/scheduler.py
index d828b90b..e776e59e 100644
--- a/packages/core/openexecutive/monitoring/research/scheduler.py
+++ b/packages/core/openexecutive/monitoring/research/scheduler.py
@@ -42,6 +42,7 @@ from __future__ import annotations
import hashlib
import logging
import uuid
+from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
@@ -118,6 +119,7 @@ def compute_research_state_hash(db_path: Path | None = None) -> str:
logger.exception("research.scheduler: profile load failed")
return "error:profile"
+ initiatives: list = []
try:
from openexecutive.memory.episodic import DB_PATH, get_active_initiatives
@@ -125,10 +127,8 @@ def compute_research_state_hash(db_path: Path | None = None) -> str:
# `get_active_initiatives` binds DB_PATH at def time; route
# through the resolver so monkeypatched DB_PATH in tests
# (and any future multi-DB harness) actually takes effect.
- for i in sorted(
- get_active_initiatives(db_path=db_path or DB_PATH),
- key=lambda x: (getattr(x, "title", "") or ""),
- ):
+ initiatives = list(get_active_initiatives(db_path=db_path or DB_PATH))
+ for i in sorted(initiatives, key=lambda x: (getattr(x, "title", "") or "")):
parts.append(
f"{getattr(i, 'title', '')}::{getattr(i, 'status', '')}"
)
@@ -152,38 +152,44 @@ def compute_research_state_hash(db_path: Path | None = None) -> str:
logger.exception("research.scheduler: watchlist load failed")
parts.append("WATCHLIST_ERROR")
- departments: list = []
- try:
- from openexecutive.departments.store import list_departments
+ from openexecutive.monitoring.research import watch_policy
+
+ departments = watch_policy.load_departments(db_path)
+ _append_part(parts, "DEPARTMENTS", lambda: [
+ f"{state.config.slug}::"
+ + "|".join(state.config.watched_entities)
+ + "::" + "|".join(state.config.charter.scope)
+ + "::" + "|".join(g.key_result for g in state.goals)
+ for state in sorted(departments, key=lambda d: d.config.slug)
+ ])
+
+ def _matching_decisions() -> list[str]:
+ # The same vocabulary the policy grounds with (minus watch labels,
+ # which never ground), so a decision moves the hash exactly when it
+ # would move a classification.
+ vocabulary = watch_policy.grounding_vocabulary(profile, initiatives, [], departments)
+ return [
+ f"{getattr(decision, 'id', '')}::{str(getattr(decision, 'summary', '') or '')[:120]}"
+ for decision in watch_policy.recent_decisions(db_path=db_path)
+ if any(watch_policy.named_in_decision(term, decision) for term in vocabulary)
+ ]
+
+ _append_part(parts, "DECISIONS", _matching_decisions)
+
+ digest = hashlib.sha256("\n".join(parts).encode()).hexdigest()
+ return digest
- departments = list(list_departments())
- parts.append("DEPARTMENTS:")
- for state in sorted(departments, key=lambda d: d.config.slug):
- parts.append(
- f"{state.config.slug}::"
- + "|".join(state.config.watched_entities)
- + "::" + "|".join(state.config.charter.scope)
- + "::" + "|".join(g.key_result for g in state.goals)
- )
- except Exception:
- logger.exception("research.scheduler: departments load failed")
- parts.append("DEPARTMENTS_ERROR")
+def _append_part(parts: list[str], label: str, lines: Callable[[], list[str]]) -> None:
+ """Append one labelled fingerprint section, or a ``<LABEL>_ERROR``
+ sentinel when loading it fails (logged; the hash still differs from a
+ healthy run so the failure is visible in the audit trail)."""
+ parts.append(f"{label}:")
try:
- from openexecutive.monitoring.research import watch_policy
-
- vocabulary = watch_policy.grounding_vocabulary(profile, [], [], departments)
- parts.append("DECISIONS:")
- for decision in watch_policy.recent_decisions(db_path=db_path):
- summary = str(getattr(decision, "summary", "") or "")
- if any(watch_policy.named_in_decision(term, decision) for term in vocabulary):
- parts.append(f"{getattr(decision, 'id', '')}::{summary[:120]}")
+ parts.extend(lines())
except Exception:
- logger.exception("research.scheduler: decisions load failed")
- parts.append("DECISIONS_ERROR")
-
- digest = hashlib.sha256("\n".join(parts).encode()).hexdigest()
- return digest
+ logger.exception("research.scheduler: %s load failed", label.lower())
+ parts.append(f"{label}_ERROR")
def _last_successful_research_run() -> tuple[str | None, str | None]:
diff --git a/packages/core/openexecutive/monitoring/research/watch_policy.py b/packages/core/openexecutive/monitoring/research/watch_policy.py
index afa595dd..b93481ff 100644
--- a/packages/core/openexecutive/monitoring/research/watch_policy.py
+++ b/packages/core/openexecutive/monitoring/research/watch_policy.py
@@ -127,11 +127,14 @@ GROUNDING_KINDS: frozenset[str] = frozenset({
STRONG_GROUNDING_KINDS: frozenset[str] = frozenset({
KIND_COMPANY, KIND_COMPETITOR, KIND_VENDOR, KIND_TICKER, KIND_DEPARTMENT_ENTITY,
})
-# When one term appears in several sources the strongest kind wins.
+# When one term appears in several sources the strongest kind wins. Profile
+# kinds outrank department kinds on purpose: a vendor the profile names and
+# Finance also lists stays a `vendor` (so the policy's vendor history keeps
+# applying to it) and merely gains Finance as its department.
_KIND_RANK: dict[str, int] = {
- KIND_COMPANY: 9, KIND_COMPETITOR: 8, KIND_DEPARTMENT_ENTITY: 7, KIND_VENDOR: 6,
- KIND_TICKER: 5, KIND_INITIATIVE: 4, KIND_PRIORITY: 3, KIND_DEPARTMENT_SCOPE: 2,
- KIND_WATCH: 1,
+ KIND_COMPANY: 9, KIND_COMPETITOR: 8, KIND_VENDOR: 7, KIND_TICKER: 6,
+ KIND_DEPARTMENT_ENTITY: 5, KIND_INITIATIVE: 4, KIND_PRIORITY: 3,
+ KIND_DEPARTMENT_SCOPE: 2, KIND_WATCH: 1,
}
# Recent decisions are consulted this far back, and only this many.
DECISION_LOOKBACK_DAYS = 90
@@ -302,8 +305,11 @@ def grounding_vocabulary(
``department_entity`` term and each charter-scope phrase / goal key
result a ``department_scope`` term, all tagged with the department's
slug. When a term is both a profile kind and a department's, the
- stronger kind wins but the department is kept — a profile competitor
- that Finance also lists stays a ``competitor`` and routes to Finance."""
+ stronger kind wins and the department is kept — a profile competitor
+ that Finance also lists stays a ``competitor`` and routes to Finance.
+ When two departments claim one term, the department that supplied the
+ winning kind owns it (a watched entity beats a scope phrase); on a tie
+ the first one seen keeps it."""
vocab: Vocabulary = {}
def add(term: str, kind: str, department: str = "") -> None:
@@ -314,8 +320,10 @@ def grounding_vocabulary(
if current is None:
vocab[n] = VocabEntry(kind, department)
return
- best_kind = kind if _KIND_RANK[kind] > _KIND_RANK[current.kind] else current.kind
- vocab[n] = VocabEntry(best_kind, current.department or department)
+ if _KIND_RANK[kind] > _KIND_RANK[current.kind]:
+ vocab[n] = VocabEntry(kind, department or current.department)
+ else:
+ vocab[n] = VocabEntry(current.kind, current.department or department)
for item in existing:
for label_key in ("display_name", "feed_label", "vendor_label", "label"):
@@ -354,6 +362,18 @@ def grounding_vocabulary(
return vocab
+def load_departments(db_path: Path | None = None) -> list[Any]:
+ """Every ``DepartmentState`` (watched entities, charter, goals, head),
+ or ``[]`` when the departments store is unavailable. Never raises."""
+ try:
+ from openexecutive.departments.store import list_departments
+
+ return list(list_departments(db_path))
+ except Exception:
+ logger.exception("watch_policy: list_departments failed")
+ return []
+
+
def recent_decisions(
now: datetime | None = None, *, db_path: Path | None = None,
) -> list[Any]:
@@ -371,7 +391,9 @@ def recent_decisions(
out: list[Any] = []
for row in rows:
ts = _parse(str(getattr(row, "timestamp", "") or ""))
- if ts is not None and ts < cutoff:
+ # A row whose timestamp cannot be read must not count as "recent"
+ # forever; it is simply not recent.
+ if ts is None or ts < cutoff:
continue
out.append(row)
return out
@@ -468,8 +490,10 @@ def match_vocab(entity: str, vocabulary: dict[str, Any]) -> tuple[str, VocabEntr
def named_in_decision(entity_term: str, decision: Any) -> bool:
- """Does a recent episodic decision name this entity? Whole-term
- containment, or a distinctive token of the term as a whole word."""
+ """Does a recent episodic decision name this entity? The whole term as
+ a phrase, or EVERY distinctive token of the term as a whole word — so
+ "Acme Corp" is named by "Acme pricing review" (corp is a stopword) but
+ "Acme Security" is not named by "Review the security policy"."""
text = _norm(str(getattr(decision, "summary", "") or ""))
if not entity_term or not text:
return False
@@ -478,7 +502,19 @@ def named_in_decision(entity_term: str, decision: Any) -> bool:
entity_tokens = {
t for t in entity_term.split() if len(t) >= _MIN_PARTIAL_TOKEN and t not in _STOPWORDS
}
- return bool(entity_tokens & set(text.split()))
+ return bool(entity_tokens) and entity_tokens <= set(text.split())
+
+
+def _decision_bonus(entity_term: str, ctx: PolicyContext) -> tuple[int, str, str]:
+ """``(points, reason, department hint)`` from the recent decisions: +1
+ when one names the entity; the department it was recorded for (when
+ the policy knows that department) is offered as a routing hint."""
+ for recent in ctx.recent_decisions:
+ if not named_in_decision(entity_term, recent):
+ continue
+ decided_for = str(getattr(recent, "department", "") or "")
+ return 1, "named in a recent decision", decided_for if decided_for in ctx.departments else ""
+ return 0, "", ""
def entity_declined(entity: str, declines: list[Any]) -> bool:
@@ -648,18 +684,16 @@ def classify(
else:
reasons.append(f"entity '{proposal.grounding_entity}' is not in company data")
- if entity_term:
- for recent in ctx.recent_decisions:
- if not named_in_decision(entity_term, recent):
- continue
- score += 1
- reasons.append("named in a recent decision")
+ if grounded:
+ # Only a company-data entity earns the decision point; a bare watch
+ # label must not pick up a department from a decision.
+ bonus, why, hint = _decision_bonus(entity_term, ctx)
+ if bonus:
+ score += bonus
+ reasons.append(why)
# A decision made for a department is a routing hint when the
# term itself has none.
- decided_for = str(getattr(recent, "department", "") or "")
- if not department and decided_for in ctx.departments:
- department = decided_for
- break
+ department = department or hint
own_source = bool(entity_term) and _is_own_source(proposal, entity_term, ctx.vocabulary)
if own_source:
@@ -1132,10 +1166,11 @@ def _auto_disable(now: datetime, db_path: Path | None) -> int:
def _nudge_if_piled_up(now: datetime, db_path: Path | None) -> int:
- # Department-routed suggestions were put in front of their head when
- # they were filed; the pile-up nudge is for the principal's own.
+ # Suggestions routed to a department HEAD were put in front of that head
+ # when they were filed; everything else (the principal's own, and rows
+ # for a department without a head) is the principal's pile.
pending = [
- i for i in ms.list_pending_suggestions(db_path=db_path) if not i.route_to_department
+ i for i in ms.list_pending_suggestions(db_path=db_path) if i.route_to_person_id is None
]
if len(pending) < _NUDGE_MIN_PENDING:
return 0
@@ -1206,6 +1241,7 @@ __all__ = [
"department_refs",
"entity_declined",
"grounding_vocabulary",
+ "load_departments",
"match_entity",
"match_vocab",
"named_in_decision",
diff --git a/packages/core/openexecutive/workflows/executive_research.py b/packages/core/openexecutive/workflows/executive_research.py
index 540369a0..b5dafe64 100644
--- a/packages/core/openexecutive/workflows/executive_research.py
+++ b/packages/core/openexecutive/workflows/executive_research.py
@@ -406,15 +406,12 @@ class ExecutiveResearchWorkflow(Workflow):
# Departments (watched entities, scope, goals) and recent decisions
# are current company intent the static profile lacks; both feed
# the watch policy's grounding and routing.
- try:
- from openexecutive.departments.store import list_departments
-
- departments = list(list_departments())
- except Exception:
- logger.exception("research: list_departments failed")
- departments = []
- from openexecutive.monitoring.research.watch_policy import recent_decisions
+ from openexecutive.monitoring.research.watch_policy import (
+ load_departments,
+ recent_decisions,
+ )
+ departments = load_departments()
decisions = recent_decisions()
research_context = _render_research_context(
diff --git a/packages/core/tests/unit/test_executive_research_scheduler.py b/packages/core/tests/unit/test_executive_research_scheduler.py
index f8b370e1..2f1ecd27 100644
--- a/packages/core/tests/unit/test_executive_research_scheduler.py
+++ b/packages/core/tests/unit/test_executive_research_scheduler.py
@@ -101,7 +101,15 @@ def test_state_hash_moves_only_for_decisions_that_name_known_entities(
after = research_scheduler.compute_research_state_hash(db_path=db)
assert after != before
store_decision("finance", "Move Stripe to annual billing", db_path=db)
- assert research_scheduler.compute_research_state_hash(db_path=db) not in (before, after)
+ third = research_scheduler.compute_research_state_hash(db_path=db)
+ assert third not in (before, after)
+ # An initiative title is vocabulary too.
+ from openexecutive.memory.episodic import store_initiative
+
+ store_initiative("Project Halo", "active", db_path=db)
+ with_initiative = research_scheduler.compute_research_state_hash(db_path=db)
+ store_decision("ops", "Pause the Project Halo rollout", db_path=db)
+ assert research_scheduler.compute_research_state_hash(db_path=db) != with_initiative
def test_state_hash_changes_when_initiative_added(db: Path) -> None:
diff --git a/packages/core/tests/unit/test_watch_policy.py b/packages/core/tests/unit/test_watch_policy.py
index 79ce5ea1..6e059636 100644
--- a/packages/core/tests/unit/test_watch_policy.py
+++ b/packages/core/tests/unit/test_watch_policy.py
@@ -717,8 +717,16 @@ def test_department_vocabulary_kinds_and_department_merge() -> None:
assert vocab["expense card programs"] == wp.VocabEntry(wp.KIND_DEPARTMENT_SCOPE, "finance")
assert vocab["close the books in 5 days"].kind == wp.KIND_DEPARTMENT_SCOPE
# A profile competitor that Finance also lists keeps the stronger kind
- # and still routes to Finance.
+ # and still routes to Finance; so does a profile vendor (profile kinds
+ # outrank department kinds, so the vendor history keeps applying).
assert vocab["acme corp"] == wp.VocabEntry(wp.KIND_COMPETITOR, "finance")
+ stripe_vocab = wp.grounding_vocabulary(_profile(), [], [], [_department("finance", "Finance", watched=["Stripe"])])
+ assert stripe_vocab["stripe"] == wp.VocabEntry(wp.KIND_VENDOR, "finance")
+ # Two departments: the one supplying the winning kind owns the term.
+ eng = _department("eng", "Engineering", scope=["Acme Migration"])
+ fin = _department("finance", "Finance", watched=["Acme Migration"])
+ both = wp.grounding_vocabulary(None, [], [], [eng, fin])
+ assert both["acme migration"] == wp.VocabEntry(wp.KIND_DEPARTMENT_ENTITY, "finance")
# Legacy plain-kind dicts still match.
assert wp.match_entity("Brex Inc", vocab) == ("brex", wp.KIND_DEPARTMENT_ENTITY)
assert wp.match_entity("Brex", {"brex": wp.KIND_VENDOR}) == ("brex", wp.KIND_VENDOR)
@@ -778,7 +786,21 @@ def test_recent_decision_adds_a_point_and_a_routing_hint() -> None:
unrelated = SimpleNamespace(summary="Hire two SDRs in Q4", department="finance", timestamp="")
assert wp.classify(p, f, _dept_ctx(recent_decisions=[unrelated])).score == base.score
assert wp.named_in_decision("acme corp", SimpleNamespace(summary="Acme Corp pricing review"))
+ assert wp.named_in_decision("acme corp", SimpleNamespace(summary="Acme pricing review"))
assert not wp.named_in_decision("acme corp", SimpleNamespace(summary="The corp retreat"))
+ # Every distinctive token must appear: a common word shared with the
+ # term is not a mention, and a watch label never earns the point.
+ assert not wp.named_in_decision("acme security", SimpleNamespace(summary="Review the security policy"))
+ hijack = SimpleNamespace(summary="Adopt the new pricing policy", department="finance", timestamp="")
+ label_only = _proposal(slug="rss-hubs", signal_type="rss", target="https://hubspot.com/feed",
+ grounding_entity="HubSpot pricing")
+ ctx = _dept_ctx(existing=[SimpleNamespace(
+ slug="stock-hubs", signal_type="stock", target="HUBS", enabled=True,
+ config_json={"display_name": "HubSpot pricing"},
+ )], recent_decisions=[hijack])
+ labelled = wp.classify(label_only, _finding(), ctx)
+ assert labelled.grounding_kind == wp.KIND_WATCH and labelled.department == ""
+ assert "named in a recent decision" not in labelled.reasons
def test_recent_decisions_apply_the_lookback(db: Path) -> None:
@@ -793,6 +815,10 @@ def test_recent_decisions_apply_the_lookback(db: Path) -> None:
((datetime.now(UTC) - timedelta(days=120)).isoformat(),))
rows = wp.recent_decisions(db_path=db)
assert [r.summary for r in rows] == ["Evaluate Brex for expense cards"]
+ # An unreadable timestamp is not "recent forever".
+ with sqlite3.connect(db) as conn:
+ conn.execute("UPDATE decisions SET timestamp = 'garbage'")
+ assert wp.recent_decisions(db_path=db) == []
def test_department_head_gets_one_card_per_run_coalesced_weekly(db: Path) -> None:
@@ -843,15 +869,25 @@ def test_department_card_falls_back_to_the_principal(db: Path) -> None:
assert row is not None and row.route_to_department == "finance" and row.route_to_person_id is None
-def test_nudge_counts_only_the_principals_suggestions(db: Path) -> None:
+def test_nudge_counts_only_suggestions_no_head_received(db: Path) -> None:
from openexecutive.alerts.store import list_alerts
for i in range(3):
ms.insert_watchlist_item(
slug=f"rss-d{i}", signal_type="rss", target=f"https://d{i}.com/feed",
mode=MODE_DRY_RUN, origin=ORIGIN_RESEARCH_PROPOSED, route_to_department="finance",
- db_path=db,
+ route_to_person_id=7, db_path=db,
)
_backdate(db, f"rss-d{i}", 8)
assert wp.sweep(datetime.now(UTC), db_path=db)["nudged"] == 0
assert not [a for a in list_alerts(limit=10, db_path=db) if a.source == wp.NUDGE_ALERT_SOURCE]
+ # A department without a head is still the principal's pile: those rows
+ # count, so they can never become invisible.
+ for i in range(3):
+ ms.insert_watchlist_item(
+ slug=f"rss-n{i}", signal_type="rss", target=f"https://n{i}.com/feed",
+ mode=MODE_DRY_RUN, origin=ORIGIN_RESEARCH_PROPOSED, route_to_department="ops",
+ db_path=db,
+ )
+ _backdate(db, f"rss-n{i}", 8)
+ assert wp.sweep(datetime.now(UTC), db_path=db)["nudged"] == 1