AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
refactor(research): remove the xcrawl scrape service and finding verification
Rufusf16136476cd6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a69cb208..f4525871 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -42,7 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0policy (`monitoring.research.watch_policy`) decides from company data. Aproposal tied to a named competitor, vendor, ticker, initiative orpriority (the profile gains `vendors` and `tickers` for this) and- corroborated (own source, high-confidence / verified finding, consensus,+ corroborated (own source, high-confidence finding, consensus,the policy's own track record) is added on its own — quietly: dailycadence, medium severity floor, a keyword trigger for feeds, 5 % forstocks — and shows up in the brief's "handled overnight" block as@@ -103,6 +103,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0bumps `dismiss_count`); approving recovers it. Trust now discounts theranking and is shown to the review as evidence.+### Removed+- **The xcrawl scrape service and everything that depended on it.** The+ research run no longer has a post-dedup verify pass or an optional+ read-before-cite scrape loop, and the watch policy no longer scores a+ "verified" point; the `XCRAWL_*`, `EXTERNAL_RESEARCH_VERIFY_*`,+ `RESEARCH_AGENTIC_*`, `RESEARCH_SCRAPE_*` and `RESEARCH_LOOP_*` settings+ are gone (a deployment that still sets them is unaffected). Every+ watchlist fetch is the keyless, SSRF-guarded httpx fetcher: a page watch+ written under the old `fetch: xcrawl` marker re-captures its baseline on+ the next poll without reporting a change, and an `rss` target that is not+ a feed becomes a `page_watch` when the page already fetched by the feed+ check has readable text (otherwise it is rejected, as before).+### Changed- The research routing pass no longer carries the watchlist write tools(`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry`):diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yamlindex 0be5fb27..c2d39d1f 100644--- a/packages/core/openexecutive/architecture/architecture-facts.yaml+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml@@ -171,7 +171,7 @@ providers:``claude-haiku-4-5`` and legacy ``claude-3-5-haiku-…`` orderings; scopedto Claude names so an unrelated slug containing "haiku" is unaffected)and the agent call paths (BaseAgent.analyze,- analyze_with_tools, monitoring.research.agentic) skip the thinking fields+ analyze_with_tools) skip the thinking fieldsfor it; the Council UI disables the checkbox for the same models, clearsit when the model is switched to Haiku, masks a stale storeddeep_reasoning=true on load (so the next Save corrects the row), and@@ -194,12 +194,10 @@ providers:carried as a synthetic ``openrouter_reasoning`` content block (alwaysLAST in ``content`` so ``content[0].text`` readers are unaffected) andreplayed verbatim as ``reasoning_details`` on the assistant message by- the request translator. The one tool loop that runs with deep reasoning- on — monitoring.research.agentic._assistant_turn — echoes the block, so- it keeps reasoning continuity (the OpenRouter-path equivalent of echoing- Anthropic thinking blocks). The Executive / executive_research /+ the request translator. The Executive / executive_research /executive_reflection loops never enable thinking and drop the block- harmlessly; wire the echo in if deep reasoning is ever added to them.+ harmlessly (each consults `reasoning_replay_block`, so the echo is+ already wired should deep reasoning ever be added to them).The Anthropicserver-side web_search tool (no input_schema) cannot be executed byOpenRouter, so its intent is translated into OpenRouter's@@ -1444,10 +1442,10 @@ external_monitoring:entity's NAME: its leading distinctive word or all its words runtogether, never a generic trailing word ("Acme Payments" is notpayments.io) / a primary-source host: sec.gov, federalregister.gov,- eur-lex), +1 finding confidence high, +1 verification confirmed, +1+ eur-lex), +1 finding confidence high, +1cross-specialist consensus, ±1 policy history for (signal_type,grounding_kind) at >= 5 samples (>= 0.7 / <= 0.3 approval rate).- - finding points (high / verified / consensus) count only when the cited+ - finding points (high / consensus) count only when the citedfinding concerns THIS source: a URL target shares its site with a citedURL; a ticker/query target appears in the finding text (as a phrase —"1211.HK" is two words once normalized) or is the named entity. A strong@@ -1459,7 +1457,7 @@ external_monitoring:concerns the source by the same test the score uses(watch_policy.auto_link_finding: with the entity term classify woulduse; ranked by citing the target's site / naming the ticker, then- citing the entity's own site, confidence, verification, order) and+ citing the entity's own site, confidence, order) andrecords "linked to finding #N, which concerns this source" as thefirst reason. The caller's proposal is not mutated. A proposal nothingcites is still rejected as having no evidence.@@ -1701,7 +1699,7 @@ external_monitoring:- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."- "edgar — SEC EDGAR filings. Watchlist `target` is a ticker (e.g. 'AAPL') or CIK; polls SEC's browse-edgar Atom feed (`/cgi-bin/browse-edgar?action=getcompany&CIK=…&output=atom`, whose CIK param resolves a ticker server-side, so no ticker→CIK table is shipped), parsed with feedparser. Emits one Signal per new filing whose form is in the filter (`trigger.forms`, default {8-K, 10-K, 10-Q}; amendments like 8-K/A match their base form). Dedup on the filing's accession number (globally unique — the cleanest natural key of any source). Provenance is the filing-index URL. Severity hint: 8-K HIGH (material-event report), 10-K/10-Q MEDIUM, else LOW. Keyless, but SEC's fair-access policy requires a descriptive User-Agent with a contact — sent via the EDGAR_USER_AGENT setting (operators should override the default with a real contact). Reuses fetch_bounded (now with a per-call user_agent override), validate_target_url, and the standard dedup/Signal pattern. The entry's <updated> lands on Signal.published_at and the adapter declares `seed_on_first_poll`, so a new watch baselines past filings instead of replaying them and filings older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are `suppressed_stale`."- - "page_watch — generic web-page change detection for feed-less pages (pricing, careers, leadership, ToS). Watchlist `target` is the page URL. STATEFUL (the first stateful adapter): each poll fetches the page (bounded, SSRF-guarded), reduces it to normalized visible text (drops script/style/comments/tags, unescapes entities, collapses whitespace — so markup/layout churn doesn't register), hashes the FULL normalized text (so a change anywhere registers — no length blind spot), and compares to the last-seen hash in `page_watch_state`. First observation stores a baseline and emits NOTHING; an unchanged page emits nothing; a changed page emits ONE Signal carrying a difflib-based change summary (~N% differs + the added text; the diff/snapshot are length-capped but detection is not) and updates the baseline. Severity hint LOW (page diffs are noisy — enrichment + severity_floor filter). Dedup key folds slug + UTC date + content hash (mirrors stock — a state seen again on a later day re-alerts; same-day repeats dedup). Optional `trigger.keywords` only surfaces a change when the change itself (summary + added text) matches. Reads/writes its state via the db_path threaded through Source.poll. Optional `config.fetch = \"xcrawl\"` routes the fetch through xcrawl's scrape API (clean markdown) instead of the keyless httpx fetcher — so JS-rendered / bot-blocked feed-less pages (most modern news/marketing sites) are monitorable; off by default, a no-op when xcrawl is disabled, and the scraped markdown is size-capped to mirror the httpx byte cap."+ - "page_watch — generic web-page change detection for feed-less pages (pricing, careers, leadership, ToS). Watchlist `target` is the page URL. STATEFUL (the first stateful adapter): each poll fetches the page (bounded, SSRF-guarded), reduces it to normalized visible text (drops script/style/comments/tags, unescapes entities, collapses whitespace — so markup/layout churn doesn't register), hashes the FULL normalized text (so a change anywhere registers — no length blind spot), and compares to the last-seen hash in `page_watch_state`. First observation stores a baseline and emits NOTHING; an unchanged page emits nothing; a changed page emits ONE Signal carrying a difflib-based change summary (~N% differs + the added text; the diff/snapshot are length-capped but detection is not) and updates the baseline. Severity hint LOW (page diffs are noisy — enrichment + severity_floor filter). Dedup key folds slug + UTC date + content hash (mirrors stock — a state seen again on a later day re-alerts; same-day repeats dedup). Optional `trigger.keywords` only surfaces a change when the change itself (summary + added text) matches. Reads/writes its state via the db_path threaded through Source.poll. The keyless httpx fetcher is the only fetch path. A row written before the xcrawl scrape path was removed still carries `config.fetch = \"xcrawl\"` (PAGE_WATCH_FETCH_LEGACY_XCRAWL) and a baseline hashed from scraped markdown; the adapter treats that marker as a stale baseline — the next poll re-captures the baseline from the plain fetch WITHOUT emitting a signal and clears the marker in place (store.remove_watchlist_config_key, a SQL json_remove so a concurrent edit to another config key is kept), so the change of fetcher is never reported as a page change. An unsaved item (no id) cannot have its marker cleared and is compared normally."insert_validation: |monitoring.target_validation.validate_and_normalize_target — runs atinsert time on BOTH watchlist surfaces (POST /watchlist and the@@ -1711,30 +1709,32 @@ external_monitoring:- recognized as a feed (parsed.version set, incl. an empty/new feed)→ kept as rss;- a public server answered but it is not a feed → CONVERTED to a- scrape-backed page_watch (config.fetch=xcrawl) when xcrawl.scrape- returns content, else REJECTED (WatchlistTargetError → HTTP 422 /- tool error);+ page_watch when the body the feed check already fetched reduces to+ readable text (page_watch.html_to_text — same SSRF guard and byte+ cap the adapter itself uses; no second fetch, no fetch marker+ written), else REJECTED (WatchlistTargetError → HTTP 422 / tool+ error). An oversized body or an HTTP error status has no body to+ inspect and is rejected;- SSRF-rejected / unparseable URL → 'blocked' → rejected outright,- never handed to xcrawl (so xcrawl can't be used to reach a target- the on-box SSRF guard refused);+ never fetched;- transient fetch failure → inserted unverified (don't penalize afeed that's merely down right now; the scan loop retries).Exists because executive_research proposes feed URLs the model neververified — historically the great majority were dead, which is whatleft the briefing's monitoring section empty.- xcrawl_integration: |- integrations.xcrawl_client — async client over the xcrawl web-data API- (search / scrape). The watchlist uses scrape to read pages the keyless- httpx fetcher cannot (JS-rendered / bot-blocked, no RSS). Settings-gated- (XCRAWL_ENABLED + XCRAWL_API_KEY, OFF by default), timeout-bounded,- graceful-degradation (every failure returns []/None, never raises into- the scan loop), one `xcrawl` audit row per call. No 'required when- enabled' validator — a flag set without a key degrades to disabled- rather than crash-looping boot. NOTE: web SEARCH for the research- council is NOT this client — that runs through the provider web_search- path (Anthropic native / OpenRouter web plugin), which already works;- xcrawl is the watchlist's fetch-and-validate layer, not a research- grounding layer.+ no_external_fetch_service: |+ There is no third-party scrape / render service. The xcrawl client, the+ read-before-cite scrape loop (`monitoring/research/agentic.py`), the+ post-dedup verify pass (`monitoring/research/verification.py`) and the+ `XCRAWL_*`, `EXTERNAL_RESEARCH_VERIFY_*`, `RESEARCH_AGENTIC_*`,+ `RESEARCH_SCRAPE_*` and `RESEARCH_LOOP_*` settings were removed: the+ service had been failing on every call for weeks with nothing noticing+ (every failure degraded silently to "unverified"), and it was the only+ outbound fetch that bypassed the on-box SSRF guard. Every watchlist+ fetch is now the keyless, SSRF-guarded, byte-capped httpx fetcher, and+ research grounding is the provider web_search path (Anthropic native /+ OpenRouter web plugin). A deployment that still sets the removed+ variables is unaffected (unknown env vars are ignored).pipeline: |openexecutive.monitoring.pipeline — `run_external_monitor_scan(now)`is the per-tick entry point. It walks@@ -1916,51 +1916,17 @@ external_monitoring:specialist (agent_overrides.research_focus, seeper_specialist_research_focus); the shared contract is fixed code.Specialists DO NOT route — they surface what their domain- turned up.- READ-BEFORE-CITE (gated): when `RESEARCH_AGENTIC_SCRAPE_ENABLED`- (+ `XCRAWL_ENABLED`), `research_one_specialist` instead runs a- bounded tool-use LOOP (`monitoring/research/agentic.py`): web_search- to find candidates → `scrape_url` (xcrawl) to READ the full article- behind the best 1-3 hits → ground claims in real content →- emit_research_findings. Built because the single-shot snippet-only- turn produced findings whose cited sources didn't back them (the- verify pass measured ~all demoted). Scraping is MANDATORY, not- advisory: the model emitted on turn 1 from web_search snippets and- skipped the optional scrape (an A/B showed 0 research-phase scrapes,- loop OFF==ON), so the loop now REJECTS an `emit_research_findings`- that arrives with `scrapes_used == 0` — it returns a tool_result on- the emit ("scrape_url at least one source first") and forces another- turn. The reject only fires while a read is still possible- (`max_scrapes > 0` AND iterations remain); on the last iteration, or- when scraping is disabled, a 0-scrape emit is accepted as the safety- valve. Server web_search blocks are- echoed forward; only the client `scrape_url` tool is executed- (mirrors the Executive's main loop). Bounded by- `RESEARCH_SCRAPE_MAX_PER_SPECIALIST` (default 3) +- `RESEARCH_LOOP_MAX_ITERATIONS` (default 5); best-effort (provider/- scrape failure → empty, never sinks the run). Off by default →- the single-shot path above is unchanged. The step 3b verify pass- remains the safety net.+ turned up. The specialist turn is single-shot: there is no+ scrape loop and no separate verify pass any more (see+ no_external_fetch_service); the finding contract's "cite a dated+ source you retrieved" rule is enforced by the prompt and the+ low-confidence filter below.3. dedup — light cross-specialist collapse on(lowercased title, summary[:80]). Consensus boosts severity +confidence; source_specialist becomes a comma-joined attribution.- 3b. verify (gated, `research.verification.verify_findings`) — runs- BEFORE the low-confidence filter. For each surviving finding it- scrapes the first cited URL via xcrawl and asks a cheap model- (RESEARCH_VERIFY_MODEL, default haiku) whether the page supports the- claim: confirmed → keep; contradicted → confidence=low; unsupported /- source_unreachable (dead URL) → demote one notch. So a finding whose- source is dead or doesn't back it is dropped by the existing- low-confidence filter instead of routed to a human. Rationale:- web_search gives specialists DISCOVERY (snippets — thin under the- OpenRouter web plugin), but the finding contract demands verifying- numbers/dates against a retrieved source; this is the deep-read it- assumes. Gated by `XCRAWL_ENABLED AND EXTERNAL_RESEARCH_VERIFY_ENABLED`- (no-op + zero network otherwise); best-effort (a scrape/LLM failure- leaves the finding unchanged); bounded to RESEARCH_VERIFY_MAX_FINDINGS- per run (default 8), highest-severity first. The verdict is stamped on- `ResearchFinding.verification`. The low-confidence filter- (drop confidence=='low') runs AFTER this pass.+ The low-confidence filter (drop confidence=='low') runs here; the+ step's summary reports after_dedup / low_conf_dropped /+ to_synthesis.4. executive_synthesis — Executive's tool-use loop (mirror ofexecutive_reflection._run_executive_reflection):system prompt built per-run by `_build_synthesis_system`,diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.jsonindex 10f0bdfe..d4b4a359 100644--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json@@ -1,7 +1,7 @@{"section_id": "external_monitoring","title": "External Monitoring",- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page. Fetches via the keyless httpx fetcher by default, or via xcrawl's scrape API when the row carries `config.fetch = \"xcrawl\"` — so JS-rendered / bot-blocked pages that publish no RSS (most modern news / marketing sites) are still monitorable.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a scrape-backed `page_watch` when xcrawl can read it, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never handed to xcrawl. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (each list entry is parsed to its name: \"Tesla (TSLA) — Model Y is the benchmark\" grounds as `Tesla` plus the ticker `TSLA`, \"GM / Chevrolet (…)\" as both names, and \"Brex (brex.com)\" pins Brex's own domains so only those sites count as its own source; the description after the dash never becomes a term, short names such as BYD or GM match as whole words, and a ticker grounds only as itself) (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's leading name or all its words run together, never a generic trailing word; or a primary-source host such as sec.gov), +1 finding confidence high, +1 verification confirmed, +1 cross-specialist consensus (the three finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. It also always needs a linked finding that cites the source — company data plus a decision mention can reach the score bar alone, but nothing goes live on hearsay. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Linked finding.** `finding_index` is required on `propose_watch` and the pass is told so; when the model omits it anyway, or cites a finding that does not concern the source, the policy links the finding that best does (a cited URL on the target's site or the ticker named in its text outranks a mere mention of the entity; then the entity's own site, confidence, verification) and says so in the first reason. A proposal that no finding cites is rejected as having no evidence.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `department=<slug>` — a tag hint, deliberately not the `channel` field triage reads as a team room), so the alerts it raises queue on the head's `/today` (as action items, since routed alerts are never hidden in Monitoring) and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Fetches are SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`; xcrawl scrape (off by default, gated by `XCRAWL_ENABLED` + `XCRAWL_API_KEY`) is the one path that delegates the fetch to an external API.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",- "mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / convert→page_watch(xcrawl) / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx | xcrawl)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",- "generated_at": "2026-09-12T12:03:38.574978+00:00"+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page, fetched with the keyless httpx fetcher. A row written before the external scrape path was removed still carries `config.fetch = \"xcrawl\"`; the adapter treats that as a stale baseline, re-captures it from the plain fetch without emitting a signal, and clears the marker.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a `page_watch` when the body the feed check already fetched reduces to readable text, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never fetched. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (each list entry is parsed to its name: \"Tesla (TSLA) — Model Y is the benchmark\" grounds as `Tesla` plus the ticker `TSLA`, \"GM / Chevrolet (…)\" as both names, and \"Brex (brex.com)\" pins Brex's own domains so only those sites count as its own source; the description after the dash never becomes a term, short names such as BYD or GM match as whole words, and a ticker grounds only as itself) (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's leading name or all its words run together, never a generic trailing word; or a primary-source host such as sec.gov), +1 finding confidence high, +1 cross-specialist consensus (the finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. It also always needs a linked finding that cites the source — company data plus a decision mention can reach the score bar alone, but nothing goes live on hearsay. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Linked finding.** `finding_index` is required on `propose_watch` and the pass is told so; when the model omits it anyway, or cites a finding that does not concern the source, the policy links the finding that best does (a cited URL on the target's site or the ticker named in its text outranks a mere mention of the entity; then the entity's own site, confidence) and says so in the first reason. A proposal that no finding cites is rejected as having no evidence.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `department=<slug>` — a tag hint, deliberately not the `channel` field triage reads as a team room), so the alerts it raises queue on the head's `/today` (as action items, since routed alerts are never hidden in Monitoring) and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Every fetch is the on-box httpx fetcher: SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`. No path delegates a fetch to an external service.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",+ "mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / readable page → page_watch / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",+ "generated_at": "2026-09-12T17:10:00+00:00"}diff --git a/packages/core/openexecutive/config.py b/packages/core/openexecutive/config.pyindex db352036..adeed577 100644--- a/packages/core/openexecutive/config.py+++ b/packages/core/openexecutive/config.py@@ -466,66 +466,6 @@ class Settings(BaseSettings):default_factory=list, alias="WEB_SEARCH_BLOCKED_DOMAINS")- # ── xcrawl (external scrape / SERP API) ───────────────────────────────- # Off by default. xcrawl's scrape API reads JS-rendered / bot-blocked- # pages the keyless RSS adapter cannot, so the watchlist can monitor- # sources that have NO usable RSS feed (modern marketing / news SPAs)- # via scrape-backed change detection, and validate / repair a feed- # target at insert time so dead rows never reach the scan loop.- #- # No "required when enabled" validator: a flag set without a key- # degrades to disabled in the client rather than crash-looping boot.- xcrawl_enabled: bool = Field(False, alias="XCRAWL_ENABLED")- xcrawl_api_key: str | None = Field(None, alias="XCRAWL_API_KEY")- xcrawl_base_url: str = Field(- "https://run.xcrawl.com/v1", alias="XCRAWL_BASE_URL"- )- xcrawl_timeout_s: float = Field(30.0, alias="XCRAWL_TIMEOUT_S")-- # ── research finding verification (xcrawl scrape deep-read) ───────────- # When on (also requires xcrawl_enabled), the executive_research workflow- # runs a post-dedup pass that scrapes each surviving finding's cited URL- # and asks a cheap model whether the page actually supports the claim —- # demoting / dropping findings whose source is dead or doesn't back them,- # before the Executive routes anything. web_search gives specialists- # discovery (snippets); this gives them the deep-read the finding contract- # already assumes. Off by default; a no-op when xcrawl is disabled.- external_research_verify_enabled: bool = Field(- False, alias="EXTERNAL_RESEARCH_VERIFY_ENABLED"- )- # Cheap model for the per-finding verify call (read scraped page → verdict).- research_verify_model: str = Field(- "claude-haiku-4-5", alias="RESEARCH_VERIFY_MODEL"- )- # Hard cap on findings verified per research run (each = 1 scrape + 1 cheap- # LLM call). Bounds added cost; verified in severity order.- research_verify_max_findings: int = Field(- 8, alias="RESEARCH_VERIFY_MAX_FINDINGS"- )-- # ── agentic research (read-before-cite scrape loop) ───────────────────- # When on (also requires xcrawl_enabled), each executive_research- # specialist runs a bounded search→scrape_url→emit tool-use loop: it- # reads the FULL article behind its best web_search hits (xcrawl scrape)- # and grounds claims in real content, instead of the single-shot- # snippet-only call that produced findings whose cited sources didn't- # back them. Off by default (single-shot path unchanged); the verify- # pass remains the safety net.- research_agentic_scrape_enabled: bool = Field(- False, alias="RESEARCH_AGENTIC_SCRAPE_ENABLED"- )- # Max scrape_url calls a specialist may make per run (each = 1 xcrawl- # scrape + the page's tokens). Past this the tool refuses and tells the- # model to emit. Bounds per-run cost across the 7-specialist fan-out.- research_scrape_max_per_specialist: int = Field(- 3, alias="RESEARCH_SCRAPE_MAX_PER_SPECIALIST"- )- # Hard ceiling on tool-use turns per specialist loop (search/scrape/emit).- # Guards against a specialist that never emits.- research_loop_max_iterations: int = Field(- 5, alias="RESEARCH_LOOP_MAX_ITERATIONS"- )-@field_validator("web_search_allowed_domains", "web_search_blocked_domains", mode="before")diff --git a/packages/core/openexecutive/integrations/xcrawl_client.py b/packages/core/openexecutive/integrations/xcrawl_client.pydeleted file mode 100644index 3a05bf89..00000000--- a/packages/core/openexecutive/integrations/xcrawl_client.py+++ /dev/null@@ -1,312 +0,0 @@-"""Async client for the xcrawl web-data API (search / scrape).--xcrawl turns "the LLM guessed a URL" into "real fetched content". The-watchlist uses it to monitor sources that have NO usable RSS feed —-modern JS-rendered / bot-blocked marketing + news pages the keyless RSS-adapter cannot read — via scrape-backed change detection, and to-validate / repair a feed target at insert time so dead rows never reach-the scan loop.--``search`` (SERP) is also exposed for callers that need keyword-discovery rather than a known URL.--Posture mirrors ``memory.honcho_client``: settings-gated, timeout--bounded, graceful degradation (every failure returns an empty result,-never raises into the caller), and one audit row per call. No SSRF guard-is needed here: the ``url`` argument may be caller- or LLM-supplied (e.g.-the research ``scrape_url`` tool), but it is sent as a JSON payload field-and fetched on xcrawl's network — our process only ever connects to the-configured xcrawl base URL, never to the target.-"""-from __future__ import annotations--import logging-from typing import Any, Literal, NamedTuple--import httpx-from pydantic import BaseModel--from openexecutive.audit import log_event as audit_log-from openexecutive.config import get_settings--logger = logging.getLogger(__name__)--# Max chars of the query / url recorded in an audit row's subject_preview.-_AUDIT_SUBJECT_PREVIEW_CHARS = 160-# SERP result count: default, and xcrawl's documented 1–100 ceiling.-_DEFAULT_SEARCH_LIMIT = 5-_MAX_SEARCH_LIMIT = 100---class SearchResult(BaseModel):- """One SERP row, normalized from xcrawl's response envelope."""-- title: str- url: str- snippet: str---def _resolve_credentials() -> tuple[bool, str | None]:- """Return ``(enabled, api_key)``.-- "Enabled" requires the flag AND a non-empty key. A flag set without a- key degrades to disabled (with a warning) rather than letting a- keyless request 401 on every call — and deliberately there is no- config-level "required when enabled" validator that would crash boot.- """- settings = get_settings()- if not settings.xcrawl_enabled:- return False, None- key = (settings.xcrawl_api_key or "").strip()- if not key:- logger.warning(- "xcrawl: XCRAWL_ENABLED is true but XCRAWL_API_KEY is unset — "- "treating xcrawl as disabled",- )- return False, None- return True, key---async def _post_json(endpoint: str, key: str, payload: dict[str, Any]) -> Any:- """POST ``payload`` to an xcrawl endpoint and return the parsed JSON.-- ``raise_for_status`` and ``.json()`` run INSIDE the client context so- the response body is always read before the connection closes. Raises- on any transport / HTTP-status / decode error; callers own the- op-specific logging + audit + graceful-degradation.- """- settings = get_settings()- async with httpx.AsyncClient(timeout=settings.xcrawl_timeout_s) as client:- resp = await client.post(- endpoint,- headers={- "Authorization": f"Bearer {key}",- "Content-Type": "application/json",- },- json=payload,- )- resp.raise_for_status()- return resp.json()---def _endpoint(path: str) -> str:- return f"{get_settings().xcrawl_base_url.rstrip('/')}/{path}"---async def search(- query: str, *, limit: int = _DEFAULT_SEARCH_LIMIT,-) -> list[SearchResult]:- """Run one SERP query against xcrawl ``/search``.-- Returns the ranked results, or ``[]`` when xcrawl is disabled, the- query is blank, or the call fails. Never raises — a research run must- not abort because xcrawl is unavailable.- """- query = (query or "").strip()- if not query or limit <= 0:- # limit<=0 means "no results wanted" — return without charging xcrawl.- return []- enabled, key = _resolve_credentials()- if not enabled or key is None:- return []-- # Clamp to xcrawl's documented ceiling (floor already handled above).- bounded_limit = min(limit, _MAX_SEARCH_LIMIT)- payload = {"query": query, "limit": bounded_limit}- try:- body = await _post_json(_endpoint("search"), key, payload)- except Exception as exc:- logger.warning("xcrawl: search failed for %r: %s", query[:80], exc)- _audit("search", query, ok=False, count=0, error=str(exc)[:200])- return []-- results = _parse_search(body)- _audit(- "search", query, ok=True, count=len(results),- credits=body.get("total_credits_used") if isinstance(body, dict) else None,- )- return results[:bounded_limit]---ScrapeStatus = Literal["ok", "empty", "error", "disabled"]---class ScrapeOutcome(NamedTuple):- """Result of a scrape attempt, distinguishing *why* there is no markdown.-- ``status`` lets a caller tell an infra failure apart from a page that was- fetched but yields nothing — a distinction ``scrape``'s bare ``None``- cannot make:-- - ``"ok"`` — markdown retrieved (``markdown`` is non-None)- - ``"empty"`` — fetched OK but no usable markdown (genuinely dead /- empty page — the page's fault)- - ``"error"`` — the request failed (transport / HTTP status / decode):- transient infra (xcrawl down, 401, 429, timeout), NOT- a verdict on the page. Callers that penalise a dead- source must treat this as "couldn't check", not "dead".- - ``"disabled"`` — xcrawl off / no key / blank url; no call was made.-- ``markdown`` is non-None only for ``status == "ok"``.- """-- markdown: str | None- status: ScrapeStatus---async def scrape_detailed(url: str) -> ScrapeOutcome:- """Fetch ``url`` via xcrawl ``/scrape`` and return a status-tagged result.-- xcrawl renders JS and fetches through a residential proxy, so it reads- pages the keyless httpx fetcher cannot (JS-only SPAs, soft bot blocks)- — the basis for monitoring sources that publish no RSS feed.-- Never raises — the status carries the failure mode (see ``ScrapeOutcome``).- """- url = (url or "").strip()- if not url:- return ScrapeOutcome(None, "disabled")- enabled, key = _resolve_credentials()- if not enabled or key is None:- return ScrapeOutcome(None, "disabled")-- payload = {- "url": url,- "output": {"formats": ["markdown"]},- "request": {"only_main_content": True},- }- try:- body = await _post_json(_endpoint("scrape"), key, payload)- except Exception as exc:- # Transport / HTTP-status / decode failure — xcrawl itself misbehaved.- # This is infra, not a verdict on the page.- logger.warning("xcrawl: scrape failed for %r: %s", url[:80], exc)- _audit("scrape", url, ok=False, count=0, error=str(exc)[:200])- return ScrapeOutcome(None, "error")-- markdown = _parse_scrape(body)- if markdown is not None:- # Bound the markdown to mirror the keyless path's byte cap so a- # giant page can't blow up the downstream hash / diff / storage on- # the scan worker. (chars ≈ bytes for the latin-heavy markdown we- # store; an exact byte cap isn't needed for this DoS guard.)- max_chars = get_settings().external_monitor_max_fetch_bytes- if len(markdown) > max_chars:- logger.info(- "xcrawl: scrape markdown for %r truncated %d→%d chars",- url[:80], len(markdown), max_chars,- )- markdown = markdown[:max_chars]- _audit(- "scrape", url, ok=markdown is not None,- count=1 if markdown else 0,- credits=body.get("total_credits_used") if isinstance(body, dict) else None,- )- return ScrapeOutcome(markdown, "ok" if markdown is not None else "empty")---async def scrape(url: str) -> str | None:- """Fetch ``url`` via xcrawl ``/scrape`` and return cleaned markdown.-- Returns ``None`` when xcrawl is disabled, the url is blank, the call- fails, or the page yields no markdown. Never raises — callers treat- ``None`` as "no fetch this tick". Callers that must distinguish an infra- failure from a genuinely empty/dead page use ``scrape_detailed`` instead.- """- return (await scrape_detailed(url)).markdown---def _parse_scrape(body: dict[str, Any]) -> str | None:- """Pull markdown out of xcrawl's scrape envelope: ``{"data": {"markdown": ...}}``.-- Tolerates one extra level of nesting (``data.data.markdown``) and- treats blank / whitespace-only markdown as "no content" (``None``).- """- if not isinstance(body, dict):- return None- data = body.get("data")- if not isinstance(data, dict):- return None- for candidate in (data, data.get("data")):- if isinstance(candidate, dict):- md = candidate.get("markdown")- if isinstance(md, str) and md.strip():- return md- return None---def _parse_search(body: dict[str, Any]) -> list[SearchResult]:- """Pull the result rows out of xcrawl's nested response envelope.-- The live shape is ``{"data": {"data": [ {position, title,- description, url}, ... ]}}``. We tolerate the inner list living at- either ``data`` or ``data.data`` (shape-drift defence) and skip any- row missing a usable URL.- """- if not isinstance(body, dict):- return []- data = body.get("data")- inner: Any = None- if isinstance(data, dict):- inner = data.get("data")- elif isinstance(data, list):- inner = data- if not isinstance(inner, list):- return []-- out: list[SearchResult] = []- for row in inner:- if not isinstance(row, dict):- continue- result_url = (row.get("url") or row.get("link") or "").strip()- if not result_url:- continue- out.append(- SearchResult(- title=(row.get("title") or "").strip(),- url=result_url,- snippet=(row.get("description") or row.get("snippet") or "").strip(),- )- )- return out---def _audit(- op: str,- subject: str,- *,- ok: bool,- count: int,- credits: Any = None,- error: str | None = None,-) -> None:- """Write one ``xcrawl`` audit row so spend + failures are visible.-- ``subject`` is the query (search) or the url (scrape).- """- details: dict[str, Any] = {- "op": op,- "subject_preview": subject[:_AUDIT_SUBJECT_PREVIEW_CHARS],- "result_count": count,- "ok": ok,- }- if credits is not None:- details["credits_used"] = credits- if error:- details["error"] = error- audit_log(- "xcrawl",- f"xcrawl.{op}: results={count} ok={ok}",- actor="xcrawl",- details=details,- )---__all__ = [- "ScrapeOutcome",- "ScrapeStatus",- "SearchResult",- "scrape",- "scrape_detailed",- "search",-]diff --git a/packages/core/openexecutive/monitoring/models.py b/packages/core/openexecutive/monitoring/models.pyindex 9d45297c..65301874 100644--- a/packages/core/openexecutive/monitoring/models.py+++ b/packages/core/openexecutive/monitoring/models.py@@ -32,14 +32,14 @@ SOURCE_KIND_EDGAR = "edgar"# (pricing, careers, leadership). Stateful (see monitoring.store page_watch_state).SOURCE_KIND_PAGE_WATCH = "page_watch"-# page_watch fetch-source contract: a row whose ``config_json[FETCH_KEY] ==-# FETCH_XCRAWL`` is fetched via xcrawl's scrape API instead of the keyless-# httpx fetcher (for JS-rendered / bot-blocked pages with no usable feed).-# Single source of truth so the page_watch adapter (which reads it) and the-# insert-time validator (which writes it when converting a non-feed rss row)-# can't drift.+# page_watch fetch-source marker. Every page_watch row is now fetched with the+# keyless httpx fetcher; rows written before the xcrawl scrape path was removed+# still carry ``config_json["fetch"] == "xcrawl"`` and a baseline captured as+# xcrawl markdown. The adapter treats that marker as "baseline is stale":+# it re-captures the baseline on the next poll without emitting a signal and+# clears the marker, so no phantom change is reported for the switch.PAGE_WATCH_FETCH_KEY = "fetch"-PAGE_WATCH_FETCH_XCRAWL = "xcrawl"+PAGE_WATCH_FETCH_LEGACY_XCRAWL = "xcrawl"# Watchlist row "mode" values. v1 default is ``active`` per the user's# instruction — dry-run is wired but opt-in (an entry created withdiff --git a/packages/core/openexecutive/monitoring/research/agentic.py b/packages/core/openexecutive/monitoring/research/agentic.pydeleted file mode 100644index 23fa4c3b..00000000--- a/packages/core/openexecutive/monitoring/research/agentic.py+++ /dev/null@@ -1,265 +0,0 @@-"""Agentic research turn — a bounded search → scrape_url → emit tool-use-loop so a specialist READS its sources before emitting findings.--The single-shot path gives a specialist only web_search snippets, so it-infers a claim and bolts on a URL it never read (the verify pass measured-this: nearly every cited source failed to back its claim). Here the-specialist instead loops: web_search to find candidates, ``scrape_url`` to-read the full article, then ``emit_research_findings`` grounded in what it-actually read.--Block handling mirrors the Executive's main loop-(``orchestrator/executive.py``): server web_search blocks-(``server_tool_use`` / ``web_search_tool_result``) are resolved inline by-Anthropic (or, under OpenRouter, are the ``web`` plugin and never appear as-blocks) — we echo them forward verbatim and run no handler. Only the-client ``scrape_url`` tool is executed by us, and every client ``tool_use``-gets a matching ``tool_result`` in the next user turn.-"""-from __future__ import annotations--import logging-from typing import Any--from openexecutive.agents.base import BaseAgent-from openexecutive.config import get_settings-from openexecutive.monitoring.research.models import ResearchFinding-from openexecutive.monitoring.research.prompts import research_addendum_for-from openexecutive.monitoring.research.scrape_tool import (- SCRAPE_URL_TOOL,- SCRAPE_URL_TOOL_NAME,- handle_scrape_url,-)-from openexecutive.monitoring.research.tools import EMIT_RESEARCH_FINDINGS_TOOL-from openexecutive.orchestrator.web_search_tool import build_web_search_tool-from openexecutive.providers import get_provider, model_supports_deep_reasoning-from openexecutive.providers.translator import reasoning_replay_block--logger = logging.getLogger(__name__)--_EMIT_TOOL_NAME = "emit_research_findings"-# A scrape can push a turn past the chat-time default; match the single-shot-# research ceiling.-_RESEARCH_TIMEOUT_SECONDS = 300.0-_MAX_TOKENS = 4096---async def run_specialist_agentic(- specialist_slug: str,- agent: BaseAgent,- research_context: str,-) -> list[ResearchFinding]:- """Run one specialist as a tool-use loop. Returns the parsed findings- (possibly empty). Best-effort: any provider/scrape failure ends the loop- with whatever was emitted so far (else empty) — never raises into the- workflow's ``asyncio.gather``."""- from openexecutive.agents.research_council import (- get_research_model,- get_research_use_deep_reasoning,- )- from openexecutive.monitoring.research.specialist_research import (- _extract_findings,- )-- settings = get_settings()- model = get_research_model()- use_deep = get_research_use_deep_reasoning()- system_prompt = agent.effective_system_prompt() + research_addendum_for(- specialist_slug- )-- tools: list[dict[str, Any]] = [EMIT_RESEARCH_FINDINGS_TOOL, SCRAPE_URL_TOOL]- web_search = build_web_search_tool()- if web_search is not None:- tools.append(web_search)-- messages: list[dict[str, Any]] = [- {"role": "user", "content": research_context},- ]- scrapes_used = 0- max_scrapes = max(0, settings.research_scrape_max_per_specialist)- max_iters = max(1, settings.research_loop_max_iterations)-- for iteration in range(1, max_iters + 1):- create_kwargs: dict[str, Any] = {- "model": model,- "max_tokens": _MAX_TOKENS,- "timeout": _RESEARCH_TIMEOUT_SECONDS,- "system": [- {- "type": "text",- "text": system_prompt,- "cache_control": {"type": "ephemeral"},- }- ],- "tools": tools,- "messages": messages,- }- if use_deep and model_supports_deep_reasoning(model):- create_kwargs["thinking"] = {"type": "adaptive"}- create_kwargs["output_config"] = {- "effort": settings.specialist_effort- }- create_kwargs["max_tokens"] = max(_MAX_TOKENS, 16000)-- try:- message = await get_provider(model).messages_create(**create_kwargs)- except Exception:- logger.exception(- "research agentic: specialist=%s provider call failed (iter=%d)",- specialist_slug, iteration,- )- return []-- content = getattr(message, "content", None) or []-- emitted = _has_emit(content)- # Mandatory read-before-cite: a specialist that emits findings without- # having scraped a single source is exactly the failure the loop- # exists to fix (it cites web_search snippets it never read). Reject- # that emit and force another turn — but only when scraping is- # actually possible (budget left) and we still have iterations to- # spare, so we never trap the model in an unsatisfiable loop.- force_read = (- emitted- and scrapes_used == 0- and max_scrapes > 0- and iteration < max_iters- )-- # Terminal — accept the emit unless we're forcing a read first.- if emitted and not force_read:- return _extract_findings(message, specialist_slug)-- assistant_content, scrape_calls = _assistant_turn(content)-- if not emitted and not scrape_calls:- # No scrape and no emit → the model is done (text-only, or it- # only ran web_search inline). There is no client tool_result to- # send, so the conversation can't continue — stop here.- logger.debug(- "research agentic: specialist=%s neither scraped nor emitted "- "at iter=%d — stopping", specialist_slug, iteration,- )- return []-- # Every CLIENT tool_use in the assistant turn needs a matching- # tool_result in the next user turn or the API 400s. Execute scrapes,- # and when forcing a read, also answer the rejected emit tool_use.- tool_results: list[dict[str, Any]] = []- for call in scrape_calls:- budget_remaining = max_scrapes - scrapes_used- try:- result = await handle_scrape_url(- call["input"], budget_remaining=budget_remaining,- )- except Exception as exc:- logger.warning(- "research agentic: scrape_url crashed for %s: %s",- specialist_slug, exc,- )- result = "error: scrape failed; emit without this source."- if budget_remaining > 0:- scrapes_used += 1- tool_results.append({- "type": "tool_result",- "tool_use_id": call["id"],- "content": result,- })-- if force_read:- for emit_id in _emit_ids(content):- tool_results.append({- "type": "tool_result",- "tool_use_id": emit_id,- "content": (- "You must call scrape_url and READ at least one source "- "to ground your claims before emitting. web_search "- "snippets are not enough. Call scrape_url now, then "- "emit findings grounded in what you read."- ),- })- logger.info(- "research agentic: specialist=%s emitted with 0 scrapes at "- "iter=%d — rejecting, forcing a read", specialist_slug,- iteration,- )-- messages.append({"role": "assistant", "content": assistant_content})- messages.append({"role": "user", "content": tool_results})-- logger.info(- "research agentic: specialist=%s hit iteration cap (%d) without "- "emitting", specialist_slug, max_iters,- )- return []---def _has_emit(content: list[Any]) -> bool:- return any(- getattr(b, "type", "") == "tool_use"- and getattr(b, "name", "") == _EMIT_TOOL_NAME- for b in content- )---def _emit_ids(content: list[Any]) -> list[str]:- """tool_use ids of every emit_research_findings block in the turn — each- needs a tool_result when we reject the emit to force a read first."""- return [- getattr(b, "id", "")- for b in content- if getattr(b, "type", "") == "tool_use"- and getattr(b, "name", "") == _EMIT_TOOL_NAME- ]---def _assistant_turn(- content: list[Any],-) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:- """Rebuild the assistant message from response blocks, and collect the- scrape_url client tool_uses to execute.-- Server web_search blocks are echoed verbatim (Anthropic already resolved- them); only scrape_url is executed by us.- """- assistant_content: list[dict[str, Any]] = []- scrape_calls: list[dict[str, Any]] = []- for block in content:- btype = getattr(block, "type", "")- if btype == "text":- assistant_content.append(- {"type": "text", "text": getattr(block, "text", "")}- )- elif btype == "tool_use":- tu = {- "type": "tool_use",- "id": getattr(block, "id", ""),- "name": getattr(block, "name", ""),- "input": getattr(block, "input", None) or {},- }- assistant_content.append(tu)- if tu["name"] == SCRAPE_URL_TOOL_NAME:- scrape_calls.append({"id": tu["id"], "input": tu["input"]})- elif (replay := reasoning_replay_block(block)) is not None:- # OpenRouter-path equivalent of a thinking block: replay its- # reasoning_details verbatim or the next tool-loop turn loses- # (Anthropic: rejects) the chain of thought.- assistant_content.append(replay)- elif btype in (- "thinking",- "redacted_thinking",- "server_tool_use",- "web_search_tool_result",- ):- # Echo verbatim. thinking / redacted_thinking MUST be preserved in- # the assistant history when extended thinking is on (deep- # reasoning) or Anthropic 400s the next turn; server web_search- # blocks are already-resolved and carried for continuity.- dump = getattr(block, "model_dump", None)- if callable(dump):- assistant_content.append(block.model_dump(exclude_none=True))- return assistant_content, scrape_calls---__all__ = ["run_specialist_agentic"]diff --git a/packages/core/openexecutive/monitoring/research/models.py b/packages/core/openexecutive/monitoring/research/models.pyindex f8b38969..d92a2fea 100644--- a/packages/core/openexecutive/monitoring/research/models.py+++ b/packages/core/openexecutive/monitoring/research/models.py@@ -85,15 +85,6 @@ class ResearchFinding(BaseModel):default="",description="Specialist slug (cso/cfo/...) that surfaced this. Set by the workflow.",)- verification: str | None = Field(- default=None,- description=(- "Set by the post-dedup verify pass (None when not verified): "- "'confirmed' (source backs the claim), 'contradicted', "- "'unsupported' (page doesn't support it), or "- "'source_unreachable' (cited URL couldn't be scraped)."- ),- )class ResearchRunSummary(BaseModel):diff --git a/packages/core/openexecutive/monitoring/research/prompts.py b/packages/core/openexecutive/monitoring/research/prompts.pyindex 95cda3a6..7942f2c5 100644--- a/packages/core/openexecutive/monitoring/research/prompts.py+++ b/packages/core/openexecutive/monitoring/research/prompts.py@@ -114,42 +114,7 @@ def shared_research_addendum() -> str:"DROPPED by the workflow before the Executive sees them** — do ""not emit a finding you would not stake 'medium' on.")- return addendum + _read_before_cite_addendum()---def _read_before_cite_addendum() -> str:- """Extra contract that only applies when the agentic scrape loop is on:- the specialist has a ``scrape_url`` tool and MUST read sources before- citing. Empty otherwise so the prompt always matches the available- tools. Static per-deployment (the flag is constant in-process), so it- does not break the cached system block.- """- from openexecutive.config import get_settings-- if not get_settings().research_agentic_scrape_enabled:- return ""- return (- "\n\n## READ BEFORE YOU CITE\n"- "\n"- "You have a `scrape_url` tool. web_search returns SNIPPETS — they "- "are NOT enough to cite. Before emitting ANY finding:\n"- " 1. Use web_search to find candidate sources.\n"- " 2. Call `scrape_url` on the 1-3 most promising results to READ "- "the full article.\n"- " 3. Ground every number / date / quote in the finding's summary "- "in what the scraped page ACTUALLY says.\n"- " 4. In `relevant_urls`, cite the specific ARTICLE url you "- "scraped and read — never a homepage or section link, and never a "- "page you did not read.\n"- "\n"- "**This is enforced, not advisory: if you call "- "`emit_research_findings` before scraping at least one source, the "- "emit is REJECTED and you are sent back to scrape first.** So always "- "`scrape_url` at least one source before you emit. A finding whose "- "source you did not scrape + read will also be demoted or dropped by "- "the verification pass. When you have read enough, call "- "`emit_research_findings` once."- )+ return addendum# Per-specialist focus tails. Appended AFTER the shared addendum so thediff --git a/packages/core/openexecutive/monitoring/research/scrape_tool.py b/packages/core/openexecutive/monitoring/research/scrape_tool.pydeleted file mode 100644index 594eeff0..00000000--- a/packages/core/openexecutive/monitoring/research/scrape_tool.py+++ /dev/null@@ -1,80 +0,0 @@-"""The ``scrape_url`` client tool the research specialists use to READ a-full source page during research.--web_search gives specialists snippets — not enough to ground a claim. This-tool lets a specialist fetch the full clean markdown of an article it found-(via xcrawl, which renders JS + bypasses soft blocks) so it can verify the-exact dates / numbers / quotes before emitting a finding. Mirrors the-spec + ``handle_*`` pattern of ``orchestrator/alert_tools.py``.-"""-from __future__ import annotations--import logging-from typing import Any--from openexecutive.integrations import xcrawl_client--logger = logging.getLogger(__name__)--SCRAPE_URL_TOOL_NAME = "scrape_url"--# Chars of markdown fed back into the loop per scrape. xcrawl already caps-# the scrape at external_monitor_max_fetch_bytes; this further bounds the-# per-turn input tokens (≈ a few thousand) so a long article can't blow up-# the specialist's context across iterations.-_SCRAPE_RESULT_CHARS = 12_000--SCRAPE_URL_TOOL: dict[str, Any] = {- "name": SCRAPE_URL_TOOL_NAME,- "description": (- "Fetch the full text (clean markdown) of a web page you found via "- "web_search, so you can READ it and ground your finding's exact "- "dates, numbers, and quotes in what the page actually says BEFORE "- "you emit. Call this on the 1-3 most promising results before "- "emit_research_findings. Pass a specific ARTICLE url (not a "- "homepage or section link). Returns the page's main content as "- "markdown, or an error string if it can't be read."- ),- "input_schema": {- "type": "object",- "properties": {- "url": {- "type": "string",- "description": "Full http(s) URL of the article to read.",- },- },- "required": ["url"],- },-}---async def handle_scrape_url(- tool_input: dict[str, Any], *, budget_remaining: int,-) -> str:- """Execute one ``scrape_url`` call. Returns the page markdown, or a- short instructive string the model can act on (no raise — a scrape- miss must not break the specialist's turn).-- ``budget_remaining`` is the specialist's remaining scrape allowance;- at or below zero the tool refuses and tells the model to emit.- """- if budget_remaining <= 0:- return (- "Scrape budget exhausted for this run — emit your findings now "- "using what you have already read."- )- url = str((tool_input or {}).get("url", "")).strip()- if not url.startswith(("http://", "https://")):- return "error: provide a full http(s) article URL."-- markdown = await xcrawl_client.scrape(url)- if markdown is None:- return (- f"Could not read {url} (unreachable, blocked, or empty). Use a "- "different source, or emit without it — do not cite a page you "- "could not read."- )- return markdown[:_SCRAPE_RESULT_CHARS]---__all__ = ["SCRAPE_URL_TOOL", "SCRAPE_URL_TOOL_NAME", "handle_scrape_url"]diff --git a/packages/core/openexecutive/monitoring/research/specialist_research.py b/packages/core/openexecutive/monitoring/research/specialist_research.pyindex 9392177f..7a94cded 100644--- a/packages/core/openexecutive/monitoring/research/specialist_research.py+++ b/packages/core/openexecutive/monitoring/research/specialist_research.py@@ -11,7 +11,6 @@ import loggingfrom typing import Anyfrom openexecutive.agents.base import BaseAgent-from openexecutive.config import get_settingsfrom openexecutive.monitoring.research.models import ResearchFindingfrom openexecutive.monitoring.research.prompts import research_addendum_forfrom openexecutive.monitoring.research.tools import (@@ -38,19 +37,6 @@ async def research_one_specialist(asyncio.gather doesn't abort on one specialist's crash. Theworkflow records per-specialist outcomes separately for the artifact."""- settings = get_settings()- if settings.research_agentic_scrape_enabled and settings.xcrawl_enabled:- # Read-before-cite: the specialist runs a search→scrape_url→emit- # loop so it grounds claims in full sources it actually read. Local- # import avoids a module-load cycle (agentic imports _extract_findings- # from here). Single-shot path below is unchanged when disabled.- from openexecutive.monitoring.research.agentic import (- run_specialist_agentic,- )- return await run_specialist_agentic(- specialist_slug, agent, research_context,- )-from openexecutive.agents.research_council import (get_research_model,get_research_use_deep_reasoning,diff --git a/packages/core/openexecutive/monitoring/research/verification.py b/packages/core/openexecutive/monitoring/research/verification.pydeleted file mode 100644index ba1a54cb..00000000--- a/packages/core/openexecutive/monitoring/research/verification.py+++ /dev/null@@ -1,278 +0,0 @@-"""Post-dedup verification of research findings against their cited source.--Research specialists discover via web_search (snippets — thin under-OpenRouter), but the finding contract demands verifying "URLs, dates,-names, and numbers against a current source" and findings "backed by a-dated URL you actually retrieved". The specialists have no way to-*retrieve* a page — only snippets. This pass closes that gap: for each-surviving finding it scrapes the cited URL (xcrawl, which renders JS and-bypasses soft blocks) and asks a cheap model whether the page actually-supports the claim, then adjusts confidence:-- - confirmed → keep (page backs the claim)- - contradicted → confidence = low (dropped by the existing filter)- - unsupported → demote one notch (page doesn't support the claim)- - source_unreachable→ demote one notch (cited URL couldn't be scraped —- the same hallucinated-URL failure mode as the watchlist)--Gated by ``xcrawl_enabled AND external_research_verify_enabled`` — a no-op-otherwise. Best-effort throughout: any scrape / LLM failure leaves the-finding unchanged so a verify hiccup can never sink a research run. Cost-is bounded: only findings with a cited URL, capped at-``research_verify_max_findings`` per run, on a cheap verify model.-"""-from __future__ import annotations--import asyncio-import logging-from typing import Any--from openexecutive.agents.base import BaseAgent-from openexecutive.alerts.models import SEVERITY_RANK-from openexecutive.config import get_settings-from openexecutive.integrations import xcrawl_client-from openexecutive.monitoring.research.models import (- ConfidenceLevel,- ResearchFinding,-)--logger = logging.getLogger(__name__)--# Scrape + one cheap LLM call; generous ceiling so a slow page can't hang-# the whole verify fan-out (each finding is independent under gather).-_VERIFY_TIMEOUT_SECONDS = 120.0-# Chars of scraped markdown fed to the verify model. xcrawl already caps the-# scrape at external_monitor_max_fetch_bytes; this bounds the verify call's-# input tokens (≈ a few thousand) regardless.-_MARKDOWN_CHARS = 12_000--# One-notch demotion. 'low' is terminal (the workflow's existing filter drops-# 'low' before synthesis).-_DEMOTE: dict[str, ConfidenceLevel] = {- "high": "medium",- "medium": "low",- "low": "low",-}--_EMIT_VERIFICATION_NAME = "emit_verification"--EMIT_VERIFICATION_TOOL: dict[str, Any] = {- "name": _EMIT_VERIFICATION_NAME,- "description": (- "Report whether the source page supports the research finding. "- "Call this EXACTLY ONCE."- ),- "input_schema": {- "type": "object",- "properties": {- "verdict": {- "type": "string",- "enum": ["confirmed", "contradicted", "unsupported"],- "description": (- "'confirmed' — the page clearly states the finding's "- "specific claim (its dates / numbers / names / event). "- "'contradicted' — the page states something that "- "conflicts with the claim. 'unsupported' — the page does "- "not contain enough to support the specific claim (a "- "vaguely-related page is 'unsupported', not 'confirmed')."- ),- },- "note": {- "type": "string",- "description": "≤1 sentence: the supporting/contradicting detail.",- },- },- "required": ["verdict"],- },-}--_VERIFY_SYSTEM = (- "You verify one research finding against the full text of the single "- "source page it cites. Decide ONLY from the provided page text — do not "- "use outside knowledge — whether the page supports the finding's "- "SPECIFIC claim (the dates, numbers, names, and event in its summary).\n\n"- "Be strict: a page that is merely on the same topic but does not state "- "the claim's specifics is 'unsupported', not 'confirmed'. Call "- "emit_verification EXACTLY ONCE with your verdict."-)---class _VerifyAgent(BaseAgent):- """Minimal agent shell so the pass can reuse ``analyze_with_tools`` (its- provider resolution + system-block plumbing). Distinct ``name`` so it- never collides with a real specialist's Council override; the model is- pinned per call via ``model_override``."""-- name = "research_verify"- domain = "utility"- model = ""- use_deep_reasoning = False-- def __init__(self) -> None:- self.model = get_settings().research_verify_model-- def get_system_prompt(self) -> str:- return _VERIFY_SYSTEM---def _first_url(finding: ResearchFinding) -> str | None:- """First http(s) URL the finding cites, or None."""- for raw in finding.relevant_urls:- url = (raw or "").strip()- if url.startswith(("http://", "https://")):- return url- return None---def _demote(confidence: ConfidenceLevel) -> ConfidenceLevel:- return _DEMOTE.get(confidence, confidence)---async def verify_findings(- findings: list[ResearchFinding],-) -> list[ResearchFinding]:- """Return ``findings`` with cited-source verification applied.-- Unchanged (no network) when xcrawl or verification is disabled, or when- no finding cites a URL. Only the top ``research_verify_max_findings`` by- severity (among those with a cited URL) are verified; the rest pass- through untouched.- """- settings = get_settings()- if not (settings.xcrawl_enabled and settings.external_research_verify_enabled):- return findings-- cap = max(0, settings.research_verify_max_findings)- # Only verify findings that (a) cite a URL and (b) could actually survive.- # A finding already at 'low' confidence is dropped by the downstream- # filter regardless of any verdict, so verifying it would waste a- # scrape + LLM call AND let a doomed finding skew the confirmed/demoted- # accounting (a 'low' demote is a no-op; a 'confirmed' low still drops).- candidates = [- (idx, f)- for idx, f in enumerate(findings)- if f.confidence != "low" and _first_url(f) is not None- ]- # Verify the highest-severity findings first so the budget protects the- # ones most likely to be routed to a human.- candidates.sort(- key=lambda pair: SEVERITY_RANK.get(pair[1].severity_hint.value, 0),- reverse=True,- )- selected = candidates[:cap]- if not selected:- return findings-- verdicts = await asyncio.gather(- *(_verify_one(f) for _idx, f in selected),- return_exceptions=True,- )-- out = list(findings)- confirmed = demoted = 0- for (idx, original), result in zip(selected, verdicts, strict=True):- if isinstance(result, BaseException):- logger.warning(- "research verify: %r raised — leaving unchanged: %s",- original.title[:80], result,- )- continue- out[idx] = result- if result.verification == "confirmed":- confirmed += 1- elif result.confidence != original.confidence:- demoted += 1-- logger.info(- "research verify: %d verified, %d confirmed, %d demoted",- len(selected), confirmed, demoted,- )- return out---async def _verify_one(finding: ResearchFinding) -> ResearchFinding:- """Verify one finding against its cited source. Returns a (possibly- confidence-adjusted) copy with a ``verification`` tag set.-- An xcrawl *infra* failure (status ``error``) leaves the finding unchanged- and untagged — a verify-pass outage must never demote a real finding. Only- a page that is fetched-but-empty (a genuinely dead URL) is demoted to- ``source_unreachable``. A genuine LLM error still propagates, which the- caller treats as 'leave unchanged'."""- url = _first_url(finding)- if url is None: # defensive; callers pre-filter- return finding-- outcome = await xcrawl_client.scrape_detailed(url)- if outcome.status in ("error", "disabled"):- # Not a verdict on the source — xcrawl couldn't even be asked:- # error — xcrawl itself failed (down / 401 / 429 / timeout)- # disabled — xcrawl off, OR enabled-but-no-key (the verify_findings- # gate only checks the enabled flag, not the key)- # Demoting here would let a verify-pass outage or a config gap sink an- # entire research run, exactly what this pass promises it never does.- # Leave the finding untouched (and untagged, so the run's- # verified-count reflects that we couldn't check it).- return finding- if outcome.markdown is None:- # status == "empty": the page was fetched but yields no content — a- # genuinely dead / unreadable source (the hallucinated-URL failure- # mode this pass exists to catch).- return finding.model_copy(update={- "confidence": _demote(finding.confidence),- "verification": "source_unreachable",- })-- verdict = await _llm_verdict(finding, outcome.markdown[:_MARKDOWN_CHARS])- if verdict == "confirmed":- return finding.model_copy(update={"verification": "confirmed"})- if verdict == "contradicted":- return finding.model_copy(update={- "confidence": "low", "verification": "contradicted",- })- if verdict == "unsupported":- return finding.model_copy(update={- "confidence": _demote(finding.confidence),- "verification": "unsupported",- })- # verdict == "unknown" — model didn't return a usable verdict. Don't- # penalize confidence for a verify miss, but tag it so the run's- # verified-count reflects the attempt (vs. a finding we never touched).- return finding.model_copy(update={"verification": "inconclusive"})---async def _llm_verdict(finding: ResearchFinding, markdown: str) -> str:- """Return 'confirmed' | 'contradicted' | 'unsupported' | 'unknown'."""- agent = _VerifyAgent()- user_content = (- f"<finding>\nTITLE: {finding.title}\nSUMMARY: {finding.summary}\n"- f"</finding>\n\n<source_page>\n{markdown}\n</source_page>\n\n"- "Does the source page support the finding's specific claim? "- "Call emit_verification with your verdict."- )- message = await agent.analyze_with_tools(- user_content,- tools=[EMIT_VERIFICATION_TOOL],- max_tokens=512,- timeout_seconds=_VERIFY_TIMEOUT_SECONDS,- model_override=get_settings().research_verify_model,- deep_reasoning_override=False,- )- return _parse_verdict(message)---def _parse_verdict(message: Any) -> str:- """Pull the verdict from the first emit_verification tool_use block;- 'unknown' when the model didn't emit a usable one."""- for block in getattr(message, "content", None) or []:- if getattr(block, "type", "") != "tool_use":- continue- if getattr(block, "name", "") != _EMIT_VERIFICATION_NAME:- continue- verdict = (getattr(block, "input", None) or {}).get("verdict")- if verdict in ("confirmed", "contradicted", "unsupported"):- return verdict- return "unknown"---__all__ = ["EMIT_VERIFICATION_TOOL", "verify_findings"]diff --git a/packages/core/openexecutive/monitoring/research/watch_policy.py b/packages/core/openexecutive/monitoring/research/watch_policy.pyindex cf8dfeea..b368b20d 100644--- a/packages/core/openexecutive/monitoring/research/watch_policy.py+++ b/packages/core/openexecutive/monitoring/research/watch_policy.py@@ -848,7 +848,6 @@ def classify(+1 entity is already watched under another source (corroboration)+1 target is the entity's own source (its ticker / its domain / a primary host)+1 finding confidence is high- +1 finding verification is 'confirmed'+1 cross-specialist consensus on the finding+1 the entity is named in a recent episodic decision±1 policy history for (signal_type, grounding kind) at >=5 samples@@ -912,9 +911,6 @@ def classify(if finding.confidence == "high":score += 1reasons.append("finding confidence high")- if finding.verification == "confirmed":- score += 1- reasons.append("source verified")if "," in (finding.source_specialist or ""):score += 1reasons.append("cross-specialist consensus")@@ -1054,8 +1050,7 @@ def auto_link_finding(with the entity term :func:`classify` would use), so the link can neverlend a proposal a finding the score would not honour. Among candidates,one that cites the target's site or names the ticker outranks one that- only names the entity, then higher confidence, then verification, then- the earlier finding — so the evidence URL stamped on the row is the+ only names the entity, then higher confidence, then the earlier finding — so the evidence URL stamped on the row is thesource's own, not an unrelated company's."""match = match_vocab(proposal.grounding_entity, ctx.vocabulary)entity_term = match[0] if match else ""@@ -1064,7 +1059,7 @@ def auto_link_finding(entity_labels = set(_name_tokens(entity_term, ctx.vocabulary)) | {target_phrase.replace(" ", "")}entry_value = ctx.vocabulary.get(entity_term) if entity_term else Nonepinned = _entry(entry_value).domains if entry_value is not None else ()- best: tuple[tuple[int, int, int, int, int], int] | None = None+ best: tuple[tuple[int, int, int, int], int] | None = Nonefor index, finding in enumerate(findings):if not _finding_supports(proposal, finding, entity_term, ctx.vocabulary):continue@@ -1085,7 +1080,6 @@ def auto_link_finding(cites_source,cites_entity_site,_CONFIDENCE_RANK.get(finding.confidence, 0),- int(finding.verification == "confirmed"),-index,)if best is None or key > best[0]:diff --git a/packages/core/openexecutive/monitoring/sources/page_watch.py b/packages/core/openexecutive/monitoring/sources/page_watch.pyindex 28177ef6..9b177359 100644--- a/packages/core/openexecutive/monitoring/sources/page_watch.py+++ b/packages/core/openexecutive/monitoring/sources/page_watch.py@@ -47,7 +47,7 @@ from openexecutive.config import get_settingsfrom openexecutive.monitoring import storefrom openexecutive.monitoring.models import (PAGE_WATCH_FETCH_KEY,- PAGE_WATCH_FETCH_XCRAWL,+ PAGE_WATCH_FETCH_LEGACY_XCRAWL,SOURCE_KIND_PAGE_WATCH,Signal,WatchlistItem,@@ -63,9 +63,8 @@ logger = logging.getLogger(__name__)# Pages change slowly; 6h is polite and sufficient. Per-row cadence can tighten._DEFAULT_POLL_MINUTES = 360-# The fetch-source sentinel (config_json["fetch"] == "xcrawl") lives in-# monitoring.models so the insert-time validator and this adapter share one-# definition; see PAGE_WATCH_FETCH_KEY / PAGE_WATCH_FETCH_XCRAWL.+# The legacy fetch marker (config_json["fetch"] == "xcrawl") lives in+# monitoring.models; see PAGE_WATCH_FETCH_KEY / PAGE_WATCH_FETCH_LEGACY_XCRAWL.# Cap the text we SNAPSHOT (store) and DIFF — NOT what we hash. Detection# hashes the full normalized text so a change anywhere on the page registers;@@ -105,15 +104,7 @@ class PageWatchSource:logger.warning("page_watch: rejecting %r target — %s", item.slug, reason)return []- # Two fetch paths. Default is the keyless httpx fetcher. Rows whose- # config_json carries ``{"fetch": "xcrawl"}`` route through xcrawl's- # scrape API instead — for pages the keyless fetcher can't read- # (JS-only SPAs, soft bot blocks) that publish no RSS feed. xcrawl- # returns clean markdown, so it skips the HTML→text reduction.- if item.config_json.get(PAGE_WATCH_FETCH_KEY) == PAGE_WATCH_FETCH_XCRAWL:- text = await self._fetch_text_xcrawl(item)- else:- text = await self._fetch_text_httpx(item)+ text = await self._fetch_text_httpx(item)if text is None:# Fetch failed (already logged at the fetch site) — drop this tick.return []@@ -130,9 +121,20 @@ class PageWatchSource:snapshot = text[:_MAX_TEXT_CHARS]prior = store.get_page_watch_state(item.slug, db_path=db_path)- if prior is None:+ # A row still tagged with the legacy xcrawl marker has a baseline+ # captured as scraped markdown, which never hashes equal to the+ # httpx text of the same page. Re-capture it silently and clear the+ # marker so the switch of fetcher is not reported as a page change.+ # Only a stored row can have its marker cleared, so an unsaved item+ # (no id) is compared normally rather than re-baselined every poll.+ legacy_baseline = bool(item.id) and (+ item.config_json.get(PAGE_WATCH_FETCH_KEY) == PAGE_WATCH_FETCH_LEGACY_XCRAWL+ )+ if prior is None or legacy_baseline:# First observation — record the baseline, do not alert.store.upsert_page_watch_state(item.slug, content_hash, snapshot, db_path=db_path)+ if legacy_baseline:+ self._clear_legacy_marker(item, db_path=db_path)logger.debug("page_watch: %s baseline captured", item.slug)return []@@ -191,41 +193,27 @@ class PageWatchSource:"page_watch: %s exceeded byte cap — dropping tick", item.target,)return None- return _html_to_text(body)-- async def _fetch_text_xcrawl(self, item: WatchlistItem) -> str | None:- """Fetch via xcrawl scrape (markdown) → normalized text, or ``None``.-- Returns ``None`` when xcrawl is disabled or the scrape failed /- returned nothing. xcrawl markdown is already clean text, so we only- collapse whitespace (matching ``_html_to_text``'s output) so the- content hash is stable across trivial reformatting.- """- from openexecutive.integrations import xcrawl_client-- markdown = await xcrawl_client.scrape(item.target)- if markdown is None:- # Disabled xcrawl is an expected config state, not a fault — a- # row tagged fetch=xcrawl on a deployment with XCRAWL_ENABLED=false- # would otherwise log a WARNING every poll tick. Demote that case- # to debug; a genuine scrape failure (xcrawl on, call failed) was- # already logged inside xcrawl_client.scrape, so debug here too.- if not get_settings().xcrawl_enabled:- logger.debug(- "page_watch: %s tagged fetch=xcrawl but xcrawl disabled",- item.slug,- )- else:- logger.debug(- "page_watch: xcrawl scrape yielded nothing for %s (%s)",- item.slug, item.target,- )- return None- # `or None` (not "") so that markdown which normalizes to empty — e.g.- # truncated down to a whitespace-only prefix — is treated as "no fetch- # this tick" (drop, keep prior baseline), NOT as an empty extraction- # that would store a blank baseline and later "change" into content.- return _WS_RE.sub(" ", markdown).strip() or None+ return html_to_text(body)++ @staticmethod+ def _clear_legacy_marker(item: WatchlistItem, *, db_path: Path | None) -> None:+ """Drop the legacy ``fetch: xcrawl`` marker from the row's config so+ the next poll compares against the freshly captured baseline. The+ key is removed in place (no read-modify-write of the whole column),+ so a concurrent edit to another config key is kept. Best-effort: a+ failed update only means the baseline is re-captured once more."""+ if not item.id:+ return+ try:+ changed = store.remove_watchlist_config_key(+ item.id, PAGE_WATCH_FETCH_KEY, db_path=db_path,+ )+ except Exception: # noqa: BLE001 — never let housekeeping sink a poll+ changed = 0+ if not changed:+ logger.warning(+ "page_watch: could not clear legacy fetch marker on %s", item.slug,+ )def matches_trigger(self, signal: Signal, item: WatchlistItem) -> bool:"""Optional keyword filter — matched against the change summary + added text."""@@ -245,7 +233,7 @@ class PageWatchSource:# --------------------------------------------------------------------- #-def _html_to_text(body: bytes) -> str:+def html_to_text(body: bytes) -> str:"""Reduce an HTML page to normalized, comparable visible text.Markup-insensitive on purpose: strips <script>/<style>/<noscript> blocksdiff --git a/packages/core/openexecutive/monitoring/store.py b/packages/core/openexecutive/monitoring/store.pyindex cb2f8a23..d48c63c8 100644--- a/packages/core/openexecutive/monitoring/store.py+++ b/packages/core/openexecutive/monitoring/store.py@@ -543,6 +543,24 @@ def update_watchlist_fields(return int(cursor.rowcount)+def remove_watchlist_config_key(+ item_id: int, key: str, *, db_path: Path | None = None+) -> int:+ """Remove one top-level key from a watchlist row's ``config_json`` in+ place (SQLite ``json_remove``), so a concurrent write to another key is+ never overwritten by a read-modify-write of a stale snapshot. ``key``+ is validated as a bare identifier because it is spliced into the JSON+ path. Returns the rowcount."""+ if not key.isidentifier():+ raise ValueError(f"Cannot remove config key {key!r}")+ with _get_conn(db_path) as conn:+ cursor = conn.execute(+ "UPDATE watchlist SET config_json = json_remove(config_json, ?) WHERE id = ?",+ (f"$.{key}", item_id),+ )+ return int(cursor.rowcount)++def delete_watchlist_item(item_id: int, db_path: Path | None = None) -> bool:"""Hard-delete a watchlist row and its ``external_signals`` history.Returns True iff a row was removed.@@ -1057,6 +1075,7 @@ __all__ = ["mark_polled","get_page_watch_state","mark_signal_processed",+ "remove_watchlist_config_key","set_enabled","update_signal_enrichment","update_watchlist_fields",diff --git a/packages/core/openexecutive/monitoring/target_validation.py b/packages/core/openexecutive/monitoring/target_validation.pyindex b14b3cfa..25ca7b0b 100644--- a/packages/core/openexecutive/monitoring/target_validation.py+++ b/packages/core/openexecutive/monitoring/target_validation.py@@ -10,9 +10,9 @@ This module validates an ``rss`` target at insert time: it must fetch andparse as a feed with entries. A target the server *answers* but thatisn't a feed is either:- - converted to a scrape-backed ``page_watch`` when xcrawl can read the- page (modern JS / bot-blocked news pages publish no feed), or- - rejected (:class:`WatchlistTargetError`),+ - converted to a ``page_watch`` when the page the feed check already+ fetched yields readable text (a news or pricing page with no feed), or+ - rejected (:class:`WatchlistTargetError`) when it does not,so a dead ``rss`` row never reaches the scan loop. A transient / networkblip passes the target through unchanged rather than blocking a@@ -29,8 +29,6 @@ import httpxfrom openexecutive.config import get_settingsfrom openexecutive.monitoring.models import (- PAGE_WATCH_FETCH_KEY,- PAGE_WATCH_FETCH_XCRAWL,SOURCE_KIND_PAGE_WATCH,SOURCE_KIND_RSS,)@@ -39,9 +37,16 @@ from openexecutive.monitoring.sources._http import (fetch_bounded,validate_target_url,)+from openexecutive.monitoring.sources.page_watch import html_to_textlogger = logging.getLogger(__name__)+# A non-feed page becomes a page_watch only when its visible text is at least+# this long. Rules out empty shells, bare error strings and binary bodies that+# happen to decode to a few characters; a real article, pricing or news page+# clears it easily.+_MIN_PAGE_TEXT_CHARS = 200+class WatchlistTargetError(Exception):"""A watchlist target failed insert-time validation (definitively bad)."""@@ -57,7 +62,7 @@ async def validate_and_normalize_target("""Validate ``target`` for ``signal_type``; may convert the signal type.Returns the ``(signal_type, target, config)`` to insert — converted to- ``page_watch`` when a non-feed ``rss`` target is scrapeable. Raises+ ``page_watch`` when a non-feed ``rss`` target is a readable page. Raises:class:`WatchlistTargetError` when an ``rss`` target is definitivelybad and can't be salvaged. Only ``rss`` is validated; every other typeis returned unchanged with no network I/O.@@ -65,7 +70,7 @@ async def validate_and_normalize_target(if signal_type != SOURCE_KIND_RSS:return signal_type, target, config- verdict = await _feed_check(target)+ verdict, body = await _feed_check(target)if verdict == "feed":return signal_type, target, configif verdict == "unknown":@@ -77,58 +82,67 @@ async def validate_and_normalize_target()return signal_type, target, configif verdict == "blocked":- # SSRF-rejected / unparseable / non-public. Reject outright — do NOT- # offer the xcrawl-scrape conversion, which would fetch the target- # from xcrawl's network and bypass our SSRF guard.+ # SSRF-rejected / unparseable / non-public. Reject outright.raise WatchlistTargetError(f"{target!r} is not a fetchable public URL ""(blocked by the SSRF guard or unparseable).")# verdict == "not_feed": a public server answered, but it is not a feed.- from openexecutive.integrations import xcrawl_client-- if await xcrawl_client.scrape(target) is not None:+ # The body the feed check already fetched (same SSRF guard, same byte+ # cap the page_watch adapter uses) decides whether the page is worth+ # watching: readable text → page_watch; nothing readable → reject.+ if body is not None and _is_readable_page(body):new_config = dict(config)- new_config[PAGE_WATCH_FETCH_KEY] = PAGE_WATCH_FETCH_XCRAWL# page_watch reads config["label"]; the rss row used "feed_label".label = new_config.pop("feed_label", None)if label and "label" not in new_config:new_config["label"] = labellogger.info(- "watchlist validate: %r is not a feed but is scrapeable — "- "converting rss → page_watch (xcrawl)", target,+ "watchlist validate: %r is not a feed but is a readable page — "+ "converting rss → page_watch", target,)return SOURCE_KIND_PAGE_WATCH, target, new_configraise WatchlistTargetError(- f"{target!r} is not a valid RSS/Atom feed and could not be scraped. "- "Fix the feed URL, or add it as signal_type='page_watch'."+ f"{target!r} is not a valid RSS/Atom feed and yields no readable "+ "page text. Fix the feed URL, or add it as signal_type='page_watch'.")+def _is_readable_page(body: bytes) -> bool:+ """Whether a fetched body is a text page with enough visible content to+ watch. A NUL byte in the head marks a binary body (PDF, image, archive);+ the text threshold rejects shells and one-line error pages."""+ if b"\x00" in body[:4096]:+ return False+ return len(html_to_text(body)) >= _MIN_PAGE_TEXT_CHARS++async def _feed_check(target: str,-) -> Literal["feed", "not_feed", "blocked", "unknown"]:- """Classify ``target``.+) -> tuple[Literal["feed", "not_feed", "blocked", "unknown"], bytes | None]:+ """Classify ``target``; also return the fetched body when there is one.``feed`` — feedparser recognized it as a feed.- ``not_feed`` — a public server answered, but it isn't a feed.+ ``not_feed`` — a public server answered, but it isn't a feed (the body+ is returned so the caller can decide whether it is a readable page;+ ``None`` when the server answered with an error or an oversized body).``blocked`` — SSRF-rejected / unparseable / non-public URL (never fetched).``unknown`` — transient fetch failure; caller should not penalize it."""ok, _reason = validate_target_url(target)if not ok:- return "blocked"+ return "blocked", Nonemax_bytes = get_settings().external_monitor_max_fetch_bytestry:body = await fetch_bounded(target, max_bytes)except FetchOverflowError:- return "not_feed" # too big to be a sane feed+ return "not_feed", None # too big to be a sane feed or pageexcept httpx.HTTPStatusError:- return "not_feed" # server answered 4xx/5xx → definitively bad+ return "not_feed", None # server answered 4xx/5xx → definitively badexcept (httpx.HTTPError, httpx.InvalidURL):- return "unknown" # connection / timeout — transient+ return "unknown", None # connection / timeout — transient# feedparser sets `version` (e.g. "rss20", "atom10") for anything it# recognizes as a feed — INCLUDING a valid but currently-empty feed. Use# it rather than `.entries` so a brand-new / quiet feed isn't misread as a@@ -136,7 +150,7 @@ async def _feed_check(# attribute access): an empty body yields a FeedParserDict with no# `version` key at all, and `parsed.version` would AttributeError.parsed = feedparser.parse(body)- return "feed" if parsed.get("version") else "not_feed"+ return ("feed", body) if parsed.get("version") else ("not_feed", body)__all__ = ["WatchlistTargetError", "validate_and_normalize_target"]diff --git a/packages/core/openexecutive/workflows/executive_research.py b/packages/core/openexecutive/workflows/executive_research.pyindex 7efa412f..38deeabf 100644--- a/packages/core/openexecutive/workflows/executive_research.py+++ b/packages/core/openexecutive/workflows/executive_research.py@@ -335,16 +335,6 @@ class ExecutiveResearchWorkflow(Workflow):"specialists; consensus boosts severity / confidence."),),- WorkflowStepDef(- id="verify",- title="Verify findings against sources",- description=(- "Scrape each surviving finding's cited URL and confirm "- "the page supports the claim; demote / drop findings "- "whose source is dead or doesn't back them. No-op unless "- "xcrawl + verification are enabled."- ),- ),WorkflowStepDef(id="executive_synthesis",title="Executive routes findings",@@ -510,33 +500,7 @@ class ExecutiveResearchWorkflow(Workflow):deduped_all: list[ResearchFinding] = dedup_findings(findings_by_specialist)- yield WorkflowEvent(- type="step_done",- step_id="dedup",- summary=f"after_dedup={len(deduped_all)}",- )-- # ------------------------------------------------------------------- # Step 3b: verify findings against their cited source- # ------------------------------------------------------------------- # Runs BEFORE the low-confidence filter so a finding whose source is- # dead or doesn't support it is demoted (often to 'low' → dropped)- # here, instead of being routed to a human on snippet-level trust.- # No-op unless xcrawl + verification are enabled; best-effort (a- # verify failure leaves the finding unchanged).- yield WorkflowEvent(- type="step_start",- step_id="verify",- step_title="Verify findings against sources",- )- from openexecutive.monitoring.research.verification import (- verify_findings,- )-- deduped_all = await verify_findings(deduped_all)- verified = [f for f in deduped_all if f.verification is not None]-- # Pre-synthesis quality filter — now reflects verify demotions.+ # Pre-synthesis quality filter.# Low-confidence findings are dropped before the Executive ever sees# them — the round_2 post-mortem showed they make up most of the# noise, and the synthesis prompt cannot be trusted to ignore them@@ -548,21 +512,15 @@ class ExecutiveResearchWorkflow(Workflow):low_conf_dropped = len(deduped_all) - len(deduped)if low_conf_dropped:logger.info(- "research: dropped %d low-confidence finding(s) pre-synthesis "- "(after verify)",+ "research: dropped %d low-confidence finding(s) pre-synthesis",low_conf_dropped,)- # step_done carries the full post-verify accounting (the dedup step- # now only reports after_dedup), so the low_conf_dropped / to_synthesis- # signals observers relied on are preserved here.yield WorkflowEvent(type="step_done",- step_id="verify",+ step_id="dedup",summary=(- f"verified={len(verified)} "- f"confirmed={sum(1 for f in verified if f.verification == 'confirmed')} "- f"demoted={sum(1 for f in verified if f.verification in ('contradicted', 'unsupported', 'source_unreachable'))} "+ f"after_dedup={len(deduped_all)} "f"low_conf_dropped={low_conf_dropped} "f"to_synthesis={len(deduped)}"),@@ -1289,8 +1247,7 @@ def _render_watchlist_turn(parts: list[str] = ["FINDINGS FROM THIS RESEARCH RUN:\n"]for i, f in enumerate(findings, start=1):block = (- f"#{i} [{f.severity_hint.value} | {f.confidence}"- f"{' | verified' if f.verification == 'confirmed' else ''}] "+ f"#{i} [{f.severity_hint.value} | {f.confidence}] "f"({f.source_specialist}) {f.title}\n"f" {f.summary}\n")diff --git a/packages/core/tests/unit/test_deep_reasoning_support.py b/packages/core/tests/unit/test_deep_reasoning_support.pyindex 4cdffe2d..0d7b1b68 100644--- a/packages/core/tests/unit/test_deep_reasoning_support.py+++ b/packages/core/tests/unit/test_deep_reasoning_support.py@@ -364,17 +364,6 @@ def test_reasoning_block_is_replayed_as_reasoning_details_on_assistant_turn() ->assert assistant["content"] is None-def test_research_loop_echoes_reasoning_block_into_history() -> None:- from openexecutive.monitoring.research.agentic import _assistant_turn-- block = SimpleNamespace(type=OPENROUTER_REASONING_BLOCK, reasoning_details=_DETAILS)- tool = SimpleNamespace(type="tool_use", id="c1", name="scrape_url", input={"url": "u"})- history, scrapes = _assistant_turn([block, tool])- assert history[0] == {"type": OPENROUTER_REASONING_BLOCK, "reasoning_details": _DETAILS}- assert history[1]["type"] == "tool_use"- assert scrapes == [{"id": "c1", "input": {"url": "u"}}]--# --------------------------------------------------------------------------# 5. Remote catalog data: capability flags need a real list# --------------------------------------------------------------------------@@ -480,9 +469,8 @@ def test_every_tool_loop_replays_reasoning() -> None:helper, so continuity isn't a research-loop-only property."""import inspect- from openexecutive.monitoring.research import agenticfrom openexecutive.orchestrator import executivefrom openexecutive.workflows import executive_reflection, executive_research- for mod in (agentic, executive, executive_research, executive_reflection):+ for mod in (executive, executive_research, executive_reflection):assert "reasoning_replay_block(block)" in inspect.getsource(mod), mod.__name__diff --git a/packages/core/tests/unit/test_executive_research_workflow.py b/packages/core/tests/unit/test_executive_research_workflow.pyindex 7c8bd2f8..6811cecb 100644--- a/packages/core/tests/unit/test_executive_research_workflow.py+++ b/packages/core/tests/unit/test_executive_research_workflow.py@@ -238,11 +238,10 @@ async def test_workflow_runs_end_to_end_with_stubbed_specialists(result_data = event.datastep_starts = [e for e in events if e.type == "step_start"]- # gather_context, research_specialists, dedup, verify, executive_synthesis,- # emit_artifact. The verify step always emits (its work is a gated no-op- # when xcrawl/verification are disabled, as here).- assert len(step_starts) == 6- assert {"verify"} <= {e.step_id for e in step_starts}+ # gather_context, research_specialists, dedup, executive_synthesis,+ # emit_artifact.+ assert len(step_starts) == 5+ assert "verify" not in {e.step_id for e in step_starts}assert artifact, "workflow produced no artifact"assert "Acme raised $50M" in artifactassert "send_slack_dm" in artifactdiff --git a/packages/core/tests/unit/test_monitoring_page_watch.py b/packages/core/tests/unit/test_monitoring_page_watch.pyindex 7b7f840a..8d742b1f 100644--- a/packages/core/tests/unit/test_monitoring_page_watch.py+++ b/packages/core/tests/unit/test_monitoring_page_watch.py@@ -15,7 +15,7 @@ from openexecutive.monitoring.sources import list_registered_kindsfrom openexecutive.monitoring.sources.page_watch import (PageWatchSource,_diff,- _html_to_text,+ html_to_text,)@@ -70,7 +70,7 @@ def test_html_to_text_strips_markup_and_scripts() -> None:<body><!-- comment --><h1>Pricing</h1><p>Pro plan is £30/mo</p></body></html>"""- text = _html_to_text(body)+ text = html_to_text(body)assert "Pricing" in textassert "£30/mo" in text # and £ entities unescaped# script/style content must not survive@@ -269,72 +269,88 @@ def test_page_watch_registered() -> None:# --------------------------------------------------------------------- #-# xcrawl fetch path (config_json["fetch"] == "xcrawl")+# Legacy rows tagged config_json["fetch"] == "xcrawl"# --------------------------------------------------------------------- #-def _install_xcrawl(monkeypatch: pytest.MonkeyPatch, holder: SimpleNamespace) -> None:- """Patch validate + xcrawl_client.scrape; leave fetch_bounded alone so a- test fails loudly if the xcrawl row wrongly falls back to httpx."""- async def fake_scrape(url: str) -> str | None:+def _install_httpx(monkeypatch: pytest.MonkeyPatch, holder: SimpleNamespace) -> None:+ async def fake_fetch(url: str, max_bytes: int, **kwargs: object) -> bytes:holder.calls += 1- return holder.md-- async def boom_fetch(url: str, max_bytes: int, **kwargs: object) -> bytes:- raise AssertionError("xcrawl row must NOT fall back to httpx fetch_bounded")+ return holder.bodymonkeypatch.setattr("openexecutive.monitoring.sources.page_watch.validate_target_url",lambda u: (True, ""),)- # Make any httpx fallback blow up so the test proves the xcrawl route.- monkeypatch.setattr(- "openexecutive.monitoring.sources.page_watch.fetch_bounded", boom_fetch,- )monkeypatch.setattr(- "openexecutive.integrations.xcrawl_client.scrape", fake_scrape,+ "openexecutive.monitoring.sources.page_watch.fetch_bounded", fake_fetch,)@pytest.mark.asyncio-async def test_page_watch_xcrawl_detects_change(+async def test_legacy_xcrawl_row_rebaselines_without_signal(db: Path, monkeypatch: pytest.MonkeyPatch,) -> None:- holder = SimpleNamespace(md="# Pricing\n\nPro plan is $30/mo", calls=0)- _install_xcrawl(monkeypatch, holder)- item = _make_item(slug="page-xc", config={"fetch": "xcrawl"})+ """A row whose baseline was captured as xcrawl markdown must not report+ the switch to the plain fetcher as a page change: the first plain poll+ re-captures the baseline silently and clears the legacy marker, and+ only a later real change emits."""+ holder = SimpleNamespace(body=b"<html><body>Pro plan is $30/mo</body></html>", calls=0)+ _install_httpx(monkeypatch, holder)+ ms.insert_watchlist_item(+ slug="page-legacy", signal_type="page_watch",+ target="https://example.com/pricing", config={"fetch": "xcrawl", "label": "Pricing"},+ db_path=db,+ )+ item = next(i for i in ms.list_watchlist(db_path=db) if i.slug == "page-legacy")+ # Baseline as xcrawl would have stored it: markdown, a different hash.+ ms.upsert_page_watch_state("page-legacy", "old-markdown-hash", "# Pricing Pro plan is $30/mo", db_path=db)src = PageWatchSource()- assert await src.poll(item, db_path=db) == [] # baseline- assert await src.poll(item, db_path=db) == [] # unchanged- holder.md = "# Pricing\n\nPro plan is $40/mo"- signals = await src.poll(item, db_path=db) # changed+ assert await src.poll(item, db_path=db) == [] # re-baseline, no signal+ state = ms.get_page_watch_state("page-legacy", db_path=db)+ assert state is not None and state["content_hash"] != "old-markdown-hash"+ refreshed = next(i for i in ms.list_watchlist(db_path=db) if i.slug == "page-legacy")+ assert "fetch" not in refreshed.config_json # marker cleared+ assert refreshed.config_json.get("label") == "Pricing" # other config kept++ assert await src.poll(refreshed, db_path=db) == [] # unchanged+ holder.body = b"<html><body>Pro plan is $40/mo</body></html>"+ signals = await src.poll(refreshed, db_path=db) # real changeassert len(signals) == 1- assert signals[0].source_kind == "page_watch"assert "$40/mo" in signals[0].raw_payload["added_text"]- assert holder.calls == 3 # every poll went through xcrawl, not httpx+ assert holder.calls == 3@pytest.mark.asyncio-async def test_page_watch_xcrawl_failure_drops_tick(+async def test_legacy_marker_on_unsaved_item_does_not_rebaseline_forever(db: Path, monkeypatch: pytest.MonkeyPatch,) -> None:- holder = SimpleNamespace(md=None, calls=0) # scrape returns nothing- _install_xcrawl(monkeypatch, holder)- item = _make_item(slug="page-xc2", config={"fetch": "xcrawl"})- assert await PageWatchSource().poll(item, db_path=db) == []- # No baseline stored, so a later successful scrape is treated as first-seen.- assert ms.get_page_watch_state("page-xc2", db_path=db) is None+ """An item with no id cannot have its marker cleared, so it must be+ compared normally — otherwise it would re-baseline on every poll and+ never report a change."""+ holder = SimpleNamespace(body=b"<html><body>v1</body></html>", calls=0)+ _install_httpx(monkeypatch, holder)+ item = WatchlistItem(+ id=None, slug="page-noid", signal_type="page_watch",+ target="https://example.com/p", config_json={"fetch": "xcrawl"},+ )+ src = PageWatchSource()+ assert await src.poll(item, db_path=db) == [] # first baseline+ holder.body = b"<html><body>v2</body></html>"+ assert len(await src.poll(item, db_path=db)) == 1 # change reported-async def test_page_watch_xcrawl_whitespace_only_drops_without_baseline(- db: Path, monkeypatch: pytest.MonkeyPatch,-) -> None:- # Markdown that normalizes to empty (e.g. truncated to a whitespace prefix)- # must be treated as "no fetch", not an empty baseline.- holder = SimpleNamespace(md=" \n\t ", calls=0)- _install_xcrawl(monkeypatch, holder)- item = _make_item(slug="page-xc3", config={"fetch": "xcrawl"})- assert await PageWatchSource().poll(item, db_path=db) == []- assert ms.get_page_watch_state("page-xc3", db_path=db) is None+def test_remove_watchlist_config_key_keeps_other_keys(db: Path) -> None:+ ms.insert_watchlist_item(+ slug="page-k", signal_type="page_watch", target="https://example.com/k",+ config={"fetch": "xcrawl", "label": "K"}, db_path=db,+ )+ item = next(i for i in ms.list_watchlist(db_path=db) if i.slug == "page-k")+ assert item.id is not None+ assert ms.remove_watchlist_config_key(item.id, "fetch", db_path=db) == 1+ again = next(i for i in ms.list_watchlist(db_path=db) if i.slug == "page-k")+ assert again.config_json == {"label": "K"}+ assert ms.remove_watchlist_config_key(item.id + 1000, "fetch", db_path=db) == 0+ with pytest.raises(ValueError):+ ms.remove_watchlist_config_key(item.id, "$.evil", db_path=db)diff --git a/packages/core/tests/unit/test_research_agentic.py b/packages/core/tests/unit/test_research_agentic.pydeleted file mode 100644index 742b9bf9..00000000--- a/packages/core/tests/unit/test_research_agentic.py+++ /dev/null@@ -1,353 +0,0 @@-"""Unit tests for the agentic research scrape loop.--The provider's ``messages_create`` is mocked to return a scripted-multi-turn sequence, and ``xcrawl_client.scrape`` is mocked — no network,-no real model. Covers: scrape→emit happy path (markdown fed back),-mandatory read-before-cite (a 0-scrape emit is rejected and forced to read,-except when scraping is impossible or the iteration cap is hit), scrape-budget cap, no-progress stop, provider failure, iteration cap, the-scrape_tool handler, and the enabled/disabled dispatch branch.-"""-from __future__ import annotations--from types import SimpleNamespace-from typing import Any--import pytest--from openexecutive.monitoring.research import agentic, scrape_tool-from openexecutive.monitoring.research import specialist_research as sr--_FINDING = {- "title": "GM cut Equinox EV price",- "summary": "GM cut the Equinox EV price by $3,000 on 2026-05-20.",- "severity_hint": "medium",- "suggested_audience": "principal",- "confidence": "high",- "relevant_urls": ["https://gmauthority.com/blog/2026/05/cut"],-}---# ---- block / message factories (mimic the provider Message shape) -----------def _text(t: str = "thinking") -> Any:- return SimpleNamespace(type="text", text=t)---def _scrape(url: str, _id: str = "s1") -> Any:- return SimpleNamespace(- type="tool_use", name="scrape_url", id=_id, input={"url": url},- )---def _emit(findings: list[dict], _id: str = "e1") -> Any:- return SimpleNamespace(- type="tool_use", name="emit_research_findings", id=_id,- input={"findings": findings},- )---def _msg(blocks: list[Any]) -> Any:- return SimpleNamespace(content=blocks, stop_reason="tool_use")---class _FakeAgent:- def effective_system_prompt(self) -> str:- return "PERSONA"---class _FakeProvider:- def __init__(self, responses: list[Any]) -> None:- self.responses = list(responses)- self.calls: list[dict[str, Any]] = []-- async def messages_create(self, **kwargs: Any) -> Any:- self.calls.append(kwargs)- idx = len(self.calls) - 1- if idx < len(self.responses):- r = self.responses[idx]- if isinstance(r, Exception):- raise r- return r- return _msg([_text("done")]) # ran out → no emit → loop stops---def _install(- monkeypatch: pytest.MonkeyPatch,- *,- responses: list[Any],- scrape: Any = "ARTICLE FULL TEXT",-) -> _FakeProvider:- monkeypatch.setenv("XCRAWL_ENABLED", "true")- monkeypatch.setenv("XCRAWL_API_KEY", "dummy")- monkeypatch.setenv("RESEARCH_AGENTIC_SCRAPE_ENABLED", "true")- provider = _FakeProvider(responses)- monkeypatch.setattr(agentic, "get_provider", lambda model: provider)-- calls = {"n": 0}-- async def fake_scrape(url: str) -> str | None:- calls["n"] += 1- return scrape-- monkeypatch.setattr(scrape_tool.xcrawl_client, "scrape", fake_scrape)- provider.scrape_calls = calls # type: ignore[attr-defined]- return provider---# --------------------------------------------------------------------- #-# Loop behavior-# --------------------------------------------------------------------- #--async def test_scrapes_then_emits_and_feeds_markdown_back(- monkeypatch: pytest.MonkeyPatch,-) -> None:- provider = _install(monkeypatch, responses=[- _msg([_text("let me read this"), _scrape("https://gmauthority.com/x")]),- _msg([_emit([_FINDING])]),- ])- out = await agentic.run_specialist_agentic("cso", _FakeAgent(), "CTX")-- assert len(out) == 1 and out[0].title == _FINDING["title"]- assert provider.scrape_calls["n"] == 1 # the page was actually scraped- # The scraped markdown was fed back as a tool_result on the 2nd call.- second_call_messages = provider.calls[1]["messages"]- tool_results = second_call_messages[-1]["content"]- assert tool_results[0]["type"] == "tool_result"- assert tool_results[0]["content"] == "ARTICLE FULL TEXT"---async def test_emit_accepted_on_first_turn_when_scrape_disabled(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # max_scrapes == 0 → scraping is impossible, so a turn-1 emit can't be- # forced to read first; accept it rather than trap the model.- monkeypatch.setenv("RESEARCH_SCRAPE_MAX_PER_SPECIALIST", "0")- provider = _install(monkeypatch, responses=[_msg([_emit([_FINDING])])])- out = await agentic.run_specialist_agentic("cfo", _FakeAgent(), "CTX")- assert len(out) == 1- assert provider.scrape_calls["n"] == 0- assert len(provider.calls) == 1---async def test_emit_with_zero_scrapes_is_rejected_then_forced_to_read(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # The core mandatory-scrape contract: the model tries to emit on turn 1- # without having read anything; the loop rejects that emit, the model is- # forced to scrape, and only the post-scrape emit is accepted.- provider = _install(monkeypatch, responses=[- _msg([_emit([_FINDING], "e_early")]), # turn 1: premature emit- _msg([_scrape("https://gmauthority.com/x")]), # turn 2: forced read- _msg([_emit([_FINDING], "e_final")]), # turn 3: grounded emit- ])- out = await agentic.run_specialist_agentic("cso", _FakeAgent(), "CTX")-- assert len(out) == 1 and out[0].title == _FINDING["title"]- assert provider.scrape_calls["n"] == 1 # it was forced to read- assert len(provider.calls) == 3 # rejected, scraped, emitted- # The rejection tool_result must be paired to the premature emit's id.- # (`messages` is one list mutated in place, so scan the final history for- # the tool_result rather than index a specific call.)- final_messages = provider.calls[-1]["messages"]- results = [- block- for m in final_messages if m["role"] == "user"- and isinstance(m["content"], list)- for block in m["content"]- if isinstance(block, dict) and block.get("type") == "tool_result"- ]- rejection = next(r for r in results if r["tool_use_id"] == "e_early")- assert "scrape_url" in rejection["content"]---async def test_emit_accepted_once_a_scrape_has_happened(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # Scrape on turn 1 → scrapes_used > 0 → the turn-2 emit is accepted with- # no further forcing (regression guard for the happy path).- provider = _install(monkeypatch, responses=[- _msg([_scrape("https://gmauthority.com/x")]),- _msg([_emit([_FINDING])]),- ])- out = await agentic.run_specialist_agentic("cmo", _FakeAgent(), "CTX")- assert len(out) == 1- assert provider.scrape_calls["n"] == 1- assert len(provider.calls) == 2 # accepted on the emit turn, no re-loop---async def test_premature_emit_accepted_at_iteration_cap(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # On the final allowed iteration there's no room to force a read, so a- # 0-scrape emit is accepted as the safety valve (don't lose everything).- monkeypatch.setenv("RESEARCH_LOOP_MAX_ITERATIONS", "1")- provider = _install(monkeypatch, responses=[_msg([_emit([_FINDING])])])- out = await agentic.run_specialist_agentic("gc", _FakeAgent(), "CTX")- assert len(out) == 1- assert provider.scrape_calls["n"] == 0- assert len(provider.calls) == 1---async def test_scrape_budget_capped(monkeypatch: pytest.MonkeyPatch) -> None:- monkeypatch.setenv("RESEARCH_SCRAPE_MAX_PER_SPECIALIST", "1")- provider = _install(monkeypatch, responses=[- _msg([_scrape("https://a.example", "s1")]),- _msg([_scrape("https://b.example", "s2")]),- _msg([_emit([_FINDING])]),- ])- out = await agentic.run_specialist_agentic("coo", _FakeAgent(), "CTX")- assert len(out) == 1- # Only the first scrape consumed the budget; the second got the refusal.- assert provider.scrape_calls["n"] == 1- second_scrape_result = provider.calls[2]["messages"][-1]["content"][0]- assert "budget exhausted" in second_scrape_result["content"].lower()---async def test_no_scrape_no_emit_stops(- monkeypatch: pytest.MonkeyPatch,-) -> None:- provider = _install(monkeypatch, responses=[_msg([_text("nothing found")])])- out = await agentic.run_specialist_agentic("gc", _FakeAgent(), "CTX")- assert out == []- assert len(provider.calls) == 1 # stopped, did not loop---async def test_provider_failure_returns_empty(- monkeypatch: pytest.MonkeyPatch,-) -> None:- _install(monkeypatch, responses=[RuntimeError("boom")])- out = await agentic.run_specialist_agentic("cmo", _FakeAgent(), "CTX")- assert out == []---async def test_iteration_cap_without_emit(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("RESEARCH_LOOP_MAX_ITERATIONS", "2")- provider = _install(monkeypatch, responses=[- _msg([_scrape("https://a.example", "s1")]),- _msg([_scrape("https://b.example", "s2")]),- _msg([_scrape("https://c.example", "s3")]), # never reached- ])- out = await agentic.run_specialist_agentic("cpo", _FakeAgent(), "CTX")- assert out == []- assert len(provider.calls) == 2 # capped at 2 iterations---def test_assistant_turn_echoes_thinking_and_collects_scrape() -> None:- # thinking blocks MUST be preserved in the assistant turn (extended- # reasoning) or the next API call 400s; scrape_url must be collected.- thinking = SimpleNamespace(- type="thinking",- model_dump=lambda exclude_none=True: {- "type": "thinking", "thinking": "reasoning...",- },- )- content, scrape_calls = agentic._assistant_turn(- [thinking, _scrape("https://x.example", "s9"), _text("note")]- )- # Order preserved: thinking must precede the tool_use it reasoned about- # (Anthropic rejects a turn where tool_use comes before its thinking).- assert content[0]["type"] == "thinking"- assert [c["id"] for c in scrape_calls] == ["s9"]---# --------------------------------------------------------------------- #-# scrape_tool handler-# --------------------------------------------------------------------- #--async def test_handle_scrape_url_budget_exhausted() -> None:- out = await scrape_tool.handle_scrape_url({"url": "https://x"}, budget_remaining=0)- assert "budget exhausted" in out.lower()---async def test_handle_scrape_url_bad_url() -> None:- out = await scrape_tool.handle_scrape_url({"url": "not-a-url"}, budget_remaining=3)- assert out.startswith("error")---async def test_handle_scrape_url_unreadable(- monkeypatch: pytest.MonkeyPatch,-) -> None:- async def none_scrape(url: str) -> str | None:- return None-- monkeypatch.setattr(scrape_tool.xcrawl_client, "scrape", none_scrape)- out = await scrape_tool.handle_scrape_url(- {"url": "https://dead.example"}, budget_remaining=3,- )- assert "could not read" in out.lower()---async def test_handle_scrape_url_caps_markdown(- monkeypatch: pytest.MonkeyPatch,-) -> None:- big = "x" * 50_000-- async def big_scrape(url: str) -> str | None:- return big-- monkeypatch.setattr(scrape_tool.xcrawl_client, "scrape", big_scrape)- out = await scrape_tool.handle_scrape_url(- {"url": "https://big.example"}, budget_remaining=3,- )- assert len(out) == scrape_tool._SCRAPE_RESULT_CHARS---# --------------------------------------------------------------------- #-# Dispatch branch in research_one_specialist-# --------------------------------------------------------------------- #--async def test_branch_uses_agentic_when_enabled(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "true")- monkeypatch.setenv("XCRAWL_API_KEY", "dummy")- monkeypatch.setenv("RESEARCH_AGENTIC_SCRAPE_ENABLED", "true")- sentinel: list[Any] = []-- async def fake_agentic(slug: str, agent: Any, ctx: str) -> list[Any]:- sentinel.append(slug)- return ["AGENTIC"] # type: ignore[list-item]-- monkeypatch.setattr(agentic, "run_specialist_agentic", fake_agentic)- out = await sr.research_one_specialist("cso", _FakeAgent(), "CTX")- assert out == ["AGENTIC"]- assert sentinel == ["cso"]---async def test_branch_single_shot_when_disabled(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("RESEARCH_AGENTIC_SCRAPE_ENABLED", "false")-- async def boom_agentic(*a: Any, **k: Any) -> list[Any]:- raise AssertionError("must use single-shot when disabled")-- monkeypatch.setattr(agentic, "run_specialist_agentic", boom_agentic)-- class _SingleShotAgent:- def effective_system_prompt(self) -> str:- return "P"-- async def analyze_with_tools(self, *a: Any, **k: Any) -> Any:- return _msg([_emit([_FINDING])])-- out = await sr.research_one_specialist("cfo", _SingleShotAgent(), "CTX")- assert len(out) == 1 and out[0].title == _FINDING["title"]diff --git a/packages/core/tests/unit/test_research_verification.py b/packages/core/tests/unit/test_research_verification.pydeleted file mode 100644index 4d394e5d..00000000--- a/packages/core/tests/unit/test_research_verification.py+++ /dev/null@@ -1,297 +0,0 @@-"""Unit tests for the post-dedup research finding verification pass.--Network (xcrawl scrape) and the verify LLM call are both mocked — these-tests exercise selection/cap/gating and the verdict→confidence mapping.-"""-from __future__ import annotations--from types import SimpleNamespace-from typing import Any--import pytest--from openexecutive.alerts.models import AlertSeverity-from openexecutive.monitoring.research import verification as v-from openexecutive.monitoring.research.models import ResearchFinding---def _finding(- *,- title: str = "T",- confidence: str = "high",- severity: AlertSeverity = AlertSeverity.MEDIUM,- urls: list[str] | None = None,-) -> ResearchFinding:- return ResearchFinding(- title=title,- summary=f"summary for {title}",- severity_hint=severity,- confidence=confidence, # type: ignore[arg-type]- relevant_urls=urls if urls is not None else ["https://src.example/a"],- )---def _enable(monkeypatch: pytest.MonkeyPatch) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "true")- monkeypatch.setenv("XCRAWL_API_KEY", "dummy") # mocked; never used live- monkeypatch.setenv("EXTERNAL_RESEARCH_VERIFY_ENABLED", "true")---def _mock(- monkeypatch: pytest.MonkeyPatch,- *,- scrape: str | None = "page text",- scrape_status: str | None = None,- verdicts: dict[str, str] | None = None,- verdict_raises: bool = False,-) -> None:- """Mock the xcrawl scrape AND the verify model's provider call (one level- below _llm_verdict), so the real _verify_one → _llm_verdict →- _parse_verdict → confidence-mapping path is exercised. ``verdicts`` maps a- finding title to the verdict the model 'returns'; the sentinel 'no_tool'- makes the model return a text-only message (→ parsed as 'unknown').- ``scrape_status`` overrides the ScrapeOutcome status (defaults: a string- body → 'ok', None → 'empty'); pass 'error' to simulate an xcrawl infra- failure."""- status = scrape_status or ("ok" if scrape is not None else "empty")-- async def fake_scrape_detailed(url: str) -> Any:- return v.xcrawl_client.ScrapeOutcome(scrape, status) # type: ignore[arg-type]-- async def fake_analyze(self: Any, user_content: str, **kwargs: Any) -> Any:- if verdict_raises:- raise RuntimeError("provider boom")- verdict = "confirmed"- for title, vd in (verdicts or {}).items():- if f"TITLE: {title}" in user_content:- verdict = vd- break- if verdict == "no_tool":- return SimpleNamespace(content=[SimpleNamespace(type="text")])- return SimpleNamespace(content=[- SimpleNamespace(- type="tool_use", name="emit_verification",- input={"verdict": verdict},- ),- ])-- monkeypatch.setattr(v.xcrawl_client, "scrape_detailed", fake_scrape_detailed)- monkeypatch.setattr(v._VerifyAgent, "analyze_with_tools", fake_analyze)---# --------------------------------------------------------------------- #-# Gating-# --------------------------------------------------------------------- #--async def test_verify_disabled_is_passthrough(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "true")- monkeypatch.setenv("XCRAWL_API_KEY", "dummy")- monkeypatch.setenv("EXTERNAL_RESEARCH_VERIFY_ENABLED", "false")-- async def boom_scrape(url: str) -> Any:- raise AssertionError("must not scrape when verification disabled")-- monkeypatch.setattr(v.xcrawl_client, "scrape_detailed", boom_scrape)- findings = [_finding()]- out = await v.verify_findings(findings)- assert out is findings # unchanged, no work---async def test_xcrawl_disabled_is_passthrough(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "false")- monkeypatch.setenv("EXTERNAL_RESEARCH_VERIFY_ENABLED", "true")- out = await v.verify_findings([_finding()])- assert out[0].verification is None---# --------------------------------------------------------------------- #-# Verdict → confidence mapping-# --------------------------------------------------------------------- #--async def test_confirmed_keeps_confidence(- monkeypatch: pytest.MonkeyPatch,-) -> None:- _enable(monkeypatch)- _mock(monkeypatch, verdicts={"T": "confirmed"})- out = await v.verify_findings([_finding(confidence="medium")])- assert out[0].confidence == "medium"- assert out[0].verification == "confirmed"---async def test_contradicted_sets_low(monkeypatch: pytest.MonkeyPatch) -> None:- _enable(monkeypatch)- _mock(monkeypatch, verdicts={"T": "contradicted"})- out = await v.verify_findings([_finding(confidence="high")])- assert out[0].confidence == "low"- assert out[0].verification == "contradicted"---async def test_unsupported_demotes_one_notch(- monkeypatch: pytest.MonkeyPatch,-) -> None:- _enable(monkeypatch)- _mock(monkeypatch, verdicts={"T": "unsupported"})- out = await v.verify_findings([_finding(confidence="high")])- assert out[0].confidence == "medium"- assert out[0].verification == "unsupported"---async def test_empty_page_is_source_unreachable(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # Fetched OK but no content → genuinely dead/unreadable URL → demote.- _enable(monkeypatch)- _mock(monkeypatch, scrape=None, scrape_status="empty")- out = await v.verify_findings([_finding(confidence="medium")])- assert out[0].confidence == "low"- assert out[0].verification == "source_unreachable"---async def test_scrape_infra_error_leaves_unchanged(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # xcrawl itself failed (down / 401 / 429 / timeout) — NOT a verdict on the- # source. A verify-pass outage must never demote a real finding.- _enable(monkeypatch)- _mock(monkeypatch, scrape=None, scrape_status="error")- out = await v.verify_findings([_finding(confidence="medium")])- assert out[0].confidence == "medium" # untouched- assert out[0].verification is None # not tagged — we couldn't check---async def test_scrape_disabled_leaves_unchanged(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # XCRAWL_ENABLED=true but no key → scrape_detailed returns "disabled". The- # verify_findings gate only checks the enabled flag, so this reaches- # _verify_one — a config gap must not demote a real finding.- _enable(monkeypatch)- _mock(monkeypatch, scrape=None, scrape_status="disabled")- out = await v.verify_findings([_finding(confidence="medium")])- assert out[0].confidence == "medium" # untouched- assert out[0].verification is None---async def test_unknown_verdict_keeps_confidence_tags_inconclusive(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # Model returned no usable tool verdict → 'unknown' → confidence untouched,- # but tagged so the run's verified-count reflects the attempt.- _enable(monkeypatch)- _mock(monkeypatch, verdicts={"T": "no_tool"})- out = await v.verify_findings([_finding(confidence="high")])- assert out[0].confidence == "high"- assert out[0].verification == "inconclusive"---async def test_verify_failure_leaves_unchanged(- monkeypatch: pytest.MonkeyPatch,-) -> None:- _enable(monkeypatch)- _mock(monkeypatch, verdict_raises=True)- out = await v.verify_findings([_finding(confidence="high")])- assert out[0].confidence == "high"- assert out[0].verification is None---# --------------------------------------------------------------------- #-# Selection / cap-# --------------------------------------------------------------------- #--async def test_no_url_finding_passes_through(- monkeypatch: pytest.MonkeyPatch,-) -> None:- _enable(monkeypatch)-- async def boom_scrape(url: str) -> Any:- raise AssertionError("no cited URL → must not scrape")-- monkeypatch.setattr(v.xcrawl_client, "scrape_detailed", boom_scrape)- out = await v.verify_findings([_finding(urls=[])])- assert out[0].verification is None---async def test_low_confidence_finding_is_not_verified(- monkeypatch: pytest.MonkeyPatch,-) -> None:- # A finding already at 'low' is dropped by the downstream filter anyway —- # don't spend a scrape/LLM on it, and don't let it skew the accounting.- _enable(monkeypatch)-- async def boom_scrape(url: str) -> Any:- raise AssertionError("low-confidence finding must not be verified")-- monkeypatch.setattr(v.xcrawl_client, "scrape_detailed", boom_scrape)- out = await v.verify_findings([_finding(confidence="low")])- assert out[0].verification is None---async def test_cap_verifies_highest_severity_first(- monkeypatch: pytest.MonkeyPatch,-) -> None:- _enable(monkeypatch)- monkeypatch.setenv("RESEARCH_VERIFY_MAX_FINDINGS", "1")- _mock(monkeypatch, verdicts={"low-sev": "unsupported", "urgent-sev": "unsupported"})- findings = [- _finding(title="low-sev", severity=AlertSeverity.LOW, confidence="high"),- _finding(title="urgent-sev", severity=AlertSeverity.URGENT, confidence="high"),- ]- out = await v.verify_findings(findings)- verified = [f for f in out if f.verification is not None]- assert len(verified) == 1- assert verified[0].title == "urgent-sev" # highest severity got the budget---# --------------------------------------------------------------------- #-# _parse_verdict + helpers-# --------------------------------------------------------------------- #--def _msg(blocks: list[Any]) -> Any:- return SimpleNamespace(content=blocks)---def _tool_block(name: str, inp: dict) -> Any:- return SimpleNamespace(type="tool_use", name=name, input=inp)---def test_parse_verdict_reads_tool_use() -> None:- for verdict in ("confirmed", "contradicted", "unsupported"):- msg = _msg([_tool_block("emit_verification", {"verdict": verdict})])- assert v._parse_verdict(msg) == verdict---def test_parse_verdict_unknown_when_missing_or_bad() -> None:- assert v._parse_verdict(_msg([])) == "unknown"- assert v._parse_verdict(_msg([SimpleNamespace(type="text")])) == "unknown"- assert v._parse_verdict(- _msg([_tool_block("emit_verification", {"verdict": "maybe"})])- ) == "unknown"- assert v._parse_verdict(- _msg([_tool_block("other_tool", {"verdict": "confirmed"})])- ) == "unknown"---def test_demote_and_first_url() -> None:- assert v._demote("high") == "medium"- assert v._demote("medium") == "low"- assert v._demote("low") == "low"- assert v._first_url(_finding(urls=["not-a-url", "https://ok.example/x"])) == \- "https://ok.example/x"- assert v._first_url(_finding(urls=["ftp://x", "mailto:y"])) is Nonediff --git a/packages/core/tests/unit/test_target_validation.py b/packages/core/tests/unit/test_target_validation.pyindex 66f6f12b..d449d555 100644--- a/packages/core/tests/unit/test_target_validation.py+++ b/packages/core/tests/unit/test_target_validation.py@@ -1,8 +1,9 @@"""Unit tests for insert-time watchlist target validation.Covers: non-rss passthrough (no network), valid-feed keep, non-feed →-page_watch conversion, non-feed-non-scrapeable reject, transient-failure-keep, and the 4xx → reject path. All network is mocked.+page_watch conversion when the fetched page has readable text, non-feed+reject when it has none, transient-failure keep, and the 4xx → reject path.+All network is mocked."""from __future__ import annotations@@ -12,10 +13,7 @@ import httpximport pytestfrom openexecutive.monitoring import target_validation as tv-from openexecutive.monitoring.models import (- PAGE_WATCH_FETCH_KEY,- PAGE_WATCH_FETCH_XCRAWL,-)+from openexecutive.monitoring.models import PAGE_WATCH_FETCH_KEYfrom openexecutive.monitoring.sources import page_watchfrom openexecutive.monitoring.sources._http import FetchOverflowError@@ -29,25 +27,24 @@ _EMPTY_FEED = (b'<?xml version="1.0"?><rss version="2.0"><channel>'b"<title>Brand new feed</title></channel></rss>")-_NOT_FEED = b"<html><body><h1>Pricing</h1><p>not a feed</p></body></html>"+_NOT_FEED = (+ b"<html><body><h1>Pricing</h1><p>Our Pro plan is now priced per seat, "+ b"with annual billing available for teams of ten or more. Enterprise "+ b"customers get SSO, audit logs, a dedicated success manager and a "+ b"99.9% uptime commitment. Contact sales for volume pricing.</p></body></html>"+)+# Answers, but reduces to no visible text (a JS-only shell).+_EMPTY_SHELL = b"<html><head><script>boot()</script></head><body></body></html>"+# Answers with a one-line page: too little text to be worth watching.+_THIN_PAGE = b"<html><body>Error 404 - not found</body></html>"+# A binary body (PDF magic + NUL bytes) that would decode to garbage.+_BINARY = b"%PDF-1.7\x00\x00" + bytes(range(256)) * 8-def _install(- monkeypatch: pytest.MonkeyPatch,- *,- fetch: Any,- scrape_result: str | None = None,-) -> None:+def _install(monkeypatch: pytest.MonkeyPatch, *, fetch: Any) -> None:monkeypatch.setattr(tv, "validate_target_url", lambda u: (True, ""))monkeypatch.setattr(tv, "fetch_bounded", fetch)- async def fake_scrape(url: str) -> str | None:- return scrape_result-- monkeypatch.setattr(- "openexecutive.integrations.xcrawl_client.scrape", fake_scrape,- )-def _fetch_returning(body: bytes) -> Any:async def _f(url: str, max_bytes: int, **kw: object) -> bytes:@@ -87,20 +84,24 @@ async def test_valid_feed_kept(monkeypatch: pytest.MonkeyPatch) -> None:@pytest.mark.asyncio-async def test_non_feed_scrapeable_converts_to_page_watch(+async def test_non_feed_readable_page_converts_to_page_watch(monkeypatch: pytest.MonkeyPatch,) -> None:- _install(- monkeypatch, fetch=_fetch_returning(_NOT_FEED),- scrape_result="# Pricing\n\ncontent",- )+ calls: list[str] = []++ async def fetch(url: str, max_bytes: int, **kw: object) -> bytes:+ calls.append(url)+ return _NOT_FEED++ _install(monkeypatch, fetch=fetch)st, target, config = await tv.validate_and_normalize_target("rss", "https://news.example/page", {"feed_label": "News"},)assert st == page_watch.PageWatchSource.kind # "page_watch"- # Drift guard: the conversion must match the shared xcrawl sentinel that- # page_watch reads (both key and value).- assert config[PAGE_WATCH_FETCH_KEY] == PAGE_WATCH_FETCH_XCRAWL+ # The body the feed check fetched decides the conversion — one fetch.+ assert calls == ["https://news.example/page"]+ # No fetch-source marker is written; the plain fetcher is the only path.+ assert PAGE_WATCH_FETCH_KEY not in config# feed_label is remapped to the label key page_watch reads.assert config.get("label") == "News"assert "feed_label" not in config@@ -125,16 +126,17 @@ async def test_empty_body_is_not_feed_no_crash() -> None:# HTTP 200 with a zero-byte body: feedparser yields a dict with no# `version` key — must classify as not_feed, not raise AttributeError.- _install(monkeypatch, fetch=_fetch_returning(b""), scrape_result=None)+ _install(monkeypatch, fetch=_fetch_returning(b""))with pytest.raises(tv.WatchlistTargetError):await tv.validate_and_normalize_target("rss", "https://empty.example", {})@pytest.mark.asyncio-async def test_non_feed_not_scrapeable_rejected(- monkeypatch: pytest.MonkeyPatch,[email protected]("body", [_EMPTY_SHELL, _THIN_PAGE, _BINARY], ids=["shell", "thin", "binary"])+async def test_non_feed_without_readable_text_rejected(+ monkeypatch: pytest.MonkeyPatch, body: bytes,) -> None:- _install(monkeypatch, fetch=_fetch_returning(_NOT_FEED), scrape_result=None)+ _install(monkeypatch, fetch=_fetch_returning(body))with pytest.raises(tv.WatchlistTargetError) as ei:await tv.validate_and_normalize_target("rss", "https://dead.example/x", {},@@ -151,7 +153,7 @@ async def test_404_status_is_not_feed_then_rejected(request=httpx.Request("GET", "https://x.example"),response=httpx.Response(404),)- _install(monkeypatch, fetch=_fetch_raising(exc), scrape_result=None)+ _install(monkeypatch, fetch=_fetch_raising(exc))with pytest.raises(tv.WatchlistTargetError):await tv.validate_and_normalize_target("rss", "https://x.example", {})@@ -171,10 +173,8 @@ async def test_transient_failure_keeps_rss_unverified(async def test_oversized_body_is_not_feed(monkeypatch: pytest.MonkeyPatch,) -> None:- _install(- monkeypatch, fetch=_fetch_raising(FetchOverflowError()),- scrape_result=None,- )+ _install(monkeypatch, fetch=_fetch_raising(FetchOverflowError()))+ # Oversized: no body to inspect, so it cannot become a page_watch either.with pytest.raises(tv.WatchlistTargetError):await tv.validate_and_normalize_target("rss", "https://big.example", {})@@ -188,13 +188,7 @@ async def test_ssrf_rejected_target_is_not_feed(async def boom(*a: object, **k: object) -> bytes:raise AssertionError("must not fetch an SSRF-rejected target")- async def boom_scrape(url: str) -> str | None:- raise AssertionError("must not hand an SSRF-rejected target to xcrawl")-monkeypatch.setattr(tv, "fetch_bounded", boom)- monkeypatch.setattr(- "openexecutive.integrations.xcrawl_client.scrape", boom_scrape,- )- # Rejected outright as "blocked" — neither fetched locally nor scraped.+ # Rejected outright as "blocked" — never fetched.with pytest.raises(tv.WatchlistTargetError):await tv.validate_and_normalize_target("rss", "http://169.254.169.254", {})diff --git a/packages/core/tests/unit/test_watch_policy.py b/packages/core/tests/unit/test_watch_policy.pyindex 6e2e974a..ce07f060 100644--- a/packages/core/tests/unit/test_watch_policy.py+++ b/packages/core/tests/unit/test_watch_policy.py@@ -215,15 +215,15 @@ def _initech_finding(**kw: Any) -> ResearchFinding:def test_entity_not_in_company_data_is_never_direct() -> None:p = _proposal(grounding_entity="Initech", target="https://initech.com/feed.xml")- d = wp.classify(p, _initech_finding(verification="confirmed", source_specialist="cso,cfo"), _ctx())+ d = wp.classify(p, _initech_finding(source_specialist="cso,cfo"), _ctx())assert d.tier == wp.TIER_SUGGEST- assert d.score == 3 # own-source +1 needs an entity; high +1, verified +1, consensus +1+ assert d.score == 2 # own-source +1 needs an entity; high +1, consensus +1def test_medium_confidence_grounded_needs_more_corroboration() -> None:d = wp.classify(_proposal(), _finding(confidence="medium"), _ctx())assert d.tier == wp.TIER_SUGGEST and d.score == 3- d2 = wp.classify(_proposal(), _finding(confidence="medium", verification="confirmed"), _ctx())+ d2 = wp.classify(_proposal(), _finding(confidence="medium", source_specialist="cso,cfo"), _ctx())assert d2.tier == wp.TIER_DIRECT and d2.score == 4@@ -237,7 +237,7 @@ def test_unsure_only_downgrades() -> None:def test_query_is_always_a_suggestion() -> None:p = _proposal(slug="q-acme", signal_type="query", target="Acme Corp pricing", grounding_entity="Acme Corp")- d = wp.classify(p, _finding(verification="confirmed"), _ctx())+ d = wp.classify(p, _finding(source_specialist="cso,cfo"), _ctx())assert d.tier == wp.TIER_SUGGEST and "standing web queries" in d.reasons[-1]@@ -568,7 +568,7 @@ def test_direct_add_needs_a_finding_that_cites_the_source() -> None:def test_finding_points_need_a_finding_about_this_source() -> None:# A strong finding about Acme lends nothing to an unrelated attacker URL.p = _proposal(target="https://totally-unrelated.attacker.net/feed", grounding_entity="Acme Corp")- d = wp.classify(p, _finding(verification="confirmed", source_specialist="cso,cfo"), _ctx())+ d = wp.classify(p, _finding(source_specialist="cso,cfo"), _ctx())assert d.score == 2 and d.tier == wp.TIER_SUGGESTassert "does not mention this source" in " ".join(d.reasons)# A finding that names the ticker does support a ticker watch on it.@@ -580,7 +580,7 @@ def test_direct_add_requires_the_entitys_own_source() -> None:# A well-corroborated third-party page about a competitor is the# principal's call, never a direct add.p = _proposal(target="https://news.example.com/acme-feed.xml", grounding_entity="Acme Corp")- f = _finding(verification="confirmed", source_specialist="cso,cfo",+ f = _finding(source_specialist="cso,cfo",relevant_urls=["https://news.example.com/acme-pricing"])d = wp.classify(p, f, _ctx())assert d.score >= wp.DIRECT_THRESHOLD and d.tier == wp.TIER_SUGGEST@@ -590,7 +590,7 @@ def test_direct_add_requires_the_entitys_own_source() -> None:def test_initiative_or_priority_grounding_is_never_direct() -> None:p = _proposal(slug="rss-helios", target="https://helios.com/feed.xml", grounding_entity="Helios API")f = _finding(title="Helios API launch", summary="Launch Helios API shipped.",- verification="confirmed", relevant_urls=["https://helios.com/launch"])+ relevant_urls=["https://helios.com/launch"])d = wp.classify(p, f, _ctx())assert d.grounding_kind == wp.KIND_INITIATIVE and d.score >= wp.DIRECT_THRESHOLDassert d.tier == wp.TIER_SUGGEST and "needs a named competitor" in d.reasons[-1]@@ -769,7 +769,7 @@ def test_department_scope_grounding_is_suggestion_only(db: Path) -> None:p = _proposal(slug="rss-expense", signal_type="rss", target="https://expensecards.example/feed",grounding_entity="Expense card programs")f = _finding(title="Expense card programs shift", summary="New expense card programs launched.",- relevant_urls=["https://expensecards.example/feed"], verification="confirmed",+ relevant_urls=["https://expensecards.example/feed"],source_specialist="cfo,coo")d = wp.classify(p, f, _dept_ctx())assert d.tier == wp.TIER_SUGGEST and d.department == "finance"@@ -778,7 +778,7 @@ def test_department_scope_grounding_is_suggestion_only(db: Path) -> None:own = _proposal(slug="rss-programs", signal_type="rss", target="https://expense.example/feed",grounding_entity="Expense card programs")own_f = _finding(title="programs", summary="s", relevant_urls=["https://expense.example/feed"],- verification="confirmed", source_specialist="cfo,coo")+ source_specialist="cfo,coo")own_d = wp.classify(own, own_f, _dept_ctx())assert own_d.tier == wp.TIER_SUGGEST and any("grounded only in a department_scope" in r for r in own_d.reasons)out = wp.apply_proposals([p], [f], _dept_ctx(), db_path=db)@@ -1097,7 +1097,7 @@ def test_auto_link_prefers_the_finding_that_cites_the_source() -> None:ctx = _ctx()findings = [_finding(title="Globex beats, Acme called a laggard", summary="Acme lags Globex; Initech too.",- relevant_urls=["https://globex.com/ir/q3"], verification="confirmed",+ relevant_urls=["https://globex.com/ir/q3"],source_specialist="cfo,cso"),_finding(title="Acme Q3", summary="ACME reported Q3.", relevant_urls=["https://acme.com/ir/q3"],confidence="medium"),@@ -1115,7 +1115,7 @@ def test_auto_link_prefers_the_finding_that_cites_the_source() -> None:_finding(title="GM recall", summary="GM recalled the Blazer EV.", relevant_urls=["https://news.gm.com/x"],confidence="medium"),_finding(title="GM news", summary="GM slashes prices.", relevant_urls=["https://gm.co.ke/news"],- verification="confirmed", source_specialist="cso,cfo"),+ source_specialist="cso,cfo"),]gm = _proposal(slug="stock-gm", signal_type="stock", target="GM", grounding_entity="GM", finding_index=None)assert wp.auto_link_finding(gm, gm_findings, pctx) == 0diff --git a/packages/core/tests/unit/test_xcrawl_client.py b/packages/core/tests/unit/test_xcrawl_client.pydeleted file mode 100644index 23319cb2..00000000--- a/packages/core/tests/unit/test_xcrawl_client.py+++ /dev/null@@ -1,258 +0,0 @@-"""Unit tests for the xcrawl REST client.--Covers the gate (disabled / no-key → empty), the nested-envelope parser,-and graceful degradation on transport failure. No network: ``httpx`` is-monkeypatched with a fake client.-"""-from __future__ import annotations--from typing import Any--import pytest--from openexecutive.integrations import xcrawl_client---class _FakeResponse:- def __init__(self, body: Any, *, raise_exc: Exception | None = None) -> None:- self._body = body- self._raise = raise_exc-- def raise_for_status(self) -> None:- if self._raise is not None:- raise self._raise-- def json(self) -> Any:- return self._body---class _FakeClient:- """Stand-in for httpx.AsyncClient. Configure class attrs per test."""-- body: Any = {}- raise_on_post: Exception | None = None- raise_on_status: Exception | None = None- last_call: dict[str, Any] = {}-- def __init__(self, *args: Any, **kwargs: Any) -> None:- pass-- async def __aenter__(self) -> _FakeClient:- return self-- async def __aexit__(self, *args: Any) -> bool:- return False-- async def post(- self, url: str, *, headers: Any = None, json: Any = None,- ) -> _FakeResponse:- _FakeClient.last_call = {"url": url, "headers": headers, "json": json}- if _FakeClient.raise_on_post is not None:- raise _FakeClient.raise_on_post- return _FakeResponse(_FakeClient.body, raise_exc=_FakeClient.raise_on_status)---_NESTED_BODY = {- "status": "completed",- "total_credits_used": 2,- "data": {- "data": [- {"position": 1, "title": "USTR finalizes tariffs",- "description": "Section 301 increases", "url": "https://a.example/1"},- {"position": 2, "title": "BYD news",- "description": "EV exports", "url": "https://b.example/2"},- {"position": 3, "title": "no url row", "description": "skip me"},- ]- },-}---def _enabled(monkeypatch: pytest.MonkeyPatch) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "true")- monkeypatch.setenv("XCRAWL_API_KEY", "test-key")- _FakeClient.body = {}- _FakeClient.raise_on_post = None- _FakeClient.raise_on_status = None- _FakeClient.last_call = {}- monkeypatch.setattr(xcrawl_client.httpx, "AsyncClient", _FakeClient)---async def test_search_disabled_returns_empty(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "false")- monkeypatch.setenv("XCRAWL_API_KEY", "test-key")- assert await xcrawl_client.search("anything") == []---async def test_search_enabled_without_key_returns_empty(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "true")- # Empty env value overrides any key supplied by the repo's .env file —- # otherwise get_settings() reads the real key and this would make a- # live call. Empty key trips the disabled gate before any HTTP.- monkeypatch.setenv("XCRAWL_API_KEY", "")- assert await xcrawl_client.search("anything") == []---async def test_search_blank_query_returns_empty(_enabled: None) -> None:- assert await xcrawl_client.search(" ") == []---async def test_search_parses_nested_envelope(_enabled: None) -> None:- _FakeClient.body = _NESTED_BODY- results = await xcrawl_client.search("ev tariffs", limit=5)- # The third row has no URL and must be dropped.- assert [r.url for r in results] == [- "https://a.example/1", "https://b.example/2",- ]- assert results[0].title == "USTR finalizes tariffs"- assert results[0].snippet == "Section 301 increases"- # Endpoint + auth header wired correctly.- assert _FakeClient.last_call["url"].endswith("/search")- assert _FakeClient.last_call["headers"]["Authorization"] == "Bearer test-key"- assert _FakeClient.last_call["json"]["query"] == "ev tariffs"---async def test_search_respects_limit(_enabled: None) -> None:- _FakeClient.body = _NESTED_BODY- results = await xcrawl_client.search("ev tariffs", limit=1)- assert len(results) == 1---async def test_search_transport_failure_returns_empty(_enabled: None) -> None:- _FakeClient.raise_on_post = RuntimeError("connection reset")- assert await xcrawl_client.search("ev tariffs") == []---async def test_search_http_status_error_returns_empty(_enabled: None) -> None:- # raise_for_status() raising (e.g. 401) is handled inside the client- # context and degrades to [].- _FakeClient.raise_on_status = RuntimeError("HTTP 401 Unauthorized")- assert await xcrawl_client.search("ev tariffs") == []---async def test_search_limit_zero_returns_empty(_enabled: None) -> None:- # limit<=0 must NOT silently return a charged-but-dropped result.- _FakeClient.body = _NESTED_BODY- assert await xcrawl_client.search("ev tariffs", limit=0) == []---async def test_scrape_disabled_returns_none(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "false")- monkeypatch.setenv("XCRAWL_API_KEY", "test-key")- assert await xcrawl_client.scrape("https://x.example") is None---async def test_scrape_blank_url_returns_none(_enabled: None) -> None:- assert await xcrawl_client.scrape(" ") is None---async def test_scrape_returns_markdown(_enabled: None) -> None:- _FakeClient.body = {"status": "completed", "total_credits_used": 1,- "data": {"markdown": "# Title\n\nbody text",- "metadata": {"status_code": 200}}}- md = await xcrawl_client.scrape("https://x.example/page")- assert md == "# Title\n\nbody text"- assert _FakeClient.last_call["url"].endswith("/scrape")- assert _FakeClient.last_call["json"]["url"] == "https://x.example/page"- assert _FakeClient.last_call["json"]["output"]["formats"] == ["markdown"]---async def test_scrape_blank_markdown_is_none(_enabled: None) -> None:- _FakeClient.body = {"data": {"markdown": " \n "}}- assert await xcrawl_client.scrape("https://x.example") is None---async def test_scrape_transport_failure_returns_none(_enabled: None) -> None:- _FakeClient.raise_on_post = RuntimeError("timeout")- assert await xcrawl_client.scrape("https://x.example") is None---# --------------------------------------------------------------------- #-# scrape_detailed — status distinguishes infra failure from empty page-# --------------------------------------------------------------------- #--async def test_scrape_detailed_ok(_enabled: None) -> None:- _FakeClient.body = {"data": {"markdown": "# Title\n\nbody"}}- out = await xcrawl_client.scrape_detailed("https://x.example/page")- assert out == xcrawl_client.ScrapeOutcome("# Title\n\nbody", "ok")---async def test_scrape_detailed_empty_page(_enabled: None) -> None:- # Fetched OK (200) but no usable markdown → 'empty', not 'error'.- _FakeClient.body = {"data": {"markdown": " \n "}}- out = await xcrawl_client.scrape_detailed("https://x.example")- assert out.markdown is None- assert out.status == "empty"---async def test_scrape_detailed_transport_error(_enabled: None) -> None:- _FakeClient.raise_on_post = RuntimeError("connection reset")- out = await xcrawl_client.scrape_detailed("https://x.example")- assert out.markdown is None- assert out.status == "error"---async def test_scrape_detailed_http_status_error(_enabled: None) -> None:- # raise_for_status() raising (401/429) is an infra error, not an empty page.- _FakeClient.raise_on_status = RuntimeError("HTTP 429 Too Many Requests")- out = await xcrawl_client.scrape_detailed("https://x.example")- assert out.status == "error"---async def test_scrape_detailed_disabled(- monkeypatch: pytest.MonkeyPatch,-) -> None:- monkeypatch.setenv("XCRAWL_ENABLED", "false")- monkeypatch.setenv("XCRAWL_API_KEY", "test-key")- out = await xcrawl_client.scrape_detailed("https://x.example")- assert out == xcrawl_client.ScrapeOutcome(None, "disabled")---async def test_scrape_detailed_blank_url_disabled(_enabled: None) -> None:- out = await xcrawl_client.scrape_detailed(" ")- assert out.status == "disabled"---def test_parse_scrape_shapes() -> None:- assert xcrawl_client._parse_scrape({"data": {"markdown": "x"}}) == "x"- assert xcrawl_client._parse_scrape({"data": {"data": {"markdown": "y"}}}) == "y"- assert xcrawl_client._parse_scrape({}) is None- assert xcrawl_client._parse_scrape({"data": None}) is None- assert xcrawl_client._parse_scrape({"data": {"markdown": ""}}) is None---def test_parse_search_tolerates_flat_list() -> None:- body = {"data": [{"title": "t", "url": "https://x.example", "description": "d"}]}- results = xcrawl_client._parse_search(body)- assert len(results) == 1 and results[0].url == "https://x.example"---def test_parse_search_handles_garbage() -> None:- assert xcrawl_client._parse_search({}) == []- assert xcrawl_client._parse_search({"data": None}) == []- assert xcrawl_client._parse_search({"data": {"data": "nope"}}) == []