Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(monitoring): check every redirect hop and pin the fetched IP (#507)

Rufus49e08727195c
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index ab108d48..2848556e 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -5814,6 +5814,16 @@ external_monitoring:
research grounding is the provider web_search path (Anthropic native /
OpenRouter web plugin). A deployment that still sets the removed
variables is unaffected (unknown env vars are ignored).
+ The guard holds across redirects and DNS changes: fetch_bounded
+ (`monitoring/sources/_http.py`) follows redirects itself, at most
+ MAX_REDIRECTS (3), and resolves and checks every hop, the first
+ included, against the same public-address rule as
+ validate_target_url. It then connects to the address it checked,
+ keeping the hostname in the Host header and the TLS server name, so a
+ public URL that redirects to 127.0.0.1, 169.254.169.254 or a private
+ IPv6 address, or a name that resolves differently on a second lookup,
+ is never fetched. A refused hop raises BlockedTargetError, an
+ httpx.HTTPError, so adapters log it and skip the tick.
pipeline: |
openexecutive.monitoring.pipeline — `run_external_monitor_scan(now)`
is the per-tick entry point. It walks
diff --git a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
index 01797d1b..30f75225 100644
--- a/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
+++ b/packages/core/openexecutive/architecture/prebuilt/external_monitoring.json
@@ -1,7 +1,7 @@
{
"section_id": "external_monitoring",
"title": "External Monitoring",
- "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page, fetched with the keyless httpx fetcher. A row written before the external scrape path was removed still carries `config.fetch = \"xcrawl\"`; the adapter treats that as a stale baseline, re-captures it from the plain fetch without emitting a signal, and clears the marker.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a `page_watch` when the body the feed check already fetched reduces to readable text, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never fetched. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nThe run itself: seven specialists (or the subset named by `RESEARCH_SPECIALISTS`) each make one call with the basic `web_search` server tool, capped at `RESEARCH_WEB_SEARCH_MAX_USES` searches (default 3, separate from the chat knob; via OpenRouter, Claude or not, the tool becomes OpenRouter's `openrouter:web_search` server tool with the same cap and domain lists, and the search count is recorded there too), and emit at most three findings — the cap sits in the tool schema and is applied again after parsing. Every specialist sees what department heads asked to have watched (a `DEPARTMENT WATCH INTERESTS` block) and the ten most recent decisions, and may ground a finding in either. Findings are deduplicated, low-confidence ones dropped, then the Executive routes the rest; the routing pass cannot start a workflow (`run_workflow` is withheld, `suggest_workflow` is not). There is no external fetch or verify service: every page read is the on-box fetcher. The periodic run is considered every `WATCHLIST_RESEARCH_INTERVAL_MINUTES` (default 360) and skipped when nothing changed. Every model call the run makes is recorded as a `cache_event` audit row tagged with the run id, and the run's result and its `watchlist_research_periodic_ran` row carry a `usage` block (calls, tokens, searches, per source), so what a run did is readable from the audit log.\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (each list entry is parsed to its name: \"Tesla (TSLA) — Model Y is the benchmark\" grounds as `Tesla` plus the ticker `TSLA`, \"GM / Chevrolet (…)\" as both names, and \"Brex (brex.com)\" pins Brex's own domains so only those sites count as its own source; the description after the dash never becomes a term, short names such as BYD or GM match as whole words, and a ticker grounds only as itself) (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's leading name or all its words run together, never a generic trailing word; or a primary-source host such as sec.gov), +1 finding confidence high, +1 cross-specialist consensus (the finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. It also always needs a linked finding that cites the source — company data plus a decision mention can reach the score bar alone, but nothing goes live on hearsay. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Linked finding.** `finding_index` is required on `propose_watch` and the pass is told so; when the model omits it anyway, or cites a finding that does not concern the source, the policy links the finding that best does (a cited URL on the target's site or the ticker named in its text outranks a mere mention of the entity; then the entity's own site, confidence) and says so in the first reason. A proposal that no finding cites is rejected as having no evidence.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run — budgets and the ceiling are spent strongest first (proposals are classified, then applied by tier — direct adds before suggestions — and descending score, ties in file order; a duplicate target keeps its strongest filing), so a better-grounded proposal filed later does not lose its slot to weaker ones filed earlier. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `department=<slug>` — a tag hint, deliberately not the `channel` field triage reads as a team room), so the alerts it raises queue on the head's `/today` (as action items, since routed alerts are never hidden in Monitoring) and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Every fetch is the on-box httpx fetcher: SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`. No path delegates a fetch to an external service.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",
+ "markdown": "`openexecutive.monitoring` watches the outside world on a heartbeat and turns detected changes into briefing proposals via the same alert pipeline that handles inbound email and Slack.\n\nThe `external_monitor_scan` heartbeat (`__internal__` channel, default every 5 min) walks `list_watchlist(enabled_only=True)`, skipping rows fresher than their per-source cadence floor. For each due row it fires a read-only source adapter resolved from `_REGISTRY` by `signal_type`:\n\n- `vendor_status` — Statuspage Atom/RSS incident feeds (AWS, Stripe, GitHub, …). Reads each incident's current status out of the entry body, parses `<updated>` into `published_at`, and keys its dedup on `(entry id, <updated>)`; a **seeding** source (see below).\n- `rss` — generic Atom/RSS via feedparser (changelogs, layoffs.fyi, filings). Parses each entry's `<pubDate>`/`<updated>` into `published_at`; a **seeding** source (see below).\n- `stock` — Yahoo Finance chart endpoint; fires on `abs_change_pct_gte` moves.\n- `page_watch` — change-detection on a feed-less page, fetched with the keyless httpx fetcher. A row written before the external scrape path was removed still carries `config.fetch = \"xcrawl\"`; the adapter treats that as a stale baseline, re-captures it from the plain fetch without emitting a signal, and clears the marker.\n- `edgar` and the BILLED `query` web-search adapter.\n\nEach `Signal` is trigger-filtered, severity-clamped to the row's `[floor, ceiling]`, and INSERTed into `external_signals` where `UNIQUE(dedup_key)` drops replays. Survivors pass a suppression cascade (`baseline` → `stale` → `dry_run` → `below_floor` → capture-time relevance enrichment → promote), then `promote_signal_to_alert` builds an `AlertEvent` handed to `alerts.pipeline.schedule_evaluation` and surfaced as a `/today` proposal.\n\nEvery signal carries two timestamps that are never conflated: `captured_at` (when the monitor first saw it) and `published_at` (when the upstream says it happened — RSS `<pubDate>`, EDGAR filing date; null for `stock`, `page_watch`, `query`). Two freshness gates run first in the cascade, before anything that costs an LLM call. **Baseline:** the first poll that returns entries for a row whose adapter declares `seed_on_first_poll` (`rss`, `edgar`) records every entry already in the feed — trigger misses included, so the first entry that ever matches a keyword is news and a later, wider trigger can't resurface old ones — as `suppressed_baseline`, promotes nothing, and stamps `watchlist.baselined_at`, so a new watch reports what changes from now on instead of replaying the feed's back-catalogue (the same idea as `page_watch`'s first-observation baseline). It is keyed on `baselined_at` rather than `last_polled_at`, so a failed first fetch can't forfeit it; the claim and the recording are one transaction (a compare-and-swap on the stamp plus the rows, all-or-nothing), so of two overlapping scans exactly one records the back-catalogue and a failure part-way leaves nothing behind to replay; the other scan promotes only entries dated after that stamp (undated entries count as baseline, since the feed controls the date; entries dated more than a few minutes past its own clock are deferred rather than recorded); and baseline rows don't count against `max_signals_per_scan`, so a large feed can't be half-baselined. `vendor_status` used to sit outside both gates for a structural reason: its dedup key was `sha256(slug, entry id)`, and an incident's id never changes across its updates, so any suppressed insert muted that incident for good. Keying on `(entry id, <updated>)` instead (issue #90) makes a suppression cost one *state* of an incident rather than the incident, so both gates now apply to vendor feeds too — and an incident moving Investigating → Resolved mints a new key and surfaces, while re-polling an unchanged feed still dedups. Because a history feed's archive is mostly incidents resolved months ago, the adapter also carves live news out of its own baseline: it reads each incident's current status from the entry body (Statuspage renders the newest update first as `<strong>Resolved</strong>` / `Investigating` / `Monitoring` / … — the body is parsed by a real HTML parser, off the event loop, and the reading is deliberately asymmetric: any sign that the incident is over closes it, wherever it appears, while only a marked-up label can say it is still open. Working out which update is newest turned out to be something tag soup cannot be trusted to express, and every attempt reported a long-resolved incident as live) and exempts the ones still OPEN via the optional `promote_on_baseline` hook, so a watch added during an outage reports that outage on its first poll while the resolved archive is recorded as `suppressed_baseline`. An unrecognised status counts as closed — a vendor changing feed format must not be able to promote its whole archive at HIGH, and nothing is lost, since the next `<updated>` bump surfaces normally. An exempt entry must still pass the row's trigger filter (one that misses stays in the baseline, so widening a trigger later cannot resurface it) and is still judged by the stale gate below: the exemption buys an entry past the baseline, never past its own timestamp, because \"this is live\" is a reading of vendor-controlled markup and an entry the feed dates outside the age window is not news whatever that reading says. **Stale:** a signal whose `published_at` is older than `EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS` (default 7; `0` disables) at capture time is recorded as `suppressed_stale` and never promoted — a January article a feed resurfaces in September is not September news. An entry dated more than `EXTERNAL_MONITOR_MAX_FUTURE_SKEW_HOURS` ahead (default 24; `0` disables; `config_json.max_future_skew_hours` overrides it per row for calendar or maintenance feeds that legitimately date entries ahead) is deferred instead: skipped for the tick and not recorded (recording would burn its dedup key and let a feed mute an announcement for good by post-dating it), with one aggregated `external_signal_deferred` audit row per poll — carrying each held-back entry's summary and link, and emitted even if the poll later raises — then judged normally once the date passes. Inside the window, the alert body handed to triage carries `Published:` and `Discovered:` lines so the item is judged on when it happened. Adapters prefer a plausible date key over one more than a day in the future (and keep the future value when every key is implausible, so the pipeline defers it rather than waving an undated entry through), and the alert body collapses whitespace in the summary so no source can forge its own labeled body line.\n\nThe `query` adapter is a special case: it emits every web hit flat-LOW (it can't judge materiality per-hit), so for query signals enrichment runs BEFORE the floor gate and `_grade_query_severity` regrades from the relevance read (≥0.8 HIGH, ≥0.5 MEDIUM, else LOW). Without this, any query watch item with a `severity_floor` above LOW would suppress every hit — which is what left the Monitoring section empty even while standing queries were returning material findings.\n\nWhich lane a proposal lands in is decided by `briefing.ranking.categorize`: an unrouted, low/medium signal from a monitoring source (`stock`, `vendor_status`, `rss`, `query`, `edgar`, `page_watch`, or any `external:*` topic tag) is `monitoring` (the collapsible Monitoring section); a routed or high/urgent one is `action` (the \"Needs you\" queue). The triage prompt's External Signals rule covers all six source kinds so it trusts the source and emits the `external:*` tags that drive this split.\n\nThe watchlist is editable from chat (`add_watchlist_entry`, `tune_watchlist_entry`) and HTTP (`/watchlist`); the `executive_research` workflow proposes what to watch through a deterministic policy (next section). Because that workflow is LLM-driven and historically proposed feed URLs it never verified (most were dead — 404 / 403 / not-actually-a-feed), every `rss` target is validated at insert time (`monitoring.target_validation`): a URL that fetches and parses as a feed is kept; a public page that isn't a feed is converted to a `page_watch` when the body the feed check already fetched reduces to readable text, otherwise rejected (HTTP 422 / tool error); SSRF-blocked or unparseable URLs are rejected outright and never fetched. Every fetch (`sources._http.fetch_bounded`) follows redirects itself, at most 3, re-checks each hop against the same public-address rule, and connects to the IP it checked (hostname kept for `Host` and TLS), so a redirect or a second DNS answer pointing at loopback, link-local (169.254.169.254) or private addresses is refused. Non-`rss` types insert unchanged with no network I/O.\n\n### One watch, one card — and dismissals teach the watch\n\n`promote_signal_to_alert` stamps the `AlertEvent` with `dedup_hint=\"watch:<slug>\"`, which `alerts.pipeline` uses as the *alert* dedup key. A repeat from the same watch coalesces into the open card (`occurrence_count`, `last_seen_at`, severity max) rather than minting another, and re-pings only when severity rose into high/urgent — so a ticker that stays over its threshold or an incident that keeps updating no longer produces a new \"Needs you\" item every day. The signal-level `dedup_key` (stock's UTC-date bucket, vendor_status's `<updated>`) is untouched, so every observation still lands in `external_signals`. Dismissing a watch-sourced alert calls `store.record_dismissal(slug)` (`dismiss_count += 1`, `trust_score = max(0.2, trust × 0.8)`); approving calls `record_confirmation` (+0.05). `trust_score` discounts the briefing ranking and is shown to the Executive's alert review as evidence, alongside the newer signals for that watch (a \"Resolved\" incident state, a reversed move) that let it close the card with evidence.\n\n### What the research pass may put on the watchlist\n\nThe run itself: seven specialists (or the subset named by `RESEARCH_SPECIALISTS`) each make one call with the basic `web_search` server tool, capped at `RESEARCH_WEB_SEARCH_MAX_USES` searches (default 3, separate from the chat knob; via OpenRouter, Claude or not, the tool becomes OpenRouter's `openrouter:web_search` server tool with the same cap and domain lists, and the search count is recorded there too), and emit at most three findings — the cap sits in the tool schema and is applied again after parsing. Every specialist sees what department heads asked to have watched (a `DEPARTMENT WATCH INTERESTS` block) and the ten most recent decisions, and may ground a finding in either. Findings are deduplicated, low-confidence ones dropped, then the Executive routes the rest; the routing pass cannot start a workflow (`run_workflow` is withheld, `suggest_workflow` is not). There is no external fetch or verify service: every page read is the on-box fetcher. The periodic run is considered every `WATCHLIST_RESEARCH_INTERVAL_MINUTES` (default 360) and skipped when nothing changed. Every model call the run makes is recorded as a `cache_event` audit row tagged with the run id, and the run's result and its `watchlist_research_periodic_ran` row carry a `usage` block (calls, tokens, searches, per source), so what a run did is readable from the audit log.\n\nEvery watchlist row carries an `origin`: `manual` (the principal, via UI/API), `executive` (the chat tool acting on a request), `research` (the research pass added it on its own) or `research_proposed` (a pending suggestion). The `executive_research` workflow's watchlist pass no longer calls `add_watchlist_entry`; its only tool is `propose_watch` (slug, signal_type, target, `grounding_entity`, `rationale`, `certainty`), and `monitoring.research.watch_policy` decides what happens to each proposal from company data, not from the model's confidence in itself:\n\n- **Direct add** — the row lands live, `origin=research`, with quiet defaults the model cannot loosen (daily cadence — hourly for `vendor_status`, weekly for `page_watch`/`query` — `severity_floor=medium`, a keyword trigger of material-event words + the current-year priority terms for `rss` / `query` only — never for `page_watch`, whose summary is just \"page changed\" and would never match, and never the entity's own name, which appears in every feed entry's label and would fire on everything — and `abs_change_pct_gte=5` for stocks). Requires the `grounding_entity` to name an external entity in company data — the company itself, a `competitive_landscape.primary_competitors` entry, one of the profile's `vendors` / `tickers`, or an entity a department lists under its **watched entities** (each list entry is parsed to its name: \"Tesla (TSLA) — Model Y is the benchmark\" grounds as `Tesla` plus the ticker `TSLA`, \"GM / Chevrolet (…)\" as both names, and \"Brex (brex.com)\" pins Brex's own domains so only those sites count as its own source; the description after the dash never becomes a term, short names such as BYD or GM match as whole words, and a ticker grounds only as itself) (an initiative title, a current-year priority, a department's charter-scope phrase or a goal's key result also grounds a proposal, but only ever to a suggestion: they are bags of common words a planted finding can echo) — *and* a score ≥ 4 on a fixed ledger: +2 entity in company data (mandatory), +1 entity already watched under another source, +1 target is the entity's own source (for stock/edgar the grounding entity must be the ticker itself; a URL whose site label — `acme` in `status.acme.com` / `acme.co.uk`, never a subdomain or TLD — is the entity's leading name or all its words run together, never a generic trailing word; or a primary-source host such as sec.gov), +1 finding confidence high, +1 cross-specialist consensus (the finding points count only when the cited finding concerns this source: a URL target must share its site with one of the finding's cited URLs, a ticker must appear in its text — a strong finding cannot be borrowed for an unrelated page), +1 the entity (a company-data term, never a bare watch label) is named in one of the ten most recent episodic decisions from the last 90 days — the whole term, or every distinctive word of it, as whole words — ±1 from the policy's own approval history for that `(signal_type, grounding kind)` once it has ≥5 samples. A direct add is further reserved for the entity's **own** source (its ticker, its site, its status page): a well-corroborated third-party page about a competitor is the principal's call. It also always needs a linked finding that cites the source — company data plus a decision mention can reach the score bar alone, but nothing goes live on hearsay. The model's `certainty` can only downgrade a direct add to a suggestion; a `query` watch is always a suggestion. At most `WATCHLIST_RESEARCH_MAX_DIRECT_ADDS` (2) per run. When one term appears in several sources the strongest kind wins (company > competitor > vendor > ticker > department entity > initiative > priority > department scope > watch label); a term that came from a department carries the department along even when a stronger profile kind wins, so a profile vendor that Finance also lists stays a vendor and still routes to Finance.\n- **Linked finding.** `finding_index` is required on `propose_watch` and the pass is told so; when the model omits it anyway, or cites a finding that does not concern the source, the policy links the finding that best does (a cited URL on the target's site or the ticker named in its text outranks a mere mention of the entity; then the entity's own site, confidence) and says so in the first reason. A proposal that no finding cites is rejected as having no evidence.\n- **Suggestion** — anything that validates, has a linked finding and scores at least 1 but misses the bar is inserted in `dry_run` (polls and records signals, never alerts) with `origin=research_proposed` and the rationale in `notes`, at most `WATCHLIST_RESEARCH_MAX_PROPOSALS` (2) per run — budgets and the ceiling are spent strongest first (proposals are classified, then applied by tier — direct adds before suggestions — and descending score, ties in file order; a duplicate target keeps its strongest filing), so a better-grounded proposal filed later does not lose its slot to weaker ones filed earlier. `/watchlist` shows them in a \"Suggested by the Executive\" section with Approve (`POST /watchlist/{slug}/approve` → `active`, `origin=research`) and Decline (`POST /watchlist/{slug}/decline` with a reason: `not_relevant` blacklists the entity, `wrong_source` blacklists only this target, `too_noisy` keeps the source live with `severity_floor=high` — the one way a suggestion goes live without an approve — and counts as a decline for calibration). A suggestion nobody reviews within `WATCHLIST_PROPOSAL_TTL_DAYS` (14) is removed. The morning brief mentions the pending count in one line; a `watchlist_suggestions` alert nudges only once ≥3 suggestions have waited ≥7 days — one card for the whole pile (an open card is refreshed in place; a dismissed one is not re-minted until the next ISO week), so acting on a suggestion never spawns another nudge.\n- **Routed to a department** — a watch grounded in a department's term (or whose entity a recent decision recorded for that department names) is inserted with `route_to_department` and `route_to_person_id` set to the department and its head at insert time. `_signal_to_alert_event` now maps both onto the `AlertEvent` for every watch (`routed_to_person_id`, `department=<slug>` — a tag hint, deliberately not the `channel` field triage reads as a team room), so the alerts it raises queue on the head's `/today` (as action items, since routed alerts are never hidden in Monitoring) and carry the `department:<slug>` tag instead of landing in the principal's unrouted queue. Suggestions in a department's area are put in front of its head, not the principal: after each run, one \"Watch suggestions for <Department>\" card per department is filed through `propose_via_alert` (routed to the head, or the principal when there is no head; an open card is refreshed in place and an acknowledged one is not re-minted until the next ISO week), and the principal's pile-up nudge counts only suggestions that reached no head — the principal's own and those of a department without a head — so none can become invisible. `/watchlist` labels routed rows \"for: <department>\".\n- **Reject** — an invalid target, a second watch on the same site + signal type (disabled rows included), a declined target or entity, nothing vouching for it (no linked finding, score 0), the enabled-watch ceiling `WATCHLIST_MAX_ENABLED` (40; suggestions poll too, so the ceiling stops them as well), or a spent budget. Every decision is audited (`watchlist_research_added` / `_suggested` / `_rejected`) with the score and reasons; the stamp also lives in the row's `config_json._policy` so approve/decline/auto-disable can attribute the outcome.\n\n**Declines are remembered and enforced.** `watchlist_declines` is keyed by the normalized target (upper-cased ticker; URL with `https` forced, `www.` dropped, host and path lower-cased, query, fragment and trailing slash removed; free-text queries case- and whitespace-folded). A `not_relevant` decline also blacklists the *entity*: any later proposal whose `grounding_entity` names it is refused whatever the target. An explicit decline — the Decline button, \"Stop watching…\" on a research-added card, `remove_watchlist_entry` or `DELETE /watchlist/{slug}` on a research-origin row — is permanent; an unreviewed expiry may be re-proposed once after 90 days. The `propose_watch` handler refuses a declined target or entity before anything is fetched, so a new slug cannot route around a decline (URL keys also fold percent-encoding, dot segments and a trailing host dot); the list is also rendered to the model as DECLINED. The routing pass that runs before the watchlist pass has no watchlist write tools at all (`add_watchlist_entry`, `tune_watchlist_entry`, `remove_watchlist_entry` are withheld from synthesis), and `tune_watchlist_entry` accepts only `notes` on a pending suggestion — never `mode` / `enabled` (only the principal's approve or decline moves one out of review) nor cadence / floor / trigger (it goes live with the quiet defaults policy gave it). Every URL-typed target (`rss`, `vendor_status`, `page_watch`) is checked against the SSRF guard at insert time, and a suggestion's `source_url` link is kept only when it is a public http(s) URL.\n\n**The policy learns from outcomes.** Approve, decline, expiry and auto-disable write `watchlist_policy_outcomes` rows keyed by `(signal_type, grounding kind, specialist)`; the pass renders the tallies (\"stock watches grounded in a competitor: 4 approved / 1 declined\"; \"proposals from cso: …\") and the score ledger reads them. The research turn also shows each existing watch's `origin`, `fired_count`, `dismiss_count` and `trust_score`.\n\n**Research watches retire themselves.** The scheduler's 15-minute sweep (`watch_policy.sweep`, alongside the alert expiry sweep) disables a `research`-origin watch — audited as `watchlist_auto_disabled`, re-enable from `/watchlist` — when it fired ≥5 alerts of which ≥2 were dismissed and trust ≤ 0.5, or after 3 consecutive poll failures (silence is deliberately not a rule: an unchanged page or a healthy vendor is a working watch); the retired target is also recorded as a 90-day decline so the next run cannot re-add it under a new slug with fresh counters (a disabled row already counts as \"same source watched\"). Approve, decline and expiry are compare-and-act on \"still pending\", so a concurrent approve always wins over a decline or the sweep. Direct adds and auto-disables appear in the brief's \"handled overnight\" block as `watching` / `stopped_watching`. The research skip-if-unchanged fingerprint hashes only enabled, `active` rows, so approving or declining a suggestion never triggers a fresh 7-specialist run.\n\n### Invariants\n\n- Adapters are read-only — they never write upstream and never auto-execute.\n- Every `Signal` carries a deterministic `dedup_key` and a mandatory `provenance_url`.\n- Every fetch is the on-box httpx fetcher: SSRF-guarded and bounded by `external_monitor_max_fetch_bytes`. No path delegates a fetch to an external service.\n- Suppressed signals are still persisted so audits show the full picture.\n- A repeat from the same watch never becomes a second open card: the alert-level `watch:<slug>` key coalesces it, and only a severity escalation re-dispatches.\n- The research model never writes the watchlist: it proposes, deterministic policy adds (only when grounded in company data) or suggests, and a target the principal declined is refused at the handler.",
"mermaid": "flowchart TD\n ADD[\"add (chat tool / POST /watchlist / research)\"]\n VAL[\"target_validation: rss feed? → keep / readable page → page_watch / reject\"]\n WL[(\"watchlist\")]\n HB[\"external_monitor_scan heartbeat (~5 min)\"]\n AD[\"read-only adapters: vendor_status / rss / stock / query / edgar / page_watch (httpx)\"]\n SIG[(\"external_signals UNIQUE(dedup_key)\")]\n ENRICH[\"capture-time enrichment<br/>(relevance score + why_it_matters)\"]\n PROMO[\"promote_signal_to_alert\"]\n PIPE[\"alert pipeline (triage)\"]\n TODAY[\"/today proposals\"]\n ADD --> VAL\n VAL -->|\"insert valid rows\"| WL\n HB -->|\"walk enabled rows\"| WL\n WL -->|\"poll due rows\"| AD\n AD -->|\"emit Signal\"| SIG\n SIG -->|\"baseline / stale / dry_run / floor gates\"| ENRICH\n ENRICH -->|\"≥ relevance floor\"| PROMO\n PROMO --> PIPE\n PIPE --> TODAY\n class HB hot\n class WL storage\n class VAL compute\n class AD external\n class SIG storage\n class ENRICH compute\n class PROMO compute\n class PIPE compute\n class TODAY entry\n class ADD compute",
- "generated_at": "2026-09-12T21:05:00+00:00"
+ "generated_at": "2026-10-16T14:00:00Z"
}
diff --git a/packages/core/openexecutive/monitoring/sources/_http.py b/packages/core/openexecutive/monitoring/sources/_http.py
index 5e4d0a8b..d740afc3 100644
--- a/packages/core/openexecutive/monitoring/sources/_http.py
+++ b/packages/core/openexecutive/monitoring/sources/_http.py
@@ -7,6 +7,7 @@ directly — call :func:`fetch_bounded` instead.
"""
from __future__ import annotations
+import asyncio
import ipaddress
import socket
from urllib.parse import urlparse
@@ -35,34 +36,47 @@ class FetchOverflowError(Exception):
"""Raised when a fetched body exceeded the configured byte ceiling."""
-def validate_target_url(url: str) -> tuple[bool, str]:
- """SSRF guard: only allow public http(s) URLs.
+class BlockedTargetError(httpx.HTTPError):
+ """Raised when a fetch hop (the first URL or a redirect) is refused.
- Rejects non-http(s) schemes and any host that resolves to a
- loopback / link-local / private / reserved IP — the classic SSRF
- pivots into IMDS (169.254.169.254), local services (127.*), or
- private cloud meshes (RFC1918, IPv6 ULA). Returns
- ``(False, reason)`` to be logged at the call site.
+ Subclasses :class:`httpx.HTTPError` so every adapter's existing
+ ``except httpx.HTTPError`` logs it and skips the tick.
+ """
+
+
+def _resolve_public(url: str) -> tuple[list[str], str]:
+ """Resolve ``url``'s host and return its addresses if all are public.
- First-resolution check only — full DNS-rebind defence would need
- re-resolution at fetch time. For the watchlist threat model
- (misconfigured row, not a remote attacker timing DNS) this is a
- strong perimeter.
+ Returns ``(ips, "")`` on success or ``([], reason)`` when the URL is
+ refused: a scheme outside :data:`ALLOWED_SCHEMES`, no host, a DNS
+ failure, or any address that is loopback / link-local / private /
+ reserved / multicast / unspecified, or otherwise not globally
+ routable (e.g. the 100.64.0.0/10 shared range some clouds use for
+ metadata).
"""
try:
parsed = urlparse(url)
except ValueError as exc:
- return False, f"unparseable URL: {exc}"
+ return [], f"unparseable URL: {exc}"
scheme = (parsed.scheme or "").lower()
if scheme not in ALLOWED_SCHEMES:
- return False, f"scheme {scheme!r} not in {sorted(ALLOWED_SCHEMES)}"
- host = parsed.hostname or ""
+ return [], f"scheme {scheme!r} not in {sorted(ALLOWED_SCHEMES)}"
+ if not parsed.hostname:
+ return [], "URL has no host"
+ # Resolve the name httpx will put in Host and the TLS server name
+ # (IDNA 2008), not the str form, which getaddrinfo would encode with
+ # IDNA 2003 and could turn into a different name.
+ try:
+ host = httpx.URL(url).raw_host.decode("ascii")
+ except httpx.InvalidURL as exc:
+ return [], f"unparseable URL: {exc}"
if not host:
- return False, "URL has no host"
+ return [], "URL has no host"
try:
infos = socket.getaddrinfo(host, None)
except socket.gaierror as exc:
- return False, f"DNS resolution failed: {exc}"
+ return [], f"DNS resolution failed: {exc}"
+ ips: list[str] = []
for info in infos:
try:
ip = ipaddress.ip_address(info[4][0])
@@ -75,9 +89,32 @@ def validate_target_url(url: str) -> tuple[bool, str]:
or ip.is_reserved
or ip.is_multicast
or ip.is_unspecified
+ or not ip.is_global
):
- return False, f"host {host!r} resolves to non-public IP {ip}"
- return True, ""
+ return [], f"host {host!r} resolves to non-public IP {ip}"
+ if str(ip) not in ips:
+ ips.append(str(ip))
+ if not ips:
+ return [], f"host {host!r} has no usable address"
+ return ips, ""
+
+
+def validate_target_url(url: str) -> tuple[bool, str]:
+ """SSRF guard: only allow public http(s) URLs.
+
+ Rejects non-http(s) schemes and any host that resolves to a
+ loopback / link-local / private / reserved IP — the classic SSRF
+ pivots into IMDS (169.254.169.254), local services (127.*), or
+ private cloud meshes (RFC1918, IPv6 ULA). Returns
+ ``(False, reason)`` to be logged at the call site.
+
+ This is the check callers run before queueing a fetch.
+ :func:`fetch_bounded` repeats it on every hop, redirects included,
+ and connects to the address it vetted, so a redirect or a DNS answer
+ that changes after this check can't reach a private address.
+ """
+ ips, reason = _resolve_public(url)
+ return bool(ips), reason
def strip_url_query(url: str) -> str:
@@ -95,6 +132,47 @@ def strip_url_query(url: str) -> str:
return parsed._replace(query="", fragment="").geturl()
+def _transport() -> httpx.AsyncBaseTransport:
+ return httpx.AsyncHTTPTransport(retries=0)
+
+
+async def _get_pinned(
+ target: httpx.URL, ip: str, max_bytes: int, user_agent: str
+) -> tuple[str | None, bytes]:
+ """One GET of ``target`` sent to ``ip``: ``(location, b"")`` for a
+ redirect, else ``(None, body)``."""
+ extensions = (
+ {"sni_hostname": target.raw_host.decode("ascii")}
+ if target.scheme == "https"
+ else {}
+ )
+ # A fresh client per hop: a pooled TLS connection to the same IP
+ # would otherwise carry the previous hop's server name.
+ async with (
+ httpx.AsyncClient(
+ timeout=HTTP_TIMEOUT_SECONDS,
+ follow_redirects=False,
+ transport=_transport(),
+ headers={"User-Agent": user_agent},
+ ) as client,
+ client.stream(
+ "GET",
+ target.copy_with(host=ip),
+ headers={"Host": target.netloc.decode("ascii")},
+ extensions=extensions,
+ ) as resp,
+ ):
+ if resp.has_redirect_location:
+ return resp.headers["Location"], b""
+ resp.raise_for_status()
+ buf = bytearray()
+ async for chunk in resp.aiter_bytes():
+ buf.extend(chunk)
+ if len(buf) > max_bytes:
+ raise FetchOverflowError()
+ return None, bytes(buf)
+
+
async def fetch_bounded(
url: str, max_bytes: int, *, user_agent: str | None = None
) -> bytes:
@@ -105,33 +183,53 @@ async def fetch_bounded(
:class:`FetchOverflowError` rather than truncating-and-pretending-
it-worked so the caller can log + skip the tick honestly.
+ Redirects are followed by hand, at most :data:`MAX_REDIRECTS`. Every
+ hop, the first included, is resolved and checked like
+ :func:`validate_target_url`, and the connection goes to the address
+ that passed the check (the hostname stays in the ``Host`` header and
+ the TLS server name), so neither a redirect nor a second DNS answer
+ can point the fetch at a private address. A refused hop raises
+ :class:`BlockedTargetError`; too many hops raise
+ :class:`httpx.TooManyRedirects`.
+
``user_agent`` overrides the default UA for hosts that require a
specific identifier — e.g. SEC EDGAR rejects/ratelimits the generic
UA and asks callers to identify with a contact. Falls back to the
module ``USER_AGENT`` when None.
"""
- transport = httpx.AsyncHTTPTransport(retries=0)
- async with (
- httpx.AsyncClient(
- timeout=HTTP_TIMEOUT_SECONDS,
- follow_redirects=True,
- max_redirects=MAX_REDIRECTS,
- transport=transport,
- headers={"User-Agent": user_agent or USER_AGENT},
- ) as client,
- client.stream("GET", url) as resp,
- ):
- resp.raise_for_status()
- buf = bytearray()
- async for chunk in resp.aiter_bytes():
- buf.extend(chunk)
- if len(buf) > max_bytes:
- raise FetchOverflowError()
- return bytes(buf)
+ current = url
+ for _hop in range(MAX_REDIRECTS + 1):
+ ips, reason = await asyncio.to_thread(_resolve_public, current)
+ if not ips:
+ raise BlockedTargetError(f"refused {strip_url_query(current)}: {reason}")
+ try:
+ target = httpx.URL(current)
+ except httpx.InvalidURL as exc:
+ raise BlockedTargetError(f"refused {strip_url_query(current)}: {exc}") from exc
+ # Every address passed the check, so trying the next one when a
+ # connect fails (e.g. an IPv6 answer with no IPv6 route) keeps the
+ # guard intact.
+ for i, ip in enumerate(ips):
+ try:
+ location, body = await _get_pinned(
+ target, ip, max_bytes, user_agent or USER_AGENT
+ )
+ break
+ except httpx.ConnectError:
+ if i == len(ips) - 1:
+ raise
+ if location is None:
+ return body
+ try:
+ current = str(target.join(location))
+ except httpx.InvalidURL as exc:
+ raise BlockedTargetError(f"bad redirect from {strip_url_query(current)}: {exc}") from exc
+ raise httpx.TooManyRedirects(f"more than {MAX_REDIRECTS} redirects from {strip_url_query(url)}")
__all__ = [
"ALLOWED_SCHEMES",
+ "BlockedTargetError",
"FetchOverflowError",
"HTTP_TIMEOUT_SECONDS",
"MAX_REDIRECTS",
diff --git a/packages/core/tests/unit/test_monitoring_http_redirects.py b/packages/core/tests/unit/test_monitoring_http_redirects.py
new file mode 100644
index 00000000..435f87b5
--- /dev/null
+++ b/packages/core/tests/unit/test_monitoring_http_redirects.py
@@ -0,0 +1,205 @@
+"""fetch_bounded checks every redirect hop and connects to the vetted IP."""
+from __future__ import annotations
+
+import socket
+
+import httpx
+import pytest
+
+from openexecutive.monitoring.sources import _http
+from openexecutive.monitoring.sources._http import (
+ BlockedTargetError,
+ fetch_bounded,
+ validate_target_url,
+)
+
+# Public addresses for the made-up hostnames below. Anything not listed
+# resolves the way an IP literal does: to itself.
+DNS = {
+ "feed.example.com": "93.184.215.14",
+ "cdn.example.net": "151.101.1.69",
+ "rebind.example.org": "10.0.0.7",
+ "dual.example.com": ["2606:4700::6810:84e5", "104.16.132.229"],
+ "xn--fa-hia.example": "93.184.215.14",
+}
+
+
+def _fake_getaddrinfo(host: str, port: object, *args: object, **kwargs: object) -> list:
+ found = DNS.get(host, host)
+ infos = []
+ for addr in [found] if isinstance(found, str) else found:
+ family = socket.AF_INET6 if ":" in addr else socket.AF_INET
+ sockaddr = (addr, 0, 0, 0) if family == socket.AF_INET6 else (addr, 0)
+ infos.append((family, socket.SOCK_STREAM, 6, "", sockaddr))
+ return infos
+
+
+class _Server:
+ """Answers by Host header, records every request it was sent."""
+
+ def __init__(self, routes: dict[str, httpx.Response]) -> None:
+ self.routes = routes
+ self.requests: list[httpx.Request] = []
+
+ def __call__(self, request: httpx.Request) -> httpx.Response:
+ self.requests.append(request)
+ key = request.headers["host"] + request.url.path
+ return self.routes.get(key, httpx.Response(404))
+
+
+def server(monkeypatch: pytest.MonkeyPatch):
+ monkeypatch.setattr(socket, "getaddrinfo", _fake_getaddrinfo)
+
+ def install(routes: dict[str, httpx.Response]) -> _Server:
+ srv = _Server(routes)
+ monkeypatch.setattr(_http, "_transport", lambda: httpx.MockTransport(srv))
+ return srv
+
+ return install
+
+
+def _redirect(location: str) -> httpx.Response:
+ return httpx.Response(302, headers={"Location": location})
+
+
+ "location",
+ [
+ "http://127.0.0.1/admin",
+ "http://169.254.169.254/latest/meta-data/",
+ "http://[fdaa::3]:8080/internal",
+ "http://100.100.100.200/latest/meta-data/",
+ "http://rebind.example.org/",
+ "file:///etc/passwd",
+ ],
+)
+async def test_redirect_to_non_public_address_is_refused(server, location: str) -> None:
+ srv = server({"feed.example.com/rss": _redirect(location)})
+
+ with pytest.raises(BlockedTargetError):
+ await fetch_bounded("https://feed.example.com/rss", 10_000)
+
+ # Only the first, public hop was ever sent.
+ assert len(srv.requests) == 1
+ assert srv.requests[0].url.host == "93.184.215.14"
+
+
+async def test_blocked_redirect_is_an_httpx_error(server) -> None:
+ # Adapters catch httpx.HTTPError to log and skip the tick.
+ server({"feed.example.com/rss": _redirect("http://127.0.0.1/")})
+ with pytest.raises(httpx.HTTPError):
+ await fetch_bounded("https://feed.example.com/rss", 10_000)
+
+
[email protected]("location", ["http://[::1", "http://exa mple.com:99999/"])
+async def test_malformed_redirect_is_an_httpx_error(server, location: str) -> None:
+ # Adapters log and skip httpx.HTTPError; a bad Location must land there.
+ srv = server({"feed.example.com/rss": _redirect(location)})
+ with pytest.raises(httpx.HTTPError):
+ await fetch_bounded("https://feed.example.com/rss", 10_000)
+ assert len(srv.requests) == 1
+
+
+async def test_first_hop_is_checked_too(server) -> None:
+ srv = server({})
+ with pytest.raises(BlockedTargetError):
+ await fetch_bounded("http://169.254.169.254/latest/meta-data/", 10_000)
+ assert srv.requests == []
+
+
+async def test_public_redirect_is_followed_on_the_pinned_ip(server) -> None:
+ srv = server({
+ "feed.example.com/rss": _redirect("https://cdn.example.net/feed.xml?sig=1"),
+ "cdn.example.net/feed.xml": httpx.Response(200, content=b"<rss/>"),
+ })
+
+ body = await fetch_bounded("https://feed.example.com/rss", 10_000)
+
+ assert body == b"<rss/>"
+ first, second = srv.requests
+ assert first.url.host == "93.184.215.14"
+ assert first.headers["host"] == "feed.example.com"
+ assert first.extensions["sni_hostname"] == "feed.example.com"
+ assert second.url.host == "151.101.1.69"
+ assert second.headers["host"] == "cdn.example.net"
+ assert second.extensions["sni_hostname"] == "cdn.example.net"
+ assert second.url.query == b"sig=1"
+
+
+async def test_relative_redirect_keeps_host_and_port(server) -> None:
+ srv = server({
+ "feed.example.com:8443/old": _redirect("/new"),
+ "feed.example.com:8443/new": httpx.Response(200, content=b"ok"),
+ })
+
+ assert await fetch_bounded("https://feed.example.com:8443/old", 100) == b"ok"
+ assert srv.requests[1].url.port == 8443
+ assert srv.requests[1].headers["host"] == "feed.example.com:8443"
+
+
+async def test_plain_http_sends_no_server_name(server) -> None:
+ srv = server({"feed.example.com/": httpx.Response(200, content=b"ok")})
+ await fetch_bounded("http://feed.example.com/", 100)
+ assert "sni_hostname" not in srv.requests[0].extensions
+
+
+async def test_next_vetted_address_is_tried_when_connect_fails(server) -> None:
+ srv = server({"dual.example.com/": httpx.Response(200, content=b"ok")})
+ inner = srv.__call__
+
+ def no_ipv6_route(request: httpx.Request) -> httpx.Response:
+ if ":" in request.url.host:
+ srv.requests.append(request)
+ raise httpx.ConnectError("unreachable", request=request)
+ return inner(request)
+
+ srv.__call__ = no_ipv6_route # type: ignore[method-assign]
+ _http_transport = httpx.MockTransport(no_ipv6_route)
+ server_patch = pytest.MonkeyPatch()
+ server_patch.setattr(_http, "_transport", lambda: _http_transport)
+ try:
+ assert await fetch_bounded("https://dual.example.com/", 100) == b"ok"
+ finally:
+ server_patch.undo()
+ assert [r.url.host for r in srv.requests] == ["2606:4700::6810:84e5", "104.16.132.229"]
+
+
+async def test_idn_host_uses_the_encoded_name(server) -> None:
+ srv = server({"xn--fa-hia.example/": httpx.Response(200, content=b"ok")})
+ await fetch_bounded("https://faß.example/", 100)
+ assert srv.requests[0].headers["host"] == "xn--fa-hia.example"
+ assert srv.requests[0].extensions["sni_hostname"] == "xn--fa-hia.example"
+
+
+async def test_redirect_cap(server) -> None:
+ routes = {
+ f"feed.example.com/{i}": _redirect(f"/{i + 1}")
+ for i in range(_http.MAX_REDIRECTS + 1)
+ }
+ srv = server(routes)
+
+ with pytest.raises(httpx.TooManyRedirects):
+ await fetch_bounded("https://feed.example.com/0", 100)
+ assert len(srv.requests) == _http.MAX_REDIRECTS + 1
+
+
+async def test_body_cap_still_applies(server) -> None:
+ server({"feed.example.com/": httpx.Response(200, content=b"x" * 50)})
+ with pytest.raises(_http.FetchOverflowError):
+ await fetch_bounded("https://feed.example.com/", 10)
+
+
+async def test_error_status_still_raises(server) -> None:
+ server({"feed.example.com/": httpx.Response(503)})
+ with pytest.raises(httpx.HTTPStatusError):
+ await fetch_bounded("https://feed.example.com/", 10)
+
+
+def test_validate_target_url_unchanged(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(socket, "getaddrinfo", _fake_getaddrinfo)
+ assert validate_target_url("https://feed.example.com/") == (True, "")
+ ok, reason = validate_target_url("http://[fdaa::3]/")
+ assert not ok and "non-public" in reason
+ ok, reason = validate_target_url("ftp://feed.example.com/")
+ assert not ok and "scheme" in reason