Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(monitoring): strip page tags in one pass (#253)

Rufus336c741a8d7a
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index 2365695c..ec4992f3 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -3827,7 +3827,7 @@ external_monitoring:
- "stock — Yahoo Finance Chart endpoint (the same endpoint yfinance scrapes, used directly via httpx to avoid pandas/numpy). Watchlist `target` is the ticker; signals fire when |intraday vs previous close| ≥ `trigger.abs_change_pct_gte` (default 5%). Severity hint scales with the magnitude of the move (≥10% URGENT, ≥5% HIGH, ≥2% MEDIUM, else LOW). v2 candidate: swap behind the same Source interface for a paid Alpha Vantage / Finnhub feed if Yahoo brittleness bites."
- "query — standing web-search query. Watchlist `target` is a natural-language query (e.g. 'Acme Corp layoffs OR restructuring OR funding'); each poll runs it through `BaseAgent.analyze_with_tools` with the web_search tool (model pinned to `get_research_model()`, honouring an OpenRouter Council override — the provider layer translates native web_search into OpenRouter's `plugins:[{id:web}]`, see providers/translator). The model reports distinct results via the `emit_query_results` tool; each result with a valid clickable URL becomes a Signal (the adapter emits a flat severity hint LOW because it can't judge materiality per-hit — the pipeline regrades this from the capture-time relevance read BEFORE the floor gate via `_grade_query_severity`, so a material hit clears a medium/high floor instead of being suppressed; dedup key `query:` + first 32 hex chars of sha256(slug + NUL + canonical_url); URL-less results dropped to uphold the provenance invariant). UNLIKE the keyless feed adapters this source is BILLED (one LLM call + web searches per poll), so it polls slowly (default 360 min), is bounded by `web_search_max_uses` + per-row `config.max_results` (default 5, ceiling 10), and has its own kill switch `EXTERNAL_MONITOR_QUERY_ENABLED`. Optional `trigger.keywords` filters like rss; `config.allowed_domains`/`blocked_domains` are best-effort prompt hints only (NOT enforced, and not honoured by the OpenRouter web plugin). This is the 'watch arbitrary conditions' primitive that no feed publishes."
- "edgar — SEC EDGAR filings. Watchlist `target` is a ticker (e.g. 'AAPL') or CIK; polls SEC's browse-edgar Atom feed (`/cgi-bin/browse-edgar?action=getcompany&CIK=…&output=atom`, whose CIK param resolves a ticker server-side, so no ticker→CIK table is shipped), parsed with feedparser. Emits one Signal per new filing whose form is in the filter (`trigger.forms`, default {8-K, 10-K, 10-Q}; amendments like 8-K/A match their base form). Dedup on the filing's accession number (globally unique — the cleanest natural key of any source). Provenance is the filing-index URL. Severity hint: 8-K HIGH (material-event report), 10-K/10-Q MEDIUM, else LOW. Keyless, but SEC's fair-access policy requires a descriptive User-Agent with a contact — sent via the EDGAR_USER_AGENT setting (operators should override the default with a real contact). Reuses fetch_bounded (now with a per-call user_agent override), validate_target_url, and the standard dedup/Signal pattern. The entry's <updated> lands on Signal.published_at and the adapter declares `seed_on_first_poll`, so a new watch baselines past filings instead of replaying them and filings older than EXTERNAL_MONITOR_MAX_SIGNAL_AGE_DAYS are `suppressed_stale`."
- - "page_watch — generic web-page change detection for feed-less pages (pricing, careers, leadership, ToS). Watchlist `target` is the page URL. STATEFUL (the first stateful adapter): each poll fetches the page (bounded, SSRF-guarded), reduces it to normalized visible text (drops script/style/comments/tags, unescapes entities, collapses whitespace — so markup/layout churn doesn't register), hashes the FULL normalized text (so a change anywhere registers — no length blind spot), and compares to the last-seen hash in `page_watch_state`. First observation stores a baseline and emits NOTHING; an unchanged page emits nothing; a changed page emits ONE Signal carrying a difflib-based change summary (~N% differs + the added text; the diff/snapshot are length-capped but detection is not) and updates the baseline. Severity hint LOW (page diffs are noisy — enrichment + severity_floor filter). Dedup key folds slug + UTC date + content hash (mirrors stock — a state seen again on a later day re-alerts; same-day repeats dedup). Optional `trigger.keywords` only surfaces a change when the change itself (summary + added text) matches. Reads/writes its state via the db_path threaded through Source.poll. The keyless httpx fetcher is the only fetch path. A row written before the xcrawl scrape path was removed still carries `config.fetch = \"xcrawl\"` (PAGE_WATCH_FETCH_LEGACY_XCRAWL) and a baseline hashed from scraped markdown; the adapter treats that marker as a stale baseline — the next poll re-captures the baseline from the plain fetch WITHOUT emitting a signal and clears the marker in place (store.remove_watchlist_config_key, a SQL json_remove so a concurrent edit to another config key is kept), so the change of fetcher is never reported as a page change. An unsaved item (no id) cannot have its marker cleared and is compared normally."
+ - "page_watch — generic web-page change detection for feed-less pages (pricing, careers, leadership, ToS). Watchlist `target` is the page URL. STATEFUL (the first stateful adapter): each poll fetches the page (bounded, SSRF-guarded), reduces it to normalized visible text (drops script/style/comments/tags, unescapes entities, collapses whitespace — so markup/layout churn doesn't register), hashes the FULL normalized text (so a change anywhere registers — no length blind spot), and compares to the last-seen hash in `page_watch_state`. First observation stores a baseline and emits NOTHING; an unchanged page emits nothing; a changed page emits ONE Signal carrying a difflib-based change summary (~N% differs + the added text; the diff/snapshot are length-capped but detection is not) and updates the baseline. Severity hint LOW (page diffs are noisy — enrichment + severity_floor filter). Dedup key folds slug + UTC date + content hash (mirrors stock — a state seen again on a later day re-alerts; same-day repeats dedup). Optional `trigger.keywords` only surfaces a change when the change itself (summary + added text) matches. Reads/writes its state via the db_path threaded through Source.poll. The reduction (html_to_text, which insert-time validation also runs on the request path) takes whatever the server sends and runs synchronously on the event loop, so every step is one forward pass: comments and script/style/noscript blocks by a scan (_strip_blocks), the remaining tags by utils.html_tags.strip_tags (shared with delegation.gmail), never a backtracking regex. The `<[^>]+>` strip it replaced rescanned to the end of the page from every '<' that nothing closed — about 40 minutes of CPU for a 2 MB page, with the API frozen meanwhile. Such a '<' stays text, and a unit test times crafted pages. The keyless httpx fetcher is the only fetch path. A row written before the xcrawl scrape path was removed still carries `config.fetch = \"xcrawl\"` (PAGE_WATCH_FETCH_LEGACY_XCRAWL) and a baseline hashed from scraped markdown; the adapter treats that marker as a stale baseline — the next poll re-captures the baseline from the plain fetch WITHOUT emitting a signal and clears the marker in place (store.remove_watchlist_config_key, a SQL json_remove so a concurrent edit to another config key is kept), so the change of fetcher is never reported as a page change. An unsaved item (no id) cannot have its marker cleared and is compared normally."
insert_validation: |
monitoring.target_validation.validate_and_normalize_target — runs at
insert time on BOTH watchlist surfaces (POST /watchlist and the
@@ -6745,8 +6745,9 @@ delegation:
Google) or error. Credentials per person, see integrations.own_gmail.
html_to_text (HTML-only bodies and the sendAs signature) runs on mail
anyone can send, on the event loop, so every step is one forward pass:
- the <br> and block patterns end at the next "<" as well as ">", tags and
- script / style elements are cut by scanning forward, and a decimal
+ the <br> and block patterns end at the next "<" as well as ">", tags
+ (utils.html_tags.strip_tags, shared with page_watch) and script / style
+ elements are cut by scanning forward, and a decimal
character reference past Unicode's range becomes U+FFFD before
html.unescape (which raises past 4,300 digits). A unit test times
crafted input.
diff --git a/packages/core/openexecutive/delegation/gmail.py b/packages/core/openexecutive/delegation/gmail.py
index bde88849..8145880e 100644
--- a/packages/core/openexecutive/delegation/gmail.py
+++ b/packages/core/openexecutive/delegation/gmail.py
@@ -42,6 +42,8 @@ from urllib.parse import quote
import httpx
+from openexecutive.utils.html_tags import strip_tags
+
logger = logging.getLogger(__name__)
GMAIL_BASE = "https://gmail.googleapis.com/gmail/v1/users/me"
@@ -269,21 +271,6 @@ _DECIMAL_REF_RE = re.compile(r"&#(\d+)(;?)")
_EDGE = "\x00"
-def _strip_tags(text: str) -> str:
- """``text`` without its tags — each "<" through the next ">", with
- something between — in one forward pass."""
- out: list[str] = []
- at = 0
- while (lt := text.find("<", at)) != -1:
- gt = text.find(">", lt + 1)
- if gt == -1:
- break # nothing after this closes a tag
- out.append(text[at:gt + 1] if gt == lt + 1 else text[at:lt])
- at = gt + 1
- out.append(text[at:])
- return "".join(out)
-
-
def _short_decimal_ref(match: re.Match[str]) -> str:
# html.unescape turns the digits into an int, and Python refuses more than
# 4,300 of them (ValueError). Beyond seven significant digits the value is
@@ -316,7 +303,7 @@ def html_to_text(markup: str) -> str:
``<div><br></div>`` a blank line."""
text = _drop_hidden(markup.replace(_EDGE, ""))
text = _BLOCK_EDGE_RE.sub(_EDGE, _BR_RE.sub("\n", text))
- text = html.unescape(_DECIMAL_REF_RE.sub(_short_decimal_ref, _strip_tags(text)))
+ text = html.unescape(_DECIMAL_REF_RE.sub(_short_decimal_ref, strip_tags(text)))
out: list[str] = []
has_text = False
for piece in re.split(f"({_EDGE}|\n)", text):
diff --git a/packages/core/openexecutive/monitoring/sources/page_watch.py b/packages/core/openexecutive/monitoring/sources/page_watch.py
index bca88adb..474a5580 100644
--- a/packages/core/openexecutive/monitoring/sources/page_watch.py
+++ b/packages/core/openexecutive/monitoring/sources/page_watch.py
@@ -57,6 +57,7 @@ from openexecutive.monitoring.sources._http import (
fetch_bounded,
validate_target_url,
)
+from openexecutive.utils.html_tags import strip_tags
logger = logging.getLogger(__name__)
@@ -82,9 +83,10 @@ _ADDED_TEXT_CHARS = 1_000
# JS/CSS never counts as "visible text". Removed by a linear scan
# (_strip_blocks), not a lazy `.*?` regex: on a page full of unterminated
# openers such a regex is quadratic, and a page the server sends is the
-# attacker's to shape.
+# attacker's to shape. The remaining tags go the same way, through
+# utils.html_tags.strip_tags rather than `<[^>]+>`, which rescans to the end
+# of the page from every "<" that nothing closes.
_DROP_BLOCK_TAGS = ("script", "style", "noscript")
-_TAG_RE = re.compile(r"(?s)<[^>]+>")
_WS_RE = re.compile(r"\s+")
@@ -248,7 +250,7 @@ def html_to_text(body: bytes) -> str:
except Exception:
return ""
markup = _strip_blocks(markup)
- markup = _TAG_RE.sub(" ", markup)
+ markup = strip_tags(markup, repl=" ")
text = html.unescape(markup)
# Full normalized text — NOT truncated. The fetch is already byte-capped
# (~2MB), and hashing the whole thing means detection has no blind spot;
diff --git a/packages/core/openexecutive/utils/html_tags.py b/packages/core/openexecutive/utils/html_tags.py
new file mode 100644
index 00000000..c965abfc
--- /dev/null
+++ b/packages/core/openexecutive/utils/html_tags.py
@@ -0,0 +1,29 @@
+"""One tag strip, shared by everything that reduces someone else's HTML to text.
+
+``monitoring.sources.page_watch`` reads pages a watched site's server shapes,
+and ``delegation.gmail`` reads mail anyone can send. Both run on the event
+loop, so both drop tags with this forward scan rather than
+``re.sub(r"<[^>]+>", ...)``: on text with many "<" and no ">", that pattern runs
+from each "<" to the end of the text, fails, and starts over at the next "<" —
+quadratic, about 40 minutes of CPU for a 2 MB page.
+"""
+from __future__ import annotations
+
+
+def strip_tags(text: str, *, repl: str = "") -> str:
+ """``text`` with each tag — a "<" through the next ">", with something
+ between — replaced by ``repl``, in one forward pass.
+
+ Gives the same result as ``re.sub(r"<[^>]+>", repl, text)``: ``<>`` is not
+ a tag, and a "<" that nothing after it closes stays as text.
+ """
+ out: list[str] = []
+ at = 0
+ while (lt := text.find("<", at)) != -1:
+ gt = text.find(">", lt + 1)
+ if gt == -1:
+ break # nothing after this closes a tag
+ out.append(text[at:gt + 1] if gt == lt + 1 else text[at:lt] + repl)
+ at = gt + 1
+ out.append(text[at:])
+ return "".join(out)
diff --git a/packages/core/tests/unit/test_html_tags.py b/packages/core/tests/unit/test_html_tags.py
new file mode 100644
index 00000000..58495b45
--- /dev/null
+++ b/packages/core/tests/unit/test_html_tags.py
@@ -0,0 +1,31 @@
+"""The shared tag strip (utils.html_tags): the regex it replaced, in one pass."""
+from __future__ import annotations
+
+import itertools
+import re
+
+import pytest
+
+from openexecutive.utils.html_tags import strip_tags
+
+# What page_watch and delegation.gmail stripped tags with before: quadratic on
+# text with many "<" and no ">".
+_OLD_TAG_RE = re.compile(r"<[^>]+>")
+
+# Every string of up to seven characters over "<", ">", a letter and a newline
+# (21,845 of them), and a few shapes real markup has.
+_SAMPLES = [
+ "".join(chars) for n in range(8) for chars in itertools.product("<>a\n", repeat=n)
+] + [
+ '<a href="https://x.example/" title="<">click</a> here',
+ "if x<y then <b>bold</b>",
+ "a <> b <<c>> d",
+ "<!-- note -->text<br/>more",
+ "<p>1 < 2</p> and 3 < 4",
+]
+
+
[email protected]("repl", ["", " "])
+def test_strip_tags_matches_the_regex_it_replaced(repl: str) -> None:
+ for text in _SAMPLES:
+ assert strip_tags(text, repl=repl) == _OLD_TAG_RE.sub(repl, text), repr(text)
diff --git a/packages/core/tests/unit/test_monitoring_page_watch.py b/packages/core/tests/unit/test_monitoring_page_watch.py
index 711e9cae..0238635a 100644
--- a/packages/core/tests/unit/test_monitoring_page_watch.py
+++ b/packages/core/tests/unit/test_monitoring_page_watch.py
@@ -282,6 +282,30 @@ def test_html_to_text_is_linear_on_unterminated_openers() -> None:
) == "a b d e"
+ ("body", "expected"),
+ [
+ (b"<" * 200_000, "<" * 200_000),
+ (b"<a" * 100_000, "<a" * 100_000),
+ (b"<\n" * 100_000, "< " * 99_999 + "<"),
+ (b"<div class=x" * 40_000, "<div class=x" * 40_000),
+ (b"<p>kept</p>" + b"<" * 200_000, "kept " + "<" * 200_000),
+ (b"<>" * 100_000, "<>" * 100_000),
+ ],
+ ids=["lt", "lt-a", "lt-newline", "div-attr", "tags-then-lt", "empty-pairs"],
+)
+def test_html_to_text_is_linear_on_unclosed_tags(body: bytes, expected: str) -> None:
+ """A page full of "<" that nothing closes must reduce in one pass: the
+ poll runs this on the event loop, and the `<[^>]+>` strip it replaced
+ rescanned to the end of the page from every such "<" (quadratic — about
+ 40 minutes of CPU for a 2 MB page). Such a "<" stays text, as it did."""
+ import time
+
+ started = time.perf_counter()
+ assert html_to_text(body) == expected
+ assert time.perf_counter() - started < 2
+
+
def test_page_watch_registered() -> None:
assert "page_watch" in list_registered_kinds()