Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix: crawl4ai headless lane verifies TLS certificates (#137) (#145)
Jordan Gibbs6f05dbf90dfa
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4ea4cfb..39f81df 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3,12 +3,13 @@## [Unreleased]- Add scholarly metadata filters and metadata-only browsing to CLI and MCP search, including semantic filtering and a cache upgrade that preserves unchecked retraction status.+- **BEHAVIOR CHANGE — crawl4ai's headless browser verifies TLS certificates (#137).** crawl4ai defaults `BrowserConfig.ignore_https_errors` to True and launches Chromium with `--ignore-certificate-errors` on every start, so the provider `hpr install` switches a vault to fetched most HTML with verification off. Setting `ignore_https_errors = False` alone changes nothing (checked against a self-signed server on crawl4ai 0.8.6); the provider now also drops those launch flags, on both the plain launch and the login-profile (managed browser) launch, and refuses to start if a crawl4ai release moves the code it overrides. A refused certificate raises `CertVerificationError` naming the code (`ERR_CERT_AUTHORITY_INVALID`, ...) and the new opt-out, `[fetch] browser_verify_tls = false`. Before, it came back as an empty page and was reported as junk. As on the builtin and PDF lanes, the failure is final: no unverified retry, no escalation. In a batch it is a logged skip. The visible-window lane (`--visible` with a login profile) still ignores certificate errors, since some of the walled sites it serves have broken chains. Batch fetches also stopped mislabelling results: `arun_many` returns them in completion order, and they were paired with the requested URLs by position.- **A run tag is a slug (#116).** `Vault.run_dir()` joined the tag onto `research/runs/` unchecked, and pathlib replaces the base on an absolute segment, so `hpr run init ../../x` or `hpr run init C:/anything` scaffolded a run workspace outside the vault and every later `run` subcommand followed it there. Tags are now validated at that one seam: letters, digits, `-`, `_` and `.`, starting with a letter or digit, which is what `hpr vault-tag` mints. `run`, `levers` and `citecheck` report a bad tag as a clean error instead of a traceback. Same bug class as the `claims ingest --tag` traversal fixed in 0.11.1; exposure is low because the tag comes from the operator or the orchestrating agent, not from fetched content.- **The builtin provider fetches PDFs (#82, reported by @earldodd).** The PDF lane (`_is_pdf_url` / `_fetch_pdf`) lived inside the crawl4ai provider only, so a vault still on `provider = "builtin"` (the default until `hpr install` switches it) had no PDF handling at all: a direct `.pdf` link was decoded as HTML text and rejected by the junk gate as "Binary PDF garbage in content", identically for every mirror of the same document, and an arXiv `/abs/` link saved the 900-word abstract page instead of the paper. The lane now lives in `web/pdf.py` and both providers use it; the crawl4ai module keeps the old names. The builtin provider also detects a PDF from its bytes when the URL does not look like one. When the PDF lane declines a URL and the HTML fallback turns out to be junk, `hpr fetch -j` now says why the lane declined it (`PDF lane: HTTP 403`, `no extractable text layer`, ...) instead of the generic junk verdict, and the pymupdf document handle is closed on the exception path instead of leaking one per encrypted PDF.- **`serply` web provider.** Google organic results with page fetch through the same `SERPLY_API_KEY`; opt-in via `[web] provider = "serply"`, no new dependency.- **Quote integrity pairs short quoted spans (#120 reported by @earldodd, #122 from @akshaypal912).** `_QUOTE_SPAN_RE` required 20+ characters between quote marks, so a short quoted term ("TVL") failed to match at its opening mark; the scan then treated its closing mark as a new opening mark and reported the unquoted prose up to the next quote character as a hallucinated quote. One real report turned 38 legitimate short quotes into 42 false errors. The regex now matches any span up to 600 characters, and the existing fewer-than-5-words skip decides what gets verified.- **The SSRF gate classifies IPv4-mapped IPv6 by its embedded IPv4 (#127, from @fduple).** On Python 3.12+ the stdlib `is_*` properties see through `::ffff:a.b.c.d` to ranges they don't flag, and the CGNAT check only looked at IPv4 objects, so `::ffff:100.64.0.1` was fetchable. On 3.11 the opposite held: `::ffff:8.8.8.8` was refused. Mapped addresses are unmapped before classification on every version. An `allow_private_hosts` entry of the form `host:port` or `[v6]:port` now raises a clear error; it never matched anything, but note that it surfaces on every fetch while it is in the config. Bracketed IPv6 entries (`[::1]`) now parse.-- **A bad TLS certificate on the builtin provider raises `CertVerificationError` (#130, from @fduple).** It surfaced as a bare `httpx.ConnectError`. The failure is still final: nothing retries without verification, and the fetch is not queued as an escalation. The crawl4ai provider raises the same error on its PDF lane; its browser lanes still accept bad certificates (#137).+- **A bad TLS certificate on the builtin provider raises `CertVerificationError` (#130, from @fduple).** It surfaced as a bare `httpx.ConnectError`. The failure is still final: nothing retries without verification, and the fetch is not queued as an escalation. The crawl4ai provider raises the same error on its PDF lane, and now on its headless browser lane too (#137).- **Note ids are fixed points of the slugifier, and the link validator rejects shell fragments and traversal (#126, from @fduple).** `write_note` built the filename from the raw id while the frontmatter id was slugified, and `repair` / `graph stub` pass broken-link text through as the id, so fetched content chose the path: `[[../../outside]]` wrote outside the vault, and `[[-n "$kernel"]]` crashed on Windows. Stub files now get slug filenames. `is_valid_wiki_link_target` also refuses targets containing `$ " \` = \ < > ;`, a leading `-` or `!`, or a `.`/`..` segment. Slug ids never contain those, but a title or alias link such as `[[E=mc2]]` that used to resolve by title will drop out of the links table on the next sync.- **`note mv` keeps notes where sync can see them (#128, from @fduple).** The destination was joined onto the vault root verbatim, so a bare name landed at the root with no `.md` and left the index, `../x.md` wrote outside the vault, and an existing file was overwritten (POSIX) or crashed the command (Windows). A bare name now lands in `research/notes/` with `.md` added; destinations outside `research/notes/` and `research/temp/` are refused with `OUTSIDE_SYNCED_TREE`, and an existing file with `DESTINATION_EXISTS`. `research/index/` is refused too, since `repair` regenerates it and deletes every `.md` there. Case-only renames work on case-insensitive filesystems. The docstring no longer claims `mv` updates references; the id lives in frontmatter, so it never needed to.- **The builtin provider keeps fetched code as markdown code (#129, from @fduple).** `<pre>` blocks and inline `<code>` were flattened to prose, so bash test syntax (`[[ -n "$k" ]]`) reached the wiki-link parser and `repair` minted stub notes named after shell fragments. `<pre>` now becomes a fenced block whose fence outruns any backtick run inside it, inline `<code>` becomes a backtick span, and indentation survives in the stdlib fallback. Because `strip_markdown` drops code before indexing, code on builtin-fetched pages is no longer in full-text search, embeddings, or the quote-integrity comparison, which is how crawl4ai-fetched pages already behaved. A report that quotes a sentence containing an inline identifier can now fail quote integrity where it passed before.diff --git a/README.md b/README.mdindex 3d71380..b217479 100644--- a/README.md+++ b/README.md@@ -273,6 +273,8 @@ The `[web] provider` setting in `.hyperresearch/config.toml` picks how pages areprovider = "crawl4ai"```+Every fetch lane verifies TLS certificates by default: `builtin`, the PDF downloads, and crawl4ai's headless browser. A bad certificate fails the fetch with `CertVerificationError`; nothing retries it unverified. For a cert-broken site you trust, set the opt-out for the lane that refused it under `[fetch]`: `browser_verify_tls = false` for crawl4ai's headless browser, `pdf_verify_tls = false` for PDFs. The builtin provider has no opt-out. The visible-window browser used with a login profile (`--visible`, and the LinkedIn / Twitter-style domains) still accepts bad certificates, because some of the walled sites it exists for serve broken chains.+---## Authenticated crawling + the browser lanediff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.pyindex 5a99c87..32c48b2 100644--- a/src/hyperresearch/core/config.py+++ b/src/hyperresearch/core/config.py@@ -17,6 +17,11 @@ class FetchSettings:# disabled TLS verification for PDF downloads. Set to false only for# cert-broken mirrors you explicitly trust.pdf_verify_tls: bool = True+ # TLS verification for the crawl4ai headless browser lane (#137). Default+ # on; set false only for cert-broken sites you explicitly trust. The+ # visible-window lane (profile + visible browser) still ignores+ # certificate errors regardless of this setting.+ browser_verify_tls: bool = Truemin_pdf_bytes: int = 100# Response-size caps enforced by the SSRF gate (web/safe_http.py).# Defaults mirror the MAX_BYTES_* constants there.diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.pyindex 0ec3a9b..20949e3 100644--- a/src/hyperresearch/web/builtin.py+++ b/src/hyperresearch/web/builtin.py@@ -218,7 +218,8 @@ class BuiltinProvider:# Raise a certificate failure as its own type, the way the PDF lane# and the crawl4ai provider do, instead of letting the raw# httpx.ConnectError through looking like any other failed fetch.- # This lane always verifies TLS; pdf_verify_tls covers PDFs only.+ # This lane always verifies TLS; pdf_verify_tls covers PDFs only,+ # and browser_verify_tls the crawl4ai headless browser only.if _is_cert_error(exc):raise CertVerificationError(f"certificate verification failed for {url!r}: {exc}"diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.pyindex 94bc55e..e9ca1dd 100644--- a/src/hyperresearch/web/crawl4ai_provider.py+++ b/src/hyperresearch/web/crawl4ai_provider.py@@ -12,6 +12,7 @@ import asyncioimport loggingimport osimport pathlib+import reimport sysimport threadingfrom datetime import UTC, datetime@@ -110,6 +111,90 @@ def _smart_wait_js(settings: FetchSettings) -> str:)+# Chromium switches crawl4ai adds to EVERY launch, unconditionally+# (browser_manager.py: BrowserManager._build_browser_args for the plain launch,+# ManagedBrowser.build_browser_flags for the profile / managed-browser launch).+# They make the browser process accept any certificate, so+# BrowserConfig(ignore_https_errors=False) alone verifies nothing: checked+# against a self-signed server on crawl4ai 0.8.6, the page loads either way+# until these are removed as well.+_CERT_IGNORE_FLAGS = frozenset({+ "--ignore-certificate-errors",+ "--ignore-certificate-errors-spki-list",+})++# Chromium's certificate failures: net::ERR_CERT_* (authority, name, date,+# revoked, weak key, ...), ERR_CERTIFICATE_TRANSPARENCY_REQUIRED, and pinning.+_CERT_ERROR_RE = re.compile(+ r"net::(ERR_CERT[A-Z_]*|ERR_SSL_PINNED_KEY_NOT_IN_CERT_CHAIN)\b"+)++BROWSER_TLS_HINT = (+ "If this host is a known cert-broken site you trust, set "+ "browser_verify_tls = false under [fetch] in config.toml."+)+++def _cert_error_code(error_message: str | None) -> str | None:+ """The Chromium cert error code in a failed crawl4ai result, or None.++ crawl4ai does not raise on a navigation failure: ``arun`` returns+ ``success=False`` with the Playwright error text (``Page.goto:+ net::ERR_CERT_AUTHORITY_INVALID at https://...``) in ``error_message``.+ """+ if not error_message:+ return None+ m = _CERT_ERROR_RE.search(error_message)+ return m.group(1) if m else None+++def _browser_cert_refusal(url: str, code: str):+ from hyperresearch.web.safe_http import CertVerificationError++ return CertVerificationError(+ f"certificate verification failed for {url!r}: {code} (browser lane). "+ + BROWSER_TLS_HINT+ )+++def _strip_cert_ignore_flags(strategy: AsyncPlaywrightCrawlerStrategy) -> None:+ """Keep crawl4ai from launching Chromium with certificate errors ignored.++ Overrides the two flag builders on THIS strategy's browser manager only+ (instance attributes, not a patch of the crawl4ai classes). If a crawl4ai+ upgrade moves either builder, this raises instead of fetching with+ verification silently off.+ """+ bm = strategy.browser_manager+ build_args = getattr(bm, "_build_browser_args", None)+ if not callable(build_args):+ raise RuntimeError(+ "crawl4ai's BrowserManager._build_browser_args is missing; cannot "+ "turn off its --ignore-certificate-errors launch flag. "+ + BROWSER_TLS_HINT+ )++ def _verified_args() -> dict:+ args = build_args()+ args["args"] = [a for a in args.get("args", []) if a not in _CERT_IGNORE_FLAGS]+ return args++ bm._build_browser_args = _verified_args++ managed = getattr(bm, "managed_browser", None)+ if managed is not None:+ build_flags = getattr(type(managed), "build_browser_flags", None)+ if not callable(build_flags):+ raise RuntimeError(+ "crawl4ai's ManagedBrowser.build_browser_flags is missing; cannot "+ "turn off its --ignore-certificate-errors launch flag. "+ + BROWSER_TLS_HINT+ )+ managed.build_browser_flags = lambda config: [+ f for f in build_flags(config) if f not in _CERT_IGNORE_FLAGS+ ]++def _check_final_url(entry_url: str, final_url: str | None, settings: FetchSettings) -> None:"""Re-validate the URL the browser actually ended up on.@@ -168,6 +253,10 @@ class Crawl4AIProvider:browser_kwargs["user_data_dir"] = data_dirif cookies:browser_kwargs["cookies"] = cookies+ # crawl4ai defaults ignore_https_errors to True; verify unless the+ # operator opted out (#137). _make_crawler also strips the launch+ # flags that would otherwise override this.+ browser_kwargs["ignore_https_errors"] = not self._settings.browser_verify_tlsself._browser_config = BrowserConfig(**browser_kwargs)@@ -201,6 +290,8 @@ class Crawl4AIProvider:browser_config=self._browser_config,browser_adapter=UndetectedAdapter(),)+ if self._settings.browser_verify_tls:+ _strip_cert_ignore_flags(strategy)return AsyncWebCrawler(crawler_strategy=strategy, config=self._browser_config)def fetch(self, url: str) -> WebResult:@@ -251,6 +342,10 @@ class Crawl4AIProvider:from playwright.async_api import async_playwrightasync with async_playwright() as pw:+ # Certificate errors stay ignored here, whatever browser_verify_tls+ # says (#137): this lane exists for login- and bot-walled sites+ # reached through the user's own profile, and some of those serve+ # broken chains. The final-URL recheck below still applies.context = await pw.chromium.launch_persistent_context(user_data_dir=self._data_dir,headless=False,@@ -298,6 +393,12 @@ class Crawl4AIProvider:async def _fetch_async(self, url: str) -> WebResult:async with self._make_crawler() as crawler:result = await crawler.arun(url=url, config=self._run_config)+ # A refused certificate is final, like on the builtin and PDF+ # lanes: no unverified retry, no escalation. Without this it came+ # back as an empty page and was reported as junk.+ code = _cert_error_code(getattr(result, "error_message", None))+ if code and not getattr(result, "success", True):+ raise _browser_cert_refusal(url, code)# crawl4ai's result.url is the REQUESTED url even after the# browser followed redirects; the landing url is redirected_url# (hermetic-tier proven — rechecking result.url alone is blind).@@ -384,7 +485,8 @@ class Crawl4AIProvider:# back to the browser path — mirrors the single-fetch behaviour in# fetch() so batch callers don't silently lose academic PDFs that# need JS-rendered landing pages. Cert refusals are the exception:- # the browser lane ignores TLS errors, so falling back would be an+ # the browser lane may run with TLS errors ignored+ # (browser_verify_tls = false), so falling back would be an# automatic unverified retry. A batch has no per-URL failure# channel, so the skip is logged loudly instead.failed_pdf_urls = []@@ -410,8 +512,22 @@ class Crawl4AIProvider:if browser_urls:async with self._make_crawler() as crawler:results = await crawler.arun_many(urls=browser_urls, config=self._run_config)- for cr, url in zip(results, browser_urls, strict=False):+ requested = set(browser_urls)+ for cr, zipped_url in zip(results, browser_urls, strict=False):+ # arun_many yields results in COMPLETION order, not input+ # order, so zip() alone mislabels them. A result's .url is+ # the URL it was asked for; trust that when it is one of ours.+ cr_url = getattr(cr, "url", None)+ url = cr_url if cr_url in requested else zipped_urlif not cr.success:+ code = _cert_error_code(getattr(cr, "error_message", None))+ if code:+ log.warning(+ "SKIPPED (TLS certificate invalid): %s -- a potentially "+ "valuable source was not fetched. To include it, set "+ "browser_verify_tls = false under [fetch] in config.toml. (%s)",+ url, code,+ )continuetry:_check_final_url(diff --git a/src/hyperresearch/web/pdf.py b/src/hyperresearch/web/pdf.pyindex 7afaa18..5d48b79 100644--- a/src/hyperresearch/web/pdf.py+++ b/src/hyperresearch/web/pdf.py@@ -118,9 +118,10 @@ def safe_get_pdf(url: str, settings: FetchSettings):except Exception as exc:if settings.pdf_verify_tls and _is_cert_error(exc):# Refuse, but say how to opt out for a trusted cert-broken mirror.- # Raised as its own type: the browser lane runs with TLS errors- # ignored, so treating this like any failed PDF would hand the- # URL to a lane that amounts to an automatic unverified retry.+ # Raised as its own type: the browser lane can run with TLS errors+ # ignored (browser_verify_tls = false, or the visible lane), so+ # treating this like any failed PDF would hand the URL to a lane+ # that amounts to an automatic unverified retry.raise CertVerificationError(f"certificate verification failed for {url!r}: {exc}. ""If this host is a known cert-broken mirror you trust, set "@@ -235,7 +236,7 @@ def fetch_pdf_ex(A certificate refusal propagates as ``CertVerificationError``: callersmust not fold it into the generic failure, whose fallback is the- TLS-ignoring browser lane.+ browser lane, which may be configured to ignore TLS errors."""from hyperresearch.web.safe_http import CertVerificationError, SafeHTTPErrordiff --git a/src/hyperresearch/web/safe_http.py b/src/hyperresearch/web/safe_http.pyindex e733d43..309dd78 100644--- a/src/hyperresearch/web/safe_http.py+++ b/src/hyperresearch/web/safe_http.py@@ -57,7 +57,8 @@ class CertVerificationError(SafeHTTPError):"""TLS certificate verification failed and verification is required.Raised distinctly so callers can surface the refusal (and its- ``pdf_verify_tls = false`` opt-out) instead of treating it like any+ ``pdf_verify_tls = false`` or ``browser_verify_tls = false`` opt-out,+ depending on the lane) instead of treating it like anyother failed fetch — in particular, a cert-refused URL must never beretried through a lane that ignores TLS errors."""diff --git a/tests/test_web/test_crawl4ai_tls.py b/tests/test_web/test_crawl4ai_tls.pynew file mode 100644index 0000000..fd6a686--- /dev/null+++ b/tests/test_web/test_crawl4ai_tls.py@@ -0,0 +1,259 @@+"""The crawl4ai headless lane verifies TLS certificates by default (#137).++crawl4ai defaults ``BrowserConfig.ignore_https_errors`` to True AND launches+Chromium with ``--ignore-certificate-errors`` unconditionally, so verifying+takes both a config value and stripping those launch flags. These tests pin+both halves, the ``[fetch] browser_verify_tls = false`` opt-out, and the+refusal message. All offline: object construction and fake crawlers only,+no browser launch, no network.+"""++from __future__ import annotations++import asyncio+import logging+import threading+from pathlib import Path++import pytest++from hyperresearch.core.config import FetchSettings, VaultConfig+from hyperresearch.web.safe_http import CertVerificationError++provider = pytest.importorskip(+ "hyperresearch.web.crawl4ai_provider",+ reason="crawl4ai extra not installed",+)++CERT_FLAGS = {"--ignore-certificate-errors", "--ignore-certificate-errors-spki-list"}++# What crawl4ai 0.8.6 put in result.error_message against a self-signed server.+REAL_CERT_ERROR = (+ "Unexpected error in _crawl_web at line 500 in wrap_api_call (...):\n"+ "Error: Page.goto: net::ERR_CERT_AUTHORITY_INVALID at https://self-signed.example/\n"+ "Call log:\n - navigating to \"https://self-signed.example/\", waiting until "+ "\"domcontentloaded\"\n"+)+++def _run(coro):+ box: dict = {}++ def target() -> None:+ try:+ box["result"] = asyncio.run(coro)+ except Exception as exc:+ box["error"] = exc++ thread = threading.Thread(target=target)+ thread.start()+ thread.join()+ if "error" in box:+ raise box["error"]+ return box["result"]+++# ---------------------------------------------------------------------------+# Config+# ---------------------------------------------------------------------------+++def test_browser_verify_tls_defaults_true():+ assert FetchSettings().browser_verify_tls is True+++def test_browser_verify_tls_loads_and_round_trips(tmp_path: Path):+ p = tmp_path / "config.toml"+ p.write_text("[fetch]\nbrowser_verify_tls = false\n", encoding="utf-8")+ cfg = VaultConfig.load(p)+ assert cfg.fetch.browser_verify_tls is False+ assert cfg.fetch.pdf_verify_tls is True # independent knobs++ out = tmp_path / "saved.toml"+ cfg.save(out)+ assert "browser_verify_tls = false" in out.read_text(encoding="utf-8")+ assert VaultConfig.load(out).fetch.browser_verify_tls is False+++# ---------------------------------------------------------------------------+# BrowserConfig + launch flags+# ---------------------------------------------------------------------------++[email protected]("profile_dir", [None, "/tmp/does-not-matter"])+def test_browser_config_verifies_by_default(profile_dir):+ inst = provider.Crawl4AIProvider(headless=True, user_data_dir=profile_dir)+ assert inst._browser_config.ignore_https_errors is False++[email protected]("profile_dir", [None, "/tmp/does-not-matter"])+def test_browser_config_opt_out_ignores_errors(profile_dir):+ inst = provider.Crawl4AIProvider(+ headless=True,+ user_data_dir=profile_dir,+ settings=FetchSettings(browser_verify_tls=False),+ )+ assert inst._browser_config.ignore_https_errors is True+++def test_default_launch_drops_cert_ignore_flags():+ """ignore_https_errors=False alone is not enough: crawl4ai's own launch+ args carry --ignore-certificate-errors, which Chromium obeys over the+ context setting. This runs against the installed crawl4ai, so a release+ that moves the flag builder fails here rather than in the field."""+ bm = provider.Crawl4AIProvider(headless=True)._make_crawler().crawler_strategy.browser_manager+ args = bm._build_browser_args()["args"]+ assert not CERT_FLAGS & set(args)+ assert "--disable-blink-features=AutomationControlled" in args # rest intact+++def test_managed_browser_launch_drops_cert_ignore_flags():+ """The profile path launches Chromium as a subprocess and attaches over+ CDP; its flags come from ManagedBrowser.build_browser_flags."""+ inst = provider.Crawl4AIProvider(headless=True, user_data_dir="/tmp/does-not-matter")+ bm = inst._make_crawler().crawler_strategy.browser_manager+ assert bm.managed_browser is not None+ flags = bm.managed_browser.build_browser_flags(inst._browser_config)+ assert not CERT_FLAGS & set(flags)+ assert "--no-sandbox" in flags+++def test_stripping_is_per_instance_not_global():+ """The override must not leak into crawl4ai's classes or other crawlers."""+ from crawl4ai.browser_manager import ManagedBrowser++ inst = provider.Crawl4AIProvider(headless=True, user_data_dir="/tmp/does-not-matter")+ inst._make_crawler()+ assert set(ManagedBrowser.build_browser_flags(inst._browser_config)) >= CERT_FLAGS+++def test_opt_out_keeps_crawl4ai_launch_flags():+ inst = provider.Crawl4AIProvider(+ headless=True, settings=FetchSettings(browser_verify_tls=False)+ )+ args = inst._make_crawler().crawler_strategy.browser_manager._build_browser_args()["args"]+ assert set(args) >= CERT_FLAGS+++def test_missing_flag_builder_fails_closed():+ """If crawl4ai moves the builder, refuse rather than fetch unverified."""++ class _Strategy:+ browser_manager = object()++ with pytest.raises(RuntimeError, match="browser_verify_tls = false"):+ provider._strip_cert_ignore_flags(_Strategy())+++# ---------------------------------------------------------------------------+# Refusal mapping+# ---------------------------------------------------------------------------+++ ("message", "code"),+ [+ (REAL_CERT_ERROR, "ERR_CERT_AUTHORITY_INVALID"),+ ("Page.goto: net::ERR_CERT_COMMON_NAME_INVALID at https://x/", "ERR_CERT_COMMON_NAME_INVALID"),+ ("Page.goto: net::ERR_CERT_DATE_INVALID at https://x/", "ERR_CERT_DATE_INVALID"),+ ("net::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED", "ERR_CERTIFICATE_TRANSPARENCY_REQUIRED"),+ ("net::ERR_SSL_PINNED_KEY_NOT_IN_CERT_CHAIN", "ERR_SSL_PINNED_KEY_NOT_IN_CERT_CHAIN"),+ ("Page.goto: net::ERR_NAME_NOT_RESOLVED at https://x/", None),+ ("Page.goto: net::ERR_SSL_PROTOCOL_ERROR at https://x/", None),+ ("", None),+ (None, None),+ ],+)+def test_cert_error_code(message, code):+ assert provider._cert_error_code(message) == code+++class _FailedCR:+ def __init__(self, url: str, error_message: str):+ self.success = False+ self.url = url+ self.redirected_url = url+ self.error_message = error_message+ self.markdown = ""+ self.metadata = {}+ self.media = {}+ self.links = {}+ self.screenshot = None+ self.html = ""+++class _OkCR(_FailedCR):+ def __init__(self, url: str):+ super().__init__(url, "")+ self.success = True+ self.markdown = f"browser text for {url}"+ self.metadata = {"title": "T"}+ self.html = "<html></html>"+++class _FakeCrawler:+ def __init__(self, results: dict):+ self._results = results++ async def __aenter__(self):+ return self++ async def __aexit__(self, *exc):+ return False++ async def arun(self, url, config):+ return self._results[url]++ async def arun_many(self, urls, config):+ # Completion order, like the real dispatcher: reversed here.+ return [self._results[u] for u in reversed(urls)]+++def _bare_provider(monkeypatch, results: dict, settings: FetchSettings | None = None):+ inst = provider.Crawl4AIProvider.__new__(provider.Crawl4AIProvider)+ inst._settings = settings or FetchSettings()+ inst._gates = provider.JunkGates()+ inst._run_config = object()+ inst._headless = True+ inst._data_dir = None+ monkeypatch.setattr(inst, "_make_crawler", lambda: _FakeCrawler(results))+ return inst+++def test_single_fetch_cert_failure_is_a_clear_refusal(monkeypatch):+ url = "https://self-signed.example/"+ inst = _bare_provider(monkeypatch, {url: _FailedCR(url, REAL_CERT_ERROR)})++ with pytest.raises(CertVerificationError) as info:+ _run(inst._fetch_async(url))++ msg = str(info.value)+ assert "certificate verification failed" in msg+ assert "ERR_CERT_AUTHORITY_INVALID" in msg+ assert "browser_verify_tls = false under [fetch]" in msg+++def test_single_fetch_non_cert_failure_unchanged(monkeypatch):+ """Other navigation failures keep their old path (empty result -> junk gate)."""+ url = "https://gone.example/"+ inst = _bare_provider(+ monkeypatch, {url: _FailedCR(url, "Page.goto: net::ERR_NAME_NOT_RESOLVED")}+ )+ result = _run(inst._fetch_async(url))+ assert result.content == ""+++def test_batch_cert_failure_is_a_loud_skip_naming_the_right_url(monkeypatch, caplog):+ # IP literals: the batch entry gate resolves hostnames.+ bad, good = "https://8.8.8.8/self-signed", "https://8.8.8.8/fine"+ inst = _bare_provider(+ monkeypatch, {bad: _FailedCR(bad, REAL_CERT_ERROR), good: _OkCR(good)}+ )+ with caplog.at_level(logging.WARNING, logger="hyperresearch.web"):+ results = _run(inst._fetch_many_async([bad, good]))++ assert [r.url for r in results] == [good]+ skips = [r.getMessage() for r in caplog.records if "SKIPPED" in r.getMessage()]+ assert len(skips) == 1+ assert bad in skips[0] and good not in skips[0]+ assert "browser_verify_tls = false" in skips[0]