Universal SEO skill for Claude Code. 26 sub-skills + 19 sub-agents covering technical SEO, E-E-A-T, schema, GEO/AEO, agent readiness (Lighthouse Agentic Browsing, WebMCP, llms.txt), backlinks, local SEO, e-commerce, international SEO, Google APIs, and PDF/Excel reporting. 9 optional extensions for live SEO data.
#!/usr/bin/env python3
"""
Shared headless renderer for claude-seo.
Every subagent that fetches HTML for analysis (technical, content, schema,
geo, local, ecommerce, hreflang, images) calls this module instead of
``fetch_page.py`` whenever JS execution might change what an audit can see.
Built on Playwright Chromium with trafilatura for boilerplate-free content
extraction and htmldate for publication-date detection.
Why
===
Before v2.0.0 only ``seo-visual`` used Playwright. Every other agent
fetched raw HTML, which produces false negatives on SPAs (empty
``<div id="root">``, no schema in source, no content in source). The
gap analysis (see ``compass_artifact_*.md``) ranks "headless rendering
across all subagents" as the single highest-impact v2 change. This module
delivers it as a shared subsystem so the change is one foundation, not
eight retrofits.
Modes
=====
- ``auto`` : raw fetch first; render only when an SPA shell is detected
(see ``_is_spa``). Default. Cheapest correct behaviour.
- ``always`` : always render with Playwright, even for static HTML.
- ``never`` : raw HTML only. Equivalent to legacy ``fetch_page.py``.
Result shape
============
A dict with::
url final URL after redirects
status_code HTTP status of the main document
content HTML after JS execution (post-render DOM)
raw_content HTML before JS execution (server response)
is_spa True iff raw_content looks like a hydration shell
extracted_text trafilatura main-content extraction (or None)
publication_date htmldate ISO 8601 string (or None)
accessibility_tree Chromium accessibility-tree snapshot (or None)
accessibility_error capture failure detail (or None)
accessibility_partial True when the accessibility result is incomplete
headers response headers from the main document
redirect_chain list of {url, status_code}
console_errors list of browser console error strings
render_diagnostics list of non-fatal render degradation messages
render_engine 'playwright-chromium' or None
render_ms elapsed wall-clock for the render step
mode_used 'rendered' or 'raw'
too_large True when the raw body hit the size cap (MAX_PAGE_BYTES)
js_gap raw-versus-rendered difference (see below), or None
when no render happened (or the HTML was too large
to compare, which render_diagnostics then says)
error str or None
JavaScript content gap
======================
When a page is rendered, ``js_gap`` compares the HTML as served with the
DOM after JavaScript ran (:func:`js_gap`, a pure function over two HTML
strings)::
raw, rendered per version: words, links, schema_types, h1, title,
canonical, canonical_all, meta_robots
diff words_only_rendered and links_only_rendered with their
*_share of the rendered total (0 to 1),
schema_types_only_rendered, h1_only_rendered,
title_changed, canonical_changed, meta_robots_changed
raw_truncated True when the raw body was cut at the size cap, so
the gap may be overstated
significant True when ``reasons`` is not empty
reasons plain-language findings; empty when nothing stands out
Words are every text token outside script, style, noscript, template and
head; hidden elements are not detected. The share limits (``JS_GAP_*``) are
this tool's heuristics, not Google rules.
Every string in ``js_gap`` that comes from the page (schema type names,
title, canonical, robots directives, H1 text, and what ``reasons`` quotes)
has its whitespace collapsed and is cut at a ``JS_GAP_MAX_*`` length, and
every list has an item limit. ``--max-text`` does not apply to ``js_gap``;
these limits keep the whole object to about 24,000 characters of JSON.
SSRF
====
The URL is validated via :func:`url_safety.validate_url_strict` before
Playwright sees it. Inside Playwright a ``route()`` handler intercepts
every subresource and aborts requests whose hostname resolves to a
non-public IP. This is defence in depth against DNS rebinding inside
Chromium's resolver. The residual rebinding risk for browser fetches
is documented in SECURITY.md.
CLI
===
python render_page.py https://nuxt.com --mode always
python render_page.py https://example.com --mode auto --json
python render_page.py https://store.example.com --block image --block font
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import time
from collections import Counter
from typing import Any, Optional
from bs4 import BeautifulSoup
# Optional native dependencies. Each is checked lazily so callers that
# only need raw-mode (mode='never') don't pay the import cost.
try:
from playwright.sync_api import (
TimeoutError as PlaywrightTimeout,
)
from playwright.sync_api import (
sync_playwright,
)
except ImportError: # pragma: no cover - exercised in environments without playwright
sync_playwright = None
PlaywrightTimeout = Exception # type: ignore[assignment,misc]
try:
import trafilatura
except ImportError: # pragma: no cover
trafilatura = None
try:
from htmldate import find_date
except ImportError: # pragma: no cover
find_date = None
# Reuse the canonical safety module.
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
from url_safety import ( # noqa: E402 (sys.path massage above is intentional)
MAX_PAGE_BYTES,
URLSafetyError,
decode_body,
make_safe_playwright_route_handler,
new_guarded_context,
read_bounded,
safe_requests_get,
validate_url_strict,
)
VIEWPORTS: dict[str, dict[str, int]] = {
"desktop": {"width": 1920, "height": 1080, "device_scale": 1},
"laptop": {"width": 1366, "height": 768, "device_scale": 1},
"tablet": {"width": 768, "height": 1024, "device_scale": 1},
"mobile": {"width": 375, "height": 812, "device_scale": 2},
}
USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/150.0.7871.115 Safari/537.36 ClaudeSEO/2.0"
)
# Hydration-shell signatures. A marker is actionable only while meaningful
# body text is sparse, because SSR/SSG framework pages legitimately retain
# the same root and hydration markers after serving complete HTML.
_SPA_SHELL_PATTERNS = (
'<div id="root"></div>',
'<div id="__next">',
'<div id="app"></div>',
'<div id="__nuxt">',
'data-svelte-h=',
'<astro-island ',
'you need to enable javascript',
'please enable javascript',
)
# Builder markers are supporting evidence only. Wix, Webflow, and Squarespace
# can all serve complete HTML, so auto-render requires multiple same-builder
# markers plus sparse meaningful body text.
_BUILDER_FINGERPRINT_GROUPS = (
("wix-warmup-data", "static.parastorage.com", 'content="wix.com'),
("data-wf-page", "data-wf-site"),
('content="squarespace', "static1.squarespace.com"),
)
_TAG_STRIP = re.compile(r"<[^>]+>")
_WHITESPACE = re.compile(r"\s+")
_NON_VISIBLE_STRIP = re.compile(
r"<(script|style|template|noscript)\b[^>]*>.*?</\1>",
re.IGNORECASE | re.DOTALL,
)
_BUILDER_SPARSE_TEXT_MAX = 400
JSON_LD_MAX_BLOCKS = 50
JSON_LD_MAX_BLOCK_BYTES = 256 * 1024
JSON_LD_MAX_TOTAL_BYTES = 1024 * 1024
JSON_LD_MAX_NODES = 10_000
JSON_LD_MAX_DEPTH = 40
ACCESSIBILITY_MAX_NODES = 10_000
ACCESSIBILITY_MAX_DEPTH = 50
# JavaScript content gap. These limits decide when a raw-versus-rendered
# difference is called "significant". They are this tool's heuristics, not
# Google rules: Google publishes no threshold for either.
JS_GAP_WORDS_SHARE = 0.30 # share of rendered words that exist only after JS
JS_GAP_WORDS_MIN = 25 # and at least this many such words
JS_GAP_LINKS_SHARE = 0.30 # share of rendered links that exist only after JS
JS_GAP_LINKS_MIN = 3 # and at least this many such links
# Output bounds, so a hostile page cannot inflate the JSON.
JS_GAP_MAX_ITEMS = 12 # schema types, H1 texts, robots directives
JS_GAP_MAX_CANONICALS = 5
JS_GAP_MAX_TEXT = 300 # characters kept per title, H1 or canonical
JS_GAP_MAX_NAME = 80 # characters kept per schema type name or robots directive
JS_GAP_REASON_TEXT = 80 # characters of page text quoted inside a reason
JS_GAP_REASON_ITEMS = 5 # schema type names listed inside a reason
# The comparison parses both versions again. Past this size it is skipped
# (js_gap stays None, with a render diagnostic) so a huge DOM cannot stall it.
JS_GAP_MAX_HTML_CHARS = 5_000_000
_WORD = re.compile(r"\w+")
_NON_VISIBLE_TAGS = ("script", "style", "noscript", "template", "head", "title")
_NON_LINK_HREF = ("#", "javascript:", "mailto:", "tel:", "data:")
def _visible_body_text(lower_html: str) -> str:
body_start = lower_html.find("<body")
body_end = lower_html.rfind("</body>")
if body_start == -1 or body_end <= body_start:
return ""
body = _NON_VISIBLE_STRIP.sub(" ", lower_html[body_start:body_end])
return _WHITESPACE.sub(" ", _TAG_STRIP.sub(" ", body)).strip()
def _schema_types(data: object) -> tuple[list[str], bool]:
"""Collect bounded @type values without recursive attacker-controlled calls."""
types: set[str] = set()
stack: list[tuple[object, int]] = [(data, 0)]
visited = 0
truncated = False
while stack:
value, depth = stack.pop()
visited += 1
if visited > JSON_LD_MAX_NODES or depth > JSON_LD_MAX_DEPTH:
truncated = True
break
if isinstance(value, dict):
schema_type = value.get("@type")
if isinstance(schema_type, str):
types.add(schema_type)
elif isinstance(schema_type, list):
types.update(item for item in schema_type if isinstance(item, str))
stack.extend((item, depth + 1) for item in value.values())
elif isinstance(value, list):
stack.extend((item, depth + 1) for item in value)
return sorted(types)[:100], truncated or len(types) > 100
def _extract_json_ld(
html: Optional[str],
*,
include_full: bool = False,
soup: Optional[BeautifulSoup] = None,
) -> dict:
"""Extract full-page JSON-LD with strict block, byte, and traversal bounds.
``soup`` lets a caller that already parsed ``html`` skip a second parse.
"""
result = {
"block_count": 0,
"processed_count": 0,
"total_bytes": 0,
"truncated": False,
"blocks": [],
}
if not html:
return result
if soup is None:
soup = BeautifulSoup(html, "html.parser")
scripts = [
script for script in soup.find_all("script")
if str(script.get("type", "")).strip().lower() == "application/ld+json"
]
result["block_count"] = len(scripts)
for index, script in enumerate(scripts):
if index >= JSON_LD_MAX_BLOCKS:
result["truncated"] = True
break
raw = script.string if script.string is not None else script.get_text()
raw = str(raw or "").strip()
size_bytes = len(raw.encode("utf-8"))
if result["total_bytes"] + size_bytes > JSON_LD_MAX_TOTAL_BYTES:
result["truncated"] = True
break
result["total_bytes"] += size_bytes
result["processed_count"] += 1
entry = {"index": index + 1, "size_bytes": size_bytes}
if size_bytes > JSON_LD_MAX_BLOCK_BYTES:
entry.update({
"valid": None,
"error": "block exceeds the JSON-LD per-block byte limit",
})
result["truncated"] = True
result["blocks"].append(entry)
continue
try:
parsed = json.loads(raw)
types, types_truncated = _schema_types(parsed)
entry.update({
"valid": True,
"types": types,
"types_truncated": types_truncated,
})
if include_full:
entry["data"] = parsed
except (json.JSONDecodeError, RecursionError) as exc:
entry.update({
"valid": False,
"error": f"{type(exc).__name__}: {exc}",
})
if include_full:
entry["raw"] = raw
result["blocks"].append(entry)
return result
def _is_spa(raw_html: Optional[str]) -> bool:
"""Detect sparse hydration shells without flagging complete SSR/SSG pages."""
if not raw_html:
return True
lc = raw_html.lower()
visible_text = _visible_body_text(lc)
if (
len(visible_text) < _BUILDER_SPARSE_TEXT_MAX
and any(pattern in lc for pattern in _SPA_SHELL_PATTERNS)
):
return True
if len(visible_text) < _BUILDER_SPARSE_TEXT_MAX:
for markers in _BUILDER_FINGERPRINT_GROUPS:
if sum(marker in lc for marker in markers) >= 2:
return True
# Very thin <body> suggests JS-rendered content even without a shell.
# Threshold (100 chars) sits between typical SPA shells (0-50 chars of
# body text) and minimal informational pages like example.com (~125
# chars). Tuned conservatively to avoid false positives that would
# force a redundant Playwright render in auto mode.
body_start = lc.find("<body")
body_end = lc.rfind("</body>")
if body_start != -1 and body_end > body_start:
if len(visible_text) < 100:
return True
return False
def _clean_text(value: object, limit: Optional[int] = None) -> str:
"""Page text for js_gap: whitespace collapsed, cut at ``limit`` characters."""
return _WHITESPACE.sub(" ", str(value or "")).strip()[:limit or JS_GAP_MAX_TEXT]
def _gap_signals(html: str) -> tuple[dict, dict]:
"""Collect one version's SEO signals for :func:`js_gap`.
Returns the published signals plus the word and link multisets used to
work out what exists only after rendering. Raw and rendered HTML go
through the same code, so the two sides are counted the same way.
"""
soup = BeautifulSoup(html or "", "html.parser")
# Dicts keep first-seen order and make de-duplication linear, so a page
# with a huge number of distinct values cannot slow the comparison down.
# A type name is page text like any other: cleaned and cut before it is
# kept, so a huge @type value cannot ride into the output.
schema_types: dict[str, None] = {}
for block in _extract_json_ld(html, soup=soup)["blocks"]:
for name in block.get("types") or []:
name = _clean_text(name, JS_GAP_MAX_NAME)
if name:
schema_types[name] = None
# An <svg><title> is an accessible name, not the document title.
title = None
for tag in soup.find_all("title"):
if tag.find_parent("svg") is None:
title = _clean_text(tag.get_text(" ")) or None
break
canonicals: dict[str, None] = {}
for tag in soup.find_all("link"):
rel = tag.get("rel") or []
rel = rel.split() if isinstance(rel, str) else rel
href = _clean_text(tag.get("href"))
if href and any(str(item).lower() == "canonical" for item in rel):
canonicals[href] = None
# Several robots meta tags combine, so their directives are merged.
robots: dict[str, None] = {}
for tag in soup.find_all("meta"):
if str(tag.get("name") or "").strip().lower() != "robots":
continue
for directive in str(tag.get("content") or "").lower().split(","):
directive = _clean_text(directive, JS_GAP_MAX_NAME)
if directive:
robots[directive] = None
for tag in soup(list(_NON_VISIBLE_TAGS)):
tag.decompose()
h1: dict[str, None] = {}
for tag in soup.find_all("h1"):
text = _clean_text(tag.get_text(" ", strip=True))
if text:
h1[text] = None
hrefs: Counter = Counter()
for anchor in soup.find_all("a", href=True):
href = str(anchor.get("href") or "").strip()
if href and not href.lower().startswith(_NON_LINK_HREF):
hrefs[href] += 1
words = Counter(_WORD.findall(soup.get_text(" ", strip=True).lower()))
signals = {
"words": sum(words.values()),
"links": sum(hrefs.values()),
"schema_types": list(schema_types)[:JS_GAP_MAX_ITEMS],
"h1": list(h1)[:JS_GAP_MAX_ITEMS],
"title": title,
"canonical": next(iter(canonicals), None),
"canonical_all": list(canonicals)[:JS_GAP_MAX_CANONICALS],
"meta_robots": ", ".join(list(robots)[:JS_GAP_MAX_ITEMS]) or None,
}
return signals, {"words": words, "hrefs": hrefs}
def _only_after(raw: Counter, rendered: Counter) -> tuple[int, float]:
"""Items in ``rendered`` with no counterpart in ``raw``: count and share."""
total = sum(rendered.values())
added = sum((rendered - raw).values())
return added, (round(added / total, 3) if total else 0.0)
def js_gap(
raw_html: Optional[str],
rendered_html: Optional[str],
*,
raw_truncated: bool = False,
) -> Optional[dict]:
"""Compare the HTML as served with the DOM after JavaScript ran.
Pure: two HTML strings in, a JSON-ready dict out, no browser or network.
Returns None when either version is missing (no render happened). An
empty string is a real, empty document and is compared.
Title, canonical and meta robots are reported as changed when the raw
and rendered values differ in any way (added, removed or replaced). A
mismatch between the two is the case Google's JavaScript SEO guidance
warns about.
"""
if raw_html is None or rendered_html is None:
return None
raw, raw_sets = _gap_signals(raw_html)
rendered, rendered_sets = _gap_signals(rendered_html)
words_added, words_share = _only_after(raw_sets["words"], rendered_sets["words"])
links_added, links_share = _only_after(raw_sets["hrefs"], rendered_sets["hrefs"])
gap: dict = {
"raw": raw,
"rendered": rendered,
"diff": {
"words_only_rendered": words_added,
"words_only_rendered_share": words_share,
"links_only_rendered": links_added,
"links_only_rendered_share": links_share,
"schema_types_only_rendered": [
name for name in rendered["schema_types"]
if name not in raw["schema_types"]
],
"h1_only_rendered": [text for text in rendered["h1"] if text not in raw["h1"]],
"title_changed": raw["title"] != rendered["title"],
"canonical_changed": (
set(raw["canonical_all"]) != set(rendered["canonical_all"])
),
"meta_robots_changed": (
set((raw["meta_robots"] or "").split(", "))
!= set((rendered["meta_robots"] or "").split(", "))
),
},
"raw_truncated": bool(raw_truncated),
}
gap["reasons"] = js_gap_reasons(gap)
gap["significant"] = bool(gap["reasons"])
return gap
def _quoted(value: object) -> str:
"""Page text for a reason line: bounded, or 'none' when absent."""
if isinstance(value, list):
value = " | ".join(str(item) for item in value)
text = str(value or "")
if not text:
return "none"
if len(text) > JS_GAP_REASON_TEXT:
text = text[:JS_GAP_REASON_TEXT] + "..."
return f"'{text}'"
def js_gap_reasons(gap: Optional[dict]) -> list[str]:
"""Plain-language reasons a :func:`js_gap` result matters. Empty when none do."""
if not gap:
return []
raw, rendered, diff = gap["raw"], gap["rendered"], gap["diff"]
reasons: list[str] = []
if (
diff["words_only_rendered_share"] >= JS_GAP_WORDS_SHARE
and diff["words_only_rendered"] >= JS_GAP_WORDS_MIN
):
reasons.append(
f"{round(diff['words_only_rendered_share'] * 100)}% of visible words appear "
f"only after JavaScript runs ({raw['words']} raw, {rendered['words']} rendered)"
)
if (
diff["links_only_rendered_share"] >= JS_GAP_LINKS_SHARE
and diff["links_only_rendered"] >= JS_GAP_LINKS_MIN
):
reasons.append(
f"{diff['links_only_rendered']} of {rendered['links']} links appear only "
f"after JavaScript runs ({raw['links']} in the raw HTML)"
)
if diff["schema_types_only_rendered"]:
reasons.append(
"structured data appears only after JavaScript: "
+ ", ".join(
_quoted(name)
for name in diff["schema_types_only_rendered"][:JS_GAP_REASON_ITEMS]
)
)
if diff["h1_only_rendered"]:
reasons.append(
"the H1 changes after JavaScript" if raw["h1"]
else "the H1 appears only after JavaScript"
)
for flag, label, key in (
("title_changed", "title", "title"),
("canonical_changed", "canonical URL", "canonical_all"),
("meta_robots_changed", "meta robots value", "meta_robots"),
):
if diff[flag]:
reasons.append(
f"the {label} is not the same before and after JavaScript "
f"(raw: {_quoted(raw[key])}, rendered: {_quoted(rendered[key])})"
)
return reasons
def _wait_for_dom_stability(page, timeout_ms: int) -> bool: # type: ignore[no-untyped-def]
"""Wait up to five seconds for meaningful body text and a stable DOM."""
budget_ms = max(250, min(timeout_ms, 5000))
previous = None
stable_samples = 0
elapsed_ms = 0
while elapsed_ms < budget_ms:
try:
signature = tuple(page.evaluate(
"() => ["
"(document.body && document.body.innerText || '').trim().length,"
"document.querySelectorAll('*').length"
"]"
))
except Exception:
return False
if signature == previous and signature[0] >= 100:
stable_samples += 1
if stable_samples >= 2:
return True
else:
stable_samples = 0
previous = signature
page.wait_for_timeout(250)
elapsed_ms += 250
return False
def _ax_value(node: dict[str, Any], key: str) -> str:
value = node.get(key)
if isinstance(value, dict):
raw = value.get("value")
return str(raw) if raw is not None else ""
return str(value) if value is not None else ""
def _accessibility_tree_from_cdp(nodes: object) -> tuple[Optional[dict], bool]:
"""Convert Chrome's flat AXNode array into the tree used by the auditor."""
if not isinstance(nodes, list) or not nodes:
return None, False
usable = [node for node in nodes if isinstance(node, dict)]
truncated = len(usable) > ACCESSIBILITY_MAX_NODES
usable = usable[:ACCESSIBILITY_MAX_NODES]
by_id = {
str(node["nodeId"]): node
for node in usable
if node.get("nodeId") is not None
}
if not by_id:
return None, truncated
known_children = {
str(child_id)
for node in by_id.values()
for child_id in (node.get("childIds") or [])
if str(child_id) in by_id
}
root_id = next((node_id for node_id in by_id if node_id not in known_children), None)
if root_id is None:
return None, True
def build(node_id: str, depth: int, ancestors: frozenset[str]) -> dict:
nonlocal truncated
node = by_id[node_id]
result = {
"role": _ax_value(node, "role"),
"name": _ax_value(node, "name"),
"ignored": bool(node.get("ignored", False)),
}
child_ids = [
str(child_id)
for child_id in (node.get("childIds") or [])
if str(child_id) in by_id
]
if depth >= ACCESSIBILITY_MAX_DEPTH:
if child_ids:
truncated = True
return result
children = []
for child_id in child_ids:
if child_id in ancestors:
truncated = True
continue
children.append(build(child_id, depth + 1, ancestors | {child_id}))
if children:
result["children"] = children
return result
return build(root_id, 0, frozenset({root_id})), truncated
def _capture_accessibility_tree(context: Any, page: Any) -> tuple[Optional[dict], bool]:
"""Capture the Chromium AX tree through Playwright's supported CDP seam."""
session = context.new_cdp_session(page)
try:
session.send("Accessibility.enable")
payload = session.send(
"Accessibility.getFullAXTree", {"depth": ACCESSIBILITY_MAX_DEPTH}
)
return _accessibility_tree_from_cdp(payload.get("nodes"))
finally:
try:
session.send("Accessibility.disable")
except Exception:
pass
detach = getattr(session, "detach", None)
if detach:
try:
detach()
except Exception:
pass
def render_page(
url: str,
*,
mode: str = "auto",
viewport: str = "desktop",
timeout_ms: int = 15000,
block_resources: Optional[list[str]] = None,
extract_content: bool = True,
extract_accessibility: bool = False,
user_agent: Optional[str] = None,
) -> dict:
"""Render or fetch ``url`` per the chosen mode. See module docstring.
``extract_accessibility``: when True and the page is rendered (mode
'always' or 'auto'+SPA), the Chromium accessibility tree is captured via
a Playwright CDP session and attached to ``result['accessibility_tree']``. Used by
``agent_ux_check.py`` for agent-friendliness scoring (Google AI
optimization guide / web.dev agent UX criteria).
"""
result: dict = {
"url": url,
"status_code": None,
"content": None,
"raw_content": None,
"is_spa": None,
"extracted_text": None,
"publication_date": None,
"accessibility_tree": None,
"accessibility_error": None,
"accessibility_partial": False,
"headers": {},
"redirect_chain": [],
"console_errors": [],
"render_diagnostics": [],
"render_engine": None,
"render_ms": None,
"mode_used": None,
"too_large": False,
"js_gap": None,
"error": None,
}
if mode not in ("auto", "always", "never"):
result["error"] = f"Invalid mode: {mode!r}"
return result
if viewport not in VIEWPORTS:
result["error"] = f"Invalid viewport: {viewport!r}"
return result
# Pre-flight SSRF check.
try:
norm_url, _pinned_ip = validate_url_strict(url)
result["url"] = norm_url
except URLSafetyError as exc:
result["error"] = f"url_safety: {exc}"
return result
# Step 1: raw fetch (always; needed for SPA detection and as a baseline).
try:
resp = safe_requests_get(norm_url, timeout=30, allow_redirects=True, stream=True)
# Streamed so the body is read under the cap; .content must not be
# read after this.
try:
raw, result["too_large"] = read_bounded(resp, MAX_PAGE_BYTES)
finally:
resp.close()
result["raw_content"] = decode_body(raw, resp.headers.get("Content-Type", ""))
if result["too_large"]:
result["render_diagnostics"].append(
"raw body exceeded the size cap; raw_content holds the part read before it"
)
if resp.history:
result["redirect_chain"] = [
{"url": r.url, "status_code": r.status_code} for r in resp.history
]
raw_status = resp.status_code
raw_headers = dict(resp.headers)
final_raw_url = resp.url
except Exception as exc:
result["error"] = f"raw fetch failed: {exc}"
return result
result["is_spa"] = _is_spa(result["raw_content"])
should_render = mode == "always" or (mode == "auto" and result["is_spa"])
if not should_render:
result["mode_used"] = "raw"
result["url"] = final_raw_url
result["status_code"] = raw_status
result["headers"] = raw_headers
result["content"] = result["raw_content"]
else:
result["mode_used"] = "rendered"
def _keep_raw_on_render_failure() -> None:
# The render failed, but the raw fetch succeeded: keep its status,
# headers and URL so callers still see what the server answered.
result["url"] = final_raw_url
result["status_code"] = raw_status
result["headers"] = raw_headers
result["raw_fallback_available"] = True
if sync_playwright is None:
result["error"] = (
"playwright is required for rendered mode. "
"Install: pip install -r requirements.txt "
"&& playwright install chromium"
)
_keep_raw_on_render_failure()
return result
vp = VIEWPORTS[viewport]
blocked = set(block_resources or [])
route_handler = make_safe_playwright_route_handler(blocked)
start = time.monotonic()
try:
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
context = new_guarded_context(
browser,
route_handler,
viewport={"width": vp["width"], "height": vp["height"]},
device_scale_factor=vp["device_scale"],
user_agent=user_agent or USER_AGENT,
)
page = context.new_page()
def _on_console(msg): # type: ignore[no-untyped-def]
if msg.type == "error":
result["console_errors"].append(msg.text)
page.on("console", _on_console)
page.route("**/*", route_handler)
try:
response = page.goto(
norm_url, wait_until="domcontentloaded", timeout=timeout_ms
)
except PlaywrightTimeout:
response = None
result["render_diagnostics"].append(
f"DOMContentLoaded timed out after {timeout_ms}ms; "
"captured the available DOM"
)
if not _wait_for_dom_stability(page, timeout_ms):
result["render_diagnostics"].append(
"DOM did not reach the bounded stability threshold; "
"captured the available DOM"
)
result["url"] = page.url
result["content"] = page.content()
result["status_code"] = response.status if response else raw_status
result["headers"] = (
dict(response.all_headers()) if response else raw_headers
)
result["render_engine"] = "playwright-chromium"
if extract_accessibility:
try:
(
result["accessibility_tree"],
result["accessibility_partial"],
) = _capture_accessibility_tree(context, page)
if result["accessibility_tree"] is None:
result["accessibility_partial"] = True
result["accessibility_error"] = (
"Chromium returned no accessibility nodes"
)
elif result["accessibility_partial"]:
result["accessibility_error"] = (
"accessibility tree exceeded the capture bounds"
)
except Exception as exc:
result["accessibility_tree"] = None
result["accessibility_partial"] = True
result["accessibility_error"] = (
f"accessibility capture failed: {type(exc).__name__}: {exc}"
)
result["render_diagnostics"].append(
result["accessibility_error"]
)
browser.close()
except Exception as exc:
result["error"] = f"playwright error: {exc}"
_keep_raw_on_render_failure()
return result
finally:
result["render_ms"] = (time.monotonic() - start) * 1000.0
# Both versions exist: measure what only JavaScript provides.
try:
largest = max(len(result["raw_content"] or ""), len(result["content"] or ""))
if largest > JS_GAP_MAX_HTML_CHARS:
result["render_diagnostics"].append(
f"js_gap skipped: HTML is larger than {JS_GAP_MAX_HTML_CHARS} characters"
)
else:
result["js_gap"] = js_gap(
result["raw_content"],
result["content"],
raw_truncated=result["too_large"],
)
except Exception as exc:
# The comparison is best-effort; never block the audit on it.
result["render_diagnostics"].append(
f"js_gap comparison failed: {type(exc).__name__}: {exc}"
)
# Step 2: content extraction (works on either raw or rendered HTML).
if extract_content and result["content"]:
if trafilatura is not None:
try:
result["extracted_text"] = trafilatura.extract(
result["content"],
include_comments=False,
include_tables=True,
favor_recall=False,
)
except Exception:
# Extraction is best-effort; never block the audit on it.
pass
if find_date is not None:
try:
result["publication_date"] = find_date(result["content"])
except Exception:
pass
return result
def _non_negative_int(value: str) -> int:
parsed = int(value)
if parsed < 0:
raise argparse.ArgumentTypeError("must be zero or greater")
return parsed
def _json_summary(result: dict, *, max_text: int = 0) -> dict:
"""Return an explicit, machine-readable JSON output contract.
``max_text=0`` preserves every textual field. Positive values truncate
content fields and record the original and returned character counts.
"""
summary = dict(result)
full_content = result.get("content") or result.get("raw_content") or ""
summary["structured_data"] = _extract_json_ld(full_content)
truncation = {"limit": max_text or None, "fields": {}}
for field in ("content", "raw_content", "extracted_text"):
value = summary.get(field)
if not isinstance(value, str):
continue
original_chars = len(value)
truncated = max_text > 0 and original_chars > max_text
if truncated:
summary[field] = value[:max_text]
truncation["fields"][field] = {
"original_chars": original_chars,
"returned_chars": len(summary[field]),
"truncated": truncated,
}
summary["truncation"] = truncation
return summary
def _cli() -> None:
parser = argparse.ArgumentParser(
description="claude-seo shared headless renderer (Playwright + trafilatura)"
)
parser.add_argument("url", help="URL to render")
parser.add_argument(
"--mode",
choices=("auto", "always", "never"),
default="auto",
help="auto: render only when SPA detected; always: always render; "
"never: raw HTML only (default: auto)",
)
parser.add_argument(
"--viewport", choices=list(VIEWPORTS), default="desktop"
)
parser.add_argument(
"--timeout-ms",
type=int,
default=15000,
help="Playwright navigation timeout in ms (default: 15000)",
)
parser.add_argument(
"--block",
action="append",
default=[],
choices=("image", "media", "font", "stylesheet"),
help="resource types to block during render (faster, less accurate)",
)
parser.add_argument(
"--no-extract",
action="store_true",
help="skip trafilatura and htmldate post-processing",
)
parser.add_argument(
"--a11y-tree",
action="store_true",
help="capture Playwright accessibility-tree snapshot (forces render)",
)
parser.add_argument(
"--json",
action="store_true",
help="emit JSON with full content fields unless --max-text is set",
)
parser.add_argument(
"--max-text",
type=_non_negative_int,
default=0,
help=(
"maximum characters returned for each JSON content field; "
"0 keeps full text (default: 0)"
),
)
parser.add_argument(
"--json-ld-output",
help=(
"write bounded full JSON-LD extraction to a UTF-8 JSON file; "
"normal --json output contains summaries only"
),
)
parser.add_argument("--output", "-o", help="write HTML content to file")
args = parser.parse_args()
effective_mode = "always" if args.a11y_tree else args.mode
res = render_page(
args.url,
mode=effective_mode,
viewport=args.viewport,
timeout_ms=args.timeout_ms,
block_resources=args.block or None,
extract_content=not args.no_extract,
extract_accessibility=args.a11y_tree,
)
full_content = res.get("content") or res.get("raw_content") or ""
if args.json_ld_output:
extraction = _extract_json_ld(full_content, include_full=True)
with open(args.json_ld_output, "w", encoding="utf-8") as fh:
json.dump(extraction, fh, indent=2, ensure_ascii=False)
output_written = False
if args.output and not res["error"]:
with open(args.output, "w", encoding="utf-8") as fh:
fh.write(res["content"] or "")
output_written = True
if args.json:
summary = _json_summary(res, max_text=args.max_text)
summary["output_written"] = output_written
print(json.dumps(summary, indent=2, default=str))
sys.exit(1 if res["error"] else 0)
if res["error"]:
print(f"Error: {res['error']}", file=sys.stderr)
sys.exit(1)
if output_written:
print(f"saved to {args.output}", file=sys.stderr)
else:
print(res["content"])
print(
f"\nFinal URL: {res['url']}\n"
f"Status: {res['status_code']} | mode={res['mode_used']} | "
f"is_spa={res['is_spa']}",
file=sys.stderr,
)
if res["render_ms"]:
print(
f"Render: {res['render_ms']:.0f}ms via {res['render_engine']}",
file=sys.stderr,
)
if res["publication_date"]:
print(f"Publication date: {res['publication_date']}", file=sys.stderr)
for reason in (res.get("js_gap") or {}).get("reasons", []):
print(f"JS gap: {reason}", file=sys.stderr)
if res["console_errors"]:
print(
f"Console errors ({len(res['console_errors'])}):", file=sys.stderr
)
for err in res["console_errors"][:5]:
print(f" - {err}", file=sys.stderr)
if __name__ == "__main__":
_cli()