https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/
feat(render): one CDP backend, auto turbo/standard (delete websocket.py) (#38)
Zhifei Li7c2057a1dfdc
diff --git a/docs/internal/screenshot-optimization-notes.md b/docs/internal/screenshot-optimization-notes.mdindex 01d52c1..d16c5d9 100644--- a/docs/internal/screenshot-optimization-notes.md+++ b/docs/internal/screenshot-optimization-notes.md@@ -182,3 +182,58 @@ Lab machines have 8× H200/B200 GPUs but:To unlock GPU: `sudo usermod -aG render $USER` on lab machine.Expected impact: 4x faster DrawRenderPass based on production system data.++## Backend reconciliation & SPA-render fix (2026-06-11)++### The three render code paths (who actually runs what)+- `backends/websocket.py` — the **shipped** general-purpose renderer. The `pixelshot`+ CLI, the `pixelbrowse` skill, and the `pixelrag index` pipeline (`render_urls`,+ `backend="cdp"`/`"websocket"`) all go through it. Simple: per-worker queue, inline+ JPEG over CDP, no extra deps.+- `backends/fast_cdp.py` — high-throughput batch path (`render_articles`): phased-logic+ capture + rawFilePath to /dev/shm + ProcessPool JPEG. **No in-repo caller** — invoked+ only by an out-of-repo ops script. The 8.28M flagship Wikipedia index was built by a+ *separate* system (Playwright/GPU/4-machine, see "Production System Comparison"), not+ by either of these.+- `strategies/*` — the benchmarking menu; used only by `bench/`. Kept as research scaffolding.++### Regression fixed: websocket backend rendered SPAs / tall pages wrong+`backends/websocket.py` had drifted from the established capture pattern — it had **no+nav-completion wait** (fired `document.fonts.ready` immediately after `Page.navigate`)+and **no per-tile scroll**, both of which `fast_cdp` and the production strategies have.+Consequences:+- JS/SPA pages were measured/captured mid-hydration at a transient (often much taller)+ layout → tiled into mostly-empty space = blank tiles (this is the "tile loop overshoots+ page height" blank bug noted under "Production System Comparison", here root-caused).+- At small `tile_height` (the skill uses 1568) every tile past the first was blank,+ because content below the short device viewport is never rasterized without scrolling.++Fix (verified in `bench/` against ground truth at 100% on the smoke set):+- Wait for the `load` event before measuring/capturing (`readyState==='complete'`+ shortcut + 12s cap). SSR pages fire `load` ~as fast as `fonts.ready`, so ~0 cost+ (measured: Wikipedia render time unchanged).+- Scroll each tile into view before capture (mirrors `fast_cdp`).+- Optional `--wait-network-idle` (JS PerformanceObserver) for pages that fetch content+ after load; off by default (costs a quiet window/page), on by default in the skill.++### Raw vs inline-JPEG is the dominant throughput lever (measured, 48w, N=600, this box)+| config | correct | t/s | note |+|---|---|---|---|+| phased **raw** (fast_cdp config) | 99.7% | **306** | capture-only in bench; JPEG is decoupled/parallel |+| phased jpeg (inline) | 98.2% | 182 | Chrome encodes JPEG on the capture critical path |+| sequential raw | 99.7% | 221 | |+| sequential jpeg (inline) | 98.2% | 142 | |++Takeaways: (1) **inline JPEG encoding is the bottleneck** — bypassing it with rawFilePath++ parallel compression is ~+56-68%. (2) phased's semaphore/work-stealing buys ~+38% over+sequential **in raw mode** (in jpeg mode the encoding bottleneck masks it to ~+8% — an+earlier jpeg-only comparison was misleading). So `fast_cdp` is ~2x the simple inline path+at batch scale and is **kept**. Absolute t/s here is optimistic (capture-only, short+window, 128-core box) vs the ~91-113 production figure; the *ratios* are the point.++### Design direction+Ship **one simple backend** (`websocket.py`, inline JPEG) for the CLI/skill/`pixelrag index`+— that scale doesn't need the raw+decoupled machinery, and the flagship index uses the+separate system anyway. Keep `fast_cdp` + `strategies/` as batch/research code. The shared+capture-readiness logic (load wait, scroll) should eventually live in one place so the+shipped backend can't silently drift from the correct pattern again.diff --git a/plugin/skills/pixelbrowse/SKILL.md b/plugin/skills/pixelbrowse/SKILL.mdindex be08335..4462baf 100644--- a/plugin/skills/pixelbrowse/SKILL.md+++ b/plugin/skills/pixelbrowse/SKILL.md@@ -17,13 +17,13 @@ Use `pixelshot` to capture any URL or document as tiled JPEG images, then read t```bash# Screenshot a URL (optimized for Claude's vision: 1568px tile height)-pixelshot <url> --output /tmp/pixelbrowse --tile-height 1568+pixelshot <url> --output /tmp/pixelbrowse --tile-height 1568 --wait-network-idle# Screenshot multiple URLs in parallel-pixelshot <url1> <url2> --output /tmp/pixelbrowse --tile-height 1568 --workers 4+pixelshot <url1> <url2> --output /tmp/pixelbrowse --tile-height 1568 --wait-network-idle --workers 4# Wider viewport for desktop layouts-pixelshot <url> --output /tmp/pixelbrowse --tile-height 1568 --viewport-width 1280+pixelshot <url> --output /tmp/pixelbrowse --tile-height 1568 --viewport-width 1280 --wait-network-idle# Render a PDFpixelshot document.pdf --output /tmp/pixelbrowse@@ -33,11 +33,16 @@ IMPORTANT: Always use `--tile-height 1568` for screenshots you will read visuallClaude's vision model downscales images with long edge > 1568px (Sonnet/Haiku) or 2576px (Opus).The default 8192px tile height will be downscaled and text becomes unreadable.+IMPORTANT: Always use `--wait-network-idle` for URLs. Without it, JavaScript-heavy+pages (most modern sites / single-page apps) are captured before they finish rendering+and come back blank or half-empty. It waits for the page's load event plus a brief+network-quiet window so client-rendered content is actually on screen.+After rendering, read the tile images from the output directory to visually understand the content.## Workflow-1. Run `pixelshot <url> --output /tmp/pixelbrowse`+1. Run `pixelshot <url> --output /tmp/pixelbrowse --tile-height 1568 --wait-network-idle`2. Read `/tmp/pixelbrowse/<domain>.png.tiles/tile_0000.jpg` directly (no need to ls — the naming is deterministic)3. If the page is long, also read tile_0001.jpg, tile_0002.jpg, etc.diff --git a/render/src/pixelrag_render/backends/cdp.py b/render/src/pixelrag_render/backends/cdp.pynew file mode 100644index 0000000..9fd513f--- /dev/null+++ b/render/src/pixelrag_render/backends/cdp.py@@ -0,0 +1,563 @@+"""The CDP backend for pixelshot — the single rendering backend.++No Playwright dependency — launches Chrome via subprocess and talks CDP over a+raw websocket. Two capture paths, selected by the Chrome binary:++- STANDARD (default, portable): standard ``Page.captureScreenshot`` (JPEG over CDP).+ Works on any stock Chrome, any OS. Used unless a turbo-capable Chrome is present.+- TURBO: delegates to ``fast_cdp`` (rawFilePath + /dev/shm + parallel JPEG), ~2x at+ batch scale. Used automatically when the pixelrag-installed patched ``headless_shell``+ is selected (``chrome.is_turbo_capable``) and the request matches its capabilities.++Selection is deterministic (by Chrome provenance), with no runtime probe — so a stock+Chrome is never sent the patched-only CDP params (which would hang).++Requirements: websockets, pillow (no playwright needed)++Usage:+ from pixelrag_render.backends.cdp import render_urls+ tile_dirs = render_urls(["https://example.com"], "./tiles", workers=4)+"""++import asyncio+import base64+import io+import json+import logging+import signal+import subprocess+import time+import urllib.request+from pathlib import Path++from PIL import Image++logger = logging.getLogger("pixelrag_render.backends.cdp")++VIEWPORT_W = 875+VIEWPORT_H = 1080++BROWSER_ARGS = [+ "--disable-dev-shm-usage",+ "--no-sandbox",+ "--disable-renderer-backgrounding",+ "--disable-backgrounding-occluded-windows",+ "--disable-background-networking",+ "--disable-features=Translate,MediaRouter,OptimizationHints",+ "--enable-gpu-rasterization",+ "--force-gpu-rasterization",+]+++def _find_chrome() -> str:+ from ..chrome import find_chrome++ return find_chrome()+++async def _connect_cdp(port: int, retries: int = 5, delay: float = 1.0):+ """Connect to Chrome's CDP websocket endpoint."""+ import websockets++ for attempt in range(retries):+ try:+ data = urllib.request.urlopen(+ f"http://localhost:{port}/json", timeout=3+ ).read()+ targets = json.loads(data)+ ws = await websockets.connect(+ targets[0]["webSocketDebuggerUrl"],+ open_timeout=10,+ max_size=50 * 1024 * 1024,+ )+ return ws+ except Exception:+ if attempt < retries - 1:+ await asyncio.sleep(delay)+ raise ConnectionError(f"Failed to connect to Chrome on port {port}")+++async def _cdp_send(ws, msg_id_ref: list, method: str, params: dict | None = None):+ """Send a CDP command and wait for its response."""+ msg_id_ref[0] += 1+ mid = msg_id_ref[0]+ msg = {"id": mid, "method": method}+ if params:+ msg["params"] = params+ await ws.send(json.dumps(msg))+ while True:+ r = json.loads(await asyncio.wait_for(ws.recv(), timeout=180))+ if r.get("id") == mid:+ if "error" in r:+ raise RuntimeError(f"CDP error: {r['error']}")+ return r.get("result", {})+++# Max time to wait for the `load` event (and for the optional network-idle wait)+# before giving up and capturing whatever is there. Keeps a hanging page from+# stalling a worker.+LOAD_TIMEOUT_MS = 12_000+# Network is considered idle once no new resource has been fetched for this long.+NET_QUIET_MS = 500+++def _readiness_expr(wait_network_idle: bool) -> str:+ """Build the in-page readiness probe.++ Always waits for the `load` event before measuring (with a+ ``readyState === 'complete'`` shortcut so an already-loaded page returns+ immediately, and a hard timeout so a hanging page can't block). Without this,+ a client-rendered (SPA) page is measured/captured mid-hydration at a transient+ layout — often much taller than the settled page — producing blank tiles. SSR+ pages (e.g. Wikipedia) fire `load` almost immediately, so this adds ~no cost.++ When ``wait_network_idle`` is set, also waits (after load) until no new+ resource has been fetched for ``NET_QUIET_MS`` — for SPAs that fetch their+ content *after* load. This costs a quiet window per page, so it is opt-in+ (the pixelbrowse skill / single-page renders), not the batch default.++ Returns an async-IIFE expression resolving to the page height to tile.+ """+ idle_step = ""+ if wait_network_idle:+ idle_step = f"""+ await new Promise(res => {{+ let timer;+ let obs;+ const finish = () => {{ try {{ obs && obs.disconnect(); }} catch (e) {{}}+ clearTimeout(timer); clearTimeout(hard); res(); }};+ const bump = () => {{ clearTimeout(timer); timer = setTimeout(finish, {NET_QUIET_MS}); }};+ try {{+ obs = new PerformanceObserver(bump);+ obs.observe({{ type: 'resource', buffered: true }});+ }} catch (e) {{}}+ const hard = setTimeout(finish, {LOAD_TIMEOUT_MS});+ bump();+ }});"""+ return f"""(async () => {{+ await new Promise(res => {{+ if (document.readyState === 'complete') return res();+ const t = setTimeout(res, {LOAD_TIMEOUT_MS});+ window.addEventListener('load', () => {{ clearTimeout(t); res(); }}, {{ once: true }});+ }});{idle_step}+ await document.fonts.ready;+ await new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)));+ document.documentElement.style.scrollBehavior = 'auto';+ const sh = document.documentElement.scrollHeight;+ const body = document.body;+ if (body) {{+ const bottom = Math.ceil(body.getBoundingClientRect().bottom);+ return Math.min(sh, Math.max(bottom, 1));+ }}+ return sh;+ }})()"""+++# Before capturing a tile below the first one, scroll it into view and wait for+# its now-visible images to load. The capture clip uses absolute page coordinates,+# but Chrome only rasterizes content near the viewport — without scrolling, tiles+# past the first (e.g. on a small tile_height) come back blank. Mirrors fast_cdp.+_SCROLL_WAIT = """new Promise(resolve => {{+ window.scrollTo(0, {y});+ requestAnimationFrame(() => requestAnimationFrame(() => {{+ const imgs = Array.from(document.images).filter(i => {{+ if (i.complete) return false;+ const r = i.getBoundingClientRect();+ return r.bottom > 0 && r.top < window.innerHeight;+ }});+ if (imgs.length === 0) return resolve();+ const timeout = new Promise(r => setTimeout(r, 500));+ const loaded = Promise.all(imgs.map(i => new Promise(r => {{+ i.addEventListener('load', r, {{once: true}});+ i.addEventListener('error', r, {{once: true}});+ }})));+ Promise.race([loaded, timeout]).then(resolve);+ }}));+}})"""+++async def capture_url(+ ws,+ msg_id_ref: list,+ url: str,+ tile_dir: Path,+ *,+ tile_h: int = 8192,+ quality: int = 85,+ viewport_w: int = VIEWPORT_W,+ image_format: str = "jpeg",+ from_surface: bool = True,+ wait_network_idle: bool = False,+) -> int:+ """Capture a URL as tiled images via direct CDP websocket.++ Returns the number of tiles written.+ """+ tile_dir.mkdir(parents=True, exist_ok=True)++ await _cdp_send(ws, msg_id_ref, "Page.navigate", {"url": url})++ # Wait for load (+ optional network-idle) + fonts + layout to stabilize,+ # return the page height to tile in one call. See _readiness_expr.+ result = await _cdp_send(+ ws,+ msg_id_ref,+ "Runtime.evaluate",+ {+ "expression": _readiness_expr(wait_network_idle),+ "awaitPromise": True,+ "returnByValue": True,+ },+ )+ try:+ page_height = result["result"]["value"]+ except (KeyError, TypeError):+ page_height = tile_h++ tiles = []+ y = 0+ idx = 0++ while y < page_height:+ clip_h = min(tile_h, page_height - y)+ if clip_h <= 0:+ break++ # Scroll the tile into view so Chrome rasterizes it (tiles past the first+ # are otherwise blank). The top tile is already in view after load.+ if idx > 0:+ try:+ await _cdp_send(+ ws,+ msg_id_ref,+ "Runtime.evaluate",+ {"expression": _SCROLL_WAIT.format(y=y), "awaitPromise": True},+ )+ except Exception:+ pass++ params = {+ "format": image_format,+ "fromSurface": from_surface,+ "optimizeForSpeed": True,+ "clip": {+ "x": 0,+ "y": y,+ "width": viewport_w,+ "height": clip_h,+ "scale": 1,+ },+ }+ if image_format == "jpeg":+ params["quality"] = quality++ result = await _cdp_send(ws, msg_id_ref, "Page.captureScreenshot", params)++ img_bytes = base64.b64decode(result["data"])+ tile_path = (+ tile_dir / f"tile_{idx:04d}.{'jpg' if image_format == 'jpeg' else 'png'}"+ )++ if clip_h < tile_h:+ img = Image.open(io.BytesIO(img_bytes))+ w, h = img.size+ if h > clip_h:+ img = img.crop((0, 0, w, clip_h))+ img.save(+ tile_path, "JPEG" if image_format == "jpeg" else "PNG", quality=quality+ )+ else:+ tile_path.write_bytes(img_bytes)++ tiles.append(tile_path.name)+ idx += 1+ y += tile_h++ manifest = {+ "url": url,+ "page_height": page_height,+ "tiles": tiles,+ "complete": True,+ }+ with open(tile_dir / "tiles.json", "w") as f:+ json.dump(manifest, f)++ return len(tiles)+++async def _worker(+ chrome_path: str,+ port: int,+ work_queue: asyncio.Queue,+ output_dir: Path,+ tile_height: int,+ quality: int,+ viewport_w: int,+ image_format: str,+ from_surface: bool,+ wait_network_idle: bool,+ worker_id: int,+ stats: dict,+ results: list,+):+ """Async worker: owns a Chrome process, pulls URLs from queue."""+ proc = subprocess.Popen(+ # `--headless=new`: the bare `--headless` is deprecated and hangs on modern+ # Chrome (e.g. google-chrome 149); `=new` works on both stock Chrome and the+ # patched headless_shell.+ [chrome_path, f"--remote-debugging-port={port}", "--headless=new"]+ + BROWSER_ARGS+ + ["about:blank"],+ stdout=subprocess.DEVNULL,+ stderr=subprocess.DEVNULL,+ )++ try:+ await asyncio.sleep(3)+ ws = await _connect_cdp(port)+ msg_id_ref = [0]++ await _cdp_send(ws, msg_id_ref, "Page.enable")+ if wait_network_idle:+ # PerformanceObserver (used by the idle wait) needs no CDP domain, but+ # enabling Network keeps resource timing reliable across navigations.+ await _cdp_send(ws, msg_id_ref, "Network.enable")+ await _cdp_send(+ ws,+ msg_id_ref,+ "Emulation.setDeviceMetricsOverride",+ {+ "width": viewport_w,+ "height": tile_height,+ "deviceScaleFactor": 1,+ "mobile": False,+ },+ )++ while True:+ try:+ item = work_queue.get_nowait()+ except asyncio.QueueEmpty:+ break++ url = item["url"]+ stem = item["stem"]+ tile_dir = output_dir / f"{stem}.png.tiles"++ t0 = time.monotonic()+ try:+ n_tiles = await capture_url(+ ws,+ msg_id_ref,+ url,+ tile_dir,+ tile_h=tile_height,+ quality=quality,+ viewport_w=viewport_w,+ image_format=image_format,+ from_surface=from_surface,+ wait_network_idle=wait_network_idle,+ )+ stats["done"] += 1+ elapsed = time.monotonic() - t0+ logger.info(+ "[w%d] %s → %d tiles (%.1fs)", worker_id, url, n_tiles, elapsed+ )+ results.append(tile_dir)+ except Exception as e:+ stats["failed"] += 1+ logger.warning("[w%d] FAIL %s: %s", worker_id, url, str(e)[:200])++ await ws.close()+ finally:+ proc.send_signal(signal.SIGTERM)+ try:+ proc.wait(timeout=5)+ except subprocess.TimeoutExpired:+ proc.kill()+++def _derive_stems(urls: list[str], stems: list[str] | None) -> list[str]:+ """Output-dir stem per URL (explicit stems win; else sanitize the URL).++ Shared by the standard and turbo paths so both emit identical+ ``{stem}.png.tiles`` directory names for the same inputs.+ """+ from urllib.parse import urlparse++ out: list[str] = []+ seen: dict[str, int] = {}+ for i, url in enumerate(urls):+ if stems and i < len(stems):+ out.append(str(stems[i]))+ continue+ parsed = urlparse(url)+ raw = (parsed.netloc + parsed.path).rstrip("/")+ stem = (+ raw.replace("/", "_").replace(":", "_").replace("?", "_").replace("&", "_")+ )+ stem = stem[:200] or "page"+ count = seen.get(stem, 0)+ seen[stem] = count + 1+ if count > 0:+ stem = f"{stem}_{count}"+ out.append(stem)+ return out+++async def _run_batch(+ urls: list[str],+ output_dir: Path,+ num_workers: int,+ tile_height: int,+ quality: int,+ viewport_w: int,+ image_format: str,+ from_surface: bool,+ wait_network_idle: bool,+ stems: list[str] | None,+ chrome_path: str,+) -> list[Path]:+ work_queue: asyncio.Queue = asyncio.Queue()+ stem_list = _derive_stems(urls, stems)+ for url, stem in zip(urls, stem_list):+ work_queue.put_nowait({"url": url, "stem": stem})++ stats = {"done": 0, "failed": 0}+ results: list[Path] = []+ base_port = 9400++ actual_workers = min(num_workers, len(urls))+ workers = [+ _worker(+ chrome_path,+ base_port + wid,+ work_queue,+ output_dir,+ tile_height,+ quality,+ viewport_w,+ image_format,+ from_surface,+ wait_network_idle,+ wid,+ stats,+ results,+ )+ for wid in range(actual_workers)+ ]+ await asyncio.gather(*workers, return_exceptions=True)++ logger.info("Batch complete: done=%d failed=%d", stats["done"], stats["failed"])+ return results+++def render_urls(+ urls: list[str],+ output_dir: str | Path,+ *,+ stems: list[str] | None = None,+ tile_height: int = 8192,+ quality: int = 85,+ viewport_width: int = VIEWPORT_W,+ workers: int = 4,+ image_format: str = "jpeg",+ from_surface: bool = True,+ wait_network_idle: bool = False,+ turbo: bool | None = None,+ chrome_path: str | None = None,+) -> list[Path]:+ """Render URLs to tiled images via CDP.++ Uses the TURBO path (fast_cdp: rawFilePath + parallel JPEG) when a turbo-capable+ patched Chrome is present and the request matches its capture profile; otherwise+ the portable STANDARD path. Both emit ``{stem}.png.tiles/`` with a tiles.json.++ Args:+ urls: URLs to capture.+ output_dir: Output directory for tile subdirectories.+ stems: Optional output directory name per URL.+ tile_height: Max tile height in pixels (default 8192).+ quality: JPEG quality 1-100 (default 85).+ viewport_width: Browser viewport width (default 875).+ workers: Number of parallel Chrome processes (default 4).+ image_format: 'jpeg' or 'png' (default 'jpeg').+ from_surface: CDP fromSurface param. True for batch (throughput),+ False for serve (low latency). Default True.+ wait_network_idle: After the load event, also wait until the network has+ been quiet (~500ms) before capturing (SPAs that fetch after+ load). Standard path only; off by default.+ turbo: None = auto (turbo when the Chrome is turbo-capable), True/False to+ force. Turbo only applies to the default capture profile+ (jpeg, default viewport, fromSurface, no network-idle wait);+ other options always use the standard path.+ chrome_path: Path to Chrome binary. Auto-detected if None.++ Returns:+ List of Path objects for created tile directories.+ """+ output_dir = Path(output_dir)+ output_dir.mkdir(parents=True, exist_ok=True)++ if not urls:+ return []++ chrome = chrome_path or _find_chrome()++ # Turbo only covers fast_cdp's capture profile; anything else → standard path.+ from ..chrome import is_turbo_capable++ use_turbo = is_turbo_capable(chrome) if turbo is None else turbo+ if use_turbo and (+ image_format != "jpeg"+ or viewport_width != VIEWPORT_W+ or wait_network_idle+ or not from_surface+ ):+ use_turbo = False++ if use_turbo:+ from .fast_cdp import render_articles++ stem_list = _derive_stems(urls, stems)++ # path "{stem}.png" makes fast_cdp emit "{stem}.png.tiles" — the same+ # layout the standard path / CLI / index pipeline expect. fast_cdp prepends+ # file:// to non-http inputs, so hand it a plain path for file:// URIs.+ def _navtarget(u: str) -> str:+ if u.startswith("http"):+ return u+ return u[len("file://") :] if u.startswith("file://") else u++ articles = [+ {"path": f"{stem}.png", "file": _navtarget(url)}+ for stem, url in zip(stem_list, urls)+ ]+ logger.info("Using turbo (fast_cdp) path for %d URL(s)", len(urls))+ asyncio.run(+ render_articles(+ articles,+ str(output_dir),+ chrome_path=chrome,+ n_workers=workers,+ tile_height=tile_height,+ jpeg_quality=quality,+ )+ )+ return [output_dir / f"{stem}.png.tiles" for stem in stem_list]++ return asyncio.run(+ _run_batch(+ urls,+ output_dir,+ workers,+ tile_height,+ quality,+ viewport_width,+ image_format,+ from_surface,+ wait_network_idle,+ stems,+ chrome,+ )+ )diff --git a/render/src/pixelrag_render/backends/fast_cdp.py b/render/src/pixelrag_render/backends/fast_cdp.pyindex 790df0c..bf4d38d 100644--- a/render/src/pixelrag_render/backends/fast_cdp.py+++ b/render/src/pixelrag_render/backends/fast_cdp.py@@ -119,7 +119,8 @@ async def _launch_chrome(chrome_path: str, port: int) -> tuple:import websocketsargs = (- [chrome_path, f"--remote-debugging-port={port}", "--headless"]+ # `--headless=new`: bare `--headless` is deprecated and hangs on modern Chrome.+ [chrome_path, f"--remote-debugging-port={port}", "--headless=new"]+ CHROME_ARGS+ ["about:blank"])diff --git a/render/src/pixelrag_render/backends/websocket.py b/render/src/pixelrag_render/backends/websocket.pydeleted file mode 100644index 849f272..0000000--- a/render/src/pixelrag_render/backends/websocket.py+++ /dev/null@@ -1,399 +0,0 @@-"""Direct websocket CDP backend for pixelshot.--No Playwright dependency — uses subprocess to launch Chrome and websockets-to communicate via CDP directly. ~35% faster than the Playwright-based cdp.py-backend due to eliminating the Node.js IPC layer.--Requirements: websockets, pillow (no playwright needed)--Usage:- from pixelrag_render.backends.websocket import render_urls- tile_dirs = render_urls(["https://example.com"], "./tiles", workers=4)-"""--import asyncio-import base64-import io-import json-import logging-import signal-import subprocess-import time-import urllib.request-from pathlib import Path--from PIL import Image--logger = logging.getLogger("pixelrag_render.backends.websocket")--VIEWPORT_W = 875-VIEWPORT_H = 1080--BROWSER_ARGS = [- "--disable-dev-shm-usage",- "--no-sandbox",- "--disable-renderer-backgrounding",- "--disable-backgrounding-occluded-windows",- "--disable-background-networking",- "--disable-features=Translate,MediaRouter,OptimizationHints",- "--enable-gpu-rasterization",- "--force-gpu-rasterization",-]---def _find_chrome() -> str:- from ..chrome import find_chrome-- return find_chrome()---async def _connect_cdp(port: int, retries: int = 5, delay: float = 1.0):- """Connect to Chrome's CDP websocket endpoint."""- import websockets-- for attempt in range(retries):- try:- data = urllib.request.urlopen(- f"http://localhost:{port}/json", timeout=3- ).read()- targets = json.loads(data)- ws = await websockets.connect(- targets[0]["webSocketDebuggerUrl"],- open_timeout=10,- max_size=50 * 1024 * 1024,- )- return ws- except Exception:- if attempt < retries - 1:- await asyncio.sleep(delay)- raise ConnectionError(f"Failed to connect to Chrome on port {port}")---async def _cdp_send(ws, msg_id_ref: list, method: str, params: dict | None = None):- """Send a CDP command and wait for its response."""- msg_id_ref[0] += 1- mid = msg_id_ref[0]- msg = {"id": mid, "method": method}- if params:- msg["params"] = params- await ws.send(json.dumps(msg))- while True:- r = json.loads(await asyncio.wait_for(ws.recv(), timeout=180))- if r.get("id") == mid:- if "error" in r:- raise RuntimeError(f"CDP error: {r['error']}")- return r.get("result", {})---async def capture_url(- ws,- msg_id_ref: list,- url: str,- tile_dir: Path,- *,- tile_h: int = 8192,- quality: int = 85,- viewport_w: int = VIEWPORT_W,- image_format: str = "jpeg",- from_surface: bool = True,-) -> int:- """Capture a URL as tiled images via direct CDP websocket.-- Returns the number of tiles written.- """- tile_dir.mkdir(parents=True, exist_ok=True)-- await _cdp_send(ws, msg_id_ref, "Page.navigate", {"url": url})-- # Wait for fonts + layout to stabilize, return scrollHeight in one call- result = await _cdp_send(- ws,- msg_id_ref,- "Runtime.evaluate",- {- "expression": """new Promise(resolve => {- document.fonts.ready.then(() => {- requestAnimationFrame(() => {- requestAnimationFrame(() => {- document.documentElement.style.scrollBehavior = 'auto';- const sh = document.documentElement.scrollHeight;- const body = document.body;- if (body) {- const bottom = Math.ceil(body.getBoundingClientRect().bottom);- resolve(Math.min(sh, Math.max(bottom, 1)));- } else {- resolve(sh);- }- });- });- });- })""",- "awaitPromise": True,- "returnByValue": True,- },- )- try:- page_height = result["result"]["value"]- except (KeyError, TypeError):- page_height = tile_h-- tiles = []- y = 0- idx = 0-- while y < page_height:- clip_h = min(tile_h, page_height - y)- if clip_h <= 0:- break-- params = {- "format": image_format,- "fromSurface": from_surface,- "optimizeForSpeed": True,- "clip": {- "x": 0,- "y": y,- "width": viewport_w,- "height": clip_h,- "scale": 1,- },- }- if image_format == "jpeg":- params["quality"] = quality-- result = await _cdp_send(ws, msg_id_ref, "Page.captureScreenshot", params)-- img_bytes = base64.b64decode(result["data"])- tile_path = (- tile_dir / f"tile_{idx:04d}.{'jpg' if image_format == 'jpeg' else 'png'}"- )-- if clip_h < tile_h:- img = Image.open(io.BytesIO(img_bytes))- w, h = img.size- if h > clip_h:- img = img.crop((0, 0, w, clip_h))- img.save(- tile_path, "JPEG" if image_format == "jpeg" else "PNG", quality=quality- )- else:- tile_path.write_bytes(img_bytes)-- tiles.append(tile_path.name)- idx += 1- y += tile_h-- manifest = {- "url": url,- "page_height": page_height,- "tiles": tiles,- "complete": True,- }- with open(tile_dir / "tiles.json", "w") as f:- json.dump(manifest, f)-- return len(tiles)---async def _worker(- chrome_path: str,- port: int,- work_queue: asyncio.Queue,- output_dir: Path,- tile_height: int,- quality: int,- viewport_w: int,- image_format: str,- from_surface: bool,- worker_id: int,- stats: dict,- results: list,-):- """Async worker: owns a Chrome process, pulls URLs from queue."""- proc = subprocess.Popen(- [chrome_path, f"--remote-debugging-port={port}", "--headless"]- + BROWSER_ARGS- + ["about:blank"],- stdout=subprocess.DEVNULL,- stderr=subprocess.DEVNULL,- )-- try:- await asyncio.sleep(3)- ws = await _connect_cdp(port)- msg_id_ref = [0]-- await _cdp_send(ws, msg_id_ref, "Page.enable")- await _cdp_send(- ws,- msg_id_ref,- "Emulation.setDeviceMetricsOverride",- {- "width": viewport_w,- "height": tile_height,- "deviceScaleFactor": 1,- "mobile": False,- },- )-- while True:- try:- item = work_queue.get_nowait()- except asyncio.QueueEmpty:- break-- url = item["url"]- stem = item["stem"]- tile_dir = output_dir / f"{stem}.png.tiles"-- t0 = time.monotonic()- try:- n_tiles = await capture_url(- ws,- msg_id_ref,- url,- tile_dir,- tile_h=tile_height,- quality=quality,- viewport_w=viewport_w,- image_format=image_format,- from_surface=from_surface,- )- stats["done"] += 1- elapsed = time.monotonic() - t0- logger.info(- "[w%d] %s → %d tiles (%.1fs)", worker_id, url, n_tiles, elapsed- )- results.append(tile_dir)- except Exception as e:- stats["failed"] += 1- logger.warning("[w%d] FAIL %s: %s", worker_id, url, str(e)[:200])-- await ws.close()- finally:- proc.send_signal(signal.SIGTERM)- try:- proc.wait(timeout=5)- except subprocess.TimeoutExpired:- proc.kill()---async def _run_batch(- urls: list[str],- output_dir: Path,- num_workers: int,- tile_height: int,- quality: int,- viewport_w: int,- image_format: str,- from_surface: bool,- stems: list[str] | None,- chrome_path: str,-) -> list[Path]:- work_queue: asyncio.Queue = asyncio.Queue()- seen_stems: dict[str, int] = {}- for i, url in enumerate(urls):- if stems and i < len(stems):- stem = str(stems[i])- else:- from urllib.parse import urlparse-- parsed = urlparse(url)- raw = (parsed.netloc + parsed.path).rstrip("/")- stem = (- raw.replace("/", "_")- .replace(":", "_")- .replace("?", "_")- .replace("&", "_")- )- stem = stem[:200] or "page"- count = seen_stems.get(stem, 0)- seen_stems[stem] = count + 1- if count > 0:- stem = f"{stem}_{count}"- work_queue.put_nowait({"url": url, "stem": stem})-- stats = {"done": 0, "failed": 0}- results: list[Path] = []- base_port = 9400-- actual_workers = min(num_workers, len(urls))- workers = [- _worker(- chrome_path,- base_port + wid,- work_queue,- output_dir,- tile_height,- quality,- viewport_w,- image_format,- from_surface,- wid,- stats,- results,- )- for wid in range(actual_workers)- ]- await asyncio.gather(*workers, return_exceptions=True)-- logger.info("Batch complete: done=%d failed=%d", stats["done"], stats["failed"])- return results---def render_urls(- urls: list[str],- output_dir: str | Path,- *,- stems: list[str] | None = None,- tile_height: int = 8192,- quality: int = 85,- viewport_width: int = VIEWPORT_W,- workers: int = 4,- image_format: str = "jpeg",- from_surface: bool = True,- chrome_path: str | None = None,-) -> list[Path]:- """Render URLs to tiled images using direct CDP websocket.-- No Playwright dependency. Each worker launches its own Chrome process- and communicates via CDP over websocket.-- Args:- urls: URLs to capture.- output_dir: Output directory for tile subdirectories.- stems: Optional output directory name per URL.- tile_height: Max tile height in pixels (default 8192).- quality: JPEG quality 1-100 (default 85).- viewport_width: Browser viewport width (default 875).- workers: Number of parallel Chrome processes (default 4).- image_format: 'jpeg' or 'png' (default 'jpeg').- from_surface: CDP fromSurface param. True for batch (throughput),- False for serve (low latency). Default True.- chrome_path: Path to Chrome binary. Auto-detected if None.-- Returns:- List of Path objects for created tile directories.- """- output_dir = Path(output_dir)- output_dir.mkdir(parents=True, exist_ok=True)-- if not urls:- return []-- chrome = chrome_path or _find_chrome()-- return asyncio.run(- _run_batch(- urls,- output_dir,- workers,- tile_height,- quality,- viewport_width,- image_format,- from_surface,- stems,- chrome,- )- )diff --git a/render/src/pixelrag_render/chrome.py b/render/src/pixelrag_render/chrome.pyindex bbeb1a1..18408a0 100644--- a/render/src/pixelrag_render/chrome.py+++ b/render/src/pixelrag_render/chrome.py@@ -89,6 +89,25 @@ def get_installed_version() -> str | None:return None+def is_turbo_capable(chrome_path: str) -> bool:+ """Whether ``chrome_path`` is the pixelrag-installed patched headless_shell,+ which supports the turbo capture extensions (rawFilePath/directClip/skipRedraw).++ Deterministic by provenance — the patched binary lives only at the install path+ (with its version.json marker). No runtime probe, so a stock Chrome is never+ mistaken for a turbo-capable one (and a turbo run is never tried on a binary+ that would hang on the unknown CDP params).+ """+ try:+ installed = (INSTALL_DIR / "headless_shell").resolve()+ return (+ Path(chrome_path).resolve() == installed+ and (INSTALL_DIR / VERSION_FILE).exists()+ )+ except Exception:+ return False++def install_chrome(version: str | None = None, force: bool = False) -> Path:"""Download and install the patched headless_shell binary.diff --git a/render/src/pixelrag_render/render.py b/render/src/pixelrag_render/render.pyindex a6d54f3..35a4753 100644--- a/render/src/pixelrag_render/render.py+++ b/render/src/pixelrag_render/render.py@@ -86,13 +86,13 @@ def render_urls(Returns:List of Path objects pointing to created tile directories."""- if backend in ("cdp", "websocket"):- from .backends.websocket import render_urls as _render_urls+ if backend in ("cdp", "websocket"): # "websocket" kept as a back-compat alias+ from .backends.cdp import render_urls as _render_urlselse:raise ValueError(- f"Unknown backend: {backend!r}. Choose 'cdp' or 'websocket'."- " For high-throughput batch rendering with custom Chrome,"- " use pixelrag_render.backends.fast_cdp.render_articles() directly."+ f"Unknown backend: {backend!r}. Choose 'cdp'."+ " The cdp backend auto-selects a turbo capture path when a turbo-capable"+ " Chrome is present.")return _render_urls(@@ -276,6 +276,14 @@ def main() -> None:default=875,help="Browser viewport width in pixels (default: 875).",)+ parser.add_argument(+ "--wait-network-idle",+ action="store_true",+ help="After the page's load event, also wait until the network is quiet "+ "(~500ms) before capturing. Helps JS/SPA pages that fetch content after "+ "load; adds a quiet window per page, so off by default. Recommended for "+ "single-page renders (e.g. the pixelbrowse skill).",+ )parser.add_argument("--dpi",type=int,@@ -313,6 +321,7 @@ def main() -> None:quality=args.quality,viewport_width=args.viewport_width,workers=args.workers,+ wait_network_idle=args.wait_network_idle,)results.extend(tile_dirs)@@ -334,6 +343,7 @@ def main() -> None:quality=args.quality,viewport_width=args.viewport_width,workers=1,+ wait_network_idle=args.wait_network_idle,)elif suffix in {".png", ".jpg", ".jpeg", ".webp"}:tile_dirs = render_file(fpath, output_dir)