Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

9.4 KB
Raw Blame

Site Scrape

pinchtab scrape turns a whole site into a page tree of markdown content. Pages are discovered and extracted over plain HTTP first (via SeaPortal: sitemap or link crawl, URL-pattern sampling). Only pages whose HTTP extraction comes back thin, blocked, or failed are re-rendered in the real browser and re-extracted from the rendered DOM — so JavaScript-only content still lands in the report, while static pages never pay for a browser.

The pipeline:

seaportal (HTTP crawl & extraction)  →  routing  →  pinchtab (browser)  →  report
   sitemap/link crawl, URL-pattern       thin /        render + re-extract   json / md
   sampling, per-page markdown           blocked?       JS-only pages         page tree

SeaPortal does the cheap HTTP work; the browser — the expensive resource — runs only where HTTP was not enough. Every page records where its content came from (http or browser) and the routing verdict that put it there.

Quick start

pinchtab scrape https://example.com --output-dir ./scrape
# → ./scrape/report.json

Also write a human-readable digest and print a summary:

pinchtab scrape https://example.com --format md --output-dir ./scrape
# → ./scrape/report.json + ./scrape/report.md

Preview → Expand (large sites)

Scraping a large site at full fidelity is expensive twice over: shipping every page's markdown dominates the token cost, and browser-rendering every routed page dominates the wall-clock cost. To survey first and drill down second, run the scrape in two cheap-then-deep passes.

1. Preview — the HTTP crawl and per-page routing verdict, but no browser rendering and no full page bodies. Each page's markdown is withheld and replaced by a charCount (how heavy a full expand would be) and a leading snippet:

pinchtab scrape https://example.com --preview
#   https://example.com/            · 4229 chars
#       Browser control for AI agents. 12MB Go binary…
#   https://example.com/docs/app    · 87 chars · needs browser: thin-content
#       Loading…

2. Expand — scrape exactly the URLs you picked, at full fidelity (HTTP extract + browser render per the normal routing), instead of re-crawling:

pinchtab scrape https://example.com \
  --only https://example.com/docs/app \
  --only https://example.com/pricing \
  --output-dir ./scrape

Expand is stateless: hand back the exact URLs the preview showed. No server-side session or cache is kept.

pinchtab scrape

pinchtab scrape <url> [flags]

The argument is the site URL. Discovery seeds from that host's root: SeaPortal looks for robots.txt and sitemap.xml at the host root and, failing that, link-crawls from the root page.

Flag Default Meaning
--preview false Outline only: page tree, charCount, and snippet per page — no browser rendering or full bodies
--only <url> Expand exactly these URLs at full fidelity instead of crawling (repeatable)
--max-pages <n> 50 Maximum pages sampled across the whole site
--max-per-pattern <n> 8 Maximum pages sampled per URL-pattern group (e.g. /blog/*)
--include <regex> Only crawl URLs matching this regex (repeatable)
--exclude <regex> Skip URLs matching this regex (repeatable)
--enrich-all false Browser-render every reachable page, ignoring routing
--no-browser false HTTP crawl only; record routing verdicts without browser rendering
--concurrency <n> 2 Pages browser-rendered in parallel (max 8)
--timeout <s> 60 Overall HTTP crawl timeout in seconds
--output-dir <dir> Write report.json (and report.md with --format md) to this directory
--format <f> json Report format: json or md
--json false Print the full report JSON to stdout
--cookie name=value Inject a cookie into an isolated temporary browser instance before the run (repeatable)
--cookies-file <file> Inject cookies into an isolated temporary browser instance from a JSON array of {name, value, domain, ...} objects
--profile <name> Run against the instance of this browser profile (cannot be combined with --cookie or --cookies-file)

Failure contract: a page that fails in both engines does not fail the run — the command exits 0 and that page's report entry carries an error field. A failed browser enrichment keeps the HTTP extraction and records browserError. The run itself only errors when the crawl fails outright or discovers no pages.

Routing

Each HTTP-extracted page is scored for whether the browser should re-render it:

  • not-found (404 / 410) never routes — re-rendering cannot recover content.
  • fetch-error or a blocked status (401, 403, 407, 429, 503) routes: a real browser with stealth and challenge handling may succeed where a plain HTTP client was refused.
  • thin-content (extraction shorter than the static-ok threshold) routes: a probable JavaScript shell.

--enrich-all forces every page except not-found (404 / 410) to the browser; --no-browser records the verdict on each page but renders nothing.

Report anatomy

report.json is a versioned scrape report (schemaVersion):

schemaVersion, generatedAt, input
site:      baseUrl, title, sitemapFound, totalURLsInSitemap, sampledPages
pageGroups[]:  pattern, total, sampled, urls[]        # site tree by URL pattern
pages[]:
  url, title, statusCode, contentType
  markdown                     # withheld in preview mode
  charCount, snippet           # set in preview mode instead of markdown
  meta, schema, internalLinks, externalLinks
  source                       # "http" or "browser"
  browserRecommended, browserReasons   # the routing verdict
  browserError?                # browser enrichment failed; HTTP content kept
  error?                       # page failed in both engines
summary:   contentTypes, httpPages, browserPages, failedPages, recommendations

--format md writes report.md next to report.json (or prints to stdout without --output-dir): a single digest with the site overview, the page tree by URL pattern, and each page's content (or its snippet in preview mode).

Summary counters

httpPages, browserPages and failedPages partition the pages: every page falls in exactly one, and the three sum to the page total.

  • failedPages counts the pages that did not return usable content — a page that failed in both engines (a transport error) or one whose statusCode is >= 400 (a 4xx/5xx response). A 3xx redirect and a page that never carried a status (statusCode 0/unset) are not failures and stay in httpPages/browserPages. A non-2xx page the browser then rendered into content (source: "browser") is not counted as failed either — the recovery cleared the failure, though the page keeps its original statusCode. When failedPages is non-zero, the recommendations include "N of M pages returned errors or 4xx/5xx responses", so broken links surface in the summary rather than being counted as successful pages.
  • contentTypes is the taxonomy of the successful pages only: a non-2xx page is excluded, so an error body is never reported as an ordinary page. The page's own statusCode and markdown are left untouched.

Schema history

schemaVersion is stamped into every report so a consumer can detect format changes. The current version is 2.0.

1.0 → 2.0 (breaking). site.totalDiscovered was removed and replaced by site.totalURLsInSitemap.

The number itself did not change — the name did. totalDiscovered only ever counted URLs listed in the site's sitemap, never everything the crawl found, so on a site with no sitemap it read 0 while pages were plainly being scraped (N page(s) discovered of 0). It was renamed to state what it holds.

Two things to change in a 1.0 reader:

  • Read totalURLsInSitemap where you read totalDiscovered; the value is the same measurement.
  • It is now omitempty, so it is absent rather than 0 when the site has no sitemap. Pair it with sitemapFound rather than treating a missing key as zero.

If what you actually wanted was how many pages the crawl reached, that is site.sampledPages (or the pages[] length) — it never was totalDiscovered.

HTTP API

The CLI is a thin client over one endpoint:

  • POST /scrape {"url", "preview", "only", "maxPages", "maxPerPattern", "includePatterns", "excludePatterns", "concurrency", "enrichAll", "noBrowser", "timeoutSeconds", "browser"} → the scrape report

Every crawl fetch and every expand fetch goes through the same SSRF/redirect navigation guard as browser navigation.

MCP

Agents can drive the same flow over MCP with the pinchtab_scrape tool. Prefer preview=true for a cheap outline, then expand with only (comma-separated URLs) — the full report can be large. See MCP Server.

pinchtab_scrape { "url": "https://example.com", "preview": true }
pinchtab_scrape { "url": "https://example.com", "only": "https://example.com/a, https://example.com/b" }

Docker / CI

The scrape CLI is a thin client over a running pinchtab server, so in Docker run the server container first and exec the scrape against it (see guides/docker.md for the server setup):

docker run -d --name pinchtab -v pinchtab-data:/data --shm-size=2g pinchtab/pinchtab
docker exec pinchtab pinchtab scrape https://example.com --output-dir /data/scrape
docker cp pinchtab:/data/scrape ./scrape