High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
remove docs/ — research notes no longer needed
botf1bc1424643c
diff --git a/docs/RESEARCH.md b/docs/RESEARCH.mddeleted file mode 100644index e94da03b..00000000--- a/docs/RESEARCH.md+++ /dev/null@@ -1,111 +0,0 @@-# Browser Bridge — Interface Research--## The Question: Why Screenshots?--Screenshots are the worst way for an AI agent to interact with a browser. Here's why, and what's better.--## Three Approaches Compared--### 1. Screenshots (Vision-based)-**How it works:** Render page → PNG → base64 → send to vision model → model interprets pixels--**Problems:**-- **Expensive:** A single screenshot = ~1,000-2,000 tokens (image tokens). A 10-step task burns 10-20k tokens just on screenshots-- **Slow:** Render → encode → transfer → vision model inference. Each step adds latency-- **Unreliable:** Vision models hallucinate button positions, misread text, can't see off-screen content-- **No element refs:** Model says "click the blue button at (342, 187)" — coordinates drift between renders-- **Requires vision model:** Not all LLMs have vision. Limits which models can use the bridge--**When it's actually useful:**-- Visual verification (does this page *look* right?)-- CAPTCHA solving-- Canvas/WebGL content (no DOM equivalent)-- Debugging — human wants to see what the agent sees--### 2. Accessibility Tree Snapshots (What we already have)-**How it works:** CDP `Accessibility.getFullAXTree()` → structured tree → flat JSON with refs--**Advantages:**-- **~4x cheaper than screenshots** (Playwright team benchmarks: ~27k vs ~114k tokens for same task via MCP)-- **Fast:** Pure data, no rendering/encoding-- **Deterministic refs:** `e0`, `e1`, `e2`... — stable element references the agent can click/type-- **Works with any LLM:** Text-only, no vision required-- **Semantic:** Role + name + value tells the agent what things *are*, not what they look like--**Limitations:**-- Doesn't capture visual layout (relative positioning, colors, spacing)-- Custom widgets with poor ARIA labeling become invisible-- Very large pages = very large trees (need pruning)--### 3. Playwright CLI Approach (Files on Disk) — NEW-**How it works:** Snapshots/screenshots saved to disk files, agent reads only what it needs--**Key insight from Playwright CLI (Microsoft, Feb 2026):**-- MCP returns full snapshot inline → bloats context window-- CLI saves to `.playwright-cli/snapshot.yaml` → agent reads if needed-- **4x token reduction** vs MCP for same tasks (27k vs 114k tokens)-- Screenshots never enter context unless agent explicitly reads the file--**Implication for Browser Bridge:**-- Add `/snapshot?format=file` — save to disk, return path only-- Add `/screenshot?save=true` — save PNG to disk, return path-- Agent decides whether to read the file or just act on previous knowledge--## Hybrid Strategy (Recommended)--The best approach combines methods based on the task:--| Task | Best Method |-|------|------------|-| Navigate + interact | Accessibility snapshot (default) |-| Verify visual appearance | Screenshot (on demand) |-| Fill forms, click buttons | Snapshot refs (`e0`, `e1`) |-| Read page content | Snapshot + `/evaluate` for innerText |-| Debug / human review | Screenshot saved to disk |--### Cost Comparison (Estimated per 10-step task)--| Method | Tokens | Cost (Claude) | Latency |-|--------|--------|---------------|---------|-| Screenshot every step | ~20,000 | ~$0.06 | ~500ms/step |-| A11y snapshot every step (inline) | ~5,000 | ~$0.015 | ~100ms/step |-| A11y snapshot (file, read 2 of 10) | ~1,500 | ~$0.005 | ~80ms/step |-| Hybrid (snapshot + 1 screenshot) | ~7,000 | ~$0.02 | ~150ms avg |--## What the Industry Is Doing--### Playwright MCP (Microsoft)-- **Default:** Accessibility snapshots, no screenshots-- "Bypasses the need for screenshots or visually-tuned models"-- Flat YAML format with element refs-- Vision mode available but opt-in--### Browser Use (open source, 8k+ stars)-- Uses accessibility tree + optional screenshots-- Their benchmark: 100 hard tasks, best models hit ~60% success-- Cost: $10-100 per 100-task evaluation run depending on model-- Key finding: structured data >> pixel interpretation for reliability--### OpenClaw (what we use)-- Already does this! `browser` tool uses accessibility snapshots by default-- `refs="aria"` for stable cross-call references-- Screenshots available but secondary--### Computer Use (Anthropic/OpenAI)-- Full screenshot-based approach (most expensive)-- Works for anything but costs ~10x more tokens-- Designed for general computer control, not browser-specific tasks--## Key Takeaways--1. **Accessibility tree is the primary interface** — already implemented in our `/snapshot`-2. **Screenshots should be opt-in, not default** — keep `/screenshot` but don't use it as the main interaction loop-3. **File-based output reduces tokens dramatically** — add save-to-disk option for both snapshots and screenshots-4. **Ref resolution is the hardest part** — our current `resolveRef()` uses heuristics. Need to improve this-5. **Pruning large trees** — add depth/filter params to `/snapshot` to keep token count low--## Sources-- Playwright MCP: github.com/microsoft/playwright-mcp-- Playwright CLI benchmarks: testcollab.com/blog/playwright-cli (Feb 2026)-- Browser Use benchmark: browser-use.com/posts/ai-browser-agent-benchmark-- "Building Browser Agents" paper: arxiv.org/abs/2511.19477 (Vardanyan, 2025)diff --git a/docs/STEEL_ANALYSIS.md b/docs/STEEL_ANALYSIS.mddeleted file mode 100644index ee94f667..00000000--- a/docs/STEEL_ANALYSIS.md+++ /dev/null@@ -1,120 +0,0 @@-# Steel Browser Analysis--**Repo:** github.com/steel-dev/steel-browser-**Codebase:** ~12,000 lines TypeScript (API only), plus React UI-**Stack:** Fastify + Puppeteer-core + LevelDB + DuckDB + React--## What Steel Actually Does--Steel is a **scraping/extraction** service that happens to use a browser. It is NOT an agent interaction layer.--### Their 4 Endpoints (that's it)-1. `POST /scrape` — Navigate to URL, extract HTML/markdown/readability/cleaned_html-2. `POST /screenshot` — Navigate to URL, return JPEG screenshot-3. `POST /pdf` — Navigate to URL, return PDF-4. `POST /search` — Scrape Brave Search results for a query--### What They DON'T Have-- ❌ **No accessibility tree / snapshot** — Zero references to `accessibility`, `snapshot`, `a11y`, or `AXTree` in the entire codebase-- ❌ **No element interaction** — No click, type, fill, hover, select-- ❌ **No action API** — Can't interact with pages at all-- ❌ **No JS evaluation** — Can't run arbitrary JavaScript-- ❌ **No tab management** — Single primary page, not multi-tab--### What They DO Have (that we don't)-- ✅ Plugin system (BasePlugin with lifecycle hooks)-- ✅ Session management (isolated browser contexts, cookies, localStorage persistence via LevelDB)-- ✅ Proxy chain support (per-request proxy rotation)-- ✅ Anti-detection / stealth (fingerprinting, user-agent management)-- ✅ Selenium WebDriver compatibility layer-- ✅ File upload/download management-- ✅ Request logging with DuckDB storage-- ✅ PDF-to-HTML conversion-- ✅ HTML-to-Markdown (Turndown + Readability/Defuddle)-- ✅ React UI for session monitoring-- ✅ OpenAPI spec (Scalar docs)-- ✅ Docker packaging--## Architecture Issues--### Overengineered for what it does-- 12,000 lines of TypeScript for 4 endpoints-- Plugin system with 7 lifecycle hooks — but ships with no plugins-- DuckDB + LevelDB + in-memory storage abstractions — for request logs-- Selenium compatibility layer — adds complexity for legacy support-- WebSocket proxy infrastructure — for live viewing, not agent control--### Missing the core problem-Steel solves "how do I scrape web pages reliably" — which is a solved problem (Firecrawl, Crawlee, etc). It does NOT solve "how does an AI agent interact with a web page" — which is the actual hard problem.--**No accessibility tree = no agent interaction.** An agent can't fill a form, click a button, or navigate a SPA with Steel. It can only fetch content.--### Cloud-first mindset baked in-- Session management designed for multi-tenant cloud (session IDs, isolation)-- Proxy support for bypassing rate limits at scale-- Anti-detection for scraping hostile sites-- All of this is irrelevant for local agent use--## What Browser Bridge Should Be (First Principles)--### The actual problem-An AI agent needs to:-1. **See** what's on a page (cheaply, structured)-2. **Act** on elements (click, type, select)-3. **Navigate** between pages-4. **Extract** specific data-5. **Verify** results--### Minimum viable surface-```-GET /tabs # What's open-GET /snapshot?tabId=X # Accessibility tree (primary interface)-POST /action {kind, ref, text} # Click/type/fill by a11y ref-POST /navigate {url} # Go somewhere-POST /evaluate {expression} # Run JS (escape hatch)-GET /screenshot?tabId=X # Visual verification (opt-in)-GET /text?tabId=X # Readable text extraction-```--That's 7 endpoints. Not 4 wrong ones + 30 supporting files.--### What we should steal from Steel-1. **HTML-to-Markdown extraction** — Useful for content reading. But as a `/text` endpoint, not the primary interface-2. **Docker packaging** — Single `docker run` to start-3. **OpenAPI spec** — Good for any HTTP client to auto-discover--### What we should NOT copy-1. Session management (overkill for local use)-2. Plugin system (YAGNI — add when needed)-3. Selenium compatibility (legacy baggage)-4. Proxy chains (not our problem)-5. Anti-detection (not our problem)-6. React UI (curl is the UI)--## Competitive Position--| Feature | Steel | Browser Bridge | Playwright MCP |-|---------|-------|---------------|----------------|-| A11y tree snapshot | ❌ | ✅ | ✅ |-| Element interaction | ❌ | ✅ | ✅ |-| Content extraction | ✅ | ⚡ TODO | ❌ |-| Screenshot | ✅ | ✅ | ✅ (opt-in) |-| JS evaluation | ❌ | ✅ | ✅ |-| Multi-tab | ❌ | ✅ | ❌ |-| Protocol | HTTP | HTTP | MCP (JSON-RPC) |-| Any agent can use | ✅ | ✅ | ❌ (MCP only) |-| Docker | ✅ | TODO | ❌ |-| Lines of code | ~12,000 | ~350 | ~5,000 |-| Zero config | ❌ | ✅ | ❌ |--**Browser Bridge is already more useful for AI agents than Steel, in 350 lines vs 12,000.**--Steel is a scraping tool marketed as a browser API. We're building an actual agent-browser interface.--## Conclusion--Don't copy Steel. Don't compete with Steel. They solve a different problem (web scraping infrastructure) and solve it with 30x more code than necessary.--Our angle: **the simplest possible bridge between any AI agent and a real browser.** Accessibility-first, HTTP API, zero config, works with curl.--The closest competitor is actually **Playwright MCP** — but that requires MCP protocol support. We're HTTP, which means anything can use us.