The agent that turns websites into APIs!
Make Playwright an optional [manual] extra to slim the base install (#113)
kalil032149612a06f388
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0cca7f5..8936a5b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Changed+- **Playwright is now an optional `[manual]` extra**: agent mode (the default) captures through `npx`-launched browser tooling (browser MCP servers or the `agent-browser` CLI) and never imports the Python Playwright package, so `playwright` and `playwright-stealth` moved out of the base dependencies into a `[manual]` optional-dependency group. This keeps the base install (and any runtime that bundles the package) lightweight. Manual capture mode now requires `pip install "reverse-api-engineer[manual]"` followed by `playwright install chromium`. Entering manual mode without the extra fails fast — before any run is recorded — with an actionable hint listing both steps, so a base-only install never leaves a phantom `manual` run in history.+## [0.11.0] - 2026-07-22### Changeddiff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 066ca72..3c4db77 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -12,8 +12,8 @@ Thank you for your interest in contributing! This document provides guidelines a2. **Install dependencies**```bash- uv sync- playwright install chromium+ uv sync --all-extras+ uv run playwright install chromium # for manual-mode capture (Playwright ships in the [manual] extra)```3. **Run locally**diff --git a/README.md b/README.mdindex 1f19a33..b93de15 100644--- a/README.md+++ b/README.md@@ -41,6 +41,15 @@ No more manually opening DevTools, copying cURL commands, and gluing together a```bashuv tool install reverse-api-engineer # or: pip install reverse-api-engineer+```++Agent mode (the default) captures through `npx`-launched browser tooling —+browser MCP servers (Playwright or Chrome DevTools) or the Vercel `agent-browser`+CLI — and needs no extra Python packages. **Manual mode** drives a local+Playwright browser, which ships in the optional `[manual]` extra:++```bash+uv tool install "reverse-api-engineer[manual]" # or: pip install "reverse-api-engineer[manual]"playwright install chromium```@@ -177,7 +186,7 @@ uv syncuv run reverse-api-engineer```-Build: `./scripts/clean_build.sh`. Requires Python 3.11+, Playwright browsers, and an API key for agent mode.+Build: `./scripts/clean_build.sh`. Requires Python 3.11+ and an API key for agent mode. Manual mode additionally needs the `[manual]` extra plus `playwright install chromium`.## Licensediff --git a/pyproject.toml b/pyproject.tomlindex 675157f..46b5039 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -44,8 +44,6 @@ classifiers = ["Operating System :: OS Independent",]dependencies = [- "playwright>=1.40.0",- "playwright-stealth>=1.0.0","click>=8.1.0","claude-agent-sdk>=0.1.48","rich>=13.0.0",@@ -59,6 +57,13 @@ dependencies = [][project.optional-dependencies]+# Manual capture mode drives a local Playwright browser. Agent mode (the+# default) uses external MCP servers over npx and needs none of this, so the+# heavy browser stack is opt-in — keeping the base install lightweight.+manual = [+ "playwright>=1.40.0",+ "playwright-stealth>=1.0.0",+]collector = ["markdownify>=0.12.0","beautifulsoup4>=4.12.0",diff --git a/src/reverse_api/browser.py b/src/reverse_api/browser.pyindex 8986147..e094e8b 100644--- a/src/reverse_api/browser.py+++ b/src/reverse_api/browser.py@@ -226,6 +226,37 @@ class ManualBrowser:"""Inject stealth scripts into page before any other scripts run."""page.add_init_script(STEALTH_JS)+ @staticmethod+ def _normalize_url(url: str) -> str:+ """Prepend https:// when the user typed a bare host (e.g.+ ``jobs.example.com/x``); Playwright's goto rejects scheme-less URLs."""+ url = url.strip()+ if url and "://" not in url:+ return "https://" + url+ return url++ def _abort_playwright(self) -> None:+ """Quietly tear down the browser/Playwright after a failed start.++ Unlike close(), this saves nothing and prints nothing — its only job is+ to stop the sync-Playwright event loop so it doesn't leak into the+ caller. A leaked loop makes the caller's next asyncio.run() (the REPL+ prompt) fail with "asyncio.run() cannot be called from a running event+ loop". Safe to call with partially-initialized state.+ """+ for teardown in (+ lambda: self._context and self._context.close(),+ lambda: self._browser and not self._using_persistent and self._browser.close(),+ lambda: self._playwright and self._playwright.stop(),+ ):+ try:+ teardown()+ except Exception:+ pass+ self._context = None+ self._browser = None+ self._playwright = None+def _start_with_real_chrome(self, start_url: str | None = None) -> Path:"""Start using the real Chrome browser with user's profile."""import shutil@@ -393,14 +424,24 @@ class ManualBrowser:console.print(" [dim]close browser or ctrl+c to finalize[/dim]")console.print()+ if start_url:+ start_url = self._normalize_url(start_url)+self._playwright = sync_playwright().start()# Try real Chrome first (better for avoiding detection)# Fall back to stealth Chromium if Chrome not available- if self.use_real_chrome:- return self._start_with_real_chrome(start_url)- else:- return self._start_with_stealth_chromium(start_url)+ try:+ if self.use_real_chrome:+ return self._start_with_real_chrome(start_url)+ else:+ return self._start_with_stealth_chromium(start_url)+ except Exception:+ # Startup failed (e.g. an invalid URL) before close() could stop+ # Playwright. Tear it down here so its sync event loop doesn't leak+ # and break the caller's next asyncio.run() (the REPL prompt).+ self._abort_playwright()+ raisedef close(self) -> Path:"""Close the browser and save HAR file. Returns HAR path."""diff --git a/src/reverse_api/cli.py b/src/reverse_api/cli.pyindex a54bfe8..a756e35 100644--- a/src/reverse_api/cli.py+++ b/src/reverse_api/cli.py@@ -17,9 +17,9 @@ from prompt_toolkit.key_binding import KeyBindingsfrom prompt_toolkit.styles import Style as PtStylefrom questionary import Choicefrom rich.console import Console+from rich.markup import escapefrom . import __version__-from .browser import ManualBrowserfrom .config import DEFAULT_OPENCODE_MODEL, DEFAULT_OPENCODE_PROVIDER, ConfigManagerfrom .engineer import run_reverse_engineeringfrom .messages import MessageStore@@ -70,7 +70,7 @@ def _select_ollama_model_for_settings() -> str | None:try:status = asyncio.run(ensure_ollama_models())except OllamaSetupError as e:- console.print(f" [yellow]error:[/yellow] {e}\n")+ console.print(f" [yellow]error:[/yellow] {escape(str(e))}\n")return Nonemodels = status.compatible_models@@ -154,7 +154,7 @@ def _select_opencode_pair_for_settings(mode_color=THEME_PRIMARY) -> tuple[str, s"Set OPENCODE_SERVER_PASSWORD (and OPENCODE_SERVER_USERNAME when customized).\n")else:- console.print(f" [yellow]error:[/yellow] could not load OpenCode providers and models: {e}\n")+ console.print(f" [yellow]error:[/yellow] could not load OpenCode providers and models: {escape(str(e))}\n")return Noneproviders = [provider for provider in catalog.get("providers", []) if isinstance(provider, dict)]@@ -1149,7 +1149,7 @@ def repl_loop():console.print("\n [dim]terminated[/dim]")returnexcept Exception as e:- console.print(f" [red]error:[/red] {e}")+ console.print(f" [red]error:[/red] {escape(str(e))}")console.print(f" [dim]{ERROR_CTA}[/dim]")@@ -1632,7 +1632,7 @@ def handle_messages(run_id: str, mode_color=THEME_PRIMARY):display += "..."console.print(f" [dim]{timestamp} .. {display}[/dim]")elif msg_type == "error":- console.print(f" [dim]{timestamp}[/dim] [red]error: {content}[/red]")+ console.print(f" [dim]{timestamp}[/dim] [red]error: {escape(str(content))}[/red]")elif msg_type == "result":console.print(f" [dim]{timestamp}[/dim] [white]complete[/white]")if isinstance(content, dict):@@ -1842,6 +1842,20 @@ def run_manual_capture(prompt=None, url=None, reverse_engineer=True, model=None,"""Shared logic for manual capture."""output_dir = output_dir or config_manager.get("output_dir")+ # Manual mode drives a local Playwright browser, which ships in the optional+ # [manual] extra. Fail fast — before prompting the user or recording a run —+ # so a base-only install never leaves a phantom "manual" run in history.+ try:+ from .browser import ManualBrowser+ except ImportError as exc:+ raise click.ClickException(+ "Manual capture mode requires the Playwright browser stack, which is "+ "not installed by default. Install it with:\n\n"+ " pip install 'reverse-api-engineer[manual]'\n"+ " playwright install chromium\n\n"+ "(Agent mode does not need this.)"+ ) from exc+if prompt is None:options = prompt_interactive_options(prompt=prompt,@@ -1978,7 +1992,7 @@ def run_collector(prompt=None, model=None, output_dir=None):)return resultexcept Exception as e:- console.print(f" [red]collector error: {e}[/red]")+ console.print(f" [red]collector error: {escape(str(e))}[/red]")console.print(f" [dim]{ERROR_CTA}[/dim]")import traceback@@ -2184,7 +2198,7 @@ def run_auto_capture(}except Exception as e:- console.print(f" [red]auto mode error: {e}[/red]")+ console.print(f" [red]auto mode error: {escape(str(e))}[/red]")console.print(f" [dim]{ERROR_CTA}[/dim]")import tracebackdiff --git a/tests/test_cli_manual_missing_extra.py b/tests/test_cli_manual_missing_extra.pynew file mode 100644index 0000000..0383082--- /dev/null+++ b/tests/test_cli_manual_missing_extra.py@@ -0,0 +1,60 @@+"""Manual mode must fail fast when the optional ``[manual]`` extra is absent.++Playwright moved out of the base install into the ``[manual]`` extra. On a+base-only install, entering manual mode must raise an actionable install hint+*before* recording anything — otherwise it leaves a phantom ``manual`` run in+history that later "operate on the latest run" flows could target.+"""++import io+import sys+from unittest.mock import MagicMock, patch++import click+import pytest+from rich.console import Console+from rich.markup import escape++from reverse_api.cli import run_manual_capture+++class TestManualModeMissingExtra:+ def _run(self):+ # `sys.modules[name] = None` makes `from .browser import ...` raise+ # ImportError, simulating a base-only install without Playwright.+ with patch.dict(sys.modules, {"reverse_api.browser": None}), patch(+ "reverse_api.cli.session_manager"+ ) as mock_session:+ with pytest.raises(click.ClickException) as excinfo:+ run_manual_capture(+ prompt="fetch things",+ url="http://example.com",+ reverse_engineer=False,+ output_dir="/tmp/rae-test-does-not-matter",+ )+ return excinfo.value, mock_session++ def test_raises_actionable_hint(self):+ err, _ = self._run()+ message = err.format_message()+ assert "reverse-api-engineer[manual]" in message+ # The pip command alone is insufficient — Chromium must also be fetched.+ assert "playwright install chromium" in message++ def test_records_no_run(self):+ _, mock_session = self._run()+ mock_session.add_run.assert_not_called()++ def test_extra_name_survives_rich_rendering(self):+ """The CLI prints errors through Rich, which parses `[manual]` as a+ markup tag and would drop it — leaving `pip install+ 'reverse-api-engineer'` (wrong). The render path must escape the message+ so the extra name reaches the user intact.+ """+ err, _ = self._run()+ buf = io.StringIO()+ # Mirror the REPL error handler: exception text into a markup string.+ Console(file=buf, force_terminal=False, no_color=True).print(+ f" [red]error:[/red] {escape(str(err))}"+ )+ assert "reverse-api-engineer[manual]" in buf.getvalue()diff --git a/tests/test_manual_browser_startup_failure.py b/tests/test_manual_browser_startup_failure.pynew file mode 100644index 0000000..be267d5--- /dev/null+++ b/tests/test_manual_browser_startup_failure.py@@ -0,0 +1,41 @@+"""A failed manual-capture start must not leak Playwright's event loop.++ManualBrowser.start() calls sync_playwright().start() and only stops it inside+close() on the happy path. If startup fails first (e.g. a bad URL makes+page.goto raise), Playwright's sync API leaves an asyncio loop running; the+REPL's next prompt then dies forever with "asyncio.run() cannot be called from+a running event loop". start() must stop Playwright before propagating.+"""++from unittest.mock import MagicMock, patch++import pytest++pytest.importorskip("playwright") # ships in the [manual] extra only++from reverse_api.browser import ManualBrowser # noqa: E402+++def _browser(tmp_path):+ return ManualBrowser(run_id="testrun", prompt="grab jobs", output_dir=str(tmp_path))+++class TestStartupFailureCleanup:+ def test_playwright_stopped_when_start_raises(self, tmp_path):+ browser = _browser(tmp_path)+ fake_pw = MagicMock()++ with patch("reverse_api.browser.sync_playwright") as sp, patch.object(+ browser, "_start_with_real_chrome", side_effect=RuntimeError("bad URL")+ ):+ sp.return_value.start.return_value = fake_pw+ with pytest.raises(RuntimeError, match="bad URL"):+ browser.start(start_url="https://example.com")++ fake_pw.stop.assert_called_once()+ assert browser._playwright is None++ def test_scheme_added_to_bare_host(self):+ assert ManualBrowser._normalize_url("jobs.example.com/x") == "https://jobs.example.com/x"+ assert ManualBrowser._normalize_url("http://a.com") == "http://a.com"+ assert ManualBrowser._normalize_url(" https://b.com ") == "https://b.com"diff --git a/uv.lock b/uv.lockindex e0faf81..1eba036 100644--- a/uv.lock+++ b/uv.lock@@ -1558,15 +1558,13 @@ wheels = [[[package]]name = "reverse-api-engineer"-version = "0.10.0"+version = "0.11.0"source = { editable = "." }dependencies = [{ name = "aiohttp" },{ name = "claude-agent-sdk" },{ name = "click" },{ name = "cryptography" },- { name = "playwright" },- { name = "playwright-stealth" },{ name = "pygments" },{ name = "questionary" },{ name = "requests" },@@ -1583,6 +1581,10 @@ collector = [copilot = [{ name = "github-copilot-sdk" },]+manual = [+ { name = "playwright" },+ { name = "playwright-stealth" },+][package.dev-dependencies]dev = [@@ -1602,8 +1604,8 @@ requires-dist = [{ name = "cryptography", specifier = ">=46.0.7" },{ name = "github-copilot-sdk", marker = "extra == 'copilot'", specifier = ">=0.1.0" },{ name = "markdownify", marker = "extra == 'collector'", specifier = ">=0.12.0" },- { name = "playwright", specifier = ">=1.40.0" },- { name = "playwright-stealth", specifier = ">=1.0.0" },+ { name = "playwright", marker = "extra == 'manual'", specifier = ">=1.40.0" },+ { name = "playwright-stealth", marker = "extra == 'manual'", specifier = ">=1.0.0" },{ name = "pygments", specifier = ">=2.20.0" },{ name = "questionary", specifier = ">=2.0.0" },{ name = "requests", specifier = ">=2.33.0" },@@ -1611,7 +1613,7 @@ requires-dist = [{ name = "setproctitle", specifier = ">=1.3.0" },{ name = "watchdog", specifier = ">=3.0.0" },]-provides-extras = ["collector", "copilot"]+provides-extras = ["manual", "collector", "copilot"][package.metadata.requires-dev]dev = [diff --git a/website/content/docs/index.mdx b/website/content/docs/index.mdxindex 68b653d..383836d 100644--- a/website/content/docs/index.mdx+++ b/website/content/docs/index.mdx@@ -60,9 +60,12 @@ and `run` can stream NDJSON progress with `--json-stream`, and `list` plus### Browser automation with stealth-Built on **Playwright** with anti-detection patches and viewport-randomisation. Captures every request as HAR: headers, cookies, request-bodies, response bodies, timing.+Manual mode drives a local **Playwright** browser (optional `[manual]` extra)+with anti-detection patches and viewport randomisation, recording every request+as a HAR file — headers, cookies, request bodies, response bodies, timing. Agent+mode captures the same detail autonomously (see below); the Playwright MCP and+`agent-browser` providers also produce a HAR, while Chrome DevTools MCP reads+network traffic directly through its MCP tools.### Autonomous agent loopdiff --git a/website/content/docs/installation.mdx b/website/content/docs/installation.mdxindex 0e017d6..4194818 100644--- a/website/content/docs/installation.mdx+++ b/website/content/docs/installation.mdx@@ -1,6 +1,6 @@---title: Installation-description: Install Reverse API Engineer with uv or pip, then set up Playwright browsers.+description: Install Reverse API Engineer with uv or pip; add the [manual] extra for manual-mode capture.---<Callout type="info" title="What you'll need">@@ -28,13 +28,28 @@ description: Install Reverse API Engineer with uv or pip, then set up Playwright</Tab></Tabs>-## Install Playwright browsers+## Manual mode: install the browser stack-This downloads the Chromium build that powers manual and agent capture:+Agent mode (the default) drives capture through `npx`-launched browser tooling —+browser MCP servers (Playwright or Chrome DevTools) or the Vercel `agent-browser`+CLI — and needs no extra Python packages. **Manual mode** drives a local+Playwright browser, which ships in the optional `[manual]` extra. Install the+extra, then download the Chromium build:-```bash-playwright install chromium-```+<Tabs items={['uv (recommended)', 'pip']}>+ <Tab value="uv (recommended)">+ ```bash+ uv tool install "reverse-api-engineer[manual]"+ playwright install chromium+ ```+ </Tab>+ <Tab value="pip">+ ```bash+ pip install "reverse-api-engineer[manual]"+ playwright install chromium+ ```+ </Tab>+</Tabs>## Verify the installdiff --git a/website/content/docs/modes/manual.mdx b/website/content/docs/modes/manual.mdxindex 84a037a..d1c68c6 100644--- a/website/content/docs/modes/manual.mdx+++ b/website/content/docs/modes/manual.mdx@@ -11,6 +11,16 @@ Manual mode runs the full pipeline with you in the driver's seat.caption="Drive the browser yourself, get a client when you're done."/>+<Callout type="info" title="Requires the browser extra">+Manual mode drives a local Playwright browser, which ships in the optional+`[manual]` extra (agent mode doesn't need it). Install it once:++```bash+uv tool install "reverse-api-engineer[manual]" # or: pip install "reverse-api-engineer[manual]"+playwright install chromium+```+</Callout>+## How it works<Steps>diff --git a/website/content/docs/quick-start.mdx b/website/content/docs/quick-start.mdxindex dcc387f..01ee80c 100644--- a/website/content/docs/quick-start.mdx+++ b/website/content/docs/quick-start.mdx@@ -23,6 +23,13 @@ description: From a fresh install to a working API client in under five minutes.<Step>### Run your first manual capture+ <Callout type="info">+ Manual mode needs the Playwright browser stack from the optional `[manual]`+ extra: `uv tool install "reverse-api-engineer[manual]"` (or add `[manual]`+ to your `pip install`) followed by `playwright install chromium`. Agent+ mode needs neither.+ </Callout>+Press **`Shift+Tab`** once to switch from agent mode to manual mode, thentype a goal: what you want from the site.diff --git a/website/src/app/llms.txt/route.ts b/website/src/app/llms.txt/route.tsindex 1f23619..9f4860e 100644--- a/website/src/app/llms.txt/route.ts+++ b/website/src/app/llms.txt/route.ts@@ -15,7 +15,7 @@ export function GET() {> ${appTagline}-${appName} is an open-source CLI that captures browser traffic with Playwright (or Chrome DevTools MCP) and uses your configured AI SDK to generate a typed API client from the recorded HAR. Output languages: Python, JavaScript, TypeScript, Go, Java, C#, PHP, Ruby, and C.+${appName} is an open-source CLI that captures browser traffic — in the default agent mode via a browser MCP server (Playwright or Chrome DevTools) or the Vercel agent-browser CLI, or in manual mode via a local Playwright browser (optional \`[manual]\` extra) — and uses your configured AI SDK to generate a typed API client from the captured requests. Output languages: Python, JavaScript, TypeScript, Go, Java, C#, PHP, Ruby, and C.## Documentation