Skip to content

The agent that turns websites into APIs!

Make Playwright an optional [manual] extra to slim the base install (#113)

kalil032149612a06f388
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0cca7f5..8936a5b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Changed
+- **Playwright is now an optional `[manual]` extra**: agent mode (the default) captures through `npx`-launched browser tooling (browser MCP servers or the `agent-browser` CLI) and never imports the Python Playwright package, so `playwright` and `playwright-stealth` moved out of the base dependencies into a `[manual]` optional-dependency group. This keeps the base install (and any runtime that bundles the package) lightweight. Manual capture mode now requires `pip install "reverse-api-engineer[manual]"` followed by `playwright install chromium`. Entering manual mode without the extra fails fast — before any run is recorded — with an actionable hint listing both steps, so a base-only install never leaves a phantom `manual` run in history.
+
## [0.11.0] - 2026-07-22
### Changed
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 066ca72..3c4db77 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -12,8 +12,8 @@ Thank you for your interest in contributing! This document provides guidelines a
2. **Install dependencies**
```bash
- uv sync
- playwright install chromium
+ uv sync --all-extras
+ uv run playwright install chromium # for manual-mode capture (Playwright ships in the [manual] extra)
```
3. **Run locally**
diff --git a/README.md b/README.md
index 1f19a33..b93de15 100644
--- a/README.md
+++ b/README.md
@@ -41,6 +41,15 @@ No more manually opening DevTools, copying cURL commands, and gluing together a
```bash
uv tool install reverse-api-engineer # or: pip install reverse-api-engineer
+```
+
+Agent mode (the default) captures through `npx`-launched browser tooling —
+browser MCP servers (Playwright or Chrome DevTools) or the Vercel `agent-browser`
+CLI — and needs no extra Python packages. **Manual mode** drives a local
+Playwright browser, which ships in the optional `[manual]` extra:
+
+```bash
+uv tool install "reverse-api-engineer[manual]" # or: pip install "reverse-api-engineer[manual]"
playwright install chromium
```
@@ -177,7 +186,7 @@ uv sync
uv run reverse-api-engineer
```
-Build: `./scripts/clean_build.sh`. Requires Python 3.11+, Playwright browsers, and an API key for agent mode.
+Build: `./scripts/clean_build.sh`. Requires Python 3.11+ and an API key for agent mode. Manual mode additionally needs the `[manual]` extra plus `playwright install chromium`.
## License
diff --git a/pyproject.toml b/pyproject.toml
index 675157f..46b5039 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -44,8 +44,6 @@ classifiers = [
"Operating System :: OS Independent",
]
dependencies = [
- "playwright>=1.40.0",
- "playwright-stealth>=1.0.0",
"click>=8.1.0",
"claude-agent-sdk>=0.1.48",
"rich>=13.0.0",
@@ -59,6 +57,13 @@ dependencies = [
]
[project.optional-dependencies]
+# Manual capture mode drives a local Playwright browser. Agent mode (the
+# default) uses external MCP servers over npx and needs none of this, so the
+# heavy browser stack is opt-in — keeping the base install lightweight.
+manual = [
+ "playwright>=1.40.0",
+ "playwright-stealth>=1.0.0",
+]
collector = [
"markdownify>=0.12.0",
"beautifulsoup4>=4.12.0",
diff --git a/src/reverse_api/browser.py b/src/reverse_api/browser.py
index 8986147..e094e8b 100644
--- a/src/reverse_api/browser.py
+++ b/src/reverse_api/browser.py
@@ -226,6 +226,37 @@ class ManualBrowser:
"""Inject stealth scripts into page before any other scripts run."""
page.add_init_script(STEALTH_JS)
+ @staticmethod
+ def _normalize_url(url: str) -> str:
+ """Prepend https:// when the user typed a bare host (e.g.
+ ``jobs.example.com/x``); Playwright's goto rejects scheme-less URLs."""
+ url = url.strip()
+ if url and "://" not in url:
+ return "https://" + url
+ return url
+
+ def _abort_playwright(self) -> None:
+ """Quietly tear down the browser/Playwright after a failed start.
+
+ Unlike close(), this saves nothing and prints nothing — its only job is
+ to stop the sync-Playwright event loop so it doesn't leak into the
+ caller. A leaked loop makes the caller's next asyncio.run() (the REPL
+ prompt) fail with "asyncio.run() cannot be called from a running event
+ loop". Safe to call with partially-initialized state.
+ """
+ for teardown in (
+ lambda: self._context and self._context.close(),
+ lambda: self._browser and not self._using_persistent and self._browser.close(),
+ lambda: self._playwright and self._playwright.stop(),
+ ):
+ try:
+ teardown()
+ except Exception:
+ pass
+ self._context = None
+ self._browser = None
+ self._playwright = None
+
def _start_with_real_chrome(self, start_url: str | None = None) -> Path:
"""Start using the real Chrome browser with user's profile."""
import shutil
@@ -393,14 +424,24 @@ class ManualBrowser:
console.print(" [dim]close browser or ctrl+c to finalize[/dim]")
console.print()
+ if start_url:
+ start_url = self._normalize_url(start_url)
+
self._playwright = sync_playwright().start()
# Try real Chrome first (better for avoiding detection)
# Fall back to stealth Chromium if Chrome not available
- if self.use_real_chrome:
- return self._start_with_real_chrome(start_url)
- else:
- return self._start_with_stealth_chromium(start_url)
+ try:
+ if self.use_real_chrome:
+ return self._start_with_real_chrome(start_url)
+ else:
+ return self._start_with_stealth_chromium(start_url)
+ except Exception:
+ # Startup failed (e.g. an invalid URL) before close() could stop
+ # Playwright. Tear it down here so its sync event loop doesn't leak
+ # and break the caller's next asyncio.run() (the REPL prompt).
+ self._abort_playwright()
+ raise
def close(self) -> Path:
"""Close the browser and save HAR file. Returns HAR path."""
diff --git a/src/reverse_api/cli.py b/src/reverse_api/cli.py
index a54bfe8..a756e35 100644
--- a/src/reverse_api/cli.py
+++ b/src/reverse_api/cli.py
@@ -17,9 +17,9 @@ from prompt_toolkit.key_binding import KeyBindings
from prompt_toolkit.styles import Style as PtStyle
from questionary import Choice
from rich.console import Console
+from rich.markup import escape
from . import __version__
-from .browser import ManualBrowser
from .config import DEFAULT_OPENCODE_MODEL, DEFAULT_OPENCODE_PROVIDER, ConfigManager
from .engineer import run_reverse_engineering
from .messages import MessageStore
@@ -70,7 +70,7 @@ def _select_ollama_model_for_settings() -> str | None:
try:
status = asyncio.run(ensure_ollama_models())
except OllamaSetupError as e:
- console.print(f" [yellow]error:[/yellow] {e}\n")
+ console.print(f" [yellow]error:[/yellow] {escape(str(e))}\n")
return None
models = status.compatible_models
@@ -154,7 +154,7 @@ def _select_opencode_pair_for_settings(mode_color=THEME_PRIMARY) -> tuple[str, s
"Set OPENCODE_SERVER_PASSWORD (and OPENCODE_SERVER_USERNAME when customized).\n"
)
else:
- console.print(f" [yellow]error:[/yellow] could not load OpenCode providers and models: {e}\n")
+ console.print(f" [yellow]error:[/yellow] could not load OpenCode providers and models: {escape(str(e))}\n")
return None
providers = [provider for provider in catalog.get("providers", []) if isinstance(provider, dict)]
@@ -1149,7 +1149,7 @@ def repl_loop():
console.print("\n [dim]terminated[/dim]")
return
except Exception as e:
- console.print(f" [red]error:[/red] {e}")
+ console.print(f" [red]error:[/red] {escape(str(e))}")
console.print(f" [dim]{ERROR_CTA}[/dim]")
@@ -1632,7 +1632,7 @@ def handle_messages(run_id: str, mode_color=THEME_PRIMARY):
display += "..."
console.print(f" [dim]{timestamp} .. {display}[/dim]")
elif msg_type == "error":
- console.print(f" [dim]{timestamp}[/dim] [red]error: {content}[/red]")
+ console.print(f" [dim]{timestamp}[/dim] [red]error: {escape(str(content))}[/red]")
elif msg_type == "result":
console.print(f" [dim]{timestamp}[/dim] [white]complete[/white]")
if isinstance(content, dict):
@@ -1842,6 +1842,20 @@ def run_manual_capture(prompt=None, url=None, reverse_engineer=True, model=None,
"""Shared logic for manual capture."""
output_dir = output_dir or config_manager.get("output_dir")
+ # Manual mode drives a local Playwright browser, which ships in the optional
+ # [manual] extra. Fail fast — before prompting the user or recording a run —
+ # so a base-only install never leaves a phantom "manual" run in history.
+ try:
+ from .browser import ManualBrowser
+ except ImportError as exc:
+ raise click.ClickException(
+ "Manual capture mode requires the Playwright browser stack, which is "
+ "not installed by default. Install it with:\n\n"
+ " pip install 'reverse-api-engineer[manual]'\n"
+ " playwright install chromium\n\n"
+ "(Agent mode does not need this.)"
+ ) from exc
+
if prompt is None:
options = prompt_interactive_options(
prompt=prompt,
@@ -1978,7 +1992,7 @@ def run_collector(prompt=None, model=None, output_dir=None):
)
return result
except Exception as e:
- console.print(f" [red]collector error: {e}[/red]")
+ console.print(f" [red]collector error: {escape(str(e))}[/red]")
console.print(f" [dim]{ERROR_CTA}[/dim]")
import traceback
@@ -2184,7 +2198,7 @@ def run_auto_capture(
}
except Exception as e:
- console.print(f" [red]auto mode error: {e}[/red]")
+ console.print(f" [red]auto mode error: {escape(str(e))}[/red]")
console.print(f" [dim]{ERROR_CTA}[/dim]")
import traceback
diff --git a/tests/test_cli_manual_missing_extra.py b/tests/test_cli_manual_missing_extra.py
new file mode 100644
index 0000000..0383082
--- /dev/null
+++ b/tests/test_cli_manual_missing_extra.py
@@ -0,0 +1,60 @@
+"""Manual mode must fail fast when the optional ``[manual]`` extra is absent.
+
+Playwright moved out of the base install into the ``[manual]`` extra. On a
+base-only install, entering manual mode must raise an actionable install hint
+*before* recording anything — otherwise it leaves a phantom ``manual`` run in
+history that later "operate on the latest run" flows could target.
+"""
+
+import io
+import sys
+from unittest.mock import MagicMock, patch
+
+import click
+import pytest
+from rich.console import Console
+from rich.markup import escape
+
+from reverse_api.cli import run_manual_capture
+
+
+class TestManualModeMissingExtra:
+ def _run(self):
+ # `sys.modules[name] = None` makes `from .browser import ...` raise
+ # ImportError, simulating a base-only install without Playwright.
+ with patch.dict(sys.modules, {"reverse_api.browser": None}), patch(
+ "reverse_api.cli.session_manager"
+ ) as mock_session:
+ with pytest.raises(click.ClickException) as excinfo:
+ run_manual_capture(
+ prompt="fetch things",
+ url="http://example.com",
+ reverse_engineer=False,
+ output_dir="/tmp/rae-test-does-not-matter",
+ )
+ return excinfo.value, mock_session
+
+ def test_raises_actionable_hint(self):
+ err, _ = self._run()
+ message = err.format_message()
+ assert "reverse-api-engineer[manual]" in message
+ # The pip command alone is insufficient — Chromium must also be fetched.
+ assert "playwright install chromium" in message
+
+ def test_records_no_run(self):
+ _, mock_session = self._run()
+ mock_session.add_run.assert_not_called()
+
+ def test_extra_name_survives_rich_rendering(self):
+ """The CLI prints errors through Rich, which parses `[manual]` as a
+ markup tag and would drop it — leaving `pip install
+ 'reverse-api-engineer'` (wrong). The render path must escape the message
+ so the extra name reaches the user intact.
+ """
+ err, _ = self._run()
+ buf = io.StringIO()
+ # Mirror the REPL error handler: exception text into a markup string.
+ Console(file=buf, force_terminal=False, no_color=True).print(
+ f" [red]error:[/red] {escape(str(err))}"
+ )
+ assert "reverse-api-engineer[manual]" in buf.getvalue()
diff --git a/tests/test_manual_browser_startup_failure.py b/tests/test_manual_browser_startup_failure.py
new file mode 100644
index 0000000..be267d5
--- /dev/null
+++ b/tests/test_manual_browser_startup_failure.py
@@ -0,0 +1,41 @@
+"""A failed manual-capture start must not leak Playwright's event loop.
+
+ManualBrowser.start() calls sync_playwright().start() and only stops it inside
+close() on the happy path. If startup fails first (e.g. a bad URL makes
+page.goto raise), Playwright's sync API leaves an asyncio loop running; the
+REPL's next prompt then dies forever with "asyncio.run() cannot be called from
+a running event loop". start() must stop Playwright before propagating.
+"""
+
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+pytest.importorskip("playwright") # ships in the [manual] extra only
+
+from reverse_api.browser import ManualBrowser # noqa: E402
+
+
+def _browser(tmp_path):
+ return ManualBrowser(run_id="testrun", prompt="grab jobs", output_dir=str(tmp_path))
+
+
+class TestStartupFailureCleanup:
+ def test_playwright_stopped_when_start_raises(self, tmp_path):
+ browser = _browser(tmp_path)
+ fake_pw = MagicMock()
+
+ with patch("reverse_api.browser.sync_playwright") as sp, patch.object(
+ browser, "_start_with_real_chrome", side_effect=RuntimeError("bad URL")
+ ):
+ sp.return_value.start.return_value = fake_pw
+ with pytest.raises(RuntimeError, match="bad URL"):
+ browser.start(start_url="https://example.com")
+
+ fake_pw.stop.assert_called_once()
+ assert browser._playwright is None
+
+ def test_scheme_added_to_bare_host(self):
+ assert ManualBrowser._normalize_url("jobs.example.com/x") == "https://jobs.example.com/x"
+ assert ManualBrowser._normalize_url("http://a.com") == "http://a.com"
+ assert ManualBrowser._normalize_url(" https://b.com ") == "https://b.com"
diff --git a/uv.lock b/uv.lock
index e0faf81..1eba036 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1558,15 +1558,13 @@ wheels = [
[[package]]
name = "reverse-api-engineer"
-version = "0.10.0"
+version = "0.11.0"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },
{ name = "claude-agent-sdk" },
{ name = "click" },
{ name = "cryptography" },
- { name = "playwright" },
- { name = "playwright-stealth" },
{ name = "pygments" },
{ name = "questionary" },
{ name = "requests" },
@@ -1583,6 +1581,10 @@ collector = [
copilot = [
{ name = "github-copilot-sdk" },
]
+manual = [
+ { name = "playwright" },
+ { name = "playwright-stealth" },
+]
[package.dev-dependencies]
dev = [
@@ -1602,8 +1604,8 @@ requires-dist = [
{ name = "cryptography", specifier = ">=46.0.7" },
{ name = "github-copilot-sdk", marker = "extra == 'copilot'", specifier = ">=0.1.0" },
{ name = "markdownify", marker = "extra == 'collector'", specifier = ">=0.12.0" },
- { name = "playwright", specifier = ">=1.40.0" },
- { name = "playwright-stealth", specifier = ">=1.0.0" },
+ { name = "playwright", marker = "extra == 'manual'", specifier = ">=1.40.0" },
+ { name = "playwright-stealth", marker = "extra == 'manual'", specifier = ">=1.0.0" },
{ name = "pygments", specifier = ">=2.20.0" },
{ name = "questionary", specifier = ">=2.0.0" },
{ name = "requests", specifier = ">=2.33.0" },
@@ -1611,7 +1613,7 @@ requires-dist = [
{ name = "setproctitle", specifier = ">=1.3.0" },
{ name = "watchdog", specifier = ">=3.0.0" },
]
-provides-extras = ["collector", "copilot"]
+provides-extras = ["manual", "collector", "copilot"]
[package.metadata.requires-dev]
dev = [
diff --git a/website/content/docs/index.mdx b/website/content/docs/index.mdx
index 68b653d..383836d 100644
--- a/website/content/docs/index.mdx
+++ b/website/content/docs/index.mdx
@@ -60,9 +60,12 @@ and `run` can stream NDJSON progress with `--json-stream`, and `list` plus
### Browser automation with stealth
-Built on **Playwright** with anti-detection patches and viewport
-randomisation. Captures every request as HAR: headers, cookies, request
-bodies, response bodies, timing.
+Manual mode drives a local **Playwright** browser (optional `[manual]` extra)
+with anti-detection patches and viewport randomisation, recording every request
+as a HAR file — headers, cookies, request bodies, response bodies, timing. Agent
+mode captures the same detail autonomously (see below); the Playwright MCP and
+`agent-browser` providers also produce a HAR, while Chrome DevTools MCP reads
+network traffic directly through its MCP tools.
### Autonomous agent loop
diff --git a/website/content/docs/installation.mdx b/website/content/docs/installation.mdx
index 0e017d6..4194818 100644
--- a/website/content/docs/installation.mdx
+++ b/website/content/docs/installation.mdx
@@ -1,6 +1,6 @@
---
title: Installation
-description: Install Reverse API Engineer with uv or pip, then set up Playwright browsers.
+description: Install Reverse API Engineer with uv or pip; add the [manual] extra for manual-mode capture.
---
<Callout type="info" title="What you'll need">
@@ -28,13 +28,28 @@ description: Install Reverse API Engineer with uv or pip, then set up Playwright
</Tab>
</Tabs>
-## Install Playwright browsers
+## Manual mode: install the browser stack
-This downloads the Chromium build that powers manual and agent capture:
+Agent mode (the default) drives capture through `npx`-launched browser tooling —
+browser MCP servers (Playwright or Chrome DevTools) or the Vercel `agent-browser`
+CLI — and needs no extra Python packages. **Manual mode** drives a local
+Playwright browser, which ships in the optional `[manual]` extra. Install the
+extra, then download the Chromium build:
-```bash
-playwright install chromium
-```
+<Tabs items={['uv (recommended)', 'pip']}>
+ <Tab value="uv (recommended)">
+ ```bash
+ uv tool install "reverse-api-engineer[manual]"
+ playwright install chromium
+ ```
+ </Tab>
+ <Tab value="pip">
+ ```bash
+ pip install "reverse-api-engineer[manual]"
+ playwright install chromium
+ ```
+ </Tab>
+</Tabs>
## Verify the install
diff --git a/website/content/docs/modes/manual.mdx b/website/content/docs/modes/manual.mdx
index 84a037a..d1c68c6 100644
--- a/website/content/docs/modes/manual.mdx
+++ b/website/content/docs/modes/manual.mdx
@@ -11,6 +11,16 @@ Manual mode runs the full pipeline with you in the driver's seat.
caption="Drive the browser yourself, get a client when you're done."
/>
+<Callout type="info" title="Requires the browser extra">
+Manual mode drives a local Playwright browser, which ships in the optional
+`[manual]` extra (agent mode doesn't need it). Install it once:
+
+```bash
+uv tool install "reverse-api-engineer[manual]" # or: pip install "reverse-api-engineer[manual]"
+playwright install chromium
+```
+</Callout>
+
## How it works
<Steps>
diff --git a/website/content/docs/quick-start.mdx b/website/content/docs/quick-start.mdx
index dcc387f..01ee80c 100644
--- a/website/content/docs/quick-start.mdx
+++ b/website/content/docs/quick-start.mdx
@@ -23,6 +23,13 @@ description: From a fresh install to a working API client in under five minutes.
<Step>
### Run your first manual capture
+ <Callout type="info">
+ Manual mode needs the Playwright browser stack from the optional `[manual]`
+ extra: `uv tool install "reverse-api-engineer[manual]"` (or add `[manual]`
+ to your `pip install`) followed by `playwright install chromium`. Agent
+ mode needs neither.
+ </Callout>
+
Press **`Shift+Tab`** once to switch from agent mode to manual mode, then
type a goal: what you want from the site.
diff --git a/website/src/app/llms.txt/route.ts b/website/src/app/llms.txt/route.ts
index 1f23619..9f4860e 100644
--- a/website/src/app/llms.txt/route.ts
+++ b/website/src/app/llms.txt/route.ts
@@ -15,7 +15,7 @@ export function GET() {
> ${appTagline}
-${appName} is an open-source CLI that captures browser traffic with Playwright (or Chrome DevTools MCP) and uses your configured AI SDK to generate a typed API client from the recorded HAR. Output languages: Python, JavaScript, TypeScript, Go, Java, C#, PHP, Ruby, and C.
+${appName} is an open-source CLI that captures browser traffic — in the default agent mode via a browser MCP server (Playwright or Chrome DevTools) or the Vercel agent-browser CLI, or in manual mode via a local Playwright browser (optional \`[manual]\` extra) — and uses your configured AI SDK to generate a typed API client from the captured requests. Output languages: Python, JavaScript, TypeScript, Go, Java, C#, PHP, Ruby, and C.
## Documentation