Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(browser): browser-harness plugin + skill (Wave 19)
trohitgcc5a43c71d9f
diff --git a/CLAUDE.md b/CLAUDE.mdindex 9b0205eb..05a306c1 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -26,6 +26,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Pricing Service](./docs-internal/pricing_service.md)** | Centralized cost tracking for LLM tokens and API services (Twitter, Google Maps) with HTTPX event hooks || **[Proxy Service](./docs-internal/proxy_service.md)** | Residential proxy provider management with template-based URL formatting, health scoring, and transparent HTTP node injection || **[Email Service](./docs-internal/email_service.md)** | IMAP/SMTP integration via Himalaya CLI with EmailService orchestrator, provider presets, custom credential fallback, and polling triggers |+| **[Browser Harness](./docs-internal/browser_harness.md)** | Wave 19 — `browserHarness` node integrating browser-use/browser-harness (alpha, raw CDP against the user's REAL Chrome; no Playwright). Primary op `run_python` pipes agent-authored Python to the CLI stdin against ~25 pre-imported helpers (see→act→verify loop: `capture_screenshot` → `click_at_xy` → `wait_for_load` → `js`). uv-tool install under `packages/browser-harness/`; daemon state pinned to `daemons/browser-harness/` via `BH_RUNTIME_DIR`/`BH_TMP_DIR`; Windows OK (token-auth TCP loopback IPC). Sibling of the stable `browser` node (agent-browser `@eN` refs) — NOT a replacement. Chrome must be CDP-reachable (`BU_CDP_URL` / chrome://inspect / `--remote-debugging-port`); `doctor` op diagnoses. Skill: `skills/web_agent/browser-harness-skill/`. || **[Stripe Service](./docs-internal/stripe_service.md)** | Stripe CLI integration — `stripeAction` (CLI pass-through) + `stripeReceive` (signed-webhook trigger). Reference plugin for the Wave 12 event framework AND the **CLI-managed-auth pattern**: browser OAuth via `stripe login --non-interactive` + `--complete <url>` (URL extracted from `next_step` shell command via `shlex.split`) + auto-installed binary (`ensure_stripe_cli` from GitHub releases) + marker-token reuse of `auth_service.store_oauth_tokens` + CloudEvents-shaped `broadcast_credential_event("credential.oauth.connected" \| ".disconnected", provider=...)` — zero per-provider hardcoding in the frontend. Daemon stdout/stderr ingested via `ProcessService`'s `line_handler` callback (no log-file tailing); credential gate routed through `await self.has_credential()` so non-api-key auth (`is_logged_in()`) plugs in via subclass override. || **[Vercel Service](./docs-internal/vercel_service.md)** | Vercel CLI integration — `vercelAction` (deploy / inspect / list + `custom` CLI passthrough; AI tool `vercel`). Second reference for the CLI-managed-auth pattern: the **device-flow variant** (`vercel login` is one blocking process — no two-step `--complete`; direct `create_subprocess_exec` + chunk-based banner read for the code-embedding URL, pumps drain pipes for the process lifetime, success gate = auth.json mtime advance + sniff). Dual auth: optional `vercel_token` field (injected as `VERCEL_TOKEN` env, never argv) OR CLI login with marker tokens. All invocations pin `--global-config <DATA_DIR>/vercel/` (the `CLAUDE_CONFIG_DIR` isolation idiom). npm install into the shared packages tree (pinned version). First-deploy project guard raises `NodeUserError` instead of letting Vercel derive an invalid name from the workspace dir. Shipped with the generic OAuthConnect change: Login gates on **required** fields only, so optional fields never block CLI login. || **[GitHub Service](./docs-internal/github_service.md)** | GitHub CLI integration — `githubAction` (repo_clone / pr_create / pr_list / pr_merge / issue_create / issue_list + `custom` passthrough; AI tool `github`). **Strict Stripe pattern: the gh CLI owns its own auth** — no token stored/injected by MachinaOs, no auth pre-flight (gh's own error surfaces), marker OAuth row for the modal badge, fieldless catalogue entry. Login = spawned `gh auth login --hostname github.com --git-protocol https --web` — source-verified: headless takes the `isInteractive=false` branch (no Press-Enter), prints one-time code + `github.com/login/device` URL on stderr; the modal shows the code via the generic `verificationCode` plumbing (`useCredentialPanel` → `OAuthConnect`). `login_env()` strips `GH_TOKEN`/`GITHUB_TOKEN` (login aborts otherwise). Success gate = `gh auth status` exit 0, then best-effort **`gh auth setup-git`** (official bridge — the future git node needs zero auth code). gh binary: **project-local, pooch-driven** (temporal `_install.py` idiom) — pinned release under `package_dir("gh")`, system gh never consulted (auth state still shared: gh config paths are user-level). Palette group `vcs`. |diff --git a/docs-internal/browser_harness.md b/docs-internal/browser_harness.mdnew file mode 100644index 00000000..a1bba8cd--- /dev/null+++ b/docs-internal/browser_harness.md@@ -0,0 +1,56 @@+# Browser Harness (browser-use/browser-harness)++Wave 19 integration of [browser-use/browser-harness](https://github.com/browser-use/browser-harness) — a minimal (~1,000 LoC, MIT, **alpha**) Python CDP bridge that drives the user's **real Chrome** with no Playwright and no bundled Chromium. Sibling of the `browser` node (agent-browser), **not** a replacement.++## Architecture++```+browserHarness node ──> BrowserHarnessService.run_code(code)+ │ (subprocess: pipe Python to CLI stdin; CLI exits after exec)+ ▼+ browser-harness CLI ──IPC──> daemon (detached, persists)+ │ │ AF_UNIX (POSIX) / token-auth TCP loopback (Windows)+ │ ▼+ │ one CDP WebSocket+ ▼ ▼+ stdout (result) user's Chrome+```++- **Plugin**: [`server/nodes/browser/browser_harness/`](../server/nodes/browser/browser_harness/) — `_install.py` (uv tool install → `<DATA_DIR>/packages/browser-harness/{tools,bin}`, lazy on first use), `_service.py` (subprocess wrapper + `NodeUserError` mapping + daemon shutdown hook), `__init__.py` (`BrowserHarnessNode`).+- **Node ops**: `run_python` (primary — Python piped verbatim to the CLI, executed against ~25 pre-imported helpers), plus `goto` / `screenshot` / `js` / `tabs` shortcuts and `doctor` diagnostics. `task_queue=BROWSER`, `usable_as_tool=True`, `tool_name="browser_harness"`.+- **Skill**: [`server/skills/web_agent/browser-harness-skill/SKILL.md`](../server/skills/web_agent/browser-harness-skill/SKILL.md) — the see→act→verify loop (`capture_screenshot` → `click_at_xy` → `wait_for_load` → `js` fallback), helper reference, print-JSON-last convention.+- **State isolation**: `BH_RUNTIME_DIR` + `BH_TMP_DIR` are pinned to `<DATA_DIR>/daemons/browser-harness/` on every spawn — daemon pid/port/sock + screenshots + logs never land in the user-global `~/.config/browser-harness/`.+- **Windows**: verified working (2026-07-10 spike) — upstream `_ipc.py` uses AF_UNIX on POSIX and **token-authenticated TCP loopback on Windows**; daemon detaches with `CREATE_NO_WINDOW`.++## Chrome prerequisite++The harness needs a CDP-reachable Chrome, discovered in this order (upstream `daemon.py`):+1. `BU_CDP_URL` / `BU_CDP_WS` env override (dedicated automation Chrome) — passed through from the MachinaOS process env untouched.+2. `DevToolsActivePort` written by Chrome when the user enables **chrome://inspect/#remote-debugging** ("Allow remote debugging for this browser instance").+3. Port probe on 9222/9223 (e.g. `chrome --remote-debugging-port=9222`).++The node's `doctor` operation runs `browser-harness doctor` and returns the full checklist — the first thing to try when calls fail with connection guidance.++## vs the `browser` node (agent-browser)++| | `browser` (agent-browser) | `browserHarness` |+|---|---|---|+| Engine | npm CLI + bundled Chrome-for-Testing (or system browser) | user's real Chrome over raw CDP |+| Interaction | accessibility tree, `@eN` refs | screenshots + coordinate clicks + `js()` |+| Sessions | named sessions, instance cap, idle timeout | one shared browser (daemon-held CDP socket) |+| Agent surface | 15 structured operations | freeform Python against helpers |+| Maturity | stable, Windows-tested | **alpha (v0.1.x)** |++Default to `browser`; reach for `browserHarness` when the task needs the user's logins/profile, bot-hostile or shadow-DOM-heavy sites, iframes (`iframe_target`), or raw CDP.++## Security caveats (know before enabling widely)++- Upstream is **alpha**; the HN launch thread reported an unaddressed RCE and prompt-injection exposure. The `run_python` op executes agent-authored Python **outside** the Monty sandbox (same trust level as `pythonExecutor`, but with the user's real browser attached).+- It drives the user's real, logged-in Chrome — the skill instructs agents to never touch accounts/purchases beyond the explicit task, but treat the node as `destructive + open_world` (its annotations say so).+- Node is visible in **dev mode only** (not in `enabled_nodes` — same posture as `browser`).++## Ops notes++- Install is lazy: first use runs `uv tool install --python 3.12 browser-harness` (requires `uv` on PATH — already a MachinaOS toolchain dependency).+- Daemon shutdown: FastAPI lifespan runs the `browser_harness` shutdown hook → kills the daemon via `<DATA_DIR>/daemons/browser-harness/bu.pid`.+- Upgrades: delete `<DATA_DIR>/packages/browser-harness/` and let the next use reinstall, or run the CLI's `--update`.diff --git a/server/nodes/browser/__init__.py b/server/nodes/browser/__init__.pyindex 392945c5..4337eb34 100644--- a/server/nodes/browser/__init__.py+++ b/server/nodes/browser/__init__.py@@ -19,4 +19,9 @@ from services.plugin.shutdown_hooks import register_shutdown_hookfrom ._service import shutdown_browser_servicefrom . import browser as _browser # noqa: F401 — side-effect: registers BrowserNode+# Wave 19: browser-harness sibling plugin (browser-use/browser-harness).+# Registers BrowserHarnessNode + its own "browser_harness" shutdown hook+# from inside the subpackage; eager import keeps discovery deterministic.+from . import browser_harness as _browser_harness # noqa: F401+register_shutdown_hook("browser_service", shutdown_browser_service)diff --git a/server/nodes/browser/browser_harness/__init__.py b/server/nodes/browser/browser_harness/__init__.pynew file mode 100644index 00000000..e4789e68--- /dev/null+++ b/server/nodes/browser/browser_harness/__init__.py@@ -0,0 +1,164 @@+"""Browser Harness — browser-use/browser-harness integration (Wave 19).++Drives the user's REAL Chrome over raw CDP via the ``browser-harness``+CLI (PyPI, alpha). The upstream philosophy is "the LLM writes Python+against ~25 pre-imported helpers" — so the primary operation here is+``run_python`` (code piped verbatim to the CLI stdin, the documented+driving model), with a few convenience operations for structured+workflow use. The companion skill+(``server/skills/web_agent/browser-harness-skill/``) teaches agents the+helper API and the screenshot -> click_at_xy -> wait_for_load loop.++Sibling to the ``browser`` node (agent-browser), NOT a replacement:+that node's accessibility-tree ``@eN`` model stays the stable default;+this one trades structure for raw-CDP freedom on the user's own Chrome+profile/session state.+"""++from __future__ import annotations++from typing import Any, Literal, Optional++from pydantic import BaseModel, ConfigDict, Field++from core.logging import get_logger+from services.plugin import ActionNode, NodeContext, Operation, TaskQueue+from services.plugin.base import NodeUserError+from services.plugin.shutdown_hooks import register_shutdown_hook++from ._service import shutdown_browser_harness_service++logger = get_logger(__name__)++register_shutdown_hook("browser_harness", shutdown_browser_harness_service)+++def _code_for(params: "BrowserHarnessParams") -> str:+ """Translate convenience operations into harness helper one-liners.++ ``repr()`` embeds user strings safely (quotes/backslashes) since the+ generated text is executed as Python by the harness.+ """+ op = params.operation+ if op == "run_python":+ return params.code+ if op == "goto":+ url = (params.url or "").strip()+ if not url:+ raise NodeUserError("url is required for the goto operation")+ return f"goto_url({url!r})\nwait_for_load()\nprint(page_info())"+ if op == "screenshot":+ return f"p = capture_screenshot(full={bool(params.full_page)})\nprint(p)"+ if op == "js":+ expr = (params.expression or "").strip()+ if not expr:+ raise NodeUserError("expression is required for the js operation")+ return f"print(js({expr!r}))"+ if op == "tabs":+ return "print(list_tabs())"+ raise NodeUserError(f"Unknown operation: {op}")+++class BrowserHarnessParams(BaseModel):+ operation: Literal["run_python", "goto", "screenshot", "js", "tabs", "doctor"] = Field(+ default="run_python",+ description=(+ "run_python executes a Python snippet against the harness "+ "helpers (goto_url, click_at_xy, capture_screenshot, js, "+ "fill_input, wait_for_load, ...). The other operations are "+ "single-helper shortcuts; doctor diagnoses the Chrome/CDP "+ "connection."+ ),+ )+ code: str = Field(+ default="",+ description=(+ "Python for run_python. Helpers are pre-imported. Print a "+ "JSON object as the final line for structured output."+ ),+ json_schema_extra={+ "editor": "code",+ "rows": 8,+ "displayOptions": {"show": {"operation": ["run_python"]}},+ },+ )+ url: str = Field(+ default="",+ description="URL to open.",+ json_schema_extra={"displayOptions": {"show": {"operation": ["goto"]}}},+ )+ expression: str = Field(+ default="",+ description="JavaScript to evaluate in the page.",+ json_schema_extra={+ "rows": 4,+ "displayOptions": {"show": {"operation": ["js"]}},+ },+ )+ full_page: bool = Field(+ default=False,+ description="Capture the full scrollable page.",+ json_schema_extra={"displayOptions": {"show": {"operation": ["screenshot"]}}},+ )+ timeout: int = Field(+ default=60,+ ge=5,+ le=600,+ description="Script timeout in seconds.",+ )++ model_config = ConfigDict(extra="ignore")+++class BrowserHarnessOutput(BaseModel):+ operation: Optional[str] = None+ data: Optional[Any] = None++ model_config = ConfigDict(extra="allow")+++class BrowserHarnessNode(ActionNode):+ type = "browserHarness"+ display_name = "Browser Harness"+ subtitle = "Raw-CDP Chrome control"+ group = ("browser", "tool")+ description = "Drive your real Chrome over raw CDP via browser-use/browser-harness (alpha)"+ component_kind = "square"+ tool_name = "browser_harness"+ tool_description = (+ "Control the user's real Chrome by writing Python against pre-imported "+ "CDP helpers. Loop: capture_screenshot() to SEE the page, "+ "click_at_xy(x, y) to interact, wait_for_load(), then js(...) for "+ "DOM reads. Prefer operation=run_python with a short script; print a "+ "JSON object as the last line for structured results. Use "+ "operation=doctor if the browser seems unreachable."+ )+ handles = (+ {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},+ {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},+ )+ annotations = {"destructive": True, "readonly": False, "open_world": True}+ task_queue = TaskQueue.BROWSER+ usable_as_tool = True++ Params = BrowserHarnessParams+ Output = BrowserHarnessOutput++ @Operation("dispatch")+ async def dispatch(self, ctx: NodeContext, params: BrowserHarnessParams) -> BrowserHarnessOutput:+ from ._service import get_browser_harness_service++ svc = get_browser_harness_service()+ if not svc:+ raise NodeUserError(+ "browser-harness could not be installed — uv is required on "+ "PATH (https://docs.astral.sh/uv/). It installs automatically "+ "on first use once uv is available."+ )++ if params.operation == "doctor":+ data = await svc.doctor()+ else:+ data = await svc.run_code(_code_for(params), timeout=params.timeout)++ return BrowserHarnessOutput(operation=params.operation, data=data)diff --git a/server/nodes/browser/browser_harness/_install.py b/server/nodes/browser/browser_harness/_install.pynew file mode 100644index 00000000..4caf759d--- /dev/null+++ b/server/nodes/browser/browser_harness/_install.py@@ -0,0 +1,73 @@+"""``browser-harness`` local install — a uv *tool* (Python), not npm.++Unlike ``agent-browser`` (npm, shared tree) the harness is a PyPI+package installed as an isolated uv tool. It lands under+:func:`core.paths.package_dir`::++ <DATA_DIR>/packages/browser-harness/tools/ (venv-per-tool)+ <DATA_DIR>/packages/browser-harness/bin/ (browser-harness[.exe])++— the same non-npm-binary precedent as ``packages/stripe/`` and+``packages/temporal/``. ``uv tool install`` is idempotent and owns+upgrades; we never hand-manage the venv.++Windows note (verified 2026-07-10 spike): upstream is fully+Windows-aware — daemon IPC is AF_UNIX on POSIX and token-authenticated+TCP loopback on Windows (``browser_harness/_ipc.py`` line 1).+"""++from __future__ import annotations++import os+import shutil+import subprocess+import sys+from typing import Optional++from core.logging import get_logger+from core.paths import package_dir++logger = get_logger(__name__)++_PYPI_SPEC = "browser-harness"+_PYTHON_PIN = "3.12" # upstream install.md pins 3.12+++def _bin_path():+ name = "browser-harness.exe" if sys.platform == "win32" else "browser-harness"+ return package_dir("browser-harness") / "bin" / name+++def browser_harness_binary_path() -> Optional[str]:+ """Return path to the browser-harness CLI, installing on miss."""+ bin_path = _bin_path()+ if bin_path.exists():+ return str(bin_path)++ uv_cmd = shutil.which("uv")+ if not uv_cmd:+ logger.warning("[browser-harness] uv not on PATH; cannot install %s", _PYPI_SPEC)+ return None++ root = package_dir("browser-harness")+ root.mkdir(parents=True, exist_ok=True)+ logger.info("[browser-harness] installing %s into %s", _PYPI_SPEC, root)+ result = subprocess.run(+ [uv_cmd, "tool", "install", "--python", _PYTHON_PIN, "--upgrade", _PYPI_SPEC],+ capture_output=True,+ text=True,+ env={+ **os.environ,+ "UV_TOOL_DIR": str(root / "tools"),+ "UV_TOOL_BIN_DIR": str(root / "bin"),+ },+ )+ if result.returncode != 0 or not bin_path.exists():+ logger.error("[browser-harness] uv tool install failed: %s", result.stderr.strip())+ return None++ logger.info("[browser-harness] installed at %s", bin_path)+ return str(bin_path)+++__all__ = ["browser_harness_binary_path"]diff --git a/server/nodes/browser/browser_harness/_service.py b/server/nodes/browser/browser_harness/_service.pynew file mode 100644index 00000000..c9459a4e--- /dev/null+++ b/server/nodes/browser/browser_harness/_service.py@@ -0,0 +1,219 @@+"""Thin wrapper around the browser-harness CLI (browser-use/browser-harness).++Stateless client — finds the binary via :mod:`._install`, pipes Python+code to the CLI's stdin (the upstream-documented driving model), and+returns the captured output. The harness's background daemon holds the+single CDP WebSocket to Chrome and persists across CLI invocations;+this service never manages the daemon directly beyond best-effort+shutdown.++Differences from the sibling ``agent-browser`` service worth knowing:++- The CLI **exits** after executing the piped code (the daemon is a+ separate detached process), so a plain ``subprocess.run`` with+ timeout works — no first-line-then-kill-tree dance.+- Interaction model is screenshot + coordinate clicks + ``js()``, not+ accessibility-tree ``@eN`` refs.+- Chrome must be reachable over CDP: either the user's Chrome grants+ remote debugging (chrome://inspect), a dedicated Chrome runs with+ ``--remote-debugging-port=9222``, or ``BU_CDP_URL``/``BU_CDP_WS``+ point at one. Failures surface as :class:`NodeUserError` with that+ guidance.++Runtime/state isolation: ``BH_RUNTIME_DIR`` (daemon sock/port/pid) and+``BH_TMP_DIR`` (screenshots, daemon log) are pinned under+``<DATA_DIR>/daemons/browser-harness/`` so the harness never writes to+the user-global ``~/.config/browser-harness`` runtime location.+"""++from __future__ import annotations++import asyncio+import json+import os+import subprocess+from typing import Any, Dict, Optional++from core.logging import get_logger+from core.paths import daemons_dir+from services.plugin.base import NodeUserError++from ._install import browser_harness_binary_path++logger = get_logger(__name__)++_MAX_OUTPUT = 100_000++# stderr fragments that indicate a user-correctable environment problem+# (vs a genuine bug). Matched case-insensitively.+_CHROME_HINTS = (+ "devtoolsactiveport",+ "unreachable",+ "is the dedicated automation chrome running",+ "chrome://inspect",+ "no browser connection",+ "connection refused",+)++_CHROME_GUIDANCE = (+ "browser-harness cannot reach Chrome over CDP. Either start a "+ "dedicated Chrome with --remote-debugging-port=9222, enable "+ "chrome://inspect/#remote-debugging in your running Chrome, or set "+ "BU_CDP_URL. Run the 'doctor' operation for a full diagnosis."+)+++def _runtime_dir():+ d = daemons_dir() / "browser-harness"+ d.mkdir(parents=True, exist_ok=True)+ return d+++def _spawn_env() -> Dict[str, str]:+ """Env for harness spawns. The daemon inherits this on auto-start,+ so the runtime/tmp isolation must ride every invocation that could+ be the one that starts it. BU_CDP_URL / BU_CDP_WS pass through from+ the process env untouched (operator-controlled overrides)."""+ root = _runtime_dir()+ tmp = root / "tmp"+ tmp.mkdir(parents=True, exist_ok=True)+ return {+ **os.environ,+ "BH_RUNTIME_DIR": str(root),+ "BH_TMP_DIR": str(tmp),+ }+++class BrowserHarnessService:+ """Subprocess wrapper for the browser-harness CLI."""++ def __init__(self, binary: str) -> None:+ self._binary = binary++ async def run_code(self, code: str, timeout: int = 60) -> Dict[str, Any]:+ """Pipe Python ``code`` to the harness and return its output.++ Convention (taught by the skill): scripts print a JSON object as+ their final line for structured output. When the last stdout+ line parses as JSON it is returned under ``result``; the full+ stdout always rides along under ``output``.+ """+ if not code.strip():+ raise NodeUserError("code is required — Python to run against the browser-harness helpers")++ raw = await asyncio.to_thread(self._run_sync, [self._binary], code, timeout)+ return self._shape_output(raw)++ async def doctor(self) -> Dict[str, Any]:+ """Run ``browser-harness doctor``. Exit 1 = checks failed, which+ is a *report*, not an error — always return the output."""+ raw = await asyncio.to_thread(self._run_sync, [self._binary, "doctor"], None, 30, check=False)+ return {"output": raw}++ @staticmethod+ def _shape_output(raw: str) -> Dict[str, Any]:+ if len(raw) > _MAX_OUTPUT:+ raw = raw[:_MAX_OUTPUT] + "\n...(truncated)"+ lines = [ln for ln in raw.strip().splitlines() if ln.strip()]+ if lines:+ try:+ return {"result": json.loads(lines[-1]), "output": raw.strip()}+ except json.JSONDecodeError:+ pass+ return {"output": raw.strip()}++ def _run_sync(+ self,+ argv: list,+ stdin_text: Optional[str],+ timeout: int,+ check: bool = True,+ ) -> str:+ """Run the CLI to completion (it exits after executing the piped+ code; the daemon detaches separately). shell=False with list+ argv per the BatBadBut-safe invocation rule."""+ try:+ proc = subprocess.run(+ argv,+ input=stdin_text,+ capture_output=True,+ text=True,+ encoding="utf-8",+ errors="replace",+ timeout=timeout,+ shell=False,+ env=_spawn_env(),+ )+ except subprocess.TimeoutExpired as e:+ raise NodeUserError(+ f"browser-harness timed out after {timeout}s — the page may be slow; "+ "raise the timeout parameter or simplify the script"+ ) from e+ except FileNotFoundError as e:+ raise NodeUserError("browser-harness binary vanished — it reinstalls automatically on next use") from e++ if check and proc.returncode != 0:+ err = (proc.stderr or "").strip() or (proc.stdout or "").strip()+ low = err.lower()+ if any(h in low for h in _CHROME_HINTS):+ raise NodeUserError(f"{_CHROME_GUIDANCE}\n\nDetail: {err[-500:]}")+ # Python errors from the user's snippet are user-correctable too.+ if "traceback" in low or "error" in low:+ raise NodeUserError(f"browser-harness script failed:\n{err[-1500:]}")+ raise RuntimeError(f"browser-harness exited {proc.returncode}: {err[-1500:]}")++ out = (proc.stdout or "").strip()+ stderr = (proc.stderr or "").strip()+ if not out and stderr:+ # doctor + some diagnostics write to stderr+ return stderr+ return out+++# -- Module-level lazy singleton (BrowserService pattern) ----------------++_instance: Optional[BrowserHarnessService] = None+++def get_browser_harness_service() -> Optional[BrowserHarnessService]:+ """Return the singleton, or None if the CLI cannot be installed."""+ global _instance+ if _instance is None:+ binary = browser_harness_binary_path()+ if binary:+ _instance = BrowserHarnessService(binary)+ return _instance+++async def shutdown_browser_harness_service() -> None:+ """Best-effort daemon stop on FastAPI lifespan shutdown.++ The harness has no ``stop`` CLI verb; its daemon writes+ ``<BH_RUNTIME_DIR>/bu.pid`` (per-instance stem because we isolate+ BH_RUNTIME_DIR). Kill the tree if present; missing/stale pid files+ are silently ignored.+ """+ pid_file = _runtime_dir() / "bu.pid"+ try:+ pid = int(pid_file.read_text().strip())+ except (FileNotFoundError, ValueError, OSError):+ return+ try:+ from services._supervisor.util import kill_tree++ kill_tree(pid)+ logger.info("[browser-harness] daemon pid=%d stopped", pid)+ except Exception as e: # noqa: BLE001 — shutdown must never raise+ logger.debug("[browser-harness] daemon stop skipped: %s", e)+ finally:+ try:+ pid_file.unlink(missing_ok=True)+ except OSError:+ pass+++__all__ = [+ "BrowserHarnessService",+ "get_browser_harness_service",+ "shutdown_browser_harness_service",+]diff --git a/server/nodes/browser/browser_harness/icon.svg b/server/nodes/browser/browser_harness/icon.svgnew file mode 100644index 00000000..4ca3a4be--- /dev/null+++ b/server/nodes/browser/browser_harness/icon.svg@@ -0,0 +1,9 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="#bd93f9" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">+ <rect x="2" y="3" width="20" height="16" rx="2"/>+ <line x1="2" y1="7.5" x2="22" y2="7.5"/>+ <circle cx="4.6" cy="5.3" r="0.4" fill="#bd93f9"/>+ <circle cx="7" cy="5.3" r="0.4" fill="#bd93f9"/>+ <polyline points="7 11 9.5 13.5 7 16"/>+ <line x1="12" y1="16" x2="16.5" y2="16"/>+ <path d="M18.5 11.5l2 2-2 2" stroke="#ff79c6"/>+</svg>diff --git a/server/nodes/browser/browser_harness/meta.json b/server/nodes/browser/browser_harness/meta.jsonnew file mode 100644index 00000000..8f309ba5--- /dev/null+++ b/server/nodes/browser/browser_harness/meta.json@@ -0,0 +1,3 @@+{+ "color": "#bd93f9"+}diff --git a/server/skills/web_agent/browser-harness-skill/SKILL.md b/server/skills/web_agent/browser-harness-skill/SKILL.mdnew file mode 100644index 00000000..498235f6--- /dev/null+++ b/server/skills/web_agent/browser-harness-skill/SKILL.md@@ -0,0 +1,114 @@+---+name: browser-harness-skill+description: Drive the user's real Chrome over raw CDP by writing Python against browser-harness helpers - screenshot, coordinate clicks, JS evaluation, form fill, tabs. For tasks needing full freedom or the user's own logins.+allowed-tools: browserHarness+metadata:+ author: machina+ version: "1.0"+ category: web++---++# Browser Harness Skill++Controls the user's **real Chrome** through raw CDP (browser-use/browser-harness). Unlike the `browser` tool (accessibility tree + `@eN` refs), this tool is **vision-first**: you look at screenshots and click coordinates, with `js()` for reading the DOM. Use it when you need the user's logged-in browser, sites that fight automation frameworks, or interactions the structured tool can't express.++## Core Workflow++The **see -> act -> verify** loop, written as Python in `operation: run_python`:++1. `capture_screenshot()` to SEE the page (the returned path is an image you can view)+2. `click_at_xy(x, y)` at coordinates you identified from the screenshot+3. `wait_for_load()` after anything that navigates+4. `capture_screenshot()` again to verify the result+5. `js(...)` when you need exact text/attributes instead of pixels++```json+{"operation": "run_python", "code": "ensure_real_tab()\ngoto_url('https://example.com')\nwait_for_load()\nprint(page_info())"}+```++## browserHarness Tool++| Field | Type | Required | Description |+|-------|------|----------|-------------|+| operation | string | Yes | run_python (primary), goto, screenshot, js, tabs, doctor |+| code | string | run_python | Python executed with helpers pre-imported. Print a JSON object as the LAST line for structured output |+| url | string | goto | URL to open |+| expression | string | js | JavaScript to evaluate in the page |+| full_page | bool | screenshot | Capture the full scrollable page (default: false) |+| timeout | int | no | Script timeout in seconds (default: 60) |++## Helper Reference (inside run_python)++All pre-imported — never `import` them.++**Navigation & tabs**+- `goto_url(url)` — navigate current tab+- `new_tab(url="about:blank")` / `close_tab(target=None)` / `switch_tab(target)`+- `list_tabs(include_chrome=True)` — enumerate open tabs+- `current_tab()` / `ensure_real_tab()` — make sure a real page tab is focused (call first!)+- `page_info()` — `{url, title, w, h, sx, sy, pw, ph}` (viewport + scroll + page size)+- `iframe_target(url_substr)` — target for js/interaction inside an iframe++**Seeing**+- `capture_screenshot(path=None, full=False, max_dim=None)` — returns the saved PNG path. Use `max_dim=1200` to keep images small++**Acting**+- `click_at_xy(x, y, button="left", clicks=1)` — coordinates from the screenshot+- `type_text(text)` — types into the focused element (click the field first)+- `fill_input(selector, text, clear_first=True)` — CSS-selector form fill+- `press_key(key, modifiers=0)` / `dispatch_key(selector, key="Enter")`+- `scroll(x, y, dy=-300, dx=0)` — scroll at a point (negative dy = down varies; verify with page_info)+- `upload_file(selector, path)`++**Reading & escape hatches**+- `js(expression, target_id=None)` — evaluate JS, returns the value. Your precision tool for text, attributes, element rects+- `cdp(method, session_id=None, **params)` — raw CDP call when nothing else fits+- `http_get(...)` — plain HTTP fetch without the browser++**Waiting**+- `wait_for_load(timeout=15.0)` — after navigation/clicks that navigate+- `wait_for_element(selector, ...)` / `wait_for_network_idle(...)` / `wait()`++## Patterns++**Find something on a page** (screenshot first, then read precisely):+```python+ensure_real_tab()+goto_url("https://news.ycombinator.com")+wait_for_load()+titles = js("[...document.querySelectorAll('.titleline > a')].slice(0,5).map(a => a.textContent)")+import json # stdlib imports are fine; helpers are pre-imported+print(json.dumps({"top5": titles}))+```++**Click something you can see** (get coordinates from the element itself, not by eye):+```python+r = js("(() => { const el = [...document.querySelectorAll('a')].find(a => a.textContent.includes('More')); const b = el.getBoundingClientRect(); return {x: b.x + b.width/2, y: b.y + b.height/2}; })()")+click_at_xy(r["x"], r["y"])+wait_for_load()+print(page_info())+```++**Fill a login/search form**:+```python+fill_input("input[name=q]", "machina os")+dispatch_key("input[name=q]", "Enter")+wait_for_load()+print(page_info())+```++## Rules++- **Always start with `ensure_real_tab()`** in the first call of a session — the daemon may be attached to a chrome:// or extension target.+- **Print a JSON object as the final line** — the tool parses it into `result` for you; everything else lands in `output`.+- **Prefer `js()`-computed coordinates** (getBoundingClientRect) over guessing pixels from a screenshot; screenshots confirm state, JS locates precisely.+- **One logical step per call.** Short scripts fail cleanly; long ones lose all progress on any exception.+- Screenshots return a **file path** under the harness tmp dir — mention the path in your reply if the user should look at it.+- If a call fails with a Chrome-connection error, run `{"operation": "doctor"}` and relay its report: the user may need to start Chrome with `--remote-debugging-port=9222` or allow remote debugging via chrome://inspect.+- The harness drives **one shared browser** — do not assume parallel calls get isolated windows. State (cookies, logins, open tabs) persists across calls by design.+- This is the user's REAL browser: never touch logged-in accounts, submit forms, or make purchases beyond what the task explicitly asks.++## When to use `browser` instead++Prefer the structured `browser` tool when the site is simple and the accessibility tree works — its `@eN` snapshot loop is cheaper (no screenshots) and more deterministic. Come to `browserHarness` for: the user's own Chrome profile/logins, canvas/shadow-DOM-heavy or bot-hostile sites, iframes needing `iframe_target`, or anything requiring raw CDP.diff --git a/server/tests/nodes/test_browser_harness.py b/server/tests/nodes/test_browser_harness.pynew file mode 100644index 00000000..a2b16a94--- /dev/null+++ b/server/tests/nodes/test_browser_harness.py@@ -0,0 +1,234 @@+"""Contract tests for the browserHarness node (Wave 19).++Freezes the node's input -> output behaviour + the op -> generated-code+mapping. All subprocess side-effects are mocked:+ - node tests patch `get_browser_harness_service` with a fake exposing+ async `run_code` / `doctor` (mirrors _FakeBrowserService in+ test_web_automation.py).+ - service tests patch `BrowserHarnessService._run_sync` so no+ subprocess is spawned.+"""++from __future__ import annotations++from unittest.mock import patch++import pytest++pytestmark = pytest.mark.node_contract+++# ============================================================================+# Helpers+# ============================================================================+++class _FakeHarnessService:+ def __init__(self, canned: dict | None = None, error: Exception | None = None):+ self._canned = canned if canned is not None else {"output": "ok"}+ self._error = error+ self.code_calls: list[tuple[str, int]] = []+ self.doctor_calls: int = 0++ async def run_code(self, code, timeout=60):+ self.code_calls.append((code, timeout))+ if self._error:+ raise self._error+ return self._canned++ async def doctor(self):+ self.doctor_calls += 1+ if self._error:+ raise self._error+ return {"output": "doctor report"}+++def _patch_service(svc):+ return patch(+ "nodes.browser.browser_harness._service.get_browser_harness_service",+ return_value=svc,+ )+++# ============================================================================+# Node contract+# ============================================================================+++class TestBrowserHarnessNode:+ async def test_run_python_pipes_code_verbatim(self, harness):+ fake = _FakeHarnessService(canned={"result": {"title": "Example"}, "output": "{}"})+ code = "ensure_real_tab()\nprint(page_info())"++ with _patch_service(fake):+ result = await harness.execute(+ "browserHarness",+ {"operation": "run_python", "code": code, "timeout": 45},+ )++ harness.assert_envelope(result, success=True)+ harness.assert_output_shape(result, ["operation", "data"])+ payload = result["result"]+ assert payload["operation"] == "run_python"+ assert payload["data"]["result"] == {"title": "Example"}+ assert fake.code_calls == [(code, 45)]++ async def test_goto_generates_navigation_snippet(self, harness):+ fake = _FakeHarnessService()++ with _patch_service(fake):+ result = await harness.execute(+ "browserHarness",+ {"operation": "goto", "url": "https://example.com"},+ )++ harness.assert_envelope(result, success=True)+ code, _ = fake.code_calls[-1]+ assert "goto_url('https://example.com')" in code+ assert "wait_for_load()" in code+ assert "page_info()" in code++ async def test_goto_url_with_quotes_is_repr_safe(self, harness):+ """repr()-embedding must survive quotes in user input — the+ generated text is executed as Python by the harness."""+ fake = _FakeHarnessService()+ tricky = "https://example.com/?q='a'\"b\""++ with _patch_service(fake):+ await harness.execute("browserHarness", {"operation": "goto", "url": tricky})++ code, _ = fake.code_calls[-1]+ assert repr(tricky) in code++ async def test_js_requires_expression(self, harness):+ fake = _FakeHarnessService()++ with _patch_service(fake):+ result = await harness.execute("browserHarness", {"operation": "js"})++ harness.assert_envelope(result, success=False)+ assert "expression is required" in (result.get("error") or "")++ async def test_screenshot_maps_full_page(self, harness):+ fake = _FakeHarnessService()++ with _patch_service(fake):+ await harness.execute(+ "browserHarness",+ {"operation": "screenshot", "full_page": True},+ )++ code, _ = fake.code_calls[-1]+ assert "capture_screenshot(full=True)" in code++ async def test_doctor_bypasses_code_path(self, harness):+ fake = _FakeHarnessService()++ with _patch_service(fake):+ result = await harness.execute("browserHarness", {"operation": "doctor"})++ harness.assert_envelope(result, success=True)+ assert fake.doctor_calls == 1+ assert fake.code_calls == []+ assert result["result"]["data"]["output"] == "doctor report"++ async def test_service_unavailable_is_user_error(self, harness):+ with patch(+ "nodes.browser.browser_harness._service.get_browser_harness_service",+ return_value=None,+ ):+ result = await harness.execute(+ "browserHarness",+ {"operation": "run_python", "code": "print(1)"},+ )++ harness.assert_envelope(result, success=False)+ assert result.get("error_type") == "NodeUserError"+ assert "uv" in (result.get("error") or "")++ async def test_node_user_error_passthrough(self, harness):+ from services.plugin.base import NodeUserError++ fake = _FakeHarnessService(error=NodeUserError("browser-harness cannot reach Chrome over CDP. detail"))++ with _patch_service(fake):+ result = await harness.execute(+ "browserHarness",+ {"operation": "run_python", "code": "print(1)"},+ )++ harness.assert_envelope(result, success=False)+ assert result.get("error_type") == "NodeUserError"+ assert "Chrome" in (result.get("error") or "")+++# ============================================================================+# Service unit tests (no subprocess)+# ============================================================================+++class TestBrowserHarnessService:+ def _svc(self):+ from nodes.browser.browser_harness._service import BrowserHarnessService++ return BrowserHarnessService("browser-harness")++ async def test_shape_output_parses_trailing_json(self):+ svc = self._svc()+ with patch.object(svc, "_run_sync", return_value='navigating\n{"title": "Example"}'):+ out = await svc.run_code("print(1)")+ assert out["result"] == {"title": "Example"}+ assert "navigating" in out["output"]++ async def test_shape_output_plain_text_fallback(self):+ svc = self._svc()+ with patch.object(svc, "_run_sync", return_value="C:\\shots\\shot.png"):+ out = await svc.run_code("print(p)")+ assert out == {"output": "C:\\shots\\shot.png"}++ async def test_empty_code_is_user_error(self):+ from services.plugin.base import NodeUserError++ svc = self._svc()+ with pytest.raises(NodeUserError, match="code is required"):+ await svc.run_code(" ")++ def test_chrome_hint_maps_to_user_error(self):+ """stderr containing a Chrome-connection fragment must raise+ NodeUserError with the doctor guidance, not a raw RuntimeError."""+ import subprocess as _sp+ from unittest.mock import MagicMock++ from services.plugin.base import NodeUserError++ svc = self._svc()+ proc = MagicMock(returncode=1, stdout="", stderr="DevToolsActivePort not found in [...]")+ with patch("nodes.browser.browser_harness._service.subprocess.run", return_value=proc):+ with pytest.raises(NodeUserError, match="remote-debugging-port"):+ svc._run_sync(["browser-harness"], "print(1)", 30)+ assert _sp # keep import referenced++ def test_timeout_maps_to_user_error(self):+ import subprocess as _sp++ from services.plugin.base import NodeUserError++ svc = self._svc()+ with patch(+ "nodes.browser.browser_harness._service.subprocess.run",+ side_effect=_sp.TimeoutExpired(cmd="browser-harness", timeout=30),+ ):+ with pytest.raises(NodeUserError, match="timed out"):+ svc._run_sync(["browser-harness"], "print(1)", 30)++ async def test_doctor_tolerates_exit_1(self):+ """doctor exits 1 when checks fail — that's a report, not an error."""+ import subprocess as _sp+ from unittest.mock import MagicMock++ svc = self._svc()+ proc = MagicMock(returncode=1, stdout="", stderr="[FAIL] daemon alive")+ with patch("nodes.browser.browser_harness._service.subprocess.run", return_value=proc):+ out = await svc.doctor()+ assert "[FAIL] daemon alive" in out["output"]+ assert _sp # keep import referenced