Accessibility-first browser & computer use for AI agents — one MCP server for every harness (Codex, Claude Code, OpenCode, Antigravity). Instruments the DOM/accessibility tree via CDP instead of screenshot loops: diffed snapshots, stable element refs, trusted input, typed errors. Zero dependencies
JavaScript agentic-automationai-agentsbrowser-automationbrowser-usecdpchrome-devtools-protocolcomputer-usecuamcpmodel-context-protocolrpaweb-automation
Server mode round 3 — input realism + per-instance egress
CTOL Team BJbaffd0756ea2
diff --git a/README.md b/README.mdindex 5eea57d..fa408f0 100644--- a/README.md+++ b/README.md@@ -87,6 +87,7 @@ element refs like `[ref=e7]`. Each action returns only what changed.| `AXUSE_DISPLAY_SIZE` | virtual display geometry for server mode (default `1920x1080x24`; `random` = a stable geometry per machine, so a fleet is not all 1920x1080) || `AXUSE_TIMEZONE` | IANA `TZ` for the Chrome process — `Date`/`Intl` in pages and workers match the egress persona (validated by `doctor`) || `AXUSE_LANG` | `--accept-lang` for Chrome — `navigator.language(s)` and the `Accept-Language` header match the egress persona |+| `AXUSE_PROXY` | route every launched browser through a proxy (`http://`·`socks5://`, credentials supported — answered over CDP); `ax_open {proxy}` sets it per launch || `AXUSE_NO_SANDBOX=1` | force `--no-sandbox` (auto for root/container server runs) || `AXUSE_NEUTRALIZE_WEBDRIVER=0` | disable the `navigator.webdriver` posture-fix fallback (the `AutomationControlled` launch flag already fixes it at the source) || `AXUSE_HEADLESS=1` | launch the browser without a window — **trusted internal pages only**: challenge/anti-bot pages detect headless first; servers get server mode instead |diff --git a/SKILL.md b/SKILL.mdindex 89be229..32ab478 100644--- a/SKILL.md+++ b/SKILL.md@@ -66,8 +66,11 @@ private Xvfb and keeps Chrome fully **headed** on it — never pass`headless: true` for challenge/anti-bot pages (they check headless first).The `ax_open` result reports the fact: `display: :N (virtual …)`. Personacoherence with the egress persona is configured server-side-(`AXUSE_TIMEZONE`, `AXUSE_LANG`; `navigator.webdriver` stays `false` via the-`AutomationControlled` launch flag). Solvers work unchanged; vision needs+(`AXUSE_TIMEZONE`, `AXUSE_LANG`, `AXUSE_PROXY`; `navigator.webdriver`+stays `false` via the `AutomationControlled` launch flag). For+dynamics-sensitive targets pass `human: true` on `ax_click` (jittered+curved mouse approach) and `ax_type` (per-key cadence) — solver plugins+already use it on challenge widgets. Solvers work unchanged; vision needs`<PREFIX>_VISION_KEY` (any OpenAI-compatible endpoint) or the owner-only`~/.config/axuse/secrets.json`. Full lane: `docs/server-mode.md`.diff --git a/docs/server-mode.md b/docs/server-mode.mdindex 155fabb..5e55663 100644--- a/docs/server-mode.md+++ b/docs/server-mode.md@@ -95,6 +95,8 @@ For one-shot / cron-style runs no service is needed — `axuse solve`,| `AXUSE_TIMEZONE` | IANA zone for the Chrome process (`TZ`): drives `Date`/`Intl` in pages AND workers. Set it to the **egress persona's** zone — a UTC server behind a US residential exit is a classic timezone-mismatch score. Validated by `doctor`. || `AXUSE_LANG` | `--accept-lang` for Chrome (verified live: `--lang` alone does **not** move `navigator.language`): sets `navigator.language`/`languages` **and** the `Accept-Language` header. || `AXUSE_FP_CORES` / `AXUSE_FP_MEMORY` | Opt-in persona caps: a 96-core server's honest `hardwareConcurrency` is a persona mismatch for residential-looking traffic. `AXUSE_FP_CORES=8` overrides `navigator.hardwareConcurrency`; `AXUSE_FP_MEMORY` overrides `navigator.deviceMemory` — only where the platform exposes the API (never invents it). Honest hardware stays the default. |+| `AXUSE_PROXY` | Per-process default for the per-launch `proxy` param: `"http://host:port"`, `"socks5://user:pass@host:port"` — one browser instance per spec, credentials answered over CDP. |+| `AXUSE_PROXY_BYPASS` | Chrome proxy bypass list override (default: loopback bypassed so local fixtures work; `"<-loopback>"` includes localhost). || `AXUSE_NO_SANDBOX` | `1` forces `--no-sandbox`. Auto-applied anyway when running as root (containers) — Chrome refuses root otherwise. || `AXUSE_BROWSER` | Path to the browser binary (as on any host). || `AXUSE_PROFILE_DIR` | Where browser profiles go (defaults to the OS temp dir). |@@ -142,6 +144,52 @@ chmod 600 ~/.config/axuse/secrets.json`AXUSE_SECRETS_FILE` overrides the path. A file the group/world can read isrefused (fix: `chmod 600`). Typed failure messages name the exact options.+## Input realism (human-shaped trusted events)++All input is CDP-trusted (`isTrusted` events). The **default** paths stay+fast for agent productivity (bulk insert, direct press). Two lanes add the+dynamics human-sampling systems look for — used automatically by the solver+plugins where it matters:++- **`ax_click {human: true}`** — a jittered curved mouse approach+ (3–6 eased moves, 120–320 ms, 1–3 px overshoot-and-settle, dwell) before+ the press. The captcha-pro Turnstile checkbox click and the+ challenge-handler puzzle/confirm clicks use it.+- **`ax_type {human: true}`** — per-key trusted events (keydown→hold→keyup+ per char, 45–160 ms gaps, 6% thinking pauses, word-boundary slowdowns)+ with real layout key codes — for login/challenge fields where keystroke+ dynamics are sampled. Solver context: `ctx.type({..., human: true})`.+- **Sliders/interstitials** already move humanly (the challenge-handler's+ bezier trajectory engine with ≥8 velocity maxima and sub-8 ms jitter).++## Egress: one browser, one proxy (per-instance routing)++For fleets, each browser instance can leave through its own proxy —+`ax_open {url, proxy: "socks5://user:pass@host:port"}` or `AXUSE_PROXY`+for the process. Credentials never ride the launch flag (Chrome ignores+them there); they are answered over CDP when the proxy challenges (the+full 407 → auth → credentialed-retry round trip is covered by a live test+against a local fixture proxy). Loopback stays bypassed by default so+local fixtures keep working; override with `proxy_bypass` /+`AXUSE_PROXY_BYPASS` (e.g. `"<-loopback>"` for proxy testing). One proxy+spec = one browser instance (different specs never share a launch).++Combine with `AXUSE_TIMEZONE`/`AXUSE_LANG` so the whole persona — IP,+clock, languages — is coherent per instance.++Note: a plugin that re-enables CDP `Fetch` with its own interception+patterns takes over interception for that session and may drop the proxy+auth subscription for those requests (documented integration seam).++## Audio posture (honest, measured)++On a GPU-less, sound-server-less host: `AudioContext` starts `suspended`+and `resume()` before any user gesture stays pending — **exactly what a+real desktop Chrome does** under the default autoplay policy.+`OfflineAudioContext` — the actual audio-fingerprint vector — renders+identically (pure math, verified live). If your flow needs audible audio+after a gesture, run a sound server (`sudo apt install pipewire-pulse`).+## Persona coherence (the parts headless-vs-headed does not cover)Being headed removes the *headless* class of signals. A second class isdiff --git a/src/core/actuate.js b/src/core/actuate.jsindex ff2215e..b4cd64c 100644--- a/src/core/actuate.js+++ b/src/core/actuate.js@@ -136,11 +136,99 @@ export async function pressCombo(ctx, combo) {const BUTTON_MAP = { left: "left", middle: "middle", right: "right", 1: "left", 2: "middle", 3: "right" };-/** Dispatch a trusted click (hover-move → press → release). */-export async function click(ctx, { x, y, button = "left", count = 1, modifiers = 0 }) {+const sleep = (ms) => new Promise((ok) => setTimeout(ok, ms));++/* --------------------------------------------------- human input realism */++/** Key descriptor for one printable character: known layout codes when the+ * key exists on a US layout (letters, digits, space), a plain text-bearing+ * key event otherwise (e.code "" is what IME/dead-key input reports too). */+function charKey(ch) {+ const known = KEYS[ch];+ if (known && known.text === ch) return known;+ return { vk: 0, key: ch, code: "", text: ch };+}++/** Human keystroke plan: per-char { …key, delayMs (approach gap), holdMs+ * (down→up) }. The shape keystroke-dynamics systems sample: mostly+ * 45–160 ms gaps, a few thinking pauses (6%), slightly longer word+ * boundaries and 25–90 ms holds. Pure — injectable randomness for tests. */+export function planHumanKeystrokes(text, { random = Math.random } = {}) {+ const chars = [...String(text)];+ return chars.map((ch, i) => {+ const wordBoundary = i > 0 && chars[i - 1] === " ";+ const thinking = random() < 0.06;+ const base = 45 + random() * 115;+ const delayMs = thinking+ ? 180 + random() * 270+ : Math.round(base + (wordBoundary ? 20 + random() * 40 : 0));+ return { ...charKey(ch), delayMs: i === 0 ? Math.round(base / 2) : delayMs, holdMs: Math.round(25 + random() * 65) };+ });+}++/** Per-key trusted typing (keydown→hold→keyup per char, human cadence).+ * The default type path stays bulk insertText (fast, agent-productivity);+ * this is the lane for challenge/login fields where keystroke dynamics+ * are sampled (DataDome-class). */+export async function humanType(ctx, text, { random = Math.random } = {}) {+ for (const k of planHumanKeystrokes(text, { random })) {+ await sleep(k.delayMs);+ await keyEvent(ctx, "keyDown", k, 0, true);+ await sleep(k.holdMs);+ await keyEvent(ctx, "keyUp", k, 0, false);+ }+}++/** Human mouse approach before a click: a jittered quadratic curve from a+ * random offset (40–130 px away), 3–6 eased moves over 120–320 ms, a+ * 1–3 px overshoot-and-settle and a short dwell — the trajectory shape+ * pointer-dynamics systems expect to see BEFORE a press. Pure planner. */+export function planHumanApproach({ x, y }, { random = Math.random } = {}) {+ const angle = random() * Math.PI * 2;+ const dist = 40 + random() * 90;+ const sx = x + Math.cos(angle) * dist;+ const sy = y + Math.sin(angle) * dist;+ const steps = 3 + Math.floor(random() * 4); // 3–6 moves+ // Control point: mid-point pushed perpendicular — a curved, not laser-straight path.+ const mx = (sx + x) / 2, my = (sy + y) / 2;+ const nx = -(y - sy), ny = x - sx;+ const nlen = Math.hypot(nx, ny) || 1;+ const bend = (random() - 0.5) * 2 * (dist * 0.35);+ const cx = mx + (nx / nlen) * bend, cy = my + (ny / nlen) * bend;+ // Overshoot 1–3 px past the target on the incoming direction, then settle.+ const olen = 1 + random() * 2;+ const ox = x + ((x - cx) / (Math.hypot(x - cx, y - cy) || 1)) * olen;+ const oy = y + ((y - cy) / (Math.hypot(x - cx, y - cy) || 1)) * olen;+ const totalMs = 120 + random() * 200;+ const events = [];+ for (let i = 1; i <= steps; i++) {+ const t = i / steps;+ const ease = 1 - (1 - t) * (1 - t); // ease-out+ const px = (1 - ease) * (1 - ease) * sx + 2 * (1 - ease) * ease * cx + ease * ease * x;+ const py = (1 - ease) * (1 - ease) * sy + 2 * (1 - ease) * ease * cy + ease * ease * y;+ events.push({ x: Math.round(px), y: Math.round(py), dtMs: Math.round((totalMs / steps) * (0.7 + random() * 0.6)) });+ }+ events.push({ x: Math.round(ox), y: Math.round(oy), dtMs: 40 + Math.round(random() * 50) }); // overshoot+ events.push({ x: Math.round(x), y: Math.round(y), dtMs: 30 + Math.round(random() * 40) }); // settle+ return { events, dwellMs: 40 + Math.round(random() * 90), from: { x: Math.round(sx), y: Math.round(sy) } };+}++/** Dispatch a trusted click. `human` adds the approach trajectory first —+ * for challenge widgets (Turnstile checkbox, puzzle tiles) and other+ * pointer-dynamics-sensitive targets. */+export async function click(ctx, { x, y, button = "left", count = 1, modifiers = 0, human = false }) {const b = BUTTON_MAP[button];if (!b) throw new AxError(Codes.COMMAND_REJECTED, `button must be left|middle|right, got "${button}"`);- await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseMoved", x, y, button: "none", modifiers }, ctx.sessionId);+ if (human) {+ const plan = planHumanApproach({ x, y });+ for (const e of plan.events) {+ await sleep(e.dtMs);+ await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: e.x, y: e.y, button: "none", modifiers }, ctx.sessionId);+ }+ await sleep(plan.dwellMs);+ } else {+ await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseMoved", x, y, button: "none", modifiers }, ctx.sessionId);+ }const base = { x, y, button: b, clickCount: count, modifiers };await ctx.conn.send("Input.dispatchMouseEvent", { type: "mousePressed", ...base }, ctx.sessionId);await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseReleased", ...base }, ctx.sessionId);diff --git a/src/core/cdp.js b/src/core/cdp.jsindex 153225c..13451b8 100644--- a/src/core/cdp.js+++ b/src/core/cdp.js@@ -238,6 +238,34 @@ export function locateBrowser(options = {}) {return null;}+/** Parse a proxy spec for one browser instance:+ * http://host:port · socks5://host:port · http://user:pass@host:port ·+ * host:port (scheme defaults to http)+ * → { server: "scheme://host:port", username?, password? } or throws typed+ * (COMMAND_REJECTED — fix the spec, don't retry). Credentials are NEVER+ * embedded in --proxy-server (Chrome ignores them there); they are+ * answered over CDP Fetch.authRequired instead. */+export function parseProxySpec(spec) {+ const raw = String(spec ?? "").trim();+ if (!raw) return null;+ const withScheme = /^[a-z0-9+.-]+:\/\//i.test(raw) ? raw : `http://${raw}`;+ let url;+ try { url = new URL(withScheme); } catch {+ throw new AxError(Codes.COMMAND_REJECTED, `proxy is not a valid URL: "${raw}" (expected e.g. "http://host:port" or "socks5://user:pass@host:port")`, { retry: "never" });+ }+ const scheme = url.protocol.replace(":", "").toLowerCase();+ if (!["http", "https", "socks4", "socks5"].includes(scheme)) {+ throw new AxError(Codes.COMMAND_REJECTED, `proxy scheme must be http, https, socks4 or socks5 (got "${url.protocol}")`, { retry: "never" });+ }+ if (!url.hostname || !url.port || url.port === "0" || Number(url.port) > 65535) {+ throw new AxError(Codes.COMMAND_REJECTED, `proxy needs a host and an explicit port: "${raw}"`, { retry: "never" });+ }+ const out = { server: `${scheme}://${url.hostname}:${url.port}` };+ if (url.username) out.username = decodeURIComponent(url.username);+ if (url.password) out.password = decodeURIComponent(url.password);+ return out;+}+/** A single browser-level WebSocket connection with flat session routing. */export class CdpConnection {/** @param {string} wsUrl */@@ -331,6 +359,7 @@ export class BrowserHandle {this.userDataDir = null;this.ephemeral = true; // false: a persistent A2 profile — never deletedthis.launched = false;+ this.proxyAuth = null; // { username, password } when launched with proxy credsthis.tabs = new Map(); // tab_id → session bookkeeping (owned by controller)}@@ -339,7 +368,7 @@ export class BrowserHandle {* philosophy); headless is opt-in for servers and tests.* `userDataDir` (plugin prerequisite A2, session-vault): reuse a PERSISTENT* profile — not deleted on dispose, only swept by nobody but the owner. */- static async launch({ executablePath, headless = false, extraArgs = [], userDataDir, profileRoot: profileRootOverride } = {}) {+ static async launch({ executablePath, headless = false, extraArgs = [], userDataDir, profileRoot: profileRootOverride, proxy, proxyBypass } = {}) {const exe = executablePath || locateBrowser();if (!exe) {throw new AxError(Codes.LAUNCH_FAILED,@@ -379,6 +408,18 @@ export class BrowserHandle {"--remote-allow-origins=*", // modern Chrome rejects unknown ws Origins];if (headless) args.push("--headless");+ // Per-instance egress (fleet lane): one browser through one proxy.+ // Credentials never ride in --proxy-server (Chrome ignores them there);+ // they are answered over CDP Fetch.authRequired at attach time.+ const proxySpec = parseProxySpec(proxy ?? process.env.AXUSE_PROXY ?? "");+ if (proxySpec) {+ args.push(`--proxy-server=${proxySpec.server}`);+ // Loopback stays bypassed by default (local fixtures, the bridge hub);+ // AXUSE_PROXY_BYPASS / proxy_bypass overrides, e.g. "<-loopback>" to+ // include localhost for proxy testing.+ const bypass = proxyBypass ?? process.env.AXUSE_PROXY_BYPASS;+ if (bypass) args.push(`--proxy-bypass-list=${bypass}`);+ }// Source-level posture fix (verified live, Chrome 151): with remote// debugging active, Chrome otherwise reports navigator.webdriver ===// true in EVERY context (main frames, cross-origin iframes) — a@@ -480,6 +521,7 @@ export class BrowserHandle {h.userDataDir = dir;h.ephemeral = !persistent; // persistent (A2) profiles are never deleted on disposeh.launched = true;+ if (proxySpec?.username) h.proxyAuth = { username: proxySpec.username, password: proxySpec.password ?? "" };h.conn = new CdpConnection(wsUrl);try {await h.conn.connect();@@ -487,6 +529,12 @@ export class BrowserHandle {await h.dispose();throw e;}+ // Proxy credentials over CDP: we subscribe to Fetch.authRequired at the+ // CONNECTION level (one handler for every session) and answer Proxy+ // challenges with the spec's credentials; Server challenges are cancelled+ // (there is no dialog to show on a server, and a modal auth dialog would+ // hang the tab). Registered once, before any tab exists.+ if (h.proxyAuth) installProxyAuthHandler(h);return h;}@@ -544,6 +592,16 @@ export class BrowserHandle {await this.conn.send("Target.setAutoAttach",{ autoAttach: true, waitForDebuggerOnStart: false, flatten: true }, sessionId);}+ // Proxy credentials: subscribe this session to auth challenges. NO+ // patterns key — an explicit empty list disables the subscription on+ // some builds, and an absent key only pauses (which the connection-+ // level router passes through) while still delivering authRequired.+ // (A plugin that later re-enables Fetch with its own patterns keeps+ // interception but may drop this subscription — documented in+ // docs/server-mode.md.)+ if (this.proxyAuth) {+ try { await this.conn.send("Fetch.enable", { handleAuthRequests: true }, sessionId); } catch {}+ }return { targetId, sessionId };}@@ -567,6 +625,45 @@ export class BrowserHandle {}}+/** Connection-level proxy-auth router for one launched browser: Proxy-source+ * challenges get the spec's credentials; Server-source (401 from the site)+ * are cancelled — a server has no dialog to show, and Chrome's native auth+ * dialog would hang the tab. Any Fetch.requestPaused that reaches us (an+ * artifact of the auth subscription on some builds) is passed through —+ * this stays an auth router, never an interceptor. Registered once per+ * connection. */+function installProxyAuthHandler(handle) {+ handle.conn.onEvent((msg) => {+ if (!handle.proxyAuth) return;+ if (msg.method === "Fetch.requestPaused") {+ handle.conn.send("Fetch.continueRequest", { requestId: msg.params?.requestId }, msg.sessionId)+ .catch(() => {});+ return;+ }+ if (msg.method !== "Fetch.authRequired") return;+ const { requestId, authChallenge } = msg.params ?? {};+ const isProxy = authChallenge?.source === "Proxy";+ // Chrome builds disagree on the enum's casing (151: PascalCase+ // "ProvideCredentials"; newer protocol revisions: lowerCamel). Answer in+ // the handle's remembered style; flip it once on an "Unrecognized+ // authChallengeResponse" error and retry — then it sticks.+ const style = handle.proxyAuthStyle ?? "pascal";+ const word = (p, l) => (style === "pascal" ? p : l);+ const answer = isProxy+ ? { response: word("ProvideCredentials", "provideCredentials"), username: handle.proxyAuth.username, password: handle.proxyAuth.password }+ : { response: word("CancelAuth", "cancelAuth") };+ handle.conn.send("Fetch.continueWithAuth", { requestId, authChallengeResponse: answer }, msg.sessionId)+ .catch((e) => {+ if (/authChallengeResponse/i.test(String(e.message)) && style === "pascal") {+ handle.proxyAuthStyle = "lower";+ handle.conn.send("Fetch.continueWithAuth", { requestId, authChallengeResponse: { ...answer, response: isProxy ? "provideCredentials" : "cancelAuth" } }, msg.sessionId)+ .catch(() => { /* request already gone */ });+ }+ /* request already gone — nothing to answer */+ });+ });+}+/** Chrome's helper processes can hold profile files briefly after exit. */async function removeDirWithRetry(dir, attempts = 10) {for (let i = 0; i < attempts; i++) {diff --git a/src/core/commands.js b/src/core/commands.jsindex 9548f30..a7e5cc2 100644--- a/src/core/commands.js+++ b/src/core/commands.js@@ -28,6 +28,8 @@ export const Contract = Object.freeze({params: {url: { t: "string", req: true, desc: "http(s), file, about, data or blob URL." },headless: { t: "boolean", desc: "No window (cdp backend). On a display-less Linux server axuse auto-boots a virtual display and stays headed (anti-bot-safe) — keep headless off for challenge pages; use it only for trusted internal pages." },+ proxy: { t: "string", desc: 'Route this browser through a proxy: "http://host:port", "socks5://user:pass@host:port" (per-launch, credentials answered over CDP; loopback bypassed by default; AXUSE_PROXY otherwise).' },+ proxy_bypass: { t: "string", desc: 'Chrome proxy bypass list (advanced), e.g. "<-loopback>" to include localhost.' },cdp_url: { t: "string", desc: 'Attach instead of launching, e.g. "http://127.0.0.1:9222".' },backend: { t: "enum", values: ["cdp", "extension"], desc: "cdp (default) or extension." },executable_path: { t: "string", desc: "Browser binary (default: auto-detect)." },@@ -101,6 +103,7 @@ export const Contract = Object.freeze({button: { t: "enum", values: ["left", "middle", "right"], desc: "Default left." },count: { t: "number", desc: "2 = double click." },modifiers: { t: "enum", values: ["alt", "ctrl", "meta", "shift"], multi: true, desc: "Held modifiers." },+ human: { t: "boolean", desc: "Human mouse approach first (jittered curved trajectory, overshoot, dwell) — use for challenge widgets and pointer-dynamics-sensitive targets." },state_id: STATE, tab_id: TAB, settle_ms: SETTLE,},observed: "diff",@@ -147,6 +150,7 @@ export const Contract = Object.freeze({text: { t: "string", req: true, desc: "Text to type." },replace: { t: "boolean", desc: "Replace content (default true)." },submit: { t: "boolean", desc: "Press Enter after." },+ human: { t: "boolean", desc: "Per-key trusted events with human cadence (keydown→hold→keyup per char, 45–160 ms gaps, thinking pauses) — for challenge/login fields where keystroke dynamics are sampled; default is fast bulk insert." },state_id: STATE, tab_id: TAB, settle_ms: SETTLE,},observed: "diff",diff --git a/src/core/controller.js b/src/core/controller.jsindex c7b6f6c..43a67d9 100644--- a/src/core/controller.js+++ b/src/core/controller.js@@ -18,7 +18,7 @@ import { captureScreenshot, DEFAULT_BUDGET_BYTES } from "./screenshot.js";import { virtualDisplayStatus } from "./vdisplay.js";import {click as doClick, hover as doHover, drag as doDrag, wheel as doWheel,- insertText, pressCombo, platformShortcut, selectAllCombo,+ insertText, humanType, pressCombo, platformShortcut, selectAllCombo,refResolveExpression, selectOptionExpression, elementInfoExpression, fieldKindExpression, findOptionExpression,sliderValueExpression, settleExpression, waitForExpression, MOD,} from "./actuate.js";@@ -289,7 +289,7 @@ export class Controller {throw new AxError(Codes.COMMAND_REJECTED, `url must start with http(s)://, file://, about:, data: or blob:`, { retry: "never" });}const headless = a.headless ?? /^(1|true)$/i.test(process.env.AXUSE_HEADLESS ?? "");- const browser = await this.ensureBrowser({ headless, cdpUrl: a.cdp_url, executablePath: a.executable_path, backend: a.backend, profileDir: a.profile_dir });+ const browser = await this.ensureBrowser({ headless, cdpUrl: a.cdp_url, executablePath: a.executable_path, backend: a.backend, profileDir: a.profile_dir, proxy: a.proxy, proxyBypass: a.proxy_bypass });const { targetId, sessionId } = await browser.handle.createTab("about:blank");const tab = this.registerTab(browser, targetId, sessionId, "about:blank");const tabId = tab.tabId;@@ -454,7 +454,7 @@ export class Controller {}}- async ensureBrowser({ headless, cdpUrl, executablePath, backend, profileDir }) {+ async ensureBrowser({ headless, cdpUrl, executablePath, backend, profileDir, proxy, proxyBypass }) {// Extension backend: the user's real Chrome through the axuse bridge// (Codex pattern: one browser, the user's own, tabs visible to them).if (backend === "extension") {@@ -474,7 +474,9 @@ export class Controller {this.browserIndex.set(key, browserId);return rec;}- const key = cdpUrl ? `attach:${cdpUrl}` : `launch:${headless ? "headless" : "headed"}:${profileDir ?? "ephemeral"}`;+ const key = cdpUrl+ ? `attach:${cdpUrl}`+ : `launch:${headless ? "headless" : "headed"}:${proxy ?? process.env.AXUSE_PROXY ?? "direct"}:${profileDir ?? "ephemeral"}`;const existing = this.browserIndex.get(key);if (existing) {const b = this.browsers.get(existing);@@ -484,7 +486,7 @@ export class Controller {}const handle = cdpUrl? await BrowserHandle.attach(cdpUrl)- : await BrowserHandle.launch({ headless, executablePath, userDataDir: profileDir });+ : await BrowserHandle.launch({ headless, executablePath, userDataDir: profileDir, proxy, proxyBypass });const browserId = `b${++this.browserSeq}`;const rec = { browserId, handle, tabs: new Set(), key };this.browsers.set(browserId, rec);@@ -766,6 +768,7 @@ export class Controller {button: a.button ?? "left",count: a.count ?? 1,modifiers: this.modifierMask(a.modifiers),+ human: !!a.human,}));const observation = await this.autoObserve(tab, a.settle_ms ?? 150);const popup = await this.popupReport(tab);@@ -961,7 +964,14 @@ export class Controller {if (a.replace !== false) {await this.doSent(tab, () => pressCombo(this.ctx(tab, session), selectAllCombo()));}- await this.doSent(tab, () => insertText(this.ctx(tab, session), String(a.text)));+ // human: per-key trusted events with a human cadence — for challenge and+ // login fields where keystroke dynamics are sampled (DataDome-class).+ // Default stays bulk insertText: fast, and what agent productivity needs.+ if (a.human) {+ await this.doSent(tab, () => humanType(this.ctx(tab, session), String(a.text)));+ } else {+ await this.doSent(tab, () => insertText(this.ctx(tab, session), String(a.text)));+ }if (a.submit) {await this.doSent(tab, () => pressCombo(this.ctx(tab, session), "enter"));}diff --git a/src/plugins/index.js b/src/plugins/index.jsindex 827d6f8..dedde13 100644--- a/src/plugins/index.js+++ b/src/plugins/index.js@@ -25,7 +25,9 @@* ctx.evaluate(expr) → parsed JSON value of the in-page expression* (single-encoded: the raw ax_evaluate JSON string* is parsed for you — a real trap we hit live)- * ctx.click({x,y}|{ref}, {settleMs})+ * ctx.click({x,y}|{ref}, {settleMs, human})+ * → human: true adds a jittered curved mouse+ * approach before the press (challenge widgets)* ctx.screenshot({crop:{x,y,w,h}, quality, budgetBytes})* → { base64, mime, bytes, quality, crop }* (crop-first: never vision-tax a whole page)@@ -213,10 +215,11 @@ export function buildSolveContext(controller, tabId) {return r.value ?? null; // truncated / non-JSON: hand back the raw string}},- async click(target, { settleMs } = {}) {+ async click(target, { settleMs, human } = {}) {const r = await controller.execute("click", {target: "ref" in target ? { ref: target.ref } : { x: target.x, y: target.y },...(settleMs !== undefined ? { settle_ms: settleMs } : {}),+ ...(human !== undefined ? { human } : {}),...(tabId ? { tab_id: tabId } : {}),});return { actionSent: r.action_sent ?? null };@@ -264,12 +267,13 @@ export function buildSolveContext(controller, tabId) {}));return { actionSent: r.action_sent ?? null };},- async type({ target, text, replace, submit, settleMs, tab_id } = {}) {+ async type({ target, text, replace, submit, settleMs, tab_id, human } = {}) {const r = await controller.execute("type", withTab({text: String(text),...(target !== undefined ? { target } : {}),...(replace !== undefined ? { replace } : {}),...(submit !== undefined ? { submit } : {}),+ ...(human !== undefined ? { human } : {}),...(settleMs !== undefined ? { settle_ms: settleMs } : {}),...(tab_id ? { tab_id } : {}),}));@@ -315,15 +319,17 @@ export function buildSolveContext(controller, tabId) {}));},/** Open a URL in a new tab (profileDir: persistent A2 profile). */- async open(url, { headless, profileDir, settleMs, executablePath } = {}) {+ async open(url, { headless, profileDir, settleMs, executablePath, proxy, proxyBypass } = {}) {const r = await controller.execute("open", {url: String(url),...(headless !== undefined ? { headless } : {}),...(profileDir !== undefined ? { profile_dir: profileDir } : {}),...(settleMs !== undefined ? { settle_ms: settleMs } : {}),...(executablePath !== undefined ? { executable_path: executablePath } : {}),+ ...(proxy !== undefined ? { proxy } : {}),+ ...(proxyBypass !== undefined ? { proxy_bypass: proxyBypass } : {}),});- return { tabId: r.tab_id, browserId: r.browser_id, observation: r.observation, dialog: r.dialog ?? null };+ return { tabId: r.tab_id, browserId: r.browser_id, observation: r.observation, dialog: r.dialog ?? null, display: r.display ?? null };},async navigate(url, { settleMs, tab_id } = {}) {const r = await controller.execute("navigate", withTab({diff --git a/test/input-realism.test.js b/test/input-realism.test.jsnew file mode 100644index 0000000..6f142f0--- /dev/null+++ b/test/input-realism.test.js@@ -0,0 +1,180 @@+/**+ * Input realism + per-instance proxy — the lanes massive server operations+ * need beyond the base trusted-input path:+ *+ * - unit (always): the human planners are pure and in-shape (keystroke+ * cadence, approach trajectory), the proxy spec parser is typed.+ * - live (browser): per-key human typing produces the event stream a+ * keystroke-dynamics sampler expects; human clicks arrive on a real+ * trajectory; a browser launched with proxy credentials routes through a+ * local 407-challenging proxy (exercising the CDP Fetch auth path).+ */+import { test } from "node:test";+import assert from "node:assert/strict";+import http from "node:http";+import { planHumanKeystrokes, planHumanApproach } from "../src/core/actuate.js";+import { locateBrowser, parseProxySpec } from "../src/core/cdp.js";+import { Controller } from "../src/core/controller.js";++const hasBrowser = !!locateBrowser();++/* ---------------------------------------------------------------- unit */++test("human typing: per-char plan with human cadence", () => {+ let i = 0;+ const seq = () => [0.5, 0.5, 0.5, 0.5, 0.5][i++ % 5]; // deterministic draws+ const plan = planHumanKeystrokes("ab 12", { random: seq });+ assert.equal(plan.length, 5);+ // known layout keys carry full descriptors (letters, digits, space)+ assert.deepEqual([plan[0].key, plan[0].code, plan[0].vk], ["A", "KeyA", 65]);+ assert.equal(plan[4].key, "2");+ assert.equal(plan[2].text, " ");+ // every char: a pre-gap, a hold, and text for the insert+ for (const [n, k] of plan.entries()) {+ assert.ok(k.delayMs >= 20 && k.delayMs <= 500, `gap ${n}: ${k.delayMs}`);+ assert.ok(k.holdMs >= 25 && k.holdMs <= 90, `hold ${n}: ${k.holdMs}`);+ assert.ok(typeof k.text === "string" && k.text.length === 1);+ }+ // unknown-to-layout chars still carry text (the insert path)+ const umlaut = planHumanKeystrokes("é", { random: () => 0.5 })[0];+ assert.equal(umlaut.text, "é");+ assert.equal(umlaut.code, "");+});++test("human click: approach plan is a curved, settled trajectory", () => {+ let i = 0;+ const seq = () => [0.3, 0.7, 0.5, 0.9, 0.1, 0.4][i++ % 6]; // varying draws → a real bend+ const p = planHumanApproach({ x: 400, y: 300 }, { random: seq });+ // 3–6 curve moves + overshoot + settle+ assert.ok(p.events.length >= 5 && p.events.length <= 8, `events: ${p.events.length}`);+ const last = p.events.at(-1);+ assert.equal(last.x, 400);+ assert.equal(last.y, 300, "settles exactly on the target");+ const [overshoot] = [p.events.at(-2)];+ assert.ok(Math.hypot(overshoot.x - 400, overshoot.y - 300) <= 4, "overshoot is 1–3 px");+ // start point is 40–130 px away+ const d = Math.hypot(p.from.x - 400, p.from.y - 300);+ assert.ok(d >= 40 && d <= 130, `start distance ${d}`);+ // per-move gaps are human-scale, total under ~0.6 s+ const total = p.events.reduce((a, e) => a + e.dtMs, 0) + p.dwellMs;+ assert.ok(total >= 120 && total <= 700, `total ${total}ms`);+ // the path bends (not a laser-straight line)+ const first = p.events[0];+ const straight = Math.abs(+ (last.y - p.from.y) * first.x - (last.x - p.from.x) * first.y + last.x * p.from.y - last.y * p.from.x,+ ) / Math.hypot(last.x - p.from.x, last.y - p.from.y);+ assert.ok(straight > 0.5, `curve deviation ${straight.toFixed(1)}px`);+});++test("proxy: spec parsing is typed and credentials never ride the flag", () => {+ assert.deepEqual(parseProxySpec("http://1.2.3.4:8080"), { server: "http://1.2.3.4:8080" });+ assert.deepEqual(parseProxySpec("1.2.3.4:8080"), { server: "http://1.2.3.4:8080" }, "scheme defaults to http");+ assert.deepEqual(parseProxySpec("socks5://u:p%[email protected]:1080"), {+ server: "socks5://example.com:1080", username: "u", password: "p@x",+ }, "creds split out, percent-decoded");+ assert.equal(parseProxySpec(""), null);+ for (const bad of ["ftp://h:1", "http://host", "http://host:99999", "://x", "not a url at all://"]) {+ assert.throws(() => parseProxySpec(bad), (e) => e.code === "COMMAND_REJECTED" && e.retry === "never", `must reject: ${bad}`);+ }+});++/* ---------------------------------------------------------------- live */++test("human input: typing and clicking land as human-shaped trusted events", { skip: !hasBrowser ? "live-gate-browser: needs any Chromium-family browser" : false }, async (t) => {+ const c = new Controller();+ t.after(() => c.dispose());+ const html = `data:text/html,<input id="t" style="margin:120px">+ <script>+ window.__k = []; window.__m = [];+ const t = document.getElementById("t");+ ["keydown","keyup","input"].forEach((n) => t.addEventListener(n, (e) =>+ window.__k.push({ n, key: e.key, code: e.code || null, trusted: e.isTrusted })));+ ["mousemove","mousedown","mouseup","click"].forEach((n) => document.addEventListener(n, (e) =>+ window.__m.push({ n, x: Math.round(e.clientX), y: Math.round(e.clientY), trusted: e.isTrusted })));+ </script>`;+ await c.execute("open", { url: html, headless: true, settle_ms: 150 });+ const ev = async (e) => JSON.parse((await c.execute("evaluate", { expression: e })).value ?? "null");++ // Human typing: per-char keydown/keyup pairs with layout codes, human pace.+ // (replace: false — the select-all combo would add its own key events.)+ const t0 = Date.now();+ await c.execute("type", { target: { ref: "e1" }, text: "he12", human: true, replace: false, settle_ms: 100 });+ const typingMs = Date.now() - t0;+ const keys = await ev("window.__k");+ const downs = keys.filter((k) => k.n === "keydown");+ const ups = keys.filter((k) => k.n === "keyup");+ assert.equal(downs.length, 4, "one keydown per character");+ assert.equal(ups.length, 4, "one keyup per character");+ assert.ok(downs.every((k) => k.trusted), "all key events trusted");+ assert.deepEqual(downs.map((k) => k.code), ["KeyH", "KeyE", "Digit1", "Digit2"], "US layout codes");+ assert.ok(typingMs >= 4 * 45, `human pace (${typingMs}ms for 4 chars) — not a bulk insert`);+ assert.deepEqual(await ev("document.getElementById('t').value"), "he12");++ // Human click: a trajectory of moves BEFORE the press, ending on the point.+ // (lastIndexOf: the type() focus-click earlier also recorded a press.)+ const input = await ev("(() => { const r = document.getElementById('t').getBoundingClientRect(); return { x: Math.round(r.x + 4), y: Math.round(r.y + 4) }; })()");+ await c.execute("click", { target: { x: input.x, y: input.y }, human: true, settle_ms: 100 });+ const m = await ev("window.__m");+ const pressIdx = m.map((e) => e.n).lastIndexOf("mousedown");+ assert.ok(pressIdx >= 3, `approach trajectory before the press (${pressIdx} prior events)`);+ const moves = m.slice(0, pressIdx).filter((e) => e.n === "mousemove");+ assert.ok(moves.length >= 3, `≥3 approach moves (got ${moves.length})`);+ assert.ok(moves.every((e) => e.trusted), "all approach moves trusted");+ const settled = moves.at(-1);+ assert.ok(Math.hypot(settled.x - input.x, settled.y - input.y) <= 1, "last approach move is on target");+ assert.ok(m[pressIdx + 1].n === "mouseup", "press → release");+});++test("proxy: browser egress routes through a credentialed local proxy (407 → CDP auth)", { skip: !hasBrowser ? "live-gate-browser: needs any Chromium-family browser" : false }, async (t) => {+ // Plain origin server (the "site").+ const site = http.createServer((req, res) => {+ res.writeHead(200, { "content-type": "text/html" });+ res.end("<h1 id=\"ok\">hello-through-proxy</h1>");+ });+ await new Promise((ok) => site.listen(0, "127.0.0.1", ok));+ t.after(() => site.close());+ const sitePort = site.address().port;++ // Minimal forwarding proxy with a 407 Basic challenge: first request has+ // no Proxy-Authorization → 407; Chrome answers via our CDP auth handler →+ // retry with credentials → forwarded.+ const USER = "fleet", PASS = "s3cret";+ const seen = { challenges: 0, authorized: 0 };+ const proxy = http.createServer((req, res) => {+ const auth = req.headers["proxy-authorization"];+ if (!auth) {+ seen.challenges++;+ res.writeHead(407, { "proxy-authenticate": 'Basic realm="axuse-test"' });+ res.end();+ return;+ }+ seen.authorized++;+ const expect = `Basic ${Buffer.from(`${USER}:${PASS}`).toString("base64")}`;+ assert.equal(auth, expect, "credentials answered over CDP arrive as Basic Proxy-Authorization");+ const target = new URL(req.url); // absolute URI (plain HTTP proxying)+ const upstream = http.request(+ { host: target.hostname, port: target.port, path: target.pathname, method: req.method, headers: { ...req.headers, host: target.host } },+ (ur) => { res.writeHead(ur.statusCode, ur.headers); ur.pipe(res); },+ );+ upstream.on("error", () => { res.writeHead(502); res.end(); });+ req.pipe(upstream);+ });+ await new Promise((ok) => proxy.listen(0, "127.0.0.1", ok));+ t.after(() => proxy.close());+ const proxyPort = proxy.address().port;++ const c = new Controller();+ t.after(() => c.dispose());+ const opened = await c.execute("open", {+ url: `http://127.0.0.1:${sitePort}/`,+ headless: true,+ proxy: `http://${USER}:${PASS}@127.0.0.1:${proxyPort}`,+ proxy_bypass: "<-loopback>", // include localhost so the fixture is proxied+ settle_ms: 300,+ });+ // The page rendered THROUGH the proxy (its content, not a Chrome error page).+ assert.match(opened.observation.text, /hello-through-proxy/);+ assert.ok(seen.challenges >= 1, "the proxy challenged (407)");+ assert.ok(seen.authorized >= 1, "the CDP-answered retry carried credentials");+ await c.execute("close", { browser: true });+});diff --git a/test/mcp.test.js b/test/mcp.test.jsindex 0e187ef..5a67ecb 100644--- a/test/mcp.test.js+++ b/test/mcp.test.js@@ -53,25 +53,6 @@ function startServer() {}test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser }, async (t) => {- const srv = startServer();- t.after(async () => { await srv.end(); });-- const init = await srv.call({- jsonrpc: "2.0", id: 1, method: "initialize",- params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "test", version: "0" } },- });- assert.equal(init.result.serverInfo.name, "axuse");- assert.ok(init.result.capabilities.tools);-- srv.proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");-- const list = await srv.call({ jsonrpc: "2.0", id: 2, method: "tools/list" });- const names = list.result.tools.map((x) => x.name);- assert.ok(names.includes("ax_open"));- assert.ok(names.includes("ax_observe"));- assert.ok(names.includes("ax_click"));- assert.ok(names.includes("ax_screenshot"));- assert.ok(names.includes("ax_open_many"));// Desktop tools are advertised when the platform driver is enabled. On// Linux the driver mounts whenever DISPLAY is set AND an X server answers// (CI runs a real Xvfb on :99; dead DISPLAY values must not count).@@ -96,8 +77,40 @@ test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser },const desktopExpected = process.platform === "win32"|| (process.platform === "darwin" && process.env.AXUSE_ENABLE_MACOS_CUA === "1")|| linuxDesktopOk;++ // NOTE on stability: this process probes X liveness AND the spawned+ // server probes it independently at startup. Under full-suite parallel+ // load the two probes can transiently disagree (one X connect beats a+ // 1-2 s timeout, the other does not) — a cross-process race, not a+ // product flake. On disagreement we respawn the server ONCE and re-list;+ // a second disagreement fails.+ const boot = async () => {+ const srv = startServer();+ const init = await srv.call({+ jsonrpc: "2.0", id: 1, method: "initialize",+ params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "test", version: "0" } },+ });+ assert.equal(init.result.serverInfo.name, "axuse");+ assert.ok(init.result.capabilities.tools);+ srv.proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");+ const list = await srv.call({ jsonrpc: "2.0", id: 2, method: "tools/list" });+ return { srv, names: list.result.tools.map((x) => x.name) };+ };+ let { srv, names } = await boot();+ t.after(async () => { await srv.end(); });+ if (!!names.includes("ax_desktop_screenshot") !== desktopExpected && process.platform === "linux" && process.env.DISPLAY) {+ await srv.end();+ ({ srv, names } = await boot());+ t.after(async () => { await srv.end(); });+ }++ assert.ok(names.includes("ax_open"));+ assert.ok(names.includes("ax_observe"));+ assert.ok(names.includes("ax_click"));+ assert.ok(names.includes("ax_screenshot"));+ assert.ok(names.includes("ax_open_many"));if (desktopExpected) {- assert.ok(names.includes("ax_desktop_screenshot"));+ assert.ok(names.includes("ax_desktop_screenshot"), "desktop tools mounted (parent probe saw a live display)");} else {assert.ok(!names.includes("ax_desktop_screenshot"));}diff --git a/test/skip-allowlist.json b/test/skip-allowlist.jsonindex 799e92d..5b33198 100644--- a/test/skip-allowlist.json+++ b/test/skip-allowlist.json@@ -26,6 +26,8 @@"captcha-pro loads and its Turnstile path solves the official demo (test keys)": "live-gate-external: needs AXUSE_TEST_LIVE_CAPTCHA=1, a vision API key, and the public demo","extension bridge: install → native messaging → drive the browser": "live-gate-browser: needs an extension-capable Chromium-family browser","browser flow": "live-gate-browser: needs any Chromium-family browser (auto-detect or AXUSE_BROWSER)",+ "human input: typing and clicking land as human-shaped trusted events": "live-gate-browser: needs any Chromium-family browser (headless hermetic)",+ "proxy: browser egress": "live-gate-browser: needs any Chromium-family browser (local fixture proxy — no external service)","fill: ARIA and native sliders via trusted Arrow keys": "live-gate-browser: needs any Chromium-family browser","fill: foreign-tab refs fail with a hint naming the owning tab": "live-gate-browser: needs any Chromium-family browser","mcp: handshake, tools/list, tools/call end-to-end": "live-gate-browser: needs any Chromium-family browser",