Skip to content

armgit / axuse

Public Imported from GitHub
0

Accessibility-first browser & computer use for AI agents — one MCP server for every harness (Codex, Claude Code, OpenCode, Antigravity). Instruments the DOM/accessibility tree via CDP instead of screenshot loops: diffed snapshots, stable element refs, trusted input, typed errors. Zero dependencies

Server mode round 3 — input realism + per-instance egress

CTOL Team BJbaffd0756ea2
diff --git a/README.md b/README.md
index 5eea57d..fa408f0 100644
--- a/README.md
+++ b/README.md
@@ -87,6 +87,7 @@ element refs like `[ref=e7]`. Each action returns only what changed.
| `AXUSE_DISPLAY_SIZE` | virtual display geometry for server mode (default `1920x1080x24`; `random` = a stable geometry per machine, so a fleet is not all 1920x1080) |
| `AXUSE_TIMEZONE` | IANA `TZ` for the Chrome process — `Date`/`Intl` in pages and workers match the egress persona (validated by `doctor`) |
| `AXUSE_LANG` | `--accept-lang` for Chrome — `navigator.language(s)` and the `Accept-Language` header match the egress persona |
+| `AXUSE_PROXY` | route every launched browser through a proxy (`http://`·`socks5://`, credentials supported — answered over CDP); `ax_open {proxy}` sets it per launch |
| `AXUSE_NO_SANDBOX=1` | force `--no-sandbox` (auto for root/container server runs) |
| `AXUSE_NEUTRALIZE_WEBDRIVER=0` | disable the `navigator.webdriver` posture-fix fallback (the `AutomationControlled` launch flag already fixes it at the source) |
| `AXUSE_HEADLESS=1` | launch the browser without a window — **trusted internal pages only**: challenge/anti-bot pages detect headless first; servers get server mode instead |
diff --git a/SKILL.md b/SKILL.md
index 89be229..32ab478 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -66,8 +66,11 @@ private Xvfb and keeps Chrome fully **headed** on it — never pass
`headless: true` for challenge/anti-bot pages (they check headless first).
The `ax_open` result reports the fact: `display: :N (virtual …)`. Persona
coherence with the egress persona is configured server-side
-(`AXUSE_TIMEZONE`, `AXUSE_LANG`; `navigator.webdriver` stays `false` via the
-`AutomationControlled` launch flag). Solvers work unchanged; vision needs
+(`AXUSE_TIMEZONE`, `AXUSE_LANG`, `AXUSE_PROXY`; `navigator.webdriver`
+stays `false` via the `AutomationControlled` launch flag). For
+dynamics-sensitive targets pass `human: true` on `ax_click` (jittered
+curved mouse approach) and `ax_type` (per-key cadence) — solver plugins
+already use it on challenge widgets. Solvers work unchanged; vision needs
`<PREFIX>_VISION_KEY` (any OpenAI-compatible endpoint) or the owner-only
`~/.config/axuse/secrets.json`. Full lane: `docs/server-mode.md`.
diff --git a/docs/server-mode.md b/docs/server-mode.md
index 155fabb..5e55663 100644
--- a/docs/server-mode.md
+++ b/docs/server-mode.md
@@ -95,6 +95,8 @@ For one-shot / cron-style runs no service is needed — `axuse solve`,
| `AXUSE_TIMEZONE` | IANA zone for the Chrome process (`TZ`): drives `Date`/`Intl` in pages AND workers. Set it to the **egress persona's** zone — a UTC server behind a US residential exit is a classic timezone-mismatch score. Validated by `doctor`. |
| `AXUSE_LANG` | `--accept-lang` for Chrome (verified live: `--lang` alone does **not** move `navigator.language`): sets `navigator.language`/`languages` **and** the `Accept-Language` header. |
| `AXUSE_FP_CORES` / `AXUSE_FP_MEMORY` | Opt-in persona caps: a 96-core server's honest `hardwareConcurrency` is a persona mismatch for residential-looking traffic. `AXUSE_FP_CORES=8` overrides `navigator.hardwareConcurrency`; `AXUSE_FP_MEMORY` overrides `navigator.deviceMemory` — only where the platform exposes the API (never invents it). Honest hardware stays the default. |
+| `AXUSE_PROXY` | Per-process default for the per-launch `proxy` param: `"http://host:port"`, `"socks5://user:pass@host:port"` — one browser instance per spec, credentials answered over CDP. |
+| `AXUSE_PROXY_BYPASS` | Chrome proxy bypass list override (default: loopback bypassed so local fixtures work; `"<-loopback>"` includes localhost). |
| `AXUSE_NO_SANDBOX` | `1` forces `--no-sandbox`. Auto-applied anyway when running as root (containers) — Chrome refuses root otherwise. |
| `AXUSE_BROWSER` | Path to the browser binary (as on any host). |
| `AXUSE_PROFILE_DIR` | Where browser profiles go (defaults to the OS temp dir). |
@@ -142,6 +144,52 @@ chmod 600 ~/.config/axuse/secrets.json
`AXUSE_SECRETS_FILE` overrides the path. A file the group/world can read is
refused (fix: `chmod 600`). Typed failure messages name the exact options.
+## Input realism (human-shaped trusted events)
+
+All input is CDP-trusted (`isTrusted` events). The **default** paths stay
+fast for agent productivity (bulk insert, direct press). Two lanes add the
+dynamics human-sampling systems look for — used automatically by the solver
+plugins where it matters:
+
+- **`ax_click {human: true}`** — a jittered curved mouse approach
+ (3–6 eased moves, 120–320 ms, 1–3 px overshoot-and-settle, dwell) before
+ the press. The captcha-pro Turnstile checkbox click and the
+ challenge-handler puzzle/confirm clicks use it.
+- **`ax_type {human: true}`** — per-key trusted events (keydown→hold→keyup
+ per char, 45–160 ms gaps, 6% thinking pauses, word-boundary slowdowns)
+ with real layout key codes — for login/challenge fields where keystroke
+ dynamics are sampled. Solver context: `ctx.type({..., human: true})`.
+- **Sliders/interstitials** already move humanly (the challenge-handler's
+ bezier trajectory engine with ≥8 velocity maxima and sub-8 ms jitter).
+
+## Egress: one browser, one proxy (per-instance routing)
+
+For fleets, each browser instance can leave through its own proxy —
+`ax_open {url, proxy: "socks5://user:pass@host:port"}` or `AXUSE_PROXY`
+for the process. Credentials never ride the launch flag (Chrome ignores
+them there); they are answered over CDP when the proxy challenges (the
+full 407 → auth → credentialed-retry round trip is covered by a live test
+against a local fixture proxy). Loopback stays bypassed by default so
+local fixtures keep working; override with `proxy_bypass` /
+`AXUSE_PROXY_BYPASS` (e.g. `"<-loopback>"` for proxy testing). One proxy
+spec = one browser instance (different specs never share a launch).
+
+Combine with `AXUSE_TIMEZONE`/`AXUSE_LANG` so the whole persona — IP,
+clock, languages — is coherent per instance.
+
+Note: a plugin that re-enables CDP `Fetch` with its own interception
+patterns takes over interception for that session and may drop the proxy
+auth subscription for those requests (documented integration seam).
+
+## Audio posture (honest, measured)
+
+On a GPU-less, sound-server-less host: `AudioContext` starts `suspended`
+and `resume()` before any user gesture stays pending — **exactly what a
+real desktop Chrome does** under the default autoplay policy.
+`OfflineAudioContext` — the actual audio-fingerprint vector — renders
+identically (pure math, verified live). If your flow needs audible audio
+after a gesture, run a sound server (`sudo apt install pipewire-pulse`).
+
## Persona coherence (the parts headless-vs-headed does not cover)
Being headed removes the *headless* class of signals. A second class is
diff --git a/src/core/actuate.js b/src/core/actuate.js
index ff2215e..b4cd64c 100644
--- a/src/core/actuate.js
+++ b/src/core/actuate.js
@@ -136,11 +136,99 @@ export async function pressCombo(ctx, combo) {
const BUTTON_MAP = { left: "left", middle: "middle", right: "right", 1: "left", 2: "middle", 3: "right" };
-/** Dispatch a trusted click (hover-move → press → release). */
-export async function click(ctx, { x, y, button = "left", count = 1, modifiers = 0 }) {
+const sleep = (ms) => new Promise((ok) => setTimeout(ok, ms));
+
+/* --------------------------------------------------- human input realism */
+
+/** Key descriptor for one printable character: known layout codes when the
+ * key exists on a US layout (letters, digits, space), a plain text-bearing
+ * key event otherwise (e.code "" is what IME/dead-key input reports too). */
+function charKey(ch) {
+ const known = KEYS[ch];
+ if (known && known.text === ch) return known;
+ return { vk: 0, key: ch, code: "", text: ch };
+}
+
+/** Human keystroke plan: per-char { …key, delayMs (approach gap), holdMs
+ * (down→up) }. The shape keystroke-dynamics systems sample: mostly
+ * 45–160 ms gaps, a few thinking pauses (6%), slightly longer word
+ * boundaries and 25–90 ms holds. Pure — injectable randomness for tests. */
+export function planHumanKeystrokes(text, { random = Math.random } = {}) {
+ const chars = [...String(text)];
+ return chars.map((ch, i) => {
+ const wordBoundary = i > 0 && chars[i - 1] === " ";
+ const thinking = random() < 0.06;
+ const base = 45 + random() * 115;
+ const delayMs = thinking
+ ? 180 + random() * 270
+ : Math.round(base + (wordBoundary ? 20 + random() * 40 : 0));
+ return { ...charKey(ch), delayMs: i === 0 ? Math.round(base / 2) : delayMs, holdMs: Math.round(25 + random() * 65) };
+ });
+}
+
+/** Per-key trusted typing (keydown→hold→keyup per char, human cadence).
+ * The default type path stays bulk insertText (fast, agent-productivity);
+ * this is the lane for challenge/login fields where keystroke dynamics
+ * are sampled (DataDome-class). */
+export async function humanType(ctx, text, { random = Math.random } = {}) {
+ for (const k of planHumanKeystrokes(text, { random })) {
+ await sleep(k.delayMs);
+ await keyEvent(ctx, "keyDown", k, 0, true);
+ await sleep(k.holdMs);
+ await keyEvent(ctx, "keyUp", k, 0, false);
+ }
+}
+
+/** Human mouse approach before a click: a jittered quadratic curve from a
+ * random offset (40–130 px away), 3–6 eased moves over 120–320 ms, a
+ * 1–3 px overshoot-and-settle and a short dwell — the trajectory shape
+ * pointer-dynamics systems expect to see BEFORE a press. Pure planner. */
+export function planHumanApproach({ x, y }, { random = Math.random } = {}) {
+ const angle = random() * Math.PI * 2;
+ const dist = 40 + random() * 90;
+ const sx = x + Math.cos(angle) * dist;
+ const sy = y + Math.sin(angle) * dist;
+ const steps = 3 + Math.floor(random() * 4); // 3–6 moves
+ // Control point: mid-point pushed perpendicular — a curved, not laser-straight path.
+ const mx = (sx + x) / 2, my = (sy + y) / 2;
+ const nx = -(y - sy), ny = x - sx;
+ const nlen = Math.hypot(nx, ny) || 1;
+ const bend = (random() - 0.5) * 2 * (dist * 0.35);
+ const cx = mx + (nx / nlen) * bend, cy = my + (ny / nlen) * bend;
+ // Overshoot 1–3 px past the target on the incoming direction, then settle.
+ const olen = 1 + random() * 2;
+ const ox = x + ((x - cx) / (Math.hypot(x - cx, y - cy) || 1)) * olen;
+ const oy = y + ((y - cy) / (Math.hypot(x - cx, y - cy) || 1)) * olen;
+ const totalMs = 120 + random() * 200;
+ const events = [];
+ for (let i = 1; i <= steps; i++) {
+ const t = i / steps;
+ const ease = 1 - (1 - t) * (1 - t); // ease-out
+ const px = (1 - ease) * (1 - ease) * sx + 2 * (1 - ease) * ease * cx + ease * ease * x;
+ const py = (1 - ease) * (1 - ease) * sy + 2 * (1 - ease) * ease * cy + ease * ease * y;
+ events.push({ x: Math.round(px), y: Math.round(py), dtMs: Math.round((totalMs / steps) * (0.7 + random() * 0.6)) });
+ }
+ events.push({ x: Math.round(ox), y: Math.round(oy), dtMs: 40 + Math.round(random() * 50) }); // overshoot
+ events.push({ x: Math.round(x), y: Math.round(y), dtMs: 30 + Math.round(random() * 40) }); // settle
+ return { events, dwellMs: 40 + Math.round(random() * 90), from: { x: Math.round(sx), y: Math.round(sy) } };
+}
+
+/** Dispatch a trusted click. `human` adds the approach trajectory first —
+ * for challenge widgets (Turnstile checkbox, puzzle tiles) and other
+ * pointer-dynamics-sensitive targets. */
+export async function click(ctx, { x, y, button = "left", count = 1, modifiers = 0, human = false }) {
const b = BUTTON_MAP[button];
if (!b) throw new AxError(Codes.COMMAND_REJECTED, `button must be left|middle|right, got "${button}"`);
- await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseMoved", x, y, button: "none", modifiers }, ctx.sessionId);
+ if (human) {
+ const plan = planHumanApproach({ x, y });
+ for (const e of plan.events) {
+ await sleep(e.dtMs);
+ await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: e.x, y: e.y, button: "none", modifiers }, ctx.sessionId);
+ }
+ await sleep(plan.dwellMs);
+ } else {
+ await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseMoved", x, y, button: "none", modifiers }, ctx.sessionId);
+ }
const base = { x, y, button: b, clickCount: count, modifiers };
await ctx.conn.send("Input.dispatchMouseEvent", { type: "mousePressed", ...base }, ctx.sessionId);
await ctx.conn.send("Input.dispatchMouseEvent", { type: "mouseReleased", ...base }, ctx.sessionId);
diff --git a/src/core/cdp.js b/src/core/cdp.js
index 153225c..13451b8 100644
--- a/src/core/cdp.js
+++ b/src/core/cdp.js
@@ -238,6 +238,34 @@ export function locateBrowser(options = {}) {
return null;
}
+/** Parse a proxy spec for one browser instance:
+ * http://host:port · socks5://host:port · http://user:pass@host:port ·
+ * host:port (scheme defaults to http)
+ * → { server: "scheme://host:port", username?, password? } or throws typed
+ * (COMMAND_REJECTED — fix the spec, don't retry). Credentials are NEVER
+ * embedded in --proxy-server (Chrome ignores them there); they are
+ * answered over CDP Fetch.authRequired instead. */
+export function parseProxySpec(spec) {
+ const raw = String(spec ?? "").trim();
+ if (!raw) return null;
+ const withScheme = /^[a-z0-9+.-]+:\/\//i.test(raw) ? raw : `http://${raw}`;
+ let url;
+ try { url = new URL(withScheme); } catch {
+ throw new AxError(Codes.COMMAND_REJECTED, `proxy is not a valid URL: "${raw}" (expected e.g. "http://host:port" or "socks5://user:pass@host:port")`, { retry: "never" });
+ }
+ const scheme = url.protocol.replace(":", "").toLowerCase();
+ if (!["http", "https", "socks4", "socks5"].includes(scheme)) {
+ throw new AxError(Codes.COMMAND_REJECTED, `proxy scheme must be http, https, socks4 or socks5 (got "${url.protocol}")`, { retry: "never" });
+ }
+ if (!url.hostname || !url.port || url.port === "0" || Number(url.port) > 65535) {
+ throw new AxError(Codes.COMMAND_REJECTED, `proxy needs a host and an explicit port: "${raw}"`, { retry: "never" });
+ }
+ const out = { server: `${scheme}://${url.hostname}:${url.port}` };
+ if (url.username) out.username = decodeURIComponent(url.username);
+ if (url.password) out.password = decodeURIComponent(url.password);
+ return out;
+}
+
/** A single browser-level WebSocket connection with flat session routing. */
export class CdpConnection {
/** @param {string} wsUrl */
@@ -331,6 +359,7 @@ export class BrowserHandle {
this.userDataDir = null;
this.ephemeral = true; // false: a persistent A2 profile — never deleted
this.launched = false;
+ this.proxyAuth = null; // { username, password } when launched with proxy creds
this.tabs = new Map(); // tab_id → session bookkeeping (owned by controller)
}
@@ -339,7 +368,7 @@ export class BrowserHandle {
* philosophy); headless is opt-in for servers and tests.
* `userDataDir` (plugin prerequisite A2, session-vault): reuse a PERSISTENT
* profile — not deleted on dispose, only swept by nobody but the owner. */
- static async launch({ executablePath, headless = false, extraArgs = [], userDataDir, profileRoot: profileRootOverride } = {}) {
+ static async launch({ executablePath, headless = false, extraArgs = [], userDataDir, profileRoot: profileRootOverride, proxy, proxyBypass } = {}) {
const exe = executablePath || locateBrowser();
if (!exe) {
throw new AxError(Codes.LAUNCH_FAILED,
@@ -379,6 +408,18 @@ export class BrowserHandle {
"--remote-allow-origins=*", // modern Chrome rejects unknown ws Origins
];
if (headless) args.push("--headless");
+ // Per-instance egress (fleet lane): one browser through one proxy.
+ // Credentials never ride in --proxy-server (Chrome ignores them there);
+ // they are answered over CDP Fetch.authRequired at attach time.
+ const proxySpec = parseProxySpec(proxy ?? process.env.AXUSE_PROXY ?? "");
+ if (proxySpec) {
+ args.push(`--proxy-server=${proxySpec.server}`);
+ // Loopback stays bypassed by default (local fixtures, the bridge hub);
+ // AXUSE_PROXY_BYPASS / proxy_bypass overrides, e.g. "<-loopback>" to
+ // include localhost for proxy testing.
+ const bypass = proxyBypass ?? process.env.AXUSE_PROXY_BYPASS;
+ if (bypass) args.push(`--proxy-bypass-list=${bypass}`);
+ }
// Source-level posture fix (verified live, Chrome 151): with remote
// debugging active, Chrome otherwise reports navigator.webdriver ===
// true in EVERY context (main frames, cross-origin iframes) — a
@@ -480,6 +521,7 @@ export class BrowserHandle {
h.userDataDir = dir;
h.ephemeral = !persistent; // persistent (A2) profiles are never deleted on dispose
h.launched = true;
+ if (proxySpec?.username) h.proxyAuth = { username: proxySpec.username, password: proxySpec.password ?? "" };
h.conn = new CdpConnection(wsUrl);
try {
await h.conn.connect();
@@ -487,6 +529,12 @@ export class BrowserHandle {
await h.dispose();
throw e;
}
+ // Proxy credentials over CDP: we subscribe to Fetch.authRequired at the
+ // CONNECTION level (one handler for every session) and answer Proxy
+ // challenges with the spec's credentials; Server challenges are cancelled
+ // (there is no dialog to show on a server, and a modal auth dialog would
+ // hang the tab). Registered once, before any tab exists.
+ if (h.proxyAuth) installProxyAuthHandler(h);
return h;
}
@@ -544,6 +592,16 @@ export class BrowserHandle {
await this.conn.send("Target.setAutoAttach",
{ autoAttach: true, waitForDebuggerOnStart: false, flatten: true }, sessionId);
}
+ // Proxy credentials: subscribe this session to auth challenges. NO
+ // patterns key — an explicit empty list disables the subscription on
+ // some builds, and an absent key only pauses (which the connection-
+ // level router passes through) while still delivering authRequired.
+ // (A plugin that later re-enables Fetch with its own patterns keeps
+ // interception but may drop this subscription — documented in
+ // docs/server-mode.md.)
+ if (this.proxyAuth) {
+ try { await this.conn.send("Fetch.enable", { handleAuthRequests: true }, sessionId); } catch {}
+ }
return { targetId, sessionId };
}
@@ -567,6 +625,45 @@ export class BrowserHandle {
}
}
+/** Connection-level proxy-auth router for one launched browser: Proxy-source
+ * challenges get the spec's credentials; Server-source (401 from the site)
+ * are cancelled — a server has no dialog to show, and Chrome's native auth
+ * dialog would hang the tab. Any Fetch.requestPaused that reaches us (an
+ * artifact of the auth subscription on some builds) is passed through —
+ * this stays an auth router, never an interceptor. Registered once per
+ * connection. */
+function installProxyAuthHandler(handle) {
+ handle.conn.onEvent((msg) => {
+ if (!handle.proxyAuth) return;
+ if (msg.method === "Fetch.requestPaused") {
+ handle.conn.send("Fetch.continueRequest", { requestId: msg.params?.requestId }, msg.sessionId)
+ .catch(() => {});
+ return;
+ }
+ if (msg.method !== "Fetch.authRequired") return;
+ const { requestId, authChallenge } = msg.params ?? {};
+ const isProxy = authChallenge?.source === "Proxy";
+ // Chrome builds disagree on the enum's casing (151: PascalCase
+ // "ProvideCredentials"; newer protocol revisions: lowerCamel). Answer in
+ // the handle's remembered style; flip it once on an "Unrecognized
+ // authChallengeResponse" error and retry — then it sticks.
+ const style = handle.proxyAuthStyle ?? "pascal";
+ const word = (p, l) => (style === "pascal" ? p : l);
+ const answer = isProxy
+ ? { response: word("ProvideCredentials", "provideCredentials"), username: handle.proxyAuth.username, password: handle.proxyAuth.password }
+ : { response: word("CancelAuth", "cancelAuth") };
+ handle.conn.send("Fetch.continueWithAuth", { requestId, authChallengeResponse: answer }, msg.sessionId)
+ .catch((e) => {
+ if (/authChallengeResponse/i.test(String(e.message)) && style === "pascal") {
+ handle.proxyAuthStyle = "lower";
+ handle.conn.send("Fetch.continueWithAuth", { requestId, authChallengeResponse: { ...answer, response: isProxy ? "provideCredentials" : "cancelAuth" } }, msg.sessionId)
+ .catch(() => { /* request already gone */ });
+ }
+ /* request already gone — nothing to answer */
+ });
+ });
+}
+
/** Chrome's helper processes can hold profile files briefly after exit. */
async function removeDirWithRetry(dir, attempts = 10) {
for (let i = 0; i < attempts; i++) {
diff --git a/src/core/commands.js b/src/core/commands.js
index 9548f30..a7e5cc2 100644
--- a/src/core/commands.js
+++ b/src/core/commands.js
@@ -28,6 +28,8 @@ export const Contract = Object.freeze({
params: {
url: { t: "string", req: true, desc: "http(s), file, about, data or blob URL." },
headless: { t: "boolean", desc: "No window (cdp backend). On a display-less Linux server axuse auto-boots a virtual display and stays headed (anti-bot-safe) — keep headless off for challenge pages; use it only for trusted internal pages." },
+ proxy: { t: "string", desc: 'Route this browser through a proxy: "http://host:port", "socks5://user:pass@host:port" (per-launch, credentials answered over CDP; loopback bypassed by default; AXUSE_PROXY otherwise).' },
+ proxy_bypass: { t: "string", desc: 'Chrome proxy bypass list (advanced), e.g. "<-loopback>" to include localhost.' },
cdp_url: { t: "string", desc: 'Attach instead of launching, e.g. "http://127.0.0.1:9222".' },
backend: { t: "enum", values: ["cdp", "extension"], desc: "cdp (default) or extension." },
executable_path: { t: "string", desc: "Browser binary (default: auto-detect)." },
@@ -101,6 +103,7 @@ export const Contract = Object.freeze({
button: { t: "enum", values: ["left", "middle", "right"], desc: "Default left." },
count: { t: "number", desc: "2 = double click." },
modifiers: { t: "enum", values: ["alt", "ctrl", "meta", "shift"], multi: true, desc: "Held modifiers." },
+ human: { t: "boolean", desc: "Human mouse approach first (jittered curved trajectory, overshoot, dwell) — use for challenge widgets and pointer-dynamics-sensitive targets." },
state_id: STATE, tab_id: TAB, settle_ms: SETTLE,
},
observed: "diff",
@@ -147,6 +150,7 @@ export const Contract = Object.freeze({
text: { t: "string", req: true, desc: "Text to type." },
replace: { t: "boolean", desc: "Replace content (default true)." },
submit: { t: "boolean", desc: "Press Enter after." },
+ human: { t: "boolean", desc: "Per-key trusted events with human cadence (keydown→hold→keyup per char, 45–160 ms gaps, thinking pauses) — for challenge/login fields where keystroke dynamics are sampled; default is fast bulk insert." },
state_id: STATE, tab_id: TAB, settle_ms: SETTLE,
},
observed: "diff",
diff --git a/src/core/controller.js b/src/core/controller.js
index c7b6f6c..43a67d9 100644
--- a/src/core/controller.js
+++ b/src/core/controller.js
@@ -18,7 +18,7 @@ import { captureScreenshot, DEFAULT_BUDGET_BYTES } from "./screenshot.js";
import { virtualDisplayStatus } from "./vdisplay.js";
import {
click as doClick, hover as doHover, drag as doDrag, wheel as doWheel,
- insertText, pressCombo, platformShortcut, selectAllCombo,
+ insertText, humanType, pressCombo, platformShortcut, selectAllCombo,
refResolveExpression, selectOptionExpression, elementInfoExpression, fieldKindExpression, findOptionExpression,
sliderValueExpression, settleExpression, waitForExpression, MOD,
} from "./actuate.js";
@@ -289,7 +289,7 @@ export class Controller {
throw new AxError(Codes.COMMAND_REJECTED, `url must start with http(s)://, file://, about:, data: or blob:`, { retry: "never" });
}
const headless = a.headless ?? /^(1|true)$/i.test(process.env.AXUSE_HEADLESS ?? "");
- const browser = await this.ensureBrowser({ headless, cdpUrl: a.cdp_url, executablePath: a.executable_path, backend: a.backend, profileDir: a.profile_dir });
+ const browser = await this.ensureBrowser({ headless, cdpUrl: a.cdp_url, executablePath: a.executable_path, backend: a.backend, profileDir: a.profile_dir, proxy: a.proxy, proxyBypass: a.proxy_bypass });
const { targetId, sessionId } = await browser.handle.createTab("about:blank");
const tab = this.registerTab(browser, targetId, sessionId, "about:blank");
const tabId = tab.tabId;
@@ -454,7 +454,7 @@ export class Controller {
}
}
- async ensureBrowser({ headless, cdpUrl, executablePath, backend, profileDir }) {
+ async ensureBrowser({ headless, cdpUrl, executablePath, backend, profileDir, proxy, proxyBypass }) {
// Extension backend: the user's real Chrome through the axuse bridge
// (Codex pattern: one browser, the user's own, tabs visible to them).
if (backend === "extension") {
@@ -474,7 +474,9 @@ export class Controller {
this.browserIndex.set(key, browserId);
return rec;
}
- const key = cdpUrl ? `attach:${cdpUrl}` : `launch:${headless ? "headless" : "headed"}:${profileDir ?? "ephemeral"}`;
+ const key = cdpUrl
+ ? `attach:${cdpUrl}`
+ : `launch:${headless ? "headless" : "headed"}:${proxy ?? process.env.AXUSE_PROXY ?? "direct"}:${profileDir ?? "ephemeral"}`;
const existing = this.browserIndex.get(key);
if (existing) {
const b = this.browsers.get(existing);
@@ -484,7 +486,7 @@ export class Controller {
}
const handle = cdpUrl
? await BrowserHandle.attach(cdpUrl)
- : await BrowserHandle.launch({ headless, executablePath, userDataDir: profileDir });
+ : await BrowserHandle.launch({ headless, executablePath, userDataDir: profileDir, proxy, proxyBypass });
const browserId = `b${++this.browserSeq}`;
const rec = { browserId, handle, tabs: new Set(), key };
this.browsers.set(browserId, rec);
@@ -766,6 +768,7 @@ export class Controller {
button: a.button ?? "left",
count: a.count ?? 1,
modifiers: this.modifierMask(a.modifiers),
+ human: !!a.human,
}));
const observation = await this.autoObserve(tab, a.settle_ms ?? 150);
const popup = await this.popupReport(tab);
@@ -961,7 +964,14 @@ export class Controller {
if (a.replace !== false) {
await this.doSent(tab, () => pressCombo(this.ctx(tab, session), selectAllCombo()));
}
- await this.doSent(tab, () => insertText(this.ctx(tab, session), String(a.text)));
+ // human: per-key trusted events with a human cadence — for challenge and
+ // login fields where keystroke dynamics are sampled (DataDome-class).
+ // Default stays bulk insertText: fast, and what agent productivity needs.
+ if (a.human) {
+ await this.doSent(tab, () => humanType(this.ctx(tab, session), String(a.text)));
+ } else {
+ await this.doSent(tab, () => insertText(this.ctx(tab, session), String(a.text)));
+ }
if (a.submit) {
await this.doSent(tab, () => pressCombo(this.ctx(tab, session), "enter"));
}
diff --git a/src/plugins/index.js b/src/plugins/index.js
index 827d6f8..dedde13 100644
--- a/src/plugins/index.js
+++ b/src/plugins/index.js
@@ -25,7 +25,9 @@
* ctx.evaluate(expr) → parsed JSON value of the in-page expression
* (single-encoded: the raw ax_evaluate JSON string
* is parsed for you — a real trap we hit live)
- * ctx.click({x,y}|{ref}, {settleMs})
+ * ctx.click({x,y}|{ref}, {settleMs, human})
+ * → human: true adds a jittered curved mouse
+ * approach before the press (challenge widgets)
* ctx.screenshot({crop:{x,y,w,h}, quality, budgetBytes})
* → { base64, mime, bytes, quality, crop }
* (crop-first: never vision-tax a whole page)
@@ -213,10 +215,11 @@ export function buildSolveContext(controller, tabId) {
return r.value ?? null; // truncated / non-JSON: hand back the raw string
}
},
- async click(target, { settleMs } = {}) {
+ async click(target, { settleMs, human } = {}) {
const r = await controller.execute("click", {
target: "ref" in target ? { ref: target.ref } : { x: target.x, y: target.y },
...(settleMs !== undefined ? { settle_ms: settleMs } : {}),
+ ...(human !== undefined ? { human } : {}),
...(tabId ? { tab_id: tabId } : {}),
});
return { actionSent: r.action_sent ?? null };
@@ -264,12 +267,13 @@ export function buildSolveContext(controller, tabId) {
}));
return { actionSent: r.action_sent ?? null };
},
- async type({ target, text, replace, submit, settleMs, tab_id } = {}) {
+ async type({ target, text, replace, submit, settleMs, tab_id, human } = {}) {
const r = await controller.execute("type", withTab({
text: String(text),
...(target !== undefined ? { target } : {}),
...(replace !== undefined ? { replace } : {}),
...(submit !== undefined ? { submit } : {}),
+ ...(human !== undefined ? { human } : {}),
...(settleMs !== undefined ? { settle_ms: settleMs } : {}),
...(tab_id ? { tab_id } : {}),
}));
@@ -315,15 +319,17 @@ export function buildSolveContext(controller, tabId) {
}));
},
/** Open a URL in a new tab (profileDir: persistent A2 profile). */
- async open(url, { headless, profileDir, settleMs, executablePath } = {}) {
+ async open(url, { headless, profileDir, settleMs, executablePath, proxy, proxyBypass } = {}) {
const r = await controller.execute("open", {
url: String(url),
...(headless !== undefined ? { headless } : {}),
...(profileDir !== undefined ? { profile_dir: profileDir } : {}),
...(settleMs !== undefined ? { settle_ms: settleMs } : {}),
...(executablePath !== undefined ? { executable_path: executablePath } : {}),
+ ...(proxy !== undefined ? { proxy } : {}),
+ ...(proxyBypass !== undefined ? { proxy_bypass: proxyBypass } : {}),
});
- return { tabId: r.tab_id, browserId: r.browser_id, observation: r.observation, dialog: r.dialog ?? null };
+ return { tabId: r.tab_id, browserId: r.browser_id, observation: r.observation, dialog: r.dialog ?? null, display: r.display ?? null };
},
async navigate(url, { settleMs, tab_id } = {}) {
const r = await controller.execute("navigate", withTab({
diff --git a/test/input-realism.test.js b/test/input-realism.test.js
new file mode 100644
index 0000000..6f142f0
--- /dev/null
+++ b/test/input-realism.test.js
@@ -0,0 +1,180 @@
+/**
+ * Input realism + per-instance proxy — the lanes massive server operations
+ * need beyond the base trusted-input path:
+ *
+ * - unit (always): the human planners are pure and in-shape (keystroke
+ * cadence, approach trajectory), the proxy spec parser is typed.
+ * - live (browser): per-key human typing produces the event stream a
+ * keystroke-dynamics sampler expects; human clicks arrive on a real
+ * trajectory; a browser launched with proxy credentials routes through a
+ * local 407-challenging proxy (exercising the CDP Fetch auth path).
+ */
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import http from "node:http";
+import { planHumanKeystrokes, planHumanApproach } from "../src/core/actuate.js";
+import { locateBrowser, parseProxySpec } from "../src/core/cdp.js";
+import { Controller } from "../src/core/controller.js";
+
+const hasBrowser = !!locateBrowser();
+
+/* ---------------------------------------------------------------- unit */
+
+test("human typing: per-char plan with human cadence", () => {
+ let i = 0;
+ const seq = () => [0.5, 0.5, 0.5, 0.5, 0.5][i++ % 5]; // deterministic draws
+ const plan = planHumanKeystrokes("ab 12", { random: seq });
+ assert.equal(plan.length, 5);
+ // known layout keys carry full descriptors (letters, digits, space)
+ assert.deepEqual([plan[0].key, plan[0].code, plan[0].vk], ["A", "KeyA", 65]);
+ assert.equal(plan[4].key, "2");
+ assert.equal(plan[2].text, " ");
+ // every char: a pre-gap, a hold, and text for the insert
+ for (const [n, k] of plan.entries()) {
+ assert.ok(k.delayMs >= 20 && k.delayMs <= 500, `gap ${n}: ${k.delayMs}`);
+ assert.ok(k.holdMs >= 25 && k.holdMs <= 90, `hold ${n}: ${k.holdMs}`);
+ assert.ok(typeof k.text === "string" && k.text.length === 1);
+ }
+ // unknown-to-layout chars still carry text (the insert path)
+ const umlaut = planHumanKeystrokes("é", { random: () => 0.5 })[0];
+ assert.equal(umlaut.text, "é");
+ assert.equal(umlaut.code, "");
+});
+
+test("human click: approach plan is a curved, settled trajectory", () => {
+ let i = 0;
+ const seq = () => [0.3, 0.7, 0.5, 0.9, 0.1, 0.4][i++ % 6]; // varying draws → a real bend
+ const p = planHumanApproach({ x: 400, y: 300 }, { random: seq });
+ // 3–6 curve moves + overshoot + settle
+ assert.ok(p.events.length >= 5 && p.events.length <= 8, `events: ${p.events.length}`);
+ const last = p.events.at(-1);
+ assert.equal(last.x, 400);
+ assert.equal(last.y, 300, "settles exactly on the target");
+ const [overshoot] = [p.events.at(-2)];
+ assert.ok(Math.hypot(overshoot.x - 400, overshoot.y - 300) <= 4, "overshoot is 1–3 px");
+ // start point is 40–130 px away
+ const d = Math.hypot(p.from.x - 400, p.from.y - 300);
+ assert.ok(d >= 40 && d <= 130, `start distance ${d}`);
+ // per-move gaps are human-scale, total under ~0.6 s
+ const total = p.events.reduce((a, e) => a + e.dtMs, 0) + p.dwellMs;
+ assert.ok(total >= 120 && total <= 700, `total ${total}ms`);
+ // the path bends (not a laser-straight line)
+ const first = p.events[0];
+ const straight = Math.abs(
+ (last.y - p.from.y) * first.x - (last.x - p.from.x) * first.y + last.x * p.from.y - last.y * p.from.x,
+ ) / Math.hypot(last.x - p.from.x, last.y - p.from.y);
+ assert.ok(straight > 0.5, `curve deviation ${straight.toFixed(1)}px`);
+});
+
+test("proxy: spec parsing is typed and credentials never ride the flag", () => {
+ assert.deepEqual(parseProxySpec("http://1.2.3.4:8080"), { server: "http://1.2.3.4:8080" });
+ assert.deepEqual(parseProxySpec("1.2.3.4:8080"), { server: "http://1.2.3.4:8080" }, "scheme defaults to http");
+ assert.deepEqual(parseProxySpec("socks5://u:p%[email protected]:1080"), {
+ server: "socks5://example.com:1080", username: "u", password: "p@x",
+ }, "creds split out, percent-decoded");
+ assert.equal(parseProxySpec(""), null);
+ for (const bad of ["ftp://h:1", "http://host", "http://host:99999", "://x", "not a url at all://"]) {
+ assert.throws(() => parseProxySpec(bad), (e) => e.code === "COMMAND_REJECTED" && e.retry === "never", `must reject: ${bad}`);
+ }
+});
+
+/* ---------------------------------------------------------------- live */
+
+test("human input: typing and clicking land as human-shaped trusted events", { skip: !hasBrowser ? "live-gate-browser: needs any Chromium-family browser" : false }, async (t) => {
+ const c = new Controller();
+ t.after(() => c.dispose());
+ const html = `data:text/html,<input id="t" style="margin:120px">
+ <script>
+ window.__k = []; window.__m = [];
+ const t = document.getElementById("t");
+ ["keydown","keyup","input"].forEach((n) => t.addEventListener(n, (e) =>
+ window.__k.push({ n, key: e.key, code: e.code || null, trusted: e.isTrusted })));
+ ["mousemove","mousedown","mouseup","click"].forEach((n) => document.addEventListener(n, (e) =>
+ window.__m.push({ n, x: Math.round(e.clientX), y: Math.round(e.clientY), trusted: e.isTrusted })));
+ </script>`;
+ await c.execute("open", { url: html, headless: true, settle_ms: 150 });
+ const ev = async (e) => JSON.parse((await c.execute("evaluate", { expression: e })).value ?? "null");
+
+ // Human typing: per-char keydown/keyup pairs with layout codes, human pace.
+ // (replace: false — the select-all combo would add its own key events.)
+ const t0 = Date.now();
+ await c.execute("type", { target: { ref: "e1" }, text: "he12", human: true, replace: false, settle_ms: 100 });
+ const typingMs = Date.now() - t0;
+ const keys = await ev("window.__k");
+ const downs = keys.filter((k) => k.n === "keydown");
+ const ups = keys.filter((k) => k.n === "keyup");
+ assert.equal(downs.length, 4, "one keydown per character");
+ assert.equal(ups.length, 4, "one keyup per character");
+ assert.ok(downs.every((k) => k.trusted), "all key events trusted");
+ assert.deepEqual(downs.map((k) => k.code), ["KeyH", "KeyE", "Digit1", "Digit2"], "US layout codes");
+ assert.ok(typingMs >= 4 * 45, `human pace (${typingMs}ms for 4 chars) — not a bulk insert`);
+ assert.deepEqual(await ev("document.getElementById('t').value"), "he12");
+
+ // Human click: a trajectory of moves BEFORE the press, ending on the point.
+ // (lastIndexOf: the type() focus-click earlier also recorded a press.)
+ const input = await ev("(() => { const r = document.getElementById('t').getBoundingClientRect(); return { x: Math.round(r.x + 4), y: Math.round(r.y + 4) }; })()");
+ await c.execute("click", { target: { x: input.x, y: input.y }, human: true, settle_ms: 100 });
+ const m = await ev("window.__m");
+ const pressIdx = m.map((e) => e.n).lastIndexOf("mousedown");
+ assert.ok(pressIdx >= 3, `approach trajectory before the press (${pressIdx} prior events)`);
+ const moves = m.slice(0, pressIdx).filter((e) => e.n === "mousemove");
+ assert.ok(moves.length >= 3, `≥3 approach moves (got ${moves.length})`);
+ assert.ok(moves.every((e) => e.trusted), "all approach moves trusted");
+ const settled = moves.at(-1);
+ assert.ok(Math.hypot(settled.x - input.x, settled.y - input.y) <= 1, "last approach move is on target");
+ assert.ok(m[pressIdx + 1].n === "mouseup", "press → release");
+});
+
+test("proxy: browser egress routes through a credentialed local proxy (407 → CDP auth)", { skip: !hasBrowser ? "live-gate-browser: needs any Chromium-family browser" : false }, async (t) => {
+ // Plain origin server (the "site").
+ const site = http.createServer((req, res) => {
+ res.writeHead(200, { "content-type": "text/html" });
+ res.end("<h1 id=\"ok\">hello-through-proxy</h1>");
+ });
+ await new Promise((ok) => site.listen(0, "127.0.0.1", ok));
+ t.after(() => site.close());
+ const sitePort = site.address().port;
+
+ // Minimal forwarding proxy with a 407 Basic challenge: first request has
+ // no Proxy-Authorization → 407; Chrome answers via our CDP auth handler →
+ // retry with credentials → forwarded.
+ const USER = "fleet", PASS = "s3cret";
+ const seen = { challenges: 0, authorized: 0 };
+ const proxy = http.createServer((req, res) => {
+ const auth = req.headers["proxy-authorization"];
+ if (!auth) {
+ seen.challenges++;
+ res.writeHead(407, { "proxy-authenticate": 'Basic realm="axuse-test"' });
+ res.end();
+ return;
+ }
+ seen.authorized++;
+ const expect = `Basic ${Buffer.from(`${USER}:${PASS}`).toString("base64")}`;
+ assert.equal(auth, expect, "credentials answered over CDP arrive as Basic Proxy-Authorization");
+ const target = new URL(req.url); // absolute URI (plain HTTP proxying)
+ const upstream = http.request(
+ { host: target.hostname, port: target.port, path: target.pathname, method: req.method, headers: { ...req.headers, host: target.host } },
+ (ur) => { res.writeHead(ur.statusCode, ur.headers); ur.pipe(res); },
+ );
+ upstream.on("error", () => { res.writeHead(502); res.end(); });
+ req.pipe(upstream);
+ });
+ await new Promise((ok) => proxy.listen(0, "127.0.0.1", ok));
+ t.after(() => proxy.close());
+ const proxyPort = proxy.address().port;
+
+ const c = new Controller();
+ t.after(() => c.dispose());
+ const opened = await c.execute("open", {
+ url: `http://127.0.0.1:${sitePort}/`,
+ headless: true,
+ proxy: `http://${USER}:${PASS}@127.0.0.1:${proxyPort}`,
+ proxy_bypass: "<-loopback>", // include localhost so the fixture is proxied
+ settle_ms: 300,
+ });
+ // The page rendered THROUGH the proxy (its content, not a Chrome error page).
+ assert.match(opened.observation.text, /hello-through-proxy/);
+ assert.ok(seen.challenges >= 1, "the proxy challenged (407)");
+ assert.ok(seen.authorized >= 1, "the CDP-answered retry carried credentials");
+ await c.execute("close", { browser: true });
+});
diff --git a/test/mcp.test.js b/test/mcp.test.js
index 0e187ef..5a67ecb 100644
--- a/test/mcp.test.js
+++ b/test/mcp.test.js
@@ -53,25 +53,6 @@ function startServer() {
}
test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser }, async (t) => {
- const srv = startServer();
- t.after(async () => { await srv.end(); });
-
- const init = await srv.call({
- jsonrpc: "2.0", id: 1, method: "initialize",
- params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "test", version: "0" } },
- });
- assert.equal(init.result.serverInfo.name, "axuse");
- assert.ok(init.result.capabilities.tools);
-
- srv.proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
-
- const list = await srv.call({ jsonrpc: "2.0", id: 2, method: "tools/list" });
- const names = list.result.tools.map((x) => x.name);
- assert.ok(names.includes("ax_open"));
- assert.ok(names.includes("ax_observe"));
- assert.ok(names.includes("ax_click"));
- assert.ok(names.includes("ax_screenshot"));
- assert.ok(names.includes("ax_open_many"));
// Desktop tools are advertised when the platform driver is enabled. On
// Linux the driver mounts whenever DISPLAY is set AND an X server answers
// (CI runs a real Xvfb on :99; dead DISPLAY values must not count).
@@ -96,8 +77,40 @@ test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser },
const desktopExpected = process.platform === "win32"
|| (process.platform === "darwin" && process.env.AXUSE_ENABLE_MACOS_CUA === "1")
|| linuxDesktopOk;
+
+ // NOTE on stability: this process probes X liveness AND the spawned
+ // server probes it independently at startup. Under full-suite parallel
+ // load the two probes can transiently disagree (one X connect beats a
+ // 1-2 s timeout, the other does not) — a cross-process race, not a
+ // product flake. On disagreement we respawn the server ONCE and re-list;
+ // a second disagreement fails.
+ const boot = async () => {
+ const srv = startServer();
+ const init = await srv.call({
+ jsonrpc: "2.0", id: 1, method: "initialize",
+ params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "test", version: "0" } },
+ });
+ assert.equal(init.result.serverInfo.name, "axuse");
+ assert.ok(init.result.capabilities.tools);
+ srv.proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
+ const list = await srv.call({ jsonrpc: "2.0", id: 2, method: "tools/list" });
+ return { srv, names: list.result.tools.map((x) => x.name) };
+ };
+ let { srv, names } = await boot();
+ t.after(async () => { await srv.end(); });
+ if (!!names.includes("ax_desktop_screenshot") !== desktopExpected && process.platform === "linux" && process.env.DISPLAY) {
+ await srv.end();
+ ({ srv, names } = await boot());
+ t.after(async () => { await srv.end(); });
+ }
+
+ assert.ok(names.includes("ax_open"));
+ assert.ok(names.includes("ax_observe"));
+ assert.ok(names.includes("ax_click"));
+ assert.ok(names.includes("ax_screenshot"));
+ assert.ok(names.includes("ax_open_many"));
if (desktopExpected) {
- assert.ok(names.includes("ax_desktop_screenshot"));
+ assert.ok(names.includes("ax_desktop_screenshot"), "desktop tools mounted (parent probe saw a live display)");
} else {
assert.ok(!names.includes("ax_desktop_screenshot"));
}
diff --git a/test/skip-allowlist.json b/test/skip-allowlist.json
index 799e92d..5b33198 100644
--- a/test/skip-allowlist.json
+++ b/test/skip-allowlist.json
@@ -26,6 +26,8 @@
"captcha-pro loads and its Turnstile path solves the official demo (test keys)": "live-gate-external: needs AXUSE_TEST_LIVE_CAPTCHA=1, a vision API key, and the public demo",
"extension bridge: install → native messaging → drive the browser": "live-gate-browser: needs an extension-capable Chromium-family browser",
"browser flow": "live-gate-browser: needs any Chromium-family browser (auto-detect or AXUSE_BROWSER)",
+ "human input: typing and clicking land as human-shaped trusted events": "live-gate-browser: needs any Chromium-family browser (headless hermetic)",
+ "proxy: browser egress": "live-gate-browser: needs any Chromium-family browser (local fixture proxy — no external service)",
"fill: ARIA and native sliders via trusted Arrow keys": "live-gate-browser: needs any Chromium-family browser",
"fill: foreign-tab refs fail with a hint naming the owning tab": "live-gate-browser: needs any Chromium-family browser",
"mcp: handshake, tools/list, tools/call end-to-end": "live-gate-browser: needs any Chromium-family browser",