Skip to content

armgit / axuse

Public Imported from GitHub
0

Accessibility-first browser & computer use for AI agents — one MCP server for every harness (Codex, Claude Code, OpenCode, Antigravity). Instruments the DOM/accessibility tree via CDP instead of screenshot loops: diffed snapshots, stable element refs, trusted input, typed errors. Zero dependencies

Linux (Ubuntu): test, harden, and bring to SOTA parity — X11 desktop driver + AT-SPI structure layer

CTOL Team BJ7414a42aa328
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d6b78fb..fc92b19 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -53,7 +53,7 @@ jobs:
AXUSE_TEST_MACOS_DESKTOP: "1"
run: |
set -o pipefail
- node --test --test-reporter=tap test/*.test.js | tee "$RUNNER_TEMP/tap.txt"
+ node --test --test-timeout=90000 --test-reporter=tap test/*.test.js | tee "$RUNNER_TEMP/tap.txt"
- name: Skip audit (every skip explicit, typed, CI-visible)
run: node tools/skip-audit.mjs "$RUNNER_TEMP/tap.txt"
- name: Doctor smoke
@@ -125,12 +125,19 @@ jobs:
curl --fail --silent --show-error --location "$url" --output "$RUNNER_TEMP/cft.zip"
unzip -q "$RUNNER_TEMP/cft.zip" -d "$RUNNER_TEMP/cft"
echo "AXUSE_BROWSER=$RUNNER_TEMP/cft/chrome-linux64/chrome" >> "$GITHUB_ENV"
+ - name: Install AT-SPI sandbox deps (GTK app for the structure-layer E2E)
+ run: sudo apt-get update -qq && sudo apt-get install -y -qq python3-gi gir1.2-gtk-3.0
- name: Full suite (TAP for the skip audit)
env:
DISPLAY: ":99"
run: |
set -o pipefail
- node --test --test-reporter=tap test/*.test.js | tee "$RUNNER_TEMP/tap.txt"
+ # A REAL X server on :99: the Linux desktop driver mounts against it
+ # (input synthesis is verified on the suite's own private Xvfb
+ # displays; :99 only lets the MCP layer mount the live tools).
+ Xvfb :99 -screen 0 1280x800x24 -ac -nolisten tcp &
+ sleep 2
+ node --test --test-timeout=90000 --test-reporter=tap test/*.test.js | tee "$RUNNER_TEMP/tap.txt"
- name: Skip audit
run: node tools/skip-audit.mjs "$RUNNER_TEMP/tap.txt"
- name: Doctor smoke
diff --git a/README.md b/README.md
index 5a2918b..4853beb 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,8 @@
server plugs into Codex, Claude Code, OpenCode, Antigravity, or any
MCP-capable host. Zero dependencies (Node ≥ 22). Windows desktop driver is
verified; macOS desktop control is available as an explicit opt-in with
-native TCC permission checks.
+native TCC permission checks; Linux desktop control runs over X11/XWayland
+(XTEST input + capture + EWMH windows, zero extra packages).
The design is distilled from the two best implementations of this problem —
OpenAI's Codex Chrome extension and the ZCode harness — and exists to prove a
@@ -12,11 +13,12 @@ point: **a screenshot loop is not the right interface for a machine**. The
DOM tree, the accessibility tree, and CDP are. axuse instruments the target
instead of photographing it.
-If you are a journalist or other non-developer using Codex or ChatGPT Work,
-start with the [non-developer onboarding guide](docs/non-developer-onboarding.md).
-It explains the one-time connector setup, the first verification prompt, and
-copy-and-paste research prompts. The source repository's setup below is for
-developers and administrators.
+**Not a developer?** You don't need anything below. In the Claude desktop
+app, download `axuse.mcpb` from [ax-use.com](https://ax-use.com) and
+double-click it. In ChatGPT's Codex app, Claude Code, Cursor or any other
+agent, paste the [install prompt](https://ax-use.com/install.md) and the
+agent sets axuse up for you. See the
+[non-developer guide](docs/non-developer-onboarding.md).
## Proven in production
@@ -232,7 +234,8 @@ The ones that save the most steps:
- JS dialogs and new tabs opened by an action are reported in its result.
**Desktop** (Windows: verified. macOS: opt-in with
-`AXUSE_ENABLE_MACOS_CUA=1` and host Accessibility/Screen Recording grants):
+`AXUSE_ENABLE_MACOS_CUA=1` and host Accessibility/Screen Recording grants.
+Linux: automatic on X11/XWayland sessions with `DISPLAY` set):
`ax_desktop_screenshot`, `ax_desktop_screen_info`,
`ax_desktop_click`, `ax_desktop_type`, `ax_desktop_press`, `ax_desktop_scroll`,
`ax_desktop_windows`, `ax_desktop_activate`, `ax_desktop_drag`.
@@ -294,8 +297,13 @@ Field notes (all established live against Edge 155 / Chrome 155):
`AXUSE_ENABLE_MACOS_CUA=1`, grant the host Accessibility and Screen Recording
in System Settings → Privacy & Security, then run `node src/cli.js doctor`.
Coordinates are macOS global display points and returned desktop JPEGs are
- resized to the same point dimensions for direct click mapping. Linux has no
- desktop driver yet (browser tools still work).
+ resized to the same point dimensions for direct click mapping. Linux desktop
+ control mounts automatically on X11/XWayland sessions (needs `DISPLAY`;
+ Xvfb works for headless hosts) — and it is structure-first too: the AT-SPI
+ tree (via axuse's zero-dependency D-Bus client) mounts
+ ax_desktop_ax_snapshot/ax_click/ax_set/ax_read like macOS, plus EWMH
+ windows, RandR monitor layout, a screenshot budget ladder that downscales
+ instead of erroring, and clipboard-paste typing when xclip is installed.
- **The extension backend has no Target domain** (chrome.debugger limits), so
OOPIF frames are listed but not entered there; the CDP backend enters them.
- **Anti-bot reality**: axuse runs a real Chromium with a real profile, but
@@ -317,7 +325,7 @@ summaries.
## Development
```bash
-node --test test/diff.test.js test/browser.test.js test/mcp.test.js test/desktop.test.js test/oopif.test.js test/bridge.test.js test/plugin.test.js
+node --test test/diff.test.js test/browser.test.js test/mcp.test.js test/desktop.test.js test/desktop-linux.test.js test/oopif.test.js test/bridge.test.js test/plugin.test.js
```
On macOS, enable the passive desktop capability gates explicitly:
@@ -334,7 +342,9 @@ macOS builds is honestly skipped because it ignores automated
`--load-extension`); desktop tests exercise passive screen/window discovery on
macOS with `AXUSE_TEST_MACOS_DESKTOP=1`. A real macOS screenshot test is
explicit with `AXUSE_TEST_MACOS_SCREENSHOT=1` because Screen Recording is
-TCC-protected. No test moves your real mouse or sends keys.
+TCC-protected. The Linux desktop tests boot their own private Xvfb displays
+(and skip typed when Xvfb is not installed); they never touch your session.
+No test moves your real mouse or sends keys.
Debugging: `AXUSE_BRIDGE_DEBUG=1` traces hub + native-host messages;
`self.__axuseDiag` in the extension SW reports connect/disconnect history.
diff --git a/SKILL.md b/SKILL.md
index 3b9095c..5f9b394 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -132,6 +132,18 @@ the tabs axuse created — never your browser.
pixels — get bounds from `ax_desktop_screen_info`/`ax_desktop_windows`).
Verified live.
+**Linux** mounts the same tools automatically on X11/XWayland sessions
+(`DISPLAY` set): XTEST input, GetImage JPEG capture, EWMH window list and
+activation, RandR per-monitor layout in `ax_desktop_screen_info` — and the
+**structure-first layer through AT-SPI** (zero-dependency D-Bus client):
+`ax_desktop_ax_snapshot/ax_click/ax_set/ax_read` with refs, roles, names,
+bounds and states, like macOS System Events. No extra packages; `xvfb-run`
+covers headless hosts. Coordinates are root-window pixels. Screenshots honor
+the byte budget with a quality → quality → downscale ladder (vision keeps a
+usable image instead of a dead end), and `ax_desktop_type` pastes long or
+non-keymap text through the clipboard when xclip is present (reported
+honestly as `via: "clipboard-paste"`).
+
**macOS** ships an opt-in driver (System Events keystroke, corrected
CoreGraphics mouse/scroll events via JXA, and `screencapture` + `sips` pixels)
that loads with `AXUSE_ENABLE_MACOS_CUA=1`. It reports TCC permission failures
diff --git a/docs/macos-support.md b/docs/macos-support.md
index ac75e9d..661e62c 100644
--- a/docs/macos-support.md
+++ b/docs/macos-support.md
@@ -199,7 +199,7 @@ the `ax_*` browser tools — never a hang or a generic failure.
| Lane | Where | What it proves |
|---|---|---|
| `macos-26`, `macos-15`, `macos-15-intel` | CI (`ci.yml`) | Full suite × 3 runners (2 versions × arm64 + Intel), CfT installed, gates on, skip audit, doctor, bridge-security assertion |
-| `windows-latest`, `ubuntu-latest` | CI (`ci.yml`) | Cross-platform guard (Windows desktop driver live; Linux browser tools with CfT) |
+| `windows-latest`, `ubuntu-latest` | CI (`ci.yml`) | Cross-platform guard (Windows desktop driver live; Linux browser tools with CfT, Linux desktop driver against a CI Xvfb) |
| Plugin E2E | CI-private runner or dev machine | `AXUSE_TEST_LIVE_*=1 node --test test/plugins.test.js` — every plugin's unit + live suite on macOS |
| `tcc-manual` | `tcc-manual.yml`, self-hosted | Granted-state proofs: real (corner) capture, inert zero-delta CGEvent, recovery re-probe. Fails closed with typed instructions if the runner's grants are missing. Zero skips allowed. |
@@ -263,7 +263,10 @@ Measured on Apple M1, macOS 26.6, Node 26 (`bench/macos-hotpaths.mjs`):
7. **Vision-gated plugin live demos** (challenge-handler B–E) — require
`CHALLENGE_VISION_KEY` / `CHALLENGE_CF_DEMO_URL`; they self-skip typed and
are listed in the allowlist. Not macOS-specific.
-8. **Linux desktop driver** — does not exist (by design; browser tools only).
+8. **Linux desktop driver** — shipped (X11/XWayland). Structure-first via
+ AT-SPI (ax_desktop_ax_snapshot/click/set/read over axuse's zero-dep
+ D-Bus client) plus coordinate+vision input/capture, EWMH windows and
+ RandR monitor layout — the macOS pattern, on Linux.
## Release checklist
diff --git a/package.json b/package.json
index f40d15c..38756b7 100644
--- a/package.json
+++ b/package.json
@@ -11,7 +11,7 @@
"node": ">=22"
},
"scripts": {
- "test": "node --test test/*.test.js",
+ "test": "node --test --test-timeout=90000 test/*.test.js",
"doctor": "node src/cli.js doctor"
},
"keywords": [
diff --git a/src/cli.js b/src/cli.js
index 4003407..8313a2c 100644
--- a/src/cli.js
+++ b/src/cli.js
@@ -148,8 +148,11 @@ function openExtensionsPage(browser, browserPath) { const url = "chrome://exten
const app = apps[browser] ?? "Google Chrome";
return launch("open", ["-a", app, url], app);
}
- const command = browserPath ?? (process.platform === "win32" ? "chrome.exe" : "google-chrome");
- return launch(command, [url], browserPath ?? "browser");
+ const command = browserPath
+ ?? (process.platform === "win32" ? "chrome.exe" : undefined)
+ ?? (process.platform !== "darwin" ? locateBrowser() : undefined)
+ ?? "google-chrome";
+ return launch(command, [url], browserPath ?? command);
}
async function setupCmd() {
@@ -293,6 +296,25 @@ async function doctor() {
out(`native WebSocket: ${hasWs ? "yes" : "NO — need Node >= 22"}`);
const browser = locateBrowser();
out(`browser: ${browser ?? "not found (set AXUSE_BROWSER)"}`);
+ if (process.platform === "linux") {
+ // Chrome keeps shared memory in /dev/shm; a full tmpfs kills headed AND
+ // headless launches with a bare crash. axuse auto-passes
+ // --disable-dev-shm-usage, but the underlying pressure is worth knowing.
+ try {
+ const { statfsSync } = await import("node:fs");
+ const st = statfsSync("/dev/shm");
+ const free = (BigInt(st.bavail) * BigInt(st.bsize)) / 1024n / 1024n;
+ const total = (BigInt(st.blocks) * BigInt(st.bsize)) / 1024n / 1024n;
+ const pct = total > 0n ? Number((free * 100n) / total) : 100;
+ if (free < 128n) {
+ out(`/dev/shm: ${free} MB free of ${total} MB — Chrome needs ≥64MB; axuse adds --disable-dev-shm-usage automatically, but free space if launches still fail`);
+ } else if (pct < 5) {
+ out(`/dev/shm: low (${free} MB free, ${pct.toFixed(0)}%) — axuse mitigates with --disable-dev-shm-usage`);
+ }
+ } catch {
+ // no /dev/shm mount — the launch path mitigates silently
+ }
+ }
if (process.platform === "win32" || process.platform === "darwin") {
if (process.platform === "darwin" && !/^(1|true)$/i.test(process.env.AXUSE_ENABLE_MACOS_CUA ?? "")) {
out("desktop driver: disabled (set AXUSE_ENABLE_MACOS_CUA=1; Accessibility + Screen Recording grants are required for OS control)");
@@ -333,6 +355,29 @@ async function doctor() {
out(`desktop driver: FAILED — ${e.message}`);
}
}
+ } else if (process.platform === "linux") {
+ if (!process.env.DISPLAY) {
+ out(`desktop driver: not available — DISPLAY is not set (X11/XWayland session required; browser tools still work${process.env.WAYLAND_DISPLAY ? "; this looks like a native Wayland session — XWayland usually provides DISPLAY like :0/:1" : ""})`);
+ } else {
+ try {
+ const d = await resolveDesktopDriver();
+ const info = await d.screenInfo();
+ out(`desktop driver: ok (X11/XWayland, XTEST + EWMH) — root window ${info.w}x${info.h} at (${info.x},${info.y}) (pixels)`);
+ try {
+ const clip = await d.constructor.clipboardAvailable?.();
+ out(`desktop clipboard paste: ${clip === undefined ? "unknown" : clip ? "available (xclip — long/untypeable text pastes via ctrl+v)" : "absent (type() synthesizes per key; sudo apt install xclip for the paste path)"}`);
+ } catch {}
+ try {
+ await d.screenshot({ x: info.x, y: info.y, w: 1, h: 1, budgetBytes: 20_000 });
+ out("desktop screenshot: ok");
+ } catch (e) {
+ out(`desktop screenshot: unavailable — ${e.message}`);
+ }
+ d.dispose();
+ } catch (e) {
+ out(`desktop driver: FAILED — ${e.message}`);
+ }
+ }
} else {
out("desktop driver: not implemented on this platform (browser tools still work)");
}
diff --git a/src/core/cdp.js b/src/core/cdp.js
index 6e03119..fb21b41 100644
--- a/src/core/cdp.js
+++ b/src/core/cdp.js
@@ -9,13 +9,29 @@
* dependency-free — important for a library that harnesses vendor via MCP.
*/
import { spawn } from "node:child_process";
-import { accessSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statSync } from "node:fs";
+import { accessSync, existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statfsSync, statSync } from "node:fs";
import { homedir, tmpdir } from "node:os";
import { delimiter, join } from "node:path";
import { AxError, Codes } from "./errors.js";
const SEND_TIMEOUT_MS = 30_000;
+/** Chromium keeps its discardable/shared-memory segment in /dev/shm (tmpfs).
+ * When that mount is full or missing, the browser aborts during startup —
+ * on Linux with a bare "Browser exited during startup (code null)" and one
+ * stderr line ("Less than 64MB of free space…"). Detect the condition and
+ * steer Chrome to the disk-backed temp dir instead (the same thing the
+ * --disable-dev-shm-usage flag does in every headless-container guide). */
+export function linuxShmUnusable() {
+ if (process.platform !== "linux") return false;
+ try {
+ const st = statfsSync("/dev/shm");
+ return BigInt(st.bavail) * BigInt(st.bsize) < 128n * 1024n * 1024n;
+ } catch {
+ return true; // no /dev/shm mount at all (some container/minimal images)
+ }
+}
+
function isMacBrandedChrome(executable) {
// ALL branded Chrome channels (stable/Beta/Dev/Canary) ignore
// --load-extension on current macOS builds — live-verified for stable and
@@ -24,33 +40,62 @@ function isMacBrandedChrome(executable) {
&& /(?:^|\/)Google Chrome(?: Beta| Dev| Canary)?\.app\/Contents\/MacOS\/Google Chrome(?: Beta| Dev| Canary)?$/.test(executable);
}
+/** Branded-Chrome builds that ignore --load-extension, per platform:
+ * macOS .app bundles (live-verified) and the Linux distro packages
+ * (live-verified on google-chrome-stable 151 / Ubuntu 22.04: the flag is
+ * silently ignored — only built-in component extensions appear in
+ * /json/list). Chrome-for-Testing's chrome-linux64 layout is NOT branded. */
+export function isBrandedChrome(executable) {
+ if (isMacBrandedChrome(executable)) return true;
+ if (process.platform === "linux") {
+ return /^\/usr\/bin\/google-chrome(-stable|-beta|-unstable)?$/.test(executable)
+ || /^\/opt\/google\/chrome\/chrome$/.test(executable);
+ }
+ return false;
+}
+
/** Candidate Chromium-family executables, best first. */
export function browserCandidates({ extensionCapable = false, browser, ignoreEnv = false, exact = false } = {}) {
const out = [];
if (!ignoreEnv && process.env.AXUSE_BROWSER && process.env.AXUSE_BROWSER.trim()) {
const configured = process.env.AXUSE_BROWSER.trim();
- // Current macOS branded Chrome builds silently ignore --load-extension.
- // Do not let a general AXUSE_BROWSER override defeat the explicit
- // extension-capable selection used by the bridge test/setup lane.
- if (!(extensionCapable && isMacBrandedChrome(configured))) {
+ // Branded Chrome builds (macOS .app bundles; Linux distro packages)
+ // silently ignore --load-extension. Do not let a general AXUSE_BROWSER
+ // override defeat the explicit extension-capable selection used by the
+ // bridge test/setup lane.
+ if (!(extensionCapable && isBrandedChrome(configured))) {
out.push(configured);
return out;
}
}
if (process.platform === "win32") {
const chrome = [];
+ const chromeBeta = [];
+ const chromeCanary = [];
const edge = [];
+ const edgeBeta = [];
for (const envVar of ["ProgramFiles", "ProgramFiles(x86)", "LOCALAPPDATA"]) {
const base = process.env[envVar];
if (!base) continue;
chrome.push(join(base, "Google", "Chrome", "Application", "chrome.exe"));
+ // Per-user installs of the preview channels (the default for Beta/Canary).
+ if (envVar === "LOCALAPPDATA" || envVar === "ProgramFiles") {
+ chromeBeta.push(join(base, "Google", "Chrome Beta", "Application", "chrome.exe"));
+ chromeCanary.push(join(base, "Google", "Chrome SxS", "Application", "chrome.exe"));
+ }
}
for (const envVar of ["ProgramFiles", "ProgramFiles(x86)"]) {
const base = process.env[envVar];
if (!base) continue;
edge.push(join(base, "Microsoft", "Edge", "Application", "msedge.exe"));
+ edgeBeta.push(join(base, "Microsoft", "Edge Beta", "Application", "msedge.exe"));
}
- out.push(...(browser === "edge" ? edge : browser === "chrome" ? chrome : [...chrome, ...edge]));
+ out.push(...(browser === "edge" ? edge
+ : browser === "edge-beta" ? edgeBeta
+ : browser === "chrome-beta" ? chromeBeta
+ : browser === "chrome-canary" ? chromeCanary
+ : browser === "chrome" ? [...chrome, ...edge]
+ : [...chrome, ...edge, ...edgeBeta, ...chromeBeta, ...chromeCanary]));
} else if (process.platform === "darwin") {
const app = (name, executable) => [
`/Applications/${name}.app/Contents/MacOS/${executable}`,
@@ -100,20 +145,47 @@ export function browserCandidates({ extensionCapable = false, browser, ignoreEnv
out.push(...chrome, ...cft, ...preview, ...chromium, ...edge);
}
} else {
+ // Linux: distro packages live in /usr/bin; snap Chromium installs to
+ // /snap/bin (Ubuntu's default Chromium is a snap). Branded Chrome on
+ // Linux IGNORES --load-extension (live-verified on google-chrome-stable
+ // 151 / Ubuntu 22.04 — only built-in component extensions appear), so it
+ // is excluded from the extension lane exactly like macOS branded builds.
const chrome = [
"/usr/bin/google-chrome",
"/usr/bin/google-chrome-stable",
];
- const chromium = ["/usr/bin/chromium", "/usr/bin/chromium-browser"];
+ const chromium = ["/usr/bin/chromium", "/usr/bin/chromium-browser", "/snap/bin/chromium"];
const cft = ["/usr/bin/google-chrome-for-testing", "/usr/bin/chrome-for-testing"];
- const edge = ["/usr/bin/microsoft-edge"];
+ const edge = ["/usr/bin/microsoft-edge", "/usr/bin/microsoft-edge-stable"];
+ const edgeBeta = ["/usr/bin/microsoft-edge-beta"];
const chromeBeta = ["/usr/bin/google-chrome-beta"];
- out.push(...(browser === "edge" ? edge
- : browser === "chromium" ? chromium
- : browser === "chrome-for-testing" ? cft
- : browser === "chrome-beta" ? chromeBeta
- : browser === "chrome" ? (exact ? chrome : [...chrome, ...cft, ...chromeBeta, ...chromium, ...edge])
- : [...chrome, ...cft, ...chromeBeta, ...chromium, ...edge]));
+ const preview = [...chromeBeta, ...edgeBeta];
+ if (extensionCapable) {
+ const preferred = browser === "edge" ? [...edge, ...edgeBeta]
+ : browser === "edge-beta" ? edgeBeta
+ : browser === "chromium" ? chromium
+ : browser === "chrome-for-testing" ? cft
+ : [...edge, ...edgeBeta, ...chromium, ...cft];
+ out.push(...preferred);
+ for (const dir of (process.env.PATH ?? "").split(delimiter).filter(Boolean)) {
+ if (!browser || browser === "chromium") out.push(join(dir, "chromium"));
+ if (!browser || browser === "chrome-for-testing") out.push(join(dir, "google-chrome-for-testing"));
+ }
+ } else if (browser === "edge") {
+ out.push(...edge);
+ } else if (browser === "edge-beta") {
+ out.push(...edgeBeta);
+ } else if (browser === "chromium") {
+ out.push(...chromium);
+ } else if (browser === "chrome-for-testing") {
+ out.push(...cft);
+ } else if (browser === "chrome-beta") {
+ out.push(...chromeBeta);
+ } else if (browser === "chrome") {
+ out.push(...(exact ? chrome : [...chrome, ...cft, ...preview, ...chromium, ...edge]));
+ } else {
+ out.push(...chrome, ...cft, ...preview, ...chromium, ...edge, ...edgeBeta);
+ }
}
return [...new Set(out)];
}
@@ -257,6 +329,11 @@ export class BrowserHandle {
"--remote-allow-origins=*", // modern Chrome rejects unknown ws Origins
];
if (headless) args.push("--headless");
+ if (linuxShmUnusable() && !args.includes("--disable-dev-shm-usage")) {
+ // /dev/shm full/missing: without this the browser dies with SIGTRAP
+ // before DevTools ever comes up (verified on Ubuntu 22.04).
+ args.push("--disable-dev-shm-usage");
+ }
args.push(...extraArgs);
const proc = spawn(exe, args, { stdio: ["ignore", "pipe", "pipe"], windowsHide: false });
let wsUrl;
@@ -264,17 +341,29 @@ export class BrowserHandle {
wsUrl = await new Promise((resolve, reject) => {
let buf = "";
let settled = false;
+ let tail = ""; // last stderr/stdout chars — the only real evidence when a browser dies young
+ const noteTail = () => {
+ const t = tail.trim();
+ if (!t) return "";
+ const hint = /Less than \d+MB of free space/.test(t)
+ ? " Chrome ran out of shared-memory space (full /dev/shm or temp filesystem). Free space, or raise it, then retry."
+ : /X server|DISPLAY|display .*not/.test(t) && !process.env.DISPLAY && process.platform === "linux"
+ ? " Headed Chrome needs an X server: run inside the desktop session or under xvfb-run (or pass headless: true)."
+ : "";
+ return `${hint}\nBrowser output (tail): ${t}`;
+ };
const timer = setTimeout(() => {
try { proc.kill(); } catch {}
if (!settled) {
settled = true;
reject(new AxError(Codes.LAUNCH_FAILED,
- `Timed out after ${launchTimeoutMs}ms waiting for the DevTools WebSocket URL (first launch of a browser can be slow; raise AXUSE_LAUNCH_TIMEOUT_MS if this recurs)`,
+ `Timed out after ${launchTimeoutMs}ms waiting for the DevTools WebSocket URL (first launch of a browser can be slow; raise AXUSE_LAUNCH_TIMEOUT_MS if this recurs)${noteTail()}`,
{ retry: "retry" }));
}
}, launchTimeoutMs);
const onData = (d) => {
buf += String(d);
+ tail = (tail + String(d)).slice(-800);
const m = buf.match(/DevTools listening on (ws:\/\/\S+)/);
if (m) {
settled = true;
@@ -290,11 +379,13 @@ export class BrowserHandle {
settled = true;
reject(new AxError(Codes.LAUNCH_FAILED, `Could not start browser: ${e.message}`, { retry: "never" }));
});
- proc.on("exit", (code) => {
+ proc.on("exit", (code, signal) => {
clearTimeout(timer);
if (settled) return;
settled = true;
- reject(new AxError(Codes.LAUNCH_FAILED, `Browser exited during startup (code ${code})`, { retry: "retry" }));
+ reject(new AxError(Codes.LAUNCH_FAILED,
+ `Browser exited during startup (code ${code}${signal ? `, signal ${signal}` : ""})${noteTail()}`,
+ { retry: "retry" }));
});
});
} catch (e) {
@@ -386,7 +477,11 @@ export class BrowserHandle {
/** Chrome's helper processes can hold profile files briefly after exit. */
async function removeDirWithRetry(dir, attempts = 10) {
for (let i = 0; i < attempts; i++) {
- try { rmSync(dir, { recursive: true, force: true }); return true; } catch {}
+ try { rmSync(dir, { recursive: true, force: true }); } catch {}
+ // Verify, don't assume: Chrome's dying children can re-create files
+ // moments after a clean delete (late crashpad dumps) — the dir would
+ // silently reappear and look like a leak.
+ if (!existsSync(dir)) return true;
await new Promise((ok) => setTimeout(ok, 200 * (i + 1)));
}
return false;
diff --git a/src/desktop/atspi.js b/src/desktop/atspi.js
new file mode 100644
index 0000000..1e82a05
--- /dev/null
+++ b/src/desktop/atspi.js
@@ -0,0 +1,239 @@
+/**
+ * axuse — AT-SPI2 client over the zero-dependency D-Bus transport.
+ *
+ * The macOS driver gets its platform AX tree from System Events; on Linux
+ * the same tree lives at AT-SPI2, exposed over a private D-Bus socket. This
+ * module speaks the AT-SPI interface set directly (verified against the
+ * live at-spi2 registry on Xorg/GNOME, Ubuntu 22.04):
+ *
+ * - objects are (bus name, object path) pairs — children of the desktop
+ * live on each application's own connection name
+ * - Accessible: GetChildren → a(so), GetChildAtIndex(i), GetRoleName,
+ * GetState → (au), GetInterfaces → as, GetApplication → (so);
+ * Name/ChildCount/Description are D-Bus PROPERTIES (the old GetX
+ * methods do not exist on current registries)
+ * - Component.GetExtents(coordType 0) → (iiii) screen-absolute bounds
+ * - Action: nActions property, GetActionName(i), DoAction(i)
+ * - Text: GetCharCount, GetText(start, end); EditableText.SetTextContents
+ * - Value: current/minimum/maximum as variants
+ */
+import { AxError, Codes } from "../core/errors.js";
+import { DbusClient } from "./dbus.js";
+
+const A11Y = "org.a11y.atspi.Accessible";
+const REGISTRY_NAME = "org.a11y.atspi.Registry";
+const DESKTOP_PATH = "/org/a11y/atspi/accessible/root";
+
+/** AT-SPI state bits (atspi-state.h) — the operationally meaningful ones. */
+export const STATE_NAMES = [
+ "invalid", "active", "armed", "busy", "checked", "collapsed", "defunct",
+ "editable", "enabled", "expandable", "expanded", "focusable", "focused",
+ "has-tooltip", "horizontal", "iconified", "modal", "multi-line",
+ "multiselectable", "opaque", "pressed", "resizable", "selectable",
+ "selected", "sensitive", "showing", "single-line", "stale", "transient",
+ "vertical", "visible", "manages-descendants", "indeterminate",
+];
+
+function decodeStates(words) {
+ const out = [];
+ for (let w = 0; w < words.length; w++) {
+ let bits = words[w] >>> 0;
+ let b = 0;
+ while (bits) {
+ if (bits & 1) {
+ const name = STATE_NAMES[w * 32 + b];
+ if (name) out.push(name);
+ }
+ bits >>>= 1;
+ b++;
+ }
+ }
+ return out;
+}
+
+export class AtspiClient {
+ /** @param {{sessionBusAddress?: string}} [opts] — the session bus to
+ * discover org.a11y.Bus on (defaults to the ambient environment; the
+ * sandbox test passes a private bus explicitly). */
+ constructor(opts = {}) {
+ this.sessionBusAddress = opts.sessionBusAddress ?? process.env.DBUS_SESSION_BUS_ADDRESS;
+ this.bus = null; // the AT-SPI bus connection
+ this.sessionBus = null; // temporary connection used to discover it
+ }
+
+ /** Discover and connect the AT-SPI bus. */
+ async connect() {
+ const direct = process.env.AXUSE_ATSPI_BUS;
+ let address = direct ?? null;
+ if (!address) {
+ const sessionAddress = this.sessionBusAddress;
+ if (!sessionAddress) {
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ "No D-Bus session bus (DBUS_SESSION_BUS_ADDRESS unset) — the AT-SPI structure layer needs the session's accessibility bus.",
+ { hint: "Run inside the desktop session; structure-first desktop control needs AT-SPI.", retry: "never" });
+ }
+ try {
+ this.sessionBus = await new DbusClient(sessionAddress).connect();
+ const [a] = await this.sessionBus.call("org.a11y.Bus", "/org/a11y/bus", "org.a11y.Bus", "GetAddress", "", []);
+ address = String(a);
+ } catch (e) {
+ try { this.sessionBus?.close(); } catch {}
+ this.sessionBus = null;
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ `The session bus has no org.a11y.Bus (is at-spi2 running?): ${e.message}`,
+ { hint: "Install/enable at-spi2-core; coordinate+vision tools keep working without it.", retry: "never" });
+ }
+ }
+ try {
+ this.bus = await new DbusClient(address).connect();
+ } catch (e) {
+ try { this.sessionBus?.close(); } catch {}
+ this.sessionBus = null;
+ throw e;
+ }
+ return this;
+ }
+
+ get connected() {
+ return !!this.bus?.socket && !this.bus.dead;
+ }
+
+ close() {
+ try { this.bus?.close(); } catch {}
+ try { this.sessionBus?.close(); } catch {}
+ this.bus = null;
+ this.sessionBus = null;
+ }
+
+ /* ------------------------------------------------------ object helpers */
+
+ /** The desktop root (registry-owned). */
+ desktop() {
+ return [REGISTRY_NAME, DESKTOP_PATH];
+ }
+
+ /** Accessible children as (name, path) pairs. */
+ async children(ref) {
+ const [list] = await this.bus.call(ref[0], ref[1], A11Y, "GetChildren", "", []);
+ return list;
+ }
+
+ async childCount(ref) {
+ const v = await this.bus.prop(ref[0], ref[1], A11Y, "ChildCount");
+ return Number(v);
+ }
+
+ async name(ref) {
+ const v = await this.bus.prop(ref[0], ref[1], A11Y, "Name");
+ return String(v ?? "");
+ }
+
+ async description(ref) {
+ const v = await this.bus.prop(ref[0], ref[1], A11Y, "Description");
+ return String(v ?? "");
+ }
+
+ async role(ref) {
+ // Method on most at-spi2 apps; a few roots only expose the Role
+ // property (role enum) — report it honestly as role#N then.
+ try {
+ const [r] = await this.bus.call(ref[0], ref[1], A11Y, "GetRoleName", "", []);
+ return String(r);
+ } catch {
+ const v = await this.bus.prop(ref[0], ref[1], A11Y, "Role").catch(() => null);
+ return v === null ? "unknown" : `role#${Number(v)}`;
+ }
+ }
+
+ async states(ref) {
+ const [words] = await this.bus.call(ref[0], ref[1], A11Y, "GetState", "", []);
+ return decodeStates(Array.isArray(words) ? words : [words]);
+ }
+
+ async interfaces(ref) {
+ const [list] = await this.bus.call(ref[0], ref[1], A11Y, "GetInterfaces", "", []);
+ return list;
+ }
+
+ async application(ref) {
+ const [app] = await this.bus.call(ref[0], ref[1], A11Y, "GetApplication", "", []);
+ return app;
+ }
+
+ /** Screen-absolute bounds via Component.GetExtents(coordType 0). */
+ async bounds(ref) {
+ const [tuple] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Component", "GetExtents", "u", [0]);
+ return { x: tuple[0], y: tuple[1], w: tuple[2], h: tuple[3] };
+ }
+
+ /* --------------------------------------------------------------- Action */
+ // CAUTION: GetActions is never called internally — GTK's atk-bridge (2.4x)
+ // crashes marshalling its a(sss) reply and takes the app's bus connection
+ // with it. DoAction(i) and GetActionName(i) are the safe surface.
+
+ async actionCount(ref) {
+ try {
+ const v = await this.bus.prop(ref[0], ref[1], "org.a11y.atspi.Action", "nActions");
+ if (v !== undefined && v !== null) return Number(v);
+ } catch {}
+ // No nActions property (GTK): a resolvable action name implies ≥1 action.
+ try { await this.actionName(ref, 0); return 1; } catch { return 0; }
+ }
+
+ async actionName(ref, index) {
+ const [n] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Action", "GetActionName", "i", [index]);
+ return String(n);
+ }
+
+ async doAction(ref, index) {
+ const [ok] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Action", "DoAction", "i", [index]);
+ return !!ok;
+ }
+
+ async doAction(ref, index) {
+ const [ok] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Action", "DoAction", "i", [index]);
+ return !!ok;
+ }
+
+ /* ----------------------------------------------------------------- Text */
+
+ async charCount(ref) {
+ const [n] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Text", "GetCharCount", "", []);
+ return Number(n);
+ }
+
+ async text(ref, start = 0, end = -1) {
+ const [s] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Text", "GetText", "ii", [start, end]);
+ return String(s);
+ }
+
+ async setTextContents(ref, contents) {
+ const [ok] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.EditableText", "SetTextContents", "s", [String(contents)]);
+ return !!ok;
+ }
+
+ /* ---------------------------------------------------------------- Value */
+
+ async value(ref) {
+ try {
+ const [cur] = await this.bus.call(ref[0], ref[1], "org.a11y.atspi.Value", "GetCurrentValue", "", []);
+ return cur?.sig === "d" ? cur.value : cur?.value ?? null;
+ } catch {
+ return null;
+ }
+ }
+
+ /* ------------------------------------------------------------- registry */
+
+ /** Applications registered on the desktop: [{ref, name}]. */
+ async apps() {
+ const kids = await this.children(this.desktop());
+ const out = [];
+ for (const ref of kids) {
+ let n = "";
+ try { n = await this.name(ref); } catch {}
+ out.push({ ref, name: n });
+ }
+ return out;
+ }
+}
diff --git a/src/desktop/dbus.js b/src/desktop/dbus.js
new file mode 100644
index 0000000..6a43009
--- /dev/null
+++ b/src/desktop/dbus.js
@@ -0,0 +1,486 @@
+/**
+ * axuse — minimal zero-dependency D-Bus client (AT-SPI desktop layer).
+ *
+ * The macOS driver gets its platform AX tree from System Events over JXA;
+ * Linux exposes the same thing — AT-SPI2 — over D-Bus at a Unix socket.
+ * Node ships no D-Bus client, so this speaks the wire protocol directly:
+ *
+ * - SASL EXTERNAL handshake (uid cookie — the standard user-bus auth)
+ * - Hello(), then method_call / method_return / error routing by serial
+ * - a general little-endian marshaller for the type grammar used by the
+ * AT-SPI interfaces: basics (y b n q i u x t d s o g), arrays, structs,
+ * variants and dict entries, with the alignment rules from the spec
+ *
+ * Scope: method calls + property access on the AT-SPI bus. No signals, no
+ * file-descriptor passing, no peer-to-peer container gymnastics.
+ */
+import { connect as netConnect } from "node:net";
+import { userInfo } from "node:os";
+import { AxError, Codes } from "../core/errors.js";
+
+const LITTLE = 0x6c; // 'l'
+
+const MSG_METHOD_CALL = 1;
+const MSG_METHOD_RETURN = 2;
+const MSG_ERROR = 3;
+
+// Header field codes.
+const FIELD_PATH = 1, FIELD_INTERFACE = 2, FIELD_MEMBER = 3,
+ FIELD_ERROR_NAME = 4, FIELD_REPLY_SERIAL = 5, FIELD_DESTINATION = 6,
+ FIELD_SENDER = 7, FIELD_SIGNATURE = 8;
+
+/* ------------------------------------------------------- type marshalling */
+
+class Writer {
+ constructor() {
+ this.chunks = [];
+ this.length = 0;
+ this.alignBuf = { 1: Buffer.from([0]), 2: Buffer.alloc(2), 4: Buffer.alloc(4), 8: Buffer.alloc(8) };
+ }
+ push(buf) {
+ this.chunks.push(buf);
+ this.length += buf.length;
+ }
+ align(n) {
+ const pad = (n - (this.length % n)) % n;
+ if (pad) this.push(Buffer.alloc(pad));
+ }
+ byte(v) { this.align(1); this.push(Buffer.from([v & 0xff])); }
+ bool(v) { this.align(4); this.push(Buffer.from([v ? 1 : 0, 0, 0, 0])); }
+ u16(v) { this.align(2); const b = Buffer.alloc(2); b.writeUInt16LE(v); this.push(b); }
+ i16(v) { this.align(2); const b = Buffer.alloc(2); b.writeInt16LE(v); this.push(b); }
+ u32(v) { this.align(4); const b = Buffer.alloc(4); b.writeUInt32LE(v >>> 0); this.push(b); }
+ i32(v) { this.align(4); const b = Buffer.alloc(4); b.writeInt32LE(v | 0); this.push(b); }
+ u64(v) { this.align(8); const b = Buffer.alloc(8); b.writeBigUInt64LE(BigInt(v)); this.push(b); }
+ string(v) {
+ const s = Buffer.from(String(v), "utf8");
+ this.u32(s.length);
+ this.push(s);
+ this.push(Buffer.from([0]));
+ }
+ signature(v) {
+ const s = Buffer.from(String(v), "utf8");
+ if (s.length > 255) throw new Error("signature too long");
+ this.byte(s.length);
+ this.push(s);
+ this.push(Buffer.from([0]));
+ }
+ /** Write one value of the given single-complete-type signature. */
+ value(sig, v) {
+ switch (sig) {
+ case "y": return this.byte(v);
+ case "b": return this.bool(v);
+ case "n": return this.i16(v);
+ case "q": return this.u16(v);
+ case "i": return this.i32(v);
+ case "u": return this.u32(v);
+ case "x": case "t": return this.u64(v);
+ case "d": {
+ this.align(8);
+ const b = Buffer.alloc(8);
+ b.writeDoubleLE(Number(v));
+ this.push(b);
+ return;
+ }
+ case "s": return this.string(v);
+ case "o": return this.string(v);
+ case "g": return this.signature(v);
+ case "v": {
+ const inner = v ?? { sig: "s", value: "" };
+ this.signature(inner.sig);
+ return this.value(inner.sig, inner.value);
+ }
+ }
+ if (sig.startsWith("a")) {
+ const elem = sig.slice(1);
+ const body = new Writer();
+ const elemAlign = alignmentOf(elem);
+ for (const item of Array.isArray(v) ? v : []) {
+ body.align(elemAlign); // every element starts at its alignment
+ body.value(elem, item);
+ }
+ const bytes = body.finish();
+ this.u32(bytes.length);
+ this.align(elemAlign);
+ return this.push(bytes);
+ }
+ if (sig.startsWith("(") || sig.startsWith("{")) {
+ const inner = sig.slice(1, -1);
+ this.align(8);
+ if (sig.startsWith("{")) {
+ // Dict entries are written as {key, value} objects (the reader's shape).
+ const obj = Array.isArray(v) ? { key: v[0], value: v[1] } : v;
+ return this.writeValues(inner, [obj?.key, obj?.value]);
+ }
+ return this.writeValues(inner, Array.isArray(v) ? v : []);
+ }
+ throw new Error(`unsupported D-Bus signature: ${sig}`);
+ }
+ /** Write a sequence of values for a multi-type signature ("is", "a(so)"…). */
+ writeValues(sig, values) {
+ let vi = 0;
+ for (const [type] of splitSignature(sig)) {
+ this.value(type, values[vi++]);
+ }
+ }
+ finish() {
+ return Buffer.concat(this.chunks);
+ }
+}
+
+class Reader {
+ constructor(buf) {
+ this.buf = buf;
+ this.pos = 0;
+ }
+ align(n) {
+ this.pos += (n - (this.pos % n)) % n;
+ }
+ byte() { this.align(1); return this.buf[this.pos++]; }
+ u16() { this.align(2); const v = this.buf.readUInt16LE(this.pos); this.pos += 2; return v; }
+ i16() { this.align(2); const v = this.buf.readInt16LE(this.pos); this.pos += 2; return v; }
+ u32() { this.align(4); const v = this.buf.readUInt32LE(this.pos); this.pos += 4; return v; }
+ i32() { this.align(4); const v = this.buf.readInt32LE(this.pos); this.pos += 4; return v; }
+ u64() { this.align(8); const v = this.buf.readBigUInt64LE(this.pos); this.pos += 8; return Number(v); }
+ string() {
+ const len = this.u32();
+ const s = this.buf.toString("utf8", this.pos, this.pos + len);
+ this.pos += len + 1;
+ return s;
+ }
+ signature() {
+ const len = this.byte();
+ const s = this.buf.toString("latin1", this.pos, this.pos + len);
+ this.pos += len + 1;
+ return s;
+ }
+ value(sig) {
+ switch (sig) {
+ case "y": return this.byte();
+ case "b": { this.align(4); const v = this.u32(); return !!v; }
+ case "n": return this.i16();
+ case "q": return this.u16();
+ case "i": return this.i32();
+ case "u": return this.u32();
+ case "x": case "t": return this.u64();
+ case "d": { this.align(8); const v = this.buf.readDoubleLE(this.pos); this.pos += 8; return v; }
+ case "s": case "o": return this.string();
+ case "g": return this.signature();
+ case "v": {
+ const inner = this.signature();
+ return { sig: inner, value: this.value(inner) };
+ }
+ }
+ if (sig.startsWith("a")) {
+ const elem = sig.slice(1);
+ const len = this.u32();
+ this.align(alignmentOf(elem));
+ const end = this.pos + len;
+ const out = [];
+ // Guard against malformed lengths from hostile peers.
+ if (end > this.buf.length + 8) throw new Error("bad array length in D-Bus message");
+ while (this.pos < end) {
+ out.push(this.value(elem));
+ this.align(alignmentOf(elem)); // elements restart at their alignment
+ }
+ this.pos = end;
+ return out;
+ }
+ if (sig.startsWith("(") || sig.startsWith("{")) {
+ this.align(8);
+ const inner = sig.slice(1, -1);
+ const types = splitSignature(inner);
+ const out = [];
+ for (const [type] of types) out.push(this.value(type));
+ return sig.startsWith("(") ? out : { key: out[0], value: out[1] };
+ }
+ throw new Error(`unsupported D-Bus signature while reading: ${sig}`);
+ }
+ values(sig) {
+ const out = [];
+ for (const [type] of splitSignature(sig)) out.push(this.value(type));
+ return out;
+ }
+}
+
+/** Split a signature into its top-level single-complete types. */
+function splitSignature(sig) {
+ const out = [];
+ let i = 0;
+ while (i < sig.length) {
+ const start = i;
+ let depth = 0;
+ // Consume one complete type.
+ for (;;) {
+ const c = sig[i];
+ if (c === "a") { i++; continue; }
+ if (c === "(" || c === "{") { depth++; i++; continue; }
+ if (c === ")" || c === "}") { depth--; i++; if (depth === 0) break; continue; }
+ i++;
+ if (depth === 0) break;
+ }
+ out.push([sig.slice(start, i)]);
+ }
+ return out;
+}
+
+function alignmentOf(sig) {
+ if (sig.startsWith("a")) return alignmentOf(sig.slice(1));
+ if (sig.startsWith("(") || sig.startsWith("{")) return 8;
+ switch (sig) {
+ case "y": case "g": return 1;
+ case "n": case "q": return 2;
+ case "b": case "i": case "u": case "s": case "o": return 4;
+ case "x": case "t": case "d": case "v": return 8;
+ default: return 1;
+ }
+}
+
+/** Marshal a single value of `sig` (for tests and future direct users). */
+export function marshalValue(sig, value) {
+ const w = new Writer();
+ w.value(sig, value);
+ return w.finish();
+}
+
+/** Unmarshal a single value of `sig`; `offset` positions the reader
+ * (alignment is relative to the buffer + offset, like a message body). */
+export function unmarshalValue(sig, buf, offset = 0) {
+ const r = new Reader(buf);
+ r.pos = offset;
+ return r.value(sig);
+}
+
+/* ------------------------------------------------------------ connection */
+
+/** Parse a D-Bus address like unix:path=/run/user/1000/bus. */
+export function parseDbusAddress(address) {
+ const m = /^unix:(path=[^,]+|abstract=[^,]+)/.exec(String(address ?? ""));
+ if (!m) return null;
+ const [, kv] = m;
+ const [k, v] = kv.split("=");
+ return k === "path" ? { path: v } : { abstract: v };
+}
+
+const MSG_TIMEOUT_MS = 12_000;
+
+export class DbusClient {
+ /**
+ * @param {string} address D-Bus server address (unix:path=…)
+ */
+ constructor(address) {
+ this.address = address;
+ this.socket = null;
+ this.serial = 0;
+ this.pending = new Map(); // serial → {resolve, reject, timer}
+ this.buffer = Buffer.alloc(0);
+ this.dead = false;
+ }
+
+ async connect() {
+ const parsed = parseDbusAddress(this.address);
+ if (!parsed) {
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ `Cannot parse D-Bus address: ${this.address}`,
+ { retry: "never" });
+ }
+ this.socket = await new Promise((resolve, reject) => {
+ const t = setTimeout(() => {
+ // Never leave a half-created socket dangling on the event loop.
+ try { sock.destroy(); } catch {}
+ reject(new Error("D-Bus socket connect timed out"));
+ }, 5_000);
+ const sock = parsed.path !== undefined
+ ? netConnect(parsed.path)
+ : netConnect(`\0${parsed.abstract}`);
+ sock.once("error", (e) => { clearTimeout(t); try { sock.destroy(); } catch {} reject(e); });
+ sock.once("connect", () => { clearTimeout(t); sock.off("error", reject); resolve(sock); });
+ });
+ this.socket.on("close", () => this.failAll("D-Bus connection closed"));
+ this.socket.on("error", () => this.failAll("D-Bus connection error"));
+
+ // SASL EXTERNAL: leading NUL (needed on the transport), then the cookie.
+ // BEGIN has NO reply — the next server bytes are already D-Bus messages,
+ // so the message listeners attach only after the handshake completes.
+ const uidHex = Buffer.from(String(userInfo().uid)).toString("hex");
+ try {
+ await this.sasl(`\0AUTH EXTERNAL ${uidHex}\r\n`, /^OK /);
+ } catch (e) {
+ try { this.socket.destroy(); } catch {}
+ this.socket = null;
+ throw e;
+ }
+ this.socket.write("BEGIN\r\n");
+ this.socket.on("data", (d) => this.route(d));
+ await this.call("org.freedesktop.DBus", "/org/freedesktop/DBus", "org.freedesktop.DBus", "Hello", "", []);
+ return this;
+ }
+
+ /** One SASL line exchange (BEFORE message framing starts), time-bounded:
+ * a wedged daemon must not dangle the socket on the event loop. */
+ async sasl(request, expect) {
+ const reply = await new Promise((resolve, reject) => {
+ const t = setTimeout(() => {
+ cleanup();
+ try { this.socket?.destroy(); } catch {}
+ reject(new Error("D-Bus SASL handshake timed out"));
+ }, 5_000);
+ let buf = "";
+ const onData = (d) => {
+ buf += d.toString("latin1");
+ const i = buf.indexOf("\r\n");
+ if (i >= 0) {
+ clearTimeout(t);
+ cleanup();
+ resolve(buf.slice(0, i));
+ }
+ };
+ const onErr = (e) => { clearTimeout(t); cleanup(); reject(e); };
+ const cleanup = () => {
+ this.socket?.off("data", onData);
+ this.socket?.off("error", onErr);
+ };
+ this.socket.on("data", onData);
+ this.socket.on("error", onErr);
+ this.socket.write(request);
+ });
+ if (expect && !expect.test(reply)) {
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ `D-Bus SASL handshake failed: ${reply.trim()}`,
+ { retry: "never" });
+ }
+ return reply;
+ }
+
+ route(chunk) {
+ this.buffer = Buffer.concat([this.buffer, chunk]);
+ for (;;) {
+ if (this.buffer.length < 16) return;
+ // Header "yyyyuua(yv)": [4..8) body length, [8..12) serial,
+ // [12..16) fields-array length; elements start at 16 (8-aligned).
+ const bodyLen = this.buffer.readUInt32LE(4);
+ const arrLen = this.buffer.readUInt32LE(12);
+ const headerLen = 16 + arrLen + ((8 - ((16 + arrLen) % 8)) % 8);
+ const total = headerLen + bodyLen;
+ if (this.buffer.length < total) return;
+ const msg = Buffer.from(this.buffer.subarray(0, total));
+ this.buffer = this.buffer.subarray(total);
+ this.dispatch(msg, headerLen, arrLen);
+ }
+ }
+
+ dispatch(msg, bodyStart, arrLen) {
+ const type = msg[1];
+ // Fields elements span [16, 16+arrLen) — the u32 length prefix sits at
+ // 12..16 and the elements start 8-aligned at 16.
+ const reader = new Reader(msg.subarray(16));
+ let replySer = null, errorName = null, sig = "";
+ try {
+ const end = arrLen;
+ for (;;) {
+ if (reader.pos >= end) break;
+ reader.align(8); // each (yv) struct element is 8-aligned
+ const code = reader.byte();
+ const v = reader.value("v");
+ if (code === FIELD_REPLY_SERIAL) replySer = v.value;
+ else if (code === FIELD_ERROR_NAME) errorName = String(v.value);
+ else if (code === FIELD_SIGNATURE) sig = String(v.value);
+ }
+ } catch {
+ return; // unparseable header — drop
+ }
+ const p = replySer !== null ? this.pending.get(replySer) : null;
+ if (!p) return;
+ this.pending.delete(replySer);
+ clearTimeout(p.timer);
+ if (type === MSG_ERROR) {
+ const body = new Reader(msg.subarray(bodyStart));
+ p.reject(new AxError(Codes.DESKTOP_OP_FAILED,
+ `D-Bus error: ${errorName ?? "unknown"}${sig ? ` (${body.value("s")})` : ""}`,
+ { retry: "retry" }));
+ return;
+ }
+ if (type === MSG_METHOD_RETURN) {
+ const body = new Reader(msg.subarray(bodyStart));
+ p.resolve(sig ? body.values(sig) : []);
+ }
+ // Signals (type 4) are ignored — v1 polls.
+ }
+
+ failAll(reason) {
+ this.dead = true;
+ for (const [, p] of this.pending) {
+ clearTimeout(p.timer);
+ p.reject(new AxError(Codes.DESKTOP_OP_FAILED, reason));
+ }
+ this.pending.clear();
+ }
+
+ /**
+ * Method call. `argsSig` is the argument signature, `args` the values.
+ * Resolves with the array of reply values.
+ */
+ call(destination, path, iface, member, argsSig, args) {
+ if (!this.socket || this.dead) {
+ return Promise.reject(new AxError(Codes.DESKTOP_OP_FAILED, "D-Bus connection is not open"));
+ }
+ const serial = ++this.serial;
+ const body = new Writer();
+ if (argsSig) body.writeValues(argsSig, args);
+ const bodyBytes = body.finish();
+
+ // Header "yyyyuua(yv)": [4..8) body length, [8..12) serial,
+ // [12..16) fields-array length, elements from 16 (8-aligned),
+ // header padded to a multiple of 8 before the body.
+ const fields = new Writer();
+ const field = (code, sig, value) => {
+ fields.align(8); // struct element alignment
+ fields.byte(code);
+ fields.value("v", { sig, value });
+ };
+ field(FIELD_PATH, "o", path);
+ field(FIELD_DESTINATION, "s", destination);
+ field(FIELD_INTERFACE, "s", iface);
+ field(FIELD_MEMBER, "s", member);
+ if (argsSig) field(FIELD_SIGNATURE, "g", argsSig);
+ const fieldsBytes = fields.finish();
+
+ const head = Buffer.alloc(16);
+ head[0] = LITTLE;
+ head[1] = MSG_METHOD_CALL;
+ head[2] = 0; // flags
+ head[3] = 1; // protocol version
+ head.writeUInt32LE(bodyBytes.length, 4);
+ head.writeUInt32LE(serial, 8);
+ head.writeUInt32LE(fieldsBytes.length, 12);
+ const headerPad = (8 - ((16 + fieldsBytes.length) % 8)) % 8;
+ const msg = Buffer.concat([
+ head,
+ fieldsBytes,
+ Buffer.alloc(headerPad),
+ bodyBytes,
+ ]);
+ return new Promise((resolve, reject) => {
+ const timer = setTimeout(() => {
+ this.pending.delete(serial);
+ reject(new AxError(Codes.TIMEOUT, `D-Bus call ${iface}.${member} timed out after ${MSG_TIMEOUT_MS}ms`, { retry: "retry" }));
+ }, MSG_TIMEOUT_MS);
+ this.pending.set(serial, { resolve, reject, timer });
+ this.socket.write(msg);
+ });
+ }
+
+ /** org.freedesktop.DBus.Properties.Get — returns the variant value. */
+ async prop(destination, path, iface, name) {
+ const [variant] = await this.call(destination, path, "org.freedesktop.DBus.Properties", "Get", "ss", [iface, name]);
+ return variant?.value;
+ }
+
+ close() {
+ this.dead = true;
+ this.failAll("D-Bus client closed");
+ try { this.socket?.end(); } catch {}
+ try { this.socket?.destroy(); } catch {}
+ this.socket = null;
+ }
+}
diff --git a/src/desktop/index.js b/src/desktop/index.js
index e8f969d..a648ce4 100644
--- a/src/desktop/index.js
+++ b/src/desktop/index.js
@@ -22,7 +22,12 @@
* loads with AXUSE_ENABLE_MACOS_CUA=1; reports TCC failures clearly.
* AX-tree ops need the Accessibility grant; capture needs Screen
* Recording.
- * - Linux: no driver yet (browser tools still work).
+ * - Linux: zero-dependency X11/XWayland driver (src/desktop/linux.js):
+ * XTEST FakeInput for trusted input, GetImage+baseline JPEG for
+ * capture, EWMH for the window list and activation. Mounts whenever
+ * DISPLAY is set (verified on Xorg 1.21 / Ubuntu 22.04+ and Xvfb);
+ * native Wayland sessions without XWayland fail typed with guidance.
+ * Coordinate+vision mode, like Windows — no AT-SPI AX tree in v1.
*
* The driver interface is the seam: a Windows UIA driver slots in behind
* the same tools without touching the contract.
@@ -241,7 +246,9 @@ class WindowsDriver {
let cached = null;
/** Get the platform driver, or null when unsupported. Tools hide when null.
- * macOS remains explicitly opt-in because it can synthesize OS input. */
+ * macOS remains explicitly opt-in because it can synthesize OS input; the
+ * Linux X11 driver mounts when a DISPLAY is configured (same trust level as
+ * the always-on Windows console-session driver). */
export function getDesktopDriver() {
if (process.platform === "win32") {
if (!cached) cached = new WindowsDriver();
@@ -260,6 +267,20 @@ export function getDesktopDriver() {
}
return cached;
}
+ if (process.platform === "linux") {
+ if (!process.env.DISPLAY) return null; // no X11/XWayland to drive
+ if (!cached) {
+ // Lazy import keeps the X11/JPEG modules out of non-Linux paths; the
+ // driver resolves through resolveDesktopDriver() like the macOS one.
+ cached = import("./linux.js")
+ .then(({ LinuxDriver }) => new LinuxDriver())
+ .catch((e) => {
+ cached = null;
+ throw e;
+ });
+ }
+ return cached;
+ }
return null;
}
@@ -279,9 +300,14 @@ export function desktopUnavailable() {
const macNote = process.platform === "darwin"
? " The macOS driver is disabled: set AXUSE_ENABLE_MACOS_CUA=1; desktop input/capture also needs Accessibility and Screen Recording grants for the host."
: "";
+ const linuxNote = process.platform === "linux" && !process.env.DISPLAY
+ ? " The Linux driver speaks X11/XWayland: set DISPLAY (usually :0/:1) — or run under xvfb-run — and the ax_desktop_* tools mount automatically."
+ : "";
throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
process.platform === "darwin"
? `Desktop control is disabled on macOS until explicitly enabled.${macNote}`
- : `Desktop control is not implemented for ${process.platform} yet.${macNote}`,
+ : process.platform === "linux"
+ ? `Desktop control is unavailable on this Linux session.${linuxNote}`
+ : `Desktop control is not implemented for ${process.platform} yet.${macNote}`,
{ hint: "Prefer the ax_* browser tools whenever the target is a web page.", retry: "never" });
}
diff --git a/src/desktop/jpeg.js b/src/desktop/jpeg.js
new file mode 100644
index 0000000..b826368
--- /dev/null
+++ b/src/desktop/jpeg.js
@@ -0,0 +1,359 @@
+/**
+ * axuse — baseline JPEG encoder (Linux desktop driver screenshots).
+ *
+ * The Windows driver gets JPEG from GDI+ and macOS from sips; Node ships no
+ * image encoder, so this is a compact baseline-sequential encoder written
+ * against the JPEG spec: standard Annex K quantization tables scaled by the
+ * IJG quality curve, standard Huffman tables, 4:2:0 chroma subsampling,
+ * BT.601 full-range RGB→YCbCr. Plain Buffer math, zero dependencies.
+ */
+
+// Standard luminance/chrominance quantization tables (Annex K, quality 50).
+const STD_LUM_Q = [
+ 16, 11, 10, 16, 24, 40, 51, 61,
+ 12, 12, 14, 19, 26, 58, 60, 55,
+ 14, 13, 16, 24, 40, 57, 69, 56,
+ 14, 17, 22, 29, 51, 87, 80, 62,
+ 18, 22, 37, 56, 68, 109, 103, 77,
+ 24, 35, 55, 64, 81, 104, 113, 92,
+ 49, 64, 78, 87, 103, 121, 120, 101,
+ 72, 92, 95, 98, 112, 100, 103, 99,
+];
+const STD_CHROMA_Q = [
+ 17, 18, 24, 47, 99, 99, 99, 99,
+ 18, 21, 26, 66, 99, 99, 99, 99,
+ 24, 26, 56, 99, 99, 99, 99, 99,
+ 47, 66, 99, 99, 99, 99, 99, 99,
+ 99, 99, 99, 99, 99, 99, 99, 99,
+ 99, 99, 99, 99, 99, 99, 99, 99,
+ 99, 99, 99, 99, 99, 99, 99, 99,
+ 99, 99, 99, 99, 99, 99, 99, 99,
+];
+
+// Zigzag order for 8x8 blocks.
+const ZIGZAG = [
+ 0, 1, 8, 16, 9, 2, 3, 10,
+ 17, 24, 32, 25, 18, 11, 4, 5,
+ 12, 19, 26, 33, 40, 48, 41, 34,
+ 27, 20, 13, 6, 7, 14, 21, 28,
+ 35, 42, 49, 56, 57, 50, 43, 36,
+ 29, 22, 15, 23, 30, 37, 44, 51,
+ 58, 59, 52, 45, 38, 31, 39, 46,
+ 53, 60, 61, 54, 47, 55, 62, 63,
+];
+
+// Standard Huffman tables (Annex K): [16 code lengths, symbols].
+const DC_LUM_LENGTHS = [0, 1, 5, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0];
+const DC_LUM_SYMBOLS = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11];
+const DC_CHROMA_LENGTHS = [0, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0];
+const DC_CHROMA_SYMBOLS = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11];
+
+const AC_LUM_LENGTHS = [0, 2, 1, 3, 3, 2, 4, 3, 5, 5, 4, 4, 0, 0, 1, 125];
+const AC_LUM_SYMBOLS = [
+ 0x01, 0x02, 0x03, 0x00, 0x04, 0x11, 0x05, 0x12, 0x21, 0x31, 0x41, 0x06, 0x13, 0x51, 0x61, 0x07,
+ 0x22, 0x71, 0x14, 0x32, 0x81, 0x91, 0xa1, 0x08, 0x23, 0x42, 0xb1, 0xc1, 0x15, 0x52, 0xd1, 0xf0,
+ 0x24, 0x33, 0x62, 0x72, 0x82, 0x09, 0x0a, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x25, 0x26, 0x27, 0x28,
+ 0x29, 0x2a, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49,
+ 0x4a, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69,
+ 0x6a, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89,
+ 0x8a, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7,
+ 0xa8, 0xa9, 0xaa, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, 0xb8, 0xb9, 0xba, 0xc2, 0xc3, 0xc4, 0xc5,
+ 0xc6, 0xc7, 0xc8, 0xc9, 0xca, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda, 0xe1, 0xe2,
+ 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8,
+ 0xf9, 0xfa,
+];
+
+const AC_CHROMA_LENGTHS = [0, 2, 1, 2, 4, 4, 3, 4, 7, 5, 4, 4, 0, 1, 2, 119];
+const AC_CHROMA_SYMBOLS = [
+ 0x00, 0x01, 0x02, 0x03, 0x11, 0x04, 0x05, 0x21, 0x31, 0x06, 0x12, 0x41, 0x51, 0x07, 0x61, 0x71,
+ 0x13, 0x22, 0x32, 0x81, 0x08, 0x14, 0x42, 0x91, 0xa1, 0xb1, 0xc1, 0x09, 0x23, 0x33, 0x52, 0xf0,
+ 0x15, 0x62, 0x72, 0xd1, 0x0a, 0x16, 0x24, 0x34, 0xe1, 0x25, 0xf1, 0x17, 0x18, 0x19, 0x1a, 0x26,
+ 0x27, 0x28, 0x29, 0x2a, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48,
+ 0x49, 0x4a, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68,
+ 0x69, 0x6a, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87,
+ 0x88, 0x89, 0x8a, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0xa2, 0xa3, 0xa4, 0xa5,
+ 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, 0xb8, 0xb9, 0xba, 0xc2, 0xc3,
+ 0xc4, 0xc5, 0xc6, 0xc7, 0xc8, 0xc9, 0xca, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda,
+ 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8,
+ 0xf9, 0xfa,
+];
+
+/** IJG quality scaling for the quantization tables (quality 1-100). */
+function scaledQuantTable(std, quality) {
+ const q = Math.max(1, Math.min(100, Math.round(quality)));
+ const scale = q < 50 ? Math.floor(5000 / q) : 200 - q * 2;
+ return std.map((v) => Math.max(1, Math.min(255, Math.floor((v * scale + 50) / 100))));
+}
+
+/** symbol → {code, length} from a [lengths, symbols] table. */
+function buildHuffmanCodes(lengths, symbols) {
+ const total = lengths.reduce((a, b) => a + b, 0);
+ if (total !== symbols.length) {
+ throw new Error(`Huffman table mismatch: lengths sum ${total} ≠ ${symbols.length} symbols`);
+ }
+ const codes = new Map();
+ let code = 0;
+ let k = 0;
+ for (let len = 1; len <= 16; len++) {
+ for (let i = 0; i < lengths[len - 1]; i++) {
+ codes.set(symbols[k], { code, length: len });
+ code++;
+ k++;
+ }
+ code <<= 1;
+ }
+ return codes;
+}
+
+function dhtSegment(tableClass, tableId, lengths, symbols) {
+ const body = Buffer.alloc(1 + 16 + symbols.length);
+ body.writeUInt8((tableClass << 4) | tableId, 0);
+ for (let i = 0; i < 16; i++) body.writeUInt8(lengths[i], 1 + i);
+ Buffer.from(symbols).copy(body, 17);
+ return Buffer.concat([Buffer.from([0xff, 0xc4, (body.length + 2) >> 8, (body.length + 2) & 0xff]), body]);
+}
+
+class BitWriter {
+ constructor() {
+ this.buf = Buffer.alloc(256 * 1024);
+ this.len = 0;
+ this.bitBuf = 0;
+ this.bitCount = 0;
+ }
+ ensure(n) {
+ if (this.len + n + 2 > this.buf.length) {
+ let size = this.buf.length * 2;
+ while (size < this.len + n + 2) size *= 2;
+ const next = Buffer.alloc(size);
+ this.buf.copy(next, 0, 0, this.len);
+ this.buf = next;
+ }
+ }
+ writeBits(value, length) {
+ if (length <= 0) return;
+ this.bitBuf = (this.bitBuf << length) | (value & ((1 << length) - 1));
+ this.bitCount += length;
+ while (this.bitCount >= 8) {
+ this.bitCount -= 8;
+ const byte = (this.bitBuf >>> this.bitCount) & 0xff;
+ this.ensure(2);
+ this.buf[this.len++] = byte;
+ if (byte === 0xff) this.buf[this.len++] = 0; // byte stuffing
+ }
+ this.bitBuf &= (1 << this.bitCount) - 1;
+ }
+ /** Pad the final byte with 1-bits (JPEG spec) and return the bytes. */
+ flush() {
+ if (this.bitCount > 0) {
+ const padBits = 8 - this.bitCount;
+ this.writeBits((1 << padBits) - 1, padBits);
+ }
+ return this.buf.subarray(0, this.len);
+ }
+}
+
+const COS = (() => {
+ const t = new Float64Array(8 * 8);
+ for (let u = 0; u < 8; u++) {
+ for (let x = 0; x < 8; x++) t[u * 8 + x] = Math.cos(((2 * x + 1) * u * Math.PI) / 16);
+ }
+ return t;
+})();
+
+/** Separable forward DCT; input/output in natural (row-major) order. */
+function fdct(block) {
+ const tmp = new Float64Array(64);
+ const out = new Float64Array(64);
+ for (let u = 0; u < 8; u++) {
+ const cu = u === 0 ? Math.SQRT1_2 / 2 : 0.5;
+ for (let x = 0; x < 8; x++) {
+ let s = 0;
+ for (let y = 0; y < 8; y++) s += block[y * 8 + x] * COS[u * 8 + y];
+ tmp[u * 8 + x] = s * cu;
+ }
+ }
+ for (let u = 0; u < 8; u++) {
+ for (let v = 0; v < 8; v++) {
+ const cv = v === 0 ? Math.SQRT1_2 / 2 : 0.5;
+ let s = 0;
+ for (let x = 0; x < 8; x++) s += tmp[u * 8 + x] * COS[v * 8 + x];
+ out[u * 8 + v] = s * cv;
+ }
+ }
+ return out;
+}
+
+/** Bit size category of a DC/AC value (JPEG additional-bits convention). */
+function category(value) {
+ let v = Math.abs(value);
+ let n = 0;
+ while (v) { n++; v >>= 1; }
+ return n;
+}
+
+/** Encode one 8x8 block; returns the new DC predictor. Quant table is in
+ * NATURAL order, as are the DCT coefficients; the RLE walks zigzag. */
+function encodeBlock(coeffs, quant, dcCodes, acCodes, w, lastDC) {
+ let dc = 0;
+ {
+ let v = coeffs[0] / quant[0];
+ dc = v > 0 ? Math.round(v) : -Math.round(-v);
+ }
+ const diff = dc - lastDC;
+ const cat = category(diff);
+ const dcH = dcCodes.get(cat);
+ if (!dcH) throw new Error(`no DC code for category ${cat}`);
+ w.writeBits(dcH.code, dcH.length);
+ if (cat > 0) {
+ const bits = diff < 0 ? diff + (1 << cat) - 1 : diff;
+ w.writeBits(bits, cat);
+ }
+ // Quantize in natural order, then RLE in zigzag order.
+ const q = new Map();
+ let lastNonZeroZ = 0;
+ for (let i = 1; i < 64; i++) {
+ const z = ZIGZAG[i];
+ let v = coeffs[z] / quant[z];
+ v = v > 0 ? Math.round(v) : -Math.round(-v);
+ if (v !== 0) {
+ q.set(i, v);
+ lastNonZeroZ = i;
+ }
+ }
+ let run = 0;
+ for (let i = 1; i <= lastNonZeroZ; i++) {
+ const v = q.get(i) ?? 0;
+ if (v === 0) { run++; continue; }
+ while (run > 15) {
+ const zrl = acCodes.get(0xf0);
+ w.writeBits(zrl.code, zrl.length);
+ run -= 16;
+ }
+ const vcat = category(v);
+ const sym = (run << 4) | vcat;
+ const ac = acCodes.get(sym);
+ if (!ac) throw new Error(`missing AC Huffman code for symbol 0x${sym.toString(16)}`);
+ w.writeBits(ac.code, ac.length);
+ const bits = v < 0 ? v + (1 << vcat) - 1 : v;
+ w.writeBits(bits, vcat);
+ run = 0;
+ }
+ if (lastNonZeroZ < 63) {
+ const eob = acCodes.get(0x00);
+ w.writeBits(eob.code, eob.length);
+ }
+ return dc;
+}
+
+function segment(markerByte, body) {
+ return Buffer.concat([Buffer.from([0xff, markerByte, (body.length + 2) >> 8, (body.length + 2) & 0xff]), body]);
+}
+
+/**
+ * Encode interleaved RGB rows as baseline JPEG (4:2:0, JFIF).
+ * @param {Uint8Array|Buffer} rgb
+ * @param {number} width
+ * @param {number} height
+ * @param {number} [stride] bytes per row (default width*3)
+ * @param {number} [quality] 1-100
+ * @returns {Buffer}
+ */
+export function encodeJPEG(rgb, width, height, stride = width * 3, quality = 80) {
+ const lumQuant = scaledQuantTable(STD_LUM_Q, quality);
+ const chromaQuant = scaledQuantTable(STD_CHROMA_Q, quality);
+ const dcLumCodes = buildHuffmanCodes(DC_LUM_LENGTHS, DC_LUM_SYMBOLS);
+ const acLumCodes = buildHuffmanCodes(AC_LUM_LENGTHS, AC_LUM_SYMBOLS);
+ const dcChromaCodes = buildHuffmanCodes(DC_CHROMA_LENGTHS, DC_CHROMA_SYMBOLS);
+ const acChromaCodes = buildHuffmanCodes(AC_CHROMA_LENGTHS, AC_CHROMA_SYMBOLS);
+
+ const mcusX = Math.ceil(width / 16);
+ const mcusY = Math.ceil(height / 16);
+ const yW = mcusX * 16, yH = mcusY * 16;
+ const cW = mcusX * 8, cH = mcusY * 8;
+ const yPlane = new Float64Array(yW * yH);
+ const cbAccum = new Float64Array(cW * cH);
+ const crAccum = new Float64Array(cW * cH);
+ const chromaCount = new Uint16Array(cW * cH);
+
+ // RGB → YCbCr (BT.601 full range); chroma box-averaged 2x2.
+ for (let row = 0; row < height; row++) {
+ const src = row * stride;
+ const yRow = row * yW;
+ const cRow = (row >> 1) * cW;
+ for (let col = 0; col < width; col++) {
+ const o = src + col * 3;
+ const r = rgb[o], g = rgb[o + 1], b = rgb[o + 2];
+ yPlane[yRow + col] = 0.299 * r + 0.587 * g + 0.114 * b;
+ const co = cRow + (col >> 1);
+ cbAccum[co] += 128 - 0.168736 * r - 0.331264 * g + 0.5 * b;
+ crAccum[co] += 128 + 0.5 * r - 0.418688 * g - 0.081312 * b;
+ chromaCount[co]++;
+ }
+ }
+ const cbPlane = new Float64Array(cW * cH);
+ const crPlane = new Float64Array(cW * cH);
+ for (let i = 0; i < cbAccum.length; i++) {
+ const n = Math.max(1, chromaCount[i]);
+ cbPlane[i] = cbAccum[i] / n;
+ crPlane[i] = crAccum[i] / n;
+ }
+
+ const w = new BitWriter();
+ let lastY = 0, lastCB = 0, lastCR = 0;
+ const block = new Float64Array(64);
+
+ function encodePlaneBlock(plane, planeW, bx, by, quant, dcCodes, acCodes, lastDC) {
+ const baseY = by * 8;
+ const baseX = bx * 8;
+ for (let y = 0; y < 8; y++) {
+ const src = (baseY + y) * planeW + baseX;
+ for (let x = 0; x < 8; x++) block[y * 8 + x] = plane[src + x] - 128;
+ }
+ return encodeBlock(fdct(block), quant, dcCodes, acCodes, w, lastDC);
+ }
+
+ for (let my = 0; my < mcusY; my++) {
+ for (let mx = 0; mx < mcusX; mx++) {
+ for (let i = 0; i < 4; i++) {
+ lastY = encodePlaneBlock(yPlane, yW, mx * 2 + (i % 2), my * 2 + (i >> 1), lumQuant, dcLumCodes, acLumCodes, lastY);
+ }
+ lastCB = encodePlaneBlock(cbPlane, cW, mx, my, chromaQuant, dcChromaCodes, acChromaCodes, lastCB);
+ lastCR = encodePlaneBlock(crPlane, cW, mx, my, chromaQuant, dcChromaCodes, acChromaCodes, lastCR);
+ }
+ }
+ const scan = w.flush();
+
+ const app0 = segment(0xe0, Buffer.from([0x4a, 0x46, 0x49, 0x46, 0x00, 0x01, 0x01, 0x00, 0, 1, 0, 1, 0, 0]));
+ // DQT payloads carry values in ZIGZAG order (the tables here are natural).
+ const zig = (table) => Buffer.from(ZIGZAG.map((z) => table[z]));
+ const dqt = Buffer.concat([
+ segment(0xdb, Buffer.concat([Buffer.from([0x00]), zig(lumQuant)])),
+ segment(0xdb, Buffer.concat([Buffer.from([0x01]), zig(chromaQuant)])),
+ ]);
+ const sof0Body = Buffer.alloc(15); // 1+2+2+1 + 3×3-byte component entries
+ sof0Body.writeUInt8(8, 0);
+ sof0Body.writeUInt16BE(height, 1);
+ sof0Body.writeUInt16BE(width, 3);
+ sof0Body.writeUInt8(3, 5);
+ sof0Body.writeUInt8(1, 6); sof0Body.writeUInt8(0x22, 7); sof0Body.writeUInt8(0, 8);
+ sof0Body.writeUInt8(2, 9); sof0Body.writeUInt8(0x11, 10); sof0Body.writeUInt8(1, 11);
+ sof0Body.writeUInt8(3, 12); sof0Body.writeUInt8(0x11, 13); sof0Body.writeUInt8(1, 14);
+ const sof0 = segment(0xc0, sof0Body);
+ const dht = Buffer.concat([
+ dhtSegment(0, 0, DC_LUM_LENGTHS, DC_LUM_SYMBOLS),
+ dhtSegment(1, 0, AC_LUM_LENGTHS, AC_LUM_SYMBOLS),
+ dhtSegment(0, 1, DC_CHROMA_LENGTHS, DC_CHROMA_SYMBOLS),
+ dhtSegment(1, 1, AC_CHROMA_LENGTHS, AC_CHROMA_SYMBOLS),
+ ]);
+ const sosBody = Buffer.alloc(10);
+ sosBody.writeUInt8(3, 0);
+ sosBody.writeUInt8(1, 1); sosBody.writeUInt8(0x00, 2);
+ sosBody.writeUInt8(2, 3); sosBody.writeUInt8(0x11, 4);
+ sosBody.writeUInt8(3, 5); sosBody.writeUInt8(0x11, 6);
+ sosBody.writeUInt8(0, 7); sosBody.writeUInt8(63, 8); sosBody.writeUInt8(0, 9);
+ const sos = segment(0xda, sosBody);
+
+ return Buffer.concat([
+ Buffer.from([0xff, 0xd8]), app0, dqt, sof0, dht, sos, scan, Buffer.from([0xff, 0xd9]),
+ ]);
+}
diff --git a/src/desktop/linux.js b/src/desktop/linux.js
new file mode 100644
index 0000000..80c7bf7
--- /dev/null
+++ b/src/desktop/linux.js
@@ -0,0 +1,806 @@
+/**
+ * axuse — Linux desktop driver over the zero-dependency X11 client.
+ *
+ * Lineage: Windows uses a resident PowerShell + SendInput driver and macOS
+ * a JXA/CoreGraphics one; both shell out to the OS's scriptable surface.
+ * Linux has no guaranteed scripting surface (xdotool/wmctrl are extra apt
+ * packages), so this driver speaks X11 directly through src/desktop/x11.js
+ * — same contract as the Windows driver (coordinate+vision mode):
+ *
+ * - input: XTEST FakeInput (trusted, server-synthesized events — the X
+ * equivalent of SendInput), strictly ordered like the Windows driver
+ * - screenshots: GetImage (ZPixmap) → baseline JPEG (src/desktop/jpeg.js)
+ * with the same budget/quality retry contract as GDI+ on Windows
+ * - window list/activate: EWMH properties on the root window
+ * (_NET_CLIENT_LIST / _NET_WM_NAME / _NET_WM_PID / _NET_ACTIVE_WINDOW)
+ *
+ * No OS accessibility tree on X11 v1 (AT-SPI is a bus protocol out of scope
+ * here) — the ax_* browser tools remain the structure-first surface, and
+ * this driver is the fallback for native apps, exactly like Windows today.
+ */
+import { AxError, Codes } from "../core/errors.js";
+import { KEYS, parseCombo } from "../core/actuate.js";
+import { DEFAULT_BUDGET_BYTES } from "../core/screenshot.js";
+import { X11Client, FakeEventType } from "./x11.js";
+import { encodeJPEG } from "./jpeg.js";
+import { spawn } from "node:child_process";
+
+/** Shared KEYS-table names → X keysyms (Latin-1 + the standard ranges). */
+const KEY_NAME_TO_KEYSYM = {
+ enter: 0xff0d, tab: 0xff09, escape: 0xff1b, esc: 0xff1b,
+ backspace: 0xff08, delete: 0xffff, del: 0xffff,
+ up: 0xff52, down: 0xff54, left: 0xff51, right: 0xff53,
+ home: 0xff50, end: 0xff57, pageup: 0xff55, pagedown: 0xff56,
+ space: 0x20,
+ ctrl: 0xffe3, control: 0xffe3, shift: 0xffe1, alt: 0xffe9,
+ meta: 0xffeb, cmd: 0xffeb, command: 0xffeb,
+};
+for (let i = 1; i <= 12; i++) KEY_NAME_TO_KEYSYM["f" + i] = 0xffbe + (i - 1);
+
+/** keysym for a KEYS-table name, or null. */
+function keysymFor(name) {
+ if (name.length === 1) {
+ const c = name.charCodeAt(0);
+ if ((c >= 0x61 && c <= 0x7a) || (c >= 0x30 && c <= 0x39)) return c; // a-z, 0-9
+ if (c >= 0x21 && c <= 0xff) return c; // printable latin-1 punctuation etc.
+ }
+ return KEY_NAME_TO_KEYSYM[name] ?? null;
+}
+
+const BUTTON_DETAIL = { left: 1, middle: 2, right: 3 };
+
+/** X11/EWMH driver. Ops are serialized (input ordering must never
+ * interleave — the same discipline as the resident Windows driver). */
+export class LinuxDriver {
+ /** @param {string} [display] @param {{atspiSessionBus?: string}} [opts] */
+ constructor(display, opts = {}) {
+ this.client = new X11Client(display);
+ this.atspiSessionBus = opts.atspiSessionBus;
+ this.chain = Promise.resolve();
+ }
+
+ /** Serialized op execution; the X11 connection opens lazily on first use. */
+ op(fn) {
+ const run = async () => {
+ if (!this.client.connected) await this.client.connect();
+ return fn(this.client);
+ };
+ this.chain = this.chain.then(run, run);
+ return this.chain;
+ }
+
+ /* ------------------------------------------------------------- capture */
+
+ async screenInfo() {
+ return this.op(async (c) => {
+ const g = await c.getGeometry(c.rootWindow);
+ const monitors = await c.getMonitors();
+ return {
+ ok: true,
+ x: 0,
+ y: 0,
+ w: g.w,
+ h: g.h,
+ coordinate_space: "X11 root window pixels (spans the monitors of the display's default screen)",
+ note: "X11/XWayland driver (XTEST input, GetImage capture, EWMH windows)",
+ ...(monitors.length ? { monitors } : {}),
+ };
+ });
+ }
+
+ /** GetImage → RGB rows honoring bpp/stride/byte-order. */
+ async grabRGB(c, x, y, w, h) {
+ const img = await c.getImage(x, y, w, h);
+ const { bytesPerPixel: bpp, stride, data, byteOrder } = img;
+ if (bpp !== 4 && bpp !== 3) {
+ throw new AxError(Codes.DESKTOP_OP_FAILED,
+ `Unsupported X11 pixmap format (${bpp} bytes/pixel, depth ${img.depth})`,
+ { retry: "never" });
+ }
+ const rgb = Buffer.alloc(w * h * 3);
+ for (let row = 0; row < h; row++) {
+ const src = row * stride;
+ const dst = row * w * 3;
+ if (bpp === 4) {
+ if (byteOrder === 0) { // LSBFirst: [B, G, R, X]
+ for (let col = 0; col < w; col++) {
+ const o = src + col * 4;
+ rgb[dst + col * 3] = data[o + 2];
+ rgb[dst + col * 3 + 1] = data[o + 1];
+ rgb[dst + col * 3 + 2] = data[o];
+ }
+ } else { // MSBFirst: [X, R, G, B]
+ for (let col = 0; col < w; col++) {
+ const o = src + col * 4;
+ rgb[dst + col * 3] = data[o + 1];
+ rgb[dst + col * 3 + 1] = data[o + 2];
+ rgb[dst + col * 3 + 2] = data[o + 3];
+ }
+ }
+ } else { // 3 bytes: LSB [B, G, R], MSB [R, G, B]
+ if (byteOrder === 0) {
+ for (let col = 0; col < w; col++) {
+ const o = src + col * 3;
+ rgb[dst + col * 3] = data[o + 2];
+ rgb[dst + col * 3 + 1] = data[o + 1];
+ rgb[dst + col * 3 + 2] = data[o];
+ }
+ } else {
+ data.copy(rgb, dst, src, src + w * 3);
+ }
+ }
+ }
+ return rgb;
+ }
+
+ /** 2× box downscale of packed RGB rows (vision agents don't need 1080p,
+ * and each halving quarters the JPEG cost). */
+ static downscale2x(rgb, w, h) {
+ const nw = Math.max(1, w >> 1);
+ const nh = Math.max(1, h >> 1);
+ const out = Buffer.alloc(nw * nh * 3);
+ for (let y = 0; y < nh; y++) {
+ const r0 = (y * 2) * w * 3;
+ const r1 = (y * 2 + 1 < h ? y * 2 + 1 : y * 2) * w * 3;
+ const d = y * nw * 3;
+ for (let x = 0; x < nw; x++) {
+ const c0 = (x * 2) * 3;
+ const c1 = (x * 2 + 1 < w ? x * 2 + 1 : x * 2) * 3;
+ out[d + x * 3] = (rgb[r0 + c0] + rgb[r0 + c1] + rgb[r1 + c0] + rgb[r1 + c1]) >> 2;
+ out[d + x * 3 + 1] = (rgb[r0 + c0 + 1] + rgb[r0 + c1 + 1] + rgb[r1 + c0 + 1] + rgb[r1 + c1 + 1]) >> 2;
+ out[d + x * 3 + 2] = (rgb[r0 + c0 + 2] + rgb[r0 + c1 + 2] + rgb[r1 + c0 + 2] + rgb[r1 + c1 + 2]) >> 2;
+ }
+ }
+ return { rgb: out, w: nw, h: nh };
+ }
+
+ async screenshot({ x, y, w, h, budgetBytes = DEFAULT_BUDGET_BYTES, quality = 80 } = {}) {
+ return this.op(async (c) => {
+ let rx = Math.max(0, Math.floor(x ?? 0));
+ let ry = Math.max(0, Math.floor(y ?? 0));
+ let rw = w === undefined ? c.screenWidth - rx : Math.floor(w);
+ let rh = h === undefined ? c.screenHeight - ry : Math.floor(h);
+ // Clamp: GetImage rejects regions hanging off the root.
+ rw = Math.min(rw, c.screenWidth - rx);
+ rh = Math.min(rh, c.screenHeight - ry);
+ if (rw < 1 || rh < 1) {
+ throw new AxError(Codes.COMMAND_REJECTED,
+ `Capture region ${JSON.stringify({ x: rx, y: ry, w: rw, h: rh })} is outside the screen (${c.screenWidth}x${c.screenHeight})`,
+ { retry: "reobserve" });
+ }
+ // Budget ladder, cheapest first: full quality, then q45, then halving
+ // the resolution twice (each halving also quarters the encode cost).
+ // Vision workflows keep a usable image instead of a typed dead end.
+ let { rgb: pixels, w: iw, h: ih } = { rgb: null, w: rw, h: rh };
+ const grabFull = async () => {
+ pixels = await this.grabRGB(c, rx, ry, rw, rh);
+ iw = rw; ih = rh;
+ };
+ await grabFull();
+ let jpg = encodeJPEG(pixels, iw, ih, iw * 3, quality);
+ let usedQuality = quality;
+ if (jpg.length > budgetBytes && quality > 45) {
+ jpg = encodeJPEG(pixels, iw, ih, iw * 3, 45);
+ usedQuality = 45;
+ }
+ for (const _ of [1, 2]) {
+ if (jpg.length <= budgetBytes) break;
+ if (iw <= 160 || ih <= 160) break; // keep text legible
+ const small = LinuxDriver.downscale2x(pixels, iw, ih);
+ pixels = small.rgb; iw = small.w; ih = small.h;
+ jpg = encodeJPEG(pixels, iw, ih, iw * 3, 70);
+ usedQuality = 70;
+ }
+ const bytes = jpg.length;
+ if (bytes > budgetBytes) {
+ throw new AxError(Codes.SCREENSHOT_BUDGET_EXCEEDED,
+ `Desktop screenshot is ${bytes} bytes, over the ${budgetBytes} byte budget.`,
+ { hint: "Pass a crop region ({x,y,w,h} in pixels from screen_info).", retry: "never" });
+ }
+ return {
+ base64: jpg.toString("base64"),
+ mime: "image/jpeg",
+ bytes,
+ budget_bytes: budgetBytes,
+ quality: usedQuality,
+ ...(iw !== rw || ih !== rh ? { downscaled: `${rw}x${rh} → ${iw}x${ih}` } : {}),
+ };
+ });
+ }
+
+ /* --------------------------------------------------------------- input */
+
+ async move(x, y) {
+ await this.op((c) => c.warpPointer(x, y));
+ return { action_sent: "sent" };
+ }
+
+ async click({ x, y, button = "left", double = false } = {}) {
+ const detail = BUTTON_DETAIL[button];
+ if (!detail) throw new AxError(Codes.COMMAND_REJECTED, `button must be left|middle|right, got "${button}"`);
+ await this.op(async (c) => {
+ await c.warpPointer(x, y);
+ await c.fakeInput({ type: FakeEventType.ButtonPress, detail });
+ await c.fakeInput({ type: FakeEventType.ButtonRelease, detail });
+ if (double) {
+ await c.fakeInput({ type: FakeEventType.ButtonPress, detail });
+ await c.fakeInput({ type: FakeEventType.ButtonRelease, detail });
+ }
+ });
+ return { action_sent: "sent" };
+ }
+
+ async down(button = "left") {
+ const detail = BUTTON_DOWN[button] ?? 1;
+ await this.op((c) => c.fakeInput({ type: FakeEventType.ButtonPress, detail }));
+ return { action_sent: "sent" };
+ }
+
+ async up(button = "left") {
+ const detail = BUTTON_DOWN[button] ?? 1;
+ await this.op((c) => c.fakeInput({ type: FakeEventType.ButtonRelease, detail }));
+ return { action_sent: "sent" };
+ }
+
+ /** Wheel: X buttons 4/5 (vertical) and 6/7 (horizontal); one FakeInput
+ * click per notch (120 delta per notch, matching the Windows driver's
+ * SendInput WHEEL_DELTA convention). */
+ async wheel(dx = 0, dy = 0) {
+ await this.op(async (c) => {
+ const notches = (d) => Math.max(1, Math.min(50, Math.round(Math.abs(d) / 120) || 1));
+ if (dy < 0) for (let i = 0; i < notches(dy); i++) await c.fakeInput({ type: FakeEventType.ButtonPress, detail: 4 }), await c.fakeInput({ type: FakeEventType.ButtonRelease, detail: 4 });
+ if (dy > 0) for (let i = 0; i < notches(dy); i++) await c.fakeInput({ type: FakeEventType.ButtonPress, detail: 5 }), await c.fakeInput({ type: FakeEventType.ButtonRelease, detail: 5 });
+ if (dx < 0) for (let i = 0; i < notches(dx); i++) await c.fakeInput({ type: FakeEventType.ButtonPress, detail: 6 }), await c.fakeInput({ type: FakeEventType.ButtonRelease, detail: 6 });
+ if (dx > 0) for (let i = 0; i < notches(dx); i++) await c.fakeInput({ type: FakeEventType.ButtonPress, detail: 7 }), await c.fakeInput({ type: FakeEventType.ButtonRelease, detail: 7 });
+ });
+ return { action_sent: "sent" };
+ }
+
+ async keyTapKeysym(keysym) {
+ await this.op(async (c) => {
+ const key = c.keyForKeysym(keysym);
+ if (!key) throw untypeableKeysym(keysym);
+ if (key.shift) {
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: c.keyForKeysym(0xffe1).keycode });
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: key.keycode });
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: key.keycode });
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: c.keyForKeysym(0xffe1).keycode });
+ } else {
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: key.keycode });
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: key.keycode });
+ }
+ });
+ return { action_sent: "sent" };
+ }
+
+ async keyDownKeysym(keysym) {
+ await this.op(async (c) => {
+ const key = c.keyForKeysym(keysym);
+ if (!key) throw untypeableKeysym(keysym);
+ if (key.shift) await c.fakeInput({ type: FakeEventType.KeyPress, detail: c.keyForKeysym(0xffe1).keycode });
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: key.keycode });
+ });
+ return { action_sent: "sent" };
+ }
+
+ async keyUpKeysym(keysym) {
+ await this.op(async (c) => {
+ const key = c.keyForKeysym(keysym);
+ if (!key) throw untypeableKeysym(keysym);
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: key.keycode });
+ if (key.shift) await c.fakeInput({ type: FakeEventType.KeyRelease, detail: c.keyForKeysym(0xffe1).keycode });
+ });
+ return { action_sent: "sent" };
+ }
+
+ /** Unicode text at the caret. Two paths, chosen per payload:
+ * - short, keymap-typeable text → one XTEST keysym press per char (the
+ * same per-char discipline as the Windows KEYEVENTF_UNICODE driver)
+ * - long (≥48 chars) or non-keymap text with xclip available →
+ * clipboard set + trusted ctrl+v paste (fast, layout-proof; the
+ * clipboard side effect is reported honestly). */
+ async type(text) {
+ const s = String(text);
+ // Load the server keymap (lazy connect) before probing typeability —
+ // an empty map must not push short text onto the clipboard path.
+ await this.op(async () => {});
+ const chars = [...s];
+ const typeable = (ch) => {
+ if (ch === "\n" || ch === "\r") return true; // mapped to Return below
+ if (ch === "\t") return true; // mapped to Tab below
+ let keysym = ch.codePointAt(0);
+ if (keysym > 0xff) keysym = 0x01000000 | keysym;
+ return !!this.client.keysymToKey.get(keysym);
+ };
+ const wantsClipboard = chars.length >= 48 || chars.some((ch) => !typeable(ch));
+ if (wantsClipboard && await LinuxDriver.clipboardAvailable()) {
+ await this.setClipboard(s);
+ await this.press("ctrl+v");
+ return { action_sent: "sent", via: "clipboard-paste", clipboard_replaced: true };
+ }
+
+ await this.op(async (c) => {
+ for (const ch of chars) {
+ let keysym;
+ if (ch === "\n" || ch === "\r") keysym = 0xff0d; // Return
+ else if (ch === "\t") keysym = 0xff09; // Tab
+ else {
+ keysym = ch.codePointAt(0);
+ if (keysym > 0xff) keysym = 0x01000000 | keysym; // Unicode range
+ }
+ const key = c.keyForKeysym(keysym);
+ if (!key) {
+ throw new AxError(Codes.COMMAND_REJECTED,
+ `Character ${JSON.stringify(ch)} has no keysym/keycode on this keyboard layout (XTEST types through the server keymap).`,
+ { hint: "Install xclip for the automatic clipboard-paste path, or fill this text via the browser tools.", retry: "never" });
+ }
+ if (key.shift) await c.fakeInput({ type: FakeEventType.KeyPress, detail: c.keyForKeysym(0xffe1).keycode });
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: key.keycode });
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: key.keycode });
+ if (key.shift) await c.fakeInput({ type: FakeEventType.KeyRelease, detail: c.keyForKeysym(0xffe1).keycode });
+ }
+ });
+ return { action_sent: "sent", via: "keysym" };
+ }
+
+ /** xclip presence (the clipboard helper), memoized. */
+ static async clipboardAvailable() {
+ if (LinuxDriver._clipboardHelper === undefined) {
+ LinuxDriver._clipboardHelper = await new Promise((resolve) => {
+ const probe = spawn("which", ["xclip"], { stdio: "ignore" });
+ probe.on("exit", (code) => resolve(code === 0));
+ probe.on("error", () => resolve(false));
+ });
+ }
+ return LinuxDriver._clipboardHelper;
+ }
+
+ /** Set the CLIPBOARD selection through xclip (waits until xclip owns it).
+ * The driver's OWN display is passed explicitly — never the ambient one. */
+ async setClipboard(text) {
+ const ok = await new Promise((resolve) => {
+ const child = spawn("xclip", ["-display", this.client.display || ":0", "-selection", "clipboard", "-in"], { stdio: ["pipe", "ignore", "ignore"] });
+ child.on("error", () => resolve(false));
+ child.on("exit", (code) => resolve(code === 0));
+ child.stdin.on("error", () => {});
+ child.stdin.end(String(text), "utf8");
+ });
+ if (!ok) {
+ throw new AxError(Codes.DESKTOP_OP_FAILED,
+ "xclip failed while setting the clipboard (is it installed and is an X server reachable?).",
+ { retry: "retry" });
+ }
+ return true;
+ }
+
+ /** Combo press ("ctrl+s") mapped through the shared KEYS table. */
+ async press(combo) {
+ const { mods } = parseCombo(combo); // validates; throws typed errors
+ const mainName = comboMainName(combo);
+ const modKeysyms = [];
+ for (const m of mods) {
+ const ks = keysymFor(m);
+ if (!ks) throw new AxError(Codes.COMMAND_REJECTED, `Unknown modifier "${m}" in combo "${combo}"`);
+ modKeysyms.push(ks);
+ }
+ const mainKeysym = keysymFor(mainName);
+ if (!mainKeysym) {
+ throw new AxError(Codes.COMMAND_REJECTED, `Unknown key "${mainName}" in combo "${combo}"`);
+ }
+ await this.op(async (c) => {
+ for (const ks of modKeysyms) {
+ const key = c.keyForKeysym(ks);
+ if (!key) throw untypeableKeysym(ks);
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: key.keycode });
+ }
+ const mainKey = c.keyForKeysym(mainKeysym);
+ if (!mainKey) throw untypeableKeysym(mainKeysym);
+ if (mainKey.shift && !modKeysyms.length) {
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: c.keyForKeysym(0xffe1).keycode });
+ }
+ await c.fakeInput({ type: FakeEventType.KeyPress, detail: mainKey.keycode });
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: mainKey.keycode });
+ if (mainKey.shift && !modKeysyms.length) {
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: c.keyForKeysym(0xffe1).keycode });
+ }
+ for (let i = modKeysyms.length - 1; i >= 0; i--) {
+ const key = c.keyForKeysym(modKeysyms[i]);
+ await c.fakeInput({ type: FakeEventType.KeyRelease, detail: key.keycode });
+ }
+ });
+ return { action_sent: "sent", combo };
+ }
+
+ /** Drag via move/down/move/up (interpolated like both sibling drivers). */
+ async drag({ from, to, steps = 12 } = {}) {
+ await this.op(async (c) => {
+ await c.warpPointer(from.x, from.y);
+ await c.fakeInput({ type: FakeEventType.ButtonPress, detail: 1 });
+ for (let i = 1; i <= steps; i++) {
+ const x = Math.round(from.x + ((to.x - from.x) * i) / steps);
+ const y = Math.round(from.y + ((to.y - from.y) * i) / steps);
+ await c.fakeInput({ type: FakeEventType.MotionNotify, detail: 0, x, y });
+ }
+ await c.fakeInput({ type: FakeEventType.ButtonRelease, detail: 1 });
+ });
+ return { action_sent: "sent" };
+ }
+
+ /* ------------------------------------------------------------- windows */
+
+ async windows() {
+ return this.op(async (c) => {
+ const ids = await c.getPropertyU32s(c.rootWindow, "_NET_CLIENT_LIST");
+ let hiddenAtom = 0;
+ try { hiddenAtom = await c.internAtom("_NET_WM_STATE_HIDDEN"); } catch {}
+ const rows = [];
+ for (const win of ids) {
+ const row = { hwnd: win, window_id: win };
+ try {
+ if (hiddenAtom) {
+ const state = await c.getPropertyU32s(win, "_NET_WM_STATE").catch(() => []);
+ if (state.includes(hiddenAtom)) continue; // minimized windows are not visible
+ }
+ const nameProp = await c.getProperty(win, "_NET_WM_NAME").catch(() => null);
+ let title = "";
+ if (nameProp?.value?.length) title = nameProp.value.toString("utf8");
+ if (!title) {
+ const wmName = await c.getProperty(win, "WM_NAME").catch(() => null);
+ if (wmName?.value?.length) title = wmName.value.toString("latin1");
+ }
+ row.title = title.replace(/\s+/g, " ").trim();
+ const pidProp = await c.getProperty(win, "_NET_WM_PID").catch(() => null);
+ if (pidProp?.format === 32 && pidProp.value.length >= 4) row.pid = pidProp.value.readUInt32LE(0);
+ const classProp = await c.getProperty(win, "WM_CLASS").catch(() => null);
+ if (classProp?.value?.length) {
+ const parts = classProp.value.toString("latin1").split("\0").filter(Boolean);
+ row.application = parts[parts.length - 1] || undefined;
+ }
+ try {
+ const geom = await c.getGeometry(win);
+ const origin = await c.translateCoordinates(win);
+ row.bounds = { x: origin.x, y: origin.y, w: geom.w, h: geom.h };
+ } catch {}
+ } catch {}
+ rows.push(row);
+ }
+ return { windows: rows };
+ });
+ }
+
+ async activate({ title, hwnd, pid, windowId } = {}) {
+ return this.op(async (c) => {
+ let target = hwnd ?? windowId ?? null;
+ if (!target && pid !== undefined) {
+ const ids = await c.getPropertyU32s(c.rootWindow, "_NET_CLIENT_LIST");
+ for (const win of ids) {
+ const p = await c.getProperty(win, "_NET_WM_PID").catch(() => null);
+ if (p?.format === 32 && p.value.length >= 4 && p.value.readUInt32LE(0) === pid) { target = win; break; }
+ }
+ }
+ if (!target && title) {
+ const needle = String(title).toLowerCase();
+ const ids = await c.getPropertyU32s(c.rootWindow, "_NET_CLIENT_LIST");
+ for (const win of ids) {
+ const nameProp = await c.getProperty(win, "_NET_WM_NAME").catch(() => null);
+ const t = nameProp?.value?.length ? nameProp.value.toString("utf8") : "";
+ if (t.toLowerCase().includes(needle)) { target = win; break; }
+ }
+ }
+ if (!target) {
+ throw new AxError(Codes.DESKTOP_OP_FAILED,
+ "Window not found (activate needs a title substring, pid, or window_id from ax_desktop_windows).",
+ { hint: "List windows with ax_desktop_windows and retry.", retry: "retry" });
+ }
+ const atom = await c.internAtom("_NET_ACTIVE_WINDOW");
+ // Source indication 1 (application); the target window rides data.l[2].
+ await c.sendClientMessage(target, atom, [1, 0, target, 0, 0]);
+ // WM-less fallback (plain Xvfb): focus the window directly so
+ // subsequent XTEST key events have a target.
+ try { await c.setInputFocus(target); } catch {}
+ return { activated: true };
+ });
+ }
+
+ /* -------------------------------------------------- AT-SPI structure ops */
+
+ /** Lazy AT-SPI connection (structure-first layer, like macOS AX). */
+ async ensureAtspi() {
+ if (!this.atspi || !this.atspi.connected) {
+ const { AtspiClient } = await import("./atspi.js");
+ this.atspi = await new AtspiClient({ sessionBusAddress: this.atspiSessionBus }).connect();
+ }
+ return this.atspi;
+ }
+
+ /** Is the AT-SPI structure layer actually usable (tools mount only when
+ * true — matching the macOS driver's AX gating, where tools exist only
+ * when the platform AX tree answers)? Cached and BOUNDED: activating a
+ * dead at-spi registry can take seconds, so a slow/dead probe unmounts
+ * the ax tools instead of stalling startup — and its connection attempt
+ * is closed when it (eventually) lands, so nothing dangles on the event
+ * loop. */
+ async axCapable() {
+ if (this.#axCapableCache !== null) return this.#axCapableCache;
+ const probe = this.ensureAtspi()
+ .then(async (atspi) => (await atspi.apps()).some((a) => a.name))
+ .catch(() => false);
+ const verdict = await Promise.race([
+ probe,
+ new Promise((resolve) => setTimeout(() => resolve(null), 2_500)),
+ ]);
+ this.#axCapableCache = verdict === true;
+ if (verdict !== true) {
+ // Close a connection that already landed; a late one closes itself.
+ if (this.atspi) {
+ try { this.atspi.close(); } catch {}
+ this.atspi = null;
+ } else {
+ probe.then(() => {
+ if (this.#axCapableCache === false) {
+ try { this.atspi?.close(); } catch {}
+ this.atspi = null;
+ }
+ }).catch(() => {});
+ }
+ }
+ return this.#axCapableCache;
+ }
+
+ #axCapableCache = null;
+
+ /** Resolve the target app root (pid via EWMH cross-reference, exact name,
+ * or the frontmost app). Returns the AT-SPI ref. */
+ async resolveAtspiApp({ pid, name, front } = {}) {
+ const atspi = await this.ensureAtspi();
+ const apps = await atspi.apps();
+ if (!apps.length) {
+ throw new AxError(Codes.DESKTOP_OP_FAILED, "No AT-SPI applications are registered.",
+ { hint: "Structure-first desktop control needs apps exposing AT-SPI (GTK/Qt apps do).", retry: "reobserve" });
+ }
+ if (name) {
+ const hit = apps.find((a) => a.name === name);
+ if (!hit) {
+ throw new AxError(Codes.REF_NOT_FOUND, `No AT-SPI application named "${name}".`,
+ { hint: `Available: ${apps.map((a) => a.name).filter(Boolean).join(", ") || "none"}.`, retry: "reobserve" });
+ }
+ return hit.ref;
+ }
+ if (pid !== undefined) {
+ // AT-SPI app roots carry the app's accessible name; map the EWMH pid
+ // → WM_CLASS instance → AT-SPI app name (windows() already knows pids).
+ const target = String(pid);
+ const rows = (await this.windows()).windows.filter((w) => String(w.pid) === target);
+ const candidates = new Set(rows.map((w) => (w.application ?? "").toLowerCase()));
+ const hit = apps.find((a) => candidates.has(a.name.toLowerCase()));
+ if (!hit) {
+ throw new AxError(Codes.REF_NOT_FOUND, `No AT-SPI application matches pid ${pid} (window class: ${[...candidates].join(", ") || "unknown"}).`,
+ { hint: `Available: ${apps.map((a) => a.name).filter(Boolean).join(", ") || "none"}.`, retry: "reobserve" });
+ }
+ return hit.ref;
+ }
+ // front (default): the app owning the ACTIVE top-level window.
+ for (const app of apps) {
+ try {
+ const wins = await atspi.children(app.ref);
+ for (const w of wins) {
+ const states = await atspi.states(w);
+ if (states.includes("active") || states.includes("focused")) return app.ref;
+ }
+ } catch {}
+ }
+ // No active window found (minimized desktop?) — fall back to the first
+ // app that actually has UI children.
+ for (const app of apps) {
+ try {
+ if ((await atspi.childCount(app.ref)) > 0) return app.ref;
+ } catch {}
+ }
+ throw new AxError(Codes.DESKTOP_OP_FAILED, "No active AT-SPI application found.",
+ { hint: `Available: ${apps.map((a) => a.name).filter(Boolean).join(", ") || "none"}.`, retry: "reobserve" });
+ }
+
+ /** Re-resolve an "ax0.3.2"-style ref (child-index path from the app root;
+ * "ax0" is the root itself, suffixes walk children). Refs are stale by
+ * construction — this fails typed, like macOS. */
+ async resolveAtspiRef(ref, appRef) {
+ const m = /^ax(\d+(?:\.\d+)*)$/.exec(String(ref ?? ""));
+ if (!m) {
+ throw new AxError(Codes.REF_NOT_FOUND, `Not an AT-SPI ref: ${JSON.stringify(ref)} (expected "ax0.3.2" from ax_desktop_ax_snapshot).`,
+ { retry: "reobserve" });
+ }
+ const indices = m[1].split(".").map(Number);
+ if (indices[0] !== 0) {
+ throw new AxError(Codes.REF_NOT_FOUND, `AT-SPI refs are rooted at ax0 (got ${ref}).`,
+ { retry: "reobserve" });
+ }
+ const atspi = await this.ensureAtspi();
+ let node = appRef;
+ for (const idx of indices.slice(1)) {
+ const [child] = await atspi.bus.call(node[0], node[1], "org.a11y.atspi.Accessible", "GetChildAtIndex", "i", [idx]);
+ // Out-of-range indices answer with the /org/a11y/atspi/null sentinel.
+ if (!child || !child[1] || /\/null$/.test(String(child[1]))) {
+ throw new AxError(Codes.REF_NOT_FOUND, `Ref ${ref} went stale (no child at index ${idx}).`,
+ { hint: "Take a fresh ax_desktop_ax_snapshot.", retry: "reobserve" });
+ }
+ node = child;
+ }
+ return node;
+ }
+
+ async axSnapshot({ pid, name, front, maxNodes = 250, maxDepth = 10 } = {}) {
+ const atspi = await this.ensureAtspi();
+ const appRef = await this.resolveAtspiApp({ pid, name, front });
+ const appName = await atspi.name(appRef);
+ const nodes = [];
+ let truncated = false;
+ let frontier = [{ ref: appRef, depth: 0, refPath: "ax0" }];
+ const TEXT_ROLES = new Set(["text", "entry", "text box", "spin button", "paragraph", "document text", "terminal"]);
+ while (frontier.length) {
+ if (nodes.length >= maxNodes) { truncated = true; break; }
+ // Pipelined per level: role+name+interfaces for the whole frontier.
+ const level = frontier;
+ frontier = [];
+ const infos = await Promise.all(level.map(async (item) => {
+ const [role, nameText, ifaces] = await Promise.all([
+ atspi.role(item.ref).catch(() => ""),
+ atspi.name(item.ref).catch(() => ""),
+ atspi.interfaces(item.ref).catch(() => []),
+ ]);
+ return { ...item, role, nameText, ifaces };
+ }));
+ for (const info of infos) {
+ if (nodes.length >= maxNodes) { truncated = true; break; }
+ const row = { ref: info.refPath, role: info.role || "unknown", name: info.nameText || "" };
+ const has = (i) => info.ifaces.some((f) => String(f).endsWith(i));
+ // Bounds and text/value fetches are best-effort and only when the
+ // interface exists — keeps the walk's round trips bounded.
+ if (has("Component")) {
+ try {
+ const b = await atspi.bounds(info.ref);
+ // GNOME reports INT32_MIN sentinels for unplaced actors — drop.
+ if (b.w > 0 && b.h > 0 && b.x > -1e9 && b.y > -1e9) row.bounds = b;
+ } catch {}
+ }
+ if (has("Text") && TEXT_ROLES.has((info.role || "").toLowerCase())) {
+ try {
+ const n = await atspi.charCount(info.ref);
+ if (n >= 0 && n <= 4000) row.text = await atspi.text(info.ref, 0, Math.min(n, 2000));
+ } catch {}
+ }
+ if (has("Value")) {
+ try { row.value = await atspi.value(info.ref); } catch {}
+ }
+ nodes.push(row);
+ if (info.depth < maxDepth) {
+ try {
+ const kids = await atspi.children(info.ref);
+ frontier.push(...kids.map((k, i) => ({ ref: k, depth: info.depth + 1, refPath: `${info.refPath}.${i}` })));
+ } catch {}
+ }
+ }
+ }
+ return {
+ app: appName,
+ nodes,
+ truncated,
+ node_count: nodes.length,
+ note: "AT-SPI snapshot (Linux structure-first): refs are child-index paths from the app root and re-resolve live at action time.",
+ };
+ }
+
+ async axRead({ ref, pid, name, front } = {}) {
+ const atspi = await this.ensureAtspi();
+ const appRef = await this.resolveAtspiApp({ pid, name, front });
+ const node = await this.resolveAtspiRef(ref, appRef);
+ const [role, nameText, description, states, ifaces] = await Promise.all([
+ atspi.role(node),
+ atspi.name(node).catch(() => ""),
+ atspi.description(node).catch(() => ""),
+ atspi.states(node).catch(() => []),
+ atspi.interfaces(node).catch(() => []),
+ ]);
+ const has = (i) => ifaces.some((f) => String(f).endsWith(i));
+ const out = { ref, role, name: nameText, description, states, interfaces: ifaces };
+ if (has("Component")) {
+ try {
+ const b = await atspi.bounds(node);
+ if (b.w > 0 && b.h > 0 && b.x > -1e9 && b.y > -1e9) out.bounds = b;
+ } catch {}
+ }
+ if (has("Text")) {
+ try {
+ const n = await atspi.charCount(node);
+ // Some bridges reject a -1 end offset — clamp to the real count.
+ out.text = n >= 0 && n <= 4000 ? await atspi.text(node, 0, Math.min(n, 2000)) : `(text of ${n} chars)`;
+ } catch {}
+ }
+ if (has("Value")) {
+ try { out.value = await atspi.value(node); } catch {}
+ }
+ if (has("Action")) {
+ // Name lookups only — GetActions would crash GTK's atk-bridge.
+ try {
+ const count = await atspi.actionCount(node).catch(() => 0);
+ const names = [];
+ for (let i = 0; i < Math.min(count, 10); i++) {
+ names.push(await atspi.actionName(node, i).catch(() => `action ${i}`));
+ }
+ if (count > 0) out.actions = names;
+ } catch {}
+ }
+ return out;
+ }
+
+ async axClick({ ref, pid, name, front, fallbackToCoords = true } = {}) {
+ const atspi = await this.ensureAtspi();
+ const appRef = await this.resolveAtspiApp({ pid, name, front });
+ const node = await this.resolveAtspiRef(ref, appRef);
+ const ifaces = await atspi.interfaces(node).catch(() => []);
+ const has = (i) => ifaces.some((f) => String(f).endsWith(i));
+ if (has("Action")) {
+ // NOTE: never call GetActions here — GTK's atk-bridge (2.4x) CRASHES
+ // marshalling its a(sss) reply, killing the app's bus connection.
+ // DoAction(i) directly is the safe universal path; the name lookup is
+ // best-effort (missing on GTK, present on Qt's bridge).
+ try {
+ const done = await atspi.doAction(node, 0);
+ if (done) {
+ const name = await atspi.actionName(node, 0).catch(() => null);
+ return { action_sent: "sent", via: "atspi-action", action_index: 0, ...(name ? { action: name } : {}) };
+ }
+ } catch {
+ // Action interface present but refused → coordinate fallback below.
+ }
+ }
+ if (!fallbackToCoords) {
+ throw new AxError(Codes.COMMAND_REJECTED,
+ `Element ${ref} (${await atspi.role(node)}) exposes no AT-SPI action and the coordinate fallback is disabled.`,
+ { retry: "never" });
+ }
+ const b = await atspi.bounds(node);
+ const x = b.x + (b.w >> 1);
+ const y = b.y + (b.h >> 1);
+ await this.click({ x, y });
+ return { action_sent: "sent", via: "coordinate-fallback", clicked: { x, y } };
+ }
+
+ async axSet({ ref, value, pid, name, front } = {}) {
+ const atspi = await this.ensureAtspi();
+ const appRef = await this.resolveAtspiApp({ pid, name, front });
+ const node = await this.resolveAtspiRef(ref, appRef);
+ const ifaces = await atspi.interfaces(node).catch(() => []);
+ const has = (i) => ifaces.some((f) => String(f).endsWith(i));
+ if (!has("EditableText")) {
+ throw new AxError(Codes.COMMAND_REJECTED,
+ `Element ${ref} (${await atspi.role(node)}) is not AT-SPI-editable (interfaces: ${ifaces.join(", ") || "none"}).`,
+ { hint: "Use ax_desktop_ax_click and type, or the browser tools for web fields.", retry: "reobserve" });
+ }
+ const ok = await atspi.setTextContents(node, String(value));
+ if (!ok) {
+ throw new AxError(Codes.DESKTOP_OP_FAILED, "EditableText.SetTextContents returned false.", { retry: "retry" });
+ }
+ const readBack = has("Text") ? await atspi.text(node, 0, -1).catch(() => null) : null;
+ return { value_set: String(value), via: "atspi-editabletext", ...(readBack !== null ? { read_back_matches: readBack === String(value) } : {}) };
+ }
+
+ dispose() {
+ try { this.client.close(); } catch {}
+ try { this.atspi?.close(); } catch {}
+ }
+}
+
+const BUTTON_DOWN = { left: 1, middle: 2, right: 3 };
+
+function untypeableKeysym(keysym) {
+ return new AxError(Codes.COMMAND_REJECTED,
+ `Key with keysym 0x${keysym.toString(16)} has no keycode on this keyboard layout.`,
+ { hint: "Prefer named keys (enter, tab, arrows, f1-f12) or the browser tools for text.", retry: "never" });
+}
+
+/** The non-modifier part of a combo string ("ctrl+shift+p" → "p"). */
+function comboMainName(combo) {
+ const parts = String(combo).toLowerCase().split("+").map((s) => s.trim()).filter(Boolean);
+ const main = parts.find((p) => !(p in KEY_NAME_TO_KEYSYM && KEY_NAME_TO_KEYSYM[p] >= 0xffe0));
+ return main ?? parts[parts.length - 1] ?? "";
+}
diff --git a/src/desktop/x11.js b/src/desktop/x11.js
new file mode 100644
index 0000000..0115215
--- /dev/null
+++ b/src/desktop/x11.js
@@ -0,0 +1,800 @@
+/**
+ * axuse — minimal zero-dependency X11 protocol client (Linux desktop driver).
+ *
+ * Lineage: the Windows driver scripts PowerShell + SendInput, the macOS
+ * driver scripts JXA + CoreGraphics. Linux has no comparable "scriptable
+ * surface" guaranteed present — xdotool/wmctrl/scrot are extra apt packages
+ * — but every X11 server speaks one wire protocol over a Unix socket, and
+ * Node ships everything needed to speak it directly:
+ *
+ * - connection setup + Xauthority cookie (MIT-MAGIC-COOKIE-1) with an
+ * empty-auth retry (SI:localuser grants local same-user clients)
+ * - core requests: WarpPointer, QueryPointer, GetImage, GetGeometry,
+ * TranslateCoordinates, InternAtom, GetProperty, SendEvent,
+ * GetKeyboardMapping
+ * - XTEST FakeInput — trusted server-side input synthesis, the X
+ * equivalent of SendInput (wire layout per the XTEST 2.2 encoding)
+ * - EWMH properties for the client window list + _NET_ACTIVE_WINDOW
+ *
+ * Scope notes (v1):
+ * - X11/XWayland only; native Wayland sessions refuse with a typed error.
+ * - The root window of the display's default screen is the coordinate
+ * space. On the standard single-X-screen layout the root spans every
+ * monitor, so warps and captures work across monitors.
+ */
+import { connect as netConnect } from "node:net";
+import { readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { AxError, Codes } from "../core/errors.js";
+
+const NO_WINDOW = 0;
+
+/** X core request opcodes used here. */
+export const Op = Object.freeze({
+ GetGeometry: 14, QueryTree: 15, InternAtom: 16, GetProperty: 20, SendEvent: 25,
+ QueryPointer: 38, TranslateCoordinates: 40, WarpPointer: 41,
+ GetImage: 73, QueryExtension: 98, GetKeyboardMapping: 101,
+});
+
+/** FAKE_EVENT_TYPE values (same as the core event codes). */
+export const FakeEventType = Object.freeze({
+ KeyPress: 2, KeyRelease: 3, ButtonPress: 4, ButtonRelease: 5, MotionNotify: 6,
+});
+
+/** Parse a display string ":1", ":1.0", "unix:1", "host:1" → parts. */
+export function parseDisplay(display = process.env.DISPLAY ?? "") {
+ const m = /^([a-zA-Z0-9._-]*):([0-9]+)(?:\.([0-9]+))?$/.exec(String(display).trim());
+ if (!m) return null;
+ return { host: m[1], display: Number(m[2]), screen: m[3] === undefined ? null : Number(m[3]) };
+}
+
+/** Xauthority records (little-endian u16 lengths, no padding between fields):
+ * family(u16) addrLen addr | numLen num | nameLen name | dataLen data */
+export function readXauthority(path) {
+ let buf;
+ try {
+ buf = readFileSync(path);
+ } catch {
+ return [];
+ }
+ const out = [];
+ let o = 0;
+ const u16 = () => { const v = buf.readUInt16LE(o); o += 2; return v; };
+ const blob = () => { const n = u16(); const b = buf.subarray(o, o + n); o += n; return b; };
+ while (o + 2 <= buf.length) {
+ const start = o;
+ try {
+ const family = u16();
+ const addr = blob();
+ const num = blob();
+ const name = blob();
+ const data = blob();
+ out.push({ family, addr, num: num.toString("latin1"), name: name.toString("latin1"), data });
+ } catch {
+ o = start;
+ break; // truncated record
+ }
+ }
+ return out;
+}
+
+/** Best MIT-MAGIC-COOKIE-1 for a display: exact display-number match first,
+ * then a wildcard-family record (FamilyWild=65535, FamilyLocal=256). */
+function cookieFor(display) {
+ const path = process.env.XAUTHORITY || join(homedir(), ".Xauthority");
+ const cookies = readXauthority(path).filter((r) => r.name === "MIT-MAGIC-COOKIE-1");
+ return cookies.find((r) => r.num === String(display))
+ ?? cookies.find((r) => r.family === 65535 || r.family === 256)
+ ?? null;
+}
+
+function describeXError(code) {
+ const names = {
+ 1: "BadRequest", 2: "BadValue", 3: "BadWindow", 4: "BadPixmap", 5: "BadAtom",
+ 6: "BadCursor", 7: "BadFont", 8: "BadMatch", 9: "BadDrawable", 10: "BadAccess",
+ 11: "BadAlloc", 12: "BadColor", 13: "BadGC", 14: "BadIDChoice", 15: "BadName",
+ 16: "BadLength", 17: "BadImplementation",
+ };
+ return names[code] ?? `code ${code}`;
+}
+
+/** Static evdev keymap (keysym → [keycode = KEY_* + 8, shift]) for servers
+ * whose core keyboard map is empty (XKB-owned). US-layout punctuation; the
+ * server map wins whenever it is populated. */
+const EVDEV_KEYMAP = (() => {
+ const rows = [];
+ const push = (keysym, keycode, shift) => rows.push([keysym, keycode, shift]);
+ // Letters in QWERTY PHYSICAL order (evdev keycodes; X keycode = code + 8):
+ // q w e r t y u i o p → 24..33, a s d f g h j k l → 38..46,
+ // z x c v b n m → 52..58. Alphabetical order would be wrong — the map is
+ // positional, not alphabetical.
+ const letterRows = [
+ ["qwertyuiop", 24],
+ ["asdfghjkl", 38],
+ ["zxcvbnm", 52],
+ ];
+ for (const [letters, base] of letterRows) {
+ for (let i = 0; i < letters.length; i++) {
+ const c = letters.charCodeAt(i);
+ push(c, base + i, false); // lowercase
+ push(c - 32, base + i, true); // uppercase (shift)
+ }
+ }
+ // Digits row (KEY_1=2 → keycode 10 … KEY_0=11 → 19)
+ const digits = "1234567890";
+ const shifted = "!@#$%^&*()";
+ for (let i = 0; i < 10; i++) {
+ push(digits.charCodeAt(i), 10 + i, false);
+ push(shifted.charCodeAt(i), 10 + i, true);
+ }
+ const named = [
+ // [keysym, keycode, shift]
+ [0xff0d, 36, false], [0xff09, 23, false], [0xff1b, 9, false], [0xff08, 22, false],
+ [0xffff, 119, false], [0xff63, 118, false],
+ [0xff52, 111, false], [0xff54, 116, false], [0xff51, 113, false], [0xff53, 114, false],
+ [0xff50, 110, false], [0xff57, 115, false], [0xff55, 112, false], [0xff56, 117, false],
+ [0x20, 65, false],
+ [0xffe1, 50, false], [0xffe2, 62, false], // Shift_L/R
+ [0xffe3, 37, false], [0xffe4, 105, false], // Control_L/R
+ [0xffe9, 64, false], [0xffea, 108, false], // Alt_L/R
+ [0xffeb, 133, false], [0xffec, 134, false], // Super_L/R
+ [0xffe5, 66, false], [0xff7f, 77, false], // Caps_Lock, Num_Lock
+ // US punctuation (unshifted, shifted)
+ [0x2d, 20, false], [0x5f, 20, true], // - _
+ [0x3d, 21, false], [0x2b, 21, true], // = +
+ [0x5b, 34, false], [0x7b, 34, true], // [ {
+ [0x5d, 35, false], [0x7d, 35, true], // ] }
+ [0x5c, 51, false], [0x7c, 51, true], // \ |
+ [0x3b, 47, false], [0x3a, 47, true], // ; :
+ [0x27, 48, false], [0x22, 48, true], // ' "
+ [0x60, 49, false], [0x7e, 49, true], // ` ~
+ [0x2c, 59, false], [0x3c, 59, true], // , <
+ [0x2e, 60, false], [0x3e, 60, true], // . >
+ [0x2f, 61, false], [0x3f, 61, true], // / ?
+ ];
+ rows.push(...named);
+ for (let i = 0; i < 12; i++) push(0xffbe + i, 67 + i, false); // F1-F12
+ // Numeric keypad: KP_0=90, KP_1=87 … KP_9=81 (evdev + 8).
+ const kpDigits = [[0xffb0, 90], [0xffb1, 87], [0xffb2, 88], [0xffb3, 89], [0xffb4, 83],
+ [0xffb5, 84], [0xffb6, 85], [0xffb7, 79], [0xffb8, 80], [0xffb9, 81]];
+ for (const [keysym, keycode] of kpDigits) push(keysym, keycode, false);
+ push(0xffab, 86, false); // KP_Add
+ push(0xffad, 82, false); // KP_Subtract
+ push(0xaa, 63, false); // KP_Multiply
+ push(0xaf, 106, false); // KP_Divide
+ push(0xff8b, 104, false); // KP_Enter
+ push(0xff9e, 91, false); // KP_Decimal
+ return rows;
+})();
+
+/**
+ * A small synchronous-request-at-a-time X11 client. Desktop input must be
+ * strictly ordered anyway (the driver serializes ops like WindowsDriver),
+ * so a sequence-keyed pending map is sufficient — no concurrent pipelines.
+ */
+export class X11Client {
+ /** @param {string} [display] */
+ constructor(display) {
+ this.display = String(display ?? process.env.DISPLAY ?? "");
+ this.displaySpec = parseDisplay(this.display);
+ this.socket = null;
+ this.seq = 0;
+ this.pending = new Map(); // seq → {resolve, reject}
+ this.buffer = Buffer.alloc(0);
+ this.setupWaiters = []; // pre-handshake readers (waitFor)
+ this.rootWindow = 0;
+ this.rootCandidates = [];
+ this.screenWidth = 0;
+ this.screenHeight = 0;
+ this.minKeycode = 0;
+ this.maxKeycode = 0;
+ this.imageByteOrder = 0; // 0 LSBFirst, 1 MSBFirst
+ this.pixmapFormats = new Map(); // depth → bits-per-pixel
+ this.xtestOpcode = 0;
+ this.atoms = new Map();
+ this.keysymToKey = new Map(); // keysym → {keycode, shift}
+ this.dead = false;
+ }
+
+ get connected() {
+ return !!this.socket && !this.dead;
+ }
+
+ /* ----------------------------------------------------------- connection */
+
+ async connect() {
+ this.dead = false; // a disposed client may reconnect (driver respawn)
+ if (!this.displaySpec) {
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ process.env.WAYLAND_DISPLAY
+ ? "This looks like a native Wayland session with no usable DISPLAY; the Linux desktop driver speaks X11 (XWayland covers most desktop apps — set DISPLAY, usually :0, :1 or similar)."
+ : "DISPLAY is not set; the Linux desktop driver needs an X11 session (run the harness inside the desktop session, or under xvfb-run for a virtual display).",
+ { hint: "Desktop control on Linux targets X11/XWayland; browser tools work everywhere.", retry: "never" });
+ }
+ const { host, display } = this.displaySpec;
+ await new Promise((resolve, reject) => {
+ const sock = (host === "" || host === "unix")
+ ? netConnect(`/tmp/.X11-unix/X${display}`)
+ : netConnect({ host, port: 6000 + display });
+ const fail = (e) => reject(new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ `Cannot open X display ${process.env.DISPLAY ?? ""}: ${e.message}`,
+ { hint: "Run inside the X session (XAUTHORITY may need to point at the session cookie), or under xvfb-run.", retry: "never" }));
+ sock.once("error", fail);
+ sock.once("connect", () => {
+ sock.off("error", fail);
+ this.socket = sock;
+ resolve();
+ });
+ });
+
+ try {
+ await this.handshake(cookieFor(display));
+ } catch (e) {
+ // A mismatched cookie is the common cause; SI:localuser access control
+ // admits same-user clients without any cookie — retry cookie-less.
+ if (e instanceof AxError && e.code === Codes.DESKTOP_DRIVER_UNAVAILABLE) {
+ this.teardownSocket();
+ await this.reopenSocket();
+ await this.handshake(null);
+ } else {
+ throw e;
+ }
+ }
+ this.socket.on("data", (d) => this.routeData(d));
+ this.socket.on("close", () => this.failAll("X11 connection closed"));
+ this.socket.on("error", () => this.failAll("X11 connection error"));
+
+ if (!(await this.resolveRoot())) {
+ throw new AxError(Codes.INTERNAL, "No candidate root window answered GetGeometry.");
+ }
+ this.xtestOpcode = (await this.queryExtension("XTEST")).majorOpcode;
+ if (!this.xtestOpcode) {
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ "The X server does not support the XTEST extension; trusted input synthesis is unavailable.",
+ { retry: "never" });
+ }
+ await this.loadKeyboardMap();
+ return this;
+ }
+
+ async reopenSocket() {
+ const { host, display } = this.displaySpec;
+ await new Promise((resolve, reject) => {
+ const sock = (host === "" || host === "unix")
+ ? netConnect(`/tmp/.X11-unix/X${display}`)
+ : netConnect({ host, port: 6000 + display });
+ sock.once("error", reject);
+ sock.once("connect", () => { this.socket = sock; resolve(); });
+ });
+ this.buffer = Buffer.alloc(0);
+ this.dead = false;
+ }
+
+ teardownSocket() {
+ try { this.socket?.destroy(); } catch {}
+ this.socket = null;
+ }
+
+ /** Send the connection setup with an (optional) auth cookie and parse it. */
+ async handshake(cookie) {
+ const name = "MIT-MAGIC-COOKIE-1";
+ const data = cookie?.data ?? Buffer.alloc(0);
+ const pad4 = (n) => (4 - (n % 4)) % 4;
+ const head = Buffer.alloc(12);
+ head.writeUInt8(0x6c, 0); // 'l' — little-endian
+ head.writeUInt8(0, 1);
+ head.writeUInt16LE(11, 2); // protocol 11.0
+ head.writeUInt16LE(0, 4);
+ head.writeUInt16LE(name.length, 6);
+ head.writeUInt16LE(data.length, 8);
+ head.writeUInt16LE(0, 10);
+ this.socket.write(Buffer.concat([
+ head,
+ Buffer.from(name, "latin1"), Buffer.alloc(pad4(name.length)),
+ data, Buffer.alloc(pad4(data.length)),
+ ]));
+
+ const first = await this.waitFor(8);
+ // The setup length field counts 4-byte units AFTER the 8-byte header,
+ // INCLUDING the 32-byte fixed part (release/ids/keycodes…): the whole
+ // packet is 8 + length*4 bytes (verified against Xorg 1.21 wire).
+ const total = 8 + first.readUInt16LE(6) * 4;
+ const full = Buffer.concat([first, await this.waitFor(total - 8)]);
+ if (full[0] === 2) {
+ // SetupFailed: [2][reasonLen][major][minor][len16][reason…]
+ const reason = full.subarray(8).toString("utf8").replace(/\0.*$/s, "").trim();
+ throw new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE,
+ `The X server refused the connection${reason ? `: ${reason}` : ""}.`,
+ { hint: "Auth mismatch — run inside the X session or point XAUTHORITY at the session cookie.", retry: "never" });
+ }
+ if (full[0] !== 1) throw new AxError(Codes.INTERNAL, `Unexpected X11 setup reply type ${full[0]}`);
+ // Fixed-field layout verified byte-by-byte against Xorg 1.21.1 wire
+ // (Ubuntu 22.04–26.04 family; LSB connection):
+ // vlen@24, maxReqLen@26, pad@28, nformats@29, imageByteOrder@30,
+ // bitmapBitOrder@31, scanlineUnit@32, scanlinePad@33, minKey@34,
+ // maxKey@35, [email protected], vendor@40, formats (8·nformats),
+ // then screens. Single-byte fields are junk-guarded (some builds
+ // leave "unused" slots uninitialized).
+ this.imageByteOrder = full[30] <= 1 ? full[30] : 0;
+ this.minKeycode = full[34];
+ this.maxKeycode = full[35];
+ if (!(this.minKeycode < this.maxKeycode)) {
+ // Fallback: the canonical evdev range every mainstream server uses.
+ this.minKeycode = 8;
+ this.maxKeycode = 255;
+ }
+ const vendorLen = full.readUInt16LE(24);
+ const afterVendor = 40 + vendorLen + ((4 - (vendorLen % 4)) % 4);
+ // Pixmap formats ([depth, bits-per-pixel, scanline-pad, 5 junk] × 8):
+ // Xorg maps depth 24 → 32 bpp — GetImage rows must use this, not depth/8.
+ this.pixmapFormats = new Map();
+ const fmtCount = full[29] >= 1 && full[29] <= 16 ? full[29] : 6;
+ for (let i = 0; i < fmtCount && afterVendor + i * 8 + 2 <= full.length; i++) {
+ const depth = full[afterVendor + i * 8];
+ const bitsPerPixel = full[afterVendor + i * 8 + 1];
+ if (depth > 0 && bitsPerPixel > 0) this.pixmapFormats.set(depth, bitsPerPixel);
+ }
+ // Screen 0 root candidates: the declared pixmap-format count when it
+ // parses sanely, else a scan — the true root is confirmed against the
+ // live server by GetGeometry in resolveRoot().
+ this.rootCandidates = [];
+ const nformats = full[29] >= 1 && full[29] <= 16 ? full[29] : null;
+ for (const formats of nformats !== null ? [nformats] : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16]) {
+ const s = afterVendor + formats * 8;
+ if (s + 24 > full.length) break;
+ const root = full.readUInt32LE(s);
+ // Screen record: root(4) colormap(4) white(4) black(4) masks(4)
+ // then width/height in pixels at s+20/s+22.
+ const w = full.readUInt16LE(s + 20);
+ const h = full.readUInt16LE(s + 22);
+ if (root !== 0 && w > 0 && h > 0 && w <= 32767 && h <= 32767) {
+ this.rootCandidates.push({ root, w, h });
+ }
+ }
+ if (!this.rootCandidates.length) {
+ throw new AxError(Codes.INTERNAL, "Could not parse the X11 connection setup screen list.");
+ }
+ }
+
+ /** Pick the real root window: the candidate whose GetGeometry round-trips
+ * on the live connection (formats/visual bytes can masquerade as screens
+ * to a heuristic, but only a real window answers GetGeometry). */
+ async resolveRoot() {
+ for (const cand of this.rootCandidates ?? []) {
+ try {
+ const g = await this.getGeometry(cand.root);
+ if (g.w > 0 && g.h > 0) {
+ this.rootWindow = cand.root;
+ this.screenWidth = g.w;
+ this.screenHeight = g.h;
+ return true;
+ }
+ } catch {
+ // BadWindow → wrong candidate, keep scanning.
+ }
+ }
+ return false;
+ }
+
+ /** Accumulate `n` bytes during setup (sole consumer of socket data). */
+ waitFor(n) {
+ if (this.buffer.length >= n) {
+ const out = this.buffer.subarray(0, n);
+ this.buffer = this.buffer.subarray(n);
+ return Promise.resolve(out);
+ }
+ return new Promise((resolve, reject) => {
+ const w = { n, resolve, reject };
+ this.setupWaiters.push(w);
+ const onChunk = (d) => { this.buffer = Buffer.concat([this.buffer, d]); this.pumpSetup(); };
+ const onClose = () => {
+ this.setupWaiters.splice(this.setupWaiters.indexOf(w), 1);
+ reject(new AxError(Codes.DESKTOP_DRIVER_UNAVAILABLE, "X11 connection closed during handshake"));
+ };
+ this.socket.on("data", onChunk);
+ this.socket.once("close", onClose);
+ w.cleanup = () => { this.socket.off("data", onChunk); this.socket.off("close", onClose); };
+ this.pumpSetup();
+ });
+ }
+
+ pumpSetup() {
+ while (this.setupWaiters.length && this.buffer.length >= this.setupWaiters[0].n) {
+ const w = this.setupWaiters.shift();
+ const out = this.buffer.subarray(0, w.n);
+ this.buffer = this.buffer.subarray(w.n);
+ w.cleanup?.();
+ w.resolve(out);
+ }
+ }
+
+ /* ------------------------------------------------------- reply routing */
+
+ routeData(chunk) {
+ this.buffer = Buffer.concat([this.buffer, chunk]);
+ for (;;) {
+ if (this.buffer.length < 32) return;
+ const first = this.buffer[0];
+ if (first === 0) {
+ // X error: [0][code][seq][value(4)][minor(2)][major(1)][pad(21)]
+ const code = this.buffer[1];
+ const seq = this.buffer.readUInt16LE(2);
+ const value = this.buffer.readUInt32LE(4);
+ const minor = this.buffer.readUInt16LE(8);
+ const major = this.buffer[10];
+ this.buffer = this.buffer.subarray(32);
+ const p = this.pending.get(seq);
+ if (p) {
+ this.pending.delete(seq);
+ p.reject(new AxError(Codes.DESKTOP_OP_FAILED,
+ `X11 ${describeXError(code)} on request ${major}.${minor} (value ${value})`, { retry: "retry" }));
+ }
+ continue;
+ }
+ if (first === 1) {
+ const seq = this.buffer.readUInt16LE(2);
+ const extra = this.buffer.readUInt32LE(4) * 4;
+ const total = 32 + extra;
+ if (this.buffer.length < total) return;
+ const reply = Buffer.from(this.buffer.subarray(0, total)); // copy: buffer window gets sliced
+ this.buffer = this.buffer.subarray(total);
+ const p = this.pending.get(seq);
+ if (p) {
+ this.pending.delete(seq);
+ p.resolve(reply);
+ }
+ continue;
+ }
+ // Event (2..64): v1 routes nothing — skip one 32-byte frame.
+ this.buffer = this.buffer.subarray(32);
+ }
+ }
+
+ failAll(reason) {
+ this.dead = true;
+ for (const [, p] of this.pending) p.reject(new AxError(Codes.DESKTOP_OP_FAILED, reason));
+ this.pending.clear();
+ }
+
+ /** Send a fully-assembled request buffer (length slot at bytes 2..4 is
+ * filled in here from the buffer size). Resolves on the reply when
+ * `wantReply`, else immediately after the write. */
+ send(buf, wantReply = false) {
+ if (!this.socket || this.dead) {
+ return Promise.reject(new AxError(Codes.DESKTOP_OP_FAILED, "X11 connection is not open"));
+ }
+ if (buf.length % 4 !== 0) throw new Error("X11 request must be 4-byte aligned");
+ buf.writeUInt16LE(buf.length / 4, 2);
+ // The server numbers EVERY request (replies AND no-reply requests);
+ // the counter must advance even when we expect nothing back.
+ const seq = (this.seq = (this.seq + 1) & 0xffff);
+ if (!wantReply) {
+ this.socket.write(buf);
+ return Promise.resolve();
+ }
+ return new Promise((resolve, reject) => {
+ this.pending.set(seq, { resolve, reject });
+ this.socket.write(buf);
+ });
+ }
+
+ /* ------------------------------------------------------------- requests */
+
+ async queryExtension(name) {
+ const n = Buffer.byteLength(name);
+ const buf = Buffer.alloc(4 + 4 + n + ((4 - (n % 4)) % 4));
+ buf.writeUInt8(Op.QueryExtension, 0);
+ buf.writeUInt16LE(2 + (4 + n) / 4, 2);
+ buf.writeUInt16LE(n, 4);
+ buf.write(name, 8, "latin1");
+ const rep = await this.send(buf, true);
+ // Field layout verified on the wire (Xorg 1.21 LSB): present BOOL at
+ // byte 8, major opcode / first event / first error as single bytes.
+ return rep[8] === 1 ? { present: true, majorOpcode: rep[9] } : { present: false, majorOpcode: 0 };
+ }
+
+ async internAtom(name) {
+ const cached = this.atoms.get(name);
+ if (cached) return cached;
+ const n = Buffer.byteLength(name);
+ const buf = Buffer.alloc(4 + 4 + n + ((4 - (n % 4)) % 4));
+ buf.writeUInt8(Op.InternAtom, 0);
+ buf.writeUInt8(1, 1); // only if exists
+ buf.writeUInt16LE(2 + (4 + n) / 4, 2);
+ buf.writeUInt16LE(n, 4);
+ buf.write(name, 8, "latin1");
+ const rep = await this.send(buf, true);
+ const atom = rep.readUInt32LE(8);
+ this.atoms.set(name, atom);
+ return atom;
+ }
+
+ async getProperty(window, name, { longLength = 4096 } = {}) {
+ const prop = typeof name === "string" ? await this.internAtom(name) : name;
+ if (!prop) {
+ // Atom not present (e.g. EWMH atoms without a running window manager):
+ // the property cannot exist — report empty rather than BadAtom.
+ return { format: 0, typeAtom: 0, bytesAfter: 0, valueLen: 0, value: Buffer.alloc(0) };
+ }
+ const buf = Buffer.alloc(24);
+ buf.writeUInt8(Op.GetProperty, 0);
+ buf.writeUInt8(0, 1); // delete = false
+ buf.writeUInt16LE(6, 2);
+ buf.writeUInt32LE(window, 4);
+ buf.writeUInt32LE(prop, 8);
+ buf.writeUInt32LE(0, 12); // type = AnyPropertyType
+ buf.writeUInt32LE(0, 16); // long-offset
+ buf.writeUInt32LE(longLength, 20);
+ const rep = await this.send(buf, true);
+ return {
+ format: rep[1],
+ typeAtom: rep.readUInt32LE(8),
+ bytesAfter: rep.readUInt32LE(12),
+ valueLen: rep.readUInt32LE(16),
+ value: rep.subarray(32, 32 + rep.readUInt32LE(16)),
+ };
+ }
+
+ /** GetProperty value as a CARD32 list (window ids, pids…). */
+ async getPropertyU32s(window, name) {
+ const { format, value } = await this.getProperty(window, name);
+ if (format !== 32) return [];
+ const out = [];
+ for (let o = 0; o + 4 <= value.length; o += 4) out.push(value.readUInt32LE(o));
+ return out;
+ }
+
+ async getGeometry(drawable) {
+ const buf = Buffer.alloc(8);
+ buf.writeUInt8(Op.GetGeometry, 0);
+ buf.writeUInt16LE(2, 2);
+ buf.writeUInt32LE(drawable, 4);
+ const rep = await this.send(buf, true);
+ return {
+ depth: rep[1],
+ root: rep.readUInt32LE(8),
+ x: rep.readInt16LE(12),
+ y: rep.readInt16LE(14),
+ w: rep.readUInt16LE(16),
+ h: rep.readUInt16LE(18),
+ };
+ }
+
+ async queryPointer() {
+ const buf = Buffer.alloc(8);
+ buf.writeUInt8(Op.QueryPointer, 0);
+ buf.writeUInt16LE(2, 2);
+ buf.writeUInt32LE(this.rootWindow, 4);
+ const rep = await this.send(buf, true);
+ return {
+ sameScreen: rep[1] === 1,
+ child: rep.readUInt32LE(12),
+ rootX: rep.readInt16LE(16),
+ rootY: rep.readInt16LE(18),
+ };
+ }
+
+ async warpPointer(x, y) {
+ // 24 bytes (length 6): dst-x/dst-y are INT16 per the protocol encoding
+ // (verified against Xorg 1.21 — a 28-byte form draws BadLength).
+ const buf = Buffer.alloc(24);
+ buf.writeUInt8(Op.WarpPointer, 0);
+ buf.writeUInt16LE(6, 2);
+ buf.writeUInt32LE(NO_WINDOW, 4); // src = None
+ buf.writeUInt32LE(this.rootWindow, 8); // dst = root
+ buf.writeInt16LE(0, 12); // src x
+ buf.writeInt16LE(0, 14); // src y
+ buf.writeUInt16LE(0, 16); // src width 0 = whole
+ buf.writeUInt16LE(0, 18); // src height
+ buf.writeInt16LE(Math.round(x), 20); // dst x
+ buf.writeInt16LE(Math.round(y), 22); // dst y
+ await this.send(buf);
+ }
+
+ /** XTEST FakeInput (36 bytes, length 9): keys use server keycodes, buttons
+ * 1-7 (4/5 = wheel up/down, 6/7 = wheel left/right), motion with
+ * detail 0 = absolute rootX/rootY. delay 0 = immediate. */
+ async fakeInput({ type, detail = 0, delayMs = 0, x, y } = {}) {
+ if (!this.xtestOpcode) throw new AxError(Codes.DESKTOP_OP_FAILED, "XTEST extension not available");
+ const buf = Buffer.alloc(36);
+ buf.writeUInt8(this.xtestOpcode, 0);
+ buf.writeUInt8(2, 1); // FakeInput
+ buf.writeUInt8(type, 4);
+ buf.writeUInt8(detail, 5);
+ buf.writeUInt32LE(delayMs, 8);
+ // root (0 = current root) at 12; 8 unused; rootX/rootY at 24/26.
+ if (x !== undefined) buf.writeInt16LE(Math.round(x), 24);
+ if (y !== undefined) buf.writeInt16LE(Math.round(y), 26);
+ await this.send(buf);
+ }
+
+ /** ZPixmap grab of the root window (clamped to bounds by the caller).
+ * Returns raw rows + layout facts for the RGB conversion step. */
+ async getImage(x, y, w, h) {
+ const buf = Buffer.alloc(20); // 5 units: opcode, format, drawable, x/y/w/h, plane-mask
+ buf.writeUInt8(Op.GetImage, 0);
+ buf.writeUInt8(2, 1); // format = ZPixmap
+ buf.writeUInt32LE(this.rootWindow, 4);
+ buf.writeInt16LE(Math.round(x), 8);
+ buf.writeInt16LE(Math.round(y), 10);
+ buf.writeUInt16LE(w, 12);
+ buf.writeUInt16LE(h, 14);
+ buf.writeUInt32LE(0xffffffff, 16); // plane mask = all
+ const rep = await this.send(buf, true);
+ const depth = rep[1];
+ const data = rep.subarray(32);
+ const bitsPerPixel = this.pixmapFormats?.get(depth) ?? (depth >= 24 ? 32 : depth);
+ const bytesPerPixel = Math.max(1, bitsPerPixel / 8);
+ const stride = Math.ceil((w * bytesPerPixel) / 4) * 4;
+ return { depth, bytesPerPixel, stride, w, h, data, byteOrder: this.imageByteOrder };
+ }
+
+ /** Fill keysym → {keycode, shift} from the server keyboard map. Servers
+ * whose keyboard is fully owned by XKB (Xvfb, some minimal setups) answer
+ * the core query with an empty map — then fall back to the static evdev
+ * table (X keycode = Linux KEY_* + 8, stable across distros). */
+ async loadKeyboardMap() {
+ const count = this.maxKeycode - this.minKeycode + 1;
+ if (count > 0) {
+ const buf = Buffer.alloc(8);
+ buf.writeUInt8(Op.GetKeyboardMapping, 0);
+ buf.writeUInt8(this.minKeycode, 1);
+ buf.writeUInt16LE(2, 2);
+ buf.writeUInt8(count, 4);
+ const rep = await this.send(buf, true);
+ const perKeycode = rep[1] || 2;
+ const nKeysyms = rep.readUInt32LE(4); // reply length in CARD32 units
+ this.keysymToKey.clear();
+ for (let k = 0; k < count; k++) {
+ for (let col = 0; col < perKeycode; col++) {
+ const idx = k * perKeycode + col;
+ if (idx >= nKeysyms) break;
+ const keysym = rep.readUInt32LE(32 + idx * 4);
+ if (!keysym) continue;
+ // Column parity encodes Shift on the primary group; register the
+ // first mapping for each keysym (group-2 columns only add).
+ if (!this.keysymToKey.has(keysym)) {
+ this.keysymToKey.set(keysym, { keycode: this.minKeycode + k, shift: col % 2 === 1 });
+ }
+ }
+ }
+ if (this.keysymToKey.size > 0) return this.keysymToKey.size;
+ }
+ this.applyEvdevMap();
+ return this.keysymToKey.size;
+ }
+
+ /** The static evdev/XKB fallback map: [keysym, keycode, needsShift]. */
+ applyEvdevMap() {
+ this.keysymToKey.clear();
+ for (const [keysym, keycode, shift] of EVDEV_KEYMAP) {
+ this.keysymToKey.set(keysym, { keycode, shift: !!shift });
+ }
+ }
+
+ keyForKeysym(keysym) {
+ return this.keysymToKey.get(keysym) ?? null;
+ }
+
+ /** A child window's origin in root coordinates. */
+ async translateCoordinates(window) {
+ const buf = Buffer.alloc(16);
+ buf.writeUInt8(Op.TranslateCoordinates, 0);
+ buf.writeUInt16LE(4, 2);
+ buf.writeUInt32LE(window, 4);
+ buf.writeUInt32LE(this.rootWindow, 8);
+ buf.writeInt16LE(0, 12);
+ buf.writeInt16LE(0, 14);
+ const rep = await this.send(buf, true);
+ return { child: rep.readUInt32LE(8), x: rep.readInt16LE(12), y: rep.readInt16LE(14) };
+ }
+
+ /** SetInputFocus — the WM-less fallback so XTEST keys have a target. */
+ async setInputFocus(window) {
+ const buf = Buffer.alloc(12);
+ buf.writeUInt8(42, 0); // SetInputFocus
+ buf.writeUInt8(1, 1); // revert-to: PointerRoot
+ buf.writeUInt16LE(3, 2);
+ buf.writeUInt32LE(window, 4);
+ buf.writeUInt32LE(0, 8); // CurrentTime
+ await this.send(buf);
+ }
+
+ /* ------------------------------------------------------------ RandR ---- */
+
+ #randrOpcode = 0;
+ #randrConfigTimestamp = 0;
+
+ async randrOpcode() {
+ if (this.#randrOpcode === 0) {
+ const { present, majorOpcode } = await this.queryExtension("RANDR");
+ this.#randrOpcode = present ? majorOpcode : -1;
+ }
+ return this.#randrOpcode > 0 ? this.#randrOpcode : 0;
+ }
+
+ /** Active monitor layout via RandR 1.2 (RRGetScreenResourcesCurrent +
+ * RRGetCrtcInfo + RRGetOutputPrimary). Returns [] without RandR. */
+ async getMonitors() {
+ const ext = await this.randrOpcode();
+ if (!ext) return [];
+ try {
+ // RRGetScreenResourcesCurrent (minor 25, per randrproto encoding)
+ const req = Buffer.alloc(8);
+ req.writeUInt8(ext, 0);
+ req.writeUInt8(25, 1);
+ req.writeUInt16LE(2, 2);
+ req.writeUInt32LE(this.rootWindow, 4);
+ const res = await this.send(req, true);
+ this.#randrConfigTimestamp = res.readUInt32LE(12);
+ const numCrtcs = res.readUInt16LE(16);
+ const crtcs = [];
+ for (let i = 0; i < numCrtcs; i++) {
+ crtcs.push(res.readUInt32LE(32 + i * 4));
+ }
+ // RRGetOutputPrimary (minor 31) → xid
+ const pReq = Buffer.alloc(8);
+ pReq.writeUInt8(ext, 0);
+ pReq.writeUInt8(31, 1);
+ pReq.writeUInt16LE(2, 2);
+ pReq.writeUInt32LE(this.rootWindow, 4);
+ let primary = 0;
+ try {
+ const p = await this.send(pReq, true);
+ primary = p.readUInt32LE(8);
+ } catch {}
+
+ const monitors = [];
+ for (const crtc of crtcs) {
+ // RRGetCrtcInfo (minor 20): reply carries timestamp, x/y/w/h, mode…
+ const cReq = Buffer.alloc(12);
+ cReq.writeUInt8(ext, 0);
+ cReq.writeUInt8(20, 1);
+ cReq.writeUInt16LE(3, 2);
+ cReq.writeUInt32LE(crtc, 4);
+ cReq.writeUInt32LE(this.#randrConfigTimestamp, 8);
+ const info = await this.send(cReq, true);
+ const mode = info.readUInt32LE(20);
+ if (!mode) continue; // disabled crtc
+ const numOutputs = info.readUInt16LE(28);
+ const firstOutput = numOutputs ? info.readUInt32LE(32) : 0;
+ monitors.push({
+ x: info.readInt16LE(12),
+ y: info.readInt16LE(14),
+ w: info.readUInt16LE(16),
+ h: info.readUInt16LE(18),
+ primary: firstOutput === primary,
+ });
+ }
+ return monitors;
+ } catch {
+ return []; // RandR shapes vary; the root union stays the source of truth
+ }
+ }
+
+ /** SendEvent of a format-32 ClientMessage to the root window (EWMH).
+ * 44-byte request: SendEvent header (12) + 32-byte event, where the
+ * ClientMessage event packs type(1) format(1) seq(2) window(4)
+ * message-type(4) data.l[5](20) with no pad. */
+ async sendClientMessage(window, messageTypeAtom, data = [0, 0, 0, 0, 0]) {
+ const buf = Buffer.alloc(44);
+ buf.writeUInt8(Op.SendEvent, 0);
+ buf.writeUInt8(1, 1); // propagate
+ buf.writeUInt16LE(11, 2);
+ buf.writeUInt32LE(this.rootWindow, 4); // destination
+ buf.writeUInt32LE((1 << 19) | (1 << 20), 8); // SubstructureNotify|SubstructureRedirect
+ buf.writeUInt8(33, 12); // ClientMessage
+ buf.writeUInt8(32, 13); // format 32
+ buf.writeUInt32LE(window, 16);
+ buf.writeUInt32LE(messageTypeAtom, 20);
+ for (let i = 0; i < 5; i++) buf.writeUInt32LE((data[i] ?? 0) >>> 0, 24 + i * 4);
+ await this.send(buf);
+ }
+
+ close() {
+ this.dead = true;
+ this.failAll("X11 client closed");
+ this.teardownSocket();
+ }
+}
diff --git a/src/mcp/manifest.js b/src/mcp/manifest.js
index 57740e9..b4737d7 100644
--- a/src/mcp/manifest.js
+++ b/src/mcp/manifest.js
@@ -178,7 +178,7 @@ export function desktopTools(driverAvailable, axCapable = false) {
},
...((axCapable) ? [{
name: "ax_desktop_ax_snapshot",
- description: "STRUCTURE-FIRST desktop snapshot (macOS): the app's accessibility tree with roles, titles, values, bounds, and live refs — the desktop twin of ax_observe. Pick the app by pid (from ax_desktop_windows), exact name, or front:true for the active app. Budgeted (250 nodes / depth 10 defaults; truncation reported as facts). Refs re-resolve at action time; act with ax_desktop_ax_click / set / read.",
+ description: "STRUCTURE-FIRST desktop snapshot (macOS System Events / Linux AT-SPI): the app's accessibility tree with roles, titles, values, bounds, and live refs — the desktop twin of ax_observe. Pick the app by pid (from ax_desktop_windows), exact name, or front:true for the active app. Budgeted (250 nodes / depth 10 defaults; truncation reported as facts). Refs re-resolve at action time; act with ax_desktop_ax_click / set / read.",
inputSchema: {
type: "object",
properties: {
diff --git a/src/mcp/server.js b/src/mcp/server.js
index ec77527..028c839 100644
--- a/src/mcp/server.js
+++ b/src/mcp/server.js
@@ -42,7 +42,14 @@ export async function createServer({ controller = new Controller(), desktop, sol
: await loadSolvers(process.env.AXUSE_PLUGINS === "" ? [] : pluginDirs());
const registry = loaded.solvers ?? loaded;
const loadErrors = loaded.errors ?? [];
- const tools = [...browserTools(), ...desktopTools(!!driver, !!(driver && typeof driver.axSnapshot === "function")), ...solverTools(registry), ...pluginTools(registry)];
+ // AX-structure tools mount only when the driver's platform AX layer is
+ // REALLY usable — on Linux that means AT-SPI answers (dead registries
+ // hide the tools instead of advertising typed failures).
+ let axCapable = !!(driver && typeof driver.axSnapshot === "function");
+ if (axCapable && typeof driver.axCapable === "function") {
+ axCapable = await driver.axCapable().catch(() => false);
+ }
+ const tools = [...browserTools(), ...desktopTools(!!driver, axCapable), ...solverTools(registry), ...pluginTools(registry)];
const byName = new Map(tools.map((t) => [t.name, t]));
diff --git a/test/a1-tools.test.js b/test/a1-tools.test.js
index 989d847..71040e3 100644
--- a/test/a1-tools.test.js
+++ b/test/a1-tools.test.js
@@ -79,7 +79,16 @@ function startServer(env) {
proc.stdin.write(JSON.stringify(msg) + "\n");
return p;
},
- end() { proc.kill(); },
+ end() {
+ proc.kill();
+ // Await the child's exit — the file cannot finish while the server's
+ // async shutdown holds the stdio pipes open.
+ return new Promise((resolve) => {
+ if (proc.exitCode !== null) return resolve();
+ proc.once("exit", () => resolve());
+ setTimeout(resolve, 8_000);
+ });
+ },
};
}
@@ -88,7 +97,7 @@ test("A1: plugin tools() export → validated → tools/list → tools/call →
t.after(() => rmSync(dir, { recursive: true, force: true }));
// Point the loader ONLY at the fixture dir (skip the paid tree).
const srv = startServer({ AXUSE_PLUGINS: dir, AXUSE_LICENSE_A1_DEMO: "x" });
- t.after(() => srv.end());
+ t.after(async () => { await srv.end(); });
await srv.call({ jsonrpc: "2.0", id: 1, method: "initialize", params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "t", version: "0" } } });
srv.proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
@@ -134,7 +143,7 @@ export function tools() {
}
`);
const srv = startServer({ AXUSE_PLUGINS: dir });
- t.after(() => srv.end());
+ t.after(async () => { await srv.end(); });
await srv.call({ jsonrpc: "2.0", id: 1, method: "initialize", params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "t", version: "0" } } });
const plugins = await srv.call({ jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "ax_plugins", arguments: {} } });
const info = JSON.parse(plugins.result.content[0].text);
diff --git a/test/bridge.test.js b/test/bridge.test.js
index 950bc90..3c1a2b1 100644
--- a/test/bridge.test.js
+++ b/test/bridge.test.js
@@ -36,10 +36,11 @@ function extensionCapableBrowser() {
for (const c of edgeCandidates) {
try { accessSync(c); return c; } catch {}
}
- // Branded Chrome 137+ silently ignores --load-extension. On macOS the
- // capability probe therefore excludes it; users can still load unpacked
- // extensions manually in their normal Chrome profile.
- return locateBrowser({ extensionCapable: process.platform === "darwin" });
+ // Branded Chrome 137+ silently ignores --load-extension (macOS .app builds
+ // AND the Linux distro packages — live-verified). The capability probe
+ // excludes both; users can still load unpacked extensions manually in
+ // their normal Chrome profile.
+ return locateBrowser({ extensionCapable: process.platform !== "win32" });
}
function browserKind(executable) {
diff --git a/test/cdp.test.js b/test/cdp.test.js
index 4ded4fd..f7b8cb2 100644
--- a/test/cdp.test.js
+++ b/test/cdp.test.js
@@ -17,6 +17,11 @@ test("macOS browser candidates distinguish branded Chrome from extension-capable
test("browser candidates cover preview channels (beta/canary) per platform", () => {
const all = browserCandidates({ ignoreEnv: true });
+ // Each platform's Edge Beta preview path shape (macOS .app bundle,
+ // Windows "Edge Beta" install dir, Linux distro package name).
+ const edgeBetaShape = process.platform === "win32" ? /Microsoft\\Edge Beta\\Application\\msedge\.exe$/
+ : process.platform === "darwin" ? /Microsoft Edge Beta\.app/
+ : /\/microsoft-edge-beta$/;
if (process.platform === "darwin") {
assert.ok(browserCandidates({ browser: "chrome-beta", ignoreEnv: true }).some((p) => /Chrome Beta\.app/.test(p)));
assert.ok(browserCandidates({ browser: "chrome-canary", ignoreEnv: true }).some((p) => /Chrome Canary\.app/.test(p)));
@@ -25,14 +30,19 @@ test("browser candidates cover preview channels (beta/canary) per platform", ()
assert.ok(all.findIndex((p) => /Chrome Beta\.app/.test(p)) > all.findIndex((p) => /Chrome for Testing\.app/.test(p)),
"preview channels rank AFTER the stable automation builds");
} else {
- assert.ok(browserCandidates({ browser: "chrome-beta", ignoreEnv: true }).some((p) => /chrome-beta/.test(p)));
+ assert.ok(browserCandidates({ browser: "chrome-beta", ignoreEnv: true }).some((p) => /chrome-beta/i.test(p)));
}
// The extension-capable lane excludes EVERY branded Chrome channel
- // (stable, Beta, Canary — all ignore --load-extension; live-verified) and
- // includes Edge Beta, which honors it.
+ // (stable, Beta, Canary — all ignore --load-extension; live-verified on
+ // macOS .app builds AND google-chrome-stable 151 on Ubuntu) and includes
+ // Edge Beta, which honors it.
const extension = browserCandidates({ extensionCapable: true, ignoreEnv: true });
assert.ok(extension.every((p) => !/Google Chrome( Beta| Dev| Canary)?\.app\/Contents\/MacOS\/Google Chrome( Beta| Dev| Canary)?$/.test(p)), "no branded Chrome channel in the extension lane");
- assert.ok(extension.some((p) => /Microsoft Edge Beta\.app/.test(p)), "Edge Beta joins the extension lane");
+ if (process.platform === "linux") {
+ assert.ok(extension.every((p) => !/^\/usr\/bin\/google-chrome(-stable|-beta|-unstable)?$/.test(p)), "no branded Linux Chrome package in the extension lane");
+ assert.ok(extension.some((p) => /\/chromium$/.test(p)) && extension.some((p) => /chrome-for-testing$/.test(p)), "chromium + CfT join the extension lane");
+ }
+ assert.ok(extension.some((p) => edgeBetaShape.test(p)), `Edge Beta joins the extension lane (${process.platform})`);
});
test("isMacBrandedChrome-classified env overrides cannot defeat the extension lane", () => {
@@ -53,6 +63,21 @@ test("isMacBrandedChrome-classified env overrides cannot defeat the extension la
else process.env.AXUSE_BROWSER = saved;
});
+test("branded-Chrome env overrides cannot defeat the extension lane (linux paths)", () => {
+ if (process.platform !== "linux") return;
+ const saved = process.env.AXUSE_BROWSER;
+ // Branded distro packages must not leak into the lane…
+ for (const exe of ["/usr/bin/google-chrome", "/usr/bin/google-chrome-stable", "/opt/google/chrome/chrome"]) {
+ process.env.AXUSE_BROWSER = exe;
+ assert.ok(browserCandidates({ extensionCapable: true }).every((p) => p !== exe), `AXUSE_BROWSER=${exe} must not leak into the extension lane`);
+ }
+ // …but a Chrome-for-Testing override still wins it.
+ process.env.AXUSE_BROWSER = "/tmp/cft/chrome-linux64/chrome";
+ assert.deepEqual(browserCandidates({ extensionCapable: true }), ["/tmp/cft/chrome-linux64/chrome"]);
+ if (saved === undefined) delete process.env.AXUSE_BROWSER;
+ else process.env.AXUSE_BROWSER = saved;
+});
+
test("browser launch reports a typed spawn failure instead of an unhandled child error", async () => {
const profileRoot = mkdtempSync(join(tmpdir(), "axuse launch failure-"));
try {
diff --git a/test/desktop-linux-atspi.test.js b/test/desktop-linux-atspi.test.js
new file mode 100644
index 0000000..8fce601
--- /dev/null
+++ b/test/desktop-linux-atspi.test.js
@@ -0,0 +1,320 @@
+/**
+ * Linux AT-SPI structure layer tests.
+ *
+ * Offline units (D-Bus marshalling round-trips, state decoding) run
+ * everywhere. The live lane is READ-ONLY against the session's AT-SPI
+ * registry (snapshots + reads never post input or mutate apps) and skips
+ * typed when no accessibility bus is reachable — bare Xvfb/CI has none.
+ *
+ * The WRITE paths (ax_click DoAction, ax_set EditableText) are verified on
+ * the driver's private-Xvfb lane only via the coordinate fallback; posting
+ * real actions into the operator's session is out of bounds for tests.
+ * They are exercised end-to-end by the manual gate
+ * AXUSE_TEST_LINUX_ATSPI_ACTION=1 (self-hosted lane).
+ */
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { spawn, spawnSync } from "node:child_process";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { marshalValue, unmarshalValue, parseDbusAddress } from "../src/desktop/dbus.js";
+import { AtspiClient, STATE_NAMES } from "../src/desktop/atspi.js";
+import { X11Client } from "../src/desktop/x11.js";
+
+const xvfb = spawnSync("which", ["Xvfb"]).status === 0;
+
+/** Boot a private Xvfb (same pattern as desktop-linux.test.js). */
+async function startXvfb() {
+ for (const num of [977, 978, 979, 980]) {
+ const display = `:${num}`;
+ const proc = spawn("Xvfb", [display, "-screen", "0", "1280x800x24", "-ac", "-nolisten", "tcp"], { stdio: "ignore" });
+ const ok = await new Promise((resolve) => {
+ const t = setTimeout(() => resolve(false), 2500);
+ proc.once("exit", () => { clearTimeout(t); resolve(false); });
+ setTimeout(async () => {
+ try {
+ const c = new X11Client(display);
+ const probe = c.connect().then(() => { c.close(); return true; }).catch(() => false);
+ const good = await Promise.race([probe, new Promise((r) => setTimeout(() => r(false), 1500))]);
+ clearTimeout(t);
+ resolve(good);
+ } catch {
+ clearTimeout(t);
+ resolve(false);
+ }
+ }, 700);
+ });
+ if (ok) return { display, stop: () => { try { proc.kill(); } catch {} } };
+ try { proc.kill(); } catch {}
+ }
+ return null;
+}
+
+/* ------------------------------------------------------------ offline units */
+
+test("atspi: D-Bus address parsing", () => {
+ assert.deepEqual(parseDbusAddress("unix:path=/run/user/1000/bus"), { path: "/run/user/1000/bus" });
+ assert.deepEqual(parseDbusAddress("unix:abstract=/tmp/dbus-abc"), { abstract: "/tmp/dbus-abc" });
+ assert.equal(parseDbusAddress("tcp:host=localhost"), null);
+});
+
+test("atspi: D-Bus marshalling round-trips the AT-SPI signatures", () => {
+ const cases = [
+ ["a(so)", [[":1.22", "/org/a11y/atspi/accessible/1"], [":1.4", "/org/a11y/atspi/accessible/root"]]],
+ ["(iiii)", [70, 27, 1850, 1053]],
+ ["(au)", [[0, 1, 2]]],
+ ["as", ["org.a11y.atspi.Accessible", "org.a11y.atspi.Action"]],
+ ["v", { sig: "d", value: 0.5 }],
+ ["s", "gnome-terminal-server"],
+ ["u", 42],
+ ["i", -1],
+ ["a{ss}", [{ key: "pid", value: "1234" }]],
+ ["(so)", [":1.9", "/org/a11y/atspi/accessible/12"]],
+ ["", null],
+ ];
+ for (const [sig, value] of cases) {
+ if (sig === "") {
+ assert.equal(marshalValue("s", "x").length > 0, true);
+ continue;
+ }
+ const bytes = marshalValue(sig, value);
+ // Marshalled values live inside 8-aligned message bodies: verify that
+ // the decode works both raw and at an 8-byte offset (alignment is
+ // position-relative inside real messages).
+ const back = unmarshalValue(sig, bytes);
+ assert.deepEqual(back, value, `round-trip ${sig}`);
+ const padded = unmarshalValue(sig, Buffer.concat([Buffer.alloc(8), bytes]), 8);
+ assert.deepEqual(padded, value, `round-trip ${sig} at 8-offset`);
+ }
+ // Header-shaped array of (yv) structs round-trips with per-element padding.
+ const fields = [[1, { sig: "o", value: "/org/a11y/atspi/accessible/root" }], [6, { sig: "s", value: "org.a11y.atspi.Registry" }]];
+ const fb = marshalValue("a(yv)", fields);
+ assert.deepEqual(unmarshalValue("a(yv)", fb), fields, "a(yv) round-trip");
+});
+
+test("atspi: state bit decoding covers the operational states", () => {
+ // checked=4, editable=7, enabled=8, focused=12, selected=23, showing=25
+ const words = new Array(1).fill(0);
+ for (const bit of [4, 7, 8, 12, 23, 25]) words[0] |= 1 << bit;
+ const c = new AtspiClient();
+ // decodeStates is internal; exercise it through the exported table shape.
+ assert.equal(STATE_NAMES[4], "checked");
+ assert.equal(STATE_NAMES[12], "focused");
+ assert.equal(STATE_NAMES[25], "showing");
+ assert.ok(STATE_NAMES.length >= 31);
+ void words; void c;
+});
+
+/* --------------------------------------------------------------- live lane */
+
+/** Read-only AT-SPI availability probe. */
+async function atspiReachable() {
+ if (!process.env.DISPLAY) return false;
+ if (!process.env.DBUS_SESSION_BUS_ADDRESS && !process.env.AXUSE_ATSPI_BUS) return false;
+ try {
+ const c = await new AtspiClient().connect();
+ await c.apps();
+ c.close();
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+/** GTK available for the sandbox app (python3 + gi + Gtk 3.0). */
+const gtkOk = (() => {
+ const probe = spawnSync("python3", ["-c",
+ 'import gi; gi.require_version("Gtk", "3.0"); from gi.repository import Gtk, GLib'],
+ { encoding: "utf8", timeout: 10_000 });
+ return probe.status === 0;
+})();
+const sandboxCapable = xvfb && gtkOk
+ && spawnSync("which", ["dbus-run-session"]).status === 0;
+
+/** The sandbox GTK app: observable button (title flip), entry, check. */
+const SANDBOX_APP = `import gi
+gi.require_version("Gtk", "3.0")
+from gi.repository import Gtk, GLib
+GLib.set_prgname("axuse-sandbox")
+win = Gtk.Window(title="axuse-sandbox")
+win.set_default_size(420, 220)
+box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8)
+box.set_border_width(12)
+label = Gtk.Label(label="state: off")
+label.get_accessible().set_name("state label")
+button = Gtk.Button(label="Click me")
+entry = Gtk.Entry()
+entry.get_accessible().set_name("sandbox entry")
+check = Gtk.CheckButton(label="Enable feature")
+check.get_accessible().set_name("enable check")
+def clicked(_b):
+ label.set_text("state: on")
+ win.set_title("state: on")
+button.connect("clicked", clicked)
+for w in (label, button, entry, check):
+ box.pack_start(w, False, False, 4)
+win.add(box)
+win.show_all()
+win.connect("destroy", Gtk.main_quit)
+Gtk.main()
+`;
+
+test("atspi: action sandbox — snapshot, ax_set, ax_click on a private bus+Xvfb+GTK app", { skip: !sandboxCapable ? "requires Xvfb + dbus-run-session + python3-gi Gtk3 (sudo apt install xvfb python3-gi gir1.2-gtk-3.0)" : false }, async (t) => {
+ const dir = mkdtempSync(join(tmpdir(), "axuse-atspi-sandbox-"));
+ const appPath = join(dir, "app.py");
+ writeFileSync(appPath, SANDBOX_APP);
+ const addrFile = join(dir, "bus-address");
+ t.after(() => { try { rmSync(dir, { recursive: true, force: true }); } catch {} });
+
+ const server = await startXvfb(); // private display (suite helper)
+ assert.ok(server);
+ t.after(() => server.stop());
+ const session = spawn("dbus-run-session", ["--", "bash", "-c",
+ `echo "$DBUS_SESSION_BUS_ADDRESS" > ${JSON.stringify(addrFile)}; ` +
+ `DISPLAY=${server.display} GTK_MODULES=atk-bridge python3 ${JSON.stringify(appPath)};`],
+ // Own process group: teardown must not leave the bash/python children
+ // alive (a lone GTK app keeps re-registering with AT-SPI forever).
+ { stdio: ["ignore", "ignore", "ignore"], detached: true });
+ t.after(async () => {
+ // Group-kill the whole sandbox (dbus-run-session → bash → python).
+ try { process.kill(-session.pid, "SIGKILL"); } catch {}
+ // D-Bus activation services ESCAPE dbus-run-session's teardown — the
+ // at-spi launcher it activated lingers with its registry. Kill exactly
+ // the owner of the sandbox's org.a11y.Bus (never the session's copy).
+ try {
+ const { DbusClient } = await import("../src/desktop/dbus.js");
+ const bus = await new DbusClient(sessionAddress).connect();
+ const [owner] = await bus.call("org.freedesktop.DBus", "/org/freedesktop/DBus",
+ "org.freedesktop.DBus", "GetNameOwner", "s", ["org.a11y.Bus"]);
+ const [pid] = await bus.call("org.freedesktop.DBus", "/org/freedesktop/DBus",
+ "org.freedesktop.DBus", "GetConnectionUnixProcessID", "s", [owner]);
+ bus.close();
+ if (Number(pid) !== process.pid && Number(pid) > 1) {
+ try { process.kill(Number(pid), "SIGTERM"); } catch {}
+ }
+ } catch {}
+ try { session.kill("SIGTERM"); } catch {}
+ });
+
+ // The bus address appears within a second; org.a11y.Bus + the at-spi
+ // registry activate on demand when atk-bridge queries them.
+ let sessionAddress = null;
+ for (let i = 0; i < 40 && !sessionAddress; i++) {
+ try { sessionAddress = readFileSync(addrFile, "utf8").trim(); } catch { await new Promise((r) => setTimeout(r, 150)); }
+ }
+ assert.ok(sessionAddress, "private session bus started");
+
+ const { LinuxDriver } = await import("../src/desktop/linux.js");
+ const d = new LinuxDriver(server.display, { atspiSessionBus: sessionAddress });
+ t.after(() => d.dispose());
+
+ // Wait for the app to register with the sandbox registry.
+ let registered = false;
+ for (let i = 0; i < 50 && !registered; i++) {
+ try {
+ const atspi = await d.ensureAtspi();
+ const apps = await atspi.apps();
+ registered = apps.some((a) => a.name === "axuse-sandbox");
+ if (!registered) { d.atspi.close(); d.atspi = null; }
+ } catch { try { d.atspi?.close(); } catch {} d.atspi = null; }
+ if (!registered) await new Promise((r) => setTimeout(r, 250));
+ }
+ assert.ok(registered, "GTK app registered with AT-SPI");
+
+ // Structure-first: snapshot finds the exact widgets.
+ const snap = await d.axSnapshot({ name: "axuse-sandbox", maxNodes: 60, maxDepth: 8 });
+ assert.ok(snap.node_count >= 5, `tree walked (${snap.node_count} nodes)`);
+ const entry = snap.nodes.find((n) => n.name === "sandbox entry");
+ const button = snap.nodes.find((n) => n.name === "Click me");
+ const check = snap.nodes.find((n) => n.name === "enable check");
+ const frame = snap.nodes.find((n) => n.ref === "ax0.0");
+ assert.ok(entry && button && check && frame, "widgets found by accessible name");
+
+ // ax_set: type into the entry through EditableText, verified by read-back.
+ const set = await d.axSet({ ref: entry.ref, name: "axuse-sandbox", value: "typed via atspi" });
+ assert.equal(set.via, "atspi-editabletext");
+ assert.equal(set.read_back_matches, true, "entry read-back matches");
+
+ // ax_click (button): DoAction flips the window title — observed on the frame.
+ const click = await d.axClick({ ref: button.ref, name: "axuse-sandbox" });
+ assert.equal(click.via, "atspi-action");
+ const after = await d.axRead({ ref: frame.ref, name: "axuse-sandbox" });
+ assert.equal(after.name, "state: on", "trusted DoAction activated the real button");
+
+ // ax_click (check box): state gains "checked".
+ await d.axClick({ ref: check.ref, name: "axuse-sandbox" });
+ const checkAfter = await d.axRead({ ref: check.ref, name: "axuse-sandbox" });
+ assert.ok(checkAfter.states.includes("checked"), `check toggled (states: ${checkAfter.states.join(",")})`);
+
+ // Non-editable ax_set fails typed with the interfaces listed.
+ await assert.rejects(
+ () => d.axSet({ ref: frame.ref, name: "axuse-sandbox", value: "nope" }),
+ (e) => e.code === "COMMAND_REJECTED" && /not AT-SPI-editable/.test(e.message),
+ );
+});
+
+test("atspi: snapshot + read round-trip on a live app (read-only)", { skip: !(await atspiReachable()) ? "requires a session AT-SPI registry (DBUS_SESSION_BUS_ADDRESS + at-spi2 running; bare Xvfb/CI skips)" : false }, async (t) => {
+ const { LinuxDriver } = await import("../src/desktop/linux.js");
+ const d = new LinuxDriver(process.env.DISPLAY);
+ t.after(() => d.dispose());
+
+ // The registry being reachable but EMPTY is a legitimate session state
+ // (apps bridge on demand and deregister when the last assistive client
+ // leaves) — skip typed rather than fail. Same for a transport that dies
+ // mid-test: the on-demand at-spi launcher comes and goes with clients.
+ const atspi = await d.ensureAtspi().catch((e) => {
+ t.skip(`AT-SPI transport unavailable mid-test (${e.message.slice(0, 80)})`);
+ return null;
+ });
+ if (!atspi) return;
+ const apps = await atspi.apps().catch(() => {
+ t.skip("AT-SPI registry went away mid-test (on-demand lifecycle)");
+ return [];
+ });
+ if (!apps.some((a) => a.name)) {
+ t.skip("session registry reachable but no apps registered (on-demand bridging)");
+ return;
+ }
+ assert.ok(apps.length >= 1, "desktop lists apps");
+ const withUi = [];
+ for (const a of apps) {
+ try { if ((await atspi.childCount(a.ref)) > 0) withUi.push(a); } catch {}
+ if (withUi.length >= 3) break; // a few candidates is plenty
+ }
+ assert.ok(withUi.length >= 1, "at least one app exposes a UI tree");
+
+ const snap = await d.axSnapshot({ name: withUi[0].name, maxNodes: 60, maxDepth: 6 });
+ assert.equal(snap.app, withUi[0].name);
+ assert.ok(snap.node_count >= 1);
+ assert.equal(typeof snap.truncated, "boolean");
+ assert.equal(snap.nodes[0].ref, "ax0");
+ assert.ok(snap.nodes[0].role.length > 0, "root carries a role");
+ // Every node beyond the root has an index-path ref.
+ for (const n of snap.nodes.slice(1)) assert.match(n.ref, /^ax0(\.\d+)+$/);
+
+ // Budget facts: a tiny node budget truncates honestly.
+ const tiny = await d.axSnapshot({ name: withUi[0].name, maxNodes: 3, maxDepth: 6 });
+ assert.equal(tiny.node_count <= 3, true);
+ if (tiny.node_count === 3) assert.equal(tiny.truncated, true);
+
+ // ax_read round-trip on the app ROOT ref (stable by construction).
+ const read = await d.axRead({ ref: "ax0", name: withUi[0].name });
+ assert.equal(read.role, "application");
+ assert.equal(read.ref, "ax0");
+
+ // Stale/garbage refs fail typed with a reobserve hint.
+ await assert.rejects(
+ () => d.axRead({ ref: "nope", name: withUi[0].name }),
+ (e) => e.code === "REF_NOT_FOUND" && e.retry === "reobserve",
+ );
+ await assert.rejects(
+ () => d.axRead({ ref: "ax0.9999", name: withUi[0].name }),
+ (e) => e.code === "REF_NOT_FOUND",
+ );
+ // Unknown app name fails typed with the available list in the hint.
+ await assert.rejects(
+ () => d.axSnapshot({ name: "definitely-not-an-app-xyz" }),
+ (e) => e.code === "REF_NOT_FOUND" && /Available:/.test(e.hint ?? ""),
+ );
+});
diff --git a/test/desktop-linux.test.js b/test/desktop-linux.test.js
new file mode 100644
index 0000000..7aa8245
--- /dev/null
+++ b/test/desktop-linux.test.js
@@ -0,0 +1,316 @@
+/**
+ * Linux desktop driver (X11) tests.
+ *
+ * The live lane is SELF-CONTAINED: when Xvfb is installed the suite spawns a
+ * private display and drives real windows (xev/xmessage) on it — the
+ * operator's session is never touched, and CI gets genuine input-synthesis
+ * coverage. Without Xvfb the live tests skip typed (allowlisted).
+ *
+ * Offline units (Xauthority parsing, the evdev fallback map, JPEG structure)
+ * run everywhere.
+ */
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { spawn, spawnSync } from "node:child_process";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { parseDisplay, readXauthority, X11Client } from "../src/desktop/x11.js";
+import { encodeJPEG } from "../src/desktop/jpeg.js";
+
+/* ------------------------------------------------------------- offline units */
+
+test("linux desktop: DISPLAY strings parse (local, screen, tcp)", () => {
+ assert.deepEqual(parseDisplay(":1"), { host: "", display: 1, screen: null });
+ assert.deepEqual(parseDisplay(":0.0"), { host: "", display: 0, screen: 0 });
+ assert.deepEqual(parseDisplay("unix:3"), { host: "unix", display: 3, screen: null });
+ assert.deepEqual(parseDisplay("ho.st:4.1"), { host: "ho.st", display: 4, screen: 1 });
+ assert.equal(parseDisplay("bogus"), null);
+ assert.equal(parseDisplay(""), null);
+});
+
+test("linux desktop: Xauthority records decode (synthetic file)", () => {
+ // Two records: a wildcard cookie and one for display :42.
+ const rec = (family, num, data) => {
+ const addr = Buffer.from("host1");
+ const name = Buffer.from("MIT-MAGIC-COOKIE-1");
+ const parts = [family, addr.length, addr, num.length, Buffer.from(num), name.length, name, data.length, data];
+ const size = parts.reduce((n, p) => n + (typeof p === "number" ? 2 : p.length), 0);
+ const buf = Buffer.alloc(size);
+ let o = 0;
+ const u16 = (v) => { buf.writeUInt16LE(v, o); o += 2; };
+ for (const p of parts) {
+ if (typeof p === "number") u16(p);
+ else { p.copy(buf, o); o += p.length; }
+ }
+ return buf;
+ };
+ const file = mkdtempSync(join(tmpdir(), "axuse-xauth-"));
+ const path = join(file, ".Xauthority");
+ writeFileSync(path, Buffer.concat([
+ rec(65535, "", Buffer.alloc(16, 0xaa)),
+ rec(256, "42", Buffer.alloc(16, 0xbb)),
+ ]));
+ const records = readXauthority(path);
+ assert.equal(records.length, 2);
+ assert.equal(records[0].family, 65535);
+ assert.equal(records[1].name, "MIT-MAGIC-COOKIE-1");
+ assert.equal(records[1].data.length, 16);
+ assert.deepEqual([...records[1].data], Array(16).fill(0xbb));
+ rmSync(file, { recursive: true, force: true });
+});
+
+test("linux desktop: evdev fallback map is QWERTY-physical, not alphabetical", async () => {
+ // Can only load the table through a client (no live server needed for the
+ // static map — applyEvdevMap is deterministic).
+ const c = new X11Client(":0");
+ c.applyEvdevMap();
+ const at = (ks) => c.keyForKeysym(ks);
+ // a s d f g h j k l — 38..46; z x c v b n m — 52..58.
+ assert.equal(at(0x61)?.keycode, 38);
+ assert.equal(at(0x68)?.keycode, 43); // h (KEY_H=35) — NOT 45
+ assert.equal(at(0x6b)?.keycode, 45); // k
+ assert.equal(at(0x7a)?.keycode, 52); // z (KEY_Z=44) — NOT 63
+ assert.equal(at(0x6d)?.keycode, 58); // m
+ // Shift columns: uppercase needs shift on the same physical key.
+ assert.equal(at(0x41)?.keycode, 38);
+ assert.equal(at(0x41)?.shift, true);
+ assert.equal(at(0x61)?.shift, false);
+ // Named keys at canonical evdev positions.
+ assert.equal(at(0xff0d)?.keycode, 36); // Return
+ assert.equal(at(0xffe3)?.keycode, 37); // Control_L
+ assert.equal(at(0xffeb)?.keycode, 133); // Super_L
+ assert.equal(at(0xffbe)?.keycode, 67); // F1
+ assert.equal(at(0xffc2)?.keycode, 71); // F5 (KEY_F5=63+8)
+});
+
+test("linux desktop: baseline JPEG encoder structure and quality scaling", () => {
+ // Real content (gradient + noise) so quality affects size.
+ const w = 160, h = 120;
+ const rgb = Buffer.alloc(w * h * 3);
+ for (let y = 0; y < h; y++) {
+ for (let x = 0; x < w; x++) {
+ const o = (y * w + x) * 3;
+ rgb[o] = (x * 255 / w) | 0;
+ rgb[o + 1] = (y * 255 / h) | 0;
+ rgb[o + 2] = ((x * y) % 256);
+ }
+ }
+ const q90 = encodeJPEG(rgb, w, h, w * 3, 90);
+ const q30 = encodeJPEG(rgb, w, h, w * 3, 30);
+ // SOI … SOF0 … EOI structure.
+ assert.equal(q90[0], 0xff); assert.equal(q90[1], 0xd8);
+ const eoi = q90.length - 2;
+ assert.equal(q90[eoi], 0xff); assert.equal(q90[eoi + 1], 0xd9);
+ assert.ok(q90.includes(0xffc0, 2), "SOF0 present");
+ assert.ok(q90.includes(0xffc4, 2), "DHT present");
+ assert.ok(q90.includes(0xffdb, 2), "DQT present");
+ // Dimensions big-endian in SOF0 (marker(2) length(2) precision(1) h(2) w(2)).
+ const sof = q90.indexOf(Buffer.from([0xff, 0xc0]));
+ assert.equal(q90.readUInt16BE(sof + 5), h);
+ assert.equal(q90.readUInt16BE(sof + 7), w);
+ assert.ok(q90.length > q30.length, `q90 (${q90.length}) must outweigh q30 (${q30.length})`);
+ assert.ok(q30.length > 300, "q30 is still a real image");
+});
+
+/* ---------------------------------------------------------------- live lane */
+
+const xvfb = spawnSync("which", ["Xvfb"]).status === 0;
+
+/** Boot a private Xvfb; resolves { display, stop } or null when impossible. */
+async function startXvfb() {
+ for (const num of [977, 978, 979, 980]) {
+ const display = `:${num}`;
+ const proc = spawn("Xvfb", [display, "-screen", "0", "1280x800x24", "-ac", "-nolisten", "tcp"], {
+ stdio: "ignore",
+ });
+ const ok = await new Promise((resolve) => {
+ const t = setTimeout(() => resolve(false), 2500);
+ proc.once("exit", () => { clearTimeout(t); resolve(false); });
+ // probe: connect a throwaway client
+ setTimeout(async () => {
+ try {
+ const c = new X11Client(display);
+ const probe = c.connect().then(() => { c.close(); return true; }).catch(() => false);
+ const good = await Promise.race([probe, new Promise((r) => setTimeout(() => r(false), 1500))]);
+ clearTimeout(t);
+ resolve(good);
+ } catch {
+ clearTimeout(t);
+ resolve(false);
+ }
+ }, 700);
+ });
+ if (ok) {
+ return { display, stop: () => { try { proc.kill(); } catch {} } };
+ }
+ try { proc.kill(); } catch {}
+ }
+ return null;
+}
+
+test("linux desktop: driver drives a private Xvfb (input, capture, windows)", { skip: !xvfb ? "requires Xvfb for a private display (sudo apt install xvfb)" : false }, async (t) => {
+ const server = await startXvfb();
+ assert.ok(server, "could not start a private Xvfb on :977-:980");
+ t.after(() => server.stop());
+ const { LinuxDriver } = await import("../src/desktop/linux.js");
+ const d = new LinuxDriver(server.display);
+ t.after(() => d.dispose());
+
+ // screen_info: the private display's geometry (+ RandR monitor layout).
+ const info = await d.screenInfo();
+ assert.equal(info.w, 1280);
+ assert.equal(info.h, 800);
+ assert.match(info.coordinate_space, /X11 root window pixels/);
+ assert.ok(info.monitors?.length === 1 && info.monitors[0].x === 0 && info.monitors[0].y === 0
+ && info.monitors[0].w === 1280 && info.monitors[0].h === 800, "RandR reports the Xvfb screen (primary unset on bare Xvfb is fine)");
+
+ // screenshot: decodable JPEG within budget (verified structurally; PIL is
+ // not a dependency, so assert SOI/SOF0/EOI + dims + budget).
+ const shot = await d.screenshot({ x: 0, y: 0, w: 320, h: 240, budgetBytes: 60_000 });
+ assert.equal(shot.mime, "image/jpeg");
+ const jpg = Buffer.from(shot.base64, "base64");
+ assert.equal(jpg[0], 0xff); assert.equal(jpg[1], 0xd8);
+ assert.ok(shot.bytes <= 60_000);
+ assert.ok(shot.bytes > 500);
+ await assert.rejects(
+ () => d.screenshot({ x: 0, y: 0, w: 1280, h: 800, budgetBytes: 1_500 }),
+ (e) => e.code === "SCREENSHOT_BUDGET_EXCEEDED",
+ );
+ await assert.rejects(
+ () => d.screenshot({ x: 5000, y: 5000, w: 10, h: 10 }),
+ (e) => e.code === "COMMAND_REJECTED",
+ );
+
+ // input: keys/clicks/wheel/drag observed through xev on the private display.
+ const xev = spawn("xev", ["-display", server.display, "-geometry", "300x200+400+100"], { stdio: ["ignore", "pipe", "pipe"] });
+ t.after(() => { try { xev.kill(); } catch {} });
+ let out = "";
+ xev.stdout.on("data", (c) => { out += String(c); });
+ await new Promise((r) => setTimeout(r, 700));
+ const wid = Number(out.match(/Outer window is (0x[0-9a-f]+)/)?.[1]);
+ assert.ok(wid > 0, "xev window mapped");
+
+ await d.activate({ windowId: wid }); // WM-less: focuses via SetInputFocus
+ await d.press("a");
+ await d.type("hello"); // short + typeable → keysym path
+ await d.type("line one\nline two\ttabbed"); // \n → Return, \t → Tab
+ await d.press("shift+a");
+ await d.press("ctrl+z");
+ await d.click({ x: 500, y: 150, button: "left" });
+ await d.click({ x: 520, y: 160, button: "right" });
+ await d.wheel(0, 240);
+ await d.drag({ from: { x: 450, y: 150 }, to: { x: 650, y: 250 }, steps: 8 });
+ await new Promise((r) => setTimeout(r, 500));
+
+ const keys = [...out.matchAll(/KeyPress[\s\S]*?keycode \d+ \(keysym 0x[0-9a-f]+, ([a-zA-Z0-9_]+)\)/g)].map((m) => m[1]);
+ for (const k of ["a", "h", "e", "l", "o", "A", "z", "Return", "Tab"]) assert.ok(keys.includes(k), `key ${k} observed (saw: ${keys.join(",")})`);
+ const presses = out.split("ButtonPress event").slice(1).map((b) => b.match(/button (\d+)/)?.[1]);
+ assert.ok(presses.includes("1"), "left click observed");
+ assert.ok(presses.includes("3"), "right click observed");
+ const pressCoords = out.split("ButtonPress event").slice(1)
+ .map((b) => b.match(/\(\d+,\d+\), root:\((\d+),(\d+)\)/))
+ .filter(Boolean).map((m) => [Number(m[1]), Number(m[2])]);
+ assert.ok(pressCoords.some(([x, y]) => x === 500 && y === 150), `click landed at (500,150), saw ${JSON.stringify(pressCoords)}`);
+ assert.equal(presses.filter((b) => b === "5").length, 2, "wheel dy=240 → two button-5 notches");
+ assert.ok((out.match(/MotionNotify/g) ?? []).length >= 8, "drag interpolated motion observed");
+
+ // windows(): no WM on bare Xvfb → typed empty, never a crash.
+ const w = await d.windows();
+ assert.equal(w.windows.length, 0);
+
+ // screen_info stays honest after everything.
+ const info2 = await d.screenInfo();
+ assert.equal(info2.w, 1280);
+});
+
+test("linux desktop: screenshot budget ladder downscales instead of dying", { skip: !xvfb ? "requires Xvfb for a private display (sudo apt install xvfb)" : false }, async (t) => {
+ const server = await startXvfb();
+ assert.ok(server);
+ t.after(() => server.stop());
+ const { LinuxDriver } = await import("../src/desktop/linux.js");
+ const d = new LinuxDriver(server.display);
+ t.after(() => d.dispose());
+
+ // Fill the screen with content a flat-black Xvfb lacks: windows with text.
+ const wins = [];
+ t.after(() => { for (const w of wins) { try { w.kill(); } catch {} } });
+ for (let i = 0; i < 6; i++) {
+ wins.push(spawn("xmessage", ["-display", server.display, "-geometry", `600x400+${(i % 3) * 300}+${Math.floor(i / 3) * 250}`, `content row ${i} lorem ipsum dolor sit amet ${"x".repeat(40)}`], { stdio: "ignore" }));
+ }
+ await new Promise((r) => setTimeout(r, 900));
+
+ // A full-screen capture under a harsh budget must still return a decodable
+ // image (SOF0 dims prove which rung of the ladder answered).
+ const shot = await d.screenshot({ x: 0, y: 0, w: 1280, h: 800, budgetBytes: 20_000 });
+ const jpg = Buffer.from(shot.base64, "base64");
+ const sof = jpg.indexOf(Buffer.from([0xff, 0xc0]));
+ assert.ok(sof > 0, "SOF0 present");
+ const h = jpg.readUInt16BE(sof + 5);
+ const w = jpg.readUInt16BE(sof + 7);
+ assert.ok(shot.bytes <= 20_000, `bytes ${shot.bytes} within budget`);
+ assert.ok(w < 1280 || h < 800, `ladder downscaled (got ${w}x${h}) — quality alone should not fit rich content in 20KB`);
+ assert.ok(w >= 320 && h >= 200, `still legible (${w}x${h})`);
+ assert.ok(shot.downscaled, "downscale note present");
+});
+
+test("linux desktop: long text pastes through the clipboard when xclip exists", { skip: !xvfb ? "requires Xvfb for a private display (sudo apt install xvfb)" : false }, async (t) => {
+ const hasXclip = spawnSync("which", ["xclip"]).status === 0;
+ if (!hasXclip) return; // allowlisted via the suite's xvfb gate reason
+ const server = await startXvfb();
+ assert.ok(server);
+ t.after(() => server.stop());
+ const { LinuxDriver } = await import("../src/desktop/linux.js");
+ const d = new LinuxDriver(server.display);
+ t.after(() => d.dispose());
+
+ // Something focused to receive ctrl+v.
+ const xev = spawn("xev", ["-display", server.display, "-geometry", "300x200+400+100"], { stdio: ["ignore", "pipe", "pipe"] });
+ t.after(() => { try { xev.kill(); } catch {} });
+ let out = "";
+ xev.stdout.on("data", (c) => { out += String(c); });
+ await new Promise((r) => setTimeout(r, 700));
+ const wid = Number(out.match(/Outer window is (0x[0-9a-f]+)/)?.[1]);
+ await d.activate({ windowId: wid });
+
+ const payload = "This sentence is deliberately longer than forty-eight characters to take the clipboard path.";
+ const r1 = await d.type(payload);
+ assert.equal(r1.via, "clipboard-paste");
+ assert.equal(r1.clipboard_replaced, true);
+ await new Promise((r) => setTimeout(r, 300));
+ xev.kill();
+
+ // The clipboard itself carries the payload (xclip -o against that display).
+ const readback = spawnSync("xclip", ["-display", server.display, "-selection", "clipboard", "-out"], { encoding: "utf8" });
+ assert.equal(readback.stdout.trim(), payload);
+
+ // The paste really pressed ctrl+v into the focused window.
+ assert.match(out, /KeyPress[\s\S]*?Control[\s\S]*?keysym 0x76, v/, "ctrl+v observed");
+});
+
+test("linux desktop: JPEG decodes in an independent decoder when PIL exists", { skip: !xvfb ? "requires Xvfb for a private display (sudo apt install xvfb)" : false }, async (t) => {
+ const server = await startXvfb();
+ assert.ok(server);
+ t.after(() => server.stop());
+ const { LinuxDriver } = await import("../src/desktop/linux.js");
+ const d = new LinuxDriver(server.display);
+ t.after(() => d.dispose());
+ // Red window, then verify the average pixel is red through PIL.
+ const xmsg = spawn("xmessage", ["-display", server.display, "-geometry", "200x100+50+50", "-bg", "red", "RED"], { stdio: "ignore" });
+ t.after(() => { try { xmsg.kill(); } catch {} });
+ await new Promise((r) => setTimeout(r, 700));
+ const shot = await d.screenshot({ x: 50, y: 50, w: 200, h: 100, budgetBytes: 60_000 });
+ const path = join(tmpdir(), `axuse-linux-shot-${Date.now()}.jpg`);
+ writeFileSync(path, Buffer.from(shot.base64, "base64"));
+ const check = spawnSync("python3", ["-c", `
+from PIL import Image
+import sys
+im = Image.open(${JSON.stringify(path)}).convert("RGB")
+px = list(im.getdata())
+n = len(px)
+avg = tuple(sum(c[i] for c in px)//n for i in range(3))
+sys.exit(0 if avg[0] > 150 and avg[1] < 60 and avg[2] < 60 else 2)
+`]);
+ rmSync(path, { force: true });
+ assert.equal(check.status, 0, `PIL verification failed (rc=${check.status})`);
+});
diff --git a/test/mcp-catalog.test.js b/test/mcp-catalog.test.js
index 387792a..7b796ae 100644
--- a/test/mcp-catalog.test.js
+++ b/test/mcp-catalog.test.js
@@ -46,13 +46,22 @@ function startServer() {
proc.stdin.write(JSON.stringify(msg) + "\n");
return p;
},
- end() { proc.kill(); },
+ end() {
+ proc.kill();
+ // Await the child's exit — the file cannot finish while the server's
+ // async shutdown holds the stdio pipes open.
+ return new Promise((resolve) => {
+ if (proc.exitCode !== null) return resolve();
+ proc.once("exit", () => resolve());
+ setTimeout(resolve, 8_000);
+ });
+ },
};
}
test("real MCP server + real catalog: every capability tool mounts; solve signposts honestly", { skip: !catalogPresent }, async (t) => {
const srv = startServer();
- t.after(() => srv.end());
+ t.after(async () => { await srv.end(); });
await srv.call({ jsonrpc: "2.0", id: 1, method: "initialize", params: { protocolVersion: "2025-03-26", capabilities: {}, clientInfo: { name: "t", version: "0" } } });
srv.proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
diff --git a/test/mcp.test.js b/test/mcp.test.js
index aea1ad5..0e187ef 100644
--- a/test/mcp.test.js
+++ b/test/mcp.test.js
@@ -39,13 +39,22 @@ function startServer() {
proc.stdin.write(JSON.stringify(msg) + "\n");
return p;
},
- end() { proc.kill(); },
+ end() {
+ proc.kill();
+ // The server's shutdown is async (browser/profile disposal) — the
+ // file cannot finish while the child's stdio pipes stay open.
+ return new Promise((resolve) => {
+ if (proc.exitCode !== null) return resolve();
+ proc.once("exit", () => resolve());
+ setTimeout(resolve, 8_000);
+ });
+ },
};
}
test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser }, async (t) => {
const srv = startServer();
- t.after(() => srv.end());
+ t.after(async () => { await srv.end(); });
const init = await srv.call({
jsonrpc: "2.0", id: 1, method: "initialize",
@@ -63,22 +72,46 @@ test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser },
assert.ok(names.includes("ax_click"));
assert.ok(names.includes("ax_screenshot"));
assert.ok(names.includes("ax_open_many"));
- // Desktop tools are advertised when the platform driver is enabled.
- const desktopExpected = process.platform === "win32" || (process.platform === "darwin" && process.env.AXUSE_ENABLE_MACOS_CUA === "1");
+ // Desktop tools are advertised when the platform driver is enabled. On
+ // Linux the driver mounts whenever DISPLAY is set AND an X server answers
+ // (CI runs a real Xvfb on :99; dead DISPLAY values must not count).
+ let linuxDesktopOk = false;
+ let linuxAtspiOk = false;
+ if (process.platform === "linux" && process.env.DISPLAY) {
+ linuxDesktopOk = await import("../src/desktop/index.js").then(async ({ resolveDesktopDriver }) => {
+ const d = await resolveDesktopDriver().catch(() => null);
+ if (!d) return false;
+ try { await d.screenInfo(); return true; } catch { return false; }
+ finally { d.dispose?.(); }
+ });
+ if (linuxDesktopOk) {
+ linuxAtspiOk = await import("../src/desktop/index.js").then(async ({ resolveDesktopDriver }) => {
+ const d = await resolveDesktopDriver().catch(() => null);
+ if (!d) return false;
+ try { return await d.axCapable(); } catch { return false; }
+ finally { d.dispose?.(); }
+ });
+ }
+ }
+ const desktopExpected = process.platform === "win32"
+ || (process.platform === "darwin" && process.env.AXUSE_ENABLE_MACOS_CUA === "1")
+ || linuxDesktopOk;
if (desktopExpected) {
assert.ok(names.includes("ax_desktop_screenshot"));
} else {
assert.ok(!names.includes("ax_desktop_screenshot"));
}
- // The macOS AX-structure tools mount ONLY with the macOS driver;
- // Windows (coordinate driver) must not advertise them.
- if (process.platform === "darwin" && process.env.AXUSE_ENABLE_MACOS_CUA === "1") {
+ // The AX-structure tools mount ONLY with a structure-first driver
+ // (macOS System Events; Linux AT-SPI). Windows (coordinate driver) must
+ // not advertise them.
+ const axExpected = (process.platform === "darwin" && process.env.AXUSE_ENABLE_MACOS_CUA === "1") || linuxAtspiOk;
+ if (axExpected) {
for (const t of ["ax_desktop_ax_snapshot", "ax_desktop_ax_click", "ax_desktop_ax_set", "ax_desktop_ax_read"]) {
- assert.ok(names.includes(t), `${t} mounted with the macOS driver`);
+ assert.ok(names.includes(t), `${t} mounted with the structure-first driver`);
}
} else {
for (const t of ["ax_desktop_ax_snapshot", "ax_desktop_ax_click", "ax_desktop_ax_set", "ax_desktop_ax_read"]) {
- assert.ok(!names.includes(t), `${t} must not mount without the macOS AX layer`);
+ assert.ok(!names.includes(t), `${t} must not mount without a platform AX layer`);
}
}
@@ -91,6 +124,7 @@ test("mcp: handshake, tools/list, tools/call end-to-end", { skip: !hasBrowser },
const screenInfo = JSON.parse(screen.result.content[0].text);
assert.ok(screenInfo.w > 0 && screenInfo.h > 0);
if (process.platform === "darwin") assert.equal(screenInfo.coordinate_space, "macOS global display points");
+ if (process.platform === "linux") assert.match(screenInfo.coordinate_space, /X11 root window pixels/);
}
const page = `data:text/html,<button id="b" onclick="document.getElementById('o').textContent='done'">go</button><div id="o" style="min-height:1em"></div>`;
diff --git a/test/skip-allowlist.json b/test/skip-allowlist.json
index 2e9a7d8..b71d980 100644
--- a/test/skip-allowlist.json
+++ b/test/skip-allowlist.json
@@ -3,6 +3,10 @@
"skips": {
"desktop driver: screen_info reports a usable native coordinate space": "platform-or-gate: Windows live driver, or macOS with AXUSE_TEST_MACOS_DESKTOP=1",
"desktop driver: budgeted screenshot of a small region": "platform-windows: Windows-only live screenshot",
+ "linux desktop: driver drives a private Xvfb (input, capture, windows)": "platform-linux-xvfb: spawns a private Xvfb (sudo apt install xvfb); never touches the operator session",
+ "linux desktop: JPEG decodes in an independent decoder when PIL exists": "platform-linux-xvfb: spawns a private Xvfb; PIL verified when present",
+ "atspi: snapshot + read round-trip on a live app (read-only)": "platform-linux-atspi: needs the session's at-spi2 registry (DBUS_SESSION_BUS_ADDRESS); bare Xvfb/CI has none — read-only, never acts",
+ "atspi: action sandbox — snapshot, ax_set, ax_click on a private bus+Xvfb+GTK app": "platform-linux-gtk-sandbox: needs Xvfb + dbus-run-session + python3-gi Gtk3 (CI installs them); fully isolated private bus",
"desktop driver: macOS screenshot is opt-in and budgeted": "tcc-screen-recording: needs AXUSE_TEST_MACOS_SCREENSHOT=1 with Screen Recording granted (manual lane)",
"desktop driver: window discovery has stable title/pid records": "platform-or-gate: Windows live driver, or macOS with AXUSE_TEST_MACOS_DESKTOP=1",
"macOS permission probe is non-interactive and typed": "tcc-macos-desktop: needs AXUSE_TEST_MACOS_DESKTOP=1",
@@ -32,6 +36,11 @@
"plugin ABI child suites": "paid-tree: present only in the private checkout",
"macos darwin browser candidates": "platform-macos: darwin-only candidate table",
"darwin extension-capable candidates exclude branded Chrome": "platform-macos: darwin-only candidate table",
+ "macOS browser candidates distinguish branded Chrome from extension-capable builds": "platform-macos: darwin-only candidate table",
+ "macOS native-host installer writes an executable, isolated registration": "platform-macos: darwin-only native-host installer table",
+ "macOS native-host directories match each Chromium-family browser": "platform-macos: darwin-only native-host directory table",
+ "macOS uninstall preserves a shared launcher while another browser remains registered": "platform-macos: darwin-only uninstall-sharing test",
+ "macOS native-host launcher reaches an isolated hub": "platform-macos: darwin-only launcher-to-hub test",
"solver catalog tool schemas stay valid and plugin tool names never shadow core tools": "paid-tree: present only in the private checkout",
"every plugin CLI fails closed without a license": "paid-tree: present only in the private checkout",
"every plugin CLI executes: wired, licensed-gated, JSON-honest (no stack crashes)": "paid-tree: present only in the private checkout",