graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
nanocua: click web pages with events, because Chrome ignores presses
Rach Pradhan7be9eb21d476
diff --git a/apps/nanocua/README.md b/apps/nanocua/README.mdindex 8d4b8c16..9cbfe6a2 100644--- a/apps/nanocua/README.md+++ b/apps/nanocua/README.md@@ -50,8 +50,12 @@ so your pointer stays where it is and your keyboard stays yours. Screenshotscapture that app's window, even behind others, with the mapping from image pixelsback to screen points. A window on another Space or minimized is out of reach:macOS shows neither its picture nor its elements, and the reply says so. This isn't a sandbox: it's your real Mac, your apps and-your logins. And some apps ignore events they get in the background (Chromium's-web content is the likely one); refs still work there.+your logins. Some apps may ignore events they get in the background, so a reply+says when a click isn't confirmed. Chrome works in the background: clicks, typing,+keys, `set_value` and scrolling all reached a test page. A button in a web page gets+a click event rather than an accessibility press, because Chrome reports a press+as done without acting on it. The first click in a window that wasn't key can be+spent on making it key.## Toolsdiff --git a/apps/nanocua/src/mac/backend.zig b/apps/nanocua/src/mac/backend.zigindex d3874d90..97e70a03 100644--- a/apps/nanocua/src/mac/backend.zig+++ b/apps/nanocua/src/mac/backend.zig@@ -298,7 +298,7 @@ const Job = struct {if (j.ref) |r| {const el = j.element(r) orelse return;const t = j.events(ax.pidOf(el)) orelse return;- if (button == .left and count == 1 and ax.perform(el, "AXPress")) return w.print("Pressed [{d}].", .{r});+ if (button == .left and count == 1 and !hit.inWebPage(el) and ax.perform(el, "AXPress")) return w.print("Pressed [{d}].", .{r});const f = ax.frame(el) orelse return j.fail("[{d}] has no frame to click.", .{r});const c = j.inWindow(t, f) orelse return j.fail("[{d}] is out of sight in its window: scroll to it first.", .{r});const aimed = j.aim(t, c[0], c[1]) orelse return;diff --git a/apps/nanocua/src/mac/hit.zig b/apps/nanocua/src/mac/hit.zigindex 319e82e6..2ae373ba 100644--- a/apps/nanocua/src/mac/hit.zig+++ b/apps/nanocua/src/mac/hit.zig@@ -24,11 +24,28 @@ fn parent(el: cf.AXUIElementRef) ?cf.AXUIElementRef {return @ptrCast(ax.copy(el, "AXParent"));}+/// Whether `el` is inside a web page. Chrome answers AXPress on a page's button+/// with success and doesn't act on it, so a page's controls get a click event,+/// whose reply doesn't claim more than it knows.+pub fn inWebPage(el: cf.AXUIElementRef) bool {+ var buf: [64]u8 = undefined;+ var at_el: cf.AXUIElementRef = @ptrCast(cf.CFRetain(el));+ defer cf.CFRelease(at_el);+ for (0..64) |_| {+ if (std.mem.eql(u8, role(&buf, at_el), "AXWebArea")) return true;+ const up = parent(at_el) orelse return false;+ cf.CFRelease(at_el);+ at_el = up;+ }+ return false;+}+/// Presses what's at x, y if it's a control, going up from the label or icon/// inside it. Writes the control's role into `buf`; null if nothing was pressed.pub fn press(buf: []u8, pid: i32, x: f64, y: f64) ?[]const u8 {var el = at(pid, x, y) orelse return null;defer cf.CFRelease(el);+ if (inWebPage(el)) return null;for (0..4) |_| {const r = role(buf, el);if (tree.pressable(r)) return if (ax.perform(el, "AXPress")) tree.bareRole(r) else null;diff --git a/apps/nanocua/src/tools.zig b/apps/nanocua/src/tools.zigindex 39c08f5d..b731da91 100644--- a/apps/nanocua/src/tools.zig+++ b/apps/nanocua/src/tools.zig@@ -61,7 +61,8 @@ pub const Tool = enum {,.click =>\\Clicks in the app you act in, without moving the user's pointer: an element by ref, or x,y in screen- \\points, inside that app's window. Controls are pressed through accessibility, which is exact. Anything+ \\points, inside that app's window. Controls are pressed through accessibility, which is exact, except in+ \\web pages, which get a click event because browsers can ignore a press while reporting success. Anything\\else gets a click event that apps can ignore in the background, so the reply says it isn't confirmed:\\check the result. In a window that isn't its app's key window, that first click may only make it key.\\button: left (default) or right; count: 2 for a double click.diff --git a/docs/adr/0314-nanocua-is-grafts-computer-use.md b/docs/adr/0314-nanocua-is-grafts-computer-use.mdindex bc3be6c6..cf6958cb 100644--- a/docs/adr/0314-nanocua-is-grafts-computer-use.md+++ b/docs/adr/0314-nanocua-is-grafts-computer-use.md@@ -79,8 +79,10 @@ macOS's own permissions stay in System Settings; nothing here can flip them.answers that each tool is unsupported.- A release built without the signing secrets has no app, so that install has nocomputer use. An ad hoc app would lose its permissions on every update.-- Background input is delivered per process. Apps may ignore it, Chromium's web- content most likely; agents fall back to refs or ask for the whole screen.+- Background input is delivered per process, and an app may ignore it. Replies say+ when a click isn't confirmed. Chrome acted on it in testing, except that a press+ through accessibility on a page's button reports success and does nothing, so+ web pages get click events instead.- Agents share one app, calls one at a time, but each connection has its own refsand target. Harness's graff chats and a terminal graff use the same app.- Beta releases are not signed, so they carry no app.