Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

nanocua: click web pages with events, because Chrome ignores presses

Rach Pradhan7be9eb21d476
diff --git a/apps/nanocua/README.md b/apps/nanocua/README.md
index 8d4b8c16..9cbfe6a2 100644
--- a/apps/nanocua/README.md
+++ b/apps/nanocua/README.md
@@ -50,8 +50,12 @@ so your pointer stays where it is and your keyboard stays yours. Screenshots
capture that app's window, even behind others, with the mapping from image pixels
back to screen points. A window on another Space or minimized is out of reach:
macOS shows neither its picture nor its elements, and the reply says so. This isn't a sandbox: it's your real Mac, your apps and
-your logins. And some apps ignore events they get in the background (Chromium's
-web content is the likely one); refs still work there.
+your logins. Some apps may ignore events they get in the background, so a reply
+says when a click isn't confirmed. Chrome works in the background: clicks, typing,
+keys, `set_value` and scrolling all reached a test page. A button in a web page gets
+a click event rather than an accessibility press, because Chrome reports a press
+as done without acting on it. The first click in a window that wasn't key can be
+spent on making it key.
## Tools
diff --git a/apps/nanocua/src/mac/backend.zig b/apps/nanocua/src/mac/backend.zig
index d3874d90..97e70a03 100644
--- a/apps/nanocua/src/mac/backend.zig
+++ b/apps/nanocua/src/mac/backend.zig
@@ -298,7 +298,7 @@ const Job = struct {
if (j.ref) |r| {
const el = j.element(r) orelse return;
const t = j.events(ax.pidOf(el)) orelse return;
- if (button == .left and count == 1 and ax.perform(el, "AXPress")) return w.print("Pressed [{d}].", .{r});
+ if (button == .left and count == 1 and !hit.inWebPage(el) and ax.perform(el, "AXPress")) return w.print("Pressed [{d}].", .{r});
const f = ax.frame(el) orelse return j.fail("[{d}] has no frame to click.", .{r});
const c = j.inWindow(t, f) orelse return j.fail("[{d}] is out of sight in its window: scroll to it first.", .{r});
const aimed = j.aim(t, c[0], c[1]) orelse return;
diff --git a/apps/nanocua/src/mac/hit.zig b/apps/nanocua/src/mac/hit.zig
index 319e82e6..2ae373ba 100644
--- a/apps/nanocua/src/mac/hit.zig
+++ b/apps/nanocua/src/mac/hit.zig
@@ -24,11 +24,28 @@ fn parent(el: cf.AXUIElementRef) ?cf.AXUIElementRef {
return @ptrCast(ax.copy(el, "AXParent"));
}
+/// Whether `el` is inside a web page. Chrome answers AXPress on a page's button
+/// with success and doesn't act on it, so a page's controls get a click event,
+/// whose reply doesn't claim more than it knows.
+pub fn inWebPage(el: cf.AXUIElementRef) bool {
+ var buf: [64]u8 = undefined;
+ var at_el: cf.AXUIElementRef = @ptrCast(cf.CFRetain(el));
+ defer cf.CFRelease(at_el);
+ for (0..64) |_| {
+ if (std.mem.eql(u8, role(&buf, at_el), "AXWebArea")) return true;
+ const up = parent(at_el) orelse return false;
+ cf.CFRelease(at_el);
+ at_el = up;
+ }
+ return false;
+}
+
/// Presses what's at x, y if it's a control, going up from the label or icon
/// inside it. Writes the control's role into `buf`; null if nothing was pressed.
pub fn press(buf: []u8, pid: i32, x: f64, y: f64) ?[]const u8 {
var el = at(pid, x, y) orelse return null;
defer cf.CFRelease(el);
+ if (inWebPage(el)) return null;
for (0..4) |_| {
const r = role(buf, el);
if (tree.pressable(r)) return if (ax.perform(el, "AXPress")) tree.bareRole(r) else null;
diff --git a/apps/nanocua/src/tools.zig b/apps/nanocua/src/tools.zig
index 39c08f5d..b731da91 100644
--- a/apps/nanocua/src/tools.zig
+++ b/apps/nanocua/src/tools.zig
@@ -61,7 +61,8 @@ pub const Tool = enum {
,
.click =>
\\Clicks in the app you act in, without moving the user's pointer: an element by ref, or x,y in screen
- \\points, inside that app's window. Controls are pressed through accessibility, which is exact. Anything
+ \\points, inside that app's window. Controls are pressed through accessibility, which is exact, except in
+ \\web pages, which get a click event because browsers can ignore a press while reporting success. Anything
\\else gets a click event that apps can ignore in the background, so the reply says it isn't confirmed:
\\check the result. In a window that isn't its app's key window, that first click may only make it key.
\\button: left (default) or right; count: 2 for a double click.
diff --git a/docs/adr/0314-nanocua-is-grafts-computer-use.md b/docs/adr/0314-nanocua-is-grafts-computer-use.md
index bc3be6c6..cf6958cb 100644
--- a/docs/adr/0314-nanocua-is-grafts-computer-use.md
+++ b/docs/adr/0314-nanocua-is-grafts-computer-use.md
@@ -79,8 +79,10 @@ macOS's own permissions stay in System Settings; nothing here can flip them.
answers that each tool is unsupported.
- A release built without the signing secrets has no app, so that install has no
computer use. An ad hoc app would lose its permissions on every update.
-- Background input is delivered per process. Apps may ignore it, Chromium's web
- content most likely; agents fall back to refs or ask for the whole screen.
+- Background input is delivered per process, and an app may ignore it. Replies say
+ when a click isn't confirmed. Chrome acted on it in testing, except that a press
+ through accessibility on a page's button reports success and does nothing, so
+ web pages get click events instead.
- Agents share one app, calls one at a time, but each connection has its own refs
and target. Harness's graff chats and a terminal graff use the same app.
- Beta releases are not signed, so they carry no app.