graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
Let graff and Harness show and stop computer use
Rach Pradhanf34f61f4fbd0
diff --git a/apps/nanocua/README.md b/apps/nanocua/README.mdindex 03a8e792..df6c8730 100644--- a/apps/nanocua/README.md+++ b/apps/nanocua/README.md@@ -19,6 +19,10 @@ agent ──stdio──▶ nanocua mcp ──unix socket──▶ nanocua (the aaccessibility, input, screenshots```+Everything goes to the one app, so a terminal `graff`, Harness's graff chats and+any other agent can use it at the same time. Calls run one at a time, and each+agent keeps its own refs and target app.+The app does all the work, and the agent's `nanocua mcp` only carries lines.That matters on macOS: permissions (Accessibility, Screen Recording) belongto the app that asks, so you grant them to nanocua once, not to each terminal@@ -69,10 +73,9 @@ brings the app forward.## What you control-- **Stop** (or **⌃⌥⌘.** from anywhere) refuses every call, even looking,- including calls already waiting, and stops text being typed, until you press- Resume. An agent may be moving the pointer, which is why the keyboard shortcut- exists.+- **Stop** refuses every call, even looking, including calls already waiting, and+ stops text being typed, until you press Resume. An agent may be moving the+ pointer, so it's also on a key, **⌃⌥⌘.**, and outside the panel (below).- **Allow input** is off at launch: agents can look but not click, type oropen apps until you turn it on.- **Allow whole screen** is off at launch too: agents stay inside one app.@@ -84,6 +87,34 @@ brings the app forward.Calls run one at a time on a queue of their own, so a hung app neverfreezes the panel (accessibility calls give up after two seconds).+## From graff and Harness++The switches also show in `graff computer`, `/computer` and Harness's MCP servers+settings page. All of them run the app's `nanocua ctl`:++```sh+nanocua ctl status [--json] what's on, what macOS has granted+nanocua ctl stop refuse every call, at once+nanocua ctl off input turn a switch off, at once+nanocua ctl allow input ask; you answer in nanocua's window+nanocua ctl grant screen open macOS's prompt+nanocua ctl open bring the window forward+```++Anything can stop nanocua or turn things off. **Nothing outside the window can+turn things on.** `allow` only puts a question in nanocua's window, and it takes a+click there, from your own mouse or trackpad. Agents have shells and could run+this command too; if it turned input on, the switch would protect you from nothing.+Allow arms after 0.7s, so a click meant for something else can't answer it. A+question goes away after a minute, and a Deny refuses new ones for 15 seconds.++The same check covers the panel's own switches and Resume, on the way on only.+Switch Control and Voice Control post their own clicks, so they can turn things+off here but not on. Release builds refuse nanoui's `--nanoui-*` flags, which+would otherwise drive this window from a command line. None of this stops a+determined program running as you, which could tamper with the app itself: it+closes the easy route. macOS's own permissions stay in System Settings.+## Adding it to an agentWith `graff` there's nothing to add: on macOS its installer puts nanocua in@@ -116,7 +147,8 @@ sh macos/package.sh aarch64-macos /tmp # /tmp/nanocua.appNothing in the tests moves the pointer, types or captures the screen: the policy,where calls go, MCP sessions, keys, the tree's text, the socket (over a socket-pair) and the panel are tested; the macOS backend is compiled, not run.+pair), the switches from outside and the panel are tested; the macOS backend is+compiled, not run.macOS keeps permissions across updates only for an app signed by the sameidentity. Set `CODESIGN_IDENTITY` for `package.sh`; an ad hoc build is asked againdiff --git a/apps/nanocua/src/app.zig b/apps/nanocua/src/app.zigindex 1d4f1cdb..1bef9d94 100644--- a/apps/nanocua/src/app.zig+++ b/apps/nanocua/src/app.zig@@ -1,11 +1,14 @@//! nanocua's state: the hub agents talk to, the socket it listens on, the-//! backend that runs the tools, the kill switch and its hot key, and what-//! the panel shows (permissions, the command for agents, the log).+//! backend that runs the tools, the kill switch and its hot key, the switches+//! as graff and Harness see them (control.zig), and what the panel shows+//! (permissions, the command for agents, the log).const std = @import("std");const builtin = @import("builtin");const nanoui = @import("nanoui");const mcp = @import("mcp.zig");const server_mod = @import("server.zig");+const control = @import("control.zig");+const tools = @import("tools.zig");pub const mac = builtin.os.tag == .macos;const backend_mod = if (mac) @import("mac/backend.zig") else @import("backend_none.zig");@@ -37,6 +40,21 @@ pub const App = struct {perms: Perms = .{},read_perms: ?*const fn () Perms = null,ask: ?*const fn (which: Ask) void = null,+ /// The user asked for Screen Recording: the panel says what's left to do.+ asked_screen: bool = false,+ /// The question a program asked, waiting in the panel.+ gate: control.Gate = .{},+ /// Whether the click being handled came from a person (none in tests:+ /// every click does). Turning something on needs one.+ real_pointer: ?*const fn () bool = null,+ /// Running under `--nanoui-hidden`: nothing comes forward, whoever asks.+ hidden: bool = false,+ /// Keeps the window above others while a question waits, or stops.+ raise: ?*const fn (window: *nanoui.Window, on: bool) void = null,+ /// A click that wasn't real was just ignored: the panel says so until then.+ ignored_until: f64 = 0,+ /// macOS's prompts open no more often than this.+ grant_after: f64 = 0,/// "Copied" shows for a while after copying the command.copied_until: f64 = 0,/// Light or dark, fixed (tests); else as the system is.@@ -53,6 +71,7 @@ pub const App = struct {};a.hub.sink = a.server.sink();a.hub.changed = .{ .ptr = a, .func = changed };+ a.hub.control = .{ .ptr = a, .func = onControl };}pub fn deinit(a: *App) void {@@ -110,11 +129,168 @@ pub const App = struct {a.redraw();}+ /// Turning something on from the panel: only a person's click counts.+ /// Turning it off always does.+ pub fn fromPanel(a: *App, want: control.Want, on: bool) void {+ if (on and !a.clickIsReal()) return;+ switch (want) {+ .input => a.setInput(on),+ .whole_screen => a.setWholeScreen(on),+ .@"resume" => a.setStopped(!on),+ }+ }++ fn clickIsReal(a: *App) bool {+ const real = if (a.real_pointer) |f| f() else true;+ if (!real) a.ignored_until = a.hub.clock() + 5;+ return real;+ }++ pub fn status(a: *App, outcome: ?[]const u8) control.Status {+ a.refreshPerms();+ const p = a.hub.policy;+ return .{+ .stopped = p.stopped,+ .input = p.allow_input,+ .wholeScreen = p.allow_whole_screen,+ .accessibility = a.perms.accessibility,+ .screenRecording = a.perms.screen,+ .agents = a.server.clients(),+ .calls = a.hub.log.calls,+ .stopKey = a.hotkey != null,+ .pending = if (a.gate.request) |r| @tagName(r.want) else null,+ .outcome = outcome,+ };+ }++ fn isOn(a: *const App, want: control.Want) bool {+ const p = a.hub.policy;+ return switch (want) {+ .input => p.allow_input,+ .whole_screen => p.allow_whole_screen,+ .@"resume" => !p.stopped,+ };+ }++ fn reply(a: *App, conn: u64, id: []const u8, outcome: ?[]const u8) void {+ const line = control.statusJson(a.gpa, a.status(outcome)) catch return;+ defer a.gpa.free(line);+ a.hub.sendResult(conn, id, line);+ }++ fn onControl(ptr: *anyopaque, conn: u64, id: []const u8, method: []const u8, params: ?std.json.Value) void {+ const a: *App = @ptrCast(@alignCast(ptr));+ a.server.markControl(conn);+ const args = tools.Args.of(params);+ const eql = std.mem.eql;+ const name = method["nanocua/".len..];+ if (eql(u8, name, "status")) {+ a.settleRequest();+ return a.reply(conn, id, null);+ }+ if (eql(u8, name, "stop")) {+ a.setStopped(true);+ return a.reply(conn, id, null);+ }+ if (eql(u8, name, "off")) {+ const what = control.Want.parse(args.string("what") orelse "") orelse .@"resume";+ switch (what) {+ .input => a.setInput(false),+ .whole_screen => a.setWholeScreen(false),+ // Off means off: nothing here can turn anything on.+ .@"resume" => return a.hub.sendError(conn, id, -32602, "Say input or whole_screen"),+ }+ return a.reply(conn, id, null);+ }+ if (eql(u8, name, "ask")) return a.onAsk(conn, id, args);+ if (eql(u8, name, "grant")) return a.onGrant(conn, id, args);+ if (eql(u8, name, "open")) {+ a.bringForward(false);+ return a.reply(conn, id, null);+ }+ a.hub.sendError(conn, id, -32601, "Method not found");+ }++ fn onAsk(a: *App, conn: u64, id: []const u8, args: tools.Args) void {+ const want = control.Want.parse(args.string("what") orelse "") orelse+ return a.hub.sendError(conn, id, -32602, "Say input, whole_screen or resume");+ if (a.isOn(want)) return a.reply(conn, id, "already_on");+ a.settleRequest();+ switch (a.gate.ask(want, args.string("from") orelse "", conn, id, a.hub.clock())) {+ .busy => a.reply(conn, id, "busy"),+ .quiet => a.reply(conn, id, "quiet"),+ // The answer comes when the user clicks (answer).+ .waiting => {+ a.bringForward(true);+ a.redraw();+ },+ }+ }++ fn onGrant(a: *App, conn: u64, id: []const u8, args: tools.Args) void {+ const which = args.string("which") orelse "";+ const ask_for: Ask = if (std.mem.eql(u8, which, "accessibility"))+ .accessibility+ else if (std.mem.eql(u8, which, "screen"))+ .screen+ else+ return a.hub.sendError(conn, id, -32602, "Say accessibility or screen");+ a.refreshPerms();+ const has = if (ask_for == .accessibility) a.perms.accessibility else a.perms.screen;+ if (has) return a.reply(conn, id, "already_allowed");+ const now = a.hub.clock();+ if (now < a.grant_after) return a.reply(conn, id, "quiet");+ a.grant_after = now + 5;+ a.request(ask_for);+ a.bringForward(false);+ a.reply(conn, id, "opened");+ }++ fn bringForward(a: *App, stay_above: bool) void {+ if (a.hidden) return;+ const w = a.window orelse return;+ if (stay_above) if (a.raise) |f| f(w, true);+ w.focus();+ }++ /// The panel's Allow or Deny. Allow needs a person's click, and the question+ /// to have been on screen a moment.+ pub fn answer(a: *App, allow: bool) void {+ const now = a.hub.clock();+ if (allow) {+ if (!a.gate.armed(now) or !a.clickIsReal()) return;+ const r = a.gate.take() orelse return;+ switch (r.want) {+ .input => a.setInput(true),+ .whole_screen => a.setWholeScreen(true),+ .@"resume" => a.setStopped(false),+ }+ a.finishRequest(r, "allowed");+ } else {+ const r = a.gate.deny(now) orelse return;+ a.finishRequest(r, "denied");+ }+ }++ /// A question nobody answered goes away, and whoever asked is told.+ pub fn settleRequest(a: *App) void {+ if (!a.gate.expired(a.hub.clock())) return;+ const r = a.gate.take() orelse return;+ a.finishRequest(r, "expired");+ }++ fn finishRequest(a: *App, r: control.Request, outcome: []const u8) void {+ if (!a.hidden) if (a.window) |w| if (a.raise) |f| f(w, false);+ a.reply(r.conn, r.id(), outcome);+ a.redraw();+ }+pub fn refreshPerms(a: *App) void {if (a.read_perms) |f| a.perms = f();}pub fn request(a: *App, which: Ask) void {+ if (which == .screen) a.asked_screen = true;if (a.ask) |f| f(which);}@@ -144,6 +320,8 @@ pub const Platform = struct {a.whole_flag = &p.impl.whole_screen;a.read_perms = readPerms;a.ask = ask;+ a.real_pointer = @import("mac/pointer.zig").real;+ a.raise = backend_mod.raise;}}diff --git a/apps/nanocua/src/control.zig b/apps/nanocua/src/control.zignew file mode 100644index 00000000..fbe90ea4--- /dev/null+++ b/apps/nanocua/src/control.zig@@ -0,0 +1,192 @@+//! The switches from outside the panel, so graff and Harness can show them.+//! Anything on the user's socket may turn something off or read the state.+//! Nothing but a real click in nanocua's own window turns something on: a+//! program only asks, and the question waits in the panel. Agents have shells,+//! so a command that turned input on would let them turn on their own input.+const std = @import("std");++pub const Want = enum {+ input,+ whole_screen,+ @"resume",++ pub fn parse(s: []const u8) ?Want {+ return std.meta.stringToEnum(Want, s);+ }++ pub fn question(w: Want) []const u8 {+ return switch (w) {+ .input => "Turn on input? Agents could click, type, press keys and open apps, one app at a time.",+ .whole_screen => "Turn on the whole screen? Agents could use your pointer and keyboard, and take pictures of everything.",+ .@"resume" => "Resume? Agents' calls run again, as far as the switches below allow.",+ };+ }+};++/// Allow does nothing this long after the question appears, so a click meant+/// for whatever was under the pointer can't answer it.+pub const armed_after: f64 = 0.7;+/// An unanswered question goes away: it can't be answered by accident later.+pub const lifetime: f64 = 60;+/// After a Deny, new questions are refused this long, so nothing can keep+/// raising the window until the user gives in.+pub const quiet_after_deny: f64 = 15;++/// One question, and who is waiting for its answer.+pub const Request = struct {+ want: Want,+ at: f64,+ conn: u64,+ from_buf: [32]u8 = undefined,+ from_len: u8 = 0,+ id_buf: [64]u8 = undefined,+ id_len: u8 = 0,++ /// Who says it's asking. Programs name themselves: this isn't proof.+ pub fn from(r: *const Request) []const u8 {+ return r.from_buf[0..r.from_len];+ }++ pub fn id(r: *const Request) []const u8 {+ return r.id_buf[0..r.id_len];+ }+};++pub const Gate = struct {+ request: ?Request = null,+ quiet_until: f64 = 0,++ pub const Asked = enum { waiting, busy, quiet };++ pub fn ask(g: *Gate, want: Want, from: []const u8, conn: u64, id: []const u8, now: f64) Asked {+ if (g.request != null) return if (g.expired(now)) .busy else .busy;+ if (now < g.quiet_until) return .quiet;+ if (id.len > 64) return .busy;+ var r: Request = .{ .want = want, .at = now, .conn = conn };+ @memcpy(r.id_buf[0..id.len], id);+ r.id_len = @intCast(id.len);+ // Shown to the user: plain characters only, and never empty.+ for (from[0..@min(from.len, r.from_buf.len)]) |ch| {+ if (ch < 0x20 or ch > 0x7e) continue;+ r.from_buf[r.from_len] = ch;+ r.from_len += 1;+ }+ if (r.from_len == 0) {+ const unknown = "A program";+ @memcpy(r.from_buf[0..unknown.len], unknown);+ r.from_len = unknown.len;+ }+ g.request = r;+ return .waiting;+ }++ pub fn armed(g: *const Gate, now: f64) bool {+ const r = g.request orelse return false;+ return now - r.at >= armed_after;+ }++ pub fn expired(g: *const Gate, now: f64) bool {+ const r = g.request orelse return false;+ return now - r.at >= lifetime;+ }++ /// The question, answered with Allow or gone away.+ pub fn take(g: *Gate) ?Request {+ const r = g.request;+ g.request = null;+ return r;+ }++ pub fn deny(g: *Gate, now: f64) ?Request {+ g.quiet_until = now + quiet_after_deny;+ return g.take();+ }+};++/// What graff and Harness read: the reply to every control call. Its field+/// names are the wire format.+pub const Status = struct {+ installed: bool = true,+ running: bool = true,+ stopped: bool = false,+ input: bool = false,+ wholeScreen: bool = false,+ accessibility: bool = false,+ screenRecording: bool = false,+ agents: usize = 0,+ calls: u64 = 0,+ /// Whether ⌃⌥⌘. was registered.+ stopKey: bool = false,+ /// What's waiting for an answer in the panel.+ pending: ?[]const u8 = null,+ /// How this call went: allowed, denied, expired, busy, quiet, already_on,+ /// opened, already_allowed, not_running, unreachable.+ outcome: ?[]const u8 = null,+};++pub fn statusJson(a: std.mem.Allocator, s: Status) ![]u8 {+ return std.json.Stringify.valueAlloc(a, s, .{});+}++/// nanoui's `--nanoui-*` flags drive a window from the command line, which is+/// for tests. In a release they'd be a way round the panel: `nanocua+/// --nanoui-play=…` clicks its own switches.+pub fn devFlagRefused(mode: std.lang.Optimize, arg: []const u8) bool {+ return mode != .debug and std.mem.startsWith(u8, arg, "--nanoui-");+}++test "a question: asked once, armed after a moment, gone after a while" {+ var g: Gate = .{};+ try std.testing.expect(!g.armed(10));+ try std.testing.expectEqual(Gate.Asked.waiting, g.ask(.input, "graff", 7, "\"a\"", 10));+ try std.testing.expectEqualStrings("graff", g.request.?.from());+ try std.testing.expectEqualStrings("\"a\"", g.request.?.id());+ try std.testing.expectEqual(@as(u64, 7), g.request.?.conn);+ try std.testing.expect(!g.armed(10.5));+ try std.testing.expect(g.armed(10.71));+ try std.testing.expectEqual(Gate.Asked.busy, g.ask(.whole_screen, "other", 8, "1", 10.8));+ try std.testing.expectEqual(Want.input, g.request.?.want);+ try std.testing.expect(!g.expired(69));+ try std.testing.expect(g.expired(70));+ try std.testing.expectEqual(Want.input, g.take().?.want);+ try std.testing.expect(g.take() == null);+}++test "a deny keeps new questions away for a while" {+ var g: Gate = .{};+ _ = g.ask(.input, "x", 1, "1", 0);+ try std.testing.expect(g.deny(1) != null);+ try std.testing.expectEqual(Gate.Asked.quiet, g.ask(.input, "x", 1, "2", 15));+ try std.testing.expectEqual(Gate.Asked.waiting, g.ask(.input, "x", 1, "2", 16));+}++test "who's asking is cut to plain characters" {+ var g: Gate = .{};+ _ = g.ask(.input, "gr\x1baff\xc3\xa9", 1, "1", 0);+ try std.testing.expectEqualStrings("graff", g.request.?.from());+ _ = g.take();+ _ = g.ask(.input, "\x01", 1, "1", 0);+ try std.testing.expectEqualStrings("A program", g.request.?.from());+ _ = g.take();+ const long: [80]u8 = @splat('x');+ _ = g.ask(.input, &long, 1, "1", 0);+ try std.testing.expectEqual(@as(usize, 32), g.request.?.from().len);+}++test "the status is the wire format" {+ const gpa = std.testing.allocator;+ const out = try statusJson(gpa, .{ .input = true, .pending = "resume" });+ defer gpa.free(out);+ const parsed = try std.json.parseFromSlice(Status, gpa, out, .{});+ defer parsed.deinit();+ try std.testing.expect(parsed.value.input and !parsed.value.wholeScreen);+ try std.testing.expectEqualStrings("resume", parsed.value.pending.?);+ try std.testing.expect(std.mem.indexOf(u8, out, "\"wholeScreen\":false") != null);+}++test "dev flags only in debug builds" {+ try std.testing.expect(!devFlagRefused(.debug, "--nanoui-play=click:Stop"));+ try std.testing.expect(devFlagRefused(.fast, "--nanoui-play=click:Stop"));+ try std.testing.expect(devFlagRefused(.safe, "--nanoui-script=x"));+ try std.testing.expect(!devFlagRefused(.fast, "mcp"));+}diff --git a/apps/nanocua/src/ctl.zig b/apps/nanocua/src/ctl.zignew file mode 100644index 00000000..bed0b092--- /dev/null+++ b/apps/nanocua/src/ctl.zig@@ -0,0 +1,196 @@+//! `nanocua ctl`: the switches from a command line, for `graff computer` and+//! Harness. It reads the state and turns things off at once. `allow` only asks:+//! the question appears in nanocua's window and this waits for the click.+const std = @import("std");+const control = @import("control.zig");+const proxy = @import("proxy.zig");+const server = @import("server.zig");+const sockpath = @import("sockpath.zig");++extern "c" fn close(fd: c_int) c_int;++pub const usage =+ \\usage: nanocua ctl <command> [--json] [--from=NAME]+ \\ status what's on, and what macOS has granted+ \\ stop refuse every call, at once+ \\ off <input|whole_screen> turn a switch off, at once+ \\ allow <input|whole_screen|resume> ask; you answer in nanocua's window+ \\ grant <accessibility|screen> open macOS's prompt for a permission+ \\ open bring nanocua's window forward+ \\+;++/// How long `allow` waits for the click: a little past the question's own end.+const answer_ms: i32 = @intFromFloat((control.lifetime + 5) * 1000);+const reply_ms: i32 = 5_000;++pub const Command = struct {+ method: []const u8,+ /// Its `params`, as JSON.+ params: []const u8 = "{}",+ /// Starts the app if it isn't running; else there's nothing to do.+ opens_app: bool = false,+ wait_ms: i32 = reply_ms,+};++/// What a command line asks for; null if it makes no sense.+pub fn parse(a: std.mem.Allocator, args: []const []const u8, from: []const u8) !?Command {+ if (args.len == 0) return null;+ const verb = args[0];+ const object = if (args.len > 1) args[1] else "";+ if (std.mem.eql(u8, verb, "status") and args.len == 1) return .{ .method = "nanocua/status" };+ if (std.mem.eql(u8, verb, "stop") and args.len == 1) return .{ .method = "nanocua/stop" };+ if (std.mem.eql(u8, verb, "open") and args.len == 1) return .{ .method = "nanocua/open", .opens_app = true };+ if (args.len != 2) return null;+ if (std.mem.eql(u8, verb, "off")) {+ const w = control.Want.parse(object) orelse return null;+ if (w == .@"resume") return null;+ return .{ .method = "nanocua/off", .params = try paramsJson(a, .{ .what = object }) };+ }+ if (std.mem.eql(u8, verb, "allow")) {+ _ = control.Want.parse(object) orelse return null;+ return .{ .method = "nanocua/ask", .params = try paramsJson(a, .{ .what = object, .from = from }), .opens_app = true, .wait_ms = answer_ms };+ }+ if (std.mem.eql(u8, verb, "grant")) {+ if (!std.mem.eql(u8, object, "accessibility") and !std.mem.eql(u8, object, "screen")) return null;+ return .{ .method = "nanocua/grant", .params = try paramsJson(a, .{ .which = object }), .opens_app = true };+ }+ return null;+}++fn paramsJson(a: std.mem.Allocator, v: anytype) ![]const u8 {+ return std.json.Stringify.valueAlloc(a, v, .{});+}++/// The status as a person reads it.+pub fn describe(a: std.mem.Allocator, s: control.Status) ![]const u8 {+ var aw: std.Io.Writer.Allocating = .init(a);+ const w = &aw.writer;+ if (s.outcome) |o| {+ const line: ?[]const u8 = if (std.mem.eql(u8, o, "allowed"))+ "Allowed."+ else if (std.mem.eql(u8, o, "denied"))+ "Denied in nanocua's window."+ else if (std.mem.eql(u8, o, "expired"))+ "Nobody answered in nanocua's window."+ else if (std.mem.eql(u8, o, "busy"))+ "nanocua's window is already waiting for an answer."+ else if (std.mem.eql(u8, o, "quiet"))+ "Just denied: try again in a few seconds."+ else if (std.mem.eql(u8, o, "unreachable"))+ "Couldn't reach nanocua: open nanocua.app."+ else if (std.mem.eql(u8, o, "opened"))+ "macOS's prompt or its settings should be open: turn nanocua on there."+ else+ null;+ if (line) |l| try w.print("{s}\n", .{l});+ }+ if (!s.running) {+ try w.writeAll("nanocua isn't running.\n");+ return aw.written();+ }+ const state = if (s.stopped) "stopped" else if (s.input) "acting" else "looking only";+ try w.print("nanocua: {s}, whole screen {s}, {d} agent{s}, {d} call{s}\n", .{ state, if (s.wholeScreen) "on" else "off", s.agents, if (s.agents == 1) "" else "s", s.calls, if (s.calls == 1) "" else "s" });+ try w.print("macOS: Accessibility {s}, Screen Recording {s}\n", .{ if (s.accessibility) "allowed" else "not allowed", if (s.screenRecording) "allowed" else "not allowed" });+ if (s.pending) |p| try w.print("Waiting in nanocua's window: {s}\n", .{p});+ return aw.written();+}++fn show(a: std.mem.Allocator, s: control.Status, json: bool) !void {+ const text = if (json) blk: {+ const line = try control.statusJson(a, s);+ break :blk try std.fmt.allocPrint(a, "{s}\n", .{line});+ } else try describe(a, s);+ _ = server.writeAll(1, text);+}++pub fn run(init: std.process.Init, args: []const [:0]const u8) !u8 {+ const a = init.arena.allocator();+ var json = false;+ var from: []const u8 = "nanocua ctl";+ var words: std.ArrayList([]const u8) = .empty;+ for (args) |arg| {+ if (std.mem.eql(u8, arg, "--json")) {+ json = true;+ } else if (std.mem.startsWith(u8, arg, "--from=")) {+ from = arg["--from=".len..];+ } else try words.append(a, arg);+ }+ const cmd = try parse(a, words.items, from) orelse {+ _ = server.writeAll(2, usage);+ return 2;+ };+ var buf: [512]u8 = undefined;+ const path = try sockpath.path(&buf, init.environ_map, sockpath.realHome());+ const fd = (if (cmd.opens_app) proxy.connectOrOpen(path) else proxy.connect(path)) orelse {+ try show(a, .{ .running = false, .outcome = if (cmd.opens_app) "unreachable" else "not_running" }, json);+ return 0;+ };+ defer _ = close(fd);+ const line = try std.fmt.allocPrint(a, "{{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"{s}\",\"params\":{s}}}\n", .{ cmd.method, cmd.params });+ if (!server.writeAll(fd, line)) return unreachableApp(a, json);+ const reply = readLine(a, fd, cmd.wait_ms) catch return unreachableApp(a, json);+ const parsed = std.json.parseFromSliceLeaky(std.json.Value, a, reply, .{}) catch return unreachableApp(a, json);+ const result = (if (parsed == .object) parsed.object.get("result") else null) orelse return unreachableApp(a, json);+ if (json) {+ const text = try std.json.Stringify.valueAlloc(a, result, .{});+ _ = server.writeAll(1, text);+ _ = server.writeAll(1, "\n");+ return 0;+ }+ const s = try std.json.parseFromValueLeaky(control.Status, a, result, .{ .ignore_unknown_fields = true });+ try show(a, s, false);+ return 0;+}++fn unreachableApp(a: std.mem.Allocator, json: bool) !u8 {+ try show(a, .{ .running = false, .outcome = "unreachable" }, json);+ return 0;+}++fn readLine(a: std.mem.Allocator, fd: c_int, wait_ms: i32) ![]const u8 {+ var got: std.ArrayList(u8) = .empty;+ var chunk: [16 * 1024]u8 = undefined;+ while (true) {+ var fds = [_]std.posix.pollfd{.{ .fd = fd, .events = std.posix.POLL.IN, .revents = 0 }};+ if (try std.posix.poll(&fds, wait_ms) == 0) return error.TimedOut;+ const n = std.c.read(fd, &chunk, chunk.len);+ if (n <= 0) return error.Closed;+ try got.appendSlice(a, chunk[0..@intCast(n)]);+ if (std.mem.indexOfScalar(u8, got.items, '\n')) |nl| return got.items[0..nl];+ }+}++test "command lines" {+ var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);+ defer arena.deinit();+ const a = arena.allocator();+ const p = struct {+ fn f(al: std.mem.Allocator, args: []const []const u8) !?Command {+ return parse(al, args, "graff");+ }+ }.f;+ try std.testing.expectEqualStrings("nanocua/status", (try p(a, &.{"status"})).?.method);+ try std.testing.expect(!(try p(a, &.{"stop"})).?.opens_app);+ const allow = (try p(a, &.{ "allow", "whole_screen" })).?;+ try std.testing.expect(allow.opens_app and allow.wait_ms > 60_000);+ try std.testing.expectEqualStrings("{\"what\":\"whole_screen\",\"from\":\"graff\"}", allow.params);+ try std.testing.expectEqualStrings("{\"which\":\"screen\"}", (try p(a, &.{ "grant", "screen" })).?.params);+ try std.testing.expect((try p(a, &.{ "off", "resume" })) == null);+ try std.testing.expect((try p(a, &.{ "allow", "everything" })) == null);+ try std.testing.expect((try p(a, &.{ "grant", "camera" })) == null);+ try std.testing.expect((try p(a, &.{ "stop", "now" })) == null);+ try std.testing.expect((try p(a, &.{})) == null);+}++test "what a person reads" {+ var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);+ defer arena.deinit();+ const a = arena.allocator();+ try std.testing.expectEqualStrings("nanocua isn't running.\n", try describe(a, .{ .running = false, .outcome = "not_running" }));+ const on = try describe(a, .{ .input = true, .agents = 1, .calls = 2, .accessibility = true, .outcome = "allowed", .pending = "resume" });+ try std.testing.expect(std.mem.startsWith(u8, on, "Allowed.\nnanocua: acting, whole screen off, 1 agent, 2 calls\n"));+ try std.testing.expect(std.mem.indexOf(u8, on, "Accessibility allowed, Screen Recording not allowed") != null);+ try std.testing.expect(std.mem.endsWith(u8, on, "Waiting in nanocua's window: resume\n"));+ try std.testing.expect(std.mem.indexOf(u8, try describe(a, .{ .stopped = true }), "nanocua: stopped") != null);+}diff --git a/apps/nanocua/src/mac/backend.zig b/apps/nanocua/src/mac/backend.zigindex 57d96564..c18092d3 100644--- a/apps/nanocua/src/mac/backend.zig+++ b/apps/nanocua/src/mac/backend.zig@@ -16,6 +16,7 @@ const input = @import("input.zig");const capture = @import("capture.zig");const apps = @import("apps.zig");const windows = @import("windows.zig");+const hit = @import("hit.zig");const target = @import("../target.zig");extern "c" fn dispatch_queue_create(label: [*:0]const u8, attr: ?*anyopaque) objc.dispatch_queue_t;@@ -23,26 +24,62 @@ extern "c" fn dispatch_async_f(queue: objc.dispatch_queue_t, ctx: ?*anyopaque, wextern "c" fn getpid() c_int;extern "c" fn usleep(us: c_uint) c_int;+/// What one agent is working on. Agents connect one socket each: refs from+/// one's tree mean nothing to another, and neither moves the other's target.+pub const Session = struct {+ refs: ax.Refs,+ /// The app it last read or opened.+ last: ?i32 = null,+};++pub const Sessions = struct {+ gpa: std.mem.Allocator,+ map: std.AutoHashMapUnmanaged(u64, *Session) = .empty,++ pub fn get(s: *Sessions, conn: u64) !*Session {+ const gop = try s.map.getOrPut(s.gpa, conn);+ if (gop.found_existing) return gop.value_ptr.*;+ errdefer _ = s.map.remove(conn);+ const session = try s.gpa.create(Session);+ session.* = .{ .refs = .{ .gpa = s.gpa } };+ gop.value_ptr.* = session;+ return session;+ }++ pub fn remove(s: *Sessions, conn: u64) void {+ const kv = s.map.fetchRemove(conn) orelse return;+ kv.value.refs.deinit();+ s.gpa.destroy(kv.value);+ }++ pub fn deinit(s: *Sessions) void {+ var it = s.map.valueIterator();+ while (it.next()) |v| {+ v.*.refs.deinit();+ s.gpa.destroy(v.*);+ }+ s.map.deinit(s.gpa);+ }+};+pub const Mac = struct {gpa: std.mem.Allocator,queue: objc.dispatch_queue_t,- /// The last tree's elements: the queue's alone.- refs: ax.Refs,+ /// Each agent's refs and target: the queue's alone.+ sessions: Sessions,/// The kill switch as the queue sees it.stopped: std.atomic.Value(bool) = .init(false),/// The whole-screen switch as the queue sees it.whole_screen: std.atomic.Value(bool) = .init(false),- /// The app the agent last read or opened: the queue's alone.- last: ?i32 = null,self_pid: i32,pub fn init(gpa: std.mem.Allocator) Mac {ax.setTimeout(2.0);- return .{ .gpa = gpa, .queue = dispatch_queue_create("nanocua.tools", null), .refs = .{ .gpa = gpa }, .self_pid = getpid() };+ return .{ .gpa = gpa, .queue = dispatch_queue_create("nanocua.tools", null), .sessions = .{ .gpa = gpa }, .self_pid = getpid() };}pub fn backend(m: *Mac) mcp.Backend {- return .{ .ptr = m, .call = call };+ return .{ .ptr = m, .call = call, .closed = closed };}fn call(ptr: *anyopaque, tool: tools.Tool, args: tools.Args, reply: *mcp.Reply) void {@@ -50,6 +87,25 @@ pub const Mac = struct {const job = Job.create(m, tool, args, reply) catch return reply.fail("nanocua is out of memory");dispatch_async_f(m.queue, job, Job.run);}++ const Closed = struct {+ mac: *Mac,+ conn: u64,++ fn run(ctx: ?*anyopaque) callconv(.c) void {+ const c: *Closed = @ptrCast(@alignCast(ctx));+ c.mac.sessions.remove(c.conn);+ c.mac.gpa.destroy(c);+ }+ };++ /// After any calls of its still waiting on the queue.+ fn closed(ptr: *anyopaque, conn: u64) void {+ const m: *Mac = @ptrCast(@alignCast(ptr));+ const c = m.gpa.create(Closed) catch return;+ c.* = .{ .mac = m, .conn = conn };+ dispatch_async_f(m.queue, c, Closed.run);+ }};/// Permissions, as the panel shows them.@@ -61,14 +117,45 @@ pub fn screenRecording() bool {return capture.allowed();}-/// Asks for a permission: the system's prompt, then its Settings pane.-pub fn request(which: enum { accessibility, screen }) void {+/// Keeps nanocua's window above other apps' while a question waits: macOS may+/// refuse to bring an app forward that nobody clicked.+pub fn raise(window: *nanoui.Window, on: bool) void {+ const floating: objc.NSInteger = 3;+ objc.msg(void, window.nswindow, "setLevel:", .{if (on) floating else @as(objc.NSInteger, 0)});+ if (on) objc.msg(void, window.nswindow, "orderFrontRegardless", .{});+}++pub const Permission = enum(usize) { accessibility = 1, screen = 2 };++/// Asks for a permission: the system's prompt, then its Settings pane. After+/// this frame, not during it: some of these calls run the event loop while they+/// wait, and a frame started inside a frame leaves its boxes open.+pub fn request(which: Permission) void {+ objc.dispatchMain(@ptrFromInt(@backingInt(which)), asked);+}++fn asked(ctx: ?*anyopaque) callconv(.c) void {+ const which: Permission = @fromBackingInt(@intCast(@intFromPtr(ctx)));switch (which) {.accessibility => if (!cf.trusted(true)) openSettings("Privacy_Accessibility"),- .screen => if (!cf.CGRequestScreenCaptureAccess()) openSettings("Privacy_ScreenCapture"),+ .screen => if (!cf.CGRequestScreenCaptureAccess()) {+ capture.register();+ openSettings("Privacy_ScreenCapture");+ // macOS may not list the app even so, and only a person can add it.+ revealApp();+ },}}+/// Shows this app in Finder, ready to drag into a System Settings list.+fn revealApp() void {+ const pool = objc.Pool.push();+ defer pool.pop();+ const bundle = objc.msg(objc.Object, objc.class("NSBundle"), "mainBundle", .{});+ const path = objc.fromNSString(objc.msg(?objc.Object, bundle, "bundlePath", .{}));+ if (std.mem.endsWith(u8, path, ".app")) nanoui.shell.showInFolder(path);+}+fn openSettings(pane: []const u8) void {var buf: [128]u8 = undefined;const url = std.fmt.bufPrint(&buf, "x-apple.systempreferences:com.apple.preference.security?{s}", .{pane}) catch return;@@ -77,6 +164,7 @@ fn openSettings(pane: []const u8) void {const Job = struct {mac: *Mac,+ session: *Session = undefined,tool: tools.Tool,reply: *mcp.Reply,arena: std.heap.ArenaAllocator,@@ -140,7 +228,10 @@ const Job = struct {fn run(ctx: ?*anyopaque) callconv(.c) void {const j: *Job = @ptrCast(@alignCast(ctx));const pool = objc.Pool.push();- j.perform() catch |e| j.fail("{s}", .{@errorName(e)});+ if (j.mac.sessions.get(j.reply.conn)) |session| {+ j.session = session;+ j.perform() catch |e| j.fail("{s}", .{@errorName(e)});+ } else |e| j.fail("{s}", .{@errorName(e)});pool.pop();objc.dispatchMain(j, deliver);}@@ -178,12 +269,12 @@ const Job = struct {.get_ui_tree => {if (!accessibility()) return j.fail("nanocua has no Accessibility permission: ask the user to grant it in nanocua's window.", .{});const named = j.namedPid() orelse return;- const pid = named orelse m.last orelse+ const pid = named orelse j.session.last orelseax.focusedPid() orelse return j.fail("No app has the keyboard focus: pass pid or app.", .{});if (pid == m.self_pid) return j.fail("That is nanocua itself, which agents don't see. Pass pid or app (see list_apps).", .{});const max: usize = @intCast(std.math.clamp(j.max_nodes orelse 400, 1, 2000));- try ax.writeTree(&m.refs, w, pid, .{ .max_nodes = max });- m.last = pid;+ try ax.writeTree(&j.session.refs, w, pid, .{ .max_nodes = max });+ j.session.last = pid;},.screenshot => try j.screenshot(),.click => {@@ -193,14 +284,22 @@ const Job = struct {const el = j.element(r) orelse return;const t = j.events(ax.pidOf(el)) orelse return;if (button == .left and count == 1 and ax.perform(el, "AXPress")) return w.print("Pressed [{d}].", .{r});- const c = ax.center(el) orelse return j.fail("[{d}] has no frame to click.", .{r});- input.click(t, c[0], c[1], button, count);- return w.print("Clicked [{d}] at {d:.0},{d:.0}.", .{ r, c[0], c[1] });+ const f = ax.frame(el) orelse return j.fail("[{d}] has no frame to click.", .{r});+ const c = j.inWindow(t, f) orelse return j.fail("[{d}] is out of sight in its window: scroll to it first.", .{r});+ const aimed = j.aim(t, c[0], c[1]) orelse return;+ input.click(aimed, c[0], c[1], button, count);+ return j.sent(aimed, "a click to", c);}const t = j.events(null) orelse return;const p = j.point(t) orelse return;- input.click(t, p[0], p[1], button, count);- try w.print("Clicked at {d:.0},{d:.0}.", .{ p[0], p[1] });+ const aimed = j.aim(t, p[0], p[1]) orelse return;+ if (aimed == .app and button == .left and count == 1) {+ var buf: [64]u8 = undefined;+ if (hit.press(&buf, aimed.app.pid, p[0], p[1])) |what|+ return w.print("Pressed the {s} at {d:.0},{d:.0}.", .{ what, p[0], p[1] });+ }+ input.click(aimed, p[0], p[1], button, count);+ try j.sent(aimed, "a click to", p);},.type_text => {const text = j.text orelse return j.fail("type_text needs text.", .{});@@ -212,7 +311,7 @@ const Job = struct {} else null;const t = j.events(el_pid) orelse return;if (el) |e| {- if (!ax.focus(e)) if (ax.center(e)) |c| input.click(t, c[0], c[1], .left, 1);+ if (!ax.focus(e)) if (ax.center(e)) |c| if (j.aim(t, c[0], c[1])) |aimed| input.click(aimed, c[0], c[1], .left, 1);_ = usleep(60_000);}if (!j.keyboardIsOthers(t)) return;@@ -230,23 +329,28 @@ const Job = struct {},.scroll => {var el_pid: ?i32 = null;- const center = if (j.ref) |r| blk: {+ var el_frame: ?[4]f64 = null;+ if (j.ref) |r| {const el = j.element(r) orelse return;el_pid = ax.pidOf(el);- break :blk ax.center(el) orelse return j.fail("[{d}] has no frame.", .{r});- } else null;+ el_frame = ax.frame(el) orelse return j.fail("[{d}] has no frame.", .{r});+ }const t = j.events(el_pid) orelse return;- const p = center orelse j.point(t) orelse return;+ const p = if (el_frame) |f|+ j.inWindow(t, f) orelse return j.fail("[{d}] is out of sight in its window.", .{j.ref.?})+ else+ j.point(t) orelse return;+ const aimed = j.aim(t, p[0], p[1]) orelse return;const dx = j.dx orelse 0;const dy: f64 = j.dy orelse if (dx == 0) 3 else 0;- input.scroll(t, p[0], p[1], dx, dy);- try w.print("Scrolled {d} down, {d} right at {d:.0},{d:.0}.", .{ dy, dx, p[0], p[1] });+ try j.scrollAt(aimed, p, dx, dy);},.move_mouse => {const t = j.events(null) orelse return;const p = j.point(t) orelse return;- input.move(t, p[0], p[1]);- try w.print("Moved to {d:.0},{d:.0}.", .{ p[0], p[1] });+ const aimed = j.aim(t, p[0], p[1]) orelse return;+ input.move(aimed, p[0], p[1]);+ try if (aimed == .hid) w.print("Moved the pointer to {d:.0},{d:.0}.", .{ p[0], p[1] }) else j.sent(aimed, "a hover to", p);},.set_value => {const r = j.ref orelse return j.fail("set_value needs ref.", .{});@@ -267,7 +371,7 @@ const Job = struct {// A later get_ui_tree without app reads this one. (One just// launching isn't in the running list yet.)if (apps.pidNamed(n)) |pid| if (pid != m.self_pid) {- m.last = pid;+ j.session.last = pid;};},}@@ -285,7 +389,7 @@ const Job = struct {/// The element behind ref `r`, unless it is missing or nanocua's.fn element(j: *Job, r: i64) ?cf.AXUIElementRef {- const el = j.mac.refs.get(r) orelse {+ const el = j.session.refs.get(r) orelse {j.fail("No element [{d}]: refs come from the last get_ui_tree.", .{r});return null;};@@ -315,7 +419,7 @@ const Job = struct {.whole_screen = j.whole_screen,.allowed = m.whole_screen.load(.acquire),.pid = named orelse el_pid,- .last = m.last,+ .last = j.session.last,.self = m.self_pid,});switch (r) {@@ -325,12 +429,67 @@ const Job = struct {},.whole_screen => return .hid,.app => |pid| {- m.last = pid;+ j.session.last = pid;return .{ .app = .{ .pid = pid, .window = if (windows.main(pid)) |w| w.id else 0 } };},}}+ /// Where to aim at an element so that the event lands in its window: on the+ /// real pointer its middle, in an app the middle of what shows.+ fn inWindow(j: *Job, t: input.Target, f: [4]f64) ?[2]f64 {+ _ = j;+ const middle: [2]f64 = .{ f[0] + f[2] / 2, f[1] + f[3] / 2 };+ if (t == .hid) return middle;+ const w = windows.at(t.app.pid, middle[0], middle[1]) orelse windows.main(t.app.pid) orelse return null;+ return target.visiblePoint(f, windows.box(w));+ }++ /// `t` aimed at the window under x, y. An app's window must be there: an+ /// event anywhere else goes nowhere, and saying so beats pretending.+ fn aim(j: *Job, t: input.Target, x: f64, y: f64) ?input.Target {+ if (t == .hid) return t;+ const w = windows.at(t.app.pid, x, y) orelse {+ if (windows.main(t.app.pid)) |m| {+ const b = windows.box(m);+ j.fail("{d:.0},{d:.0} isn't in a window of that app. Its main window is at {d:.0},{d:.0} sized {d:.0}x{d:.0}.", .{ x, y, b[0], b[1], b[2], b[3] });+ } else j.fail("That app has no window to act in.", .{});+ return null;+ };+ return .{ .app = .{ .pid = t.app.pid, .window = w.id, .origin = .{ w.frame.origin.x, w.frame.origin.y } } };+ }++ /// Reports events that can't be confirmed, which is every event sent into+ /// an app: macOS doesn't say whether it acted on them.+ fn sent(j: *Job, t: input.Target, what: []const u8, p: [2]f64) !void {+ const w = &j.out.writer;+ if (t == .hid) return w.print("Sent {s} the screen at {d:.0},{d:.0}.", .{ what, p[0], p[1] });+ try w.print("Sent {s} the app at {d:.0},{d:.0}. Not confirmed: apps can ignore events in the background, so check with get_ui_tree or screenshot.", .{ what, p[0], p[1] });+ }++ /// Scrolls and says what really happened: with a scroll bar to read, it knows.+ fn scrollAt(j: *Job, t: input.Target, p: [2]f64, dx: f64, dy: f64) !void {+ const w = &j.out.writer;+ if (t == .hid) {+ input.scroll(t, p[0], p[1], dx, dy);+ return w.print("Scrolled the screen {d} down, {d} right at {d:.0},{d:.0}.", .{ dy, dx, p[0], p[1] });+ }+ const pid = t.app.pid;+ const before = hit.position(pid, p[0], p[1]);+ input.scroll(t, p[0], p[1], dx, dy);+ if (!before.known()) return j.sent(t, "a scroll to", p);+ // Scrolling can animate.+ _ = usleep(200_000);+ if (hit.position(pid, p[0], p[1]).moved(before))+ return w.print("Scrolled {d} down, {d} right at {d:.0},{d:.0}.", .{ dy, dx, p[0], p[1] });+ if (before.atEnd(dx, dy)) return w.print("Nothing moved: it's already at the end that way.", .{});+ hit.scrollBars(pid, p[0], p[1], dx, dy);+ _ = usleep(50_000);+ if (hit.position(pid, p[0], p[1]).moved(before))+ return w.print("Scrolled at {d:.0},{d:.0} through its scroll bars: about {d} down, {d} right.", .{ p[0], p[1], dy, dx });+ j.fail("Nothing scrolled at {d:.0},{d:.0}: the app ignored it. It may not be scrollable there, or need whole_screen.", .{ p[0], p[1] });+ }+/// x and y. On the real pointer, not over nanocua's own windows.fn point(j: *Job, t: input.Target) ?[2]f64 {const x = j.x orelse {@@ -365,7 +524,7 @@ const Job = struct {.whole_screen = j.whole_screen,.allowed = m.whole_screen.load(.acquire),.pid = named,- .last = m.last,+ .last = j.session.last,.self = m.self_pid,});const max: u32 = @intCast(std.math.clamp(j.max_size orelse 1280, 64, 2560));@@ -387,7 +546,7 @@ const Job = struct {.app => |pid| blk: {const w = windows.main(pid) orelse return j.fail("App {d} has no window to capture.", .{pid});rect = w.frame;- m.last = pid;+ j.session.last = pid;break :blk capture.window(m.gpa, w.id, w.frame, max);},};@@ -401,3 +560,17 @@ const Job = struct {try j.out.writer.print("The image is {d}x{d} px of the region at {d:.0},{d:.0} sized {d:.0}x{d:.0} points: pixel (px, py) is the point ({d:.0} + px*{d:.4}, {d:.0} + py*{d:.4}).", .{ shot.width, shot.height, rect.origin.x, rect.origin.y, rect.size.width, rect.size.height, rect.origin.x, sx, rect.origin.y, sy });}};++test "agents don't share refs or a target; a closed agent's are freed" {+ var s: Sessions = .{ .gpa = std.testing.allocator };+ defer s.deinit();+ const a = try s.get(1);+ const b = try s.get(2);+ try std.testing.expect(a != b);+ a.last = 42;+ try std.testing.expectEqual(@as(?i32, null), b.last);+ try std.testing.expectEqual(a, try s.get(1));+ s.remove(1);+ s.remove(1);+ try std.testing.expectEqual(@as(?i32, null), (try s.get(1)).last);+}diff --git a/apps/nanocua/src/mac/capture.zig b/apps/nanocua/src/mac/capture.zigindex 45bff006..17a9291d 100644--- a/apps/nanocua/src/mac/capture.zig+++ b/apps/nanocua/src/mac/capture.zig@@ -43,6 +43,19 @@ pub fn allowed() bool {return cf.CGPreflightScreenCaptureAccess();}+var nothing: u8 = 0;++fn ignore(_: *u8, _: ?objc.Object, _: ?objc.Object) void {}++/// Asks ScreenCaptureKit for what it could capture and ignores the answer:+/// without permission, this is what gets the app listed in System Settings,+/// where the permission request alone may not.+pub fn register() void {+ const cls = objc_getClass("SCShareableContent") orelse return;+ var block = objc.Block(u8, &.{ ?objc.Object, ?objc.Object }).init(¬hing, ignore);+ objc.msg(void, cls, "getShareableContentWithCompletionHandler:", .{@as(*anyopaque, @ptrCast(&block))});+}+/// A rectangle of the screen.pub fn capture(gpa: std.mem.Allocator, rect: cf.CGRect, max_size: u32) Error!Shot {// Asking would put up the system's prompt for an agent: only the user@@ -119,8 +132,8 @@ const Shareable = struct {const answer_timeout: u64 = 10 * std.time.ns_per_s;-/// ScreenCaptureKit's window with number `id`, retained.-fn scWindow(gpa: std.mem.Allocator, cls: objc.Class, id: u32) !?objc.Object {+/// What ScreenCaptureKit could capture, retained.+fn shareableContent(gpa: std.mem.Allocator, cls: objc.Class) !?objc.Object {const wait = try gpa.create(Shareable);wait.* = .{ .sem = objc.dispatch_semaphore_create(0) };var block = objc.Block(Shareable, &.{ ?objc.Object, ?objc.Object }).init(wait, Shareable.done);@@ -129,25 +142,55 @@ fn scWindow(gpa: std.mem.Allocator, cls: objc.Class, id: u32) !?objc.Object {if (objc.dispatch_semaphore_wait(wait.sem, dispatchTime(answer_timeout)) != 0) return null;defer gpa.destroy(wait);defer objc.dispatch_release(@ptrCast(wait.sem));- const content = wait.content orelse return null;- defer objc.release(content);+ return wait.content;+}++fn findWindow(content: objc.Object, id: u32) ?objc.Object {const list = objc.msg(objc.Object, content, "windows", .{});for (0..objc.msg(usize, list, "count", .{})) |i| {const w = objc.msg(objc.Object, list, "objectAtIndex:", .{i});- if (objc.msg(u32, w, "windowID", .{}) == id) return objc.retain(w);+ if (objc.msg(u32, w, "windowID", .{}) == id) return w;}return null;}+/// How much of `frame` lies in `other`, in square points.+pub fn overlap(frame: objc.CGRect, other: objc.CGRect) f64 {+ const w = @min(frame.origin.x + frame.size.width, other.origin.x + other.size.width) - @max(frame.origin.x, other.origin.x);+ const h = @min(frame.origin.y + frame.size.height, other.origin.y + other.size.height) - @max(frame.origin.y, other.origin.y);+ return if (w > 0 and h > 0) w * h else 0;+}++/// The display showing most of `frame`.+fn displayOf(content: objc.Object, frame: objc.CGRect) ?objc.Object {+ const list = objc.msg(objc.Object, content, "displays", .{});+ var best: ?objc.Object = null;+ var most: f64 = 0;+ for (0..objc.msg(usize, list, "count", .{})) |i| {+ const d = objc.msg(objc.Object, list, "objectAtIndex:", .{i});+ const shared = overlap(frame, objc.msg(objc.CGRect, d, "frame", .{}));+ if (shared > most) {+ most = shared;+ best = d;+ }+ }+ return best;+}++/// `frame` in the coordinates of a display at `display`.+pub fn inDisplay(frame: objc.CGRect, display: objc.CGRect) objc.CGRect {+ return .{ .origin = .{ .x = frame.origin.x - display.origin.x, .y = frame.origin.y - display.origin.y }, .size = frame.size };+}+fn grabWindow(gpa: std.mem.Allocator, id: u32, frame: cf.CGRect, max_size: u32) !?CGImageRef {const manager = objc_getClass("SCScreenshotManager");const shareable = objc_getClass("SCShareableContent");const modern = if (manager) |c| objc.msg(bool, c, "respondsToSelector:", .{objc.sel("captureImageWithFilter:configuration:completionHandler:")}) else false;if (!modern or shareable == null) return CGWindowListCreateImage(CGRectNull, list_including_window, id, image_bounds_ignore_framing);- const sc = try scWindow(gpa, shareable.?, id) orelse return null;- defer objc.release(sc);- const filter = objc.msg(objc.Object, objc.alloc("SCContentFilter"), "initWithDesktopIndependentWindow:", .{sc});- defer objc.release(filter);+ const content = try shareableContent(gpa, shareable.?) orelse return null;+ defer objc.release(content);+ const sc = findWindow(content, id) orelse return null;+ const rect: objc.CGRect = .{ .origin = .{ .x = frame.origin.x, .y = frame.origin.y }, .size = .{ .width = frame.size.width, .height = frame.size.height } };const config = objc.new("SCStreamConfiguration");defer objc.release(config);// Retina's pixels, unless that's more than the picture will keep.@@ -156,6 +199,20 @@ fn grabWindow(gpa: std.mem.Allocator, id: u32, frame: cf.CGRect, max_size: u32)objc.msg(void, config, "setWidth:", .{@as(usize, @intFromFloat(@max(1, @round(frame.size.width * scale))))});objc.msg(void, config, "setHeight:", .{@as(usize, @intFromFloat(@max(1, @round(frame.size.height * scale))))});objc.msg(void, config, "setShowsCursor:", .{false});+ // On a display: that display with only this window on it, whatever is in+ // front, cut to its frame, so pixels map exactly to points. The window's own+ // filter widens to fit child windows (a floating orb, a popover) and shrinks+ // the picture, and its source rect isn't in one coordinate system. A window+ // off the displays (another Space) has only that filter.+ const display = if (objc.msg(bool, sc, "isOnScreen", .{})) displayOf(content, rect) else null;+ const filter = if (display) |d| blk: {+ objc.msg(void, config, "setSourceRect:", .{inDisplay(rect, objc.msg(objc.CGRect, d, "frame", .{}))});+ if (objc.msg(bool, config, "respondsToSelector:", .{objc.sel("setIncludeChildWindows:")}))+ objc.msg(void, config, "setIncludeChildWindows:", .{false});+ const only = objc.msg(objc.Object, objc.class("NSArray"), "arrayWithObject:", .{sc});+ break :blk objc.msg(objc.Object, objc.alloc("SCContentFilter"), "initWithDisplay:includingWindows:", .{ d, only });+ } else objc.msg(objc.Object, objc.alloc("SCContentFilter"), "initWithDesktopIndependentWindow:", .{sc});+ defer objc.release(filter);const wait = try gpa.create(Wait);wait.* = .{ .sem = objc.dispatch_semaphore_create(0) };var block = objc.Block(Wait, &.{ ?*anyopaque, ?objc.Object }).init(wait, Wait.done);@@ -190,3 +247,15 @@ extern "c" fn dispatch_time(when: u64, delta: i64) u64;fn dispatchTime(ns: u64) u64 {return dispatch_time(0, @intCast(ns));}++test "a window's place on its display" {+ const win: objc.CGRect = .{ .origin = .{ .x = 1500, .y = 100 }, .size = .{ .width = 400, .height = 300 } };+ const left: objc.CGRect = .{ .size = .{ .width = 1440, .height = 900 } };+ const right: objc.CGRect = .{ .origin = .{ .x = 1440 }, .size = .{ .width = 1920, .height = 1080 } };+ try std.testing.expectEqual(@as(f64, 0), overlap(win, left));+ try std.testing.expectEqual(@as(f64, 120000), overlap(win, right));+ const placed = inDisplay(win, right);+ try std.testing.expectEqual(@as(f64, 60), placed.origin.x);+ try std.testing.expectEqual(@as(f64, 100), placed.origin.y);+ try std.testing.expectEqual(@as(f64, 400), placed.size.width);+}diff --git a/apps/nanocua/src/mac/hit.zig b/apps/nanocua/src/mac/hit.zignew file mode 100644index 00000000..319e82e6--- /dev/null+++ b/apps/nanocua/src/mac/hit.zig@@ -0,0 +1,180 @@+//! What's at a point in an app, through accessibility: an exact way to act+//! when events in the background are only a hope. Pressing what's there, and+//! scrolling what's there and reading whether it moved.+const std = @import("std");+const cf = @import("cf.zig");+const ax = @import("ax.zig");+const tree = @import("../tree.zig");++/// App `pid`'s element at x, y; release it.+fn at(pid: i32, x: f64, y: f64) ?cf.AXUIElementRef {+ const app = cf.AXUIElementCreateApplication(pid);+ defer cf.CFRelease(app);+ var el: ?cf.AXUIElementRef = null;+ if (cf.AXUIElementCopyElementAtPosition(app, @floatCast(x), @floatCast(y), &el) != 0) return null;+ return el;+}++fn role(buf: []u8, el: cf.AXUIElementRef) []const u8 {+ return ax.copyText(buf, el, "AXRole");+}++/// `el`'s parent; release it.+fn parent(el: cf.AXUIElementRef) ?cf.AXUIElementRef {+ return @ptrCast(ax.copy(el, "AXParent"));+}++/// Presses what's at x, y if it's a control, going up from the label or icon+/// inside it. Writes the control's role into `buf`; null if nothing was pressed.+pub fn press(buf: []u8, pid: i32, x: f64, y: f64) ?[]const u8 {+ var el = at(pid, x, y) orelse return null;+ defer cf.CFRelease(el);+ for (0..4) |_| {+ const r = role(buf, el);+ if (tree.pressable(r)) return if (ax.perform(el, "AXPress")) tree.bareRole(r) else null;+ if (!tree.insideControl(r)) return null;+ const up = parent(el) orelse return null;+ cf.CFRelease(el);+ el = up;+ }+ return null;+}++pub const Bar = enum { vertical, horizontal };++fn barName(bar: Bar) []const u8 {+ return switch (bar) {+ .vertical => "AXVerticalScrollBar",+ .horizontal => "AXHorizontalScrollBar",+ };+}++/// The scroll area around x, y; release it.+fn scrollArea(pid: i32, x: f64, y: f64) ?cf.AXUIElementRef {+ var el = at(pid, x, y) orelse return null;+ var buf: [64]u8 = undefined;+ for (0..32) |_| {+ if (std.mem.eql(u8, role(&buf, el), "AXScrollArea")) return el;+ const up = parent(el) orelse break;+ cf.CFRelease(el);+ el = up;+ }+ cf.CFRelease(el);+ return null;+}++fn value(bar: cf.AXUIElementRef) ?f64 {+ const v = ax.copy(bar, "AXValue") orelse return null;+ defer cf.CFRelease(v);+ if (cf.CFGetTypeID(v) != cf.CFNumberGetTypeID()) return null;+ var d: f64 = 0;+ return if (cf.CFNumberGetValue(@ptrCast(v), cf.number_double, &d)) d else null;+}++/// How far along it is scrolled, 0 to 1, on each axis, of what has a scroll bar.+pub const Position = struct {+ vertical: ?f64 = null,+ horizontal: ?f64 = null,++ pub fn known(p: Position) bool {+ return p.vertical != null or p.horizontal != null;+ }++ /// Whether anything moved since `before`.+ pub fn moved(p: Position, before: Position) bool {+ const near = struct {+ fn f(a: ?f64, b: ?f64) bool {+ return @abs((a orelse 0) - (b orelse 0)) < 1e-6;+ }+ }.f;+ return !near(p.vertical, before.vertical) or !near(p.horizontal, before.horizontal);+ }++ /// Whether it can't go that way anyway: the end of the document, not a failure.+ pub fn atEnd(p: Position, dx: f64, dy: f64) bool {+ const stuck = struct {+ fn f(v: ?f64, d: f64) bool {+ if (d == 0) return true;+ const at_ = v orelse return false;+ return if (d > 0) at_ >= 0.999 else at_ <= 0.001;+ }+ }.f;+ return stuck(p.vertical, dy) and stuck(p.horizontal, dx);+ }+};++/// The scroll position at x, y.+pub fn position(pid: i32, x: f64, y: f64) Position {+ const area = scrollArea(pid, x, y) orelse return .{};+ defer cf.CFRelease(area);+ var out: Position = .{};+ if (ax.copy(area, barName(.vertical))) |b| {+ defer cf.CFRelease(b);+ out.vertical = value(@ptrCast(b));+ }+ if (ax.copy(area, barName(.horizontal))) |b| {+ defer cf.CFRelease(b);+ out.horizontal = value(@ptrCast(b));+ }+ return out;+}++/// The share of the whole a move of `lines` lines is, in a view with `hidden`+/// points scrolled out of sight (a guess at the line's height).+pub fn lineShare(lines: f64, hidden: f64) f64 {+ const line_points = 16;+ return if (hidden > 1) lines * line_points / hidden else lines * 0.05;+}++/// How many points of the area's content are out of sight on an axis.+fn hiddenPoints(area: cf.AXUIElementRef, bar: Bar) f64 {+ const view = ax.frame(area) orelse return 0;+ const contents = cf.array(ax.copy(area, "AXContents")) orelse return 0;+ defer cf.CFRelease(contents);+ if (cf.CFArrayGetCount(contents) < 1) return 0;+ const first = cf.CFArrayGetValueAtIndex(contents, 0) orelse return 0;+ const inner = ax.frame(@ptrCast(first)) orelse return 0;+ return switch (bar) {+ .vertical => inner[3] - view[3],+ .horizontal => inner[2] - view[2],+ };+}++fn moveBar(area: cf.AXUIElementRef, bar: Bar, lines: f64) void {+ if (lines == 0) return;+ const b = ax.copy(area, barName(bar)) orelse return;+ defer cf.CFRelease(b);+ const now = value(@ptrCast(b)) orelse return;+ const next = std.math.clamp(now + lineShare(lines, hiddenPoints(area, bar)), 0, 1);+ const n = cf.CFNumberCreate(null, cf.number_double, &next) orelse return;+ defer cf.CFRelease(n);+ _ = ax.set(@ptrCast(b), "AXValue", n);+}++/// Scrolls through the scroll bars themselves, for an app that ignored the+/// wheel. The caller reads whether that worked.+pub fn scrollBars(pid: i32, x: f64, y: f64, dx: f64, dy: f64) void {+ const area = scrollArea(pid, x, y) orelse return;+ defer cf.CFRelease(area);+ moveBar(area, .vertical, dy);+ moveBar(area, .horizontal, dx);+}++test "scroll positions: moved, and the end of the document" {+ const p: Position = .{ .vertical = 0.5 };+ try std.testing.expect(p.known());+ try std.testing.expect(!(Position{}).known());+ try std.testing.expect(!p.moved(.{ .vertical = 0.5 }));+ try std.testing.expect(p.moved(.{ .vertical = 0.25 }));+ try std.testing.expect(!(Position{ .vertical = 0.5 }).atEnd(0, 10));+ try std.testing.expect((Position{ .vertical = 1 }).atEnd(0, 10));+ try std.testing.expect((Position{ .vertical = 0 }).atEnd(0, -10));+ try std.testing.expect(!(Position{ .vertical = 0 }).atEnd(0, 10));+ try std.testing.expect(!(Position{ .vertical = 1 }).atEnd(5, 10)); // no way to know sideways+}++test "a line's share of what is out of sight" {+ try std.testing.expectEqual(@as(f64, 0.1), lineShare(10, 1600));+ try std.testing.expectEqual(@as(f64, -0.1), lineShare(-10, 1600));+ try std.testing.expectEqual(@as(f64, 0.5), lineShare(10, 0)); // unknown: a guess+}diff --git a/apps/nanocua/src/mac/hotkey.zig b/apps/nanocua/src/mac/hotkey.zigindex 973e450d..371c9932 100644--- a/apps/nanocua/src/mac/hotkey.zig+++ b/apps/nanocua/src/mac/hotkey.zig@@ -8,7 +8,7 @@ const EventTypeSpec = extern struct { class: u32, kind: u32 };const HotKeyID = extern struct { signature: u32, id: u32 };const Handler = *const fn (next: ?*anyopaque, event: ?*anyopaque, user: ?*anyopaque) callconv(.c) i32;extern "c" fn GetApplicationEventTarget() *anyopaque;-extern "c" fn InstallEventHandler(target: *anyopaque, handler: Handler, n: u32, list: [*]const EventTypeSpec, user: ?*anyopaque, out: ?*?*anyopaque) i32;+extern "c" fn InstallEventHandler(target: *anyopaque, handler: Handler, n: usize, list: [*]const EventTypeSpec, user: ?*anyopaque, out: ?*?*anyopaque) i32;extern "c" fn RegisterEventHotKey(code: u32, modifiers: u32, id: HotKeyID, target: *anyopaque, options: u32, out: *?*anyopaque) i32;extern "c" fn UnregisterEventHotKey(ref: *anyopaque) i32;diff --git a/apps/nanocua/src/mac/input.zig b/apps/nanocua/src/mac/input.zigindex c51f113e..49bbfb08 100644--- a/apps/nanocua/src/mac/input.zig+++ b/apps/nanocua/src/mac/input.zig@@ -5,6 +5,7 @@const std = @import("std");const keys = @import("../keys.zig");const cf = @import("cf.zig");+const skylight = @import("skylight.zig");const CGEventRef = *opaque {};extern "c" fn CGEventCreateMouseEvent(source: ?*anyopaque, kind: u32, at: cf.CGPoint, button: u32) ?CGEventRef;@@ -19,19 +20,39 @@ extern "c" fn CGEventPostToPid(pid: i32, ev: CGEventRef) void;extern "c" fn usleep(us: c_uint) c_int;const hid_tap: u32 = 0;+const mouse_moved: u32 = 5;+// CGEvent's fields. Those without a public name are what WindowServer stamps on+// an event it routes itself, and an app in the background goes by them.const field_click_state: u32 = 1;+const field_button: u32 = 3;+const field_subtype: u32 = 7;+const field_target_pid: u32 = 40;+const field_window_number: u32 = 51;+const field_click_group: u32 = 58;const field_window_under_pointer: u32 = 91;const field_window_handling: u32 = 92;-const mouse_moved: u32 = 5;+const subtype_touch: i64 = 3;+/// A scroll event carries at most this many lines: more arrive coalesced or not at all.+const lines_per_event: i32 = 10;pub const Button = enum { left, right };+/// One app's window, where mouse events land.+pub const App = struct {+ pid: i32,+ /// The window under the event, and its top-left in screen points.+ window: u32 = 0,+ origin: [2]f64 = .{ 0, 0 },+ /// Ties the events of one gesture together.+ group: i64 = 0,+};+/// Where events go.pub const Target = union(enum) {/// The real pointer and keyboard.hid,- /// Into one app. `window` is what a mouse event says it is over (0: unknown).- app: struct { pid: i32, window: u32 = 0 },+ /// Into one app.+ app: App,fn post(t: Target, ev: ?CGEventRef) void {const e = ev orelse return;@@ -42,38 +63,78 @@ pub const Target = union(enum) {cf.CFRelease(e);}- /// A mouse event at x, y, saying which window it is over: an app in the- /// background otherwise goes by where the real pointer is.- fn mouse(t: Target, kind: u32, x: f64, y: f64, button: u32) ?CGEventRef {- const ev = CGEventCreateMouseEvent(null, kind, .{ .x = x, .y = y }, button) orelse return null;- if (t == .app and t.app.window != 0) {- CGEventSetIntegerValueField(ev, field_window_under_pointer, t.app.window);- CGEventSetIntegerValueField(ev, field_window_handling, t.app.window);+ /// Posts a mouse or scroll event, with what an app in the background needs+ /// to act on it at x, y.+ fn postPointer(t: Target, ev: ?CGEventRef, x: f64, y: f64, state: i64, button: i64) void {+ const e = ev orelse return;+ defer cf.CFRelease(e);+ const a = switch (t) {+ .hid => return CGEventPost(hid_tap, e),+ .app => |app| app,+ };+ // Exactly no modifiers: an event otherwise inherits what the user holds.+ CGEventSetFlags(e, 0);+ skylight.setField(e, field_target_pid, a.pid);+ if (a.window != 0) {+ // Relative to the window: the live test missed with screen points or none.+ skylight.setWindowLocation(e, x - a.origin[0], y - a.origin[1]);+ skylight.setField(e, field_click_state, state);+ skylight.setField(e, field_button, button);+ skylight.setField(e, field_subtype, subtype_touch);+ skylight.setField(e, field_window_number, a.window);+ skylight.setField(e, field_click_group, a.group);+ skylight.setField(e, field_window_under_pointer, a.window);+ skylight.setField(e, field_window_handling, a.window);}- return ev;+ skylight.post(a.pid, e);}};+/// A mouse event's type and button number.+fn press_kinds(button: Button) struct { down: u32, up: u32, number: u32 } {+ return switch (button) {+ .left => .{ .down = 1, .up = 2, .number = 0 },+ .right => .{ .down = 3, .up = 4, .number = 1 },+ };+}+pub fn move(t: Target, x: f64, y: f64) void {- t.post(t.mouse(mouse_moved, x, y, 0));+ t.postPointer(CGEventCreateMouseEvent(null, mouse_moved, .{ .x = x, .y = y }, 0), x, y, 0, 0);+}++/// Gestures so far. Only the backend's queue sends events.+var next_group: i64 = 0;++/// Starts a gesture: a move first, so an app in the background knows where the+/// pointer now is before a press arrives.+fn prepare(t: Target, x: f64, y: f64) Target {+ var out = t;+ if (out == .app) {+ next_group += 1;+ out.app.group = next_group;+ }+ move(out, x, y);+ _ = usleep(15_000);+ return out;}/// Clicks `count` times at x, y (a double click is two, the second with/// click state 2, as the system makes it).pub fn click(t: Target, x: f64, y: f64, button: Button, count: u32) void {- const down: u32, const up: u32, const b: u32 = switch (button) {- .left => .{ 1, 2, 0 },- .right => .{ 3, 4, 1 },- };- move(t, x, y);- _ = usleep(15_000);+ const k = press_kinds(button);+ const at: cf.CGPoint = .{ .x = x, .y = y };+ const ready = prepare(t, x, y);for (1..@min(count, 3) + 1) |n| {- for ([_]u32{ down, up }) |kind| {- const ev = t.mouse(kind, x, y, b) orelse continue;- CGEventSetIntegerValueField(ev, field_click_state, @intCast(n));- t.post(ev);- _ = usleep(8_000);- }+ const state: i64 = @intCast(n);+ const down = CGEventCreateMouseEvent(null, k.down, at, k.number);+ if (ready == .hid) if (down) |d| CGEventSetIntegerValueField(d, field_click_state, state);+ ready.postPointer(down, x, y, state, k.number);+ // A control's tracking loop polls for the release: too soon and it's missed.+ _ = usleep(28_000);+ const up = CGEventCreateMouseEvent(null, k.up, at, k.number);+ if (ready == .hid) if (up) |u| CGEventSetIntegerValueField(u, field_click_state, state);+ ready.postPointer(up, x, y, state, k.number);+ _ = usleep(30_000);}}@@ -131,21 +192,32 @@ fn unicode(t: Target, units: []const u16) void {_ = usleep(4_000);}+/// The next wheel event's share of `left` lines.+pub fn wheelStep(left: i32) i32 {+ return std.math.clamp(left, -lines_per_event, lines_per_event);+}+/// Scrolls at x, y: `dy` lines down, `dx` lines right.pub fn scroll(t: Target, x: f64, y: f64, dx: f64, dy: f64) void {- if (t == .hid) {- move(t, x, y);- _ = usleep(15_000);- }- const clamp = struct {+ const ready = prepare(t, x, y);+ const whole = struct {fn f(v: f64) i32 {return @intFromFloat(std.math.clamp(@round(v), -200, 200));}}.f;- // Wheel 1 is vertical, positive up; wheel 2 horizontal, positive left.- const ev = CGEventCreateScrollWheelEvent2(null, 1, 2, -clamp(dy), -clamp(dx), 0) orelse return;- CGEventSetLocation(ev, .{ .x = x, .y = y });- t.post(ev);+ var down = whole(dy);+ var right = whole(dx);+ while (down != 0 or right != 0) {+ const sy = wheelStep(down);+ const sx = wheelStep(right);+ down -= sy;+ right -= sx;+ // Wheel 1 is vertical, positive up; wheel 2 horizontal, positive left.+ const ev = CGEventCreateScrollWheelEvent2(null, 1, 2, -sy, -sx, 0);+ if (ev) |e| CGEventSetLocation(e, .{ .x = x, .y = y });+ ready.postPointer(ev, x, y, 0, 0);+ _ = usleep(30_000);+ }}test "characters in UTF-16" {@@ -156,3 +228,10 @@ test "characters in UTF-16" {try std.testing.expectEqual(@as(u16, 0xD83D), u[0]);try std.testing.expectEqual(@as(u16, 0xDE00), u[1]);}++test "a long scroll goes out in steps" {+ try std.testing.expectEqual(@as(i32, 10), wheelStep(40));+ try std.testing.expectEqual(@as(i32, -10), wheelStep(-40));+ try std.testing.expectEqual(@as(i32, 3), wheelStep(3));+ try std.testing.expectEqual(@as(i32, 0), wheelStep(0));+}diff --git a/apps/nanocua/src/mac/pointer.zig b/apps/nanocua/src/mac/pointer.zignew file mode 100644index 00000000..afa6d061--- /dev/null+++ b/apps/nanocua/src/mac/pointer.zig@@ -0,0 +1,80 @@+//! Whether the click nanocua's window is handling came from a person's mouse+//! or trackpad. Turning a switch on only counts for a real click, so a program+//! that posts mouse events can't press Allow. It's a watch, not a filter: Stop+//! and the off positions work however they're clicked. Assistive tools that post+//! their own clicks (Switch Control, Voice Control) count as programs here.+const std = @import("std");+const objc = @import("nanoui").objc;++extern "c" fn CGEventGetIntegerValueField(ev: *anyopaque, field: u32) i64;++const field_source_pid: u32 = 41;+const field_source_state: u32 = 45;+/// kCGEventSourceStateHIDSystemState: the hardware's own state.+const hid_state: i64 = 1;++const left_down: usize = 1 << 1;+const left_up: usize = 1 << 2;+const type_left_down: usize = 1;++/// What a real click looked like in the live test: posted by no process, in+/// the hardware's state. Anything posted by a program names its process.+pub fn isHardware(source_pid: i64, source_state: i64) bool {+ return source_pid == 0 and source_state == hid_state;+}++var down_real = false;+var up_real = false;++/// Whether the press and the release just handled were both real. Main thread.+pub fn real() bool {+ return down_real and up_real;+}++fn note(event: objc.Object) void {+ const cg = objc.msg(?*anyopaque, event, "CGEvent", .{});+ const is_real = if (cg) |e| isHardware(CGEventGetIntegerValueField(e, field_source_pid), CGEventGetIntegerValueField(e, field_source_state)) else false;+ if (objc.msg(usize, event, "type", .{}) == type_left_down) {+ down_real = is_real;+ up_real = false;+ } else up_real = is_real;+}++const Descriptor = extern struct { reserved: usize = 0, size: usize };++/// A block returning the event: objc.Block's return nothing, and this API+/// drops an event its handler returns nothing for.+const Monitor = extern struct {+ isa: *anyopaque,+ flags: i32 = 0,+ reserved: i32 = 0,+ invoke: *const fn (*const Monitor, objc.Object) callconv(.c) objc.Object,+ descriptor: *const Descriptor,+};++const descriptor: Descriptor = .{ .size = @sizeOf(Monitor) };+extern "c" var _NSConcreteStackBlock: opaque {};++fn seen(_: *const Monitor, event: objc.Object) callconv(.c) objc.Object {+ note(event);+ return event;+}++/// Starts watching this process's clicks, before its windows handle them.+pub fn watch() void {+ var block: Monitor = .{ .isa = @ptrCast(&_NSConcreteStackBlock), .invoke = seen, .descriptor = &descriptor };+ // AppKit copies the block and holds the monitor for the life of the app.+ const monitor = objc.msg(?objc.Object, objc.class("NSEvent"), "addLocalMonitorForEventsMatchingMask:handler:", .{ left_down | left_up, @as(*anyopaque, @ptrCast(&block)) });+ if (monitor) |m| _ = objc.retain(m);+}++test "only an event no process posted, in the hardware's state, is real" {+ try std.testing.expect(isHardware(0, 1));+ try std.testing.expect(!isHardware(4242, 1));+ try std.testing.expect(!isHardware(0, 0));+ try std.testing.expect(!isHardware(4242, 0));+}++test "nothing has clicked yet: not real" {+ try std.testing.expect(!real());+}diff --git a/apps/nanocua/src/mac/skylight.zig b/apps/nanocua/src/mac/skylight.zignew file mode 100644index 00000000..b5067d4a--- /dev/null+++ b/apps/nanocua/src/mac/skylight.zig@@ -0,0 +1,50 @@+//! What plain CGEventPostToPid can't do for a window in the background: mouse+//! events an app acts on. These are private functions, looked up at run time, so+//! a macOS without them falls back to the public post (and the reply already says+//! a background click isn't confirmed). The recipe follows trycua's cua-driver (MIT).+const std = @import("std");+const cf = @import("cf.zig");++extern "c" fn dlopen(path: [*:0]const u8, mode: c_int) ?*anyopaque;+extern "c" fn dlsym(handle: ?*anyopaque, name: [*:0]const u8) ?*anyopaque;+extern "c" fn CGEventPostToPid(pid: i32, ev: *anyopaque) void;+extern "c" fn CGEventSetIntegerValueField(ev: *anyopaque, field: u32, value: i64) void;++const rtld_default: ?*anyopaque = @ptrFromInt(@as(usize, @bitCast(@as(isize, -2))));+const rtld_lazy: c_int = 1;++const Api = struct {+ post_to_pid: ?*const fn (pid: i32, ev: *anyopaque) callconv(.c) void = null,+ set_window_location: ?*const fn (ev: *anyopaque, at: cf.CGPoint) callconv(.c) void = null,+};++var api: Api = .{};+var loaded = false;++/// The backend's queue is the only caller, so no lock.+fn get() *const Api {+ if (loaded) return &api;+ loaded = true;+ _ = dlopen("/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight", rtld_lazy);+ api = .{+ .post_to_pid = @ptrCast(@alignCast(dlsym(rtld_default, "SLEventPostToPid"))),+ .set_window_location = @ptrCast(@alignCast(dlsym(rtld_default, "CGEventSetWindowLocation"))),+ };+ return &api;+}++/// Hands a mouse or scroll event to one app.+pub fn post(pid: i32, ev: *anyopaque) void {+ if (get().post_to_pid) |f| return f(pid, ev);+ CGEventPostToPid(pid, ev);+}++/// Says where in its window an event is, which an app in the background can't+/// work out from the pointer.+pub fn setWindowLocation(ev: *anyopaque, x: f64, y: f64) void {+ if (get().set_window_location) |f| f(ev, .{ .x = x, .y = y });+}++pub fn setField(ev: *anyopaque, field: u32, value: i64) void {+ CGEventSetIntegerValueField(ev, field, value);+}diff --git a/apps/nanocua/src/mac/windows.zig b/apps/nanocua/src/mac/windows.zigindex 3bacd7cb..a7dedd92 100644--- a/apps/nanocua/src/mac/windows.zig+++ b/apps/nanocua/src/mac/windows.zig@@ -36,6 +36,17 @@ fn int(v: ?*anyopaque) ?i64 {return if (cf.CFNumberGetValue(@ptrCast(x), number_sint64, &n)) n else null;}+fn read(d: cf.CFDictionaryRef, pid: i32) ?Window {+ if (int(CFDictionaryGetValue(d, kCGWindowOwnerPID)) != pid) return null;+ if (int(CFDictionaryGetValue(d, kCGWindowLayer)) != 0) return null;+ var w: Window = .{ .id = @intCast(int(CFDictionaryGetValue(d, kCGWindowNumber)) orelse return null), .frame = .{} };+ const bounds = CFDictionaryGetValue(d, kCGWindowBounds) orelse return null;+ if (!CGRectMakeWithDictionaryRepresentation(@ptrCast(bounds), &w.frame)) return null;+ if (w.frame.size.width < 2 or w.frame.size.height < 2) return null;+ w.onscreen = cf.boolean(CFDictionaryGetValue(d, kCGWindowIsOnscreen)) orelse false;+ return w;+}+/// App `pid`'s main window, if it has one.pub fn main(pid: i32) ?Window {const list = CGWindowListCopyWindowInfo(exclude_desktop, 0) orelse return null;@@ -44,25 +55,53 @@ pub fn main(pid: i32) ?Window {const n: usize = @intCast(cf.CFArrayGetCount(list));for (0..n) |i| {const d: cf.CFDictionaryRef = @ptrCast(cf.CFArrayGetValueAtIndex(list, @intCast(i)) orelse continue);- if (int(CFDictionaryGetValue(d, kCGWindowOwnerPID)) != pid) continue;- if (int(CFDictionaryGetValue(d, kCGWindowLayer)) != 0) continue;- var w: Window = .{ .id = @intCast(int(CFDictionaryGetValue(d, kCGWindowNumber)) orelse continue), .frame = .{} };- const bounds = CFDictionaryGetValue(d, kCGWindowBounds) orelse continue;- if (!CGRectMakeWithDictionaryRepresentation(@ptrCast(bounds), &w.frame)) continue;- if (w.frame.size.width < 2 or w.frame.size.height < 2) continue;- w.onscreen = cf.boolean(CFDictionaryGetValue(d, kCGWindowIsOnscreen)) orelse false;+ const w = read(d, pid) orelse continue;if (best == null or better(w, best.?)) best = w;}return best;}+pub fn contains(w: Window, x: f64, y: f64) bool {+ const f = w.frame;+ return x >= f.origin.x and y >= f.origin.y and x < f.origin.x + f.size.width and y < f.origin.y + f.size.height;+}++/// App `pid`'s window at x, y: the frontmost that shows there, as the list runs+/// front to back. Else one that isn't on screen just now (another Space).+pub fn at(pid: i32, x: f64, y: f64) ?Window {+ const list = CGWindowListCopyWindowInfo(exclude_desktop, 0) orelse return null;+ defer cf.CFRelease(list);+ var hidden: ?Window = null;+ const n: usize = @intCast(cf.CFArrayGetCount(list));+ for (0..n) |i| {+ const d: cf.CFDictionaryRef = @ptrCast(cf.CFArrayGetValueAtIndex(list, @intCast(i)) orelse continue);+ const w = read(d, pid) orelse continue;+ if (!contains(w, x, y)) continue;+ if (w.onscreen) return w;+ if (hidden == null) hidden = w;+ }+ return hidden;+}++/// A window's frame as x, y, width, height.+pub fn box(w: Window) [4]f64 {+ return .{ w.frame.origin.x, w.frame.origin.y, w.frame.size.width, w.frame.size.height };+}+test "the main window: on screen first, then the largest" {- const at = struct {+ const of = struct {fn f(w: f64, h: f64, on: bool) Window {return .{ .id = 1, .frame = .{ .size = .{ .width = w, .height = h } }, .onscreen = on };}}.f;- try std.testing.expect(better(at(1352, 802, true), at(1352, 47, true)));- try std.testing.expect(!better(at(1352, 47, true), at(1352, 802, true)));- try std.testing.expect(better(at(10, 10, true), at(1352, 849, false)));+ try std.testing.expect(better(of(1352, 802, true), of(1352, 47, true)));+ try std.testing.expect(!better(of(1352, 47, true), of(1352, 802, true)));+ try std.testing.expect(better(of(10, 10, true), of(1352, 849, false)));+}++test "a window contains the points in its frame" {+ const w: Window = .{ .id = 1, .frame = .{ .origin = .{ .x = 100, .y = 50 }, .size = .{ .width = 200, .height = 100 } } };+ try std.testing.expect(contains(w, 100, 50) and contains(w, 299, 149));+ try std.testing.expect(!contains(w, 300, 100) and !contains(w, 10, 10));+ try std.testing.expectEqual([4]f64{ 100, 50, 200, 100 }, box(w));}diff --git a/apps/nanocua/src/main.zig b/apps/nanocua/src/main.zigindex 238ecaf1..9c09911e 100644--- a/apps/nanocua/src/main.zig+++ b/apps/nanocua/src/main.zig@@ -1,17 +1,26 @@//! nanocua: lets AI agents see and use the computer, as far as its user//! allows. `nanocua` is the app: the control panel, and the server agents//! reach over a Unix socket. `nanocua mcp` is what an agent runs: an MCP-//! server on stdio that carries its lines to the app.+//! server on stdio that carries its lines to the app. `nanocua ctl` is what+//! graff and Harness run to show and change the switches.const std = @import("std");const nanoui = @import("nanoui");const app_mod = @import("app.zig");const theme = @import("theme.zig");+const builtin = @import("builtin");pub fn main(init: std.process.Init) !u8 {const args = try init.minimal.args.toSlice(init.arena.allocator());if (args.len > 1 and std.mem.eql(u8, args[1], "mcp"))return if (comptime @import("server.zig").supported) @import("proxy.zig").run(init) else 1;+ if (args.len > 1 and std.mem.eql(u8, args[1], "ctl"))+ return if (comptime @import("server.zig").supported) @import("ctl.zig").run(init, args[2..]) else 1;+ for (args) |arg| if (@import("control.zig").devFlagRefused(builtin.mode, arg)) {+ std.debug.print("nanocua: {s} is only for debug builds\n", .{arg});+ return 2;+ };+const app = try nanoui.App.create(init, .{ .name = "nanocua" });defer app.destroy();var plat: app_mod.Platform = .init(init.gpa);@@ -20,12 +29,16 @@ pub fn main(init: std.process.Init) !u8 {plat.attach(&a);defer a.deinit();var path_buf: [512]u8 = undefined;- if (@import("sockpath.zig").path(&path_buf, init.environ_map)) |path| a.listen(path) else |_| {+ if (@import("sockpath.zig").path(&path_buf, init.environ_map, @import("sockpath.zig").realHome())) |path| a.listen(path) else |_| {a.listen_error = "No folder for the socket (HOME is unset).";}var exe_buf: [1024]u8 = undefined;if (std.process.executablePath(init.io, &exe_buf)) |n| a.setCommand(exe_buf[0..n]) else |_| {}a.stopKey();+ if (comptime app_mod.mac) {+ @import("mac/pointer.zig").watch();+ a.hidden = app.flags.hidden;+ }a.window = try app.newWindow(.{.title = "nanocua",.width = 560,diff --git a/apps/nanocua/src/mcp.zig b/apps/nanocua/src/mcp.zigindex 545f9b92..88132f58 100644--- a/apps/nanocua/src/mcp.zig+++ b/apps/nanocua/src/mcp.zig@@ -21,6 +21,14 @@ pub const Sink = struct {pub const Backend = struct {ptr: *anyopaque,call: *const fn (ptr: *anyopaque, tool: tools.Tool, args: tools.Args, reply: *Reply) void,+ /// A connection went away: what it held can go too.+ closed: ?*const fn (ptr: *anyopaque, conn: u64) void = null,+};++/// The switches, for `nanocua/*` calls (control.zig). `id` lasts only the call.+pub const Control = struct {+ ptr: *anyopaque,+ func: *const fn (ptr: *anyopaque, conn: u64, id: []const u8, method: []const u8, params: ?std.json.Value) void,};pub const Content = union(enum) {@@ -46,6 +54,13 @@ pub const Hub = struct {changed: ?Callback = null,/// Calls not yet answered.pending: usize = 0,+ /// Null where nothing can answer them (`nanocua mcp`'s own hub): agents+ /// reach the switches only by opening the socket themselves.+ control: ?Control = null,++ pub fn closed(h: *Hub, conn: u64) void {+ if (h.backend.closed) |f| f(h.backend.ptr, conn);+ }/// Handles one line from connection `conn`.pub fn handle(h: *Hub, conn: u64, line: []const u8) void {@@ -74,6 +89,8 @@ pub const Hub = struct {h.sendResult(conn, id, aw.written());} else if (std.mem.eql(u8, method, "tools/call")) {h.call(conn, id, params);+ } else if (std.mem.startsWith(u8, method, "nanocua/") and h.control != null) {+ h.control.?.func(h.control.?.ptr, conn, id, method, params);} else h.sendError(conn, id, -32601, "Method not found");}@@ -101,14 +118,14 @@ pub const Hub = struct {if (h.changed) |c| c.func(c.ptr);}- fn sendResult(h: *Hub, conn: u64, id: []const u8, result: []const u8) void {+ pub fn sendResult(h: *Hub, conn: u64, id: []const u8, result: []const u8) void {var aw: std.Io.Writer.Allocating = .init(h.gpa);defer aw.deinit();aw.writer.print("{{\"jsonrpc\":\"2.0\",\"id\":{s},\"result\":{s}}}", .{ id, result }) catch return;h.sink.send(h.sink.ptr, conn, aw.written());}- fn sendError(h: *Hub, conn: u64, id: []const u8, code: i32, message: []const u8) void {+ pub fn sendError(h: *Hub, conn: u64, id: []const u8, code: i32, message: []const u8) void {var buf: [256]u8 = undefined;const line = std.fmt.bufPrint(&buf, "{{\"jsonrpc\":\"2.0\",\"id\":{s},\"error\":{{\"code\":{d},\"message\":\"{s}\"}}}}", .{ id, code, message }) catch return;h.sink.send(h.sink.ptr, conn, line);diff --git a/apps/nanocua/src/proxy.zig b/apps/nanocua/src/proxy.zigindex fc1e7907..16fa4d1a 100644--- a/apps/nanocua/src/proxy.zig+++ b/apps/nanocua/src/proxy.zig@@ -23,7 +23,7 @@ pub fn run(init: std.process.Init) !u8 {var buf: [512]u8 = undefined;var p: Proxy = .{.gpa = init.gpa,- .path = try sockpath.path(&buf, init.environ_map),+ .path = try sockpath.path(&buf, init.environ_map, sockpath.realHome()),.hub = .{.gpa = init.gpa,.sink = .{ .ptr = undefined, .send = toStdout },@@ -147,7 +147,20 @@ const Proxy = struct {}};-fn connect(path: [:0]const u8) ?c_int {+/// The app's socket, opening the app if it isn't running; null if it can't.+pub fn connectOrOpen(path: [:0]const u8) ?c_int {+ if (connect(path)) |fd| return fd;+ if (!launch()) return null;+ // The app starts in well under this; its first launch may ask the user+ // something first.+ for (0..100) |_| {+ _ = usleep(100_000);+ if (connect(path)) |fd| return fd;+ }+ return null;+}++pub fn connect(path: [:0]const u8) ?c_int {var addr: std.c.sockaddr.un = .{ .path = @splat(0) };if (path.len >= addr.path.len) return null;@memcpy(addr.path[0..path.len], path);diff --git a/apps/nanocua/src/server.zig b/apps/nanocua/src/server.zigindex 9db4f941..ffdc03d7 100644--- a/apps/nanocua/src/server.zig+++ b/apps/nanocua/src/server.zig@@ -50,6 +50,8 @@ const Conn = struct {fd: c_int,watch: ?fdwatch.Watch = null,lines: Lines = .{},+ /// It uses the switches, so it isn't an agent.+ control: bool = false,fn readable(ctx: ?*anyopaque) callconv(.c) void {if (comptime !supported) return;@@ -124,8 +126,15 @@ pub const Server = struct {return c.id;}+ /// Agents connected.pub fn clients(s: *const Server) usize {- return s.conns.count();+ var n: usize = 0;+ for (s.conns.values()) |c| n += @intFromBool(!c.control);+ return n;+ }++ pub fn markControl(s: *Server, id: u64) void {+ if (s.conns.get(id)) |c| c.control = true;}fn changed(s: *Server) void {@@ -141,6 +150,7 @@ pub const Server = struct {fn drop(s: *Server, id: u64) void {const kv = s.conns.fetchSwapRemove(id) orelse return;const c = kv.value;+ s.hub.closed(id);if (c.watch) |w| w.cancel();_ = close(c.fd);c.lines.deinit(s.gpa);diff --git a/apps/nanocua/src/sockpath.zig b/apps/nanocua/src/sockpath.zigindex 88ef9202..7420083a 100644--- a/apps/nanocua/src/sockpath.zig+++ b/apps/nanocua/src/sockpath.zig@@ -1,16 +1,25 @@//! Where the app listens and `nanocua mcp` connects: a Unix socket only//! this user can open, in nanocua's Application Support folder on macOS,-//! the user's runtime folder on Linux; NANOCUA_SOCKET overrides it.+//! the user's runtime folder on Linux; NANOCUA_SOCKET overrides it. `nanocua+//! mcp` and `nanocua ctl` find it the same way, whoever starts them.const std = @import("std");const builtin = @import("builtin");extern "c" fn mkdir(path: [*:0]const u8, mode: c_uint) c_int;extern "c" fn getuid() c_uint;-pub fn path(buf: []u8, env: *const std.process.Environ.Map) ![:0]const u8 {+/// The user's real home, not $HOME: Harness runs graff with HOME set per account,+/// and every program on one Mac has to find the one app.+pub fn realHome() ?[]const u8 {+ if (builtin.os.tag != .macos) return null;+ const entry = std.c.getpwuid(getuid()) orelse return null;+ return std.mem.span(entry.dir orelse return null);+}++pub fn path(buf: []u8, env: *const std.process.Environ.Map, real_home: ?[]const u8) ![:0]const u8 {if (env.get("NANOCUA_SOCKET")) |p| return std.fmt.bufPrintSentinel(buf, "{s}", .{p}, 0);if (builtin.os.tag == .macos) {- const home = env.get("HOME") orelse return error.NoHome;+ const home = real_home orelse env.get("HOME") orelse return error.NoHome;return std.fmt.bufPrintSentinel(buf, "{s}/Library/Application Support/nanocua/mcp.sock", .{home}, 0);}if (env.get("XDG_RUNTIME_DIR")) |d| return std.fmt.bufPrintSentinel(buf, "{s}/nanocua.sock", .{d}, 0);@@ -32,11 +41,21 @@ test "the path: overridden, or the platform's" {var buf: [256]u8 = undefined;try env.put("HOME", "/Users/ada");try env.put("XDG_RUNTIME_DIR", "/run/user/501");- const p = try path(&buf, &env);+ const p = try path(&buf, &env, null);if (builtin.os.tag == .macos)try std.testing.expectEqualStrings("/Users/ada/Library/Application Support/nanocua/mcp.sock", p)elsetry std.testing.expectEqualStrings("/run/user/501/nanocua.sock", p);try env.put("NANOCUA_SOCKET", "/tmp/x.sock");- try std.testing.expectEqualStrings("/tmp/x.sock", try path(&buf, &env));+ try std.testing.expectEqualStrings("/tmp/x.sock", try path(&buf, &env, null));+}++test "another $HOME doesn't move the socket" {+ if (builtin.os.tag != .macos) return error.SkipZigTest;+ var env: std.process.Environ.Map = .init(std.testing.allocator);+ defer env.deinit();+ var buf: [256]u8 = undefined;+ try env.put("HOME", "/Users/ada/.accounts/two");+ try std.testing.expectEqualStrings("/Users/ada/Library/Application Support/nanocua/mcp.sock", try path(&buf, &env, "/Users/ada"));+ try std.testing.expect(realHome() != null);}diff --git a/apps/nanocua/src/target.zig b/apps/nanocua/src/target.zigindex 3549d5eb..01af54fc 100644--- a/apps/nanocua/src/target.zig+++ b/apps/nanocua/src/target.zig@@ -31,6 +31,26 @@ pub fn route(a: Ask) Route {return .{ .app = pid };}+/// Where in `el` (x, y, width, height) to send an event so that it lands inside+/// `window`: the middle of what they share. An element can be far larger than its+/// window (a long document's text), and its own middle out of sight.+pub fn visiblePoint(el: [4]f64, window: [4]f64) ?[2]f64 {+ const left = @max(el[0], window[0]);+ const top = @max(el[1], window[1]);+ const right = @min(el[0] + el[2], window[0] + window[2]);+ const bottom = @min(el[1] + el[3], window[1] + window[3]);+ if (right <= left or bottom <= top) return null;+ return .{ (left + right) / 2, (top + bottom) / 2 };+}++test "a long element's point stays inside its window" {+ const window = [4]f64{ 128, 61, 586, 488 };+ // A document 3000 points tall, as the live test met it.+ try std.testing.expectEqual(@as(?[2]f64, .{ 421, 305 }), visiblePoint(.{ 128, 61, 586, 3000 }, window));+ try std.testing.expectEqual(@as(?[2]f64, .{ 150, 80 }), visiblePoint(.{ 140, 70, 20, 20 }, window));+ try std.testing.expectEqual(@as(?[2]f64, null), visiblePoint(.{ 128, 900, 586, 40 }, window));+}+test "a call goes into the app it names, else the last one read" {try std.testing.expectEqual(Route{ .app = 7 }, route(.{ .pid = 7, .last = 9, .self = 1 }));try std.testing.expectEqual(Route{ .app = 9 }, route(.{ .last = 9, .self = 1 }));diff --git a/apps/nanocua/src/tests.zig b/apps/nanocua/src/tests.zigindex 9e7a251d..de4c8409 100644--- a/apps/nanocua/src/tests.zig+++ b/apps/nanocua/src/tests.zig@@ -13,10 +13,14 @@ test {_ = @import("keys.zig");_ = @import("tree.zig");_ = @import("target.zig");+ _ = @import("control.zig");_ = @import("server.zig");_ = @import("sockpath.zig");_ = @import("tests_app.zig");- if (comptime @import("server.zig").supported) _ = @import("proxy.zig");+ if (comptime @import("server.zig").supported) {+ _ = @import("proxy.zig");+ _ = @import("ctl.zig");+ }std.testing.refAllDecls(@import("server.zig").Server);std.testing.refAllDecls(@import("app.zig").App);std.testing.refAllDecls(@import("app.zig").Platform);@@ -24,6 +28,10 @@ test {_ = @import("mac/cf.zig");_ = @import("mac/windows.zig");_ = @import("mac/input.zig");+ _ = @import("mac/skylight.zig");+ _ = @import("mac/hit.zig");+ _ = @import("mac/pointer.zig");+ _ = @import("mac/backend.zig");inline for (.{ @import("mac/backend.zig"), @import("mac/ax.zig"), @import("mac/input.zig"), @import("mac/capture.zig"), @import("mac/apps.zig"), @import("mac/windows.zig"), @import("mac/hotkey.zig"), @import("proxy.zig") }) |m| std.testing.refAllDecls(m);std.testing.refAllDecls(@import("mac/backend.zig").Mac);std.testing.refAllDecls(@import("mac/hotkey.zig").HotKey);diff --git a/apps/nanocua/src/tests_app.zig b/apps/nanocua/src/tests_app.zigindex 1d8fa12a..c9313fa5 100644--- a/apps/nanocua/src/tests_app.zig+++ b/apps/nanocua/src/tests_app.zig@@ -1,5 +1,6 @@//! The panel through the Tester (Stop and Resume, input, the log, the-//! command's Copy) and the socket end to end over a socket pair, with a+//! command's Copy), the switches from outside, and the socket end to end over+//! a socket pair, with a//! backend that only records: nothing here touches the real screen.const std = @import("std");const builtin = @import("builtin");@@ -51,6 +52,15 @@ test "the panel: stop and resume, input, the log, copying the command" {try std.testing.expect(t.hasText("Allow…"));}+ // Asking for Screen Recording says what's left to do, which is the user's.+ if (comptime builtin.os.tag == .macos) {+ try std.testing.expect(!t.hasText(view.screen_hint));+ try t.clickText("Allow…");+ t.settle(0.2);+ try std.testing.expect(a.asked_screen);+ try std.testing.expect(t.hasText(view.screen_hint));+ }+// Looking works; acting is refused until input is on.call(&a, 1, "get_ui_tree", "{\"app\":\"Notes\"}");call(&a, 2, "click", "{\"ref\":9}");@@ -104,6 +114,195 @@ test "the panel: stop and resume, input, the log, copying the command" {try std.testing.expect(t.hasText("Nothing yet. Every call an agent makes shows here."));}+test "the switches stay as the user left them: calls and redraws don't flip them" {+ const gpa = std.testing.allocator;+ var rec: Recorder = .{};+ var a: App = undefined;+ a.init(gpa, .{ .ptr = &rec, .call = Recorder.call });+ defer a.deinit();+ a.dark = false;+ var t: ui.Tester = undefined;+ try t.init(gpa, .init(&a, view.view), .{ .width = 560, .height = 720, .theme = theme.ui(false) });+ defer t.deinit();+ const whole = "Allow whole screen: your pointer and keyboard, pictures of everything";++ // Turning input on leaves the whole screen off, whatever comes after.+ try t.clickText("Allow input: clicks, typing, keys, opening apps");+ t.settle(0.5);+ try std.testing.expect(a.hub.policy.allow_input);+ try std.testing.expect(!a.hub.policy.allow_whole_screen);+ call(&a, 1, "screenshot", "{\"whole_screen\":true}");+ t.settle(0.5);+ try std.testing.expect(!a.hub.policy.allow_whole_screen);++ // One click, one flip; off stays off through calls, redraws and the pointer.+ try t.clickText(whole);+ t.settle(0.5);+ try std.testing.expect(a.hub.policy.allow_whole_screen);+ try t.clickText(whole);+ t.settle(0.5);+ try std.testing.expect(!a.hub.policy.allow_whole_screen);+ try std.testing.expect(!t.hasText("Whole screen"));+ for (0..5) |i| {+ call(&a, @intCast(i + 2), "get_ui_tree", "{}");+ t.frame();+ t.moveTo(300, @floatFromInt(300 + i * 10));+ t.settle(0.2);+ try std.testing.expect(!a.hub.policy.allow_whole_screen);+ try std.testing.expect(a.hub.policy.allow_input);+ }+}++/// The hub's clock and what it sends, for the control tests.+const Wire = struct {+ gpa: std.mem.Allocator,+ sent: std.ArrayList(u8) = .empty,+ var now: f64 = 0;+ fn clock() f64 {+ return now;+ }+ fn send(ptr: *anyopaque, conn: u64, line: []const u8) void {+ const w: *Wire = @ptrCast(@alignCast(ptr));+ w.sent.print(w.gpa, "{d}:{s}\n", .{ conn, line }) catch {};+ }+ fn saw(w: *Wire, text: []const u8) bool {+ return std.mem.indexOf(u8, w.sent.items, text) != null;+ }+};++fn control(a: *App, id: u32, method: []const u8, params: []const u8) void {+ var buf: [256]u8 = undefined;+ a.hub.handle(9, std.fmt.bufPrint(&buf, "{{\"jsonrpc\":\"2.0\",\"id\":{d},\"method\":\"nanocua/{s}\",\"params\":{s}}}", .{ id, method, params }) catch unreachable);+}++fn notReal() bool {+ return false;+}++test "from outside: off at once, on only by a click in the window" {+ const gpa = std.testing.allocator;+ var rec: Recorder = .{};+ var wire: Wire = .{ .gpa = gpa };+ defer wire.sent.deinit(gpa);+ var a: App = undefined;+ a.init(gpa, .{ .ptr = &rec, .call = Recorder.call });+ defer a.deinit();+ a.dark = false;+ Wire.now = 100;+ a.hub.clock = Wire.clock;+ a.hub.sink = .{ .ptr = &wire, .send = Wire.send };+ var t: ui.Tester = undefined;+ try t.init(gpa, .init(&a, view.view), .{ .width = 560, .height = 720, .theme = theme.ui(false) });+ defer t.deinit();++ // Asking changes nothing, answers nothing yet, and shows the question.+ control(&a, 1, "ask", "{\"what\":\"input\",\"from\":\"graff\"}");+ try std.testing.expect(!a.hub.policy.allow_input);+ try std.testing.expectEqual(@as(usize, 0), wire.sent.items.len);+ t.settle(0.2);+ try std.testing.expect(t.hasText("A request from graff"));+ try std.testing.expect(t.hasText(@import("control.zig").Want.input.question()));++ // A click right away can't answer: it may have been meant for something else.+ try t.clickText("Allow");+ t.settle(0.2);+ try std.testing.expect(!a.hub.policy.allow_input);++ // The status says what's waiting; a second question has to wait its turn.+ control(&a, 2, "status", "{}");+ try std.testing.expect(wire.saw("9:{\"jsonrpc\":\"2.0\",\"id\":2,\"result\":{"));+ try std.testing.expect(wire.saw("\"pending\":\"input\""));+ control(&a, 3, "ask", "{\"what\":\"whole_screen\"}");+ try std.testing.expect(wire.saw("\"outcome\":\"busy\""));++ // A program's click, once it's armed, is ignored, and the panel says so.+ Wire.now += 1;+ a.real_pointer = notReal;+ t.frame();+ try t.clickText("Allow");+ t.settle(0.2);+ try std.testing.expect(!a.hub.policy.allow_input);+ try std.testing.expect(a.gate.request != null);+ try std.testing.expect(t.hasText(view.ignored_click));++ // So are the panel's own switches, on the way on...+ try t.clickText("Allow input: clicks, typing, keys, opening apps");+ t.settle(0.2);+ try std.testing.expect(!a.hub.policy.allow_input);++ // A person's click answers it, and whoever asked hears.+ a.real_pointer = null;+ wire.sent.clearRetainingCapacity();+ try t.clickText("Allow");+ t.settle(0.2);+ try std.testing.expect(a.hub.policy.allow_input);+ try std.testing.expect(a.gate.request == null);+ try std.testing.expect(wire.saw("9:{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{"));+ try std.testing.expect(wire.saw("\"outcome\":\"allowed\""));+ try std.testing.expect(!t.hasText("A request from graff"));++ // ...but a program can turn things off, here and from outside.+ a.real_pointer = notReal;+ try t.clickText("Allow input: clicks, typing, keys, opening apps");+ t.settle(0.2);+ try std.testing.expect(!a.hub.policy.allow_input);+ a.real_pointer = null;+ a.setInput(true);+ a.setWholeScreen(true);+ control(&a, 4, "off", "{\"what\":\"whole_screen\"}");+ try std.testing.expect(!a.hub.policy.allow_whole_screen and a.hub.policy.allow_input);+ control(&a, 5, "off", "{\"what\":\"input\"}");+ try std.testing.expect(!a.hub.policy.allow_input);+ wire.sent.clearRetainingCapacity();+ control(&a, 6, "off", "{\"what\":\"resume\"}");+ try std.testing.expect(wire.saw("-32602"));++ // Stop works from outside; Resume is a question like the rest.+ control(&a, 7, "stop", "{}");+ try std.testing.expect(a.hub.policy.stopped);+ control(&a, 8, "ask", "{\"what\":\"resume\",\"from\":\"Harness\"}");+ try std.testing.expect(a.hub.policy.stopped);+ t.settle(0.2);+ try std.testing.expect(t.hasText("A request from Harness"));++ // Deny: whoever asked hears, and asking again is refused for a while.+ wire.sent.clearRetainingCapacity();+ try t.clickText("Deny");+ t.settle(0.2);+ try std.testing.expect(a.hub.policy.stopped);+ try std.testing.expect(wire.saw("\"outcome\":\"denied\""));+ wire.sent.clearRetainingCapacity();+ control(&a, 10, "ask", "{\"what\":\"resume\"}");+ try std.testing.expect(wire.saw("\"outcome\":\"quiet\""));++ // Unanswered, it goes away and says so. Already on needs no question.+ Wire.now += 20;+ a.setStopped(false);+ wire.sent.clearRetainingCapacity();+ control(&a, 11, "ask", "{\"what\":\"resume\"}");+ try std.testing.expect(wire.saw("\"outcome\":\"already_on\""));+ control(&a, 12, "ask", "{\"what\":\"input\",\"from\":\"graff\"}");+ Wire.now += 61;+ wire.sent.clearRetainingCapacity();+ t.frame();+ t.settle(0.2);+ try std.testing.expect(a.gate.request == null);+ try std.testing.expect(!a.hub.policy.allow_input);+ try std.testing.expect(wire.saw("9:{\"jsonrpc\":\"2.0\",\"id\":12,\"result\":{"));+ try std.testing.expect(wire.saw("\"outcome\":\"expired\""));+}++test "an agent's own connection can't reach the switches through MCP" {+ // `nanocua mcp` has a hub of its own with no control behind it.+ const gpa = std.testing.allocator;+ var wire: Wire = .{ .gpa = gpa };+ defer wire.sent.deinit(gpa);+ var rec: Recorder = .{};+ var h: mcp.Hub = .{ .gpa = gpa, .clock = Wire.clock, .sink = .{ .ptr = &wire, .send = Wire.send }, .backend = .{ .ptr = &rec, .call = Recorder.call } };+ h.handle(1, "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"nanocua/ask\",\"params\":{\"what\":\"input\"}}");+ try std.testing.expect(wire.saw("-32601"));+}+extern "c" fn socketpair(domain: c_int, kind: c_int, protocol: c_int, fds: *[2]c_int) c_int;extern "c" fn close(fd: c_int) c_int;diff --git a/apps/nanocua/src/tools.zig b/apps/nanocua/src/tools.zigindex 58e5f955..e85d2fcd 100644--- a/apps/nanocua/src/tools.zig+++ b/apps/nanocua/src/tools.zig@@ -52,9 +52,11 @@ pub const Tool = enum {\\need (canvases, images).,.click =>- \\Clicks in the app you act in, without moving the user's pointer: an element by ref (pressing it through- \\accessibility when it can, else at its center) or x,y in screen points. button: left (default) or right;- \\count: 2 for a double click. Some apps ignore background clicks: then use refs, or ask for whole_screen.+ \\Clicks in the app you act in, without moving the user's pointer: an element by ref, or x,y in screen+ \\points, inside that app's window. Controls are pressed through accessibility, which is exact. Anything+ \\else gets a click event that apps can ignore in the background, so the reply says it isn't confirmed:+ \\check the result. In a window that isn't its app's key window, that first click may only make it key.+ \\button: left (default) or right; count: 2 for a double click.,.type_text =>\\Types text into the app you act in, without taking the user's keyboard: into its focused element, or@@ -66,7 +68,7 @@ pub const Tool = enum {,.scroll =>\\Scrolls in the app you act in, at x,y (or over element ref), by dy lines (positive scrolls down) and dx- \\lines (positive scrolls right).+ \\lines (positive scrolls right). The reply says whether it moved, when the app has scroll bars to read.,.move_mouse =>\\Hovers at x,y in screen points in the app you act in. The user's pointer stays put unless whole_screen.diff --git a/apps/nanocua/src/tree.zig b/apps/nanocua/src/tree.zigindex 8a2fff36..53fed4d5 100644--- a/apps/nanocua/src/tree.zig+++ b/apps/nanocua/src/tree.zig@@ -32,6 +32,21 @@ fn bare(role: []const u8) []const u8 {return if (std.mem.startsWith(u8, role, "AX")) role[2..] else role;}+/// A role as it reads in a reply: "Button".+pub const bareRole = bare;++/// Roles an accessibility press does what a click would.+const pressed = [_][]const u8{ "Button", "CheckBox", "RadioButton", "PopUpButton", "MenuButton", "MenuItem", "MenuBarItem", "Link", "DisclosureTriangle", "Incrementor" };++pub fn pressable(role: []const u8) bool {+ return in(bare(role), &pressed);+}++/// Roles that are likely a control's label or icon, so its control is above.+pub fn insideControl(role: []const u8) bool {+ return in(bare(role), &.{ "StaticText", "Image" });+}+fn in(s: []const u8, list: []const []const u8) bool {for (list) |x| if (std.mem.eql(u8, s, x)) return true;return false;@@ -79,6 +94,13 @@ fn quoted(w: *std.Io.Writer, prefix: []const u8, s: []const u8, max: usize) !voitry w.writeByte('"');}+test "what a press reaches" {+ try std.testing.expect(pressable("AXButton") and pressable("AXLink"));+ try std.testing.expect(!pressable("AXTextArea") and !pressable("AXGroup"));+ try std.testing.expect(insideControl("AXStaticText") and !insideControl("AXTextArea"));+ try std.testing.expectEqualStrings("Button", bareRole("AXButton"));+}+test "lines name what can be acted on and skip plumbing" {var buf: [512]u8 = undefined;var w: std.Io.Writer = .fixed(&buf);diff --git a/apps/nanocua/src/view.zig b/apps/nanocua/src/view.zigindex cca14b88..3418aeda 100644--- a/apps/nanocua/src/view.zig+++ b/apps/nanocua/src/view.zig@@ -6,6 +6,7 @@ const nanoui = @import("nanoui");const ui = nanoui.ui;const theme = @import("theme.zig");const policy = @import("policy.zig");+const control = @import("control.zig");const App = @import("app.zig").App;const mac = @import("app.zig").mac;@@ -15,6 +16,7 @@ pub fn view(a: *App, c: *ui.Context) void {const page = ui.div(c).column().sizeFull().p(20).gap(14).bg(k.page);defer page.end();header(c, a, k);+ question(c, a, k);permissions(c, a, k);agents(c, a, k);activity(c, a, k);@@ -45,10 +47,35 @@ fn header(c: *ui.Context, a: *App, k: theme.Colors) void {ui.text(c, "Lets AI agents see and use this computer, as far as you allow here.", .{ .size = 12, .color = k.secondary, .wrap = true });}if (p.stopped) {- if (ui.button(c, "Resume", .{ .variant = .primary }).clicked) a.setStopped(false);+ if (ui.button(c, "Resume", .{ .variant = .primary }).clicked) a.fromPanel(.@"resume", true);} else if (ui.button(c, "Stop", .{ .variant = .danger }).clicked) a.setStopped(true);}+pub const screen_hint = "Turn on nanocua in the list System Settings opened. Not there? Drag nanocua.app from the Finder window into it.";++/// What a program asked for from outside. Only this click turns it on.+fn question(c: *ui.Context, a: *App, k: theme.Colors) void {+ const now = a.hub.clock();+ a.settleRequest();+ if (now < a.ignored_until) {+ c.frameIn(@floatCast(a.ignored_until - now));+ ui.text(c, ignored_click, .{ .size = 12, .color = k.red, .wrap = true });+ }+ const r = a.gate.request orelse return;+ const armed = a.gate.armed(now);+ c.frameIn(@floatCast(if (armed) r.at + control.lifetime - now else r.at + control.armed_after - now));+ const d = ui.div(c).column().wFull().p(14).gap(10).bg(k.card).border(k.accent).borderW(2).rounded(12);+ defer d.end();+ ui.text(c, c.print("A request from {s}", .{r.from()}), .{ .size = 13, .weight = .semibold, .color = k.accent });+ ui.text(c, r.want.question(), .{ .size = 14, .color = k.text, .wrap = true });+ const row = ui.div(c).row().itemsCenter().gap(8);+ defer row.end();+ if (ui.button(c, "Allow", .{ .variant = .primary, .disabled = !armed }).clicked) a.answer(true);+ if (ui.button(c, "Deny", .{}).clicked) a.answer(false);+}++pub const ignored_click = "That click came from a program, so it was ignored. Only your own mouse or trackpad turns things on.";+fn permissions(c: *ui.Context, a: *App, k: theme.Colors) void {if (comptime !mac) return;const d = card(c, k);@@ -56,6 +83,8 @@ fn permissions(c: *ui.Context, a: *App, k: theme.Colors) void {title(c, k, "Permissions");permission(c, a, k, "Accessibility", "To read apps' controls, click and type", a.perms.accessibility, .accessibility);permission(c, a, k, "Screen Recording", "For screenshots", a.perms.screen, .screen);+ if (a.asked_screen and !a.perms.screen)+ ui.text(c, screen_hint, .{ .size = 12, .color = k.accent, .wrap = true });}fn permission(c: *ui.Context, a: *App, k: theme.Colors, name: []const u8, what: []const u8, on: bool, which: App.Ask) void {@@ -77,9 +106,9 @@ fn agents(c: *ui.Context, a: *App, k: theme.Colors) void {defer d.end();title(c, k, "Agents");var input = a.hub.policy.allow_input;- if (ui.toggle(c, "Allow input: clicks, typing, keys, opening apps", &input)) a.setInput(input);+ if (ui.toggle(c, "Allow input: clicks, typing, keys, opening apps", &input)) a.fromPanel(.input, input);var whole = a.hub.policy.allow_whole_screen;- if (ui.toggle(c, "Allow whole screen: your pointer and keyboard, pictures of everything", &whole)) a.setWholeScreen(whole);+ if (ui.toggle(c, "Allow whole screen: your pointer and keyboard, pictures of everything", &whole)) a.fromPanel(.whole_screen, whole);ui.text(c, "Off, agents work inside one app at a time, in the background.", .{ .size = 12, .color = k.secondary, .wrap = true });if (a.listen_error) |e| {ui.text(c, e, .{ .size = 12, .color = k.red, .wrap = true });@@ -106,10 +135,12 @@ fn agents(c: *ui.Context, a: *App, k: theme.Colors) void {}}if (comptime mac) {- const r = ui.div(c).row().itemsCenter().gap(6);- defer r.end();- ui.text(c, "Stop everything from anywhere:", .{ .size = 12, .color = k.secondary });- ui.kbd(c, @import("mac/hotkey.zig").label);+ if (a.hotkey != null) {+ const r = ui.div(c).row().itemsCenter().gap(6);+ defer r.end();+ ui.text(c, "Stop everything from anywhere:", .{ .size = 12, .color = k.secondary });+ ui.kbd(c, @import("mac/hotkey.zig").label);+ } else ui.text(c, "The stop key couldn't be set up: use Stop above, or `graff computer stop`.", .{ .size = 12, .color = k.secondary, .wrap = true });}}diff --git a/docs/adr/0314-nanocua-is-grafts-computer-use.md b/docs/adr/0314-nanocua-is-grafts-computer-use.mdindex 17dd521b..3402e6ef 100644--- a/docs/adr/0314-nanocua-is-grafts-computer-use.md+++ b/docs/adr/0314-nanocua-is-grafts-computer-use.md@@ -41,6 +41,29 @@ codegraff holds the original. nanoui's `apps/nanocua` is generated from it byMIT. nanoui does not import it, so nanoui itself stays MIT. Linking the other wayis fine: `apps/nanocua` depends on nanoui, pinned, like any MIT library.+## Who changes the switches++`nanocua ctl` is the one way in from outside the panel. `graff computer` and+`/computer` run it, and Harness runs it directly, because an older graff would+take `graff computer` for a prompt. Status, stop and turning things off happen at+once, for anyone on the user's socket. Turning something on (input, whole screen,+resume) only asks: a question appears in nanocua's window, and only a person's+click on Allow there answers it.++That split exists because Harness runs graff with `--yolo` and agents have shells:+a command that turned input on would let an agent turn on its own input, so the+switches would protect against nothing. Allow arms after 0.7s, so a click meant+for what was under the pointer can't answer it. A question goes away after 60s,+and a Deny refuses new ones for 15s. nanocua checks that a click came from+hardware, not a program, and only for turning things on: Stop works however it's+clicked. Switch Control and Voice Control post their own clicks, so they can't+turn things on. nanoui's `--nanoui-*` flags are refused outside debug builds,+since they drive the same window.++A process running as the user can still attack this in ways nanocua can't see,+such as editing what the app loads. The check closes the easy route, not every one.+macOS's own permissions stay in System Settings; nothing here can flip them.+## Consequences- The desktop app runs `graff acp`, so it offers nanocua too once the app is in@@ -53,5 +76,6 @@ is fine: `apps/nanocua` depends on nanoui, pinned, like any MIT library.computer use. An ad hoc app would lose its permissions on every update.- Background input is delivered per process. Apps may ignore it, Chromium's webcontent most likely; agents fall back to refs or ask for the whole screen.-- Two agents share one app, one target and one set of refs, calls one at a time.+- Agents share one app, calls one at a time, but each connection has its own refs+ and target. Harness's graff chats and a terminal graff use the same app.- Beta releases are not signed, so they carry no app.diff --git a/docs/adr/README.md b/docs/adr/README.mdindex 14728301..aacd821a 100644--- a/docs/adr/README.md+++ b/docs/adr/README.md@@ -338,7 +338,7 @@ record only when you need the evidence or the edge cases.| [0311](0311-tool-refresh-and-addressed-wakes.md) | Claude Agent refreshes a changed tool list with MCP `tools/list_changed` (served before anything else; fresh session if not re-read in 15 s) instead of removing and re-adding its server, which failed held calls. `attempt_completion` settles a task like a final answer: room mail waits for the next human prompt, addressed mail can still start a turn; `yield_turn` stays strict. A peer wake preloads `peer_message`. || [0312](0312-openai-12h-prompt-cache-is-opt-in.md) | OpenAI Responses GPT-5.6+ can put a 12h explicit breakpoint on the stable prefix (`GRAFF_OPENAI_CACHE_TTL=12h` or `openai_cache_ttl`); the default stays 30m (ADR 0009 bytes) until a live probe confirms OpenAI accepts the field and keeps the entry. Usage parses cache writes by TTL. || [0313](0313-graff-does-not-load-codex-computer-use.md) | Graff does not load Codex's `computer-use` or `unified-computer-use` plugins; every MCP elicitation is declined. Supersedes 0036 and 0119. |-| [0314](0314-nanocua-is-grafts-computer-use.md) | Computer use on macOS is nanocua, a signed app shipped beside graff and offered as one MCP server; it works inside one app in the background, and the app's panel, not graff, says what it may do. nanoui's copy is generated from codegraff's and keeps codegraff's license. |+| [0314](0314-nanocua-is-grafts-computer-use.md) | Computer use on macOS is nanocua, a signed app shipped beside graff and offered as one MCP server; it works inside one app in the background, and the app's panel, not graff, says what it may do: graff and Harness can read its state and turn things off, but turning anything on takes a person's click in its window. nanoui's copy is generated from codegraff's and keeps codegraff's license. || [0302](0302-sse-idle-watchdogs-follow-body-byte-progress.md) | SSE idle watchdogs observe partial body-byte progress, while separate unfinished-event time and size bounds preserve recovery. |diff --git a/src/args.zig b/src/args.zigindex 2a718607..d8edfcc3 100644--- a/src/args.zig+++ b/src/args.zig@@ -123,7 +123,7 @@ pub fn parse(init: std.process.Init) !Flags {_ = it.next(); // argv[0]while (it.next()) |arg| {if (flags.positionals.items.len > 0 and- (std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "learn") or std.mem.eql(u8, flags.positionals.items[0], "peer") or std.mem.eql(u8, flags.positionals.items[0], "keys") or std.mem.eql(u8, flags.positionals.items[0], "harness-tree")))+ (std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "computer") or std.mem.eql(u8, flags.positionals.items[0], "learn") or std.mem.eql(u8, flags.positionals.items[0], "peer") or std.mem.eql(u8, flags.positionals.items[0], "keys") or std.mem.eql(u8, flags.positionals.items[0], "harness-tree"))){try flags.positionals.append(arena, try arena.dupe(u8, arg));} else if (std.mem.startsWith(u8, arg, "-")) {@@ -288,7 +288,7 @@ pub fn parse(init: std.process.Init) !Flags {// One-shot print mode: `harness -p "prompt"` or a bare positional prompt// (`harness "say hi"`). Subcommands (login/key) are not prompts.flags.is_subcommand = flags.positionals.items.len > 0 and- (std.mem.eql(u8, flags.positionals.items[0], "login") or std.mem.eql(u8, flags.positionals.items[0], "logout") or std.mem.eql(u8, flags.positionals.items[0], "key") or std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "learn") or+ (std.mem.eql(u8, flags.positionals.items[0], "login") or std.mem.eql(u8, flags.positionals.items[0], "computer") or std.mem.eql(u8, flags.positionals.items[0], "logout") or std.mem.eql(u8, flags.positionals.items[0], "key") or std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "learn") orstd.mem.eql(u8, flags.positionals.items[0], "serve") or std.mem.eql(u8, flags.positionals.items[0], "update") or std.mem.eql(u8, flags.positionals.items[0], "title") or std.mem.eql(u8, flags.positionals.items[0], "repl") or (!flags.print_flag and std.mem.eql(u8, flags.positionals.items[0], "tui")) orstd.mem.eql(u8, flags.positionals.items[0], "worktree") or std.mem.eql(u8, flags.positionals.items[0], "harness-tree") or std.mem.eql(u8, flags.positionals.items[0], "sandboxes") or std.mem.eql(u8, flags.positionals.items[0], "cube") or std.mem.eql(u8, flags.positionals.items[0], "models") orstd.mem.eql(u8, flags.positionals.items[0], "remote-control") or std.mem.eql(u8, flags.positionals.items[0], "remote") or @import("acp.zig").isAcpSubcommand(flags.positionals.items[0])); // `acp` also arms ACP's stdout discipline (json_mode) — see acp.isAcpSubcommanddiff --git a/src/cli_text_source.zig b/src/cli_text_source.zigindex b0c97e29..8d32603c 100644--- a/src/cli_text_source.zig+++ b/src/cli_text_source.zig@@ -334,6 +334,7 @@ pub const usage_text =\\ graff mcp install configure local HTTP service and MCP clients\\ graff mcp serve [--http] [--port N] expose run_task to MCP clients\\ graff plugins [load <name>] list Claude/Cursor/Grok/Codex plugin trees (in place)+ \\ graff computer [status|stop|off|allow|grant|open] nanocua's switches (macOS)\\++ learn_usage ++\\ graff worktree list list -w tabs and experiment-pool trees (tagged)diff --git a/src/command_catalog.zig b/src/command_catalog.zigindex 10c56269..1b2fe968 100644--- a/src/command_catalog.zig+++ b/src/command_catalog.zig@@ -128,6 +128,7 @@ pub const commands = [_]Item{.{ .name = "/theme", .usage = "/theme [name]", .desc = "pick a color theme; /theme off resets to your terminal default; persists" },.{ .name = "/fleet", .usage = "/fleet [on|off]", .desc = "federated DGM contribution (propose/submit/elite_pull)" },.{ .name = "/mcp", .usage = "/mcp [add … | reconnect <name>]", .desc = "list MCP servers/tools; /mcp add <name> <cmd> [args...] connects one live; /mcp reconnect <name> starts one again" },+ .{ .name = "/computer", .usage = "/computer [status | stop | off … | allow … | grant … | open]", .desc = "nanocua's switches: what's on, stop it, ask to turn things on (macOS)" },.{ .name = "/import-claude", .desc = "copy Claude/Cursor setup and import this project's Claude conversations for /resume" },.{ .name = "/help", .desc = "list every command" },};diff --git a/src/commands_misc.zig b/src/commands_misc.zigindex 846099b4..219d839e 100644--- a/src/commands_misc.zig+++ b/src/commands_misc.zig@@ -211,6 +211,10 @@ pub fn tryHandle(root: *Agent, keys: *Keys, arena: Allocator, line: []const u8,try out.flush();return true;}+ if (std.mem.eql(u8, line, "/computer") or std.mem.startsWith(u8, line, "/computer ")) {+ try @import("computer_cli.zig").slash(root.io, root.gpa, arena, root.home, line, out);+ return true;+ }if (std.mem.startsWith(u8, line, "/mcp")) {const arg = std.mem.trim(u8, line["/mcp".len..], " \t");const reg = root.registry.?; // always present nowdiff --git a/src/computer_cli.zig b/src/computer_cli.zignew file mode 100644index 00000000..72fe9c3c--- /dev/null+++ b/src/computer_cli.zig@@ -0,0 +1,204 @@+//! `graff computer` and `/computer` (ADR 0314): nanocua's switches from graff,+//! by running the app's own `nanocua ctl`. Status, stop and turning things off+//! happen at once. `allow` only asks: the question appears in nanocua's window+//! and only a person's click there answers it, because agents have shells and+//! could run this command as easily as anyone. macOS's permissions stay the+//! user's to flip in System Settings: `grant` opens the prompt.++const std = @import("std");+const builtin = @import("builtin");+const Io = std.Io;+const Allocator = std.mem.Allocator;+const nanocua_default = @import("nanocua_default.zig");+const process_runner = @import("process_runner.zig");++pub const usage =+ \\usage: graff computer [status] [--json]+ \\ status what's on, and what macOS has granted+ \\ stop refuse every agent's calls, at once+ \\ off <input|whole_screen> turn a switch off, at once+ \\ allow <input|whole_screen|resume> ask; you answer in nanocua's window+ \\ grant <accessibility|screen> open macOS's prompt for a permission+ \\ open bring nanocua's window forward+ \\+;++/// `allow` waits for the click: a question lasts a minute, and starting the+/// app may come first.+const deadline_ms: u64 = 90_000;+const output_cap: usize = 64 * 1024;++const not_installed_text = "nanocua isn't installed. It ships with graff's signed macOS releases: run `graff update`.\n";+const not_macos_text = "Computer use is macOS-only for now.\n";++pub const Parsed = struct {+ /// What goes after `nanocua ctl`.+ words: []const []const u8,+ json: bool = false,+ from: []const u8 = "graff",+};++/// Null asks for the usage. Words are checked here, so nothing but the commands+/// above ever reaches the app, and a mistake answers here without starting it.+pub fn parse(arena: Allocator, args: []const []const u8) !?Parsed {+ var out: Parsed = .{ .words = &.{} };+ var words: std.ArrayList([]const u8) = .empty;+ for (args) |arg| {+ if (std.mem.eql(u8, arg, "--json")) {+ out.json = true;+ } else if (std.mem.startsWith(u8, arg, "--from=")) {+ out.from = arg["--from=".len..];+ } else try words.append(arena, arg);+ }+ if (words.items.len == 0) try words.append(arena, "status");+ if (!known(words.items)) return null;+ out.words = words.items;+ return out;+}++fn oneOf(word: []const u8, comptime options: []const []const u8) bool {+ for (options) |o| if (std.mem.eql(u8, word, o)) return true;+ return false;+}++fn known(words: []const []const u8) bool {+ if (words.len == 1) return oneOf(words[0], &.{ "status", "stop", "open" });+ if (words.len != 2) return false;+ if (std.mem.eql(u8, words[0], "off")) return oneOf(words[1], &.{ "input", "whole_screen" });+ if (std.mem.eql(u8, words[0], "allow")) return oneOf(words[1], &.{ "input", "whole_screen", "resume" });+ if (std.mem.eql(u8, words[0], "grant")) return oneOf(words[1], &.{ "accessibility", "screen" });+ return false;+}++pub fn argv(arena: Allocator, program: []const u8, p: Parsed) ![]const []const u8 {+ var out: std.ArrayList([]const u8) = .empty;+ try out.appendSlice(arena, &.{ program, "ctl" });+ try out.appendSlice(arena, p.words);+ if (p.json) try out.append(arena, "--json");+ try out.append(arena, try std.fmt.allocPrint(arena, "--from={s}", .{p.from}));+ return out.items;+}++/// What's said when there's no app to ask.+fn absent(json: bool, supported: bool) []const u8 {+ if (json) return if (supported)+ "{\"installed\":false,\"running\":false}\n"+ else+ "{\"installed\":false,\"running\":false,\"outcome\":\"unsupported\"}\n";+ return if (supported) not_installed_text else not_macos_text;+}++/// True if the command ran; its output or the reason it didn't goes to `out`.+pub fn run(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, args: []const []const u8, out: *Io.Writer) !bool {+ const p = try parse(arena, args) orelse {+ try out.writeAll(usage);+ return false;+ };+ const supported = builtin.os.tag == .macos;+ const program = (if (supported) nanocua_default.find(io, arena, home) else null) orelse {+ try out.writeAll(absent(p.json, supported));+ return false;+ };+ if (!p.json and std.mem.eql(u8, p.words[0], "allow")) {+ try out.writeAll("Asking: answer in nanocua's window. The question lasts a minute.\n");+ try out.flush();+ }+ const r = process_runner.runCapped(gpa, io, try argv(arena, program, p), output_cap, output_cap, deadline_ms) catch |err| {+ try out.print("Couldn't run nanocua: {t}\n", .{err});+ return false;+ };+ defer gpa.free(r.stdout);+ defer gpa.free(r.stderr);+ try out.writeAll(r.stdout);+ if (r.timed_out) try out.writeAll("nanocua didn't answer in time.\n");+ const exited_ok = switch (r.term) {+ .exited => |code| code == 0,+ else => false,+ };+ if (!exited_ok and r.stderr.len > 0) try out.writeAll(r.stderr);+ return exited_ok and !r.timed_out;+}++/// `graff computer …`.+pub fn command(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, args: []const []const u8) !void {+ var buf: [4096]u8 = undefined;+ var w = Io.File.stdout().writer(io, &buf);+ const ok = try run(io, gpa, arena, home, args, &w.interface);+ try w.interface.flush();+ if (!ok) std.process.exit(1);+}++/// `/computer …`, on the REPL and the TUI.+pub fn slash(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, line: []const u8, out: *Io.Writer) !void {+ var words: std.ArrayList([]const u8) = .empty;+ var it = std.mem.tokenizeAny(u8, line["/computer".len..], " \t");+ while (it.next()) |word| try words.append(arena, word);+ _ = try run(io, gpa, arena, home, words.items, out);+ try out.flush();+}++const testing = std.testing;++test "only the commands nanocua knows get through" {+ var state = std.heap.ArenaAllocator.init(testing.allocator);+ defer state.deinit();+ const a = state.allocator();+ try testing.expectEqualStrings("status", (try parse(a, &.{})).?.words[0]);+ const allow = (try parse(a, &.{ "allow", "whole_screen", "--json", "--from=Harness" })).?;+ try testing.expect(allow.json);+ try testing.expectEqualStrings("Harness", allow.from);+ try testing.expectEqualStrings("whole_screen", allow.words[1]);+ for ([_][]const []const u8{+ &.{"nonsense"},+ &.{"allow"},+ &.{ "allow", "everything" },+ &.{ "off", "resume" },+ &.{ "grant", "camera" },+ &.{ "stop", "now" },+ &.{ "status", "extra", "words" },+ }) |bad| try testing.expect((try parse(a, bad)) == null);+}++test "what runs: the app's ctl, saying who asks" {+ var state = std.heap.ArenaAllocator.init(testing.allocator);+ defer state.deinit();+ const a = state.allocator();+ const got = try argv(a, "/x/nanocua", (try parse(a, &.{ "grant", "screen", "--json" })).?);+ const want = [_][]const u8{ "/x/nanocua", "ctl", "grant", "screen", "--json", "--from=graff" };+ try testing.expectEqual(want.len, got.len);+ for (want, got) |w, g| try testing.expectEqualStrings(w, g);+}++test "no app: a person is told how, a program reads that it's missing" {+ var tmp = testing.tmpDir(.{});+ defer tmp.cleanup();+ var state = std.heap.ArenaAllocator.init(testing.allocator);+ defer state.deinit();+ const a = state.allocator();+ var buf: [std.fs.max_path_bytes]u8 = undefined;+ const n = try tmp.dir.realPath(testing.io, &buf);+ const home = try a.dupe(u8, buf[0..n]);+ var aw: Io.Writer.Allocating = .init(a);+ try testing.expect(!try run(testing.io, testing.allocator, a, home, &.{}, &aw.writer));+ try testing.expect(std.mem.indexOf(u8, aw.written(), "isn't") != null or std.mem.indexOf(u8, aw.written(), "macOS-only") != null);+ aw.clearRetainingCapacity();+ try testing.expect(!try run(testing.io, testing.allocator, a, home, &.{"--json"}, &aw.writer));+ const parsed = try std.json.parseFromSliceLeaky(std.json.Value, a, aw.written(), .{});+ try testing.expect(!parsed.object.get("installed").?.bool);+ aw.clearRetainingCapacity();+ try testing.expect(!try run(testing.io, testing.allocator, a, home, &.{"nonsense"}, &aw.writer));+ try testing.expectEqualStrings(usage, aw.written());+}++test "/computer splits its words" {+ var tmp = testing.tmpDir(.{});+ defer tmp.cleanup();+ var state = std.heap.ArenaAllocator.init(testing.allocator);+ defer state.deinit();+ const a = state.allocator();+ var buf: [std.fs.max_path_bytes]u8 = undefined;+ const n = try tmp.dir.realPath(testing.io, &buf);+ var aw: Io.Writer.Allocating = .init(a);+ try slash(testing.io, testing.allocator, a, try a.dupe(u8, buf[0..n]), "/computer allow \t nonsense", &aw.writer);+ try testing.expectEqualStrings(usage, aw.written());+}diff --git a/src/help.zig b/src/help.zigindex 5d9a4b67..14c436cb 100644--- a/src/help.zig+++ b/src/help.zig@@ -32,7 +32,7 @@ pub const sections = [_]Section{.{ .title = "the model", .names = &.{ "/model", "/models", "/effort", "/reasoning", "/fast", "/thinking", "/keepcontext", "/fallback", "/routes" } },.{ .title = "working autonomously", .names = &.{ "/goal", "/goals", "/schedule", "/review", "/issue", "/plan", "/todo", "/jobs", "/workers", "/ultracode", "/strict", "/yolo", "/never" } },.{ .title = "talking to other graffs", .names = &.{ "/tell", "/daddy", "/peek", "/adapter" }, .blurb = peers_blurb },- .{ .title = "your setup", .names = &.{ "/login", "/key", "/cost", "/usage", "/version", "/update", "/privacy", "/mcp", "/import-claude", "/skills", "/plugins", "/agents", "/hooks", "/tools", "/fleet" } },+ .{ .title = "your setup", .names = &.{ "/login", "/key", "/cost", "/usage", "/version", "/update", "/privacy", "/mcp", "/computer", "/import-claude", "/skills", "/plugins", "/agents", "/hooks", "/tools", "/fleet" } },.{ .title = "context & history", .names = &.{ "/compact", "/compact-at", "/btw", "/doctor", "/debug", "/cache", "/trace", "/trajectory" } },.{ .title = "shell & images", .names = &.{ "/bash", "/image", "/images", "/paste" } },.{ .title = "look & feel", .names = &.{ "/theme", "/animation", "/title" } },diff --git a/src/nanocua_default.zig b/src/nanocua_default.zigindex 077fe14a..b408b35a 100644--- a/src/nanocua_default.zig+++ b/src/nanocua_default.zig@@ -12,6 +12,25 @@ const Allocator = std.mem.Allocator;pub const server_name = "nanocua";const executable = "nanocua.app/Contents/MacOS/nanocua";+/// Whether `home` is `real` or a folder inside it, as an account's HOME under+/// Harness is. A $HOME anywhere else is a sandbox, and stays one.+pub fn within(home: []const u8, real: []const u8) bool {+ const h = std.mem.trimEnd(u8, home, "/");+ const r = std.mem.trimEnd(u8, real, "/");+ if (r.len == 0 or !std.mem.startsWith(u8, h, r)) return false;+ return h.len == r.len or h[r.len] == '/';+}++/// The home the one app on this Mac is under: the user's real one when $HOME is+/// inside it. A sandbox's HOME is used as it is, so a test running graff can't+/// find or replace the app installed on the machine running it.+pub fn userHome(home: []const u8) []const u8 {+ if (builtin.is_test or builtin.os.tag != .macos) return home;+ const user = std.c.getpwuid(std.c.getuid()) orelse return home;+ const real = std.mem.span(user.dir orelse return home);+ return if (within(home, real)) real else home;+}+/// Where install.sh and `graff update` put the app.pub fn installRoot(arena: Allocator, home: []const u8) ?[]const u8 {if (home.len == 0) return null;@@ -24,11 +43,14 @@ fn inRoot(io: Io, arena: Allocator, root: []const u8) ?[]const u8 {return path;}-/// The shipped app's program. Tests look only under their own home, so an app-/// installed on the machine running them doesn't change what they see.+/// The shipped app's program. Unit tests and a sandbox's HOME look only under+/// that home, so an app installed on the machine doesn't change what they see.pub fn find(io: Io, arena: Allocator, home: []const u8) ?[]const u8 {if (installRoot(arena, home)) |root| if (inRoot(io, arena, root)) |p| return p;if (builtin.is_test) return null;+ const user = userHome(home);+ if (!within(home, user)) return null;+ if (installRoot(arena, user)) |root| if (inRoot(io, arena, root)) |p| return p;if (inRoot(io, arena, "/Applications")) |p| return p;// An unpacked release tarball, which has it beside graff.const exe = std.process.executablePathAlloc(io, arena) catch return null;@@ -75,6 +97,16 @@ fn tmpHome(io: Io, arena: Allocator, tmp: *testing.TmpDir, with_app: bool) ![]coreturn home;}+test "only a home inside the user's own leads back to it" {+ try testing.expect(within("/Users/ada", "/Users/ada"));+ try testing.expect(within("/Users/ada/", "/Users/ada"));+ try testing.expect(within("/Users/ada/.harness/accounts/two", "/Users/ada"));+ try testing.expect(!within("/Users/adam", "/Users/ada"));+ try testing.expect(!within("/tmp/sandbox", "/Users/ada"));+ try testing.expect(!within("/var/folders/x/T/home", "/Users/ada"));+ try testing.expect(!within("/Users/ada", ""));+}+test "the shipped app becomes the nanocua server" {var tmp = testing.tmpDir(.{});defer tmp.cleanup();diff --git a/src/repl_model_commands.zig b/src/repl_model_commands.zigindex 8cfd0adb..61fd948d 100644--- a/src/repl_model_commands.zig+++ b/src/repl_model_commands.zig@@ -111,6 +111,8 @@ pub fn runCommand(self: *Model, line: []const u8) void {self.push(.info, "running shell commands needs the agent loop — use the main `graff` session, or your shell") catch {};} else if (std.mem.eql(u8, cmd, "/mcp")) {self.push(.info, "MCP servers attach to the agent loop, not the chat repl") catch {};+ } else if (std.mem.eql(u8, cmd, "/computer")) {+ self.push(.info, "computer use attaches to the agent loop, not the chat repl; `graff computer` works from a shell") catch {};} else if (std.mem.eql(u8, cmd, "/skills")) {self.push(.info, "skills (codedb, zigrep, …) attach to the agent loop, not the chat repl") catch {};} else if (std.mem.eql(u8, cmd, "/agents")) {diff --git a/src/startup.zig b/src/startup.zigindex 8f7e5c5a..1756aeab 100644--- a/src/startup.zig+++ b/src/startup.zig@@ -318,6 +318,12 @@ pub fn runSubcommand(io: Io, gpa: Allocator, arena: Allocator, init: std.processreturn true;}+ if (flags.positionals.items.len > 0 and std.mem.eql(u8, flags.positionals.items[0], "computer")) {+ const home = keys_cli.homeEnv(init.environ_map) orelse "";+ try @import("computer_cli.zig").command(io, gpa, arena, home, flags.positionals.items[1..]);+ return true;+ }+if (flags.positionals.items.len > 0 and std.mem.eql(u8, flags.positionals.items[0], "plugins")) {const home = keys_cli.homeEnv(init.environ_map) orelse "";plugins.applyEnv(init.environ_map);diff --git a/src/test_hooks.zig b/src/test_hooks.zigindex 15ce902c..0743914b 100644--- a/src/test_hooks.zig+++ b/src/test_hooks.zig@@ -404,6 +404,7 @@ test {_ = @import("update_windows.zig");_ = @import("mcp_elicitation.zig");_ = @import("nanocua_default.zig");+ _ = @import("computer_cli.zig");// POSIX file modes; only macOS installs it, Linux runs its tests too.if (@import("builtin").os.tag != .windows) _ = @import("nanocua_install.zig");_ = @import("edit_worktree_tests.zig"); // #747: edit lands on the selected worktreediff --git a/src/update_service.zig b/src/update_service.zigindex 717ebfab..0182ec92 100644--- a/src/update_service.zig+++ b/src/update_service.zig@@ -308,7 +308,7 @@ pub fn install(opts: Opts, consent: Consent) Report {// The app that ships with graff (ADR 0314). It has its own uses, so failing// to place it doesn't stop graff updating.- if (builtin.os.tag == .macos) _ = @import("nanocua_install.zig").fromArchive(opts.io, opts.gpa, opts.home, tarball) catch {};+ if (builtin.os.tag == .macos) _ = @import("nanocua_install.zig").fromArchive(opts.io, opts.gpa, @import("nanocua_default.zig").userHome(opts.home), tarball) catch {};if (existingMatches(opts.io, opts.gpa, opts.target_path, bin)) {writeInstalledVersion(opts.io, opts.target_path, version_status.stripV(tag_copy));