Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

Let graff and Harness show and stop computer use

Rach Pradhanf34f61f4fbd0
diff --git a/apps/nanocua/README.md b/apps/nanocua/README.md
index 03a8e792..df6c8730 100644
--- a/apps/nanocua/README.md
+++ b/apps/nanocua/README.md
@@ -19,6 +19,10 @@ agent ──stdio──▶ nanocua mcp ──unix socket──▶ nanocua (the a
accessibility, input, screenshots
```
+Everything goes to the one app, so a terminal `graff`, Harness's graff chats and
+any other agent can use it at the same time. Calls run one at a time, and each
+agent keeps its own refs and target app.
+
The app does all the work, and the agent's `nanocua mcp` only carries lines.
That matters on macOS: permissions (Accessibility, Screen Recording) belong
to the app that asks, so you grant them to nanocua once, not to each terminal
@@ -69,10 +73,9 @@ brings the app forward.
## What you control
-- **Stop** (or **⌃⌥⌘.** from anywhere) refuses every call, even looking,
- including calls already waiting, and stops text being typed, until you press
- Resume. An agent may be moving the pointer, which is why the keyboard shortcut
- exists.
+- **Stop** refuses every call, even looking, including calls already waiting, and
+ stops text being typed, until you press Resume. An agent may be moving the
+ pointer, so it's also on a key, **⌃⌥⌘.**, and outside the panel (below).
- **Allow input** is off at launch: agents can look but not click, type or
open apps until you turn it on.
- **Allow whole screen** is off at launch too: agents stay inside one app.
@@ -84,6 +87,34 @@ brings the app forward.
Calls run one at a time on a queue of their own, so a hung app never
freezes the panel (accessibility calls give up after two seconds).
+## From graff and Harness
+
+The switches also show in `graff computer`, `/computer` and Harness's MCP servers
+settings page. All of them run the app's `nanocua ctl`:
+
+```sh
+nanocua ctl status [--json] what's on, what macOS has granted
+nanocua ctl stop refuse every call, at once
+nanocua ctl off input turn a switch off, at once
+nanocua ctl allow input ask; you answer in nanocua's window
+nanocua ctl grant screen open macOS's prompt
+nanocua ctl open bring the window forward
+```
+
+Anything can stop nanocua or turn things off. **Nothing outside the window can
+turn things on.** `allow` only puts a question in nanocua's window, and it takes a
+click there, from your own mouse or trackpad. Agents have shells and could run
+this command too; if it turned input on, the switch would protect you from nothing.
+Allow arms after 0.7s, so a click meant for something else can't answer it. A
+question goes away after a minute, and a Deny refuses new ones for 15 seconds.
+
+The same check covers the panel's own switches and Resume, on the way on only.
+Switch Control and Voice Control post their own clicks, so they can turn things
+off here but not on. Release builds refuse nanoui's `--nanoui-*` flags, which
+would otherwise drive this window from a command line. None of this stops a
+determined program running as you, which could tamper with the app itself: it
+closes the easy route. macOS's own permissions stay in System Settings.
+
## Adding it to an agent
With `graff` there's nothing to add: on macOS its installer puts nanocua in
@@ -116,7 +147,8 @@ sh macos/package.sh aarch64-macos /tmp # /tmp/nanocua.app
Nothing in the tests moves the pointer, types or captures the screen: the policy,
where calls go, MCP sessions, keys, the tree's text, the socket (over a socket
-pair) and the panel are tested; the macOS backend is compiled, not run.
+pair), the switches from outside and the panel are tested; the macOS backend is
+compiled, not run.
macOS keeps permissions across updates only for an app signed by the same
identity. Set `CODESIGN_IDENTITY` for `package.sh`; an ad hoc build is asked again
diff --git a/apps/nanocua/src/app.zig b/apps/nanocua/src/app.zig
index 1d4f1cdb..1bef9d94 100644
--- a/apps/nanocua/src/app.zig
+++ b/apps/nanocua/src/app.zig
@@ -1,11 +1,14 @@
//! nanocua's state: the hub agents talk to, the socket it listens on, the
-//! backend that runs the tools, the kill switch and its hot key, and what
-//! the panel shows (permissions, the command for agents, the log).
+//! backend that runs the tools, the kill switch and its hot key, the switches
+//! as graff and Harness see them (control.zig), and what the panel shows
+//! (permissions, the command for agents, the log).
const std = @import("std");
const builtin = @import("builtin");
const nanoui = @import("nanoui");
const mcp = @import("mcp.zig");
const server_mod = @import("server.zig");
+const control = @import("control.zig");
+const tools = @import("tools.zig");
pub const mac = builtin.os.tag == .macos;
const backend_mod = if (mac) @import("mac/backend.zig") else @import("backend_none.zig");
@@ -37,6 +40,21 @@ pub const App = struct {
perms: Perms = .{},
read_perms: ?*const fn () Perms = null,
ask: ?*const fn (which: Ask) void = null,
+ /// The user asked for Screen Recording: the panel says what's left to do.
+ asked_screen: bool = false,
+ /// The question a program asked, waiting in the panel.
+ gate: control.Gate = .{},
+ /// Whether the click being handled came from a person (none in tests:
+ /// every click does). Turning something on needs one.
+ real_pointer: ?*const fn () bool = null,
+ /// Running under `--nanoui-hidden`: nothing comes forward, whoever asks.
+ hidden: bool = false,
+ /// Keeps the window above others while a question waits, or stops.
+ raise: ?*const fn (window: *nanoui.Window, on: bool) void = null,
+ /// A click that wasn't real was just ignored: the panel says so until then.
+ ignored_until: f64 = 0,
+ /// macOS's prompts open no more often than this.
+ grant_after: f64 = 0,
/// "Copied" shows for a while after copying the command.
copied_until: f64 = 0,
/// Light or dark, fixed (tests); else as the system is.
@@ -53,6 +71,7 @@ pub const App = struct {
};
a.hub.sink = a.server.sink();
a.hub.changed = .{ .ptr = a, .func = changed };
+ a.hub.control = .{ .ptr = a, .func = onControl };
}
pub fn deinit(a: *App) void {
@@ -110,11 +129,168 @@ pub const App = struct {
a.redraw();
}
+ /// Turning something on from the panel: only a person's click counts.
+ /// Turning it off always does.
+ pub fn fromPanel(a: *App, want: control.Want, on: bool) void {
+ if (on and !a.clickIsReal()) return;
+ switch (want) {
+ .input => a.setInput(on),
+ .whole_screen => a.setWholeScreen(on),
+ .@"resume" => a.setStopped(!on),
+ }
+ }
+
+ fn clickIsReal(a: *App) bool {
+ const real = if (a.real_pointer) |f| f() else true;
+ if (!real) a.ignored_until = a.hub.clock() + 5;
+ return real;
+ }
+
+ pub fn status(a: *App, outcome: ?[]const u8) control.Status {
+ a.refreshPerms();
+ const p = a.hub.policy;
+ return .{
+ .stopped = p.stopped,
+ .input = p.allow_input,
+ .wholeScreen = p.allow_whole_screen,
+ .accessibility = a.perms.accessibility,
+ .screenRecording = a.perms.screen,
+ .agents = a.server.clients(),
+ .calls = a.hub.log.calls,
+ .stopKey = a.hotkey != null,
+ .pending = if (a.gate.request) |r| @tagName(r.want) else null,
+ .outcome = outcome,
+ };
+ }
+
+ fn isOn(a: *const App, want: control.Want) bool {
+ const p = a.hub.policy;
+ return switch (want) {
+ .input => p.allow_input,
+ .whole_screen => p.allow_whole_screen,
+ .@"resume" => !p.stopped,
+ };
+ }
+
+ fn reply(a: *App, conn: u64, id: []const u8, outcome: ?[]const u8) void {
+ const line = control.statusJson(a.gpa, a.status(outcome)) catch return;
+ defer a.gpa.free(line);
+ a.hub.sendResult(conn, id, line);
+ }
+
+ fn onControl(ptr: *anyopaque, conn: u64, id: []const u8, method: []const u8, params: ?std.json.Value) void {
+ const a: *App = @ptrCast(@alignCast(ptr));
+ a.server.markControl(conn);
+ const args = tools.Args.of(params);
+ const eql = std.mem.eql;
+ const name = method["nanocua/".len..];
+ if (eql(u8, name, "status")) {
+ a.settleRequest();
+ return a.reply(conn, id, null);
+ }
+ if (eql(u8, name, "stop")) {
+ a.setStopped(true);
+ return a.reply(conn, id, null);
+ }
+ if (eql(u8, name, "off")) {
+ const what = control.Want.parse(args.string("what") orelse "") orelse .@"resume";
+ switch (what) {
+ .input => a.setInput(false),
+ .whole_screen => a.setWholeScreen(false),
+ // Off means off: nothing here can turn anything on.
+ .@"resume" => return a.hub.sendError(conn, id, -32602, "Say input or whole_screen"),
+ }
+ return a.reply(conn, id, null);
+ }
+ if (eql(u8, name, "ask")) return a.onAsk(conn, id, args);
+ if (eql(u8, name, "grant")) return a.onGrant(conn, id, args);
+ if (eql(u8, name, "open")) {
+ a.bringForward(false);
+ return a.reply(conn, id, null);
+ }
+ a.hub.sendError(conn, id, -32601, "Method not found");
+ }
+
+ fn onAsk(a: *App, conn: u64, id: []const u8, args: tools.Args) void {
+ const want = control.Want.parse(args.string("what") orelse "") orelse
+ return a.hub.sendError(conn, id, -32602, "Say input, whole_screen or resume");
+ if (a.isOn(want)) return a.reply(conn, id, "already_on");
+ a.settleRequest();
+ switch (a.gate.ask(want, args.string("from") orelse "", conn, id, a.hub.clock())) {
+ .busy => a.reply(conn, id, "busy"),
+ .quiet => a.reply(conn, id, "quiet"),
+ // The answer comes when the user clicks (answer).
+ .waiting => {
+ a.bringForward(true);
+ a.redraw();
+ },
+ }
+ }
+
+ fn onGrant(a: *App, conn: u64, id: []const u8, args: tools.Args) void {
+ const which = args.string("which") orelse "";
+ const ask_for: Ask = if (std.mem.eql(u8, which, "accessibility"))
+ .accessibility
+ else if (std.mem.eql(u8, which, "screen"))
+ .screen
+ else
+ return a.hub.sendError(conn, id, -32602, "Say accessibility or screen");
+ a.refreshPerms();
+ const has = if (ask_for == .accessibility) a.perms.accessibility else a.perms.screen;
+ if (has) return a.reply(conn, id, "already_allowed");
+ const now = a.hub.clock();
+ if (now < a.grant_after) return a.reply(conn, id, "quiet");
+ a.grant_after = now + 5;
+ a.request(ask_for);
+ a.bringForward(false);
+ a.reply(conn, id, "opened");
+ }
+
+ fn bringForward(a: *App, stay_above: bool) void {
+ if (a.hidden) return;
+ const w = a.window orelse return;
+ if (stay_above) if (a.raise) |f| f(w, true);
+ w.focus();
+ }
+
+ /// The panel's Allow or Deny. Allow needs a person's click, and the question
+ /// to have been on screen a moment.
+ pub fn answer(a: *App, allow: bool) void {
+ const now = a.hub.clock();
+ if (allow) {
+ if (!a.gate.armed(now) or !a.clickIsReal()) return;
+ const r = a.gate.take() orelse return;
+ switch (r.want) {
+ .input => a.setInput(true),
+ .whole_screen => a.setWholeScreen(true),
+ .@"resume" => a.setStopped(false),
+ }
+ a.finishRequest(r, "allowed");
+ } else {
+ const r = a.gate.deny(now) orelse return;
+ a.finishRequest(r, "denied");
+ }
+ }
+
+ /// A question nobody answered goes away, and whoever asked is told.
+ pub fn settleRequest(a: *App) void {
+ if (!a.gate.expired(a.hub.clock())) return;
+ const r = a.gate.take() orelse return;
+ a.finishRequest(r, "expired");
+ }
+
+ fn finishRequest(a: *App, r: control.Request, outcome: []const u8) void {
+ if (!a.hidden) if (a.window) |w| if (a.raise) |f| f(w, false);
+ a.reply(r.conn, r.id(), outcome);
+ a.redraw();
+ }
+
pub fn refreshPerms(a: *App) void {
if (a.read_perms) |f| a.perms = f();
}
pub fn request(a: *App, which: Ask) void {
+ if (which == .screen) a.asked_screen = true;
if (a.ask) |f| f(which);
}
@@ -144,6 +320,8 @@ pub const Platform = struct {
a.whole_flag = &p.impl.whole_screen;
a.read_perms = readPerms;
a.ask = ask;
+ a.real_pointer = @import("mac/pointer.zig").real;
+ a.raise = backend_mod.raise;
}
}
diff --git a/apps/nanocua/src/control.zig b/apps/nanocua/src/control.zig
new file mode 100644
index 00000000..fbe90ea4
--- /dev/null
+++ b/apps/nanocua/src/control.zig
@@ -0,0 +1,192 @@
+//! The switches from outside the panel, so graff and Harness can show them.
+//! Anything on the user's socket may turn something off or read the state.
+//! Nothing but a real click in nanocua's own window turns something on: a
+//! program only asks, and the question waits in the panel. Agents have shells,
+//! so a command that turned input on would let them turn on their own input.
+const std = @import("std");
+
+pub const Want = enum {
+ input,
+ whole_screen,
+ @"resume",
+
+ pub fn parse(s: []const u8) ?Want {
+ return std.meta.stringToEnum(Want, s);
+ }
+
+ pub fn question(w: Want) []const u8 {
+ return switch (w) {
+ .input => "Turn on input? Agents could click, type, press keys and open apps, one app at a time.",
+ .whole_screen => "Turn on the whole screen? Agents could use your pointer and keyboard, and take pictures of everything.",
+ .@"resume" => "Resume? Agents' calls run again, as far as the switches below allow.",
+ };
+ }
+};
+
+/// Allow does nothing this long after the question appears, so a click meant
+/// for whatever was under the pointer can't answer it.
+pub const armed_after: f64 = 0.7;
+/// An unanswered question goes away: it can't be answered by accident later.
+pub const lifetime: f64 = 60;
+/// After a Deny, new questions are refused this long, so nothing can keep
+/// raising the window until the user gives in.
+pub const quiet_after_deny: f64 = 15;
+
+/// One question, and who is waiting for its answer.
+pub const Request = struct {
+ want: Want,
+ at: f64,
+ conn: u64,
+ from_buf: [32]u8 = undefined,
+ from_len: u8 = 0,
+ id_buf: [64]u8 = undefined,
+ id_len: u8 = 0,
+
+ /// Who says it's asking. Programs name themselves: this isn't proof.
+ pub fn from(r: *const Request) []const u8 {
+ return r.from_buf[0..r.from_len];
+ }
+
+ pub fn id(r: *const Request) []const u8 {
+ return r.id_buf[0..r.id_len];
+ }
+};
+
+pub const Gate = struct {
+ request: ?Request = null,
+ quiet_until: f64 = 0,
+
+ pub const Asked = enum { waiting, busy, quiet };
+
+ pub fn ask(g: *Gate, want: Want, from: []const u8, conn: u64, id: []const u8, now: f64) Asked {
+ if (g.request != null) return if (g.expired(now)) .busy else .busy;
+ if (now < g.quiet_until) return .quiet;
+ if (id.len > 64) return .busy;
+ var r: Request = .{ .want = want, .at = now, .conn = conn };
+ @memcpy(r.id_buf[0..id.len], id);
+ r.id_len = @intCast(id.len);
+ // Shown to the user: plain characters only, and never empty.
+ for (from[0..@min(from.len, r.from_buf.len)]) |ch| {
+ if (ch < 0x20 or ch > 0x7e) continue;
+ r.from_buf[r.from_len] = ch;
+ r.from_len += 1;
+ }
+ if (r.from_len == 0) {
+ const unknown = "A program";
+ @memcpy(r.from_buf[0..unknown.len], unknown);
+ r.from_len = unknown.len;
+ }
+ g.request = r;
+ return .waiting;
+ }
+
+ pub fn armed(g: *const Gate, now: f64) bool {
+ const r = g.request orelse return false;
+ return now - r.at >= armed_after;
+ }
+
+ pub fn expired(g: *const Gate, now: f64) bool {
+ const r = g.request orelse return false;
+ return now - r.at >= lifetime;
+ }
+
+ /// The question, answered with Allow or gone away.
+ pub fn take(g: *Gate) ?Request {
+ const r = g.request;
+ g.request = null;
+ return r;
+ }
+
+ pub fn deny(g: *Gate, now: f64) ?Request {
+ g.quiet_until = now + quiet_after_deny;
+ return g.take();
+ }
+};
+
+/// What graff and Harness read: the reply to every control call. Its field
+/// names are the wire format.
+pub const Status = struct {
+ installed: bool = true,
+ running: bool = true,
+ stopped: bool = false,
+ input: bool = false,
+ wholeScreen: bool = false,
+ accessibility: bool = false,
+ screenRecording: bool = false,
+ agents: usize = 0,
+ calls: u64 = 0,
+ /// Whether ⌃⌥⌘. was registered.
+ stopKey: bool = false,
+ /// What's waiting for an answer in the panel.
+ pending: ?[]const u8 = null,
+ /// How this call went: allowed, denied, expired, busy, quiet, already_on,
+ /// opened, already_allowed, not_running, unreachable.
+ outcome: ?[]const u8 = null,
+};
+
+pub fn statusJson(a: std.mem.Allocator, s: Status) ![]u8 {
+ return std.json.Stringify.valueAlloc(a, s, .{});
+}
+
+/// nanoui's `--nanoui-*` flags drive a window from the command line, which is
+/// for tests. In a release they'd be a way round the panel: `nanocua
+/// --nanoui-play=…` clicks its own switches.
+pub fn devFlagRefused(mode: std.lang.Optimize, arg: []const u8) bool {
+ return mode != .debug and std.mem.startsWith(u8, arg, "--nanoui-");
+}
+
+test "a question: asked once, armed after a moment, gone after a while" {
+ var g: Gate = .{};
+ try std.testing.expect(!g.armed(10));
+ try std.testing.expectEqual(Gate.Asked.waiting, g.ask(.input, "graff", 7, "\"a\"", 10));
+ try std.testing.expectEqualStrings("graff", g.request.?.from());
+ try std.testing.expectEqualStrings("\"a\"", g.request.?.id());
+ try std.testing.expectEqual(@as(u64, 7), g.request.?.conn);
+ try std.testing.expect(!g.armed(10.5));
+ try std.testing.expect(g.armed(10.71));
+ try std.testing.expectEqual(Gate.Asked.busy, g.ask(.whole_screen, "other", 8, "1", 10.8));
+ try std.testing.expectEqual(Want.input, g.request.?.want);
+ try std.testing.expect(!g.expired(69));
+ try std.testing.expect(g.expired(70));
+ try std.testing.expectEqual(Want.input, g.take().?.want);
+ try std.testing.expect(g.take() == null);
+}
+
+test "a deny keeps new questions away for a while" {
+ var g: Gate = .{};
+ _ = g.ask(.input, "x", 1, "1", 0);
+ try std.testing.expect(g.deny(1) != null);
+ try std.testing.expectEqual(Gate.Asked.quiet, g.ask(.input, "x", 1, "2", 15));
+ try std.testing.expectEqual(Gate.Asked.waiting, g.ask(.input, "x", 1, "2", 16));
+}
+
+test "who's asking is cut to plain characters" {
+ var g: Gate = .{};
+ _ = g.ask(.input, "gr\x1baff\xc3\xa9", 1, "1", 0);
+ try std.testing.expectEqualStrings("graff", g.request.?.from());
+ _ = g.take();
+ _ = g.ask(.input, "\x01", 1, "1", 0);
+ try std.testing.expectEqualStrings("A program", g.request.?.from());
+ _ = g.take();
+ const long: [80]u8 = @splat('x');
+ _ = g.ask(.input, &long, 1, "1", 0);
+ try std.testing.expectEqual(@as(usize, 32), g.request.?.from().len);
+}
+
+test "the status is the wire format" {
+ const gpa = std.testing.allocator;
+ const out = try statusJson(gpa, .{ .input = true, .pending = "resume" });
+ defer gpa.free(out);
+ const parsed = try std.json.parseFromSlice(Status, gpa, out, .{});
+ defer parsed.deinit();
+ try std.testing.expect(parsed.value.input and !parsed.value.wholeScreen);
+ try std.testing.expectEqualStrings("resume", parsed.value.pending.?);
+ try std.testing.expect(std.mem.indexOf(u8, out, "\"wholeScreen\":false") != null);
+}
+
+test "dev flags only in debug builds" {
+ try std.testing.expect(!devFlagRefused(.debug, "--nanoui-play=click:Stop"));
+ try std.testing.expect(devFlagRefused(.fast, "--nanoui-play=click:Stop"));
+ try std.testing.expect(devFlagRefused(.safe, "--nanoui-script=x"));
+ try std.testing.expect(!devFlagRefused(.fast, "mcp"));
+}
diff --git a/apps/nanocua/src/ctl.zig b/apps/nanocua/src/ctl.zig
new file mode 100644
index 00000000..bed0b092
--- /dev/null
+++ b/apps/nanocua/src/ctl.zig
@@ -0,0 +1,196 @@
+//! `nanocua ctl`: the switches from a command line, for `graff computer` and
+//! Harness. It reads the state and turns things off at once. `allow` only asks:
+//! the question appears in nanocua's window and this waits for the click.
+const std = @import("std");
+const control = @import("control.zig");
+const proxy = @import("proxy.zig");
+const server = @import("server.zig");
+const sockpath = @import("sockpath.zig");
+
+extern "c" fn close(fd: c_int) c_int;
+
+pub const usage =
+ \\usage: nanocua ctl <command> [--json] [--from=NAME]
+ \\ status what's on, and what macOS has granted
+ \\ stop refuse every call, at once
+ \\ off <input|whole_screen> turn a switch off, at once
+ \\ allow <input|whole_screen|resume> ask; you answer in nanocua's window
+ \\ grant <accessibility|screen> open macOS's prompt for a permission
+ \\ open bring nanocua's window forward
+ \\
+;
+
+/// How long `allow` waits for the click: a little past the question's own end.
+const answer_ms: i32 = @intFromFloat((control.lifetime + 5) * 1000);
+const reply_ms: i32 = 5_000;
+
+pub const Command = struct {
+ method: []const u8,
+ /// Its `params`, as JSON.
+ params: []const u8 = "{}",
+ /// Starts the app if it isn't running; else there's nothing to do.
+ opens_app: bool = false,
+ wait_ms: i32 = reply_ms,
+};
+
+/// What a command line asks for; null if it makes no sense.
+pub fn parse(a: std.mem.Allocator, args: []const []const u8, from: []const u8) !?Command {
+ if (args.len == 0) return null;
+ const verb = args[0];
+ const object = if (args.len > 1) args[1] else "";
+ if (std.mem.eql(u8, verb, "status") and args.len == 1) return .{ .method = "nanocua/status" };
+ if (std.mem.eql(u8, verb, "stop") and args.len == 1) return .{ .method = "nanocua/stop" };
+ if (std.mem.eql(u8, verb, "open") and args.len == 1) return .{ .method = "nanocua/open", .opens_app = true };
+ if (args.len != 2) return null;
+ if (std.mem.eql(u8, verb, "off")) {
+ const w = control.Want.parse(object) orelse return null;
+ if (w == .@"resume") return null;
+ return .{ .method = "nanocua/off", .params = try paramsJson(a, .{ .what = object }) };
+ }
+ if (std.mem.eql(u8, verb, "allow")) {
+ _ = control.Want.parse(object) orelse return null;
+ return .{ .method = "nanocua/ask", .params = try paramsJson(a, .{ .what = object, .from = from }), .opens_app = true, .wait_ms = answer_ms };
+ }
+ if (std.mem.eql(u8, verb, "grant")) {
+ if (!std.mem.eql(u8, object, "accessibility") and !std.mem.eql(u8, object, "screen")) return null;
+ return .{ .method = "nanocua/grant", .params = try paramsJson(a, .{ .which = object }), .opens_app = true };
+ }
+ return null;
+}
+
+fn paramsJson(a: std.mem.Allocator, v: anytype) ![]const u8 {
+ return std.json.Stringify.valueAlloc(a, v, .{});
+}
+
+/// The status as a person reads it.
+pub fn describe(a: std.mem.Allocator, s: control.Status) ![]const u8 {
+ var aw: std.Io.Writer.Allocating = .init(a);
+ const w = &aw.writer;
+ if (s.outcome) |o| {
+ const line: ?[]const u8 = if (std.mem.eql(u8, o, "allowed"))
+ "Allowed."
+ else if (std.mem.eql(u8, o, "denied"))
+ "Denied in nanocua's window."
+ else if (std.mem.eql(u8, o, "expired"))
+ "Nobody answered in nanocua's window."
+ else if (std.mem.eql(u8, o, "busy"))
+ "nanocua's window is already waiting for an answer."
+ else if (std.mem.eql(u8, o, "quiet"))
+ "Just denied: try again in a few seconds."
+ else if (std.mem.eql(u8, o, "unreachable"))
+ "Couldn't reach nanocua: open nanocua.app."
+ else if (std.mem.eql(u8, o, "opened"))
+ "macOS's prompt or its settings should be open: turn nanocua on there."
+ else
+ null;
+ if (line) |l| try w.print("{s}\n", .{l});
+ }
+ if (!s.running) {
+ try w.writeAll("nanocua isn't running.\n");
+ return aw.written();
+ }
+ const state = if (s.stopped) "stopped" else if (s.input) "acting" else "looking only";
+ try w.print("nanocua: {s}, whole screen {s}, {d} agent{s}, {d} call{s}\n", .{ state, if (s.wholeScreen) "on" else "off", s.agents, if (s.agents == 1) "" else "s", s.calls, if (s.calls == 1) "" else "s" });
+ try w.print("macOS: Accessibility {s}, Screen Recording {s}\n", .{ if (s.accessibility) "allowed" else "not allowed", if (s.screenRecording) "allowed" else "not allowed" });
+ if (s.pending) |p| try w.print("Waiting in nanocua's window: {s}\n", .{p});
+ return aw.written();
+}
+
+fn show(a: std.mem.Allocator, s: control.Status, json: bool) !void {
+ const text = if (json) blk: {
+ const line = try control.statusJson(a, s);
+ break :blk try std.fmt.allocPrint(a, "{s}\n", .{line});
+ } else try describe(a, s);
+ _ = server.writeAll(1, text);
+}
+
+pub fn run(init: std.process.Init, args: []const [:0]const u8) !u8 {
+ const a = init.arena.allocator();
+ var json = false;
+ var from: []const u8 = "nanocua ctl";
+ var words: std.ArrayList([]const u8) = .empty;
+ for (args) |arg| {
+ if (std.mem.eql(u8, arg, "--json")) {
+ json = true;
+ } else if (std.mem.startsWith(u8, arg, "--from=")) {
+ from = arg["--from=".len..];
+ } else try words.append(a, arg);
+ }
+ const cmd = try parse(a, words.items, from) orelse {
+ _ = server.writeAll(2, usage);
+ return 2;
+ };
+ var buf: [512]u8 = undefined;
+ const path = try sockpath.path(&buf, init.environ_map, sockpath.realHome());
+ const fd = (if (cmd.opens_app) proxy.connectOrOpen(path) else proxy.connect(path)) orelse {
+ try show(a, .{ .running = false, .outcome = if (cmd.opens_app) "unreachable" else "not_running" }, json);
+ return 0;
+ };
+ defer _ = close(fd);
+ const line = try std.fmt.allocPrint(a, "{{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"{s}\",\"params\":{s}}}\n", .{ cmd.method, cmd.params });
+ if (!server.writeAll(fd, line)) return unreachableApp(a, json);
+ const reply = readLine(a, fd, cmd.wait_ms) catch return unreachableApp(a, json);
+ const parsed = std.json.parseFromSliceLeaky(std.json.Value, a, reply, .{}) catch return unreachableApp(a, json);
+ const result = (if (parsed == .object) parsed.object.get("result") else null) orelse return unreachableApp(a, json);
+ if (json) {
+ const text = try std.json.Stringify.valueAlloc(a, result, .{});
+ _ = server.writeAll(1, text);
+ _ = server.writeAll(1, "\n");
+ return 0;
+ }
+ const s = try std.json.parseFromValueLeaky(control.Status, a, result, .{ .ignore_unknown_fields = true });
+ try show(a, s, false);
+ return 0;
+}
+
+fn unreachableApp(a: std.mem.Allocator, json: bool) !u8 {
+ try show(a, .{ .running = false, .outcome = "unreachable" }, json);
+ return 0;
+}
+
+fn readLine(a: std.mem.Allocator, fd: c_int, wait_ms: i32) ![]const u8 {
+ var got: std.ArrayList(u8) = .empty;
+ var chunk: [16 * 1024]u8 = undefined;
+ while (true) {
+ var fds = [_]std.posix.pollfd{.{ .fd = fd, .events = std.posix.POLL.IN, .revents = 0 }};
+ if (try std.posix.poll(&fds, wait_ms) == 0) return error.TimedOut;
+ const n = std.c.read(fd, &chunk, chunk.len);
+ if (n <= 0) return error.Closed;
+ try got.appendSlice(a, chunk[0..@intCast(n)]);
+ if (std.mem.indexOfScalar(u8, got.items, '\n')) |nl| return got.items[0..nl];
+ }
+}
+
+test "command lines" {
+ var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const p = struct {
+ fn f(al: std.mem.Allocator, args: []const []const u8) !?Command {
+ return parse(al, args, "graff");
+ }
+ }.f;
+ try std.testing.expectEqualStrings("nanocua/status", (try p(a, &.{"status"})).?.method);
+ try std.testing.expect(!(try p(a, &.{"stop"})).?.opens_app);
+ const allow = (try p(a, &.{ "allow", "whole_screen" })).?;
+ try std.testing.expect(allow.opens_app and allow.wait_ms > 60_000);
+ try std.testing.expectEqualStrings("{\"what\":\"whole_screen\",\"from\":\"graff\"}", allow.params);
+ try std.testing.expectEqualStrings("{\"which\":\"screen\"}", (try p(a, &.{ "grant", "screen" })).?.params);
+ try std.testing.expect((try p(a, &.{ "off", "resume" })) == null);
+ try std.testing.expect((try p(a, &.{ "allow", "everything" })) == null);
+ try std.testing.expect((try p(a, &.{ "grant", "camera" })) == null);
+ try std.testing.expect((try p(a, &.{ "stop", "now" })) == null);
+ try std.testing.expect((try p(a, &.{})) == null);
+}
+
+test "what a person reads" {
+ var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ try std.testing.expectEqualStrings("nanocua isn't running.\n", try describe(a, .{ .running = false, .outcome = "not_running" }));
+ const on = try describe(a, .{ .input = true, .agents = 1, .calls = 2, .accessibility = true, .outcome = "allowed", .pending = "resume" });
+ try std.testing.expect(std.mem.startsWith(u8, on, "Allowed.\nnanocua: acting, whole screen off, 1 agent, 2 calls\n"));
+ try std.testing.expect(std.mem.indexOf(u8, on, "Accessibility allowed, Screen Recording not allowed") != null);
+ try std.testing.expect(std.mem.endsWith(u8, on, "Waiting in nanocua's window: resume\n"));
+ try std.testing.expect(std.mem.indexOf(u8, try describe(a, .{ .stopped = true }), "nanocua: stopped") != null);
+}
diff --git a/apps/nanocua/src/mac/backend.zig b/apps/nanocua/src/mac/backend.zig
index 57d96564..c18092d3 100644
--- a/apps/nanocua/src/mac/backend.zig
+++ b/apps/nanocua/src/mac/backend.zig
@@ -16,6 +16,7 @@ const input = @import("input.zig");
const capture = @import("capture.zig");
const apps = @import("apps.zig");
const windows = @import("windows.zig");
+const hit = @import("hit.zig");
const target = @import("../target.zig");
extern "c" fn dispatch_queue_create(label: [*:0]const u8, attr: ?*anyopaque) objc.dispatch_queue_t;
@@ -23,26 +24,62 @@ extern "c" fn dispatch_async_f(queue: objc.dispatch_queue_t, ctx: ?*anyopaque, w
extern "c" fn getpid() c_int;
extern "c" fn usleep(us: c_uint) c_int;
+/// What one agent is working on. Agents connect one socket each: refs from
+/// one's tree mean nothing to another, and neither moves the other's target.
+pub const Session = struct {
+ refs: ax.Refs,
+ /// The app it last read or opened.
+ last: ?i32 = null,
+};
+
+pub const Sessions = struct {
+ gpa: std.mem.Allocator,
+ map: std.AutoHashMapUnmanaged(u64, *Session) = .empty,
+
+ pub fn get(s: *Sessions, conn: u64) !*Session {
+ const gop = try s.map.getOrPut(s.gpa, conn);
+ if (gop.found_existing) return gop.value_ptr.*;
+ errdefer _ = s.map.remove(conn);
+ const session = try s.gpa.create(Session);
+ session.* = .{ .refs = .{ .gpa = s.gpa } };
+ gop.value_ptr.* = session;
+ return session;
+ }
+
+ pub fn remove(s: *Sessions, conn: u64) void {
+ const kv = s.map.fetchRemove(conn) orelse return;
+ kv.value.refs.deinit();
+ s.gpa.destroy(kv.value);
+ }
+
+ pub fn deinit(s: *Sessions) void {
+ var it = s.map.valueIterator();
+ while (it.next()) |v| {
+ v.*.refs.deinit();
+ s.gpa.destroy(v.*);
+ }
+ s.map.deinit(s.gpa);
+ }
+};
+
pub const Mac = struct {
gpa: std.mem.Allocator,
queue: objc.dispatch_queue_t,
- /// The last tree's elements: the queue's alone.
- refs: ax.Refs,
+ /// Each agent's refs and target: the queue's alone.
+ sessions: Sessions,
/// The kill switch as the queue sees it.
stopped: std.atomic.Value(bool) = .init(false),
/// The whole-screen switch as the queue sees it.
whole_screen: std.atomic.Value(bool) = .init(false),
- /// The app the agent last read or opened: the queue's alone.
- last: ?i32 = null,
self_pid: i32,
pub fn init(gpa: std.mem.Allocator) Mac {
ax.setTimeout(2.0);
- return .{ .gpa = gpa, .queue = dispatch_queue_create("nanocua.tools", null), .refs = .{ .gpa = gpa }, .self_pid = getpid() };
+ return .{ .gpa = gpa, .queue = dispatch_queue_create("nanocua.tools", null), .sessions = .{ .gpa = gpa }, .self_pid = getpid() };
}
pub fn backend(m: *Mac) mcp.Backend {
- return .{ .ptr = m, .call = call };
+ return .{ .ptr = m, .call = call, .closed = closed };
}
fn call(ptr: *anyopaque, tool: tools.Tool, args: tools.Args, reply: *mcp.Reply) void {
@@ -50,6 +87,25 @@ pub const Mac = struct {
const job = Job.create(m, tool, args, reply) catch return reply.fail("nanocua is out of memory");
dispatch_async_f(m.queue, job, Job.run);
}
+
+ const Closed = struct {
+ mac: *Mac,
+ conn: u64,
+
+ fn run(ctx: ?*anyopaque) callconv(.c) void {
+ const c: *Closed = @ptrCast(@alignCast(ctx));
+ c.mac.sessions.remove(c.conn);
+ c.mac.gpa.destroy(c);
+ }
+ };
+
+ /// After any calls of its still waiting on the queue.
+ fn closed(ptr: *anyopaque, conn: u64) void {
+ const m: *Mac = @ptrCast(@alignCast(ptr));
+ const c = m.gpa.create(Closed) catch return;
+ c.* = .{ .mac = m, .conn = conn };
+ dispatch_async_f(m.queue, c, Closed.run);
+ }
};
/// Permissions, as the panel shows them.
@@ -61,14 +117,45 @@ pub fn screenRecording() bool {
return capture.allowed();
}
-/// Asks for a permission: the system's prompt, then its Settings pane.
-pub fn request(which: enum { accessibility, screen }) void {
+/// Keeps nanocua's window above other apps' while a question waits: macOS may
+/// refuse to bring an app forward that nobody clicked.
+pub fn raise(window: *nanoui.Window, on: bool) void {
+ const floating: objc.NSInteger = 3;
+ objc.msg(void, window.nswindow, "setLevel:", .{if (on) floating else @as(objc.NSInteger, 0)});
+ if (on) objc.msg(void, window.nswindow, "orderFrontRegardless", .{});
+}
+
+pub const Permission = enum(usize) { accessibility = 1, screen = 2 };
+
+/// Asks for a permission: the system's prompt, then its Settings pane. After
+/// this frame, not during it: some of these calls run the event loop while they
+/// wait, and a frame started inside a frame leaves its boxes open.
+pub fn request(which: Permission) void {
+ objc.dispatchMain(@ptrFromInt(@backingInt(which)), asked);
+}
+
+fn asked(ctx: ?*anyopaque) callconv(.c) void {
+ const which: Permission = @fromBackingInt(@intCast(@intFromPtr(ctx)));
switch (which) {
.accessibility => if (!cf.trusted(true)) openSettings("Privacy_Accessibility"),
- .screen => if (!cf.CGRequestScreenCaptureAccess()) openSettings("Privacy_ScreenCapture"),
+ .screen => if (!cf.CGRequestScreenCaptureAccess()) {
+ capture.register();
+ openSettings("Privacy_ScreenCapture");
+ // macOS may not list the app even so, and only a person can add it.
+ revealApp();
+ },
}
}
+/// Shows this app in Finder, ready to drag into a System Settings list.
+fn revealApp() void {
+ const pool = objc.Pool.push();
+ defer pool.pop();
+ const bundle = objc.msg(objc.Object, objc.class("NSBundle"), "mainBundle", .{});
+ const path = objc.fromNSString(objc.msg(?objc.Object, bundle, "bundlePath", .{}));
+ if (std.mem.endsWith(u8, path, ".app")) nanoui.shell.showInFolder(path);
+}
+
fn openSettings(pane: []const u8) void {
var buf: [128]u8 = undefined;
const url = std.fmt.bufPrint(&buf, "x-apple.systempreferences:com.apple.preference.security?{s}", .{pane}) catch return;
@@ -77,6 +164,7 @@ fn openSettings(pane: []const u8) void {
const Job = struct {
mac: *Mac,
+ session: *Session = undefined,
tool: tools.Tool,
reply: *mcp.Reply,
arena: std.heap.ArenaAllocator,
@@ -140,7 +228,10 @@ const Job = struct {
fn run(ctx: ?*anyopaque) callconv(.c) void {
const j: *Job = @ptrCast(@alignCast(ctx));
const pool = objc.Pool.push();
- j.perform() catch |e| j.fail("{s}", .{@errorName(e)});
+ if (j.mac.sessions.get(j.reply.conn)) |session| {
+ j.session = session;
+ j.perform() catch |e| j.fail("{s}", .{@errorName(e)});
+ } else |e| j.fail("{s}", .{@errorName(e)});
pool.pop();
objc.dispatchMain(j, deliver);
}
@@ -178,12 +269,12 @@ const Job = struct {
.get_ui_tree => {
if (!accessibility()) return j.fail("nanocua has no Accessibility permission: ask the user to grant it in nanocua's window.", .{});
const named = j.namedPid() orelse return;
- const pid = named orelse m.last orelse
+ const pid = named orelse j.session.last orelse
ax.focusedPid() orelse return j.fail("No app has the keyboard focus: pass pid or app.", .{});
if (pid == m.self_pid) return j.fail("That is nanocua itself, which agents don't see. Pass pid or app (see list_apps).", .{});
const max: usize = @intCast(std.math.clamp(j.max_nodes orelse 400, 1, 2000));
- try ax.writeTree(&m.refs, w, pid, .{ .max_nodes = max });
- m.last = pid;
+ try ax.writeTree(&j.session.refs, w, pid, .{ .max_nodes = max });
+ j.session.last = pid;
},
.screenshot => try j.screenshot(),
.click => {
@@ -193,14 +284,22 @@ const Job = struct {
const el = j.element(r) orelse return;
const t = j.events(ax.pidOf(el)) orelse return;
if (button == .left and count == 1 and ax.perform(el, "AXPress")) return w.print("Pressed [{d}].", .{r});
- const c = ax.center(el) orelse return j.fail("[{d}] has no frame to click.", .{r});
- input.click(t, c[0], c[1], button, count);
- return w.print("Clicked [{d}] at {d:.0},{d:.0}.", .{ r, c[0], c[1] });
+ const f = ax.frame(el) orelse return j.fail("[{d}] has no frame to click.", .{r});
+ const c = j.inWindow(t, f) orelse return j.fail("[{d}] is out of sight in its window: scroll to it first.", .{r});
+ const aimed = j.aim(t, c[0], c[1]) orelse return;
+ input.click(aimed, c[0], c[1], button, count);
+ return j.sent(aimed, "a click to", c);
}
const t = j.events(null) orelse return;
const p = j.point(t) orelse return;
- input.click(t, p[0], p[1], button, count);
- try w.print("Clicked at {d:.0},{d:.0}.", .{ p[0], p[1] });
+ const aimed = j.aim(t, p[0], p[1]) orelse return;
+ if (aimed == .app and button == .left and count == 1) {
+ var buf: [64]u8 = undefined;
+ if (hit.press(&buf, aimed.app.pid, p[0], p[1])) |what|
+ return w.print("Pressed the {s} at {d:.0},{d:.0}.", .{ what, p[0], p[1] });
+ }
+ input.click(aimed, p[0], p[1], button, count);
+ try j.sent(aimed, "a click to", p);
},
.type_text => {
const text = j.text orelse return j.fail("type_text needs text.", .{});
@@ -212,7 +311,7 @@ const Job = struct {
} else null;
const t = j.events(el_pid) orelse return;
if (el) |e| {
- if (!ax.focus(e)) if (ax.center(e)) |c| input.click(t, c[0], c[1], .left, 1);
+ if (!ax.focus(e)) if (ax.center(e)) |c| if (j.aim(t, c[0], c[1])) |aimed| input.click(aimed, c[0], c[1], .left, 1);
_ = usleep(60_000);
}
if (!j.keyboardIsOthers(t)) return;
@@ -230,23 +329,28 @@ const Job = struct {
},
.scroll => {
var el_pid: ?i32 = null;
- const center = if (j.ref) |r| blk: {
+ var el_frame: ?[4]f64 = null;
+ if (j.ref) |r| {
const el = j.element(r) orelse return;
el_pid = ax.pidOf(el);
- break :blk ax.center(el) orelse return j.fail("[{d}] has no frame.", .{r});
- } else null;
+ el_frame = ax.frame(el) orelse return j.fail("[{d}] has no frame.", .{r});
+ }
const t = j.events(el_pid) orelse return;
- const p = center orelse j.point(t) orelse return;
+ const p = if (el_frame) |f|
+ j.inWindow(t, f) orelse return j.fail("[{d}] is out of sight in its window.", .{j.ref.?})
+ else
+ j.point(t) orelse return;
+ const aimed = j.aim(t, p[0], p[1]) orelse return;
const dx = j.dx orelse 0;
const dy: f64 = j.dy orelse if (dx == 0) 3 else 0;
- input.scroll(t, p[0], p[1], dx, dy);
- try w.print("Scrolled {d} down, {d} right at {d:.0},{d:.0}.", .{ dy, dx, p[0], p[1] });
+ try j.scrollAt(aimed, p, dx, dy);
},
.move_mouse => {
const t = j.events(null) orelse return;
const p = j.point(t) orelse return;
- input.move(t, p[0], p[1]);
- try w.print("Moved to {d:.0},{d:.0}.", .{ p[0], p[1] });
+ const aimed = j.aim(t, p[0], p[1]) orelse return;
+ input.move(aimed, p[0], p[1]);
+ try if (aimed == .hid) w.print("Moved the pointer to {d:.0},{d:.0}.", .{ p[0], p[1] }) else j.sent(aimed, "a hover to", p);
},
.set_value => {
const r = j.ref orelse return j.fail("set_value needs ref.", .{});
@@ -267,7 +371,7 @@ const Job = struct {
// A later get_ui_tree without app reads this one. (One just
// launching isn't in the running list yet.)
if (apps.pidNamed(n)) |pid| if (pid != m.self_pid) {
- m.last = pid;
+ j.session.last = pid;
};
},
}
@@ -285,7 +389,7 @@ const Job = struct {
/// The element behind ref `r`, unless it is missing or nanocua's.
fn element(j: *Job, r: i64) ?cf.AXUIElementRef {
- const el = j.mac.refs.get(r) orelse {
+ const el = j.session.refs.get(r) orelse {
j.fail("No element [{d}]: refs come from the last get_ui_tree.", .{r});
return null;
};
@@ -315,7 +419,7 @@ const Job = struct {
.whole_screen = j.whole_screen,
.allowed = m.whole_screen.load(.acquire),
.pid = named orelse el_pid,
- .last = m.last,
+ .last = j.session.last,
.self = m.self_pid,
});
switch (r) {
@@ -325,12 +429,67 @@ const Job = struct {
},
.whole_screen => return .hid,
.app => |pid| {
- m.last = pid;
+ j.session.last = pid;
return .{ .app = .{ .pid = pid, .window = if (windows.main(pid)) |w| w.id else 0 } };
},
}
}
+ /// Where to aim at an element so that the event lands in its window: on the
+ /// real pointer its middle, in an app the middle of what shows.
+ fn inWindow(j: *Job, t: input.Target, f: [4]f64) ?[2]f64 {
+ _ = j;
+ const middle: [2]f64 = .{ f[0] + f[2] / 2, f[1] + f[3] / 2 };
+ if (t == .hid) return middle;
+ const w = windows.at(t.app.pid, middle[0], middle[1]) orelse windows.main(t.app.pid) orelse return null;
+ return target.visiblePoint(f, windows.box(w));
+ }
+
+ /// `t` aimed at the window under x, y. An app's window must be there: an
+ /// event anywhere else goes nowhere, and saying so beats pretending.
+ fn aim(j: *Job, t: input.Target, x: f64, y: f64) ?input.Target {
+ if (t == .hid) return t;
+ const w = windows.at(t.app.pid, x, y) orelse {
+ if (windows.main(t.app.pid)) |m| {
+ const b = windows.box(m);
+ j.fail("{d:.0},{d:.0} isn't in a window of that app. Its main window is at {d:.0},{d:.0} sized {d:.0}x{d:.0}.", .{ x, y, b[0], b[1], b[2], b[3] });
+ } else j.fail("That app has no window to act in.", .{});
+ return null;
+ };
+ return .{ .app = .{ .pid = t.app.pid, .window = w.id, .origin = .{ w.frame.origin.x, w.frame.origin.y } } };
+ }
+
+ /// Reports events that can't be confirmed, which is every event sent into
+ /// an app: macOS doesn't say whether it acted on them.
+ fn sent(j: *Job, t: input.Target, what: []const u8, p: [2]f64) !void {
+ const w = &j.out.writer;
+ if (t == .hid) return w.print("Sent {s} the screen at {d:.0},{d:.0}.", .{ what, p[0], p[1] });
+ try w.print("Sent {s} the app at {d:.0},{d:.0}. Not confirmed: apps can ignore events in the background, so check with get_ui_tree or screenshot.", .{ what, p[0], p[1] });
+ }
+
+ /// Scrolls and says what really happened: with a scroll bar to read, it knows.
+ fn scrollAt(j: *Job, t: input.Target, p: [2]f64, dx: f64, dy: f64) !void {
+ const w = &j.out.writer;
+ if (t == .hid) {
+ input.scroll(t, p[0], p[1], dx, dy);
+ return w.print("Scrolled the screen {d} down, {d} right at {d:.0},{d:.0}.", .{ dy, dx, p[0], p[1] });
+ }
+ const pid = t.app.pid;
+ const before = hit.position(pid, p[0], p[1]);
+ input.scroll(t, p[0], p[1], dx, dy);
+ if (!before.known()) return j.sent(t, "a scroll to", p);
+ // Scrolling can animate.
+ _ = usleep(200_000);
+ if (hit.position(pid, p[0], p[1]).moved(before))
+ return w.print("Scrolled {d} down, {d} right at {d:.0},{d:.0}.", .{ dy, dx, p[0], p[1] });
+ if (before.atEnd(dx, dy)) return w.print("Nothing moved: it's already at the end that way.", .{});
+ hit.scrollBars(pid, p[0], p[1], dx, dy);
+ _ = usleep(50_000);
+ if (hit.position(pid, p[0], p[1]).moved(before))
+ return w.print("Scrolled at {d:.0},{d:.0} through its scroll bars: about {d} down, {d} right.", .{ p[0], p[1], dy, dx });
+ j.fail("Nothing scrolled at {d:.0},{d:.0}: the app ignored it. It may not be scrollable there, or need whole_screen.", .{ p[0], p[1] });
+ }
+
/// x and y. On the real pointer, not over nanocua's own windows.
fn point(j: *Job, t: input.Target) ?[2]f64 {
const x = j.x orelse {
@@ -365,7 +524,7 @@ const Job = struct {
.whole_screen = j.whole_screen,
.allowed = m.whole_screen.load(.acquire),
.pid = named,
- .last = m.last,
+ .last = j.session.last,
.self = m.self_pid,
});
const max: u32 = @intCast(std.math.clamp(j.max_size orelse 1280, 64, 2560));
@@ -387,7 +546,7 @@ const Job = struct {
.app => |pid| blk: {
const w = windows.main(pid) orelse return j.fail("App {d} has no window to capture.", .{pid});
rect = w.frame;
- m.last = pid;
+ j.session.last = pid;
break :blk capture.window(m.gpa, w.id, w.frame, max);
},
};
@@ -401,3 +560,17 @@ const Job = struct {
try j.out.writer.print("The image is {d}x{d} px of the region at {d:.0},{d:.0} sized {d:.0}x{d:.0} points: pixel (px, py) is the point ({d:.0} + px*{d:.4}, {d:.0} + py*{d:.4}).", .{ shot.width, shot.height, rect.origin.x, rect.origin.y, rect.size.width, rect.size.height, rect.origin.x, sx, rect.origin.y, sy });
}
};
+
+test "agents don't share refs or a target; a closed agent's are freed" {
+ var s: Sessions = .{ .gpa = std.testing.allocator };
+ defer s.deinit();
+ const a = try s.get(1);
+ const b = try s.get(2);
+ try std.testing.expect(a != b);
+ a.last = 42;
+ try std.testing.expectEqual(@as(?i32, null), b.last);
+ try std.testing.expectEqual(a, try s.get(1));
+ s.remove(1);
+ s.remove(1);
+ try std.testing.expectEqual(@as(?i32, null), (try s.get(1)).last);
+}
diff --git a/apps/nanocua/src/mac/capture.zig b/apps/nanocua/src/mac/capture.zig
index 45bff006..17a9291d 100644
--- a/apps/nanocua/src/mac/capture.zig
+++ b/apps/nanocua/src/mac/capture.zig
@@ -43,6 +43,19 @@ pub fn allowed() bool {
return cf.CGPreflightScreenCaptureAccess();
}
+var nothing: u8 = 0;
+
+fn ignore(_: *u8, _: ?objc.Object, _: ?objc.Object) void {}
+
+/// Asks ScreenCaptureKit for what it could capture and ignores the answer:
+/// without permission, this is what gets the app listed in System Settings,
+/// where the permission request alone may not.
+pub fn register() void {
+ const cls = objc_getClass("SCShareableContent") orelse return;
+ var block = objc.Block(u8, &.{ ?objc.Object, ?objc.Object }).init(&nothing, ignore);
+ objc.msg(void, cls, "getShareableContentWithCompletionHandler:", .{@as(*anyopaque, @ptrCast(&block))});
+}
+
/// A rectangle of the screen.
pub fn capture(gpa: std.mem.Allocator, rect: cf.CGRect, max_size: u32) Error!Shot {
// Asking would put up the system's prompt for an agent: only the user
@@ -119,8 +132,8 @@ const Shareable = struct {
const answer_timeout: u64 = 10 * std.time.ns_per_s;
-/// ScreenCaptureKit's window with number `id`, retained.
-fn scWindow(gpa: std.mem.Allocator, cls: objc.Class, id: u32) !?objc.Object {
+/// What ScreenCaptureKit could capture, retained.
+fn shareableContent(gpa: std.mem.Allocator, cls: objc.Class) !?objc.Object {
const wait = try gpa.create(Shareable);
wait.* = .{ .sem = objc.dispatch_semaphore_create(0) };
var block = objc.Block(Shareable, &.{ ?objc.Object, ?objc.Object }).init(wait, Shareable.done);
@@ -129,25 +142,55 @@ fn scWindow(gpa: std.mem.Allocator, cls: objc.Class, id: u32) !?objc.Object {
if (objc.dispatch_semaphore_wait(wait.sem, dispatchTime(answer_timeout)) != 0) return null;
defer gpa.destroy(wait);
defer objc.dispatch_release(@ptrCast(wait.sem));
- const content = wait.content orelse return null;
- defer objc.release(content);
+ return wait.content;
+}
+
+fn findWindow(content: objc.Object, id: u32) ?objc.Object {
const list = objc.msg(objc.Object, content, "windows", .{});
for (0..objc.msg(usize, list, "count", .{})) |i| {
const w = objc.msg(objc.Object, list, "objectAtIndex:", .{i});
- if (objc.msg(u32, w, "windowID", .{}) == id) return objc.retain(w);
+ if (objc.msg(u32, w, "windowID", .{}) == id) return w;
}
return null;
}
+/// How much of `frame` lies in `other`, in square points.
+pub fn overlap(frame: objc.CGRect, other: objc.CGRect) f64 {
+ const w = @min(frame.origin.x + frame.size.width, other.origin.x + other.size.width) - @max(frame.origin.x, other.origin.x);
+ const h = @min(frame.origin.y + frame.size.height, other.origin.y + other.size.height) - @max(frame.origin.y, other.origin.y);
+ return if (w > 0 and h > 0) w * h else 0;
+}
+
+/// The display showing most of `frame`.
+fn displayOf(content: objc.Object, frame: objc.CGRect) ?objc.Object {
+ const list = objc.msg(objc.Object, content, "displays", .{});
+ var best: ?objc.Object = null;
+ var most: f64 = 0;
+ for (0..objc.msg(usize, list, "count", .{})) |i| {
+ const d = objc.msg(objc.Object, list, "objectAtIndex:", .{i});
+ const shared = overlap(frame, objc.msg(objc.CGRect, d, "frame", .{}));
+ if (shared > most) {
+ most = shared;
+ best = d;
+ }
+ }
+ return best;
+}
+
+/// `frame` in the coordinates of a display at `display`.
+pub fn inDisplay(frame: objc.CGRect, display: objc.CGRect) objc.CGRect {
+ return .{ .origin = .{ .x = frame.origin.x - display.origin.x, .y = frame.origin.y - display.origin.y }, .size = frame.size };
+}
+
fn grabWindow(gpa: std.mem.Allocator, id: u32, frame: cf.CGRect, max_size: u32) !?CGImageRef {
const manager = objc_getClass("SCScreenshotManager");
const shareable = objc_getClass("SCShareableContent");
const modern = if (manager) |c| objc.msg(bool, c, "respondsToSelector:", .{objc.sel("captureImageWithFilter:configuration:completionHandler:")}) else false;
if (!modern or shareable == null) return CGWindowListCreateImage(CGRectNull, list_including_window, id, image_bounds_ignore_framing);
- const sc = try scWindow(gpa, shareable.?, id) orelse return null;
- defer objc.release(sc);
- const filter = objc.msg(objc.Object, objc.alloc("SCContentFilter"), "initWithDesktopIndependentWindow:", .{sc});
- defer objc.release(filter);
+ const content = try shareableContent(gpa, shareable.?) orelse return null;
+ defer objc.release(content);
+ const sc = findWindow(content, id) orelse return null;
+ const rect: objc.CGRect = .{ .origin = .{ .x = frame.origin.x, .y = frame.origin.y }, .size = .{ .width = frame.size.width, .height = frame.size.height } };
const config = objc.new("SCStreamConfiguration");
defer objc.release(config);
// Retina's pixels, unless that's more than the picture will keep.
@@ -156,6 +199,20 @@ fn grabWindow(gpa: std.mem.Allocator, id: u32, frame: cf.CGRect, max_size: u32)
objc.msg(void, config, "setWidth:", .{@as(usize, @intFromFloat(@max(1, @round(frame.size.width * scale))))});
objc.msg(void, config, "setHeight:", .{@as(usize, @intFromFloat(@max(1, @round(frame.size.height * scale))))});
objc.msg(void, config, "setShowsCursor:", .{false});
+ // On a display: that display with only this window on it, whatever is in
+ // front, cut to its frame, so pixels map exactly to points. The window's own
+ // filter widens to fit child windows (a floating orb, a popover) and shrinks
+ // the picture, and its source rect isn't in one coordinate system. A window
+ // off the displays (another Space) has only that filter.
+ const display = if (objc.msg(bool, sc, "isOnScreen", .{})) displayOf(content, rect) else null;
+ const filter = if (display) |d| blk: {
+ objc.msg(void, config, "setSourceRect:", .{inDisplay(rect, objc.msg(objc.CGRect, d, "frame", .{}))});
+ if (objc.msg(bool, config, "respondsToSelector:", .{objc.sel("setIncludeChildWindows:")}))
+ objc.msg(void, config, "setIncludeChildWindows:", .{false});
+ const only = objc.msg(objc.Object, objc.class("NSArray"), "arrayWithObject:", .{sc});
+ break :blk objc.msg(objc.Object, objc.alloc("SCContentFilter"), "initWithDisplay:includingWindows:", .{ d, only });
+ } else objc.msg(objc.Object, objc.alloc("SCContentFilter"), "initWithDesktopIndependentWindow:", .{sc});
+ defer objc.release(filter);
const wait = try gpa.create(Wait);
wait.* = .{ .sem = objc.dispatch_semaphore_create(0) };
var block = objc.Block(Wait, &.{ ?*anyopaque, ?objc.Object }).init(wait, Wait.done);
@@ -190,3 +247,15 @@ extern "c" fn dispatch_time(when: u64, delta: i64) u64;
fn dispatchTime(ns: u64) u64 {
return dispatch_time(0, @intCast(ns));
}
+
+test "a window's place on its display" {
+ const win: objc.CGRect = .{ .origin = .{ .x = 1500, .y = 100 }, .size = .{ .width = 400, .height = 300 } };
+ const left: objc.CGRect = .{ .size = .{ .width = 1440, .height = 900 } };
+ const right: objc.CGRect = .{ .origin = .{ .x = 1440 }, .size = .{ .width = 1920, .height = 1080 } };
+ try std.testing.expectEqual(@as(f64, 0), overlap(win, left));
+ try std.testing.expectEqual(@as(f64, 120000), overlap(win, right));
+ const placed = inDisplay(win, right);
+ try std.testing.expectEqual(@as(f64, 60), placed.origin.x);
+ try std.testing.expectEqual(@as(f64, 100), placed.origin.y);
+ try std.testing.expectEqual(@as(f64, 400), placed.size.width);
+}
diff --git a/apps/nanocua/src/mac/hit.zig b/apps/nanocua/src/mac/hit.zig
new file mode 100644
index 00000000..319e82e6
--- /dev/null
+++ b/apps/nanocua/src/mac/hit.zig
@@ -0,0 +1,180 @@
+//! What's at a point in an app, through accessibility: an exact way to act
+//! when events in the background are only a hope. Pressing what's there, and
+//! scrolling what's there and reading whether it moved.
+const std = @import("std");
+const cf = @import("cf.zig");
+const ax = @import("ax.zig");
+const tree = @import("../tree.zig");
+
+/// App `pid`'s element at x, y; release it.
+fn at(pid: i32, x: f64, y: f64) ?cf.AXUIElementRef {
+ const app = cf.AXUIElementCreateApplication(pid);
+ defer cf.CFRelease(app);
+ var el: ?cf.AXUIElementRef = null;
+ if (cf.AXUIElementCopyElementAtPosition(app, @floatCast(x), @floatCast(y), &el) != 0) return null;
+ return el;
+}
+
+fn role(buf: []u8, el: cf.AXUIElementRef) []const u8 {
+ return ax.copyText(buf, el, "AXRole");
+}
+
+/// `el`'s parent; release it.
+fn parent(el: cf.AXUIElementRef) ?cf.AXUIElementRef {
+ return @ptrCast(ax.copy(el, "AXParent"));
+}
+
+/// Presses what's at x, y if it's a control, going up from the label or icon
+/// inside it. Writes the control's role into `buf`; null if nothing was pressed.
+pub fn press(buf: []u8, pid: i32, x: f64, y: f64) ?[]const u8 {
+ var el = at(pid, x, y) orelse return null;
+ defer cf.CFRelease(el);
+ for (0..4) |_| {
+ const r = role(buf, el);
+ if (tree.pressable(r)) return if (ax.perform(el, "AXPress")) tree.bareRole(r) else null;
+ if (!tree.insideControl(r)) return null;
+ const up = parent(el) orelse return null;
+ cf.CFRelease(el);
+ el = up;
+ }
+ return null;
+}
+
+pub const Bar = enum { vertical, horizontal };
+
+fn barName(bar: Bar) []const u8 {
+ return switch (bar) {
+ .vertical => "AXVerticalScrollBar",
+ .horizontal => "AXHorizontalScrollBar",
+ };
+}
+
+/// The scroll area around x, y; release it.
+fn scrollArea(pid: i32, x: f64, y: f64) ?cf.AXUIElementRef {
+ var el = at(pid, x, y) orelse return null;
+ var buf: [64]u8 = undefined;
+ for (0..32) |_| {
+ if (std.mem.eql(u8, role(&buf, el), "AXScrollArea")) return el;
+ const up = parent(el) orelse break;
+ cf.CFRelease(el);
+ el = up;
+ }
+ cf.CFRelease(el);
+ return null;
+}
+
+fn value(bar: cf.AXUIElementRef) ?f64 {
+ const v = ax.copy(bar, "AXValue") orelse return null;
+ defer cf.CFRelease(v);
+ if (cf.CFGetTypeID(v) != cf.CFNumberGetTypeID()) return null;
+ var d: f64 = 0;
+ return if (cf.CFNumberGetValue(@ptrCast(v), cf.number_double, &d)) d else null;
+}
+
+/// How far along it is scrolled, 0 to 1, on each axis, of what has a scroll bar.
+pub const Position = struct {
+ vertical: ?f64 = null,
+ horizontal: ?f64 = null,
+
+ pub fn known(p: Position) bool {
+ return p.vertical != null or p.horizontal != null;
+ }
+
+ /// Whether anything moved since `before`.
+ pub fn moved(p: Position, before: Position) bool {
+ const near = struct {
+ fn f(a: ?f64, b: ?f64) bool {
+ return @abs((a orelse 0) - (b orelse 0)) < 1e-6;
+ }
+ }.f;
+ return !near(p.vertical, before.vertical) or !near(p.horizontal, before.horizontal);
+ }
+
+ /// Whether it can't go that way anyway: the end of the document, not a failure.
+ pub fn atEnd(p: Position, dx: f64, dy: f64) bool {
+ const stuck = struct {
+ fn f(v: ?f64, d: f64) bool {
+ if (d == 0) return true;
+ const at_ = v orelse return false;
+ return if (d > 0) at_ >= 0.999 else at_ <= 0.001;
+ }
+ }.f;
+ return stuck(p.vertical, dy) and stuck(p.horizontal, dx);
+ }
+};
+
+/// The scroll position at x, y.
+pub fn position(pid: i32, x: f64, y: f64) Position {
+ const area = scrollArea(pid, x, y) orelse return .{};
+ defer cf.CFRelease(area);
+ var out: Position = .{};
+ if (ax.copy(area, barName(.vertical))) |b| {
+ defer cf.CFRelease(b);
+ out.vertical = value(@ptrCast(b));
+ }
+ if (ax.copy(area, barName(.horizontal))) |b| {
+ defer cf.CFRelease(b);
+ out.horizontal = value(@ptrCast(b));
+ }
+ return out;
+}
+
+/// The share of the whole a move of `lines` lines is, in a view with `hidden`
+/// points scrolled out of sight (a guess at the line's height).
+pub fn lineShare(lines: f64, hidden: f64) f64 {
+ const line_points = 16;
+ return if (hidden > 1) lines * line_points / hidden else lines * 0.05;
+}
+
+/// How many points of the area's content are out of sight on an axis.
+fn hiddenPoints(area: cf.AXUIElementRef, bar: Bar) f64 {
+ const view = ax.frame(area) orelse return 0;
+ const contents = cf.array(ax.copy(area, "AXContents")) orelse return 0;
+ defer cf.CFRelease(contents);
+ if (cf.CFArrayGetCount(contents) < 1) return 0;
+ const first = cf.CFArrayGetValueAtIndex(contents, 0) orelse return 0;
+ const inner = ax.frame(@ptrCast(first)) orelse return 0;
+ return switch (bar) {
+ .vertical => inner[3] - view[3],
+ .horizontal => inner[2] - view[2],
+ };
+}
+
+fn moveBar(area: cf.AXUIElementRef, bar: Bar, lines: f64) void {
+ if (lines == 0) return;
+ const b = ax.copy(area, barName(bar)) orelse return;
+ defer cf.CFRelease(b);
+ const now = value(@ptrCast(b)) orelse return;
+ const next = std.math.clamp(now + lineShare(lines, hiddenPoints(area, bar)), 0, 1);
+ const n = cf.CFNumberCreate(null, cf.number_double, &next) orelse return;
+ defer cf.CFRelease(n);
+ _ = ax.set(@ptrCast(b), "AXValue", n);
+}
+
+/// Scrolls through the scroll bars themselves, for an app that ignored the
+/// wheel. The caller reads whether that worked.
+pub fn scrollBars(pid: i32, x: f64, y: f64, dx: f64, dy: f64) void {
+ const area = scrollArea(pid, x, y) orelse return;
+ defer cf.CFRelease(area);
+ moveBar(area, .vertical, dy);
+ moveBar(area, .horizontal, dx);
+}
+
+test "scroll positions: moved, and the end of the document" {
+ const p: Position = .{ .vertical = 0.5 };
+ try std.testing.expect(p.known());
+ try std.testing.expect(!(Position{}).known());
+ try std.testing.expect(!p.moved(.{ .vertical = 0.5 }));
+ try std.testing.expect(p.moved(.{ .vertical = 0.25 }));
+ try std.testing.expect(!(Position{ .vertical = 0.5 }).atEnd(0, 10));
+ try std.testing.expect((Position{ .vertical = 1 }).atEnd(0, 10));
+ try std.testing.expect((Position{ .vertical = 0 }).atEnd(0, -10));
+ try std.testing.expect(!(Position{ .vertical = 0 }).atEnd(0, 10));
+ try std.testing.expect(!(Position{ .vertical = 1 }).atEnd(5, 10)); // no way to know sideways
+}
+
+test "a line's share of what is out of sight" {
+ try std.testing.expectEqual(@as(f64, 0.1), lineShare(10, 1600));
+ try std.testing.expectEqual(@as(f64, -0.1), lineShare(-10, 1600));
+ try std.testing.expectEqual(@as(f64, 0.5), lineShare(10, 0)); // unknown: a guess
+}
diff --git a/apps/nanocua/src/mac/hotkey.zig b/apps/nanocua/src/mac/hotkey.zig
index 973e450d..371c9932 100644
--- a/apps/nanocua/src/mac/hotkey.zig
+++ b/apps/nanocua/src/mac/hotkey.zig
@@ -8,7 +8,7 @@ const EventTypeSpec = extern struct { class: u32, kind: u32 };
const HotKeyID = extern struct { signature: u32, id: u32 };
const Handler = *const fn (next: ?*anyopaque, event: ?*anyopaque, user: ?*anyopaque) callconv(.c) i32;
extern "c" fn GetApplicationEventTarget() *anyopaque;
-extern "c" fn InstallEventHandler(target: *anyopaque, handler: Handler, n: u32, list: [*]const EventTypeSpec, user: ?*anyopaque, out: ?*?*anyopaque) i32;
+extern "c" fn InstallEventHandler(target: *anyopaque, handler: Handler, n: usize, list: [*]const EventTypeSpec, user: ?*anyopaque, out: ?*?*anyopaque) i32;
extern "c" fn RegisterEventHotKey(code: u32, modifiers: u32, id: HotKeyID, target: *anyopaque, options: u32, out: *?*anyopaque) i32;
extern "c" fn UnregisterEventHotKey(ref: *anyopaque) i32;
diff --git a/apps/nanocua/src/mac/input.zig b/apps/nanocua/src/mac/input.zig
index c51f113e..49bbfb08 100644
--- a/apps/nanocua/src/mac/input.zig
+++ b/apps/nanocua/src/mac/input.zig
@@ -5,6 +5,7 @@
const std = @import("std");
const keys = @import("../keys.zig");
const cf = @import("cf.zig");
+const skylight = @import("skylight.zig");
const CGEventRef = *opaque {};
extern "c" fn CGEventCreateMouseEvent(source: ?*anyopaque, kind: u32, at: cf.CGPoint, button: u32) ?CGEventRef;
@@ -19,19 +20,39 @@ extern "c" fn CGEventPostToPid(pid: i32, ev: CGEventRef) void;
extern "c" fn usleep(us: c_uint) c_int;
const hid_tap: u32 = 0;
+const mouse_moved: u32 = 5;
+// CGEvent's fields. Those without a public name are what WindowServer stamps on
+// an event it routes itself, and an app in the background goes by them.
const field_click_state: u32 = 1;
+const field_button: u32 = 3;
+const field_subtype: u32 = 7;
+const field_target_pid: u32 = 40;
+const field_window_number: u32 = 51;
+const field_click_group: u32 = 58;
const field_window_under_pointer: u32 = 91;
const field_window_handling: u32 = 92;
-const mouse_moved: u32 = 5;
+const subtype_touch: i64 = 3;
+/// A scroll event carries at most this many lines: more arrive coalesced or not at all.
+const lines_per_event: i32 = 10;
pub const Button = enum { left, right };
+/// One app's window, where mouse events land.
+pub const App = struct {
+ pid: i32,
+ /// The window under the event, and its top-left in screen points.
+ window: u32 = 0,
+ origin: [2]f64 = .{ 0, 0 },
+ /// Ties the events of one gesture together.
+ group: i64 = 0,
+};
+
/// Where events go.
pub const Target = union(enum) {
/// The real pointer and keyboard.
hid,
- /// Into one app. `window` is what a mouse event says it is over (0: unknown).
- app: struct { pid: i32, window: u32 = 0 },
+ /// Into one app.
+ app: App,
fn post(t: Target, ev: ?CGEventRef) void {
const e = ev orelse return;
@@ -42,38 +63,78 @@ pub const Target = union(enum) {
cf.CFRelease(e);
}
- /// A mouse event at x, y, saying which window it is over: an app in the
- /// background otherwise goes by where the real pointer is.
- fn mouse(t: Target, kind: u32, x: f64, y: f64, button: u32) ?CGEventRef {
- const ev = CGEventCreateMouseEvent(null, kind, .{ .x = x, .y = y }, button) orelse return null;
- if (t == .app and t.app.window != 0) {
- CGEventSetIntegerValueField(ev, field_window_under_pointer, t.app.window);
- CGEventSetIntegerValueField(ev, field_window_handling, t.app.window);
+ /// Posts a mouse or scroll event, with what an app in the background needs
+ /// to act on it at x, y.
+ fn postPointer(t: Target, ev: ?CGEventRef, x: f64, y: f64, state: i64, button: i64) void {
+ const e = ev orelse return;
+ defer cf.CFRelease(e);
+ const a = switch (t) {
+ .hid => return CGEventPost(hid_tap, e),
+ .app => |app| app,
+ };
+ // Exactly no modifiers: an event otherwise inherits what the user holds.
+ CGEventSetFlags(e, 0);
+ skylight.setField(e, field_target_pid, a.pid);
+ if (a.window != 0) {
+ // Relative to the window: the live test missed with screen points or none.
+ skylight.setWindowLocation(e, x - a.origin[0], y - a.origin[1]);
+ skylight.setField(e, field_click_state, state);
+ skylight.setField(e, field_button, button);
+ skylight.setField(e, field_subtype, subtype_touch);
+ skylight.setField(e, field_window_number, a.window);
+ skylight.setField(e, field_click_group, a.group);
+ skylight.setField(e, field_window_under_pointer, a.window);
+ skylight.setField(e, field_window_handling, a.window);
}
- return ev;
+ skylight.post(a.pid, e);
}
};
+/// A mouse event's type and button number.
+fn press_kinds(button: Button) struct { down: u32, up: u32, number: u32 } {
+ return switch (button) {
+ .left => .{ .down = 1, .up = 2, .number = 0 },
+ .right => .{ .down = 3, .up = 4, .number = 1 },
+ };
+}
+
pub fn move(t: Target, x: f64, y: f64) void {
- t.post(t.mouse(mouse_moved, x, y, 0));
+ t.postPointer(CGEventCreateMouseEvent(null, mouse_moved, .{ .x = x, .y = y }, 0), x, y, 0, 0);
+}
+
+/// Gestures so far. Only the backend's queue sends events.
+var next_group: i64 = 0;
+
+/// Starts a gesture: a move first, so an app in the background knows where the
+/// pointer now is before a press arrives.
+fn prepare(t: Target, x: f64, y: f64) Target {
+ var out = t;
+ if (out == .app) {
+ next_group += 1;
+ out.app.group = next_group;
+ }
+ move(out, x, y);
+ _ = usleep(15_000);
+ return out;
}
/// Clicks `count` times at x, y (a double click is two, the second with
/// click state 2, as the system makes it).
pub fn click(t: Target, x: f64, y: f64, button: Button, count: u32) void {
- const down: u32, const up: u32, const b: u32 = switch (button) {
- .left => .{ 1, 2, 0 },
- .right => .{ 3, 4, 1 },
- };
- move(t, x, y);
- _ = usleep(15_000);
+ const k = press_kinds(button);
+ const at: cf.CGPoint = .{ .x = x, .y = y };
+ const ready = prepare(t, x, y);
for (1..@min(count, 3) + 1) |n| {
- for ([_]u32{ down, up }) |kind| {
- const ev = t.mouse(kind, x, y, b) orelse continue;
- CGEventSetIntegerValueField(ev, field_click_state, @intCast(n));
- t.post(ev);
- _ = usleep(8_000);
- }
+ const state: i64 = @intCast(n);
+ const down = CGEventCreateMouseEvent(null, k.down, at, k.number);
+ if (ready == .hid) if (down) |d| CGEventSetIntegerValueField(d, field_click_state, state);
+ ready.postPointer(down, x, y, state, k.number);
+ // A control's tracking loop polls for the release: too soon and it's missed.
+ _ = usleep(28_000);
+ const up = CGEventCreateMouseEvent(null, k.up, at, k.number);
+ if (ready == .hid) if (up) |u| CGEventSetIntegerValueField(u, field_click_state, state);
+ ready.postPointer(up, x, y, state, k.number);
+ _ = usleep(30_000);
}
}
@@ -131,21 +192,32 @@ fn unicode(t: Target, units: []const u16) void {
_ = usleep(4_000);
}
+/// The next wheel event's share of `left` lines.
+pub fn wheelStep(left: i32) i32 {
+ return std.math.clamp(left, -lines_per_event, lines_per_event);
+}
+
/// Scrolls at x, y: `dy` lines down, `dx` lines right.
pub fn scroll(t: Target, x: f64, y: f64, dx: f64, dy: f64) void {
- if (t == .hid) {
- move(t, x, y);
- _ = usleep(15_000);
- }
- const clamp = struct {
+ const ready = prepare(t, x, y);
+ const whole = struct {
fn f(v: f64) i32 {
return @intFromFloat(std.math.clamp(@round(v), -200, 200));
}
}.f;
- // Wheel 1 is vertical, positive up; wheel 2 horizontal, positive left.
- const ev = CGEventCreateScrollWheelEvent2(null, 1, 2, -clamp(dy), -clamp(dx), 0) orelse return;
- CGEventSetLocation(ev, .{ .x = x, .y = y });
- t.post(ev);
+ var down = whole(dy);
+ var right = whole(dx);
+ while (down != 0 or right != 0) {
+ const sy = wheelStep(down);
+ const sx = wheelStep(right);
+ down -= sy;
+ right -= sx;
+ // Wheel 1 is vertical, positive up; wheel 2 horizontal, positive left.
+ const ev = CGEventCreateScrollWheelEvent2(null, 1, 2, -sy, -sx, 0);
+ if (ev) |e| CGEventSetLocation(e, .{ .x = x, .y = y });
+ ready.postPointer(ev, x, y, 0, 0);
+ _ = usleep(30_000);
+ }
}
test "characters in UTF-16" {
@@ -156,3 +228,10 @@ test "characters in UTF-16" {
try std.testing.expectEqual(@as(u16, 0xD83D), u[0]);
try std.testing.expectEqual(@as(u16, 0xDE00), u[1]);
}
+
+test "a long scroll goes out in steps" {
+ try std.testing.expectEqual(@as(i32, 10), wheelStep(40));
+ try std.testing.expectEqual(@as(i32, -10), wheelStep(-40));
+ try std.testing.expectEqual(@as(i32, 3), wheelStep(3));
+ try std.testing.expectEqual(@as(i32, 0), wheelStep(0));
+}
diff --git a/apps/nanocua/src/mac/pointer.zig b/apps/nanocua/src/mac/pointer.zig
new file mode 100644
index 00000000..afa6d061
--- /dev/null
+++ b/apps/nanocua/src/mac/pointer.zig
@@ -0,0 +1,80 @@
+//! Whether the click nanocua's window is handling came from a person's mouse
+//! or trackpad. Turning a switch on only counts for a real click, so a program
+//! that posts mouse events can't press Allow. It's a watch, not a filter: Stop
+//! and the off positions work however they're clicked. Assistive tools that post
+//! their own clicks (Switch Control, Voice Control) count as programs here.
+const std = @import("std");
+const objc = @import("nanoui").objc;
+
+extern "c" fn CGEventGetIntegerValueField(ev: *anyopaque, field: u32) i64;
+
+const field_source_pid: u32 = 41;
+const field_source_state: u32 = 45;
+/// kCGEventSourceStateHIDSystemState: the hardware's own state.
+const hid_state: i64 = 1;
+
+const left_down: usize = 1 << 1;
+const left_up: usize = 1 << 2;
+const type_left_down: usize = 1;
+
+/// What a real click looked like in the live test: posted by no process, in
+/// the hardware's state. Anything posted by a program names its process.
+pub fn isHardware(source_pid: i64, source_state: i64) bool {
+ return source_pid == 0 and source_state == hid_state;
+}
+
+var down_real = false;
+var up_real = false;
+
+/// Whether the press and the release just handled were both real. Main thread.
+pub fn real() bool {
+ return down_real and up_real;
+}
+
+fn note(event: objc.Object) void {
+ const cg = objc.msg(?*anyopaque, event, "CGEvent", .{});
+ const is_real = if (cg) |e| isHardware(CGEventGetIntegerValueField(e, field_source_pid), CGEventGetIntegerValueField(e, field_source_state)) else false;
+ if (objc.msg(usize, event, "type", .{}) == type_left_down) {
+ down_real = is_real;
+ up_real = false;
+ } else up_real = is_real;
+}
+
+const Descriptor = extern struct { reserved: usize = 0, size: usize };
+
+/// A block returning the event: objc.Block's return nothing, and this API
+/// drops an event its handler returns nothing for.
+const Monitor = extern struct {
+ isa: *anyopaque,
+ flags: i32 = 0,
+ reserved: i32 = 0,
+ invoke: *const fn (*const Monitor, objc.Object) callconv(.c) objc.Object,
+ descriptor: *const Descriptor,
+};
+
+const descriptor: Descriptor = .{ .size = @sizeOf(Monitor) };
+extern "c" var _NSConcreteStackBlock: opaque {};
+
+fn seen(_: *const Monitor, event: objc.Object) callconv(.c) objc.Object {
+ note(event);
+ return event;
+}
+
+/// Starts watching this process's clicks, before its windows handle them.
+pub fn watch() void {
+ var block: Monitor = .{ .isa = @ptrCast(&_NSConcreteStackBlock), .invoke = seen, .descriptor = &descriptor };
+ // AppKit copies the block and holds the monitor for the life of the app.
+ const monitor = objc.msg(?objc.Object, objc.class("NSEvent"), "addLocalMonitorForEventsMatchingMask:handler:", .{ left_down | left_up, @as(*anyopaque, @ptrCast(&block)) });
+ if (monitor) |m| _ = objc.retain(m);
+}
+
+test "only an event no process posted, in the hardware's state, is real" {
+ try std.testing.expect(isHardware(0, 1));
+ try std.testing.expect(!isHardware(4242, 1));
+ try std.testing.expect(!isHardware(0, 0));
+ try std.testing.expect(!isHardware(4242, 0));
+}
+
+test "nothing has clicked yet: not real" {
+ try std.testing.expect(!real());
+}
diff --git a/apps/nanocua/src/mac/skylight.zig b/apps/nanocua/src/mac/skylight.zig
new file mode 100644
index 00000000..b5067d4a
--- /dev/null
+++ b/apps/nanocua/src/mac/skylight.zig
@@ -0,0 +1,50 @@
+//! What plain CGEventPostToPid can't do for a window in the background: mouse
+//! events an app acts on. These are private functions, looked up at run time, so
+//! a macOS without them falls back to the public post (and the reply already says
+//! a background click isn't confirmed). The recipe follows trycua's cua-driver (MIT).
+const std = @import("std");
+const cf = @import("cf.zig");
+
+extern "c" fn dlopen(path: [*:0]const u8, mode: c_int) ?*anyopaque;
+extern "c" fn dlsym(handle: ?*anyopaque, name: [*:0]const u8) ?*anyopaque;
+extern "c" fn CGEventPostToPid(pid: i32, ev: *anyopaque) void;
+extern "c" fn CGEventSetIntegerValueField(ev: *anyopaque, field: u32, value: i64) void;
+
+const rtld_default: ?*anyopaque = @ptrFromInt(@as(usize, @bitCast(@as(isize, -2))));
+const rtld_lazy: c_int = 1;
+
+const Api = struct {
+ post_to_pid: ?*const fn (pid: i32, ev: *anyopaque) callconv(.c) void = null,
+ set_window_location: ?*const fn (ev: *anyopaque, at: cf.CGPoint) callconv(.c) void = null,
+};
+
+var api: Api = .{};
+var loaded = false;
+
+/// The backend's queue is the only caller, so no lock.
+fn get() *const Api {
+ if (loaded) return &api;
+ loaded = true;
+ _ = dlopen("/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight", rtld_lazy);
+ api = .{
+ .post_to_pid = @ptrCast(@alignCast(dlsym(rtld_default, "SLEventPostToPid"))),
+ .set_window_location = @ptrCast(@alignCast(dlsym(rtld_default, "CGEventSetWindowLocation"))),
+ };
+ return &api;
+}
+
+/// Hands a mouse or scroll event to one app.
+pub fn post(pid: i32, ev: *anyopaque) void {
+ if (get().post_to_pid) |f| return f(pid, ev);
+ CGEventPostToPid(pid, ev);
+}
+
+/// Says where in its window an event is, which an app in the background can't
+/// work out from the pointer.
+pub fn setWindowLocation(ev: *anyopaque, x: f64, y: f64) void {
+ if (get().set_window_location) |f| f(ev, .{ .x = x, .y = y });
+}
+
+pub fn setField(ev: *anyopaque, field: u32, value: i64) void {
+ CGEventSetIntegerValueField(ev, field, value);
+}
diff --git a/apps/nanocua/src/mac/windows.zig b/apps/nanocua/src/mac/windows.zig
index 3bacd7cb..a7dedd92 100644
--- a/apps/nanocua/src/mac/windows.zig
+++ b/apps/nanocua/src/mac/windows.zig
@@ -36,6 +36,17 @@ fn int(v: ?*anyopaque) ?i64 {
return if (cf.CFNumberGetValue(@ptrCast(x), number_sint64, &n)) n else null;
}
+fn read(d: cf.CFDictionaryRef, pid: i32) ?Window {
+ if (int(CFDictionaryGetValue(d, kCGWindowOwnerPID)) != pid) return null;
+ if (int(CFDictionaryGetValue(d, kCGWindowLayer)) != 0) return null;
+ var w: Window = .{ .id = @intCast(int(CFDictionaryGetValue(d, kCGWindowNumber)) orelse return null), .frame = .{} };
+ const bounds = CFDictionaryGetValue(d, kCGWindowBounds) orelse return null;
+ if (!CGRectMakeWithDictionaryRepresentation(@ptrCast(bounds), &w.frame)) return null;
+ if (w.frame.size.width < 2 or w.frame.size.height < 2) return null;
+ w.onscreen = cf.boolean(CFDictionaryGetValue(d, kCGWindowIsOnscreen)) orelse false;
+ return w;
+}
+
/// App `pid`'s main window, if it has one.
pub fn main(pid: i32) ?Window {
const list = CGWindowListCopyWindowInfo(exclude_desktop, 0) orelse return null;
@@ -44,25 +55,53 @@ pub fn main(pid: i32) ?Window {
const n: usize = @intCast(cf.CFArrayGetCount(list));
for (0..n) |i| {
const d: cf.CFDictionaryRef = @ptrCast(cf.CFArrayGetValueAtIndex(list, @intCast(i)) orelse continue);
- if (int(CFDictionaryGetValue(d, kCGWindowOwnerPID)) != pid) continue;
- if (int(CFDictionaryGetValue(d, kCGWindowLayer)) != 0) continue;
- var w: Window = .{ .id = @intCast(int(CFDictionaryGetValue(d, kCGWindowNumber)) orelse continue), .frame = .{} };
- const bounds = CFDictionaryGetValue(d, kCGWindowBounds) orelse continue;
- if (!CGRectMakeWithDictionaryRepresentation(@ptrCast(bounds), &w.frame)) continue;
- if (w.frame.size.width < 2 or w.frame.size.height < 2) continue;
- w.onscreen = cf.boolean(CFDictionaryGetValue(d, kCGWindowIsOnscreen)) orelse false;
+ const w = read(d, pid) orelse continue;
if (best == null or better(w, best.?)) best = w;
}
return best;
}
+pub fn contains(w: Window, x: f64, y: f64) bool {
+ const f = w.frame;
+ return x >= f.origin.x and y >= f.origin.y and x < f.origin.x + f.size.width and y < f.origin.y + f.size.height;
+}
+
+/// App `pid`'s window at x, y: the frontmost that shows there, as the list runs
+/// front to back. Else one that isn't on screen just now (another Space).
+pub fn at(pid: i32, x: f64, y: f64) ?Window {
+ const list = CGWindowListCopyWindowInfo(exclude_desktop, 0) orelse return null;
+ defer cf.CFRelease(list);
+ var hidden: ?Window = null;
+ const n: usize = @intCast(cf.CFArrayGetCount(list));
+ for (0..n) |i| {
+ const d: cf.CFDictionaryRef = @ptrCast(cf.CFArrayGetValueAtIndex(list, @intCast(i)) orelse continue);
+ const w = read(d, pid) orelse continue;
+ if (!contains(w, x, y)) continue;
+ if (w.onscreen) return w;
+ if (hidden == null) hidden = w;
+ }
+ return hidden;
+}
+
+/// A window's frame as x, y, width, height.
+pub fn box(w: Window) [4]f64 {
+ return .{ w.frame.origin.x, w.frame.origin.y, w.frame.size.width, w.frame.size.height };
+}
+
test "the main window: on screen first, then the largest" {
- const at = struct {
+ const of = struct {
fn f(w: f64, h: f64, on: bool) Window {
return .{ .id = 1, .frame = .{ .size = .{ .width = w, .height = h } }, .onscreen = on };
}
}.f;
- try std.testing.expect(better(at(1352, 802, true), at(1352, 47, true)));
- try std.testing.expect(!better(at(1352, 47, true), at(1352, 802, true)));
- try std.testing.expect(better(at(10, 10, true), at(1352, 849, false)));
+ try std.testing.expect(better(of(1352, 802, true), of(1352, 47, true)));
+ try std.testing.expect(!better(of(1352, 47, true), of(1352, 802, true)));
+ try std.testing.expect(better(of(10, 10, true), of(1352, 849, false)));
+}
+
+test "a window contains the points in its frame" {
+ const w: Window = .{ .id = 1, .frame = .{ .origin = .{ .x = 100, .y = 50 }, .size = .{ .width = 200, .height = 100 } } };
+ try std.testing.expect(contains(w, 100, 50) and contains(w, 299, 149));
+ try std.testing.expect(!contains(w, 300, 100) and !contains(w, 10, 10));
+ try std.testing.expectEqual([4]f64{ 100, 50, 200, 100 }, box(w));
}
diff --git a/apps/nanocua/src/main.zig b/apps/nanocua/src/main.zig
index 238ecaf1..9c09911e 100644
--- a/apps/nanocua/src/main.zig
+++ b/apps/nanocua/src/main.zig
@@ -1,17 +1,26 @@
//! nanocua: lets AI agents see and use the computer, as far as its user
//! allows. `nanocua` is the app: the control panel, and the server agents
//! reach over a Unix socket. `nanocua mcp` is what an agent runs: an MCP
-//! server on stdio that carries its lines to the app.
+//! server on stdio that carries its lines to the app. `nanocua ctl` is what
+//! graff and Harness run to show and change the switches.
const std = @import("std");
const nanoui = @import("nanoui");
const app_mod = @import("app.zig");
const theme = @import("theme.zig");
+const builtin = @import("builtin");
pub fn main(init: std.process.Init) !u8 {
const args = try init.minimal.args.toSlice(init.arena.allocator());
if (args.len > 1 and std.mem.eql(u8, args[1], "mcp"))
return if (comptime @import("server.zig").supported) @import("proxy.zig").run(init) else 1;
+ if (args.len > 1 and std.mem.eql(u8, args[1], "ctl"))
+ return if (comptime @import("server.zig").supported) @import("ctl.zig").run(init, args[2..]) else 1;
+ for (args) |arg| if (@import("control.zig").devFlagRefused(builtin.mode, arg)) {
+ std.debug.print("nanocua: {s} is only for debug builds\n", .{arg});
+ return 2;
+ };
+
const app = try nanoui.App.create(init, .{ .name = "nanocua" });
defer app.destroy();
var plat: app_mod.Platform = .init(init.gpa);
@@ -20,12 +29,16 @@ pub fn main(init: std.process.Init) !u8 {
plat.attach(&a);
defer a.deinit();
var path_buf: [512]u8 = undefined;
- if (@import("sockpath.zig").path(&path_buf, init.environ_map)) |path| a.listen(path) else |_| {
+ if (@import("sockpath.zig").path(&path_buf, init.environ_map, @import("sockpath.zig").realHome())) |path| a.listen(path) else |_| {
a.listen_error = "No folder for the socket (HOME is unset).";
}
var exe_buf: [1024]u8 = undefined;
if (std.process.executablePath(init.io, &exe_buf)) |n| a.setCommand(exe_buf[0..n]) else |_| {}
a.stopKey();
+ if (comptime app_mod.mac) {
+ @import("mac/pointer.zig").watch();
+ a.hidden = app.flags.hidden;
+ }
a.window = try app.newWindow(.{
.title = "nanocua",
.width = 560,
diff --git a/apps/nanocua/src/mcp.zig b/apps/nanocua/src/mcp.zig
index 545f9b92..88132f58 100644
--- a/apps/nanocua/src/mcp.zig
+++ b/apps/nanocua/src/mcp.zig
@@ -21,6 +21,14 @@ pub const Sink = struct {
pub const Backend = struct {
ptr: *anyopaque,
call: *const fn (ptr: *anyopaque, tool: tools.Tool, args: tools.Args, reply: *Reply) void,
+ /// A connection went away: what it held can go too.
+ closed: ?*const fn (ptr: *anyopaque, conn: u64) void = null,
+};
+
+/// The switches, for `nanocua/*` calls (control.zig). `id` lasts only the call.
+pub const Control = struct {
+ ptr: *anyopaque,
+ func: *const fn (ptr: *anyopaque, conn: u64, id: []const u8, method: []const u8, params: ?std.json.Value) void,
};
pub const Content = union(enum) {
@@ -46,6 +54,13 @@ pub const Hub = struct {
changed: ?Callback = null,
/// Calls not yet answered.
pending: usize = 0,
+ /// Null where nothing can answer them (`nanocua mcp`'s own hub): agents
+ /// reach the switches only by opening the socket themselves.
+ control: ?Control = null,
+
+ pub fn closed(h: *Hub, conn: u64) void {
+ if (h.backend.closed) |f| f(h.backend.ptr, conn);
+ }
/// Handles one line from connection `conn`.
pub fn handle(h: *Hub, conn: u64, line: []const u8) void {
@@ -74,6 +89,8 @@ pub const Hub = struct {
h.sendResult(conn, id, aw.written());
} else if (std.mem.eql(u8, method, "tools/call")) {
h.call(conn, id, params);
+ } else if (std.mem.startsWith(u8, method, "nanocua/") and h.control != null) {
+ h.control.?.func(h.control.?.ptr, conn, id, method, params);
} else h.sendError(conn, id, -32601, "Method not found");
}
@@ -101,14 +118,14 @@ pub const Hub = struct {
if (h.changed) |c| c.func(c.ptr);
}
- fn sendResult(h: *Hub, conn: u64, id: []const u8, result: []const u8) void {
+ pub fn sendResult(h: *Hub, conn: u64, id: []const u8, result: []const u8) void {
var aw: std.Io.Writer.Allocating = .init(h.gpa);
defer aw.deinit();
aw.writer.print("{{\"jsonrpc\":\"2.0\",\"id\":{s},\"result\":{s}}}", .{ id, result }) catch return;
h.sink.send(h.sink.ptr, conn, aw.written());
}
- fn sendError(h: *Hub, conn: u64, id: []const u8, code: i32, message: []const u8) void {
+ pub fn sendError(h: *Hub, conn: u64, id: []const u8, code: i32, message: []const u8) void {
var buf: [256]u8 = undefined;
const line = std.fmt.bufPrint(&buf, "{{\"jsonrpc\":\"2.0\",\"id\":{s},\"error\":{{\"code\":{d},\"message\":\"{s}\"}}}}", .{ id, code, message }) catch return;
h.sink.send(h.sink.ptr, conn, line);
diff --git a/apps/nanocua/src/proxy.zig b/apps/nanocua/src/proxy.zig
index fc1e7907..16fa4d1a 100644
--- a/apps/nanocua/src/proxy.zig
+++ b/apps/nanocua/src/proxy.zig
@@ -23,7 +23,7 @@ pub fn run(init: std.process.Init) !u8 {
var buf: [512]u8 = undefined;
var p: Proxy = .{
.gpa = init.gpa,
- .path = try sockpath.path(&buf, init.environ_map),
+ .path = try sockpath.path(&buf, init.environ_map, sockpath.realHome()),
.hub = .{
.gpa = init.gpa,
.sink = .{ .ptr = undefined, .send = toStdout },
@@ -147,7 +147,20 @@ const Proxy = struct {
}
};
-fn connect(path: [:0]const u8) ?c_int {
+/// The app's socket, opening the app if it isn't running; null if it can't.
+pub fn connectOrOpen(path: [:0]const u8) ?c_int {
+ if (connect(path)) |fd| return fd;
+ if (!launch()) return null;
+ // The app starts in well under this; its first launch may ask the user
+ // something first.
+ for (0..100) |_| {
+ _ = usleep(100_000);
+ if (connect(path)) |fd| return fd;
+ }
+ return null;
+}
+
+pub fn connect(path: [:0]const u8) ?c_int {
var addr: std.c.sockaddr.un = .{ .path = @splat(0) };
if (path.len >= addr.path.len) return null;
@memcpy(addr.path[0..path.len], path);
diff --git a/apps/nanocua/src/server.zig b/apps/nanocua/src/server.zig
index 9db4f941..ffdc03d7 100644
--- a/apps/nanocua/src/server.zig
+++ b/apps/nanocua/src/server.zig
@@ -50,6 +50,8 @@ const Conn = struct {
fd: c_int,
watch: ?fdwatch.Watch = null,
lines: Lines = .{},
+ /// It uses the switches, so it isn't an agent.
+ control: bool = false,
fn readable(ctx: ?*anyopaque) callconv(.c) void {
if (comptime !supported) return;
@@ -124,8 +126,15 @@ pub const Server = struct {
return c.id;
}
+ /// Agents connected.
pub fn clients(s: *const Server) usize {
- return s.conns.count();
+ var n: usize = 0;
+ for (s.conns.values()) |c| n += @intFromBool(!c.control);
+ return n;
+ }
+
+ pub fn markControl(s: *Server, id: u64) void {
+ if (s.conns.get(id)) |c| c.control = true;
}
fn changed(s: *Server) void {
@@ -141,6 +150,7 @@ pub const Server = struct {
fn drop(s: *Server, id: u64) void {
const kv = s.conns.fetchSwapRemove(id) orelse return;
const c = kv.value;
+ s.hub.closed(id);
if (c.watch) |w| w.cancel();
_ = close(c.fd);
c.lines.deinit(s.gpa);
diff --git a/apps/nanocua/src/sockpath.zig b/apps/nanocua/src/sockpath.zig
index 88ef9202..7420083a 100644
--- a/apps/nanocua/src/sockpath.zig
+++ b/apps/nanocua/src/sockpath.zig
@@ -1,16 +1,25 @@
//! Where the app listens and `nanocua mcp` connects: a Unix socket only
//! this user can open, in nanocua's Application Support folder on macOS,
-//! the user's runtime folder on Linux; NANOCUA_SOCKET overrides it.
+//! the user's runtime folder on Linux; NANOCUA_SOCKET overrides it. `nanocua
+//! mcp` and `nanocua ctl` find it the same way, whoever starts them.
const std = @import("std");
const builtin = @import("builtin");
extern "c" fn mkdir(path: [*:0]const u8, mode: c_uint) c_int;
extern "c" fn getuid() c_uint;
-pub fn path(buf: []u8, env: *const std.process.Environ.Map) ![:0]const u8 {
+/// The user's real home, not $HOME: Harness runs graff with HOME set per account,
+/// and every program on one Mac has to find the one app.
+pub fn realHome() ?[]const u8 {
+ if (builtin.os.tag != .macos) return null;
+ const entry = std.c.getpwuid(getuid()) orelse return null;
+ return std.mem.span(entry.dir orelse return null);
+}
+
+pub fn path(buf: []u8, env: *const std.process.Environ.Map, real_home: ?[]const u8) ![:0]const u8 {
if (env.get("NANOCUA_SOCKET")) |p| return std.fmt.bufPrintSentinel(buf, "{s}", .{p}, 0);
if (builtin.os.tag == .macos) {
- const home = env.get("HOME") orelse return error.NoHome;
+ const home = real_home orelse env.get("HOME") orelse return error.NoHome;
return std.fmt.bufPrintSentinel(buf, "{s}/Library/Application Support/nanocua/mcp.sock", .{home}, 0);
}
if (env.get("XDG_RUNTIME_DIR")) |d| return std.fmt.bufPrintSentinel(buf, "{s}/nanocua.sock", .{d}, 0);
@@ -32,11 +41,21 @@ test "the path: overridden, or the platform's" {
var buf: [256]u8 = undefined;
try env.put("HOME", "/Users/ada");
try env.put("XDG_RUNTIME_DIR", "/run/user/501");
- const p = try path(&buf, &env);
+ const p = try path(&buf, &env, null);
if (builtin.os.tag == .macos)
try std.testing.expectEqualStrings("/Users/ada/Library/Application Support/nanocua/mcp.sock", p)
else
try std.testing.expectEqualStrings("/run/user/501/nanocua.sock", p);
try env.put("NANOCUA_SOCKET", "/tmp/x.sock");
- try std.testing.expectEqualStrings("/tmp/x.sock", try path(&buf, &env));
+ try std.testing.expectEqualStrings("/tmp/x.sock", try path(&buf, &env, null));
+}
+
+test "another $HOME doesn't move the socket" {
+ if (builtin.os.tag != .macos) return error.SkipZigTest;
+ var env: std.process.Environ.Map = .init(std.testing.allocator);
+ defer env.deinit();
+ var buf: [256]u8 = undefined;
+ try env.put("HOME", "/Users/ada/.accounts/two");
+ try std.testing.expectEqualStrings("/Users/ada/Library/Application Support/nanocua/mcp.sock", try path(&buf, &env, "/Users/ada"));
+ try std.testing.expect(realHome() != null);
}
diff --git a/apps/nanocua/src/target.zig b/apps/nanocua/src/target.zig
index 3549d5eb..01af54fc 100644
--- a/apps/nanocua/src/target.zig
+++ b/apps/nanocua/src/target.zig
@@ -31,6 +31,26 @@ pub fn route(a: Ask) Route {
return .{ .app = pid };
}
+/// Where in `el` (x, y, width, height) to send an event so that it lands inside
+/// `window`: the middle of what they share. An element can be far larger than its
+/// window (a long document's text), and its own middle out of sight.
+pub fn visiblePoint(el: [4]f64, window: [4]f64) ?[2]f64 {
+ const left = @max(el[0], window[0]);
+ const top = @max(el[1], window[1]);
+ const right = @min(el[0] + el[2], window[0] + window[2]);
+ const bottom = @min(el[1] + el[3], window[1] + window[3]);
+ if (right <= left or bottom <= top) return null;
+ return .{ (left + right) / 2, (top + bottom) / 2 };
+}
+
+test "a long element's point stays inside its window" {
+ const window = [4]f64{ 128, 61, 586, 488 };
+ // A document 3000 points tall, as the live test met it.
+ try std.testing.expectEqual(@as(?[2]f64, .{ 421, 305 }), visiblePoint(.{ 128, 61, 586, 3000 }, window));
+ try std.testing.expectEqual(@as(?[2]f64, .{ 150, 80 }), visiblePoint(.{ 140, 70, 20, 20 }, window));
+ try std.testing.expectEqual(@as(?[2]f64, null), visiblePoint(.{ 128, 900, 586, 40 }, window));
+}
+
test "a call goes into the app it names, else the last one read" {
try std.testing.expectEqual(Route{ .app = 7 }, route(.{ .pid = 7, .last = 9, .self = 1 }));
try std.testing.expectEqual(Route{ .app = 9 }, route(.{ .last = 9, .self = 1 }));
diff --git a/apps/nanocua/src/tests.zig b/apps/nanocua/src/tests.zig
index 9e7a251d..de4c8409 100644
--- a/apps/nanocua/src/tests.zig
+++ b/apps/nanocua/src/tests.zig
@@ -13,10 +13,14 @@ test {
_ = @import("keys.zig");
_ = @import("tree.zig");
_ = @import("target.zig");
+ _ = @import("control.zig");
_ = @import("server.zig");
_ = @import("sockpath.zig");
_ = @import("tests_app.zig");
- if (comptime @import("server.zig").supported) _ = @import("proxy.zig");
+ if (comptime @import("server.zig").supported) {
+ _ = @import("proxy.zig");
+ _ = @import("ctl.zig");
+ }
std.testing.refAllDecls(@import("server.zig").Server);
std.testing.refAllDecls(@import("app.zig").App);
std.testing.refAllDecls(@import("app.zig").Platform);
@@ -24,6 +28,10 @@ test {
_ = @import("mac/cf.zig");
_ = @import("mac/windows.zig");
_ = @import("mac/input.zig");
+ _ = @import("mac/skylight.zig");
+ _ = @import("mac/hit.zig");
+ _ = @import("mac/pointer.zig");
+ _ = @import("mac/backend.zig");
inline for (.{ @import("mac/backend.zig"), @import("mac/ax.zig"), @import("mac/input.zig"), @import("mac/capture.zig"), @import("mac/apps.zig"), @import("mac/windows.zig"), @import("mac/hotkey.zig"), @import("proxy.zig") }) |m| std.testing.refAllDecls(m);
std.testing.refAllDecls(@import("mac/backend.zig").Mac);
std.testing.refAllDecls(@import("mac/hotkey.zig").HotKey);
diff --git a/apps/nanocua/src/tests_app.zig b/apps/nanocua/src/tests_app.zig
index 1d8fa12a..c9313fa5 100644
--- a/apps/nanocua/src/tests_app.zig
+++ b/apps/nanocua/src/tests_app.zig
@@ -1,5 +1,6 @@
//! The panel through the Tester (Stop and Resume, input, the log, the
-//! command's Copy) and the socket end to end over a socket pair, with a
+//! command's Copy), the switches from outside, and the socket end to end over
+//! a socket pair, with a
//! backend that only records: nothing here touches the real screen.
const std = @import("std");
const builtin = @import("builtin");
@@ -51,6 +52,15 @@ test "the panel: stop and resume, input, the log, copying the command" {
try std.testing.expect(t.hasText("Allow…"));
}
+ // Asking for Screen Recording says what's left to do, which is the user's.
+ if (comptime builtin.os.tag == .macos) {
+ try std.testing.expect(!t.hasText(view.screen_hint));
+ try t.clickText("Allow…");
+ t.settle(0.2);
+ try std.testing.expect(a.asked_screen);
+ try std.testing.expect(t.hasText(view.screen_hint));
+ }
+
// Looking works; acting is refused until input is on.
call(&a, 1, "get_ui_tree", "{\"app\":\"Notes\"}");
call(&a, 2, "click", "{\"ref\":9}");
@@ -104,6 +114,195 @@ test "the panel: stop and resume, input, the log, copying the command" {
try std.testing.expect(t.hasText("Nothing yet. Every call an agent makes shows here."));
}
+test "the switches stay as the user left them: calls and redraws don't flip them" {
+ const gpa = std.testing.allocator;
+ var rec: Recorder = .{};
+ var a: App = undefined;
+ a.init(gpa, .{ .ptr = &rec, .call = Recorder.call });
+ defer a.deinit();
+ a.dark = false;
+ var t: ui.Tester = undefined;
+ try t.init(gpa, .init(&a, view.view), .{ .width = 560, .height = 720, .theme = theme.ui(false) });
+ defer t.deinit();
+ const whole = "Allow whole screen: your pointer and keyboard, pictures of everything";
+
+ // Turning input on leaves the whole screen off, whatever comes after.
+ try t.clickText("Allow input: clicks, typing, keys, opening apps");
+ t.settle(0.5);
+ try std.testing.expect(a.hub.policy.allow_input);
+ try std.testing.expect(!a.hub.policy.allow_whole_screen);
+ call(&a, 1, "screenshot", "{\"whole_screen\":true}");
+ t.settle(0.5);
+ try std.testing.expect(!a.hub.policy.allow_whole_screen);
+
+ // One click, one flip; off stays off through calls, redraws and the pointer.
+ try t.clickText(whole);
+ t.settle(0.5);
+ try std.testing.expect(a.hub.policy.allow_whole_screen);
+ try t.clickText(whole);
+ t.settle(0.5);
+ try std.testing.expect(!a.hub.policy.allow_whole_screen);
+ try std.testing.expect(!t.hasText("Whole screen"));
+ for (0..5) |i| {
+ call(&a, @intCast(i + 2), "get_ui_tree", "{}");
+ t.frame();
+ t.moveTo(300, @floatFromInt(300 + i * 10));
+ t.settle(0.2);
+ try std.testing.expect(!a.hub.policy.allow_whole_screen);
+ try std.testing.expect(a.hub.policy.allow_input);
+ }
+}
+
+/// The hub's clock and what it sends, for the control tests.
+const Wire = struct {
+ gpa: std.mem.Allocator,
+ sent: std.ArrayList(u8) = .empty,
+ var now: f64 = 0;
+ fn clock() f64 {
+ return now;
+ }
+ fn send(ptr: *anyopaque, conn: u64, line: []const u8) void {
+ const w: *Wire = @ptrCast(@alignCast(ptr));
+ w.sent.print(w.gpa, "{d}:{s}\n", .{ conn, line }) catch {};
+ }
+ fn saw(w: *Wire, text: []const u8) bool {
+ return std.mem.indexOf(u8, w.sent.items, text) != null;
+ }
+};
+
+fn control(a: *App, id: u32, method: []const u8, params: []const u8) void {
+ var buf: [256]u8 = undefined;
+ a.hub.handle(9, std.fmt.bufPrint(&buf, "{{\"jsonrpc\":\"2.0\",\"id\":{d},\"method\":\"nanocua/{s}\",\"params\":{s}}}", .{ id, method, params }) catch unreachable);
+}
+
+fn notReal() bool {
+ return false;
+}
+
+test "from outside: off at once, on only by a click in the window" {
+ const gpa = std.testing.allocator;
+ var rec: Recorder = .{};
+ var wire: Wire = .{ .gpa = gpa };
+ defer wire.sent.deinit(gpa);
+ var a: App = undefined;
+ a.init(gpa, .{ .ptr = &rec, .call = Recorder.call });
+ defer a.deinit();
+ a.dark = false;
+ Wire.now = 100;
+ a.hub.clock = Wire.clock;
+ a.hub.sink = .{ .ptr = &wire, .send = Wire.send };
+ var t: ui.Tester = undefined;
+ try t.init(gpa, .init(&a, view.view), .{ .width = 560, .height = 720, .theme = theme.ui(false) });
+ defer t.deinit();
+
+ // Asking changes nothing, answers nothing yet, and shows the question.
+ control(&a, 1, "ask", "{\"what\":\"input\",\"from\":\"graff\"}");
+ try std.testing.expect(!a.hub.policy.allow_input);
+ try std.testing.expectEqual(@as(usize, 0), wire.sent.items.len);
+ t.settle(0.2);
+ try std.testing.expect(t.hasText("A request from graff"));
+ try std.testing.expect(t.hasText(@import("control.zig").Want.input.question()));
+
+ // A click right away can't answer: it may have been meant for something else.
+ try t.clickText("Allow");
+ t.settle(0.2);
+ try std.testing.expect(!a.hub.policy.allow_input);
+
+ // The status says what's waiting; a second question has to wait its turn.
+ control(&a, 2, "status", "{}");
+ try std.testing.expect(wire.saw("9:{\"jsonrpc\":\"2.0\",\"id\":2,\"result\":{"));
+ try std.testing.expect(wire.saw("\"pending\":\"input\""));
+ control(&a, 3, "ask", "{\"what\":\"whole_screen\"}");
+ try std.testing.expect(wire.saw("\"outcome\":\"busy\""));
+
+ // A program's click, once it's armed, is ignored, and the panel says so.
+ Wire.now += 1;
+ a.real_pointer = notReal;
+ t.frame();
+ try t.clickText("Allow");
+ t.settle(0.2);
+ try std.testing.expect(!a.hub.policy.allow_input);
+ try std.testing.expect(a.gate.request != null);
+ try std.testing.expect(t.hasText(view.ignored_click));
+
+ // So are the panel's own switches, on the way on...
+ try t.clickText("Allow input: clicks, typing, keys, opening apps");
+ t.settle(0.2);
+ try std.testing.expect(!a.hub.policy.allow_input);
+
+ // A person's click answers it, and whoever asked hears.
+ a.real_pointer = null;
+ wire.sent.clearRetainingCapacity();
+ try t.clickText("Allow");
+ t.settle(0.2);
+ try std.testing.expect(a.hub.policy.allow_input);
+ try std.testing.expect(a.gate.request == null);
+ try std.testing.expect(wire.saw("9:{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{"));
+ try std.testing.expect(wire.saw("\"outcome\":\"allowed\""));
+ try std.testing.expect(!t.hasText("A request from graff"));
+
+ // ...but a program can turn things off, here and from outside.
+ a.real_pointer = notReal;
+ try t.clickText("Allow input: clicks, typing, keys, opening apps");
+ t.settle(0.2);
+ try std.testing.expect(!a.hub.policy.allow_input);
+ a.real_pointer = null;
+ a.setInput(true);
+ a.setWholeScreen(true);
+ control(&a, 4, "off", "{\"what\":\"whole_screen\"}");
+ try std.testing.expect(!a.hub.policy.allow_whole_screen and a.hub.policy.allow_input);
+ control(&a, 5, "off", "{\"what\":\"input\"}");
+ try std.testing.expect(!a.hub.policy.allow_input);
+ wire.sent.clearRetainingCapacity();
+ control(&a, 6, "off", "{\"what\":\"resume\"}");
+ try std.testing.expect(wire.saw("-32602"));
+
+ // Stop works from outside; Resume is a question like the rest.
+ control(&a, 7, "stop", "{}");
+ try std.testing.expect(a.hub.policy.stopped);
+ control(&a, 8, "ask", "{\"what\":\"resume\",\"from\":\"Harness\"}");
+ try std.testing.expect(a.hub.policy.stopped);
+ t.settle(0.2);
+ try std.testing.expect(t.hasText("A request from Harness"));
+
+ // Deny: whoever asked hears, and asking again is refused for a while.
+ wire.sent.clearRetainingCapacity();
+ try t.clickText("Deny");
+ t.settle(0.2);
+ try std.testing.expect(a.hub.policy.stopped);
+ try std.testing.expect(wire.saw("\"outcome\":\"denied\""));
+ wire.sent.clearRetainingCapacity();
+ control(&a, 10, "ask", "{\"what\":\"resume\"}");
+ try std.testing.expect(wire.saw("\"outcome\":\"quiet\""));
+
+ // Unanswered, it goes away and says so. Already on needs no question.
+ Wire.now += 20;
+ a.setStopped(false);
+ wire.sent.clearRetainingCapacity();
+ control(&a, 11, "ask", "{\"what\":\"resume\"}");
+ try std.testing.expect(wire.saw("\"outcome\":\"already_on\""));
+ control(&a, 12, "ask", "{\"what\":\"input\",\"from\":\"graff\"}");
+ Wire.now += 61;
+ wire.sent.clearRetainingCapacity();
+ t.frame();
+ t.settle(0.2);
+ try std.testing.expect(a.gate.request == null);
+ try std.testing.expect(!a.hub.policy.allow_input);
+ try std.testing.expect(wire.saw("9:{\"jsonrpc\":\"2.0\",\"id\":12,\"result\":{"));
+ try std.testing.expect(wire.saw("\"outcome\":\"expired\""));
+}
+
+test "an agent's own connection can't reach the switches through MCP" {
+ // `nanocua mcp` has a hub of its own with no control behind it.
+ const gpa = std.testing.allocator;
+ var wire: Wire = .{ .gpa = gpa };
+ defer wire.sent.deinit(gpa);
+ var rec: Recorder = .{};
+ var h: mcp.Hub = .{ .gpa = gpa, .clock = Wire.clock, .sink = .{ .ptr = &wire, .send = Wire.send }, .backend = .{ .ptr = &rec, .call = Recorder.call } };
+ h.handle(1, "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"nanocua/ask\",\"params\":{\"what\":\"input\"}}");
+ try std.testing.expect(wire.saw("-32601"));
+}
+
extern "c" fn socketpair(domain: c_int, kind: c_int, protocol: c_int, fds: *[2]c_int) c_int;
extern "c" fn close(fd: c_int) c_int;
diff --git a/apps/nanocua/src/tools.zig b/apps/nanocua/src/tools.zig
index 58e5f955..e85d2fcd 100644
--- a/apps/nanocua/src/tools.zig
+++ b/apps/nanocua/src/tools.zig
@@ -52,9 +52,11 @@ pub const Tool = enum {
\\need (canvases, images).
,
.click =>
- \\Clicks in the app you act in, without moving the user's pointer: an element by ref (pressing it through
- \\accessibility when it can, else at its center) or x,y in screen points. button: left (default) or right;
- \\count: 2 for a double click. Some apps ignore background clicks: then use refs, or ask for whole_screen.
+ \\Clicks in the app you act in, without moving the user's pointer: an element by ref, or x,y in screen
+ \\points, inside that app's window. Controls are pressed through accessibility, which is exact. Anything
+ \\else gets a click event that apps can ignore in the background, so the reply says it isn't confirmed:
+ \\check the result. In a window that isn't its app's key window, that first click may only make it key.
+ \\button: left (default) or right; count: 2 for a double click.
,
.type_text =>
\\Types text into the app you act in, without taking the user's keyboard: into its focused element, or
@@ -66,7 +68,7 @@ pub const Tool = enum {
,
.scroll =>
\\Scrolls in the app you act in, at x,y (or over element ref), by dy lines (positive scrolls down) and dx
- \\lines (positive scrolls right).
+ \\lines (positive scrolls right). The reply says whether it moved, when the app has scroll bars to read.
,
.move_mouse =>
\\Hovers at x,y in screen points in the app you act in. The user's pointer stays put unless whole_screen.
diff --git a/apps/nanocua/src/tree.zig b/apps/nanocua/src/tree.zig
index 8a2fff36..53fed4d5 100644
--- a/apps/nanocua/src/tree.zig
+++ b/apps/nanocua/src/tree.zig
@@ -32,6 +32,21 @@ fn bare(role: []const u8) []const u8 {
return if (std.mem.startsWith(u8, role, "AX")) role[2..] else role;
}
+/// A role as it reads in a reply: "Button".
+pub const bareRole = bare;
+
+/// Roles an accessibility press does what a click would.
+const pressed = [_][]const u8{ "Button", "CheckBox", "RadioButton", "PopUpButton", "MenuButton", "MenuItem", "MenuBarItem", "Link", "DisclosureTriangle", "Incrementor" };
+
+pub fn pressable(role: []const u8) bool {
+ return in(bare(role), &pressed);
+}
+
+/// Roles that are likely a control's label or icon, so its control is above.
+pub fn insideControl(role: []const u8) bool {
+ return in(bare(role), &.{ "StaticText", "Image" });
+}
+
fn in(s: []const u8, list: []const []const u8) bool {
for (list) |x| if (std.mem.eql(u8, s, x)) return true;
return false;
@@ -79,6 +94,13 @@ fn quoted(w: *std.Io.Writer, prefix: []const u8, s: []const u8, max: usize) !voi
try w.writeByte('"');
}
+test "what a press reaches" {
+ try std.testing.expect(pressable("AXButton") and pressable("AXLink"));
+ try std.testing.expect(!pressable("AXTextArea") and !pressable("AXGroup"));
+ try std.testing.expect(insideControl("AXStaticText") and !insideControl("AXTextArea"));
+ try std.testing.expectEqualStrings("Button", bareRole("AXButton"));
+}
+
test "lines name what can be acted on and skip plumbing" {
var buf: [512]u8 = undefined;
var w: std.Io.Writer = .fixed(&buf);
diff --git a/apps/nanocua/src/view.zig b/apps/nanocua/src/view.zig
index cca14b88..3418aeda 100644
--- a/apps/nanocua/src/view.zig
+++ b/apps/nanocua/src/view.zig
@@ -6,6 +6,7 @@ const nanoui = @import("nanoui");
const ui = nanoui.ui;
const theme = @import("theme.zig");
const policy = @import("policy.zig");
+const control = @import("control.zig");
const App = @import("app.zig").App;
const mac = @import("app.zig").mac;
@@ -15,6 +16,7 @@ pub fn view(a: *App, c: *ui.Context) void {
const page = ui.div(c).column().sizeFull().p(20).gap(14).bg(k.page);
defer page.end();
header(c, a, k);
+ question(c, a, k);
permissions(c, a, k);
agents(c, a, k);
activity(c, a, k);
@@ -45,10 +47,35 @@ fn header(c: *ui.Context, a: *App, k: theme.Colors) void {
ui.text(c, "Lets AI agents see and use this computer, as far as you allow here.", .{ .size = 12, .color = k.secondary, .wrap = true });
}
if (p.stopped) {
- if (ui.button(c, "Resume", .{ .variant = .primary }).clicked) a.setStopped(false);
+ if (ui.button(c, "Resume", .{ .variant = .primary }).clicked) a.fromPanel(.@"resume", true);
} else if (ui.button(c, "Stop", .{ .variant = .danger }).clicked) a.setStopped(true);
}
+pub const screen_hint = "Turn on nanocua in the list System Settings opened. Not there? Drag nanocua.app from the Finder window into it.";
+
+/// What a program asked for from outside. Only this click turns it on.
+fn question(c: *ui.Context, a: *App, k: theme.Colors) void {
+ const now = a.hub.clock();
+ a.settleRequest();
+ if (now < a.ignored_until) {
+ c.frameIn(@floatCast(a.ignored_until - now));
+ ui.text(c, ignored_click, .{ .size = 12, .color = k.red, .wrap = true });
+ }
+ const r = a.gate.request orelse return;
+ const armed = a.gate.armed(now);
+ c.frameIn(@floatCast(if (armed) r.at + control.lifetime - now else r.at + control.armed_after - now));
+ const d = ui.div(c).column().wFull().p(14).gap(10).bg(k.card).border(k.accent).borderW(2).rounded(12);
+ defer d.end();
+ ui.text(c, c.print("A request from {s}", .{r.from()}), .{ .size = 13, .weight = .semibold, .color = k.accent });
+ ui.text(c, r.want.question(), .{ .size = 14, .color = k.text, .wrap = true });
+ const row = ui.div(c).row().itemsCenter().gap(8);
+ defer row.end();
+ if (ui.button(c, "Allow", .{ .variant = .primary, .disabled = !armed }).clicked) a.answer(true);
+ if (ui.button(c, "Deny", .{}).clicked) a.answer(false);
+}
+
+pub const ignored_click = "That click came from a program, so it was ignored. Only your own mouse or trackpad turns things on.";
+
fn permissions(c: *ui.Context, a: *App, k: theme.Colors) void {
if (comptime !mac) return;
const d = card(c, k);
@@ -56,6 +83,8 @@ fn permissions(c: *ui.Context, a: *App, k: theme.Colors) void {
title(c, k, "Permissions");
permission(c, a, k, "Accessibility", "To read apps' controls, click and type", a.perms.accessibility, .accessibility);
permission(c, a, k, "Screen Recording", "For screenshots", a.perms.screen, .screen);
+ if (a.asked_screen and !a.perms.screen)
+ ui.text(c, screen_hint, .{ .size = 12, .color = k.accent, .wrap = true });
}
fn permission(c: *ui.Context, a: *App, k: theme.Colors, name: []const u8, what: []const u8, on: bool, which: App.Ask) void {
@@ -77,9 +106,9 @@ fn agents(c: *ui.Context, a: *App, k: theme.Colors) void {
defer d.end();
title(c, k, "Agents");
var input = a.hub.policy.allow_input;
- if (ui.toggle(c, "Allow input: clicks, typing, keys, opening apps", &input)) a.setInput(input);
+ if (ui.toggle(c, "Allow input: clicks, typing, keys, opening apps", &input)) a.fromPanel(.input, input);
var whole = a.hub.policy.allow_whole_screen;
- if (ui.toggle(c, "Allow whole screen: your pointer and keyboard, pictures of everything", &whole)) a.setWholeScreen(whole);
+ if (ui.toggle(c, "Allow whole screen: your pointer and keyboard, pictures of everything", &whole)) a.fromPanel(.whole_screen, whole);
ui.text(c, "Off, agents work inside one app at a time, in the background.", .{ .size = 12, .color = k.secondary, .wrap = true });
if (a.listen_error) |e| {
ui.text(c, e, .{ .size = 12, .color = k.red, .wrap = true });
@@ -106,10 +135,12 @@ fn agents(c: *ui.Context, a: *App, k: theme.Colors) void {
}
}
if (comptime mac) {
- const r = ui.div(c).row().itemsCenter().gap(6);
- defer r.end();
- ui.text(c, "Stop everything from anywhere:", .{ .size = 12, .color = k.secondary });
- ui.kbd(c, @import("mac/hotkey.zig").label);
+ if (a.hotkey != null) {
+ const r = ui.div(c).row().itemsCenter().gap(6);
+ defer r.end();
+ ui.text(c, "Stop everything from anywhere:", .{ .size = 12, .color = k.secondary });
+ ui.kbd(c, @import("mac/hotkey.zig").label);
+ } else ui.text(c, "The stop key couldn't be set up: use Stop above, or `graff computer stop`.", .{ .size = 12, .color = k.secondary, .wrap = true });
}
}
diff --git a/docs/adr/0314-nanocua-is-grafts-computer-use.md b/docs/adr/0314-nanocua-is-grafts-computer-use.md
index 17dd521b..3402e6ef 100644
--- a/docs/adr/0314-nanocua-is-grafts-computer-use.md
+++ b/docs/adr/0314-nanocua-is-grafts-computer-use.md
@@ -41,6 +41,29 @@ codegraff holds the original. nanoui's `apps/nanocua` is generated from it by
MIT. nanoui does not import it, so nanoui itself stays MIT. Linking the other way
is fine: `apps/nanocua` depends on nanoui, pinned, like any MIT library.
+## Who changes the switches
+
+`nanocua ctl` is the one way in from outside the panel. `graff computer` and
+`/computer` run it, and Harness runs it directly, because an older graff would
+take `graff computer` for a prompt. Status, stop and turning things off happen at
+once, for anyone on the user's socket. Turning something on (input, whole screen,
+resume) only asks: a question appears in nanocua's window, and only a person's
+click on Allow there answers it.
+
+That split exists because Harness runs graff with `--yolo` and agents have shells:
+a command that turned input on would let an agent turn on its own input, so the
+switches would protect against nothing. Allow arms after 0.7s, so a click meant
+for what was under the pointer can't answer it. A question goes away after 60s,
+and a Deny refuses new ones for 15s. nanocua checks that a click came from
+hardware, not a program, and only for turning things on: Stop works however it's
+clicked. Switch Control and Voice Control post their own clicks, so they can't
+turn things on. nanoui's `--nanoui-*` flags are refused outside debug builds,
+since they drive the same window.
+
+A process running as the user can still attack this in ways nanocua can't see,
+such as editing what the app loads. The check closes the easy route, not every one.
+macOS's own permissions stay in System Settings; nothing here can flip them.
+
## Consequences
- The desktop app runs `graff acp`, so it offers nanocua too once the app is in
@@ -53,5 +76,6 @@ is fine: `apps/nanocua` depends on nanoui, pinned, like any MIT library.
computer use. An ad hoc app would lose its permissions on every update.
- Background input is delivered per process. Apps may ignore it, Chromium's web
content most likely; agents fall back to refs or ask for the whole screen.
-- Two agents share one app, one target and one set of refs, calls one at a time.
+- Agents share one app, calls one at a time, but each connection has its own refs
+ and target. Harness's graff chats and a terminal graff use the same app.
- Beta releases are not signed, so they carry no app.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 14728301..aacd821a 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -338,7 +338,7 @@ record only when you need the evidence or the edge cases.
| [0311](0311-tool-refresh-and-addressed-wakes.md) | Claude Agent refreshes a changed tool list with MCP `tools/list_changed` (served before anything else; fresh session if not re-read in 15 s) instead of removing and re-adding its server, which failed held calls. `attempt_completion` settles a task like a final answer: room mail waits for the next human prompt, addressed mail can still start a turn; `yield_turn` stays strict. A peer wake preloads `peer_message`. |
| [0312](0312-openai-12h-prompt-cache-is-opt-in.md) | OpenAI Responses GPT-5.6+ can put a 12h explicit breakpoint on the stable prefix (`GRAFF_OPENAI_CACHE_TTL=12h` or `openai_cache_ttl`); the default stays 30m (ADR 0009 bytes) until a live probe confirms OpenAI accepts the field and keeps the entry. Usage parses cache writes by TTL. |
| [0313](0313-graff-does-not-load-codex-computer-use.md) | Graff does not load Codex's `computer-use` or `unified-computer-use` plugins; every MCP elicitation is declined. Supersedes 0036 and 0119. |
-| [0314](0314-nanocua-is-grafts-computer-use.md) | Computer use on macOS is nanocua, a signed app shipped beside graff and offered as one MCP server; it works inside one app in the background, and the app's panel, not graff, says what it may do. nanoui's copy is generated from codegraff's and keeps codegraff's license. |
+| [0314](0314-nanocua-is-grafts-computer-use.md) | Computer use on macOS is nanocua, a signed app shipped beside graff and offered as one MCP server; it works inside one app in the background, and the app's panel, not graff, says what it may do: graff and Harness can read its state and turn things off, but turning anything on takes a person's click in its window. nanoui's copy is generated from codegraff's and keeps codegraff's license. |
| [0302](0302-sse-idle-watchdogs-follow-body-byte-progress.md) | SSE idle watchdogs observe partial body-byte progress, while separate unfinished-event time and size bounds preserve recovery. |
diff --git a/src/args.zig b/src/args.zig
index 2a718607..d8edfcc3 100644
--- a/src/args.zig
+++ b/src/args.zig
@@ -123,7 +123,7 @@ pub fn parse(init: std.process.Init) !Flags {
_ = it.next(); // argv[0]
while (it.next()) |arg| {
if (flags.positionals.items.len > 0 and
- (std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "learn") or std.mem.eql(u8, flags.positionals.items[0], "peer") or std.mem.eql(u8, flags.positionals.items[0], "keys") or std.mem.eql(u8, flags.positionals.items[0], "harness-tree")))
+ (std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "computer") or std.mem.eql(u8, flags.positionals.items[0], "learn") or std.mem.eql(u8, flags.positionals.items[0], "peer") or std.mem.eql(u8, flags.positionals.items[0], "keys") or std.mem.eql(u8, flags.positionals.items[0], "harness-tree")))
{
try flags.positionals.append(arena, try arena.dupe(u8, arg));
} else if (std.mem.startsWith(u8, arg, "-")) {
@@ -288,7 +288,7 @@ pub fn parse(init: std.process.Init) !Flags {
// One-shot print mode: `harness -p "prompt"` or a bare positional prompt
// (`harness "say hi"`). Subcommands (login/key) are not prompts.
flags.is_subcommand = flags.positionals.items.len > 0 and
- (std.mem.eql(u8, flags.positionals.items[0], "login") or std.mem.eql(u8, flags.positionals.items[0], "logout") or std.mem.eql(u8, flags.positionals.items[0], "key") or std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "learn") or
+ (std.mem.eql(u8, flags.positionals.items[0], "login") or std.mem.eql(u8, flags.positionals.items[0], "computer") or std.mem.eql(u8, flags.positionals.items[0], "logout") or std.mem.eql(u8, flags.positionals.items[0], "key") or std.mem.eql(u8, flags.positionals.items[0], "mcp") or std.mem.eql(u8, flags.positionals.items[0], "learn") or
std.mem.eql(u8, flags.positionals.items[0], "serve") or std.mem.eql(u8, flags.positionals.items[0], "update") or std.mem.eql(u8, flags.positionals.items[0], "title") or std.mem.eql(u8, flags.positionals.items[0], "repl") or (!flags.print_flag and std.mem.eql(u8, flags.positionals.items[0], "tui")) or
std.mem.eql(u8, flags.positionals.items[0], "worktree") or std.mem.eql(u8, flags.positionals.items[0], "harness-tree") or std.mem.eql(u8, flags.positionals.items[0], "sandboxes") or std.mem.eql(u8, flags.positionals.items[0], "cube") or std.mem.eql(u8, flags.positionals.items[0], "models") or
std.mem.eql(u8, flags.positionals.items[0], "remote-control") or std.mem.eql(u8, flags.positionals.items[0], "remote") or @import("acp.zig").isAcpSubcommand(flags.positionals.items[0])); // `acp` also arms ACP's stdout discipline (json_mode) — see acp.isAcpSubcommand
diff --git a/src/cli_text_source.zig b/src/cli_text_source.zig
index b0c97e29..8d32603c 100644
--- a/src/cli_text_source.zig
+++ b/src/cli_text_source.zig
@@ -334,6 +334,7 @@ pub const usage_text =
\\ graff mcp install configure local HTTP service and MCP clients
\\ graff mcp serve [--http] [--port N] expose run_task to MCP clients
\\ graff plugins [load <name>] list Claude/Cursor/Grok/Codex plugin trees (in place)
+ \\ graff computer [status|stop|off|allow|grant|open] nanocua's switches (macOS)
\\
++ learn_usage ++
\\ graff worktree list list -w tabs and experiment-pool trees (tagged)
diff --git a/src/command_catalog.zig b/src/command_catalog.zig
index 10c56269..1b2fe968 100644
--- a/src/command_catalog.zig
+++ b/src/command_catalog.zig
@@ -128,6 +128,7 @@ pub const commands = [_]Item{
.{ .name = "/theme", .usage = "/theme [name]", .desc = "pick a color theme; /theme off resets to your terminal default; persists" },
.{ .name = "/fleet", .usage = "/fleet [on|off]", .desc = "federated DGM contribution (propose/submit/elite_pull)" },
.{ .name = "/mcp", .usage = "/mcp [add … | reconnect <name>]", .desc = "list MCP servers/tools; /mcp add <name> <cmd> [args...] connects one live; /mcp reconnect <name> starts one again" },
+ .{ .name = "/computer", .usage = "/computer [status | stop | off … | allow … | grant … | open]", .desc = "nanocua's switches: what's on, stop it, ask to turn things on (macOS)" },
.{ .name = "/import-claude", .desc = "copy Claude/Cursor setup and import this project's Claude conversations for /resume" },
.{ .name = "/help", .desc = "list every command" },
};
diff --git a/src/commands_misc.zig b/src/commands_misc.zig
index 846099b4..219d839e 100644
--- a/src/commands_misc.zig
+++ b/src/commands_misc.zig
@@ -211,6 +211,10 @@ pub fn tryHandle(root: *Agent, keys: *Keys, arena: Allocator, line: []const u8,
try out.flush();
return true;
}
+ if (std.mem.eql(u8, line, "/computer") or std.mem.startsWith(u8, line, "/computer ")) {
+ try @import("computer_cli.zig").slash(root.io, root.gpa, arena, root.home, line, out);
+ return true;
+ }
if (std.mem.startsWith(u8, line, "/mcp")) {
const arg = std.mem.trim(u8, line["/mcp".len..], " \t");
const reg = root.registry.?; // always present now
diff --git a/src/computer_cli.zig b/src/computer_cli.zig
new file mode 100644
index 00000000..72fe9c3c
--- /dev/null
+++ b/src/computer_cli.zig
@@ -0,0 +1,204 @@
+//! `graff computer` and `/computer` (ADR 0314): nanocua's switches from graff,
+//! by running the app's own `nanocua ctl`. Status, stop and turning things off
+//! happen at once. `allow` only asks: the question appears in nanocua's window
+//! and only a person's click there answers it, because agents have shells and
+//! could run this command as easily as anyone. macOS's permissions stay the
+//! user's to flip in System Settings: `grant` opens the prompt.
+
+const std = @import("std");
+const builtin = @import("builtin");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+const nanocua_default = @import("nanocua_default.zig");
+const process_runner = @import("process_runner.zig");
+
+pub const usage =
+ \\usage: graff computer [status] [--json]
+ \\ status what's on, and what macOS has granted
+ \\ stop refuse every agent's calls, at once
+ \\ off <input|whole_screen> turn a switch off, at once
+ \\ allow <input|whole_screen|resume> ask; you answer in nanocua's window
+ \\ grant <accessibility|screen> open macOS's prompt for a permission
+ \\ open bring nanocua's window forward
+ \\
+;
+
+/// `allow` waits for the click: a question lasts a minute, and starting the
+/// app may come first.
+const deadline_ms: u64 = 90_000;
+const output_cap: usize = 64 * 1024;
+
+const not_installed_text = "nanocua isn't installed. It ships with graff's signed macOS releases: run `graff update`.\n";
+const not_macos_text = "Computer use is macOS-only for now.\n";
+
+pub const Parsed = struct {
+ /// What goes after `nanocua ctl`.
+ words: []const []const u8,
+ json: bool = false,
+ from: []const u8 = "graff",
+};
+
+/// Null asks for the usage. Words are checked here, so nothing but the commands
+/// above ever reaches the app, and a mistake answers here without starting it.
+pub fn parse(arena: Allocator, args: []const []const u8) !?Parsed {
+ var out: Parsed = .{ .words = &.{} };
+ var words: std.ArrayList([]const u8) = .empty;
+ for (args) |arg| {
+ if (std.mem.eql(u8, arg, "--json")) {
+ out.json = true;
+ } else if (std.mem.startsWith(u8, arg, "--from=")) {
+ out.from = arg["--from=".len..];
+ } else try words.append(arena, arg);
+ }
+ if (words.items.len == 0) try words.append(arena, "status");
+ if (!known(words.items)) return null;
+ out.words = words.items;
+ return out;
+}
+
+fn oneOf(word: []const u8, comptime options: []const []const u8) bool {
+ for (options) |o| if (std.mem.eql(u8, word, o)) return true;
+ return false;
+}
+
+fn known(words: []const []const u8) bool {
+ if (words.len == 1) return oneOf(words[0], &.{ "status", "stop", "open" });
+ if (words.len != 2) return false;
+ if (std.mem.eql(u8, words[0], "off")) return oneOf(words[1], &.{ "input", "whole_screen" });
+ if (std.mem.eql(u8, words[0], "allow")) return oneOf(words[1], &.{ "input", "whole_screen", "resume" });
+ if (std.mem.eql(u8, words[0], "grant")) return oneOf(words[1], &.{ "accessibility", "screen" });
+ return false;
+}
+
+pub fn argv(arena: Allocator, program: []const u8, p: Parsed) ![]const []const u8 {
+ var out: std.ArrayList([]const u8) = .empty;
+ try out.appendSlice(arena, &.{ program, "ctl" });
+ try out.appendSlice(arena, p.words);
+ if (p.json) try out.append(arena, "--json");
+ try out.append(arena, try std.fmt.allocPrint(arena, "--from={s}", .{p.from}));
+ return out.items;
+}
+
+/// What's said when there's no app to ask.
+fn absent(json: bool, supported: bool) []const u8 {
+ if (json) return if (supported)
+ "{\"installed\":false,\"running\":false}\n"
+ else
+ "{\"installed\":false,\"running\":false,\"outcome\":\"unsupported\"}\n";
+ return if (supported) not_installed_text else not_macos_text;
+}
+
+/// True if the command ran; its output or the reason it didn't goes to `out`.
+pub fn run(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, args: []const []const u8, out: *Io.Writer) !bool {
+ const p = try parse(arena, args) orelse {
+ try out.writeAll(usage);
+ return false;
+ };
+ const supported = builtin.os.tag == .macos;
+ const program = (if (supported) nanocua_default.find(io, arena, home) else null) orelse {
+ try out.writeAll(absent(p.json, supported));
+ return false;
+ };
+ if (!p.json and std.mem.eql(u8, p.words[0], "allow")) {
+ try out.writeAll("Asking: answer in nanocua's window. The question lasts a minute.\n");
+ try out.flush();
+ }
+ const r = process_runner.runCapped(gpa, io, try argv(arena, program, p), output_cap, output_cap, deadline_ms) catch |err| {
+ try out.print("Couldn't run nanocua: {t}\n", .{err});
+ return false;
+ };
+ defer gpa.free(r.stdout);
+ defer gpa.free(r.stderr);
+ try out.writeAll(r.stdout);
+ if (r.timed_out) try out.writeAll("nanocua didn't answer in time.\n");
+ const exited_ok = switch (r.term) {
+ .exited => |code| code == 0,
+ else => false,
+ };
+ if (!exited_ok and r.stderr.len > 0) try out.writeAll(r.stderr);
+ return exited_ok and !r.timed_out;
+}
+
+/// `graff computer …`.
+pub fn command(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, args: []const []const u8) !void {
+ var buf: [4096]u8 = undefined;
+ var w = Io.File.stdout().writer(io, &buf);
+ const ok = try run(io, gpa, arena, home, args, &w.interface);
+ try w.interface.flush();
+ if (!ok) std.process.exit(1);
+}
+
+/// `/computer …`, on the REPL and the TUI.
+pub fn slash(io: Io, gpa: Allocator, arena: Allocator, home: []const u8, line: []const u8, out: *Io.Writer) !void {
+ var words: std.ArrayList([]const u8) = .empty;
+ var it = std.mem.tokenizeAny(u8, line["/computer".len..], " \t");
+ while (it.next()) |word| try words.append(arena, word);
+ _ = try run(io, gpa, arena, home, words.items, out);
+ try out.flush();
+}
+
+const testing = std.testing;
+
+test "only the commands nanocua knows get through" {
+ var state = std.heap.ArenaAllocator.init(testing.allocator);
+ defer state.deinit();
+ const a = state.allocator();
+ try testing.expectEqualStrings("status", (try parse(a, &.{})).?.words[0]);
+ const allow = (try parse(a, &.{ "allow", "whole_screen", "--json", "--from=Harness" })).?;
+ try testing.expect(allow.json);
+ try testing.expectEqualStrings("Harness", allow.from);
+ try testing.expectEqualStrings("whole_screen", allow.words[1]);
+ for ([_][]const []const u8{
+ &.{"nonsense"},
+ &.{"allow"},
+ &.{ "allow", "everything" },
+ &.{ "off", "resume" },
+ &.{ "grant", "camera" },
+ &.{ "stop", "now" },
+ &.{ "status", "extra", "words" },
+ }) |bad| try testing.expect((try parse(a, bad)) == null);
+}
+
+test "what runs: the app's ctl, saying who asks" {
+ var state = std.heap.ArenaAllocator.init(testing.allocator);
+ defer state.deinit();
+ const a = state.allocator();
+ const got = try argv(a, "/x/nanocua", (try parse(a, &.{ "grant", "screen", "--json" })).?);
+ const want = [_][]const u8{ "/x/nanocua", "ctl", "grant", "screen", "--json", "--from=graff" };
+ try testing.expectEqual(want.len, got.len);
+ for (want, got) |w, g| try testing.expectEqualStrings(w, g);
+}
+
+test "no app: a person is told how, a program reads that it's missing" {
+ var tmp = testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var state = std.heap.ArenaAllocator.init(testing.allocator);
+ defer state.deinit();
+ const a = state.allocator();
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const n = try tmp.dir.realPath(testing.io, &buf);
+ const home = try a.dupe(u8, buf[0..n]);
+ var aw: Io.Writer.Allocating = .init(a);
+ try testing.expect(!try run(testing.io, testing.allocator, a, home, &.{}, &aw.writer));
+ try testing.expect(std.mem.indexOf(u8, aw.written(), "isn't") != null or std.mem.indexOf(u8, aw.written(), "macOS-only") != null);
+ aw.clearRetainingCapacity();
+ try testing.expect(!try run(testing.io, testing.allocator, a, home, &.{"--json"}, &aw.writer));
+ const parsed = try std.json.parseFromSliceLeaky(std.json.Value, a, aw.written(), .{});
+ try testing.expect(!parsed.object.get("installed").?.bool);
+ aw.clearRetainingCapacity();
+ try testing.expect(!try run(testing.io, testing.allocator, a, home, &.{"nonsense"}, &aw.writer));
+ try testing.expectEqualStrings(usage, aw.written());
+}
+
+test "/computer splits its words" {
+ var tmp = testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var state = std.heap.ArenaAllocator.init(testing.allocator);
+ defer state.deinit();
+ const a = state.allocator();
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const n = try tmp.dir.realPath(testing.io, &buf);
+ var aw: Io.Writer.Allocating = .init(a);
+ try slash(testing.io, testing.allocator, a, try a.dupe(u8, buf[0..n]), "/computer allow \t nonsense", &aw.writer);
+ try testing.expectEqualStrings(usage, aw.written());
+}
diff --git a/src/help.zig b/src/help.zig
index 5d9a4b67..14c436cb 100644
--- a/src/help.zig
+++ b/src/help.zig
@@ -32,7 +32,7 @@ pub const sections = [_]Section{
.{ .title = "the model", .names = &.{ "/model", "/models", "/effort", "/reasoning", "/fast", "/thinking", "/keepcontext", "/fallback", "/routes" } },
.{ .title = "working autonomously", .names = &.{ "/goal", "/goals", "/schedule", "/review", "/issue", "/plan", "/todo", "/jobs", "/workers", "/ultracode", "/strict", "/yolo", "/never" } },
.{ .title = "talking to other graffs", .names = &.{ "/tell", "/daddy", "/peek", "/adapter" }, .blurb = peers_blurb },
- .{ .title = "your setup", .names = &.{ "/login", "/key", "/cost", "/usage", "/version", "/update", "/privacy", "/mcp", "/import-claude", "/skills", "/plugins", "/agents", "/hooks", "/tools", "/fleet" } },
+ .{ .title = "your setup", .names = &.{ "/login", "/key", "/cost", "/usage", "/version", "/update", "/privacy", "/mcp", "/computer", "/import-claude", "/skills", "/plugins", "/agents", "/hooks", "/tools", "/fleet" } },
.{ .title = "context & history", .names = &.{ "/compact", "/compact-at", "/btw", "/doctor", "/debug", "/cache", "/trace", "/trajectory" } },
.{ .title = "shell & images", .names = &.{ "/bash", "/image", "/images", "/paste" } },
.{ .title = "look & feel", .names = &.{ "/theme", "/animation", "/title" } },
diff --git a/src/nanocua_default.zig b/src/nanocua_default.zig
index 077fe14a..b408b35a 100644
--- a/src/nanocua_default.zig
+++ b/src/nanocua_default.zig
@@ -12,6 +12,25 @@ const Allocator = std.mem.Allocator;
pub const server_name = "nanocua";
const executable = "nanocua.app/Contents/MacOS/nanocua";
+/// Whether `home` is `real` or a folder inside it, as an account's HOME under
+/// Harness is. A $HOME anywhere else is a sandbox, and stays one.
+pub fn within(home: []const u8, real: []const u8) bool {
+ const h = std.mem.trimEnd(u8, home, "/");
+ const r = std.mem.trimEnd(u8, real, "/");
+ if (r.len == 0 or !std.mem.startsWith(u8, h, r)) return false;
+ return h.len == r.len or h[r.len] == '/';
+}
+
+/// The home the one app on this Mac is under: the user's real one when $HOME is
+/// inside it. A sandbox's HOME is used as it is, so a test running graff can't
+/// find or replace the app installed on the machine running it.
+pub fn userHome(home: []const u8) []const u8 {
+ if (builtin.is_test or builtin.os.tag != .macos) return home;
+ const user = std.c.getpwuid(std.c.getuid()) orelse return home;
+ const real = std.mem.span(user.dir orelse return home);
+ return if (within(home, real)) real else home;
+}
+
/// Where install.sh and `graff update` put the app.
pub fn installRoot(arena: Allocator, home: []const u8) ?[]const u8 {
if (home.len == 0) return null;
@@ -24,11 +43,14 @@ fn inRoot(io: Io, arena: Allocator, root: []const u8) ?[]const u8 {
return path;
}
-/// The shipped app's program. Tests look only under their own home, so an app
-/// installed on the machine running them doesn't change what they see.
+/// The shipped app's program. Unit tests and a sandbox's HOME look only under
+/// that home, so an app installed on the machine doesn't change what they see.
pub fn find(io: Io, arena: Allocator, home: []const u8) ?[]const u8 {
if (installRoot(arena, home)) |root| if (inRoot(io, arena, root)) |p| return p;
if (builtin.is_test) return null;
+ const user = userHome(home);
+ if (!within(home, user)) return null;
+ if (installRoot(arena, user)) |root| if (inRoot(io, arena, root)) |p| return p;
if (inRoot(io, arena, "/Applications")) |p| return p;
// An unpacked release tarball, which has it beside graff.
const exe = std.process.executablePathAlloc(io, arena) catch return null;
@@ -75,6 +97,16 @@ fn tmpHome(io: Io, arena: Allocator, tmp: *testing.TmpDir, with_app: bool) ![]co
return home;
}
+test "only a home inside the user's own leads back to it" {
+ try testing.expect(within("/Users/ada", "/Users/ada"));
+ try testing.expect(within("/Users/ada/", "/Users/ada"));
+ try testing.expect(within("/Users/ada/.harness/accounts/two", "/Users/ada"));
+ try testing.expect(!within("/Users/adam", "/Users/ada"));
+ try testing.expect(!within("/tmp/sandbox", "/Users/ada"));
+ try testing.expect(!within("/var/folders/x/T/home", "/Users/ada"));
+ try testing.expect(!within("/Users/ada", ""));
+}
+
test "the shipped app becomes the nanocua server" {
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
diff --git a/src/repl_model_commands.zig b/src/repl_model_commands.zig
index 8cfd0adb..61fd948d 100644
--- a/src/repl_model_commands.zig
+++ b/src/repl_model_commands.zig
@@ -111,6 +111,8 @@ pub fn runCommand(self: *Model, line: []const u8) void {
self.push(.info, "running shell commands needs the agent loop — use the main `graff` session, or your shell") catch {};
} else if (std.mem.eql(u8, cmd, "/mcp")) {
self.push(.info, "MCP servers attach to the agent loop, not the chat repl") catch {};
+ } else if (std.mem.eql(u8, cmd, "/computer")) {
+ self.push(.info, "computer use attaches to the agent loop, not the chat repl; `graff computer` works from a shell") catch {};
} else if (std.mem.eql(u8, cmd, "/skills")) {
self.push(.info, "skills (codedb, zigrep, …) attach to the agent loop, not the chat repl") catch {};
} else if (std.mem.eql(u8, cmd, "/agents")) {
diff --git a/src/startup.zig b/src/startup.zig
index 8f7e5c5a..1756aeab 100644
--- a/src/startup.zig
+++ b/src/startup.zig
@@ -318,6 +318,12 @@ pub fn runSubcommand(io: Io, gpa: Allocator, arena: Allocator, init: std.process
return true;
}
+ if (flags.positionals.items.len > 0 and std.mem.eql(u8, flags.positionals.items[0], "computer")) {
+ const home = keys_cli.homeEnv(init.environ_map) orelse "";
+ try @import("computer_cli.zig").command(io, gpa, arena, home, flags.positionals.items[1..]);
+ return true;
+ }
+
if (flags.positionals.items.len > 0 and std.mem.eql(u8, flags.positionals.items[0], "plugins")) {
const home = keys_cli.homeEnv(init.environ_map) orelse "";
plugins.applyEnv(init.environ_map);
diff --git a/src/test_hooks.zig b/src/test_hooks.zig
index 15ce902c..0743914b 100644
--- a/src/test_hooks.zig
+++ b/src/test_hooks.zig
@@ -404,6 +404,7 @@ test {
_ = @import("update_windows.zig");
_ = @import("mcp_elicitation.zig");
_ = @import("nanocua_default.zig");
+ _ = @import("computer_cli.zig");
// POSIX file modes; only macOS installs it, Linux runs its tests too.
if (@import("builtin").os.tag != .windows) _ = @import("nanocua_install.zig");
_ = @import("edit_worktree_tests.zig"); // #747: edit lands on the selected worktree
diff --git a/src/update_service.zig b/src/update_service.zig
index 717ebfab..0182ec92 100644
--- a/src/update_service.zig
+++ b/src/update_service.zig
@@ -308,7 +308,7 @@ pub fn install(opts: Opts, consent: Consent) Report {
// The app that ships with graff (ADR 0314). It has its own uses, so failing
// to place it doesn't stop graff updating.
- if (builtin.os.tag == .macos) _ = @import("nanocua_install.zig").fromArchive(opts.io, opts.gpa, opts.home, tarball) catch {};
+ if (builtin.os.tag == .macos) _ = @import("nanocua_install.zig").fromArchive(opts.io, opts.gpa, @import("nanocua_default.zig").userHome(opts.home), tarball) catch {};
if (existingMatches(opts.io, opts.gpa, opts.target_path, bin)) {
writeInstalledVersion(opts.io, opts.target_path, version_status.stripV(tag_copy));