Skip to content

A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.

TypeScript

feat(computer): register the macOS control helper and show its state

Alan Buscaglia5b59c0efe4f7
diff --git a/docs/design.md b/docs/design.md
index 6f8fa3f..db0ba40 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -158,6 +158,20 @@ Every daemon message carries `seq` (monotonic per connection) so the UI can dete
The UI shows one endless chat: no conversations list, no new conversation, and the header shows the assistant (the rose glyph and "Gentle Dot"). The code and protocol for several conversations stay behind `features.conversations` (`GENTLE_DOT_CONVERSATIONS=1`); with it off, the UI also ignores the menu bar's `dot://new-conversation`. The daemon rotates the session underneath when it grows (§7), and `history` / `get_earlier` read through the chain of session files, so earlier messages, including those before any compaction, stay reachable with "Show earlier".
+### Computer control (macOS, S24)
+
+The assistant can see the screen and operate apps through a built-in MCP server that the desktop app serves. macOS only (L59); the web page and a VPS never offer it.
+
+- S24.1 Helper: native Rust inside the Tauri app (`apps/desktop/src-tauri`), never in the daemon or the engine. It serves MCP (JSON-RPC 2.0 over streamable HTTP) on `http://127.0.0.1:<random port>/mcp` and requires `Authorization: Bearer <key>`, a fresh random key per app launch.
+- S24.2 Tools: `screenshot`, `click`, `move`, `drag`, `scroll`, `type`, `key`, `open_app`, `list_apps`, `wait`, each with an optional `intent`. Screenshots of the display under the pointer are downscaled to a long edge of 1280 px; coordinates are pixels of the latest screenshot; actions return a fresh one; Gentle Dot's own windows are left out.
+- S24.3 Session grant: the first call without a session shows a native dialog ("Allow Gentle Dot to see and control this Mac for 30 minutes?"). A session ends on timeout, Stop, the tray item, or the panic shortcut ⌥⇧Esc, and queued actions are dropped.
+- S24.4 Guards: a fixed blocklist (Gentle Dot itself, System Settings, SecurityAgent, loginwindow, Keychain Access, password managers), 10 actions per second, and a native per-action confirmation for risky actions (send, pay, buy, delete, submit, confirm, transfer, and their Spanish equivalents, by Accessibility role and title, by `return` on a default button, or by the declared `intent`).
+- S24.5 Permissions: the app reports and requests Accessibility and Screen Recording (`computer_permissions`, `computer_request_permission {kind}`). The Connectors screen shows a built-in "Computer" entry with both statuses, Grant buttons, Check again, the note that debug builds lose the grants on every rebuild (macOS ties them to the code signature), and a note when the current model does not accept images (`ConnectorInfo.noImages`, from the model's `input` in the engine's `get_state` and `set_model` answers).
+- S24.6 Visible state: the app emits `computer://state {active, endsAt?, reason?}` (`computer_status` gives the current one). While active, the rose shows the "in control" state (`dot-in_control`, mood `control`: accent rim, steady traveling light) and the panel shows "Controlling your Mac · mm:ss left" with Stop (`computer_stop`).
+- S24.7 Wiring: on every connection the desktop panel calls `computer_endpoint` and, when it returns `{url, token}`, sends `computer_register {url, token}` (the protocol accepts only `http://127.0.0.1:<port>/mcp` and a header-safe key). The daemon keeps the endpoint in memory, never in `connectors.json`, and renders `computer: {url, headers: {Authorization: "Bearer <key>"}, exposure: "direct"}` into `mcp.json`, then restarts the engine when idle. The guard's policy lists it under `builtin`, so its tools run without in-engine approval cards. It is removed on `computer_unregister` or when the window that registered it disconnects. The key is never logged or sent to a window: `connectors` lists only a built-in "Computer" entry (`builtin: true`) that the connector messages cannot change or remove, no saved or imported connector can take the name `computer`, and the import scan never offers the helper.
+
+Why enforcement lives in the app: everything inside the engine, and the daemon's WebSocket, is within the agent's reach (it has a shell, and with the daemon's access key it could answer its own approval cards, L49). The session grant, panic stop, blocklist, rate limit, and risky-action confirmations therefore run in the helper, behind native dialogs the agent cannot answer, and the in-engine guard adds nothing for these tools. macOS also attributes Accessibility and Screen Recording to the app that holds them, Gentle Dot.app. The key in `mcp.json` only reaches the helper, which still asks the user before anything happens.
+
## 5. White-label layer
- **Identity**: `--append-system-prompt` with `identity.md`: the assistant is "Gentle Dot", a personal assistant; it must not mention Gentle Shell, Pi, el Gentleman, ODD, or Engram by name, and describes them as "my workflow" and "my memory" when needed. It keeps every behavior.
diff --git a/packages/daemon/src/bridge.ts b/packages/daemon/src/bridge.ts
index f7c4cda..a91cecb 100644
--- a/packages/daemon/src/bridge.ts
+++ b/packages/daemon/src/bridge.ts
@@ -126,6 +126,7 @@ export class DotBridge {
if (options.connectors) {
options.connectors.onChanged = (restart) => this.afterConnectorsChange(restart);
options.connectors.onBlocked = () => this.connectorChangeBlocked();
+ options.connectors.imagesSupported = () => supervisor.modelImages;
}
this.state = this.deriveState();
}
@@ -152,6 +153,7 @@ export class DotBridge {
this.clients.delete(client);
this.options.auth?.cancelOwnedBy(client);
this.options.connectors?.cancelOwnedBy(client);
+ this.options.connectors?.unregisterComputer(client);
};
}
@@ -350,6 +352,12 @@ export class DotBridge {
case "connectors_scan":
this.deliver(client, { type: "connector_imports", found: this.requireConnectors().scan() });
return;
+ case "computer_register":
+ this.requireConnectors().registerComputer(client, { url: message.url, token: message.token });
+ return;
+ case "computer_unregister":
+ this.requireConnectors().unregisterComputer(client);
+ return;
case "connector_import":
this.deliver(client, {
type: "connector_imported",
diff --git a/packages/daemon/src/connectors.ts b/packages/daemon/src/connectors.ts
index 23c7ce8..8dc8545 100644
--- a/packages/daemon/src/connectors.ts
+++ b/packages/daemon/src/connectors.ts
@@ -300,6 +300,23 @@ export const CATALOG: readonly CatalogEntry[] = [
const catalogEntry = (id: string) => CATALOG.find((entry) => entry.id === id);
+/** The desktop app's computer-control helper (S24): its loopback address and its key for this launch. */
+export interface ComputerEndpoint {
+ url: string;
+ token: string;
+}
+
+/** The built-in server's name in `mcp.json`; no connector can take it. */
+export const COMPUTER_ID = "computer";
+
+const COMPUTER_INFO = {
+ id: COMPUTER_ID,
+ name: "Computer",
+ reads: "See your screen during a session you allow in the app.",
+ sends:
+ "Click, type, and open apps on this Mac during that session. The app asks you again before risky actions.",
+} as const;
+
/** A server the user added from the assistant's draft or an import. */
export interface CustomConnector {
name: string;
@@ -432,7 +449,7 @@ function ordered(state: ConnectorsState): [string, SavedConnector][] {
* user's own has no curated list, so all of its tools are hidden. A connector that still waits for
* a value the user types is left out.
*/
-export function renderMcpJson(state: ConnectorsState): string {
+export function renderMcpJson(state: ConnectorsState, computer?: ComputerEndpoint): string {
const servers: Record<string, unknown> = {};
for (const [id, saved] of ordered(state)) {
const spec = specOf(id, saved);
@@ -452,6 +469,13 @@ export function renderMcpJson(state: ConnectorsState): string {
...(saved.enabled ? {} : { enabled: false }),
};
}
+ // The helper checks its own key and asks the user itself, so all of its tools are direct.
+ if (computer)
+ servers[COMPUTER_ID] = {
+ url: computer.url,
+ headers: { Authorization: `Bearer ${literal(computer.token)}` },
+ exposure: "direct",
+ };
return `${JSON.stringify({ mcpServers: servers }, null, 2)}\n`;
}
@@ -487,6 +511,8 @@ export class ConnectorStore {
/** Called after a change made outside the Connectors screen was put back. */
onReverted: () => void = () => {};
private saved: ConnectorsState;
+ /** The desktop app's helper while a window has it registered; only in memory and `mcp.json`. */
+ private computer: ComputerEndpoint | undefined;
private watchers: FSWatcher[] = [];
private readonly options: ConnectorStoreOptions;
@@ -513,7 +539,22 @@ export class ConnectorStore {
this.saved = state;
mkdirSync(this.options.agentHome, { recursive: true, mode: 0o700 });
writePrivate(this.file, this.recordText());
- writePrivate(this.mcpFile, renderMcpJson(state));
+ writePrivate(this.mcpFile, renderMcpJson(state, this.computer));
+ }
+
+ /** Adds, replaces, or removes the computer helper in `mcp.json`; true when that changed it. */
+ setComputer(endpoint: ComputerEndpoint | undefined): boolean {
+ const same = endpoint?.url === this.computer?.url && endpoint?.token === this.computer?.token;
+ if (same) return false;
+ this.computer = endpoint ? { ...endpoint } : undefined;
+ mkdirSync(this.options.agentHome, { recursive: true, mode: 0o700 });
+ writePrivate(this.mcpFile, renderMcpJson(this.saved, this.computer));
+ return true;
+ }
+
+ /** The helper's address while it is registered (never its key). */
+ computerUrl(): string | undefined {
+ return this.computer?.url;
}
/**
@@ -521,11 +562,11 @@ export class ConnectorStore {
* `report: false` only writes them (the daemon's start, where the files are what it just read).
*/
enforce(report = true): boolean {
- const untouched = Object.keys(this.saved.connectors).length === 0;
+ const untouched = Object.keys(this.saved.connectors).length === 0 && !this.computer;
const changed: string[] = [];
for (const [file, text] of [
[this.file, this.recordText()],
- [this.mcpFile, renderMcpJson(this.saved)],
+ [this.mcpFile, renderMcpJson(this.saved, this.computer)],
] as const) {
const current = existsSync(file) ? readFileSync(file, "utf8") : undefined;
if (current === text || (current === undefined && untouched)) continue;
@@ -599,6 +640,7 @@ export class ConnectorStore {
join(this.options.dataDir, "token"),
...(this.options.guardPath ? [this.options.guardPath] : []),
],
+ ...(this.computer ? { builtin: [COMPUTER_ID] } : {}),
};
}
@@ -672,6 +714,7 @@ export class ConnectorStore {
};
const connectors: ConnectorsState["connectors"] = {};
for (const [id, value] of Object.entries(saved.connectors ?? {})) {
+ if (id === COMPUTER_ID) continue;
const custom = catalogEntry(id) ? undefined : parseCustom(value?.custom);
if (!catalogEntry(id) && !custom) continue;
const entry: SavedConnector = {
@@ -825,7 +868,11 @@ export class ConnectorManager {
onChanged: (restart: boolean) => void = () => {};
/** A change to the connector files made outside the Connectors screen was put back. */
onBlocked: () => void = () => {};
+ /** Whether the current model accepts images; undefined when that is not known. */
+ imagesSupported: () => boolean | undefined = () => undefined;
private flow: Flow | undefined;
+ /** The window that registered the computer helper; it goes away with that window. */
+ private computerOwner: object | undefined;
private setupFlow: Setup | undefined;
private readonly failed = new Set<string>();
private readonly waiting = new Map<string, ConnectorDraft>();
@@ -854,7 +901,7 @@ export class ConnectorManager {
.map(([id]) => id)
.filter((id) => !catalogEntry(id)),
];
- return ids.flatMap((id) => {
+ const infos = ids.flatMap((id) => {
const saved = connectors[id];
const spec = specOf(id, saved);
if (!spec) return [];
@@ -873,6 +920,38 @@ export class ConnectorManager {
if (spec.custom) info.custom = { origin: spec.custom.origin, summary: summarize(spec.custom.server) };
return [info];
});
+ if (this.options.store.computerUrl() === undefined) return infos;
+ const computer: ConnectorInfo = {
+ ...COMPUTER_INFO,
+ added: true,
+ enabled: true,
+ mode: "read_write",
+ status: "connected",
+ noSignIn: true,
+ builtin: true,
+ };
+ if (this.imagesSupported() === false) computer.noImages = true;
+ return [...infos, computer];
+ }
+
+ /**
+ * The desktop app's computer helper, registered by one of its windows (S24.7). The engine reads it
+ * at its next start; its key is written only to `mcp.json`, never logged or sent to a window.
+ */
+ registerComputer(owner: object, endpoint: ComputerEndpoint): void {
+ this.computerOwner = owner;
+ if (!this.options.store.setComputer(endpoint)) return;
+ this.log("the desktop app registered computer control");
+ this.onChanged(true);
+ }
+
+ /** Removes the helper; only the window that registered it can (it also goes when that window does). */
+ unregisterComputer(owner: object): void {
+ if (this.computerOwner !== owner) return;
+ this.computerOwner = undefined;
+ if (!this.options.store.setComputer(undefined)) return;
+ this.log("computer control was unregistered");
+ this.onChanged(true);
}
/** Adds or turns on a connector: asks for the values it still needs, then signs in when it has no sign-in yet. */
@@ -1052,8 +1131,12 @@ export class ConnectorManager {
const home = this.options.importHome ?? homedir();
const groups = new Map<string, { candidate: ImportCandidate; server: ScannedServer }>();
const found = new Map<string, ScannedServer>();
+ const computerUrl = this.options.store.computerUrl();
+ const computer = computerUrl === undefined ? undefined : fingerprint({ url: computerUrl });
for (const server of scanClientConfigs(home)) {
const key = server.server ? fingerprint(server.server) : `${server.source}:${server.name}`;
+ // The helper is built in; a copy of it elsewhere is never offered.
+ if (key === computer) continue;
const group = groups.get(key);
if (group) {
if (!group.candidate.sources.includes(server.source)) group.candidate.sources.push(server.source);
@@ -1140,7 +1223,7 @@ export class ConnectorManager {
.slice(0, 32)
.replace(/-+$/, "") || "server";
const taken = (id: string) =>
- catalogEntry(id) !== undefined || id in this.options.store.state().connectors;
+ id === COMPUTER_ID || catalogEntry(id) !== undefined || id in this.options.store.state().connectors;
if (!taken(base)) return base;
for (let n = 2; ; n++) if (!taken(`${base}-${n}`)) return `${base}-${n}`;
}
diff --git a/packages/daemon/src/extensions/approval-guard.ts b/packages/daemon/src/extensions/approval-guard.ts
index c730bdd..e2c651b 100644
--- a/packages/daemon/src/extensions/approval-guard.ts
+++ b/packages/daemon/src/extensions/approval-guard.ts
@@ -5,7 +5,8 @@
* the server marks them `readOnlyHint: true` AND they are on the connector's curated list.
* Anything else asks the user through a confirm card with a preview, and is blocked when the
* user declines, when nobody can answer, or when the connector is set to read only. Calls made
- * by other tools (nested calls) go through the same hook.
+ * by other tools (nested calls) go through the same hook. The built-in computer server (the desktop
+ * app's helper, S24) runs without cards: the app grants sessions and confirms risky actions itself.
*
* The policy (approved connectors, modes, curated lists, protected files) comes from the
* daemon in this process's environment, which the assistant cannot change. Without a valid
@@ -114,8 +115,16 @@ export interface ConnectorPolicy {
connectors?: Record<string, { name: string; mode: ConnectorMode; readOnlyTools: string[] }>;
/** Files only the daemon may read or change: credentials and connector control files. */
protectedPaths: string[];
+ /**
+ * Built-in servers whose tools run without approval cards: only `computer`, the desktop app's
+ * helper, which asks the user itself (session grant, risky-action confirmations) outside the engine.
+ */
+ builtin?: string[];
}
+/** The only server that can be built in. */
+export const BUILTIN_SERVERS = ["computer"] as const;
+
export type GuardDecision =
| { action: "pass" }
| { action: "ask"; title: string; message: string }
@@ -190,7 +199,12 @@ export function parsePolicy(raw: string | undefined): ConnectorPolicy | undefine
entry.readOnlyTools.every((tool) => typeof tool === "string");
if (!valid) return undefined;
}
- return { connectors, protectedPaths: policy.protectedPaths };
+ const { builtin } = policy;
+ if (builtin === undefined) return { connectors, protectedPaths: policy.protectedPaths };
+ const known: readonly string[] = BUILTIN_SERVERS;
+ if (!Array.isArray(builtin) || !builtin.every((id) => typeof id === "string" && known.includes(id)))
+ return undefined;
+ return { connectors, protectedPaths: policy.protectedPaths, builtin };
}
export function decide(call: GuardCall, policy: ConnectorPolicy): GuardDecision {
@@ -214,6 +228,7 @@ export function decide(call: GuardCall, policy: ConnectorPolicy): GuardDecision
: { action: "pass" };
}
if (!toolName.startsWith("mcp__")) return { action: "pass" };
+ if (policy.builtin?.some((id) => toolName.startsWith(toolId(id, "")))) return { action: "pass" };
const approved = policy.connectors;
const server = approved
diff --git a/packages/daemon/src/supervisor.ts b/packages/daemon/src/supervisor.ts
index d708446..5ef53b0 100644
--- a/packages/daemon/src/supervisor.ts
+++ b/packages/daemon/src/supervisor.ts
@@ -67,6 +67,8 @@ export class AgentSupervisor {
sessionFile: string | undefined;
sessionId: string | undefined;
model: string | undefined;
+ /** Whether the current model accepts images (its `input` lists `image`); undefined when not reported. */
+ modelImages: boolean | undefined;
private child: ChildProcess | undefined;
private readonly pending = new Map<string, Pending>();
@@ -136,6 +138,9 @@ export class AgentSupervisor {
this.busy = true;
}
if (command.type === "new_session" || command.type === "switch_session") await this.refreshSession();
+ // A switch answers with the new model.
+ if (command.type === "set_model") this.applyModel(response.data);
+ if (command.type === "cycle_model") this.applyModel((response.data as { model?: unknown } | null)?.model);
return response;
}
@@ -307,9 +312,9 @@ export class AgentSupervisor {
const state = (data ?? {}) as {
sessionFile?: string;
sessionId?: string;
- model?: { name?: string; id?: string };
+ model?: unknown;
};
- if (state.model) this.model = state.model.name ?? state.model.id;
+ if (state.model) this.applyModel(state.model);
if (state.sessionId) this.sessionId = state.sessionId;
if (state.sessionFile && state.sessionFile !== this.sessionFile) {
this.sessionFile = state.sessionFile;
@@ -317,6 +322,13 @@ export class AgentSupervisor {
if (this.sessionFile) this.persistSession(this.sessionFile);
}
+ private applyModel(data: unknown): void {
+ if (typeof data !== "object" || data === null) return;
+ const model = data as { name?: string; id?: string; input?: unknown };
+ this.model = model.name ?? model.id;
+ this.modelImages = Array.isArray(model.input) ? model.input.includes("image") : undefined;
+ }
+
private readPersistedSession(): string | undefined {
try {
const saved = JSON.parse(readFileSync(this.stateFile, "utf8")) as {
diff --git a/packages/daemon/test/approval-guard.test.ts b/packages/daemon/test/approval-guard.test.ts
index aba0317..a316ec4 100644
--- a/packages/daemon/test/approval-guard.test.ts
+++ b/packages/daemon/test/approval-guard.test.ts
@@ -193,6 +193,37 @@ function credentialTree() {
return { data, agent, workspace, files, guarded, at };
}
+describe("built-in computer server (S24.7)", () => {
+ const withComputer = (): ConnectorPolicy => ({ ...policy("read_only"), builtin: ["computer"] });
+
+ it("runs its tools without approval cards, since the desktop app enforces its own checks", () => {
+ for (const tool of ["mcp__computer__screenshot", "mcp__computer__click", "mcp__computer__type"]) {
+ expect(decide(call(tool, { x: 1, y: 2, text: "send" }), withComputer()), tool).toEqual({
+ action: "pass",
+ });
+ }
+ });
+
+ it("leaves every other server's rules as they were", () => {
+ expect(decide(call("mcp__notion__notion_create_pages", { title: "x" }), withComputer()).action).toBe(
+ "block",
+ );
+ expect(decide(call("mcp__evil__send", { to: "x" }, true), withComputer()).action).toBe("block");
+ // A look-alike server name is not the built-in one.
+ expect(decide(call("mcp__computer_x__click", {}), withComputer()).action).toBe("block");
+ // Without the built-in entry, the computer's tools are an unapproved server.
+ expect(decide(call("mcp__computer__click", {}), policy()).action).toBe("block");
+ });
+
+ it("reads the built-in list from the daemon's policy and rejects a malformed one", () => {
+ expect(parsePolicy(JSON.stringify(withComputer()))).toEqual(withComputer());
+ expect(parsePolicy(JSON.stringify({ ...policy(), builtin: "computer" }))).toBeUndefined();
+ expect(parsePolicy(JSON.stringify({ ...policy(), builtin: [3] }))).toBeUndefined();
+ // Only the computer server can be built in.
+ expect(parsePolicy(JSON.stringify({ ...policy(), builtin: ["notion"] }))).toBeUndefined();
+ });
+});
+
describe("credential and control files (B2)", () => {
it("blocks reading them with the read tool, by any path spelling", () => {
const { files, guarded, at, workspace, data } = credentialTree();
diff --git a/packages/daemon/test/computer.test.ts b/packages/daemon/test/computer.test.ts
new file mode 100644
index 0000000..411461f
--- /dev/null
+++ b/packages/daemon/test/computer.test.ts
@@ -0,0 +1,333 @@
+import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import type { ConnectorInfo, ServerMessage } from "@gentle-dot/protocol";
+import { afterEach, describe, expect, it } from "vitest";
+import WebSocket from "ws";
+import { type DotDaemon, startDaemon } from "../src/daemon.ts";
+import { fakeAuthRuntime } from "./fake-auth-runtime.ts";
+import { FAKE_AGENT, tempDir, waitFor } from "./helpers.ts";
+
+const FAKE_CLI = fileURLToPath(new URL("./fixtures/fake-mcp-cli.ts", import.meta.url));
+const HELPER = "http://127.0.0.1:51234/mcp";
+const KEY = "s3cr3t-Computer-Key-0123456789abcdef";
+
+const daemons: DotDaemon[] = [];
+afterEach(async () => {
+ await Promise.all(daemons.splice(0).map((d) => d.close()));
+});
+
+interface Setup {
+ cliMode?: "auto";
+ /** Written to `<data>/connectors.json` before the daemon starts. */
+ saved?: object;
+ /** Files below the home folder the import scan reads, by relative path. */
+ importFiles?: Record<string, object>;
+}
+
+async function setup(options: Setup = {}) {
+ const dataDir = tempDir();
+ const agentHome = join(dataDir, "agent");
+ const importHome = join(dataDir, "import-home");
+ const envFile = join(dataDir, "agent-env.json");
+ const logs: string[] = [];
+ if (options.saved) writeFileSync(join(dataDir, "connectors.json"), JSON.stringify(options.saved));
+ for (const [path, content] of Object.entries(options.importFiles ?? {})) {
+ mkdirSync(dirname(join(importHome, path)), { recursive: true });
+ writeFileSync(join(importHome, path), JSON.stringify(content));
+ }
+ const fake = fakeAuthRuntime();
+ const d = await startDaemon({
+ port: 0,
+ host: "127.0.0.1",
+ dataDir,
+ workspace: join(dataDir, "workspace"),
+ uiDir: dataDir,
+ agentCommand: process.execPath,
+ agentArgs: [FAKE_AGENT],
+ agentHome,
+ backoffMs: [50],
+ agentEnv: {
+ ...process.env,
+ FAKE_AGENT_ENV_FILE: envFile,
+ ...(options.cliMode ? { FAKE_MCP_CLI_MODE: options.cliMode } : {}),
+ },
+ authRuntime: async () => fake.runtime,
+ connectorCli: { command: process.execPath, args: [FAKE_CLI] },
+ importHome,
+ log: (line) => logs.push(line),
+ });
+ daemons.push(d);
+ const mcpFile = join(agentHome, "mcp.json");
+ const servers = (): Record<string, unknown> =>
+ existsSync(mcpFile) ? JSON.parse(readFileSync(mcpFile, "utf8")).mcpServers : {};
+ const agentEnv = () => readFileSync(envFile, "utf8");
+ const policy = () =>
+ JSON.parse(JSON.parse(agentEnv()).GENTLE_DOT_CONNECTOR_POLICY ?? "{}") as { builtin?: string[] };
+ return {
+ d,
+ dataDir,
+ logs,
+ mcpFile,
+ servers,
+ agentEnv,
+ policy,
+ ...(await connect(d)),
+ connect: () => connect(d),
+ };
+}
+
+async function connect(d: DotDaemon) {
+ const ws = new WebSocket(`ws://127.0.0.1:${d.port}/ws`);
+ const messages: ServerMessage[] = [];
+ ws.on("message", (data) => messages.push(JSON.parse(String(data)) as ServerMessage));
+ await new Promise((resolve) => ws.on("open", resolve));
+ ws.send(JSON.stringify({ type: "hello", token: d.token, protocol: 1 }));
+ await waitFor(() => messages.some((m) => m.type === "ready"));
+ const send = (m: object) => ws.send(JSON.stringify(m));
+ function find<T extends ServerMessage["type"]>(
+ type: T,
+ where: (m: Extract<ServerMessage, { type: T }>) => boolean = () => true,
+ ) {
+ return waitFor(() =>
+ messages.find(
+ (m): m is Extract<ServerMessage, { type: T }> =>
+ m.type === type && where(m as Extract<ServerMessage, { type: T }>),
+ ),
+ );
+ }
+ /** The next connector list after the ones already received. */
+ async function list(where: (connectors: ConnectorInfo[]) => boolean = () => true) {
+ const seen = messages.length;
+ send({ type: "connectors_list" });
+ const found = await waitFor(() =>
+ messages
+ .slice(seen)
+ .find(
+ (m): m is Extract<ServerMessage, { type: "connectors" }> =>
+ m.type === "connectors" && where(m.connectors),
+ ),
+ );
+ return found.connectors;
+ }
+ return { ws, messages, send, find, list };
+}
+
+const computerOf = (connectors: ConnectorInfo[]) => connectors.find((c) => c.id === "computer");
+const settle = () => new Promise((resolve) => setTimeout(resolve, 150));
+
+async function restarted(d: DotDaemon, pid: number | undefined) {
+ await waitFor(() => d.supervisor.pid !== pid && d.supervisor.state === "ready");
+}
+
+describe("computer control registration (S24.7)", () => {
+ it("renders the helper as a built-in direct server with the key in its header, and the key goes nowhere else", async () => {
+ const {
+ d,
+ dataDir,
+ logs,
+ mcpFile,
+ servers,
+ agentEnv,
+ policy,
+ send,
+ find,
+ messages,
+ connect: other,
+ } = await setup();
+ const second = await other();
+ const pid = d.supervisor.pid;
+ expect(policy().builtin).toBeUndefined();
+
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ const listed = await second.find("connectors", (m) => computerOf(m.connectors) !== undefined);
+ expect(computerOf(listed.connectors)).toEqual({
+ id: "computer",
+ name: "Computer",
+ reads: expect.any(String),
+ sends: expect.any(String),
+ added: true,
+ enabled: true,
+ mode: "read_write",
+ status: "connected",
+ noSignIn: true,
+ builtin: true,
+ });
+ // Direct exposure, nothing hidden, and the key only in the header the engine sends.
+ expect(servers().computer).toEqual({
+ url: HELPER,
+ headers: { Authorization: `Bearer ${KEY}` },
+ exposure: "direct",
+ });
+ expect(statSync(mcpFile).mode & 0o777).toBe(0o600);
+ // The engine restarts to read it, and its guard lets the computer's tools run without cards.
+ await restarted(d, pid);
+ await waitFor(() => policy().builtin?.includes("computer"));
+ expect(policy().builtin).toEqual(["computer"]);
+
+ await find("connectors", (m) => computerOf(m.connectors) !== undefined);
+ expect(computerOf(await second.list())).toBeDefined();
+ expect(readFileSync(join(dataDir, "connectors.json"), "utf8")).not.toContain("computer");
+ for (const text of [
+ JSON.stringify(messages),
+ JSON.stringify(second.messages),
+ logs.join("\n"),
+ agentEnv(),
+ readFileSync(join(dataDir, "connectors.json"), "utf8"),
+ ]) {
+ expect(text).not.toContain(KEY);
+ expect(text).not.toContain("51234");
+ }
+ });
+
+ it("removes it on computer_unregister, which only the window that registered it can send", async () => {
+ const { d, servers, send, list, connect: other } = await setup();
+ const second = await other();
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ await waitFor(() => servers().computer);
+ const pid = d.supervisor.pid;
+
+ second.send({ type: "computer_unregister" });
+ await settle();
+ expect(servers().computer).toBeDefined();
+
+ send({ type: "computer_unregister" });
+ await waitFor(() => !servers().computer);
+ expect(computerOf(await list())).toBeUndefined();
+ await waitFor(() => d.supervisor.pid !== pid && d.supervisor.state === "ready");
+ });
+
+ it("removes it when the window that registered it disconnects, not when another one does", async () => {
+ const { servers, send, ws, connect: other } = await setup();
+ const second = await other();
+ const third = await other();
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ await waitFor(() => servers().computer);
+
+ third.ws.close();
+ await settle();
+ expect(servers().computer).toBeDefined();
+
+ ws.close();
+ await waitFor(() => !servers().computer);
+ await second.find("connectors", (m) => computerOf(m.connectors) === undefined);
+ expect(computerOf(await second.list())).toBeUndefined();
+ });
+
+ it("cannot be changed or removed through the connector messages, and no saved connector takes its name", async () => {
+ const { servers, send, messages, list } = await setup({
+ saved: {
+ version: 1,
+ connectors: {
+ computer: {
+ enabled: true,
+ mode: "read_write",
+ custom: { name: "Computer", origin: "x", server: { url: "http://127.0.0.1:9/mcp" }, fields: [] },
+ },
+ },
+ },
+ });
+ expect(computerOf(await list())).toBeUndefined();
+ expect(servers().computer).toBeUndefined();
+
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ await waitFor(() => servers().computer);
+ const before = JSON.stringify(servers().computer);
+ const commands = [
+ { type: "connector_connect" },
+ { type: "connector_signin" },
+ { type: "connector_setup" },
+ { type: "connector_disconnect" },
+ { type: "connector_mode", mode: "read_only" },
+ { type: "connector_remove" },
+ ];
+ const seen = messages.length;
+ for (const command of commands) send({ ...command, connectorId: "computer" });
+ const refusals = await waitFor(() => {
+ const errors = messages.slice(seen).filter((m) => m.type === "error");
+ return errors.length >= commands.length && errors;
+ });
+ expect(refusals.map((m) => m.type === "error" && m.code)).toEqual(
+ commands.map(() => "unknown_connector"),
+ );
+ await settle();
+ expect(messages.slice(seen).some((m) => m.type === "auth_prompt" || m.type === "auth_event")).toBe(false);
+ expect(JSON.stringify(servers().computer)).toBe(before);
+ expect(computerOf(await list())).toMatchObject({ builtin: true, enabled: true, mode: "read_write" });
+ });
+
+ it("never appears in the import scan, and an imported server never gets its name", async () => {
+ const { servers, send, find } = await setup({
+ importFiles: {
+ "Library/Application Support/Claude/claude_desktop_config.json": {
+ mcpServers: {
+ helper: { url: HELPER, headers: { Authorization: `Bearer ${KEY}` } },
+ computer: { command: "npx", args: ["-y", "[email protected]"] },
+ },
+ },
+ },
+ });
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ await waitFor(() => servers().computer);
+
+ send({ type: "connectors_scan" });
+ const { found } = await find("connector_imports");
+ expect(found.map((c) => c.name)).toEqual(["computer"]);
+ expect(JSON.stringify(found)).not.toContain("51234");
+
+ send({ type: "connector_import", ids: found.map((c) => c.id) });
+ expect((await find("connector_imported")).names).toEqual(["computer-2"]);
+ await waitFor(() => servers()["computer-2"]);
+ expect(servers().computer).toEqual({
+ url: HELPER,
+ headers: { Authorization: `Bearer ${KEY}` },
+ exposure: "direct",
+ });
+ });
+
+ it("says when the current model cannot see images", async () => {
+ const { d, send, list } = await setup();
+ const pid = d.supervisor.pid;
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ await restarted(d, pid);
+ // The fake engine's default model accepts images.
+ expect(computerOf(await list((c) => computerOf(c) !== undefined))?.noImages).toBeUndefined();
+ await d.supervisor.request({ type: "set_model", provider: "fake", modelId: "fake-fast" });
+ expect(computerOf(await list())?.noImages).toBe(true);
+ await d.supervisor.request({ type: "set_model", provider: "fake", modelId: "fake-model" });
+ expect(computerOf(await list())?.noImages).toBeUndefined();
+ });
+});
+
+describe("connectors next to the computer (regression)", () => {
+ it("connects, changes, and removes a connector while the helper is registered, and puts back a tampered mcp.json", async () => {
+ const { mcpFile, servers, send, find, list } = await setup({ cliMode: "auto" });
+ send({ type: "computer_register", url: HELPER, token: KEY });
+ await waitFor(() => servers().computer);
+
+ send({ type: "connector_connect", connectorId: "notion" });
+ expect(await find("auth_done")).toMatchObject({ ok: true, providerId: "notion" });
+ const connected = await list((c) => c.find((x) => x.id === "notion")?.status === "connected");
+ expect(connected.map((c) => [c.id, c.status])).toEqual([
+ ["notion", "connected"],
+ ["linear", "off"],
+ ["atlassian", "off"],
+ ["discord", "off"],
+ ["slack", "off"],
+ ["gmail", "off"],
+ ["computer", "connected"],
+ ]);
+ expect(Object.keys(servers())).toEqual(["notion", "computer"]);
+ expect(servers().notion).toMatchObject({ url: "https://mcp.notion.com/mcp", exposure: "direct" });
+
+ send({ type: "connector_mode", connectorId: "notion", mode: "read_write" });
+ await waitFor(() => !(servers().notion as { toolExposure?: unknown }).toolExposure);
+
+ writeFileSync(mcpFile, '{"mcpServers":{}}\n');
+ await waitFor(() => servers().computer && servers().notion);
+
+ send({ type: "connector_remove", connectorId: "notion" });
+ await waitFor(() => !servers().notion);
+ expect(Object.keys(servers())).toEqual(["computer"]);
+ });
+});
diff --git a/packages/daemon/test/fixtures/fake-agent.ts b/packages/daemon/test/fixtures/fake-agent.ts
index 9125bcc..ddadb7c 100644
--- a/packages/daemon/test/fixtures/fake-agent.ts
+++ b/packages/daemon/test/fixtures/fake-agent.ts
@@ -37,7 +37,8 @@
// FAKE_AGENT_START_DELAY_MS answer the first get_state only after this delay
// FAKE_AGENT_DROP_CUSTOM load session files without their custom messages (a format drift)
// get_available_models lists MODELS; set_model and set_thinking_level change what
-// get_state reports, so tests can read the last values back.
+// get_state reports, so tests can read the last values back. A model's `input` says whether it
+// accepts images, like the engine's models.
import { randomUUID } from "node:crypto";
import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
@@ -51,12 +52,13 @@ const MODELS: Rec[] = [
name: "Fake Model",
provider: "fake",
reasoning: true,
+ input: ["text", "image"],
// Real models may carry credentials in headers; they must never reach a client.
headers: { Authorization: "Bearer fake-secret-header" },
baseUrl: "https://internal.fake.example",
cost: { input: 1, output: 2 },
},
- { id: "fake-fast", name: "Fake Fast", provider: "fake", reasoning: false },
+ { id: "fake-fast", name: "Fake Fast", provider: "fake", reasoning: false, input: ["text"] },
{ id: "big", name: "Other Big", provider: "other", reasoning: true },
];
let model = MODELS[0] as Rec;
@@ -379,7 +381,7 @@ function handle(rec: Rec) {
}
started = true;
return respond(id, "get_state", {
- model: { id: model.id, name: model.name, provider: model.provider },
+ model: { id: model.id, name: model.name, provider: model.provider, input: model.input },
thinkingLevel,
isStreaming: busy,
isCompacting: false,
diff --git a/packages/protocol/src/index.ts b/packages/protocol/src/index.ts
index 5d69613..92f268b 100644
--- a/packages/protocol/src/index.ts
+++ b/packages/protocol/src/index.ts
@@ -122,6 +122,13 @@ export interface ConnectorInfo {
custom?: { origin: string; summary: string };
/** It has no sign-in of its own (it uses a token, or none). */
noSignIn?: true;
+ /**
+ * Built into the desktop app (Computer): listed only while the app has it registered, and not
+ * changed or removed through the connector messages. Its address and key are never sent.
+ */
+ builtin?: true;
+ /** The current model does not accept images, so the assistant cannot see the screenshots. */
+ noImages?: true;
}
/** The daemon keeps at most this many drafts waiting; a newer one drops the oldest. */
@@ -269,7 +276,10 @@ export type ClientMessage =
| { type: "connector_mode"; connectorId: string; mode: ConnectorMode }
| { type: "connector_draft_reply"; draftId: string; approve: boolean }
| { type: "connectors_scan" }
- | { type: "connector_import"; ids: string[] };
+ | { type: "connector_import"; ids: string[] }
+ /** The desktop app's computer-control helper (macOS), at `http://127.0.0.1:<port>/mcp` with its key. */
+ | { type: "computer_register"; url: string; token: string }
+ | { type: "computer_unregister" };
export type ServerPayload =
| { type: "ready"; agentState: AgentState; conversationId?: string; model?: string; features?: Features }
@@ -334,6 +344,15 @@ const CONNECTOR_ID = /^[a-z0-9][a-z0-9-]{0,39}$/;
const isConnectorId = (value: unknown): value is string => isString(value) && CONNECTOR_ID.test(value);
const isShortString = (value: unknown): value is string => isString(value) && value.length <= 200;
const MAX_IMPORTS = 200;
+const COMPUTER_URL = /^http:\/\/127\.0\.0\.1:([1-9][0-9]{0,4})\/mcp$/;
+/** The helper's key goes into an `Authorization` header, so only header-safe characters. */
+const COMPUTER_TOKEN = /^[A-Za-z0-9._~+/=-]{16,512}$/;
+
+/** Only the helper's loopback address: `http://127.0.0.1:<port>/mcp`, nothing else. */
+export function isComputerUrl(value: unknown): value is string {
+ const port = isString(value) ? COMPUTER_URL.exec(value)?.[1] : undefined;
+ return port !== undefined && Number(port) <= 65_535;
+}
/**
* Reads the complete queues from the engine's `queue_update` record. A missing
@@ -437,6 +456,11 @@ export function parseClientMessage(raw: string): ClientMessage | undefined {
return Array.isArray(m.ids) && m.ids.length <= MAX_IMPORTS && m.ids.every(isShortString)
? { type: "connector_import", ids: m.ids }
: undefined;
+ case "computer_register":
+ return isComputerUrl(m.url) && isString(m.token) && COMPUTER_TOKEN.test(m.token)
+ ? { type: "computer_register", url: m.url, token: m.token }
+ : undefined;
+ case "computer_unregister":
case "connectors_list":
case "connectors_scan":
case "profiles_list":
diff --git a/packages/protocol/test/protocol.test.ts b/packages/protocol/test/protocol.test.ts
index f6dad08..65ac9bf 100644
--- a/packages/protocol/test/protocol.test.ts
+++ b/packages/protocol/test/protocol.test.ts
@@ -251,3 +251,70 @@ describe("connector messages", () => {
}
});
});
+
+describe("computer control messages", () => {
+ const parse = (value: object) => parseClientMessage(JSON.stringify(value));
+ const token = "k3Y_0123456789abcdef-ABCDEF.~+/=";
+
+ it("registers the desktop app's helper at a loopback MCP address, and unregisters it", () => {
+ expect(parse({ type: "computer_register", url: "http://127.0.0.1:51234/mcp", token })).toEqual({
+ type: "computer_register",
+ url: "http://127.0.0.1:51234/mcp",
+ token,
+ });
+ expect(parse({ type: "computer_register", url: "http://127.0.0.1:1/mcp", token, extra: 1 })).toEqual({
+ type: "computer_register",
+ url: "http://127.0.0.1:1/mcp",
+ token,
+ });
+ expect(parse({ type: "computer_register", url: "http://127.0.0.1:65535/mcp", token })).toMatchObject({
+ url: "http://127.0.0.1:65535/mcp",
+ });
+ expect(parse({ type: "computer_unregister" })).toEqual({ type: "computer_unregister" });
+ });
+
+ it("rejects any address other than http://127.0.0.1:<port>/mcp", () => {
+ for (const url of [
+ "https://127.0.0.1:51234/mcp",
+ "http://localhost:51234/mcp",
+ "http://[::1]:51234/mcp",
+ "http://127.0.0.2:51234/mcp",
+ "http://0.0.0.0:51234/mcp",
+ "http://example.com:51234/mcp",
+ "http://127.0.0.1/mcp",
+ "http://127.0.0.1:0/mcp",
+ "http://127.0.0.1:65536/mcp",
+ "http://127.0.0.1:051234/mcp",
+ "http://127.0.0.1:51234/",
+ "http://127.0.0.1:51234/mcp/",
+ "http://127.0.0.1:51234/mcp/x",
+ "http://127.0.0.1:51234/MCP",
+ "http://127.0.0.1:51234/mcp?x=1",
+ "http://127.0.0.1:51234/mcp#x",
+ "http://user:[email protected]:51234/mcp",
+ "http://127.0.0.1:[email protected]/mcp",
+ " http://127.0.0.1:51234/mcp",
+ "http://127.0.0.1:51234/mcp\n",
+ ]) {
+ expect(parse({ type: "computer_register", url, token }), url).toBeUndefined();
+ }
+ expect(parse({ type: "computer_register", token })).toBeUndefined();
+ expect(parse({ type: "computer_register", url: 51234, token })).toBeUndefined();
+ });
+
+ it("rejects a key that is missing, too short or long, or not safe for a header", () => {
+ const url = "http://127.0.0.1:51234/mcp";
+ for (const bad of [
+ undefined,
+ 7,
+ "",
+ "short",
+ "x".repeat(513),
+ `${"a".repeat(20)} b`,
+ `${"a".repeat(20)}\r\nX-Evil: 1`,
+ `${"a".repeat(20)}"`,
+ ]) {
+ expect(parse({ type: "computer_register", url, token: bad }), String(bad)).toBeUndefined();
+ }
+ });
+});
diff --git a/packages/ui/src/App.tsx b/packages/ui/src/App.tsx
index 83bfca1..872d6f0 100644
--- a/packages/ui/src/App.tsx
+++ b/packages/ui/src/App.tsx
@@ -1,7 +1,9 @@
import { useEffect, useRef, useState } from "react";
import { type ConnectionInfo, tokenFromLocation } from "./client.ts";
import { ChatSurface } from "./components/ChatSurface.tsx";
+import type { ComputerControls } from "./components/Computer.tsx";
import { DotSurface } from "./components/DotSurface.tsx";
+import { computerPermissions, requestComputerPermission, stopComputer } from "./computer.ts";
import {
connectionInfo,
hidePanel,
@@ -11,6 +13,7 @@ import {
startDragging,
togglePanel,
} from "./desktop.ts";
+import { useComputerRegistration, useComputerSession } from "./useComputer.ts";
import { useDot } from "./useDot.ts";
type Surface = "dot" | "panel" | "web";
@@ -47,6 +50,24 @@ export function App() {
const { state, send, dismiss, dispatch } = useDot(info);
const conversations = useRef(state.features.conversations);
conversations.current = state.features.conversations;
+ // Computer control lives in the desktop app only (S24.7); the panel registers the helper.
+ const desktop = inDesktop();
+ const computerSession = useComputerSession(desktop && surface !== "web");
+ const computerAvailable = useComputerRegistration(
+ desktop && surface === "panel",
+ state.connection === "open",
+ send,
+ );
+ const computer: ComputerControls | undefined =
+ desktop && surface === "panel"
+ ? {
+ available: computerAvailable,
+ ...(computerSession ? { session: computerSession } : {}),
+ stop: stopComputer,
+ permissions: computerPermissions,
+ requestPermission: requestComputerPermission,
+ }
+ : undefined;
useEffect(() => {
if (inDesktop()) {
@@ -91,6 +112,7 @@ export function App() {
<DotSurface
agentState={state.agentState}
connected={state.connection === "open"}
+ inControl={computerSession !== undefined}
toggle={togglePanel}
startDrag={() => void startDragging()}
/>
@@ -114,6 +136,7 @@ export function App() {
dispatch={dispatch}
focusKey={focusKey}
{...(surface === "panel" ? { onHide: hidePanel } : {})}
+ {...(computer ? { computer } : {})}
/>
);
}
diff --git a/packages/ui/src/components/ChatSurface.tsx b/packages/ui/src/components/ChatSurface.tsx
index ece9c9d..041dcf6 100644
--- a/packages/ui/src/components/ChatSurface.tsx
+++ b/packages/ui/src/components/ChatSurface.tsx
@@ -5,6 +5,7 @@ import { activeTitle, type DotAction, type DotState, isBusy, needsAccount } from
import { AccountsPanel } from "./AccountsPanel.tsx";
import { AskCard } from "./AskCard.tsx";
import { Composer } from "./Composer.tsx";
+import { ComputerBanner, type ComputerControls } from "./Computer.tsx";
import { ConnectorDraftCard } from "./ConnectorDraftCard.tsx";
import { ConnectorsPanel } from "./ConnectorsPanel.tsx";
import {
@@ -29,6 +30,8 @@ interface ChatSurfaceProps {
focusKey?: number;
dispatch?: (action: DotAction) => void;
openUrl?: (url: string) => void;
+ /** Computer control, in the desktop panel only. */
+ computer?: ComputerControls;
}
const CONTINUE_TEXT = "Continue where you left off.";
@@ -55,6 +58,7 @@ export function ChatSurface({
focusKey,
dispatch = () => {},
openUrl = openExternal,
+ computer,
}: ChatSurfaceProps) {
const openAccounts = () => {
send({ type: "auth_list" });
@@ -152,6 +156,8 @@ export function ChatSurface({
) : null}
</header>
+ {computer?.session ? <ComputerBanner session={computer.session} stop={computer.stop} /> : null}
+
{conversations && showConversations ? (
<nav className="conversations" aria-label="Earlier conversations">
<p className="eyebrow">Conversations</p>
@@ -200,7 +206,13 @@ export function ChatSurface({
{state.connectors.open && !state.auth.open && !state.profiles.open ? (
<main className="chat-body">
- <ConnectorsPanel connectors={state.connectors} send={send} dispatch={dispatch} openUrl={openUrl} />
+ <ConnectorsPanel
+ connectors={state.connectors}
+ send={send}
+ dispatch={dispatch}
+ openUrl={openUrl}
+ {...(computer?.available ? { computer } : {})}
+ />
</main>
) : null}
diff --git a/packages/ui/src/components/Computer.tsx b/packages/ui/src/components/Computer.tsx
new file mode 100644
index 0000000..401f53a
--- /dev/null
+++ b/packages/ui/src/components/Computer.tsx
@@ -0,0 +1,112 @@
+import type { ConnectorInfo } from "@gentle-dot/protocol";
+import { useCallback, useEffect, useState } from "react";
+import {
+ type ComputerPermission,
+ type ComputerPermissions,
+ type ComputerSession,
+ timeLeft,
+} from "../computer.ts";
+
+/** Computer control in the desktop panel (S24.5, S24.6); the web page never gets it. */
+export interface ComputerControls {
+ /** The app has a helper (macOS), so the Connectors screen offers it. */
+ available: boolean;
+ /** The session the user allowed, while it lasts. */
+ session?: ComputerSession;
+ stop: () => void;
+ permissions: () => Promise<ComputerPermissions>;
+ requestPermission: (kind: ComputerPermission) => Promise<unknown>;
+}
+
+const PERMISSIONS: [ComputerPermission, string][] = [
+ ["accessibility", "Accessibility"],
+ ["screenRecording", "Screen Recording"],
+];
+
+export const DEBUG_BUILD_NOTE =
+ "Debug builds lose these permissions on every rebuild because macOS ties them to the app's signature; grant them again after updating.";
+const NO_IMAGES_NOTE =
+ "The current model does not accept images, so the assistant cannot see your screen. Choose a model that does in Profiles.";
+
+/** "Controlling your Mac · mm:ss left" with a Stop button, while a session is active. */
+export function ComputerBanner({ session, stop }: { session: ComputerSession; stop: () => void }) {
+ const [now, setNow] = useState(() => Date.now());
+ useEffect(() => {
+ const timer = setInterval(() => setNow(Date.now()), 1000);
+ return () => clearInterval(timer);
+ }, []);
+ const left = session.endsAt === undefined ? "" : ` · ${timeLeft(session.endsAt, now)} left`;
+ return (
+ <section className="computer-banner" aria-label="Computer control">
+ <span className="computer-banner-light" aria-hidden="true" />
+ <span>Controlling your Mac{left}</span>
+ <button type="button" className="primary" onClick={stop}>
+ Stop
+ </button>
+ </section>
+ );
+}
+
+interface ComputerEntryProps {
+ /** The daemon's built-in entry, once the helper is registered. */
+ info?: ConnectorInfo;
+ controls: ComputerControls;
+}
+
+/** The built-in Computer connector: the macOS permissions it needs and how to grant them. */
+export function ComputerEntry({ info, controls }: ComputerEntryProps) {
+ const { permissions, requestPermission } = controls;
+ const [granted, setGranted] = useState<ComputerPermissions>();
+ const check = useCallback(() => {
+ permissions().then(setGranted, () => setGranted(undefined));
+ }, [permissions]);
+ useEffect(check, [check]);
+
+ return (
+ <li className="computer-entry" aria-labelledby="connector-computer">
+ <div className="provider-name">
+ <b id="connector-computer">Computer</b>
+ <span className="chip connector-connected">Built in</span>
+ </div>
+ {info ? (
+ <>
+ <p className="connector-note">{info.reads}</p>
+ <p className="connector-note">{info.sends}</p>
+ </>
+ ) : null}
+ <ul className="computer-permissions">
+ {PERMISSIONS.map(([kind, label]) => {
+ const allowed = granted?.[kind] === true;
+ const id = `computer-permission-${kind}`;
+ return (
+ <li key={kind} aria-labelledby={id}>
+ <span id={id}>{label}</span>
+ <span className={`chip ${allowed ? "connector-connected" : "connector-needs_setup"}`}>
+ {granted === undefined ? "Checking…" : allowed ? "Allowed" : "Not allowed"}
+ </span>
+ {granted !== undefined && !allowed ? (
+ <button
+ type="button"
+ className="primary"
+ aria-label={`Grant ${label}`}
+ onClick={() => {
+ requestPermission(kind).then(check, check);
+ }}
+ >
+ Grant
+ </button>
+ ) : null}
+ </li>
+ );
+ })}
+ </ul>
+ <div className="provider-actions">
+ <button type="button" onClick={check}>
+ Check again
+ </button>
+ </div>
+ <p className="connector-note">{DEBUG_BUILD_NOTE}</p>
+ {info?.noImages ? <p className="connector-note computer-warning">{NO_IMAGES_NOTE}</p> : null}
+ </li>
+ );
+}
diff --git a/packages/ui/src/components/ConnectorsPanel.tsx b/packages/ui/src/components/ConnectorsPanel.tsx
index 8eae156..805a701 100644
--- a/packages/ui/src/components/ConnectorsPanel.tsx
+++ b/packages/ui/src/components/ConnectorsPanel.tsx
@@ -3,6 +3,7 @@ import { useState } from "react";
import type { ConnectorsState, DotAction } from "../store.ts";
import "../accounts.css";
import { FlowEvent, FlowPrompt } from "./AccountsPanel.tsx";
+import { type ComputerControls, ComputerEntry } from "./Computer.tsx";
import type { Send } from "./types.ts";
interface ConnectorsPanelProps {
@@ -10,6 +11,8 @@ interface ConnectorsPanelProps {
send: Send;
dispatch: (action: DotAction) => void;
openUrl: (url: string) => void;
+ /** Computer control, in the desktop app on macOS only. */
+ computer?: ComputerControls;
}
const STATUS_LABELS: Record<ConnectorStatus, string> = {
@@ -25,7 +28,7 @@ export const addConnectorRequest = (what: string) =>
`Please help me add a connector for ${what}. Find its MCP server and draft it for me to review.`;
/** The apps the assistant can use, each signed in from here and set to read only or read and send. */
-export function ConnectorsPanel({ connectors, send, dispatch, openUrl }: ConnectorsPanelProps) {
+export function ConnectorsPanel({ connectors, send, dispatch, openUrl, computer }: ConnectorsPanelProps) {
const [guideFor, setGuideFor] = useState<string>();
const [adding, setAdding] = useState(false);
const flow = connectors.flow;
@@ -129,20 +132,29 @@ export function ConnectorsPanel({ connectors, send, dispatch, openUrl }: Connect
/>
) : null}
<ul className="providers">
- {(connectors.list ?? []).map((connector) => (
- <ConnectorRow
- key={connector.id}
- connector={connector}
- send={send}
- connect={() =>
- connector.guide && !connector.added
- ? setGuideFor(connector.id)
- : start(connector, "connector_connect")
- }
- signIn={() => start(connector, "connector_signin")}
- setUp={() => start(connector, "connector_setup")}
+ {computer ? (
+ <ComputerEntry
+ controls={computer}
+ {...withInfo(connectors.list?.find((c) => c.builtin && c.id === "computer"))}
/>
- ))}
+ ) : null}
+ {/* Built-in entries are the desktop app's; the web page never shows them. */}
+ {(connectors.list ?? [])
+ .filter((c) => !c.builtin)
+ .map((connector) => (
+ <ConnectorRow
+ key={connector.id}
+ connector={connector}
+ send={send}
+ connect={() =>
+ connector.guide && !connector.added
+ ? setGuideFor(connector.id)
+ : start(connector, "connector_connect")
+ }
+ signIn={() => start(connector, "connector_signin")}
+ setUp={() => start(connector, "connector_setup")}
+ />
+ ))}
</ul>
</>
)}
@@ -150,6 +162,8 @@ export function ConnectorsPanel({ connectors, send, dispatch, openUrl }: Connect
);
}
+const withInfo = (info: ConnectorInfo | undefined) => (info ? { info } : {});
+
interface ConnectorGuideViewProps {
connector: ConnectorInfo;
openUrl: (url: string) => void;
diff --git a/packages/ui/src/components/DotSurface.tsx b/packages/ui/src/components/DotSurface.tsx
index 6715c63..de86bb1 100644
--- a/packages/ui/src/components/DotSurface.tsx
+++ b/packages/ui/src/components/DotSurface.tsx
@@ -27,20 +27,22 @@ const DRAG_THRESHOLD_PX = 3;
interface DotSurfaceProps {
agentState: AgentState;
connected: boolean;
+ /** The assistant controls the Mac (S24.6); it shows over every other state. */
+ inControl?: boolean;
toggle: () => void;
startDrag: () => void;
}
/** The collapsed, always-on-top rose in its black circle. A click opens the panel; a drag moves the window. */
-export function DotSurface({ agentState, connected, toggle, startDrag }: DotSurfaceProps) {
+export function DotSurface({ agentState, connected, inControl = false, toggle, startDrag }: DotSurfaceProps) {
const press = useRef<{ x: number; y: number; dragging: boolean } | undefined>(undefined);
const state = connected ? agentState : "restarting";
return (
<button
type="button"
- className={`dot dot-${state}`}
- aria-label={`Gentle Dot: ${LABELS[state]}`}
+ className={`dot dot-${state}${inControl ? " dot-in_control" : ""}`}
+ aria-label={`Gentle Dot: ${inControl ? "controlling your Mac" : LABELS[state]}`}
onPointerDown={(event) => {
if (event.button !== 0) return;
press.current = { x: event.clientX, y: event.clientY, dragging: false };
@@ -66,7 +68,7 @@ export function DotSurface({ agentState, connected, toggle, startDrag }: DotSurf
}}
>
<span className="dot-disc" aria-hidden="true">
- <Rose mood={MOODS[state]} />
+ <Rose mood={inControl ? "control" : MOODS[state]} />
</span>
{state === "needs_you" ? (
<span className="dot-badge" aria-hidden="true">
diff --git a/packages/ui/src/computer.ts b/packages/ui/src/computer.ts
new file mode 100644
index 0000000..3a0d272
--- /dev/null
+++ b/packages/ui/src/computer.ts
@@ -0,0 +1,81 @@
+import { invoke, isTauri } from "@tauri-apps/api/core";
+
+/**
+ * Computer control on macOS (S24): the desktop app's helper, reached through its Tauri commands.
+ * Outside the desktop app (the web page, a VPS) every call is a no-op, so computer control is never offered.
+ */
+
+/** The helper's MCP address and its key for this launch of the app. */
+export interface ComputerEndpoint {
+ url: string;
+ token: string;
+}
+
+export interface ComputerPermissions {
+ accessibility: boolean;
+ screenRecording: boolean;
+}
+
+export type ComputerPermission = keyof ComputerPermissions;
+
+/** A control session the user allowed; `endsAt` in milliseconds since the epoch. */
+export interface ComputerSession {
+ endsAt?: number;
+}
+
+/** What the app reports on `computer://state` and from `computer_status`. */
+interface ComputerStateEvent {
+ active?: unknown;
+ endsAt?: unknown;
+}
+
+/** The helper's endpoint, or null when the app has none (off macOS, or the helper did not start). */
+export async function computerEndpoint(): Promise<ComputerEndpoint | null> {
+ if (!isTauri()) return null;
+ const endpoint = await invoke<Partial<ComputerEndpoint> | null>("computer_endpoint");
+ return typeof endpoint?.url === "string" && typeof endpoint.token === "string"
+ ? { url: endpoint.url, token: endpoint.token }
+ : null;
+}
+
+export function computerPermissions(): Promise<ComputerPermissions> {
+ if (!isTauri()) return Promise.resolve({ accessibility: false, screenRecording: false });
+ return invoke<ComputerPermissions>("computer_permissions");
+}
+
+/** Prompts for the permission and opens its System Settings pane. */
+export function requestComputerPermission(kind: ComputerPermission): Promise<unknown> {
+ if (!isTauri()) return Promise.resolve();
+ return invoke("computer_request_permission", { kind });
+}
+
+/** Ends the session now; queued actions are dropped by the app. */
+export function stopComputer(): void {
+ if (isTauri()) void invoke("computer_stop");
+}
+
+export async function computerStatus(): Promise<ComputerSession | undefined> {
+ if (!isTauri()) return undefined;
+ return sessionOf(await invoke<ComputerStateEvent>("computer_status"));
+}
+
+export async function onComputerState(handler: (session: ComputerSession | undefined) => void) {
+ if (!isTauri()) return () => {};
+ const { listen } = await import("@tauri-apps/api/event");
+ return listen<ComputerStateEvent>("computer://state", (event) => handler(sessionOf(event.payload)));
+}
+
+/** The session while it is active; an `endsAt` in seconds is read as such. */
+function sessionOf(state: ComputerStateEvent | null | undefined): ComputerSession | undefined {
+ if (state?.active !== true) return undefined;
+ const endsAt = state.endsAt;
+ if (typeof endsAt !== "number" || !Number.isFinite(endsAt)) return {};
+ return { endsAt: endsAt < 1e12 ? endsAt * 1000 : endsAt };
+}
+
+/** `mm:ss` left until `endsAt`, never below zero. */
+export function timeLeft(endsAt: number, now: number): string {
+ const seconds = Math.max(0, Math.ceil((endsAt - now) / 1000));
+ const pad = (n: number) => String(n).padStart(2, "0");
+ return `${pad(Math.floor(seconds / 60))}:${pad(seconds % 60)}`;
+}
diff --git a/packages/ui/src/rose/Rose.tsx b/packages/ui/src/rose/Rose.tsx
index 5106574..204c36d 100644
--- a/packages/ui/src/rose/Rose.tsx
+++ b/packages/ui/src/rose/Rose.tsx
@@ -2,7 +2,7 @@ import type { CSSProperties } from "react";
import { ROSE_SPARKLES, ROSE_STROKES } from "./strokes.ts";
/** How the rose looks (design sheet `docs/brand/rose-design.html`). */
-export type RoseMood = "idle" | "thinking" | "working" | "needs" | "restarting" | "offline";
+export type RoseMood = "idle" | "thinking" | "working" | "needs" | "restarting" | "offline" | "control";
/**
* The square crop that frames the rose in the Dot's circle: centered on the
diff --git a/packages/ui/src/styles.css b/packages/ui/src/styles.css
index 828e695..ed27a9a 100644
--- a/packages/ui/src/styles.css
+++ b/packages/ui/src/styles.css
@@ -263,6 +263,12 @@ textarea:focus {
--rim: rgba(255, 255, 255, 0.06);
}
+/* In control of the Mac (S24.6): the accent rim glows steadily over every other state. */
+.dot-in_control .dot-disc {
+ --rim: rgba(240, 149, 200, 0.8);
+ --rim-glow: rgba(240, 149, 200, 0.35);
+}
+
.rose {
position: relative;
display: block;
@@ -403,6 +409,27 @@ textarea:focus {
filter: none;
}
+/* In control: the whole rose lit and calling, with a steady, quick light along every stroke. */
+.rose-control .rose-base {
+ opacity: 1;
+ animation: rose-call 2s ease-in-out infinite;
+}
+
+.rose-control .rose-light {
+ opacity: 1;
+}
+
+.rose-control .rose-light path {
+ stroke-dasharray: 0.3 0.4;
+ animation: rose-run calc(var(--d) * 0.8) linear infinite;
+ animation-delay: calc(var(--i) * -0.2s);
+}
+
+.rose-control .rose-sparkles circle {
+ animation: rose-twinkle 1.2s ease-in-out infinite;
+ animation-delay: calc(var(--i) * -0.17s);
+}
+
/* On the circle's edge, at the top right. */
.dot-badge {
position: absolute;
@@ -430,7 +457,8 @@ textarea:focus {
}
.rose-working .rose-base,
- .rose-needs .rose-base {
+ .rose-needs .rose-base,
+ .rose-control .rose-base {
filter: drop-shadow(0 0 1.5px var(--neon)) drop-shadow(0 0 4px var(--neon));
}
}
@@ -630,6 +658,68 @@ html[data-effects="none"] .chat-panel {
line-height: 16px;
}
+/* While the assistant controls the Mac (S24.6): time left and Stop, always in view. */
+.computer-banner {
+ display: flex;
+ align-items: center;
+ gap: 10px;
+ margin: 10px 12px 0;
+ padding: 8px 8px 8px 12px;
+ border-radius: var(--radius-md);
+ border: 1px solid var(--accent-line);
+ background: var(--accent-soft);
+ color: var(--text);
+ font-size: 13px;
+ font-weight: 600;
+ line-height: 18px;
+}
+
+.computer-banner > span:nth-child(2) {
+ flex: 1;
+ font-variant-numeric: tabular-nums;
+}
+
+.computer-banner button {
+ padding: 4px 14px;
+}
+
+.computer-banner-light {
+ width: 8px;
+ height: 8px;
+ border-radius: var(--radius-pill);
+ background: var(--accent);
+ box-shadow: 0 0 8px var(--accent);
+ animation: pulse 1.6s ease-in-out infinite;
+}
+
+.computer-permissions {
+ display: grid;
+ gap: 6px;
+ margin: 8px 0;
+ padding: 0;
+ list-style: none;
+}
+
+.computer-permissions li {
+ display: flex;
+ align-items: center;
+ gap: 8px;
+ font-size: 13px;
+}
+
+.computer-permissions li > span:first-child {
+ flex: 1;
+}
+
+.computer-permissions button {
+ padding: 3px 12px;
+ font-size: 12px;
+}
+
+.computer-warning {
+ color: var(--yellow);
+}
+
.chat-body {
flex: 1;
overflow-y: auto;
diff --git a/packages/ui/src/useComputer.ts b/packages/ui/src/useComputer.ts
new file mode 100644
index 0000000..2f9bd9f
--- /dev/null
+++ b/packages/ui/src/useComputer.ts
@@ -0,0 +1,52 @@
+import type { ClientMessage } from "@gentle-dot/protocol";
+import { useEffect, useState } from "react";
+import { type ComputerSession, computerEndpoint, computerStatus, onComputerState } from "./computer.ts";
+
+/** The control session the app reports, while one is active (desktop windows only). */
+export function useComputerSession(enabled: boolean): ComputerSession | undefined {
+ const [session, setSession] = useState<ComputerSession | undefined>(undefined);
+ useEffect(() => {
+ if (!enabled) return;
+ let live = true;
+ const update = (next: ComputerSession | undefined) => {
+ if (live) setSession(next);
+ };
+ computerStatus().then(update, () => {});
+ const unlisten = onComputerState(update);
+ return () => {
+ live = false;
+ void unlisten.then((stop) => stop());
+ };
+ }, [enabled]);
+ return session;
+}
+
+/**
+ * Registers the app's helper with the daemon on every connection (S24.7); the daemon drops it when
+ * this window disconnects. True when the app has a helper. The key is passed on, never kept.
+ */
+export function useComputerRegistration(
+ enabled: boolean,
+ connected: boolean,
+ send: (message: ClientMessage) => void,
+): boolean {
+ const [available, setAvailable] = useState(false);
+ useEffect(() => {
+ if (!enabled || !connected) return;
+ let live = true;
+ computerEndpoint().then(
+ (endpoint) => {
+ if (!live) return;
+ setAvailable(endpoint !== null);
+ if (endpoint) send({ type: "computer_register", url: endpoint.url, token: endpoint.token });
+ },
+ () => {
+ if (live) setAvailable(false);
+ },
+ );
+ return () => {
+ live = false;
+ };
+ }, [enabled, connected, send]);
+ return available;
+}
diff --git a/packages/ui/test/computer.test.tsx b/packages/ui/test/computer.test.tsx
new file mode 100644
index 0000000..cf44c37
--- /dev/null
+++ b/packages/ui/test/computer.test.tsx
@@ -0,0 +1,270 @@
+import { readFileSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+import type { ConnectorInfo } from "@gentle-dot/protocol";
+import { act, render, screen, waitFor, within } from "@testing-library/react";
+import userEvent from "@testing-library/user-event";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { type DotState, initialState } from "../src/store.ts";
+
+const HELPER = "http://127.0.0.1:51234/mcp";
+const KEY = "s3cr3t-Computer-Key-0123456789abcdef";
+
+const h = vi.hoisted(() => ({
+ tauri: true,
+ endpoint: null as unknown,
+ permissions: { accessibility: false, screenRecording: true },
+ status: { active: false } as unknown,
+ invoke: vi.fn(),
+ send: vi.fn(),
+ listeners: new Map<string, (event: { payload: unknown }) => void>(),
+ state: undefined as unknown as DotState,
+}));
+
+vi.mock("@tauri-apps/api/core", () => ({
+ isTauri: () => h.tauri,
+ invoke: (command: string, args?: unknown) => h.invoke(command, args),
+}));
+
+vi.mock("@tauri-apps/api/event", () => ({
+ listen: async (name: string, handler: (event: { payload: unknown }) => void) => {
+ h.listeners.set(name, handler);
+ return () => h.listeners.delete(name);
+ },
+}));
+
+// The panel's other desktop events are not under test; concurrent first imports of the mocked
+// event module can reach the real one, so only computer.ts imports it here.
+vi.mock("../src/desktop.ts", async (importOriginal) => ({
+ ...(await importOriginal<typeof import("../src/desktop.ts")>()),
+ onDesktopEvent: async () => () => {},
+}));
+
+vi.mock("../src/useDot.ts", () => ({
+ useDot: () => ({ state: h.state, send: h.send, dismiss: () => {}, dispatch: () => {} }),
+}));
+
+const computerEntry = (overrides: Partial<ConnectorInfo> = {}): ConnectorInfo => ({
+ id: "computer",
+ name: "Computer",
+ reads: "See your screen during a session you allow in the app.",
+ sends: "Click, type, and open apps on this Mac during that session.",
+ added: true,
+ enabled: true,
+ mode: "read_write",
+ status: "connected",
+ noSignIn: true,
+ builtin: true,
+ ...overrides,
+});
+
+const notion: ConnectorInfo = {
+ id: "notion",
+ name: "Notion",
+ reads: "Search and read your pages.",
+ sends: "Create and edit pages, after you approve each one.",
+ added: true,
+ enabled: true,
+ mode: "read_only",
+ status: "connected",
+};
+
+const DEBUG_NOTE =
+ "Debug builds lose these permissions on every rebuild because macOS ties them to the app's signature; grant them again after updating.";
+
+function connectorsOpen(list: ConnectorInfo[]): DotState {
+ return { ...h.state, connectors: { open: true, list } };
+}
+
+async function renderApp(search: string) {
+ window.history.replaceState(null, "", `/${search}`);
+ const { App } = await import("../src/App.tsx");
+ return render(<App />);
+}
+
+const registrations = () => h.send.mock.calls.filter(([m]) => m.type === "computer_register");
+const invoked = (command: string) => h.invoke.mock.calls.filter(([c]) => c === command);
+
+async function emitState(payload: unknown) {
+ await waitFor(() => expect(h.listeners.has("computer://state")).toBe(true));
+ await act(async () => h.listeners.get("computer://state")?.({ payload }));
+}
+
+beforeEach(() => {
+ h.tauri = true;
+ h.endpoint = { url: HELPER, token: KEY };
+ h.permissions = { accessibility: false, screenRecording: true };
+ h.status = { active: false };
+ h.listeners.clear();
+ h.send = vi.fn();
+ h.state = { ...initialState, connection: "open", agentState: "idle" };
+ h.invoke = vi.fn(async (command: string) => {
+ switch (command) {
+ case "connection_info":
+ return new Promise(() => {});
+ case "computer_endpoint":
+ return h.endpoint;
+ case "computer_permissions":
+ return h.permissions;
+ case "computer_status":
+ return h.status;
+ default:
+ return null;
+ }
+ });
+});
+
+afterEach(() => {
+ vi.useRealTimers();
+ window.history.replaceState(null, "", "/");
+ window.sessionStorage.clear();
+});
+
+describe("registering the helper (S24.7)", () => {
+ it("the desktop panel asks the app for the endpoint once connected and registers it with the daemon", async () => {
+ h.state = { ...h.state, connection: "connecting" };
+ const view = await renderApp("?surface=panel");
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(invoked("computer_endpoint")).toHaveLength(0);
+
+ h.state = { ...h.state, connection: "open" };
+ const { App } = await import("../src/App.tsx");
+ view.rerender(<App />);
+ await waitFor(() => expect(registrations()).toHaveLength(1));
+ expect(registrations()[0]?.[0]).toEqual({ type: "computer_register", url: HELPER, token: KEY });
+ expect(invoked("computer_endpoint")).toHaveLength(1);
+ });
+
+ it("registers nothing when the app has no helper", async () => {
+ h.endpoint = null;
+ h.state = connectorsOpen([notion]);
+ await renderApp("?surface=panel");
+ await waitFor(() => expect(invoked("computer_endpoint")).toHaveLength(1));
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(registrations()).toHaveLength(0);
+ expect(screen.queryByRole("listitem", { name: "Computer" })).toBeNull();
+ });
+
+ it("the Dot window does not register it; only the panel does", async () => {
+ await renderApp("?surface=dot");
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(invoked("computer_endpoint")).toHaveLength(0);
+ expect(registrations()).toHaveLength(0);
+ });
+
+ it("the web page never offers computer control, even when the daemon lists it", async () => {
+ h.tauri = false;
+ h.state = connectorsOpen([notion, computerEntry()]);
+ await renderApp("?#token=web-key");
+ await screen.findByRole("region", { name: "Connectors" });
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(screen.queryByText("Computer")).toBeNull();
+ expect(screen.getByText("Notion")).toBeInTheDocument();
+ expect(h.invoke).not.toHaveBeenCalled();
+ expect(registrations()).toHaveLength(0);
+ });
+});
+
+describe("the Computer entry in Connectors (S24.5)", () => {
+ it("shows both permissions with grant buttons, a re-check, the debug-build note, and nothing to remove", async () => {
+ h.state = connectorsOpen([notion, computerEntry()]);
+ await renderApp("?surface=panel");
+ const entry = await screen.findByRole("listitem", { name: "Computer" });
+ const accessibility = within(entry).getByRole("listitem", { name: "Accessibility" });
+ const recording = within(entry).getByRole("listitem", { name: "Screen Recording" });
+ await waitFor(() => expect(accessibility).toHaveTextContent("Not allowed"));
+ expect(recording).toHaveTextContent("Allowed");
+ expect(within(recording).queryByRole("button")).toBeNull();
+ expect(within(entry).getByText(DEBUG_NOTE)).toBeInTheDocument();
+ expect(within(entry).queryByText(/does not accept images/)).toBeNull();
+ for (const name of [/^Connect/, /^Disconnect/, /^Remove/, /^Sign in/]) {
+ expect(within(entry).queryByRole("button", { name })).toBeNull();
+ }
+ expect(within(entry).queryByRole("radio")).toBeNull();
+ // The other connectors keep their usual controls.
+ expect(screen.getByRole("button", { name: "Remove Notion" })).toBeInTheDocument();
+
+ const checks = invoked("computer_permissions").length;
+ await userEvent.click(within(accessibility).getByRole("button", { name: "Grant Accessibility" }));
+ expect(h.invoke).toHaveBeenCalledWith("computer_request_permission", { kind: "accessibility" });
+ await waitFor(() => expect(invoked("computer_permissions").length).toBe(checks + 1));
+
+ h.permissions = { accessibility: true, screenRecording: true };
+ await userEvent.click(within(entry).getByRole("button", { name: "Check again" }));
+ await waitFor(() => expect(accessibility).toHaveTextContent("Allowed"));
+ expect(accessibility).not.toHaveTextContent("Not allowed");
+ expect(within(accessibility).queryByRole("button")).toBeNull();
+ });
+
+ it("asks for Screen Recording with its own kind", async () => {
+ h.permissions = { accessibility: true, screenRecording: false };
+ h.state = connectorsOpen([computerEntry()]);
+ await renderApp("?surface=panel");
+ const recording = await screen.findByRole("listitem", { name: "Screen Recording" });
+ await userEvent.click(await within(recording).findByRole("button", { name: "Grant Screen Recording" }));
+ expect(h.invoke).toHaveBeenCalledWith("computer_request_permission", { kind: "screenRecording" });
+ });
+
+ it("says when the current model does not accept images", async () => {
+ h.state = connectorsOpen([computerEntry({ noImages: true })]);
+ await renderApp("?surface=panel");
+ const entry = await screen.findByRole("listitem", { name: "Computer" });
+ expect(within(entry).getByText(/current model does not accept images/)).toBeInTheDocument();
+ });
+});
+
+describe("while the assistant controls the Mac (S24.6)", () => {
+ it("shows a banner with the time left and a Stop button, until the session ends", async () => {
+ await renderApp("?surface=panel");
+ await waitFor(() => expect(h.listeners.has("computer://state")).toBe(true));
+ expect(screen.queryByRole("region", { name: "Computer control" })).toBeNull();
+
+ vi.useFakeTimers({ toFake: ["Date", "setInterval", "clearInterval"] });
+ vi.setSystemTime(1_800_000_000_000);
+ await emitState({ active: true, endsAt: Date.now() + 90_000, reason: "granted" });
+ const banner = screen.getByRole("region", { name: "Computer control" });
+ expect(banner).toHaveTextContent("Controlling your Mac · 01:30 left");
+ await act(async () => vi.advanceTimersByTime(1000));
+ expect(banner).toHaveTextContent("Controlling your Mac · 01:29 left");
+
+ within(banner).getByRole("button", { name: "Stop" }).click();
+ expect(invoked("computer_stop")).toHaveLength(1);
+ await emitState({ active: false, reason: "stopped" });
+ expect(screen.queryByRole("region", { name: "Computer control" })).toBeNull();
+ });
+
+ it("shows a session that was already running when the panel opened, with endsAt in seconds too", async () => {
+ vi.useFakeTimers({ toFake: ["Date", "setInterval", "clearInterval"] });
+ vi.setSystemTime(1_800_000_000_000);
+ h.status = { active: true, endsAt: 1_800_000_000 + 25 * 60 };
+ await renderApp("?surface=panel");
+ const banner = await screen.findByRole("region", { name: "Computer control" });
+ expect(banner).toHaveTextContent("Controlling your Mac · 25:00 left");
+ });
+
+ it("the rose shows the in-control state and goes back when the session ends", async () => {
+ await renderApp("?surface=dot");
+ const dot = screen.getByRole("button");
+ expect(dot.querySelector(".rose")).toHaveClass("rose-idle");
+
+ await emitState({ active: true, endsAt: Date.now() + 60_000, reason: "granted" });
+ expect(dot).toHaveAccessibleName("Gentle Dot: controlling your Mac");
+ expect(dot).toHaveClass("dot-in_control");
+ expect(dot.querySelector(".rose")).toHaveClass("rose-control");
+
+ await emitState({ active: false, reason: "panic" });
+ expect(dot).toHaveAccessibleName("Gentle Dot: ready");
+ expect(dot).not.toHaveClass("dot-in_control");
+ expect(dot.querySelector(".rose")).toHaveClass("rose-idle");
+ });
+
+ it("styles the in-control rose and the banner with the site's tokens", () => {
+ const stylesheet = "../src/styles.css";
+ const css = readFileSync(fileURLToPath(new URL(stylesheet, import.meta.url)), "utf8");
+ const rule = (selector: string) =>
+ css.match(new RegExp(`\\n${selector.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} \\{([^}]*)\\}`))?.[1] ?? "";
+ expect(rule(".dot-in_control .dot-disc")).toMatch(/--rim: /);
+ expect(rule(".rose-control .rose-light path")).toMatch(/animation: rose-run/);
+ expect(rule(".computer-banner")).toMatch(/var\(--accent/);
+ expect(rule(".computer-banner")).toMatch(/border-radius: var\(--radius-md\)/);
+ });
+});