Skip to content

A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.

TypeScript

docs(odd): specify macOS computer control (S24.1-S24.7)

Alan Buscaglia85bd25dfb9c9
diff --git a/odd/tasks/gentle-dot.md b/odd/tasks/gentle-dot.md
index 3850a7e..465c789 100644
--- a/odd/tasks/gentle-dot.md
+++ b/odd/tasks/gentle-dot.md
@@ -76,7 +76,14 @@ Branch: `feat/gentle-dot` · Delivery strategy: `single-pr` (user choice, L7; fo
- Mechanism (to verify in the implementation): `ENGRAM_DATA_DIR=<real ~/.engram>` in the child env so the plugin's instance-id check matches the global server; the project forced through the plugin's supported override (Engram has `ENGRAM_PROJECT` / `process_override`; check that gentle-engram honors it), or a workspace whose detected project is `gentle-dot` as the fallback.
- **S24 — Computer control (last item).** User (L43): "un item mas al final de todo, que haga como codex y que permitas controlar tu computadora, que pueda manejar tus aplicaciones y tal".
- - Acceptance (draft, to be refined before implementation): the assistant can see the screen and operate apps (screenshots, mouse, keyboard, opening and switching apps) after the user grants macOS Accessibility and Screen Recording permissions, with an in-app guide for granting them. Safety: a visible "in control" state on the rose, a global panic shortcut that stops control instantly, approval per control session plus mandatory confirmation for risky actions (send, pay, delete, submit), an optional allowlist of apps, and a clear note when the selected model cannot use images. Likely delivered as a built-in MCP connector behind the S18 approval guard. Scope (L59): macOS only for now (user: "lo de controlar la computadora por ahora solo para mac"); Linux is out of scope.
+ - Acceptance (original draft, refined below in L60): the assistant can see the screen and operate apps (screenshots, mouse, keyboard, opening and switching apps) after the user grants macOS Accessibility and Screen Recording permissions, with an in-app guide for granting them. Safety: a visible "in control" state on the rose, a global panic shortcut that stops control instantly, approval per control session plus mandatory confirmation for risky actions (send, pay, delete, submit), an optional allowlist of apps, and a clear note when the selected model cannot use images. Likely delivered as a built-in MCP connector behind the S18 approval guard. Scope (L59): macOS only for now (user: "lo de controlar la computadora por ahora solo para mac"); Linux is out of scope.
+ - **S24.1 Helper location.** A native helper inside the Tauri app (Rust, `apps/desktop/src-tauri`), never in the daemon or the engine, so macOS attributes Accessibility and Screen Recording to Gentle Dot.app and every rule below lives outside the agent's reach. It serves MCP (JSON-RPC 2.0 over streamable HTTP, `initialize`, `tools/list`, `tools/call`) on `127.0.0.1:<random port>/mcp`, requiring `Authorization: Bearer <token>` with a fresh random token per app launch. Off macOS the helper does not start.
+ - **S24.2 Tools.** `screenshot {}`, `click {x, y, button?: "left"|"right", count?: 1|2|3}`, `move {x, y}`, `drag {fromX, fromY, toX, toY}`, `scroll {x, y, dx, dy}`, `type {text}`, `key {combo}` (for example `cmd+shift+t`, `return`), `open_app {name}` (name or bundle id), `list_apps {}`, `wait {ms ≤ 5000}`. Every action accepts an optional `intent` string. Coordinates are pixels of the latest screenshot; the helper maps them to logical points with the scale and origin of that screenshot. Screenshots of the display under the pointer are downscaled to a long edge of at most 1280 px (Anthropic: "pre downscale your screenshots before sending them to the API"; Pi resizes above 2000×2000), returned as an MCP `image` block (PNG or JPEG) plus a text block with `{imageWidth, imageHeight, scale, frontApp}`. Actions return a fresh screenshot. Gentle Dot's own windows are excluded from screenshots.
+ - **S24.3 Session grant.** Any tool call (screenshots included) without an active session shows a native dialog owned by the app: "Allow Gentle Dot to see and control this Mac for 30 minutes?" (Allow / Deny). Deny returns a tool error and starts nothing. A session ends on timeout (30 min), the Stop button, the tray item "Stop computer control", or the panic shortcut **⌥⇧Esc**; ending drops queued actions immediately, and the next call asks again.
+ - **S24.4 Guards enforced by the helper.** A fixed blocklist: no action targets Gentle Dot's own windows, System Settings, SecurityAgent, loginwindow, Keychain Access, or known password managers (1Password, Bitwarden, Dashlane, LastPass, Apple Passwords); a target is the frontmost app for keyboard actions and the app owning the window under the point for pointer actions. A rate limit of 10 actions per second. Risky actions need a native per-action confirmation showing the action and target: a click on an Accessibility element whose role is a button or menu item and whose title or description matches send, pay, buy, purchase, delete, remove, submit, confirm, transfer, place order, checkout, or their Spanish equivalents (enviar, pagar, comprar, eliminar, borrar, confirmar, transferir); `return` or `cmd+return` while the focused element is a default button or a send-like field; or any declared `intent` matching those words. The intent only adds checks, never removes them.
+ - **S24.5 Permissions guide.** The app reports Accessibility (`AXIsProcessTrusted`) and Screen Recording (`CGPreflightScreenCaptureAccess`) status, can prompt for them, and deep-links to the System Settings panes. The connectors screen shows a built-in "Computer" entry with both statuses, grant buttons, a note that ad-hoc signed debug builds lose the grant on every rebuild (macOS ties it to the code signature), and a note when the current model does not accept images.
+ - **S24.6 Visible state.** While a session is active, the rose shows an "in control" state, and the panel shows a banner with the remaining time and a Stop button.
+ - **S24.7 Wiring.** The desktop webview asks the app for the endpoint and registers it with the daemon (`computer_register {url, token}`); the daemon renders a built-in `computer` MCP server into the engine's `mcp.json` (direct exposure, no in-engine approval cards, since the helper enforces) and removes it on `computer_unregister` or when that client disconnects. The web UI (VPS) never offers computer control.
- **S25 — Connector enforcement outside the agent's reach (stage 2, before publishing to other users).** Decided in L48. The daemon proxies MCP traffic: connector tokens live in the OS secure store (macOS Keychain, with an equivalent on Linux and the VPS), the engine sees only local proxy URLs, and enabled state, mode, and approvals are enforced by the daemon, not by files or an in-engine hook. Subagent child processes are covered too. Approvals must reach the user through a channel the agent cannot reach (L49: with the daemon access key, the agent could connect to the WebSocket and answer its own approval cards), for example a native OS dialog shown by the desktop app, or a per-window key never stored in a file the agent can read.
@@ -104,7 +111,9 @@ Branch: `feat/gentle-dot` · Delivery strategy: `single-pr` (user choice, L7; fo
| T18 | S18–S20 | explore (delegated) → T18a core + T18b guides/drafts/import, each verified independently | done | `e49bd9d`, `8651bba` (T22), `994abce`, `63c8e9c`; see L46–L55 | Connectors: map the engine's MCP integration and the providers first |
| T19 | S22 (+ design §4/§6/§7 refresh) | delegated writer (context backstop) | done | `225596d`; parent re-ran 260 tests, typecheck, lint + real-engine handoff probe | Single continuous chat: hide conversations, transparent session rotation, recent-only history |
| T20 | S23 | delegated writer, after T17 (the runtime refused it in parallel: test surface overlap with T17's `packages/daemon/test/**`) | done | `823363e`, follow-up `400ec73`; parent re-ran 235 tests, typecheck, e2e 12/12 | Global Engram, project gentle-dot |
-| T21 | S24 (macOS only) | explore (delegated, read-only) → spec → implement | in progress | — | Computer control with permissions guide, panic stop, and approvals |
+| T21 | S24 (macOS only) | explore (delegated, read-only) → spec → implement | in progress | map done (L60) | Computer control with permissions guide, panic stop, and approvals |
+| T21a | S24.1–S24.5 (app side), S24.6 events | delegated writer (parallel with T21b, disjoint surface `apps/desktop/src-tauri/**`) | in progress | — | Rust helper: MCP server, session grant, guards, permissions, panic |
+| T21b | S24.5 UI, S24.6, S24.7 | delegated writer (parallel with T21a, surface `packages/**`, `docs/**`) | in progress | — | Protocol, daemon registration, connectors entry, rose state, banner |
| T22 | S18 (hardening, L47 B1/B2) | delegated writer | done (accepted with documented gaps, L52) | `8651bba`; parent re-ran 328 tests + typecheck | Option A: authoritative in-memory connector state with revert, credential reads blocked, path-independent bash checks, full approval preview, redirect manual, private Engram stop, onboarding overlap fix |
| T23 | S25 | explore → spec → implement (stage 2) | pending | — | Daemon-side MCP proxy with Keychain tokens |
| T9 | S1–S6, S9–S12 | delegated verifier (high risk: process supervision, auth) | done | L24 + L31 | Full verification: unit, integration, Playwright, `@real-agent`, `tauri build` + launch, manual macOS checklist with observed results |
@@ -172,4 +181,5 @@ Environment notes: Node v24.14.1 and pnpm are present. `cargo` is on PATH but `r
- **L57 (2026-10-08, Docker check):** The server image builds at HEAD, and a container smoke run reached `idle`, setup 3/3, no token in the logs (only the key file path), and the workspace `.engram/config.json` with `gentle-dot`. T14 was launched (from source, with setup scripts, the tester checklist, and a Debian container build check).
- **L58 (2026-10-08, T14 evidence + parent fix):** Commit `3c002e3`. A pure `platform.rs` decides the Wayland strategy: `GDK_BACKEND=x11` set before GTK on GNOME Wayland when XWayland exists, and no self-positioning on native Wayland (Hyprland), where window rules are used. Single-instance with `gentle-dot --toggle`; Linux tray glyphs; an opaque panel without vibrancy; `.cargo` overrides limited to macOS. `macos-private-api` must stay cross-platform (tauri-build allowlist). Setup scripts: Debian/Ubuntu via apt and Arch/Omarchy via pacman; Node 24 from the official tarball with a sha256 check in a private dir (no sudo), pnpm via corepack, rustup; sudo only for packages, after a prompt; plus an uninstall script. Hyprland snippets in Lua (Omarchy v4.0.4, cited) and hyprlang (0.53–0.54 and the older forms). docs/linux-testing.md is the tester checklist. Container evidence (debian:bookworm arm64): the real setup script ran end to end and was idempotent, cargo 53/53, a 16 MB release binary; an X11 smoke run under Xvfb showed the Dot at 72×72 on the right edge and `--toggle` showing and hiding the panel, with one instance. It found and fixed two GTK bugs (a non-resizable 72×72 window drawn at 200×200; a never-shown panel reporting 0×0). Arch package names were checked (libappindicator-gtk3 → libayatana-appindicator). Not verified: a real GNOME/Hyprland session, an Arch end-to-end run, x86_64. Writer-reported engine issue, "Pi requires the `pi` executable in PATH" on clean machines, was fixed by the parent test-first (RED → GREEN): the daemon prepends the bundled `node_modules/.bin` (pi 1.0.4) to the engine PATH. Real-engine check with no `pi` on PATH: 0 errors, the 4 companion packages installed, setup 3/3. 376 tests, typecheck, lint.
- **L59 (2026-10-08, user):** "lo de controlar la computadora por ahora solo para mac". S24 is scoped to macOS; the read-only map was launched.
-- **Next step:** the T21 map, then the S24 spec and implementation; the community tester runs docs/linux-testing.md when the user shares it; T23/S25 before any public release.
+- **L60 (2026-10-08):** T21 map returned (no web access in the explorer; key facts re-checked by the parent): Pi forwards MCP image blocks to vision models and resizes above 2000×2000 (pi-mcp README:32, pi-coding-agent docs/models.md:70-89); Anthropic recommends downscaling screenshots (claude.com best-practices article); ad-hoc signed builds lose TCC grants on every rebuild because the grant is keyed to the cdhash (several public fixes, e.g. github.com/ericwbailey/what-cant-i-press 750a2ff); ScreenCaptureKit `SCContentFilter`/`SCScreenshotManager` can exclude windows (developer.apple.com). Decision: option (a), a native Rust helper in the Tauri app serving MCP over loopback, because only there do the grant, panic, blocklist, and risky-action checks live outside the agent (the engine guard and the WS are reachable by the agent, L49). Parent defaults, open to the user's change: 30-minute sessions, panic ⌥⇧Esc, 10 actions per second, long edge 1280 px. S24 refined into S24.1–S24.7; split into T21a (Rust) and T21b (TypeScript), launched in parallel with disjoint surfaces; contract: Tauri commands `computer_endpoint` → `{url, token} | null`, `computer_permissions` → `{accessibility, screenRecording}`, `computer_request_permission {kind: "accessibility"|"screenRecording"}`, `computer_stop`, `computer_status` → `{active, endsAt?}`; event `computer://state` `{active, endsAt?, reason?: "granted"|"stopped"|"panic"|"timeout"}`; client messages `computer_register {url, token}` and `computer_unregister`.
+- **Next step:** T21a and T21b, then a seam check and a real-Mac run; the community tester runs docs/linux-testing.md when the user shares it; T23/S25 before any public release.