Skip to content

A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.

TypeScript

docs(design): number Windows computer control S24.11 and describe per-screenshot exclusion

Marcelitofd96a5159804
diff --git a/docs/design.md b/docs/design.md
index 375bb68..f32cbf3 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -167,7 +167,7 @@ The UI shows one endless chat: no conversations list, no new conversation, and t
### Computer control (macOS and Windows, S24)
-The assistant can see the screen and operate apps through a built-in MCP server that the desktop app serves. macOS (L59) and Windows (S24.10); Linux, the web page and a VPS never offer it.
+The assistant can see the screen and operate apps through a built-in MCP server that the desktop app serves. macOS (L59) and Windows (S24.11); Linux, the web page and a VPS never offer it.
- S24.1 Helper: native Rust inside the Tauri app (`apps/desktop/src-tauri`), never in the daemon or the engine. It serves MCP (JSON-RPC 2.0 over streamable HTTP) on `http://127.0.0.1:<random port>/mcp` and requires `Authorization: Bearer <key>`, a fresh random key per app launch.
- S24.2 Tools: `screenshot`, `click`, `move`, `drag`, `scroll`, `type`, `key`, `open_app`, `list_apps`, `wait`, each with an optional `intent`. Screenshots of the display under the pointer are downscaled to a long edge of 1280 px; coordinates are pixels of the latest screenshot; actions return a fresh one; Gentle Dot's own windows are left out.
@@ -185,9 +185,9 @@ The assistant can see the screen and operate apps through a built-in MCP server
- Working edge: a separate non-fading layer lights a thin neon line along the top edge of the controlled display, with a light traveling along it like the light on the rose's strokes, during a prepared tool call, not for the entire 30-minute authorization. No text or logo. With "Reduce motion" the edge stays lit and the traveling light is left out. The overlay covers the controlled display before a call starts, even before its first action mark. An internal `computer://working {active, operationId}` event targets only the glow; a matching rendered-readiness acknowledgement (`computer_working_ready`, granted only to the glow window) is required before the interactive windows are temporarily hidden. Stale acknowledgements or queued preparation cannot hide windows after cancellation. The working edge clears when the call or session ends.
- Guards and screenshots: the overlay is a Gentle Dot window covering the display, so pointer-owner resolution skips it by its window number (`kCGWindowNumber`) and nothing else. The panel and the Dot still resolve to Gentle Dot and are refused. When the system-wide Accessibility hit test lands on the overlay while another app owns the window under the point, the risk check hit-tests that app's own element instead (`AXUIElementCreateApplication`). Screenshots leave the overlay out with every other window of our process (ScreenCaptureKit `excludingWindows`). The overlay is on screen for the whole session, so it is in the capture's window list before it draws anything.
-- S24.10 Windows: the helper uses the same session and guards, with Windows-specific input, screenshots, app discovery and UI Automation.
+- S24.11 Windows: the helper uses the same session and guards, with Windows-specific input, screenshots, app discovery and UI Automation.
- The panic shortcut is Alt+Shift+Esc; the panel says "Controlling your PC", and Connectors shows no macOS permission rows.
- - Screenshots require Windows 10 version 2004 (build 19041) or newer for `WDA_EXCLUDEFROMCAPTURE`. The Dot and panel are excluded during a session and restored after it ends; the glow and dialogs remain excluded.
+ - Screenshots require Windows 10 version 2004 (build 19041) or newer for `WDA_EXCLUDEFROMCAPTURE`. Each screenshot leaves out every visible Gentle Dot window and, right after it is taken, lets the ones it excluded show again, so the user's own screenshots still show the Dot and the panel; windows that are always left out (the glow, the approval dialogs) stay out. A dialog that cannot be left out holds screenshots off while it is open, and a build that cannot be read counts as too old.
- Keyboard input to an elevated app is refused: "{app} runs as administrator, so Gentle Dot cannot send it keys."
- `open_app` resolves names through Start menu shortcuts and App Paths, not PATH; it refuses unsafe targets and checks the resolved executable against the blocklist.
- UI Automation failures are fail-closed: if any element in an ancestor chain is unreadable, the entire chain is unreadable and asks for confirmation. A default-button scan cut short is also unreadable.
diff --git a/odd/tasks/gentle-dot.md b/odd/tasks/gentle-dot.md
index 90fb5aa..b0d57b3 100644
--- a/odd/tasks/gentle-dot.md
+++ b/odd/tasks/gentle-dot.md
@@ -86,6 +86,7 @@ Branch: `feat/gentle-dot` · Delivery strategy: `single-pr` (user choice, L7; fo
- **S24.8 Speed (L66, user: "anda muy bien pero lento, bastante lento"; "hagamos todas si").** (a) A `batch {steps: [{tool, ...args}], intent?}` tool runs up to 20 steps of the action tools (click, move, drag, scroll, type, key, open_app, wait) in one call and returns one screenshot at the end; every step goes through the same checks as a single call (session, epoch/panic, rate limit, blocklist, risky confirmation, coordinate mapping against the screenshot taken before the batch); the batch stops at the first refused, declined, or failed step and reports which steps ran. The tool descriptions steer the model to batch predictable sequences. (b) The helper gets faster: dependencies that encode and resize images are optimized in debug builds, and the settle wait after an action drops from 300 ms to about 150 ms (open_app keeps its longer wait). (c) Screenshots get lighter (lower JPEG quality) without losing legibility of UI text. (d) Explain and fix the 18 s `type` seen in L66 if it is the helper's doing.
- **S24.9 Yolo mode (L67, user: "y mete un modo yolo").** User choices: "Solo las confirmaciones (recomendado)" and "Interruptor, se apaga solo (recomendado)". While on, risky actions (S24.4) run without the per-action confirmation; the session grant, the blocklist, the rate limit, the panic shortcut, and Stop stay as they are. A switch in Connectors → Computer and a tray item turn it on and off; turning it on asks a native confirmation owned by the app; it turns itself off when the app quits, after 1 hour, or on the panic shortcut (L74; Stop leaves it on), never persists to disk, and only the app (Tauri command or tray) can change it, never the engine or the WS. The rose and the session banner show when it is on.
- **S24.10 Prune old screenshots from the model's context (L95).** User: "si lo pruebo desde pi funciona, en gentle dot esta MUY lento". Before each model call, only the most recent computer-control screenshot is sent as an image (L97, user: "no seria mejor solo enviar la ultima y de las anteriores mandar un resumen?"); each older one is replaced, in the request only, by a one-line text summary built from data already in the history (the tool's front app, action, and result), e.g. "[earlier screenshot omitted: Safari, after click at (412, 230)]", with no extra model calls. The saved session keeps every image. Applies to any model.
+ - **S24.11 Computer control on Windows (L131).** The same helper, session and guards as on macOS, with Windows input, screenshots, app discovery and UI Automation; the details are in `docs/design.md` S24.11.
- **S24.7 Wiring.** The desktop webview asks the app for the endpoint and registers it with the daemon (`computer_register {url, token}`); the daemon renders a built-in `computer` MCP server into the engine's `mcp.json` (direct exposure, no in-engine approval cards, since the helper enforces) and removes it on `computer_unregister` or when that client disconnects. The web UI (VPS) never offers computer control.
- **S25 — Connector enforcement outside the agent's reach (stage 2, before publishing to other users).** Decided in L48. The daemon proxies MCP traffic: connector tokens live in the OS secure store (macOS Keychain, with an equivalent on Linux and the VPS), the engine sees only local proxy URLs, and enabled state, mode, and approvals are enforced by the daemon, not by files or an in-engine hook. Subagent child processes are covered too. Approvals must reach the user through a channel the agent cannot reach (L49: with the daemon access key, the agent could connect to the WebSocket and answer its own approval cards), for example a native OS dialog shown by the desktop app, or a per-window key never stored in a file the agent can read.
@@ -324,6 +325,7 @@ Environment notes: Node v24.14.1 and pnpm are present. `cargo` is on PATH but `r
- **L102 (2026-10-09, user):** "bien pero, mientras estoy hablando no me aparece lo que se va enviar como cuando uso el bui† in de mac" — Parakeet transcribes correctly with our capture, so the earlier garbage came from Apple Speech with our converted 16 kHz audio (open: feed Apple the device's native buffers). Live text with the local model (`e027b19`): the recording so far (last 30 s at most) is decoded again every 0.8 s of new audio, skipping when the model is still loading or a decode runs (never blocking the audio thread), serialized with the final decode, and never shown after stop/cancel; RED (compile) → GREEN, cargo test 296. The Voice chip now names the engine ("Offline", "macOS", "Off"; `1396457`). Rebuilt, signed, restarted (also applies the model-chip fix `7ab0c0a`).
- **L103 (2026-10-09, user):** "estaria bueno que si pongo el boton de grabar de nuevo en vez de enviarlo, que se quede escrito en el input para confirmarlo, como se comporta ahora estaria bueno que sea una opcion que uno elijiese". S30.4 updated: review in the input is the default; "Send when I stop" (switch in the recording strip, stored in localStorage) keeps send-on-stop; the mic button reads "Stop and review" or "Stop and send". `e0010eb`: RED 3 failing → GREEN, pnpm test 593 (the earlier tests now cover the option). Rebuilt, signed, restarted.
- **L104 (2026-10-09, user):** "sube todo esto al repo y fijate el readme de gentle ai como es para copiar el estilo y hacer el propio para Gentle Dot"; choices "Gentleman-Programming/gentle-dot, público" and "main + feat/gentle-dot, y abro el PR (recomendado)"; then "usa la rosa con glow como banner". TruffleHog over the whole history: no verified secrets (unverified hits are masking-test fixtures). README rewritten in Gentle-AI's layout with an honest Security status; banner and divider rendered from the Dot's own rose strokes and glow (`docs/brand/tools/build-banner.mjs`); MIT LICENSE as in Gentle-AI. Created the public repo github.com/Gentleman-Programming/gentle-dot, `main` at the first commit `5ec596c`, pushed `feat/gentle-dot` (153 commits, 286 files, +57,416), and opened PR #1 (single-pr). Merge is the user's decision.
+- **L131 (2026-11-02):** Review of #170 (egdev6): S24.10 is already "Prune old screenshots" (T32), and `docs/design.md` reused it for Windows; the Windows computer-control spec is S24.11. The same review asked the screenshot sentence to match #169: exclusion per screenshot, restored right after, not for the whole session.
- **L130 (2026-10-31):** Review of #159 (egdev6) asked to record the keyboard guards in S24.4: the screenshot binding (#89), the recipient recheck per chunk, Return and key, and the 500 ms restore after a confirmation, all observable on macOS and Linux. It also found that a stop in the middle of `type` said "nothing was sent" next to "Typing stopped after 20 of 45 characters"; the guard now says "the rest was not sent" once characters landed. S24.4 rewritten; docs/design.md S24.4 updated.
- **L129 (2026-10-10):** T23h committed (`35b942b`): server mode `GENTLE_DOT_VPS=1` (image only; refuses without root); daemon root with SETUID, SETGID, CHOWN, FOWNER, DAC_OVERRIDE, KILL and no-new-privileges; engine `dot` (uid 1001), stdio connector servers `dotmcp` (uid 999) via setpriv with `<data>/connector-home` (they cannot see the engine's workspace; documented); data dir root 0711, token/state/connectors/secrets/PIN root 0600; secrets AES-256-GCM per record (id as AAD) under `GENTLE_DOT_SECRETS_KEY`, scrubbed from every child, old file secrets migrated once; web PIN `pin_set`/`pin_command`/`pin_approve` (6-12 digits, scrypt + salt, constant-time, 5 failures -> 15 min persisted lockout; declining needs no PIN); desktop with the app channel unchanged. RED across protocol (4), UI (5), env scrub, spawn uid, new modules, vps-gateway 10/10, migration, CLI -> GREEN; pnpm test 778. Writer deviation: one `git stash` to observe a RED (checked: stash empty, HEAD intact; told not to repeat). The first independent verifier crashed twice (runtime errors) after a lead the parent confirmed by code: **B1** the root daemon ran Pi's ModelRuntime on dot-owned `auth.json`/`models.json` (`resolveConfigValue` runs `!cmd` values; voice refresh calls it on every provider list) and wrote engine files dot could replace with links. Correction `162473c` (class fix): ModelRuntime in `cli.ts --auth-helper` spawned as dot (no capabilities, scrubbed env, Node IPC, plain data back); every other daemon access to engine folders (profiles/settings, mcp.json + shim, `.pi`, `.engram`, history/rotation, uploads + prompt images, legacy mcp-auth.json, isolation) inside a synchronous `engineAccess` bracket (setgroups/setegid/seteuid to dot and back, exit 70 if it cannot restore); root files through `writePrivateFile` (O_EXCL temp, owner check, refused inside a bracket); per-launch hand-over removed; static guard against async fs (seteuid applies to every thread). RED: 2 non-root + 5 root tests in the container against a mktemp copy of `35b942b`; GREEN: pnpm test 790 (13 root-only skipped on macOS), root tests 18/18 in the image, 3 mutations caught. Fresh full verifier (7ff8021..162473c): **no blockers**; S25.8 PASS, desktop S25.1/2/5/6 unchanged; as dot every daemon file EACCES, environ denied, links and hardlinks refused, planted `!cmd` ran only as uid 1001, PIN lockout persisted, PIN never in logs. Advisories fixed in `4d1035e`: the user switch moved inside its try (root tests 18/18 in the image), entrypoint `.engram` hand-over only of root-owned files (`--from=0:0`), capability reasons in compose.yaml/deploy-vps.md corrected, stale token-folder comment. Advisories kept: connector `!cmd` values (user- or root-import-supplied only) resolve as root; root `existsSync` on engine-reported paths (boolean only); upgrade-only chmod in `prepareVpsLayout` follows a link left by an older image; `docker exec` shells inherit the key (documented); dot can ptrace/kill the helper (spoofed data or DoS, no root execution); a future library doing async fs as root is not caught by the guard. Not run: startup hand-over link probe (the safety policy blocks recursive chown commands), a daemon restart for lockout, a live connector `pin_approve`. S25 stage 2 complete (T23a-T23h). **Next step:** PR for T23f-T23h (about 4,700 changed lines; delivery decision pending), then the user's real-Mac round.
- **L128 (2026-10-10):** T23g independent verifier: S25.1 (Linux channel) and S25.7 store PASS, Docker Linux check rerun green (379 tests, clippy, GNOME Keyring cases); two blockers. B1: the new docs said a keyring installed later needs no restart and that the assistant names it, but a keyring missing (or a prompt dismissed) at start leaves `unchecked` for the whole run (T23e fail-closed, by design), so the assistant gets INTEGRITY_UNCHECKED until a restart. B2: the daemon's mapping matched every `the secret store is unavailable: ` answer, changing macOS's locked-keychain wording and dropping the proxy's log line. Correction `920455b`: only the Linux answer (`/^the secret store is unavailable: .*Secret Service/`) is passed on; macOS keeps its wording (regression test); the proxy logs `connector <id> cannot get its secrets: …`; install.md, linux-testing.md, and design.md say to unlock/install the keyring and restart Gentle Dot. RED 2 → GREEN 43/43. Recheck: B1 and B2 closed (probes against a `git archive` copy). Advisories kept: getrlimit not on the POSIX async-signal-safe list (safe in glibc/musl); fcntl fallback never exercised; the 120 s bound is per prompt and the crate never dismisses a timed-out prompt; no-prompter maps to Denied; provider error text reaches the model unbounded (same-user only). T23h design accepted (writer's notice): `GENTLE_DOT_VPS=1` only in the container; daemon root, engine `dot`, stdio connector servers `dotmcp` via `setpriv`; `/app` root-owned read-only; root 0600 token, connectors.json, secrets, PIN; AES-256-GCM per record with `GENTLE_DOT_SECRETS_KEY` scrubbed from env; PIN over the WebSocket, scrypt + salt, 5 failures → 15 min persisted lockout; desktop unchanged. Parent added: `no-new-privileges`, `cap_drop: ALL`, only justified `cap_add`, and a smoke check that the engine and `dotmcp` have no effective or ambient capabilities.
diff --git a/packages/ui/test/computer.test.tsx b/packages/ui/test/computer.test.tsx
index 12a1c30..d613d0a 100644
--- a/packages/ui/test/computer.test.tsx
+++ b/packages/ui/test/computer.test.tsx
@@ -8,6 +8,8 @@ import { type DotState, initialState } from "../src/store.ts";
const HELPER = "http://127.0.0.1:51234/mcp";
const KEY = "s3cr3t-Computer-Key-0123456789abcdef";
+const WINDOWS_2004_NOTE =
+ "Computer control needs Windows 10 version 2004 (build 19041) or newer for screenshots.";
const h = vi.hoisted(() => ({
tauri: true,
@@ -294,7 +296,15 @@ describe("on Windows", () => {
h.permissions.screenshotsSupported = false;
h.state = connectorsOpen([computerEntry()]);
await renderApp("?surface=panel");
- expect(await screen.findByText(/Windows 10 version 2004/)).toBeInTheDocument();
+ expect(await screen.findByText(WINDOWS_2004_NOTE)).toBeInTheDocument();
+ });
+
+ it("says nothing about Windows 10 version 2004 when screenshots are supported", async () => {
+ h.state = connectorsOpen([computerEntry()]);
+ await renderApp("?surface=panel");
+ const entry = await screen.findByRole("listitem", { name: "Computer" });
+ await within(entry).findByText("Windows asks for no permission for computer control.");
+ expect(within(entry).queryByText(WINDOWS_2004_NOTE)).toBeNull();
});
it("names the PC in the session banner", async () => {
@@ -304,6 +314,34 @@ describe("on Windows", () => {
});
});
+describe("on macOS", () => {
+ beforeEach(() => {
+ vi.spyOn(navigator, "userAgent", "get").mockReturnValue(
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko)",
+ );
+ });
+ afterEach(() => vi.restoreAllMocks());
+
+ it("shows none of the Windows texts, even when screenshots are reported unsupported", async () => {
+ h.permissions.screenshotsSupported = false;
+ h.state = connectorsOpen([computerEntry()]);
+ await renderApp("?surface=panel");
+ const entry = await screen.findByRole("listitem", { name: "Computer" });
+ await within(entry).findByText("Accessibility");
+ expect(within(entry).queryByText("Windows asks for no permission for computer control.")).toBeNull();
+ expect(within(entry).queryByText(WINDOWS_2004_NOTE)).toBeNull();
+ expect(within(entry).queryByText(/Alt\+Shift\+Esc/)).toBeNull();
+ expect(within(entry).getByText(/Stop and ⌥⇧Esc still work/)).toBeInTheDocument();
+ });
+
+ it("names the Mac in the session banner", async () => {
+ h.status = { active: true };
+ await renderApp("?surface=panel");
+ expect(await screen.findByText("Controlling your Mac")).toBeInTheDocument();
+ expect(screen.queryByText("Controlling your PC")).toBeNull();
+ });
+});
+
describe("yolo mode (S24.9)", () => {
const yoloSwitch = async () => {
const entry = await screen.findByRole("listitem", { name: "Computer" });