Skip to content

A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.

TypeScript

docs(odd): specify pruning old screenshots from the model context (S24.10, T32)

Alan Buscaglia11291f288e2d
diff --git a/odd/tasks/gentle-dot.md b/odd/tasks/gentle-dot.md
index b5b2d1f..ca12531 100644
--- a/odd/tasks/gentle-dot.md
+++ b/odd/tasks/gentle-dot.md
@@ -85,6 +85,7 @@ Branch: `feat/gentle-dot` · Delivery strategy: `single-pr` (user choice, L7; fo
- **S24.6 Visible state.** While a session is active, the rose shows an "in control" state, and the panel shows a banner with the remaining time and a Stop button.
- **S24.8 Speed (L66, user: "anda muy bien pero lento, bastante lento"; "hagamos todas si").** (a) A `batch {steps: [{tool, ...args}], intent?}` tool runs up to 20 steps of the action tools (click, move, drag, scroll, type, key, open_app, wait) in one call and returns one screenshot at the end; every step goes through the same checks as a single call (session, epoch/panic, rate limit, blocklist, risky confirmation, coordinate mapping against the screenshot taken before the batch); the batch stops at the first refused, declined, or failed step and reports which steps ran. The tool descriptions steer the model to batch predictable sequences. (b) The helper gets faster: dependencies that encode and resize images are optimized in debug builds, and the settle wait after an action drops from 300 ms to about 150 ms (open_app keeps its longer wait). (c) Screenshots get lighter (lower JPEG quality) without losing legibility of UI text. (d) Explain and fix the 18 s `type` seen in L66 if it is the helper's doing.
- **S24.9 Yolo mode (L67, user: "y mete un modo yolo").** User choices: "Solo las confirmaciones (recomendado)" and "Interruptor, se apaga solo (recomendado)". While on, risky actions (S24.4) run without the per-action confirmation; the session grant, the blocklist, the rate limit, the panic shortcut, and Stop stay as they are. A switch in Connectors → Computer and a tray item turn it on and off; turning it on asks a native confirmation owned by the app; it turns itself off when the app quits, after 1 hour, or on the panic shortcut (L74; Stop leaves it on), never persists to disk, and only the app (Tauri command or tray) can change it, never the engine or the WS. The rose and the session banner show when it is on.
+ - **S24.10 Prune old screenshots from the model's context (L95).** User: "si lo pruebo desde pi funciona, en gentle dot esta MUY lento". Before each model call, only the most recent computer-control screenshots (assumption: the last 3) are sent as images; older ones are replaced by a short text placeholder in the request only. The saved session keeps every image. Applies to any model.
- **S24.7 Wiring.** The desktop webview asks the app for the endpoint and registers it with the daemon (`computer_register {url, token}`); the daemon renders a built-in `computer` MCP server into the engine's `mcp.json` (direct exposure, no in-engine approval cards, since the helper enforces) and removes it on `computer_unregister` or when that client disconnects. The web UI (VPS) never offers computer control.
- **S25 — Connector enforcement outside the agent's reach (stage 2, before publishing to other users).** Decided in L48. The daemon proxies MCP traffic: connector tokens live in the OS secure store (macOS Keychain, with an equivalent on Linux and the VPS), the engine sees only local proxy URLs, and enabled state, mode, and approvals are enforced by the daemon, not by files or an in-engine hook. Subagent child processes are covered too. Approvals must reach the user through a channel the agent cannot reach (L49: with the daemon access key, the agent could connect to the WebSocket and answer its own approval cards), for example a native OS dialog shown by the desktop app, or a per-window key never stored in a file the agent can read.
@@ -171,7 +172,8 @@ Branch: `feat/gentle-dot` · Delivery strategy: `single-pr` (user choice, L7; fo
| T29b | S30.2–S30.4 | delegated writer, parallel with T29a (surface `packages/**`) | done (real Mac pending) | `fd7c25f`; RED → GREEN; parent re-ran pnpm test 520 | Daemon OpenAI-key transcription/TTS, mic button, spoken replies |
| T29c | S30.5 | delegated writer (Rust: engine, download, commands), parallel with T30; UI hook after T30 | in progress | — | Optional Parakeet local model via sherpa-onnx: download, verify, remove, prefer |
| T30 | S31 | delegated writer, parallel with T29c (surface `packages/**`); independent verifier (high: authenticated upload writing to disk) | done (verify running) | `8a79b48`; RED → GREEN; 2 mutation checks; parent re-ran pnpm test 562 | Attach files to messages: upload, workspace storage, images to the model |
-| T31 | S32 | delegated writer, parallel with T29c | in progress | — | Model picker in the chat |
+| T31 | S32 | delegated writer, parallel with T29c | in progress | — |
+| T32 | S24.10 | inline, after T31 (T31 owns `packages/daemon/src/**` now) | pending | — | Model picker in the chat |
| T22 | S18 (hardening, L47 B1/B2) | delegated writer | done (accepted with documented gaps, L52) | `8651bba`; parent re-ran 328 tests + typecheck | Option A: authoritative in-memory connector state with revert, credential reads blocked, path-independent bash checks, full approval preview, redirect manual, private Engram stop, onboarding overlap fix |
| T23 | S25 | explore → spec → implement (stage 2) | pending | — | Daemon-side MCP proxy with Keychain tokens |
| T9 | S1–S6, S9–S12 | delegated verifier (high risk: process supervision, auth) | done | L24 + L31 | Full verification: unit, integration, Playwright, `@real-agent`, `tauri build` + launch, manual macOS checklist with observed results |
@@ -274,4 +276,5 @@ Environment notes: Node v24.14.1 and pnpm are present. `cargo` is on PATH but `r
- **L92 (2026-10-09):** T29a done (`93ee637`): SFSpeechRecognizer (SpeechAnalyzer is Swift-only), native 16 kHz mono capture behind a `Recognizer` trait, Info.plist usage strings, `voice` capability for the panel only. T29b done (`fd7c25f`): protocol voice messages and capability, daemon VoiceService using only an OpenAI API key (`checkAuth("openai").type === "api_key"`; the ChatGPT sign-in is refused), mic button, source choice, spoken replies, mute. The lint failure T29b reported was T29a's capabilities formatting, fixed. User: "una cosa mas, no veo para conectar NaN, fijate que si lo tenemos en gentle shell, hasta tenemos nuestro propio plugin". Cause: the Accounts screen's ModelRuntime runs without gentle-pi's extensions, and NaN is registered by `extensions/nan-provider.ts`. Fix `627f2ad`: the daemon registers gentle-pi's NaN provider as a native provider, loaded through jiti 2.7.0 (gentle-pi ships TypeScript inside node_modules); RED → GREEN, pnpm test 521; jiti added to the runtime's dependencies. App restarted.
- **L93 (2026-10-09, user):** "una cosa mas, da la posibilidad de cambiar los modelos en el chat mismo". Checked: the engine RPC has `get_available_models`, `set_model`, `set_thinking_level` (rpc-types.d.ts:45-56) and profile apply already switches live (profiles.ts:670-698), so this is mostly UI. Recorded as S32/T31, queued after T30.
- **L94 (2026-10-09):** T30 done (`8a79b48`): `POST /upload` (Bearer only, WS Origin allowlist, CORS preflight), streaming limits with cleanup, private modes, one-time uploads, `<attachments>` block in the prompt hidden from history, images only when the model accepts them. Pending parent change after T29c: `.disable_drag_drop_handler()` on the panel's WebviewWindowBuilder (shell.rs) so HTML5 drops reach the panel. Verifier launched on 4e2efc5..8a79b48; T31 writer launched. User: "con deepseek v4 no anda de NaN". daemon.log: signed in to nan (api_key) at 10:57, then three `assistant error: 524` Cloudflare "A timeout occurred" responses (11:02, 11:04, 11:15) with model `nan/deepseek-v4-flash`; a 524 means NaN's origin did not answer within Cloudflare's limit. Not yet known whether NaN was slow for everyone or for this request (large prompt/tools).
-- **Next step:** NaN diagnosis with the user; T29c, T31, T30 verdict; then the T29c UI hook, the drag-drop change, and a rebuild for a real-Mac round; grant both permissions, take a screenshot (the Dot must be absent), click and type in a harmless app, a risky "Send" confirmation, ⌥⇧Esc panic, a blocked app (System Settings), and `screencapture -x /tmp/t.png` from the agent's bash must fail (R1); the community tester runs docs/linux-testing.md when the user shares it; T23/S25 before any public release.
+- **L95 (2026-10-09, user):** "si lo pruebo desde pi funciona, en gentle dot esta MUY lento" (screenshot: computer control active in yolo mode, several "Something went wrong while answering"). Measured: the current session holds 91 screenshots, 11.1 MB of base64 images, and the engine resends the whole history with every call, which plain Pi sessions do not carry; the 524s are NaN's origin not answering within Cloudflare's limit on those heavy requests. Pi's extension `context` event can replace the messages sent to the model without touching the session (pi-coding-agent extensions/types.d.ts:659-662, 1047-1049). Recorded as S24.10/T32, inline after T31.
+- **Next step:** T32 after T31; T29c, T31, T30 verdict; then the T29c UI hook, the drag-drop change, and a rebuild for a real-Mac round; grant both permissions, take a screenshot (the Dot must be absent), click and type in a harmless app, a risky "Send" confirmation, ⌥⇧Esc panic, a blocked app (System Settings), and `screencapture -x /tmp/t.png` from the agent's bash must fail (R1); the community tester runs docs/linux-testing.md when the user shares it; T23/S25 before any public release.