Skip to content

A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.

TypeScript

docs(odd): keep only the last screenshot in the model context (S24.10, L97)

Alan Buscagliaa339bd058df9
diff --git a/odd/tasks/gentle-dot.md b/odd/tasks/gentle-dot.md
index 7d271dd..b705f2f 100644
--- a/odd/tasks/gentle-dot.md
+++ b/odd/tasks/gentle-dot.md
@@ -85,7 +85,7 @@ Branch: `feat/gentle-dot` · Delivery strategy: `single-pr` (user choice, L7; fo
- **S24.6 Visible state.** While a session is active, the rose shows an "in control" state, and the panel shows a banner with the remaining time and a Stop button.
- **S24.8 Speed (L66, user: "anda muy bien pero lento, bastante lento"; "hagamos todas si").** (a) A `batch {steps: [{tool, ...args}], intent?}` tool runs up to 20 steps of the action tools (click, move, drag, scroll, type, key, open_app, wait) in one call and returns one screenshot at the end; every step goes through the same checks as a single call (session, epoch/panic, rate limit, blocklist, risky confirmation, coordinate mapping against the screenshot taken before the batch); the batch stops at the first refused, declined, or failed step and reports which steps ran. The tool descriptions steer the model to batch predictable sequences. (b) The helper gets faster: dependencies that encode and resize images are optimized in debug builds, and the settle wait after an action drops from 300 ms to about 150 ms (open_app keeps its longer wait). (c) Screenshots get lighter (lower JPEG quality) without losing legibility of UI text. (d) Explain and fix the 18 s `type` seen in L66 if it is the helper's doing.
- **S24.9 Yolo mode (L67, user: "y mete un modo yolo").** User choices: "Solo las confirmaciones (recomendado)" and "Interruptor, se apaga solo (recomendado)". While on, risky actions (S24.4) run without the per-action confirmation; the session grant, the blocklist, the rate limit, the panic shortcut, and Stop stay as they are. A switch in Connectors → Computer and a tray item turn it on and off; turning it on asks a native confirmation owned by the app; it turns itself off when the app quits, after 1 hour, or on the panic shortcut (L74; Stop leaves it on), never persists to disk, and only the app (Tauri command or tray) can change it, never the engine or the WS. The rose and the session banner show when it is on.
- - **S24.10 Prune old screenshots from the model's context (L95).** User: "si lo pruebo desde pi funciona, en gentle dot esta MUY lento". Before each model call, only the most recent computer-control screenshots (assumption: the last 3) are sent as images; older ones are replaced by a short text placeholder in the request only. The saved session keeps every image. Applies to any model.
+ - **S24.10 Prune old screenshots from the model's context (L95).** User: "si lo pruebo desde pi funciona, en gentle dot esta MUY lento". Before each model call, only the most recent computer-control screenshot is sent as an image (L97, user: "no seria mejor solo enviar la ultima y de las anteriores mandar un resumen?"); each older one is replaced, in the request only, by a one-line text summary built from data already in the history (the tool's front app, action, and result), e.g. "[earlier screenshot omitted: Safari, after click at (412, 230)]", with no extra model calls. The saved session keeps every image. Applies to any model.
- **S24.7 Wiring.** The desktop webview asks the app for the endpoint and registers it with the daemon (`computer_register {url, token}`); the daemon renders a built-in `computer` MCP server into the engine's `mcp.json` (direct exposure, no in-engine approval cards, since the helper enforces) and removes it on `computer_unregister` or when that client disconnects. The web UI (VPS) never offers computer control.
- **S25 — Connector enforcement outside the agent's reach (stage 2, before publishing to other users).** Decided in L48. The daemon proxies MCP traffic: connector tokens live in the OS secure store (macOS Keychain, with an equivalent on Linux and the VPS), the engine sees only local proxy URLs, and enabled state, mode, and approvals are enforced by the daemon, not by files or an in-engine hook. Subagent child processes are covered too. Approvals must reach the user through a channel the agent cannot reach (L49: with the daemon access key, the agent could connect to the WebSocket and answer its own approval cards), for example a native OS dialog shown by the desktop app, or a per-window key never stored in a file the agent can read.
@@ -278,4 +278,5 @@ Environment notes: Node v24.14.1 and pnpm are present. `cargo` is on PATH but `r
- **L94 (2026-10-09):** T30 done (`8a79b48`): `POST /upload` (Bearer only, WS Origin allowlist, CORS preflight), streaming limits with cleanup, private modes, one-time uploads, `<attachments>` block in the prompt hidden from history, images only when the model accepts them. Pending parent change after T29c: `.disable_drag_drop_handler()` on the panel's WebviewWindowBuilder (shell.rs) so HTML5 drops reach the panel. Verifier launched on 4e2efc5..8a79b48; T31 writer launched. User: "con deepseek v4 no anda de NaN". daemon.log: signed in to nan (api_key) at 10:57, then three `assistant error: 524` Cloudflare "A timeout occurred" responses (11:02, 11:04, 11:15) with model `nan/deepseek-v4-flash`; a 524 means NaN's origin did not answer within Cloudflare's limit. Not yet known whether NaN was slow for everyone or for this request (large prompt/tools).
- **L95 (2026-10-09, user):** "si lo pruebo desde pi funciona, en gentle dot esta MUY lento" (screenshot: computer control active in yolo mode, several "Something went wrong while answering"). Measured: the current session holds 91 screenshots, 11.1 MB of base64 images, and the engine resends the whole history with every call, which plain Pi sessions do not carry; the 524s are NaN's origin not answering within Cloudflare's limit on those heavy requests. Pi's extension `context` event can replace the messages sent to the model without touching the session (pi-coding-agent extensions/types.d.ts:659-662, 1047-1049). Recorded as S24.10/T32, inline after T31.
- **L96 (2026-10-09):** T29c Rust done: sherpa-onnx 1.13.8 Parakeet backend preferred when installed (Apple Speech otherwise on macOS; parec/arecord capture on Linux), ModelManager with pinned asset (k2-fsa/sherpa-onnx release asr-models, 487,170,055 bytes, SHA-256 5793d0fd…16bf, matches GitHub's digest), atomic install, cancel, remove; commands `voice_model_status|download|cancel|remove`, event `voice://model`. Real model: English and Spanish clips transcribed correctly, load ~0.6 s, decode ~0.3 s; install 24 s; release binary 14.3 → 42.2 MB; ~0.7 GB RAM while loaded (unloads after 5 min idle). Risks: sherpa-onnx's build.rs downloads unverified static libs (pin with SHERPA_ONNX_ARCHIVE_DIR in the packaging scripts); Linux voice not compiled here; a model that fails to load returns empty text silently. Pending UI: download button with progress, a "Transcribing…" state while voice_stop runs.
+- **L97 (2026-10-09, user):** "no seria mejor solo enviar la ultima y de las anteriores mandar un resumen?" after "no te acueras que no podemos tener mas de una conversacion? esto es justo lo que te decia con lo de autocompactar y el costo". Parent's earlier advice to open a new conversation was wrong (one continuous chat, T19); the session (11.8 MB) had not rotated (20 MB or 10 compactions). S24.10 updated: keep only the last screenshot, summarize older ones from existing text, no extra model calls. T32 is first priority after T31.
- **Next step:** T32 after T31; the T29c UI hook; pin sherpa-onnx libs in packaging; drag-drop handler change; T29c, T31, T30 verdict; then the T29c UI hook, the drag-drop change, and a rebuild for a real-Mac round; grant both permissions, take a screenshot (the Dot must be absent), click and type in a harmless app, a risky "Send" confirmation, ⌥⇧Esc panic, a blocked app (System Settings), and `screencapture -x /tmp/t.png` from the agent's bash must fail (R1); the community tester runs docs/linux-testing.md when the user shares it; T23/S25 before any public release.