Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
Visual references in coding work orders
workspace-orchestrate accepts image_refs on create, open_workspace
and send. These are opaque IDs from the current conversation, not paths,
URLs, or descriptions. Select the images used by the particular assignment.
An explicit empty list means the assignment does not use images. When usable
images exist and the field is omitted, the tool requests a selection before
creating a pane or sending a task. No global latest-appshot lookup is used.
Sources and authorization
take_appshotreturns its filtered pixels, an image ID, and handoff guidance. The regular tool loop and Live both deliver pixels as image input rather than serializing them as tool-result text.- Images on the current brain request receive IDs before tool routing.
- A Dodaios chat upload registers the bytes of the images selected on that message. Paths must resolve inside that chat's working folder. Missing, oversized, unsupported or escaped files produce an unavailable notice.
- Appshots attached directly to a live call belong to that call's scope, in every live mode (backend thinking, native, and subscription). The ID rides beside the pixels wherever the call keeps them in model context, including a capture's retained snapshot after its tool output left the history. Chat, live-call and ordinary conversation scopes cannot resolve each other's IDs. A live-call reference lives as long as the call keeps the picture in context: closing the call clears it, and a one-hour bound expires a scope whose call never closed. Other captures retain their configured TTL; other temporary references expire after 120 seconds. One active timer removes expired bytes even when there are no later calls.
The existing ToolExecutor authorization, capture denylist/redaction, coding pane permissions and prompt readiness checks remain in force. Pixels are untrusted reference data. The assistant must not derive permission from text inside them. A requested image-based coding handoff authorizes copying the selected images to that coding workspace; taking a screenshot alone does not.
Delivery and retention
Before typing the task, the coding gateway validates the image and copies the
actual bytes to .jarvis/visual-references inside the pane's working folder
(including a pane-specific worktree). It writes an ignore file, refuses
redirected directories, verifies the copied bytes, and includes the readable
absolute path in the CLI prompt using the CLI registry's file-reference syntax.
The prompt tells the agent to open every image and report an inability to view
it. No provider-specific clipboard or undocumented image flag is required.
For a pane on a connected computer, the copy uses that computer's existing authenticated SFTP connection and remote working folder. It verifies the remote bytes before sending the remote path. A failed transfer never falls back to a local path or a description. Moving a pane during transfer/readiness invalidates delivery. Long prompts that would truncate an image path are refused.
These explicit task attachments are separate from transient Screen Context: copies survive a source expiry, process restart and ordinary session resume. They are eligible for deletion after seven days; a later handoff sweeps expired copies. This is not a promise of timed deletion from disk. Users can delete the copies earlier, in which case the coding agent must report a missing file. There is no extra local disk copy for a remote handoff. File permissions restrict copies on systems that enforce POSIX permissions; Windows retains the workspace's ACL boundary.
Receipts include the selected ID, destination path/computer, byte count, SHA-256,
and copy retention period. accepted proves CLI prompt submission and verified
file availability at handoff, not that a model opened the image or finished the
task. Refused/uncertain submission must never be reported as completed.
Image selection and conversation scope are part of the durable delivery claim;
retries cannot silently replace or duplicate an attachment.
Verification
tests/contract/test_coding_image_handoff.py exercises upload, appshot-tool and
live-call appshot sources with new/existing sessions through the real workspace orchestrator, coding
gateway and registry into a fake PTY. It opens the delivered PNG, compares bytes,
checks the typed file path, receipt and retry behavior, and covers multiple images,
unrelated images, expiry, conversation isolation, invalid images, path traversal,
prompt truncation, pane-specific worktrees and moving destinations. SFTP tests
exercise verified transfers and corrupt-read refusals using an isolated transport
and a real in-process SSH/SFTP server bound to loopback.
These tests do not spend a provider key, capture the user's real screen, connect to a user's remote host, or prove that an actual vendor model used its image tool.
Implementation map
| Area | Files |
|---|---|
| Scoped references and expiry | dodaios/core/image_references.py |
| Upload intake | dodaios/agent_chat/attachments.py, service.py |
| Conversation and tool routing | dodaios/brain/manager.py, dispatcher.py, tool_use_loop.py, workspace_tool.py |
| Captures and live calls | dodaios/plugins/tool/appshot.py, dodaios/live/tools.py, session.py, native.py, subscription.py |
| Work order and CLI delivery | dodaios/agentic_ide/orchestration.py, control.py, session.py, visual_handoff.py |
| Regression evidence | tests/contract/test_coding_image_handoff.py, tests/fakes/fake_visual_sftp.py, tests/unit/appshot/test_appshot_surfaces.py, tests/unit/live/test_subscription_session.py |