Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

43.2 KB
Raw Blame

Architecture & product overview

This file holds the reference detail that used to live inline in AGENTS.md: the phase status table, the full layer model, the deep architecture subsections, the optimistic-execution decisions, the cross-platform desktop ports, and the platform/brand/wake specifics. AGENTS.md keeps only the binding rules an agent must respect on every change and links here for the depth.

Status drift moves fast. The filesystem + git log -- <module> is the source of truth; verify against it rather than trusting any table below.


Project

Dodaios — voice-driven meta-orchestrator. Not a classical voice assistant: the core pattern is a Supervisor Agent that routes heavy work through the Mission Manager to capability-selected Dodaios-Agent workers. MCP servers and marketplace integrations contribute gated tools to that same path; they are not alternate worker harnesses. The voice layer is just the interface.

Master plan: ~/.claude/plans/also-er-muss-auch-lexical-pond.md — binding for all design decisions. On conflict between plan and code, the plan wins; code deviations must be documented back in the plan.

Binding architecture contracts:

  • Realtime, chat, and voice delegate heavy work through the same spawn-worker → Mission Manager → worker → Critic/Kontrollierer lifecycle. The retired pre-rename (OpenClaw) sub-tier and phantom harness paths must not be restored.
  • docs/anti-drift-three-layer.md — five-layer enum pattern (Python ↔ SQL ↔ Pydantic ↔ TS ↔ UI). Mandatory for any string crossing module boundaries.

Phase Status

Phase Live? Pointers
0–4 — Foundations ✅ Plugin system + protocols, FastAPI/React desktop app, speech pipeline, skill system, tool-use loop, risk-tier executor, core memory, harness dispatch. Detail in docs/phase{0,1,1a,1c,2,4}-*.md.
5 — Vision/Action/Admin/Async/Control + Tiered Routing ✅ dodaios/{vision,admin,tasks,control,telemetry}/. Computer-Use enabled. Tiered routing via ROUTER_TOOLS frozenset. ADR-0001..0011.
6 — Self-Healing Worker-Critic ✅ dodaios/missions/ (event store, manager, recovery, state machine, budget, cleanup, workers, critic, kontrollierer, safety, voice, isolation). ADR-0009, docs/phase6-*.md. Wired into REST + voice path via bootstrap_missions. Live progress (2026-06-15): both the Codex worker and ClaudeDirectWorker stream stdout line-by-line; the orchestrator drain loop emits throttled WorkerProgress (dodaios/missions/events.py) → WS → ReasoningPanel. Re-run (2026-06-15): POST /api/missions/{id}/rerun re-dispatches a terminal mission's stored prompt as a NEW mission linked via parent_mission_id ("Continue" cancelled / "Restart" failed) — the source card is untouched audit; no state-machine or idempotency change (deliberate, avoids AP-14).
7 — Self-Mod (foundation + writer + tools) ✅ dodaios/core/self_mod/ (audit, errors, pending, registry, schema, writer). Three router-tier tools: list_mutable_settings, get_config_value, set_config_value. ADR + writeup in docs/self_mod.md. 7.5 skill authoring by voice/chat is the create-skill router tool (2026-08-18; pyproject.toml → dodaios.plugins.tool.create_skill:CreateSkillTool, router-tier, monitor; one bounded brain call through SkillCreatorService, the same service behind POST /api/skills/creator/* and dodaios skills create). The older spawn-skill-author class is NOT wired. Generated skills land as state="draft" and are never auto-activated (AP-15).
Awareness A0–A5 removed 2026-09-30 The window watcher, episode summaries and the awareness-snapshot / awareness-recall tools were deleted: they spent tokens every turn for a recall feature the product no longer wants. ADR-0009/0010/0012 are superseded; see the ADR-0011 amendment "Awareness tools retired".
Wiki B0/B1/B2/B3/B5/B7/B8/B9 ✅ dodaios/memory/wiki/ (curator, atomic_writer, page_repository, integration, session_rollup, voice_bridge, telemetry, scheduler). Three router-tier tools: wiki-recall, wiki-page-read, wiki-ingest. ADR-0013/0014/0015. B2 = docs/obsidian-setup.md + B9-Wizard. B3 = WikiView.tsx + 6-endpoint wiki_routes.py + wiki_ws.py live-reload. B4 soft-disabled 2026-05-17 via [memory.legacy_curator] enabled = false — legacy Curator package + data/workspace/ snapshot stay on disk; Hart-Cut (vault migration + reader refactor + package delete) remains open. B6 not started.
Realtime → Dodaios-Agents bridge ✅ canonical path live Hybrid tool mode (ADR-0035, 2026-08-19, default): the live model calls every router tool itself through RealtimeToolBridge (catalog minus the computer-use vehicles, compact declarations under [voice].realtime_tool_declaration_budget_tokens) and keeps ONE provider-neutral jarvis_action for computer use, screen looks and any capability it has no function for; the Tool Model runs only those delegated turns. delegate (one function, every action through the router Brain) and direct stay as explicit modes. BrainManager delegates heavy work through spawn-worker; dodaios/missions/ owns worker selection, isolation, retries, Critic review, and Kontrollierer sign-off. MCP and marketplace tools are granted by capability, not exposed as worker harness names.
Ack-Brain (pre-thinking) ✅ dodaios/brain/ack_brain/ — sub-second short acknowledgment before the deep brain replies. Gemini Flash Lite primary. UI preamble bubble. Suppress-if-fast gate at 2000ms ([ack_brain].suppress_if_brain_faster_than_ms). ADR-0014 (flash-brain).
CLI catalog + terminal view ✅ dodaios/clis/ (catalog, installer, loader, prober, registry, risk_integration, usage_log) + dodaios/terminal/ (cross-platform PTY via terminal.backend.make_pty_backend — ConPTY/pywinpty on Windows, ptyprocess on POSIX). Router tool cli-tools (virtual loader → one cli_<name> tool per connected CLI). UI views: ClisView, TerminalView. spawn-cli-worker was REMOVED 2026-05-24 (dead entry point; heavy multi-step CLI work goes through spawn-worker, single-step through the cli_<name> tools — never re-add a CLI spawn tool, it is a D9 recursion vector, AP-5/AP-14).
Board / Profile ("Knows-you" dashboard) ✅ dodaios/board/ (aggregator, store, achievements, evaluator, bio prompts, scheduler, schema.sql) → data/board/personal.db. Parses FlightRecorder JSONL into daily stats / personal records / 10 achievements + an anti-cliché AI-bio. Routes board_routes.py (/api/board/*) + profile_routes.py; views BoardView.tsx, ProfileView.tsx, frontend/src/views/profile/. Deterministic profile writes via the update-profile tool. Separate standalone service stub in board-backend/ (FastAPI + Docker). CHANGELOG v1.0.0-board.
Channels (Web / Telegram / Discord) ✅ dodaios/channels/ (base.py ChannelAdapter, manager.py, bootstrap.py bootstrap_channels, chat_bridge.py, web.py/telegram.py/discord.py). Bridges a DM/guild message into the normal Dodaios chat path. Entry points web (base), telegram (base dep python-telegram-bot), discord (optional [channels] extra, lazy-imported, graceful ChannelStartError when absent). Tokens via the OS credential store.
Friends + Socials ✅ dodaios/friends/ (registry.py FriendRegistry on aiosqlite, status_publisher.py, status_filter.py, messages.py, schemas.py). Routes friends_routes.py + socials_routes.py; views FriendsView.tsx, frontend/src/views/friends/ + socials/. Telegram channel is the live transport (F-FRIENDS F0/F1).
Contacts + Telephony ✅ dodaios/contacts/ (store.py, schema.py, notify.py) — contacts mirror to guaranteed Wiki person pages people/<slug>.md on ContactChanged (PII stays out of the page). Tools contact-lookup (safe), contact-upsert (monitor, write), call-contact (ask, echo-confirm). dodaios/telephony/ (outbound.py, twiml.py, provisioning.py, security.py, session.py) places real outbound calls via Twilio. Twilio is the optional [telephony] extra — routes (telephony_routes.py) + TelephonyManager degrade gracefully when absent (AD-T8). Views TelephonyView.tsx, frontend/src/views/contacts/.
Marketplace plugins ✅ dodaios/marketplace/ (catalog + auth/ OAuth + oauth_callback_server.py + plugin_loader.py/plugin_registry.py/plugin_relevance.py + mcp_bridge.py). The plugin-tools entry-point loader expands connected marketplace plugins into live brain tools. Native REST tools where a catalog transport was insufficient: gmail (gmail_rest, ask — send is consequential) + vercel (vercel_rest, monitor — read-only). Router-tier, never a spawn (AP-5/AP-14). Route marketplace_routes.py; views MarketplaceView.tsx (the storefront section: the community index for plugins and skills under one search), PluginsView.tsx, ExtensionsView.tsx.
Workflows ✅ dodaios/workflows/ (runner.py, scheduler.py, store.py, schema.sql, seed.py). Imperative cron/manual-triggered multi-step pipelines (brain-prompt / harness-dispatch / shell / tool-call / speak steps) — distinct from Phase-6 missions (single persistent self-healing action). Route workflows_routes.py (CRUD); bootstrap_workflows on app.state. View WorkflowsView.tsx.
Conductor ✅ Separate root package conductor/ (api/, core/, jobs/, seed/, cli.py) with its own SQLite store — a YAML-first agentic-workflow canvas (shell/http/agent jobs, cron/webhook/manual triggers, timeline view). Dodaios mounts the Conductor router inside its own FastAPI server → ConductorView.tsx. Do not confuse with Workflows (imperative, in-dodaios/) — Conductor is YAML-first and standalone-capable.
Dodaios-Agents / Outputs ✅ dodaios/agents/registry.py builds an in-RAM agent event tree from the EventBus (harness/Brain/Tool signals; TTL-cached, no DB) → sub_agents_routes.py (/api/sub-agents/tree) + SubAgentsView.tsx. Outputs (outputs_routes.py + OutputsView.tsx) list a mission's deliverables from the filesystem (<repo_parent>/sub-agents-outputs/<slug>/, NOT a DB). An "artifact" is a .md/PDF/HTML/code file a worker produced. Per-artifact download (Content-Disposition attachment) / view (server-rendered markdown→HTML under a strict default-src 'none' CSP) / desktop-only reveal+open-with-default-app via dodaios/platform/open_path.py; native actions are off on headless/VPS (native_file_actions launcher flag).
Frontier (model auto-switch) ✅ dodaios/brain/frontier_{resolver,autoswitch}.py query each provider's /v1/models at boot, detect newer models, and propose switching BrainProviderConfig; the user acknowledges via a modal → POST /api/frontier/ack. Route frontier_routes.py (/api/frontier/{pending,ack}); cache data/frontier_cache.json (24h TTL). Aligns with the "frontier-quality-before-cost" user preference.
Preview / Pointer / Federation ✅ dodaios/preview/registry.py + preview_routes.py — registry of dev-server iframes (Vite :5173, Storybook) surfaced in the sidebar; paired with the start-preview-server / verify-localhost self-verification tools. dodaios/pointer/ (intent.py, context.py, turn.py) — "AI Pointer": resolves the UI element under the mouse cursor via the OS accessibility tree (not a screenshot), attached only on deictic intent ("what's that", "click there"); router tool inspect-pointer. federation_proxy_routes.py — local signing proxy to the Board-federation backend (frontend has no privkey; signs with the credential-store key; path-whitelist anti-traversal).

Infra-only (no UI, consumed internally): dodaios/hardware/detection.py (CPU/GPU/VRAM/CUDA probe + Whisper-model sizing for the wizard / --check); dodaios/orchestrator/ (currently a thin L6 seam — most supervisor logic lives in dodaios/missions/); dodaios/diagnostics/.

Still unrowed (verify with ls dodaios/ui/web/*routes*.py + git log): chats/sessions. Treat the absence of a row as "undocumented, not absent".


Architecture (the parts an agent must respect)

8-Layer model

L7 UI/UX           Tray, Toasts, Admin-API, Desktop-App (FastAPI+React+pywebview), Orb-Overlay (bar / mascot / voice orb / desktop pet)
L6 Orchestrator    State-Machine, Router, BrainManager, Supervisor, Mission-Manager
L5 Harness-Adapter Capability-gated Computer Use and universal Python Script
L4 Brain           5 providers (Claude-API, OpenRouter, OpenAI, Gemini, NVIDIA) + Ack-Brain sub-second tier
L3 Intent/Risk     Classifier, Risk-Tier-Policy, Approval, Rate-Limit-Tracker
L2 Speech          Wake → VAD (Silero) → STT (faster-whisper / Google) → TTS (Gemini Flash / SAPI5)
L1 Audio I/O       WASAPI via sounddevice, Device-Routing, Chime-Feedback
L0 OS/Hardware     Win32, CUDA, Mic/Speakers, global-hotkeys

Dependency rule (strict): higher layers reach lower layers only via protocols (dodaios/core/protocols.py). Lateral communication is only via typed events on EventBus (dodaios/core/bus.py) with frozen=True dataclasses carrying trace_id + timestamp_ns. Subscriber exceptions are swallowed in _safe_dispatch — they must never propagate.

Plugin system (structural, not nominal)

Plugins live under dodaios/plugins/<group>/<name>.py, register via pyproject.toml [project.entry-points."dodaios.<group>"], and must not import from dodaios.* inside the plugin module — only structural compatibility with the protocol (registry at dodaios/core/registry.py). After editing entry-points: pip install -e . --no-deps.

Every group has two names. dodaios.<group> is the one to register a new plugin under; jarvis.<group> is the name plugins were published under before, and it stays valid: both are read in one scan of the installed distributions (dodaios/core/entry_points.py), and the first-party tables in pyproject.toml still use it. A name registered under both loads once, and if the two registrations point at different objects the dodaios.<group> one loads.

Environment variables, for a plugin and for any program the app starts. A person may set a variable under either prefix: DODAIOS_<NAME> or JARVIS_<NAME> (DODAIOS__<SECTION>__<KEY> or JARVIS__<SECTION>__<KEY> for a config override, DODAIOSCTL_<NAME> or JARVISCTL_<NAME> for the command-line client). Inside the app and in every child process only the JARVIS_ spelling is present: on its first import the jarvis package moves each DODAIOS_ variable to its JARVIS_ name and removes the DODAIOS_ one (dodaios/core/env_prefix.py; when both were set to different values the DODAIOS_ value is the one kept). So a plugin reads os.environ["JARVIS_<NAME>"] and never looks for a DODAIOS_ variable. One spelling is kept on purpose: code that starts an external tool removes credentials from the child's environment by name, and a second copy under another name would pass it. A program that runs before the package is imported does the same move itself for the names it reads: the frozen runtime hook, install.sh / install.ps1, the uninstallers and the AppImage's AppRun.

Groups (frozen in PLUGIN_GROUPS): jarvis.wakeword, jarvis.stt, jarvis.tts, jarvis.brain, jarvis.harness, jarvis.tool, jarvis.channel.

Streaming first

All Brain, STT, TTS, Harness provider methods return AsyncIterator[...]. Non-streaming providers yield exactly one element. Consumers always write async for chunk in provider.xxx().

Event-Bus

  • Events are frozen=True dataclasses (dodaios/core/events.py) with trace_id: UUID + timestamp_ns. Immutability enables flight-recorder replay.
  • subscribe_all receives every event — the flight recorder is a wildcard subscriber.
  • A broken subscriber is logged, never propagated.

Secrets

Access via dodaios.core.config.get_secret(key, env_fallback) only. Hierarchy: OS credential store (keyring, service dodaios, with the earlier personal-jarvis read as a fallback) → ENV → .env (dev fallback) → local-file fallback (config._ensure_keyring_backend, 0600 JSON, headless hosts with no Secret Service). The wizard (dodaios/setup/wizard.py) populates the store. Never put API keys in code, jarvis.toml, commits, or .claude/ files. Voice/chat must never accept secrets (AP-2 — STT log leak vector).

Brain providers + ack-brain

Multi-provider is mandatory — never hardcode Anthropic/Claude. Config under [brain.providers.*] in jarvis.toml. Runtime switch via voice ("switch to Gemini") is a plan requirement; BrainManager must support it. Smart fallback chain in dodaios/brain/manager.py. Workers use a claude-cli backend via Claude Max OAuth (user has no Anthropic API account).

Provider-agnostic features (no provider/model hardcoding) — BINDING

Every feature must work with whatever brain provider the user has selected/activated, and across all configured providers — there are five API providers (claude-api, openrouter, openai, gemini, nvidia) plus two subscription-CLI brains (codex over ChatGPT, antigravity over the Google login), and any of them may be the active one. Never branch on a provider name or a model id to enable or disable behavior — no if provider == "grok", no hardcoded grok-4.3, no provider-specific code path. Gate on a capability instead: supports_vision, supports_tools, the runtime can_call_tools() (and codex's runtime supports_vision). If the capability you need doesn't exist yet, add a capability flag — do not name-check the provider.

When the active/selected provider lacks a needed capability — e.g. a text-only CLI brain (antigravity / codex-CLI) cannot see images for Computer-Use — fall through to the first available provider that has the capability, provider-agnostically and never pinned to a favorite; if none is available, degrade gracefully with an honest message. A provider or model literal may appear only as a documented default/fallback (a plugin's own DEFAULT_MODEL) or behind a runtime capability probe — never as the gate that decides whether a feature runs. The Computer-Use "no vision" incident was a capability-flag bug (grok's supports_vision was wrongly False) — the correct fix was to set the flag, not to pin Computer-Use to grok. This generalizes AP-6 (don't hardcode Claude) to don't hardcode any provider (see AP-21).

Open-source single-provider resilience (BINDING). This is an open-source project: it must work for any downloader with whatever single provider key they happen to have. So the same fall-through is mandatory when the active/selected provider fails at runtime — its key is missing/empty, it is rate-limited (HTTP 429), out of credit (HTTP 402), or unreachable: the chain MUST advance to the next available provider in a different family, never retry the same dead one, never give up. No single provider being absent or depleted may brick a core path — and "core path" is not only the deep answer: it is the router, the ack/flash tier, the STT (voice input), the Dodaios-Agent/mission worker, and the mission critic. A tier whose primary AND its fallback resolve to the same provider family is a single-provider brick (see AP-22). Build every tier's chain from the providers that actually have a usable key at runtime (model it on manager._build_fallback_chain + the pre-boot key check), never from hardcoded names. Recovery from a dead provider must be reachable in-app, never via a hand-edited jarvis.toml or a spun-up cloud instance.

Ack-Brain (dodaios/brain/ack_brain/) emits a sub-second short acknowledgment before the deep brain replies. Suppress-if-fast gate at 2000ms keeps it out of the way when the deep brain is already fast.

Persona / custom system prompt: the live persona comes from dodaios/brain/persona_loader.py. base_persona_prompt() returns an editable override (data/custom_system_prompt.md, written atomically via the Settings UI / settings_routes.py) when present, else the packaged DODAIOS_PERSONA.md. load_effective_persona_prompt() is what the brains call: the base plus the active mode's character block. Edits apply on the next turn (no restart); invalidate_cache() clears the in-process cache. Never hardcode the persona string elsewhere.

The typed chat: two surfaces, three runners (2026-08-26)

dodaios/agent_chat/ serves two chats that look the same and mean different things — every session carries a surface (store.SURFACES, mirrored in Pydantic and TypeScript, parity-tested):

Surface What a typed turn is Runner per seat Permission ladder
"jarvis" — the front page's Chat section Dodaios with a keyboard: the same assistant the microphone talks to (soul, persona, profile, identity card, memory, wiki, skills, ALL tools through the risk tiers) on the model the composer picked for THIS chat EVERY seat → runner_brain (BrainManager.generate + a TurnOverride). No CLI seats at all (SurfaceKit.cli_seats = False) ONE ladder in Claude Code's words — ask / accept-edits / bypass + plan
"agent" — the Agentic IDE's chat mode a plain coding-agent session in a folder runner_api (tool loop) or a plain runner_cli the vendor's own ladder

Which providers a surface may be seated on is catalog.rows_for(surface), and it gates on the capability, never on a name (AP-21): a surface with CLI seats gets every row; one without gets only the rows a brain plugin drives (runner_api.BRAIN_BY_PROVIDER). The front page is the second case (maintainer, 2026-08-26) — one pipeline, one place the model is picked, one usage ledger, and a model change is a TurnOverride rather than a different vendor process. Its claude-api row therefore always resolves to the Anthropic endpoint, Claude Code installed or not. Chats seated on a CLI before the change move to the API row of the same brand once per database (service._retire_cli_seats, marked by SQLite's user_version), keeping their transcript and dropping only a model id the endpoint cannot take.

The per-turn override contract (binding). A chat's pick of provider / model / effort NEVER moves the live brain: BrainManager.generate(turn_override=…) carries it in a ContextVar (dodaios/brain/turn_override.py), read in exactly three places — the chain (exactly the pick, no cross-provider stand-in; the intelligent-router lead stays for a tool-incapable pick), the tool surface (tools_extra merged after every gate, tool_filter last) and the dispatcher (reasoning_effort, tool_context, max_turns) — and it writes nothing on the manager: not _active_name, not the provider config, not the dead-lists. The pick runs on its own cached instance (_get_brain(scope=)), so its credential (the Agents-tab key, a task-local override_provider_secrets) is never the voice's. switch(persist=False) / apply_provider_model per chat turn is the mistake that got the first version reverted (ce3fb9673).

Hands and approvals. The folder chip is Dodaios' working folder for the chat: folder_tools.py wraps Read/Write/Edit/Ls/Glob/Grep/RunCommand as real Tools (reads safe, writes and the shell ask) so they run through ToolExecutor like every other Dodaios tool — one safety path (AP-3), the blacklist matching "RunCommand <cmd>". A gate the executor raises is answered by the chat's clickable card, not the voice's two-turn "yes": the turn's tool context declares approval_surface: interactive + an approval_ref ("agent-chat:<session_id>") + approval_timeout_s, the executor echoes the ref on ActionApprovalRequired, and approval_bridge.ChatApprovalBridge (one persistent subscriber, grants keyed by ref, never awaiting the person inside a handler) turns the stance into a decision: ask cards everything consequential, accept-edits waves the folder's Write/Edit through, bypass answers every gate itself; plan offers reading hands only. The blacklist is never reached from here in any stance.

A CLI seat as Dodaios (built, currently unreachable). No surface seats a CLI as Dodaios any more — the front page dropped its CLI seats on 2026-08-26 and the IDE's chat runs a plain coding agent — so run_cli_turn(identity=…) is False on every turn today. The machinery stays wired to the kit (kit.brain_runner and kit.cli_seats) rather than to a surface name, so a surface that wants both works without a rebuild. How it works, when reached: a subscription CLI keeps its own loop (that is what the subscription pays for; only the vendor's official binary spends it) but runs with Dodaios' head: BrainManager.render_surface_prompt renders the voice turn's prompt layers read-only, dodaios_harness.build_identity adds the chat's transcript on a fresh conversation and the surface addendum, and each CLI takes it the way it can (Claude Code: --append-system-prompt-file; Codex and agy: in front of the stdin prompt; Grok Build: a compact cut on argv). The CLI's MCP config names the session in an X-Jarvis-Chat-Session header; mcp_server_routes reads it into dodaios_tools_server.CHAT_SESSION_REF, and a Dodaios tool the CLI reaches over MCP asks on the chat's card instead of failing fast as unattended.

Risk-Tier system

Four levels: safe / monitor / ask / block. Priority is blacklist > whitelist > tool default (dodaios/safety/risk_tier.py). Whitelist downgrades a tier to safe with approved_by="whitelist" — this is the anti-confirmation-fatigue contract. Direct calls to Tool.execute() are a bug; only ToolExecutor.execute() is authorized.

Router discipline (binding, ADR-0011 amended)

The router-tier brain is a pure dispatcher. Tool surface is the ROUTER_TOOLS frozenset in dodaios/brain/factory.py. Direct actions outside this set are delegated to the agent harness via the harness-spawn tool.

Force-spawn heuristic in BrainManager._should_force_spawn:

  • Smalltalk allowlist wins → never spawn.
  • Action verb (lies/baue/installiere/öffne/mach/zeig + repair words) → spawn.
  • External-system marker (PR/Repo/GitHub/Issue) → spawn.

Patterns configurable under [brain.routing]. The sub-tier was deleted in Welle 4 — only "router" remains. Resurrecting SUB_TOOLS or adding the harness-spawn / dispatch-with-review / run-skill tool to any worker set breaks the D9 recursion guard. When extending ROUTER_TOOLS: amend ADR-0011 + extend tests/unit/brain/test_routing.py.

Output filter discipline (voice path)

Brain output → TTS goes through scrub_for_voice in dodaios/brain/output_filter.py — regex only, no LLM calls (latency mandate). Two TTS paths are wired through scrub:

  • _handle_utterance → _speak() → tts.synthesize (pipeline.py:1330).
  • _on_announcement (skill/agent announcements, harness summary_de readback) (pipeline.py:647).

Whitelist (sacred, never scrubbed): Datei, Email, Browser, Terminal, Notiz, Termin, Kalender. Hyphen-compounds preserved (Browser-Provider stays). For the full blacklist see the module docstring and tests/unit/brain/test_output_filter.py (40 cases). ADR-0010.

Web search (search_web, router-tier)

The search-web tool (dodaios/plugins/tool/search_web.py) runs a priority backend chain in dodaios/plugins/tool/search_backends.py: optional prepaid Apifare hop (live Google via DataForSEO, if apifare_api_key / APIPAY_TOKEN is set) → real DuckDuckGo SERP via the key-free ddgs dependency (default) → DuckDuckGo Instant Answer (last-resort encyclopedic abstract). The chain stays key-free so the base VPS install still searches; there is no Tavily, Serper or Brave key requirement. A missing, empty-balance (HTTP 402) or failed prepaid hop degrades to DuckDuckGo. Honesty contract: each attempt returns a SearchOutcome with status ok / empty / unavailable. empty = searched, genuinely nothing; unavailable = backend unreachable — the brain must NOT say "no results" for unavailable, it must say search is down.

Atomic config writes

Mutations of jarvis.toml go via dodaios/core/config_writer.py only — tomlkit-based (preserves comments), _WRITE_LOCK mutex, BOM-aware read/write, tempfile + os.replace. For Phase-7 self-mod, the pipeline is non-negotiable: Allowlist → Read → Apply → Pre-Validate (JarvisConfig.model_validate) → Backup → Tempfile+replace → sync reload-test → Rollback-on-fail → ConfigReloaded dispatch → Backup GC → Audit (AD-5; AP-3/4/5/13/14). Backup directory must be outside the watchdog scope (AP-13). Reload-test is synchronous, not watchdog-driven (AP-14).

Multi-layer enum drift prevention

When a vocabulary spans Python ↔ SQL ↔ Pydantic ↔ TypeScript ↔ UI label (e.g. HangupReason), use the five-layer pattern from docs/anti-drift-three-layer.md. Reference: dodaios/sessions/constants.py is the single source of truth; models.py runtime-asserts the Pydantic Literal against it. Regression guards: tests/unit/sessions/test_hangup_reason_parity.py + tests/integration/test_sessions_db_compatibility.py. BUG-008 recurred four times because this scaffolding was missing — apply it preemptively for any new wire-format enum (mission status, skill lifecycle, voice tier).

Phase-6 isolation invariants

  • Every worker runs in a fresh git worktree add -b agent/<task-slug> under <repo_parent>/sub-agents-outputs/ (≤200-char path cap). No writes to the user's working tree.
  • Every worker subprocess is contained for kill-on-crash: Windows via a Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE (kernel guarantee — no zombies even on a hard kill); macOS/Linux via a POSIX process-group reaper (start_new_session + os.killpg-on-close) that reaps the worker tree on a clean shutdown / cancel / timeout / exception but, being userspace, leaks on a hard kill -9 of the orchestrator itself.
  • MAX_CRITIC_LOOPS = 3 is hardcoded. Not parameterizable. Changing it requires a new ADR.
  • Action/Observation invariant (ADR-0009): the LLM never authors its own Observation. Voice readback reads only Kontrollierer-signed MissionApproved.summary_de, never correction_instruction from the Critic-LLM.

Optimistic Execution & the "Oops" Protocol (binding)

The core UX contract is one uninterrupted spoken conversation. The Talker (router-brain + ack-brain) acknowledges optimistically and never blocks on an MCP round-trip; the Heavy-Duty Worker (Mission-Manager + claude-cli Sonnet) executes in the background off the chat transcript. Reality-aligned plan + KPIs (M1–M5) + 4-wave execution: docs/plans/optimistic-execution-v1/README.md.

Architecture Decisions

  • AD-OE1 The optimistic ACK ("Geht klar") is emitted before the worker dispatch returns — never after. Audit every _handle_utterance return path (BUG-007/BUG-020 territory).
  • AD-OE2 The Talker never awaits an MCP/network call on the voice path. The talker↔worker queue is the in-process EventBus + mission event store — no external broker.
  • AD-OE3 Dumb tools (local scripts) resolve in-process via local_action_gate; they MUST NOT wake the worker (false-spawn rate = 0).
  • AD-OE4 Smart tools: the worker issues the MCP call, never the Talker.
  • AD-OE5 Oops loop: worker failure → frozen WorkerCorrectionNeeded event → inject into Talker context → speak ONLY at the next Silero-VAD turn-boundary → through scrub_for_voice. Never interrupt mid-utterance.
  • AD-OE6 Zero silent drops: every worker/MCP failure yields a silent retry OR a spoken correction OR an audited apology (anti-BUG-020 invariant).

Coding Standards

  • Latency budgets are SLO-gated: p95 wake→ACK < 1.2 s, intent→ACK < 3.0 s, router decision < 150 ms. Regressions fail CI.
  • Every spoken path (utterance + announcement) goes through scrub_for_voice (regex only, no LLM call — AP-11).
  • New wire-format vocab (correction reasons, mission status) uses the five-layer enum pattern + parity test.
  • ROUTER_TOOLS stays a frozenset; no spawn-tool ever enters a worker set (AP-5/AP-14). Every subprocess uses NO_WINDOW_CREATIONFLAGS (AP-1). Config writes go through config_writer (AP-7).

Cross-platform desktop features (the six ports, behind dodaios/platform/)

The six desktop power-user features that were historically Windows-only are now cross-platform behind the shared dodaios/platform/ capability seam (detect_platform() + a cached frozen Capabilities snapshot, AD-5). Each feature is one Protocol + one per-OS implementation + a sys.platform factory + a graceful logged null-fallback (AD-6); the Windows implementations are untouched (AD-7). The migration plan is docs/plans/cross-platform-mac-linux/; ADR-0020 (cross-platform elevation) supersedes ADR-0001 but reuses the HMAC / Pydantic-argv / shell=False security core unchanged.

Feature Factory Windows macOS Linux Verification
Terminal (PTY) terminal.backend.make_pty_backend ConPTY (pywinpty) ptyprocess ptyprocess CI-provable (real PTY, EK-4)
App-launch plugins.tool.app_resolver.resolve_app_launch_target App Paths open -a xdg-open/exec CI-provable (resolution)
UI-element-click vision.tree_factory.make_ui_tree_source UIA AX (pyobjc) AT-SPI (pyatspi) live sign-off (AX/AT-SPI tree)
Orb overlay overlay.surface.make_overlay_surface Tk color-key Tk -transparentcolor best-effort + tray live sign-off (transparency)
Hotkey trigger.backends.make_hotkey_backend global-hotkeys pynput pynput (X11); Wayland no-op live sign-off (capture)
CU screen indicator cu.indicator.controller.wire_cu_indicator PySide6 sidecar + capture-affinity PySide6 sidecar + capture guard PySide6 sidecar (X11); Wayland no-op live sign-off (glow + Esc)
Admin/elevation admin.transport.make_admin_transport + admin.elevator.make_elevator UAC + SDDL pipe Authorization Services + unix socket pkexec/sudo + unix socket live sign-off (prompt); never CI-E2E

Verification is honestly labelled per feature in docs/plans/cross-platform-mac-linux/SIGNOFF-LOG.md. As of this writing the maintainer has Windows only and nothing is pushed, so every macOS/Linux live GUI/permission behavior is unverified-on-real-desktop and the ci.yml matrix is CI-configured (first green run pending push) — never claim "CI-verified" or "live-verified" until that log says so.

  • Dependency reality (AD-14 — do not "fix" this): pynput + ptyprocess live in the [desktop] extra (ptyprocess gated sys_platform != 'win32'); pyobjc-framework-{Quartz,ApplicationServices,Accessibility} in [desktop-macos] (sys_platform == 'darwin'). Linux pyatspi is NOT on PyPI — never add it as a pip dependency. It is GObject-Introspection, distro-packaged (apt install python3-pyatspi gir1.2-atspi-2.0), surfaced via the capabilities.has_ax_tree runtime probe.
  • Doctrine intact: the headless €5-VPS base install ships none of these desktop extras and still boots on a fresh python:3.11-slim Linux container — every port is extras-gated and degrades to a logged no-op (AD-6) when its capability is absent.
  • Overlay display styles: [ui] orb_style picks the on-screen surface — the slim Dodaios bar, or one of three looks of the floating orb window (mascot, voice_orb, pet), or none. The vocabulary lives in dodaios/ui/overlay_styles.py; ui/orb/bus_bridge.py drives every surface from bus events. Every switch applies live: orb-window looks re-style one window, and a surface the app has not built yet starts in its own host process (dodaios/ui/dodaiosbar/host.py), because a second in-process Tk root aborts the app (BUG-031). The desktop pet (animated pixel-art sprites, control strip, status bubble) is specified in docs/pets.md.
  • Computer-Use screen indicator (ADR-0028): while a CU mission drives the local mouse/keyboard, a breathing Dodaios-gold border glows on every monitor edge with a localized "Esc to cancel" pill. It is a minimal PySide6 sidecar (dodaios/cu/indicator, pyside6-essentials in [desktop]) spawned lazily per mission off the CUControlStarted/Ended events at the ComputerUseHarness.invoke() boundary; a global Escape (existing hotkey backends, armed only while a mission runs) cancels all active missions through the CU-scoped token registry. Windows hides the border from all capture via WDA_EXCLUDEFROMCAPTURE; macOS/Linux blank it around CU's own frame grabs (fail-open). Headless/Wayland → logged no-op.
  • CPU-first device selection (ADR-0024): the default compute device is always cpu. A GPU is used only on an explicit config request (device = "cuda") with a verified capability; auto/empty/unknown resolve to CPU, and a known-bad GPU degrades to CPU with a logged warning. One central policy — dodaios/core/device.py::resolve_device — expresses this; the capability verdict is injected so the always-on wake path keeps its strict out-of-process inference gate (AP-25) and the policy module adds no torch/ctranslate2 import to any path (AP-26).

Windows specifics (do not skip)

  • Unicode stdout: cp1252 default. New CLI modules must call sys.stdout.reconfigure(encoding='utf-8') or stick to ASCII.
  • Subprocess hygiene: every subprocess.* / asyncio.create_subprocess_exec call must pass creationflags=NO_WINDOW_CREATIONFLAGS from dodaios/core/process_utils.py. Missing this triggers the BUG-012 flicker storm under pythonw.exe.
  • Audio: WASAPI via sounddevice. WDM-KS host-API is forbidden (_FORBIDDEN_OUTPUT_HOSTAPIS in dodaios/audio/player.py) — PortAudio's blocking write API crashes there (BUG-014). Pattern-match device names on shortest unique token ("PRO X"), not marketing name.
  • Hotkeys: global-hotkeys. Avoid Alt+F4, Ctrl+C, Win+*. Safe combos: ctrl+right_alt+<letter>.
  • No Windows Service. SYSTEM user has no headset/mic access. Dodaios is a tray app in the user session under pythonw.exe; autostart via shortcut in shell:startup.
  • UAC manifest: asInvoker. Elevate per-action, never globally.

Brand mark / logo (BINDING)

The official Dodaios logo is the hooded figure in assets/brand/dodaios-logo.svg (black ink on white, one even-odd path). Every app icon — assets/icons/dodaios.ico/.icns/dodaios-1024.png, the bundled dodaios/assets/icons/dodaios.png, the DEV copy, the frontend favicon, splash and dodaios-mark-256.png — is rendered from it by python scripts/make_app_icon.py, which also writes src/components/brand/dodaiosLogoPath.ts for the in-app mark; tests/test_app_icon.py fails when the two drift. Edit the SVG and re-run the script; never hand-edit a PNG or ICO.


Wake word — works with ANY user-chosen phrase (BINDING)

The wake word is whatever the user configures ([trigger.wake_word]), and EVERY part of the wake path must work with ANY such phrase — never hardcode, assume, or special-case a specific wake word. There is no built-in/trademarked default ("Hey Jarvis", "Alexa", etc. are NOT assumed); the maintainer's own is a custom phrase, but the code must behave identically for any phrase the user picks. This applies end-to-end: the wake-plan resolver, the OpenWakeWord vs. custom-phrase (stt_match rolling-Whisper) routing, the phrase matcher/verifier, AND the wake transcription itself.

Concretely, for the custom-phrase (stt_match) path: the rolling-Whisper wake must transcribe in the user's wake language ([stt].wake_language, default de) and must NOT silently auto-detect — auto-detect mis-hears a German/short wake phrase as English and mangles it, so the phrase never matches and the wake never fires. A wake word that the user set but that Dodaios cannot recognize is a release-blocking bug. Regression-guard any change here against at least one non-English custom phrase.


Screenshots & scratch captures

All development/verification screenshots go in screenshots/ at the repo root — never the repo root itself, never a random cwd. dodaios/core/screenshots.py defines screenshots_dir() plus a boot sweep (sweep_screenshots, wired into SingleInstance._on_primary_claim) that (a) consolidates any stray root-level image into screenshots/ and (b) prunes captures older than 10 days by mtime. App-runtime Vision frames are separate — they live in data/flight_recorder/blobs/ and are pruned by dodaios/telemetry/retention.py. The folder is git-ignored and self-tidying.


Memory + Wiki

This project has an auto-memory at ~/.claude/projects/<your-claude-project-dir>/memory/. Check MEMORY.md before larger decisions — stable user preferences (multi-provider brain, hybrid privacy, bilingual, anti-confirmation-fatigue, no-Anthropic-API, frontier-quality-before-cost) live there.

The Knowledge Wiki is the long-term memory tier (B0/B1/B5/B7/B8/B9 live). Three router-tier tools: wiki-recall (search), wiki-page-read (read by vault path), wiki-ingest (deterministic save-fact). Vault root configured in [wiki_integration].vault_root; default wiki/obsidian-vault/. Telemetry snapshot at GET /api/wiki/telemetry. Trigger contract in ADR-0014.

Realtime and chat memory use a durable two-stage path. A recall-biased extractor reviews a turn only when the user explicitly asked to keep it (the brain acknowledged the save) or when the user starts a backfill; the automatic review of every turn, the end-of-call sweep and the periodic auto-backfill were removed on 2026-09-30 because they billed paid keys around the clock. Every model call the wiki makes on its own bills only what background work may bill (dodaios/brain/background_policy.py) and passes the runaway guard in dodaios/memory/wiki/background_guard.py (backoff plus a daily cap on the number of calls, default 200). Candidates carry an exact user-turn ID plus a bounded, secret-redacted user-only evidence excerpt in data/jarvis.db; assistant text can resolve a reference but is never evidence. The body-aware consolidator is the binding cleanliness gate: it compares that evidence with complete related pages, then chooses ADD, UPDATE, NOOP, or INVALIDATE through the guarded AtomicWriter. Same-target candidates are serialized, token-capped batches are bisected, transient edit conflicts remain pending, and successful pages receive deterministic session/turn provenance. Topic questions cannot become user-interest claims without an explicit self-disclosure, even when a provider proposes one. Stage 2 also rejects new numeric values absent from the candidate, exact evidence, or existing page. An explicit remember/note/add-to-wiki request makes its supported fact binding: a NOOP is valid only for an unchanged duplicate or unsupported evidence; the control command itself is never stored as knowledge. POST /api/wiki/backfill safely re-reviews recent persisted Realtime sessions under policy-v3 review keys without draining unrelated journal rows.

The legacy Curator-Merger is soft-disabled since 2026-05-17 ([memory.legacy_curator] enabled = false, gated in dodaios/brain/factory.py). The data/workspace/ snapshot stays on disk and ~35 reader sites keep rendering it — but no new writes land there. Hart-Cut (migrate snapshot to wiki/obsidian-vault/, refactor readers, delete dodaios/memory/curator/) remains open.