Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

feat(pets): contracts for the desktop pet overlay style

rubenluetke10-beep05d2ca9b5f29
diff --git a/docs/pets.md b/docs/pets.md
new file mode 100644
index 0000000000..8bfe067e9b
--- /dev/null
+++ b/docs/pets.md
@@ -0,0 +1,204 @@
+# Desktop pets
+
+A desktop pet is an animated pixel-art companion that floats on the desktop
+while Jarvis runs. It is the overlay style `pet`, an alternative to the slim
+Jarvis bar (`jarvis_bar`), the Gigi mascot (`mascot`) and the voice orb
+(`voice_orb`). Users pick the style and the pet under
+**Settings → My Pets**.
+
+The pet sits in the same frameless, always-on-top window as the mascot and the
+voice orb (`ui/orb/overlay.py`, `OrbOverlay`). Switching between those three
+styles, or between pets, applies live. Switching between the bar and any
+orb-window style needs an app restart, because a second Tk root created at
+runtime aborts the process (BUG-031). The settings page offers the restart in
+one click.
+
+## What the user sees
+
+- **The figure.** It reacts to Jarvis in real time (see *States*). It can be
+ dragged anywhere, and its position is remembered per monitor
+ (`[overlay.mascot] position_*`, shared with the mascot).
+- **The control strip** under the figure, from left to right:
+ - pen: raise the main window and open a new chat (`ComposeRequested`);
+ - microphone: mute Jarvis's microphone (`VoiceMuteToggleRequested`, mirrored
+ from `VoiceMuteChanged`);
+ - orb: start a conversation, or hang up the running one; it pulses with the
+ live audio level;
+ - speaker: silence the assistant's voice for this session (TTS volume 0,
+ mirrored from `VoiceSpeakerMuteChanged`).
+- **The status bubble** under the strip. It shows short, condensed lines of
+ what Jarvis is doing: the live transcript while listening, a "Thinking …"
+ header with the current reason, tool or progress line while thinking, and
+ the condensed reply while talking. It collapses to two lines, expands to six
+ on click, and fades out six seconds after the last update while idle. It can
+ be switched off (`[ui] pet_bubble`).
+- **The pet "None"** (`pet_id = "none"`) shows the control strip and the
+ bubble without a figure.
+
+The pet stays on screen while Jarvis is idle. The global shortcut
+(`[trigger] hotkey_pet_toggle`, default `alt+win+p`) hides it or brings it
+back and to the front; hiding lasts until the next app start. On Wayland,
+global shortcuts are a no-op, as for every other shortcut.
+
+## States
+
+The vocabulary lives in `jarvis/ui/pets/states.py` (a module with no imports)
+and is mirrored in `frontend/src/lib/petStates.ts`; a parity test keeps them
+equal.
+
+| State | Driven by |
+|---|---|
+| `idle` | `SystemStateChanged(IDLE)`, no session |
+| `listening` | wake word, `VoiceSessionStarted`, `SystemStateChanged(LISTENING)`, dictation |
+| `thinking` | `SystemStateChanged(THINKING)` |
+| `talking` | `AudioOutFirst`, `SystemStateChanged(SPEAKING)` with audible output |
+| `success` (one-shot, 1.5 s) | `ActionExecuted(success=True)`, `SpeechSpoken.spoken_kind` in `action_done`/`completion`, `JarvisAgentBackgroundCompleted(success=True)` |
+| `error` (one-shot, 2 s) | `ErrorOccurred`, `ActionExecuted(success=False)`, `SpeechSpoken.spoken_kind` in `timeout`/`unavailable`/`stt_unavailable`, `VoiceSessionEnded(hangup_reason="error")` |
+| `sleeping` | five minutes without any Jarvis event; any event wakes it |
+
+States come only from real bus events (`ui/orb/bus_bridge.py`); nothing is
+simulated. The classic voice pipeline does not publish `ErrorOccurred`, so on
+that path an error shows through the `SpeechSpoken` kinds.
+
+## Sprite format `jarvis-pet/1`
+
+A pet is a folder with two files:
+
+```text
+<pet-id>/
+ pet.json manifest
+ sheet.png sprite sheet (RGBA, real transparency)
+```
+
+The sheet is a grid of square cells. `frame_size` is 32, 48 or 64 source
+pixels. Each state is one row with up to eight frames. By convention the rows
+follow the order of `PET_STATES`: `idle`, `listening`, `thinking`, `talking`,
+`success`, `error`, `sleeping`.
+
+```json
+{
+ "format": "jarvis-pet/1",
+ "id": "gigi",
+ "name": "Gigi",
+ "description": "The Jarvis ghost, eight bits tall.",
+ "frame_size": 48,
+ "sheet": "sheet.png",
+ "animations": {
+ "idle": {"row": 0, "frames": 4, "fps": 4, "loop": true},
+ "listening": {"row": 1, "frames": 4, "fps": 8, "loop": true},
+ "thinking": {"row": 2, "frames": 6, "fps": 8, "loop": true},
+ "talking": {"row": 3, "frames": 4, "fps": 10, "loop": true},
+ "success": {"row": 4, "frames": 6, "fps": 10, "loop": false},
+ "error": {"row": 5, "frames": 4, "fps": 8, "loop": false},
+ "sleeping": {"row": 6, "frames": 2, "fps": 2, "loop": true}
+ }
+}
+```
+
+Rules the loader enforces (`jarvis/ui/pets/manifest.py`):
+
+- `format` must be `jarvis-pet/1`; `id` matches `^[a-z0-9][a-z0-9-]{0,31}$`
+ (built-in) or `^u[0-9a-f]{16}$` (user-created).
+- `name` is 1–40 characters, `description` at most 140.
+- `idle` is required. A missing state borrows another row through
+ `STATE_FALLBACKS` (for example `sleeping` → `idle`).
+- `frames` is 1–8, `fps` is 1–12, and every referenced cell lies inside the
+ sheet. The sheet is at most 512 × 512 pixels and 2 MB.
+- Transparency is binary on screen: a pixel is either opaque or gone
+ (alpha ≥ 128 counts as opaque). The overlay keys out pure magenta
+ (`#FF00FF`), so the loader nudges any exact-magenta sprite pixel to
+ `#FE00FE`.
+
+Built-in pets live in `jarvis/ui/pets/builtin/<id>/` and are generated from
+pixel data by `scripts/pets/build_pets.py` (deterministic; a test checks the
+committed PNGs match the script). User-created pets live in
+`<data dir>/pets/<u…id>/`.
+
+### Built-in pets
+
+| Id | Name | Idea |
+|---|---|---|
+| `gigi` | Gigi | The Jarvis ghost as a pixel sprite (default) |
+| `miso` | Miso | A cat: ears up while listening, curls up to sleep |
+| `brew` | Brew | A teapot: steams while thinking, whistles while talking |
+| `bolt` | Bolt | A battery: charges while thinking, sparks on success, runs flat asleep |
+| `mochi` | Mochi | A jelly blob that wobbles while listening |
+| `shelly` | Shelly | A snail: the shell spins while thinking, it withdraws to sleep |
+
+## Adding a pet
+
+- **As a user:** Settings → My Pets → Create pet. Upload a sheet in the
+ layout above (a template PNG is offered for download), add a name and a
+ one-sentence description, optionally a `pet.json`. Without a manifest the
+ rows are read in `PET_STATES` order and the frame count per row is the
+ number of non-empty cells.
+- **As a developer:** add a builder to `scripts/pets/build_pets.py`, run it,
+ and commit the generated folder. The parity and completeness tests require
+ every built-in pet to provide all seven states.
+
+## Interfaces
+
+### Surface methods
+
+`OrbOverlay` gains these methods. The bridge calls them through `getattr`, so
+the bar and `NullOverlay` stay untouched. The macOS companion process
+(`jarvis/ui/jarvisbar/host.py`) forwards each one as a command of the same
+name.
+
+| Method | Meaning |
+|---|---|
+| `set_pet(pet_id)` | Swap the figure live; `"none"` shows the strip only |
+| `set_pet_look(scale, bubble)` | Apply size and bubble on/off live |
+| `set_pet_outcome(kind)` | Play the one-shot `success` or `error` |
+| `show_status(header, line)` | Update the status bubble (already condensed) |
+| `set_muted(muted)` | Mirror the microphone mute on the strip |
+| `set_speaker_muted(muted)` | Mirror the speaker mute on the strip |
+| `set_visible(visible)` | Hide or show the whole pet (shortcut, settings) |
+| `toggle_visible()` | Shortcut action: hide, or show and raise |
+
+Control-strip actions reported back through callbacks: `compose`, `mic_mute`,
+`talk`/`hangup` (the orb) and `speaker`.
+
+### Events (`jarvis/core/events.py`)
+
+- `VoiceSpeakerMuteChanged(muted, source)`: published by
+ `SpeechPipeline.set_tts_volume` when the muted-ness flips.
+- `ComposeRequested(source)`: the pen control; DesktopApp raises the window,
+ the frontend opens a new chat.
+- `PetChanged(pet_id, scale, bubble, visible, source)`: published by the pets
+ routes after a change is on disk and applied.
+
+### Configuration
+
+| Key | Default | Meaning |
+|---|---|---|
+| `[ui] orb_style` | `jarvis_bar` | `pet` selects the pet |
+| `[ui] pet_id` | `gigi` | Active pet (built-in id, `u…` id or `none`) |
+| `[ui] pet_scale` | `1.0` | Size multiplier, 0.5–2.0 |
+| `[ui] pet_bubble` | `true` | Show the status bubble |
+| `[trigger] hotkey_pet_toggle` | `alt+win+p` | Hide / show the pet; empty disables it |
+
+### REST (`jarvis/ui/web/pets_routes.py`)
+
+| Method and path | Body / result |
+|---|---|
+| `GET /api/pets` | `{active, scale, bubble, visible, pets: [{id, name, description, builtin, frame_size, animations, sheet_url}]}` |
+| `GET /api/pets/{id}/sheet.png` | The sprite sheet |
+| `PUT /api/pets/active` | `{pet_id}` → saves `[ui] pet_id`, applies live |
+| `PUT /api/pets/settings` | `{scale?, bubble?}` → saves, applies live |
+| `POST /api/pets/visibility` | `{visible}` → runtime only |
+| `POST /api/pets` | multipart: `sheet` (PNG), `name`, `description`, optional `manifest` (JSON), optional `frame_size` → the new pet |
+| `DELETE /api/pets/{id}` | user-created pets only |
+
+## Performance budget
+
+The pet must cost next to nothing while idle:
+
+- every frame is scaled once (nearest-neighbour, integer factor from the
+ monitor DPI times `pet_scale`) and cached as a Tk image;
+- the window repaints only when the frame index changes, and the timer sleeps
+ until the next frame boundary: 4 fps idle, 2 fps asleep, 8–10 fps active,
+ nothing while hidden;
+- the control strip's orb animates only while listening or talking.
+
+Target: an idle pet adds less than 1 % of one CPU core to the app process.
diff --git a/jarvis/core/events.py b/jarvis/core/events.py
index bcbb699ab5..58e93cba3e 100644
--- a/jarvis/core/events.py
+++ b/jarvis/core/events.py
@@ -1120,6 +1120,47 @@ class VoiceMuteChanged(Event):
source: str = ""
+@dataclass(frozen=True, slots=True)
+class VoiceSpeakerMuteChanged(Event):
+ """Authoritative broadcast that the assistant's voice went silent or audible.
+
+ Speaker mute is TTS volume 0 for the running session (nothing is written
+ to ``jarvis.toml`` by a mute). Every writer — the orb/pet speaker disc, the
+ in-app speaker button through ``PUT /api/settings/tts-volume`` — ends in
+ ``SpeechPipeline.set_tts_volume``, and that one choke point publishes this
+ event whenever the muted-ness flips. Surfaces mirror it; none of them keeps
+ its own truth.
+ """
+ muted: bool = False
+ source: str = ""
+
+
+@dataclass(frozen=True, slots=True)
+class ComposeRequested(Event):
+ """A surface asked for a fresh typed conversation.
+
+ Publishers: the pet's pen control, wired through ``OrbBusBridge``. The
+ DesktopApp raises the main window; the frontend (``useWebSocket``) starts
+ a new chat and focuses the composer. Null-safe when no window exists.
+ """
+ source: str = ""
+
+
+@dataclass(frozen=True, slots=True)
+class PetChanged(Event):
+ """The active desktop pet or its look settings changed.
+
+ Emitted by ``jarvis/ui/web/pets_routes.py`` after the value is on disk and
+ applied live, so every open settings page redraws from one event. ``pet_id``
+ is a built-in id, a custom ``u…`` id, or ``"none"`` (control strip only).
+ """
+ pet_id: str = ""
+ scale: float = 1.0
+ bubble: bool = True
+ visible: bool = True
+ source: str = ""
+
+
# Kill-Switch (ADR-0004)
@dataclass(frozen=True, slots=True)
diff --git a/jarvis/ui/overlay_styles.py b/jarvis/ui/overlay_styles.py
index 0eda858d69..73c0b232ff 100644
--- a/jarvis/ui/overlay_styles.py
+++ b/jarvis/ui/overlay_styles.py
@@ -20,9 +20,15 @@ BAR_STYLE = "jarvis_bar"
#: Styles drawn by the floating orb window (``ui.orb.overlay.OrbOverlay``):
#: ``mascot`` is the Gigi ghost, ``voice_orb`` the procedural weather sphere —
-#: the desktop twin of the in-app orb. Both live in the SAME frameless,
-#: always-on-top window, so both can be dragged onto any monitor.
-ORB_STYLES: tuple[str, ...] = ("mascot", "voice_orb")
+#: the desktop twin of the in-app orb — and ``pet`` an animated pixel-art
+#: companion with its own control strip (``jarvis.ui.pets``). All three live in
+#: the SAME frameless, always-on-top window, so all can be dragged onto any
+#: monitor and swapped live without a restart.
+ORB_STYLES: tuple[str, ...] = ("mascot", "voice_orb", "pet")
+
+#: Orb-window styles that stay on screen while Jarvis is idle. The others pop
+#: up only for a conversation.
+PERSISTENT_ORB_STYLES: tuple[str, ...] = ("pet",)
#: No on-screen overlay at all.
HIDDEN_STYLE = "none"
diff --git a/jarvis/ui/pets/__init__.py b/jarvis/ui/pets/__init__.py
new file mode 100644
index 0000000000..b3e6ebae82
--- /dev/null
+++ b/jarvis/ui/pets/__init__.py
@@ -0,0 +1,5 @@
+"""Desktop pets — animated pixel-art companions for the floating overlay window.
+
+See ``docs/pets.md`` for the sprite format, the state vocabulary and how a pet
+reacts to Jarvis. The vocabulary lives in :mod:`jarvis.ui.pets.states`.
+"""
diff --git a/jarvis/ui/pets/states.py b/jarvis/ui/pets/states.py
new file mode 100644
index 0000000000..670bd81a89
--- /dev/null
+++ b/jarvis/ui/pets/states.py
@@ -0,0 +1,67 @@
+"""The desktop pet's vocabulary — ONE definition, imported everywhere.
+
+A pet state is what the floating pet *does* on screen (its animation row). It
+is derived from real Jarvis events by ``jarvis.ui.pets.state_machine`` and it
+crosses Python, the sprite manifest on disk, the REST payload, TypeScript and
+the i18n labels — the five-layer enum this repo has been bitten by four times
+(AP-4 / BUG-008). Same remedy as ``jarvis.ui.jarvisbar.modes``: the tuples live
+in a module with NO imports at all, every layer imports these names instead of
+restating them, and ``tests/unit/ui/pets/test_pet_state_parity.py`` fails the
+build when a layer drifts (TS mirror ``frontend/src/lib/petStates.ts``, the
+``pets.states.*`` locale labels, the built-in manifests).
+
+The ORDER of ``PET_STATES`` is also the row order of a sprite sheet: row 0 is
+``idle``, row 1 ``listening`` and so on. A manifest may name another row per
+state, but the template sheet and every built-in pet follow this order.
+
+Adding a state = add it here (at the END, so existing sheets keep their rows),
+give it a fallback, teach the state machine when it applies, and add the label
+to every locale. Nothing else needs touching.
+"""
+from __future__ import annotations
+
+#: Every animation state a pet can show, in sprite-sheet row order.
+PET_STATES: tuple[str, ...] = (
+ "idle",
+ "listening",
+ "thinking",
+ "talking",
+ "success",
+ "error",
+ "sleeping",
+)
+
+#: States that play once and then hand back to the underlying state.
+ONE_SHOT_STATES: tuple[str, ...] = ("success", "error")
+
+#: How long a one-shot state stays on screen, in seconds.
+ONE_SHOT_SECONDS: dict[str, float] = {"success": 1.5, "error": 2.0}
+
+#: Idle time without any Jarvis event before the pet falls asleep, in seconds.
+SLEEP_AFTER_SECONDS: float = 300.0
+
+#: Which row to borrow when a manifest has no animation for a state. Every
+#: chain ends at ``idle``, the one state a manifest must provide.
+STATE_FALLBACKS: dict[str, str] = {
+ "listening": "idle",
+ "thinking": "listening",
+ "talking": "listening",
+ "success": "idle",
+ "error": "idle",
+ "sleeping": "idle",
+}
+
+#: The pet id that shows the control strip with no figure ("None" in the UI).
+NO_PET_ID: str = "none"
+
+#: The pet a fresh install shows.
+DEFAULT_PET_ID: str = "gigi"
+
+#: Manifest format tag written into every ``pet.json``.
+PET_FORMAT: str = "jarvis-pet/1"
+
+#: Frame edge lengths a sprite sheet may use, in source pixels.
+FRAME_SIZES: tuple[int, ...] = (32, 48, 64)
+
+#: Upper bound of frames in one animation row.
+MAX_FRAMES_PER_STATE: int = 8