Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

feat(appshot): show the assistant the front window on a shortcut, button or request

rubenluetke10-beep3e605159539e
diff --git a/docs/appshots.md b/docs/appshots.md
new file mode 100644
index 0000000000..cc5e7c3e77
--- /dev/null
+++ b/docs/appshots.md
@@ -0,0 +1,81 @@
+# Appshots
+
+An appshot shows the assistant the window you are working in. It captures the
+front window once — picture and on-screen text — and hands it to the
+conversation as context. Settings live under **Settings > Appshots**.
+
+## Three ways to take one
+
+| Trigger | What happens |
+|---|---|
+| **Shortcut** — both Alt keys at once by default (both Option keys on a Mac) | The front window is captured and delivered per **Appshot destination**. |
+| **Voice or chat** — "take an appshot", "mach einen Appshot", "haz un appshot" | The turn that asked captures the front window and answers with it. |
+| **Try it** button on the Appshots page | Waits three seconds so you can switch windows, then behaves like the shortcut. |
+
+A spoken "what do you see?" is the same look (Screen Context); it also plays
+the shutter and shows up as the last appshot.
+
+## Where a shortcut appshot goes
+
+| Destination | Running voice call | No voice call |
+|---|---|---|
+| **Automatic** (default) | Into the call | Onto your next message |
+| **Next message** | Onto your next message | Onto your next message |
+| **Voice call only** | Into the call | Not sent |
+
+- *Into the call* is silent: GPT-Live puts the picture into its thinking
+ backend's context, native live models (Gemini, local servers with image
+ input) receive it as a video frame. Your next words are the question.
+- *Onto your next message*: the next spoken turn uses it, or — while the
+ front-page chat is open — it appears in the composer as an attachment you
+ can remove. It is single use and expires after `[screen_context].ttl_s`.
+- Scheduled tasks, workflows and background agents never take a parked
+ appshot; it waits for a person.
+
+## Privacy
+
+Appshots capture through the Screen Context engine
+([screen-context.md](screen-context.md)), so everything there applies
+unchanged: the app denylist, redaction of password fields and sensitive
+patterns, the capture indicator before the shutter, and no image ever written
+to disk. **Allow appshots** on the Appshots page is `[screen_context].enabled`
+— one switch for every screen look.
+
+The shutter effect's thumbnail is cut from the frame in memory and piped only
+to the local overlay process, which is excluded from screen capture on
+Windows and hidden before any capture elsewhere. The last appshot shown on the
+Appshots page is kept in memory for `[screen_context].deck_preview_s` seconds
+and served with `Cache-Control: no-store`.
+
+## Settings
+
+`[appshot]` in `jarvis.toml`, written only through the app or
+`jarvis api appshot put-settings`:
+
+| Key | Default | Meaning |
+|---|---|---|
+| `hotkey` | `"alt+alt"` | `alt+alt` = both Alt keys; any other combo in the shared hotkey syntax; `""` = off |
+| `target` | `"auto"` | `auto` · `message` · `voice` (table above) |
+| `sound` | `true` | Shutter sound; also needs `[ui].sound_effects` |
+| `effect` | `true` | Flash and corner thumbnail |
+
+## Operating systems
+
+| | Windows | macOS | Linux/X11 | Wayland / headless |
+|---|---|---|---|---|
+| Both-Alt shortcut | `GetAsyncKeyState` (AltGr counts as right Alt) | `CGEventSourceKeyState`, needs the Input Monitoring grant | `XQueryKeymap` via python-xlib | Unavailable, reason shown on the page; voice and the button still work where capture works |
+| Other shortcuts | Shared hotkey backends (`jarvis/trigger/backends`) | same | same | same as above |
+| Flash + thumbnail | PySide6 overlay, excluded from capture | PySide6 overlay | PySide6 overlay | No overlay; the appshot is still taken where capture works |
+| Capture | Screen Context engine | Needs Screen Recording | X11 | Honest refusal |
+
+Only the instance that owns ambient duties (the default app, not the dev
+instance) arms the shortcut.
+
+## Code
+
+`jarvis/appshot/` — `service.py` (take and deliver), `store.py` (pending and
+last appshot, memory only), `gesture.py` (both-Alt watcher), `hotkey.py`
+(shortcut lifecycle), `effect.py` (shutter hook), `delivery.py` (voice calls).
+The live model's `take_appshot` tool is `jarvis/plugins/tool/appshot.py`; the
+REST surface is `jarvis/ui/web/appshot_routes.py`; the page is
+`frontend/src/views/AppshotsView.tsx`.
diff --git a/docs/os-parity.md b/docs/os-parity.md
index d9d584226a..5dd26a33b6 100644
--- a/docs/os-parity.md
+++ b/docs/os-parity.md
@@ -532,6 +532,7 @@ implementations, not stubs.
|---|---|
| Computer-Use / desktop actions (click, type, hotkey, scroll, drag, windows, apps, screenshots, UI trees) | Full per-OS backends (Win32/UIA, Quartz/AX, xdotool/AT-SPI); honest degradation on Wayland/headless/missing TCC grants |
| On-demand Screen Context | One-shot capture is wired into the production brain on Windows, macOS, and Linux/X11; UIA/AX/AT-SPI text is source-filtered, the indicator precedes capture, and Wayland/headless/missing grants refuse honestly |
+| Appshots (front-window capture on a shortcut, button or request) | Capture, privacy and delivery are OS-neutral (Screen Context engine, `jarvis/appshot`). The both-Alt shortcut reads key state per OS: Windows `GetAsyncKeyState`, macOS `CGEventSourceKeyState` (Input Monitoring grant), Linux/X11 `XQueryKeymap`; Wayland/headless report it unavailable on the Appshots page. The flash is the PySide6 overlay where one can run. Verified live on Windows only; see `docs/appshots.md` |
| Voice / audio (capture, playback, VAD, wake, STT, TTS, realtime) | Clean; headless disables voice honestly; WASAPI logic is inert-by-data off Windows |
| Core (launcher, config, keyring, restart, autostart, tray, elevation, paths) | Clean; per-OS autostart (Registry / LaunchAgent / XDG `.desktop`), keyring falls back to a 0600 file on headless hosts |
| Data / agents (wiki, contacts, telephony, sessions, missions, skills, self-mod, channels, MCP) | Clean; mission workers run on POSIX with a real process-group reaper |
diff --git a/docs/screen-context.md b/docs/screen-context.md
index bcc0e2ca15..595fbc3b39 100644
--- a/docs/screen-context.md
+++ b/docs/screen-context.md
@@ -11,6 +11,13 @@ platform through the same adapter seam.
## 1. What this is
+> **Appshots** (2026-09-29) are the user-facing face of this engine: a
+> global shortcut, a button and the word "appshot" take one front-window
+> capture through the service below, with a shutter effect and sound, and
+> deliver it to a running voice call or the next message. See
+> [appshots.md](appshots.md). The Settings card described in Wave 4 moved
+> onto the Appshots page as **Allow appshots**.
+
When the user says something that unambiguously asks Jarvis to *look* — "can you
see this?", "what does that say?", "look at the error" — Jarvis takes **one**
capture of the screen the user is actually working on, enriches it with the
diff --git a/jarvis/appshot/__init__.py b/jarvis/appshot/__init__.py
new file mode 100644
index 0000000000..9b71875947
--- /dev/null
+++ b/jarvis/appshot/__init__.py
@@ -0,0 +1,10 @@
+"""Appshots — show the assistant the window you are working in.
+
+An appshot is one capture of the front window, taken on a global shortcut
+(both Alt keys by default), from the Appshots page, or when the user asks for
+one, and handed to the conversation as context: straight into a running voice
+call, otherwise onto the next message. Capture, privacy filtering and
+retention are the Screen Context engine's (``jarvis/screen_context``); this
+package adds the triggers, the delivery, the shutter effect and the sound
+gate. Nothing here is imported at boot (AP-26). See ``docs/appshots.md``.
+"""
diff --git a/jarvis/appshot/delivery.py b/jarvis/appshot/delivery.py
new file mode 100644
index 0000000000..99e1a26bdf
--- /dev/null
+++ b/jarvis/appshot/delivery.py
@@ -0,0 +1,49 @@
+"""Hand an appshot to a running voice call.
+
+Every continuous voice session (``jarvis.live``) that can take a picture
+exposes ``attach_appshot(image, mime, note)``: GPT-Live puts it into its
+thinking backend's context, native live providers receive it as a video
+frame. The call stays silent — an appshot is context for what the user says
+next, not a question of its own.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any
+
+log = logging.getLogger(__name__)
+
+
+def live_call_running() -> bool:
+ try:
+ from jarvis.live.runtime import active # noqa: PLC0415
+
+ return any(getattr(session, "is_active", False) for session in active())
+ except Exception: # noqa: BLE001 - a probe failure means "no call"
+ log.debug("appshot: live-session probe failed", exc_info=True)
+ return False
+
+
+async def deliver_to_live(image: bytes, mime: str, note: str) -> bool:
+ """``True`` when a running voice call accepted the picture."""
+ try:
+ from jarvis.live.runtime import active # noqa: PLC0415
+
+ sessions: tuple[Any, ...] = active()
+ except Exception: # noqa: BLE001 - no live package state means no call
+ log.debug("appshot: live runtime unavailable", exc_info=True)
+ return False
+ for session in sessions:
+ attach = getattr(session, "attach_appshot", None)
+ if not callable(attach) or not getattr(session, "is_active", False):
+ continue
+ try:
+ if await attach(image, mime, note):
+ return True
+ except Exception: # noqa: BLE001 - fall back to the next message
+ log.warning("appshot: the voice call rejected the appshot", exc_info=True)
+ return False
+
+
+__all__ = ["deliver_to_live", "live_call_running"]
diff --git a/jarvis/appshot/effect.py b/jarvis/appshot/effect.py
new file mode 100644
index 0000000000..d97460151d
--- /dev/null
+++ b/jarvis/appshot/effect.py
@@ -0,0 +1,120 @@
+"""The shutter effect: flash, then the picture flies into the corner.
+
+Installed as the Screen Context shutter hook, so it fires at the true capture
+moment for every look the shared service takes — a shortcut appshot, a spoken
+"take an appshot", a "what do you see?". The thumbnail is cut from the raw
+frame in memory and piped to the local indicator sidecar; it is never stored
+and never published on the event bus.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import io
+import logging
+from typing import Any
+
+log = logging.getLogger(__name__)
+
+#: Longest edge of the thumbnail sent to the sidecar. The card is painted at
+#: ~250 logical px, so this stays sharp on a 2x display and small on the pipe.
+_THUMB_EDGE = 560
+
+_tasks: set[asyncio.Task[None]] = set()
+
+
+def on_shutter(target: Any, size: tuple[int, int], rgb: bytes, monitors: list[dict]) -> None:
+ """Screen Context shutter hook — synchronous, schedules the slow part."""
+ from jarvis.cu.indicator.controller import get_indicator_controller # noqa: PLC0415
+
+ controller = get_indicator_controller()
+ if controller is None:
+ return
+ # Before the capture's own border dismissal can quit the sidecar.
+ controller.hold_for_snap()
+ task = asyncio.get_running_loop().create_task(
+ _play(controller, tuple(target.bbox), size, rgb, monitors),
+ name="appshot-effect",
+ )
+ _tasks.add(task)
+ task.add_done_callback(_tasks.discard)
+
+
+async def _play(
+ controller: Any,
+ bbox: tuple[int, int, int, int],
+ size: tuple[int, int],
+ rgb: bytes,
+ monitors: list[dict],
+) -> None:
+ try:
+ from jarvis.core.config import load_config # noqa: PLC0415
+
+ config = await asyncio.to_thread(load_config)
+ if not bool(getattr(getattr(config, "appshot", None), "effect", True)):
+ return
+ thumb = await asyncio.to_thread(thumbnail_jpeg, size, rgb)
+ monitor, rect = placement(bbox, monitors)
+ shown = await controller.snap(
+ monitor=monitor,
+ rect=rect,
+ thumb_b64=base64.b64encode(thumb).decode("ascii"),
+ )
+ if not shown:
+ log.info("appshot: shutter effect could not be shown on this desktop")
+ except Exception: # noqa: BLE001 - the effect is decoration, never a failure
+ log.warning("appshot: shutter effect failed", exc_info=True)
+
+
+def thumbnail_jpeg(size: tuple[int, int], rgb: bytes) -> bytes:
+ """A small JPEG of the raw frame, for the local effect only."""
+ from PIL import Image # noqa: PLC0415
+
+ image = Image.frombytes("RGB", size, rgb)
+ image.thumbnail((_THUMB_EDGE, _THUMB_EDGE), Image.Resampling.BILINEAR)
+ buffer = io.BytesIO()
+ image.save(buffer, format="JPEG", quality=82)
+ return buffer.getvalue()
+
+
+def placement(
+ bbox: tuple[int, int, int, int], monitors: list[dict]
+) -> tuple[list[int], list[float]]:
+ """The monitor holding ``bbox`` and ``bbox`` as fractions of it.
+
+ Fractions keep the sidecar independent of the capture's pixel units
+ (physical on Windows, points on macOS). ``monitors`` is mss-shaped:
+ ``[0]`` is the virtual desktop, ``[1:]`` the physical screens.
+ """
+ left, top, width, height = bbox
+ cx, cy = left + width / 2.0, top + height / 2.0
+ screens = [m for m in monitors[1:] if isinstance(m, dict)] or [
+ m for m in monitors[:1] if isinstance(m, dict)
+ ]
+ chosen = None
+ for mon in screens:
+ ml, mt = mon.get("left", 0), mon.get("top", 0)
+ mw, mh = mon.get("width", 0), mon.get("height", 0)
+ if ml <= cx < ml + mw and mt <= cy < mt + mh:
+ chosen = mon
+ break
+ if chosen is None:
+ if not screens:
+ return [left, top, max(1, width), max(1, height)], [0.0, 0.0, 1.0, 1.0]
+ chosen = screens[0]
+ ml, mt = int(chosen.get("left", 0)), int(chosen.get("top", 0))
+ mw, mh = max(1, int(chosen.get("width", 1))), max(1, int(chosen.get("height", 1)))
+ x0, y0 = max(left, ml), max(top, mt)
+ x1, y1 = min(left + width, ml + mw), min(top + height, mt + mh)
+ if x1 <= x0 or y1 <= y0:
+ return [ml, mt, mw, mh], [0.0, 0.0, 1.0, 1.0]
+ return [ml, mt, mw, mh], [
+ (x0 - ml) / mw,
+ (y0 - mt) / mh,
+ (x1 - x0) / mw,
+ (y1 - y0) / mh,
+ ]
+
+
+__all__ = ["on_shutter", "placement", "thumbnail_jpeg"]
diff --git a/jarvis/appshot/gesture.py b/jarvis/appshot/gesture.py
new file mode 100644
index 0000000000..21bd2318e6
--- /dev/null
+++ b/jarvis/appshot/gesture.py
@@ -0,0 +1,185 @@
+"""The "both Alt keys at once" gesture.
+
+The shared hotkey backends fold left and right Alt into one token (the
+Windows library knows no difference), so a chord of the two Alt keys cannot be
+expressed there. This module watches exactly that gesture with the cheapest
+primitive each OS offers — a key-state read every 50 ms, no hook, no tap:
+
+* Windows: ``GetAsyncKeyState`` on ``VK_LMENU`` / ``VK_RMENU`` (AltGr raises
+ ``VK_RMENU`` too, so the gesture works on AltGr layouts).
+* macOS: ``CGEventSourceKeyState`` on the left/right Option key codes. Needs
+ the same Input Monitoring grant as every other global shortcut; without it
+ the read stays false and nothing fires.
+* Linux/X11: ``XQueryKeymap`` through python-xlib (pynput's own dependency).
+* Wayland, headless, missing packages: :func:`make_probe` returns ``None``
+ with the reason, and the caller reports the shortcut as unavailable.
+"""
+
+from __future__ import annotations
+
+import logging
+import threading
+from collections.abc import Callable
+
+log = logging.getLogger(__name__)
+
+#: Poll period. A deliberate two-key press lasts far longer than this.
+POLL_S = 0.05
+#: A second gesture within this window is the same press, not a new one.
+_REFIRE_GUARD_S = 0.6
+
+Probe = Callable[[], tuple[bool, bool] | None]
+
+
+def _windows_probe() -> Probe | None:
+ import ctypes # noqa: PLC0415
+
+ user32 = ctypes.WinDLL("user32", use_last_error=True)
+ get_state = user32.GetAsyncKeyState
+ get_state.argtypes = [ctypes.c_int]
+ get_state.restype = ctypes.c_short
+ vk_lmenu, vk_rmenu = 0xA4, 0xA5
+
+ def probe() -> tuple[bool, bool]:
+ return bool(get_state(vk_lmenu) & 0x8000), bool(get_state(vk_rmenu) & 0x8000)
+
+ return probe
+
+
+def _macos_probe() -> Probe | None:
+ import Quartz # type: ignore[import-untyped] # noqa: PLC0415
+
+ source = Quartz.kCGEventSourceStateHIDSystemState
+ key_state = Quartz.CGEventSourceKeyState
+ left_option, right_option = 58, 61
+
+ def probe() -> tuple[bool, bool]:
+ return bool(key_state(source, left_option)), bool(key_state(source, right_option))
+
+ return probe
+
+
+def _x11_probe() -> Probe | None:
+ from Xlib import XK # type: ignore[import-untyped] # noqa: PLC0415
+ from Xlib.display import Display # type: ignore[import-untyped] # noqa: PLC0415
+
+ display = Display()
+ left = display.keysym_to_keycode(XK.XK_Alt_L)
+ rights = {
+ code
+ for code in (
+ display.keysym_to_keycode(XK.XK_Alt_R),
+ display.keysym_to_keycode(XK.string_to_keysym("ISO_Level3_Shift")),
+ )
+ if code
+ }
+ if not left or not rights:
+ display.close()
+ return None
+
+ def down(keymap: list[int], code: int) -> bool:
+ return bool(keymap[code // 8] & (1 << (code % 8)))
+
+ def probe() -> tuple[bool, bool]:
+ keymap = display.query_keymap()
+ return down(keymap, left), any(down(keymap, code) for code in rights)
+
+ return probe
+
+
+def make_probe() -> tuple[Probe | None, str]:
+ """The key-state reader for this host, or ``None`` and why not."""
+ from jarvis.platform import detect_platform # noqa: PLC0415
+ from jarvis.platform.probes import display_present, is_wayland # noqa: PLC0415
+
+ platform = detect_platform()
+ try:
+ if platform == "win32":
+ return _windows_probe(), ""
+ if not display_present():
+ return None, "No display on this computer, so there is no keyboard to watch."
+ if platform == "darwin":
+ return _macos_probe(), ""
+ if is_wayland():
+ return None, (
+ "Wayland does not let apps watch global keys. Pick a different "
+ "shortcut in an X11 session, or ask for an appshot by voice."
+ )
+ probe = _x11_probe()
+ if probe is None:
+ return None, "This keyboard layout has no second Alt key."
+ return probe, ""
+ except ImportError as exc: # the missing package IS the answer; the caller shows it
+ return None, f"The key reader is not installed ({exc.name})."
+ except Exception as exc: # noqa: BLE001 - an unreadable keyboard is "unavailable"
+ log.warning("appshot: key-state probe failed to start", exc_info=True)
+ return None, f"The keyboard could not be read ({type(exc).__name__})."
+
+
+class BothAltWatcher:
+ """Calls ``on_fire`` once each time both Alt keys go down together."""
+
+ def __init__(
+ self,
+ on_fire: Callable[[], None],
+ *,
+ probe: Probe,
+ poll_s: float = POLL_S,
+ clock: Callable[[], float] | None = None,
+ ) -> None:
+ import time # noqa: PLC0415
+
+ self._on_fire = on_fire
+ self._probe = probe
+ self._poll_s = poll_s
+ self._clock = clock or time.monotonic
+ self._stop = threading.Event()
+ self._thread: threading.Thread | None = None
+ self._armed = True
+ self._last_fire = float("-inf")
+
+ def start(self) -> None:
+ if self._thread is not None and self._thread.is_alive():
+ return
+ self._stop.clear()
+ self._thread = threading.Thread(target=self._run, name="appshot-alt-alt", daemon=True)
+ self._thread.start()
+
+ def stop(self) -> None:
+ self._stop.set()
+ thread, self._thread = self._thread, None
+ if thread is not None and thread is not threading.current_thread():
+ thread.join(timeout=1.0)
+
+ def step(self) -> bool:
+ """One poll. ``True`` when this poll fired. Exposed for tests."""
+ state = self._probe()
+ if state is None:
+ return False
+ left, right = state
+ if not (left or right):
+ self._armed = True
+ return False
+ if left and right and self._armed:
+ self._armed = False
+ now = self._clock()
+ if now - self._last_fire < _REFIRE_GUARD_S:
+ return False
+ from jarvis.platform.self_input import synthetic_input_recent # noqa: PLC0415
+
+ if synthetic_input_recent():
+ return False
+ self._last_fire = now
+ self._on_fire()
+ return True
+ return False
+
+ def _run(self) -> None:
+ while not self._stop.wait(self._poll_s):
+ try:
+ self.step()
+ except Exception: # noqa: BLE001 - one bad read must not kill the shortcut
+ log.debug("appshot: key-state read failed", exc_info=True)
+
+
+__all__ = ["POLL_S", "BothAltWatcher", "make_probe"]
diff --git a/jarvis/appshot/hotkey.py b/jarvis/appshot/hotkey.py
new file mode 100644
index 0000000000..c7d3b62038
--- /dev/null
+++ b/jarvis/appshot/hotkey.py
@@ -0,0 +1,223 @@
+"""The global appshot shortcut: armed after boot, re-armed on a settings change.
+
+``[appshot].hotkey`` is either ``alt+alt`` (both Alt keys, watched by
+:mod:`jarvis.appshot.gesture`), any combo in the shared hotkey syntax (armed
+through the regular per-OS :class:`~jarvis.trigger.hotkey.HotkeyTrigger`), or
+empty (off). Only the instance that owns ambient duties arms it — a dev app
+beside the live one would otherwise take every appshot twice.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+log = logging.getLogger(__name__)
+
+BOTH_ALT = "alt+alt"
+
+
+@dataclass(frozen=True, slots=True)
+class ShortcutStatus:
+ hotkey: str
+ armed: bool
+ detail: str = ""
+
+ def to_json(self) -> dict[str, Any]:
+ return {"hotkey": self.hotkey, "armed": self.armed, "detail": self.detail}
+
+
+def normalize_hotkey(value: str) -> str:
+ """Canonical spelling: lower case, no spaces, both-Alt aliases folded."""
+ combo = "+".join(part.strip().lower() for part in str(value or "").split("+") if part.strip())
+ if combo in {"alt+alt", "left_alt+right_alt", "right_alt+left_alt", "alt+altgr", "altgr+alt"}:
+ return BOTH_ALT
+ return combo
+
+
+class AppshotShortcut:
+ """Owns whichever listener the configured shortcut needs."""
+
+ def __init__(self, bus: Any) -> None:
+ self._bus = bus
+ self._loop: asyncio.AbstractEventLoop | None = None
+ self._watcher: Any | None = None
+ self._trigger_task: asyncio.Task[None] | None = None
+ self._busy = False
+ self._status = ShortcutStatus(hotkey="", armed=False, detail="Not started yet.")
+ self._subscribed = False
+
+ @property
+ def status(self) -> ShortcutStatus:
+ return self._status
+
+ async def start(self) -> ShortcutStatus:
+ self._loop = asyncio.get_running_loop()
+ self._subscribe_reload()
+ return await self.reload()
+
+ async def reload(self) -> ShortcutStatus:
+ """Re-read ``[appshot].hotkey`` and re-arm. Never raises."""
+ await self.stop()
+ try:
+ from jarvis.core.config import load_config # noqa: PLC0415
+ from jarvis.core.instance import current_instance # noqa: PLC0415
+
+ config = await asyncio.to_thread(load_config)
+ hotkey = normalize_hotkey(config.appshot.hotkey)
+ if not hotkey:
+ self._status = ShortcutStatus(hotkey="", armed=False, detail="No shortcut set.")
+ return self._status
+ if not current_instance().owns_ambient_duties:
+ self._status = ShortcutStatus(
+ hotkey=hotkey,
+ armed=False,
+ detail="The main app owns global shortcuts; this instance does not arm them.",
+ )
+ return self._status
+ if hotkey == BOTH_ALT:
+ self._status = await self._arm_both_alt()
+ else:
+ self._status = self._arm_combo(hotkey)
+ except Exception as exc: # noqa: BLE001 - a bad shortcut must not break boot
+ log.warning("appshot: shortcut could not be armed", exc_info=True)
+ self._status = ShortcutStatus(
+ hotkey="",
+ armed=False,
+ detail=f"The shortcut failed to start ({type(exc).__name__}).",
+ )
+ log.info(
+ "appshot: shortcut %s (%s)",
+ self._status.hotkey or "off",
+ "armed" if self._status.armed else self._status.detail,
+ )
+ return self._status
+
+ async def stop(self) -> None:
+ watcher, self._watcher = self._watcher, None
+ if watcher is not None:
+ await asyncio.to_thread(watcher.stop)
+ task, self._trigger_task = self._trigger_task, None
+ if task is not None and not task.done():
+ task.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await task
+
+ async def _arm_both_alt(self) -> ShortcutStatus:
+ from jarvis.appshot.gesture import BothAltWatcher, make_probe # noqa: PLC0415
+
+ probe, reason = await asyncio.to_thread(make_probe)
+ if probe is None:
+ return ShortcutStatus(hotkey=BOTH_ALT, armed=False, detail=reason)
+ watcher = BothAltWatcher(self._fire_threadsafe, probe=probe)
+ watcher.start()
+ self._watcher = watcher
+ return ShortcutStatus(hotkey=BOTH_ALT, armed=True)
+
+ def _arm_combo(self, hotkey: str) -> ShortcutStatus:
+ from jarvis.platform.probes import has_hotkey # noqa: PLC0415
+ from jarvis.trigger.hotkey import validate_hotkey # noqa: PLC0415
+
+ verdict = validate_hotkey(hotkey)
+ if not getattr(verdict, "ok", True):
+ return ShortcutStatus(
+ hotkey=hotkey, armed=False, detail=str(getattr(verdict, "reason", "") or "")
+ )
+ if not has_hotkey():
+ return ShortcutStatus(
+ hotkey=hotkey,
+ armed=False,
+ detail="Global shortcuts are not available on this desktop.",
+ )
+ self._trigger_task = asyncio.get_running_loop().create_task(
+ self._run_combo(hotkey), name="appshot-hotkey"
+ )
+ return ShortcutStatus(hotkey=hotkey, armed=True)
+
+ async def _run_combo(self, hotkey: str) -> None:
+ from jarvis.trigger.hotkey import HotkeyTrigger # noqa: PLC0415
+
+ try:
+ trigger = HotkeyTrigger({"appshot": [hotkey]})
+ async with trigger:
+ async for name in trigger.events():
+ if name == "appshot":
+ self._fire()
+ except asyncio.CancelledError:
+ raise
+ except Exception: # noqa: BLE001 - voice and chat keep working without it
+ log.warning("appshot: shortcut listener stopped", exc_info=True)
+
+ def _fire_threadsafe(self) -> None:
+ loop = self._loop
+ if loop is None or loop.is_closed():
+ return
+ with contextlib.suppress(RuntimeError):
+ loop.call_soon_threadsafe(self._fire)
+
+ def _fire(self) -> None:
+ if self._busy:
+ return
+ self._busy = True
+ task = asyncio.get_running_loop().create_task(self._take(), name="appshot-take")
+ task.add_done_callback(lambda _t: setattr(self, "_busy", False))
+
+ async def _take(self) -> None:
+ from jarvis.appshot.service import take_appshot # noqa: PLC0415
+
+ result = await take_appshot(trigger="hotkey", bus=self._bus)
+ if not result.ok:
+ log.info("appshot: shortcut press refused — %s", result.message)
+ await self._publish_refusal(result.message)
+
+ async def _publish_refusal(self, message: str) -> None:
+ try:
+ from jarvis.core.events import AppshotTaken # noqa: PLC0415
+
+ await self._bus.publish(
+ AppshotTaken(source_layer="appshot", trigger="hotkey", delivered_to="refused",
+ target_label=message[:200])
+ )
+ except Exception: # noqa: BLE001 - the log line above already records it
+ log.debug("appshot: refusal receipt failed", exc_info=True)
+
+ def _subscribe_reload(self) -> None:
+ if self._subscribed or not hasattr(self._bus, "subscribe"):
+ return
+ from jarvis.core.events import ConfigReloaded # noqa: PLC0415
+
+ async def _on_reload(event: ConfigReloaded) -> None:
+ if any(key.startswith("appshot.") for key in event.changed_keys):
+ await self.reload()
+
+ self._bus.subscribe(ConfigReloaded, _on_reload)
+ self._subscribed = True
+
+
+_shortcut: AppshotShortcut | None = None
+
+
+def get_shortcut() -> AppshotShortcut | None:
+ return _shortcut
+
+
+async def start_appshot_shortcut(bus: Any) -> AppshotShortcut:
+ """Boot hook (scheduled after the app is ready, never on the boot path)."""
+ global _shortcut
+ if _shortcut is None:
+ _shortcut = AppshotShortcut(bus)
+ await _shortcut.start()
+ return _shortcut
+
+
+__all__ = [
+ "BOTH_ALT",
+ "AppshotShortcut",
+ "ShortcutStatus",
+ "get_shortcut",
+ "normalize_hotkey",
+ "start_appshot_shortcut",
+]
diff --git a/jarvis/appshot/service.py b/jarvis/appshot/service.py
new file mode 100644
index 0000000000..2eb0b428ee
--- /dev/null
+++ b/jarvis/appshot/service.py
@@ -0,0 +1,198 @@
+"""Take an appshot and put it where the conversation will see it.
+
+One entry point for every trigger — the global shortcut, the Appshots page's
+test button, the live model's ``take_appshot`` tool — plus
+:func:`record_turn_capture` for looks a conversation turn took itself. The
+capture always runs through the shared Screen Context service, so the privacy
+denylist, redaction, the pre-shutter indicator and the no-disk retention rule
+apply unchanged; this module only decides what happens to the picture.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+import uuid
+from dataclasses import dataclass, replace
+from typing import Any, Literal
+
+from jarvis.appshot.store import Appshot, get_store
+
+log = logging.getLogger(__name__)
+
+Trigger = Literal["hotkey", "voice", "tool", "button"]
+
+#: Trusted framing in front of the untrusted screen evidence block.
+_APPSHOT_PREAMBLE = (
+ "APPSHOT: the user deliberately captured their front window to give you "
+ "context. Use it for their request. If they only asked you to take an "
+ "appshot, confirm it in one short sentence and ask what they want to know."
+)
+
+
+@dataclass(frozen=True, slots=True)
+class AppshotResult:
+ status: Literal["captured", "refused"]
+ shot: Appshot | None = None
+ message: str = ""
+ reason_code: str = ""
+
+ @property
+ def ok(self) -> bool:
+ return self.status == "captured" and self.shot is not None
+
+
+def _load_config() -> Any:
+ from jarvis.core.config import load_config # noqa: PLC0415
+
+ return load_config()
+
+
+def shot_from_context(context: Any, *, trigger: str) -> Appshot:
+ """Wrap a finished, redacted Screen Context capture as an appshot."""
+ from jarvis.screen_context.turn import model_note # noqa: PLC0415
+
+ target = context.target
+ label = "active window" if str(target.kind) == "window" else (
+ f"monitor {target.monitor_name}" if target.monitor_name else "selected monitor"
+ )
+ return Appshot(
+ id=uuid.uuid4().hex,
+ image=context.image,
+ mime=context.mime,
+ width=int(context.size[0]),
+ height=int(context.size[1]),
+ label=label,
+ app_name=str(getattr(target.window, "app_name", "") or ""),
+ note=f"{_APPSHOT_PREAMBLE}\n{model_note(context)}",
+ ui_text=context.ui_text,
+ trigger=trigger,
+ taken_at=time.time(),
+ )
+
+
+async def take_appshot(
+ *,
+ trigger: Trigger,
+ bus: Any | None = None,
+ deliver: bool = True,
+ trace_id: uuid.UUID | None = None,
+) -> AppshotResult:
+ """Capture the front window once and deliver it per ``[appshot].target``.
+
+ ``deliver=False`` is for a caller that consumes the picture itself (the
+ live model's tool). Never raises; a refusal carries the user-facing reason.
+ """
+ from jarvis.screen_context.models import IntentVerdict, VisualIntent # noqa: PLC0415
+ from jarvis.screen_context.turn import get_service # noqa: PLC0415
+
+ try:
+ config = await asyncio.to_thread(_load_config)
+ if not config.screen_context.enabled:
+ return AppshotResult(
+ status="refused",
+ reason_code="disabled",
+ message="Appshots are switched off. Turn them on under Settings > Appshots.",
+ )
+ service = get_service(bus=bus)
+ outcome = await service.capture(
+ verdict=IntentVerdict(intent=VisualIntent.WINDOW, evidence=("appshot",)),
+ trace_id=trace_id or uuid.uuid4(),
+ )
+ if outcome.status != "captured" or outcome.context is None:
+ return AppshotResult(
+ status="refused",
+ reason_code=outcome.reason_code or outcome.reason_kind or "refused",
+ message=outcome.message or "The appshot could not be taken.",
+ )
+ if outcome.handle_id:
+ service.consume(outcome.handle_id)
+ shot = shot_from_context(outcome.context, trigger=trigger)
+ except Exception: # noqa: BLE001 - a shortcut press must never crash the app
+ log.error("appshot: capture failed", exc_info=True)
+ return AppshotResult(
+ status="refused",
+ reason_code="failure",
+ message="The appshot failed. Nothing was captured.",
+ )
+
+ store = get_store()
+ store.remember(shot, keep_s=float(config.screen_context.deck_preview_s))
+ delivered_to = "turn"
+ if deliver:
+ delivered_to = await _deliver(
+ shot,
+ target=str(config.appshot.target),
+ ttl_s=float(config.screen_context.ttl_s),
+ )
+ store.mark_delivered(shot.id, delivered_to)
+ await _publish(bus, shot, delivered_to)
+ log.info(
+ "appshot: %s %dx%d via %s -> %s",
+ shot.label,
+ shot.width,
+ shot.height,
+ trigger,
+ delivered_to,
+ )
+ return AppshotResult(status="captured", shot=replace(shot, delivered_to=delivered_to))
+
+
+async def record_turn_capture(context: Any, *, bus: Any | None, trigger: Trigger) -> None:
+ """A conversation turn looked at the screen itself — show it as an appshot."""
+ try:
+ config = await asyncio.to_thread(_load_config)
+ shot = shot_from_context(context, trigger=trigger)
+ store = get_store()
+ store.remember(shot, keep_s=float(config.screen_context.deck_preview_s))
+ store.mark_delivered(shot.id, "turn")
+ await _publish(bus, shot, "turn")
+ except Exception: # noqa: BLE001 - the turn already has its picture
+ log.warning("appshot: could not record the turn's capture", exc_info=True)
+
+
+async def _deliver(shot: Appshot, *, target: str, ttl_s: float) -> str:
+ from jarvis.appshot.delivery import deliver_to_live # noqa: PLC0415
+
+ if target in ("auto", "voice") and await deliver_to_live(shot.image, shot.mime, shot.note):
+ return "voice"
+ if target == "voice":
+ return "none"
+ get_store().park(shot, ttl_s=ttl_s)
+ return "message"
+
+
+async def _publish(bus: Any | None, shot: Appshot, delivered_to: str) -> None:
+ if bus is None:
+ return
+ try:
+ from jarvis.core.events import AppshotTaken # noqa: PLC0415
+
+ await bus.publish(
+ AppshotTaken(
+ source_layer="appshot",
+ appshot_id=shot.id,
+ trigger=shot.trigger,
+ delivered_to=delivered_to,
+ target_label=shot.label,
+ width=shot.width,
+ height=shot.height,
+ )
+ )
+ except Exception: # noqa: BLE001 - a lost receipt cannot undo the appshot
+ log.warning("appshot: receipt publication failed", exc_info=True)
+
+
+def take_pending_for_turn() -> Appshot | None:
+ """The appshot waiting for the next message, removed on the way out."""
+ return get_store().take_pending()
+
+
+__all__ = [
+ "AppshotResult",
+ "record_turn_capture",
+ "shot_from_context",
+ "take_appshot",
+ "take_pending_for_turn",
+]
diff --git a/jarvis/appshot/store.py b/jarvis/appshot/store.py
new file mode 100644
index 0000000000..e5cc260688
--- /dev/null
+++ b/jarvis/appshot/store.py
@@ -0,0 +1,127 @@
+"""In-memory home of appshots — nothing here ever touches the disk.
+
+Two slots, with two different promises:
+
+* **pending** — the appshot waiting for the next message. Single use: the
+ first turn (or the chat composer) that takes it removes it, and it expires
+ after ``[screen_context].ttl_s`` like any other unconsumed capture.
+* **latest** — the one picture the Appshots page may show, so the user can see
+ what the assistant was shown. Replaced by the next appshot and dropped after
+ ``[screen_context].deck_preview_s`` (``0`` = never kept).
+
+Both hold bytes, never paths (the Screen Context retention contract).
+"""
+
+from __future__ import annotations
+
+import threading
+import time
+from dataclasses import dataclass, replace
+
+
+@dataclass(frozen=True, slots=True)
+class Appshot:
+ """One captured, already-redacted window or monitor."""
+
+ id: str
+ image: bytes
+ mime: str
+ width: int
+ height: int
+ #: Metadata-safe label ("active window" / "monitor X") for events and logs.
+ label: str
+ #: Front application name — shown in the app only, never published.
+ app_name: str
+ #: Model-facing evidence block that rides with the image.
+ note: str
+ #: Scrubbed on-screen text, when the accessibility layer had any.
+ ui_text: str
+ #: ``hotkey`` | ``voice`` | ``tool`` | ``button``.
+ trigger: str
+ #: Wall-clock seconds.
+ taken_at: float
+ delivered_to: str = ""
+
+ def meta(self) -> dict[str, object]:
+ """What the app may show about it — no pixels."""
+ return {
+ "id": self.id,
+ "width": self.width,
+ "height": self.height,
+ "label": self.label,
+ "app_name": self.app_name,
+ "trigger": self.trigger,
+ "taken_at": self.taken_at,
+ "delivered_to": self.delivered_to,
+ }
+
+
+class AppshotStore:
+ """Thread-safe pending + latest slots with monotonic expiry."""
+
+ def __init__(self, *, clock=time.monotonic) -> None:
+ self._clock = clock
+ self._lock = threading.Lock()
+ self._pending: Appshot | None = None
+ self._pending_until = 0.0
+ self._latest: Appshot | None = None
+ self._latest_until = 0.0
+
+ def remember(self, shot: Appshot, *, keep_s: float) -> None:
+ """Make ``shot`` the picture the Appshots page shows."""
+ with self._lock:
+ if keep_s <= 0:
+ self._latest = None
+ return
+ self._latest = shot
+ self._latest_until = self._clock() + keep_s
+
+ def park(self, shot: Appshot, *, ttl_s: float) -> None:
+ """Hold ``shot`` for the next message, replacing an older one."""
+ with self._lock:
+ self._pending = shot
+ self._pending_until = self._clock() + max(1.0, ttl_s)
+
+ def take_pending(self, shot_id: str | None = None) -> Appshot | None:
+ """Remove and return the pending appshot (optionally only a given id)."""
+ with self._lock:
+ shot = self._live_pending()
+ if shot is None or (shot_id and shot.id != shot_id):
+ return None
+ self._pending = None
+ return shot
+
+ def peek_pending(self) -> Appshot | None:
+ with self._lock:
+ return self._live_pending()
+
+ def latest(self) -> Appshot | None:
+ with self._lock:
+ if self._latest is not None and self._clock() >= self._latest_until:
+ self._latest = None
+ return self._latest
+
+ def mark_delivered(self, shot_id: str, delivered_to: str) -> None:
+ with self._lock:
+ if self._latest is not None and self._latest.id == shot_id:
+ self._latest = replace(self._latest, delivered_to=delivered_to)
+
+ def clear(self) -> None:
+ with self._lock:
+ self._pending = None
+ self._latest = None
+
+ def _live_pending(self) -> Appshot | None:
+ if self._pending is not None and self._clock() >= self._pending_until:
+ self._pending = None
+ return self._pending
+
+
+_STORE = AppshotStore()
+
+
+def get_store() -> AppshotStore:
+ return _STORE
+
+
+__all__ = ["Appshot", "AppshotStore", "get_store"]
diff --git a/jarvis/audio/effects.py b/jarvis/audio/effects.py
index 97fa5c21b9..898aec28b5 100644
--- a/jarvis/audio/effects.py
+++ b/jarvis/audio/effects.py
@@ -65,6 +65,10 @@ class AudioEffectsService:
config = await asyncio.to_thread(self._load_config)
if not bool(getattr(getattr(config, "ui", None), "sound_effects", True)):
return
+ # Every shared-service capture is an appshot to the user, so the
+ # Appshots page's "play sound" switch owns this cue.
+ if not bool(getattr(getattr(config, "appshot", None), "sound", True)):
+ return
player = self._bound_player()
if player is None:
player = await asyncio.to_thread(self._make_player, config)
diff --git a/jarvis/brain/manager.py b/jarvis/brain/manager.py
index aa89940dc7..8fe5785077 100644
--- a/jarvis/brain/manager.py
+++ b/jarvis/brain/manager.py
@@ -2697,6 +2697,25 @@ _MID_ANSWER_ERROR_PHRASES: dict[str, str] = {
}
+#: Source layers that run without a person waiting on the reply. An appshot
+#: parked for "the next message" belongs to the user's next message, never to
+#: a scheduled task or a background worker that happens to run first.
+_AUTOMATED_LAYER_PREFIXES = (
+ "tasks",
+ "workflows",
+ "skills",
+ "missions",
+ "society",
+ "supervisor",
+ "tool.",
+)
+
+
+def _takes_pending_appshot(source_layer: str | None) -> bool:
+ layer = (source_layer or "").strip().lower()
+ return not layer.startswith(_AUTOMATED_LAYER_PREFIXES)
+
+
class BrainManager:
"""Top-level orchestrator with intent router and smart fallback."""
@@ -10028,6 +10047,7 @@ class BrainManager:
locale=locale,
bus=self._bus,
trace_id=trace_id,
+ allow_pending_appshot=_takes_pending_appshot(source_layer),
)
if outcome.status == "clarify":
pending_map[confirm_key] = _PendingScreenConfirm(
diff --git a/jarvis/brain/tool_gateway.py b/jarvis/brain/tool_gateway.py
index 53417ad42e..7319867ae7 100644
--- a/jarvis/brain/tool_gateway.py
+++ b/jarvis/brain/tool_gateway.py
@@ -105,6 +105,7 @@ class BrainSupervisorToolGateway:
def _voice_tools(self) -> dict[str, Any]:
from jarvis.harness.computer_use_context import peek_computer_use_context
+ from jarvis.plugins.tool.appshot import AppshotTool
from jarvis.plugins.tool.live_screen import LiveScreenTool
tools = self._live_tools()
@@ -113,6 +114,10 @@ class BrainSupervisorToolGateway:
if context is not None and getattr(computer_use, "enabled", True):
tools.update(context.tools or {})
tools["screen_snapshot"] = LiveScreenTool()
+ # "Take an appshot" said in a live call: the shortcut's capture, with
+ # its effect and sound. Voice-only — a brain turn gets the same
+ # picture from its own Screen Context step.
+ tools["take_appshot"] = AppshotTool()
if self._workspace_tool is not None:
# Live delegates coding to one addressed service. The old prompt
# tools silently choose an ambient pane and cannot safely coexist.
diff --git a/jarvis/core/config.py b/jarvis/core/config.py
index b3bc4822fb..88a044ae68 100644
--- a/jarvis/core/config.py
+++ b/jarvis/core/config.py
@@ -2575,6 +2575,38 @@ class ScreenContextConfig(BaseModel):
ocr_enabled: bool = False
+class AppshotConfig(BaseModel):
+ """Top-level ``[appshot]`` config — show the assistant the front window.
+
+ An appshot is one capture of the window the user is working in, taken on
+ a global shortcut or when the user asks for one, and handed to the
+ conversation as context (``jarvis/appshot/``). It runs through the Screen
+ Context capture engine, so every ``[screen_context]`` privacy rule applies.
+
+ The master switch is ``[screen_context].enabled``: an appshot and a spoken
+ "what do you see?" are the same look, so one switch governs both. Every
+ key below is read by ``jarvis.appshot`` (AP-31).
+ """
+
+ model_config = {"extra": "allow"}
+
+ #: Global shortcut. ``alt+alt`` means both Alt keys pressed together (read
+ #: by ``jarvis.appshot.gesture``); any other value is an ordinary combo in
+ #: the shared hotkey syntax; an empty string switches the shortcut off.
+ hotkey: str = "alt+alt"
+
+ #: Where a shortcut appshot goes. ``auto``: into the running voice call,
+ #: otherwise onto the next message. ``message``: always onto the next
+ #: message. ``voice``: only into a running voice call.
+ target: Literal["auto", "message", "voice"] = "auto"
+
+ #: Shutter sound on every appshot (also gated by ``[ui].sound_effects``).
+ sound: bool = True
+
+ #: Flash and thumbnail animation on the captured window.
+ effect: bool = True
+
+
class ComputerUseConfig(BaseModel):
"""Top-level ``[computer_use]`` config for the Computer-Use harness.
@@ -4366,6 +4398,9 @@ class JarvisConfig(BaseModel):
# One-shot, intent-driven screen look (jarvis/screen_context/). Distinct
# from ``[vision]`` above, which governs the always-on observation path.
screen_context: ScreenContextConfig = Field(default_factory=ScreenContextConfig)
+ # Appshots: the front window as conversation context, on a shortcut or on
+ # request (jarvis/appshot/). Captures through Screen Context above.
+ appshot: AppshotConfig = Field(default_factory=AppshotConfig)
# Phase 5/6 — Computer-Use-POAV-Harness (ADR-0008).
computer_use: ComputerUseConfig = Field(default_factory=ComputerUseConfig)
# Low-latency local-action gate. Hidden tools only; never exposed in the
diff --git a/jarvis/core/config_writer.py b/jarvis/core/config_writer.py
index 2538b783c1..9445eac3db 100644
--- a/jarvis/core/config_writer.py
+++ b/jarvis/core/config_writer.py
@@ -993,6 +993,41 @@ def set_screen_context_settings(
_atomic_write(path, out)
+#: Keys ``[appshot]`` accepts — same allowlist reasoning as Screen Context.
+#: The master switch is ``[screen_context].enabled``, written by the setter above.
+APPSHOT_SETTING_KEYS: frozenset[str] = frozenset({"hotkey", "target", "sound", "effect"})
+
+
+def set_appshot_settings(
+ values: dict[str, str | bool],
+ *,
+ path: Path = DEFAULT_CONFIG_FILE,
+) -> None:
+ """Persist a validated ``[appshot]`` patch in one atomic replacement."""
+ unknown = set(values).difference(APPSHOT_SETTING_KEYS)
+ if unknown:
+ raise ValueError(f"unknown appshot setting(s): {sorted(unknown)!r}")
+ if not values:
+ return
+ path = _ensure_writable_config_path(path)
+ with _WRITE_LOCK:
+ raw = path.read_text(encoding="utf-8")
+ had_bom = raw.startswith(_BOM)
+ if had_bom:
+ raw = raw[len(_BOM) :]
+ doc: TOMLDocument = tomlkit.parse(raw)
+ section = doc.get("appshot")
+ if section is None:
+ section = tomlkit.table()
+ doc["appshot"] = section
+ for key, value in values.items():
+ section[key] = value
+ out = tomlkit.dumps(doc)
+ if had_bom:
+ out = _BOM + out
+ _atomic_write(path, out)
+
+
def set_reply_language(name: str, *, path: Path = DEFAULT_CONFIG_FILE) -> None:
"""Persist the user-facing reply-language pin in ``[brain] reply_language``.
diff --git a/jarvis/core/events.py b/jarvis/core/events.py
index 63a09884a8..e16ac9dc9d 100644
--- a/jarvis/core/events.py
+++ b/jarvis/core/events.py
@@ -1227,6 +1227,25 @@ class ScreenCaptureCompleted(Event):
ui_text_source: str = "none"
+@dataclass(frozen=True, slots=True)
+class AppshotTaken(Event):
+ """An appshot was captured and handed to the conversation.
+
+ Metadata only, like ``ScreenCaptureCompleted``: pixels stay in
+ ``jarvis.appshot`` memory and are served to the app on request.
+ """
+
+ appshot_id: str = ""
+ #: What started it: ``hotkey`` | ``voice`` | ``tool`` | ``button``.
+ trigger: str = ""
+ #: Where it went: ``voice`` (running call) | ``message`` (next turn) |
+ #: ``turn`` (the turn that asked for it) | ``none``.
+ delivered_to: str = ""
+ target_label: str = ""
+ width: int = 0
+ height: int = 0
+
+
@dataclass(frozen=True, slots=True)
class ActionPlanned(Event):
"""The CU loop planner proposed the next action (before execution)."""
diff --git a/jarvis/cu/indicator/controller.py b/jarvis/cu/indicator/controller.py
index 6c34074472..41ebef2343 100644
--- a/jarvis/cu/indicator/controller.py
+++ b/jarvis/cu/indicator/controller.py
@@ -31,6 +31,7 @@ import queue
import subprocess
import sys
import threading
+import time
from contextlib import contextmanager, suppress
from typing import Any
from uuid import UUID
@@ -51,6 +52,9 @@ _ESC_HINTS: dict[str, str] = {
_QUIT_GRACE_S = 1.5
_BLANK_ACK_TIMEOUT_S = 0.15
_SHOW_ACK_TIMEOUT_S = 1.2
+#: How long the sidecar stays up for one appshot shutter effect (flash, rest
+#: in the corner, slide out — see ``renderer._SNAP_TOTAL_MS``) plus slack.
+_SNAP_LIFETIME_S = 3.4
def screen_indicator_capability() -> tuple[bool, str]:
@@ -109,6 +113,9 @@ class CUIndicatorController:
self._acks: queue.Queue[str] = queue.Queue()
self._esc_task: asyncio.Task | None = None
self._sidecar_warned = False
+ # Monotonic deadline until which an appshot effect owns the sidecar.
+ self._snap_until = 0.0
+ self._idle_quit_task: asyncio.Task | None = None
# ------------------------------------------------------------------ wiring
def wire(self) -> None:
@@ -229,6 +236,15 @@ class CUIndicatorController:
protocol.CMD_HIDE,
_SHOW_ACK_TIMEOUT_S,
)
+ if time.monotonic() < self._snap_until:
+ # An appshot effect is still on screen; quit once it has played.
+ self._schedule_idle_quit()
+ return
+ await self._quit_sidecar()
+
+ async def _quit_sidecar(self) -> None:
+ if self._proc is None:
+ return
await asyncio.to_thread(
self._send_and_wait,
protocol.CMD_QUIT,
@@ -239,6 +255,59 @@ class CUIndicatorController:
if proc is not None:
await asyncio.to_thread(self._reap, proc)
+ # ---------------------------------------------------------- appshot snap
+ def hold_for_snap(self) -> None:
+ """Keep the sidecar alive for an effect that is about to be sent.
+
+ Called synchronously at the shutter, BEFORE the capture's own border
+ dismissal can quit the sidecar — the thumbnail is encoded a moment
+ later, and a respawn would cost the effect a second of Qt start-up.
+ """
+ self._snap_until = max(self._snap_until, time.monotonic() + _SNAP_LIFETIME_S)
+
+ async def snap(self, *, monitor: list[int], rect: list[float], thumb_b64: str) -> bool:
+ """Play the appshot shutter effect. ``False`` when it cannot run here."""
+ ok, reason = self._border_capability()
+ if not ok:
+ log.debug("[appshot-effect] unavailable: %s", reason)
+ return False
+ async with self._lock:
+ self.hold_for_snap()
+ await asyncio.to_thread(self._spawn_sidecar)
+ if self._proc is None:
+ return False
+ shown = await asyncio.to_thread(
+ self._send_and_wait,
+ protocol.CMD_SNAP,
+ _SHOW_ACK_TIMEOUT_S,
+ monitor=list(monitor),
+ rect=list(rect),
+ thumb=thumb_b64,
+ )
+ self._schedule_idle_quit()
+ return shown
+
+ def _schedule_idle_quit(self) -> None:
+ task = self._idle_quit_task
+ if task is not None and not task.done():
+ return
+ try:
+ loop = asyncio.get_running_loop()
+ except RuntimeError: # no loop = interpreter shutdown; the sidecar dies with us
+ return
+ self._idle_quit_task = loop.create_task(self._idle_quit(), name="appshot-effect-quit")
+
+ async def _idle_quit(self) -> None:
+ while True:
+ remaining = self._snap_until - time.monotonic()
+ if remaining <= 0:
+ break
+ await asyncio.sleep(remaining + 0.05)
+ async with self._lock:
+ if self._active or self._screen_active or time.monotonic() < self._snap_until:
+ return
+ await self._quit_sidecar()
+
@staticmethod
def _border_capability() -> tuple[bool, str]:
try:
@@ -463,6 +532,11 @@ class CUIndicatorController:
_controller: CUIndicatorController | None = None
+def get_indicator_controller() -> CUIndicatorController | None:
+ """The wired controller, or ``None`` before boot wired one."""
+ return _controller
+
+
def wire_cu_indicator(bus: Any) -> CUIndicatorController | None:
"""Idempotent boot hook: subscribe the indicator controller to ``bus``.
@@ -484,4 +558,4 @@ def wire_cu_indicator(bus: Any) -> CUIndicatorController | None:
return controller
-__all__ = ["CUIndicatorController", "wire_cu_indicator"]
+__all__ = ["CUIndicatorController", "get_indicator_controller", "wire_cu_indicator"]
diff --git a/jarvis/cu/indicator/protocol.py b/jarvis/cu/indicator/protocol.py
index 6010c4d5c6..0fed2596a6 100644
--- a/jarvis/cu/indicator/protocol.py
+++ b/jarvis/cu/indicator/protocol.py
@@ -8,6 +8,9 @@ One JSON object per line on the sidecar's stdin::
{"cmd": "blank"} # hide INSTANTLY (capture guard)
{"cmd": "unblank"} # restore after a frame grab
{"cmd": "quit"} # exit the sidecar
+ {"cmd": "snap", "monitor": [l, t, w, h], # appshot shutter effect:
+ "rect": [fx, fy, fw, fh], # flash the captured rect and
+ "thumb": "<base64 jpeg>"} # fly its thumbnail to a corner
The sidecar answers each command with one JSON line on stdout::
@@ -28,9 +31,10 @@ CMD_HIDE = "hide"
CMD_BLANK = "blank"
CMD_UNBLANK = "unblank"
CMD_QUIT = "quit"
+CMD_SNAP = "snap"
ALL_COMMANDS = frozenset(
- {CMD_SHOW, CMD_HIDE, CMD_BLANK, CMD_UNBLANK, CMD_QUIT}
+ {CMD_SHOW, CMD_HIDE, CMD_BLANK, CMD_UNBLANK, CMD_QUIT, CMD_SNAP}
)
#: Sidecar exit code when no usable GUI stack exists (PySide6 missing or
@@ -81,6 +85,7 @@ __all__ = [
"CMD_HIDE",
"CMD_QUIT",
"CMD_SHOW",
+ "CMD_SNAP",
"CMD_UNBLANK",
"EXIT_NO_GUI",
"decode_ack",
diff --git a/jarvis/cu/indicator/renderer.py b/jarvis/cu/indicator/renderer.py
index 60d22cde9e..e1d11c3b5a 100644
--- a/jarvis/cu/indicator/renderer.py
+++ b/jarvis/cu/indicator/renderer.py
@@ -25,8 +25,10 @@ from collections import deque
from contextlib import suppress
from PySide6.QtCore import (
+ QByteArray,
QEasingCurve,
QObject,
+ QRectF,
Qt,
QVariantAnimation,
Signal,
@@ -37,8 +39,10 @@ from PySide6.QtGui import (
QFont,
QFontMetricsF,
QGuiApplication,
+ QImage,
QLinearGradient,
QPainter,
+ QPainterPath,
QPen,
)
from PySide6.QtWidgets import QApplication, QWidget
@@ -178,6 +182,184 @@ class _GlowWindow(QWidget):
)
+# ---------------------------------------------------------------------------
+# Appshot shutter effect — the phone-screenshot moment.
+#
+# A white flash over the captured surface, then the picture shrinks into the
+# bottom-right corner of that monitor, rests there as a rounded thumbnail and
+# slides out. Purely local: the thumbnail arrives over this process's stdin,
+# is painted, and is dropped with the window. Click-through like the glow.
+# ---------------------------------------------------------------------------
+
+_SNAP_FLASH_MS = 200
+_SNAP_FLY_START_MS = 90
+_SNAP_FLY_MS = 430
+_SNAP_HOLD_UNTIL_MS = 2500
+_SNAP_OUT_MS = 320
+_SNAP_TOTAL_MS = _SNAP_HOLD_UNTIL_MS + _SNAP_OUT_MS
+_SNAP_THUMB_W = 320
+_SNAP_THUMB_MAX_H = 240
+_SNAP_MARGIN = 28
+_SNAP_RADIUS = 12.0
+
+
+def _ease_out_cubic(x: float) -> float:
+ x = max(0.0, min(1.0, x))
+ return 1.0 - (1.0 - x) ** 3
+
+
+def _lerp(a: float, b: float, t: float) -> float:
+ return a + (b - a) * t
+
+
+def _match_screen(monitor: list[float]):
+ """The QScreen that best matches a capture-coordinate monitor rect.
+
+ Capture coordinates are physical pixels on Windows and points on macOS,
+ while Qt reports logical geometry — so every plausible interpretation of
+ each screen is scored and the closest one wins. Primary on no data.
+ """
+ screens = QGuiApplication.screens()
+ primary = QGuiApplication.primaryScreen()
+ if not screens or len(monitor) != 4:
+ return primary
+ left, top, width, height = (float(v) for v in monitor)
+ best, best_score = primary, float("inf")
+ for screen in screens:
+ g = screen.geometry()
+ dpr = float(screen.devicePixelRatio() or 1.0)
+ candidates = (
+ (g.x(), g.y(), g.width(), g.height()),
+ (g.x(), g.y(), g.width() * dpr, g.height() * dpr),
+ (g.x() * dpr, g.y() * dpr, g.width() * dpr, g.height() * dpr),
+ )
+ for cx, cy, cw, ch in candidates:
+ score = abs(cx - left) + abs(cy - top) + abs(cw - width) + abs(ch - height)
+ if score < best_score:
+ best, best_score = screen, score
+ return best
+
+
+class _SnapWindow(QWidget):
+ """One monitor-sized, click-through canvas for a single shutter effect."""
+
+ def __init__(self, screen, rect_frac: list[float], thumb: QImage, on_done) -> None:
+ super().__init__(None)
+ self.setWindowFlags(
+ Qt.WindowType.FramelessWindowHint
+ | Qt.WindowType.WindowStaysOnTopHint
+ | Qt.WindowType.WindowTransparentForInput
+ | Qt.WindowType.WindowDoesNotAcceptFocus
+ | Qt.WindowType.Tool
+ | Qt.WindowType.NoDropShadowWindowHint
+ )
+ self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground)
+ self.setAttribute(Qt.WidgetAttribute.WA_TransparentForMouseEvents)
+ self.setAttribute(Qt.WidgetAttribute.WA_ShowWithoutActivating)
+ if sys.platform == "darwin":
+ mac_always_show = getattr(Qt.WidgetAttribute, "WA_MacAlwaysShowToolWindow", None)
+ if mac_always_show is not None:
+ self.setAttribute(mac_always_show)
+ self.setScreen(screen)
+ self.setGeometry(screen.geometry())
+ self._thumb = thumb
+ self._on_done = on_done
+ w, h = float(screen.geometry().width()), float(screen.geometry().height())
+ fx, fy, fw, fh = (max(0.0, min(1.0, float(v))) for v in rect_frac)
+ self._src = QRectF(fx * w, fy * h, max(1.0, fw * w), max(1.0, fh * h))
+ aspect = (
+ thumb.height() / max(1, thumb.width())
+ if not thumb.isNull()
+ else (fh * h) / max(fw * w, 1e-6)
+ )
+ tw = float(_SNAP_THUMB_W)
+ th = tw * aspect
+ if th > _SNAP_THUMB_MAX_H:
+ th = float(_SNAP_THUMB_MAX_H)
+ tw = th / max(aspect, 1e-6)
+ self._dst = QRectF(w - tw - _SNAP_MARGIN, h - th - _SNAP_MARGIN, tw, th)
+ self._t = 0.0
+ self._anim = QVariantAnimation(self)
+ self._anim.setStartValue(0.0)
+ self._anim.setEndValue(float(_SNAP_TOTAL_MS))
+ self._anim.setDuration(_SNAP_TOTAL_MS)
+ self._anim.valueChanged.connect(self._on_tick)
+ self._anim.finished.connect(self._finish)
+
+ def start(self) -> None:
+ self.show()
+ self._anim.start()
+
+ def finish_now(self) -> None:
+ self._anim.stop()
+ self._finish()
+
+ def _on_tick(self, value) -> None:
+ self._t = float(value)
+ self.update()
+
+ def _finish(self) -> None:
+ self.hide()
+ callback, self._on_done = self._on_done, None
+ if callback is not None:
+ callback(self)
+ self.deleteLater()
+
+ def showEvent(self, event) -> None: # noqa: N802 (Qt override)
+ super().showEvent(event)
+ hwnd = int(self.winId())
+ harden_window(hwnd)
+ exclude_from_capture(hwnd)
+
+ def paintEvent(self, event) -> None: # noqa: N802 (Qt override)
+ del event
+ t = self._t
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ painter.setRenderHint(QPainter.RenderHint.SmoothPixmapTransform)
+
+ fly = _ease_out_cubic((t - _SNAP_FLY_START_MS) / _SNAP_FLY_MS)
+ out = _ease_out_cubic((t - _SNAP_HOLD_UNTIL_MS) / _SNAP_OUT_MS)
+ rect = QRectF(
+ _lerp(self._src.x(), self._dst.x(), fly) + 48.0 * out,
+ _lerp(self._src.y(), self._dst.y(), fly),
+ _lerp(self._src.width(), self._dst.width(), fly),
+ _lerp(self._src.height(), self._dst.height(), fly),
+ )
+ radius = _SNAP_RADIUS * fly
+ opacity = 1.0 - out
+
+ if not self._thumb.isNull() and opacity > 0.0:
+ painter.setOpacity(opacity)
+ # Soft shadow grows in as the card lifts off the surface.
+ if fly > 0.0:
+ for spread, alpha in ((10.0, 18), (5.0, 30), (2.0, 46)):
+ shadow = rect.adjusted(-spread, -spread + 4, spread, spread + 4)
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(0, 0, 0, int(alpha * fly)))
+ painter.drawRoundedRect(shadow, radius + spread, radius + spread)
+ clip = QPainterPath()
+ clip.addRoundedRect(rect, radius, radius)
+ painter.save()
+ painter.setClipPath(clip)
+ painter.drawImage(rect, self._thumb)
+ painter.restore()
+ if fly > 0.0:
+ pen = QPen(QColor(255, 255, 255, int(235 * fly)))
+ pen.setWidthF(2.5)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawRoundedRect(rect, radius, radius)
+
+ flash = 1.0 - _ease_out_cubic(t / _SNAP_FLASH_MS)
+ if flash > 0.0:
+ painter.setOpacity(1.0)
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(255, 255, 255, int(225 * flash)))
+ painter.drawRect(self._src)
+ painter.end()
+
+
class Renderer(QObject):
"""Owns the per-monitor windows, the animations, and the IPC slots."""
@@ -185,6 +367,7 @@ class Renderer(QObject):
super().__init__()
self._app = app
self._windows: list[_GlowWindow] = []
+ self._snaps: list[_SnapWindow] = []
self._hint = ""
self._active = False # "show" was requested and not yet "hide"
self._blanked = False # capture guard currently hiding the border
@@ -242,6 +425,8 @@ class Renderer(QObject):
self._blank()
elif cmd == protocol.CMD_UNBLANK:
self._unblank()
+ elif cmd == protocol.CMD_SNAP:
+ self._snap(payload)
elif cmd == protocol.CMD_QUIT:
_ack(cmd)
self._app.quit()
@@ -274,7 +459,33 @@ class Renderer(QObject):
self._active = False
self._fade_to(0.0)
+ def _snap(self, payload: dict) -> None:
+ thumb = QImage()
+ raw = payload.get("thumb")
+ if isinstance(raw, str) and raw:
+ thumb.loadFromData(QByteArray.fromBase64(raw.encode("ascii")))
+ rect = payload.get("rect")
+ if not isinstance(rect, list) or len(rect) != 4:
+ rect = [0.0, 0.0, 1.0, 1.0]
+ monitor = payload.get("monitor")
+ screen = _match_screen(monitor if isinstance(monitor, list) else [])
+ if screen is None:
+ return
+ # One effect at a time: a second appshot replaces the resting card.
+ for old in list(self._snaps):
+ old.finish_now()
+ win = _SnapWindow(screen, rect, thumb, self._snap_done)
+ self._snaps.append(win)
+ win.start()
+
+ def _snap_done(self, win) -> None:
+ with suppress(ValueError):
+ self._snaps.remove(win)
+
def _blank(self) -> None:
+ # A resting thumbnail must never end up inside the next capture.
+ for snap in list(self._snaps):
+ snap.finish_now()
if not self._active:
return
self._blanked = True
diff --git a/jarvis/live/config.py b/jarvis/live/config.py
index a135806d11..0ddff98295 100644
--- a/jarvis/live/config.py
+++ b/jarvis/live/config.py
@@ -60,8 +60,10 @@ class LiveConfig(BaseModel):
"Backchannel policy: Use moderate backchannels. "
"Interruption policy: Listen when interrupted. "
"Stopping speech does not cancel work. "
- "Delegation policy: Backend tools: files, applications, screen, settings, memory, "
- "connected services, web search and agents. Delegate when a request needs these "
+ "Delegation policy: Backend tools: files, applications, screen, appshots, "
+ "settings, memory, connected services, web search and agents. An appshot is a "
+ "picture of the user's front window; asking for one or about one is a backend "
+ "request. Delegate when a request needs these "
"capabilities, careful reasoning, or changes an ongoing task. Do not delegate "
"greetings, simple conversation, or repeating a still-current result. Delegate "
"before answering anything dependent on tools; never guess their results. "
diff --git a/jarvis/live/native.py b/jarvis/live/native.py
index 5a5cb1bc55..ec2ab98e76 100644
--- a/jarvis/live/native.py
+++ b/jarvis/live/native.py
@@ -59,6 +59,7 @@ class NativeLiveVoiceSession(LiveVoiceSession):
"call_tool for any tool not declared directly. For computer control, capture "
"screen_snapshot, inspect the image, call the desktop primitives, and verify "
"the result with a new snapshot. Do not call a separate computer-use harness. "
+ "When the user asks for an appshot, call take_appshot. "
"Request confirmation for pending approvals. Use confirm_action only after "
"explicit approval. A started job is not complete. Never invent tool results. "
"Use workspace-orchestrate for coding tasks: inspect and resolve project, "
@@ -375,6 +376,20 @@ class NativeLiveVoiceSession(LiveVoiceSession):
await self._connection.send_text(text)
return True
+ async def attach_appshot(self, image: bytes, mime: str, note: str) -> bool:
+ """Hand the appshot to the native model as a video frame, silently.
+
+ A text turn would make the model answer right away; the frame alone is
+ what it looks at when the user asks. Servers without image input
+ decline, and the caller parks the appshot for the next message.
+ """
+ del note
+ send_image = getattr(self._connection, "send_image", None)
+ if not self.is_active or not callable(send_image):
+ return False
+ await send_image(image, mime)
+ return True
+
async def end(self, *, reason: str = "client_stop") -> None:
if self._ended:
return
diff --git a/jarvis/live/session.py b/jarvis/live/session.py
index 7c7a82480a..5d9292a837 100644
--- a/jarvis/live/session.py
+++ b/jarvis/live/session.py
@@ -937,6 +937,46 @@ class LiveVoiceSession:
)
return True
+ async def attach_appshot(self, image: bytes, mime: str, note: str) -> bool:
+ """Put an appshot into the thinking backend's context, silently.
+
+ The voice model only hears that it exists; the picture itself goes to
+ the backend conversation, which answers every question about it. No
+ response is requested: the user's next words are the question.
+ """
+ if not self.is_active or self._recovering or self._resume_needs_input:
+ return False
+ await self._connection.send(
+ {
+ "type": "session.thinking.append",
+ "delegation_id": None,
+ "content": (
+ "The user just took an appshot of their front window. It is "
+ "in your backend's context. Delegate any question about what "
+ "they are looking at; do not describe it yourself, and do "
+ "not comment on the appshot unless asked."
+ ),
+ }
+ )
+ await self._connection.send(
+ {
+ "type": "response.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [
+ {"type": "input_text", "text": note},
+ {
+ "type": "input_image",
+ "image_url": f"data:{mime};base64,"
+ + base64.b64encode(image).decode("ascii"),
+ },
+ ],
+ },
+ }
+ )
+ return True
+
async def end(self, *, reason: str = "client_stop") -> None:
if self._ended:
return
diff --git a/jarvis/plugins/tool/appshot.py b/jarvis/plugins/tool/appshot.py
new file mode 100644
index 0000000000..e5d8300970
--- /dev/null
+++ b/jarvis/plugins/tool/appshot.py
@@ -0,0 +1,60 @@
+"""``take_appshot`` — the live voice model's way to honour "take an appshot".
+
+A running voice call does not pass through the brain's turn pipeline, so a
+spoken "take an appshot" reaches the live model as words. This tool gives it
+the same capture the global shortcut takes — the front window, privacy
+filtered, with the shutter effect and sound — and returns the picture as an
+image input for the answer.
+"""
+
+from __future__ import annotations
+
+
+def _app_bus():
+ from jarvis.core.runtime_refs import get_brain_manager, get_web_app
+
+ app = get_web_app()
+ bus = getattr(getattr(app, "state", None), "bus", None)
+ if bus is not None:
+ return bus
+ return getattr(get_brain_manager(), "_bus", None)
+
+
+class AppshotTool:
+ name = "take_appshot"
+ description = (
+ "Take an appshot: capture the user's front window once, with the usual "
+ "privacy filter, so you can see what they are working on. Use when the "
+ "user asks for an appshot or asks you to look at their window or screen."
+ )
+ risk_tier = "monitor"
+ schema = {"type": "object", "properties": {}, "additionalProperties": False}
+
+ async def execute(self, args, ctx):
+ import base64
+
+ from jarvis.appshot.service import take_appshot
+ from jarvis.core.protocols import ToolResult
+
+ del args
+ result = await take_appshot(
+ trigger="tool",
+ bus=_app_bus(),
+ deliver=False,
+ trace_id=getattr(ctx, "trace_id", None),
+ )
+ if not result.ok or result.shot is None:
+ return ToolResult(False, None, result.message or "The appshot could not be taken.")
+ shot = result.shot
+ return ToolResult(
+ True,
+ {
+ "description": f"Appshot of the {shot.label}, {shot.width}x{shot.height}.",
+ "app": shot.app_name,
+ "evidence": shot.note,
+ "_image": {
+ "mime": shot.mime,
+ "data": base64.b64encode(shot.image).decode("ascii"),
+ },
+ },
+ )
diff --git a/jarvis/plugins/tool/navigate.py b/jarvis/plugins/tool/navigate.py
index c1e3800e74..d8494a8995 100644
--- a/jarvis/plugins/tool/navigate.py
+++ b/jarvis/plugins/tool/navigate.py
@@ -54,6 +54,8 @@ KNOWN: frozenset[str] = frozenset(
"feedback",
"agent-instructions",
"wallpaper",
+ # Appshots: the shortcut and destination for showing the front window.
+ "appshots",
"dictionary",
"dictation",
"voice-shortcuts",
@@ -199,6 +201,9 @@ _ALIASES: dict[str, str] = {
"hintergrund": "wallpaper", # i18n-allow: input vocab
"hintergrundbild": "wallpaper", # i18n-allow: input vocab
"fondo de pantalla": "wallpaper", # i18n-allow: input vocab
+ "appshot": "appshots",
+ "app shots": "appshots",
+ "app shot": "appshots",
"task bar": "taskbar",
"taskleiste": "taskbar",
# The Artifacts section (section id kept as "visualization" — its 2026-08
diff --git a/jarvis/screen_context/intent.py b/jarvis/screen_context/intent.py
index c3010ced7d..ce9f20cb38 100644
--- a/jarvis/screen_context/intent.py
+++ b/jarvis/screen_context/intent.py
@@ -349,6 +349,44 @@ _SCREEN_INTENT_RE: re.Pattern[str] = re.compile(
)
+# --------------------------------------------------------------------------
+# Appshot — the feature named outright
+# --------------------------------------------------------------------------
+
+# "Appshot" is the product's own word for "show the assistant my front
+# window", so naming it is the least ambiguous look request there is. Speech
+# recognition spells it every way ("App Shot", "App-Shot", "Epp Shot").
+_APPSHOT_WORD = r"\b(?:app|apps|ap|epp|ep)[\s-]?shots?\b"
+_APPSHOT_RE: re.Pattern[str] = re.compile(_APPSHOT_WORD, re.IGNORECASE)
+
+# Asking ABOUT the feature is not asking for one.
+_APPSHOT_META_RE: re.Pattern[str] = re.compile(
+ r"(?:"
+ rf"\bwhat\s+(?:is|are)\s+(?:an?\s+)?{_APPSHOT_WORD}"
+ rf"|\bhow\s+(?:do|does|can|to)\b[^.?!]{{0,48}}{_APPSHOT_WORD}"
+ # i18n-allow: German speech-input matching data
+ rf"|\bwas\s+(?:ist|sind)\s+(?:ein(?:e)?\s+)?{_APPSHOT_WORD}"
+ rf"|\bwie\s+(?:funktioniert|funktionieren|geht|macht\s+man)\b[^.?!]{{0,48}}{_APPSHOT_WORD}"
+ # i18n-allow: Spanish speech-input matching data
+ rf"|\bque\s+(?:es|son)\s+(?:un\s+)?{_APPSHOT_WORD}"
+ rf"|\bcomo\s+(?:funciona|funcionan|se\s+hace)\b[^.?!]{{0,48}}{_APPSHOT_WORD}"
+ # Opening the Appshots settings page is navigation, not a capture.
+ r"|\b(?:open|go\s+to|navigate\s+to|oeffne|geh\s+(?:zu|in|auf)" # i18n-allow: DE input
+ r"|wechsel\s+(?:zu|in)|abre|ve\s+a)\b[^.?!]{0,24}?"
+ rf"(?:the\s+|die\s+|den\s+|los\s+)?{_APPSHOT_WORD}"
+ r"(?:\s+(?:settings|page|section|einstellungen|seite|ajustes|pagina))?\s*(?:$|[.?!;:])"
+ rf"|{_APPSHOT_WORD}[\s-]?(?:settings|page|section|einstellungen|seite|ajustes)\b"
+ r")",
+ re.IGNORECASE,
+)
+
+
+def mentions_appshot(text: str) -> bool:
+ """Does ``text`` ask for an appshot (not merely ask about the feature)?"""
+ normalized = _normalize(text or "")
+ return bool(_APPSHOT_RE.search(normalized)) and not _APPSHOT_META_RE.search(normalized)
+
+
# --------------------------------------------------------------------------
# Ownership boundary — observing is not operating
# --------------------------------------------------------------------------
@@ -588,6 +626,14 @@ def classify(text: str, *, locale: str = "") -> IntentVerdict:
if requests_screen_operation(normalized):
return IntentVerdict(intent=VisualIntent.NONE, locale=locale)
+ # An appshot is the front window by definition.
+ if _APPSHOT_RE.search(normalized) and not _APPSHOT_META_RE.search(normalized):
+ return IntentVerdict(
+ intent=VisualIntent.WINDOW,
+ evidence=_evidence(_APPSHOT_RE, normalized),
+ locale=locale,
+ )
+
window_hits = _evidence(_WINDOW_SCOPE_RE, normalized)
screen_hits = _evidence(_SCREEN_INTENT_RE, normalized)
diff --git a/jarvis/screen_context/service.py b/jarvis/screen_context/service.py
index c8c7c2e9fd..4fa7c8338b 100644
--- a/jarvis/screen_context/service.py
+++ b/jarvis/screen_context/service.py
@@ -203,6 +203,7 @@ class ScreenContextService:
self._expiry_timers: dict[str, asyncio.TimerHandle | threading.Timer] = {}
self._handle_lock = threading.RLock()
self._closed = False
+ self._shutter_hook: Any | None = None
self._patterns = redaction.build_patterns(
self._settings.extra_patterns,
include_defaults=self._settings.include_default_patterns,
@@ -263,6 +264,16 @@ class ScreenContextService:
attach_audio_effects(bus)
+ def set_shutter_hook(self, hook: Any | None) -> None:
+ """Run ``hook(target, size, rgb, monitors)`` at every shutter.
+
+ Called synchronously right after the pixels are grabbed, on the event
+ loop, before redaction. It exists for the appshot effect, which must
+ start at the true shutter moment; the hook stays in-process and must
+ schedule anything slow itself. Its failures never affect the capture.
+ """
+ self._shutter_hook = hook
+
# ---- intent ----------------------------------------------------------
def classify(self, text: str, *, locale: str = "") -> IntentVerdict:
@@ -424,10 +435,11 @@ class ScreenContextService:
if callable(getter):
window_handle = await asyncio.to_thread(getter)
+ monitors = await asyncio.to_thread(self.displays.monitors)
try:
target, target_degradations = resolve_target(
verdict.intent,
- monitors=await asyncio.to_thread(self.displays.monitors),
+ monitors=monitors,
cursor_point=cursor_point,
bar_point=bar_point,
window=window_facts,
@@ -536,6 +548,7 @@ class ScreenContextService:
# This is the shutter boundary: pixels exist now. Publish only
# metadata so the shared audio layer can play its cue at the
# truthful moment without receiving or retaining screen content.
+ self._run_shutter_hook(target, size, rgb, monitors)
await self._publish_grabbed(size, trace_id=event_trace_id)
# Treat a post-shutter identity change as untrusted: discard the
@@ -614,6 +627,21 @@ class ScreenContextService:
finally:
await self._dismiss_indicator(trace_id=event_trace_id)
+ def _run_shutter_hook(
+ self,
+ target: CaptureTarget,
+ size: tuple[int, int],
+ rgb: bytes,
+ monitors: list[dict],
+ ) -> None:
+ hook = self._shutter_hook
+ if hook is None:
+ return
+ try:
+ hook(target, size, rgb, monitors)
+ except Exception: # noqa: BLE001 - an effect must never cost the capture
+ log.warning("screen_context: shutter hook failed", exc_info=True)
+
# ---- assembly --------------------------------------------------------
def _monitor_privacy_error(self, target: CaptureTarget) -> str | None:
diff --git a/jarvis/screen_context/turn.py b/jarvis/screen_context/turn.py
index 50306a07bd..eab4f1e239 100644
--- a/jarvis/screen_context/turn.py
+++ b/jarvis/screen_context/turn.py
@@ -71,6 +71,11 @@ def get_service(*, bus: Any | None = None) -> ScreenContextService:
settings=settings_from_config(load_config()),
bus=bus,
)
+ # Every look through the shared service is an appshot to the user:
+ # the shutter effect starts at the true capture moment.
+ from jarvis.appshot.effect import on_shutter # noqa: PLC0415
+
+ _service.set_shutter_hook(on_shutter)
elif bus is not None:
# REST may create the service before the conversational brain is ready.
# Attach the one application bus later so receipts are not lost.
@@ -154,7 +159,7 @@ class TurnScreenContext:
return self.status in ("clarify", "refused", "unavailable", "cancelled")
-def _model_note(context: ScreenContext) -> str:
+def model_note(context: ScreenContext) -> str:
"""The English preamble that rides with the image.
Three jobs, all of them about not letting the model invent things:
@@ -223,6 +228,7 @@ async def screen_context_for_turn(
service: ScreenContextService | None = None,
force: bool = False,
trace_id: UUID | None = None,
+ allow_pending_appshot: bool = True,
) -> TurnScreenContext:
"""Resolve the screen question for one conversation turn.
@@ -231,11 +237,21 @@ async def screen_context_for_turn(
derives a language, so a clarifying question cannot flip the conversation's
language mid-session (CLAUDE.md §1.3).
+ An appshot waiting for the next message (``jarvis.appshot``) is this
+ turn's picture, unless the utterance asks for a fresh appshot. Automated
+ callers pass ``allow_pending_appshot=False`` so a scheduled task cannot
+ swallow the picture the user took for their own next question.
+
Never raises. Ordinary non-visual turns return before capture infrastructure
is initialized. Once a visual request is established, an unexpected defect
fails closed with an honest reply so no older screen path can bypass the
privacy policy.
"""
+ if not force and allow_pending_appshot:
+ pending = _pending_appshot(utterance)
+ if pending is not None:
+ return pending
+
if not force:
try:
verdict = intent_module.classify(utterance, locale=locale)
@@ -304,11 +320,15 @@ async def screen_context_for_turn(
if outcome.handle_id:
svc.consume(outcome.handle_id)
+ from jarvis.appshot.service import record_turn_capture # noqa: PLC0415
+
+ await record_turn_capture(context, bus=bus, trigger="voice")
+
return TurnScreenContext(
status="captured",
image=context.image,
mime=context.mime,
- note=_model_note(context),
+ note=model_note(context),
text=context.ui_text,
receipt=context.describe(),
source_hash=f"{context.captured_at_ns:x}",
@@ -326,10 +346,41 @@ async def screen_context_for_turn(
)
+def _pending_appshot(utterance: str) -> TurnScreenContext | None:
+ """The appshot parked for this message, as a captured turn context."""
+ try:
+ from jarvis.appshot.store import get_store # noqa: PLC0415
+
+ store = get_store()
+ if store.peek_pending() is None:
+ return None
+ if intent_module.mentions_appshot(utterance):
+ # "Take an appshot" asks for a NEW picture; the parked one is stale.
+ store.take_pending()
+ return None
+ shot = store.take_pending()
+ except Exception: # noqa: BLE001 - a store fault means "no parked picture"
+ log.warning("screen_context: pending appshot lookup failed", exc_info=True)
+ return None
+ if shot is None:
+ return None
+ log.info("screen_context: attaching the pending appshot %s", shot.id)
+ return TurnScreenContext(
+ status="captured",
+ image=shot.image,
+ mime=shot.mime,
+ note=shot.note,
+ text=shot.ui_text,
+ receipt=f"appshot {shot.width}x{shot.height}",
+ source_hash=shot.id,
+ )
+
+
__all__ = [
"TurnScreenContext",
"TurnStatus",
"get_service",
+ "model_note",
"reset_service",
"screen_context_for_turn",
]
diff --git a/jarvis/ui/web/appshot_routes.py b/jarvis/ui/web/appshot_routes.py
new file mode 100644
index 0000000000..e55a7c6af1
--- /dev/null
+++ b/jarvis/ui/web/appshot_routes.py
@@ -0,0 +1,207 @@
+"""REST API for appshots — the front window as conversation context.
+
+Endpoints (mounted by the WebServer in ``_build_app()``):
+
+ GET /api/appshot/settings → switches, shortcut state, readiness.
+ PUT /api/appshot/settings → change one or more switches.
+ POST /api/appshot/take → take one appshot now (optional delay).
+ GET /api/appshot/latest → metadata of the last appshot.
+ GET /api/appshot/latest/image → its picture (never cached).
+ GET /api/appshot/pending → the appshot waiting for the next message.
+ POST /api/appshot/pending/claim → hand that one to the chat composer.
+ DELETE /api/appshot → forget every held appshot now.
+
+Under the CLI-first contract every action here is also a
+``jarvis api appshot <op>`` command. Pixels stay in memory
+(``jarvis.appshot.store``); nothing here writes an image to disk.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any, Literal
+
+from fastapi import APIRouter, HTTPException, Request, Response
+from pydantic import BaseModel, Field
+
+log = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/appshot", tags=["appshot"])
+
+_NO_STORE = {"Cache-Control": "no-store"}
+
+
+class SettingsPatch(BaseModel):
+ """Every field optional; only the ones sent are written."""
+
+ enabled: bool | None = None
+ hotkey: str | None = Field(default=None, max_length=64)
+ target: Literal["auto", "message", "voice"] | None = None
+ sound: bool | None = None
+ effect: bool | None = None
+
+
+class TakeRequest(BaseModel):
+ #: Seconds to wait first, so the user can bring the window they mean to
+ #: the front after pressing the button inside this app.
+ delay_s: float = Field(default=0.0, ge=0.0, le=10.0)
+
+
+class ClaimRequest(BaseModel):
+ id: str = Field(min_length=1, max_length=64)
+
+
+def _bus(request: Request) -> Any | None:
+ return getattr(request.app.state, "bus", None)
+
+
+def _capability() -> dict[str, Any]:
+ import importlib.util # noqa: PLC0415
+
+ from jarvis.cu.indicator.controller import screen_indicator_capability # noqa: PLC0415
+
+ capture_ok = importlib.util.find_spec("mss") is not None
+ effect_ok, effect_reason = screen_indicator_capability()
+ return {
+ "capture": capture_ok,
+ "capture_detail": "" if capture_ok else "The screen-capture package is not installed.",
+ "effect": effect_ok,
+ "effect_detail": effect_reason,
+ }
+
+
+def _settings_payload() -> dict[str, Any]:
+ from jarvis.appshot.hotkey import get_shortcut, normalize_hotkey # noqa: PLC0415
+ from jarvis.core.config import load_config # noqa: PLC0415
+
+ config = load_config()
+ block = config.appshot
+ shortcut = get_shortcut()
+ status = (
+ shortcut.status.to_json()
+ if shortcut is not None
+ else {"hotkey": normalize_hotkey(block.hotkey), "armed": False, "detail": "Starting."}
+ )
+ return {
+ "enabled": bool(config.screen_context.enabled),
+ "hotkey": normalize_hotkey(block.hotkey),
+ "target": block.target,
+ "sound": bool(block.sound),
+ "effect": bool(block.effect),
+ "sound_effects_master": bool(getattr(config.ui, "sound_effects", True)),
+ "shortcut": status,
+ "readiness": _capability(),
+ }
+
+
[email protected]("/settings")
+async def get_settings() -> dict[str, Any]:
+ return await asyncio.to_thread(_settings_payload)
+
+
[email protected]("/settings")
+async def put_settings(request: Request, patch: SettingsPatch) -> dict[str, Any]:
+ """Write the changed switches, then re-arm the shortcut in place."""
+ from jarvis.appshot.hotkey import get_shortcut, normalize_hotkey # noqa: PLC0415
+ from jarvis.core.config_writer import ( # noqa: PLC0415
+ set_appshot_settings,
+ set_screen_context_settings,
+ )
+
+ changes = patch.model_dump(exclude_none=True)
+ if not changes:
+ raise HTTPException(status_code=400, detail="No settings were provided.")
+ enabled = changes.pop("enabled", None)
+ if "hotkey" in changes:
+ changes["hotkey"] = normalize_hotkey(changes["hotkey"])
+ if changes["hotkey"] and changes["hotkey"] != "alt+alt":
+ from jarvis.trigger.hotkey import validate_hotkey # noqa: PLC0415
+
+ verdict = validate_hotkey(changes["hotkey"])
+ if not verdict.ok:
+ raise HTTPException(status_code=400, detail=verdict.reason or "Invalid shortcut.")
+
+ def _write() -> None:
+ if changes:
+ set_appshot_settings(changes)
+ if enabled is not None:
+ set_screen_context_settings({"enabled": bool(enabled)})
+
+ try:
+ await asyncio.to_thread(_write)
+ except Exception as exc: # noqa: BLE001 - surface the write failure honestly
+ log.error("appshot: settings write failed", exc_info=True)
+ raise HTTPException(status_code=500, detail=f"Could not save the settings: {exc}") from exc
+
+ if enabled is not None:
+ from jarvis.screen_context.turn import reset_service # noqa: PLC0415
+
+ reset_service()
+ shortcut = get_shortcut()
+ if shortcut is not None and "hotkey" in changes:
+ await shortcut.reload()
+ return await asyncio.to_thread(_settings_payload)
+
+
+async def take(request: Request, body: TakeRequest | None = None) -> dict[str, Any]:
+ """Take one appshot of the front window and deliver it like the shortcut."""
+ from jarvis.appshot.service import take_appshot # noqa: PLC0415
+
+ delay = (body or TakeRequest()).delay_s
+ if delay:
+ await asyncio.sleep(delay)
+ result = await take_appshot(trigger="button", bus=_bus(request))
+ if not result.ok or result.shot is None:
+ return {"ok": False, "reason": result.reason_code, "message": result.message}
+ return {"ok": True, "appshot": result.shot.meta()}
+
+
+async def latest() -> dict[str, Any]:
+ from jarvis.appshot.store import get_store # noqa: PLC0415
+
+ shot = await asyncio.to_thread(get_store().latest)
+ return {"appshot": shot.meta() if shot is not None else None}
+
+
[email protected]("/latest/image")
+async def latest_image() -> Response:
+ from jarvis.appshot.store import get_store # noqa: PLC0415
+
+ shot = await asyncio.to_thread(get_store().latest)
+ if shot is None:
+ raise HTTPException(status_code=404, detail="No appshot is being kept right now.")
+ return Response(content=shot.image, media_type=shot.mime, headers=_NO_STORE)
+
+
[email protected]("/pending")
+async def pending() -> dict[str, Any]:
+ from jarvis.appshot.store import get_store # noqa: PLC0415
+
+ shot = await asyncio.to_thread(get_store().peek_pending)
+ return {"appshot": shot.meta() if shot is not None else None}
+
+
[email protected]("/pending/claim")
+async def claim_pending(body: ClaimRequest) -> Response:
+ """Move the waiting appshot into the chat composer. Single use."""
+ from jarvis.appshot.store import get_store # noqa: PLC0415
+
+ shot = await asyncio.to_thread(get_store().take_pending, body.id)
+ if shot is None:
+ raise HTTPException(
+ status_code=404,
+ detail="That appshot was already used or has expired.",
+ )
+ return Response(content=shot.image, media_type=shot.mime, headers=_NO_STORE)
+
+
[email protected]("", openapi_extra={"x-jarvis-dangerous": True})
+async def forget_all() -> dict[str, Any]:
+ """Drop the waiting and the last appshot right now."""
+ from jarvis.appshot.store import get_store # noqa: PLC0415
+
+ await asyncio.to_thread(get_store().clear)
+ return {"ok": True}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx b/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx
index d74c52d8c4..ddaf7005a4 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx
@@ -34,6 +34,7 @@ import { DictationButton } from "@/components/agentchat/DictationButton";
import { ComposerAddMenu } from "@/components/agentchat/ComposerAddMenu";
import { ComposerChipField, type ComposerChipFieldHandle } from "@/components/agentchat/ComposerChipField";
import type { ToolChoice } from "@/components/agentchat/toolChoices";
+import { useAppshotClaim } from "@/components/agentchat/useAppshotClaim";
import { GigiMark } from "@/components/GigiMark";
import { fill, useT } from "@/i18n";
import { cn } from "@/lib/utils";
@@ -166,6 +167,8 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {
},
onAttachProblem,
);
+ // A shortcut appshot parked for "the next message" joins this one.
+ useAppshotClaim(files.attachFiles, surface === "jarvis");
const fileInputRef = useRef<HTMLInputElement | null>(null);
const pasteRescue = usePasteRescue();
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/useAppshotClaim.ts b/jarvis/ui/web/frontend/src/components/agentchat/useAppshotClaim.ts
new file mode 100644
index 0000000000..60dc89bfdc
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/useAppshotClaim.ts
@@ -0,0 +1,47 @@
+/**
+ * An appshot taken for "the next message" lands in the open chat composer.
+ *
+ * The backend parks a shortcut appshot for the next turn. While the front-page
+ * chat is on screen, that next turn is the one being typed here, so the
+ * composer claims the picture (single use — the backend drops its copy) and
+ * holds it like a pasted screenshot: visible, removable, sent with the
+ * sentence through the same attachment path every seat already understands.
+ */
+import { useEffect, useRef } from "react";
+
+import { useT } from "@/i18n";
+import { claimPendingAppshot, fetchPendingAppshot } from "@/lib/appshotApi";
+import { useEventStore } from "@/store/events";
+
+export function useAppshotClaim(attachFiles: (files: File[]) => void, enabled: boolean): void {
+ const t = useT();
+ const pushToast = useEventStore((s) => s.pushToast);
+ const eventId = useEventStore((s) => {
+ const event = s.events.find((item) => item.name === "AppshotTaken");
+ const payload = (event?.payload ?? {}) as { delivered_to?: unknown };
+ return event && payload.delivered_to === "message" ? event.id : "";
+ });
+ const claiming = useRef(false);
+
+ useEffect(() => {
+ if (!enabled || claiming.current) return;
+ claiming.current = true;
+ void (async () => {
+ try {
+ const { appshot } = await fetchPendingAppshot();
+ if (!appshot) return;
+ // Once claimed the backend no longer holds it, so it is attached
+ // even if this effect was superseded meanwhile — never dropped.
+ const file = await claimPendingAppshot(appshot);
+ if (!file) return;
+ attachFiles([file]);
+ pushToast("info", t("appshots.chip_label"));
+ } catch {
+ // Nothing parked or the backend is restarting: the picture stays
+ // where it was, for the next spoken turn to use.
+ } finally {
+ claiming.current = false;
+ }
+ })();
+ }, [enabled, eventId, attachFiles, pushToast, t]);
+}
diff --git a/jarvis/ui/web/frontend/src/components/layout/MainView.tsx b/jarvis/ui/web/frontend/src/components/layout/MainView.tsx
index 495f2f0d7c..bbc3f6d5f6 100644
--- a/jarvis/ui/web/frontend/src/components/layout/MainView.tsx
+++ b/jarvis/ui/web/frontend/src/components/layout/MainView.tsx
@@ -512,6 +512,7 @@ function SwitchOnActiveSection({ active }: { active: string }) {
case "local-models":
case "computers":
case "wallpaper":
+ case "appshots":
case "costs":
case "feedback":
return <SettingsHubView />;
diff --git a/jarvis/ui/web/frontend/src/components/layout/navGroups.ts b/jarvis/ui/web/frontend/src/components/layout/navGroups.ts
index 7d859a7719..bd90915309 100644
--- a/jarvis/ui/web/frontend/src/components/layout/navGroups.ts
+++ b/jarvis/ui/web/frontend/src/components/layout/navGroups.ts
@@ -33,6 +33,7 @@ import {
Wallet,
Workflow,
Image as ImageIcon,
+ ScanLine,
type LucideIcon,
} from "lucide-react";
import { OllamaIcon } from "@/components/icons/OllamaIcon";
@@ -265,6 +266,15 @@ export const NAV_FOOTER_ITEMS: NavItem[] = [
{ id: "feedback", labelKey: "nav.feedback", icon: MessageSquareWarning },
];
+/**
+ * Rows that exist only inside the Settings hub's own navigation, not in the
+ * app sidebar: settings for one feature, reached through Settings the way the
+ * Appshots page is.
+ */
+export const SETTINGS_HUB_ONLY_ITEMS: NavItem[] = [
+ { id: "appshots", labelKey: "nav.appshots", icon: ScanLine, fallbackLabel: "Appshots" },
+];
+
/**
* Every section id rendered inside the Settings hub (`SettingsHubView`).
*
@@ -289,6 +299,7 @@ export const SETTINGS_HUB_IDS: readonly SectionId[] = [
"local-models",
"computers",
"wallpaper",
+ "appshots",
"costs",
"feedback",
];
diff --git a/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx b/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx
index 927ce44ae3..b474f16f9f 100644
--- a/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx
+++ b/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx
@@ -257,13 +257,15 @@ describe("useWebSocket VoiceBootStatus handling", () => {
expect(voiceInputLevelRef.current).toBe(0);
});
- it("shows a metadata-only receipt for one-shot screen capture", async () => {
+ it("shows one appshot receipt per capture and no window title", async () => {
render(<Harness />);
await Promise.resolve();
+ // The capture events themselves stay silent: the desktop overlay flashes
+ // at the shutter, and AppshotTaken is the single in-app receipt.
MockWebSocket.last!.deliver(
envelope("ScreenCaptureAnnounced", {
- target_kind: "monitor",
+ target_kind: "window",
target_label: "private title must not enter the toast",
}),
);
@@ -275,15 +277,38 @@ describe("useWebSocket VoiceBootStatus handling", () => {
target_label: "private title must not enter the toast",
}),
);
+ MockWebSocket.last!.deliver(
+ envelope("AppshotTaken", {
+ appshot_id: "a1",
+ trigger: "hotkey",
+ delivered_to: "message",
+ target_label: "active window",
+ }),
+ );
const toasts = useEventStore.getState().toasts;
- expect(toasts).toHaveLength(2);
- expect(toasts[0].message).toContain("requested surface");
- expect(toasts[1].message).toContain("1600×900");
- expect(toasts[1].message).toContain("3 redaction");
- expect(toasts.map((toast) => toast.message).join(" ")).not.toContain(
- "private title",
+ expect(toasts).toHaveLength(1);
+ expect(toasts[0].kind).toBe("success");
+ expect(toasts[0].message).toContain("Appshot");
+ expect(toasts[0].message).not.toContain("private title");
+ });
+
+ it("says why a shortcut appshot was refused", async () => {
+ render(<Harness />);
+ await Promise.resolve();
+
+ MockWebSocket.last!.deliver(
+ envelope("AppshotTaken", {
+ trigger: "hotkey",
+ delivered_to: "refused",
+ target_label: "Appshots are switched off.",
+ }),
);
+
+ const toasts = useEventStore.getState().toasts;
+ expect(toasts).toHaveLength(1);
+ expect(toasts[0].kind).toBe("warning");
+ expect(toasts[0].message).toContain("Appshots are switched off.");
});
it("stages the newest picture when the brain opens the Visualization section", async () => {
diff --git a/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts b/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts
index 0f969e1f04..e40ad47e8a 100644
--- a/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts
+++ b/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts
@@ -140,29 +140,16 @@ export function useWebSocket(): void {
payload: env.payload,
});
- if (env.event_name === "ScreenCaptureAnnounced") {
- pushToast("info", translate("settings_view.screen_context.capturing"));
- }
-
- if (env.event_name === "ScreenCaptureCompleted") {
- const payload = env.payload as {
- width?: unknown;
- height?: unknown;
- redaction_count?: unknown;
- };
- const width = typeof payload.width === "number" ? payload.width : 0;
- const height = typeof payload.height === "number" ? payload.height : 0;
- const redactions =
- typeof payload.redaction_count === "number"
- ? payload.redaction_count
- : 0;
- pushToast(
- "success",
- translate("settings_view.screen_context.completed")
- .replace("{0}", String(width))
- .replace("{1}", String(height))
- .replace("{2}", String(redactions)),
- );
+ // Every screen look is an appshot to the user: the desktop overlay
+ // flashes at the shutter, and this one line is the in-app receipt.
+ if (env.event_name === "AppshotTaken") {
+ const payload = env.payload as { delivered_to?: unknown; target_label?: unknown };
+ const label = typeof payload.target_label === "string" ? payload.target_label : "";
+ if (payload.delivered_to === "refused") {
+ pushToast("warning", translate("appshots.toast_refused").replace("{0}", label));
+ } else {
+ pushToast("success", translate("appshots.try_done"));
+ }
}
// Live reasoning trace: while the text chat is waiting on a reply,
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/de.json b/jarvis/ui/web/frontend/src/i18n/locales/de.json
index 105db74462..d55aef8298 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/de.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/de.json
@@ -161,6 +161,7 @@
"group_system": "System",
"agent_instructions": "{name}.md",
"wallpaper": "Hintergrund",
+ "appshots": "Appshots",
"marketplace": "Marketplace",
"agentic_ide": "Agentic IDE",
"agentic_ide_beta": "Beta",
@@ -5921,5 +5922,50 @@
"idle": "Im Leerlauf"
}
}
+ },
+ "appshots": {
+ "title": "Appshots",
+ "subtitle": "Zeig {name} das Fenster, in dem du gerade arbeitest.",
+ "hero_title": "Mach einen Appshot, um {name} dein vorderstes Fenster zu zeigen",
+ "hero_body": "Ein Appshot nimmt das Fenster im Vordergrund auf, Bild und sichtbaren Text, und gibt es ins Gespräch. Deine Datenschutz-Regeln gelten weiter, und nichts wird auf der Festplatte gespeichert.",
+ "enabled_label": "Appshots erlauben",
+ "enabled_hint": "Gilt auch für Fragen wie \"Was siehst du?\". Aus heißt: {name} schaut nie auf deinen Bildschirm.",
+ "shortcut_label": "Tastenkürzel",
+ "shortcut_both_alt_hint": "Beide Alt-Tasten gleichzeitig drücken",
+ "shortcut_both_option_hint": "Beide Wahltasten gleichzeitig drücken",
+ "shortcut_combo_hint": "{0} drücken",
+ "shortcut_hint_off": "Kein Tastenkürzel. Frag per Stimme oder nutze den Knopf unten.",
+ "shortcut_off": "Aus",
+ "shortcut_unavailable": "Nicht aktiv: {0}",
+ "target_label": "Appshot-Ziel",
+ "target_auto": "Automatisch",
+ "target_message": "Nächste Nachricht",
+ "target_voice": "Nur Sprachgespräch",
+ "target_auto_hint": "In ein laufendes Sprachgespräch, sonst an deine nächste Nachricht.",
+ "target_message_hint": "Immer an deine nächste gesprochene oder getippte Nachricht.",
+ "target_voice_hint": "Nur in ein laufendes Sprachgespräch. Ohne Gespräch wird er nicht gesendet.",
+ "sound_label": "Soundeffekt abspielen",
+ "sound_master_off": "Soundeffekte sind für die ganze App unter Einstellungen ausgeschaltet.",
+ "effect_label": "Blitz zeigen",
+ "effect_hint": "Ein kurzer Blitz und eine Mini-Vorschau in der Bildschirmecke.",
+ "effect_unavailable": "Hier nicht verfügbar: {0}",
+ "try_label": "Ausprobieren",
+ "try_hint": "Macht nach 3 Sekunden einen Appshot vom Fenster im Vordergrund. Wechsle jetzt dorthin.",
+ "try_counting": "Wechsle zu deinem Fenster: {0}",
+ "try_button": "Appshot in 3 s",
+ "try_done": "Appshot aufgenommen",
+ "preview_title": "Letzter Appshot",
+ "preview_empty": "Dein letzter Appshot erscheint hier für zwei Minuten, danach ist er weg.",
+ "preview_front_window": "Vorderstes Fenster",
+ "preview_alt": "Appshot von {0}",
+ "preview_sent_voice": "Ins Sprachgespräch gegeben",
+ "preview_sent_message": "Wartet auf deine nächste Nachricht",
+ "preview_sent_turn": "Im Gespräch verwendet",
+ "preview_sent_none": "Nicht gesendet, es lief kein Sprachgespräch",
+ "forget": "Vergessen",
+ "voice_hint": "Du kannst auch einfach sagen: \"Mach einen Appshot\".",
+ "save_error": "Speichern fehlgeschlagen: {0}",
+ "toast_refused": "Kein Appshot: {0}",
+ "chip_label": "Appshot an deine Nachricht gehängt"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/en.json b/jarvis/ui/web/frontend/src/i18n/locales/en.json
index da2a97655f..5253983ce4 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/en.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/en.json
@@ -161,6 +161,7 @@
"group_system": "System",
"agent_instructions": "{name}.md",
"wallpaper": "Wallpaper",
+ "appshots": "Appshots",
"marketplace": "Marketplace",
"agentic_ide": "Agentic IDE",
"agentic_ide_beta": "Beta",
@@ -5954,5 +5955,50 @@
"idle": "Idle"
}
}
+ },
+ "appshots": {
+ "title": "Appshots",
+ "subtitle": "Show {name} the window you are working in.",
+ "hero_title": "Take an appshot to show {name} your front window",
+ "hero_body": "An appshot captures the window in front, picture and on-screen text, and hands it to the conversation. Your privacy rules still apply, and nothing is saved to disk.",
+ "enabled_label": "Allow appshots",
+ "enabled_hint": "Also covers questions like \"what do you see?\". Off means {name} never looks at your screen.",
+ "shortcut_label": "Keyboard shortcut",
+ "shortcut_both_alt_hint": "Press both Alt keys at the same time",
+ "shortcut_both_option_hint": "Press both Option keys at the same time",
+ "shortcut_combo_hint": "Press {0}",
+ "shortcut_hint_off": "No shortcut. Ask by voice or use the button below.",
+ "shortcut_off": "Off",
+ "shortcut_unavailable": "Not active: {0}",
+ "target_label": "Appshot destination",
+ "target_auto": "Automatic",
+ "target_message": "Next message",
+ "target_voice": "Voice call only",
+ "target_auto_hint": "Into a running voice call, otherwise onto your next message.",
+ "target_message_hint": "Always onto your next spoken or typed message.",
+ "target_voice_hint": "Only into a running voice call. Without one, it is not sent.",
+ "sound_label": "Play sound effect",
+ "sound_master_off": "Sound effects are off for the whole app under Settings.",
+ "effect_label": "Show the flash",
+ "effect_hint": "A short flash and a thumbnail in the corner of your screen.",
+ "effect_unavailable": "Not available here: {0}",
+ "try_label": "Try it",
+ "try_hint": "Takes an appshot of the window in front after 3 seconds. Switch to it now.",
+ "try_counting": "Switch to your window: {0}",
+ "try_button": "Take appshot in 3 s",
+ "try_done": "Appshot taken",
+ "preview_title": "Last appshot",
+ "preview_empty": "Your last appshot shows up here for two minutes, then it is gone.",
+ "preview_front_window": "Front window",
+ "preview_alt": "Appshot of {0}",
+ "preview_sent_voice": "Sent into the voice call",
+ "preview_sent_message": "Waiting for your next message",
+ "preview_sent_turn": "Used in the conversation",
+ "preview_sent_none": "Not sent, no voice call was running",
+ "forget": "Forget",
+ "voice_hint": "You can also just say \"take an appshot\".",
+ "save_error": "Could not save: {0}",
+ "toast_refused": "No appshot: {0}",
+ "chip_label": "Appshot added to your message"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/es.json b/jarvis/ui/web/frontend/src/i18n/locales/es.json
index 0b7b0c93cb..f250fb16a8 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/es.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/es.json
@@ -161,6 +161,7 @@
"group_system": "Sistema",
"agent_instructions": "{name}.md",
"wallpaper": "Fondo de pantalla",
+ "appshots": "Appshots",
"marketplace": "Marketplace",
"agentic_ide": "IDE agéntico",
"agentic_ide_beta": "Beta",
@@ -5921,5 +5922,50 @@
"idle": "Inactivo"
}
}
+ },
+ "appshots": {
+ "title": "Appshots",
+ "subtitle": "Muestra a {name} la ventana en la que trabajas.",
+ "hero_title": "Haz un appshot para mostrar a {name} tu ventana principal",
+ "hero_body": "Un appshot captura la ventana en primer plano, imagen y texto visible, y la pasa a la conversación. Tus reglas de privacidad siguen aplicándose y no se guarda nada en el disco.",
+ "enabled_label": "Permitir appshots",
+ "enabled_hint": "También cubre preguntas como \"¿qué ves?\". Desactivado significa que {name} nunca mira tu pantalla.",
+ "shortcut_label": "Atajo de teclado",
+ "shortcut_both_alt_hint": "Pulsa las dos teclas Alt a la vez",
+ "shortcut_both_option_hint": "Pulsa las dos teclas Opción a la vez",
+ "shortcut_combo_hint": "Pulsa {0}",
+ "shortcut_hint_off": "Sin atajo. Pídelo por voz o usa el botón de abajo.",
+ "shortcut_off": "Desactivado",
+ "shortcut_unavailable": "No activo: {0}",
+ "target_label": "Destino del appshot",
+ "target_auto": "Automático",
+ "target_message": "Siguiente mensaje",
+ "target_voice": "Solo llamada de voz",
+ "target_auto_hint": "A una llamada de voz en curso; si no hay, a tu siguiente mensaje.",
+ "target_message_hint": "Siempre a tu siguiente mensaje, hablado o escrito.",
+ "target_voice_hint": "Solo a una llamada de voz en curso. Sin llamada, no se envía.",
+ "sound_label": "Reproducir efecto de sonido",
+ "sound_master_off": "Los efectos de sonido están desactivados para toda la app en Ajustes.",
+ "effect_label": "Mostrar el destello",
+ "effect_hint": "Un destello breve y una miniatura en la esquina de la pantalla.",
+ "effect_unavailable": "No disponible aquí: {0}",
+ "try_label": "Probar",
+ "try_hint": "Hace un appshot de la ventana en primer plano tras 3 segundos. Cambia a ella ahora.",
+ "try_counting": "Cambia a tu ventana: {0}",
+ "try_button": "Appshot en 3 s",
+ "try_done": "Appshot hecho",
+ "preview_title": "Último appshot",
+ "preview_empty": "Tu último appshot aparece aquí durante dos minutos y luego desaparece.",
+ "preview_front_window": "Ventana principal",
+ "preview_alt": "Appshot de {0}",
+ "preview_sent_voice": "Enviado a la llamada de voz",
+ "preview_sent_message": "Esperando tu siguiente mensaje",
+ "preview_sent_turn": "Usado en la conversación",
+ "preview_sent_none": "No enviado, no había llamada de voz",
+ "forget": "Olvidar",
+ "voice_hint": "También puedes decir \"haz un appshot\".",
+ "save_error": "No se pudo guardar: {0}",
+ "toast_refused": "Sin appshot: {0}",
+ "chip_label": "Appshot añadido a tu mensaje"
}
}
diff --git a/jarvis/ui/web/frontend/src/index.css b/jarvis/ui/web/frontend/src/index.css
index 374624a70d..91d37853dd 100644
--- a/jarvis/ui/web/frontend/src/index.css
+++ b/jarvis/ui/web/frontend/src/index.css
@@ -237,6 +237,61 @@ label,
}
}
+ /* Appshots page demo: the same shutter the desktop overlay plays — a white
+ flash, then the picture shrinks into the corner, rests, and slides out. */
+ .appshot-demo-flash {
+ animation: appshot-demo-flash 4.8s ease-out infinite;
+ }
+
+ .appshot-demo-card {
+ transform-origin: 100% 100%;
+ animation: appshot-demo-card 4.8s cubic-bezier(0.22, 1, 0.36, 1) infinite;
+ }
+
+ @media (prefers-reduced-motion: reduce) {
+ .appshot-demo-flash,
+ .appshot-demo-card {
+ animation: none;
+ opacity: 0;
+ }
+ }
+
+ @keyframes appshot-demo-flash {
+ 0%,
+ 6% {
+ opacity: 0;
+ }
+ 9% {
+ opacity: 0.85;
+ }
+ 22%,
+ 100% {
+ opacity: 0;
+ }
+ }
+
+ @keyframes appshot-demo-card {
+ 0%,
+ 8% {
+ opacity: 0;
+ transform: translate(0, 0) scale(1);
+ }
+ 10% {
+ opacity: 1;
+ transform: translate(0, 0) scale(1);
+ }
+ 28%,
+ 78% {
+ opacity: 1;
+ transform: translate(-6%, -7%) scale(0.34);
+ }
+ 90%,
+ 100% {
+ opacity: 0;
+ transform: translate(4%, -7%) scale(0.34);
+ }
+ }
+
@keyframes agentic-transcript-in {
from {
opacity: 0;
diff --git a/jarvis/ui/web/frontend/src/lib/appshotApi.ts b/jarvis/ui/web/frontend/src/lib/appshotApi.ts
new file mode 100644
index 0000000000..3233d009a7
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/appshotApi.ts
@@ -0,0 +1,132 @@
+/**
+ * Client for `/api/appshot` — the front window as conversation context.
+ *
+ * Pixels only ever come from the two image endpoints, which the backend serves
+ * with `Cache-Control: no-store`; nothing here keeps a copy.
+ */
+
+export type AppshotTarget = "auto" | "message" | "voice";
+
+export interface AppshotShortcutStatus {
+ hotkey: string;
+ armed: boolean;
+ detail: string;
+}
+
+export interface AppshotSettings {
+ enabled: boolean;
+ hotkey: string;
+ target: AppshotTarget;
+ sound: boolean;
+ effect: boolean;
+ sound_effects_master: boolean;
+ shortcut: AppshotShortcutStatus;
+ readiness: {
+ capture: boolean;
+ capture_detail: string;
+ effect: boolean;
+ effect_detail: string;
+ };
+}
+
+export interface AppshotMeta {
+ id: string;
+ width: number;
+ height: number;
+ label: string;
+ app_name: string;
+ trigger: string;
+ taken_at: number;
+ delivered_to: string;
+}
+
+export type AppshotSettingsPatch = Partial<
+ Pick<AppshotSettings, "enabled" | "hotkey" | "target" | "sound" | "effect">
+>;
+
+async function request<T>(url: string, init?: RequestInit): Promise<T> {
+ const response = await fetch(url, init);
+ const body = (await response.json().catch(() => null)) as T | { detail?: unknown } | null;
+ if (!response.ok) {
+ const detail =
+ body && typeof body === "object" && "detail" in body && typeof body.detail === "string"
+ ? body.detail
+ : "";
+ throw new Error(detail || `Appshot request failed (${response.status}).`);
+ }
+ return body as T;
+}
+
+export function fetchAppshotSettings(): Promise<AppshotSettings> {
+ return request<AppshotSettings>("/api/appshot/settings");
+}
+
+export function saveAppshotSettings(patch: AppshotSettingsPatch): Promise<AppshotSettings> {
+ return request<AppshotSettings>("/api/appshot/settings", {
+ method: "PUT",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify(patch),
+ });
+}
+
+export function fetchLatestAppshot(): Promise<{ appshot: AppshotMeta | null }> {
+ return request<{ appshot: AppshotMeta | null }>("/api/appshot/latest");
+}
+
+export function latestAppshotImageUrl(id: string): string {
+ // The id only busts the <img> cache between appshots; the server keeps one.
+ return `/api/appshot/latest/image?v=${encodeURIComponent(id)}`;
+}
+
+export function takeAppshot(delaySeconds = 0): Promise<
+ { ok: true; appshot: AppshotMeta } | { ok: false; reason: string; message: string }
+> {
+ return request("/api/appshot/take", {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify({ delay_s: delaySeconds }),
+ });
+}
+
+export function forgetAppshots(): Promise<{ ok: boolean }> {
+ return request<{ ok: boolean }>("/api/appshot", { method: "DELETE" });
+}
+
+export function fetchPendingAppshot(): Promise<{ appshot: AppshotMeta | null }> {
+ return request<{ appshot: AppshotMeta | null }>("/api/appshot/pending");
+}
+
+/** Take the waiting appshot out of the backend, as a file for the composer. */
+export async function claimPendingAppshot(meta: AppshotMeta): Promise<File | null> {
+ const response = await fetch("/api/appshot/pending/claim", {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify({ id: meta.id }),
+ });
+ if (!response.ok) return null;
+ const blob = await response.blob();
+ const stamp = new Date(meta.taken_at * 1000)
+ .toISOString()
+ .replace(/[-:]/g, "")
+ .replace(/\..*$/, "")
+ .replace("T", "-");
+ const extension = blob.type === "image/png" ? "png" : "jpg";
+ return new File([blob], `appshot-${stamp}.${extension}`, { type: blob.type || "image/jpeg" });
+}
+
+/** "alt+alt" → "Alt + Alt" (⌥ on a Mac); any other combo, title-cased. */
+export function formatAppshotHotkey(hotkey: string, isMac: boolean): string {
+ if (!hotkey) return "";
+ if (hotkey === "alt+alt") return isMac ? "⌥ + ⌥" : "Alt + Alt";
+ return hotkey
+ .split("+")
+ .map((part) => {
+ const key = part.trim();
+ if (key === "ctrl") return isMac ? "⌃" : "Ctrl";
+ if (key === "alt") return isMac ? "⌥" : "Alt";
+ if (key === "shift") return isMac ? "⇧" : "Shift";
+ if (key === "win" || key === "cmd" || key === "super") return isMac ? "⌘" : "Win";
+ return key.length === 1 ? key.toUpperCase() : key.charAt(0).toUpperCase() + key.slice(1);
+ })
+ .join(" + ");
+}
diff --git a/jarvis/ui/web/frontend/src/store/events.ts b/jarvis/ui/web/frontend/src/store/events.ts
index e76d98bc06..811a0dabce 100644
--- a/jarvis/ui/web/frontend/src/store/events.ts
+++ b/jarvis/ui/web/frontend/src/store/events.ts
@@ -85,6 +85,9 @@ export type SectionId =
| "feedback"
| "agent-instructions"
| "wallpaper"
+ // Appshots: the shortcut, destination, sound and flash for showing the
+ // assistant the front window. A Settings-hub page.
+ | "appshots"
| "dictionary"
| "dictation"
// The three tabs added by the merged voice section. "dictation" (default
@@ -139,6 +142,7 @@ export const SECTION_IDS = [
"feedback",
"agent-instructions",
"wallpaper",
+ "appshots",
"dictionary",
"dictation",
// `satisfies` only catches array entries that are missing from the union,
@@ -229,6 +233,7 @@ export const SECTION_LABELS: Record<SectionId, string> = {
feedback: "Feedback",
"agent-instructions": "Agent Instructions",
wallpaper: "Wallpaper",
+ appshots: "Appshots",
dictionary: "Dictionary",
dictation: "Dictation",
// Plain English, deliberately NOT the "{name} Voice" brand: these labels are
diff --git a/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx b/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx
new file mode 100644
index 0000000000..49c92c4e45
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx
@@ -0,0 +1,93 @@
+import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+import { formatAppshotHotkey } from "@/lib/appshotApi";
+import { useEventStore } from "@/store/events";
+import { AppshotsView } from "@/views/AppshotsView";
+
+const SETTINGS = {
+ enabled: true,
+ hotkey: "alt+alt",
+ target: "auto",
+ sound: true,
+ effect: true,
+ sound_effects_master: true,
+ shortcut: { hotkey: "alt+alt", armed: true, detail: "" },
+ readiness: { capture: true, capture_detail: "", effect: true, effect_detail: "" },
+};
+
+function json(body: unknown, status = 200): Response {
+ return { ok: status < 400, status, json: async () => body } as Response;
+}
+
+describe("AppshotsView", () => {
+ let fetchMock: ReturnType<typeof vi.fn>;
+
+ beforeEach(() => {
+ useEventStore.setState({ events: [], toasts: [], assistantName: "Jarvis" });
+ fetchMock = vi.fn(async (url: string, init?: RequestInit) => {
+ if (url === "/api/appshot/settings" && init?.method === "PUT") {
+ return json({ ...SETTINGS, ...JSON.parse(String(init.body)) });
+ }
+ if (url === "/api/appshot/settings") return json(SETTINGS);
+ if (url === "/api/appshot/latest") return json({ appshot: null });
+ return json({}, 404);
+ });
+ vi.stubGlobal("fetch", fetchMock);
+ });
+
+ afterEach(() => {
+ cleanup();
+ vi.unstubAllGlobals();
+ });
+
+ it("shows the switches once the settings arrive", async () => {
+ render(<AppshotsView />);
+ await waitFor(() => expect(screen.getByTestId("appshots-enabled")).toBeDefined());
+ expect(screen.getByTestId("appshots-sound").getAttribute("data-state")).toBe("checked");
+ expect(screen.getByTestId("appshots-effect").getAttribute("data-state")).toBe("checked");
+ expect(screen.getByTestId("appshots-try")).toBeDefined();
+ });
+
+ it("saves one switch with a PUT of just that key", async () => {
+ render(<AppshotsView />);
+ const sound = await screen.findByTestId("appshots-sound");
+ fireEvent.click(sound);
+ await waitFor(() =>
+ expect(
+ fetchMock.mock.calls.some(
+ ([url, init]) =>
+ url === "/api/appshot/settings" &&
+ init?.method === "PUT" &&
+ JSON.parse(String(init.body)).sound === false,
+ ),
+ ).toBe(true),
+ );
+ const put = fetchMock.mock.calls.find(([, init]) => init?.method === "PUT")!;
+ expect(Object.keys(JSON.parse(String(put[1].body)))).toEqual(["sound"]);
+ });
+
+ it("disables every other control while appshots are switched off", async () => {
+ fetchMock.mockImplementation(async (url: string) =>
+ url === "/api/appshot/settings"
+ ? json({ ...SETTINGS, enabled: false })
+ : json({ appshot: null }),
+ );
+ render(<AppshotsView />);
+ const tryButton = await screen.findByTestId("appshots-try");
+ expect((tryButton as HTMLButtonElement).disabled).toBe(true);
+ expect(screen.getByTestId("appshots-sound").hasAttribute("disabled")).toBe(true);
+ });
+});
+
+describe("formatAppshotHotkey", () => {
+ it("names the both-Alt gesture per platform", () => {
+ expect(formatAppshotHotkey("alt+alt", false)).toBe("Alt + Alt");
+ expect(formatAppshotHotkey("alt+alt", true)).toBe("⌥ + ⌥");
+ });
+
+ it("title-cases an ordinary combo", () => {
+ expect(formatAppshotHotkey("ctrl+alt+a", false)).toBe("Ctrl + Alt + A");
+ expect(formatAppshotHotkey("", false)).toBe("");
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/views/AppshotsView.tsx b/jarvis/ui/web/frontend/src/views/AppshotsView.tsx
new file mode 100644
index 0000000000..d5cc2b9ba8
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/AppshotsView.tsx
@@ -0,0 +1,437 @@
+import { useCallback, useEffect, useMemo, useRef, useState } from "react";
+import { Loader2, Trash2 } from "lucide-react";
+
+import { PageHeader } from "@/components/layout/PageHeader";
+import { Button } from "@/components/ui/button";
+import { BrandedSelect, type BrandedSelectOption } from "@/components/ui/select";
+import { Switch } from "@/components/ui/switch";
+import { useT } from "@/i18n";
+import {
+ fetchAppshotSettings,
+ fetchLatestAppshot,
+ forgetAppshots,
+ formatAppshotHotkey,
+ latestAppshotImageUrl,
+ saveAppshotSettings,
+ takeAppshot,
+ type AppshotMeta,
+ type AppshotSettings,
+ type AppshotSettingsPatch,
+} from "@/lib/appshotApi";
+import { cn } from "@/lib/utils";
+import { useEventStore } from "@/store/events";
+
+/**
+ * Appshots — show the assistant the window you are working in.
+ *
+ * One page for the whole feature: the master switch (it is also the switch
+ * for every other screen look, `[screen_context].enabled`), the global
+ * shortcut, where a shortcut appshot goes, the sound and the flash, a try-it
+ * button, and the last appshot so the user sees exactly what was handed over.
+ * That picture lives in backend memory for `deck_preview_s` and is fetched
+ * with `no-store`; this view keeps no copy.
+ */
+
+const IS_MAC = typeof navigator !== "undefined" && /Mac/i.test(navigator.platform || "");
+const PRESET_HOTKEYS = ["alt+alt", "ctrl+alt+a", ""] as const;
+const TRY_DELAY_S = 3;
+
+export function AppshotGlyph({ className }: { className?: string }) {
+ // Four camera-frame corners around a pill: "this window, captured".
+ return (
+ <svg viewBox="0 0 32 32" fill="none" aria-hidden className={className}>
+ <path
+ d="M4 11V8a4 4 0 0 1 4-4h3M21 4h3a4 4 0 0 1 4 4v3M28 21v3a4 4 0 0 1-4 4h-3M11 28H8a4 4 0 0 1-4-4v-3"
+ stroke="currentColor"
+ strokeWidth="2.6"
+ strokeLinecap="round"
+ />
+ <rect x="9.5" y="13" width="13" height="6" rx="3" fill="currentColor" />
+ </svg>
+ );
+}
+
+function Row({
+ label,
+ hint,
+ control,
+ children,
+}: {
+ label: string;
+ hint?: string;
+ control: React.ReactNode;
+ children?: React.ReactNode;
+}) {
+ return (
+ <div className="px-5 py-4">
+ <div className="flex items-center justify-between gap-6">
+ <div className="min-w-0">
+ <p className="text-base font-medium text-foreground">{label}</p>
+ {hint && <p className="mt-0.5 text-sm text-muted-foreground">{hint}</p>}
+ </div>
+ <div className="shrink-0">{control}</div>
+ </div>
+ {children}
+ </div>
+ );
+}
+
+function DemoWindow({ className }: { className?: string }) {
+ // A theme-token mock of an app window; no screenshot of anyone's screen.
+ return (
+ <div className={cn("flex flex-col overflow-hidden rounded-lg border border-border bg-card shadow-sm", className)}>
+ <div className="flex items-center gap-1.5 border-b border-border px-3 py-2">
+ <span className="h-2 w-2 rounded-full bg-foreground/20" />
+ <span className="h-2 w-2 rounded-full bg-foreground/20" />
+ <span className="h-2 w-2 rounded-full bg-foreground/20" />
+ <span className="ml-3 h-2 w-1/3 rounded-full bg-foreground/10" />
+ </div>
+ <div className="flex flex-1 gap-3 p-3">
+ <div className="w-1/4 space-y-2">
+ {[0, 1, 2, 3].map((i) => (
+ <span key={i} className="block h-2 rounded-full bg-foreground/10" />
+ ))}
+ </div>
+ <div className="flex-1 space-y-2">
+ <span className="block h-3 w-1/2 rounded-full bg-foreground/15" />
+ {[0, 1, 2, 3, 4].map((i) => (
+ <span key={i} className={cn("block h-2 rounded-full bg-foreground/10", i % 2 ? "w-4/5" : "w-full")} />
+ ))}
+ <span className="mt-3 block h-14 rounded-md bg-accent/15" />
+ </div>
+ </div>
+ </div>
+ );
+}
+
+function PreviewDemo() {
+ const t = useT();
+ return (
+ <div className="flex h-full flex-col">
+ <div className="relative aspect-[16/10] w-full overflow-hidden rounded-lg bg-secondary/60 p-5">
+ <DemoWindow className="h-full w-full" />
+ <div className="appshot-demo-card pointer-events-none absolute inset-5">
+ <DemoWindow className="h-full w-full ring-4 ring-white/90" />
+ </div>
+ <div className="appshot-demo-flash pointer-events-none absolute inset-5 rounded-lg bg-white" />
+ </div>
+ <p className="mt-3 text-sm text-muted-foreground">{t("appshots.preview_empty")}</p>
+ </div>
+ );
+}
+
+function deliveredLabel(t: (key: string) => string, deliveredTo: string): string {
+ switch (deliveredTo) {
+ case "voice":
+ return t("appshots.preview_sent_voice");
+ case "message":
+ return t("appshots.preview_sent_message");
+ case "turn":
+ return t("appshots.preview_sent_turn");
+ case "none":
+ return t("appshots.preview_sent_none");
+ default:
+ return "";
+ }
+}
+
+function LatestPreview({ shot, onForget }: { shot: AppshotMeta; onForget: () => void }) {
+ const t = useT();
+ const time = new Date(shot.taken_at * 1000).toLocaleTimeString([], {
+ hour: "2-digit",
+ minute: "2-digit",
+ });
+ const where = shot.app_name || t("appshots.preview_front_window");
+ const delivered = deliveredLabel(t, shot.delivered_to);
+ return (
+ <div className="flex h-full flex-col">
+ <div className="flex aspect-[16/10] w-full items-center justify-center overflow-hidden rounded-lg bg-secondary/60 p-3">
+ <img
+ src={latestAppshotImageUrl(shot.id)}
+ alt={t("appshots.preview_alt").replace("{0}", where)}
+ className="max-h-full max-w-full rounded-md object-contain shadow-sm ring-1 ring-border"
+ />
+ </div>
+ <div className="mt-3 flex items-start justify-between gap-3">
+ <div className="min-w-0">
+ <p className="truncate text-base font-medium text-foreground">{where}</p>
+ <p className="text-sm text-muted-foreground">
+ {[time, `${shot.width} × ${shot.height}`, delivered].filter(Boolean).join(" · ")}
+ </p>
+ </div>
+ <Button type="button" variant="ghost" size="sm" onClick={onForget}>
+ <Trash2 aria-hidden />
+ {t("appshots.forget")}
+ </Button>
+ </div>
+ </div>
+ );
+}
+
+export function AppshotsView() {
+ const t = useT();
+ const pushToast = useEventStore((s) => s.pushToast);
+ const lastAppshotEvent = useEventStore(
+ (s) => s.events.find((event) => event.name === "AppshotTaken")?.id ?? "",
+ );
+ const [settings, setSettings] = useState<AppshotSettings | null>(null);
+ const [latest, setLatest] = useState<AppshotMeta | null>(null);
+ const [saving, setSaving] = useState(false);
+ const [countdown, setCountdown] = useState<number | null>(null);
+ const countdownTimer = useRef<number | null>(null);
+
+ useEffect(() => {
+ let active = true;
+ fetchAppshotSettings()
+ .then((next) => active && setSettings(next))
+ .catch((error: Error) => pushToast("error", error.message));
+ return () => {
+ active = false;
+ };
+ }, [pushToast]);
+
+ // Refetch the last appshot whenever the backend reports a new one.
+ useEffect(() => {
+ let active = true;
+ fetchLatestAppshot()
+ .then((body) => active && setLatest(body.appshot))
+ .catch(() => active && setLatest(null));
+ return () => {
+ active = false;
+ };
+ }, [lastAppshotEvent]);
+
+ useEffect(
+ () => () => {
+ if (countdownTimer.current !== null) window.clearInterval(countdownTimer.current);
+ },
+ [],
+ );
+
+ const save = useCallback(
+ async (patch: AppshotSettingsPatch) => {
+ setSaving(true);
+ setSettings((current) => (current ? { ...current, ...patch } : current));
+ try {
+ setSettings(await saveAppshotSettings(patch));
+ } catch (error) {
+ pushToast("error", t("appshots.save_error").replace("{0}", (error as Error).message));
+ fetchAppshotSettings().then(setSettings).catch(() => undefined);
+ } finally {
+ setSaving(false);
+ }
+ },
+ [pushToast, t],
+ );
+
+ const tryIt = useCallback(async () => {
+ if (countdown !== null) return;
+ setCountdown(TRY_DELAY_S);
+ countdownTimer.current = window.setInterval(() => {
+ setCountdown((value) => (value !== null && value > 1 ? value - 1 : value));
+ }, 1000);
+ try {
+ const result = await takeAppshot(TRY_DELAY_S);
+ if (result.ok) {
+ setLatest(result.appshot);
+ pushToast("success", t("appshots.try_done"));
+ } else {
+ pushToast("warning", t("appshots.toast_refused").replace("{0}", result.message));
+ }
+ } catch (error) {
+ pushToast("error", (error as Error).message);
+ } finally {
+ if (countdownTimer.current !== null) window.clearInterval(countdownTimer.current);
+ countdownTimer.current = null;
+ setCountdown(null);
+ }
+ }, [countdown, pushToast, t]);
+
+ const forget = useCallback(async () => {
+ try {
+ await forgetAppshots();
+ setLatest(null);
+ } catch (error) {
+ pushToast("error", (error as Error).message);
+ }
+ }, [pushToast]);
+
+ const hotkeyOptions = useMemo<BrandedSelectOption[]>(() => {
+ const options: BrandedSelectOption[] = PRESET_HOTKEYS.map((value) => ({
+ value: value || "off",
+ label: value ? formatAppshotHotkey(value, IS_MAC) : t("appshots.shortcut_off"),
+ }));
+ const current = settings?.hotkey ?? "";
+ if (current && !PRESET_HOTKEYS.includes(current as (typeof PRESET_HOTKEYS)[number])) {
+ options.unshift({ value: current, label: formatAppshotHotkey(current, IS_MAC) });
+ }
+ return options;
+ }, [settings?.hotkey, t]);
+
+ const targetOptions = useMemo<BrandedSelectOption[]>(
+ () => [
+ { value: "auto", label: t("appshots.target_auto") },
+ { value: "message", label: t("appshots.target_message") },
+ { value: "voice", label: t("appshots.target_voice") },
+ ],
+ [t],
+ );
+
+ const shortcutHint = (() => {
+ if (!settings) return "";
+ if (!settings.hotkey) return t("appshots.shortcut_hint_off");
+ if (settings.enabled && !settings.shortcut.armed && settings.shortcut.detail) {
+ return t("appshots.shortcut_unavailable").replace("{0}", settings.shortcut.detail);
+ }
+ if (settings.hotkey === "alt+alt") {
+ return IS_MAC ? t("appshots.shortcut_both_option_hint") : t("appshots.shortcut_both_alt_hint");
+ }
+ return t("appshots.shortcut_combo_hint").replace(
+ "{0}",
+ formatAppshotHotkey(settings.hotkey, IS_MAC),
+ );
+ })();
+
+ const targetHint =
+ settings?.target === "message"
+ ? t("appshots.target_message_hint")
+ : settings?.target === "voice"
+ ? t("appshots.target_voice_hint")
+ : t("appshots.target_auto_hint");
+
+ const disabled = !settings || !settings.enabled;
+
+ return (
+ <div data-testid="appshots-view" className="flex h-full flex-col overflow-y-auto bg-background px-8 pb-10 scrollbar-jarvis">
+ <PageHeader
+ icon={<AppshotGlyph />}
+ title={t("appshots.title")}
+ description={t("appshots.subtitle")}
+ />
+
+ <div className="flex items-start gap-4 rounded-xl border border-border bg-card p-5">
+ <AppshotGlyph className="mt-0.5 h-8 w-8 shrink-0 text-accent" />
+ <div className="min-w-0">
+ <p className="text-lg font-semibold text-foreground">{t("appshots.hero_title")}</p>
+ <p className="mt-1 text-base text-muted-foreground">{t("appshots.hero_body")}</p>
+ </div>
+ </div>
+
+ <div className="mt-5 grid gap-5 lg:grid-cols-2">
+ <div className="divide-y divide-border self-start rounded-xl border border-border bg-card">
+ {!settings ? (
+ <div className="flex h-40 items-center justify-center" role="status" aria-busy="true">
+ <Loader2 className="h-5 w-5 animate-spin text-muted-foreground" aria-hidden />
+ </div>
+ ) : (
+ <>
+ <Row
+ label={t("appshots.enabled_label")}
+ hint={t("appshots.enabled_hint")}
+ control={
+ <Switch
+ checked={settings.enabled}
+ disabled={saving}
+ aria-label={t("appshots.enabled_label")}
+ data-testid="appshots-enabled"
+ onCheckedChange={(enabled) => void save({ enabled })}
+ />
+ }
+ />
+ <Row
+ label={t("appshots.shortcut_label")}
+ hint={shortcutHint}
+ control={
+ <BrandedSelect
+ value={settings.hotkey || "off"}
+ options={hotkeyOptions}
+ ariaLabel={t("appshots.shortcut_label")}
+ disabled={disabled || saving}
+ testId="appshots-hotkey"
+ className="w-44"
+ onValueChange={(value) => void save({ hotkey: value === "off" ? "" : value })}
+ />
+ }
+ />
+ <Row
+ label={t("appshots.target_label")}
+ hint={targetHint}
+ control={
+ <BrandedSelect
+ value={settings.target}
+ options={targetOptions}
+ ariaLabel={t("appshots.target_label")}
+ disabled={disabled || saving}
+ testId="appshots-target"
+ className="w-44"
+ onValueChange={(value) =>
+ void save({ target: value as AppshotSettings["target"] })
+ }
+ />
+ }
+ />
+ <Row
+ label={t("appshots.sound_label")}
+ hint={settings.sound_effects_master ? undefined : t("appshots.sound_master_off")}
+ control={
+ <Switch
+ checked={settings.sound}
+ disabled={disabled || saving}
+ aria-label={t("appshots.sound_label")}
+ data-testid="appshots-sound"
+ onCheckedChange={(sound) => void save({ sound })}
+ />
+ }
+ />
+ <Row
+ label={t("appshots.effect_label")}
+ hint={
+ settings.readiness.effect
+ ? t("appshots.effect_hint")
+ : t("appshots.effect_unavailable").replace("{0}", settings.readiness.effect_detail)
+ }
+ control={
+ <Switch
+ checked={settings.effect}
+ disabled={disabled || saving}
+ aria-label={t("appshots.effect_label")}
+ data-testid="appshots-effect"
+ onCheckedChange={(effect) => void save({ effect })}
+ />
+ }
+ />
+ <Row
+ label={t("appshots.try_label")}
+ hint={
+ countdown !== null
+ ? t("appshots.try_counting").replace("{0}", String(countdown))
+ : t("appshots.try_hint")
+ }
+ control={
+ <Button
+ type="button"
+ variant="secondary"
+ size="sm"
+ disabled={disabled || countdown !== null}
+ onClick={() => void tryIt()}
+ data-testid="appshots-try"
+ >
+ {countdown !== null && <Loader2 className="animate-spin" aria-hidden />}
+ {t("appshots.try_button")}
+ </Button>
+ }
+ />
+ </>
+ )}
+ </div>
+
+ <div className="rounded-xl border border-border bg-card p-5">
+ <p className="mb-3 text-base font-medium text-foreground">{t("appshots.preview_title")}</p>
+ {latest ? <LatestPreview shot={latest} onForget={() => void forget()} /> : <PreviewDemo />}
+ </div>
+ </div>
+
+ <p className="mt-5 text-sm text-muted-foreground">{t("appshots.voice_hint")}</p>
+ </div>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx b/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx
index 558b442d2a..ab4993964e 100644
--- a/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx
+++ b/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx
@@ -3,6 +3,7 @@ import { ArrowLeft, Loader2, Search, Settings as SettingsIcon, X } from "lucide-
import {
NAV_FOOTER_ITEMS,
NAV_GROUPS,
+ SETTINGS_HUB_ONLY_ITEMS,
resolveNavLabel,
type NavItem,
} from "@/components/layout/navGroups";
@@ -18,7 +19,7 @@ import { cn } from "@/lib/utils";
* searchable left navigation (Personal · System · Activity) and the selected
* section on the right:
*
- * General: Settings, Profile, {name}.md, Contacts, Socials
+ * General: Settings, Appshots, Profile, {name}.md, Contacts, Socials
* System: Computers, API Keys, Local models, Wallpaper
* Activity: Spend, Feedback
*
@@ -77,6 +78,9 @@ const ComputersTab = lazy(() =>
const WallpaperTab = lazy(() =>
import("@/views/WallpaperView").then((m) => ({ default: m.WallpaperView })),
);
+const AppshotsTab = lazy(() =>
+ import("@/views/AppshotsView").then((m) => ({ default: m.AppshotsView })),
+);
const CostsTab = lazy(() =>
import("@/views/CostsView").then((m) => ({ default: m.CostsView })),
);
@@ -89,6 +93,7 @@ const FeedbackTab = lazy(() =>
/** The eleven entries of the left navigation, in display order. */
type HubNavId =
| "settings"
+ | "appshots"
| "profile"
| "agent-instructions"
| "contacts"
@@ -103,7 +108,7 @@ type HubNavId =
const HUB_NAV_GROUPS: readonly { labelKey: string; ids: readonly HubNavId[] }[] = [
{
labelKey: "settings_hub.group_general",
- ids: ["settings", "profile", "agent-instructions", "contacts", "socials"],
+ ids: ["settings", "appshots", "profile", "agent-instructions", "contacts", "socials"],
},
{
labelKey: "settings_hub.group_system",
@@ -126,6 +131,7 @@ const TAB_CONTENT: Record<HubNavId | "telephony-setup", LazyExoticComponent<Comp
"local-models": LocalModelsTab,
computers: ComputersTab,
wallpaper: WallpaperTab,
+ appshots: AppshotsTab,
costs: CostsTab,
feedback: FeedbackTab,
};
@@ -156,6 +162,8 @@ function resolveHubTab(active: string): { content: HubNavId | "telephony-setup";
return { content: "computers", highlight: "computers" };
case "wallpaper":
return { content: "wallpaper", highlight: "wallpaper" };
+ case "appshots":
+ return { content: "appshots", highlight: "appshots" };
case "costs":
return { content: "costs", highlight: "costs" };
case "feedback":
@@ -171,7 +179,11 @@ function resolveHubTab(active: string): { content: HubNavId | "telephony-setup";
// `NAV_GROUPS` plus the footer: "feedback" lives in `NAV_FOOTER_ITEMS`, not in
// a group (same lookup as TopBar/DockRail) — without it the hub cannot resolve
// its own tenth entry.
-const ALL_NAV_ITEMS: readonly NavItem[] = [...NAV_GROUPS.flat(), ...NAV_FOOTER_ITEMS];
+const ALL_NAV_ITEMS: readonly NavItem[] = [
+ ...NAV_GROUPS.flat(),
+ ...NAV_FOOTER_ITEMS,
+ ...SETTINGS_HUB_ONLY_ITEMS,
+];
function findNavItem(id: HubNavId): NavItem {
const item = ALL_NAV_ITEMS.find((row) => row.id === id);
diff --git a/jarvis/ui/web/frontend/src/views/SettingsView.tsx b/jarvis/ui/web/frontend/src/views/SettingsView.tsx
index 5ed69cfcef..cda8e6ca3c 100644
--- a/jarvis/ui/web/frontend/src/views/SettingsView.tsx
+++ b/jarvis/ui/web/frontend/src/views/SettingsView.tsx
@@ -22,7 +22,6 @@ import { VolumeGroup } from "@/views/settings/VolumeGroup";
import { AudioDevicesGroup } from "@/views/settings/AudioDevicesGroup";
import { SystemPromptGroup } from "@/views/settings/SystemPromptGroup";
import { SettingsGroupBoundary } from "@/views/settings/SettingsGroupBoundary";
-import { ScreenContextGroup } from "@/views/settings/ScreenContextGroup";
import { settingsInputCls } from "@/views/settings/SettingsBlock";
import {
useWakeWord,
@@ -161,7 +160,6 @@ const SECTIONS: readonly { id: string; labelKey: string; render: () => React.Rea
{ id: "languages", labelKey: "settings_view.nav.languages", render: () => <LanguagesGroup /> },
{ id: "app", labelKey: "settings_view.nav.app", render: () => <AppSettingsGroup /> },
{ id: "permissions", labelKey: "settings_view.nav.permissions", render: () => <PermissionsPanel /> },
- { id: "screen-context", labelKey: "settings_view.nav.screen_context", render: () => <ScreenContextGroup /> },
{ id: "realtime-voice", labelKey: "settings_view.nav.realtime_voice", render: () => <RealtimeVoiceGroup /> },
{ id: "system-prompt", labelKey: "settings_view.nav.system_prompt", render: () => <SystemPromptGroup /> },
{ id: "wake-word", labelKey: "settings_view.nav.wake_word", render: () => <WakeWordPanel /> },
diff --git a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.test.tsx b/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.test.tsx
deleted file mode 100644
index 0f2a7d6b81..0000000000
--- a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.test.tsx
+++ /dev/null
@@ -1,128 +0,0 @@
-import { fireEvent, render, screen, waitFor } from "@testing-library/react";
-import { afterEach, describe, expect, it, vi } from "vitest";
-
-import { ScreenContextGroup } from "./ScreenContextGroup";
-
-/**
- * Screen Context is a ONE-SWITCH card (maintainer mandate 2026-08-02). These
- * tests pin that contract from the outside: the only control is the switch, a
- * blocker is reported honestly while the feature is on, and nothing on the card
- * asks the user to write privacy rules by hand.
- */
-
-const SETTINGS = { enabled: true };
-
-function status(available: boolean, enabled = true) {
- return {
- enabled,
- available,
- blocked_reason: available ? null : "No vision-capable provider is configured.",
- blocked_reasons: available
- ? []
- : ["No vision-capable provider is configured."],
- monitor_count: 2,
- };
-}
-
-function response(body: unknown) {
- return { ok: true, json: async () => body };
-}
-
-afterEach(() => vi.restoreAllMocks());
-
-describe("ScreenContextGroup", () => {
- it("reports the real blocker while the feature is switched on", async () => {
- vi.stubGlobal(
- "fetch",
- vi.fn(async (url: string) =>
- response(url.endsWith("/settings") ? SETTINGS : status(false)),
- ),
- );
-
- render(<ScreenContextGroup />);
-
- await waitFor(() =>
- expect(
- screen.getByText("No vision-capable provider is configured."),
- ).toBeTruthy(),
- );
- });
-
- it("offers exactly one control and no privacy-rule editors", async () => {
- vi.stubGlobal(
- "fetch",
- vi.fn(async (url: string) =>
- response(url.endsWith("/settings") ? SETTINGS : status(true)),
- ),
- );
-
- const { container } = render(<ScreenContextGroup />);
-
- await waitFor(() => expect(screen.getByText(/Ready on 2/)).toBeTruthy());
- expect(container.querySelectorAll("textarea")).toHaveLength(0);
- expect(container.querySelectorAll("input")).toHaveLength(0);
- // The old card shipped Save / Test / Discard buttons plus three more
- // switches next to this one.
- const buttons = Array.from(container.querySelectorAll("button"));
- expect(buttons).toHaveLength(1);
- expect(buttons[0].getAttribute("role")).toBe("switch");
- });
-
- it("runs full width like every neighbouring settings card", async () => {
- vi.stubGlobal(
- "fetch",
- vi.fn(async (url: string) =>
- response(url.endsWith("/settings") ? SETTINGS : status(true)),
- ),
- );
-
- const { container } = render(<ScreenContextGroup />);
- await waitFor(() => expect(screen.getByText(/Ready on 2/)).toBeTruthy());
-
- // A leftover `max-w-5xl` from the multi-field card capped this one at half
- // the window and parked its switch mid-row next to full-width neighbours.
- // The settings column owns the width; the card must not cap it.
- const root = container.firstElementChild;
- expect(root).toBeTruthy();
- const capped = Array.from(container.querySelectorAll("*"))
- .concat(root ? [root] : [])
- .filter((element) =>
- Array.from(element.classList).some((name) => name.startsWith("max-w-")),
- );
- expect(capped).toHaveLength(0);
- // Same shell as RealtimeVoiceGroup, so the two read as one list.
- expect(root?.className).toBe(
- "mt-2 rounded-lg border border-border bg-card p-4",
- );
- });
-
- it("writes only the enabled flag when the switch is flipped", async () => {
- const fetchMock = vi.fn(async (url: string, init?: RequestInit) => {
- if (url.endsWith("/settings") && init?.method === "PUT") {
- return response({ ok: true, changed: ["enabled"] });
- }
- if (url.endsWith("/settings")) return response(SETTINGS);
- if (url.endsWith("/status")) return response(status(true));
- throw new Error(`Unexpected request: ${url}`);
- });
- vi.stubGlobal("fetch", fetchMock);
-
- const { container } = render(<ScreenContextGroup />);
- const toggle = await waitFor(() => {
- const found = container.querySelector("button[role=switch]");
- if (!found) throw new Error("switch not rendered yet");
- return found;
- });
- fireEvent.click(toggle);
-
- await waitFor(() => {
- const put = fetchMock.mock.calls.find(
- ([, init]) => (init as RequestInit | undefined)?.method === "PUT",
- );
- expect(put).toBeTruthy();
- expect(JSON.parse(String((put?.[1] as RequestInit).body))).toEqual({
- enabled: false,
- });
- });
- });
-});
diff --git a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.tsx b/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.tsx
deleted file mode 100644
index 9393ad47b2..0000000000
--- a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.tsx
+++ /dev/null
@@ -1,149 +0,0 @@
-import { useEffect, useState } from "react";
-import { Eye, Loader2 } from "lucide-react";
-
-import { Switch } from "@/components/ui/switch";
-import { useT } from "@/i18n";
-import { useEventStore } from "@/store/events";
-
-/**
- * Screen Context is deliberately a ONE-SWITCH feature (maintainer mandate
- * 2026-08-02). The privacy rules it used to expose here — denylist, extra
- * patterns, OCR, retention seconds — are still enforced at their shipped
- * defaults and remain editable through `jarvis api screen-context put-settings`
- * for anyone who wants them, but a settings card that asks a non-technical user
- * to write regular expressions is a card they switch off instead of using.
- */
-interface ScreenContextSettings {
- enabled: boolean;
-}
-
-interface ScreenContextStatus {
- enabled: boolean;
- available: boolean;
- blocked_reason: string | null;
- monitor_count: number;
-}
-
-async function jsonRequest<T>(url: string, init?: RequestInit): Promise<T> {
- const response = await fetch(url, init);
- const body = (await response.json().catch(() => null)) as
- | T
- | { detail?: string }
- | null;
- if (!response.ok) {
- const detail =
- body !== null &&
- typeof body === "object" &&
- "detail" in body &&
- typeof body.detail === "string"
- ? body.detail
- : "";
- throw new Error(
- detail || `Screen Context request failed (${response.status}).`,
- );
- }
- return body as T;
-}
-
-export function ScreenContextGroup() {
- const t = useT();
- const pushToast = useEventStore((state) => state.pushToast);
- const [settings, setSettings] = useState<ScreenContextSettings | null>(null);
- const [status, setStatus] = useState<ScreenContextStatus | null>(null);
- const [loading, setLoading] = useState(true);
- const [saving, setSaving] = useState(false);
-
- useEffect(() => {
- let active = true;
- void Promise.all([
- jsonRequest<ScreenContextSettings>("/api/screen-context/settings"),
- jsonRequest<ScreenContextStatus>("/api/screen-context/status"),
- ])
- .then(([nextSettings, nextStatus]) => {
- if (!active) return;
- setSettings(nextSettings);
- setStatus(nextStatus);
- })
- .catch((error: Error) => pushToast("error", error.message))
- .finally(() => {
- if (active) setLoading(false);
- });
- return () => {
- active = false;
- };
- }, [pushToast]);
-
- async function toggle(enabled: boolean) {
- setSaving(true);
- try {
- await jsonRequest("/api/screen-context/settings", {
- method: "PUT",
- headers: { "content-type": "application/json" },
- body: JSON.stringify({ enabled }),
- });
- setSettings((current) => (current === null ? current : { enabled }));
- setStatus(
- await jsonRequest<ScreenContextStatus>("/api/screen-context/status"),
- );
- pushToast("success", t("settings_view.screen_context.saved"));
- } catch (error) {
- pushToast("error", (error as Error).message);
- } finally {
- setSaving(false);
- }
- }
-
- // A blocker is only worth showing while the feature is meant to be running:
- // "switched off" is not a problem to report, it is what the switch says.
- const showBlocker = Boolean(settings?.enabled && status && !status.available);
-
- // Layout is deliberately byte-for-byte the neighbouring RealtimeVoiceGroup
- // shell: same wrapper, same icon placement, same spacing. The card carried a
- // `max-w-5xl` from its multi-field days, which capped it at half the window
- // and parked its switch mid-row while every other card ran full width — a
- // mismatch that only became visible once the card got short. Do not
- // reintroduce a width cap here; the settings column owns the width.
- return (
- <div className="mt-2 rounded-lg border border-border bg-card p-4">
- <div className="flex items-start gap-3">
- <Eye className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" />
- <div className="min-w-0 flex-1">
- <div className="flex items-center justify-between gap-4">
- <h4 className="font-medium">
- {t("settings_view.screen_context.title")}
- </h4>
- {loading || !settings ? (
- <Loader2 className="h-4 w-4 animate-spin text-muted-foreground" />
- ) : (
- <Switch
- checked={settings.enabled}
- disabled={saving}
- aria-label={t("settings_view.screen_context.title")}
- onCheckedChange={(enabled) => void toggle(enabled)}
- />
- )}
- </div>
- <p className="mt-0.5 text-xs text-muted-foreground">
- {t("settings_view.screen_context.description")}
- </p>
- {settings?.enabled && status && (
- <p
- className={`mt-1.5 text-micro ${
- status.available ? "text-muted-foreground" : "text-foreground"
- }`}
- aria-live="polite"
- >
- {showBlocker
- ? status.blocked_reason ||
- t("settings_view.screen_context.unavailable")
- : t("settings_view.screen_context.available").replace(
- "{0}",
- String(status.monitor_count),
- )}
- </p>
- )}
- </div>
- </div>
- </div>
- );
-}
diff --git a/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts b/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts
index a2d0dc3a44..8de537f254 100644
--- a/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts
+++ b/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts
@@ -12,7 +12,6 @@ const SEARCH_GROUPS = [
{ id: "languages", keys: ["language", "languages_group_title"] },
{ id: "app", keys: ["app_settings_group_title", "autostart", "appearance", "jarvis_api"] },
{ id: "permissions", keys: ["nav.permissions"] },
- { id: "screen-context", keys: ["screen_context"] },
{ id: "realtime-voice", keys: ["realtime_voice"] },
{ id: "system-prompt", keys: ["system_prompt"] },
{ id: "wake-word", keys: ["wake_word"] },
@@ -32,6 +31,7 @@ const SEARCH_PAGES = [
{ id: "socials", keys: ["socials"] },
{ id: "apikeys", keys: ["apikeys_view", "apikeys_voice", "apikeys_model", "apikeys_cu_model"] },
{ id: "wallpaper", keys: ["home.background_label", "home.background_hint"] },
+ { id: "appshots", keys: ["appshots"] },
{ id: "costs", keys: ["costs_view"] },
{ id: "feedback", keys: ["feedback"] },
] as const;
diff --git a/jarvis/ui/web/server.py b/jarvis/ui/web/server.py
index 36eae721ab..4c7ff92b4e 100644
--- a/jarvis/ui/web/server.py
+++ b/jarvis/ui/web/server.py
@@ -391,6 +391,7 @@ class WebServer:
from .provider_routes import router as provider_router
from .review_routes import router as review_router
from .routine_hooks_routes import router as routine_hooks_router
+ from .appshot_routes import router as appshot_router
from .screen_context_routes import router as screen_context_router
from .self_mod_routes import router as self_mod_router
from .sessions_routes import router as sessions_router
@@ -565,6 +566,9 @@ class WebServer:
# machine with no display, so `jarvis api screen-context status` is a
# valid capability probe everywhere.
app.include_router(screen_context_router)
+ # Appshots: the front window as conversation context, on a shortcut,
+ # a button or a spoken request. Captures through Screen Context.
+ app.include_router(appshot_router)
# The mission deck's pictures: the last Screen-Context capture (one
# frame, in memory, TTL) and Computer-Use frames by content hash.
app.include_router(deck_router)
@@ -2392,6 +2396,30 @@ class WebServer:
"will be renewed in the background."
)
+ def _schedule_appshot_shortcut(self) -> None:
+ """Arm the global appshot shortcut once the wake model has loaded.
+
+ Off the boot path (AP-26): the key reader and the hotkey backend load
+ native libraries. Headless hosts arm nothing and say so in the log.
+ """
+ try:
+ loop = asyncio.get_running_loop()
+ except RuntimeError:
+ logger.debug("Appshot shortcut not scheduled — no running event loop.")
+ return
+
+ async def _arm() -> None:
+ from jarvis.appshot.hotkey import start_appshot_shortcut
+ from jarvis.core import runtime_refs as _rr
+
+ await _rr.await_wake_model_ready(timeout=12.0)
+ try:
+ await start_appshot_shortcut(self.bus)
+ except Exception as exc: # noqa: BLE001 - voice/chat work without it
+ logger.opt(exception=exc).warning("Appshot shortcut could not start")
+
+ self._appshot_shortcut_task = loop.create_task(_arm(), name="appshot-shortcut")
+
def _schedule_realtime_transport_warm(self) -> None:
"""Pre-open the selected realtime transports off the boot path.
@@ -2831,6 +2859,7 @@ class WebServer:
# browser-only install has no desktop shell to warm the realtime
# transport for it.
self._schedule_realtime_transport_warm()
+ self._schedule_appshot_shortcut()
# Defer provisioning until the boot chain returns control to the server.
async def prepare_browser() -> None:
from jarvis.society.browser import install
diff --git a/tests/unit/appshot/__init__.py b/tests/unit/appshot/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/unit/appshot/test_appshot_core.py b/tests/unit/appshot/test_appshot_core.py
new file mode 100644
index 0000000000..b0ec8dc842
--- /dev/null
+++ b/tests/unit/appshot/test_appshot_core.py
@@ -0,0 +1,425 @@
+"""Appshots — store, gesture, placement, shortcut spelling and turn delivery.
+
+Everything here runs without a display: the key reader, the capture service
+and the voice session are small fakes, so the contract is checked on a
+headless CI box exactly as on a desktop.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from jarvis.appshot import service as appshot_service
+from jarvis.appshot.effect import placement
+from jarvis.appshot.gesture import BothAltWatcher
+from jarvis.appshot.hotkey import normalize_hotkey
+from jarvis.appshot.store import Appshot, AppshotStore, get_store
+from jarvis.brain.manager import _takes_pending_appshot
+from jarvis.screen_context.models import (
+ CaptureTarget,
+ IntentVerdict,
+ ScreenContext,
+ TargetKind,
+ TargetReason,
+ VisualIntent,
+ WindowFacts,
+)
+from jarvis.screen_context.service import CaptureOutcome
+from jarvis.screen_context.turn import screen_context_for_turn
+
+
+class Clock:
+ def __init__(self) -> None:
+ self.now = 100.0
+
+ def __call__(self) -> float:
+ return self.now
+
+
+def make_shot(shot_id: str = "a1") -> Appshot:
+ return Appshot(
+ id=shot_id,
+ image=b"jpeg",
+ mime="image/jpeg",
+ width=800,
+ height=600,
+ label="active window",
+ app_name="Editor",
+ note="APPSHOT: evidence",
+ ui_text="hello",
+ trigger="hotkey",
+ taken_at=1.0,
+ )
+
+
+def make_context() -> ScreenContext:
+ return ScreenContext(
+ image=b"jpeg-bytes",
+ mime="image/jpeg",
+ size=(1280, 720),
+ target=CaptureTarget(
+ kind=TargetKind.WINDOW,
+ bbox=(10, 20, 1280, 720),
+ reason=TargetReason.FOCUSED_WINDOW,
+ window=WindowFacts(app_name="Editor", title="notes.md"),
+ ),
+ ui_text="Build failed",
+ captured_at_ns=1,
+ )
+
+
+# --------------------------------------------------------------------- store
+
+
+def test_pending_appshot_is_single_use_and_expires() -> None:
+ clock = Clock()
+ store = AppshotStore(clock=clock)
+ store.park(make_shot("a1"), ttl_s=60)
+
+ assert store.peek_pending() is not None
+ assert store.take_pending("other") is None, "a foreign id never claims it"
+ assert store.take_pending("a1").id == "a1"
+ assert store.take_pending() is None, "the second take gets nothing"
+
+ store.park(make_shot("a2"), ttl_s=60)
+ clock.now += 61
+ assert store.take_pending() is None, "an expired appshot is gone"
+
+
+def test_latest_is_kept_for_its_budget_and_zero_keeps_nothing() -> None:
+ clock = Clock()
+ store = AppshotStore(clock=clock)
+ store.remember(make_shot(), keep_s=120)
+ store.mark_delivered("a1", "voice")
+ assert store.latest().delivered_to == "voice"
+ clock.now += 121
+ assert store.latest() is None
+
+ store.remember(make_shot(), keep_s=0)
+ assert store.latest() is None
+
+
+# ------------------------------------------------------------------- gesture
+
+
+class Keys:
+ def __init__(self) -> None:
+ self.state: tuple[bool, bool] | None = (False, False)
+
+ def __call__(self):
+ return self.state
+
+
+def test_both_alt_fires_once_per_press_and_rearms_on_release(monkeypatch) -> None:
+ import jarvis.platform.self_input as self_input
+
+ monkeypatch.setattr(self_input, "synthetic_input_recent", lambda *a, **k: False)
+ keys, clock, fired = Keys(), Clock(), []
+ watcher = BothAltWatcher(lambda: fired.append(1), probe=keys, clock=clock)
+
+ keys.state = (True, False)
+ assert not watcher.step(), "one Alt key alone is ordinary typing"
+ keys.state = (True, True)
+ assert watcher.step()
+ assert not watcher.step(), "holding the keys does not repeat"
+ keys.state = (False, False)
+ watcher.step()
+ clock.now += 1.0
+ keys.state = (True, True)
+ assert watcher.step(), "a new press after release fires again"
+ assert len(fired) == 2
+
+
+def test_both_alt_ignores_a_bounce_and_synthetic_input(monkeypatch) -> None:
+ import jarvis.platform.self_input as self_input
+
+ keys, clock, fired = Keys(), Clock(), []
+ watcher = BothAltWatcher(lambda: fired.append(1), probe=keys, clock=clock)
+ monkeypatch.setattr(self_input, "synthetic_input_recent", lambda *a, **k: False)
+ keys.state = (True, True)
+ watcher.step()
+ keys.state = (False, False)
+ watcher.step()
+ clock.now += 0.1
+ keys.state = (True, True)
+ assert not watcher.step(), "a key bounce inside the guard is the same press"
+
+ keys.state = (False, False)
+ watcher.step()
+ clock.now += 5
+ monkeypatch.setattr(self_input, "synthetic_input_recent", lambda *a, **k: True)
+ keys.state = (True, True)
+ assert not watcher.step(), "Jarvis's own keystrokes never take an appshot"
+ assert len(fired) == 1
+
+
+def test_unreadable_keyboard_never_fires() -> None:
+ keys, fired = Keys(), []
+ keys.state = None
+ watcher = BothAltWatcher(lambda: fired.append(1), probe=keys)
+ assert not watcher.step()
+ assert fired == []
+
+
+# ------------------------------------------------------------ effect placing
+
+
+MONITORS = [
+ {"left": 0, "top": 0, "width": 3840, "height": 1080},
+ {"left": 0, "top": 0, "width": 1920, "height": 1080},
+ {"left": 1920, "top": 0, "width": 1920, "height": 1080},
+]
+
+
+def test_placement_picks_the_monitor_under_the_window_centre() -> None:
+ monitor, rect = placement((2400, 100, 960, 540), MONITORS)
+ assert monitor == [1920, 0, 1920, 1080]
+ assert rect == pytest.approx([0.25, 100 / 1080, 0.5, 0.5])
+
+
+def test_placement_clips_a_window_hanging_off_its_monitor() -> None:
+ monitor, rect = placement((1500, 0, 800, 1080), MONITORS)
+ assert monitor == [0, 0, 1920, 1080]
+ assert rect[0] + rect[2] == pytest.approx(1.0)
+
+
+def test_placement_without_monitors_covers_the_capture() -> None:
+ monitor, rect = placement((0, 0, 100, 50), [])
+ assert monitor == [0, 0, 100, 50]
+ assert rect == [0.0, 0.0, 1.0, 1.0]
+
+
+# ---------------------------------------------------------------- shortcuts
+
+
+ ("raw", "expected"),
+ [
+ ("alt+alt", "alt+alt"),
+ ("Left_Alt + Right_Alt", "alt+alt"),
+ ("altgr+alt", "alt+alt"),
+ ("Ctrl + Alt + A", "ctrl+alt+a"),
+ ("", ""),
+ ],
+)
+def test_shortcut_spelling_is_canonical(raw: str, expected: str) -> None:
+ assert normalize_hotkey(raw) == expected
+
+
+ ("layer", "takes"),
+ [
+ ("speech", True),
+ ("ui.web.chat", True),
+ (None, True),
+ ("tasks.runner", False),
+ ("workflows.scheduler", False),
+ ("society.lead", False),
+ ("tool.run-skill", False),
+ ],
+)
+def test_only_a_person_waiting_takes_the_parked_appshot(layer, takes) -> None:
+ assert _takes_pending_appshot(layer) is takes
+
+
+# ------------------------------------------------------------ turn delivery
+
+
+def clean_store():
+ get_store().clear()
+ yield get_store()
+ get_store().clear()
+
+
+class NeverCaptures:
+ """A service that fails the test if the turn tries to take a new picture."""
+
+ async def capture_for_turn(self, *args, **kwargs):
+ raise AssertionError("the parked appshot should have answered this turn")
+
+
+async def test_the_next_message_carries_the_parked_appshot(clean_store) -> None:
+ clean_store.park(make_shot("p1"), ttl_s=60)
+
+ result = await screen_context_for_turn(
+ "what is wrong here?", locale="en", service=NeverCaptures()
+ )
+
+ assert result.has_image
+ assert result.image == b"jpeg"
+ assert result.note.startswith("APPSHOT")
+ assert clean_store.peek_pending() is None, "single use"
+
+
+async def test_an_automated_turn_leaves_the_parked_appshot_alone(clean_store) -> None:
+ clean_store.park(make_shot("p1"), ttl_s=60)
+
+ class NoVisualIntent:
+ async def capture_for_turn(self, *args, **kwargs):
+ return CaptureOutcome(
+ status="not_requested", verdict=IntentVerdict(intent=VisualIntent.NONE)
+ )
+
+ result = await screen_context_for_turn(
+ "summarise the news",
+ locale="en",
+ service=NoVisualIntent(),
+ allow_pending_appshot=False,
+ )
+
+ assert result.status == "none"
+ assert clean_store.peek_pending() is not None
+
+
+async def test_asking_for_a_new_appshot_drops_the_stale_one(clean_store) -> None:
+ clean_store.park(make_shot("old"), ttl_s=60)
+
+ class Captures:
+ def __init__(self) -> None:
+ self.asked = False
+
+ async def capture_for_turn(self, *args, **kwargs):
+ self.asked = True
+ return CaptureOutcome(
+ status="refused",
+ verdict=IntentVerdict(intent=VisualIntent.WINDOW),
+ reason_kind="policy",
+ message="switched off",
+ )
+
+ service = Captures()
+ await screen_context_for_turn("take an appshot", locale="en", service=service)
+
+ assert service.asked, "a fresh capture was attempted"
+ assert clean_store.peek_pending() is None
+
+
+# ----------------------------------------------------- shortcut destination
+
+
+class Config:
+ class screen_context: # noqa: N801 - mirrors the config attribute
+ enabled = True
+ ttl_s = 120.0
+ deck_preview_s = 120.0
+
+ class appshot: # noqa: N801
+ target = "auto"
+
+
+class FakeCaptureService:
+ def __init__(self) -> None:
+ self.consumed: list[str] = []
+ self.verdicts: list[IntentVerdict] = []
+
+ async def capture(self, *, verdict=None, trace_id=None):
+ self.verdicts.append(verdict)
+ return CaptureOutcome(
+ status="captured", verdict=verdict, context=make_context(), handle_id="h1"
+ )
+
+ def consume(self, handle_id):
+ self.consumed.append(handle_id)
+
+
+class Bus:
+ def __init__(self) -> None:
+ self.events: list = []
+
+ async def publish(self, event) -> None:
+ self.events.append(event)
+
+
+def wired(monkeypatch, clean_store):
+ import jarvis.appshot.delivery as delivery
+ import jarvis.screen_context.turn as turn
+
+ service = FakeCaptureService()
+ config = Config()
+ live = {"accepts": False, "calls": 0}
+
+ async def deliver_to_live(image, mime, note):
+ live["calls"] += 1
+ return live["accepts"]
+
+ monkeypatch.setattr(turn, "get_service", lambda bus=None: service)
+ monkeypatch.setattr(appshot_service, "_load_config", lambda: config)
+ monkeypatch.setattr(delivery, "deliver_to_live", deliver_to_live)
+ return service, config, live, clean_store
+
+
+async def test_without_a_call_the_appshot_waits_for_the_next_message(wired) -> None:
+ service, _config, _live, store = wired
+ bus = Bus()
+
+ result = await appshot_service.take_appshot(trigger="hotkey", bus=bus)
+
+ assert result.ok and result.shot.delivered_to == "message"
+ assert service.verdicts[0].intent is VisualIntent.WINDOW, "an appshot is the front window"
+ assert service.consumed == ["h1"], "the capture handle is not left in memory"
+ assert store.peek_pending().id == result.shot.id
+ assert store.latest().app_name == "Editor"
+ event = bus.events[-1]
+ assert event.delivered_to == "message"
+ assert "notes.md" not in repr(event), "a window title never enters the bus"
+
+
+async def test_a_running_call_receives_the_appshot_directly(wired) -> None:
+ _service, _config, live, store = wired
+ live["accepts"] = True
+
+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())
+
+ assert result.shot.delivered_to == "voice"
+ assert store.peek_pending() is None, "a delivered appshot is not also parked"
+
+
+async def test_voice_only_target_without_a_call_sends_nothing(wired) -> None:
+ _service, config, _live, store = wired
+ config.appshot.target = "voice"
+ try:
+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())
+ finally:
+ config.appshot.target = "auto"
+
+ assert result.shot.delivered_to == "none"
+ assert store.peek_pending() is None
+
+
+async def test_message_target_never_interrupts_a_call(wired) -> None:
+ _service, config, live, _store = wired
+ live["accepts"] = True
+ config.appshot.target = "message"
+ try:
+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())
+ finally:
+ config.appshot.target = "auto"
+
+ assert result.shot.delivered_to == "message"
+ assert live["calls"] == 0
+
+
+async def test_switched_off_appshots_refuse_without_capturing(wired) -> None:
+ service, config, _live, _store = wired
+ config.screen_context.enabled = False
+ try:
+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())
+ finally:
+ config.screen_context.enabled = True
+
+ assert not result.ok
+ assert result.reason_code == "disabled"
+ assert service.verdicts == []
+
+
+async def test_the_tool_keeps_the_picture_for_its_own_answer(wired) -> None:
+ _service, _config, live, store = wired
+ live["accepts"] = True
+
+ result = await appshot_service.take_appshot(trigger="tool", bus=Bus(), deliver=False)
+
+ assert result.shot.delivered_to == "turn"
+ assert live["calls"] == 0
+ assert store.peek_pending() is None
diff --git a/tests/unit/appshot/test_appshot_surfaces.py b/tests/unit/appshot/test_appshot_surfaces.py
new file mode 100644
index 0000000000..b4b1a310e8
--- /dev/null
+++ b/tests/unit/appshot/test_appshot_surfaces.py
@@ -0,0 +1,226 @@
+"""Appshots on every surface: speech vocabulary, voice calls, config, REST."""
+
+from __future__ import annotations
+
+import base64
+import tomllib
+from types import SimpleNamespace
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.core.config_writer import set_appshot_settings
+from jarvis.live.native import NativeLiveVoiceSession
+from jarvis.live.session import LiveVoiceSession
+from jarvis.screen_context.intent import classify, mentions_appshot
+from jarvis.screen_context.models import VisualIntent
+
+# ------------------------------------------------------------------ speech
+
+
+ "utterance",
+ [
+ "take an appshot",
+ "Mach mal einen Appshot", # i18n-allow: German speech-input test data
+ "mach einen App Shot und sag mir was da steht", # i18n-allow: DE input
+ "Appshot bitte", # i18n-allow: German speech-input test data
+ "hazme un app shot",
+ ],
+)
+def test_naming_the_feature_takes_the_front_window(utterance: str) -> None:
+ assert classify(utterance, locale="en").intent is VisualIntent.WINDOW
+ assert mentions_appshot(utterance)
+
+
+ "utterance",
+ [
+ "what is an appshot?",
+ "how do appshots work",
+ "Was ist ein Appshot?", # i18n-allow: German speech-input test data
+ "Öffne die Appshots Einstellungen", # i18n-allow: DE input
+ "geh zu Appshots", # i18n-allow: German speech-input test data
+ "open the appshot settings",
+ "the snapshot looks fine",
+ "happy shots all round",
+ ],
+)
+def test_asking_about_or_opening_appshots_takes_nothing(utterance: str) -> None:
+ assert classify(utterance, locale="en").intent is VisualIntent.NONE
+
+
+# ------------------------------------------------------------- voice calls
+
+
+class Recorder:
+ def __init__(self) -> None:
+ self.sent: list[dict] = []
+ self.images: list[tuple[bytes, str]] = []
+ self.answer_sdp = ""
+
+ async def send(self, event: dict) -> None:
+ self.sent.append(event)
+
+ async def send_image(self, image: bytes, mime: str) -> None:
+ self.images.append((image, mime))
+
+
+async def _noop(*_args, **_kwargs) -> None:
+ return None
+
+
+def _config():
+ return SimpleNamespace(brain=SimpleNamespace(reply_language="en"))
+
+
+async def test_gpt_live_puts_the_appshot_into_the_backend_silently() -> None:
+ session = LiveVoiceSession(
+ session_id="s",
+ send_binary=_noop,
+ send_json=_noop,
+ providers=[SimpleNamespace(name="openai")],
+ config=_config(),
+ )
+ session._connection = Recorder()
+
+ assert await session.attach_appshot(b"jpeg", "image/jpeg", "APPSHOT: note")
+
+ kinds = [event["type"] for event in session._connection.sent]
+ assert kinds == ["session.thinking.append", "response.item.create"]
+ assert "response.create" not in kinds, "the user's next words are the question"
+ content = session._connection.sent[1]["item"]["content"]
+ assert content[0] == {"type": "input_text", "text": "APPSHOT: note"}
+ assert content[1]["image_url"] == "data:image/jpeg;base64," + base64.b64encode(
+ b"jpeg"
+ ).decode("ascii")
+
+
+async def test_gpt_live_declines_while_it_is_recovering() -> None:
+ session = LiveVoiceSession(
+ session_id="s",
+ send_binary=_noop,
+ send_json=_noop,
+ providers=[SimpleNamespace(name="openai")],
+ config=_config(),
+ )
+ session._connection = Recorder()
+ session._recovering = True
+
+ assert not await session.attach_appshot(b"jpeg", "image/jpeg", "note")
+ assert session._connection.sent == []
+
+
+async def test_native_live_sends_a_frame_and_no_text_turn() -> None:
+ session = NativeLiveVoiceSession(
+ session_id="s",
+ send_binary=_noop,
+ send_json=_noop,
+ providers=[SimpleNamespace(name="gemini")],
+ config=_config(),
+ )
+ session._connection = Recorder()
+
+ assert await session.attach_appshot(b"png", "image/png", "note")
+ assert session._connection.images == [(b"png", "image/png")]
+ assert session._connection.sent == []
+
+
+async def test_native_live_without_image_input_declines() -> None:
+ session = NativeLiveVoiceSession(
+ session_id="s",
+ send_binary=_noop,
+ send_json=_noop,
+ providers=[SimpleNamespace(name="local")],
+ config=_config(),
+ )
+ session._connection = SimpleNamespace(answer_sdp="")
+
+ assert not await session.attach_appshot(b"png", "image/png", "note")
+
+
+# ------------------------------------------------------------------ config
+
+
+def test_appshot_patch_writes_one_table(tmp_path) -> None:
+ path = tmp_path / "jarvis.toml"
+ path.write_text("[ui]\nsound_effects = true\n", encoding="utf-8")
+
+ set_appshot_settings({"hotkey": "alt+alt", "target": "message", "sound": False}, path=path)
+
+ data = tomllib.loads(path.read_text(encoding="utf-8"))
+ assert data["appshot"] == {"hotkey": "alt+alt", "target": "message", "sound": False}
+ assert data["ui"] == {"sound_effects": True}
+
+
+def test_appshot_patch_rejects_unknown_keys_before_writing(tmp_path) -> None:
+ path = tmp_path / "jarvis.toml"
+ path.write_text("", encoding="utf-8")
+
+ with pytest.raises(ValueError, match="unknown appshot"):
+ set_appshot_settings({"enabled": True}, path=path)
+ assert path.read_text(encoding="utf-8") == ""
+
+
+# -------------------------------------------------------------------- REST
+
+
+def client() -> TestClient:
+ from jarvis.ui.web.appshot_routes import router
+
+ app = FastAPI()
+ app.include_router(router)
+ app.state.bus = None
+ return TestClient(app)
+
+
+def test_an_empty_settings_patch_is_refused_without_writing(client) -> None:
+ assert client.put("/api/appshot/settings", json={}).status_code == 400
+
+
+def test_an_unknown_destination_is_refused(client) -> None:
+ assert client.put("/api/appshot/settings", json={"target": "email"}).status_code == 422
+
+
+def test_take_reports_a_refusal_as_data(client, monkeypatch) -> None:
+ import jarvis.appshot.service as service
+
+ async def refuse(**_kwargs):
+ return service.AppshotResult(status="refused", reason_code="disabled", message="off")
+
+ monkeypatch.setattr(service, "take_appshot", refuse)
+ body = client.post("/api/appshot/take", json={"delay_s": 0}).json()
+ assert body == {"ok": False, "reason": "disabled", "message": "off"}
+
+
+def test_the_parked_appshot_is_claimed_once(client) -> None:
+ from jarvis.appshot.store import Appshot, get_store
+
+ store = get_store()
+ store.clear()
+ store.park(
+ Appshot(
+ id="c1",
+ image=b"jpeg",
+ mime="image/jpeg",
+ width=1,
+ height=1,
+ label="active window",
+ app_name="",
+ note="",
+ ui_text="",
+ trigger="hotkey",
+ taken_at=0.0,
+ ),
+ ttl_s=60,
+ )
+ try:
+ assert client.get("/api/appshot/pending").json()["appshot"]["id"] == "c1"
+ first = client.post("/api/appshot/pending/claim", json={"id": "c1"})
+ assert first.status_code == 200 and first.content == b"jpeg"
+ assert first.headers["cache-control"] == "no-store"
+ assert client.post("/api/appshot/pending/claim", json={"id": "c1"}).status_code == 404
+ finally:
+ store.clear()