Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
Python agent-orchestrationai-agentai-coding-assistantclaude-codecoding-agentcomputer-usedesktop-applocal-firstmcppersonal-jarvisvoice-assistantvoice-coding
feat(appshot): show the assistant the front window on a shortcut, button or request
rubenluetke10-beep3e605159539e
diff --git a/docs/appshots.md b/docs/appshots.mdnew file mode 100644index 0000000000..cc5e7c3e77--- /dev/null+++ b/docs/appshots.md@@ -0,0 +1,81 @@+# Appshots++An appshot shows the assistant the window you are working in. It captures the+front window once — picture and on-screen text — and hands it to the+conversation as context. Settings live under **Settings > Appshots**.++## Three ways to take one++| Trigger | What happens |+|---|---|+| **Shortcut** — both Alt keys at once by default (both Option keys on a Mac) | The front window is captured and delivered per **Appshot destination**. |+| **Voice or chat** — "take an appshot", "mach einen Appshot", "haz un appshot" | The turn that asked captures the front window and answers with it. |+| **Try it** button on the Appshots page | Waits three seconds so you can switch windows, then behaves like the shortcut. |++A spoken "what do you see?" is the same look (Screen Context); it also plays+the shutter and shows up as the last appshot.++## Where a shortcut appshot goes++| Destination | Running voice call | No voice call |+|---|---|---|+| **Automatic** (default) | Into the call | Onto your next message |+| **Next message** | Onto your next message | Onto your next message |+| **Voice call only** | Into the call | Not sent |++- *Into the call* is silent: GPT-Live puts the picture into its thinking+ backend's context, native live models (Gemini, local servers with image+ input) receive it as a video frame. Your next words are the question.+- *Onto your next message*: the next spoken turn uses it, or — while the+ front-page chat is open — it appears in the composer as an attachment you+ can remove. It is single use and expires after `[screen_context].ttl_s`.+- Scheduled tasks, workflows and background agents never take a parked+ appshot; it waits for a person.++## Privacy++Appshots capture through the Screen Context engine+([screen-context.md](screen-context.md)), so everything there applies+unchanged: the app denylist, redaction of password fields and sensitive+patterns, the capture indicator before the shutter, and no image ever written+to disk. **Allow appshots** on the Appshots page is `[screen_context].enabled`+— one switch for every screen look.++The shutter effect's thumbnail is cut from the frame in memory and piped only+to the local overlay process, which is excluded from screen capture on+Windows and hidden before any capture elsewhere. The last appshot shown on the+Appshots page is kept in memory for `[screen_context].deck_preview_s` seconds+and served with `Cache-Control: no-store`.++## Settings++`[appshot]` in `jarvis.toml`, written only through the app or+`jarvis api appshot put-settings`:++| Key | Default | Meaning |+|---|---|---|+| `hotkey` | `"alt+alt"` | `alt+alt` = both Alt keys; any other combo in the shared hotkey syntax; `""` = off |+| `target` | `"auto"` | `auto` · `message` · `voice` (table above) |+| `sound` | `true` | Shutter sound; also needs `[ui].sound_effects` |+| `effect` | `true` | Flash and corner thumbnail |++## Operating systems++| | Windows | macOS | Linux/X11 | Wayland / headless |+|---|---|---|---|---|+| Both-Alt shortcut | `GetAsyncKeyState` (AltGr counts as right Alt) | `CGEventSourceKeyState`, needs the Input Monitoring grant | `XQueryKeymap` via python-xlib | Unavailable, reason shown on the page; voice and the button still work where capture works |+| Other shortcuts | Shared hotkey backends (`jarvis/trigger/backends`) | same | same | same as above |+| Flash + thumbnail | PySide6 overlay, excluded from capture | PySide6 overlay | PySide6 overlay | No overlay; the appshot is still taken where capture works |+| Capture | Screen Context engine | Needs Screen Recording | X11 | Honest refusal |++Only the instance that owns ambient duties (the default app, not the dev+instance) arms the shortcut.++## Code++`jarvis/appshot/` — `service.py` (take and deliver), `store.py` (pending and+last appshot, memory only), `gesture.py` (both-Alt watcher), `hotkey.py`+(shortcut lifecycle), `effect.py` (shutter hook), `delivery.py` (voice calls).+The live model's `take_appshot` tool is `jarvis/plugins/tool/appshot.py`; the+REST surface is `jarvis/ui/web/appshot_routes.py`; the page is+`frontend/src/views/AppshotsView.tsx`.diff --git a/docs/os-parity.md b/docs/os-parity.mdindex d9d584226a..5dd26a33b6 100644--- a/docs/os-parity.md+++ b/docs/os-parity.md@@ -532,6 +532,7 @@ implementations, not stubs.|---|---|| Computer-Use / desktop actions (click, type, hotkey, scroll, drag, windows, apps, screenshots, UI trees) | Full per-OS backends (Win32/UIA, Quartz/AX, xdotool/AT-SPI); honest degradation on Wayland/headless/missing TCC grants || On-demand Screen Context | One-shot capture is wired into the production brain on Windows, macOS, and Linux/X11; UIA/AX/AT-SPI text is source-filtered, the indicator precedes capture, and Wayland/headless/missing grants refuse honestly |+| Appshots (front-window capture on a shortcut, button or request) | Capture, privacy and delivery are OS-neutral (Screen Context engine, `jarvis/appshot`). The both-Alt shortcut reads key state per OS: Windows `GetAsyncKeyState`, macOS `CGEventSourceKeyState` (Input Monitoring grant), Linux/X11 `XQueryKeymap`; Wayland/headless report it unavailable on the Appshots page. The flash is the PySide6 overlay where one can run. Verified live on Windows only; see `docs/appshots.md` || Voice / audio (capture, playback, VAD, wake, STT, TTS, realtime) | Clean; headless disables voice honestly; WASAPI logic is inert-by-data off Windows || Core (launcher, config, keyring, restart, autostart, tray, elevation, paths) | Clean; per-OS autostart (Registry / LaunchAgent / XDG `.desktop`), keyring falls back to a 0600 file on headless hosts || Data / agents (wiki, contacts, telephony, sessions, missions, skills, self-mod, channels, MCP) | Clean; mission workers run on POSIX with a real process-group reaper |diff --git a/docs/screen-context.md b/docs/screen-context.mdindex bcc0e2ca15..595fbc3b39 100644--- a/docs/screen-context.md+++ b/docs/screen-context.md@@ -11,6 +11,13 @@ platform through the same adapter seam.## 1. What this is+> **Appshots** (2026-09-29) are the user-facing face of this engine: a+> global shortcut, a button and the word "appshot" take one front-window+> capture through the service below, with a shutter effect and sound, and+> deliver it to a running voice call or the next message. See+> [appshots.md](appshots.md). The Settings card described in Wave 4 moved+> onto the Appshots page as **Allow appshots**.+When the user says something that unambiguously asks Jarvis to *look* — "can yousee this?", "what does that say?", "look at the error" — Jarvis takes **one**capture of the screen the user is actually working on, enriches it with thediff --git a/jarvis/appshot/__init__.py b/jarvis/appshot/__init__.pynew file mode 100644index 0000000000..9b71875947--- /dev/null+++ b/jarvis/appshot/__init__.py@@ -0,0 +1,10 @@+"""Appshots — show the assistant the window you are working in.++An appshot is one capture of the front window, taken on a global shortcut+(both Alt keys by default), from the Appshots page, or when the user asks for+one, and handed to the conversation as context: straight into a running voice+call, otherwise onto the next message. Capture, privacy filtering and+retention are the Screen Context engine's (``jarvis/screen_context``); this+package adds the triggers, the delivery, the shutter effect and the sound+gate. Nothing here is imported at boot (AP-26). See ``docs/appshots.md``.+"""diff --git a/jarvis/appshot/delivery.py b/jarvis/appshot/delivery.pynew file mode 100644index 0000000000..99e1a26bdf--- /dev/null+++ b/jarvis/appshot/delivery.py@@ -0,0 +1,49 @@+"""Hand an appshot to a running voice call.++Every continuous voice session (``jarvis.live``) that can take a picture+exposes ``attach_appshot(image, mime, note)``: GPT-Live puts it into its+thinking backend's context, native live providers receive it as a video+frame. The call stays silent — an appshot is context for what the user says+next, not a question of its own.+"""++from __future__ import annotations++import logging+from typing import Any++log = logging.getLogger(__name__)+++def live_call_running() -> bool:+ try:+ from jarvis.live.runtime import active # noqa: PLC0415++ return any(getattr(session, "is_active", False) for session in active())+ except Exception: # noqa: BLE001 - a probe failure means "no call"+ log.debug("appshot: live-session probe failed", exc_info=True)+ return False+++async def deliver_to_live(image: bytes, mime: str, note: str) -> bool:+ """``True`` when a running voice call accepted the picture."""+ try:+ from jarvis.live.runtime import active # noqa: PLC0415++ sessions: tuple[Any, ...] = active()+ except Exception: # noqa: BLE001 - no live package state means no call+ log.debug("appshot: live runtime unavailable", exc_info=True)+ return False+ for session in sessions:+ attach = getattr(session, "attach_appshot", None)+ if not callable(attach) or not getattr(session, "is_active", False):+ continue+ try:+ if await attach(image, mime, note):+ return True+ except Exception: # noqa: BLE001 - fall back to the next message+ log.warning("appshot: the voice call rejected the appshot", exc_info=True)+ return False+++__all__ = ["deliver_to_live", "live_call_running"]diff --git a/jarvis/appshot/effect.py b/jarvis/appshot/effect.pynew file mode 100644index 0000000000..d97460151d--- /dev/null+++ b/jarvis/appshot/effect.py@@ -0,0 +1,120 @@+"""The shutter effect: flash, then the picture flies into the corner.++Installed as the Screen Context shutter hook, so it fires at the true capture+moment for every look the shared service takes — a shortcut appshot, a spoken+"take an appshot", a "what do you see?". The thumbnail is cut from the raw+frame in memory and piped to the local indicator sidecar; it is never stored+and never published on the event bus.+"""++from __future__ import annotations++import asyncio+import base64+import io+import logging+from typing import Any++log = logging.getLogger(__name__)++#: Longest edge of the thumbnail sent to the sidecar. The card is painted at+#: ~250 logical px, so this stays sharp on a 2x display and small on the pipe.+_THUMB_EDGE = 560++_tasks: set[asyncio.Task[None]] = set()+++def on_shutter(target: Any, size: tuple[int, int], rgb: bytes, monitors: list[dict]) -> None:+ """Screen Context shutter hook — synchronous, schedules the slow part."""+ from jarvis.cu.indicator.controller import get_indicator_controller # noqa: PLC0415++ controller = get_indicator_controller()+ if controller is None:+ return+ # Before the capture's own border dismissal can quit the sidecar.+ controller.hold_for_snap()+ task = asyncio.get_running_loop().create_task(+ _play(controller, tuple(target.bbox), size, rgb, monitors),+ name="appshot-effect",+ )+ _tasks.add(task)+ task.add_done_callback(_tasks.discard)+++async def _play(+ controller: Any,+ bbox: tuple[int, int, int, int],+ size: tuple[int, int],+ rgb: bytes,+ monitors: list[dict],+) -> None:+ try:+ from jarvis.core.config import load_config # noqa: PLC0415++ config = await asyncio.to_thread(load_config)+ if not bool(getattr(getattr(config, "appshot", None), "effect", True)):+ return+ thumb = await asyncio.to_thread(thumbnail_jpeg, size, rgb)+ monitor, rect = placement(bbox, monitors)+ shown = await controller.snap(+ monitor=monitor,+ rect=rect,+ thumb_b64=base64.b64encode(thumb).decode("ascii"),+ )+ if not shown:+ log.info("appshot: shutter effect could not be shown on this desktop")+ except Exception: # noqa: BLE001 - the effect is decoration, never a failure+ log.warning("appshot: shutter effect failed", exc_info=True)+++def thumbnail_jpeg(size: tuple[int, int], rgb: bytes) -> bytes:+ """A small JPEG of the raw frame, for the local effect only."""+ from PIL import Image # noqa: PLC0415++ image = Image.frombytes("RGB", size, rgb)+ image.thumbnail((_THUMB_EDGE, _THUMB_EDGE), Image.Resampling.BILINEAR)+ buffer = io.BytesIO()+ image.save(buffer, format="JPEG", quality=82)+ return buffer.getvalue()+++def placement(+ bbox: tuple[int, int, int, int], monitors: list[dict]+) -> tuple[list[int], list[float]]:+ """The monitor holding ``bbox`` and ``bbox`` as fractions of it.++ Fractions keep the sidecar independent of the capture's pixel units+ (physical on Windows, points on macOS). ``monitors`` is mss-shaped:+ ``[0]`` is the virtual desktop, ``[1:]`` the physical screens.+ """+ left, top, width, height = bbox+ cx, cy = left + width / 2.0, top + height / 2.0+ screens = [m for m in monitors[1:] if isinstance(m, dict)] or [+ m for m in monitors[:1] if isinstance(m, dict)+ ]+ chosen = None+ for mon in screens:+ ml, mt = mon.get("left", 0), mon.get("top", 0)+ mw, mh = mon.get("width", 0), mon.get("height", 0)+ if ml <= cx < ml + mw and mt <= cy < mt + mh:+ chosen = mon+ break+ if chosen is None:+ if not screens:+ return [left, top, max(1, width), max(1, height)], [0.0, 0.0, 1.0, 1.0]+ chosen = screens[0]+ ml, mt = int(chosen.get("left", 0)), int(chosen.get("top", 0))+ mw, mh = max(1, int(chosen.get("width", 1))), max(1, int(chosen.get("height", 1)))+ x0, y0 = max(left, ml), max(top, mt)+ x1, y1 = min(left + width, ml + mw), min(top + height, mt + mh)+ if x1 <= x0 or y1 <= y0:+ return [ml, mt, mw, mh], [0.0, 0.0, 1.0, 1.0]+ return [ml, mt, mw, mh], [+ (x0 - ml) / mw,+ (y0 - mt) / mh,+ (x1 - x0) / mw,+ (y1 - y0) / mh,+ ]+++__all__ = ["on_shutter", "placement", "thumbnail_jpeg"]diff --git a/jarvis/appshot/gesture.py b/jarvis/appshot/gesture.pynew file mode 100644index 0000000000..21bd2318e6--- /dev/null+++ b/jarvis/appshot/gesture.py@@ -0,0 +1,185 @@+"""The "both Alt keys at once" gesture.++The shared hotkey backends fold left and right Alt into one token (the+Windows library knows no difference), so a chord of the two Alt keys cannot be+expressed there. This module watches exactly that gesture with the cheapest+primitive each OS offers — a key-state read every 50 ms, no hook, no tap:++* Windows: ``GetAsyncKeyState`` on ``VK_LMENU`` / ``VK_RMENU`` (AltGr raises+ ``VK_RMENU`` too, so the gesture works on AltGr layouts).+* macOS: ``CGEventSourceKeyState`` on the left/right Option key codes. Needs+ the same Input Monitoring grant as every other global shortcut; without it+ the read stays false and nothing fires.+* Linux/X11: ``XQueryKeymap`` through python-xlib (pynput's own dependency).+* Wayland, headless, missing packages: :func:`make_probe` returns ``None``+ with the reason, and the caller reports the shortcut as unavailable.+"""++from __future__ import annotations++import logging+import threading+from collections.abc import Callable++log = logging.getLogger(__name__)++#: Poll period. A deliberate two-key press lasts far longer than this.+POLL_S = 0.05+#: A second gesture within this window is the same press, not a new one.+_REFIRE_GUARD_S = 0.6++Probe = Callable[[], tuple[bool, bool] | None]+++def _windows_probe() -> Probe | None:+ import ctypes # noqa: PLC0415++ user32 = ctypes.WinDLL("user32", use_last_error=True)+ get_state = user32.GetAsyncKeyState+ get_state.argtypes = [ctypes.c_int]+ get_state.restype = ctypes.c_short+ vk_lmenu, vk_rmenu = 0xA4, 0xA5++ def probe() -> tuple[bool, bool]:+ return bool(get_state(vk_lmenu) & 0x8000), bool(get_state(vk_rmenu) & 0x8000)++ return probe+++def _macos_probe() -> Probe | None:+ import Quartz # type: ignore[import-untyped] # noqa: PLC0415++ source = Quartz.kCGEventSourceStateHIDSystemState+ key_state = Quartz.CGEventSourceKeyState+ left_option, right_option = 58, 61++ def probe() -> tuple[bool, bool]:+ return bool(key_state(source, left_option)), bool(key_state(source, right_option))++ return probe+++def _x11_probe() -> Probe | None:+ from Xlib import XK # type: ignore[import-untyped] # noqa: PLC0415+ from Xlib.display import Display # type: ignore[import-untyped] # noqa: PLC0415++ display = Display()+ left = display.keysym_to_keycode(XK.XK_Alt_L)+ rights = {+ code+ for code in (+ display.keysym_to_keycode(XK.XK_Alt_R),+ display.keysym_to_keycode(XK.string_to_keysym("ISO_Level3_Shift")),+ )+ if code+ }+ if not left or not rights:+ display.close()+ return None++ def down(keymap: list[int], code: int) -> bool:+ return bool(keymap[code // 8] & (1 << (code % 8)))++ def probe() -> tuple[bool, bool]:+ keymap = display.query_keymap()+ return down(keymap, left), any(down(keymap, code) for code in rights)++ return probe+++def make_probe() -> tuple[Probe | None, str]:+ """The key-state reader for this host, or ``None`` and why not."""+ from jarvis.platform import detect_platform # noqa: PLC0415+ from jarvis.platform.probes import display_present, is_wayland # noqa: PLC0415++ platform = detect_platform()+ try:+ if platform == "win32":+ return _windows_probe(), ""+ if not display_present():+ return None, "No display on this computer, so there is no keyboard to watch."+ if platform == "darwin":+ return _macos_probe(), ""+ if is_wayland():+ return None, (+ "Wayland does not let apps watch global keys. Pick a different "+ "shortcut in an X11 session, or ask for an appshot by voice."+ )+ probe = _x11_probe()+ if probe is None:+ return None, "This keyboard layout has no second Alt key."+ return probe, ""+ except ImportError as exc: # the missing package IS the answer; the caller shows it+ return None, f"The key reader is not installed ({exc.name})."+ except Exception as exc: # noqa: BLE001 - an unreadable keyboard is "unavailable"+ log.warning("appshot: key-state probe failed to start", exc_info=True)+ return None, f"The keyboard could not be read ({type(exc).__name__})."+++class BothAltWatcher:+ """Calls ``on_fire`` once each time both Alt keys go down together."""++ def __init__(+ self,+ on_fire: Callable[[], None],+ *,+ probe: Probe,+ poll_s: float = POLL_S,+ clock: Callable[[], float] | None = None,+ ) -> None:+ import time # noqa: PLC0415++ self._on_fire = on_fire+ self._probe = probe+ self._poll_s = poll_s+ self._clock = clock or time.monotonic+ self._stop = threading.Event()+ self._thread: threading.Thread | None = None+ self._armed = True+ self._last_fire = float("-inf")++ def start(self) -> None:+ if self._thread is not None and self._thread.is_alive():+ return+ self._stop.clear()+ self._thread = threading.Thread(target=self._run, name="appshot-alt-alt", daemon=True)+ self._thread.start()++ def stop(self) -> None:+ self._stop.set()+ thread, self._thread = self._thread, None+ if thread is not None and thread is not threading.current_thread():+ thread.join(timeout=1.0)++ def step(self) -> bool:+ """One poll. ``True`` when this poll fired. Exposed for tests."""+ state = self._probe()+ if state is None:+ return False+ left, right = state+ if not (left or right):+ self._armed = True+ return False+ if left and right and self._armed:+ self._armed = False+ now = self._clock()+ if now - self._last_fire < _REFIRE_GUARD_S:+ return False+ from jarvis.platform.self_input import synthetic_input_recent # noqa: PLC0415++ if synthetic_input_recent():+ return False+ self._last_fire = now+ self._on_fire()+ return True+ return False++ def _run(self) -> None:+ while not self._stop.wait(self._poll_s):+ try:+ self.step()+ except Exception: # noqa: BLE001 - one bad read must not kill the shortcut+ log.debug("appshot: key-state read failed", exc_info=True)+++__all__ = ["POLL_S", "BothAltWatcher", "make_probe"]diff --git a/jarvis/appshot/hotkey.py b/jarvis/appshot/hotkey.pynew file mode 100644index 0000000000..c7d3b62038--- /dev/null+++ b/jarvis/appshot/hotkey.py@@ -0,0 +1,223 @@+"""The global appshot shortcut: armed after boot, re-armed on a settings change.++``[appshot].hotkey`` is either ``alt+alt`` (both Alt keys, watched by+:mod:`jarvis.appshot.gesture`), any combo in the shared hotkey syntax (armed+through the regular per-OS :class:`~jarvis.trigger.hotkey.HotkeyTrigger`), or+empty (off). Only the instance that owns ambient duties arms it — a dev app+beside the live one would otherwise take every appshot twice.+"""++from __future__ import annotations++import asyncio+import contextlib+import logging+from dataclasses import dataclass+from typing import Any++log = logging.getLogger(__name__)++BOTH_ALT = "alt+alt"+++@dataclass(frozen=True, slots=True)+class ShortcutStatus:+ hotkey: str+ armed: bool+ detail: str = ""++ def to_json(self) -> dict[str, Any]:+ return {"hotkey": self.hotkey, "armed": self.armed, "detail": self.detail}+++def normalize_hotkey(value: str) -> str:+ """Canonical spelling: lower case, no spaces, both-Alt aliases folded."""+ combo = "+".join(part.strip().lower() for part in str(value or "").split("+") if part.strip())+ if combo in {"alt+alt", "left_alt+right_alt", "right_alt+left_alt", "alt+altgr", "altgr+alt"}:+ return BOTH_ALT+ return combo+++class AppshotShortcut:+ """Owns whichever listener the configured shortcut needs."""++ def __init__(self, bus: Any) -> None:+ self._bus = bus+ self._loop: asyncio.AbstractEventLoop | None = None+ self._watcher: Any | None = None+ self._trigger_task: asyncio.Task[None] | None = None+ self._busy = False+ self._status = ShortcutStatus(hotkey="", armed=False, detail="Not started yet.")+ self._subscribed = False++ @property+ def status(self) -> ShortcutStatus:+ return self._status++ async def start(self) -> ShortcutStatus:+ self._loop = asyncio.get_running_loop()+ self._subscribe_reload()+ return await self.reload()++ async def reload(self) -> ShortcutStatus:+ """Re-read ``[appshot].hotkey`` and re-arm. Never raises."""+ await self.stop()+ try:+ from jarvis.core.config import load_config # noqa: PLC0415+ from jarvis.core.instance import current_instance # noqa: PLC0415++ config = await asyncio.to_thread(load_config)+ hotkey = normalize_hotkey(config.appshot.hotkey)+ if not hotkey:+ self._status = ShortcutStatus(hotkey="", armed=False, detail="No shortcut set.")+ return self._status+ if not current_instance().owns_ambient_duties:+ self._status = ShortcutStatus(+ hotkey=hotkey,+ armed=False,+ detail="The main app owns global shortcuts; this instance does not arm them.",+ )+ return self._status+ if hotkey == BOTH_ALT:+ self._status = await self._arm_both_alt()+ else:+ self._status = self._arm_combo(hotkey)+ except Exception as exc: # noqa: BLE001 - a bad shortcut must not break boot+ log.warning("appshot: shortcut could not be armed", exc_info=True)+ self._status = ShortcutStatus(+ hotkey="",+ armed=False,+ detail=f"The shortcut failed to start ({type(exc).__name__}).",+ )+ log.info(+ "appshot: shortcut %s (%s)",+ self._status.hotkey or "off",+ "armed" if self._status.armed else self._status.detail,+ )+ return self._status++ async def stop(self) -> None:+ watcher, self._watcher = self._watcher, None+ if watcher is not None:+ await asyncio.to_thread(watcher.stop)+ task, self._trigger_task = self._trigger_task, None+ if task is not None and not task.done():+ task.cancel()+ with contextlib.suppress(asyncio.CancelledError):+ await task++ async def _arm_both_alt(self) -> ShortcutStatus:+ from jarvis.appshot.gesture import BothAltWatcher, make_probe # noqa: PLC0415++ probe, reason = await asyncio.to_thread(make_probe)+ if probe is None:+ return ShortcutStatus(hotkey=BOTH_ALT, armed=False, detail=reason)+ watcher = BothAltWatcher(self._fire_threadsafe, probe=probe)+ watcher.start()+ self._watcher = watcher+ return ShortcutStatus(hotkey=BOTH_ALT, armed=True)++ def _arm_combo(self, hotkey: str) -> ShortcutStatus:+ from jarvis.platform.probes import has_hotkey # noqa: PLC0415+ from jarvis.trigger.hotkey import validate_hotkey # noqa: PLC0415++ verdict = validate_hotkey(hotkey)+ if not getattr(verdict, "ok", True):+ return ShortcutStatus(+ hotkey=hotkey, armed=False, detail=str(getattr(verdict, "reason", "") or "")+ )+ if not has_hotkey():+ return ShortcutStatus(+ hotkey=hotkey,+ armed=False,+ detail="Global shortcuts are not available on this desktop.",+ )+ self._trigger_task = asyncio.get_running_loop().create_task(+ self._run_combo(hotkey), name="appshot-hotkey"+ )+ return ShortcutStatus(hotkey=hotkey, armed=True)++ async def _run_combo(self, hotkey: str) -> None:+ from jarvis.trigger.hotkey import HotkeyTrigger # noqa: PLC0415++ try:+ trigger = HotkeyTrigger({"appshot": [hotkey]})+ async with trigger:+ async for name in trigger.events():+ if name == "appshot":+ self._fire()+ except asyncio.CancelledError:+ raise+ except Exception: # noqa: BLE001 - voice and chat keep working without it+ log.warning("appshot: shortcut listener stopped", exc_info=True)++ def _fire_threadsafe(self) -> None:+ loop = self._loop+ if loop is None or loop.is_closed():+ return+ with contextlib.suppress(RuntimeError):+ loop.call_soon_threadsafe(self._fire)++ def _fire(self) -> None:+ if self._busy:+ return+ self._busy = True+ task = asyncio.get_running_loop().create_task(self._take(), name="appshot-take")+ task.add_done_callback(lambda _t: setattr(self, "_busy", False))++ async def _take(self) -> None:+ from jarvis.appshot.service import take_appshot # noqa: PLC0415++ result = await take_appshot(trigger="hotkey", bus=self._bus)+ if not result.ok:+ log.info("appshot: shortcut press refused — %s", result.message)+ await self._publish_refusal(result.message)++ async def _publish_refusal(self, message: str) -> None:+ try:+ from jarvis.core.events import AppshotTaken # noqa: PLC0415++ await self._bus.publish(+ AppshotTaken(source_layer="appshot", trigger="hotkey", delivered_to="refused",+ target_label=message[:200])+ )+ except Exception: # noqa: BLE001 - the log line above already records it+ log.debug("appshot: refusal receipt failed", exc_info=True)++ def _subscribe_reload(self) -> None:+ if self._subscribed or not hasattr(self._bus, "subscribe"):+ return+ from jarvis.core.events import ConfigReloaded # noqa: PLC0415++ async def _on_reload(event: ConfigReloaded) -> None:+ if any(key.startswith("appshot.") for key in event.changed_keys):+ await self.reload()++ self._bus.subscribe(ConfigReloaded, _on_reload)+ self._subscribed = True+++_shortcut: AppshotShortcut | None = None+++def get_shortcut() -> AppshotShortcut | None:+ return _shortcut+++async def start_appshot_shortcut(bus: Any) -> AppshotShortcut:+ """Boot hook (scheduled after the app is ready, never on the boot path)."""+ global _shortcut+ if _shortcut is None:+ _shortcut = AppshotShortcut(bus)+ await _shortcut.start()+ return _shortcut+++__all__ = [+ "BOTH_ALT",+ "AppshotShortcut",+ "ShortcutStatus",+ "get_shortcut",+ "normalize_hotkey",+ "start_appshot_shortcut",+]diff --git a/jarvis/appshot/service.py b/jarvis/appshot/service.pynew file mode 100644index 0000000000..2eb0b428ee--- /dev/null+++ b/jarvis/appshot/service.py@@ -0,0 +1,198 @@+"""Take an appshot and put it where the conversation will see it.++One entry point for every trigger — the global shortcut, the Appshots page's+test button, the live model's ``take_appshot`` tool — plus+:func:`record_turn_capture` for looks a conversation turn took itself. The+capture always runs through the shared Screen Context service, so the privacy+denylist, redaction, the pre-shutter indicator and the no-disk retention rule+apply unchanged; this module only decides what happens to the picture.+"""++from __future__ import annotations++import asyncio+import logging+import time+import uuid+from dataclasses import dataclass, replace+from typing import Any, Literal++from jarvis.appshot.store import Appshot, get_store++log = logging.getLogger(__name__)++Trigger = Literal["hotkey", "voice", "tool", "button"]++#: Trusted framing in front of the untrusted screen evidence block.+_APPSHOT_PREAMBLE = (+ "APPSHOT: the user deliberately captured their front window to give you "+ "context. Use it for their request. If they only asked you to take an "+ "appshot, confirm it in one short sentence and ask what they want to know."+)+++@dataclass(frozen=True, slots=True)+class AppshotResult:+ status: Literal["captured", "refused"]+ shot: Appshot | None = None+ message: str = ""+ reason_code: str = ""++ @property+ def ok(self) -> bool:+ return self.status == "captured" and self.shot is not None+++def _load_config() -> Any:+ from jarvis.core.config import load_config # noqa: PLC0415++ return load_config()+++def shot_from_context(context: Any, *, trigger: str) -> Appshot:+ """Wrap a finished, redacted Screen Context capture as an appshot."""+ from jarvis.screen_context.turn import model_note # noqa: PLC0415++ target = context.target+ label = "active window" if str(target.kind) == "window" else (+ f"monitor {target.monitor_name}" if target.monitor_name else "selected monitor"+ )+ return Appshot(+ id=uuid.uuid4().hex,+ image=context.image,+ mime=context.mime,+ width=int(context.size[0]),+ height=int(context.size[1]),+ label=label,+ app_name=str(getattr(target.window, "app_name", "") or ""),+ note=f"{_APPSHOT_PREAMBLE}\n{model_note(context)}",+ ui_text=context.ui_text,+ trigger=trigger,+ taken_at=time.time(),+ )+++async def take_appshot(+ *,+ trigger: Trigger,+ bus: Any | None = None,+ deliver: bool = True,+ trace_id: uuid.UUID | None = None,+) -> AppshotResult:+ """Capture the front window once and deliver it per ``[appshot].target``.++ ``deliver=False`` is for a caller that consumes the picture itself (the+ live model's tool). Never raises; a refusal carries the user-facing reason.+ """+ from jarvis.screen_context.models import IntentVerdict, VisualIntent # noqa: PLC0415+ from jarvis.screen_context.turn import get_service # noqa: PLC0415++ try:+ config = await asyncio.to_thread(_load_config)+ if not config.screen_context.enabled:+ return AppshotResult(+ status="refused",+ reason_code="disabled",+ message="Appshots are switched off. Turn them on under Settings > Appshots.",+ )+ service = get_service(bus=bus)+ outcome = await service.capture(+ verdict=IntentVerdict(intent=VisualIntent.WINDOW, evidence=("appshot",)),+ trace_id=trace_id or uuid.uuid4(),+ )+ if outcome.status != "captured" or outcome.context is None:+ return AppshotResult(+ status="refused",+ reason_code=outcome.reason_code or outcome.reason_kind or "refused",+ message=outcome.message or "The appshot could not be taken.",+ )+ if outcome.handle_id:+ service.consume(outcome.handle_id)+ shot = shot_from_context(outcome.context, trigger=trigger)+ except Exception: # noqa: BLE001 - a shortcut press must never crash the app+ log.error("appshot: capture failed", exc_info=True)+ return AppshotResult(+ status="refused",+ reason_code="failure",+ message="The appshot failed. Nothing was captured.",+ )++ store = get_store()+ store.remember(shot, keep_s=float(config.screen_context.deck_preview_s))+ delivered_to = "turn"+ if deliver:+ delivered_to = await _deliver(+ shot,+ target=str(config.appshot.target),+ ttl_s=float(config.screen_context.ttl_s),+ )+ store.mark_delivered(shot.id, delivered_to)+ await _publish(bus, shot, delivered_to)+ log.info(+ "appshot: %s %dx%d via %s -> %s",+ shot.label,+ shot.width,+ shot.height,+ trigger,+ delivered_to,+ )+ return AppshotResult(status="captured", shot=replace(shot, delivered_to=delivered_to))+++async def record_turn_capture(context: Any, *, bus: Any | None, trigger: Trigger) -> None:+ """A conversation turn looked at the screen itself — show it as an appshot."""+ try:+ config = await asyncio.to_thread(_load_config)+ shot = shot_from_context(context, trigger=trigger)+ store = get_store()+ store.remember(shot, keep_s=float(config.screen_context.deck_preview_s))+ store.mark_delivered(shot.id, "turn")+ await _publish(bus, shot, "turn")+ except Exception: # noqa: BLE001 - the turn already has its picture+ log.warning("appshot: could not record the turn's capture", exc_info=True)+++async def _deliver(shot: Appshot, *, target: str, ttl_s: float) -> str:+ from jarvis.appshot.delivery import deliver_to_live # noqa: PLC0415++ if target in ("auto", "voice") and await deliver_to_live(shot.image, shot.mime, shot.note):+ return "voice"+ if target == "voice":+ return "none"+ get_store().park(shot, ttl_s=ttl_s)+ return "message"+++async def _publish(bus: Any | None, shot: Appshot, delivered_to: str) -> None:+ if bus is None:+ return+ try:+ from jarvis.core.events import AppshotTaken # noqa: PLC0415++ await bus.publish(+ AppshotTaken(+ source_layer="appshot",+ appshot_id=shot.id,+ trigger=shot.trigger,+ delivered_to=delivered_to,+ target_label=shot.label,+ width=shot.width,+ height=shot.height,+ )+ )+ except Exception: # noqa: BLE001 - a lost receipt cannot undo the appshot+ log.warning("appshot: receipt publication failed", exc_info=True)+++def take_pending_for_turn() -> Appshot | None:+ """The appshot waiting for the next message, removed on the way out."""+ return get_store().take_pending()+++__all__ = [+ "AppshotResult",+ "record_turn_capture",+ "shot_from_context",+ "take_appshot",+ "take_pending_for_turn",+]diff --git a/jarvis/appshot/store.py b/jarvis/appshot/store.pynew file mode 100644index 0000000000..e5cc260688--- /dev/null+++ b/jarvis/appshot/store.py@@ -0,0 +1,127 @@+"""In-memory home of appshots — nothing here ever touches the disk.++Two slots, with two different promises:++* **pending** — the appshot waiting for the next message. Single use: the+ first turn (or the chat composer) that takes it removes it, and it expires+ after ``[screen_context].ttl_s`` like any other unconsumed capture.+* **latest** — the one picture the Appshots page may show, so the user can see+ what the assistant was shown. Replaced by the next appshot and dropped after+ ``[screen_context].deck_preview_s`` (``0`` = never kept).++Both hold bytes, never paths (the Screen Context retention contract).+"""++from __future__ import annotations++import threading+import time+from dataclasses import dataclass, replace+++@dataclass(frozen=True, slots=True)+class Appshot:+ """One captured, already-redacted window or monitor."""++ id: str+ image: bytes+ mime: str+ width: int+ height: int+ #: Metadata-safe label ("active window" / "monitor X") for events and logs.+ label: str+ #: Front application name — shown in the app only, never published.+ app_name: str+ #: Model-facing evidence block that rides with the image.+ note: str+ #: Scrubbed on-screen text, when the accessibility layer had any.+ ui_text: str+ #: ``hotkey`` | ``voice`` | ``tool`` | ``button``.+ trigger: str+ #: Wall-clock seconds.+ taken_at: float+ delivered_to: str = ""++ def meta(self) -> dict[str, object]:+ """What the app may show about it — no pixels."""+ return {+ "id": self.id,+ "width": self.width,+ "height": self.height,+ "label": self.label,+ "app_name": self.app_name,+ "trigger": self.trigger,+ "taken_at": self.taken_at,+ "delivered_to": self.delivered_to,+ }+++class AppshotStore:+ """Thread-safe pending + latest slots with monotonic expiry."""++ def __init__(self, *, clock=time.monotonic) -> None:+ self._clock = clock+ self._lock = threading.Lock()+ self._pending: Appshot | None = None+ self._pending_until = 0.0+ self._latest: Appshot | None = None+ self._latest_until = 0.0++ def remember(self, shot: Appshot, *, keep_s: float) -> None:+ """Make ``shot`` the picture the Appshots page shows."""+ with self._lock:+ if keep_s <= 0:+ self._latest = None+ return+ self._latest = shot+ self._latest_until = self._clock() + keep_s++ def park(self, shot: Appshot, *, ttl_s: float) -> None:+ """Hold ``shot`` for the next message, replacing an older one."""+ with self._lock:+ self._pending = shot+ self._pending_until = self._clock() + max(1.0, ttl_s)++ def take_pending(self, shot_id: str | None = None) -> Appshot | None:+ """Remove and return the pending appshot (optionally only a given id)."""+ with self._lock:+ shot = self._live_pending()+ if shot is None or (shot_id and shot.id != shot_id):+ return None+ self._pending = None+ return shot++ def peek_pending(self) -> Appshot | None:+ with self._lock:+ return self._live_pending()++ def latest(self) -> Appshot | None:+ with self._lock:+ if self._latest is not None and self._clock() >= self._latest_until:+ self._latest = None+ return self._latest++ def mark_delivered(self, shot_id: str, delivered_to: str) -> None:+ with self._lock:+ if self._latest is not None and self._latest.id == shot_id:+ self._latest = replace(self._latest, delivered_to=delivered_to)++ def clear(self) -> None:+ with self._lock:+ self._pending = None+ self._latest = None++ def _live_pending(self) -> Appshot | None:+ if self._pending is not None and self._clock() >= self._pending_until:+ self._pending = None+ return self._pending+++_STORE = AppshotStore()+++def get_store() -> AppshotStore:+ return _STORE+++__all__ = ["Appshot", "AppshotStore", "get_store"]diff --git a/jarvis/audio/effects.py b/jarvis/audio/effects.pyindex 97fa5c21b9..898aec28b5 100644--- a/jarvis/audio/effects.py+++ b/jarvis/audio/effects.py@@ -65,6 +65,10 @@ class AudioEffectsService:config = await asyncio.to_thread(self._load_config)if not bool(getattr(getattr(config, "ui", None), "sound_effects", True)):return+ # Every shared-service capture is an appshot to the user, so the+ # Appshots page's "play sound" switch owns this cue.+ if not bool(getattr(getattr(config, "appshot", None), "sound", True)):+ returnplayer = self._bound_player()if player is None:player = await asyncio.to_thread(self._make_player, config)diff --git a/jarvis/brain/manager.py b/jarvis/brain/manager.pyindex aa89940dc7..8fe5785077 100644--- a/jarvis/brain/manager.py+++ b/jarvis/brain/manager.py@@ -2697,6 +2697,25 @@ _MID_ANSWER_ERROR_PHRASES: dict[str, str] = {}+#: Source layers that run without a person waiting on the reply. An appshot+#: parked for "the next message" belongs to the user's next message, never to+#: a scheduled task or a background worker that happens to run first.+_AUTOMATED_LAYER_PREFIXES = (+ "tasks",+ "workflows",+ "skills",+ "missions",+ "society",+ "supervisor",+ "tool.",+)+++def _takes_pending_appshot(source_layer: str | None) -> bool:+ layer = (source_layer or "").strip().lower()+ return not layer.startswith(_AUTOMATED_LAYER_PREFIXES)++class BrainManager:"""Top-level orchestrator with intent router and smart fallback."""@@ -10028,6 +10047,7 @@ class BrainManager:locale=locale,bus=self._bus,trace_id=trace_id,+ allow_pending_appshot=_takes_pending_appshot(source_layer),)if outcome.status == "clarify":pending_map[confirm_key] = _PendingScreenConfirm(diff --git a/jarvis/brain/tool_gateway.py b/jarvis/brain/tool_gateway.pyindex 53417ad42e..7319867ae7 100644--- a/jarvis/brain/tool_gateway.py+++ b/jarvis/brain/tool_gateway.py@@ -105,6 +105,7 @@ class BrainSupervisorToolGateway:def _voice_tools(self) -> dict[str, Any]:from jarvis.harness.computer_use_context import peek_computer_use_context+ from jarvis.plugins.tool.appshot import AppshotToolfrom jarvis.plugins.tool.live_screen import LiveScreenTooltools = self._live_tools()@@ -113,6 +114,10 @@ class BrainSupervisorToolGateway:if context is not None and getattr(computer_use, "enabled", True):tools.update(context.tools or {})tools["screen_snapshot"] = LiveScreenTool()+ # "Take an appshot" said in a live call: the shortcut's capture, with+ # its effect and sound. Voice-only — a brain turn gets the same+ # picture from its own Screen Context step.+ tools["take_appshot"] = AppshotTool()if self._workspace_tool is not None:# Live delegates coding to one addressed service. The old prompt# tools silently choose an ambient pane and cannot safely coexist.diff --git a/jarvis/core/config.py b/jarvis/core/config.pyindex b3bc4822fb..88a044ae68 100644--- a/jarvis/core/config.py+++ b/jarvis/core/config.py@@ -2575,6 +2575,38 @@ class ScreenContextConfig(BaseModel):ocr_enabled: bool = False+class AppshotConfig(BaseModel):+ """Top-level ``[appshot]`` config — show the assistant the front window.++ An appshot is one capture of the window the user is working in, taken on+ a global shortcut or when the user asks for one, and handed to the+ conversation as context (``jarvis/appshot/``). It runs through the Screen+ Context capture engine, so every ``[screen_context]`` privacy rule applies.++ The master switch is ``[screen_context].enabled``: an appshot and a spoken+ "what do you see?" are the same look, so one switch governs both. Every+ key below is read by ``jarvis.appshot`` (AP-31).+ """++ model_config = {"extra": "allow"}++ #: Global shortcut. ``alt+alt`` means both Alt keys pressed together (read+ #: by ``jarvis.appshot.gesture``); any other value is an ordinary combo in+ #: the shared hotkey syntax; an empty string switches the shortcut off.+ hotkey: str = "alt+alt"++ #: Where a shortcut appshot goes. ``auto``: into the running voice call,+ #: otherwise onto the next message. ``message``: always onto the next+ #: message. ``voice``: only into a running voice call.+ target: Literal["auto", "message", "voice"] = "auto"++ #: Shutter sound on every appshot (also gated by ``[ui].sound_effects``).+ sound: bool = True++ #: Flash and thumbnail animation on the captured window.+ effect: bool = True++class ComputerUseConfig(BaseModel):"""Top-level ``[computer_use]`` config for the Computer-Use harness.@@ -4366,6 +4398,9 @@ class JarvisConfig(BaseModel):# One-shot, intent-driven screen look (jarvis/screen_context/). Distinct# from ``[vision]`` above, which governs the always-on observation path.screen_context: ScreenContextConfig = Field(default_factory=ScreenContextConfig)+ # Appshots: the front window as conversation context, on a shortcut or on+ # request (jarvis/appshot/). Captures through Screen Context above.+ appshot: AppshotConfig = Field(default_factory=AppshotConfig)# Phase 5/6 — Computer-Use-POAV-Harness (ADR-0008).computer_use: ComputerUseConfig = Field(default_factory=ComputerUseConfig)# Low-latency local-action gate. Hidden tools only; never exposed in thediff --git a/jarvis/core/config_writer.py b/jarvis/core/config_writer.pyindex 2538b783c1..9445eac3db 100644--- a/jarvis/core/config_writer.py+++ b/jarvis/core/config_writer.py@@ -993,6 +993,41 @@ def set_screen_context_settings(_atomic_write(path, out)+#: Keys ``[appshot]`` accepts — same allowlist reasoning as Screen Context.+#: The master switch is ``[screen_context].enabled``, written by the setter above.+APPSHOT_SETTING_KEYS: frozenset[str] = frozenset({"hotkey", "target", "sound", "effect"})+++def set_appshot_settings(+ values: dict[str, str | bool],+ *,+ path: Path = DEFAULT_CONFIG_FILE,+) -> None:+ """Persist a validated ``[appshot]`` patch in one atomic replacement."""+ unknown = set(values).difference(APPSHOT_SETTING_KEYS)+ if unknown:+ raise ValueError(f"unknown appshot setting(s): {sorted(unknown)!r}")+ if not values:+ return+ path = _ensure_writable_config_path(path)+ with _WRITE_LOCK:+ raw = path.read_text(encoding="utf-8")+ had_bom = raw.startswith(_BOM)+ if had_bom:+ raw = raw[len(_BOM) :]+ doc: TOMLDocument = tomlkit.parse(raw)+ section = doc.get("appshot")+ if section is None:+ section = tomlkit.table()+ doc["appshot"] = section+ for key, value in values.items():+ section[key] = value+ out = tomlkit.dumps(doc)+ if had_bom:+ out = _BOM + out+ _atomic_write(path, out)++def set_reply_language(name: str, *, path: Path = DEFAULT_CONFIG_FILE) -> None:"""Persist the user-facing reply-language pin in ``[brain] reply_language``.diff --git a/jarvis/core/events.py b/jarvis/core/events.pyindex 63a09884a8..e16ac9dc9d 100644--- a/jarvis/core/events.py+++ b/jarvis/core/events.py@@ -1227,6 +1227,25 @@ class ScreenCaptureCompleted(Event):ui_text_source: str = "none"+@dataclass(frozen=True, slots=True)+class AppshotTaken(Event):+ """An appshot was captured and handed to the conversation.++ Metadata only, like ``ScreenCaptureCompleted``: pixels stay in+ ``jarvis.appshot`` memory and are served to the app on request.+ """++ appshot_id: str = ""+ #: What started it: ``hotkey`` | ``voice`` | ``tool`` | ``button``.+ trigger: str = ""+ #: Where it went: ``voice`` (running call) | ``message`` (next turn) |+ #: ``turn`` (the turn that asked for it) | ``none``.+ delivered_to: str = ""+ target_label: str = ""+ width: int = 0+ height: int = 0++@dataclass(frozen=True, slots=True)class ActionPlanned(Event):"""The CU loop planner proposed the next action (before execution)."""diff --git a/jarvis/cu/indicator/controller.py b/jarvis/cu/indicator/controller.pyindex 6c34074472..41ebef2343 100644--- a/jarvis/cu/indicator/controller.py+++ b/jarvis/cu/indicator/controller.py@@ -31,6 +31,7 @@ import queueimport subprocessimport sysimport threading+import timefrom contextlib import contextmanager, suppressfrom typing import Anyfrom uuid import UUID@@ -51,6 +52,9 @@ _ESC_HINTS: dict[str, str] = {_QUIT_GRACE_S = 1.5_BLANK_ACK_TIMEOUT_S = 0.15_SHOW_ACK_TIMEOUT_S = 1.2+#: How long the sidecar stays up for one appshot shutter effect (flash, rest+#: in the corner, slide out — see ``renderer._SNAP_TOTAL_MS``) plus slack.+_SNAP_LIFETIME_S = 3.4def screen_indicator_capability() -> tuple[bool, str]:@@ -109,6 +113,9 @@ class CUIndicatorController:self._acks: queue.Queue[str] = queue.Queue()self._esc_task: asyncio.Task | None = Noneself._sidecar_warned = False+ # Monotonic deadline until which an appshot effect owns the sidecar.+ self._snap_until = 0.0+ self._idle_quit_task: asyncio.Task | None = None# ------------------------------------------------------------------ wiringdef wire(self) -> None:@@ -229,6 +236,15 @@ class CUIndicatorController:protocol.CMD_HIDE,_SHOW_ACK_TIMEOUT_S,)+ if time.monotonic() < self._snap_until:+ # An appshot effect is still on screen; quit once it has played.+ self._schedule_idle_quit()+ return+ await self._quit_sidecar()++ async def _quit_sidecar(self) -> None:+ if self._proc is None:+ returnawait asyncio.to_thread(self._send_and_wait,protocol.CMD_QUIT,@@ -239,6 +255,59 @@ class CUIndicatorController:if proc is not None:await asyncio.to_thread(self._reap, proc)+ # ---------------------------------------------------------- appshot snap+ def hold_for_snap(self) -> None:+ """Keep the sidecar alive for an effect that is about to be sent.++ Called synchronously at the shutter, BEFORE the capture's own border+ dismissal can quit the sidecar — the thumbnail is encoded a moment+ later, and a respawn would cost the effect a second of Qt start-up.+ """+ self._snap_until = max(self._snap_until, time.monotonic() + _SNAP_LIFETIME_S)++ async def snap(self, *, monitor: list[int], rect: list[float], thumb_b64: str) -> bool:+ """Play the appshot shutter effect. ``False`` when it cannot run here."""+ ok, reason = self._border_capability()+ if not ok:+ log.debug("[appshot-effect] unavailable: %s", reason)+ return False+ async with self._lock:+ self.hold_for_snap()+ await asyncio.to_thread(self._spawn_sidecar)+ if self._proc is None:+ return False+ shown = await asyncio.to_thread(+ self._send_and_wait,+ protocol.CMD_SNAP,+ _SHOW_ACK_TIMEOUT_S,+ monitor=list(monitor),+ rect=list(rect),+ thumb=thumb_b64,+ )+ self._schedule_idle_quit()+ return shown++ def _schedule_idle_quit(self) -> None:+ task = self._idle_quit_task+ if task is not None and not task.done():+ return+ try:+ loop = asyncio.get_running_loop()+ except RuntimeError: # no loop = interpreter shutdown; the sidecar dies with us+ return+ self._idle_quit_task = loop.create_task(self._idle_quit(), name="appshot-effect-quit")++ async def _idle_quit(self) -> None:+ while True:+ remaining = self._snap_until - time.monotonic()+ if remaining <= 0:+ break+ await asyncio.sleep(remaining + 0.05)+ async with self._lock:+ if self._active or self._screen_active or time.monotonic() < self._snap_until:+ return+ await self._quit_sidecar()+@staticmethoddef _border_capability() -> tuple[bool, str]:try:@@ -463,6 +532,11 @@ class CUIndicatorController:_controller: CUIndicatorController | None = None+def get_indicator_controller() -> CUIndicatorController | None:+ """The wired controller, or ``None`` before boot wired one."""+ return _controller++def wire_cu_indicator(bus: Any) -> CUIndicatorController | None:"""Idempotent boot hook: subscribe the indicator controller to ``bus``.@@ -484,4 +558,4 @@ def wire_cu_indicator(bus: Any) -> CUIndicatorController | None:return controller-__all__ = ["CUIndicatorController", "wire_cu_indicator"]+__all__ = ["CUIndicatorController", "get_indicator_controller", "wire_cu_indicator"]diff --git a/jarvis/cu/indicator/protocol.py b/jarvis/cu/indicator/protocol.pyindex 6010c4d5c6..0fed2596a6 100644--- a/jarvis/cu/indicator/protocol.py+++ b/jarvis/cu/indicator/protocol.py@@ -8,6 +8,9 @@ One JSON object per line on the sidecar's stdin::{"cmd": "blank"} # hide INSTANTLY (capture guard){"cmd": "unblank"} # restore after a frame grab{"cmd": "quit"} # exit the sidecar+ {"cmd": "snap", "monitor": [l, t, w, h], # appshot shutter effect:+ "rect": [fx, fy, fw, fh], # flash the captured rect and+ "thumb": "<base64 jpeg>"} # fly its thumbnail to a cornerThe sidecar answers each command with one JSON line on stdout::@@ -28,9 +31,10 @@ CMD_HIDE = "hide"CMD_BLANK = "blank"CMD_UNBLANK = "unblank"CMD_QUIT = "quit"+CMD_SNAP = "snap"ALL_COMMANDS = frozenset(- {CMD_SHOW, CMD_HIDE, CMD_BLANK, CMD_UNBLANK, CMD_QUIT}+ {CMD_SHOW, CMD_HIDE, CMD_BLANK, CMD_UNBLANK, CMD_QUIT, CMD_SNAP})#: Sidecar exit code when no usable GUI stack exists (PySide6 missing or@@ -81,6 +85,7 @@ __all__ = ["CMD_HIDE","CMD_QUIT","CMD_SHOW",+ "CMD_SNAP","CMD_UNBLANK","EXIT_NO_GUI","decode_ack",diff --git a/jarvis/cu/indicator/renderer.py b/jarvis/cu/indicator/renderer.pyindex 60d22cde9e..e1d11c3b5a 100644--- a/jarvis/cu/indicator/renderer.py+++ b/jarvis/cu/indicator/renderer.py@@ -25,8 +25,10 @@ from collections import dequefrom contextlib import suppressfrom PySide6.QtCore import (+ QByteArray,QEasingCurve,QObject,+ QRectF,Qt,QVariantAnimation,Signal,@@ -37,8 +39,10 @@ from PySide6.QtGui import (QFont,QFontMetricsF,QGuiApplication,+ QImage,QLinearGradient,QPainter,+ QPainterPath,QPen,)from PySide6.QtWidgets import QApplication, QWidget@@ -178,6 +182,184 @@ class _GlowWindow(QWidget):)+# ---------------------------------------------------------------------------+# Appshot shutter effect — the phone-screenshot moment.+#+# A white flash over the captured surface, then the picture shrinks into the+# bottom-right corner of that monitor, rests there as a rounded thumbnail and+# slides out. Purely local: the thumbnail arrives over this process's stdin,+# is painted, and is dropped with the window. Click-through like the glow.+# ---------------------------------------------------------------------------++_SNAP_FLASH_MS = 200+_SNAP_FLY_START_MS = 90+_SNAP_FLY_MS = 430+_SNAP_HOLD_UNTIL_MS = 2500+_SNAP_OUT_MS = 320+_SNAP_TOTAL_MS = _SNAP_HOLD_UNTIL_MS + _SNAP_OUT_MS+_SNAP_THUMB_W = 320+_SNAP_THUMB_MAX_H = 240+_SNAP_MARGIN = 28+_SNAP_RADIUS = 12.0+++def _ease_out_cubic(x: float) -> float:+ x = max(0.0, min(1.0, x))+ return 1.0 - (1.0 - x) ** 3+++def _lerp(a: float, b: float, t: float) -> float:+ return a + (b - a) * t+++def _match_screen(monitor: list[float]):+ """The QScreen that best matches a capture-coordinate monitor rect.++ Capture coordinates are physical pixels on Windows and points on macOS,+ while Qt reports logical geometry — so every plausible interpretation of+ each screen is scored and the closest one wins. Primary on no data.+ """+ screens = QGuiApplication.screens()+ primary = QGuiApplication.primaryScreen()+ if not screens or len(monitor) != 4:+ return primary+ left, top, width, height = (float(v) for v in monitor)+ best, best_score = primary, float("inf")+ for screen in screens:+ g = screen.geometry()+ dpr = float(screen.devicePixelRatio() or 1.0)+ candidates = (+ (g.x(), g.y(), g.width(), g.height()),+ (g.x(), g.y(), g.width() * dpr, g.height() * dpr),+ (g.x() * dpr, g.y() * dpr, g.width() * dpr, g.height() * dpr),+ )+ for cx, cy, cw, ch in candidates:+ score = abs(cx - left) + abs(cy - top) + abs(cw - width) + abs(ch - height)+ if score < best_score:+ best, best_score = screen, score+ return best+++class _SnapWindow(QWidget):+ """One monitor-sized, click-through canvas for a single shutter effect."""++ def __init__(self, screen, rect_frac: list[float], thumb: QImage, on_done) -> None:+ super().__init__(None)+ self.setWindowFlags(+ Qt.WindowType.FramelessWindowHint+ | Qt.WindowType.WindowStaysOnTopHint+ | Qt.WindowType.WindowTransparentForInput+ | Qt.WindowType.WindowDoesNotAcceptFocus+ | Qt.WindowType.Tool+ | Qt.WindowType.NoDropShadowWindowHint+ )+ self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground)+ self.setAttribute(Qt.WidgetAttribute.WA_TransparentForMouseEvents)+ self.setAttribute(Qt.WidgetAttribute.WA_ShowWithoutActivating)+ if sys.platform == "darwin":+ mac_always_show = getattr(Qt.WidgetAttribute, "WA_MacAlwaysShowToolWindow", None)+ if mac_always_show is not None:+ self.setAttribute(mac_always_show)+ self.setScreen(screen)+ self.setGeometry(screen.geometry())+ self._thumb = thumb+ self._on_done = on_done+ w, h = float(screen.geometry().width()), float(screen.geometry().height())+ fx, fy, fw, fh = (max(0.0, min(1.0, float(v))) for v in rect_frac)+ self._src = QRectF(fx * w, fy * h, max(1.0, fw * w), max(1.0, fh * h))+ aspect = (+ thumb.height() / max(1, thumb.width())+ if not thumb.isNull()+ else (fh * h) / max(fw * w, 1e-6)+ )+ tw = float(_SNAP_THUMB_W)+ th = tw * aspect+ if th > _SNAP_THUMB_MAX_H:+ th = float(_SNAP_THUMB_MAX_H)+ tw = th / max(aspect, 1e-6)+ self._dst = QRectF(w - tw - _SNAP_MARGIN, h - th - _SNAP_MARGIN, tw, th)+ self._t = 0.0+ self._anim = QVariantAnimation(self)+ self._anim.setStartValue(0.0)+ self._anim.setEndValue(float(_SNAP_TOTAL_MS))+ self._anim.setDuration(_SNAP_TOTAL_MS)+ self._anim.valueChanged.connect(self._on_tick)+ self._anim.finished.connect(self._finish)++ def start(self) -> None:+ self.show()+ self._anim.start()++ def finish_now(self) -> None:+ self._anim.stop()+ self._finish()++ def _on_tick(self, value) -> None:+ self._t = float(value)+ self.update()++ def _finish(self) -> None:+ self.hide()+ callback, self._on_done = self._on_done, None+ if callback is not None:+ callback(self)+ self.deleteLater()++ def showEvent(self, event) -> None: # noqa: N802 (Qt override)+ super().showEvent(event)+ hwnd = int(self.winId())+ harden_window(hwnd)+ exclude_from_capture(hwnd)++ def paintEvent(self, event) -> None: # noqa: N802 (Qt override)+ del event+ t = self._t+ painter = QPainter(self)+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)+ painter.setRenderHint(QPainter.RenderHint.SmoothPixmapTransform)++ fly = _ease_out_cubic((t - _SNAP_FLY_START_MS) / _SNAP_FLY_MS)+ out = _ease_out_cubic((t - _SNAP_HOLD_UNTIL_MS) / _SNAP_OUT_MS)+ rect = QRectF(+ _lerp(self._src.x(), self._dst.x(), fly) + 48.0 * out,+ _lerp(self._src.y(), self._dst.y(), fly),+ _lerp(self._src.width(), self._dst.width(), fly),+ _lerp(self._src.height(), self._dst.height(), fly),+ )+ radius = _SNAP_RADIUS * fly+ opacity = 1.0 - out++ if not self._thumb.isNull() and opacity > 0.0:+ painter.setOpacity(opacity)+ # Soft shadow grows in as the card lifts off the surface.+ if fly > 0.0:+ for spread, alpha in ((10.0, 18), (5.0, 30), (2.0, 46)):+ shadow = rect.adjusted(-spread, -spread + 4, spread, spread + 4)+ painter.setPen(Qt.PenStyle.NoPen)+ painter.setBrush(QColor(0, 0, 0, int(alpha * fly)))+ painter.drawRoundedRect(shadow, radius + spread, radius + spread)+ clip = QPainterPath()+ clip.addRoundedRect(rect, radius, radius)+ painter.save()+ painter.setClipPath(clip)+ painter.drawImage(rect, self._thumb)+ painter.restore()+ if fly > 0.0:+ pen = QPen(QColor(255, 255, 255, int(235 * fly)))+ pen.setWidthF(2.5)+ painter.setPen(pen)+ painter.setBrush(Qt.BrushStyle.NoBrush)+ painter.drawRoundedRect(rect, radius, radius)++ flash = 1.0 - _ease_out_cubic(t / _SNAP_FLASH_MS)+ if flash > 0.0:+ painter.setOpacity(1.0)+ painter.setPen(Qt.PenStyle.NoPen)+ painter.setBrush(QColor(255, 255, 255, int(225 * flash)))+ painter.drawRect(self._src)+ painter.end()++class Renderer(QObject):"""Owns the per-monitor windows, the animations, and the IPC slots."""@@ -185,6 +367,7 @@ class Renderer(QObject):super().__init__()self._app = appself._windows: list[_GlowWindow] = []+ self._snaps: list[_SnapWindow] = []self._hint = ""self._active = False # "show" was requested and not yet "hide"self._blanked = False # capture guard currently hiding the border@@ -242,6 +425,8 @@ class Renderer(QObject):self._blank()elif cmd == protocol.CMD_UNBLANK:self._unblank()+ elif cmd == protocol.CMD_SNAP:+ self._snap(payload)elif cmd == protocol.CMD_QUIT:_ack(cmd)self._app.quit()@@ -274,7 +459,33 @@ class Renderer(QObject):self._active = Falseself._fade_to(0.0)+ def _snap(self, payload: dict) -> None:+ thumb = QImage()+ raw = payload.get("thumb")+ if isinstance(raw, str) and raw:+ thumb.loadFromData(QByteArray.fromBase64(raw.encode("ascii")))+ rect = payload.get("rect")+ if not isinstance(rect, list) or len(rect) != 4:+ rect = [0.0, 0.0, 1.0, 1.0]+ monitor = payload.get("monitor")+ screen = _match_screen(monitor if isinstance(monitor, list) else [])+ if screen is None:+ return+ # One effect at a time: a second appshot replaces the resting card.+ for old in list(self._snaps):+ old.finish_now()+ win = _SnapWindow(screen, rect, thumb, self._snap_done)+ self._snaps.append(win)+ win.start()++ def _snap_done(self, win) -> None:+ with suppress(ValueError):+ self._snaps.remove(win)+def _blank(self) -> None:+ # A resting thumbnail must never end up inside the next capture.+ for snap in list(self._snaps):+ snap.finish_now()if not self._active:returnself._blanked = Truediff --git a/jarvis/live/config.py b/jarvis/live/config.pyindex a135806d11..0ddff98295 100644--- a/jarvis/live/config.py+++ b/jarvis/live/config.py@@ -60,8 +60,10 @@ class LiveConfig(BaseModel):"Backchannel policy: Use moderate backchannels. ""Interruption policy: Listen when interrupted. ""Stopping speech does not cancel work. "- "Delegation policy: Backend tools: files, applications, screen, settings, memory, "- "connected services, web search and agents. Delegate when a request needs these "+ "Delegation policy: Backend tools: files, applications, screen, appshots, "+ "settings, memory, connected services, web search and agents. An appshot is a "+ "picture of the user's front window; asking for one or about one is a backend "+ "request. Delegate when a request needs these ""capabilities, careful reasoning, or changes an ongoing task. Do not delegate ""greetings, simple conversation, or repeating a still-current result. Delegate ""before answering anything dependent on tools; never guess their results. "diff --git a/jarvis/live/native.py b/jarvis/live/native.pyindex 5a5cb1bc55..ec2ab98e76 100644--- a/jarvis/live/native.py+++ b/jarvis/live/native.py@@ -59,6 +59,7 @@ class NativeLiveVoiceSession(LiveVoiceSession):"call_tool for any tool not declared directly. For computer control, capture ""screen_snapshot, inspect the image, call the desktop primitives, and verify ""the result with a new snapshot. Do not call a separate computer-use harness. "+ "When the user asks for an appshot, call take_appshot. ""Request confirmation for pending approvals. Use confirm_action only after ""explicit approval. A started job is not complete. Never invent tool results. ""Use workspace-orchestrate for coding tasks: inspect and resolve project, "@@ -375,6 +376,20 @@ class NativeLiveVoiceSession(LiveVoiceSession):await self._connection.send_text(text)return True+ async def attach_appshot(self, image: bytes, mime: str, note: str) -> bool:+ """Hand the appshot to the native model as a video frame, silently.++ A text turn would make the model answer right away; the frame alone is+ what it looks at when the user asks. Servers without image input+ decline, and the caller parks the appshot for the next message.+ """+ del note+ send_image = getattr(self._connection, "send_image", None)+ if not self.is_active or not callable(send_image):+ return False+ await send_image(image, mime)+ return True+async def end(self, *, reason: str = "client_stop") -> None:if self._ended:returndiff --git a/jarvis/live/session.py b/jarvis/live/session.pyindex 7c7a82480a..5d9292a837 100644--- a/jarvis/live/session.py+++ b/jarvis/live/session.py@@ -937,6 +937,46 @@ class LiveVoiceSession:)return True+ async def attach_appshot(self, image: bytes, mime: str, note: str) -> bool:+ """Put an appshot into the thinking backend's context, silently.++ The voice model only hears that it exists; the picture itself goes to+ the backend conversation, which answers every question about it. No+ response is requested: the user's next words are the question.+ """+ if not self.is_active or self._recovering or self._resume_needs_input:+ return False+ await self._connection.send(+ {+ "type": "session.thinking.append",+ "delegation_id": None,+ "content": (+ "The user just took an appshot of their front window. It is "+ "in your backend's context. Delegate any question about what "+ "they are looking at; do not describe it yourself, and do "+ "not comment on the appshot unless asked."+ ),+ }+ )+ await self._connection.send(+ {+ "type": "response.item.create",+ "item": {+ "type": "message",+ "role": "user",+ "content": [+ {"type": "input_text", "text": note},+ {+ "type": "input_image",+ "image_url": f"data:{mime};base64,"+ + base64.b64encode(image).decode("ascii"),+ },+ ],+ },+ }+ )+ return True+async def end(self, *, reason: str = "client_stop") -> None:if self._ended:returndiff --git a/jarvis/plugins/tool/appshot.py b/jarvis/plugins/tool/appshot.pynew file mode 100644index 0000000000..e5d8300970--- /dev/null+++ b/jarvis/plugins/tool/appshot.py@@ -0,0 +1,60 @@+"""``take_appshot`` — the live voice model's way to honour "take an appshot".++A running voice call does not pass through the brain's turn pipeline, so a+spoken "take an appshot" reaches the live model as words. This tool gives it+the same capture the global shortcut takes — the front window, privacy+filtered, with the shutter effect and sound — and returns the picture as an+image input for the answer.+"""++from __future__ import annotations+++def _app_bus():+ from jarvis.core.runtime_refs import get_brain_manager, get_web_app++ app = get_web_app()+ bus = getattr(getattr(app, "state", None), "bus", None)+ if bus is not None:+ return bus+ return getattr(get_brain_manager(), "_bus", None)+++class AppshotTool:+ name = "take_appshot"+ description = (+ "Take an appshot: capture the user's front window once, with the usual "+ "privacy filter, so you can see what they are working on. Use when the "+ "user asks for an appshot or asks you to look at their window or screen."+ )+ risk_tier = "monitor"+ schema = {"type": "object", "properties": {}, "additionalProperties": False}++ async def execute(self, args, ctx):+ import base64++ from jarvis.appshot.service import take_appshot+ from jarvis.core.protocols import ToolResult++ del args+ result = await take_appshot(+ trigger="tool",+ bus=_app_bus(),+ deliver=False,+ trace_id=getattr(ctx, "trace_id", None),+ )+ if not result.ok or result.shot is None:+ return ToolResult(False, None, result.message or "The appshot could not be taken.")+ shot = result.shot+ return ToolResult(+ True,+ {+ "description": f"Appshot of the {shot.label}, {shot.width}x{shot.height}.",+ "app": shot.app_name,+ "evidence": shot.note,+ "_image": {+ "mime": shot.mime,+ "data": base64.b64encode(shot.image).decode("ascii"),+ },+ },+ )diff --git a/jarvis/plugins/tool/navigate.py b/jarvis/plugins/tool/navigate.pyindex c1e3800e74..d8494a8995 100644--- a/jarvis/plugins/tool/navigate.py+++ b/jarvis/plugins/tool/navigate.py@@ -54,6 +54,8 @@ KNOWN: frozenset[str] = frozenset("feedback","agent-instructions","wallpaper",+ # Appshots: the shortcut and destination for showing the front window.+ "appshots","dictionary","dictation","voice-shortcuts",@@ -199,6 +201,9 @@ _ALIASES: dict[str, str] = {"hintergrund": "wallpaper", # i18n-allow: input vocab"hintergrundbild": "wallpaper", # i18n-allow: input vocab"fondo de pantalla": "wallpaper", # i18n-allow: input vocab+ "appshot": "appshots",+ "app shots": "appshots",+ "app shot": "appshots","task bar": "taskbar","taskleiste": "taskbar",# The Artifacts section (section id kept as "visualization" — its 2026-08diff --git a/jarvis/screen_context/intent.py b/jarvis/screen_context/intent.pyindex c3010ced7d..ce9f20cb38 100644--- a/jarvis/screen_context/intent.py+++ b/jarvis/screen_context/intent.py@@ -349,6 +349,44 @@ _SCREEN_INTENT_RE: re.Pattern[str] = re.compile()+# --------------------------------------------------------------------------+# Appshot — the feature named outright+# --------------------------------------------------------------------------++# "Appshot" is the product's own word for "show the assistant my front+# window", so naming it is the least ambiguous look request there is. Speech+# recognition spells it every way ("App Shot", "App-Shot", "Epp Shot").+_APPSHOT_WORD = r"\b(?:app|apps|ap|epp|ep)[\s-]?shots?\b"+_APPSHOT_RE: re.Pattern[str] = re.compile(_APPSHOT_WORD, re.IGNORECASE)++# Asking ABOUT the feature is not asking for one.+_APPSHOT_META_RE: re.Pattern[str] = re.compile(+ r"(?:"+ rf"\bwhat\s+(?:is|are)\s+(?:an?\s+)?{_APPSHOT_WORD}"+ rf"|\bhow\s+(?:do|does|can|to)\b[^.?!]{{0,48}}{_APPSHOT_WORD}"+ # i18n-allow: German speech-input matching data+ rf"|\bwas\s+(?:ist|sind)\s+(?:ein(?:e)?\s+)?{_APPSHOT_WORD}"+ rf"|\bwie\s+(?:funktioniert|funktionieren|geht|macht\s+man)\b[^.?!]{{0,48}}{_APPSHOT_WORD}"+ # i18n-allow: Spanish speech-input matching data+ rf"|\bque\s+(?:es|son)\s+(?:un\s+)?{_APPSHOT_WORD}"+ rf"|\bcomo\s+(?:funciona|funcionan|se\s+hace)\b[^.?!]{{0,48}}{_APPSHOT_WORD}"+ # Opening the Appshots settings page is navigation, not a capture.+ r"|\b(?:open|go\s+to|navigate\s+to|oeffne|geh\s+(?:zu|in|auf)" # i18n-allow: DE input+ r"|wechsel\s+(?:zu|in)|abre|ve\s+a)\b[^.?!]{0,24}?"+ rf"(?:the\s+|die\s+|den\s+|los\s+)?{_APPSHOT_WORD}"+ r"(?:\s+(?:settings|page|section|einstellungen|seite|ajustes|pagina))?\s*(?:$|[.?!;:])"+ rf"|{_APPSHOT_WORD}[\s-]?(?:settings|page|section|einstellungen|seite|ajustes)\b"+ r")",+ re.IGNORECASE,+)+++def mentions_appshot(text: str) -> bool:+ """Does ``text`` ask for an appshot (not merely ask about the feature)?"""+ normalized = _normalize(text or "")+ return bool(_APPSHOT_RE.search(normalized)) and not _APPSHOT_META_RE.search(normalized)++# --------------------------------------------------------------------------# Ownership boundary — observing is not operating# --------------------------------------------------------------------------@@ -588,6 +626,14 @@ def classify(text: str, *, locale: str = "") -> IntentVerdict:if requests_screen_operation(normalized):return IntentVerdict(intent=VisualIntent.NONE, locale=locale)+ # An appshot is the front window by definition.+ if _APPSHOT_RE.search(normalized) and not _APPSHOT_META_RE.search(normalized):+ return IntentVerdict(+ intent=VisualIntent.WINDOW,+ evidence=_evidence(_APPSHOT_RE, normalized),+ locale=locale,+ )+window_hits = _evidence(_WINDOW_SCOPE_RE, normalized)screen_hits = _evidence(_SCREEN_INTENT_RE, normalized)diff --git a/jarvis/screen_context/service.py b/jarvis/screen_context/service.pyindex c8c7c2e9fd..4fa7c8338b 100644--- a/jarvis/screen_context/service.py+++ b/jarvis/screen_context/service.py@@ -203,6 +203,7 @@ class ScreenContextService:self._expiry_timers: dict[str, asyncio.TimerHandle | threading.Timer] = {}self._handle_lock = threading.RLock()self._closed = False+ self._shutter_hook: Any | None = Noneself._patterns = redaction.build_patterns(self._settings.extra_patterns,include_defaults=self._settings.include_default_patterns,@@ -263,6 +264,16 @@ class ScreenContextService:attach_audio_effects(bus)+ def set_shutter_hook(self, hook: Any | None) -> None:+ """Run ``hook(target, size, rgb, monitors)`` at every shutter.++ Called synchronously right after the pixels are grabbed, on the event+ loop, before redaction. It exists for the appshot effect, which must+ start at the true shutter moment; the hook stays in-process and must+ schedule anything slow itself. Its failures never affect the capture.+ """+ self._shutter_hook = hook+# ---- intent ----------------------------------------------------------def classify(self, text: str, *, locale: str = "") -> IntentVerdict:@@ -424,10 +435,11 @@ class ScreenContextService:if callable(getter):window_handle = await asyncio.to_thread(getter)+ monitors = await asyncio.to_thread(self.displays.monitors)try:target, target_degradations = resolve_target(verdict.intent,- monitors=await asyncio.to_thread(self.displays.monitors),+ monitors=monitors,cursor_point=cursor_point,bar_point=bar_point,window=window_facts,@@ -536,6 +548,7 @@ class ScreenContextService:# This is the shutter boundary: pixels exist now. Publish only# metadata so the shared audio layer can play its cue at the# truthful moment without receiving or retaining screen content.+ self._run_shutter_hook(target, size, rgb, monitors)await self._publish_grabbed(size, trace_id=event_trace_id)# Treat a post-shutter identity change as untrusted: discard the@@ -614,6 +627,21 @@ class ScreenContextService:finally:await self._dismiss_indicator(trace_id=event_trace_id)+ def _run_shutter_hook(+ self,+ target: CaptureTarget,+ size: tuple[int, int],+ rgb: bytes,+ monitors: list[dict],+ ) -> None:+ hook = self._shutter_hook+ if hook is None:+ return+ try:+ hook(target, size, rgb, monitors)+ except Exception: # noqa: BLE001 - an effect must never cost the capture+ log.warning("screen_context: shutter hook failed", exc_info=True)+# ---- assembly --------------------------------------------------------def _monitor_privacy_error(self, target: CaptureTarget) -> str | None:diff --git a/jarvis/screen_context/turn.py b/jarvis/screen_context/turn.pyindex 50306a07bd..eab4f1e239 100644--- a/jarvis/screen_context/turn.py+++ b/jarvis/screen_context/turn.py@@ -71,6 +71,11 @@ def get_service(*, bus: Any | None = None) -> ScreenContextService:settings=settings_from_config(load_config()),bus=bus,)+ # Every look through the shared service is an appshot to the user:+ # the shutter effect starts at the true capture moment.+ from jarvis.appshot.effect import on_shutter # noqa: PLC0415++ _service.set_shutter_hook(on_shutter)elif bus is not None:# REST may create the service before the conversational brain is ready.# Attach the one application bus later so receipts are not lost.@@ -154,7 +159,7 @@ class TurnScreenContext:return self.status in ("clarify", "refused", "unavailable", "cancelled")-def _model_note(context: ScreenContext) -> str:+def model_note(context: ScreenContext) -> str:"""The English preamble that rides with the image.Three jobs, all of them about not letting the model invent things:@@ -223,6 +228,7 @@ async def screen_context_for_turn(service: ScreenContextService | None = None,force: bool = False,trace_id: UUID | None = None,+ allow_pending_appshot: bool = True,) -> TurnScreenContext:"""Resolve the screen question for one conversation turn.@@ -231,11 +237,21 @@ async def screen_context_for_turn(derives a language, so a clarifying question cannot flip the conversation'slanguage mid-session (CLAUDE.md §1.3).+ An appshot waiting for the next message (``jarvis.appshot``) is this+ turn's picture, unless the utterance asks for a fresh appshot. Automated+ callers pass ``allow_pending_appshot=False`` so a scheduled task cannot+ swallow the picture the user took for their own next question.+Never raises. Ordinary non-visual turns return before capture infrastructureis initialized. Once a visual request is established, an unexpected defectfails closed with an honest reply so no older screen path can bypass theprivacy policy."""+ if not force and allow_pending_appshot:+ pending = _pending_appshot(utterance)+ if pending is not None:+ return pending+if not force:try:verdict = intent_module.classify(utterance, locale=locale)@@ -304,11 +320,15 @@ async def screen_context_for_turn(if outcome.handle_id:svc.consume(outcome.handle_id)+ from jarvis.appshot.service import record_turn_capture # noqa: PLC0415++ await record_turn_capture(context, bus=bus, trigger="voice")+return TurnScreenContext(status="captured",image=context.image,mime=context.mime,- note=_model_note(context),+ note=model_note(context),text=context.ui_text,receipt=context.describe(),source_hash=f"{context.captured_at_ns:x}",@@ -326,10 +346,41 @@ async def screen_context_for_turn()+def _pending_appshot(utterance: str) -> TurnScreenContext | None:+ """The appshot parked for this message, as a captured turn context."""+ try:+ from jarvis.appshot.store import get_store # noqa: PLC0415++ store = get_store()+ if store.peek_pending() is None:+ return None+ if intent_module.mentions_appshot(utterance):+ # "Take an appshot" asks for a NEW picture; the parked one is stale.+ store.take_pending()+ return None+ shot = store.take_pending()+ except Exception: # noqa: BLE001 - a store fault means "no parked picture"+ log.warning("screen_context: pending appshot lookup failed", exc_info=True)+ return None+ if shot is None:+ return None+ log.info("screen_context: attaching the pending appshot %s", shot.id)+ return TurnScreenContext(+ status="captured",+ image=shot.image,+ mime=shot.mime,+ note=shot.note,+ text=shot.ui_text,+ receipt=f"appshot {shot.width}x{shot.height}",+ source_hash=shot.id,+ )++__all__ = ["TurnScreenContext","TurnStatus","get_service",+ "model_note","reset_service","screen_context_for_turn",]diff --git a/jarvis/ui/web/appshot_routes.py b/jarvis/ui/web/appshot_routes.pynew file mode 100644index 0000000000..e55a7c6af1--- /dev/null+++ b/jarvis/ui/web/appshot_routes.py@@ -0,0 +1,207 @@+"""REST API for appshots — the front window as conversation context.++Endpoints (mounted by the WebServer in ``_build_app()``):++ GET /api/appshot/settings → switches, shortcut state, readiness.+ PUT /api/appshot/settings → change one or more switches.+ POST /api/appshot/take → take one appshot now (optional delay).+ GET /api/appshot/latest → metadata of the last appshot.+ GET /api/appshot/latest/image → its picture (never cached).+ GET /api/appshot/pending → the appshot waiting for the next message.+ POST /api/appshot/pending/claim → hand that one to the chat composer.+ DELETE /api/appshot → forget every held appshot now.++Under the CLI-first contract every action here is also a+``jarvis api appshot <op>`` command. Pixels stay in memory+(``jarvis.appshot.store``); nothing here writes an image to disk.+"""++from __future__ import annotations++import asyncio+import logging+from typing import Any, Literal++from fastapi import APIRouter, HTTPException, Request, Response+from pydantic import BaseModel, Field++log = logging.getLogger(__name__)++router = APIRouter(prefix="/api/appshot", tags=["appshot"])++_NO_STORE = {"Cache-Control": "no-store"}+++class SettingsPatch(BaseModel):+ """Every field optional; only the ones sent are written."""++ enabled: bool | None = None+ hotkey: str | None = Field(default=None, max_length=64)+ target: Literal["auto", "message", "voice"] | None = None+ sound: bool | None = None+ effect: bool | None = None+++class TakeRequest(BaseModel):+ #: Seconds to wait first, so the user can bring the window they mean to+ #: the front after pressing the button inside this app.+ delay_s: float = Field(default=0.0, ge=0.0, le=10.0)+++class ClaimRequest(BaseModel):+ id: str = Field(min_length=1, max_length=64)+++def _bus(request: Request) -> Any | None:+ return getattr(request.app.state, "bus", None)+++def _capability() -> dict[str, Any]:+ import importlib.util # noqa: PLC0415++ from jarvis.cu.indicator.controller import screen_indicator_capability # noqa: PLC0415++ capture_ok = importlib.util.find_spec("mss") is not None+ effect_ok, effect_reason = screen_indicator_capability()+ return {+ "capture": capture_ok,+ "capture_detail": "" if capture_ok else "The screen-capture package is not installed.",+ "effect": effect_ok,+ "effect_detail": effect_reason,+ }+++def _settings_payload() -> dict[str, Any]:+ from jarvis.appshot.hotkey import get_shortcut, normalize_hotkey # noqa: PLC0415+ from jarvis.core.config import load_config # noqa: PLC0415++ config = load_config()+ block = config.appshot+ shortcut = get_shortcut()+ status = (+ shortcut.status.to_json()+ if shortcut is not None+ else {"hotkey": normalize_hotkey(block.hotkey), "armed": False, "detail": "Starting."}+ )+ return {+ "enabled": bool(config.screen_context.enabled),+ "hotkey": normalize_hotkey(block.hotkey),+ "target": block.target,+ "sound": bool(block.sound),+ "effect": bool(block.effect),+ "sound_effects_master": bool(getattr(config.ui, "sound_effects", True)),+ "shortcut": status,+ "readiness": _capability(),+ }++[email protected]("/settings")+async def get_settings() -> dict[str, Any]:+ return await asyncio.to_thread(_settings_payload)++[email protected]("/settings")+async def put_settings(request: Request, patch: SettingsPatch) -> dict[str, Any]:+ """Write the changed switches, then re-arm the shortcut in place."""+ from jarvis.appshot.hotkey import get_shortcut, normalize_hotkey # noqa: PLC0415+ from jarvis.core.config_writer import ( # noqa: PLC0415+ set_appshot_settings,+ set_screen_context_settings,+ )++ changes = patch.model_dump(exclude_none=True)+ if not changes:+ raise HTTPException(status_code=400, detail="No settings were provided.")+ enabled = changes.pop("enabled", None)+ if "hotkey" in changes:+ changes["hotkey"] = normalize_hotkey(changes["hotkey"])+ if changes["hotkey"] and changes["hotkey"] != "alt+alt":+ from jarvis.trigger.hotkey import validate_hotkey # noqa: PLC0415++ verdict = validate_hotkey(changes["hotkey"])+ if not verdict.ok:+ raise HTTPException(status_code=400, detail=verdict.reason or "Invalid shortcut.")++ def _write() -> None:+ if changes:+ set_appshot_settings(changes)+ if enabled is not None:+ set_screen_context_settings({"enabled": bool(enabled)})++ try:+ await asyncio.to_thread(_write)+ except Exception as exc: # noqa: BLE001 - surface the write failure honestly+ log.error("appshot: settings write failed", exc_info=True)+ raise HTTPException(status_code=500, detail=f"Could not save the settings: {exc}") from exc++ if enabled is not None:+ from jarvis.screen_context.turn import reset_service # noqa: PLC0415++ reset_service()+ shortcut = get_shortcut()+ if shortcut is not None and "hotkey" in changes:+ await shortcut.reload()+ return await asyncio.to_thread(_settings_payload)++[email protected]("/take")+async def take(request: Request, body: TakeRequest | None = None) -> dict[str, Any]:+ """Take one appshot of the front window and deliver it like the shortcut."""+ from jarvis.appshot.service import take_appshot # noqa: PLC0415++ delay = (body or TakeRequest()).delay_s+ if delay:+ await asyncio.sleep(delay)+ result = await take_appshot(trigger="button", bus=_bus(request))+ if not result.ok or result.shot is None:+ return {"ok": False, "reason": result.reason_code, "message": result.message}+ return {"ok": True, "appshot": result.shot.meta()}++[email protected]("/latest")+async def latest() -> dict[str, Any]:+ from jarvis.appshot.store import get_store # noqa: PLC0415++ shot = await asyncio.to_thread(get_store().latest)+ return {"appshot": shot.meta() if shot is not None else None}++[email protected]("/latest/image")+async def latest_image() -> Response:+ from jarvis.appshot.store import get_store # noqa: PLC0415++ shot = await asyncio.to_thread(get_store().latest)+ if shot is None:+ raise HTTPException(status_code=404, detail="No appshot is being kept right now.")+ return Response(content=shot.image, media_type=shot.mime, headers=_NO_STORE)++[email protected]("/pending")+async def pending() -> dict[str, Any]:+ from jarvis.appshot.store import get_store # noqa: PLC0415++ shot = await asyncio.to_thread(get_store().peek_pending)+ return {"appshot": shot.meta() if shot is not None else None}++[email protected]("/pending/claim")+async def claim_pending(body: ClaimRequest) -> Response:+ """Move the waiting appshot into the chat composer. Single use."""+ from jarvis.appshot.store import get_store # noqa: PLC0415++ shot = await asyncio.to_thread(get_store().take_pending, body.id)+ if shot is None:+ raise HTTPException(+ status_code=404,+ detail="That appshot was already used or has expired.",+ )+ return Response(content=shot.image, media_type=shot.mime, headers=_NO_STORE)++[email protected]("", openapi_extra={"x-jarvis-dangerous": True})+async def forget_all() -> dict[str, Any]:+ """Drop the waiting and the last appshot right now."""+ from jarvis.appshot.store import get_store # noqa: PLC0415++ await asyncio.to_thread(get_store().clear)+ return {"ok": True}diff --git a/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx b/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsxindex d74c52d8c4..ddaf7005a4 100644--- a/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx+++ b/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx@@ -34,6 +34,7 @@ import { DictationButton } from "@/components/agentchat/DictationButton";import { ComposerAddMenu } from "@/components/agentchat/ComposerAddMenu";import { ComposerChipField, type ComposerChipFieldHandle } from "@/components/agentchat/ComposerChipField";import type { ToolChoice } from "@/components/agentchat/toolChoices";+import { useAppshotClaim } from "@/components/agentchat/useAppshotClaim";import { GigiMark } from "@/components/GigiMark";import { fill, useT } from "@/i18n";import { cn } from "@/lib/utils";@@ -166,6 +167,8 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {},onAttachProblem,);+ // A shortcut appshot parked for "the next message" joins this one.+ useAppshotClaim(files.attachFiles, surface === "jarvis");const fileInputRef = useRef<HTMLInputElement | null>(null);const pasteRescue = usePasteRescue();diff --git a/jarvis/ui/web/frontend/src/components/agentchat/useAppshotClaim.ts b/jarvis/ui/web/frontend/src/components/agentchat/useAppshotClaim.tsnew file mode 100644index 0000000000..60dc89bfdc--- /dev/null+++ b/jarvis/ui/web/frontend/src/components/agentchat/useAppshotClaim.ts@@ -0,0 +1,47 @@+/**+ * An appshot taken for "the next message" lands in the open chat composer.+ *+ * The backend parks a shortcut appshot for the next turn. While the front-page+ * chat is on screen, that next turn is the one being typed here, so the+ * composer claims the picture (single use — the backend drops its copy) and+ * holds it like a pasted screenshot: visible, removable, sent with the+ * sentence through the same attachment path every seat already understands.+ */+import { useEffect, useRef } from "react";++import { useT } from "@/i18n";+import { claimPendingAppshot, fetchPendingAppshot } from "@/lib/appshotApi";+import { useEventStore } from "@/store/events";++export function useAppshotClaim(attachFiles: (files: File[]) => void, enabled: boolean): void {+ const t = useT();+ const pushToast = useEventStore((s) => s.pushToast);+ const eventId = useEventStore((s) => {+ const event = s.events.find((item) => item.name === "AppshotTaken");+ const payload = (event?.payload ?? {}) as { delivered_to?: unknown };+ return event && payload.delivered_to === "message" ? event.id : "";+ });+ const claiming = useRef(false);++ useEffect(() => {+ if (!enabled || claiming.current) return;+ claiming.current = true;+ void (async () => {+ try {+ const { appshot } = await fetchPendingAppshot();+ if (!appshot) return;+ // Once claimed the backend no longer holds it, so it is attached+ // even if this effect was superseded meanwhile — never dropped.+ const file = await claimPendingAppshot(appshot);+ if (!file) return;+ attachFiles([file]);+ pushToast("info", t("appshots.chip_label"));+ } catch {+ // Nothing parked or the backend is restarting: the picture stays+ // where it was, for the next spoken turn to use.+ } finally {+ claiming.current = false;+ }+ })();+ }, [enabled, eventId, attachFiles, pushToast, t]);+}diff --git a/jarvis/ui/web/frontend/src/components/layout/MainView.tsx b/jarvis/ui/web/frontend/src/components/layout/MainView.tsxindex 495f2f0d7c..bbc3f6d5f6 100644--- a/jarvis/ui/web/frontend/src/components/layout/MainView.tsx+++ b/jarvis/ui/web/frontend/src/components/layout/MainView.tsx@@ -512,6 +512,7 @@ function SwitchOnActiveSection({ active }: { active: string }) {case "local-models":case "computers":case "wallpaper":+ case "appshots":case "costs":case "feedback":return <SettingsHubView />;diff --git a/jarvis/ui/web/frontend/src/components/layout/navGroups.ts b/jarvis/ui/web/frontend/src/components/layout/navGroups.tsindex 7d859a7719..bd90915309 100644--- a/jarvis/ui/web/frontend/src/components/layout/navGroups.ts+++ b/jarvis/ui/web/frontend/src/components/layout/navGroups.ts@@ -33,6 +33,7 @@ import {Wallet,Workflow,Image as ImageIcon,+ ScanLine,type LucideIcon,} from "lucide-react";import { OllamaIcon } from "@/components/icons/OllamaIcon";@@ -265,6 +266,15 @@ export const NAV_FOOTER_ITEMS: NavItem[] = [{ id: "feedback", labelKey: "nav.feedback", icon: MessageSquareWarning },];+/**+ * Rows that exist only inside the Settings hub's own navigation, not in the+ * app sidebar: settings for one feature, reached through Settings the way the+ * Appshots page is.+ */+export const SETTINGS_HUB_ONLY_ITEMS: NavItem[] = [+ { id: "appshots", labelKey: "nav.appshots", icon: ScanLine, fallbackLabel: "Appshots" },+];+/*** Every section id rendered inside the Settings hub (`SettingsHubView`).*@@ -289,6 +299,7 @@ export const SETTINGS_HUB_IDS: readonly SectionId[] = ["local-models","computers","wallpaper",+ "appshots","costs","feedback",];diff --git a/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx b/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsxindex 927ce44ae3..b474f16f9f 100644--- a/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx+++ b/jarvis/ui/web/frontend/src/hooks/useWebSocket.test.tsx@@ -257,13 +257,15 @@ describe("useWebSocket VoiceBootStatus handling", () => {expect(voiceInputLevelRef.current).toBe(0);});- it("shows a metadata-only receipt for one-shot screen capture", async () => {+ it("shows one appshot receipt per capture and no window title", async () => {render(<Harness />);await Promise.resolve();+ // The capture events themselves stay silent: the desktop overlay flashes+ // at the shutter, and AppshotTaken is the single in-app receipt.MockWebSocket.last!.deliver(envelope("ScreenCaptureAnnounced", {- target_kind: "monitor",+ target_kind: "window",target_label: "private title must not enter the toast",}),);@@ -275,15 +277,38 @@ describe("useWebSocket VoiceBootStatus handling", () => {target_label: "private title must not enter the toast",}),);+ MockWebSocket.last!.deliver(+ envelope("AppshotTaken", {+ appshot_id: "a1",+ trigger: "hotkey",+ delivered_to: "message",+ target_label: "active window",+ }),+ );const toasts = useEventStore.getState().toasts;- expect(toasts).toHaveLength(2);- expect(toasts[0].message).toContain("requested surface");- expect(toasts[1].message).toContain("1600×900");- expect(toasts[1].message).toContain("3 redaction");- expect(toasts.map((toast) => toast.message).join(" ")).not.toContain(- "private title",+ expect(toasts).toHaveLength(1);+ expect(toasts[0].kind).toBe("success");+ expect(toasts[0].message).toContain("Appshot");+ expect(toasts[0].message).not.toContain("private title");+ });++ it("says why a shortcut appshot was refused", async () => {+ render(<Harness />);+ await Promise.resolve();++ MockWebSocket.last!.deliver(+ envelope("AppshotTaken", {+ trigger: "hotkey",+ delivered_to: "refused",+ target_label: "Appshots are switched off.",+ }),);++ const toasts = useEventStore.getState().toasts;+ expect(toasts).toHaveLength(1);+ expect(toasts[0].kind).toBe("warning");+ expect(toasts[0].message).toContain("Appshots are switched off.");});it("stages the newest picture when the brain opens the Visualization section", async () => {diff --git a/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts b/jarvis/ui/web/frontend/src/hooks/useWebSocket.tsindex 0f969e1f04..e40ad47e8a 100644--- a/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts+++ b/jarvis/ui/web/frontend/src/hooks/useWebSocket.ts@@ -140,29 +140,16 @@ export function useWebSocket(): void {payload: env.payload,});- if (env.event_name === "ScreenCaptureAnnounced") {- pushToast("info", translate("settings_view.screen_context.capturing"));- }-- if (env.event_name === "ScreenCaptureCompleted") {- const payload = env.payload as {- width?: unknown;- height?: unknown;- redaction_count?: unknown;- };- const width = typeof payload.width === "number" ? payload.width : 0;- const height = typeof payload.height === "number" ? payload.height : 0;- const redactions =- typeof payload.redaction_count === "number"- ? payload.redaction_count- : 0;- pushToast(- "success",- translate("settings_view.screen_context.completed")- .replace("{0}", String(width))- .replace("{1}", String(height))- .replace("{2}", String(redactions)),- );+ // Every screen look is an appshot to the user: the desktop overlay+ // flashes at the shutter, and this one line is the in-app receipt.+ if (env.event_name === "AppshotTaken") {+ const payload = env.payload as { delivered_to?: unknown; target_label?: unknown };+ const label = typeof payload.target_label === "string" ? payload.target_label : "";+ if (payload.delivered_to === "refused") {+ pushToast("warning", translate("appshots.toast_refused").replace("{0}", label));+ } else {+ pushToast("success", translate("appshots.try_done"));+ }}// Live reasoning trace: while the text chat is waiting on a reply,diff --git a/jarvis/ui/web/frontend/src/i18n/locales/de.json b/jarvis/ui/web/frontend/src/i18n/locales/de.jsonindex 105db74462..d55aef8298 100644--- a/jarvis/ui/web/frontend/src/i18n/locales/de.json+++ b/jarvis/ui/web/frontend/src/i18n/locales/de.json@@ -161,6 +161,7 @@"group_system": "System","agent_instructions": "{name}.md","wallpaper": "Hintergrund",+ "appshots": "Appshots","marketplace": "Marketplace","agentic_ide": "Agentic IDE","agentic_ide_beta": "Beta",@@ -5921,5 +5922,50 @@"idle": "Im Leerlauf"}}+ },+ "appshots": {+ "title": "Appshots",+ "subtitle": "Zeig {name} das Fenster, in dem du gerade arbeitest.",+ "hero_title": "Mach einen Appshot, um {name} dein vorderstes Fenster zu zeigen",+ "hero_body": "Ein Appshot nimmt das Fenster im Vordergrund auf, Bild und sichtbaren Text, und gibt es ins Gespräch. Deine Datenschutz-Regeln gelten weiter, und nichts wird auf der Festplatte gespeichert.",+ "enabled_label": "Appshots erlauben",+ "enabled_hint": "Gilt auch für Fragen wie \"Was siehst du?\". Aus heißt: {name} schaut nie auf deinen Bildschirm.",+ "shortcut_label": "Tastenkürzel",+ "shortcut_both_alt_hint": "Beide Alt-Tasten gleichzeitig drücken",+ "shortcut_both_option_hint": "Beide Wahltasten gleichzeitig drücken",+ "shortcut_combo_hint": "{0} drücken",+ "shortcut_hint_off": "Kein Tastenkürzel. Frag per Stimme oder nutze den Knopf unten.",+ "shortcut_off": "Aus",+ "shortcut_unavailable": "Nicht aktiv: {0}",+ "target_label": "Appshot-Ziel",+ "target_auto": "Automatisch",+ "target_message": "Nächste Nachricht",+ "target_voice": "Nur Sprachgespräch",+ "target_auto_hint": "In ein laufendes Sprachgespräch, sonst an deine nächste Nachricht.",+ "target_message_hint": "Immer an deine nächste gesprochene oder getippte Nachricht.",+ "target_voice_hint": "Nur in ein laufendes Sprachgespräch. Ohne Gespräch wird er nicht gesendet.",+ "sound_label": "Soundeffekt abspielen",+ "sound_master_off": "Soundeffekte sind für die ganze App unter Einstellungen ausgeschaltet.",+ "effect_label": "Blitz zeigen",+ "effect_hint": "Ein kurzer Blitz und eine Mini-Vorschau in der Bildschirmecke.",+ "effect_unavailable": "Hier nicht verfügbar: {0}",+ "try_label": "Ausprobieren",+ "try_hint": "Macht nach 3 Sekunden einen Appshot vom Fenster im Vordergrund. Wechsle jetzt dorthin.",+ "try_counting": "Wechsle zu deinem Fenster: {0}",+ "try_button": "Appshot in 3 s",+ "try_done": "Appshot aufgenommen",+ "preview_title": "Letzter Appshot",+ "preview_empty": "Dein letzter Appshot erscheint hier für zwei Minuten, danach ist er weg.",+ "preview_front_window": "Vorderstes Fenster",+ "preview_alt": "Appshot von {0}",+ "preview_sent_voice": "Ins Sprachgespräch gegeben",+ "preview_sent_message": "Wartet auf deine nächste Nachricht",+ "preview_sent_turn": "Im Gespräch verwendet",+ "preview_sent_none": "Nicht gesendet, es lief kein Sprachgespräch",+ "forget": "Vergessen",+ "voice_hint": "Du kannst auch einfach sagen: \"Mach einen Appshot\".",+ "save_error": "Speichern fehlgeschlagen: {0}",+ "toast_refused": "Kein Appshot: {0}",+ "chip_label": "Appshot an deine Nachricht gehängt"}}diff --git a/jarvis/ui/web/frontend/src/i18n/locales/en.json b/jarvis/ui/web/frontend/src/i18n/locales/en.jsonindex da2a97655f..5253983ce4 100644--- a/jarvis/ui/web/frontend/src/i18n/locales/en.json+++ b/jarvis/ui/web/frontend/src/i18n/locales/en.json@@ -161,6 +161,7 @@"group_system": "System","agent_instructions": "{name}.md","wallpaper": "Wallpaper",+ "appshots": "Appshots","marketplace": "Marketplace","agentic_ide": "Agentic IDE","agentic_ide_beta": "Beta",@@ -5954,5 +5955,50 @@"idle": "Idle"}}+ },+ "appshots": {+ "title": "Appshots",+ "subtitle": "Show {name} the window you are working in.",+ "hero_title": "Take an appshot to show {name} your front window",+ "hero_body": "An appshot captures the window in front, picture and on-screen text, and hands it to the conversation. Your privacy rules still apply, and nothing is saved to disk.",+ "enabled_label": "Allow appshots",+ "enabled_hint": "Also covers questions like \"what do you see?\". Off means {name} never looks at your screen.",+ "shortcut_label": "Keyboard shortcut",+ "shortcut_both_alt_hint": "Press both Alt keys at the same time",+ "shortcut_both_option_hint": "Press both Option keys at the same time",+ "shortcut_combo_hint": "Press {0}",+ "shortcut_hint_off": "No shortcut. Ask by voice or use the button below.",+ "shortcut_off": "Off",+ "shortcut_unavailable": "Not active: {0}",+ "target_label": "Appshot destination",+ "target_auto": "Automatic",+ "target_message": "Next message",+ "target_voice": "Voice call only",+ "target_auto_hint": "Into a running voice call, otherwise onto your next message.",+ "target_message_hint": "Always onto your next spoken or typed message.",+ "target_voice_hint": "Only into a running voice call. Without one, it is not sent.",+ "sound_label": "Play sound effect",+ "sound_master_off": "Sound effects are off for the whole app under Settings.",+ "effect_label": "Show the flash",+ "effect_hint": "A short flash and a thumbnail in the corner of your screen.",+ "effect_unavailable": "Not available here: {0}",+ "try_label": "Try it",+ "try_hint": "Takes an appshot of the window in front after 3 seconds. Switch to it now.",+ "try_counting": "Switch to your window: {0}",+ "try_button": "Take appshot in 3 s",+ "try_done": "Appshot taken",+ "preview_title": "Last appshot",+ "preview_empty": "Your last appshot shows up here for two minutes, then it is gone.",+ "preview_front_window": "Front window",+ "preview_alt": "Appshot of {0}",+ "preview_sent_voice": "Sent into the voice call",+ "preview_sent_message": "Waiting for your next message",+ "preview_sent_turn": "Used in the conversation",+ "preview_sent_none": "Not sent, no voice call was running",+ "forget": "Forget",+ "voice_hint": "You can also just say \"take an appshot\".",+ "save_error": "Could not save: {0}",+ "toast_refused": "No appshot: {0}",+ "chip_label": "Appshot added to your message"}}diff --git a/jarvis/ui/web/frontend/src/i18n/locales/es.json b/jarvis/ui/web/frontend/src/i18n/locales/es.jsonindex 0b7b0c93cb..f250fb16a8 100644--- a/jarvis/ui/web/frontend/src/i18n/locales/es.json+++ b/jarvis/ui/web/frontend/src/i18n/locales/es.json@@ -161,6 +161,7 @@"group_system": "Sistema","agent_instructions": "{name}.md","wallpaper": "Fondo de pantalla",+ "appshots": "Appshots","marketplace": "Marketplace","agentic_ide": "IDE agéntico","agentic_ide_beta": "Beta",@@ -5921,5 +5922,50 @@"idle": "Inactivo"}}+ },+ "appshots": {+ "title": "Appshots",+ "subtitle": "Muestra a {name} la ventana en la que trabajas.",+ "hero_title": "Haz un appshot para mostrar a {name} tu ventana principal",+ "hero_body": "Un appshot captura la ventana en primer plano, imagen y texto visible, y la pasa a la conversación. Tus reglas de privacidad siguen aplicándose y no se guarda nada en el disco.",+ "enabled_label": "Permitir appshots",+ "enabled_hint": "También cubre preguntas como \"¿qué ves?\". Desactivado significa que {name} nunca mira tu pantalla.",+ "shortcut_label": "Atajo de teclado",+ "shortcut_both_alt_hint": "Pulsa las dos teclas Alt a la vez",+ "shortcut_both_option_hint": "Pulsa las dos teclas Opción a la vez",+ "shortcut_combo_hint": "Pulsa {0}",+ "shortcut_hint_off": "Sin atajo. Pídelo por voz o usa el botón de abajo.",+ "shortcut_off": "Desactivado",+ "shortcut_unavailable": "No activo: {0}",+ "target_label": "Destino del appshot",+ "target_auto": "Automático",+ "target_message": "Siguiente mensaje",+ "target_voice": "Solo llamada de voz",+ "target_auto_hint": "A una llamada de voz en curso; si no hay, a tu siguiente mensaje.",+ "target_message_hint": "Siempre a tu siguiente mensaje, hablado o escrito.",+ "target_voice_hint": "Solo a una llamada de voz en curso. Sin llamada, no se envía.",+ "sound_label": "Reproducir efecto de sonido",+ "sound_master_off": "Los efectos de sonido están desactivados para toda la app en Ajustes.",+ "effect_label": "Mostrar el destello",+ "effect_hint": "Un destello breve y una miniatura en la esquina de la pantalla.",+ "effect_unavailable": "No disponible aquí: {0}",+ "try_label": "Probar",+ "try_hint": "Hace un appshot de la ventana en primer plano tras 3 segundos. Cambia a ella ahora.",+ "try_counting": "Cambia a tu ventana: {0}",+ "try_button": "Appshot en 3 s",+ "try_done": "Appshot hecho",+ "preview_title": "Último appshot",+ "preview_empty": "Tu último appshot aparece aquí durante dos minutos y luego desaparece.",+ "preview_front_window": "Ventana principal",+ "preview_alt": "Appshot de {0}",+ "preview_sent_voice": "Enviado a la llamada de voz",+ "preview_sent_message": "Esperando tu siguiente mensaje",+ "preview_sent_turn": "Usado en la conversación",+ "preview_sent_none": "No enviado, no había llamada de voz",+ "forget": "Olvidar",+ "voice_hint": "También puedes decir \"haz un appshot\".",+ "save_error": "No se pudo guardar: {0}",+ "toast_refused": "Sin appshot: {0}",+ "chip_label": "Appshot añadido a tu mensaje"}}diff --git a/jarvis/ui/web/frontend/src/index.css b/jarvis/ui/web/frontend/src/index.cssindex 374624a70d..91d37853dd 100644--- a/jarvis/ui/web/frontend/src/index.css+++ b/jarvis/ui/web/frontend/src/index.css@@ -237,6 +237,61 @@ label,}}+ /* Appshots page demo: the same shutter the desktop overlay plays — a white+ flash, then the picture shrinks into the corner, rests, and slides out. */+ .appshot-demo-flash {+ animation: appshot-demo-flash 4.8s ease-out infinite;+ }++ .appshot-demo-card {+ transform-origin: 100% 100%;+ animation: appshot-demo-card 4.8s cubic-bezier(0.22, 1, 0.36, 1) infinite;+ }++ @media (prefers-reduced-motion: reduce) {+ .appshot-demo-flash,+ .appshot-demo-card {+ animation: none;+ opacity: 0;+ }+ }++ @keyframes appshot-demo-flash {+ 0%,+ 6% {+ opacity: 0;+ }+ 9% {+ opacity: 0.85;+ }+ 22%,+ 100% {+ opacity: 0;+ }+ }++ @keyframes appshot-demo-card {+ 0%,+ 8% {+ opacity: 0;+ transform: translate(0, 0) scale(1);+ }+ 10% {+ opacity: 1;+ transform: translate(0, 0) scale(1);+ }+ 28%,+ 78% {+ opacity: 1;+ transform: translate(-6%, -7%) scale(0.34);+ }+ 90%,+ 100% {+ opacity: 0;+ transform: translate(4%, -7%) scale(0.34);+ }+ }+@keyframes agentic-transcript-in {from {opacity: 0;diff --git a/jarvis/ui/web/frontend/src/lib/appshotApi.ts b/jarvis/ui/web/frontend/src/lib/appshotApi.tsnew file mode 100644index 0000000000..3233d009a7--- /dev/null+++ b/jarvis/ui/web/frontend/src/lib/appshotApi.ts@@ -0,0 +1,132 @@+/**+ * Client for `/api/appshot` — the front window as conversation context.+ *+ * Pixels only ever come from the two image endpoints, which the backend serves+ * with `Cache-Control: no-store`; nothing here keeps a copy.+ */++export type AppshotTarget = "auto" | "message" | "voice";++export interface AppshotShortcutStatus {+ hotkey: string;+ armed: boolean;+ detail: string;+}++export interface AppshotSettings {+ enabled: boolean;+ hotkey: string;+ target: AppshotTarget;+ sound: boolean;+ effect: boolean;+ sound_effects_master: boolean;+ shortcut: AppshotShortcutStatus;+ readiness: {+ capture: boolean;+ capture_detail: string;+ effect: boolean;+ effect_detail: string;+ };+}++export interface AppshotMeta {+ id: string;+ width: number;+ height: number;+ label: string;+ app_name: string;+ trigger: string;+ taken_at: number;+ delivered_to: string;+}++export type AppshotSettingsPatch = Partial<+ Pick<AppshotSettings, "enabled" | "hotkey" | "target" | "sound" | "effect">+>;++async function request<T>(url: string, init?: RequestInit): Promise<T> {+ const response = await fetch(url, init);+ const body = (await response.json().catch(() => null)) as T | { detail?: unknown } | null;+ if (!response.ok) {+ const detail =+ body && typeof body === "object" && "detail" in body && typeof body.detail === "string"+ ? body.detail+ : "";+ throw new Error(detail || `Appshot request failed (${response.status}).`);+ }+ return body as T;+}++export function fetchAppshotSettings(): Promise<AppshotSettings> {+ return request<AppshotSettings>("/api/appshot/settings");+}++export function saveAppshotSettings(patch: AppshotSettingsPatch): Promise<AppshotSettings> {+ return request<AppshotSettings>("/api/appshot/settings", {+ method: "PUT",+ headers: { "content-type": "application/json" },+ body: JSON.stringify(patch),+ });+}++export function fetchLatestAppshot(): Promise<{ appshot: AppshotMeta | null }> {+ return request<{ appshot: AppshotMeta | null }>("/api/appshot/latest");+}++export function latestAppshotImageUrl(id: string): string {+ // The id only busts the <img> cache between appshots; the server keeps one.+ return `/api/appshot/latest/image?v=${encodeURIComponent(id)}`;+}++export function takeAppshot(delaySeconds = 0): Promise<+ { ok: true; appshot: AppshotMeta } | { ok: false; reason: string; message: string }+> {+ return request("/api/appshot/take", {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({ delay_s: delaySeconds }),+ });+}++export function forgetAppshots(): Promise<{ ok: boolean }> {+ return request<{ ok: boolean }>("/api/appshot", { method: "DELETE" });+}++export function fetchPendingAppshot(): Promise<{ appshot: AppshotMeta | null }> {+ return request<{ appshot: AppshotMeta | null }>("/api/appshot/pending");+}++/** Take the waiting appshot out of the backend, as a file for the composer. */+export async function claimPendingAppshot(meta: AppshotMeta): Promise<File | null> {+ const response = await fetch("/api/appshot/pending/claim", {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({ id: meta.id }),+ });+ if (!response.ok) return null;+ const blob = await response.blob();+ const stamp = new Date(meta.taken_at * 1000)+ .toISOString()+ .replace(/[-:]/g, "")+ .replace(/\..*$/, "")+ .replace("T", "-");+ const extension = blob.type === "image/png" ? "png" : "jpg";+ return new File([blob], `appshot-${stamp}.${extension}`, { type: blob.type || "image/jpeg" });+}++/** "alt+alt" → "Alt + Alt" (⌥ on a Mac); any other combo, title-cased. */+export function formatAppshotHotkey(hotkey: string, isMac: boolean): string {+ if (!hotkey) return "";+ if (hotkey === "alt+alt") return isMac ? "⌥ + ⌥" : "Alt + Alt";+ return hotkey+ .split("+")+ .map((part) => {+ const key = part.trim();+ if (key === "ctrl") return isMac ? "⌃" : "Ctrl";+ if (key === "alt") return isMac ? "⌥" : "Alt";+ if (key === "shift") return isMac ? "⇧" : "Shift";+ if (key === "win" || key === "cmd" || key === "super") return isMac ? "⌘" : "Win";+ return key.length === 1 ? key.toUpperCase() : key.charAt(0).toUpperCase() + key.slice(1);+ })+ .join(" + ");+}diff --git a/jarvis/ui/web/frontend/src/store/events.ts b/jarvis/ui/web/frontend/src/store/events.tsindex e76d98bc06..811a0dabce 100644--- a/jarvis/ui/web/frontend/src/store/events.ts+++ b/jarvis/ui/web/frontend/src/store/events.ts@@ -85,6 +85,9 @@ export type SectionId =| "feedback"| "agent-instructions"| "wallpaper"+ // Appshots: the shortcut, destination, sound and flash for showing the+ // assistant the front window. A Settings-hub page.+ | "appshots"| "dictionary"| "dictation"// The three tabs added by the merged voice section. "dictation" (default@@ -139,6 +142,7 @@ export const SECTION_IDS = ["feedback","agent-instructions","wallpaper",+ "appshots","dictionary","dictation",// `satisfies` only catches array entries that are missing from the union,@@ -229,6 +233,7 @@ export const SECTION_LABELS: Record<SectionId, string> = {feedback: "Feedback","agent-instructions": "Agent Instructions",wallpaper: "Wallpaper",+ appshots: "Appshots",dictionary: "Dictionary",dictation: "Dictation",// Plain English, deliberately NOT the "{name} Voice" brand: these labels arediff --git a/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx b/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsxnew file mode 100644index 0000000000..49c92c4e45--- /dev/null+++ b/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx@@ -0,0 +1,93 @@+import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";++import { formatAppshotHotkey } from "@/lib/appshotApi";+import { useEventStore } from "@/store/events";+import { AppshotsView } from "@/views/AppshotsView";++const SETTINGS = {+ enabled: true,+ hotkey: "alt+alt",+ target: "auto",+ sound: true,+ effect: true,+ sound_effects_master: true,+ shortcut: { hotkey: "alt+alt", armed: true, detail: "" },+ readiness: { capture: true, capture_detail: "", effect: true, effect_detail: "" },+};++function json(body: unknown, status = 200): Response {+ return { ok: status < 400, status, json: async () => body } as Response;+}++describe("AppshotsView", () => {+ let fetchMock: ReturnType<typeof vi.fn>;++ beforeEach(() => {+ useEventStore.setState({ events: [], toasts: [], assistantName: "Jarvis" });+ fetchMock = vi.fn(async (url: string, init?: RequestInit) => {+ if (url === "/api/appshot/settings" && init?.method === "PUT") {+ return json({ ...SETTINGS, ...JSON.parse(String(init.body)) });+ }+ if (url === "/api/appshot/settings") return json(SETTINGS);+ if (url === "/api/appshot/latest") return json({ appshot: null });+ return json({}, 404);+ });+ vi.stubGlobal("fetch", fetchMock);+ });++ afterEach(() => {+ cleanup();+ vi.unstubAllGlobals();+ });++ it("shows the switches once the settings arrive", async () => {+ render(<AppshotsView />);+ await waitFor(() => expect(screen.getByTestId("appshots-enabled")).toBeDefined());+ expect(screen.getByTestId("appshots-sound").getAttribute("data-state")).toBe("checked");+ expect(screen.getByTestId("appshots-effect").getAttribute("data-state")).toBe("checked");+ expect(screen.getByTestId("appshots-try")).toBeDefined();+ });++ it("saves one switch with a PUT of just that key", async () => {+ render(<AppshotsView />);+ const sound = await screen.findByTestId("appshots-sound");+ fireEvent.click(sound);+ await waitFor(() =>+ expect(+ fetchMock.mock.calls.some(+ ([url, init]) =>+ url === "/api/appshot/settings" &&+ init?.method === "PUT" &&+ JSON.parse(String(init.body)).sound === false,+ ),+ ).toBe(true),+ );+ const put = fetchMock.mock.calls.find(([, init]) => init?.method === "PUT")!;+ expect(Object.keys(JSON.parse(String(put[1].body)))).toEqual(["sound"]);+ });++ it("disables every other control while appshots are switched off", async () => {+ fetchMock.mockImplementation(async (url: string) =>+ url === "/api/appshot/settings"+ ? json({ ...SETTINGS, enabled: false })+ : json({ appshot: null }),+ );+ render(<AppshotsView />);+ const tryButton = await screen.findByTestId("appshots-try");+ expect((tryButton as HTMLButtonElement).disabled).toBe(true);+ expect(screen.getByTestId("appshots-sound").hasAttribute("disabled")).toBe(true);+ });+});++describe("formatAppshotHotkey", () => {+ it("names the both-Alt gesture per platform", () => {+ expect(formatAppshotHotkey("alt+alt", false)).toBe("Alt + Alt");+ expect(formatAppshotHotkey("alt+alt", true)).toBe("⌥ + ⌥");+ });++ it("title-cases an ordinary combo", () => {+ expect(formatAppshotHotkey("ctrl+alt+a", false)).toBe("Ctrl + Alt + A");+ expect(formatAppshotHotkey("", false)).toBe("");+ });+});diff --git a/jarvis/ui/web/frontend/src/views/AppshotsView.tsx b/jarvis/ui/web/frontend/src/views/AppshotsView.tsxnew file mode 100644index 0000000000..d5cc2b9ba8--- /dev/null+++ b/jarvis/ui/web/frontend/src/views/AppshotsView.tsx@@ -0,0 +1,437 @@+import { useCallback, useEffect, useMemo, useRef, useState } from "react";+import { Loader2, Trash2 } from "lucide-react";++import { PageHeader } from "@/components/layout/PageHeader";+import { Button } from "@/components/ui/button";+import { BrandedSelect, type BrandedSelectOption } from "@/components/ui/select";+import { Switch } from "@/components/ui/switch";+import { useT } from "@/i18n";+import {+ fetchAppshotSettings,+ fetchLatestAppshot,+ forgetAppshots,+ formatAppshotHotkey,+ latestAppshotImageUrl,+ saveAppshotSettings,+ takeAppshot,+ type AppshotMeta,+ type AppshotSettings,+ type AppshotSettingsPatch,+} from "@/lib/appshotApi";+import { cn } from "@/lib/utils";+import { useEventStore } from "@/store/events";++/**+ * Appshots — show the assistant the window you are working in.+ *+ * One page for the whole feature: the master switch (it is also the switch+ * for every other screen look, `[screen_context].enabled`), the global+ * shortcut, where a shortcut appshot goes, the sound and the flash, a try-it+ * button, and the last appshot so the user sees exactly what was handed over.+ * That picture lives in backend memory for `deck_preview_s` and is fetched+ * with `no-store`; this view keeps no copy.+ */++const IS_MAC = typeof navigator !== "undefined" && /Mac/i.test(navigator.platform || "");+const PRESET_HOTKEYS = ["alt+alt", "ctrl+alt+a", ""] as const;+const TRY_DELAY_S = 3;++export function AppshotGlyph({ className }: { className?: string }) {+ // Four camera-frame corners around a pill: "this window, captured".+ return (+ <svg viewBox="0 0 32 32" fill="none" aria-hidden className={className}>+ <path+ d="M4 11V8a4 4 0 0 1 4-4h3M21 4h3a4 4 0 0 1 4 4v3M28 21v3a4 4 0 0 1-4 4h-3M11 28H8a4 4 0 0 1-4-4v-3"+ stroke="currentColor"+ strokeWidth="2.6"+ strokeLinecap="round"+ />+ <rect x="9.5" y="13" width="13" height="6" rx="3" fill="currentColor" />+ </svg>+ );+}++function Row({+ label,+ hint,+ control,+ children,+}: {+ label: string;+ hint?: string;+ control: React.ReactNode;+ children?: React.ReactNode;+}) {+ return (+ <div className="px-5 py-4">+ <div className="flex items-center justify-between gap-6">+ <div className="min-w-0">+ <p className="text-base font-medium text-foreground">{label}</p>+ {hint && <p className="mt-0.5 text-sm text-muted-foreground">{hint}</p>}+ </div>+ <div className="shrink-0">{control}</div>+ </div>+ {children}+ </div>+ );+}++function DemoWindow({ className }: { className?: string }) {+ // A theme-token mock of an app window; no screenshot of anyone's screen.+ return (+ <div className={cn("flex flex-col overflow-hidden rounded-lg border border-border bg-card shadow-sm", className)}>+ <div className="flex items-center gap-1.5 border-b border-border px-3 py-2">+ <span className="h-2 w-2 rounded-full bg-foreground/20" />+ <span className="h-2 w-2 rounded-full bg-foreground/20" />+ <span className="h-2 w-2 rounded-full bg-foreground/20" />+ <span className="ml-3 h-2 w-1/3 rounded-full bg-foreground/10" />+ </div>+ <div className="flex flex-1 gap-3 p-3">+ <div className="w-1/4 space-y-2">+ {[0, 1, 2, 3].map((i) => (+ <span key={i} className="block h-2 rounded-full bg-foreground/10" />+ ))}+ </div>+ <div className="flex-1 space-y-2">+ <span className="block h-3 w-1/2 rounded-full bg-foreground/15" />+ {[0, 1, 2, 3, 4].map((i) => (+ <span key={i} className={cn("block h-2 rounded-full bg-foreground/10", i % 2 ? "w-4/5" : "w-full")} />+ ))}+ <span className="mt-3 block h-14 rounded-md bg-accent/15" />+ </div>+ </div>+ </div>+ );+}++function PreviewDemo() {+ const t = useT();+ return (+ <div className="flex h-full flex-col">+ <div className="relative aspect-[16/10] w-full overflow-hidden rounded-lg bg-secondary/60 p-5">+ <DemoWindow className="h-full w-full" />+ <div className="appshot-demo-card pointer-events-none absolute inset-5">+ <DemoWindow className="h-full w-full ring-4 ring-white/90" />+ </div>+ <div className="appshot-demo-flash pointer-events-none absolute inset-5 rounded-lg bg-white" />+ </div>+ <p className="mt-3 text-sm text-muted-foreground">{t("appshots.preview_empty")}</p>+ </div>+ );+}++function deliveredLabel(t: (key: string) => string, deliveredTo: string): string {+ switch (deliveredTo) {+ case "voice":+ return t("appshots.preview_sent_voice");+ case "message":+ return t("appshots.preview_sent_message");+ case "turn":+ return t("appshots.preview_sent_turn");+ case "none":+ return t("appshots.preview_sent_none");+ default:+ return "";+ }+}++function LatestPreview({ shot, onForget }: { shot: AppshotMeta; onForget: () => void }) {+ const t = useT();+ const time = new Date(shot.taken_at * 1000).toLocaleTimeString([], {+ hour: "2-digit",+ minute: "2-digit",+ });+ const where = shot.app_name || t("appshots.preview_front_window");+ const delivered = deliveredLabel(t, shot.delivered_to);+ return (+ <div className="flex h-full flex-col">+ <div className="flex aspect-[16/10] w-full items-center justify-center overflow-hidden rounded-lg bg-secondary/60 p-3">+ <img+ src={latestAppshotImageUrl(shot.id)}+ alt={t("appshots.preview_alt").replace("{0}", where)}+ className="max-h-full max-w-full rounded-md object-contain shadow-sm ring-1 ring-border"+ />+ </div>+ <div className="mt-3 flex items-start justify-between gap-3">+ <div className="min-w-0">+ <p className="truncate text-base font-medium text-foreground">{where}</p>+ <p className="text-sm text-muted-foreground">+ {[time, `${shot.width} × ${shot.height}`, delivered].filter(Boolean).join(" · ")}+ </p>+ </div>+ <Button type="button" variant="ghost" size="sm" onClick={onForget}>+ <Trash2 aria-hidden />+ {t("appshots.forget")}+ </Button>+ </div>+ </div>+ );+}++export function AppshotsView() {+ const t = useT();+ const pushToast = useEventStore((s) => s.pushToast);+ const lastAppshotEvent = useEventStore(+ (s) => s.events.find((event) => event.name === "AppshotTaken")?.id ?? "",+ );+ const [settings, setSettings] = useState<AppshotSettings | null>(null);+ const [latest, setLatest] = useState<AppshotMeta | null>(null);+ const [saving, setSaving] = useState(false);+ const [countdown, setCountdown] = useState<number | null>(null);+ const countdownTimer = useRef<number | null>(null);++ useEffect(() => {+ let active = true;+ fetchAppshotSettings()+ .then((next) => active && setSettings(next))+ .catch((error: Error) => pushToast("error", error.message));+ return () => {+ active = false;+ };+ }, [pushToast]);++ // Refetch the last appshot whenever the backend reports a new one.+ useEffect(() => {+ let active = true;+ fetchLatestAppshot()+ .then((body) => active && setLatest(body.appshot))+ .catch(() => active && setLatest(null));+ return () => {+ active = false;+ };+ }, [lastAppshotEvent]);++ useEffect(+ () => () => {+ if (countdownTimer.current !== null) window.clearInterval(countdownTimer.current);+ },+ [],+ );++ const save = useCallback(+ async (patch: AppshotSettingsPatch) => {+ setSaving(true);+ setSettings((current) => (current ? { ...current, ...patch } : current));+ try {+ setSettings(await saveAppshotSettings(patch));+ } catch (error) {+ pushToast("error", t("appshots.save_error").replace("{0}", (error as Error).message));+ fetchAppshotSettings().then(setSettings).catch(() => undefined);+ } finally {+ setSaving(false);+ }+ },+ [pushToast, t],+ );++ const tryIt = useCallback(async () => {+ if (countdown !== null) return;+ setCountdown(TRY_DELAY_S);+ countdownTimer.current = window.setInterval(() => {+ setCountdown((value) => (value !== null && value > 1 ? value - 1 : value));+ }, 1000);+ try {+ const result = await takeAppshot(TRY_DELAY_S);+ if (result.ok) {+ setLatest(result.appshot);+ pushToast("success", t("appshots.try_done"));+ } else {+ pushToast("warning", t("appshots.toast_refused").replace("{0}", result.message));+ }+ } catch (error) {+ pushToast("error", (error as Error).message);+ } finally {+ if (countdownTimer.current !== null) window.clearInterval(countdownTimer.current);+ countdownTimer.current = null;+ setCountdown(null);+ }+ }, [countdown, pushToast, t]);++ const forget = useCallback(async () => {+ try {+ await forgetAppshots();+ setLatest(null);+ } catch (error) {+ pushToast("error", (error as Error).message);+ }+ }, [pushToast]);++ const hotkeyOptions = useMemo<BrandedSelectOption[]>(() => {+ const options: BrandedSelectOption[] = PRESET_HOTKEYS.map((value) => ({+ value: value || "off",+ label: value ? formatAppshotHotkey(value, IS_MAC) : t("appshots.shortcut_off"),+ }));+ const current = settings?.hotkey ?? "";+ if (current && !PRESET_HOTKEYS.includes(current as (typeof PRESET_HOTKEYS)[number])) {+ options.unshift({ value: current, label: formatAppshotHotkey(current, IS_MAC) });+ }+ return options;+ }, [settings?.hotkey, t]);++ const targetOptions = useMemo<BrandedSelectOption[]>(+ () => [+ { value: "auto", label: t("appshots.target_auto") },+ { value: "message", label: t("appshots.target_message") },+ { value: "voice", label: t("appshots.target_voice") },+ ],+ [t],+ );++ const shortcutHint = (() => {+ if (!settings) return "";+ if (!settings.hotkey) return t("appshots.shortcut_hint_off");+ if (settings.enabled && !settings.shortcut.armed && settings.shortcut.detail) {+ return t("appshots.shortcut_unavailable").replace("{0}", settings.shortcut.detail);+ }+ if (settings.hotkey === "alt+alt") {+ return IS_MAC ? t("appshots.shortcut_both_option_hint") : t("appshots.shortcut_both_alt_hint");+ }+ return t("appshots.shortcut_combo_hint").replace(+ "{0}",+ formatAppshotHotkey(settings.hotkey, IS_MAC),+ );+ })();++ const targetHint =+ settings?.target === "message"+ ? t("appshots.target_message_hint")+ : settings?.target === "voice"+ ? t("appshots.target_voice_hint")+ : t("appshots.target_auto_hint");++ const disabled = !settings || !settings.enabled;++ return (+ <div data-testid="appshots-view" className="flex h-full flex-col overflow-y-auto bg-background px-8 pb-10 scrollbar-jarvis">+ <PageHeader+ icon={<AppshotGlyph />}+ title={t("appshots.title")}+ description={t("appshots.subtitle")}+ />++ <div className="flex items-start gap-4 rounded-xl border border-border bg-card p-5">+ <AppshotGlyph className="mt-0.5 h-8 w-8 shrink-0 text-accent" />+ <div className="min-w-0">+ <p className="text-lg font-semibold text-foreground">{t("appshots.hero_title")}</p>+ <p className="mt-1 text-base text-muted-foreground">{t("appshots.hero_body")}</p>+ </div>+ </div>++ <div className="mt-5 grid gap-5 lg:grid-cols-2">+ <div className="divide-y divide-border self-start rounded-xl border border-border bg-card">+ {!settings ? (+ <div className="flex h-40 items-center justify-center" role="status" aria-busy="true">+ <Loader2 className="h-5 w-5 animate-spin text-muted-foreground" aria-hidden />+ </div>+ ) : (+ <>+ <Row+ label={t("appshots.enabled_label")}+ hint={t("appshots.enabled_hint")}+ control={+ <Switch+ checked={settings.enabled}+ disabled={saving}+ aria-label={t("appshots.enabled_label")}+ data-testid="appshots-enabled"+ onCheckedChange={(enabled) => void save({ enabled })}+ />+ }+ />+ <Row+ label={t("appshots.shortcut_label")}+ hint={shortcutHint}+ control={+ <BrandedSelect+ value={settings.hotkey || "off"}+ options={hotkeyOptions}+ ariaLabel={t("appshots.shortcut_label")}+ disabled={disabled || saving}+ testId="appshots-hotkey"+ className="w-44"+ onValueChange={(value) => void save({ hotkey: value === "off" ? "" : value })}+ />+ }+ />+ <Row+ label={t("appshots.target_label")}+ hint={targetHint}+ control={+ <BrandedSelect+ value={settings.target}+ options={targetOptions}+ ariaLabel={t("appshots.target_label")}+ disabled={disabled || saving}+ testId="appshots-target"+ className="w-44"+ onValueChange={(value) =>+ void save({ target: value as AppshotSettings["target"] })+ }+ />+ }+ />+ <Row+ label={t("appshots.sound_label")}+ hint={settings.sound_effects_master ? undefined : t("appshots.sound_master_off")}+ control={+ <Switch+ checked={settings.sound}+ disabled={disabled || saving}+ aria-label={t("appshots.sound_label")}+ data-testid="appshots-sound"+ onCheckedChange={(sound) => void save({ sound })}+ />+ }+ />+ <Row+ label={t("appshots.effect_label")}+ hint={+ settings.readiness.effect+ ? t("appshots.effect_hint")+ : t("appshots.effect_unavailable").replace("{0}", settings.readiness.effect_detail)+ }+ control={+ <Switch+ checked={settings.effect}+ disabled={disabled || saving}+ aria-label={t("appshots.effect_label")}+ data-testid="appshots-effect"+ onCheckedChange={(effect) => void save({ effect })}+ />+ }+ />+ <Row+ label={t("appshots.try_label")}+ hint={+ countdown !== null+ ? t("appshots.try_counting").replace("{0}", String(countdown))+ : t("appshots.try_hint")+ }+ control={+ <Button+ type="button"+ variant="secondary"+ size="sm"+ disabled={disabled || countdown !== null}+ onClick={() => void tryIt()}+ data-testid="appshots-try"+ >+ {countdown !== null && <Loader2 className="animate-spin" aria-hidden />}+ {t("appshots.try_button")}+ </Button>+ }+ />+ </>+ )}+ </div>++ <div className="rounded-xl border border-border bg-card p-5">+ <p className="mb-3 text-base font-medium text-foreground">{t("appshots.preview_title")}</p>+ {latest ? <LatestPreview shot={latest} onForget={() => void forget()} /> : <PreviewDemo />}+ </div>+ </div>++ <p className="mt-5 text-sm text-muted-foreground">{t("appshots.voice_hint")}</p>+ </div>+ );+}diff --git a/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx b/jarvis/ui/web/frontend/src/views/SettingsHubView.tsxindex 558b442d2a..ab4993964e 100644--- a/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx+++ b/jarvis/ui/web/frontend/src/views/SettingsHubView.tsx@@ -3,6 +3,7 @@ import { ArrowLeft, Loader2, Search, Settings as SettingsIcon, X } from "lucide-import {NAV_FOOTER_ITEMS,NAV_GROUPS,+ SETTINGS_HUB_ONLY_ITEMS,resolveNavLabel,type NavItem,} from "@/components/layout/navGroups";@@ -18,7 +19,7 @@ import { cn } from "@/lib/utils";* searchable left navigation (Personal · System · Activity) and the selected* section on the right:*- * General: Settings, Profile, {name}.md, Contacts, Socials+ * General: Settings, Appshots, Profile, {name}.md, Contacts, Socials* System: Computers, API Keys, Local models, Wallpaper* Activity: Spend, Feedback*@@ -77,6 +78,9 @@ const ComputersTab = lazy(() =>const WallpaperTab = lazy(() =>import("@/views/WallpaperView").then((m) => ({ default: m.WallpaperView })),);+const AppshotsTab = lazy(() =>+ import("@/views/AppshotsView").then((m) => ({ default: m.AppshotsView })),+);const CostsTab = lazy(() =>import("@/views/CostsView").then((m) => ({ default: m.CostsView })),);@@ -89,6 +93,7 @@ const FeedbackTab = lazy(() =>/** The eleven entries of the left navigation, in display order. */type HubNavId =| "settings"+ | "appshots"| "profile"| "agent-instructions"| "contacts"@@ -103,7 +108,7 @@ type HubNavId =const HUB_NAV_GROUPS: readonly { labelKey: string; ids: readonly HubNavId[] }[] = [{labelKey: "settings_hub.group_general",- ids: ["settings", "profile", "agent-instructions", "contacts", "socials"],+ ids: ["settings", "appshots", "profile", "agent-instructions", "contacts", "socials"],},{labelKey: "settings_hub.group_system",@@ -126,6 +131,7 @@ const TAB_CONTENT: Record<HubNavId | "telephony-setup", LazyExoticComponent<Comp"local-models": LocalModelsTab,computers: ComputersTab,wallpaper: WallpaperTab,+ appshots: AppshotsTab,costs: CostsTab,feedback: FeedbackTab,};@@ -156,6 +162,8 @@ function resolveHubTab(active: string): { content: HubNavId | "telephony-setup";return { content: "computers", highlight: "computers" };case "wallpaper":return { content: "wallpaper", highlight: "wallpaper" };+ case "appshots":+ return { content: "appshots", highlight: "appshots" };case "costs":return { content: "costs", highlight: "costs" };case "feedback":@@ -171,7 +179,11 @@ function resolveHubTab(active: string): { content: HubNavId | "telephony-setup";// `NAV_GROUPS` plus the footer: "feedback" lives in `NAV_FOOTER_ITEMS`, not in// a group (same lookup as TopBar/DockRail) — without it the hub cannot resolve// its own tenth entry.-const ALL_NAV_ITEMS: readonly NavItem[] = [...NAV_GROUPS.flat(), ...NAV_FOOTER_ITEMS];+const ALL_NAV_ITEMS: readonly NavItem[] = [+ ...NAV_GROUPS.flat(),+ ...NAV_FOOTER_ITEMS,+ ...SETTINGS_HUB_ONLY_ITEMS,+];function findNavItem(id: HubNavId): NavItem {const item = ALL_NAV_ITEMS.find((row) => row.id === id);diff --git a/jarvis/ui/web/frontend/src/views/SettingsView.tsx b/jarvis/ui/web/frontend/src/views/SettingsView.tsxindex 5ed69cfcef..cda8e6ca3c 100644--- a/jarvis/ui/web/frontend/src/views/SettingsView.tsx+++ b/jarvis/ui/web/frontend/src/views/SettingsView.tsx@@ -22,7 +22,6 @@ import { VolumeGroup } from "@/views/settings/VolumeGroup";import { AudioDevicesGroup } from "@/views/settings/AudioDevicesGroup";import { SystemPromptGroup } from "@/views/settings/SystemPromptGroup";import { SettingsGroupBoundary } from "@/views/settings/SettingsGroupBoundary";-import { ScreenContextGroup } from "@/views/settings/ScreenContextGroup";import { settingsInputCls } from "@/views/settings/SettingsBlock";import {useWakeWord,@@ -161,7 +160,6 @@ const SECTIONS: readonly { id: string; labelKey: string; render: () => React.Rea{ id: "languages", labelKey: "settings_view.nav.languages", render: () => <LanguagesGroup /> },{ id: "app", labelKey: "settings_view.nav.app", render: () => <AppSettingsGroup /> },{ id: "permissions", labelKey: "settings_view.nav.permissions", render: () => <PermissionsPanel /> },- { id: "screen-context", labelKey: "settings_view.nav.screen_context", render: () => <ScreenContextGroup /> },{ id: "realtime-voice", labelKey: "settings_view.nav.realtime_voice", render: () => <RealtimeVoiceGroup /> },{ id: "system-prompt", labelKey: "settings_view.nav.system_prompt", render: () => <SystemPromptGroup /> },{ id: "wake-word", labelKey: "settings_view.nav.wake_word", render: () => <WakeWordPanel /> },diff --git a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.test.tsx b/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.test.tsxdeleted file mode 100644index 0f2a7d6b81..0000000000--- a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.test.tsx+++ /dev/null@@ -1,128 +0,0 @@-import { fireEvent, render, screen, waitFor } from "@testing-library/react";-import { afterEach, describe, expect, it, vi } from "vitest";--import { ScreenContextGroup } from "./ScreenContextGroup";--/**- * Screen Context is a ONE-SWITCH card (maintainer mandate 2026-08-02). These- * tests pin that contract from the outside: the only control is the switch, a- * blocker is reported honestly while the feature is on, and nothing on the card- * asks the user to write privacy rules by hand.- */--const SETTINGS = { enabled: true };--function status(available: boolean, enabled = true) {- return {- enabled,- available,- blocked_reason: available ? null : "No vision-capable provider is configured.",- blocked_reasons: available- ? []- : ["No vision-capable provider is configured."],- monitor_count: 2,- };-}--function response(body: unknown) {- return { ok: true, json: async () => body };-}--afterEach(() => vi.restoreAllMocks());--describe("ScreenContextGroup", () => {- it("reports the real blocker while the feature is switched on", async () => {- vi.stubGlobal(- "fetch",- vi.fn(async (url: string) =>- response(url.endsWith("/settings") ? SETTINGS : status(false)),- ),- );-- render(<ScreenContextGroup />);-- await waitFor(() =>- expect(- screen.getByText("No vision-capable provider is configured."),- ).toBeTruthy(),- );- });-- it("offers exactly one control and no privacy-rule editors", async () => {- vi.stubGlobal(- "fetch",- vi.fn(async (url: string) =>- response(url.endsWith("/settings") ? SETTINGS : status(true)),- ),- );-- const { container } = render(<ScreenContextGroup />);-- await waitFor(() => expect(screen.getByText(/Ready on 2/)).toBeTruthy());- expect(container.querySelectorAll("textarea")).toHaveLength(0);- expect(container.querySelectorAll("input")).toHaveLength(0);- // The old card shipped Save / Test / Discard buttons plus three more- // switches next to this one.- const buttons = Array.from(container.querySelectorAll("button"));- expect(buttons).toHaveLength(1);- expect(buttons[0].getAttribute("role")).toBe("switch");- });-- it("runs full width like every neighbouring settings card", async () => {- vi.stubGlobal(- "fetch",- vi.fn(async (url: string) =>- response(url.endsWith("/settings") ? SETTINGS : status(true)),- ),- );-- const { container } = render(<ScreenContextGroup />);- await waitFor(() => expect(screen.getByText(/Ready on 2/)).toBeTruthy());-- // A leftover `max-w-5xl` from the multi-field card capped this one at half- // the window and parked its switch mid-row next to full-width neighbours.- // The settings column owns the width; the card must not cap it.- const root = container.firstElementChild;- expect(root).toBeTruthy();- const capped = Array.from(container.querySelectorAll("*"))- .concat(root ? [root] : [])- .filter((element) =>- Array.from(element.classList).some((name) => name.startsWith("max-w-")),- );- expect(capped).toHaveLength(0);- // Same shell as RealtimeVoiceGroup, so the two read as one list.- expect(root?.className).toBe(- "mt-2 rounded-lg border border-border bg-card p-4",- );- });-- it("writes only the enabled flag when the switch is flipped", async () => {- const fetchMock = vi.fn(async (url: string, init?: RequestInit) => {- if (url.endsWith("/settings") && init?.method === "PUT") {- return response({ ok: true, changed: ["enabled"] });- }- if (url.endsWith("/settings")) return response(SETTINGS);- if (url.endsWith("/status")) return response(status(true));- throw new Error(`Unexpected request: ${url}`);- });- vi.stubGlobal("fetch", fetchMock);-- const { container } = render(<ScreenContextGroup />);- const toggle = await waitFor(() => {- const found = container.querySelector("button[role=switch]");- if (!found) throw new Error("switch not rendered yet");- return found;- });- fireEvent.click(toggle);-- await waitFor(() => {- const put = fetchMock.mock.calls.find(- ([, init]) => (init as RequestInit | undefined)?.method === "PUT",- );- expect(put).toBeTruthy();- expect(JSON.parse(String((put?.[1] as RequestInit).body))).toEqual({- enabled: false,- });- });- });-});diff --git a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.tsx b/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.tsxdeleted file mode 100644index 9393ad47b2..0000000000--- a/jarvis/ui/web/frontend/src/views/settings/ScreenContextGroup.tsx+++ /dev/null@@ -1,149 +0,0 @@-import { useEffect, useState } from "react";-import { Eye, Loader2 } from "lucide-react";--import { Switch } from "@/components/ui/switch";-import { useT } from "@/i18n";-import { useEventStore } from "@/store/events";--/**- * Screen Context is deliberately a ONE-SWITCH feature (maintainer mandate- * 2026-08-02). The privacy rules it used to expose here — denylist, extra- * patterns, OCR, retention seconds — are still enforced at their shipped- * defaults and remain editable through `jarvis api screen-context put-settings`- * for anyone who wants them, but a settings card that asks a non-technical user- * to write regular expressions is a card they switch off instead of using.- */-interface ScreenContextSettings {- enabled: boolean;-}--interface ScreenContextStatus {- enabled: boolean;- available: boolean;- blocked_reason: string | null;- monitor_count: number;-}--async function jsonRequest<T>(url: string, init?: RequestInit): Promise<T> {- const response = await fetch(url, init);- const body = (await response.json().catch(() => null)) as- | T- | { detail?: string }- | null;- if (!response.ok) {- const detail =- body !== null &&- typeof body === "object" &&- "detail" in body &&- typeof body.detail === "string"- ? body.detail- : "";- throw new Error(- detail || `Screen Context request failed (${response.status}).`,- );- }- return body as T;-}--export function ScreenContextGroup() {- const t = useT();- const pushToast = useEventStore((state) => state.pushToast);- const [settings, setSettings] = useState<ScreenContextSettings | null>(null);- const [status, setStatus] = useState<ScreenContextStatus | null>(null);- const [loading, setLoading] = useState(true);- const [saving, setSaving] = useState(false);-- useEffect(() => {- let active = true;- void Promise.all([- jsonRequest<ScreenContextSettings>("/api/screen-context/settings"),- jsonRequest<ScreenContextStatus>("/api/screen-context/status"),- ])- .then(([nextSettings, nextStatus]) => {- if (!active) return;- setSettings(nextSettings);- setStatus(nextStatus);- })- .catch((error: Error) => pushToast("error", error.message))- .finally(() => {- if (active) setLoading(false);- });- return () => {- active = false;- };- }, [pushToast]);-- async function toggle(enabled: boolean) {- setSaving(true);- try {- await jsonRequest("/api/screen-context/settings", {- method: "PUT",- headers: { "content-type": "application/json" },- body: JSON.stringify({ enabled }),- });- setSettings((current) => (current === null ? current : { enabled }));- setStatus(- await jsonRequest<ScreenContextStatus>("/api/screen-context/status"),- );- pushToast("success", t("settings_view.screen_context.saved"));- } catch (error) {- pushToast("error", (error as Error).message);- } finally {- setSaving(false);- }- }-- // A blocker is only worth showing while the feature is meant to be running:- // "switched off" is not a problem to report, it is what the switch says.- const showBlocker = Boolean(settings?.enabled && status && !status.available);-- // Layout is deliberately byte-for-byte the neighbouring RealtimeVoiceGroup- // shell: same wrapper, same icon placement, same spacing. The card carried a- // `max-w-5xl` from its multi-field days, which capped it at half the window- // and parked its switch mid-row while every other card ran full width — a- // mismatch that only became visible once the card got short. Do not- // reintroduce a width cap here; the settings column owns the width.- return (- <div className="mt-2 rounded-lg border border-border bg-card p-4">- <div className="flex items-start gap-3">- <Eye className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" />- <div className="min-w-0 flex-1">- <div className="flex items-center justify-between gap-4">- <h4 className="font-medium">- {t("settings_view.screen_context.title")}- </h4>- {loading || !settings ? (- <Loader2 className="h-4 w-4 animate-spin text-muted-foreground" />- ) : (- <Switch- checked={settings.enabled}- disabled={saving}- aria-label={t("settings_view.screen_context.title")}- onCheckedChange={(enabled) => void toggle(enabled)}- />- )}- </div>- <p className="mt-0.5 text-xs text-muted-foreground">- {t("settings_view.screen_context.description")}- </p>- {settings?.enabled && status && (- <p- className={`mt-1.5 text-micro ${- status.available ? "text-muted-foreground" : "text-foreground"- }`}- aria-live="polite"- >- {showBlocker- ? status.blocked_reason ||- t("settings_view.screen_context.unavailable")- : t("settings_view.screen_context.available").replace(- "{0}",- String(status.monitor_count),- )}- </p>- )}- </div>- </div>- </div>- );-}diff --git a/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts b/jarvis/ui/web/frontend/src/views/settings/settingsSearch.tsindex a2d0dc3a44..8de537f254 100644--- a/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts+++ b/jarvis/ui/web/frontend/src/views/settings/settingsSearch.ts@@ -12,7 +12,6 @@ const SEARCH_GROUPS = [{ id: "languages", keys: ["language", "languages_group_title"] },{ id: "app", keys: ["app_settings_group_title", "autostart", "appearance", "jarvis_api"] },{ id: "permissions", keys: ["nav.permissions"] },- { id: "screen-context", keys: ["screen_context"] },{ id: "realtime-voice", keys: ["realtime_voice"] },{ id: "system-prompt", keys: ["system_prompt"] },{ id: "wake-word", keys: ["wake_word"] },@@ -32,6 +31,7 @@ const SEARCH_PAGES = [{ id: "socials", keys: ["socials"] },{ id: "apikeys", keys: ["apikeys_view", "apikeys_voice", "apikeys_model", "apikeys_cu_model"] },{ id: "wallpaper", keys: ["home.background_label", "home.background_hint"] },+ { id: "appshots", keys: ["appshots"] },{ id: "costs", keys: ["costs_view"] },{ id: "feedback", keys: ["feedback"] },] as const;diff --git a/jarvis/ui/web/server.py b/jarvis/ui/web/server.pyindex 36eae721ab..4c7ff92b4e 100644--- a/jarvis/ui/web/server.py+++ b/jarvis/ui/web/server.py@@ -391,6 +391,7 @@ class WebServer:from .provider_routes import router as provider_routerfrom .review_routes import router as review_routerfrom .routine_hooks_routes import router as routine_hooks_router+ from .appshot_routes import router as appshot_routerfrom .screen_context_routes import router as screen_context_routerfrom .self_mod_routes import router as self_mod_routerfrom .sessions_routes import router as sessions_router@@ -565,6 +566,9 @@ class WebServer:# machine with no display, so `jarvis api screen-context status` is a# valid capability probe everywhere.app.include_router(screen_context_router)+ # Appshots: the front window as conversation context, on a shortcut,+ # a button or a spoken request. Captures through Screen Context.+ app.include_router(appshot_router)# The mission deck's pictures: the last Screen-Context capture (one# frame, in memory, TTL) and Computer-Use frames by content hash.app.include_router(deck_router)@@ -2392,6 +2396,30 @@ class WebServer:"will be renewed in the background.")+ def _schedule_appshot_shortcut(self) -> None:+ """Arm the global appshot shortcut once the wake model has loaded.++ Off the boot path (AP-26): the key reader and the hotkey backend load+ native libraries. Headless hosts arm nothing and say so in the log.+ """+ try:+ loop = asyncio.get_running_loop()+ except RuntimeError:+ logger.debug("Appshot shortcut not scheduled — no running event loop.")+ return++ async def _arm() -> None:+ from jarvis.appshot.hotkey import start_appshot_shortcut+ from jarvis.core import runtime_refs as _rr++ await _rr.await_wake_model_ready(timeout=12.0)+ try:+ await start_appshot_shortcut(self.bus)+ except Exception as exc: # noqa: BLE001 - voice/chat work without it+ logger.opt(exception=exc).warning("Appshot shortcut could not start")++ self._appshot_shortcut_task = loop.create_task(_arm(), name="appshot-shortcut")+def _schedule_realtime_transport_warm(self) -> None:"""Pre-open the selected realtime transports off the boot path.@@ -2831,6 +2859,7 @@ class WebServer:# browser-only install has no desktop shell to warm the realtime# transport for it.self._schedule_realtime_transport_warm()+ self._schedule_appshot_shortcut()# Defer provisioning until the boot chain returns control to the server.async def prepare_browser() -> None:from jarvis.society.browser import installdiff --git a/tests/unit/appshot/__init__.py b/tests/unit/appshot/__init__.pynew file mode 100644index 0000000000..e69de29bb2diff --git a/tests/unit/appshot/test_appshot_core.py b/tests/unit/appshot/test_appshot_core.pynew file mode 100644index 0000000000..b0ec8dc842--- /dev/null+++ b/tests/unit/appshot/test_appshot_core.py@@ -0,0 +1,425 @@+"""Appshots — store, gesture, placement, shortcut spelling and turn delivery.++Everything here runs without a display: the key reader, the capture service+and the voice session are small fakes, so the contract is checked on a+headless CI box exactly as on a desktop.+"""++from __future__ import annotations++import pytest++from jarvis.appshot import service as appshot_service+from jarvis.appshot.effect import placement+from jarvis.appshot.gesture import BothAltWatcher+from jarvis.appshot.hotkey import normalize_hotkey+from jarvis.appshot.store import Appshot, AppshotStore, get_store+from jarvis.brain.manager import _takes_pending_appshot+from jarvis.screen_context.models import (+ CaptureTarget,+ IntentVerdict,+ ScreenContext,+ TargetKind,+ TargetReason,+ VisualIntent,+ WindowFacts,+)+from jarvis.screen_context.service import CaptureOutcome+from jarvis.screen_context.turn import screen_context_for_turn+++class Clock:+ def __init__(self) -> None:+ self.now = 100.0++ def __call__(self) -> float:+ return self.now+++def make_shot(shot_id: str = "a1") -> Appshot:+ return Appshot(+ id=shot_id,+ image=b"jpeg",+ mime="image/jpeg",+ width=800,+ height=600,+ label="active window",+ app_name="Editor",+ note="APPSHOT: evidence",+ ui_text="hello",+ trigger="hotkey",+ taken_at=1.0,+ )+++def make_context() -> ScreenContext:+ return ScreenContext(+ image=b"jpeg-bytes",+ mime="image/jpeg",+ size=(1280, 720),+ target=CaptureTarget(+ kind=TargetKind.WINDOW,+ bbox=(10, 20, 1280, 720),+ reason=TargetReason.FOCUSED_WINDOW,+ window=WindowFacts(app_name="Editor", title="notes.md"),+ ),+ ui_text="Build failed",+ captured_at_ns=1,+ )+++# --------------------------------------------------------------------- store+++def test_pending_appshot_is_single_use_and_expires() -> None:+ clock = Clock()+ store = AppshotStore(clock=clock)+ store.park(make_shot("a1"), ttl_s=60)++ assert store.peek_pending() is not None+ assert store.take_pending("other") is None, "a foreign id never claims it"+ assert store.take_pending("a1").id == "a1"+ assert store.take_pending() is None, "the second take gets nothing"++ store.park(make_shot("a2"), ttl_s=60)+ clock.now += 61+ assert store.take_pending() is None, "an expired appshot is gone"+++def test_latest_is_kept_for_its_budget_and_zero_keeps_nothing() -> None:+ clock = Clock()+ store = AppshotStore(clock=clock)+ store.remember(make_shot(), keep_s=120)+ store.mark_delivered("a1", "voice")+ assert store.latest().delivered_to == "voice"+ clock.now += 121+ assert store.latest() is None++ store.remember(make_shot(), keep_s=0)+ assert store.latest() is None+++# ------------------------------------------------------------------- gesture+++class Keys:+ def __init__(self) -> None:+ self.state: tuple[bool, bool] | None = (False, False)++ def __call__(self):+ return self.state+++def test_both_alt_fires_once_per_press_and_rearms_on_release(monkeypatch) -> None:+ import jarvis.platform.self_input as self_input++ monkeypatch.setattr(self_input, "synthetic_input_recent", lambda *a, **k: False)+ keys, clock, fired = Keys(), Clock(), []+ watcher = BothAltWatcher(lambda: fired.append(1), probe=keys, clock=clock)++ keys.state = (True, False)+ assert not watcher.step(), "one Alt key alone is ordinary typing"+ keys.state = (True, True)+ assert watcher.step()+ assert not watcher.step(), "holding the keys does not repeat"+ keys.state = (False, False)+ watcher.step()+ clock.now += 1.0+ keys.state = (True, True)+ assert watcher.step(), "a new press after release fires again"+ assert len(fired) == 2+++def test_both_alt_ignores_a_bounce_and_synthetic_input(monkeypatch) -> None:+ import jarvis.platform.self_input as self_input++ keys, clock, fired = Keys(), Clock(), []+ watcher = BothAltWatcher(lambda: fired.append(1), probe=keys, clock=clock)+ monkeypatch.setattr(self_input, "synthetic_input_recent", lambda *a, **k: False)+ keys.state = (True, True)+ watcher.step()+ keys.state = (False, False)+ watcher.step()+ clock.now += 0.1+ keys.state = (True, True)+ assert not watcher.step(), "a key bounce inside the guard is the same press"++ keys.state = (False, False)+ watcher.step()+ clock.now += 5+ monkeypatch.setattr(self_input, "synthetic_input_recent", lambda *a, **k: True)+ keys.state = (True, True)+ assert not watcher.step(), "Jarvis's own keystrokes never take an appshot"+ assert len(fired) == 1+++def test_unreadable_keyboard_never_fires() -> None:+ keys, fired = Keys(), []+ keys.state = None+ watcher = BothAltWatcher(lambda: fired.append(1), probe=keys)+ assert not watcher.step()+ assert fired == []+++# ------------------------------------------------------------ effect placing+++MONITORS = [+ {"left": 0, "top": 0, "width": 3840, "height": 1080},+ {"left": 0, "top": 0, "width": 1920, "height": 1080},+ {"left": 1920, "top": 0, "width": 1920, "height": 1080},+]+++def test_placement_picks_the_monitor_under_the_window_centre() -> None:+ monitor, rect = placement((2400, 100, 960, 540), MONITORS)+ assert monitor == [1920, 0, 1920, 1080]+ assert rect == pytest.approx([0.25, 100 / 1080, 0.5, 0.5])+++def test_placement_clips_a_window_hanging_off_its_monitor() -> None:+ monitor, rect = placement((1500, 0, 800, 1080), MONITORS)+ assert monitor == [0, 0, 1920, 1080]+ assert rect[0] + rect[2] == pytest.approx(1.0)+++def test_placement_without_monitors_covers_the_capture() -> None:+ monitor, rect = placement((0, 0, 100, 50), [])+ assert monitor == [0, 0, 100, 50]+ assert rect == [0.0, 0.0, 1.0, 1.0]+++# ---------------------------------------------------------------- shortcuts+++ ("raw", "expected"),+ [+ ("alt+alt", "alt+alt"),+ ("Left_Alt + Right_Alt", "alt+alt"),+ ("altgr+alt", "alt+alt"),+ ("Ctrl + Alt + A", "ctrl+alt+a"),+ ("", ""),+ ],+)+def test_shortcut_spelling_is_canonical(raw: str, expected: str) -> None:+ assert normalize_hotkey(raw) == expected+++ ("layer", "takes"),+ [+ ("speech", True),+ ("ui.web.chat", True),+ (None, True),+ ("tasks.runner", False),+ ("workflows.scheduler", False),+ ("society.lead", False),+ ("tool.run-skill", False),+ ],+)+def test_only_a_person_waiting_takes_the_parked_appshot(layer, takes) -> None:+ assert _takes_pending_appshot(layer) is takes+++# ------------------------------------------------------------ turn delivery+++def clean_store():+ get_store().clear()+ yield get_store()+ get_store().clear()+++class NeverCaptures:+ """A service that fails the test if the turn tries to take a new picture."""++ async def capture_for_turn(self, *args, **kwargs):+ raise AssertionError("the parked appshot should have answered this turn")+++async def test_the_next_message_carries_the_parked_appshot(clean_store) -> None:+ clean_store.park(make_shot("p1"), ttl_s=60)++ result = await screen_context_for_turn(+ "what is wrong here?", locale="en", service=NeverCaptures()+ )++ assert result.has_image+ assert result.image == b"jpeg"+ assert result.note.startswith("APPSHOT")+ assert clean_store.peek_pending() is None, "single use"+++async def test_an_automated_turn_leaves_the_parked_appshot_alone(clean_store) -> None:+ clean_store.park(make_shot("p1"), ttl_s=60)++ class NoVisualIntent:+ async def capture_for_turn(self, *args, **kwargs):+ return CaptureOutcome(+ status="not_requested", verdict=IntentVerdict(intent=VisualIntent.NONE)+ )++ result = await screen_context_for_turn(+ "summarise the news",+ locale="en",+ service=NoVisualIntent(),+ allow_pending_appshot=False,+ )++ assert result.status == "none"+ assert clean_store.peek_pending() is not None+++async def test_asking_for_a_new_appshot_drops_the_stale_one(clean_store) -> None:+ clean_store.park(make_shot("old"), ttl_s=60)++ class Captures:+ def __init__(self) -> None:+ self.asked = False++ async def capture_for_turn(self, *args, **kwargs):+ self.asked = True+ return CaptureOutcome(+ status="refused",+ verdict=IntentVerdict(intent=VisualIntent.WINDOW),+ reason_kind="policy",+ message="switched off",+ )++ service = Captures()+ await screen_context_for_turn("take an appshot", locale="en", service=service)++ assert service.asked, "a fresh capture was attempted"+ assert clean_store.peek_pending() is None+++# ----------------------------------------------------- shortcut destination+++class Config:+ class screen_context: # noqa: N801 - mirrors the config attribute+ enabled = True+ ttl_s = 120.0+ deck_preview_s = 120.0++ class appshot: # noqa: N801+ target = "auto"+++class FakeCaptureService:+ def __init__(self) -> None:+ self.consumed: list[str] = []+ self.verdicts: list[IntentVerdict] = []++ async def capture(self, *, verdict=None, trace_id=None):+ self.verdicts.append(verdict)+ return CaptureOutcome(+ status="captured", verdict=verdict, context=make_context(), handle_id="h1"+ )++ def consume(self, handle_id):+ self.consumed.append(handle_id)+++class Bus:+ def __init__(self) -> None:+ self.events: list = []++ async def publish(self, event) -> None:+ self.events.append(event)+++def wired(monkeypatch, clean_store):+ import jarvis.appshot.delivery as delivery+ import jarvis.screen_context.turn as turn++ service = FakeCaptureService()+ config = Config()+ live = {"accepts": False, "calls": 0}++ async def deliver_to_live(image, mime, note):+ live["calls"] += 1+ return live["accepts"]++ monkeypatch.setattr(turn, "get_service", lambda bus=None: service)+ monkeypatch.setattr(appshot_service, "_load_config", lambda: config)+ monkeypatch.setattr(delivery, "deliver_to_live", deliver_to_live)+ return service, config, live, clean_store+++async def test_without_a_call_the_appshot_waits_for_the_next_message(wired) -> None:+ service, _config, _live, store = wired+ bus = Bus()++ result = await appshot_service.take_appshot(trigger="hotkey", bus=bus)++ assert result.ok and result.shot.delivered_to == "message"+ assert service.verdicts[0].intent is VisualIntent.WINDOW, "an appshot is the front window"+ assert service.consumed == ["h1"], "the capture handle is not left in memory"+ assert store.peek_pending().id == result.shot.id+ assert store.latest().app_name == "Editor"+ event = bus.events[-1]+ assert event.delivered_to == "message"+ assert "notes.md" not in repr(event), "a window title never enters the bus"+++async def test_a_running_call_receives_the_appshot_directly(wired) -> None:+ _service, _config, live, store = wired+ live["accepts"] = True++ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())++ assert result.shot.delivered_to == "voice"+ assert store.peek_pending() is None, "a delivered appshot is not also parked"+++async def test_voice_only_target_without_a_call_sends_nothing(wired) -> None:+ _service, config, _live, store = wired+ config.appshot.target = "voice"+ try:+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())+ finally:+ config.appshot.target = "auto"++ assert result.shot.delivered_to == "none"+ assert store.peek_pending() is None+++async def test_message_target_never_interrupts_a_call(wired) -> None:+ _service, config, live, _store = wired+ live["accepts"] = True+ config.appshot.target = "message"+ try:+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())+ finally:+ config.appshot.target = "auto"++ assert result.shot.delivered_to == "message"+ assert live["calls"] == 0+++async def test_switched_off_appshots_refuse_without_capturing(wired) -> None:+ service, config, _live, _store = wired+ config.screen_context.enabled = False+ try:+ result = await appshot_service.take_appshot(trigger="hotkey", bus=Bus())+ finally:+ config.screen_context.enabled = True++ assert not result.ok+ assert result.reason_code == "disabled"+ assert service.verdicts == []+++async def test_the_tool_keeps_the_picture_for_its_own_answer(wired) -> None:+ _service, _config, live, store = wired+ live["accepts"] = True++ result = await appshot_service.take_appshot(trigger="tool", bus=Bus(), deliver=False)++ assert result.shot.delivered_to == "turn"+ assert live["calls"] == 0+ assert store.peek_pending() is Nonediff --git a/tests/unit/appshot/test_appshot_surfaces.py b/tests/unit/appshot/test_appshot_surfaces.pynew file mode 100644index 0000000000..b4b1a310e8--- /dev/null+++ b/tests/unit/appshot/test_appshot_surfaces.py@@ -0,0 +1,226 @@+"""Appshots on every surface: speech vocabulary, voice calls, config, REST."""++from __future__ import annotations++import base64+import tomllib+from types import SimpleNamespace++import pytest+from fastapi import FastAPI+from fastapi.testclient import TestClient++from jarvis.core.config_writer import set_appshot_settings+from jarvis.live.native import NativeLiveVoiceSession+from jarvis.live.session import LiveVoiceSession+from jarvis.screen_context.intent import classify, mentions_appshot+from jarvis.screen_context.models import VisualIntent++# ------------------------------------------------------------------ speech+++ "utterance",+ [+ "take an appshot",+ "Mach mal einen Appshot", # i18n-allow: German speech-input test data+ "mach einen App Shot und sag mir was da steht", # i18n-allow: DE input+ "Appshot bitte", # i18n-allow: German speech-input test data+ "hazme un app shot",+ ],+)+def test_naming_the_feature_takes_the_front_window(utterance: str) -> None:+ assert classify(utterance, locale="en").intent is VisualIntent.WINDOW+ assert mentions_appshot(utterance)+++ "utterance",+ [+ "what is an appshot?",+ "how do appshots work",+ "Was ist ein Appshot?", # i18n-allow: German speech-input test data+ "Öffne die Appshots Einstellungen", # i18n-allow: DE input+ "geh zu Appshots", # i18n-allow: German speech-input test data+ "open the appshot settings",+ "the snapshot looks fine",+ "happy shots all round",+ ],+)+def test_asking_about_or_opening_appshots_takes_nothing(utterance: str) -> None:+ assert classify(utterance, locale="en").intent is VisualIntent.NONE+++# ------------------------------------------------------------- voice calls+++class Recorder:+ def __init__(self) -> None:+ self.sent: list[dict] = []+ self.images: list[tuple[bytes, str]] = []+ self.answer_sdp = ""++ async def send(self, event: dict) -> None:+ self.sent.append(event)++ async def send_image(self, image: bytes, mime: str) -> None:+ self.images.append((image, mime))+++async def _noop(*_args, **_kwargs) -> None:+ return None+++def _config():+ return SimpleNamespace(brain=SimpleNamespace(reply_language="en"))+++async def test_gpt_live_puts_the_appshot_into_the_backend_silently() -> None:+ session = LiveVoiceSession(+ session_id="s",+ send_binary=_noop,+ send_json=_noop,+ providers=[SimpleNamespace(name="openai")],+ config=_config(),+ )+ session._connection = Recorder()++ assert await session.attach_appshot(b"jpeg", "image/jpeg", "APPSHOT: note")++ kinds = [event["type"] for event in session._connection.sent]+ assert kinds == ["session.thinking.append", "response.item.create"]+ assert "response.create" not in kinds, "the user's next words are the question"+ content = session._connection.sent[1]["item"]["content"]+ assert content[0] == {"type": "input_text", "text": "APPSHOT: note"}+ assert content[1]["image_url"] == "data:image/jpeg;base64," + base64.b64encode(+ b"jpeg"+ ).decode("ascii")+++async def test_gpt_live_declines_while_it_is_recovering() -> None:+ session = LiveVoiceSession(+ session_id="s",+ send_binary=_noop,+ send_json=_noop,+ providers=[SimpleNamespace(name="openai")],+ config=_config(),+ )+ session._connection = Recorder()+ session._recovering = True++ assert not await session.attach_appshot(b"jpeg", "image/jpeg", "note")+ assert session._connection.sent == []+++async def test_native_live_sends_a_frame_and_no_text_turn() -> None:+ session = NativeLiveVoiceSession(+ session_id="s",+ send_binary=_noop,+ send_json=_noop,+ providers=[SimpleNamespace(name="gemini")],+ config=_config(),+ )+ session._connection = Recorder()++ assert await session.attach_appshot(b"png", "image/png", "note")+ assert session._connection.images == [(b"png", "image/png")]+ assert session._connection.sent == []+++async def test_native_live_without_image_input_declines() -> None:+ session = NativeLiveVoiceSession(+ session_id="s",+ send_binary=_noop,+ send_json=_noop,+ providers=[SimpleNamespace(name="local")],+ config=_config(),+ )+ session._connection = SimpleNamespace(answer_sdp="")++ assert not await session.attach_appshot(b"png", "image/png", "note")+++# ------------------------------------------------------------------ config+++def test_appshot_patch_writes_one_table(tmp_path) -> None:+ path = tmp_path / "jarvis.toml"+ path.write_text("[ui]\nsound_effects = true\n", encoding="utf-8")++ set_appshot_settings({"hotkey": "alt+alt", "target": "message", "sound": False}, path=path)++ data = tomllib.loads(path.read_text(encoding="utf-8"))+ assert data["appshot"] == {"hotkey": "alt+alt", "target": "message", "sound": False}+ assert data["ui"] == {"sound_effects": True}+++def test_appshot_patch_rejects_unknown_keys_before_writing(tmp_path) -> None:+ path = tmp_path / "jarvis.toml"+ path.write_text("", encoding="utf-8")++ with pytest.raises(ValueError, match="unknown appshot"):+ set_appshot_settings({"enabled": True}, path=path)+ assert path.read_text(encoding="utf-8") == ""+++# -------------------------------------------------------------------- REST+++def client() -> TestClient:+ from jarvis.ui.web.appshot_routes import router++ app = FastAPI()+ app.include_router(router)+ app.state.bus = None+ return TestClient(app)+++def test_an_empty_settings_patch_is_refused_without_writing(client) -> None:+ assert client.put("/api/appshot/settings", json={}).status_code == 400+++def test_an_unknown_destination_is_refused(client) -> None:+ assert client.put("/api/appshot/settings", json={"target": "email"}).status_code == 422+++def test_take_reports_a_refusal_as_data(client, monkeypatch) -> None:+ import jarvis.appshot.service as service++ async def refuse(**_kwargs):+ return service.AppshotResult(status="refused", reason_code="disabled", message="off")++ monkeypatch.setattr(service, "take_appshot", refuse)+ body = client.post("/api/appshot/take", json={"delay_s": 0}).json()+ assert body == {"ok": False, "reason": "disabled", "message": "off"}+++def test_the_parked_appshot_is_claimed_once(client) -> None:+ from jarvis.appshot.store import Appshot, get_store++ store = get_store()+ store.clear()+ store.park(+ Appshot(+ id="c1",+ image=b"jpeg",+ mime="image/jpeg",+ width=1,+ height=1,+ label="active window",+ app_name="",+ note="",+ ui_text="",+ trigger="hotkey",+ taken_at=0.0,+ ),+ ttl_s=60,+ )+ try:+ assert client.get("/api/appshot/pending").json()["appshot"]["id"] == "c1"+ first = client.post("/api/appshot/pending/claim", json={"id": "c1"})+ assert first.status_code == 200 and first.content == b"jpeg"+ assert first.headers["cache-control"] == "no-store"+ assert client.post("/api/appshot/pending/claim", json={"id": "c1"}).status_code == 404+ finally:+ store.clear()