Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

chore(release): capture pending local desktop updates

rubenluetke10-beep4880f4ebb21a
diff --git a/README.md b/README.md
index d54ed5277c..682d5fda46 100644
--- a/README.md
+++ b/README.md
@@ -1,21 +1,88 @@
-<p align="center">
- <img src="https://github.com/PersonalJarvis/PersonalJarvis/raw/main/assets/brand/banner.png" alt="Personal Jarvis" width="860" />
-</p>
+# Personal Jarvis
+
+## Your computer becomes an AI agent.
+
+Personal Jarvis is an open-source desktop app that connects your computer, AI
+models, tools, services, and specialized agents. Give it a task in chat or by
+voice. Jarvis can use the browser and desktop, call connected tools, delegate
+longer work, and bring the result back to one workspace.
<p align="center">
- <a href="https://github.com/PersonalJarvis/PersonalJarvis/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-666666?labelColor=333333" alt="Apache 2.0 license" /></a>
- <a href="https://github.com/PersonalJarvis/PersonalJarvis/releases/latest"><img src="https://img.shields.io/github/v/release/PersonalJarvis/PersonalJarvis?label=release&labelColor=333333&color=666666" alt="latest release" /></a>
- <a href="https://github.com/PersonalJarvis/PersonalJarvis/stargazers"><img src="https://img.shields.io/github/stars/PersonalJarvis/PersonalJarvis?labelColor=333333&color=666666&logo=github" alt="GitHub stars" /></a>
- <a href="https://x.com/PersonalJarvis"><img src="https://img.shields.io/badge/follow-%40PersonalJarvis-000000?logo=x&logoColor=white" alt="follow @PersonalJarvis on X" /></a>
+ <a href="https://pypi.org/project/personal-jarvis/"><img alt="PyPI" src="https://img.shields.io/pypi/v/personal-jarvis?labelColor=0A0A0A&amp;color=F7F7F4" /></a>
+ <a href="https://github.com/PersonalJarvis/PersonalJarvis/blob/main/LICENSE"><img alt="Apache 2.0" src="https://img.shields.io/badge/License-Apache_2.0-F7F7F4?labelColor=0A0A0A" /></a>
+ <a href="https://github.com/PersonalJarvis/PersonalJarvis/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/PersonalJarvis/PersonalJarvis/actions/workflows/ci.yml/badge.svg" /></a>
</p>
-**Personal Jarvis is an open-source AI assistant that lives on your desktop and gets real work done.**
+![Flow from a user request through Jarvis to the computer, browser, apps, agents, models, and services, then back as a result with run history.](assets/brand/request-to-result.svg)
+
+**Why it is different:** Jarvis coordinates the tools you already have. A request
+can move from voice or chat to [computer use](#computer-use-and-connected-channels),
+[Codex or Claude Code](#coding-workspace), [local or hosted models](#local-models),
+[plugins and MCP](#plugins-skills-and-mcp), or a [background agent](#jarvis-agents).
+[Memory](#memory-and-knowledge), [agent learning](#jarvis-agents), and
+[routines](#scheduled-work-and-workflows) provide context and recurring work.
+You can inspect actions, approvals, and output in the app. What runs depends on
+your setup, permissions, and the tools you connect.
+
+**Try it:** [Install on Windows, macOS, or Linux](#install) ·
+[See the app and demos](#see-it-in-action) ·
+[Read the first-run guide](#your-first-steps-in-the-desktop-app)
+
+## Install
+
+**Windows — PowerShell**
+
+```powershell
+irm https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.ps1 | iex
+```
+
+**macOS and Linux**
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.sh | bash
+```
+
+The installer checks Python 3.11+ and Git, offers to install missing prerequisites
+through the host package manager, installs the applicable desktop components,
+registers the desktop launcher, and opens the app. Language, wake phrase, and provider setup happen in the app.
+OS permissions and hardware capabilities affect voice and desktop control.
+Re-running the installer updates an existing installation.
The idea is simple: one place on your computer where you say what you need, and it happens. You talk to Jarvis or type to it, and it works out whether to just answer, do something on your computer for you, or pass the job to an agent that keeps at it while you get on with your day. You can always see what it's doing, and it asks before it touches anything that matters.
It runs on your own machine with whichever model you like, local ones included, and it's free. There's no account to create, and nothing sends your data anywhere you didn't connect yourself.
-It's built in the open by a very small team, and it still has rough edges. If something breaks for you, please open an issue. That really is how it gets better.
+## See it in action
+
+The [desktop screenshot](assets/screenshots/home-2026-09-09.webp) shows the
+workspace; the [agent demo](assets/demo/readme-2026-09/jarvis-agents-v4.mp4)
+shows a brief, progress, and output. The demos are **illustrative interface
+recreations**, not recordings of a completed live task or timing benchmarks.
+The [demo source and still previews](scripts/readme-video/README.md) explain
+what is shown.
+
+<p align="center">
+ <a href="assets/demo/readme-2026-09/jarvis-agents-v4.mp4">
+ <img src="assets/demo/readme-2026-09/jarvis-agents-v4.png" alt="Illustrative Jarvis Agents workspace showing a brief, agent activity, and a resulting plan" width="850" />
+ </a>
+</p>
+
+## One request, several ways to get it done
+
+| Stage | What Jarvis connects |
+|---|---|
+| **Ask** | Start in chat or Voice Chat. Jarvis Voice separately dictates into other apps. |
+| **Act** | Use the host desktop and browser through Computer Use, or call a connected plugin, skill, CLI, or MCP tool. |
+| **Delegate** | Hand longer work to a background mission, a persistent specialist, or a Codex/Claude Code coding session when configured. Bounded parallel workers can split suitable work. |
+| **Keep context** | Use the local wiki and private agent lessons; schedule routines for recurring tasks. |
+| **Review** | Follow approvals and activity, then open the answer, file, or report in the conversation or Artifacts. |
+
+The app runs on Windows, macOS, and Linux, with headless use for server-capable
+features. Desktop control needs a supported graphical session and OS permission.
+Local models and speech are optional; hosted providers and connected services
+receive the content required for their requests. See [privacy and local data](docs/product/privacy-safety-and-support/privacy-and-local-data.md).
+
+## Start with chat or voice
[Website](https://personaljarvis.ai) · [Docs](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/docs/product/start-here/welcome-to-personal-jarvis.md) · [Getting started](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/docs/product/start-here/first-run-setup.md) · [How it works](#how-it-works) · [Discord](https://discord.gg/x7USduHxbc) · [X](https://x.com/PersonalJarvis)
@@ -37,7 +104,49 @@ irm https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install
curl -fsSL https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.sh | bash
```
-The installer checks for Python 3.11+ and Git, offers to install what is missing, sets up the app and opens it. Run it again to update. [install options →](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/install/README.md)
+For a minimal server installation, use `pip install personal-jarvis` and
+`jarvis serve`. Open the local address reported at startup; the default is
+`http://localhost:47821`. Remote browser microphone access requires HTTPS.
+See the [headless deployment guide](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/docs/headless-vps-deployment.md).
+
+</details>
+
+## Jarvis Agents
+
+Build a team you can return to. Each agent has an identity, a direct conversation,
+standing instructions, and access to the tools you grant it. Pick a connected
+model or supported agent account for the work, and keep the conversations in
+one workspace.
+
+<br />
+
+<p align="center">
+ <a href="https://github.com/PersonalJarvis/PersonalJarvis/blob/main/assets/demo/readme-2026-09/jarvis-agents-v4.mp4">
+ <img src="https://github.com/PersonalJarvis/PersonalJarvis/raw/main/assets/demo/readme-2026-09/jarvis-agents-v4.gif" alt="Agents workspace recreation: send a brief, watch the live thinking trace, then read the streaming reply and completed plan" width="1000" />
+ </a>
+</p>
+
+<p align="center">
+ <sub>An illustrative agent conversation, from brief to draft. The GIF plays once and holds the reply; click to replay the video.</sub>
+</p>
+
+<br />
+
+- **Talk directly to a specialist.** Select an agent from the roster and continue its chat.
+- **Give it a standing brief.** Configure its instructions, model access, and tools.
+- **Set up recurring work.** Per-agent routines expose instructions, scheduling, and execution history.
+- **Inspect its work.** Read messages and tool activity, and open produced files in Artifacts.
+- **Explore the world view.** The workspace also has a visual map of the team.
+
+Persistent agents and isolated coding missions have different lifecycles.
+Coding missions can use worktree isolation and critic review; an ordinary
+agent chat is not a new isolated worktree on every message.
+For suitable tasks, a mission can fan out to a bounded group of child workers.
+Each child has its own result; this is controlled delegation rather than an
+unlimited self-spawning swarm.
+
+[Agent guide](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/docs/product/extend-and-automate/jarvis-agents.md) ·
+[Agent learning](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/docs/agent-society/self-learning.md) · [Routines](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/docs/routines.md)
You need Windows, macOS or Linux, plus one API key from a supported provider or a local model. A microphone helps if you want to talk to it. You don't need a GPU; speech recognition and local models run fine on a normal machine, just slower.
@@ -70,7 +179,12 @@ Start with a [good first issue](https://github.com/PersonalJarvis/PersonalJarvis
If you are an AI agent helping with this repository, read [`AGENTS.md`](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/AGENTS.md) first.
-## thanks
+Panes have call signs so you can address a particular session through Jarvis:
+*"Tell T1 to run the tests"* or *"What is T2 working on?"* Return to the workspace
+to inspect output, respond to a prompt, or take over manually. A terminal becoming
+idle is not proof that its result is correct; inspect its changes and validation.
+You can also send one brief to several coding panes, with a delivery result for
+each pane. This is the coding workspace's parallel-agent path.
If Personal Jarvis is useful to you, a star helps other people find it. Sponsors are listed in [SPONSORS.md](https://github.com/PersonalJarvis/PersonalJarvis/blob/main/SPONSORS.md).
diff --git a/assets/brand/request-to-result.svg b/assets/brand/request-to-result.svg
new file mode 100644
index 0000000000..08ad271fed
--- /dev/null
+++ b/assets/brand/request-to-result.svg
@@ -0,0 +1,46 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 244" role="img" aria-labelledby="title desc">
+ <title id="title">From request to result</title>
+ <desc id="desc">A user asks Jarvis. Jarvis coordinates the computer, browser, apps, agents, models, and services, then returns a result with run history.</desc>
+ <rect width="1200" height="244" rx="18" fill="#0d1117"/>
+ <g font-family="Arial, Helvetica, sans-serif" text-anchor="middle">
+ <rect x="24" y="78" width="142" height="88" rx="12" fill="#19212c" stroke="#718096" stroke-width="2"/>
+ <text x="95" y="114" fill="#f7f7f4" font-size="24" font-weight="700">You</text>
+ <text x="95" y="142" fill="#c0c9d3" font-size="16">task</text>
+
+ <path d="M174 122h47" fill="none" stroke="#eac56b" stroke-width="3"/>
+ <path d="m221 122-12-8v16z" fill="#eac56b"/>
+
+ <rect x="231" y="78" width="174" height="88" rx="12" fill="#302a1d" stroke="#eac56b" stroke-width="2"/>
+ <text x="318" y="114" fill="#fff0bf" font-size="24" font-weight="700">Jarvis</text>
+ <text x="318" y="142" fill="#f0dca4" font-size="16">coordinate + act</text>
+
+ <path d="M413 122h45" fill="none" stroke="#eac56b" stroke-width="3"/>
+ <path d="m458 122-12-8v16z" fill="#eac56b"/>
+
+ <rect x="468" y="30" width="509" height="184" rx="12" fill="#151d27" stroke="#718096" stroke-width="2"/>
+ <text x="722" y="65" fill="#f7f7f4" font-size="19" font-weight="700">Your connected workspace</text>
+ <g fill="#222d3a" stroke="#758397" stroke-width="1.5">
+ <rect x="487" y="83" width="145" height="48" rx="8"/>
+ <rect x="650" y="83" width="145" height="48" rx="8"/>
+ <rect x="813" y="83" width="145" height="48" rx="8"/>
+ <rect x="487" y="147" width="145" height="48" rx="8"/>
+ <rect x="650" y="147" width="145" height="48" rx="8"/>
+ <rect x="813" y="147" width="145" height="48" rx="8"/>
+ </g>
+ <g fill="#f7f7f4" font-size="17" font-weight="600">
+ <text x="560" y="113">Computer</text>
+ <text x="722" y="113">Browser</text>
+ <text x="885" y="113">Apps</text>
+ <text x="560" y="177">Agents</text>
+ <text x="722" y="177">Models</text>
+ <text x="885" y="177">Services</text>
+ </g>
+
+ <path d="M985 122h39" fill="none" stroke="#eac56b" stroke-width="3"/>
+ <path d="m1024 122-12-8v16z" fill="#eac56b"/>
+
+ <rect x="1034" y="78" width="142" height="88" rx="12" fill="#19212c" stroke="#718096" stroke-width="2"/>
+ <text x="1105" y="114" fill="#f7f7f4" font-size="23" font-weight="700">Result</text>
+ <text x="1105" y="142" fill="#c0c9d3" font-size="16">+ run history</text>
+ </g>
+</svg>
diff --git a/assets/brand/social-preview.html b/assets/brand/social-preview.html
index 4021bcc015..ea9f22f135 100644
--- a/assets/brand/social-preview.html
+++ b/assets/brand/social-preview.html
@@ -19,8 +19,8 @@
.glow {
position: absolute; inset: 0;
background:
- radial-gradient(ellipse 90% 55% at 50% -8%, rgba(61, 139, 255, 0.10), transparent 62%),
- radial-gradient(ellipse 120% 40% at 50% 112%, rgba(61, 139, 255, 0.04), transparent 60%);
+ radial-gradient(ellipse 90% 55% at 50% -8%, rgba(255, 214, 10, 0.10), transparent 62%),
+ radial-gradient(ellipse 120% 40% at 50% 112%, rgba(255, 214, 10, 0.04), transparent 60%);
}
.card {
position: relative;
@@ -32,13 +32,13 @@
.mascot-wrap { position: relative; width: 128px; height: 128px; }
.mascot-wrap .ring {
position: absolute; border-radius: 50%;
- border: 1px solid rgba(250, 250, 250, 0.32);
+ border: 1px solid rgba(255, 214, 10, 0.32);
}
.ring.r1 { inset: -14px; }
- .ring.r2 { inset: -30px; border-color: rgba(250, 250, 250, 0.16); }
+ .ring.r2 { inset: -30px; border-color: rgba(255, 214, 10, 0.16); }
.mascot {
width: 128px; height: 128px; object-fit: contain;
- filter: drop-shadow(0 0 26px rgba(61, 139, 255, 0.30));
+ filter: sepia(1) saturate(8) hue-rotate(354deg) drop-shadow(0 0 26px rgba(255, 214, 10, 0.30));
}
.wordmark {
width: 760px; display: block;
@@ -53,7 +53,7 @@
color: #FAFAFA; letter-spacing: -0.01em;
text-align: center;
}
- .tagline .accent { color: #3D8BFF; }
+ .tagline .accent { color: #FFD60A; }
.meta {
font-family: 'JetBrains Mono', monospace;
font-size: 16px; font-weight: 500;
@@ -71,7 +71,7 @@
<img class="mascot" src="../icons/jarvis-gigi-256.png" alt="" />
</div>
<img class="wordmark" src="banner.png" alt="Personal Jarvis" />
- <div class="tagline">Talk to your computer — <span class="accent">and watch it do the work.</span></div>
+ <div class="tagline">Your computer becomes <span class="accent">an AI agent.</span></div>
<div class="meta">OPEN SOURCE · APACHE 2.0 · LINUX · MACOS · WINDOWS</div>
</div>
</body>
diff --git a/assets/brand/social-preview.png b/assets/brand/social-preview.png
index 46503bf69a..53126922d8 100644
Binary files a/assets/brand/social-preview.png and b/assets/brand/social-preview.png differ
diff --git a/docs/agent-society/MASTERPLAN.md b/docs/agent-society/MASTERPLAN.md
index 121868a514..2f1521dbb4 100644
--- a/docs/agent-society/MASTERPLAN.md
+++ b/docs/agent-society/MASTERPLAN.md
@@ -32,7 +32,8 @@ carries the team card (roster + hands, byte-stable, current on the next turn aft
write), the router's decision table has a DELEGATE way ahead of `spawn_worker`, the realtime
session gets a names directive and the turn planner a `SOCIETY` reason, `delegate_to_agent`
picks the fitting agent when no name is given, and a lead-assigned result comes back as a
-notice in the front-page chat, not spoken (2026-09-30; `agent-definition.md` §3.5). **M3
+notice in its originating chat and, for voice requests, a batched update at a conversational
+pause (`agent-definition.md` §3.5). **M3
world** is the maintainer's parallel track. Open in M4: rooms live under the scheduler, the
curator.
diff --git a/docs/agent-society/agent-definition.md b/docs/agent-society/agent-definition.md
index 6ff21e12c3..5d21c83885 100644
--- a/docs/agent-society/agent-definition.md
+++ b/docs/agent-society/agent-definition.md
@@ -313,8 +313,11 @@ Gmail agent stood on the island. The team card (`jarvis/society/lead_card.py`) i
name the roster does not know, "email agent") the lead picks the agent whose focus fits the
task (`SocietyRuntime.pick_agent`, `derive_focus` over the task text) or says that nobody
fits. When a lead-assigned run ends, `SocietyRuntime.report_to_lead` posts a `notice` event
- into the newest front-page chat session (rendered as a muted result line). It is not
- spoken: since 2026-09-30 no background result is read aloud, and no model call phrases one.
+ into the originating front-page chat session (rendered as a muted result line).
+ Explicit voice requests also return through the active voice conversation, with the
+ original request and evidence. Results are batched and wait for a conversational pause;
+ no separate paid summary model is called. See
+ [delegation follow-through](../architecture/delegation-followthrough.md).
- **Words.** The section is "Agents" (`nav.agents`); the Agentic IDE holds "coding terminals"
in every prompt and directive; `navigate` maps "society", "team", "island", "my agents" to
the section and "terminals", "terminal grid" to the IDE.
@@ -327,8 +330,8 @@ Gmail agent stood on the island. The team card (`jarvis/society/lead_card.py`) i
- **User ↔ agent:** the canonical chat (card's right column), `society:<agent_id>` session on
the agent_chat store; the composer's `/` lists the agent's enabled skills, `@` the roster.
- **Jarvis ↔ agent (voice):** the `delegate-to-agent` router tool (M4): ACK < 5 s, work goes
- through the scheduler as `ASSIGN`, completion comes back as a notice in the front-page
- chat (not spoken, 2026-09-30).
+ through the scheduler as `ASSIGN`, completion returns through the floor-aware voice
+ result queue and a chat notice. Jarvis stays available while the assignment runs.
- **Agent ↔ agent:** exactly one tool, `message_agent(target, text, kind)` → one `SAY` /
`QUERY` / `PROPOSE` envelope to ONE teammate; the scheduler wakes the target's chat.
`@name` in a canonical chat by the USER also produces a `SAY` from the user's identity.
diff --git a/docs/architecture/delegation-followthrough.md b/docs/architecture/delegation-followthrough.md
new file mode 100644
index 0000000000..698041b309
--- /dev/null
+++ b/docs/architecture/delegation-followthrough.md
@@ -0,0 +1,110 @@
+# Delegation follow-through
+
+## What was present
+
+Jarvis already dispatched Society assignments without waiting for their model
+turns, retained their request IDs, watched turn completion, and recorded results
+on the board. Internal questions had parent-message correlation and explicit
+reply policies. Coding panes had submission receipts, a process-aware activity
+watcher, structured transcript adapters and notifications. The speech pipeline
+already supported deferred announcements, live-model report synthesis and
+speaker-drain confirmation.
+
+The missing connection was result delivery: Society and coding-pane speech had
+been removed, while dispatch acknowledgements could still promise a report.
+Society notices targeted the newest front-page chat, rather than the requesting
+chat. Those notices were not included when rebuilding model conversation history.
+Re-enabling the old implementation would also have restored a separate paid
+summary composer and inherited weak freshness checks for terminal answers.
+
+## Resulting behavior
+
+- Explicit Jarvis voice delegations return through the existing live voice
+ conversation. The original task, source, reported status and evidence accompany
+ every result. The active model summarizes these in its own voice. Classic TTS
+ has a deterministic short fallback. There is no additional summary model call.
+- Text-chat delegations return to their originating conversation and remain in
+ its subsequent model context. They do not speak into an unrelated voice call.
+ A deleted origin is never redirected into a newer chat.
+- A 750 ms coalescing window groups up to four results. Longer lists drain in
+ subsequent batches. Each result keeps a separate context budget, including its
+ beginning and ending. The queue never waits for unfinished sibling tasks.
+- Delivery waits for LISTENING, including a second check after the coalescing
+ window. User speech, foreground processing, existing speech, mute and hangup
+ retain results instead of interrupting. Only one live readback can be in flight.
+ Speaker completion confirms delivery; interrupted readbacks retain the existing
+ retry-on-next-call behavior. Reports already enter the active conversation's
+ context while awaiting speech.
+- Direct agent chat, agent-to-agent traffic, manual terminal input and work
+ owned by a Society coding supervisor retain their own reporting owner. Reply
+ policies (`always`, `on_error`, `none`) remain authoritative.
+
+## Completion and uncertainty
+
+Society completion is a typed lifecycle event, not the dispatch receipt. A
+correlated question or proposal reaches the user as needing input; the following
+turn end records a blocker rather than manufacturing success or a duplicate report.
+
+Coding receipts belong to one submission and process generation. A replacement
+process or subsequent manual prompt invalidates them. A recorded answer must be
+new relative to the pre-submission transcript and answer the exact submitted
+prompt. An old answer to an identical prior prompt is insufficient. Short jobs
+that finish between activity sweeps are recovered from fresh recorded evidence.
+Remote panes never read their stale local transcript as a current result.
+
+A settled terminal is not independent proof of task success. When the actual
+answer cannot be established, the report explicitly identifies that uncertainty.
+Questions, process exit and startup failure have separate outcomes. No result
+automatically grants permission or starts another tool operation.
+
+## Implementation boundaries
+
+- `core/delegation.py`: trusted origin capture, evidence envelopes and bounded
+ batching/deduplication; no provider or boot-time initialization.
+- `society/runtime.py`: request-correlated answers and completion reports;
+ existing scheduler and board remain the execution owners.
+- `agentic_ide/followthrough.py`: submission ownership and transcript freshness;
+ existing notification sweeps supply activity evidence independently of the bell.
+- `speech/pipeline.py` and `realtime/report_prompt.py`: conversational scheduling
+ and synthesis through the selected live model.
+- `agent_chat/runner_api.py` and the server's result bridge: existing persisted
+ notice events carry results back into text-chat history.
+
+No REST route, provider, permission tier, model default or frontend appearance
+changes. Result tracking does not hold a speech turn open and does not poll a model.
+
+## Verification and limits
+
+Focused tests cover batching, deduplication, user/assistant floor ownership,
+mute/hangup, arrivals during playback, prompt freshness, superseded work,
+questions, failed submission and text-chat origin. Integration tests exercise
+the real Society scheduler, SQLite board, event bus and speech inbox with fake
+providers, including two concurrent assignments and question-before-completion.
+Existing provider readback and speech lifecycle tests exercise the shared adapter
+contract. All tests are run without paid provider probes or browser processes.
+
+The terminal tracker and voice queue are process-local. Society board results and
+chat notices persist, but automatic spoken replay after an application restart
+is not promised. Terminal detection retains the existing activity watcher's
+settle latency and uncertainty for unreadable or remote transcripts. Text notices
+use deterministic excerpts; live voice performs contextual synthesis. Real-device
+voice quality and non-Windows runtime behavior require separate live validation.
+
+The broad Windows regression runs passed 995 and 698 tests respectively (one
+skip in the first run). Two failures were reproduced individually on the exact
+unchanged base commit `ab09f5b13`, with the same assertions and causes:
+
+- `test_routine_approval_delegation_is_only_for_society`: the existing routine
+ permission expectation no longer matches the base implementation.
+- `test_jarvis_chat_receives_controller_without_becoming_a_coding_cli`: its fake
+ session lacks the provider field required by the existing browser tool builder.
+
+Neither failure was hidden or added to a baseline. The final focused pane and
+integration rerun passed 68 tests; changed Python files passed Ruff and the
+documentation passed the privacy scan. The four required routing, output-filter,
+hangup-parity and turn-language guards are included in the second broad run.
+
+After integrating with GitHub main `2971510a5`, 102 focused lifecycle, Society,
+workspace, speech and integration checks passed again. The existing guard against
+claiming success from an empty agent report was preserved during integration.
+CLI route coverage passed; the terminal-prompt route keeps its public schema.
diff --git a/docs/commands-reference.md b/docs/commands-reference.md
index be977f6b29..ed1730c2cc 100644
--- a/docs/commands-reference.md
+++ b/docs/commands-reference.md
@@ -13,10 +13,10 @@ Commands marked **requires confirmation** never run on a bare voice request —
## `society-create-agent` — Create a persistent team agent
-Create an agent in the user's existing Agents team, only when requested. This creates a roster profile and starts no work. Responsibilities derive capabilities through the same service as the Agents UI. When a team agent creates a teammate, the new agent inherits that creator's model seat (provider, model, effort, subscription account) and permission setup (ceiling, grant mode, grants, denies) and cannot exceed them. Read the returned agent and created flag: an existing name is adopted, never duplicated. Use society-switch-agent-model only if a different provider is requested. Design the agent before calling: turn the user's words into a complete brief — mission, 3-6 responsibilities, working rules naming the data sources, output format and boundaries — inferring sensible defaults instead of copying the request verbatim. Read readback.not_connected and tell the user which services to connect. A requested schedule is a separate society-create-routine call.
+Create an agent in the user's existing Agents team, only when requested. This creates a roster profile and starts no work. Responsibilities derive capabilities through the same service as the Agents UI. When a team agent creates a teammate, the new agent inherits that creator's model seat (provider, model, effort, subscription account) and permission setup (approval mode, ceiling, grant mode, grants, denies) and cannot exceed them. Read the returned agent and created flag: an existing name is adopted, never duplicated. Use society-switch-agent-model only if a different provider is requested. Design the agent before calling: turn the user's words into a complete brief — mission, 3-6 responsibilities, working rules naming the data sources, output format and boundaries — inferring sensible defaults instead of copying the request verbatim. Read readback.not_connected and tell the user which services to connect. A requested schedule is a separate society-create-routine call.
- **Endpoint:** `POST /api/society/agents`
-- **Arguments:** `name` (string; required); `title` (string; optional); `description` (string; optional); `mission` (string; optional); `responsibilities` (array; optional); `working_rules` (array; optional); `output_format` (string; optional); `boundaries` (array; optional); `success_criteria` (string; optional); `tier` (one of: specialist, orchestrator; optional); `focus` (array; optional); `skills` (array; optional); `daily_budget_usd` (number; optional); `max_concurrent_runs` (integer; optional); `permission_ceiling` (one of: safe, monitor, ask; optional); `grant_mode` (one of: all, allowlist; optional); `grants` (array; optional); `denies` (array; optional)
+- **Arguments:** `name` (string; required); `title` (string; optional); `description` (string; optional); `mission` (string; optional); `responsibilities` (array; optional); `working_rules` (array; optional); `output_format` (string; optional); `boundaries` (array; optional); `success_criteria` (string; optional); `tier` (one of: specialist, orchestrator; optional); `focus` (array; optional); `skills` (array; optional); `approval_mode` (one of: bypass, ask, always_ask; optional); `daily_budget_usd` (number; optional); `max_concurrent_runs` (integer; optional); `permission_ceiling` (one of: safe, monitor, ask; optional); `grant_mode` (one of: all, allowlist; optional); `grants` (array; optional); `denies` (array; optional)
- **Requires confirmation:** no
- **Desktop UI section:** `agents`
- **Voice example (EN):** "create an agent for researching suppliers"
@@ -26,7 +26,7 @@ Create an agent in the user's existing Agents team, only when requested. This cr
Change an existing agent's responsibilities, skills, focus, budget or paused/active state as requested by the user. Resolve its real id with society_status first. Only send requested fields. Existing learned focus and approval rules survive prose edits. Report the returned stored agent.
- **Endpoint:** `PATCH /api/society/agents/{agent_id}`
-- **Arguments:** `agent_id` (string; required); `title` (string; optional); `description` (string; optional); `mission` (string; optional); `responsibilities` (array; optional); `working_rules` (array; optional); `output_format` (string; optional); `boundaries` (array; optional); `success_criteria` (string; optional); `tier` (one of: specialist, orchestrator; optional); `focus` (array; optional); `skills` (array; optional); `daily_budget_usd` (number; optional); `max_concurrent_runs` (integer; optional); `permission_ceiling` (one of: safe, monitor, ask; optional); `grant_mode` (one of: all, allowlist; optional); `grants` (array; optional); `denies` (array; optional); `state` (one of: active, paused; optional)
+- **Arguments:** `agent_id` (string; required); `title` (string; optional); `description` (string; optional); `mission` (string; optional); `responsibilities` (array; optional); `working_rules` (array; optional); `output_format` (string; optional); `boundaries` (array; optional); `success_criteria` (string; optional); `tier` (one of: specialist, orchestrator; optional); `focus` (array; optional); `skills` (array; optional); `approval_mode` (one of: bypass, ask, always_ask; optional); `daily_budget_usd` (number; optional); `max_concurrent_runs` (integer; optional); `permission_ceiling` (one of: safe, monitor, ask; optional); `grant_mode` (one of: all, allowlist; optional); `grants` (array; optional); `denies` (array; optional); `state` (one of: active, paused; optional)
- **Requires confirmation:** no
- **Desktop UI section:** `agents`
- **Voice example (EN):** "update Scout's responsibilities"
diff --git a/docs/os-parity.md b/docs/os-parity.md
index 6e26c953be..dc8498e318 100644
--- a/docs/os-parity.md
+++ b/docs/os-parity.md
@@ -1,5 +1,46 @@
# OS Feature Parity — macOS / Linux Gap Register
+## Remote agents retain Jarvis tools (2026-10-01, T3)
+
+Remote coding turns carry their Jarvis MCP tools over a reverse forward on
+the existing pinned SSH connection. Both listeners bind to loopback. The
+remote credential belongs to one active chat turn; the Control API key stays
+on the main computer. Session headers cannot change the owner, and stale
+requests cannot acquire the next turn's user context. The existing session
+catalog and ToolExecutor still enforce grants, approvals and cancellation.
+Routine execution chats retain their owner's computer placement.
+
+The transport uses portable AsyncSSH and asyncio/uvicorn, with no native OS
+imports or boot-time initialization. Windows remote launchers and POSIX argv
+receive the remote endpoint through their existing launch paths. Claude,
+Codex, Antigravity, Grok, OpenCode, Kimi and Cursor have MCP configuration
+adapters; other runners fail before starting model work. Project-discovered
+configurations are restored after the turn and cannot overlap in one
+workspace. Native files and society_shell remain remote; Jarvis's browser,
+memory and file-tool workspace remain on the main computer.
+
+Verification: real in-process SSH forwarding tests cover authentication,
+session isolation, expired and revoked credentials, connection refusal,
+cleanup, configuration adapters, and routine creation/readback through MCP
+with a real SQLite store and scheduler. Reopening the store preserves the
+calendar schedule. Existing Windows launcher tests cover the launch path.
+Native macOS/Linux hosts and all seven real vendor CLIs are not qualified
+by those tests; SSH servers must permit TCP forwarding. No browser is
+started by this transport.
+
+Live qualification on Windows: a resumed Claude subscription chat on a
+connected Windows computer discovered the Jarvis tools, created a temporary
+daily 09:00 Europe/Berlin routine, read back its persisted ID and next run,
+deleted it and verified the empty routine list. The final CLI configuration
+uses an explicit HTTP transport type; omitting it hid the server from the
+actual Claude tool catalog despite successful protocol-level tests. Browser
+actions were not executed in this qualification.
+
+Protocol references: [AsyncSSH port forwarding](https://asyncssh.readthedocs.io/en/latest/),
+[OpenCode MCP](https://opencode.ai/docs/mcp-servers/),
+[Kimi MCP configuration](https://github.com/MoonshotAI/kimi-cli/blob/main/README.md),
+[Cursor MCP](https://docs.cursor.com/en/cli/mcp).
+
## Connected computers on Linux, macOS and Windows (2026-10-01, T3)
This one is about the REMOTE side: the machine Jarvis connects to under
@@ -621,7 +662,7 @@ experiences today.
|---|---|---|---|---|---|
| P-29 | Low | Subscription voice | The dedicated ChatGPT-subscription voice login is an interactive browser flow, so a headless Linux host — and a graphical Linux desktop that ships no terminal emulator able to host the login for its full lifetime — can never CONNECT the profile there (an existing login still reports ready and calls work through the browser voice bridge) | `jarvis/codex_app_server.py::_login_required_state`, `_linux_login_terminal_missing`, `start_codex_subscription_login`, `jarvis/codex_auth.py::_LINUX_LOGIN_TERMINALS` | Both cases report the same `lifecycle_unavailable` truth on every surface (card, activation, voice-mode, Test), each with its own actionable reason — "run Jarvis on a desktop" or "install one of these terminals" — and never an enabled Connect button that can only produce an error toast |
| P-24 | Medium | Dictation shortcut | The global dictation/call shortcut needs `pynput` on Linux/X11, and `pynput` hard-requires `evdev` — which is published **source-only** (verified on PyPI 2026-07-28: evdev 1.9.3 ships an sdist and no wheels) and compiles against the kernel headers. Putting it in `[full]` would break the one advertised install path on a stock `python:3.11-slim`, so it is the opt-in `[desktop-linux]` extra instead. Wayland is a separate, unfixable-by-install case: the compositor owns global shortcuts by design (the XDG `GlobalShortcuts` portal lets the *compositor* assign the keys, and no wlroots compositor implements it at all) | `pyproject.toml` (`desktop-linux`), `jarvis/platform/probes.py::has_hotkey`, `jarvis/trigger/backends/noop.py::explain_unavailable` | X11 without the extra: no global shortcut, and the log/UI now names the actual cause and the exact `pip install` that fixes it (it used to blame Wayland unconditionally). Wayland: no global shortcut at all — bind a compositor shortcut to `jarvis api dictation start`. On both, dictation still works from the Jarvis Bar, the Dictation view and the CLI, and voice still works via the wake word |
-| P-25 | Medium | Dictation insertion | Pasting the transcript into another application is blocked, silently, in three OS-specific situations: Windows UIPI when the foreground window is elevated and Jarvis is not (`SendInput` reports success and the input is discarded), macOS Secure Input while a password field is focused, and Wayland outright (no synthetic input). Detection exists for the first two; Wayland is refused up front. Two further silent failures are Windows-only in their FIX: a chord the target does not bind as "paste" (an xterm.js terminal in a Tauri/Electron app swallows Ctrl+V as `^V`), and a target that reads the clipboard late (an async WebView bridge on a busy machine) after the 120 ms restore timer had already put the previous clipboard back | `jarvis/dictation/insert.py::describe_target`, `_insert_windows_verified`, `jarvis/platform/clipboard_offer.py`, `jarvis/platform/input_isolation.py::windows_foreground_window_is_elevated`, `macos_secure_input_enabled` | All three blocks degrade to the SAME honest outcome instead of silence: the transcript is left on the clipboard, the result is reported as `clipboard_only`, and the bar plus the Dictation view say why and that Ctrl+V will paste it. **Windows** additionally offers the text with delayed rendering and watches who reads it: on a host without a clipboard watcher (no Remote Desktop client, clipboard history off — the default) a paste is proven by the target's read, silence cascades Ctrl+V → Ctrl+Shift+V → Shift+Insert → typing (line breaks as Shift+Enter), and the route is remembered per executable; on a host with a watcher the offer is blind, ONE chord goes out (never a guessed second paste) and the previous clipboard is restored after a 2 s grace only if the dictated text is still on it. **macOS / Linux X11**: plain chord + 120 ms timer restore, unchanged — no delayed-rendering equivalent exists there (NSPasteboard promises and X11 selections notify the owner too, but are a follow-up). macOS Secure Input detection is implemented but has not been verified on real hardware from this machine |
+| P-25 | Medium | Dictation insertion | Pasting the transcript into another application is blocked, silently, in three OS-specific situations: Windows UIPI when the foreground window is elevated and Jarvis is not (`SendInput` reports success and the input is discarded), macOS Secure Input while a password field is focused, and Wayland outright (no synthetic input). Detection exists for the first two; Wayland is refused up front. Two further silent failures are Windows-only in their FIX: a chord the target does not bind as "paste" (an xterm.js terminal in a Tauri/Electron app swallows Ctrl+V as `^V`), and a target that reads the clipboard late (an async WebView bridge on a busy machine) after the 120 ms restore timer had already put the previous clipboard back | `jarvis/dictation/insert.py::describe_target`, `_insert_windows_verified`, `jarvis/platform/clipboard_offer.py`, `jarvis/platform/input_isolation.py::windows_foreground_window_is_elevated`, `macos_secure_input_enabled` | All three blocks degrade to the SAME honest outcome instead of silence: the transcript is left on the clipboard, the result is reported as `clipboard_only`, and the bar plus the Dictation view say why and that Ctrl+V will paste it. **Windows** offers the text with delayed rendering and sends exactly one configured paste shortcut. Only a render requested by the captured target process confirms clipboard consumption; merely opening the clipboard or a watcher reading it is not confirmation. A missing acknowledgement never triggers another shortcut, character typing, or an executable-wide learned route. Unconfirmed delivery reports `paste_sent` and leaves the transcript on the clipboard for a late or manual paste, without a restore timer. Focus changes or held modifiers before key emission leave the text available as `clipboard_only`. Confirmed reads may restore the previous clipboard only while the offer still owns it. The explicit typing setting remains available. Regression tests cover delayed readers, watchers, focus changes, and offer shutdown; these checks do not establish compatibility with every third-party editor. **macOS / Linux X11**: plain chord + 120 ms timer restore, unchanged — no delayed-rendering equivalent exists there (NSPasteboard promises and X11 selections notify the owner too, but are a follow-up). macOS Secure Input detection is implemented but has not been verified on real hardware from this machine |
| P-04 | Medium | CU typing | Linux desktop Unicode text input needs the system `xdotool` binary (pip cannot install it); the pyautogui fallback used on Linux drops non-ASCII chars (umlauts, CJK, emoji) without it | `jarvis/cu/actuate/posix.py::type_text`, `jarvis/plugins/tool/type_text.py` | With `xdotool` (installer provisions it since 2026-07-15): fine. Without, the drop is now reported HONESTLY (2026-07-23): an all-non-ASCII text fails with an actionable "install xdotool" error, and a mixed text types its ASCII portion and warns that the rest was dropped — no more silent success |
| P-05 | Low | Wiki | Wiki search hard-fails (RuntimeError with actionable apt/pysqlite3 remediation) on distros whose system SQLite lacks FTS5 | `jarvis/memory/wiki/fts_index.py:279` | `python:3.11-slim` and macOS ship FTS5 — only exotic/old distros affected; message is honest. Decision 2026-07-16: kept as honest hard error — a pysqlite3 shim would rewire seven wiki modules for an exotic audience |
| P-07 | Low | Audio | No macOS/Linux host-API preference exists (the Windows-name-driven tables are intentionally inert off Windows — documented in-code since 2026-07-16), and headset-name heuristics are Windows-centric | `jarvis/audio/player.py`, `jarvis/audio/capture.py` | Device auto-pick falls back to OS default order — works, less clever than on Windows |
diff --git a/docs/product/start-here/welcome-to-personal-jarvis.md b/docs/product/start-here/welcome-to-personal-jarvis.md
index aee68b4949..667ae54d64 100644
--- a/docs/product/start-here/welcome-to-personal-jarvis.md
+++ b/docs/product/start-here/welcome-to-personal-jarvis.md
@@ -1,7 +1,7 @@
---
title: "Welcome to Personal Jarvis"
slug: welcome-to-personal-jarvis
-summary: "Understand what Personal Jarvis can do, what stays under your control, and where to begin."
+summary: "See how Jarvis turns your computer into an AI agent, what stays under your control, and where to begin."
section: "Start here"
section_order: 1
order: 1
@@ -15,13 +15,13 @@ tags: [overview, getting-started, privacy, safety]
related: [install-personal-jarvis, desktop-app-tour, start-your-first-chat]
---
-Personal Jarvis is an open-source assistant you can use in its desktop window
-or through its browser interface. You choose how it reasons, which services it
-can reach, and which device permissions it receives.
+Personal Jarvis turns your computer into an AI agent. Give it a task in chat or
+by voice. It can use the desktop and browser on its host, call connected tools,
+delegate longer work to specialized agents, and return results in one workspace.
+You choose the models, services, and device permissions it can use.
-You can begin with one working chat provider or a local model. Voice, connected
-services, long-running Agents, knowledge, and computer control are optional
-capabilities you can add when you need them.
+Start with one working chat provider or local model. Add computer control,
+voice, connected services, agents, and recurring work as you need them.
Several visible labels follow the assistant name you chose, including the
Agents area and Voice hub. This guide uses the neutral words **assistant**,
diff --git a/jarvis/__main__.py b/jarvis/__main__.py
index 67c4623ae3..1b9553111a 100644
--- a/jarvis/__main__.py
+++ b/jarvis/__main__.py
@@ -52,7 +52,7 @@ if TYPE_CHECKING:
def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="jarvis",
- description="Personal Jarvis — voice-driven meta-orchestrator.",
+ description="Personal Jarvis — turn your computer into an AI agent.",
)
parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
parser.add_argument("--wizard", action="store_true", help="Restart the setup wizard.")
diff --git a/jarvis/agent_chat/approval_bridge.py b/jarvis/agent_chat/approval_bridge.py
index 0a602cc79d..a4376f778d 100644
--- a/jarvis/agent_chat/approval_bridge.py
+++ b/jarvis/agent_chat/approval_bridge.py
@@ -90,6 +90,8 @@ class ChatGrant:
#: this turn (Claude Code's ``can_use_tool``): the executor's gate for the
#: same tool is answered without a second card.
pre_approved: set[str] = field(default_factory=set)
+ #: Society's explicit require-approval rules override Bypass.
+ force_ask: Callable[[str, dict[str, Any]], bool] = field(default=lambda _name, _args: False)
class ChatApprovalBridge:
@@ -138,8 +140,10 @@ class ChatApprovalBridge:
grant = self._grants.get(ref) if ref else None
if grant is None:
return
- decision = self._decide_inline(grant, event.tool_name)
+ args = self._args.get(event.trace_id) or {}
+ decision = self._decide_inline(grant, event.tool_name, args)
if decision is not None:
+ self._args.pop(event.trace_id, None)
await self._bus.publish(
ActionApproved(
trace_id=event.trace_id,
@@ -160,10 +164,16 @@ class ChatApprovalBridge:
# ------------------------------------------------------------ policy
@staticmethod
- def _decide_inline(grant: ChatGrant, tool_name: str) -> str | None:
+ def _decide_inline(
+ grant: ChatGrant, tool_name: str, args: dict[str, Any] | None = None
+ ) -> str | None:
"""The ``approved_by`` label when the stance decides without a card, else None."""
name = _bare(tool_name)
+ if grant.stance == "always_ask":
+ return None
if grant.stance == "bypass":
+ if grant.force_ask(name, args or {}):
+ return None
return "chat-bypass"
if name in grant.always_allowed:
return "user"
diff --git a/jarvis/agent_chat/jarvis_harness.py b/jarvis/agent_chat/jarvis_harness.py
index 730df35231..ffe328e1fc 100644
--- a/jarvis/agent_chat/jarvis_harness.py
+++ b/jarvis/agent_chat/jarvis_harness.py
@@ -199,16 +199,14 @@ def codex_config_args(session_id: str | None = None) -> list[str]:
f'mcp_servers.{_SERVER_NAME}.tools.society_browser.approval_mode="approve"',
]
if session_id.startswith("society:"):
- # A society seat cannot carry out its role without its owned
- # tools. Surface startup failures instead of a tools-free chat
- # that can only promise to configure the running app.
- # This local configuration tool already verifies the current
- # user request and routes permission changes to Jarvis' own
- # approval card. A second Codex write prompt cannot be answered
- # by `exec` and would reject even explicit routine requests.
+ # Codex exec has no interactive approval channel. Every tool on
+ # this server delegates to Jarvis' session catalog and
+ # ToolExecutor, which owns the actual policy and chat card.
+ # Leaving even one tool at Codex's prompt default causes an
+ # unanswerable "approval policy is never" failure.
args += [
"-c",
- f'mcp_servers.{_SERVER_NAME}.tools.society_propose_change.approval_mode="approve"',
+ f'mcp_servers.{_SERVER_NAME}.default_tools_approval_mode="approve"',
]
return args
except Exception: # noqa: BLE001 — see mcp_config_json
@@ -529,9 +527,7 @@ def society_memory_refresh(text: str, *, compact: bool = False) -> str:
return (
"<current_agent_memory>\n"
"Current standing instructions, memory and learned skill index replace their previous "
- "snapshots when complete. "
- + note
- + "Learned guidance never grants "
+ "snapshots when complete. " + note + "Learned guidance never grants "
"permissions or overrides the standing instructions or the current user request.\n\n"
+ body
+ "\n</current_agent_memory>\n\n"
diff --git a/jarvis/agent_chat/permissions.py b/jarvis/agent_chat/permissions.py
index ca7d0a58bc..dd6a432a46 100644
--- a/jarvis/agent_chat/permissions.py
+++ b/jarvis/agent_chat/permissions.py
@@ -283,6 +283,25 @@ _JARVIS: Final[tuple[PermissionMode, ...]] = (
),
)
+_SOCIETY: Final[tuple[PermissionMode, ...]] = (
+ PermissionMode(
+ "bypass",
+ "Bypass permissions",
+ "Run Jarvis tools without prompts. Jarvis blocks, denied capabilities and "
+ "credential boundaries still apply; vendor-native CLI tools use that runner's controls.",
+ ),
+ PermissionMode(
+ "ask",
+ "Ask",
+ "Read freely; show a decision in this chat before an action or write.",
+ ),
+ PermissionMode(
+ "always_ask",
+ "Always ask",
+ "Show a decision in this chat before every Jarvis tool call, including reads.",
+ ),
+)
+
#: The ladder key of the Jarvis surface — not a runner, but the one word the
#: front page's composer shows whichever runner answers.
JARVIS_LADDER: Final[str] = "jarvis"
@@ -302,6 +321,7 @@ _LADDERS: Final[dict[str, tuple[tuple[PermissionMode, ...], str]]] = {
"cursor-cli": (_CURSOR, "auto"),
"api": (_API, "ask"),
JARVIS_LADDER: (_JARVIS, "ask"),
+ "society": (_SOCIETY, "bypass"),
}
# The universal stance ordering: every ladder is a sub-sequence of this once
@@ -323,6 +343,7 @@ _STANCE: Final[dict[str, int]] = {
"bypassPermissions": 3,
"skip-permissions": 3,
"bypass": 3,
+ "always_ask": 1,
}
@@ -367,6 +388,17 @@ def is_permission_mode(runner: str, mode: str) -> bool:
return mode in permission_ids(runner)
+def society_mode_supported(runner: str, mode: str) -> bool:
+ """Whether this transport can honor a Society mode without hidden prompts."""
+ if mode == "bypass":
+ return True
+ if mode == "ask":
+ return runner in {"brain", "codex-cli", "claude-cli", "glm-cli"}
+ if mode == "always_ask":
+ return runner in {"brain", "codex-cli"}
+ return False
+
+
def normalize_permission(runner: str, mode: str | None) -> str:
"""Fold ``mode`` onto the closest mode ``runner`` accepts.
@@ -407,5 +439,6 @@ __all__ = [
"normalize_permission",
"permission_ids",
"permission_modes",
+ "society_mode_supported",
"stance_of",
]
diff --git a/jarvis/agent_chat/remote_cli.py b/jarvis/agent_chat/remote_cli.py
index 4170d887a6..b53929bbc3 100644
--- a/jarvis/agent_chat/remote_cli.py
+++ b/jarvis/agent_chat/remote_cli.py
@@ -14,9 +14,9 @@ What does NOT travel:
* Local paths and secrets. The environment is an allow-list of a few
prompt-suppressing variables; the CLI uses the login it has on the remote
machine (``claude login`` / ``codex login`` there), never this box's.
-* Jarvis' own MCP tools. That endpoint listens on this computer's localhost
- and a remote CLI cannot reach it, so ``--mcp-config`` / the Codex
- ``mcp_servers`` overrides are dropped for remote turns.
+Jarvis tools travel through a turn-scoped reverse SSH forward. Local MCP
+configuration is replaced with a remote loopback endpoint; the main computer's
+control credential never leaves this process.
On every OS the turn is started by a small launcher uploaded first, with the
system prompt as a file next to it (``jarvis.computers.remote_os``): the SSH
@@ -189,6 +189,7 @@ class RemoteCliProcess:
stack: contextlib.AsyncExitStack,
*,
stop: Callable[[], Awaitable[None]] | None = None,
+ revoke: Callable[[], None] | None = None,
) -> None:
self._process = process
self._stack = stack
@@ -199,6 +200,20 @@ class RemoteCliProcess:
#: Ends the remote process tree on a computer where hanging up does not.
self._stop = stop
self._stopping: asyncio.Task[None] | None = None
+ self._revoke = revoke
+ self._reaper = (
+ asyncio.create_task(self._reap(), name="remote-tools-cleanup") if revoke else None
+ )
+
+ async def _reap(self) -> None:
+ try:
+ await self._process.wait_closed()
+ except Exception: # noqa: BLE001 — a disconnect still revokes this turn
+ log.debug("remote tools: process channel closed with an error", exc_info=True)
+ finally:
+ assert self._revoke is not None
+ self._revoke()
+ await self._release()
@property
def returncode(self) -> int | None:
@@ -212,10 +227,14 @@ class RemoteCliProcess:
await asyncio.shield(self._stopping)
with contextlib.suppress(Exception): # a torn-down channel reports no status
await self._process.wait_closed()
+ if self._reaper is not None:
+ await asyncio.shield(self._reaper)
await self._release()
return self.returncode
def kill(self) -> None:
+ if self._revoke is not None:
+ self._revoke()
if self._stop is not None:
if self._stopping is None:
self._stopping = asyncio.get_running_loop().create_task(
@@ -257,7 +276,7 @@ class RemoteCliProcess:
await self._stack.aclose()
-def placement_note(computer: Any, remote_cwd: str) -> str:
+def placement_note(computer: Any, remote_cwd: str, *, tools_connected: bool = False) -> str:
"""The system-prompt section that tells a remote CLI where it runs.
The briefing it rides on was written for this computer; without the note
@@ -278,8 +297,16 @@ def placement_note(computer: Any, remote_cwd: str) -> str:
"\n\n## Where you run\n\n"
f"You run on the connected computer {where}, reached over SSH from the "
f"person's main computer — not on the main computer itself. Your working "
- f"folder there is {remote_cwd}. Jarvis' own tools (the jarvis MCP tools) "
- "are not available on that computer; work with the files and programs there.\n"
+ f"folder there is {remote_cwd}. "
+ + (
+ "Jarvis tools are connected over SSH under your own chat identity. "
+ "Use society_routines and society_propose_change for persistent routines; "
+ "use society_browser for the visible Jarvis browser on the main computer. "
+ "Native file tools run here. Use society_shell for remote commands. "
+ "Jarvis file tools, memory and app services remain on the main computer.\n"
+ if tools_connected else
+ "Jarvis tools are not connected for this turn; work with the files and programs here.\n"
+ )
)
@@ -293,6 +320,7 @@ async def _launch(
remote_cwd: str,
env: dict[str, str],
prompts: dict[str, str],
+ tool_config: Any = None,
) -> str:
"""Upload the system prompt and a launcher; the command line that runs it.
@@ -315,9 +343,13 @@ async def _launch(
remote_cwd=remote_cwd,
uploaded_prompt_files=uploaded,
)
+ command_env = remote_env(env)
+ if tool_config is not None:
+ command_argv = tool_config.apply(command_argv)
+ command_env.update(tool_config.env)
launcher = f"{remote_os.LAUNCH_DIR}/{remote_os.launcher_name(agent_id)}"
script = remote_os.launcher_script(
- host, remote_cwd, command_argv, remote_env(env), pid_file=_pid_file(agent_id)
+ host, remote_cwd, command_argv, command_env, pid_file=_pid_file(agent_id)
)
await remote_os.upload_text(opened, host, launcher, script, private=True)
return host.launcher_command(launcher)
@@ -339,6 +371,8 @@ async def spawn(
local_cwd: str,
env: dict[str, str],
system_prompt_files: dict[str, str] | None = None,
+ session_id: str = "",
+ tools_enabled: bool = True,
) -> RemoteCliProcess:
"""Start the CLI on ``computer_id`` inside the agent's remote workspace."""
from jarvis.computers import remote_os
@@ -346,8 +380,17 @@ async def spawn(
from jarvis.computers.ssh import SshError
from jarvis.society.remote import remote_workspace_expr
+ from .remote_mcp import RemoteMcpBridge, RemoteToolsUnavailable
+ from .remote_mcp_config import SUPPORTED_RUNNERS, configure
+
stack = contextlib.AsyncExitStack()
+ bridge = None
try:
+ if session_id and tools_enabled and runner not in SUPPORTED_RUNNERS:
+ raise RemoteCliUnavailable(
+ "This coding CLI cannot connect to Jarvis tools remotely. "
+ "Choose a CLI with MCP support in the agent's model settings."
+ )
opened = await stack.enter_async_context(get_service().session(computer_id))
computer = get_service().get(computer_id)
host = await remote_os.remote_host(computer_id, opened)
@@ -377,13 +420,27 @@ async def spawn(
"Ready for coding agents) and log in once, or set the agent to run on "
"this computer."
)
- note = placement_note(computer, remote_cwd)
+ tool_config = None
+ if session_id and tools_enabled:
+ bridge = await RemoteMcpBridge.open(opened.conn, session_id)
+ stack.push_async_callback(bridge.aclose)
+ tool_config = await configure(
+ opened, host, stack, runner=runner, cwd=remote_cwd,
+ url=bridge.url, token=bridge.app.token,
+ )
+ # Revocation precedes potentially slow SFTP cleanup on disconnect.
+ stack.callback(bridge.app.revoke)
+ note = placement_note(computer, remote_cwd, tools_connected=tool_config is not None)
prompts = {path: text + note for path, text in (system_prompt_files or {}).items()}
command = await _launch(
- opened, host, agent_id, argv, binary, local_cwd, remote_cwd, env, prompts
+ opened, host, agent_id, argv, binary, local_cwd, remote_cwd, env, prompts,
+ tool_config=tool_config,
)
log.info("agent chat: %s turn for %s runs on %s", runner, agent_id, computer.name)
process = await opened.conn.create_process(command, encoding=None)
+ except RemoteToolsUnavailable as exc:
+ await stack.aclose()
+ raise RemoteCliUnavailable(str(exc)) from exc
except (ComputerError, SshError) as exc:
await stack.aclose()
raise RemoteCliUnavailable(f"Could not reach the agent's computer: {exc.message}") from exc
@@ -397,4 +454,6 @@ async def spawn(
# (measured), and a POSIX CLI may ignore the hang-up until it writes.
await remote_os.run_script(opened, host, stop_command, timeout_s=30)
- return RemoteCliProcess(process, stack, stop=stop)
+ return RemoteCliProcess(
+ process, stack, stop=stop, revoke=bridge.app.revoke if bridge is not None else None,
+ )
diff --git a/jarvis/agent_chat/remote_mcp.py b/jarvis/agent_chat/remote_mcp.py
new file mode 100644
index 0000000000..41b88b06b2
--- /dev/null
+++ b/jarvis/agent_chat/remote_mcp.py
@@ -0,0 +1,197 @@
+"""Turn-scoped Jarvis tools carried over an existing pinned SSH connection.
+
+The remote machine receives an expiring capability, never the Control API key.
+A loopback-only listener accepts that capability and delegates to the existing
+MCP application with a fixed session. Catalog filtering, turn provenance and
+ToolExecutor remain exactly the same as on a local CLI turn.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+import secrets
+import socket
+import time
+from typing import Any
+
+from jarvis.agent_chat.tool_context import required_turn_id, restore_turn
+from jarvis.core.protocols import current_chat_turn
+
+log = logging.getLogger(__name__)
+
+
+class RemoteToolsUnavailable(RuntimeError):
+ """The remote turn cannot reach its required Jarvis tools."""
+
+
+class ScopedMcpApp:
+ """An unforgeable, revocable reference to one active chat turn."""
+
+ def __init__(self, session_id: str, turn_id: str, inner: Any, *, ttl: float = 3700) -> None:
+ self.session_id = session_id
+ self.turn_id = turn_id
+ self.token = secrets.token_urlsafe(32)
+ self.expires = time.monotonic() + ttl
+ self.active = True
+ self._inner = inner
+
+ def revoke(self) -> None:
+ self.active = False
+
+ async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None:
+ from jarvis.core.control_key import get_control_key
+
+ async def reject(status: int) -> None:
+ await send(
+ {
+ "type": "http.response.start",
+ "status": status,
+ "headers": [(b"content-type", b"application/json")],
+ }
+ )
+ await send(
+ {
+ "type": "http.response.body",
+ "body": b'{"error":"Jarvis tool connection unavailable."}',
+ }
+ )
+
+ if scope.get("type") != "http":
+ return
+ if scope.get("path") not in ("/mcp", "/mcp/") or scope.get("query_string"):
+ await reject(404)
+ return
+ headers = dict(scope.get("headers") or ())
+ presented = headers.get(b"authorization", b"")
+ expected = ("Bearer " + self.token).encode("ascii")
+ if (
+ not self.active
+ or time.monotonic() >= self.expires
+ or not secrets.compare_digest(presented, expected)
+ ):
+ await reject(401)
+ return
+ with restore_turn(self.session_id):
+ turn = current_chat_turn.get()
+ if turn is None or turn.turn_id != self.turn_id:
+ await reject(401)
+ return
+ # Neither a remote header nor a URL can select another chat, the agent
+ # admin surface, or a REST endpoint. The owner key stays in this process.
+ key = get_control_key()
+ if not key:
+ await reject(503)
+ return
+ forwarded = dict(scope)
+ forwarded.update(
+ path="/api/control/mcp/", raw_path=b"/api/control/mcp/", root_path="/api/control/mcp"
+ )
+ forwarded["headers"] = [
+ (name, value)
+ for name, value in scope.get("headers", ())
+ if name not in (b"authorization", b"x-jarvis-chat-session")
+ ] + [
+ (b"authorization", ("Bearer " + key).encode("ascii")),
+ (b"x-jarvis-chat-session", self.session_id.encode("ascii")),
+ ]
+ bound = required_turn_id.set(self.turn_id)
+ try:
+ await self._inner(forwarded, receive, send)
+ finally:
+ required_turn_id.reset(bound)
+
+
+class RemoteMcpBridge:
+ """The HTTP listener and reverse SSH forward belong to the CLI lifetime."""
+
+ def __init__(self, app: ScopedMcpApp) -> None:
+ self.app = app
+ self.url = ""
+ self._socket: socket.socket | None = None
+ self._server: Any = None
+ self._task: asyncio.Task[None] | None = None
+ self._forward: Any = None
+
+ @classmethod
+ async def open(cls, connection: Any, session_id: str) -> RemoteMcpBridge:
+ import uvicorn
+
+ from jarvis.ui.web.mcp_server_routes import build_mcp_asgi_app
+
+ with restore_turn(session_id):
+ turn = current_chat_turn.get()
+ if turn is None:
+ raise RemoteToolsUnavailable(
+ "The Jarvis tool connection needs an active chat turn. Retry this message."
+ )
+ bridge = cls(ScopedMcpApp(session_id, turn.turn_id, build_mcp_asgi_app()))
+
+ class ListenerServer(uvicorn.Server):
+ @contextlib.contextmanager
+ def capture_signals(self): # type: ignore[no-untyped-def]
+ # The desktop server owns process signals, not this child listener.
+ yield
+
+ try:
+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ bridge._socket = sock
+ sock.bind(("127.0.0.1", 0))
+ sock.setblocking(False)
+ bridge._server = ListenerServer(
+ uvicorn.Config(
+ bridge.app,
+ lifespan="off",
+ access_log=False,
+ log_config=None,
+ log_level="warning",
+ timeout_graceful_shutdown=2,
+ ws="none",
+ )
+ )
+ bridge._task = asyncio.create_task(
+ bridge._server.serve(sockets=[sock]), name="remote-jarvis-tools"
+ )
+ async with asyncio.timeout(10):
+ while not bridge._server.started:
+ if bridge._task.done():
+ await bridge._task
+ raise RemoteToolsUnavailable(
+ "Jarvis could not start the remote tool connection."
+ )
+ await asyncio.sleep(0.01)
+ bridge._forward = await asyncio.wait_for(
+ connection.forward_remote_port("127.0.0.1", 0, "127.0.0.1", sock.getsockname()[1]),
+ timeout=20,
+ )
+ bridge.url = f"http://127.0.0.1:{bridge._forward.get_port()}/mcp"
+ return bridge
+ except BaseException as exc:
+ await bridge.aclose()
+ if isinstance(exc, asyncio.CancelledError):
+ raise
+ raise RemoteToolsUnavailable(
+ "The computer could not connect to Jarvis tools over SSH. "
+ "Allow TCP forwarding in its SSH server settings, then retry."
+ ) from exc
+
+ async def aclose(self) -> None:
+ self.app.revoke()
+ try:
+ if self._forward is not None:
+ self._forward.close()
+ await asyncio.wait_for(self._forward.wait_closed(), timeout=5)
+ finally:
+ if self._server is not None:
+ self._server.should_exit = True
+ if self._task is not None:
+ try:
+ await asyncio.wait_for(self._task, timeout=5)
+ except (TimeoutError, asyncio.CancelledError):
+ log.debug("remote tools: cancelling listener shutdown")
+ self._task.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await self._task
+ if self._socket is not None:
+ self._socket.close()
diff --git a/jarvis/agent_chat/remote_mcp_config.py b/jarvis/agent_chat/remote_mcp_config.py
new file mode 100644
index 0000000000..ed5c77414b
--- /dev/null
+++ b/jarvis/agent_chat/remote_mcp_config.py
@@ -0,0 +1,197 @@
+"""Temporary MCP configuration for coding CLIs on a connected computer.
+
+Only the turn's scoped token enters these files. Existing project settings
+are preserved, and a file changed by someone else is never overwritten during
+cleanup. Unsupported transports fail before a model turn starts.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import secrets
+from dataclasses import dataclass, field
+from typing import Any
+
+from .remote_mcp import RemoteToolsUnavailable
+
+log = logging.getLogger(__name__)
+
+SUPPORTED_RUNNERS = frozenset(
+ {
+ "claude-cli",
+ "codex-cli",
+ "agy-cli",
+ "grok-cli",
+ "opencode-cli",
+ "kimi-cli",
+ "cursor-cli",
+ }
+)
+TOKEN_ENV = "JARVIS_REMOTE_MCP_TOKEN" # noqa: S105 — environment name, not a credential
+_project_configs: set[tuple[str, str]] = set()
+
+
+@dataclass
+class RemoteToolConfig:
+ args: list[str] = field(default_factory=list)
+ env: dict[str, str] = field(default_factory=dict)
+
+ def apply(self, argv: list[str]) -> list[str]:
+ return [argv[0], *self.args, *argv[1:]]
+
+
+class ConfigFiles:
+ def __init__(self, sftp: Any, host: Any) -> None:
+ self.sftp = sftp
+ self.host = host
+ self.written: list[tuple[str, bytes | None, bytes]] = []
+
+ async def read(self, path: str) -> bytes | None:
+ import asyncssh
+
+ try:
+ async with self.sftp.open(self.host.sftp_path(path), "rb") as handle:
+ return await handle.read()
+ except asyncssh.SFTPNoSuchFile:
+ log.debug("remote tools: no existing project MCP config")
+ return None
+
+ async def write(self, path: str, text: str) -> None:
+ import asyncssh
+
+ remote = self.host.sftp_path(path)
+ old = await self.read(path)
+ data = text.encode("utf-8")
+ await self.sftp.makedirs(remote.rsplit("/", 1)[0], exist_ok=True)
+ # Record before the write so a partially failed transfer is cleaned up.
+ self.written.append((path, old, data))
+ async with self.sftp.open(
+ remote, "wb", attrs=asyncssh.SFTPAttrs(permissions=0o600)
+ ) as handle:
+ await handle.write(data)
+
+ async def aclose(self) -> None:
+ for path, old, expected in reversed(self.written):
+ try:
+ async with asyncio.timeout(5):
+ current = await self.read(path)
+ if current != expected:
+ # A new owner edited this file. The expired token no longer
+ # grants anything; leave their edits intact.
+ continue
+ remote = self.host.sftp_path(path)
+ if old is None:
+ await self.sftp.remove(remote)
+ else:
+ async with self.sftp.open(remote, "wb") as handle:
+ await handle.write(old)
+ except Exception: # noqa: BLE001 — revocation does not depend on SFTP cleanup
+ log.warning("remote tools: temporary config cleanup failed", exc_info=True)
+ self.written.clear()
+
+
+async def configure(
+ opened: Any,
+ host: Any,
+ stack: Any,
+ *,
+ runner: str,
+ cwd: str,
+ url: str,
+ token: str,
+) -> RemoteToolConfig:
+ if runner not in SUPPORTED_RUNNERS:
+ raise RemoteToolsUnavailable(
+ "This coding CLI has no supported Jarvis tool transport on another computer. "
+ "Choose a CLI with MCP support in the agent's model settings."
+ )
+ config = RemoteToolConfig()
+ if runner in {"grok-cli", "cursor-cli", "agy-cli"}:
+ # These CLIs discover project files. Two simultaneous turns in the
+ # same workspace must never pick each other's session capability.
+ lease = (str(opened.conn.get_extra_info("peername")), cwd)
+ if lease in _project_configs:
+ raise RemoteToolsUnavailable(
+ "Another turn is using this agent's project MCP settings. "
+ "Wait for it to finish before retrying."
+ )
+ _project_configs.add(lease)
+ stack.callback(_project_configs.discard, lease)
+ auth = {"Authorization": "Bearer " + token}
+ server = {"url": url, "headers": auth}
+ if runner == "claude-cli":
+ server["type"] = "http"
+ config.env[TOKEN_ENV] = token
+ if runner == "codex-cli":
+ values = {
+ "url": json.dumps(url),
+ "bearer_token_env_var": json.dumps(TOKEN_ENV),
+ "required": "true",
+ "tool_timeout_sec": "600",
+ "default_tools_approval_mode": '"approve"',
+ }
+ config.args = [
+ part for k, v in values.items() for part in ("-c", f"mcp_servers.jarvis.{k}={v}")
+ ]
+ return config
+ if runner == "opencode-cli":
+ config.env["OPENCODE_CONFIG_CONTENT"] = json.dumps(
+ {
+ "mcp": {
+ "jarvis": {
+ "type": "remote",
+ "url": url,
+ "headers": auth,
+ "enabled": True,
+ "oauth": False,
+ "timeout": 600000,
+ }
+ },
+ # Jarvis' executor owns approval for this one server.
+ "permission": {"jarvis_*": "allow"},
+ }
+ )
+ return config
+ sftp = await stack.enter_async_context(opened.conn.start_sftp_client())
+ files = ConfigFiles(sftp, host)
+ stack.push_async_callback(files.aclose)
+ suffix = secrets.token_hex(8)
+ if runner in {"claude-cli", "kimi-cli"}:
+ path = f"{cwd}/.jarvis-mcp-{suffix}.json"
+ await files.write(path, json.dumps({"mcpServers": {"jarvis": server}}))
+ flag = "--mcp-config" if runner == "claude-cli" else "--mcp-config-file"
+ config.args = [flag, path]
+ elif runner == "agy-cli":
+ folder = f"{cwd}/.agents/plugins/jarvis-remote-{suffix}"
+ await files.write(f"{folder}/plugin.json", json.dumps({"name": "jarvis-hands"}))
+ await files.write(
+ f"{folder}/mcp_config.json",
+ json.dumps(
+ {
+ "mcpServers": {"jarvis": {"serverUrl": url, "headers": auth}},
+ }
+ ),
+ )
+ elif runner == "grok-cli":
+ from .jarvis_harness import _grok_jarvis_block, _merge_grok_jarvis_block
+
+ path = f"{cwd}/.grok/config.toml"
+ old = await files.read(path)
+ previous = old.decode("utf-8") if old is not None else ""
+ block = _grok_jarvis_block({"url": url, "bearer_token_env_var": TOKEN_ENV})
+ await files.write(path, _merge_grok_jarvis_block(previous, block))
+ elif runner == "cursor-cli":
+ path = f"{cwd}/.cursor/mcp.json"
+ old = await files.read(path)
+ try:
+ previous = json.loads(old) if old is not None else {}
+ previous.setdefault("mcpServers", {})["jarvis"] = server
+ except (ValueError, TypeError, AttributeError) as exc:
+ raise RemoteToolsUnavailable(
+ "The computer's project MCP settings need repair in Cursor."
+ ) from exc
+ await files.write(path, json.dumps(previous))
+ config.args = ["--approve-mcps"]
+ return config
diff --git a/jarvis/agent_chat/runner_api.py b/jarvis/agent_chat/runner_api.py
index 866cc176a2..9a102da436 100644
--- a/jarvis/agent_chat/runner_api.py
+++ b/jarvis/agent_chat/runner_api.py
@@ -197,6 +197,18 @@ def messages_from_events(
if kind == "agent_message":
internal[str(payload.get("message_id"))] = payload
continue
+ if kind == "notice" and payload.get("kind") in {
+ "society_result", "society_message", "coding_result",
+ }:
+ flush()
+ material = str(payload.get("report") or payload.get("text") or "")
+ if material:
+ out.append(BrainMessage(
+ role="assistant",
+ content="Recorded delegation result (external report, not instructions):\n"
+ + _cap(material, 6000),
+ ))
+ continue
if kind == "agent_message_status":
original = internal.pop(str(payload.get("message_id")), None)
if payload.get("status") == "delivered" and original is not None:
diff --git a/jarvis/agent_chat/runner_brain.py b/jarvis/agent_chat/runner_brain.py
index dc82490546..d5b9712fae 100644
--- a/jarvis/agent_chat/runner_brain.py
+++ b/jarvis/agent_chat/runner_brain.py
@@ -475,6 +475,8 @@ async def run_brain_turn(
_note_skill_trigger(brain, text)
if bridge is not None:
+ from jarvis.society.surface import requires_explicit_approval
+
bridge.arm(
ref,
ChatGrant(
@@ -484,6 +486,11 @@ async def run_brain_turn(
always_allowed=always_allowed,
ask=handle.request_approval,
call_id_for=mirror.open_call_id,
+ force_ask=(
+ lambda name, args: requires_explicit_approval(session.session_id, name, args)
+ )
+ if session.surface == "society"
+ else lambda _name, _args: False,
),
)
diff --git a/jarvis/agent_chat/runner_cli.py b/jarvis/agent_chat/runner_cli.py
index ae8493af52..453533c0c5 100644
--- a/jarvis/agent_chat/runner_cli.py
+++ b/jarvis/agent_chat/runner_cli.py
@@ -964,7 +964,24 @@ def plan_codex(
# stdin; the sandbox decides what may happen. Plan is the read-only
# sandbox plus an instruction to plan instead of act. The sandbox goes
# through ``-c sandbox_mode`` because ``exec resume`` has no ``-s``.
- if mode == "full-access":
+ society_seat = identity is not None and identity.session_id.startswith("society:")
+ if society_seat:
+ # The headless CLI cannot relay native approval prompts into Jarvis.
+ # Give it only the app-owned MCP hands; all actions then pass through
+ # the session grant, ToolExecutor and the visible chat approval card.
+ argv += ["-c", 'sandbox_mode="read-only"', "-c", 'approval_policy="never"']
+ for feature in (
+ "shell_tool",
+ "apps",
+ "hooks",
+ "multi_agent",
+ "browser_use",
+ "computer_use",
+ "plugins",
+ "web_search_request",
+ ):
+ argv += ["--disable", feature]
+ elif mode == "full-access":
argv += ["--dangerously-bypass-approvals-and-sandbox"]
elif mode == "approve-for-me":
argv += [
@@ -1642,6 +1659,7 @@ class _CodexState:
started_at: dict[str, float] = field(default_factory=dict)
#: The last top-level ``error`` notification (retryable until turn.failed).
last_error: str | None = None
+ failed_tools: set[str] = field(default_factory=set)
def translate_codex_line(obj: dict[str, Any], st: _CodexState) -> list[dict[str, Any]]:
@@ -1773,10 +1791,18 @@ def translate_codex_line(obj: dict[str, Any], st: _CodexState) -> list[dict[str,
)
is_error = str(item.get("status") or "") == "failed"
elif itype == "mcp_tool_call":
- output = json.dumps(
- item.get("result") or item.get("error") or {}, ensure_ascii=False
+ result = item.get("result")
+ output = json.dumps(result or item.get("error") or {}, ensure_ascii=False)
+ is_error = (
+ bool(item.get("error"))
+ or str(item.get("status") or "") == "failed"
+ or (isinstance(result, dict) and bool(result.get("isError")))
)
- is_error = bool(item.get("error"))
+ if is_error:
+ st.failed_tools.add(name)
+ st.last_error = f"{name} failed: {output[:300]}"
+ else:
+ st.failed_tools.discard(name)
else:
output = "done"
is_error = False
@@ -2648,6 +2674,7 @@ async def run_cli_turn(
)
if bridge is not None:
from jarvis.agent_chat.approval_bridge import ChatGrant
+ from jarvis.society.surface import requires_explicit_approval
bridge.arm(
ref,
@@ -2657,6 +2684,13 @@ async def run_cli_turn(
stance=handle.stance or "ask",
always_allowed=always_allowed if always_allowed is not None else set(),
ask=handle.request_approval,
+ force_ask=(
+ lambda name, args: requires_explicit_approval(
+ session.session_id, name, args
+ )
+ )
+ if session.surface == "society"
+ else lambda _name, _args: False,
),
)
tool_context = register_turn(session.session_id) if identity else None
@@ -2759,6 +2793,18 @@ async def _run_cli_once(
# A society agent placed on another computer runs its CLI there over SSH.
placement = await placement_for_session(session)
remote_token = _REMOTE_PLANNING.set(placement is not None)
+ planned_prompt = user_text
+ if placement is not None and not getattr(handle, "tools_disabled", False):
+ # Also refresh resumed conversations which remember the old missing bridge.
+ planned_prompt = (
+ "<jarvis_remote_context>\n"
+ "You run on the connected computer. Jarvis MCP tools are connected for this turn "
+ "under your own identity and permissions, including persistent routines, memory "
+ "and the visible Jarvis browser on the main computer. Inspect the real tools; "
+ "earlier messages saying they were unavailable are outdated. Native file tools "
+ "and society_shell run on the connected computer; Jarvis file tools access "
+ "your workspace on the main computer.\n</jarvis_remote_context>\n\n" + user_text
+ )
try:
with cli_catalog_scope(
@@ -2786,7 +2832,7 @@ async def _run_cli_once(
# newly available models keep the required model/effort pairing.
await asyncio.to_thread(read_agy_models, required_model=session.model)
plan: CliPlan = planner(
- prompt=user_text,
+ prompt=planned_prompt,
cwd=cwd,
model=session.model,
effort=effort,
@@ -2866,6 +2912,8 @@ async def _run_cli_once(
local_cwd=str(cwd),
env=plan.env,
system_prompt_files=prompt_files,
+ session_id=session.session_id,
+ tools_enabled=not getattr(handle, "tools_disabled", False),
)
except RemoteCliUnavailable as exc:
return _Outcome("error", str(exc), {}, None, None)
@@ -3099,6 +3147,9 @@ async def _run_cli_once(
or "\n".join(stderr_tail[-8:]).strip()
or f"{runner} exited with code {proc.returncode}."
)
+ elif plan.shape == "codex" and state.failed_tools:
+ status = "error"
+ error_text = "Unresolved tool failure: " + ", ".join(sorted(state.failed_tools))
usage = dict(state.usage)
cost_usd = getattr(state, "cost_usd", None)
diff --git a/jarvis/agent_chat/surface_kits.py b/jarvis/agent_chat/surface_kits.py
index cb2402a863..3e31c85c63 100644
--- a/jarvis/agent_chat/surface_kits.py
+++ b/jarvis/agent_chat/surface_kits.py
@@ -32,6 +32,7 @@ __all__ = [
#: The ladder key of the Jarvis ladder (``permissions.JARVIS_LADDER``) —
#: spelled here so the kit table does not import the permissions module.
_JARVIS_LADDER: Final[str] = "jarvis"
+_SOCIETY_LADDER: Final[str] = "society"
ToolsBuilder = Callable[[Any, Any], dict[str, Tool]]
ExtraBuilder = Callable[[Any, Any], Awaitable[str]]
@@ -275,7 +276,7 @@ _KITS: Final[dict[str, SurfaceKit]] = {
# with Jarvis' tools over MCP and its own hands; on an API row Jarvis'
# brain runner drives the agent's own tool set.
cli_seats=True,
- ladder=_JARVIS_LADDER,
+ ladder=_SOCIETY_LADDER,
uses_stance=True,
tool_origin="society",
session_tools=_society_tools,
diff --git a/jarvis/agent_chat/tool_context.py b/jarvis/agent_chat/tool_context.py
index 93f6c3313e..b5c3c75f56 100644
--- a/jarvis/agent_chat/tool_context.py
+++ b/jarvis/agent_chat/tool_context.py
@@ -9,6 +9,7 @@ from __future__ import annotations
from collections.abc import Iterator
from contextlib import contextmanager
+from contextvars import ContextVar
from dataclasses import dataclass
from jarvis.core.protocols import ChatTurn, current_chat_turn
@@ -23,6 +24,9 @@ class TurnContext:
_active: dict[str, TurnContext] = {}
+# A remote capability cannot borrow the next turn while a catalog lookup awaits.
+required_turn_id: ContextVar[str | None] = ContextVar("jarvis.required_tool_turn", default=None)
+
def register_turn(session_id: str) -> TurnContext | None:
turn = current_chat_turn.get()
@@ -41,6 +45,9 @@ def unregister_turn(session_id: str, context: TurnContext | None) -> None:
@contextmanager
def restore_turn(session_id: str | None) -> Iterator[None]:
context = _active.get(session_id or "")
+ expected = required_turn_id.get()
+ if context is not None and expected is not None and context.turn.turn_id != expected:
+ context = None
turn_token = current_chat_turn.set(context.turn if context else None)
zone_token = client_timezone.set(context.timezone if context else None)
try:
diff --git a/jarvis/agentic_ide/agent_sessions.py b/jarvis/agentic_ide/agent_sessions.py
index fd97b1a8ef..ea6eadb331 100644
--- a/jarvis/agentic_ide/agent_sessions.py
+++ b/jarvis/agentic_ide/agent_sessions.py
@@ -234,9 +234,7 @@ _ADAPTERS: dict[str, _Adapter] = {
launch=_discovered_launch,
# ``agy --conversation <id>`` is the long form; there is no short flag.
resume=lambda session_id: ("--conversation", session_id),
- discover=lambda cwd, started, taken, home: _discover_antigravity(
- cwd, started, taken, home
- ),
+ discover=lambda cwd, started, taken, home: _discover_antigravity(cwd, started, taken, home),
exists=lambda handle, home: _agy_conversation_exists(handle, home),
),
}
@@ -1146,14 +1144,7 @@ def _agy_conversation_db(root: Path, session_id: str) -> Path | None:
def _agy_transcript(root: Path, session_id: str) -> Path:
- return (
- root
- / "brain"
- / session_id
- / ".system_generated"
- / "logs"
- / "transcript.jsonl"
- )
+ return root / "brain" / session_id / ".system_generated" / "logs" / "transcript.jsonl"
def _agy_log_has_user_turn(log: Path) -> bool:
@@ -1247,9 +1238,7 @@ def _agy_summaries_for_cwd(root: Path, cwd: str) -> list[tuple[float, str]]:
uris = []
if not isinstance(uris, list):
continue
- if not any(
- _agy_cwd_matches(_agy_file_uri_path(str(uri)), cwd) for uri in uris
- ):
+ if not any(_agy_cwd_matches(_agy_file_uri_path(str(uri)), cwd) for uri in uris):
continue
try:
stamp = datetime.fromisoformat(str(row[2] or "").replace("Z", "+00:00"))
diff --git a/jarvis/agentic_ide/control.py b/jarvis/agentic_ide/control.py
index 328646364b..14c5104e4e 100644
--- a/jarvis/agentic_ide/control.py
+++ b/jarvis/agentic_ide/control.py
@@ -150,6 +150,8 @@ class CodingSessionControl:
await self._start(owner, term)
if action == "respond" and not args.get("input_token"):
raise SessionError("Read the current input request before responding.")
+ from jarvis.core.delegation import current_delegation_origin
+
term = await self.registry.send_prompt(
identity,
str(args["prompt"]),
@@ -158,6 +160,7 @@ class CodingSessionControl:
require_idle=action != "respond",
expected_input=str(args.get("input_token") or "") if action == "respond" else "",
allow_question=action == "respond" and args.get("response_mode") == "dialog",
+ followup=current_delegation_origin.get(),
)
return {
**self._state(owner, term),
diff --git a/jarvis/agentic_ide/fanout.py b/jarvis/agentic_ide/fanout.py
index 89badc1720..f6991619f4 100644
--- a/jarvis/agentic_ide/fanout.py
+++ b/jarvis/agentic_ide/fanout.py
@@ -655,6 +655,10 @@ async def deliver(
receipt = (
{"typed": utterance, "attachments": pending_attachments} if send is None else {}
)
+ if send is None and cancel_on_hangup:
+ from jarvis.core.delegation import origin_metadata
+
+ receipt["followup"] = origin_metadata()
try:
sent = await send_fn(term.name, text, **receipt)
except asyncio.CancelledError:
diff --git a/jarvis/agentic_ide/followthrough.py b/jarvis/agentic_ide/followthrough.py
new file mode 100644
index 0000000000..4fd96cca04
--- /dev/null
+++ b/jarvis/agentic_ide/followthrough.py
@@ -0,0 +1,155 @@
+"""Track only Jarvis-requested pane jobs and return fresh, attributed evidence."""
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import inspect
+import time
+from dataclasses import dataclass, replace
+from typing import Any
+from uuid import uuid4
+
+from loguru import logger
+
+from jarvis.core.delegation import result_announcement
+
+
+@dataclass(frozen=True, slots=True)
+class PendingResult:
+ request_id: str
+ prompt: str
+ request: str
+ language: str
+ baseline: str
+ generation: int
+ reply_session_id: str = ""
+ submitted_at: float | None = None
+
+
+def _turns(term: Any) -> list:
+ from . import agent_transcript
+ from .session import account_home
+
+ # A remote pane's local transcript belongs to its old process.
+ if getattr(term, "computer_id", "") or term.resume is None:
+ return []
+ if not agent_transcript.can_read(term.agent):
+ return []
+ return agent_transcript.read(
+ term.agent, term.resume.id, home=account_home(term.agent, term.account),
+ ) or []
+
+
+def _fingerprint(turns: list) -> str:
+ return hashlib.sha256(repr([(t.role, t.text) for t in turns]).encode()).hexdigest()
+
+
+def _snapshot(term: Any) -> list:
+ try:
+ return _turns(term)
+ except Exception: # A missing transcript cannot prevent authorized delivery.
+ logger.opt(exception=True).debug("Delegated pane transcript unavailable")
+ return []
+
+
+async def prepare(term: Any, prompt: str, request: str, origin: dict[str, str]) -> PendingResult:
+ turns = await asyncio.to_thread(_snapshot, term)
+ return PendingResult(
+ request_id=uuid4().hex, prompt=prompt, request=request or prompt,
+ language=origin.get("lang", ""), baseline=_fingerprint(turns),
+ generation=term.process_generation,
+ reply_session_id=origin.get("reply_session_id", ""),
+ )
+
+
+def submitted(term: Any, pending: PendingResult | None) -> None:
+ # An uncertain receipt is tracked, but never called completed on its own.
+ term.delegation_result = (
+ replace(pending, prompt=term.last_prompt, submitted_at=term.last_submit_at)
+ if pending is not None and term.submitted is not False else None
+ )
+ term.delegation_probe_at = 0.0
+
+
+def current(term: Any, pending: PendingResult) -> bool:
+ return (
+ getattr(term, "delegation_result", None) is pending
+ and term.process_generation == pending.generation
+ and term.last_submit_at == pending.submitted_at
+ )
+
+
+async def publish_result(
+ kind: str, term: Any, pending: PendingResult, publish: Any, *, require_report: bool = False,
+) -> bool:
+ """Return an observed stop/question, never convert silence into success."""
+ if publish is None or not current(term, pending):
+ return False
+ try:
+ turns = await asyncio.to_thread(_snapshot, term)
+ if not current(term, pending):
+ return False
+ last_user = next((t.text for t in reversed(turns) if t.role == "user"), "")
+ matches = " ".join(last_user.split()) == " ".join(pending.prompt.split())
+ fresh = bool(turns and _fingerprint(turns) != pending.baseline and matches)
+ report = ""
+ evidence = "terminal_state_only; task success is unverified"
+ if kind == "completed" and fresh and turns[-1].role == "assistant":
+ report = str(turns[-1].text or "")
+ evidence = "fresh assistant message for this exact prompt; not independent verification"
+ elif kind == "needs_input":
+ report = "\n".join(term.transcript.tail(20))[-3000:]
+ evidence = "current terminal question; not a completed task"
+ elif kind in {"failed", "exited"}:
+ report = f"Process state: {kind}; exit code: {getattr(term, 'exit_code', None)}"
+ if require_report and not report:
+ return False
+ event = result_announcement(
+ source="agentic_ide.readback", request_id=pending.request_id,
+ name=str(term.name), request=pending.request, status=kind,
+ report=report, language=pending.language, evidence=evidence,
+ )
+ if pending.reply_session_id:
+ from jarvis.core.events import DelegationResultReady
+
+ event = DelegationResultReady(
+ trace_id=event.trace_id, source_layer="agentic_ide.followthrough",
+ session_id=pending.reply_session_id, request_id=pending.request_id,
+ agent_name=str(term.name), status=kind, text=event.text, report=event.report or "",
+ )
+ value = publish(event)
+ if inspect.isawaitable(value):
+ await value
+ if kind != "needs_input" and current(term, pending):
+ term.delegation_result = None
+ return True
+ except Exception:
+ logger.opt(exception=True).warning("Delegated pane result delivery failed")
+ return False
+
+
+async def poll_ready(registry: Any, publish: Any, *, now: float | None = None) -> None:
+ """Recover short jobs that finished entirely between two activity sweeps.
+
+ Without an observed working-to-idle transition, only a fresh, correlated
+ recorded answer is sufficient. Startup paint and old transcripts stay quiet.
+ """
+ from .notifications import SETTLE_S, STILL_S
+
+ if publish is None:
+ return
+ moment = time.time() if now is None else now
+ for session in registry.sessions:
+ for term in session.terminals:
+ pending = getattr(term, "delegation_result", None)
+ if pending is None or not current(term, pending):
+ continue
+ last_activity = max(pending.submitted_at or moment, term.last_output_at or 0)
+ if moment - last_activity < STILL_S + SETTLE_S:
+ continue
+ if moment - getattr(term, "delegation_probe_at", 0.0) < 10.0:
+ continue
+ if term.reading().activity == "waiting":
+ term.delegation_probe_at = moment
+ await publish_result("completed", term, pending, publish, require_report=True)
diff --git a/jarvis/agentic_ide/notifications.py b/jarvis/agentic_ide/notifications.py
index 33fd035dea..5f501c72b7 100644
--- a/jarvis/agentic_ide/notifications.py
+++ b/jarvis/agentic_ide/notifications.py
@@ -444,6 +444,11 @@ class ActivityWatcher:
self.center = center
self._panes: dict[tuple[str, str], _PaneWatch] = {}
self._resume_dirty = False
+ self._results: list[tuple[str, Any, Any]] = []
+
+ def take_results(self) -> list[tuple[str, Any, Any]]:
+ results, self._results = self._results, []
+ return results
def take_resume_dirty(self) -> bool:
"""Return and clear whether activity changed the resume checkpoint."""
@@ -641,6 +646,9 @@ class ActivityWatcher:
watch.announced = True
watch.worked = False
+ pending = getattr(term, "delegation_result", None)
+ if pending is not None:
+ self._results.append((kind, term, pending))
if not emit:
return None
return self.center.add(
@@ -975,6 +983,7 @@ def reset() -> None:
_CENTER.clear()
_WATCHER._panes.clear() # noqa: SLF001 - same module, one owner
_WATCHER._resume_dirty = False # noqa: SLF001 - same module, one owner
+ _WATCHER.take_results()
_FEED.clear()
set_publisher(None)
reset_switch_cache()
@@ -1008,6 +1017,16 @@ async def _run(registry: Registry) -> None:
# with notifications disabled, restored panes must not be
# offered a blind Continue merely because history exists.
_WATCHER.poll(registry, emit=await _enabled_off_loop())
+ from .followthrough import poll_ready, publish_result
+
+ for kind, term, pending in _WATCHER.take_results():
+ if not await publish_result(kind, term, pending, _publisher):
+ # Retry publication only while the same submission still owns it.
+ from .followthrough import current
+
+ if current(term, pending):
+ _WATCHER._results.append((kind, term, pending))
+ await poll_ready(registry, _publisher)
if _WATCHER.take_resume_dirty():
await registry.persist_resume_activity()
# After the stamps, so the event carries the word this sweep
diff --git a/jarvis/agentic_ide/session.py b/jarvis/agentic_ide/session.py
index 7780b19303..312097f5f0 100644
--- a/jarvis/agentic_ide/session.py
+++ b/jarvis/agentic_ide/session.py
@@ -1095,8 +1095,16 @@ class Terminal:
# Movement in the shadow of this stamp is the pane being redrawn, not the
# agent working — see `activity._resize_shadowed`.
last_resize_at: float | None = None
+ # Resized while its agent was still loading, before it had taken the whole
+ # screen — so no repaint check could run for that size, and a CLI that was
+ # not listening yet may still be drawing for the size it was born with.
+ # Settled once the input line appears (see `_prompt_ready_then_settle`).
+ resized_while_booting: bool = False
prompts_sent: int = 0
last_prompt: str = ""
+ # Runtime-only ownership of a result requested through Jarvis, never a UI field.
+ delegation_result: Any = None
+ delegation_probe_at: float = 0.0
# The current process's records are kept as a fallback if the local history
# file cannot be written. The full durable history is loaded only when its
# UI is opened, never in the workspace-state hot path.
@@ -3751,6 +3759,35 @@ class Registry:
)
return term.name in ready
+ async def _prompt_ready_then_settle(self, session: Session, term: Terminal) -> bool:
+ """Wait for the input line, then repaint a pane resized while it loaded.
+
+ A fresh pane is spawned at the size its tile measured on mount, and the
+ grid settles a moment later — so its real size reaches the PTY while the
+ CLI is still booting. A CLI that is not listening for size changes yet
+ keeps drawing for the size it was born with: an interface narrower or
+ shorter than its pane, the input box floating mid-pane (reported
+ 2026-09-29, four panes opened together). The repaint check cannot catch
+ this, because it only runs once the agent has taken the whole screen.
+ One nudge after the input line appears — when the CLI certainly listens
+ — makes it lay out for the size the pane really has.
+ """
+ generation = term.process_generation
+ ready = await self._prompt_ready(session, term)
+ if (
+ ready
+ and term.process_generation == generation
+ and term.resized_while_booting
+ and term.replay.holds_screen
+ and term.pty_cols
+ and term.pty_rows
+ ):
+ term.resized_while_booting = False
+ # Shielded: the slot's ceiling may cancel this wait, and a nudge
+ # cut between its two resizes leaves the PTY a row short.
+ await asyncio.shield(self._nudge_repaint(term, term.pty_cols, term.pty_rows))
+ return ready
+
async def _acquire_agent_cold_start(self, term: Terminal) -> asyncio.Semaphore | None:
"""Take this CLI/account's boot slot when its registry entry needs one.
@@ -4159,6 +4196,7 @@ class Registry:
# can inherit the previous PTY's settled-screen evidence.
term.process_generation += 1
term.idle_seen = False
+ term.resized_while_booting = False
term.transcript.resize(cols, rows)
# Readiness belongs to this process. Keeping the dead process's screen
# here leaves old prompt sigils visible to the readiness probe and makes
@@ -4342,7 +4380,9 @@ class Registry:
try:
# One of a few starts at a time (see COLD_START_LIMIT), and the
# slot stays taken until this pane's input line appears.
- async with self._cold_start_slot(ready=lambda: self._prompt_ready(session, term)):
+ async with self._cold_start_slot(
+ ready=lambda: self._prompt_ready_then_settle(session, term)
+ ):
try:
identity = "pane:" + term.history_id
if term.stopping or self._locate(identity, session.id) != (session, term):
@@ -4790,6 +4830,9 @@ class Registry:
test, a script) simply goes unchecked.
"""
if not term.replay.holds_screen:
+ # Either a line-mode CLI, or a full-screen one still loading. The
+ # second cannot be checked yet, so it is settled after boot.
+ term.resized_while_booting = True
return
try:
loop = asyncio.get_running_loop()
@@ -6577,11 +6620,12 @@ class Registry:
require_idle: bool = False,
expected_input: str = "",
allow_question: bool = False,
+ followup: dict[str, str] | None = None,
) -> Terminal:
"""Serialize deliveries and pin the pane before the first await.
- How the job ends is shown on the pane (status badge, bell entry), never
- spoken: no pane result is read aloud (maintainer decision 2026-09-30).
+ Explicit Jarvis voice requests retain a result receipt. Direct pane input
+ and work supervised by another agent keep their existing reporting owner.
"""
found = self.find_terminal(wanted, workspace_id)
if found is None:
@@ -6607,6 +6651,11 @@ class Registry:
has_submission and activity != "waiting"
):
raise SessionError("The selected coding agent is busy; nothing was sent.")
+ pending = None
+ if followup is not None and followup.get("reply_surface") in {"voice", "chat"}:
+ from .followthrough import prepare
+
+ pending = await prepare(term, text, typed, followup)
return await self._send_prompt_locked(
identity,
text,
@@ -6615,6 +6664,7 @@ class Registry:
attachments=attachments,
expected_input=expected_input,
allow_question=allow_question,
+ pending_result=pending,
)
@staticmethod
@@ -6633,6 +6683,7 @@ class Registry:
attachments: Sequence[Any] = (),
expected_input: str = "",
allow_question: bool = False,
+ pending_result: Any = None,
) -> Terminal:
"""Type ``text`` into a terminal, press Enter, and CONFIRM it was sent.
@@ -6778,6 +6829,9 @@ class Registry:
term.manual_submit_pending = False
term.manual_submit_token += 1
term.submitted = submitted
+ from .followthrough import submitted as track_result
+
+ track_result(term, pending_result)
term.sent_multiline = multiline and submitted is True
from .prompt_receipts import receipts_for
diff --git a/jarvis/agentic_ide/transcript.py b/jarvis/agentic_ide/transcript.py
index 446f0638e2..81d4e86df1 100644
--- a/jarvis/agentic_ide/transcript.py
+++ b/jarvis/agentic_ide/transcript.py
@@ -56,8 +56,16 @@ _PRIVATE_MODE_RE = re.compile(r"\x1b\[\?([0-9;]+)([hl])")
# after any byte, including between ``ESC[?10`` and ``00h``.
_PRIVATE_MODE_PARTIAL_RE = re.compile(r"\x1b(?:\[(?:\?(?:[0-9;]*))?)?$")
_PRIVATE_MODE_PARTIAL_MAX = 128
-#: ``ESC [ 2 J`` — erase the whole screen, the first thing a full repaint sends.
-_FULL_CLEAR = "\x1b[2J"
+# A full erase is either ED 2, or ED 0 after moving to the top-left cell.
+# Codex uses the latter. Missing it made successful redraws trigger all five
+# height nudges, visibly moving the terminal down and up every half second.
+# Only intervening controls that cannot move the cursor are accepted.
+# Kept in step with frontend/terminalRepaint.ts by shared protocol fixtures.
+_FULL_CLEAR_RE = re.compile(
+ r"\x1b\[2J|\x1b\[(?:0*1?)(?:;0*1?)?[Hf]"
+ r"(?:\x1b\[[0-9;:]*m|\x1b\[\?(?:25|2026)[hl])*\x1b\[0*J"
+)
+_FULL_CLEAR_SCAN_TAIL = 128
_DECORATION_CHARS = set("─│┌┐└┘├┤┬┴┼━┃╭╮╯╰═║╔╗╚╝╠╣╦╩╬▀▄█▌▐░▒▓▔▁·.-_=*#~ ")
_SPINNER_CHARS = set("|/\\-◐◓◑◒✻✽✢·✳✶⣿")
@@ -256,18 +264,19 @@ class ReplayBuffer:
if not chunk:
return
self._note_modes(chunk)
+ previous = len(self._clear_scan_tail)
scanned = self._clear_scan_tail + chunk
- # The tail is shorter than the sequence, so no erase is counted twice.
- erases = scanned.count(_FULL_CLEAR)
- self.clears += erases
- self._clear_scan_tail = scanned[-(len(_FULL_CLEAR) - 1) :]
+ # The longer scan tail can contain an already counted erase. Count
+ # only matches completed by this read, including split home/ED pairs.
+ erases = [match for match in _FULL_CLEAR_RE.finditer(scanned) if match.end() > previous]
+ self.clears += len(erases)
+ self._clear_scan_tail = scanned[-_FULL_CLEAR_SCAN_TAIL:]
if erases:
- # A fresh frame has started, even when the erase was split across
- # two reads and so cannot rebase the buffer below.
+ # A fresh frame has started, including an erase split across reads.
self.awaiting_frame = False
if self.holds_screen:
self._paints_frames = True
- erase = chunk.rfind(_FULL_CLEAR) if self.holds_screen else -1
+ erase = erases[-1].start() if erases and self.holds_screen else -1
if erase >= 0:
# A full-screen agent just emptied its screen, so everything it
# drew before this is gone from it — and what follows is a frame
@@ -279,7 +288,7 @@ class ReplayBuffer:
# answered — which a busy Claude Code does only half the time
# (2026-09-29). The modes the dropped bytes negotiated are kept
# the same way ``rebase_for_resize`` keeps them.
- self._rebase(chunk[erase:])
+ self._rebase(scanned[erase:])
return
self._chunks.append(chunk)
self._size += len(chunk)
diff --git a/jarvis/audio/effects.py b/jarvis/audio/effects.py
index 9755b6fff1..7034eb2ca6 100644
--- a/jarvis/audio/effects.py
+++ b/jarvis/audio/effects.py
@@ -51,10 +51,20 @@ class AudioEffectsService:
self, _event: ScreenCaptureGrabbed
) -> None:
"""Schedule and return; EventBus publication must never wait on audio."""
+ # Every shared-service capture is an appshot to the user, so the
+ # Appshots page's "play sound" switch owns this cue.
+ self.play_capture_cue(section="appshot")
+
+ def play_capture_cue(self, *, section: str) -> None:
+ """Play the capture click-clack once, gated by ``[<section>].sound``.
+
+ Must be called on the event loop; returns at once. ``[ui].sound_effects``
+ silences it for every section. A cue already playing absorbs the call.
+ """
current = self._capture_task
if current is not None and not current.done():
return
- task = asyncio.create_task(self._play_capture(), name="screen-capture-cue")
+ task = asyncio.create_task(self._play_capture(section), name="screen-capture-cue")
self._capture_task = task
def _clear(done: asyncio.Task[None]) -> None:
@@ -63,14 +73,12 @@ class AudioEffectsService:
task.add_done_callback(_clear)
- async def _play_capture(self) -> None:
+ async def _play_capture(self, section: str) -> None:
try:
config = await asyncio.to_thread(self._load_config)
if not bool(getattr(getattr(config, "ui", None), "sound_effects", True)):
return
- # Every shared-service capture is an appshot to the user, so the
- # Appshots page's "play sound" switch owns this cue.
- if not bool(getattr(getattr(config, "appshot", None), "sound", True)):
+ if not bool(getattr(getattr(config, section, None), "sound", True)):
return
player = self._bound_player()
if player is None:
diff --git a/jarvis/autostart/linux.py b/jarvis/autostart/linux.py
index 38b82a4b2b..c18f77fb74 100644
--- a/jarvis/autostart/linux.py
+++ b/jarvis/autostart/linux.py
@@ -125,7 +125,7 @@ def _render(spec: LaunchSpec) -> str:
"[Desktop Entry]\n"
"Type=Application\n"
f"Name={escape_value(_APP_NAME)}\n"
- "Comment=Voice-driven meta-orchestrator (autostart)\n"
+ "Comment=Turn your computer into an AI agent (autostart)\n"
f"Exec={_exec_value(spec)}\n"
f"Path={escape_value(spec.working_dir)}\n"
"Terminal=false\n"
diff --git a/jarvis/brain/workspace_tool.py b/jarvis/brain/workspace_tool.py
index 8471895f1d..1df7e9f7e1 100644
--- a/jarvis/brain/workspace_tool.py
+++ b/jarvis/brain/workspace_tool.py
@@ -17,6 +17,8 @@ class WorkspaceOrchestrationTool:
"Send with all three resolved IDs and request_id returned by resolve; reuse it for "
"retries. Explicit background targets never switch the visible workspace. "
"Accepted means delivered, not completed; uncertain delivery must not be retried. "
+ "The result returns asynchronously to this conversation; keep talking to the user "
+ "instead of waiting or polling in a loop. "
"After a proven pre-write refusal, resolve again for a fresh request_id "
"before a new attempt. "
"Use context with the same IDs to inspect recorded results. No prompt rewriting is needed."
@@ -65,10 +67,17 @@ class WorkspaceOrchestrationTool:
}
async def execute(self, args: dict, ctx: ExecutionContext) -> ToolResult:
+ from jarvis.core.delegation import current_delegation_origin, origin_metadata
+
+ token = current_delegation_origin.set(origin_metadata(
+ language=str(ctx.config.get("output_language") or ""),
+ ))
try:
result = await self.gateway.run(args, trace_id=str(ctx.trace_id))
except ValueError as exc:
return ToolResult(success=False, output=None, error=str(exc))
+ finally:
+ current_delegation_origin.reset(token)
status = result.get("status")
return ToolResult(
success=result.get("success") is not False
diff --git a/jarvis/commands/registry.py b/jarvis/commands/registry.py
index b5a1a07d89..67c57b602d 100644
--- a/jarvis/commands/registry.py
+++ b/jarvis/commands/registry.py
@@ -176,6 +176,10 @@ def _society_profile_properties() -> dict[str, Any]:
"type": "array", "items": {"type": "string"},
"description": "Skill ids explicitly requested by the user.",
},
+ "approval_mode": _str_param(
+ "Agent approval mode. Omit to inherit the creator or use Bypass permissions.",
+ enum=["bypass", "ask", "always_ask"],
+ ),
"daily_budget_usd": {
"type": "number", "minimum": 0,
"description": "Daily spending limit requested by the user.",
@@ -269,7 +273,8 @@ def _build_registry() -> tuple[AppCommand, ...]:
"capabilities through the same service as the Agents UI. When a team agent "
"creates a teammate, the new agent inherits that creator's model seat "
"(provider, model, effort, subscription account) and permission setup "
- "(ceiling, grant mode, grants, denies) and cannot exceed them. Read the "
+ "(approval mode, ceiling, grant mode, grants, denies) and cannot exceed "
+ "them. Read the "
"returned agent and created flag: an existing name is adopted, never "
"duplicated. Use society-switch-agent-model only if a different provider "
"is requested. " + _AGENT_DESIGN_GUIDE
diff --git a/jarvis/core/config.py b/jarvis/core/config.py
index 97f7c9e457..85904c2d4a 100644
--- a/jarvis/core/config.py
+++ b/jarvis/core/config.py
@@ -2609,6 +2609,56 @@ class AppshotConfig(BaseModel):
effect: bool = True
+class JarvisXConfig(BaseModel):
+ """Top-level ``[jarvisx]`` config — the built-in screenshot and screen recorder.
+
+ Jarvis X is a plain capture tool (``jarvis/jarvisx/``): region, window and
+ full-screen screenshots plus region / full-screen recordings, each on its
+ own global shortcut, saved to a folder and kept in a local library. It
+ never involves the assistant: no Screen Context privacy pipeline, no
+ delivery into a conversation. Every key below is read by
+ ``jarvis.jarvisx`` (AP-31).
+ """
+
+ model_config = {"extra": "allow"}
+
+ #: Master switch: off disarms every shortcut and refuses new captures.
+ #: The library of earlier captures stays browsable.
+ enabled: bool = True
+
+ #: Global shortcuts in the shared hotkey syntax; an empty string turns
+ #: that one shortcut off. The Ctrl+Shift+digit row stays clear of the OS
+ #: screenshot keys (Win+Shift+S, Cmd+Shift+3/4/5) and every other
+ #: Jarvis default.
+ hotkey_region: str = "ctrl+shift+2"
+ hotkey_window: str = "ctrl+shift+3"
+ hotkey_fullscreen: str = "ctrl+shift+1"
+ hotkey_record_region: str = "ctrl+shift+5"
+ hotkey_record_fullscreen: str = "ctrl+shift+6"
+ #: Stops a running recording (pressing its record shortcut again does too).
+ hotkey_stop_recording: str = "ctrl+shift+4"
+
+ #: ``true``: the corner thumbnail stays until it is dismissed. ``false``:
+ #: it fades after ``thumbnail_dismiss_s`` seconds.
+ thumbnail_persist: bool = False
+ #: Clamped to 1..3600 where it is read, so a hand-edited value can never
+ #: make the config unloadable.
+ thumbnail_dismiss_s: int = 30
+
+ #: Folder new captures are saved to. Empty = the user's Pictures folder
+ #: (``Pictures/Jarvis X``), or the Jarvis data folder where there is none.
+ save_dir: str = ""
+
+ #: Also put each new screenshot on the system clipboard.
+ copy_to_clipboard: bool = True
+
+ #: Shutter sound on every capture (also gated by ``[ui].sound_effects``).
+ sound: bool = True
+
+ #: Flash plus the corner thumbnail card after every capture.
+ effect: bool = True
+
+
class ComputerUseConfig(BaseModel):
"""Top-level ``[computer_use]`` config for the Computer-Use harness.
@@ -4406,6 +4456,9 @@ class JarvisConfig(BaseModel):
# Appshots: the front window as conversation context, on a shortcut or on
# request (jarvis/appshot/). Captures through Screen Context above.
appshot: AppshotConfig = Field(default_factory=AppshotConfig)
+ # Jarvis X: the built-in screenshot / screen-recording tool
+ # (jarvis/jarvisx/). Independent of Screen Context and the assistant.
+ jarvisx: JarvisXConfig = Field(default_factory=JarvisXConfig)
# Phase 5/6 — Computer-Use-POAV-Harness (ADR-0008).
computer_use: ComputerUseConfig = Field(default_factory=ComputerUseConfig)
# Low-latency local-action gate. Hidden tools only; never exposed in the
diff --git a/jarvis/core/config_writer.py b/jarvis/core/config_writer.py
index 9445eac3db..371587a557 100644
--- a/jarvis/core/config_writer.py
+++ b/jarvis/core/config_writer.py
@@ -1028,6 +1028,58 @@ def set_appshot_settings(
_atomic_write(path, out)
+#: Keys ``[jarvisx]`` accepts (``jarvis.core.config.JarvisXConfig``). The
+#: route validates types and hotkeys first; this allowlist keeps a typo from
+#: ever landing in the file.
+JARVISX_SETTING_KEYS: frozenset[str] = frozenset(
+ {
+ "enabled",
+ "hotkey_region",
+ "hotkey_window",
+ "hotkey_fullscreen",
+ "hotkey_record_region",
+ "hotkey_record_fullscreen",
+ "hotkey_stop_recording",
+ "thumbnail_persist",
+ "thumbnail_dismiss_s",
+ "save_dir",
+ "copy_to_clipboard",
+ "sound",
+ "effect",
+ }
+)
+
+
+def set_jarvisx_settings(
+ values: dict[str, str | bool | int],
+ *,
+ path: Path = DEFAULT_CONFIG_FILE,
+) -> None:
+ """Persist a validated ``[jarvisx]`` patch in one atomic replacement."""
+ unknown = set(values).difference(JARVISX_SETTING_KEYS)
+ if unknown:
+ raise ValueError(f"unknown jarvisx setting(s): {sorted(unknown)!r}")
+ if not values:
+ return
+ path = _ensure_writable_config_path(path)
+ with _WRITE_LOCK:
+ raw = path.read_text(encoding="utf-8")
+ had_bom = raw.startswith(_BOM)
+ if had_bom:
+ raw = raw[len(_BOM) :]
+ doc: TOMLDocument = tomlkit.parse(raw)
+ section = doc.get("jarvisx")
+ if section is None:
+ section = tomlkit.table()
+ doc["jarvisx"] = section
+ for key, value in values.items():
+ section[key] = value
+ out = tomlkit.dumps(doc)
+ if had_bom:
+ out = _BOM + out
+ _atomic_write(path, out)
+
+
def set_reply_language(name: str, *, path: Path = DEFAULT_CONFIG_FILE) -> None:
"""Persist the user-facing reply-language pin in ``[brain] reply_language``.
diff --git a/jarvis/core/delegation.py b/jarvis/core/delegation.py
new file mode 100644
index 0000000000..55e40b91f0
--- /dev/null
+++ b/jarvis/core/delegation.py
@@ -0,0 +1,123 @@
+"""Correlated delegation results, without a background model or provider call."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections import OrderedDict
+from contextvars import ContextVar
+from dataclasses import replace
+from uuid import NAMESPACE_URL, uuid5
+
+from jarvis.core.events import AnnouncementRequested
+from jarvis.core.protocols import current_chat_turn
+from jarvis.core.turn_language import DEFAULT_LOCALE
+
+RESULT_SOURCES = frozenset({"society.lead", "agentic_ide.readback"})
+BATCH_SOURCE = "delegation.batch"
+BATCH_WINDOW_S = 0.75
+MAX_BATCH = 4
+current_delegation_origin: ContextVar[dict[str, str] | None] = ContextVar(
+ "current_delegation_origin", default=None,
+)
+
+
+def origin_metadata(*, language: str = "") -> dict[str, str]:
+ """Capture the actual caller before a background task outlives its turn.
+
+ An internal/scheduled chat cannot promote itself into a spoken user task.
+ The metadata is supplied by the application, never a tool argument.
+ """
+ turn = current_chat_turn.get()
+ return {
+ "reply_surface": "voice" if turn is None else ("chat" if turn.direct_user else "none"),
+ "reply_session_id": turn.session_id if turn is not None else "",
+ "lang": language or DEFAULT_LOCALE,
+ }
+
+
+def bounded(text: str, limit: int = 4000) -> str:
+ text = str(text or "").strip()
+ if len(text) <= limit:
+ return text
+ half = (limit - 5) // 2
+ return text[:half] + "\n[…]\n" + text[-half:]
+
+
+def result_announcement(
+ *, source: str, request_id: str, name: str, request: str,
+ status: str, report: str, language: str = "", evidence: str = "agent_report",
+) -> AnnouncementRequested:
+ """Keep provenance, the original request and the outcome together.
+
+ The live conversation does the synthesis. Classic speech gets a short,
+ deterministic excerpt; neither path needs a separate paid composer.
+ """
+ lang = language or DEFAULT_LOCALE
+ labels = {
+ "en": {"done": "reports", "completed": "has stopped", "needs_input": "needs your input",
+ "blocked": "is blocked", "failed": "failed", "exited": "has exited"},
+ "de": { # i18n-allow: deterministic spoken fallback
+ "done": "meldet", "completed": "hat angehalten",
+ "needs_input": "braucht deine Antwort", # i18n-allow
+ "blocked": "ist blockiert", "failed": "ist fehlgeschlagen", # i18n-allow
+ "exited": "wurde beendet", # i18n-allow
+ },
+ "es": {
+ "done": "informa", "completed": "se ha detenido",
+ "needs_input": "necesita tu respuesta", "blocked": "está bloqueado",
+ "failed": "ha fallado", "exited": "ha terminado",
+ },
+ }
+ table = labels.get(lang, labels["en"])
+ label = table.get(status, table["done"])
+ excerpt = " ".join(str(report or "").split()[:45])
+ text = f"{name} {label}" + (f": {excerpt}" if excerpt else ".")
+ material = json.dumps({
+ "task_id": request_id, "agent": name, "status": status, "evidence": evidence,
+ "request": bounded(request, 1800), "report": bounded(report),
+ }, ensure_ascii=False)
+ digest = hashlib.sha256(material.encode()).hexdigest()[:16]
+ return AnnouncementRequested(
+ trace_id=uuid5(NAMESPACE_URL, f"jarvis:delegation:{request_id}"),
+ source_layer=source, kind="completion", priority="normal", language=lang,
+ text=text,
+ detail=f"request={request_id} status={status} result={digest}",
+ report=material,
+ )
+
+
+class ResultInbox:
+ """Coalesce distinct completed tasks; never wait for unfinished siblings."""
+
+ def __init__(self) -> None:
+ self.pending: OrderedDict[tuple, AnnouncementRequested] = OrderedDict()
+ self.seen: OrderedDict[tuple, None] = OrderedDict()
+
+ def add(self, event: AnnouncementRequested) -> bool:
+ key = (event.source_layer, event.trace_id, event.detail)
+ if key in self.seen or key in self.pending:
+ return False
+ self.pending[key] = event
+ self.seen[key] = None
+ while len(self.seen) > 2048:
+ self.seen.popitem(last=False)
+ return True
+
+ def take(self) -> AnnouncementRequested | None:
+ if not self.pending:
+ return None
+ events = [
+ self.pending.popitem(last=False)[1]
+ for _ in range(min(MAX_BATCH, len(self.pending)))
+ ]
+ # Allocate each report its own budget so clipping cannot erase a middle task.
+ report = "\n\n".join(
+ f"Result {i + 1}:\n{bounded(event.report or event.text, 5500 // len(events))}"
+ for i, event in enumerate(events)
+ )
+ return replace(
+ events[0], source_layer=BATCH_SOURCE,
+ text=" ".join(event.text for event in events), report=report,
+ detail="; ".join(event.detail or "" for event in events),
+ )
diff --git a/jarvis/core/events.py b/jarvis/core/events.py
index bcbb699ab5..888ecb11a0 100644
--- a/jarvis/core/events.py
+++ b/jarvis/core/events.py
@@ -996,6 +996,18 @@ class AnnouncementRequested(Event):
report: str | None = None
+@dataclass(frozen=True, slots=True)
+class DelegationResultReady(Event):
+ """A coding result addressed to one existing Jarvis text conversation."""
+
+ session_id: str = ""
+ request_id: str = ""
+ agent_name: str = ""
+ status: str = ""
+ text: str = ""
+ report: str = ""
+
+
# Mission completion — bridged from the per-mission MissionBus to drive When-Then rules
@dataclass(frozen=True, slots=True)
@@ -1266,6 +1278,48 @@ class AppshotTaken(Event):
height: int = 0
+@dataclass(frozen=True, slots=True)
+class JarvisXItemCreated(Event):
+ """Jarvis X saved a new screenshot or recording to its library.
+
+ Metadata only; the file is served by ``/api/jarvisx/items/<id>/file``.
+ """
+
+ id: str = ""
+ #: ``image`` | ``video``.
+ kind: str = ""
+ #: ``region`` | ``window`` | ``fullscreen``.
+ mode: str = ""
+
+
+@dataclass(frozen=True, slots=True)
+class JarvisXItemUpdated(Event):
+ """A library item changed (an annotated copy was saved)."""
+
+ id: str = ""
+
+
+@dataclass(frozen=True, slots=True)
+class JarvisXItemDeleted(Event):
+ """A library item and its files were deleted."""
+
+ id: str = ""
+
+
+@dataclass(frozen=True, slots=True)
+class JarvisXRecordingChanged(Event):
+ """A Jarvis X screen recording started or stopped.
+
+ Published on start and on stop only (not per second); a UI that shows a
+ running timer counts ``elapsed_s`` forward itself.
+ """
+
+ recording: bool = False
+ #: ``region`` | ``fullscreen`` while recording, ``""`` once stopped.
+ mode: str = ""
+ elapsed_s: float = 0.0
+
+
@dataclass(frozen=True, slots=True)
class ActionPlanned(Event):
"""The CU loop planner proposed the next action (before execution)."""
diff --git a/jarvis/dictation/insert.py b/jarvis/dictation/insert.py
index 1de27da96e..dd2e1e6cb2 100644
--- a/jarvis/dictation/insert.py
+++ b/jarvis/dictation/insert.py
@@ -32,6 +32,7 @@ dependency. Nothing is imported at module scope that a headless host lacks.
from __future__ import annotations
import logging
+import os
import sys
import threading
import time
@@ -39,6 +40,7 @@ from dataclasses import dataclass
from typing import Literal
log = logging.getLogger(__name__)
+_INSERT_LOCK = threading.Lock()
#: Paste chords by name. ``auto`` resolves per platform at call time.
#: Ctrl+V is wrong in many terminals (it is an interrupt or a literal there),
@@ -117,13 +119,8 @@ CUSTOM_CHORD_KEYS: dict[str, str] = {
"down": "down",
}
-#: ``paste_sent`` exists because Jarvis does not paste — it asks the foreground
-#: application to paste by sending a synthetic chord, and an application that
-#: does not bind that chord simply ignores it. There is no error and nothing to
-#: read back (see the module docstring), so for a chord we did not curate the
-#: only true statement is "the keystroke went out; what the app did with it is
-#: unknown". Reporting ``inserted`` there would be the exact lie this module
-#: exists to avoid.
+#: ``paste_sent`` means the chord was sent but its delivery was not observed.
+#: Keep the transcript available for manual recovery and never retry on silence.
InsertStatus = Literal["inserted", "paste_sent", "clipboard_only", "unavailable"]
@@ -145,7 +142,7 @@ class InsertResult:
``clipboard_only`` is a SUCCESS state, not a failure: the text is one
Ctrl+V away and the user was told so. ``paste_sent`` is the honest middle
- ground for a user-recorded chord (the keystroke went out, the outcome is
+ ground for unconfirmed delivery (the keystroke went out, the outcome is
unknown, so the transcript is left on the clipboard). Only ``unavailable``
means the text could not even be parked.
"""
@@ -236,9 +233,7 @@ def normalize_paste_chord(value: str) -> tuple[str, str]:
def paste_chord_is_curated(label: str) -> bool:
"""Is this the label of a chord we know actually means "paste"?
- The curated names (and ``auto``, which resolves to one of them) are chords
- that paste in real applications, so a sent-without-error chord may honestly
- be reported as ``inserted``. A recorded combo carries no such warrant.
+ This describes a known binding, not whether a particular field pasted.
"""
return str(label or "").strip().lower() in PASTE_CHORDS
@@ -416,13 +411,39 @@ def insert_text(
delay_ms: int = 120,
delay_after_ms: int = 120,
restore_clipboard: bool = True,
+) -> InsertResult:
+ """Deliver once; overlapping repeat requests must not replace an active offer."""
+ if not _INSERT_LOCK.acquire(blocking=False):
+ return InsertResult(
+ "unavailable",
+ "Another dictation is being pasted. This text remains in dictation history.",
+ False,
+ )
+ try:
+ return _insert_text(
+ text, method=method, paste_chord=paste_chord, delay_ms=delay_ms,
+ delay_after_ms=delay_after_ms, restore_clipboard=restore_clipboard,
+ )
+ finally:
+ _INSERT_LOCK.release()
+
+
+def _insert_text(
+ text: str,
+ *,
+ method: str,
+ paste_chord: str,
+ delay_ms: int,
+ delay_after_ms: int,
+ restore_clipboard: bool,
) -> InsertResult:
"""Insert ``text`` into the focused field. Never raises.
- The clipboard route (default) writes the text, sends the paste chord and
- puts the previous clipboard content back. The ``type`` route synthesises
- the characters instead — correct for the rare control that ignores paste,
- but slow and easily mangled by editor autocomplete, so it is opt-in.
+ The clipboard route writes text before sending one paste chord. Windows
+ restores the previous text only after the target requests the offer;
+ unconfirmed delivery keeps the transcript available for manual paste.
+ The ``type`` route synthesises characters instead; it remains opt-in
+ because character delivery is slow and can trigger editor autocomplete.
Either way the text is written to the clipboard FIRST, so every failure
path below degrades to "it is one Ctrl+V away" rather than to silence.
@@ -441,6 +462,7 @@ def insert_text(
)
from jarvis.platform import clipboard
+ target = _foreground_target()
# 1. Remember what was there. ``None`` means the clipboard is unreachable
# (not that it was empty) — restoring on that would CLEAR it, so the
@@ -508,6 +530,9 @@ def insert_text(
)
if method == "type":
+ blocked = _input_block_reason(target)
+ if blocked:
+ return InsertResult("clipboard_only" if parked else "unavailable", blocked, parked)
try:
actuator.type_text(text)
except Exception as exc: # noqa: BLE001
@@ -520,11 +545,15 @@ def insert_text(
),
clipboard_holds_text=parked,
)
- restored = _restore(clipboard, previous) if restore_clipboard else False
+ restored = (
+ _restore(clipboard, previous)
+ if restore_clipboard and _clipboard_still_holds(clipboard, text)
+ else False
+ )
return InsertResult(
status="inserted",
detail="",
- clipboard_holds_text=not restored,
+ clipboard_holds_text=parked and not restored,
method="type",
clipboard_restored=restored,
)
@@ -537,6 +566,7 @@ def insert_text(
delay_ms=delay_ms,
delay_after_ms=delay_after_ms,
previous=previous if restore_clipboard else None,
+ target=target,
)
if verified is not None:
return verified
@@ -546,6 +576,9 @@ def insert_text(
# Load-bearing: without it the target app can still be holding the
# PREVIOUS clipboard content when the chord arrives, and pastes that.
time.sleep(delay_ms / 1000.0)
+ blocked = _input_block_reason(target)
+ if blocked:
+ return InsertResult("clipboard_only", blocked, parked)
try:
actuator.key_combo(chord)
except Exception as exc: # noqa: BLE001
@@ -563,10 +596,9 @@ def insert_text(
# snatches the text away before the target app has read it.
time.sleep(delay_after_ms / 1000.0)
- if not paste_chord_is_curated(chord_name):
- # A recorded chord. It was sent without error, but "sent" is not
- # "pasted": an application that does not bind this combination simply
- # ignores it, and there is nothing to read back (module docstring).
+ if os.name == "nt" or not paste_chord_is_curated(chord_name):
+ # Without Windows rendering evidence, or with an arbitrary recorded
+ # chord, sending input does not establish that the field pasted it.
# So the clipboard is deliberately NOT restored — putting the previous
# content back here would delete the one copy the user can still reach
# if the chord landed nowhere.
@@ -582,7 +614,11 @@ def insert_text(
clipboard_restored=False,
)
- restored = _restore(clipboard, previous) if restore_clipboard else False
+ restored = (
+ _restore(clipboard, previous)
+ if restore_clipboard and _clipboard_still_holds(clipboard, text)
+ else False
+ )
return InsertResult(
status="inserted",
detail="",
@@ -592,125 +628,85 @@ def insert_text(
)
-#: Chords tried, in order, when the configured one is not answered by a
-#: clipboard read. Windows only — each of them means "paste" somewhere real:
-#: Ctrl+V nearly everywhere, Ctrl+Shift+V in terminals and "paste as plain
-#: text" fields, Shift+Insert in the classic controls and most terminals.
-WINDOWS_CHORD_CASCADE: tuple[str, ...] = ("ctrl_v", "ctrl_shift_v", "shift_insert")
-
-#: How long the target application gets to read the clipboard after a chord
-#: before that chord is declared "not a paste here". An xterm.js paste through
-#: a Tauri/Electron IPC bridge answers in well under 100 ms; the margin covers
-#: a busy machine.
-PASTE_READ_WAIT_S: float = 0.6
-
-#: On a blind host (see ``_insert_windows_verified``) the previous clipboard
-#: content comes back after this long, in the background, and only if the
-#: clipboard still holds the dictated text. Long enough for a WebView that
-#: pastes through an async IPC bridge on a busy machine; a copy the user
-#: makes in the meantime is never overwritten.
-RESTORE_GRACE_S: float = 2.0
-
-#: Processes that read EVERY clipboard change and therefore prove nothing —
-#: remote-desktop clipboard sync and the Windows clipboard-history service.
-#: Measured: ``msrdc.exe`` reads within 5 ms of every write (2026-08-24).
-_CLIPBOARD_WATCHER_EXES: frozenset[str] = frozenset(
- {"msrdc.exe", "mstsc.exe", "rdpclip.exe", "svchost.exe", "vmware-tray.exe", "vmtoolsd.exe"}
-)
-
-#: What worked last time for a given foreground executable, so the second
-#: dictation into the same app skips straight to the route that landed. A
-#: value is a curated chord name or ``"type"``. Process-local on purpose: an
-#: app update can change its bindings, and a stale file would outlive it.
-_LEARNED_ROUTES: dict[str, str] = {}
+# A missing render event never proves a failed paste. Slow applications and
+# clipboard watchers can both consume the same offer without another event.
+PASTE_READ_WAIT_S: float = 0.35
def _clipboard_offer_factory():
- """The delayed-rendering clipboard offer, or ``None`` off Windows.
-
- A function rather than an import so tests can swap the factory and so a
- host without the mechanism costs one attribute check, not an import.
- """
- if sys.platform != "win32":
+ """The delayed-rendering clipboard offer, or ``None`` off Windows."""
+ if os.name != "nt" or sys.platform != "win32":
return None
try:
from jarvis.platform.clipboard_offer import ClipboardOffer, available
- except Exception: # noqa: BLE001 — a missing helper is the plain path, not an error
+ except Exception: # noqa: BLE001 - preserve the plain paste fallback
log.debug("clipboard offer unavailable", exc_info=True)
return None
return ClipboardOffer if available() else None
-def _foreground_exe() -> str:
- """Executable name of the foreground window's owner, ``""`` when unknown."""
- if sys.platform != "win32":
- return ""
+def _foreground_target() -> tuple[int, int] | None:
+ """Snapshot the foreground HWND and PID without UIA or focus changes."""
+ if os.name != "nt" or sys.platform != "win32":
+ return None
try:
- import ctypes # noqa: PLC0415 — lazy (HN-7)
+ import ctypes # noqa: PLC0415
from ctypes import wintypes # noqa: PLC0415
- from jarvis.platform.clipboard_offer import _exe_name
-
user32 = ctypes.WinDLL("user32", use_last_error=True)
+ user32.GetForegroundWindow.argtypes = []
user32.GetForegroundWindow.restype = wintypes.HWND
+ user32.GetWindowThreadProcessId.argtypes = [
+ wintypes.HWND, ctypes.POINTER(wintypes.DWORD),
+ ]
+ user32.GetWindowThreadProcessId.restype = wintypes.DWORD
hwnd = user32.GetForegroundWindow()
if not hwnd:
- return ""
+ return None
pid = wintypes.DWORD()
user32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid))
- return _exe_name(int(pid.value))
- except Exception: # noqa: BLE001 — unknown app, no learning, still a paste
- log.debug("could not read the foreground executable", exc_info=True)
- return ""
+ return (int(hwnd), int(pid.value)) if pid.value else None
+ except Exception: # noqa: BLE001 - no probe means no delivery evidence
+ log.debug("could not read the foreground target", exc_info=True)
+ return None
-def _type_with_soft_newlines(actuator: object, text: str) -> None:
- """Type *text*, sending each line break as Shift+Enter.
+def _input_block_reason(target: tuple[int, int] | None) -> str:
+ """Do not send a chord to a changed window or with physically held modifiers."""
+ if os.name != "nt" or sys.platform != "win32":
+ return ""
+ if target is not None and _foreground_target() != target:
+ return (
+ "The focused window changed. The text is on your clipboard; "
+ "paste it where you want it."
+ )
+ try:
+ import ctypes # noqa: PLC0415
+ from ctypes import wintypes # noqa: PLC0415
- A plain Enter SUBMITS in the places that refuse to paste — a coding agent's
- terminal prompt, a chat composer — so a dictated paragraph with a line
- break would go out half-written. Shift+Enter is the line break in those
- places and a harmless line break in an ordinary editor.
- """
- lines = text.replace("\r\n", "\n").replace("\r", "\n").split("\n")
- for index, line in enumerate(lines):
- if index:
- actuator.key_combo(["shift", "enter"]) # type: ignore[attr-defined]
- if line:
- actuator.type_text(line, delay_s=0.002) # type: ignore[attr-defined]
+ user32 = ctypes.WinDLL("user32", use_last_error=True)
+ user32.GetAsyncKeyState.argtypes = [ctypes.c_int]
+ user32.GetAsyncKeyState.restype = wintypes.SHORT
+ # SendInput does not reset held keys. Never manufacture key-up events
+ # for the user's keys: retain the transcript for a deliberate paste.
+ if any(user32.GetAsyncKeyState(vk) & 0x8000 for vk in (0x10, 0x11, 0x12, 0x5B, 0x5C)):
+ return (
+ "A modifier key is still held. Release it and paste the text "
+ "from your clipboard."
+ )
+ except Exception: # noqa: BLE001 - optional keyboard-state probe
+ log.debug("could not inspect modifier keys before dictation paste", exc_info=True)
+ return ""
def _clipboard_still_holds(clipboard_module: object, text: str) -> bool:
- """Is our text still what the clipboard shows? Unreadable counts as yes."""
+ """Restore only positively identified text; unreadable means do not write."""
try:
current = clipboard_module.read_text() # type: ignore[attr-defined]
- except Exception: # noqa: BLE001 — cannot check, do not clobber
- return True
- if current is None:
- return True
- return current.replace("\r\n", "\n") == text.replace("\r\n", "\n")
-
-
-def _restore_later(
- clipboard_module: object, previous: str | None, text: str, grace_s: float
-) -> None:
- """Put the previous clipboard back after *grace_s*, off the delivery path.
-
- Only if the clipboard still holds OUR text: if the user copied something
- else in the meantime, that copy is theirs and stays. This is the blind-host
- answer to a target that reads the clipboard late (an async WebView bridge
- on a busy machine): the old timer fired at 120 ms and handed such an app
- the previous content — the maintainer's 2026-08-24 report.
- """
- if previous is None:
- return
-
- def _run() -> None:
- time.sleep(grace_s)
- if _clipboard_still_holds(clipboard_module, text):
- _restore(clipboard_module, previous)
-
- threading.Thread(target=_run, name="jarvis-clipboard-restore", daemon=True).start()
+ except Exception: # noqa: BLE001 - unknown ownership must not clobber a copy
+ log.debug("clipboard ownership check failed; skipping restore", exc_info=True)
+ return False
+ return current is not None and current.replace("\r\n", "\n") == text.replace("\r\n", "\n")
def _insert_windows_verified(
@@ -722,165 +718,86 @@ def _insert_windows_verified(
delay_ms: int,
delay_after_ms: int,
previous: str | None,
+ target: tuple[int, int] | None = None,
) -> InsertResult | None:
- """Paste with evidence, and fall back to typing only when it is SAFE.
-
- Windows only; ``None`` on any other host or when the delayed-rendering
- offer cannot be made, and the caller uses the plain chord path.
-
- The text is offered with delayed rendering
- (:mod:`jarvis.platform.clipboard_offer`), which makes the first clipboard
- read visible with the reader's process name. Two hosts follow:
-
- * **Sighted** — nobody read during the settle window. A paste now shows up
- as a read by the foreground app; silence for :data:`PASTE_READ_WAIT_S`
- proves the chord was not a paste there, so the next chord of
- :data:`WINDOWS_CHORD_CASCADE` is tried and, when none is answered, the
- text is typed in — every focused field accepts keystrokes. The route
- that landed is remembered per executable for this process.
- * **Blind** — a watcher (Remote Desktop clipboard sync, clipboard history)
- read during the settle window, so the system has cached the text and a
- later paste leaves no render event. Then ONE chord goes out and nothing
- is inferred from silence: a second chord or typing on a guess would
- paste the prompt twice. What the blind host still gets is the restore
- fix — the previous clipboard comes back only after a read was seen or
- after a long grace, never on the old 120 ms timer.
+ """Send one paste request and retain the transcript unless delivery is observed.
+
+ Only a render requested by the captured foreground process is evidence.
+ A watcher reading, merely opening the clipboard, or no event at all says
+ nothing about whether the focused field inserted text. In particular, a
+ timeout must never emit another chord or type a duplicate transcript.
"""
offer_cls = _clipboard_offer_factory()
if offer_cls is None:
return None
-
- import os # noqa: PLC0415
-
- exe = _foreground_exe()
- configured, _keys = resolve_paste_chord(paste_chord)
- order: list[str] = []
- learned = _LEARNED_ROUTES.get(exe) if exe else None
- for name in ((learned,) if learned else ()) + (configured,) + WINDOWS_CHORD_CASCADE:
- if name and name != "type" and name not in order:
- order.append(name)
- if learned == "type":
- order = []
-
- own_pid = os.getpid()
- for chord_name in order:
- offer = offer_cls(text)
- blind = False
- try:
- if not offer.start():
- log.debug("clipboard offer could not take the clipboard; plain paste path")
- return None
- time.sleep(max(delay_ms, 20) / 1000.0)
- # Whoever read during the settle window is a watcher, not the
- # paste — and it has just consumed the one render there is.
- exclude = {own_pid}
- for early in offer.reads():
- exclude.add(early.pid)
- if early.observed == "render":
- blind = True
- sent_at = offer.elapsed()
- _name, keys = resolve_paste_chord(chord_name)
- try:
- actuator.key_combo(keys) # type: ignore[attr-defined]
- except Exception as exc: # noqa: BLE001
- log.warning("paste chord %s failed: %s", chord_name, exc)
- return InsertResult(
- status="clipboard_only",
- detail=(
- "The paste shortcut could not be sent. The text is on your "
- "clipboard — press Ctrl+V."
- ),
- clipboard_holds_text=True,
- )
- read = offer.wait_for_read(
- exclude_pids=exclude, after_s=sent_at, timeout_s=PASTE_READ_WAIT_S
+ chord_name, keys = resolve_paste_chord(paste_chord)
+ offer = offer_cls(text)
+ sent = False
+ read = None
+ restore_text = None
+ lost_ownership = False
+ try:
+ if not offer.start():
+ return None
+ if delay_ms > 0:
+ time.sleep(delay_ms / 1000.0)
+ blocked = _input_block_reason(target)
+ if blocked or offer.lost_ownership:
+ return InsertResult(
+ "clipboard_only" if not offer.lost_ownership else "unavailable",
+ blocked or (
+ "The clipboard changed before pasting. "
+ "Recover the text from dictation history."
+ ),
+ not offer.lost_ownership,
)
- while read is not None and read.exe in _CLIPBOARD_WATCHER_EXES:
- # A slow watcher answering after the chord: from here on the
- # host is blind for the same reason as above.
- exclude.add(read.pid)
- if read.observed == "render":
- blind = True
- read = offer.wait_for_read(
- exclude_pids=exclude, after_s=sent_at, timeout_s=PASTE_READ_WAIT_S
- )
- finally:
- # Renders the text for real: the clipboard keeps it either way.
- offer.stop()
-
- if read is not None:
- log.info(
- "dictation paste: %s read the clipboard %.0f ms after %s (%s)",
- read.exe or f"pid {read.pid}",
- (read.at - sent_at) * 1000.0,
- chord_name,
- read.observed,
+ early_render = any(item.observed == "render" for item in offer.reads())
+ sent_at = offer.elapsed()
+ # Mark before sending: an exception may follow partial input. Never
+ # fall back to another chord after input could have reached the app.
+ sent = True
+ actuator.key_combo(keys) # type: ignore[attr-defined]
+ if not early_render:
+ candidate = offer.wait_for_read(
+ exclude_pids={os.getpid()}, after_s=sent_at, timeout_s=PASTE_READ_WAIT_S,
)
- if exe:
- _LEARNED_ROUTES[exe] = chord_name
+ if (
+ candidate is not None
+ and candidate.observed == "render"
+ and target is not None
+ and candidate.pid == target[1]
+ ):
+ read = candidate
+ lost_ownership = offer.lost_ownership
+ if read is not None and not lost_ownership:
if delay_after_ms > 0:
time.sleep(delay_after_ms / 1000.0)
- restored = _restore(clipboard_module, previous)
- return InsertResult(
- status="inserted",
- detail="",
- clipboard_holds_text=not restored,
- method=f"clipboard+{chord_name}",
- clipboard_restored=restored,
- )
-
- if blind:
- log.info(
- "dictation paste: %s sent to %s; a clipboard watcher makes this host "
- "blind, so the paste is assumed and the clipboard is restored after "
- "%.1f s at the earliest",
- chord_name,
- exe or "the foreground app",
- RESTORE_GRACE_S,
- )
- _restore_later(clipboard_module, previous, text, RESTORE_GRACE_S)
- return InsertResult(
- status="inserted" if paste_chord_is_curated(chord_name) else "paste_sent",
- detail="",
- clipboard_holds_text=True,
- method=f"clipboard+{chord_name}",
- clipboard_restored=False,
- )
-
- log.info(
- "dictation paste: %s was not answered by a clipboard read in %s; trying the next route",
- chord_name,
- exe or "the foreground app",
- )
+ restore_text = previous
+ except Exception: # noqa: BLE001 - preserve the transcript on a failed probe/send
+ log.warning("dictation paste could not be confirmed", exc_info=True)
+ if not sent:
+ return None
+ finally:
+ offer.stop(restore_text=restore_text)
+ lost_ownership = offer.lost_ownership
- # Sighted host, and no chord was a paste in this application: type it in.
- # The clipboard still holds the text (the offers rendered it) and stays
- # that way — if the keystrokes land nowhere, that copy is the way back.
- try:
- _type_with_soft_newlines(actuator, text)
- except Exception as exc: # noqa: BLE001
- log.warning("synthetic typing failed: %s", exc)
+ if read is not None:
+ restored = offer.restored
+ log.info("dictation paste: target requested text after %s", chord_name)
return InsertResult(
- status="clipboard_only",
- detail=(
- "This app does not paste on any known shortcut and typing into it "
- "did not work. The text is on your clipboard."
- ),
- clipboard_holds_text=True,
+ "inserted", "", not restored and not lost_ownership,
+ f"clipboard+{chord_name}", restored,
)
- if exe:
- _LEARNED_ROUTES[exe] = "type"
- log.info(
- "dictation paste: no paste shortcut is answered in %s; typed %d chars",
- exe or "the foreground app",
- len(text),
- )
+
+ log.info("dictation paste: %s sent; delivery unconfirmed, retaining transcript", chord_name)
return InsertResult(
- status="inserted",
- detail="",
- clipboard_holds_text=True,
- method="type",
- clipboard_restored=False,
+ "paste_sent",
+ (
+ "The paste shortcut was sent. If no text appeared, paste it from "
+ "your clipboard or dictation history."
+ ),
+ not lost_ownership,
+ f"clipboard+{chord_name}",
)
diff --git a/jarvis/dictation/outcomes.py b/jarvis/dictation/outcomes.py
index b2e6386098..df42ecddb0 100644
--- a/jarvis/dictation/outcomes.py
+++ b/jarvis/dictation/outcomes.py
@@ -17,13 +17,13 @@ What each value means
``inserted``
The text was typed or pasted into the focused window. The happy path.
``paste_sent``
- A user-recorded paste shortcut was sent to the focused window and no error
+ A paste shortcut was sent to the focused window and no error
came back — but Jarvis does not paste, it asks the application in front to
paste, and an application that does not bind that shortcut simply ignores
it. There is nothing to read back, so this is as far as honesty reaches:
the keystroke went out, the result is unknown, and the text is left on the
- clipboard instead of being restored away. Only ever produced for a custom
- chord; the curated ones report ``inserted``.
+ clipboard instead of being restored away. This includes curated Windows
+ shortcuts when clipboard consumption cannot be attributed to the target.
``clipboard_only``
Insertion was not possible, so the text was left on the clipboard. The
user still has their words; they just have to paste them.
diff --git a/jarvis/jarvisx/__init__.py b/jarvis/jarvisx/__init__.py
new file mode 100644
index 0000000000..7cac05d877
--- /dev/null
+++ b/jarvis/jarvisx/__init__.py
@@ -0,0 +1,24 @@
+"""Jarvis X — the built-in screenshot and screen-recording tool.
+
+A plain capture tool in the spirit of the macOS screenshot utilities: region,
+window and full-screen screenshots plus region / full-screen recordings, each
+on its own global shortcut. Captures are saved to a folder, indexed in a small
+local library, optionally copied to the clipboard, and announced with the same
+flash-and-fly-into-the-corner card the appshots use. Clicking the card opens
+the annotation editor window.
+
+Jarvis X never involves the assistant: no Screen Context privacy pipeline, no
+delivery into a conversation. Nothing here is imported at boot (AP-26); the
+shortcuts are armed after the app is ready. See ``docs/jarvisx.md``.
+
+Modules:
+
+- ``paths`` / ``store`` — the save folder and the item index.
+- ``geometry`` — pure coordinate math (HiDPI mapping, card layout, timing).
+- ``capture`` — region / window / monitor grabs (mss, native window capture).
+- ``recorder`` — the screen recorder and its encoder probe (PyAV).
+- ``clipboard`` — put an image on the system clipboard.
+- ``overlay`` — the PySide6 sidecar: selection overlay, cards, recording pill.
+- ``service`` — the one orchestrator every trigger goes through.
+- ``hotkeys`` — the global shortcuts, re-armed live on a settings change.
+"""
diff --git a/jarvis/jarvisx/capture.py b/jarvis/jarvisx/capture.py
new file mode 100644
index 0000000000..10553f7171
--- /dev/null
+++ b/jarvis/jarvisx/capture.py
@@ -0,0 +1,259 @@
+"""Screen grabs for Jarvis X: a rectangle, a monitor, the front window.
+
+Reuses the shared platform seams — ``mss`` for pixels, the Screen Context
+cursor/window probes for *where* — but none of Screen Context's privacy
+machinery: this is the user's own screenshot tool, and what they point it at
+is what they get. Every function runs on a worker thread (blocking native
+calls) and raises :class:`CaptureError` with a user-facing English message
+when no pixels can be produced.
+
+No platform package at module scope (``mss``, ``PIL`` and ctypes load lazily),
+so ``import jarvis.jarvisx.capture`` stays clean on a headless base install.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass
+
+from jarvis.jarvisx.geometry import Rect
+
+log = logging.getLogger(__name__)
+
+
+class CaptureError(RuntimeError):
+ """No pixels could be produced; the message is shown to the user."""
+
+
+@dataclass(frozen=True, slots=True)
+class Frame:
+ """Raw RGB pixels plus where they came from (capture space)."""
+
+ width: int
+ height: int
+ rgb: bytes
+ bbox: Rect
+
+
+@contextmanager
+def input_space() -> Iterator[None]:
+ """Pin native geometry and capture to one coordinate convention.
+
+ The same pinning Screen Context uses: per-monitor DPI awareness on
+ Windows, so monitor rects, window rects and ``mss`` all speak physical
+ pixels; a no-op on macOS and X11.
+ """
+ try:
+ from jarvis.core.win32_dpi import ensure_dpi_awareness # noqa: PLC0415
+
+ ensure_dpi_awareness()
+ except Exception: # noqa: BLE001 - non-Windows and minimal hosts are valid
+ log.debug("DPI-awareness probe unavailable", exc_info=True)
+ try:
+ from jarvis.cu.geometry import input_space as cu_input_space # noqa: PLC0415
+ except Exception: # noqa: BLE001 - base/headless install
+ log.debug("input-space pinning unavailable", exc_info=True)
+ yield
+ return
+ with cu_input_space():
+ yield
+
+
+def capability() -> tuple[bool, str]:
+ """Whether this host can take a screenshot at all, and why not."""
+ import importlib.util # noqa: PLC0415
+
+ try:
+ from jarvis.platform.probes import display_present, is_wayland # noqa: PLC0415
+
+ if not display_present():
+ return False, (
+ "There is no screen on this computer (headless), so nothing can be captured."
+ )
+ if is_wayland():
+ return False, (
+ "Screen capture is not available in a Wayland session. "
+ "Log in to an X11 session to use Jarvis X."
+ )
+ except Exception: # noqa: BLE001 - probes missing: let the grab decide
+ log.debug("jarvisx: platform probes unavailable", exc_info=True)
+ if importlib.util.find_spec("mss") is None:
+ return False, (
+ "Screen capture needs the 'mss' package (part of the desktop install: "
+ "pip install personal-jarvis[desktop])."
+ )
+ return True, ""
+
+
+def permission_problem() -> str:
+ """macOS Screen Recording permission (and similar), as a message or ``""``."""
+ try:
+ from jarvis.screen_context.ports import capture_permission_error # noqa: PLC0415
+
+ issue = capture_permission_error()
+ except Exception: # noqa: BLE001 - an unavailable probe must not block capture
+ log.debug("jarvisx: permission probe failed", exc_info=True)
+ return ""
+ return issue.message if issue is not None else ""
+
+
+def list_monitors() -> list[dict]:
+ """mss-shaped monitor list (``[0]`` virtual desktop), ``[]`` when none."""
+ try:
+ import mss # type: ignore[import-not-found] # noqa: PLC0415
+
+ with input_space(), mss.mss() as sct:
+ return [dict(m) for m in sct.monitors]
+ except Exception: # noqa: BLE001 - no display / no mss
+ log.debug("jarvisx: monitor enumeration failed", exc_info=True)
+ return []
+
+
+def cursor_position() -> tuple[int, int] | None:
+ try:
+ from jarvis.screen_context.ports import PlatformCursorLocator # noqa: PLC0415
+
+ return PlatformCursorLocator().position()
+ except Exception: # noqa: BLE001 - cursor unreadable: caller falls back
+ log.debug("jarvisx: cursor probe failed", exc_info=True)
+ return None
+
+
+def monitor_under_cursor(monitors: list[dict]) -> dict | None:
+ """The monitor holding the mouse pointer, else the primary one."""
+ physical = monitors[1:] if len(monitors) > 1 else monitors
+ if not physical:
+ return None
+ point = cursor_position()
+ if point is not None:
+ from jarvis.screen_context.targeting import monitor_for_point # noqa: PLC0415
+
+ found = monitor_for_point(monitors, point)
+ if found is not None:
+ return found
+ try:
+ from jarvis.platform.monitors import resolve_primary_monitor # noqa: PLC0415
+
+ return resolve_primary_monitor(monitors)
+ except Exception: # noqa: BLE001 - any screen beats no screen
+ log.debug("jarvisx: primary monitor resolution failed", exc_info=True)
+ return physical[0]
+
+
+def grab_rect(bbox: Rect) -> Frame:
+ """Grab ``bbox`` (capture space) from the desktop."""
+ left, top, width, height = (int(v) for v in bbox)
+ if width <= 0 or height <= 0:
+ raise CaptureError("The area to capture has no size.")
+ try:
+ import mss # type: ignore[import-not-found] # noqa: PLC0415
+ except ImportError as exc:
+ raise CaptureError(
+ "Screen capture needs the 'mss' package, which is not installed."
+ ) from exc
+ try:
+ with input_space(), mss.mss() as sct:
+ raw = sct.grab({"left": left, "top": top, "width": width, "height": height})
+ return Frame(int(raw.size[0]), int(raw.size[1]), bytes(raw.rgb), (left, top, width, height))
+ except Exception as exc: # noqa: BLE001 - GDI/X/Quartz errors vary widely
+ raise CaptureError(
+ f"The screen could not be captured right now ({exc}). A locked screen, "
+ "a sleeping display or a resolution change are the usual causes."
+ ) from exc
+
+
+def front_window() -> tuple[Rect, int | None, str]:
+ """The foreground window's frame rect, native handle and title.
+
+ Jarvis X captures the *foreground* window rather than the one under the
+ pointer: the shortcut is pressed while working in a window, so the
+ focused window is the one meant, and "under the pointer" picks the wrong
+ window whenever the mouse rests over a neighbour.
+ """
+ from jarvis.screen_context.ports import PlatformWindowProbe # noqa: PLC0415
+
+ snapshot = PlatformWindowProbe().foreground_snapshot()
+ if snapshot is None or snapshot.facts.frame_rect is None:
+ raise CaptureError("No focused window could be found to capture.")
+ left, top, width, height = (int(v) for v in snapshot.facts.frame_rect)
+ if width < 16 or height < 16:
+ raise CaptureError("The focused window is minimized or too small to capture.")
+ return (left, top, width, height), snapshot.handle, snapshot.facts.title
+
+
+def grab_window(bbox: Rect, handle: int | None) -> Frame:
+ """The window's own pixels where the OS offers that, else its rectangle."""
+ if handle is not None:
+ try:
+ from jarvis.platform.window_capture import grab_window as native_grab # noqa: PLC0415
+
+ left, top, width, height = bbox
+ with input_space():
+ native = native_grab(
+ int(handle), {"left": left, "top": top, "width": width, "height": height}
+ )
+ if native is not None:
+ (w, h), rgb = native
+ return Frame(int(w), int(h), rgb, bbox)
+ except Exception: # noqa: BLE001 - the rectangle grab below still works
+ log.debug("jarvisx: native window capture failed, using the rectangle", exc_info=True)
+ return grab_rect(clip_to_desktop(bbox, list_monitors()))
+
+
+def clip_to_desktop(bbox: Rect, monitors: list[dict]) -> Rect:
+ """Clip a window rect to the virtual desktop (a window may hang off-screen)."""
+ if not monitors:
+ return bbox
+ desk = monitors[0]
+ dl, dt = int(desk.get("left", 0)), int(desk.get("top", 0))
+ dr, db = dl + int(desk.get("width", 0)), dt + int(desk.get("height", 0))
+ left, top, width, height = bbox
+ x0, y0 = max(left, dl), max(top, dt)
+ x1, y1 = min(left + width, dr), min(top + height, db)
+ if x1 <= x0 or y1 <= y0:
+ raise CaptureError("The focused window is not on any screen.")
+ return (x0, y0, x1 - x0, y1 - y0)
+
+
+def encode_png(frame: Frame) -> bytes:
+ import io # noqa: PLC0415
+
+ from PIL import Image # noqa: PLC0415
+
+ image = Image.frombytes("RGB", (frame.width, frame.height), frame.rgb)
+ buffer = io.BytesIO()
+ image.save(buffer, format="PNG", optimize=False, compress_level=6)
+ return buffer.getvalue()
+
+
+def thumbnail_jpeg(width: int, height: int, rgb: bytes, *, edge: int = 560) -> bytes:
+ """A small JPEG preview (library tiles and the corner card)."""
+ import io # noqa: PLC0415
+
+ from PIL import Image # noqa: PLC0415
+
+ image = Image.frombytes("RGB", (width, height), rgb)
+ image.thumbnail((edge, edge), Image.Resampling.BILINEAR)
+ buffer = io.BytesIO()
+ image.save(buffer, format="JPEG", quality=84)
+ return buffer.getvalue()
+
+
+__all__ = [
+ "CaptureError",
+ "Frame",
+ "capability",
+ "clip_to_desktop",
+ "cursor_position",
+ "encode_png",
+ "front_window",
+ "grab_rect",
+ "grab_window",
+ "input_space",
+ "list_monitors",
+ "monitor_under_cursor",
+ "permission_problem",
+ "thumbnail_jpeg",
+]
diff --git a/jarvis/jarvisx/clipboard.py b/jarvis/jarvisx/clipboard.py
new file mode 100644
index 0000000000..34179573b8
--- /dev/null
+++ b/jarvis/jarvisx/clipboard.py
@@ -0,0 +1,178 @@
+"""Put a PNG image on the system clipboard (Windows, macOS, Linux).
+
+The app's WebView cannot be trusted with ``navigator.clipboard`` for images,
+so the backend does it natively:
+
+- **Windows** — ``CF_DIB`` (every app pastes it) plus the registered ``PNG``
+ format (keeps transparency for apps that prefer it), via ctypes.
+- **macOS** — ``osascript`` with the PNG as ``«class PNGf»``; no extra
+ package needed.
+- **Linux** — ``wl-copy`` on Wayland, else ``xclip``; without either the copy
+ degrades to a clear message.
+
+Every function returns ``(ok, message)``, never raises, and logs why a copy
+did not happen. Only stdlib at module scope; Pillow loads lazily.
+"""
+
+from __future__ import annotations
+
+import ctypes
+import io
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS
+
+log = logging.getLogger(__name__)
+
+_TIMEOUT_S = 5.0
+_RETRIES = 10
+
+
+def copy_png(png: bytes) -> tuple[bool, str]:
+ """Replace the clipboard with this PNG image."""
+ try:
+ from jarvis.platform.probes import display_present # noqa: PLC0415
+
+ if not display_present():
+ return False, "There is no desktop on this computer, so there is no clipboard."
+ except Exception: # noqa: BLE001 - probe missing: try the copy anyway
+ log.debug("jarvisx: display probe failed", exc_info=True)
+ if sys.platform == "win32":
+ return _copy_windows(png)
+ if sys.platform == "darwin":
+ return _copy_macos(png)
+ return _copy_linux(png)
+
+
+def _run(command: list[str], data: bytes | None = None) -> tuple[bool, str]:
+ try:
+ completed = subprocess.run( # noqa: S603 - fixed OS clipboard command
+ command,
+ input=data,
+ capture_output=True,
+ timeout=_TIMEOUT_S,
+ check=False,
+ close_fds=True,
+ creationflags=NO_WINDOW_CREATIONFLAGS,
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ log.warning("jarvisx: clipboard command %s failed to run (%s)", command[0], exc)
+ return False, "The clipboard tool could not be started."
+ if completed.returncode != 0:
+ detail = completed.stderr.decode("utf-8", errors="replace").strip()[:300]
+ log.warning(
+ "jarvisx: clipboard command %s exited %d %s", command[0], completed.returncode, detail
+ )
+ return False, "The clipboard refused the image."
+ return True, "Copied to the clipboard."
+
+
+def _copy_linux(png: bytes) -> tuple[bool, str]:
+ if os.environ.get("WAYLAND_DISPLAY") and shutil.which("wl-copy"):
+ return _run([str(shutil.which("wl-copy")), "--type", "image/png"], png)
+ xclip = shutil.which("xclip")
+ if xclip:
+ return _run([xclip, "-selection", "clipboard", "-t", "image/png", "-i"], png)
+ log.info("jarvisx: no wl-copy or xclip; image copy unavailable")
+ return False, "Copying images needs 'xclip' (X11) or 'wl-clipboard' (Wayland) installed."
+
+
+def _copy_macos(png: bytes) -> tuple[bool, str]:
+ handle, name = tempfile.mkstemp(prefix="jarvisx-", suffix=".png")
+ path = Path(name)
+ try:
+ with os.fdopen(handle, "wb") as out:
+ out.write(png)
+ posix = str(path).replace("\\", "\\\\").replace('"', '\\"')
+ script = f'set the clipboard to (read (POSIX file "{posix}") as «class PNGf»)'
+ return _run(["/usr/bin/osascript", "-e", script])
+ finally:
+ path.unlink(missing_ok=True)
+
+
+def _dib_bytes(png: bytes) -> bytes:
+ """The PNG as a packed DIB (a BMP without its 14-byte file header)."""
+ from PIL import Image # noqa: PLC0415
+
+ with Image.open(io.BytesIO(png)) as image:
+ rgb = image.convert("RGB")
+ buffer = io.BytesIO()
+ rgb.save(buffer, format="BMP")
+ return buffer.getvalue()[14:]
+
+
+def _copy_windows(png: bytes) -> tuple[bool, str]:
+ try:
+ dib = _dib_bytes(png)
+ except Exception as exc: # noqa: BLE001 - a broken image is reported, not raised
+ log.warning("jarvisx: image could not be prepared for the clipboard", exc_info=True)
+ return False, f"The image could not be read ({exc})."
+ try:
+ from ctypes import wintypes # noqa: PLC0415
+
+ user32 = ctypes.WinDLL("user32", use_last_error=True)
+ kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
+ user32.OpenClipboard.argtypes = [wintypes.HWND]
+ user32.OpenClipboard.restype = wintypes.BOOL
+ user32.EmptyClipboard.restype = wintypes.BOOL
+ user32.CloseClipboard.restype = wintypes.BOOL
+ user32.SetClipboardData.argtypes = [wintypes.UINT, wintypes.HANDLE]
+ user32.SetClipboardData.restype = wintypes.HANDLE
+ user32.RegisterClipboardFormatW.argtypes = [wintypes.LPCWSTR]
+ user32.RegisterClipboardFormatW.restype = wintypes.UINT
+ kernel32.GlobalAlloc.argtypes = [wintypes.UINT, ctypes.c_size_t]
+ kernel32.GlobalAlloc.restype = wintypes.HGLOBAL
+ kernel32.GlobalLock.argtypes = [wintypes.HGLOBAL]
+ kernel32.GlobalLock.restype = ctypes.c_void_p
+ kernel32.GlobalUnlock.argtypes = [wintypes.HGLOBAL]
+ kernel32.GlobalUnlock.restype = wintypes.BOOL
+ kernel32.GlobalFree.argtypes = [wintypes.HGLOBAL]
+ kernel32.GlobalFree.restype = wintypes.HGLOBAL
+ except (AttributeError, OSError) as exc:
+ log.warning("jarvisx: Win32 clipboard API unavailable (%s)", exc)
+ return False, "The Windows clipboard is not reachable."
+
+ def _put(fmt: int, data: bytes) -> bool:
+ handle = kernel32.GlobalAlloc(0x0002, len(data)) # GMEM_MOVEABLE
+ if not handle:
+ return False
+ target = kernel32.GlobalLock(handle)
+ if not target:
+ kernel32.GlobalFree(handle)
+ return False
+ try:
+ ctypes.memmove(target, data, len(data))
+ finally:
+ kernel32.GlobalUnlock(handle)
+ if not user32.SetClipboardData(fmt, handle):
+ kernel32.GlobalFree(handle)
+ return False
+ return True # the clipboard owns the memory now
+
+ for _attempt in range(_RETRIES):
+ if user32.OpenClipboard(None):
+ break
+ time.sleep(0.02)
+ else:
+ return False, "Another app is holding the clipboard; try again."
+ try:
+ if not user32.EmptyClipboard():
+ return False, "The clipboard could not be cleared."
+ if not _put(8, dib): # CF_DIB
+ return False, "The clipboard refused the image."
+ png_format = user32.RegisterClipboardFormatW("PNG")
+ if png_format:
+ _put(int(png_format), png)
+ return True, "Copied to the clipboard."
+ finally:
+ user32.CloseClipboard()
+
+
+__all__ = ["copy_png"]
diff --git a/jarvis/jarvisx/geometry.py b/jarvis/jarvisx/geometry.py
new file mode 100644
index 0000000000..7a8646baa7
--- /dev/null
+++ b/jarvis/jarvisx/geometry.py
@@ -0,0 +1,288 @@
+"""Pure coordinate and timing math for Jarvis X — no Qt, no OS calls.
+
+Two coordinate worlds meet here:
+
+- **capture space** — what ``mss`` and the native window APIs use: physical
+ pixels on Windows (the process is per-monitor DPI aware), points on macOS,
+ root-window pixels on X11. ``monitors`` below are mss-shaped dicts
+ (``[0]`` = the virtual desktop, ``[1:]`` = the physical screens).
+- **Qt space** — the overlay sidecar's logical pixels, one ``QScreen`` per
+ monitor with its own ``devicePixelRatio``.
+
+A selection never crosses between them as absolute coordinates. The overlay
+reports *which screen* (its Qt geometry and scale) and the selection as
+*fractions of that screen*; :func:`match_monitor` finds the same screen in
+capture space and :func:`fraction_to_bbox` turns the fractions back into
+capture pixels. Fractions survive every mixed-DPI layout, because a screen's
+content scales uniformly within itself even when the virtual-desktop origins
+of different screens disagree between the two worlds.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+#: A drag smaller than this (logical px, either side) is treated as a cancel.
+MIN_SELECTION_PX = 4
+
+#: At most this many thumbnail cards rest on screen; the oldest leaves first.
+MAX_CARDS = 5
+
+#: Bounds for ``[jarvisx].thumbnail_dismiss_s``.
+DISMISS_MIN_S = 1
+DISMISS_MAX_S = 3600
+
+Rect = tuple[int, int, int, int]
+FRect = tuple[float, float, float, float]
+
+
+# --------------------------------------------------------------------------
+# Selection → capture pixels
+# --------------------------------------------------------------------------
+
+
+def selection_fractions(
+ x0: float, y0: float, x1: float, y1: float, width: float, height: float
+) -> FRect | None:
+ """A drag from ``(x0, y0)`` to ``(x1, y1)`` on a ``width × height`` screen.
+
+ Returns ``(fx, fy, fw, fh)`` clamped to the screen, or ``None`` when the
+ drag is too small to be a selection (a click, a slip of the mouse).
+ Works for a drag in any direction.
+ """
+ if width <= 0 or height <= 0:
+ return None
+ left = max(0.0, min(float(x0), float(x1)))
+ top = max(0.0, min(float(y0), float(y1)))
+ right = min(float(width), max(float(x0), float(x1)))
+ bottom = min(float(height), max(float(y0), float(y1)))
+ if right - left < MIN_SELECTION_PX or bottom - top < MIN_SELECTION_PX:
+ return None
+ return (left / width, top / height, (right - left) / width, (bottom - top) / height)
+
+
+def fraction_to_bbox(monitor: dict, frac: FRect | list[float]) -> Rect:
+ """Fractions of ``monitor`` → ``(left, top, width, height)`` in capture space.
+
+ Edges are rounded independently so two adjacent selections never overlap
+ or leave a gap, and the result is clamped to the monitor.
+ """
+ ml, mt = int(monitor.get("left", 0)), int(monitor.get("top", 0))
+ mw, mh = max(1, int(monitor.get("width", 1))), max(1, int(monitor.get("height", 1)))
+ fx, fy, fw, fh = (max(0.0, min(1.0, float(v))) for v in frac)
+ x0 = ml + round(fx * mw)
+ y0 = mt + round(fy * mh)
+ x1 = ml + round(min(1.0, fx + fw) * mw)
+ y1 = mt + round(min(1.0, fy + fh) * mh)
+ return (x0, y0, max(1, x1 - x0), max(1, y1 - y0))
+
+
+def match_monitor(screen: dict, monitors: list[dict]) -> dict | None:
+ """The capture-space monitor that is the overlay's Qt ``screen``.
+
+ ``screen`` carries the Qt geometry ``x, y, w, h`` (logical) and ``dpr``.
+ Each monitor is compared with every plausible reading of the Qt geometry
+ — logical as-is (macOS points, X11 at scale 1), logical origin with
+ physical size, and fully physical (Windows per-monitor DPI) — and the
+ closest one wins. ``None`` only when there are no monitors at all.
+ """
+ physical = [m for m in (monitors[1:] if len(monitors) > 1 else monitors) if isinstance(m, dict)]
+ if not physical:
+ return None
+ return min(physical, key=lambda mon: screen_monitor_score(screen, mon))
+
+
+def screen_monitor_score(screen: dict, monitor: dict) -> float:
+ """How far a Qt screen is from a capture-space monitor (0 = identical).
+
+ Used in both directions: by the service to find the monitor of a
+ selection, and by the overlay to find the screen of a monitor.
+ """
+ x = float(screen.get("x", 0))
+ y = float(screen.get("y", 0))
+ w = float(screen.get("w", 0))
+ h = float(screen.get("h", 0))
+ dpr = float(screen.get("dpr", 1.0) or 1.0)
+ ml, mt = float(monitor.get("left", 0)), float(monitor.get("top", 0))
+ mw, mh = float(monitor.get("width", 0)), float(monitor.get("height", 0))
+ best = float("inf")
+ for rx, ry, rw, rh in (
+ (x, y, w, h),
+ (x, y, w * dpr, h * dpr),
+ (x * dpr, y * dpr, w * dpr, h * dpr),
+ ):
+ # Size mismatches weigh more than origin mismatches: origins are the
+ # part mixed-DPI layouts disagree about.
+ score = 2.0 * (abs(rw - mw) + abs(rh - mh)) + abs(rx - ml) + abs(ry - mt)
+ best = min(best, score)
+ return best
+
+
+def monitor_fraction(bbox: Rect, monitor: dict) -> FRect:
+ """``bbox`` (capture space) as fractions of ``monitor``, clipped to it."""
+ left, top, width, height = bbox
+ ml, mt = int(monitor.get("left", 0)), int(monitor.get("top", 0))
+ mw, mh = max(1, int(monitor.get("width", 1))), max(1, int(monitor.get("height", 1)))
+ x0, y0 = max(left, ml), max(top, mt)
+ x1, y1 = min(left + width, ml + mw), min(top + height, mt + mh)
+ if x1 <= x0 or y1 <= y0:
+ return (0.0, 0.0, 1.0, 1.0)
+ return ((x0 - ml) / mw, (y0 - mt) / mh, (x1 - x0) / mw, (y1 - y0) / mh)
+
+
+def monitor_rect(monitor: dict) -> list[int]:
+ return [
+ int(monitor.get("left", 0)),
+ int(monitor.get("top", 0)),
+ int(monitor.get("width", 0)),
+ int(monitor.get("height", 0)),
+ ]
+
+
+def even_size(width: int, height: int) -> tuple[int, int]:
+ """H.264 with 4:2:0 chroma needs even dimensions; trim one pixel if odd."""
+ return (max(2, int(width) - int(width) % 2), max(2, int(height) - int(height) % 2))
+
+
+# --------------------------------------------------------------------------
+# Cards
+# --------------------------------------------------------------------------
+
+
+def clamp_dismiss_s(value: object) -> int:
+ try:
+ seconds = int(float(value)) # type: ignore[arg-type]
+ except (TypeError, ValueError):
+ return 30
+ return max(DISMISS_MIN_S, min(DISMISS_MAX_S, seconds))
+
+
+def card_stack(
+ screen_w: float,
+ screen_h: float,
+ sizes: list[tuple[float, float]],
+ *,
+ margin: float = 28.0,
+ gap: float = 12.0,
+) -> list[tuple[float, float]]:
+ """Top-left corners for cards stacked up from the bottom-right corner.
+
+ ``sizes`` is ordered newest first; the newest card sits lowest, older
+ ones rise above it. A card that would leave the top of the screen is
+ placed at the top margin (the caller caps the stack at ``MAX_CARDS``).
+ """
+ corners: list[tuple[float, float]] = []
+ bottom = screen_h - margin
+ for w, h in sizes:
+ x = screen_w - margin - w
+ y = max(margin, bottom - h)
+ corners.append((x, y))
+ bottom = y - gap
+ return corners
+
+
+class CardTimer:
+ """When a resting thumbnail card fades out.
+
+ ``persist`` cards never expire. Otherwise the card expires ``dismiss_s``
+ seconds after it came to rest; while the pointer hovers it, the clock is
+ paused, and leaving restarts the full countdown — a card never vanishes
+ from under the cursor that is about to click it.
+ """
+
+ def __init__(self, *, persist: bool, dismiss_s: object, now: float) -> None:
+ self.persist = bool(persist)
+ self.dismiss_s = float(clamp_dismiss_s(dismiss_s))
+ self._deadline: float | None = None if self.persist else now + self.dismiss_s
+ self._hovered = False
+
+ def hover(self, hovered: bool, now: float) -> None:
+ if self.persist or hovered == self._hovered:
+ return
+ self._hovered = hovered
+ self._deadline = None if hovered else now + self.dismiss_s
+
+ def remaining(self, now: float) -> float | None:
+ """Seconds left, or ``None`` while it cannot expire."""
+ if self._deadline is None:
+ return None
+ return max(0.0, self._deadline - now)
+
+ def expired(self, now: float) -> bool:
+ return self._deadline is not None and now >= self._deadline
+
+
+# --------------------------------------------------------------------------
+# Recording chrome
+# --------------------------------------------------------------------------
+
+
+@dataclass(frozen=True, slots=True)
+class PillPlacement:
+ x: float
+ y: float
+ #: ``True`` when no spot outside the recorded area existed; the pill then
+ #: relies on the OS capture exclusion to stay out of the video.
+ overlaps: bool
+
+
+def pill_placement(
+ region: FRect | None,
+ screen_w: float,
+ screen_h: float,
+ pill_w: float,
+ pill_h: float,
+ *,
+ margin: float = 10.0,
+) -> PillPlacement:
+ """Where the "Stop" pill goes so it stays out of the recorded area.
+
+ ``region`` is the recorded area as screen fractions (``None`` = the whole
+ screen). Preference: centred below the region, then above it, then beside
+ it; a full-screen recording gets the bottom centre and ``overlaps=True``.
+ """
+ if region is None:
+ return PillPlacement((screen_w - pill_w) / 2.0, screen_h - pill_h - 24.0, True)
+ fx, fy, fw, fh = region
+ left, top = fx * screen_w, fy * screen_h
+ right, bottom = left + fw * screen_w, top + fh * screen_h
+ cx = min(max(margin, (left + right) / 2.0 - pill_w / 2.0), screen_w - pill_w - margin)
+ if bottom + margin + pill_h + margin <= screen_h:
+ return PillPlacement(cx, bottom + margin, False)
+ if top - margin - pill_h - margin >= 0:
+ return PillPlacement(cx, top - margin - pill_h, False)
+ cy = min(max(margin, (top + bottom) / 2.0 - pill_h / 2.0), screen_h - pill_h - margin)
+ if right + margin + pill_w + margin <= screen_w:
+ return PillPlacement(right + margin, cy, False)
+ if left - margin - pill_w - margin >= 0:
+ return PillPlacement(left - margin - pill_w, cy, False)
+ return PillPlacement((screen_w - pill_w) / 2.0, screen_h - pill_h - 24.0, True)
+
+
+def format_elapsed(seconds: float) -> str:
+ """``00:12`` / ``1:02:03`` for the recording pill."""
+ total = max(0, int(seconds))
+ hours, rest = divmod(total, 3600)
+ minutes, secs = divmod(rest, 60)
+ if hours:
+ return f"{hours}:{minutes:02d}:{secs:02d}"
+ return f"{minutes:02d}:{secs:02d}"
+
+
+__all__ = [
+ "MAX_CARDS",
+ "MIN_SELECTION_PX",
+ "CardTimer",
+ "PillPlacement",
+ "card_stack",
+ "clamp_dismiss_s",
+ "even_size",
+ "format_elapsed",
+ "fraction_to_bbox",
+ "match_monitor",
+ "monitor_fraction",
+ "monitor_rect",
+ "pill_placement",
+ "screen_monitor_score",
+ "selection_fractions",
+]
diff --git a/jarvis/jarvisx/hotkeys.py b/jarvis/jarvisx/hotkeys.py
new file mode 100644
index 0000000000..6a589813e7
--- /dev/null
+++ b/jarvis/jarvisx/hotkeys.py
@@ -0,0 +1,283 @@
+"""The Jarvis X global shortcuts: armed after boot, re-armed on a settings change.
+
+Six named shortcuts from ``[jarvisx]`` share ONE :class:`HotkeyTrigger`
+(one backend registration, one listener task). Each keeps its own status so
+the settings page can show which ones are live and why one is not: empty
+(off), invalid, the same keys as another Jarvis X shortcut, no global-hotkey
+backend on this desktop, or a second Jarvis instance that does not own the
+global shortcuts.
+
+The record shortcuts toggle: pressed while that recording runs, they stop it.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any
+
+log = logging.getLogger(__name__)
+
+#: Shortcut name -> ``[jarvisx]`` key. Order is the settings page order.
+SHORTCUT_KEYS: dict[str, str] = {
+ "region": "hotkey_region",
+ "window": "hotkey_window",
+ "fullscreen": "hotkey_fullscreen",
+ "record_region": "hotkey_record_region",
+ "record_fullscreen": "hotkey_record_fullscreen",
+ "stop_recording": "hotkey_stop_recording",
+}
+
+_EVENT_PREFIX = "jarvisx_"
+
+
+@dataclass(frozen=True, slots=True)
+class ShortcutStatus:
+ hotkey: str
+ armed: bool
+ detail: str = ""
+
+ def to_json(self) -> dict[str, Any]:
+ return {"hotkey": self.hotkey, "armed": self.armed, "detail": self.detail}
+
+
+def normalize_hotkey(value: object) -> str:
+ """Canonical spelling: lower case, no spaces, ``+``-joined."""
+ return "+".join(part.strip().lower() for part in str(value or "").split("+") if part.strip())
+
+
+def configured_hotkeys(block: Any) -> dict[str, str]:
+ return {name: normalize_hotkey(getattr(block, key, "")) for name, key in SHORTCUT_KEYS.items()}
+
+
+def check_hotkeys(hotkeys: dict[str, str]) -> dict[str, str]:
+ """Per-shortcut problem text (``""`` = fine). Empty shortcuts are fine (off)."""
+ from jarvis.trigger.hotkey import combos_collide, validate_hotkey # noqa: PLC0415
+
+ problems = {name: "" for name in hotkeys}
+ for name, combo in hotkeys.items():
+ if not combo:
+ continue
+ if combo in {"alt+alt", "left_alt+right_alt", "right_alt+left_alt"}:
+ problems[name] = "Both Alt keys together is the appshot gesture; pick another shortcut."
+ continue
+ verdict = validate_hotkey(combo)
+ if not verdict.ok:
+ problems[name] = verdict.reason or "This shortcut is not valid."
+ names = list(hotkeys)
+ for index, name in enumerate(names):
+ for other in names[:index]:
+ if problems[name] or problems[other]:
+ continue
+ if combos_collide(hotkeys[name], hotkeys[other]):
+ problems[name] = f"Uses the same keys as the {other.replace('_', ' ')} shortcut."
+ return problems
+
+
+def _default_trigger_factory(bindings: dict[str, list[str]]) -> Any:
+ from jarvis.trigger.hotkey import HotkeyTrigger # noqa: PLC0415
+
+ return HotkeyTrigger(bindings)
+
+
+def _default_has_hotkey() -> bool:
+ from jarvis.platform.probes import has_hotkey # noqa: PLC0415
+
+ return bool(has_hotkey())
+
+
+def _default_owns_shortcuts() -> bool:
+ from jarvis.core.instance import current_instance # noqa: PLC0415
+
+ return bool(current_instance().owns_ambient_duties)
+
+
+def _load_config() -> Any:
+ from jarvis.core.config import load_config # noqa: PLC0415
+
+ return load_config()
+
+
+class JarvisXShortcuts:
+ """Owns the listener for all six shortcuts."""
+
+ def __init__(
+ self,
+ on_press: Callable[[str], Any],
+ *,
+ bus: Any | None = None,
+ config_loader: Callable[[], Any] = _load_config,
+ trigger_factory: Callable[[dict[str, list[str]]], Any] = _default_trigger_factory,
+ has_hotkey: Callable[[], bool] = _default_has_hotkey,
+ owns_shortcuts: Callable[[], bool] = _default_owns_shortcuts,
+ ) -> None:
+ self._on_press = on_press
+ self._bus = bus
+ self._config_loader = config_loader
+ self._trigger_factory = trigger_factory
+ self._has_hotkey = has_hotkey
+ self._owns_shortcuts = owns_shortcuts
+ self._task: asyncio.Task[None] | None = None
+ self._subscribed = False
+ self._statuses: dict[str, ShortcutStatus] = {
+ name: ShortcutStatus("", False, "Not started yet.") for name in SHORTCUT_KEYS
+ }
+
+ @property
+ def statuses(self) -> dict[str, ShortcutStatus]:
+ return dict(self._statuses)
+
+ async def start(self) -> dict[str, ShortcutStatus]:
+ self._subscribe_reload()
+ return await self.reload()
+
+ async def reload(self) -> dict[str, ShortcutStatus]:
+ """Re-read ``[jarvisx]`` and re-arm. Never raises."""
+ await self.stop()
+ try:
+ config = await asyncio.to_thread(self._config_loader)
+ block = config.jarvisx
+ hotkeys = configured_hotkeys(block)
+ self._statuses = self._plan(bool(block.enabled), hotkeys)
+ bindings = {
+ f"{_EVENT_PREFIX}{name}": [status.hotkey]
+ for name, status in self._statuses.items()
+ if status.armed
+ }
+ if bindings:
+ self._task = asyncio.get_running_loop().create_task(
+ self._listen(bindings), name="jarvisx-hotkeys"
+ )
+ except Exception as exc: # noqa: BLE001 - a bad shortcut must not break the app
+ log.warning("jarvisx: shortcuts could not be armed", exc_info=True)
+ detail = f"The shortcuts failed to start ({type(exc).__name__})."
+ self._statuses = {name: ShortcutStatus("", False, detail) for name in SHORTCUT_KEYS}
+ armed = [f"{n}={s.hotkey}" for n, s in self._statuses.items() if s.armed]
+ log.info("jarvisx: shortcuts %s", ", ".join(armed) if armed else "none armed")
+ return self.statuses
+
+ def _plan(self, enabled: bool, hotkeys: dict[str, str]) -> dict[str, ShortcutStatus]:
+ problems = check_hotkeys(hotkeys)
+ owns = self._owns_shortcuts()
+ backend = self._has_hotkey()
+ plan: dict[str, ShortcutStatus] = {}
+ for name, combo in hotkeys.items():
+ if not combo:
+ plan[name] = ShortcutStatus("", False, "No shortcut set.")
+ elif not enabled:
+ plan[name] = ShortcutStatus(combo, False, "Jarvis X is switched off.")
+ elif problems[name]:
+ plan[name] = ShortcutStatus(combo, False, problems[name])
+ elif not owns:
+ plan[name] = ShortcutStatus(
+ combo,
+ False,
+ "The main app owns global shortcuts; this instance does not arm them.",
+ )
+ elif not backend:
+ plan[name] = ShortcutStatus(
+ combo, False, "Global shortcuts are not available on this desktop."
+ )
+ else:
+ plan[name] = ShortcutStatus(combo, True)
+ return plan
+
+ async def stop(self) -> None:
+ task, self._task = self._task, None
+ if task is not None and not task.done():
+ task.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await task
+
+ async def _listen(self, bindings: dict[str, list[str]]) -> None:
+ try:
+ trigger = self._trigger_factory(bindings)
+ async with trigger:
+ async for event in trigger.events():
+ if event.startswith(_EVENT_PREFIX):
+ self._fire(event[len(_EVENT_PREFIX) :])
+ except asyncio.CancelledError:
+ raise
+ except Exception: # noqa: BLE001 - the app keeps working without them
+ log.warning("jarvisx: shortcut listener stopped", exc_info=True)
+
+ def _fire(self, name: str) -> None:
+ try:
+ result = self._on_press(name)
+ if asyncio.iscoroutine(result):
+ task = asyncio.get_running_loop().create_task(result, name=f"jarvisx-{name}")
+ _TASKS.add(task)
+ task.add_done_callback(_TASKS.discard)
+ except Exception: # noqa: BLE001 - one press failing must not stop the listener
+ log.warning("jarvisx: shortcut %s failed", name, exc_info=True)
+
+ def _subscribe_reload(self) -> None:
+ if self._subscribed or self._bus is None or not hasattr(self._bus, "subscribe"):
+ return
+ from jarvis.core.events import ConfigReloaded # noqa: PLC0415
+
+ async def _on_reload(event: ConfigReloaded) -> None:
+ if any(key.startswith("jarvisx.") for key in event.changed_keys):
+ await self.reload()
+
+ self._bus.subscribe(ConfigReloaded, _on_reload)
+ self._subscribed = True
+
+
+_TASKS: set[asyncio.Task[Any]] = set()
+
+
+async def press(name: str) -> None:
+ """What each shortcut does."""
+ from jarvis.jarvisx.service import get_service # noqa: PLC0415
+
+ service = get_service()
+ if name in ("region", "window", "fullscreen"):
+ result = await service.capture(name) # type: ignore[arg-type]
+ elif name == "record_region":
+ result = await service.toggle_recording("region")
+ elif name == "record_fullscreen":
+ result = await service.toggle_recording("fullscreen")
+ elif name == "stop_recording":
+ if not service.recording:
+ return
+ result = await service.stop_recording()
+ else:
+ return
+ if not result.ok:
+ log.info("jarvisx: %s shortcut: %s", name, result.message)
+
+
+_shortcuts: JarvisXShortcuts | None = None
+
+
+def get_shortcuts() -> JarvisXShortcuts | None:
+ return _shortcuts
+
+
+async def start_jarvisx(bus: Any, app_state: Any | None = None) -> JarvisXShortcuts:
+ """Boot hook (scheduled after the app is ready, never on the boot path)."""
+ global _shortcuts
+ from jarvis.jarvisx.service import get_service # noqa: PLC0415
+
+ get_service().bind(bus=bus, app_state=app_state)
+ if _shortcuts is None:
+ _shortcuts = JarvisXShortcuts(press, bus=bus)
+ await _shortcuts.start()
+ return _shortcuts
+
+
+__all__ = [
+ "SHORTCUT_KEYS",
+ "JarvisXShortcuts",
+ "ShortcutStatus",
+ "check_hotkeys",
+ "configured_hotkeys",
+ "get_shortcuts",
+ "normalize_hotkey",
+ "press",
+ "start_jarvisx",
+]
diff --git a/jarvis/jarvisx/overlay/__init__.py b/jarvis/jarvisx/overlay/__init__.py
new file mode 100644
index 0000000000..9b99745e8c
--- /dev/null
+++ b/jarvis/jarvisx/overlay/__init__.py
@@ -0,0 +1,16 @@
+"""The Jarvis X overlay sidecar — everything Jarvis X draws on the desktop.
+
+- ``renderer`` + ``__main__`` — a PySide6 process (``python -m
+ jarvis.jarvisx.overlay``): the dimmed region-selection overlay, the flash
+ and fly-into-the-corner thumbnail cards (the appshot visual language, but
+ clickable and optionally persistent), and the recording border plus the
+ "Stop" pill. PySide6 is imported only inside that process.
+- ``controller`` — main-process glue: spawns the sidecar on first use, sends
+ commands, turns the sidecar's events (card clicked, stop clicked, region
+ selected) into calls on the Jarvis X service.
+- ``protocol`` — the JSON-lines vocabulary between the two.
+
+A separate process from the Computer-Use indicator on purpose: that sidecar
+is strictly click-through and command-only, while this one takes mouse input
+and reports back.
+"""
diff --git a/jarvis/jarvisx/overlay/__main__.py b/jarvis/jarvisx/overlay/__main__.py
new file mode 100644
index 0000000000..97643cd0b2
--- /dev/null
+++ b/jarvis/jarvisx/overlay/__main__.py
@@ -0,0 +1,27 @@
+"""Sidecar entry: ``python -m jarvis.jarvisx.overlay``.
+
+Guards the PySide6 import so a host without the GUI stack exits with
+``protocol.EXIT_NO_GUI`` and one English line on stderr, never a traceback.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from jarvis.jarvisx.overlay.protocol import EXIT_NO_GUI
+
+
+def main() -> int:
+ try:
+ from jarvis.jarvisx.overlay.renderer import run # noqa: PLC0415
+ except ImportError as exc:
+ sys.stderr.write(
+ f"jarvisx-overlay: PySide6 unavailable ({exc}); install the [desktop] extra "
+ "for the Jarvis X selection overlay and thumbnails.\n"
+ )
+ return EXIT_NO_GUI
+ return run()
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/jarvis/jarvisx/overlay/controller.py b/jarvis/jarvisx/overlay/controller.py
new file mode 100644
index 0000000000..8db839cac4
--- /dev/null
+++ b/jarvis/jarvisx/overlay/controller.py
@@ -0,0 +1,520 @@
+"""Main-process side of the Jarvis X overlay sidecar.
+
+Spawns ``python -m jarvis.jarvisx.overlay`` on first use, writes commands,
+waits for acknowledgements, and routes the sidecar's events (a region was
+selected, a card was clicked, the Stop pill was clicked) back to the Jarvis X
+service. The sidecar quits itself when it has been idle for a while — no card
+on screen, no selection, no recording — and is respawned on the next capture.
+
+Degradations (each logged once, never raised): no display, a Wayland session
+or a missing PySide6 mean no overlay — screenshots of a window or a screen
+still work (without the flash and card), a region selection reports the
+reason instead.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import importlib.util
+import itertools
+import logging
+import os
+import queue
+import subprocess
+import sys
+import threading
+import time
+from collections.abc import Awaitable, Callable, Iterator
+from dataclasses import dataclass
+from typing import Any
+
+from jarvis.jarvisx.overlay import protocol
+
+log = logging.getLogger(__name__)
+
+_ACK_TIMEOUT_S = 1.5
+_BLANK_ACK_TIMEOUT_S = 0.25
+#: A selection nobody finishes is abandoned after this long.
+_SELECT_TIMEOUT_S = 300.0
+#: How long an idle sidecar (nothing on screen) stays up before quitting.
+_IDLE_QUIT_S = 90.0
+_QUIT_GRACE_S = 1.5
+#: Qt start-up budget of a fresh sidecar (cold disk, first PySide6 import).
+_STARTUP_S = 8.0
+
+# Overlay copy for every supported interface language ([ui].language).
+_TEXTS: dict[str, dict[str, str]] = {
+ "en": {
+ "select_capture": "Drag to capture · Esc to cancel",
+ "select_record": "Drag the area to record · Esc to cancel",
+ "edit": "Click to edit",
+ "stop": "Stop",
+ },
+ "de": {
+ "select_capture": "Bereich aufziehen · Esc bricht ab", # i18n-allow: overlay UI
+ "select_record": "Aufnahmebereich aufziehen · Esc bricht ab", # i18n-allow: overlay UI
+ "edit": "Zum Bearbeiten klicken", # i18n-allow: localized overlay UI string
+ "stop": "Stopp", # i18n-allow: localized overlay UI string
+ },
+ "es": {
+ "select_capture": "Arrastra para capturar · Esc para cancelar", # i18n-allow: overlay UI
+ "select_record": "Arrastra el área a grabar · Esc para cancelar", # i18n-allow: UI
+ "edit": "Haz clic para editar", # i18n-allow: localized overlay UI string
+ "stop": "Detener", # i18n-allow: localized overlay UI string
+ },
+}
+
+
+def texts_for(language: str) -> dict[str, str]:
+ return dict(_TEXTS.get(str(language or "en"), _TEXTS["en"]))
+
+
+def _ui_texts() -> dict[str, str]:
+ try:
+ from jarvis.core.config import load_config # noqa: PLC0415
+
+ return texts_for(str(getattr(load_config().ui, "language", "en")))
+ except Exception: # noqa: BLE001 - English is always a correct fallback
+ log.debug("jarvisx: interface language unavailable", exc_info=True)
+ return texts_for("en")
+
+
+def overlay_capability() -> tuple[bool, str]:
+ """Whether the overlay can run here, and the reason when it cannot."""
+ try:
+ from jarvis.platform.probes import display_present, is_wayland # noqa: PLC0415
+
+ if not display_present():
+ return False, "no display on this host (headless)"
+ if is_wayland():
+ return False, "Wayland session (no always-on-top overlay surface)"
+ except Exception: # noqa: BLE001
+ return False, "platform probes unavailable"
+ if importlib.util.find_spec("PySide6") is None:
+ return False, "PySide6 is not installed (install the [desktop] extra)"
+ return True, ""
+
+
+@dataclass(frozen=True, slots=True)
+class Selection:
+ """A finished region selection: which Qt screen, which part of it."""
+
+ screen: dict[str, float]
+ rect: tuple[float, float, float, float]
+
+
+class SelectionUnavailable(RuntimeError):
+ """No selection overlay can run here; the message says why."""
+
+
+EventHandler = Callable[[str, dict[str, Any]], Awaitable[None] | None]
+
+
+class OverlayController:
+ """Owns the sidecar process and the request/response plumbing."""
+
+ def __init__(self) -> None:
+ self._proc: subprocess.Popen[str] | None = None
+ self._stdin_lock = threading.Lock()
+ self._command_lock = threading.Lock()
+ self._acks: queue.Queue[str] = queue.Queue()
+ self._loop: asyncio.AbstractEventLoop | None = None
+ self._handler: EventHandler | None = None
+ self._selections: dict[int, asyncio.Future[Selection | None]] = {}
+ self._req_ids = itertools.count(1)
+ self._warned = False
+ self._cards: set[str] = set()
+ self._recording = False
+ self._last_activity = time.monotonic()
+ self._idle_task: asyncio.Task[None] | None = None
+ self._spawned_at = 0.0
+
+ # ----------------------------------------------------------------- setup
+ def bind(self, loop: asyncio.AbstractEventLoop, handler: EventHandler) -> None:
+ """Route sidecar events (card clicks, stop clicks) to ``handler``."""
+ self._loop = loop
+ self._handler = handler
+
+ @property
+ def available(self) -> bool:
+ return overlay_capability()[0]
+
+ # ------------------------------------------------------------- commands
+ async def select_region(self, *, purpose: str) -> Selection | None:
+ """Let the user drag a rectangle. ``None`` = cancelled.
+
+ Raises :class:`SelectionUnavailable` when no overlay can run here.
+ """
+ ok, reason = overlay_capability()
+ if not ok:
+ raise SelectionUnavailable(f"A region cannot be selected here: {reason}.")
+ loop = asyncio.get_running_loop()
+ self._loop = self._loop or loop
+ await asyncio.to_thread(self._spawn)
+ if self._proc is None:
+ raise SelectionUnavailable("The selection overlay could not be started.")
+ req = next(self._req_ids)
+ future: asyncio.Future[Selection | None] = loop.create_future()
+ self._selections[req] = future
+ texts = _ui_texts()
+ hint = texts["select_record" if purpose == "record" else "select_capture"]
+ sent = await asyncio.to_thread(
+ self._send_and_wait,
+ protocol.CMD_SELECT,
+ _ACK_TIMEOUT_S,
+ req=req,
+ purpose=purpose,
+ hint=hint,
+ texts=texts,
+ )
+ if not sent:
+ self._selections.pop(req, None)
+ raise SelectionUnavailable("The selection overlay did not respond.")
+ escape = asyncio.create_task(self._escape_cancels(req), name="jarvisx-select-esc")
+ try:
+ return await asyncio.wait_for(future, timeout=_SELECT_TIMEOUT_S)
+ except TimeoutError:
+ await asyncio.to_thread(self._send_and_wait, protocol.CMD_CANCEL_SELECT, _ACK_TIMEOUT_S)
+ return None
+ finally:
+ self._selections.pop(req, None)
+ escape.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await escape
+ self._touch()
+
+ async def _escape_cancels(self, req: int) -> None:
+ """A global Escape also cancels: the overlay may not get keyboard focus.
+
+ Windows refuses to hand the foreground to a background process, so the
+ overlay's own Esc handler can be deaf; the shared hotkey backend is not.
+ """
+ try:
+ from jarvis.platform.probes import has_hotkey # noqa: PLC0415
+
+ if not has_hotkey():
+ return
+ from jarvis.trigger.hotkey import HotkeyTrigger # noqa: PLC0415
+
+ key = "escape" if sys.platform == "win32" else "esc"
+ trigger = HotkeyTrigger({"jarvisx_cancel": [key]})
+ async with trigger:
+ async for name in trigger.events():
+ if name == "jarvisx_cancel" and req in self._selections:
+ await asyncio.to_thread(
+ self._send_and_wait, protocol.CMD_CANCEL_SELECT, _ACK_TIMEOUT_S
+ )
+ return
+ except asyncio.CancelledError:
+ raise
+ except Exception: # noqa: BLE001 - the overlay's own Esc and right-click still cancel
+ log.debug("jarvisx: global Escape for the selection unavailable", exc_info=True)
+
+ async def show_card(
+ self,
+ *,
+ item_id: str,
+ monitor: list[int],
+ rect: list[float] | None,
+ thumb_b64: str,
+ persist: bool,
+ dismiss_s: int,
+ badge: str = "",
+ ) -> bool:
+ if not self.available:
+ return False
+ await asyncio.to_thread(self._spawn)
+ if self._proc is None:
+ return False
+ shown = await asyncio.to_thread(
+ self._send_and_wait,
+ protocol.CMD_CARD,
+ _ACK_TIMEOUT_S,
+ id=item_id,
+ monitor=list(monitor),
+ rect=list(rect) if rect is not None else None,
+ thumb=thumb_b64,
+ persist=bool(persist),
+ dismiss_s=int(dismiss_s),
+ badge=badge,
+ texts=_ui_texts(),
+ )
+ if shown:
+ self._cards.add(item_id)
+ self._touch()
+ return shown
+
+ async def remove_card(self, item_id: str) -> None:
+ self._cards.discard(item_id)
+ if self._running():
+ await asyncio.to_thread(
+ self._send_and_wait, protocol.CMD_REMOVE_CARD, _ACK_TIMEOUT_S, id=item_id
+ )
+
+ async def show_recording(self, *, monitor: list[int], rect: list[float] | None) -> bool:
+ if not self.available:
+ return False
+ await asyncio.to_thread(self._spawn)
+ if self._proc is None:
+ return False
+ self._recording = True
+ return await asyncio.to_thread(
+ self._send_and_wait,
+ protocol.CMD_REC_SHOW,
+ _ACK_TIMEOUT_S,
+ monitor=list(monitor),
+ rect=list(rect) if rect is not None else None,
+ texts=_ui_texts(),
+ )
+
+ async def hide_recording(self) -> None:
+ self._recording = False
+ self._touch()
+ if self._running():
+ await asyncio.to_thread(self._send_and_wait, protocol.CMD_REC_HIDE, _ACK_TIMEOUT_S)
+
+ @contextlib.contextmanager
+ def cards_hidden(self) -> Iterator[None]:
+ """Hide the resting cards around one screenshot (blocking; worker thread).
+
+ Fail-open: a dead or slow sidecar costs at most the short ack wait,
+ never the capture itself.
+ """
+ blanked = (
+ self._running()
+ and bool(self._cards)
+ and self._send_and_wait(protocol.CMD_BLANK, _BLANK_ACK_TIMEOUT_S)
+ )
+ try:
+ if blanked:
+ # One compositor frame so the hidden cards are off the glass.
+ time.sleep(0.05)
+ yield
+ finally:
+ if blanked:
+ self._send_and_wait(protocol.CMD_UNBLANK, _BLANK_ACK_TIMEOUT_S)
+
+ async def shutdown(self) -> None:
+ if self._idle_task is not None:
+ self._idle_task.cancel()
+ await asyncio.to_thread(self._quit)
+
+ # -------------------------------------------------------------- process
+ def _running(self) -> bool:
+ proc = self._proc
+ return proc is not None and proc.poll() is None
+
+ def _spawn(self) -> None:
+ if self._running():
+ return
+ try:
+ from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS # noqa: PLC0415
+
+ self._proc = subprocess.Popen(
+ [sys.executable, "-m", "jarvis.jarvisx.overlay"],
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL,
+ text=True,
+ encoding="utf-8",
+ creationflags=NO_WINDOW_CREATIONFLAGS,
+ env=os.environ.copy(),
+ )
+ except Exception: # noqa: BLE001 - reported once; captures still work
+ self._proc = None
+ if not self._warned:
+ self._warned = True
+ log.warning("jarvisx: overlay sidecar could not start", exc_info=True)
+ return
+ self._cards.clear()
+ self._spawned_at = time.monotonic()
+ threading.Thread(
+ target=self._pump, args=(self._proc,), name="jarvisx-overlay-out", daemon=True
+ ).start()
+ self._schedule_idle_quit()
+
+ def _pump(self, proc: subprocess.Popen[str]) -> None:
+ try:
+ assert proc.stdout is not None
+ for line in proc.stdout:
+ reply = protocol.decode_reply(line)
+ if reply is None:
+ continue
+ kind, payload = reply
+ if kind == "ack":
+ self._acks.put(str(payload["ok"]))
+ else:
+ self._dispatch(payload)
+ except Exception: # noqa: BLE001 - the pipe closes when the sidecar exits
+ log.debug("jarvisx: overlay output closed", exc_info=True)
+ finally:
+ self._dispatch({"event": "_exited"})
+
+ def _dispatch(self, payload: dict[str, Any]) -> None:
+ loop = self._loop
+ if loop is None or loop.is_closed():
+ return
+ with contextlib.suppress(RuntimeError):
+ loop.call_soon_threadsafe(self._on_event, payload)
+
+ def _on_event(self, payload: dict[str, Any]) -> None:
+ """Runs on the event loop."""
+ event = str(payload.get("event", ""))
+ self._touch()
+ if event == protocol.EVENT_SELECTION:
+ future = self._selections.get(int(payload.get("req", -1) or -1))
+ if future is not None and not future.done():
+ future.set_result(_selection_from(payload))
+ return
+ if event == "_exited":
+ # The sidecar died: fail every open selection as "cancelled".
+ for future in self._selections.values():
+ if not future.done():
+ future.set_result(None)
+ self._cards.clear()
+ return
+ if event in (protocol.EVENT_CARD_CLICK, protocol.EVENT_CARD_CLOSED):
+ self._cards.discard(str(payload.get("id", "")))
+ handler = self._handler
+ if handler is None:
+ return
+ try:
+ result = handler(event, payload)
+ if asyncio.iscoroutine(result):
+ task = asyncio.get_running_loop().create_task(result, name=f"jarvisx-{event}")
+ task.add_done_callback(_log_task_failure)
+ except Exception: # noqa: BLE001 - one bad handler must not stop the pump
+ log.warning("jarvisx: overlay event handler failed (%s)", event, exc_info=True)
+
+ def _send(self, cmd: str, **fields: Any) -> bool:
+ try:
+ with self._stdin_lock:
+ proc = self._proc
+ if proc is None or proc.stdin is None or proc.poll() is not None:
+ return False
+ proc.stdin.write(protocol.encode_command(cmd, **fields))
+ proc.stdin.flush()
+ return True
+ except Exception: # noqa: BLE001 - a broken pipe means the sidecar is gone
+ log.info("jarvisx: overlay pipe broke; it restarts on the next capture")
+ with self._stdin_lock:
+ self._proc = None
+ return False
+
+ def _send_and_wait(self, cmd: str, timeout_s: float, **fields: Any) -> bool:
+ with self._command_lock:
+ while True:
+ try:
+ self._acks.get_nowait()
+ except queue.Empty: # stale acks drained
+ break
+ if not self._send(cmd, **fields):
+ return False
+ # A freshly spawned sidecar first has to bring Qt up; its first
+ # commands wait in the pipe meanwhile.
+ startup_left = self._spawned_at + _STARTUP_S - time.monotonic()
+ try:
+ return self._acks.get(timeout=max(timeout_s, startup_left)) == cmd
+ except queue.Empty: # a slow sidecar is reported as "not done"
+ return False
+
+ def _quit(self) -> None:
+ if not self._running():
+ return
+ self._send_and_wait(protocol.CMD_QUIT, _ACK_TIMEOUT_S)
+ with self._stdin_lock:
+ proc, self._proc = self._proc, None
+ if proc is None:
+ return
+ try:
+ proc.wait(timeout=_QUIT_GRACE_S)
+ except Exception: # noqa: BLE001 - escalate to a kill
+ with contextlib.suppress(Exception):
+ proc.kill()
+ proc.wait(timeout=1.0)
+ for pipe in (proc.stdin, proc.stdout):
+ if pipe is not None:
+ with contextlib.suppress(Exception):
+ pipe.close()
+
+ # ----------------------------------------------------------------- idle
+ def _touch(self) -> None:
+ self._last_activity = time.monotonic()
+
+ def _idle(self) -> bool:
+ return (
+ not self._cards
+ and not self._recording
+ and not self._selections
+ and time.monotonic() - self._last_activity >= _IDLE_QUIT_S
+ )
+
+ def _schedule_idle_quit(self) -> None:
+ loop = self._loop
+ if loop is None or loop.is_closed():
+ return
+ task = self._idle_task
+ if task is not None and not task.done():
+ return
+
+ def _start() -> None:
+ self._idle_task = asyncio.get_running_loop().create_task(
+ self._idle_watch(), name="jarvisx-overlay-idle"
+ )
+
+ with contextlib.suppress(RuntimeError):
+ loop.call_soon_threadsafe(_start)
+
+ async def _idle_watch(self) -> None:
+ # Cards that time out inside the sidecar are not reported back, so
+ # the card set is only an upper bound; a persistent card keeps the
+ # sidecar alive until the user closes or clicks it.
+ while self._running():
+ await asyncio.sleep(15.0)
+ if self._idle():
+ await asyncio.to_thread(self._quit)
+ return
+
+
+def _selection_from(payload: dict[str, Any]) -> Selection | None:
+ if payload.get("cancelled"):
+ return None
+ screen = payload.get("screen")
+ rect = payload.get("rect")
+ if not isinstance(screen, dict) or not isinstance(rect, list) or len(rect) != 4:
+ return None
+ try:
+ values = tuple(float(v) for v in rect)
+ info = {k: float(screen.get(k, 0.0)) for k in ("x", "y", "w", "h", "dpr")}
+ except (TypeError, ValueError):
+ return None
+ return Selection(screen=info, rect=(values[0], values[1], values[2], values[3]))
+
+
+def _log_task_failure(task: asyncio.Task[Any]) -> None:
+ if task.cancelled():
+ return
+ exc = task.exception()
+ if exc is not None:
+ log.warning("jarvisx: overlay event handling failed", exc_info=exc)
+
+
+_controller: OverlayController | None = None
+
+
+def get_overlay() -> OverlayController:
+ global _controller
+ if _controller is None:
+ _controller = OverlayController()
+ return _controller
+
+
+__all__ = [
+ "OverlayController",
+ "Selection",
+ "SelectionUnavailable",
+ "get_overlay",
+ "overlay_capability",
+ "texts_for",
+]
diff --git a/jarvis/jarvisx/overlay/protocol.py b/jarvis/jarvisx/overlay/protocol.py
new file mode 100644
index 0000000000..efb5f38378
--- /dev/null
+++ b/jarvis/jarvisx/overlay/protocol.py
@@ -0,0 +1,135 @@
+"""JSON-lines IPC between the Jarvis X controller and its overlay sidecar.
+
+One JSON object per line on the sidecar's stdin (commands)::
+
+ {"cmd": "select", "req": 7, "purpose": "capture"|"record"}
+ # dim every screen, let the user drag a rectangle
+ {"cmd": "cancel_select"} # abort a running selection
+ {"cmd": "card", "id": "...", "monitor": [l, t, w, h], "rect": [fx, fy, fw, fh],
+ "thumb": "<base64 jpeg>", "persist": bool, "dismiss_s": int,
+ "badge": "" | "0:12"} # flash + fly into a corner card
+ {"cmd": "remove_card", "id": "..."} # the item was deleted
+ {"cmd": "blank"} / {"cmd": "unblank"} # hide cards around a grab
+ {"cmd": "rec_show", "monitor": [l, t, w, h], "rect": [fx, fy, fw, fh] | null}
+ # recording chrome: border around the area + "Stop" pill with timer
+ {"cmd": "rec_hide"}
+ {"cmd": "quit"}
+
+The sidecar acknowledges each command with ``{"ok": "<cmd>"}`` and reports
+user actions as events::
+
+ {"event": "selection", "req": 7, "screen": {"x", "y", "w", "h", "dpr"},
+ "rect": [fx, fy, fw, fh]} # or "cancelled": true
+ {"event": "card_click", "id": "..."} # open the editor
+ {"event": "card_closed", "id": "..."} # dismissed with its (x)
+ {"event": "stop_clicked"} # the recording pill was clicked
+
+It exits on stdin EOF (parent gone) even without ``quit``. Garbled lines are
+ignored on both sides.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+CMD_SELECT = "select"
+CMD_CANCEL_SELECT = "cancel_select"
+CMD_CARD = "card"
+CMD_REMOVE_CARD = "remove_card"
+CMD_BLANK = "blank"
+CMD_UNBLANK = "unblank"
+CMD_REC_SHOW = "rec_show"
+CMD_REC_HIDE = "rec_hide"
+CMD_QUIT = "quit"
+
+ALL_COMMANDS = frozenset(
+ {
+ CMD_SELECT,
+ CMD_CANCEL_SELECT,
+ CMD_CARD,
+ CMD_REMOVE_CARD,
+ CMD_BLANK,
+ CMD_UNBLANK,
+ CMD_REC_SHOW,
+ CMD_REC_HIDE,
+ CMD_QUIT,
+ }
+)
+
+EVENT_SELECTION = "selection"
+EVENT_CARD_CLICK = "card_click"
+EVENT_CARD_CLOSED = "card_closed"
+EVENT_STOP_CLICKED = "stop_clicked"
+
+ALL_EVENTS = frozenset({EVENT_SELECTION, EVENT_CARD_CLICK, EVENT_CARD_CLOSED, EVENT_STOP_CLICKED})
+
+#: Sidecar exit code when no GUI stack exists (PySide6 missing, no display).
+EXIT_NO_GUI = 3
+
+
+def encode_command(cmd: str, **fields: Any) -> str:
+ return json.dumps({"cmd": cmd, **fields}, ensure_ascii=False) + "\n"
+
+
+def decode_command(line: str) -> dict[str, Any] | None:
+ payload = _load(line)
+ if payload is None or payload.get("cmd") not in ALL_COMMANDS:
+ return None
+ return payload
+
+
+def encode_ack(cmd: str) -> str:
+ return json.dumps({"ok": cmd}) + "\n"
+
+
+def encode_event(event: str, **fields: Any) -> str:
+ return json.dumps({"event": event, **fields}, ensure_ascii=False) + "\n"
+
+
+def decode_reply(line: str) -> tuple[str, dict[str, Any]] | None:
+ """``("ack", {"ok": cmd})`` or ``("event", payload)``; ``None`` if neither."""
+ payload = _load(line)
+ if payload is None:
+ return None
+ if isinstance(payload.get("ok"), str):
+ return "ack", payload
+ if payload.get("event") in ALL_EVENTS:
+ return "event", payload
+ return None
+
+
+def _load(line: str) -> dict[str, Any] | None:
+ text = line.strip()
+ if not text:
+ return None
+ try:
+ payload = json.loads(text)
+ except (ValueError, TypeError):
+ return None
+ return payload if isinstance(payload, dict) else None
+
+
+__all__ = [
+ "ALL_COMMANDS",
+ "ALL_EVENTS",
+ "CMD_BLANK",
+ "CMD_CANCEL_SELECT",
+ "CMD_CARD",
+ "CMD_QUIT",
+ "CMD_REC_HIDE",
+ "CMD_REC_SHOW",
+ "CMD_REMOVE_CARD",
+ "CMD_SELECT",
+ "CMD_UNBLANK",
+ "EVENT_CARD_CLICK",
+ "EVENT_CARD_CLOSED",
+ "EVENT_SELECTION",
+ "EVENT_STOP_CLICKED",
+ "EXIT_NO_GUI",
+ "decode_command",
+ "decode_reply",
+ "encode_ack",
+ "encode_command",
+ "encode_event",
+]
diff --git a/jarvis/jarvisx/overlay/renderer.py b/jarvis/jarvisx/overlay/renderer.py
new file mode 100644
index 0000000000..04f915b8c4
--- /dev/null
+++ b/jarvis/jarvisx/overlay/renderer.py
@@ -0,0 +1,1086 @@
+"""PySide6 renderer for the Jarvis X overlay sidecar.
+
+Runs ONLY inside ``python -m jarvis.jarvisx.overlay``; the main process never
+imports this module (AP-26). Four surfaces, all frameless and always on top:
+
+- **Selection** — one dimmed window per screen with a crosshair; drag a
+ rectangle (live size readout in real pixels), release to confirm, Esc or
+ right-click to cancel. The selection is reported as fractions of its screen
+ (see ``jarvis.jarvisx.geometry``), so mixed-DPI layouts stay exact.
+- **Flash + fly** — the appshot shutter language: a white flash over the
+ captured area, then the picture shrinks into the bottom-right corner.
+- **Cards** — where the picture lands: a clickable thumbnail (opens the
+ editor), a close (x) on hover, draggable, stacked newest-lowest, fading
+ after the configured time unless it persists or the pointer rests on it.
+- **Recording chrome** — a thin border just outside the recorded area and a
+ "Stop" pill with a running timer, placed outside the area where possible.
+
+Cards, border and pill are excluded from screen capture where the OS allows
+(Windows display affinity, macOS window sharing); the controller also hides
+cards around its own screenshots, and the recording border sits outside the
+recorded area, so none of them ends up in a capture.
+"""
+
+from __future__ import annotations
+
+import math
+import os
+import sys
+import threading
+import time
+from collections import deque
+from collections.abc import Callable
+from contextlib import suppress
+from typing import Any
+
+from PySide6.QtCore import (
+ QByteArray,
+ QObject,
+ QPoint,
+ QPointF,
+ QRectF,
+ Qt,
+ QTimer,
+ QVariantAnimation,
+ Signal,
+ Slot,
+)
+from PySide6.QtGui import (
+ QColor,
+ QCursor,
+ QFont,
+ QFontMetricsF,
+ QGuiApplication,
+ QImage,
+ QPainter,
+ QPainterPath,
+ QPen,
+)
+from PySide6.QtWidgets import QApplication, QWidget
+
+from jarvis.jarvisx import geometry
+from jarvis.jarvisx.overlay import protocol
+
+# Jarvis gold (matches the Computer-Use indicator and the orb border).
+_GOLD = (255, 229, 0)
+_SOFT_GOLD = (231, 196, 110)
+
+_DIM_ALPHA = 105
+_FADE_MS = 260
+
+# Appshot shutter timings (see jarvis/cu/indicator/renderer.py) — the same
+# visual language, ending in a resting card instead of sliding away.
+_FLASH_MS = 200
+_FLY_START_MS = 90
+_FLY_MS = 430
+_FLY_TOTAL_MS = _FLY_START_MS + _FLY_MS
+
+_CARD_W = 300.0
+_CARD_MAX_H = 220.0
+_CARD_PAD = 14.0 # transparent margin that holds the shadow and the (x)
+_CARD_RADIUS = 12.0
+_CARD_MARGIN = 28.0
+_CARD_GAP = 10.0
+_CLOSE_R = 11.0
+_DRAG_SLOP_PX = 5
+
+
+# --------------------------------------------------------------------------
+# Helpers
+# --------------------------------------------------------------------------
+
+
+def _ease_out_cubic(x: float) -> float:
+ x = max(0.0, min(1.0, x))
+ return 1.0 - (1.0 - x) ** 3
+
+
+def _lerp(a: float, b: float, t: float) -> float:
+ return a + (b - a) * t
+
+
+def _screen_info(screen) -> dict[str, float]:
+ g = screen.geometry()
+ return {
+ "x": float(g.x()),
+ "y": float(g.y()),
+ "w": float(g.width()),
+ "h": float(g.height()),
+ "dpr": float(screen.devicePixelRatio() or 1.0),
+ }
+
+
+def _screen_for_monitor(monitor: Any):
+ """The QScreen that is the capture-space ``monitor`` ([l, t, w, h])."""
+ screens = QGuiApplication.screens()
+ primary = QGuiApplication.primaryScreen()
+ if not screens or not isinstance(monitor, list) or len(monitor) != 4:
+ return primary
+ mon = {"left": monitor[0], "top": monitor[1], "width": monitor[2], "height": monitor[3]}
+ return min(screens, key=lambda s: geometry.screen_monitor_score(_screen_info(s), mon))
+
+
+def _frac(rect: Any) -> tuple[float, float, float, float] | None:
+ if not isinstance(rect, list) or len(rect) != 4:
+ return None
+ try:
+ fx, fy, fw, fh = (max(0.0, min(1.0, float(v))) for v in rect)
+ except (TypeError, ValueError):
+ return None
+ return (fx, fy, fw, fh)
+
+
+def _base_flags(*, click_through: bool) -> Qt.WindowType:
+ flags = (
+ Qt.WindowType.FramelessWindowHint
+ | Qt.WindowType.WindowStaysOnTopHint
+ | Qt.WindowType.Tool
+ | Qt.WindowType.NoDropShadowWindowHint
+ )
+ if click_through:
+ flags |= Qt.WindowType.WindowTransparentForInput | Qt.WindowType.WindowDoesNotAcceptFocus
+ return flags
+
+
+def _prepare(widget: QWidget, *, click_through: bool, activates: bool = False) -> None:
+ widget.setWindowFlags(_base_flags(click_through=click_through))
+ widget.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground)
+ if click_through:
+ widget.setAttribute(Qt.WidgetAttribute.WA_TransparentForMouseEvents)
+ if not activates:
+ widget.setAttribute(Qt.WidgetAttribute.WA_ShowWithoutActivating)
+ if sys.platform == "darwin":
+ # Qt::Tool is an NSPanel on macOS; without this it stays invisible
+ # while this (never activated) process is in the background.
+ always_show = getattr(Qt.WidgetAttribute, "WA_MacAlwaysShowToolWindow", None)
+ if always_show is not None:
+ widget.setAttribute(always_show)
+
+
+def _exclude_from_capture(widget: QWidget) -> None:
+ """Keep chrome out of every screenshot and recording where the OS can."""
+ if os.environ.get("JARVIS_JARVISX_OVERLAY_CAPTURABLE", "").strip() in {"1", "true", "yes"}:
+ return # live verification needs to SEE the chrome in a screenshot
+ try:
+ if sys.platform == "win32":
+ from jarvis.platform.capture_exclusion import exclude_hwnd_from_capture # noqa: PLC0415
+
+ exclude_hwnd_from_capture(int(widget.winId()))
+ elif sys.platform == "darwin":
+ from jarvis.platform.capture_exclusion import exclude_macos_app_windows # noqa: PLC0415
+
+ exclude_macos_app_windows()
+ except Exception: # noqa: BLE001 - cosmetic; the controller also hides cards
+ _log("capture exclusion failed")
+
+
+def _log(message: str) -> None:
+ with suppress(Exception):
+ sys.stderr.write(f"jarvisx-overlay: {message}\n")
+ sys.stderr.flush()
+
+
+# --------------------------------------------------------------------------
+# Selection
+# --------------------------------------------------------------------------
+
+
+class _SelectWindow(QWidget):
+ """One screen's dimmed selection canvas."""
+
+ def __init__(self, screen, hint: str, owner: Renderer) -> None:
+ super().__init__(None)
+ _prepare(self, click_through=False, activates=True)
+ self.setScreen(screen)
+ self.setGeometry(screen.geometry())
+ self.setCursor(Qt.CursorShape.CrossCursor)
+ self.setMouseTracking(True)
+ self.setFocusPolicy(Qt.FocusPolicy.StrongFocus)
+ self._screen = screen
+ self._hint = hint
+ self._owner = owner
+ self._start: QPointF | None = None
+ self._end: QPointF | None = None
+ self._pointer: QPointF | None = None
+
+ @property
+ def info(self) -> dict[str, float]:
+ return _screen_info(self._screen)
+
+ def showEvent(self, event) -> None: # noqa: N802 (Qt override)
+ super().showEvent(event)
+ _exclude_from_capture(self)
+
+ def keyPressEvent(self, event) -> None: # noqa: N802
+ if event.key() == Qt.Key.Key_Escape:
+ self._owner.finish_selection(None, None)
+ return
+ super().keyPressEvent(event)
+
+ def mousePressEvent(self, event) -> None: # noqa: N802
+ if event.button() == Qt.MouseButton.RightButton:
+ self._owner.finish_selection(None, None)
+ return
+ if event.button() == Qt.MouseButton.LeftButton:
+ self._start = event.position()
+ self._end = event.position()
+ self._owner.selection_started(self)
+ self.update()
+
+ def mouseMoveEvent(self, event) -> None: # noqa: N802
+ self._pointer = event.position()
+ if self._start is not None:
+ self._end = event.position()
+ self._owner.pointer_moved(self)
+ self.update()
+
+ def mouseReleaseEvent(self, event) -> None: # noqa: N802
+ if event.button() != Qt.MouseButton.LeftButton or self._start is None:
+ return
+ end = event.position()
+ frac = geometry.selection_fractions(
+ self._start.x(), self._start.y(), end.x(), end.y(), self.width(), self.height()
+ )
+ self._start = None
+ self._owner.finish_selection(self, frac)
+
+ def clear_pointer(self) -> None:
+ if self._pointer is not None or self._start is not None:
+ self._pointer = None
+ self._start = None
+ self.update()
+
+ def _selection(self) -> QRectF | None:
+ if self._start is None or self._end is None:
+ return None
+ return QRectF(self._start, self._end).normalized()
+
+ def paintEvent(self, event) -> None: # noqa: N802
+ del event
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ full = QRectF(0, 0, self.width(), self.height())
+ sel = self._selection()
+ dim = QPainterPath()
+ dim.addRect(full)
+ if sel is not None:
+ hole = QPainterPath()
+ hole.addRect(sel)
+ dim = dim.subtracted(hole)
+ painter.fillPath(dim, QColor(0, 0, 0, _DIM_ALPHA))
+ if sel is not None:
+ pen = QPen(QColor(*_GOLD, 235))
+ pen.setWidthF(1.5)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawRect(sel)
+ self._paint_size(painter, sel)
+ elif self._pointer is not None:
+ pen = QPen(QColor(255, 255, 255, 120))
+ pen.setWidthF(1.0)
+ painter.setPen(pen)
+ p = self._pointer
+ painter.drawLine(QPointF(0, p.y()), QPointF(self.width(), p.y()))
+ painter.drawLine(QPointF(p.x(), 0), QPointF(p.x(), self.height()))
+ if self._hint and self._owner.hint_screen() is self:
+ self._paint_hint(painter)
+ painter.end()
+
+ def _paint_size(self, painter: QPainter, sel: QRectF) -> None:
+ dpr = float(self._screen.devicePixelRatio() or 1.0)
+ text = f"{round(sel.width() * dpr)} × {round(sel.height() * dpr)}"
+ font = QFont()
+ font.setPointSizeF(9.5)
+ font.setWeight(QFont.Weight.DemiBold)
+ painter.setFont(font)
+ metrics = QFontMetricsF(font)
+ w = metrics.horizontalAdvance(text) + 16.0
+ h = metrics.height() + 8.0
+ x = sel.right() - w
+ y = sel.bottom() + 6.0
+ if y + h > self.height():
+ y = sel.bottom() - h - 6.0
+ x = max(4.0, min(x, self.width() - w - 4.0))
+ box = QRectF(x, y, w, h)
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(18, 18, 18, 210))
+ painter.drawRoundedRect(box, h / 2.0, h / 2.0)
+ painter.setPen(QColor(255, 244, 205, 240))
+ painter.drawText(box, Qt.AlignmentFlag.AlignCenter, text)
+
+ def _paint_hint(self, painter: QPainter) -> None:
+ font = QFont()
+ font.setPointSizeF(10.5)
+ font.setWeight(QFont.Weight.Medium)
+ painter.setFont(font)
+ metrics = QFontMetricsF(font)
+ w = metrics.horizontalAdvance(self._hint) + 32.0
+ h = metrics.height() + 14.0
+ box = QRectF((self.width() - w) / 2.0, 24.0, w, h)
+ painter.setPen(QPen(QColor(*_SOFT_GOLD, 200), 1.0))
+ painter.setBrush(QColor(18, 18, 18, 190))
+ painter.drawRoundedRect(box, h / 2.0, h / 2.0)
+ painter.setPen(QColor(255, 240, 200, 240))
+ painter.drawText(box, Qt.AlignmentFlag.AlignCenter, self._hint)
+
+
+# --------------------------------------------------------------------------
+# Flash + fly
+# --------------------------------------------------------------------------
+
+
+class _FlyWindow(QWidget):
+ """Monitor-sized, click-through: flash the area, fly it to ``dst``."""
+
+ def __init__(
+ self,
+ screen,
+ src: QRectF,
+ dst: QRectF,
+ thumb: QImage,
+ on_landed: Callable[[_FlyWindow], None],
+ ) -> None:
+ super().__init__(None)
+ _prepare(self, click_through=True)
+ self.setScreen(screen)
+ self.setGeometry(screen.geometry())
+ self._src = src
+ self._dst = dst
+ self._thumb = thumb
+ self._on_landed: Callable[[_FlyWindow], None] | None = on_landed
+ self._t = 0.0
+ self._anim = QVariantAnimation(self)
+ self._anim.setStartValue(0.0)
+ self._anim.setEndValue(float(_FLY_TOTAL_MS))
+ self._anim.setDuration(_FLY_TOTAL_MS)
+ self._anim.valueChanged.connect(self._tick)
+ self._anim.finished.connect(self._landed)
+
+ def start(self) -> None:
+ self.show()
+ self._anim.start()
+
+ def retarget(self, dst: QRectF) -> None:
+ self._dst = dst
+
+ def finish_now(self) -> None:
+ self._anim.stop()
+ self._landed()
+
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ _exclude_from_capture(self)
+
+ def _tick(self, value) -> None:
+ self._t = float(value)
+ self.update()
+
+ def _landed(self) -> None:
+ callback, self._on_landed = self._on_landed, None
+ self.hide()
+ if callback is not None:
+ callback(self)
+ self.deleteLater()
+
+ def paintEvent(self, event) -> None: # noqa: N802
+ del event
+ t = self._t
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ painter.setRenderHint(QPainter.RenderHint.SmoothPixmapTransform)
+ fly = _ease_out_cubic((t - _FLY_START_MS) / _FLY_MS)
+ rect = QRectF(
+ _lerp(self._src.x(), self._dst.x(), fly),
+ _lerp(self._src.y(), self._dst.y(), fly),
+ _lerp(self._src.width(), self._dst.width(), fly),
+ _lerp(self._src.height(), self._dst.height(), fly),
+ )
+ radius = _CARD_RADIUS * fly
+ if not self._thumb.isNull():
+ if fly > 0.0:
+ for spread, alpha in ((10.0, 18), (5.0, 30), (2.0, 46)):
+ shadow = rect.adjusted(-spread, -spread + 4, spread, spread + 4)
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(0, 0, 0, int(alpha * fly)))
+ painter.drawRoundedRect(shadow, radius + spread, radius + spread)
+ clip = QPainterPath()
+ clip.addRoundedRect(rect, radius, radius)
+ painter.save()
+ painter.setClipPath(clip)
+ painter.drawImage(rect, self._thumb)
+ painter.restore()
+ if fly > 0.0:
+ pen = QPen(QColor(255, 255, 255, int(235 * fly)))
+ pen.setWidthF(2.5)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawRoundedRect(rect, radius, radius)
+ flash = 1.0 - _ease_out_cubic(t / _FLASH_MS)
+ if flash > 0.0:
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(255, 255, 255, int(225 * flash)))
+ painter.drawRect(self._src)
+ painter.end()
+
+
+# --------------------------------------------------------------------------
+# Cards
+# --------------------------------------------------------------------------
+
+
+class _CardWindow(QWidget):
+ """A resting, clickable thumbnail of one capture."""
+
+ def __init__(
+ self,
+ owner: Renderer,
+ item_id: str,
+ screen,
+ thumb: QImage,
+ card_size: tuple[float, float],
+ *,
+ persist: bool,
+ dismiss_s: int,
+ badge: str,
+ ) -> None:
+ super().__init__(None)
+ _prepare(self, click_through=False)
+ self.setScreen(screen)
+ self.setMouseTracking(True)
+ self.setCursor(Qt.CursorShape.PointingHandCursor)
+ self.item_id = item_id
+ self.screen_ref = screen
+ self.card_size = card_size
+ self.moved_by_user = False
+ self._owner = owner
+ self._thumb = thumb
+ self._badge = badge
+ self._hover = False
+ self._press_global: QPoint | None = None
+ self._press_origin: QPoint | None = None
+ self._dragging = False
+ self._opacity_anim: QVariantAnimation | None = None
+ self._leaving = False
+ self.timer = geometry.CardTimer(persist=persist, dismiss_s=dismiss_s, now=time.monotonic())
+ w, h = card_size
+ self.resize(int(math.ceil(w + 2 * _CARD_PAD)), int(math.ceil(h + 2 * _CARD_PAD)))
+
+ # -- placement ---------------------------------------------------------
+ def place_card_at(self, x: float, y: float) -> None:
+ """Move so the card itself (not its padding) starts at screen-local x, y."""
+ g = self.screen_ref.geometry()
+ self.move(int(g.x() + x - _CARD_PAD), int(g.y() + y - _CARD_PAD))
+
+ def card_rect_local(self) -> QRectF:
+ return QRectF(_CARD_PAD, _CARD_PAD, self.card_size[0], self.card_size[1])
+
+ def _close_center(self) -> QPointF:
+ return QPointF(_CARD_PAD + 2.0, _CARD_PAD + 2.0)
+
+ def _on_close(self, pos: QPointF) -> bool:
+ c = self._close_center()
+ return self._hover and math.hypot(pos.x() - c.x(), pos.y() - c.y()) <= _CLOSE_R + 3.0
+
+ # -- lifecycle -----------------------------------------------------------
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ _exclude_from_capture(self)
+
+ def leave(self, *, notify: bool) -> None:
+ """Fade out and remove; ``notify`` reports a user dismissal."""
+ if self._leaving:
+ return
+ self._leaving = True
+ self._owner.card_leaving(self, notify=notify)
+ anim = QVariantAnimation(self)
+ anim.setStartValue(self.windowOpacity())
+ anim.setEndValue(0.0)
+ anim.setDuration(_FADE_MS)
+ anim.valueChanged.connect(lambda v: self.setWindowOpacity(float(v)))
+ anim.finished.connect(self._gone)
+ self._opacity_anim = anim
+ anim.start()
+
+ def _gone(self) -> None:
+ self.hide()
+ self.deleteLater()
+
+ # -- input ---------------------------------------------------------------
+ def enterEvent(self, event) -> None: # noqa: N802
+ self._hover = True
+ self.timer.hover(True, time.monotonic())
+ self.update()
+ super().enterEvent(event)
+
+ def leaveEvent(self, event) -> None: # noqa: N802
+ self._hover = False
+ self.timer.hover(False, time.monotonic())
+ self.update()
+ super().leaveEvent(event)
+
+ def mousePressEvent(self, event) -> None: # noqa: N802
+ if event.button() != Qt.MouseButton.LeftButton:
+ return
+ self._press_global = event.globalPosition().toPoint()
+ self._press_origin = self.pos()
+ self._dragging = False
+
+ def mouseMoveEvent(self, event) -> None: # noqa: N802
+ self.update()
+ if self._press_global is None or self._press_origin is None:
+ return
+ delta = event.globalPosition().toPoint() - self._press_global
+ if not self._dragging and delta.manhattanLength() > _DRAG_SLOP_PX:
+ self._dragging = True
+ self.moved_by_user = True
+ if self._dragging:
+ self.move(self._press_origin + delta)
+
+ def mouseReleaseEvent(self, event) -> None: # noqa: N802
+ if event.button() != Qt.MouseButton.LeftButton or self._press_global is None:
+ return
+ dragged = self._dragging
+ self._press_global = None
+ self._press_origin = None
+ self._dragging = False
+ if dragged or self._leaving:
+ return
+ if self._on_close(event.position()):
+ self.leave(notify=True)
+ return
+ if self.card_rect_local().contains(event.position()):
+ _emit(protocol.EVENT_CARD_CLICK, id=self.item_id)
+ self.leave(notify=False)
+
+ # -- painting ------------------------------------------------------------
+ def paintEvent(self, event) -> None: # noqa: N802
+ del event
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ painter.setRenderHint(QPainter.RenderHint.SmoothPixmapTransform)
+ rect = self.card_rect_local()
+ for spread, alpha in ((10.0, 18), (5.0, 30), (2.0, 46)):
+ shadow = rect.adjusted(-spread, -spread + 4, spread, spread + 4)
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(0, 0, 0, alpha))
+ painter.drawRoundedRect(shadow, _CARD_RADIUS + spread, _CARD_RADIUS + spread)
+ clip = QPainterPath()
+ clip.addRoundedRect(rect, _CARD_RADIUS, _CARD_RADIUS)
+ painter.save()
+ painter.setClipPath(clip)
+ if self._thumb.isNull():
+ painter.fillRect(rect, QColor(40, 40, 40))
+ else:
+ painter.drawImage(rect, self._thumb)
+ if self._hover:
+ painter.fillRect(rect, QColor(0, 0, 0, 70))
+ painter.restore()
+ pen = QPen(QColor(255, 255, 255, 235))
+ pen.setWidthF(2.5)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawRoundedRect(rect, _CARD_RADIUS, _CARD_RADIUS)
+ if self._badge:
+ self._paint_badge(painter, rect)
+ if self._hover:
+ self._paint_edit_hint(painter, rect)
+ self._paint_close(painter)
+ painter.end()
+
+ def _pill(self, painter: QPainter, box: QRectF, text: str) -> None:
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(18, 18, 18, 200))
+ painter.drawRoundedRect(box, box.height() / 2.0, box.height() / 2.0)
+ painter.setPen(QColor(255, 255, 255, 240))
+ painter.drawText(box, Qt.AlignmentFlag.AlignCenter, text)
+
+ def _paint_badge(self, painter: QPainter, rect: QRectF) -> None:
+ font = QFont()
+ font.setPointSizeF(8.5)
+ font.setWeight(QFont.Weight.DemiBold)
+ painter.setFont(font)
+ text = f"▶ {self._badge}"
+ metrics = QFontMetricsF(font)
+ box = QRectF(
+ rect.left() + 8.0,
+ rect.bottom() - metrics.height() - 14.0,
+ metrics.horizontalAdvance(text) + 16.0,
+ metrics.height() + 6.0,
+ )
+ self._pill(painter, box, text)
+
+ def _paint_edit_hint(self, painter: QPainter, rect: QRectF) -> None:
+ font = QFont()
+ font.setPointSizeF(9.5)
+ font.setWeight(QFont.Weight.DemiBold)
+ painter.setFont(font)
+ text = self._owner.text("edit")
+ metrics = QFontMetricsF(font)
+ w = metrics.horizontalAdvance(text) + 22.0
+ h = metrics.height() + 10.0
+ box = QRectF(rect.center().x() - w / 2.0, rect.center().y() - h / 2.0, w, h)
+ self._pill(painter, box, text)
+
+ def _paint_close(self, painter: QPainter) -> None:
+ c = self._close_center()
+ painter.setPen(QPen(QColor(255, 255, 255, 230), 1.2))
+ painter.setBrush(QColor(28, 28, 28, 235))
+ painter.drawEllipse(c, _CLOSE_R, _CLOSE_R)
+ pen = QPen(QColor(255, 255, 255, 240))
+ pen.setWidthF(1.8)
+ pen.setCapStyle(Qt.PenCapStyle.RoundCap)
+ painter.setPen(pen)
+ d = _CLOSE_R * 0.42
+ painter.drawLine(QPointF(c.x() - d, c.y() - d), QPointF(c.x() + d, c.y() + d))
+ painter.drawLine(QPointF(c.x() - d, c.y() + d), QPointF(c.x() + d, c.y() - d))
+
+
+# --------------------------------------------------------------------------
+# Recording chrome
+# --------------------------------------------------------------------------
+
+
+class _BorderWindow(QWidget):
+ """Click-through outline drawn just OUTSIDE the recorded area."""
+
+ def __init__(self, screen, area: QRectF) -> None:
+ super().__init__(None)
+ _prepare(self, click_through=True)
+ self.setScreen(screen)
+ self.setGeometry(screen.geometry())
+ self._area = area
+
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ _exclude_from_capture(self)
+
+ def paintEvent(self, event) -> None: # noqa: N802
+ del event
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ pen = QPen(QColor(*_GOLD, 230))
+ pen.setWidthF(2.0)
+ pen.setStyle(Qt.PenStyle.DashLine)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawRect(self._area.adjusted(-3.0, -3.0, 3.0, 3.0))
+ painter.end()
+
+
+class _PillWindow(QWidget):
+ """The "Stop ● 00:12" pill; a click stops the recording."""
+
+ def __init__(self, owner: Renderer, screen, started: float) -> None:
+ super().__init__(None)
+ _prepare(self, click_through=False)
+ self.setScreen(screen)
+ self.setCursor(Qt.CursorShape.PointingHandCursor)
+ self._owner = owner
+ self._started = started
+ self._hover = False
+ font = QFont()
+ font.setPointSizeF(10.0)
+ font.setWeight(QFont.Weight.DemiBold)
+ self._font = font
+ metrics = QFontMetricsF(font)
+ sample = f"{owner.text('stop')} 00:00:00"
+ self.resize(int(metrics.horizontalAdvance(sample) + 58), int(metrics.height() + 18))
+ self._tick = QTimer(self)
+ self._tick.setInterval(250)
+ self._tick.timeout.connect(self.update)
+ self._tick.start()
+
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ _exclude_from_capture(self)
+
+ def enterEvent(self, event) -> None: # noqa: N802
+ self._hover = True
+ self.update()
+ super().enterEvent(event)
+
+ def leaveEvent(self, event) -> None: # noqa: N802
+ self._hover = False
+ self.update()
+ super().leaveEvent(event)
+
+ def mouseReleaseEvent(self, event) -> None: # noqa: N802
+ if event.button() == Qt.MouseButton.LeftButton:
+ _emit(protocol.EVENT_STOP_CLICKED)
+
+ def paintEvent(self, event) -> None: # noqa: N802
+ del event
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ box = QRectF(1.0, 1.0, self.width() - 2.0, self.height() - 2.0)
+ painter.setPen(QPen(QColor(*_SOFT_GOLD, 210), 1.0))
+ painter.setBrush(QColor(18, 18, 18, 235 if self._hover else 210))
+ painter.drawRoundedRect(box, box.height() / 2.0, box.height() / 2.0)
+ elapsed = time.monotonic() - self._started
+ pulse = 0.55 + 0.45 * (0.5 + 0.5 * math.cos(elapsed * math.pi))
+ dot = QPointF(box.left() + 18.0, box.center().y())
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(235, 64, 52, int(255 * pulse)))
+ painter.drawEllipse(dot, 5.5, 5.5)
+ painter.setFont(self._font)
+ painter.setPen(QColor(255, 244, 214, 245))
+ text = f"{self._owner.text('stop')} {geometry.format_elapsed(elapsed)}"
+ painter.drawText(
+ box.adjusted(32.0, 0.0, -12.0, 0.0),
+ Qt.AlignmentFlag.AlignVCenter | Qt.AlignmentFlag.AlignLeft,
+ text,
+ )
+ painter.end()
+
+
+# --------------------------------------------------------------------------
+# Renderer
+# --------------------------------------------------------------------------
+
+
+class Renderer(QObject):
+ """Owns every overlay surface and applies the controller's commands."""
+
+ def __init__(self, app: QApplication) -> None:
+ super().__init__()
+ self._app = app
+ self._commands: deque[dict] = deque()
+ self._draining = False
+ self._texts: dict[str, str] = {"edit": "Click to edit", "stop": "Stop"}
+ # selection
+ self._select_req: int | None = None
+ self._select_windows: list[_SelectWindow] = []
+ self._hint_window: _SelectWindow | None = None
+ # cards, newest first
+ self._cards: list[_CardWindow] = []
+ self._flights: dict[str, tuple[_FlyWindow, _CardWindow]] = {}
+ self._blanked = False
+ # recording
+ self._border: _BorderWindow | None = None
+ self._pill: _PillWindow | None = None
+ self._expiry = QTimer(self)
+ self._expiry.setInterval(250)
+ self._expiry.timeout.connect(self._expire_cards)
+ self._expiry.start()
+
+ def text(self, key: str) -> str:
+ return self._texts.get(key, key)
+
+ # -- IPC -----------------------------------------------------------------
+ @Slot(str)
+ def on_line(self, raw: str) -> None:
+ payload = protocol.decode_command(raw)
+ if payload is None:
+ return
+ self._commands.append(payload)
+ if not self._draining:
+ self._drain()
+
+ def _drain(self) -> None:
+ self._draining = True
+ try:
+ while self._commands:
+ payload = self._commands.popleft()
+ cmd = payload["cmd"]
+ texts = payload.get("texts")
+ if isinstance(texts, dict):
+ self._texts.update({str(k): str(v) for k, v in texts.items()})
+ try:
+ self._apply(cmd, payload)
+ except Exception as exc: # noqa: BLE001 - one bad command must not kill the overlay
+ _log(f"{cmd} failed: {type(exc).__name__}: {exc}")
+ if cmd == protocol.CMD_QUIT:
+ _ack(cmd)
+ self._app.quit()
+ self._commands.clear()
+ return
+ # Flush native show/hide work before acknowledging, so an ACK
+ # of "blank" really means "not on screen any more".
+ self._app.processEvents()
+ _ack(cmd)
+ finally:
+ self._draining = False
+
+ def _apply(self, cmd: str, payload: dict) -> None:
+ if cmd == protocol.CMD_SELECT:
+ self._start_selection(payload)
+ elif cmd == protocol.CMD_CANCEL_SELECT:
+ self.finish_selection(None, None)
+ elif cmd == protocol.CMD_CARD:
+ self._add_card(payload)
+ elif cmd == protocol.CMD_REMOVE_CARD:
+ for card in list(self._cards):
+ if card.item_id == str(payload.get("id", "")):
+ card.leave(notify=False)
+ elif cmd == protocol.CMD_BLANK:
+ self._set_blank(True)
+ elif cmd == protocol.CMD_UNBLANK:
+ self._set_blank(False)
+ elif cmd == protocol.CMD_REC_SHOW:
+ self._show_recording(payload)
+ elif cmd == protocol.CMD_REC_HIDE:
+ self._hide_recording()
+
+ # -- selection -----------------------------------------------------------
+ def _start_selection(self, payload: dict) -> None:
+ if self._select_req is not None:
+ self.finish_selection(None, None)
+ try:
+ self._select_req = int(payload.get("req", 0))
+ except (TypeError, ValueError):
+ self._select_req = 0
+ hint = str(payload.get("hint", ""))
+ self._select_windows = [_SelectWindow(s, hint, self) for s in QGuiApplication.screens()]
+ cursor_screen = QGuiApplication.screenAt(QCursor.pos()) or QGuiApplication.primaryScreen()
+ self._hint_window = next(
+ (w for w in self._select_windows if w._screen is cursor_screen), None
+ )
+ for win in self._select_windows:
+ win.show()
+ focus = self._hint_window or (self._select_windows[0] if self._select_windows else None)
+ if focus is not None:
+ focus.raise_()
+ focus.activateWindow()
+ focus.setFocus()
+
+ def hint_screen(self) -> _SelectWindow | None:
+ return self._hint_window
+
+ def selection_started(self, window: _SelectWindow) -> None:
+ for other in self._select_windows:
+ if other is not window:
+ other.clear_pointer()
+ if self._hint_window is not window:
+ self._hint_window = window
+ for win in self._select_windows:
+ win.update()
+
+ def pointer_moved(self, window: _SelectWindow) -> None:
+ for other in self._select_windows:
+ if other is not window:
+ other.clear_pointer()
+
+ def finish_selection(
+ self, window: _SelectWindow | None, frac: tuple[float, float, float, float] | None
+ ) -> None:
+ req = self._select_req
+ if req is None:
+ return
+ self._select_req = None
+ info = window.info if window is not None else None
+ for win in self._select_windows:
+ win.hide()
+ win.deleteLater()
+ self._select_windows = []
+ self._hint_window = None
+ # The dimmed overlay must be gone from the glass before the grab.
+ self._app.processEvents()
+ if info is None or frac is None:
+ _emit(protocol.EVENT_SELECTION, req=req, cancelled=True)
+ return
+ _emit(protocol.EVENT_SELECTION, req=req, screen=info, rect=list(frac))
+
+ # -- cards ---------------------------------------------------------------
+ def _add_card(self, payload: dict) -> None:
+ item_id = str(payload.get("id", ""))
+ if not item_id:
+ return
+ thumb = QImage()
+ raw = payload.get("thumb")
+ if isinstance(raw, str) and raw:
+ thumb.loadFromData(QByteArray.fromBase64(raw.encode("ascii")))
+ screen = _screen_for_monitor(payload.get("monitor"))
+ if screen is None:
+ return
+ aspect = thumb.height() / max(1, thumb.width()) if not thumb.isNull() else 0.6
+ cw = _CARD_W
+ ch = cw * aspect
+ if ch > _CARD_MAX_H:
+ ch = _CARD_MAX_H
+ cw = ch / max(aspect, 1e-6)
+ card = _CardWindow(
+ self,
+ item_id,
+ screen,
+ thumb,
+ (cw, ch),
+ persist=bool(payload.get("persist", False)),
+ dismiss_s=geometry.clamp_dismiss_s(payload.get("dismiss_s", 30)),
+ badge=str(payload.get("badge", "") or ""),
+ )
+ self._cards.insert(0, card)
+ while len(self._cards) > geometry.MAX_CARDS:
+ self._cards[-1].leave(notify=False)
+ self._layout_cards()
+ frac = _frac(payload.get("rect"))
+ g = screen.geometry()
+ if frac is None or self._blanked:
+ self._land(card)
+ return
+ fx, fy, fw, fh = frac
+ src = QRectF(
+ fx * g.width(),
+ fy * g.height(),
+ max(1.0, fw * g.width()),
+ max(1.0, fh * g.height()),
+ )
+ dst = self._card_local_rect(card)
+ flight = _FlyWindow(screen, src, dst, thumb, lambda _w, c=card: self._land(c))
+ self._flights[item_id] = (flight, card)
+ flight.start()
+
+ def _card_local_rect(self, card: _CardWindow) -> QRectF:
+ g = card.screen_ref.geometry()
+ return QRectF(
+ card.x() + _CARD_PAD - g.x(),
+ card.y() + _CARD_PAD - g.y(),
+ card.card_size[0],
+ card.card_size[1],
+ )
+
+ def _land(self, card: _CardWindow) -> None:
+ self._flights.pop(card.item_id, None)
+ if card not in self._cards:
+ return
+ card.timer = geometry.CardTimer(
+ persist=card.timer.persist, dismiss_s=card.timer.dismiss_s, now=time.monotonic()
+ )
+ if not self._blanked:
+ card.setWindowOpacity(1.0)
+ card.show()
+
+ def _layout_cards(self) -> None:
+ """Stack the auto-placed cards per screen, newest lowest."""
+ by_screen: dict[int, list[_CardWindow]] = {}
+ for card in self._cards:
+ if not card.moved_by_user:
+ by_screen.setdefault(id(card.screen_ref), []).append(card)
+ for cards in by_screen.values():
+ g = cards[0].screen_ref.geometry()
+ corners = geometry.card_stack(
+ float(g.width()),
+ float(g.height()),
+ [c.card_size for c in cards],
+ margin=_CARD_MARGIN,
+ gap=_CARD_GAP + 2 * _CARD_PAD - _CARD_PAD,
+ )
+ for card, (x, y) in zip(cards, corners, strict=True):
+ card.place_card_at(x, y)
+ flight = self._flights.get(card.item_id)
+ if flight is not None:
+ flight[0].retarget(self._card_local_rect(card))
+
+ def card_leaving(self, card: _CardWindow, *, notify: bool) -> None:
+ with suppress(ValueError):
+ self._cards.remove(card)
+ flight = self._flights.pop(card.item_id, None)
+ if flight is not None:
+ flight[0].finish_now()
+ if notify:
+ _emit(protocol.EVENT_CARD_CLOSED, id=card.item_id)
+ self._layout_cards()
+
+ def _expire_cards(self) -> None:
+ now = time.monotonic()
+ for card in list(self._cards):
+ if card.item_id in self._flights:
+ continue
+ if card.timer.expired(now):
+ card.leave(notify=False)
+
+ def _set_blank(self, blank: bool) -> None:
+ self._blanked = blank
+ for flight, _card in list(self._flights.values()):
+ flight.finish_now()
+ for card in self._cards:
+ if blank:
+ card.hide()
+ else:
+ card.show()
+
+ # -- recording -----------------------------------------------------------
+ def _show_recording(self, payload: dict) -> None:
+ self._hide_recording()
+ screen = _screen_for_monitor(payload.get("monitor"))
+ if screen is None:
+ return
+ g = screen.geometry()
+ frac = _frac(payload.get("rect"))
+ if frac is not None:
+ fx, fy, fw, fh = frac
+ area = QRectF(fx * g.width(), fy * g.height(), fw * g.width(), fh * g.height())
+ self._border = _BorderWindow(screen, area)
+ self._border.show()
+ pill = _PillWindow(self, screen, time.monotonic())
+ spot = geometry.pill_placement(
+ frac, float(g.width()), float(g.height()), float(pill.width()), float(pill.height())
+ )
+ pill.move(int(g.x() + spot.x), int(g.y() + spot.y))
+ pill.show()
+ self._pill = pill
+
+ def _hide_recording(self) -> None:
+ for widget in (self._border, self._pill):
+ if widget is not None:
+ widget.hide()
+ widget.deleteLater()
+ self._border = None
+ self._pill = None
+
+
+# --------------------------------------------------------------------------
+# Process plumbing
+# --------------------------------------------------------------------------
+
+_out_lock = threading.Lock()
+
+
+def _write(line: str) -> None:
+ # A failed write means the parent is gone; the stdin EOF path quits.
+ with suppress(Exception), _out_lock:
+ sys.stdout.write(line)
+ sys.stdout.flush()
+
+
+def _ack(cmd: str) -> None:
+ _write(protocol.encode_ack(cmd))
+
+
+def _emit(event: str, **fields: Any) -> None:
+ _write(protocol.encode_event(event, **fields))
+
+
+class _StdinPump(QObject):
+ line = Signal(str)
+ eof = Signal()
+
+ def start(self) -> None:
+ threading.Thread(target=self._run, name="jarvisx-overlay-stdin", daemon=True).start()
+
+ def _run(self) -> None:
+ with suppress(Exception):
+ for raw in sys.stdin:
+ self.line.emit(raw)
+ self.eof.emit()
+
+
+def run() -> int:
+ """Sidecar main loop; returns the process exit code."""
+ QGuiApplication.setHighDpiScaleFactorRoundingPolicy(
+ Qt.HighDpiScaleFactorRoundingPolicy.PassThrough
+ )
+ try:
+ app = QApplication(sys.argv[:1] or ["jarvisx-overlay"])
+ except Exception as exc: # noqa: BLE001 - no display / no platform plugin
+ sys.stderr.write(f"jarvisx-overlay: no usable display ({exc!r}).\n")
+ return protocol.EXIT_NO_GUI
+ app.setQuitOnLastWindowClosed(False)
+ renderer = Renderer(app)
+ pump = _StdinPump()
+ pump.line.connect(renderer.on_line, Qt.ConnectionType.QueuedConnection)
+ pump.eof.connect(app.quit, Qt.ConnectionType.QueuedConnection)
+ pump.start()
+ return app.exec()
+
+
+__all__ = ["Renderer", "run"]
diff --git a/jarvis/jarvisx/paths.py b/jarvis/jarvisx/paths.py
new file mode 100644
index 0000000000..03e4430f17
--- /dev/null
+++ b/jarvis/jarvisx/paths.py
@@ -0,0 +1,182 @@
+"""Where Jarvis X keeps its files.
+
+Two locations:
+
+- the **save folder** for the captures themselves — ``[jarvisx].save_dir`` when
+ set, else ``Pictures/Jarvis X`` resolved per OS (the Windows known-folder
+ API, so a OneDrive-redirected Pictures folder is honoured; the XDG
+ ``PICTURES`` entry on Linux; ``~/Pictures`` on macOS), else a folder in the
+ Jarvis data directory when the machine has no Pictures folder at all (a
+ headless server);
+- the **library folder** in the Jarvis data directory, holding the index and
+ the small preview thumbnails.
+
+Only stdlib at module scope; the Windows path uses ctypes lazily.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import re
+import sys
+from datetime import datetime
+from pathlib import Path
+
+log = logging.getLogger(__name__)
+
+#: Folder name inside Pictures (and the file-name prefix of every capture).
+FOLDER_NAME = "Jarvis X"
+
+
+def library_dir() -> Path:
+ """``<data dir>/jarvisx`` — the index database and the thumbnails."""
+ from jarvis.core import config as config_module # noqa: PLC0415
+
+ return Path(config_module.DATA_DIR) / "jarvisx"
+
+
+def default_save_dir() -> Path:
+ """``Pictures/Jarvis X`` when a Pictures folder exists, else the data dir."""
+ pictures = pictures_dir()
+ if pictures is not None:
+ return pictures / FOLDER_NAME
+ return library_dir() / "captures"
+
+
+def resolve_save_dir(configured: str) -> Path:
+ """The folder new captures go to, created on demand.
+
+ A configured folder that cannot be created falls back to the default one
+ (logged), so a stale setting never loses a capture.
+ """
+ candidates: list[Path] = []
+ text = str(configured or "").strip()
+ if text:
+ candidates.append(Path(os.path.expandvars(text)).expanduser())
+ candidates.append(default_save_dir())
+ candidates.append(library_dir() / "captures")
+ for candidate in candidates:
+ try:
+ candidate.mkdir(parents=True, exist_ok=True)
+ return candidate
+ except OSError:
+ log.warning("jarvisx: save folder %s is not usable, trying the next one", candidate)
+ # The last candidate lives in the data dir; if even that fails the caller's
+ # write reports the real OS error.
+ return candidates[-1]
+
+
+def pictures_dir() -> Path | None:
+ """The user's Pictures folder, or ``None`` when this machine has none."""
+ try:
+ if sys.platform == "win32":
+ found = _windows_pictures()
+ elif sys.platform == "darwin":
+ found = Path.home() / "Pictures"
+ else:
+ found = _xdg_pictures()
+ except Exception: # noqa: BLE001 - a probe failure means "use the fallback"
+ log.debug("jarvisx: Pictures folder probe failed", exc_info=True)
+ return None
+ if found is not None and found.is_dir():
+ return found
+ return None
+
+
+def _windows_pictures() -> Path | None:
+ """``FOLDERID_Pictures`` through ``SHGetKnownFolderPath`` (OneDrive-aware)."""
+ import ctypes # noqa: PLC0415
+ from ctypes import wintypes # noqa: PLC0415
+
+ class _GUID(ctypes.Structure):
+ _fields_ = [
+ ("Data1", wintypes.DWORD),
+ ("Data2", wintypes.WORD),
+ ("Data3", wintypes.WORD),
+ ("Data4", ctypes.c_ubyte * 8),
+ ]
+
+ # {33E28130-4E1E-4676-835A-98395C3BC3BB}
+ folder_id = _GUID(
+ 0x33E28130,
+ 0x4E1E,
+ 0x4676,
+ (ctypes.c_ubyte * 8)(0x83, 0x5A, 0x98, 0x39, 0x5C, 0x3B, 0xC3, 0xBB),
+ )
+ shell32 = ctypes.WinDLL("shell32", use_last_error=True)
+ ole32 = ctypes.WinDLL("ole32", use_last_error=True)
+ shell32.SHGetKnownFolderPath.argtypes = [
+ ctypes.POINTER(_GUID),
+ wintypes.DWORD,
+ wintypes.HANDLE,
+ ctypes.POINTER(ctypes.c_wchar_p),
+ ]
+ shell32.SHGetKnownFolderPath.restype = ctypes.c_long
+ ole32.CoTaskMemFree.argtypes = [ctypes.c_void_p]
+ ole32.CoTaskMemFree.restype = None
+ out = ctypes.c_wchar_p()
+ if shell32.SHGetKnownFolderPath(ctypes.byref(folder_id), 0, None, ctypes.byref(out)) != 0:
+ return Path.home() / "Pictures"
+ try:
+ return Path(out.value) if out.value else Path.home() / "Pictures"
+ finally:
+ ole32.CoTaskMemFree(out)
+
+
+def _xdg_pictures() -> Path | None:
+ """``XDG_PICTURES_DIR`` from ``user-dirs.dirs``, else ``~/Pictures``."""
+ home = Path.home()
+ config_home = Path(os.environ.get("XDG_CONFIG_HOME") or home / ".config")
+ dirs_file = config_home / "user-dirs.dirs"
+ try:
+ text = dirs_file.read_text(encoding="utf-8")
+ except OSError:
+ return home / "Pictures"
+ match = re.search(r'^XDG_PICTURES_DIR="([^"]*)"', text, re.MULTILINE)
+ if not match:
+ return home / "Pictures"
+ value = match.group(1).replace("$HOME", str(home))
+ path = Path(value)
+ # user-dirs.dirs points Pictures at $HOME itself when the folder was
+ # removed; saving loose files into the home folder is not what anyone wants.
+ return None if path == home else path
+
+
+def capture_filename(kind: str, when: datetime, ext: str) -> str:
+ """``Jarvis X 2026-09-29 at 14.03.22.png`` (``Recording`` for videos)."""
+ label = "Recording" if kind == "video" else FOLDER_NAME
+ stamp = when.strftime("%Y-%m-%d at %H.%M.%S")
+ return f"{label} {stamp}.{ext}"
+
+
+def unique_path(folder: Path, filename: str) -> Path:
+ """``folder/filename``, suffixed `` (2)``, `` (3)`` … when it already exists."""
+ candidate = folder / filename
+ if not candidate.exists():
+ return candidate
+ stem, dot, ext = filename.rpartition(".")
+ if not dot:
+ stem, ext = filename, ""
+ for index in range(2, 1000):
+ candidate = folder / (f"{stem} ({index}).{ext}" if ext else f"{stem} ({index})")
+ if not candidate.exists():
+ return candidate
+ return folder / (f"{stem} {os.getpid()}.{ext}" if ext else f"{stem} {os.getpid()}")
+
+
+def edited_path_for(original: Path) -> Path:
+ """``<name>-edited.png`` next to the original."""
+ return original.with_name(f"{original.stem}-edited.png")
+
+
+__all__ = [
+ "FOLDER_NAME",
+ "capture_filename",
+ "default_save_dir",
+ "edited_path_for",
+ "library_dir",
+ "pictures_dir",
+ "resolve_save_dir",
+ "unique_path",
+]
diff --git a/jarvis/jarvisx/recorder.py b/jarvis/jarvisx/recorder.py
new file mode 100644
index 0000000000..f45292084a
--- /dev/null
+++ b/jarvis/jarvisx/recorder.py
@@ -0,0 +1,466 @@
+"""Screen recording for Jarvis X: grab frames, encode a video, stop cleanly.
+
+The recorder is two seams and a loop:
+
+- a :class:`FrameSource` hands out BGRA frames of one fixed screen area
+ (``mss`` in production);
+- an :class:`Encoder` turns them into a file (PyAV in production);
+- :class:`Recorder` runs the loop on its own thread at up to ``fps`` frames a
+ second. Timestamps come from the wall clock, so a slow machine produces
+ fewer frames of the right duration, never a sped-up video.
+
+Encoder choice (:func:`probe_encoder`): PyAV is already installed with the
+base install on every platform that ships wheels for it (it is the WebRTC
+dependency of the voice stack), so recording needs nothing extra. The probe
+tries H.264 encoders in order of quality — ``libx264``, then the OS encoders
+(Media Foundation on Windows, VideoToolbox on macOS), then ``libopenh264`` —
+and falls back to MPEG-4 Part 2 in an ``.mp4``, then VP9 in a ``.webm``. The
+first one that really encodes a test frame wins, and the settings report which
+one it is. Without PyAV (Windows on ARM) recording is reported unavailable
+with the reason; screenshots keep working.
+
+Audio is not recorded, and the mouse pointer is not drawn into the frames.
+"""
+
+from __future__ import annotations
+
+import io
+import logging
+import queue
+import threading
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Protocol
+
+from jarvis.jarvisx.geometry import Rect, even_size
+
+log = logging.getLogger(__name__)
+
+#: Frames per second the recorder aims for.
+DEFAULT_FPS = 30
+
+#: Consecutive failed grabs after which a recording gives up (display gone).
+_MAX_GRAB_FAILURES = 60
+
+#: Frames waiting for the encoder before new ones are dropped.
+_QUEUE_DEPTH = 3
+
+
+# --------------------------------------------------------------------------
+# Seams
+# --------------------------------------------------------------------------
+
+
+class FrameSource(Protocol):
+ def grab(self) -> tuple[int, int, bytes]:
+ """``(width, height, bgra_bytes)`` of the recorded area, now."""
+ ...
+
+ def close(self) -> None: ...
+
+
+class Encoder(Protocol):
+ name: str
+ extension: str
+
+ def open(self, path: Path, width: int, height: int, fps: int) -> None: ...
+
+ def write(self, bgra: bytes, width: int, height: int, pts: int) -> None:
+ """One frame; ``pts`` counts in units of ``1 / fps`` seconds."""
+ ...
+
+ def close(self) -> None: ...
+
+
+@dataclass(frozen=True, slots=True)
+class EncoderChoice:
+ """The encoder the probe settled on (or why there is none)."""
+
+ available: bool
+ codec: str = ""
+ extension: str = "mp4"
+ pix_fmt: str = "yuv420p"
+ detail: str = ""
+
+
+# --------------------------------------------------------------------------
+# Production seams
+# --------------------------------------------------------------------------
+
+
+class MssFrameSource:
+ """Grabs one fixed capture-space rectangle with ``mss``.
+
+ Created on the recording thread: an ``mss`` handle belongs to the thread
+ that opened it, and the per-monitor DPI pinning is thread-local too.
+ """
+
+ def __init__(self, bbox: Rect) -> None:
+ import mss # type: ignore[import-not-found] # noqa: PLC0415
+
+ from jarvis.jarvisx.capture import input_space # noqa: PLC0415
+
+ left, top, width, height = (int(v) for v in bbox)
+ self._area = {"left": left, "top": top, "width": width, "height": height}
+ self._space = input_space()
+ self._space.__enter__()
+ factory = getattr(mss, "MSS", None) or mss.mss
+ self._sct = factory()
+
+ def grab(self) -> tuple[int, int, bytes]:
+ shot = self._sct.grab(self._area)
+ # ``raw`` is already a fresh BGRA buffer; copying 33 MB per 4K frame
+ # would cost a frame of its own.
+ return int(shot.size[0]), int(shot.size[1]), shot.raw
+
+ def close(self) -> None:
+ try:
+ self._sct.close()
+ finally:
+ self._space.__exit__(None, None, None)
+
+
+_CANDIDATES: tuple[tuple[str, str, str, dict[str, str]], ...] = (
+ # (codec, container extension, pixel format, codec options)
+ ("libx264", "mp4", "yuv420p", {"preset": "superfast", "tune": "zerolatency", "crf": "23"}),
+ ("h264_mf", "mp4", "nv12", {}),
+ ("h264_videotoolbox", "mp4", "nv12", {"realtime": "1"}),
+ ("libopenh264", "mp4", "yuv420p", {}),
+ ("mpeg4", "mp4", "yuv420p", {"qscale": "3"}),
+ ("libvpx-vp9", "webm", "yuv420p", {"deadline": "realtime", "cpu-used": "8"}),
+)
+
+
+class PyAvEncoder:
+ """Encodes BGRA frames into a video file with PyAV (FFmpeg)."""
+
+ def __init__(self, choice: EncoderChoice) -> None:
+ self.name = choice.codec
+ self.extension = choice.extension
+ self._pix_fmt = choice.pix_fmt
+ self._options = next((c[3] for c in _CANDIDATES if c[0] == choice.codec), {})
+ self._container: Any = None
+ self._stream: Any = None
+ self._size = (0, 0)
+
+ def open(self, path: Path, width: int, height: int, fps: int) -> None:
+ import av # noqa: PLC0415
+
+ container_options = {"movflags": "+faststart"} if self.extension == "mp4" else {}
+ self._container = av.open(str(path), mode="w", container_options=container_options)
+ stream = self._container.add_stream(self.name, rate=fps, options=dict(self._options))
+ self._size = even_size(width, height)
+ stream.width, stream.height = self._size
+ stream.pix_fmt = self._pix_fmt
+ self._stream = stream
+
+ def write(self, bgra: bytes, width: int, height: int, pts: int) -> None:
+ import av # noqa: PLC0415
+ import numpy as np # noqa: PLC0415
+
+ w, h = self._size
+ array = np.frombuffer(bgra, dtype=np.uint8).reshape(height, width, 4)[:h, :w]
+ frame = av.VideoFrame.from_ndarray(np.ascontiguousarray(array), format="bgra")
+ frame = frame.reformat(format=self._pix_fmt)
+ frame.pts = pts
+ for packet in self._stream.encode(frame):
+ self._container.mux(packet)
+
+ def close(self) -> None:
+ if self._container is None:
+ return
+ try:
+ for packet in self._stream.encode(None):
+ self._container.mux(packet)
+ finally:
+ self._container.close()
+ self._container = None
+
+
+_probe_lock = threading.Lock()
+_probe_result: EncoderChoice | None = None
+
+
+def probe_encoder(*, refresh: bool = False) -> EncoderChoice:
+ """The best video encoder that really works here (cached per process)."""
+ global _probe_result
+ with _probe_lock:
+ if _probe_result is not None and not refresh:
+ return _probe_result
+ _probe_result = _probe()
+ return _probe_result
+
+
+def _probe() -> EncoderChoice:
+ try:
+ import av # noqa: PLC0415
+ except Exception as exc: # noqa: BLE001 - ImportError or a broken native build
+ log.info("jarvisx: recording unavailable, PyAV missing (%s)", exc)
+ return EncoderChoice(
+ available=False,
+ detail=(
+ "Screen recording needs the PyAV package, which is not installed on this "
+ "computer (pip install av). Screenshots work without it."
+ ),
+ )
+ available = set(getattr(av, "codecs_available", ()) or ())
+ for codec, extension, pix_fmt, options in _CANDIDATES:
+ if available and codec not in available:
+ continue
+ if _encodes(av, codec, extension, pix_fmt, options):
+ label = "H.264" if "264" in codec else ("MPEG-4" if codec == "mpeg4" else "VP9")
+ return EncoderChoice(
+ available=True,
+ codec=codec,
+ extension=extension,
+ pix_fmt=pix_fmt,
+ detail=f"Recordings are saved as {extension.upper()} ({label}, {codec}).",
+ )
+ return EncoderChoice(
+ available=False,
+ detail="No working video encoder was found in this PyAV build, so recording is off.",
+ )
+
+
+def _encodes(av: Any, codec: str, extension: str, pix_fmt: str, options: dict[str, str]) -> bool:
+ """Encode one tiny frame into memory; any failure means "not usable"."""
+ try:
+ import numpy as np # noqa: PLC0415
+
+ buffer = io.BytesIO()
+ container = av.open(buffer, mode="w", format=extension)
+ try:
+ stream = container.add_stream(codec, rate=DEFAULT_FPS, options=dict(options))
+ stream.width, stream.height = 64, 64
+ stream.pix_fmt = pix_fmt
+ array = np.zeros((64, 64, 4), dtype=np.uint8)
+ for pts in range(2):
+ frame = av.VideoFrame.from_ndarray(array, format="bgra").reformat(format=pix_fmt)
+ frame.pts = pts
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode(None):
+ container.mux(packet)
+ finally:
+ container.close()
+ return buffer.tell() > 0 or len(buffer.getvalue()) > 0
+ except Exception: # noqa: BLE001 - probing: an encoder that fails is skipped
+ log.debug("jarvisx: encoder %s is not usable here", codec, exc_info=True)
+ return False
+
+
+# --------------------------------------------------------------------------
+# The recording loop
+# --------------------------------------------------------------------------
+
+
+@dataclass(frozen=True, slots=True)
+class RecordingResult:
+ path: Path
+ width: int
+ height: int
+ duration_s: float
+ frames: int
+ #: The first frame as ``(width, height, bgra)``, for the thumbnail.
+ first_frame: tuple[int, int, bytes] | None
+ error: str = ""
+
+ @property
+ def ok(self) -> bool:
+ return not self.error and self.frames > 0
+
+
+class Recorder:
+ """Records one area until :meth:`stop`. One use per instance."""
+
+ def __init__(
+ self,
+ *,
+ source_factory: Callable[[], FrameSource],
+ encoder: Encoder,
+ path: Path,
+ fps: int = DEFAULT_FPS,
+ clock: Callable[[], float] = time.monotonic,
+ sleep: Callable[[float], None] = time.sleep,
+ ) -> None:
+ self._source_factory = source_factory
+ self._encoder = encoder
+ self.path = path
+ self.fps = max(1, int(fps))
+ self._clock = clock
+ self._sleep = sleep
+ self._stop = threading.Event()
+ self._thread: threading.Thread | None = None
+ self._started_at: float | None = None
+ self._stopped_at: float | None = None
+ self._frames = 0
+ self._size = (0, 0)
+ self._first: tuple[int, int, bytes] | None = None
+ self._error = ""
+ self._dropped = 0
+ self._opened = threading.Event()
+
+ @property
+ def elapsed_s(self) -> float:
+ if self._started_at is None:
+ return 0.0
+ end = self._stopped_at if self._stopped_at is not None else self._clock()
+ return max(0.0, end - self._started_at)
+
+ @property
+ def running(self) -> bool:
+ return self._thread is not None and self._thread.is_alive()
+
+ def start(self, *, wait_s: float = 5.0) -> str:
+ """Start the loop; returns ``""`` or the reason it could not start."""
+ self._thread = threading.Thread(target=self._run, name="jarvisx-recorder", daemon=True)
+ self._thread.start()
+ self._opened.wait(timeout=wait_s)
+ return self._error
+
+ def stop(self, *, timeout_s: float = 30.0) -> RecordingResult:
+ self._stop.set()
+ thread = self._thread
+ if thread is not None:
+ thread.join(timeout=timeout_s)
+ if thread.is_alive():
+ self._error = self._error or "The recording did not finish writing in time."
+ width, height = self._size
+ return RecordingResult(
+ path=self.path,
+ width=width,
+ height=height,
+ duration_s=round(self.elapsed_s, 2),
+ frames=self._frames,
+ first_frame=self._first,
+ error=self._error,
+ )
+
+ def _run(self) -> None:
+ source: FrameSource | None = None
+ opened = False
+ frames: queue.Queue[tuple[int, bytes] | None] = queue.Queue(maxsize=_QUEUE_DEPTH)
+ writer: threading.Thread | None = None
+ try:
+ source = self._source_factory()
+ width, height, first = source.grab()
+ self._size = even_size(width, height)
+ self._encoder.open(self.path, width, height, self.fps)
+ opened = True
+ self._first = (width, height, first)
+ self._started_at = self._clock()
+ writer = threading.Thread(
+ target=self._write_frames,
+ args=(frames, width, height),
+ name="jarvisx-encoder",
+ daemon=True,
+ )
+ writer.start()
+ self._opened.set()
+ self._grab_frames(source, frames, width, height, first)
+ except Exception as exc: # noqa: BLE001 - reported through the result
+ log.warning("jarvisx: recording failed", exc_info=True)
+ self._error = self._error or f"The recording failed ({type(exc).__name__}: {exc})."
+ finally:
+ self._stopped_at = self._clock() if self._started_at is not None else None
+ self._opened.set()
+ if writer is not None:
+ frames.put(None)
+ writer.join()
+ if opened:
+ try:
+ self._encoder.close()
+ except Exception as exc: # noqa: BLE001 - reported through the result
+ log.warning("jarvisx: finishing the video failed", exc_info=True)
+ self._error = self._error or f"The video could not be finished ({exc})."
+ if source is not None:
+ try:
+ source.close()
+ except Exception: # noqa: BLE001 - the video is already written
+ log.debug("jarvisx: frame source close failed", exc_info=True)
+
+ def _grab_frames(
+ self,
+ source: FrameSource,
+ frames: queue.Queue[tuple[int, bytes] | None],
+ width: int,
+ height: int,
+ first: bytes,
+ ) -> None:
+ """Grab on this thread, encode on the writer thread.
+
+ The two run in parallel (both native calls release the GIL), which
+ roughly doubles the frame rate of a large screen. When the encoder
+ falls behind, a frame is dropped rather than queued: the timestamps
+ keep the video's duration true.
+ """
+ assert self._started_at is not None
+ interval = 1.0 / self.fps
+ last_pts = -1
+ failures = 0
+ pending: bytes | None = first
+ while True:
+ pts = int((self._clock() - self._started_at) * self.fps)
+ if pending is not None and pts > last_pts:
+ try:
+ frames.put_nowait((pts, pending))
+ last_pts = pts
+ except queue.Full:
+ self._dropped += 1
+ if self._stop.is_set() or self._error:
+ return
+ next_due = self._started_at + (max(last_pts, pts) + 1) * interval
+ delay = next_due - self._clock()
+ if delay > 0:
+ if self._sleep is time.sleep:
+ # Wakes early on stop(), so stopping never waits a frame.
+ self._stop.wait(delay)
+ else:
+ self._sleep(delay)
+ try:
+ w, h, pending = source.grab()
+ failures = 0
+ if (w, h) != (width, height):
+ # The screen changed resolution mid-recording; frames of
+ # another size cannot join this video.
+ pending = None
+ except Exception: # noqa: BLE001 - a transient grab failure skips a frame
+ failures += 1
+ pending = None
+ log.debug("jarvisx: frame grab failed", exc_info=True)
+ if failures >= _MAX_GRAB_FAILURES:
+ self._error = "The screen stopped delivering frames, so the recording ended."
+ return
+
+ def _write_frames(
+ self, frames: queue.Queue[tuple[int, bytes] | None], width: int, height: int
+ ) -> None:
+ while True:
+ item = frames.get()
+ if item is None:
+ return
+ if self._error:
+ continue
+ pts, bgra = item
+ try:
+ self._encoder.write(bgra, width, height, pts)
+ self._frames += 1
+ except Exception as exc: # noqa: BLE001 - ends the recording, reported
+ log.warning("jarvisx: encoding a frame failed", exc_info=True)
+ self._error = f"The video encoder failed ({type(exc).__name__}: {exc})."
+ self._stop.set()
+
+
+__all__ = [
+ "DEFAULT_FPS",
+ "Encoder",
+ "EncoderChoice",
+ "FrameSource",
+ "MssFrameSource",
+ "PyAvEncoder",
+ "Recorder",
+ "RecordingResult",
+ "probe_encoder",
+]
diff --git a/jarvis/jarvisx/service.py b/jarvis/jarvisx/service.py
new file mode 100644
index 0000000000..4de6f55e0c
--- /dev/null
+++ b/jarvis/jarvisx/service.py
@@ -0,0 +1,627 @@
+"""The Jarvis X orchestrator — every trigger (shortcut, REST, card click) ends here.
+
+One capture at a time, one recording at a time. A screenshot runs:
+
+1. resolve the area — a dragged region (overlay), the front window, or the
+ monitor under the pointer;
+2. grab it with the resting thumbnail cards hidden;
+3. save the PNG to the save folder, a preview to the library, index it;
+4. play the capture cue, show the flash + corner card, copy to the clipboard
+ (each behind its own switch);
+5. publish ``JarvisXItemCreated``.
+
+A recording starts the same way for the area, then runs the recorder until a
+stop (shortcut, pill click or REST) and lands in the library like a picture.
+
+Every public coroutine returns a result object and never raises: a shortcut
+press must never take the app down.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import contextlib
+import logging
+import os
+import uuid
+from collections.abc import Callable
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Literal
+
+from jarvis.jarvisx import capture, geometry, paths
+from jarvis.jarvisx.store import Item, ItemStore, get_store
+
+log = logging.getLogger(__name__)
+
+CaptureMode = Literal["region", "window", "fullscreen"]
+RecordMode = Literal["region", "fullscreen"]
+
+#: Largest annotated PNG the editor may save.
+MAX_EDITED_BYTES = 80 * 1024 * 1024
+
+#: Pause after the selection overlay closed, so the compositor has removed
+#: the dimmed layer from the glass before the grab.
+_AFTER_SELECTION_S = 0.12
+
+
+@dataclass(frozen=True, slots=True)
+class Result:
+ ok: bool
+ message: str = ""
+ item: Item | None = None
+
+
+@dataclass
+class _ActiveRecording:
+ recorder: Any
+ mode: RecordMode
+ monitor: dict
+ rect: list[float] | None
+
+
+def _load_config() -> Any:
+ from jarvis.core.config import load_config # noqa: PLC0415
+
+ return load_config()
+
+
+def _now() -> datetime:
+ return datetime.now().astimezone()
+
+
+class JarvisXService:
+ """Capture, record and library actions. Obtain it with :func:`get_service`."""
+
+ def __init__(
+ self,
+ *,
+ store: ItemStore | None = None,
+ overlay: Any | None = None,
+ config_loader: Callable[[], Any] = _load_config,
+ ) -> None:
+ self._store = store
+ self._overlay = overlay
+ self._config_loader = config_loader
+ self._bus: Any | None = None
+ self._app_state: Any | None = None
+ self._busy = asyncio.Lock()
+ self._recording: _ActiveRecording | None = None
+ self._record_lock = asyncio.Lock()
+ self._bound_loop: asyncio.AbstractEventLoop | None = None
+
+ # ---------------------------------------------------------------- wiring
+ def bind(self, *, bus: Any | None = None, app_state: Any | None = None) -> None:
+ """Attach the event bus and the web app state (for the editor window)."""
+ if bus is not None:
+ self._bus = bus
+ if app_state is not None:
+ self._app_state = app_state
+ with contextlib.suppress(RuntimeError):
+ loop = asyncio.get_running_loop()
+ if self._bound_loop is not loop:
+ self.overlay.bind(loop, self._on_overlay_event)
+ self._bound_loop = loop
+
+ @property
+ def store(self) -> ItemStore:
+ return self._store if self._store is not None else get_store()
+
+ @property
+ def overlay(self) -> Any:
+ if self._overlay is None:
+ from jarvis.jarvisx.overlay.controller import get_overlay # noqa: PLC0415
+
+ self._overlay = get_overlay()
+ return self._overlay
+
+ async def _config(self) -> Any:
+ return await asyncio.to_thread(self._config_loader)
+
+ # --------------------------------------------------------------- capture
+ async def capture(self, mode: CaptureMode, *, delay_s: float = 0.0) -> Result:
+ """Take one screenshot. Never raises."""
+ self.bind()
+ if self._busy.locked():
+ return Result(False, "A capture is already in progress.")
+ async with self._busy:
+ try:
+ return await self._capture(mode, delay_s)
+ except Exception: # noqa: BLE001 - a shortcut press must never crash the app
+ log.error("jarvisx: capture failed", exc_info=True)
+ return Result(False, "The capture failed. Nothing was saved.")
+
+ async def _capture(self, mode: CaptureMode, delay_s: float) -> Result:
+ config = await self._config()
+ block = config.jarvisx
+ if not block.enabled:
+ return Result(False, "Jarvis X is switched off. Turn it on in its settings.")
+ refusal = await asyncio.to_thread(_capture_refusal)
+ if refusal:
+ return Result(False, refusal)
+ if delay_s > 0:
+ await asyncio.sleep(min(float(delay_s), 10.0))
+
+ monitors = await asyncio.to_thread(capture.list_monitors)
+ if not monitors:
+ return Result(False, "No screen could be found to capture.")
+ handle: int | None = None
+ if mode == "region":
+ area = await self._select_area(monitors, purpose="capture")
+ if isinstance(area, str):
+ return Result(False, area)
+ monitor, bbox, rect = area
+ elif mode == "window":
+ try:
+ bbox, handle, _title = await asyncio.to_thread(capture.front_window)
+ except capture.CaptureError as exc:
+ return Result(False, str(exc))
+ from jarvis.appshot.effect import placement # noqa: PLC0415
+
+ mon_rect, frac = placement(bbox, monitors)
+ monitor = dict(zip(("left", "top", "width", "height"), mon_rect, strict=True))
+ rect = list(frac)
+ else:
+ found = await asyncio.to_thread(capture.monitor_under_cursor, monitors)
+ if found is None:
+ return Result(False, "No screen could be found to capture.")
+ monitor = found
+ bbox = tuple(geometry.monitor_rect(found)) # type: ignore[assignment]
+ rect = [0.0, 0.0, 1.0, 1.0]
+
+ try:
+ frame = await asyncio.to_thread(self._grab, mode, bbox, handle)
+ except capture.CaptureError as exc:
+ return Result(False, str(exc))
+
+ item, png = await asyncio.to_thread(self._save_image, mode, frame, block.save_dir)
+ self._after_capture(config, item, monitor, rect, png=png)
+ await self._publish_created(item)
+ log.info("jarvisx: %s screenshot %dx%d saved", mode, item.width, item.height)
+ return Result(True, "Saved.", item)
+
+ def _grab(self, mode: str, bbox: Any, handle: int | None) -> capture.Frame:
+ with self.overlay.cards_hidden():
+ if mode == "window":
+ return capture.grab_window(bbox, handle)
+ return capture.grab_rect(bbox)
+
+ async def _select_area(
+ self, monitors: list[dict], *, purpose: str
+ ) -> tuple[dict, tuple[int, int, int, int], list[float]] | str:
+ """The dragged area as ``(monitor, bbox, fractions)`` or a message."""
+ from jarvis.jarvisx.overlay.controller import SelectionUnavailable # noqa: PLC0415
+
+ try:
+ selection = await self.overlay.select_region(purpose=purpose)
+ except SelectionUnavailable as exc:
+ return str(exc)
+ if selection is None:
+ return "Cancelled."
+ monitor = geometry.match_monitor(selection.screen, monitors)
+ if monitor is None:
+ return "The selected screen could not be found."
+ bbox = geometry.fraction_to_bbox(monitor, selection.rect)
+ await asyncio.sleep(_AFTER_SELECTION_S)
+ return monitor, bbox, list(selection.rect)
+
+ def _save_image(self, mode: str, frame: capture.Frame, save_dir: str) -> tuple[Item, bytes]:
+ png = capture.encode_png(frame)
+ when = _now()
+ folder = paths.resolve_save_dir(save_dir)
+ target = paths.unique_path(folder, paths.capture_filename("image", when, "png"))
+ _atomic_write(target, png)
+ item_id = uuid.uuid4().hex
+ thumb_path = self.store.thumb_path_for(item_id)
+ try:
+ thumb_path.parent.mkdir(parents=True, exist_ok=True)
+ thumb_path.write_bytes(capture.thumbnail_jpeg(frame.width, frame.height, frame.rgb))
+ except OSError:
+ log.warning("jarvisx: preview could not be written", exc_info=True)
+ item = Item(
+ id=item_id,
+ kind="image",
+ mode=mode, # type: ignore[arg-type]
+ created_at=when.isoformat(timespec="seconds"),
+ width=frame.width,
+ height=frame.height,
+ duration_s=None,
+ path=str(target),
+ thumb_path=str(thumb_path),
+ )
+ return self.store.add(item), png
+
+ def _after_capture(
+ self,
+ config: Any,
+ item: Item,
+ monitor: dict,
+ rect: list[float] | None,
+ *,
+ png: bytes | None,
+ badge: str = "",
+ ) -> None:
+ """Sound, card and clipboard — each optional, none can fail the capture."""
+ block = config.jarvisx
+ if block.sound:
+ self._play_cue()
+ if block.effect:
+ self._spawn(self._show_card(config, item, monitor, rect, badge), "jarvisx-card")
+ if png is not None and block.copy_to_clipboard:
+ self._spawn(self._copy_bytes(png), "jarvisx-clipboard")
+
+ def _play_cue(self) -> None:
+ try:
+ from jarvis.audio.effects import get_audio_effects_service # noqa: PLC0415
+
+ get_audio_effects_service().play_capture_cue(section="jarvisx")
+ except Exception: # noqa: BLE001 - audio is decoration
+ log.info("jarvisx: capture cue unavailable", exc_info=True)
+
+ async def _show_card(
+ self, config: Any, item: Item, monitor: dict, rect: list[float] | None, badge: str
+ ) -> None:
+ try:
+ thumb = await asyncio.to_thread(_read_bytes, item.thumb_path)
+ await self.overlay.show_card(
+ item_id=item.id,
+ monitor=geometry.monitor_rect(monitor),
+ rect=rect,
+ thumb_b64=base64.b64encode(thumb).decode("ascii") if thumb else "",
+ persist=bool(config.jarvisx.thumbnail_persist),
+ dismiss_s=geometry.clamp_dismiss_s(config.jarvisx.thumbnail_dismiss_s),
+ badge=badge,
+ )
+ except Exception: # noqa: BLE001 - the card is decoration
+ log.warning("jarvisx: thumbnail card failed", exc_info=True)
+
+ async def _copy_bytes(self, png: bytes) -> tuple[bool, str]:
+ from jarvis.jarvisx.clipboard import copy_png # noqa: PLC0415
+
+ ok, message = await asyncio.to_thread(copy_png, png)
+ if not ok:
+ log.info("jarvisx: clipboard copy skipped: %s", message)
+ return ok, message
+
+ # ------------------------------------------------------------- recording
+ @property
+ def recording(self) -> bool:
+ return self._recording is not None
+
+ def recording_status(self) -> dict[str, Any]:
+ active = self._recording
+ if active is None:
+ return {"recording": False, "mode": None, "elapsed_s": 0.0}
+ return {
+ "recording": True,
+ "mode": active.mode,
+ "elapsed_s": round(float(active.recorder.elapsed_s), 2),
+ }
+
+ async def start_recording(self, mode: RecordMode) -> Result:
+ """Start recording the chosen area. Never raises."""
+ self.bind()
+ async with self._record_lock:
+ if self._recording is not None:
+ return Result(False, "A recording is already running.")
+ if self._busy.locked():
+ return Result(False, "A capture is already in progress.")
+ async with self._busy:
+ try:
+ return await self._start_recording(mode)
+ except Exception: # noqa: BLE001 - a shortcut press must never crash the app
+ log.error("jarvisx: recording could not start", exc_info=True)
+ return Result(False, "The recording could not start.")
+
+ async def _start_recording(self, mode: RecordMode) -> Result:
+ from jarvis.jarvisx import recorder as rec # noqa: PLC0415
+
+ config = await self._config()
+ block = config.jarvisx
+ if not block.enabled:
+ return Result(False, "Jarvis X is switched off. Turn it on in its settings.")
+ refusal = await asyncio.to_thread(_capture_refusal)
+ if refusal:
+ return Result(False, refusal)
+ choice = await asyncio.to_thread(rec.probe_encoder)
+ if not choice.available:
+ return Result(False, choice.detail)
+ monitors = await asyncio.to_thread(capture.list_monitors)
+ if not monitors:
+ return Result(False, "No screen could be found to record.")
+ rect: list[float] | None
+ if mode == "region":
+ area = await self._select_area(monitors, purpose="record")
+ if isinstance(area, str):
+ return Result(False, area)
+ monitor, bbox, rect = area
+ else:
+ found = await asyncio.to_thread(capture.monitor_under_cursor, monitors)
+ if found is None:
+ return Result(False, "No screen could be found to record.")
+ monitor = found
+ bbox = tuple(geometry.monitor_rect(found)) # type: ignore[assignment]
+ rect = None
+ folder = await asyncio.to_thread(paths.resolve_save_dir, block.save_dir)
+ target = paths.unique_path(
+ folder, paths.capture_filename("video", _now(), choice.extension)
+ )
+ recorder = self._make_recorder(choice, bbox, target)
+ error = await asyncio.to_thread(recorder.start)
+ if error:
+ await asyncio.to_thread(_unlink, target)
+ return Result(False, error)
+ self._recording = _ActiveRecording(recorder=recorder, mode=mode, monitor=monitor, rect=rect)
+ await self.overlay.show_recording(monitor=geometry.monitor_rect(monitor), rect=rect)
+ if block.sound:
+ self._play_cue()
+ await self._publish_recording(True, mode, 0.0)
+ log.info("jarvisx: %s recording started -> %s", mode, target.name)
+ return Result(True, "Recording.")
+
+ def _make_recorder(self, choice: Any, bbox: Any, target: Path) -> Any:
+ from jarvis.jarvisx import recorder as rec # noqa: PLC0415
+
+ area = tuple(int(v) for v in bbox)
+ return rec.Recorder(
+ source_factory=lambda: rec.MssFrameSource(area), # type: ignore[arg-type]
+ encoder=rec.PyAvEncoder(choice),
+ path=target,
+ )
+
+ async def stop_recording(self) -> Result:
+ """Stop and save the running recording. Never raises."""
+ async with self._record_lock:
+ active, self._recording = self._recording, None
+ if active is None:
+ return Result(False, "No recording is running.")
+ try:
+ return await self._finish_recording(active)
+ except Exception: # noqa: BLE001 - reported, the app keeps running
+ log.error("jarvisx: recording could not be saved", exc_info=True)
+ await self._publish_recording(False, "", 0.0)
+ return Result(False, "The recording could not be saved.")
+
+ async def _finish_recording(self, active: _ActiveRecording) -> Result:
+ await self.overlay.hide_recording()
+ result = await asyncio.to_thread(active.recorder.stop)
+ await self._publish_recording(False, "", float(result.duration_s))
+ if result.frames <= 0 or not result.path.is_file():
+ await asyncio.to_thread(_unlink, result.path)
+ return Result(False, result.error or "Nothing was recorded.")
+ item = await asyncio.to_thread(self._index_video, active, result)
+ config = await self._config()
+ self._after_capture(
+ config,
+ item,
+ active.monitor,
+ active.rect,
+ png=None,
+ badge=geometry.format_elapsed(result.duration_s),
+ )
+ await self._publish_created(item)
+ log.info(
+ "jarvisx: recording saved (%.1f s, %d frames) %s",
+ result.duration_s,
+ result.frames,
+ result.path.name,
+ )
+ message = "Saved." if not result.error else f"Saved, but: {result.error}"
+ return Result(True, message, item)
+
+ def _index_video(self, active: _ActiveRecording, result: Any) -> Item:
+ item_id = uuid.uuid4().hex
+ thumb_path = self.store.thumb_path_for(item_id)
+ if result.first_frame is not None:
+ try:
+ width, height, bgra = result.first_frame
+ thumb_path.parent.mkdir(parents=True, exist_ok=True)
+ thumb_path.write_bytes(_bgra_thumbnail(width, height, bgra))
+ except Exception: # noqa: BLE001 - a missing preview is not a lost video
+ log.warning("jarvisx: video preview could not be written", exc_info=True)
+ item = Item(
+ id=item_id,
+ kind="video",
+ mode=active.mode,
+ created_at=_now().isoformat(timespec="seconds"),
+ width=int(result.width),
+ height=int(result.height),
+ duration_s=float(result.duration_s),
+ path=str(result.path),
+ thumb_path=str(thumb_path),
+ )
+ return self.store.add(item)
+
+ async def toggle_recording(self, mode: RecordMode) -> Result:
+ """A record shortcut: start, or stop when a recording is running."""
+ if self._recording is not None:
+ return await self.stop_recording()
+ return await self.start_recording(mode)
+
+ # --------------------------------------------------------------- library
+ async def save_edited(self, item_id: str, png: bytes) -> Result:
+ item = await asyncio.to_thread(self.store.get, item_id)
+ if item is None:
+ return Result(False, "That capture no longer exists.")
+ if item.kind != "image":
+ return Result(False, "Only screenshots can be annotated.")
+ target = paths.edited_path_for(Path(item.path))
+ await asyncio.to_thread(_atomic_write, target, png)
+ updated = await asyncio.to_thread(self.store.set_edited, item_id, str(target))
+ await self._publish(_event("JarvisXItemUpdated", id=item_id))
+ return Result(True, "Saved.", updated)
+
+ async def copy_item(self, item_id: str, *, edited: bool) -> Result:
+ item = await asyncio.to_thread(self.store.get, item_id)
+ if item is None:
+ return Result(False, "That capture no longer exists.")
+ if item.kind != "image":
+ return Result(False, "Only screenshots can be copied as a picture.")
+ source = item.edited_path if edited and item.has_edited() else item.path
+ try:
+ png = await asyncio.to_thread(Path(source).read_bytes)
+ except OSError as exc:
+ return Result(False, f"The file could not be read ({exc.strerror or exc}).")
+ ok, message = await self._copy_bytes(png)
+ return Result(ok, message, item)
+
+ async def reveal(self, item_id: str) -> Result:
+ item = await asyncio.to_thread(self.store.get, item_id)
+ if item is None:
+ return Result(False, "That capture no longer exists.")
+ from jarvis.platform.open_path import reveal_in_folder # noqa: PLC0415
+
+ opened = await asyncio.to_thread(reveal_in_folder, Path(item.path))
+ if not opened:
+ return Result(False, "The file manager could not be opened on this computer.", item)
+ return Result(True, "Opened.", item)
+
+ async def delete(self, item_id: str) -> Result:
+ item = await asyncio.to_thread(self.store.delete, item_id)
+ if item is None:
+ return Result(False, "That capture no longer exists.")
+ with contextlib.suppress(Exception):
+ await self.overlay.remove_card(item_id)
+ await self._publish(_event("JarvisXItemDeleted", id=item_id))
+ return Result(True, "Deleted.", item)
+
+ async def open_editor(self, item_id: str) -> Result:
+ """Open the annotation editor window on ``item_id`` (desktop app only)."""
+ item = await asyncio.to_thread(self.store.get, item_id)
+ if item is None:
+ return Result(False, "That capture no longer exists.")
+ desktop = getattr(self._app_state, "desktop_app", None) if self._app_state else None
+ opener = getattr(desktop, "open_jarvisx_editor", None)
+ if not callable(opener):
+ return Result(
+ False,
+ "The editor window needs the desktop app; this Jarvis runs without one. "
+ f"Open /?view=jarvisx-editor&solo=1&item={item_id} in a browser instead.",
+ item,
+ )
+ # pywebview creates runtime windows only off the main thread, and every
+ # window call blocks its caller until the GUI thread services it.
+ outcome = await asyncio.to_thread(opener, item_id)
+ if isinstance(outcome, dict) and not outcome.get("ok", False):
+ return Result(False, str(outcome.get("reason") or "The editor could not open."), item)
+ return Result(True, "Opened.", item)
+
+ # ---------------------------------------------------------- overlay I/O
+ async def _on_overlay_event(self, event: str, payload: dict[str, Any]) -> None:
+ if event == "card_click":
+ result = await self.open_editor(str(payload.get("id", "")))
+ if not result.ok:
+ log.info("jarvisx: editor not opened: %s", result.message)
+ elif event == "stop_clicked":
+ await self.stop_recording()
+
+ # --------------------------------------------------------------- events
+ async def _publish_created(self, item: Item) -> None:
+ await self._publish(
+ _event("JarvisXItemCreated", id=item.id, kind=item.kind, mode=item.mode)
+ )
+
+ async def _publish_recording(self, recording: bool, mode: str, elapsed_s: float) -> None:
+ await self._publish(
+ _event(
+ "JarvisXRecordingChanged",
+ recording=recording,
+ mode=mode,
+ elapsed_s=round(float(elapsed_s), 2),
+ )
+ )
+
+ async def _publish(self, event: Any) -> None:
+ bus = self._bus
+ if bus is None or event is None:
+ return
+ try:
+ await bus.publish(event)
+ except Exception: # noqa: BLE001 - a lost receipt cannot undo the capture
+ log.warning("jarvisx: event publication failed", exc_info=True)
+
+ @staticmethod
+ def _spawn(coro: Any, name: str) -> None:
+ task = asyncio.get_running_loop().create_task(coro, name=name)
+ _TASKS.add(task)
+ task.add_done_callback(_TASKS.discard)
+
+ async def shutdown(self) -> None:
+ """Finish a running recording and close the overlay (app quit)."""
+ if self._recording is not None:
+ await self.stop_recording()
+ if self._overlay is not None:
+ await self._overlay.shutdown()
+
+
+_TASKS: set[asyncio.Task[Any]] = set()
+
+
+def _event(name: str, **fields: Any) -> Any:
+ from jarvis.core import events # noqa: PLC0415
+
+ return getattr(events, name)(source_layer="jarvisx", **fields)
+
+
+def _capture_refusal() -> str:
+ ok, reason = capture.capability()
+ if not ok:
+ return reason
+ return capture.permission_problem()
+
+
+def _atomic_write(target: Path, data: bytes) -> None:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ tmp = target.with_name(f".{target.name}.{uuid.uuid4().hex[:8]}.tmp")
+ try:
+ tmp.write_bytes(data)
+ os.replace(tmp, target)
+ finally:
+ if tmp.exists():
+ tmp.unlink(missing_ok=True)
+
+
+def _read_bytes(path: str) -> bytes:
+ try:
+ return Path(path).read_bytes() if path else b""
+ except OSError:
+ return b""
+
+
+def _unlink(path: Path) -> None:
+ try:
+ path.unlink(missing_ok=True)
+ except OSError:
+ log.warning("jarvisx: could not remove %s", path, exc_info=True)
+
+
+def _bgra_thumbnail(width: int, height: int, bgra: bytes) -> bytes:
+ import io # noqa: PLC0415
+
+ from PIL import Image # noqa: PLC0415
+
+ image = Image.frombuffer("RGB", (width, height), bytes(bgra), "raw", "BGRX", 0, 1)
+ image.thumbnail((560, 560), Image.Resampling.BILINEAR)
+ buffer = io.BytesIO()
+ image.save(buffer, format="JPEG", quality=84)
+ return buffer.getvalue()
+
+
+_service: JarvisXService | None = None
+
+
+def get_service() -> JarvisXService:
+ global _service
+ if _service is None:
+ _service = JarvisXService()
+ return _service
+
+
+def set_service_for_tests(service: JarvisXService | None) -> None:
+ global _service
+ _service = service
+
+
+__all__ = ["JarvisXService", "Result", "get_service", "set_service_for_tests"]
diff --git a/jarvis/jarvisx/store.py b/jarvis/jarvisx/store.py
new file mode 100644
index 0000000000..4dbe4bcd8a
--- /dev/null
+++ b/jarvis/jarvisx/store.py
@@ -0,0 +1,230 @@
+"""The Jarvis X library index — one SQLite table of captures.
+
+The pictures and videos themselves live in the user's save folder under their
+readable names; the index remembers what each file is (kind, mode, size,
+duration), where its preview thumbnail sits, and whether an annotated copy
+exists. A file deleted outside the app drops out of the index on the next
+listing, so the library never shows a tile that cannot load.
+
+Stdlib only. Every call opens its own short-lived connection under one lock:
+the store is touched from worker threads (``asyncio.to_thread``) and the
+recorder's finishing thread, never from the event loop itself.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import logging
+import sqlite3
+import threading
+from collections.abc import Iterator
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Literal
+
+log = logging.getLogger(__name__)
+
+Kind = Literal["image", "video"]
+Mode = Literal["region", "window", "fullscreen"]
+
+_SCHEMA = """
+CREATE TABLE IF NOT EXISTS items (
+ id TEXT PRIMARY KEY,
+ kind TEXT NOT NULL,
+ mode TEXT NOT NULL,
+ created_at TEXT NOT NULL,
+ width INTEGER NOT NULL,
+ height INTEGER NOT NULL,
+ duration_s REAL,
+ path TEXT NOT NULL,
+ thumb_path TEXT NOT NULL DEFAULT '',
+ edited_path TEXT
+);
+"""
+
+_COLUMNS = "id, kind, mode, created_at, width, height, duration_s, path, thumb_path, edited_path"
+_INSERT = f"INSERT OR REPLACE INTO items ({_COLUMNS}) VALUES (?,?,?,?,?,?,?,?,?,?)" # noqa: S608
+_SELECT_ONE = f"SELECT {_COLUMNS} FROM items WHERE id = ?" # noqa: S608
+_SELECT_RECENT = f"SELECT {_COLUMNS} FROM items ORDER BY rowid DESC LIMIT ?" # noqa: S608
+
+
+@dataclass(frozen=True, slots=True)
+class Item:
+ """One capture in the library."""
+
+ id: str
+ kind: Kind
+ mode: Mode
+ #: ISO-8601 with the local UTC offset, e.g. ``2026-09-29T14:03:22+02:00``.
+ created_at: str
+ width: int
+ height: int
+ duration_s: float | None
+ path: str
+ thumb_path: str = ""
+ edited_path: str | None = None
+
+ @property
+ def filename(self) -> str:
+ return Path(self.path).name
+
+ def has_edited(self) -> bool:
+ return bool(self.edited_path) and Path(str(self.edited_path)).is_file()
+
+ def to_json(self) -> dict[str, Any]:
+ """The REST shape (``/api/jarvisx/items``)."""
+ base = f"/api/jarvisx/items/{self.id}"
+ return {
+ "id": self.id,
+ "kind": self.kind,
+ "mode": self.mode,
+ "created_at": self.created_at,
+ "width": int(self.width),
+ "height": int(self.height),
+ "duration_s": None if self.duration_s is None else round(float(self.duration_s), 2),
+ "filename": self.filename,
+ "path": self.path,
+ "url": f"{base}/file",
+ "thumb_url": f"{base}/thumb",
+ "edited_url": f"{base}/edited" if self.has_edited() else None,
+ }
+
+
+class ItemStore:
+ """SQLite-backed index. ``root`` holds ``index.sqlite3`` and ``thumbs/``."""
+
+ def __init__(self, root: Path) -> None:
+ self.root = Path(root)
+ self._lock = threading.Lock()
+ self._ready = False
+
+ @property
+ def db_path(self) -> Path:
+ return self.root / "index.sqlite3"
+
+ @property
+ def thumbs_dir(self) -> Path:
+ return self.root / "thumbs"
+
+ def thumb_path_for(self, item_id: str) -> Path:
+ return self.thumbs_dir / f"{item_id}.jpg"
+
+ @contextlib.contextmanager
+ def _connect(self) -> Iterator[sqlite3.Connection]:
+ with self._lock:
+ if not self._ready:
+ self.root.mkdir(parents=True, exist_ok=True)
+ self.thumbs_dir.mkdir(parents=True, exist_ok=True)
+ conn = sqlite3.connect(self.db_path, timeout=5.0)
+ try:
+ if not self._ready:
+ conn.executescript(_SCHEMA)
+ self._ready = True
+ yield conn
+ conn.commit()
+ finally:
+ conn.close()
+
+ # ----------------------------------------------------------------- write
+ def add(self, item: Item) -> Item:
+ with self._connect() as conn:
+ conn.execute(
+ _INSERT,
+ (
+ item.id,
+ item.kind,
+ item.mode,
+ item.created_at,
+ int(item.width),
+ int(item.height),
+ item.duration_s,
+ item.path,
+ item.thumb_path,
+ item.edited_path,
+ ),
+ )
+ return item
+
+ def set_edited(self, item_id: str, edited_path: str) -> Item | None:
+ with self._connect() as conn:
+ conn.execute("UPDATE items SET edited_path = ? WHERE id = ?", (edited_path, item_id))
+ return self.get(item_id)
+
+ def delete(self, item_id: str) -> Item | None:
+ """Remove the row and every file it names. Returns the removed item."""
+ item = self.get(item_id)
+ if item is None:
+ return None
+ for name in (item.path, item.thumb_path, item.edited_path):
+ if not name:
+ continue
+ try:
+ Path(name).unlink(missing_ok=True)
+ except OSError:
+ # The row still goes: a locked file must not pin a dead tile.
+ log.warning("jarvisx: could not delete %s", name, exc_info=True)
+ with self._connect() as conn:
+ conn.execute("DELETE FROM items WHERE id = ?", (item_id,))
+ return item
+
+ # ------------------------------------------------------------------ read
+ def get(self, item_id: str) -> Item | None:
+ with self._connect() as conn:
+ row = conn.execute(_SELECT_ONE, (item_id,)).fetchone()
+ return _row_to_item(row) if row else None
+
+ def recent(self, limit: int = 100) -> list[Item]:
+ """Newest first. Rows whose file vanished are pruned on the way."""
+ limit = max(1, min(int(limit), 1000))
+ with self._connect() as conn:
+ rows = conn.execute(_SELECT_RECENT, (limit,)).fetchall()
+ items: list[Item] = []
+ gone: list[Item] = []
+ for row in rows:
+ item = _row_to_item(row)
+ (items if Path(item.path).is_file() else gone).append(item)
+ for item in gone:
+ log.info("jarvisx: %s was removed outside the app; dropping it", item.filename)
+ self.delete(item.id)
+ return items
+
+
+def _row_to_item(row: tuple[Any, ...]) -> Item:
+ kind = "video" if row[1] == "video" else "image"
+ mode = row[2] if row[2] in ("region", "window", "fullscreen") else "region"
+ return Item(
+ id=str(row[0]),
+ kind=kind,
+ mode=mode,
+ created_at=str(row[3]),
+ width=int(row[4]),
+ height=int(row[5]),
+ duration_s=None if row[6] is None else float(row[6]),
+ path=str(row[7]),
+ thumb_path=str(row[8] or ""),
+ edited_path=str(row[9]) if row[9] else None,
+ )
+
+
+_store: ItemStore | None = None
+_store_lock = threading.Lock()
+
+
+def get_store() -> ItemStore:
+ """The process-wide store under ``<data dir>/jarvisx``."""
+ global _store
+ with _store_lock:
+ if _store is None:
+ from jarvis.jarvisx.paths import library_dir # noqa: PLC0415
+
+ _store = ItemStore(library_dir())
+ return _store
+
+
+def set_store_for_tests(store: ItemStore | None) -> None:
+ global _store
+ with _store_lock:
+ _store = store
+
+
+__all__ = ["Item", "ItemStore", "Kind", "Mode", "get_store", "set_store_for_tests"]
diff --git a/jarvis/local_models/autostart.py b/jarvis/local_models/autostart.py
index b87cd4f7d0..8fe01306b1 100644
--- a/jarvis/local_models/autostart.py
+++ b/jarvis/local_models/autostart.py
@@ -230,7 +230,10 @@ async def run_once(
if not ok:
return record
record["started"] = True
- model = _chat_model(cfg)
+ # Local voice owns its own model and load order. Warming a dormant chat
+ # pick here competes with speech models and can evict the voice brain.
+ primary = str(getattr(getattr(cfg, "brain", None), "primary", "") or "")
+ model = _chat_model(cfg) if primary == SERVER_PROVIDER_ID else ""
if model and await (warm or _default_warm)(root, model, _keep_alive(cfg, model)):
record["warmed"] = model
return record
diff --git a/jarvis/mcp/jarvis_tools_server.py b/jarvis/mcp/jarvis_tools_server.py
index 75dfce2cb1..c75c71d5af 100644
--- a/jarvis/mcp/jarvis_tools_server.py
+++ b/jarvis/mcp/jarvis_tools_server.py
@@ -223,37 +223,41 @@ def build_server() -> Any:
async def _call_tool(name: str, arguments: dict[str, Any] | None) -> Any:
from jarvis.core.protocols import SupervisorToolRequest, current_chat_turn
+ def failed(message: str) -> Any:
+ return types.CallToolResult(
+ content=[types.TextContent(type="text", text=message)], isError=True
+ )
+
gateway = _gateway()
if gateway is None:
- return [types.TextContent(type="text", text="Jarvis is still starting up.")]
+ return failed("Jarvis is still starting up.")
if name in _WITHHELD:
- return [
- types.TextContent(
- type="text",
- text=(
- f"{name} is not available from the chat — you ARE the worker. "
- "Do the work yourself."
- ),
- )
- ]
+ return failed(
+ f"{name} is not available from the chat — you ARE the worker. Do the work yourself."
+ )
try:
- from jarvis.agent_chat.tool_context import restore_turn
+ from jarvis.agent_chat.tool_context import required_turn_id, restore_turn
entry = (await _session_wire_catalog()).get(name)
if entry is None:
- return [
- types.TextContent(type="text", text="Tool is no longer available in this chat.")
- ]
+ return failed("Tool is no longer available in this chat.")
name = str(entry.name)
with restore_turn(CHAT_SESSION_REF.get()):
turn = current_chat_turn.get()
+ if required_turn_id.get() is not None and turn is None:
+ return types.CallToolResult(
+ content=[types.TextContent(
+ type="text", text="The requesting chat turn has ended."
+ )],
+ isError=True,
+ )
from jarvis.core.task_agent import scope_for
task_scope = scope_for(CHAT_SESSION_REF.get())
if (CHAT_SESSION_REF.get() or "").startswith("task-") and task_scope is None:
- return [types.TextContent(type="text", text="The task grant has expired.")]
+ return failed("The task grant has expired.")
if task_scope is not None and name not in task_scope.names:
- return [types.TextContent(type="text", text="Tool outside the task grant.")]
+ return failed("Tool outside the task grant.")
trace_id = task_scope.trace_id if task_scope else uuid4()
if name == "society_browser" and turn is not None and task_scope is None:
try:
@@ -271,9 +275,7 @@ def build_server() -> Any:
result = await gateway.execute(name, dict(arguments or {}), request)
except Exception as exc: # noqa: BLE001 — a tool failure is data for the model, not a crash
log.warning("jarvis MCP: %s raised", name, exc_info=True)
- return [
- types.TextContent(type="text", text=f"Tool failed: {type(exc).__name__}: {exc}")
- ]
+ return failed(f"Tool failed: {type(exc).__name__}: {exc}")
if name == "society_browser":
success = bool(getattr(result, "success", False))
text = _render(result)
@@ -294,7 +296,10 @@ def build_server() -> Any:
return types.CallToolResult(
content=[types.TextContent(type="text", text=text)], isError=not success
)
- return [types.TextContent(type="text", text=_render(result))]
+ return types.CallToolResult(
+ content=[types.TextContent(type="text", text=_render(result))],
+ isError=not bool(getattr(result, "success", False)),
+ )
return server
diff --git a/jarvis/platform/clipboard_offer.py b/jarvis/platform/clipboard_offer.py
index 04db8424af..3de8504fd5 100644
--- a/jarvis/platform/clipboard_offer.py
+++ b/jarvis/platform/clipboard_offer.py
@@ -1,51 +1,20 @@
-"""Offer text on the Windows clipboard and OBSERVE who reads it.
-
-Why this exists
----------------
-Jarvis does not paste; it asks the foreground application to paste by sending
-a synthetic chord. Two things about that are unknowable from the outside:
-whether the chord means "paste" in that application at all (an xterm.js
-terminal left to itself swallows Ctrl+V as ``^V`` and cancels the browser's
-paste), and WHEN the application reads the clipboard — a WebView that pastes
-through an async IPC bridge can read hundreds of milliseconds later on a busy
-machine, after a timer-based "restore the previous clipboard" has already put
-the old content back (the 2026-08-24 BridgeMind report).
-
-Windows offers one mechanism that turns both into a measurement: **delayed
-rendering**. A clipboard owner may publish a format with a ``NULL`` handle;
-the data is requested through ``WM_RENDERFORMAT`` only when some process
-actually calls ``GetClipboardData``. Because the reader must hold the
-clipboard open at that moment, ``GetOpenClipboardWindow`` names it. So
-"did the paste land, and when?" becomes "which process read the clipboard
-after the chord went out?" — a fact, not a guess.
-
-The one limit, measured: the system renders ONCE. After the first reader the
-text is cached and later readers are served silently. A host with a clipboard
-watcher — a Remote Desktop client (``msrdc.exe`` reads within 5 ms of every
-write) or the clipboard-history service — therefore consumes the render before
-the chord is even sent, and from then on the offer is *blind*: a paste can no
-longer be proven absent, only (by polling for who holds the clipboard open)
-occasionally proven present. Callers must treat the two states differently;
-:mod:`jarvis.dictation.insert` does.
-
-Design
-------
-* One hidden message-only window on a dedicated thread owns the clipboard for
- the duration of the offer and answers ``WM_RENDERFORMAT`` with the text.
-* Every render request is recorded with the reader's pid and executable name.
-* On :meth:`ClipboardOffer.stop` the text is rendered for real before the
- window goes away (``WM_RENDERALLFORMATS`` contract), so the clipboard still
- holds the text afterwards — the "one Ctrl+V away" guarantee of
- :mod:`jarvis.dictation.insert` survives.
-* Losing ownership (another app copied something) is recorded, never raised.
-
-Windows only, imported lazily by its one caller. On any other OS
-:func:`available` is ``False`` and nothing here is constructed.
+"""Observe real Windows clipboard rendering without treating silence as failure.
+
+A message-only window owns a delayed Unicode text offer. The first successful
+render records the reader PID; cached reads produce no further events. Clipboard
+watchers can consume that first render, so no event is never proof that a paste
+failed, and merely opening the clipboard is never insertion evidence.
+
+The dedicated thread waits on native messages. Stopping renders the transcript
+for later manual paste. A caller with delivery evidence may request restoration;
+ownership is checked under the same clipboard lock as the replacement, preserving
+any newer user copy. All native imports are lazy and this helper is inert off Windows.
"""
from __future__ import annotations
import logging
+import os
import sys
import threading
import time
@@ -72,50 +41,13 @@ class ClipboardRead:
#: Seconds since the offer went up.
at: float
#: ``render`` — the system asked us for the text (a real ``GetClipboardData``);
- #: ``open`` — the process merely held the clipboard open, seen by polling.
- #: Once the text has been rendered and cached, ``open`` is the only
- #: evidence left (see :meth:`ClipboardOffer.rendered`).
+ #: Historical consumers may supply ``open``; it is not delivery evidence.
observed: str = "render"
def available() -> bool:
"""Can a delayed-rendering offer be made on this host?"""
- return sys.platform == "win32"
-
-
-def _exe_name(pid: int) -> str:
- """Executable name for *pid*, ``""`` when it cannot be read. Never raises."""
- if pid <= 0:
- return ""
- try:
- import ctypes # noqa: PLC0415 — lazy (HN-7)
- from ctypes import wintypes # noqa: PLC0415
-
- kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
- kernel32.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
- kernel32.OpenProcess.restype = wintypes.HANDLE
- kernel32.QueryFullProcessImageNameW.argtypes = [
- wintypes.HANDLE,
- wintypes.DWORD,
- wintypes.LPWSTR,
- ctypes.POINTER(wintypes.DWORD),
- ]
- kernel32.QueryFullProcessImageNameW.restype = wintypes.BOOL
- kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
- handle = kernel32.OpenProcess(0x1000, False, pid) # QUERY_LIMITED_INFORMATION
- if not handle:
- return ""
- try:
- buf = ctypes.create_unicode_buffer(1024)
- size = wintypes.DWORD(1024)
- if not kernel32.QueryFullProcessImageNameW(handle, 0, buf, ctypes.byref(size)):
- return ""
- return buf.value.replace("/", "\\").rsplit("\\", 1)[-1].lower()
- finally:
- kernel32.CloseHandle(handle)
- except Exception: # noqa: BLE001 — a nameless reader is still a reader
- log.debug("could not resolve exe name for pid %s", pid, exc_info=True)
- return ""
+ return os.name == "nt" and sys.platform == "win32"
class ClipboardOffer:
@@ -140,6 +72,8 @@ class ClipboardOffer:
self._read_event = threading.Event()
self._ready = threading.Event()
self._done = threading.Event()
+ self._stop_requested = threading.Event()
+ self._lifecycle_lock = threading.Lock()
self._thread: threading.Thread | None = None
self._hwnd: int = 0
self._started_at = 0.0
@@ -148,6 +82,8 @@ class ClipboardOffer:
#: True once the text was handed to a reader. From then on the system
#: caches it and answers later readers itself — no more render events.
self.rendered = False
+ self.restored = False
+ self._restore_text: str | None = None
self._wndproc_ref: object = None # keeps the ctypes callback alive
# -- public ---------------------------------------------------------
@@ -156,12 +92,20 @@ class ClipboardOffer:
"""Take the clipboard. ``True`` when the offer is up."""
if not available():
return False
- self._thread = threading.Thread(
- target=self._run, name="jarvis-clipboard-offer", daemon=True
- )
- self._thread.start()
- self._ready.wait(timeout_s)
- return self._ok
+ with self._lifecycle_lock:
+ if self._stop_requested.is_set():
+ return False
+ if self._thread is None:
+ self._thread = threading.Thread(
+ target=self._run, name="jarvis-clipboard-offer", daemon=True
+ )
+ self._thread.start()
+ if not self._ready.wait(timeout_s):
+ # A late worker must never take the clipboard after our caller
+ # already chose its fallback delivery route.
+ self.stop()
+ return False
+ return self._ok and not self._done.is_set()
def reads(self) -> list[ClipboardRead]:
with self._lock:
@@ -183,30 +127,55 @@ class ClipboardOffer:
return read
self._read_event.clear()
remaining = deadline - time.monotonic()
- if remaining <= 0 or self._done.is_set():
+ if remaining <= 0 or self._done.is_set() or self.lost_ownership:
return None
- self._read_event.wait(min(remaining, 0.05))
+ self._read_event.wait(remaining)
def elapsed(self) -> float:
"""Seconds since the offer went up."""
return time.monotonic() - self._started_at if self._started_at else 0.0
- def stop(self, *, timeout_s: float = 1.0) -> None:
- """Render the text for real and give the window up. Idempotent."""
- if self._thread is None:
+ def stop(self, *, timeout_s: float = 2.0, restore_text: str | None = None) -> None:
+ """Finish the offer, optionally restoring text only while we still own it."""
+ self._restore_text = restore_text
+ self._stop_requested.set()
+ thread = self._thread
+ if thread is None:
return
if self._hwnd:
try:
import ctypes # noqa: PLC0415
-
- ctypes.WinDLL("user32").PostMessageW(self._hwnd, _WM_APP_STOP, 0, 0)
+ from ctypes import wintypes # noqa: PLC0415
+
+ user32 = ctypes.WinDLL("user32", use_last_error=True)
+ user32.PostMessageW.argtypes = [
+ wintypes.HWND, wintypes.UINT, wintypes.WPARAM, wintypes.LPARAM,
+ ]
+ user32.PostMessageW.restype = wintypes.BOOL
+ if not user32.PostMessageW(self._hwnd, _WM_APP_STOP, 0, 0):
+ log.debug("clipboard offer stop message was refused")
except Exception: # noqa: BLE001 — the thread's own timeout covers it
log.debug("could not post stop to the clipboard offer window", exc_info=True)
- self._thread.join(timeout_s)
- self._thread = None
+ thread.join(timeout_s)
+ if not thread.is_alive():
+ self._thread = None
# -- thread body ----------------------------------------------------
+ def _restore_owned_text(self, user32, hwnd: int, render) -> None:
+ """Check ownership and replace text under one native clipboard lock."""
+ if self._restore_text is None or not user32.OpenClipboard(hwnd):
+ return
+ try:
+ if user32.GetClipboardOwner() == hwnd and user32.EmptyClipboard():
+ self.restored = render(self._restore_text)
+ if self.restored:
+ self._text = self._restore_text
+ else:
+ render()
+ finally:
+ user32.CloseClipboard()
+
def _run(self) -> None:
try:
self._pump()
@@ -219,6 +188,8 @@ class ClipboardOffer:
self._read_event.set()
def _pump(self) -> None:
+ if not available() or self._stop_requested.is_set():
+ return
import ctypes # noqa: PLC0415
from ctypes import wintypes # noqa: PLC0415
@@ -251,6 +222,17 @@ class ClipboardOffer:
]
user32.CreateWindowExW.restype = wintypes.HWND
user32.DestroyWindow.argtypes = [wintypes.HWND]
+ user32.IsWindow.argtypes = [wintypes.HWND]
+ user32.IsWindow.restype = wintypes.BOOL
+ user32.UnregisterClassW.argtypes = [wintypes.LPCWSTR, wintypes.HINSTANCE]
+ user32.UnregisterClassW.restype = wintypes.BOOL
+ user32.PeekMessageW.argtypes = [
+ ctypes.POINTER(wintypes.MSG), wintypes.HWND, wintypes.UINT, wintypes.UINT,
+ wintypes.UINT,
+ ]
+ user32.TranslateMessage.argtypes = [ctypes.POINTER(wintypes.MSG)]
+ user32.DispatchMessageW.argtypes = [ctypes.POINTER(wintypes.MSG)]
+ user32.DispatchMessageW.restype = ctypes.c_ssize_t
user32.OpenClipboard.argtypes = [wintypes.HWND]
user32.SetClipboardData.argtypes = [wintypes.UINT, wintypes.HANDLE]
user32.SetClipboardData.restype = wintypes.HANDLE
@@ -266,9 +248,9 @@ class ClipboardOffer:
kernel32.GetModuleHandleW.argtypes = [wintypes.LPCWSTR]
kernel32.GetModuleHandleW.restype = wintypes.HMODULE
- def render() -> bool:
+ def render(value: str | None = None) -> bool:
"""Hand the real text to whoever asked. Clipboard must be open."""
- buf = ctypes.create_unicode_buffer(self._text)
+ buf = ctypes.create_unicode_buffer(self._text if value is None else value)
size = ctypes.sizeof(buf)
handle = kernel32.GlobalAlloc(_GMEM_MOVEABLE, size)
if not handle:
@@ -298,12 +280,14 @@ class ClipboardOffer:
try:
if msg == _WM_RENDERFORMAT:
pid = reader_pid()
- read = ClipboardRead(pid=pid, exe=_exe_name(pid), at=self.elapsed())
- with self._lock:
- self._reads.append(read)
- self._read_event.set()
if int(wparam) == _CF_UNICODETEXT and render():
self.rendered = True
+ # Publish only successfully rendered data, and keep
+ # filesystem/process queries off the target's read.
+ read = ClipboardRead(pid=pid, exe="", at=self.elapsed())
+ with self._lock:
+ self._reads.append(read)
+ self._read_event.set()
return 0
if msg == _WM_RENDERALLFORMATS:
# We are about to stop owning the clipboard: leave the text
@@ -317,9 +301,10 @@ class ClipboardOffer:
return 0
if msg == _WM_DESTROYCLIPBOARD:
self.lost_ownership = True
+ self._read_event.set()
return 0
if msg == _WM_APP_STOP:
- user32.DestroyWindow(hwnd)
+ self._stop_requested.set()
return 0
if msg == _WM_DESTROY:
user32.PostQuitMessage(0)
@@ -363,6 +348,8 @@ class ClipboardOffer:
taken = False
for _attempt in range(10):
+ if self._stop_requested.is_set():
+ break
if user32.OpenClipboard(hwnd):
try:
user32.EmptyClipboard()
@@ -386,12 +373,8 @@ class ClipboardOffer:
self._ok = True
self._ready.set()
- # A message pump that also POLLS: once the text is rendered the system
- # serves later readers from its cache and sends no more render events,
- # so the only trace a paste leaves is the reader briefly holding the
- # clipboard open. Sampling that every millisecond is cheap and catches
- # a WebView's multi-format read comfortably; a missed sample only
- # means "no positive evidence", never a wrong claim.
+ # Clipboard opening is not a read or an insertion acknowledgment.
+ # Wait for native messages; never sample the clipboard every millisecond.
msg = wintypes.MSG()
user32.MsgWaitForMultipleObjectsEx.argtypes = [
wintypes.DWORD,
@@ -400,38 +383,28 @@ class ClipboardOffer:
wintypes.DWORD,
wintypes.DWORD,
]
- seen_open: set[int] = set()
running = True
try:
- while running:
+ while running and not self._stop_requested.is_set():
while user32.PeekMessageW(ctypes.byref(msg), None, 0, 0, 1):
if msg.message == 0x0012: # WM_QUIT
running = False
break
user32.TranslateMessage(ctypes.byref(msg))
user32.DispatchMessageW(ctypes.byref(msg))
- if not running:
+ if not running or self._stop_requested.is_set():
break
- holder = user32.GetOpenClipboardWindow()
- if holder and holder != hwnd:
- pid = wintypes.DWORD()
- user32.GetWindowThreadProcessId(holder, ctypes.byref(pid))
- if pid.value and pid.value not in seen_open:
- seen_open.add(int(pid.value))
- read = ClipboardRead(
- pid=int(pid.value),
- exe=_exe_name(int(pid.value)),
- at=self.elapsed(),
- observed="open",
- )
- with self._lock:
- self._reads.append(read)
- self._read_event.set()
# QS_ALLINPUT = 0x04FF, MWMO_INPUTAVAILABLE = 0x0004
- user32.MsgWaitForMultipleObjectsEx(0, None, 1, 0x04FF, 0x0004)
+ user32.MsgWaitForMultipleObjectsEx(0, None, 1000, 0x04FF, 0x0004)
finally:
- user32.UnregisterClassW(class_name, wc.hInstance)
- self._hwnd = 0
+ # A newer copy of identical text still belongs to its new owner.
+ try:
+ self._restore_owned_text(user32, hwnd, render)
+ finally:
+ if user32.IsWindow(hwnd):
+ user32.DestroyWindow(hwnd)
+ user32.UnregisterClassW(class_name, wc.hInstance)
+ self._hwnd = 0
__all__ = ["ClipboardOffer", "ClipboardRead", "available"]
diff --git a/jarvis/plugins/realtime/openai_realtime.py b/jarvis/plugins/realtime/openai_realtime.py
index 04b6da7f77..591a4270b2 100644
--- a/jarvis/plugins/realtime/openai_realtime.py
+++ b/jarvis/plugins/realtime/openai_realtime.py
@@ -362,6 +362,11 @@ class _OpenAIRealtimeSession:
_RESPONSE_STALL_S if response_start_timeout_s is None else response_start_timeout_s
),
)
+ # A local cascade can emit its text before the TTS has produced PCM.
+ # Its declared startup budget covers that first-audio preparation too.
+ self._first_audio_timeout_s = (
+ self._response_start_timeout_s if response_start_timeout_s is not None else None
+ )
# A capacity-one server must release the old pipeline before opening a
# replacement. Hosted endpoints retain make-before-break; self-hosted
# endpoints can opt into serial replacement plus bounded drain retries.
@@ -418,7 +423,7 @@ class _OpenAIRealtimeSession:
now = time.monotonic()
self._response_started_at = now
self._last_response_activity = now
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
self._last_item_id = ""
# Whether THIS response's transcript arrived as a delta stream. Servers
# split here: OpenAI streams it token by token, a self-hosted stack that
@@ -542,6 +547,8 @@ class _OpenAIRealtimeSession:
self._last_response_activity = time.monotonic()
if event_type in _RESPONSE_OUTPUT_EVENTS:
self._response_output_started = True
+ if event_type == "response.output_audio.delta":
+ self._response_audio_started = True
if event_type == "response.output_audio.delta":
self._last_item_id = str(getattr(event, "item_id", "") or "")
yield _ProviderEvent(
@@ -636,7 +643,7 @@ class _OpenAIRealtimeSession:
elif len(self._accepted_response_ids) == 1:
self._accepted_response_ids.pop()
self._response_idle.set()
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
usage = _usage_from_response(getattr(event, "response", None))
if usage is not None:
# Every response bills its own pass over the session context,
@@ -816,7 +823,7 @@ class _OpenAIRealtimeSession:
await self._conn.response.cancel()
finally:
self._response_idle.set()
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
async def send_tool_result(self, call_id: str, name: str, result: dict[str, Any]) -> None:
del name
@@ -848,7 +855,7 @@ class _OpenAIRealtimeSession:
now = time.monotonic()
self._response_started_at = now
self._last_response_activity = now
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
marker = uuid4().hex
self._pending_response_markers.add(marker)
response: dict[str, Any] = {
@@ -861,7 +868,7 @@ class _OpenAIRealtimeSession:
except BaseException:
self._pending_response_markers.discard(marker)
self._response_idle.set()
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
raise
@staticmethod
@@ -924,7 +931,7 @@ class _OpenAIRealtimeSession:
now = time.monotonic()
self._response_started_at = now
self._last_response_activity = now
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
self._server_heard_user_since_response = False
self._auto_adopted_unanswered_input = True
await self._rearm_session_contract()
@@ -1051,6 +1058,20 @@ class _OpenAIRealtimeSession:
return False
return time.monotonic() >= self._transcript_deadline
+ def _response_stall_budget_s(self) -> float:
+ contract = self._session_contract or {}
+ modalities = contract.get(
+ "output_modalities", contract.get("modalities", ["audio"])
+ )
+ if (
+ self._first_audio_timeout_s is not None
+ and not self._response_audio_started
+ and not self._response_had_tool_calls
+ and "audio" in modalities
+ ):
+ return self._first_audio_timeout_s
+ return _RESPONSE_STALL_S
+
def _response_lifecycle_stalled(self) -> bool:
"""Whether an in-flight response exceeded its current phase budget.
@@ -1068,7 +1089,8 @@ class _OpenAIRealtimeSession:
if self._response_idle.is_set():
return False
if self._response_output_started:
- return time.monotonic() - self._last_response_activity >= _RESPONSE_STALL_S
+ gap = time.monotonic() - self._last_response_activity
+ return gap >= self._response_stall_budget_s()
return time.monotonic() - self._response_started_at >= self._response_start_timeout_s
def _maybe_begin_rebuild(self) -> None:
@@ -1076,7 +1098,7 @@ class _OpenAIRealtimeSession:
if self._response_output_started:
reason = (
"in-flight response stopped producing events for "
- f"{_RESPONSE_STALL_S:.0f} s after output began — "
+ f"{self._response_stall_budget_s():.0f} s after output began — "
"response.done is not coming"
)
else:
@@ -1246,7 +1268,7 @@ class _OpenAIRealtimeSession:
now = time.monotonic()
self._response_started_at = now
self._last_response_activity = now
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
self._server_heard_user_since_response = False
self._auto_adopted_unanswered_input = False
self._response_idle.set()
@@ -1278,7 +1300,7 @@ class _OpenAIRealtimeSession:
self._pending_response_markers.clear()
self._accepted_response_ids.clear()
self._response_idle.set()
- self._response_output_started = False
+ self._response_output_started = self._response_audio_started = False
try:
if self._connection_is_open:
try:
diff --git a/jarvis/plugins/tool/delegate_to_agent.py b/jarvis/plugins/tool/delegate_to_agent.py
index fe9606f810..de8470d6b6 100644
--- a/jarvis/plugins/tool/delegate_to_agent.py
+++ b/jarvis/plugins/tool/delegate_to_agent.py
@@ -23,6 +23,7 @@ import logging
from collections.abc import Callable
from typing import Any, Final
+from jarvis.core.delegation import origin_metadata
from jarvis.core.protocols import ExecutionContext, ToolResult
from jarvis.society.communication import REPLY_POLICY_SCHEMA, select_reply_policy
@@ -133,6 +134,7 @@ class DelegateToAgentTool:
"to the team'. Use when a background task fits the persistent team; leave `agent` "
"empty to let the lead pick the agent whose hands fit the task. The agent works in "
"the background; you acknowledge now and its result follows reply_policy. "
+ "Stay available for conversation while it runs; completion is delivered asynchronously. "
"Never for inventory/status questions or tasks the user wants done right here. "
"Turn the user's intent into an actionable brief: objective, known target, relevant "
"context, scope, constraints and completion evidence. Do not merely paraphrase or "
@@ -225,7 +227,10 @@ class DelegateToAgentTool:
text=brief,
trace_id=f"voice:{ctx.trace_id.hex[:12]}",
msg_type=MsgType.ASSIGN,
- payload={"text": brief, "lang": lang, "refs": refs, "reply_policy": policy},
+ payload={
+ "text": brief, "refs": refs, "reply_policy": policy,
+ **origin_metadata(language=lang),
+ },
)
# The scheduler answered synchronously on the same trace: a CLAIM
# means the agent took it, a VETO says why not — say so, no waiting.
diff --git a/jarvis/realtime/local_server/brain_link.py b/jarvis/realtime/local_server/brain_link.py
index 608ce3ddbc..9aebdf1911 100644
--- a/jarvis/realtime/local_server/brain_link.py
+++ b/jarvis/realtime/local_server/brain_link.py
@@ -217,6 +217,8 @@ def list_brain_choices(
base = _ollama_base()
if current_model.endswith(_MANAGED_VOICE_SUFFIX):
current_model = current_model[: -len(_MANAGED_VOICE_SUFFIX)]
+ from jarvis.brain.ollama_inventory import is_hidden_alias
+
served = _ollama_models(base, timeout)
reachable = served is not None
choices: list[dict[str, object]] = []
@@ -225,7 +227,7 @@ def list_brain_choices(
# The managed 8k alias is an implementation detail derived from its
# base tag. Listing both made one selection look like two models and
# encouraged users to select an alias that is recreated automatically.
- if name.endswith(_MANAGED_VOICE_SUFFIX) or not _usable_tag(name):
+ if (is_hidden_alias(name) and name != current_model) or not _usable_tag(name):
continue
seen.add(name)
fits = _fits(size, usable_gb)
diff --git a/jarvis/realtime/local_server/supervisor.py b/jarvis/realtime/local_server/supervisor.py
index cd1ec07f08..2dc3bfa26b 100644
--- a/jarvis/realtime/local_server/supervisor.py
+++ b/jarvis/realtime/local_server/supervisor.py
@@ -895,10 +895,10 @@ def _boot_status(*, booting: bool) -> dict[str, object]:
)
expected = boot_progress.expected_boot_s(stats)
remaining: float | None = None
- # Past twice the historical boot time the countdown would be a lie;
- # showing only the stage is the honest degradation.
- if expected is not None and elapsed <= 2.0 * expected:
- remaining = max(5.0, expected - elapsed)
+ # Once the measured estimate is exceeded, show the loading stage rather
+ # than an endless five-second countdown that promises imminent readiness.
+ if expected is not None and elapsed < expected:
+ remaining = expected - elapsed
payload.update(
{
"starting": True,
diff --git a/jarvis/realtime/report_prompt.py b/jarvis/realtime/report_prompt.py
index aab27f123c..444f8a26ad 100644
--- a/jarvis/realtime/report_prompt.py
+++ b/jarvis/realtime/report_prompt.py
@@ -2,9 +2,8 @@
An announcement that carries an agent's report (``AnnouncementRequested.report``)
has two things: a short deterministic line (what classic TTS speaks) and the
-agent's full report. Coding panes and Jarvis agents no longer send one — their
-results are shown, not spoken (2026-09-30) — so only a caller that still
-attaches a report reaches this path. A live model is handed
+agent's full report. Explicit agent and coding-pane delegations attach their
+request and evidence; unsolicited background messages remain quiet. A live model is handed
both and asked to work out what the user actually needs to hear, instead of
reading either one out. Every live engine (the realtime wrapper, GPT-Live,
native Gemini) builds its request here, so they all ask the same thing.
@@ -56,6 +55,12 @@ def report_update_prompt(
"If the agent asks something, pass the question on clearly. Keep "
"success, failure and uncertainty exactly as the report states them "
"and never claim more than it says. "
+ "Several results may be grouped here. Briefly identify each agent or task, "
+ "combine related outcomes, and give blockers or questions priority. "
+ "Do not imply that unfinished sibling tasks are done. A stopped terminal "
+ "or exited process alone is not proof of success; say when its result "
+ "could not be verified. Use the original request and our conversation "
+ "to decide which details matter. "
"Say it as yourself, in exactly the same voice, tone, and pace as your "
"previous replies; do not imitate another person and do not change or "
"dramatize your voice. "
diff --git a/jarvis/society/agent_tools.py b/jarvis/society/agent_tools.py
index 2140db35ed..b9f423eefd 100644
--- a/jarvis/society/agent_tools.py
+++ b/jarvis/society/agent_tools.py
@@ -219,6 +219,10 @@ class MessageAgentTool:
FailureReason.BLOCKED_BY_POLICY, "no success reply was requested"
)
trace_id = parent.trace_id if parent is not None else None
+ if caller.agent_id == rt.lead_id and parent is None:
+ from jarvis.core.delegation import origin_metadata
+ config = getattr(ctx, "config", None) or {}
+ payload.update(origin_metadata(language=str(config.get("output_language") or "")))
env = await rt.say(
from_agent=caller.agent_id,
to_agent=target.agent_id,
diff --git a/jarvis/society/approvals.py b/jarvis/society/approvals.py
index 3455bbfd12..2a7d4a62e9 100644
--- a/jarvis/society/approvals.py
+++ b/jarvis/society/approvals.py
@@ -9,9 +9,9 @@ person decides.
Rule evaluation (``decide``), in order: global blacklist (not here — the
ToolExecutor raises before anything), the agent's ``require_approval``
-patterns (queue), its ``always_allow`` patterns (run when the tool's own
-tier is at most ``ask`` — the person's standing yes), then the tool's risk
-tier against the ceiling.
+patterns (queue), its explicit approval mode, then the legacy
+``always_allow`` patterns and risk ceiling. Bypass still blocks actions above
+an inherited ceiling; Always ask queues even safe reads.
Patterns are capability ids (``plugin:gmail``), ``capability:verb``
(``plugin:gmail:send``) or ``capability:*``.
"""
@@ -67,11 +67,21 @@ def decide(agent: AgentRecord, capability_id: str, tool_tier: str, *, verb: str
rules = agent.approval_rules
if any(matches(p, capability_id, verb) for p in rules.get("require_approval", [])):
return Verdict.QUEUE
+ mode = str(agent.approval_mode) if agent.approval_mode is not None else ""
+ if mode == "always_ask":
+ return Verdict.QUEUE
if any(matches(p, capability_id, verb) for p in rules.get("always_allow", [])):
- # An explicit always-allow rule is the person's standing "yes" for this
- # exact action (the card's "Always allow", agent-definition §3.4): it
- # runs up to the ask tier. Block stays block — it never had a card.
+ # The person's explicit standing yes can authorize this action up to
+ # the ask tier. Always ask above intentionally ignores standing yes.
return Verdict.RUN if _TIER_RANK.get(tool_tier, 1) <= 2 else Verdict.QUEUE
+ if mode == "bypass":
+ # Bypass removes prompts, not an inherited risk ceiling. An action
+ # above that ceiling is denied instead of being silently escalated.
+ if _TIER_RANK.get(tool_tier, 1) > _TIER_RANK[str(agent.permission_ceiling)]:
+ return Verdict.BLOCK
+ return Verdict.RUN
+ if mode == "ask":
+ return Verdict.RUN if tool_tier == "safe" else Verdict.QUEUE
ceiling = _TIER_RANK[str(agent.permission_ceiling)]
tier = _TIER_RANK.get(tool_tier, 1)
if tier <= ceiling and tool_tier != str(PermissionCeiling.ASK):
diff --git a/jarvis/society/chat_binding.py b/jarvis/society/chat_binding.py
index 15cfaa7089..a883f157e4 100644
--- a/jarvis/society/chat_binding.py
+++ b/jarvis/society/chat_binding.py
@@ -41,9 +41,8 @@ __all__ = [
SURFACE: Final[str] = "society"
-#: Permission ceiling → the Jarvis-ladder stance the session runs in
-#: (``jarvis/agent_chat/permissions.py``): safe reads only, monitor may edit
-#: and runs commands after asking, ask asks for everything mutating.
+#: Legacy permission ceiling → the pre-migration Jarvis stance. Explicit
+#: approval modes instead come from the roster's own persisted choice.
_CEILING_TO_MODE: Final[dict[str, str]] = {
str(PermissionCeiling.SAFE): "plan",
str(PermissionCeiling.MONITOR): "accept-edits",
@@ -80,15 +79,30 @@ def _workspace(cfg: Any, agent: AgentRecord) -> str:
def ensure_session(svc: Any, cfg: Any, agent: AgentRecord) -> Any:
"""The agent's canonical session, created or re-seated to the roster row."""
from jarvis.agent_chat.effort import default_effort
- from jarvis.agent_chat.permissions import ladder_key, normalize_permission
+ from jarvis.agent_chat.permissions import (
+ ladder_key,
+ normalize_permission,
+ society_mode_supported,
+ )
from jarvis.agent_chat.service import resolve_runner
provider, model, effort = pair_for(cfg, agent)
+ runner = resolve_runner(provider, surface=SURFACE)
+ if agent.approval_mode is not None and not society_mode_supported(
+ runner, str(agent.approval_mode)
+ ):
+ raise PermissionError(
+ f"{runner} cannot provide an actionable approval for {agent.approval_mode}"
+ )
session_id = agent.session_id
existing = svc.store.get_session(session_id)
if existing is None:
- ladder = ladder_key(SURFACE, resolve_runner(provider, surface=SURFACE))
- mode = normalize_permission(ladder, _CEILING_TO_MODE[str(agent.permission_ceiling)])
+ ladder = ladder_key(SURFACE, runner)
+ if agent.approval_mode is None:
+ # A pre-migration row keeps the old Jarvis ladder, including Plan.
+ mode = normalize_permission("jarvis", _CEILING_TO_MODE[str(agent.permission_ceiling)])
+ else:
+ mode = normalize_permission(ladder, str(agent.approval_mode))
return svc.store.create_session(
provider=provider,
model=model,
@@ -104,6 +118,8 @@ def ensure_session(svc: Any, cfg: Any, agent: AgentRecord) -> Any:
svc.store.reseat_session(session_id, provider=provider, model=model or existing.model)
existing = svc.store.get_session(session_id)
updates: dict[str, str] = {}
+ if agent.approval_mode is not None and existing.permission_mode != str(agent.approval_mode):
+ updates["permission_mode"] = str(agent.approval_mode)
if getattr(existing, "account_id", "") != agent.account_id:
updates["account_id"] = agent.account_id
if effort and existing.effort != effort:
diff --git a/jarvis/society/coding_tool.py b/jarvis/society/coding_tool.py
index 91593d8c2d..e8aa1a09d6 100644
--- a/jarvis/society/coding_tool.py
+++ b/jarvis/society/coding_tool.py
@@ -174,9 +174,18 @@ class CodingSessionTool:
record = {"fingerprint": fingerprint}
await self.runtime.store.set_meta(key, json.dumps(record))
# Cancellation leaves a durable pending record. Retrying cannot repeat the side effect.
- result = await self._perform(
- gateway, args, owner_session, trace_id=str(getattr(ctx, "trace_id", ""))
- )
+ from jarvis.core.delegation import current_delegation_origin, origin_metadata
+
+ config = getattr(ctx, "config", None) or {}
+ origin = origin_metadata(language=str(config.get("output_language") or ""))
+ owns_result = self.agent_id == "jarvis" and args.get("action") in {"send", "respond"}
+ token = current_delegation_origin.set(origin if owns_result else None)
+ try:
+ result = await self._perform(
+ gateway, args, owner_session, trace_id=str(getattr(ctx, "trace_id", ""))
+ )
+ finally:
+ current_delegation_origin.reset(token)
record["result"] = {
"success": result.success,
"output": result.output,
diff --git a/jarvis/society/conversation.py b/jarvis/society/conversation.py
index 36ff4f85ee..4d8ee1ad39 100644
--- a/jarvis/society/conversation.py
+++ b/jarvis/society/conversation.py
@@ -6,6 +6,7 @@ reads and a parameterized text-search fallback on every supported OS.
from __future__ import annotations
+import asyncio
import json
import logging
import re
@@ -30,11 +31,40 @@ def event_text(event: dict[str, Any]) -> str:
class ConversationArchive:
- def __init__(self, path: Path) -> None:
- path.parent.mkdir(parents=True, exist_ok=True)
- self._db = sqlite3.connect(path, check_same_thread=False)
- self._db.row_factory = sqlite3.Row
+ def __init__(self, path: Path, *, defer_open: bool = False) -> None:
+ self._path = path
self._lock = threading.RLock()
+ self._connection: sqlite3.Connection | None = None
+ self._closed = False
+ self.fts_available = False
+ if not defer_open:
+ self.open()
+
+ @property
+ def _db(self) -> sqlite3.Connection:
+ self.open()
+ assert self._connection is not None
+ return self._connection
+
+ def open(self) -> None:
+ """Initialize storage once. Async owners must call this off their loop."""
+ with self._lock:
+ if self._closed:
+ raise sqlite3.ProgrammingError("The conversation archive is closed")
+ if self._connection is not None:
+ return
+ self._path.parent.mkdir(parents=True, exist_ok=True)
+ self._connection = sqlite3.connect(self._path, check_same_thread=False)
+ self._connection.row_factory = sqlite3.Row
+ try:
+ self._initialize_schema()
+ except BaseException:
+ self._connection.close()
+ self._connection = None
+ self.fts_available = False
+ raise
+
+ def _initialize_schema(self) -> None:
self._db.executescript("""
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS messages (
@@ -80,7 +110,10 @@ class ConversationArchive:
def close(self) -> None:
with self._lock:
- self._db.close()
+ self._closed = True
+ if self._connection is not None:
+ self._connection.close()
+ self._connection = None
def ingest(self, session: str, events: list[dict[str, Any]]) -> None:
with self._lock, self._db:
@@ -104,6 +137,7 @@ class ConversationArchive:
return []
limit = max(1, min(20, limit))
with self._lock:
+ self.open()
if self.fts_available:
expression = " OR ".join('"' + w + '"' for w in words[:32])
rows = self._db.execute(
@@ -267,11 +301,11 @@ async def prepare_history(
archive = runtime.conversations
sid = session.session_id
- archive.ingest(sid, events)
+ await asyncio.to_thread(archive.ingest, sid, events)
window = max(1024, int(getattr(provider, "context_window", 0) or 32768))
# Reserve room for the system/tool surface, current request and generated output.
budget = max(1024, window * 3 - prompt_chars - len(query))
- through, summary = archive.checkpoint(sid)
+ through, summary = await asyncio.to_thread(archive.checkpoint, sid)
remaining = [e for e in events if int(e.get("seq") or 0) > through]
size = sum(len(event_text(e)) + 32 for e in remaining)
if size + len(summary) > budget or len(summary) > budget // 3:
@@ -347,7 +381,7 @@ async def prepare_history(
raise ValueError("An empty summary cannot replace conversation context")
if boundary is not None:
through = boundary
- archive.save_checkpoint(sid, through, summary)
+ await asyncio.to_thread(archive.save_checkpoint, sid, through, summary)
remaining = [e for e in events if int(e.get("seq") or 0) > through]
if retained_request is not None and int(retained_request.get("seq") or 0) <= through:
# Keep the last human request verbatim even if its unusually long
@@ -355,7 +389,7 @@ async def prepare_history(
remaining.insert(0, retained_request)
history = brain_history_from_events(remaining, max_messages=None)
if summary:
- recalled = archive.search(sid, query, limit=4)
+ recalled = await asyncio.to_thread(archive.search, sid, query, limit=4)
excerpts = "\n".join(
f"{h['source']}: {h['text'][:1500]}" for h in recalled if h["seq"] <= through
)
diff --git a/jarvis/society/conversation_tool.py b/jarvis/society/conversation_tool.py
index 7e388466fe..9a696ece0d 100644
--- a/jarvis/society/conversation_tool.py
+++ b/jarvis/society/conversation_tool.py
@@ -2,6 +2,7 @@
from __future__ import annotations
+import asyncio
from typing import Any
from jarvis.core.protocols import ToolResult
@@ -35,14 +36,14 @@ class ConversationRecallTool:
service = self._runtime.chat_service()
archive = self._runtime.conversations
if service is not None:
- archive.ingest(session, service.store.list_events(session))
+ events = await asyncio.to_thread(service.store.list_events, session)
+ await asyncio.to_thread(archive.ingest, session, events)
limit = max(1, min(20, int(args.get("limit") or 5)))
if str(args.get("query") or "").strip():
- return ToolResult(
- True, {"hits": archive.search(session, str(args["query"]), limit=limit)}
- )
- events = archive.read(
- session, after_seq=max(0, int(args.get("after_seq") or 0)), limit=limit
+ hits = await asyncio.to_thread(archive.search, session, str(args["query"]), limit=limit)
+ return ToolResult(True, {"hits": hits})
+ events = await asyncio.to_thread(
+ archive.read, session, after_seq=max(0, int(args.get("after_seq") or 0)), limit=limit
)
return ToolResult(
True, {"events": events, "next_after_seq": events[-1]["seq"] if events else None}
diff --git a/jarvis/society/events.py b/jarvis/society/events.py
index a1d9da0b24..69d095f764 100644
--- a/jarvis/society/events.py
+++ b/jarvis/society/events.py
@@ -21,6 +21,7 @@ from pydantic import BaseModel, ConfigDict, Field
from jarvis.missions.ids import uuid7_str
__all__ = [
+ "AgentApprovalMode",
"AgentState",
"ApprovalState",
"BrowserMode",
@@ -119,6 +120,14 @@ class PermissionCeiling(StrEnum):
ASK = "ask"
+class AgentApprovalMode(StrEnum):
+ """Per-agent approval behavior; capability denies and global blocks still apply."""
+
+ BYPASS = "bypass"
+ ASK = "ask"
+ ALWAYS_ASK = "always_ask"
+
+
class BrowserMode(StrEnum):
"""``own`` = the agent's persistent Chromium profile; ``attach`` = the
person's running Chrome over CDP."""
diff --git a/jarvis/society/inherit.py b/jarvis/society/inherit.py
index 602b1153f3..f5146a702d 100644
--- a/jarvis/society/inherit.py
+++ b/jarvis/society/inherit.py
@@ -3,13 +3,14 @@
When a society agent creates a teammate — typed chat or a subscription CLI
calling ``society-create-agent`` — the new row starts on the creator's model
seat (provider, model, effort, subscription account) and may not exceed the
-creator's permission ceiling, grant mode, grants or denies. The Agents UI
+creator's explicit approval mode, permission ceiling, grant mode, grants or denies. The Agents UI
sends those fields itself, so an explicit choice still wins, and a blank
create from the UI (no society session) keeps the usual defaults.
"""
from __future__ import annotations
+import logging
from typing import Any
from .events import GrantMode, PermissionCeiling
@@ -22,6 +23,8 @@ __all__ = [
"session_agent_id",
]
+log = logging.getLogger(__name__)
+
_CEILING_RANK: dict[str, int] = {
str(PermissionCeiling.SAFE): 0,
str(PermissionCeiling.MONITOR): 1,
@@ -52,7 +55,7 @@ def caller_session_id() -> str | None:
if stamped:
return str(stamped)
except Exception: # noqa: BLE001 — optional; a missing MCP server is not a create failure
- pass
+ log.debug("Society caller has no MCP session context", exc_info=True)
try:
from jarvis.core.chat_turn import current_chat_turn
@@ -60,7 +63,7 @@ def caller_session_id() -> str | None:
if turn is not None and turn.session_id:
return str(turn.session_id)
except Exception: # noqa: BLE001 — optional; a missing turn is not a create failure
- pass
+ log.debug("Society caller has no chat turn context", exc_info=True)
return None
@@ -105,6 +108,18 @@ def inherit_creator_fields(fields: dict[str, Any], creator: AgentRecord) -> dict
else creator_ceiling
)
+ # A child may ask more often, but cannot silently bypass a parent's
+ # explicit Ask setting. Legacy parents made no approval-mode choice, so
+ # their children use the new Bypass default under the inherited ceiling.
+ creator_approval = str(creator.approval_mode or "bypass")
+ requested_approval = str(fields.get("approval_mode") or creator_approval)
+ approval_rank = {"always_ask": 0, "ask": 1, "bypass": 2}
+ if requested_approval not in approval_rank:
+ requested_approval = creator_approval
+ fields["approval_mode"] = min(
+ (creator_approval, requested_approval), key=lambda mode: approval_rank[mode]
+ )
+
creator_mode = str(creator.grant_mode)
requested_mode = str(fields.get("grant_mode") or "").strip()
if creator_mode == str(GrantMode.ALLOWLIST):
diff --git a/jarvis/society/remote.py b/jarvis/society/remote.py
index 40fa3d61b0..fe2ceeb0b2 100644
--- a/jarvis/society/remote.py
+++ b/jarvis/society/remote.py
@@ -212,7 +212,8 @@ async def placement_for_session(session: Any) -> tuple[str, str] | None:
session_id = str(getattr(session, "session_id", "") or "")
if not session_id.startswith("society:"):
return None
- agent_id = session_id.split(":", 1)[1]
+ # Scheduled executions retain their owner's remote placement too.
+ agent_id = session_id.split(":", 2)[1]
from .runtime import current_runtime
runtime = current_runtime()
diff --git a/jarvis/society/roster.py b/jarvis/society/roster.py
index 657824dd0c..d29638d14e 100644
--- a/jarvis/society/roster.py
+++ b/jarvis/society/roster.py
@@ -24,6 +24,7 @@ from pydantic import ValidationError
from .companion import validate_avatar_companion
from .events import (
+ AgentApprovalMode,
AgentState,
BrowserMode,
Checkpoint,
@@ -119,6 +120,8 @@ class AgentRecord:
created_ms: int
updated_ms: int
stats: dict[str, Any] = field(default_factory=dict)
+ # NULL on pre-migration rows: their existing chat mode remains authoritative.
+ approval_mode: AgentApprovalMode | None = None
#: Where the agent's work executes: ``None`` = this computer, else the id
#: of a connected machine (``jarvis.computers``) reached over SSH.
computer_id: str | None = None
@@ -155,6 +158,7 @@ class AgentRecord:
"wiki_namespace": self.wiki_namespace,
"knowledge_scope": str(self.knowledge_scope),
"permission_ceiling": str(self.permission_ceiling),
+ "approval_mode": str(self.approval_mode) if self.approval_mode else None,
"approval_rules": {
"require_approval": list(self.approval_rules.get("require_approval", [])),
"always_allow": list(self.approval_rules.get("always_allow", [])),
@@ -200,6 +204,9 @@ class AgentRecord:
wiki_namespace=str(row.get("wiki_namespace") or ""),
knowledge_scope=KnowledgeScope(str(row.get("knowledge_scope") or "shared")),
permission_ceiling=PermissionCeiling(str(row.get("permission_ceiling") or "monitor")),
+ approval_mode=(
+ AgentApprovalMode(str(row["approval_mode"])) if row.get("approval_mode") else None
+ ),
approval_rules={
"require_approval": [str(x) for x in rules.get("require_approval", [])],
"always_allow": [str(x) for x in rules.get("always_allow", [])],
@@ -210,8 +217,8 @@ class AgentRecord:
browser_allowed_domains=[
str(x) for x in _loads(row.get("browser_allowed_domains_json"), [])
],
- created_ms=int(row.get("created_ms") or 0),
computer_id=str(row["computer_id"]) if row.get("computer_id") else None,
+ created_ms=int(row.get("created_ms") or 0),
updated_ms=int(row.get("updated_ms") or 0),
)
@@ -239,6 +246,7 @@ _EDITABLE: Final[frozenset[str]] = frozenset(
"wiki_namespace",
"knowledge_scope",
"permission_ceiling",
+ "approval_mode",
"approval_rules",
"daily_budget_usd",
"max_concurrent_runs",
@@ -329,6 +337,8 @@ def _coerce(field_name: str, value: Any) -> Any:
return _enum(KnowledgeScope, value, field_name)
if field_name == "permission_ceiling":
return _enum(PermissionCeiling, value, field_name)
+ if field_name == "approval_mode":
+ return _enum(AgentApprovalMode, value, field_name)
if field_name == "browser_mode":
return _enum(BrowserMode, value, field_name)
if field_name == "browser_allowed_domains":
@@ -445,13 +455,18 @@ class Roster:
return await self._hydrate(existing), False
tier_value = Tier(_enum(Tier, tier, "tier"))
agent_id = slugify(clean_name)
+ if tier_value is Tier.LEAD and agent_id != LEAD_AGENT_ID:
+ raise RosterError(
+ FailureReason.TIER_NOT_ALLOWED, "exactly one lead exists and it is Jarvis"
+ )
if tier_value is Tier.LEAD and fields.get("computer_id"):
raise RosterError(
FailureReason.TIER_NOT_ALLOWED, "the lead always runs on this computer"
)
- if tier_value is Tier.LEAD and agent_id != LEAD_AGENT_ID:
+ if tier_value is Tier.LEAD and "approval_mode" in fields:
raise RosterError(
- FailureReason.TIER_NOT_ALLOWED, "exactly one lead exists and it is Jarvis"
+ FailureReason.BLOCKED_BY_POLICY,
+ "the Jarvis lead uses the app chat permission policy",
)
same_slug = await self._store.get_agent_row(agent_id)
if same_slug is not None:
@@ -471,6 +486,12 @@ class Roster:
"updated_ms": now,
"workspace_dir": f"society/{agent_id}/workspace",
"wiki_namespace": f"society/{agent_id}/",
+ "approval_mode": (None if tier_value is Tier.LEAD else str(AgentApprovalMode.BYPASS)),
+ "permission_ceiling": (
+ str(PermissionCeiling.MONITOR)
+ if tier_value is Tier.LEAD
+ else str(PermissionCeiling.ASK)
+ ),
}
if tier_value is Tier.ORCHESTRATOR:
row["max_concurrent_runs"] = 3
@@ -522,6 +543,11 @@ class Roster:
raise RosterError(FailureReason.BLOCKED_BY_POLICY, f"field {key!r} is not editable")
if key == "tier" and agent_id == LEAD_AGENT_ID and str(value) != str(Tier.LEAD):
raise RosterError(FailureReason.TIER_NOT_ALLOWED, "Jarvis stays the lead")
+ if key == "approval_mode" and agent_id == LEAD_AGENT_ID:
+ raise RosterError(
+ FailureReason.BLOCKED_BY_POLICY,
+ "the Jarvis lead uses the app chat permission policy",
+ )
if key == "computer_id" and agent_id == LEAD_AGENT_ID and value:
raise RosterError(
FailureReason.TIER_NOT_ALLOWED, "the lead always runs on this computer"
diff --git a/jarvis/society/runtime.py b/jarvis/society/runtime.py
index 9a90818078..6dcef3d039 100644
--- a/jarvis/society/runtime.py
+++ b/jarvis/society/runtime.py
@@ -18,6 +18,7 @@ the ownership map letting the bridge attribute every mission event back.
from __future__ import annotations
import asyncio
+import inspect
import logging
from collections.abc import Callable, Iterable, Mapping
from contextlib import AsyncExitStack
@@ -169,9 +170,12 @@ class SocietyRuntime:
# ``event_publish`` is the app bus the WebSocket forwards (server.py hands
# it in); without it the engine falls back to the process default bus.
self.checkpoints = CheckpointEngine(self, publish=event_publish)
+ self._publish_event = event_publish
#: The society's one memory service; every touch moves the figure to the Memory House.
self.memory = SocietyMemory(self, on_activity=self.checkpoints.note_memory_activity)
- self.conversations = ConversationArchive(self._data_dir / "society-conversations.db")
+ self.conversations = ConversationArchive(
+ self._data_dir / "society-conversations.db", defer_open=True
+ )
self._review_lock = asyncio.Lock()
#: The Quest Board: the person's jobs, routed to one taker, read back off the board.
self.quests = Quests(self)
@@ -250,6 +254,10 @@ class SocietyRuntime:
async def _start_runtime(self) -> SocietyRuntime:
await self.store.open()
self._require_open_owner()
+ # FTS backfill can take seconds on a busy disk. It must never stall
+ # HTTP, microphone controls or the first response from the desktop.
+ await asyncio.to_thread(self.conversations.open)
+ self._require_open_owner()
self.scheduler._budget = self._get_budget() # noqa: SLF001 — the runtime owns its scheduler
self.scheduler.attach()
self._delivery_unsubscribe = self.store.bus.subscribe_all(self._delivery_failed)
@@ -325,50 +333,74 @@ class SocietyRuntime:
)
if request is None:
return
- if request.msg_type is MsgType.ASSIGN and "reply_policy" in request.payload:
+ if (
+ request.msg_type is MsgType.ASSIGN and "reply_policy" in request.payload
+ and env.msg_type is MsgType.ANSWER
+ ):
# The turn watcher owns this completion, including semantic blockers.
return
- status = str(env.payload.get("reply_status") or "done")
+ status = (
+ "needs_input" if env.msg_type in (MsgType.QUERY, MsgType.PROPOSE)
+ else str(env.payload.get("reply_status") or "done")
+ )
if not should_report(request, status):
return
- await self.announce_lead_message(sender, env)
+ await self.announce_lead_message(sender, env, request=request)
except Exception: # noqa: BLE001 - a silent message is a lost courtesy, not a lost result
log.warning("society: lead incoming announcement failed", exc_info=True)
- async def announce_lead_message(self, sender: AgentRecord, env: SocietyEnvelope) -> None:
- """Close the loop on an agent talking back to Jarvis: show the person.
+ async def announce_lead_message(
+ self, sender: AgentRecord, env: SocietyEnvelope, *, request: SocietyEnvelope,
+ ) -> None:
+ """Return a correlated answer or clarification to its requesting conversation."""
+ status = (
+ "needs_input" if env.msg_type in (MsgType.QUERY, MsgType.PROPOSE)
+ else str(env.payload.get("reply_status") or "done")
+ )
+ await self._deliver_lead_result(
+ sender, request, status=status, summary=env.text,
+ kind="society_message", message_type=str(env.msg_type).lower(),
+ )
- Twin of report_to_lead for conversational envelopes (SAY / QUERY /
- ANSWER / PROPOSE): the text is posted as a notice into the newest
- front-page chat, where the order came from. It is not spoken — no
- background result is read aloud (maintainer decision 2026-09-30).
- A failed notice never fails the run.
- """
- summary = " ".join(str(env.text or "").split())
- if not summary:
- return
+ async def _deliver_lead_result(
+ self, target: AgentRecord, request: SocietyEnvelope, *, status: str,
+ summary: str, kind: str, message_type: str = "",
+ ) -> None:
+ from jarvis.core.delegation import result_announcement
+
+ event = result_announcement(
+ source="society.lead", request_id=request.event_id, name=target.name,
+ request=request.text, status=status, report=summary,
+ language=str(request.payload.get("lang") or ""),
+ )
svc = self._get_chat()
post = getattr(svc, "post_notice", None)
- if svc is None or post is None:
- return
- try:
- sessions = svc.store.list_sessions(limit=1, surface="jarvis")
- if sessions:
- await post(
- sessions[0].session_id,
- {
- "kind": "society_message",
- "agent_id": sender.agent_id,
- "agent_name": sender.name,
- "msg_type": str(env.msg_type).lower(),
- "status": "done",
- "text": summary[:1000],
- "session_id": sender.session_id,
- "trace_id": env.trace_id,
- },
- )
- except Exception: # noqa: BLE001 - a lost notice is a lost courtesy, not a lost result
- log.warning("society: incoming message notice for the lead chat failed", exc_info=True)
+ if post is not None:
+ try:
+ session_id = str(request.payload.get("reply_session_id") or "")
+ if not session_id:
+ sessions = svc.store.list_sessions(limit=1, surface="jarvis")
+ session_id = sessions[0].session_id if sessions else ""
+ # Never redirect a result from a deleted chat into an unrelated one.
+ session = svc.store.get_session(session_id) if session_id else None
+ if session is not None and session.surface == "jarvis":
+ await post(session_id, {
+ "kind": kind, "agent_id": target.agent_id, "agent_name": target.name,
+ "msg_type": message_type, "status": status, "text": summary[:1000],
+ "session_id": target.session_id, "trace_id": request.trace_id,
+ "assignment_id": request.event_id, "report": event.report,
+ })
+ except Exception: # A chat failure must not suppress the voice result.
+ log.warning("society: result notice delivery failed", exc_info=True)
+ surface = request.payload.get("reply_surface")
+ if surface == "voice" or (surface is None and request.trace_id.startswith("voice:")):
+ if self._publish_event is not None:
+ try:
+ value = self._publish_event(event)
+ if inspect.isawaitable(value):
+ await value
+ except Exception: # The durable board and chat still retain the report.
+ log.warning("society: result announcement failed", exc_info=True)
async def _deliver_pending(self) -> None:
"""Recover committed messages and retry busy chats without opening sockets."""
@@ -408,13 +440,13 @@ class SocietyRuntime:
async with AsyncExitStack() as cleanup:
cleanup.callback(clear_runtime)
cleanup.push_async_callback(self.store.close)
+ cleanup.push_async_callback(asyncio.to_thread, self.conversations.close)
for release in (
self.world_feed.detach,
self.quests.detach,
self.checkpoints.detach,
self.bridge.detach,
self.scheduler.detach,
- self.conversations.close,
):
cleanup.callback(release)
cleanup.push_async_callback(self.browser.close)
@@ -458,7 +490,7 @@ class SocietyRuntime:
import json
events = json.loads(completion.events_json)
- self.conversations.ingest(session.session_id, events)
+ await asyncio.to_thread(self.conversations.ingest, session.session_id, events)
await self._complete_message_reply(session, completion, events)
terminal = [e for e in events if e.get("kind") == "turn_finished"]
if not terminal or terminal[-1].get("payload", {}).get("status") not in {
@@ -483,7 +515,8 @@ class SocietyRuntime:
# Reviewing them again wastes a model call and can
# duplicate a standing instruction as a conflicting memory.
return
- if self.conversations.queue_review(
+ if await asyncio.to_thread(
+ self.conversations.queue_review,
session.session_id,
completion.turn.turn_id,
events,
@@ -799,18 +832,24 @@ class SocietyRuntime:
summary = (final_text or error or "turn finished").strip()
# A successful model turn can still report an unfinished task. Use the
# correlated, typed report, never a keyword guess over the final prose.
- reports = [
+ correlated = [
item
for item in await self.store.events_for_trace(env.trace_id)
if item.parent_event_id == env.event_id
and item.from_agent == target.agent_id
and item.to_agent == env.from_agent
- and item.msg_type is MsgType.ANSWER
- and item.payload.get("reply_status") == "blocked"
+ ]
+ reports = [item for item in correlated if item.msg_type is MsgType.ANSWER
+ and item.payload.get("reply_status") == "blocked"]
+ questions = [
+ item for item in correlated if item.msg_type in (MsgType.QUERY, MsgType.PROPOSE)
]
if reports:
status = "blocked"
error = summary = reports[-1].text
+ elif questions:
+ status = "blocked"
+ error = summary = questions[-1].text
try:
await self.store.append_and_publish(
SocietyEnvelope(
@@ -833,7 +872,7 @@ class SocietyRuntime:
)
except Exception: # noqa: BLE001 - the slot is free either way; the loss is one RESULT row
log.warning("society: RESULT for %s could not be written", run_id, exc_info=True)
- if env.from_agent == LEAD_AGENT_ID:
+ if env.from_agent == LEAD_AGENT_ID and (reports or not questions):
await self.report_to_lead(target, env, status=status, summary=summary)
digest = TurnDigest(
task=env.text or str(env.payload.get("task") or ""),
@@ -857,34 +896,18 @@ class SocietyRuntime:
Jarvis delegates by voice or from the front-page chat and acknowledges
at once ("Scout is on it"); the work then ends on the board, where the
person only sees it by opening the Agents section. So the RESULT of a
- lead-assigned task is also posted where the order came from: a notice
- in the newest front-page chat. It is not spoken — no background result
- is read aloud (maintainer decision 2026-09-30). A failed notice never
- fails the run.
+ lead-assigned task returns to its original chat. Voice requests also
+ enter the live conversation's floor-aware result queue. No separate
+ model is called to phrase the report. A failed notice never fails the run.
"""
- if not should_report(env, status):
- return
- svc = self._get_chat()
- post = getattr(svc, "post_notice", None)
- if svc is None or post is None:
+ if (
+ env.from_agent != LEAD_AGENT_ID or env.to_agent != target.agent_id
+ or not should_report(env, status)
+ ):
return
- try:
- sessions = svc.store.list_sessions(limit=1, surface="jarvis")
- if sessions:
- await post(
- sessions[0].session_id,
- {
- "kind": "society_result",
- "agent_id": target.agent_id,
- "agent_name": target.name,
- "status": status,
- "text": summary[:1000],
- "session_id": target.session_id,
- "trace_id": env.trace_id,
- },
- )
- except Exception: # noqa: BLE001 - a lost notice is a lost courtesy, not a lost result
- log.warning("society: result notice for the lead chat failed", exc_info=True)
+ await self._deliver_lead_result(
+ target, env, status=status, summary=summary, kind="society_result",
+ )
def pick_agent(self, task: str) -> AgentRecord | None:
"""The active agent whose hands fit ``task`` best, or ``None``.
diff --git a/jarvis/society/society_schema.sql b/jarvis/society/society_schema.sql
index 90e5b9db7e..9f4d5e1af6 100644
--- a/jarvis/society/society_schema.sql
+++ b/jarvis/society/society_schema.sql
@@ -38,6 +38,8 @@ CREATE TABLE IF NOT EXISTS society_agents (
CHECK (knowledge_scope IN ('shared', 'own')),
permission_ceiling TEXT NOT NULL DEFAULT 'monitor'
CHECK (permission_ceiling IN ('safe', 'monitor', 'ask')),
+ approval_mode TEXT DEFAULT NULL
+ CHECK (approval_mode IN ('bypass', 'ask', 'always_ask')),
approval_rules_json TEXT NOT NULL DEFAULT '{}',
daily_budget_usd REAL NOT NULL DEFAULT 2.0,
browser_mode TEXT NOT NULL DEFAULT 'own'
diff --git a/jarvis/society/store.py b/jarvis/society/store.py
index 2e52844fc1..49ec42e64e 100644
--- a/jarvis/society/store.py
+++ b/jarvis/society/store.py
@@ -135,6 +135,13 @@ class SocietyStore:
"TEXT NOT NULL DEFAULT '[]'"
)
log.info("society store: migration applied — added browser_allowed_domains")
+ if "approval_mode" not in existing:
+ # NULL preserves legacy sessions until the person chooses a mode.
+ await self.conn.execute(
+ "ALTER TABLE society_agents ADD COLUMN approval_mode TEXT DEFAULT NULL "
+ "CHECK (approval_mode IN ('bypass', 'ask', 'always_ask'))"
+ )
+ log.info("society store: migration applied — added approval_mode")
if "computer_id" not in existing:
# NULL = this computer; every existing agent keeps running here.
await self.conn.execute(
diff --git a/jarvis/society/surface.py b/jarvis/society/surface.py
index 8b2d57d402..e9333918c8 100644
--- a/jarvis/society/surface.py
+++ b/jarvis/society/surface.py
@@ -68,8 +68,14 @@ _OWN_PREFIX: Final[str] = "society_"
#: through its own contained shell (never the free-cwd shell tools).
_SOCIETY_DENIED: Final[frozenset[str]] = frozenset(
{
- "wiki-ingest", "run-shell", "run_shell", "RunCommand",
- "remember", "update_profile", "profile-update", "update-profile",
+ "wiki-ingest",
+ "run-shell",
+ "run_shell",
+ "RunCommand",
+ "remember",
+ "update_profile",
+ "profile-update",
+ "update-profile",
}
)
@@ -554,22 +560,59 @@ def society_tool_filter(session: Any) -> Callable[[dict[str, Tool]], dict[str, T
focus=agent.focus,
denies=agent.denies,
)
- # Every granted hand obeys the agent's own approval rules and ceiling
- # (agent-definition §3.4): the gate rides on the executor's per-call
- # tier hook, so the chat card and the queue stay the one approval path.
+ # Every granted hand obeys the agent's own approval rules and mode.
+ # The gate rides on the executor's per-call tier hook, so the chat
+ # card and the queue stay the one approval path.
picked = {
name: cast(Tool, _GatedTool(tool, agent, cap_id))
for name, tool in picked.items()
if (cap_id := capability_id_for_tool(name)) is not None
}
ordered: dict[str, Tool] = {}
- ordered.update(own)
+ if agent.approval_mode is not None:
+ ordered.update(
+ {
+ name: cast(
+ Tool,
+ _GatedTool(tool, agent, capability_id_for_tool(name) or "core:society"),
+ )
+ for name, tool in own.items()
+ # Asking the user IS the person's decision; gating it
+ # behind an approval card would ask twice.
+ if name != ASK_USER_TOOL_NAME
+ }
+ )
+ if ASK_USER_TOOL_NAME in own:
+ ordered[ASK_USER_TOOL_NAME] = own[ASK_USER_TOOL_NAME]
+ else:
+ ordered.update(own)
ordered.update(picked)
return ordered
return _apply
+def requires_explicit_approval(session_id: str, tool_name: str, args: dict[str, Any]) -> bool:
+ """Keep an agent's explicit ask rules effective even in Bypass mode."""
+ from .approvals import matches
+
+ rt = current_runtime()
+ agent_id = agent_id_of(session_id)
+ if rt is None or agent_id is None:
+ return True
+ agent = rt.cached_agent(agent_id)
+ if agent is None:
+ return True
+ bare = tool_name.split("__", 2)[-1] if tool_name.startswith("mcp__") else tool_name
+ capability = capability_id_for_tool(bare)
+ if capability is None:
+ return True
+ return any(
+ matches(pattern, capability, _verb_of(args))
+ for pattern in agent.approval_rules.get("require_approval", [])
+ )
+
+
async def society_system_extra(cfg: Any, brain: Any, session: Any) -> str:
rt = current_runtime()
agent_id = agent_id_of(getattr(session, "session_id", "") or "")
diff --git a/jarvis/speech/pipeline.py b/jarvis/speech/pipeline.py
index 40d3d342e0..46e5b43d1f 100644
--- a/jarvis/speech/pipeline.py
+++ b/jarvis/speech/pipeline.py
@@ -392,11 +392,13 @@ _READBACK_KINDS: frozenset[str] = frozenset(
#: Readback sources that never speak outside a call (see ``_is_agent_reply``).
#: A mission the user just asked for is not here: its answer may still punch
-#: through the hangup gate (AD-OE5/OE6). A coding pane or a Jarvis agent that
-#: finishes is not here either: since 2026-09-30 neither is spoken at all, only
-#: shown (pane badge and bell, the agent chat notice).
+#: through the hangup gate (AD-OE5/OE6). Explicit delegations wait for a call
+#: and a conversational pause; their results never open a call themselves.
_HELD_FOR_CALL_SOURCES: frozenset[str] = frozenset(
{
+ "society.lead",
+ "agentic_ide.readback",
+ "delegation.batch",
"tasks.runner",
"workflows.runner",
"workflows.scheduler",
@@ -1396,11 +1398,15 @@ _DICTATION_TAIL_REREAD_BACK_S = 0.5
# How long the release waits for the incremental polish worker to finish the
# windows it already has — normally the last one, whose formatting started the
-# moment it was read. One configured polish budget plus a little: past that
-# the formatter is not answering and the whole-text pass (which has its own
-# ceiling and fails open to the raw text) takes over.
+# moment it was read. Past that the worker is retired and its completed prefix
+# is retained, leaving only unfinished text for the bounded final polish pass.
_DICTATION_PREFIX_POLISH_WAIT_S = 3.0
+# Dictation notifications update UI state; they must not inherit the bus's
+# five-second observer timeout before a finished transcript can be delivered.
+_DICTATION_EVENT_TIMEOUT_S = 1.0
+_DICTATION_POLISH_CANCEL_WAIT_S = 0.25
+
class _SessionInputBuffer:
"""Replayable bounded handoff for one continuously captured mic stream.
@@ -3826,11 +3832,14 @@ class SpeechPipeline:
):
self._last_answer_floor_monotonic = time.monotonic()
self._turn_state = new_state
+ if new_state is TurnTakingState.LISTENING:
+ self._schedule_delegation_results()
if new_state is TurnTakingState.IDLE:
- retry = getattr(self, "_agent_reply_retry_task", None)
- if retry is not None and retry is not asyncio.current_task():
- retry.cancel()
- await asyncio.gather(retry, return_exceptions=True)
+ for task_name in ("_agent_reply_retry_task", "_delegation_result_task"):
+ retry = getattr(self, task_name, None)
+ if retry is not None and retry is not asyncio.current_task():
+ retry.cancel()
+ await asyncio.gather(retry, return_exceptions=True)
await self._transition(self._supervisor_state_for_turn(new_state))
# Turn-boundary: the floor has cleared → flush any announcements that
# were deferred while the user was speaking (AD-OE6 zero-silent-drop).
@@ -4973,9 +4982,24 @@ class SpeechPipeline:
"Realtime announcement context mirror failed",
exc_info=True,
)
+ from jarvis.core.delegation import RESULT_SOURCES, ResultInbox
+
+ if event.source_layer in RESULT_SOURCES:
+ inbox = getattr(self, "_delegation_inbox", None)
+ if inbox is None:
+ inbox = self._delegation_inbox = ResultInbox()
+ inbox.add(event)
+ self._schedule_delegation_results()
+ return
if is_agent_reply:
if event == getattr(self, "_agent_reply_inflight", None):
return
+ if (
+ event.source_layer == "delegation.batch"
+ and self._turn_state is not TurnTakingState.LISTENING
+ ):
+ self._defer_agent_reply(event)
+ return
if self._agent_reply_needs_session():
self._defer_agent_reply(event)
log.info("Agent reply retained until an unmuted voice session is available")
@@ -5387,6 +5411,8 @@ class SpeechPipeline:
chunks,
should_play=lambda: (
not self._agent_reply_needs_session()
+ and (event.source_layer != "delegation.batch"
+ or self._turn_state is TurnTakingState.LISTENING)
if is_agent_reply
else getattr(self, "_turn_state", TurnTakingState.IDLE)
is not TurnTakingState.JARVIS_SPEAKING
@@ -5429,6 +5455,49 @@ class SpeechPipeline:
session = getattr(self, "_active_realtime_handle", None)
return session is not None
+ def _schedule_delegation_results(self) -> None:
+ """One coalescing task; a user's conversation always keeps the floor."""
+ inbox = getattr(self, "_delegation_inbox", None)
+ task = getattr(self, "_delegation_result_task", None)
+ if (
+ inbox is None or not inbox.pending
+ or (task is not None and not task.done())
+ or self._agent_reply_needs_session()
+ or self._turn_state is not TurnTakingState.LISTENING
+ or getattr(self, "_agent_reply_inflight", None) is not None
+ ):
+ return
+ self._delegation_result_task = asyncio.create_task(
+ self._flush_delegation_results(), name="delegation-results"
+ )
+
+ async def _flush_delegation_results(self) -> None:
+ from jarvis.core.delegation import BATCH_WINDOW_S
+
+ event = None
+ try:
+ await asyncio.sleep(BATCH_WINDOW_S)
+ if (
+ self._agent_reply_needs_session()
+ or self._turn_state is not TurnTakingState.LISTENING
+ or getattr(self, "_agent_reply_inflight", None) is not None
+ ):
+ return
+ event = self._delegation_inbox.take()
+ if event is not None:
+ await self._on_announcement(event)
+ except asyncio.CancelledError:
+ if event is not None:
+ self._defer_agent_reply(event)
+ raise
+ except Exception:
+ if event is not None:
+ self._defer_agent_reply(event)
+ log.warning("Delegation result delivery deferred", exc_info=True)
+ finally:
+ self._delegation_result_task = None
+ self._schedule_delegation_results()
+
async def _deliver_announcement_via_realtime(
self,
event: AnnouncementRequested,
@@ -11833,6 +11902,8 @@ class SpeechPipeline:
stt_models: list[str] = []
detected_languages: list[str] = []
stt_latency_ms = 0.0
+ warmup_wait_ms = 0.0
+ stt_queue_wait_ms = 0.0
stt_calls = 0
final_window_count = 0
# Windows whose transcript stopped at a mid-recording pause and were
@@ -11906,6 +11977,7 @@ class SpeechPipeline:
prefix_polish_windows = 0
prefix_polish_deltas = 0
prefix_polish_failed = False
+ prefix_polish_stopped = False
prefix_polish_task: asyncio.Task[None] | None = None
probe_task: asyncio.Task[None] | None = None
@@ -12013,7 +12085,7 @@ class SpeechPipeline:
# ``stt_failures`` is appended to, never rebound, so it needs no
# ``nonlocal`` — the list object itself is the shared state.
nonlocal stt, stt_error, stt_error_detail, session_language
- nonlocal stt_calls, stt_latency_ms
+ nonlocal stt_calls, stt_latency_ms, warmup_wait_ms, stt_queue_wait_ms
ceiling = max(
float(getattr(self, "_stt_final_timeout_s", 8.0) or 8.0),
(len(pcm) / bytes_per_second)
@@ -12026,8 +12098,14 @@ class SpeechPipeline:
# Startup/live-switch warm-up and this call share one native task.
# Joining it avoids both an 11 s cold final decode and a concurrent
# model call that would return TranscribeBusy (AP-24).
+ queued_at = time.perf_counter()
async with stt_gate:
- stt = await self._join_dictation_warmup(stt)
+ stt_queue_wait_ms += (time.perf_counter() - queued_at) * 1000.0
+ warmup_started = time.perf_counter()
+ try:
+ stt = await self._join_dictation_warmup(stt)
+ finally:
+ warmup_wait_ms += (time.perf_counter() - warmup_started) * 1000.0
if probe:
inference_active.set()
call_started = time.perf_counter()
@@ -12380,7 +12458,7 @@ class SpeechPipeline:
continue
last_published = live
try:
- await self._publish_event(
+ await self._publish_dictation_event(
DictationTranscript(
source_layer="speech.dictation",
text=live,
@@ -12678,7 +12756,7 @@ class SpeechPipeline:
nonlocal prefix_polish_deltas, prefix_polish_failed
from jarvis.dictation.merge import merge_transcripts
- while not prefix_polish_failed:
+ while not prefix_polish_failed and not prefix_polish_stopped:
reading = final_reads.get(prefix_polish_windows)
if reading is None:
return
@@ -12696,6 +12774,8 @@ class SpeechPipeline:
delta = merged[len(prefix_polish_raw) :].strip()
if delta:
piece, _status = await _polish_delta(delta, prefix_polish_text)
+ if prefix_polish_stopped:
+ return
if piece:
prefix_polish_text = " ".join(
part for part in (prefix_polish_text, piece) if part
@@ -12707,7 +12787,7 @@ class SpeechPipeline:
def _kick_prefix_polish() -> None:
"""Start the formatting worker unless it is running or given up."""
nonlocal prefix_polish_task
- if prefix_polish_failed or not final_quality_pass:
+ if prefix_polish_failed or prefix_polish_stopped or not final_quality_pass:
return
if not bool(getattr(cfg, "polish", True)) or bool(getattr(cfg, "translate", False)):
return
@@ -12720,10 +12800,11 @@ class SpeechPipeline:
async def _settle_prefix_polish() -> None:
"""Let the worker finish the windows it already has — normally the
last one, started the moment it was read — within one polish budget.
- Past that the whole-text pass takes over rather than making the
- user wait on a formatter that is not answering.
+ Past that, retain the completed prefix and retire its worker. The
+ finish path formats only the remaining tail instead of repeating
+ every completed window while the old formatter is still running.
"""
- nonlocal prefix_polish_failed
+ nonlocal prefix_polish_failed, prefix_polish_stopped
_kick_prefix_polish()
task = prefix_polish_task
if task is None:
@@ -12733,10 +12814,26 @@ class SpeechPipeline:
asyncio.shield(task), timeout=_DICTATION_PREFIX_POLISH_WAIT_S
)
except TimeoutError:
- prefix_polish_failed = True
+ prefix_polish_stopped = True
+ task.cancel()
+ done, _pending = await asyncio.wait(
+ (task,), timeout=_DICTATION_POLISH_CANCEL_WAIT_S
+ )
+
+ def consume(done_task: asyncio.Task) -> None:
+ if done_task.cancelled():
+ return
+ if done_task.exception() is not None:
+ log.debug("Retired dictation formatter finished with an error")
+
+ if done:
+ consume(task)
+ else:
+ # No late result may mutate the captured prefix above.
+ task.add_done_callback(consume)
log.info(
"incremental dictation polish did not finish within %.1fs; "
- "formatting the whole text instead.",
+ "keeping its completed prefix and formatting the remaining tail.",
_DICTATION_PREFIX_POLISH_WAIT_S,
)
except asyncio.CancelledError:
@@ -13179,8 +13276,9 @@ class SpeechPipeline:
# sets an event and does not know when the stream actually stopped.
# A hangup skips it: nothing will be transcribed, and
# ``DictationCompleted`` follows immediately.
+ capture_closed_at = time.perf_counter()
if not hung_up:
- await self._publish_event(
+ await self._publish_dictation_event(
DictationTranscribing(source_layer="speech.dictation")
)
@@ -13296,6 +13394,10 @@ class SpeechPipeline:
f"final_windows:{final_window_count}",
f"final_windows_prefetched:{final_prefetched}",
f"release_wait_ms:{release_wait_ms}",
+ f"warmup_wait_ms:{round(warmup_wait_ms)}",
+ f"stt_queue_wait_ms:{round(stt_queue_wait_ms)}",
+ "post_recording_wait_ms:"
+ f"{round((time.perf_counter() - capture_closed_at) * 1000.0)}",
f"truncation_repairs:{truncation_repairs}",
f"tail_repairs:{tail_repairs}",
f"pause_trim_ms:{round(pause_trim_bytes * 1000 / bytes_per_second)}",
@@ -13522,6 +13624,26 @@ class SpeechPipeline:
return ""
return counts[0][0]
+ async def _publish_dictation_event(self, event: Any) -> bool:
+ """Deliver to healthy observers in order without waiting on a dead UI.
+
+ The bus fans out concurrently, so an unrelated stalled observer cannot
+ delay a healthy recipient. The bounded await still joins cancellation;
+ no publication task is left behind to reorder a later dictation.
+ """
+ try:
+ await asyncio.wait_for(
+ self._publish_event(event), timeout=_DICTATION_EVENT_TIMEOUT_S
+ )
+ except TimeoutError:
+ log.warning(
+ "Dictation event %s exceeded %.1fs; abandoning stalled observers.",
+ type(event).__name__,
+ _DICTATION_EVENT_TIMEOUT_S,
+ )
+ return False
+ return True
+
async def _finish_dictation(
self,
*,
@@ -13596,6 +13718,7 @@ class SpeechPipeline:
(``[dictation].keep_failed_audio``), which is what a later Restore
transcribes again.
"""
+ finish_started = time.perf_counter()
stt_error = normalize_stt_failure(stt_error)
cleaned = raw_text
removed_words = 0
@@ -13906,20 +14029,52 @@ class SpeechPipeline:
log.debug("dictation target resolution failed", exc_info=True)
resolved_target = "insert"
- # Publish the final transcript before inserting: the app's own fields,
- # the chat composer and the bar all listen for it, and they should
- # update even if insertion fails.
- try:
- await self._publish_event(
- DictationTranscript(
- source_layer="speech.dictation",
- text=cleaned,
- is_final=True,
- target=resolved_target,
+ formatting_ms = round((time.perf_counter() - finish_started) * 1000.0)
+ notification_ms = 0
+ insertion_ms = 0
+ final_notification_status = "complete"
+ insert_result = None
+
+ async def notify_final() -> None:
+ nonlocal notification_ms, final_notification_status
+ started = time.perf_counter()
+ try:
+ complete = await self._publish_dictation_event(
+ DictationTranscript(
+ source_layer="speech.dictation",
+ text=cleaned,
+ is_final=True,
+ target=resolved_target,
+ )
)
- )
+ if not complete:
+ final_notification_status = "observer_timeout"
+ except Exception:
+ final_notification_status = "failed"
+ log.warning("dictation final publish failed", exc_info=True)
+ finally:
+ notification_ms = round((time.perf_counter() - started) * 1000.0)
+
+ async def insert_external() -> None:
+ nonlocal insert_result, insertion_ms
+ started = time.perf_counter()
+ try:
+ insert_result = await asyncio.to_thread(self._insert_dictation, cleaned)
+ except Exception:
+ log.warning("dictation insertion failed", exc_info=True)
+ finally:
+ insertion_ms = round((time.perf_counter() - started) * 1000.0)
+
+ # Healthy UI clients receive the final text even if OS insertion fails.
+ # An external paste starts alongside notification: its destination must
+ # not drift while an unrelated UI observer is holding up the bus.
+ try:
+ if resolved_target == "insert" and cleaned.strip() and not hung_up:
+ await asyncio.gather(notify_final(), insert_external())
+ else:
+ await notify_final()
except Exception as exc: # noqa: BLE001
- log.debug("dictation final publish failed: %s", exc)
+ log.warning("dictation final delivery failed: %s", exc)
outcome_name = "chat"
detail = ""
@@ -13944,10 +14099,15 @@ class SpeechPipeline:
# an unexplained empty result after the user clearly spoke.
detail = rejected_detail
elif resolved_target == "insert":
- insert_result = await asyncio.to_thread(self._insert_dictation, cleaned)
- outcome_name = insert_result.status
- detail = insert_result.detail
- method = insert_result.method
+ if insert_result is None:
+ outcome_name = "failed"
+ detail = "The text could not be inserted."
+ if bool(getattr(cfg, "history_enabled", True)):
+ detail += " It is available in dictation history."
+ else:
+ outcome_name = insert_result.status
+ detail = insert_result.detail
+ method = insert_result.method
# A dictation that delivered SOME words and permanently lost others is
# not a success, whichever way the surviving fragment was delivered.
@@ -14023,12 +14183,21 @@ class SpeechPipeline:
dropped_audio_s,
)
+ delivery_audit = (
+ *stt_audit,
+ f"formatting_wait_ms:{formatting_ms}",
+ f"final_notification_wait_ms:{notification_ms}",
+ f"insertion_wait_ms:{insertion_ms}",
+ f"finish_wait_ms:{round((time.perf_counter() - finish_started) * 1000.0)}",
+ f"final_notification:{final_notification_status}",
+ )
+
# Mark the turn closed BEFORE the publish attempt: this flag answers
# "does the teardown still owe a terminal event", and a publish that
# raised is not a reason to fire a second, contradictory completion.
self._dictation_completion_published = True
try:
- await self._publish_event(
+ await self._publish_dictation_event(
DictationCompleted(
source_layer="speech.dictation",
text=cleaned,
@@ -14049,7 +14218,7 @@ class SpeechPipeline:
stt_latency_ms=max(0, int(stt_latency_ms)),
stt_calls=max(0, int(stt_calls)),
stt_errors=stt_errors,
- stt_audit=stt_audit,
+ stt_audit=delivery_audit,
audio_sample_rate_hz=max(0, int(audio_sample_rate_hz)),
audio_rms=max(0.0, float(audio_rms)),
audio_clipping_ratio=max(0.0, float(audio_clipping_ratio)),
@@ -14080,7 +14249,7 @@ class SpeechPipeline:
stt_latency_ms=max(0, int(stt_latency_ms)),
stt_calls=max(0, int(stt_calls)),
stt_errors=stt_errors,
- stt_audit=stt_audit,
+ stt_audit=delivery_audit,
audio_sample_rate_hz=max(0, int(audio_sample_rate_hz)),
audio_rms=max(0.0, float(audio_rms)),
audio_clipping_ratio=max(0.0, float(audio_clipping_ratio)),
diff --git a/jarvis/ui/desktop_app.py b/jarvis/ui/desktop_app.py
index 344181361c..8f3958bdc6 100644
--- a/jarvis/ui/desktop_app.py
+++ b/jarvis/ui/desktop_app.py
@@ -109,6 +109,11 @@ DETACHABLE_VIEWS: dict[str, str] = {
# picture is something you keep looking at WHILE you carry on working, and
# on a second monitor it stops competing with the section that produced it.
"visualization": "Visualization",
+ # The Jarvis X annotation editor. Never a section of the main window: it
+ # opens from a capture's thumbnail card or the library, always on one
+ # item (``&item=<id>``), and ONE editor window is reused — opening another
+ # capture navigates it instead of stacking windows (open_jarvisx_editor).
+ "jarvisx-editor": "Jarvis X Editor",
}
META_FILE_PATH = DATA_DIR / ".jarvis-running"
#: Timeout for the initial lock acquire, in seconds. 0 = non-blocking,
@@ -4850,7 +4855,7 @@ class DesktopApp:
return resolve_theme(self._configured_theme())
- def open_detached_window(self, view: str) -> dict[str, Any]:
+ def open_detached_window(self, view: str, query: str = "") -> dict[str, Any]:
"""Open ``view`` in its own solo window — MUST run on a worker thread.
pywebview materializes runtime windows only from a thread whose name is
@@ -4862,11 +4867,22 @@ class DesktopApp:
honest degrade on hosts whose webview backend cannot create runtime
windows.
"""
- fallback = f"/?view={view}&solo=1"
+ suffix = f"&{query}" if query else ""
+ fallback = f"/?view={view}&solo=1{suffix}"
if view not in DETACHABLE_VIEWS:
return {"ok": False, "reason": "unknown_view"}
existing = self._detached_windows.get(view)
if existing is not None:
+ if query:
+ # Same window, new content: navigate it rather than open a
+ # second one (the title is the window's identity on Windows).
+ try:
+ existing.load_url(f"{self._url()}{fallback}")
+ except Exception as exc: # noqa: BLE001
+ from loguru import logger
+
+ logger.opt(exception=exc).warning("Detached '{}' could not navigate", view)
+ return {"ok": False, "reason": f"{type(exc).__name__}: {exc}"}
_bring_window_to_front_by_title(self._detached_title(view))
return {"ok": True, "already_open": True, "view": view}
if self._window is None and not self._detached_windows:
@@ -4880,7 +4896,7 @@ class DesktopApp:
title = self._detached_title(view)
window = webview.create_window(
title,
- f"{self._url()}/?view={view}&solo=1",
+ f"{self._url()}{fallback}",
width=1100,
height=750,
min_size=(800, 520),
@@ -4910,6 +4926,16 @@ class DesktopApp:
self._publish_detached_event_threadsafe(view, opened=True)
return {"ok": True, "already_open": False, "view": view}
+ def open_jarvisx_editor(self, item_id: str) -> dict[str, Any]:
+ """Open (or re-point) the Jarvis X editor window on one capture.
+
+ Worker-thread only, like :meth:`open_detached_window`: the card click
+ and ``POST /api/jarvisx/items/<id>/open-editor`` both reach it through
+ ``asyncio.to_thread``. ``item_id`` is the library's hex id, validated
+ by the caller, so it is safe to put into the URL as is.
+ """
+ return self.open_detached_window("jarvisx-editor", query=f"item={item_id}")
+
def close_detached_window(self, view: str) -> dict[str, Any]:
"""Close the detached window for ``view`` — worker-thread only.
@@ -5065,6 +5091,9 @@ class DesktopApp:
def _hook_main_window_lifecycle(self) -> None:
"""Attach the closing/closed contract to the current main window."""
+ from jarvis.ui.winforms_errors import register_winforms_error_logging
+
+ register_winforms_error_logging(self._window)
self._window.events.closing += self._on_window_closing
self._window.events.closed += self._on_main_window_closed
# Real paths for dropped files/folders (see jarvis/ui/native_drop.py).
diff --git a/jarvis/ui/icon_utils.py b/jarvis/ui/icon_utils.py
index f5e1c6f958..15e0bc380f 100644
--- a/jarvis/ui/icon_utils.py
+++ b/jarvis/ui/icon_utils.py
@@ -1605,7 +1605,7 @@ def ensure_linux_desktop_entry(applications_dir: Path | None = None) -> bool:
"[Desktop Entry]\n"
"Type=Application\n"
f"Name={escape_value(APP_DISPLAY_NAME)}\n"
- "Comment=Voice-driven meta-orchestrator\n"
+ "Comment=Turn your computer into an AI agent\n"
f"Exec={exec_value(sys.executable, ('-m', _LAUNCHER_MODULE, *_LAUNCHER_ARGS))}\n"
f"Path={escape_value(str(PROJECT_ROOT))}\n"
"Terminal=false\n"
diff --git a/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.offscreen.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.offscreen.test.tsx
index 8ed8e939cb..94beb1ab4e 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.offscreen.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.offscreen.test.tsx
@@ -39,7 +39,7 @@ vi.mock("@xterm/xterm", () => ({
options: Record<string, unknown> = {};
unicode = { activeVersion: "" };
buffer = {
- active: { type: "normal", baseY: 0, viewportY: 0 },
+ active: { type: "normal", baseY: 0, viewportY: 0, cursorX: 0, cursorY: 0, getLine: () => undefined },
};
// A pane takes over the terminal's protocol replies on mount, because the
// backend answers those instead (see ./terminalQueries). A stand-in without
@@ -63,6 +63,9 @@ vi.mock("@xterm/xterm", () => ({
getSelection() {
return "";
}
+ getSelectionPosition() { return undefined; }
+ onSelectionChange() { return { dispose() {} }; }
+ onRender() { return { dispose() {} }; }
onData() {
return { dispose() {} };
}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.test.tsx
index edc3b858fa..69c4fec198 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.test.tsx
@@ -6,6 +6,8 @@ const terminalHarness = vi.hoisted(() => ({
host: { current: null as HTMLElement | null },
observe: vi.fn(),
fit: vi.fn(),
+ clearTextureAtlas: vi.fn(),
+ spacingAtOpen: [] as unknown[],
/** What the terminal reports after a fit — a test moves it to grow the pane. */
size: { cols: 80, rows: 24 },
/** Every frame the pane hands its socket. Returns whether it went out. */
@@ -70,7 +72,7 @@ vi.mock("@xterm/xterm", () => ({
}
get buffer() {
return {
- active: { type: terminalHarness.bufferType, ...terminalHarness.viewport },
+ active: { type: terminalHarness.bufferType, ...terminalHarness.viewport, cursorX: 0, cursorY: 0, getLine: () => undefined },
};
}
scrollLines(amount: number) {
@@ -107,6 +109,7 @@ vi.mock("@xterm/xterm", () => ({
loadAddon() {}
open(host: HTMLElement) {
terminalHarness.open(host);
+ terminalHarness.spacingAtOpen.push(this.options.letterSpacing);
terminalHarness.host.current = host;
}
focus() {
@@ -125,6 +128,9 @@ vi.mock("@xterm/xterm", () => ({
getSelection() {
return "";
}
+ getSelectionPosition() { return undefined; }
+ onSelectionChange() { return { dispose() {} }; }
+ onRender() { return { dispose() {} }; }
onData() {
return { dispose() {} };
}
@@ -150,7 +156,7 @@ vi.mock("@xterm/xterm", () => ({
terminalHarness.resize(cols, rows);
}
dispose() {}
- clearTextureAtlas() {}
+ clearTextureAtlas() { terminalHarness.clearTextureAtlas(this); }
},
}));
@@ -717,7 +723,7 @@ describe("AgenticTerminal layout", () => {
expect(region?.className).not.toContain("invisible");
});
- it("waits for a promised repaint rather than showing the broken tail", () => {
+ it.each(["\x1b[2", "\x1b[1;1H\x1b["])("waits for a promised repaint and recognizes split erases: %j", (eraseStart) => {
// A replay the server marks as needing a repaint cannot rebuild the screen
// by itself. A busy agent may answer the repaint request late (the server
// repeats it after half a second), and revealing on quiet in between showed
@@ -749,7 +755,7 @@ describe("AgenticTerminal layout", () => {
// The erase arrives split across two chunks; the quiet window follows it.
act(() => {
- terminalHarness.handlers.current?.onOutput?.("\x1b[2" as never);
+ terminalHarness.handlers.current?.onOutput?.(eraseStart as never);
terminalHarness.handlers.current?.onOutput?.("Jthe repainted screen" as never);
});
expect(region?.className).toContain("invisible");
@@ -2638,6 +2644,8 @@ describe("renaming a pane", () => {
describe("terminal text size across a rebuild", () => {
beforeEach(() => {
terminalHarness.instances.length = 0;
+ terminalHarness.clearTextureAtlas.mockClear();
+ terminalHarness.spacingAtOpen.length = 0;
globalThis.ResizeObserver = ResizeObserverHarness;
});
@@ -2647,6 +2655,51 @@ describe("terminal text size across a rebuild", () => {
const newest = () => terminalHarness.instances[terminalHarness.instances.length - 1];
+ it.each(["light", "dark"] as const)("keeps the glyph cache warm when opening and restarting %s panes", (appearance) => {
+ const pane = (name: string, restartToken = 0) => (
+ <AgenticTerminal key={name} name={name} displayName="Codex" appearance={appearance} fontSize={13} restartToken={restartToken} />
+ );
+ const view = render(<>{pane("First")}</>);
+ expect(terminalHarness.clearTextureAtlas).not.toHaveBeenCalled();
+ const first = newest();
+
+ view.rerender(<>{pane("First")}{pane("Second")}</>);
+ expect(terminalHarness.instances).toHaveLength(2);
+ expect(terminalHarness.instances[0]).toBe(first);
+ expect(terminalHarness.clearTextureAtlas).not.toHaveBeenCalled();
+
+ view.rerender(<>{pane("First")}{pane("Second", 1)}</>);
+ expect(terminalHarness.instances).toHaveLength(3);
+ expect(terminalHarness.clearTextureAtlas).not.toHaveBeenCalled();
+ });
+
+ it("still refreshes glyphs on real theme and font changes without reconnecting", () => {
+ const view = render(<AgenticTerminal name="Dana" displayName="Codex" appearance="dark" fontSize={13} />);
+ const terminal = newest();
+ terminalHarness.clearTextureAtlas.mockClear();
+
+ view.rerender(<AgenticTerminal name="Dana" displayName="Codex" appearance="light" fontSize={13} />);
+ expect(terminalHarness.clearTextureAtlas).toHaveBeenCalledTimes(1);
+ expect(newest().options.theme).not.toBeUndefined();
+
+ view.rerender(<AgenticTerminal name="Dana" displayName="Codex" appearance="light" fontSize={18} />);
+ expect(terminalHarness.clearTextureAtlas).toHaveBeenCalledTimes(2);
+ expect(newest().options.fontSize).toBe(18);
+ expect(newest()).toBe(terminal);
+ expect(terminalHarness.instances).toHaveLength(1);
+ });
+
+ it("aligns fractional font cells before opening the renderer and measuring the first grid", () => {
+ vi.spyOn(HTMLCanvasElement.prototype, "getContext").mockReturnValue({
+ font: "",
+ measureText: () => ({ width: 249.6 }),
+ } as unknown as CanvasRenderingContext2D);
+
+ render(<AgenticTerminal name="Dana" displayName="Codex" appearance="dark" fontSize={13} />);
+ expect(terminalHarness.spacingAtOpen).toEqual([1]);
+ expect(terminalHarness.clearTextureAtlas).not.toHaveBeenCalled();
+ });
+
it("keeps the xterm canvas clear over the shared translucent pane shell", () => {
render(
<AgenticTerminal
diff --git a/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.tsx b/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.tsx
index 3c70c402bf..c6220ccad4 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/AgenticTerminal.tsx
@@ -136,11 +136,14 @@ import {
PARKED_RECHECK_MS,
} from "./offscreenBuffer";
import { installQuerySuppression } from "./terminalQueries";
+import { installSynchronizedOutput } from "./terminalSynchronizedOutput";
+import { FULL_SCREEN_ERASE_SCAN_TAIL, hasFullScreenErase } from "./terminalRepaint";
import {
bindTerminalScrollRegion,
captureWheelForTerminalHistory,
} from "./terminalScrollSurface";
import { installMouseSelection } from "./terminalMouseSelection";
+import { installPromptSelectionBridge } from "./terminalPromptSelection";
import {
openPaneSocket,
type PaneSocket,
@@ -150,6 +153,7 @@ import { PromptReceipt } from "./PromptReceipt";
import { PromptHistoryButton } from "./PromptHistoryButton";
import { PaneConversationDialog } from "./PaneConversationDialog";
import { WorkspaceTerminalHeader } from "./WorkspaceTerminalHeader";
+import { usePaneContextMenu } from "./usePaneContextMenu";
import { useT } from "@/i18n";
/**
@@ -243,9 +247,6 @@ export const REBUILD_SETTLE_MAX_MS = 450;
*/
export const REPAINT_WAIT_MAX_MS = 1_400;
-/** The whole-screen erase a full-screen agent's repaint starts with. */
-const FULL_SCREEN_ERASE = "\x1b[2J";
-
/**
* The longest an active pane's surface may stay behind a curtain, full stop.
*
@@ -489,6 +490,7 @@ export interface PaneRecapActions {
}
interface AgenticTerminalProps {
+ onSwapWithFocused?: () => void;
/** Terminal call-sign — also the WS path segment. */
name: string;
/**
@@ -658,6 +660,7 @@ interface AgenticTerminalProps {
}
export function AgenticTerminal({
+ onSwapWithFocused,
name,
workspaceId,
displayName,
@@ -701,6 +704,7 @@ export function AgenticTerminal({
layoutBusy = false,
sizeLead = false,
}: AgenticTerminalProps) {
+ const paneMenu = usePaneContextMenu(headerMode === "compact" || headerMode === "minimal");
const containerRef = useRef<HTMLDivElement | null>(null);
const terminalRegionRef = useRef<HTMLDivElement | null>(null);
const terminalRegionId = useId();
@@ -1036,6 +1040,10 @@ export function AgenticTerminal({
} catch {
/* proposed API unavailable in this build — widths stay at Unicode 6 */
}
+ // Align before the renderer joins the shared glyph atlas and before the
+ // first fit/handshake. Correcting spacing afterwards changes the grid and
+ // invalidates every existing pane's cache just to open an empty terminal.
+ alignTerminalCells(term);
term.open(container);
// The wheel always moves xterm's own history, even while a normal-buffer
// CLI has negotiated mouse tracking — otherwise scrolling only "works"
@@ -1053,6 +1061,7 @@ export function AgenticTerminal({
// The fallback renderer starts on an empty surface.
term.refresh(0, term.rows - 1);
});
+ const synchronizedOutput = installSynchronizedOutput(term);
termRef.current = term;
fitRef.current = fit;
setTerminalEpoch((current) => current + 1);
@@ -1073,6 +1082,7 @@ export function AgenticTerminal({
// A plain left-button drag selects text even while the agent's CLI has
// mouse tracking on, which every coding agent negotiates at start-up.
const disposeMouseSelection = installMouseSelection(container, term, isMac);
+ const disposePromptSelection = installPromptSelectionBridge(term, keys.add, isMac);
// The desktop IDE reserves its platform copy chord for copying. In
// particular, an unselected Ctrl+C on Windows/Linux must not reach Codex
// as `^C`, where it cancels the current turn or exits the pane.
@@ -1368,12 +1378,12 @@ export function AgenticTerminal({
const noteRepaintOutput = (text: string) => {
if (!repaintPending) return;
const scanned = eraseScanTail + text;
- if (scanned.includes(FULL_SCREEN_ERASE)) {
+ if (hasFullScreenErase(scanned)) {
repaintPending = false;
eraseScanTail = "";
return;
}
- eraseScanTail = scanned.slice(-(FULL_SCREEN_ERASE.length - 1));
+ eraseScanTail = scanned.slice(-FULL_SCREEN_ERASE_SCAN_TAIL);
};
const clearSettleTimers = () => {
@@ -1523,6 +1533,7 @@ export function AgenticTerminal({
setTailReady(false);
armCurtainWatchdog();
}
+ synchronizedOutput.reset();
term.reset();
// A normal-buffer CLI's replay is its whole scrollback — up to the
// server's 128 KB (see `ReplayBuffer`) — and xterm parses it in time
@@ -2284,10 +2295,12 @@ export function AgenticTerminal({
io?.disconnect();
disposeFontSync();
disposeMouseSelection();
+ disposePromptSelection();
disposeCopyBridge();
disposePasteBridge();
disposeNewlineBridge();
disposeQuerySuppression();
+ synchronizedOutput.dispose();
try {
socket?.close();
} catch {
@@ -2414,24 +2427,25 @@ export function AgenticTerminal({
* change has to invalidate it or the old palette keeps being painted.
*
* `terminalEpoch` is in here, and in the size effect below, for a reason the
- * appearance prop alone cannot cover: these effects fire on CHANGES, and the
- * terminal underneath them can be replaced without one. Every rebuild bumps
- * the epoch, so the pane restates the current theme and size to the new
- * terminal instead of trusting that it was born with them.
+ * appearance prop alone cannot cover: the terminal can be replaced without
+ * either prop changing. Check the replacement's actual options, but leave a
+ * correctly initialized terminal alone. Clearing its shared atlas on mount
+ * makes every existing WebGL pane rebuild its glyphs too.
*/
useEffect(() => {
const term = termRef.current;
if (!term) return;
- term.options.theme = themeFor(appearance);
+ const theme = themeFor(appearance);
+ if (term.options.theme === theme) return;
+ term.options.theme = theme;
clearTerminalTextureAtlas(term);
}, [appearance, terminalEpoch]);
useEffect(() => {
const term = termRef.current;
- if (!term) return;
- // A no-op on a terminal already built at this size (xterm's setter drops a
- // write of the identical value), which is what makes restating it on every
- // rebuild free.
+ if (!term || term.options.fontSize === fontSize) return;
+ // The setter itself skips unchanged values; the alignment, atlas clear and
+ // resize below must also run only for an actual size change.
term.options.fontSize = fontSize;
// A new size is a new glyph advance, and so a new fraction of a pixel for
// the canvas renderer to floor away. Re-align before the fit below, or the
@@ -2622,6 +2636,10 @@ export function AgenticTerminal({
const minimal = headerMode === "minimal";
const tile = PANE_TILE[appearance];
const headerProps = {
+ contextMenuRequest: paneMenu.request,
+ sendRightClicks: paneMenu.sendRightClicks,
+ onToggleSendRightClicks: paneMenu.toggleSendRightClicks,
+ onSwapWithFocused,
name,
workspaceId,
promptCount,
@@ -2789,6 +2807,7 @@ export function AgenticTerminal({
>
<div
ref={containerRef}
+ {...paneMenu.handlers}
data-testid={`agentic-terminal-host-${name}`}
// Read by ./index.css, which anchors the contents to the bottom for
// the length of a drag — see the rule there for why that is the side
diff --git a/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.test.tsx
index 18f63a83cb..be20a5ba33 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.test.tsx
@@ -17,8 +17,11 @@ import {
extractDropPayload,
isAbsolutePath,
joinPath,
+ lastUsedLabel,
+ leafName,
looksLikePath,
normalizeTypedPath,
+ pathCrumbs,
resolveTypedPath,
separatorOf,
splitTypedPath,
@@ -177,7 +180,7 @@ describe("FolderPicker", () => {
const card = await screen.findByText("/home/ruben/webshop");
expect(screen.getByText("Recent folders")).toBeTruthy();
- expect(screen.getByTestId("recent-folder-star")).toBeTruthy();
+ expect(screen.getByTestId("recent-folder-icon")).toBeTruthy();
fireEvent.click(card);
expect(onSelect).toHaveBeenCalledWith("/home/ruben/webshop");
expect(onSelectRecent).toHaveBeenCalledWith(
@@ -377,6 +380,7 @@ describe("typing a path", () => {
render(<FolderPicker selected={null} onSelect={vi.fn()} />);
await screen.findByText("webshop");
+ fireEvent.click(screen.getByTestId("edit-path"));
fireEvent.change(screen.getByTestId("folder-path-input"), {
target: { value: "web" },
});
@@ -396,6 +400,7 @@ describe("typing a path", () => {
render(<FolderPicker selected={null} onSelect={vi.fn()} />);
await screen.findByText("webshop");
+ fireEvent.click(screen.getByTestId("edit-path"));
const input = screen.getByTestId("folder-path-input") as HTMLInputElement;
fireEvent.change(input, { target: { value: "web" } });
await screen.findByTestId("path-suggestions");
@@ -454,6 +459,7 @@ describe("typing a path", () => {
render(<FolderPicker selected={null} onSelect={onSelect} />);
await screen.findByText("webshop");
+ fireEvent.click(screen.getByTestId("edit-path"));
const input = screen.getByTestId("folder-path-input");
fireEvent.change(input, { target: { value: "notes" } });
fireEvent.keyDown(input, { key: "Enter" });
@@ -466,6 +472,7 @@ describe("typing a path", () => {
render(<FolderPicker selected={null} onSelect={onSelect} />);
await screen.findByText("webshop");
+ fireEvent.click(screen.getByTestId("edit-path"));
const input = screen.getByTestId("folder-path-input");
fireEvent.change(input, { target: { value: "/srv/deploy" } });
fireEvent.keyDown(input, { key: "Enter" });
@@ -492,6 +499,7 @@ describe("typing a path", () => {
render(<FolderPicker selected={null} onSelect={onSelect} />);
await screen.findByText("webshop");
+ fireEvent.click(screen.getByTestId("edit-path"));
const input = screen.getByTestId("folder-path-input");
fireEvent.change(input, { target: { value: "nope" } });
fireEvent.keyDown(input, { key: "Enter" });
@@ -640,3 +648,137 @@ describe("hidden folders", () => {
await waitFor(() => expect(api.fetchFolders).toHaveBeenLastCalledWith(null, true));
});
});
+
+describe("the address bar", () => {
+ it("splits a path into crumbs that keep their root", () => {
+ expect(pathCrumbs("C:\\Users\\me")).toEqual([
+ { label: "C:", path: "C:\\" },
+ { label: "Users", path: "C:\\Users" },
+ { label: "me", path: "C:\\Users\\me" },
+ ]);
+ expect(pathCrumbs("/home/ruben")).toEqual([
+ { label: "/", path: "/" },
+ { label: "home", path: "/home" },
+ { label: "ruben", path: "/home/ruben" },
+ ]);
+ // A server alone is not a folder anyone can open: the share is the root.
+ expect(pathCrumbs("\\\\server\\share\\dir")).toEqual([
+ { label: "\\\\server\\share", path: "\\\\server\\share" },
+ { label: "dir", path: "\\\\server\\share\\dir" },
+ ]);
+ expect(leafName("C:\\Users\\me\\")).toBe("me");
+ expect(leafName("/")).toBe("/");
+ });
+
+ it("opens a crumb and chooses it, the same as opening a folder", async () => {
+ vi.mocked(api.fetchFolders).mockImplementation(async (target) => ({
+ ...LISTING,
+ path: target ?? "/home/ruben",
+ parent: "/home",
+ }));
+ const onSelect = vi.fn();
+ render(<FolderPicker selected={null} onSelect={onSelect} />);
+ await screen.findByText("webshop");
+
+ fireEvent.click(screen.getByRole("button", { name: "home" }));
+
+ await waitFor(() => expect(onSelect).toHaveBeenCalledWith("/home"));
+ expect(api.fetchFolders).toHaveBeenLastCalledWith("/home", false);
+ });
+
+ it("chooses the folder above when going up", async () => {
+ // The folder on screen is the folder chosen — going up must not leave the
+ // choice behind on a folder that is no longer shown.
+ vi.mocked(api.fetchFolders).mockImplementation(async (target) => ({
+ ...LISTING,
+ path: target ?? "/home/ruben",
+ parent: "/home",
+ }));
+ const onSelect = vi.fn();
+ render(<FolderPicker selected={null} onSelect={onSelect} />);
+ await screen.findByText("webshop");
+
+ fireEvent.click(screen.getByRole("button", { name: "Go up one folder" }));
+
+ await waitFor(() => expect(onSelect).toHaveBeenCalledWith("/home"));
+ });
+
+ it("gives Escape to the path field, which puts the crumbs back", async () => {
+ render(<FolderPicker selected={null} onSelect={vi.fn()} />);
+ await screen.findByText("webshop");
+
+ fireEvent.click(screen.getByTestId("edit-path"));
+ const input = screen.getByTestId("folder-path-input");
+ // The host dialog checks for this before closing itself on Escape.
+ expect(input.closest("[data-escape-local]")).toBeTruthy();
+ fireEvent.keyDown(input, { key: "Escape" });
+
+ await waitFor(() => expect(screen.queryByTestId("folder-path-input")).toBeNull());
+ expect(screen.getByText("Rubens MacBook")).toBeTruthy();
+ });
+});
+
+describe("the chosen folder", () => {
+ it("is named under the list", async () => {
+ render(<FolderPicker selected="/home/ruben/webshop" onSelect={vi.fn()} />);
+ const strip = await screen.findByTestId("folder-selection");
+ expect(strip.textContent).toContain("webshop");
+ expect(strip.textContent).toContain("/home/ruben/webshop");
+ });
+
+ it("says plainly when nothing is chosen yet", async () => {
+ render(<FolderPicker selected={null} onSelect={vi.fn()} />);
+ expect((await screen.findByTestId("folder-selection")).textContent).toMatch(
+ /no folder chosen/i,
+ );
+ });
+
+ it("is left to a host that reports it itself", async () => {
+ render(<FolderPicker selected="/x" onSelect={vi.fn()} showSelection={false} />);
+ await screen.findByText("webshop");
+ expect(screen.queryByTestId("folder-selection")).toBeNull();
+ });
+});
+
+describe("recent folders", () => {
+ it("belong to the start view, not to every folder", async () => {
+ vi.mocked(api.fetchRecents).mockResolvedValue({
+ device_name: "Rubens MacBook",
+ recents: [
+ {
+ path: "/home/ruben/shop",
+ name: "shop",
+ terminals: 1,
+ agents: {},
+ last_used: 1,
+ exists: true,
+ },
+ ],
+ });
+ vi.mocked(api.fetchFolders).mockImplementation(async (target) => ({
+ ...LISTING,
+ path: target ?? null,
+ parent: target ? "/home/ruben" : null,
+ }));
+ render(<FolderPicker selected={null} onSelect={vi.fn()} />);
+ expect(await screen.findByText("Recent folders")).toBeTruthy();
+
+ fireEvent.click(screen.getByText("notes"));
+
+ await waitFor(() => expect(screen.queryByText("Recent folders")).toBeNull());
+ });
+
+ it("say when they were last used, in words", () => {
+ const now = new Date(2026, 8, 30, 10, 0).getTime();
+ expect(lastUsedLabel(now / 1000, now)).toBe("Today");
+ expect(lastUsedLabel(new Date(2026, 8, 29, 23, 0).getTime() / 1000, now)).toBe(
+ "Yesterday",
+ );
+ // Milliseconds are read as well as the seconds the backend stores.
+ expect(lastUsedLabel(new Date(2026, 8, 26, 12, 0).getTime(), now)).toBe("4 days ago");
+ expect(lastUsedLabel(new Date(2026, 8, 16, 12, 0).getTime() / 1000, now)).toBe(
+ "2 weeks ago",
+ );
+ expect(lastUsedLabel(0, now)).toBe("");
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.tsx b/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.tsx
index 16772af6ea..74c549abcb 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/FolderPicker.tsx
@@ -1,7 +1,7 @@
/**
* Which folder the agents work in.
*
- * Six ways to arrive at one, because people arrive differently:
+ * Seven ways to arrive at one, because people arrive differently:
*
* 1. **The system folder window** — Explorer on Windows, Finder on macOS, the
* desktop's own dialog on Linux. The one people already know. Offered only
@@ -15,7 +15,8 @@
* that (which found nothing, and was reported as "search is broken").
* 4. **Browsing** — the list, projects and repositories sorted first. Hidden
* folders (`.claude`, `.config`) stay out of the way until asked for.
- * 5. **Typing a path** — with completion, the way `cd` works in a terminal.
+ * 5. **Typing a path** — the address bar turns into a field with completion,
+ * the way `cd` works in a terminal.
* 6. **Drag and drop** — drop a folder (or a file inside it) anywhere on the
* panel. Inside the desktop shell the host reports the real path
* (`waitForNativeDrop`); in a browser only the NAME is known, and the
@@ -29,27 +30,45 @@
* over a network and absent on a headless server. Everything here therefore
* works without it, and the button appears only where it can deliver.
*
- * ## Six ways, one list
+ * ## Laid out like a file window
*
- * Those six used to be six regions stacked down a column — a grid of recent
- * cards, a search row, a breadcrumb, the listing, a drop result, a path field,
- * and a bar repeating what was selected — each with its own border and its own
- * heading. Every one of them answers the same question, so a reader had to scan
- * seven bordered boxes to find the one route they wanted.
+ * Every route answers the same question, so they share one shape — the one
+ * people know from Explorer and Finder — instead of seven bordered regions:
*
- * Now there is one scrolling list. Recents are a labelled group at the top of
- * it, not a separate control with a different shape; search filters that same
- * list in place; the path field sits under it as a single line. What is above
- * the list is only ever navigation, what is below it is only ever typing, and
- * the folder that is chosen is reported once — in the launcher's header, where
- * the button that acts on it is.
+ * * a toolbar: search, the system window, a new folder;
+ * * one panel: an address bar over the folder list. The bar shows where you
+ * are as clickable crumbs and turns into a path field when clicked, which is
+ * where typing a path lives (it used to be a second text field under the
+ * list that looked exactly like the search above it);
+ * * the choice, stated once under the panel.
+ *
+ * **The folder on screen is the folder chosen.** Opening a folder, clicking a
+ * crumb, or going up all choose what is then on screen, so "what will be used"
+ * never differs from "where I am" — the old list chose a folder on click and
+ * then showed its CONTENTS, which left people looking at the wrong thing and
+ * guessing what the button would connect. The start view is the one exception:
+ * it is a list of places, not a folder, and choosing nothing there keeps the
+ * previous choice.
+ *
+ * Recents belong to the start view only. Pinned above every folder listing,
+ * they pushed what someone was browsing half a screen down; one click on the
+ * machine's name brings them back.
*
* The start view is labelled with the machine's own name rather than the account
* folder: "Administrator" says nothing about which computer you are looking at,
* "Ruben's MacBook" does.
+ *
+ * ## Escape
+ *
+ * Every host is a dialog that closes on Escape. The new-folder field and the
+ * path field use Escape to back out of THEMSELVES, so they carry
+ * `data-escape-local`; a host checks for it before closing (AgenticIdeView's
+ * key handler, `onEscapeKeyDown` on the Radix dialogs).
*/
-import { useCallback, useEffect, useMemo, useRef, useState } from "react";
+import { Fragment, useCallback, useEffect, useMemo, useRef, useState } from "react";
import {
+ Check,
+ ChevronRight,
CornerDownLeft,
CornerLeftUp,
Eye,
@@ -58,10 +77,12 @@ import {
FolderGit2,
FolderOpen,
FolderPlus,
+ History,
Loader2,
+ Monitor,
+ Pencil,
RefreshCw,
Search,
- Star,
X,
} from "lucide-react";
import { cn } from "@/lib/utils";
@@ -87,11 +108,20 @@ interface FolderPickerProps {
onSelect: (path: string) => void;
/** A recent workspace was picked — its previous layout can be replayed. */
onSelectRecent?: (recent: RecentWorkspace) => void;
+ /**
+ * Whether the picker states the chosen folder under its list. A host that
+ * already reports the choice beside its own action (the launcher's step
+ * list) turns it off so the folder is not named twice.
+ */
+ showSelection?: boolean;
+ className?: string;
}
/** Below this many characters the search stays a local filter of the open list. */
const SEARCH_MIN = 2;
const SEARCH_DEBOUNCE_MS = 250;
+/** More crumbs than this and the middle ones fold into "…". */
+const CRUMBS_MAX = 4;
/**
* Pull whatever identifies the dropped folder out of a DataTransfer.
@@ -148,6 +178,8 @@ export function FolderPicker({
selected,
onSelect,
onSelectRecent,
+ showSelection = true,
+ className,
}: FolderPickerProps) {
const [path, setPath] = useState<string | null>(null);
const [parent, setParent] = useState<string | null>(null);
@@ -192,6 +224,8 @@ export function FolderPicker({
const [newName, setNewName] = useState("");
const [creating, setCreating] = useState(false);
const [createNote, setCreateNote] = useState<string | null>(null);
+ // The address bar is showing the path field instead of the crumbs.
+ const [editingPath, setEditingPath] = useState(false);
// Read through a ref by the search effect: the launcher may hand in a fresh
// callback every render, and an effect keyed on it would re-run — and
@@ -242,6 +276,21 @@ export function FolderPicker({
});
}, [load]);
+ /**
+ * Go to a folder, and — because the folder on screen is the folder chosen —
+ * choose it. `null` is the start view, which is a list of places rather
+ * than a folder, so it leaves the choice as it was.
+ */
+ const navigate = (target: string | null) => {
+ if (target) onSelect(target);
+ setQuery("");
+ setSearchHits(null);
+ setPathLeaf(null);
+ setCreateOffer(null);
+ setEditingPath(false);
+ void load(target);
+ };
+
/**
* Hand over to the operating system's own folder window.
*
@@ -257,10 +306,7 @@ export function FolderPicker({
void openNativePicker(selected ?? path)
.then((res) => {
if (res.path) {
- onSelect(res.path);
- setQuery("");
- setSearchHits(null);
- void load(res.path);
+ navigate(res.path);
} else if (res.error) {
setNativeNote(res.error);
}
@@ -350,31 +396,36 @@ export function FolderPicker({
return () => window.clearTimeout(handle);
}, [query, goToTypedPath]);
+ const typedPath = looksLikePath(normalizeTypedPath(query));
+ // A bare word narrows what is on screen; a path is a place, not a filter.
+ const needle = typedPath ? "" : query.trim().toLowerCase();
+
const visible = useMemo(() => {
if (searchHits !== null) return searchHits;
if (pathLeaf !== null) {
- const needle = pathLeaf.toLowerCase();
- return entries.filter((e) => e.name.toLowerCase().startsWith(needle));
+ const leafNeedle = pathLeaf.toLowerCase();
+ return entries.filter((e) => e.name.toLowerCase().startsWith(leafNeedle));
}
- const typed = normalizeTypedPath(query);
- // A path on its way to being listed: the folder itself is what is shown,
- // not the list filtered by a string no folder is called.
- if (looksLikePath(typed)) return entries;
- const needle = query.trim().toLowerCase();
if (!needle) return entries;
return entries.filter((e) => e.name.toLowerCase().includes(needle));
- }, [entries, query, searchHits, pathLeaf]);
+ }, [entries, needle, searchHits, pathLeaf]);
- const open = (item: FolderItem) => {
- onSelect(item.path);
- setQuery("");
- setSearchHits(null);
- setPathLeaf(null);
- setCreateOffer(null);
- void load(item.path);
- };
+ const searchingMachine = searchHits !== null;
+ const atStart = path === null && pathLeaf === null && !searchingMachine;
+ const visibleRecents = useMemo(
+ () =>
+ atStart
+ ? recents.filter(
+ (r) =>
+ !needle ||
+ r.name.toLowerCase().includes(needle) ||
+ r.path.toLowerCase().includes(needle),
+ )
+ : [],
+ [atStart, recents, needle],
+ );
- /** Make the folder, then treat it like any other folder that was clicked. */
+ /** Make the folder, then treat it like any other folder that was opened. */
const makeFolder = (parent: string | null, name: string) => {
if (creating) return;
setCreating(true);
@@ -388,7 +439,7 @@ export function FolderPicker({
setNaming(false);
setNewName("");
setError(null);
- open(res.folder);
+ navigate(res.folder.path);
})
.catch((e) => setCreateNote((e as Error).message))
.finally(() => setCreating(false));
@@ -407,14 +458,16 @@ export function FolderPicker({
* the review step as the workspace, with the error shown next to it. Now the
* folder is listed first; only when it really is one does it become the
* selection, and otherwise the message stays and the previous choice stands.
+ *
+ * Resolves to whether it worked, so the path field stays open for a fix.
*/
- const usePath = (raw: string) => {
+ const usePath = (raw: string): Promise<boolean> => {
const value = raw.trim();
- if (!value) return;
+ if (!value) return Promise.resolve(false);
setLoading(true);
setError(null);
const { dir, leaf } = splitTypedPath(value);
- fetchFolders(value, showHiddenRef.current || leaf.startsWith("."))
+ return fetchFolders(value, showHiddenRef.current || leaf.startsWith("."))
.then((res) => {
if (res.device_name) setDeviceName(res.device_name);
if (res.error) {
@@ -423,22 +476,26 @@ export function FolderPicker({
// message, so a typo is corrected and a new project is created from
// the same line.
setCreateOffer(dir && leaf ? { parent: dir, name: leaf } : null);
- return;
+ return false;
}
setPath(res.path);
setParent(res.parent);
setEntries(res.entries);
setCreateOffer(null);
+ setPathLeaf(null);
onSelect(res.path ?? value);
+ return true;
+ })
+ .catch((e) => {
+ setError((e as Error).message);
+ return false;
})
- .catch((e) => setError((e as Error).message))
.finally(() => setLoading(false));
};
const pickRecent = (recent: RecentWorkspace) => {
- onSelect(recent.path);
+ navigate(recent.path);
onSelectRecent?.(recent);
- void load(recent.path);
};
// ------------------------------------------------------------ drag & drop
@@ -472,8 +529,7 @@ export function FolderPicker({
.then((res) => {
if (res.resolved) {
setDropNote(res.detail || null);
- onSelect(res.resolved);
- void load(res.resolved);
+ navigate(res.resolved);
} else {
setDropChoices(res.candidates);
setDropNote(res.detail || "Could not find that folder.");
@@ -491,11 +547,20 @@ export function FolderPicker({
}, []);
useDragSessionEnd(dragOver, clearDragOver);
- const searchingMachine = searchHits !== null;
- // What a full path looks like on THIS machine, for the placeholder and the
- // "nothing found" hints. Built from the real home folder, never from a
- // hard-coded `C:\…` — the same UI is opened against Macs and Linux boxes.
+ // What a full path looks like on THIS machine, for the "nothing found"
+ // hints. Built from the real home folder, never from a hard-coded `C:\…` —
+ // the same UI is opened against Macs and Linux boxes.
const pathExample = home ? joinPath(home, "my-project") : null;
+ const machine = deviceName || "This machine";
+ const crumbs = path ? foldCrumbs(pathCrumbs(path)) : [];
+ const listLabel = searchingMachine
+ ? null
+ : pathLeaf !== null
+ ? `Folders starting with “${pathLeaf}”`
+ : path
+ ? `Folders in ${leafName(path)}`
+ : "Places";
+ const hasMessages = Boolean(error || nativeNote || dropNote || createNote);
return (
<div
@@ -515,46 +580,44 @@ export function FolderPicker({
}}
onDrop={onDrop}
data-testid="folder-drop-zone"
- className="relative flex min-h-0 flex-1 flex-col"
+ className={cn("relative flex min-h-0 flex-1 flex-col p-3", className)}
>
{dragOver && (
- <div className="pointer-events-none absolute inset-0 z-20 flex items-center justify-center rounded-surface bg-background ring-2 ring-inset ring-primary/60">
- <span className="flex items-center gap-2 text-sm font-medium text-primary">
- <FolderOpen className="h-4 w-4" />
- Let go to use this folder
+ <div className="pointer-events-none absolute inset-0 z-40 flex flex-col items-center justify-center gap-2 rounded-surface border-2 border-dashed border-primary/60 bg-background/95">
+ <FolderOpen className="h-8 w-8 text-primary" />
+ <span className="text-sm font-medium text-foreground">
+ Drop to choose this folder
</span>
</div>
)}
- {/* --------------------------------------------------------- navigation */}
- <div className="flex items-center gap-1.5 p-3 pb-2">
+ {/* ------------------------------------------------------------ toolbar */}
+ <div className="flex items-center gap-2">
<div className="relative min-w-0 flex-1">
{searching ? (
- <Loader2 className="pointer-events-none absolute left-2.5 top-1/2 h-3.5 w-3.5 -translate-y-1/2 animate-spin text-muted-foreground" />
+ <Loader2 className="pointer-events-none absolute left-3 top-1/2 h-4 w-4 -translate-y-1/2 animate-spin text-muted-foreground" />
) : (
- <Search className="pointer-events-none absolute left-2.5 top-1/2 h-3.5 w-3.5 -translate-y-1/2 text-muted-foreground" />
+ <Search className="pointer-events-none absolute left-3 top-1/2 h-4 w-4 -translate-y-1/2 text-muted-foreground" />
)}
<Field
value={query}
onChange={(e) => setQuery(e.target.value)}
- placeholder={
- pathExample
- ? `Search by name, or type a path like ${pathExample}`
- : "Search folders by name, or type a path…"
- }
- aria-label="Search folders by name, or type a path"
+ placeholder="Search by name or paste a path"
+ aria-label="Search folders by name, or paste a path"
data-testid="folder-search"
- className="w-full pl-8 pr-7"
+ data-autofocus
+ className="h-9 w-full pl-9 pr-8"
spellCheck={false}
+ autoComplete="off"
/>
{query && (
<IconButton
size="sm"
label="Clear search"
onClick={() => setQuery("")}
- className="absolute right-1 top-1/2 -translate-y-1/2"
+ className="absolute right-1.5 top-1/2 -translate-y-1/2"
>
- <X className="h-3 w-3" />
+ <X className="h-3.5 w-3.5" />
</IconButton>
)}
</div>
@@ -563,14 +626,15 @@ export function FolderPicker({
onClick={browseNatively}
disabled={nativeOpen}
data-testid="native-browse"
- title="Choose the folder in a normal folder window"
+ title="Choose the folder in this computer's own folder window"
+ className="h-9"
>
{nativeOpen ? (
- <Loader2 className="h-3.5 w-3.5 animate-spin" />
+ <Loader2 className="h-4 w-4 animate-spin" />
) : (
- <FolderOpen className="h-3.5 w-3.5" />
+ <FolderOpen className="h-4 w-4" />
)}
- {nativeOpen ? "Window open…" : "Browse"}
+ {nativeOpen ? "Window open…" : "Browse…"}
</Button>
)}
<Button
@@ -582,331 +646,421 @@ export function FolderPicker({
disabled={naming}
data-testid="new-folder"
title={`Make a new folder in ${path ?? "your home folder"}`}
+ className="h-9"
>
- <FolderPlus className="h-3.5 w-3.5" />
+ <FolderPlus className="h-4 w-4" />
New folder
</Button>
- <IconButton
- label={showHidden ? "Hide hidden folders" : "Show hidden folders"}
- title={
- showHidden
- ? "Hidden folders (names starting with a dot) are shown"
- : "Also list hidden folders — names starting with a dot"
- }
- aria-pressed={showHidden}
- onClick={toggleHidden}
- disabled={loading}
- data-testid="toggle-hidden"
- className={cn(showHidden && "text-primary")}
- >
- {showHidden ? (
- <Eye className="h-4 w-4" />
- ) : (
- <EyeOff className="h-4 w-4" />
- )}
- </IconButton>
- <IconButton
- label="Go up one folder"
- onClick={() => void load(parent)}
- disabled={loading || searchingMachine || (!parent && path === null)}
- >
- <CornerLeftUp className="h-4 w-4" />
- </IconButton>
- <IconButton
- label="Reload this folder"
- onClick={() => void load(path)}
- disabled={loading}
- >
- {loading ? (
- <Loader2 className="h-4 w-4 animate-spin" />
- ) : (
- <RefreshCw className="h-4 w-4" />
- )}
- </IconButton>
- </div>
-
- {/* ------------------------------------------------------- where you are */}
- <div className="flex min-w-0 items-center gap-1.5 px-3 pb-2 text-xs text-muted-foreground">
- {searchingMachine ? (
- <span className="truncate">
- {visible.length === 0 && !searching
- ? `Nothing matches “${query.trim()}”`
- : `Matches for “${query.trim()}”, anywhere on this machine`}
- </span>
- ) : (
- <>
- <span className="shrink-0 font-medium text-foreground">
- {deviceName || "This machine"}
- </span>
- {path && (
- <>
- <span className="shrink-0 opacity-50">›</span>
- <code className="min-w-0 truncate font-mono">{path}</code>
- </>
- )}
- </>
- )}
</div>
{/* The window can still end up behind the app despite being opened
topmost — a user who does not know it is there just sees a button
that hung. Saying so costs one line and saves the confusion. */}
{nativeOpen && (
- <p className="px-3 pb-2 text-xs text-primary" role="status">
+ <p className="mt-2 text-xs text-primary" role="status">
A folder window has opened. Choose your folder there. If you cannot
see it, it is behind this window or in the taskbar.
</p>
)}
- {/* --------------------------------------------------------- new folder */}
- {naming && (
- <form
- data-testid="new-folder-form"
- className="flex items-center gap-1.5 px-3 pb-2"
- onSubmit={(e) => {
- e.preventDefault();
- if (newName.trim()) makeFolder(path, newName);
- }}
- >
- <FolderPlus className="h-4 w-4 shrink-0 text-primary" />
- <Field
- autoFocus
- value={newName}
- onChange={(e) => setNewName(e.target.value)}
- onKeyDown={(e) => {
- if (e.key === "Escape") setNaming(false);
- }}
- placeholder={`Name of the new folder in ${path ?? "your home folder"}`}
- aria-label="Name of the new folder"
- data-testid="new-folder-name"
- className="min-w-0 flex-1"
- spellCheck={false}
- autoComplete="off"
- />
- <Button
- type="submit"
- variant="primary"
- disabled={creating || !newName.trim()}
- data-testid="new-folder-create"
+ {/* -------------------------------------------------------------- panel */}
+ {/* Not overflow-hidden: the path field's suggestions hang below the
+ address bar, over the list, and would be clipped by the panel. */}
+ <div className="mt-3 flex min-h-0 flex-1 flex-col rounded-surface border border-border">
+ {/* ------------------------------------------------------ address bar */}
+ <div className="relative flex h-10 shrink-0 items-center gap-0.5 rounded-t-surface border-b border-border bg-secondary/40 px-1">
+ <IconButton
+ label="Go up one folder"
+ onClick={() => navigate(parent)}
+ disabled={loading || searchingMachine || path === null}
>
- {creating ? (
- <Loader2 className="h-3.5 w-3.5 animate-spin" />
+ <CornerLeftUp className="h-4 w-4" />
+ </IconButton>
+
+ {editingPath ? (
+ <PathInput
+ base={path}
+ initial={path ? withTrailingSeparator(path) : ""}
+ showHidden={showHidden}
+ onUse={(value) =>
+ usePath(value).then((ok) => {
+ if (ok) setEditingPath(false);
+ })
+ }
+ onCancel={() => setEditingPath(false)}
+ />
+ ) : searchingMachine ? (
+ <span className="min-w-0 flex-1 truncate px-2 text-xs text-muted-foreground">
+ {visible.length === 0 && !searching
+ ? `Nothing on ${machine} is called “${query.trim()}”`
+ : `Folders called “${query.trim()}” anywhere on ${machine}`}
+ </span>
+ ) : (
+ <nav
+ aria-label="Folder path"
+ className="flex h-full min-w-0 flex-1 items-center"
+ >
+ <button
+ type="button"
+ onClick={() => navigate(null)}
+ title={`Start: recent folders and places on ${machine}`}
+ aria-current={path === null ? "location" : undefined}
+ className={cn(
+ CRUMB,
+ "max-w-[11rem] shrink",
+ path === null && "font-medium text-foreground",
+ )}
+ >
+ <Monitor className="h-3.5 w-3.5 shrink-0" />
+ <span className="truncate">{machine}</span>
+ </button>
+ {crumbs.map((crumb, index) => {
+ const last = index === crumbs.length - 1;
+ return (
+ <Fragment key={crumb.path ?? `fold-${index}`}>
+ <ChevronRight className="h-3.5 w-3.5 shrink-0 text-muted-foreground/60" />
+ {crumb.path === null ? (
+ <button
+ type="button"
+ onClick={() => setEditingPath(true)}
+ title={`${crumb.hidden}\nClick to see and edit the whole path`}
+ aria-label="Show the whole path"
+ className={cn(CRUMB, "shrink-0")}
+ >
+ …
+ </button>
+ ) : (
+ <button
+ type="button"
+ onClick={() => navigate(crumb.path)}
+ title={crumb.path}
+ aria-current={last ? "location" : undefined}
+ className={cn(
+ CRUMB,
+ "min-w-[2.5rem] max-w-[12rem] shrink",
+ last && "font-medium text-foreground",
+ )}
+ >
+ <span className="truncate">{crumb.label}</span>
+ </button>
+ )}
+ </Fragment>
+ );
+ })}
+ {/* The empty rest of the bar is where a path is typed, the way
+ clicking beside the crumbs in Explorer turns them into text. */}
+ <button
+ type="button"
+ tabIndex={-1}
+ aria-hidden="true"
+ onClick={() => setEditingPath(true)}
+ className="h-full min-w-4 flex-1 cursor-text"
+ />
+ </nav>
+ )}
+
+ {!editingPath && (
+ <IconButton
+ label="Type a path"
+ title="Type or paste a path — Tab completes folder names"
+ onClick={() => setEditingPath(true)}
+ data-testid="edit-path"
+ >
+ <Pencil className="h-3.5 w-3.5" />
+ </IconButton>
+ )}
+ <IconButton
+ label={showHidden ? "Hide hidden folders" : "Show hidden folders"}
+ title={
+ showHidden
+ ? "Hidden folders (names starting with a dot) are shown"
+ : "Also list hidden folders — names starting with a dot"
+ }
+ aria-pressed={showHidden}
+ onClick={toggleHidden}
+ disabled={loading}
+ data-testid="toggle-hidden"
+ className={cn(showHidden && "text-primary")}
+ >
+ {showHidden ? (
+ <Eye className="h-4 w-4" />
) : (
- <FolderPlus className="h-3.5 w-3.5" />
+ <EyeOff className="h-4 w-4" />
)}
- Create
- </Button>
+ </IconButton>
<IconButton
- label="Cancel new folder"
- onClick={() => {
- setNaming(false);
- setCreateNote(null);
- }}
+ label="Reload this folder"
+ onClick={() => void load(path)}
+ disabled={loading}
>
- <X className="h-4 w-4" />
+ {loading ? (
+ <Loader2 className="h-4 w-4 animate-spin" />
+ ) : (
+ <RefreshCw className="h-4 w-4" />
+ )}
</IconButton>
- </form>
- )}
+ </div>
- {/* --------------------------------------------------------------- list */}
- <div className="min-h-[16rem] flex-1 overflow-y-auto scrollbar-jarvis border-y border-border">
- {recents.length > 0 && !searchingMachine && (
- <>
- <div className="sticky top-0 z-10 bg-card px-3 py-1.5 backdrop-blur">
- <SectionLabel>Recent folders</SectionLabel>
+ {/* ------------------------------------------------------- new folder */}
+ {naming && (
+ <form
+ data-testid="new-folder-form"
+ data-escape-local
+ className="shrink-0 border-b border-border px-3 py-2.5"
+ onSubmit={(e) => {
+ e.preventDefault();
+ if (newName.trim()) makeFolder(path, newName);
+ }}
+ >
+ <div className="flex items-center gap-2">
+ <FolderPlus className="h-4 w-4 shrink-0 text-primary" />
+ <Field
+ autoFocus
+ value={newName}
+ onChange={(e) => setNewName(e.target.value)}
+ onKeyDown={(e) => {
+ if (e.key === "Escape") {
+ setNaming(false);
+ setCreateNote(null);
+ }
+ }}
+ placeholder="Name of the new folder"
+ aria-label="Name of the new folder"
+ data-testid="new-folder-name"
+ className="min-w-0 flex-1"
+ spellCheck={false}
+ autoComplete="off"
+ />
+ <Button
+ type="submit"
+ variant="primary"
+ disabled={creating || !newName.trim()}
+ data-testid="new-folder-create"
+ >
+ {creating && <Loader2 className="h-3.5 w-3.5 animate-spin" />}
+ Create
+ </Button>
+ <IconButton
+ label="Cancel new folder"
+ onClick={() => {
+ setNaming(false);
+ setCreateNote(null);
+ }}
+ >
+ <X className="h-4 w-4" />
+ </IconButton>
</div>
- <ul>
- {recents.map((recent) => (
- <li key={recent.path} className="group/row relative">
- <button
- type="button"
- onClick={() => pickRecent(recent)}
- className={cn(
- "flex w-full items-center gap-2.5 px-3 py-1.5 text-left transition-colors",
- selected === recent.path
- ? "bg-primary/10"
- : "hover:bg-secondary",
- )}
- >
- <Star
- data-testid="recent-folder-star"
- className="h-3.5 w-3.5 shrink-0 fill-current text-primary"
- />
- <span className="min-w-0 flex-1">
- <span className="block truncate text-sm">
- {recent.name}
+ <p className="mt-1.5 truncate pl-6 text-xs text-muted-foreground">
+ Created in{" "}
+ <span className="font-mono">{path ?? "your home folder"}</span>
+ </p>
+ </form>
+ )}
+
+ {/* ------------------------------------------------------------- list */}
+ <div className="min-h-[14rem] flex-1 overflow-y-auto scrollbar-jarvis p-1.5">
+ {visibleRecents.length > 0 && (
+ <section aria-label="Recent folders" className="pb-2">
+ <GroupLabel>Recent folders</GroupLabel>
+ <ul>
+ {visibleRecents.map((recent) => (
+ <li key={recent.path} className="group/row relative">
+ <button
+ type="button"
+ onClick={() => pickRecent(recent)}
+ title={recent.path}
+ className={cn(
+ ROW,
+ "pr-9",
+ selected === recent.path
+ ? "bg-primary/10"
+ : "hover:bg-secondary",
+ )}
+ >
+ <History
+ data-testid="recent-folder-icon"
+ className="h-4 w-4 shrink-0 text-muted-foreground"
+ />
+ <span className="min-w-0 flex-1">
+ <span className="block truncate font-medium">
+ {recent.name}
+ </span>
+ <span className="block truncate font-mono text-micro text-muted-foreground">
+ {recent.path}
+ </span>
</span>
- <span className="block truncate font-mono text-micro text-muted-foreground">
- {recent.path}
+ <span className="shrink-0 text-right text-micro text-muted-foreground">
+ {lastUsedLabel(recent.last_used)}
+ {/* The terminal count only means something where the
+ layout is replayed — the launcher. */}
+ {onSelectRecent && (
+ <span className="block tabular-nums">
+ {recent.terminals}{" "}
+ {recent.terminals === 1 ? "terminal" : "terminals"}
+ </span>
+ )}
</span>
- </span>
- <span className="shrink-0 pr-5 font-mono text-micro tabular-nums text-muted-foreground">
- {recent.terminals}
- </span>
- </button>
- <IconButton
- size="sm"
- label={`Forget ${recent.name}`}
- title="Remove from this list (the folder itself stays)"
- onClick={(e) => {
- e.stopPropagation();
- setRecents((prev) =>
- prev.filter((r) => r.path !== recent.path),
- );
- void forgetRecent(recent.path).catch(() => {
- /* list refreshes on next visit */
- });
- }}
- className="absolute right-1.5 top-1/2 -translate-y-1/2 opacity-0 focus-visible:opacity-100 group-hover/row:opacity-100"
- >
- <X className="h-3 w-3" />
- </IconButton>
- </li>
- ))}
- </ul>
- {/*
- Not the machine's name a second time: it is already in the line
- above the list, and a label repeated twelve pixels apart reads as
- two different things being named rather than one.
- */}
- <div className="sticky top-0 z-10 bg-card px-3 py-1.5 backdrop-blur">
- <SectionLabel>Browse</SectionLabel>
- </div>
- </>
- )}
+ </button>
+ <IconButton
+ size="sm"
+ label={`Forget ${recent.name}`}
+ title="Remove from this list (the folder itself stays)"
+ onClick={(e) => {
+ e.stopPropagation();
+ setRecents((prev) =>
+ prev.filter((r) => r.path !== recent.path),
+ );
+ void forgetRecent(recent.path).catch(() => {
+ /* list refreshes on next visit */
+ });
+ }}
+ className="absolute right-2 top-1/2 -translate-y-1/2 opacity-0 focus-visible:opacity-100 group-hover/row:opacity-100"
+ >
+ <X className="h-3.5 w-3.5" />
+ </IconButton>
+ </li>
+ ))}
+ </ul>
+ </section>
+ )}
- {/*
- The typed path's last segment is not a folder in the folder on
- screen, and nothing there starts with it either: the one thing left
- to do with that name is make it.
- */}
- {createOffer && pathLeaf !== null && visible.length === 0 && !searching && (
- <button
- type="button"
- data-testid="create-offer"
- onClick={() => makeFolder(createOffer.parent, createOffer.name)}
- disabled={creating}
- className="flex w-full items-center gap-2.5 px-3 py-2 text-left text-sm text-primary transition-colors hover:bg-secondary"
- >
- {creating ? (
- <Loader2 className="h-4 w-4 shrink-0 animate-spin" />
- ) : (
- <FolderPlus className="h-4 w-4 shrink-0" />
- )}
- <span className="min-w-0 flex-1">
- <span className="block truncate">
- Create folder “{createOffer.name}”
- </span>
- <span className="block truncate font-mono text-micro text-muted-foreground">
- in {createOffer.parent ?? "your home folder"}
+ {/*
+ The typed path's last segment is not a folder in the folder on
+ screen, and nothing there starts with it either: the one thing left
+ to do with that name is make it.
+ */}
+ {createOffer && pathLeaf !== null && visible.length === 0 && !searching && (
+ <button
+ type="button"
+ data-testid="create-offer"
+ onClick={() => makeFolder(createOffer.parent, createOffer.name)}
+ disabled={creating}
+ className={cn(ROW, "text-primary hover:bg-secondary")}
+ >
+ {creating ? (
+ <Loader2 className="h-4 w-4 shrink-0 animate-spin" />
+ ) : (
+ <FolderPlus className="h-4 w-4 shrink-0" />
+ )}
+ <span className="min-w-0 flex-1">
+ <span className="block truncate">
+ Create folder “{createOffer.name}”
+ </span>
+ <span className="block truncate font-mono text-micro text-muted-foreground">
+ in {createOffer.parent ?? "your home folder"}
+ </span>
</span>
- </span>
- </button>
- )}
+ </button>
+ )}
- {visible.length === 0 && !loading && !searching ? (
- <p className="px-3 py-4 text-sm text-muted-foreground">
- {searchingMachine
- ? `No folder with that name was found.${
- pathExample
- ? ` To open one by where it is, type its full path — like ${pathExample}.`
- : ""
- }`
- : pathLeaf !== null
- ? `No folder here starts with “${pathLeaf}”.`
- : "Nothing to list here — search above, drop a folder, or type its path below."}
- </p>
- ) : (
- <ul>
- {visible.map((item) => {
- const isSelected = selected === item.path;
- return (
- <li key={item.path}>
- <button
- type="button"
- onClick={() => open(item)}
- className={cn(
- "flex w-full items-center gap-2.5 px-3 py-1.5 text-left text-sm transition-colors",
- isSelected ? "bg-primary/10" : "hover:bg-secondary",
- )}
- >
- {item.is_repo ? (
- <FolderGit2
+ {listLabel && visible.length > 0 && (
+ <GroupLabel>{listLabel}</GroupLabel>
+ )}
+
+ {visible.length === 0 && !loading && !searching ? (
+ <p className="px-2.5 py-6 text-center text-sm text-muted-foreground">
+ {searchingMachine
+ ? `No folder with that name was found.${
+ pathExample
+ ? ` To open one by where it is, type its full path — like ${pathExample}.`
+ : ""
+ }`
+ : pathLeaf !== null
+ ? `No folder here starts with “${pathLeaf}”.`
+ : needle
+ ? `No folder here matches “${query.trim()}”. Keep typing to search the whole machine.`
+ : path
+ ? "This folder has no folders inside. You can choose it as it is, or make a new folder in it."
+ : "Nothing to list here — search above, drop a folder, or type its path."}
+ </p>
+ ) : (
+ <ul>
+ {visible.map((item) => {
+ const isSelected = selected === item.path;
+ const Icon = item.is_repo ? FolderGit2 : Folder;
+ return (
+ <li key={item.path}>
+ <button
+ type="button"
+ onClick={() => navigate(item.path)}
+ title={item.path}
+ className={cn(
+ ROW,
+ "group/item",
+ isSelected ? "bg-primary/10" : "hover:bg-secondary",
+ )}
+ >
+ <Icon
className={cn(
"h-4 w-4 shrink-0",
isSelected ? "text-primary" : "text-muted-foreground",
)}
/>
- ) : (
- <Folder
- className={cn(
- "h-4 w-4 shrink-0",
- isSelected
- ? "text-primary"
- : "text-muted-foreground",
- )}
- />
- )}
- <span className="min-w-0 flex-1">
- <span
- className={cn(
- "block truncate",
- isSelected && "text-primary",
+ <span className="min-w-0 flex-1">
+ <span
+ className={cn(
+ "block truncate",
+ isSelected && "font-medium text-primary",
+ )}
+ >
+ {item.name}
+ </span>
+ {searchingMachine && (
+ <span className="block truncate font-mono text-micro text-muted-foreground">
+ {item.path}
+ </span>
)}
- >
- {item.name}
</span>
- {searchingMachine && (
- <span className="block truncate font-mono text-micro text-muted-foreground">
- {item.path}
+ {item.is_repo && (
+ <span className="shrink-0 rounded border border-border px-1.5 font-mono text-micro text-muted-foreground">
+ git
</span>
)}
- </span>
- {item.is_repo && (
- <span className="shrink-0 font-mono text-micro text-muted-foreground">
- git
- </span>
- )}
- </button>
- </li>
- );
- })}
- </ul>
- )}
+ <ChevronRight className="h-4 w-4 shrink-0 text-muted-foreground opacity-0 transition-opacity group-hover/item:opacity-100" />
+ </button>
+ </li>
+ );
+ })}
+ </ul>
+ )}
+ </div>
</div>
- {/* ------------------------------------------------------------- typing */}
- <PathInput base={path} showHidden={showHidden} onUse={usePath} />
+ {/* ------------------------------------------------------------ choice */}
+ {showSelection && <SelectedFolder selected={selected} />}
{/* ---------------------------------------------------------- reporting */}
- {(error || nativeNote || dropNote || createNote) && (
- <div className="space-y-1 px-3 pb-3">
+ {hasMessages ? (
+ <div className="mt-2 space-y-1.5">
{error && (
- <p className="text-xs text-destructive" role="alert">
- {error}
- </p>
- )}
- {error && pathExample && (
- <p className="text-xs text-muted-foreground">
- A full path on this machine looks like{" "}
- <code className="font-mono">{pathExample}</code>.
- </p>
- )}
- {/* The path field's "no such folder", answered: make it. */}
- {error && createOffer && pathLeaf === null && (
- <Button
- onClick={() => makeFolder(createOffer.parent, createOffer.name)}
- disabled={creating}
- data-testid="create-offer"
- className="h-7 text-xs"
+ <div
+ className="rounded-control border border-destructive/30 bg-destructive/5 px-3 py-2 text-xs"
+ role="alert"
>
- {creating ? (
- <Loader2 className="h-3.5 w-3.5 animate-spin" />
- ) : (
- <FolderPlus className="h-3.5 w-3.5" />
+ <p className="text-destructive">{error}</p>
+ {pathExample && (
+ <p className="mt-0.5 text-muted-foreground">
+ A full path on this machine looks like{" "}
+ <code className="font-mono">{pathExample}</code>.
+ </p>
+ )}
+ {/* The path field's "no such folder", answered: make it. */}
+ {createOffer && pathLeaf === null && (
+ <Button
+ onClick={() => makeFolder(createOffer.parent, createOffer.name)}
+ disabled={creating}
+ data-testid="create-offer"
+ className="mt-2 h-7 text-xs"
+ >
+ {creating ? (
+ <Loader2 className="h-3.5 w-3.5 animate-spin" />
+ ) : (
+ <FolderPlus className="h-3.5 w-3.5" />
+ )}
+ Create folder “{createOffer.name}”
+ </Button>
)}
- Create folder “{createOffer.name}”
- </Button>
+ </div>
)}
{createNote && (
<p className="text-xs text-destructive" role="alert">
@@ -930,9 +1084,9 @@ export function FolderPicker({
onClick={() => {
setDropNote(null);
setDropChoices([]);
- open(choice);
+ navigate(choice.path);
}}
- className="flex w-full items-center gap-2 rounded px-1 py-0.5 text-left text-xs hover:bg-secondary"
+ className="flex w-full items-center gap-2 rounded-control px-1.5 py-1 text-left text-xs hover:bg-secondary"
>
<FolderOpen className="h-3.5 w-3.5 shrink-0 text-muted-foreground" />
<code className="min-w-0 truncate font-mono">
@@ -946,6 +1100,75 @@ export function FolderPicker({
</>
)}
</div>
+ ) : (
+ <p className="mt-2 text-xs text-muted-foreground">
+ Tip: you can also drag a folder here from your file manager.
+ </p>
+ )}
+ </div>
+ );
+}
+
+const CRUMB =
+ "inline-flex h-7 min-w-0 items-center gap-1.5 rounded-control px-1.5 text-xs " +
+ "text-muted-foreground transition-colors hover:bg-secondary hover:text-foreground " +
+ "focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring/60";
+
+const ROW =
+ "flex min-h-9 w-full items-center gap-3 rounded-control px-2.5 py-1.5 text-left text-sm " +
+ "transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring/60";
+
+function GroupLabel({ children }: { children: React.ReactNode }) {
+ return (
+ <div className="px-2.5 pb-1 pt-2">
+ <SectionLabel>{children}</SectionLabel>
+ </div>
+ );
+}
+
+/**
+ * The folder that will be used, said once, right above whatever acts on it.
+ *
+ * The list alone never answered this: a folder that is opened is chosen AND
+ * replaced on screen by its contents, so the one row that could have shown
+ * the choice is gone the moment it is made.
+ */
+function SelectedFolder({ selected }: { selected: string | null }) {
+ return (
+ <div
+ data-testid="folder-selection"
+ role="status"
+ aria-live="polite"
+ className={cn(
+ "mt-3 flex min-h-[3.25rem] shrink-0 items-center gap-3 rounded-surface border px-3 py-2",
+ selected ? "border-primary/40 bg-primary/5" : "border-dashed border-border",
+ )}
+ >
+ {selected ? (
+ <>
+ <span className="flex h-6 w-6 shrink-0 items-center justify-center rounded-full bg-primary text-primary-foreground">
+ <Check className="h-3.5 w-3.5" strokeWidth={3} />
+ </span>
+ <span className="min-w-0 flex-1">
+ <span className="block truncate text-sm font-medium">
+ <span className="sr-only">Chosen folder: </span>
+ {leafName(selected)}
+ </span>
+ <span
+ className="block truncate font-mono text-micro text-muted-foreground"
+ title={selected}
+ >
+ {selected}
+ </span>
+ </span>
+ </>
+ ) : (
+ <>
+ <Folder className="h-4 w-4 shrink-0 text-muted-foreground" />
+ <span className="text-sm text-muted-foreground">
+ No folder chosen yet. Open a folder in the list to choose it.
+ </span>
+ </>
)}
</div>
);
@@ -975,6 +1198,97 @@ export function joinPath(base: string, name: string): string {
return base.endsWith(sep) ? `${base}${name}` : `${base}${sep}${name}`;
}
+function withTrailingSeparator(fullPath: string): string {
+ return /[\\/]$/.test(fullPath) ? fullPath : `${fullPath}${separatorOf(fullPath)}`;
+}
+
+/** The last segment of a path — what a person calls the folder. */
+export function leafName(fullPath: string): string {
+ const trimmed = fullPath.replace(/[\\/]+$/, "");
+ return trimmed.split(/[\\/]/).pop() || fullPath;
+}
+
+/**
+ * A path as the crumbs of an address bar, each one a place to go.
+ *
+ * The root keeps its own shape — `C:` on Windows, `/` elsewhere, the whole
+ * `\\server\share` of a network path, since a server alone is not a folder
+ * anyone can open.
+ */
+export function pathCrumbs(fullPath: string): { label: string; path: string }[] {
+ const sep = separatorOf(fullPath);
+ let root = "";
+ let rest = fullPath;
+ const unc = fullPath.match(/^([\\/]{2}[^\\/]+[\\/]+[^\\/]+)[\\/]*(.*)$/);
+ const drive = fullPath.match(/^([a-zA-Z]:)[\\/]*(.*)$/);
+ if (unc) {
+ root = unc[1];
+ rest = unc[2];
+ } else if (drive) {
+ root = `${drive[1]}${sep}`;
+ rest = drive[2];
+ } else if (/^[\\/]/.test(fullPath)) {
+ root = sep;
+ rest = fullPath.replace(/^[\\/]+/, "");
+ }
+ const crumbs: { label: string; path: string }[] = [];
+ if (root) {
+ crumbs.push({ label: root === sep ? sep : root.replace(/[\\/]+$/, ""), path: root });
+ }
+ let acc = root;
+ for (const part of rest.split(/[\\/]+/).filter(Boolean)) {
+ acc = acc ? joinPath(acc, part) : part;
+ crumbs.push({ label: part, path: acc });
+ }
+ return crumbs;
+}
+
+/**
+ * A long path keeps its root and the two folders nearest the end; the middle
+ * folds into one "…" crumb (`path: null`) that names what it hides.
+ */
+function foldCrumbs(
+ crumbs: { label: string; path: string }[],
+): { label: string; path: string | null; hidden?: string }[] {
+ if (crumbs.length <= CRUMBS_MAX) return crumbs;
+ const folded = crumbs.slice(1, -2);
+ return [
+ crumbs[0],
+ { label: "…", path: null, hidden: folded[folded.length - 1].path },
+ ...crumbs.slice(-2),
+ ];
+}
+
+/**
+ * When a recent folder was last opened, the way people say it.
+ *
+ * Counted in calendar days on this clock, so something opened late last night
+ * is "Yesterday" this morning, not "Today". Accepts seconds (what the backend
+ * stores) or milliseconds.
+ */
+export function lastUsedLabel(lastUsed: number, now: number = Date.now()): string {
+ if (!lastUsed || !Number.isFinite(lastUsed)) return "";
+ const ms = lastUsed > 1e12 ? lastUsed : lastUsed * 1000;
+ const dayStart = (t: number) => {
+ const d = new Date(t);
+ d.setHours(0, 0, 0, 0);
+ return d.getTime();
+ };
+ const days = Math.round((dayStart(now) - dayStart(ms)) / 86_400_000);
+ if (days <= 0) return "Today";
+ if (days === 1) return "Yesterday";
+ if (days < 7) return `${days} days ago`;
+ if (days < 30) {
+ const weeks = Math.floor(days / 7);
+ return weeks === 1 ? "Last week" : `${weeks} weeks ago`;
+ }
+ return new Date(ms).toLocaleDateString(undefined, {
+ day: "numeric",
+ month: "short",
+ year: days > 300 ? "numeric" : undefined,
+ });
+}
+
/**
* What was typed, read the way a shell would read it.
*
@@ -1025,31 +1339,40 @@ export function resolveTypedPath(raw: string, base: string | null): string {
}
/**
- * A path field that completes as you type, the way `cd` does in a terminal.
+ * The address bar as a path field that completes as you type, the way `cd`
+ * does in a terminal.
*
- * Typing a bare name completes against the folder currently on screen, so
- * `cd`-style navigation works without ever writing an absolute path; Tab takes
- * the highlighted suggestion and appends a separator so the next segment can be
- * typed straight away. Arrow keys move through the list, Enter opens what is
- * highlighted — or, with nothing highlighted, whatever was typed.
+ * It opens holding the folder on screen plus a separator, so the next segment
+ * can be typed straight away; a bare name still completes against the folder
+ * on screen, so `cd`-style navigation works without an absolute path. Tab
+ * takes the highlighted suggestion and appends a separator. Arrow keys move
+ * through the list, Enter opens what is highlighted — or, with nothing
+ * highlighted, whatever was typed. Escape and leaving the field put the crumbs
+ * back.
*
* Completion is deliberately a plain folder listing rather than the name search
- * above it: `cd` shows what is in THIS folder, and a field that answered with
- * matches from across the disk would be a different tool wearing the same shape.
+ * in the toolbar: `cd` shows what is in THIS folder, and a field that answered
+ * with matches from across the disk would be a different tool wearing the same
+ * shape.
*/
function PathInput({
base,
+ initial,
showHidden,
onUse,
+ onCancel,
}: {
base: string | null;
+ initial: string;
showHidden: boolean;
- onUse: (value: string) => void;
+ onUse: (value: string) => Promise<void>;
+ onCancel: () => void;
}) {
- const [value, setValue] = useState("");
+ const [value, setValue] = useState(initial);
const [options, setOptions] = useState<FolderItem[]>([]);
const [active, setActive] = useState(-1);
- const [open, setOpen] = useState(false);
+ const [open, setOpen] = useState(true);
+ const inputRef = useRef<HTMLInputElement>(null);
const { dir, leaf } = splitTypedPath(normalizeTypedPath(value));
// An empty `dir` means a bare name was typed: complete inside the folder the
@@ -1061,6 +1384,14 @@ function PathInput({
: joinPath(base, dir)
: base || "";
+ useEffect(() => {
+ const input = inputRef.current;
+ if (!input) return;
+ input.focus();
+ // The caret at the end, ready for the next segment.
+ input.setSelectionRange(input.value.length, input.value.length);
+ }, []);
+
useEffect(() => {
if (!value.trim() && !base) {
setOptions([]);
@@ -1100,11 +1431,11 @@ function PathInput({
const submit = () => {
if (active >= 0 && options[active]) {
- onUse(options[active].path);
+ void onUse(options[active].path);
} else {
// A relative path is relative to what is on screen, the way cd reads it.
const absolute = resolveTypedPath(value, base);
- if (absolute) onUse(absolute);
+ if (absolute) void onUse(absolute);
}
setOpen(false);
};
@@ -1132,45 +1463,51 @@ function PathInput({
submit();
return;
}
- if (e.key === "Escape") setOpen(false);
+ if (e.key === "Escape") {
+ e.preventDefault();
+ // First Escape closes the suggestions, the next one the field.
+ if (open && options.length > 0) setOpen(false);
+ else onCancel();
+ }
};
return (
- <div className="relative p-3">
+ <div className="relative min-w-0 flex-1" data-escape-local>
<Field
+ ref={inputRef}
value={value}
onChange={(e) => {
setValue(e.target.value);
setOpen(true);
}}
onKeyDown={onKeyDown}
- onFocus={() => setOpen(true)}
- // Closing on blur is delayed: a click on a suggestion blurs the input
- // first, and an immediate close would remove the element under the
- // pointer before the click lands on it.
- onBlur={() => window.setTimeout(() => setOpen(false), 150)}
- placeholder="…or type a folder name here — Tab completes it"
+ // Leaving is delayed: a click on a suggestion blurs the input first,
+ // and an immediate close would remove the element under the pointer
+ // before the click lands on it.
+ onBlur={() =>
+ window.setTimeout(() => {
+ if (document.activeElement !== inputRef.current) onCancel();
+ }, 150)
+ }
+ placeholder="Type a path — Tab completes folder names"
aria-label="Folder path"
data-testid="folder-path-input"
- className="w-full pr-8 font-mono text-xs"
+ className="h-8 w-full pr-8 font-mono text-xs"
spellCheck={false}
autoComplete="off"
/>
{/*
- The field's own confirm, in place of the "Use this path" button that
- used to sit beside it. Enter already does this, but only for someone who
- guesses that it will — and a control that can only be reached by guessing
- is not a route, so the affordance stays. It is inside the field because
- it belongs to it: as a sibling it was a second full-width control for a
- line of text nobody had typed yet.
+ The field's own confirm. Enter already does this, but only for someone
+ who guesses that it will — and a control that can only be reached by
+ guessing is not a route, so the affordance stays.
*/}
{value.trim() && (
<IconButton
size="sm"
- label="Use this path"
+ label="Go to this path"
onMouseDown={(e) => e.preventDefault()}
onClick={submit}
- className="absolute right-4 top-1/2 -translate-y-1/2 text-primary"
+ className="absolute right-1 top-1/2 -translate-y-1/2 text-primary"
>
<CornerDownLeft className="h-3.5 w-3.5" />
</IconButton>
@@ -1178,7 +1515,7 @@ function PathInput({
{open && options.length > 0 && (
<ul
data-testid="path-suggestions"
- className="absolute bottom-full left-3 right-3 z-30 mb-1 max-h-48 overflow-y-auto scrollbar-jarvis rounded-control border border-border bg-popover"
+ className="absolute left-0 right-0 top-full z-30 mt-1.5 max-h-56 overflow-y-auto scrollbar-jarvis rounded-control border border-border bg-popover p-1 shadow-lg"
>
{options.slice(0, 40).map((item, index) => (
<li key={item.path}>
@@ -1187,7 +1524,7 @@ function PathInput({
onMouseDown={(e) => e.preventDefault()}
onClick={() => complete(item)}
className={cn(
- "flex w-full items-center gap-2 px-2.5 py-1.5 text-left text-xs",
+ "flex w-full items-center gap-2 rounded-control px-2 py-1.5 text-left text-xs",
index === active ? "bg-primary/15" : "hover:bg-secondary",
)}
>
diff --git a/jarvis/ui/web/frontend/src/components/agentic/PaneContextMenu.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/PaneContextMenu.test.tsx
new file mode 100644
index 0000000000..3f37c5b3e3
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/PaneContextMenu.test.tsx
@@ -0,0 +1,93 @@
+import { useEffect, useRef } from "react";
+import { cleanup, fireEvent, render, screen } from "@testing-library/react";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { WorkspaceTerminalHeader } from "./WorkspaceTerminalHeader";
+import { usePaneContextMenu } from "./usePaneContextMenu";
+import { PANE_CHROME } from "./terminalThemes";
+
+afterEach(cleanup);
+
+function Harness({ appearance = "dark", forwarded, selected, swap, enabled = true }: {
+ appearance?: "light" | "dark"; forwarded: () => void; selected: () => void;
+ swap?: () => void; enabled?: boolean;
+}) {
+ const menu = usePaneContextMenu(enabled);
+ const host = useRef<HTMLDivElement>(null);
+ // A native target listener models xterm consuming mouse events before React bubbles.
+ useEffect(() => {
+ const node = host.current!;
+ node.addEventListener("mousedown", forwarded);
+ return () => node.removeEventListener("mousedown", forwarded);
+ }, [forwarded]);
+ return <div onMouseDown={selected}>
+ <WorkspaceTerminalHeader name="Dana" agent="codex" displayName="Codex" status="live"
+ appearance={appearance} contextMenuRequest={menu.request} onSwapWithFocused={swap}
+ sendRightClicks={menu.sendRightClicks} onToggleSendRightClicks={menu.toggleSendRightClicks}
+ onRename={async () => true} onAdd={() => {}} onToggleMaximize={() => {}} onClose={() => {}} />
+ <div {...menu.handlers}><div ref={host} data-testid="terminal-content" /></div>
+ </div>;
+}
+
+describe("terminal surface context menu", () => {
+ it.each(["light", "dark"] as const)("opens the pane actions in %s without selecting or sending a right-click", (appearance) => {
+ const forwarded = vi.fn(), selected = vi.fn(), swap = vi.fn();
+ render(<Harness appearance={appearance} forwarded={forwarded} selected={selected} swap={swap} />);
+ const host = screen.getByTestId("terminal-content");
+ fireEvent.mouseDown(host, { button: 2 });
+ const event = new MouseEvent("contextmenu", { button: 2, bubbles: true, cancelable: true, clientX: 30, clientY: 40 });
+ fireEvent(host, event);
+ expect(event.defaultPrevented).toBe(true);
+ expect(forwarded).not.toHaveBeenCalled();
+ expect(selected).not.toHaveBeenCalled();
+ const menu = screen.getByRole("menu");
+ const expectedColor = document.createElement("div");
+ expectedColor.style.background = PANE_CHROME[appearance].float;
+ expect(menu.style.background).toBe(expectedColor.style.background);
+ for (const name of ["Rename pane", "Swap with focused pane", "Split right…", "Split down…", "Zoom", "Close pane"])
+ expect(screen.getByRole("menuitem", { name })).toBeTruthy();
+ expect(screen.getByRole("menuitemcheckbox", { name: "Send right-clicks to pane" })).toBeTruthy();
+ const swapItem = screen.getByRole("menuitem", { name: "Swap with focused pane" });
+ fireEvent.mouseDown(swapItem);
+ expect(selected).not.toHaveBeenCalled();
+ fireEvent.click(swapItem);
+ expect(swap).toHaveBeenCalledOnce();
+ expect(screen.queryByRole("menu")).toBeNull();
+ });
+
+ it("forwards only after opt-in and retains Shift+right-click as a reversible escape hatch", () => {
+ const forwarded = vi.fn(), selected = vi.fn();
+ render(<Harness forwarded={forwarded} selected={selected} />);
+ const host = screen.getByTestId("terminal-content");
+ fireEvent.contextMenu(host);
+ expect((screen.getByRole("menuitem", { name: "Swap with focused pane" }) as HTMLButtonElement).disabled).toBe(true);
+ fireEvent.click(screen.getByRole("menuitemcheckbox"));
+ fireEvent.mouseDown(host, { button: 2 });
+ expect(forwarded).toHaveBeenCalledOnce();
+ fireEvent.contextMenu(host);
+ expect(screen.queryByRole("menu")).toBeNull();
+ fireEvent.mouseDown(host, { button: 2, shiftKey: true });
+ fireEvent.contextMenu(host, { shiftKey: true });
+ expect(forwarded).toHaveBeenCalledOnce();
+ expect(screen.getByRole("menuitemcheckbox").getAttribute("aria-checked")).toBe("true");
+ fireEvent.click(screen.getByRole("menuitemcheckbox"));
+ fireEvent.contextMenu(host);
+ expect(screen.getByRole("menuitemcheckbox").getAttribute("aria-checked")).toBe("false");
+ fireEvent.keyDown(screen.getByRole("menuitem", { name: "Close pane" }), { key: "ArrowUp" });
+ // Prompt history remains between forwarding and close; Home/End skip disabled actions.
+ fireEvent.keyDown(document.activeElement!, { key: "Home" });
+ fireEvent.keyDown(document.activeElement!, { key: "ArrowDown" });
+ expect(document.activeElement).toBe(screen.getByRole("menuitem", { name: "Split right…" }));
+ });
+
+ it("leaves legacy terminal gestures untouched", () => {
+ const forwarded = vi.fn();
+ render(<Harness enabled={false} forwarded={forwarded} selected={() => {}} />);
+ const host = screen.getByTestId("terminal-content");
+ fireEvent.mouseDown(host, { button: 2 });
+ const event = new MouseEvent("contextmenu", { bubbles: true, cancelable: true });
+ fireEvent(host, event);
+ expect(forwarded).toHaveBeenCalledOnce();
+ expect(event.defaultPrevented).toBe(false);
+ expect(screen.queryByRole("menu")).toBeNull();
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/RetainedWorkspaceGrid.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/RetainedWorkspaceGrid.test.tsx
new file mode 100644
index 0000000000..b2c497d1c2
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/RetainedWorkspaceGrid.test.tsx
@@ -0,0 +1,93 @@
+import { useEffect } from "react";
+import { cleanup, fireEvent, render, screen } from "@testing-library/react";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import type { SessionState } from "@/lib/agenticIdeApi";
+
+const lifecycle = vi.hoisted(() => ({ mount: vi.fn(), unmount: vi.fn() }));
+vi.mock("./WorkspaceTerminalGrid", () => ({
+ WorkspaceTerminalGrid: ({ session, active, disabled }: { session: SessionState; active: boolean; disabled: boolean }) => {
+ useEffect(() => {
+ lifecycle.mount(session.id);
+ return () => lifecycle.unmount(session.id);
+ }, []);
+ return <input aria-label={session.id} data-active={String(active)} disabled={disabled} defaultValue={session.name} />;
+ },
+}));
+import { RetainedWorkspaceGrid } from "./RetainedWorkspaceGrid";
+
+const session = (id: string, count = 1) => ({
+ id, name: id,
+ terminals: Array.from({ length: count }, (_, index) => ({ key: `${id}-${index}`, history_id: `${id}-${index}` })),
+}) as SessionState;
+const props = {
+ onScreen: true, selected: "", fontSize: 13, appearance: null,
+ onChanged: vi.fn(), onAdd: vi.fn(), onClose: vi.fn(), onSelect: vi.fn(),
+};
+beforeEach(() => { vi.clearAllMocks(); });
+afterEach(cleanup);
+
+describe("retained workspace terminals", () => {
+ it("keeps the same terminal view and input when switching away and back", () => {
+ const first = session("first");
+ const second = session("second");
+ const view = render(<RetainedWorkspaceGrid {...props} session={first} />);
+ const input = screen.getByLabelText("first");
+ fireEvent.change(input, { target: { value: "unfinished input" } });
+ view.rerender(<RetainedWorkspaceGrid {...props} session={second} />);
+ expect(input.closest("[hidden]")).not.toBeNull();
+ expect(input.getAttribute("data-active")).toBe("false");
+ expect(input.hasAttribute("disabled")).toBe(true);
+ view.rerender(<RetainedWorkspaceGrid {...props} session={{ ...first }} />);
+ expect(screen.getByLabelText("first")).toBe(input);
+ expect((input as HTMLInputElement).value).toBe("unfinished input");
+ expect(input.closest("[hidden]")).toBeNull();
+ expect(input.getAttribute("data-active")).toBe("true");
+ expect(lifecycle.mount).toHaveBeenCalledTimes(2);
+ expect(lifecycle.unmount).not.toHaveBeenCalled();
+ });
+
+ it("parks every retained view when the IDE is offscreen", () => {
+ const view = render(<RetainedWorkspaceGrid {...props} session={session("first")} />);
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session("second")} />);
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session("second")} onScreen={false} />);
+ expect(document.querySelectorAll("[data-active=true]")).toHaveLength(0);
+ expect(lifecycle.unmount).not.toHaveBeenCalled();
+ });
+
+ it("evicts the least recently visited workspace after three views", () => {
+ const view = render(<RetainedWorkspaceGrid {...props} session={session("first")} />);
+ for (const id of ["second", "third", "first", "fourth"]) {
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session(id)} />);
+ }
+ expect(lifecycle.unmount).toHaveBeenCalledExactlyOnceWith("second");
+ expect(screen.queryByLabelText("second")).toBeNull();
+ expect(document.querySelectorAll("input")).toHaveLength(3);
+ });
+
+ it("bounds retained panes while always allowing the active workspace", () => {
+ const view = render(<RetainedWorkspaceGrid {...props} session={session("first", 6)} />);
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session("second", 6)} />);
+ expect(lifecycle.unmount).not.toHaveBeenCalled();
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session("large", 16)} />);
+ expect(screen.queryByLabelText("first")).toBeNull();
+ expect(screen.queryByLabelText("second")).toBeNull();
+ expect(screen.getByLabelText("large")).toBeTruthy();
+ });
+
+ it("releases closed workspaces and views made stale by a transferred pane", () => {
+ const first = session("first");
+ const view = render(<RetainedWorkspaceGrid {...props} session={first} workspaceIds={["first", "second"]} />);
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session("second")} workspaceIds={["second"]} />);
+ expect(screen.queryByLabelText("first")).toBeNull();
+ view.rerender(<RetainedWorkspaceGrid {...props} session={{ ...session("third"), terminals: session("second").terminals }} />);
+ expect(screen.queryByLabelText("second")).toBeNull();
+ expect(screen.getByLabelText("third")).toBeTruthy();
+ });
+
+ it("releases all retained views on unmount", () => {
+ const view = render(<RetainedWorkspaceGrid {...props} session={session("first")} />);
+ view.rerender(<RetainedWorkspaceGrid {...props} session={session("second")} />);
+ view.unmount();
+ expect(lifecycle.unmount.mock.calls).toEqual([["second"], ["first"]]);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/RetainedWorkspaceGrid.tsx b/jarvis/ui/web/frontend/src/components/agentic/RetainedWorkspaceGrid.tsx
new file mode 100644
index 0000000000..1705fa6269
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/RetainedWorkspaceGrid.tsx
@@ -0,0 +1,41 @@
+import { useLayoutEffect, useRef, type ComponentProps } from "react";
+import { WorkspaceTerminalGrid } from "./WorkspaceTerminalGrid";
+
+type GridProps = ComponentProps<typeof WorkspaceTerminalGrid>;
+
+// Bound retained renderers and scrollback. A larger active workspace is always
+// allowed, but then keeps no background workspace alive.
+const MAX_WARM_PANES = 12;
+const MAX_WARM_WORKSPACES = 3;
+
+type Props = GridProps & {
+ onScreen: boolean;
+ workspaceIds?: readonly string[];
+};
+
+/** Keep recent terminal views alive instead of replaying their history on every switch. */
+export function RetainedWorkspaceGrid({ onScreen, workspaceIds, ...current }: Props) {
+ const retained = useRef<GridProps[]>([]);
+ const activePaneIds = new Set(current.session.terminals.flatMap((pane) => pane.history_id ? [pane.history_id] : []));
+ const entries = [current];
+ let panes = current.session.terminals.length;
+ for (const previous of retained.current) {
+ if (previous.session.id === current.session.id) continue;
+ if (workspaceIds && !workspaceIds.includes(previous.session.id)) continue;
+ // A transferred pane belongs to its new workspace. Do not retain another
+ // viewer from the source workspace's stale snapshot.
+ if (previous.session.terminals.some((pane) => pane.history_id && activePaneIds.has(pane.history_id))) continue;
+ if (entries.length >= MAX_WARM_WORKSPACES || panes + previous.session.terminals.length > MAX_WARM_PANES) continue;
+ entries.push(previous);
+ panes += previous.session.terminals.length;
+ }
+ useLayoutEffect(() => { retained.current = entries; });
+
+ return <>{entries.map((entry) => {
+ const visible = entry.session.id === current.session.id;
+ const active = onScreen && visible;
+ return <div key={entry.session.id} hidden={!visible} className="h-full min-h-0" data-workspace-terminal-view={entry.session.id}>
+ <WorkspaceTerminalGrid {...entry} active={active} disabled={!active || entry.disabled} />
+ </div>;
+ })}</>;
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceLauncher.tsx b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceLauncher.tsx
index 13dbf9177a..1d15d0d0ee 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceLauncher.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceLauncher.tsx
@@ -328,6 +328,7 @@ export function WorkspaceLauncher({
selected={folder}
onSelect={onSelectFolder}
onSelectRecent={onSelectRecent}
+ showSelection={false}
/>
</div>
)}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.test.tsx
index bdcf78404e..cb6b34c668 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.test.tsx
@@ -8,10 +8,13 @@ vi.mock("@/store/events", () => ({ useEventStore: (select: (state: unknown) => u
vi.mock("./AgenticTerminal", () => ({ AgenticTerminal: (props: {
name: string; onToggleMaximize: () => void; onRestart: () => void; restartToken: number;
onFocus: () => void;
+ active?: boolean; focused?: boolean;
+ onSwapWithFocused?: () => void;
onArrangeStart?: (event: React.PointerEvent) => void;
headerMode?: string; markFocus?: boolean;
-}) => <div onMouseDown={props.onFocus} data-testid={`pane-${props.name}`} data-header-mode={props.headerMode} data-mark-focus={String(props.markFocus)}>
+}) => <div onMouseDown={props.onFocus} data-testid={`pane-${props.name}`} data-header-mode={props.headerMode} data-mark-focus={String(props.markFocus)} data-active={String(props.active)} data-focused={String(props.focused)}>
<button onClick={props.onToggleMaximize}>Maximize {props.name}</button>
+ <button disabled={!props.onSwapWithFocused} onClick={props.onSwapWithFocused}>Swap {props.name}</button>
<button type="button" data-ide-drag-handle="true" onPointerDown={props.onArrangeStart}>Move {props.name}</button>
<button onClick={props.onRestart}>Restart {props.name}</button>
<output data-testid={`restart-${props.name}`}>{props.restartToken}</output>
@@ -52,6 +55,29 @@ function Controlled() {
beforeEach(() => { vi.clearAllMocks(); });
afterEach(cleanup);
+it("parks retained terminals and releases keyboard focus while their workspace is inactive", () => {
+ const session = makeSession(["T1"]);
+ const view = render(<WorkspaceTerminalGrid {...props} session={session} selected="T1" />);
+ const pane = screen.getByTestId("pane-T1");
+ expect(pane.getAttribute("data-active")).toBe("true");
+ expect(pane.getAttribute("data-focused")).toBe("true");
+ view.rerender(<WorkspaceTerminalGrid {...props} session={session} selected="T1" active={false} />);
+ expect(screen.getByTestId("pane-T1")).toBe(pane);
+ expect(pane.getAttribute("data-active")).toBe("false");
+ expect(pane.getAttribute("data-focused")).toBe("false");
+});
+
+it("swaps the context pane with the selected pane by stable identity", async () => {
+ api.move.mockResolvedValue(makeSession());
+ const { rerender } = render(<WorkspaceTerminalGrid {...props} selected="T1" session={makeSession()} />);
+ expect((screen.getByRole("button", { name: "Swap T1" }) as HTMLButtonElement).disabled).toBe(true);
+ fireEvent.click(screen.getByRole("button", { name: "Swap T3" }));
+ await waitFor(() => expect(api.move).toHaveBeenCalledExactlyOnceWith("pane:T3", "pane:T1", "swap"));
+ await waitFor(() => expect((screen.getByRole("button", { name: "Swap T3" }) as HTMLButtonElement).disabled).toBe(false));
+ rerender(<WorkspaceTerminalGrid {...props} selected="T1" disabled session={makeSession()} />);
+ expect((screen.getByRole("button", { name: "Swap T3" }) as HTMLButtonElement).disabled).toBe(true);
+});
+
it("swaps immediately, persists once and preserves mounted terminal nodes", async () => {
let finish!: (state: SessionState) => void;
api.move.mockImplementation(() => new Promise((resolve) => { finish = resolve; }));
@@ -222,6 +248,16 @@ it("drags the seam between two panes to resize them and saves the new sizes", as
} finally { vi.useRealTimers(); }
});
+it("leaves a maximize request pending until its retained workspace is active", () => {
+ act(() => useIdeChatStore.setState({ paneRequest: { workspaceId: "w1", pane: "T1", nonce: 6, maximize: true } }));
+ const session = makeSession();
+ const view = render(<WorkspaceTerminalGrid {...props} session={session} active={false} />);
+ expect(useIdeChatStore.getState().paneRequest?.maximize).toBe(true);
+ view.rerender(<WorkspaceTerminalGrid {...props} session={session} active />);
+ expect(useIdeChatStore.getState().paneRequest?.maximize).toBe(false);
+ act(() => useIdeChatStore.setState({ paneRequest: null }));
+});
+
it("maximizes the pane the office asked for, once, and only in its own workspace", () => {
act(() => useIdeChatStore.setState({ paneRequest: { workspaceId: "w2", pane: "T3", nonce: 7, maximize: true } }));
const { unmount } = render(<WorkspaceTerminalGrid {...props} session={makeSession()} />);
diff --git a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.tsx b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.tsx
index c52c574e32..7e859c3de4 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalGrid.tsx
@@ -39,6 +39,8 @@ function writeStyle(node: HTMLElement, style: React.CSSProperties) {
}
interface Props {
+ /** Hidden retained workspaces keep their sockets but park output and focus. */
+ active?: boolean;
session: SessionState;
onChanged: (session: SessionState) => void;
/** Open the agent picker; with an anchor, the new pane splits off that pane. */
@@ -69,7 +71,7 @@ interface DragFeedback { id: string; target: DropTarget | null; x: number; y: nu
/** The attribute a sidebar row carries when a dragged pane may be dropped on it. */
const WORKSPACE_DROP = "data-pane-drop-workspace";
-export function WorkspaceTerminalGrid({ session, onChanged, onAdd, onClose, onSelect, selected, maxPanes = MAX_WORKSPACE_PANES, fontSize, appearance, disabled = false, onMutationStart, onMutationEnd, paneStyle: look = "classic", workspaces = [] }: Props) {
+export function WorkspaceTerminalGrid({ session, onChanged, onAdd, onClose, onSelect, selected, maxPanes = MAX_WORKSPACE_PANES, fontSize, appearance, disabled = false, onMutationStart, onMutationEnd, paneStyle: look = "classic", workspaces = [], active = true }: Props) {
const theme = useThemeValue();
const pushToast = useEventStore((state) => state.pushToast);
// The pane an agent card in the side panel pointed at, framed in blue.
@@ -145,12 +147,12 @@ export function WorkspaceTerminalGrid({ session, onChanged, onAdd, onClose, onSe
const paneRequest = useIdeChatStore((state) => state.paneRequest);
const settlePaneMaximize = useIdeChatStore((state) => state.settlePaneMaximize);
useEffect(() => {
- if (!paneRequest?.maximize || paneRequest.workspaceId !== session.id) return;
+ if (!active || !paneRequest?.maximize || paneRequest.workspaceId !== session.id) return;
const terminal = session.terminals.find((entry) => entry.name === paneRequest.pane);
if (!terminal) return;
settlePaneMaximize(paneRequest.nonce);
setMaximized(idOf(terminal));
- }, [paneRequest, session.id, session.terminals, settlePaneMaximize]);
+ }, [active, paneRequest, session.id, session.terminals, settlePaneMaximize]);
useEffect(() => { if (disabled) dragCleanup.current?.(); }, [disabled]);
useEffect(() => {
const node = frame.current;
@@ -446,11 +448,16 @@ export function WorkspaceTerminalGrid({ session, onChanged, onAdd, onClose, onSe
drag?.id === id && "opacity-50",
spotlitPane === terminal.name && "ring-2 ring-accent ring-offset-2 ring-offset-background",
visibleMaximized && visibleMaximized !== id && "hidden")}>
- <AgenticTerminal headerMode={minimal ? "minimal" : "compact"} agent={terminal.agent}
+ <AgenticTerminal active={active} headerMode={minimal ? "minimal" : "compact"} agent={terminal.agent}
name={terminal.name} workspaceId={session.id} displayName={terminal.display_name}
recap={terminal.recap} promptCount={terminal.prompts_sent} appearance={appearance ?? theme} fontSize={fontSize}
- focused={selected === terminal.name} markFocus={!panelInUse}
+ focused={active && selected === terminal.name} markFocus={!panelInUse}
onFocus={() => { if (spotlitPane && spotlitPane !== terminal.name) setSpotlight(null); setPanelInUse(false); onSelect(terminal.name); }}
+ onSwapWithFocused={saving || disabled || selected === terminal.name || !tiles.some((entry) => entry.name === selected)
+ ? undefined : () => {
+ const target = tiles.find((entry) => entry.name === selected);
+ if (target) void move(id, idOf(target));
+ }}
layoutBusy={resizing}
maximized={visibleMaximized === id} onToggleMaximize={() => setMaximized((current) => current === id ? null : id)}
onArrangeStart={visibleMaximized || saving || disabled ? undefined : (event) => startDrag(id, event)} arranging={drag?.id === id}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.test.tsx
index 64692d471c..57c50b3348 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.test.tsx
@@ -79,10 +79,10 @@ describe("compact workspace terminal header", () => {
const event = new MouseEvent("contextmenu", { bubbles: true, cancelable: true, clientX: 40, clientY: 30 });
fireEvent(screen.getByText("Dana"), event);
expect(event.defaultPrevented).toBe(true);
- expect(activate).toHaveBeenCalled();
+ expect(activate).not.toHaveBeenCalled();
const menu = screen.getByRole("menu", { name: "Actions for Dana" });
expect(within(menu).getAllByRole("menuitem").map((item) => item.textContent)).toEqual(expect.arrayContaining(
- ["Split right…", "Split down…", "Split left…", "Split up…", "Maximize", "Close pane"]));
+ ["Split right…", "Split down…", "Split left…", "Split up…", "Zoom", "Close pane"]));
fireEvent.click(within(menu).getByRole("menuitem", { name: "Split left…" }));
expect(add).toHaveBeenCalledWith("left");
fireEvent(screen.getByText("Dana"), new MouseEvent("contextmenu", { bubbles: true, cancelable: true }));
@@ -103,7 +103,7 @@ describe("compact workspace terminal header", () => {
render(<WorkspaceTerminalHeader {...BASE} onRename={async () => true} onOpenConversation={() => {}} />);
const more = screen.getByRole("button", { name: "More actions for Dana" });
fireEvent.click(more);
- expect(document.activeElement).toBe(screen.getByRole("menuitem", { name: "Rename" }));
+ expect(document.activeElement).toBe(screen.getByRole("menuitem", { name: "Rename pane" }));
fireEvent.keyDown(document.activeElement!, { key: "ArrowDown" });
expect(document.activeElement).toBe(screen.getByRole("menuitem", { name: "Conversation history" }));
fireEvent.keyDown(document.activeElement!, { key: "Escape" });
@@ -129,7 +129,7 @@ describe("compact workspace terminal header", () => {
const rename = vi.fn().mockResolvedValue(false);
render(<WorkspaceTerminalHeader {...BASE} onRename={rename} />);
fireEvent.click(screen.getByRole("button", { name: "More actions for Dana" }));
- fireEvent.click(screen.getByRole("menuitem", { name: "Rename" }));
+ fireEvent.click(screen.getByRole("menuitem", { name: "Rename pane" }));
fireEvent.change(screen.getByRole("textbox", { name: "Name for Dana" }), { target: { value: "Installer" } });
await act(async () => { fireEvent.click(screen.getByRole("button", { name: "Save name" })); });
expect(rename).toHaveBeenCalledWith("Installer");
diff --git a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.tsx b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.tsx
index cc0527c3f3..3b671571e6 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/WorkspaceTerminalHeader.tsx
@@ -7,10 +7,15 @@ import { usePaneTitle } from "@/store/paneRecaps";
import { PromptHistoryButton } from "./PromptHistoryButton";
import { SplitAboveIcon, SplitBelowIcon, SplitLeftIcon, SplitRightIcon } from "./splitIcons";
import { PANE_BRAND, PANE_CHROME, PANE_TILE, themeFor, type PaneEdgeState, type TerminalAppearance } from "./terminalThemes";
+import type { PaneMenuRequest } from "./usePaneContextMenu";
export type PaneSplitDirection = "right" | "down" | "left" | "above";
interface Props {
+ contextMenuRequest?: PaneMenuRequest | null;
+ onSwapWithFocused?: () => void;
+ sendRightClicks?: boolean;
+ onToggleSendRightClicks?: () => void;
name: string;
workspaceId?: string;
promptCount?: number;
@@ -52,7 +57,7 @@ interface Props {
}
type MenuIcon = ComponentType<SVGProps<SVGSVGElement>>;
-interface MenuItem { label: string; run: () => void; Icon?: MenuIcon; separated?: boolean }
+interface MenuItem { label: string; run: () => void; Icon?: MenuIcon; separated?: boolean; disabled?: boolean }
const SPLIT_ITEMS: { direction: PaneSplitDirection; label: string; Icon: MenuIcon }[] = [
{ direction: "right", label: "Split right", Icon: SplitRightIcon },
@@ -65,6 +70,7 @@ const ACTION_CLASS = "flex h-7 w-7 shrink-0 items-center justify-center rounded
/** Quiet workspace chrome; terminal rendering and connection state stay in AgenticTerminal. */
export function WorkspaceTerminalHeader({
+ contextMenuRequest, onSwapWithFocused, sendRightClicks = false, onToggleSendRightClicks,
name, workspaceId, promptCount = 0, agent, agentLogoUrl, displayName, status, appearance, arranging = false,
maximized = false, addDisabled = false, onArrangeStart, onActivate, onToggleMaximize,
onAdd, onClose, onRename, onOpenConversation, onOpenChat, onRestart, onFork, branch,
@@ -86,6 +92,9 @@ export function WorkspaceTerminalHeader({
const [renameError, setRenameError] = useState("");
const stopped = status === "exited" || status === "error";
const menuOpen = menuPosition !== null;
+ useEffect(() => {
+ if (contextMenuRequest) setMenuPosition(contextMenuRequest);
+ }, [contextMenuRequest]);
const variables = {
"--pane-ink": brand.ink,
"--pane-ink-muted": brand.inkMuted,
@@ -95,7 +104,7 @@ export function WorkspaceTerminalHeader({
useEffect(() => {
if (!menuOpen) return;
- menuRef.current?.querySelector<HTMLButtonElement>("[role=menuitem]")?.focus();
+ menuRef.current?.querySelector<HTMLButtonElement>("button[role^=menuitem]:not(:disabled)")?.focus();
const outside = (event: globalThis.PointerEvent) => {
if (event.target instanceof Node && !menuRef.current?.contains(event.target) && !moreRef.current?.contains(event.target)) setMenuPosition(null);
};
@@ -130,7 +139,6 @@ export function WorkspaceTerminalHeader({
if (event.shiftKey || draft !== null) return;
event.preventDefault();
event.stopPropagation();
- onActivate?.();
setMenuPosition({ left: event.clientX, top: event.clientY });
};
const choose = (action: () => void) => { setMenuPosition(null); action(); };
@@ -164,7 +172,7 @@ export function WorkspaceTerminalHeader({
</form>;
const menu = createPortal(<div ref={menuRef} id={menuId} role="menu" hidden={!menuOpen} aria-label={`Actions for ${name}`}
- className={`fixed z-[90] w-[200px] border p-1 shadow-lg ${tile ? "rounded-none" : "rounded-lg"}`}
+ className={`fixed z-[90] w-[260px] max-w-[calc(100vw-16px)] border p-1 shadow-lg ${tile ? "rounded-none" : "rounded-lg"}`}
style={{ ...variables, ...(menuPosition ?? { left: 0, top: 0 }), borderColor: chrome.border, background: chrome.float }}
onPointerDown={(event) => event.stopPropagation()}
onMouseDown={(event) => event.stopPropagation()}
@@ -176,18 +184,19 @@ export function WorkspaceTerminalHeader({
if (event.key === "Tab") { setMenuPosition(null); moreRef.current?.focus(); return; }
if (!["ArrowUp", "ArrowDown", "Home", "End"].includes(event.key)) return;
event.preventDefault();
- const items = Array.from(menuRef.current?.querySelectorAll<HTMLButtonElement>("[role=menuitem]") ?? []);
+ const items = Array.from(menuRef.current?.querySelectorAll<HTMLButtonElement>("button[role^=menuitem]:not(:disabled)") ?? []);
const at = items.indexOf(document.activeElement as HTMLButtonElement);
const index = event.key === "Home" ? 0 : event.key === "End" ? items.length - 1 : (at + (event.key === "ArrowDown" ? 1 : -1) + items.length) % items.length;
items[index]?.focus();
}}>
{([
- onRename && { label: "Rename", run: () => { setDraft(name); setRenameError(""); } },
+ onRename && { label: "Rename pane", run: () => { setDraft(name); setRenameError(""); } },
+ { label: "Swap with focused pane", disabled: !onSwapWithFocused, run: () => onSwapWithFocused?.() },
onFork && !addDisabled && { label: "Fork…", Icon: BranchIcon, run: onFork },
...(onAdd && !addDisabled ? SPLIT_ITEMS.map((item, index) => ({
label: `${item.label}…`, Icon: item.Icon, separated: index === 0, run: () => onAdd(item.direction),
})) : []),
- onToggleMaximize && { label: maximized ? "Restore size" : "Maximize", separated: true,
+ onToggleMaximize && { label: maximized ? "Unzoom" : "Zoom", separated: true,
Icon: maximized ? Minimize2 : Maximize2, run: onToggleMaximize },
onOpenConversation && { label: "Conversation history", run: onOpenConversation },
onOpenChat && { label: "Open as chat", run: onOpenChat },
@@ -195,11 +204,18 @@ export function WorkspaceTerminalHeader({
...(workspaceItems ?? []).map((item, index) => ({ ...item, Icon: FolderInput, separated: index === 0 })),
...(placementItems ?? []).map((item, index) => ({ ...item, Icon: Server, separated: index === 0 })),
] as (MenuItem | false | undefined | null)[]).filter((item): item is MenuItem => Boolean(item)).map((item) =>
- <button type="button" role="menuitem" key={item.label} onClick={() => choose(item.run)}
- className={`flex w-full items-center gap-2 rounded px-2.5 py-2 text-left text-xs hover:bg-[color:var(--pane-chip)] focus:bg-[color:var(--pane-chip)] focus:outline-none ${item.separated ? "mt-1 border-t border-[color:var(--pane-chip)] pt-2" : ""}`}>
+ <button type="button" role="menuitem" key={item.label} disabled={item.disabled} onClick={() => choose(item.run)}
+ className={`flex w-full items-center gap-2 rounded px-2.5 py-2 text-left text-xs disabled:opacity-40 hover:bg-[color:var(--pane-chip)] focus:bg-[color:var(--pane-chip)] focus:outline-none ${item.separated ? "mt-1 border-t border-[color:var(--pane-chip)] pt-2" : ""}`}>
{item.Icon && <item.Icon className="h-3.5 w-3.5 shrink-0" aria-hidden="true" />}{item.label}
</button>,
)}
+ {onToggleSendRightClicks && <button type="button" role="menuitemcheckbox" aria-checked={sendRightClicks}
+ onClick={() => choose(onToggleSendRightClicks)}
+ title="Shift+right-click opens the pane menu while forwarding is enabled."
+ className="flex w-full items-center gap-2 rounded px-2.5 py-2 text-left text-xs hover:bg-[color:var(--pane-chip)] focus:bg-[color:var(--pane-chip)] focus:outline-none">
+ <Check aria-hidden="true" className={`h-3.5 w-3.5 shrink-0 ${sendRightClicks ? "" : "invisible"}`} />
+ <span>Send right-clicks to pane{sendRightClicks && <span className="block text-[10px] text-[color:var(--pane-ink-muted)]">Shift + right-click: pane menu</span>}</span>
+ </button>}
<PromptHistoryButton terminal={name} workspaceId={workspaceId} count={promptCount} triggerMode="menu-item"
onOpen={() => setMenuPosition(null)} restoreFocus={() => moreRef.current?.focus()} />
{onClose && <button type="button" role="menuitem" onClick={() => choose(onClose)}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/paneFont.ts b/jarvis/ui/web/frontend/src/components/agentic/paneFont.ts
index 477f4dea92..8c2ac45d80 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/paneFont.ts
+++ b/jarvis/ui/web/frontend/src/components/agentic/paneFont.ts
@@ -49,7 +49,11 @@ export function storedFontSize(): number | null {
}
}
-/** The size a pane will actually open at — the stored one, or the default. */
+/**
+ * The size a pane opens at: always the default. The workspace terminals use
+ * one fixed, dense size (maintainer decision 2026-09-28), so a size stored by
+ * an older build is ignored.
+ */
export function paneFontSize(): number {
- return storedFontSize() ?? FONT_DEFAULT;
+ return FONT_DEFAULT;
}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/terminalPromptSelection.test.ts b/jarvis/ui/web/frontend/src/components/agentic/terminalPromptSelection.test.ts
new file mode 100644
index 0000000000..3813e89e2b
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/terminalPromptSelection.test.ts
@@ -0,0 +1,232 @@
+import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
+import { Terminal } from "@xterm/xterm";
+import type { IBufferRange } from "@xterm/xterm";
+import { installPromptSelectionBridge } from "./terminalPromptSelection";
+
+// These tests parse cells without rendering. xterm's colour fallback supports
+// a missing canvas context; jsdom otherwise logs its unimplemented method.
+const restoreCanvas = vi.hoisted(() => {
+ const original = HTMLCanvasElement.prototype.getContext;
+ HTMLCanvasElement.prototype.getContext = () => null;
+ return () => { HTMLCanvasElement.prototype.getContext = original; };
+});
+afterAll(restoreCanvas);
+
+const terminals: Terminal[] = [];
+afterEach(() => { terminals.splice(0).forEach((term) => term.dispose()); });
+
+/** Parse real terminal cells, with a fake selection/input transport. No browser. */
+async function setup(output = "› hello world", isMac = false, cols = 40) {
+ const terminal = new Terminal({ cols, rows: 10, allowProposedApi: true });
+ terminals.push(terminal);
+ const write = (text: string) => new Promise<void>((resolve) => terminal.write(text, resolve));
+ await write(output);
+ let selected: IBufferRange | undefined;
+ let onSelection = () => {};
+ let onRender = () => {};
+ let keyHandler = (_event: KeyboardEvent) => true;
+ const sent: string[] = [];
+ let listenerDisposed = false;
+ let handlerDisposed = false;
+ const cleanup = installPromptSelectionBridge({
+ cols,
+ buffer: terminal.buffer,
+ getSelectionPosition: () => selected,
+ select: (x, y, length) => {
+ const end = y * cols + x + length;
+ selected = { start: { x, y }, end: { x: end % cols, y: Math.floor(end / cols) } };
+ onSelection();
+ },
+ clearSelection: () => { selected = undefined; onSelection(); },
+ input: (data) => sent.push(data),
+ onSelectionChange: (listener) => {
+ onSelection = listener;
+ return { dispose: () => { listenerDisposed = true; onSelection = () => {}; } };
+ },
+ onRender: (listener) => {
+ onRender = listener;
+ return { dispose: () => { onRender = () => {}; } };
+ },
+ }, (handler) => {
+ keyHandler = handler;
+ return () => { handlerDisposed = true; keyHandler = () => true; };
+ }, isMac);
+ return {
+ write, sent, cleanup, renderFrame: () => onRender(),
+ selection: () => selected,
+ disposed: () => listenerDisposed && handlerDisposed,
+ select: (start: number, end: number, row = 0, endRow = row) => {
+ selected = { start: { x: start, y: row }, end: { x: end, y: endRow } };
+ onSelection();
+ },
+ press: (key: string, mods: KeyboardEventInit = {}, type = "keydown") => {
+ const event = new KeyboardEvent(type, { key, cancelable: true, ...mods });
+ return { passthrough: keyHandler(event), prevented: event.defaultPrevented };
+ },
+ };
+}
+
+describe("terminal prompt selection", () => {
+ it.each(["Backspace", "Delete"])("deletes the whole selected draft with one %s press", async (key) => {
+ const pane = await setup();
+ pane.select(2, 13);
+ expect(pane.press(key)).toEqual({ passthrough: false, prevented: true });
+ expect(pane.sent).toEqual(["\x7f".repeat(11)]);
+ expect(pane.selection()).toBeUndefined();
+ pane.press(key, {}, "keyup");
+ expect(pane.sent).toHaveLength(1);
+ });
+
+ it("deletes a middle selection while leaving the suffix in place", async () => {
+ const pane = await setup();
+ pane.select(2, 7);
+ pane.press("Delete");
+ expect(pane.sent).toEqual(["\x1b[D".repeat(6) + "\x7f".repeat(5)]);
+ });
+
+ it("moves right when the caret precedes the selected text", async () => {
+ const pane = await setup("❯ hello world\x1b[11D");
+ pane.select(8, 13);
+ pane.press("Backspace");
+ expect(pane.sent).toEqual(["\x1b[C".repeat(11) + "\x7f".repeat(5)]);
+ });
+
+ it.each([false, true])("selects only input using the platform select-all chord (mac=%s)", async (isMac) => {
+ const pane = await setup("old output\r\n› hello world", isMac);
+ expect(pane.press("a", isMac ? { metaKey: true } : { ctrlKey: true }).passthrough).toBe(false);
+ expect(pane.selection()).toEqual({ start: { x: 2, y: 1 }, end: { x: 13, y: 1 } });
+ pane.press("Backspace");
+ expect(pane.sent).toEqual(["\x7f".repeat(11)]);
+ });
+
+ it("excludes the prompt marker and terminal padding from a full-row drag", async () => {
+ const pane = await setup(" > hello \x1b[5D");
+ pane.select(0, 39);
+ pane.press("Delete");
+ expect(pane.sent).toEqual(["\x7f".repeat(5)]);
+ });
+
+ it("handles an input line wrapped by the terminal", async () => {
+ const pane = await setup("› hello world", false, 10);
+ pane.select(2, 3, 0, 1);
+ pane.press("Backspace");
+ expect(pane.sent).toEqual(["\x7f".repeat(11)]);
+ });
+
+ it("handles select-all ending at the last terminal column", async () => {
+ const pane = await setup("› abcdefgh", false, 10);
+ pane.press("a", { ctrlKey: true });
+ pane.press("Delete");
+ expect(pane.sent).toEqual(["\x7f".repeat(8)]);
+ });
+
+ it("edits a prompt drawn in the alternate screen too", async () => {
+ const pane = await setup("\x1b[?1049h› hello");
+ pane.select(2, 7);
+ pane.press("Backspace");
+ expect(pane.sent).toEqual(["\x7f".repeat(5)]);
+ });
+
+ it("counts wide and combining characters as editing units, not cells or bytes", async () => {
+ const pane = await setup("› 界e\u0301z");
+ pane.select(2, 5);
+ pane.press("Delete");
+ expect(pane.sent).toEqual(["\x1b[D" + "\x7f\x7f"]);
+ });
+
+ it("does not split a wide character", async () => {
+ const pane = await setup("› 界z");
+ pane.select(3, 4);
+ expect(pane.press("Delete").passthrough).toBe(false);
+ expect(pane.sent).toEqual([]);
+ });
+
+ it("never edits the draft when output is selected", async () => {
+ const pane = await setup("earlier output\r\n› hello");
+ pane.select(0, 7);
+ expect(pane.press("Backspace").passthrough).toBe(false);
+ expect(pane.sent).toEqual([]);
+ });
+
+ it("rejects a selection made before a CLI redraw changed the draft", async () => {
+ const pane = await setup();
+ pane.select(2, 13);
+ await pane.write("\r\x1b[2K› new draft");
+ pane.renderFrame();
+ pane.press("Delete");
+ expect(pane.sent).toEqual([]);
+ });
+
+ it("keeps a mouse selection made while the CLI is painting another row", async () => {
+ const pane = await setup("status\r\n› dsadas");
+ // A synchronized TUI frame leaves the old prompt visible while the parser
+ // cursor is temporarily on a status row. Mouseup can happen in that gap.
+ await pane.write("\x1b[1;1HWorking");
+ pane.select(2, 8, 1);
+ await pane.write("\x1b[2;9H");
+ pane.renderFrame();
+ pane.press("Delete");
+ expect(pane.sent).toEqual(["\x7f".repeat(6)]);
+ });
+
+ it("defers Delete until an in-flight repaint restores the editor cursor", async () => {
+ const pane = await setup("status\r\n› dsadas");
+ pane.select(2, 8, 1);
+ await pane.write("\x1b[1;1HWorking");
+ expect(pane.press("Delete").passthrough).toBe(false);
+ expect(pane.sent).toEqual([]);
+ await pane.write("\x1b[2;9H");
+ pane.renderFrame();
+ expect(pane.sent).toEqual(["\x7f".repeat(6)]);
+ pane.cleanup();
+ });
+
+ it("does not apply deferred deletion to a changed draft", async () => {
+ const pane = await setup("status\r\n› dsadas");
+ pane.select(2, 8, 1);
+ await pane.write("\x1b[1;1HWorking");
+ pane.press("Delete");
+ await pane.write("\x1b[2;1H\x1b[2K› new text");
+ pane.renderFrame();
+ expect(pane.sent).toEqual([]);
+ pane.cleanup();
+ });
+
+ it.each(["selection", "key", "dispose"])("cancels deferred deletion after a later %s action", async (action) => {
+ const pane = await setup("status\r\n› dsadas");
+ pane.select(2, 8, 1);
+ await pane.write("\x1b[1;1HWorking");
+ pane.press("Delete");
+ if (action === "selection") pane.select(2, 4, 1);
+ if (action === "key") pane.press("ArrowLeft");
+ if (action === "dispose") pane.cleanup();
+ await pane.write("\x1b[2;9H");
+ pane.renderFrame();
+ expect(pane.sent).toEqual([]);
+ pane.cleanup();
+ });
+
+ it.each(["shell output", "› [Pasted text 20 lines]", "› [Image #1]"])(
+ "leaves an unsupported editor alone: %s", async (output) => {
+ const pane = await setup(output);
+ expect(pane.press("a", { ctrlKey: true }).passthrough).toBe(true);
+ pane.select(2, 7);
+ pane.press("Delete");
+ expect(pane.sent).toEqual([]);
+ },
+ );
+
+ it("leaves ordinary editing, modified keys and IME composition to the CLI", async () => {
+ const pane = await setup();
+ expect(pane.press("Backspace").passthrough).toBe(true);
+ pane.select(2, 7);
+ for (const mods of [{ ctrlKey: true }, { altKey: true }, { metaKey: true }, { isComposing: true }]) {
+ expect(pane.press("Backspace", mods).passthrough).toBe(true);
+ }
+ expect(pane.press("Enter").passthrough).toBe(true);
+ expect(pane.sent).toEqual([]);
+ pane.cleanup();
+ expect(pane.disposed()).toBe(true);
+ expect(pane.press("Delete").passthrough).toBe(true);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/terminalPromptSelection.ts b/jarvis/ui/web/frontend/src/components/agentic/terminalPromptSelection.ts
new file mode 100644
index 0000000000..aaebe50fe8
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/terminalPromptSelection.ts
@@ -0,0 +1,178 @@
+import type { IBuffer, IBufferCellPosition, IBufferRange } from "@xterm/xterm";
+
+/** The public xterm surface needed to edit a selected, visible prompt. */
+export interface PromptSelectionTerminal {
+ cols: number;
+ buffer: { active: IBuffer };
+ getSelectionPosition(): IBufferRange | undefined;
+ select(column: number, row: number, length: number): void;
+ clearSelection(): void;
+ input(data: string): void;
+ onSelectionChange(listener: () => void): { dispose(): void };
+ onRender(listener: () => void): { dispose(): void };
+}
+
+interface PromptLine {
+ text: string;
+ start: IBufferCellPosition;
+ end: IBufferCellPosition;
+ cursor: number;
+ offsets: Map<number, number>;
+}
+
+const segmenter = typeof Intl.Segmenter === "function"
+ ? new Intl.Segmenter(undefined, { granularity: "grapheme" }) : null;
+const graphemes = (text: string) => [...segmenter!.segment(text)].length;
+
+/**
+ * Only the logical input line containing the live cursor is editable. A
+ * terminal selection otherwise belongs to output, not to the CLI's editor.
+ * Recognise the same input markers as agentic_ide/session.py, without using
+ * provider names. Wrapped buffer rows are one line; separate TUI rows are not
+ * guessed to be input (they can be menus, permission questions or output).
+ */
+function readPrompt(term: PromptSelectionTerminal): PromptLine | null {
+ const buffer = term.buffer.active;
+ if (!segmenter) return null;
+ const cursorRow = buffer.baseY + buffer.cursorY;
+ let first = cursorRow;
+ while (first > buffer.baseY && buffer.getLine(first)?.isWrapped) first--;
+ const head = buffer.getLine(first)?.translateToString(false, 0, term.cols);
+ const prefix = head?.match(/^ *[❯›>] /)?.[0];
+ if (!prefix) return null;
+ let last = cursorRow;
+ while (buffer.getLine(last + 1)?.isWrapped) last++;
+
+ const offsets = new Map<number, number>();
+ let text = "";
+ const start = { x: prefix.length, y: first };
+ let end = start;
+ for (let row = first; row <= last; row++) {
+ const line = buffer.getLine(row);
+ if (!line) return null;
+ const left = row === first ? start.x : 0;
+ // Keep typed spaces through the caret, but omit empty cells after input.
+ let right = term.cols;
+ if (row === last) {
+ right = left;
+ for (let col = left; col < term.cols; col++) {
+ const cell = line.getCell(col);
+ if (cell?.getChars().trim()) right = col + cell.getWidth();
+ }
+ if (row === cursorRow) right = Math.max(right, buffer.cursorX);
+ }
+ for (let col = left; col < right; col++) {
+ const cell = line.getCell(col);
+ if (!cell || cell.getWidth() === 0) continue;
+ offsets.set(row * term.cols + col, text.length);
+ text += cell.getChars() || " ";
+ }
+ offsets.set(row * term.cols + right, text.length);
+ end = { x: right, y: row };
+ }
+ const cursor = offsets.get(cursorRow * term.cols + buffer.cursorX);
+ // Collapsed paste/attachment tokens have editor-specific cursor widths.
+ // Sending one Backspace per displayed letter could erase adjacent input.
+ if (cursor === undefined || /\[(?:Pasted|Image|Attachment)\b/i.test(text)) return null;
+ return { text, start, end, cursor, offsets };
+}
+
+const samePrompt = (a: PromptLine | null, b: PromptLine | null) =>
+ a !== null && b !== null && a.text === b.text && a.cursor === b.cursor &&
+ a.start.x === b.start.x && a.start.y === b.start.y &&
+ a.end.x === b.end.x && a.end.y === b.end.y;
+
+/**
+ * Turn mouse selection + Backspace/Delete into editor cursor moves and deletes.
+ * Never erase the rendered buffer: the CLI owns the draft and must redraw it.
+ */
+export function installPromptSelectionBridge(
+ term: PromptSelectionTerminal,
+ addKeyHandler: (handler: (event: KeyboardEvent) => boolean) => () => void,
+ isMac: boolean,
+): () => void {
+ let selectedPrompt: PromptLine | null = null;
+ // A TUI moves the parser cursor while repainting. Selection belongs to the
+ // last presented frame, not to that temporary drawing cursor.
+ let paintedPrompt = readPrompt(term);
+ let pending: { prompt: PromptLine; selection: IBufferRange } | null = null;
+ let pendingTimer: ReturnType<typeof setTimeout> | undefined;
+ const cancelPending = () => {
+ pending = null;
+ if (pendingTimer !== undefined) clearTimeout(pendingTimer);
+ pendingTimer = undefined;
+ };
+ const selectionListener = term.onSelectionChange(() => {
+ cancelPending();
+ selectedPrompt = term.getSelectionPosition() ? paintedPrompt : null;
+ });
+ const erase = (prompt: PromptLine, selection: IBufferRange) => {
+ const key = (pos: IBufferCellPosition) => pos.y * term.cols + pos.x;
+ if (selection.start.y < prompt.start.y ||
+ key(selection.end) > (prompt.end.y + 1) * term.cols) return false;
+ const start = prompt.offsets.get(Math.max(key(selection.start), key(prompt.start)));
+ const end = prompt.offsets.get(Math.min(key(selection.end), key(prompt.end)));
+ if (start === undefined || end === undefined || start >= end) return false;
+ // A cell boundary can be inside an emoji composed of several code points.
+ const boundaries = new Set([0, ...[...segmenter!.segment(prompt.text)].map((s) => s.index + s.segment.length)]);
+ if (![start, end, prompt.cursor].every((offset) => boundaries.has(offset))) return false;
+ const move = end < prompt.cursor
+ ? "\x1b[D".repeat(graphemes(prompt.text.slice(end, prompt.cursor)))
+ : "\x1b[C".repeat(graphemes(prompt.text.slice(prompt.cursor, end)));
+ const backspaces = "\x7f".repeat(graphemes(prompt.text.slice(start, end)));
+ term.clearSelection();
+ selectedPrompt = null;
+ // One input frame, no Enter, no Ctrl+C and no clipboard round trip.
+ term.input(move + backspaces);
+ return false;
+ };
+ const renderListener = term.onRender(() => {
+ paintedPrompt = readPrompt(term);
+ if (!pending || !paintedPrompt) return;
+ const request = pending;
+ const selection = term.getSelectionPosition();
+ cancelPending();
+ if (samePrompt(request.prompt, paintedPrompt) && selection &&
+ selection.start.x === request.selection.start.x && selection.start.y === request.selection.start.y &&
+ selection.end.x === request.selection.end.x && selection.end.y === request.selection.end.y) {
+ erase(paintedPrompt, selection);
+ }
+ });
+ const removeKeyHandler = addKeyHandler((event) => {
+ if (event.type !== "keydown") return true;
+ cancelPending();
+ if (event.isComposing || event.altKey) return true;
+ const selectAll = event.key.toLowerCase() === "a" && !event.shiftKey &&
+ (isMac ? event.metaKey && !event.ctrlKey : event.ctrlKey && !event.metaKey);
+ if (selectAll) {
+ const prompt = paintedPrompt;
+ if (!prompt) return true;
+ event.preventDefault();
+ const { start, end } = prompt;
+ term.select(start.x, start.y, (end.y - start.y) * term.cols + end.x - start.x);
+ selectedPrompt = prompt;
+ return false;
+ }
+ if (event.ctrlKey || event.metaKey || !["Backspace", "Delete"].includes(event.key)) return true;
+ const selection = term.getSelectionPosition();
+ if (!selection) return true;
+ // A selected output line must never delete unrelated prompt text.
+ event.preventDefault();
+ if (!selectedPrompt) return false;
+ const prompt = readPrompt(term);
+ if (prompt && samePrompt(selectedPrompt, prompt)) return erase(prompt, selection);
+ // Wait for one completed frame instead of silently discarding Delete in
+ // the middle of a synchronized repaint. Never retry a changed draft.
+ if (samePrompt(selectedPrompt, paintedPrompt)) {
+ pending = { prompt: selectedPrompt, selection };
+ pendingTimer = setTimeout(cancelPending, 1_500);
+ }
+ return false;
+ });
+ return () => {
+ cancelPending();
+ selectionListener.dispose();
+ renderListener.dispose();
+ removeKeyHandler();
+ };
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/terminalRepaint.test.ts b/jarvis/ui/web/frontend/src/components/agentic/terminalRepaint.test.ts
new file mode 100644
index 0000000000..9020104109
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/terminalRepaint.test.ts
@@ -0,0 +1,17 @@
+import { describe, expect, it } from "vitest";
+import cases from "../../../../../../../tests/fakes/terminal_repaint_cases.json";
+import { FULL_SCREEN_ERASE_SCAN_TAIL, hasFullScreenErase } from "./terminalRepaint";
+
+describe("full terminal repaint acknowledgement", () => {
+ it.each(cases.full)("accepts a full redraw across any chunk boundary: %j", (erase) => {
+ for (let split = 1; split < erase.length; split++) {
+ expect(hasFullScreenErase(erase.slice(0, split))).toBe(false);
+ const tail = erase.slice(0, split).slice(-FULL_SCREEN_ERASE_SCAN_TAIL);
+ expect(hasFullScreenErase(tail + erase.slice(split))).toBe(true);
+ }
+ });
+
+ it.each(cases.partial)("does not release the curtain for a partial erase: %j", (erase) => {
+ expect(hasFullScreenErase(erase)).toBe(false);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/terminalRepaint.ts b/jarvis/ui/web/frontend/src/components/agentic/terminalRepaint.ts
new file mode 100644
index 0000000000..a3254a446f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/terminalRepaint.ts
@@ -0,0 +1,10 @@
+/** Shared protocol cases also test ReplayBuffer's backend acknowledgement. */
+const FULL_SCREEN_ERASE = /\x1b\[2J|\x1b\[(?:0*1?)(?:;0*1?)?[Hf](?:\x1b\[[0-9;:]*m|\x1b\[\?(?:25|2026)[hl])*\x1b\[0*J/;
+
+/** Retain enough of a split cursor-home/erase pair and its color controls. */
+export const FULL_SCREEN_ERASE_SCAN_TAIL = 128;
+
+/** ED 0 clears the whole screen only immediately after a cursor-home. */
+export function hasFullScreenErase(text: string): boolean {
+ return FULL_SCREEN_ERASE.test(text);
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/terminalSynchronizedOutput.test.ts b/jarvis/ui/web/frontend/src/components/agentic/terminalSynchronizedOutput.test.ts
new file mode 100644
index 0000000000..e6c892d5ee
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/terminalSynchronizedOutput.test.ts
@@ -0,0 +1,148 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { Terminal } from "@xterm/xterm";
+import { installSynchronizedOutput, SYNCHRONIZED_OUTPUT_TIMEOUT_MS } from "./terminalSynchronizedOutput";
+
+function harness() {
+ const csi = new Map<string, (params: number[]) => boolean>();
+ let reset = () => false;
+ const render = vi.fn();
+ const refresh = vi.fn();
+ const service = { _renderRows: render };
+ const term = {
+ rows: 24, refresh, _core: { _renderService: service },
+ parser: {
+ registerCsiHandler(id: { final: string }, callback: (params: number[]) => boolean) {
+ csi.set(id.final, callback);
+ return { dispose: () => csi.delete(id.final) };
+ },
+ registerEscHandler(_id: unknown, callback: () => boolean) {
+ reset = callback;
+ return { dispose: () => { reset = () => false; } };
+ },
+ },
+ } as unknown as Terminal;
+ const bridge = installSynchronizedOutput(term);
+ return { bridge, render, refresh, service, csi, reset: () => reset() };
+}
+
+afterEach(() => { vi.useRealTimers(); vi.restoreAllMocks(); });
+
+describe("synchronized terminal output", () => {
+ it("keeps the last frame while several output chunks build the next one", () => {
+ const h = harness();
+ h.service._renderRows(0, 23);
+ h.render.mockClear();
+ expect(h.csi.get("h")!([2026, 25])).toBe(false);
+ h.service._renderRows(0, 12);
+ h.service._renderRows(13, 23);
+ expect(h.render).not.toHaveBeenCalled();
+ expect(h.csi.get("l")!([2026, 25])).toBe(false);
+ expect(h.refresh).toHaveBeenCalledExactlyOnceWith(0, 23);
+ h.service._renderRows(0, 23);
+ expect(h.render).toHaveBeenCalledExactlyOnceWith(0, 23);
+ h.bridge.dispose();
+ });
+
+ it("leaves ordinary output and unrelated private modes unchanged", () => {
+ const h = harness();
+ h.csi.get("h")!([25]);
+ h.service._renderRows(3, 7);
+ h.csi.get("l")!([2026]);
+ expect(h.render).toHaveBeenCalledExactlyOnceWith(3, 7);
+ expect(h.refresh).not.toHaveBeenCalled();
+ h.bridge.dispose();
+ });
+
+ it("bounds a missing end marker even when begin is repeated", () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.csi.get("h")!([2026]);
+ vi.advanceTimersByTime(SYNCHRONIZED_OUTPUT_TIMEOUT_MS - 1);
+ h.csi.get("h")!([2026]);
+ h.service._renderRows(0, 23);
+ expect(h.render).not.toHaveBeenCalled();
+ vi.advanceTimersByTime(1);
+ h.service._renderRows(0, 23);
+ expect(h.render).toHaveBeenCalledOnce();
+ expect(h.refresh).toHaveBeenCalledOnce();
+ h.bridge.dispose();
+ });
+
+ it("starts a fresh deadline for the next frame and releases on reset", () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.csi.get("h")!([2026]);
+ vi.advanceTimersByTime(900);
+ h.csi.get("l")!([2026]);
+ h.csi.get("h")!([2026]);
+ vi.advanceTimersByTime(101);
+ h.service._renderRows(0, 23);
+ expect(h.render).not.toHaveBeenCalled();
+ h.bridge.reset();
+ h.service._renderRows(0, 23);
+ expect(h.render).toHaveBeenCalledOnce();
+ h.csi.get("h")!([2026]);
+ expect(h.reset()).toBe(false);
+ h.service._renderRows(0, 23);
+ expect(h.render).toHaveBeenCalledTimes(2);
+ h.bridge.dispose();
+ });
+
+ it("restores rendering and removes timers and parser handlers on disposal", () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.csi.get("h")!([2026]);
+ h.bridge.dispose();
+ h.bridge.dispose();
+ expect(h.service._renderRows).toBe(h.render);
+ expect(h.csi.size).toBe(0);
+ expect(vi.getTimerCount()).toBe(0);
+ expect(h.refresh).not.toHaveBeenCalled();
+ });
+
+ it("degrades to ordinary rendering when xterm's private seam is unavailable", () => {
+ const bridge = installSynchronizedOutput({} as Terminal);
+ expect(() => { bridge.reset(); bridge.dispose(); }).not.toThrow();
+ });
+
+ it("honors Codex frame markers split across writes through the real xterm parser", async () => {
+ vi.spyOn(HTMLCanvasElement.prototype, "getContext").mockReturnValue(null);
+ vi.stubGlobal("matchMedia", () => ({ matches: false, addListener() {}, removeListener() {} }));
+ const host = document.createElement("div");
+ document.body.append(host);
+ const term = new Terminal({ cols: 80, rows: 24, allowProposedApi: true });
+ term.open(host);
+ const service = (term as unknown as { _core: { _renderService: { _renderRows(start: number, end: number): void } } })._core._renderService;
+ const render = vi.spyOn(service, "_renderRows");
+ let bridge: ReturnType<typeof installSynchronizedOutput> | undefined;
+ const write = (text: string) => new Promise<void>((resolve) => term.write(text, resolve));
+ try {
+ // The installed xterm ignores the frame marker without the bridge:
+ // the same split repaint can present its incomplete first half.
+ await write("\x1b[?2026h\x1b[1;1H\x1b[Jbaseline partial");
+ service._renderRows(0, 23);
+ expect(render).toHaveBeenCalled();
+ await write("\x1b[?2026l");
+ render.mockClear();
+ bridge = installSynchronizedOutput(term);
+ await write("\x1b[?2026h\x1b[?25l\x1b[1;1H\x1b[Jpartial");
+ // Discard a baseline repaint that ran before the async parser consumed
+ // the new begin marker; every presentation from this point must wait.
+ render.mockClear();
+ service._renderRows(0, 23);
+ expect(render).not.toHaveBeenCalled();
+ await write(" frame\x1b[?20");
+ service._renderRows(0, 23);
+ expect(render).not.toHaveBeenCalled();
+ await write("26l\x1b[?25h");
+ service._renderRows(0, 23);
+ expect(render).toHaveBeenCalled();
+ expect(term.buffer.active.getLine(0)?.translateToString(true)).toBe("partial frame");
+ } finally {
+ bridge?.dispose();
+ term.dispose();
+ host.remove();
+ vi.unstubAllGlobals();
+ }
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/terminalSynchronizedOutput.ts b/jarvis/ui/web/frontend/src/components/agentic/terminalSynchronizedOutput.ts
new file mode 100644
index 0000000000..2624c887e6
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/terminalSynchronizedOutput.ts
@@ -0,0 +1,85 @@
+import type { Terminal } from "@xterm/xterm";
+
+/** A missing end marker must not leave a terminal frozen. */
+export const SYNCHRONIZED_OUTPUT_TIMEOUT_MS = 1_000;
+
+interface RenderService {
+ _renderRows(start: number, end: number): void;
+}
+
+export interface SynchronizedOutput {
+ reset(): void;
+ dispose(): void;
+}
+
+/**
+ * Honor DEC synchronized output (CSI ? 2026 h/l) on xterm 5.5.
+ *
+ * Codex splits one repaint across several PTY/WebSocket chunks. Without this
+ * mode, xterm paints the intermediate erase and cursor moves before the rest
+ * of the frame arrives. Parsing must continue; only presentation waits.
+ *
+ * RenderService is shared by the DOM, canvas and WebGL renderers, including
+ * renderer replacements after context loss. Its debouncer calls _renderRows
+ * through the service, so this also holds refreshes queued before the marker.
+ * Keep the previous frame visible instead of hiding or clearing the canvas.
+ */
+export function installSynchronizedOutput(term: Terminal): SynchronizedOutput {
+ const service = (term as unknown as {
+ _core?: { _renderService?: RenderService };
+ })._core?._renderService;
+ if (typeof service?._renderRows !== "function") {
+ // Unknown xterm internals: ordinary rendering remains available.
+ return { reset() {}, dispose() {} };
+ }
+
+ const original = service._renderRows;
+ let holding = false;
+ let disposed = false;
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const release = () => {
+ if (timer !== undefined) clearTimeout(timer);
+ timer = undefined;
+ if (!holding) return;
+ holding = false;
+ if (!disposed) term.refresh(0, Math.max(0, term.rows - 1));
+ };
+ const begin = () => {
+ if (disposed || holding) return;
+ holding = true;
+ // Repeated set-mode commands are not nested frames and must not extend
+ // the deadline indefinitely when a program forgets to send its end marker.
+ timer = setTimeout(release, SYNCHRONIZED_OUTPUT_TIMEOUT_MS);
+ };
+ const renderRows = function (this: RenderService, start: number, end: number) {
+ if (!holding) original.call(this, start, end);
+ };
+ service._renderRows = renderRows;
+
+ const handlers = [
+ term.parser.registerCsiHandler({ prefix: "?", final: "h" }, (params) => {
+ if (params.includes(2026)) begin();
+ // Let xterm apply other modes combined with this one, e.g. cursor mode.
+ return false;
+ }),
+ term.parser.registerCsiHandler({ prefix: "?", final: "l" }, (params) => {
+ if (params.includes(2026)) release();
+ return false;
+ }),
+ term.parser.registerEscHandler({ final: "c" }, () => {
+ release();
+ return false;
+ }),
+ ];
+
+ return {
+ reset: release,
+ dispose() {
+ if (disposed) return;
+ disposed = true;
+ release();
+ for (const handler of handlers) handler.dispose();
+ if (service._renderRows === renderRows) service._renderRows = original;
+ },
+ };
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/usePaneContextMenu.ts b/jarvis/ui/web/frontend/src/components/agentic/usePaneContextMenu.ts
new file mode 100644
index 0000000000..f04f148e5e
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/usePaneContextMenu.ts
@@ -0,0 +1,34 @@
+import { useState, type MouseEvent } from "react";
+
+export interface PaneMenuRequest { left: number; top: number }
+
+/** Intercept before xterm can consume a right-click or send it to the PTY. */
+export function usePaneContextMenu(enabled: boolean) {
+ const [request, setRequest] = useState<PaneMenuRequest | null>(null);
+ const [sendRightClicks, setSendRightClicks] = useState(false);
+ const interceptButton = (event: MouseEvent) => {
+ if (enabled && event.button === 2 && (!sendRightClicks || event.shiftKey)) {
+ event.preventDefault();
+ event.stopPropagation();
+ }
+ };
+ return {
+ request,
+ sendRightClicks,
+ toggleSendRightClicks: () => setSendRightClicks((current) => !current),
+ handlers: {
+ onPointerDownCapture: interceptButton,
+ onPointerUpCapture: interceptButton,
+ onMouseDownCapture: interceptButton,
+ onMouseUpCapture: interceptButton,
+ onContextMenuCapture: (event: MouseEvent) => {
+ if (!enabled) return;
+ event.preventDefault();
+ event.stopPropagation();
+ if (!sendRightClicks || event.shiftKey) {
+ setRequest({ left: event.clientX, top: event.clientY });
+ }
+ },
+ },
+ };
+}
diff --git a/jarvis/ui/web/frontend/src/components/layout/MainView.prefetch.test.tsx b/jarvis/ui/web/frontend/src/components/layout/MainView.prefetch.test.tsx
new file mode 100644
index 0000000000..c409b8e916
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/layout/MainView.prefetch.test.tsx
@@ -0,0 +1,98 @@
+import { act, cleanup, render, screen } from "@testing-library/react";
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { afterEach, expect, test, vi } from "vitest";
+
+import { MainView } from "./MainView";
+import { useEventStore } from "@/store/events";
+
+const loads = vi.hoisted(() => [] as string[]);
+vi.mock("@/views/ChatsSurface", () => ({ ChatsSurface: () => <div>home</div> }));
+vi.mock("@/views/SettingsHubView", () => {
+ loads.push("settings");
+ return { SettingsHubDialog: () => null };
+});
+vi.mock("@/views/society/SocietyView", () => {
+ loads.push("agents");
+ return { SocietyView: () => <div>agents ready</div> };
+});
+vi.mock("@/views/WikiView", () => {
+ loads.push("memory");
+ return { WikiView: () => null };
+});
+vi.mock("@/views/PluginsDialog", () => {
+ loads.push("plugins");
+ return { PluginsDialog: () => null };
+});
+vi.mock("@/views/AgenticIdeView", () => {
+ loads.push("ide");
+ return { AgenticIdeView: () => null };
+});
+vi.mock("@/views/AutomationsView", () => {
+ loads.push("tasks");
+ return { AutomationsView: () => null };
+});
+vi.mock("@/views/SessionsView", () => {
+ loads.push("sessions");
+ return { SessionsView: () => null };
+});
+vi.mock("@/views/ClisHubView", () => {
+ loads.push("clis");
+ return { ClisHubView: () => null };
+});
+vi.mock("@/views/DocsView", () => {
+ loads.push("docs");
+ return { DocsView: () => null };
+});
+vi.mock("@/views/BoardView", () => {
+ loads.push("board");
+ return { BoardView: () => null };
+});
+vi.mock("@/views/RunInspectorView", () => {
+ loads.push("runs");
+ return { RunInspectorView: () => null };
+});
+vi.mock("@/views/VoiceHubView", () => {
+ loads.push("voice");
+ return { VoiceHubView: () => null };
+});
+vi.mock("@/views/VisualizationView", () => {
+ loads.push("artifacts");
+ return { VisualizationView: () => null };
+});
+vi.mock("@/views/MarketplaceView", () => {
+ loads.push("marketplace");
+ return { MarketplaceView: () => null };
+});
+
+afterEach(() => {
+ cleanup();
+ vi.useRealTimers();
+});
+
+test("an idle home window loads no unused sections or speculative data; navigation still loads its view", async () => {
+ vi.useFakeTimers();
+ useEventStore.setState({ activeSection: "chats", solo: false, detachedViews: [] });
+ const client = new QueryClient();
+ const prefetch = vi.spyOn(client, "prefetchQuery").mockResolvedValue(undefined);
+ render(<QueryClientProvider client={client}><MainView /></QueryClientProvider>);
+
+ // Give both idle-callback and timer implementations enough slots to drain
+ // the old blanket warm-up, without measuring wall time on a busy host.
+ for (let i = 0; i < 15; i += 1) {
+ await act(async () => {
+ await vi.advanceTimersByTimeAsync(4_000);
+ await vi.dynamicImportSettled();
+ });
+ }
+ expect(loads).toEqual([]);
+ expect(prefetch).not.toHaveBeenCalled();
+ expect(screen.getByText("home")).toBeDefined();
+
+ await act(async () => {
+ useEventStore.getState().setActiveSection("agents");
+ await vi.dynamicImportSettled();
+ });
+ expect(loads).toEqual(["agents"]);
+ expect(screen.getByText("agents ready")).toBeDefined();
+ client.clear();
+}, 30_000);
diff --git a/jarvis/ui/web/frontend/src/components/layout/MainView.tsx b/jarvis/ui/web/frontend/src/components/layout/MainView.tsx
index 4db2654f87..135c0e0906 100644
--- a/jarvis/ui/web/frontend/src/components/layout/MainView.tsx
+++ b/jarvis/ui/web/frontend/src/components/layout/MainView.tsx
@@ -1,22 +1,14 @@
import {
lazy,
Suspense,
- useContext,
useEffect,
useState,
type ComponentType,
type LazyExoticComponent,
} from "react";
-import { QueryClientContext } from "@tanstack/react-query";
import { Loader2 } from "lucide-react";
-import {
- costDailyQueryOptions,
- costSummaryQueryOptions,
- EMPTY_FILTERS,
-} from "@/hooks/useCosts";
-import { overviewQueryOptions } from "@/hooks/useLocalModels";
-import { readLocalModelsSeed } from "@/lib/localModelsSeed";
-import { useEventStore } from "@/store/events";
+import { useEventStore, type SectionId } from "@/store/events";
+import { sectionPrefetch } from "@/lib/sectionPrefetch";
import { useT } from "@/i18n";
import { cn } from "@/lib/utils";
import { ViewErrorBoundary } from "@/components/ViewErrorBoundary";
@@ -32,33 +24,18 @@ import type { AgenticIdeViewProps } from "@/views/AgenticIdeView";
import { ChatsSurface } from "@/views/ChatsSurface";
/**
- * Section views are CODE-SPLIT, then prefetched while the app is idle.
- *
- * Why (measured 2026-07-26): importing all 21 views statically linked every
- * section — plus the terminal emulator and the charting library — into a single
- * 2.83 MB entry chunk. A WebView must download, parse AND execute all of it
- * before React paints, which is the "the app feels sluggish on start" report.
- * The boot splash in index.html only hid that wait.
- *
- * Splitting alone would move the cost rather than remove it: the first visit to
- * a section would then pay for its chunk. `useIdleViewPrefetch` closes that gap
- * by pulling the remaining chunks in during idle time after the first paint, so
- * switching sections stays instant while startup only pays for what it shows.
+ * Keep unused views out of every window's startup work. Navigation loads its
+ * view through React.lazy; a deliberate sidebar hover/focus warms only that
+ * destination. Blanket idle imports used to execute all 14 views per window.
*/
type ViewModule = { default: ComponentType };
type ViewLoader = () => Promise<ViewModule>;
-/**
- * Loaders for every split view, in the order the idle prefetch warms them.
- *
- * Typed as returning `unknown` because warming a chunk only cares that the
- * import RAN — the module's shape is the caller's business, and pinning the
- * props-free shape here would keep `lazyPropView` out of the warm-up.
- */
-const prefetchQueue: (() => Promise<unknown>)[] = [];
-
-function lazyView(loader: ViewLoader): LazyExoticComponent<ComponentType> {
- prefetchQueue.push(loader);
+function lazyView(
+ sections: readonly SectionId[],
+ loader: ViewLoader,
+): LazyExoticComponent<ComponentType> {
+ sectionPrefetch.register(sections, loader);
return lazy(loader);
}
@@ -72,32 +49,31 @@ function lazyView(loader: ViewLoader): LazyExoticComponent<ComponentType> {
* answer for it (see `MainView`).
*/
function lazyPropView<P>(
+ sections: readonly SectionId[],
loader: () => Promise<{ default: ComponentType<P> }>,
): LazyExoticComponent<ComponentType<P>> {
- prefetchQueue.push(loader);
+ sectionPrefetch.register(sections, loader);
return lazy(loader);
}
-// Ordered roughly by how likely a section is to be opened, so the warm-up
-// front-loads what the user reaches for first. Views are named exports, hence
-// the explicit unwrap into the { default } shape React.lazy expects.
+// Views are named exports, hence the explicit React.lazy default unwrap.
// The Settings hub — Profile, {name}.md, Contacts, Socials, API Keys, Local
// models, Spend and Feedback behind one left-nav dialog. The hub
// statically owns only its shell; every tab stays its own lazy chunk (see
// SettingsHubView), so this one import replaces the eleven per-view imports
// below without merging their chunks back together.
-const SettingsHubDialog = lazyPropView<{ onClose: () => void }>(() =>
+const SettingsHubDialog = lazyPropView<{ onClose: () => void }>(SETTINGS_HUB_IDS, () =>
import("@/views/SettingsHubView").then((m) => ({ default: m.SettingsHubDialog })),
);
// The Agents section is the society (MASTERPLAN §4.1): stage + agents rail +
// model cards. The board it replaced is the society's stage until the island
// lands, loaded by SocietyView itself.
-const SocietyView = lazyView(() =>
+const SocietyView = lazyView(["agents"], () =>
import("@/views/society/SocietyView").then((m) => ({
default: m.SocietyView,
})),
);
-const WikiView = lazyView(() =>
+const WikiView = lazyView(["memory"], () =>
import("@/views/WikiView").then((m) => ({ default: m.WikiView })),
);
type PluginArea = "plugins" | "mcps" | "skills";
@@ -109,31 +85,31 @@ const isSettingsHub = (section: string) =>
// replacing it.
const isOverlaySection = (section: string) => isPluginArea(section) || isSettingsHub(section);
-const PluginsDialog = lazyPropView<{ onClose: () => void; area: PluginArea; onAreaChange: (area: PluginArea) => void }>(() =>
+const PluginsDialog = lazyPropView<{ onClose: () => void; area: PluginArea; onAreaChange: (area: PluginArea) => void }>(["plugins", "mcps", "skills"], () =>
import("@/views/PluginsDialog").then((m) => ({ default: m.PluginsDialog })),
);
// The prop type is named rather than inferred: inferring it from the loader's
// return value is circular (the loader's contextual type is what depends on it),
// and TypeScript resolves that by falling back to `never`.
-const AgenticIdeView = lazyPropView<AgenticIdeViewProps>(() =>
+const AgenticIdeView = lazyPropView<AgenticIdeViewProps>(["agentic-ide", "agentic-ide-classic", "chat-workspace"], () =>
import("@/views/AgenticIdeView").then((m) => ({ default: m.AgenticIdeView })),
);
-const AutomationsView = lazyView(() =>
+const AutomationsView = lazyView(["tasks"], () =>
import("@/views/AutomationsView").then((m) => ({ default: m.AutomationsView })),
);
-const SessionsView = lazyView(() =>
+const SessionsView = lazyView(["sessions"], () =>
import("@/views/SessionsView").then((m) => ({ default: m.SessionsView })),
);
-const ClisHubView = lazyView(() =>
+const ClisHubView = lazyView(["clis", "cli-test-hub"], () =>
import("@/views/ClisHubView").then((m) => ({ default: m.ClisHubView })),
);
-const DocsView = lazyView(() =>
+const DocsView = lazyView(["docs"], () =>
import("@/views/DocsView").then((m) => ({ default: m.DocsView })),
);
-const BoardView = lazyView(() =>
+const BoardView = lazyView(["board"], () =>
import("@/views/BoardView").then((m) => ({ default: m.BoardView })),
);
-const RunInspectorView = lazyView(() =>
+const RunInspectorView = lazyView(["run_inspector"], () =>
import("@/views/RunInspectorView").then((m) => ({
default: m.RunInspectorView,
})),
@@ -142,123 +118,20 @@ const RunInspectorView = lazyView(() =>
// behind the one "{name} Voice" sidebar entry. Only the hub is split out here —
// it statically imports its five tabs, so they travel in its chunk instead of
// being prefetched as separate ones.
-const VoiceHubView = lazyView(() =>
+const VoiceHubView = lazyView(["dictation", "dictionary", "voice-shortcuts", "voice-language", "voice-api-keys"], () =>
import("@/views/VoiceHubView").then((m) => ({ default: m.VoiceHubView })),
);
-const VisualizationView = lazyView(() =>
+const VisualizationView = lazyView(["visualization"], () =>
import("@/views/VisualizationView").then((m) => ({
default: m.VisualizationView,
})),
);
-const MarketplaceView = lazyView(() =>
+const MarketplaceView = lazyView(["marketplace"], () =>
import("@/views/MarketplaceView").then((m) => ({
default: m.MarketplaceView,
})),
);
-type IdleWindow = Window & {
- requestIdleCallback?: (
- cb: () => void,
- opts?: { timeout: number },
- ) => number;
- cancelIdleCallback?: (handle: number) => void;
-};
-
-/**
- * Run `task` when the browser is idle, falling back to a timer.
- *
- * `requestIdleCallback` is absent on older Safari/WebKit, which is exactly the
- * macOS WebView this app also ships in, so the timer fallback is load-bearing
- * rather than cosmetic. The returned function cancels a pending slot.
- */
-function scheduleIdle(task: () => void): () => void {
- const w = window as IdleWindow;
- if (typeof w.requestIdleCallback === "function") {
- // The timeout caps how long a busy main thread may starve the warm-up.
- const handle = w.requestIdleCallback(task, { timeout: 2_000 });
- return () => w.cancelIdleCallback?.(handle);
- }
- const handle = window.setTimeout(task, 300);
- return () => window.clearTimeout(handle);
-}
-
-/**
- * Warm the split view chunks one at a time, each in its own idle slot.
- *
- * Sequential rather than parallel on purpose: firing 20 imports at once would
- * compete with the requests the visible section is making, which is the very
- * stall this is meant to remove. A failed prefetch is ignored — the chunk is
- * simply fetched again on navigation, where Suspense handles it.
- */
-/**
- * How long after mount the Spend warm-up runs. Past the boot burst — this app
- * makes about thirty API calls before it settles — and well inside the time it
- * takes anyone to read the home screen and reach for the sidebar.
- */
-const SPEND_WARM_DELAY_MS = 2_500;
-
-function useIdleViewPrefetch(): void {
- // Optional on purpose: the shell renders without a QueryClient in some
- // tests, and the data warm-up is a bonus, never a requirement.
- const queryClient = useContext(QueryClientContext);
- useEffect(() => {
- let cancelled = false;
- let cancelSlot: (() => void) | null = null;
- let index = 0;
- const queue = [...prefetchQueue];
- if (queryClient) {
- // After the chunks: the Local models overview, when a previous open
- // left a seed — so the section paints with server truth, not just the
- // on-disk snapshot, the moment it is opened.
- queue.push(() => {
- const seed = readLocalModelsSeed();
- return seed
- ? queryClient.prefetchQuery(overviewQueryOptions(seed))
- : Promise.resolve();
- });
- }
-
- const pump = () => {
- if (cancelled || index >= queue.length) return;
- const loader = queue[index++];
- void loader()
- .catch(() => undefined)
- .then(() => {
- if (cancelled) return;
- cancelSlot = scheduleIdle(pump);
- });
- };
-
- cancelSlot = scheduleIdle(pump);
-
- // Spend, on its own clock rather than at the end of that queue.
- //
- // It is the one section whose first read is measured in a second rather
- // than milliseconds — it aggregates every store the app writes plus the
- // coding-CLI transcripts — so it is the one where arriving to a finished
- // page instead of a loading one is the whole difference. Queued behind
- // twenty module imports, each waiting for its own idle slot, it had not
- // run twenty-five seconds into a fresh bundle. A plain timer is late
- // enough to stay out of the boot burst and early enough to beat a click.
- // The section opens on `EMPTY_FILTERS`, so these are the keys it asks for.
- const warmSpend = window.setTimeout(() => {
- if (cancelled || !queryClient) return;
- void queryClient
- .prefetchQuery(costSummaryQueryOptions(EMPTY_FILTERS))
- .catch(() => undefined);
- void queryClient
- .prefetchQuery(costDailyQueryOptions(EMPTY_FILTERS))
- .catch(() => undefined);
- }, SPEND_WARM_DELAY_MS);
-
- return () => {
- cancelled = true;
- cancelSlot?.();
- window.clearTimeout(warmSpend);
- };
- }, [queryClient]);
-}
-
/** Quiet time before a loading section admits it is loading. */
export const LOADING_HINT_MS = 400;
@@ -391,8 +264,6 @@ export function MainView() {
if (!isOverlaySection(active) && backgroundSection !== active) setBackgroundSection(active);
const displayed = isOverlaySection(active) ? backgroundSection : active;
- useIdleViewPrefetch();
-
/*
* While a coding view lives in a detached solo window, THIS (non-solo)
* window must not keep a mounted IDE instance around — not even hidden: the
diff --git a/jarvis/ui/web/frontend/src/components/layout/Sidebar.test.tsx b/jarvis/ui/web/frontend/src/components/layout/Sidebar.test.tsx
index 1c7080b343..63980455ce 100644
--- a/jarvis/ui/web/frontend/src/components/layout/Sidebar.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/layout/Sidebar.test.tsx
@@ -13,6 +13,34 @@ import { isSectionId, useEventStore } from "@/store/events";
import { useHomeStore } from "@/store/home";
import { useIdeChatStore } from "@/store/ideChat";
import { useIdeProjectsStore } from "@/store/ideProjects";
+import { sectionPrefetch } from "@/lib/sectionPrefetch";
+import { SECTION_PREFETCH_DELAY_MS } from "@/hooks/useSectionPrefetch";
+
+test.each([SIDEBAR_DEFAULT_WIDTH, SIDEBAR_RAIL_WIDTH])(
+ "sidebar intent warms a destination without navigating (width=%s)",
+ (width) => {
+ vi.useFakeTimers();
+ const warm = vi.spyOn(sectionPrefetch, "prefetch").mockResolvedValue();
+ try {
+ useEventStore.setState({ activeSection: "chats" });
+ renderSidebar(width);
+ expect(warm).not.toHaveBeenCalled();
+ fireEvent.mouseEnter(screen.getByTestId("nav-row-agents"));
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS); });
+ expect(warm).toHaveBeenCalledExactlyOnceWith("agents");
+ expect(useEventStore.getState().activeSection).toBe("chats");
+ fireEvent.focus(screen.getByTestId("sidebar-profile-toggle"));
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS); });
+ expect(warm).toHaveBeenLastCalledWith("profile");
+ fireEvent.click(screen.getByTestId("nav-row-agents"));
+ expect(useEventStore.getState().activeSection).toBe("agents");
+ } finally {
+ cleanup();
+ warm.mockRestore();
+ vi.useRealTimers();
+ }
+ },
+);
test("IDE rail keeps workspace options and Jarvis Live reachable", () => {
act(() => {
diff --git a/jarvis/ui/web/frontend/src/components/layout/Sidebar.tsx b/jarvis/ui/web/frontend/src/components/layout/Sidebar.tsx
index 2b5ff05851..f251a1de8e 100644
--- a/jarvis/ui/web/frontend/src/components/layout/Sidebar.tsx
+++ b/jarvis/ui/web/frontend/src/components/layout/Sidebar.tsx
@@ -17,6 +17,7 @@ import {
type NavItem,
} from "@/components/layout/navGroups";
import { useEventStore } from "@/store/events";
+import { useSectionPrefetch } from "@/hooks/useSectionPrefetch";
import { useVoiceReadiness } from "@/hooks/useVoiceReadiness";
import { useVoiceMode } from "@/hooks/useVoiceMode";
import { useSectionHealth } from "@/hooks/useProviders";
@@ -167,6 +168,8 @@ export function Sidebar({
collapsed = false,
}: SidebarProps = {}) {
const t = useT();
+ const profilePrefetch = useSectionPrefetch("profile");
+ const marketplacePrefetch = useSectionPrefetch("marketplace");
const active = useEventStore((s) => s.activeSection);
const setActive = useEventStore((s) => s.setActiveSection);
const activeIdeWorkspaceId = useIdeProjectsStore((s) => s.activeWorkspaceId);
@@ -548,7 +551,7 @@ export function Sidebar({
needs care, must be visible without opening anything. */}
<div className="shrink-0 border-t border-border p-2">
<div className={cn("flex items-center gap-1", railed && "flex-col")}>
- <button type="button" onClick={() => setActive("profile")} title={t("nav.profile")}
+ <button type="button" {...profilePrefetch} onClick={() => setActive("profile")} title={t("nav.profile")}
data-testid="sidebar-profile-toggle"
data-tour="settings"
className={cn(rowClass, "min-w-0 flex-1", hubActive && "jarvis-nav-active bg-secondary text-foreground")}>
@@ -560,7 +563,7 @@ export function Sidebar({
</span>
{!railed && <span className="min-w-0 flex-1 truncate text-left">{identity.data?.signed_in ? identity.data.login || t("nav.profile") : t("nav.profile")}</span>}
</button>
- <button type="button" onClick={() => setActive("marketplace")} title={t("nav.marketplace")}
+ <button type="button" {...marketplacePrefetch} onClick={() => setActive("marketplace")} title={t("nav.marketplace")}
aria-label={t("nav.marketplace")} data-testid="nav-row-marketplace"
className="flex h-9 w-9 shrink-0 items-center justify-center rounded-md text-muted-foreground hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
<Store aria-hidden className="h-5 w-5" />
@@ -612,6 +615,7 @@ function NavRow({
warnTitle?: string;
onClick: () => void;
}) {
+ const prefetch = useSectionPrefetch(item.id);
const Icon = item.icon;
const hint = alert ? alertTitle : warn ? warnTitle : undefined;
/*
@@ -640,6 +644,7 @@ function NavRow({
<button
type="button"
data-testid={`nav-row-${item.id}`}
+ {...prefetch}
data-tour={`nav-${item.id}`}
onClick={onClick}
title={compact ? `${label}${hint ? ` — ${hint}` : ""}` : hint}
diff --git a/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.test.tsx b/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.test.tsx
index 101abb60d0..61d8f867e5 100644
--- a/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.test.tsx
@@ -1145,6 +1145,40 @@ describe("managed local realtime model setup", () => {
expect(screen.getByText("MOSS-TTS-Nano 100M")).toBeTruthy();
expect(screen.getByText("Integration pending")).toBeTruthy();
});
+
+ it.each([true, false])("activates local voice only after a successful speech test (%s)", async (success) => {
+ const calls: string[] = [];
+ vi.stubGlobal("fetch", vi.fn(async (input: RequestInfo | URL) => {
+ const url = String(input);
+ calls.push(url);
+ const body = url.endsWith("/model-catalog") ? {
+ brain: { current: "voice-brain", models: [{ id: "voice-brain", label: "Voice brain", installed: true, fits: true, size_gb: 3 }] },
+ current: "voice-tts", models: [{ id: "voice-tts", label: "Voice TTS", selectable: true, runtime_ready: true, languages: ["en"] }],
+ hearing: { label: "Local recognition" },
+ } : url.endsWith("/setup") ? (success ? { smoke: { first_audio_ms: 100 } } : { detail: "Speech test failed" })
+ : url.endsWith("/realtime/switch") ? { ok: true, restart_required: false }
+ : {};
+ return { ok: success || !url.endsWith("/setup"), status: success ? 200 : 409,
+ json: async () => body, text: async () => JSON.stringify(body) } as Response;
+ }));
+ const card = localRealtimeCard();
+ card.active = false;
+ card.experimental = true;
+ card.managed_server!.ready = true;
+ renderCard(card);
+ fireEvent.click(await screen.findByRole("button", { name: "Apply, test & use local voice" }));
+ await waitFor(() => expect(calls.some((url) => url.endsWith("/setup"))).toBe(true));
+ if (success) {
+ await waitFor(() => expect(calls).toContain("/api/realtime/switch"));
+ await waitFor(() => expect(calls).toContain("/api/settings/voice-mode"));
+ expect(calls.findIndex((url) => url.endsWith("/setup"))).toBeLessThan(calls.indexOf("/api/realtime/switch"));
+ } else {
+ await screen.findByText("Speech test failed");
+ expect(calls).not.toContain("/api/realtime/switch");
+ expect(calls).not.toContain("/api/settings/voice-mode");
+ expect(screen.queryByText(enLocale.apikeys_view.managed_setup_testing)).toBeNull();
+ }
+ });
});
describe("managed server badge", () => {
diff --git a/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.tsx b/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.tsx
index eac63a0a5a..6a71fcf294 100644
--- a/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.tsx
+++ b/jarvis/ui/web/frontend/src/components/providers/ProviderTierSection.tsx
@@ -5,6 +5,7 @@ import { ApiKeyForm } from "@/components/ApiKeyForm";
import { BrainModelSelector } from "@/components/BrainModelSelector";
import { OpenRouterTtsControls } from "@/components/OpenRouterTtsVoicePicker";
import { RealtimeOptionsControl } from "@/components/RealtimeOptionsControl";
+import { putVoiceMode } from "@/lib/voiceEngineMode";
import { ProviderLogo } from "@/components/providers/ProviderLogo";
import { useRowGestures } from "@/components/providers/rowGestures";
import { Button } from "@/components/ui/button";
@@ -1593,10 +1594,18 @@ export function ProviderCard({
// probe so the card does not visibly flicker while saying
// "one moment".
descriptor.codex_status?.reason_code === "busy") && (
- <RealtimeOptionsControl
- providerId={descriptor.id}
- healthActive={descriptor.active}
- />
+ descriptor.managed_server ? (
+ <details className="text-xs text-muted-foreground">
+ <summary className="cursor-pointer">
+ {t("apikeys_view.managed_protocol_options")}
+ </summary>
+ <div className="pt-3">
+ <RealtimeOptionsControl providerId={descriptor.id} healthActive={descriptor.active} />
+ </div>
+ </details>
+ ) : (
+ <RealtimeOptionsControl providerId={descriptor.id} healthActive={descriptor.active} />
+ )
)}
{/* Footer: the live connectivity test, visually separated from the
@@ -2228,21 +2237,28 @@ function ManagedServerPanel({
useEffect(() => {
if (running || !installed) return;
let cancelled = false;
+ let timer: number | undefined;
const read = async () => {
+ let delay = 20_000;
try {
const next = await managedServerStatus();
if (!cancelled) setRuntime(next.runtime ?? null);
+ if (next.runtime?.boot?.starting || next.runtime?.ready === false || setupBusy || lifecycleBusy) {
+ delay = 1500;
+ }
} catch {
if (!cancelled) setRuntime(null);
+ delay = 5000;
+ } finally {
+ if (!cancelled) timer = window.setTimeout(read, delay + Math.random() * 500);
}
};
void read();
- const timer = window.setInterval(read, 20_000);
return () => {
cancelled = true;
- window.clearInterval(timer);
+ window.clearTimeout(timer);
};
- }, [running, installed]);
+ }, [running, installed, setupBusy, lifecycleBusy]);
if (!status) return null;
@@ -2364,7 +2380,7 @@ function ManagedServerPanel({
) => {
if (!brainModel || !voiceModel) return;
setError(null);
- setSetupNote(null);
+ setSetupNote(t("apikeys_view.managed_setup_testing"));
setSetupBusy(true);
try {
const choice = catalog?.brain.models.find((item) => item.id === brainModel);
@@ -2374,8 +2390,14 @@ function ManagedServerPanel({
await downloadBrain(brainModel);
}
const result = await managedServerSetup(brainModel, voiceModel);
+ if (!descriptor.active) {
+ await switchRealtimeProvider(descriptor.id, descriptor.experimental === true);
+ }
+ await putVoiceMode("realtime");
+ window.dispatchEvent(new CustomEvent("jarvis:realtime-switched"));
const latency = result.smoke.first_audio_ms;
setSelectedBrain(brainModel);
+ setSelectedVoice(voiceModel);
setSetupNote(
latency === null
? t("apikeys_view.managed_setup_complete")
@@ -2387,6 +2409,7 @@ function ManagedServerPanel({
await loadCatalog();
onChanged();
} catch (err) {
+ setSetupNote(null);
setError(err instanceof Error ? err.message : String(err));
} finally {
setSetupBusy(false);
@@ -2573,6 +2596,9 @@ function ManagedServerPanel({
{!running && catalog && (
<div className="space-y-3" data-testid="managed-model-picker">
+ <p className="text-xs text-muted-foreground" role="status">
+ {t(descriptor.active ? "apikeys_view.managed_selected" : "apikeys_view.managed_not_selected")}
+ </p>
<div className="grid gap-2 sm:grid-cols-3">
<label className="space-y-1.5">
<span className="flex items-center gap-1.5 text-xs font-medium">
diff --git a/jarvis/ui/web/frontend/src/components/society/card/AgentLimits.test.tsx b/jarvis/ui/web/frontend/src/components/society/card/AgentLimits.test.tsx
index 20216b16f4..847f22ceac 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/AgentLimits.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/AgentLimits.test.tsx
@@ -103,6 +103,20 @@ describe("the card's limits", () => {
});
});
+ test("saves an explicit approval mode while preserving the ceiling", async () => {
+ renderCard({ approvalMode: "bypass" });
+ fireEvent.click(await screen.findByTestId("agent-limits-edit"));
+ fireEvent.click(screen.getByText("Always ask"));
+ fireEvent.click(screen.getByText("Save"));
+
+ await waitFor(() => {
+ const patch = fetchMock.mock.calls.find((call) => call[1]?.method === "PATCH");
+ const body = JSON.parse(String(patch?.[1]?.body));
+ expect(body.approval_mode).toBe("always_ask");
+ expect(body.permission_ceiling).toBe("monitor");
+ });
+ });
+
test("an empty budget means no cap rather than a broken number", async () => {
renderCard();
fireEvent.click(await screen.findByTestId("agent-limits-edit"));
diff --git a/jarvis/ui/web/frontend/src/components/society/card/AgentSpecSheet.tsx b/jarvis/ui/web/frontend/src/components/society/card/AgentSpecSheet.tsx
index fc32fb5689..9ba0786a69 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/AgentSpecSheet.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/AgentSpecSheet.tsx
@@ -31,6 +31,7 @@
* Every string goes through the locale files.
*/
import { useMemo, useState } from "react";
+import { useQuery } from "@tanstack/react-query";
import { MessageSquare, Pause, Play, Server } from "lucide-react";
import { ProviderLogo } from "@/components/providers/ProviderLogo";
@@ -46,7 +47,7 @@ import { AgentRoutinesList } from "./AgentRoutinesList";
import { RetireButton } from "./RetireButton";
import { CapabilityChip } from "../CapabilityChip";
import { useAgentActivity, useAgentSkills, type AgentActivity } from "../cardData";
-import { PERMISSION_CEILINGS } from "@/lib/societyApi";
+import { AGENT_APPROVAL_MODES, PERMISSION_CEILINGS, fetchSocietyProviders, type AgentApprovalMode } from "@/lib/societyApi";
import {
useSetAgentComputer,
@@ -526,6 +527,9 @@ function LimitsSection({
const [saving, setSaving] = useState(false);
const [error, setError] = useState("");
const [ceiling, setCeiling] = useState<PermissionCeiling>(agent.permissionCeiling);
+ const [approvalMode, setApprovalMode] = useState<AgentApprovalMode | null>(agent.approvalMode ?? null);
+ const providerRows = useQuery({ queryKey: ["society", "providers"], queryFn: fetchSocietyProviders });
+ const runner = providerRows.data?.find((row) => row.id === agent.provider)?.runner;
const [budget, setBudget] = useState(String(agent.dailyBudgetUsd));
// The stored 0 IS 'no cap' (the scheduler skips the gate), so the switch
// reads it back rather than asking the person to know that.
@@ -534,6 +538,7 @@ function LimitsSection({
const begin = () => {
setCeiling(agent.permissionCeiling);
+ setApprovalMode(agent.approvalMode ?? null);
setBudget(String(agent.dailyBudgetUsd || 2));
setBudgetOn(agent.dailyBudgetUsd > 0);
setJobs(String(agent.maxConcurrentRuns));
@@ -546,6 +551,7 @@ function LimitsSection({
try {
await update(agent, {
permissionCeiling: ceiling,
+ approvalMode,
dailyBudgetUsd: budgetOn ? Math.max(0, Number.parseFloat(budget) || 0) : 0,
maxConcurrentRuns: Number.parseInt(jobs, 10) || 1,
});
@@ -575,8 +581,37 @@ function LimitsSection({
{editing ? (
<div className="ac-prose flex flex-col gap-3">
- <div className="flex flex-col gap-1">
- <span className="text-xs text-muted-foreground">{t("society.card.permission")}</span>
+ {agent.tier !== "lead" && <div className="flex flex-col gap-1">
+ <span className="text-xs text-muted-foreground">{t("society.approval_mode.title")}</span>
+ <div className="flex flex-wrap gap-1">
+ {AGENT_APPROVAL_MODES.map((mode) => {
+ const unknownRunner = Boolean(agent.provider) && runner == null;
+ const unavailable = mode === "always_ask"
+ ? unknownRunner || (runner != null && runner !== "codex-cli" && runner !== "brain")
+ : mode === "ask" && (unknownRunner || (runner != null
+ && !["codex-cli", "claude-cli", "glm-cli", "brain"].includes(runner)));
+ return <button
+ key={mode}
+ type="button"
+ disabled={saving || unavailable}
+ title={unavailable ? t("society.approval_mode.unavailable") : t(`society.approval_mode.${mode}_hint`)}
+ onClick={() => setApprovalMode(mode)}
+ aria-pressed={approvalMode === mode}
+ className={cn(
+ "rounded-md border px-2.5 py-1 text-xs transition-colors disabled:opacity-50",
+ approvalMode === mode
+ ? "border-border-strong bg-secondary text-foreground"
+ : "border-border text-muted-foreground hover:bg-secondary",
+ )}
+ >{t(`society.approval_mode.${mode}`)}</button>;
+ })}
+ </div>
+ <p className="text-xs text-muted-foreground">
+ {approvalMode ? t(`society.approval_mode.${approvalMode}_hint`) : t("society.approval_mode.legacy_hint")}
+ </p>
+ </div>}
+ {(agent.tier === "lead" || !agent.approvalMode) && <div className="flex flex-col gap-1">
+ <span className="text-xs text-muted-foreground">{t("society.approval_mode.legacy_ceiling")}</span>
<div className="flex flex-wrap gap-1">
{PERMISSION_CEILINGS.map((step) => (
<button
@@ -596,7 +631,7 @@ function LimitsSection({
</button>
))}
</div>
- </div>
+ </div>}
<div className="flex flex-wrap items-end gap-4">
<div className="flex flex-col gap-1">
@@ -666,12 +701,18 @@ function LimitsSection({
</div>
) : (
<>
- <StepMeter
- label={t("society.card.permission")}
- steps={3}
- filled={CEILING_STEP[agent.permissionCeiling] ?? 0}
- value={t(`society.ceiling.${agent.permissionCeiling}`)}
- />
+ {agent.tier !== "lead" && agent.approvalMode ? (
+ <div className="text-xs text-muted-foreground">
+ {t("society.approval_mode.title")}: <span className="text-foreground">{t(`society.approval_mode.${agent.approvalMode}`)}</span>
+ </div>
+ ) : (
+ <StepMeter
+ label={t("society.approval_mode.legacy_ceiling")}
+ steps={3}
+ filled={CEILING_STEP[agent.permissionCeiling] ?? 0}
+ value={t(`society.ceiling.${agent.permissionCeiling}`)}
+ />
+ )}
<StepMeter
label={t("society.card.meter_reach")}
steps={REACH_STEPS}
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/CodingProjectChoice.tsx b/jarvis/ui/web/frontend/src/components/society/chat/CodingProjectChoice.tsx
index 90a884cc01..5e4e9ce7e2 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/CodingProjectChoice.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/CodingProjectChoice.tsx
@@ -41,13 +41,13 @@ export function CodingProjectChoice({ agents, folder, onFolder }: {
<Dialog.Root open={open} onOpenChange={setOpen}>
<Dialog.Portal>
<Dialog.Overlay className="fixed inset-0 z-[90] bg-[rgb(var(--scrim-rgb)/0.4)]" />
- <Dialog.Content className="fixed left-1/2 top-1/2 z-[91] flex max-h-[85vh] w-[min(680px,calc(100vw-24px))] -translate-x-1/2 -translate-y-1/2 flex-col rounded-xl border border-border bg-popover p-4 text-foreground shadow-float">
+ <Dialog.Content onEscapeKeyDown={(e) => { if (e.target instanceof Element && e.target.closest("[data-escape-local]")) e.preventDefault(); }} className="fixed left-1/2 top-1/2 z-[91] flex max-h-[85vh] w-[min(680px,calc(100vw-24px))] -translate-x-1/2 -translate-y-1/2 flex-col rounded-xl border border-border bg-popover p-4 text-foreground shadow-float">
<Dialog.Title className="pr-8 text-sm font-semibold">{t("society.chat.coding_choose_folder")}</Dialog.Title>
<Dialog.Description className="mt-1 text-xs text-muted-foreground">{t("society.chat.coding_project_optional")}</Dialog.Description>
<Dialog.Close aria-label={t("society.chat.coding_close")} className="absolute right-3 top-3 rounded p-1 hover:bg-secondary"><X className="h-4 w-4" /></Dialog.Close>
<div className="my-3 min-h-0 overflow-auto">
<Suspense fallback={<p className="text-xs text-muted-foreground">{t("society.chat.mention_loading")}</p>}>
- {open && <FolderPicker selected={candidate} onSelect={setCandidate} />}
+ {open && <FolderPicker selected={candidate} onSelect={setCandidate} className="p-0" />}
</Suspense>
</div>
<button type="button" disabled={!candidate} onClick={() => { if (candidate) onFolder(candidate); setOpen(false); }}
diff --git a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx
index eb1f1cc121..83110ee5ce 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx
@@ -46,7 +46,7 @@ import {
type AgentConnectionRow,
type CuratedModel,
} from "@/lib/agentChatApi";
-import { fetchSocietyProviders, type SocietyProviderRow } from "@/lib/societyApi";
+import { AGENT_APPROVAL_MODES, fetchSocietyProviders, type AgentApprovalMode, type SocietyProviderRow } from "@/lib/societyApi";
import { cn } from "@/lib/utils";
import { joinProviderOptions } from "@/store/agentChat";
@@ -60,7 +60,7 @@ import {
type BrainSeat,
} from "./brainPicker";
import { modelSeats } from "../chat/modelChoices";
-import { useCreateAgent, type PermissionCeiling } from "../data";
+import { useCreateAgent } from "../data";
import { CompanionEditor } from "../companion/CompanionEditor";
import { resolveCompanion } from "../companion/appearance";
import { AgentFigureViewer } from "../figures/AgentFigureViewer";
@@ -85,8 +85,6 @@ import {
} from "../figures/figureRegistry";
import type { FigureArchetype } from "../figures/figureRecipe";
-const CEILINGS: readonly PermissionCeiling[] = ["safe", "monitor", "ask"];
-
/**
* How tall a figure may be made, per archetype. One 1.5–2.1 m band fits a
* person and nothing else: a fox at 1.75 m is a horse, and Gigi's own scale
@@ -131,7 +129,7 @@ export function CreateAgentDialog({ open, onClose, onCreated }: CreateAgentDialo
const [accountId, setAccountId] = useState("");
// "" = this computer; otherwise a connected computer (VPS / local VM) id.
const [computerId, setComputerId] = useState("");
- const [ceiling, setCeiling] = useState<PermissionCeiling>("monitor");
+ const [approvalMode, setApprovalMode] = useState<AgentApprovalMode>("bypass");
const [budget, setBudget] = useState("2");
// A cap is the default because an agent that can spend without one is the
// surprising case, not the ordinary one. Off sends 0, which is exactly what
@@ -226,6 +224,7 @@ export function CreateAgentDialog({ open, onClose, onCreated }: CreateAgentDialo
setModel(next ? next.provider.default_model || modelsFor(next)[0]?.id || "" : "");
setEffort(next?.provider.default_effort ?? "");
setAccountId("");
+ setApprovalMode("bypass");
};
// A fresh dialog starts on the brain marked active, else the first seat —
@@ -252,6 +251,7 @@ export function CreateAgentDialog({ open, onClose, onCreated }: CreateAgentDialo
setModel("");
setEffort("");
setAccountId("");
+ setApprovalMode("bypass");
setImportProblems(null);
setImportedName(null);
setError(null);
@@ -381,7 +381,8 @@ export function CreateAgentDialog({ open, onClose, onCreated }: CreateAgentDialo
grantMode: "all",
toolGrants: [],
focus: [],
- permissionCeiling: ceiling,
+ permissionCeiling: "ask",
+ approvalMode,
dailyBudgetUsd: budgetOn ? Math.max(0, Number.parseFloat(budget) || 0) : 0,
});
onCreated(agent.agentId);
@@ -587,7 +588,9 @@ export function CreateAgentDialog({ open, onClose, onCreated }: CreateAgentDialo
type="button"
className="flex w-full items-center justify-between rounded-md border border-border px-3 py-2 text-sm font-medium text-foreground hover:bg-secondary"
>
- {t("society.create.advanced")}
+ <span>
+ {t("society.create.advanced")} · {t(`society.approval_mode.${approvalMode}`)}
+ </span>
<ChevronDown
className={cn("h-4 w-4 transition-transform", advanced && "rotate-180")}
aria-hidden
@@ -596,12 +599,23 @@ export function CreateAgentDialog({ open, onClose, onCreated }: CreateAgentDialo
</Collapsible.Trigger>
<Collapsible.Content className="flex flex-col gap-4 pt-4">
<div>
- <span className={labelClass}>{t("society.create.ceiling")}</span>
+ <span className={labelClass}>{t("society.approval_mode.title")}</span>
<Segmented
- value={ceiling}
- options={CEILINGS.map((c) => ({ value: c, label: t(`society.ceiling.${c}`) }))}
- onChange={(v) => setCeiling(v as PermissionCeiling)}
+ value={approvalMode}
+ options={AGENT_APPROVAL_MODES.map((mode) => ({
+ value: mode,
+ label: t(`society.approval_mode.${mode}`),
+ disabled: mode === "always_ask"
+ ? seat?.kind === "subscription" && seat.provider.runner !== "codex-cli"
+ : mode === "ask" && seat?.kind === "subscription"
+ && !["codex-cli", "claude-cli", "glm-cli"].includes(seat.provider.runner),
+ hint: t(`society.approval_mode.${mode}_hint`),
+ }))}
+ onChange={(v) => setApprovalMode(v as AgentApprovalMode)}
/>
+ <p className="mt-1 text-xs text-muted-foreground">
+ {t(`society.approval_mode.${approvalMode}_hint`)}
+ </p>
</div>
<div>
<div className="mb-2 flex items-center gap-2">
diff --git a/jarvis/ui/web/frontend/src/components/society/data.ts b/jarvis/ui/web/frontend/src/components/society/data.ts
index 072352d7cd..ff7e2cd5ff 100644
--- a/jarvis/ui/web/frontend/src/components/society/data.ts
+++ b/jarvis/ui/web/frontend/src/components/society/data.ts
@@ -16,7 +16,7 @@ import { useCallback, useEffect } from "react";
import { useQuery, useQueryClient, type QueryClient } from "@tanstack/react-query";
import type { MentionPlugin } from "./chat/mentionItems";
-import type { Checkpoint, SocietyAgentRow } from "@/lib/societyApi";
+import type { AgentApprovalMode, Checkpoint, SocietyAgentRow } from "@/lib/societyApi";
import { NEUTRAL_ASSISTANT_NAME } from "@/lib/assistantNameCache";
import { useEventStore } from "@/store/events";
@@ -43,6 +43,7 @@ export type AgentRunState = "idle" | "working" | "waiting" | "paused";
/** §6.2 — the unattended ceiling; "block" never appears (block is block). */
export type PermissionCeiling = "safe" | "monitor" | "ask";
+export type { AgentApprovalMode };
/** agent-definition.md §3.2 — everything the tiers allow, or only an allow-list. */
export type GrantMode = "all" | "allowlist";
@@ -116,6 +117,7 @@ export interface SocietyAgent {
denies: string[];
approvalRules: ApprovalRules;
permissionCeiling: PermissionCeiling;
+ approvalMode?: AgentApprovalMode | null;
dailyBudgetUsd: number;
checkpoint: AgentCheckpoint;
state: AgentRunState;
@@ -154,6 +156,7 @@ export interface NewAgentInput {
toolGrants: string[];
focus: string[];
permissionCeiling: PermissionCeiling;
+ approvalMode: AgentApprovalMode;
dailyBudgetUsd: number;
}
@@ -239,6 +242,7 @@ export function rowToAgent(row: SocietyAgentRow): SocietyAgent {
alwaysAllow: row.approval_rules?.always_allow ?? [],
},
permissionCeiling: row.permission_ceiling as PermissionCeiling,
+ approvalMode: row.approval_mode ?? null,
dailyBudgetUsd: row.daily_budget_usd,
checkpoint: row.checkpoint as AgentCheckpoint,
state: runState,
@@ -417,6 +421,7 @@ export function useCreateAgent() {
grants: input.grantMode === "allowlist" ? input.toolGrants : undefined,
focus: input.focus.length ? input.focus : undefined,
permission_ceiling: input.permissionCeiling,
+ approval_mode: input.approvalMode,
daily_budget_usd: input.dailyBudgetUsd,
};
try {
@@ -464,6 +469,7 @@ export function useCreateAgent() {
denies: [],
approvalRules: { requireApproval: [], alwaysAllow: [] },
permissionCeiling: input.permissionCeiling,
+ approvalMode: input.approvalMode,
dailyBudgetUsd: input.dailyBudgetUsd,
checkpoint: "idle",
state: "idle",
@@ -559,6 +565,7 @@ export interface AgentLimits {
/** 0 means no cap — the scheduler skips the budget gate entirely. */
dailyBudgetUsd: number;
permissionCeiling: PermissionCeiling;
+ approvalMode?: AgentApprovalMode | null;
maxConcurrentRuns: number;
}
@@ -583,6 +590,7 @@ export function useUpdateAgentLimits() {
const body = {
daily_budget_usd: Math.max(0, limits.dailyBudgetUsd),
permission_ceiling: limits.permissionCeiling,
+ ...(limits.approvalMode ? { approval_mode: limits.approvalMode } : {}),
max_concurrent_runs: Math.max(1, Math.round(limits.maxConcurrentRuns)),
};
if (!sample) {
@@ -595,6 +603,7 @@ export function useUpdateAgentLimits() {
} else {
agent.dailyBudgetUsd = body.daily_budget_usd;
agent.permissionCeiling = body.permission_ceiling;
+ if (limits.approvalMode) agent.approvalMode = limits.approvalMode;
agent.maxConcurrentRuns = body.max_concurrent_runs;
}
await client.invalidateQueries({ queryKey: ROSTER_QUERY_KEY });
diff --git a/jarvis/ui/web/frontend/src/components/society/office/NewWorkspaceFields.tsx b/jarvis/ui/web/frontend/src/components/society/office/NewWorkspaceFields.tsx
index 47e82a203b..70126a83b4 100644
--- a/jarvis/ui/web/frontend/src/components/society/office/NewWorkspaceFields.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/office/NewWorkspaceFields.tsx
@@ -71,13 +71,13 @@ export function NewWorkspaceFields({ projects, target, onTarget, name, onName, d
<Dialog.Root open={picking} onOpenChange={setPicking}>
<Dialog.Portal>
<Dialog.Overlay className="fixed inset-0 z-[90] bg-[rgb(var(--scrim-rgb)/0.4)]" />
- <Dialog.Content className="fixed left-1/2 top-1/2 z-[91] flex max-h-[85vh] w-[min(680px,calc(100vw-24px))] -translate-x-1/2 -translate-y-1/2 flex-col rounded-xl border border-border bg-popover p-4 text-foreground shadow-float">
+ <Dialog.Content onEscapeKeyDown={(e) => { if (e.target instanceof Element && e.target.closest("[data-escape-local]")) e.preventDefault(); }} className="fixed left-1/2 top-1/2 z-[91] flex max-h-[85vh] w-[min(680px,calc(100vw-24px))] -translate-x-1/2 -translate-y-1/2 flex-col rounded-xl border border-border bg-popover p-4 text-foreground shadow-float">
<Dialog.Title className="pr-8 text-sm font-semibold">{t("society.office.mission_ws_choose_title")}</Dialog.Title>
<Dialog.Description className="mt-1 text-xs text-muted-foreground">{t("society.office.mission_ws_choose_hint")}</Dialog.Description>
<Dialog.Close aria-label={t("society.office.mission_ws_close")} className="absolute right-3 top-3 rounded p-1 hover:bg-secondary"><X className="h-4 w-4" /></Dialog.Close>
<div className="my-3 min-h-0 overflow-auto">
<Suspense fallback={<p className="text-xs text-muted-foreground">{t("society.office.mission_loading")}</p>}>
- {picking && <FolderPicker selected={candidate} onSelect={setCandidate} />}
+ {picking && <FolderPicker selected={candidate} onSelect={setCandidate} className="p-0" />}
</Suspense>
</div>
<button type="button" disabled={!candidate}
diff --git a/jarvis/ui/web/frontend/src/components/society/roster/GroupRosterActions.tsx b/jarvis/ui/web/frontend/src/components/society/roster/GroupRosterActions.tsx
new file mode 100644
index 0000000000..cb0f8aba5f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/roster/GroupRosterActions.tsx
@@ -0,0 +1,108 @@
+import { useEffect, useLayoutEffect, useRef, useState, type KeyboardEvent } from "react";
+import { createPortal } from "react-dom";
+import * as Dialog from "@radix-ui/react-dialog";
+import { useQueryClient } from "@tanstack/react-query";
+import { Pencil, Trash2 } from "lucide-react";
+
+import { Button } from "@/components/ui/button";
+import { useT } from "@/i18n";
+import { deleteSocietyChatGroup, type SocietyChatGroup } from "@/lib/societyChatGroups";
+
+interface Props {
+ group: SocietyChatGroup;
+ x: number;
+ y: number;
+ onEdit: () => void;
+ onDismiss: () => void;
+}
+
+export function GroupRosterActions({ group, x, y, onEdit, onDismiss }: Props) {
+ const t = useT();
+ const client = useQueryClient();
+ const menuRef = useRef<HTMLDivElement>(null);
+ const [confirmUngroup, setConfirmUngroup] = useState(false);
+ const [busy, setBusy] = useState(false);
+ const [error, setError] = useState("");
+
+ useEffect(() => {
+ if (confirmUngroup) return;
+ const onPointerDown = (event: PointerEvent) => {
+ if (!menuRef.current?.contains(event.target as Node)) onDismiss();
+ };
+ const onKeyDown = (event: globalThis.KeyboardEvent) => {
+ if (event.key === "Escape") onDismiss();
+ };
+ document.addEventListener("pointerdown", onPointerDown, true);
+ document.addEventListener("keydown", onKeyDown, true);
+ document.addEventListener("scroll", onDismiss, true);
+ window.addEventListener("resize", onDismiss);
+ return () => {
+ document.removeEventListener("pointerdown", onPointerDown, true);
+ document.removeEventListener("keydown", onKeyDown, true);
+ document.removeEventListener("scroll", onDismiss, true);
+ window.removeEventListener("resize", onDismiss);
+ };
+ }, [confirmUngroup, onDismiss]);
+
+ useLayoutEffect(() => {
+ const menu = menuRef.current;
+ if (!menu || confirmUngroup) return;
+ const { width, height } = menu.getBoundingClientRect();
+ menu.style.left = `${Math.max(8, Math.min(x, window.innerWidth - width - 8))}px`;
+ menu.style.top = `${Math.max(8, Math.min(y, window.innerHeight - height - 8))}px`;
+ menu.style.visibility = "visible";
+ menu.querySelector<HTMLButtonElement>("button")?.focus();
+ }, [confirmUngroup, x, y]);
+
+ const onMenuKeyDown = (event: KeyboardEvent<HTMLDivElement>) => {
+ if (event.key !== "ArrowDown" && event.key !== "ArrowUp") return;
+ event.preventDefault();
+ const items = Array.from(menuRef.current?.querySelectorAll<HTMLButtonElement>("button") ?? []);
+ const index = items.indexOf(document.activeElement as HTMLButtonElement);
+ items[(index + (event.key === "ArrowDown" ? 1 : -1) + items.length) % items.length]?.focus();
+ };
+
+ const remove = async () => {
+ if (busy) return;
+ setBusy(true);
+ setError("");
+ try {
+ await deleteSocietyChatGroup(group.group_id);
+ await client.invalidateQueries({ queryKey: ["society", "chat-groups"] });
+ onDismiss();
+ } catch (cause) {
+ setError(cause instanceof Error ? cause.message : String(cause));
+ } finally {
+ setBusy(false);
+ }
+ };
+
+ return createPortal(<>
+ {!confirmUngroup && <div ref={menuRef} role="menu" aria-label={group.name} onKeyDown={onMenuKeyDown}
+ style={{ width: 240, visibility: "hidden" }}
+ className="fixed z-[100] overflow-hidden rounded-md border border-border bg-popover p-1 text-popover-foreground shadow-float">
+ <button type="button" role="menuitem" onClick={() => { onEdit(); onDismiss(); }}
+ className="flex w-full items-center gap-2 rounded px-3 py-2 text-left text-sm hover:bg-secondary focus:bg-secondary focus:outline-none">
+ <Pencil className="h-4 w-4" aria-hidden />{t("society.groups.edit")}
+ </button>
+ <button type="button" role="menuitem" onClick={() => setConfirmUngroup(true)}
+ className="flex w-full items-center gap-2 rounded px-3 py-2 text-left text-sm text-destructive hover:bg-destructive/10 focus:bg-destructive/10 focus:outline-none">
+ <Trash2 className="h-4 w-4" aria-hidden />{t("society.groups.delete")}
+ </button>
+ </div>}
+ <Dialog.Root open={confirmUngroup} onOpenChange={(open) => { if (!open && !busy) onDismiss(); }}>
+ <Dialog.Portal>
+ <Dialog.Overlay className="fixed inset-0 z-[101] bg-scrim/60" />
+ <Dialog.Content className="fixed left-1/2 top-1/2 z-[102] w-[min(400px,calc(100vw-24px))] -translate-x-1/2 -translate-y-1/2 rounded-xl border border-border bg-popover p-5 text-popover-foreground shadow-float outline-none">
+ <Dialog.Title className="text-base font-semibold">{t("society.groups.delete")}</Dialog.Title>
+ <Dialog.Description className="mt-2 text-sm text-muted-foreground">{t("society.groups.delete_confirm")}</Dialog.Description>
+ {error && <p role="alert" className="mt-2 text-sm text-destructive">{error}</p>}
+ <div className="mt-5 flex justify-end gap-2">
+ <Button variant="ghost" disabled={busy} onClick={onDismiss}>{t("society.groups.cancel")}</Button>
+ <Button variant="destructive" disabled={busy} onClick={() => void remove()}>{t("society.groups.delete")}</Button>
+ </div>
+ </Dialog.Content>
+ </Dialog.Portal>
+ </Dialog.Root>
+ </>, document.body);
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.test.tsx b/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.test.tsx
index 9fa6a23c71..38081b72b6 100644
--- a/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.test.tsx
@@ -69,6 +69,31 @@ describe("RosterRail status", () => {
fireEvent.click(screen.getByTestId("society-group-team"));
expect(openGroup).toHaveBeenCalledWith("team");
});
+ it("opens team actions on right-click and only ungroups after confirmation", async () => {
+ const request = vi.fn(async () => ({ ok: true, json: async () => ({}) }));
+ vi.stubGlobal("fetch", request);
+ const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
+ render(<QueryClientProvider client={client}><RosterRail {...baseProps} activeAgentId={null}
+ agents={[agent({ agentId: "scout", name: "Scout" }), agent({ agentId: "writer", name: "Writer" })]}
+ groups={[{ group_id: "team", name: "Launch team", members: ["scout", "writer"], created_ms: 1, updated_ms: 1 }]}
+ onOpenGroup={() => undefined} /></QueryClientProvider>);
+
+ const row = screen.getByTestId("society-group-team");
+ fireEvent.contextMenu(row, { clientX: 32, clientY: 56 });
+ expect(screen.getByRole("menu", { name: "Launch team" })).toBeTruthy();
+ expect(screen.getByRole("menuitem", { name: "society.groups.edit" })).toBeTruthy();
+ fireEvent.click(screen.getByRole("menuitem", { name: "society.groups.delete" }));
+ expect(screen.getByRole("dialog", { name: "society.groups.delete" })).toBeTruthy();
+ expect(request).not.toHaveBeenCalled();
+ fireEvent.click(screen.getByRole("button", { name: "society.groups.cancel" }));
+ expect(request).not.toHaveBeenCalled();
+
+ fireEvent.contextMenu(row, { clientX: 32, clientY: 56 });
+ fireEvent.click(screen.getByRole("menuitem", { name: "society.groups.delete" }));
+ fireEvent.click(screen.getByRole("button", { name: "society.groups.delete" }));
+ await waitFor(() => expect(request).toHaveBeenCalledWith("/api/society/chat-groups/team", { method: "DELETE" }));
+ await waitFor(() => expect(screen.queryByRole("dialog")).toBeNull());
+ });
it("shows a loading spinner while an agent is thinking", () => {
render(
<RosterRail
diff --git a/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx b/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx
index 3321427ae3..cfa4bee0c4 100644
--- a/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx
@@ -35,6 +35,7 @@ import type { SocietyChatGroup } from "@/lib/societyChatGroups";
import { AgentSwatch } from "../AgentSwatch";
import type { AgentRunState, SocietyAgent } from "../data";
import { AgentRosterActions } from "./AgentRosterActions";
+import { GroupRosterActions } from "./GroupRosterActions";
import { useRosterUnread } from "./useRosterUnread";
import { ChatGroupDialog } from "../chat/ChatGroupDialog";
@@ -131,8 +132,12 @@ export function RosterRail({
const [groupReady, setGroupReady] = useState(false);
const [showHidden, setShowHidden] = useState(false);
const [menu, setMenu] = useState<{ agentId: string; x: number; y: number } | null>(null);
+ const [groupMenu, setGroupMenu] = useState<{ groupId: string; x: number; y: number } | null>(null);
+ const [editingGroupId, setEditingGroupId] = useState<string | null>(null);
const profile = agents.find((agent) => agent.agentId === profileId);
const menuAgent = agents.find((agent) => agent.agentId === menu?.agentId);
+ const menuGroup = groups.find((group) => group.group_id === groupMenu?.groupId);
+ const editingGroup = groups.find((group) => group.group_id === editingGroupId);
const unread = useRosterUnread(agents, activeAgentId);
const assistantName = useEventStore((state) => state.assistantName);
@@ -147,8 +152,15 @@ export function RosterRail({
const openMenu = (event: MouseEvent, agentId: string) => {
event.preventDefault();
event.stopPropagation();
+ setGroupMenu(null);
setMenu({ agentId, x: event.clientX, y: event.clientY });
};
+ const openGroupMenu = (event: MouseEvent, groupId: string) => {
+ event.preventDefault();
+ event.stopPropagation();
+ setMenu(null);
+ setGroupMenu({ groupId, x: event.clientX, y: event.clientY });
+ };
const lead = useMemo(() => agents.find((a) => a.tier === "lead") ?? null, [agents]);
const groupedIds = useMemo(() => new Set(groups.flatMap((group) => group.members)), [groups]);
@@ -562,7 +574,7 @@ export function RosterRail({
) : null}
{visibleGroups.length > 0 && <ul className="flex flex-col gap-0.5 px-2 pb-2">
{visibleGroups.map((group) => <li key={group.group_id}>
- <button type="button" data-group-id={group.group_id} onClick={() => onOpenGroup?.(group.group_id)} aria-current={activeGroupId === group.group_id ? "true" : undefined}
+ <button type="button" data-group-id={group.group_id} onClick={() => onOpenGroup?.(group.group_id)} onContextMenu={(event) => openGroupMenu(event, group.group_id)} aria-current={activeGroupId === group.group_id ? "true" : undefined}
className={cn("flex w-full items-center gap-2.5 rounded-md px-2 py-2 text-left hover:bg-secondary focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring", activeGroupId === group.group_id && "bg-secondary", groupHover?.kind === "group" && groupHover.id === group.group_id && (groupReady ? "ring-2 ring-primary" : "ring-1 ring-border-strong"))}
data-testid={`society-group-${group.group_id}`}>
<span className="flex w-12 shrink-0 items-center justify-center">
@@ -646,8 +658,11 @@ export function RosterRail({
{footer}
{menu && menuAgent && <AgentRosterActions key={menu.agentId} agent={menuAgent} roster={agents} sample={sample}
hidden={hiddenIds.includes(menu.agentId)} x={menu.x} y={menu.y} onVisibilityChange={setHidden} onDismiss={closeMenu} />}
+ {groupMenu && menuGroup && <GroupRosterActions key={groupMenu.groupId} group={menuGroup} x={groupMenu.x} y={groupMenu.y}
+ onEdit={() => setEditingGroupId(menuGroup.group_id)} onDismiss={() => setGroupMenu(null)} />}
{profile && <Suspense fallback={null}><AgentProfileDialog key={profile.agentId} agent={profile} sample={sample} onClose={() => setProfileId(null)} /></Suspense>}
{creatingGroup && <ChatGroupDialog agents={agents} onClose={() => setCreatingGroup(false)} onSaved={(id) => onOpenGroup?.(id)} />}
+ {editingGroup && <ChatGroupDialog group={editingGroup} agents={agents} onClose={() => setEditingGroupId(null)} onSaved={(id) => onOpenGroup?.(id)} />}
</aside>
);
}
diff --git a/jarvis/ui/web/frontend/src/hooks/useSectionPrefetch.test.tsx b/jarvis/ui/web/frontend/src/hooks/useSectionPrefetch.test.tsx
new file mode 100644
index 0000000000..5042dd7cec
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/hooks/useSectionPrefetch.test.tsx
@@ -0,0 +1,51 @@
+import { act, cleanup, fireEvent, render, screen } from "@testing-library/react";
+import { afterEach, beforeEach, expect, test, vi } from "vitest";
+import { sectionPrefetch } from "@/lib/sectionPrefetch";
+import { SECTION_PREFETCH_DELAY_MS, useSectionPrefetch } from "./useSectionPrefetch";
+import type { SectionId } from "@/store/events";
+
+function Link({ section = "agents" }: { section?: SectionId }) {
+ return <button {...useSectionPrefetch(section)}>destination</button>;
+}
+
+beforeEach(() => {
+ vi.useFakeTimers();
+ vi.spyOn(sectionPrefetch, "prefetch").mockResolvedValue();
+});
+afterEach(() => {
+ cleanup();
+ vi.restoreAllMocks();
+ vi.useRealTimers();
+});
+
+test("mounting and a passing pointer do no work", () => {
+ render(<Link />);
+ act(() => { vi.advanceTimersByTime(60_000); });
+ const button = screen.getByRole("button");
+ fireEvent.mouseEnter(button);
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS - 1); });
+ fireEvent.mouseLeave(button);
+ act(() => { vi.advanceTimersByTime(60_000); });
+ expect(sectionPrefetch.prefetch).not.toHaveBeenCalled();
+});
+
+test.each(["mouseEnter", "focus"] as const)("%s warms only the intended destination", (event) => {
+ render(<Link />);
+ fireEvent[event](screen.getByRole("button"));
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS); });
+ expect(sectionPrefetch.prefetch).toHaveBeenCalledExactlyOnceWith("agents");
+});
+
+test("blur, a new destination and unmount cancel pending work", () => {
+ const { rerender, unmount } = render(<Link />);
+ fireEvent.focus(screen.getByRole("button"));
+ fireEvent.blur(screen.getByRole("button"));
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS); });
+ fireEvent.mouseEnter(screen.getByRole("button"));
+ rerender(<Link section="docs" />);
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS); });
+ fireEvent.focus(screen.getByRole("button"));
+ unmount();
+ act(() => { vi.advanceTimersByTime(SECTION_PREFETCH_DELAY_MS); });
+ expect(sectionPrefetch.prefetch).not.toHaveBeenCalled();
+});
diff --git a/jarvis/ui/web/frontend/src/hooks/useSectionPrefetch.ts b/jarvis/ui/web/frontend/src/hooks/useSectionPrefetch.ts
new file mode 100644
index 0000000000..5b47cce645
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/hooks/useSectionPrefetch.ts
@@ -0,0 +1,30 @@
+import { useEffect, useRef } from "react";
+import { sectionPrefetch } from "@/lib/sectionPrefetch";
+import type { SectionId } from "@/store/events";
+
+export const SECTION_PREFETCH_DELAY_MS = 150;
+
+/** A brief hover/focus warms one destination; passing over the rail warms none. */
+export function useSectionPrefetch(section: SectionId) {
+ const timer = useRef<ReturnType<typeof setTimeout> | null>(null);
+ const cancel = () => {
+ if (timer.current !== null) clearTimeout(timer.current);
+ timer.current = null;
+ };
+ useEffect(() => cancel, [section]);
+
+ const schedule = () => {
+ cancel();
+ timer.current = setTimeout(() => {
+ timer.current = null;
+ void sectionPrefetch.prefetch(section);
+ }, SECTION_PREFETCH_DELAY_MS);
+ };
+
+ return {
+ onMouseEnter: schedule,
+ onMouseLeave: cancel,
+ onFocus: schedule,
+ onBlur: cancel,
+ };
+}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/de.json b/jarvis/ui/web/frontend/src/i18n/locales/de.json
index f3ce0aaaf9..35b749185f 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/de.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/de.json
@@ -161,6 +161,7 @@
"group_system": "System",
"agent_instructions": "{name}.md",
"appshots": "Appshots",
+ "jarvisx": "Jarvis X",
"jarvis_actions": "Jarvis-Aktionen",
"marketplace": "Marketplace",
"agentic_ide": "Agentic IDE",
@@ -1923,7 +1924,11 @@
"managed_speaking_title": "Sprechen",
"managed_voice_unavailable": "noch nicht sprachgetestet",
"managed_voice_install_required": "Laufzeit-Installation nötig",
- "managed_apply_test": "Anwenden & Sprache testen",
+ "managed_apply_test": "Anwenden, testen & lokal sprechen",
+ "managed_protocol_options": "Erweiterte Modell- und Stimmoptionen des Servers",
+ "managed_setup_testing": "Die ausgewählten Modelle werden vorbereitet und die Sprachausgabe geprüft. Beim ersten Start kann das länger dauern.",
+ "managed_selected": "Für Sprache ausgewählt. Diese Modelle starten automatisch mit Jarvis.",
+ "managed_not_selected": "Dieser Server ist noch nicht für Sprache ausgewählt. Zwei Modelle auswählen und anwenden, um lokal zu sprechen.",
"managed_full_catalog": "Vollständigen Ollama-Katalog durchsuchen",
"managed_voice_catalog_title": "Open-Source-Sprachmodelle durchsuchen",
"managed_voice_catalog_search": "Nach Modell, Backend, Sprache oder Lizenz suchen…",
@@ -2059,7 +2064,7 @@
"realtime_options_error": "Die Sprachoptionen konnten nicht geladen werden.",
"realtime_options_retry": "Erneut versuchen",
"realtime_model_managed": "Verwaltet",
- "realtime_model_managed_hint": "ChatGPT wählt das Subscription-Modell. Bei einem API-Realtime-Anbieter kannst du ein GPT-Realtime-Modell festlegen.",
+ "realtime_model_managed_hint": "Dieser Anbieter verwaltet das Modell. Die Auswahl lässt sich über die Einrichtung oben ändern.",
"guide_realtime_title": "Welches Modell soll ich nehmen?",
"guide_realtime_body": "OpenAI API: GPT Realtime 2.1 ist das aktuelle Flaggschiff, 2.1 Mini die günstigere Wahl. Google: Gemini 3.1 Flash Live ist das aktuelle Live-Flaggschiff.",
"guide_computer_use_title": "Welcher Provider soll das Tool Model antreiben?",
@@ -5172,6 +5177,18 @@
"monitor": "Beobachten",
"ask": "Erst fragen"
},
+ "approval_mode": {
+ "title": "Agentenberechtigungen",
+ "bypass": "Berechtigungen umgehen",
+ "ask": "Fragen",
+ "always_ask": "Immer fragen",
+ "bypass_hint": "Jarvis-Werkzeuge laufen ohne Nachfragen; Sperren und verweigerte Werkzeuge gelten weiter. Eigene Werkzeuge einer Abo-CLI folgen deren Regeln.",
+ "ask_hint": "Lesen ohne Nachfrage; vor Änderungen und anderen Aktionen hier im Chat fragen.",
+ "always_ask_hint": "Vor jedem Jarvis-Werkzeugaufruf hier im Chat fragen, auch beim Lesen.",
+ "legacy_hint": "Dieser Agent behält seine bisherigen Chat-Berechtigungen, bis du einen Modus auswählst.",
+ "legacy_ceiling": "Bisherige Berechtigungsgrenze",
+ "unavailable": "Dieser Runner kann für diesen Modus keine nutzbare Freigabekarte anzeigen."
+ },
"grant": {
"all": "Alle verbundenen Werkzeuge",
"allowlist": "Nur Erlaubnisliste"
@@ -5797,5 +5814,124 @@
"save_error": "Speichern fehlgeschlagen: {0}",
"toast_refused": "Kein Appshot: {0}",
"chip_label": "Appshot an deine Nachricht gehängt"
+ },
+ "jarvisx": {
+ "title": "Jarvis X",
+ "subtitle": "Bildschirmfotos und Bildschirmaufnahmen, mit einem schnellen Editor zum Markieren.",
+ "tab_library": "Bibliothek",
+ "tab_settings": "Einstellungen",
+ "time_now": "Gerade eben",
+ "time_minutes": "vor {0} Min.",
+ "action_failed": "Das hat nicht geklappt.",
+ "deleted": "Gelöscht",
+ "copied": "In die Zwischenablage kopiert",
+ "copy_failed": "Kopieren fehlgeschlagen.",
+ "capture_region": "Bereich",
+ "capture_window": "Fenster",
+ "capture_fullscreen": "Ganzer Bildschirm",
+ "recording": "Aufnahme läuft",
+ "record_stop": "Stopp",
+ "record_region": "Bereich aufnehmen",
+ "record_fullscreen": "Bildschirm aufnehmen",
+ "recording_unavailable": "Bildschirmaufnahmen sind auf diesem Computer nicht verfügbar.",
+ "disabled_hint": "Jarvis X ist aus. Schalte es unter Einstellungen ein.",
+ "load_failed": "Deine Aufnahmen konnten nicht geladen werden: {0}",
+ "empty_title": "Noch keine Aufnahmen",
+ "empty_body": "Mach ein Bildschirmfoto per Tastenkürzel oder mit den Knöpfen oben. Es erscheint hier, bereit zum Bearbeiten.",
+ "delete_title": "Diese Aufnahme löschen?",
+ "delete_body": "Die Datei wird von deiner Festplatte entfernt. Das lässt sich nicht rückgängig machen.",
+ "delete": "Löschen",
+ "open_video": "{0} abspielen",
+ "open_editor": "{0} bearbeiten",
+ "mode_region": "Bereich",
+ "mode_window": "Fenster",
+ "mode_fullscreen": "Ganzer Bildschirm",
+ "edited": "Bearbeitet",
+ "kind_video": "Video",
+ "kind_image": "Bildschirmfoto",
+ "edit": "Bearbeiten",
+ "open_window": "In eigenem Fenster öffnen",
+ "copy": "Kopieren",
+ "reveal": "Im Ordner zeigen",
+ "editor": {
+ "title": "Jarvis-X-Editor",
+ "no_item": "Keine Aufnahme angegeben. Öffne den Editor aus der Jarvis-X-Bibliothek.",
+ "image_failed": "Das Bild konnte nicht geladen werden.",
+ "tools": "Werkzeuge",
+ "tool_select": "Auswählen und verschieben",
+ "tool_rect": "Rechteck",
+ "tool_highlight": "Markierungsfeld",
+ "tool_ellipse": "Ellipse",
+ "tool_arrow": "Pfeil",
+ "tool_line": "Linie",
+ "tool_pen": "Stift",
+ "tool_marker": "Textmarker",
+ "tool_text": "Text",
+ "tool_counter": "Nummerierter Schritt",
+ "tool_blur": "Verpixeln",
+ "tool_crop": "Zuschneiden",
+ "undo": "Rückgängig",
+ "redo": "Wiederholen",
+ "save": "Speichern",
+ "done": "Speichern & schließen",
+ "save_failed": "Speichern fehlgeschlagen: {0}",
+ "saved": "Gespeichert",
+ "color": "Farbe",
+ "stroke": "Linienstärke",
+ "font_size": "Textgröße",
+ "shadow": "Schatten",
+ "crop_hint": "Zieh über den Teil, den du behalten willst.",
+ "crop_reset": "Zuschnitt aufheben",
+ "crop_apply": "Fertig",
+ "select_hint": "Klick eine Markierung an, um sie zu verschieben. Doppelklick auf Text, um ihn zu ändern.",
+ "delete_shape": "Markierung löschen",
+ "base": "Ausgangsbild",
+ "base_edited": "Bearbeitet",
+ "base_original": "Original",
+ "zoom_out": "Verkleinern",
+ "zoom_in": "Vergrößern",
+ "zoom_fit": "An Fenster anpassen",
+ "load_failed": "Diese Aufnahme ließ sich nicht öffnen.",
+ "dev_hint": "Vorschau: Dieses Bild ist keine gespeicherte Aufnahme und kann nicht gespeichert werden.",
+ "discard_title": "Änderungen verwerfen?",
+ "discard_body": "Deine Markierungen seit dem letzten Speichern gehen verloren.",
+ "discard_confirm": "Verwerfen",
+ "switch_title": "Ausgangsbild wechseln?",
+ "switch_body": "Nicht gespeicherte Markierungen gehen verloren. Gespeicherte Änderungen bleiben in der bearbeiteten Version."
+ },
+ "settings": {
+ "load_failed": "Die Jarvis-X-Einstellungen konnten nicht geladen werden.",
+ "section_general": "Allgemein",
+ "enabled_label": "Jarvis X verwenden",
+ "enabled_hint": "Schaltet die Aufnahme-Tastenkürzel ein. Aus heißt: Die Kürzel tun nichts.",
+ "section_shortcuts": "Tastenkürzel",
+ "shortcuts_off": "Die Kürzel funktionieren nur, wenn Jarvis X eingeschaltet ist.",
+ "hotkey_region": "Bereich aufnehmen (Foto)",
+ "hotkey_window": "Fenster aufnehmen (Foto)",
+ "hotkey_fullscreen": "Ganzen Bildschirm aufnehmen (Foto)",
+ "hotkey_record_region": "Bereich als Video aufnehmen",
+ "hotkey_record_fullscreen": "Ganzen Bildschirm als Video aufnehmen",
+ "hotkey_stop_recording": "Videoaufnahme beenden",
+ "hotkey_armed": "Aktiv",
+ "hotkey_unarmed": "Nicht aktiv: {0}",
+ "hotkey_unarmed_plain": "Gerade nicht aktiv",
+ "save_error": "Speichern fehlgeschlagen: {0}",
+ "section_thumbnail": "Vorschaubild",
+ "persist_label": "Vorschaubild behalten, bis ich es schließe",
+ "persist_hint": "Nach einer Aufnahme erscheint eine kleine Vorschau in der Ecke deines Bildschirms.",
+ "dismiss_label": "Vorschaubild ausblenden nach",
+ "dismiss_hint": "Es verschwindet nach {0} Sekunden von selbst.",
+ "dismiss_seconds": "Sekunden",
+ "seconds_unit": "s",
+ "section_after": "Nach einer Aufnahme",
+ "save_dir_label": "Speicherordner",
+ "save_dir_hint": "Hier werden neue Bildschirmfotos und Aufnahmen abgelegt.",
+ "clipboard_label": "In die Zwischenablage kopieren",
+ "clipboard_hint": "Jedes neue Bildschirmfoto lässt sich sofort einfügen.",
+ "sound_label": "Auslöser-Geräusch abspielen",
+ "effect_label": "Animation zeigen",
+ "effect_hint": "Ein kurzer Blitz, dann fliegt das Vorschaubild in die Ecke.",
+ "recording_unavailable": "Bildschirmaufnahmen sind hier nicht verfügbar: {0}"
+ }
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/en.json b/jarvis/ui/web/frontend/src/i18n/locales/en.json
index 61ecf17509..fc894e93c9 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/en.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/en.json
@@ -161,6 +161,7 @@
"group_system": "System",
"agent_instructions": "{name}.md",
"appshots": "Appshots",
+ "jarvisx": "Jarvis X",
"jarvis_actions": "Jarvis actions",
"marketplace": "Marketplace",
"agentic_ide": "Agentic IDE",
@@ -1956,7 +1957,11 @@
"managed_speaking_title": "Speak",
"managed_voice_unavailable": "not voice-tested yet",
"managed_voice_install_required": "runtime install required",
- "managed_apply_test": "Apply & voice-test",
+ "managed_apply_test": "Apply, test & use local voice",
+ "managed_protocol_options": "Advanced server model and voice options",
+ "managed_setup_testing": "Preparing the selected models and testing speech. This can take longer on the first start.",
+ "managed_selected": "Selected for voice. These models start automatically with Jarvis.",
+ "managed_not_selected": "This server is not selected for voice yet. Choose two models and apply them to use local voice.",
"managed_full_catalog": "Browse the full Ollama catalog",
"managed_voice_catalog_title": "Browse open-source speech models",
"managed_voice_catalog_search": "Search by model, backend, language, or license…",
@@ -2093,7 +2098,7 @@
"realtime_options_error": "Voice options could not be loaded.",
"realtime_options_retry": "Try again",
"realtime_model_managed": "Managed",
- "realtime_model_managed_hint": "ChatGPT selects the subscription model. Choose an API Realtime provider to pin a GPT Realtime model.",
+ "realtime_model_managed_hint": "This provider manages the model. Use its setup controls above to change the selection.",
"guide_realtime_title": "Which model should I pick?",
"guide_realtime_body": "OpenAI API: GPT Realtime 2.1 is the current flagship and 2.1 Mini is the budget pick. Google: Gemini 3.1 Flash Live is the current Live flagship.",
"guide_computer_use_title": "Which provider should power the Tool Model?",
@@ -5205,6 +5210,18 @@
"monitor": "Monitor",
"ask": "Ask first"
},
+ "approval_mode": {
+ "title": "Agent permissions",
+ "bypass": "Bypass permissions",
+ "ask": "Ask",
+ "always_ask": "Always ask",
+ "bypass_hint": "Jarvis tools run without prompts; Jarvis blocks and denied tools still apply. A subscription CLI's native tools follow that runner's controls.",
+ "ask_hint": "Read freely; ask in this chat before changes or other actions.",
+ "always_ask_hint": "Ask in this chat before every Jarvis tool call, including reads.",
+ "legacy_hint": "This agent keeps its existing chat permissions until you choose a mode.",
+ "legacy_ceiling": "Existing permission ceiling",
+ "unavailable": "This runner cannot show a usable approval card for this mode."
+ },
"grant": {
"all": "All connected tools",
"allowlist": "Allow-list only"
@@ -5830,5 +5847,124 @@
"save_error": "Could not save: {0}",
"toast_refused": "No appshot: {0}",
"chip_label": "Appshot added to your message"
+ },
+ "jarvisx": {
+ "title": "Jarvis X",
+ "subtitle": "Screenshots and screen recordings, with a quick editor to mark them up.",
+ "tab_library": "Library",
+ "tab_settings": "Settings",
+ "time_now": "Just now",
+ "time_minutes": "{0} min ago",
+ "action_failed": "That did not work.",
+ "deleted": "Deleted",
+ "copied": "Copied to the clipboard",
+ "copy_failed": "Could not copy.",
+ "capture_region": "Region",
+ "capture_window": "Window",
+ "capture_fullscreen": "Full screen",
+ "recording": "Recording",
+ "record_stop": "Stop",
+ "record_region": "Record region",
+ "record_fullscreen": "Record screen",
+ "recording_unavailable": "Screen recording is not available on this computer.",
+ "disabled_hint": "Jarvis X is off. Turn it on under Settings.",
+ "load_failed": "Could not load your captures: {0}",
+ "empty_title": "No captures yet",
+ "empty_body": "Take a screenshot with a shortcut or the buttons above. It shows up here, ready to edit.",
+ "delete_title": "Delete this capture?",
+ "delete_body": "The file is removed from your disk. This cannot be undone.",
+ "delete": "Delete",
+ "open_video": "Play {0}",
+ "open_editor": "Edit {0}",
+ "mode_region": "Region",
+ "mode_window": "Window",
+ "mode_fullscreen": "Full screen",
+ "edited": "Edited",
+ "kind_video": "Video",
+ "kind_image": "Screenshot",
+ "edit": "Edit",
+ "open_window": "Open in its own window",
+ "copy": "Copy",
+ "reveal": "Show in folder",
+ "editor": {
+ "title": "Jarvis X editor",
+ "no_item": "No capture was named. Open the editor from the Jarvis X library.",
+ "image_failed": "The picture could not be loaded.",
+ "tools": "Tools",
+ "tool_select": "Select and move",
+ "tool_rect": "Rectangle",
+ "tool_highlight": "Highlight box",
+ "tool_ellipse": "Ellipse",
+ "tool_arrow": "Arrow",
+ "tool_line": "Line",
+ "tool_pen": "Pen",
+ "tool_marker": "Highlighter",
+ "tool_text": "Text",
+ "tool_counter": "Numbered step",
+ "tool_blur": "Pixelate",
+ "tool_crop": "Crop",
+ "undo": "Undo",
+ "redo": "Redo",
+ "save": "Save",
+ "done": "Save & close",
+ "save_failed": "Could not save: {0}",
+ "saved": "Saved",
+ "color": "Colour",
+ "stroke": "Line width",
+ "font_size": "Text size",
+ "shadow": "Shadow",
+ "crop_hint": "Drag over the part you want to keep.",
+ "crop_reset": "Reset crop",
+ "crop_apply": "Done",
+ "select_hint": "Click a mark to move it. Double-click text to edit it.",
+ "delete_shape": "Delete mark",
+ "base": "Start from",
+ "base_edited": "Edited",
+ "base_original": "Original",
+ "zoom_out": "Zoom out",
+ "zoom_in": "Zoom in",
+ "zoom_fit": "Fit to window",
+ "load_failed": "This capture could not be opened.",
+ "dev_hint": "Preview mode: this picture is not a stored capture, so it cannot be saved.",
+ "discard_title": "Discard your changes?",
+ "discard_body": "Your marks since the last save will be lost.",
+ "discard_confirm": "Discard",
+ "switch_title": "Switch the starting picture?",
+ "switch_body": "Your unsaved marks will be lost. Saved edits stay in the edited version."
+ },
+ "settings": {
+ "load_failed": "The Jarvis X settings could not be loaded.",
+ "section_general": "General",
+ "enabled_label": "Use Jarvis X",
+ "enabled_hint": "Turns the capture shortcuts on. Off means the shortcuts do nothing.",
+ "section_shortcuts": "Keyboard shortcuts",
+ "shortcuts_off": "The shortcuts only work while Jarvis X is on.",
+ "hotkey_region": "Capture a region",
+ "hotkey_window": "Capture a window",
+ "hotkey_fullscreen": "Capture the full screen",
+ "hotkey_record_region": "Record a region",
+ "hotkey_record_fullscreen": "Record the full screen",
+ "hotkey_stop_recording": "Stop recording",
+ "hotkey_armed": "Active",
+ "hotkey_unarmed": "Not active: {0}",
+ "hotkey_unarmed_plain": "Not active right now",
+ "save_error": "Could not save: {0}",
+ "section_thumbnail": "Thumbnail",
+ "persist_label": "Keep the thumbnail until I close it",
+ "persist_hint": "After a capture, a small preview sits in the corner of your screen.",
+ "dismiss_label": "Hide the thumbnail after",
+ "dismiss_hint": "It disappears on its own after {0} seconds.",
+ "dismiss_seconds": "Seconds",
+ "seconds_unit": "s",
+ "section_after": "After a capture",
+ "save_dir_label": "Save folder",
+ "save_dir_hint": "Where new screenshots and recordings are stored.",
+ "clipboard_label": "Copy to the clipboard",
+ "clipboard_hint": "Every new screenshot is ready to paste right away.",
+ "sound_label": "Play the shutter sound",
+ "effect_label": "Show the animation",
+ "effect_hint": "A short flash, then the thumbnail flies into the corner.",
+ "recording_unavailable": "Screen recording is not available here: {0}"
+ }
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/es.json b/jarvis/ui/web/frontend/src/i18n/locales/es.json
index 51f4e157fa..60d32cdf48 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/es.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/es.json
@@ -161,6 +161,7 @@
"group_system": "Sistema",
"agent_instructions": "{name}.md",
"appshots": "Appshots",
+ "jarvisx": "Jarvis X",
"jarvis_actions": "Acciones de Jarvis",
"marketplace": "Marketplace",
"agentic_ide": "IDE agéntico",
@@ -1923,7 +1924,11 @@
"managed_speaking_title": "Hablar",
"managed_voice_unavailable": "aún sin prueba de voz",
"managed_voice_install_required": "requiere instalar el runtime",
- "managed_apply_test": "Aplicar y probar la voz",
+ "managed_apply_test": "Aplicar, probar y usar voz local",
+ "managed_protocol_options": "Opciones avanzadas de modelo y voz del servidor",
+ "managed_setup_testing": "Preparando los modelos seleccionados y probando la voz. El primer inicio puede tardar más.",
+ "managed_selected": "Seleccionado para voz. Estos modelos se inician automáticamente con Jarvis.",
+ "managed_not_selected": "Este servidor aún no está seleccionado para voz. Elige dos modelos y aplícalos para usar voz local.",
"managed_full_catalog": "Explorar todo el catálogo de Ollama",
"managed_voice_catalog_title": "Explorar modelos de voz de código abierto",
"managed_voice_catalog_search": "Buscar por modelo, backend, idioma o licencia…",
@@ -2059,7 +2064,7 @@
"realtime_options_error": "No se pudieron cargar las opciones de voz.",
"realtime_options_retry": "Intentar de nuevo",
"realtime_model_managed": "Gestionado",
- "realtime_model_managed_hint": "ChatGPT selecciona el modelo de suscripción. Elige un proveedor Realtime por API para fijar un modelo GPT Realtime.",
+ "realtime_model_managed_hint": "Este proveedor gestiona el modelo. Usa los controles de configuración anteriores para cambiar la selección.",
"guide_realtime_title": "¿Qué modelo debería elegir?",
"guide_realtime_body": "API de OpenAI: GPT Realtime 2.1 es el modelo insignia actual y 2.1 Mini es la opción económica. Google: Gemini 3.1 Flash Live es el modelo insignia actual de Live.",
"guide_computer_use_title": "¿Qué proveedor debería impulsar el Tool Model?",
@@ -5172,6 +5177,18 @@
"monitor": "Supervisar",
"ask": "Preguntar primero"
},
+ "approval_mode": {
+ "title": "Permisos del agente",
+ "bypass": "Omitir permisos",
+ "ask": "Preguntar",
+ "always_ask": "Preguntar siempre",
+ "bypass_hint": "Las herramientas de Jarvis se ejecutan sin preguntas; sus bloqueos siguen activos. Las herramientas nativas de una CLI de suscripción siguen sus propias reglas.",
+ "ask_hint": "Lee libremente; pregunta en este chat antes de cambios u otras acciones.",
+ "always_ask_hint": "Pregunta en este chat antes de cada herramienta de Jarvis, incluso para leer.",
+ "legacy_hint": "Este agente conserva los permisos actuales del chat hasta que elijas un modo.",
+ "legacy_ceiling": "Límite de permisos anterior",
+ "unavailable": "Este ejecutor no puede mostrar una aprobación utilizable para este modo."
+ },
"grant": {
"all": "Todas las herramientas conectadas",
"allowlist": "Solo lista permitida"
@@ -5797,5 +5814,124 @@
"save_error": "No se pudo guardar: {0}",
"toast_refused": "Sin appshot: {0}",
"chip_label": "Appshot añadido a tu mensaje"
+ },
+ "jarvisx": {
+ "title": "Jarvis X",
+ "subtitle": "Capturas y grabaciones de pantalla, con un editor rápido para marcarlas.",
+ "tab_library": "Biblioteca",
+ "tab_settings": "Ajustes",
+ "time_now": "Ahora mismo",
+ "time_minutes": "hace {0} min",
+ "action_failed": "No ha funcionado.",
+ "deleted": "Eliminado",
+ "copied": "Copiado al portapapeles",
+ "copy_failed": "No se pudo copiar.",
+ "capture_region": "Región",
+ "capture_window": "Ventana",
+ "capture_fullscreen": "Pantalla completa",
+ "recording": "Grabando",
+ "record_stop": "Detener",
+ "record_region": "Grabar región",
+ "record_fullscreen": "Grabar pantalla",
+ "recording_unavailable": "La grabación de pantalla no está disponible en este equipo.",
+ "disabled_hint": "Jarvis X está desactivado. Actívalo en Ajustes.",
+ "load_failed": "No se pudieron cargar tus capturas: {0}",
+ "empty_title": "Aún no hay capturas",
+ "empty_body": "Haz una captura con un atajo o con los botones de arriba. Aparece aquí, lista para editar.",
+ "delete_title": "¿Eliminar esta captura?",
+ "delete_body": "El archivo se borra de tu disco. No se puede deshacer.",
+ "delete": "Eliminar",
+ "open_video": "Reproducir {0}",
+ "open_editor": "Editar {0}",
+ "mode_region": "Región",
+ "mode_window": "Ventana",
+ "mode_fullscreen": "Pantalla completa",
+ "edited": "Editada",
+ "kind_video": "Vídeo",
+ "kind_image": "Captura",
+ "edit": "Editar",
+ "open_window": "Abrir en su propia ventana",
+ "copy": "Copiar",
+ "reveal": "Mostrar en la carpeta",
+ "editor": {
+ "title": "Editor de Jarvis X",
+ "no_item": "No se indicó ninguna captura. Abre el editor desde la biblioteca de Jarvis X.",
+ "image_failed": "No se pudo cargar la imagen.",
+ "tools": "Herramientas",
+ "tool_select": "Seleccionar y mover",
+ "tool_rect": "Rectángulo",
+ "tool_highlight": "Recuadro resaltado",
+ "tool_ellipse": "Elipse",
+ "tool_arrow": "Flecha",
+ "tool_line": "Línea",
+ "tool_pen": "Lápiz",
+ "tool_marker": "Rotulador",
+ "tool_text": "Texto",
+ "tool_counter": "Paso numerado",
+ "tool_blur": "Pixelar",
+ "tool_crop": "Recortar",
+ "undo": "Deshacer",
+ "redo": "Rehacer",
+ "save": "Guardar",
+ "done": "Guardar y cerrar",
+ "save_failed": "No se pudo guardar: {0}",
+ "saved": "Guardado",
+ "color": "Color",
+ "stroke": "Grosor",
+ "font_size": "Tamaño del texto",
+ "shadow": "Sombra",
+ "crop_hint": "Arrastra sobre la parte que quieres conservar.",
+ "crop_reset": "Quitar recorte",
+ "crop_apply": "Listo",
+ "select_hint": "Haz clic en una marca para moverla. Doble clic en un texto para editarlo.",
+ "delete_shape": "Eliminar marca",
+ "base": "Partir de",
+ "base_edited": "Editada",
+ "base_original": "Original",
+ "zoom_out": "Alejar",
+ "zoom_in": "Acercar",
+ "zoom_fit": "Ajustar a la ventana",
+ "load_failed": "No se pudo abrir esta captura.",
+ "dev_hint": "Vista previa: esta imagen no es una captura guardada y no se puede guardar.",
+ "discard_title": "¿Descartar los cambios?",
+ "discard_body": "Se perderán tus marcas desde el último guardado.",
+ "discard_confirm": "Descartar",
+ "switch_title": "¿Cambiar la imagen de partida?",
+ "switch_body": "Se perderán las marcas sin guardar. Las ediciones guardadas siguen en la versión editada."
+ },
+ "settings": {
+ "load_failed": "No se pudieron cargar los ajustes de Jarvis X.",
+ "section_general": "General",
+ "enabled_label": "Usar Jarvis X",
+ "enabled_hint": "Activa los atajos de captura. Desactivado, los atajos no hacen nada.",
+ "section_shortcuts": "Atajos de teclado",
+ "shortcuts_off": "Los atajos solo funcionan con Jarvis X activado.",
+ "hotkey_region": "Capturar una región",
+ "hotkey_window": "Capturar una ventana",
+ "hotkey_fullscreen": "Capturar la pantalla completa",
+ "hotkey_record_region": "Grabar una región",
+ "hotkey_record_fullscreen": "Grabar la pantalla completa",
+ "hotkey_stop_recording": "Detener la grabación",
+ "hotkey_armed": "Activo",
+ "hotkey_unarmed": "No activo: {0}",
+ "hotkey_unarmed_plain": "No activo ahora mismo",
+ "save_error": "No se pudo guardar: {0}",
+ "section_thumbnail": "Miniatura",
+ "persist_label": "Mantener la miniatura hasta que la cierre",
+ "persist_hint": "Tras una captura, una pequeña vista previa aparece en la esquina de la pantalla.",
+ "dismiss_label": "Ocultar la miniatura tras",
+ "dismiss_hint": "Desaparece sola a los {0} segundos.",
+ "dismiss_seconds": "Segundos",
+ "seconds_unit": "s",
+ "section_after": "Tras una captura",
+ "save_dir_label": "Carpeta de guardado",
+ "save_dir_hint": "Donde se guardan las nuevas capturas y grabaciones.",
+ "clipboard_label": "Copiar al portapapeles",
+ "clipboard_hint": "Cada captura nueva está lista para pegar al instante.",
+ "sound_label": "Reproducir el sonido del obturador",
+ "effect_label": "Mostrar la animación",
+ "effect_hint": "Un destello breve y la miniatura vuela a la esquina.",
+ "recording_unavailable": "La grabación de pantalla no está disponible aquí: {0}"
+ }
}
}
diff --git a/jarvis/ui/web/frontend/src/lib/jarvisxApi.test.ts b/jarvis/ui/web/frontend/src/lib/jarvisxApi.test.ts
new file mode 100644
index 0000000000..9da90e4fc2
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/jarvisxApi.test.ts
@@ -0,0 +1,113 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+import {
+ clampDismissSeconds,
+ copyJarvisXItem,
+ errorDetail,
+ fetchJarvisXItems,
+ formatDuration,
+ isJarvisXEvent,
+ jarvisXEditorUrl,
+ normalizeRecordStatus,
+ saveJarvisXEdited,
+ saveJarvisXSettings,
+ withVersion,
+ type JarvisXItem,
+} from "@/lib/jarvisxApi";
+
+function json(body: unknown, status = 200): Response {
+ return { ok: status < 400, status, json: async () => body } as Response;
+}
+
+function item(id: string, created_at: string): JarvisXItem {
+ return {
+ id,
+ kind: "image",
+ mode: "region",
+ created_at,
+ width: 10,
+ height: 10,
+ duration_s: null,
+ filename: `${id}.png`,
+ url: `/api/jarvisx/items/${id}/file`,
+ thumb_url: `/api/jarvisx/items/${id}/thumb`,
+ edited_url: null,
+ };
+}
+
+describe("jarvisxApi", () => {
+ let fetchMock: ReturnType<typeof vi.fn>;
+
+ beforeEach(() => {
+ fetchMock = vi.fn();
+ vi.stubGlobal("fetch", fetchMock);
+ });
+
+ afterEach(() => vi.unstubAllGlobals());
+
+ it("lists items newest first", async () => {
+ fetchMock.mockResolvedValue(
+ json({ items: [item("old", "2026-09-01T10:00:00Z"), item("new", "2026-09-29T10:00:00Z")] }),
+ );
+ const items = await fetchJarvisXItems(50);
+ expect(fetchMock).toHaveBeenCalledWith("/api/jarvisx/items?limit=50", undefined);
+ expect(items.map((i) => i.id)).toEqual(["new", "old"]);
+ });
+
+ it("sends a partial settings patch and surfaces a 422 reason", async () => {
+ fetchMock.mockResolvedValue(json({ detail: "Ctrl+Alt+Del is reserved by the system." }, 422));
+ await expect(saveJarvisXSettings({ hotkeys: { region: "ctrl+alt+delete" } })).rejects.toThrow(
+ "Ctrl+Alt+Del is reserved by the system.",
+ );
+ const [url, init] = fetchMock.mock.calls[0];
+ expect(url).toBe("/api/jarvisx/settings");
+ expect(init.method).toBe("PUT");
+ expect(JSON.parse(init.body)).toEqual({ hotkeys: { region: "ctrl+alt+delete" } });
+ });
+
+ it("uploads the edited picture as a raw PNG body", async () => {
+ fetchMock.mockResolvedValue(json(item("a", "2026-09-29T10:00:00Z")));
+ const png = new Blob(["x"], { type: "image/png" });
+ await saveJarvisXEdited("a b", png);
+ const [url, init] = fetchMock.mock.calls[0];
+ expect(url).toBe("/api/jarvisx/items/a%20b/edited");
+ expect(init.method).toBe("PUT");
+ expect(init.headers["content-type"]).toBe("image/png");
+ expect(init.body).toBe(png);
+ });
+
+ it("asks the backend to copy the edited or the original version", async () => {
+ fetchMock.mockResolvedValue(json({ ok: true, message: "" }));
+ await copyJarvisXItem("a", true);
+ expect(JSON.parse(fetchMock.mock.calls[0][1].body)).toEqual({ edited: true });
+ });
+
+ it("reads FastAPI's string and list error bodies", () => {
+ expect(errorDetail({ detail: "nope" })).toBe("nope");
+ expect(errorDetail({ detail: [{ msg: "bad hotkey" }, { msg: "again" }] })).toBe("bad hotkey again");
+ expect(errorDetail(null)).toBe("");
+ });
+
+ it("normalises recording status from a body or an event payload", () => {
+ expect(normalizeRecordStatus({ recording: true, mode: "region", elapsed_s: 4.2 })).toEqual({
+ recording: true,
+ mode: "region",
+ elapsed_s: 4.2,
+ });
+ expect(normalizeRecordStatus({ mode: "sideways" })).toEqual({ recording: false, mode: null, elapsed_s: 0 });
+ expect(normalizeRecordStatus(undefined)).toEqual({ recording: false, mode: null, elapsed_s: 0 });
+ });
+
+ it("formats durations, clamps the dismiss delay and builds URLs", () => {
+ expect(formatDuration(83.4)).toBe("1:23");
+ expect(formatDuration(3723)).toBe("1:02:03");
+ expect(formatDuration(null)).toBe("");
+ expect(clampDismissSeconds(1)).toBe(3);
+ expect(clampDismissSeconds(1000)).toBe(300);
+ expect(clampDismissSeconds(Number.NaN)).toBe(30);
+ expect(jarvisXEditorUrl("a/b")).toBe("/?view=jarvisx-editor&solo=1&item=a%2Fb");
+ expect(withVersion("/x?y=1", 2)).toBe("/x?y=1&v=2");
+ expect(isJarvisXEvent("JarvisXItemCreated")).toBe(true);
+ expect(isJarvisXEvent("AppshotTaken")).toBe(false);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/lib/jarvisxApi.ts b/jarvis/ui/web/frontend/src/lib/jarvisxApi.ts
new file mode 100644
index 0000000000..86eca0a228
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/jarvisxApi.ts
@@ -0,0 +1,248 @@
+/**
+ * Client for `/api/jarvisx` — the screenshot and screen-recording tool.
+ *
+ * Jarvis X never involves the assistant: it captures, keeps a library of the
+ * captures on disk, and hands an image to the annotation editor. Every call
+ * here is a thin, typed wrapper over one route; a failed call throws an Error
+ * carrying the backend's own `detail` when it sent one.
+ */
+
+export type JarvisXKind = "image" | "video";
+export type JarvisXMode = "region" | "window" | "fullscreen";
+export type JarvisXRecordMode = "region" | "fullscreen";
+
+/** The six global shortcuts, in the order the settings page lists them. */
+export const JARVISX_HOTKEY_ACTIONS = [
+ "region",
+ "window",
+ "fullscreen",
+ "record_region",
+ "record_fullscreen",
+ "stop_recording",
+] as const;
+
+export type JarvisXHotkeyAction = (typeof JARVISX_HOTKEY_ACTIONS)[number];
+
+export interface JarvisXShortcutStatus {
+ hotkey: string;
+ armed: boolean;
+ detail: string;
+}
+
+export interface JarvisXSettings {
+ enabled: boolean;
+ hotkeys: Record<JarvisXHotkeyAction, string>;
+ thumbnail_persist: boolean;
+ thumbnail_dismiss_s: number;
+ save_dir: string;
+ copy_to_clipboard: boolean;
+ sound: boolean;
+ effect: boolean;
+ recording_available: boolean;
+ recording_detail: string;
+ shortcuts: Partial<Record<string, JarvisXShortcutStatus>>;
+}
+
+export type JarvisXSettingsPatch = Partial<
+ Pick<
+ JarvisXSettings,
+ | "enabled"
+ | "thumbnail_persist"
+ | "thumbnail_dismiss_s"
+ | "save_dir"
+ | "copy_to_clipboard"
+ | "sound"
+ | "effect"
+ >
+> & { hotkeys?: Partial<Record<JarvisXHotkeyAction, string>> };
+
+export interface JarvisXItem {
+ id: string;
+ kind: JarvisXKind;
+ mode: JarvisXMode;
+ created_at: string;
+ width: number;
+ height: number;
+ duration_s: number | null;
+ filename: string;
+ url: string;
+ thumb_url: string;
+ edited_url: string | null;
+}
+
+export interface JarvisXRecordStatus {
+ recording: boolean;
+ mode: JarvisXRecordMode | null;
+ elapsed_s: number;
+}
+
+export interface JarvisXActionResult {
+ ok: boolean;
+ message?: string;
+}
+
+/** The thumbnail-dismiss range the settings page offers, in seconds. */
+export const DISMISS_MIN_S = 3;
+export const DISMISS_MAX_S = 300;
+export const DISMISS_DEFAULT_S = 30;
+
+/** Clamp a typed or dragged dismiss delay into the offered range. */
+export function clampDismissSeconds(value: number): number {
+ if (!Number.isFinite(value)) return DISMISS_DEFAULT_S;
+ return Math.min(DISMISS_MAX_S, Math.max(DISMISS_MIN_S, Math.round(value)));
+}
+
+/** The websocket event names the backend publishes for this feature. */
+export const JARVISX_EVENTS = {
+ created: "JarvisXItemCreated",
+ updated: "JarvisXItemUpdated",
+ deleted: "JarvisXItemDeleted",
+ recording: "JarvisXRecordingChanged",
+} as const;
+
+export function isJarvisXEvent(name: string): boolean {
+ return (Object.values(JARVISX_EVENTS) as string[]).includes(name);
+}
+
+const BASE = "/api/jarvisx";
+
+/** The readable reason out of a FastAPI error body (string or 422 list). */
+export function errorDetail(body: unknown): string {
+ if (!body || typeof body !== "object" || !("detail" in body)) return "";
+ const detail = (body as { detail?: unknown }).detail;
+ if (typeof detail === "string") return detail;
+ if (Array.isArray(detail)) {
+ return detail
+ .map((entry) =>
+ entry && typeof entry === "object" && "msg" in entry ? String((entry as { msg: unknown }).msg) : "",
+ )
+ .filter(Boolean)
+ .join(" ");
+ }
+ if (detail && typeof detail === "object" && "message" in detail) {
+ return String((detail as { message: unknown }).message);
+ }
+ return "";
+}
+
+async function request<T>(url: string, init?: RequestInit): Promise<T> {
+ const response = await fetch(url, init);
+ const body = (await response.json().catch(() => null)) as unknown;
+ if (!response.ok) {
+ throw new Error(errorDetail(body) || `Jarvis X request failed (${response.status}).`);
+ }
+ return body as T;
+}
+
+function jsonInit(method: string, payload?: unknown): RequestInit {
+ return payload === undefined
+ ? { method }
+ : { method, headers: { "content-type": "application/json" }, body: JSON.stringify(payload) };
+}
+
+const itemPath = (id: string) => `${BASE}/items/${encodeURIComponent(id)}`;
+
+export function fetchJarvisXSettings(): Promise<JarvisXSettings> {
+ return request<JarvisXSettings>(`${BASE}/settings`);
+}
+
+export function saveJarvisXSettings(patch: JarvisXSettingsPatch): Promise<JarvisXSettings> {
+ return request<JarvisXSettings>(`${BASE}/settings`, jsonInit("PUT", patch));
+}
+
+export async function fetchJarvisXItems(limit = 100): Promise<JarvisXItem[]> {
+ const body = await request<{ items?: JarvisXItem[] }>(`${BASE}/items?limit=${limit}`);
+ return sortNewestFirst(Array.isArray(body.items) ? body.items : []);
+}
+
+export function fetchJarvisXItem(id: string): Promise<JarvisXItem> {
+ return request<JarvisXItem>(itemPath(id));
+}
+
+/** Upload the flattened annotation result as the item's edited version. */
+export function saveJarvisXEdited(id: string, png: Blob): Promise<JarvisXItem> {
+ return request<JarvisXItem>(`${itemPath(id)}/edited`, {
+ method: "PUT",
+ headers: { "content-type": "image/png" },
+ body: png,
+ });
+}
+
+export function copyJarvisXItem(id: string, edited: boolean): Promise<JarvisXActionResult> {
+ return request<JarvisXActionResult>(`${itemPath(id)}/copy`, jsonInit("POST", { edited }));
+}
+
+export function revealJarvisXItem(id: string): Promise<JarvisXActionResult> {
+ return request<JarvisXActionResult>(`${itemPath(id)}/reveal`, jsonInit("POST"));
+}
+
+export function deleteJarvisXItem(id: string): Promise<JarvisXActionResult> {
+ return request<JarvisXActionResult>(itemPath(id), jsonInit("DELETE"));
+}
+
+export function openJarvisXEditor(id: string): Promise<JarvisXActionResult> {
+ return request<JarvisXActionResult>(`${itemPath(id)}/open-editor`, jsonInit("POST"));
+}
+
+export function captureJarvisX(
+ mode: JarvisXMode,
+ delaySeconds?: number,
+): Promise<{ ok: boolean; item?: JarvisXItem | null; message?: string }> {
+ const payload = delaySeconds ? { mode, delay_s: delaySeconds } : { mode };
+ return request(`${BASE}/capture`, jsonInit("POST", payload));
+}
+
+export function startJarvisXRecording(mode: JarvisXRecordMode): Promise<JarvisXActionResult> {
+ return request<JarvisXActionResult>(`${BASE}/record/start`, jsonInit("POST", { mode }));
+}
+
+export function stopJarvisXRecording(): Promise<JarvisXActionResult> {
+ return request<JarvisXActionResult>(`${BASE}/record/stop`, jsonInit("POST"));
+}
+
+export async function fetchJarvisXRecordStatus(): Promise<JarvisXRecordStatus> {
+ const body = await request<Partial<JarvisXRecordStatus>>(`${BASE}/record/status`);
+ return normalizeRecordStatus(body);
+}
+
+/** Coerce a status body or a JarvisXRecordingChanged payload into one shape. */
+export function normalizeRecordStatus(raw: unknown): JarvisXRecordStatus {
+ const body = (raw && typeof raw === "object" ? raw : {}) as Record<string, unknown>;
+ const mode = body.mode === "region" || body.mode === "fullscreen" ? body.mode : null;
+ const elapsed = typeof body.elapsed_s === "number" && Number.isFinite(body.elapsed_s) ? body.elapsed_s : 0;
+ return { recording: body.recording === true, mode, elapsed_s: Math.max(0, elapsed) };
+}
+
+function createdMs(item: JarvisXItem): number {
+ const ms = Date.parse(item.created_at);
+ return Number.isNaN(ms) ? 0 : ms;
+}
+
+export function sortNewestFirst(items: readonly JarvisXItem[]): JarvisXItem[] {
+ return [...items].sort((a, b) => createdMs(b) - createdMs(a));
+}
+
+/** The editor window's own URL for one capture. */
+export function jarvisXEditorUrl(id: string): string {
+ return `/?view=jarvisx-editor&solo=1&item=${encodeURIComponent(id)}`;
+}
+
+/** "83.4" → "1:23"; an hour or longer → "1:02:03". */
+export function formatDuration(seconds: number | null | undefined): string {
+ if (seconds === null || seconds === undefined || !Number.isFinite(seconds) || seconds < 0) return "";
+ const total = Math.round(seconds);
+ const h = Math.floor(total / 3600);
+ const m = Math.floor((total % 3600) / 60);
+ const s = total % 60;
+ const pad = (n: number) => String(n).padStart(2, "0");
+ return h > 0 ? `${h}:${pad(m)}:${pad(s)}` : `${m}:${pad(s)}`;
+}
+
+/**
+ * Cache-bust a media URL with a version token. The edited image keeps its URL
+ * across saves, so without this an `<img>` shows the previous edit.
+ */
+export function withVersion(url: string, version: string | number): string {
+ const sep = url.includes("?") ? "&" : "?";
+ return `${url}${sep}v=${encodeURIComponent(String(version))}`;
+}
diff --git a/jarvis/ui/web/frontend/src/lib/jarvisxEditorModel.test.ts b/jarvis/ui/web/frontend/src/lib/jarvisxEditorModel.test.ts
new file mode 100644
index 0000000000..0d6c576a52
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/jarvisxEditorModel.test.ts
@@ -0,0 +1,203 @@
+import { describe, expect, it } from "vitest";
+
+import {
+ EMPTY_DOC,
+ addShape,
+ canRedo,
+ canUndo,
+ commit,
+ createHistory,
+ cropFromDrag,
+ deleteShape,
+ exportSize,
+ finishGesture,
+ fitScale,
+ hitHandle,
+ hitTest,
+ imageToView,
+ isMeaningful,
+ moveShape,
+ nextCounterNumber,
+ redo,
+ replacePresent,
+ resizeShape,
+ restyleShape,
+ setCrop,
+ stepZoom,
+ toolForKey,
+ undo,
+ updateShape,
+ viewToImage,
+ visibleArea,
+ type BoxShape,
+ type CounterShape,
+ type SegmentShape,
+ type Shape,
+} from "@/lib/jarvisxEditorModel";
+
+const style = { color: "#ff3b30", width: 4, shadow: false };
+
+function box(id: string, x: number, y: number, w: number, h: number, kind: BoxShape["kind"] = "rect"): BoxShape {
+ return { id, kind, x, y, w, h, ...style };
+}
+
+function counter(id: string, n: number): CounterShape {
+ return { id, kind: "counter", x: 10 * n, y: 10, n, ...style };
+}
+
+describe("shape edits", () => {
+ it("adds, moves, resizes and deletes shapes without mutating the old document", () => {
+ const doc0 = EMPTY_DOC;
+ const doc1 = addShape(doc0, box("a", 10, 10, 100, 50));
+ expect(doc0.shapes).toHaveLength(0);
+ expect(doc1.shapes).toHaveLength(1);
+
+ const doc2 = updateShape(doc1, "a", (s) => moveShape(s, 5, -5));
+ expect(doc2.shapes[0]).toMatchObject({ x: 15, y: 5, w: 100, h: 50 });
+ expect(doc1.shapes[0]).toMatchObject({ x: 10, y: 10 });
+
+ const doc3 = updateShape(doc2, "a", (s) => resizeShape(s, "se", { x: 215, y: 105 }));
+ expect(doc3.shapes[0]).toMatchObject({ x: 15, y: 5, w: 200, h: 100 });
+
+ expect(deleteShape(doc3, "a").shapes).toHaveLength(0);
+ // Deleting something that is not there changes nothing.
+ expect(deleteShape(doc3, "missing")).toBe(doc3);
+ });
+
+ it("normalises a box dragged through its opposite corner", () => {
+ const flipped = resizeShape(box("a", 100, 100, 50, 50), "nw", { x: 200, y: 180 });
+ expect(flipped).toMatchObject({ x: 150, y: 150, w: 50, h: 30 });
+ });
+
+ it("moves both ends of a segment and drags one end by its handle", () => {
+ const arrow: SegmentShape = { id: "s", kind: "arrow", x1: 0, y1: 0, x2: 100, y2: 0, ...style };
+ expect(moveShape(arrow, 10, 20)).toMatchObject({ x1: 10, y1: 20, x2: 110, y2: 20 });
+ expect(resizeShape(arrow, "end", { x: 50, y: 50 })).toMatchObject({ x1: 0, y1: 0, x2: 50, y2: 50 });
+ expect(hitHandle(arrow, { x: 99, y: 1 }, 4)).toBe("end");
+ });
+
+ it("moves every point of a freehand stroke", () => {
+ const pen: Shape = { id: "p", kind: "pen", points: [{ x: 0, y: 0 }, { x: 5, y: 5 }], ...style };
+ expect(moveShape(pen, 1, 2)).toMatchObject({ points: [{ x: 1, y: 2 }, { x: 6, y: 7 }] });
+ });
+
+ it("restyles only what a shape uses", () => {
+ const blur = box("b", 0, 0, 10, 10, "blur");
+ expect(restyleShape(blur, { color: "#ffffff" }).color).toBe(style.color);
+ const text: Shape = { id: "t", kind: "text", x: 0, y: 0, text: "hi", fontSize: 16, ...style };
+ expect(restyleShape(text, { fontSize: 48 })).toMatchObject({ fontSize: 48 });
+ expect(restyleShape(box("r", 0, 0, 5, 5), { fontSize: 48 })).not.toHaveProperty("fontSize");
+ });
+
+ it("drops a click that never became a shape", () => {
+ expect(isMeaningful(box("a", 0, 0, 1, 1))).toBe(false);
+ expect(isMeaningful(box("a", 0, 0, 20, 20))).toBe(true);
+ expect(isMeaningful({ id: "t", kind: "text", x: 0, y: 0, text: " ", fontSize: 16, ...style })).toBe(false);
+ });
+});
+
+describe("hit testing", () => {
+ const shapes: Shape[] = [box("under", 0, 0, 200, 200, "highlight"), box("outline", 50, 50, 100, 100)];
+
+ it("grabs a hollow rectangle by its outline only", () => {
+ expect(hitTest(shapes, { x: 50, y: 100 }, 3)).toBe("outline");
+ // Inside the hollow rect, the filled highlight underneath is what is hit.
+ expect(hitTest(shapes, { x: 100, y: 100 }, 3)).toBe("under");
+ expect(hitTest(shapes, { x: 400, y: 400 }, 3)).toBeNull();
+ });
+
+ it("returns the topmost shape", () => {
+ const stacked = [box("a", 0, 0, 50, 50, "highlight"), box("b", 0, 0, 50, 50, "highlight")];
+ expect(hitTest(stacked, { x: 25, y: 25 }, 2)).toBe("b");
+ });
+});
+
+describe("counters", () => {
+ it("numbers the next badge one past the highest", () => {
+ expect(nextCounterNumber([])).toBe(1);
+ expect(nextCounterNumber([counter("a", 1), counter("b", 2)])).toBe(3);
+ // A deleted middle badge does not get its number reused while a higher one exists.
+ expect(nextCounterNumber([counter("a", 1), counter("c", 3)])).toBe(4);
+ });
+});
+
+describe("history", () => {
+ it("undoes and redoes whole edits", () => {
+ let h = createHistory();
+ h = commit(h, addShape(h.present, box("a", 0, 0, 10, 10)));
+ h = commit(h, addShape(h.present, box("b", 0, 0, 10, 10)));
+ expect(h.present.shapes.map((s) => s.id)).toEqual(["a", "b"]);
+ h = undo(h);
+ expect(h.present.shapes.map((s) => s.id)).toEqual(["a"]);
+ expect(canRedo(h)).toBe(true);
+ h = redo(h);
+ expect(h.present.shapes.map((s) => s.id)).toEqual(["a", "b"]);
+ h = undo(undo(h));
+ expect(h.present.shapes).toHaveLength(0);
+ expect(canUndo(h)).toBe(false);
+ // Undo at the start is a no-op.
+ expect(undo(h)).toBe(h);
+ });
+
+ it("records a drag as ONE step however many frames it painted", () => {
+ let h = createHistory(addShape(EMPTY_DOC, box("a", 0, 0, 10, 10)));
+ const before = h.present;
+ for (let i = 1; i <= 5; i += 1) {
+ h = replacePresent(h, updateShape(h.present, "a", (s) => moveShape(s, 1, 0)));
+ }
+ expect(h.past).toHaveLength(0);
+ h = finishGesture(h, before);
+ expect(h.past).toHaveLength(1);
+ expect(h.present.shapes[0]).toMatchObject({ x: 5 });
+ expect(undo(h).present.shapes[0]).toMatchObject({ x: 0 });
+ });
+
+ it("a new edit clears the redo stack", () => {
+ let h = createHistory();
+ h = commit(h, addShape(h.present, box("a", 0, 0, 10, 10)));
+ h = undo(h);
+ h = commit(h, addShape(h.present, box("b", 0, 0, 10, 10)));
+ expect(canRedo(h)).toBe(false);
+ });
+});
+
+describe("crop, zoom and export math", () => {
+ it("clamps a crop to the image and rounds it to whole pixels", () => {
+ expect(cropFromDrag({ x: -20, y: 10.4 }, { x: 50.6, y: 500 }, 100, 80)).toEqual({ x: 0, y: 10, w: 51, h: 70 });
+ // Dragged backwards: same rect.
+ expect(cropFromDrag({ x: 50.6, y: 500 }, { x: -20, y: 10.4 }, 100, 80)).toEqual({ x: 0, y: 10, w: 51, h: 70 });
+ // Too small to be a crop.
+ expect(cropFromDrag({ x: 10, y: 10 }, { x: 12, y: 12 }, 100, 80)).toBeNull();
+ });
+
+ it("exports at native resolution, cropped or not, whatever the zoom", () => {
+ expect(exportSize(EMPTY_DOC, 2880, 1800)).toEqual({ width: 2880, height: 1800 });
+ const cropped = setCrop(EMPTY_DOC, { x: 100, y: 50, w: 640, h: 360 });
+ expect(exportSize(cropped, 2880, 1800)).toEqual({ width: 640, height: 360 });
+ expect(visibleArea(cropped, 2880, 1800)).toEqual({ x: 100, y: 50, w: 640, h: 360 });
+ });
+
+ it("maps view pixels to image pixels and back through zoom and crop", () => {
+ const area = { x: 100, y: 50, w: 640, h: 360 };
+ const img = viewToImage({ x: 32, y: 18 }, 0.5, area);
+ expect(img).toEqual({ x: 164, y: 86 });
+ expect(imageToView(img, 0.5, area)).toEqual({ x: 32, y: 18 });
+ });
+
+ it("fits content into the stage without ever enlarging it", () => {
+ expect(fitScale(1000, 800, 2000, 1000, 0)).toBe(0.5);
+ expect(fitScale(1000, 800, 200, 100, 0)).toBe(1);
+ });
+
+ it("steps zoom through the fixed ladder", () => {
+ expect(stepZoom(1, 1)).toBe(1.5);
+ expect(stepZoom(1, -1)).toBe(0.75);
+ expect(stepZoom(0.6, 1)).toBe(0.75);
+ });
+
+ it("maps a tool key to its tool", () => {
+ expect(toolForKey("r")).toBe("rect");
+ expect(toolForKey("N")).toBe("counter");
+ expect(toolForKey("q")).toBeNull();
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/lib/jarvisxEditorModel.ts b/jarvis/ui/web/frontend/src/lib/jarvisxEditorModel.ts
new file mode 100644
index 0000000000..c039dcea72
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/jarvisxEditorModel.ts
@@ -0,0 +1,607 @@
+/**
+ * The Jarvis X annotation editor's document model — pure, no DOM.
+ *
+ * Every shape lives in IMAGE pixel coordinates (the capture's native
+ * resolution), so the on-screen zoom never changes what is exported. The
+ * document is a list of shapes plus an optional crop; the editor keeps an
+ * undo history of whole documents, which stays cheap because shapes are small
+ * immutable records and unchanged ones are shared between snapshots.
+ *
+ * Kept free of React and canvas so the rules — hit testing, resizing, counter
+ * numbering, crop and export math, undo — are unit-testable on their own.
+ */
+
+export interface Point {
+ x: number;
+ y: number;
+}
+
+export interface Rect {
+ x: number;
+ y: number;
+ w: number;
+ h: number;
+}
+
+export type Tool =
+ | "select"
+ | "rect"
+ | "highlight"
+ | "ellipse"
+ | "arrow"
+ | "line"
+ | "pen"
+ | "marker"
+ | "text"
+ | "counter"
+ | "blur"
+ | "crop";
+
+interface ShapeBase {
+ id: string;
+ color: string;
+ /** Stroke width in image pixels. */
+ width: number;
+ shadow: boolean;
+}
+
+export interface BoxShape extends ShapeBase {
+ kind: "rect" | "highlight" | "ellipse" | "blur";
+ x: number;
+ y: number;
+ w: number;
+ h: number;
+}
+
+export interface SegmentShape extends ShapeBase {
+ kind: "arrow" | "line";
+ x1: number;
+ y1: number;
+ x2: number;
+ y2: number;
+}
+
+export interface StrokeShape extends ShapeBase {
+ kind: "pen" | "marker";
+ points: Point[];
+}
+
+export interface TextShape extends ShapeBase {
+ kind: "text";
+ x: number;
+ y: number;
+ text: string;
+ fontSize: number;
+}
+
+export interface CounterShape extends ShapeBase {
+ kind: "counter";
+ x: number;
+ y: number;
+ n: number;
+}
+
+export type Shape = BoxShape | SegmentShape | StrokeShape | TextShape | CounterShape;
+export type ShapeKind = Shape["kind"];
+
+export interface EditorDoc {
+ shapes: Shape[];
+ /** The kept part of the image, in image pixels; null keeps all of it. */
+ crop: Rect | null;
+}
+
+export const EMPTY_DOC: EditorDoc = { shapes: [], crop: null };
+
+/** Tools that draw a shape (select and crop do not). */
+export const DRAWING_TOOLS: readonly Tool[] = [
+ "rect",
+ "highlight",
+ "ellipse",
+ "arrow",
+ "line",
+ "pen",
+ "marker",
+ "text",
+ "counter",
+ "blur",
+];
+
+/** One key per tool, shown in the toolbar tooltips. */
+export const TOOL_SHORTCUTS: Record<Tool, string> = {
+ select: "V",
+ rect: "R",
+ highlight: "H",
+ ellipse: "O",
+ arrow: "A",
+ line: "L",
+ pen: "P",
+ marker: "M",
+ text: "T",
+ counter: "N",
+ blur: "B",
+ crop: "C",
+};
+
+export function toolForKey(key: string): Tool | null {
+ const upper = key.toUpperCase();
+ const hit = (Object.keys(TOOL_SHORTCUTS) as Tool[]).find((tool) => TOOL_SHORTCUTS[tool] === upper);
+ return hit ?? null;
+}
+
+/** The annotation palette. Content colours drawn INTO the image, not theme ink. */
+export const PALETTE = [
+ "#ff3b30",
+ "#ff9500",
+ "#ffcc00",
+ "#34c759",
+ "#0a84ff",
+ "#af52de",
+ "#ffffff",
+ "#1c1c1e",
+] as const;
+
+export const DEFAULT_COLOR = PALETTE[0];
+export const STROKE_WIDTHS = [2, 4, 6, 10] as const;
+export const DEFAULT_STROKE = 4;
+export const FONT_SIZES = [16, 24, 32, 48, 72] as const;
+export const DEFAULT_FONT_SIZE = 32;
+/** Opacity of the translucent marker tools. */
+export const HIGHLIGHT_ALPHA = 0.35;
+/** Smallest box a drag must span to become a shape, in image pixels. */
+export const MIN_DRAG = 3;
+
+let idCounter = 0;
+export function newShapeId(): string {
+ idCounter += 1;
+ return `s${Date.now().toString(36)}${idCounter.toString(36)}`;
+}
+
+// ---------------------------------------------------------------------------
+// Geometry
+// ---------------------------------------------------------------------------
+
+/** A rect with non-negative width and height from any two corners. */
+export function rectFromPoints(a: Point, b: Point): Rect {
+ return {
+ x: Math.min(a.x, b.x),
+ y: Math.min(a.y, b.y),
+ w: Math.abs(b.x - a.x),
+ h: Math.abs(b.y - a.y),
+ };
+}
+
+export function normalizeRect(r: Rect): Rect {
+ return rectFromPoints({ x: r.x, y: r.y }, { x: r.x + r.w, y: r.y + r.h });
+}
+
+/** Intersect a rect with the image so a crop can never reach outside it. */
+export function clampRect(r: Rect, width: number, height: number): Rect {
+ const n = normalizeRect(r);
+ const x1 = Math.max(0, Math.min(width, n.x));
+ const y1 = Math.max(0, Math.min(height, n.y));
+ const x2 = Math.max(0, Math.min(width, n.x + n.w));
+ const y2 = Math.max(0, Math.min(height, n.y + n.h));
+ return { x: x1, y: y1, w: x2 - x1, h: y2 - y1 };
+}
+
+/** Whole-pixel crop inside the image, or null when it is too small to keep. */
+export function cropFromDrag(a: Point, b: Point, width: number, height: number): Rect | null {
+ const r = clampRect(rectFromPoints(a, b), width, height);
+ const x = Math.round(r.x);
+ const y = Math.round(r.y);
+ const w = Math.round(r.x + r.w) - x;
+ const h = Math.round(r.y + r.h) - y;
+ if (w < MIN_DRAG * 2 || h < MIN_DRAG * 2) return null;
+ return { x, y, w, h };
+}
+
+/** The part of the image the document shows and exports. */
+export function visibleArea(doc: EditorDoc, width: number, height: number): Rect {
+ return doc.crop ?? { x: 0, y: 0, w: width, h: height };
+}
+
+/** Exported bitmap size — always native resolution, never the view zoom. */
+export function exportSize(doc: EditorDoc, width: number, height: number): { width: number; height: number } {
+ const area = visibleArea(doc, width, height);
+ return { width: Math.max(1, Math.round(area.w)), height: Math.max(1, Math.round(area.h)) };
+}
+
+/** Scale that fits content into a box with a margin; never enlarges past 1. */
+export function fitScale(
+ boxW: number,
+ boxH: number,
+ contentW: number,
+ contentH: number,
+ margin = 24,
+): number {
+ if (contentW <= 0 || contentH <= 0) return 1;
+ const w = Math.max(1, boxW - margin * 2);
+ const h = Math.max(1, boxH - margin * 2);
+ return Math.min(1, w / contentW, h / contentH);
+}
+
+export const MIN_ZOOM = 0.05;
+export const MAX_ZOOM = 8;
+const ZOOM_STEPS = [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 4, 6, 8];
+
+export function clampZoom(z: number): number {
+ return Math.min(MAX_ZOOM, Math.max(MIN_ZOOM, z));
+}
+
+/** Next zoom step in a direction, from wherever the zoom is now. */
+export function stepZoom(current: number, direction: 1 | -1): number {
+ if (direction > 0) return ZOOM_STEPS.find((z) => z > current + 1e-6) ?? MAX_ZOOM;
+ const lower = [...ZOOM_STEPS].reverse().find((z) => z < current - 1e-6);
+ return lower ?? Math.min(current, ZOOM_STEPS[0]);
+}
+
+/** View (CSS pixel, relative to the canvas) → image pixels. */
+export function viewToImage(p: Point, scale: number, area: Rect): Point {
+ return { x: p.x / scale + area.x, y: p.y / scale + area.y };
+}
+
+/** Image pixels → view (CSS pixel, relative to the canvas). */
+export function imageToView(p: Point, scale: number, area: Rect): Point {
+ return { x: (p.x - area.x) * scale, y: (p.y - area.y) * scale };
+}
+
+/** Rough text box: the editor measures real text; this keeps tests DOM-free. */
+export function estimateTextSize(text: string, fontSize: number): { w: number; h: number } {
+ const lines = (text || " ").split("\n");
+ const longest = Math.max(...lines.map((line) => line.length), 1);
+ return { w: longest * fontSize * 0.58, h: lines.length * fontSize * 1.25 };
+}
+
+export function counterRadius(shape: Pick<CounterShape, "width">): number {
+ return 12 + shape.width * 2;
+}
+
+export function shapeBounds(
+ shape: Shape,
+ measure: (s: TextShape) => { w: number; h: number } = (s) => estimateTextSize(s.text, s.fontSize),
+): Rect {
+ switch (shape.kind) {
+ case "rect":
+ case "highlight":
+ case "ellipse":
+ case "blur":
+ return normalizeRect(shape);
+ case "arrow":
+ case "line":
+ return rectFromPoints({ x: shape.x1, y: shape.y1 }, { x: shape.x2, y: shape.y2 });
+ case "pen":
+ case "marker": {
+ if (shape.points.length === 0) return { x: 0, y: 0, w: 0, h: 0 };
+ const xs = shape.points.map((p) => p.x);
+ const ys = shape.points.map((p) => p.y);
+ const x = Math.min(...xs);
+ const y = Math.min(...ys);
+ return { x, y, w: Math.max(...xs) - x, h: Math.max(...ys) - y };
+ }
+ case "text": {
+ const size = measure(shape);
+ return { x: shape.x, y: shape.y, w: size.w, h: size.h };
+ }
+ case "counter": {
+ const r = counterRadius(shape);
+ return { x: shape.x - r, y: shape.y - r, w: r * 2, h: r * 2 };
+ }
+ }
+}
+
+function distanceToSegment(p: Point, a: Point, b: Point): number {
+ const dx = b.x - a.x;
+ const dy = b.y - a.y;
+ const len2 = dx * dx + dy * dy;
+ const t = len2 === 0 ? 0 : Math.max(0, Math.min(1, ((p.x - a.x) * dx + (p.y - a.y) * dy) / len2));
+ return Math.hypot(p.x - (a.x + t * dx), p.y - (a.y + t * dy));
+}
+
+function inside(p: Point, r: Rect, pad = 0): boolean {
+ return p.x >= r.x - pad && p.x <= r.x + r.w + pad && p.y >= r.y - pad && p.y <= r.y + r.h + pad;
+}
+
+/** Does a point (image pixels) touch this shape? `tolerance` in image pixels. */
+export function hitShape(
+ shape: Shape,
+ p: Point,
+ tolerance: number,
+ measure?: (s: TextShape) => { w: number; h: number },
+): boolean {
+ const reach = tolerance + shape.width / 2;
+ switch (shape.kind) {
+ case "highlight":
+ case "blur":
+ case "text":
+ case "counter":
+ // Filled things are grabbed anywhere inside.
+ return inside(p, shapeBounds(shape, measure), tolerance);
+ case "rect": {
+ const r = normalizeRect(shape);
+ if (!inside(p, r, reach)) return false;
+ // Hollow: only the outline, so a click inside selects what is under it.
+ return (
+ Math.abs(p.x - r.x) <= reach ||
+ Math.abs(p.x - (r.x + r.w)) <= reach ||
+ Math.abs(p.y - r.y) <= reach ||
+ Math.abs(p.y - (r.y + r.h)) <= reach
+ );
+ }
+ case "ellipse": {
+ const r = normalizeRect(shape);
+ const rx = r.w / 2;
+ const ry = r.h / 2;
+ if (rx <= 0 || ry <= 0) return false;
+ const cx = r.x + rx;
+ const cy = r.y + ry;
+ const d = Math.hypot((p.x - cx) / rx, (p.y - cy) / ry);
+ // Distance from the outline, approximated in the ellipse's own scale.
+ return Math.abs(d - 1) * Math.min(rx, ry) <= reach;
+ }
+ case "arrow":
+ case "line":
+ return distanceToSegment(p, { x: shape.x1, y: shape.y1 }, { x: shape.x2, y: shape.y2 }) <= reach;
+ case "pen":
+ case "marker": {
+ const pts = shape.points;
+ if (pts.length === 1) return Math.hypot(p.x - pts[0].x, p.y - pts[0].y) <= reach;
+ for (let i = 1; i < pts.length; i += 1) {
+ if (distanceToSegment(p, pts[i - 1], pts[i]) <= reach) return true;
+ }
+ return false;
+ }
+ }
+}
+
+/** Topmost shape under the point, or null. Later shapes paint on top. */
+export function hitTest(
+ shapes: readonly Shape[],
+ p: Point,
+ tolerance: number,
+ measure?: (s: TextShape) => { w: number; h: number },
+): string | null {
+ for (let i = shapes.length - 1; i >= 0; i -= 1) {
+ if (hitShape(shapes[i], p, tolerance, measure)) return shapes[i].id;
+ }
+ return null;
+}
+
+// ---------------------------------------------------------------------------
+// Handles and transforms
+// ---------------------------------------------------------------------------
+
+export type Handle = "nw" | "n" | "ne" | "e" | "se" | "s" | "sw" | "w" | "start" | "end";
+
+/** The resize handles a shape offers, with their image-pixel positions. */
+export function shapeHandles(shape: Shape): { handle: Handle; x: number; y: number }[] {
+ if (shape.kind === "arrow" || shape.kind === "line") {
+ return [
+ { handle: "start", x: shape.x1, y: shape.y1 },
+ { handle: "end", x: shape.x2, y: shape.y2 },
+ ];
+ }
+ if (shape.kind === "rect" || shape.kind === "highlight" || shape.kind === "ellipse" || shape.kind === "blur") {
+ const r = normalizeRect(shape);
+ const mx = r.x + r.w / 2;
+ const my = r.y + r.h / 2;
+ return [
+ { handle: "nw", x: r.x, y: r.y },
+ { handle: "n", x: mx, y: r.y },
+ { handle: "ne", x: r.x + r.w, y: r.y },
+ { handle: "e", x: r.x + r.w, y: my },
+ { handle: "se", x: r.x + r.w, y: r.y + r.h },
+ { handle: "s", x: mx, y: r.y + r.h },
+ { handle: "sw", x: r.x, y: r.y + r.h },
+ { handle: "w", x: r.x, y: my },
+ ];
+ }
+ // Strokes, text and counters move as a whole; their size is a style.
+ return [];
+}
+
+export function hitHandle(shape: Shape, p: Point, tolerance: number): Handle | null {
+ const hit = shapeHandles(shape).find((h) => Math.hypot(p.x - h.x, p.y - h.y) <= tolerance);
+ return hit ? hit.handle : null;
+}
+
+/** Move a shape by an offset. Returns a new shape. */
+export function moveShape<S extends Shape>(shape: S, dx: number, dy: number): S {
+ switch (shape.kind) {
+ case "arrow":
+ case "line":
+ return { ...shape, x1: shape.x1 + dx, y1: shape.y1 + dy, x2: shape.x2 + dx, y2: shape.y2 + dy };
+ case "pen":
+ case "marker":
+ return { ...shape, points: shape.points.map((p) => ({ x: p.x + dx, y: p.y + dy })) };
+ default:
+ return { ...shape, x: (shape as BoxShape).x + dx, y: (shape as BoxShape).y + dy };
+ }
+}
+
+/**
+ * Drag one handle of `original` to `p`. Box shapes may flip through zero —
+ * the result is normalised, so the box stays a valid rect either way.
+ */
+export function resizeShape<S extends Shape>(original: S, handle: Handle, p: Point): S {
+ if (original.kind === "arrow" || original.kind === "line") {
+ if (handle === "start") return { ...original, x1: p.x, y1: p.y };
+ if (handle === "end") return { ...original, x2: p.x, y2: p.y };
+ return original;
+ }
+ if (
+ original.kind !== "rect" &&
+ original.kind !== "highlight" &&
+ original.kind !== "ellipse" &&
+ original.kind !== "blur"
+ ) {
+ return original;
+ }
+ const r = normalizeRect(original);
+ let x1 = r.x;
+ let y1 = r.y;
+ let x2 = r.x + r.w;
+ let y2 = r.y + r.h;
+ if (handle.includes("w")) x1 = p.x;
+ if (handle.includes("e")) x2 = p.x;
+ if (handle.includes("n")) y1 = p.y;
+ if (handle.includes("s")) y2 = p.y;
+ return { ...original, ...rectFromPoints({ x: x1, y: y1 }, { x: x2, y: y2 }) };
+}
+
+// ---------------------------------------------------------------------------
+// Document edits
+// ---------------------------------------------------------------------------
+
+export function addShape(doc: EditorDoc, shape: Shape): EditorDoc {
+ return { ...doc, shapes: [...doc.shapes, shape] };
+}
+
+export function updateShape(doc: EditorDoc, id: string, patch: (shape: Shape) => Shape): EditorDoc {
+ let changed = false;
+ const shapes = doc.shapes.map((shape) => {
+ if (shape.id !== id) return shape;
+ const next = patch(shape);
+ if (next !== shape) changed = true;
+ return next;
+ });
+ return changed ? { ...doc, shapes } : doc;
+}
+
+export function deleteShape(doc: EditorDoc, id: string): EditorDoc {
+ const shapes = doc.shapes.filter((shape) => shape.id !== id);
+ return shapes.length === doc.shapes.length ? doc : { ...doc, shapes };
+}
+
+export function setCrop(doc: EditorDoc, crop: Rect | null): EditorDoc {
+ return { ...doc, crop };
+}
+
+/** The number the next counter badge gets: one past the highest on the page. */
+export function nextCounterNumber(shapes: readonly Shape[]): number {
+ let max = 0;
+ for (const shape of shapes) if (shape.kind === "counter" && shape.n > max) max = shape.n;
+ return max + 1;
+}
+
+/** Apply a style change to a shape, touching only what that kind uses. */
+export function restyleShape(
+ shape: Shape,
+ style: Partial<{ color: string; width: number; shadow: boolean; fontSize: number }>,
+): Shape {
+ const next: Shape = { ...shape };
+ if (style.color !== undefined && shape.kind !== "blur") next.color = style.color;
+ if (style.width !== undefined) next.width = style.width;
+ if (style.shadow !== undefined) next.shadow = style.shadow;
+ if (style.fontSize !== undefined && next.kind === "text") next.fontSize = style.fontSize;
+ return next;
+}
+
+/** Is a freshly drawn shape big enough to keep (a click is not a rectangle)? */
+export function isMeaningful(shape: Shape): boolean {
+ switch (shape.kind) {
+ case "rect":
+ case "highlight":
+ case "ellipse":
+ case "blur":
+ return Math.abs(shape.w) >= MIN_DRAG && Math.abs(shape.h) >= MIN_DRAG;
+ case "arrow":
+ case "line":
+ return Math.hypot(shape.x2 - shape.x1, shape.y2 - shape.y1) >= MIN_DRAG * 2;
+ case "pen":
+ case "marker":
+ return shape.points.length >= 2;
+ case "text":
+ return shape.text.trim().length > 0;
+ case "counter":
+ return true;
+ }
+}
+
+// ---------------------------------------------------------------------------
+// History
+// ---------------------------------------------------------------------------
+
+export interface History {
+ past: EditorDoc[];
+ present: EditorDoc;
+ future: EditorDoc[];
+}
+
+export const HISTORY_LIMIT = 200;
+
+export function createHistory(doc: EditorDoc = EMPTY_DOC): History {
+ return { past: [], present: doc, future: [] };
+}
+
+/** A new undoable state. A no-op edit (same object) records nothing. */
+export function commit(history: History, next: EditorDoc): History {
+ if (next === history.present) return history;
+ const past = [...history.past, history.present];
+ if (past.length > HISTORY_LIMIT) past.splice(0, past.length - HISTORY_LIMIT);
+ return { past, present: next, future: [] };
+}
+
+/** Replace the present without a history entry — the live frames of a drag. */
+export function replacePresent(history: History, next: EditorDoc): History {
+ return next === history.present ? history : { ...history, present: next };
+}
+
+/**
+ * Close a gesture whose live frames went through `replacePresent`: ONE undo
+ * step back to `before`, however many frames the drag painted.
+ */
+export function finishGesture(history: History, before: EditorDoc): History {
+ if (history.present === before) return history;
+ const past = [...history.past, before];
+ if (past.length > HISTORY_LIMIT) past.splice(0, past.length - HISTORY_LIMIT);
+ return { past, present: history.present, future: [] };
+}
+
+export function undo(history: History): History {
+ if (history.past.length === 0) return history;
+ const previous = history.past[history.past.length - 1];
+ return {
+ past: history.past.slice(0, -1),
+ present: previous,
+ future: [history.present, ...history.future],
+ };
+}
+
+export function redo(history: History): History {
+ if (history.future.length === 0) return history;
+ const [next, ...rest] = history.future;
+ return { past: [...history.past, history.present], present: next, future: rest };
+}
+
+export const canUndo = (history: History) => history.past.length > 0;
+export const canRedo = (history: History) => history.future.length > 0;
+
+/** Anything to save — a shape or a crop on the page. */
+export function isDirty(doc: EditorDoc): boolean {
+ return doc.shapes.length > 0 || doc.crop !== null;
+}
+
+// ---------------------------------------------------------------------------
+// Drawing helpers shared by the view and the export
+// ---------------------------------------------------------------------------
+
+/** The two barb points of an arrow head at (x2, y2). */
+export function arrowHead(shape: Pick<SegmentShape, "x1" | "y1" | "x2" | "y2" | "width">): {
+ left: Point;
+ right: Point;
+ length: number;
+} {
+ const angle = Math.atan2(shape.y2 - shape.y1, shape.x2 - shape.x1);
+ const length = Math.max(12, shape.width * 4);
+ const spread = Math.PI / 7;
+ return {
+ left: { x: shape.x2 - length * Math.cos(angle - spread), y: shape.y2 - length * Math.sin(angle - spread) },
+ right: { x: shape.x2 - length * Math.cos(angle + spread), y: shape.y2 - length * Math.sin(angle + spread) },
+ length,
+ };
+}
+
+/** Pixelation block size for a blur region, from the stroke width setting. */
+export function pixelBlock(shape: Pick<BoxShape, "width">): number {
+ return Math.max(6, shape.width * 3);
+}
diff --git a/jarvis/ui/web/frontend/src/lib/jarvisxRender.ts b/jarvis/ui/web/frontend/src/lib/jarvisxRender.ts
new file mode 100644
index 0000000000..ad66529f06
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/jarvisxRender.ts
@@ -0,0 +1,261 @@
+/**
+ * Canvas painting for the Jarvis X editor.
+ *
+ * One painter for both the screen and the export: the caller sets a transform
+ * that maps image pixels onto its canvas (zoomed and high-DPI on screen, 1:1
+ * for the export), then `paintDocument` draws the base image, every blur
+ * region, and the shapes on top. Because the export goes through the same
+ * code, what the user sees is what gets saved.
+ */
+import {
+ HIGHLIGHT_ALPHA,
+ arrowHead,
+ counterRadius,
+ exportSize,
+ pixelBlock,
+ visibleArea,
+ type BoxShape,
+ type EditorDoc,
+ type Shape,
+ type TextShape,
+} from "@/lib/jarvisxEditorModel";
+
+export const TEXT_FONT_FAMILY = '"Inter Variable", Inter, system-ui, sans-serif';
+const TEXT_LINE_HEIGHT = 1.25;
+
+export function textFont(fontSize: number): string {
+ return `600 ${fontSize}px ${TEXT_FONT_FAMILY}`;
+}
+
+let measureCtx: CanvasRenderingContext2D | null = null;
+
+/** Real text box for a text shape, measured with the font it is drawn in. */
+export function measureTextShape(shape: Pick<TextShape, "text" | "fontSize">): { w: number; h: number } {
+ const lines = (shape.text || " ").split("\n");
+ if (!measureCtx && typeof document !== "undefined") {
+ measureCtx = document.createElement("canvas").getContext("2d");
+ }
+ if (!measureCtx) {
+ const longest = Math.max(...lines.map((l) => l.length), 1);
+ return { w: longest * shape.fontSize * 0.58, h: lines.length * shape.fontSize * TEXT_LINE_HEIGHT };
+ }
+ measureCtx.font = textFont(shape.fontSize);
+ const w = Math.max(...lines.map((line) => measureCtx!.measureText(line || " ").width), shape.fontSize * 0.5);
+ return { w, h: lines.length * shape.fontSize * TEXT_LINE_HEIGHT };
+}
+
+/** Black or white, whichever reads on top of `hex`. */
+export function inkOn(hex: string): string {
+ const m = /^#?([0-9a-f]{6})$/i.exec(hex);
+ if (!m) return "#ffffff";
+ const n = parseInt(m[1], 16);
+ const r = (n >> 16) & 255;
+ const g = (n >> 8) & 255;
+ const b = n & 255;
+ const luminance = (0.299 * r + 0.587 * g + 0.114 * b) / 255;
+ return luminance > 0.62 ? "#1c1c1e" : "#ffffff";
+}
+
+function withShadow(ctx: CanvasRenderingContext2D, shape: Shape, draw: () => void) {
+ ctx.save();
+ if (shape.shadow) {
+ ctx.shadowColor = "rgba(0, 0, 0, 0.45)";
+ ctx.shadowBlur = Math.max(6, shape.width * 2.5);
+ ctx.shadowOffsetY = Math.max(2, shape.width * 0.6);
+ }
+ draw();
+ ctx.restore();
+}
+
+function paintBlur(ctx: CanvasRenderingContext2D, image: CanvasImageSource, shape: BoxShape) {
+ const w = Math.round(Math.abs(shape.w));
+ const h = Math.round(Math.abs(shape.h));
+ if (w < 1 || h < 1 || typeof document === "undefined") return;
+ const x = Math.round(Math.min(shape.x, shape.x + shape.w));
+ const y = Math.round(Math.min(shape.y, shape.y + shape.h));
+ const block = pixelBlock(shape);
+ const small = document.createElement("canvas");
+ small.width = Math.max(1, Math.ceil(w / block));
+ small.height = Math.max(1, Math.ceil(h / block));
+ const sctx = small.getContext("2d");
+ if (!sctx) return;
+ sctx.imageSmoothingEnabled = true;
+ sctx.drawImage(image, x, y, w, h, 0, 0, small.width, small.height);
+ ctx.save();
+ ctx.imageSmoothingEnabled = false;
+ ctx.drawImage(small, 0, 0, small.width, small.height, x, y, w, h);
+ ctx.restore();
+}
+
+function strokePath(ctx: CanvasRenderingContext2D, points: { x: number; y: number }[]) {
+ ctx.beginPath();
+ ctx.moveTo(points[0].x, points[0].y);
+ if (points.length === 1) {
+ ctx.lineTo(points[0].x + 0.01, points[0].y);
+ } else {
+ // Midpoint quadratic smoothing: a hand-drawn line without the jaggies.
+ for (let i = 1; i < points.length - 1; i += 1) {
+ const mx = (points[i].x + points[i + 1].x) / 2;
+ const my = (points[i].y + points[i + 1].y) / 2;
+ ctx.quadraticCurveTo(points[i].x, points[i].y, mx, my);
+ }
+ const last = points[points.length - 1];
+ ctx.lineTo(last.x, last.y);
+ }
+ ctx.stroke();
+}
+
+export function paintShape(ctx: CanvasRenderingContext2D, shape: Shape) {
+ ctx.lineCap = "round";
+ ctx.lineJoin = "round";
+ switch (shape.kind) {
+ case "blur":
+ return;
+ case "rect":
+ withShadow(ctx, shape, () => {
+ ctx.strokeStyle = shape.color;
+ ctx.lineWidth = shape.width;
+ ctx.lineJoin = "miter";
+ ctx.strokeRect(shape.x, shape.y, shape.w, shape.h);
+ });
+ return;
+ case "highlight":
+ ctx.save();
+ ctx.globalAlpha = HIGHLIGHT_ALPHA;
+ ctx.fillStyle = shape.color;
+ ctx.fillRect(shape.x, shape.y, shape.w, shape.h);
+ ctx.restore();
+ return;
+ case "ellipse":
+ withShadow(ctx, shape, () => {
+ ctx.strokeStyle = shape.color;
+ ctx.lineWidth = shape.width;
+ ctx.beginPath();
+ ctx.ellipse(
+ shape.x + shape.w / 2,
+ shape.y + shape.h / 2,
+ Math.abs(shape.w / 2),
+ Math.abs(shape.h / 2),
+ 0,
+ 0,
+ Math.PI * 2,
+ );
+ ctx.stroke();
+ });
+ return;
+ case "line":
+ withShadow(ctx, shape, () => {
+ ctx.strokeStyle = shape.color;
+ ctx.lineWidth = shape.width;
+ ctx.beginPath();
+ ctx.moveTo(shape.x1, shape.y1);
+ ctx.lineTo(shape.x2, shape.y2);
+ ctx.stroke();
+ });
+ return;
+ case "arrow":
+ withShadow(ctx, shape, () => {
+ const head = arrowHead(shape);
+ const angle = Math.atan2(shape.y2 - shape.y1, shape.x2 - shape.x1);
+ // Stop the shaft inside the head so the tip stays sharp.
+ const inset = head.length * 0.6;
+ ctx.strokeStyle = shape.color;
+ ctx.fillStyle = shape.color;
+ ctx.lineWidth = shape.width;
+ ctx.beginPath();
+ ctx.moveTo(shape.x1, shape.y1);
+ ctx.lineTo(shape.x2 - inset * Math.cos(angle), shape.y2 - inset * Math.sin(angle));
+ ctx.stroke();
+ ctx.beginPath();
+ ctx.moveTo(shape.x2, shape.y2);
+ ctx.lineTo(head.left.x, head.left.y);
+ ctx.lineTo(head.right.x, head.right.y);
+ ctx.closePath();
+ ctx.fill();
+ });
+ return;
+ case "pen":
+ if (shape.points.length === 0) return;
+ withShadow(ctx, shape, () => {
+ ctx.strokeStyle = shape.color;
+ ctx.lineWidth = shape.width;
+ strokePath(ctx, shape.points);
+ });
+ return;
+ case "marker":
+ if (shape.points.length === 0) return;
+ ctx.save();
+ ctx.globalAlpha = HIGHLIGHT_ALPHA;
+ ctx.strokeStyle = shape.color;
+ ctx.lineWidth = shape.width * 4;
+ ctx.lineCap = "butt";
+ strokePath(ctx, shape.points);
+ ctx.restore();
+ return;
+ case "text":
+ withShadow(ctx, shape, () => {
+ ctx.font = textFont(shape.fontSize);
+ ctx.fillStyle = shape.color;
+ ctx.textBaseline = "top";
+ shape.text.split("\n").forEach((line, i) => {
+ ctx.fillText(line, shape.x, shape.y + i * shape.fontSize * TEXT_LINE_HEIGHT);
+ });
+ });
+ return;
+ case "counter": {
+ const r = counterRadius(shape);
+ withShadow(ctx, shape, () => {
+ ctx.fillStyle = shape.color;
+ ctx.beginPath();
+ ctx.arc(shape.x, shape.y, r, 0, Math.PI * 2);
+ ctx.fill();
+ });
+ ctx.save();
+ ctx.fillStyle = inkOn(shape.color);
+ ctx.font = `700 ${Math.round(r * 1.1)}px ${TEXT_FONT_FAMILY}`;
+ ctx.textAlign = "center";
+ ctx.textBaseline = "middle";
+ ctx.fillText(String(shape.n), shape.x, shape.y + r * 0.05);
+ ctx.restore();
+ return;
+ }
+ }
+}
+
+/**
+ * Paint base image + blur + shapes. The context's transform must already map
+ * image pixels to its own space; `skipId` leaves out the shape being edited
+ * in place (the text field shows it instead).
+ */
+export function paintDocument(
+ ctx: CanvasRenderingContext2D,
+ image: CanvasImageSource,
+ doc: EditorDoc,
+ skipId?: string | null,
+) {
+ ctx.drawImage(image, 0, 0);
+ for (const shape of doc.shapes) {
+ if (shape.kind === "blur" && shape.id !== skipId) paintBlur(ctx, image, shape);
+ }
+ for (const shape of doc.shapes) {
+ if (shape.kind !== "blur" && shape.id !== skipId) paintShape(ctx, shape);
+ }
+}
+
+/** Flatten the document into a PNG at the capture's native resolution. */
+export function exportPng(image: HTMLImageElement, doc: EditorDoc): Promise<Blob> {
+ const natW = image.naturalWidth;
+ const natH = image.naturalHeight;
+ const size = exportSize(doc, natW, natH);
+ const area = visibleArea(doc, natW, natH);
+ const canvas = document.createElement("canvas");
+ canvas.width = size.width;
+ canvas.height = size.height;
+ const ctx = canvas.getContext("2d");
+ if (!ctx) return Promise.reject(new Error("Canvas is unavailable."));
+ ctx.setTransform(1, 0, 0, 1, -Math.round(area.x), -Math.round(area.y));
+ paintDocument(ctx, image, doc);
+ return new Promise((resolve, reject) => {
+ canvas.toBlob((blob) => (blob ? resolve(blob) : reject(new Error("Export failed."))), "image/png");
+ });
+}
diff --git a/jarvis/ui/web/frontend/src/lib/sectionPrefetch.test.ts b/jarvis/ui/web/frontend/src/lib/sectionPrefetch.test.ts
new file mode 100644
index 0000000000..5e7a619834
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/sectionPrefetch.test.ts
@@ -0,0 +1,46 @@
+import { describe, expect, it, vi } from "vitest";
+import { createSectionPrefetcher } from "./sectionPrefetch";
+
+describe("section prefetch", () => {
+ it("does no work at registration and only warms the requested section", async () => {
+ const registry = createSectionPrefetcher();
+ const agents = vi.fn().mockResolvedValue({});
+ const docs = vi.fn().mockResolvedValue({});
+ registry.register(["agents"], agents);
+ registry.register(["docs"], docs);
+ expect(agents).not.toHaveBeenCalled();
+ expect(docs).not.toHaveBeenCalled();
+ await registry.prefetch("agents");
+ await registry.prefetch("chats");
+ expect(agents).toHaveBeenCalledTimes(1);
+ expect(docs).not.toHaveBeenCalled();
+ });
+
+ it("deduplicates in-flight and completed requests across section aliases", async () => {
+ const registry = createSectionPrefetcher();
+ let resolve!: () => void;
+ const loader = vi.fn(() => new Promise<void>((done) => { resolve = done; }));
+ registry.register(["agentic-ide", "agentic-ide-classic"], loader);
+ const first = registry.prefetch("agentic-ide");
+ expect(registry.prefetch("agentic-ide-classic")).toBe(first);
+ await Promise.resolve();
+ resolve();
+ await first;
+ await registry.prefetch("agentic-ide");
+ expect(loader).toHaveBeenCalledTimes(1);
+ });
+
+ it("permits retry after a rejected or synchronously throwing loader", async () => {
+ const registry = createSectionPrefetcher();
+ const loader = vi.fn()
+ .mockRejectedValueOnce(new Error("offline"))
+ .mockImplementationOnce(() => { throw new Error("unavailable"); })
+ .mockResolvedValue({});
+ registry.register(["docs"], loader);
+ await registry.prefetch("docs");
+ await registry.prefetch("docs");
+ await registry.prefetch("docs");
+ await registry.prefetch("docs");
+ expect(loader).toHaveBeenCalledTimes(3);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/lib/sectionPrefetch.ts b/jarvis/ui/web/frontend/src/lib/sectionPrefetch.ts
new file mode 100644
index 0000000000..a1c3c6b351
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/sectionPrefetch.ts
@@ -0,0 +1,33 @@
+import type { SectionId } from "@/store/events";
+
+type Loader = () => Promise<unknown>;
+
+/** Register without executing: unopened windows must not warm every feature. */
+export function createSectionPrefetcher() {
+ const loaders = new Map<SectionId, Loader>();
+ const pending = new Map<Loader, Promise<void>>();
+
+ return {
+ register(sections: readonly SectionId[], loader: Loader) {
+ for (const section of sections) loaders.set(section, loader);
+ },
+ prefetch(section: SectionId): Promise<void> {
+ const loader = loaders.get(section);
+ if (!loader) return Promise.resolve();
+ const existing = pending.get(loader);
+ if (existing) return existing;
+ const request = Promise.resolve().then(loader).then(
+ () => undefined,
+ () => {
+ // Speculation is optional. Navigation retries through React.lazy
+ // and its error boundary; a failed warm-up must not poison it.
+ pending.delete(loader);
+ },
+ );
+ pending.set(loader, request);
+ return request;
+ },
+ };
+}
+
+export const sectionPrefetch = createSectionPrefetcher();
diff --git a/jarvis/ui/web/frontend/src/lib/societyApi.ts b/jarvis/ui/web/frontend/src/lib/societyApi.ts
index 122995c6db..762e744cb7 100644
--- a/jarvis/ui/web/frontend/src/lib/societyApi.ts
+++ b/jarvis/ui/web/frontend/src/lib/societyApi.ts
@@ -52,6 +52,9 @@ export type Checkpoint = (typeof CHECKPOINTS)[number];
export const PERMISSION_CEILINGS = ["safe", "monitor", "ask"] as const;
export type PermissionCeiling = (typeof PERMISSION_CEILINGS)[number];
+export const AGENT_APPROVAL_MODES = ["bypass", "ask", "always_ask"] as const;
+export type AgentApprovalMode = (typeof AGENT_APPROVAL_MODES)[number];
+
export const BROWSER_MODES = ["own", "attach"] as const;
export type BrowserMode = (typeof BROWSER_MODES)[number];
@@ -155,6 +158,8 @@ export interface SocietyAgentRow {
wiki_namespace: string;
knowledge_scope: KnowledgeScope;
permission_ceiling: PermissionCeiling;
+ /** Null only for an existing agent whose chat retains its legacy setting. */
+ approval_mode: AgentApprovalMode | null;
approval_rules: ApprovalRules;
daily_budget_usd: number;
max_concurrent_runs: number;
diff --git a/jarvis/ui/web/frontend/src/views/AgenticIdeView.tsx b/jarvis/ui/web/frontend/src/views/AgenticIdeView.tsx
index 1026e91506..9266c89129 100644
--- a/jarvis/ui/web/frontend/src/views/AgenticIdeView.tsx
+++ b/jarvis/ui/web/frontend/src/views/AgenticIdeView.tsx
@@ -2,7 +2,7 @@ import { useCallback, useEffect, useRef, useState } from "react";
import { FolderPlus, Loader2, X } from "lucide-react";
import { FolderPicker } from "@/components/agentic/FolderPicker";
import { VoiceBubble, storedVoiceBubbleOpen, storeVoiceBubbleOpen } from "@/components/agentic/VoiceBubble";
-import { WorkspaceTerminalGrid } from "@/components/agentic/WorkspaceTerminalGrid";
+import { RetainedWorkspaceGrid } from "@/components/agentic/RetainedWorkspaceGrid";
import { WorkspaceAgentSetup } from "@/components/agentic/WorkspaceAgentSetup";
import { RunOnPicker, storeRunOn, storedRunOn } from "@/components/agentic/RunOnPicker";
import { WorkspaceOptionsDialog } from "@/components/agentic/WorkspaceOptionsDialog";
@@ -138,11 +138,14 @@ export function AgenticIdeView({ onScreen = true }: AgenticIdeViewProps) {
const dialog = document.querySelector<HTMLElement>("[data-ide-dialog]");
if (!dialog) return;
const focusable = () => [...dialog.querySelectorAll<HTMLElement>('button:not([disabled]), input:not([disabled]), select:not([disabled]), [tabindex]:not([tabindex="-1"])')];
- (focusable()[0] ?? dialog).focus();
+ (dialog.querySelector<HTMLElement>("[data-autofocus]") ?? focusable()[0] ?? dialog).focus();
const onKeyDown = (event: KeyboardEvent) => {
if (event.key === "Escape") {
+ // A field that backs out of itself on Escape (the folder picker's path
+ // and new-folder fields) gets the key instead of the whole dialog closing.
+ if (event.target instanceof Element && event.target.closest("[data-escape-local]")) return;
event.preventDefault();
- if (workspaceProject && busy) { event.stopPropagation(); return; }
+ if ((workspaceProject || projectDialog) && busy) { event.stopPropagation(); return; }
setProjectDialog(false); setWorkspaceProject(null); setRenameOpen(false); setAgentPicker(null);
return;
}
@@ -485,7 +488,7 @@ export function AgenticIdeView({ onScreen = true }: AgenticIdeViewProps) {
<main className="min-h-0 flex-1">
<IdeSidePanelFrame markInUse={paneStyle === "minimal"}>
- {session ? <WorkspaceTerminalGrid key={session.id} session={session} onChanged={(next) => setState((current) => current?.session?.id === next.id ? { ...current, session: next } : current)}
+ {session ? <RetainedWorkspaceGrid session={session} onScreen={onScreen} workspaceIds={state.workspaces?.map((workspace) => workspace.id)} onChanged={(next) => setState((current) => current?.session?.id === next.id ? { ...current, session: next } : current)}
onAdd={openAgentPicker} onClose={closeAgent} onSelect={setSelected} selected={selected} maxPanes={maxPanes} fontSize={fontSize} appearance={appearance} disabled={busy}
onMutationStart={beginGridMutation} onMutationEnd={endGridMutation} paneStyle={paneStyle} workspaces={state.workspaces ?? []} />
: <div className="flex h-full flex-col items-center justify-center gap-3 px-6 text-center">
@@ -565,13 +568,34 @@ export function AgenticIdeView({ onScreen = true }: AgenticIdeViewProps) {
</section>
</div>}
- {projectDialog && <div className="fixed inset-0 z-[80] flex items-center justify-center bg-background/75 p-4 backdrop-blur-sm" role="presentation" onMouseDown={(event) => { if (event.target === event.currentTarget) setProjectDialog(false); }}>
- <section data-ide-dialog tabIndex={-1} role="dialog" aria-modal="true" aria-label="Connect project" className="flex max-h-[85vh] w-full max-w-2xl flex-col overflow-hidden rounded-xl border border-border bg-card shadow-2xl">
- <div className="flex items-center justify-between border-b border-border px-5 py-3"><h2 className="text-sm font-semibold">Connect project folder</h2><button type="button" aria-label="Close" onClick={() => setProjectDialog(false)}><X className="h-4 w-4" /></button></div>
- <div className="min-h-0 flex-1 overflow-auto px-5 py-3"><FolderPicker selected={projectPath} onSelect={setProjectPath} /></div>
- <div className="border-t border-border px-5 py-3"><label className="block text-xs text-muted-foreground">Project name (optional)<input value={projectName} onChange={(event) => setProjectName(event.target.value)} placeholder={projectPath?.split(/[\\/]/).filter(Boolean).at(-1) ?? "Project name"} className="mt-1 w-full rounded-md border border-input bg-background px-3 py-2 text-sm text-foreground" /></label>
- <div className="mt-3 flex justify-end gap-2"><button type="button" onClick={() => setProjectDialog(false)} className="rounded-md px-3 py-1.5 text-sm text-muted-foreground hover:bg-accent">Cancel</button><button type="button" disabled={busy || !projectPath} onClick={connect} className="rounded-md bg-primary px-3 py-1.5 text-sm text-primary-foreground disabled:opacity-50">Connect project</button></div>
- </div>
+ {projectDialog && <div className="fixed inset-0 z-[80] flex items-center justify-center bg-background/75 p-4 backdrop-blur-sm" role="presentation" onMouseDown={(event) => { if (!busy && event.target === event.currentTarget) setProjectDialog(false); }}>
+ {/* A fixed height, so the dialog does not jump each time a folder with
+ a different number of subfolders is opened. */}
+ <section data-ide-dialog tabIndex={-1} role="dialog" aria-modal="true" aria-label="Connect project" aria-describedby="connect-project-hint" aria-busy={busy}
+ className="flex h-[min(46rem,90vh)] w-full max-w-3xl flex-col overflow-hidden rounded-2xl border border-border bg-card shadow-2xl">
+ <header className="shrink-0 px-6 pb-1 pt-6">
+ <div className="flex items-center justify-between gap-4">
+ <h2 className="text-xl font-semibold tracking-tight">Connect a project</h2>
+ <button type="button" aria-label="Close" disabled={busy} onClick={() => setProjectDialog(false)}
+ className="rounded-lg p-2 text-muted-foreground hover:bg-muted hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring"><X className="h-5 w-5" /></button>
+ </div>
+ <p id="connect-project-hint" className="mt-1 text-sm text-muted-foreground">Choose the folder your coding agents will work in. Open it in the list, find it by name, or make a new one.</p>
+ </header>
+ <div className="flex min-h-0 flex-1 flex-col px-3"><FolderPicker selected={projectPath} onSelect={setProjectPath} /></div>
+ <footer className="flex shrink-0 flex-wrap items-end gap-3 border-t border-border bg-muted/20 px-6 py-4">
+ <label className="min-w-[14rem] flex-1 text-xs font-medium text-muted-foreground">Project name <span className="font-normal opacity-70">(optional)</span>
+ <input value={projectName} onChange={(event) => setProjectName(event.target.value)}
+ onKeyDown={(event) => { if (event.key === "Enter" && projectPath && !busy) { event.preventDefault(); connect(); } }}
+ placeholder={projectPath?.split(/[\\/]/).filter(Boolean).at(-1) ?? "Uses the folder name"}
+ className="mt-1.5 h-10 w-full rounded-lg border border-input bg-background/60 px-3 text-sm text-foreground outline-none placeholder:text-muted-foreground focus:border-ring focus:ring-1 focus:ring-ring/30" />
+ </label>
+ <div className="flex gap-2">
+ <button type="button" disabled={busy} onClick={() => setProjectDialog(false)} className="h-10 rounded-lg px-4 text-sm text-muted-foreground hover:bg-muted disabled:opacity-50">Cancel</button>
+ <button type="button" disabled={busy || !projectPath} onClick={connect} title={projectPath ? undefined : "Choose a folder first"}
+ className="inline-flex h-10 items-center gap-2 rounded-lg bg-primary px-4 text-sm font-medium text-primary-foreground disabled:opacity-50">
+ {busy && <Loader2 className="h-4 w-4 animate-spin" aria-hidden="true" />}Connect project</button>
+ </div>
+ </footer>
</section>
</div>}
diff --git a/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXConfirmDialog.tsx b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXConfirmDialog.tsx
new file mode 100644
index 0000000000..3d7a7905e6
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXConfirmDialog.tsx
@@ -0,0 +1,65 @@
+import { useEffect, useRef } from "react";
+
+import { Button } from "@/components/ui/button";
+import { useT } from "@/i18n";
+
+/**
+ * An in-app confirmation. Never `window.confirm`: a native dialog blocks the
+ * desktop WebView's loop and looks foreign in both themes.
+ */
+export function JarvisXConfirmDialog({
+ title,
+ body,
+ confirmLabel,
+ onCancel,
+ onConfirm,
+}: {
+ title: string;
+ body: string;
+ confirmLabel: string;
+ onCancel: () => void;
+ onConfirm: () => void;
+}) {
+ const t = useT();
+ const cancelRef = useRef<HTMLButtonElement>(null);
+
+ useEffect(() => {
+ cancelRef.current?.focus();
+ const onKey = (event: KeyboardEvent) => {
+ if (event.key === "Escape") {
+ event.stopPropagation();
+ onCancel();
+ }
+ };
+ window.addEventListener("keydown", onKey, true);
+ return () => window.removeEventListener("keydown", onKey, true);
+ }, [onCancel]);
+
+ return (
+ <div
+ className="fixed inset-0 z-[200] flex items-center justify-center bg-scrim/60 p-4"
+ onPointerDown={(event) => {
+ if (event.target === event.currentTarget) onCancel();
+ }}
+ >
+ <div
+ role="alertdialog"
+ aria-modal="true"
+ aria-label={title}
+ data-testid="jarvisx-confirm"
+ className="w-full max-w-[420px] rounded-xl border border-border bg-card p-6 shadow-2xl"
+ >
+ <h3 className="text-base font-semibold text-foreground">{title}</h3>
+ <p className="mt-2 text-sm text-muted-foreground">{body}</p>
+ <div className="mt-5 flex justify-end gap-2">
+ <Button ref={cancelRef} type="button" size="sm" variant="ghost" onClick={onCancel}>
+ {t("common.cancel")}
+ </Button>
+ <Button type="button" size="sm" variant="destructive" onClick={onConfirm} data-testid="jarvisx-confirm-ok">
+ {confirmLabel}
+ </Button>
+ </div>
+ </div>
+ </div>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXEditor.tsx b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXEditor.tsx
new file mode 100644
index 0000000000..df141c0f5f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXEditor.tsx
@@ -0,0 +1,1233 @@
+import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState, type ReactNode } from "react";
+import {
+ ArrowUpRight,
+ Check,
+ Circle,
+ Copy,
+ Crop,
+ FolderOpen,
+ Grid3x3,
+ Highlighter,
+ Loader2,
+ Minus,
+ MousePointer2,
+ PenLine,
+ Plus,
+ Redo2,
+ Slash,
+ Square,
+ Trash2,
+ Type,
+ Undo2,
+ X,
+} from "lucide-react";
+
+import { Button } from "@/components/ui/button";
+import { QuickTooltip } from "@/components/ui/tooltip";
+import { useElementSize } from "@/hooks/useElementSize";
+import { useT } from "@/i18n";
+import {
+ copyJarvisXItem,
+ deleteJarvisXItem,
+ fetchJarvisXItem,
+ formatDuration,
+ revealJarvisXItem,
+ saveJarvisXEdited,
+ withVersion,
+ type JarvisXItem,
+} from "@/lib/jarvisxApi";
+import {
+ DEFAULT_COLOR,
+ DEFAULT_FONT_SIZE,
+ DEFAULT_STROKE,
+ FONT_SIZES,
+ PALETTE,
+ STROKE_WIDTHS,
+ TOOL_SHORTCUTS,
+ addShape,
+ canRedo,
+ canUndo,
+ clampZoom,
+ commit,
+ createHistory,
+ cropFromDrag,
+ deleteShape,
+ finishGesture,
+ fitScale,
+ hitHandle,
+ hitTest,
+ imageToView,
+ isDirty,
+ isMeaningful,
+ moveShape,
+ newShapeId,
+ nextCounterNumber,
+ rectFromPoints,
+ redo,
+ replacePresent,
+ resizeShape,
+ restyleShape,
+ setCrop,
+ shapeBounds,
+ shapeHandles,
+ stepZoom,
+ toolForKey,
+ undo,
+ updateShape,
+ viewToImage,
+ visibleArea,
+ type EditorDoc,
+ type Handle,
+ type History,
+ type Point,
+ type Rect,
+ type Shape,
+ type TextShape,
+ type Tool,
+} from "@/lib/jarvisxEditorModel";
+import { exportPng, measureTextShape, paintDocument, textFont } from "@/lib/jarvisxRender";
+import { cn } from "@/lib/utils";
+import { useEventStore } from "@/store/events";
+import { JarvisXConfirmDialog } from "@/views/jarvisx/JarvisXConfirmDialog";
+
+/**
+ * The Jarvis X annotation editor.
+ *
+ * One component for both places it appears: its own desktop window
+ * (`/?view=jarvisx-editor&solo=1&item=<id>`) and an overlay over the library.
+ * Annotations stay editable objects until Save flattens them into a PNG at the
+ * capture's native resolution; an item that already has an edited version
+ * opens that version as the base picture (earlier annotations are part of its
+ * pixels by then), and "Original" starts over from the untouched capture.
+ */
+
+const IS_MAC = typeof navigator !== "undefined" && /Mac/i.test(navigator.platform || "");
+const MOD = IS_MAC ? "⌘" : "Ctrl";
+/** Hit and handle tolerance, in SCREEN pixels (converted per zoom). */
+const HIT_SLOP_PX = 6;
+const HANDLE_PX = 5;
+
+interface ToolDef {
+ tool: Tool;
+ labelKey: string;
+ icon: ReactNode;
+}
+
+function CounterGlyph() {
+ return (
+ <svg viewBox="0 0 24 24" aria-hidden className="h-4 w-4">
+ <circle cx="12" cy="12" r="9" fill="none" stroke="currentColor" strokeWidth="2" />
+ <text x="12" y="16.2" textAnchor="middle" fontSize="11" fontWeight="700" fill="currentColor">
+ 1
+ </text>
+ </svg>
+ );
+}
+
+function HighlightGlyph() {
+ return (
+ <svg viewBox="0 0 24 24" aria-hidden className="h-4 w-4">
+ <rect x="3" y="6" width="18" height="12" rx="2" fill="currentColor" opacity="0.35" />
+ <rect x="3" y="6" width="18" height="12" rx="2" fill="none" stroke="currentColor" strokeWidth="1.5" />
+ </svg>
+ );
+}
+
+const TOOLS: ToolDef[] = [
+ { tool: "select", labelKey: "jarvisx.editor.tool_select", icon: <MousePointer2 aria-hidden /> },
+ { tool: "rect", labelKey: "jarvisx.editor.tool_rect", icon: <Square aria-hidden /> },
+ { tool: "highlight", labelKey: "jarvisx.editor.tool_highlight", icon: <HighlightGlyph /> },
+ { tool: "ellipse", labelKey: "jarvisx.editor.tool_ellipse", icon: <Circle aria-hidden /> },
+ { tool: "arrow", labelKey: "jarvisx.editor.tool_arrow", icon: <ArrowUpRight aria-hidden /> },
+ { tool: "line", labelKey: "jarvisx.editor.tool_line", icon: <Slash aria-hidden /> },
+ { tool: "pen", labelKey: "jarvisx.editor.tool_pen", icon: <PenLine aria-hidden /> },
+ { tool: "marker", labelKey: "jarvisx.editor.tool_marker", icon: <Highlighter aria-hidden /> },
+ { tool: "text", labelKey: "jarvisx.editor.tool_text", icon: <Type aria-hidden /> },
+ { tool: "counter", labelKey: "jarvisx.editor.tool_counter", icon: <CounterGlyph /> },
+ { tool: "blur", labelKey: "jarvisx.editor.tool_blur", icon: <Grid3x3 aria-hidden /> },
+ { tool: "crop", labelKey: "jarvisx.editor.tool_crop", icon: <Crop aria-hidden /> },
+];
+
+interface Style {
+ color: string;
+ width: number;
+ shadow: boolean;
+ fontSize: number;
+}
+
+type Gesture =
+ | { type: "draw"; before: EditorDoc; id: string; origin: Point }
+ | { type: "move"; before: EditorDoc; id: string; origin: Point; original: Shape }
+ | { type: "resize"; before: EditorDoc; id: string; handle: Handle; original: Shape }
+ | { type: "crop"; origin: Point; current: Point };
+
+function isTypingTarget(target: EventTarget | null): boolean {
+ if (!(target instanceof HTMLElement)) return false;
+ return target.isContentEditable || ["INPUT", "TEXTAREA", "SELECT"].includes(target.tagName);
+}
+
+function handleCursor(handle: Handle): string {
+ switch (handle) {
+ case "nw":
+ case "se":
+ return "nwse-resize";
+ case "ne":
+ case "sw":
+ return "nesw-resize";
+ case "n":
+ case "s":
+ return "ns-resize";
+ case "e":
+ case "w":
+ return "ew-resize";
+ default:
+ return "move";
+ }
+}
+
+/** Constrain a drag with Shift: square boxes, 45-degree lines. */
+function constrain(tool: Tool, origin: Point, p: Point): Point {
+ const dx = p.x - origin.x;
+ const dy = p.y - origin.y;
+ if (tool === "arrow" || tool === "line") {
+ const angle = Math.round(Math.atan2(dy, dx) / (Math.PI / 4)) * (Math.PI / 4);
+ const len = Math.hypot(dx, dy);
+ return { x: origin.x + len * Math.cos(angle), y: origin.y + len * Math.sin(angle) };
+ }
+ const side = Math.max(Math.abs(dx), Math.abs(dy));
+ return { x: origin.x + Math.sign(dx || 1) * side, y: origin.y + Math.sign(dy || 1) * side };
+}
+
+function draftShape(tool: Tool, id: string, origin: Point, style: Style): Shape | null {
+ const base = { id, color: style.color, width: style.width, shadow: style.shadow };
+ switch (tool) {
+ case "rect":
+ case "highlight":
+ case "ellipse":
+ case "blur":
+ return { ...base, kind: tool, x: origin.x, y: origin.y, w: 0, h: 0 };
+ case "arrow":
+ case "line":
+ return { ...base, kind: tool, x1: origin.x, y1: origin.y, x2: origin.x, y2: origin.y };
+ case "pen":
+ case "marker":
+ return { ...base, kind: tool, points: [origin] };
+ default:
+ return null;
+ }
+}
+
+function extendDraft(shape: Shape, origin: Point, p: Point): Shape {
+ switch (shape.kind) {
+ case "rect":
+ case "highlight":
+ case "ellipse":
+ case "blur":
+ return { ...shape, ...rectFromPoints(origin, p) };
+ case "arrow":
+ case "line":
+ return { ...shape, x2: p.x, y2: p.y };
+ case "pen":
+ case "marker": {
+ const last = shape.points[shape.points.length - 1];
+ if (last && Math.hypot(p.x - last.x, p.y - last.y) < 1) return shape;
+ return { ...shape, points: [...shape.points, p] };
+ }
+ default:
+ return shape;
+ }
+}
+
+export interface JarvisXEditorProps {
+ /** The capture to edit; absent together with `devSrc` opens a local picture read-only. */
+ itemId: string | null;
+ /** Dev/verification fallback: a same-origin image URL to annotate without an item. */
+ devSrc?: string | null;
+ variant: "window" | "overlay";
+ onClose: () => void;
+ /** Extra controls for the leading edge of the toolbar row (the window caption). */
+ leading?: ReactNode;
+ /** Controls for the trailing edge (the window buttons). */
+ trailing?: ReactNode;
+}
+
+export function JarvisXEditor({ itemId, devSrc, variant, onClose, leading, trailing }: JarvisXEditorProps) {
+ const t = useT();
+ const pushToast = useEventStore((s) => s.pushToast);
+
+ const [item, setItem] = useState<JarvisXItem | null>(null);
+ const [loadError, setLoadError] = useState<string | null>(null);
+ const [base, setBase] = useState<"edited" | "original">("edited");
+ const [image, setImage] = useState<HTMLImageElement | null>(null);
+ const [history, setHistory] = useState<History>(() => createHistory());
+ const [savedDoc, setSavedDoc] = useState<EditorDoc>(history.present);
+ const [tool, setTool] = useState<Tool>("rect");
+ const [style, setStyle] = useState<Style>({
+ color: DEFAULT_COLOR,
+ width: DEFAULT_STROKE,
+ shadow: true,
+ fontSize: DEFAULT_FONT_SIZE,
+ });
+ const [selectedId, setSelectedId] = useState<string | null>(null);
+ const [editingTextId, setEditingTextId] = useState<string | null>(null);
+ const [zoom, setZoom] = useState<number | "fit">("fit");
+ const [cropPreview, setCropPreview] = useState<Rect | null>(null);
+ const [busy, setBusy] = useState<null | "save" | "copy" | "close">(null);
+ const [confirm, setConfirm] = useState<null | "discard" | "delete" | "original">(null);
+ const [cursor, setCursor] = useState("crosshair");
+ const [reloadKey, setReloadKey] = useState(0);
+
+ const stageRef = useRef<HTMLDivElement>(null);
+ const canvasRef = useRef<HTMLCanvasElement>(null);
+ const gestureRef = useRef<Gesture | null>(null);
+ const historyRef = useRef(history);
+ historyRef.current = history;
+ const stage = useElementSize(stageRef);
+
+ const doc = history.present;
+ const isVideo = item?.kind === "video";
+ const canPersist = Boolean(itemId && item);
+
+ // ---- loading -----------------------------------------------------------
+
+ useEffect(() => {
+ if (!itemId) {
+ setItem(null);
+ setLoadError(devSrc ? null : t("jarvisx.editor.no_item"));
+ return;
+ }
+ let live = true;
+ setLoadError(null);
+ fetchJarvisXItem(itemId)
+ .then((next) => {
+ if (!live) return;
+ setItem(next);
+ setBase(next.edited_url ? "edited" : "original");
+ })
+ .catch((error: Error) => live && setLoadError(error.message));
+ return () => {
+ live = false;
+ };
+ }, [itemId, devSrc, reloadKey, t]);
+
+ const imageSrc = useMemo(() => {
+ if (item && item.kind === "image") {
+ if (base === "edited" && item.edited_url) return withVersion(item.edited_url, reloadKey);
+ return item.url;
+ }
+ return !itemId && devSrc ? devSrc : null;
+ }, [item, base, itemId, devSrc, reloadKey]);
+
+ useEffect(() => {
+ if (!imageSrc) {
+ setImage(null);
+ return;
+ }
+ let live = true;
+ const img = new Image();
+ img.decoding = "async";
+ img.onload = () => {
+ if (!live) return;
+ setImage(img);
+ const fresh = createHistory();
+ setHistory(fresh);
+ setSavedDoc(fresh.present);
+ setSelectedId(null);
+ setEditingTextId(null);
+ setZoom("fit");
+ };
+ img.onerror = () => live && setLoadError(t("jarvisx.editor.image_failed"));
+ img.src = imageSrc;
+ return () => {
+ live = false;
+ };
+ }, [imageSrc, t]);
+
+ // ---- geometry ----------------------------------------------------------
+
+ const natW = image?.naturalWidth ?? 0;
+ const natH = image?.naturalHeight ?? 0;
+ // The crop tool shows the whole picture so the crop can grow again.
+ const area: Rect = tool === "crop" ? { x: 0, y: 0, w: natW, h: natH } : visibleArea(doc, natW, natH);
+ const fit = fitScale(stage.width, stage.height, area.w, area.h);
+ const scale = zoom === "fit" ? fit : zoom;
+ const viewW = Math.max(1, Math.round(area.w * scale));
+ const viewH = Math.max(1, Math.round(area.h * scale));
+ const slop = HIT_SLOP_PX / scale;
+
+ const measure = useCallback((shape: TextShape) => measureTextShape(shape), []);
+
+ // ---- painting ----------------------------------------------------------
+
+ useLayoutEffect(() => {
+ const canvas = canvasRef.current;
+ if (!canvas || !image) return;
+ const dpr = typeof window !== "undefined" ? window.devicePixelRatio || 1 : 1;
+ canvas.width = Math.round(viewW * dpr);
+ canvas.height = Math.round(viewH * dpr);
+ const ctx = canvas.getContext("2d");
+ if (!ctx) return;
+ ctx.setTransform(1, 0, 0, 1, 0, 0);
+ ctx.clearRect(0, 0, canvas.width, canvas.height);
+ const k = scale * dpr;
+ ctx.setTransform(k, 0, 0, k, -area.x * k, -area.y * k);
+ ctx.imageSmoothingQuality = "high";
+ paintDocument(ctx, image, doc, editingTextId);
+
+ // Overlays in device pixels so they stay crisp at any zoom.
+ ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
+ if (tool === "crop") {
+ const crop = cropPreview ?? doc.crop;
+ if (crop) {
+ const a = imageToView({ x: crop.x, y: crop.y }, scale, area);
+ const w = crop.w * scale;
+ const h = crop.h * scale;
+ ctx.fillStyle = "rgba(0, 0, 0, 0.55)";
+ ctx.beginPath();
+ ctx.rect(0, 0, viewW, viewH);
+ ctx.rect(a.x, a.y, w, h);
+ ctx.fill("evenodd");
+ ctx.strokeStyle = "#ffffff";
+ ctx.lineWidth = 1.5;
+ ctx.setLineDash([6, 4]);
+ ctx.strokeRect(a.x, a.y, w, h);
+ ctx.setLineDash([]);
+ }
+ }
+ const selected = selectedId && selectedId !== editingTextId ? doc.shapes.find((s) => s.id === selectedId) : null;
+ if (selected && tool !== "crop") {
+ const b = shapeBounds(selected, measure);
+ const pad = selected.width / 2 + 3 / scale;
+ const tl = imageToView({ x: b.x - pad, y: b.y - pad }, scale, area);
+ ctx.strokeStyle = "#ffffff";
+ ctx.lineWidth = 1;
+ ctx.setLineDash([4, 3]);
+ ctx.strokeRect(tl.x, tl.y, (b.w + pad * 2) * scale, (b.h + pad * 2) * scale);
+ ctx.strokeStyle = "rgba(0, 0, 0, 0.6)";
+ ctx.lineDashOffset = 3.5;
+ ctx.strokeRect(tl.x, tl.y, (b.w + pad * 2) * scale, (b.h + pad * 2) * scale);
+ ctx.setLineDash([]);
+ ctx.lineDashOffset = 0;
+ for (const h of shapeHandles(selected)) {
+ const v = imageToView(h, scale, area);
+ ctx.fillStyle = "#ffffff";
+ ctx.strokeStyle = "rgba(0, 0, 0, 0.7)";
+ ctx.lineWidth = 1;
+ ctx.beginPath();
+ ctx.arc(v.x, v.y, HANDLE_PX, 0, Math.PI * 2);
+ ctx.fill();
+ ctx.stroke();
+ }
+ }
+ }, [image, doc, scale, viewW, viewH, area.x, area.y, tool, cropPreview, selectedId, editingTextId, measure]);
+
+ // ---- edits -------------------------------------------------------------
+
+ const apply = useCallback((next: (d: EditorDoc) => EditorDoc) => {
+ setHistory((h) => commit(h, next(h.present)));
+ }, []);
+
+ const doUndo = useCallback(() => {
+ setEditingTextId(null);
+ setHistory((h) => undo(h));
+ }, []);
+ const doRedo = useCallback(() => setHistory((h) => redo(h)), []);
+
+ const deleteSelected = useCallback(() => {
+ if (!selectedId) return;
+ apply((d) => deleteShape(d, selectedId));
+ setSelectedId(null);
+ }, [apply, selectedId]);
+
+ const changeStyle = useCallback(
+ (patch: Partial<Style>) => {
+ setStyle((s) => ({ ...s, ...patch }));
+ if (selectedId) apply((d) => updateShape(d, selectedId, (shape) => restyleShape(shape, patch)));
+ },
+ [apply, selectedId],
+ );
+
+ const chooseTool = useCallback((next: Tool) => {
+ setTool(next);
+ setCropPreview(null);
+ if (next !== "select") setSelectedId(null);
+ }, []);
+
+ // ---- pointer -----------------------------------------------------------
+
+ const toImage = (event: React.PointerEvent | React.MouseEvent): Point => {
+ const rect = canvasRef.current!.getBoundingClientRect();
+ return viewToImage({ x: event.clientX - rect.left, y: event.clientY - rect.top }, scale, area);
+ };
+
+ const commitText = useCallback(
+ (id: string, text: string) => {
+ setEditingTextId(null);
+ setHistory((h) => {
+ const shape = h.present.shapes.find((s) => s.id === id);
+ if (!shape || shape.kind !== "text") return h;
+ // A new text shape entered the document without a history step; its
+ // undo point is the document WITHOUT it.
+ const isNew = shape.text === "";
+ const without = deleteShape(h.present, id);
+ if (!text.trim()) return isNew ? replacePresent(h, without) : commit(h, without);
+ if (shape.text === text) return h;
+ const next = updateShape(h.present, id, (s) => ({ ...(s as TextShape), text }));
+ return isNew ? finishGesture(replacePresent(h, next), without) : commit(h, next);
+ });
+ },
+ [],
+ );
+
+ const onPointerDown = (event: React.PointerEvent<HTMLCanvasElement>) => {
+ if (!image || event.button !== 0) return;
+ if (editingTextId) {
+ // A click outside the text field ends the edit; the textarea's blur commits it.
+ (document.activeElement as HTMLElement | null)?.blur();
+ return;
+ }
+ const p = toImage(event);
+ const present = historyRef.current.present;
+ event.currentTarget.setPointerCapture(event.pointerId);
+
+ if (tool === "crop") {
+ gestureRef.current = { type: "crop", origin: p, current: p };
+ setCropPreview(null);
+ return;
+ }
+
+ // The selected shape's handles win in every tool: resize right after drawing.
+ const selected = selectedId ? present.shapes.find((s) => s.id === selectedId) : null;
+ if (selected) {
+ const handle = hitHandle(selected, p, (HANDLE_PX + 3) / scale);
+ if (handle) {
+ gestureRef.current = { type: "resize", before: present, id: selected.id, handle, original: selected };
+ return;
+ }
+ }
+
+ if (tool === "select") {
+ const hit = hitTest(present.shapes, p, slop, measure);
+ setSelectedId(hit);
+ if (hit) {
+ const original = present.shapes.find((s) => s.id === hit)!;
+ gestureRef.current = { type: "move", before: present, id: hit, origin: p, original };
+ if (original.kind === "text") {
+ setStyle((s) => ({ ...s, color: original.color, fontSize: original.fontSize }));
+ } else {
+ setStyle((s) => ({ ...s, color: original.color, width: original.width, shadow: original.shadow }));
+ }
+ }
+ return;
+ }
+
+ if (tool === "text") {
+ const existing = hitTest(present.shapes, p, slop, measure);
+ const hitShape = existing ? present.shapes.find((s) => s.id === existing) : null;
+ if (hitShape && hitShape.kind === "text") {
+ setSelectedId(hitShape.id);
+ setEditingTextId(hitShape.id);
+ return;
+ }
+ const id = newShapeId();
+ const shape: TextShape = {
+ id,
+ kind: "text",
+ color: style.color,
+ width: style.width,
+ shadow: style.shadow,
+ fontSize: style.fontSize,
+ x: p.x,
+ y: p.y - (style.fontSize * 1.25) / 2,
+ text: "",
+ };
+ // Not in the history until it has text: an empty click leaves no step.
+ setHistory((h) => replacePresent(h, addShape(h.present, shape)));
+ setSelectedId(id);
+ setEditingTextId(id);
+ return;
+ }
+
+ if (tool === "counter") {
+ const id = newShapeId();
+ apply((d) =>
+ addShape(d, {
+ id,
+ kind: "counter",
+ color: style.color,
+ width: style.width,
+ shadow: style.shadow,
+ x: p.x,
+ y: p.y,
+ n: nextCounterNumber(d.shapes),
+ }),
+ );
+ setSelectedId(id);
+ return;
+ }
+
+ const id = newShapeId();
+ const draft = draftShape(tool, id, p, style);
+ if (!draft) return;
+ gestureRef.current = { type: "draw", before: present, id, origin: p };
+ setHistory((h) => replacePresent(h, addShape(h.present, draft)));
+ };
+
+ const onPointerMove = (event: React.PointerEvent<HTMLCanvasElement>) => {
+ if (!image) return;
+ let p = toImage(event);
+ const gesture = gestureRef.current;
+ if (!gesture) {
+ // Hover feedback only.
+ const present = historyRef.current.present;
+ const selected = selectedId ? present.shapes.find((s) => s.id === selectedId) : null;
+ const handle = selected && tool !== "crop" ? hitHandle(selected, p, (HANDLE_PX + 3) / scale) : null;
+ if (handle) setCursor(handleCursor(handle));
+ else if (tool === "select") setCursor(hitTest(present.shapes, p, slop, measure) ? "move" : "default");
+ else if (tool === "text") setCursor("text");
+ else setCursor("crosshair");
+ return;
+ }
+ if (gesture.type === "crop") {
+ gesture.current = p;
+ setCropPreview(rectFromPoints(gesture.origin, p));
+ return;
+ }
+ if (gesture.type === "draw") {
+ if (event.shiftKey) p = constrain(tool, gesture.origin, p);
+ setHistory((h) =>
+ replacePresent(h, updateShape(h.present, gesture.id, (s) => extendDraft(s, gesture.origin, p))),
+ );
+ return;
+ }
+ if (gesture.type === "move") {
+ const dx = p.x - gesture.origin.x;
+ const dy = p.y - gesture.origin.y;
+ setHistory((h) => replacePresent(h, updateShape(h.present, gesture.id, () => moveShape(gesture.original, dx, dy))));
+ return;
+ }
+ if (gesture.type === "resize") {
+ setHistory((h) =>
+ replacePresent(h, updateShape(h.present, gesture.id, () => resizeShape(gesture.original, gesture.handle, p))),
+ );
+ }
+ };
+
+ const onPointerUp = (event: React.PointerEvent<HTMLCanvasElement>) => {
+ const gesture = gestureRef.current;
+ gestureRef.current = null;
+ if (event.currentTarget.hasPointerCapture(event.pointerId)) {
+ event.currentTarget.releasePointerCapture(event.pointerId);
+ }
+ if (!gesture || !image) return;
+ if (gesture.type === "crop") {
+ setCropPreview(null);
+ const crop = cropFromDrag(gesture.origin, gesture.current, natW, natH);
+ if (crop) apply((d) => setCrop(d, crop));
+ return;
+ }
+ if (gesture.type === "draw") {
+ setHistory((h) => {
+ const shape = h.present.shapes.find((s) => s.id === gesture.id);
+ if (!shape || !isMeaningful(shape)) return replacePresent(h, gesture.before);
+ return finishGesture(h, gesture.before);
+ });
+ setSelectedId(gesture.id);
+ return;
+ }
+ setHistory((h) => finishGesture(h, gesture.before));
+ };
+
+ const onDoubleClick = (event: React.MouseEvent<HTMLCanvasElement>) => {
+ if (tool !== "select") return;
+ const p = toImage(event);
+ const hit = hitTest(doc.shapes, p, slop, measure);
+ const shape = hit ? doc.shapes.find((s) => s.id === hit) : null;
+ if (shape?.kind === "text") {
+ setSelectedId(shape.id);
+ setEditingTextId(shape.id);
+ }
+ };
+
+ const onWheel = (event: React.WheelEvent) => {
+ if (!event.ctrlKey && !event.metaKey) return;
+ event.preventDefault();
+ setZoom(clampZoom(scale * (event.deltaY < 0 ? 1.1 : 1 / 1.1)));
+ };
+
+ // ---- actions -----------------------------------------------------------
+
+ const dirtySinceSave = doc !== savedDoc && isDirty(doc);
+
+ const save = useCallback(async (): Promise<boolean> => {
+ if (!image || !item || !itemId) return false;
+ setEditingTextId(null);
+ try {
+ const png = await exportPng(image, historyRef.current.present);
+ const next = await saveJarvisXEdited(itemId, png);
+ setItem((current) => ({ ...(current ?? item), ...(next ?? {}) }));
+ setSavedDoc(historyRef.current.present);
+ return true;
+ } catch (error) {
+ pushToast("error", t("jarvisx.editor.save_failed").replace("{0}", (error as Error).message));
+ return false;
+ }
+ }, [image, item, itemId, pushToast, t]);
+
+ const onSave = useCallback(async () => {
+ if (!canPersist || busy) return;
+ setBusy("save");
+ if (await save()) pushToast("success", t("jarvisx.editor.saved"));
+ setBusy(null);
+ }, [busy, canPersist, pushToast, save, t]);
+
+ const onCopy = useCallback(async () => {
+ if (!canPersist || busy || !itemId) return;
+ setBusy("copy");
+ try {
+ let edited = base === "edited" && Boolean(item?.edited_url);
+ if (isVideo) edited = false;
+ else if (dirtySinceSave) {
+ if (!(await save())) return;
+ edited = true;
+ } else if (isDirty(doc)) edited = true;
+ const result = await copyJarvisXItem(itemId, edited);
+ if (result.ok === false) pushToast("error", result.message || t("jarvisx.copy_failed"));
+ else pushToast("success", t("jarvisx.copied"));
+ } catch (error) {
+ pushToast("error", (error as Error).message);
+ } finally {
+ setBusy(null);
+ }
+ }, [base, busy, canPersist, dirtySinceSave, doc, isVideo, item, itemId, pushToast, save, t]);
+
+ const onSaveAndClose = useCallback(async () => {
+ if (!canPersist || busy) return;
+ setBusy("close");
+ const ok = !dirtySinceSave || (await save());
+ setBusy(null);
+ if (ok) onClose();
+ }, [busy, canPersist, dirtySinceSave, onClose, save]);
+
+ const onReveal = useCallback(async () => {
+ if (!itemId) return;
+ try {
+ const result = await revealJarvisXItem(itemId);
+ if (result.ok === false && result.message) pushToast("error", result.message);
+ } catch (error) {
+ pushToast("error", (error as Error).message);
+ }
+ }, [itemId, pushToast]);
+
+ const onDelete = useCallback(async () => {
+ if (!itemId) return;
+ try {
+ await deleteJarvisXItem(itemId);
+ pushToast("success", t("jarvisx.deleted"));
+ onClose();
+ } catch (error) {
+ pushToast("error", (error as Error).message);
+ }
+ }, [itemId, onClose, pushToast, t]);
+
+ const requestClose = useCallback(() => {
+ if (dirtySinceSave) setConfirm("discard");
+ else onClose();
+ }, [dirtySinceSave, onClose]);
+
+ // ---- keyboard ----------------------------------------------------------
+
+ useEffect(() => {
+ const onKey = (event: KeyboardEvent) => {
+ if (confirm || isTypingTarget(event.target)) return;
+ const mod = event.ctrlKey || event.metaKey;
+ const key = event.key.toLowerCase();
+ if (mod && key === "z") {
+ event.preventDefault();
+ if (event.shiftKey) doRedo();
+ else doUndo();
+ return;
+ }
+ if (mod && key === "y") {
+ event.preventDefault();
+ doRedo();
+ return;
+ }
+ if (mod && key === "s") {
+ event.preventDefault();
+ void onSave();
+ return;
+ }
+ if (mod && key === "c") {
+ event.preventDefault();
+ void onCopy();
+ return;
+ }
+ if (mod && (key === "0" || key === "1")) {
+ event.preventDefault();
+ setZoom(key === "0" ? "fit" : 1);
+ return;
+ }
+ if (mod && (key === "=" || key === "+" || key === "-")) {
+ event.preventDefault();
+ setZoom(clampZoom(stepZoom(scale, key === "-" ? -1 : 1)));
+ return;
+ }
+ if (mod || event.altKey) return;
+ if ((event.key === "Delete" || event.key === "Backspace") && selectedId) {
+ event.preventDefault();
+ deleteSelected();
+ return;
+ }
+ if (event.key === "Escape") {
+ if (selectedId) setSelectedId(null);
+ else if (tool === "crop") chooseTool("select");
+ else requestClose();
+ return;
+ }
+ if (event.key === "Enter" && tool === "crop") {
+ chooseTool("select");
+ return;
+ }
+ const next = toolForKey(event.key);
+ if (next && !isVideo) chooseTool(next);
+ };
+ window.addEventListener("keydown", onKey);
+ return () => window.removeEventListener("keydown", onKey);
+ }, [chooseTool, confirm, deleteSelected, doRedo, doUndo, isVideo, onCopy, onSave, requestClose, scale, selectedId, tool]);
+
+ // ---- render ------------------------------------------------------------
+
+ const selectedShape = selectedId ? doc.shapes.find((s) => s.id === selectedId) ?? null : null;
+ const styleTarget: Tool | Shape["kind"] = selectedShape?.kind ?? tool;
+ const showsColor = styleTarget !== "blur" && styleTarget !== "crop" && styleTarget !== "select";
+ const showsWidth = !["text", "crop", "select", "highlight"].includes(styleTarget);
+ const showsFont = styleTarget === "text";
+ const showsShadow = ["rect", "ellipse", "arrow", "line", "pen", "text", "counter"].includes(styleTarget);
+ const editingText = editingTextId ? (doc.shapes.find((s) => s.id === editingTextId) as TextShape | undefined) : undefined;
+ const title = item?.filename ?? (devSrc ? devSrc.split("/").pop() ?? "" : "");
+
+ return (
+ <div
+ data-testid="jarvisx-editor"
+ className={cn(
+ "flex h-full min-h-0 w-full flex-col bg-background text-foreground",
+ variant === "overlay" && "rounded-xl border border-border shadow-2xl",
+ )}
+ >
+ {/* Toolbar row. In the window variant its empty middle is the drag handle. */}
+ <div className="flex h-12 shrink-0 items-center gap-2 border-b border-border bg-card pl-2">
+ {leading}
+ {!isVideo && (
+ <div role="toolbar" aria-label={t("jarvisx.editor.tools")} className="flex items-center gap-0.5">
+ {TOOLS.map((def) => {
+ const label = `${t(def.labelKey)} (${TOOL_SHORTCUTS[def.tool]})`;
+ return (
+ <QuickTooltip key={def.tool} content={label} side="bottom">
+ <button
+ type="button"
+ aria-label={label}
+ aria-pressed={tool === def.tool}
+ data-testid={`jarvisx-tool-${def.tool}`}
+ onClick={() => chooseTool(def.tool)}
+ className={cn(
+ "grid h-8 w-8 place-items-center rounded-md text-muted-foreground transition-colors [&>svg]:h-4 [&>svg]:w-4",
+ "hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring",
+ tool === def.tool && "bg-accent-soft text-accent hover:bg-accent-soft hover:text-accent",
+ )}
+ >
+ {def.icon}
+ </button>
+ </QuickTooltip>
+ );
+ })}
+ </div>
+ )}
+ <div
+ className={cn("h-full min-w-6 flex-1 truncate px-2 text-center text-sm leading-[3rem] text-muted-foreground", variant === "window" && "pywebview-drag-region")}
+ title={title}
+ >
+ {title}
+ </div>
+ <div className="flex shrink-0 items-center gap-1 pr-2">
+ {!isVideo && (
+ <>
+ <IconButton label={`${t("jarvisx.editor.undo")} (${MOD}+Z)`} disabled={!canUndo(history)} onClick={doUndo}>
+ <Undo2 aria-hidden />
+ </IconButton>
+ <IconButton label={`${t("jarvisx.editor.redo")} (${MOD}+Shift+Z)`} disabled={!canRedo(history)} onClick={doRedo}>
+ <Redo2 aria-hidden />
+ </IconButton>
+ <span className="mx-1 h-5 w-px bg-border" aria-hidden />
+ </>
+ )}
+ {canPersist && (
+ <>
+ <IconButton label={t("jarvisx.reveal")} onClick={() => void onReveal()}>
+ <FolderOpen aria-hidden />
+ </IconButton>
+ <IconButton label={t("jarvisx.delete")} onClick={() => setConfirm("delete")} testId="jarvisx-editor-delete">
+ <Trash2 aria-hidden />
+ </IconButton>
+ <Button type="button" size="sm" variant="outline" disabled={busy !== null} onClick={() => void onCopy()} data-testid="jarvisx-editor-copy">
+ {busy === "copy" ? <Loader2 className="animate-spin" aria-hidden /> : <Copy aria-hidden />}
+ {t("jarvisx.copy")}
+ </Button>
+ {!isVideo && (
+ <Button type="button" size="sm" variant="secondary" disabled={busy !== null || !dirtySinceSave} onClick={() => void onSave()} data-testid="jarvisx-editor-save">
+ {busy === "save" && <Loader2 className="animate-spin" aria-hidden />}
+ {t("jarvisx.editor.save")}
+ </Button>
+ )}
+ <Button type="button" size="sm" disabled={busy !== null} onClick={() => void (isVideo ? onClose() : onSaveAndClose())} data-testid="jarvisx-editor-done">
+ {busy === "close" ? <Loader2 className="animate-spin" aria-hidden /> : <Check aria-hidden />}
+ {isVideo ? t("common.close") : t("jarvisx.editor.done")}
+ </Button>
+ </>
+ )}
+ {!canPersist && (
+ <IconButton label={t("common.close")} onClick={requestClose}>
+ <X aria-hidden />
+ </IconButton>
+ )}
+ </div>
+ {trailing}
+ </div>
+
+ {/* Style row: only what the active tool or selection actually uses. */}
+ {!isVideo && image && (
+ <div className="flex h-11 shrink-0 items-center gap-3 overflow-x-auto border-b border-border bg-card px-3 scrollbar-jarvis">
+ {showsColor && (
+ <div role="radiogroup" aria-label={t("jarvisx.editor.color")} className="flex items-center gap-1">
+ {PALETTE.map((color) => (
+ <button
+ key={color}
+ type="button"
+ role="radio"
+ aria-checked={style.color === color}
+ aria-label={color}
+ onClick={() => changeStyle({ color })}
+ className={cn(
+ "h-5 w-5 rounded-full border border-border-strong transition-transform focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring",
+ style.color === color && "ring-2 ring-accent ring-offset-2 ring-offset-card",
+ )}
+ style={{ backgroundColor: color }}
+ />
+ ))}
+ </div>
+ )}
+ {showsWidth && (
+ <div role="radiogroup" aria-label={t("jarvisx.editor.stroke")} className="flex items-center gap-0.5">
+ {STROKE_WIDTHS.map((width) => (
+ <button
+ key={width}
+ type="button"
+ role="radio"
+ aria-checked={style.width === width}
+ aria-label={`${t("jarvisx.editor.stroke")} ${width}`}
+ onClick={() => changeStyle({ width })}
+ className={cn(
+ "grid h-7 w-7 place-items-center rounded-md text-muted-foreground hover:bg-secondary hover:text-foreground",
+ style.width === width && "bg-secondary text-foreground",
+ )}
+ >
+ <span className="rounded-full bg-current" style={{ width: 4 + width * 1.2, height: 4 + width * 1.2 }} />
+ </button>
+ ))}
+ </div>
+ )}
+ {showsFont && (
+ <div role="radiogroup" aria-label={t("jarvisx.editor.font_size")} className="flex items-center gap-0.5">
+ {FONT_SIZES.map((size) => (
+ <button
+ key={size}
+ type="button"
+ role="radio"
+ aria-checked={style.fontSize === size}
+ onClick={() => changeStyle({ fontSize: size })}
+ className={cn(
+ "h-7 min-w-8 rounded-md px-1.5 text-sm tabular-nums text-muted-foreground hover:bg-secondary hover:text-foreground",
+ style.fontSize === size && "bg-secondary text-foreground",
+ )}
+ >
+ {size}
+ </button>
+ ))}
+ </div>
+ )}
+ {showsShadow && (
+ <label className="flex cursor-pointer items-center gap-1.5 text-sm text-muted-foreground">
+ <input
+ type="checkbox"
+ checked={style.shadow}
+ onChange={(event) => changeStyle({ shadow: event.target.checked })}
+ className="h-3.5 w-3.5 accent-[hsl(var(--accent))]"
+ />
+ {t("jarvisx.editor.shadow")}
+ </label>
+ )}
+ {tool === "crop" && (
+ <div className="flex items-center gap-2 text-sm text-muted-foreground">
+ <span>{doc.crop ? `${doc.crop.w} × ${doc.crop.h}` : t("jarvisx.editor.crop_hint")}</span>
+ {doc.crop && (
+ <Button type="button" size="sm" variant="ghost" onClick={() => apply((d) => setCrop(d, null))}>
+ {t("jarvisx.editor.crop_reset")}
+ </Button>
+ )}
+ <Button type="button" size="sm" variant="secondary" onClick={() => chooseTool("select")}>
+ {t("jarvisx.editor.crop_apply")}
+ </Button>
+ </div>
+ )}
+ {tool === "select" && !selectedShape && (
+ <span className="text-sm text-muted-foreground">{t("jarvisx.editor.select_hint")}</span>
+ )}
+ {selectedShape && (
+ <IconButton label={`${t("jarvisx.editor.delete_shape")} (Del)`} onClick={deleteSelected}>
+ <Trash2 aria-hidden />
+ </IconButton>
+ )}
+ <div className="ml-auto flex shrink-0 items-center gap-1">
+ {item?.edited_url && (
+ <div className="mr-2 flex items-center rounded-md bg-secondary p-0.5 text-sm" role="radiogroup" aria-label={t("jarvisx.editor.base")}>
+ {(["edited", "original"] as const).map((value) => (
+ <button
+ key={value}
+ type="button"
+ role="radio"
+ aria-checked={base === value}
+ onClick={() => {
+ if (base === value) return;
+ if (isDirty(doc) && doc !== savedDoc) setConfirm("original");
+ else setBase(value);
+ }}
+ className={cn(
+ "h-6 rounded px-2 text-muted-foreground",
+ base === value && "bg-card text-foreground shadow-sm",
+ )}
+ >
+ {t(`jarvisx.editor.base_${value}`)}
+ </button>
+ ))}
+ </div>
+ )}
+ <IconButton label={`${t("jarvisx.editor.zoom_out")} (${MOD}+−)`} onClick={() => setZoom(clampZoom(stepZoom(scale, -1)))}>
+ <Minus aria-hidden />
+ </IconButton>
+ <button
+ type="button"
+ onClick={() => setZoom(zoom === "fit" ? 1 : "fit")}
+ title={`${t("jarvisx.editor.zoom_fit")} (${MOD}+0) · 100 % (${MOD}+1)`}
+ className="h-7 min-w-14 rounded-md px-2 text-sm tabular-nums text-muted-foreground hover:bg-secondary hover:text-foreground"
+ data-testid="jarvisx-zoom"
+ >
+ {Math.round(scale * 100)} %
+ </button>
+ <IconButton label={`${t("jarvisx.editor.zoom_in")} (${MOD}++)`} onClick={() => setZoom(clampZoom(stepZoom(scale, 1)))}>
+ <Plus aria-hidden />
+ </IconButton>
+ </div>
+ </div>
+ )}
+
+ {/* Stage */}
+ <div
+ ref={stageRef}
+ onWheel={onWheel}
+ className="relative min-h-0 flex-1 overflow-auto bg-secondary/50 scrollbar-jarvis"
+ >
+ {loadError ? (
+ <div className="flex h-full flex-col items-center justify-center gap-3 p-6 text-center">
+ <p className="text-base text-foreground">{t("jarvisx.editor.load_failed")}</p>
+ <p className="max-w-md text-sm text-muted-foreground">{loadError}</p>
+ {itemId && (
+ <Button type="button" size="sm" variant="secondary" onClick={() => setReloadKey((k) => k + 1)}>
+ {t("common.retry")}
+ </Button>
+ )}
+ </div>
+ ) : isVideo && item ? (
+ <div className="flex h-full items-center justify-center p-6">
+ <video
+ src={item.url}
+ controls
+ className="max-h-full max-w-full rounded-lg bg-scrim shadow-lg"
+ data-testid="jarvisx-video"
+ />
+ {item.duration_s !== null && (
+ <span className="sr-only">{formatDuration(item.duration_s)}</span>
+ )}
+ </div>
+ ) : !image ? (
+ <div className="flex h-full items-center justify-center" role="status" aria-busy="true">
+ <Loader2 className="h-5 w-5 animate-spin text-muted-foreground" aria-hidden />
+ </div>
+ ) : (
+ <div
+ className="flex min-h-full min-w-full items-center justify-center p-6"
+ style={{ width: viewW + 48, height: viewH + 48 }}
+ >
+ <div className="relative shrink-0 shadow-lg ring-1 ring-border" style={{ width: viewW, height: viewH }}>
+ <canvas
+ ref={canvasRef}
+ data-testid="jarvisx-canvas"
+ style={{ width: viewW, height: viewH, cursor, touchAction: "none" }}
+ className="block"
+ onPointerDown={onPointerDown}
+ onPointerMove={onPointerMove}
+ onPointerUp={onPointerUp}
+ onPointerCancel={onPointerUp}
+ onDoubleClick={onDoubleClick}
+ />
+ {editingText && (
+ <TextEditor
+ key={editingText.id}
+ shape={editingText}
+ scale={scale}
+ area={area}
+ onCommit={(text) => commitText(editingText.id, text)}
+ />
+ )}
+ </div>
+ </div>
+ )}
+ </div>
+
+ {!canPersist && devSrc && !itemId && (
+ <p className="shrink-0 border-t border-border bg-card px-4 py-2 text-sm text-muted-foreground">
+ {t("jarvisx.editor.dev_hint")}
+ </p>
+ )}
+
+ {confirm === "discard" && (
+ <JarvisXConfirmDialog
+ title={t("jarvisx.editor.discard_title")}
+ body={t("jarvisx.editor.discard_body")}
+ confirmLabel={t("jarvisx.editor.discard_confirm")}
+ onCancel={() => setConfirm(null)}
+ onConfirm={() => {
+ setConfirm(null);
+ onClose();
+ }}
+ />
+ )}
+ {confirm === "original" && (
+ <JarvisXConfirmDialog
+ title={t("jarvisx.editor.switch_title")}
+ body={t("jarvisx.editor.switch_body")}
+ confirmLabel={t("jarvisx.editor.discard_confirm")}
+ onCancel={() => setConfirm(null)}
+ onConfirm={() => {
+ setConfirm(null);
+ setBase(base === "edited" ? "original" : "edited");
+ }}
+ />
+ )}
+ {confirm === "delete" && (
+ <JarvisXConfirmDialog
+ title={t("jarvisx.delete_title")}
+ body={t("jarvisx.delete_body")}
+ confirmLabel={t("jarvisx.delete")}
+ onCancel={() => setConfirm(null)}
+ onConfirm={() => {
+ setConfirm(null);
+ void onDelete();
+ }}
+ />
+ )}
+ </div>
+ );
+}
+
+function IconButton({
+ label,
+ onClick,
+ disabled,
+ children,
+ testId,
+}: {
+ label: string;
+ onClick: () => void;
+ disabled?: boolean;
+ children: ReactNode;
+ testId?: string;
+}) {
+ return (
+ <QuickTooltip content={label} side="bottom">
+ <button
+ type="button"
+ aria-label={label}
+ disabled={disabled}
+ onClick={onClick}
+ data-testid={testId}
+ className="grid h-8 w-8 place-items-center rounded-md text-muted-foreground transition-colors hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:pointer-events-none disabled:opacity-40 [&>svg]:h-4 [&>svg]:w-4"
+ >
+ {children}
+ </button>
+ </QuickTooltip>
+ );
+}
+
+/** In-place text field over a text shape; Enter commits, Shift+Enter breaks the line. */
+function TextEditor({
+ shape,
+ scale,
+ area,
+ onCommit,
+}: {
+ shape: TextShape;
+ scale: number;
+ area: Rect;
+ onCommit: (text: string) => void;
+}) {
+ const [value, setValue] = useState(shape.text);
+ const ref = useRef<HTMLTextAreaElement>(null);
+ const done = useRef(false);
+ const pos = imageToView({ x: shape.x, y: shape.y }, scale, area);
+ const size = measureTextShape({ text: value || "M", fontSize: shape.fontSize });
+
+ useEffect(() => {
+ // Focus after the pointer gesture that created it has finished.
+ const id = window.setTimeout(() => ref.current?.focus(), 0);
+ return () => window.clearTimeout(id);
+ }, []);
+
+ const finish = () => {
+ if (done.current) return;
+ done.current = true;
+ onCommit(value);
+ };
+
+ return (
+ <textarea
+ ref={ref}
+ value={value}
+ data-testid="jarvisx-text-input"
+ onChange={(event) => setValue(event.target.value)}
+ onBlur={finish}
+ onKeyDown={(event) => {
+ event.stopPropagation();
+ if (event.key === "Enter" && !event.shiftKey) {
+ event.preventDefault();
+ finish();
+ } else if (event.key === "Escape") {
+ event.preventDefault();
+ finish();
+ }
+ }}
+ spellCheck={false}
+ rows={Math.max(1, value.split("\n").length)}
+ className="absolute resize-none overflow-hidden border border-dashed border-accent bg-transparent p-0 outline-none"
+ style={{
+ left: pos.x,
+ top: pos.y,
+ width: size.w * scale + 24,
+ height: size.h * scale + 4,
+ font: textFont(shape.fontSize * scale),
+ lineHeight: 1.25,
+ color: shape.color,
+ caretColor: shape.color,
+ }}
+ />
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXEditorWindow.tsx b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXEditorWindow.tsx
new file mode 100644
index 0000000000..67dbfd10cf
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXEditorWindow.tsx
@@ -0,0 +1,78 @@
+import { useCallback, useEffect, useState } from "react";
+
+import { ToastLayer } from "@/components/ToastLayer";
+import { EditContextMenu } from "@/components/EditContextMenu";
+import { useDesktopChrome, WindowControls } from "@/components/layout/WindowControls";
+import { hydrateUiLanguage } from "@/i18n";
+import { JarvisXEditor } from "@/views/jarvisx/JarvisXEditor";
+
+/** The view id the backend opens the editor window with. */
+export const JARVISX_EDITOR_VIEW = "jarvisx-editor";
+
+/** Is this document the Jarvis X editor window (`?view=jarvisx-editor`)? */
+export function isJarvisXEditorSearch(search: string): boolean {
+ return new URLSearchParams(search).get("view") === JARVISX_EDITOR_VIEW;
+}
+
+/**
+ * The editor as its own desktop window.
+ *
+ * Mounted by `main.tsx` INSTEAD of the app shell: the editor is not a section,
+ * so none of the shell's machinery (voice broker, websocket, onboarding, dock)
+ * has a job here, and a solo shell would have mounted the front page's voice
+ * broker for an unknown view id. The window keeps its title strip — the
+ * toolbar row doubles as the drag handle — and the toast layer.
+ */
+export function JarvisXEditorWindow() {
+ const params = new URLSearchParams(window.location.search);
+ const itemId = params.get("item");
+ // Verification fallback: a same-origin picture opened without a stored item.
+ const devSrc = itemId ? null : params.get("src");
+ const chrome = useDesktopChrome();
+ const controls = chrome.frameless ? chrome.controls : "none";
+
+ useEffect(() => {
+ void hydrateUiLanguage();
+ document.title = "Jarvis X";
+ }, []);
+
+ const [maximized, setMaximized] = useState(false);
+
+ // Always names THIS window. The shared `chrome.command` derives the target
+ // from the section store, which knows nothing about the editor, and a
+ // command without a known view lands on the main window.
+ const command = useCallback((action: "minimize" | "maximize" | "close") => {
+ void fetch("/api/window/command", {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ action, view: JARVISX_EDITOR_VIEW }),
+ })
+ .then((res) => (res.ok ? res.json() : null))
+ .then((body: { maximized?: unknown } | null) => {
+ if (body === null && action === "close") window.close();
+ if (body && typeof body.maximized === "boolean") setMaximized(body.maximized);
+ })
+ .catch(() => {
+ if (action === "close") window.close();
+ });
+ }, []);
+ const close = useCallback(() => command("close"), [command]);
+
+ const windowButtons = (side: "leading" | "trailing") =>
+ controls === side ? <WindowControls controls={controls} maximized={maximized} onCommand={command} /> : null;
+
+ return (
+ <div className="flex h-screen w-screen flex-col overflow-hidden bg-background text-foreground">
+ <JarvisXEditor
+ itemId={itemId}
+ devSrc={devSrc}
+ variant="window"
+ onClose={close}
+ leading={windowButtons("leading")}
+ trailing={windowButtons("trailing")}
+ />
+ <ToastLayer />
+ <EditContextMenu />
+ </div>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXSettingsPanel.tsx b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXSettingsPanel.tsx
new file mode 100644
index 0000000000..9d43171a84
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXSettingsPanel.tsx
@@ -0,0 +1,388 @@
+import { useCallback, useEffect, useMemo, useState, type ReactNode } from "react";
+import { Loader2 } from "lucide-react";
+
+import { Button } from "@/components/ui/button";
+import { Input } from "@/components/ui/input";
+import { Switch } from "@/components/ui/switch";
+import type { KeybindAction, KeybindsConfig, KeybindSaveResult } from "@/hooks/useHotkey";
+import { useT } from "@/i18n";
+import {
+ DISMISS_MAX_S,
+ DISMISS_MIN_S,
+ JARVISX_HOTKEY_ACTIONS,
+ clampDismissSeconds,
+ saveJarvisXSettings,
+ type JarvisXHotkeyAction,
+ type JarvisXSettings,
+ type JarvisXSettingsPatch,
+} from "@/lib/jarvisxApi";
+import { useEventStore } from "@/store/events";
+import { KeybindRow } from "@/views/settings/KeybindRow";
+
+/**
+ * The Jarvis X settings: the master switch, the six global shortcuts (the
+ * same recorder the voice shortcuts use), the corner thumbnail, where captures
+ * are saved, and the three after-capture effects.
+ *
+ * Every control saves on its own with a PUT of just its key; the backend's
+ * answer replaces the local copy, so what is on screen is what is stored.
+ */
+
+const HOTKEY_LABEL_KEY: Record<JarvisXHotkeyAction, string> = {
+ region: "jarvisx.settings.hotkey_region",
+ window: "jarvisx.settings.hotkey_window",
+ fullscreen: "jarvisx.settings.hotkey_fullscreen",
+ record_region: "jarvisx.settings.hotkey_record_region",
+ record_fullscreen: "jarvisx.settings.hotkey_record_fullscreen",
+ stop_recording: "jarvisx.settings.hotkey_stop_recording",
+};
+
+function Row({
+ label,
+ hint,
+ control,
+ children,
+ testId,
+}: {
+ label: string;
+ hint?: ReactNode;
+ control?: ReactNode;
+ children?: ReactNode;
+ testId?: string;
+}) {
+ return (
+ <div className="px-5 py-4" data-testid={testId}>
+ <div className="flex items-center justify-between gap-6">
+ <div className="min-w-0">
+ <p className="text-base font-medium text-foreground">{label}</p>
+ {hint && <p className="mt-0.5 text-sm text-muted-foreground">{hint}</p>}
+ </div>
+ {control && <div className="shrink-0">{control}</div>}
+ </div>
+ {children}
+ </div>
+ );
+}
+
+function Section({ title, children }: { title: string; children: ReactNode }) {
+ return (
+ <section>
+ <h2 className="mb-2 px-1 text-sm font-medium uppercase tracking-wide text-foreground-faint">{title}</h2>
+ <div className="divide-y divide-border rounded-xl border border-border bg-card">{children}</div>
+ </section>
+ );
+}
+
+export function JarvisXSettingsPanel({
+ settings,
+ error,
+ onRetry,
+ onSettings,
+}: {
+ settings: JarvisXSettings | null;
+ /** Why the settings could not be loaded, if they could not. */
+ error?: string | null;
+ onRetry?: () => void;
+ onSettings: (next: JarvisXSettings) => void;
+}) {
+ const t = useT();
+ const pushToast = useEventStore((s) => s.pushToast);
+ const [saving, setSaving] = useState(false);
+ const [hotkeyErrors, setHotkeyErrors] = useState<Partial<Record<JarvisXHotkeyAction, string>>>({});
+ const [dismissDraft, setDismissDraft] = useState<number | null>(null);
+ const [dirDraft, setDirDraft] = useState<string | null>(null);
+
+ const save = useCallback(
+ async (patch: JarvisXSettingsPatch) => {
+ if (!settings) return;
+ setSaving(true);
+ onSettings({ ...settings, ...patch, hotkeys: { ...settings.hotkeys, ...(patch.hotkeys ?? {}) } });
+ try {
+ onSettings(await saveJarvisXSettings(patch));
+ } catch (error) {
+ pushToast("error", t("jarvisx.settings.save_error").replace("{0}", (error as Error).message));
+ onSettings(settings);
+ } finally {
+ setSaving(false);
+ }
+ },
+ [onSettings, pushToast, settings, t],
+ );
+
+ // The recorder's config shape, filled with this feature's shortcuts.
+ const keybindConfig = useMemo<KeybindsConfig | null>(
+ () =>
+ settings
+ ? {
+ keybinds: (settings.hotkeys ?? {}) as unknown as KeybindsConfig["keybinds"],
+ defaults: {},
+ suggestions: [],
+ restart_required: false,
+ }
+ : null,
+ [settings],
+ );
+
+ const saveHotkey = useCallback(
+ async (action: KeybindAction, combo: string): Promise<KeybindSaveResult> => {
+ const name = action as unknown as JarvisXHotkeyAction;
+ try {
+ const next = await saveJarvisXSettings({ hotkeys: { [name]: combo } });
+ onSettings(next);
+ setHotkeyErrors((errors) => ({ ...errors, [name]: undefined }));
+ return { ok: true, action, hotkey: combo, persisted: true, restart_required: false };
+ } catch (error) {
+ // The backend's 422 reason stays under the row, not only in a toast.
+ setHotkeyErrors((errors) => ({ ...errors, [name]: (error as Error).message }));
+ throw error;
+ }
+ },
+ [onSettings],
+ );
+
+ const actionLabel = useCallback(
+ (action: string) => {
+ const key = HOTKEY_LABEL_KEY[action as JarvisXHotkeyAction];
+ return key ? t(key) : undefined;
+ },
+ [t],
+ );
+
+ useEffect(() => {
+ setDismissDraft(null);
+ setDirDraft(null);
+ }, [settings?.thumbnail_dismiss_s, settings?.save_dir]);
+
+ if (!settings && error) {
+ return (
+ <div className="flex flex-col items-start gap-3 rounded-xl border border-border bg-card p-5" role="alert">
+ <p className="text-base text-foreground">{t("jarvisx.settings.load_failed")}</p>
+ <p className="text-sm text-muted-foreground">{error}</p>
+ {onRetry && (
+ <Button type="button" size="sm" variant="secondary" onClick={onRetry}>
+ {t("common.retry")}
+ </Button>
+ )}
+ </div>
+ );
+ }
+
+ if (!settings) {
+ return (
+ <div className="flex h-40 items-center justify-center" role="status" aria-busy="true">
+ <Loader2 className="h-5 w-5 animate-spin text-muted-foreground" aria-hidden />
+ </div>
+ );
+ }
+
+ const off = !settings.enabled;
+ const dismiss = dismissDraft ?? settings.thumbnail_dismiss_s;
+ const commitDismiss = () => {
+ if (dismissDraft === null) return;
+ const next = clampDismissSeconds(dismissDraft);
+ setDismissDraft(null);
+ if (next !== settings.thumbnail_dismiss_s) void save({ thumbnail_dismiss_s: next });
+ };
+ const dir = dirDraft ?? settings.save_dir;
+ const commitDir = () => {
+ if (dirDraft === null) return;
+ const next = dirDraft.trim();
+ setDirDraft(null);
+ if (next !== settings.save_dir) void save({ save_dir: next });
+ };
+
+ const shortcutStatus = (action: JarvisXHotkeyAction): ReactNode => {
+ const error = hotkeyErrors[action];
+ if (error) {
+ return (
+ <p className="px-1 text-sm text-destructive" data-testid={`jarvisx-hotkey-error-${action}`}>
+ {error}
+ </p>
+ );
+ }
+ const combo = settings.hotkeys?.[action];
+ const status = settings.shortcuts?.[action];
+ if (!combo || off || !status) return null;
+ if (status.armed) {
+ return (
+ <p className="flex items-center gap-1.5 px-1 text-sm text-muted-foreground" data-testid={`jarvisx-hotkey-armed-${action}`}>
+ <span className="h-1.5 w-1.5 rounded-full bg-success" aria-hidden />
+ {t("jarvisx.settings.hotkey_armed")}
+ </p>
+ );
+ }
+ return (
+ <p className="flex items-center gap-1.5 px-1 text-sm text-warning" data-testid={`jarvisx-hotkey-unarmed-${action}`}>
+ <span className="h-1.5 w-1.5 rounded-full bg-warning" aria-hidden />
+ {status.detail
+ ? t("jarvisx.settings.hotkey_unarmed").replace("{0}", status.detail)
+ : t("jarvisx.settings.hotkey_unarmed_plain")}
+ </p>
+ );
+ };
+
+ return (
+ <div className="flex max-w-3xl flex-col gap-6" data-testid="jarvisx-settings">
+ <Section title={t("jarvisx.settings.section_general")}>
+ <Row
+ label={t("jarvisx.settings.enabled_label")}
+ hint={t("jarvisx.settings.enabled_hint")}
+ control={
+ <Switch
+ checked={settings.enabled}
+ disabled={saving}
+ aria-label={t("jarvisx.settings.enabled_label")}
+ data-testid="jarvisx-enabled"
+ onCheckedChange={(enabled) => void save({ enabled })}
+ />
+ }
+ />
+ </Section>
+
+ <section>
+ <h2 className="mb-2 px-1 text-sm font-medium uppercase tracking-wide text-foreground-faint">
+ {t("jarvisx.settings.section_shortcuts")}
+ </h2>
+ {off && <p className="mb-2 px-1 text-sm text-muted-foreground">{t("jarvisx.settings.shortcuts_off")}</p>}
+ <div className="flex flex-col gap-3">
+ {JARVISX_HOTKEY_ACTIONS.map((action) => (
+ <div key={action} className="flex flex-col gap-1">
+ <KeybindRow
+ action={action as unknown as KeybindAction}
+ variant="voice"
+ label={t(HOTKEY_LABEL_KEY[action])}
+ config={keybindConfig}
+ loading={false}
+ onSave={saveHotkey}
+ actionLabel={actionLabel}
+ />
+ {shortcutStatus(action)}
+ </div>
+ ))}
+ </div>
+ </section>
+
+ <Section title={t("jarvisx.settings.section_thumbnail")}>
+ <Row
+ label={t("jarvisx.settings.persist_label")}
+ hint={t("jarvisx.settings.persist_hint")}
+ control={
+ <Switch
+ checked={settings.thumbnail_persist}
+ disabled={saving}
+ aria-label={t("jarvisx.settings.persist_label")}
+ data-testid="jarvisx-persist"
+ onCheckedChange={(thumbnail_persist) => void save({ thumbnail_persist })}
+ />
+ }
+ />
+ {!settings.thumbnail_persist && (
+ <Row
+ testId="jarvisx-dismiss-row"
+ label={t("jarvisx.settings.dismiss_label")}
+ hint={t("jarvisx.settings.dismiss_hint").replace("{0}", String(dismiss))}
+ >
+ <div className="mt-3 flex items-center gap-4">
+ <input
+ type="range"
+ min={DISMISS_MIN_S}
+ max={DISMISS_MAX_S}
+ step={1}
+ value={dismiss}
+ aria-label={t("jarvisx.settings.dismiss_label")}
+ data-testid="jarvisx-dismiss-slider"
+ onChange={(event) => setDismissDraft(Number(event.target.value))}
+ onPointerUp={commitDismiss}
+ onKeyUp={commitDismiss}
+ onBlur={commitDismiss}
+ className="h-1.5 min-w-0 flex-1 cursor-pointer accent-[hsl(var(--accent))]"
+ />
+ <div className="flex shrink-0 items-center gap-1.5">
+ <Input
+ type="number"
+ min={DISMISS_MIN_S}
+ max={DISMISS_MAX_S}
+ value={dismiss}
+ aria-label={t("jarvisx.settings.dismiss_seconds")}
+ data-testid="jarvisx-dismiss-input"
+ onChange={(event) => setDismissDraft(Number(event.target.value))}
+ onBlur={commitDismiss}
+ onKeyDown={(event) => {
+ if (event.key === "Enter") commitDismiss();
+ }}
+ className="h-8 w-20 text-right tabular-nums"
+ />
+ <span className="text-sm text-muted-foreground">{t("jarvisx.settings.seconds_unit")}</span>
+ </div>
+ </div>
+ </Row>
+ )}
+ </Section>
+
+ <Section title={t("jarvisx.settings.section_after")}>
+ <Row label={t("jarvisx.settings.save_dir_label")} hint={t("jarvisx.settings.save_dir_hint")}>
+ <Input
+ value={dir}
+ spellCheck={false}
+ aria-label={t("jarvisx.settings.save_dir_label")}
+ data-testid="jarvisx-save-dir"
+ onChange={(event) => setDirDraft(event.target.value)}
+ onBlur={commitDir}
+ onKeyDown={(event) => {
+ if (event.key === "Enter") commitDir();
+ if (event.key === "Escape") setDirDraft(null);
+ }}
+ className="mt-3 font-mono text-sm"
+ />
+ </Row>
+ <Row
+ label={t("jarvisx.settings.clipboard_label")}
+ hint={t("jarvisx.settings.clipboard_hint")}
+ control={
+ <Switch
+ checked={settings.copy_to_clipboard}
+ disabled={saving}
+ aria-label={t("jarvisx.settings.clipboard_label")}
+ data-testid="jarvisx-clipboard"
+ onCheckedChange={(copy_to_clipboard) => void save({ copy_to_clipboard })}
+ />
+ }
+ />
+ <Row
+ label={t("jarvisx.settings.sound_label")}
+ control={
+ <Switch
+ checked={settings.sound}
+ disabled={saving}
+ aria-label={t("jarvisx.settings.sound_label")}
+ data-testid="jarvisx-sound"
+ onCheckedChange={(sound) => void save({ sound })}
+ />
+ }
+ />
+ <Row
+ label={t("jarvisx.settings.effect_label")}
+ hint={t("jarvisx.settings.effect_hint")}
+ control={
+ <Switch
+ checked={settings.effect}
+ disabled={saving}
+ aria-label={t("jarvisx.settings.effect_label")}
+ data-testid="jarvisx-effect"
+ onCheckedChange={(effect) => void save({ effect })}
+ />
+ }
+ />
+ </Section>
+
+ {!settings.recording_available && (
+ <p className="rounded-lg border border-border bg-card px-4 py-3 text-sm text-muted-foreground" data-testid="jarvisx-recording-unavailable">
+ {t("jarvisx.settings.recording_unavailable").replace(
+ "{0}",
+ settings.recording_detail || t("jarvisx.recording_unavailable"),
+ )}
+ </p>
+ )}
+ </div>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXView.test.tsx b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXView.test.tsx
new file mode 100644
index 0000000000..51e2170ec6
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXView.test.tsx
@@ -0,0 +1,185 @@
+import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+import { useEventStore } from "@/store/events";
+import { JarvisXView } from "@/views/jarvisx/JarvisXView";
+
+const SETTINGS = {
+ enabled: true,
+ hotkeys: {
+ region: "ctrl+shift+4",
+ window: "ctrl+shift+5",
+ fullscreen: "ctrl+shift+3",
+ record_region: "ctrl+shift+6",
+ record_fullscreen: "",
+ stop_recording: "ctrl+shift+7",
+ },
+ thumbnail_persist: false,
+ thumbnail_dismiss_s: 30,
+ save_dir: "/home/me/Pictures/Jarvis X",
+ copy_to_clipboard: true,
+ sound: true,
+ effect: true,
+ recording_available: true,
+ recording_detail: "",
+ shortcuts: {
+ region: { hotkey: "ctrl+shift+4", armed: true, detail: "" },
+ window: { hotkey: "ctrl+shift+5", armed: false, detail: "taken by another app" },
+ },
+};
+
+const ITEMS = [
+ {
+ id: "old",
+ kind: "image",
+ mode: "window",
+ created_at: "2026-09-01T10:00:00Z",
+ width: 800,
+ height: 600,
+ duration_s: null,
+ filename: "old.png",
+ url: "/api/jarvisx/items/old/file",
+ thumb_url: "/api/jarvisx/items/old/thumb",
+ edited_url: null,
+ },
+ {
+ id: "vid",
+ kind: "video",
+ mode: "fullscreen",
+ created_at: "2026-09-29T10:00:00Z",
+ width: 1920,
+ height: 1080,
+ duration_s: 83,
+ filename: "vid.mp4",
+ url: "/api/jarvisx/items/vid/file",
+ thumb_url: "/api/jarvisx/items/vid/thumb",
+ edited_url: null,
+ },
+];
+
+function json(body: unknown, status = 200): Response {
+ return { ok: status < 400, status, json: async () => body } as Response;
+}
+
+describe("JarvisXView", () => {
+ let fetchMock: ReturnType<typeof vi.fn>;
+ let items: typeof ITEMS;
+
+ beforeEach(() => {
+ items = [...ITEMS];
+ useEventStore.setState({ events: [], toasts: [], assistantName: "Jarvis" });
+ fetchMock = vi.fn(async (url: string, init?: RequestInit) => {
+ if (url === "/api/jarvisx/settings" && init?.method === "PUT") {
+ const patch = JSON.parse(String(init.body));
+ return json({ ...SETTINGS, ...patch, hotkeys: { ...SETTINGS.hotkeys, ...(patch.hotkeys ?? {}) } });
+ }
+ if (url === "/api/jarvisx/settings") return json(SETTINGS);
+ if (url.startsWith("/api/jarvisx/items?")) return json({ items });
+ if (url === "/api/jarvisx/record/status") return json({ recording: false, mode: null, elapsed_s: 0 });
+ if (url === "/api/jarvisx/items/old" && init?.method === "DELETE") {
+ items = items.filter((i) => i.id !== "old");
+ return json({ ok: true });
+ }
+ return json({}, 404);
+ });
+ vi.stubGlobal("fetch", fetchMock);
+ });
+
+ afterEach(() => {
+ cleanup();
+ vi.unstubAllGlobals();
+ });
+
+ it("lists captures newest first with a video duration", async () => {
+ render(<JarvisXView />);
+ const cards = await screen.findAllByTestId("jarvisx-item");
+ expect(cards).toHaveLength(2);
+ expect(cards[0].textContent).toContain("1:23");
+ expect(screen.getByTestId("jarvisx-capture-region")).toBeDefined();
+ expect(screen.getByTestId("jarvisx-record-region")).toBeDefined();
+ });
+
+ it("deletes only after the in-app confirmation", async () => {
+ const confirmSpy = vi.fn();
+ vi.stubGlobal("confirm", confirmSpy);
+ render(<JarvisXView />);
+ await screen.findAllByTestId("jarvisx-item");
+ const deleteButtons = screen.getAllByTestId("jarvisx-item-delete");
+ fireEvent.click(deleteButtons[1]);
+ expect(screen.getByTestId("jarvisx-confirm")).toBeDefined();
+ expect(fetchMock.mock.calls.some(([, init]) => init?.method === "DELETE")).toBe(false);
+ fireEvent.click(screen.getByTestId("jarvisx-confirm-ok"));
+ await waitFor(() => expect(screen.getAllByTestId("jarvisx-item")).toHaveLength(1));
+ expect(confirmSpy).not.toHaveBeenCalled();
+ });
+
+ it("refreshes the library when a capture event arrives", async () => {
+ render(<JarvisXView />);
+ await screen.findAllByTestId("jarvisx-item");
+ const before = fetchMock.mock.calls.filter(([url]) => String(url).startsWith("/api/jarvisx/items?")).length;
+ act(() => {
+ useEventStore.getState().pushEvent({
+ id: "e1",
+ name: "JarvisXItemCreated",
+ ts: Date.now(),
+ payload: { id: "new", kind: "image", mode: "region" },
+ });
+ });
+ await waitFor(() =>
+ expect(fetchMock.mock.calls.filter(([url]) => String(url).startsWith("/api/jarvisx/items?")).length).toBe(
+ before + 1,
+ ),
+ );
+ });
+
+ it("shows a recording indicator from the websocket event", async () => {
+ render(<JarvisXView />);
+ await screen.findAllByTestId("jarvisx-item");
+ act(() => {
+ useEventStore.getState().pushEvent({
+ id: "e2",
+ name: "JarvisXRecordingChanged",
+ ts: Date.now(),
+ payload: { recording: true, mode: "region", elapsed_s: 5 },
+ });
+ });
+ expect(await screen.findByTestId("jarvisx-recording")).toBeDefined();
+ expect(screen.getByTestId("jarvisx-record-stop")).toBeDefined();
+ });
+
+ it("hides the seconds field while the thumbnail stays until dismissed", async () => {
+ render(<JarvisXView />);
+ fireEvent.click(await screen.findByText("Settings"));
+ const persist = await screen.findByTestId("jarvisx-persist");
+ expect(screen.getByTestId("jarvisx-dismiss-row")).toBeDefined();
+ fireEvent.click(persist);
+ await waitFor(() => expect(screen.queryByTestId("jarvisx-dismiss-row")).toBeNull());
+ const put = fetchMock.mock.calls.find(
+ ([url, init]) => url === "/api/jarvisx/settings" && init?.method === "PUT",
+ );
+ expect(JSON.parse(String(put![1].body))).toEqual({ thumbnail_persist: true });
+ fireEvent.click(screen.getByTestId("jarvisx-persist"));
+ await waitFor(() => expect(screen.getByTestId("jarvisx-dismiss-row")).toBeDefined());
+ });
+
+ it("saves a clamped dismiss delay", async () => {
+ render(<JarvisXView />);
+ fireEvent.click(await screen.findByText("Settings"));
+ const input = await screen.findByTestId("jarvisx-dismiss-input");
+ fireEvent.change(input, { target: { value: "999" } });
+ fireEvent.blur(input);
+ await waitFor(() => {
+ const put = fetchMock.mock.calls.find(
+ ([url, init]) => url === "/api/jarvisx/settings" && init?.method === "PUT",
+ );
+ expect(JSON.parse(String(put![1].body))).toEqual({ thumbnail_dismiss_s: 300 });
+ });
+ });
+
+ it("shows whether each shortcut is armed", async () => {
+ render(<JarvisXView />);
+ fireEvent.click(await screen.findByText("Settings"));
+ expect(await screen.findByTestId("jarvisx-hotkey-armed-region")).toBeDefined();
+ expect(screen.getByTestId("jarvisx-hotkey-unarmed-window").textContent).toContain("taken by another app");
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXView.tsx b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXView.tsx
new file mode 100644
index 0000000000..bfebb0032f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/jarvisx/JarvisXView.tsx
@@ -0,0 +1,558 @@
+import { useCallback, useEffect, useMemo, useRef, useState } from "react";
+import {
+ AppWindow,
+ Aperture,
+ Copy,
+ Crop,
+ ExternalLink,
+ FolderOpen,
+ Loader2,
+ Monitor,
+ Pencil,
+ Play,
+ Square,
+ Trash2,
+ Video,
+} from "lucide-react";
+
+import { PageHeader } from "@/components/layout/PageHeader";
+import { TabBar } from "@/components/layout/SectionTabBar";
+import { Button } from "@/components/ui/button";
+import { QuickTooltip } from "@/components/ui/tooltip";
+import { useT } from "@/i18n";
+import {
+ JARVISX_EVENTS,
+ captureJarvisX,
+ copyJarvisXItem,
+ deleteJarvisXItem,
+ fetchJarvisXItems,
+ fetchJarvisXRecordStatus,
+ fetchJarvisXSettings,
+ formatDuration,
+ isJarvisXEvent,
+ normalizeRecordStatus,
+ openJarvisXEditor,
+ revealJarvisXItem,
+ startJarvisXRecording,
+ stopJarvisXRecording,
+ withVersion,
+ type JarvisXItem,
+ type JarvisXMode,
+ type JarvisXRecordMode,
+ type JarvisXRecordStatus,
+ type JarvisXSettings,
+} from "@/lib/jarvisxApi";
+import { cn } from "@/lib/utils";
+import { useEventStore } from "@/store/events";
+import { JarvisXConfirmDialog } from "@/views/jarvisx/JarvisXConfirmDialog";
+import { JarvisXEditor } from "@/views/jarvisx/JarvisXEditor";
+import { JarvisXSettingsPanel } from "@/views/jarvisx/JarvisXSettingsPanel";
+
+/**
+ * Jarvis X — screenshots and screen recordings, without the assistant.
+ *
+ * Two tabs: the library of everything captured (newest first, live over the
+ * websocket) with the capture and record buttons on top, and the settings
+ * (shortcuts, the corner thumbnail, the save folder). A click on a capture
+ * opens the annotation editor over the page; "Open in window" hands the same
+ * editor to its own desktop window.
+ */
+
+type Tab = "library" | "settings";
+
+function relativeTime(iso: string, t: (key: string) => string, now: number): string {
+ const ms = Date.parse(iso);
+ if (Number.isNaN(ms)) return "";
+ const diff = Math.max(0, Math.round((now - ms) / 1000));
+ if (diff < 45) return t("jarvisx.time_now");
+ if (diff < 3600) return t("jarvisx.time_minutes").replace("{0}", String(Math.max(1, Math.round(diff / 60))));
+ const date = new Date(ms);
+ const sameDay = new Date(now).toDateString() === date.toDateString();
+ return sameDay
+ ? date.toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" })
+ : date.toLocaleDateString([], { day: "numeric", month: "short", hour: "2-digit", minute: "2-digit" });
+}
+
+export function JarvisXView() {
+ const t = useT();
+ const [tab, setTab] = useState<Tab>("library");
+ const [settings, setSettings] = useState<JarvisXSettings | null>(null);
+ const [settingsError, setSettingsError] = useState<string | null>(null);
+ const [settingsAttempt, setSettingsAttempt] = useState(0);
+
+ useEffect(() => {
+ let live = true;
+ setSettingsError(null);
+ fetchJarvisXSettings()
+ .then((next) => live && setSettings(next))
+ .catch((error: Error) => live && setSettingsError(error.message));
+ return () => {
+ live = false;
+ };
+ }, [settingsAttempt]);
+
+ return (
+ <div data-testid="jarvisx-view" className="flex h-full flex-col overflow-y-auto bg-background px-8 pb-10 scrollbar-jarvis">
+ <div className="w-full max-w-[1400px]">
+ <PageHeader
+ icon={<Aperture />}
+ title={t("jarvisx.title")}
+ description={t("jarvisx.subtitle")}
+ tabs={
+ <TabBar
+ tabs={[
+ { id: "library", label: t("jarvisx.tab_library") },
+ { id: "settings", label: t("jarvisx.tab_settings") },
+ ]}
+ active={tab}
+ onChange={(id) => setTab(id as Tab)}
+ />
+ }
+ />
+ <div className="pt-5">
+ {tab === "library" ? (
+ <JarvisXLibrary settings={settings} />
+ ) : (
+ <JarvisXSettingsPanel
+ settings={settings}
+ error={settingsError}
+ onRetry={() => setSettingsAttempt((n) => n + 1)}
+ onSettings={setSettings}
+ />
+ )}
+ </div>
+ </div>
+ </div>
+ );
+}
+
+function CaptureButton({
+ icon,
+ label,
+ disabled,
+ busy,
+ onClick,
+ testId,
+ title,
+}: {
+ icon: React.ReactNode;
+ label: string;
+ disabled?: boolean;
+ busy?: boolean;
+ onClick: () => void;
+ testId: string;
+ title?: string;
+}) {
+ return (
+ <Button type="button" variant="outline" size="sm" disabled={disabled} onClick={onClick} data-testid={testId} title={title}>
+ {busy ? <Loader2 className="animate-spin" aria-hidden /> : icon}
+ {label}
+ </Button>
+ );
+}
+
+export function JarvisXLibrary({ settings }: { settings: JarvisXSettings | null }) {
+ const t = useT();
+ const pushToast = useEventStore((s) => s.pushToast);
+ // The newest Jarvis X event — its id changes on every capture, edit, delete
+ // or recording change, which is all this view needs to know to refresh.
+ const lastEvent = useEventStore((s) => s.events.find((event) => isJarvisXEvent(event.name)));
+ const [items, setItems] = useState<JarvisXItem[] | null>(null);
+ const [error, setError] = useState<string | null>(null);
+ const [record, setRecord] = useState<JarvisXRecordStatus>({ recording: false, mode: null, elapsed_s: 0 });
+ const [recordSince, setRecordSince] = useState(() => Date.now());
+ const [now, setNow] = useState(() => Date.now());
+ const [busy, setBusy] = useState<string | null>(null);
+ const [editing, setEditing] = useState<string | null>(null);
+ const [confirmDelete, setConfirmDelete] = useState<JarvisXItem | null>(null);
+ const versions = useRef(new Map<string, number>());
+
+ const load = useCallback(async () => {
+ try {
+ setItems(await fetchJarvisXItems(100));
+ setError(null);
+ } catch (err) {
+ setError((err as Error).message);
+ setItems((current) => current ?? []);
+ }
+ }, []);
+
+ useEffect(() => {
+ void load();
+ fetchJarvisXRecordStatus()
+ .then((status) => {
+ setRecord(status);
+ setRecordSince(Date.now());
+ })
+ .catch(() => undefined);
+ }, [load]);
+
+ useEffect(() => {
+ if (!lastEvent) return;
+ if (lastEvent.name === JARVISX_EVENTS.recording) {
+ setRecord(normalizeRecordStatus(lastEvent.payload));
+ setRecordSince(Date.now());
+ return;
+ }
+ if (lastEvent.name === JARVISX_EVENTS.updated) {
+ const id = (lastEvent.payload as { id?: unknown } | undefined)?.id;
+ if (typeof id === "string") versions.current.set(id, (versions.current.get(id) ?? 0) + 1);
+ }
+ void load();
+ }, [lastEvent, load]);
+
+ // One clock for the relative times and the recording timer.
+ useEffect(() => {
+ const id = window.setInterval(() => setNow(Date.now()), record.recording ? 1000 : 30_000);
+ return () => window.clearInterval(id);
+ }, [record.recording]);
+
+ const run = useCallback(
+ async (key: string, action: () => Promise<{ ok?: boolean; message?: string } | void>, success?: string) => {
+ setBusy(key);
+ try {
+ const result = await action();
+ if (result && result.ok === false) pushToast("warning", result.message || t("jarvisx.action_failed"));
+ else if (success) pushToast("success", success);
+ } catch (err) {
+ pushToast("error", (err as Error).message);
+ } finally {
+ setBusy(null);
+ }
+ },
+ [pushToast, t],
+ );
+
+ const capture = (mode: JarvisXMode) =>
+ run(`capture-${mode}`, async () => {
+ const result = await captureJarvisX(mode);
+ if (result.ok && result.item) {
+ setItems((current) => [result.item!, ...(current ?? []).filter((i) => i.id !== result.item!.id)]);
+ }
+ return result;
+ });
+
+ const startRecording = (mode: JarvisXRecordMode) =>
+ run(`record-${mode}`, async () => {
+ const result = await startJarvisXRecording(mode);
+ if (result.ok !== false) {
+ setRecord({ recording: true, mode, elapsed_s: 0 });
+ setRecordSince(Date.now());
+ }
+ return result;
+ });
+
+ const stopRecording = () =>
+ run("record-stop", async () => {
+ const result = await stopJarvisXRecording();
+ if (result.ok !== false) setRecord({ recording: false, mode: null, elapsed_s: 0 });
+ return result;
+ });
+
+ const openInWindow = (item: JarvisXItem) =>
+ run(`open-${item.id}`, async () => {
+ try {
+ const result = await openJarvisXEditor(item.id);
+ if (result.ok === false) setEditing(item.id);
+ } catch {
+ // No desktop shell to open a window in: edit right here instead.
+ setEditing(item.id);
+ }
+ });
+
+ const remove = async (item: JarvisXItem) => {
+ setConfirmDelete(null);
+ await run(`delete-${item.id}`, async () => {
+ const result = await deleteJarvisXItem(item.id);
+ setItems((current) => (current ?? []).filter((i) => i.id !== item.id));
+ return result;
+ }, t("jarvisx.deleted"));
+ };
+
+ const disabled = settings !== null && !settings.enabled;
+ const recordingAvailable = settings?.recording_available ?? true;
+ const recordingTitle = !recordingAvailable ? settings?.recording_detail || t("jarvisx.recording_unavailable") : undefined;
+ const elapsed = record.elapsed_s + (record.recording ? (now - recordSince) / 1000 : 0);
+
+ const sorted = useMemo(() => items ?? [], [items]);
+
+ return (
+ <div className="flex flex-col gap-5">
+ <div className="flex flex-wrap items-center gap-2 rounded-xl border border-border bg-card p-3">
+ <CaptureButton
+ icon={<Crop aria-hidden />}
+ label={t("jarvisx.capture_region")}
+ disabled={disabled || busy !== null}
+ busy={busy === "capture-region"}
+ onClick={() => void capture("region")}
+ testId="jarvisx-capture-region"
+ />
+ <CaptureButton
+ icon={<AppWindow aria-hidden />}
+ label={t("jarvisx.capture_window")}
+ disabled={disabled || busy !== null}
+ busy={busy === "capture-window"}
+ onClick={() => void capture("window")}
+ testId="jarvisx-capture-window"
+ />
+ <CaptureButton
+ icon={<Monitor aria-hidden />}
+ label={t("jarvisx.capture_fullscreen")}
+ disabled={disabled || busy !== null}
+ busy={busy === "capture-fullscreen"}
+ onClick={() => void capture("fullscreen")}
+ testId="jarvisx-capture-fullscreen"
+ />
+ <span className="mx-1 hidden h-6 w-px bg-border sm:block" aria-hidden />
+ {record.recording ? (
+ <div className="flex items-center gap-3" data-testid="jarvisx-recording">
+ <span className="flex items-center gap-2 text-sm font-medium text-foreground" role="status">
+ <span className="relative flex h-2.5 w-2.5">
+ <span className="absolute inline-flex h-full w-full animate-ping rounded-full bg-destructive opacity-60 motion-reduce:animate-none" />
+ <span className="relative inline-flex h-2.5 w-2.5 rounded-full bg-destructive" />
+ </span>
+ {t("jarvisx.recording")}
+ <span className="tabular-nums text-muted-foreground">{formatDuration(elapsed)}</span>
+ </span>
+ <Button
+ type="button"
+ size="sm"
+ variant="destructive"
+ disabled={busy === "record-stop"}
+ onClick={() => void stopRecording()}
+ data-testid="jarvisx-record-stop"
+ >
+ <Square aria-hidden className="fill-current" />
+ {t("jarvisx.record_stop")}
+ </Button>
+ </div>
+ ) : (
+ <>
+ <CaptureButton
+ icon={<Video aria-hidden />}
+ label={t("jarvisx.record_region")}
+ disabled={disabled || busy !== null || !recordingAvailable}
+ busy={busy === "record-region"}
+ onClick={() => void startRecording("region")}
+ testId="jarvisx-record-region"
+ title={recordingTitle}
+ />
+ <CaptureButton
+ icon={<Video aria-hidden />}
+ label={t("jarvisx.record_fullscreen")}
+ disabled={disabled || busy !== null || !recordingAvailable}
+ busy={busy === "record-fullscreen"}
+ onClick={() => void startRecording("fullscreen")}
+ testId="jarvisx-record-fullscreen"
+ title={recordingTitle}
+ />
+ </>
+ )}
+ {disabled && <span className="ml-auto text-sm text-muted-foreground">{t("jarvisx.disabled_hint")}</span>}
+ </div>
+
+ {error && (
+ <p role="alert" className="rounded-lg border border-border bg-card px-4 py-3 text-sm text-muted-foreground">
+ {t("jarvisx.load_failed").replace("{0}", error)}
+ </p>
+ )}
+
+ {items === null ? (
+ <div className="flex h-40 items-center justify-center" role="status" aria-busy="true">
+ <Loader2 className="h-5 w-5 animate-spin text-muted-foreground" aria-hidden />
+ </div>
+ ) : sorted.length === 0 ? (
+ !error && (
+ <div className="flex flex-col items-center justify-center rounded-xl border border-dashed border-border px-6 py-16 text-center">
+ <Aperture className="h-8 w-8 text-muted-foreground" aria-hidden />
+ <p className="mt-3 text-base font-medium text-foreground">{t("jarvisx.empty_title")}</p>
+ <p className="mt-1 max-w-md text-sm text-muted-foreground">{t("jarvisx.empty_body")}</p>
+ </div>
+ )
+ ) : (
+ <ul data-testid="jarvisx-grid" className="grid grid-cols-[repeat(auto-fill,minmax(220px,1fr))] gap-4">
+ {sorted.map((item) => (
+ <LibraryCard
+ key={item.id}
+ item={item}
+ version={versions.current.get(item.id) ?? 0}
+ now={now}
+ busy={busy?.endsWith(item.id) ?? false}
+ onOpen={() => setEditing(item.id)}
+ onOpenWindow={() => void openInWindow(item)}
+ onCopy={() =>
+ void run(`copy-${item.id}`, () => copyJarvisXItem(item.id, Boolean(item.edited_url)), t("jarvisx.copied"))
+ }
+ onReveal={() => void run(`reveal-${item.id}`, () => revealJarvisXItem(item.id))}
+ onDelete={() => setConfirmDelete(item)}
+ />
+ ))}
+ </ul>
+ )}
+
+ {confirmDelete && (
+ <JarvisXConfirmDialog
+ title={t("jarvisx.delete_title")}
+ body={t("jarvisx.delete_body")}
+ confirmLabel={t("jarvisx.delete")}
+ onCancel={() => setConfirmDelete(null)}
+ onConfirm={() => void remove(confirmDelete)}
+ />
+ )}
+
+ {editing && (
+ <div
+ className="fixed inset-0 z-[150] flex bg-scrim/60 p-4 md:p-8"
+ role="dialog"
+ aria-modal="true"
+ aria-label={t("jarvisx.editor.title")}
+ >
+ <JarvisXEditor
+ itemId={editing}
+ variant="overlay"
+ onClose={() => {
+ setEditing(null);
+ void load();
+ }}
+ />
+ </div>
+ )}
+ </div>
+ );
+}
+
+function LibraryCard({
+ item,
+ version,
+ now,
+ busy,
+ onOpen,
+ onOpenWindow,
+ onCopy,
+ onReveal,
+ onDelete,
+}: {
+ item: JarvisXItem;
+ version: number;
+ now: number;
+ busy: boolean;
+ onOpen: () => void;
+ onOpenWindow: () => void;
+ onCopy: () => void;
+ onReveal: () => void;
+ onDelete: () => void;
+}) {
+ const t = useT();
+ const [broken, setBroken] = useState(false);
+ const isVideo = item.kind === "video";
+ const thumb = version ? withVersion(item.thumb_url, version) : item.thumb_url;
+ const modeLabel = t(`jarvisx.mode_${item.mode}`);
+
+ return (
+ <li
+ data-testid="jarvisx-item"
+ className="group relative flex flex-col overflow-hidden rounded-xl border border-border bg-card transition-colors hover:border-border-strong focus-within:border-border-strong"
+ >
+ <button
+ type="button"
+ onClick={onOpen}
+ aria-label={t(isVideo ? "jarvisx.open_video" : "jarvisx.open_editor").replace("{0}", item.filename)}
+ className="relative block aspect-[16/10] w-full overflow-hidden bg-secondary focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-ring"
+ >
+ {broken ? (
+ <span className="flex h-full w-full items-center justify-center text-muted-foreground">
+ {isVideo ? <Video className="h-6 w-6" aria-hidden /> : <Aperture className="h-6 w-6" aria-hidden />}
+ </span>
+ ) : (
+ <img
+ src={thumb}
+ alt=""
+ loading="lazy"
+ draggable={false}
+ onError={() => setBroken(true)}
+ className="h-full w-full object-contain"
+ />
+ )}
+ <span className="absolute bottom-2 left-2 flex items-center gap-1">
+ <span className="rounded bg-scrim/70 px-1.5 py-0.5 text-micro font-medium text-white">{modeLabel}</span>
+ {item.edited_url && (
+ <span className="rounded bg-accent px-1.5 py-0.5 text-micro font-medium text-accent-foreground">
+ {t("jarvisx.edited")}
+ </span>
+ )}
+ </span>
+ {isVideo && (
+ <span className="absolute bottom-2 right-2 flex items-center gap-1 rounded bg-scrim/70 px-1.5 py-0.5 text-micro font-medium tabular-nums text-white">
+ <Play className="h-3 w-3 fill-current" aria-hidden />
+ {formatDuration(item.duration_s) || t("jarvisx.kind_video")}
+ </span>
+ )}
+ </button>
+ <div className="flex items-center justify-between gap-2 px-3 py-2.5">
+ <div className="min-w-0">
+ <p className="truncate text-sm font-medium text-foreground" title={item.filename}>
+ {relativeTime(item.created_at, t, now)}
+ </p>
+ <p className="truncate text-micro text-muted-foreground">
+ {[isVideo ? t("jarvisx.kind_video") : t("jarvisx.kind_image"), item.width && item.height ? `${item.width} × ${item.height}` : ""]
+ .filter(Boolean)
+ .join(" · ")}
+ </p>
+ </div>
+ {busy && <Loader2 className="h-4 w-4 shrink-0 animate-spin text-muted-foreground" aria-hidden />}
+ </div>
+ {/* Actions float over the picture on hover or keyboard focus. */}
+ <div className="pointer-events-none absolute right-2 top-2 flex gap-1 opacity-0 transition-opacity group-hover:pointer-events-auto group-hover:opacity-100 group-focus-within:pointer-events-auto group-focus-within:opacity-100">
+ {!isVideo && (
+ <CardAction label={t("jarvisx.edit")} onClick={onOpen} testId="jarvisx-item-edit">
+ <Pencil aria-hidden />
+ </CardAction>
+ )}
+ <CardAction label={t("jarvisx.open_window")} onClick={onOpenWindow} testId="jarvisx-item-window">
+ <ExternalLink aria-hidden />
+ </CardAction>
+ <CardAction label={t("jarvisx.copy")} onClick={onCopy} testId="jarvisx-item-copy">
+ <Copy aria-hidden />
+ </CardAction>
+ <CardAction label={t("jarvisx.reveal")} onClick={onReveal} testId="jarvisx-item-reveal">
+ <FolderOpen aria-hidden />
+ </CardAction>
+ <CardAction label={t("jarvisx.delete")} onClick={onDelete} testId="jarvisx-item-delete" danger>
+ <Trash2 aria-hidden />
+ </CardAction>
+ </div>
+ </li>
+ );
+}
+
+function CardAction({
+ label,
+ onClick,
+ children,
+ testId,
+ danger,
+}: {
+ label: string;
+ onClick: () => void;
+ children: React.ReactNode;
+ testId: string;
+ danger?: boolean;
+}) {
+ return (
+ <QuickTooltip content={label} side="bottom">
+ <button
+ type="button"
+ aria-label={label}
+ onClick={onClick}
+ data-testid={testId}
+ className={cn(
+ "grid h-7 w-7 place-items-center rounded-md border border-border bg-popover text-foreground shadow-sm transition-colors [&>svg]:h-3.5 [&>svg]:w-3.5",
+ "hover:bg-secondary focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring",
+ danger && "hover:text-destructive",
+ )}
+ >
+ {children}
+ </button>
+ </QuickTooltip>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/settings/KeybindRow.tsx b/jarvis/ui/web/frontend/src/views/settings/KeybindRow.tsx
index 530dcbe52f..16a7e66f7f 100644
--- a/jarvis/ui/web/frontend/src/views/settings/KeybindRow.tsx
+++ b/jarvis/ui/web/frontend/src/views/settings/KeybindRow.tsx
@@ -194,14 +194,16 @@ export function ComboChips({ combo }: { combo: string }) {
export function validationText(
v: ComboValidation,
t: (key: string) => string,
+ actionLabel?: (action: string) => string | undefined,
): string | null {
if (v.status === "error" && v.reason === "collision" && v.conflict) {
// The conflict carries the ACTION ID (unique); the message names it the way
// the UI labels that row. An id the frontend does not know yet (a newer
// backend) falls back to the raw id instead of rendering an empty name.
const labelKey = ACTION_LABEL_KEY[v.conflict.action as KeybindAction];
+ const named = actionLabel?.(v.conflict.action) ?? (labelKey ? t(labelKey) : undefined);
return t("settings_view.keybinds.validation.collision")
- .replace("{action}", labelKey ? t(labelKey) : v.conflict.action)
+ .replace("{action}", named ?? v.conflict.action)
.replace("{combo}", formatCombo(v.conflict.combo));
}
const cautions = v.cautions ?? [];
@@ -235,6 +237,12 @@ export interface KeybindRowProps {
* pins the dictation mode to "hold"). Failures are the owner's to report.
*/
onSaved?: (combo: string) => void | Promise<void>;
+ /**
+ * Names a sibling row for "already used by …" when the rows are not the
+ * voice keybinds (whose labels `ACTION_LABEL_KEY` knows) — another feature's
+ * shortcut list reusing this recorder.
+ */
+ actionLabel?: (action: string) => string | undefined;
}
/**
@@ -253,6 +261,7 @@ export function KeybindRow({
variant = "settings",
suggestions,
onSaved,
+ actionLabel,
}: KeybindRowProps) {
const t = useT();
const pushToast = useEventStore((s) => s.pushToast);
@@ -287,11 +296,11 @@ export function KeybindRow({
for (const [act, c] of Object.entries(config.keybinds)) {
if (act === action) continue;
const labelKey = ACTION_LABEL_KEY[act as KeybindAction];
- const lbl = labelKey ? t(labelKey) : act;
+ const lbl = actionLabel?.(act) ?? (labelKey ? t(labelKey) : act);
for (const tok of comboTokens(c ?? "")) out[tok] = lbl;
}
return out;
- }, [config, action, t]);
+ }, [config, action, t, actionLabel]);
// The OTHER actions' combos keyed by their ACTION ID. Keying by the translated
// label collapsed two rows that happen to share a label into one entry, so one
@@ -316,7 +325,7 @@ export function KeybindRow({
() => validateCombo(combo, otherCombos),
[combo, otherCombos],
);
- const validationMsg = validationText(validation, t);
+ const validationMsg = validationText(validation, t, actionLabel);
// Click-to-assign: toggle a key in/out of the combo without a physical press.
// Functional update — toggles dispatched before the next render must each
diff --git a/jarvis/ui/web/jarvisx_routes.py b/jarvis/ui/web/jarvisx_routes.py
new file mode 100644
index 0000000000..a6456ac17f
--- /dev/null
+++ b/jarvis/ui/web/jarvisx_routes.py
@@ -0,0 +1,405 @@
+"""REST API for Jarvis X — the built-in screenshot and screen-recording tool.
+
+Endpoints (mounted by the WebServer in ``_build_app()``):
+
+ GET /api/jarvisx/settings → switches, shortcuts, readiness.
+ PUT /api/jarvisx/settings → change any subset; shortcuts re-arm live.
+ GET /api/jarvisx/items?limit=100 → the library, newest first.
+ GET /api/jarvisx/items/{id} → one item.
+ GET /api/jarvisx/items/{id}/file → the original PNG / video (Range-aware).
+ GET /api/jarvisx/items/{id}/thumb → a small JPEG preview.
+ GET /api/jarvisx/items/{id}/edited → the annotated PNG (404 when none).
+ PUT /api/jarvisx/items/{id}/edited → save an annotated PNG (raw image/png body).
+ POST /api/jarvisx/items/{id}/copy → copy the picture to the OS clipboard.
+ POST /api/jarvisx/items/{id}/reveal → show the file in the OS file manager.
+ POST /api/jarvisx/items/{id}/open-editor → open the annotation editor window.
+ DELETE /api/jarvisx/items/{id} → delete the files and the entry.
+ POST /api/jarvisx/capture → take a screenshot (region/window/fullscreen).
+ POST /api/jarvisx/record/start → start recording (region/fullscreen).
+ POST /api/jarvisx/record/stop → stop and save the recording.
+ GET /api/jarvisx/record/status → is a recording running, and for how long.
+
+Under the CLI-first contract every route here is also a ``jarvis api jarvisx
+<op>`` command (``operation_id`` = the command name). Clipboard, reveal and the
+editor act on the machine the backend runs on, so they answer ``ok: false``
+on a headless server rather than acting on a desktop nobody sees.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from pathlib import Path
+from typing import Annotated, Any, Literal
+
+from fastapi import APIRouter, HTTPException, Query, Request, Response
+from fastapi import Path as PathParam
+from fastapi.responses import FileResponse
+from pydantic import BaseModel, Field
+
+log = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/jarvisx", tags=["jarvisx"])
+
+_ITEM_ID = r"^[0-9a-f]{32}$"
+ItemId = Annotated[str, PathParam(pattern=_ITEM_ID, description="Library item id.")]
+_NO_STORE = {"Cache-Control": "no-store"}
+_PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"
+
+
+class HotkeysPatch(BaseModel):
+ region: str | None = Field(default=None, max_length=64)
+ window: str | None = Field(default=None, max_length=64)
+ fullscreen: str | None = Field(default=None, max_length=64)
+ record_region: str | None = Field(default=None, max_length=64)
+ record_fullscreen: str | None = Field(default=None, max_length=64)
+ stop_recording: str | None = Field(default=None, max_length=64)
+
+
+class SettingsPatch(BaseModel):
+ """Every field optional; only the ones sent are written."""
+
+ enabled: bool | None = None
+ hotkeys: HotkeysPatch | None = None
+ thumbnail_persist: bool | None = None
+ thumbnail_dismiss_s: int | None = Field(default=None, ge=1, le=3600)
+ save_dir: str | None = Field(default=None, max_length=1024)
+ copy_to_clipboard: bool | None = None
+ sound: bool | None = None
+ effect: bool | None = None
+
+
+class CaptureRequest(BaseModel):
+ mode: Literal["region", "window", "fullscreen"]
+ #: Seconds to wait first, so the user can bring the right window forward
+ #: after clicking the button inside this app.
+ delay_s: float = Field(default=0.0, ge=0.0, le=10.0)
+
+
+class RecordRequest(BaseModel):
+ mode: Literal["region", "fullscreen"]
+
+
+class CopyRequest(BaseModel):
+ edited: bool = False
+
+
+# --------------------------------------------------------------------------
+# helpers
+# --------------------------------------------------------------------------
+
+
+def _service(request: Request) -> Any:
+ from jarvis.jarvisx.service import get_service # noqa: PLC0415
+
+ service = get_service()
+ service.bind(bus=getattr(request.app.state, "bus", None), app_state=request.app.state)
+ return service
+
+
+def _native_actions(request: Request) -> bool:
+ return bool(getattr(request.app.state, "native_file_actions", False))
+
+
+def _item_or_404(item_id: str) -> Any:
+ from jarvis.jarvisx.store import get_store # noqa: PLC0415
+
+ item = get_store().get(item_id)
+ if item is None:
+ raise HTTPException(status_code=404, detail="That capture no longer exists.")
+ return item
+
+
+def _is_full_path(text: str) -> bool:
+ """An absolute path on this OS (``~`` allowed); a bare ``shots`` is not."""
+ return Path(text).expanduser().is_absolute()
+
+
+def _media_type(path: Path) -> str:
+ return {
+ ".png": "image/png",
+ ".mp4": "video/mp4",
+ ".webm": "video/webm",
+ ".jpg": "image/jpeg",
+ ".jpeg": "image/jpeg",
+ }.get(path.suffix.lower(), "application/octet-stream")
+
+
+def _settings_payload() -> dict[str, Any]:
+ from jarvis.core.config import load_config # noqa: PLC0415
+ from jarvis.jarvisx import capture, geometry, paths # noqa: PLC0415
+ from jarvis.jarvisx.hotkeys import ( # noqa: PLC0415
+ SHORTCUT_KEYS,
+ configured_hotkeys,
+ get_shortcuts,
+ )
+ from jarvis.jarvisx.overlay.controller import overlay_capability # noqa: PLC0415
+ from jarvis.jarvisx.recorder import probe_encoder # noqa: PLC0415
+
+ config = load_config()
+ block = config.jarvisx
+ hotkeys = configured_hotkeys(block)
+ shortcuts = get_shortcuts()
+ if shortcuts is not None:
+ statuses = {name: status.to_json() for name, status in shortcuts.statuses.items()}
+ else:
+ statuses = {
+ name: {"hotkey": hotkeys[name], "armed": False, "detail": "Not started yet."}
+ for name in SHORTCUT_KEYS
+ }
+ encoder = probe_encoder()
+ capture_ok, capture_detail = capture.capability()
+ overlay_ok, overlay_detail = overlay_capability()
+ return {
+ "enabled": bool(block.enabled),
+ "hotkeys": hotkeys,
+ "thumbnail_persist": bool(block.thumbnail_persist),
+ "thumbnail_dismiss_s": geometry.clamp_dismiss_s(block.thumbnail_dismiss_s),
+ "save_dir": str(block.save_dir or ""),
+ "save_dir_effective": str(paths.resolve_save_dir(block.save_dir)),
+ "save_dir_default": str(paths.default_save_dir()),
+ "copy_to_clipboard": bool(block.copy_to_clipboard),
+ "sound": bool(block.sound),
+ "effect": bool(block.effect),
+ "sound_effects_master": bool(getattr(config.ui, "sound_effects", True)),
+ "recording_available": bool(encoder.available),
+ "recording_detail": encoder.detail,
+ "capture_available": capture_ok,
+ "capture_detail": capture_detail,
+ "overlay_available": overlay_ok,
+ "overlay_detail": overlay_detail,
+ "shortcuts": statuses,
+ }
+
+
+# --------------------------------------------------------------------------
+# settings
+# --------------------------------------------------------------------------
+
+
[email protected]("/settings", operation_id="settings_get")
+async def get_settings() -> dict[str, Any]:
+ return await asyncio.to_thread(_settings_payload)
+
+
[email protected]("/settings", operation_id="settings_set")
+async def put_settings(patch: SettingsPatch) -> dict[str, Any]:
+ """Write the changed settings, then re-arm the shortcuts in place."""
+ from jarvis.core.config import load_config # noqa: PLC0415
+ from jarvis.core.config_writer import set_jarvisx_settings # noqa: PLC0415
+ from jarvis.jarvisx.hotkeys import ( # noqa: PLC0415
+ SHORTCUT_KEYS,
+ check_hotkeys,
+ configured_hotkeys,
+ get_shortcuts,
+ normalize_hotkey,
+ )
+
+ body = patch.model_dump(exclude_none=True)
+ hotkey_patch = body.pop("hotkeys", None) or {}
+ if not body and not hotkey_patch:
+ raise HTTPException(status_code=400, detail="No settings were provided.")
+ changes: dict[str, Any] = dict(body)
+ if "save_dir" in changes:
+ changes["save_dir"] = str(changes["save_dir"]).strip()
+ if changes["save_dir"] and not _is_full_path(changes["save_dir"]):
+ raise HTTPException(status_code=422, detail="The save folder must be a full path.")
+ if hotkey_patch:
+ current = configured_hotkeys((await asyncio.to_thread(load_config)).jarvisx)
+ merged = {**current, **{k: normalize_hotkey(v) for k, v in hotkey_patch.items()}}
+ problems = check_hotkeys(merged)
+ bad = [
+ f"{name.replace('_', ' ')}: {problems[name]}" for name in hotkey_patch if problems[name]
+ ]
+ if bad:
+ raise HTTPException(status_code=422, detail=" ".join(bad))
+ for name in hotkey_patch:
+ changes[SHORTCUT_KEYS[name]] = merged[name]
+ try:
+ await asyncio.to_thread(set_jarvisx_settings, changes)
+ except Exception as exc: # noqa: BLE001 - surface the write failure honestly
+ log.error("jarvisx: settings write failed", exc_info=True)
+ raise HTTPException(status_code=500, detail=f"Could not save the settings: {exc}") from exc
+ shortcuts = get_shortcuts()
+ if shortcuts is not None and (hotkey_patch or "enabled" in changes):
+ await shortcuts.reload()
+ return await asyncio.to_thread(_settings_payload)
+
+
+# --------------------------------------------------------------------------
+# library
+# --------------------------------------------------------------------------
+
+
[email protected]("/items", operation_id="items_list")
+async def list_items(limit: int = Query(default=100, ge=1, le=1000)) -> dict[str, Any]:
+ from jarvis.jarvisx.store import get_store # noqa: PLC0415
+
+ items = await asyncio.to_thread(get_store().recent, limit)
+ return {"items": [item.to_json() for item in items]}
+
+
[email protected]("/items/{item_id}", operation_id="item_get")
+async def get_item(item_id: ItemId) -> dict[str, Any]:
+ item = await asyncio.to_thread(_item_or_404, item_id)
+ return item.to_json()
+
+
[email protected]("/items/{item_id}/file", operation_id="item_file")
+async def item_file(item_id: ItemId) -> FileResponse:
+ item = await asyncio.to_thread(_item_or_404, item_id)
+ path = Path(item.path)
+ if not await asyncio.to_thread(Path.is_file, path):
+ raise HTTPException(status_code=404, detail="The file was moved or deleted.")
+ return FileResponse(path, media_type=_media_type(path), filename=path.name)
+
+
[email protected]("/items/{item_id}/thumb", operation_id="item_thumb")
+async def item_thumb(item_id: ItemId) -> Response:
+ item = await asyncio.to_thread(_item_or_404, item_id)
+ data = await asyncio.to_thread(_thumb_bytes, item)
+ if not data:
+ raise HTTPException(status_code=404, detail="No preview is available.")
+ return Response(content=data, media_type="image/jpeg", headers=_NO_STORE)
+
+
+def _thumb_bytes(item: Any) -> bytes:
+ """The stored preview, rebuilt from the picture when it went missing."""
+ thumb = Path(item.thumb_path) if item.thumb_path else None
+ if thumb is not None and thumb.is_file():
+ return thumb.read_bytes()
+ if item.kind != "image":
+ return b""
+ try:
+ from PIL import Image # noqa: PLC0415
+
+ from jarvis.jarvisx.capture import thumbnail_jpeg # noqa: PLC0415
+
+ with Image.open(item.path) as image:
+ rgb = image.convert("RGB")
+ data = thumbnail_jpeg(rgb.width, rgb.height, rgb.tobytes())
+ if thumb is not None:
+ thumb.parent.mkdir(parents=True, exist_ok=True)
+ thumb.write_bytes(data)
+ return data
+ except Exception: # noqa: BLE001 - a broken picture has no preview; the 404 says so
+ log.warning("jarvisx: preview rebuild failed for %s", item.id, exc_info=True)
+ return b""
+
+
[email protected]("/items/{item_id}/edited", operation_id="item_edited_get")
+async def get_edited(item_id: ItemId) -> FileResponse:
+ item = await asyncio.to_thread(_item_or_404, item_id)
+ if not item.has_edited():
+ raise HTTPException(status_code=404, detail="This capture has no annotated copy.")
+ path = Path(str(item.edited_path))
+ return FileResponse(path, media_type="image/png", filename=path.name, headers=_NO_STORE)
+
+
[email protected]("/items/{item_id}/edited", operation_id="item_edited_save")
+async def put_edited(request: Request, item_id: ItemId) -> dict[str, Any]:
+ """Save the editor's annotated PNG as ``<name>-edited.png`` next to the original."""
+ from jarvis.jarvisx.service import MAX_EDITED_BYTES # noqa: PLC0415
+
+ content_type = request.headers.get("content-type", "").split(";")[0].strip().lower()
+ if content_type != "image/png":
+ raise HTTPException(status_code=415, detail="Send the picture as image/png.")
+ data = await request.body()
+ if len(data) > MAX_EDITED_BYTES:
+ raise HTTPException(status_code=413, detail="The picture is too large.")
+ if not data.startswith(_PNG_SIGNATURE) or not await asyncio.to_thread(_is_png, data):
+ raise HTTPException(status_code=422, detail="That is not a readable PNG picture.")
+ result = await _service(request).save_edited(item_id, data)
+ if not result.ok:
+ code = 404 if result.item is None and "no longer" in result.message else 409
+ raise HTTPException(status_code=code, detail=result.message)
+ return result.item.to_json()
+
+
+def _is_png(data: bytes) -> bool:
+ import io # noqa: PLC0415
+
+ try:
+ from PIL import Image # noqa: PLC0415
+
+ with Image.open(io.BytesIO(data)) as image:
+ image.verify()
+ return True
+ except Exception: # noqa: BLE001 - any decoder complaint means "not a PNG we keep"
+ log.info("jarvisx: rejected an unreadable annotated picture")
+ return False
+
+
[email protected]("/items/{item_id}/copy", operation_id="item_copy")
+async def copy_item(
+ request: Request, item_id: ItemId, body: CopyRequest | None = None
+) -> dict[str, Any]:
+ if not _native_actions(request):
+ return {"ok": False, "message": "The clipboard belongs to the computer Jarvis runs on."}
+ result = await _service(request).copy_item(item_id, edited=(body or CopyRequest()).edited)
+ return {"ok": result.ok, "message": result.message}
+
+
[email protected]("/items/{item_id}/reveal", operation_id="item_reveal")
+async def reveal_item(request: Request, item_id: ItemId) -> dict[str, Any]:
+ if not _native_actions(request):
+ return {"ok": False, "message": "Showing files needs the desktop app."}
+ result = await _service(request).reveal(item_id)
+ return {"ok": result.ok, "message": result.message}
+
+
[email protected]("/items/{item_id}/open-editor", operation_id="item_open_editor")
+async def open_editor(request: Request, item_id: ItemId) -> dict[str, Any]:
+ result = await _service(request).open_editor(item_id)
+ return {
+ "ok": result.ok,
+ "message": result.message,
+ "url": f"/?view=jarvisx-editor&solo=1&item={item_id}",
+ }
+
+
[email protected]("/items/{item_id}", operation_id="item_delete")
+async def delete_item(request: Request, item_id: ItemId) -> dict[str, Any]:
+ result = await _service(request).delete(item_id)
+ if not result.ok:
+ raise HTTPException(status_code=404, detail=result.message)
+ return {"ok": True, "message": result.message}
+
+
+# --------------------------------------------------------------------------
+# capture + recording
+# --------------------------------------------------------------------------
+
+
[email protected]("/capture", operation_id="capture")
+async def take_capture(request: Request, body: CaptureRequest) -> dict[str, Any]:
+ result = await _service(request).capture(body.mode, delay_s=body.delay_s)
+ return {
+ "ok": result.ok,
+ "item": result.item.to_json() if result.item is not None else None,
+ "message": result.message,
+ }
+
+
[email protected]("/record/start", operation_id="record_start")
+async def record_start(request: Request, body: RecordRequest) -> dict[str, Any]:
+ service = _service(request)
+ result = await service.start_recording(body.mode)
+ return {"ok": result.ok, "message": result.message, **service.recording_status()}
+
+
[email protected]("/record/stop", operation_id="record_stop")
+async def record_stop(request: Request) -> dict[str, Any]:
+ service = _service(request)
+ result = await service.stop_recording()
+ return {
+ "ok": result.ok,
+ "item": result.item.to_json() if result.item is not None else None,
+ "message": result.message,
+ **service.recording_status(),
+ }
+
+
[email protected]("/record/status", operation_id="record_status")
+def record_status(request: Request) -> dict[str, Any]:
+ return _service(request).recording_status()
diff --git a/jarvis/ui/web/mcp_server_routes.py b/jarvis/ui/web/mcp_server_routes.py
index b18328b390..e9c771ded8 100644
--- a/jarvis/ui/web/mcp_server_routes.py
+++ b/jarvis/ui/web/mcp_server_routes.py
@@ -137,7 +137,10 @@ def session_ref(scope: dict[str, Any]) -> str | None:
value = raw_value.decode("latin-1").strip()
# Society chats use a namespaced roster slug, while ordinary chats
# use opaque alphanumeric IDs. Preserve that scope for tool lookup.
- if len(value) <= 128 and re.fullmatch(r"society:[a-z0-9]+(?:-[a-z0-9]+)*", value):
+ if len(value) <= 128 and re.fullmatch(
+ r"society:[a-z0-9]+(?:-[a-z0-9]+)*(?::routine:[a-zA-Z0-9_-]+:[a-zA-Z0-9_-]+)?",
+ value,
+ ):
return value
if len(value) <= 64 and re.fullmatch(r"[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*", value):
return value
diff --git a/jarvis/ui/web/provider_spec.py b/jarvis/ui/web/provider_spec.py
index 1051f0669d..8c5d96df96 100644
--- a/jarvis/ui/web/provider_spec.py
+++ b/jarvis/ui/web/provider_spec.py
@@ -953,7 +953,7 @@ PROVIDERS: tuple[ProviderSpec, ...] = (
# and bundles no server.
ProviderSpec(
id="local-realtime",
- label="Self-hosted realtime (OpenAI-compatible)",
+ label="Local voice",
tier="realtime",
auth_mode="none",
secret_keys=(),
diff --git a/jarvis/ui/web/server.py b/jarvis/ui/web/server.py
index e1feb290ca..249a2fa466 100644
--- a/jarvis/ui/web/server.py
+++ b/jarvis/ui/web/server.py
@@ -193,6 +193,20 @@ class WebServer:
self.app: FastAPI = self._build_app()
self.app.state.refresh_scheduler = None
+ async def _forward_delegation_to_chat(self, event: Any) -> None:
+ """Persist a result in its originating chat without starting another model turn."""
+ service = getattr(self.app.state, "agent_chat", None)
+ if service is None:
+ return
+ session = service.store.get_session(event.session_id)
+ if session is None or session.surface != "jarvis":
+ return
+ await service.post_notice(event.session_id, {
+ "kind": "coding_result", "agent_name": event.agent_name,
+ "assignment_id": event.request_id, "status": event.status,
+ "text": event.text, "report": event.report,
+ })
+
async def _forward_preamble_to_chat(self, event: AnnouncementRequested) -> None:
"""Bridge AnnouncementRequested(kind="preamble") → MessageSent.
@@ -333,6 +347,8 @@ class WebServer:
from .agentic_ide_git_routes import router as agentic_ide_git_router
from .agentic_ide_routes import router as agentic_ide_router
from .antigravity_routes import router as antigravity_router
+ from .app_actions_routes import router as app_actions_router
+ from .appshot_routes import router as appshot_router
from .board_routes import (
board_router as board_meta_router,
)
@@ -388,8 +404,6 @@ class WebServer:
from .provider_routes import router as provider_router
from .review_routes import router as review_router
from .routine_hooks_routes import router as routine_hooks_router
- from .app_actions_routes import router as app_actions_router
- from .appshot_routes import router as appshot_router
from .screen_context_routes import router as screen_context_router
from .self_mod_routes import router as self_mod_router
from .sessions_routes import router as sessions_router
@@ -398,9 +412,9 @@ class WebServer:
from .setup_routes import router as setup_router
from .skills_routes import router as skills_router
from .socials_routes import router as socials_router
- from .society_routes import router as society_router
from .society_browser_routes import router as society_browser_router
from .society_figure_routes import router as society_figure_router
+ from .society_routes import router as society_router
from .starter_plan_routes import router as starter_plan_router
from .sub_agents_routes import router as sub_agents_router
from .tasks_routes import router as tasks_router
@@ -3390,7 +3404,10 @@ class WebServer:
from jarvis.harness.manager import HarnessManager
def workflow_services():
- return (getattr(self.app.state, "workflow_store", None), getattr(self.app.state, "workflow_runner", None))
+ return (
+ getattr(self.app.state, "workflow_store", None),
+ getattr(self.app.state, "workflow_runner", None),
+ )
runner = TaskRunner(
store=store,
@@ -3403,7 +3420,9 @@ class WebServer:
owned_action_guard=self._guard_society_routine_action,
workflow_services=workflow_services,
)
- scheduler = TaskScheduler(store=store, bus=self.bus, runner=runner, workflow_services=workflow_services)
+ scheduler = TaskScheduler(
+ store=store, bus=self.bus, runner=runner, workflow_services=workflow_services,
+ )
scheduler.bind_bus()
await scheduler.hydrate()
@@ -3761,6 +3780,11 @@ class WebServer:
except Exception: # noqa: BLE001 — the name is cosmetic
return "Jarvis"
+ from jarvis.core.events import DelegationResultReady
+
+ # Install on first chat use, keeping result tracking off the boot path.
+ self.bus.unsubscribe(DelegationResultReady, self._forward_delegation_to_chat)
+ self.bus.subscribe(DelegationResultReady, self._forward_delegation_to_chat)
return AgentChatService(store, assistant_name=_name, bus=lambda: self.bus)
async def stop(self) -> None:
diff --git a/jarvis/ui/web/society_routes.py b/jarvis/ui/web/society_routes.py
index e24db48975..14d89c9d45 100644
--- a/jarvis/ui/web/society_routes.py
+++ b/jarvis/ui/web/society_routes.py
@@ -90,6 +90,7 @@ class CreateAgentBody(BaseModel):
denies: list[str] | None = None
skills: list[str] | None = None
permission_ceiling: str | None = None
+ approval_mode: str | None = None
approval_rules: dict[str, list[str]] | None = None
daily_budget_usd: float | None = None
max_concurrent_runs: int | None = None
@@ -125,6 +126,7 @@ class PatchAgentBody(BaseModel):
skills: list[str] | None = None
knowledge_scope: str | None = None
permission_ceiling: str | None = None
+ approval_mode: str | None = None
approval_rules: dict[str, list[str]] | None = None
daily_budget_usd: float | None = None
max_concurrent_runs: int | None = None
@@ -262,6 +264,16 @@ async def create_agent(body: CreateAgentBody, request: Request) -> dict[str, Any
from jarvis.society.inherit import inherit_creator_fields
fields = inherit_creator_fields(fields, creator)
+ requested_mode = str(fields.get("approval_mode") or "bypass")
+ provider = str(fields.get("provider") or body.provider)
+ if provider:
+ from jarvis.agent_chat.permissions import society_mode_supported
+ from jarvis.agent_chat.service import resolve_runner
+
+ if not society_mode_supported(resolve_runner(provider, surface="society"), requested_mode):
+ raise HTTPException(
+ 422, "This runner cannot provide an actionable approval for that mode."
+ )
derived_focus, derived_rules = rt.derive(body.title, description)
if body.focus is None and derived_focus:
fields["focus"] = derived_focus
@@ -315,6 +327,18 @@ async def patch_agent(agent_id: str, body: PatchAgentBody, request: Request) ->
if has_brief(brief):
# A brief rewrites the standing instructions as a whole.
fields["description"] = compose_description(body.description or "", brief)
+ requested_mode = body.approval_mode or (
+ str(agent.approval_mode) if body.provider and agent.approval_mode else ""
+ )
+ if requested_mode:
+ from jarvis.agent_chat.permissions import society_mode_supported
+ from jarvis.agent_chat.service import resolve_runner
+
+ provider = body.provider or agent.provider
+ if not society_mode_supported(resolve_runner(provider, surface="society"), requested_mode):
+ raise HTTPException(
+ 422, "This runner cannot provide an actionable approval for that mode."
+ )
if ("title" in fields or "description" in fields) and "focus" not in fields:
# A prose edit must not wipe what the agent earned in its chat: the
# derived focus is APPENDED to the existing order (existing first,
@@ -336,6 +360,10 @@ async def patch_agent(agent_id: str, body: PatchAgentBody, request: Request) ->
updated = await rt.roster.update(agent.agent_id, fields)
except RosterError as exc:
raise _typed_error(exc) from exc
+ if "approval_mode" in fields:
+ svc = rt.chat_service()
+ if svc is not None and svc.store.get_session(updated.session_id) is not None:
+ svc.store.update_session(updated.session_id, permission_mode=str(updated.approval_mode))
if "state" in fields:
await rt.checkpoints.refresh(agent.agent_id)
updated = await rt.roster.get(agent.agent_id) or updated
@@ -356,7 +384,10 @@ async def bind_agent_chat(agent_id: str, request: Request) -> dict[str, Any]:
raise HTTPException(503, "agent chat service unavailable")
from jarvis.society.chat_binding import ensure_session
- session = ensure_session(svc, rt.config(), agent)
+ try:
+ session = ensure_session(svc, rt.config(), agent)
+ except PermissionError as exc:
+ raise HTTPException(422, str(exc)) from exc
return {"session": session.to_dict(), "agent_id": agent.agent_id}
@@ -685,6 +716,7 @@ async def switch_agent_model(agent_id: str, body: ModelBody, request: Request) -
"""Move the agent onto another provider / model / effort / subscription seat.
Its canonical chat is re-seated at once (transcript kept)."""
from jarvis.agent_chat.catalog import offers
+ from jarvis.agent_chat.permissions import society_mode_supported
from jarvis.agent_chat.service import resolve_runner
rt = await _runtime(request)
@@ -696,6 +728,10 @@ async def switch_agent_model(agent_id: str, body: ModelBody, request: Request) -
422,
{"reason": str(FailureReason.BLOCKED_BY_POLICY), "detail": "provider not offered"},
)
+ if agent.approval_mode and not society_mode_supported(
+ resolve_runner(body.provider, surface="society"), str(agent.approval_mode)
+ ):
+ raise HTTPException(422, "This runner cannot provide an actionable approval for that mode.")
fields = {
"provider": body.provider.strip().lower(),
"model": body.model.strip(),
diff --git a/jarvis/ui/winforms_errors.py b/jarvis/ui/winforms_errors.py
new file mode 100644
index 0000000000..da319fc769
--- /dev/null
+++ b/jarvis/ui/winforms_errors.py
@@ -0,0 +1,92 @@
+"""Keep the details of a Windows Forms error after its dialog is dismissed.
+
+WinForms catches UI callback failures inside its managed message loop. Neither
+``sys.excepthook`` nor ``threading.excepthook`` sees them. Subscribe on the GUI
+thread (pywebview's synchronous ``before_show`` event), not the Python main
+thread. Keep the normal Continue/Quit dialog; logging is not crash recovery.
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+from typing import Any
+
+log = logging.getLogger(__name__)
+_handlers: dict[int, tuple[Any, Any]] = {}
+
+
+def _report_exception(exception: Any) -> None:
+ from jarvis.core.redact import safe_preview
+
+ # ToString includes managed inner exceptions and stack traces. Python's
+ # traceback formatter cannot reconstruct those from a pythonnet proxy.
+ detail = safe_preview(str(exception.ToString()), max_chars=32_768)
+ log.critical("Unhandled Windows Forms UI exception:\n%s", detail)
+
+
+def _present_exception(forms: Any, exception: Any) -> None:
+ dialog = forms.ThreadExceptionDialog(exception)
+ try:
+ result = dialog.ShowDialog()
+ finally:
+ dialog.Dispose()
+ if result == forms.DialogResult.Abort:
+ # Run normal window-closing hooks, including Jarvis's bounded shutdown.
+ forms.Application.Exit()
+
+
+def _subscribe(forms: Any, thread_id: int) -> None:
+ """Install once per managed UI thread; release delegates on ThreadExit."""
+ if thread_id in _handlers:
+ return
+
+ def on_exception(_sender: Any, args: Any) -> None:
+ try:
+ _report_exception(args.Exception)
+ except Exception: # noqa: BLE001 — reporting must not hide the original dialog
+ log.exception("Could not record the Windows Forms exception")
+ _present_exception(forms, args.Exception)
+
+ def on_exit(_sender: Any, _args: Any) -> None:
+ # ThreadExit is process-wide; another GUI thread must keep its handler.
+ from System.Threading import Thread # type: ignore[import-not-found]
+
+ if int(Thread.CurrentThread.ManagedThreadId) != thread_id:
+ return
+ forms.Application.ThreadException -= on_exception
+ forms.Application.ThreadExit -= on_exit
+ _handlers.pop(thread_id, None)
+
+ forms.Application.ThreadException += on_exception
+ try:
+ forms.Application.ThreadExit += on_exit
+ except Exception:
+ forms.Application.ThreadException -= on_exception
+ raise
+ # Hold the Python callables for the full managed delegate lifetime.
+ _handlers[thread_id] = (on_exception, on_exit)
+ log.info("Windows Forms UI error logging installed")
+
+
+def _install_on_gui_thread() -> None:
+ try:
+ # The real window already loaded WinForms; importing here adds no CLR
+ # dependency to Linux, macOS, the API server, or Python module import.
+ import System.Windows.Forms as forms # type: ignore[import-not-found]
+ from System.Threading import Thread # type: ignore[import-not-found]
+
+ _subscribe(forms, int(Thread.CurrentThread.ManagedThreadId))
+ except Exception: # noqa: BLE001 — retain the framework's default error handling
+ log.exception("Windows Forms error logging could not be installed")
+
+
+def register_winforms_error_logging(window: Any) -> None:
+ """Arm logging before the first shown window, without loading .NET early."""
+ if sys.platform != "win32":
+ return
+ before_show = getattr(window.events, "before_show", None)
+ if before_show is None:
+ log.warning("Windows Forms error logging needs a synchronous before_show event")
+ return
+ before_show += _install_on_gui_thread
diff --git a/jarvis/workspace/agents.py b/jarvis/workspace/agents.py
index ba28a30b17..3a7f807804 100644
--- a/jarvis/workspace/agents.py
+++ b/jarvis/workspace/agents.py
@@ -790,6 +790,13 @@ _AGENTS: dict[str, WorkspaceAgent] = {
binary="codex",
npm_package="@openai/codex",
homepage="https://github.com/openai/codex",
+ # Codex's startup update box can exceed a narrow pane's width: its
+ # unbroken release URL widens the border after the content was wrapped
+ # (reproduced on 0.159.2 at 41 columns). Keep this extra startup UI out
+ # of embedded panes. A per-launch override leaves standalone Codex and
+ # the account's saved settings untouched; installation remains available
+ # through Jarvis's CLI controls.
+ launch_args=("-c", "check_for_update_on_startup=false"),
trust=CODEX_TRUST,
account=AccountSpec(env=(("CODEX_HOME", "{dir}"),), native_dir="~/.codex"),
win_shim=WinShim(
diff --git a/packaging/linux/build.sh b/packaging/linux/build.sh
index 387e7d100c..4a908e1667 100755
--- a/packaging/linux/build.sh
+++ b/packaging/linux/build.sh
@@ -291,8 +291,8 @@ Architecture: amd64
Depends: libc6
Recommends: libportaudio2
Maintainer: Personal Jarvis contributors <https://github.com/PersonalJarvis/PersonalJarvis>
-Description: Voice-driven meta-orchestrator
- Personal Jarvis turns one spoken request into a fleet of self-checking AI
+Description: AI agent for your computer
+ Personal Jarvis coordinates desktop apps, tools, models and specialized
agents. This package ships the self-contained desktop build; it needs no
system Python and no pip installation.
EOF
diff --git a/packaging/linux/make_desktop_entry.py b/packaging/linux/make_desktop_entry.py
index 4188bfb04c..fa88ed32ff 100644
--- a/packaging/linux/make_desktop_entry.py
+++ b/packaging/linux/make_desktop_entry.py
@@ -41,7 +41,7 @@ from jarvis.core.branding import ( # noqa: E402
from jarvis.core.desktop_entry import escape_value, exec_value # noqa: E402
#: Same one-line summary the source install's menu entry uses.
-COMMENT = "Voice-driven meta-orchestrator"
+COMMENT = "Turn your computer into an AI agent"
#: Freedesktop main category. "Utility" is what the source install already
#: registers; changing it would move the app in the menu between install kinds.
diff --git a/pyproject.toml b/pyproject.toml
index a6a714a171..7331ba39a8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "personal-jarvis"
version = "2.3.2"
-description = "Voice-driven meta-orchestrator that turns one spoken request into a fleet of self-checking AI agents"
+description = "Turn your computer into an AI agent that coordinates apps, tools, models, and other agents"
requires-python = ">=3.11,<3.15"
readme = "README.md"
# Apache-2.0 from 2.0.0 on. Releases up to 1.6.0 shipped MIT and stay MIT —
diff --git a/scripts/config-soll.json b/scripts/config-soll.json
index 5ff6c15866..de3faccbc6 100644
--- a/scripts/config-soll.json
+++ b/scripts/config-soll.json
@@ -33,7 +33,7 @@
"allow_sapi5_fallback": false
},
"brain": {
- "primary": "openrouter",
+ "primary": "openai",
"fallback": "openrouter",
"deep_brain": "claude-cli",
"routing_provider": "grok",
@@ -56,7 +56,7 @@
"deep_model": "claude-opus-4-8"
},
"brain.worker": {
- "provider": "openai-codex",
+ "provider": "claude-api",
"model": "",
"fallback_provider": "gemini",
"fallback_model": "gemini-3.1-pro-preview",
@@ -65,7 +65,7 @@
},
"stt": {
"provider": "faster-whisper",
- "model": "large-v3",
+ "model": "whisper-large-v3",
"bias_prompt": "Jarvis, OpenClaw, Codex, Gemini, Grok, Claude, Anthropic, OpenAI, OpenRouter, Whisper, Subagent, Sub-Agent, Mission-Manager, Critic-Loop, Critic, Worker, Router-Brain, Router, Kontrollierer, Voice-Pipeline, Wake-Word, Silero, VAD, TTS, STT, Telemetry, Transkript, Token, Provider, Plugin, Sprechblase, Orb, Mascot, Quip, Browser, Terminal, Repository, Branch, Commit, Push, Pull-Request, Issue, Merge, Refactor, Deployment, Screenshot, Debuggen, Workflow, Microservice, Container, Docker, Kubernetes, FastAPI, React, TypeScript, Python, JSON, Markdown, Logfile, Konfiguration, Endpoint, Webhook, API-Key, Service-Account, Cloud, Google Cloud, Vertex AI, BigQuery, GitHub, Twilio, ElevenLabs, OAuth, Credential Manager, Frontend, Backend, Skript, Daemon, Hangup, Auflegen, Denkpause, submitted, distanziert, erkennt",
"language": "auto",
"models": {
@@ -113,7 +113,7 @@
"provider": "vertex"
},
"brain.computer_use": {
- "provider": "grok"
+ "provider": "openai"
},
"brain.providers.grok-realtime": {
"voice": "lux"
@@ -143,7 +143,7 @@
"live": {
"model": "gpt-live-1",
"voice": "gleam",
- "backend_model": "gpt-5.6-terra",
+ "backend_model": "gpt-6-luna",
"reasoning_effort": "medium",
"web_search": true,
"instructions": "",
diff --git a/tests/contract/test_society_communication.py b/tests/contract/test_society_communication.py
index c6b7807985..1bc1b89ef7 100644
--- a/tests/contract/test_society_communication.py
+++ b/tests/contract/test_society_communication.py
@@ -150,7 +150,7 @@ async def test_question_returns_once_and_answer_ends_exchange(world):
assert reply_policy(answers[0]) == "none"
assert "No reply is requested" in frame_incoming(answers[0], "Scout")
assert len(chat.notices) == 1
- assert published == [], "a reply to the lead is shown, never spoken"
+ assert len(published) == 1
refused = await MessageAgentTool(rt, "jarvis").execute(
{
"target": "scout",
@@ -211,7 +211,7 @@ async def test_message_reporting_matrix(world, policy, status, expected):
request = await send(rt, reply_policy=policy)
await finish_message(rt, chat, request, status=status)
assert len(chat.notices) == expected
- assert published == [], "a reply to the lead is shown, never spoken"
+ assert len(published) == expected
if expected and status == "error":
replies = await rt.store.events_for_trace(request.trace_id)
assert replies[-1].payload["reply_status"] == "blocked"
@@ -293,7 +293,7 @@ async def test_assignment_reporting_preserves_result_and_releases_slot(
assert len(results) == 1 and results[0].payload["status"] == status
assert rt.scheduler.active_runs("scout") == 0
assert len(chat.notices) == expected
- assert published == [], "a reply to the lead is shown, never spoken"
+ assert len(published) == expected
@pytest.mark.parametrize(
diff --git a/tests/fakes/fake_ssh_server.py b/tests/fakes/fake_ssh_server.py
index b361e0b4b1..0c7c5d44d4 100644
--- a/tests/fakes/fake_ssh_server.py
+++ b/tests/fakes/fake_ssh_server.py
@@ -70,6 +70,7 @@ class FakeSshState:
username: str = "root"
#: False plays a server with ``PasswordAuthentication no`` (keys only).
password_login: bool = True
+ port_forwarding: bool = False
authorized: set[str] = field(default_factory=set)
commands: list[str] = field(default_factory=list)
#: The raw SSH command lines, as the user's login shell would parse them.
@@ -91,6 +92,9 @@ class _Server(asyncssh.SSHServer):
def begin_auth(self, username: str) -> bool:
return True
+ def server_requested(self, listen_host: str, listen_port: int) -> bool:
+ return self._state.port_forwarding and listen_host == "127.0.0.1"
+
def password_auth_supported(self) -> bool:
return self._state.password_login
diff --git a/tests/fakes/terminal_repaint_cases.json b/tests/fakes/terminal_repaint_cases.json
new file mode 100644
index 0000000000..4e278b2cd6
--- /dev/null
+++ b/tests/fakes/terminal_repaint_cases.json
@@ -0,0 +1,19 @@
+{
+ "full": [
+ "\u001b[2J",
+ "\u001b[1;1H\u001b[J",
+ "\u001b[H\u001b[J",
+ "\u001b[0;0H\u001b[0J",
+ "\u001b[1H\u001b[0m\u001b[J",
+ "\u001b[H\u001b[?25l\u001b[J"
+ ],
+ "partial": [
+ "\u001b[J",
+ "\u001b[2;1H\u001b[J",
+ "\u001b[1;2H\u001b[J",
+ "\u001b[Htext\u001b[J",
+ "\u001b[H\u001b[1B\u001b[J",
+ "\u001b[H\u001b[?1049l\u001b[J",
+ "\u001b[H\u001b[3J"
+ ]
+}
diff --git a/tests/integration/test_delegation_followthrough.py b/tests/integration/test_delegation_followthrough.py
new file mode 100644
index 0000000000..be615f88cf
--- /dev/null
+++ b/tests/integration/test_delegation_followthrough.py
@@ -0,0 +1,135 @@
+"""Real delegation lifecycle through the speech inbox, with fake providers only."""
+
+import asyncio
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.agent_chat.runner_api import messages_from_events
+from jarvis.core.bus import EventBus
+from jarvis.core.events import AnnouncementRequested, DelegationResultReady
+from jarvis.plugins.tool.delegate_to_agent import DelegateToAgentTool
+from jarvis.society.events import MsgType
+from jarvis.speech.pipeline import TurnTakingState
+from jarvis.ui.web.server import WebServer
+from tests.contract import test_society_communication as communication
+from tests.unit.speech.test_realtime_announcement_bridge import _pipeline
+
+world = communication.world
+
+
+async def test_two_delegations_return_after_completion_without_blocking_conversation(
+ world, monkeypatch,
+):
+ runtime, chat, _ = world
+ pipeline, tts, _, live = _pipeline(accepted=True)
+ bus = EventBus()
+ bus.subscribe(AnnouncementRequested, pipeline._on_announcement)
+ runtime._publish_event = bus.publish
+ monkeypatch.setattr("jarvis.core.delegation.BATCH_WINDOW_S", 0.005)
+ tool = DelegateToAgentTool(runtime_resolver=lambda: runtime)
+ receipts = []
+ for agent in ("Scout", "Archivist"):
+ receipt = await asyncio.wait_for(tool.execute(
+ {"agent": agent, "task": f"Research {agent}", "reply_policy": "always"},
+ communication.ctx(),
+ ), timeout=1)
+ assert receipt.success and receipt.output["state"] == "running"
+ receipts.append(receipt)
+ assert not live.calls # Dispatch is acknowledged before either agent has finished.
+ pipeline._turn_state = TurnTakingState.USER_SPEAKING
+ for name in ("scout", "archivist"):
+ await chat.queues[f"society:{name}"].put({
+ "kind": "assistant_text", "payload": {"text": f"Recorded {name} findings."},
+ })
+ await chat.queues[f"society:{name}"].put({
+ "kind": "turn_finished", "payload": {"status": "completed"},
+ })
+ await asyncio.wait_for(asyncio.gather(*list(runtime._watchers)), timeout=3)
+ assert not live.calls
+ assert len(chat.notices) == 2
+ await pipeline._set_turn_state(TurnTakingState.LISTENING)
+ await asyncio.sleep(0.04)
+ assert len(live.calls) == 1 and not tts.calls
+ for receipt in receipts:
+ assert receipt.output["assignment_id"] in live.calls[0]["report"]
+ assert "scout findings" in live.calls[0]["report"]
+ assert "archivist findings" in live.calls[0]["report"]
+
+
+async def test_agent_question_is_returned_and_turn_end_does_not_claim_success(world):
+ runtime, chat, published = world
+ receipt = await DelegateToAgentTool(runtime_resolver=lambda: runtime).execute(
+ {"agent": "Scout", "task": "Research it", "reply_policy": "on_error"},
+ communication.ctx(),
+ )
+ request = await runtime.store.get_event(receipt.output["assignment_id"])
+ await runtime.say(
+ from_agent="scout", to_agent="jarvis", msg_type=MsgType.QUERY,
+ text="Which repository do you mean?", trace_id=request.trace_id,
+ parent_event_id=request.event_id,
+ )
+ await chat.queues["society:scout"].put({
+ "kind": "turn_finished", "payload": {"status": "completed"},
+ })
+ await asyncio.wait_for(asyncio.gather(*list(runtime._watchers)), timeout=3)
+ assert len(published) == 1
+ assert "needs_input" in published[0].report
+ assert len(chat.notices) == 1
+ results = [e for e in await runtime.store.events_for_trace(request.trace_id)
+ if e.msg_type is MsgType.RESULT]
+ assert results[0].payload["status"] == "blocked"
+
+
+async def test_coding_result_survives_chat_event_storage_and_enters_followup_context(world):
+ _, chat, _ = world
+ original = chat.store.list_sessions(surface="jarvis")[0]
+ notices = []
+
+ async def post(session_id, payload):
+ notices.append((session_id, payload))
+ await chat._emit(session_id, {"kind": "notice", "payload": payload})
+
+ chat.post_notice = post
+ server = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace(agent_chat=chat)))
+ event = DelegationResultReady(
+ session_id=original.session_id, request_id="task-1", agent_name="T1",
+ status="completed", text="T1 reports: Fixed login.",
+ report='{"request":"Fix login", "report":"Tests passed; deployment pending"}',
+ )
+ await WebServer._forward_delegation_to_chat(server, event)
+ assert notices[0][0] == original.session_id
+ messages = messages_from_events([{"kind": "notice", "payload": notices[0][1]}])
+ assert "deployment pending" in messages[0].content
+ assert "Fix login" in messages[0].content
+
+
+async def test_chat_factory_installs_one_lazy_result_bridge(tmp_path):
+ server = WebServer.__new__(WebServer)
+ server.cfg = SimpleNamespace(memory=SimpleNamespace(data_dir=str(tmp_path)))
+ server.bus = EventBus()
+ server.app = SimpleNamespace(state=SimpleNamespace(agent_chat=None))
+ first = server._build_agent_chat_service()
+ first.store.close()
+ service = server._build_agent_chat_service()
+ server.app.state.agent_chat = service
+ session = service.store.create_session(
+ provider="openai", model="", effort="", cwd=str(tmp_path), surface="jarvis",
+ )
+ queue = service.subscribe(session.session_id)
+ try:
+ await server.bus.publish(DelegationResultReady(
+ session_id=session.session_id, request_id="job", agent_name="T1",
+ status="completed", text="Reported outcome", report="Task and evidence",
+ ))
+ assert queue.qsize() == 1
+ notice = queue.get_nowait()
+ assert notice["kind"] == "notice"
+ assert notice["payload"]["report"] == "Task and evidence"
+ finally:
+ service.unsubscribe(session.session_id, queue)
+ service.store.close()
diff --git a/tests/unit/agent_chat/test_jarvis_identity.py b/tests/unit/agent_chat/test_jarvis_identity.py
index 2d028b3c49..b2d01e68b9 100644
--- a/tests/unit/agent_chat/test_jarvis_identity.py
+++ b/tests/unit/agent_chat/test_jarvis_identity.py
@@ -301,7 +301,16 @@ def test_the_mcp_route_reads_the_session_header_and_ignores_junk():
assert mcp_server_routes.session_ref({"headers": [(b"x-jarvis-chat-session", b"../x")]}) is None
[email protected]("value", ["society:nala", "society:gmail-agent", "sess-1", "0123abcd"])
+ "value",
+ [
+ "society:nala",
+ "society:gmail-agent",
+ "society:reddit-community-agent:routine:task123:exec456",
+ "sess-1",
+ "0123abcd",
+ ],
+)
def test_session_header_preserves_canonical_chat_scopes(value):
assert (
mcp_server_routes.session_ref({"headers": [(b"x-jarvis-chat-session", value.encode())]})
diff --git a/tests/unit/agentic_ide/test_followthrough.py b/tests/unit/agentic_ide/test_followthrough.py
new file mode 100644
index 0000000000..01989b7569
--- /dev/null
+++ b/tests/unit/agentic_ide/test_followthrough.py
@@ -0,0 +1,115 @@
+"""Only a fresh answer to the tracked prompt can be read back as its report."""
+
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.agentic_ide import followthrough as ft
+from jarvis.agentic_ide.agent_transcript import Turn
+from jarvis.core.events import AnnouncementRequested, DelegationResultReady
+
+
+def pane(monkeypatch):
+ turns = [Turn("user", "Old task"), Turn("assistant", "Old answer")]
+ term = SimpleNamespace(
+ name="T2", process_generation=1, last_submit_at=100.0, last_prompt="Fix the bug",
+ submitted=True, delegation_result=None, exit_code=None,
+ last_output_at=100.0, reading=lambda: SimpleNamespace(activity="waiting"),
+ transcript=SimpleNamespace(tail=lambda n: ["Which repository should I change?"]),
+ )
+ monkeypatch.setattr(ft, "_turns", lambda t: list(turns))
+ return term, turns
+
+
+async def arm(pane, **origin):
+ term, _ = pane
+ pending = await ft.prepare(term, "Fix the bug", "Fix the bug", {"lang": "en", **origin})
+ ft.submitted(term, pending)
+ return term.delegation_result
+
+
+async def test_fresh_correlated_answer_reaches_voice_once(pane):
+ pending = await arm(pane)
+ term, turns = pane
+ turns.extend([Turn("user", "Fix the bug"), Turn("assistant", "Fixed login; tests passed.")])
+ events = []
+ assert await ft.publish_result("completed", term, pending, events.append)
+ assert not await ft.publish_result("completed", term, pending, events.append)
+ assert len(events) == 1 and isinstance(events[0], AnnouncementRequested)
+ assert "Fixed login" in events[0].report
+ assert "Fix the bug" in events[0].report
+
+
[email protected]("stale", ["old", "same_prompt", "unrelated"])
+async def test_stale_transcript_never_becomes_the_current_answer(pane, stale):
+ term, turns = pane
+ if stale == "same_prompt":
+ turns[:] = [Turn("user", "Fix the bug"), Turn("assistant", "Old answer")]
+ pending = await arm(pane)
+ if stale == "unrelated":
+ turns.extend([Turn("user", "Other task"), Turn("assistant", "Unrelated answer")])
+ events = []
+ await ft.publish_result("completed", term, pending, events.append)
+ assert "Old answer" not in events[0].report and "Unrelated answer" not in events[0].report
+ assert "unverified" in events[0].report
+
+
[email protected]("change", ["manual_prompt", "restarted", "replaced"])
+async def test_later_prompt_or_process_cannot_claim_previous_result(pane, change):
+ pending = await arm(pane)
+ term, _ = pane
+ if change == "manual_prompt":
+ term.last_submit_at += 1
+ elif change == "restarted":
+ term.process_generation += 1
+ else:
+ term.delegation_result = None
+ events = []
+ assert not await ft.publish_result("completed", term, pending, events.append)
+ assert not events
+
+
+async def test_question_does_not_consume_later_completion(pane):
+ pending = await arm(pane)
+ term, turns = pane
+ events = []
+ await ft.publish_result("needs_input", term, pending, events.append)
+ assert "Which repository" in events[0].report
+ assert term.delegation_result is pending
+ turns.extend([Turn("user", "Fix the bug"), Turn("assistant", "Resolved.")])
+ await ft.publish_result("completed", term, pending, events.append)
+ assert len(events) == 2 and "Resolved" in events[1].report
+
+
+async def test_text_origin_is_returned_only_to_its_original_chat(pane):
+ pending = await arm(pane, reply_session_id="original-chat")
+ events = []
+ await ft.publish_result("exited", pane[0], pending, events.append)
+ assert isinstance(events[0], DelegationResultReady)
+ assert events[0].session_id == "original-chat"
+ assert events[0].status == "exited"
+
+
+async def test_refused_submission_has_no_pending_job(pane):
+ pane[0].submitted = False
+ assert await arm(pane) is None
+
+
+async def test_fast_job_finishing_between_sweeps_still_returns_its_fresh_answer(pane):
+ await arm(pane)
+ term, turns = pane
+ registry = SimpleNamespace(sessions=[SimpleNamespace(terminals=[term])])
+ events = []
+ await ft.poll_ready(registry, events.append, now=130.0)
+ assert not events # An idle terminal and an old answer are not completion evidence.
+ turns.extend([Turn("user", "Fix the bug"), Turn("assistant", "Fixed immediately.")])
+ await ft.poll_ready(registry, events.append, now=141.0)
+ assert len(events) == 1 and "Fixed immediately" in events[0].report
diff --git a/tests/unit/agentic_ide/test_pane_notifications.py b/tests/unit/agentic_ide/test_pane_notifications.py
index 75d9531092..d3398ca1b3 100644
--- a/tests/unit/agentic_ide/test_pane_notifications.py
+++ b/tests/unit/agentic_ide/test_pane_notifications.py
@@ -1089,16 +1089,36 @@ def _entry(entry_id: str) -> notifications.Notification:
)
-def test_a_finished_pane_job_is_shown_never_spoken() -> None:
- """Maintainer decision 2026-09-30: how a pane's job ended is shown (badge,
- bell entry) and never read aloud. The spoken readback also cost a model
- call on an API key per job; nothing may bring it back unnoticed."""
+def test_manual_pane_jobs_remain_silent_and_no_paid_composer_returns() -> None:
+ """Only explicit Jarvis requests opt into floor-aware result delivery."""
import importlib.util
import inspect
from jarvis.speech.pipeline import _HELD_FOR_CALL_SOURCES
assert importlib.util.find_spec("jarvis.agentic_ide.voice_readback") is None
- assert "readback" not in inspect.signature(Registry.send_prompt).parameters
- assert not hasattr(notifications.watcher(), "take_readbacks")
- assert "agentic_ide.readback" not in _HELD_FOR_CALL_SOURCES
+ assert inspect.signature(Registry.send_prompt).parameters["followup"].default is None
+ assert importlib.util.find_spec("jarvis.voice.report_readback") is None
+ assert "agentic_ide.readback" in _HELD_FOR_CALL_SOURCES
+
+
[email protected]("tracked", [False, True])
+async def test_result_tracking_is_independent_of_the_optional_bell(registry, tmp_path, tracked):
+ watcher = notifications.watcher()
+ _session, term = await _pane(registry, tmp_path)
+ pending = object() if tracked else None
+ term.delegation_result = pending
+ _draw(term, REAL_WORKING)
+ term.last_submit_at = 999.0
+ term.submit_generation = term.process_generation
+ term.last_output_at = 1000.0
+ watcher.poll(registry, now=1000.5, emit=False)
+ term.transcript.clear()
+ _draw(term, REAL_FINISHED)
+ _quiet_since(term, 1002.0)
+ watcher.poll(registry, now=1002.5, emit=False)
+ watcher.poll(registry, now=1010.0, emit=False)
+ watcher.poll(registry, now=1010.0 + notifications.SETTLE_S + 1, emit=False)
+ assert notifications.center().list() == []
+ assert watcher.take_results() == ([("completed", term, pending)] if tracked else [])
+ assert watcher.take_results() == []
diff --git a/tests/unit/agentic_ide/test_panes.py b/tests/unit/agentic_ide/test_panes.py
index 201a240aa6..d42296f7a4 100644
--- a/tests/unit/agentic_ide/test_panes.py
+++ b/tests/unit/agentic_ide/test_panes.py
@@ -131,6 +131,15 @@ async def test_split_down_pushes_the_existing_stack_down(
assert slots[bottom.name] == (0, 2), "the older pane moved down"
+async def test_split_left_and_above(registry: Registry, tmp_path: Path) -> None:
+ await _open_in_a_row(registry, tmp_path, 2)
+ left_term = await registry.add_terminal(anchor="T1", direction="left")
+ assert _layout(registry)[0][0] == left_term.name
+ above_term = await registry.add_terminal(anchor="T2", direction="above")
+ slots = {name: (column, slot) for name, column, slot in _layout(registry)}
+ assert slots[above_term.name][1] < slots["T2"][1]
+
+
# --------------------------------------------------------------------- naming
async def test_new_panes_take_the_next_free_call_sign(registry: Registry, tmp_path: Path) -> None:
await _open(registry, tmp_path, 2) # T1, T2
diff --git a/tests/unit/agentic_ide/test_session.py b/tests/unit/agentic_ide/test_session.py
index e157f7b22b..c8bf874f98 100644
--- a/tests/unit/agentic_ide/test_session.py
+++ b/tests/unit/agentic_ide/test_session.py
@@ -86,9 +86,10 @@ def test_codex_npm_shim_is_bypassed_with_absolute_node(tmp_path, monkeypatch) ->
argv = session_mod.agent_argv("codex")
- assert tuple(os.path.normcase(part) for part in argv) == tuple(
+ assert tuple(os.path.normcase(part) for part in argv[:2]) == tuple(
os.path.normcase(str(path)) for path in (node_exe, codex_js)
)
+ assert argv[2:] == ("-c", "check_for_update_on_startup=false")
assert not any(part.lower().endswith((".cmd", ".bat")) for part in argv)
@@ -360,10 +361,10 @@ async def test_a_geometry_change_rebases_an_intact_replay_and_repaints(
async def _rejoin_cut_fullscreen_pane(
- registry: Registry, fake_pty: FakePtyManager, tmp_path: Path
+ registry: Registry, fake_pty: FakePtyManager, tmp_path: Path, agent: str = "claude"
) -> tuple[object, str]:
"""A full-screen agent whose replay lost its start, re-joined at 80x24."""
- session = await _open(registry, tmp_path, [{"agent": "claude"}])
+ session = await _open(registry, tmp_path, [{"agent": agent}])
term = session.terminals[0]
await registry.attach(term.name, 80, 24, _noop_output, _noop_exit)
pty = term.pty_id
@@ -377,6 +378,32 @@ async def _rejoin_cut_fullscreen_pane(
return term, pty
+async def test_codex_home_then_erase_stops_repeated_height_nudges(
+ registry: Registry,
+ fake_pty: FakePtyManager,
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """A real Codex redraw must not cause five down/up size changes."""
+ monkeypatch.setattr(session_mod, "REPAINT_CONFIRM_S", 0.02)
+ monkeypatch.setattr(session_mod, "REPAINT_POLL_S", 0.005)
+ plain_resize = FakePtyManager.resize
+
+ def answer_first_nudge(self: FakePtyManager, tid: str, cols: int, rows: int) -> bool:
+ done = plain_resize(self, tid, cols, rows)
+ if len(self.resizes) == 1:
+ asyncio.get_running_loop().create_task(
+ self.emit(tid, "\x1b[?2026h\x1b[1;1H\x1b[Jnew Codex frame\x1b[?2026l")
+ )
+ return done
+
+ monkeypatch.setattr(FakePtyManager, "resize", answer_first_nudge)
+ _term, pty = await _rejoin_cut_fullscreen_pane(registry, fake_pty, tmp_path, agent="codex")
+ await asyncio.gather(*registry._repaint_checks)
+ sizes = [(cols, rows) for tid, cols, rows in fake_pty.resizes if tid == pty]
+ assert sizes == [(80, 23), (80, 24)]
+
+
async def test_an_ignored_repaint_nudge_is_sent_again(
registry: Registry,
fake_pty: FakePtyManager,
@@ -427,6 +454,63 @@ async def test_repaint_nudges_stop_at_the_bound_and_leave_the_real_size(
assert sizes[-1] == (80, 24), "the pane must be left at the size it really is"
+async def _ready_now(_session: object, _term: object) -> bool:
+ return True
+
+
+async def test_a_pane_resized_while_its_agent_loaded_is_repainted_once_ready(
+ registry: Registry,
+ fake_pty: FakePtyManager,
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The grid settles while the CLI boots; its real size must be drawn for.
+
+ Four panes opened together were spawned at the mount-time size and resized
+ a moment later, before Claude Code listened for size changes — so it kept
+ laying out for the old size (2026-09-29). No repaint check can run before
+ the agent takes the whole screen, so the pane is nudged once it is ready.
+ """
+ monkeypatch.setattr(session_mod, "REPAINT_CONFIRM_S", 0.02)
+ monkeypatch.setattr(session_mod, "REPAINT_POLL_S", 0.005)
+ monkeypatch.setattr(registry, "_prompt_ready", _ready_now)
+ session = await _open(registry, tmp_path, [{"agent": "claude"}])
+ term = session.terminals[0]
+ await registry.attach(term.name, 114, 30, _noop_output, _noop_exit)
+ pty = term.pty_id
+ # Still loading: nothing has taken the screen yet.
+ assert registry.resize(term.key, 104, 30, session.id)
+ assert term.resized_while_booting
+ await fake_pty.emit(pty, "\x1b[?1049h\x1b[2J\x1b[Hdrawn for 114 columns")
+ fake_pty.resizes.clear()
+
+ assert await registry._prompt_ready_then_settle(session, term)
+
+ sizes = [(cols, rows) for tid, cols, rows in fake_pty.resizes if tid == pty]
+ assert sizes[:2] == [(104, 29), (104, 30)]
+ assert not term.resized_while_booting
+ await asyncio.gather(*registry._repaint_checks)
+ assert fake_pty.resizes[-1][1:] == (104, 30)
+
+
+async def test_a_pane_never_resized_while_loading_is_not_nudged(
+ registry: Registry,
+ fake_pty: FakePtyManager,
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(registry, "_prompt_ready", _ready_now)
+ session = await _open(registry, tmp_path, [{"agent": "claude"}])
+ term = session.terminals[0]
+ await registry.attach(term.name, 104, 30, _noop_output, _noop_exit)
+ await fake_pty.emit(term.pty_id, "\x1b[?1049h\x1b[2J\x1b[Hdrawn for 104 columns")
+ fake_pty.resizes.clear()
+
+ assert await registry._prompt_ready_then_settle(session, term)
+
+ assert fake_pty.resizes == []
+
+
async def test_a_viewer_resize_the_agent_ignored_is_nudged_again(
registry: Registry,
fake_pty: FakePtyManager,
diff --git a/tests/unit/agentic_ide/test_transcript.py b/tests/unit/agentic_ide/test_transcript.py
index c8cf1ca250..6b017cb412 100644
--- a/tests/unit/agentic_ide/test_transcript.py
+++ b/tests/unit/agentic_ide/test_transcript.py
@@ -6,9 +6,46 @@ is Alex doing?" gets answered from control codes and repainted frames.
"""
from __future__ import annotations
+import json
+from pathlib import Path
+
+import pytest
+
from jarvis.agentic_ide.screen import ScreenBuffer
from jarvis.agentic_ide.transcript import ReplayBuffer, Transcript, is_noise, strip_ansi
+REPAINT_CASES = json.loads(
+ (Path(__file__).parents[2] / "fakes" / "terminal_repaint_cases.json").read_text(
+ encoding="utf-8"
+ )
+)
+
+
[email protected]("erase", REPAINT_CASES["full"])
+def test_equivalent_full_redraws_acknowledge_a_repaint_across_every_read_boundary(
+ erase: str,
+) -> None:
+ for split in range(1, len(erase)):
+ buffer = ReplayBuffer()
+ buffer.feed("\x1b[?1049hOLD FRAME")
+ buffer.feed(erase[:split])
+ buffer.feed(erase[split:] + "new frame")
+ assert buffer.clears == 1, (erase, split)
+ assert "OLD FRAME" not in buffer.text()
+ assert buffer.text().endswith(erase + "new frame")
+ buffer.feed("more output")
+ assert buffer.clears == 1, "the retained scan tail must not count an erase twice"
+
+
[email protected]("erase", REPAINT_CASES["partial"])
+def test_a_partial_erase_does_not_acknowledge_a_full_redraw(erase: str) -> None:
+ buffer = ReplayBuffer()
+ buffer.feed("\x1b[?1049hOLD FRAME")
+ for character in erase:
+ buffer.feed(character)
+ assert buffer.clears == 0
+ assert "OLD FRAME" in buffer.text()
+
def test_strip_ansi_removes_colour_and_cursor_sequences() -> None:
coloured = "\x1b[32mgreen\x1b[0m and \x1b[1;31mred\x1b[0m"
diff --git a/tests/unit/core/test_dictation_paste_chord_custom.py b/tests/unit/core/test_dictation_paste_chord_custom.py
index 33d79a1a78..c6866e1669 100644
--- a/tests/unit/core/test_dictation_paste_chord_custom.py
+++ b/tests/unit/core/test_dictation_paste_chord_custom.py
@@ -2,10 +2,9 @@
Jarvis does not paste. It asks whatever application is in front to paste, by
sending a synthetic chord — and an application that does not bind that chord
-ignores it silently. That is fine for the curated chords (they mean "paste"
-somewhere real), and it is the whole problem for a chord the user recorded:
-there is no error and nothing to read back, so reporting ``inserted`` would be
-a lie in the one module whose entire docstring is about not lying.
+ignores it silently. A recorded chord without delivery evidence is therefore
+reported as ``paste_sent``. On Windows, this also applies to curated chords
+when the native clipboard observation mechanism is unavailable.
So the vocabulary widened in two directions at once and both are pinned here:
the config field accepts a combo, and the delivery report gains ``paste_sent``
@@ -14,6 +13,8 @@ for the case where the keystroke went out and the result is unknown.
from __future__ import annotations
+from types import SimpleNamespace
+
import pytest
import jarvis.platform
@@ -43,7 +44,14 @@ def _install_fake_clipboard(monkeypatch: pytest.MonkeyPatch, fake: object) -> No
would be ignored and the test would drive the REAL system clipboard of
whoever ran the suite. Patch the attribute the code reads.
"""
+ from jarvis.dictation import insert as insert_mod
+
monkeypatch.setattr(jarvis.platform, "clipboard", fake)
+ # The delayed-rendering offer bypasses the clipboard facade. Without this
+ # seam, a fake clipboard can coexist with a REAL native clipboard owner.
+ monkeypatch.setattr(insert_mod, "_clipboard_offer_factory", lambda: None)
+ monkeypatch.setattr(insert_mod, "_foreground_target", lambda: None)
+ monkeypatch.setattr(insert_mod, "_input_block_reason", lambda target: "")
# ----------------------------------------------------------------------
@@ -135,11 +143,19 @@ def test_paste_sent_is_part_of_the_outcome_vocabulary() -> None:
assert "paste_sent" in DICTATION_OUTCOMES
+ ("os_name", "platform"), [("nt", "win32"), ("posix", "darwin"), ("posix", "linux")],
+)
def test_a_custom_chord_reports_paste_sent_and_keeps_the_clipboard(
monkeypatch: pytest.MonkeyPatch,
+ os_name: str,
+ platform: str,
) -> None:
from jarvis.dictation import insert as insert_mod
+ monkeypatch.setattr(insert_mod, "os", SimpleNamespace(name=os_name))
+ monkeypatch.setattr(insert_mod, "sys", SimpleNamespace(platform=platform))
+
class _Clipboard:
def __init__(self) -> None:
self.content = "something the user had copied"
@@ -187,12 +203,20 @@ def test_a_custom_chord_reports_paste_sent_and_keeps_the_clipboard(
assert result.detail
-def test_a_curated_chord_still_reports_inserted_and_restores(
+ ("os_name", "platform"), [("nt", "win32"), ("posix", "darwin"), ("posix", "linux")],
+)
+def test_a_curated_chord_without_native_evidence_preserves_platform_contract(
monkeypatch: pytest.MonkeyPatch,
+ os_name: str,
+ platform: str,
) -> None:
- """The honesty change must not cost the normal path its clean report."""
+ """Windows retains unconfirmed text; existing POSIX delivery is unchanged."""
from jarvis.dictation import insert as insert_mod
+ monkeypatch.setattr(insert_mod, "os", SimpleNamespace(name=os_name))
+ monkeypatch.setattr(insert_mod, "sys", SimpleNamespace(platform=platform))
+
class _Clipboard:
def __init__(self) -> None:
self.content = "previous"
@@ -226,7 +250,14 @@ def test_a_curated_chord_still_reports_inserted_and_restores(
"dictated text", paste_chord="ctrl_v", delay_ms=0, delay_after_ms=0
)
- assert result.status == "inserted"
- assert result.clipboard_restored is True
- assert clipboard.writes == ["dictated text", "previous"]
- assert clipboard.content == "previous"
+ if platform == "win32":
+ assert result.status == "paste_sent"
+ assert result.clipboard_restored is False
+ assert result.clipboard_holds_text is True
+ assert clipboard.writes == ["dictated text"]
+ assert clipboard.content == "dictated text"
+ else:
+ assert result.status == "inserted"
+ assert result.clipboard_restored is True
+ assert clipboard.writes == ["dictated text", "previous"]
+ assert clipboard.content == "previous"
diff --git a/tests/unit/dictation/test_clipboard_offer_lifecycle.py b/tests/unit/dictation/test_clipboard_offer_lifecycle.py
new file mode 100644
index 0000000000..76a66b80e0
--- /dev/null
+++ b/tests/unit/dictation/test_clipboard_offer_lifecycle.py
@@ -0,0 +1,218 @@
+"""Clipboard offers must not outlive a timeout or leak their owning thread."""
+from __future__ import annotations
+
+import threading
+
+from jarvis.platform import clipboard_offer as mod
+
+
+class EventOffer(mod.ClipboardOffer):
+ def __init__(self, *, delayed=False):
+ super().__init__("test")
+ self.delayed = delayed
+ self.runs = 0
+ self.offered = False
+
+ def _pump(self):
+ self.runs += 1
+ if self.delayed:
+ self._stop_requested.wait(0.2)
+ if self._stop_requested.is_set():
+ return
+ self.offered = True
+ self._ok = True
+ self._ready.set()
+ self._stop_requested.wait(2)
+
+
+def test_start_and_stop_are_idempotent(monkeypatch):
+ monkeypatch.setattr(mod, "available", lambda: True)
+ offer = EventOffer()
+ assert offer.start()
+ thread = offer._thread
+ assert offer.start()
+ assert offer._thread is thread
+ assert offer.runs == 1
+ offer.stop()
+ offer.stop()
+ assert not thread.is_alive()
+ assert offer._thread is None
+ assert not offer.start()
+
+
+def test_start_timeout_cancels_before_delayed_worker_can_take_clipboard(monkeypatch):
+ monkeypatch.setattr(mod, "available", lambda: True)
+ offer = EventOffer(delayed=True)
+ assert not offer.start(timeout_s=0.001)
+ assert offer._stop_requested.is_set()
+ assert offer._done.wait(1)
+ assert not offer.offered
+ assert offer._thread is None
+
+
+def test_stop_timeout_keeps_handle_for_later_join(monkeypatch):
+ monkeypatch.setattr(mod, "available", lambda: True)
+ release = threading.Event()
+ offer = EventOffer()
+
+ def pump():
+ offer._ok = True
+ offer._ready.set()
+ release.wait(1)
+
+ offer._pump = pump
+ try:
+ assert offer.start()
+ thread = offer._thread
+ offer.stop(timeout_s=0)
+ assert offer._thread is thread
+ finally:
+ release.set()
+ offer.stop()
+ assert not thread.is_alive()
+ assert offer._thread is None
+
+
+def test_read_wait_wakes_when_offer_loses_ownership():
+ offer = EventOffer()
+ offer.lost_ownership = True
+ assert offer.wait_for_read(exclude_pids=set(), after_s=0, timeout_s=10) is None
+
+
+class OwnedClipboard:
+ def __init__(self, owner):
+ self.owner = owner
+ self.locked = False
+ self.emptied = False
+
+ def OpenClipboard(self, hwnd):
+ self.locked = True
+ return True
+
+ def GetClipboardOwner(self):
+ assert self.locked
+ return self.owner
+
+ def EmptyClipboard(self):
+ assert self.locked
+ self.emptied = True
+ return True
+
+ def CloseClipboard(self):
+ self.locked = False
+
+
+def test_restore_cannot_overwrite_a_new_owner_even_with_same_text():
+ clipboard = OwnedClipboard(owner=456)
+ offer = EventOffer()
+ offer._restore_text = "previous"
+ writes = []
+ offer._restore_owned_text(clipboard, 123, lambda text=None: writes.append(text))
+ assert not writes
+ assert not clipboard.emptied
+ assert not clipboard.locked
+ assert not offer.restored
+
+
+def test_restore_checks_and_writes_under_the_same_lock():
+ clipboard = OwnedClipboard(owner=123)
+ offer = EventOffer()
+ offer._restore_text = "previous"
+ writes = []
+
+ def render(text=None):
+ assert clipboard.locked
+ writes.append(text)
+ return True
+
+ offer._restore_owned_text(clipboard, 123, render)
+ assert writes == ["previous"]
+ assert offer.restored
+ assert offer._text == "previous"
+ assert not clipboard.locked
+
+
+def test_failed_restore_republishes_transcript_before_unlocking():
+ clipboard = OwnedClipboard(owner=123)
+ offer = EventOffer()
+ offer._restore_text = "previous"
+ writes = []
+
+ def render(text=None):
+ assert clipboard.locked
+ writes.append(text)
+ return text is None
+
+ offer._restore_owned_text(clipboard, 123, render)
+ assert writes == ["previous", None]
+ assert not offer.restored
+ assert not clipboard.locked
+
+
+class NativeFunction:
+ """ctypes-callable stand-in retaining the signatures assigned by the pump."""
+ def __init__(self, call=lambda *args: 0):
+ self.call = call
+ self.argtypes = None
+ self.restype = None
+
+ def __call__(self, *args):
+ return self.call(*args)
+
+
+class NativeLibrary:
+ def __init__(self, **calls):
+ self.calls = {name: NativeFunction(call) for name, call in calls.items()}
+
+ def __getattr__(self, name):
+ return self.calls.setdefault(name, NativeFunction())
+
+
+def test_stop_dispatched_during_message_drain_never_enters_one_second_wait(monkeypatch):
+ """Exercise the actual native pump with a stop arriving inside DispatchMessage."""
+ import ctypes
+
+ offer = mod.ClipboardOffer("test")
+ waits = []
+ callbacks = []
+ drained = False
+
+ def register_class(pointer):
+ callbacks.append(pointer._obj.lpfnWndProc)
+ return 1
+
+ def peek_message(pointer, *args):
+ nonlocal drained
+ if drained:
+ return 0
+ drained = True
+ pointer._obj.message = mod._WM_APP_STOP
+ return 1
+
+ def dispatch_message(pointer):
+ return callbacks[0](123, pointer._obj.message, 0, 0)
+
+ user32 = NativeLibrary(
+ RegisterClassW=register_class,
+ CreateWindowExW=lambda *args: 123,
+ OpenClipboard=lambda *args: 1,
+ PeekMessageW=peek_message,
+ DispatchMessageW=dispatch_message,
+ IsWindow=lambda *args: 1,
+ MsgWaitForMultipleObjectsEx=lambda *args: waits.append(args[2]),
+ )
+ kernel32 = NativeLibrary(GetModuleHandleW=lambda *args: 1)
+ monkeypatch.setattr(mod, "available", lambda: True)
+ monkeypatch.setattr(ctypes, "WINFUNCTYPE", ctypes.CFUNCTYPE, raising=False)
+ monkeypatch.setattr(ctypes, "set_last_error", lambda value: None, raising=False)
+ monkeypatch.setattr(ctypes, "get_last_error", lambda: 0, raising=False)
+ monkeypatch.setattr(
+ ctypes, "WinDLL", lambda name, **kwargs: user32 if name == "user32" else kernel32,
+ raising=False,
+ )
+
+ offer._pump()
+
+ assert offer._stop_requested.is_set()
+ assert waits == []
+ assert offer._hwnd == 0
diff --git a/tests/unit/dictation/test_delivery_latency.py b/tests/unit/dictation/test_delivery_latency.py
new file mode 100644
index 0000000000..6f53d7622a
--- /dev/null
+++ b/tests/unit/dictation/test_delivery_latency.py
@@ -0,0 +1,130 @@
+"""Finished text is independent of a stalled observer, with ordered recovery."""
+
+from __future__ import annotations
+
+import asyncio
+import threading
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.core.bus import EventBus
+from jarvis.core.config import DictationConfig
+from jarvis.core.events import DictationCompleted, DictationTranscript
+from jarvis.speech import pipeline as pipeline_mod
+from jarvis.speech.pipeline import SpeechPipeline
+
+
+def _pipeline(monkeypatch):
+ monkeypatch.setattr(pipeline_mod, "_DICTATION_EVENT_TIMEOUT_S", 0.03)
+ pipe = SpeechPipeline.__new__(SpeechPipeline)
+ pipe._bus = EventBus()
+ pipe._dictation_cfg = DictationConfig(polish=False)
+ recorded = []
+
+ async def record(**values):
+ recorded.append(values)
+
+ pipe._record_dictation = record
+ return pipe, recorded
+
+
+async def _finish(pipe, *, target="chat"):
+ return await pipe._finish_dictation(
+ raw_text="These words must arrive.", language="en", duration_s=2,
+ target=target, hung_up=False,
+ )
+
+
+async def test_healthy_chat_receives_final_and_completion_beside_stalled_observer(monkeypatch):
+ pipe, recorded = _pipeline(monkeypatch)
+ healthy = []
+ cancelled = []
+
+ async def receive(event):
+ healthy.append(event)
+
+ async def stalled(event):
+ try:
+ await asyncio.Event().wait()
+ finally:
+ cancelled.append(type(event))
+
+ pipe._bus.subscribe_all(receive)
+ pipe._bus.subscribe_all(stalled)
+ result = await asyncio.wait_for(_finish(pipe), timeout=1)
+
+ assert result == "These words must arrive."
+ assert [type(event) for event in healthy] == [DictationTranscript, DictationCompleted]
+ assert cancelled == [DictationTranscript, DictationCompleted]
+ assert recorded[0]["cleaned"] == result
+ assert "final_notification:observer_timeout" in recorded[0]["stt_audit"]
+ assert recorded[0]["stt_audit"] == healthy[-1].stt_audit
+
+
+async def test_external_insertion_does_not_wait_for_notification(monkeypatch):
+ pipe, recorded = _pipeline(monkeypatch)
+ inserted = threading.Event()
+ observer_started = asyncio.Event()
+ observer_release = asyncio.Event()
+ monkeypatch.setattr(pipeline_mod, "_DICTATION_EVENT_TIMEOUT_S", 1)
+
+ async def stalled(event):
+ if isinstance(event, DictationTranscript):
+ observer_started.set()
+ await observer_release.wait()
+
+ def insert(text):
+ inserted.set()
+ return SimpleNamespace(status="inserted", detail="", method="fake")
+
+ pipe._bus.subscribe_all(stalled)
+ pipe._insert_dictation = insert
+ task = asyncio.create_task(_finish(pipe, target="insert"))
+ try:
+ await asyncio.wait_for(observer_started.wait(), timeout=1)
+ assert await asyncio.to_thread(inserted.wait, 0.5)
+ assert not task.done()
+ finally:
+ observer_release.set()
+ await asyncio.wait_for(task, timeout=1)
+ assert recorded[0]["outcome_name"] == "inserted"
+
+
+async def test_an_insertion_exception_still_preserves_text_and_terminal_event(monkeypatch):
+ pipe, recorded = _pipeline(monkeypatch)
+ events = []
+
+ async def receive(event):
+ events.append(event)
+
+ def insert(text):
+ raise RuntimeError("fake unavailable desktop")
+
+ pipe._bus.subscribe_all(receive)
+ pipe._insert_dictation = insert
+ result = await _finish(pipe, target="insert")
+ assert result == "These words must arrive."
+ assert events[-1].outcome == "failed"
+ assert recorded[0]["cleaned"] == result
+ assert recorded[0]["outcome_name"] == "failed"
+
+
+async def test_cancellation_retires_notification_before_next_turn(monkeypatch):
+ pipe, _recorded = _pipeline(monkeypatch)
+ entered, retired = asyncio.Event(), asyncio.Event()
+
+ async def stalled(event):
+ entered.set()
+ try:
+ await asyncio.Event().wait()
+ finally:
+ retired.set()
+
+ pipe._bus.subscribe_all(stalled)
+ task = asyncio.create_task(_finish(pipe))
+ await entered.wait()
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert retired.is_set()
diff --git a/tests/unit/dictation/test_incremental_polish.py b/tests/unit/dictation/test_incremental_polish.py
index 0603812edb..14447bb544 100644
--- a/tests/unit/dictation/test_incremental_polish.py
+++ b/tests/unit/dictation/test_incremental_polish.py
@@ -285,3 +285,48 @@ async def test_translation_keeps_the_whole_text_pass(
assert calls[0].get("translate_to") == "en"
assert not calls[0].get("preceding_text")
assert _audit(events, "polish_mode") == "whole"
+
+
+async def test_stalled_prefix_worker_is_retired_without_reformatting_completed_text(
+ _no_preview: None, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ import jarvis.dictation.polish as polish
+ import jarvis.speech.pipeline as pipeline_mod
+
+ monkeypatch.setattr(pipeline_mod, "_DICTATION_PREFIX_POLISH_WAIT_S", 0.03)
+ blocked, retired = asyncio.Event(), asyncio.Event()
+ calls: list[dict[str, Any]] = []
+
+ async def format_piece(raw: str, **kwargs: Any) -> PolishOutcome:
+ calls.append({"raw": raw, **kwargs})
+ if len(calls) == 2:
+ blocked.set()
+ try:
+ await asyncio.Event().wait()
+ finally:
+ retired.set()
+ elif len(calls) > 2:
+ assert retired.is_set(), "replacement overlapped the retired formatter"
+ return PolishOutcome(text=raw.upper(), status="applied")
+
+ monkeypatch.setattr(polish, "polish_transcript", format_piece)
+ mic = _FakeMic(_voiced(12.0))
+ pipe, events = _session_pipeline(_ScriptedSTT(), mic)
+ task = asyncio.create_task(pipe._dictation_session())
+ try:
+ await asyncio.wait_for(mic.delivered.wait(), timeout=3)
+ await asyncio.wait_for(blocked.wait(), timeout=3)
+ pipe._dictation_stop_event.set()
+ await asyncio.wait_for(task, timeout=3)
+ finally:
+ if not task.done():
+ task.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await task
+
+ assert retired.is_set()
+ assert len(calls) == 3
+ assert "alpha" not in calls[-1]["raw"]
+ assert calls[-1]["preceding_text"].startswith("ALPHA")
+ assert _final(events).text.count("ALPHA") == 1
+ assert _audit(events, "polish_mode") == "incremental"
diff --git a/tests/unit/dictation/test_insert.py b/tests/unit/dictation/test_insert.py
index cf82ad3421..710cdebe3b 100644
--- a/tests/unit/dictation/test_insert.py
+++ b/tests/unit/dictation/test_insert.py
@@ -8,6 +8,7 @@ the clipboard before any keystroke, and it stays there when the paste fails.**
from __future__ import annotations
import sys
+from types import SimpleNamespace
import pytest
@@ -75,6 +76,9 @@ def wired(monkeypatch: pytest.MonkeyPatch):
# These tests pin the PLAIN chord path (every OS). The Windows path that
# watches who reads the clipboard has its own file: test_insert_verified.py.
monkeypatch.setattr(insert_mod, "_clipboard_offer_factory", lambda: None)
+ # Preserve coverage of the existing POSIX plain route. Windows degrades
+ # honestly to paste_sent if its read-observation helper is unavailable.
+ monkeypatch.setattr(insert_mod, "os", SimpleNamespace(name="posix"))
return clipboard, actuator
diff --git a/tests/unit/dictation/test_insert_verified.py b/tests/unit/dictation/test_insert_verified.py
index 0b64f0840c..e22e7e550b 100644
--- a/tests/unit/dictation/test_insert_verified.py
+++ b/tests/unit/dictation/test_insert_verified.py
@@ -1,310 +1,244 @@
-"""The Windows paste path that watches WHO reads the clipboard.
-
-A synthetic paste chord is a request, not a paste: an xterm.js terminal in a
-Tauri/Electron app swallows Ctrl+V, and a WebView that pastes through an async
-bridge may read the clipboard long after a 120 ms "restore the old clipboard"
-timer has fired. The delayed-rendering offer turns both into observations. These
-tests drive ``_insert_windows_verified`` with a fake offer so each host state
-(sighted / blind) and each route (chord, cascade, typing) is pinned without a
-real clipboard. They run on every OS — the platform gate is the factory, and
-the factory is what is swapped.
-"""
-
+"""A paste timeout must never duplicate input or erase an unconfirmed transcript."""
from __future__ import annotations
from dataclasses import dataclass, field
+from types import SimpleNamespace
import pytest
from jarvis.dictation import insert as insert_mod
from jarvis.dictation.insert import insert_text
-from .test_insert import FakeActuator, FakeClipboard # noqa: TID252 — sibling fakes
+from .test_insert import FakeActuator, FakeClipboard # noqa: TID252
-SELF_PID = 4242
-WATCHER_PID = 77
-TARGET_PID = 9001
+TARGET = (123, 9001)
@dataclass
class FakeRead:
pid: int
- exe: str
- at: float
+ exe: str = ""
+ at: float = 0.1
observed: str = "render"
@dataclass
class FakeOffer:
- """Scripted clipboard offer: who reads, and when (offer-relative seconds)."""
-
text: str
script: list[FakeRead] = field(default_factory=list)
- started: bool = False
stopped: bool = False
- clock: float = 0.0
+ lost_ownership: bool = False
+ restored: bool = False
+ clipboard: FakeClipboard | None = None
+ waits: list[float] = field(default_factory=list)
- def start(self) -> bool:
- self.started = True
+ def start(self):
return True
- def reads(self) -> list[FakeRead]:
- return [r for r in self.script if r.at <= self.clock]
+ def reads(self):
+ return [r for r in self.script if r.at <= 0.05]
- def elapsed(self) -> float:
- return self.clock
+ def elapsed(self):
+ return 0.05
def wait_for_read(self, *, exclude_pids, after_s, timeout_s):
- self.clock = after_s + timeout_s
- for read in self.script:
- if after_s <= read.at <= self.clock and read.pid not in exclude_pids:
- return read
- return None
+ self.waits.append(timeout_s)
+ return next((r for r in self.script if after_s <= r.at <= after_s + timeout_s
+ and r.pid not in exclude_pids), None)
- def stop(self) -> None:
+ def stop(self, *, restore_text=None):
self.stopped = True
-
-
-class OfferFactory:
- """Hands out one scripted offer per chord attempt, in order."""
-
- def __init__(self, *scripts: list[FakeRead]) -> None:
- self.scripts = list(scripts)
- self.offers: list[FakeOffer] = []
-
- def __call__(self, text: str) -> FakeOffer:
- script = self.scripts.pop(0) if self.scripts else []
- offer = FakeOffer(text, script)
- self.offers.append(offer)
- return offer
+ if restore_text is not None and not self.lost_ownership:
+ self.restored = self.clipboard.write_text(restore_text)
@pytest.fixture()
-def verified(monkeypatch: pytest.MonkeyPatch):
- """insert_text with the verified path active and every OS call faked."""
- clipboard = FakeClipboard()
- actuator = FakeActuator()
-
+def verified(monkeypatch):
+ clipboard, actuator = FakeClipboard(), FakeActuator()
import jarvis.platform.clipboard as real_clipboard
-
monkeypatch.setattr(real_clipboard, "read_text", clipboard.read_text)
monkeypatch.setattr(real_clipboard, "write_text", clipboard.write_text)
monkeypatch.setattr(
- insert_mod, "describe_target", lambda: insert_mod.TargetReport(True, "", "")
+ insert_mod, "describe_target", lambda: insert_mod.TargetReport(True, "", ""),
)
monkeypatch.setattr("jarvis.cu.actuate.get_actuator", lambda: actuator)
- monkeypatch.setattr(insert_mod.time, "sleep", lambda _s: None)
- monkeypatch.setattr(insert_mod, "_foreground_exe", lambda: "target.exe")
- monkeypatch.setattr("os.getpid", lambda: SELF_PID)
- monkeypatch.setattr(insert_mod, "_LEARNED_ROUTES", {})
-
- # The settle window advances the fake clock so early reads become visible.
- def install(factory: OfferFactory) -> None:
- def make(text: str) -> FakeOffer:
- offer = factory(text)
- offer.clock = 0.05
- return offer
-
- monkeypatch.setattr(insert_mod, "_clipboard_offer_factory", lambda: make)
-
- restores: list[tuple[str | None, str, float]] = []
- monkeypatch.setattr(
- insert_mod,
- "_restore_later",
- lambda cb, previous, text, grace: restores.append((previous, text, grace)),
- )
- return clipboard, actuator, install, restores
-
-
-# --------------------------------------------------------------------------
-# Sighted host: nobody read during the settle window, so silence is proof.
-# --------------------------------------------------------------------------
-
+ monkeypatch.setattr(insert_mod.time, "sleep", lambda _: None)
+ monkeypatch.setattr(insert_mod, "_foreground_target", lambda: TARGET)
+ monkeypatch.setattr(insert_mod, "_input_block_reason", lambda _: "")
+ monkeypatch.setattr(insert_mod, "os", SimpleNamespace(name="nt", getpid=lambda: 4242))
+ offers = []
+
+ def install(*reads):
+ offer = FakeOffer("dictated text", list(reads))
+ offer.clipboard = clipboard
+ offers.append(offer)
+ monkeypatch.setattr(insert_mod, "_clipboard_offer_factory", lambda: lambda text: offer)
+ return offer
-def test_sighted_paste_is_proven_by_the_target_reading(verified) -> None:
- clipboard, actuator, install, restores = verified
- factory = OfferFactory([FakeRead(TARGET_PID, "app.exe", at=0.1)])
- install(factory)
+ return clipboard, actuator, install
- result = insert_text("dictated text", paste_chord="ctrl_v")
+def test_target_render_confirms_one_paste_and_restores(verified):
+ clipboard, actuator, install = verified
+ offer = install(FakeRead(TARGET[1]))
+ result = insert_text("dictated text")
assert result.status == "inserted"
- assert result.method == "clipboard+ctrl_v"
- assert actuator.combos == [["ctrl", "v"]]
- # Restored right away — the read was seen, no need to wait.
- assert result.clipboard_restored is True
+ assert result.clipboard_restored
assert clipboard.content == "previous contents"
- assert restores == []
- assert insert_mod._LEARNED_ROUTES == {"target.exe": "ctrl_v"}
-
-
-def test_sighted_silence_cascades_to_the_next_chord(verified) -> None:
- """Ctrl+V unanswered, Ctrl+Shift+V answered: one paste, the right one."""
- _clipboard, actuator, install, _restores = verified
- factory = OfferFactory([], [FakeRead(TARGET_PID, "app.exe", at=0.2)])
- install(factory)
-
- result = insert_text("dictated text", paste_chord="ctrl_v")
-
- assert result.status == "inserted"
- assert result.method == "clipboard+ctrl_shift_v"
- assert actuator.combos == [["ctrl", "v"], ["ctrl", "shift", "v"]]
- assert all(o.stopped for o in factory.offers)
- assert insert_mod._LEARNED_ROUTES == {"target.exe": "ctrl_shift_v"}
-
-
-def test_sighted_no_chord_answered_types_with_soft_newlines(verified) -> None:
- """An app that binds no paste chord still gets the words — typed."""
- clipboard, actuator, install, _restores = verified
- install(OfferFactory([], [], []))
-
- result = insert_text("first line\nsecond line", paste_chord="ctrl_v")
-
- assert result.status == "inserted"
- assert result.method == "type"
- assert actuator.combos == [
- ["ctrl", "v"],
- ["ctrl", "shift", "v"],
- ["shift", "insert"],
- ["shift", "enter"], # the line break, never a submitting Enter
- ]
- assert actuator.typed == ["first line", "second line"]
- # The text stays on the clipboard: if the keystrokes landed nowhere it is
- # the user's way back.
- assert result.clipboard_holds_text is True
- assert clipboard.content == "first line\nsecond line"
- assert insert_mod._LEARNED_ROUTES == {"target.exe": "type"}
-
-
-def test_learned_type_route_skips_the_chords(verified) -> None:
- _clipboard, actuator, install, _restores = verified
- insert_mod._LEARNED_ROUTES["target.exe"] = "type"
- install(OfferFactory())
-
- result = insert_text("again", paste_chord="ctrl_v")
-
- assert result.method == "type"
- assert actuator.combos == []
- assert actuator.typed == ["again"]
+ assert actuator.combos == [["ctrl", "v"]]
+ assert offer.stopped
+
+
+ [],
+ [FakeRead(TARGET[1], at=3.0)],
+ [FakeRead(77)],
+ [FakeRead(TARGET[1], observed="open")],
+])
+def test_missing_late_unrelated_or_open_evidence_never_retries_or_erases(verified, reads):
+ clipboard, actuator, install = verified
+ offer = install(*reads)
+ result = insert_text("dictated text")
+ assert result.status == "paste_sent"
+ assert result.clipboard_holds_text
+ assert not result.clipboard_restored
+ assert clipboard.writes == ["dictated text"]
+ assert actuator.combos == [["ctrl", "v"]]
+ assert actuator.typed == []
+ assert sum(offer.waits) <= insert_mod.PASTE_READ_WAIT_S
+ assert offer.stopped
-def test_learned_chord_goes_first(verified) -> None:
- _clipboard, actuator, install, _restores = verified
- insert_mod._LEARNED_ROUTES["target.exe"] = "shift_insert"
- install(OfferFactory([FakeRead(TARGET_PID, "app.exe", at=0.1)]))
+def test_early_watcher_returns_without_waiting_and_keeps_text(verified):
+ clipboard, actuator, install = verified
+ offer = install(FakeRead(77, at=0.004))
+ result = insert_text("dictated text")
+ assert result.status == "paste_sent"
+ assert offer.waits == []
+ assert clipboard.content == "dictated text"
+ assert actuator.combos == [["ctrl", "v"]]
- result = insert_text("again", paste_chord="ctrl_v")
+def test_failure_in_one_field_cannot_change_route_for_later_field(verified):
+ _, actuator, install = verified
+ install()
+ insert_text("first")
+ install(FakeRead(TARGET[1]))
+ result = insert_text("second", paste_chord="shift_insert")
+ assert actuator.combos == [["ctrl", "v"], ["shift", "insert"]]
+ assert actuator.typed == []
assert result.method == "clipboard+shift_insert"
- assert actuator.combos == [["shift", "insert"]]
-
-
-# --------------------------------------------------------------------------
-# Blind host: a watcher consumed the one render, silence proves nothing.
-# --------------------------------------------------------------------------
-
-
-def test_blind_host_sends_one_chord_and_restores_late(verified) -> None:
- """RDP clipboard sync read at arm time: no cascade, no typing, no 120 ms timer."""
- clipboard, actuator, install, restores = verified
- install(OfferFactory([FakeRead(WATCHER_PID, "msrdc.exe", at=0.004)]))
- result = insert_text("dictated text", paste_chord="ctrl_v")
- assert result.status == "inserted"
- assert result.method == "clipboard+ctrl_v"
- assert actuator.combos == [["ctrl", "v"]] # exactly one — never a double paste
- assert actuator.typed == []
- # Not restored on the spot; scheduled with the long grace instead.
- assert result.clipboard_restored is False
- assert result.clipboard_holds_text is True
- assert clipboard.content == "dictated text"
- assert restores == [("previous contents", "dictated text", insert_mod.RESTORE_GRACE_S)]
- assert insert_mod._LEARNED_ROUTES == {}
+def test_custom_chord_remains_exactly_the_configured_chord(verified):
+ _, actuator, install = verified
+ install()
+ result = insert_text("dictated text", paste_chord="ctrl+alt+insert")
+ assert result.status == "paste_sent"
+ assert actuator.combos == [["ctrl", "alt", "insert"]]
-def test_slow_watcher_after_the_chord_also_means_blind(verified) -> None:
- _clipboard, actuator, install, restores = verified
- install(OfferFactory([FakeRead(WATCHER_PID, "svchost.exe", at=0.2)]))
+def test_native_probe_failure_after_sending_cannot_send_again(verified):
+ clipboard, actuator, install = verified
+ offer = install()
- result = insert_text("dictated text", paste_chord="ctrl_v")
+ def fail(**kwargs):
+ raise OSError("probe unavailable")
- assert result.status == "inserted"
+ offer.wait_for_read = fail
+ result = insert_text("dictated text")
+ assert result.status == "paste_sent"
assert actuator.combos == [["ctrl", "v"]]
- assert len(restores) == 1
-
-
-def test_blind_host_still_proves_a_paste_by_polling(verified) -> None:
- """The watcher cached the text, but the target was SEEN holding the clipboard."""
- clipboard, actuator, install, restores = verified
- install(
- OfferFactory(
- [
- FakeRead(WATCHER_PID, "msrdc.exe", at=0.004),
- FakeRead(TARGET_PID, "webview.exe", at=0.3, observed="open"),
- ]
- )
- )
+ assert clipboard.content == "dictated text"
+ assert offer.stopped
- result = insert_text("dictated text", paste_chord="ctrl_v")
- assert result.status == "inserted"
+def test_failed_offer_uses_one_plain_chord_but_keeps_windows_clipboard(verified):
+ clipboard, actuator, install = verified
+ offer = install()
+ offer.start = lambda: False
+ result = insert_text("dictated text")
+ assert result.status == "paste_sent"
assert actuator.combos == [["ctrl", "v"]]
- assert result.clipboard_restored is True
- assert clipboard.content == "previous contents"
- assert restores == []
-
-
-def test_blind_custom_chord_stays_honest(verified) -> None:
- """A user-recorded chord with no evidence is ``paste_sent``, not ``inserted``."""
- _clipboard, _actuator, install, _restores = verified
- install(OfferFactory([FakeRead(WATCHER_PID, "msrdc.exe", at=0.004)]))
+ assert clipboard.content == "dictated text"
+ assert offer.stopped
- result = insert_text("dictated text", paste_chord="ctrl+alt+insert")
- assert result.status == "paste_sent"
- assert result.method == "clipboard+ctrl+alt+insert"
+def test_changed_focus_or_held_modifier_stops_before_input(verified, monkeypatch):
+ clipboard, actuator, install = verified
+ offer = install()
+ monkeypatch.setattr(insert_mod, "_input_block_reason", lambda _: "A modifier is held.")
+ result = insert_text("dictated text")
+ assert result.status == "clipboard_only"
+ assert actuator.combos == []
+ assert clipboard.content == "dictated text"
+ assert offer.stopped
-# --------------------------------------------------------------------------
-# Degradations
-# --------------------------------------------------------------------------
+def test_user_copy_during_confirmed_paste_is_not_overwritten(verified):
+ clipboard, _, install = verified
+ offer = install(FakeRead(TARGET[1]))
+ def stop(*, restore_text=None):
+ clipboard.content = "user copy"
+ offer.lost_ownership = True
+ offer.stop = stop
+ result = insert_text("dictated text")
+ assert not result.clipboard_restored
+ assert clipboard.content == "user copy"
-def test_offer_that_cannot_take_the_clipboard_uses_the_plain_path(verified) -> None:
- clipboard, actuator, install, _restores = verified
+def test_unreadable_clipboard_prevents_restore(monkeypatch):
+ clipboard = FakeClipboard(initial=None)
+ assert not insert_mod._clipboard_still_holds(clipboard, "dictated text")
- class Refusing(FakeOffer):
- def start(self) -> bool:
- return False
+ def fail():
+ raise OSError("clipboard busy")
- install(OfferFactory())
- monkeypatch_factory = lambda: Refusing # noqa: E731
- insert_mod._clipboard_offer_factory = monkeypatch_factory # type: ignore[assignment]
+ monkeypatch.setattr(clipboard, "read_text", fail)
+ assert not insert_mod._clipboard_still_holds(clipboard, "dictated text")
- result = insert_text("dictated text", paste_chord="ctrl_v")
- assert result.status == "inserted"
- assert actuator.combos == [["ctrl", "v"]]
+def test_overlapping_delivery_does_not_replace_active_clipboard(verified):
+ clipboard, actuator, install = verified
+ install()
+ insert_mod._INSERT_LOCK.acquire()
+ try:
+ result = insert_text("other text")
+ finally:
+ insert_mod._INSERT_LOCK.release()
+ assert result.status == "unavailable"
assert clipboard.content == "previous contents"
+ assert actuator.combos == []
-def test_restore_later_only_restores_our_own_text(monkeypatch: pytest.MonkeyPatch) -> None:
- """A copy the user makes during the grace window is never overwritten."""
- clipboard = FakeClipboard(initial="dictated text")
- monkeypatch.setattr(insert_mod.time, "sleep", lambda _s: None)
+def test_explicit_typing_is_still_available(verified):
+ _, actuator, install = verified
+ install()
+ result = insert_text("dictated text", method="type")
+ assert result.method == "type"
+ assert actuator.typed == ["dictated text"]
+ assert actuator.combos == []
+
- import threading
[email protected]("held_vk", [0x10, 0x11, 0x12, 0x5B, 0x5C])
+def test_native_modifier_snapshot_refuses_modified_paste(monkeypatch, held_vk):
+ import ctypes
- class Immediate(threading.Thread):
- def start(self) -> None: # run inline for the test
- self.run()
+ def key_state(vk):
+ return 0x8000 if vk == held_vk else 0
- monkeypatch.setattr(insert_mod.threading, "Thread", Immediate)
+ monkeypatch.setattr(insert_mod, "os", SimpleNamespace(name="nt"))
+ monkeypatch.setattr(insert_mod, "sys", SimpleNamespace(platform="win32"))
+ monkeypatch.setattr(insert_mod, "_foreground_target", lambda: TARGET)
+ monkeypatch.setattr(
+ ctypes, "WinDLL", lambda *a, **kw: SimpleNamespace(GetAsyncKeyState=key_state),
+ raising=False,
+ )
+ assert "modifier" in insert_mod._input_block_reason(TARGET)
- insert_mod._restore_later(clipboard, "previous", "dictated text", 0.0)
- assert clipboard.content == "previous"
- clipboard.content = "something the user copied"
- insert_mod._restore_later(clipboard, "previous", "dictated text", 0.0)
- assert clipboard.content == "something the user copied"
+def test_foreground_change_prevents_keyboard_probe_and_paste(monkeypatch):
+ monkeypatch.setattr(insert_mod, "os", SimpleNamespace(name="nt"))
+ monkeypatch.setattr(insert_mod, "sys", SimpleNamespace(platform="win32"))
+ monkeypatch.setattr(insert_mod, "_foreground_target", lambda: (124, 9002))
+ assert "window changed" in insert_mod._input_block_reason(TARGET)
diff --git a/tests/unit/jarvisx/__init__.py b/tests/unit/jarvisx/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/unit/jarvisx/test_geometry.py b/tests/unit/jarvisx/test_geometry.py
new file mode 100644
index 0000000000..68ca889820
--- /dev/null
+++ b/tests/unit/jarvisx/test_geometry.py
@@ -0,0 +1,132 @@
+"""Jarvis X coordinate and timing math (HiDPI mapping, cards, pill)."""
+
+from __future__ import annotations
+
+import pytest
+
+from jarvis.jarvisx import geometry
+
+# A 4K panel at 150 % left of a 1080p panel at 100 %, as Windows reports it to
+# a per-monitor DPI aware process (physical pixels).
+_WIN_MONITORS = [
+ {"left": 0, "top": 0, "width": 5760, "height": 2160},
+ {"left": 0, "top": 0, "width": 3840, "height": 2160},
+ {"left": 3840, "top": 0, "width": 1920, "height": 1080},
+]
+
+
+def test_selection_fractions_normalize_any_drag_direction() -> None:
+ assert geometry.selection_fractions(300, 200, 100, 50, 400, 400) == pytest.approx(
+ (0.25, 0.125, 0.5, 0.375)
+ )
+
+
+def test_selection_is_clamped_to_the_screen() -> None:
+ frac = geometry.selection_fractions(-50, -50, 200, 100, 400, 200)
+ assert frac == pytest.approx((0.0, 0.0, 0.5, 0.5))
+
+
+def test_a_click_is_not_a_selection() -> None:
+ assert geometry.selection_fractions(10, 10, 12, 40, 400, 400) is None
+
+
+def test_hidpi_screen_maps_to_its_physical_monitor() -> None:
+ # Qt reports the 4K panel at 150 % as 2560x1440 logical, dpr 1.5.
+ qt_4k = {"x": 0, "y": 0, "w": 2560, "h": 1440, "dpr": 1.5}
+ # Qt's logical origin of the second screen follows the first's logical width.
+ qt_hd = {"x": 2560, "y": 0, "w": 1920, "h": 1080, "dpr": 1.0}
+ assert geometry.match_monitor(qt_4k, _WIN_MONITORS) is _WIN_MONITORS[1]
+ assert geometry.match_monitor(qt_hd, _WIN_MONITORS) is _WIN_MONITORS[2]
+
+
+def test_fractions_become_physical_pixels_on_the_matched_monitor() -> None:
+ monitor = _WIN_MONITORS[1]
+ # A 1000x500 logical drag at (100, 100) on the 150 % screen.
+ frac = geometry.selection_fractions(100, 100, 1100, 600, 2560, 1440)
+ assert frac is not None
+ assert geometry.fraction_to_bbox(monitor, frac) == (150, 150, 1500, 750)
+
+
+def test_fraction_to_bbox_respects_monitor_origin() -> None:
+ assert geometry.fraction_to_bbox(_WIN_MONITORS[2], (0.5, 0.5, 0.5, 0.5)) == (
+ 4800,
+ 540,
+ 960,
+ 540,
+ )
+
+
+def test_macos_points_match_without_scaling() -> None:
+ monitors = [
+ {"left": 0, "top": 0, "width": 1512, "height": 982},
+ {"left": 0, "top": 0, "width": 1512, "height": 982},
+ ]
+ qt = {"x": 0, "y": 0, "w": 1512, "h": 982, "dpr": 2.0}
+ assert geometry.match_monitor(qt, monitors) is monitors[1]
+
+
+def test_match_monitor_without_monitors_is_none() -> None:
+ assert geometry.match_monitor({"x": 0, "y": 0, "w": 1, "h": 1, "dpr": 1}, []) is None
+
+
+def test_monitor_fraction_clips_to_the_monitor() -> None:
+ frac = geometry.monitor_fraction((3740, 0, 300, 1080), _WIN_MONITORS[2])
+ assert frac == pytest.approx((0.0, 0.0, 200 / 1920, 1.0))
+
+
+def test_even_size_trims_odd_dimensions() -> None:
+ assert geometry.even_size(1921, 1081) == (1920, 1080)
+ assert geometry.even_size(1, 1) == (2, 2)
+
+
+def test_cards_stack_upward_newest_lowest() -> None:
+ corners = geometry.card_stack(1000, 800, [(300, 200), (300, 100)], margin=20, gap=10)
+ assert corners == [(680, 580), (680, 470)]
+
+
+def test_card_timer_expires_after_the_dismiss_time() -> None:
+ timer = geometry.CardTimer(persist=False, dismiss_s=30, now=100.0)
+ assert not timer.expired(129.9)
+ assert timer.expired(130.0)
+
+
+def test_persistent_card_never_expires() -> None:
+ timer = geometry.CardTimer(persist=True, dismiss_s=1, now=0.0)
+ assert timer.remaining(10_000.0) is None
+ assert not timer.expired(10_000.0)
+
+
+def test_hover_pauses_and_leaving_restarts_the_countdown() -> None:
+ timer = geometry.CardTimer(persist=False, dismiss_s=10, now=0.0)
+ timer.hover(True, 8.0)
+ assert not timer.expired(50.0)
+ timer.hover(False, 50.0)
+ assert timer.remaining(55.0) == pytest.approx(5.0)
+ assert timer.expired(60.0)
+
+
[email protected](("raw", "expected"), [(0, 1), (99999, 3600), ("x", 30), (12.7, 12)])
+def test_dismiss_seconds_are_clamped(raw: object, expected: int) -> None:
+ assert geometry.clamp_dismiss_s(raw) == expected
+
+
+def test_pill_goes_below_the_region_when_there_is_room() -> None:
+ spot = geometry.pill_placement((0.25, 0.25, 0.5, 0.5), 1000, 1000, 120, 30)
+ assert not spot.overlaps
+ assert spot.y >= 750
+
+
+def test_pill_goes_above_when_the_region_reaches_the_bottom() -> None:
+ spot = geometry.pill_placement((0.25, 0.25, 0.5, 0.75), 1000, 1000, 120, 30)
+ assert not spot.overlaps
+ assert spot.y + 30 <= 250
+
+
+def test_full_screen_pill_overlaps_and_says_so() -> None:
+ assert geometry.pill_placement(None, 1000, 1000, 120, 30).overlaps
+ assert geometry.pill_placement((0.0, 0.0, 1.0, 1.0), 1000, 1000, 120, 30).overlaps
+
+
+def test_elapsed_format() -> None:
+ assert geometry.format_elapsed(12.4) == "00:12"
+ assert geometry.format_elapsed(3723) == "1:02:03"
diff --git a/tests/unit/jarvisx/test_hotkeys.py b/tests/unit/jarvisx/test_hotkeys.py
new file mode 100644
index 0000000000..1169ddbc70
--- /dev/null
+++ b/tests/unit/jarvisx/test_hotkeys.py
@@ -0,0 +1,146 @@
+"""Jarvis X shortcuts: planning, arming, re-arming, dispatch."""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+
+from jarvis.jarvisx.hotkeys import JarvisXShortcuts, check_hotkeys, normalize_hotkey
+
+
+def _config(**overrides) -> SimpleNamespace:
+ block = {
+ "enabled": True,
+ "hotkey_region": "ctrl+shift+2",
+ "hotkey_window": "ctrl+shift+3",
+ "hotkey_fullscreen": "ctrl+shift+1",
+ "hotkey_record_region": "ctrl+shift+5",
+ "hotkey_record_fullscreen": "ctrl+shift+6",
+ "hotkey_stop_recording": "ctrl+shift+4",
+ }
+ block.update(overrides)
+ return SimpleNamespace(jarvisx=SimpleNamespace(**block))
+
+
+class FakeTrigger:
+ """Stands in for HotkeyTrigger: records bindings, replays queued presses."""
+
+ instances: list[FakeTrigger] = []
+
+ def __init__(self, bindings: dict[str, list[str]]) -> None:
+ self.bindings = bindings
+ self.queue: asyncio.Queue[str] = asyncio.Queue()
+ self.entered = False
+ self.exited = False
+ FakeTrigger.instances.append(self)
+
+ async def __aenter__(self) -> FakeTrigger:
+ self.entered = True
+ return self
+
+ async def __aexit__(self, *exc) -> None:
+ self.exited = True
+
+ async def events(self):
+ while True:
+ yield await self.queue.get()
+
+
+def _shortcuts(config, presses, *, backend=True, owns=True) -> JarvisXShortcuts:
+ FakeTrigger.instances.clear()
+ return JarvisXShortcuts(
+ presses.append,
+ config_loader=lambda: config,
+ trigger_factory=FakeTrigger,
+ has_hotkey=lambda: backend,
+ owns_shortcuts=lambda: owns,
+ )
+
+
+def test_normalize_hotkey() -> None:
+ assert normalize_hotkey(" Ctrl + Shift+2 ") == "ctrl+shift+2"
+ assert normalize_hotkey(None) == ""
+
+
+def test_defaults_are_valid_and_distinct() -> None:
+ hotkeys = {
+ name: normalize_hotkey(value)
+ for name, value in {
+ "region": "ctrl+shift+2",
+ "window": "ctrl+shift+3",
+ "fullscreen": "ctrl+shift+1",
+ "record_region": "ctrl+shift+5",
+ "record_fullscreen": "ctrl+shift+6",
+ "stop_recording": "ctrl+shift+4",
+ }.items()
+ }
+ assert all(problem == "" for problem in check_hotkeys(hotkeys).values())
+
+
+def test_duplicate_and_both_alt_shortcuts_are_flagged() -> None:
+ problems = check_hotkeys(
+ {"region": "ctrl+shift+2", "window": "shift+ctrl+2", "fullscreen": "alt+alt"}
+ )
+ assert problems["region"] == ""
+ assert "same keys" in problems["window"]
+ assert "appshot" in problems["fullscreen"]
+
+
+async def test_all_valid_shortcuts_arm_in_one_trigger_and_dispatch() -> None:
+ presses: list[str] = []
+ shortcuts = _shortcuts(_config(hotkey_window=""), presses)
+ statuses = await shortcuts.start()
+ await asyncio.sleep(0)
+
+ assert statuses["region"].armed and statuses["region"].hotkey == "ctrl+shift+2"
+ assert not statuses["window"].armed and statuses["window"].detail == "No shortcut set."
+ trigger = FakeTrigger.instances[-1]
+ assert trigger.entered
+ assert set(trigger.bindings) == {
+ "jarvisx_region",
+ "jarvisx_fullscreen",
+ "jarvisx_record_region",
+ "jarvisx_record_fullscreen",
+ "jarvisx_stop_recording",
+ }
+ trigger.queue.put_nowait("jarvisx_record_region")
+ trigger.queue.put_nowait("someone_else")
+ for _ in range(5):
+ await asyncio.sleep(0)
+ assert presses == ["record_region"]
+ await shortcuts.stop()
+ assert trigger.exited
+
+
+async def test_reload_rearms_with_the_new_keys() -> None:
+ presses: list[str] = []
+ config = _config()
+ shortcuts = _shortcuts(config, presses)
+ await shortcuts.start()
+ await asyncio.sleep(0)
+ first = FakeTrigger.instances[-1]
+
+ config.jarvisx.hotkey_region = "ctrl+alt+r"
+ await shortcuts.reload()
+ await asyncio.sleep(0)
+ second = FakeTrigger.instances[-1]
+ assert first is not second and first.exited
+ assert second.bindings["jarvisx_region"] == ["ctrl+alt+r"]
+ await shortcuts.stop()
+
+
+async def test_switched_off_or_unsupported_hosts_arm_nothing() -> None:
+ off = _shortcuts(_config(enabled=False), [])
+ statuses = await off.start()
+ assert not any(s.armed for s in statuses.values())
+ assert statuses["region"].detail == "Jarvis X is switched off."
+ assert FakeTrigger.instances == []
+
+ headless = _shortcuts(_config(), [], backend=False)
+ statuses = await headless.start()
+ assert "not available" in statuses["region"].detail
+
+ second_instance = _shortcuts(_config(), [], owns=False)
+ statuses = await second_instance.start()
+ assert "main app" in statuses["region"].detail
+ assert FakeTrigger.instances == []
diff --git a/tests/unit/jarvisx/test_service_and_routes.py b/tests/unit/jarvisx/test_service_and_routes.py
new file mode 100644
index 0000000000..06c76c67e6
--- /dev/null
+++ b/tests/unit/jarvisx/test_service_and_routes.py
@@ -0,0 +1,370 @@
+"""Jarvis X service flows and the REST contract, with fakes for every OS seam."""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import io
+import tomllib
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from PIL import Image
+
+from jarvis.core.config_writer import set_jarvisx_settings
+from jarvis.jarvisx import capture
+from jarvis.jarvisx import service as service_module
+from jarvis.jarvisx import store as store_module
+from jarvis.jarvisx.overlay.controller import Selection
+from jarvis.jarvisx.service import JarvisXService
+
+_MONITORS = [
+ {"left": 0, "top": 0, "width": 200, "height": 100},
+ {"left": 0, "top": 0, "width": 200, "height": 100},
+]
+
+
+class FakeBus:
+ def __init__(self) -> None:
+ self.events: list = []
+
+ async def publish(self, event) -> None:
+ self.events.append(event)
+
+ def names(self) -> list[str]:
+ return [type(e).__name__ for e in self.events]
+
+
+class FakeOverlay:
+ def __init__(self, selection: Selection | None = None) -> None:
+ self.selection = selection
+ self.cards: list[dict] = []
+ self.hidden_grabs = 0
+ self.recording_shown: dict | None = None
+ self.removed: list[str] = []
+
+ def bind(self, loop, handler) -> None:
+ self.handler = handler
+
+ async def select_region(self, *, purpose: str):
+ return self.selection
+
+ @contextlib.contextmanager
+ def cards_hidden(self):
+ self.hidden_grabs += 1
+ yield
+
+ async def show_card(self, **kwargs) -> bool:
+ self.cards.append(kwargs)
+ return True
+
+ async def remove_card(self, item_id: str) -> None:
+ self.removed.append(item_id)
+
+ async def show_recording(self, *, monitor, rect) -> bool:
+ self.recording_shown = {"monitor": monitor, "rect": rect}
+ return True
+
+ async def hide_recording(self) -> None:
+ self.recording_shown = None
+
+ async def shutdown(self) -> None:
+ return None
+
+
+def _config(tmp_path: Path, **overrides) -> SimpleNamespace:
+ block = {
+ "enabled": True,
+ "save_dir": str(tmp_path / "shots"),
+ "thumbnail_persist": True,
+ "thumbnail_dismiss_s": 12,
+ "copy_to_clipboard": False,
+ "sound": False,
+ "effect": True,
+ }
+ block.update(overrides)
+ return SimpleNamespace(jarvisx=SimpleNamespace(**block))
+
+
+def fake_screen(monkeypatch):
+ grabs: list[tuple] = []
+
+ def grab_rect(bbox):
+ grabs.append(tuple(bbox))
+ _l, _t, w, h = bbox
+ return capture.Frame(w, h, bytes([200, 10, 10]) * (w * h), tuple(bbox))
+
+ monkeypatch.setattr(capture, "capability", lambda: (True, ""))
+ monkeypatch.setattr(capture, "permission_problem", lambda: "")
+ monkeypatch.setattr(capture, "list_monitors", lambda: _MONITORS)
+ monkeypatch.setattr(capture, "monitor_under_cursor", lambda monitors: monitors[1])
+ monkeypatch.setattr(capture, "grab_rect", grab_rect)
+ return grabs
+
+
+def _service(tmp_path: Path, overlay: FakeOverlay, **config) -> tuple[JarvisXService, FakeBus]:
+ bus = FakeBus()
+ svc = JarvisXService(
+ store=store_module.ItemStore(tmp_path / "lib"),
+ overlay=overlay,
+ config_loader=lambda: _config(tmp_path, **config),
+ )
+ svc.bind(bus=bus)
+ return svc, bus
+
+
+def _saved_png_size(path: str, folder: Path) -> tuple[int, int]:
+ saved = Path(path)
+ assert saved.parent == folder and saved.suffix == ".png"
+ with Image.open(saved) as image:
+ return image.size
+
+
+def _is_file(path: str) -> bool:
+ return Path(path).is_file()
+
+
+async def test_region_capture_saves_indexes_announces_and_shows_a_card(
+ tmp_path, fake_screen
+) -> None:
+ selection = Selection(
+ screen={"x": 0, "y": 0, "w": 200, "h": 100, "dpr": 1.0}, rect=(0.25, 0.5, 0.5, 0.25)
+ )
+ overlay = FakeOverlay(selection)
+ svc, bus = _service(tmp_path, overlay)
+
+ result = await svc.capture("region")
+ await asyncio.gather(*list(service_module._TASKS))
+
+ assert result.ok, result.message
+ assert fake_screen == [(50, 50, 100, 25)]
+ assert overlay.hidden_grabs == 1, "resting cards are hidden during the grab"
+ item = result.item
+ assert item is not None and item.kind == "image" and item.mode == "region"
+ assert (item.width, item.height) == (100, 25)
+ assert _saved_png_size(item.path, tmp_path / "shots") == (100, 25)
+ assert _is_file(item.thumb_path)
+ assert bus.names() == ["JarvisXItemCreated"]
+ assert bus.events[0].id == item.id and bus.events[0].mode == "region"
+ assert overlay.cards[0]["item_id"] == item.id
+ assert overlay.cards[0]["persist"] is True and overlay.cards[0]["dismiss_s"] == 12
+ assert overlay.cards[0]["rect"] == [0.25, 0.5, 0.5, 0.25]
+
+
+async def test_cancelled_selection_saves_nothing(tmp_path, fake_screen) -> None:
+ svc, bus = _service(tmp_path, FakeOverlay(None))
+ result = await svc.capture("region")
+ assert not result.ok and result.message == "Cancelled."
+ assert fake_screen == [] and bus.events == []
+
+
+async def test_switched_off_refuses(tmp_path, fake_screen) -> None:
+ svc, _bus = _service(tmp_path, FakeOverlay(), enabled=False)
+ result = await svc.capture("fullscreen")
+ assert not result.ok and "switched off" in result.message
+
+
+async def test_effect_off_means_no_card(tmp_path, fake_screen) -> None:
+ overlay = FakeOverlay()
+ svc, _bus = _service(tmp_path, overlay, effect=False)
+ result = await svc.capture("fullscreen")
+ assert result.ok and fake_screen == [(0, 0, 200, 100)]
+ assert overlay.cards == []
+
+
+async def test_recording_lifecycle_with_a_fake_recorder(tmp_path, fake_screen, monkeypatch) -> None:
+ from jarvis.jarvisx import recorder as rec
+
+ monkeypatch.setattr(
+ rec,
+ "probe_encoder",
+ lambda: rec.EncoderChoice(available=True, codec="fake", extension="mp4"),
+ )
+
+ class FakeRecorder:
+ def __init__(self, path: Path) -> None:
+ self.path = path
+ self.elapsed_s = 1.5
+
+ def start(self) -> str:
+ return ""
+
+ def stop(self):
+ self.path.write_bytes(b"video")
+ return rec.RecordingResult(
+ path=self.path,
+ width=200,
+ height=100,
+ duration_s=3.2,
+ frames=96,
+ first_frame=(2, 1, bytes(8)),
+ )
+
+ overlay = FakeOverlay()
+ svc, bus = _service(tmp_path, overlay)
+ monkeypatch.setattr(svc, "_make_recorder", lambda choice, bbox, target: FakeRecorder(target))
+
+ started = await svc.start_recording("fullscreen")
+ assert started.ok, started.message
+ assert svc.recording_status() == {"recording": True, "mode": "fullscreen", "elapsed_s": 1.5}
+ assert overlay.recording_shown == {"monitor": [0, 0, 200, 100], "rect": None}
+ assert (await svc.start_recording("region")).message == "A recording is already running."
+
+ stopped = await svc.stop_recording()
+ assert stopped.ok and stopped.item is not None
+ assert stopped.item.kind == "video" and stopped.item.duration_s == 3.2
+ assert overlay.recording_shown is None
+ assert svc.recording_status()["recording"] is False
+ assert bus.names() == [
+ "JarvisXRecordingChanged",
+ "JarvisXRecordingChanged",
+ "JarvisXItemCreated",
+ ]
+ assert bus.events[1].recording is False and bus.events[1].elapsed_s == 3.2
+ assert (await svc.stop_recording()).message == "No recording is running."
+
+
+# ------------------------------------------------------------------ config
+
+
+def test_settings_writer_round_trip_and_allowlist(tmp_path) -> None:
+ path = tmp_path / "jarvis.toml"
+ path.write_text("[ui]\nsound_effects = true\n", encoding="utf-8")
+ set_jarvisx_settings({"hotkey_region": "ctrl+alt+r", "thumbnail_dismiss_s": 9}, path=path)
+ data = tomllib.loads(path.read_text(encoding="utf-8"))
+ assert data["jarvisx"] == {"hotkey_region": "ctrl+alt+r", "thumbnail_dismiss_s": 9}
+ with pytest.raises(ValueError, match="unknown jarvisx"):
+ set_jarvisx_settings({"bogus": 1}, path=path)
+
+
+# ------------------------------------------------------------------- REST
+
+
+def client(tmp_path, monkeypatch, fake_screen):
+ from jarvis.jarvisx import recorder as rec
+ from jarvis.ui.web.jarvisx_routes import router
+
+ config_path = tmp_path / "jarvis.toml"
+ config_path.write_text(
+ f"[jarvisx]\nsave_dir = '{(tmp_path / 'shots').as_posix()}'\n"
+ "copy_to_clipboard = false\nsound = false\neffect = false\n",
+ encoding="utf-8",
+ )
+ monkeypatch.setenv("JARVIS_CONFIG", str(config_path))
+ monkeypatch.setattr(
+ rec, "probe_encoder", lambda: rec.EncoderChoice(False, detail="No encoder here.")
+ )
+ store = store_module.ItemStore(tmp_path / "lib")
+ store_module.set_store_for_tests(store)
+ svc = JarvisXService(store=store, overlay=FakeOverlay())
+ service_module.set_service_for_tests(svc)
+ app = FastAPI()
+ app.include_router(router)
+ app.state.bus = FakeBus()
+ app.state.native_file_actions = False
+ try:
+ yield TestClient(app)
+ finally:
+ store_module.set_store_for_tests(None)
+ service_module.set_service_for_tests(None)
+
+
+def test_settings_contract_and_partial_update(client) -> None:
+ body = client.get("/api/jarvisx/settings").json()
+ for key in (
+ "enabled",
+ "hotkeys",
+ "thumbnail_persist",
+ "thumbnail_dismiss_s",
+ "save_dir",
+ "copy_to_clipboard",
+ "sound",
+ "effect",
+ "recording_available",
+ "recording_detail",
+ "shortcuts",
+ ):
+ assert key in body
+ assert body["hotkeys"]["fullscreen"] == "ctrl+shift+1"
+ assert body["recording_available"] is False and body["recording_detail"] == "No encoder here."
+ assert set(body["shortcuts"]) == set(body["hotkeys"])
+
+ updated = client.put(
+ "/api/jarvisx/settings",
+ json={"hotkeys": {"window": "Ctrl+Alt+W"}, "thumbnail_persist": True},
+ ).json()
+ assert updated["hotkeys"]["window"] == "ctrl+alt+w"
+ assert updated["hotkeys"]["region"] == "ctrl+shift+2"
+ assert updated["thumbnail_persist"] is True
+
+
+def test_invalid_or_colliding_hotkeys_are_422(client) -> None:
+ clash = client.put("/api/jarvisx/settings", json={"hotkeys": {"window": "ctrl+shift+2"}})
+ assert clash.status_code == 422 and "same keys" in clash.json()["detail"]
+ assert client.put("/api/jarvisx/settings", json={}).status_code == 400
+ relative = client.put("/api/jarvisx/settings", json={"save_dir": "shots"})
+ assert relative.status_code == 422
+
+
+def test_capture_library_edit_and_delete(client, tmp_path) -> None:
+ taken = client.post("/api/jarvisx/capture", json={"mode": "fullscreen"}).json()
+ assert taken["ok"], taken
+ item = taken["item"]
+ item_id = item["id"]
+ assert item["url"] == f"/api/jarvisx/items/{item_id}/file"
+
+ listing = client.get("/api/jarvisx/items?limit=10").json()["items"]
+ assert [i["id"] for i in listing] == [item_id]
+ assert client.get(f"/api/jarvisx/items/{item_id}").json()["width"] == 200
+
+ original = client.get(f"/api/jarvisx/items/{item_id}/file")
+ assert original.status_code == 200 and original.headers["content-type"] == "image/png"
+ thumb = client.get(f"/api/jarvisx/items/{item_id}/thumb")
+ assert thumb.status_code == 200 and thumb.headers["content-type"] == "image/jpeg"
+ assert client.get(f"/api/jarvisx/items/{item_id}/edited").status_code == 404
+
+ buffer = io.BytesIO()
+ Image.new("RGB", (4, 4), "red").save(buffer, format="PNG")
+ saved = client.put(
+ f"/api/jarvisx/items/{item_id}/edited",
+ content=buffer.getvalue(),
+ headers={"Content-Type": "image/png"},
+ )
+ assert saved.status_code == 200
+ assert saved.json()["edited_url"] == f"/api/jarvisx/items/{item_id}/edited"
+ assert (
+ Path(listing[0]["path"]).with_name(Path(listing[0]["path"]).stem + "-edited.png").is_file()
+ )
+ assert client.get(f"/api/jarvisx/items/{item_id}/edited").content == buffer.getvalue()
+ bad = client.put(
+ f"/api/jarvisx/items/{item_id}/edited",
+ content=b"not a png",
+ headers={"Content-Type": "image/png"},
+ )
+ assert bad.status_code == 422
+
+ headless = client.post(f"/api/jarvisx/items/{item_id}/copy", json={"edited": True}).json()
+ assert headless["ok"] is False
+ editor = client.post(f"/api/jarvisx/items/{item_id}/open-editor").json()
+ assert editor["ok"] is False and editor["url"].endswith(f"item={item_id}")
+
+ assert client.delete(f"/api/jarvisx/items/{item_id}").json()["ok"] is True
+ assert client.get(f"/api/jarvisx/items/{item_id}").status_code == 404
+ assert client.get("/api/jarvisx/items/not-an-id").status_code == 422
+
+
+def test_record_status_and_refusal_without_encoder(client) -> None:
+ assert client.get("/api/jarvisx/record/status").json() == {
+ "recording": False,
+ "mode": None,
+ "elapsed_s": 0.0,
+ }
+ started = client.post("/api/jarvisx/record/start", json={"mode": "fullscreen"}).json()
+ assert started["ok"] is False and started["message"] == "No encoder here."
+ stopped = client.post("/api/jarvisx/record/stop").json()
+ assert stopped["ok"] is False and stopped["item"] is None
diff --git a/tests/unit/jarvisx/test_store_and_recorder.py b/tests/unit/jarvisx/test_store_and_recorder.py
new file mode 100644
index 0000000000..e4bf902c3d
--- /dev/null
+++ b/tests/unit/jarvisx/test_store_and_recorder.py
@@ -0,0 +1,183 @@
+"""Jarvis X library index, save paths and the recording loop (fake encoder)."""
+
+from __future__ import annotations
+
+import time
+from dataclasses import replace
+from datetime import datetime
+from pathlib import Path
+
+from jarvis.jarvisx import paths
+from jarvis.jarvisx.recorder import Recorder
+from jarvis.jarvisx.store import Item, ItemStore
+
+
+def _item(tmp_path: Path, item_id: str, *, kind: str = "image") -> Item:
+ original = tmp_path / f"{item_id}.png"
+ original.write_bytes(b"png")
+ return Item(
+ id=item_id,
+ kind=kind, # type: ignore[arg-type]
+ mode="region",
+ created_at="2026-09-29T14:03:22+02:00",
+ width=10,
+ height=20,
+ duration_s=None if kind == "image" else 3.5,
+ path=str(original),
+ thumb_path="",
+ )
+
+
+def test_store_lists_newest_first_and_serializes_the_contract(tmp_path) -> None:
+ store = ItemStore(tmp_path / "lib")
+ store.add(_item(tmp_path, "a" * 32))
+ store.add(_item(tmp_path, "b" * 32, kind="video"))
+
+ items = store.recent()
+ assert [i.id for i in items] == ["b" * 32, "a" * 32]
+ payload = items[0].to_json()
+ assert payload["kind"] == "video"
+ assert payload["duration_s"] == 3.5
+ assert payload["url"] == f"/api/jarvisx/items/{'b' * 32}/file"
+ assert payload["thumb_url"] == f"/api/jarvisx/items/{'b' * 32}/thumb"
+ assert payload["edited_url"] is None
+ assert payload["filename"] == f"{'b' * 32}.png"
+
+
+def test_edited_url_appears_only_when_the_file_exists(tmp_path) -> None:
+ store = ItemStore(tmp_path / "lib")
+ item = store.add(_item(tmp_path, "c" * 32))
+ edited = tmp_path / "c-edited.png"
+ updated = store.set_edited(item.id, str(edited))
+ assert updated is not None and updated.to_json()["edited_url"] is None
+ edited.write_bytes(b"png")
+ assert store.get(item.id).to_json()["edited_url"] == f"/api/jarvisx/items/{'c' * 32}/edited"
+
+
+def test_a_file_removed_outside_the_app_drops_out(tmp_path) -> None:
+ store = ItemStore(tmp_path / "lib")
+ item = store.add(_item(tmp_path, "d" * 32))
+ Path(item.path).unlink()
+ assert store.recent() == []
+ assert store.get(item.id) is None
+
+
+def test_delete_removes_every_file(tmp_path) -> None:
+ store = ItemStore(tmp_path / "lib")
+ item = _item(tmp_path, "e" * 32)
+ thumb = store.thumb_path_for(item.id)
+ thumb.parent.mkdir(parents=True, exist_ok=True)
+ thumb.write_bytes(b"jpg")
+ store.add(replace(item, thumb_path=str(thumb)))
+ removed = store.delete(item.id)
+ assert removed is not None
+ assert not Path(item.path).exists() and not thumb.exists()
+ assert store.delete(item.id) is None
+
+
+def test_capture_names_and_unique_paths(tmp_path) -> None:
+ when = datetime(2026, 9, 29, 14, 3, 22)
+ name = paths.capture_filename("image", when, "png")
+ assert name == "Jarvis X 2026-09-29 at 14.03.22.png"
+ assert paths.capture_filename("video", when, "mp4").startswith("Recording ")
+ (tmp_path / name).write_bytes(b"x")
+ assert paths.unique_path(tmp_path, name).name == "Jarvis X 2026-09-29 at 14.03.22 (2).png"
+ edited = paths.edited_path_for(tmp_path / name)
+ assert edited.name == "Jarvis X 2026-09-29 at 14.03.22-edited.png"
+
+
+def test_unusable_save_dir_falls_back(tmp_path, monkeypatch) -> None:
+ blocker = tmp_path / "file"
+ blocker.write_text("not a folder", encoding="utf-8")
+ monkeypatch.setattr(paths, "default_save_dir", lambda: tmp_path / "default")
+ assert paths.resolve_save_dir(str(blocker / "sub")) == tmp_path / "default"
+ assert paths.resolve_save_dir("") == tmp_path / "default"
+
+
+# ---------------------------------------------------------------- recorder
+
+
+class FakeSource:
+ def __init__(self) -> None:
+ self.closed = False
+ self.grabs = 0
+
+ def grab(self) -> tuple[int, int, bytes]:
+ self.grabs += 1
+ return 5, 3, bytes(5 * 3 * 4)
+
+ def close(self) -> None:
+ self.closed = True
+
+
+class FakeEncoder:
+ name = "fake"
+ extension = "mp4"
+
+ def __init__(self) -> None:
+ self.opened: tuple | None = None
+ self.pts: list[int] = []
+ self.closed = False
+
+ def open(self, path: Path, width: int, height: int, fps: int) -> None:
+ self.opened = (path, width, height, fps)
+
+ def write(self, bgra: bytes, width: int, height: int, pts: int) -> None:
+ self.pts.append(pts)
+
+ def close(self) -> None:
+ self.closed = True
+
+
+def _wait_for(predicate, timeout: float = 5.0) -> None:
+ deadline = time.monotonic() + timeout
+ while not predicate() and time.monotonic() < deadline:
+ time.sleep(0.01)
+
+
+def test_recorder_lifecycle_with_a_fake_encoder(tmp_path) -> None:
+ source = FakeSource()
+ encoder = FakeEncoder()
+ recorder = Recorder(
+ source_factory=lambda: source, encoder=encoder, path=tmp_path / "v.mp4", fps=50
+ )
+ assert recorder.start() == ""
+ assert recorder.running
+ _wait_for(lambda: len(encoder.pts) >= 5)
+ result = recorder.stop()
+
+ assert not recorder.running
+ assert encoder.opened == (tmp_path / "v.mp4", 5, 3, 50)
+ assert encoder.closed and source.closed
+ assert result.frames == len(encoder.pts) >= 5
+ assert encoder.pts == sorted(set(encoder.pts)), "timestamps strictly increase"
+ assert (result.width, result.height) == (4, 2), "H.264 needs even dimensions"
+ assert result.first_frame is not None and result.first_frame[:2] == (5, 3)
+ assert result.ok and result.duration_s > 0
+
+
+def test_recorder_reports_a_source_that_cannot_start(tmp_path) -> None:
+ def broken():
+ raise RuntimeError("no display")
+
+ encoder = FakeEncoder()
+ recorder = Recorder(source_factory=broken, encoder=encoder, path=tmp_path / "v.mp4")
+ error = recorder.start()
+ assert "no display" in error
+ result = recorder.stop()
+ assert not result.ok and result.frames == 0
+ assert encoder.opened is None
+
+
+def test_an_encoder_failure_ends_the_recording_with_a_reason(tmp_path) -> None:
+ class Exploding(FakeEncoder):
+ def write(self, bgra, width, height, pts) -> None:
+ raise OSError("disk full")
+
+ encoder = Exploding()
+ recorder = Recorder(source_factory=FakeSource, encoder=encoder, path=tmp_path / "v.mp4", fps=50)
+ recorder.start()
+ _wait_for(lambda: not recorder.running, timeout=3.0)
+ result = recorder.stop()
+ assert "disk full" in result.error
+ assert encoder.closed
diff --git a/tests/unit/local_models/test_autostart.py b/tests/unit/local_models/test_autostart.py
index 4b144c2dca..865fd390a2 100644
--- a/tests/unit/local_models/test_autostart.py
+++ b/tests/unit/local_models/test_autostart.py
@@ -99,7 +99,7 @@ def test_the_default_is_on_even_without_an_ollama_card() -> None:
@pytest.mark.asyncio
async def test_run_once_starts_a_stopped_server_and_warms_the_chat_pick() -> None:
- cfg = _cfg(chat="qwen3.5:4b")
+ cfg = _cfg(primary="ollama", chat="qwen3.5:4b")
cfg.brain.providers["ollama"].models["qwen3.5:4b"] = OllamaModelOptions(keep_alive="2h")
started: list[str] = []
warmed: list[tuple[str, str, Any]] = []
@@ -129,7 +129,7 @@ async def test_run_once_only_warms_a_running_server() -> None:
return True
record = await autostart.run_once(
- _cfg(chat="qwen3.5:4b"),
+ _cfg(primary="ollama", chat="qwen3.5:4b"),
status=lambda: {"installed": True, "running": True},
start=lambda: (_ for _ in ()).throw(AssertionError("must not start")),
warm=_warm,
@@ -138,6 +138,23 @@ async def test_run_once_only_warms_a_running_server() -> None:
assert warmed == [f"qwen3.5:4b:{autostart.DEFAULT_KEEP_ALIVE}"]
+async def test_local_voice_does_not_prewarm_an_unselected_chat_model() -> None:
+ warmed: list[str] = []
+
+ async def warm(_root: str, model: str, _keep_alive: Any) -> bool:
+ warmed.append(model)
+ return True
+
+ result = await autostart.run_once(
+ _cfg(realtime="local-realtime", chat="unused-large-chat-model"),
+ status=lambda: {"installed": True, "running": True},
+ warm=warm,
+ )
+ assert result["warmed"] == ""
+ assert warmed == []
+
+
@pytest.mark.asyncio
async def test_run_once_never_installs_and_never_starts_a_remote_server() -> None:
absent = await autostart.run_once(
diff --git a/tests/unit/realtime/local_server/test_engine.py b/tests/unit/realtime/local_server/test_engine.py
index 5e7d68d5c5..796775e620 100644
--- a/tests/unit/realtime/local_server/test_engine.py
+++ b/tests/unit/realtime/local_server/test_engine.py
@@ -112,6 +112,9 @@ class TestBrainResolution:
"_ollama_models",
lambda base, timeout: [
("qwen3.5:4b", 3.4),
+ ("qwen3.5:4b-voice-32k", 3.4),
+ ("qwen3.5:4b-voice-64k", 3.4),
+ ("qwen3.5-4b-jarvis-ab12cd34:latest", 3.4),
("nemotron-cascade-2:latest", 24.0),
("nomic-embed-text:latest", 0.3), # never a voice brain
],
@@ -122,6 +125,7 @@ class TestBrainResolution:
assert payload["reachable"] is True
by_id = {entry["id"]: entry for entry in payload["models"]}
assert "nomic-embed-text:latest" not in by_id
+ assert not any("-voice-" in name or "-jarvis-" in name for name in by_id)
assert by_id["qwen3.5:4b"]["installed"] is True
assert by_id["qwen3.5:4b"]["fits"] is True
assert by_id["qwen3.5:4b"]["current"] is True
diff --git a/tests/unit/realtime/local_server/test_supervisor.py b/tests/unit/realtime/local_server/test_supervisor.py
index 3185469d60..a3a91d78d7 100644
--- a/tests/unit/realtime/local_server/test_supervisor.py
+++ b/tests/unit/realtime/local_server/test_supervisor.py
@@ -42,6 +42,13 @@ def _local_models_switch_on(monkeypatch: pytest.MonkeyPatch) -> None:
provider.autostart = True
cfg.brain.providers["ollama"] = provider
monkeypatch.setattr("jarvis.core.config.load_config", lambda: cfg)
+ # Lifecycle fixtures must not depend on or query the developer's GPU and
+ # live model server. Individual probe tests supply their own fake client.
+ monkeypatch.setattr(
+ "jarvis.hardware.detection.free_accelerator_gb",
+ lambda: (16.0, "nvidia-smi"),
+ )
+ monkeypatch.setattr(supervisor, "_probe_client", lambda: None)
def _spawn_ready(monkeypatch, tmp_path: Path) -> list[dict[str, Any]]:
@@ -160,16 +167,7 @@ def test_pool_url_normalizes_every_supported_input_shape() -> None:
def test_runtime_probe_requires_a_valid_loaded_pool(monkeypatch) -> None:
- import http.client
-
- class _Response:
- status = 200
-
- def __init__(self, payload: object) -> None:
- self._payload = json.dumps(payload).encode("utf-8")
-
- def read(self, limit: int) -> bytes:
- return self._payload[:limit]
+ import httpx
payload = {
"size": 4,
@@ -182,34 +180,24 @@ def test_runtime_probe_requires_a_valid_loaded_pool(monkeypatch) -> None:
],
}
- class _Connection:
- def __init__(self, *args: Any, **kwargs: Any) -> None:
- pass
-
- def request(self, *args: Any, **kwargs: Any) -> None:
- pass
-
- def getresponse(self) -> _Response:
- return _Response(payload)
-
- def close(self) -> None:
- pass
-
- monkeypatch.setattr(http.client, "HTTPConnection", _Connection)
- assert supervisor.probe_runtime("http://localhost:8765") == {
- "size": 4,
- "in_use": 3,
- "available": 1,
- "active": 1,
- "draining": 1,
- "stuck": 1,
- }
+ with httpx.Client(
+ transport=httpx.MockTransport(lambda request: httpx.Response(200, json=payload))
+ ) as client:
+ monkeypatch.setattr(supervisor, "_probe_client", lambda: client)
+ assert supervisor.probe_runtime("http://localhost:8765") == {
+ "size": 4,
+ "in_use": 3,
+ "available": 1,
+ "active": 1,
+ "draining": 1,
+ "stuck": 1,
+ }
- payload["in_use"] = 2
- assert supervisor.probe_runtime("http://localhost:8765") is None
- payload["in_use"] = 3
- payload["units"] = []
- assert supervisor.probe_runtime("http://localhost:8765") is None
+ payload["in_use"] = 2
+ assert supervisor.probe_runtime("http://localhost:8765") is None
+ payload["in_use"] = 3
+ payload["units"] = []
+ assert supervisor.probe_runtime("http://localhost:8765") is None
def test_only_an_abandoned_full_pool_has_no_usable_capacity() -> None:
@@ -2104,12 +2092,15 @@ def _write_booting_pidfile(tmp_path: Path, *, spawned_ago_s: float) -> None:
)
-def test_status_reports_stage_and_eta_while_an_owned_child_boots(monkeypatch, tmp_path) -> None:
[email protected]("spawned_ago_s", [20.0, 90.0])
+def test_status_reports_stage_and_eta_while_an_owned_child_boots(
+ monkeypatch, tmp_path, spawned_ago_s
+) -> None:
monkeypatch.setenv("JARVIS_DATA_DIR", str(tmp_path))
monkeypatch.setattr(supervisor, "_port_open", lambda port, timeout=1.0: False)
monkeypatch.setattr(supervisor, "probe_runtime", lambda *args, **kwargs: None)
monkeypatch.setattr(supervisor, "_process_create_time", lambda pid: 1000.0)
- _write_booting_pidfile(tmp_path, spawned_ago_s=20.0)
+ _write_booting_pidfile(tmp_path, spawned_ago_s=spawned_ago_s)
stamp = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
(tmp_path / "local_realtime_server.log").write_text(
f"{stamp},100 - speech_to_speech.TTS.qwen3_tts_handler - INFO - "
@@ -2131,7 +2122,10 @@ def test_status_reports_stage_and_eta_while_an_owned_child_boots(monkeypatch, tm
assert boot["stage"] == "voice-model"
assert boot["stage_label"] == "loading the speaking voice"
assert boot["expected_total_s"] == 80.0
- assert 50.0 <= boot["remaining_s"] <= 65.0
+ if spawned_ago_s < 80:
+ assert 50.0 <= boot["remaining_s"] <= 65.0
+ else:
+ assert boot["remaining_s"] is None
assert boot["failed_streak"] == 0
diff --git a/tests/unit/realtime/test_openai_realtime.py b/tests/unit/realtime/test_openai_realtime.py
index c831e23af4..825ec248a7 100644
--- a/tests/unit/realtime/test_openai_realtime.py
+++ b/tests/unit/realtime/test_openai_realtime.py
@@ -1574,6 +1574,47 @@ async def test_slow_first_output_uses_the_declared_response_start_budget(
await session.close()
+async def test_local_text_does_not_expire_the_first_audio_budget(monkeypatch) -> None:
+ """A transcript may precede local TTS audio by more than eight seconds."""
+ client = _FakeAsyncOpenAI(api_key="local")
+ session = await openai_realtime._open_realtime_session(
+ client, RealtimeSessionConfig(model="local"), model="local",
+ response_start_timeout_s=90.0,
+ )
+ try:
+ await session.request_response()
+ marker = session._conn.response_create_payloads[0]["response"]["metadata"][
+ "jarvis_request_id"
+ ]
+ created = SimpleNamespace(
+ type="response.created",
+ response=SimpleNamespace(id="local-answer", metadata={"jarvis_request_id": marker}),
+ )
+ _ = [event async for event in session._dispatch_event(created)]
+ transcript = SimpleNamespace(
+ type="response.output_audio_transcript.delta", response_id="local-answer", delta="Five."
+ )
+ _ = [event async for event in session._dispatch_event(transcript)]
+ text_at = session._last_response_activity
+ monkeypatch.setattr(openai_realtime.time, "monotonic", lambda: text_at + 20)
+ assert not session._response_lifecycle_stalled()
+ monkeypatch.setattr(openai_realtime.time, "monotonic", lambda: text_at + 91)
+ assert session._response_lifecycle_stalled(), "Missing audio still has a finite deadline"
+ audio = SimpleNamespace(
+ type="response.output_audio.delta", response_id="local-answer", delta="AAA="
+ )
+ _ = [event async for event in session._dispatch_event(audio)]
+ audio_at = session._last_response_activity
+ monkeypatch.setattr(openai_realtime.time, "monotonic", lambda: audio_at + 9)
+ assert session._response_lifecycle_stalled(), "Started audio retains the strict guard"
+ session._response_idle.set()
+ await session.request_response()
+ assert not session._response_audio_started, "Every response gets a fresh audio clock"
+ finally:
+ await session.close()
+
+
@pytest.mark.asyncio
async def test_serial_rebuild_releases_single_slot_before_retrying() -> None:
"""The managed server owns one pipeline slot. A rebuild must disconnect
diff --git a/tests/unit/society/test_conversation_responsiveness.py b/tests/unit/society/test_conversation_responsiveness.py
new file mode 100644
index 0000000000..f7a90ea1fd
--- /dev/null
+++ b/tests/unit/society/test_conversation_responsiveness.py
@@ -0,0 +1,100 @@
+"""Busy conversation storage must not freeze the desktop's serving loop."""
+from __future__ import annotations
+
+import asyncio
+import sqlite3
+import threading
+
+import pytest
+
+from jarvis.society.conversation import ConversationArchive
+from jarvis.society.runtime import SocietyRuntime
+
+
+def test_deferred_archive_does_not_touch_storage_until_used(tmp_path):
+ path = tmp_path / "unused" / "archive.db"
+ archive = ConversationArchive(path, defer_open=True)
+ assert not path.parent.exists()
+ archive.open()
+ try:
+ archive.ingest(
+ "agent-a", [{"seq": 1, "kind": "user_message", "payload": {"text": "hello"}}]
+ )
+ assert archive.search("agent-a", "hello")[0]["seq"] == 1
+ assert archive.search("agent-b", "hello") == []
+ finally:
+ archive.close()
+
+
+def test_closed_deferred_archive_cannot_create_storage(tmp_path):
+ path = tmp_path / "unused" / "archive.db"
+ archive = ConversationArchive(path, defer_open=True)
+ archive.close()
+ archive.close()
+ with pytest.raises(sqlite3.ProgrammingError, match="closed"):
+ archive.open()
+ assert not path.parent.exists()
+
+
+def test_failed_initialization_releases_connection_and_can_retry(tmp_path, monkeypatch):
+ archive = ConversationArchive(tmp_path / "archive.db", defer_open=True)
+ initialize = archive._initialize_schema
+ attempts = 0
+
+ def initialize_once():
+ nonlocal attempts
+ attempts += 1
+ if attempts == 1:
+ raise sqlite3.OperationalError("test storage unavailable")
+ initialize()
+
+ monkeypatch.setattr(archive, "_initialize_schema", initialize_once)
+ with pytest.raises(sqlite3.OperationalError):
+ archive.open()
+ assert archive._connection is None
+ archive.open()
+ try:
+ assert archive.read("agent-a") == []
+ finally:
+ archive.close()
+
+
+async def test_slow_archive_start_and_cancelled_cleanup_keep_loop_responsive(tmp_path, monkeypatch):
+ loop_thread = threading.get_ident()
+ entered = threading.Event()
+ release = threading.Event()
+ initialize = ConversationArchive._initialize_schema
+
+ def slow_schema(self):
+ assert threading.get_ident() != loop_thread
+ entered.set()
+ assert release.wait(10), "test must release the database initializer"
+ initialize(self)
+
+ monkeypatch.setattr(ConversationArchive, "_initialize_schema", slow_schema)
+ runtime = SocietyRuntime(tmp_path)
+ assert not (tmp_path / "society-conversations.db").exists()
+ startup = asyncio.create_task(runtime.ensure_started())
+ closing = None
+ try:
+ assert await asyncio.to_thread(entered.wait, 5)
+ heartbeat = asyncio.Event()
+ asyncio.get_running_loop().call_soon(heartbeat.set)
+ await asyncio.wait_for(heartbeat.wait(), 1)
+ startup.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await startup
+ closing = asyncio.create_task(runtime.close())
+ heartbeat.clear()
+ asyncio.get_running_loop().call_soon(heartbeat.set)
+ await asyncio.wait_for(heartbeat.wait(), 1)
+ finally:
+ release.set()
+ await asyncio.gather(startup, return_exceptions=True)
+ if closing is not None:
+ await closing
+ else:
+ await runtime.close()
+ assert runtime.conversations._connection is None
+ with pytest.raises(sqlite3.ProgrammingError, match="closed"):
+ runtime.conversations.open()
diff --git a/tests/unit/society/test_delegation_followthrough.py b/tests/unit/society/test_delegation_followthrough.py
new file mode 100644
index 0000000000..173deeb2d3
--- /dev/null
+++ b/tests/unit/society/test_delegation_followthrough.py
@@ -0,0 +1,62 @@
+"""Results follow trusted turn provenance, even after another chat opens."""
+
+from jarvis.core.delegation import origin_metadata
+from jarvis.core.protocols import ChatTurn, current_chat_turn
+from jarvis.society.events import MsgType, SocietyEnvelope
+from tests.unit.society import test_lead_card
+
+rt = test_lead_card.rt
+
+
+def session(runtime):
+ return runtime.chat.store.create_session(
+ provider="openai", model="test", effort="", cwd=str(runtime.data_dir), surface="jarvis",
+ )
+
+
+async def test_result_returns_to_origin_after_user_opens_another_chat(rt):
+ original = session(rt)
+ newer = session(rt)
+ target, _ = await rt.roster.create(name="Scout")
+ request = SocietyEnvelope(
+ msg_type=MsgType.ASSIGN, from_agent="jarvis", to_agent=target.agent_id,
+ trace_id="voice:legacy-prefix",
+ payload={"text": "Find the release", "reply_surface": "chat",
+ "reply_session_id": original.session_id, "reply_policy": "always"},
+ )
+ await rt.report_to_lead(target, request, status="done", summary="Release found.")
+ assert rt.chat.notices[0][0] == original.session_id != newer.session_id
+ assert not [e for e in rt.published if type(e).__name__ == "AnnouncementRequested"]
+ assert "Find the release" in rt.chat.notices[0][1]["report"]
+
+
+async def test_voice_report_does_not_depend_on_an_open_text_chat(rt):
+ target, _ = await rt.roster.create(name="Scout")
+ request = SocietyEnvelope(
+ msg_type=MsgType.ASSIGN, from_agent="jarvis", to_agent=target.agent_id,
+ trace_id="request",
+ payload={"text": "Find the release", "reply_surface": "voice", "reply_policy": "always"},
+ )
+ await rt.report_to_lead(target, request, status="blocked", summary="Access unavailable.")
+ reports = [e for e in rt.published if type(e).__name__ == "AnnouncementRequested"]
+ assert len(reports) == 1 and "blocked" in reports[0].report
+
+
+async def test_deleted_origin_does_not_redirect_to_new_chat(rt):
+ session(rt)
+ target, _ = await rt.roster.create(name="Scout")
+ request = SocietyEnvelope(
+ msg_type=MsgType.ASSIGN, from_agent="jarvis", to_agent=target.agent_id,
+ trace_id="request",
+ payload={"text": "Task", "reply_surface": "chat", "reply_session_id": "deleted-chat"},
+ )
+ await rt.report_to_lead(target, request, status="done", summary="Finished.")
+ assert not rt.chat.notices
+
+
+def test_scheduled_turn_cannot_promote_itself_to_voice():
+ token = current_chat_turn.set(ChatTurn("society:scout", "turn", "Task", False, "trace"))
+ try:
+ assert origin_metadata()["reply_surface"] == "none"
+ finally:
+ current_chat_turn.reset(token)
diff --git a/tests/unit/society/test_lead_card.py b/tests/unit/society/test_lead_card.py
index 4e126b3007..f17e4046e3 100644
--- a/tests/unit/society/test_lead_card.py
+++ b/tests/unit/society/test_lead_card.py
@@ -12,8 +12,7 @@ society. The card is the fix, and these tests pin its contract:
5. the rule block says what "agent" means and where the old system went;
6. the realtime helper hands the planner the live names;
7. the lead picks the fitting agent for a nameless delegation;
-8. a lead-assigned RESULT reaches the person as a chat notice, never spoken
- (no background result is read aloud since 2026-09-30).
+8. a lead-assigned RESULT reaches its chat and a requested voice conversation.
"""
from __future__ import annotations
@@ -227,15 +226,18 @@ async def test_a_lead_assigned_result_reaches_the_person(rt: SocietyRuntime):
assert payload["agent_name"] == "Gmail agent"
assert payload["status"] == "done"
assert payload["text"] == "Invoice answered."
- # Shown, never spoken (2026-09-30): no announcement, no phrasing model call.
+ # The active voice conversation receives evidence, with no extra composer call.
published = rt.published # type: ignore[attr-defined]
- assert [e for e in published if type(e).__name__ == "AnnouncementRequested"] == []
+ spoken = [e for e in published if type(e).__name__ == "AnnouncementRequested"]
+ assert len(spoken) == 1
+ assert "Answer the invoice mail" in spoken[0].report
+ assert "Invoice answered" in spoken[0].report
async def test_an_agent_answer_to_the_lead_reaches_the_person(rt: SocietyRuntime):
"""Live 2026-09-09: the ack promised a report, the ANSWER arrived, and
nothing reached the person. The lead's incoming message is owed a notice
- too — in the chat, not by voice."""
+ too, with the original task context for the active voice model."""
import asyncio
chat: FakeChat = rt.chat # type: ignore[attr-defined]
@@ -272,8 +274,7 @@ async def test_an_agent_answer_to_the_lead_reaches_the_person(rt: SocietyRuntime
assert payload["agent_name"] == "Gmail agent"
assert payload["msg_type"] == "answer"
assert payload["text"] == "Invoice answered."
- # Shown, never spoken (2026-09-30).
- assert [e for e in published if type(e).__name__ == "AnnouncementRequested"] == []
+ assert len([e for e in published if type(e).__name__ == "AnnouncementRequested"]) == 1
def test_no_model_phrases_a_background_report_anymore() -> None:
diff --git a/tests/unit/society/test_remote_placement.py b/tests/unit/society/test_remote_placement.py
index 569fc2cdb4..53b8022a5d 100644
--- a/tests/unit/society/test_remote_placement.py
+++ b/tests/unit/society/test_remote_placement.py
@@ -377,5 +377,8 @@ async def test_placement_follows_the_society_roster(monkeypatch: pytest.MonkeyPa
return SimpleNamespace(surface=surface, session_id=sid)
assert await remote.placement_for_session(chat("society", "society:scout")) == ("c_1", "scout")
+ assert await remote.placement_for_session(
+ chat("society", "society:scout:routine:task1:run1")
+ ) == ("c_1", "scout")
assert await remote.placement_for_session(chat("society", "society:here")) is None
assert await remote.placement_for_session(chat("jarvis", "society:scout")) is None
diff --git a/tests/unit/society/test_remote_tools.py b/tests/unit/society/test_remote_tools.py
new file mode 100644
index 0000000000..8d4b5e7eba
--- /dev/null
+++ b/tests/unit/society/test_remote_tools.py
@@ -0,0 +1,442 @@
+"""Real SSH forwarding, turn isolation, CLI wiring and durable routine proof."""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import json
+import shlex
+from types import SimpleNamespace
+
+import httpx
+import pytest
+
+from jarvis.agent_chat.remote_mcp import RemoteMcpBridge, RemoteToolsUnavailable, ScopedMcpApp
+from jarvis.agent_chat.remote_mcp_config import SUPPORTED_RUNNERS, TOKEN_ENV, configure
+from jarvis.agent_chat.tool_context import register_turn, unregister_turn
+from jarvis.computers.remote_os import RemoteHost
+from jarvis.computers.ssh import SshTarget, open_session
+from jarvis.core.protocols import ChatTurn, current_chat_turn
+from tests.fakes.fake_ssh_server import TEST_PASSWORD, FakeSshServer, FakeSshState
+from tests.unit.society.test_cli_routines import world # noqa: F401 — reusable world fixture
+
+
+def active_turn():
+ turn = ChatTurn("society:scout", "remote-turn", "Check issues daily at 09:00", True, "trace")
+ token = current_chat_turn.set(turn)
+ registered = register_turn(turn.session_id)
+ current_chat_turn.reset(token)
+ try:
+ yield turn
+ finally:
+ unregister_turn(turn.session_id, registered)
+
+
+async def connection(tmp_path):
+ root = tmp_path / "remote"
+ root.mkdir()
+ server = FakeSshServer(FakeSshState(port_forwarding=True), sftp_root=root)
+ await server.start()
+ opened = await open_session(
+ SshTarget(
+ "127.0.0.1",
+ server.port,
+ "root",
+ password=TEST_PASSWORD,
+ host_key=server.host_key.export_public_key().decode(),
+ )
+ )
+ try:
+ yield SimpleNamespace(conn=opened.conn, server=server, root=root)
+ finally:
+ opened.conn.close()
+ await opened.conn.wait_closed()
+ await server.stop()
+
+
+async def echo(scope, receive, send):
+ headers = dict(scope["headers"])
+ await send({"type": "http.response.start", "status": 200, "headers": []})
+ await send(
+ {
+ "type": "http.response.body",
+ "body": json.dumps(
+ {
+ "session": headers[b"x-jarvis-chat-session"].decode(),
+ "path": scope["path"],
+ }
+ ).encode(),
+ }
+ )
+
+
+def scoped_app(active_turn, monkeypatch):
+ monkeypatch.setattr("jarvis.core.control_key.get_control_key", lambda: "fixture-owner-key")
+ return ScopedMcpApp(active_turn.session_id, active_turn.turn_id, echo)
+
+
[email protected]("path", ["/api/control/config", "/mcp/agents", "/mcp?session=other"])
+async def test_only_tools_path_is_exposed(scoped_app, path):
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=scoped_app), base_url="http://test"
+ ) as c:
+ response = await c.post(path, headers={"Authorization": "Bearer " + scoped_app.token})
+ assert response.status_code == 404
+
+
+async def test_remote_headers_cannot_change_session(scoped_app):
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=scoped_app), base_url="http://test"
+ ) as c:
+ response = await c.post(
+ "/mcp",
+ headers={
+ "Authorization": "Bearer " + scoped_app.token,
+ "X-Jarvis-Chat-Session": "society:other",
+ },
+ )
+ assert response.json() == {"session": "society:scout", "path": "/api/control/mcp/"}
+
+
[email protected]("state", ["wrong-token", "revoked", "expired", "new-turn", "no-turn"])
+async def test_stale_or_invalid_capability_is_rejected(scoped_app, active_turn, state):
+ if state == "revoked":
+ scoped_app.revoke()
+ if state == "expired":
+ scoped_app.expires = 0
+ if state in {"new-turn", "no-turn"}:
+ from jarvis.agent_chat import tool_context
+
+ tool_context._active.pop(active_turn.session_id)
+ context = None
+ if state == "new-turn":
+ token = current_chat_turn.set(ChatTurn(active_turn.session_id, "new", "yes", True, "trace"))
+ context = register_turn(active_turn.session_id)
+ current_chat_turn.reset(token)
+ credential = "wrong" if state == "wrong-token" else scoped_app.token
+ try:
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=scoped_app), base_url="http://test"
+ ) as c:
+ response = await c.post("/mcp", headers={"Authorization": "Bearer " + credential})
+ assert response.status_code == 401
+ finally:
+ unregister_turn(active_turn.session_id, context)
+
+
+async def test_real_reverse_ssh_forward_and_cleanup(connection, active_turn, monkeypatch):
+ monkeypatch.setattr("jarvis.core.control_key.get_control_key", lambda: "fixture-owner-key")
+ monkeypatch.setattr("jarvis.ui.web.mcp_server_routes.build_mcp_asgi_app", lambda: echo)
+ bridge = await RemoteMcpBridge.open(connection.conn, active_turn.session_id)
+ try:
+ async with httpx.AsyncClient() as c:
+ response = await c.post(
+ bridge.url, headers={"Authorization": "Bearer " + bridge.app.token}
+ )
+ assert response.json()["session"] == active_turn.session_id
+ assert (await c.post(bridge.url)).status_code == 401
+ finally:
+ await bridge.aclose()
+ assert not bridge.app.active
+ assert bridge._task.done()
+ async with httpx.AsyncClient(timeout=1) as c:
+ with pytest.raises(httpx.TransportError):
+ await c.post(bridge.url)
+
+
+async def test_forward_refusal_closes_listener(connection, active_turn):
+ connection.server.state.port_forwarding = False
+ before = {t for t in asyncio.all_tasks() if t.get_name() == "remote-jarvis-tools"}
+ with pytest.raises(RemoteToolsUnavailable, match="TCP forwarding"):
+ await RemoteMcpBridge.open(connection.conn, active_turn.session_id)
+ assert {t for t in asyncio.all_tasks() if t.get_name() == "remote-jarvis-tools"} == before
+
+
+async def test_bridge_requires_registered_turn(connection):
+ with pytest.raises(RemoteToolsUnavailable, match="active chat turn"):
+ await RemoteMcpBridge.open(connection.conn, "society:scout")
+
+
+async def test_remote_cli_spawn_mounts_tools_and_reaps_without_caller_wait(
+ connection,
+ active_turn,
+ monkeypatch,
+):
+ from jarvis.agent_chat import remote_cli
+
+ @contextlib.asynccontextmanager
+ async def session(_computer):
+ yield connection
+
+ async def host(_computer, _opened):
+ return RemoteHost()
+
+ monkeypatch.setattr(
+ "jarvis.computers.service.get_service",
+ lambda: SimpleNamespace(
+ session=session,
+ get=lambda _: SimpleNamespace(name="Test computer", facts=None),
+ ),
+ )
+ monkeypatch.setattr("jarvis.computers.remote_os.remote_host", host)
+ monkeypatch.setattr("jarvis.core.control_key.get_control_key", lambda: "fixture-owner-key")
+ monkeypatch.setattr("jarvis.ui.web.mcp_server_routes.build_mcp_asgi_app", lambda: echo)
+
+ async def handler(command, process):
+ script = await process.stdin.read() if command.endswith(" -s") else command
+ if "command -v" in script:
+ process.stdout.write("/workspace\nfound\n")
+ else:
+ parts = shlex.split(command)
+ if "--mcp-config" not in parts:
+ launcher = (connection.root / parts[-1].lstrip("/")).read_text()
+ parts = shlex.split(next(line[5:] for line in launcher.splitlines()
+ if line.startswith("exec ")))
+ config_path = parts[parts.index("--mcp-config") + 1]
+ config = json.loads((connection.root / config_path.lstrip("/")).read_text())
+ server = config["mcpServers"]["jarvis"]
+ async with httpx.AsyncClient() as client:
+ response = await client.post(server["url"], headers=server["headers"])
+ process.stdout.write(response.text + "\n")
+ process.exit(0)
+ return True
+
+ connection.server.state.handler = handler
+ proc = await remote_cli.spawn(
+ "fixture",
+ agent_id="scout",
+ runner="claude-cli",
+ binary="claude",
+ argv=["claude", "--print"],
+ local_cwd="/local",
+ env={"SECRET": "must-stay-local"},
+ session_id=active_turn.session_id,
+ )
+ response = json.loads(await proc.stdout.readline())
+ assert response["session"] == active_turn.session_id
+ # Resource cleanup must not depend on the runner reaching proc.wait().
+ await asyncio.wait_for(proc._reaper, timeout=10)
+ assert proc._closed
+ assert not list((connection.root / "workspace").glob(".jarvis-mcp-*.json"))
+ assert not any("must-stay-local" in command for command in connection.server.state.commands)
+ assert await proc.wait() == 0
+
+
+async def test_disconnect_and_cancel_revoke_bridge(connection, active_turn):
+ bridge = await RemoteMcpBridge.open(connection.conn, active_turn.session_id)
+ connection.conn.close()
+ await connection.conn.wait_closed()
+ await bridge.aclose()
+ assert not bridge.app.active
+ assert bridge._task.done()
+
+
+async def test_project_configs_cannot_cross_concurrent_sessions(connection):
+ async with contextlib.AsyncExitStack() as first, contextlib.AsyncExitStack() as second:
+ kwargs = dict(
+ runner="grok-cli",
+ cwd="/workspace",
+ url="http://127.0.0.1:23456/mcp",
+ token="fixture", # noqa: S106 — test capability
+ )
+ await configure(connection, RemoteHost(), first, **kwargs)
+ with pytest.raises(RemoteToolsUnavailable, match="Another turn"):
+ await configure(connection, RemoteHost(), second, **kwargs)
+
+
[email protected]("runner", sorted(SUPPORTED_RUNNERS))
+async def test_runner_wiring_and_file_cleanup(connection, runner):
+ root = connection.root
+ workspace = root / "workspace"
+ workspace.mkdir()
+ old_files = {
+ ".grok/config.toml": 'theme = "dark"\n',
+ ".cursor/mcp.json": '{"mcpServers":{"other":{"url":"https://example.test"}}}',
+ }
+ for name, body in old_files.items():
+ path = workspace / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(body)
+ async with contextlib.AsyncExitStack() as stack:
+ config = await configure(
+ connection,
+ RemoteHost(),
+ stack,
+ runner=runner,
+ cwd="/workspace",
+ url="http://127.0.0.1:23456/mcp",
+ token="turn-only", # noqa: S106 — isolated fake SSH server
+ )
+ assert config.apply(["cli", "--print"])[0] == "cli"
+ assert config.env[TOKEN_ENV] == "turn-only"
+ content = " ".join(p.read_text() for p in workspace.rglob("*") if p.is_file())
+ encoded = json.dumps(config.env) + json.dumps(config.args) + content
+ assert "http://127.0.0.1:23456/mcp" in encoded
+ assert "fixture-owner-key" not in encoded
+ if runner == "claude-cli":
+ loaded = json.loads((connection.root / config.args[1].lstrip("/")).read_text())
+ assert loaded["mcpServers"]["jarvis"]["type"] == "http"
+ if runner == "codex-cli":
+ assert "mcp_servers.jarvis.required=true" in config.args
+ assert 'mcp_servers.jarvis.default_tools_approval_mode="approve"' in config.args
+ if runner == "cursor-cli":
+ assert '"other"' in content
+ if runner == "grok-cli":
+ assert 'theme = "dark"' in content
+ remaining = {
+ p.relative_to(workspace).as_posix(): p.read_text()
+ for p in workspace.rglob("*")
+ if p.is_file()
+ }
+ assert remaining == old_files
+
+
+async def test_cleanup_preserves_concurrent_changes(connection):
+ async with contextlib.AsyncExitStack() as stack:
+ await configure(
+ connection,
+ RemoteHost(),
+ stack,
+ runner="cursor-cli",
+ cwd="/workspace",
+ url="http://127.0.0.1:23456/mcp",
+ token="turn-only", # noqa: S106 — isolated fake SSH server
+ )
+ path = connection.root / "workspace/.cursor/mcp.json"
+ path.write_text('{"user":"changed"}')
+ assert json.loads(path.read_text()) == {"user": "changed"}
+
+
+async def test_routine_roundtrip_over_ssh_persists_after_reopen(
+ connection,
+ active_turn,
+ world, # noqa: F811 — imported pytest fixture
+ monkeypatch,
+ tmp_path,
+):
+ from jarvis.core.bus import EventBus
+ from jarvis.tasks.scheduler import TaskScheduler
+ from jarvis.tasks.store import TaskStore
+ from jarvis.ui.web import mcp_server_routes as routes
+
+ # Real persistence and scheduler; no runner or scheduling loop is started.
+ store = TaskStore(tmp_path / "persistent-tasks.db")
+ await store.init()
+ scheduler = TaskScheduler(store, EventBus())
+ world.rt.task_services = lambda: (store, scheduler)
+ monkeypatch.setattr("jarvis.core.control_key.get_control_key", lambda: "fixture-owner-key")
+ monkeypatch.setattr(
+ "jarvis.core.control_key.verify_control_key", lambda key: key == "fixture-owner-key"
+ )
+ surface = routes._Surface("tools", routes._build_tools_server)
+ monkeypatch.setattr(routes, "_TOOLS_SURFACE", surface)
+ monkeypatch.setitem(routes._SUFFIX_SURFACES, "", surface)
+ bridge = await RemoteMcpBridge.open(connection.conn, active_turn.session_id)
+ headers = {
+ "Authorization": "Bearer " + bridge.app.token,
+ "Accept": "application/json, text/event-stream",
+ }
+ try:
+ async with httpx.AsyncClient(headers=headers) as c:
+
+ async def call(method, params):
+ response = await c.post(
+ bridge.url, json={"jsonrpc": "2.0", "id": 1, "method": method, "params": params}
+ )
+ assert response.status_code == 200, response.text
+ return response.json()["result"]
+
+ catalog = await call("tools/list", {})
+ assert {"society_routines", "society_propose_change"} <= {
+ t["name"] for t in catalog["tools"]
+ }
+ result = await call(
+ "tools/call",
+ {
+ "name": "society_propose_change",
+ "arguments": {
+ "kind": "routine",
+ "mode": "apply",
+ "request_quote": active_turn.user_text,
+ "payload": {
+ "title": "Issue report",
+ "prompt": "Read and rank repository issues.",
+ "schedule": {
+ "kind": "calendar",
+ "local_time": "09:00",
+ "timezone": "Europe/Berlin",
+ },
+ },
+ },
+ },
+ )
+ assert not result["isError"], result
+ assert json.loads(result["content"][0]["text"])["applied"] is True
+ result = await call("tools/call", {"name": "society_routines", "arguments": {}})
+ saved = json.loads(result["content"][0]["text"])["routines"]
+ assert len(saved) == 1
+ assert saved[0]["trigger"]["timezone"] == "Europe/Berlin"
+ assert saved[0]["state"] == "scheduled"
+ assert saved[0]["due_at_ns"] > 0
+ await scheduler.shutdown()
+ await store.close()
+ await store.init()
+ result = await call("tools/call", {"name": "society_routines", "arguments": {}})
+ restored = json.loads(result["content"][0]["text"])["routines"]
+ assert restored[0]["id"] == saved[0]["id"]
+ assert restored[0]["trigger"]["local_time"] == "09:00"
+ finally:
+ await bridge.aclose()
+ await scheduler.shutdown()
+ if surface._task is not None:
+ surface._task.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await surface._task
+ await store.close()
+
+
+async def test_expired_request_cannot_borrow_next_turn(world, active_turn): # noqa: F811
+ import mcp.types as types
+
+ from jarvis.agent_chat.tool_context import required_turn_id
+ from jarvis.mcp.jarvis_tools_server import CHAT_SESSION_REF
+
+ entered, release = asyncio.Event(), asyncio.Event()
+ original = world.gateway.session_catalog
+
+ async def delayed(session_id):
+ entered.set()
+ await release.wait()
+ return await original(session_id)
+
+ world.gateway.session_catalog = delayed
+ session = CHAT_SESSION_REF.set(active_turn.session_id)
+ bound = required_turn_id.set(active_turn.turn_id)
+ task = asyncio.create_task(
+ world.server.request_handlers[types.CallToolRequest](
+ types.CallToolRequest(
+ method="tools/call",
+ params=types.CallToolRequestParams(
+ name="society_routines",
+ arguments={},
+ ),
+ )
+ )
+ )
+ required_turn_id.reset(bound)
+ CHAT_SESSION_REF.reset(session)
+ await entered.wait()
+ token = current_chat_turn.set(ChatTurn(active_turn.session_id, "next", "yes", True, "trace"))
+ registered = register_turn(active_turn.session_id)
+ current_chat_turn.reset(token)
+ release.set()
+ try:
+ result = (await task).root
+ assert result.isError
+ assert "turn has ended" in result.content[0].text
+ assert world.executor.calls == []
+ finally:
+ unregister_turn(active_turn.session_id, registered)
diff --git a/tests/unit/speech/test_delegation_results.py b/tests/unit/speech/test_delegation_results.py
new file mode 100644
index 0000000000..ccb07782d4
--- /dev/null
+++ b/tests/unit/speech/test_delegation_results.py
@@ -0,0 +1,120 @@
+"""Delegation completion never occupies the user's conversational floor."""
+
+import asyncio
+import json
+from dataclasses import replace
+
+import pytest
+
+from jarvis.core.delegation import ResultInbox, result_announcement
+from jarvis.speech.pipeline import TurnTakingState
+from tests.unit.speech.test_realtime_announcement_bridge import _pipeline
+
+
+def result(name="Scout", status="done"):
+ return result_announcement(
+ source="society.lead", request_id=name, name=name,
+ request=f"Research {name}", status=status, report=f"Findings from {name}", language="en",
+ )
+
+
[email protected](autouse=True)
+def quick_batch(monkeypatch):
+ monkeypatch.setattr("jarvis.core.delegation.BATCH_WINDOW_S", 0.005)
+
+
+async def test_simultaneous_results_make_one_natural_live_request_with_both_tasks():
+ pipe, tts, _, live = _pipeline(accepted=True)
+ first, second = result(), result("Coder", "blocked")
+ await asyncio.gather(pipe._on_announcement(first), pipe._on_announcement(second))
+ assert not live.calls # Producers return without waiting for speech or a model.
+ await asyncio.sleep(0.03)
+ assert len(live.calls) == 1
+ assert "Research Scout" in live.calls[0]["report"]
+ assert "Research Coder" in live.calls[0]["report"]
+ assert "blocked" in live.calls[0]["report"]
+ assert not tts.calls
+ pipe._settle_agent_reply(completed=True)
+ # A replayed transport callback must not read the same result twice.
+ await pipe._on_announcement(replace(first, timestamp_ns=first.timestamp_ns + 1))
+ await asyncio.sleep(0.02)
+ assert len(live.calls) == 1
+
+
+ TurnTakingState.IDLE, TurnTakingState.USER_SPEAKING,
+ TurnTakingState.PROCESSING, TurnTakingState.JARVIS_SPEAKING,
+])
+async def test_result_waits_while_user_keeps_talking_or_another_answer_is_running(state):
+ pipe, tts, player, live = _pipeline(accepted=True)
+ pipe._turn_state = state
+ await pipe._on_announcement(result())
+ await asyncio.sleep(0.02)
+ assert not live.calls and not tts.calls and not player.plays
+ assert live.remembered # Follow-up context is available even before it is spoken.
+ await pipe._set_turn_state(TurnTakingState.LISTENING)
+ await asyncio.sleep(0.03)
+ assert len(live.calls) == 1
+
+
+async def test_user_begins_speaking_during_coalescing_window():
+ pipe, _, _, live = _pipeline(accepted=True)
+ await pipe._on_announcement(result())
+ pipe._turn_state = TurnTakingState.USER_SPEAKING
+ await asyncio.sleep(0.03)
+ assert not live.calls
+ await pipe._set_turn_state(TurnTakingState.LISTENING)
+ await asyncio.sleep(0.03)
+ assert len(live.calls) == 1
+
+
+async def test_muted_or_hung_up_result_waits_for_next_call():
+ pipe, tts, _, live = _pipeline(accepted=True)
+ pipe._hangup_event.set()
+ pipe._muted = True
+ await pipe._on_announcement(result())
+ await asyncio.sleep(0.03)
+ assert not live.calls and not tts.calls
+ pipe._hangup_event.clear()
+ pipe._muted = False
+ await pipe._set_turn_state(TurnTakingState.LISTENING)
+ await asyncio.sleep(0.03)
+ assert len(live.calls) == 1
+
+
+async def test_arrivals_during_readback_wait_until_audio_finishes():
+ pipe, _, _, live = _pipeline(accepted=True)
+ await pipe._on_announcement(result())
+ await asyncio.sleep(0.03)
+ await pipe._on_announcement(result("Coder"))
+ await asyncio.sleep(0.03)
+ assert len(live.calls) == 1
+ pipe._settle_agent_reply(completed=True)
+ await pipe._set_turn_state(TurnTakingState.LISTENING)
+ await asyncio.sleep(0.03)
+ assert len(live.calls) == 2
+
+
+def test_batch_has_bounded_fair_context_and_preserves_overflow():
+ inbox = ResultInbox()
+ for i in range(5):
+ inbox.add(replace(result(str(i)), report=f"task-{i} " + "x" * 8000 + f" verdict-{i}"))
+ batch = inbox.take()
+ assert len(batch.report) < 6000
+ for i in range(4):
+ assert f"task-{i}" in batch.report and f"verdict-{i}" in batch.report
+ assert "task-4" in inbox.take().report
+ assert inbox.take() is None
+
+
+def test_report_is_data_and_stopped_is_not_claimed_success():
+ event = result("Coder", "completed")
+ assert "is done" not in event.text
+ data = json.loads(event.report)
+ assert data["request"] == "Research Coder"
+ assert data["status"] == "completed"
diff --git a/tests/unit/ui/test_winforms_errors.py b/tests/unit/ui/test_winforms_errors.py
new file mode 100644
index 0000000000..a6467b871a
--- /dev/null
+++ b/tests/unit/ui/test_winforms_errors.py
@@ -0,0 +1,145 @@
+"""Managed UI failures retain their details and their Continue/Quit behavior."""
+
+from __future__ import annotations
+
+import sys
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.ui import winforms_errors as errors
+
+
+class Event:
+ def __init__(self):
+ self.handlers = []
+
+ def __iadd__(self, handler):
+ self.handlers.append(handler)
+ return self
+
+ def __isub__(self, handler):
+ self.handlers.remove(handler)
+ return self
+
+ def fire(self, args):
+ for handler in self.handlers[:]:
+ handler(None, args)
+
+
+def forms(monkeypatch):
+ monkeypatch.setattr(errors, "_handlers", {})
+ current = SimpleNamespace(ManagedThreadId=42)
+ monkeypatch.setitem(sys.modules, "System.Threading", SimpleNamespace(
+ Thread=SimpleNamespace(CurrentThread=current),
+ ))
+ calls = []
+
+ class Dialog:
+ def __init__(self, exception):
+ calls.append(("dialog", exception))
+
+ def ShowDialog(self):
+ return api.result
+
+ def Dispose(self):
+ calls.append(("dispose", None))
+
+ api = SimpleNamespace(
+ Application=SimpleNamespace(
+ ThreadException=Event(), ThreadExit=Event(),
+ Exit=lambda: calls.append(("exit", None)),
+ ),
+ ThreadExceptionDialog=Dialog,
+ DialogResult=SimpleNamespace(Abort="abort"),
+ result="ignore", calls=calls, current=current,
+ )
+ return api
+
+
+def test_managed_details_logged_before_continue_dialog(forms, monkeypatch, caplog):
+ exception = SimpleNamespace(ToString=lambda: "OuterException\nInnerException\n at CloseWindow")
+ original = errors._present_exception
+
+ def present(api, exc):
+ assert "InnerException\n at CloseWindow" in caplog.text
+ original(api, exc)
+
+ monkeypatch.setattr(errors, "_present_exception", present)
+ errors._subscribe(forms, 42)
+ forms.Application.ThreadException.fire(SimpleNamespace(Exception=exception))
+ assert forms.calls == [("dialog", exception), ("dispose", None)]
+
+
+def test_quit_runs_application_exit_after_disposing_dialog(forms):
+ forms.result = "abort"
+ exception = SimpleNamespace(ToString=lambda: "Close failed")
+ errors._subscribe(forms, 42)
+ forms.Application.ThreadException.fire(SimpleNamespace(Exception=exception))
+ assert forms.calls == [("dialog", exception), ("dispose", None), ("exit", None)]
+
+
+def test_managed_error_details_redact_credentials(caplog):
+ credential_example = "example-credential-value"
+ exception = SimpleNamespace(
+ ToString=lambda: f"InvalidOperationException: password={credential_example}"
+ )
+ errors._report_exception(exception)
+ assert credential_example not in caplog.text
+ assert "<redacted:labelled_secret>" in caplog.text
+
+
+def test_partial_subscription_is_rolled_back(forms):
+ class UnavailableEvent:
+ def __iadd__(self, _handler):
+ raise RuntimeError("event unavailable")
+
+ forms.Application.ThreadExit = UnavailableEvent()
+ with pytest.raises(RuntimeError, match="event unavailable"):
+ errors._subscribe(forms, 42)
+ assert not forms.Application.ThreadException.handlers
+ assert not errors._handlers
+
+
+def test_logging_failure_still_presents_original_exception(forms, monkeypatch):
+ def broken(_exc):
+ raise OSError("log unavailable")
+
+ monkeypatch.setattr(errors, "_report_exception", broken)
+ exception = object()
+ errors._subscribe(forms, 42)
+ forms.Application.ThreadException.fire(SimpleNamespace(Exception=exception))
+ assert forms.calls[0] == ("dialog", exception)
+
+
+def test_duplicate_windows_do_not_duplicate_handler_and_thread_exit_releases_it(forms):
+ errors._subscribe(forms, 42)
+ errors._subscribe(forms, 42)
+ assert len(forms.Application.ThreadException.handlers) == 1
+ forms.current.ManagedThreadId = 99
+ forms.Application.ThreadExit.fire(None)
+ assert 42 in errors._handlers
+ forms.current.ManagedThreadId = 42
+ forms.Application.ThreadExit.fire(None)
+ assert not errors._handlers
+ assert not forms.Application.ThreadException.handlers
+ assert not forms.Application.ThreadExit.handlers
+
+
[email protected]("platform", ["linux", "darwin"])
+def test_non_windows_does_not_touch_native_dependencies(monkeypatch, platform):
+ monkeypatch.setattr(errors.sys, "platform", platform)
+ errors.register_winforms_error_logging(object())
+
+
+def test_install_waits_for_synchronous_gui_event(monkeypatch):
+ monkeypatch.setattr(errors.sys, "platform", "win32")
+ calls = []
+ monkeypatch.setattr(errors, "_install_on_gui_thread", lambda: calls.append("installed"))
+ event = Event()
+ window = SimpleNamespace(events=SimpleNamespace(before_show=event))
+ errors.register_winforms_error_logging(window)
+ assert calls == []
+ event.handlers[0]()
+ assert calls == ["installed"]
diff --git a/tests/unit/workspace/test_codex_embedded_launch.py b/tests/unit/workspace/test_codex_embedded_launch.py
new file mode 100644
index 0000000000..ede93f4666
--- /dev/null
+++ b/tests/unit/workspace/test_codex_embedded_launch.py
@@ -0,0 +1,35 @@
+"""Keep Codex's oversized startup update box out of embedded terminal panes."""
+
+import pytest
+
+from jarvis.agentic_ide import session
+from jarvis.agentic_ide.agent_sessions import ResumeHandle, resume_argv
+from jarvis.workspace.agents import get_agent
+
+
[email protected]("remote", [False, True])
[email protected]("continuing", [False, True])
+def test_embedded_codex_suppresses_update_box_on_every_launch(
+ monkeypatch: pytest.MonkeyPatch, remote: bool, continuing: bool
+) -> None:
+ # An executable path takes the same argv path on POSIX and Windows;
+ # test_session covers the Windows npm shim separately.
+ monkeypatch.setattr(session.shutil, "which", lambda _name: "/tools/codex")
+ launcher = session.remote_agent_argv if remote else session.agent_argv
+ argv = launcher("codex")
+ assert argv is not None
+ if continuing:
+ resume = resume_argv(
+ "codex", ResumeHandle(kind="codex_rollout", id="conversation", captured_at=1.0)
+ )
+ assert resume is not None
+ argv += resume
+ assert argv[-2:] == ("resume", "conversation")
+
+ assert argv[1:3] == ("-c", "check_for_update_on_startup=false")
+ # Only pane launches get the override, never detection or the plain CLI
+ # command used outside the embedded IDE.
+ agent = get_agent("codex")
+ assert agent is not None and agent.spec is not None
+ assert agent.launch_command == "codex"
+ assert agent.spec.check_command == ("codex", "--version")