Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

feat: integrate local changes for the 3.0.0 release (#500)

Ruben Lütke7f00ab95c3fa
diff --git a/.agents/README.md b/.agents/README.md
index 0011fe7a3e..712ce7147d 100644
--- a/.agents/README.md
+++ b/.agents/README.md
@@ -16,4 +16,4 @@ This directory is the source for the versioned agent definitions and skills.
committed.
Edit this tree only. Run the sync scripts after editing or let the pre-commit
-hook create the compatibility copies.
+hook create the compatibility copies.
\ No newline at end of file
diff --git a/.github/workflows/browser-runtime.yml b/.github/workflows/browser-runtime.yml
index 5985f58c26..4ddc2cb80d 100644
--- a/.github/workflows/browser-runtime.yml
+++ b/.github/workflows/browser-runtime.yml
@@ -2,7 +2,17 @@ name: Browser runtime
# Called by ci.yml's `browser` lane (path-gated there) and runnable by hand.
on:
workflow_call:
+ inputs:
+ target:
+ type: string
+ default: all
workflow_dispatch:
+ inputs:
+ target:
+ description: "All platforms, or a focused Windows capture investigation"
+ type: choice
+ options: [all, windows-latest]
+ default: all
permissions:
contents: read
jobs:
@@ -10,7 +20,7 @@ jobs:
strategy:
fail-fast: false
matrix:
- os: [ubuntu-latest, ubuntu-24.04-arm, windows-latest, windows-11-arm, macos-latest, macos-15-intel]
+ os: ${{ inputs.target == 'windows-latest' && fromJSON('["windows-latest"]') || fromJSON('["ubuntu-latest", "ubuntu-24.04-arm", "windows-latest", "windows-11-arm", "macos-latest", "macos-15-intel"]') }}
runs-on: ${{ matrix.os }}
timeout-minutes: 30
steps:
@@ -39,6 +49,26 @@ jobs:
stream.write(f"JARVIS_BROWSER_TEST_EXECUTABLE={browser_executable()}\n")
- name: Test real frames, ownership and protocol
run: python -m pytest tests/contract/test_browser_live.py tests/contract/test_browser_contract.py tests/contract/test_browser_desktop_mcp.py tests/contract/test_browser_cli_model.py tests/unit/society/test_browser.py tests/unit/society/test_browser_install.py tests/unit/society/test_approvals.py tests/unit/agent_chat/test_jarvis_identity.py tests/unit/agent_chat/test_agent_chat_ladders.py tests/unit/brain/test_tool_gateway.py -q --tb=short --junitxml=browser-results.xml
+ - name: Diagnose Windows capture style support after a failure
+ if: failure() && matrix.os == 'windows-latest' && env.JARVIS_BROWSER_TEST_PYTHON != ''
+ shell: python
+ run: |
+ import os
+ import subprocess
+ from pathlib import Path
+ from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS
+ from jarvis.society.browser.install import worker_env
+ result = subprocess.run(
+ [os.environ["JARVIS_BROWSER_TEST_PYTHON"],
+ "tests/contract/native_window_probe.py",
+ os.environ["JARVIS_BROWSER_TEST_EXECUTABLE"], "--capture-styles"],
+ env=worker_env(Path(os.environ["RUNNER_TEMP"])),
+ encoding="utf-8", capture_output=True, creationflags=NO_WINDOW_CREATIONFLAGS,
+ timeout=150, check=False,
+ )
+ print(result.stdout, flush=True)
+ print(result.stderr, flush=True)
+ result.check_returncode()
- name: Preserve browser test evidence
if: always()
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a82bcfcd44..bd938aca50 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -249,6 +249,8 @@ jobs:
python scripts/ci/run_gates.py "${args[@]}"
- name: Validate workflow syntax, expressions and action inputs
if: always()
+ env:
+ GOTOOLCHAIN: go1.27.1
run: |
GOBIN="$RUNNER_TEMP/actionlint" go install github.com/rhysd/actionlint/cmd/[email protected]
"$RUNNER_TEMP/actionlint/actionlint" -shellcheck= -pyflakes=
diff --git a/.gitignore b/.gitignore
index 975f94c0b2..58c84b5946 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,4 @@
-# Secrets (nie committen — Secrets liegen im Windows Credential Manager)
+# Secrets (never commit; stored in the system credential manager)
.env
.env.local
.env.*.local
diff --git a/AGENTS.md b/AGENTS.md
index 9d2f5be497..9ffb5e3a03 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -78,6 +78,12 @@ the install base, and "works on my machine" is the defect. (AP-23)
- **Any single key must work.** Gate on capability, never a provider name or
model id (AP-21). A tier whose primary AND fallback share one provider family
is a brick — every chain crosses families or degrades honestly (AP-22).
+- **Bypass permissions is the project default.** Fresh installs, chats, agents,
+ delegated work and coding panes execute permitted work without asking for
+ permission. Carry this default into native CLI launch flags too; an omitted
+ pick must not restore a vendor's ask-first default. Preserve explicitly
+ selected restrictions and blocked actions. Questions for missing task
+ information remain available; do not turn them into permission prompts.
- **Nothing the user did not start may bill a key on its own.** Health dots
come from real calls, never a paid probe. Once a subscription is connected,
background work runs only on subscriptions or local models
diff --git a/docs/agent-server.md b/docs/agent-server.md
new file mode 100644
index 0000000000..58ef557830
--- /dev/null
+++ b/docs/agent-server.md
@@ -0,0 +1,87 @@
+# Independent agent server
+
+The independent server owns agent execution. Desktop windows and browsers are
+clients of the same HTTP and WebSocket application. Closing a client does not
+cancel a turn, release an approval, stop a runtime, or transfer the databases
+to another process.
+
+## Start and connect
+
+In Settings, enable **Independent agent server** for the next desktop launch.
+Leave its address empty to start a server on this computer. Existing integrated
+desktop installations retain their current mode unless this setting is enabled.
+The integrated desktop must finish closing before the independent server takes
+ownership; a client refuses to evict an existing owner.
+
+The equivalent entry points are:
+
+```console
+python -m jarvis --persistent-server
+python -m jarvis --connect
+python -m jarvis --connect https://jarvis.example.com
+```
+
+A local client starts or reuses the server, guarded by a launch lock. Multiple
+clients may connect. A remote client uses the server's login page; it never
+sends its local control credential, provider keys or subscription login to the
+remote server. Remote origins require HTTPS; a local SSH tunnel may use HTTP.
+Credentials must not be embedded in the address.
+
+The server remains available when no work is scheduled. It stops only when
+explicitly stopped, the operating system exits, or the process fails. The tray's
+stop action and the authenticated `POST /api/agent-server/stop` stop the server;
+closing a client does neither. `GET /api/agent-server/status` distinguishes
+server ownership, completed agent startup and a startup failure. Both routes
+are available through the dynamic `jarvis api agent-server` command group.
+
+For login startup, the existing background-only option includes the persistent
+server flag when a local independent server is selected. On a headless host,
+use the operating system's process supervisor for restart after process or host
+failure. This is a user process on Windows, not a SYSTEM service.
+
+## Ownership and recovery
+
+The server owns the roster, chat events, routine scheduler, conversations,
+memory, skills, delegation, provider access, model gateway and tool authorization.
+Jarvis, Hermes and OpenClaw use those same services. External runtimes live on
+the server and reach its local model and MCP endpoints; a remote UI does not
+need either runtime installed.
+
+Agent services start after the HTTP surface is available, even if no client
+opens the Society page. Startup itself makes no model warm-up request.
+Runtime preparation and provider calls retain their existing account and
+background billing policies.
+
+The existing chat stream replays stored events after a sequence cursor. Live
+approvals remain owned by the server when a client disconnects. Reconnection
+includes pending approvals and current turn state. A missing client never grants
+permission. Completed effects and terminal events remain stored across server
+restarts. A server crash is different from a client disconnect: unfinished turns
+use the existing interruption recovery and are not blindly replayed. Check their
+stored result before explicitly continuing.
+
+## Devices and deployment boundaries
+
+The independent client provides the web application and browser voice. Local
+wake word, overlays, native capture and desktop automation still require an
+available device and its permissions. The integrated desktop remains available
+for those features. A remote server cannot access the client's screen merely
+because the client opened a page. Server-side file and shell operations address
+the server; connected-computer operations keep their explicit target.
+
+The SSH placement of a coding CLI is separate from connecting a UI to a Jarvis
+server. The server connection does not expand the supported SSH runtime list
+or grant new device access.
+
+## Verification
+
+The disconnect contract is exercised for Jarvis, Hermes and OpenClaw dispatches
+with scripted runners: disconnect during approval, reconnect with a cursor,
+approve once, collect one result, and reopen the durable store. The runtime tests
+exercise ACP subprocess translation and session restoration against scripted
+agents. These checks do not claim live model-provider or remote-host execution.
+
+Server ownership tests cover concurrent startup boundaries, refusal to take over
+a persistent owner, explicit stopping, atomic configuration and endpoint validation.
+The local process proof must additionally start the real backend, close and
+reopen clients, and verify that the same server remains healthy.
diff --git a/docs/appshots.md b/docs/appshots.md
index f644fe8815..d1cc59f007 100644
--- a/docs/appshots.md
+++ b/docs/appshots.md
@@ -43,9 +43,12 @@ Jarvis X screenshots (`[jarvisx].copy_to_clipboard`) use the same code.
## Selecting an area
-The area picker is a short-lived PySide6 process (`python -m
-jarvis.appshot.picker`) that starts on the shortcut and exits after one
-selection, so nothing stays resident. The picker provides:
+The area picker uses a prepared PySide6 process (`python -m
+jarvis.appshot.picker --resident`). The main instance prepares it after the
+wake-ready gate when Appshots are enabled. Standby creates no selection
+windows and captures no pixels. Each shortcut samples the current screens
+and window layout; completing or cancelling releases the frozen images,
+blur previews, markings and windows while keeping Qt loaded. The picker provides:
- every screen is frozen under a light dim the moment the picker opens, so
nothing moves under the selection (where a frozen frame cannot be grabbed,
@@ -277,7 +280,8 @@ it outside the editor; **not done** = not built, with the reason.
| Corner card (copy / save / annotate / drag after capture) | Elsewhere: the corner card (click = editor, drag = file) |
| Editor window: drag handle, zoom, Save / Done | Done: "Drag me" (Windows, macOS), zoom Fit/50/100/200 %, Save to Downloads, Done = use in the next message |
| Scrolling capture, self-timer | Not done (the page's "in 3 s" button is the only timer) |
-| Screen recording, video editor, GIF | Not done |
+| Screen recording and video editor | Local MP4 recordings with quality controls and optional system audio |
+| GIF export | Not done |
| Cloud upload and sharing | Not done — appshots stay on this machine by design |
| Text recognition (OCR), QR codes | Not done in the editor; the model reads the picture itself when it gets it |
| Pin to screen (floating screenshot) | Not done: needs an always-on-top native window per OS |
@@ -309,14 +313,19 @@ to the assistant; they are the screenshot-tool half.
## The gallery
-Below the settings, **Your appshots** shows every appshot taken with the
+Below the settings, **Your captures** shows every appshot taken with the
shortcuts, the page's buttons or the assistant's `take_appshot` tool, and every
edit saved in the editor, newest first. An edit sits right before its original
and wears an **Edited** badge; the **Edited** filter shows only those. A look
-a conversation turn took on its own is never kept.
+a conversation turn took on its own is never kept. Finished screen recordings
+appear in the same grid, with a video badge and duration. The **Recordings**
+filter shows only videos. Existing recordings appear automatically, including
+ones created before this gallery supported video.
- **Click** a picture to open it in the editor again. Saving there keeps the
new edit beside the original and hands it to the assistant like any edit.
+- **Click** a video to play it with playback and seeking controls. The player
+ also offers an MP4 download. Closing it returns to the same gallery.
- **Drag** a picture into a chat composer or a terminal pane to attach it.
The drag carries the picture's real path the way a row from the workspace
explorer does, so both drop targets take it unchanged. In a browser the
@@ -324,7 +333,9 @@ a conversation turn took on its own is never kept.
Inside the desktop shell the grip on a tile starts a native file drag that
reaches any other app (mail, Explorer/Finder, a browser upload).
- **Delete** on an edited tile removes only that edit; on an original it
- removes the appshot and its edit. **Delete all** asks once more first.
+ removes the appshot and its edit. On a video it removes that recording.
+ **Delete all** confirms that both pictures and videos will be removed;
+ a recording still in progress is never included.
The pictures live in `<data dir>/appshots/<id>/` (`original.<ext>`,
`edited.png`, `meta.json`, small thumbnails made on first view). Only the
@@ -333,6 +344,50 @@ on-screen text. The library keeps the newest 500 appshots and removes the
oldest first. **Keep appshot history** (`[appshot].library`) turns it off;
what is already kept stays until it is deleted.
+Screen recordings are finalized as MP4 files under the app's user-data
+directory (`appshot-recordings/`) and remain there across navigation and app
+restarts, independently of the screenshot-history switch. The gallery indexes
+these existing files rather than duplicating them. It loads only small video
+posters for tiles and streams the full file when playback is opened. The image
+editor's Add Picture panel continues to offer images only.
+
+### Recording quality and system audio
+
+The recording panel saves preferences for both button and shortcut starts.
+Defaults are **1080p, 60 FPS, 12 Mbps**, with system audio off. Available
+resolutions are 720p, 1080p, 1440p, 2160p and native (including displays above
+4K). Presets are bounding boxes: the source keeps its aspect ratio, portrait
+sources use the corresponding portrait bounds, and smaller areas are never
+enlarged. Encoders may add a single padding pixel for even H.264 dimensions.
+
+Frame-rate choices are 30, 60 and 120 FPS. The monitor on which the user selects
+an area determines the limit; another monitor's faster refresh rate does not
+raise it. Selection remains confined to one monitor per recording. If the
+capture portal cannot expose its refresh rate, the conservative cap is 60 FPS.
+Actual output dimensions and FPS are shown while recording. Capture and encoding
+can skip frames under load; the completion state reports skipped frames.
+
+The video target bitrate accepts whole values from 1 to 100 Mbps. It is an
+encoding target, not a promised file size: simple scenes can need fewer bits.
+MP4 prefers an available hardware H.264 encoder, falls back to software H.264,
+and finally MPEG-4. A codec must open successfully at the selected dimensions;
+the presence of a GPU or codec name alone is not treated as support.
+Streaming Qt Multimedia capture replaces repeated screenshots. High-resolution
+timestamps preserve real frames despite desktop timer jitter. The video editor retains
+the source timing and includes audio when trimming or changing playback speed.
+
+**Include system audio** captures the default playback device using
+[SoundCard output loopback](https://github.com/bastibe/SoundCard), at 48 kHz
+stereo, encoded as AAC. This includes applications on other monitors using that
+output. It never falls back to a microphone. Device selection is fixed for the
+recording; switch outputs before starting a new recording. An audio failure
+reports an error instead of silently producing a successful video without sound.
+Windows output loopback was verified locally with an audible test signal.
+Linux uses a PulseAudio/PipeWire monitor source and still requires live device
+verification. macOS system audio is currently unavailable and is disabled in
+the UI; silent screen recording remains available. Optional desktop packages
+are lazy imports and are not needed for base/headless startup.
+
## Where a shortcut appshot goes
| Destination | Running voice call | No voice call |
@@ -408,12 +463,37 @@ original any-order behaviour.
Only the instance that owns ambient duties (the default app, not the dev
instance) arms the shortcuts.
+## Capture startup latency
+
+Previously every area shortcut started Python, imported Qt and initialized
+the GUI before freezing the screens. The shutter effect also started its
+own process when the previous card had closed. Both startup costs sat on
+the interaction path and could show the operating system's busy cursor.
+
+The area picker now reuses its prepared runtime. The shutter runtime stays
+ready while Appshots and their effect are enabled. Neither preparation
+takes a screenshot or opens an overlay. Disabling Appshots releases the
+picker and the idle shutter runtime; active Computer-Use borders and open
+cards keep their existing ownership. Application shutdown reaps both helpers.
+Timeouts and interrupted requests discard the picker before a later request
+can use it, and a dead idle process is recreated on the next request.
+
+Windows verification on Python 3.11 measured about 399 ms from cold picker
+spawn to its visible-ready acknowledgement, versus 94–133 ms per selection
+in a prepared process. A separate native drag/toolbar/Enter check measured
+141–168 ms and verified no remaining selection windows after either pick.
+These are local renderer measurements, not a universal shortcut latency
+guarantee; configuration, privacy checks, display capture and image processing
+still take time. macOS and X11 share the lifecycle but were not tested live;
+Wayland, headless hosts and installations without PySide6 remain unavailable.
+
## Code
`jarvis/appshot/` — `service.py` (take and deliver), `store.py` (pending and
last appshot, memory only), `library.py` (the gallery's history on disk), `gesture.py` (both-Alt watcher), `hotkey.py`
(both shortcuts' lifecycle), `region.py` (area selection and its coordinate
-mapping), `picker/` (the area picker sidecar), `effect.py` (shutter hook),
+mapping), `picker_host.py` (prepared-process lifecycle), `picker/` (the area
+picker sidecar), `effect.py` (shutter hook),
`delivery.py` (voice calls). The corner card lives in the indicator sidecar
(`jarvis/cu/indicator/renderer.py`, `_CardWindow`); the editor is
`frontend/src/views/AppshotEditor.tsx`, hosted by
diff --git a/docs/igentic-workspaces.md b/docs/igentic-workspaces.md
index 53d1714c37..5acb0fb5a4 100644
--- a/docs/igentic-workspaces.md
+++ b/docs/igentic-workspaces.md
@@ -96,6 +96,26 @@ with a different `while_busy`); an accepted, queued or uncertain receipt is
returned instead of a second delivery. Claude Code records a mid-turn message as
a human `queued_command`; `context` shows it as a user message.
+### Failed turns and stopped agents
+
+A turn that ended in an error (Codex records `task_complete` with an `error`)
+shows the pane as `failed`, but its CLI is back at its prompt: a `send` delivers
+the next task as usual.
+
+An agent whose process is gone (`exited`, or `error` when it could not start)
+takes nothing. A `send` returns `not_accepted` with `stopped: true` and
+`restartable: true` and types nothing. `restart` (by call-sign or the resolved
+IDs) starts that same pane again: same `terminal_id`, account and launch picks,
+continuing its own conversation through the CLI's resume where one exists
+(`conversation: continued`), fresh otherwise. Its receipt reads `restarted`;
+nothing is typed. A pane whose agent is running is left alone and the receipt
+reads `already_running`, so a repeated or concurrent restart starts at most one
+process and never cuts a turn short. The refused send is then retried with the
+same `request_id` and prompt: its refusal proved nothing was typed, so the retry
+delivers once, and the accepted receipt answers every later retry. A restart is
+a logged `monitor` action like `interrupt` and `close`. Nothing restarts an agent
+on its own: a send never does it implicitly.
+
## Boundaries and verification
This is a T3 shared contract change. No new OS adapter, credential or model
diff --git a/docs/marketplace/plugin-auth-audit.md b/docs/marketplace/plugin-auth-audit.md
index 9a8b04c6db..c2b0ddf3cf 100644
--- a/docs/marketplace/plugin-auth-audit.md
+++ b/docs/marketplace/plugin-auth-audit.md
@@ -67,7 +67,7 @@ Twenty-six further hosted MCP services (Atlassian, Sentry, PayPal, Zapier and ot
| `salesforce` (Salesforce) | Connect entry observed; completion blocked | Not reached | Not verified | Not verified | Not run | Not completed | Windows UI only; other OSes not exercised | **BLOCKED** | Publisher OAuth client is missing; the real UI displays publisher setup pending. **Owner:** Project publisher. **Next:** Provision the official shared OAuth registration and any required review, then repeat the full clean browser journey. Observation: Connect Salesforce Browser sign-in is not available yet. Browser sign-in is pending publisher setup. No developer account or client registration is required from you. Use your own OAuth client (advanced) Cancel Continue |
| `granola` (Granola) | Connect entry observed; completion blocked | Provider page reached; authorization incomplete | Not verified | Not verified | Not run | Not completed | Windows UI only; other OSes not exercised | **BLOCKED** | Provider account sign-in has not been completed. **Owner:** Provider account holder. **Next:** Complete official sign-in and consent, then verify all seven stages with a real read-only plugin action. Observation: Official mcp-auth.granola.ai sign-in reached from real Connect; account sign-in required. |
| `zoom` (Zoom) | Connect entry observed; completion blocked | Not reached | Not verified | Not verified | Not run | Not completed | Windows UI only; other OSes not exercised | **BLOCKED** | Publisher OAuth client is missing; the real UI displays publisher setup pending. **Owner:** Project publisher. **Next:** Provision the official shared OAuth registration and any required review, then repeat the full clean browser journey. Observation: Connect Zoom Browser sign-in is not available yet. Browser sign-in is pending publisher setup. No developer account or client registration is required from you. Use your own OAuth client (advanced) Cancel Continue |
-| `figma` (Figma) | Connect entry observed; completion blocked | Not reached | Not verified | Not verified | Not run | Not completed | Windows UI only; other OSes not exercised | **BLOCKED** | Publisher OAuth client is not provisioned: the draft Figma OAuth app "Personal Jarvis" has the loopback redirect URL and the read scopes (`current_user:read`, `file_content:read`, `file_comments:read`, `folders:read`) since 2026-10-07, but no stored client secret. Figma's token endpoint requires the secret (HTTP Basic) and refreshes at `/v1/oauth/refresh`, which the catalog now uses. **Owner:** Project publisher. **Next:** Regenerate the secret, add a figma token-broker route and ship the client id, submit the app for public review, then repeat the full clean browser journey. Observation: Connect Figma Browser sign-in is not available yet. Browser sign-in is pending publisher setup. No developer account or client registration is required from you. Use your own OAuth client (advanced) Expert token alternative Cancel Continue |
+| `figma` (Figma) | Connect entry observed; completion blocked | Not reached | Not verified | Not verified | Not run | Not completed | Windows UI only; other OSes not exercised | **BLOCKED** | On 2026-10-08 the owner connected Figma through the shipped client and the deployed token broker; the stored grant carries a refresh token and the `GET /v1/me` read check passed. The app is still a draft that only its own team can authorize; reconnect and restart were not observed. **Owner:** Project publisher. **Next:** Submit the Figma app for public review, then repeat the clean browser journey with a non-team account. Observation: Connect Figma Browser sign-in is not available yet. Browser sign-in is pending publisher setup. No developer account or client registration is required from you. Use your own OAuth client (advanced) Expert token alternative Cancel Continue |
| `amd_gpu` (AMD GPU Status) | Connect entry observed; completion blocked | Not reached | Not verified | Not verified | Not run | Not completed | Supported Linux AMD host required | **BLOCKED** | Real UI reports unsupported OS: this connector needs a compatible Linux AMD host with AMD SMI. **Owner:** Hardware test operator. **Next:** Run the no-login connection and real telemetry read on a supported Linux AMD host, then verify lifecycle. Observation: Plugins Skills Plugins MCPs Plugins AMD GPU Status Unsupported on this device · Home & Devices Connect AMD GPU telemetry is unavailable on this operating system. This connector requires a compatible Linux AMD host with AMD SMI. Read AMD GPU utilization, temperature and driver data through AMD SMI... |
| `shopify` (Shopify) | Connect entry observed; card renders with logo, Connect fails honestly | Provider page not reached (wrong flow, now corrected) | Not verified | Not verified | Not run | Not completed | Windows UI only; other OSes not exercised | **BLOCKED** | The first configuration used Dynamic Client Registration, but Shopify publishes no DCR registration_endpoint, so Connect failed before any browser page opened ("provider request failed"). The entry now uses Authorization Code + PKCE loopback against the publisher-shared app (loopback http://127.0.0.1:3130/oauth/callback, RFC 8707 resource https://setup.shopify.com/mcp, comma-joined read scopes). The shared app is not provisioned yet, so the dialog reports browser sign-in pending; no store-owner sign-in has been completed either. **Owner:** Project publisher. **Next:** Register the official shared Shopify app (redirect URI above, read scopes for products, orders, customers and discounts), provision publisher_shopify_oauth_client_id/secret, then repeat the full clean browser journey with a real read-only plugin action. Until then an expert can connect with their own Shopify app client through the dialog's advanced option. Observation: Connect on the Shopify card returned the generic failure dialog on 2026-09-21; live metadata at setup.shopify.com serves OAuth discovery but no registration endpoint. |
diff --git a/docs/marketplace/plugin-e2e-audit.json b/docs/marketplace/plugin-e2e-audit.json
index f3686d45bf..c826b2fb28 100644
--- a/docs/marketplace/plugin-e2e-audit.json
+++ b/docs/marketplace/plugin-e2e-audit.json
@@ -2265,9 +2265,9 @@
}
},
"blocker": {
- "reason": "Publisher OAuth client is not provisioned. The draft Figma OAuth app \"Personal Jarvis\" (client id RkG83E4jPtFsiRQZBku3SD) has the loopback redirect URL and the four read scopes since 2026-10-07, but its client secret is not stored anywhere; Figma's token endpoint requires the secret (HTTP Basic), so a shipped public client also needs a token broker.",
+ "reason": "On 2026-10-08 the owner connected Figma through the shipped client and the deployed token broker (stored grant bound to client QnSTQR5QK5hAOjxBmp6sIY with a refresh token; GET /v1/me read check passed). The app is still a draft that only its own team can authorize, and reconnect and restart were not observed.",
"owner": "Project publisher",
- "next_action": "Regenerate the client secret in the Figma developer console, add a figma route to the token broker and ship the client id, submit the app for public review, then repeat the full clean browser journey with a real read-only action.",
+ "next_action": "Submit the Figma app for public review (Publish tab, steps 2-5), then repeat the clean browser journey with a non-team account and observe reconnect and restart.",
"evidence": "Connect Figma\n\nBrowser sign-in is not available yet.\n\nBrowser sign-in is pending publisher setup. No developer account or client registration is required from you.\n\nUse your own OAuth client (advanced)\nExpert token alternative\nCancel\nContinue"
}
},
diff --git a/docs/os-parity.md b/docs/os-parity.md
index 6958c21aea..b9bfea286f 100644
--- a/docs/os-parity.md
+++ b/docs/os-parity.md
@@ -236,6 +236,92 @@ Protocol references: [AsyncSSH port forwarding](https://asyncssh.readthedocs.io/
[Kimi MCP configuration](https://github.com/MoonshotAI/kimi-cli/blob/main/README.md),
[Cursor MCP](https://docs.cursor.com/en/cli/mcp).
+## AppShot capture pixel integrity (2026-10-04, T2)
+
+Packed RGB capture frames are mapped read-only, copied with their actual row
+stride, and unmapped before entering the encoder queue. They do not go through
+Qt's GPU presentation conversion. This fixes intermittent striped/color-shifted
+frames seen in native Windows area recordings. Zero-filled startup buffers are
+skipped; Windows opaque-desktop sources also reject uninitialized alpha values.
+Valid opaque black frames remain recordable. The first usable video frame starts
+at timestamp zero, avoiding an empty startup interval in MP4 playback.
+
+A separate visible Windows test application supplied moving content to the real
+picker/capture/encoder flow. A 15.5-second, 1918-by-948 capture produced 947
+decoded frames with no color-patch corruption, a zero first timestamp, and 226
+distinct positions of the moving marker. Regression tests cover row padding,
+buffer reuse, map failure/cleanup, rotation, mirror, startup transparency and
+opaque black frames. YUV portal sources retain Qt conversion with a detached
+copy; macOS/Linux native capture remains unverified. This evidence supersedes
+the earlier frame-count-only check for visual recording correctness.
+
+## AppShot recording quality and output audio (2026-10-04, T2)
+
+Recording presets default to 1080p at 60 FPS and a 12 Mbps video target.
+Resolution choices through 4K plus native preserve source aspect ratio without
+upscaling; 30/60/120 FPS choices are capped by the selected monitor's refresh
+rate. Missing portal refresh information caps at 60 FPS. The same validation,
+size calculation, MP4 encoding and settings routes apply on every OS. Qt
+Multimedia supplies streaming frames after explicit selection; encoder startup
+tries available hardware H.264 backends, then software H.264/MPEG-4. Missing
+drivers or unsupported dimensions fall back before any packet is written.
+Windows full-screen capture from a 4K/60 Hz monitor to Full HD measured about
+59.6 captured frames/second, with a 120 FPS request correctly capped at 60.
+High-resolution timestamps avoid discarding distinct frames due to coarse
+Windows clock ticks. Native macOS/Linux performance is still unverified.
+
+System audio is opt-in and captures only the default playback device's loopback.
+Windows uses WASAPI through the optional SoundCard desktop dependency. A real
+Windows capture produced an MP4 with synchronized video and audible AAC stereo
+at 48 kHz; deterministic tests cover silent recording, rescaling, all three
+frame rates, invalid settings, and audio retention through trim/speed edits.
+Linux uses PulseAudio/PipeWire monitor sources; native Linux audio remains
+unverified. macOS audio loopback is unavailable and is visibly disabled, while
+silent recording keeps working. No fallback ever opens a microphone. Headless
+installs need no audio backend and keep their existing capture-unavailable state.
+
+## AppShot recording startup latency (2026-10-03, T2)
+
+The primary desktop instance prepares one idle recording worker after shortcut
+arming when AppShots are enabled. Standby loads Qt and the encoder libraries;
+it creates no capture session, windows, video files, or permission prompts and
+takes no pixels. A user start still checks current permissions before sending
+the selection command. After a recording finishes, a fresh worker is prepared.
+Disabling AppShots releases standby; application shutdown also finalizes any
+active recording and reaps its process.
+
+Windows native checks measured about 150–260 ms from a prepared start to the
+visible picker, versus about 1.7 seconds for a cold process on the same desktop.
+Two consecutive selected-area recordings produced playable MP4s and reused their
+prepared workers. Timeout, cancellation during spawn, dead-worker recovery,
+permission revocation and shutdown have focused regression coverage. Standby
+was checked to create neither widgets nor screen grabs. macOS and Linux use the
+same ownership protocol; their native latency remains unmeasured, and a first
+permission dialog or Wayland portal selection still requires user interaction.
+
+## AppShot recording frame and completion card (2026-10-03, T2)
+
+The selected recording area keeps a click-through border until recording stops.
+Elapsed time and a stop button sit below it; the toolbar moves above the area or
+inside the work area when the selection reaches the screen edges. A finalized
+video uses the existing screenshot flight animation to enter a bottom-right
+preview card. Its poster shows the first video frame with a prominent play button
+and a video/duration label. The card can play the local file, save a separate copy
+to Downloads, or drag the MP4 to another application. Screenshot and video previews share the
+indicator process and its shutdown/capture-suppression lifecycle.
+
+Windows area and 4K full-screen recordings were verified with the desktop Python
+runtime: persistent frame, elapsed time, stop-button completion, playable MP4,
+and a visible corner card. Both the frame and toolbar successfully enabled
+native capture exclusion. A separate explicitly capturable visual pass verified
+their geometry and appearance without confusing that pass with clean video output.
+
+macOS and X11 use the same Qt positioning code but remain unverified on real
+desktops. Native capture exclusion is Windows-specific; controls are placed outside
+the recorded area when space permits. Wayland portals do not expose global source
+coordinates: the normal stop window remains available there, while a desktop-wide
+boundary and corner flight retain the indicator's existing Wayland limitation.
+
## Live computer control by the session's own model (2026-10-03, T3)
Voice sessions operate the screen through `computer` (ADR-0039). The same
@@ -287,6 +373,17 @@ input follows the owned focus. Offscreen positions from older builds recover
onto a monitor without activating Chrome. Hooks and captures have bounded
cleanup, and stale geometry rejects input.
+The owned Windows browser and its native popups use zero desktop opacity and
+non-activating tool-window styles while their original pixels remain available
+to Windows Graphics Capture. This prevents a separate window or edge strip
+from appearing outside the Jarvis preview. Regular Chrome keeps rendering while
+occluded so the embedded view can update. This applies only to Jarvis-owned
+windows; attached user Chrome windows and the page streams on other operating
+systems are unchanged. Fake Win32 tests cover ownership, popup visibility,
+input, and failure handling; a live Windows check covers invisible window
+startup and continued capture. macOS and Linux native window capture remains
+unverified.
+
Manual website sign-in stays inside the embedded viewer. Vision-capable agents
with unrestricted website access can use approved native window controls;
domain-restricted agents keep page-only tools because a complete Chrome frame
@@ -1147,6 +1244,8 @@ implementations, not stubs.
| On-demand Screen Context | One-shot capture is wired into the production brain on Windows, macOS, and Linux/X11; UIA/AX/AT-SPI text is source-filtered, the indicator precedes capture, and Wayland/headless/missing grants refuse honestly |
| Appshots (front-window capture on a shortcut, button or request) | Capture, privacy and delivery are OS-neutral (Screen Context engine, `jarvis/appshot`). The two-sided shortcuts (both Alt, both Shift, both Ctrl) read key state per OS: Windows `GetAsyncKeyState`, macOS `CGEventSourceKeyState` (Input Monitoring grant), Linux/X11 `XQueryKeymap`; Wayland/headless report it unavailable on the Appshots page. The flash and the area picker (both Shift keys, drag a rectangle, then mark it up with its toolbar) are PySide6 overlays where one can run; the markings are burnt into the capture with Pillow on every OS (`jarvis/appshot/markup.py`); Wayland/headless report the area picker unavailable. The appshot editor is web code and behaves the same everywhere; its Copy, and every shortcut or button appshot, writes the picture natively in several formats (`jarvis/platform/clipboard_image.py`): Windows the registered PNG format + `CF_DIB`, macOS `public.png` + `public.tiff` via `osascript -l JavaScript`, Linux `wl-copy` or `xclip`; the editor falls back to the browser clipboard without them. Save goes to `~/Downloads` on every OS. Verified live on Windows only; see `docs/appshots.md` |
| Voice / audio (capture, playback, VAD, wake, STT, TTS, realtime) | Clean; a headless host has no local voice stack, so the browser that presses Start holds the call over `/ws/audio` (realtime, or the classic STT → brain → TTS bridge while `[browser_voice].enabled`, the default — issue #399); WASAPI logic is inert-by-data off Windows |
+| Appshots (front-window capture on a shortcut, button or request) | Capture, privacy and delivery are OS-neutral (Screen Context engine, `jarvis/appshot`). The two-sided shortcuts (both Alt, both Shift, both Ctrl) read key state per OS: Windows `GetAsyncKeyState`, macOS `CGEventSourceKeyState` (Input Monitoring grant), Linux/X11 `XQueryKeymap`; Wayland/headless report it unavailable on the Appshots page. The flash and the area picker (both Shift keys, drag a rectangle, then mark it up with its toolbar) are PySide6 overlays where one can run. The main instance prepares their runtimes after wake-ready without capturing pixels; selections reuse Qt but release all screen images and windows after completion. The markings are burnt into the capture with Pillow on every OS (`jarvis/appshot/markup.py`); Wayland/headless report the area picker unavailable. The appshot editor is web code and behaves the same everywhere; its Copy, and every shortcut or button appshot, writes the picture natively in several formats (`jarvis/platform/clipboard_image.py`): Windows the registered PNG format + `CF_DIB`, macOS `public.png` + `public.tiff` via `osascript -l JavaScript`, Linux `wl-copy` or `xclip`; the editor falls back to the browser clipboard without them. Save goes to `~/Downloads` on every OS. Verified live on Windows only; see `docs/appshots.md` |
+| Voice / audio (capture, playback, VAD, wake, STT, TTS, realtime) | Clean; headless disables voice honestly; WASAPI logic is inert-by-data off Windows |
| Core (launcher, config, keyring, restart, autostart, tray, elevation, paths) | Clean; per-OS autostart (Registry / LaunchAgent / XDG `.desktop`), keyring falls back to a 0600 file on headless hosts |
| Data / agents (wiki, contacts, telephony, sessions, missions, skills, self-mod, channels, MCP) | Clean; mission workers run on POSIX with a real process-group reaper |
| Agent society hands (own shell, browser via browser-use, learned skills) | Shell: local subprocess in the agent's workspace on every OS (Git Bash/PowerShell/bash/sh pick as the chat's folder tools), no container by decision. Browser: browser-use lives in a managed venv under the data dir (its pins collide with the app's), installed on demand — `uv`/`venv`, a 3.11–3.13 interpreter preferred, Chromium downloaded once; headless runs need no display, so a headless Linux box runs agents' browsers; the headed login session needs a display (409 without one is the follow-up); attach mode needs a running Chrome with `--remote-debugging-port`. Learning is pure files + the brain, OS-neutral |
@@ -1636,6 +1735,25 @@ sessions and releases it when the worker exits. This prevents the capture
library's cached WinRT factories from outliving their apartment during manual
handover. Each window still stops its capture and event threads on close.
+The shared turn-host client accepts a replacement reader's output only after
+the host acknowledges attachment. Earlier live frames remain in the host's
+replay buffer, so a fast process cannot displace the first replayed line.
+Failed attachments release their local reader and can retry with the host's
+authoritative process metadata. Socket tests exercise both paths without
+depending on Windows, macOS, or Linux scheduling speed.
+
+A stopped Windows popup capture discards its stale pixels and retries once while
+the same owned popup remains visible. Parked owned windows remain fully
+transparent and mouse-transparent. They clear the tool-window, app-window,
+and no-activate extended styles: native capture rejects the no-activate style
+when the app-window style is absent. Positioning still uses SWP_NOACTIVATE
+and keeps the owned window behind the application, including popup recovery.
+Input waits for the replacement's first
+frame. A repeated failure, failed restart, or unfinished capture cleanup reports
+an error instead of retaining an unusable preview or starting overlapping captures.
+Closing the popup resets its retry allowance. Regression tests use native-window
+fakes; this recovery does not establish a successful live Chrome sync or sign-in.
+
macOS and Linux keep their existing managed browser behavior. They do not expose
this sign-in capability because the current native capture/input implementation
is Windows-only. The module imports without Win32 dependencies on those systems;
@@ -1726,3 +1844,19 @@ Agent notebook migration explicitly normalizes the final `USER.md` and
`MEMORY.md` filename spelling on case-insensitive filesystems. The migration
journal and backups continue to preserve the original content on every OS.
The inherited-lock and migration recovery contracts run in the OS test shards.
+
+## Independent agent server and clients
+
+The opt-in independent server uses the existing headless backend on Windows,
+macOS and Linux. Its lifetime no longer depends on a desktop client, an idle
+timer or a desktop handover. Windows runs it in the user session rather than
+as a SYSTEM service. The client uses pywebview where installed; the same server
+also serves ordinary browsers. Remote origins require HTTPS or a local tunnel.
+
+Windows evidence covers the real headless process, readiness and explicit stop,
+the startup budget, and scripted disconnect/reconnect contracts for Jarvis,
+Hermes and OpenClaw. The shared lifecycle and configuration tests are portable;
+native macOS/GTK client windows and live remote-host/provider execution have
+not been verified by these checks. Device capture, wake word and overlays keep
+their existing device and permission requirements; opening a client does not
+grant the server access to that client's desktop.
diff --git a/docs/pets.md b/docs/pets.md
index 7d4a947560..8d5930407b 100644
--- a/docs/pets.md
+++ b/docs/pets.md
@@ -33,7 +33,11 @@ one click.
three, each a light-blue sky with soft white clouds in it): start a conversation, or hang up the running one. The
strokes stand still and dimmed at rest, follow the live audio level while
listening, dictating or talking, and carry a travelling highlight while
- thinking or transcribing;
+ thinking or transcribing. While a realtime call is still connecting
+ (`SystemStateChanged(CONNECTING)`) the three strokes bend into the arcs
+ of one ring and spin as the loading loop (a live voice lengthens the
+ arcs); once the provider takes the call they flash green, unbend and
+ slide back into the listening row. The Jarvis Bar shows the same loop;
- speaker: silence the assistant's voice for this session (TTS volume 0,
mirrored from `VoiceSpeakerMuteChanged`).
- **The thought bubble** above the pet's head while Jarvis thinks, drawn in
@@ -53,6 +57,17 @@ one click.
character (`docs/agent-society/office-map.md`, *Jarvis is the person's
pet*). A pet the person drew becomes a voxel figure cut from its idle
frames.
+- **Worn by an agent.** In an agent's look dialog (*Companion → Look →
+ Pet*) any pet from My Pets can replace the agent's shape. It is stored as
+ `companion.pet` in the agent's avatar JSON (`jarvis/society/companion.py`,
+ mirrored by `components/society/companion/appearance.ts`) and drawn by
+ `CompanionMark` wherever the agent appears: roster, chats, mentions, the
+ level hall, the bot stage (each scene plays a pet row, `PET_STATE` in
+ `botStage/scenes.ts`), the live line of its routine traces, and as its 3D
+ companion in the Verse with the pet's own model and gait. The agent's
+ colour stays for its chat bubbles. A pet that does not exist on this
+ machine (deleted, or drawn on another one) shows the shape again; a shared
+ agent template keeps a built-in pet and drops a drawn one.
The pet stays on screen while Jarvis is idle. The global shortcut
(`[trigger] hotkey_pet_toggle`, default `alt+win+p`) hides it or brings it
@@ -148,7 +163,7 @@ equal.
|---|---|
| `idle` | `SystemStateChanged(IDLE)`, no session |
| `listening` | wake word, `VoiceSessionStarted`, `SystemStateChanged(LISTENING)`, dictation |
-| `thinking` | `SystemStateChanged(THINKING)` |
+| `thinking` | `SystemStateChanged(THINKING)`; also `SystemStateChanged(CONNECTING)` while a realtime call connects (the pet nods once it is connected) |
| `talking` | `AudioOutFirst`, `SystemStateChanged(SPEAKING)` with audible output |
| `success` (one-shot, 1.5 s) | `SpeechSpoken.spoken_kind` in `action_done`/`completion`, `JarvisAgentBackgroundCompleted(success=True)`, `ActionExecuted(success=True)` while no turn is running |
| `error` (one-shot, 2 s) | `ErrorOccurred(recoverable=False)`, `SpeechSpoken.spoken_kind` in `timeout`/`unavailable`/`stt_unavailable`, `VoiceSessionEnded(hangup_reason="error")`, `ActionExecuted(success=False)` while no turn is running |
@@ -352,6 +367,6 @@ The pet must cost next to nothing while idle:
- the window repaints only when the frame key changes (frame index, plus the
level bucket while talking), and the timer sleeps until the next frame
boundary: 6 fps idle, 3 fps asleep, 8–10 fps active, nothing while hidden;
-- the control strip's indicator animates only while listening, talking or thinking.
+- the control strip's indicator animates only while connecting, listening, talking or thinking.
Target: an idle pet adds less than 1 % of one CPU core to the app process.
diff --git a/jarvis.toml.example b/jarvis.toml.example
index 231a176219..363e5c3d86 100644
--- a/jarvis.toml.example
+++ b/jarvis.toml.example
@@ -155,6 +155,7 @@ base_url = "http://127.0.0.1:8080"
# ── Safety: anti-confirmation-fatigue whitelist + hard blacklist ──
[safety]
+approval_mode = "bypass"
default_tier = "safe"
always_confirm_tiers = ["ask"]
always_block_tiers = ["block"]
diff --git a/jarvis/__main__.py b/jarvis/__main__.py
index dc2a11c165..5469f622b3 100644
--- a/jarvis/__main__.py
+++ b/jarvis/__main__.py
@@ -50,7 +50,7 @@ if len(sys.argv) >= 2 and sys.argv[1] == "--relauncher":
# The background agent service of a frozen build, same reason: no ``-m`` to
# start ``jarvis.ui.web.launcher --background-service`` with
# (see jarvis/core/background_service.py).
-if len(sys.argv) >= 2 and sys.argv[1] == "--background-service":
+if len(sys.argv) >= 2 and sys.argv[1] in {"--background-service", "--persistent-server", "--connect"}:
from jarvis.ui.web.launcher import main as _launcher_main
raise SystemExit(_launcher_main(sys.argv[1:]))
diff --git a/jarvis/agent_chat/credential_requests.py b/jarvis/agent_chat/credential_requests.py
new file mode 100644
index 0000000000..50c033d271
--- /dev/null
+++ b/jarvis/agent_chat/credential_requests.py
@@ -0,0 +1,287 @@
+"""User-owned secure fields whose values never enter the chat event stream.
+
+A field stays open across turn completion, interruption and reconnect until
+its owner saves a validated value or explicitly cancels. Only secret-free
+metadata is persisted, allowing the service to rebuild fields after a restart.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import inspect
+import uuid
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass
+from typing import Any, Final
+
+from jarvis.agent_chat.events import make_event
+
+# Retained for callers that used the old argument. It no longer expires fields.
+CREDENTIAL_TIMEOUT_S: Final[float] = 0.0
+MAX_REQUESTS_PER_TURN: Final[int] = 3
+MAX_PENDING_PER_SESSION: Final[int] = 12
+MAX_PENDING_REQUESTS: Final[int] = 256
+MAX_CLOSED_REQUESTS: Final[int] = 256
+SAVED: Final[str] = "saved"
+DECLINED: Final[str] = "declined"
+TIMEOUT: Final[str] = "timeout" # Historical receipts only.
+CANCELLED: Final[str] = "cancelled" # Historical receipts only.
+
+Save = Callable[[str], Awaitable[None] | None]
+Emit = Callable[[str, dict[str, Any]], Awaitable[None]]
+RunningTurn = Callable[[str], str | None]
+History = Callable[[str], list[dict[str, Any]]]
+RestoreSave = Callable[[str, "CredentialSpec"], Save | None]
+
+
+class TooManyCredentialRequests(RuntimeError):
+ """The turn or session already has enough credential fields."""
+
+
+class CredentialRequestBusy(RuntimeError):
+ """Validation or storage is running; cancellation must not race its commit."""
+
+
+def _text(value: Any, limit: int) -> str:
+ return " ".join(str(value or "").split())[:limit]
+
+
+@dataclass(frozen=True, slots=True)
+class CredentialSpec:
+ """What the field asks for. ``env`` is validated by the asking tool."""
+
+ env: str
+ label: str
+ description: str = ""
+ placeholder: str = ""
+ replace: bool = False
+
+ @classmethod
+ def build(
+ cls, env: str, label: Any, description: Any = "", placeholder: Any = "",
+ *, replace: bool = False,
+ ) -> CredentialSpec:
+ name = _text(label, 80)
+ if not name:
+ raise ValueError("label is required: say what the credential is, e.g. 'GitHub token'")
+ return cls(env, name, _text(description, 400), _text(placeholder, 80), replace)
+
+ def to_payload(self) -> dict[str, Any]:
+ return {
+ "env": self.env, "label": self.label, "description": self.description,
+ "placeholder": self.placeholder, "replace": self.replace,
+ }
+
+
+class _OpenCredential:
+ def __init__(
+ self, session_id: str, turn_id: str, spec: CredentialSpec, save: Save,
+ ) -> None:
+ self.session_id = session_id
+ self.turn_id = turn_id
+ self.spec = spec
+ self.save = save
+ self.result: asyncio.Future[str] = asyncio.get_running_loop().create_future()
+ self.saving: asyncio.Task[bool] | None = None
+
+ @property
+ def done(self) -> bool:
+ return self.result.done()
+
+
+class CredentialRequests:
+ """Bounded pending fields, independent of the lifetime of any agent turn."""
+
+ def __init__(
+ self, emit: Emit, running_turn: RunningTurn, *, history: History | None = None,
+ restore_save: RestoreSave | None = None,
+ ) -> None:
+ self._emit = emit
+ self._running_turn = running_turn
+ self._history = history
+ self._restore_save = restore_save
+ self._open: dict[str, _OpenCredential] = {}
+ self._asked: dict[tuple[str, str], int] = {}
+ self._restored: set[str] = set()
+ self._restore_lock = asyncio.Lock()
+
+ async def restore(self, session_id: str) -> None:
+ if session_id in self._restored or self._history is None or self._restore_save is None:
+ return
+ async with self._restore_lock:
+ if session_id in self._restored:
+ return
+ events = await asyncio.to_thread(self._history, session_id)
+ self._restore_events(session_id, events)
+ self._restored.add(session_id)
+
+ def _restore_events(self, session_id: str, events: list[dict[str, Any]]) -> None:
+ pending: dict[str, dict[str, Any]] = {}
+ statuses: dict[str, str] = {}
+ for event in events:
+ payload = event.get("payload") or {}
+ rid = str(payload.get("request_id") or "")
+ if not rid:
+ continue
+ if event.get("kind") == "credential_required":
+ pending[rid] = payload
+ elif event.get("kind") == "credential_resolved":
+ statuses[rid] = str(payload.get("status") or CANCELLED)
+ for rid, payload in pending.items():
+ if rid in self._open:
+ continue
+ spec = CredentialSpec.build(
+ str(payload.get("env") or ""), payload.get("label"),
+ payload.get("description"), payload.get("placeholder"),
+ replace=bool(payload.get("replace")),
+ )
+ assert self._restore_save is not None
+ save = self._restore_save(session_id, spec)
+ if save is not None:
+ card = self._open[rid] = _OpenCredential(
+ session_id, str(payload.get("turn_id") or ""), spec, save,
+ )
+ if rid in statuses:
+ card.result.set_result(statuses[rid])
+
+ async def open(
+ self, session_id: str, spec: CredentialSpec, save: Save, *, asker: str = "",
+ timeout_s: float = CREDENTIAL_TIMEOUT_S,
+ ) -> str:
+ """Show a field; legacy ``timeout_s`` deliberately has no closing effect."""
+ await self.restore(session_id)
+ turn_id = self._running_turn(session_id)
+ if not turn_id:
+ raise RuntimeError("this chat has no running turn to ask in")
+ # Repeated asks for the same variable reuse the person's existing field.
+ for rid, card in self._open.items():
+ if card.session_id == session_id and not card.done and card.spec.env == spec.env:
+ return rid
+ key = (session_id, turn_id)
+ if self._asked.get(key, 0) >= MAX_REQUESTS_PER_TURN:
+ raise TooManyCredentialRequests("this turn already asked for three credentials")
+ if len(self.pending(session_id)) >= MAX_PENDING_PER_SESSION or sum(
+ not card.done for card in self._open.values()
+ ) >= MAX_PENDING_REQUESTS:
+ raise TooManyCredentialRequests("complete or cancel existing credential fields first")
+ rid = uuid.uuid4().hex
+ self._open[rid] = _OpenCredential(session_id, turn_id, spec, save)
+ try:
+ await self._emit(session_id, make_event("credential_required", {
+ "turn_id": turn_id, "request_id": rid, "asker": asker,
+ **spec.to_payload(), "timeout_s": 0, "expires_ms": 0,
+ }))
+ except BaseException:
+ self._open.pop(rid, None)
+ raise
+ self._asked[key] = self._asked.get(key, 0) + 1
+ return rid
+
+ async def wait(
+ self, session_id: str, request_id: str, timeout_s: float | None = None,
+ ) -> str | None:
+ await self.restore(session_id)
+ card = self._card(session_id, request_id)
+ try:
+ return await asyncio.wait_for(asyncio.shield(card.result), timeout_s)
+ except TimeoutError: # Only the caller's polling slice ends.
+ return None
+
+ def spec(self, session_id: str, request_id: str) -> CredentialSpec:
+ return self._card(session_id, request_id).spec
+
+ def pending(self, session_id: str) -> list[str]:
+ return [rid for rid, c in self._open.items() if c.session_id == session_id and not c.done]
+
+ async def submit(self, session_id: str, request_id: str, value: str) -> bool:
+ await self.restore(session_id)
+ card = self._open.get(request_id)
+ if card is None or card.session_id != session_id:
+ return False
+ if card.done:
+ return card.result.result() == SAVED
+ if card.saving is not None:
+ raise CredentialRequestBusy("The credential is still being checked. Please wait.")
+ # A dropped HTTP connection cannot interrupt a storage transaction.
+ card.saving = asyncio.create_task(self._save(request_id, card, value))
+ card.saving.add_done_callback(self._observe_save)
+ return await asyncio.shield(card.saving)
+
+ @staticmethod
+ def _observe_save(task: asyncio.Task[bool]) -> None:
+ if not task.cancelled():
+ task.exception() # Retrieve failures even if the HTTP caller disconnected.
+
+ async def _save(self, request_id: str, card: _OpenCredential, value: str) -> bool:
+ try:
+ outcome = card.save(value)
+ if inspect.isawaitable(outcome):
+ await outcome
+ await self._close(request_id, SAVED)
+ return True
+ finally:
+ card.saving = None
+
+ async def decline(self, session_id: str, request_id: str) -> bool:
+ await self.restore(session_id)
+ card = self._open.get(request_id)
+ if card is None or card.session_id != session_id:
+ return False
+ if card.done:
+ return card.result.result() == DECLINED
+ if card.saving is not None:
+ raise CredentialRequestBusy("The credential is still being checked. Please wait.")
+ card.saving = asyncio.create_task(self._decline(request_id, card))
+ card.saving.add_done_callback(self._observe_save)
+ return await asyncio.shield(card.saving)
+
+ async def _decline(self, request_id: str, card: _OpenCredential) -> bool:
+ try:
+ await self._close(request_id, DECLINED)
+ return True
+ finally:
+ card.saving = None
+
+ def finish_turn(self, session_id: str) -> None:
+ """Release turn accounting without closing the person's pending fields."""
+ for key in [key for key in self._asked if key[0] == session_id]:
+ self._asked.pop(key, None)
+ closed = [rid for rid, card in self._open.items() if card.done]
+ for rid in closed[:-MAX_CLOSED_REQUESTS]:
+ self._open.pop(rid, None)
+
+ async def discard_session(self, session_id: str) -> None:
+ """Explicit session deletion waits for storage before forgetting fields."""
+ saves = [
+ card.saving for card in self._open.values()
+ if card.session_id == session_id and card.saving is not None
+ ]
+ if saves:
+ # Save errors have already reached their HTTP caller; deletion still
+ # removes their now-unneeded metadata after all writes have stopped.
+ await asyncio.gather(*(asyncio.shield(task) for task in saves), return_exceptions=True)
+ for rid, card in list(self._open.items()):
+ if card.session_id == session_id:
+ if not card.done:
+ card.result.set_result(CANCELLED)
+ self._open.pop(rid, None)
+ self._restored.discard(session_id)
+ self.finish_turn(session_id)
+
+ def _card(self, session_id: str, request_id: str) -> _OpenCredential:
+ card = self._open.get(request_id)
+ if card is None or card.session_id != session_id:
+ raise KeyError(request_id)
+ return card
+
+ async def _close(self, request_id: str, status: str) -> None:
+ card = self._open.get(request_id)
+ if card is None or card.done:
+ return
+ # Persist before reporting success. A failed receipt remains retryable,
+ # so a reload cannot resurrect a field that was reported closed.
+ await self._emit(card.session_id, make_event("credential_resolved", {
+ "turn_id": card.turn_id, "request_id": request_id,
+ "env": card.spec.env, "status": status,
+ }))
+ card.result.set_result(status)
diff --git a/jarvis/agent_chat/events.py b/jarvis/agent_chat/events.py
index 045e66ad19..0f33223626 100644
--- a/jarvis/agent_chat/events.py
+++ b/jarvis/agent_chat/events.py
@@ -34,6 +34,11 @@ Kinds (``payload`` keys in brackets):
whose answers go to the agent as the next message
``question_progress`` [turn_id, question_id, answers]
``question_resolved`` [turn_id, question_id, answers]
+``credential_required`` [turn_id, request_id, asker, env, label, description, placeholder,
+ replace, expires_ms] — a secure field for a secret the agent
+ needs (credential_requests.py); the value never appears in any event
+``credential_resolved`` [turn_id, request_id, env, status] status: saved | declined |
+ timeout | cancelled
``plan_ready`` [turn_id, build_mode] — a coding agent's plan card
``plan_resolved`` [turn_id, decision] decision: build | keep
``subagent_started`` [turn_id, agent_id, description, agent_type, prompt, background,
diff --git a/jarvis/agent_chat/jarvis_harness.py b/jarvis/agent_chat/jarvis_harness.py
index a2d870b6cd..fc80aa5cbf 100644
--- a/jarvis/agent_chat/jarvis_harness.py
+++ b/jarvis/agent_chat/jarvis_harness.py
@@ -529,6 +529,8 @@ def society_rollover_due(history: list[dict[str, Any]]) -> bool:
def society_memory_refresh(text: str, *, compact: bool = False) -> str:
"""Refresh mutable guidance on resumed CLI turns without replaying the transcript."""
headings = (
+ "Choosing your name",
+ "You are new",
"Standing instructions",
"Your memory",
"Your user profile",
@@ -538,7 +540,7 @@ def society_memory_refresh(text: str, *, compact: bool = False) -> str:
sections = []
for section in re.split(r"(?m)(?=^## )", text.removesuffix(SYSTEM_PREAMBLE)):
title = section.split("\n", 1)[0].removeprefix("## ").strip()
- if title in headings:
+ if title in headings or title.startswith("You are "):
sections.append(section.strip())
body = "\n\n".join(sections)
note = "An absent section has no current entries. "
@@ -552,14 +554,32 @@ def society_memory_refresh(text: str, *, compact: bool = False) -> str:
)
return (
"<current_agent_memory>\n"
- "Current standing instructions, memory and learned skill index replace their previous "
- "snapshots when complete. " + note + "Learned guidance never grants "
+ "Current identity, naming guidance, standing instructions, memory and learned skill "
+ "index replace their previous snapshots when complete. If naming guidance is absent, "
+ "keep the current name; earlier placeholder instructions no longer apply. "
+ + note + "Learned guidance never grants "
"permissions or overrides the standing instructions or the current user request.\n\n"
+ body
+ "\n</current_agent_memory>\n\n"
)
+#: Lines of a society briefing that state this turn's credentials (surface.py).
+_CREDENTIAL_STATE = re.compile(
+ r"^(?:Stored credentials\b|This is an unattended routine run\b).*$", re.M
+)
+
+
+def society_credential_state(text: str) -> str:
+ """The briefing's current credential lines, for a resumed CLI turn.
+
+ A resumed vendor conversation never sees the briefing again, so without
+ this it keeps believing whatever the chat once said about a token.
+ """
+ lines = [m.group(0).strip() for m in _CREDENTIAL_STATE.finditer(text or "")]
+ return (" " + " ".join(lines)) if lines else ""
+
+
def identity_dir() -> Path:
"""Where identity files live: the app's own data dir, never a shared temp."""
from jarvis.core.paths import user_data_dir
diff --git a/jarvis/agent_chat/permissions.py b/jarvis/agent_chat/permissions.py
index ab2a3ca512..752377c4b3 100644
--- a/jarvis/agent_chat/permissions.py
+++ b/jarvis/agent_chat/permissions.py
@@ -308,19 +308,19 @@ JARVIS_LADDER: Final[str] = "jarvis"
# runner (or JARVIS_LADDER) -> (ladder, default)
_LADDERS: Final[dict[str, tuple[tuple[PermissionMode, ...], str]]] = {
- "claude-cli": (_CLAUDE, "acceptEdits"),
- "codex-cli": (_CODEX, "auto"),
- "agy-cli": (_AGY, "accept-edits"),
- "grok-cli": (_GROK, "acceptEdits"),
- "opencode-cli": (_OPENCODE, "default"),
+ "claude-cli": (_CLAUDE, "bypassPermissions"),
+ "codex-cli": (_CODEX, "full-access"),
+ "agy-cli": (_AGY, "skip-permissions"),
+ "grok-cli": (_GROK, "bypassPermissions"),
+ "opencode-cli": (_OPENCODE, "auto"),
"kimi-cli": (_KIMI, "auto"),
# GLM Coding Plan drives the Claude Code binary, so it takes Claude
# Code's own words.
- "glm-cli": (_CLAUDE, "acceptEdits"),
+ "glm-cli": (_CLAUDE, "bypassPermissions"),
"dsh-cli": (_DSH, "auto"),
"cursor-cli": (_CURSOR, "auto"),
- "api": (_API, "ask"),
- JARVIS_LADDER: (_JARVIS, "ask"),
+ "api": (_API, "auto"),
+ JARVIS_LADDER: (_JARVIS, "bypass"),
"society": (_SOCIETY, "bypass"),
}
diff --git a/jarvis/agent_chat/runner_acp.py b/jarvis/agent_chat/runner_acp.py
index 4d084d26e7..5bc3f1efd5 100644
--- a/jarvis/agent_chat/runner_acp.py
+++ b/jarvis/agent_chat/runner_acp.py
@@ -195,7 +195,7 @@ async def plan_runtime_turn(
except RuntimeUnavailable as exc:
raise CliUnavailable(str(exc)) from exc
try:
- failure = gateway.watch_failure(route.api_key or "")
+ failure = gateway.watch_failure(route.api_key or "", effort=session.effort or "")
except Exception:
if launch.release is not None:
launch.release()
@@ -203,6 +203,13 @@ async def plan_runtime_turn(
def release() -> None:
gateway.unwatch_failure(route.api_key or "", failure)
+ if launch.invalidate is not None and (
+ handle.cancel.is_set()
+ or (failure.done() and not failure.cancelled())
+ or not acp.saw_result
+ or acp.status != "done"
+ ):
+ launch.invalidate()
if launch.release is not None:
launch.release()
diff --git a/jarvis/agent_chat/runner_cli.py b/jarvis/agent_chat/runner_cli.py
index 82a9126c22..2b99654e76 100644
--- a/jarvis/agent_chat/runner_cli.py
+++ b/jarvis/agent_chat/runner_cli.py
@@ -94,6 +94,7 @@ from jarvis.agent_chat.tool_context import register_turn, unregister_turn
from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS
from jarvis.core.response_style import (
AGENT_QUESTION_GUIDANCE,
+ CREDENTIAL_GUIDANCE,
KEEP_GOING_ON_TOOL_FAILURE,
TASK_EXECUTION_GUIDANCE,
)
@@ -1376,6 +1377,9 @@ def _with_identity(
+ AGENT_QUESTION_GUIDANCE + "\n"
+ KEEP_GOING_ON_TOOL_FAILURE
+ " Existing permission rules still apply.\n"
+ + CREDENTIAL_GUIDANCE
+ + jarvis_harness.society_credential_state(identity.text)
+ + "\n"
+ CONVERSATIONAL_TURN_REMINDER
+ "\n</jarvis_turn_context>\n\n"
+ prompt
@@ -3781,7 +3785,7 @@ async def _drive_cli(
translate(obj, state)
continue
if plan.acp is not None:
- if _provider_error():
+ if _provider_error() or handle.cancel.is_set():
# Do not present a runtime's synthetic error prose as an
# assistant answer, or process tools after the failure.
continue
@@ -3991,6 +3995,9 @@ async def _drive_cli(
run lives in its Gateway and keeps calling the model and running
tools after its bridge dies. The caller kills afterwards either way.
"""
+ if plan.provider_failure is not None:
+ # Revoke inference before waiting for the runtime to acknowledge stop.
+ plan.provider_failure.cancel()
if plan.acp is None or hosted or proc.returncode is not None:
return
frame = plan.acp.cancel_frame()
@@ -4128,6 +4135,7 @@ async def _drive_cli(
# in the turn host and the next app start carries the turn on.
proc.detach()
else:
+ await _cancel_acp()
_kill(proc)
raise
finally:
@@ -4190,7 +4198,7 @@ async def _drive_cli(
elif plan.shape == "codex" and state.failed_tools:
status = "error"
error_text = "Unresolved tool failure: " + ", ".join(sorted(state.failed_tools))
- elif runner in {"claude-cli", "glm-cli"} and not state.saw_result:
+ elif (runner in {"claude-cli", "glm-cli"} or plan.acp is not None) and not state.saw_result:
status = "error"
error_text = f"{runner} exited without a terminal result; its output may be incomplete."
diff --git a/jarvis/agent_chat/secret_requests.py b/jarvis/agent_chat/secret_requests.py
new file mode 100644
index 0000000000..00811319e8
--- /dev/null
+++ b/jarvis/agent_chat/secret_requests.py
@@ -0,0 +1,99 @@
+"""Spot an agent that asks for a secret in plain text instead of the secure field.
+
+A society agent gets credentials only through ``society_request_credential``
+(jarvis/society/credential_tool.py): the chat shows a password field and the
+value never enters the conversation. Models still fall back to old habits:
+"please paste your bot token", "enter it in a secure field", a question card
+whose free-text answer would carry the token, or "the token is already
+stored" when nothing is. These checks catch those replies so the turn can be
+steered back to the tool (turn_completion.py) or a question card refused
+(society/ask_tool.py).
+
+Deliberately lexical and narrow: a sentence must name a secret AND ask the
+user to hand it over (or claim one is stored). Mentioning a stored variable by
+name, or explaining what a token is, does not match.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Final
+
+__all__ = ["asks_for_secret", "claims_stored_secret", "SECRET_NOUN"]
+
+#: Something only the user can supply and that must never travel through the chat.
+SECRET_NOUN: Final[re.Pattern[str]] = re.compile(
+ r"(?:\b(?:api[\s_-]?keys?|access[\s_-]?keys?|secret[\s_-]?keys?|private[\s_-]?keys?|"
+ r"client[\s_-]?secrets?|secrets?|tokens?|passwords?|passphrases?|credentials?|"
+ r"webhook[\s_-]?urls?|connection[\s_-]?strings?|ssh[\s_-]?keys?|"
+ r"\w*token\w*|\w*passwor[dt]\w*|"
+ r"kennw[oö]rt\w*|zugangsdaten|anmeldedaten|\w*schl[uü]ssel\w*|" # i18n-allow
+ r"contrase[nñ]as?|claves?|credenciales)\b" # i18n-allow
+ r"|令牌|密钥|密码|凭据|凭证)", # i18n-allow
+ re.I,
+)
+
+#: The user is asked to hand something over, or told where to put it.
+_HAND_OVER: Final[re.Pattern[str]] = re.compile(
+ r"(?:\b(?:please|send|paste|enter|provide|share|give|copy|"
+ r"i need|we need|i'?ll need|need your|need a|need the|do you have|reply with|"
+ r"secure (?:input )?field|"
+ r"bitte|gib|gebe|schick\w*|sende|teil\w*|f[uü]g\w*|kopier\w*|trag\w*|" # i18n-allow
+ r"hinterleg\w*|eingeben|einf[uü]gen|brauche|ben[oö]tige|hast du|" # i18n-allow
+ r"sicheres?n? (?:eingabe)?feld|" # i18n-allow
+ r"env[ií]a\w*|pega\w*|introduc\w*|necesito|proporciona\w*|compart\w*|ingresa\w*|" # i18n-allow
+ r"por favor)\b" # i18n-allow
+ r"|\?|请|提供|输入|粘贴|需要)", # i18n-allow
+ re.I,
+)
+
+#: Only the value itself is asked for (the answer would BE the secret).
+_VALUE: Final[re.Pattern[str]] = re.compile(
+ r"(?:\b(?:send|paste|enter|provide|share|give me|type|reply with|"
+ r"(?:what|which) (?:is|are) (?:your|the)|"
+ r"gib|schick\w*|sende|teil\w* mir|nenn\w*|eingeben|einf[uü]gen|wie lautet|" # i18n-allow
+ r"was ist dein\w*|" # i18n-allow
+ r"env[ií]a\w*|pega\w*|introduc\w*|proporciona\w*|cu[aá]l es (?:tu|la|el))\b" # i18n-allow
+ r"|请提供|输入|粘贴)", # i18n-allow
+ re.I,
+)
+
+#: A sentence that reports a credential as already saved.
+_STORED: Final[re.Pattern[str]] = re.compile(
+ r"\b(?:(?:is|was|has been|are|have been) (?:already |now |securely |safely )*"
+ r"(?:stored|saved|set|configured)|"
+ r"(?:ist|wurde|sind|wurden) (?:\w+ ){0,3}(?:gespeichert|hinterlegt|gesetzt|" # i18n-allow
+ r"eingerichtet)|" # i18n-allow
+ r"(?:est[aá]|fue|ya est[aá]) (?:\w+ )?(?:guardad[oa]|almacenad[oa]))\b", # i18n-allow
+ re.I,
+)
+
+_SENTENCE: Final[re.Pattern[str]] = re.compile(r"[^.!?。!?\n]+[.!?。!?]?")
+
+
+def _sentences(text: str) -> list[str]:
+ return [s.strip() for s in _SENTENCE.findall(text or "") if s.strip()]
+
+
+def asks_for_secret(text: str, *, value_only: bool = False) -> bool:
+ """``text`` asks the user to hand over a secret, or says where to put one.
+
+ ``value_only``: only a request for the value itself counts (a question card
+ whose typed answer would be the secret), not "token or browser login?".
+ A sentence that only reports a credential as stored does not count; see
+ ``claims_stored_secret`` for that case.
+ """
+ cue = _VALUE if value_only else _HAND_OVER
+ for sentence in _sentences(text):
+ if not SECRET_NOUN.search(sentence) or _STORED.search(sentence):
+ continue
+ if cue.search(sentence):
+ return True
+ return False
+
+
+def claims_stored_secret(text: str) -> bool:
+ """``text`` says a credential is already stored or set."""
+ return any(
+ SECRET_NOUN.search(sentence) and _STORED.search(sentence) for sentence in _sentences(text)
+ )
diff --git a/jarvis/agent_chat/send_queue.py b/jarvis/agent_chat/send_queue.py
index 34b753b5ca..1250904a99 100644
--- a/jarvis/agent_chat/send_queue.py
+++ b/jarvis/agent_chat/send_queue.py
@@ -5,12 +5,9 @@ teammates and routines runs there too, so the person often writes while a
turn is still running. Instead of a "turn already running" refusal, the
message waits here and starts as the next turn once the agent's seat is free.
-The queue lives in memory and is bounded per chat, in size and in waiting
-time. While the chat's own turn runs, that turn bounds the wait (its setup,
-slot and run timeouts: a Hermes or OpenClaw agent's first turn may install
-its runtime for many minutes); :data:`MAX_WAIT_S` counts only the time the
-chat is idle yet still cannot take the message, and :data:`MAX_TOTAL_WAIT_S`
-caps the wait as a whole. Every queued message is a ``message_queued`` notice in the chat; when it
+The queue lives in memory and is bounded per chat in size. Waiting for a
+credential or approval never expires an accepted message. Every queued
+message is a ``message_queued`` notice in the chat; when it
starts (or cannot be sent) a ``message_dequeued`` notice with the same
``queue_id`` replaces it, so the timeline shows a waiting message exactly
while it waits. A restart forgets the queue, so opening the chat afterwards
@@ -29,30 +26,12 @@ from typing import Any, Final
log = logging.getLogger(__name__)
-__all__ = [
- "MAX_QUEUED",
- "MAX_TOTAL_WAIT_S",
- "MAX_WAIT_S",
- "QueueFull",
- "close_orphans",
- "send_or_queue",
-]
+__all__ = ["MAX_QUEUED", "QueueFull", "close_orphans", "send_or_queue"]
#: Messages one chat may hold while its agent works.
-MAX_QUEUED: Final[int] = 5
-#: How long a message may wait while the chat is idle but the agent's seat is
-#: still taken elsewhere (another of its chats, a delegation), before it is
-#: reported unsent.
-MAX_WAIT_S: Final[float] = 600.0
-#: The longest any message waits, a running turn included. Above the longest
-#: turn the runner allows: a runtime setup (30 min, ``agent_runtimes.manager``),
-#: the runtime's turn slot (15 min, ``agent_runtimes.base``) and the run (60 min,
-#: ``runner_cli._TURN_TIMEOUT_S``).
-MAX_TOTAL_WAIT_S: Final[float] = 2.5 * 3600
+MAX_QUEUED: Final[int] = 64
#: How often a waiting chat checks whether its agent is free (seconds).
_POLL_S: Final[float] = 0.5
-#: How far back opening a chat looks for waiting notices a restart orphaned.
-_ORPHAN_TAIL: Final[int] = 400
#: Shown in the chat for a message that never started; never provider text (AP-34).
_NOT_SENT: Final[str] = "The waiting message could not be sent. Please send it again."
@@ -67,15 +46,18 @@ class _Queued:
text: str
attachments: list[dict[str, Any]] | None
tool_choices: list[str] | None
- queued_at: float = 0.0
- #: Seconds the chat was idle yet could not take this message.
- blocked_s: float = 0.0
+ timezone: str | None = None
@dataclass(slots=True)
class _ChatQueue:
items: deque[_Queued] = field(default_factory=deque)
drain: asyncio.Task[None] | None = None
+ starting: bool = False
+
+
+def _display_text(text: str, attachments: list[dict[str, Any]] | None) -> str:
+ return text.strip() or ", ".join(str(item.get("name", "")) for item in attachments or [])
def _queues(svc: Any) -> dict[str, _ChatQueue]:
@@ -87,16 +69,16 @@ def _queues(svc: Any) -> dict[str, _ChatQueue]:
def _queueable(svc: Any, session_id: str) -> bool:
- """Only a created agent's own chat queues; every other chat refuses as before."""
- from jarvis.society.roster import LEAD_AGENT_ID, PAIR_SESSION_MARKER
+ """User chats queue; automated routine and teammate threads stay exclusive."""
+ from jarvis.society.roster import PAIR_SESSION_MARKER
from jarvis.society.routine_runner import is_routine_session
session = svc.store.get_session(session_id)
- if session is None or getattr(session, "surface", "") != "society":
+ if session is None or getattr(session, "surface", "") not in ("jarvis", "society"):
return False
if is_routine_session(session_id) or PAIR_SESSION_MARKER in session_id:
return False
- return session_id != f"society:{LEAD_AGENT_ID}"
+ return True
async def _notice(svc: Any, session_id: str, payload: dict[str, Any]) -> None:
@@ -123,29 +105,43 @@ async def send_or_queue(
that does not queue and :class:`QueueFull` when the queue is full.
"""
from jarvis.agent_chat.service import SessionBusy
+ from jarvis.tasks.context import client_timezone
queues = _queues(svc)
- pending = queues.get(session_id)
- if pending is None or not pending.items:
+ if not _queueable(svc, session_id):
+ return await svc.send(session_id, text, attachments, tool_choices=tool_choices), ""
+ pending = queues.setdefault(session_id, _ChatQueue())
+ first = False
+ if not pending.items and not pending.starting and not svc.is_running(session_id):
+ # Reserve admission before setup yields. Later HTTP requests must not
+ # overtake this message, even before the service reserves its seat.
+ pending.starting = True
try:
turn_id = await svc.send(session_id, text, attachments, tool_choices=tool_choices)
- except SessionBusy:
- if not _queueable(svc, session_id):
- raise
+ except SessionBusy: # The expected occupied seat becomes a visible queued message below.
+ first = True
else:
return turn_id, ""
+ finally:
+ pending.starting = False
+ if not pending.items and not first and queues.get(session_id) is pending:
+ queues.pop(session_id, None)
# Earlier waiting messages go first, so a new one never overtakes them.
- pending = queues.setdefault(session_id, _ChatQueue())
- if len(pending.items) >= MAX_QUEUED:
+ if len(pending.items) >= MAX_QUEUED and not first:
raise QueueFull(session_id)
- item = _Queued(
- uuid.uuid4().hex, text, attachments, tool_choices, asyncio.get_running_loop().time()
- )
- pending.items.append(item)
+ item = _Queued(uuid.uuid4().hex, text, attachments, tool_choices, client_timezone.get())
+ # Recovery reads persisted notices off-thread. A generation fence detects
+ # an admission that both starts and finishes while that snapshot is read.
+ svc._society_send_generation = getattr(svc, "_society_send_generation", 0) + 1
+ if first:
+ pending.items.appendleft(item)
+ else:
+ pending.items.append(item)
await _notice(
svc,
session_id,
- {"kind": "message_queued", "queue_id": item.queue_id, "text": text.strip()},
+ {"kind": "message_queued", "queue_id": item.queue_id,
+ "text": _display_text(text, attachments)},
)
if pending.drain is None or pending.drain.done():
pending.drain = asyncio.create_task(
@@ -160,7 +156,7 @@ async def _drain(svc: Any, session_id: str, pending: _ChatQueue) -> None:
await _drain_loop(svc, session_id, pending)
finally:
queues = _queues(svc)
- if not pending.items and queues.get(session_id) is pending:
+ if not pending.items and not pending.starting and queues.get(session_id) is pending:
queues.pop(session_id, None)
@@ -169,35 +165,26 @@ async def _fail(svc: Any, session_id: str, item: _Queued) -> None:
svc,
session_id,
{"kind": "message_dequeued", "queue_id": item.queue_id, "status": "failed",
- "text": _NOT_SENT},
+ "text": _display_text(item.text, item.attachments) or _NOT_SENT},
)
async def _drain_loop(svc: Any, session_id: str, pending: _ChatQueue) -> None:
from jarvis.agent_chat.service import SessionBusy
+ from jarvis.tasks.context import client_timezone
- loop = asyncio.get_running_loop()
- checked = loop.time()
while pending.items:
# Jitter only spreads polls of several waiting chats; it is not security.
await asyncio.sleep(_POLL_S + random.uniform(0, _POLL_S / 2)) # noqa: S311
- now = loop.time()
- step, checked = now - checked, now
- head = pending.items[0]
- if head.blocked_s > MAX_WAIT_S or now - head.queued_at > MAX_TOTAL_WAIT_S:
- # The agent stayed busy too long: report it instead of waiting forever.
- pending.items.popleft()
- await _fail(svc, session_id, head)
- continue
- if svc.is_running(session_id):
- # The chat's own turn bounds this wait with its own timeouts.
+ if pending.starting or svc.is_running(session_id):
continue
+ head = pending.items[0]
+ token = client_timezone.set(head.timezone)
try:
await svc.send(
session_id, head.text, head.attachments, tool_choices=head.tool_choices
)
except SessionBusy: # the seat is still taken: this message simply waits longer
- head.blocked_s += step
continue
except Exception: # noqa: BLE001 — reported in the chat, then the next one runs
# The detail stays in the log; a provider's text never reaches the chat.
@@ -205,6 +192,8 @@ async def _drain_loop(svc: Any, session_id: str, pending: _ChatQueue) -> None:
pending.items.popleft()
await _fail(svc, session_id, head)
continue
+ finally:
+ client_timezone.reset(token)
pending.items.popleft()
await _notice(
svc,
@@ -219,32 +208,56 @@ async def close_orphans(svc: Any, session_id: str) -> int:
Returns how many were closed. Messages still queued in this process are
left alone.
"""
- pending = _queues(svc).get(session_id)
- live = {item.queue_id for item in pending.items} if pending is not None else set()
- try:
- events = await asyncio.to_thread(svc.store.list_events, session_id, tail=_ORPHAN_TAIL)
- except Exception: # noqa: BLE001 — a chat that cannot be read keeps its notices
- log.warning("agent chat: waiting notices of %s not checked", session_id, exc_info=True)
+ lock = getattr(svc, "_society_queue_recovery_lock", None)
+ if lock is None:
+ lock = svc._society_queue_recovery_lock = asyncio.Lock()
+ async with lock:
+ return await _close_orphans_locked(svc, session_id)
+
+
+async def _close_orphans_locked(svc: Any, session_id: str) -> int:
+ """One recovery at a time prevents duplicate receipts from GET/WS reopens."""
+ live: set[str] = set()
+ for _attempt in range(2):
+ generation = getattr(svc, "_society_send_generation", 0)
+ pending = _queues(svc).get(session_id)
+ if pending is not None:
+ live.update(item.queue_id for item in pending.items)
+ try:
+ events = await asyncio.to_thread(svc.store.queue_notice_events, session_id)
+ except Exception: # noqa: BLE001 — a chat that cannot be read keeps its notices
+ log.warning("agent chat: waiting notices of %s not checked", session_id, exc_info=True)
+ return 0
+ if generation == getattr(svc, "_society_send_generation", 0):
+ break
+ else:
+ # A busy process can postpone recovery until the next open/reconnect;
+ # it must never overwrite a live message's successful sent receipt.
return 0
- waiting: dict[str, None] = {}
+ waiting: dict[str, str] = {}
for event in events:
payload = event.get("payload") or {}
if event.get("kind") != "notice":
continue
queue_id = str(payload.get("queue_id") or "")
if payload.get("kind") == "message_queued" and queue_id:
- waiting[queue_id] = None
+ waiting[queue_id] = str(payload.get("text") or _NOT_SENT)
elif payload.get("kind") == "message_dequeued":
waiting.pop(queue_id, None)
closed = 0
- for queue_id in waiting:
+ for queue_id, text in waiting.items():
+ # An admission may have arrived while the history read yielded. Keep
+ # both snapshots so a live message that just started also stays safe.
+ current = _queues(svc).get(session_id)
+ if current is not None:
+ live.update(item.queue_id for item in current.items)
if queue_id in live:
continue
await _notice(
svc,
session_id,
{"kind": "message_dequeued", "queue_id": queue_id, "status": "failed",
- "text": _NOT_SENT},
+ "text": text},
)
closed += 1
return closed
diff --git a/jarvis/agent_chat/service.py b/jarvis/agent_chat/service.py
index 8518245812..e2655b80c4 100644
--- a/jarvis/agent_chat/service.py
+++ b/jarvis/agent_chat/service.py
@@ -37,6 +37,11 @@ from jarvis.agent_chat import attachments as chat_attachments
from jarvis.agent_chat import turn_prompts
from jarvis.agent_chat.approval_bridge import ChatApprovalBridge
from jarvis.agent_chat.catalog import PROVIDER_ROWS, api_seat, offers, provider_row
+from jarvis.agent_chat.credential_requests import (
+ CREDENTIAL_TIMEOUT_S,
+ CredentialRequests,
+ CredentialSpec,
+)
from jarvis.agent_chat.effort import normalize_effort
from jarvis.agent_chat.events import make_event
from jarvis.agent_chat.permissions import default_permission, ladder_key, normalize_permission
@@ -67,6 +72,10 @@ from jarvis.core.protocols import ChatCompletion, ChatTurn, current_chat_turn
from jarvis.society.delivery import IncomingMessage
log = logging.getLogger(__name__)
+#: A ``[tools: a, b]`` capability-pin line. The list starts with a
+#: non-space character, so the padding and the list cannot both claim the
+#: same spaces: matching stays linear on long lines of spaces.
+_TOOL_PIN_LINE = re.compile(r"(?m)^\[tools:[ \t]*([^\]\s][^\]\r\n]*)\][ \t\r]*$")
def _stop_caller() -> str:
@@ -322,6 +331,47 @@ def _expires_ms(timeout_s: float) -> int:
return int(time.time() * 1000 + timeout_s * 1000)
+async def _redact_credentials(session_id: str, event: dict[str, Any]) -> dict[str, Any]:
+ """``event`` with the society agent's stored credential values masked.
+
+ A command that prints a token shows ``[credential ENV]`` in the chat, its
+ log and the live stream instead of the value (jarvis/society/credentials.py).
+ """
+ try:
+ from jarvis.society.credentials import current_vault
+ from jarvis.society.surface import agent_id_of
+
+ vault = current_vault()
+ agent_id = agent_id_of(session_id)
+ if vault is None or agent_id is None:
+ return event
+ if not vault.is_loaded(agent_id):
+ await asyncio.to_thread(vault.values, agent_id)
+ payload = event.get("payload")
+ redacted = vault.redact_payload(agent_id, payload)
+ except Exception: # noqa: BLE001 — fails closed: the text is withheld, the event stays
+ log.warning("agent chat: credential redaction failed for %s", session_id, exc_info=True)
+ return {**event, "payload": _withheld(event.get("payload"))}
+ return event if redacted is payload else {**event, "payload": redacted}
+
+
+#: Payload keys that name an event, never carry printed text.
+_EVENT_IDS: Final[frozenset[str]] = frozenset(
+ {"turn_id", "call_id", "message_id", "agent_id", "name", "request_id", "question_id"}
+)
+
+
+def _withheld(payload: Any) -> Any:
+ """``payload`` with all its text replaced: what an unmaskable event shows."""
+ if isinstance(payload, dict):
+ return {k: v if k in _EVENT_IDS else _withheld(v) for k, v in payload.items()}
+ if isinstance(payload, list):
+ return [_withheld(item) for item in payload]
+ if isinstance(payload, str):
+ return "[hidden: the credential check failed]" if payload else payload
+ return payload
+
+
def _current_task() -> asyncio.Task[Any] | None:
try:
return asyncio.current_task()
@@ -386,6 +436,11 @@ class AgentChatService:
# Questions an agent is waiting on (questions.py), by question id.
self._questions: dict[str, _OpenQuestion] = {}
self._question_tasks: set[asyncio.Task[None]] = set()
+ # Credential cards (credential_requests.py): the value never enters a chat event.
+ self._credentials = CredentialRequests(
+ self._emit, self._running_turn_id,
+ history=self.store.credential_events, restore_save=self._restore_credential_save,
+ )
# "Always allow" on the Jarvis surface: the tools a person waved through
# for the rest of the session, per session. Claude Code's "don't ask
# again for this tool" rather than a mode flip — the unified ladder has
@@ -912,6 +967,8 @@ class AgentChatService:
async def _emit(self, session_id: str, event: dict[str, Any]) -> None:
# One delivery path for every runner. Normalize only finished receipts;
# token deltas and voice-critical streaming never perform file I/O.
+ if session_id.startswith("society:"):
+ event = await _redact_credentials(session_id, event)
if event.get("kind") in {"assistant_text", "tool_result", "user_message"}:
from jarvis.agent_chat.media import normalize_media_event
from jarvis.core.paths import repo_root
@@ -1196,7 +1253,7 @@ class AgentChatService:
# browser pin to the same validated receipt used by the root composer.
if session.surface == "jarvis" and any(
"core:browser" in {item.strip() for item in match.split(",")}
- for match in re.findall(r"(?m)^\[tools:\s*([^\]\r\n]+)\]\s*$", text)
+ for match in _TOOL_PIN_LINE.findall(text)
):
tool_choices = list(dict.fromkeys([*(tool_choices or []), "tool:society_browser"]))
selected = []
@@ -2247,6 +2304,9 @@ class AgentChatService:
def _cancel_questions(self, session_id: str) -> None:
# Tolerates a service built without __init__ (test doubles), like _controls.
+ credentials: CredentialRequests | None = getattr(self, "_credentials", None)
+ if credentials is not None:
+ credentials.finish_turn(session_id)
questions: dict[str, _OpenQuestion] = getattr(self, "_questions", {})
for qid, open_q in list(questions.items()):
if open_q.session_id != session_id:
@@ -2256,6 +2316,83 @@ class AgentChatService:
open_q.wake.set()
questions.pop(qid, None)
+ # ------------------------------------------------------------ credentials
+
+ @staticmethod
+ def _restore_credential_save(session_id: str, spec: CredentialSpec) -> Callable | None:
+ """Rebuild only a Society field's save callback, never a stored secret."""
+ from jarvis.society.credentials import save_requested_credential, validate_env_name
+ from jarvis.society.surface import agent_id_of
+
+ agent_id = agent_id_of(session_id)
+ if agent_id is None:
+ return None
+ try:
+ validate_env_name(spec.env)
+ except ValueError:
+ log.warning("agent chat: ignored invalid stored credential field metadata")
+ return None
+
+ async def save(value: str) -> None:
+ await save_requested_credential(agent_id, spec.env, value, label=spec.label)
+
+ return save
+
+ def _running_turn_id(self, session_id: str) -> str | None:
+ run = self._running.get(session_id)
+ if run is None or run.task is None or run.task.done():
+ return None
+ return run.turn_id
+
+ async def open_credential_request(
+ self,
+ session_id: str,
+ spec: CredentialSpec,
+ save: Callable[[str], Any],
+ *,
+ asker: str = "",
+ timeout_s: float = CREDENTIAL_TIMEOUT_S,
+ ) -> str:
+ """Show a secure credential field in the running turn; returns its request id.
+
+ ``save`` receives the pasted value and is its only destination. Raises
+ ``RuntimeError`` without a running turn and ``TooManyCredentialRequests``
+ once the turn used its cards.
+ """
+ return await self._credentials.open(
+ session_id, spec, save, asker=asker, timeout_s=timeout_s
+ )
+
+ async def wait_credential_request(
+ self, session_id: str, request_id: str, timeout_s: float | None = None
+ ) -> str | None:
+ """``saved`` or ``declined``, or ``None`` while the field remains open."""
+ return await self._credentials.wait(session_id, request_id, timeout_s)
+
+ def credential_request_spec(self, session_id: str, request_id: str) -> CredentialSpec:
+ return self._credentials.spec(session_id, request_id)
+
+ async def restore_credential_requests(self, session_id: str) -> None:
+ await self._credentials.restore(session_id)
+
+ async def submit_credential(self, session_id: str, request_id: str, value: str) -> bool:
+ """The person's pasted value; ``ValueError`` when the vault refuses it."""
+ if self.store.get_session(session_id) is None:
+ return False
+ return await self._credentials.submit(session_id, request_id, value)
+
+ async def decline_credential(self, session_id: str, request_id: str) -> bool:
+ if self.store.get_session(session_id) is None:
+ return False
+ return await self._credentials.decline(session_id, request_id)
+
+ async def discard_credential_requests(self, session_id: str) -> None:
+ await self._credentials.discard_session(session_id)
+
+ def pending_credential_requests(self, session_id: str) -> list[str]:
+ credentials: CredentialRequests | None = getattr(self, "_credentials", None)
+ return credentials.pending(session_id) if credentials is not None else []
+
# ------------------------------------------------------------ end-of-turn cards
#: How far back a finished turn's reply is looked for. The question block
diff --git a/jarvis/agent_chat/store.py b/jarvis/agent_chat/store.py
index 7237386110..52a85bf4cc 100644
--- a/jarvis/agent_chat/store.py
+++ b/jarvis/agent_chat/store.py
@@ -46,7 +46,7 @@ CREATE TABLE IF NOT EXISTS agent_chat_sessions (
model TEXT NOT NULL DEFAULT '',
effort TEXT NOT NULL DEFAULT '',
cwd TEXT NOT NULL DEFAULT '',
- permission_mode TEXT NOT NULL DEFAULT 'ask',
+ permission_mode TEXT NOT NULL DEFAULT 'bypass',
vendor_session TEXT,
created_ms INTEGER NOT NULL,
updated_ms INTEGER NOT NULL,
@@ -266,7 +266,7 @@ class AgentChatStore:
now = now_ms()
# The route validated the mode against the runner's ladder; the store
# keeps the id as given.
- mode = (permission_mode or "").strip()
+ mode = (permission_mode or "").strip() or "bypass"
with self._lock:
self._conn.execute(
"INSERT INTO agent_chat_sessions (session_id, title, provider, model, effort, "
@@ -649,6 +649,41 @@ class AgentChatStore:
).fetchone()
return row is not None
+ def queue_notice_events(self, session_id: str) -> list[dict[str, Any]]:
+ """Queue receipts only, including waits hidden behind long tool streams."""
+ with self._lock:
+ rows = self._conn.execute(
+ "SELECT kind, payload FROM agent_chat_events WHERE session_id = ? "
+ "AND kind = 'notice' AND json_extract(payload, '$.kind') "
+ "IN ('message_queued', 'message_dequeued') ORDER BY seq",
+ (session_id,),
+ ).fetchall()
+ return [{"kind": row["kind"], "payload": json.loads(row["payload"])} for row in rows]
+
+ def credential_events(self, session_id: str) -> list[dict[str, Any]]:
+ """Pending fields and recent receipts, without loading the chat's text.
+
+ Keep every pending field even in an old turn, plus only the newest 256
+ closed fields for idempotent save retries across a process restart.
+ """
+ with self._lock:
+ rows = self._conn.execute(
+ "WITH fields AS ("
+ "SELECT json_extract(payload, '$.request_id') AS rid, "
+ "max(CASE WHEN kind = 'credential_required' THEN seq END) AS opened, "
+ "max(CASE WHEN kind = 'credential_resolved' THEN seq END) AS closed "
+ "FROM agent_chat_events WHERE session_id = ? "
+ "AND kind IN ('credential_required', 'credential_resolved') GROUP BY rid"
+ "), retained AS ("
+ "SELECT opened, closed FROM fields WHERE closed IS NULL UNION ALL "
+ "SELECT opened, closed FROM (SELECT opened, closed FROM fields "
+ "WHERE closed IS NOT NULL ORDER BY closed DESC LIMIT 256)"
+ ") SELECT e.kind, e.payload FROM retained r JOIN agent_chat_events e "
+ "ON e.session_id = ? AND (e.seq = r.opened OR e.seq = r.closed) ORDER BY e.seq",
+ (session_id, session_id),
+ ).fetchall()
+ return [{"kind": row["kind"], "payload": json.loads(row["payload"])} for row in rows]
+
def list_events(
self, session_id: str, *, after_seq: int = 0, tail: int | None = None
) -> list[dict[str, Any]]:
diff --git a/jarvis/agent_chat/turn_completion.py b/jarvis/agent_chat/turn_completion.py
index 8baad1a60a..4161f8e23e 100644
--- a/jarvis/agent_chat/turn_completion.py
+++ b/jarvis/agent_chat/turn_completion.py
@@ -10,13 +10,17 @@ from __future__ import annotations
import asyncio
import json
+import logging
import re
import time
from typing import Any
from .questions import CANCELLED, MAX_ASKS_PER_TURN
+from .secret_requests import asks_for_secret, claims_stored_secret
from .task_recovery import ToolRecovery
+log = logging.getLogger(__name__)
+
_REQUEST = re.compile(
r"^\s*(?:(?:hello|hi|hallo|hola)[,! ]+)?" # i18n-allow
r"(?:(?:please|bitte|por favor)\s+)?" # i18n-allow
@@ -110,6 +114,11 @@ class TurnCompletion:
self.calls: dict[str, tuple[str, dict[str, Any]]] = {}
self.receipts = ToolRecovery()
self.did_work = False
+ # Credential guard: everything the agent said this turn, and whether it
+ # opened a secure credential field (society_request_credential).
+ self.texts: list[str] = []
+ self.credential_asked = False
+ self.credential_corrected = False
self.background_corrected = False
self.background_unregistered = False
self.background_tasks: set[str] = set()
@@ -130,6 +139,7 @@ class TurnCompletion:
self.receipts.observe(event)
if kind == "assistant_text":
self.last_text = str(payload.get("text") or "")
+ self.texts.append(self.last_text)
from .background_work import has_scheduled_work, promises_background_work
self.background_unregistered = (
@@ -149,6 +159,8 @@ class TurnCompletion:
elif kind == "tool_call":
name = str(payload.get("name") or "").rsplit("__", 1)[-1].lower()
self.calls[str(payload.get("call_id") or "")] = (name, payload.get("input") or {})
+ if name == CREDENTIAL_TOOL:
+ self.credential_asked = True
elif kind == "tool_result":
name, _ = self.calls.pop(str(payload.get("call_id") or ""), ("", {}))
if name == "society_propose_change" and not payload.get("is_error"):
@@ -210,6 +222,9 @@ class TurnCompletion:
+ "\nOriginal request:\n"
+ self.context
)
+ correction = await self._credential_correction(sid)
+ if correction is not None:
+ return correction
if self.background_unregistered:
if (
self.allow_correction and not self.background_corrected
@@ -254,6 +269,40 @@ class TurnCompletion:
)
return None
+ async def _credential_correction(self, sid: str) -> str | None:
+ """Steer a reply that asks for a secret in plain text back to the secure field.
+
+ Once per turn, only when the agent did not open a credential field
+ itself. A reply that claims a credential is stored counts too when the
+ agent has none stored.
+ """
+ if (
+ not self.allow_correction
+ or self.credential_corrected
+ or self.credential_asked
+ or self.receipts.declined
+ or self.receipts.blocked
+ ):
+ return None
+ said = "\n".join(self.texts)
+ stored = await _stored_credentials(sid)
+ if stored is None:
+ return None # no credential tool in this chat (a routine, or no society)
+ if not asks_for_secret(said) and not (not stored and claims_stored_secret(said)):
+ return None
+ self.credential_corrected = True
+ return (
+ "Your reply asks the user for a credential, says where to put one, or claims one is "
+ f"stored, but you did not call {CREDENTIAL_TOOL}. Secrets reach you only through "
+ f"that tool. Call {CREDENTIAL_TOOL} now for each credential the task needs (env, "
+ "label, and a description of what it is for and where the user creates it), wait "
+ "for its result, then continue the original task. Do not ask for the value in the "
+ "chat and do not repeat the request in plain text. Stored credentials right now: "
+ + (", ".join(stored) if stored else "none")
+ + ". If the task needs no credential from the user, finish without mentioning one."
+ "\nOriginal request:\n" + self.context
+ )
+
async def publish(self) -> None:
if self.finish_event is not None:
payload = {
@@ -264,3 +313,25 @@ class TurnCompletion:
if self.cost is not None:
payload["cost_usd"] = self.cost
await self.handle.emit({**self.finish_event, "payload": payload})
+
+
+#: The secure credential field's tool (jarvis/society/credential_tool.py).
+CREDENTIAL_TOOL = "society_request_credential"
+
+
+async def _stored_credentials(session_id: str) -> list[str] | None:
+ """Env names stored for the chat's society agent; ``None`` without the tool."""
+ from jarvis.society.credentials import current_vault
+ from jarvis.society.routine_runner import is_routine_session
+ from jarvis.society.surface import agent_id_of
+
+ agent_id = agent_id_of(session_id)
+ vault = current_vault()
+ if agent_id is None or vault is None or is_routine_session(session_id):
+ return None
+ try:
+ rows = await asyncio.to_thread(vault.list, agent_id)
+ except Exception: # noqa: BLE001 — an unreadable index skips the guard, never the turn
+ log.warning("credential guard: index unavailable for %s", session_id, exc_info=True)
+ return None
+ return [row.env for row in rows]
diff --git a/jarvis/agent_chat/turn_host_client.py b/jarvis/agent_chat/turn_host_client.py
index acc69ece61..ff1a93d0c5 100644
--- a/jarvis/agent_chat/turn_host_client.py
+++ b/jarvis/agent_chat/turn_host_client.py
@@ -218,6 +218,7 @@ class HostedCli:
self._exit: asyncio.Future[int] = asyncio.get_running_loop().create_future()
self._fed = 0
self._acked = 0
+ self._attaching = False
self.detached = False
self.host_lost = False
#: Another app process took the host over and carries this turn on.
@@ -269,6 +270,18 @@ class HostedCli:
self._acked = seq
self._send({"op": "ack", "seq": seq})
+ def _on_attach_reply(self, reply: dict[str, Any]) -> None:
+ if reply.get("ok") and reply.get("known"):
+ self.meta = dict(reply.get("meta") or self.meta)
+ self.pid = int(reply.get("pid") or self.pid)
+ if self.stdin is None and self.meta.get("keep_stdin"):
+ self.stdin = _HostStdin(self)
+ self._acked = int(reply.get("acked") or 0)
+ self.stdout.replay_upto = self._acked
+ for line in reply.get("stderr") or ():
+ self._on_err(str(line))
+ self._attaching = False
+
def _on_line(self, seq: int, text: str) -> None:
if seq <= self._fed:
return
@@ -312,6 +325,7 @@ class TurnHostClient:
# Events for an id whose spawn reply has not been handled yet.
self._early: dict[str, list[dict[str, Any]]] = {}
self._futures: dict[int, asyncio.Future[dict[str, Any]]] = {}
+ self._reply_callbacks: dict[int, Callable[[dict[str, Any]], None]] = {}
self._next_rid = 1
self._closed = False
self.host_pid = host_pid
@@ -366,16 +380,21 @@ class TurnHostClient:
)
# Registered BEFORE the request: the replayed lines follow the reply
# immediately and must find their reader.
+ cli._attaching = True
self._register(cli)
- reply = await self._request({"op": "attach", "id": host_id})
+ try:
+ reply = await self._request(
+ {"op": "attach", "id": host_id},
+ on_reply=cli._on_attach_reply,
+ )
+ except BaseException:
+ if self._procs.get(host_id) is cli:
+ cli.detach()
+ cli._on_lost()
+ raise
if not reply.get("ok") or not reply.get("known"):
self._forget(host_id)
return None
- acked = int(reply.get("acked") or 0)
- cli.stdout.replay_upto = acked
- cli._acked = acked
- for line in reply.get("stderr") or ():
- cli._on_err(str(line))
return cli
def _register(self, cli: HostedCli) -> None:
@@ -408,15 +427,21 @@ class TurnHostClient:
return False
return True
- async def _request(self, frame: dict[str, Any]) -> dict[str, Any]:
+ async def _request(
+ self, frame: dict[str, Any],
+ *, on_reply: Callable[[dict[str, Any]], None] | None = None,
+ ) -> dict[str, Any]:
if self._closed:
raise ConnectionError("The turn host is not connected.")
rid = self._next_rid
self._next_rid += 1
future: asyncio.Future[dict[str, Any]] = self._loop.create_future()
self._futures[rid] = future
+ if on_reply is not None:
+ self._reply_callbacks[rid] = on_reply
if not self._send({**frame, "rid": rid}):
self._futures.pop(rid, None)
+ self._reply_callbacks.pop(rid, None)
raise ConnectionError("The turn host is not connected.")
try:
await self._writer.drain()
@@ -425,6 +450,7 @@ class TurnHostClient:
raise ConnectionError("The turn host did not answer in time.") from exc
finally:
self._futures.pop(rid, None)
+ self._reply_callbacks.pop(rid, None)
async def _read_loop(self) -> None:
error: BaseException | None = None
@@ -452,6 +478,9 @@ class TurnHostClient:
if rid is not None:
future = self._futures.get(int(rid))
if future is not None and not future.done():
+ callback = self._reply_callbacks.pop(int(rid), None)
+ if callback is not None:
+ callback(frame)
future.set_result(frame)
return
if frame.get("ev") == "replaced":
@@ -468,6 +497,11 @@ class TurnHostClient:
@staticmethod
def _deliver(cli: HostedCli, frame: dict[str, Any]) -> None:
+ if cli._attaching:
+ # Live frames before the attach reply belong to the old reader.
+ # The host replays them after that reply; accepting a later live
+ # sequence now would make the new reader discard earlier replay.
+ return
event = frame.get("ev")
if event == "line":
cli._on_line(int(frame.get("seq") or 0), str(frame.get("d", "")))
@@ -484,6 +518,7 @@ class TurnHostClient:
if not future.done():
future.set_exception(ConnectionError("The turn host went away."))
self._futures.clear()
+ self._reply_callbacks.clear()
if was_open:
log.warning(
"turn host connection lost (%s) — %d turn(s) reported as ended",
@@ -507,6 +542,7 @@ class TurnHostClient:
if not future.done():
future.set_exception(ConnectionError("Another app took the turn host over."))
self._futures.clear()
+ self._reply_callbacks.clear()
for cli in procs:
cli._on_handed_over()
diff --git a/jarvis/agent_resets.py b/jarvis/agent_resets.py
new file mode 100644
index 0000000000..d6a2557f28
--- /dev/null
+++ b/jarvis/agent_resets.py
@@ -0,0 +1,238 @@
+"""Account-scoped earned resets; never start a model turn or buy usage.
+
+Codex's documented app-server protocol supplies inventory and an idempotent
+redemption operation. Other providers remain explicit browser handoffs until
+their subscription transport exposes a verified reset contract.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import json
+import logging
+import os
+import time
+from typing import Any
+
+from jarvis.agent_accounts import AgentAccount
+
+log = logging.getLogger(__name__)
+TIMEOUT_S = 25
+_OUTCOMES = {"reset", "alreadyRedeemed", "nothingToReset", "noCredit"}
+_MANAGE_URLS = {
+ "codex": "https://chatgpt.com/codex/settings/usage",
+ "claude": "https://claude.ai/settings/usage",
+ "grok-build": "https://grok.com/",
+}
+
+
+class ResetError(ValueError):
+ """A safe, displayable failure; provider error bodies never leave here."""
+
+ def __init__(self, message: str, code: str = "uncertain") -> None:
+ super().__init__(message)
+ self.code = code
+
+
+def _empty(account: AgentAccount, status: str) -> dict[str, Any]:
+ return {
+ "account_id": account.id,
+ "status": status,
+ "available_count": None,
+ "credits": None,
+ "can_redeem": False,
+ "account_key": None,
+ "manage_url": _MANAGE_URLS.get(account.platform),
+ "as_of": time.time(),
+ "usage": None,
+ }
+
+
+def _identity(account: AgentAccount, remote: dict[str, Any]) -> str:
+ from jarvis.agent_usage import _read_json_file
+
+ auth = _read_json_file(account.config_dir / "auth.json") or {}
+ tokens = auth.get("tokens") or {}
+ ref = tokens.get("account_id") if isinstance(tokens, dict) else None
+ material = [str(account.config_dir.resolve()), ref, remote.get("email"), remote.get("type")]
+ return hashlib.sha256(json.dumps(material, sort_keys=True).encode()).hexdigest()
+
+
+def _snapshot(account: AgentAccount, payload: dict[str, Any], key: str) -> dict[str, Any]:
+ from jarvis.agent_usage import AccountUsage, _codex_rate_limit_windows, _iso_from_epoch
+
+ result = _empty(account, "unavailable")
+ result["account_key"] = key
+ summary = payload.get("rateLimitResetCredits")
+ if isinstance(summary, dict):
+ count = summary.get("availableCount")
+ if type(count) is int and count >= 0:
+ result.update(status="ok", available_count=count, can_redeem=count > 0)
+ rows = summary.get("credits")
+ if isinstance(rows, list):
+ result["credits"] = []
+ for row in rows:
+ if not isinstance(row, dict) or not isinstance(row.get("id"), str):
+ continue
+ expiry = row.get("expiresAt")
+ expiry_valid = expiry is None or (type(expiry) is int and expiry > time.time())
+ result["credits"].append(
+ {
+ "id": row["id"],
+ "title": row.get("title")
+ if isinstance(row.get("title"), str)
+ else None,
+ "description": row.get("description")
+ if isinstance(row.get("description"), str)
+ else None,
+ "expires_at": _iso_from_epoch(expiry),
+ "redeemable": bool(row["id"])
+ and row.get("status") == "available"
+ and row.get("resetType") == "codexRateLimits"
+ and expiry_valid,
+ }
+ )
+ # Empty/unknown detail rows are not permission to consume an
+ # unspecified credit. Count-only responses have their own path.
+ result["can_redeem"] = count > 0 and any(
+ row["redeemable"] for row in result["credits"]
+ )
+ buckets = payload.get("rateLimitsByLimitId")
+ limits = buckets.get("codex") if isinstance(buckets, dict) else None
+ if not isinstance(limits, dict):
+ limits = payload.get("rateLimits")
+ if isinstance(limits, dict):
+ normalized = {"limit_id": limits.get("limitId")}
+ for slot in ("primary", "secondary"):
+ window = limits.get(slot)
+ if isinstance(window, dict):
+ normalized[slot] = {
+ "used_percent": window.get("usedPercent"),
+ "window_minutes": window.get("windowDurationMins"),
+ "resets_at": window.get("resetsAt"),
+ }
+ windows = _codex_rate_limit_windows(normalized)
+ if windows:
+ result["usage"] = AccountUsage(
+ account.id,
+ account.platform,
+ "ok",
+ windows,
+ as_of=result["as_of"],
+ plan=limits.get("planType"),
+ ).to_dict()
+ return result
+
+
+async def _codex(
+ account: AgentAccount,
+ *,
+ attempt: str | None = None,
+ credit_id: str | None = None,
+ account_key: str | None = None,
+) -> dict[str, Any]:
+ from jarvis.agent_chat.native_control import GoalRpc
+ from jarvis.agent_chat.runner_cli import codex_argv_prefix
+ from jarvis.codex_app_server import _subscription_env_allowed
+
+ env = {k: v for k, v in os.environ.items() if _subscription_env_allowed(k)}
+ env["CODEX_HOME"] = str(account.config_dir)
+ # File storage prevents a missing redirected login from falling through to
+ # another account in a shared OS credential store. No thread is ever opened.
+ argv = [
+ *codex_argv_prefix(),
+ "app-server",
+ "-c",
+ 'model_provider="openai"',
+ "-c",
+ 'cli_auth_credentials_store="file"',
+ ]
+
+ async def deny(*_: Any) -> str:
+ return "deny"
+
+ async with GoalRpc(argv, env, str(account.config_dir), deny) as rpc:
+
+ async def request(method: str, params: dict[str, Any]) -> dict[str, Any]:
+ limit = TIMEOUT_S if method.endswith("/consume") else 8
+ return await asyncio.wait_for(rpc.request(method, params), timeout=limit)
+
+ await request(
+ "initialize",
+ {
+ "clientInfo": {"name": "jarvis-subscription-resets", "version": "1"},
+ "capabilities": {"experimentalApi": True},
+ },
+ )
+ await rpc.send({"method": "initialized"})
+ auth = await request("account/read", {"refreshToken": False})
+ remote = auth.get("account")
+ if not isinstance(remote, dict) or remote.get("type") != "chatgpt":
+ raise ResetError(
+ "Sign in to this Codex subscription before using a reset.", "signed_out"
+ )
+ key = _identity(account, remote)
+ if attempt is not None:
+ if account_key != key:
+ raise ResetError(
+ "This subscription login changed. Refresh before using a reset.",
+ "account_changed",
+ )
+ params = {"idempotencyKey": attempt}
+ if credit_id is not None:
+ params["creditId"] = credit_id
+ # Do not recheck inventory before replaying: a successful first
+ # attempt may have consumed the last credit and lost its reply.
+ answer = await request("account/rateLimitResetCredit/consume", params)
+ outcome = answer.get("outcome")
+ if not isinstance(outcome, str) or outcome not in _OUTCOMES:
+ raise ResetError(
+ "The reset outcome is unknown. Retry the same request to check it."
+ )
+ # Preserve the confirmed receipt even if the refresh fails.
+ try:
+ payload = await request("account/rateLimits/read", {})
+ snapshot = _snapshot(account, payload, key)
+ except (ValueError, ConnectionError, TimeoutError):
+ log.info("Reset completed; usage refresh unavailable for %s", account.id)
+ snapshot = _empty(account, "unavailable")
+ return {"outcome": outcome, "resets": snapshot}
+ payload = await request("account/rateLimits/read", {})
+ return _snapshot(account, payload, key)
+
+
+async def read_resets(account: AgentAccount) -> dict[str, Any]:
+ """Read only this login; missing support is distinct from zero credits."""
+ if account.platform != "codex":
+ return _empty(account, "browser" if account.platform == "claude" else "unsupported")
+ try:
+ return await _codex(account)
+ except (OSError, ValueError, RuntimeError, ConnectionError, TimeoutError) as exc:
+ log.info("Subscription reset inventory unavailable (%s)", type(exc).__name__)
+ return _empty(account, "unavailable")
+
+
+async def consume_reset(
+ account: AgentAccount, *, attempt: str, credit_id: str | None, account_key: str
+) -> dict[str, Any]:
+ """One deliberate attempt. A retry MUST carry the same idempotency key."""
+ if account.platform != "codex":
+ raise ResetError(
+ "This provider requires its own usage page to manage resets.", "unsupported"
+ )
+ try:
+ result = await _codex(
+ account, attempt=attempt, credit_id=credit_id, account_key=account_key
+ )
+ except ResetError:
+ raise
+ except (OSError, ValueError, RuntimeError, ConnectionError, TimeoutError) as exc:
+ log.info("Subscription reset response unavailable (%s)", type(exc).__name__)
+ raise ResetError(
+ "No confirmed reset response. Retry the same request to check it."
+ ) from exc
+ from jarvis.agent_usage import clear_cache
+
+ clear_cache()
+ return result
diff --git a/jarvis/agent_runtimes/acp.py b/jarvis/agent_runtimes/acp.py
index 67409906fb..b86a106618 100644
--- a/jarvis/agent_runtimes/acp.py
+++ b/jarvis/agent_runtimes/acp.py
@@ -323,7 +323,8 @@ class AcpTurn:
elif self.stop_reason in _STOPPED_EARLY and not self.emitted_text:
self._fail(_STOPPED_EARLY[self.stop_reason])
elif self.stop_reason in _STOPPED_EARLY:
- # The answer stands, but the chat says it is not the whole one.
+ self._fail(f"{_STOPPED_EARLY[self.stop_reason]} ({self.stop_reason})")
+ # Retain the partial answer while reporting that the turn is incomplete.
await io.emit(
make_event(
"notice",
@@ -337,6 +338,14 @@ class AcpTurn:
)
return
+ async def cancel(self, io: AcpIO) -> None:
+ """Abort the upstream session before its stdio bridge is terminated."""
+ if self._acp_session and not self.saw_result:
+ await io.write({
+ "jsonrpc": "2.0", "method": "session/cancel",
+ "params": {"sessionId": self._acp_session},
+ })
+
async def _on_agent_request(self, obj: dict[str, Any], method: str, io: AcpIO) -> None:
rid = obj.get("id")
params = obj.get("params") if isinstance(obj.get("params"), dict) else {}
diff --git a/jarvis/agent_runtimes/anthropic_history.py b/jarvis/agent_runtimes/anthropic_history.py
new file mode 100644
index 0000000000..f217fff6c0
--- /dev/null
+++ b/jarvis/agent_runtimes/anthropic_history.py
@@ -0,0 +1,87 @@
+"""Preserve signed Claude tool continuations across an OpenAI-shaped runtime.
+
+Entries are scoped to the runtime grant and model, and replayed only against
+the unchanged request prefix and assistant content that produced them.
+Nothing is persisted or sent as visible reasoning text to the runtime.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import threading
+from collections import OrderedDict
+from dataclasses import asdict, replace
+from typing import Any
+
+_LOCK = threading.Lock()
+_HISTORY: OrderedDict[
+ tuple[Any, str, str], tuple[str, str, str, list[dict[str, Any]]]
+] = OrderedDict()
+_MAX_ENTRIES = 128
+
+
+def _prefix(request: Any, messages: Any) -> str:
+ encoded = json.dumps({"system": request.system, "tools": request.tools,
+ "messages": [asdict(m) for m in messages]},
+ sort_keys=True, separators=(",", ":"))
+ return hashlib.sha256(encoded.encode()).hexdigest()
+
+
+def _visible(blocks: list[dict[str, Any]]) -> str:
+ # Chat Completions folds text and tools into separate fields.
+ return json.dumps({
+ "text": "".join(str(b.get("text") or "") for b in blocks if b.get("type") == "text"),
+ "tools": [{k: b.get(k) for k in ("id", "name", "input")}
+ for b in blocks if b.get("type") == "tool_use"],
+ }, sort_keys=True, separators=(",", ":"))
+
+
+def remember(
+ grant: Any, model: str, request: Any, call: dict[str, Any], *, provider_request: Any,
+) -> None:
+ blocks = call.get("_anthropic_blocks")
+ if not isinstance(blocks, list) or not call.get("id"):
+ return
+ key = (grant, model, str(call["id"]))
+ # Copy the provider envelope so later mutation cannot change signed data.
+ copied = json.loads(json.dumps(blocks))
+ entry = (
+ _prefix(request, request.messages),
+ _prefix(provider_request, provider_request.messages),
+ _visible(copied), copied,
+ )
+ with _LOCK:
+ _HISTORY[key] = entry
+ _HISTORY.move_to_end(key)
+ while len(_HISTORY) > _MAX_ENTRIES:
+ _HISTORY.popitem(last=False)
+
+
+def restore(grant: Any, model: str, request: Any) -> Any:
+ messages = list(request.messages)
+ for index, message in enumerate(request.messages):
+ if message.role != "assistant" or not isinstance(message.content, list):
+ continue
+ ids = [str(b.get("id")) for b in message.content if b.get("type") == "tool_use"]
+ with _LOCK:
+ entry = next((_HISTORY[(grant, model, call_id)] for call_id in ids
+ if (grant, model, call_id) in _HISTORY), None)
+ if entry is None:
+ continue
+ prefix, provider_prefix, visible, blocks = entry
+ if (
+ prefix != _prefix(request, request.messages[:index])
+ or provider_prefix != _prefix(request, messages[:index])
+ or visible != _visible(message.content)
+ ):
+ # Edited/compacted history must never receive another prefix's
+ # signature. Start a fresh Claude turn using the runtime's history.
+ continue
+ messages[index] = replace(message, content=json.loads(json.dumps(blocks)))
+ return replace(request, messages=tuple(messages))
+
+
+def reset() -> None:
+ with _LOCK:
+ _HISTORY.clear()
diff --git a/jarvis/agent_runtimes/base.py b/jarvis/agent_runtimes/base.py
index 930de00ea3..1a992352e6 100644
--- a/jarvis/agent_runtimes/base.py
+++ b/jarvis/agent_runtimes/base.py
@@ -184,6 +184,8 @@ class RuntimeLaunch:
#: Called exactly once when the turn's process is gone: frees the turn
#: slot (and lets an idle Gateway be reaped).
release: Callable[[], None] | None = None
+ #: Prevent reusing a persistent gateway after an unconfirmed termination.
+ invalidate: Callable[[], None] | None = None
class AgentRuntimeDriver(Protocol):
diff --git a/jarvis/agent_runtimes/gateway.py b/jarvis/agent_runtimes/gateway.py
index c6d0ab0977..7ccab6e994 100644
--- a/jarvis/agent_runtimes/gateway.py
+++ b/jarvis/agent_runtimes/gateway.py
@@ -84,9 +84,12 @@ class Grant:
account_id: str = ""
#: The runtime profile: direct chat and serialized scheduled runs differ.
scope: str = ""
+ #: Persistent runtimes may infer only while their runner owns a live turn.
+ require_active_turn: bool = False
_LOCK = threading.Lock()
+_UNBOUND: Final = object()
_GRANTS: dict[str, Grant] = {}
_TOKENS: OrderedDict[Grant, str] = OrderedDict()
#: Grants kept at once. Each chat session mints its own; the least recently
@@ -96,6 +99,7 @@ _CLIENTS: dict[str, Any] = {}
_MODEL_LIMITS: dict[Grant, dict[str, ModelLimits]] = {}
_CATALOG: Any = None
_FAILURES: dict[Grant, Future[str]] = {}
+_TURN_EFFORTS: dict[Grant, str] = {}
_COOLDOWNS: OrderedDict[tuple[str, str, str], float] = OrderedDict()
_DEFAULT_COOLDOWN_S: Final = 30.0
@@ -137,10 +141,13 @@ _EFFORTS: Final[frozenset[str]] = frozenset(
)
-def grant_token(agent_id: str, provider: str, account_id: str = "", *, scope: str = "") -> str:
+def grant_token(
+ agent_id: str, provider: str, account_id: str = "", *, scope: str = "",
+ require_active_turn: bool = False,
+) -> str:
"""The token one agent's runtime presents (stable for the app's lifetime,
so a runtime that keeps a process between turns is not restarted)."""
- grant = Grant(agent_id, provider, account_id, scope)
+ grant = Grant(agent_id, provider, account_id, scope, require_active_turn)
with _LOCK:
token = _TOKENS.get(grant)
if token is None:
@@ -263,7 +270,7 @@ async def refresh_model_limits(grant: Grant, model: str, config: Any) -> ModelLi
return resolve_limits(config, grant.provider, model, metadata)
-def watch_failure(token: str) -> Future[str]:
+def watch_failure(token: str, *, effort: str = "") -> Future[str]:
"""Observe this session's current turn, across the HTTP and runner loops."""
with _LOCK:
grant = _GRANTS[token]
@@ -271,6 +278,7 @@ def watch_failure(token: str) -> Future[str]:
raise RuntimeError("A runtime turn already owns this gateway session.")
signal: Future[str] = Future()
_FAILURES[grant] = signal
+ _TURN_EFFORTS[grant] = effort if effort in _EFFORTS else ""
return signal
@@ -279,6 +287,7 @@ def unwatch_failure(token: str, signal: Future[str]) -> None:
grant = _GRANTS.get(token)
if _FAILURES.get(grant) is signal:
del _FAILURES[grant]
+ _TURN_EFFORTS.pop(grant, None)
def _watch_for(grant: Grant) -> Future[str] | None:
@@ -286,6 +295,30 @@ def _watch_for(grant: Grant) -> Future[str] | None:
return _FAILURES.get(grant)
+def _require_turn(grant: Grant, signal: Future[str] | None) -> None:
+ if not grant.require_active_turn:
+ return
+ with _LOCK:
+ active = signal is not None and _FAILURES.get(grant) is signal and not signal.done()
+ if not active:
+ raise GatewayError(
+ "This runtime has no active turn. Start a new turn in Jarvis.",
+ status=409, code="turn_inactive",
+ )
+
+
+def _turn_effort(grant: Grant) -> str:
+ with _LOCK:
+ return _TURN_EFFORTS.get(grant, "")
+
+
+def capture_turn(grant: Grant) -> Future[str] | None:
+ """Bind inference to its ingress turn, before body/setup awaits can yield."""
+ signal = _watch_for(grant)
+ _require_turn(grant, signal)
+ return signal
+
+
def _retry_after(exc: Exception) -> float | None:
"""Read only Retry-After, never copy a provider's body or other headers."""
response = getattr(exc, "response", None)
@@ -355,9 +388,7 @@ def _report_failure(
failure = _limited(grant.provider, seconds)
from jarvis.agent_runtimes.provider_errors import is_terminal
- # Only a failure no runtime can recover from ends the turn here. A
- # timeout, an overloaded provider or a context overflow goes back to the
- # runtime, whose own retry and conversation compression handle it.
+ # Recoverable failures stay with the runtime for retry or compression.
if signal is not None and is_terminal(failure.code):
# A request captures its turn's signal before awaiting the provider.
# Late failures therefore cannot stop a later turn on the same session.
@@ -482,7 +513,9 @@ def _failed_event(message: str, code: str) -> dict[str, Any]:
}
-async def stream_response(grant: Grant, args: dict[str, Any]) -> AsyncIterator[bytes]:
+async def stream_response(
+ grant: Grant, args: dict[str, Any], *, lease: Any = _UNBOUND,
+) -> AsyncIterator[bytes]:
"""The Responses event stream as server-sent events.
Once streaming has begun a failure can only travel as an event, so it
@@ -493,17 +526,25 @@ async def stream_response(grant: Grant, args: dict[str, Any]) -> AsyncIterator[b
from jarvis.live.subscription_reasoning import SubscriptionReasoningError
model = str(args.get("model") or "")
- signal = _watch_for(grant)
+ signal = capture_turn(grant) if lease is _UNBOUND else lease
+ if effort := _turn_effort(grant):
+ args = {**args, "reasoning_effort": effort}
started = False
+ completed = False
try:
+ _require_turn(grant, signal)
check_cooldown(grant.provider, model, grant.account_id)
client = _client(grant.account_id)
args = await _snapped_effort(client, args)
async with contextlib.aclosing(client.stream(**args)) as upstream:
async for event in upstream:
started = True
+ completed = completed or event.get("type") == "response.completed"
yield _sse(event)
+ if not completed:
+ raise GatewayError("ChatGPT ended before completion.", status=502, code="incomplete")
except (SubscriptionReasoningError, SubscriptionAuthError, GatewayError) as exc:
+ log.info("runtime gateway: %s turn failed (%s)", grant.agent_id, type(exc).__name__)
failure = _report_failure(grant, model, _subscription_failure(grant, exc), signal)
if not started:
raise failure from exc
@@ -519,9 +560,14 @@ async def _snapped_effort(client: Any, args: dict[str, Any]) -> dict[str, Any]:
return {**args, "reasoning_effort": await snap(str(args.get("model") or ""), effort)}
-async def open_response_stream(grant: Grant, args: dict[str, Any]) -> AsyncIterator[bytes]:
+
+
+async def open_response_stream(
+ grant: Grant, args: dict[str, Any], *, lease: Any = _UNBOUND,
+) -> AsyncIterator[bytes]:
"""Check upstream acceptance before committing an HTTP 200 SSE response."""
- events = stream_response(grant, args)
+ lease = capture_turn(grant) if lease is _UNBOUND else lease
+ events = stream_response(grant, args, lease=lease)
try:
first = await anext(events)
except StopAsyncIteration as exc:
@@ -541,14 +587,19 @@ async def open_response_stream(grant: Grant, args: dict[str, Any]) -> AsyncItera
return accepted()
-async def complete_response(grant: Grant, args: dict[str, Any]) -> dict[str, Any]:
+async def complete_response(
+ grant: Grant, args: dict[str, Any], *, lease: Any = _UNBOUND,
+) -> dict[str, Any]:
"""The finished Responses object, for a runtime that did not ask to stream."""
from jarvis.live.subscription_auth import SubscriptionAuthError
from jarvis.live.subscription_reasoning import SubscriptionReasoningError
model = str(args.get("model") or "")
- signal = _watch_for(grant)
+ signal = capture_turn(grant) if lease is _UNBOUND else lease
+ if effort := _turn_effort(grant):
+ args = {**args, "reasoning_effort": effort}
try:
+ _require_turn(grant, signal)
check_cooldown(grant.provider, model, grant.account_id)
items: list[dict[str, Any]] = []
client = _client(grant.account_id)
@@ -764,6 +815,7 @@ def chat_request(body: Any) -> tuple[str, Any]:
raise GatewayError("The request has no messages.")
system: list[str] = []
messages: list[BrainMessage] = []
+ tool_names: dict[str, str] = {}
for message in raw_messages:
if not isinstance(message, dict):
raise GatewayError("Every message must be a JSON object.")
@@ -789,6 +841,7 @@ def chat_request(body: Any) -> tuple[str, Any]:
"name": str(function.get("name") or ""),
"input": _arguments(function.get("arguments")),
}
+ tool_names[call_id] = block["name"]
if signature := _signature(call_id):
block["thought_signature"] = signature
blocks.append(block)
@@ -802,7 +855,11 @@ def chat_request(body: Any) -> tuple[str, Any]:
"tool",
_text(content) if not isinstance(content, str) else content,
tool_call_id=str(message.get("tool_call_id") or ""),
- name=message.get("name") if isinstance(message.get("name"), str) else None,
+ name=(
+ message["name"]
+ if isinstance(message.get("name"), str) and message["name"]
+ else tool_names.get(str(message.get("tool_call_id") or ""))
+ ),
)
)
tools: list[dict[str, Any]] = []
@@ -996,7 +1053,9 @@ async def _close_brain(brain: Any) -> None:
return
-async def _deltas(grant: Grant, model: str, request: Any) -> AsyncIterator[Any]:
+async def _deltas(
+ grant: Grant, model: str, request: Any, lease: Any = _UNBOUND,
+) -> AsyncIterator[Any]:
"""The provider plugin's stream, with the Agents-tier key and cost caller.
The plugin runs in a task of its own and hands its deltas over a queue:
@@ -1008,15 +1067,22 @@ async def _deltas(grant: Grant, model: str, request: Any) -> AsyncIterator[Any]:
"""
from jarvis.agent_runtimes.model_map import login_route
from jarvis.agent_runtimes.provider_errors import login_expired
+ from jarvis.agent_runtimes import anthropic_history
from jarvis.core.config import get_jarvis_agent_secret, override_provider_secrets
from jarvis.costs.ledger import usage_context
from jarvis.costs.model import RUNTIME_CALLER
queue: asyncio.Queue[Any] = asyncio.Queue(maxsize=256)
+ signal = capture_turn(grant) if lease is _UNBOUND else lease
+ if effort := _turn_effort(grant):
+ request = replace(request, reasoning_effort=effort)
+ visible_request = request
+ request = anthropic_history.restore(grant, model, request)
async def pump() -> None:
login: str | None = None
try:
+ _require_turn(grant, signal)
secret = get_jarvis_agent_secret(grant.provider)
overrides = {grant.provider: secret} if secret else {}
with override_provider_secrets(overrides), usage_context(RUNTIME_CALLER):
@@ -1035,7 +1101,13 @@ async def _deltas(grant: Grant, model: str, request: Any) -> AsyncIterator[Any]:
if callable(configure_context):
limits = await asyncio.to_thread(model_limits, grant, model)
configure_context(limits.context_window)
+ _require_turn(grant, signal)
async for delta in brain.complete(request):
+ if delta.tool_call:
+ anthropic_history.remember(
+ grant, model, visible_request, delta.tool_call,
+ provider_request=request,
+ )
await queue.put(delta)
finally:
_release_brain(slot)
@@ -1140,7 +1212,7 @@ def _tool_call(index: int, call: dict[str, Any]) -> dict[str, Any]:
async def open_chat_stream(
- grant: Grant, model: str, request: Any, *, temperature_given: bool = True
+ grant: Grant, model: str, request: Any, *, temperature_given: bool = True, lease: Any = _UNBOUND,
) -> AsyncIterator[bytes]:
"""The answer as Chat Completions chunks.
@@ -1149,17 +1221,28 @@ async def open_chat_stream(
on; a failure after streaming began arrives as an ``error`` chunk.
``temperature_given`` is whether the runtime chose a temperature.
"""
+ signal = capture_turn(grant) if lease is _UNBOUND else lease
limits = await asyncio.to_thread(model_limits, grant, model)
request = _with_output_capacity(request, limits)
- stream = _deltas(grant, model, request)
+ stream = (_deltas(grant, model, request, signal) if grant.require_active_turn
+ else _deltas(grant, model, request))
started = time.monotonic()
- signal = _watch_for(grant)
try:
+ _require_turn(grant, signal)
check_cooldown(grant.provider, model, grant.account_id)
- # The plugin task starts on the first delta and keeps this profile.
with _with_profile(grant.provider, temperature_given):
first = await anext(stream, None)
- except Exception as exc: # noqa: BLE001 — becomes the runtime's HTTP error; _failure logs it
+ if first is None:
+ raise GatewayError(
+ "The provider ended without an answer.", status=502, code="incomplete",
+ )
+ except Exception as exc: # noqa: BLE001 — becomes the runtime's HTTP error, logged below
+ log.info(
+ "runtime gateway: %s on %s failed up front (%s)",
+ grant.agent_id,
+ grant.provider,
+ type(exc).__name__,
+ )
raise _report_failure(grant, model, _failure(grant.provider, exc, model), signal) from exc
return _chat_chunks(grant, model, stream, first, started, signal)
@@ -1175,7 +1258,7 @@ async def _chat_chunks(
chat_id = f"chatcmpl-{uuid.uuid4().hex}"
usage: dict[str, int] = {}
calls = 0
- finish = "stop"
+ finish: str | None = None
delta = first
try:
yield _chunk(chat_id, model, {"role": "assistant", "content": ""})
@@ -1190,6 +1273,10 @@ async def _chat_chunks(
if delta.finish_reason:
finish = _finish(delta.finish_reason)
delta = await anext(stream, None)
+ if finish is None:
+ raise GatewayError(
+ "The provider stream ended before completion.", status=502, code="incomplete",
+ )
except Exception as exc: # noqa: BLE001 — the stream already began: an error chunk tells the runtime
failure = _report_failure(grant, model, _failure(grant.provider, exc, model), signal)
payload = {"error": {"message": str(failure), "type": failure.code, "code": failure.code}}
@@ -1222,18 +1309,20 @@ async def _chat_chunks(
async def complete_chat(
- grant: Grant, model: str, request: Any, *, temperature_given: bool = True
+ grant: Grant, model: str, request: Any, *, temperature_given: bool = True, lease: Any = _UNBOUND,
) -> dict[str, Any]:
"""The finished ``chat.completion`` for a runtime that did not ask to stream."""
+ signal = capture_turn(grant) if lease is _UNBOUND else lease
limits = await asyncio.to_thread(model_limits, grant, model)
request = _with_output_capacity(request, limits)
- stream = _deltas(grant, model, request)
+ stream = (_deltas(grant, model, request, signal) if grant.require_active_turn
+ else _deltas(grant, model, request))
text: list[str] = []
calls: list[dict[str, Any]] = []
usage: dict[str, int] = {}
- finish = "stop"
- signal = _watch_for(grant)
+ finish: str | None = None
try:
+ _require_turn(grant, signal)
check_cooldown(grant.provider, model, grant.account_id)
with _with_profile(grant.provider, temperature_given):
# The plugin task starts on the first delta and keeps this profile.
@@ -1248,6 +1337,10 @@ async def complete_chat(
if delta.finish_reason:
finish = _finish(delta.finish_reason)
delta = await anext(stream, None)
+ if finish is None:
+ raise GatewayError(
+ "The provider stream ended before completion.", status=502, code="incomplete",
+ )
except Exception as exc: # noqa: BLE001 — becomes the runtime's HTTP error; _failure logs it
raise _report_failure(grant, model, _failure(grant.provider, exc, model), signal) from exc
message: dict[str, Any] = {"role": "assistant", "content": "".join(text) or None}
@@ -1321,6 +1414,9 @@ def reset() -> None:
slot.retired = True
if slot.users == 0:
_close_later(slot.brain)
+ from jarvis.agent_runtimes import anthropic_history
+
+ anthropic_history.reset()
with _LOCK:
_GRANTS.clear()
_TOKENS.clear()
@@ -1328,4 +1424,5 @@ def reset() -> None:
_SIGNATURES.clear()
_MODEL_LIMITS.clear()
_FAILURES.clear()
+ _TURN_EFFORTS.clear()
_COOLDOWNS.clear()
diff --git a/jarvis/agent_runtimes/model_map.py b/jarvis/agent_runtimes/model_map.py
index 6601ea29c2..41ee965e5c 100644
--- a/jarvis/agent_runtimes/model_map.py
+++ b/jarvis/agent_runtimes/model_map.py
@@ -156,7 +156,16 @@ def login_token_for(provider: str, account_id: str = "") -> str | None:
def login_providers() -> list[str]:
"""Usable providers that answer on a Claude login, not an API key: the
picker labels them as billed extra usage. Blocking (keyring)."""
- return [name for name in _ENDPOINTS if login_token_for(name)]
+ return [name for name in _ENDPOINTS if _login_available(login_token_for(name))]
+
+
+def _login_available(token: str | None) -> bool:
+ """Use the same cached, non-inference billing check as runtime startup."""
+ if not token:
+ return False
+ from jarvis.agent_runtimes.provider_errors import login_blocked
+
+ return login_blocked(token, "") is None
def access_choices() -> dict[str, list[str]]:
@@ -170,7 +179,7 @@ def access_choices() -> dict[str, list[str]]:
found: list[str] = []
if _saved_key(name, endpoint) is not None:
found.append("api")
- if claude_login_token() is not None:
+ if _login_available(claude_login_token()):
found.append("subscription")
if found:
choices[name] = found
@@ -216,6 +225,7 @@ def usable_providers(config: Any) -> list[str]:
for provider in sorted(_ENDPOINTS, key=lambda name: (not _ENDPOINTS[name].subscription, name)):
try:
_checked_model(config, provider, "probe")
+ _check_login_billing(provider, "", "")
except RouteUnavailable: # not connected: the provider is simply not offered
continue
except Exception: # noqa: BLE001 — one unreadable provider must not empty the list
@@ -365,6 +375,7 @@ def route_for(
token = gateway.grant_token(
agent_id, provider, account_id,
scope=home_key(agent_id, session_id) if session_id else "",
+ require_active_turn=True,
)
gateway.register_model(token, chosen, limits)
return ModelRoute(
diff --git a/jarvis/agent_runtimes/openclaw.py b/jarvis/agent_runtimes/openclaw.py
index 5e8295f848..22d3a80662 100644
--- a/jarvis/agent_runtimes/openclaw.py
+++ b/jarvis/agent_runtimes/openclaw.py
@@ -282,6 +282,7 @@ class _Gateway:
tracker: Any
last_used: float = field(default_factory=time.monotonic)
in_use: int = 0
+ poisoned: bool = False
def alive(self) -> bool:
return self.proc.returncode is None
@@ -514,6 +515,9 @@ class OpenClawRuntime:
# create dreaming jobs at startup even with heartbeat disabled.
# Jarvis owns memory and scheduling; neither may call this key.
"cron": {"enabled": False},
+ # Detached workshop reviews spend the model key after a turn,
+ # independently of cron and heartbeat. Jarvis owns that work.
+ "skills": {"workshop": {"autonomous": {"mode": "off"}}},
"plugins": {
"deny": list(_DENIED_PLUGINS),
"slots": {"memory": "none"},
@@ -612,6 +616,7 @@ class OpenClawRuntime:
acp_resume=None,
vendor_session=session_key(turn.agent_id, turn.session_id),
release=release,
+ invalidate=lambda: setattr(gateway, "poisoned", True),
)
async def _ensure_gateway(
@@ -642,7 +647,7 @@ class OpenClawRuntime:
write_json_if_changed, home / "openclaw.json", config
)
if current is not None and (
- not current.alive() or current.env_hash != env_hash or changed
+ not current.alive() or current.poisoned or current.env_hash != env_hash or changed
):
await self._stop_gateway(current)
current = None
diff --git a/jarvis/agent_runtimes/provider_errors.py b/jarvis/agent_runtimes/provider_errors.py
index 64b64853c7..508da1a21c 100644
--- a/jarvis/agent_runtimes/provider_errors.py
+++ b/jarvis/agent_runtimes/provider_errors.py
@@ -146,6 +146,7 @@ TERMINAL_CODES: Final[frozenset[str]] = frozenset(
"invalid_tool_schema",
"tools_unsupported",
"rate_limited",
+ "incomplete",
}
)
@@ -436,6 +437,7 @@ def login_refusal(usage: Any, model: str) -> Refusal | None:
#: The last usage report per login, so routing does not ask on every turn.
_REPORTS: dict[str, tuple[float, Any]] = {}
_REPORT_TTL_S: Final = 120.0
+_REPORT_FAILURE_TTL_S: Final = 10.0
def _claude_usage(token: str) -> Any:
@@ -473,12 +475,14 @@ def login_blocked(token: str, model: str) -> Refusal | None:
key = hashlib.sha256(token.encode()).hexdigest()
cached = _REPORTS.get(key)
- if cached is not None and time.monotonic() - cached[0] < _REPORT_TTL_S:
+ ttl = _REPORT_FAILURE_TTL_S if cached is not None and cached[1] is None else _REPORT_TTL_S
+ if cached is not None and time.monotonic() - cached[0] < ttl:
usage = cached[1]
else:
usage = _claude_usage(token)
- if usage is not None:
- _REPORTS[key] = (time.monotonic(), usage)
+ # Availability is read by several picker lists. A report outage is
+ # unknown, not blocked; briefly cache it to avoid repeated timeouts.
+ _REPORTS[key] = (time.monotonic(), usage)
return login_refusal(usage, model)
diff --git a/jarvis/agentic_ide/activity.py b/jarvis/agentic_ide/activity.py
index 7ca70a89d2..c7023d23dc 100644
--- a/jarvis/agentic_ide/activity.py
+++ b/jarvis/agentic_ide/activity.py
@@ -24,7 +24,8 @@ if TYPE_CHECKING: # pragma: no cover - typing only
#: ``working`` — its screen is moving.
#: ``waiting`` — alive and still.
#: ``asking`` — still, AND showing a question or a choice.
-#: ``failed`` — its agent could not be started.
+#: ``failed`` — its agent could not be started, or (on a live process) its last
+#: turn ended in an error; the CLI is then back at its prompt.
#: ``exited`` — its process is gone.
Activity = Literal[
"starting", "working", "waiting", "asking", "failed", "exited", "stopped", "unknown",
diff --git a/jarvis/agentic_ide/control.py b/jarvis/agentic_ide/control.py
index 8610edbfe6..5f939e6a88 100644
--- a/jarvis/agentic_ide/control.py
+++ b/jarvis/agentic_ide/control.py
@@ -145,7 +145,9 @@ class CodingSessionControl:
raise SessionError("A nonempty assignment is required.")
if term.status in ("dead", "error", "exited"):
raise SessionError(
- "The coding terminal has stopped. Inspect it before opening another."
+ "The coding terminal has stopped; nothing was sent. Restart the same "
+ "terminal (workspace-orchestrate restart, or its pane's Restart button; it "
+ "continues its own conversation) instead of opening another."
)
from jarvis.core.image_references import ImageReferenceError, get_store
diff --git a/jarvis/agentic_ide/orchestration.py b/jarvis/agentic_ide/orchestration.py
index 7b3b5606b0..a6be2d5b5d 100644
--- a/jarvis/agentic_ide/orchestration.py
+++ b/jarvis/agentic_ide/orchestration.py
@@ -57,7 +57,10 @@ _TARGET_KEYS = ("project_id", "workspace_id", "terminal_id")
# A send of the same prompt to the same pane counts as a retry only this soon;
# a later "continue" or "/compact" is a new instruction and must be typed.
_RETRY_WINDOW_S = 120
-_PANE_ACTIONS = frozenset({"observe", "respond", "keys", "interrupt", "close"})
+_PANE_ACTIONS = frozenset({"observe", "respond", "keys", "interrupt", "restart", "close"})
+# A pane whose agent process is gone (exited, or could not start). A send there
+# types nothing; restart brings the same pane back on its own conversation.
+_STOPPED = frozenset({"exited", "error"})
_WORKSPACE_ACTIONS = frozenset({"open_workspace", "restore", "show"})
# How many times one request may be re-attempted after receipts proving that
# nothing was typed (the pane was busy, closed, or not a coding pane).
@@ -294,6 +297,9 @@ class WorkspaceOrchestrator:
**({"availability": "busy", "brief_writing": True}
if term.name in writing else {}),
"accepts_tasks": accepts_prompts(term.agent) and not term.archived,
+ "restartable": accepts_prompts(term.agent)
+ and not term.archived
+ and term.status in _STOPPED,
}
for term in owner.terminals
]
@@ -710,7 +716,8 @@ class WorkspaceOrchestrator:
observe reads the screen and the newest recorded events; respond answers
the question or permission prompt the agent is showing; keys presses
keys (menus, Shift+Tab, Escape); interrupt stops the current turn
- (Escape, the pane's Stop button); close stops and removes the pane.
+ (Escape, the pane's Stop button); restart starts a stopped or failed
+ pane's agent again on its own conversation; close stops and removes it.
"""
action = str(args.get("action"))
picked = await self._pane(args)
@@ -758,6 +765,8 @@ class WorkspaceOrchestrator:
):
raise SessionError(f"{term.name} is not running.")
return {"status": "pressed", "target": target, "keys": keys, "agent": term.name}
+ if action == "restart":
+ return await self._restart(owner, term, target)
if action == "close":
from .fleet_actions import terminals_closed_event
@@ -780,6 +789,42 @@ class WorkspaceOrchestrator:
return {"status": "not_accepted", "target": target, "reason": str(exc)}
raise ValueError("Unknown pane action.")
+ async def _restart(self, owner: Any, term: Any, target: dict[str, str]) -> dict[str, Any]:
+ """Bring a stopped pane back under the same IDs, and say what happened.
+
+ Idempotent by state: a running agent is left alone and the receipt
+ says so, so a retried or duplicated restart never starts a second
+ process or cuts a turn short. Nothing is typed; the task follows as an
+ ordinary send, and a send refused earlier as stopped is retried with
+ its own request_id.
+ """
+ before = term.status
+ term, started = await self.registry.restart_terminal(
+ target["terminal_id"], owner.id
+ )
+ receipt: dict[str, Any] = {
+ "target": target,
+ "agent": term.name,
+ "cli": term.agent,
+ "previous_status": before,
+ "process_status": term.status,
+ "input_written": False,
+ }
+ if not started:
+ return {
+ **receipt,
+ "status": "already_running",
+ "reason": "The agent is running; nothing was restarted. A failed last turn "
+ "does not stop it: send the task to the same terminal_id.",
+ }
+ return {
+ **receipt,
+ "status": "restarted",
+ "conversation": "continued" if term.resumed else "fresh",
+ "next": "Send the task to the same terminal_id. A send refused earlier because "
+ "the agent was stopped is retried with the same request_id and prompt.",
+ }
+
async def workspace_action(self, args: dict[str, Any], *, trace_id: str = "") -> dict:
"""Open a new workspace, restore a closed one, or bring one on screen."""
from .session import workspace_changed_event
@@ -1354,6 +1399,21 @@ class WorkspaceOrchestrator:
"target": target,
"reason": "This session does not accept coding tasks.",
}
+ if action == "send" and found[1].status in _STOPPED:
+ # Pre-write, so the same request_id is retried after a restart.
+ return {
+ "status": "not_accepted",
+ "target": target,
+ "reason": f"{found[1].name} is not running ({found[1].status}); nothing "
+ "was sent.",
+ "completed": False,
+ "input_written": False,
+ "stopped": True,
+ "restartable": True,
+ "next": "To deliver it to this agent, call restart with this terminal_id (it "
+ "continues the agent's own conversation and stops nothing), then "
+ "resend this request with the same request_id and prompt.",
+ }
# The low-level gateway enforces idle/asking state and refuses exited
# agents. It never falls through to a shell or a different pane.
try:
diff --git a/jarvis/agentic_ide/session.py b/jarvis/agentic_ide/session.py
index 91655d0544..8544ed4ec7 100644
--- a/jarvis/agentic_ide/session.py
+++ b/jarvis/agentic_ide/session.py
@@ -2553,6 +2553,71 @@ class Registry:
self._start_in_background(session, term)
return term
+ async def restart_terminal(
+ self, wanted: str, workspace_id: str | None = None
+ ) -> tuple[Terminal, bool]:
+ """Start a stopped or failed coding pane again, as the user asked.
+
+ The explicit counterpart of :meth:`start_pending`'s "an exited or failed
+ one is the user's call". The pane keeps its identity (``pane:<history
+ id>``), account and launch picks, and the one start path
+ (:meth:`_attach_locked`) continues its own conversation where the CLI
+ can, fresh only when none exists. Whoever is watching the pane sees the
+ new process. Nothing is typed into it.
+
+ A running agent is never restarted, which would throw away work in
+ flight: the call is then a no-op, so a repeated request starts at most
+ one process. Returns the pane and whether a new process was started.
+ """
+ found = self.find_terminal(wanted, workspace_id)
+ if found is None:
+ raise self._unknown_terminal(wanted)
+ session, term = found
+ if not accepts_prompts(term.agent):
+ raise SessionError(
+ f"{term.name} is a {agent_display(term.agent).lower()}, not a coding agent; "
+ "only coding agents are restarted here."
+ )
+
+ async def _discard(_data: Any) -> None:
+ return None
+
+ async with term.attach_lock:
+ if term.status == "live" and term.pty_id:
+ return term, False
+ if term.placing:
+ raise SessionNotReady(term.placing)
+ generation, previous, error = term.process_generation, term.status, term.error
+ if not term.pty_id:
+ # A pty still recorded is re-joined below, never restarted.
+ term.status = "pending"
+ term.error = ""
+ term.stopping = False
+ viewer = term.viewer_output or _discard
+ try:
+ await self._attach_locked(
+ "pane:" + term.history_id,
+ term.pty_cols or term.transcript.cols,
+ term.pty_rows or term.transcript.rows,
+ viewer,
+ term.viewer_exit or _discard,
+ workspace_id=session.id,
+ )
+ except BaseException:
+ if term.status == "pending" and not term.pty_id:
+ # Refused before any start: the pane is as it was.
+ term.status, term.error = previous, error
+ raise
+ finally:
+ if viewer is _discard:
+ self.detach("pane:" + term.history_id, workspace_id=session.id, viewer=_discard)
+ logger.info(
+ "Agentic IDE: restarted {} on request ({})",
+ term.name,
+ "its own conversation" if term.resumed else "a fresh conversation",
+ )
+ return term, term.process_generation != generation
+
def _host_went_away(self) -> bool:
"""Did the PTY host this process was attached to just drop away?"""
current = self._pty
@@ -6755,7 +6820,8 @@ class Registry:
)
if busy in ("failed", "exited"):
raise SessionError(
- f"{term.name} is not running ({busy}); nothing was sent."
+ f"{term.name} is not running ({busy}); nothing was sent. Restart it "
+ "(it continues its own conversation), then send again."
)
if busy == "asking" and when_busy != "refuse":
raise SessionError(
@@ -6810,6 +6876,12 @@ class Registry:
term.last_submit_at is not None
and term.submit_generation == term.process_generation
)
+ if activity == "failed" and term.status == "live" and term.pty_id:
+ # The last TURN failed (Codex records ``task_complete`` with an
+ # error), not the process: the CLI is back at its prompt and takes
+ # the next instruction. Refusing it as "not running" left a failed
+ # Codex pane unreachable for good (live 2026-10-07).
+ return ""
if activity in ("working", "asking", "failed", "exited"):
return activity
# An interrupted turn ("stopped") sits at its prompt like a finished
diff --git a/jarvis/appshot/editor_window.py b/jarvis/appshot/editor_window.py
index 0fa3549425..8ab0632510 100644
--- a/jarvis/appshot/editor_window.py
+++ b/jarvis/appshot/editor_window.py
@@ -25,6 +25,8 @@ EDITOR_VIEW = "appshot-editor"
#: Appshot ids are uuid hex; nothing else may reach the window URL.
_ID = re.compile(r"^[0-9a-f]{8,64}$")
+#: Recording ids are exactly one uuid hex (``jarvis.appshot.recording``).
+_RECORDING_ID = re.compile(r"^[0-9a-f]{32}$")
_opener: Callable[[str], dict[str, Any]] | None = None
_prewarm: Callable[[], dict[str, Any]] | None = None
@@ -63,11 +65,28 @@ def can_open_window() -> bool:
async def open_editor_window(shot_id: str) -> bool:
"""Open the editor window on ``shot_id``. ``False`` = use the page instead."""
+ if not _ID.match(shot_id or ""):
+ return False
+ return await _open(f"appshot={shot_id}")
+
+
+async def open_recording_window(recording_id: str) -> bool:
+ """Open the same window as a video player and trimmer on a recording.
+
+ ``False`` where the shell cannot open windows; the caller then falls back
+ to the system's video player.
+ """
+ if not _RECORDING_ID.match(recording_id or ""):
+ return False
+ return await _open(f"recording={recording_id}")
+
+
+async def _open(query: str) -> bool:
opener = _opener
- if opener is None or not _ID.match(shot_id or ""):
+ if opener is None:
return False
try:
- result = await asyncio.to_thread(opener, f"appshot={shot_id}")
+ result = await asyncio.to_thread(opener, query)
except Exception: # noqa: BLE001 - the page editor is the honest fallback
log.warning("appshot: the editor window could not be opened", exc_info=True)
return False
@@ -81,6 +100,7 @@ __all__ = [
"EDITOR_VIEW",
"can_open_window",
"open_editor_window",
+ "open_recording_window",
"prewarm_editor_window",
"register_window_opener",
]
diff --git a/jarvis/appshot/hotkey.py b/jarvis/appshot/hotkey.py
index c2f72e0920..3a0ba0ae5f 100644
--- a/jarvis/appshot/hotkey.py
+++ b/jarvis/appshot/hotkey.py
@@ -18,6 +18,7 @@ from __future__ import annotations
import asyncio
import contextlib
import logging
+import sys
from dataclasses import dataclass
from typing import Any
@@ -161,6 +162,7 @@ class AppshotShortcut:
self._trigger_task: asyncio.Task[None] | None = None
self._trigger: Any | None = None
self._combo_scopes: set[str] = set()
+ self._warm_task: asyncio.Task[None] | None = None
self._busy = False
self._recording_busy = False
# One press that arrives while a toggle is still running. Dropping it
@@ -246,6 +248,16 @@ class AppshotShortcut:
self._trigger_task = asyncio.get_running_loop().create_task(
self._run_combos(combos), name="appshot-hotkey"
)
+ if owns and bool(getattr(getattr(config, "screen_context", None), "enabled", False)):
+ self._warm_task = asyncio.create_task(
+ self._warm_surfaces(bool(getattr(config.appshot, "effect", True))),
+ name="appshot-surfaces-warm",
+ )
+ from jarvis.appshot.recording import warm_recording_service
+
+ await warm_recording_service(
+ owns and bool(getattr(getattr(config, "screen_context", None), "enabled", False))
+ )
except Exception as exc: # noqa: BLE001 - a bad shortcut must not break boot
log.warning("appshot: shortcut could not be armed", exc_info=True)
await self.stop() # never leave half of a failed arming running
@@ -265,6 +277,9 @@ class AppshotShortcut:
return self.status
async def stop(self) -> None:
+ from jarvis.appshot.recording import warm_recording_service
+
+ await warm_recording_service(False)
watchers, self._watchers = self._watchers, []
for watcher in watchers:
await asyncio.to_thread(watcher.stop)
@@ -279,6 +294,36 @@ class AppshotShortcut:
scope: ShortcutStatus(status.hotkey, False, "The shortcut is stopped.")
for scope, status in self._statuses.items()
}
+ warm, self._warm_task = self._warm_task, None
+ if warm is not None:
+ warm.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await warm
+ from jarvis.appshot.picker_host import close_picker_host
+ from jarvis.cu.indicator.controller import get_indicator_controller
+
+ await close_picker_host()
+ controller = get_indicator_controller()
+ if controller is not None:
+ await controller.warm_for_appshots(False)
+
+ async def _warm_surfaces(self, effect: bool) -> None:
+ from jarvis.appshot.picker_host import get_picker_host
+ from jarvis.cu.indicator.controller import wire_cu_indicator
+
+ try:
+ await get_picker_host().prewarm()
+ except Exception:
+ log.warning("appshot: picker preparation failed; next press will retry", exc_info=True)
+ if effect:
+ try:
+ controller = wire_cu_indicator(self._bus)
+ if controller is not None:
+ await controller.warm_for_appshots(True)
+ except Exception:
+ log.warning(
+ "appshot: shutter preparation failed; next capture will retry", exc_info=True
+ )
async def _arm_gesture(self, scope: str, hotkey: str) -> ShortcutStatus:
from jarvis.appshot.gesture import ( # noqa: PLC0415
@@ -465,10 +510,7 @@ class AppshotShortcut:
from jarvis.core.events import ConfigReloaded # noqa: PLC0415
async def _on_reload(event: ConfigReloaded) -> None:
- if any(
- key.startswith("appshot.") or key == "screen_context.enabled"
- for key in event.changed_keys
- ):
+ if any(key.startswith(("appshot.", "screen_context.")) for key in event.changed_keys):
await self.reload()
self._bus.subscribe(ConfigReloaded, _on_reload)
@@ -491,6 +533,26 @@ async def start_appshot_shortcut(bus: Any) -> AppshotShortcut:
return _shortcut
+async def stop_appshot_shortcut() -> None:
+ """Release the native helpers before the backend event loop closes."""
+ global _shortcut
+ shortcut, _shortcut = _shortcut, None
+ # A helper whose module was never imported never started: shutdown must
+ # not import (and initialize) it just to find nothing to close.
+ picker_host = sys.modules.get("jarvis.appshot.picker_host")
+ if shortcut is not None:
+ await shortcut.stop()
+ elif picker_host is not None:
+ await picker_host.close_picker_host()
+ indicator = sys.modules.get("jarvis.cu.indicator.controller")
+ controller = indicator.get_indicator_controller() if indicator is not None else None
+ if controller is not None:
+ await controller.close()
+ recording = sys.modules.get("jarvis.appshot.recording")
+ if recording is not None:
+ await recording.close_recording_service()
+
+
__all__ = [
"BOTH_ALT",
"BOTH_CTRL",
diff --git a/jarvis/appshot/media_library.py b/jarvis/appshot/media_library.py
new file mode 100644
index 0000000000..c2acdbff44
--- /dev/null
+++ b/jarvis/appshot/media_library.py
@@ -0,0 +1,146 @@
+"""One gallery for kept screenshots and already finalized screen recordings.
+
+Videos remain in the recorder's durable directory: no copying, migration or
+dependence on the last live recording session. Namespaced IDs keep video
+actions separate from screenshot edits. Video packages are loaded on demand.
+"""
+
+from __future__ import annotations
+
+import io
+import logging
+import math
+from dataclasses import dataclass
+from functools import lru_cache
+
+from jarvis.appshot import library, recording
+
+log = logging.getLogger(__name__)
+MAX_ENTRIES = library.MAX_ENTRIES
+_PREFIX = "recording_"
+_FALLBACK_POSTER = (
+ b'<svg xmlns="http://www.w3.org/2000/svg" width="480" height="270" viewBox="0 0 480 270">'
+ b'<rect width="480" height="270" rx="8" fill="#71717a"/>'
+ b'<path d="M220 105v60l50-30z" fill="white"/></svg>'
+)
+
+
+@dataclass(frozen=True, slots=True)
+class RecordingItem(library.LibraryItem):
+ duration_s: float = 0.0
+
+ def to_json(self) -> dict[str, object]:
+ return {**library.LibraryItem.to_json(self), "duration_s": self.duration_s}
+
+
+@lru_cache(maxsize=512)
+def _metadata(path: str, modified_ns: int, size: int) -> tuple[int, int, float]:
+ del modified_ns, size # Cache identity: replacing a file invalidates its metadata.
+ try:
+ import av
+
+ with av.open(path) as container:
+ stream = container.streams.video[0]
+ duration = (
+ float(stream.duration * stream.time_base)
+ if stream.duration is not None and stream.time_base is not None
+ else float(container.duration or 0) / 1_000_000
+ )
+ duration = max(0.0, duration) if math.isfinite(duration) else 0.0
+ return stream.width, stream.height, duration
+ except Exception:
+ # A missing optional decoder or a damaged file must not hide the rest
+ # of the gallery. The original remains available for download.
+ log.debug("appshot library: recording metadata unavailable", exc_info=True)
+ return 0, 0, 0.0
+
+
+def _video_item(recording_id: str) -> RecordingItem | None:
+ path = recording.recording_file(recording_id)
+ if path is None:
+ return None
+ try:
+ stat = path.stat()
+ except OSError:
+ log.debug("appshot library: recording disappeared during listing", exc_info=True)
+ return None
+ if not stat.st_size:
+ return None
+ width, height, duration = _metadata(str(path), stat.st_mtime_ns, stat.st_size)
+ return RecordingItem(
+ id=f"{_PREFIX}{recording_id}", variant="original", path=path, mime="video/mp4",
+ width=width, height=height, label="Screen recording", app_name="", trigger="recording",
+ taken_at=stat.st_mtime, edited_at=0.0, has_edit=False, duration_s=duration,
+ )
+
+
+def list_items() -> list[library.LibraryItem]:
+ items = library.list_items()
+ for path in recording.recording_dir().glob("*.mp4"):
+ item = _video_item(path.stem)
+ if item is not None:
+ items.append(item)
+ # Stable sort retains each screenshot's edited/original ordering.
+ return sorted(items, key=lambda item: item.taken_at, reverse=True)
+
+
+def get_item(item_id: str, variant: library.Variant) -> library.LibraryItem | None:
+ if item_id.startswith(_PREFIX):
+ return _video_item(item_id[len(_PREFIX):]) if variant == "original" else None
+ return library.get_item(item_id, variant)
+
+
+@lru_cache(maxsize=64)
+def _poster(path: str, modified_ns: int, size: int) -> tuple[bytes, str]:
+ del modified_ns, size
+ try:
+ import av
+
+ with av.open(path) as container:
+ stream = container.streams.video[0]
+ stream.codec_context.thread_count = 1
+ frame = next(container.decode(stream))
+ ratio = min(1.0, library.THUMB_EDGE / max(frame.width, frame.height))
+ frame = frame.reformat(
+ width=max(1, round(frame.width * ratio)),
+ height=max(1, round(frame.height * ratio)),
+ format="rgb24",
+ )
+ out = io.BytesIO()
+ frame.to_image().save(out, "JPEG", quality=82)
+ return out.getvalue(), "image/jpeg"
+ except Exception:
+ log.debug("appshot library: recording preview unavailable", exc_info=True)
+ return _FALLBACK_POSTER, "image/svg+xml"
+
+
+def thumbnail(item: library.LibraryItem) -> tuple[bytes, str]:
+ if item.mime != "video/mp4":
+ return library.thumbnail(item)
+ stat = item.path.stat()
+ return _poster(str(item.path), stat.st_mtime_ns, stat.st_size)
+
+
+def delete(item_id: str, variant: library.Variant) -> bool:
+ if not item_id.startswith(_PREFIX):
+ return library.delete(item_id, variant)
+ if variant != "original":
+ return False
+ path = recording.recording_file(item_id[len(_PREFIX):])
+ if path is None:
+ return False
+ try:
+ path.unlink()
+ except FileNotFoundError: # Deleted between resolution and removal.
+ return False
+ _metadata.cache_clear()
+ _poster.cache_clear()
+ return True
+
+
+def clear() -> int:
+ removed = library.clear()
+ # Only finalized, valid recorder IDs; partial files and foreign entries stay.
+ for path in recording.recording_dir().glob("*.mp4"):
+ removed += int(delete(f"{_PREFIX}{path.stem}", "original"))
+ return removed
diff --git a/jarvis/appshot/picker/__init__.py b/jarvis/appshot/picker/__init__.py
index 52eefd25cb..63bad8c9d9 100644
--- a/jarvis/appshot/picker/__init__.py
+++ b/jarvis/appshot/picker/__init__.py
@@ -1,4 +1,4 @@
-"""The area picker for region appshots — a one-shot PySide6 sidecar.
+"""The area picker for region appshots — an isolated PySide6 sidecar.
``python -m jarvis.appshot.picker --lang de`` dims every screen, lets the
user drag one rectangle, mark it up in place with a toolbar and prints
@@ -6,10 +6,15 @@ the result as one JSON line on stdout, then exits. The main process never
imports PySide6 (AP-26); it talks to this process through
:mod:`jarvis.appshot.region`.
+With ``--resident``, it starts in pixel-free standby. Each ``start`` command
+opens a fresh selection; finishing releases its windows and pixels while
+keeping Qt loaded. ``quit`` or parent EOF exits the process.
+
Wire format (one JSON object per line):
stdout::
+ {"event": "standby"} # resident runtime loaded
{"event": "ready"} # overlay is up
{"event": "marking"} # area chosen, toolbar up
{"event": "selection", "screen": {"x":…, "y":…, # the Qt screen,
@@ -22,6 +27,9 @@ stdout::
stdin::
+ {"cmd": "start", "language": "en", "monitors": [], # resident: fresh capture
+ "windows": []}
+ {"cmd": "quit"} # release the runtime
{"cmd": "layout", "monitors": [...], # mss monitors and the
"windows": [[l, t, w, h], ...]} # snap targets, top first
{"cmd": "cancel"} # e.g. a global Esc
@@ -38,6 +46,7 @@ from typing import Any
EXIT_NO_GUI = 3
EVENT_READY = "ready"
+EVENT_STANDBY = "standby"
EVENT_MARKING = "marking"
EVENT_SELECTION = "selection"
#: What finishing the selection asks for besides taking the appshot.
@@ -48,6 +57,8 @@ ACTION_EDIT = "edit"
ACTIONS = (ACTION_DONE, ACTION_COPY, ACTION_SAVE, ACTION_EDIT)
CMD_CANCEL = "cancel"
CMD_LAYOUT = "layout"
+CMD_START = "start"
+CMD_QUIT = "quit"
def encode(payload: dict[str, Any]) -> str:
@@ -71,8 +82,11 @@ __all__ = [
"ACTION_SAVE",
"CMD_CANCEL",
"CMD_LAYOUT",
+ "CMD_START",
+ "CMD_QUIT",
"EVENT_MARKING",
"EVENT_READY",
+ "EVENT_STANDBY",
"EVENT_SELECTION",
"EXIT_NO_GUI",
"decode",
diff --git a/jarvis/appshot/picker/__main__.py b/jarvis/appshot/picker/__main__.py
index 82aa136770..32e294a080 100644
--- a/jarvis/appshot/picker/__main__.py
+++ b/jarvis/appshot/picker/__main__.py
@@ -19,6 +19,7 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--hint", default="")
# The toolbar's tooltip language ([ui].language); English when absent.
parser.add_argument("--lang", default="en")
+ parser.add_argument("--resident", action="store_true")
args = parser.parse_args(argv)
try:
from jarvis.appshot.picker.renderer import run # noqa: PLC0415
@@ -28,7 +29,7 @@ def main(argv: list[str] | None = None) -> int:
"extra to select an area for an appshot.\n"
)
return EXIT_NO_GUI
- return run(args.lang)
+ return run(args.lang, resident=args.resident)
if __name__ == "__main__":
diff --git a/jarvis/appshot/picker/renderer.py b/jarvis/appshot/picker/renderer.py
index 555730d72b..5c97074a90 100644
--- a/jarvis/appshot/picker/renderer.py
+++ b/jarvis/appshot/picker/renderer.py
@@ -264,6 +264,12 @@ class _SelectWindow(QWidget):
self._patches = annotate.HidePatches(frozen, self._scale)
self._toolbar: annotate.Toolbar | None = None
+ def release_capture(self) -> None:
+ """Drop pixels explicitly, including the preview's bound-method cycle."""
+ self._frozen = None
+ self._patches = annotate.HidePatches(None, self._scale)
+ self._markup = _MarkupState()
+
# -- data from the main process ------------------------------------------
def set_layout(self, monitors: list[dict], windows: list[list[int]]) -> None:
"""Map capture-space window rects onto this screen's logical pixels."""
@@ -1075,10 +1081,14 @@ class _SelectWindow(QWidget):
class Picker(QObject):
"""Owns the per-screen windows and reports exactly one result."""
- def __init__(self, app: QApplication, *, language: str = "en") -> None:
+ def __init__(
+ self, app: QApplication, *, language: str = "en", resident: bool = False
+ ) -> None:
super().__init__()
self._app = app
self.language = language
+ self._resident = resident
+ self._started = False
self._windows: list[_SelectWindow] = []
self._focused: _SelectWindow | None = None
#: The window whose area is being marked up; the others stay inert.
@@ -1086,6 +1096,10 @@ class Picker(QObject):
self._done = False
def start(self) -> None:
+ if self._started:
+ return
+ self._started = True
+ self._done = False
screens = QGuiApplication.screens()
# Freeze every screen BEFORE any overlay window exists.
frozen = {id(s): self._grab(s) for s in screens}
@@ -1158,6 +1172,16 @@ class Picker(QObject):
win.hide()
# The overlay must be off the glass before the parent grabs.
self._app.processEvents()
+ # Keep only the GUI runtime warm. Bound Qt callbacks can outlive the
+ # native widget until GC, so explicitly release its pixels first.
+ for win in self._windows:
+ win.release_capture()
+ win.close()
+ win.deleteLater()
+ self._windows.clear()
+ self._focused = None
+ self.marking = None
+ self._started = False
if info is None or frac is None:
_emit({"event": wire.EVENT_SELECTION, "cancelled": True})
else:
@@ -1170,7 +1194,8 @@ class Picker(QObject):
if markup is not None:
payload["markup"] = markup
_emit(payload)
- self._app.quit()
+ if not self._resident:
+ self._app.quit()
@Slot(str)
def on_line(self, raw: str) -> None:
@@ -1178,7 +1203,13 @@ class Picker(QObject):
if payload is None:
return
cmd = payload.get("cmd")
- if cmd == wire.CMD_CANCEL:
+ if cmd == wire.CMD_START and self._resident and not self._started:
+ self.language = str(payload.get("language") or "en")
+ self.start()
+ self.set_layout(payload.get("monitors", []), payload.get("windows", []))
+ elif cmd == wire.CMD_QUIT:
+ self.on_eof()
+ elif cmd == wire.CMD_CANCEL and self._started:
self.finish(None, None)
elif cmd == wire.CMD_LAYOUT:
monitors = payload.get("monitors")
@@ -1188,7 +1219,9 @@ class Picker(QObject):
@Slot()
def on_eof(self) -> None:
- self.finish(None, None)
+ if self._started:
+ self.finish(None, None)
+ self._app.quit()
class _StdinPump(QObject):
@@ -1208,7 +1241,7 @@ class _StdinPump(QObject):
self.eof.emit()
-def run(language: str = "en") -> int:
+def run(language: str = "en", *, resident: bool = False) -> int:
"""Picker main loop. Returns the process exit code."""
QGuiApplication.setHighDpiScaleFactorRoundingPolicy(
Qt.HighDpiScaleFactorRoundingPolicy.PassThrough
@@ -1222,11 +1255,18 @@ def run(language: str = "en") -> int:
hide_from_dock() # macOS: no "Python" Dock icon for the resident picker
app.setQuitOnLastWindowClosed(False)
- picker = Picker(app, language=language)
+ if not QGuiApplication.screens():
+ return wire.EXIT_NO_GUI
+ picker = Picker(app, language=language, resident=resident)
pump = _StdinPump()
pump.line.connect(picker.on_line, Qt.ConnectionType.QueuedConnection)
pump.eof.connect(picker.on_eof, Qt.ConnectionType.QueuedConnection)
- picker.start()
+ if resident:
+ # Import/initialize Qt without taking pixels, creating windows or
+ # stealing focus. The next start command samples the current desktop.
+ _emit({"event": wire.EVENT_STANDBY})
+ else:
+ picker.start()
pump.start()
return app.exec()
diff --git a/jarvis/appshot/picker_host.py b/jarvis/appshot/picker_host.py
new file mode 100644
index 0000000000..bc37f6e89b
--- /dev/null
+++ b/jarvis/appshot/picker_host.py
@@ -0,0 +1,166 @@
+"""A warm, pixel-free picker runtime; one visible selection at a time.
+
+Qt lives in a child process. Only an explicit pick command takes a fresh
+desktop image; standby imports the GUI without opening a window. Readers are
+joined before reuse and a cancelled or timed-out request discards its process,
+so a late result can never complete the next request.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+import subprocess
+import time
+from typing import Any
+
+from jarvis.appshot import picker as wire
+from jarvis.appshot import region
+
+log = logging.getLogger(__name__)
+_START_TIMEOUT_S = 5.0
+
+
+class PickerHost:
+ def __init__(self) -> None:
+ self._lock = asyncio.Lock()
+ self._proc: subprocess.Popen[str] | None = None
+ self._active: asyncio.Task | None = None
+
+ async def prewarm(self) -> None:
+ if not region.picker_capability()[0]:
+ return
+ async with self._lock:
+ await self._ensure()
+
+ @staticmethod
+ def _standby(proc: subprocess.Popen[str]) -> bool:
+ assert proc.stdout is not None
+ for line in proc.stdout:
+ payload = wire.decode(line)
+ if payload and payload.get("event") == wire.EVENT_STANDBY:
+ return True
+ return False
+
+ @staticmethod
+ def _send(proc: subprocess.Popen[str], payload: dict) -> None:
+ if proc.stdin is None or proc.poll() is not None:
+ raise region.RegionUnavailable("The selection overlay stopped unexpectedly.")
+ proc.stdin.write(wire.encode(payload))
+ proc.stdin.flush()
+
+ async def _dispose(self) -> None:
+ proc, self._proc = self._proc, None
+ if proc is None:
+ return
+ try:
+ await asyncio.to_thread(self._send, proc, {"cmd": wire.CMD_QUIT})
+ except (OSError, ValueError, region.RegionUnavailable):
+ log.debug("appshot: picker pipe already closed during shutdown")
+ await asyncio.to_thread(region._reap, proc)
+
+ async def _ensure(self) -> subprocess.Popen[str]:
+ if self._proc is not None and self._proc.poll() is None:
+ return self._proc
+ await self._dispose()
+ spawn = asyncio.create_task(asyncio.to_thread(region._spawn, resident=True))
+ reader = None
+ try:
+ try:
+ self._proc = await asyncio.shield(spawn)
+ except asyncio.CancelledError:
+ # to_thread cannot cancel Popen: acquire the late child first.
+ self._proc = await spawn
+ raise
+ reader = asyncio.create_task(asyncio.to_thread(self._standby, self._proc))
+ ready = await asyncio.wait_for(asyncio.shield(reader), _START_TIMEOUT_S)
+ if not ready:
+ code = self._proc.poll()
+ raise region.RegionUnavailable(region._exit_message(code or 1))
+ return self._proc
+ except BaseException:
+ await self._dispose()
+ if reader is not None:
+ await reader # reap closes stdout, including a stuck startup
+ raise
+
+ async def pick(self, timeout_s: float, language: str) -> tuple[Any, int | None, bool]:
+ async with self._lock:
+ self._active = asyncio.current_task()
+ try:
+ return await self._pick(timeout_s, language)
+ finally:
+ self._active = None
+
+ async def _pick(self, timeout_s: float, language: str) -> tuple[Any, int | None, bool]:
+ started = time.perf_counter()
+ proc = await self._ensure()
+ try:
+ layout = await asyncio.to_thread(region.snap_layout)
+ except Exception:
+ log.debug("appshot: snap layout unavailable", exc_info=True)
+ layout = {}
+ escape = None
+ reader = None
+ payload = None
+ timed_out = False
+ try:
+ await asyncio.to_thread(
+ self._send, proc, {"cmd": wire.CMD_START, "language": language, **layout}
+ )
+ loop = asyncio.get_running_loop()
+ escape = loop.create_task(region._escape_cancels(proc), name="appshot-region-esc")
+
+ def marking() -> None:
+ loop.call_soon_threadsafe(escape.cancel)
+
+ def ready() -> None:
+ log.info(
+ "appshot: picker visible in %.1f ms", (time.perf_counter() - started) * 1000
+ )
+
+ reader = asyncio.create_task(
+ asyncio.to_thread(region._read_result, proc, marking, ready)
+ )
+ try:
+ payload = await asyncio.wait_for(asyncio.shield(reader), timeout_s)
+ except TimeoutError:
+ log.info("appshot: selection timed out; closing the picker")
+ timed_out = True
+ finally:
+ if escape is not None:
+ escape.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await escape
+ if payload is None:
+ await self._dispose()
+ if reader is not None:
+ await reader
+ return payload, proc.poll(), timed_out
+
+ async def close(self) -> None:
+ active = self._active
+ if active is not None and active is not asyncio.current_task():
+ active.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await active
+ async with self._lock:
+ await self._dispose()
+
+
+_host: PickerHost | None = None
+
+
+def get_picker_host() -> PickerHost:
+ global _host
+ if _host is None:
+ _host = PickerHost()
+ return _host
+
+
+async def close_picker_host() -> None:
+ global _host
+ host, _host = _host, None
+ if host is not None:
+ await host.close()
diff --git a/jarvis/appshot/recording.py b/jarvis/appshot/recording.py
index dfd60e1271..6abeb03e0d 100644
--- a/jarvis/appshot/recording.py
+++ b/jarvis/appshot/recording.py
@@ -49,7 +49,7 @@ def recent_recordings() -> list[dict[str, Any]]:
def capability() -> dict[str, Any]:
- from jarvis.platform.probes import display_present, is_wayland
+ from jarvis.platform.probes import display_present
from jarvis.platform.qt_sidecar import missing_system_library as qt_missing
detail = ""
@@ -58,8 +58,8 @@ def capability() -> dict[str, Any]:
detail = "Screen recording requires an interactive desktop."
elif not all(importlib.util.find_spec(name) for name in ("PySide6", "av", "numpy")):
detail = "Screen recording requires the desktop capture and video packages."
- elif is_wayland() and importlib.util.find_spec("PySide6.QtMultimedia") is None:
- detail = "Wayland recording requires Qt Multimedia, PipeWire and a ScreenCast portal."
+ elif importlib.util.find_spec("PySide6.QtMultimedia") is None:
+ detail = "Screen recording requires the desktop multimedia package."
elif qt_missing():
detail = f"{qt_missing()}."
elif sys.platform == "darwin":
@@ -68,7 +68,10 @@ def capability() -> dict[str, Any]:
permission = not screen_access.state_allows_capture(screen_access.screen_recording_state())
if permission:
detail = "Allow Screen Recording for Personal Jarvis in macOS System Settings."
- return {"available": not detail, "detail": detail, "permission_required": permission}
+ from jarvis.appshot.recording_audio import capability as audio_capability
+
+ return {"available": not detail, "detail": detail, "permission_required": permission,
+ "system_audio": audio_capability()}
class RecordingService:
@@ -80,6 +83,43 @@ class RecordingService:
self._reader: asyncio.Task[None] | None = None
self._state: dict[str, Any] = {"phase": "idle", "id": "", "message": ""}
self._started: float | None = None
+ from jarvis.appshot.recording_runtime import RecordingRuntime
+
+ self._runtime = RecordingRuntime()
+ self._warm_enabled = False
+ self._warm_task: asyncio.Task[None] | None = None
+ self._requested: float | None = None
+
+ async def set_warm(self, enabled: bool) -> None:
+ """Prepare after boot; disabling/reloading shortcuts never stops an active video."""
+ self._warm_enabled = enabled
+ if enabled:
+ self._schedule_warm()
+ return
+ task, self._warm_task = self._warm_task, None
+ if task is not None:
+ task.cancel()
+ with contextlib.suppress(asyncio.CancelledError):
+ await task
+ await self._runtime.close()
+
+ def _schedule_warm(self) -> None:
+ if self._process is None and (self._warm_task is None or self._warm_task.done()):
+ self._warm_task = asyncio.create_task(self._prepare(), name="appshot-recorder-warm")
+
+ async def _prepare(self) -> None:
+ try:
+ ready = await asyncio.to_thread(capability)
+ if ready["available"] or ready.get("permission_required"):
+ await self._runtime.prewarm()
+ except Exception:
+ log.warning(
+ "appshot: recorder preparation failed; next start will retry", exc_info=True
+ )
+
+ async def close(self) -> None:
+ await self.set_warm(False)
+ await self.stop()
def status(self) -> dict[str, Any]:
state = dict(self._state)
@@ -89,7 +129,6 @@ class RecordingService:
async def start(self) -> dict[str, Any]:
from jarvis.core.config import load_config
- from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS
async with self._lock:
if self._process is not None:
@@ -110,36 +149,29 @@ class RecordingService:
ready = await asyncio.to_thread(capability)
if not ready["available"]:
raise ValueError(ready["detail"])
+ from jarvis.appshot.recording_audio import capability as audio_capability
+ from jarvis.appshot.recording_options import RecordingOptions
+
+ options = RecordingOptions.from_config(getattr(config, "appshot", None))
+ if options.system_audio:
+ audio_ready = audio_capability()
+ if not audio_ready["available"]:
+ raise ValueError(audio_ready["detail"])
recording_id = uuid.uuid4().hex
folder = recording_dir()
await asyncio.to_thread(folder.mkdir, parents=True, exist_ok=True)
self._started = None
self._state = {"phase": "selecting", "id": recording_id, "message": ""}
- # A cancelled HTTP request must not lose ownership during spawn.
- spawn = asyncio.create_task(
- asyncio.create_subprocess_exec(
- sys.executable,
- "-m",
- "jarvis.appshot.recording_worker",
- "--output",
- str(folder / f"{recording_id}.mp4"),
- "--language",
- str(getattr(config.ui, "language", "en")),
- stdin=asyncio.subprocess.PIPE,
- stdout=asyncio.subprocess.PIPE,
- stderr=asyncio.subprocess.DEVNULL,
- creationflags=NO_WINDOW_CREATIONFLAGS,
- )
- )
+ self._requested = time.perf_counter()
try:
- self._process = await asyncio.shield(spawn)
+ self._process = await self._runtime.take(
+ folder / f"{recording_id}.mp4", str(getattr(config.ui, "language", "en")),
+ options=options.model_dump(),
+ )
except asyncio.CancelledError:
- process = await spawn
- process.kill()
- await process.wait()
self._state["phase"] = "cancelled"
raise
- except OSError as exc:
+ except (OSError, TimeoutError) as exc:
self._state.update(phase="error", message="The recorder could not start.")
log.exception("appshot: recorder spawn failed")
raise ValueError(self._state["message"]) from exc
@@ -148,6 +180,7 @@ class RecordingService:
async def _read(self, process: asyncio.subprocess.Process) -> None:
assert process.stdout is not None
+ preview = None
try:
async for raw in process.stdout:
try:
@@ -158,23 +191,37 @@ class RecordingService:
if not isinstance(event, dict):
continue
phase = event.get("phase")
+ if phase == "selection_ready" and self._requested is not None:
+ log.info(
+ "appshot: recording picker visible in %.1f ms",
+ (time.perf_counter() - self._requested) * 1000,
+ )
+ continue
if phase not in {"selecting", "recording", "saved", "cancelled", "error"}:
continue
self._state.update(
{
key: event[key]
- for key in ("phase", "message", "width", "height", "duration_s")
+ for key in ("phase", "message", "width", "height", "duration_s", "fps",
+ "bitrate_mbps", "system_audio", "display_name", "refresh_hz",
+ "dropped_frames")
if key in event
}
)
if phase == "recording" and self._started is None:
self._started = time.monotonic()
+ if phase == "saved" and event.get("preview"):
+ preview = event
code = await process.wait()
if self._state["phase"] in ACTIVE_PHASES or (
code != 0 and self._state["phase"] != "error"
):
self._state.update(phase="error", message="The recorder stopped unexpectedly.")
log.warning("appshot: recorder exited with code %s", code)
+ elif preview is not None and self._state["phase"] == "saved":
+ from jarvis.appshot.recording_cards import show_recording_preview
+
+ await show_recording_preview(self._state["id"], preview)
if self._state["phase"] == "saved":
await _trim_old_captures()
except Exception:
@@ -187,6 +234,9 @@ class RecordingService:
if process.stdin:
process.stdin.close()
self._process = None
+ self._runtime.release()
+ if self._warm_enabled:
+ self._schedule_warm()
async def stop(self) -> dict[str, Any]:
async with self._lock:
@@ -237,3 +287,17 @@ def get_recording_service() -> RecordingService:
if _service is None:
_service = RecordingService()
return _service
+
+
+async def warm_recording_service(enabled: bool) -> None:
+ if enabled:
+ await get_recording_service().set_warm(True)
+ elif _service is not None:
+ await _service.set_warm(False)
+
+
+async def close_recording_service() -> None:
+ global _service
+ service, _service = _service, None
+ if service is not None:
+ await service.close()
diff --git a/jarvis/appshot/recording_audio.py b/jarvis/appshot/recording_audio.py
new file mode 100644
index 0000000000..17c3f8c0fa
--- /dev/null
+++ b/jarvis/appshot/recording_audio.py
@@ -0,0 +1,91 @@
+"""Opt-in output-loopback capture owned by the disposable recording process.
+
+No microphone fallback: only the default playback device's loopback is opened.
+The device, recorder and native backend are confined to one capture thread.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import logging
+import queue
+import sys
+import threading
+import time
+from typing import Any
+
+log = logging.getLogger(__name__)
+SAMPLE_RATE = 48000
+BLOCK_FRAMES = 960
+
+
+def capability() -> dict[str, Any]:
+ """Package/platform readiness only; never open a device during a health poll."""
+ supported = sys.platform in {"win32", "linux"}
+ installed = importlib.util.find_spec("soundcard") is not None
+ reason = "" if supported and installed else (
+ "System audio recording is not available on this operating system."
+ if not supported else "System audio recording requires the desktop audio package."
+ )
+ return {"available": supported and installed, "detail": reason}
+
+
+class SystemAudioCapture:
+ """Bounded PCM queue; startup and shutdown are bounded by the sidecar owner."""
+
+ def __init__(self, epoch: float) -> None:
+ self.epoch = epoch
+ self.chunks: queue.Queue[tuple[Any, int]] = queue.Queue(maxsize=128)
+ self.ready = threading.Event()
+ self.stopping = threading.Event()
+ self.done = threading.Event()
+ self.error = ""
+ self._thread = threading.Thread(target=self._run, name="appshot-audio", daemon=True)
+
+ def start(self) -> None:
+ self._thread.start()
+
+ def stop(self) -> None:
+ self.stopping.set()
+ self._thread.join(timeout=2)
+ if self._thread.is_alive():
+ self.error = "System audio did not stop in time. The recording could not be saved."
+ log.warning("appshot: loopback capture shutdown timed out")
+
+ def _run(self) -> None:
+ try:
+ import numpy as np
+ import soundcard as sc
+
+ speaker = sc.default_speaker()
+ if speaker is None:
+ raise RuntimeError("No default playback device.")
+ # Exact identity matters: fuzzy matching must never select a microphone.
+ loopbacks = sc.all_microphones(include_loopback=True)
+ source = next((item for item in loopbacks
+ if item.isloopback and item.id == speaker.id), None)
+ if source is None and sys.platform == "linux":
+ source = next((item for item in loopbacks
+ if item.isloopback and item.id == f"{speaker.id}.monitor"), None)
+ if source is None:
+ raise RuntimeError("The default output has no loopback source.")
+ with source.recorder(samplerate=SAMPLE_RATE, channels=2, blocksize=BLOCK_FRAMES) as mic:
+ pts = max(0, round((time.perf_counter() - self.epoch) * SAMPLE_RATE))
+ self.ready.set()
+ while not self.stopping.is_set():
+ data = mic.record(numframes=BLOCK_FRAMES)
+ samples = np.ascontiguousarray(data.T, dtype=np.float32)
+ try:
+ self.chunks.put_nowait((samples, pts))
+ except queue.Full:
+ raise RuntimeError("The audio encoder cannot keep up.") from None
+ pts += samples.shape[1]
+ except Exception:
+ log.exception("appshot: system audio capture failed")
+ self.error = (
+ "System audio could not be recorded. Check the default output device "
+ "or switch off system audio and try again."
+ )
+ finally:
+ self.ready.set()
+ self.done.set()
diff --git a/jarvis/appshot/recording_audio_edit.py b/jarvis/appshot/recording_audio_edit.py
new file mode 100644
index 0000000000..1263f207eb
--- /dev/null
+++ b/jarvis/appshot/recording_audio_edit.py
@@ -0,0 +1,77 @@
+"""Stream a recording's trimmed audio through a pitch-preserving speed filter."""
+
+from __future__ import annotations
+
+from fractions import Fraction
+from typing import Any
+
+
+def write_audio(
+ source: Any, writer: Any, output: Any, start: float, end: float, speed: float
+) -> None:
+ """Mux the selected audio range with sample-accurate cuts and a fresh timeline."""
+ import av
+ import numpy as np
+
+ rate = 48000
+ graph = av.filter.Graph()
+ entry = graph.add_abuffer(sample_rate=rate, format="fltp", layout="stereo",
+ time_base=Fraction(1, rate))
+ tempo = graph.add("atempo", str(speed))
+ sink = graph.add("abuffersink")
+ graph.link_nodes(entry, tempo, sink)
+ graph.configure()
+ pts = 0
+
+ def drain() -> None:
+ while True:
+ try:
+ frame = graph.pull()
+ except (av.error.BlockingIOError, av.error.EOFError):
+ break # The filter needs input, or its final samples were drained.
+ for packet in output.encode(frame):
+ writer.mux(packet)
+
+ def push_samples(samples: Any) -> None:
+ nonlocal pts
+ sliced = av.AudioFrame.from_ndarray(
+ np.ascontiguousarray(samples), format="fltp", layout="stereo"
+ )
+ sliced.sample_rate = rate
+ sliced.pts, sliced.time_base = pts, Fraction(1, rate)
+ pts += sliced.samples
+ graph.push(sliced)
+ drain()
+
+ with av.open(str(source)) as reader:
+ stream = reader.streams.audio[0]
+ resampler = av.AudioResampler(format="fltp", layout="stereo", rate=rate)
+ reader.seek(max(0, int(start / stream.time_base)), stream=stream, backward=True)
+
+ def submit(frame: Any) -> bool:
+ at = float(frame.pts * frame.time_base) if frame.pts is not None else 0
+ left = max(0, round((start - at) * rate))
+ right = min(frame.samples, round((end - at) * rate))
+ if right > left:
+ desired_pts = max(0, round((at - start) * rate) + left)
+ # Preserve delayed audio startup and any timestamp gaps relative to video.
+ while pts < desired_pts:
+ push_samples(np.zeros((2, min(rate, desired_pts - pts)), dtype=np.float32))
+ left += max(0, pts - desired_pts)
+ if right > left:
+ push_samples(frame.to_ndarray()[:, left:right])
+ return at >= end
+
+ finished = False
+ for frame in reader.decode(stream):
+ for converted in resampler.resample(frame):
+ finished = submit(converted)
+ if finished:
+ break
+ if not finished:
+ for converted in resampler.resample(None):
+ submit(converted)
+ graph.push(None)
+ drain()
+ for packet in output.encode(None):
+ writer.mux(packet)
diff --git a/jarvis/appshot/recording_cards.py b/jarvis/appshot/recording_cards.py
new file mode 100644
index 0000000000..cc6a26e8f9
--- /dev/null
+++ b/jarvis/appshot/recording_cards.py
@@ -0,0 +1,105 @@
+"""Hand finalized recordings to the existing screenshot corner-card sidecar."""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import shutil
+import time
+from pathlib import Path
+from typing import Any
+
+from jarvis.appshot.recording import recording_file
+
+log = logging.getLogger(__name__)
+RECORDING_PREFIX = "recording:"
+
+
+async def show_recording_preview(recording_id: str, metadata: dict[str, Any]) -> bool:
+ from jarvis.appshot.recording_labels import LABELS
+ from jarvis.core.config import load_config
+ from jarvis.cu.indicator.controller import get_indicator_controller
+
+ try:
+ path = recording_file(recording_id)
+ controller = get_indicator_controller()
+ if path is None or controller is None:
+ return False
+ config = await asyncio.to_thread(load_config)
+ if not config.appshot.effect:
+ return False
+ poster = await asyncio.to_thread(path.with_suffix(".jpg").read_bytes)
+ labels = dict(LABELS.get(config.ui.language, LABELS["en"]))
+ return await controller.show_recording(
+ recording_id=recording_id,
+ video_path=str(path),
+ thumb_b64=base64.b64encode(poster).decode("ascii"),
+ monitor=metadata.get("monitor", []),
+ rect=metadata.get("rect", [0, 0, 1, 1]),
+ screen_name=str(metadata.get("screen_name", "")),
+ duration_s=float(metadata.get("duration_s", 0)),
+ rest_ms=max(0, min(600, int(getattr(config.appshot, "card_seconds", 6)))) * 1000,
+ labels=labels,
+ )
+ except Exception:
+ # Presentation failure must never turn an already finalized video into an error.
+ log.exception("appshot: recording preview could not be shown")
+ return False
+
+
+def open_recording(recording_id: str) -> bool:
+ from jarvis.platform.open_path import open_file
+
+ path = recording_file(recording_id)
+ return path is not None and open_file(path)
+
+
+def save_recording(recording_id: str, folder: Path | None = None) -> Path | None:
+ path = recording_file(recording_id)
+ if path is None:
+ return None
+ if folder is None:
+ from jarvis.platform.user_dirs import downloads_dir
+
+ folder = downloads_dir()
+ folder.mkdir(parents=True, exist_ok=True)
+ stem = f"recording-{time.strftime('%Y%m%d-%H%M%S')}-{recording_id[:6]}"
+ counter = 0
+ while True:
+ destination = folder / f"{stem}{f'-{counter}' if counter else ''}.mp4"
+ created = False
+ try:
+ with path.open("rb") as source, destination.open("xb") as target:
+ created = True
+ shutil.copyfileobj(source, target, length=1024 * 1024)
+ return destination
+ except FileExistsError: # Pick a fresh filename; never overwrite an existing export.
+ counter += 1
+ except OSError:
+ if created:
+ destination.unlink(missing_ok=True)
+ raise
+
+
+async def run_recording_card_action(action: str, recording_id: str) -> str:
+ from jarvis.appshot.recording_labels import LABELS
+ from jarvis.core.config import load_config
+
+ config = await asyncio.to_thread(load_config)
+ labels = LABELS.get(config.ui.language, LABELS["en"])
+ try:
+ if action == "open":
+ from jarvis.appshot.editor_window import open_recording_window
+
+ # The app's own video editor first; the system player where the
+ # shell cannot open windows (browser-only or headless host).
+ if await open_recording_window(recording_id):
+ return ""
+ opened = await asyncio.to_thread(open_recording, recording_id)
+ return "" if opened else labels["failed"]
+ saved = await asyncio.to_thread(save_recording, recording_id) if action == "save" else None
+ return labels["saved"] if saved is not None else labels["failed"]
+ except OSError:
+ log.exception("appshot: recording could not be copied to Downloads")
+ return labels["failed"]
diff --git a/jarvis/appshot/recording_codec.py b/jarvis/appshot/recording_codec.py
new file mode 100644
index 0000000000..07ac5a650d
--- /dev/null
+++ b/jarvis/appshot/recording_codec.py
@@ -0,0 +1,77 @@
+"""Open a usable recording encoder, with software fallback before any packets."""
+
+from __future__ import annotations
+
+import logging
+from fractions import Fraction
+from typing import Any
+
+log = logging.getLogger(__name__)
+# Fine wall-clock timestamps avoid dropping valid frames when GUI ticks jitter.
+# 60000 is divisible by 30/60/120 and within MPEG-4's time-base denominator limit.
+VIDEO_TIME_BASE = Fraction(1, 60000)
+
+
+def open_video(
+ av: Any, path: Any, size: tuple[int, int], fps: int, bitrate: int, *, hdr: bool = False
+):
+ """Codec presence alone is insufficient: open it with the actual source size.
+
+ ``hdr`` opens a 10-bit BT.2020 PQ encoder for P010 frames
+ (:mod:`jarvis.appshot.hdr_recording`); otherwise an 8-bit one for sRGB.
+ """
+ if hdr:
+ from jarvis.appshot.hdr_recording import HDR_CANDIDATES, tag_hdr10
+
+ return _open_first(av, path, size, fps, bitrate, HDR_CANDIDATES, tag_hdr10)
+ return _open_first(av, path, size, fps, bitrate, _SDR_CANDIDATES, _tag_srgb)
+
+
+def _tag_srgb(context: Any) -> None:
+ """Screen pixels are sRGB: say so, so players do not guess another space.
+
+ The matrix stays as each encoder sets it (NVENC signals the BT.601
+ conversion it applies to RGB input; swscale converts with BT.601 too).
+ """
+ context.color_primaries = 1 # BT.709 = sRGB primaries
+ context.color_trc = 13 # IEC 61966-2-1 (sRGB)
+ context.color_range = 1 # limited, what every 8-bit path here writes
+ if getattr(context, "colorspace", 2) in (0, 2): # RGB / unspecified
+ context.colorspace = 5 # BT.470BG = the BT.601 matrix swscale uses
+
+
+_SDR_CANDIDATES = (
+ ("h264_nvenc", "rgba", {"preset": "p1", "tune": "ull", "rc": "vbr"}),
+ ("h264_amf", "rgba", {"quality": "speed", "rc": "vbr_peak"}),
+ ("h264_qsv", "nv12", {"preset": "veryfast"}),
+ ("h264_videotoolbox", "yuv420p", {"realtime": "1"}),
+ ("libx264", "yuv420p", {"preset": "ultrafast", "tune": "zerolatency"}),
+ ("mpeg4", "yuv420p", {}),
+)
+
+
+def _open_first(
+ av: Any, path: Any, size: tuple[int, int], fps: int, bitrate: int, candidates: Any, tag: Any
+):
+ for codec, pixel_format, options in candidates:
+ if codec not in av.codecs_available:
+ continue
+ container = av.open(str(path), "w", format="mp4")
+ try:
+ stream = container.add_stream(codec, rate=fps, options=options)
+ stream.width, stream.height = size
+ stream.pix_fmt = pixel_format
+ stream.bit_rate = bitrate
+ stream.codec_context.time_base = VIDEO_TIME_BASE
+ stream.codec_context.thread_count = 2
+ stream.codec_context.max_b_frames = 0
+ tag(stream.codec_context)
+ stream.codec_context.open()
+ return container, stream
+ except Exception:
+ # Missing drivers, exhausted hardware sessions, or unsupported sizes.
+ # Retry only before capture writes a header or any audio/video packet.
+ log.debug("appshot: encoder %s unavailable; trying the next codec", codec,
+ exc_info=True)
+ container.close()
+ raise RuntimeError("No video encoder could be opened for this recording.")
diff --git a/jarvis/appshot/recording_frame.py b/jarvis/appshot/recording_frame.py
new file mode 100644
index 0000000000..afcc6e9845
--- /dev/null
+++ b/jarvis/appshot/recording_frame.py
@@ -0,0 +1,63 @@
+"""Copy capture pixels into owned images before the multimedia buffer is reused."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def owned_capture_image(frame: Any, *, opaque_desktop: bool = False) -> Any:
+ """Read packed RGB frames without the GPU presentation/conversion path.
+
+ Desktop frames already contain RGB pixels. Sending them through ``toImage``
+ can introduce readback/texture-conversion artifacts. Respect the mapped row
+ stride, copy while mapped and always release the frame's mapping. YUV portal
+ formats still use Qt's conversion, but their result is detached as well.
+ """
+ from PySide6.QtGui import QImage, QTransform
+ from PySide6.QtMultimedia import QVideoFrame, QVideoFrameFormat
+
+ if not frame.isValid():
+ return QImage()
+ image_format = QVideoFrameFormat.imageFormatFromPixelFormat(frame.pixelFormat())
+ if image_format == QImage.Format.Format_Invalid:
+ image = frame.toImage().copy()
+ else:
+ if not frame.map(QVideoFrame.MapMode.ReadOnly):
+ return QImage()
+ try:
+ width, height = frame.width(), frame.height()
+ stride = frame.bytesPerLine(0)
+ if width <= 0 or height <= 0 or stride <= 0 or frame.mappedBytes(0) < height * stride:
+ return QImage()
+ image = QImage(frame.bits(0), width, height, stride, image_format).copy()
+ finally:
+ frame.unmap()
+ surface = frame.surfaceFormat()
+ viewport = surface.viewport().intersected(image.rect())
+ if not viewport.isEmpty() and viewport != image.rect():
+ image = image.copy(viewport)
+ if surface.scanLineDirection() == QVideoFrameFormat.Direction.BottomToTop:
+ image = image.mirrored(False, True)
+ rotation = frame.rotation().value
+ if rotation:
+ image = image.transformed(QTransform().rotate(rotation))
+ if frame.mirrored():
+ image = image.mirrored(True, False)
+ if image.isNull():
+ return image
+ # A newly opened desktop source can emit zero-filled transparent buffers.
+ # Keep opaque black screens: darkness alone is never an invalid-frame signal.
+ if image.hasAlphaChannel():
+ w, h = image.width() - 1, image.height() - 1
+ points = ((0, 0), (w, 0), (0, h), (w, h), (w // 2, h // 2))
+ alphas = [image.pixelColor(x, y).alpha() for x, y in points]
+ if all(alpha == 0 for alpha in alphas):
+ return QImage()
+ # Windows desktop duplication is opaque. Before its first completed
+ # GPU copy, a staging surface can contain arbitrary bytes, not just
+ # zeros. Do not encode that uninitialized buffer as a real desktop.
+ # Portal window sources may legitimately have transparency, so this
+ # stronger check is explicitly limited to opaque desktop sources.
+ if opaque_desktop and any(alpha != 255 for alpha in alphas):
+ return QImage()
+ return image
diff --git a/jarvis/appshot/recording_geometry.py b/jarvis/appshot/recording_geometry.py
new file mode 100644
index 0000000000..6c7fc076bc
--- /dev/null
+++ b/jarvis/appshot/recording_geometry.py
@@ -0,0 +1,35 @@
+"""Logical-pixel placement for the persistent recording frame and controls."""
+
+from __future__ import annotations
+
+Rect = tuple[int, int, int, int]
+
+
+def selected_rect(screen: Rect, fractions: tuple[float, ...]) -> Rect:
+ left, top, width, height = screen
+ x, y, w, h = fractions
+ x0, y0 = round(x * width), round(y * height)
+ x1, y1 = round((x + w) * width), round((y + h) * height)
+ return left + x0, top + y0, max(1, x1 - x0), max(1, y1 - y0)
+
+
+def toolbar_rect(selection: Rect, available: Rect, size: tuple[int, int]) -> Rect:
+ """Below the selection, above it if needed, and inside the work area for full screen."""
+ x, y, w, h = selection
+ left, top, aw, ah = available
+ bw, bh = min(size[0], aw), min(size[1], ah)
+ bx = max(left, min(round(x + (w - bw) / 2), left + aw - bw))
+ by = y + h + 10
+ if by + bh > top + ah:
+ by = y - bh - 10
+ if by < top:
+ by = top + ah - bh - 10
+ by = max(top, min(by, top + ah - bh))
+ return bx, by, bw, bh
+
+
+def elapsed_label(seconds: float) -> str:
+ total = max(0, int(seconds))
+ hours, remainder = divmod(total, 3600)
+ minutes, seconds = divmod(remainder, 60)
+ return f"{hours:02d}:{minutes:02d}:{seconds:02d}" if hours else f"{minutes:02d}:{seconds:02d}"
diff --git a/jarvis/appshot/recording_hud.py b/jarvis/appshot/recording_hud.py
new file mode 100644
index 0000000000..b075e22fec
--- /dev/null
+++ b/jarvis/appshot/recording_hud.py
@@ -0,0 +1,353 @@
+"""Persistent, click-through capture boundary and its separate clickable stop bar.
+
+Imported only by the recording sidecar, after QApplication has been created.
+"""
+
+from __future__ import annotations
+
+import sys
+from collections.abc import Callable
+
+from PySide6.QtCore import QEasingCurve, QObject, QRect, QRectF, QSize, Qt, QVariantAnimation
+from PySide6.QtGui import QColor, QFont, QIcon, QPainter, QPen, QPixmap
+from PySide6.QtWidgets import (
+ QFrame,
+ QGraphicsDropShadowEffect,
+ QHBoxLayout,
+ QLabel,
+ QPushButton,
+ QWidget,
+)
+
+from jarvis.appshot.picker.annotate import _dark_hud
+from jarvis.appshot.recording_geometry import elapsed_label, selected_rect, toolbar_rect
+from jarvis.cu.indicator.win32 import (
+ exclude_from_capture,
+ harden_clickable_window,
+ harden_window,
+ strip_window_frame,
+)
+
+_RED = QColor(255, 69, 58)
+#: The capture frame: ONE white line with rounded corners and nothing else —
+#: no shadow ring, which read as a second, grey frame. Where the frame window
+#: is excluded from capture, the line lies right ON the selection's edge and
+#: hides the window border beneath it (radius 8 = a Windows 11 window corner),
+#: so the two never show as a double frame. Elsewhere it stays fully outside
+#: the captured pixels, with a radius small enough to clear the selection corners.
+_LINE_WIDTH = 2.5
+_ON_EDGE = (0.5, 8.0) # line centre offset, corner radius
+_OUTSIDE = (2.25, 4.5)
+_FRAME_PAD = 12
+
+
+def _tool_window(widget: QWidget, *, click_through: bool) -> None:
+ flags = (
+ Qt.WindowType.FramelessWindowHint
+ | Qt.WindowType.WindowStaysOnTopHint
+ | Qt.WindowType.WindowDoesNotAcceptFocus
+ | Qt.WindowType.Tool
+ | Qt.WindowType.NoDropShadowWindowHint
+ )
+ if click_through:
+ flags |= Qt.WindowType.WindowTransparentForInput
+ widget.setAttribute(Qt.WidgetAttribute.WA_TransparentForMouseEvents)
+ widget.setWindowFlags(flags)
+ widget.setAttribute(Qt.WidgetAttribute.WA_ShowWithoutActivating)
+ if sys.platform == "darwin":
+ widget.setAttribute(Qt.WidgetAttribute.WA_MacAlwaysShowToolWindow)
+
+
+def hud_style() -> str:
+ """One look for every recording control: a rounded pill that follows the OS theme."""
+ dark = _dark_hud()
+ bg = "rgba(28, 28, 31, 242)" if dark else "rgba(250, 250, 252, 245)"
+ edge = "rgba(255, 255, 255, 34)" if dark else "rgba(0, 0, 0, 30)"
+ fg = "rgb(236, 236, 240)" if dark else "rgb(28, 28, 32)"
+ muted = "rgba(236, 236, 240, 170)" if dark else "rgba(28, 28, 32, 160)"
+ sep = "rgba(255, 255, 255, 40)" if dark else "rgba(0, 0, 0, 28)"
+ ghost = "rgba(255, 255, 255, 20)" if dark else "rgba(0, 0, 0, 12)"
+ ghost_hover = "rgba(255, 255, 255, 34)" if dark else "rgba(0, 0, 0, 22)"
+ return f"""
+ QFrame#pill {{ background: {bg}; border: 1px solid {edge}; border-radius: 14px; }}
+ QFrame#sep {{ background: {sep}; border: none; }}
+ QLabel {{ color: {fg}; background: transparent; }}
+ QLabel#hint {{ color: {muted}; }}
+ QPushButton {{
+ color: {fg}; background: {ghost}; border: none; border-radius: 9px;
+ padding: 6px 12px 6px 10px; font-weight: 600;
+ }}
+ QPushButton:hover {{ background: {ghost_hover}; }}
+ QPushButton#stop {{ color: white; background: #E5342B; }}
+ QPushButton#stop:hover {{ background: #F2463D; }}
+ QPushButton#stop:pressed {{ background: #C82A22; }}
+ """
+
+
+def _shadow(widget: QWidget) -> None:
+ shadow = QGraphicsDropShadowEffect(widget)
+ shadow.setBlurRadius(26)
+ shadow.setOffset(0, 5)
+ shadow.setColor(QColor(0, 0, 0, 120))
+ widget.setGraphicsEffect(shadow)
+
+
+def _separator(parent: QWidget) -> QFrame:
+ line = QFrame(parent)
+ line.setObjectName("sep")
+ line.setFixedSize(1, 20)
+ return line
+
+
+def _glyph(kind: str, colour: QColor, dpr: float) -> QIcon:
+ """Small hand-drawn icons, so the controls never depend on an icon theme."""
+ size = 14
+ pixmap = QPixmap(round(size * dpr), round(size * dpr))
+ pixmap.setDevicePixelRatio(dpr)
+ pixmap.fill(Qt.GlobalColor.transparent)
+ painter = QPainter(pixmap)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ if kind == "stop":
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(colour)
+ painter.drawRoundedRect(QRectF(2.5, 2.5, 9, 9), 2, 2)
+ else: # "screen": a monitor outline with its stand
+ pen = QPen(colour, 1.4)
+ pen.setCapStyle(Qt.PenCapStyle.RoundCap)
+ painter.setPen(pen)
+ painter.drawRoundedRect(QRectF(1.5, 2, 11, 7.5), 1.6, 1.6)
+ painter.drawLine(7, 9.5, 7, 12)
+ painter.drawLine(4.5, 12.2, 9.5, 12.2)
+ painter.end()
+ return QIcon(pixmap)
+
+
+def _font(widget: QWidget, px: int, weight: QFont.Weight, *, tabular: bool = False) -> QFont:
+ font = QFont(widget.font())
+ font.setPixelSize(px)
+ font.setWeight(weight)
+ if tabular:
+ try: # Qt >= 6.7: equal-width digits, so the clock does not wobble.
+ font.setFeature(QFont.Tag("tnum"), 1)
+ except (AttributeError, TypeError):
+ pass # Older Qt: the fixed clock width below still keeps the bar steady.
+ return font
+
+
+class RecordingDot(QWidget):
+ """A softly pulsing red dot — the universal "recording now" signal."""
+
+ def __init__(self, parent: QWidget) -> None:
+ super().__init__(parent)
+ self.setFixedSize(14, 14)
+ self._level = 1.0
+ self._pulse = QVariantAnimation(self)
+ self._pulse.setStartValue(1.0)
+ self._pulse.setKeyValueAt(0.5, 0.35)
+ self._pulse.setEndValue(1.0)
+ self._pulse.setDuration(1400)
+ self._pulse.setLoopCount(-1)
+ self._pulse.setEasingCurve(QEasingCurve.Type.InOutSine)
+ self._pulse.valueChanged.connect(self._set_level)
+ self._pulse.start()
+
+ def _set_level(self, value: float) -> None:
+ self._level = float(value)
+ self.update()
+
+ def paintEvent(self, _event) -> None: # noqa: N802
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ painter.setPen(Qt.PenStyle.NoPen)
+ halo = QColor(_RED)
+ halo.setAlpha(round(70 * (1.0 - self._level) + 20))
+ painter.setBrush(halo)
+ painter.drawEllipse(QRectF(0, 0, 14, 14))
+ core = QColor(_RED)
+ core.setAlpha(round(255 * (0.55 + 0.45 * self._level)))
+ painter.setBrush(core)
+ painter.drawEllipse(QRectF(3, 3, 8, 8))
+
+
+class RecordingFrame(QWidget):
+ def __init__(self, screen, selection: QRect, on_stop: Callable[[], None]) -> None:
+ super().__init__()
+ self.on_stop = on_stop
+ self.setObjectName("appshot-recording-frame")
+ _tool_window(self, click_through=True)
+ self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground)
+ self.setScreen(screen)
+ self.selection = QRect(selection)
+ pad = _FRAME_PAD
+ self.setGeometry(selection.adjusted(-pad, -pad, pad, pad).intersected(screen.geometry()))
+ self.capture_excluded = False
+
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ harden_window(int(self.winId()))
+ # Otherwise Windows 11 outlines the padded window: a grey second frame.
+ strip_window_frame(int(self.winId()))
+ self.capture_excluded = exclude_from_capture(int(self.winId()))
+ self.update() # The line's placement depends on the exclusion.
+
+ def paintEvent(self, _event) -> None: # noqa: N802
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ # The frame sits outside the captured pixels wherever the monitor has
+ # room. Full-screen edges stay visible just inside the monitor bounds.
+ pad = _FRAME_PAD
+ sel = QRectF(self.selection.translated(-self.x(), -self.y()))
+ outer = sel.adjusted(-pad, -pad, pad, pad).intersected(QRectF(self.rect()))
+ base = outer.adjusted(pad, pad, -pad, -pad)
+ offset, corner = _ON_EDGE if self.capture_excluded else _OUTSIDE
+ painter.setPen(QPen(QColor(255, 255, 255, 250), _LINE_WIDTH))
+ painter.drawRoundedRect(base.adjusted(-offset, -offset, offset, offset), corner, corner)
+
+ def closeEvent(self, event) -> None: # noqa: N802
+ self.on_stop()
+ event.accept()
+
+
+class RecordingToolbar(QWidget):
+ """Pulsing dot, running time and a red stop button on one floating pill."""
+
+ def __init__(
+ self, labels: dict[str, str], on_stop: Callable[[], None], dpr: float = 1.0
+ ) -> None:
+ super().__init__()
+ self.on_stop = on_stop
+ self.setObjectName("appshot-recording-toolbar")
+ _tool_window(self, click_through=False)
+ self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground)
+ # No edge line and no drop shadow: both drew a grey ring around the bar.
+ self.setStyleSheet(hud_style() + "QFrame#pill { border: none; }")
+ outer = QHBoxLayout(self)
+ outer.setContentsMargins(0, 0, 0, 0)
+ pill = self._pill = QFrame(self)
+ pill.setObjectName("pill")
+ outer.addWidget(pill)
+ layout = QHBoxLayout(pill)
+ layout.setContentsMargins(14, 7, 7, 7)
+ layout.setSpacing(10)
+ self.dot = RecordingDot(pill)
+ self.dot.setToolTip(labels["recording"])
+ layout.addWidget(self.dot)
+ self.clock = QLabel("00:00", pill)
+ self.clock.setObjectName("appshot-recording-time")
+ self.clock.setFont(_font(self.clock, 15, QFont.Weight.DemiBold, tabular=True))
+ self.clock.setAlignment(Qt.AlignmentFlag.AlignLeft | Qt.AlignmentFlag.AlignVCenter)
+ self._fit_clock("00:00")
+ layout.addWidget(self.clock)
+ layout.addWidget(_separator(pill))
+ self.stop_button = QPushButton(f" {labels['stop']}", pill)
+ self.stop_button.setObjectName("stop")
+ self.stop_button.setFont(_font(self.stop_button, 13, QFont.Weight.DemiBold))
+ self.stop_button.setIcon(_glyph("stop", QColor(255, 255, 255), dpr))
+ self.stop_button.setIconSize(QSize(14, 14))
+ self.stop_button.setCursor(Qt.CursorShape.PointingHandCursor)
+ self.stop_button.setFocusPolicy(Qt.FocusPolicy.NoFocus)
+ self.stop_button.clicked.connect(on_stop)
+ layout.addWidget(self.stop_button)
+ self.adjustSize()
+ self.capture_excluded = False
+
+ def _fit_clock(self, text: str) -> None:
+ # Size for the widest digits, so ticking seconds never shift the button.
+ metrics = self.clock.fontMetrics()
+ widest = max("0123456789", key=metrics.horizontalAdvance)
+ sample = "".join(widest if ch.isdigit() else ch for ch in text)
+ self.clock.setFixedWidth(metrics.horizontalAdvance(sample) + 2)
+
+ def set_clock(self, text: str) -> bool:
+ """Show ``text``; True when the bar changed size (the hour digits appeared)."""
+ grew = len(text) != len(self.clock.text())
+ self.clock.setText(text)
+ if grew:
+ self._fit_clock(text)
+ self._pill.layout().activate()
+ self.layout().activate()
+ self.adjustSize()
+ return grew
+
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ # No Windows 11 frame around the bar, and a click never steals focus.
+ harden_clickable_window(int(self.winId()))
+ self.capture_excluded = exclude_from_capture(int(self.winId()))
+
+ def closeEvent(self, event) -> None: # noqa: N802
+ self.on_stop()
+ event.accept()
+
+
+class RecordingHint(QFrame):
+ """The selection-time pill: what to do, plus a one-click full-screen recording."""
+
+ def __init__(
+ self, parent: QWidget, labels: dict[str, str], on_full: Callable[[], None], dpr: float
+ ) -> None:
+ super().__init__(parent)
+ self.setObjectName("pill")
+ self.setStyleSheet(hud_style())
+ self.setCursor(Qt.CursorShape.ArrowCursor)
+ _shadow(self)
+ layout = QHBoxLayout(self)
+ layout.setContentsMargins(14, 7, 7, 7)
+ layout.setSpacing(10)
+ dot = QLabel(self)
+ dot.setFixedSize(10, 10)
+ dot.setStyleSheet(f"background: {_RED.name()}; border-radius: 5px;")
+ layout.addWidget(dot)
+ hint = QLabel(labels["select"], self)
+ hint.setObjectName("hint")
+ hint.setFont(_font(hint, 13, QFont.Weight.Medium))
+ layout.addWidget(hint)
+ layout.addWidget(_separator(self))
+ dark = _dark_hud()
+ self.full_button = QPushButton(f" {labels['full']}", self)
+ self.full_button.setFont(_font(self.full_button, 13, QFont.Weight.DemiBold))
+ self.full_button.setIcon(
+ _glyph("screen", QColor(236, 236, 240) if dark else QColor(28, 28, 32), dpr)
+ )
+ self.full_button.setIconSize(QSize(14, 14))
+ self.full_button.setCursor(Qt.CursorShape.PointingHandCursor)
+ self.full_button.setFocusPolicy(Qt.FocusPolicy.NoFocus)
+ self.full_button.clicked.connect(lambda _checked=False: on_full())
+ layout.addWidget(self.full_button)
+ self.adjustSize()
+
+
+class RecordingHud(QObject):
+ def __init__(self, screen, fractions, labels, on_stop) -> None:
+ super().__init__()
+ geometry, self._available = screen.geometry(), screen.availableGeometry()
+ self._rect = selected_rect(geometry.getRect(), tuple(fractions))
+ self.frame = RecordingFrame(screen, QRect(*self._rect), on_stop)
+ self.toolbar = RecordingToolbar(labels, on_stop, screen.devicePixelRatio())
+ self.toolbar.setScreen(screen)
+ self._place_toolbar()
+
+ def _place_toolbar(self) -> None:
+ hint = self.toolbar.sizeHint()
+ x, y, w, h = toolbar_rect(
+ self._rect, self._available.getRect(), (hint.width(), hint.height())
+ )
+ self.toolbar.setGeometry(QRect(x, y, w, h))
+
+ def show(self) -> None:
+ self.frame.show()
+ self.toolbar.show()
+
+ def update_elapsed(self, seconds: float) -> None:
+ text = elapsed_label(seconds)
+ if self.toolbar.clock.text() != text and self.toolbar.set_clock(text):
+ self._place_toolbar()
+
+ def hide(self) -> None:
+ self.frame.hide()
+ self.toolbar.hide()
+
+ def close(self) -> None:
+ self.frame.close()
+ self.toolbar.close()
diff --git a/jarvis/appshot/recording_labels.py b/jarvis/appshot/recording_labels.py
index 167ce3f949..1078152379 100644
--- a/jarvis/appshot/recording_labels.py
+++ b/jarvis/appshot/recording_labels.py
@@ -3,6 +3,13 @@
LABELS = {
"en": {
"title": "AppShots — Screen recording",
+ "play": "Play video",
+ "video": "Video",
+ "save": "Save video",
+ "close": "Close",
+ "pin": "Keep on screen",
+ "saved": "Saved to Downloads",
+ "failed": "That did not work",
"select": "Drag an area to record · Esc cancels",
"full": "Record entire screen",
"portal": "Choose a screen in the system sharing dialog.",
@@ -15,6 +22,13 @@ LABELS = {
},
"de": {
"title": "AppShots — Bildschirmaufnahme", # i18n-allow: product UI
+ "play": "Video abspielen", # i18n-allow: product UI
+ "video": "Video",
+ "save": "Video speichern", # i18n-allow: product UI
+ "close": "Schließen", # i18n-allow: product UI
+ "pin": "Auf dem Bildschirm behalten", # i18n-allow: product UI
+ "saved": "In Downloads gespeichert", # i18n-allow: product UI
+ "failed": "Das hat nicht geklappt", # i18n-allow: product UI
"select": "Bereich zum Aufnehmen ziehen · Esc bricht ab", # i18n-allow: product UI
"full": "Gesamten Bildschirm aufnehmen", # i18n-allow: product UI
"portal": "Bildschirm im Freigabedialog wählen.", # i18n-allow: product UI
@@ -27,6 +41,13 @@ LABELS = {
},
"es": {
"title": "AppShots — Grabación de pantalla",
+ "play": "Reproducir vídeo",
+ "video": "Vídeo",
+ "save": "Guardar vídeo",
+ "close": "Cerrar",
+ "pin": "Mantener en pantalla",
+ "saved": "Guardado en Descargas",
+ "failed": "No se pudo completar",
"select": "Arrastra un área para grabar · Esc cancela",
"full": "Grabar pantalla completa",
"portal": "Elige una pantalla en el diálogo de compartir del sistema.",
@@ -39,6 +60,13 @@ LABELS = {
},
"zh": {
"title": "AppShots — 屏幕录制", # i18n-allow: product UI
+ "play": "播放视频", # i18n-allow: product UI
+ "video": "视频", # i18n-allow: product UI
+ "save": "保存视频", # i18n-allow: product UI
+ "close": "关闭", # i18n-allow: product UI
+ "pin": "固定在屏幕上", # i18n-allow: product UI
+ "saved": "已保存到“下载”", # i18n-allow: product UI
+ "failed": "操作未成功", # i18n-allow: product UI
"select": "拖动选择录制区域 · 按 Esc 取消", # i18n-allow: product UI
"full": "录制整个屏幕", # i18n-allow: product UI
"portal": "请在系统共享对话框中选择一个屏幕。", # i18n-allow: product UI
diff --git a/jarvis/appshot/recording_options.py b/jarvis/appshot/recording_options.py
new file mode 100644
index 0000000000..140598188e
--- /dev/null
+++ b/jarvis/appshot/recording_options.py
@@ -0,0 +1,53 @@
+"""Validated recording preferences and per-source limits, without desktop imports."""
+
+from __future__ import annotations
+
+import math
+from typing import Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+Resolution = Literal["720p", "1080p", "1440p", "2160p", "native"]
+FrameRate = Literal[30, 60, 120]
+
+
+class RecordingOptions(BaseModel):
+ """One immutable snapshot for a recording, including shortcut starts."""
+
+ model_config = ConfigDict(extra="forbid", frozen=True)
+ resolution: Resolution = "1080p"
+ fps: FrameRate = 60
+ bitrate_mbps: int = Field(default=12, ge=1, le=100, strict=True)
+ system_audio: bool = False
+
+ @classmethod
+ def from_config(cls, block: object) -> RecordingOptions:
+ return cls(**{
+ key: getattr(block, f"recording_{key}", field.default)
+ for key, field in cls.model_fields.items()
+ })
+
+
+def output_size(width: int, height: int, resolution: Resolution) -> tuple[int, int]:
+ """Fit inside an oriented preset without upscaling or distorting the source."""
+ if width < 1 or height < 1:
+ raise ValueError("The recording area must have positive dimensions.")
+ factor = 1.0
+ if resolution != "native":
+ short = int(resolution[:-1])
+ long = round(short * 16 / 9)
+ bound_w, bound_h = (long, short) if width >= height else (short, long)
+ factor = min(1.0, bound_w / width, bound_h / height)
+ # H.264 4:2:0 needs even dimensions. At most one padding pixel at native size.
+ if factor == 1:
+ return width + width % 2, height + height % 2
+ return max(2, int(width * factor) // 2 * 2), max(2, int(height * factor) // 2 * 2)
+
+
+def effective_fps(requested: int, refresh_hz: float | None) -> int:
+ """Use the selected display's current rate, never another monitor's maximum."""
+ hz = refresh_hz if refresh_hz and math.isfinite(refresh_hz) else 60.0
+ # 59.94/119.88 are the conventional 60/120 Hz modes.
+ ceiling = max(1, round(hz))
+ available = [rate for rate in (30, 60, 120) if rate <= ceiling]
+ return min(requested, max(available) if available else ceiling)
diff --git a/jarvis/appshot/recording_preview.py b/jarvis/appshot/recording_preview.py
new file mode 100644
index 0000000000..49da0c5c57
--- /dev/null
+++ b/jarvis/appshot/recording_preview.py
@@ -0,0 +1,525 @@
+"""Video corner cards in the shared screenshot effect process (Qt-only)."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+from PySide6.QtCore import QByteArray, QMimeData, QObject, QPoint, QPointF, QRectF, Qt, QTimer, QUrl
+from PySide6.QtGui import (
+ QColor,
+ QDrag,
+ QFont,
+ QFontMetricsF,
+ QGuiApplication,
+ QImage,
+ QPainter,
+ QPainterPath,
+ QPen,
+)
+from PySide6.QtWidgets import QWidget
+
+from jarvis.appshot.recording_geometry import elapsed_label
+from jarvis.cu.indicator import protocol
+from jarvis.cu.indicator import win32 as native_window
+from jarvis.cu.indicator.renderer import (
+ _CARD_INSET,
+ _CARD_ROUND,
+ _CARD_STATUS_MS,
+ _SNAP_RADIUS,
+ _card_font,
+ _paint_card,
+ _paint_icon,
+ _poly,
+)
+
+#: The big play disc in the middle of a hovered card.
+_PLAY_D = 46.0
+#: Transparent margin around the picture that holds the soft shadow.
+_PAD = 12
+
+
+def _frosted(image: QImage) -> QImage:
+ """The thumbnail blurred like frosted glass: shrink in steps, grow back smoothly."""
+ size = image.size()
+ for div in (4, 16, 4, 1):
+ image = image.scaled(
+ max(1, size.width() // div),
+ max(1, size.height() // div),
+ Qt.AspectRatioMode.IgnoreAspectRatio,
+ Qt.TransformationMode.SmoothTransformation,
+ )
+ return image
+
+
+def _paint_video_icon(painter: QPainter, name: str, area: QRectF, color: QColor) -> None:
+ """The play triangle and the save arrow, on the corner chips' 28-unit grid."""
+ painter.save()
+ painter.translate(area.center())
+ scale = area.width() / 28.0
+ painter.scale(scale, scale)
+ if name == "play":
+ # Rounded triangle, nudged right so it looks centred in the disc.
+ pen = QPen(color, 2.4)
+ pen.setJoinStyle(Qt.PenJoinStyle.RoundJoin)
+ painter.setPen(pen)
+ painter.setBrush(color)
+ painter.drawPolygon(_poly((-3.6, -6.4), (6.8, 0.0), (-3.6, 6.4)))
+ else: # "save": an arrow into a tray
+ pen = QPen(color, 2.1)
+ pen.setCapStyle(Qt.PenCapStyle.RoundCap)
+ pen.setJoinStyle(Qt.PenJoinStyle.RoundJoin)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawLine(QPointF(0.0, -7.0), QPointF(0.0, 3.0))
+ painter.drawPolyline(_poly((-4.2, -1.2), (0.0, 3.0), (4.2, -1.2)))
+ painter.drawPolyline(_poly((-6.8, 3.4), (-6.8, 7.0), (6.8, 7.0), (6.8, 3.4)))
+ painter.restore()
+
+
+def _caption_font() -> QFont:
+ font = QFont()
+ font.setPointSizeF(8.5)
+ font.setWeight(QFont.Weight.Medium)
+ return font
+
+
+class RecordingCard(QWidget):
+ """The same framed thumbnail as a screenshot card, for a finished video.
+
+ At rest it shows a small duration badge. Hover frosts the picture and shows
+ a big play disc in the middle and white chips in the corners: Pin
+ (top-left), Close (top-right) and Save (bottom-right). A click anywhere
+ else plays, a drag shares the MP4, a right-click dismisses the card.
+ """
+
+ def __init__(self, owner, screen, rect, thumb, payload) -> None:
+ super().__init__()
+ self.owner = owner
+ self.screen_name = screen.name()
+ self.recording_id = str(payload["id"])
+ self.path = Path(payload["video_path"])
+ self.thumb = thumb
+ self.closed = False
+ self.pinned = False
+ self.press = None
+ self.press_button = ""
+ self.hover = False
+ self.hot = ""
+ self.frost: QImage | None = None
+ self.status = ""
+ self.rest_ms = max(0, int(payload.get("rest_ms", 6000)))
+ self.labels = {
+ "play": "Play video",
+ "save": "Save video",
+ "close": "Close",
+ "pin": "Keep on screen",
+ "video": "Video",
+ **payload.get("labels", {}),
+ }
+ self.duration = elapsed_label(float(payload.get("duration_s", 0)))
+ self.setObjectName("appshot-recording-card")
+ self.setWindowFlags(
+ Qt.WindowType.FramelessWindowHint
+ | Qt.WindowType.WindowStaysOnTopHint
+ | Qt.WindowType.Tool
+ | Qt.WindowType.WindowDoesNotAcceptFocus
+ | Qt.WindowType.NoDropShadowWindowHint
+ )
+ self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground)
+ self.setAttribute(Qt.WidgetAttribute.WA_ShowWithoutActivating)
+ if sys.platform == "darwin":
+ self.setAttribute(Qt.WidgetAttribute.WA_MacAlwaysShowToolWindow)
+ self.setScreen(screen)
+ self.setGeometry(rect.adjusted(-_PAD, -_PAD, _PAD, _PAD).toAlignedRect())
+ self.setCursor(Qt.CursorShape.PointingHandCursor)
+ self.setMouseTracking(True)
+ self.setAccessibleName(f"{self.labels['video']} · {self.duration}")
+ self.dismiss = QTimer(self)
+ self.dismiss.setSingleShot(True)
+ self.dismiss.timeout.connect(self.close)
+ self.status_timer = QTimer(self)
+ self.status_timer.setSingleShot(True)
+ self.status_timer.timeout.connect(self.clear_status)
+ if self.rest_ms:
+ self.dismiss.start(self.rest_ms)
+
+ # -- actions -------------------------------------------------------------
+ def play(self) -> None:
+ self.owner.emit(protocol.EVENT_RECORDING_OPEN, id=self.recording_id)
+
+ def save(self) -> None:
+ self.owner.emit(protocol.EVENT_RECORDING_SAVE, id=self.recording_id)
+
+ def toggle_pin(self) -> None:
+ self.pinned = not self.pinned
+ self.dismiss.stop() # The pointer is on the card; leaving re-arms it.
+ self.update()
+
+ def show_status(self, text: str) -> None:
+ """A short line over the card ("Saved to Downloads")."""
+ self.status = text
+ self.status_timer.start(_CARD_STATUS_MS)
+ self.update()
+
+ def clear_status(self) -> None:
+ self.status = ""
+ self.update()
+
+ # -- geometry ------------------------------------------------------------
+ def card_area(self) -> QRectF:
+ return QRectF(_PAD, _PAD, self.width() - 2 * _PAD, self.height() - 2 * _PAD)
+
+ def buttons(self) -> dict[str, QRectF]:
+ """Hit areas of the hover buttons, in widget coordinates."""
+ rect = self.card_area()
+ d, inset = _CARD_ROUND, _CARD_INSET
+ left, right = rect.left() + inset, rect.right() - inset - d
+ top, bottom = rect.top() + inset, rect.bottom() - inset - d
+ play = min(_PLAY_D, max(d, rect.height() - 2 * (d + inset) - 8))
+ centre = rect.center()
+ return {
+ "pin": QRectF(left, top, d, d),
+ "close": QRectF(right, top, d, d),
+ "save": QRectF(right, bottom, d, d),
+ "play": QRectF(centre.x() - play / 2, centre.y() - play / 2, play, play),
+ }
+
+ def button_at(self, pos: QPointF) -> str:
+ if not self.hover:
+ return ""
+ for name, area in self.buttons().items():
+ if area.adjusted(-2, -2, 2, 2).contains(pos):
+ return name
+ return ""
+
+ # -- input ---------------------------------------------------------------
+ def enterEvent(self, event) -> None: # noqa: N802
+ self.hover = True
+ self.dismiss.stop()
+ self.update()
+ super().enterEvent(event)
+
+ def leaveEvent(self, event) -> None: # noqa: N802
+ self.hover = False
+ self.hot = ""
+ if not self.pinned and self.rest_ms:
+ self.dismiss.start(2500)
+ self.update()
+ super().leaveEvent(event)
+
+ def showEvent(self, event) -> None: # noqa: N802
+ super().showEvent(event)
+ harden = getattr(native_window, "harden_clickable_window", None)
+ if harden is not None:
+ harden(int(self.winId()))
+ native_window.exclude_from_capture(int(self.winId()))
+
+ def mousePressEvent(self, event) -> None: # noqa: N802
+ if event.button() == Qt.MouseButton.RightButton:
+ self.close()
+ elif event.button() == Qt.MouseButton.LeftButton:
+ self.press = event.position().toPoint()
+ self.press_button = self.button_at(event.position())
+
+ def mouseMoveEvent(self, event) -> None: # noqa: N802
+ hot = self.button_at(event.position())
+ if hot != self.hot:
+ self.hot = hot
+ self.update()
+ if self.press is None or self.press_button:
+ return
+ if (event.position().toPoint() - self.press).manhattanLength() < 6:
+ return
+ self.press = None
+ self.dismiss.stop()
+ mime = QMimeData()
+ mime.setUrls([QUrl.fromLocalFile(str(self.path))])
+ drag = QDrag(self)
+ drag.setMimeData(mime)
+ drag.exec(Qt.DropAction.CopyAction)
+ if not self.pinned and self.rest_ms:
+ self.dismiss.start(2500)
+
+ def mouseReleaseEvent(self, event) -> None: # noqa: N802
+ if event.button() != Qt.MouseButton.LeftButton or self.press is None:
+ return
+ self.press = None
+ pressed, self.press_button = self.press_button, ""
+ if pressed and pressed != self.button_at(event.position()):
+ return # Pressed a button, let go elsewhere: nothing.
+ if pressed == "close":
+ self.close()
+ elif pressed == "pin":
+ self.toggle_pin()
+ elif pressed == "save":
+ self.save()
+ else: # "play" or the picture itself
+ self.play()
+
+ def closeEvent(self, event) -> None: # noqa: N802
+ self.dismiss.stop()
+ if not self.closed:
+ self.closed = True
+ self.owner.remove(self)
+ event.accept()
+ self.deleteLater()
+
+ # -- painting ------------------------------------------------------------
+ def paintEvent(self, _event) -> None: # noqa: N802
+ painter = QPainter(self)
+ painter.setRenderHint(QPainter.RenderHint.Antialiasing)
+ painter.setRenderHint(QPainter.RenderHint.SmoothPixmapTransform)
+ rect = self.card_area()
+ painter.setPen(Qt.PenStyle.NoPen)
+ for spread, alpha in ((8.0, 18), (4.0, 30), (1.5, 46)):
+ painter.setBrush(QColor(0, 0, 0, alpha))
+ shadow = rect.adjusted(-spread, -spread + 3, spread, spread + 3)
+ painter.drawRoundedRect(shadow, _SNAP_RADIUS + spread, _SNAP_RADIUS + spread)
+ _paint_card(painter, rect, self.thumb, _SNAP_RADIUS, 1.0)
+ if self.status:
+ self.paint_veil(painter, rect, None, QColor(0, 0, 0, 150))
+ font = _card_font()
+ font.setPointSizeF(10.0)
+ painter.setFont(font)
+ painter.setPen(QColor(255, 255, 255, 245))
+ painter.drawText(
+ rect, Qt.AlignmentFlag.AlignCenter | Qt.TextFlag.TextWordWrap, self.status
+ )
+ elif self.hover:
+ if self.frost is None and not self.thumb.isNull():
+ self.frost = _frosted(self.thumb)
+ self.paint_veil(painter, rect, self.frost, QColor(18, 16, 26, 92))
+ buttons = self.buttons()
+ for name in ("pin", "close", "save"):
+ self.paint_chip(painter, name, buttons[name])
+ self.paint_play(painter, buttons["play"])
+ self.paint_caption(painter, rect)
+ else:
+ self.paint_badge(painter, rect)
+ if self.pinned:
+ # Pinned and at rest: the pin stays, so the card says why it stays.
+ self.paint_chip(painter, "pin", self.buttons()["pin"])
+ painter.end()
+
+ def paint_veil(
+ self, painter: QPainter, rect: QRectF, image: QImage | None, colour: QColor
+ ) -> None:
+ clip = QPainterPath()
+ clip.addRoundedRect(rect, _SNAP_RADIUS, _SNAP_RADIUS)
+ painter.save()
+ painter.setClipPath(clip)
+ if image is not None:
+ painter.drawImage(rect, image)
+ painter.fillRect(rect, colour)
+ if image is not None:
+ painter.fillRect(rect, QColor(255, 255, 255, 18)) # frosted, not black
+ painter.restore()
+ if image is not None:
+ pen = QPen(QColor(255, 255, 255, 150))
+ pen.setWidthF(1.2)
+ painter.setPen(pen)
+ painter.setBrush(Qt.BrushStyle.NoBrush)
+ painter.drawRoundedRect(rect.adjusted(0.6, 0.6, -0.6, -0.6), _SNAP_RADIUS, _SNAP_RADIUS)
+
+ def paint_chip(self, painter: QPainter, name: str, area: QRectF) -> None:
+ """A round white corner chip with a dark icon; a set pin is inverted."""
+ hot = name == self.hot and self.hover
+ inverted = name == "pin" and self.pinned
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(0, 0, 0, 40))
+ painter.drawEllipse(area.translated(0, 1.2))
+ if inverted:
+ ring = QPen(QColor(255, 255, 255, 120))
+ ring.setWidthF(1.0)
+ painter.setPen(ring)
+ painter.setBrush(QColor(22, 22, 26, 250 if hot else 232))
+ icon = QColor(255, 255, 255)
+ else:
+ painter.setBrush(QColor(255, 255, 255, 252) if hot else QColor(240, 240, 242, 240))
+ icon = QColor(18, 18, 20)
+ painter.drawEllipse(area)
+ if name == "save":
+ _paint_video_icon(painter, "save", area, icon)
+ else:
+ _paint_icon(painter, name, area, icon)
+
+ def paint_play(self, painter: QPainter, area: QRectF) -> None:
+ hot = self.hot == "play"
+ if hot:
+ area = area.adjusted(-2, -2, 2, 2) # a small lift under the pointer
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(0, 0, 0, 46))
+ painter.drawEllipse(area.translated(0, 1.8))
+ painter.setBrush(QColor(255, 255, 255, 252) if hot else QColor(242, 242, 244, 240))
+ painter.drawEllipse(area)
+ _paint_video_icon(painter, "play", area, QColor(18, 18, 20))
+
+ def paint_caption(self, painter: QPainter, rect: QRectF) -> None:
+ """What the button under the pointer does, else "Video · 00:12"."""
+ if self.hot == "pin" and self.pinned:
+ text = self.labels.get("unpin", self.labels["pin"])
+ elif self.hot:
+ text = self.labels.get(self.hot, "")
+ else:
+ text = f"{self.labels['video']} · {self.duration}"
+ font = _caption_font()
+ metrics = QFontMetricsF(font)
+ h = metrics.height() + 6.0
+ side = _CARD_INSET + _CARD_ROUND + 6.0
+ band = QRectF(
+ rect.left() + side,
+ rect.bottom() - _CARD_INSET - _CARD_ROUND / 2 - h / 2,
+ rect.width() - 2 * side,
+ h,
+ )
+ if not text or band.width() <= 0 or self.buttons()["play"].bottom() + 2 > band.top():
+ return
+ painter.setFont(font)
+ painter.setPen(QColor(255, 255, 255, 225))
+ painter.drawText(
+ band,
+ Qt.AlignmentFlag.AlignCenter,
+ metrics.elidedText(text, Qt.TextElideMode.ElideRight, band.width()),
+ )
+
+ def paint_badge(self, painter: QPainter, rect: QRectF) -> None:
+ """A small dark pill, bottom-left: a play mark and the length."""
+ font = _caption_font()
+ font.setWeight(QFont.Weight.DemiBold)
+ metrics = QFontMetricsF(font)
+ h = 22.0
+ icon = 9.0
+ w = 8.0 + icon + 5.0 + metrics.horizontalAdvance(self.duration) + 9.0
+ badge = QRectF(rect.left() + _CARD_INSET, rect.bottom() - _CARD_INSET - h, w, h)
+ if badge.right() > rect.right() - _CARD_INSET:
+ return
+ painter.setPen(Qt.PenStyle.NoPen)
+ painter.setBrush(QColor(12, 12, 14, 190))
+ painter.drawRoundedRect(badge, h / 2, h / 2)
+ mark = QRectF(badge.left() + 8.0, badge.center().y() - icon / 2, icon, icon)
+ painter.setBrush(QColor(255, 255, 255, 240))
+ painter.drawPolygon(
+ _poly(
+ (mark.left() + 1.0, mark.top()),
+ (mark.right(), mark.center().y()),
+ (mark.left() + 1.0, mark.bottom()),
+ )
+ )
+ painter.setFont(font)
+ painter.setPen(QColor(255, 255, 255, 240))
+ painter.drawText(
+ badge.adjusted(8.0 + icon + 5.0, 0, 0, 0),
+ Qt.AlignmentFlag.AlignVCenter | Qt.AlignmentFlag.AlignLeft,
+ self.duration,
+ )
+
+
+class RecordingPreviews(QObject):
+ """A bounded bottom-right stack, using the screenshot flight animation unchanged."""
+
+ def __init__(self, emit, activity) -> None:
+ super().__init__()
+ self.emit = emit
+ self.activity = activity
+ self.cards = []
+ self.flights = []
+ self.hidden = False
+
+ @property
+ def active(self) -> bool:
+ return bool(self.cards or self.flights)
+
+ def show(self, payload: dict) -> None:
+ from jarvis.cu.indicator.renderer import _match_screen, _SnapWindow
+
+ path = Path(str(payload.get("video_path", "")))
+ if not path.is_file() or path.suffix.lower() != ".mp4":
+ return
+ thumb = QImage.fromData(
+ QByteArray.fromBase64(str(payload.get("thumb", "")).encode("ascii"))
+ )
+ screen = next(
+ (s for s in QGuiApplication.screens() if s.name() == payload.get("screen_name")), None
+ )
+ screen = screen or _match_screen(payload.get("monitor", []))
+ if thumb.isNull() or screen is None:
+ return
+ for flight in list(self.flights):
+ self.land(flight)
+ while len(self.cards) >= protocol.MAX_CARDS:
+ self.cards[0].close()
+
+ def landed(rect, image, *_args):
+ card = RecordingCard(self, screen, rect, image, payload)
+ self.cards.append(card)
+ self.layout_cards()
+ if not self.hidden:
+ card.show()
+ self.activity()
+
+ flight = _SnapWindow(
+ screen,
+ payload.get("rect", [0, 0, 1, 1]),
+ thumb,
+ landed,
+ self.flight_done,
+ corner="right",
+ )
+ self.flights.append(flight)
+ self.activity()
+ flight.start()
+
+ def flight_done(self, flight) -> None:
+ if flight in self.flights:
+ self.flights.remove(flight)
+ self.layout_cards()
+ self.activity()
+
+ @staticmethod
+ def land(flight) -> None:
+ flight._anim.stop()
+ flight._landed()
+
+ def remove(self, card) -> None:
+ if card in self.cards:
+ self.cards.remove(card)
+ self.layout_cards()
+ self.activity()
+
+ def layout_cards(self) -> None:
+ from jarvis.cu.indicator.renderer import _SNAP_MARGIN
+
+ for screen in QGuiApplication.screens():
+ available = screen.availableGeometry()
+ bottom = available.bottom() + 1 - _SNAP_MARGIN
+ for card in reversed(self.cards):
+ if card.screen_name != screen.name():
+ continue
+ height = card.height() - 24
+ if bottom - height < available.top() + _SNAP_MARGIN:
+ card.close() # remove() lays out the remaining bounded stack again.
+ return
+ right = available.right() + 1 - _SNAP_MARGIN
+ card.move(QPoint(right - card.width() + 12, bottom - height - 12))
+ bottom -= height + 12
+
+ def set_status(self, recording_id: str, text: str) -> bool:
+ for card in self.cards:
+ if card.recording_id == recording_id:
+ card.show_status(text)
+ return True
+ return False
+
+ def suspend(self, hidden: bool) -> None:
+ self.hidden = hidden
+ for flight in list(self.flights):
+ self.land(flight)
+ for card in self.cards:
+ card.setVisible(not hidden)
+
+ def close(self) -> None:
+ for flight in list(self.flights):
+ flight.finish_now()
+ for card in list(self.cards):
+ card.close()
diff --git a/jarvis/appshot/recording_runtime.py b/jarvis/appshot/recording_runtime.py
new file mode 100644
index 0000000000..1c8f057f4e
--- /dev/null
+++ b/jarvis/appshot/recording_runtime.py
@@ -0,0 +1,99 @@
+"""Prepare one pixel-free recording process and transfer it on explicit start."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import sys
+from pathlib import Path
+
+from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS
+
+log = logging.getLogger(__name__)
+_READY_TIMEOUT_S = 20.0
+
+
+class RecordingRuntime:
+ """Own a dormant worker until RecordingService takes responsibility for it."""
+
+ def __init__(self) -> None:
+ self._lock = asyncio.Lock()
+ self._process: asyncio.subprocess.Process | None = None
+ self._active = False
+
+ async def prewarm(self) -> None:
+ async with self._lock:
+ if not self._active:
+ await self._ensure()
+
+ async def _ensure(self) -> asyncio.subprocess.Process:
+ if self._process is not None and self._process.returncode is None:
+ return self._process
+ await self._dispose()
+ spawn = asyncio.create_task(asyncio.create_subprocess_exec(
+ sys.executable, "-m", "jarvis.appshot.recording_worker", "--standby",
+ stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.DEVNULL, creationflags=NO_WINDOW_CREATIONFLAGS,
+ ))
+ try:
+ try:
+ self._process = await asyncio.shield(spawn)
+ except asyncio.CancelledError:
+ self._process = await spawn # A cancelled spawn still owns its late child.
+ raise
+ async with asyncio.timeout(_READY_TIMEOUT_S):
+ assert self._process.stdout is not None
+ while raw := await self._process.stdout.readline():
+ try:
+ event = json.loads(raw.decode("utf-8"))
+ except (ValueError, UnicodeError):
+ log.debug("appshot: ignoring non-protocol recorder startup output")
+ continue
+ if isinstance(event, dict) and event.get("phase") == "ready":
+ return self._process
+ raise OSError("The recording runtime closed before becoming ready.")
+ except BaseException:
+ await self._dispose()
+ raise
+
+ async def take(
+ self, output: Path, language: str, *, options: dict | None = None
+ ) -> asyncio.subprocess.Process:
+ async with self._lock:
+ process = await self._ensure()
+ try:
+ assert process.stdin is not None
+ command = {"cmd": "start", "output": str(output), "language": language}
+ if options is not None:
+ command["options"] = options
+ process.stdin.write((json.dumps(command) + "\n").encode("utf-8"))
+ await process.stdin.drain()
+ except BaseException:
+ await self._dispose()
+ raise
+ self._process = None # The recording service now owns shutdown and stdout.
+ self._active = True
+ return process
+
+ def release(self) -> None:
+ """The transferred worker has exited; another standby may now be prepared."""
+ self._active = False
+
+ async def _dispose(self) -> None:
+ process, self._process = self._process, None
+ if process is None:
+ return
+ if process.stdin is not None:
+ process.stdin.close() # EOF quits standby without ever starting capture.
+ try:
+ await asyncio.wait_for(process.wait(), 2)
+ except TimeoutError:
+ log.debug("appshot: terminating an unresponsive standby recorder")
+ if process.returncode is None:
+ process.kill()
+ await process.wait()
+
+ async def close(self) -> None:
+ async with self._lock:
+ await self._dispose()
diff --git a/jarvis/appshot/recording_worker.py b/jarvis/appshot/recording_worker.py
index f7b5991688..2451f1051f 100644
--- a/jarvis/appshot/recording_worker.py
+++ b/jarvis/appshot/recording_worker.py
@@ -1,4 +1,4 @@
-"""Visible capture UI, loaded only in the user-started recorder process."""
+"""Warm desktop runtime; capture and visible UI require an explicit start command."""
from __future__ import annotations
@@ -18,7 +18,10 @@ def emit(phase: str, **values: object) -> None:
pass
-def run(output: Path, language: str = "en") -> int:
+def run(
+ output: Path | None, language: str = "en", *, standby: bool = False,
+ options: dict | None = None,
+) -> int:
import importlib
# Load native DLLs on the sidecar main thread before Qt's event loop.
@@ -31,7 +34,6 @@ def run(output: Path, language: str = "en") -> int:
from PySide6.QtGui import QImage, QPainter, QPen
from PySide6.QtWidgets import (
QApplication,
- QHBoxLayout,
QLabel,
QPushButton,
QVBoxLayout,
@@ -39,25 +41,41 @@ def run(output: Path, language: str = "en") -> int:
)
from jarvis.appshot.picker.renderer import Picker, _SelectWindow
+ from jarvis.appshot.recording_frame import owned_capture_image
+ from jarvis.appshot.recording_hud import RecordingHint, RecordingHud
from jarvis.appshot.recording_labels import LABELS
+ from jarvis.appshot.recording_options import RecordingOptions, effective_fps, output_size
from jarvis.appshot.region import selection_fractions
- from jarvis.appshot.video_encoder import FPS, VideoEncoder
+ from jarvis.appshot.video_encoder import FirstFramePoster, VideoEncoder
from jarvis.platform.probes import is_wayland
+ # Importing the backend in standby opens no capture device or portal.
+ importlib.import_module("PySide6.QtMultimedia")
+
app = QApplication(["Personal Jarvis — Screen recording"])
from jarvis.platform.qt_sidecar import hide_from_dock
hide_from_dock() # macOS: no "Python" Dock icon for the standby recorder
app.setQuitOnLastWindowClosed(False)
labels = LABELS.get(language, LABELS["en"])
+ preferences = RecordingOptions.model_validate(options or {})
class ParentPipe(QObject):
stop = Signal()
+ start = Signal(dict)
def read(self) -> None:
try:
- for _line in sys.stdin:
- self.stop.emit()
+ for line in sys.stdin:
+ if line.strip() == "stop":
+ self.stop.emit()
+ continue
+ try:
+ command = json.loads(line)
+ except ValueError:
+ continue # Ignore non-protocol input; EOF still stops the worker.
+ if isinstance(command, dict) and command.get("cmd") == "start":
+ self.start.emit(command)
except OSError:
pass # A closed pipe means the parent exited: always stop below.
self.stop.emit()
@@ -143,16 +161,24 @@ def run(output: Path, language: str = "en") -> int:
def start(self) -> None:
super().start()
for window in self._windows:
- panel = QWidget(window)
- panel.setAutoFillBackground(True)
- layout = QHBoxLayout(panel)
- layout.addWidget(QLabel(labels["select"]))
- button = QPushButton(labels["full"])
- button.clicked.connect(
- lambda _checked=False, w=window: self.finish(w, (0, 0, 1, 1))
+ screen = window.screen_ref
+ dpr = screen.devicePixelRatio()
+ size = output_size(
+ round(screen.size().width() * dpr), round(screen.size().height() * dpr),
+ preferences.resolution,
+ )
+ screen_labels = dict(labels)
+ screen_labels["select"] += (
+ f"\n{screen.name()} · {size[0]} × {size[1]} · "
+ f"{effective_fps(preferences.fps, screen.refreshRate())} FPS · "
+ f"{preferences.bitrate_mbps} Mbps"
+ )
+ panel = RecordingHint(
+ window,
+ screen_labels,
+ lambda w=window: self.finish(w, (0, 0, 1, 1)),
+ window.devicePixelRatioF(),
)
- layout.addWidget(button)
- panel.adjustSize()
panel.move(max(0, (window.width() - panel.width()) // 2), 24)
panel.show()
@@ -190,6 +216,10 @@ def run(output: Path, language: str = "en") -> int:
self.completed = False
self.error = ""
self.last_image = None
+ self.poster = FirstFramePoster()
+ self.hud = None
+ #: Full-depth capture when the monitor runs HDR / wide gamut (Windows).
+ self.hdr = None
self.preview = None
self.capture = None
self.capture_session = None
@@ -198,47 +228,56 @@ def run(output: Path, language: str = "en") -> int:
self.last_frame = 0.0
self.dimensions = None
self.permission_checked = 0.0
+ self.fps = effective_fps(preferences.fps, None)
+ self.refresh_hz = None
+ self.monitor_name = ""
self.timer = QTimer(self)
+ self.timer.setTimerType(Qt.TimerType.PreciseTimer)
self.timer.timeout.connect(self.tick)
- self.timer.start(round(1000 / FPS))
- self.selection_deadline = time.monotonic() + 180
+ self.timer.start(max(1, 1000 // self.fps))
+ self.selection_deadline = time.perf_counter() + 180
def begin(self) -> None:
emit("selecting")
if is_wayland():
- from PySide6.QtMultimedia import QMediaCaptureSession, QScreenCapture, QVideoSink
-
self.show()
- self.capture_session = QMediaCaptureSession(self)
- self.capture = QScreenCapture(self)
- self.sink = QVideoSink(self)
- self.capture_session.setScreenCapture(self.capture)
- self.capture_session.setVideoSink(self.sink)
- self.sink.videoFrameChanged.connect(self.portal_frame)
- self.capture.activeChanged.connect(
- lambda active: None if active or self.finishing else self.stop()
- )
- self.capture.errorOccurred.connect(
- lambda _code, _text: self.fail(
- "Screen sharing was denied or unavailable. "
- "Check PipeWire and the ScreenCast portal."
- )
- )
- self.capture.start()
+ self.start_capture()
else:
self.picker = RecordingPicker(app)
self.picker.start()
+ emit("selection_ready")
+
+ def start_capture(self) -> None:
+ from PySide6.QtMultimedia import QMediaCaptureSession, QScreenCapture, QVideoSink
+
+ self.capture_session = QMediaCaptureSession(self)
+ self.capture = QScreenCapture(self)
+ self.sink = QVideoSink(self)
+ if self.screen is not None:
+ self.capture.setScreen(self.screen)
+ self.capture_session.setScreenCapture(self.capture)
+ self.capture_session.setVideoSink(self.sink)
+ self.sink.videoFrameChanged.connect(self.portal_frame)
+ self.capture.activeChanged.connect(
+ lambda active: None if active or self.finishing else self.stop()
+ )
+ self.capture.errorOccurred.connect(
+ lambda _code, _text: self.fail(
+ "Screen capture was denied or unavailable. Check screen sharing permissions."
+ )
+ )
+ self.capture.start()
def portal_frame(self, frame) -> None:
- now = time.monotonic()
- if now - self.last_frame < 1 / FPS or self.finishing:
+ now = time.perf_counter()
+ if self.finishing:
return
self.last_frame = now
- image = frame.toImage()
+ image = owned_capture_image(frame, opaque_desktop=sys.platform == "win32")
if image.isNull():
return
self.last_image = image
- if self.preview is None and self.frac is None:
+ if is_wayland() and self.preview is None and self.frac is None:
self.label.setText(labels["preview"])
self.preview = Preview(image, self)
self.layout_box.insertWidget(1, self.preview)
@@ -258,17 +297,44 @@ def run(output: Path, language: str = "en") -> int:
self.full_button.hide()
self.label.setText(labels["starting"])
self.stop_button.setText(labels["stop"])
- self.resize(290, 90)
- self.show()
+ if screen is not None:
+ self.hud = RecordingHud(screen, frac, labels, self.stop)
+ self.hide()
+ self.hud.show()
+ else:
+ # The portal does not expose the source's global screen position.
+ self.resize(290, 90)
+ self.show()
# Let the selection overlays disappear before capturing pixels.
QTimer.singleShot(250, self.start_encoder)
def start_encoder(self) -> None:
if self.finishing:
return
- self.encoder = VideoEncoder(output)
+ self.refresh_hz = self.screen.refreshRate() if self.screen is not None else None
+ self.monitor_name = self.screen.name() if self.screen is not None else ""
+ self.fps = effective_fps(preferences.fps, self.refresh_hz)
+ self.timer.setInterval(max(1, 1000 // self.fps))
+ if self.hud is not None:
+ from jarvis.appshot.hdr_recording import choose_hdr_capture
+
+ # An HDR or wide-gamut monitor is read at full depth instead.
+ self.hdr = choose_hdr_capture(
+ int(self.hud.frame.winId()), tuple(self.frac),
+ preferences.resolution, self.fps,
+ )
+ # Python 3.11's Windows monotonic clock can tick only every 15.6 ms.
+ # QPC keeps 60/120 FPS frames and the audio epoch distinguishable.
+ self.started = time.perf_counter()
+ self.encoder = VideoEncoder(
+ output, options=preferences, fps=self.fps, epoch=self.started,
+ hdr=self.hdr is not None,
+ )
self.encoder.start()
- self.started = time.monotonic()
+ if self.hdr is not None:
+ self.hdr.start(self.encoder, self.started)
+ elif self.capture is None:
+ self.start_capture()
def tick(self) -> None:
try:
@@ -285,13 +351,13 @@ def run(output: Path, language: str = "en") -> int:
self.finish()
return
if self.started is None or self.encoder is None:
- if time.monotonic() > self.selection_deadline:
+ if time.perf_counter() > self.selection_deadline:
self.stop()
return
if self.encoder.error:
self.fail(self.encoder.error)
return
- elapsed = time.monotonic() - self.started
+ elapsed = time.perf_counter() - self.started
if sys.platform == "darwin" and elapsed - self.permission_checked >= 1.0:
from jarvis.platform import screen_access
@@ -305,12 +371,20 @@ def run(output: Path, language: str = "en") -> int:
)
return
if self.screen is not None:
- image = self.screen.grabWindow(0).toImage()
- else:
- image = self.last_image
- if image is None or image.isNull():
- self.fail("Screen capture returned no image. Check screen recording permission.")
+ if effective_fps(preferences.fps, self.screen.refreshRate()) != self.fps:
+ self.fail("The display refresh rate changed. Start a new recording.")
+ return
+ if self.hdr is not None:
+ if self.hdr.error:
+ self.fail(self.hdr.error)
+ return
+ if self.poster.image is None and self.hdr.poster:
+ self.poster.image = self.hdr.poster
+ self.report(elapsed)
return
+ image = self.last_image
+ if image is None or image.isNull():
+ return # Startup above has a bounded first-frame deadline.
size = (image.width(), image.height())
if self.dimensions is not None and size != self.dimensions:
self.fail("The screen resolution changed. Start a new recording for the new size.")
@@ -323,16 +397,26 @@ def run(output: Path, language: str = "en") -> int:
max(1, round(width * size[0])),
max(1, round(height * size[1])),
)
- crop = image.copy(rect.intersected(image.rect())).convertToFormat(
- QImage.Format.Format_RGBA8888
- )
+ rect = rect.intersected(image.rect())
+ # QImage is implicitly shared and the encoder only reads it. Avoid
+ # copying a full 4K frame on every GUI tick when no crop is needed.
+ crop = image if rect == image.rect() else image.copy(rect)
self.encoder.submit(crop, elapsed)
+ self.poster.observe(crop)
+ self.report(elapsed)
+
+ def report(self, elapsed: float) -> None:
if self.encoder.ready.is_set() and not self.reported:
self.reported = True
- emit("recording", width=self.encoder.size[0], height=self.encoder.size[1])
+ emit("recording", width=self.encoder.size[0], height=self.encoder.size[1],
+ fps=self.fps, bitrate_mbps=preferences.bitrate_mbps,
+ system_audio=preferences.system_audio, display_name=self.monitor_name,
+ refresh_hz=self.refresh_hz)
if self.reported:
seconds = int(elapsed)
self.label.setText(f"Recording · {seconds // 60:02d}:{seconds % 60:02d}")
+ if self.hud is not None:
+ self.hud.update_elapsed(elapsed)
def fail(self, message: str) -> None:
self.error = message
@@ -342,14 +426,18 @@ def run(output: Path, language: str = "en") -> int:
if self.finishing:
return
self.finishing = True
+ if self.hud is not None:
+ self.hud.hide()
if self.picker:
self.picker.finish(None, None)
if self.capture:
self.capture.stop()
+ if self.hdr is not None:
+ self.hdr.stop()
self.label.setText(labels["saving"])
self.stop_button.setEnabled(False)
if self.encoder:
- self.encoder.stop(max(0, time.monotonic() - self.started))
+ self.encoder.stop(max(0, time.perf_counter() - self.started))
# A stuck native encoder cannot keep recording or leave an orphan.
QTimer.singleShot(15000, self.force_finish)
@@ -372,11 +460,50 @@ def run(output: Path, language: str = "en") -> int:
duration_s=round(self.encoder.duration, 1),
width=self.encoder.size[0],
height=self.encoder.size[1],
+ fps=self.fps,
+ bitrate_mbps=preferences.bitrate_mbps,
+ system_audio=preferences.system_audio,
+ display_name=self.monitor_name,
+ refresh_hz=self.refresh_hz,
+ dropped_frames=self.encoder.dropped_frames,
+ **self.save_preview(),
)
else:
emit("cancelled")
+ if self.hud is not None:
+ self.hud.close()
app.quit()
+ def save_preview(self) -> dict:
+ try:
+ return self._save_preview()
+ except Exception:
+ import logging
+
+ logging.getLogger(__name__).warning(
+ "appshot: saved video preview is unavailable", exc_info=True
+ )
+ return {}
+
+ def _save_preview(self) -> dict:
+ if self.poster.image is None:
+ return {}
+ thumbnail = self.poster.image.scaled(
+ 560, 560, Qt.AspectRatioMode.KeepAspectRatio,
+ Qt.TransformationMode.SmoothTransformation,
+ )
+ poster = output.with_suffix(".jpg")
+ if not thumbnail.save(str(poster), "JPG", 82):
+ return {} # An optional poster failure does not invalidate the video.
+ screen = self.screen or app.primaryScreen()
+ if screen is None:
+ return {}
+ return {
+ "preview": True, "screen_name": screen.name(),
+ "monitor": list(screen.geometry().getRect()),
+ "rect": list(self.frac or (0, 0, 1, 1)),
+ }
+
def closeEvent(self, event) -> None: # noqa: N802
if self.completed:
event.accept()
@@ -384,12 +511,37 @@ def run(output: Path, language: str = "en") -> int:
event.ignore()
self.stop()
- session = Session()
+ session = None
+ closing = False
pipe = ParentPipe()
- pipe.stop.connect(session.stop)
- threading.Thread(target=pipe.read, name="appshot-recording-parent", daemon=True).start()
- def begin() -> None:
+ def stop() -> None:
+ nonlocal closing
+ closing = True
+ if session is None:
+ app.quit()
+ else:
+ session.stop()
+
+ def begin(command=None) -> None:
+ nonlocal session, output, labels, preferences
+ if session is not None or closing:
+ return
+ if command is not None:
+ value = command.get("output")
+ if not isinstance(value, str) or not value:
+ emit("error", message="The recording output is missing.")
+ stop()
+ return
+ output = Path(value)
+ labels = LABELS.get(command.get("language", "en"), LABELS["en"])
+ try:
+ preferences = RecordingOptions.model_validate(command.get("options", {}))
+ except ValueError:
+ emit("error", message="The recording settings are invalid.")
+ stop()
+ return
+ session = Session()
try:
if sys.platform == "darwin":
from jarvis.platform import screen_access
@@ -407,7 +559,13 @@ def run(output: Path, language: str = "en") -> int:
logging.getLogger(__name__).exception("appshot: capture initialization failed")
session.fail("Screen capture could not start. Check desktop support and permissions.")
- QTimer.singleShot(0, begin)
+ pipe.stop.connect(stop)
+ pipe.start.connect(begin)
+ threading.Thread(target=pipe.read, name="appshot-recording-parent", daemon=True).start()
+ if standby:
+ emit("ready") # No Session, windows, timers, pixels or permission prompts in standby.
+ else:
+ QTimer.singleShot(0, begin)
return app.exec()
@@ -415,13 +573,16 @@ def main() -> int:
from jarvis.appshot.recording_labels import LABELS
parser = argparse.ArgumentParser(description=__doc__)
- parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--output", type=Path)
+ parser.add_argument("--standby", action="store_true")
parser.add_argument("--language", choices=tuple(LABELS), default="en")
args = parser.parse_args()
+ if not args.standby and args.output is None:
+ parser.error("--output is required unless --standby is used")
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8")
try:
- return run(args.output, args.language)
+ return run(args.output, args.language, standby=args.standby)
except Exception:
import logging
diff --git a/jarvis/appshot/region.py b/jarvis/appshot/region.py
index 3c9df1e414..36057c847b 100644
--- a/jarvis/appshot/region.py
+++ b/jarvis/appshot/region.py
@@ -1,6 +1,6 @@
"""Select an area of the screen for a region appshot.
-:func:`pick_region` starts the one-shot picker sidecar
+:func:`pick_region` opens a selection in the prepared picker sidecar
(:mod:`jarvis.appshot.picker`), waits for the user to drag a rectangle and
returns it; :func:`selection_to_bbox` turns that into capture coordinates.
@@ -48,7 +48,7 @@ MIN_SELECTION_PX = 4
#: The picker gives up on its own after this long without a result. Long
#: enough to mark an area up in peace; Esc still ends it at once.
PICK_TIMEOUT_S = 900.0
-#: Grace for the picker to exit (and its overlay to leave the glass).
+#: Grace for a stopped or failed picker process to exit.
_EXIT_GRACE_S = 2.0
#: Extra settle time so the compositor has repainted without the dim layer.
_SETTLE_S = 0.08
@@ -296,12 +296,13 @@ def snap_layout() -> dict[str, Any]:
return {"monitors": [dict(m) for m in monitors], "windows": windows}
-def _spawn(language: str = "en") -> subprocess.Popen[str]:
+def _spawn(language: str = "en", *, resident: bool = False) -> subprocess.Popen[str]:
from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS # noqa: PLC0415
from jarvis.platform.qt_sidecar import stderr_sink # noqa: PLC0415
return subprocess.Popen(
- [sys.executable, "-m", "jarvis.appshot.picker", "--lang", language or "en"],
+ [sys.executable, "-m", "jarvis.appshot.picker", "--lang", language or "en"]
+ + (["--resident"] if resident else []),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=stderr_sink("appshot-picker"),
@@ -313,7 +314,8 @@ def _spawn(language: str = "en") -> subprocess.Popen[str]:
def _read_result(
- proc: subprocess.Popen[str], on_marking: Callable[[], None] | None = None
+ proc: subprocess.Popen[str], on_marking: Callable[[], None] | None = None,
+ on_ready: Callable[[], None] | None = None,
) -> dict[str, Any] | None:
"""Block until the picker reports its selection; ``None`` on EOF.
@@ -330,6 +332,8 @@ def _read_result(
event = payload.get("event")
if event == wire.EVENT_SELECTION:
return payload
+ if event == wire.EVENT_READY and on_ready is not None:
+ on_ready()
if event == wire.EVENT_MARKING and on_marking is not None:
with contextlib.suppress(Exception): # a lost hint only keeps the global Esc
on_marking()
@@ -466,55 +470,15 @@ def _exit_message(code: int) -> str:
async def _run_picker(timeout_s: float) -> tuple[dict[str, Any] | None, int | None, bool]:
- """Spawn, wait for one selection, always reap. ``(payload, exit code, timed out)``."""
- try:
- layout = await asyncio.to_thread(snap_layout)
- except Exception: # noqa: BLE001 - no snapping; dragging still works
- log.debug("appshot: snap layout unavailable", exc_info=True)
- layout = None
- # Shielded: a caller cancelled mid-spawn must still get the process reaped,
- # or its overlay would cover the screens until the stdin timeout.
- spawn = asyncio.ensure_future(asyncio.to_thread(_spawn, _language.get()))
+ """Use the prepared runtime; only the explicit pick samples the desktop."""
+ from jarvis.appshot.picker_host import get_picker_host
+
try:
- proc = await asyncio.shield(spawn)
- except asyncio.CancelledError:
- with contextlib.suppress(Exception): # a failed spawn leaves nothing to reap
- late = await spawn
- await asyncio.to_thread(_cancel, late)
- await asyncio.to_thread(_reap, late)
- raise
- except Exception as exc: # noqa: BLE001 - surfaced to the user as unavailable
+ return await get_picker_host().pick(timeout_s, _language.get())
+ except (OSError, TimeoutError) as exc:
log.warning("appshot: area picker could not be started", exc_info=True)
raise RegionUnavailable("The selection overlay could not be started.") from exc
- if layout is not None:
- await asyncio.to_thread(_send_layout, proc, layout)
- loop = asyncio.get_running_loop()
- escape = loop.create_task(_escape_cancels(proc), name="appshot-region-esc")
-
- def marking() -> None:
- # The picker has keyboard focus now, and Esc there may only end a
- # text box: the global Esc must stop cancelling the whole picker.
- loop.call_soon_threadsafe(escape.cancel)
-
- reader = asyncio.ensure_future(asyncio.to_thread(_read_result, proc, marking))
- timed_out = False
- payload: dict[str, Any] | None = None
- try:
- try:
- payload = await asyncio.wait_for(asyncio.shield(reader), timeout=timeout_s)
- except TimeoutError: # the timeout is reported through timed_out
- timed_out = True
- await asyncio.to_thread(_cancel, proc)
- finally:
- escape.cancel()
- with contextlib.suppress(asyncio.CancelledError):
- await escape
- if payload is None and not timed_out:
- await asyncio.to_thread(_cancel, proc) # a cancelled caller closes the overlay
- code = await asyncio.to_thread(_reap, proc)
- with contextlib.suppress(Exception): # EOF after the reap ends the reader
- await reader
- return payload, code, timed_out
+
__all__ = [
diff --git a/jarvis/appshot/video_edit.py b/jarvis/appshot/video_edit.py
new file mode 100644
index 0000000000..eec386d768
--- /dev/null
+++ b/jarvis/appshot/video_edit.py
@@ -0,0 +1,215 @@
+"""Light edits of a finished screen recording: trim it and change its speed.
+
+The video editor plays the original file and only sends the chosen range and
+speed. This module writes that as a new recording next to the original (the
+original is never touched), so it can be saved, dragged or opened like any
+other recording. PyAV is imported lazily: a base install without the video
+extra still boots, and the routes answer with a clear error instead.
+"""
+
+from __future__ import annotations
+
+import logging
+import threading
+import uuid
+from fractions import Fraction
+from typing import Any
+
+from jarvis.appshot.recording import recording_dir, recording_file
+
+log = logging.getLogger(__name__)
+
+#: Playback speeds the editor offers; anything else is refused.
+SPEEDS = (0.5, 1.0, 1.5, 2.0)
+#: A trim this close to the original's ends counts as "not trimmed".
+_EDGE_S = 0.05
+
+_lock = threading.Lock()
+#: ``(source id, start, end, speed)`` → the clip already written for it.
+_clips: dict[tuple[str, float, float, float], str] = {}
+
+
+class ClipError(ValueError):
+ """The edit cannot be written; the message is safe to show the user."""
+
+
+def probe_duration(recording_id: str) -> float:
+ """Length of a finished recording in seconds."""
+ path = recording_file(recording_id)
+ if path is None:
+ raise FileNotFoundError(recording_id)
+ import av
+
+ with av.open(str(path)) as container:
+ if container.duration:
+ return container.duration / 1_000_000
+ stream = container.streams.video[0]
+ if stream.duration and stream.time_base:
+ return float(stream.duration * stream.time_base)
+ return 0.0
+
+
+def export_clip(recording_id: str, start_s: float, end_s: float, speed: float) -> str:
+ """Write ``[start_s, end_s]`` at ``speed`` as a new recording; returns its id.
+
+ An unchanged range at normal speed returns ``recording_id`` itself, and an
+ edit already written in this session is reused instead of encoded twice.
+ """
+ if speed not in SPEEDS:
+ raise ClipError("This playback speed is not supported.")
+ duration = probe_duration(recording_id)
+ start = max(0.0, min(start_s, duration))
+ end = max(0.0, min(end_s, duration))
+ if end - start < 0.2:
+ raise ClipError("The selected part of the video is too short.")
+ if start <= _EDGE_S and end >= duration - _EDGE_S and speed == 1.0:
+ return recording_id
+ key = (recording_id, round(start, 2), round(end, 2), speed)
+ with _lock:
+ cached = _clips.get(key)
+ if cached is not None and recording_file(cached) is not None:
+ return cached
+ clip_id = uuid.uuid4().hex
+ _write_clip(recording_id, clip_id, start, end, speed)
+ _clips[key] = clip_id
+ return clip_id
+
+
+#: Transfer characteristics of HDR video: SMPTE ST 2084 (PQ) and ARIB STD-B67 (HLG).
+_HDR_TRANSFERS = (16, 18)
+
+
+def _clip_encoder(av: Any, source: Any) -> tuple[str, str, dict[str, str]]:
+ """An encoder that keeps the recording's colour: 10-bit for HDR, 8-bit otherwise."""
+ if int(getattr(source, "color_trc", 0) or 0) in _HDR_TRANSFERS:
+ for codec, pixel_format, options in (
+ ("libsvtav1", "yuv420p10le", {"preset": "8", "crf": "26"}),
+ ("libx265", "yuv420p10le", {"preset": "veryfast", "crf": "20"}),
+ ("av1_nvenc", "p010le", {"preset": "p5", "rc": "vbr", "cq": "24"}),
+ ("hevc_nvenc", "p010le", {"preset": "p5", "rc": "vbr", "cq": "22",
+ "profile": "main10"}),
+ ):
+ if codec in av.codecs_available:
+ return codec, pixel_format, options
+ if "libx264" in av.codecs_available:
+ return "libx264", "yuv420p", {"preset": "veryfast", "crf": "20"}
+ return "mpeg4", "yuv420p", {}
+
+
+#: Whole-number rates the recorder captures at (its ``fps`` is an int); a
+#: measured rate near one of them is that rate.
+_NOMINAL_RATES = tuple(
+ Fraction(rate) for rate in (24, 25, 30, 48, 50, 60, 72, 90, 100, 120, 144, 165, 240)
+)
+
+
+def _nominal_rate(video: Any) -> Fraction:
+ """The rate the recording was captured at, not one skewed by timestamp jitter.
+
+ VFR timestamp jitter can make base_rate report 120/60000 for a 60 FPS
+ recording, and encoders that write no timing info (VideoToolbox) leave only
+ the average rate, which jitter skews to e.g. 4000/67. Prefer the codec's
+ declared rate, then the average, and snap either to a capture rate within 2 %.
+ """
+ measured = video.codec_context.framerate or video.average_rate or Fraction(60)
+ measured = Fraction(measured)
+ nearest = min(_NOMINAL_RATES, key=lambda rate: abs(rate - measured))
+ if abs(nearest - measured) <= nearest * Fraction(2, 100):
+ return nearest
+ return measured
+
+
+def _copy_colour_tags(source: Any, target: Any) -> None:
+ """The clip says the same colour space as its recording, so players match it."""
+ for name in ("color_primaries", "color_trc", "colorspace", "color_range"):
+ value = getattr(source, name, None)
+ if value:
+ setattr(target, name, value)
+
+
+def _write_clip(source_id: str, clip_id: str, start: float, end: float, speed: float) -> None:
+ import av
+
+ source = recording_file(source_id)
+ if source is None:
+ raise FileNotFoundError(source_id)
+ target = recording_dir() / f"{clip_id}.mp4"
+ partial = target.with_suffix(".partial")
+ try:
+ with av.open(str(source)) as reader, av.open(str(partial), "w", format="mp4") as writer:
+ video = reader.streams.video[0]
+ fps = _nominal_rate(video)
+ time_base = 1 / fps
+ video.thread_type = "AUTO"
+ codec, pixel_format, options = _clip_encoder(av, video.codec_context)
+ stream = writer.add_stream(codec, rate=fps, options=options)
+ stream.width = video.codec_context.width
+ stream.height = video.codec_context.height
+ stream.pix_fmt = pixel_format
+ stream.codec_context.max_b_frames = 0
+ _copy_colour_tags(video.codec_context, stream.codec_context)
+ audio = None
+ if reader.streams.audio:
+ audio = writer.add_stream("aac", rate=48000)
+ audio.layout = "stereo"
+ audio.bit_rate = 192000
+
+ def encode(frame: Any, pts: int) -> None:
+ out = frame.reformat(format=pixel_format)
+ out.pts = pts
+ out.time_base = time_base
+ for packet in stream.encode(out):
+ writer.mux(packet)
+
+ # Seek to the keyframe before the cut, then decode up to it: the
+ # frame on screen at ``start`` opens the clip.
+ reader.seek(max(0, int(start / video.time_base)), stream=video, backward=True)
+ held = None
+ last_pts = -1
+ for frame in reader.decode(video):
+ if frame.pts is None:
+ continue
+ at = float(frame.pts * video.time_base)
+ if at <= start:
+ held = frame
+ continue
+ if at > end:
+ break
+ if last_pts < 0 and held is not None:
+ encode(held, 0)
+ last_pts = 0
+ pts = round((at - start) / speed * fps)
+ if pts <= last_pts:
+ continue # Faster than real time: this frame is skipped.
+ encode(frame, pts)
+ last_pts = pts
+ held = frame
+ if held is None:
+ raise ClipError("The selected part of the video has no frames.")
+ if last_pts < 0:
+ encode(held, 0)
+ last_pts = 0
+ # Hold the last picture until the chosen end, like the recorder does.
+ final = max(0, round((end - start) / speed * fps) - 1)
+ if final > last_pts:
+ encode(held, final)
+ for packet in stream.encode(None):
+ writer.mux(packet)
+ if audio is not None:
+ from jarvis.appshot.recording_audio_edit import write_audio
+
+ write_audio(source, writer, audio, start, end, speed)
+ partial.replace(target)
+ except (ClipError, FileNotFoundError):
+ raise
+ except Exception as exc:
+ log.exception("appshot: the edited video could not be written")
+ raise ClipError("The edited video could not be written.") from exc
+ finally:
+ try:
+ partial.unlink(missing_ok=True)
+ except OSError:
+ log.warning("appshot: partial clip cleanup failed", exc_info=True)
+
+
+__all__ = ["SPEEDS", "ClipError", "export_clip", "probe_duration"]
diff --git a/jarvis/appshot/video_encoder.py b/jarvis/appshot/video_encoder.py
index a38298732c..39289f4f28 100644
--- a/jarvis/appshot/video_encoder.py
+++ b/jarvis/appshot/video_encoder.py
@@ -5,19 +5,51 @@ from __future__ import annotations
import logging
import queue
import threading
+import time
from fractions import Fraction
from pathlib import Path
from typing import Any
+from jarvis.appshot.recording_codec import VIDEO_TIME_BASE
+from jarvis.appshot.recording_options import RecordingOptions, output_size
+
log = logging.getLogger(__name__)
-FPS = 30
+FPS = 60
+
+
+class FirstFramePoster:
+ """Keep a small immutable copy of the first submitted frame for the video card."""
+
+ def __init__(self) -> None:
+ self.image: Any = None
+
+ def observe(self, image: Any) -> None:
+ if self.image is not None:
+ return
+ from PySide6.QtCore import Qt
+
+ self.image = image.scaled(
+ 560, 560, Qt.AspectRatioMode.KeepAspectRatio,
+ Qt.TransformationMode.SmoothTransformation,
+ )
class VideoEncoder:
"""Native encoder calls stay off the UI thread and in a disposable process."""
- def __init__(self, path: Path) -> None:
+ def __init__(
+ self, path: Path, *, options: RecordingOptions | None = None,
+ fps: int | None = None, epoch: float | None = None, hdr: bool = False,
+ ) -> None:
self.path = path
+ #: HDR frames arrive as finished P010 planes ``(y, uv)`` at the output
+ #: size (:class:`jarvis.appshot.hdr_recording.HdrCapture`), not QImages.
+ self.hdr = hdr
+ self.options = options or RecordingOptions()
+ self.fps = fps or self.options.fps
+ self.epoch = epoch if epoch is not None else time.perf_counter()
+ self.dropped_frames = 0
+ self._audio: Any = None
self.partial = path.with_suffix(".partial")
self.frames: queue.Queue[tuple[Any, float]] = queue.Queue(maxsize=2)
self.stopping = threading.Event()
@@ -49,60 +81,116 @@ class VideoEncoder:
try:
import av
import numpy as np
+ from PySide6.QtCore import Qt
+ from PySide6.QtGui import QImage
+ if self.options.system_audio:
+ from jarvis.appshot.recording_audio import SystemAudioCapture
+
+ self._audio = SystemAudioCapture(self.epoch)
+ self._audio.start()
+ if not self._audio.ready.wait(timeout=5) or self._audio.error:
+ raise RuntimeError(self._audio.error or "System audio startup timed out.")
stream = None
+ audio_stream = None
+ source_size = None
last_pts = -1
+ encoded_frames = 0
last_array = None
while not self.stopping.is_set() or not self.frames.empty():
+ if self._audio is not None and self._audio.error:
+ raise RuntimeError(self._audio.error)
+ if audio_stream is not None:
+ self._drain_audio(av, container, audio_stream)
try:
image, elapsed = self.frames.get(timeout=0.1)
except queue.Empty: # Normal idle tick: recheck stop without blocking shutdown.
continue
if stream is None:
- width, height = image.width(), image.height()
+ width, height = self._source_size(image)
+ source_size = (width, height)
# Pad to even encoder dimensions instead of cropping user pixels.
- self.size = (width + width % 2, height + height % 2)
- container = av.open(str(self.partial), "w", format="mp4")
- codec = "libx264" if "libx264" in av.codecs_available else "mpeg4"
- options = (
- {"preset": "ultrafast", "crf": "23", "tune": "zerolatency"}
- if codec == "libx264"
- else {}
+ self.size = (
+ (width, height) if self.hdr
+ else output_size(width, height, self.options.resolution)
)
- stream = container.add_stream(codec, rate=FPS, options=options)
- stream.width, stream.height = self.size
- stream.pix_fmt = "yuv420p"
- stream.codec_context.thread_count = 2
- stream.codec_context.max_b_frames = 0
- if image.width() > self.size[0] or image.height() > self.size[1]:
+ from jarvis.appshot.recording_codec import open_video
+
+ container, stream = open_video(
+ av, self.partial, self.size, self.fps,
+ self.options.bitrate_mbps * 1_000_000, hdr=self.hdr,
+ )
+ if self._audio is not None:
+ audio_stream = container.add_stream("aac", rate=48000)
+ audio_stream.layout = "stereo"
+ audio_stream.bit_rate = 192000
+ if self._source_size(image) != source_size:
raise ValueError("The selected display changed size during recording.")
- pts = max(last_pts + 1, round(elapsed * FPS))
+ # Hold the first usable picture from time zero instead of an
+ # empty MP4 edit-list interval during capture/driver startup.
+ pts = 0 if last_pts < 0 else round(elapsed / VIDEO_TIME_BASE)
+ if pts <= last_pts:
+ continue # Never stretch wall-clock time when a timer fires early.
+ if self.hdr:
+ last_array = image
+ encoded = self._encode(av, container, stream, image, pts)
+ encoded_frames += 1
+ last_pts = pts
+ if encoded:
+ self.ready.set()
+ continue
+ if self.size[0] < image.width() or self.size[1] < image.height():
+ # Scaling and color conversion belong off the capture/UI thread.
+ # Resize before copying into PyAV to avoid converting 4K RGBA
+ # buffers when the selected output is only Full HD.
+ image = image.scaled(*self.size, Qt.AspectRatioMode.IgnoreAspectRatio,
+ Qt.TransformationMode.SmoothTransformation)
+ image = image.convertToFormat(QImage.Format.Format_RGBA8888)
# QImage RGBA rows can include alignment padding.
array = (
np.frombuffer(image.constBits(), dtype=np.uint8)
.reshape(image.height(), image.bytesPerLine())[:, : image.width() * 4]
.reshape(image.height(), image.width(), 4)
)
- padded = np.zeros((self.size[1], self.size[0], 4), dtype=np.uint8)
- padded[: image.height(), : image.width()] = array
+ width, height = image.width(), image.height()
+ if width % 2 or height % 2:
+ padded = np.zeros((height + height % 2, width + width % 2, 4), dtype=np.uint8)
+ padded[:height, :width] = array
+ else:
+ padded = np.ascontiguousarray(array)
last_array = padded
encoded = self._encode(av, container, stream, padded, pts)
+ encoded_frames += 1
last_pts = pts
if encoded:
self.ready.set()
if stream is not None and container is not None:
- final_pts = round(self.duration * FPS)
+ final_pts = max(0, round((self.duration - 1 / self.fps) / VIDEO_TIME_BASE))
if last_array is not None and final_pts > last_pts:
self._encode(av, container, stream, last_array, final_pts)
+ encoded_frames += 1
+ self.dropped_frames = max(0, round(self.duration * self.fps) - encoded_frames)
for packet in stream.encode(None):
container.mux(packet)
+ if self._audio is not None:
+ self._audio.stop()
+ if self._audio.error:
+ raise RuntimeError(self._audio.error)
+ self._drain_audio(av, container, audio_stream)
+ for packet in audio_stream.encode(None):
+ container.mux(packet)
container.close()
container = None
self.partial.replace(self.path)
except Exception:
log.exception("appshot: video encoding failed")
- self.error = "The video could not be saved. Check free disk space and video support."
+ self.error = (
+ self._audio.error if self._audio is not None and self._audio.error else
+ "The video could not be saved. Check free disk space and video support."
+ )
finally:
+ if self._audio is not None and not self._audio.stopping.is_set():
+ self._audio.stop()
if container is not None:
try:
container.close()
@@ -114,11 +202,45 @@ class VideoEncoder:
log.warning("appshot: partial video cleanup failed", exc_info=True)
self.done.set()
- @staticmethod
- def _encode(av: Any, container: Any, stream: Any, array: Any, pts: int) -> bool:
- frame = av.VideoFrame.from_ndarray(array, format="rgba")
+ def _drain_audio(self, av: Any, container: Any, stream: Any) -> None:
+ while True:
+ try:
+ samples, pts = self._audio.chunks.get_nowait()
+ except queue.Empty: # All currently queued audio has been encoded.
+ return
+ if self.stopping.is_set():
+ remaining = max(0, round(self.duration * 48000) - pts)
+ samples = samples[:, :remaining]
+ if not samples.shape[1]:
+ continue
+ import numpy as np
+
+ frame = av.AudioFrame.from_ndarray(
+ np.ascontiguousarray(samples), format="fltp", layout="stereo"
+ )
+ frame.sample_rate = 48000
+ frame.pts, frame.time_base = pts, Fraction(1, 48000)
+ for packet in stream.encode(frame):
+ container.mux(packet)
+
+ def _source_size(self, image: Any) -> tuple[int, int]:
+ if self.hdr:
+ height, width = image[0].shape
+ return width, height
+ return image.width(), image.height()
+
+ def _encode(self, av: Any, container: Any, stream: Any, array: Any, pts: int) -> bool:
+ if self.hdr:
+ from jarvis.appshot.hdr_recording import p010_frame
+
+ frame = p010_frame(av, stream.pix_fmt, *array)
+ else:
+ frame = av.VideoFrame.from_ndarray(array, format="rgba")
+ frame = frame.reformat(
+ width=self.size[0], height=self.size[1], format=stream.pix_fmt
+ )
frame.pts = pts
- frame.time_base = Fraction(1, FPS)
+ frame.time_base = VIDEO_TIME_BASE
written = False
for packet in stream.encode(frame):
container.mux(packet)
diff --git a/jarvis/audio/ducking/controller.py b/jarvis/audio/ducking/controller.py
index 159a43f91e..433b6ab44a 100644
--- a/jarvis/audio/ducking/controller.py
+++ b/jarvis/audio/ducking/controller.py
@@ -1,8 +1,8 @@
"""AudioDuckController — mutes other apps' audio for the duration of a session.
Subscribes ``VoiceSessionStarted`` (mute others) / ``VoiceSessionEnded``
-(restore). The blocking backend work runs in ``asyncio.to_thread`` with
-``CoInitialize`` so it never touches the event loop. Our own PID is excluded
+(restore). An owned task runs blocking backend work in ``asyncio.to_thread``
+with ``CoInitialize`` without delaying the wake publisher. Our own PID is excluded
from the mute sweep, which automatically protects Jarvis's own TTS voice.
A session never asks the OS for anything: ``mute_others`` is non-interactive.
@@ -14,6 +14,8 @@ from __future__ import annotations
import asyncio
import logging
import os
+import threading
+from collections.abc import Coroutine
from typing import Any
from jarvis.audio.ducking.factory import make_audio_ducker
@@ -21,6 +23,8 @@ from jarvis.core.events import VoiceSessionEnded, VoiceSessionStarted
log = logging.getLogger("jarvis.audio.ducking")
+_SHUTDOWN_LOCK_TIMEOUT_S = 0.25
+
class AudioDuckController:
def __init__(self, bus: Any, cfg: Any, ducker: Any) -> None:
@@ -30,6 +34,12 @@ class AudioDuckController:
self._muted: list[int] = []
self._own_pid = os.getpid()
self._lock = asyncio.Lock()
+ # The worker owns both the native effect and its receipt. Cancelling an
+ # asyncio waiter must not lose a late mute, including during shutdown.
+ self._native_lock = threading.Lock()
+ self._closed = threading.Event()
+ self._tasks: set[asyncio.Task[None]] = set()
+ self._duck_requested = False
# Bumped on every session start. A delayed _on_end restore only runs if
# the generation is unchanged — so a new session that starts during the
# restore-delay window keeps the music muted (its own _on_end restores).
@@ -55,19 +65,17 @@ class AudioDuckController:
# ---- bus handlers ---------------------------------------------------
async def _on_start(self, _ev: Any) -> None:
- if not self._enabled():
+ if not self._enabled() or self._closed.is_set():
return
- async with self._lock:
- self._session_gen += 1 # every start owns the current generation
- if self._muted: # already muted (carry the existing mute over)
- return
- self._muted = await self._run(
- self._ducker.mute_others, own_pid=self._own_pid, never=self._never()
- )
- log.info("ducking: muted %d other session(s)", len(self._muted))
+ self._session_gen += 1
+ self._duck_requested = True
+ self._schedule(self._mute_locked(self._session_gen, self._never()))
async def _on_end(self, _ev: Any) -> None:
- gen = self._session_gen
+ # Keep cleanup owned even when the publisher is cancelled at hangup.
+ await asyncio.shield(self._schedule(self._restore_after_delay(self._session_gen)))
+
+ async def _restore_after_delay(self, gen: int) -> None:
delay = self._restore_delay_s()
if delay > 0:
await asyncio.sleep(delay) # let the TTS tail finish before music returns
@@ -75,7 +83,8 @@ class AudioDuckController:
# A new session started during the delay — leave the music muted; its
# own _on_end will restore. (Avoids un-muting an active session.)
return
- await self._restore_locked()
+ self._duck_requested = False
+ await self._restore_locked(gen)
# ---- public live controls ------------------------------------------
async def set_enabled(self, enabled: bool) -> Any:
@@ -94,7 +103,7 @@ class AudioDuckController:
except Exception: # noqa: BLE001
log.debug("in-memory ducking.enabled update skipped", exc_info=True)
if not enabled:
- await self._restore_locked()
+ await self.restore()
return None
pre = getattr(self._ducker, "prewarm", None)
if callable(pre):
@@ -105,47 +114,80 @@ class AudioDuckController:
async def restore(self) -> None:
"""Force-restore (live path: turning the toggle off)."""
- await self._restore_locked()
+ self._session_gen += 1
+ self._duck_requested = False
+ await asyncio.shield(self._schedule(self._restore_locked(self._session_gen)))
def restore_sync(self) -> None:
"""Synchronous force-restore for shutdown (no event loop available).
- Unmutes the sessions we muted directly on the calling thread (with COM
- init) so a quit/crash mid-session never leaves the user's music muted.
+ Close admission before briefly waiting for the native operation lock.
+ A wedged audio driver must not hold desktop shutdown indefinitely. If
+ the wait expires, the worker restores its own late mute on return,
+ independently of the event loop.
"""
- if not self._muted:
+ self._closed.set()
+ if not self._native_lock.acquire(timeout=_SHUTDOWN_LOCK_TIMEOUT_S):
+ log.warning("ducking: shutdown cleanup is pending on a native audio operation")
return
- pids, self._muted = self._muted, []
- initialized = False
try:
- import comtypes
-
- comtypes.CoInitialize()
- initialized = True
- except Exception: # noqa: BLE001, S110 — non-Windows / already-init
- pass
- try:
- self._ducker.restore(pids)
- log.info("ducking: restored %d session(s) on shutdown", len(pids))
+ self._call(self._restore_receipt)
except Exception: # noqa: BLE001
log.debug("shutdown restore failed", exc_info=True)
finally:
- if initialized:
- try:
- import comtypes
-
- comtypes.CoUninitialize()
- except Exception: # noqa: BLE001, S110 — routine teardown
- pass
+ self._native_lock.release()
# ---- internals ------------------------------------------------------
- async def _restore_locked(self) -> None:
+ def _schedule(self, work: Coroutine[Any, Any, None]) -> asyncio.Task[None]:
+ task = asyncio.create_task(work, name="audio-ducking")
+ self._tasks.add(task)
+ task.add_done_callback(self._task_done)
+ return task
+
+ def _task_done(self, task: asyncio.Task[None]) -> None:
+ self._tasks.discard(task)
+ if not task.cancelled() and (error := task.exception()) is not None:
+ log.error("ducking task failed", exc_info=(type(error), error, error.__traceback__))
+
+ async def _mute_locked(self, gen: int, never: frozenset[str]) -> None:
async with self._lock:
- if not self._muted:
+ await self._run(self._mute_native, gen, never)
+
+ def _mute_native(self, gen: int, never: frozenset[str]) -> None:
+ with self._native_lock:
+ if (
+ self._closed.is_set()
+ or not self._enabled()
+ or not self._duck_requested
+ or self._session_gen != gen
+ or self._muted
+ ):
return
- pids, self._muted = self._muted, []
- await self._run(self._ducker.restore, pids)
- log.info("ducking: restored %d session(s)", len(pids))
+ self._muted = self._ducker.mute_others(own_pid=self._own_pid, never=never)
+ log.info("ducking: muted %d other session(s)", len(self._muted))
+ if self._closed.is_set() or not self._enabled() or not self._duck_requested:
+ self._restore_receipt()
+
+ async def _restore_locked(self, gen: int) -> None:
+ async with self._lock:
+ await self._run(self._restore_native, gen)
+
+ def _restore_native(self, gen: int | None = None) -> None:
+ with self._native_lock:
+ # Recheck here too: a new start can arrive while this worker waits
+ # for an earlier native call, even if the asyncio check was current.
+ if gen is not None and self._session_gen != gen:
+ return
+ self._restore_receipt()
+
+ def _restore_receipt(self) -> None:
+ """Restore under the native lock; retain the receipt if the backend raises."""
+ if not self._muted:
+ return
+ pids = self._muted
+ self._ducker.restore(pids)
+ self._muted = []
+ log.info("ducking: restored %d session(s)", len(pids))
async def _run(self, fn: Any, *args: Any, **kwargs: Any) -> Any:
"""Run a blocking backend call off the loop, with COM init on the worker.
@@ -154,31 +196,31 @@ class AudioDuckController:
bounded ``mute_others`` / ``restore`` calls run under the lock.
"""
- def _call() -> Any:
- initialized = False
+ try:
+ return await asyncio.to_thread(self._call, fn, *args, **kwargs)
+ except Exception: # noqa: BLE001
+ log.exception("ducking COM call failed")
+ return []
+
+ @staticmethod
+ def _call(fn: Any, *args: Any, **kwargs: Any) -> Any:
+ initialized = False
+ if os.name == "nt":
try:
- import comtypes
+ import comtypes # noqa: PLC0415
comtypes.CoInitialize()
initialized = True
- except Exception: # noqa: BLE001, S110 — non-Windows/already-init, routine
- pass
- try:
- return fn(*args, **kwargs)
- finally:
- if initialized:
- try:
- import comtypes
-
- comtypes.CoUninitialize()
- except Exception: # noqa: BLE001, S110 — routine teardown
- pass
-
+ except Exception: # noqa: BLE001 — optional COM bridge or existing apartment
+ log.debug("ducking COM initialization unavailable", exc_info=True)
try:
- return await asyncio.to_thread(_call)
- except Exception: # noqa: BLE001
- log.exception("ducking COM call failed")
- return []
+ return fn(*args, **kwargs)
+ finally:
+ if initialized:
+ try:
+ comtypes.CoUninitialize()
+ except Exception: # noqa: BLE001 — routine teardown
+ log.debug("ducking COM teardown failed", exc_info=True)
def make_audio_duck_controller(bus: Any, cfg: Any) -> AudioDuckController:
diff --git a/jarvis/autostart/command.py b/jarvis/autostart/command.py
index dbcdb3e0ed..fe369e6e31 100644
--- a/jarvis/autostart/command.py
+++ b/jarvis/autostart/command.py
@@ -93,6 +93,11 @@ def resolve_launch_spec(cfg: object | None = None) -> LaunchSpec:
from jarvis.core.background_service import SERVICE_FLAG
service_args = (SERVICE_FLAG,)
+ background = getattr(cfg, "background", None)
+ if getattr(background, "persistent_server", False) and not getattr(
+ background, "server_url", ""
+ ):
+ service_args += ("--persistent-server",)
args: tuple[str, ...] = ("-m", LAUNCHER_MODULE, *service_args)
if sys.platform == "win32":
diff --git a/jarvis/brain/app_control.py b/jarvis/brain/app_control.py
index beb42d47ef..d6498bc457 100644
--- a/jarvis/brain/app_control.py
+++ b/jarvis/brain/app_control.py
@@ -277,6 +277,25 @@ def is_credential_present(spec: ProviderSpec, binary_path: str | None = None) ->
# A key is the usual answer but not the only one — see
# KEYLESS_CREDENTIAL_PROBES (Vertex's Cloud project path).
return _keyless_credential_present(spec.id)
+ if spec.auth_mode == "claude_cli":
+ from jarvis.claude_auth import ClaudeAuthService
+
+ # An unreadable keyring does not disconnect the CLI's native login.
+ try:
+ key = cfg_mod.get_jarvis_agent_secret("claude-api")
+ except Exception as exc: # noqa: BLE001 — subscription credentials live independently
+ log.debug("Claude API key status unavailable (%s)", type(exc).__name__)
+ key = None
+ try:
+ # Match the Claude connection card. An OAuth bearer in the API-key
+ # slot is not an API credential and cannot bypass a signed-out CLI.
+ api_key_present = bool(key and not key.startswith("sk-ant-oat"))
+ return ClaudeAuthService(
+ binary_path, api_key_present=api_key_present
+ ).status().connected
+ except Exception as exc: # noqa: BLE001 — status must survive a missing CLI
+ log.debug("Claude credential status unavailable (%s)", type(exc).__name__)
+ return False
if spec.auth_mode == "codex":
if any(bool(cfg_mod.get_secret(k)) for k in spec.secret_keys):
return True
diff --git a/jarvis/brain/model_catalog.py b/jarvis/brain/model_catalog.py
index 045608d456..03d92a300e 100644
--- a/jarvis/brain/model_catalog.py
+++ b/jarvis/brain/model_catalog.py
@@ -1998,7 +1998,7 @@ class ModelCatalog:
raise ValueError(f"Unsupported provider: {provider}")
ep = _ENDPOINTS[provider]
url = self._resolve_catalog_url(provider, ep)
- key = cfg.get_provider_secret(provider)
+ key = cfg.resolve_provider_endpoint(provider).credential
if not key and ep.auth in ("x-api-key", "bearer", "query"):
raise RuntimeError(f"No API key configured for {provider}.")
auth = ep.auth
@@ -2014,10 +2014,10 @@ class ModelCatalog:
headers = {"Authorization": f"Bearer {key}"}
elif auth == "query":
params = {"key": key or ""}
- elif auth == "none" and ep.secret_slot is not None:
+ elif auth == "none" and (key or ep.secret_slot is not None):
# Keyless local server with an OPTIONAL stored key (e.g. vLLM
# --api-key): attach it when present, stay anonymous otherwise.
- optional = cfg.get_secret(*ep.secret_slot)
+ optional = key or (cfg.get_secret(*ep.secret_slot) if ep.secret_slot else None)
if optional:
headers = {"Authorization": f"Bearer {optional}"}
@@ -2039,12 +2039,15 @@ class ModelCatalog:
resp.raise_for_status()
models = parse_models_response(provider, resp.json())
if provider == "ollama":
- models = await self._enrich_ollama_capabilities(client, url, models)
+ models = await self._enrich_ollama_capabilities(
+ client, url, models, headers=headers,
+ )
return models
@staticmethod
async def _enrich_ollama_capabilities(
- client: httpx.AsyncClient, tags_url: str, models: list[ModelInfo]
+ client: httpx.AsyncClient, tags_url: str, models: list[ModelInfo],
+ *, headers: dict[str, str] | None = None,
) -> list[ModelInfo]:
"""Attach each download's DECLARED capabilities from ``/api/show``.
@@ -2071,7 +2074,9 @@ class ModelCatalog:
async def probe(info: ModelInfo) -> ModelInfo | None:
async with semaphore:
try:
- resp = await client.post(f"{root}/api/show", json={"model": info.id})
+ resp = await client.post(
+ f"{root}/api/show", json={"model": info.id}, headers=headers,
+ )
resp.raise_for_status()
shown = resp.json()
caps = shown.get("capabilities")
diff --git a/jarvis/brain/ollama_library.py b/jarvis/brain/ollama_library.py
index f6080dd7c0..70fe278826 100644
--- a/jarvis/brain/ollama_library.py
+++ b/jarvis/brain/ollama_library.py
@@ -224,9 +224,10 @@ def parse_search_html(page: str) -> list[dict[str, Any]]:
description = text
break
- # Since 2026-10 a badge puts an icon before its label ("<svg>…</svg>Vision")
- # and capitalises it; dropping the icons reads both layouts as text spans.
- plain = re.sub(r"<svg\b.*?</svg>", "", block, flags=re.DOTALL)
+ # The current catalog puts an SVG before title-cased badge labels.
+ # Ignore icon markup while retaining the actual visible label, and
+ # keep accepting the older text-only lowercase badges.
+ plain = re.sub(r"<svg\b[^>]*>.*?</svg>", "", block, flags=re.DOTALL | re.IGNORECASE)
spans = [_text(m.group(1)) for m in re.finditer(r"<span\b[^>]*>([^<]*)</span>", plain)]
spans = [s for s in spans if s]
# Every text node, not only whole spans: "Cloud" now sits beside a tooltip.
@@ -241,7 +242,10 @@ def parse_search_html(page: str) -> list[dict[str, Any]]:
r'<span\b[^>]*title="[^"]*downloads"[^>]*>\s*<span\b[^>]*>\s*([\d.,]+[KMB]?)\s*</span>',
plain,
)
- # The current search page prints no update time any more; "" then.
+ if pulls_match is None:
+ pulls_match = re.search(
+ r'''\btitle=["']([\d.,]+[KMB]?)\s+downloads["']''', block, re.IGNORECASE
+ )
updated = next((s for s in spans if s.endswith(" ago") or s == "yesterday"), "")
entries.append(
diff --git a/jarvis/brain/ollama_profiles.py b/jarvis/brain/ollama_profiles.py
index ae64d40901..ed2f908e99 100644
--- a/jarvis/brain/ollama_profiles.py
+++ b/jarvis/brain/ollama_profiles.py
@@ -10,7 +10,7 @@ the native ``/api/chat``, the per-model knobs travel three ways:
the same trick the local-realtime supervisor uses for its ``-voice-8k``
alias. The alias shares the weights (metadata only, no download), and its
name carries a hash of the baked set so a changed knob yields a new alias
- and the stale one is deleted. Aliases end in ``-jarvis-<8 hex>`` so the
+ without deleting another caller's profile. Aliases end in ``-jarvis-<8 hex>`` so the
inventory can hide them from the user.
* ``keep_alive`` rides a warm ping (``POST /api/generate`` with an empty
prompt) once per process per (model, keep_alive).
@@ -40,7 +40,7 @@ log = logging.getLogger(__name__)
#: Option keys that are baked into the derived model's ``parameters``.
#: ``num_predict``, ``temperature``, ``keep_alive`` and ``think`` are NOT
#: here on purpose: they have a per-request channel, and baking them would
-#: force a new alias (and a stale-alias delete) for a knob that costs nothing
+#: force a new alias for a knob that costs nothing
#: to send.
BAKEABLE_KEYS: tuple[str, ...] = (
"num_ctx",
@@ -71,9 +71,8 @@ _OVERHEAD_GB = 1.0
#: the same rule ``ollama_pull.fit_verdict`` applies.
_RAM_SHARE = 0.6
-# Per-process memo so a turn never repeats an HTTP round-trip it has already
-# won: (root, alias) once ensured, (root, model, keep_alive) once warmed.
-_ensured: set[tuple[str, str]] = set()
+# Warm pings are memoized; profile existence is checked each time because a
+# server restart or an explicit model removal can invalidate a previous create.
_warmed: set[tuple[str, str, str]] = set()
# Creating a derived model or loading weights for the warm ping can take a
@@ -117,18 +116,6 @@ def is_profile_alias(name: str) -> bool:
return bool(_PROFILE_ALIAS_RE.match((name or "").strip().removesuffix(":latest")))
-def _stale_aliases(names: list[str], base: str, keep: str) -> list[str]:
- """Earlier ``<base>-jarvis-*`` aliases that are not ``keep``."""
- prefix = _fold(base)
- stale: list[str] = []
- for raw in names:
- name = raw.strip().removesuffix(":latest")
- match = _PROFILE_ALIAS_RE.match(name)
- if match and match.group("prefix") == prefix and name != keep:
- stale.append(name)
- return stale
-
-
async def _tag_names(client: httpx.AsyncClient, root: str) -> list[str]:
resp = await client.get(f"{root}/api/tags")
resp.raise_for_status()
@@ -141,14 +128,15 @@ async def ensure_profile(
opts: OllamaModelOptions,
*,
transport: httpx.AsyncBaseTransport | None = None,
+ headers: dict[str, str] | None = None,
) -> str:
"""Return the alias that carries ``opts`` for ``base``, creating it once.
Idempotent against ``/api/tags``: an alias that already exists is reused
without a create call; a base with NO bakeable knob returns ``base``
- itself. When the hash changed, every earlier ``<base>-jarvis-*`` alias is
- deleted (a failed delete is logged and does not fail the turn — the alias
- is a ghost in the list, not a wrong answer). Raises ``RuntimeError`` with
+ itself. Other option sets remain available for concurrent agents and
+ voice sessions. Check existence on each use to recover deleted aliases.
+ Raises ``RuntimeError`` with
an English sentence when the create itself fails; the caller then runs the
base model and says so.
@@ -158,10 +146,7 @@ async def ensure_profile(
if not has_bakeable(opts):
return base
alias = profile_name(base, opts)
- key = (root, alias)
- if key in _ensured:
- return alias
- async with httpx.AsyncClient(timeout=_TIMEOUT, transport=transport) as client:
+ async with httpx.AsyncClient(timeout=_TIMEOUT, transport=transport, headers=headers) as client:
try:
names = await _tag_names(client, root)
except Exception as exc:
@@ -191,19 +176,6 @@ async def ensure_profile(
base,
json.dumps(baked_parameters(opts), sort_keys=True),
)
- for stale in _stale_aliases(names, base, alias):
- try:
- resp = await client.request("DELETE", f"{root}/api/delete", json={"model": stale})
- resp.raise_for_status()
- log.info("ollama profile: deleted stale alias %s", stale)
- except Exception as exc: # noqa: BLE001 — a leftover alias is cosmetic
- log.warning(
- "ollama profile: could not delete stale alias %s (%s: %s)",
- stale,
- type(exc).__name__,
- exc,
- )
- _ensured.add(key)
return alias
@@ -213,6 +185,7 @@ async def warm(
keep_alive: str | int,
*,
transport: httpx.AsyncBaseTransport | None = None,
+ headers: dict[str, str] | None = None,
) -> bool:
"""Load ``model`` with ``keep_alive`` via an empty ``/api/generate`` ping.
@@ -226,7 +199,9 @@ async def warm(
return True
payload = {"model": model, "keep_alive": keep_alive, "stream": False}
try:
- async with httpx.AsyncClient(timeout=_TIMEOUT, transport=transport) as client:
+ async with httpx.AsyncClient(
+ timeout=_TIMEOUT, transport=transport, headers=headers,
+ ) as client:
resp = await client.post(f"{root}/api/generate", json=payload)
resp.raise_for_status()
except Exception as exc: # noqa: BLE001 — the warm ping is best-effort
@@ -244,8 +219,7 @@ async def warm(
def reset_process_memo() -> None:
- """Forget what was ensured/warmed (tests, or after a server restart)."""
- _ensured.clear()
+ """Forget warm pings (tests, or after a server restart)."""
_warmed.clear()
diff --git a/jarvis/brain/workspace_tool.py b/jarvis/brain/workspace_tool.py
index c3e702053f..bccd765826 100644
--- a/jarvis/brain/workspace_tool.py
+++ b/jarvis/brain/workspace_tool.py
@@ -65,8 +65,12 @@ class WorkspaceOrchestrationTool:
"terminal_id): observe reads its screen, question and newest events; respond types "
"the answer to the question or permission prompt it shows (prompt = the answer); keys "
"presses keys such as enter, escape, up/down, digits, shift+tab (permission mode); "
- "interrupt stops its current turn; close removes it. These are the user's own app "
- "actions: run them when asked, without asking back. "
+ "interrupt stops its current turn; restart starts a stopped or failed agent again "
+ "under the same terminal_id on its own conversation (a running one is left alone, so "
+ "a repeat is harmless); close removes it. These are the user's own app "
+ "actions: run them when asked, without asking back. A send refused because the agent "
+ "is not running (stopped: true) types nothing: when the user wants that agent to do "
+ "the task, restart it and resend with the same request_id and prompt. "
"Inspect the current graph (pass workspace to scope it): each session includes "
"availability, history, last user/assistant messages, recent tools and screen excerpts. "
"Empty is unused; idle has an existing conversation; stopped/failed/unknown is not free. "
@@ -122,6 +126,7 @@ class WorkspaceOrchestrationTool:
"respond",
"keys",
"interrupt",
+ "restart",
"close",
],
},
diff --git a/jarvis/core/background_service.py b/jarvis/core/background_service.py
index 32e2ef06fe..e68ded56b6 100644
--- a/jarvis/core/background_service.py
+++ b/jarvis/core/background_service.py
@@ -175,10 +175,11 @@ def _write_json(path: Path, payload: dict[str, Any]) -> None:
os.replace(tmp, path)
-def write_marker(port: int) -> None:
+def write_marker(port: int, *, persistent: bool = False) -> None:
_write_json(
marker_path(),
- {"pid": os.getpid(), "port": int(port), "started_at": time.time()},
+ {"pid": os.getpid(), "port": int(port), "started_at": time.time(),
+ "persistent": persistent},
)
@@ -210,7 +211,7 @@ def _is_service_process(pid: int) -> bool:
cmdline = psutil.Process(int(pid)).cmdline()
except Exception: # noqa: BLE001 — gone or unreadable: not our service
return False
- return SERVICE_FLAG in cmdline
+ return SERVICE_FLAG in cmdline or "--persistent-server" in cmdline
def service_pid(*, is_service: Callable[[int], bool] = _is_service_process) -> int | None:
@@ -270,6 +271,8 @@ def _windowless_python(executable: str) -> str:
def service_command(
*,
after_pid: int | None,
+ persistent: bool = False,
+ port: int | None = None,
executable: str | None = None,
frozen: bool | None = None,
) -> list[str]:
@@ -287,6 +290,10 @@ def service_command(
frozen = is_frozen()
tail = [*current_instance().launcher_args]
+ if persistent:
+ tail.append("--persistent-server")
+ if port is not None:
+ tail += ["--port", str(int(port))]
if after_pid is not None:
tail += [AFTER_PID_FLAG, str(int(after_pid))]
if frozen:
@@ -349,7 +356,9 @@ def _spawn_windows_outside_job(argv: list[str], cwd: str, env: dict[str, str]) -
return True
-def spawn_service(*, after_pid: int | None) -> bool:
+def spawn_service(
+ *, after_pid: int | None, persistent: bool = False, port: int | None = None,
+) -> bool:
"""Start the service detached from this process. Never raises."""
try:
from jarvis.core.instance import current_instance
@@ -358,7 +367,7 @@ def spawn_service(*, after_pid: int | None) -> bool:
cwd = restart_workdir(_source_root())
env = current_instance().environ(fresh_user_env())
env["PYTHONIOENCODING"] = "utf-8"
- argv = service_command(after_pid=after_pid)
+ argv = service_command(after_pid=after_pid, persistent=persistent, port=port)
if sys.platform == "win32":
return _spawn_windows_outside_job(argv, cwd, env)
spawn_detached(argv, cwd=cwd, env=env)
@@ -376,19 +385,24 @@ def hand_off_on_quit(state: Any, cfg: Any, *, pid: int | None = None) -> bool:
try:
from jarvis.core.instance import current_instance
- if not current_instance().owns_ambient_duties:
+ section = getattr(cfg, "background", None)
+ persistent = bool(getattr(section, "persistent_server", False)) and not bool(
+ getattr(section, "server_url", "")
+ )
+ if not current_instance().owns_ambient_duties and not persistent:
# A dev instance carries no channels and must not linger unseen.
return False
- if not keep_running_enabled(cfg):
+ if not persistent and not keep_running_enabled(cfg):
logger.info("background: keep_agents_running is off — everything stops with the app")
return False
work = work_from_state(state)
- if not work:
+ if not persistent and not work:
logger.info("background: no routines or channels to keep — no background service")
return False
# A leftover request would make the new service exit at once.
clear_handover_request()
- started = spawn_service(after_pid=os.getpid() if pid is None else pid)
+ options = {"persistent": True} if persistent else {}
+ started = spawn_service(after_pid=os.getpid() if pid is None else pid, **options)
logger.info(
"background: {} the background service to keep {}",
"started" if started else "could NOT start",
diff --git a/jarvis/core/config.py b/jarvis/core/config.py
index a300c89cc4..411f780a85 100644
--- a/jarvis/core/config.py
+++ b/jarvis/core/config.py
@@ -1934,6 +1934,9 @@ class SafetyBlacklistConfig(BaseModel):
class SafetyConfig(BaseModel):
+ # App-owned executions run without permission prompts unless the person
+ # selects Ask. Chat sessions keep their own explicitly selected stance.
+ approval_mode: Literal["bypass", "ask"] = "bypass"
default_tier: RiskTier = "safe"
always_confirm_tiers: list[RiskTier] = Field(default_factory=lambda: ["ask"])
always_block_tiers: list[RiskTier] = Field(default_factory=lambda: ["block"])
@@ -2382,6 +2385,19 @@ class BackgroundConfig(BaseModel):
model_config = ConfigDict(extra="allow")
keep_agents_running: bool = True
+ # Independent server ownership is opt-in for existing integrated desktops.
+ # A client never stops this server or inherits its provider credentials.
+ persistent_server: bool = False
+ server_url: str = ""
+
+ @field_validator("server_url")
+ @classmethod
+ def validate_server_url(cls, value: str) -> str:
+ if not value.strip():
+ return ""
+ from jarvis.core.server_endpoint import server_endpoint
+
+ return server_endpoint(value)
class TelemetryConfig(BaseModel):
@@ -2706,6 +2722,12 @@ class AppshotConfig(BaseModel):
#: One shortcut selects a recording area; pressing it again stops and saves.
recording_hotkey: str = "ctrl+shift+9"
+ #: Recording preferences are snapshotted when selection starts.
+ recording_resolution: Literal["720p", "1080p", "1440p", "2160p", "native"] = "1080p"
+ recording_fps: Literal[30, 60, 120] = 60
+ recording_bitrate_mbps: int = Field(default=12, ge=1, le=100, strict=True)
+ recording_system_audio: bool = False
+
#: Where a shortcut appshot goes. ``auto``: into the running voice call,
#: otherwise onto the next message. ``message``: always onto the next
#: message. ``voice``: only into a running voice call.
diff --git a/jarvis/core/config_writer.py b/jarvis/core/config_writer.py
index 5a3934eda8..8ae8e23982 100644
--- a/jarvis/core/config_writer.py
+++ b/jarvis/core/config_writer.py
@@ -1001,8 +1001,9 @@ def set_screen_context_settings(
#: The master switch is ``[screen_context].enabled``, written by the setter above.
APPSHOT_SETTING_KEYS: frozenset[str] = frozenset(
{"keep_newest",
- "hotkey", "region_hotkey", "recording_hotkey", "target", "sound", "effect", "card_seconds", "library",
- "copy_to_clipboard"}
+ "hotkey", "region_hotkey", "recording_hotkey", "target", "sound", "effect", "card_seconds",
+ "library", "copy_to_clipboard", "recording_resolution", "recording_fps", "recording_bitrate_mbps",
+ "recording_system_audio"}
)
@@ -1255,6 +1256,19 @@ def set_background_keep_running(enabled: bool, *, path: Path = DEFAULT_CONFIG_FI
_patch_table(path, "background", "keep_agents_running", bool(enabled))
+def set_agent_server(
+ *, persistent: bool, server_url: str = "", path: Path = DEFAULT_CONFIG_FILE,
+) -> None:
+ """Select independent server ownership for the next desktop launch."""
+ from jarvis.core.server_endpoint import server_endpoint
+
+ origin = server_endpoint(server_url) if server_url.strip() else ""
+ _patch_table(
+ path, "background", "persistent_server", bool(persistent),
+ extra={"server_url": origin},
+ )
+
+
def set_wiki_vault_root(vault_root: str, *, path: Path = DEFAULT_CONFIG_FILE) -> None:
"""Persist ``[wiki_integration] vault_root`` in jarvis.toml (AP-7).
diff --git a/jarvis/core/events.py b/jarvis/core/events.py
index 18bb62821d..ef9ff0147f 100644
--- a/jarvis/core/events.py
+++ b/jarvis/core/events.py
@@ -2074,6 +2074,7 @@ class BrowserVoiceRequested(Event):
"""A desktop trigger hands a voice call to the browser media owner."""
action: Literal["start", "stop"] = "start"
+ request_id: str = ""
@dataclass(frozen=True, slots=True)
diff --git a/jarvis/core/response_style.py b/jarvis/core/response_style.py
index de61bb6318..70b9ba82da 100644
--- a/jarvis/core/response_style.py
+++ b/jarvis/core/response_style.py
@@ -34,6 +34,37 @@ AGENT_QUESTION_GUIDANCE = (
"provide that requested output without executing an unrequested change."
)
+#: Society agents: the one way a secret reaches an agent (jarvis/society/credential_tool.py).
+#: Fresh briefings carry it in full; resumed vendor conversations get it every turn too,
+#: because they never see the briefing again.
+CREDENTIAL_GUIDANCE = (
+ "Credentials are mandatory tool work. Whenever a task needs a secret only the user can "
+ "supply (an API key, bot or access token, personal access token, password, client "
+ "secret, webhook URL with a secret, database connection string, SSH or private key), "
+ "call society_request_credential as soon as you know it is needed, before the work that "
+ "depends on it. It is the only way to get a secret: the chat shows the user a secure "
+ "field, the value goes to the system credential store and is set as the environment "
+ "variable you name, in every society_shell command. This applies just as much when you "
+ "set up an app, bot, script, integration or deployment for the user (a Discord or "
+ "Telegram bot, an app that calls the OpenAI or GitHub API, a cloud account): request "
+ "each credential it needs with the tool, then build it so it reads the variable "
+ "(os.environ['DISCORD_BOT_TOKEN'], process.env.DISCORD_BOT_TOKEN) and run it with "
+ "society_shell, which passes the variable on. Something that must keep running (a bot, "
+ "a server) is started detached from society_shell with its output in a log file "
+ "(PowerShell: Start-Process -WindowStyle Hidden -RedirectStandardOutput app.log ...; "
+ "bash: nohup ... > app.log 2>&1 &); then read the log to verify it is up. "
+ "One call per credential: a conventional "
+ "UPPER_SNAKE_CASE env name, a short label, and a description of what it is for and where "
+ "the user creates it with minimal permissions. If the service rejects a stored value, "
+ "call again with replace=true. Never ask for a secret in your reply or in a "
+ "society_ask_user question, never tell the user to paste it into the chat, a settings "
+ "page, a plugin dialog, a .env file or 'a secure field' yourself, never open a browser "
+ "or computer control to find a place for it, and never print, log, commit or write the "
+ "value anywhere. A service Jarvis already connects through its own sign-in (a connected "
+ "plugin in your capabilities) uses that connection instead. Only the credentials listed "
+ "as stored for this turn exist; never claim one is stored otherwise."
+)
+
KEEP_GOING_ON_TOOL_FAILURE = (
"A technical tool failure is not the end of the task. Read the result, inspect the "
"current tools and try a meaningfully different supported path when permitted. "
@@ -49,16 +80,19 @@ KEEP_GOING_ON_TOOL_FAILURE = (
)
CONVERSATIONAL_TURN_REMINDER = (
- "Keep replies short, direct and natural by default, like an everyday chat. "
- "For a routine reply, aim for one to three short sentences in one paragraph; "
- "a simple confirmation may need only one sentence. Lead with the answer or "
- "observed outcome. Expand only when the user asks for detail or the task needs "
- "it for a complete, useful answer; this is a default, not a hard length limit. "
- "Skip report headings, repeated summaries, filler and obligatory closing offers. "
- "Use a short list or headings only when requested or when they make necessary "
- "detail easier to follow. Keep important results, blockers, uncertainty and "
- "essential questions clear, even when that takes more space. Follow explicit "
- "user preferences and the turn's selected language."
+ "Write like a capable person texting in a messenger: short, warm, plain. "
+ "Open with the state in the first words (done, running, fixed, not yet, "
+ "blocked) or the direct answer; causes and details come after it. No closing "
+ "offers such as 'let me know if…'. Most replies are one or two short paragraphs "
+ "of one to three sentences each; a simple confirmation is one sentence. "
+ "Expand only when the user asks for detail or the task needs it for a complete, "
+ "useful answer; this is a default, not a hard length limit. Write so someone "
+ "who is not technical understands it on the first read: everyday words, no "
+ "code names, ids or tool names unless the person has to type or click them. "
+ "No headings, bold labels or bullet walls in a chat reply; use a short list "
+ "only for three or more parallel items. Keep important results, blockers, "
+ "uncertainty and essential questions clear, even when that takes more space. "
+ "Follow explicit user preferences and the turn's selected language."
)
# The same default reaches fresh prompts and resumed vendor sessions. Brevity
@@ -66,25 +100,45 @@ CONVERSATIONAL_TURN_REMINDER = (
# requested code, detailed answers and failure evidence must remain complete.
CONVERSATIONAL_RESPONSE_STYLE = (
CONVERSATIONAL_TURN_REMINDER
- + "\n\nSound like a capable, approachable colleague. Use ordinary words and complete "
- "sentences, not slogans, clipped status fragments or a formal report template. "
+ + "\n\nThe person must always know three things: what is happening now, what "
+ "is done, and what they can do next. The chat shows them no reasoning and no tool "
+ "log, only your messages, so shape the messages of a turn that way. "
+ "Whenever a turn needs tools, your first message comes before the first tool call: "
+ "a short yes plus what you are doing now, in one or two sentences (\"On it, I'm "
+ "setting up a daily check that announces new contributors.\", \"Checking now "
+ "whether your message arrived.\"). A progress update says in one "
+ "sentence what just finished and what comes next (\"Three automations are running; "
+ "I'm adding one more for new contributors.\"); it only appears when something "
+ "changed and work remains. Two messages in a row are fine. "
+ "The closing reply leads with the outcome, then says in concrete terms "
+ "what changes for the person (real numbers, times and names: \"every 30 minutes\", "
+ "\"tomorrow at 10:04\", \"all 17 channels\"), then names the one next step when "
+ "there is one for them (test it, send the code, click a link). Never invent a "
+ "next step or end every reply with a question.\n\n"
+ "Explain causes as plain cause and effect (\"two runs answered the same message "
+ "at once\"), never as component names or protocol terms. For a problem you fixed, "
+ "the first sentence says it is fixed, the second gives the cause, and the fix is "
+ "described by what it guarantees the person, not by how it works inside. "
+ "When the person must do something, give the exact click path "
+ "(\"Settings → Bot → Token\") instead of the theory behind it. "
+ "One step at a time: when the person has several things to do, "
+ "give only the next one or two and continue once those are done, instead of a "
+ "full manual up front. If the person did not understand you, say sorry once, then "
+ "give simpler, concrete steps. State a limit or a refusal calmly in one sentence "
+ "with its reason. A secret goes into the secure field, never into the chat; say "
+ "so when you ask for one. Mention a relevant side observation in one sentence. "
"Do not repeat the user's request or restate a fact in an introduction, a list "
- "and a conclusion. Finish when the useful information has been delivered. "
- "Interpret tool results in plain language. Do not paste raw JSON envelopes, token "
- "counts, internal provider names or empty result fields into the answer unless the "
- "person explicitly asks for raw data or debugging details. Preserve requested code, "
- "data and deliverables. Put a useful result link next to the outcome. "
- "Distinguish intended, attempted, pending and verified work: a successful tool call "
- "does not by itself prove the user's goal was achieved. State partial success or "
- "failure plainly, with the relevant reason and at most one next action. "
- "Progress updates should be brief and add meaningful new information. Do not "
- "narrate every click, repeat that you are checking, or split one thought into many "
- "messages. Never promise ongoing monitoring without a confirmed active schedule. "
- "Ask only for essential missing information that changes the next action; "
- "do not end every reply with a question or invent a next step. "
- "For a correction, acknowledge the specific change once and apply it; do not "
- "restart the explanation or defend the old answer. If evidence contradicts the "
- "person's assumption, explain that calmly rather than agreeing automatically. "
- "Keep implementation vocabulary in the tool details unless it explains a real "
- "limitation. Say what works, what is missing, and what the person needs to do."
+ "and a conclusion. Interpret tool results in plain language. Do not paste raw JSON "
+ "envelopes, token counts, internal provider names or empty result fields into the "
+ "answer unless the person explicitly asks for raw data or debugging details. "
+ "Preserve requested code, data and deliverables. Put a useful result link next to "
+ "the outcome. Distinguish intended, attempted, pending and verified work: a "
+ "successful tool call does not by itself prove the user's goal was achieved. State "
+ "partial success or failure plainly, with the relevant reason and at most one next "
+ "action. Do not narrate every click, repeat that you are checking, or split one "
+ "thought into many messages. Never promise ongoing monitoring without a confirmed "
+ "active schedule. Ask only for essential missing information that changes the next "
+ "action. For a correction, acknowledge the specific change once and apply it; do "
+ "not restart the explanation or defend the old answer. If evidence contradicts the "
+ "person's assumption, explain that calmly rather than agreeing automatically."
)
diff --git a/jarvis/core/server_endpoint.py b/jarvis/core/server_endpoint.py
new file mode 100644
index 0000000000..2c6b451c69
--- /dev/null
+++ b/jarvis/core/server_endpoint.py
@@ -0,0 +1,36 @@
+"""Validate a client connection without putting credentials into its URL."""
+
+from __future__ import annotations
+
+import ipaddress
+from urllib.parse import urlsplit, urlunsplit
+
+
+def server_endpoint(value: str) -> str:
+ """Accept TLS servers or local HTTP, with no embedded login or query."""
+ value = value.strip()
+ if any(char.isspace() for char in value):
+ raise ValueError("The server address must not contain whitespace.")
+ parsed = urlsplit(value)
+ if (
+ parsed.scheme not in {"http", "https"}
+ or not parsed.hostname
+ or parsed.username is not None
+ or parsed.password is not None
+ or parsed.query
+ or parsed.fragment
+ or parsed.path not in {"", "/"}
+ ):
+ raise ValueError("Use a server origin such as https://jarvis.example.com without a key.")
+ try:
+ port = parsed.port
+ local = parsed.hostname == "localhost" or ipaddress.ip_address(parsed.hostname).is_loopback
+ except ValueError:
+ # A DNS host is valid; an invalid port still needs to fail validation.
+ port = parsed.port
+ local = parsed.hostname == "localhost"
+ if parsed.scheme == "http" and not local:
+ raise ValueError("Remote servers require HTTPS. An SSH tunnel may use local HTTP.")
+ if port is not None and port < 1:
+ raise ValueError("The server port must be between 1 and 65535.")
+ return urlunsplit((parsed.scheme, parsed.netloc, "", "", ""))
diff --git a/jarvis/cu/indicator/__main__.py b/jarvis/cu/indicator/__main__.py
index f2dcc7bf7b..ccacc16c8d 100644
--- a/jarvis/cu/indicator/__main__.py
+++ b/jarvis/cu/indicator/__main__.py
@@ -8,7 +8,9 @@ expected degradation.
from __future__ import annotations
+import os
import sys
+from contextlib import suppress
from jarvis.cu.indicator.protocol import EXIT_NO_GUI
@@ -26,5 +28,19 @@ def main() -> int:
return run()
+def _exit(code: int) -> None:
+ """End without interpreter finalization once the sidecar has cleaned up.
+
+ ``run()`` restores the pointer and closes every card before it returns.
+ Finalization would then free Qt objects and the stdin reader thread in an
+ arbitrary order after the QApplication, which a slow machine turns into
+ an access violation (0xC0000005) on an otherwise clean exit.
+ """
+ for stream in (sys.stdout, sys.stderr):
+ with suppress(Exception): # A gone parent closed the pipe; nothing to flush.
+ stream.flush()
+ os._exit(code)
+
+
if __name__ == "__main__":
- raise SystemExit(main())
+ _exit(main())
diff --git a/jarvis/cu/indicator/controller.py b/jarvis/cu/indicator/controller.py
index 501377b4f2..0dc768313a 100644
--- a/jarvis/cu/indicator/controller.py
+++ b/jarvis/cu/indicator/controller.py
@@ -133,6 +133,7 @@ class CUIndicatorController:
self._idle_quit_task: asyncio.Task | None = None
# At least one appshot card in the corner stack is up.
self._card_open = False
+ self._appshot_warm = False
self._loop: asyncio.AbstractEventLoop | None = None
# The finished picture for the current card. It can be ready before the
# effect reached the sidecar, so ``snap`` re-sends it after the card.
@@ -266,7 +267,7 @@ class CUIndicatorController:
protocol.CMD_HIDE,
_SHOW_ACK_TIMEOUT_S,
)
- if self._card_open or time.monotonic() < self._snap_until:
+ if self._appshot_warm or self._card_open or time.monotonic() < self._snap_until:
# An appshot effect is still on screen; quit once it has played.
self._schedule_idle_quit()
return
@@ -287,6 +288,42 @@ class CUIndicatorController:
await asyncio.to_thread(self._reap, proc)
# ---------------------------------------------------------- appshot snap
+ async def warm_for_appshots(self, enabled: bool) -> None:
+ """Retain an idle GUI runtime without showing a border or taking pixels."""
+ self._loop = asyncio.get_running_loop()
+ async with self._lock:
+ self._appshot_warm = enabled and self._border_capability()[0]
+ if not self._appshot_warm:
+ self._schedule_idle_quit()
+ return
+ if self._proc is not None and self._proc.poll() is None:
+ return # Never hide an existing mission border or Appshot card.
+ spawn = asyncio.create_task(asyncio.to_thread(self._spawn_sidecar))
+ try:
+ await asyncio.shield(spawn)
+ except asyncio.CancelledError:
+ await spawn # Acquire the late Popen before the owner tears down.
+ raise
+ if self._proc is not None:
+ ready = await asyncio.to_thread(
+ self._send_and_wait, protocol.CMD_HIDE, _SHOW_ACK_TIMEOUT_S
+ )
+ if not ready:
+ log.warning("appshot: shutter runtime did not acknowledge standby")
+ await self._quit_sidecar()
+
+ async def close(self) -> None:
+ """Stop owned tasks and reap the helper when the application exits."""
+ self._appshot_warm = False
+ task, self._idle_quit_task = self._idle_quit_task, None
+ if task is not None:
+ task.cancel()
+ with suppress(asyncio.CancelledError):
+ await task
+ self._disarm_escape()
+ async with self._lock:
+ await self._quit_sidecar()
+
def hold_for_snap(self) -> None:
"""Keep the sidecar alive for an effect that is about to be sent.
@@ -404,6 +441,32 @@ class CUIndicatorController:
self._schedule_idle_quit()
return shown
+ async def show_recording(
+ self, *, recording_id: str, video_path: str, thumb_b64: str,
+ monitor: list[int], rect: list[float], screen_name: str,
+ duration_s: float, rest_ms: int, labels: dict[str, str],
+ ) -> bool:
+ """Fly a finalized video into its bottom-right card, without changing screenshot state."""
+ ok, reason = self._border_capability()
+ if not ok:
+ log.debug("[appshot-effect] recording preview unavailable: %s", reason)
+ return False
+ self._loop = asyncio.get_running_loop()
+ async with self._lock:
+ self._snap_until = max(self._snap_until, time.monotonic() + _SNAP_LIFETIME_S)
+ await asyncio.to_thread(self._spawn_sidecar)
+ if self._proc is None:
+ return False
+ capture_guard.register_hook(self._suppress_for_grab)
+ shown = await asyncio.to_thread(
+ self._send_and_wait, protocol.CMD_RECORDING, _SHOW_ACK_TIMEOUT_S,
+ id=f"recording:{recording_id}", video_path=video_path, thumb=thumb_b64,
+ monitor=list(monitor), rect=list(rect), screen_name=screen_name,
+ duration_s=duration_s, rest_ms=rest_ms, labels=labels,
+ )
+ self._schedule_idle_quit()
+ return shown
+
async def snap_image(self, image_b64: str, shot_id: str = "") -> bool:
"""Hand the finished picture to the newest card, for a drag out.
@@ -434,6 +497,13 @@ class CUIndicatorController:
def _handle_sidecar_event(self, payload: dict[str, Any]) -> None:
event = payload.get("event")
+ if event in (protocol.EVENT_RECORDING_OPEN, protocol.EVENT_RECORDING_SAVE):
+ action = "open" if event == protocol.EVENT_RECORDING_OPEN else "save"
+ asyncio.get_running_loop().create_task(
+ self._recording_action(action, str(payload.get("id", ""))),
+ name="appshot-recording-card",
+ )
+ return
if event == protocol.EVENT_CARD:
self._card_open = bool(payload.get("open"))
if not self._card_open:
@@ -454,7 +524,6 @@ class CUIndicatorController:
async def _card_action(self, action: str, shot_id: str = "") -> None:
"""Copy, save or copy text from a card, then tell that card how it went."""
from jarvis.appshot.card_actions import run_card_action # noqa: PLC0415
-
status = await run_card_action(action, shot_id)
if self._proc is None or self._proc.poll() is not None:
return
@@ -497,6 +566,19 @@ class CUIndicatorController:
except Exception: # noqa: BLE001 - a lost click must not break the sidecar
log.warning("[appshot-effect] opening the editor failed", exc_info=True)
+ async def _recording_action(self, action: str, card_id: str) -> None:
+ from jarvis.appshot.recording_cards import RECORDING_PREFIX, run_recording_card_action
+
+ try:
+ status = await run_recording_card_action(action, card_id.removeprefix(RECORDING_PREFIX))
+ if status and self._proc is not None and self._proc.poll() is None:
+ await asyncio.to_thread(
+ self._send_and_wait, protocol.CMD_RECORDING_STATUS, _SHOW_ACK_TIMEOUT_S,
+ text=status, id=card_id,
+ )
+ except Exception:
+ log.exception("[appshot-effect] recording card action failed")
+
def _schedule_idle_quit(self) -> None:
task = self._idle_quit_task
if task is not None and not task.done():
@@ -515,7 +597,8 @@ class CUIndicatorController:
await asyncio.sleep(remaining + 0.05)
async with self._lock:
if (
- self._active
+ self._appshot_warm
+ or self._active
or self._screen_active
or self._card_open
or time.monotonic() < self._snap_until
diff --git a/jarvis/cu/indicator/protocol.py b/jarvis/cu/indicator/protocol.py
index e2485f01d5..27c00012a9 100644
--- a/jarvis/cu/indicator/protocol.py
+++ b/jarvis/cu/indicator/protocol.py
@@ -54,6 +54,8 @@ CMD_UNBLANK = "unblank"
CMD_QUIT = "quit"
CMD_SNAP = "snap"
CMD_SNAP_IMAGE = "snap_image"
+CMD_RECORDING = "recording"
+CMD_RECORDING_STATUS = "recording_status"
#: Put a picture straight into the corner card (the editor closed).
CMD_CARD = "card"
#: A short status line on the card ("Copied", "Saved to Downloads").
@@ -70,6 +72,8 @@ ALL_COMMANDS = frozenset(
CMD_QUIT,
CMD_SNAP,
CMD_SNAP_IMAGE,
+ CMD_RECORDING,
+ CMD_RECORDING_STATUS,
CMD_CARD,
CMD_CARD_STATUS,
CMD_POINTER_PRESS,
@@ -85,7 +89,11 @@ CARD_ACTIONS = frozenset({"copy", "save", "copy_text"})
#: How many cards the corner stack holds — as many appshots as the store keeps
#: (``jarvis.appshot.store.MAX_RECENT``), so every card's buttons still work.
MAX_CARDS = 5
-ALL_EVENTS = frozenset({EVENT_CARD, EVENT_SNAP_OPEN, EVENT_CARD_ACTION})
+EVENT_RECORDING_OPEN = "recording_open"
+EVENT_RECORDING_SAVE = "recording_save"
+ALL_EVENTS = frozenset({
+ EVENT_CARD, EVENT_SNAP_OPEN, EVENT_CARD_ACTION, EVENT_RECORDING_OPEN, EVENT_RECORDING_SAVE,
+})
#: Sidecar exit code when no usable GUI stack exists (PySide6 missing or
#: no display). The controller logs it as an expected degradation.
@@ -156,6 +164,10 @@ __all__ = [
"CMD_SHOW",
"CMD_SNAP",
"CMD_SNAP_IMAGE",
+ "CMD_RECORDING",
+ "CMD_RECORDING_STATUS",
+ "EVENT_RECORDING_OPEN",
+ "EVENT_RECORDING_SAVE",
"EVENT_CARD",
"EVENT_CARD_ACTION",
"EVENT_SNAP_OPEN",
diff --git a/jarvis/cu/indicator/renderer.py b/jarvis/cu/indicator/renderer.py
index 7c0d721c38..d92f469cb8 100644
--- a/jarvis/cu/indicator/renderer.py
+++ b/jarvis/cu/indicator/renderer.py
@@ -341,6 +341,7 @@ class _SnapWindow(QWidget):
on_done,
*,
flash: bool = True,
+ corner: str = "right",
) -> None:
super().__init__(None)
self.setWindowFlags(
@@ -384,6 +385,8 @@ class _SnapWindow(QWidget):
right = float(avail.right() + 1 - geo.x())
bottom = float(avail.bottom() + 1 - geo.y())
self._dst = QRectF(right - tw - _SNAP_MARGIN, bottom - th - _SNAP_MARGIN, tw, th)
+ if corner == "left":
+ self._dst.moveLeft(float(avail.left() - geo.x()) + _SNAP_MARGIN)
self._t = 0.0
self._anim = QVariantAnimation(self)
self._anim.setStartValue(0.0)
@@ -1201,6 +1204,9 @@ class Renderer(QObject):
self._snaps: list[_SnapWindow] = []
#: The corner stack, oldest first; the newest sits at the bottom.
self._cards: list[_CardWindow] = []
+ from jarvis.appshot.recording_preview import RecordingPreviews
+
+ self._recordings = RecordingPreviews(_emit, self._recording_activity)
self._card_hint = ""
self._card_rest_ms = _CARD_REST_MS
self._card_labels: dict[str, str] = {}
@@ -1272,6 +1278,12 @@ class Renderer(QObject):
self._snap(payload)
elif cmd == protocol.CMD_SNAP_IMAGE:
self._snap_image(payload)
+ elif cmd == protocol.CMD_RECORDING:
+ self._recordings.show(payload)
+ elif cmd == protocol.CMD_RECORDING_STATUS:
+ self._recordings.set_status(
+ str(payload.get("id", "")), str(payload.get("text", ""))
+ )
elif cmd == protocol.CMD_CARD:
self._card_cmd(payload)
elif cmd == protocol.CMD_CARD_STATUS:
@@ -1329,6 +1341,10 @@ class Renderer(QObject):
def shutdown(self) -> None:
"""The user's pointer always comes back, whatever ended the sidecar."""
self._set_pointer(False)
+ self._recordings.close()
+
+ def _recording_activity(self) -> None:
+ _emit(protocol.EVENT_CARD, open=bool(self._cards) or self._recordings.active)
def _snap(self, payload: dict) -> None:
thumb = QImage()
@@ -1531,7 +1547,7 @@ class Renderer(QObject):
with suppress(ValueError):
self._cards.remove(card)
if not self._cards:
- _emit(protocol.EVENT_CARD, open=False)
+ self._recording_activity()
return
self._relayout() # the cards above it glide down
@@ -1565,6 +1581,7 @@ class Renderer(QObject):
return None
def _blank(self) -> None:
+ self._recordings.suspend(True)
# A resting thumbnail must never end up inside the next capture.
# A flight still on its way lands now, so a quick next appshot only
# stacks it instead of losing it; then every card hides.
@@ -1581,6 +1598,7 @@ class Renderer(QObject):
self._pointer.set_blanked(True)
def _unblank(self) -> None:
+ self._recordings.suspend(False)
for card in self._cards:
if not card.isVisible():
card.show()
diff --git a/jarvis/live/config.py b/jarvis/live/config.py
index e930393d3a..f50ddf4b2c 100644
--- a/jarvis/live/config.py
+++ b/jarvis/live/config.py
@@ -73,6 +73,10 @@ class LiveConfig(BaseModel):
from jarvis.brain.identity import name_directive
identity = name_directive("")
+ # The computer tool is declared by name whenever it exists
+ # (``live.tools._DIRECT_TOOLS``); with Computer Use switched off its
+ # rules would send the model to a tool it does not have.
+ computer = any(isinstance(t, dict) and t.get("name") == "computer" for t in tools)
backend: dict = {
"model": self.backend_model,
"instructions": (
@@ -113,13 +117,14 @@ class LiveConfig(BaseModel):
"count and the task as prompt; never an existing agent and never spawn_worker. "
+ AGENT_BRIEF_RULE
+ " "
- + COMPUTER_CONTROL_RULES
+ + (COMPUTER_CONTROL_RULES if computer else "")
+ " Appshots: when asked to take an appshot, screenshot, or look at the "
"current screen, call take_appshot for a fresh capture, even if an earlier "
"image is already in context. This tool owns the capture animation and "
"privacy filtering. Use scope window by default; scope screen only for "
- "an explicit whole-screen request. Use computer for operating the desktop. "
- "An attached image is a static snapshot, never proof that you performed a "
+ "an explicit whole-screen request. "
+ + ("Use computer for operating the desktop. " if computer else "")
+ + "An attached image is a static snapshot, never proof that you performed a "
"new capture. Describe an existing supplied image when asked about that "
"image. Confirm a requested new capture only after take_appshot succeeds "
"in this request; if it fails, explain the failure without describing the "
diff --git a/jarvis/live/native.py b/jarvis/live/native.py
index 1228dd9315..1feb195c03 100644
--- a/jarvis/live/native.py
+++ b/jarvis/live/native.py
@@ -154,7 +154,12 @@ class NativeLiveVoiceSession(LiveVoiceSession):
+ "Use your tools directly "
"for actions, private information and current facts. Use discover_tools and "
"call_tool for any tool not declared directly. "
- + COMPUTER_CONTROL_RULES
+ + (
+ # Only when Computer Use is on: the tool is then in the catalog.
+ COMPUTER_CONTROL_RULES
+ if any(d.name == "computer" for d in self._tools.catalog())
+ else ""
+ )
+ " When the user asks for an appshot, call take_appshot. "
"Request confirmation for pending approvals. Use confirm_action only after "
"explicit approval. A started job is not complete. Never invent tool results. "
diff --git a/jarvis/live/runtime.py b/jarvis/live/runtime.py
index b6a2ff613d..8ab9b7c194 100644
--- a/jarvis/live/runtime.py
+++ b/jarvis/live/runtime.py
@@ -3,13 +3,48 @@
from __future__ import annotations
import asyncio
+import threading
+from dataclasses import dataclass
from typing import Any
+from uuid import uuid4
_active: dict[str, Any] = {}
_owners: dict[str, asyncio.AbstractEventLoop] = {}
_opening: set[str] = set()
_watchers: set[tuple[asyncio.AbstractEventLoop, asyncio.Event]] = set()
_detached: set[asyncio.Task] = set()
+_request_lock = threading.Lock()
+
+
+@dataclass
+class _BrowserStart:
+ loop: asyncio.AbstractEventLoop
+ changed: asyncio.Event
+ session_id: str
+ owner_key: str
+ failed: bool = False
+ registered: bool = False
+
+
+_pending_browser_starts: dict[str, _BrowserStart] = {}
+_browser_call_requests: dict[str, str] = {}
+
+
+def browser_startup_failed(request_id: str) -> bool:
+ """Fail only the still-pending native request named by its browser owner."""
+ with _request_lock:
+ pending = _pending_browser_starts.get(request_id)
+ if (pending is None or _active or pending.loop.is_closed()
+ or _browser_call_requests.get(pending.owner_key) != request_id):
+ return False
+ _pending_browser_starts.pop(request_id, None)
+ pending.failed = True
+ try:
+ pending.loop.call_soon_threadsafe(pending.changed.set)
+ except RuntimeError:
+ # The owning event loop shut down after the closed check.
+ return False
+ return True
def retain_work(jobs: tuple[asyncio.Task, ...], ledger: Any) -> None:
@@ -39,16 +74,25 @@ def claim(session_id: str) -> None:
def register(session: Any) -> None:
- _opening.discard(session.session_id)
- _active[session.session_id] = session
- _owners[session.session_id] = asyncio.get_running_loop()
+ with _request_lock:
+ _opening.discard(session.session_id)
+ _active[session.session_id] = session
+ _owners[session.session_id] = asyncio.get_running_loop()
+ for request_id, pending in tuple(_pending_browser_starts.items()):
+ if not pending.session_id or pending.session_id == session.session_id:
+ # Registration and an early disconnect can both happen before
+ # the native waiter resumes. Remember the transition instead
+ # of waiting for a session which has already come and gone.
+ pending.registered = True
+ _pending_browser_starts.pop(request_id, None)
_notify()
def unregister(session_id: str) -> None:
- _opening.discard(session_id)
- _active.pop(session_id, None)
- _owners.pop(session_id, None)
+ with _request_lock:
+ _opening.discard(session_id)
+ _active.pop(session_id, None)
+ _owners.pop(session_id, None)
_notify()
@@ -99,6 +143,25 @@ async def run_browser_call(
changed = asyncio.Event()
watcher = (asyncio.get_running_loop(), changed)
_watchers.add(watcher)
+ request_id = str(uuid4())
+ owner_key = session_id or request_id
+ pending = _BrowserStart(watcher[0], changed, session_id, owner_key)
+ with _request_lock:
+ previous_id = _browser_call_requests.get(owner_key)
+ previous = _pending_browser_starts.pop(previous_id, None)
+ _browser_call_requests[owner_key] = request_id
+ _pending_browser_starts[request_id] = pending
+ if previous is not None:
+ previous.failed = True
+ _notify()
+
+ def owns_request() -> bool:
+ with _request_lock:
+ return _browser_call_requests.get(owner_key) == request_id
+
+ def own_sessions() -> tuple[Any, ...]:
+ return tuple(session for session in active()
+ if not session_id or session.session_id == session_id)
async def wait_change() -> None:
tasks = [asyncio.create_task(changed.wait()), asyncio.create_task(hangup.wait())]
@@ -115,21 +178,37 @@ async def run_browser_call(
return "hotkey"
if input_buffer is not None and session_id:
startup.offer(session_id, input_buffer)
- await bus.publish(BrowserVoiceRequested(action="start"))
+ await bus.publish(BrowserVoiceRequested(action="start", request_id=request_id))
# UI permission and device setup can take time. No idle billed connection.
async with asyncio.timeout(timeout_s):
- while not active() and not hangup.is_set():
+ while (not pending.registered and not own_sessions()
+ and not hangup.is_set() and owns_request()):
+ if pending.failed or (session_id and active()):
+ return "error"
await wait_change()
- while active() and not hangup.is_set():
+ if pending.failed:
+ return "error"
+ while own_sessions() and not hangup.is_set() and owns_request():
await wait_change()
except TimeoutError:
# The caller receives an explicit error outcome for this bounded wait.
return "error"
finally:
- startup.discard(session_id)
+ with _request_lock:
+ _pending_browser_starts.pop(request_id, None)
+ still_owned = _browser_call_requests.get(owner_key) == request_id
+ if still_owned:
+ sessions_to_close = own_sessions()
+ startup.discard(session_id)
+ _browser_call_requests.pop(owner_key, None)
_watchers.discard(watcher)
- # Retract pending browser starts on timeout/cancellation too. Otherwise
- # focusing a hidden window later starts a call whose owner already left.
- await bus.publish(BrowserVoiceRequested(action="stop"))
- await close_all()
+ if still_owned:
+ # Correlated retraction cannot stop a newer browser request. Only
+ # close this native session; unrelated browser calls remain owned.
+ await bus.publish(BrowserVoiceRequested(action="stop", request_id=request_id))
+ await asyncio.gather(*(
+ on_session_loop(session, session.end, reason="hotkey")
+ for session in sessions_to_close
+ if _active.get(session.session_id) is session
+ ))
return "hotkey" if hangup.is_set() else "client_stop"
diff --git a/jarvis/live/session.py b/jarvis/live/session.py
index f9a2579561..dfbf12502d 100644
--- a/jarvis/live/session.py
+++ b/jarvis/live/session.py
@@ -23,6 +23,8 @@ from jarvis.live.tools import LiveTools, take_images
from jarvis.realtime.audio import StreamingPcm16Resampler
log = logging.getLogger(__name__)
+_STARTUP_CLOSE_SEND_TIMEOUT_S = 2.0
+_STARTUP_CLOSE_TIMEOUT_S = 5.0
def _identity(config: Any) -> str:
@@ -163,6 +165,7 @@ class LiveVoiceSession:
self._brain = brain
self._surface = surface
self._connection: Any = None
+ self._opening_task: asyncio.Task | None = None
self._pump_task: asyncio.Task | None = None
self._jobs: set[asyncio.Task] = set()
self._control_tasks: set[asyncio.Task] = set()
@@ -682,6 +685,7 @@ class LiveVoiceSession:
# off the wake path instead of stacked after a cold-disk sqlite open.
self._ledger, _ = await asyncio.gather(_open_ledger(), _take_permit())
assert self._ledger is not None
+ self._startup_timing.mark("local_setup_ready")
setup_ms = (time.monotonic() - started_at) * 1000.0
self._tools = LiveTools(
gateway,
@@ -709,10 +713,6 @@ class LiveVoiceSession:
claim(self.session_id)
self._watch_input_mute()
await self._send_json({"type": "input_mute", "muted": self._input_muted})
- # The browser already captures. Finish the native handoff now,
- # before a remote handshake can fill/overflow its 30-second buffer.
- await self._take_startup_input(message)
- self._startup_timing.mark("capture_handoff")
await self._publish_phase("connecting")
from jarvis.live.recovery import seed_messages
@@ -720,21 +720,29 @@ class LiveVoiceSession:
if self._initial_seed:
config["input"] = seed_messages(self._initial_seed, [])
+ if self._closing:
+ return
open_started_at = time.monotonic()
self._startup_timing.mark("provider_open")
- self._connection = await self._provider.open_session(
- ContinuousVoiceStart(
- session=config, offer_sdp=offer,
- on_transport_ready=self._transport_ready if offer else None,
- on_startup_phase=self._startup_timing.mark,
- )
+ opening = asyncio.create_task(
+ self._open_with_input_handoff(
+ message,
+ ContinuousVoiceStart(
+ session=config, offer_sdp=offer,
+ on_transport_ready=self._transport_ready if offer else None,
+ on_startup_phase=self._startup_timing.mark,
+ ),
+ ),
+ name="live-startup",
)
+ self._opening_task = opening
+ try:
+ await opening
+ finally:
+ if self._opening_task is opening:
+ self._opening_task = None
if self._closing:
- # A hangup may arrive through another surface during open.
- try:
- await self._connection.send({"type": "session.close"})
- finally:
- await self._connection.close()
+ # end() cancels and awaits opening before closing its result.
return
self._startup_timing.mark("control_connected")
open_ms = (time.monotonic() - open_started_at) * 1000.0
@@ -755,6 +763,7 @@ class LiveVoiceSession:
"OpenAI Live rejected session setup. "
"Check the selected model and account access."
)
+ self._startup_timing.mark("provider_ready")
self._pump_task = asyncio.create_task(self._pump(), name="live-events")
from jarvis.brain import provider_health_ledger as health_ledger
from jarvis.core.events import (
@@ -784,20 +793,13 @@ class LiveVoiceSession:
turn_id=self._archive_turn_id,
)
)
- await self._bus.publish(
- RealtimeSessionReady(
- session_id=self.session_id,
- provider=self.active_provider,
- model=profile.model,
- surface=self._surface,
- input_sample_rate=24000,
- output_sample_rate=24000,
- language=self._language,
- )
- )
- await self._publish_phase()
+ self._startup_timing.mark("lifecycle_started")
if self._closing:
return
+ # Release usable audio before status fan-out and persistence. A
+ # slow observer must not hold the microphone behind a ready control
+ # channel. Session/turn events still precede input, and the browser
+ # separately waits for ICE/DTLS before releasing its local buffer.
self._startup_timing.mark("audio_ready")
await self._send_json(
{
@@ -818,6 +820,22 @@ class LiveVoiceSession:
"input_muted": self._input_muted,
}
)
+ if self._closing:
+ return
+ if self._bus is not None:
+ await self._bus.publish(
+ RealtimeSessionReady(
+ session_id=self.session_id,
+ provider=self.active_provider,
+ model=profile.model,
+ surface=self._surface,
+ input_sample_rate=24000,
+ output_sample_rate=24000,
+ language=self._language,
+ )
+ )
+ await self._publish_phase()
+ self._startup_timing.mark("ready_notifications_done")
if not offer:
# Audio already flows over this socket: agent results that
# finished before the call may be offered at once. A WebRTC
@@ -830,6 +848,64 @@ class LiveVoiceSession:
await self.end(reason="error")
raise
+ async def _open_with_input_handoff(self, message: dict, config: Any) -> None:
+ """Overlap native microphone release with the explicitly requested call.
+
+ Handoff starts first and must finish, including delivery of its local
+ prefix, before audio_ready. Own the provider result immediately so a
+ concurrent handoff failure or hangup cannot orphan an allocated call.
+ Publish the connection to end() only when the handoff also succeeds.
+ """
+ async def handoff() -> None:
+ await self._take_startup_input(message)
+ self._startup_timing.mark("capture_handoff")
+
+ connection = None
+
+ async def open_connection() -> None:
+ nonlocal connection
+ connection = await self._provider.open_session(config)
+ self._startup_timing.mark("control_connected")
+
+ tasks = (
+ asyncio.create_task(handoff(), name="live-input-handoff"),
+ asyncio.create_task(open_connection(), name="live-provider-open"),
+ )
+ try:
+ await asyncio.gather(*tasks)
+ except BaseException:
+ for task in tasks:
+ if not task.done():
+ task.cancel()
+
+ async def reap() -> None:
+ # A cancelled provider may still return an allocated call.
+ await asyncio.gather(*tasks, return_exceptions=True)
+ if connection is None:
+ return
+ try:
+ try:
+ async with asyncio.timeout(_STARTUP_CLOSE_SEND_TIMEOUT_S):
+ await connection.send({"type": "session.close"})
+ finally:
+ async with asyncio.timeout(_STARTUP_CLOSE_TIMEOUT_S):
+ await connection.close()
+ except Exception:
+ # Preserve the original cancellation/handoff error.
+ log.warning("Live handoff cleanup could not confirm the allocated call closed")
+
+ cleanup = asyncio.create_task(reap(), name="live-startup-cleanup")
+ while not cleanup.done():
+ try:
+ await asyncio.shield(cleanup)
+ except asyncio.CancelledError:
+ # Hangup and surface disconnect can cancel independently.
+ # Keep owning cleanup until the remote call is retired.
+ continue
+ await cleanup
+ raise
+ self._connection = connection
+
async def _transport_ready(self, answer_sdp: str) -> None:
"""Negotiate media early; this message never releases microphone audio."""
if self._closing:
@@ -1043,7 +1119,8 @@ class LiveVoiceSession:
while not self._closed.is_set():
try:
event = await self._connection.receive()
- except Exception: # Shutdown is quiet; active failures enter the reporting recovery loop.
+ except Exception:
+ # Shutdown is quiet; active failures enter reporting recovery.
if self._closing:
return
self._had_unconfirmed_wire = True
@@ -1138,6 +1215,15 @@ class LiveVoiceSession:
# state. Sideband generation can lead playback or include silence.
from jarvis.live.playback import AudioTimedFrame, playback_frame, source_interval
+ timing = self._startup_timing
+ if timing is not None and "first_output_signal_received" not in timing.marks:
+ # Continuous providers emit silent PCM before their first
+ # spoken response. Inspect at most one second per frame until
+ # nonzero PCM arrives; no samples or levels are retained.
+ pcm = base64.b64decode(event["delta"][:64_000])
+ if any(pcm):
+ timing.mark("first_output_signal_received")
+
timed = source_interval(event.get("start_ms"), event.get("end_ms"))
if timed:
self._last_output_audio_end = max(
@@ -1720,6 +1806,13 @@ class LiveVoiceSession:
unregister(self.session_id)
self._closing = True
self._notify_ended()
+ opening = self._opening_task
+ if opening is not None and opening is not asyncio.current_task():
+ if not opening.done() and not opening.cancelling():
+ opening.cancel()
+ # Stop a hung handoff and retire any allocation before returning
+ # from hangup, even if the provider returns during cancellation.
+ await asyncio.gather(opening, return_exceptions=True)
if self._recovering and self._pump_task is not None:
self._closed.set()
if self._pump_task is not asyncio.current_task():
diff --git a/jarvis/live/subscription.py b/jarvis/live/subscription.py
index c3efadc4d8..ed3d3331b4 100644
--- a/jarvis/live/subscription.py
+++ b/jarvis/live/subscription.py
@@ -69,6 +69,12 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
SubscriptionReasoningBrain(self._reasoning, config.live.backend_model),
)
self._delegation_lock = asyncio.Lock()
+ # Only inference may be preempted. The delegation owner retains the
+ # lock through an executing tool and its durable receipt.
+ self._reasoning_task: asyncio.Task | None = None
+ self._queued_requests: dict[str, tuple[int, str]] = {}
+ self._steering_receipts: dict[str, tuple[int, dict]] = {}
+ self._unfinished_groups: list[list[dict]] = []
self._seen_delegations: set[str] = set()
self._local_delegations: set[str] = set()
self._backend_groups: list[list[dict]] = []
@@ -128,6 +134,17 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
task.cancel()
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
+ self._queued_requests.clear()
+ self._steering_receipts.clear()
+ self._unfinished_groups.clear()
+ self._remember([{
+ "role": "assistant", "content": [{
+ "type": "output_text",
+ "text": "[Application state: the user cancelled unfinished voice requests. "
+ "Do not resume them without a new request. Already started agents keep "
+ "their tasks; inspect receipts before repeating any action.]",
+ }],
+ }])
return
if message.get("type") != "text_input" or self._connection is None:
await super().handle_control(message)
@@ -197,6 +214,13 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
self._tools.user_text,
)
)
+ if not event.get("application_event"):
+ # Admit before yielding. Intermediate requests remain context
+ # even when a later correction takes ownership of the reply.
+ self._queued_requests[identifier] = (revision, prompt)
+ if self._reasoning_task is not None:
+ self._reasoning_task.cancel()
+ following_up = not event.get("application_event") and self._delegation_lock.locked()
task = asyncio.create_task(
self._run_client_delegation(
identifier,
@@ -211,6 +235,20 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
self._subscription_report_task = task
task.add_done_callback(self._job_finished)
await self._note_thinking()
+ if following_up:
+ try:
+ await self._connection.send({
+ "type": "session.thinking.append",
+ "delegation_id": (
+ None if identifier in self._local_delegations else identifier
+ ),
+ "content": "The application accepted the follow-up. Reasoning will use "
+ "it now, or after the current tool returns its receipt. Already started "
+ "agents continue working. This is acceptance, not task completion.",
+ })
+ except Exception:
+ # Admission already succeeded; a failed status must not erase it.
+ log.warning("Live follow-up status could not be delivered", exc_info=True)
return
if kind == "error" and event.get("fatal"):
await self._failure(
@@ -416,6 +454,13 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
self._tools.language = self._language
fragments = await asyncio.to_thread(self._ledger.transcript, self.session_id)
items = seed_messages(self._initial_seed, [])
+ # History eviction is allowed only for completed conversation. Keep
+ # every input and receipt of an unfinished steering chain available.
+ retained = {id(group) for group in self._backend_groups}
+ items.extend(
+ item for group in self._unfinished_groups if id(group) not in retained
+ for item in group
+ )
items.extend(item for group in self._backend_groups for item in group)
context = {
"role": "user",
@@ -431,8 +476,22 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
latest = prompt.strip() or getattr(self._tools, "request_text", self._tools.user_text)
if not latest:
return
- request_item = {"role": "user", "content": [{"type": "input_text", "text": latest}]}
- history_prefix = [request_item]
+ # Transcript persistence above yields. Do not let an overtaken job
+ # consume the queue or start another inference with old authority.
+ if self._closing or revision != self._tools.revision:
+ return
+ requests = [latest]
+ if not application_event and identifier in self._queued_requests:
+ requests = [text for _, text in self._queued_requests.values()]
+ self._queued_requests.clear()
+ request_items = [
+ {"role": "user", "content": [{"type": "input_text", "text": text}]}
+ for text in requests
+ ]
+ if not application_event:
+ # Save accepted input before inference: cancelling an unfinished
+ # round must not erase the original task or intervening requests.
+ self._remember(request_items, unfinished=True)
if not application_event:
if self._pending_images:
self._image_context = self._pending_images
@@ -448,7 +507,7 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
]})
# Put the actual request after retained image context. A previous
# appshot's framing must not override a request for a fresh capture.
- items.append(request_item)
+ items.extend(request_items)
backend = self._config.live.backend_config(
language=self._language,
tools=self._tools.declarations(defer_catalog=True),
@@ -457,6 +516,13 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
# Search runs through Jarvis's tools; never imply a hosted API tool
# is authorized by this different Codex subscription credential.
tools = [tool for tool in backend["tools"] if tool.get("type") == "function"]
+ backend["instructions"] += (
+ " Follow-up user requests can steer unfinished work. Preserve earlier "
+ "requests unless the user changes or cancels them; the latest correction "
+ "wins. Use tool receipts and exact returned agent/session IDs to continue "
+ "work already started. Never repeat a completed send or spawn merely "
+ "because reasoning was interrupted. Ask if the target remains ambiguous."
+ )
if application_event:
tools = []
backend["instructions"] += (
@@ -472,24 +538,24 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
rounds = 0
while rounds < round_limit:
rounds += 1
- if self._closing or self._tools.cancel_token.is_cancelled():
+ if (self._closing or self._tools.cancel_token.is_cancelled()
+ or revision != self._tools.revision):
return
if len(json.dumps(items, ensure_ascii=False).encode("utf-8")) > 16_000_000:
await self._failure("context_capacity")
return
- output, text, response_id, usage = await self._reasoning_round(
- identifier,
- items,
- backend,
- tools,
+ result = await self._steerable_reasoning_round(
+ identifier, items, backend, tools, revision,
)
+ if result is None or revision != self._tools.revision:
+ return
+ output, text, response_id, usage = result
calls = [item for item in output if item.get("type") == "function_call"]
if calls and application_event:
await self._failure("subscription_unavailable")
return
items.extend(output)
- history_group = [*history_prefix, *output]
- history_prefix = []
+ history_group = list(output)
if not calls:
if not application_event:
self._remember(history_group)
@@ -531,6 +597,9 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
"content": text,
}
)
+ if not application_event and revision == self._tools.revision:
+ self._steering_receipts.clear()
+ self._unfinished_groups.clear()
log.info("Subscription result submitted to the live voice session.")
return
if round_limit == _MAX_ROUNDS and any(_is_computer_call(c) for c in calls):
@@ -561,12 +630,7 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
"error": "Invalid tool arguments.",
}
else:
- result = await self._tools.execute(
- str(call["call_id"]),
- str(call.get("name") or ""),
- args,
- revision,
- )
+ result = await self._execute_steerable_call(call, args, revision)
images = take_images(result)
result_item = {
"type": "function_call_output",
@@ -604,7 +668,7 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
if self._tools.end_requested:
await self.end(reason="tool_hangup")
return
- self._remember(history_group)
+ self._remember(history_group, unfinished=True)
if revision != self._tools.revision:
return
await self._failure("context_capacity")
@@ -619,7 +683,8 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
if not self._closing:
await self._failure(str(getattr(exc, "code", "subscription_unavailable")))
finally:
- self._thinking = False
+ current = asyncio.current_task()
+ self._thinking = any(task is not current and not task.done() for task in self._jobs)
if not self._closing:
await self._publish_phase()
log.info(
@@ -627,8 +692,64 @@ class SubscriptionLiveVoiceSession(LiveVoiceSession):
(time.monotonic() - started) * 1000,
)
- def _remember(self, group: list[dict]) -> None:
+ async def _steerable_reasoning_round(
+ self, identifier: str, items: list[dict], backend: dict, tools: list[dict], revision: int,
+ ) -> tuple[list[dict], str, str, dict] | None:
+ """A new request cancels inference, never the parent or an executing tool."""
+ task = asyncio.create_task(
+ self._reasoning_round(identifier, items, backend, tools),
+ name="live-subscription-reasoning",
+ )
+ self._reasoning_task = task
+ try:
+ return await task
+ except asyncio.CancelledError:
+ owner = asyncio.current_task()
+ if owner is not None and owner.cancelling():
+ raise
+ if self._tools is not None and revision != self._tools.revision:
+ log.info("Subscription inference yielded to a follow-up request.")
+ return None
+ raise
+ finally:
+ if self._reasoning_task is task:
+ self._reasoning_task = None
+
+ async def _execute_steerable_call(self, call: dict, args: dict, revision: int) -> dict:
+ """Retain receipts across replanning, with all effects still serialized."""
+ assert self._tools is not None
+ if revision != self._tools.revision:
+ return {
+ "success": False, "executed": False, "status": "superseded",
+ "error": "Follow-up input arrived before this tool started. Re-evaluate it.",
+ }
+ name = str(call.get("name") or "")
+ key = self._tools.steering_replay_key(name, args)
+ previous = self._steering_receipts.get(key) if key is not None else None
+ if previous is not None and previous[0] < revision:
+ return await self._tools.execute(
+ str(call["call_id"]), name, args, revision,
+ replay_result=copy.deepcopy(previous[1]),
+ )
+ approved_effect = self._tools.approval_effect_key(name, args)
+ result = await self._tools.execute(str(call["call_id"]), name, args, revision)
+ if result.get("success") and approved_effect is not None:
+ key = approved_effect
+ if (key is not None and result.get("executed") is not False
+ and not result.get("retryable")
+ and result.get("status") not in {
+ "superseded", "cancelled", "voice_closed_before_execution",
+ }
+ and not result.get("confirmation_required")):
+ # Include uncertain failures: a send can succeed before its transport
+ # fails. Replanning is not permission to replay it with a new call ID.
+ self._steering_receipts[key] = (revision, copy.deepcopy(result))
+ return result
+
+ def _remember(self, group: list[dict], *, unfinished: bool = False) -> None:
"""Evict complete response/tool groups, never leave orphan tool results."""
+ if unfinished:
+ self._unfinished_groups.append(group)
self._backend_groups.append(group)
while self._backend_groups and (
len(self._backend_groups) > 16
diff --git a/jarvis/live/timing.py b/jarvis/live/timing.py
index 170f23f097..7456a4ebf7 100644
--- a/jarvis/live/timing.py
+++ b/jarvis/live/timing.py
@@ -9,8 +9,13 @@ import time
log = logging.getLogger(__name__)
BROWSER_PHASES = frozenset({
- "capture_ready", "first_capture_frame", "offer_ready", "socket_open",
+ "microphone_ready", "worklet_ready", "capture_ready", "first_capture_frame",
+ "offer_ready", "socket_open", "control_ready",
"answer_received", "media_connected", "input_released", "first_output_audio",
+ "context_prepared", "context_created", "worklet_loaded", "context_running",
+ "input_started", "input_caught_up", "first_output_packet",
+ "local_requests_dispatched", "rtc_destination_begin", "rtc_destination_ready",
+ "peer_created", "offer_requested",
})
diff --git a/jarvis/live/tools.py b/jarvis/live/tools.py
index 79546f1343..b0ca761b5c 100644
--- a/jarvis/live/tools.py
+++ b/jarvis/live/tools.py
@@ -226,7 +226,10 @@ class LiveTools:
definitions.append(definition)
return definitions
- async def execute(self, call_id: str, name: str, args: dict, revision: int) -> dict:
+ async def execute(
+ self, call_id: str, name: str, args: dict, revision: int,
+ *, replay_result: dict | None = None,
+ ) -> dict:
async with self._lock:
if (self._hangup_confirmation.pending_turn is not None
and confirms_hangup(self.user_text) and name != "end_call"):
@@ -247,7 +250,17 @@ class LiveTools:
if receipt is not None:
return receipt
try:
- result = await self._execute(name, args, revision)
+ if replay_result is not None and self.steering_replay_key(name, args) is None:
+ result = {
+ "success": False, "executed": False,
+ "error": "Controls, approvals and reads require a fresh evaluation.",
+ }
+ elif replay_result is not None and revision == self.revision:
+ # The session retained this result from an earlier execution.
+ # Record the new model call ID without performing another effect.
+ result = {**replay_result, "reused_receipt": True}
+ else:
+ result = await self._execute(name, args, revision)
except asyncio.CancelledError:
# Leave the claim uncertain: a cancelled request may already have acted.
raise
@@ -481,13 +494,66 @@ class LiveTools:
]
def _reads_only(self, name: str, args: dict) -> bool:
- """True for a call that only reads: tool search or a safe-tier tool."""
+ """A safe permission tier does not imply a read: internal sends are safe."""
if name == "discover_tools":
return True
if name == "call_tool":
name = str(args.get("name", ""))
+ try:
+ args = json.loads(args.get("arguments_json") or "{}")
+ except (TypeError, ValueError):
+ return False # Unknown arguments cannot prove a read-only call.
canonical = self._names.get(name, name)
- return any(d.name == canonical and d.risk_tier == "safe" for d in self.catalog())
+ descriptor = next((d for d in self.catalog() if d.name == canonical), None)
+ if descriptor is None or not isinstance(args, dict):
+ return False
+ describe = getattr(descriptor, "describe_args", None)
+ if callable(describe):
+ try:
+ return describe(args).get("level") == "read"
+ except Exception:
+ log.warning("Live tool impact inspection failed for %s", canonical, exc_info=True)
+ return False
+ return descriptor.risk_tier == "safe" and not descriptor.is_action_tool
+
+ def _replay_call(self, name: str, args: dict) -> tuple[str, dict] | None:
+ """Normalize aliases and wrappers without authorizing an operation."""
+ if ":" in name:
+ prefix, suffix = name.split(":", 1)
+ if prefix.isidentifier() and (suffix in self._names or suffix in _BUILTIN_TOOLS):
+ name = suffix
+ if name == "call_tool":
+ name = str(args.get("name", ""))
+ try:
+ args = json.loads(args.get("arguments_json") or "{}")
+ except (TypeError, ValueError):
+ return None # Invalid arguments have no executable identity.
+ name = self._names.get(name, name)
+ return (name, args) if isinstance(args, dict) else None
+
+ def steering_replay_key(self, name: str, args: dict) -> str | None:
+ """Identity of an effect across replanning, independent of model call IDs.
+
+ Session controls and reads cannot be reused this way. In particular,
+ action approval and hang-up each require their own fresh consent.
+ """
+ call = self._replay_call(name, args)
+ if call is None:
+ return None
+ name, args = call
+ if name in _BUILTIN_TOOLS or self._reads_only(name, args):
+ return None
+ return json.dumps([name, args], sort_keys=True, ensure_ascii=False)
+
+ def approval_effect_key(self, name: str, args: dict) -> str | None:
+ """The pending effect's identity; the confirmation itself is never replayed."""
+ call = self._replay_call(name, args)
+ if call is None or call[0] != "confirm_action":
+ return None
+ pending = self._pending.get(str(call[1].get("approval_id", "")))
+ if pending is None and len(self._pending) == 1:
+ pending = next(iter(self._pending.values()))
+ return self.steering_replay_key(pending[1], pending[2]) if pending else None
def _user_approved(self) -> bool:
"""True when the latest user text is a short, unvetoed yes in any locale.
diff --git a/jarvis/marketplace/plugins/atlassian/plugin.json b/jarvis/marketplace/plugins/atlassian/plugin.json
index 10a01ed8a0..41932fff09 100644
--- a/jarvis/marketplace/plugins/atlassian/plugin.json
+++ b/jarvis/marketplace/plugins/atlassian/plugin.json
@@ -7,6 +7,7 @@
"extensions": {
"io.github.personaljarvis": {
"display_name": "Atlassian (Jira & Confluence)",
+ "acceptance": "verified",
"category": "Lists & Tasks",
"logo_slug": "atlassian",
"logo_color": "0052CC",
diff --git a/jarvis/marketplace/publisher_clients.py b/jarvis/marketplace/publisher_clients.py
index f04473bcd0..7dd0e5cccb 100644
--- a/jarvis/marketplace/publisher_clients.py
+++ b/jarvis/marketplace/publisher_clients.py
@@ -69,7 +69,11 @@ ClientSource = Literal["publisher", "own", "catalog", "missing"]
# Slack is the publicly distributed "Personal Jarvis" Slack app. Slack only
# allows https redirect URLs for it, and web-redirect tokens need the secret,
# so the family uses a token broker with an https redirect below.
+# Figma is the "Personal Jarvis" Figma OAuth app: loopback PKCE on
+# http://127.0.0.1:3127/oauth/callback. Its token and refresh endpoints take
+# the client secret in a Basic header, so the family uses a token broker.
SHIPPED_PUBLIC_CLIENT_IDS: dict[str, str] = {
+ "figma": "QnSTQR5QK5hAOjxBmp6sIY",
"github": "Ov23liaA2KfHbfVOelcx",
"google": "940985062784-4t8h10q3ggt6b1317ajjb9u0eriop2m7.apps.googleusercontent.com",
"microsoft": "1986efcb-f871-4ffa-8df7-64c04dd3ab56",
@@ -109,6 +113,7 @@ _BROKER_BASE = "https://token.personaljarvis.ai/oauth"
# token endpoint directly with its own secret. Revocation never needs a secret
# and always goes straight to the provider.
SHIPPED_TOKEN_BROKERS: dict[str, TokenBroker] = {
+ "figma": TokenBroker(token_url=f"{_BROKER_BASE}/figma/token"),
"google": TokenBroker(token_url=f"{_BROKER_BASE}/google/token"),
"slack": TokenBroker(
token_url=f"{_BROKER_BASE}/slack/token",
diff --git a/jarvis/marketplace/seed_catalog.json b/jarvis/marketplace/seed_catalog.json
index 6c6d6c891d..83531051c9 100644
--- a/jarvis/marketplace/seed_catalog.json
+++ b/jarvis/marketplace/seed_catalog.json
@@ -470,6 +470,7 @@
},
{
"id": "atlassian",
+ "acceptance": "verified",
"display_name": "Atlassian (Jira & Confluence)",
"description": "Jira issues and Confluence pages",
"category": "Lists & Tasks",
diff --git a/jarvis/plugins/brain/_anthropic_base.py b/jarvis/plugins/brain/_anthropic_base.py
index 998640ead6..99a231f3d7 100644
--- a/jarvis/plugins/brain/_anthropic_base.py
+++ b/jarvis/plugins/brain/_anthropic_base.py
@@ -19,7 +19,7 @@ from . import _agent_profile
# 64-char cap is stricter than Anthropic's 128 but still valid, so a slash/dot/
# colon MCP name (jarvis/mcp/adapter.py) no longer trips Anthropic's
# ``tools.N.custom.name`` 400 on the direct claude-api path.
-from ._openai_base import _openai_tool_name_map
+from ._openai_base import _openai_tool_name_map, _sanitize_openai_function_name
# Latency-sprint-2: beta header for the 1h cache TTL. The default is 5 min;
# 1h extends the effective cache duration and covers more voice sessions.
@@ -32,7 +32,9 @@ _ANTHROPIC_CACHE_TTL_BETA = "extended-cache-ttl-2025-04-11"
_ENV_PROMPT_CACHE = "JARVIS_ANTHROPIC_PROMPT_CACHE"
-def _to_anthropic_messages(messages: tuple[BrainMessage, ...]) -> list[dict[str, Any]]:
+def _to_anthropic_messages(
+ messages: tuple[BrainMessage, ...], name_map: dict[str, str] | None = None,
+) -> list[dict[str, Any]]:
"""BrainMessages → Anthropic API messages array.
Anthropic supports roles: "user", "assistant". "system" is passed
@@ -43,9 +45,20 @@ def _to_anthropic_messages(messages: tuple[BrainMessage, ...]) -> list[dict[str,
Backwards compat: without images, string content stays a string.
"""
out: list[dict[str, Any]] = []
+ name_map = dict(name_map or {})
+ taken = set(name_map.values())
for m in messages:
role = m.role
content = m.content
+ if role == "assistant" and isinstance(content, list):
+ content = [dict(block) for block in content]
+ for block in content:
+ if block.get("type") == "tool_use":
+ original = str(block.get("name") or "")
+ if original not in name_map:
+ name_map[original] = _sanitize_openai_function_name(original, taken)
+ taken.add(name_map[original])
+ block["name"] = name_map[original]
if role == "system":
continue # system is passed externally as the `system` parameter
@@ -267,11 +280,11 @@ async def stream_complete(
req: BrainRequest,
) -> AsyncIterator[BrainDelta]:
"""Runs a streaming messages.create and yields BrainDeltas."""
- messages = _to_anthropic_messages(req.messages)
system = _extract_system(req.messages, req.system)
# Sanitize tool names + keep a reverse map so the inbound tool_use name maps
# back to the ORIGINAL tool the executor knows (e.g. the "server/tool" MCP name).
name_map = _openai_tool_name_map(req.tools) if req.tools else {}
+ messages = _to_anthropic_messages(req.messages, name_map)
reverse_name_map = {safe: original for original, safe in name_map.items()}
tools_payload = _tools_anthropic_format(req.tools, name_map) if req.tools else None
@@ -325,7 +338,13 @@ async def stream_complete(
# An agent call may think for minutes; the voice path's short client
# timeout is replaced only while an agent request profile is active.
if (timeout := _agent_profile.http_timeout()) is not None:
- kwargs["timeout"] = timeout
+ from anthropic import Timeout
+
+ # The SDK owns its transport family (httpx or httpx2). Preserve every
+ # bound while constructing the timeout type that its client accepts.
+ kwargs["timeout"] = Timeout(
+ connect=timeout.connect, read=timeout.read, write=timeout.write, pool=timeout.pool,
+ )
# A requested reasoning effort (the agent chat's picker) becomes
# ``output_config.effort`` plus adaptive thinking on the models that
# take it; the voice brain never sets one and sends exactly what it did.
@@ -335,18 +354,42 @@ async def stream_complete(
if tools_payload:
kwargs["tools"] = tools_payload
if extra_headers:
+ # Per-request headers replace SDK defaults. Keep authentication betas
+ # (notably OAuth) when adding the cache capability.
+ defaults = getattr(client, "default_headers", {})
+ existing = next((str(v) for k, v in defaults.items()
+ if k.lower() == "anthropic-beta"), "")
+ extra_headers["anthropic-beta"] = ",".join(dict.fromkeys(
+ beta.strip() for beta in (existing + "," + extra_headers["anthropic-beta"]).split(",")
+ if beta.strip()
+ ))
kwargs["extra_headers"] = extra_headers
async with client.messages.stream(**kwargs) as stream:
# Tool-call accumulator (Anthropic streams tool_use as separate blocks)
current_tool: dict[str, Any] | None = None
current_tool_json = ""
+ initial_usage: dict[str, int] = {}
+ reported_usage: dict[str, int] = {}
+ response_blocks: list[dict[str, Any]] = []
+ pending_tools: list[dict[str, Any]] = []
+ current_block: dict[str, Any] | None = None
async for event in stream:
etype = getattr(event, "type", None) or getattr(event, "event", None)
+ if etype == "message_start":
+ usage = getattr(getattr(event, "message", None), "usage", None)
+ initial_usage = {
+ target: int(getattr(usage, source, 0) or 0)
+ for source, target in (
+ ("input_tokens", "input_tokens"),
+ ("cache_read_input_tokens", "cache_hit_tokens"),
+ ("cache_creation_input_tokens", "cache_write_tokens"),
+ )
+ }
# Text delta
- if etype == "content_block_delta":
+ elif etype == "content_block_delta":
delta = getattr(event, "delta", None)
if delta is None:
continue
@@ -354,15 +397,33 @@ async def stream_complete(
if dtype == "text_delta":
text = getattr(delta, "text", "") or ""
if text:
+ if current_block is not None:
+ current_block["text"] = current_block.get("text", "") + text
yield BrainDelta(content=text)
elif dtype == "input_json_delta":
if current_tool is not None:
partial = getattr(delta, "partial_json", "") or ""
current_tool_json += partial
+ elif dtype in {"thinking_delta", "signature_delta"} and current_block is not None:
+ field = "thinking" if dtype == "thinking_delta" else "signature"
+ current_block[field] = current_block.get(field, "") + (
+ getattr(delta, field, "") or ""
+ )
# Tool-use block start
elif etype == "content_block_start":
block = getattr(event, "content_block", None)
+ current_block = None
+ kind = getattr(block, "type", "")
+ if kind == "thinking":
+ current_block = {"type": kind, "thinking": getattr(block, "thinking", ""),
+ "signature": getattr(block, "signature", "")}
+ elif kind == "redacted_thinking":
+ current_block = {"type": kind, "data": getattr(block, "data", "")}
+ elif kind == "text":
+ current_block = {"type": kind, "text": getattr(block, "text", "")}
+ if current_block is not None:
+ response_blocks.append(current_block)
if block is not None and getattr(block, "type", None) == "tool_use":
_raw_name = getattr(block, "name", "")
current_tool = {
@@ -376,33 +437,59 @@ async def stream_complete(
if current_tool is not None:
try:
parsed = json.loads(current_tool_json) if current_tool_json else {}
- except json.JSONDecodeError:
- parsed = {}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ "The provider returned incomplete tool arguments.",
+ ) from exc
+ if not isinstance(parsed, dict):
+ raise ValueError("The provider returned non-object tool arguments.")
current_tool["input"] = parsed
- yield BrainDelta(tool_call=current_tool)
+ response_blocks.append({"type": "tool_use", **current_tool})
+ pending_tools.append(current_tool)
current_tool = None
current_tool_json = ""
+ current_block = None
# Message end with usage
elif etype == "message_delta":
delta = getattr(event, "delta", None)
finish = getattr(delta, "stop_reason", None) if delta else None
+ if finish and pending_tools:
+ if finish == "max_tokens":
+ raise ValueError("The provider stopped before completing its tool calls.")
+ has_thinking = any(b["type"] in {"thinking", "redacted_thinking"}
+ for b in response_blocks)
+ for call in pending_tools:
+ # Opaque continuation data stays inside Jarvis; the
+ # gateway's OpenAI wire cannot represent signed blocks.
+ if has_thinking:
+ call = {**call, "_anthropic_blocks": response_blocks}
+ yield BrainDelta(tool_call=call)
+ pending_tools.clear()
usage = getattr(event, "usage", None)
- usage_d: dict[str, int] = {}
+ usage_d: dict[str, int] = dict(initial_usage)
if usage is not None:
- usage_d = {
- "input_tokens": int(getattr(usage, "input_tokens", 0) or 0),
+ usage_d.update({
+ "input_tokens": int(getattr(usage, "input_tokens", None)
+ or initial_usage.get("input_tokens", 0)),
"output_tokens": int(getattr(usage, "output_tokens", 0) or 0),
# The protocol key is cache_hit_tokens (protocols.py) —
# this plugin used to forward Anthropic's wire name
# cache_read_input_tokens, which no consumer reads, so
# cache hits were invisible in cost and telemetry and
# cache regressions could not be measured.
- "cache_hit_tokens": int(getattr(usage, "cache_read_input_tokens", 0) or 0),
+ "cache_hit_tokens": int(getattr(usage, "cache_read_input_tokens", None)
+ or initial_usage.get("cache_hit_tokens", 0)),
# Cache writes bill above the input rate; a mission's
# paid-use cap counts them (jarvis/missions/capacity.py).
"cache_write_tokens": int(
- getattr(usage, "cache_creation_input_tokens", 0) or 0
+ getattr(usage, "cache_creation_input_tokens", None)
+ or initial_usage.get("cache_write_tokens", 0)
),
- }
- yield BrainDelta(finish_reason=finish, usage=usage_d or None)
+ })
+ # Anthropic's updates are cumulative; BrainDelta consumers sum
+ # increments across the stream.
+ increment = {key: max(0, value - reported_usage.get(key, 0))
+ for key, value in usage_d.items()}
+ reported_usage.update(usage_d)
+ yield BrainDelta(finish_reason=finish, usage=increment or None)
diff --git a/jarvis/plugins/brain/_openai_base.py b/jarvis/plugins/brain/_openai_base.py
index ab3472e9ba..95df17c5e2 100644
--- a/jarvis/plugins/brain/_openai_base.py
+++ b/jarvis/plugins/brain/_openai_base.py
@@ -678,17 +678,30 @@ async def stream_complete(
finish = getattr(choice, "finish_reason", None)
if finish:
+ if tool_buffer and finish not in {"tool_calls", "stop"}:
+ # Preserve the stop reason so the caller can retry with a
+ # larger budget; no partial tool arguments may execute.
+ tool_buffer.clear()
+ yield BrainDelta(finish_reason=finish)
+ continue
# Finalize tool calls if present
+ completed_calls = []
for idx, buf in sorted(tool_buffer.items()):
try:
parsed = json.loads(buf["arguments"]) if buf["arguments"] else {}
- except json.JSONDecodeError:
- parsed = {}
- yield BrainDelta(tool_call={
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ "The provider returned incomplete tool arguments.",
+ ) from exc
+ if not isinstance(parsed, dict):
+ raise ValueError("The provider returned non-object tool arguments.")
+ completed_calls.append({
"id": buf["id"] or f"call_{idx}",
"name": reverse_name_map.get(buf["name"], buf["name"]),
"input": parsed,
})
+ for call in completed_calls:
+ yield BrainDelta(tool_call=call)
tool_buffer.clear()
yield BrainDelta(finish_reason=finish)
diff --git a/jarvis/plugins/brain/ollama.py b/jarvis/plugins/brain/ollama.py
index d471b5d4ac..a9f123a7e9 100644
--- a/jarvis/plugins/brain/ollama.py
+++ b/jarvis/plugins/brain/ollama.py
@@ -209,6 +209,11 @@ class OllamaBrain:
)
return self._client
+ def _native_headers(self) -> dict[str, str]:
+ """Native discovery/profile calls use the same endpoint credential as chat."""
+ self._resolve_root()
+ return {"Authorization": f"Bearer {self._credential}"} if self._credential else {}
+
async def _resolve_model(self, *, need_tools: bool = False, need_vision: bool = False) -> str:
"""The configured model, else the smallest CAPABLE download.
@@ -232,7 +237,9 @@ class OllamaBrain:
return self._discovered[profile]
root = self._resolve_root()
try:
- async with httpx.AsyncClient(timeout=CLIENT_TIMEOUT) as client:
+ async with httpx.AsyncClient(
+ timeout=CLIENT_TIMEOUT, headers=self._native_headers(),
+ ) as client:
resp = await client.get(f"{root}/api/tags")
resp.raise_for_status()
models = resp.json().get("models") or []
@@ -309,7 +316,9 @@ class OllamaBrain:
return self._caps_cache[name]
caps_set: set[str] | None = None
try:
- async with httpx.AsyncClient(timeout=CLIENT_TIMEOUT) as client:
+ async with httpx.AsyncClient(
+ timeout=CLIENT_TIMEOUT, headers=self._native_headers(),
+ ) as client:
resp = await client.post(f"{root}/api/show", json={"model": name})
resp.raise_for_status()
caps = resp.json().get("capabilities")
@@ -414,7 +423,7 @@ class OllamaBrain:
run_model = model
if has_bakeable(opts):
try:
- run_model = await ensure_profile(root, model, opts)
+ run_model = await ensure_profile(root, model, opts, headers=self._native_headers())
except Exception as exc: # noqa: BLE001 — degrade to the base model, say so
if self._requested_context_window is not None:
# The runtime already budgets against this window. Sending
@@ -431,7 +440,7 @@ class OllamaBrain:
if opts.num_ctx:
self.context_window = opts.num_ctx
if opts.keep_alive is not None:
- await warm(root, run_model, opts.keep_alive)
+ await warm(root, run_model, opts.keep_alive, headers=self._native_headers())
overrides: dict[str, Any] = {}
defaults = {f.name: f.default for f in fields(BrainRequest)}
for key, value in to_v1_kwargs(opts).items():
diff --git a/jarvis/plugins/realtime/_live_transport.py b/jarvis/plugins/realtime/_live_transport.py
index f90fb6cd5c..471d219d0f 100644
--- a/jarvis/plugins/realtime/_live_transport.py
+++ b/jarvis/plugins/realtime/_live_transport.py
@@ -24,6 +24,51 @@ _TLS_MAX_AGE_S = 300.0
_cleanup_tasks: set[asyncio.Task] = set()
+def startup_http_trace(mark: Callable[[str], Any]) -> Callable[..., Any]:
+ """Record transport milestones without retaining HTTP trace payloads.
+
+ The trace metadata contains headers and request bodies. Read only the
+ fixed event name so authentication, SDP and user context never reach logs.
+ """
+ phases = {
+ "connection.connect_tcp.complete": "http_tcp_connected",
+ "connection.start_tls.complete": "http_tls_connected",
+ "http11.send_request_body.complete": "http_request_sent",
+ "http2.send_request_body.complete": "http_request_sent",
+ "http11.receive_response_headers.complete": "http_response_headers",
+ "http2.receive_response_headers.complete": "http_response_headers",
+ }
+
+ async def trace(event: str, _info: dict) -> None:
+ phase = phases.get(event)
+ if phase is not None:
+ mark(phase)
+
+ return trace
+
+
+def startup_websocket_options(mark: Callable[[str], Any]) -> dict[str, Any]:
+ """Time the supported connection lifecycle without reading wire content.
+
+ For WSS, connection_made runs after TCP/proxy and TLS setup. Separating it
+ from the opening handshake exposes whether the wait is local transport or
+ the provider's HTTP upgrade. Optional WebSocket imports remain call-owned.
+ """
+ from websockets.asyncio.client import ClientConnection
+
+ class StartupConnection(ClientConnection):
+ def connection_made(self, transport: Any) -> None:
+ super().connection_made(transport)
+ mark("control_transport_connected")
+
+ async def handshake(self, *args: Any, **kwargs: Any) -> None:
+ mark("control_handshake_started")
+ await super().handshake(*args, **kwargs)
+ mark("control_handshake_complete")
+
+ return {"create_connection": StartupConnection}
+
+
def _trust_key() -> tuple:
# Replacing an explicitly configured bundle must invalidate the context.
# OS trust-store edits are picked up within the bounded refresh interval.
@@ -60,6 +105,26 @@ async def websocket_options() -> dict[str, Any]:
return {"ssl": await asyncio.to_thread(_websocket_context)}
+@asynccontextmanager
+async def preparing_websocket_options() -> AsyncIterator[asyncio.Task]:
+ """Overlap local trust-store loading with allocation, with no remote I/O.
+
+ A cold or expired trust store must not sit between the SDP answer and
+ control attachment. The worker owns only immutable local TLS state, so
+ cancellation can abandon its result without leaving a connection open.
+ """
+ task = asyncio.create_task(websocket_options(), name="live-tls-prepare")
+ try:
+ yield task
+ finally:
+ if not task.done():
+ task.cancel()
+ # A failure reached the caller through its await, or the caller failed
+ # authentication/cancelled first. In the latter case preserve that
+ # primary failure; preparation has no remote resources to clean up.
+ await asyncio.gather(task, return_exceptions=True)
+
+
async def warm_transport() -> None:
"""Prime imports and certificate parsing without DNS, auth or remote I/O."""
def prepare() -> None:
diff --git a/jarvis/plugins/realtime/openai_live.py b/jarvis/plugins/realtime/openai_live.py
index e2e95bc38c..c6ad0d4fd4 100644
--- a/jarvis/plugins/realtime/openai_live.py
+++ b/jarvis/plugins/realtime/openai_live.py
@@ -117,7 +117,13 @@ class OpenAILiveProvider:
from websockets.asyncio.client import connect
- from ._live_transport import preparing_http_client, websocket_options
+ from ._live_transport import (
+ preparing_http_client,
+ preparing_websocket_options,
+ startup_http_trace,
+ startup_websocket_options,
+ websocket_options,
+ )
if not self._api_key:
raise ValueError("Connect OpenAI in API Keys before starting voice.")
@@ -136,13 +142,17 @@ class OpenAILiveProvider:
# Construction can load certificates/proxy settings synchronously.
# Keep the application's existing HTTP trust/cache policy;
# never retain a billed or loop-bound idle voice connection.
- async with preparing_http_client(timeout=25) as preparation:
+ async with (
+ preparing_http_client(timeout=25) as preparation,
+ preparing_websocket_options() as tls_preparation,
+ ):
client = await asyncio.shield(preparation)
mark("http_client_ready")
response = await client.post(
"https://api.openai.com/v1/live/sessions",
headers=headers,
json={"session": session, "transport": {"type": "webrtc", "sdp": offer}},
+ extensions={"trace": startup_http_trace(mark)},
)
if response.status_code >= 400:
from jarvis.brain.provider_test import classify_provider_error
@@ -161,6 +171,11 @@ class OpenAILiveProvider:
session_id = payload["session"]["id"]
answer = payload["transport"]["sdp"]
mark("session_response")
+ on_transport_ready = getattr(cfg, "on_transport_ready", None)
+ if answer and on_transport_ready is not None:
+ await on_transport_ready(answer)
+ options = await tls_preparation
+ mark("control_tls_ready")
log.info(
"OpenAI Live session created in %.0f ms.",
(time.monotonic() - started_at) * 1000.0,
@@ -171,18 +186,14 @@ class OpenAILiveProvider:
**session.get("audio", {}),
"format": {"type": "audio/pcm", "rate": 24000},
}
- on_transport_ready = getattr(cfg, "on_transport_ready", None)
- if answer and on_transport_ready is not None:
- # Let ICE/DTLS run concurrently with sideband attachment. The
- # application still withholds microphone audio and tool-ready
- # state until this method returns successfully.
- await on_transport_ready(answer)
- options = await websocket_options()
- mark("control_tls_ready")
+ options = await websocket_options()
+ mark("control_tls_ready")
attach_started_at = time.monotonic()
+ mark("control_connect_started")
socket = await connect(
url, additional_headers=headers, open_timeout=25, max_size=8_000_000,
**options,
+ **startup_websocket_options(mark),
)
log.info(
"OpenAI Live transport attached in %.0f ms.",
diff --git a/jarvis/plugins/realtime/openai_subscription_live.py b/jarvis/plugins/realtime/openai_subscription_live.py
index e271cabe21..a1802db578 100644
--- a/jarvis/plugins/realtime/openai_subscription_live.py
+++ b/jarvis/plugins/realtime/openai_subscription_live.py
@@ -613,7 +613,12 @@ class OpenAISubscriptionLiveProvider:
# Optional dependencies are loaded only for an explicitly started call.
from websockets.asyncio.client import connect
- from ._live_transport import preparing_http_client, websocket_options
+ from ._live_transport import (
+ preparing_http_client,
+ preparing_websocket_options,
+ startup_http_trace,
+ startup_websocket_options,
+ )
offer = str(cfg.offer_sdp or "")
if not _audio_only_sdp(offer) or len(offer.encode("utf-8")) > _MAX_SDP_BYTES:
@@ -635,9 +640,12 @@ class OpenAISubscriptionLiveProvider:
try:
# Local TLS/client construction and selected-account loading are
# independent. Neither operation can allocate a remote voice call.
- async with preparing_http_client(
- self._http_client_factory, timeout=25, follow_redirects=False,
- ) as preparation:
+ async with (
+ preparing_http_client(
+ self._http_client_factory, timeout=25, follow_redirects=False,
+ ) as preparation,
+ preparing_websocket_options() as tls_preparation,
+ ):
headers = {**await self._headers(), **request_headers}
credentials_ready = True
mark("credentials_ready")
@@ -649,6 +657,7 @@ class OpenAISubscriptionLiveProvider:
CALL_URL,
headers=headers,
json={"sdp": offer, "session": session},
+ extensions={"trace": startup_http_trace(mark)},
) as response:
if response.status_code == 401 and attempt == 0:
# Rejected auth allocated no call. Refresh once;
@@ -669,11 +678,13 @@ class OpenAISubscriptionLiveProvider:
raise SubscriptionLiveError("invalid_response")
mark("session_response")
break
- on_transport_ready = getattr(cfg, "on_transport_ready", None)
- if on_transport_ready is not None:
- # Negotiate browser media while the trusted sideband attaches.
- # The host keeps microphone output gated until both are ready.
- await on_transport_ready(answer)
+ on_transport_ready = getattr(cfg, "on_transport_ready", None)
+ if on_transport_ready is not None:
+ # ICE/DTLS can also overlap a cold local trust-store load.
+ # The host withholds input until media AND control are ready.
+ await on_transport_ready(answer)
+ options = await tls_preparation
+ mark("control_tls_ready")
except asyncio.CancelledError:
if allocated:
await self._retire_allocation(call_id, headers, connector)
@@ -700,9 +711,8 @@ class OpenAISubscriptionLiveProvider:
# retry pays the host's shared connection permit and includes jitter.
for attempt in range(3):
try:
- options = await websocket_options()
- mark("control_tls_ready")
attach_started_at = time.monotonic()
+ mark("control_connect_started")
socket = await connector(
SIDEBAND_BASE + call_id,
additional_headers=headers,
@@ -711,6 +721,7 @@ class OpenAISubscriptionLiveProvider:
max_size=_MAX_FRAME_BYTES,
max_queue=32,
**options,
+ **startup_websocket_options(mark),
)
log.info(
"Subscription Live control attached in %.0f ms.",
diff --git a/jarvis/plugins/tool/app_action.py b/jarvis/plugins/tool/app_action.py
index 795c6c0cb5..6cabd269cd 100644
--- a/jarvis/plugins/tool/app_action.py
+++ b/jarvis/plugins/tool/app_action.py
@@ -147,6 +147,31 @@ class RunAppActionTool:
runtime = _Runtime()
self._runtime = runtime
+ @staticmethod
+ def _reads(args: dict[str, Any]) -> bool:
+ """Does this action only read? The catalog's own rule (``default_tier``).
+
+ A GET that is not marked dangerous reads. The person's per-action mode
+ still decides whether it runs (``risk_tier_for_args``); this only says
+ what it is, so a voice read is never treated as a change, nor answered
+ from an earlier call's result. Deliberately no ``read_only_for_args``:
+ a read-only chat keeps refusing app actions as before.
+ """
+ from jarvis.app_actions.catalog import default_tier
+
+ entry = _catalog().get(str((args or {}).get("action_id") or ""))
+ return entry is not None and default_tier(entry) == "safe"
+
+ def describe_args(self, args: dict[str, Any]) -> dict[str, str]:
+ entry = _catalog().get(str((args or {}).get("action_id") or ""))
+ return {
+ "level": "read" if self._reads(args) else "modify",
+ # Named in a confirmation question when the action asks first.
+ "commands": entry.title
+ if entry is not None
+ else str((args or {}).get("action_id") or ""),
+ }
+
def risk_tier_for_args(self, args: dict[str, Any]) -> str:
from jarvis.app_actions import history
from jarvis.app_actions.policy import effective_tier
diff --git a/jarvis/plugins/tool/computer.py b/jarvis/plugins/tool/computer.py
index 60c4df25b7..b68cef9d63 100644
--- a/jarvis/plugins/tool/computer.py
+++ b/jarvis/plugins/tool/computer.py
@@ -22,10 +22,30 @@ class ComputerTool:
is_action_tool = True
schema = tool_schema()
- def describe_args(self, args: dict[str, Any]) -> str:
- steps = args.get("steps") if isinstance(args, dict) else None
- actions = [str(step.get("action")) for step in steps or [] if isinstance(step, dict)]
- return "Operate the screen: " + (", ".join(actions) or "screenshot")
+ def describe_args(self, args: dict[str, Any]) -> dict[str, str]:
+ """The impact contract (``{"level", ...}``) every caller reads as a dict.
+
+ Looking (screenshot, wait) only reads; any input step changes the
+ screen. A plain string broke the voice read check on every call, so a
+ repeated screenshot could be answered with an earlier, stale one.
+ """
+ # Read the call as ``direct.parse_steps`` does (one bare step allowed)
+ # and fail closed: only a well-formed call of pure looks is a read.
+ args = args if isinstance(args, dict) else {}
+ raw = args.get("steps")
+ if raw is None and "action" in args:
+ raw = [args]
+ steps = raw if isinstance(raw, list) else []
+ actions = [
+ str(step.get("action", "")).strip().lower() for step in steps if isinstance(step, dict)
+ ]
+ looks_only = bool(actions) and len(actions) == len(steps) and all(
+ action in {"screenshot", "wait"} for action in actions
+ )
+ return {
+ "level": "read" if looks_only else "modify",
+ "commands": "Operate the screen: " + (", ".join(actions) or "screenshot"),
+ }
async def execute(self, args, ctx):
from jarvis.core.protocols import ToolResult
diff --git a/jarvis/safety/risk_tier.py b/jarvis/safety/risk_tier.py
index 60cf8ee3bb..f7d7a906d5 100644
--- a/jarvis/safety/risk_tier.py
+++ b/jarvis/safety/risk_tier.py
@@ -185,6 +185,11 @@ class RiskTierEvaluator:
command_string=cmd,
)
+ @property
+ def bypass_permissions(self) -> bool:
+ """The app default for calls without a session-owned approval policy."""
+ return self._safety.approval_mode == "bypass"
+
def needs_user_confirmation(self, decision: TierDecision) -> bool:
"""True if a user confirmation must be obtained before execution."""
if decision.approved_by is not None:
diff --git a/jarvis/safety/tool_executor.py b/jarvis/safety/tool_executor.py
index 30610aaf31..0c3afdbd3d 100644
--- a/jarvis/safety/tool_executor.py
+++ b/jarvis/safety/tool_executor.py
@@ -20,6 +20,7 @@ from uuid import UUID, uuid4
from jarvis.core.bus import EventBus
from jarvis.core.events import (
ActionApprovalRequired,
+ ActionApproved,
ActionDenied,
ActionExecuted,
ActionProposed,
@@ -337,12 +338,19 @@ class ToolExecutor:
tool.name, exc,
)
needs_confirm = tier_confirm or plaus_confirm
+ # A bound chat owns its approval stance through ChatApprovalBridge.
+ # Calls from voice, tasks, missions and other app surfaces use the
+ # global default. This never overrides the block/read-only/cancel
+ # checks, and model-controlled tool arguments cannot set the policy.
+ bypass_default = self._evaluator.bypass_permissions and not (
+ config_snapshot or {}
+ ).get("approval_ref")
# WHO can answer this gate — declared by the calling layer, never by
# the model (see ``approval_surface``). It changes only how the
# approval is obtained, never whether one is needed: ``needs_confirm``
# above is the untouched tier decision.
surface = resolve_approval_surface(config_snapshot)
- voice_confirm = surface == CONVERSATIONAL
+ voice_confirm = surface == CONVERSATIONAL and not bypass_default
# How long a card may stay open — declared by the calling layer like
# the surface itself (a person reading a chat card must not lose the
# tool to a clock built for a spoken "ja"), clamped so no surface can
@@ -403,6 +411,17 @@ class ToolExecutor:
),
)
)
+ # Scoped authorizers have the first decision, including explicit
+ # denials. Bypass supplies an immediate standing approval only
+ # when none has answered; it never overwrites their decision.
+ if bypass_default and approval_ticket.peek() is None:
+ await self._bus.publish(
+ ActionApproved(
+ trace_id=tid,
+ tool_name=tool.name,
+ approved_by="bypass",
+ )
+ )
if needs_confirm:
# Two-turn confirmation on a conversational turn: do NOT block on the
diff --git a/jarvis/society/agent_template.py b/jarvis/society/agent_template.py
index e4853b2e0a..26abbcbb74 100644
--- a/jarvis/society/agent_template.py
+++ b/jarvis/society/agent_template.py
@@ -99,6 +99,8 @@ _SCOPES: Final[tuple[str, ...]] = ("shared", "own")
# URL served by the author's own app and never travels.
_AVATAR_ID_RE: Final[re.Pattern[str]] = re.compile(r"^[a-z0-9_-]{1,40}$")
_HEX_RE: Final[re.Pattern[str]] = re.compile(r"^#[0-9a-fA-F]{6}$")
+#: ``USER_ID_RE`` in ``jarvis/ui/pets/manifest.py``: a pet drawn on this machine.
+_DRAWN_PET_RE: Final[re.Pattern[str]] = re.compile(r"^u[0-9a-f]{16}$")
_AVATAR_STRINGS: Final[tuple[str, ...]] = (
"archetype",
"base",
@@ -108,7 +110,7 @@ _AVATAR_STRINGS: Final[tuple[str, ...]] = (
"eyewear",
)
_COMPANION_KEYS: Final[frozenset[str]] = frozenset(
- {"shape", "color", "eyes", "enabled", "sizeM", "followDistanceM"}
+ {"shape", "color", "eyes", "enabled", "sizeM", "followDistanceM", "pet"}
)
@@ -280,6 +282,9 @@ def public_avatar(avatar: Any) -> dict[str, Any]:
companion = avatar.get("companion")
if isinstance(companion, dict):
kept = {k: v for k, v in companion.items() if k in _COMPANION_KEYS}
+ # A pet the person drew lives only on their machine; a built-in one travels.
+ if isinstance(kept.get("pet"), str) and _DRAWN_PET_RE.fullmatch(kept["pet"]):
+ del kept["pet"]
# The roster validates this namespace strictly; a companion it would
# refuse is dropped here, so a template never fails to install over
# the colour of a floating shape.
diff --git a/jarvis/society/agent_tools.py b/jarvis/society/agent_tools.py
index d23f34c060..e733c534e3 100644
--- a/jarvis/society/agent_tools.py
+++ b/jarvis/society/agent_tools.py
@@ -20,6 +20,7 @@ folder.
from __future__ import annotations
+import asyncio
import logging
import re
from pathlib import Path
@@ -519,6 +520,10 @@ class ProposeChangeTool:
"you are still new (no title and no description), your first identity applies at "
"once from the "
"user's message, so propose it as soon as the user has told you what you are for. "
+ "If your name is still a temporary creation label, an identity payload containing "
+ "only name also saves automatically in your own direct user chat, even when your "
+ "role is already stored. Choose a short role-based name within the first one to "
+ "three meaningful replies; keep an already chosen name unless the user changes it. "
"Say why in 'reason'. "
"Use it when the user states a lasting preference, asks you to remember a way of "
"working, to save a procedure, or to run something regularly. Never propose the "
@@ -567,7 +572,7 @@ class ProposeChangeTool:
from jarvis.core.protocols import current_chat_turn
from .proposals import ProposalRefused, propose, resolve
- from .roster import is_fresh
+ from .roster import has_placeholder_name, is_fresh
from .surface import agent_id_of
rt = self._runtime
@@ -586,9 +591,21 @@ class ProposeChangeTool:
# A fresh agent takes its first identity from its person's turn in its
# own chat without a card; the outcome card offers undo.
payload = args.get("payload") if isinstance(args.get("payload"), dict) else {}
+ first_name = (
+ has_placeholder_name(caller)
+ and set(payload) == {"name"}
+ and turn is not None
+ and turn.session_id == caller.session_id
+ and (not self._session_id or self._session_id == caller.session_id)
+ )
+ keeps_name = (
+ "name" not in payload
+ or str(payload["name"]).strip() == caller.name
+ or has_placeholder_name(caller)
+ )
fresh_apply = (
kind == "identity"
- and is_fresh(caller)
+ and ((is_fresh(caller) and keeps_name) or first_name)
and own_user_turn
# A long role text is a card, not an automatic change.
and len(str(payload.get("description") or "")) <= FRESH_IDENTITY_MAX_CHARS
@@ -772,9 +789,17 @@ class ShellTool:
from .remote import backend_for
backend = self._backend or backend_for(caller, self._workspace)
- result = await backend.run(command, cwd=cwd, timeout_s=timeout)
+ from .credentials import current_vault
+
+ vault = current_vault()
+ secrets = await asyncio.to_thread(vault.shell_env, caller.agent_id) if vault else {}
+ if secrets and getattr(backend, "accepts_env", False):
+ result = await backend.run(command, cwd=cwd, timeout_s=timeout, extra_env=secrets)
+ else:
+ result = await backend.run(command, cwd=cwd, timeout_s=timeout)
+ output = vault.redact(caller.agent_id, result.output) if vault else result.output
body = {
- "output": result.output,
+ "output": output,
"exit_code": result.exit_code,
"seconds": round(result.seconds, 2),
"folder": str(cwd),
@@ -784,9 +809,14 @@ class ShellTool:
if runs_on:
# A remote command did not run in the local ``folder`` above; say where.
body["runs_on"] = runs_on
+ if secrets and not getattr(backend, "accepts_env", False):
+ body["credentials"] = (
+ "Your stored credentials are only set for commands on this computer, "
+ "not on the computer this command ran on."
+ )
if result.timed_out:
return ToolResult(success=False, output=body, error="command timed out")
if result.failed_to_start:
- return ToolResult(success=False, output=body, error=result.output)
+ return ToolResult(success=False, output=body, error=output)
error = None if result.ok else f"exit {result.exit_code}"
return ToolResult(success=result.ok, output=body, error=error)
diff --git a/jarvis/society/ask_tool.py b/jarvis/society/ask_tool.py
index f099a0c192..d986b1460b 100644
--- a/jarvis/society/ask_tool.py
+++ b/jarvis/society/ask_tool.py
@@ -35,6 +35,7 @@ from jarvis.agent_chat.questions import (
parse_questions,
recommended_answer,
)
+from jarvis.agent_chat.secret_requests import asks_for_secret
from jarvis.core.protocols import ToolResult
from .routine_runner import is_routine_session
@@ -103,6 +104,7 @@ class AskUserTool:
f"{_MINUTES} minutes, your recommendations are applied, so recommend the safest "
"sensible choice. Never ask for permission to do what you were asked, never ask "
"what you could look up, and never ask again what was already answered. "
+ "Never ask for a token, key or password here: use society_request_credential. "
f"A turn can show at most {MAX_ASKS_PER_TURN} question cards. While the user has "
"not answered yet, the call returns status 'waiting' with a question_id: then call "
"this tool again with only wait_for set to that id, and do nothing else meanwhile."
@@ -141,6 +143,18 @@ class AskUserTool:
except ValueError as exc:
# Bad questions go back to the agent as the tool error to fix.
return ToolResult(success=False, output=None, error=f"invalid questions: {exc}")
+ if any(asks_for_secret(spec.question, value_only=True) for spec in specs):
+ # A typed answer would carry the secret into the chat and the model.
+ return ToolResult(
+ success=False,
+ output=None,
+ error=(
+ "A question card never asks for a token, key or password: its answer is "
+ "plain chat text. Call society_request_credential instead; it opens a "
+ "secure field and its description can say where the user creates the "
+ "credential."
+ ),
+ )
if is_routine_session(self._session_id):
return _unattended(
specs,
diff --git a/jarvis/society/browser/native_window.py b/jarvis/society/browser/native_window.py
index 5db2d09765..440b694de5 100644
--- a/jarvis/society/browser/native_window.py
+++ b/jarvis/society/browser/native_window.py
@@ -18,6 +18,7 @@ from typing import Any
log = logging.getLogger(__name__)
_capture_mta_lock = threading.Lock()
_capture_mta: tuple[Any, Any] | None = None
+_PARKED_ALPHA = 0
def _release_capture_mta() -> None:
@@ -83,6 +84,7 @@ class NativeWindow:
def __init__(self, pid: int) -> None:
self.failed = False
self._closed = False
+ self._parked = False
if os.name != "nt":
self.failed = True
return
@@ -104,6 +106,7 @@ class NativeWindow:
self._capture_factory = WindowsCapture
self._captures: dict[int, tuple[Any, Any]] = {}
self._capture_keys: dict[int, object] = {}
+ self._restarted_popups: set[int] = set()
self._images: dict[int, tuple[Any, float]] = {}
self._popups: list[int] = []
self._regions: list[dict[str, Any]] = []
@@ -176,6 +179,7 @@ class NativeWindow:
if not self.hwnd:
raise RuntimeError("The owned Chrome window is unavailable")
self.input_hwnd = self.hwnd
+ self.park()
self._event_task = self._loop.create_task(self._consume_window_events())
self._hook_thread = threading.Thread(
target=self._watch_windows, name="jarvis-chrome-window-events", daemon=True
@@ -295,19 +299,41 @@ class NativeWindow:
return True
self.user32.EnumWindows(visit, 0)
+ if self._parked:
+ for hwnd in popups:
+ self._hide_from_desktop(hwnd)
# EnumWindows enumerates front to back; painting uses the reverse order.
popups.reverse()
with self.lock:
self._popups = popups
self._revision += 1
+ self._restarted_popups.intersection_update(popups)
for hwnd in list(self._captures):
- if hwnd != self.hwnd and hwnd not in popups:
+ if hwnd == self.hwnd:
+ continue
+ if hwnd not in popups:
self._stop_capture(hwnd)
+ elif hwnd not in self._capture_keys:
+ if not self._stop_capture(hwnd):
+ self.failed = True
+ raise RuntimeError("The stopped Chrome popup capture could not be released")
+ if hwnd in self._restarted_popups:
+ self.failed = True
+ raise RuntimeError("The Chrome popup capture repeatedly stopped")
+ self._restarted_popups.add(hwnd)
+ log.warning("Restarting a stopped Chrome popup capture")
for hwnd in popups:
if hwnd not in self._captures:
try:
self._start_capture(hwnd)
- except Exception:
+ except Exception as exc:
+ if (
+ hwnd in self._restarted_popups
+ and self._owned(hwnd)
+ and self.user32.IsWindowVisible(hwnd)
+ ):
+ self.failed = True
+ raise RuntimeError("The Chrome popup capture could not restart") from exc
# Native menus can close between the show event and WGC
# startup; the main browser stream remains usable.
log.debug("Chrome popup capture could not start", exc_info=True)
@@ -348,12 +374,20 @@ class NativeWindow:
@capture.event
def on_closed() -> None:
- if self._capture_keys.get(hwnd) is not token:
- return
- if hwnd == self.hwnd and not self._closed:
- self.failed = True
- self.ready.set()
- elif not self._closed:
+ with self.lock:
+ if self._closed or self._capture_keys.get(hwnd) is not token:
+ return
+ if hwnd == self.hwnd:
+ self.failed = True
+ self.ready.set()
+ return
+ # Invalidate pixels and late callbacks immediately. The owning
+ # loop reaps the native worker before a bounded replacement;
+ # stopping/joining it inside this callback would deadlock.
+ self._capture_keys.pop(hwnd, None)
+ self._images.pop(hwnd, None)
+ self._revision += 1
+ if not self._closed:
self._queue_window_event(hwnd)
try:
@@ -364,10 +398,11 @@ class NativeWindow:
self._images.pop(hwnd, None)
raise
- def _stop_capture(self, hwnd: int) -> None:
+ def _stop_capture(self, hwnd: int) -> bool:
with self.lock:
self._capture_keys.pop(hwnd, None)
capture = self._captures.pop(hwnd, None)
+ stopped = threading.Event()
if capture:
def stop() -> None:
try:
@@ -375,6 +410,7 @@ class NativeWindow:
# so the deadline must enclose both operations.
capture[1].stop()
capture[1].wait()
+ stopped.set()
except Exception:
log.warning("Chrome capture cleanup failed", exc_info=True)
@@ -387,6 +423,7 @@ class NativeWindow:
log.warning("Chrome capture did not exit before its deadline")
with self.lock:
self._images.pop(hwnd, None)
+ return capture is None or stopped.is_set()
def _bounds(self, hwnd: int) -> tuple[int, int, int, int]:
c, w = self.ctypes, self.wintypes
@@ -886,8 +923,47 @@ class NativeWindow:
for hwnd in list(self._captures):
self._stop_capture(hwnd)
+ def _hide_from_desktop(self, hwnd: int) -> None:
+ """Keep an owned window transparent while retaining native capture.
+
+ Moving offscreen makes Chrome clamp its native menus onto a monitor.
+ Hiding/minimizing stops window capture. Desktop alpha leaves the native
+ surface and its coordinates intact; WGC captures the original pixels.
+ The owning browser process retains this style until it exits.
+ """
+ if not self._owned(hwnd):
+ raise RuntimeError("Chrome window ownership changed before hiding")
+ c, w, u = self.ctypes, self.wintypes, self.user32
+ u.GetWindowLongW.argtypes = [w.HWND, c.c_int]
+ u.GetWindowLongW.restype = c.c_long
+ u.SetWindowLongW.argtypes = [w.HWND, c.c_int, c.c_long]
+ u.SetWindowLongW.restype = c.c_long
+ u.SetLayeredWindowAttributes.argtypes = [w.HWND, w.DWORD, w.BYTE, w.DWORD]
+ u.SetLayeredWindowAttributes.restype = w.BOOL
+ u.GetLayeredWindowAttributes.argtypes = [
+ w.HWND, c.POINTER(w.DWORD), c.POINTER(w.BYTE), c.POINTER(w.DWORD),
+ ]
+ u.GetLayeredWindowAttributes.restype = w.BOOL
+ style = u.GetWindowLongW(hwnd, -20) # GWL_EXSTYLE is a 32-bit style, not a pointer.
+ # WGC rejects NOACTIVATE when APPWINDOW is absent. Use SWP_NOACTIVATE
+ # in park() instead; the transparent surface also ignores desktop clicks.
+ # Clear TOOLWINDOW on owned popups so they remain capturable too.
+ hidden_style = (style | 0x00080020) & ~0x08040080
+ if style != hidden_style:
+ u.SetWindowLongW(hwnd, -20, hidden_style)
+ if u.GetWindowLongW(hwnd, -20) != hidden_style:
+ raise RuntimeError("The Chrome window could not be removed from the desktop")
+ alpha, flags = w.BYTE(), w.DWORD()
+ if (
+ u.GetLayeredWindowAttributes(hwnd, None, c.byref(alpha), c.byref(flags))
+ and flags.value & 0x2 and alpha.value == _PARKED_ALPHA
+ ):
+ return
+ if not u.SetLayeredWindowAttributes(hwnd, 0, _PARKED_ALPHA, 0x2): # LWA_ALPHA
+ raise RuntimeError("The Chrome window could not be made transparent")
+
def park(self) -> None:
- """Keep Chrome behind Jarvis while preserving native popup placement."""
+ """Hide Chrome on the desktop while preserving the embedded window stream."""
if os.name != "nt":
return
u, w = self.user32, self.wintypes
@@ -903,6 +979,8 @@ class NativeWindow:
u.SetWindowPos.restype = w.BOOL
with self.dpi():
self._check_owner()
+ self._hide_from_desktop(self.hwnd)
+ self._parked = True
# Moving far offscreen breaks Chrome's monitor-clamped profile and
# login bubbles. HWND_BOTTOM does not activate or move the browser.
flags = 0x213
diff --git a/jarvis/society/companion.py b/jarvis/society/companion.py
index 2f482aa0c5..064fb5cc57 100644
--- a/jarvis/society/companion.py
+++ b/jarvis/society/companion.py
@@ -9,6 +9,10 @@ from pydantic import BaseModel, ConfigDict, Field, StringConstraints
# stay open-ended so a look saved by a newer build survives an older one.
AccessorySlot = Literal["back", "outfit", "neck", "mouth", "face", "head", "held"]
AccessoryId = Annotated[str, StringConstraints(pattern=r"^[a-z0-9_]{1,40}$")]
+# A pet from My Pets (``jarvis/ui/pets/manifest.py``: built-in slug or ``u<hex>``
+# for a drawn one). Open-ended like accessory ids: a pet deleted later or
+# missing on another machine shows the shape instead, it never breaks the look.
+PetId = Annotated[str, StringConstraints(pattern=r"^[a-z0-9][a-z0-9-]{0,31}$")]
#: Mirrors ``COMPANION_SHAPES`` / ``COMPANION_COLORS`` in the frontend's
#: ``components/society/companion/appearance.ts`` (parity test pins both).
@@ -16,7 +20,10 @@ COMPANION_SHAPES: Final[tuple[str, ...]] = (
"circle", "squircle", "pill", "triangle", "hexagon", "cloud", "drop",
)
COMPANION_COLORS: Final[tuple[str, ...]] = (
- "#8b5cf6", "#c5dfd4", "#f2a65a", "#7ab6ef", "#ed91aa", "#b7cb78", "#bba7ed", "#79c7c4",
+ "#ffffff", "#875b36", "#dd2233", "#ff6801", "#ff9700", "#029858",
+ "#00a591", "#1174da", "#804ee1", "#df2a87", "#777777",
+ "#e8c89a", "#ff7a6b", "#ffd23f", "#9ccc3a", "#3cc4e8", "#b59cf0",
+ "#222222", "#8c1c3a", "#6b7a1f", "#0f5c63", "#1f3a8a", "#f49ac1",
)
@@ -30,6 +37,8 @@ class CompanionAppearance(BaseModel):
eyes: Literal["dots", "lines"] = "dots"
enabled: bool = True
accessories: dict[AccessorySlot, AccessoryId] = Field(default_factory=dict)
+ #: Wear this pet instead of the shape, everywhere the agent appears.
+ pet: PetId | None = None
sizeM: float = Field(default=0.5, ge=0.25, le=0.8)
followDistanceM: float = Field(default=1.0, ge=0.5, le=2.0)
@@ -42,9 +51,15 @@ def random_companion() -> dict[str, Any]:
color=rng.choice(COMPANION_COLORS),
eyes=rng.choice(("dots", "lines")), # type: ignore[arg-type]
).model_dump()
- # A new agent wears nothing yet; store it the way a bare look is stored.
+ return _bare(look)
+
+
+def _bare(look: dict[str, Any]) -> dict[str, Any]:
+ """Drop empty optional fields, so a plain look stays byte-identical to older builds."""
if not look.get("accessories"):
look.pop("accessories", None)
+ if look.get("pet") is None:
+ look.pop("pet", None)
return look
@@ -53,7 +68,4 @@ def validate_avatar_companion(avatar: dict[str, Any]) -> dict[str, Any]:
if "companion" not in avatar:
return avatar
companion = CompanionAppearance.model_validate(avatar["companion"]).model_dump()
- # A bare look stays byte-identical to what builds without accessories stored.
- if not companion["accessories"]:
- del companion["accessories"]
- return {**avatar, "companion": companion}
+ return {**avatar, "companion": _bare(companion)}
diff --git a/jarvis/society/credential_tool.py b/jarvis/society/credential_tool.py
new file mode 100644
index 0000000000..aa7c82b54b
--- /dev/null
+++ b/jarvis/society/credential_tool.py
@@ -0,0 +1,263 @@
+"""``society_request_credential``: an agent asks the person for a token or key.
+
+The chat shows a secure field ("GitHub token — stored safely, available as
+$GITHUB_TOKEN"); the person pastes the value and saves. The value goes from
+that field into the agent's credential vault (credentials.py) and never into
+the chat, a prompt or this tool's result: the agent learns only that the
+variable is set, and uses it by name inside its commands.
+
+A credential the agent already has is not asked for again unless the agent
+says the stored one stopped working (``replace``). A routine never asks: its
+chat is unattended, and the tool is not offered there.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any, Final
+
+from jarvis.agent_chat.credential_requests import (
+ CANCELLED,
+ DECLINED,
+ MAX_REQUESTS_PER_TURN,
+ SAVED,
+ CredentialSpec,
+ TooManyCredentialRequests,
+)
+from jarvis.core.protocols import ToolResult
+
+from . import credentials
+from .credentials import validate_env_name
+from .routine_runner import is_routine_session
+
+log = logging.getLogger(__name__)
+
+__all__ = ["CREDENTIAL_TOOL_NAME", "RequestCredentialTool", "usage_note"]
+
+CREDENTIAL_TOOL_NAME: Final[str] = "society_request_credential"
+
+def usage_note(env: str) -> str:
+ """How the agent uses a stored credential without ever seeing it."""
+ return (
+ f"The value is set as the environment variable {env} in every command you run with "
+ f"society_shell (only there, not in any other shell). Reference it by name inside the "
+ f"command (bash: ${env}, PowerShell: $env:{env}). Never print, echo, log, commit or "
+ "write the value anywhere, and never ask the user to paste it into the chat; Jarvis "
+ "masks it in command output."
+ )
+
+
+class RequestCredentialTool:
+ """Ask the person for a secret through a secure field in the chat."""
+
+ name: str = CREDENTIAL_TOOL_NAME
+ risk_tier: str = "safe"
+ is_action_tool: bool = False
+ description: str = (
+ "REQUIRED whenever a task needs a secret only the user can supply: an API key, bot or "
+ "access token, personal access token, password, client secret, webhook URL with a "
+ "secret or database connection string (for example a Discord bot token, an OpenAI "
+ "API key, a GitHub token). Call it as soon as you know the secret is needed, including "
+ "when you set up an app, bot, script or integration that needs one. It opens a secure "
+ "field in the user's chat; the value goes to the system credential store. It is the "
+ "ONLY way to obtain a secret: never ask for one in your reply or a question card, and "
+ "never send the user to settings, a plugin dialog, a .env file or the browser to "
+ "enter it. You NEVER see the value: it is set as the environment variable you name "
+ "in env for commands you run with society_shell; reference it by name inside them "
+ "($GITHUB_TOKEN) and have your code read it from the environment. "
+ "Choose a conventional UPPER_SNAKE_CASE name, give a short label, and say in "
+ "description what it is for and where the user creates it (with the minimal scopes), "
+ "in the user's language. This explanation appears before the secure field. "
+ "A credential you already have is not asked for again: the call returns 'available'. "
+ "Set replace only when the stored value was rejected by the service. While the user "
+ "has not answered yet, the call returns immediately with status 'waiting' and a "
+ "request_id. Do not poll, sleep, or keep the turn running for a token. The field "
+ "stays open until the user saves or cancels, even if the turn ends. Continue unrelated "
+ "work and respond to new messages while the user obtains the credential. If nothing "
+ "else can be done now, finish your reply so the user can ask questions. On a later "
+ "turn, wait_for checks the existing field's status immediately without opening another."
+ )
+ schema: dict[str, Any] = {
+ "type": "object",
+ "properties": {
+ "env": {
+ "type": "string",
+ "description": "Environment variable name, UPPER_SNAKE_CASE, e.g. GITHUB_TOKEN.",
+ },
+ "label": {
+ "type": "string",
+ "description": "What it is, 1-4 words, e.g. 'GitHub token'.",
+ },
+ "description": {
+ "type": "string",
+ "description": (
+ "One or two sentences: what you need it for and where the user creates "
+ "it, with the minimal permissions."
+ ),
+ },
+ "placeholder": {
+ "type": "string",
+ "description": "Optional hint of the expected shape, e.g. 'ghp_…'.",
+ },
+ "replace": {
+ "type": "boolean",
+ "description": "Ask again although a value is stored (the service rejected it).",
+ },
+ "wait_for": {
+ "type": "string",
+ "description": (
+ "Check a previously opened field on a later turn: the request_id from a "
+ "'waiting' result. Returns immediately. Send it alone; never poll in a loop."
+ ),
+ },
+ },
+ }
+
+ def __init__(self, runtime: Any, agent_id: str, *, session_id: str) -> None:
+ self._runtime = runtime
+ self._agent_id = agent_id
+ self._session_id = session_id
+
+ async def execute(self, args: dict[str, Any], ctx: Any) -> ToolResult:
+ wait_for = str(args.get("wait_for") or "").strip()
+ if wait_for:
+ return await self._check_existing(wait_for)
+ try:
+ env = validate_env_name(args.get("env"))
+ spec = CredentialSpec.build(
+ env,
+ args.get("label"),
+ args.get("description"),
+ args.get("placeholder"),
+ replace=bool(args.get("replace")),
+ )
+ if not spec.description:
+ raise ValueError(
+ "description is required: explain why the task needs this credential"
+ )
+ except ValueError as exc:
+ # A malformed request goes back to the agent as the error to fix.
+ return ToolResult(success=False, output=None, error=f"invalid request: {exc}")
+ vault = credentials.current_vault()
+ if vault is None:
+ return ToolResult(
+ success=False, output=None, error="The credential store is not available yet."
+ )
+ if not spec.replace and await asyncio.to_thread(vault.has, self._agent_id, env):
+ return ToolResult(
+ success=True,
+ output={"status": "available", "env": env, "note": usage_note(env)},
+ )
+ if is_routine_session(self._session_id):
+ return ToolResult(
+ success=False,
+ output={"status": "unattended", "env": env},
+ error=(
+ f"Routines run unattended and cannot ask for {env}. Report in your result "
+ "that the user has to provide it in your chat."
+ ),
+ )
+ service = self._runtime.chat_service()
+ if service is None or not callable(getattr(service, "open_credential_request", None)):
+ return _nobody(env)
+ agent_id = self._agent_id
+
+ async def save(value: str) -> None:
+ await credentials.save_requested_credential(agent_id, env, value, label=spec.label)
+
+ try:
+ request_id = await service.open_credential_request(
+ self._session_id, spec, save, asker=self._asker()
+ )
+ except TooManyCredentialRequests as exc: # Return the limit as the tool's visible error.
+ return ToolResult(
+ success=False,
+ output={"status": "limit", "env": env},
+ error=(
+ f"Credential field limit reached: {exc}. At most {MAX_REQUESTS_PER_TURN} "
+ "new fields can be opened per turn. Reuse existing pending fields."
+ ),
+ )
+ except RuntimeError as exc:
+ log.info("society credential: %s cannot ask here (%s)", agent_id, exc)
+ return _nobody(env)
+ return await self._status(service, request_id, env)
+
+ async def _check_existing(self, request_id: str) -> ToolResult:
+ service = self._runtime.chat_service()
+ try:
+ await service.restore_credential_requests(self._session_id)
+ spec = service.credential_request_spec(self._session_id, request_id)
+ except (AttributeError, KeyError):
+ # Unknown request id is reported to the agent in the result.
+ return ToolResult(
+ success=False,
+ output=None,
+ error=(
+ "No credential field with that id in this chat. Check whether the "
+ "credential is already available before opening a new field."
+ ),
+ )
+ return await self._status(service, request_id, spec.env)
+
+ async def _status(self, service: Any, request_id: str, env: str) -> ToolResult:
+ # The field owns its lifetime. Holding the turn here queues the very
+ # questions the person needs to ask before supplying the credential.
+ status = await service.wait_credential_request(self._session_id, request_id, 0)
+ if status is None:
+ return ToolResult(
+ success=True,
+ output={
+ "status": "waiting",
+ "request_id": request_id,
+ "env": env,
+ "note": (
+ "The user has not saved the credential yet. The field stays open "
+ "until they save or cancel, including after this turn ends. Continue "
+ "unrelated work if useful; otherwise finish your reply now so the user "
+ "can ask questions. Do not poll, sleep, or call this tool again in this "
+ "turn just to wait. On a later turn, check this field with "
+ f'{{"wait_for": "{request_id}"}} or use the stored credential by name.'
+ ),
+ },
+ )
+ if status == SAVED:
+ return ToolResult(
+ success=True, output={"status": SAVED, "env": env, "note": usage_note(env)}
+ )
+ if status == CANCELLED:
+ return ToolResult(
+ success=False,
+ output={"status": CANCELLED, "env": env},
+ error="The user stopped this turn. Do not continue the task.",
+ )
+ reason = (
+ "The user declined to provide it."
+ if status == DECLINED
+ else "The field closed without a value."
+ )
+ return ToolResult(
+ success=False,
+ output={"status": status, "env": env},
+ error=(
+ f"{reason} {env} is not set. Continue without it where you can, and tell the "
+ "user what needs it. Do not ask again in this turn."
+ ),
+ )
+
+ def _asker(self) -> str:
+ cached = getattr(self._runtime, "cached_agent", None)
+ agent = cached(self._agent_id) if callable(cached) else None
+ return str(getattr(agent, "name", "") or "")
+
+
+def _nobody(env: str) -> ToolResult:
+ return ToolResult(
+ success=False,
+ output={"status": "unattended", "env": env},
+ error=(
+ f"Nobody can enter {env} in this run. Report that the user has to provide it in "
+ "your chat."
+ ),
+ )
diff --git a/jarvis/society/credentials.py b/jarvis/society/credentials.py
new file mode 100644
index 0000000000..b6f788a5c8
--- /dev/null
+++ b/jarvis/society/credentials.py
@@ -0,0 +1,469 @@
+"""Credentials an agent asked the person for: stored safely, never shown to the agent.
+
+An agent that needs a token (a GitHub token, a Discord bot token, an API key
+for a service it scripts against) asks with ``society_request_credential``.
+The person pastes the value into a secure field in the chat; it goes straight
+from that field to the OS credential store through ``set_secret`` and never
+enters the chat log, the event stream, a prompt or a tool result.
+
+What the agent gets is the NAME: the value is set as the environment variable
+``<ENV>`` only in the commands it runs through ``society_shell``, which reads
+the store on every call (a credential saved mid-turn works in the very next
+command) and masks the value in the command's output BEFORE the model reads
+it. A CLI seat's own process never receives a credential: its own shell tool
+would hand ``printenv`` straight to the model, and a provider key there could
+move a subscription seat onto paid billing.
+
+Masking (``redact``) covers the plain value and its common encodings (base64,
+URL encoding). It protects against an accidental print, not an agent set on
+exfiltrating the value; every chat event of the agent is masked the same way.
+
+The index file per agent holds names, labels and timestamps only; the values
+live in the credential store under ``society_credential.<agent>.<ENV>``.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import json
+import logging
+import os
+import re
+import threading
+import time
+from collections.abc import Mapping
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Final, Protocol
+from urllib.parse import quote
+
+log = logging.getLogger(__name__)
+
+__all__ = [
+ "SecretStore",
+ "CredentialError",
+ "CredentialInfo",
+ "CredentialVault",
+ "MAX_VALUE_CHARS",
+ "validate_env_name",
+]
+
+#: A pasted secret longer than this is a file, not a token.
+MAX_VALUE_CHARS: Final[int] = 16_384
+
+#: Shorter values are never scrubbed from text: "1234" would eat every number.
+_MIN_REDACT_CHARS: Final[int] = 6
+
+_ENV_NAME: Final[re.Pattern[str]] = re.compile(r"^[A-Z][A-Z0-9_]{1,63}$")
+
+#: Names the shell or the app itself depends on: refused outright.
+_RESERVED_NAMES: Final[frozenset[str]] = frozenset(
+ {
+ "APPDATA", "COMSPEC", "HOME", "HOMEDRIVE", "HOMEPATH", "LANG", "LD_LIBRARY_PATH",
+ "LD_PRELOAD", "LOCALAPPDATA", "NODE_OPTIONS", "PATH", "PATHEXT", "PROGRAMDATA",
+ "PROGRAMFILES", "PSMODULEPATH", "PWD", "PYTHONHOME", "PYTHONIOENCODING", "PYTHONPATH",
+ "SHELL", "SYSTEMDRIVE", "SYSTEMROOT", "TEMP", "TERM", "TMP", "TMPDIR", "USER",
+ "USERNAME", "USERPROFILE", "WINDIR",
+ }
+)
+_RESERVED_PREFIXES: Final[tuple[str, ...]] = ("JARVIS", "DYLD_")
+
+_SLOT_PREFIX: Final[str] = "society_credential"
+_SAFE_ID: Final[re.Pattern[str]] = re.compile(r"[^A-Za-z0-9_.-]")
+
+
+class SecretStore(Protocol):
+ """Where the values live: the OS credential store in the app, a dict in tests."""
+
+ def get(self, slot: str) -> str | None: ...
+
+ def set(self, slot: str, value: str) -> bool: ...
+
+ def delete(self, slot: str) -> bool: ...
+
+
+class _OsSecrets:
+ """``jarvis.core.config``'s keyring → file-fallback store (AGENTS.md: get_secret only)."""
+
+ def get(self, slot: str) -> str | None:
+ from jarvis.core.config import get_secret
+
+ # "" turns off the ENV/.env fallback: only this slot's own value counts.
+ return get_secret(slot, "")
+
+ def set(self, slot: str, value: str) -> bool:
+ from jarvis.core.config import set_secret
+
+ return set_secret(slot, value)
+
+ def delete(self, slot: str) -> bool:
+ from jarvis.core.config import delete_secret
+
+ return delete_secret(slot)
+
+
+class CredentialError(ValueError):
+ """A safe, user-facing reason for a failed credential submission."""
+
+ def __init__(self, message: str, *, code: str = "invalid_token") -> None:
+ super().__init__(message)
+ self.code = code
+
+
+def normalize_value(value: str) -> str:
+ """Reject malformed input before validation or storage, without echoing it."""
+ value = value.strip()
+ if not value:
+ raise CredentialError("Enter the credential before saving.")
+ if len(value) > MAX_VALUE_CHARS:
+ raise CredentialError("The credential is too long.")
+ if any(ord(ch) < 32 or ord(ch) == 127 for ch in value):
+ raise CredentialError("A credential must be a single line without control characters.")
+ return value
+
+
+async def validate_token(env: str, value: str, *, label: str = "") -> None:
+ """Check identifiable Discord tokens only at the fixed, official endpoint.
+
+ Other credentials receive local input validation; arbitrary service names
+ and descriptions can never select a URL or send credentials elsewhere.
+ """
+ import httpx
+
+ value = normalize_value(value)
+ discord = ("DISCORD" in env.split("_") and "TOKEN" in env.split("_")) or (
+ "discord" in label.casefold() and "token" in label.casefold()
+ )
+ if not discord:
+ return
+ if not value.isascii() or any(ch.isspace() for ch in value):
+ raise CredentialError("Paste only the Discord bot token, without an authorization prefix.")
+ try:
+ # One user-triggered validation, no retry, redirect, paid generation or
+ # token in the URL. A context-managed client closes its resources.
+ async with asyncio.timeout(12):
+ async with httpx.AsyncClient(timeout=10, follow_redirects=False) as client:
+ response = await client.get(
+ "https://discord.com/api/v10/users/@me",
+ headers={"Authorization": f"Bot {value}"},
+ )
+ except (TimeoutError, httpx.TimeoutException):
+ raise CredentialError(
+ "Discord did not respond in time. Please try saving again.",
+ code="validation_timeout",
+ ) from None
+ except httpx.RequestError:
+ raise CredentialError(
+ "Discord could not be reached. Check your connection and try again.",
+ code="network_error",
+ ) from None
+ if response.status_code in {401, 403}:
+ raise CredentialError("Discord rejected this bot token. Check it or create a new one.")
+ if response.status_code != 200:
+ raise CredentialError(
+ "Discord could not validate the token right now. Please try again later.",
+ code="network_error",
+ )
+ try:
+ user = response.json()
+ except ValueError:
+ raise CredentialError(
+ "Discord returned an unreadable response. Please try again.", code="network_error",
+ ) from None
+ if not isinstance(user, dict) or not user.get("id") or user.get("bot") is not True:
+ raise CredentialError("Use a Discord bot token from the Bot page in the Developer Portal.")
+
+
+async def save_requested_credential(
+ agent_id: str, env: str, value: str, *, label: str = "",
+) -> None:
+ """The shared validation and storage path for new and restored fields."""
+ validate_env_name(env)
+ value = normalize_value(value)
+ vault = current_vault()
+ if vault is None:
+ raise CredentialError(
+ "The credential store is unavailable. Please try again shortly.", code="storage_failed",
+ )
+ await validate_token(env, value, label=label)
+ await asyncio.to_thread(vault.store, agent_id, env, value, label=label)
+
+
+def validate_env_name(raw: Any) -> str:
+ """The environment variable name ``raw`` asks for, or ``CredentialError``."""
+ name = str(raw or "").strip()
+ if not _ENV_NAME.fullmatch(name):
+ raise CredentialError(
+ "env must be an UPPER_SNAKE_CASE environment variable name such as GITHUB_TOKEN"
+ )
+ if name in _RESERVED_NAMES or name.startswith(_RESERVED_PREFIXES):
+ raise CredentialError(f"{name} is reserved for the system and cannot hold a credential")
+ return name
+
+
+def _encodings(value: str) -> list[str]:
+ """The forms a printed value commonly takes: plain, base64 (also with the
+ newline ``echo`` adds), URL-encoded."""
+ raw = value.encode("utf-8")
+ forms = {value, quote(value, safe="")}
+ for data in (raw, raw + b"\n"):
+ forms.add(base64.b64encode(data).decode("ascii"))
+ forms.add(base64.b64encode(data).decode("ascii").rstrip("="))
+ forms.add(base64.urlsafe_b64encode(data).decode("ascii").rstrip("="))
+ return [form for form in forms if len(form) >= _MIN_REDACT_CHARS]
+
+
+@dataclass(frozen=True, slots=True)
+class CredentialInfo:
+ """What the index knows about one stored credential: never its value."""
+
+ env: str
+ label: str
+ created_ms: int
+ updated_ms: int
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "env": self.env,
+ "label": self.label,
+ "created_ms": self.created_ms,
+ "updated_ms": self.updated_ms,
+ }
+
+
+class CredentialVault:
+ """Per-agent credentials: names in ``<root>/<agent>.json``, values in the store."""
+
+ def __init__(self, root: Path, secrets: SecretStore | None = None) -> None:
+ self._root = Path(root)
+ self._secrets: SecretStore = secrets or _OsSecrets()
+ self._lock = threading.Lock()
+ #: Values by agent, loaded once per process for scrubbing and env building.
+ self._values: dict[str, dict[str, str]] = {}
+ #: (needle, env) per agent, longest first, so a longer secret that
+ #: contains a shorter one is masked whole.
+ self._masks: dict[str, list[tuple[str, str]]] = {}
+ #: Bumped by every store/delete: a load that raced one is discarded.
+ self._generation = 0
+
+ # ------------------------------------------------------------- index
+
+ def _index_path(self, agent_id: str) -> Path:
+ # The name keeps no separator, but the joined path is still checked to
+ # stay inside the vault: an agent id must never address another file.
+ root = os.path.normpath(os.path.abspath(self._root))
+ path = os.path.normpath(os.path.join(root, f"{_SAFE_ID.sub('_', agent_id)}.json"))
+ if not path.startswith(root + os.sep):
+ raise ValueError("Agent credential index escapes the vault")
+ return Path(path)
+
+ def _read_index(self, agent_id: str) -> dict[str, CredentialInfo]:
+ path = self._index_path(agent_id)
+ try:
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ except FileNotFoundError: # An agent without saved credentials has no index yet.
+ return {}
+ except (OSError, ValueError):
+ log.warning("society credentials: unreadable index %s", path, exc_info=True)
+ return {}
+ rows: dict[str, CredentialInfo] = {}
+ for entry in raw.get("credentials", []) if isinstance(raw, dict) else []:
+ if not isinstance(entry, dict):
+ continue
+ env = str(entry.get("env") or "")
+ if not _ENV_NAME.fullmatch(env):
+ continue
+ rows[env] = CredentialInfo(
+ env=env,
+ label=str(entry.get("label") or "")[:120],
+ created_ms=int(entry.get("created_ms") or 0),
+ updated_ms=int(entry.get("updated_ms") or 0),
+ )
+ return rows
+
+ def _write_index(self, agent_id: str, rows: Mapping[str, CredentialInfo]) -> None:
+ path = self._index_path(agent_id)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ body = {"credentials": [rows[name].to_dict() for name in sorted(rows)]}
+ tmp = path.with_suffix(".json.tmp")
+ tmp.write_text(json.dumps(body, indent=2, ensure_ascii=False), encoding="utf-8")
+ tmp.replace(path)
+
+ @staticmethod
+ def _slot(agent_id: str, env: str) -> str:
+ return f"{_SLOT_PREFIX}.{_SAFE_ID.sub('_', agent_id)}.{env}"
+
+ # ------------------------------------------------------------- public
+
+ def list(self, agent_id: str) -> list[CredentialInfo]:
+ with self._lock:
+ rows = self._read_index(agent_id)
+ return [rows[name] for name in sorted(rows)]
+
+ def has(self, agent_id: str, env: str) -> bool:
+ """The value is really there: an index row whose store entry vanished is not."""
+ return env in self.values(agent_id)
+
+ def store(self, agent_id: str, env: str, value: str, *, label: str = "") -> CredentialInfo:
+ """Save ``value`` for ``agent_id`` as ``env``; raises ``CredentialError``."""
+ env = validate_env_name(env)
+ value = normalize_value(value)
+ now = int(time.time() * 1000)
+ with self._lock:
+ rows = self._read_index(agent_id)
+ previous = rows.get(env)
+ info = CredentialInfo(
+ env=env,
+ label=(label.strip() or (previous.label if previous else ""))[:120],
+ created_ms=previous.created_ms if previous else now,
+ updated_ms=now,
+ )
+ slot = self._slot(agent_id, env)
+ previous_value: str | None = None
+ attempted = False
+ try:
+ previous_value = self._secrets.get(slot)
+ attempted = True
+ if not self._secrets.set(slot, value) or self._secrets.get(slot) != value:
+ raise CredentialError(
+ "The credential store could not verify the saved value. Please try again.",
+ code="storage_failed",
+ )
+ rows[env] = info
+ self._write_index(agent_id, rows)
+ except Exception:
+ # Neither keyring errors nor their tracebacks are safe: a
+ # backend may include the submitted value in its exception.
+ if attempted:
+ try:
+ restored = (
+ self._secrets.set(slot, previous_value)
+ if previous_value is not None else self._secrets.delete(slot)
+ )
+ if not restored:
+ log.error("society credentials: rollback failed for %s", env)
+ except Exception:
+ log.error("society credentials: rollback failed for %s", env)
+ raise CredentialError(
+ "The credential could not be stored. Please try saving again.",
+ code="storage_failed",
+ ) from None
+ finally:
+ self._forget(agent_id)
+ log.info("society credentials: stored %s for %s", env, agent_id)
+ return info
+
+ def delete(self, agent_id: str, env: str) -> bool:
+ """Forget ``env`` for ``agent_id``; False when it was never stored."""
+ with self._lock:
+ rows = self._read_index(agent_id)
+ if env not in rows:
+ return False
+ if not self._secrets.delete(self._slot(agent_id, env)):
+ raise CredentialError("the credential store could not delete the value")
+ rows.pop(env)
+ self._write_index(agent_id, rows)
+ self._forget(agent_id)
+ log.info("society credentials: deleted %s for %s", env, agent_id)
+ return True
+
+ def _forget(self, agent_id: str) -> None:
+ """Drop the cached values (caller holds the lock); the next read reloads."""
+ self._generation += 1
+ self._values.pop(agent_id, None)
+ self._masks.pop(agent_id, None)
+
+ def values(self, agent_id: str) -> dict[str, str]:
+ """Every stored credential of ``agent_id`` by name (reads the store once)."""
+ while True:
+ with self._lock:
+ cached = self._values.get(agent_id)
+ if cached is not None:
+ return dict(cached)
+ rows = self._read_index(agent_id)
+ generation = self._generation
+ loaded = self._load(agent_id, rows)
+ with self._lock:
+ if generation != self._generation:
+ continue # a store or delete ran meanwhile: load again
+ self._values[agent_id] = loaded
+ self._masks[agent_id] = sorted(
+ ((form, env) for env, value in loaded.items() for form in _encodings(value)),
+ key=lambda pair: len(pair[0]),
+ reverse=True,
+ )
+ return dict(loaded)
+
+ def _load(self, agent_id: str, rows: Mapping[str, CredentialInfo]) -> dict[str, str]:
+ loaded: dict[str, str] = {}
+ for env in rows:
+ value = self._secrets.get(self._slot(agent_id, env))
+ if value:
+ loaded[env] = value
+ else:
+ log.warning(
+ "society credentials: %s for %s is missing from the store", env, agent_id
+ )
+ return loaded
+
+ def is_loaded(self, agent_id: str) -> bool:
+ return agent_id in self._values
+
+ def shell_env(self, agent_id: str) -> dict[str, str]:
+ """The variables an agent's own ``society_shell`` commands run with."""
+ return self.values(agent_id)
+
+ def redact(self, agent_id: str, text: str) -> str:
+ """``text`` with every credential value of ``agent_id`` masked (may read the store)."""
+ if not text or not self.values(agent_id):
+ return text
+ for needle, env in self._masks.get(agent_id, ()):
+ if needle in text:
+ text = text.replace(needle, f"[credential {env}]")
+ return text
+
+ def redact_payload(self, agent_id: str, payload: Any) -> Any:
+ """``payload`` with every string in it redacted; unchanged objects are reused."""
+ if not self.values(agent_id):
+ return payload
+ if isinstance(payload, str):
+ return self.redact(agent_id, payload)
+ if isinstance(payload, dict):
+ out: dict[Any, Any] = {}
+ changed = False
+ for key, item in payload.items():
+ new = self.redact_payload(agent_id, item)
+ changed = changed or new is not item
+ out[key] = new
+ return out if changed else payload
+ if isinstance(payload, list):
+ items = [self.redact_payload(agent_id, item) for item in payload]
+ changed = any(a is not b for a, b in zip(items, payload, strict=True))
+ return items if changed else payload
+ return payload
+
+
+_VAULT: CredentialVault | None = None
+_VAULT_ROOT: Path | None = None
+_VAULT_LOCK = threading.Lock()
+
+
+def vault_for(data_dir: Path) -> CredentialVault:
+ """The process's vault for the society data folder ``data_dir``."""
+ global _VAULT, _VAULT_ROOT
+ root = Path(data_dir) / "credentials"
+ with _VAULT_LOCK:
+ if _VAULT is None or _VAULT_ROOT != root:
+ _VAULT = CredentialVault(root)
+ _VAULT_ROOT = root
+ return _VAULT
+
+
+def current_vault() -> CredentialVault | None:
+ """The vault of the running society, or ``None`` before it started."""
+ from .runtime import current_runtime
+
+ rt = current_runtime()
+ data_dir = getattr(rt, "data_dir", None) if rt is not None else None
+ return vault_for(data_dir) if data_dir is not None else None
diff --git a/jarvis/society/proposals.py b/jarvis/society/proposals.py
index 9bd343120c..93adafc343 100644
--- a/jarvis/society/proposals.py
+++ b/jarvis/society/proposals.py
@@ -464,18 +464,20 @@ async def _apply_identity(rt: Any, agent: AgentRecord, payload: dict[str, Any])
key: list(agent.approval_rules.get(key, []))
for key in ("require_approval", "always_allow")
}
- derived_focus, derived_rules = rt.derive(
- fields.get("title", agent.title), fields.get("description", agent.description)
- )
- focus = list(agent.focus)
- for cap_id in derived_focus:
- if cap_id not in focus:
- focus.append(cap_id)
- if focus != list(agent.focus):
- fields["focus"] = focus
- has_rules = any(agent.approval_rules.get(k) for k in ("require_approval", "always_allow"))
- if not has_rules and derived_rules.get("require_approval"):
- fields["approval_rules"] = derived_rules
+ if "title" in fields or "description" in fields:
+ # Naming an existing role must not rewrite its focus or approval rules.
+ derived_focus, derived_rules = rt.derive(
+ fields.get("title", agent.title), fields.get("description", agent.description)
+ )
+ focus = list(agent.focus)
+ for cap_id in derived_focus:
+ if cap_id not in focus:
+ focus.append(cap_id)
+ if focus != list(agent.focus):
+ fields["focus"] = focus
+ has_rules = any(agent.approval_rules.get(k) for k in ("require_approval", "always_allow"))
+ if not has_rules and derived_rules.get("require_approval"):
+ fields["approval_rules"] = derived_rules
try:
updated = await rt.roster.update(agent.agent_id, fields)
except RosterError as exc: # reported on the outcome card as "failed"; nothing changed
diff --git a/jarvis/society/roster.py b/jarvis/society/roster.py
index ef2d0cb8d3..07d913d8fa 100644
--- a/jarvis/society/roster.py
+++ b/jarvis/society/roster.py
@@ -55,6 +55,7 @@ __all__ = [
"PAIR_SESSION_MARKER",
"canonical_session_id",
"conversation_session_id",
+ "has_placeholder_name",
"is_fresh",
"pair_session_id",
"slugify",
@@ -67,7 +68,7 @@ _MAX_TITLE: Final[int] = 120
_MAX_DESCRIPTION: Final[int] = 20_000
#: A visibly temporary name; the agent chooses its identity in conversation.
-_FRESH_NAME: Final[str] = "New Bot"
+_FRESH_NAME: Final[str] = "New Agent"
class RosterError(ValueError):
@@ -87,6 +88,19 @@ def slugify(name: str) -> str:
return text or "agent"
+def has_placeholder_name(agent: AgentRecord) -> bool:
+ """An unnamed creation still waiting for its first role-based name.
+
+ The random creation id distinguishes placeholders from explicit names.
+ Include the previous default so existing conversations can catch up.
+ """
+ return (
+ str(agent.tier) != "lead"
+ and re.fullmatch(r"agent-[0-9a-f]{8}", agent.agent_id) is not None
+ and re.fullmatch(r"New (?:Agent|Bot)(?: [1-9][0-9]*)?", agent.name) is not None
+ )
+
+
def is_fresh(agent: AgentRecord) -> bool:
"""An agent that does not know its role yet: no title and no description.
@@ -546,7 +560,7 @@ class Roster:
self._epoch += 1
async def _placeholder_name(self) -> str:
- """Use ``New Bot``, adding a number when another agent holds it."""
+ """Use ``New Agent``, adding a number when another agent holds it."""
candidate = _FRESH_NAME
number = 2
while await self._store.get_agent_row_by_name(candidate) is not None:
@@ -659,11 +673,7 @@ class Roster:
"workspace_dir": f"society/{agent_id}/workspace",
"wiki_namespace": f"society/{agent_id}/",
"approval_mode": (None if tier_value is Tier.LEAD else str(AgentApprovalMode.BYPASS)),
- "permission_ceiling": (
- str(PermissionCeiling.MONITOR)
- if tier_value is Tier.LEAD
- else str(PermissionCeiling.ASK)
- ),
+ "permission_ceiling": str(PermissionCeiling.ASK),
}
if tier_value is Tier.ORCHESTRATOR:
row["max_concurrent_runs"] = 3
diff --git a/jarvis/society/shell.py b/jarvis/society/shell.py
index 7458b7ab5e..16c7b2c489 100644
--- a/jarvis/society/shell.py
+++ b/jarvis/society/shell.py
@@ -20,6 +20,7 @@ import asyncio
import contextlib
import os
import time
+from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Final, Protocol
@@ -101,12 +102,23 @@ class LocalBackend:
bash / sh — the same pick the chat's folder tools make)."""
name: str = "local"
-
- async def run(self, command: str, *, cwd: Path, timeout_s: float) -> ShellResult:
+ #: ``run`` takes ``extra_env``: the agent's stored credentials (credentials.py).
+ accepts_env: bool = True
+
+ async def run(
+ self,
+ command: str,
+ *,
+ cwd: Path,
+ timeout_s: float,
+ extra_env: Mapping[str, str] | None = None,
+ ) -> ShellResult:
from jarvis.agent_chat.tools import shell_argv
timeout = max(1.0, min(float(timeout_s or DEFAULT_TIMEOUT_S), MAX_TIMEOUT_S))
env = dict(os.environ)
+ if extra_env:
+ env.update(extra_env)
env.setdefault("PYTHONIOENCODING", "utf-8")
env["CI"] = env.get("CI", "1")
env.setdefault("NO_COLOR", "1")
diff --git a/jarvis/society/surface.py b/jarvis/society/surface.py
index 917d6d448c..db278f921e 100644
--- a/jarvis/society/surface.py
+++ b/jarvis/society/surface.py
@@ -30,6 +30,7 @@ from jarvis.core.protocols import Tool, ToolResult
from jarvis.core.response_style import (
AGENT_QUESTION_GUIDANCE,
CONVERSATIONAL_RESPONSE_STYLE,
+ CREDENTIAL_GUIDANCE,
KEEP_GOING_ON_TOOL_FAILURE,
TASK_EXECUTION_GUIDANCE,
)
@@ -46,9 +47,16 @@ from .capabilities import CapabilityKind, CapabilityRow, capability_id_for_tool,
from .coding_threads import CodingThreadTool
from .communication import COMMUNICATION_GUIDANCE
from .conversation_tool import ConversationRecallTool, RoutineInvokeTool, RoutineListTool
+from .credential_tool import CREDENTIAL_TOOL_NAME, RequestCredentialTool
from .learning import RunLearnedSkillTool
from .memory import resolve_society_vault
-from .roster import PAIR_SESSION_MARKER, AgentRecord, canonical_session_id, is_fresh
+from .roster import (
+ PAIR_SESSION_MARKER,
+ AgentRecord,
+ canonical_session_id,
+ has_placeholder_name,
+ is_fresh,
+)
from .routine_runner import is_routine_session
from .runtime import current_runtime
from .share_tool import ShareTemplateTool
@@ -116,6 +124,10 @@ recommendation first with its reason. Unanswered questions take your recommendat
minutes. Never ask what you can infer or look up; decide everything else yourself.
- Shell: society_shell runs commands in YOUR workspace folder only (relative paths stay inside it; \
outside paths are refused). Destructive commands ask the user first.
+- Credentials: society_request_credential is the only way to get a token, key or password \
+from the user; the rules are under Credentials below. Which credentials are stored is listed \
+at the end of this briefing on every turn; that list is current, so trust it over anything \
+said earlier in the chat.
- Learning: after a finished task you may gain a learned skill of your own (listed \
above when present); run it with society_run_skill when a task matches.
- Memory: maintain your own USER.md (user profile and preferences, kind memory, target user) \
@@ -420,6 +432,9 @@ def society_tools(cfg: Any, brain: Any, session: Any) -> dict[str, Tool]:
# A routine runs unattended: it never gets a way to ask the user.
if not is_routine_session(session_id):
tools[ASK_USER_TOOL_NAME] = cast(Tool, AskUserTool(rt, agent_id, session_id=session_id))
+ tools[CREDENTIAL_TOOL_NAME] = cast(
+ Tool, RequestCredentialTool(rt, agent_id, session_id=session_id)
+ )
if rt.browser.is_installed() or rt.browser.live.model_resolver is not None:
from .browser.tool import BrowserTool
@@ -435,6 +450,9 @@ def society_tools(cfg: Any, brain: Any, session: Any) -> dict[str, Tool]:
return tools
+#: Tools that ask the person themselves: a gate card in front of them would ask twice.
+_UNGATED: Final[tuple[str, ...]] = (ASK_USER_TOOL_NAME, CREDENTIAL_TOOL_NAME)
+
#: Argument keys that name WHAT a mixed-action tool does (``gmail: send``).
_VERB_KEYS: Final[tuple[str, ...]] = ("action", "operation", "method", "command", "mode")
@@ -803,11 +821,12 @@ def society_tool_filter(session: Any) -> Callable[[dict[str, Tool]], dict[str, T
for name, tool in own.items()
# A legacy row still needs a live gate if its permissions change
# mid-turn. Asking the user is never gated behind its own card.
- if name != ASK_USER_TOOL_NAME
+ if name not in _UNGATED
}
)
- if ASK_USER_TOOL_NAME in own:
- ordered[ASK_USER_TOOL_NAME] = own[ASK_USER_TOOL_NAME]
+ for name in _UNGATED:
+ if name in own:
+ ordered[name] = own[name]
ordered.update(picked)
return ordered
@@ -864,12 +883,44 @@ async def society_system_extra(cfg: Any, brain: Any, session: Any) -> str:
zone = client_timezone.get() or "unknown; ask before scheduling wall-clock work"
context = f"\nClient timezone for this turn: {zone}."
+ context += await _credential_line(rt, agent.agent_id)
+ if is_routine_session(str(getattr(session, "session_id", "") or "")):
+ context += (
+ "\nThis is an unattended routine run: society_request_credential is not available. "
+ "If a credential is missing, report which one in your result."
+ )
return (
build_briefing(agent, catalog, roster, browser=browser, learned=learned, memory=memory)
+ context
)
+async def _credential_line(rt: Any, agent_id: str) -> str:
+ """The names of the agent's stored credentials, so it does not ask twice.
+
+ Also said when there are none: a resumed CLI conversation still remembers
+ a token it once saved, and without this line the agent reports a deleted
+ credential as stored instead of asking for it.
+ """
+ from .credentials import vault_for
+
+ try:
+ rows = await asyncio.to_thread(vault_for(rt.data_dir).list, agent_id)
+ except Exception: # noqa: BLE001 — an unreadable index costs this line, not the turn
+ log.warning("society: credential index unavailable for %s", agent_id, exc_info=True)
+ return ""
+ if not rows:
+ return (
+ "\nStored credentials: none. Ask with society_request_credential when a task "
+ "needs one."
+ )
+ names = ", ".join(f"{row.env} ({row.label})" if row.label else row.env for row in rows)
+ return (
+ "\nStored credentials, set as environment variables in society_shell (values are "
+ f"never shown to you): {names}."
+ )
+
+
# ------------------------------------------------------------------ briefing
@@ -886,16 +937,40 @@ def _kind_label(kind: CapabilityKind) -> str:
#: The introduction frame of a fresh agent (one-click creation): it has a
#: placeholder name and no role until its person says what it is for.
FRESH_AGENT_GUIDANCE = (
- "You were just created and have no role yet; your current name is a placeholder. "
+ "You have no assigned role yet. "
"If the person has not said what you are for, greet them in one or two sentences, say "
"you are new, and ask what you should take care of. As soon as they tell you, call "
- "society_propose_change with kind 'identity': a short fitting name, a one-line title, "
+ "society_propose_change with kind 'identity': a one-line title "
"and a description written as your standing instructions (goal, responsibilities, "
- "working style), in the person's language. Then start on the task they gave you. "
+ "working style), in the turn's output language. Then start on the task they gave you. "
+ "Keep your name unless the current naming guidance says it is a placeholder "
+ "or the person explicitly asks for another name. "
"Never invent a role the person did not describe."
)
+AGENT_NAMING_GUIDANCE = (
+ "Your current name is only a temporary creation label. Infer your ongoing responsibility "
+ "from the person's messages in this chat and your standing instructions. As soon as it "
+ "is clear, normally within the first one to three meaningful replies, replace the label "
+ "with a short descriptive name in the turn's output language. Do this on the first clear "
+ "role request; do not wait for three replies or for a request to rename you. "
+ "For example, an assigned Discord moderation role can be named Discord-Mod; apply the "
+ "same principle to any role, rather than picking an unrelated personal name. "
+ "Before your answer, call society_propose_change with kind=identity and payload={name: "
+ "the chosen name}. This first name saves automatically in your own direct user chat; "
+ "no confirmation question is needed. Check that the result says applied; if a name is "
+ "taken, choose another fitting name. If your role is already stored, send only name "
+ "and keep your standing instructions. If you also need to establish your first role, "
+ "you may include title and description. Never claim the name changed without a saved "
+ "result. A greeting, a quoted example, or a task assigned to another agent does not "
+ "establish your role: leave the label until your own responsibility is clear. "
+ "Do not create another agent to name yourself. After choosing a name, keep it through "
+ "ordinary follow-up tasks unless the person asks for a change."
+ " Honor an explicit instruction to keep the current name."
+)
+
+
def build_briefing(
agent: AgentRecord,
catalog: list[CapabilityRow],
@@ -937,12 +1012,15 @@ def build_briefing(
# API and CLI seats both consume this briefing. Put reply guidance and the
# keep-going rule before potentially long standing instructions so compact
# CLI identities retain them (a cancelled tool must not end the task).
+ if has_placeholder_name(agent):
+ parts.append("## Choosing your name\n" + AGENT_NAMING_GUIDANCE)
if is_fresh(agent):
parts.append("## You are new\n" + FRESH_AGENT_GUIDANCE)
parts.append("## Completing the user's task\n" + TASK_EXECUTION_GUIDANCE)
parts.append("## Acting and asking\n" + AGENT_QUESTION_GUIDANCE)
parts.append("## How to reply to the person\n" + CONVERSATIONAL_RESPONSE_STYLE)
parts.append("## When a tool fails\n" + KEEP_GOING_ON_TOOL_FAILURE)
+ parts.append("## Credentials\n" + CREDENTIAL_GUIDANCE)
if agent.description.strip():
parts.append("## Standing instructions\n" + agent.description.strip())
diff --git a/jarvis/tasks/approval_bridge.py b/jarvis/tasks/approval_bridge.py
index 588f2f3ba9..ffbe3f4b3d 100644
--- a/jarvis/tasks/approval_bridge.py
+++ b/jarvis/tasks/approval_bridge.py
@@ -17,9 +17,9 @@ Design (Option B from the integration analysis):
``ActionApproved`` straight onto the bus, which resolves the executor's
``wait()``. The full audit trail (Proposed -> Approved -> Executed) is
preserved — this answers the gate, it does not bypass it.
- - Anything NOT armed (read-only grants arm with an empty set; ungranted
- tools) is left to block and deny on timeout. Nothing is auto-approved
- by default.
+ - An armed task explicitly denies consequential calls outside its write
+ grants, including read-only grants. Unarmed traces use the executor's
+ app-wide policy.
Concurrency-safe: a persistent ``ActionApprovalRequired`` subscriber plus a
``trace_id -> grant`` map, so several tasks can run at once without a
@@ -31,7 +31,7 @@ from collections.abc import Iterable
from uuid import UUID
from jarvis.core.bus import EventBus
-from jarvis.core.events import ActionApprovalRequired, ActionApproved
+from jarvis.core.events import ActionApprovalRequired, ActionApproved, ActionDenied
class TaskAutoApprover:
@@ -46,9 +46,8 @@ class TaskAutoApprover:
def arm(self, trace_id: UUID, plugin_ids: Iterable[str], *, approved_by: str) -> None:
"""Pre-authorize ``plugin_ids`` for the turn identified by ``trace_id``.
- An empty ``plugin_ids`` is a valid no-op arm (a read-only task): it
- registers the trace but approves nothing, so the disarm() in the
- runner stays symmetric.
+ An empty ``plugin_ids`` is a read-only task: it registers the trace
+ and denies consequential calls until disarmed.
"""
self._active[trace_id] = (frozenset(plugin_ids), approved_by)
@@ -61,6 +60,13 @@ class TaskAutoApprover:
return
granted, approved_by = ctx
if not self._tool_is_granted(event.tool_name, granted):
+ await self._bus.publish(
+ ActionDenied(
+ trace_id=event.trace_id,
+ tool_name=event.tool_name,
+ reason="Scheduled task has no write grant for this tool",
+ )
+ )
return
await self._bus.publish(
ActionApproved(
diff --git a/jarvis/ui/jarvisbar/host.py b/jarvis/ui/jarvisbar/host.py
index 16f06c232b..0686d785e6 100644
--- a/jarvis/ui/jarvisbar/host.py
+++ b/jarvis/ui/jarvisbar/host.py
@@ -280,6 +280,7 @@ class _EchoBar:
def set_on_dictation_stop(self, cb: Any) -> None: ...
def set_feedback_publisher(self, cb: Any) -> None: ...
def set_on_show_window(self, cb: Any) -> None: ...
+ def set_on_visibility_changed(self, cb: Any) -> None: ...
def set_on_speaker_toggle(self, cb: Any) -> None: ...
def set_on_compose(self, cb: Any) -> None: ...
def set_on_notifications_toggle(self, cb: Any) -> None: ...
@@ -459,6 +460,11 @@ def _wire_surface_events(surface: Any) -> None:
lambda kind, payload: emit("feedback", kind=kind, payload=payload),
)
_call(surface, "set_on_show_window", lambda: emit("show_window"))
+ _call(
+ surface,
+ "set_on_visibility_changed",
+ lambda visible: emit("visibility_changed", visible=bool(visible)),
+ )
# The pet's pen: raising the window (and the new chat) happens in the
# parent, where the bus and the window live.
_call(surface, "set_on_compose", lambda: emit("compose"))
diff --git a/jarvis/ui/jarvisbar/interaction.py b/jarvis/ui/jarvisbar/interaction.py
index b93fe72d71..a8f90c17a4 100644
--- a/jarvis/ui/jarvisbar/interaction.py
+++ b/jarvis/ui/jarvisbar/interaction.py
@@ -18,7 +18,7 @@ from pathlib import Path
# Dependency-free import (no numpy/PIL): this module stays cheap enough for the
# IPC proxy and the pure unit tests.
-from jarvis.ui.jarvisbar.modes import DICTATION_MODES, NOTICE_MODES
+from jarvis.ui.jarvisbar.modes import ACTIVE_VOICE_MODES, DICTATION_MODES, NOTICE_MODES
# --------------------------------------------------------------------------- #
@@ -61,7 +61,7 @@ def resolve_click(
if action == "orb" and prompt_mode:
return "prompt_mode_toggle"
if action in ("orb", "call"):
- return "hangup" if mode in ("listen", "think", "speak") else "talk"
+ return "hangup" if mode in ACTIVE_VOICE_MODES else "talk"
return "none"
diff --git a/jarvis/ui/jarvisbar/modes.py b/jarvis/ui/jarvisbar/modes.py
index 1f36a9dd7c..608539a190 100644
--- a/jarvis/ui/jarvisbar/modes.py
+++ b/jarvis/ui/jarvisbar/modes.py
@@ -24,6 +24,22 @@ from __future__ import annotations
#: Voice-session modes, driven by the supervisor state machine.
VOICE_MODES: tuple[str, ...] = ("idle", "listen", "speak", "think")
+#: The call was accepted but the realtime transport is still negotiating
+#: (``SupervisorState.CONNECTING``): the wake word was heard, the opening words
+#: are being buffered, and the provider has not taken a single frame yet. On a
+#: hosted provider this lasts a moment, on a self-hosted one seconds — long
+#: enough that a bar showing the plain listening look claimed the user was
+#: already being heard.
+#:
+#: - ``connect`` — a loading loop until the provider accepts the session; the
+#: surface then plays a short "connected" flourish on its way to ``listen``.
+#: It is part of the call, so its close-X hangs up like the listening one.
+CONNECT_MODES: tuple[str, ...] = ("connect",)
+
+#: Modes during which a voice call is running: a click on the talk control or
+#: the phone hangs up instead of starting a new call.
+ACTIVE_VOICE_MODES: tuple[str, ...] = CONNECT_MODES + ("listen", "speak", "think")
+
#: Dictation modes. Dictation runs OUTSIDE the voice state machine (it raises no
#: ``SystemStateChanged``), so it gets its own modes rather than borrowing a
#: voice one — reusing ``listen``/``think`` would re-arm the close-X hang-up and
@@ -48,4 +64,4 @@ DICTATION_MODES: tuple[str, ...] = ("dictate", "dictate_transcribing")
NOTICE_MODES: tuple[str, ...] = ("notice",)
#: Every mode a surface accepts. Anything else is dropped by ``show()``.
-MODES: tuple[str, ...] = VOICE_MODES + DICTATION_MODES + NOTICE_MODES
+MODES: tuple[str, ...] = VOICE_MODES + CONNECT_MODES + DICTATION_MODES + NOTICE_MODES
diff --git a/jarvis/ui/jarvisbar/overlay.py b/jarvis/ui/jarvisbar/overlay.py
index cd674ca3ce..2d1f1bb15e 100644
--- a/jarvis/ui/jarvisbar/overlay.py
+++ b/jarvis/ui/jarvisbar/overlay.py
@@ -456,6 +456,9 @@ class JarvisBarOverlay:
# surfaces leave the opt-in gate off and retain their immediate behavior.
self._startup_gated = bool(startup_gated)
self._mode = "idle"
+ # The clock of the strokes bending into the loading loop while a call
+ # connects, and of their one-shot back out once it is connected.
+ self._connect_timeline = renderer.ConnectTimeline()
self._ext_level = 0.0
# perf_counter() of the last set_level() that carried real sound
# (>= AUDIBLE_LEVEL). Drives the sweep↔bars choice in _schedule_frame.
@@ -554,9 +557,17 @@ class JarvisBarOverlay:
# ------------------------------------------------------------------ #
# Surface API consumed by OrbBusBridge #
# ------------------------------------------------------------------ #
+ def _timeline(self) -> renderer.ConnectTimeline:
+ # ``__new__``-built test/hot-reload instances skip __init__.
+ timeline = getattr(self, "_connect_timeline", None)
+ if timeline is None:
+ timeline = self._connect_timeline = renderer.ConnectTimeline()
+ return timeline
+
def show(self, mode: str = "listen") -> None:
if mode not in renderer.MODES:
return
+ self._timeline().note_mode(self._mode, mode, time.perf_counter())
self._mode = mode
if self._root is None:
return
@@ -1397,6 +1408,9 @@ class JarvisBarOverlay:
# long before a file arrives, so the very frames the tick lives in
# are exactly the ones the fast path skips.
drop_visual = self._current_drop_visual()
+ # The "connected" flourish is time-dependent like the drop
+ # feedback: it joins the tick key and vetoes the static skip.
+ connect_look = self._timeline().look(self._mode, now)
# A stale level (the feed stopped without a zero) renders as
# silence, never as frozen dancing bars. getattr default:
# ``__new__``-built test/hot-reload instances skip __init__; a
@@ -1426,6 +1440,7 @@ class JarvisBarOverlay:
silent,
getattr(self, "_prompt_mode", False),
getattr(self, "_prompt_mode_paused", False),
+ connect_look,
)
if tick_key != self._static_tick_key:
self._static_tick_key = tick_key
@@ -1435,6 +1450,7 @@ class JarvisBarOverlay:
is_settled_static = (
static_capable
and drop_visual == renderer.DROP_STATE_NONE
+ and connect_look is None
and self._static_tick_count >= _IDLE_SETTLE_TICKS
)
@@ -1462,6 +1478,7 @@ class JarvisBarOverlay:
# skips __init__, and a missing stamp must read as "no drop
# in flight" rather than blow the frame away.
drop_elapsed=now - getattr(self, "_drop_visual_t0", 0.0),
+ connect_look=connect_look,
)
if getattr(self, "_mac_transparent", False):
# macOS: no color key — carry real per-pixel alpha instead.
diff --git a/jarvis/ui/jarvisbar/qt_overlay.py b/jarvis/ui/jarvisbar/qt_overlay.py
index e0932a0d1d..dc0ecd03ec 100644
--- a/jarvis/ui/jarvisbar/qt_overlay.py
+++ b/jarvis/ui/jarvisbar/qt_overlay.py
@@ -427,6 +427,8 @@ class QtJarvisBarOverlay:
self._user_size_scale = renderer.clamp_user_size(size_scale)
self._screen_scale = 1.0
self._mode = "idle"
+ # Clock of the strokes bending into the loading loop and back out.
+ self._connect_timeline = renderer.ConnectTimeline()
self._ext_level = 0.0
self._last_audible_t = 0.0
# Receipt stamp of the last set_level of ANY value: the frame loop
@@ -522,6 +524,7 @@ class QtJarvisBarOverlay:
def show(self, mode: str = "listen") -> None:
if mode not in renderer.MODES:
return
+ self._connect_timeline.note_mode(self._mode, mode, time.perf_counter())
self._mode = mode
self._desired_visible = not self._startup_gated and (
self._persistent_flag or mode != "idle"
@@ -799,6 +802,7 @@ class QtJarvisBarOverlay:
# Without the veto the confirmation is never seen: a settled idle
# bar skips exactly the frames the tick would live in.
drop_visual = self._current_drop_visual()
+ connect_look = self._connect_timeline.look(self._mode, now)
tick_key = (
effective_mode,
self._hovered,
@@ -809,6 +813,7 @@ class QtJarvisBarOverlay:
drop_visual,
self._prompt_mode,
self._prompt_mode_paused,
+ connect_look,
)
if tick_key != self._static_tick_key:
self._static_tick_key = tick_key
@@ -840,6 +845,7 @@ class QtJarvisBarOverlay:
prompt_mode_paused=self._prompt_mode_paused,
drop_state=drop_visual,
drop_elapsed=now - self._drop_visual_t0,
+ connect_look=connect_look,
)
rgba_frame = renderer.key_to_alpha(pil_frame)
input_mask_key = rgba_frame.getchannel("A").tobytes()
diff --git a/jarvis/ui/jarvisbar/renderer.py b/jarvis/ui/jarvisbar/renderer.py
index 2237558779..e90e88a2de 100644
--- a/jarvis/ui/jarvisbar/renderer.py
+++ b/jarvis/ui/jarvisbar/renderer.py
@@ -11,7 +11,13 @@ import math
import numpy as np
from PIL import Image, ImageDraw
-from jarvis.ui.jarvisbar.modes import DICTATION_MODES, MODES, NOTICE_MODES # noqa: F401
+from jarvis.ui.jarvisbar.modes import ( # noqa: F401
+ ACTIVE_VOICE_MODES,
+ CONNECT_MODES,
+ DICTATION_MODES,
+ MODES,
+ NOTICE_MODES,
+)
from ui.orb import controls
COLOR_KEY_RGB = (255, 0, 255)
@@ -187,8 +193,13 @@ def visual_mode(
something did not happen, and a stale level sample or an in-flight playback
must never be able to repaint it as listening or speaking — that would
replace the answer to the user's key press with a lie about the microphone.
+
+ ``connect`` passes through unchanged for the same reason: the opening words
+ are being buffered while the transport negotiates, so a live mic level is
+ real, but painting it as the equalizer would claim the provider is already
+ hearing them. The loading loop is the true state until it accepts the call.
"""
- if coarse_mode in NOTICE_MODES:
+ if coarse_mode in NOTICE_MODES or coarse_mode in CONNECT_MODES:
return coarse_mode
if coarse_mode == "idle":
return "idle"
@@ -203,6 +214,11 @@ def visual_mode(
return "speak"
+# The connect clock every bar surface keeps (``ui.orb.controls``).
+ConnectTimeline = controls.ConnectTimeline
+ConnectLook = controls.ConnectLook
+
+
def apply_display_scale(scale: float, user_size: float | None = None) -> None:
"""Keep the existing size preference and monitor calibration."""
global DISPLAY_SCALE, USER_SIZE_SCALE, WIN_W, WIN_H
@@ -252,23 +268,36 @@ class JarvisBarRenderer:
drop_state: str = DROP_STATE_NONE,
drop_elapsed: float = 0.0,
surface_mode: str | None = None,
+ connect_look: controls.ConnectLook | None = None,
) -> Image.Image:
actual = surface_mode or mode
motion = (
- "think"
+ "connect"
+ if mode in CONNECT_MODES
+ else "think"
if mode in ("think", "dictate_transcribing")
else "voice"
if mode in ("listen", "speak", "dictate")
else "rest"
)
+ phase = controls.indicator_phase(motion, t)
+ spin = 0
+ if connect_look is not None and actual in ACTIVE_VOICE_MODES:
+ # The surface's ConnectTimeline: the strokes bending into the
+ # loading loop, or the one-shot back out of it once connected.
+ motion, phase, spin = connect_look.motion, connect_look.phase, connect_look.spin
+ elif motion == "connect":
+ # No timeline (a bare render): the loop, fully bent, spinning on t.
+ phase, spin = controls.PET_MORPH_STEPS, controls.spin_step(t)
state = controls.PetStripState(
jarvis_bar=True,
mic_muted=muted,
speaker_muted=speaker_muted,
- active=actual in ("listen", "think", "speak"),
- level=controls.quantize_level(ext_level),
+ active=actual in ACTIVE_VOICE_MODES,
+ level=controls.quantize_level(ext_level) if motion in ("voice", "connect") else 0,
motion=motion,
- phase=controls.indicator_phase(motion, t),
+ phase=phase,
+ spin=spin,
hovered=hovered_action,
call_ring=call_ring,
)
diff --git a/jarvis/ui/jarvisbar/subprocess_overlay.py b/jarvis/ui/jarvisbar/subprocess_overlay.py
index 7bb8af582e..d71d803e6b 100644
--- a/jarvis/ui/jarvisbar/subprocess_overlay.py
+++ b/jarvis/ui/jarvisbar/subprocess_overlay.py
@@ -829,11 +829,8 @@ class SubprocessMascotOverlay(SubprocessBarOverlay):
self._send({"op": "set_visible", "visible": self._user_visible})
def toggle_visible(self) -> None:
- # The host knows whether the pet is on screen; this proxy only guesses.
- # The guess matters for one thing: a respawn replaying a hidden pet.
- # It is an optimistic mirror: the host sends no confirmation, so
- # ``pet_user_hidden`` (and the REST ``visible`` built on it) can drift
- # if the host drops the command; the next set_visible re-syncs it.
+ # Mirror optimistically; the host confirms the result and also reports
+ # right-click dismissal so settings and crash recovery stay in sync.
self._user_visible = not self._user_visible
self._send({"op": "toggle_visible"})
@@ -854,6 +851,9 @@ class SubprocessMascotOverlay(SubprocessBarOverlay):
self._send({"op": "set_notifications_enabled", "enabled": self._notifications_enabled})
def _dispatch_event(self, msg: dict[str, Any]) -> None:
+ if msg.get("event") == "visibility_changed":
+ self._user_visible = bool(msg.get("visible", True))
+ return
if msg.get("event") == "notify_toggle":
self._notifications_enabled = bool(msg.get("enabled", True))
return
diff --git a/jarvis/ui/pets/state_machine.py b/jarvis/ui/pets/state_machine.py
index e82c5fbfd9..f0bfbaf36f 100644
--- a/jarvis/ui/pets/state_machine.py
+++ b/jarvis/ui/pets/state_machine.py
@@ -39,6 +39,9 @@ _log = logging.getLogger(__name__)
#: plays the ``error`` one-shot on top of whatever the pet was doing.
MODE_STATES: dict[str, str] = {
"idle": "idle",
+ # The realtime transport is still negotiating: the pet is busy getting
+ # through, and turns to listening (with a nod) once the call is up.
+ "connect": "thinking",
"listen": "listening",
"dictate": "listening",
"think": "thinking",
diff --git a/jarvis/ui/server_client.py b/jarvis/ui/server_client.py
new file mode 100644
index 0000000000..9af30f0b59
--- /dev/null
+++ b/jarvis/ui/server_client.py
@@ -0,0 +1,149 @@
+"""A desktop window attached to a server it neither owns nor stops.
+
+The same origin serves the UI, API, streams and login. No provider credentials,
+databases, scheduler or agent process is created by this client. Native window
+chrome belongs to the client; server-side desktop routes retain their browser
+fallbacks instead of controlling a different machine's window.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import logging
+import random
+import sys
+import time
+from collections.abc import Callable
+
+from jarvis.core.server_endpoint import server_endpoint
+
+log = logging.getLogger(__name__)
+
+
+def wait_for_server(url: str, *, timeout_s: float = 120.0) -> None:
+ """Await the real backend through one reused HTTP client; never infer readiness."""
+ from urllib.parse import urlsplit
+
+ import httpx
+
+ # Only local startup uses the machine's control credential. Remote clients
+ # authenticate in the server's own page, never by forwarding this key.
+ server_endpoint(url)
+ if urlsplit(url).hostname not in {"localhost", "127.0.0.1", "::1"}:
+ raise ValueError("Readiness checks with local credentials require a loopback server.")
+ from jarvis.core.control_key import get_control_key
+ from jarvis.core.http_pool import SyncHttpClientPool
+
+ deadline = time.monotonic() + timeout_s
+ # An explicit transport disables environment proxy discovery in httpx:
+ # the loopback control credential must never travel through a proxy.
+ pool = SyncHttpClientPool(timeout_s=2.0, transport=httpx.HTTPTransport())
+ with pool.client() as client:
+ while time.monotonic() < deadline:
+ try:
+ key = get_control_key()
+ response = client.get(
+ url + "/api/agent-server/status",
+ headers={"Authorization": "Bearer " + key} if key else {},
+ )
+ if response.status_code == 200:
+ status = response.json()
+ if status.get("error"):
+ raise RuntimeError(status["error"])
+ if status.get("independent") and status.get("ready"):
+ return
+ except (httpx.HTTPError, ValueError):
+ log.debug("Waiting for the agent server to become ready")
+ time.sleep(random.uniform(0.2, 0.35)) # noqa: S311 - readiness jitter, not a secret
+ raise RuntimeError("The agent server did not become ready. Its log contains the startup error.")
+
+
+def local_server(*, port: int, spawn: Callable[..., bool] | None = None) -> str:
+ """Start or reuse the independently owned server of this instance."""
+ from filelock import FileLock
+
+ from jarvis.core import background_service as bg
+
+ # Concurrent clients must not start competing owners of the same databases.
+ lock_path = bg.marker_path().with_suffix(".launch.lock")
+ lock_path.parent.mkdir(parents=True, exist_ok=True)
+ with FileLock(str(lock_path), timeout=130):
+ marker = bg.read_marker() or {}
+ pid = bg.service_pid()
+ if pid is not None:
+ if not marker.get("persistent"):
+ raise RuntimeError(
+ "A legacy background session is still running. Stop it before enabling "
+ "the persistent server; opening a client will not interrupt its agents."
+ )
+ port = int(marker.get("port") or port)
+ else:
+ from jarvis.ui.desktop_app import _pid_alive, _read_meta
+
+ existing = _read_meta() or {}
+ existing_pid = int(existing.get("pid") or 0)
+ if existing_pid > 0 and _pid_alive(existing_pid):
+ raise RuntimeError(
+ "The integrated desktop is still running. Close it before starting "
+ "the independent server. No existing work has been stopped."
+ )
+ start = spawn or bg.spawn_service
+ if not start(after_pid=None, persistent=True, port=port):
+ raise RuntimeError("The independent agent server could not be started.")
+ url = server_endpoint(f"http://127.0.0.1:{port}")
+ wait_for_server(url)
+ marker = bg.read_marker() or {}
+ if not marker.get("persistent") or bg.service_pid() is None:
+ raise RuntimeError("The answering backend is not the persistent agent server.")
+ return url
+
+
+def run(url: str | None, *, port: int = 47821) -> int:
+ """Show the existing web application; closing this process only disconnects it."""
+ import webview
+
+ from jarvis.core.branding import PRODUCT_NAME
+ from jarvis.core.instance import current_instance
+ from jarvis.core.paths import user_data_dir
+
+ origin = server_endpoint(url or f"http://127.0.0.1:{port}")
+ # Origins and app instances have separate cookies, layouts and local data.
+ digest = hashlib.sha256(origin.encode()).hexdigest()[:20]
+ storage = user_data_dir() / "server-clients" / current_instance().name / digest
+ storage.mkdir(parents=True, exist_ok=True)
+ window = webview.create_window(
+ PRODUCT_NAME, origin if url else None,
+ html=None if url else _holding_page("Starting the agent server…"),
+ width=1280, height=850, min_size=(800, 600),
+ )
+
+ def connect_local() -> None:
+ try:
+ endpoint = local_server(port=port)
+ if not window.events.closed.is_set():
+ window.load_url(endpoint)
+ except Exception:
+ log.exception("The desktop client could not connect to its agent server")
+ if not window.events.closed.is_set():
+ window.load_html(_holding_page(
+ "The agent server could not start. Check the server log and reopen the client."
+ ))
+
+ webview.start(
+ func=connect_local if url is None else None,
+ gui="edgechromium" if sys.platform == "win32" else None,
+ private_mode=False,
+ storage_path=str(storage),
+ )
+ return 0
+
+
+def _holding_page(message: str) -> str:
+ import html
+
+ return (
+ '<!doctype html><meta charset="utf-8"><meta name="color-scheme" content="light dark">'
+ '<style>body{font:16px system-ui;display:grid;place-items:center;min-height:90vh;'
+ 'margin:0;padding:24px}p{max-width:42rem;text-align:center}</style>'
+ f"<p>{html.escape(message)}</p>"
+ )
diff --git a/jarvis/ui/web/agent_accounts_routes.py b/jarvis/ui/web/agent_accounts_routes.py
index b4f86e98e6..0ee10620da 100644
--- a/jarvis/ui/web/agent_accounts_routes.py
+++ b/jarvis/ui/web/agent_accounts_routes.py
@@ -21,6 +21,7 @@ import asyncio
import logging
import time
from typing import Any
+from uuid import UUID
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
@@ -143,6 +144,51 @@ async def usage(refresh: bool = False) -> dict[str, Any]:
return await asyncio.to_thread(_usage, refresh)
+class ResetConsumeRequest(BaseModel):
+ confirmed: bool = Field(
+ strict=True, description="Explicit confirmation to spend one earned reset."
+ )
+ idempotency_key: UUID = Field(description="Reuse this UUID when retrying the same reset.")
+ credit_id: str | None = Field(default=None, min_length=1, max_length=512)
+ account_key: str = Field(min_length=64, max_length=64)
+
+
+async def _reset_account(account_id: str) -> agent_accounts.AgentAccount:
+ account = await asyncio.to_thread(agent_accounts.resolve, account_id)
+ if account is None:
+ raise HTTPException(status_code=404, detail="That account no longer exists.")
+ return account
+
+
[email protected]("/{account_id}/resets", summary="Read earned usage resets for one subscription")
+async def list_account_resets(account_id: str) -> dict[str, Any]:
+ from jarvis.agent_resets import read_resets
+
+ return await read_resets(await _reset_account(account_id))
+
+
+ "/{account_id}/resets/consume",
+ summary="Confirm and use one earned subscription reset",
+ openapi_extra={"x-jarvis-dangerous": True},
+)
+async def consume_account_reset(account_id: str, req: ResetConsumeRequest) -> dict[str, Any]:
+ from jarvis.agent_resets import ResetError, consume_reset
+
+ if not req.confirmed:
+ raise HTTPException(status_code=422, detail="Confirm before consuming an earned reset.")
+ account = await _reset_account(account_id)
+ try:
+ return await consume_reset(
+ account,
+ attempt=str(req.idempotency_key),
+ credit_id=req.credit_id,
+ account_key=req.account_key,
+ )
+ except ResetError as exc:
+ raise HTTPException(status_code=409, detail={"code": exc.code}) from exc
+
+
@router.post("", summary="Add another subscription")
async def create_account(req: AccountCreateRequest) -> dict[str, Any]:
"""Register a subscription and mint its (still empty) config directory.
diff --git a/jarvis/ui/web/agent_chat_routes.py b/jarvis/ui/web/agent_chat_routes.py
index e7f6ee69b3..25801a0d5d 100644
--- a/jarvis/ui/web/agent_chat_routes.py
+++ b/jarvis/ui/web/agent_chat_routes.py
@@ -57,6 +57,7 @@ from jarvis.agent_chat import attachments as chat_attachments
from jarvis.agent_chat import runner_cli, typeahead
from jarvis.agent_chat.catalog import claude_code_models, offers, rows_for
from jarvis.agent_chat.control_types import CommandRequest, CommandResult
+from jarvis.agent_chat.credential_requests import CredentialRequestBusy
from jarvis.agent_chat.effort import normalize_effort
from jarvis.agent_chat.events import make_event
from jarvis.agent_chat.permissions import (
@@ -76,6 +77,7 @@ from jarvis.agent_chat.service import (
)
from jarvis.agent_chat.surface_kits import kit_for
from jarvis.agent_chat.tools import shell_label
+from jarvis.society.credentials import CredentialError
log = logging.getLogger(__name__)
@@ -264,6 +266,14 @@ class ApprovalBody(BaseModel):
decision: str
+class CredentialBody(BaseModel):
+ #: The secret the person pasted. It goes to the agent's vault and nowhere
+ #: else: never logged, never echoed, never part of a chat event. No length
+ #: constraint here on purpose: a validation error would echo the value in
+ #: its 422 body; the vault refuses a bad value with a 400 that names none.
+ value: Any = Field(default=None, repr=False)
+
+
class QuestionAnswerBody(BaseModel):
#: Which question of the card's series this answers.
index: int = 0
@@ -997,7 +1007,7 @@ def create_session(body: CreateSessionBody, request: Request) -> dict[str, Any]:
@router.get("/sessions/{session_id}")
-def get_session(
+async def get_session(
session_id: str,
request: Request,
tail: int | None = Query(
@@ -1008,9 +1018,14 @@ def get_session(
session = svc.store.get_session(session_id)
if session is None:
raise HTTPException(status_code=404, detail="session not found")
+ if session.surface in ("jarvis", "society"):
+ from jarvis.agent_chat.send_queue import close_orphans
+
+ await close_orphans(svc, session_id)
d = session.to_dict()
d["running"] = svc.is_running(session_id)
- return {"session": d, "events": svc.store.list_events(session_id, tail=tail)}
+ events = await asyncio.to_thread(svc.store.list_events, session_id, tail=tail)
+ return {"session": d, "events": events}
@router.get(
@@ -1157,6 +1172,7 @@ async def delete_session(session_id: str, request: Request) -> dict[str, Any]:
raise HTTPException(status_code=404, detail="session not found")
await svc.controls.pause(session_id, "Session deleted")
await svc.controls._clear_saved_native(session_id)
+ await svc.discard_credential_requests(session_id)
if not svc.store.delete_session(session_id):
raise HTTPException(status_code=404, detail="session not found")
svc.controls.store.delete(session_id)
@@ -1312,6 +1328,67 @@ async def skip_question(session_id: str, question_id: str, request: Request) ->
return {"ok": True, "question_id": question_id}
+ "/sessions/{session_id}/credentials/{request_id}",
+ summary="Save the secret an agent asked for in its secure credential field",
+ openapi_extra={"x-jarvis-dangerous": True},
+)
+async def submit_credential(
+ session_id: str, request_id: str, body: CredentialBody, request: Request
+) -> dict[str, Any]:
+ svc = _service(request)
+ if not isinstance(body.value, str):
+ raise HTTPException(status_code=400, detail={
+ "code": "invalid_token", "message": "Enter a credential as text before saving.",
+ })
+ try:
+ ok = await svc.submit_credential(session_id, request_id, body.value)
+ except CredentialRequestBusy:
+ raise HTTPException(status_code=409, detail={
+ "code": "request_busy",
+ "message": "The credential is still being checked. Please wait.",
+ }) from None
+ except CredentialError as exc:
+ status = {
+ "network_error": 503, "validation_timeout": 504, "storage_failed": 503,
+ }.get(exc.code, 400)
+ raise HTTPException(status_code=status, detail={
+ "code": exc.code, "message": str(exc),
+ }) from None
+ except Exception:
+ # A third-party storage exception may include the token. Never emit
+ # its message or traceback into logs, HTTP bodies or the agent timeline.
+ log.warning("agent chat: credential submission failed for %s", session_id)
+ raise HTTPException(status_code=503, detail={
+ "code": "storage_failed", "message": "The credential could not be stored. Try again.",
+ }) from None
+ if not ok:
+ raise HTTPException(status_code=410, detail={
+ "code": "request_closed", "message": "This credential field is already closed.",
+ })
+ return {"ok": True, "request_id": request_id, "status": "saved"}
+
+
+ "/sessions/{session_id}/credentials/{request_id}/decline",
+ summary="Close an agent's credential field without providing the secret",
+)
+async def decline_credential(session_id: str, request_id: str, request: Request) -> dict[str, Any]:
+ svc = _service(request)
+ try:
+ ok = await svc.decline_credential(session_id, request_id)
+ except CredentialRequestBusy:
+ raise HTTPException(status_code=409, detail={
+ "code": "request_busy",
+ "message": "The credential is still being checked. Please wait.",
+ }) from None
+ if not ok:
+ raise HTTPException(status_code=410, detail={
+ "code": "request_closed", "message": "This credential field is already closed.",
+ })
+ return {"ok": True, "request_id": request_id, "status": "declined"}
+
+
@router.post(
"/sessions/{session_id}/plan",
summary="Answer a coding agent's plan card: build it, or keep planning",
@@ -1501,6 +1578,10 @@ async def session_stream(ws: WebSocket, session_id: str) -> None:
# Subscribe BEFORE reading the snapshot so nothing falls between the two.
q = svc.subscribe(session_id)
try:
+ if session.surface in ("jarvis", "society"):
+ from jarvis.agent_chat.send_queue import close_orphans
+
+ await close_orphans(svc, session_id)
events = svc.store.list_events(session_id, after_seq=after)
d = session.to_dict()
d["running"] = svc.is_running(session_id)
diff --git a/jarvis/ui/web/agent_server.py b/jarvis/ui/web/agent_server.py
new file mode 100644
index 0000000000..d32c5d5d35
--- /dev/null
+++ b/jarvis/ui/web/agent_server.py
@@ -0,0 +1,31 @@
+"""Start durable agent services without waiting for a browser to open their pages."""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any
+
+log = logging.getLogger(__name__)
+
+
+async def start_agents(state: Any) -> None:
+ """Restore chats and deliveries after HTTP is ready; never call a model to warm it."""
+ from jarvis.ui.web.agent_chat_routes import _service_from_state
+
+ try:
+ service = await asyncio.to_thread(_service_from_state, state)
+ if service is None:
+ raise RuntimeError("The agent chat service is unavailable.")
+ # The route helper owns the singleton and its startup synchronization.
+ runtime = getattr(state, "society", None) or state.society_factory()
+ state.society = runtime
+ await runtime.ensure_started()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ log.exception("Independent agent server could not restore agent services")
+ state.agent_server_error = "Agent services could not start. Check the server log."
+ return
+ state.agent_server_ready = True
+ log.info("Independent agent server restored chats, routines and delivery")
diff --git a/jarvis/ui/web/agent_server_routes.py b/jarvis/ui/web/agent_server_routes.py
new file mode 100644
index 0000000000..66b8ef4cd2
--- /dev/null
+++ b/jarvis/ui/web/agent_server_routes.py
@@ -0,0 +1,34 @@
+"""Read independent server readiness and explicitly stop its owner."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter, HTTPException, Request
+
+router = APIRouter(prefix="/api/agent-server", tags=["agent-server"])
+
+
[email protected]("/status", operation_id="agent_server_status")
+def status(request: Request) -> dict[str, Any]:
+ """Read agent server ownership and startup state without calling a model."""
+ state = request.app.state
+ return {
+ "independent": bool(getattr(state, "agent_server_mode", False)),
+ "ready": bool(getattr(state, "agent_server_ready", False)),
+ "error": getattr(state, "agent_server_error", None),
+ }
+
+
+ "/stop", operation_id="agent_server_stop",
+ openapi_extra={"x-jarvis-dangerous": True},
+)
+def stop(request: Request) -> dict[str, bool]:
+ """Explicitly stop the independent server and its agents."""
+ state = request.app.state
+ callback = getattr(state, "agent_server_stop", None)
+ if not getattr(state, "agent_server_mode", False) or not callable(callback):
+ raise HTTPException(409, "This backend is not an independent agent server.")
+ callback()
+ return {"stopping": True}
diff --git a/jarvis/ui/web/agentic_ide_routes.py b/jarvis/ui/web/agentic_ide_routes.py
index f90be418f9..a7b32ec0da 100644
--- a/jarvis/ui/web/agentic_ide_routes.py
+++ b/jarvis/ui/web/agentic_ide_routes.py
@@ -4674,7 +4674,14 @@ async def _analyze_drops(
return []
[email protected]("/fanout", summary="Run one task across several coding agents")
+# Not an app action (``x-jarvis-exclude``): a model reaches coding panes through
+# workspace-orchestrate, which addresses them by ID and keeps a delivery receipt.
+# These routes pick a pane by call-sign in whichever workspace is in front.
+ "/fanout",
+ summary="Run one task across several coding agents",
+ openapi_extra={"x-jarvis-exclude": True},
+)
async def fanout(request: Request, req: FanOutRequest) -> dict:
"""Brief a fleet: open the panes if asked, divide the work, deliver it.
@@ -4908,7 +4915,11 @@ def _unknown_terminal_detail(registry: object, wanted: str) -> str:
)
[email protected]("/terminals/{name}/prompt", summary="Send a prompt to one terminal")
+ "/terminals/{name}/prompt",
+ summary="Send a prompt to one terminal",
+ openapi_extra={"x-jarvis-exclude": True}, # see /fanout
+)
async def terminal_prompt(
request: Request, name: str, req: PromptRequest, workspace: str | None = None
) -> dict:
diff --git a/jarvis/ui/web/appshot_routes.py b/jarvis/ui/web/appshot_routes.py
index 41004b0323..de64099859 100644
--- a/jarvis/ui/web/appshot_routes.py
+++ b/jarvis/ui/web/appshot_routes.py
@@ -5,6 +5,9 @@ Endpoints (mounted by the WebServer in ``_build_app()``):
GET /api/appshot/settings → switches, shortcut state, readiness.
PUT /api/appshot/settings → change one or more switches.
POST /api/appshot/take → take one appshot now (window or area).
+ POST /api/appshot/recording/{id}/export → trim / speed up a recording as a new one.
+ POST /api/appshot/recording/{id}/save → copy a recording into Downloads.
+ POST /api/appshot/recording/{id}/open-editor → play and trim it in the editor window.
GET /api/appshot/latest → metadata of the last appshot.
GET /api/appshot/latest/image → its picture (never cached).
PUT /api/appshot/latest/image → replace it with the editor's version and
@@ -52,6 +55,10 @@ class SettingsPatch(BaseModel):
hotkey: str | None = Field(default=None, max_length=64)
region_hotkey: str | None = Field(default=None, max_length=64)
recording_hotkey: str | None = Field(default=None, max_length=64)
+ recording_resolution: Literal["720p", "1080p", "1440p", "2160p", "native"] | None = None
+ recording_fps: Literal[30, 60, 120] | None = None
+ recording_bitrate_mbps: int | None = Field(default=None, ge=1, le=100, strict=True)
+ recording_system_audio: bool | None = None
target: Literal["auto", "message", "voice"] | None = None
sound: bool | None = None
effect: bool | None = None
@@ -124,6 +131,7 @@ def _capability() -> dict[str, Any]:
def _settings_payload() -> dict[str, Any]:
from jarvis.appshot.hotkey import configured_hotkeys, get_shortcut # noqa: PLC0415
+ from jarvis.appshot.recording_options import RecordingOptions
from jarvis.core.config import load_config # noqa: PLC0415
config = load_config()
@@ -141,6 +149,8 @@ def _settings_payload() -> dict[str, Any]:
"hotkey": hotkeys["window"],
"region_hotkey": hotkeys["region"],
"recording_hotkey": hotkeys["recording"],
+ **{f"recording_{key}": value
+ for key, value in RecordingOptions.from_config(block).model_dump().items()},
"target": block.target,
"sound": bool(block.sound),
"effect": bool(block.effect),
@@ -293,6 +303,55 @@ async def recording_video(recording_id: str):
headers=_NO_STORE)
+class ClipRequest(BaseModel):
+ start_s: float = Field(ge=0)
+ end_s: float = Field(gt=0)
+ speed: float = Field(default=1.0, gt=0, le=4)
+
+
[email protected]("/recording/{recording_id}/export")
+async def export_recording_clip(recording_id: str, body: ClipRequest) -> dict[str, Any]:
+ """Write the video editor's trim and speed as a new recording (the original stays)."""
+ from jarvis.appshot.video_edit import ClipError, export_clip
+
+ try:
+ clip_id = await asyncio.to_thread(
+ export_clip, recording_id, body.start_s, body.end_s, body.speed
+ )
+ except FileNotFoundError as exc:
+ raise HTTPException(status_code=404, detail="The recording is not available.") from exc
+ except ClipError as exc:
+ raise HTTPException(status_code=400, detail=str(exc)) from exc
+ except ImportError as exc:
+ raise HTTPException(
+ status_code=503, detail="Editing videos requires the desktop video packages."
+ ) from exc
+ return {"id": clip_id}
+
+
[email protected]("/recording/{recording_id}/save")
+async def save_recording_copy(recording_id: str) -> dict[str, Any]:
+ """Copy a recording into the Downloads folder of the machine running Jarvis."""
+ from jarvis.appshot.recording_cards import save_recording
+
+ try:
+ path = await asyncio.to_thread(save_recording, recording_id)
+ except OSError as exc:
+ log.warning("appshot: recording could not be saved to Downloads", exc_info=True)
+ raise HTTPException(status_code=500, detail="The video could not be saved.") from exc
+ if path is None:
+ raise HTTPException(status_code=404, detail="The recording is not available.")
+ return {"path": str(path), "filename": path.name}
+
+
[email protected]("/recording/{recording_id}/open-editor")
+async def open_recording_editor(recording_id: str) -> dict[str, Any]:
+ """Open the video editor window on a recording. ``window: false`` = not possible here."""
+ from jarvis.appshot.editor_window import open_recording_window
+
+ return {"window": await open_recording_window(recording_id)}
+
+
@router.post("/take")
async def take(request: Request, body: TakeRequest | None = None) -> dict[str, Any]:
"""Take one appshot (front window or a selected area), like the shortcuts."""
@@ -542,8 +601,8 @@ def _library_id(shot_id: str) -> str:
@router.get("/library")
async def library_list() -> dict[str, Any]:
- """Every kept appshot and edit, newest first — no pixels."""
- from jarvis.appshot import library # noqa: PLC0415
+ """Every kept screenshot, edit and recording, newest first — no pixels."""
+ from jarvis.appshot import media_library as library # noqa: PLC0415
items = await asyncio.to_thread(library.list_items)
return {
@@ -556,12 +615,20 @@ async def library_list() -> dict[str, Any]:
async def library_image(
shot_id: str, variant: LibraryVariant = "original", thumb: bool = False
) -> Response:
- """One kept picture; ``thumb=1`` sends the gallery's small JPEG."""
- from jarvis.appshot import library # noqa: PLC0415
+ """One kept image or video; ``thumb=1`` sends a small preview."""
+ from fastapi.responses import FileResponse
+
+ from jarvis.appshot import media_library as library # noqa: PLC0415
item = await asyncio.to_thread(library.get_item, _library_id(shot_id), variant)
if item is None:
raise HTTPException(status_code=404, detail="That appshot is no longer kept.")
+ if item.mime == "video/mp4" and not thumb:
+ # Stream with byte-range support for seeking; never read an entire
+ # recording into the web server's memory.
+ return FileResponse(
+ item.path, media_type=item.mime, headers={"Cache-Control": "no-store"},
+ )
if thumb:
data, mime = await asyncio.to_thread(library.thumbnail, item)
else:
@@ -591,8 +658,8 @@ async def library_open(shot_id: str, body: LibraryOpenRequest | None = None) ->
@router.delete("/library/{shot_id}", openapi_extra={"x-jarvis-dangerous": True})
async def library_delete(shot_id: str, variant: LibraryVariant = "original") -> dict[str, Any]:
- """Delete an edit only, or — for the original — the whole appshot."""
- from jarvis.appshot import library # noqa: PLC0415
+ """Delete an edit, an original screenshot with its edit, or one recording."""
+ from jarvis.appshot import media_library as library # noqa: PLC0415
removed = await asyncio.to_thread(library.delete, _library_id(shot_id), variant)
if not removed:
@@ -602,7 +669,7 @@ async def library_delete(shot_id: str, variant: LibraryVariant = "original") ->
@router.delete("/library", openapi_extra={"x-jarvis-dangerous": True})
async def library_clear() -> dict[str, Any]:
- """Delete every kept appshot and edit."""
- from jarvis.appshot import library # noqa: PLC0415
+ """Delete all kept screenshots, edits and finalized recordings."""
+ from jarvis.appshot import media_library as library # noqa: PLC0415
return {"ok": True, "removed": await asyncio.to_thread(library.clear)}
diff --git a/jarvis/ui/web/frontend/scripts/bench-realtime-audio.html b/jarvis/ui/web/frontend/scripts/bench-realtime-audio.html
new file mode 100644
index 0000000000..401a89281f
--- /dev/null
+++ b/jarvis/ui/web/frontend/scripts/bench-realtime-audio.html
@@ -0,0 +1,20 @@
+<!doctype html>
+<html lang="en">
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Local realtime audio preparation measurement</title>
+<style>
+ body { max-width: 900px; margin: 40px auto; padding: 16px; font: 16px system-ui; color-scheme: light dark; }
+ button { font: inherit; padding: 10px 14px; margin: 0 8px 12px 0; }
+ pre { white-space: pre-wrap; font-size: 13px; }
+</style>
+<h1>Local realtime audio preparation</h1>
+<p>Real AudioContext and production worklet. Synthetic input stays local and silent.
+No microphone, peer connection, credentials or provider session.</p>
+<button id="cold">Measure cold audio setup</button>
+<button id="prepare">Prepare suspended audio</button>
+<button id="warm" disabled>Measure prepared audio setup</button>
+<p id="status" role="status">Ready</p>
+<pre id="results">[]</pre>
+<script type="module" src="./bench-realtime-audio.ts"></script>
+</html>
diff --git a/jarvis/ui/web/frontend/scripts/bench-realtime-audio.ts b/jarvis/ui/web/frontend/scripts/bench-realtime-audio.ts
new file mode 100644
index 0000000000..66fb5a0bef
--- /dev/null
+++ b/jarvis/ui/web/frontend/scripts/bench-realtime-audio.ts
@@ -0,0 +1,116 @@
+/** Run via Vite's /scripts/bench-realtime-audio.html in a real visible browser. */
+import {
+ acquireRealtimeAudio,
+ registerRealtimeAudioPreparation,
+ releaseRealtimeAudio,
+ type PreparedRealtimeAudio,
+} from "../src/lib/realtimeAudioPreparation";
+
+const cold = document.querySelector<HTMLButtonElement>("#cold")!;
+const prepare = document.querySelector<HTMLButtonElement>("#prepare")!;
+const warm = document.querySelector<HTMLButtonElement>("#warm")!;
+const status = document.querySelector<HTMLElement>("#status")!;
+const results = document.querySelector<HTMLElement>("#results")!;
+const samples: Record<string, number | string | boolean>[] = [];
+let held: PreparedRealtimeAudio | null = null;
+let unregister: (() => void) | null = null;
+
+function buttons(busy: boolean): void {
+ cold.disabled = prepare.disabled = busy;
+ warm.disabled = busy || !held;
+}
+
+async function dispose(): Promise<void> {
+ unregister?.();
+ unregister = null;
+ const entry = held;
+ held = null;
+ if (entry) await releaseRealtimeAudio(entry);
+}
+
+async function run(prepared: boolean): Promise<void> {
+ buttons(true);
+ let entry: PreparedRealtimeAudio | null = null;
+ let source: ConstantSourceNode | null = null;
+ let capture: AudioWorkletNode | null = null;
+ let sink: GainNode | null = null;
+ try {
+ if (!prepared) await dispose();
+ const started = performance.now();
+ entry = held ?? acquireRealtimeAudio();
+ held = null;
+ const context = entry.context;
+ const sample: Record<string, number | string | boolean> = {
+ mode: prepared ? "prepared" : "cold",
+ used_prepared_context: entry.prepared,
+ state_before_activation: context.state,
+ acquire_ms: performance.now() - started,
+ };
+ await entry.loaded;
+ sample.module_ready_ms = performance.now() - started;
+ await context.resume();
+ sample.running_ms = performance.now() - started;
+ capture = new AudioWorkletNode(context, "pcm-capture");
+ source = context.createConstantSource();
+ source.offset.value = 0.01;
+ sink = context.createGain();
+ sink.gain.value = 0;
+ source.connect(capture);
+ capture.connect(sink);
+ sink.connect(context.destination);
+ const rendered = new Promise<void>((resolve, reject) => {
+ const timer = window.setTimeout(() => reject(new Error("No rendered capture packet")), 5000);
+ capture!.port.onmessage = (event: MessageEvent) => {
+ if (!(event.data instanceof ArrayBuffer)) return;
+ window.clearTimeout(timer);
+ resolve();
+ };
+ });
+ source.start();
+ await rendered;
+ sample.first_rendered_packet_ms = performance.now() - started;
+ sample.remote_resource_requests = performance.getEntriesByType("resource")
+ .filter(resource => new URL(resource.name, location.href).origin !== location.origin).length;
+ samples.push(sample);
+ results.textContent = JSON.stringify(samples, null, 2);
+ status.textContent = "Measured a real local audio render packet; no microphone was opened.";
+ } catch (error) {
+ status.textContent = `Measurement failed: ${String(error)}`;
+ } finally {
+ source?.stop();
+ source?.disconnect();
+ capture?.disconnect();
+ sink?.disconnect();
+ unregister?.();
+ unregister = null;
+ if (entry) await releaseRealtimeAudio(entry);
+ buttons(false);
+ }
+}
+
+cold.addEventListener("click", () => void run(false));
+warm.addEventListener("click", () => void run(true));
+prepare.addEventListener("click", async () => {
+ buttons(true);
+ try {
+ await dispose();
+ unregister = registerRealtimeAudioPreparation();
+ // The helper's idle callback is queued first. Acquiring afterward exposes
+ // whether it really prepared a context; an unsupported scheduler fails the
+ // assertion rather than being called a warm result.
+ await new Promise<void>(resolve => {
+ if (window.requestIdleCallback) window.requestIdleCallback(() => resolve(), { timeout: 2500 });
+ else window.setTimeout(resolve, 750);
+ });
+ held = acquireRealtimeAudio();
+ await held.loaded;
+ if (!held.prepared || held.context.state !== "suspended") {
+ throw new Error("Preparation did not produce a suspended context");
+ }
+ status.textContent = "Prepared: context suspended, local worklet loaded, microphone unopened.";
+ } catch (error) {
+ status.textContent = `Preparation failed: ${String(error)}`;
+ await dispose();
+ } finally { buttons(false); }
+});
+window.addEventListener("pagehide", () => { void dispose(); });
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx b/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx
index 2d04331c7c..bbfd616809 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/AgentComposer.tsx
@@ -29,7 +29,7 @@ import { offeredModels, useSavedHiddenModels } from "@/lib/agentProviderPrefs";
import { runningTurn } from "@/components/agentchat/reduce";
import { permissionModeIcon } from "@/components/agentchat/permissionIcons";
import { useComposerDictation } from "@/components/agentchat/useComposerDictation";
-import { useChatAttachments } from "@/components/agentchat/useChatAttachments";
+import { releaseHeldFiles, restoreHeldFiles, useChatAttachments } from "@/components/agentchat/useChatAttachments";
import { usePasteRescue } from "@/components/agentchat/usePasteRescue";
import { useComposerTypeahead } from "@/components/agentchat/useComposerTypeahead";
import { ComposerTypeahead } from "@/components/agentchat/ComposerTypeahead";
@@ -43,7 +43,7 @@ import {
type BrainSection,
} from "@/components/agentchat/ComposerBrainPicker";
import type { ToolChoice } from "@/components/agentchat/toolChoices";
-import { readComposerDraft, useComposerDraft, writeComposerDraft } from "./composerDrafts";
+import { captureComposerDraftTarget, readComposerDraft, useComposerDraft, writeComposerDraft } from "./composerDrafts";
import { useAppshotClaim } from "@/components/agentchat/useAppshotClaim";
import { PetMark } from "@/components/pets/PetMark";
import { useVoiceModeSwitch } from "@/components/home/assistantStatus";
@@ -253,37 +253,45 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {
const running = runningTurn(timeline) !== null;
const live = running || busy;
+ const canQueue = surface === "jarvis" || surface === "society";
async function onSend() {
const sessionAtSend = activeSessionId;
- const content = value.trim();
- if (await commands.execute(content)) return;
+ if (await commands.execute(fieldRef.current?.getDraft().text.trim() ?? value.trim())) return;
+ const submitted = fieldRef.current?.getDraft() ?? readComposerDraft(store, sessionAtSend);
+ const content = submitted.text.trim();
// A message may be files alone: dropping a screenshot and pressing Enter
// is a complete gesture, and refusing it would be the composer insisting
// on a sentence the picture already is.
- if ((!content && files.attachments.length === 0) || (running || busy) && !commands.canSteer) return;
+ if ((!content && files.attachments.length === 0) || (running || busy) && !commands.canSteer && !canQueue) return;
if (files.analyzing > 0) return; // a file still being read would be sent without its contents
if (dictating) stopDictation();
- const selected = selectedTools;
+ const selected = submitted.choices;
+ const held = files.take();
+ if (!content && held.attachments.length === 0) return;
+ const draftTarget = captureComposerDraftTarget(store, sessionAtSend);
writeComposerDraft(store, sessionAtSend, { text: "", choices: [] });
fieldRef.current?.clear();
setAttachError("");
- const attached = files.attachments;
- files.clear();
- if (selected.length)
- await send(
+ const result = selected.length ? await send(
content,
- attached,
+ held.attachments,
[...new Set(selected.map((row) => row.id))],
- );
- else await send(content, attached);
- if (sessionAtSend && store.getState().activeSessionId !== sessionAtSend) return;
- if (store.getState().lastError) {
- const target = store.getState().activeSessionId;
- const current = readComposerDraft(store, target);
- if (!current.text && !current.choices.length) {
- writeComposerDraft(store, target, { text: value, choices: selected });
+ ) : await send(content, held.attachments);
+ if (result === "failed" || result === undefined && store.getState().lastError) {
+ if (draftTarget.superseded) {
+ releaseHeldFiles(held);
+ pushToast("error", `${t("society.chat.message_not_sent").replace("{0}", submitted.text)}`);
+ return;
}
+ const current = readComposerDraft(store, draftTarget.sessionId);
+ writeComposerDraft(store, draftTarget.sessionId, {
+ text: [submitted.text, current.text].filter(Boolean).join("\n"),
+ choices: [...selected, ...current.choices.filter((row) => !selected.some((sent) => sent.id === row.id))],
+ });
+ restoreHeldFiles(store, draftTarget.sessionId ?? "", held);
+ } else {
+ releaseHeldFiles(held);
}
}
@@ -429,7 +437,9 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {
}, [minimal, providers]);
const brainSections = useMemo<BrainSection[]>(
- () => brainProviders.map((p) => ({ id: p.id, label: p.label, icon: providerIcon(p), muted: !p.connected })),
+ () => brainProviders.map((p) => ({ id: p.id, family: p.family,
+ access: p.keyless ? "local" : isApiRunner(p.runner) ? "api" : "subscription",
+ label: p.label, icon: providerIcon(p), muted: !p.connected })),
[brainProviders, providerIcon],
);
@@ -621,12 +631,12 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {
commands.isCommand || connected &&
(Boolean(value.trim()) || files.attachments.length > 0) &&
files.analyzing === 0 &&
- (!running || commands.canSteer) &&
- (!busy || commands.canSteer) &&
+ (!running || commands.canSteer || canQueue) &&
+ (!busy || commands.canSteer || canQueue) &&
Boolean(provider?.connected);
// Send while the mic is open ends the dictation and sends once the words
// land, so it is live even before the box holds any text.
- const canFinishDictation = dictating && connected && !live && Boolean(provider?.connected);
+ const canFinishDictation = dictating && connected && (!live || canQueue) && Boolean(provider?.connected);
const placeholder = dictating
? t("chats_view.dictation_listening")
: connected
@@ -827,7 +837,14 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {
{/* One round action in the signal blue: voice mode while the box is
empty, Send once there is something to send, Stop while a turn
runs. */}
- {live && !commands.isCommand && !(commands.canSteer && value.trim()) ? (
+ {live && canQueue && (
+ <button type="button" onClick={() => void cancel()} aria-label={t("agent_chat.stop")}
+ title={t("agent_chat.stop")} data-testid="composer-stop"
+ className={cn(ROUND_ACTION_CLASS, "bg-secondary text-foreground hover:bg-secondary/80")}>
+ <Square className="h-3 w-3 fill-current" />
+ </button>
+ )}
+ {live && !canQueue && !commands.isCommand && !(commands.canSteer && value.trim()) ? (
<button
type="button"
onClick={() => void cancel()}
@@ -1059,7 +1076,14 @@ export function AgentComposer({ autoFocus = false }: { autoFocus?: boolean }) {
startLabel={t("chats_view.dictation_start")}
stopLabel={t("chats_view.dictation_stop")}
/>
- {live && !commands.isCommand && !(commands.canSteer && value.trim()) ? (
+ {live && canQueue && (
+ <button type="button" onClick={() => void cancel()} aria-label={t("agent_chat.stop")}
+ title={t("agent_chat.stop")} data-testid="composer-stop"
+ className="inline-flex h-8 w-8 items-center justify-center rounded-lg bg-secondary text-foreground hover:bg-secondary/80">
+ <Square className="h-3.5 w-3.5" />
+ </button>
+ )}
+ {live && !canQueue && !commands.isCommand && !(commands.canSteer && value.trim()) ? (
<button
type="button"
onClick={() => void cancel()}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/AgentTimeline.tsx b/jarvis/ui/web/frontend/src/components/agentchat/AgentTimeline.tsx
index 1052c67f43..1bf8c0cd45 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/AgentTimeline.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/AgentTimeline.tsx
@@ -1,4 +1,4 @@
-import { Fragment, memo, useMemo } from "react";
+import { Fragment, memo, useMemo, type ReactNode } from "react";
import { ChatMarkdown, MediaPreview, mediaKind } from "@/components/agentchat/ChatMarkdown";
import { CircleAlert, FileText, ImageIcon } from "lucide-react";
import { InternalMessageBubble, type InternalParticipant } from "./InternalMessageBubble";
@@ -25,6 +25,7 @@ export function AgentTimeline({
bubbles = false,
traceLook = "rail",
traceCompanion = false,
+ traceFace,
}: {
items: TimelineItem[];
assistantName: string;
@@ -46,6 +47,8 @@ export function AgentTimeline({
traceLook?: TraceLook;
/** Jarvis's own chat: live traces show the user's pet at work. */
traceCompanion?: boolean;
+ /** An agent's own chat: live traces show that agent's face (or the pet it wears) at work. */
+ traceFace?: ReactNode;
}) {
const t = useT();
// On the rail look an agent's answer to a working turn is a quiet line in
@@ -220,6 +223,7 @@ export function AgentTimeline({
bubbles={bubbles}
traceLook={traceLook}
traceCompanion={traceCompanion}
+ traceFace={traceFace}
messages={messagesByTurn.get(item.id)}
/>
);
@@ -263,8 +267,8 @@ function stampLabel(at: Date, now: Date): string {
const NO_MESSAGES = new Map<string, TraceMessage[]>();
-const Turn = memo(function Turn({ turn, assistantName, providerLabel, onDecide, bubbles = false, traceLook, traceCompanion = false, messages }: {
- turn: TurnItem; assistantName: string; providerLabel: string; onDecide: Decide; bubbles?: boolean; traceLook: TraceLook; traceCompanion?: boolean;
+const Turn = memo(function Turn({ turn, assistantName, providerLabel, onDecide, bubbles = false, traceLook, traceCompanion = false, traceFace, messages }: {
+ turn: TurnItem; assistantName: string; providerLabel: string; onDecide: Decide; bubbles?: boolean; traceLook: TraceLook; traceCompanion?: boolean; traceFace?: ReactNode;
/** Agent messages that arrived while this turn worked. */
messages?: TraceMessage[];
}) {
@@ -282,7 +286,7 @@ const Turn = memo(function Turn({ turn, assistantName, providerLabel, onDecide,
{turn.effort ? <span>{effortLabel(turn.effort, t)}</span> : null}
</div>
)}
- <TurnTrace turn={turn} look={traceLook} companion={traceCompanion} extras={extras} onDecide={onDecide} renderText={(text, id) => <Prose block={{ kind: "text", text, id }} />} />
+ <TurnTrace turn={turn} look={traceLook} companion={traceCompanion} face={traceFace} extras={extras} onDecide={onDecide} renderText={(text, id) => <Prose block={{ kind: "text", text, id }} />} />
</div>;
});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.access.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.access.test.tsx
new file mode 100644
index 0000000000..7561453d23
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.access.test.tsx
@@ -0,0 +1,63 @@
+import { cleanup, fireEvent, render, screen, within } from "@testing-library/react";
+import { afterEach, beforeEach, expect, test, vi } from "vitest";
+import { ComposerBrainPicker, type BrainSection } from "./ComposerBrainPicker";
+import { MODEL_ACCESS_KEY } from "@/lib/modelAccess";
+
+const sections: BrainSection[] = [
+ { id: "openai", family: "openai", access: "api", label: "OpenAI", icon: <span>O</span> },
+ { id: "openai-codex", family: "openai", access: "subscription", label: "Codex", icon: <span>O</span> },
+ { id: "gemini", family: "gemini", access: "api", label: "Gemini", icon: <span>G</span> },
+];
+const groups = sections.map((section) => ({ id: section.id, label: section.label,
+ options: [{ value: `${section.id}\u0001model`, label: `${section.label} model` }],
+}));
+function mount(offered = sections, onChange = vi.fn()) {
+ return { onChange, ...render(<ComposerBrainPicker value="openai\u0001model" currentSection="openai"
+ sections={offered} groups={groups} onChange={onChange} ariaLabel="Model" fallbackLabel="Model" searchPlaceholder="Search" />) };
+}
+async function open() {
+ fireEvent.click(screen.getByTestId("composer-model"));
+ return await screen.findByTestId("composer-model-panel");
+}
+beforeEach(() => localStorage.clear());
+afterEach(() => { cleanup(); localStorage.clear(); });
+
+test("one brand mark opens subscription first and remembers API browsing across reopening", async () => {
+ const { onChange, unmount } = mount();
+ let panel = await open();
+ expect(within(panel).getByTestId("composer-model-rail").querySelectorAll("button")).toHaveLength(3);
+ expect(screen.queryByTestId("composer-model-rail-openai-codex")).toBeNull();
+ expect(within(panel).getByRole("radio", { name: "Subscription" }).getAttribute("aria-checked")).toBe("true");
+ expect(within(panel).getByRole("option").textContent).toContain("Codex model");
+ fireEvent.click(within(panel).getByRole("radio", { name: "API key" }));
+ expect(within(panel).getByRole("option").textContent).toContain("OpenAI model");
+ expect(onChange).not.toHaveBeenCalled();
+ fireEvent.keyDown(within(panel).getByRole("radio", { name: "API key" }), { key: "Escape" });
+ panel = await open();
+ expect(within(panel).getByRole("radio", { name: "API key" }).getAttribute("aria-checked")).toBe("true");
+ unmount(); mount(); panel = await open();
+ expect(within(panel).getByRole("radio", { name: "API key" }).getAttribute("aria-checked")).toBe("true");
+ fireEvent.keyDown(within(panel).getByRole("radio", { name: "API key" }), { key: "ArrowLeft" });
+ expect(within(panel).getByRole("radio", { name: "Subscription" }).getAttribute("aria-checked")).toBe("true");
+ expect(document.activeElement).toBe(within(panel).getByRole("radio", { name: "Subscription" }));
+});
+
+test("API-only access works without a switch, and a disconnected subscription is not the default", async () => {
+ const view = mount([sections[0]]);
+ let panel = await open();
+ expect(within(panel).queryByRole("radiogroup")).toBeNull();
+ expect(within(panel).getByRole("option").textContent).toContain("OpenAI model");
+ view.unmount(); mount([sections[0], { ...sections[1], muted: true }]); panel = await open();
+ expect(within(panel).getByRole("radio", { name: "API key" }).getAttribute("aria-checked")).toBe("true");
+});
+
+test("search keeps both access paths explicit and selecting a result remembers its path", async () => {
+ const { onChange } = mount();
+ const panel = await open();
+ fireEvent.change(screen.getByPlaceholderText("Search"), { target: { value: "model" } });
+ expect(within(panel).queryByRole("radiogroup")).toBeNull();
+ expect(within(panel).getAllByRole("option")).toHaveLength(3);
+ fireEvent.click(within(panel).getByRole("option", { name: /Codex model/ }));
+ expect(onChange).toHaveBeenCalledWith("openai-codex\u0001model");
+ expect(JSON.parse(localStorage.getItem(MODEL_ACCESS_KEY)!)).toEqual({ openai: "subscription" });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.tsx b/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.tsx
index 2ede595c33..17b21b010e 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/ComposerBrainPicker.tsx
@@ -5,6 +5,8 @@ import { Combobox, type ComboboxGroup, type ComboboxOption } from "@/components/
import { useT } from "@/i18n";
import { orderBy, useProviderOrder } from "@/lib/providerOrder";
import { cn } from "@/lib/utils";
+import { modelAccessFamily, preferredModelAccess, useModelAccess, type ModelAccess } from "@/lib/modelAccess";
+import { ModelAccessSwitch } from "@/components/providers/ModelAccessSwitch";
/**
* The front page's brain pick: which provider and which model answer.
@@ -23,6 +25,8 @@ import { cn } from "@/lib/utils";
export interface BrainSection {
id: string;
+ family?: string;
+ access?: ModelAccess;
label: string;
icon: ReactNode;
/** Shown greyed: the provider is offered but not connected. */
@@ -101,8 +105,24 @@ export function ComposerBrainPicker({
const t = useT();
const [favorites, toggleFavorite] = useFavoriteModels();
const [section, setSection] = useState(currentSection);
+ const [access, rememberAccess] = useModelAccess();
const [providerOrder, moveProvider] = useProviderOrder();
- const ordered = useMemo(() => orderBy(sections, (s) => s.id, providerOrder), [sections, providerOrder]);
+ const families = useMemo(() => {
+ const byFamily = new Map<string, BrainSection[]>();
+ for (const entry of orderBy(sections, (s) => s.id, providerOrder)) {
+ const family = modelAccessFamily(entry.family || entry.id);
+ byFamily.set(family, [...(byFamily.get(family) ?? []), entry]);
+ }
+ return [...byFamily].map(([family, entries]) => {
+ const options = entries.map((entry) => ({ ...entry, kind: entry.access ?? "api" as const, disabled: entry.muted }));
+ const selected = preferredModelAccess(options, access[family])!;
+ // Prefer the API row's company name over a subscription product's name.
+ const brand = entries.find((entry) => entry.access === "api") ?? entries[0];
+ return { family, entries, options, selected, brand };
+ });
+ }, [sections, providerOrder, access]);
+ const activeFamily = families.find((entry) => entry.entries.some((s) => s.id === section)) ?? families[0];
+ const activeSection = activeFamily?.selected.id;
// Every row carries its star; the favourites tab lists the starred rows in
// the order they were starred.
@@ -148,10 +168,10 @@ export function ComposerBrainPicker({
.filter((option): option is NonNullable<typeof option> => Boolean(option));
return options.length ? [{ id: FAVORITES, options }] : [];
}
- const own = starred.find((group) => group.id === section) ?? starred[0];
+ const own = starred.find((group) => group.id === activeSection) ?? starred[0];
// The rail already names the provider, so its group needs no heading.
return own ? [{ ...own, label: undefined }] : [];
- }, [section, starred, favorites]);
+ }, [section, activeSection, starred, favorites]);
const onOpenChange = useCallback(
(open: boolean) => {
@@ -177,20 +197,20 @@ export function ComposerBrainPicker({
<Star className="h-4 w-4 fill-current" aria-hidden />
</RailButton>
<span className="mx-1 my-0.5 border-b border-border" aria-hidden />
- {ordered.map((s, index) => (
+ {families.map(({ family, entries, brand: s }, index) => (
<RailButton
- key={s.id}
+ key={family}
reorder={{
id: s.id,
- onMove: (dragged, target) => moveProvider(ordered.map((o) => o.id), dragged, target),
+ onMove: (dragged, target) => moveProvider(families.map((o) => o.brand.id), dragged, target),
onStep: (step) => {
- const target = ordered[index + step];
- if (target) moveProvider(ordered.map((o) => o.id), s.id, target.id);
+ const target = families[index + step];
+ if (target) moveProvider(families.map((o) => o.brand.id), s.id, target.brand.id);
},
}}
- active={section === s.id}
+ active={section !== FAVORITES && activeFamily?.family === family}
label={s.label}
- muted={s.muted}
+ muted={entries.every((entry) => entry.muted)}
onSelect={() => setSection(s.id)}
testId={`composer-model-rail-${s.id}`}
>
@@ -206,9 +226,17 @@ export function ComposerBrainPicker({
value={value}
groups={starred}
browseGroups={browseGroups}
+ browseHeader={section !== FAVORITES && activeFamily ? <ModelAccessSwitch
+ options={activeFamily.options} value={activeFamily.selected.kind}
+ onChange={(kind) => rememberAccess(activeFamily.family, kind)} /> : undefined}
aside={rail}
onOpenChange={onOpenChange}
- onChange={onChange}
+ onChange={(next) => {
+ const chosen = sections.find((entry) => groups.some((group) => group.id === entry.id
+ && [...group.options, ...(group.more?.options ?? [])].some((option) => option.value === next)));
+ if (chosen?.access) rememberAccess(chosen.family || chosen.id, chosen.access);
+ onChange(next);
+ }}
ariaLabel={ariaLabel}
fallbackLabel={fallbackLabel}
searchPlaceholder={searchPlaceholder}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.test.tsx
index 3d18008a00..da330c0c2f 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.test.tsx
@@ -1,5 +1,5 @@
import { createRef } from "react";
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { ComposerChipField, type ComposerChipFieldHandle } from "./ComposerChipField";
@@ -74,4 +74,21 @@ describe("ComposerChipField", () => {
handle.current?.insertChip(gmail);
expect(handle.current?.getDraft().choices.map((row) => row.id)).toEqual(["plugin:gmail"]);
});
+
+ it("sends selection changes only to the current draft owner after switching sessions", () => {
+ const handle = createRef<ComposerChipFieldHandle>();
+ const blank = vi.fn();
+ const assigned = vi.fn();
+ const { rerender } = render(
+ <ComposerChipField ref={handle} placeholder="Message" onSubmit={() => {}} onDraftChange={blank} />,
+ );
+ handle.current?.hydrate("Followup draft");
+ blank.mockClear();
+ rerender(
+ <ComposerChipField ref={handle} placeholder="Message" onSubmit={() => {}} onDraftChange={assigned} />,
+ );
+ document.dispatchEvent(new Event("selectionchange"));
+ expect(blank).not.toHaveBeenCalled();
+ expect(assigned).toHaveBeenCalledWith(expect.objectContaining({ text: "Followup draft" }));
+ });
});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.tsx b/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.tsx
index 3f9f714459..e17104e342 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/ComposerChipField.tsx
@@ -179,6 +179,8 @@ export const ComposerChipField = forwardRef<
) {
const elRef = useRef<HTMLDivElement>(null);
const savedRange = useRef<Range | null>(null);
+ const onDraftChangeRef = useRef(onDraftChange);
+ onDraftChangeRef.current = onDraftChange;
const rememberRange = (root: HTMLElement) => {
const range = liveRange(root);
@@ -196,7 +198,9 @@ export const ComposerChipField = forwardRef<
const emit = () => {
const root = elRef.current;
if (!root) return;
- onDraftChange(serialize(root));
+ // The document selection listener outlives a render or session switch.
+ // Always notify the current draft owner, including after session creation.
+ onDraftChangeRef.current(serialize(root));
};
const insertNode = (node: Node, afterSpace = true) => {
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/CredentialCard.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/CredentialCard.test.tsx
new file mode 100644
index 0000000000..c24b413b45
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/CredentialCard.test.tsx
@@ -0,0 +1,228 @@
+import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { ThreadTurn } from "@/components/agentic/threads/ThreadTimeline";
+import type { AgentChatEvent } from "@/lib/agentChatApi";
+import { createAgentChatStore } from "@/store/agentChat";
+import { AgentChatStoreProvider } from "./AgentChatStoreContext";
+import { EMPTY_TIMELINE, reduceEvents, type ToolBlock, type TurnItem } from "./reduce";
+
+let seq = 0;
+function ev(kind: string, payload: Record<string, unknown>, tsMs = 1000): AgentChatEvent {
+ return { seq: ++seq, ts_ms: tsMs, kind, payload } as AgentChatEvent;
+}
+
+const SECRET = "ghp_supersecretvalue123";
+
+function asking(extra: AgentChatEvent[] = []): AgentChatEvent[] {
+ return [
+ ev("turn_started", { turn_id: "t1" }),
+ ev("tool_call", { turn_id: "t1", call_id: "c1", name: "mcp__jarvis__society_request_credential", input: { env: "GITHUB_TOKEN" } }),
+ ev("credential_required", {
+ turn_id: "t1",
+ request_id: "r1",
+ asker: "Ada",
+ env: "GITHUB_TOKEN",
+ label: "GitHub token",
+ description: "To open the pull request.",
+ placeholder: "",
+ replace: false,
+ expires_ms: Date.now() + 10 * 60_000,
+ }),
+ ...extra,
+ ];
+}
+
+afterEach(() => {
+ cleanup();
+ seq = 0;
+ vi.useRealTimers();
+ vi.restoreAllMocks();
+});
+
+describe("credential events", () => {
+ it("attach the field to the request call and record how it closed", () => {
+ const tl = reduceEvents(EMPTY_TIMELINE, asking([
+ ev("tool_call", { turn_id: "t1", call_id: "c2", name: "mcp__jarvis__society_request_credential", input: { wait_for: "r1" } }),
+ ev("credential_resolved", { turn_id: "t1", request_id: "r1", env: "GITHUB_TOKEN", status: "saved" }),
+ ]));
+ const turn = tl.items[0] as TurnItem;
+ const block = turn.blocks[0] as ToolBlock;
+ expect(block.callId).toBe("c1");
+ expect(block.credential).toMatchObject({ env: "GITHUB_TOKEN", label: "GitHub token", status: "saved" });
+ });
+
+ it.each(["done", "error", "cancelled"])("keeps an unanswered field open when its turn is %s", (status) => {
+ const tl = reduceEvents(EMPTY_TIMELINE, asking([
+ ev("turn_finished", { turn_id: "t1", status }),
+ ev("turn_started", { turn_id: "t2" }),
+ ]));
+ const block = (tl.items[0] as TurnItem).blocks[0] as ToolBlock;
+ expect(block.credential?.status).toBeNull();
+ });
+});
+
+describe("CredentialCard", () => {
+ function draw(events: AgentChatEvent[]) {
+ const store = createAgentChatStore("society");
+ store.setState({ activeSessionId: "society:ada" });
+ const view = (current: AgentChatEvent[]) => (
+ <AgentChatStoreProvider store={store}>
+ <ThreadTurn turn={reduceEvents(EMPTY_TIMELINE, current).items[0] as TurnItem} prompts="inline" />
+ </AgentChatStoreProvider>
+ );
+ const rendered = render(view(events));
+ return { rerender: (current: AgentChatEvent[]) => rendered.rerender(view(current)) };
+ }
+
+ it("sends the pasted value once and confirms saving without waiting for the event stream", async () => {
+ const fetchMock = vi.spyOn(globalThis, "fetch").mockResolvedValue(
+ new Response(JSON.stringify({ ok: true, request_id: "r1", status: "saved" }), { status: 200 }),
+ );
+ draw(asking());
+ const card = screen.getByTestId("credential-card");
+ expect(card.textContent).toContain("GitHub token");
+ const explanation = screen.getByText("To open the pull request.");
+ expect(explanation.compareDocumentPosition(card) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
+ expect(card.contains(explanation)).toBe(false);
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ expect(input.type).toBe("password");
+ expect(input.getAttribute("aria-describedby")).toBe(explanation.id);
+ const save = screen.getByRole("button", { name: "Save securely" }) as HTMLButtonElement;
+ expect(save.disabled).toBe(true);
+ fireEvent.change(input, { target: { value: ` ${SECRET} ` } });
+ fireEvent.click(save);
+ fireEvent.click(save);
+ await waitFor(() => expect(fetchMock).toHaveBeenCalledTimes(1));
+ const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit];
+ expect(url).toBe("/api/agent-chat/sessions/society%3Aada/credentials/r1");
+ expect(JSON.parse(String(init.body))).toEqual({ value: SECRET });
+ await waitFor(() => expect(screen.queryByLabelText("GitHub token")).toBeNull());
+ expect(screen.getByRole("status").textContent).toContain("GitHub token saved");
+ expect(screen.getByTestId("credential-card").textContent).not.toContain(SECRET);
+ });
+
+ it("says why a refused value was not stored, in the person's words", async () => {
+ vi.spyOn(globalThis, "fetch").mockResolvedValue(
+ new Response(JSON.stringify({ detail: "the credential is empty" }), { status: 400 }),
+ );
+ draw(asking());
+ fireEvent.change(screen.getByLabelText("GitHub token"), { target: { value: "x" } });
+ fireEvent.click(screen.getByRole("button", { name: "Save securely" }));
+ expect((await screen.findByRole("alert")).textContent).toContain("single line");
+ });
+
+ it("shows the closed field as one line without any value", () => {
+ draw(asking([ev("credential_resolved", { turn_id: "t1", request_id: "r1", env: "GITHUB_TOKEN", status: "saved" })]));
+ const card = screen.getByTestId("credential-card");
+ expect(card.getAttribute("data-state")).toBe("saved");
+ expect(card.textContent).toBe("GitHub token saved · available as GITHUB_TOKEN");
+ expect(screen.queryByRole("textbox")).toBeNull();
+ });
+
+ it("declines without sending anything secret", async () => {
+ const fetchMock = vi.spyOn(globalThis, "fetch").mockResolvedValue(
+ new Response(JSON.stringify({ ok: true, request_id: "r1", status: "declined" }), { status: 200 }),
+ );
+ draw(asking());
+ fireEvent.change(screen.getByLabelText("GitHub token"), { target: { value: SECRET } });
+ fireEvent.click(screen.getByRole("button", { name: "Cancel" }));
+ await waitFor(() => expect(fetchMock).toHaveBeenCalledTimes(1));
+ const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit];
+ expect(url).toBe("/api/agent-chat/sessions/society%3Aada/credentials/r1/decline");
+ expect(init.body).toBeUndefined();
+ await waitFor(() => expect(screen.getByTestId("credential-card").getAttribute("data-state")).toBe("declined"));
+ });
+
+ it("retains a partially pasted value when the turn finishes and its work folds", () => {
+ const events = asking([
+ ev("tool_result", { turn_id: "t1", call_id: "c1", output: "Waiting for the person" }),
+ ev("tool_call", { turn_id: "t1", call_id: "c2", name: "Read", input: { file: "README.md" } }),
+ ev("tool_result", { turn_id: "t1", call_id: "c2", output: "Read complete" }),
+ ev("text_delta", { turn_id: "t1", text: "Enter the token when you are ready." }),
+ ]);
+ const { rerender } = draw(events);
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ fireEvent.change(input, { target: { value: SECRET } });
+ rerender([...events, ev("turn_finished", { turn_id: "t1", status: "done" })]);
+ expect(screen.getByLabelText("GitHub token")).toBe(input);
+ expect(input.value).toBe(SECRET);
+ fireEvent.click(screen.getByTestId("thread-worked-for"));
+ fireEvent.click(screen.getByTestId("thread-worked-for"));
+ expect(input.value).toBe(SECRET);
+ });
+
+ it("stays open across elapsed time, blur, Escape and outside clicks", async () => {
+ vi.useFakeTimers();
+ const fetchMock = vi.spyOn(globalThis, "fetch");
+ draw(asking());
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ fireEvent.change(input, { target: { value: SECRET } });
+ fireEvent.blur(input);
+ fireEvent.keyDown(input, { key: "Escape" });
+ fireEvent.click(document.body);
+ await act(async () => { await vi.advanceTimersByTimeAsync(20 * 60_000); });
+ expect(screen.getByTestId("credential-card").getAttribute("data-state")).toBe("open");
+ expect(input.value).toBe(SECRET);
+ expect(fetchMock).not.toHaveBeenCalled();
+ });
+
+ it.each([
+ ["invalid_token", 422, "service rejected"],
+ ["network_error", 503, "reach the service"],
+ ["validation_timeout", 504, "timed out"],
+ ["storage_failed", 503, "saved securely"],
+ ["request_busy", 409, "already being checked"],
+ ])("keeps %s failures retryable without rendering response secrets", async (code, status, message) => {
+ const fetchMock = vi.spyOn(globalThis, "fetch")
+ .mockResolvedValueOnce(new Response(JSON.stringify({ detail: { code, message: SECRET } }), { status: Number(status) }))
+ .mockResolvedValueOnce(new Response(JSON.stringify({ ok: true, request_id: "r1", status: "saved" }), { status: 200 }));
+ draw(asking());
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ fireEvent.change(input, { target: { value: SECRET } });
+ fireEvent.click(screen.getByRole("button", { name: "Save securely" }));
+ expect((await screen.findByRole("alert")).textContent).toContain(message);
+ expect(screen.getByTestId("credential-card").textContent).not.toContain(SECRET);
+ expect(input.value).toBe(SECRET);
+ expect(input.disabled).toBe(false);
+ fireEvent.click(screen.getByRole("button", { name: "Save securely" }));
+ await waitFor(() => expect(screen.getByTestId("credential-card").getAttribute("data-state")).toBe("saved"));
+ expect(fetchMock).toHaveBeenCalledTimes(2);
+ });
+
+ it("keeps transport errors visible and retryable", async () => {
+ vi.spyOn(globalThis, "fetch").mockRejectedValue(new TypeError("Failed to fetch"));
+ draw(asking());
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ fireEvent.change(input, { target: { value: SECRET } });
+ fireEvent.click(screen.getByRole("button", { name: "Save securely" }));
+ expect((await screen.findByRole("alert")).textContent).toContain("Check your connection");
+ expect(input.disabled).toBe(false);
+ expect(input.value).toBe(SECRET);
+ });
+
+ it("bounds a hanging submission without expiring the input", async () => {
+ vi.useFakeTimers();
+ vi.spyOn(globalThis, "fetch").mockImplementation((_url, init) => new Promise((_resolve, reject) => {
+ init?.signal?.addEventListener("abort", () => reject(new DOMException("Aborted", "AbortError")), { once: true });
+ }));
+ draw(asking());
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ fireEvent.change(input, { target: { value: SECRET } });
+ fireEvent.click(screen.getByRole("button", { name: "Save securely" }));
+ await act(async () => { await vi.advanceTimersByTimeAsync(30_000); });
+ expect(screen.getByRole("alert").textContent).toContain("timed out");
+ expect(input.disabled).toBe(false);
+ expect(input.value).toBe(SECRET);
+ });
+
+ it("does not claim a save when the response is unconfirmed", async () => {
+ vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response("{}", { status: 200 }));
+ draw(asking());
+ const input = screen.getByLabelText("GitHub token") as HTMLInputElement;
+ fireEvent.change(input, { target: { value: SECRET } });
+ fireEvent.click(screen.getByRole("button", { name: "Save securely" }));
+ await screen.findByRole("alert");
+ expect(input.value).toBe(SECRET);
+ expect(input.disabled).toBe(false);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/CredentialCard.tsx b/jarvis/ui/web/frontend/src/components/agentchat/CredentialCard.tsx
new file mode 100644
index 0000000000..85016b6e3b
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/CredentialCard.tsx
@@ -0,0 +1,184 @@
+import { memo, useId, useRef, useState, type FormEvent } from "react";
+import { Check, KeyRound, ShieldCheck, X } from "lucide-react";
+import { fill, useT } from "@/i18n";
+import { cn } from "@/lib/utils";
+import { AgentChatApiError, declineAgentChatCredential, submitAgentChatCredential, type CredentialResult } from "@/lib/agentChatApi";
+import { useAgentChat } from "./AgentChatStoreContext";
+import type { CredentialState } from "./reduce";
+
+/**
+ * An agent's secure credential field (jarvis/agent_chat/credential_requests.py).
+ *
+ * The person pastes a token the agent asked for; the value goes from this
+ * field to the agent's vault in one request and is never kept in the chat:
+ * no event, no store and no reply carries it. The agent only learns that the
+ * environment variable is set.
+ */
+
+export const CredentialCard = memo(function CredentialCard({ credential }: { credential: CredentialState }) {
+ const t = useT();
+ const descriptionId = useId();
+ const agent = credential.asker || t("credential_card.fallback_agent");
+ return (
+ <div className="my-2 w-full max-w-2xl space-y-3">
+ <p id={descriptionId} className="whitespace-pre-wrap text-sm leading-6 text-foreground [overflow-wrap:anywhere]">
+ {credential.description || fill(t("credential_card.subtitle"), { agent, env: credential.env })}
+ </p>
+ {credential.status !== null ? <ClosedCard credential={credential} /> : (
+ <OpenCard key={credential.requestId} credential={credential} descriptionId={descriptionId} />
+ )}
+ </div>
+ );
+});
+
+function OpenCard({ credential, descriptionId }: { credential: CredentialState; descriptionId: string }) {
+ const t = useT();
+ const sessionId = useAgentChat((s) => s.activeSessionId);
+ const [value, setValue] = useState("");
+ const [busy, setBusy] = useState(false);
+ const [error, setError] = useState<string | null>(null);
+ const [resolved, setResolved] = useState<CredentialResult["status"] | null>(null);
+ const errorId = useId();
+ const sending = useRef(false);
+ const agent = credential.asker || t("credential_card.fallback_agent");
+
+ const failure = (err: unknown): string => {
+ if (err instanceof AgentChatApiError) {
+ const keys: Record<string, string> = {
+ invalid_token: "invalid_token", network_error: "network_error", validation_timeout: "validation_timeout",
+ storage_failed: "storage_failed", request_busy: "request_busy", request_closed: "expired",
+ };
+ const key = keys[err.code];
+ if (key) return t(`credential_card.${key}`);
+ }
+ if (err instanceof AgentChatApiError && (err.status === 400 || err.status === 422)) return t("credential_card.invalid");
+ if (err instanceof AgentChatApiError && (err.status === 404 || err.status === 410)) return t("credential_card.expired");
+ if (err instanceof TypeError) return t("credential_card.network_error");
+ return t("credential_card.failed");
+ };
+
+ const run = async (action: (sid: string) => Promise<CredentialResult>, expected: CredentialResult["status"]) => {
+ if (sending.current || !sessionId) return;
+ sending.current = true;
+ setBusy(true);
+ setError(null);
+ try {
+ const result = await action(sessionId);
+ if (!result.ok || result.status !== expected || result.request_id !== credential.requestId) {
+ throw new AgentChatApiError("credential-unconfirmed", 502);
+ }
+ setValue("");
+ // The HTTP receipt is sufficient even while the event stream reconnects.
+ setResolved(result.status);
+ } catch (err) {
+ setError(failure(err));
+ sending.current = false;
+ setBusy(false);
+ }
+ };
+
+ const submit = (event: FormEvent) => {
+ event.preventDefault();
+ const secret = value.trim();
+ if (secret) void run((sid) => submitAgentChatCredential(sid, credential.requestId, secret), "saved");
+ };
+
+ if (resolved) return <ClosedCard credential={{ ...credential, status: resolved }} />;
+
+ return (
+ <section
+ role="group"
+ aria-label={t("credential_card.aria")}
+ aria-describedby={descriptionId}
+ data-testid="credential-card"
+ data-state="open"
+ className="w-full rounded-2xl border border-border bg-card p-4 text-foreground shadow-sm"
+ >
+ <header className="flex items-start gap-3">
+ <div className="min-w-0 flex-1">
+ <h3 className="text-sm font-semibold leading-6 [overflow-wrap:anywhere]">{credential.label}</h3>
+ {credential.replace ? (
+ <p className="mt-1 text-xs leading-5 text-muted-foreground [overflow-wrap:anywhere]">
+ {fill(t("credential_card.replace_note"), { agent })}
+ </p>
+ ) : null}
+ </div>
+ <button
+ type="button"
+ onClick={() => void run((sid) => declineAgentChatCredential(sid, credential.requestId), "declined")}
+ disabled={busy}
+ aria-label={t("credential_card.decline")}
+ title={t("credential_card.decline")}
+ className="-mr-1 -mt-0.5 shrink-0 rounded-md p-1.5 text-muted-foreground hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-50"
+ >
+ <X aria-hidden className="h-5 w-5" />
+ </button>
+ </header>
+
+ <form onSubmit={submit} className="mt-4 flex flex-col gap-2 sm:flex-row">
+ <input
+ type="password"
+ value={value}
+ onChange={(event) => setValue(event.target.value)}
+ disabled={busy}
+ autoComplete="new-password"
+ autoCorrect="off"
+ autoCapitalize="off"
+ spellCheck={false}
+ data-1p-ignore
+ data-lpignore="true"
+ name={`credential-${credential.requestId}`}
+ maxLength={16384}
+ placeholder={credential.placeholder || fill(t("credential_card.placeholder"), { label: credential.label })}
+ aria-label={credential.label}
+ aria-invalid={Boolean(error)}
+ aria-describedby={error ? `${descriptionId} ${errorId}` : descriptionId}
+ className="min-w-0 flex-1 rounded-xl border border-border bg-background px-4 py-2.5 text-sm text-foreground placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-60"
+ />
+ <button
+ type="submit"
+ disabled={busy || !value.trim()}
+ className={cn(
+ "shrink-0 rounded-xl px-4 py-2.5 text-sm font-medium transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring",
+ "bg-primary text-primary-foreground hover:opacity-90",
+ "disabled:bg-secondary disabled:text-muted-foreground disabled:hover:opacity-100",
+ )}
+ >
+ {busy ? t("credential_card.saving") : t("credential_card.save")}
+ </button>
+ </form>
+
+ <footer className="mt-3 space-y-1 text-xs leading-5 text-muted-foreground">
+ <p className="flex items-center gap-1.5">
+ <ShieldCheck aria-hidden className="h-3.5 w-3.5 shrink-0" />
+ <span>{t("credential_card.footer")}</span>
+ </p>
+ <p>{t("credential_card.stays_open")}</p>
+ {error ? (
+ <p id={errorId} role="alert" className="text-destructive">
+ {error}
+ </p>
+ ) : null}
+ </footer>
+ </section>
+ );
+}
+
+function ClosedCard({ credential }: { credential: CredentialState }) {
+ const t = useT();
+ const saved = credential.status === "saved";
+ const key = saved ? "saved" : credential.status === "declined" ? "declined" : credential.status === "timeout" ? "timeout" : "cancelled";
+ return (
+ <div
+ role="status"
+ data-testid="credential-card"
+ data-state={credential.status ?? ""}
+ className="my-1 flex w-full max-w-2xl items-center gap-2 rounded-xl border border-border px-4 py-2.5 text-sm text-muted-foreground"
+ >
+ {saved ? <Check aria-hidden className="h-4 w-4 shrink-0 text-primary" /> : <KeyRound aria-hidden className="h-4 w-4 shrink-0" />}
+ <span className="min-w-0 [overflow-wrap:anywhere]">
+ {fill(t(`credential_card.${key}`), { label: credential.label, env: credential.env })}
+ </span>
+ </div>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/MessengerTurn.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/MessengerTurn.test.tsx
new file mode 100644
index 0000000000..bcb0d3907a
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/MessengerTurn.test.tsx
@@ -0,0 +1,114 @@
+import { cleanup, fireEvent, render, screen } from "@testing-library/react";
+import { afterEach, describe, expect, it } from "vitest";
+import { MessengerTurn } from "./MessengerTurn";
+import { actionNoticeOf, callPresence, presenceOf } from "./messengerPresence";
+import type { ReasoningBlock, TextBlock, ToolBlock, TurnBlock, TurnItem } from "./reduce";
+
+afterEach(cleanup);
+
+function turn(blocks: TurnBlock[], status: TurnItem["status"] = "running"): TurnItem {
+ return {
+ type: "turn", id: "t1", provider: "p", model: "", effort: "", runner: "api", status, blocks,
+ startedMs: 1, durationMs: null, usage: null, liveUsage: null, costUsd: null, error: null,
+ };
+}
+
+const text = (value: string, id = "x1"): TextBlock => ({ kind: "text", id, text: value });
+const thought = (value: string): ReasoningBlock => ({ kind: "reasoning", id: "r1", text: value, durationMs: null, live: true, startedMs: 1 });
+function call(name: string, input: unknown, output: string | null = null, extra: Partial<ToolBlock> = {}): ToolBlock {
+ return { kind: "tool", callId: `c-${name}`, name, input, output, isError: false, durationMs: null, approval: null, startedMs: 1, ...extra };
+}
+
+describe("presenceOf", () => {
+ it("thinks before anything happened and while a thought runs", () => {
+ expect(presenceOf(turn([]))).toBe("thinking");
+ expect(presenceOf(turn([thought("planning the schedule")]))).toBe("thinking");
+ });
+
+ it("types while a message streams", () => {
+ expect(presenceOf(turn([text("Checking what is already connected.")]))).toBe("typing");
+ });
+
+ it("names the running call by what it does", () => {
+ expect(presenceOf(turn([call("mcp__jarvis__society_wiki_note", { title: "x" })]))).toBe("writing");
+ expect(presenceOf(turn([call("society_propose_change", { kind: "rule" })]))).toBe("writing");
+ expect(presenceOf(turn([call("society_propose_change", { kind: "routine" })]))).toBe("setting_up");
+ expect(presenceOf(turn([call("web_search", {})]))).toBe("searching");
+ expect(presenceOf(turn([call("browser_navigate", {})]))).toBe("browsing");
+ expect(presenceOf(turn([call("Bash", {})]))).toBe("command");
+ expect(presenceOf(turn([call("send_message", {})]))).toBe("working");
+ });
+
+ it("goes back to thinking between steps and says nothing once done", () => {
+ expect(presenceOf(turn([call("web_search", {}, "ok")]))).toBe("thinking");
+ expect(presenceOf(turn([text("Done.")], "done"))).toBeNull();
+ });
+
+ it("leaves a waiting card to speak for itself", () => {
+ const waiting = call("society_ask_user", {}, null, { question: { closed: false } as ToolBlock["question"] });
+ expect(presenceOf(turn([waiting]))).toBeNull();
+ });
+});
+
+describe("actionNoticeOf", () => {
+ const routine = (operation?: string) => call("society_propose_change", {
+ kind: "routine", mode: "apply", payload: { title: "Daily contributor check", ...(operation ? { operation } : {}) },
+ }, "{\"applied\": true}");
+
+ it("turns an applied routine into a created or updated line", () => {
+ expect(actionNoticeOf(routine())).toEqual({ key: "routine_created", subject: "Daily contributor check", icon: "routine" });
+ expect(actionNoticeOf(routine("update"))?.key).toBe("routine_updated");
+ expect(actionNoticeOf(routine("delete"))?.key).toBe("routine_deleted");
+ });
+
+ it("leaves no line for a failed, pending or unremarkable call", () => {
+ expect(actionNoticeOf({ ...routine(), isError: true })).toBeNull();
+ expect(actionNoticeOf({ ...routine(), output: "{\"status\": \"pending\"}" })).toBeNull();
+ expect(actionNoticeOf({ ...routine(), output: null })).toBeNull();
+ expect(actionNoticeOf(call("web_search", {}, "results"))).toBeNull();
+ expect(actionNoticeOf(call("society_propose_change", { kind: "identity" }, "ok"))).toBeNull();
+ });
+
+ it("classifies a call even behind a transport prefix", () => {
+ expect(callPresence(call("mcp__jarvis__society_memory_recall", {}))).toBe("recall");
+ expect(callPresence(call("Bash", {}))).toBe("command");
+ });
+});
+
+describe("MessengerTurn", () => {
+ it("draws messages as bubbles and never shows a thought", () => {
+ render(<MessengerTurn
+ turn={turn([thought("secret reasoning"), text("Checking it now.", "a"), routineDone(), text("It runs every day at 10:04.", "b")], "done")}
+ avatar={<span data-testid="face" />} color="#7ab6ef" onDecide={() => undefined} />);
+ expect(screen.getAllByTestId("agent-message-bubble")).toHaveLength(2);
+ expect(screen.queryByText("secret reasoning")).toBeNull();
+ expect(screen.getByTestId("messenger-action").textContent).toContain("Daily contributor check");
+ expect(screen.queryByTestId("messenger-presence")).toBeNull();
+ });
+
+ it("shows the agent's face with what it is doing while it works", () => {
+ render(<MessengerTurn turn={turn([call("society_wiki_note", { title: "x" })])}
+ avatar={<span data-testid="face" />} color="#7ab6ef" onDecide={() => undefined} />);
+ const line = screen.getByTestId("messenger-presence");
+ expect(line.dataset.presence).toBe("writing");
+ expect(screen.getByTestId("face")).toBeTruthy();
+ });
+
+ it("opens memory receipts from the Noted quill lines, never as a line of their own twice", () => {
+ const opened: string[] = [];
+ const link = (key: string) => ({ key, title: `Memory updated · ${key}.md`, onOpen: () => opened.push(key) });
+ render(<MessengerTurn
+ turn={turn([call("society_wiki_note", { title: "Plan" }, "{\"ok\": true}"), text("Saved.", "a")], "done")}
+ avatar={<span />} color="#7ab6ef" onDecide={() => undefined} memoryLinks={[link("USER"), link("MEMORY")]} />);
+ const lines = screen.getAllByTestId("messenger-action");
+ expect(lines).toHaveLength(2);
+ expect(lines[0].textContent).toContain("Plan");
+ fireEvent.click(screen.getByRole("button", { name: "Memory updated · USER.md" }));
+ fireEvent.click(screen.getByRole("button", { name: "Memory updated · MEMORY.md" }));
+ expect(opened).toEqual(["USER", "MEMORY"]);
+ });
+});
+
+function routineDone(): ToolBlock {
+ return call("society_propose_change", { kind: "routine", payload: { title: "Daily contributor check" } }, "{\"applied\": true}");
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/MessengerTurn.tsx b/jarvis/ui/web/frontend/src/components/agentchat/MessengerTurn.tsx
new file mode 100644
index 0000000000..ec39dc1490
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/MessengerTurn.tsx
@@ -0,0 +1,151 @@
+import { memo, useEffect, useMemo, useState, type ReactNode } from "react";
+import { BookmarkPlus, CircleAlert, Clock3, Feather, NotebookPen, SlidersHorizontal } from "lucide-react";
+import { ChatMarkdown } from "./ChatMarkdown";
+import { CredentialCard } from "./CredentialCard";
+import { QuestionCard } from "./QuestionCard";
+import type { ApprovalDecision } from "@/lib/agentChatApi";
+import type { TurnItem } from "./reduce";
+import { actionNoticeOf, presenceOf, type ActionNotice } from "./messengerPresence";
+import { BotStage, useDoneFlourish, useTypingPresence } from "./botStage/BotStage";
+import { ApprovalPrompt, PROSE, THREAD_MEDIA } from "@/components/agentic/threads/ThreadTimeline";
+import { SubagentCard } from "@/components/agentic/threads/ThreadSubagents";
+import { isSubagentCall } from "@/components/agentic/threads/subagents";
+import { useT } from "@/i18n";
+import { cn } from "@/lib/utils";
+
+/**
+ * One agent turn read like a messenger conversation. The agent's messages
+ * are grey bubbles; what it changed for the person (a routine, a rule, a
+ * saved skill) is a quiet centred line between them; while it works, its
+ * own face plays a little scene of what it is doing on a stage below
+ * (BotStage) and hops once when the turn is done. No reasoning and no work
+ * log: the agent's own messages say what is happening, what is done and
+ * what comes next.
+ */
+export const MessengerTurn = memo(function MessengerTurn({ turn, avatar, color, onDecide, memoryLinks }: {
+ turn: TurnItem;
+ /** The agent's face at 30 px; it plays the scenes. */
+ avatar: ReactNode;
+ /** The agent's colour, for the accents of the scenes. */
+ color: string;
+ onDecide: (approvalId: string, decision: ApprovalDecision) => void | Promise<void>;
+ /** Memory receipts posted after the turn: each a "Noted" quill line that opens the file. */
+ memoryLinks?: MemoryLink[];
+}) {
+ const t = useT();
+ const textLength = useMemo(() => turn.blocks.reduce((n, block) => n + (block.kind === "text" ? block.text.length : 0), 0), [turn.blocks]);
+ const presence = useTypingPresence(presenceOf(turn), textLength);
+ const hasText = textLength > 0 && turn.blocks.some((block) => block.kind === "text" && block.text.trim());
+ const flourish = useDoneFlourish(turn.status === "running", turn.status === "done");
+ const stage = presence ?? (flourish ? "done" : null);
+ let noteIndex = 0;
+ return <div className="flex min-w-0 flex-col gap-1.5" data-testid="messenger-turn" data-status={turn.status}>
+ {turn.blocks.map((block) => {
+ if (block.kind === "text") {
+ return block.text.trim() ? <MessageBubble key={block.id} text={block.text} /> : null;
+ }
+ if (block.kind !== "tool") return null;
+ if (block.credential) return <CredentialCard key={block.callId} credential={block.credential} />;
+ if (block.question) return <QuestionCard key={block.callId} question={block.question} />;
+ if (block.approval && block.approval.decision === null) return <ApprovalPrompt key={block.callId} block={block} onDecide={onDecide} />;
+ if (isSubagentCall(block)) return <SubagentCard key={block.callId} block={block} turn={turn} />;
+ const notice = actionNoticeOf(block);
+ if (!notice) return null;
+ // The turn's own "Noted" lines open its memory receipts first, in order.
+ const link = notice.icon === "note" ? memoryLinks?.[noteIndex++] : undefined;
+ return <ActionLine key={block.callId} notice={notice} link={link} />;
+ })}
+ {memoryLinks?.slice(noteIndex).map((link) => <ActionLine key={`memory:${link.key}`} notice={NOTED} link={link} />)}
+ {stage && <BotStage presence={stage} seed={turn.id} avatar={avatar} color={color} />}
+ {turn.status === "error" && (
+ <p role="alert" className="mx-auto flex max-w-[85%] items-center gap-1.5 py-1 text-center text-xs text-destructive">
+ <CircleAlert aria-hidden className="h-3.5 w-3.5 shrink-0" />
+ <span>{turn.error || t("thread_turn.failed")}</span>
+ </p>
+ )}
+ {turn.status === "cancelled" && <SystemLine>{t("thread_turn.stopped")}</SystemLine>}
+ {turn.status === "done" && !hasText && !flourish && <SystemLine>{t("thread_turn.no_answer")}</SystemLine>}
+ </div>;
+});
+
+/** How long a sent message may wait for its turn before the stage stops reading it. */
+const AWAIT_MS = 90_000;
+
+/**
+ * The person's message is in and the agent's turn has not started yet: its
+ * face reads the message, so the chat never sits silent after a send.
+ */
+export function AwaitingTurn({ since, seed, avatar, color }: { since: number; seed: string; avatar: ReactNode; color: string }) {
+ const [, tick] = useState(0);
+ const left = AWAIT_MS - (Date.now() - since);
+ const waiting = left > 0;
+ useEffect(() => {
+ if (!waiting) return;
+ const id = window.setTimeout(() => tick((n) => n + 1), left + 50);
+ return () => window.clearTimeout(id);
+ // `left` shrinks every render; only the start of the wait matters.
+ // eslint-disable-next-line react-hooks/exhaustive-deps
+ }, [waiting, since]);
+ return waiting ? <BotStage presence="reading" seed={seed} avatar={avatar} color={color} /> : null;
+}
+
+/** One message of the agent's: a grey bubble on the left. */
+export function MessageBubble({ text }: { text: string }) {
+ return <div className="flex min-w-0" data-testid="agent-message-bubble">
+ <div className="jarvis-chat-in min-w-0 max-w-[min(85%,42rem)] rounded-[20px] px-4 py-2.5">
+ <div className={cn(PROSE, THREAD_MEDIA, "py-0")}><ChatMarkdown text={text} /></div>
+ </div>
+ </div>;
+}
+
+function SystemLine({ children }: { children: ReactNode }) {
+ return <p className="py-1 text-center text-xs text-muted-foreground">{children}</p>;
+}
+
+const NOTICE_ICON: Record<ActionNotice["icon"], typeof Clock3> = {
+ routine: Clock3,
+ rule: BookmarkPlus,
+ skill: NotebookPen,
+ settings: SlidersHorizontal,
+ note: Feather,
+};
+
+export interface MemoryLink {
+ key: string;
+ /** What the line opens ("Memory updated · USER.md"), as its tooltip. */
+ title: string;
+ onOpen: () => void;
+}
+
+const NOTED: ActionNotice = { key: "note_saved", subject: "", icon: "note" };
+
+/** A "Noted" quill line that opens a memory file, for a receipt with no turn of its own. */
+export function NotedLine({ link }: { link: MemoryLink }) {
+ return <ActionLine notice={NOTED} link={link} />;
+}
+
+/**
+ * "Created: Routine ⏱ Daily contributor check" — centred, quiet, not a
+ * message. With a memory link it is a button that opens the changed file.
+ */
+function ActionLine({ notice, link }: { notice: ActionNotice; link?: MemoryLink }) {
+ const t = useT();
+ const Icon = NOTICE_ICON[notice.icon];
+ const body = <>
+ <span>{t(`messenger_turn.${notice.key}`)}</span>
+ <Icon aria-hidden className="h-3.5 w-3.5 shrink-0" />
+ {notice.subject ? <span className="text-foreground-secondary">{notice.subject}</span> : null}
+ </>;
+ if (link) {
+ return <div className="flex justify-center py-1">
+ <button type="button" onClick={link.onOpen} title={link.title} aria-label={link.title}
+ className="flex flex-wrap items-center justify-center gap-x-1.5 rounded-md px-1.5 text-center text-xs text-muted-foreground transition-colors hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring"
+ data-testid="messenger-action" data-memory-link={link.key}>
+ {body}
+ </button>
+ </div>;
+ }
+ return <p className="flex flex-wrap items-center justify-center gap-x-1.5 py-1 text-center text-xs text-muted-foreground" data-testid="messenger-action">
+ {body}
+ </p>;
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/WorkTrace.tsx b/jarvis/ui/web/frontend/src/components/agentchat/WorkTrace.tsx
index 1ac5168774..296e768d4a 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/WorkTrace.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/WorkTrace.tsx
@@ -3,12 +3,14 @@ import { Brain, Check, ChevronRight, CircleAlert, CircleDashed, FilePenLine, Fil
import ReactMarkdown from "react-markdown";
import remarkGfm from "remark-gfm";
import { useT } from "@/i18n";
+import { PetStageContext } from "@/components/pets/petStage";
import { PetMark } from "@/components/pets/PetMark";
import type { PetState } from "@/lib/petStates";
import { cn } from "@/lib/utils";
import type { ApprovalDecision } from "@/lib/agentChatApi";
-import { isQuestionTool, type ReasoningBlock, type TextBlock, type ToolBlock, type TurnBlock, type TurnItem, type TurnStatus } from "./reduce";
+import { hasPersonCard, isCredentialTool, isQuestionTool, waitsOnCard, type ReasoningBlock, type TextBlock, type ToolBlock, type TurnBlock, type TurnItem, type TurnStatus } from "./reduce";
import { ChatMarkdown } from "./ChatMarkdown";
+import { CredentialCard } from "./CredentialCard";
import { QuestionCard } from "./QuestionCard";
import { toolDiff } from "./toolDiff";
import { formatTokens, outputTokens } from "./toolView";
@@ -57,12 +59,12 @@ function operation(name: string): string | null {
}
function attention(block: ToolBlock) {
- return block.isError || Boolean(block.approval) || Boolean(block.question);
+ return block.isError || Boolean(block.approval) || hasPersonCard(block);
}
/** An agent's question still waiting for the person — it never folds away. */
function isOpenQuestion(block: TurnBlock): block is ToolBlock {
- return block.kind === "tool" && Boolean(block.question && !block.question.closed);
+ return block.kind === "tool" && waitsOnCard(block);
}
/** Only adjacent, successful, read-only operations may lose individual rows. */
@@ -401,6 +403,7 @@ function Detail({ label, text }: { label: string; text: string }) {
export const TraceTool = memo(function TraceTool({ block, status, onDecide }: { block: ToolBlock; status: TurnStatus; onDecide?: Decide }) {
if (block.question) return <QuestionCard question={block.question} />;
+ if (block.credential) return <CredentialCard credential={block.credential} />;
return <TraceToolRow block={block} status={status} onDecide={onDecide} />;
});
@@ -554,7 +557,7 @@ function traceGroupItems({ groups, live, status, onDecide, renderText, conversat
{rail ? <Rail items={group.blocks.map((block, i) => ({
key: block.kind === "tool" ? block.callId : block.id,
node: inner[i],
- rail: !(block.kind === "tool" && block.question),
+ rail: !(block.kind === "tool" && hasPersonCard(block)),
}))} /> : inner}
</Disclosure>;
items.push({ key: group.id, rail: true, node: rail ? <div data-trace-summary>{disclosure}</div>
@@ -574,7 +577,7 @@ function traceGroupItems({ groups, live, status, onDecide, renderText, conversat
}
if (first.kind === "tool") {
const tool = <TraceTool block={first} status={status} onDecide={onDecide} />;
- items.push({ key: group.id, rail: !first.question, node: rail ? tool
+ items.push({ key: group.id, rail: !hasPersonCard(first), node: rail ? tool
: <div className={conversation ? "w-full py-1 text-xs [&_button]:text-xs" : undefined}>{tool}</div> });
continue;
}
@@ -602,7 +605,8 @@ function replyNode(block: TextBlock, conversation: boolean, renderText?: (text:
* and draw nothing.
*/
function withoutQuestionPolls(blocks: TurnBlock[]): TurnBlock[] {
- const kept = blocks.filter((block) => !(block.kind === "tool" && isQuestionTool(block.name) && !block.question && !block.isError));
+ const kept = blocks.filter((block) => !(block.kind === "tool" && !block.isError
+ && ((isQuestionTool(block.name) && !block.question) || (isCredentialTool(block.name) && !block.credential))));
return kept.length === blocks.length ? blocks : kept;
}
@@ -622,6 +626,12 @@ type WorkTraceProps = {
* and coding-pane traces keep the plain node.
*/
companion?: boolean;
+ /**
+ * An agent's own trace: its face takes the live line instead, and a face
+ * that wears a pet plays the row for what the agent is doing
+ * (PetStageContext). Wins over `companion`.
+ */
+ face?: ReactNode;
/**
* Steps that belong to this turn's work but are not blocks of it (a
* memory receipt posted after the turn). They sit on the trace above the
@@ -630,7 +640,7 @@ type WorkTraceProps = {
extras?: { key: string; node: ReactNode }[];
};
-function WorkTraceBody({ blocks: rawBlocks, status, startedMs, durationMs, error, onDecide, renderText, className, receipt, completionLabel, conversation = false, companion = false, extras }: WorkTraceProps) {
+function WorkTraceBody({ blocks: rawBlocks, status, startedMs, durationMs, error, onDecide, renderText, className, receipt, completionLabel, conversation = false, companion = false, face, extras }: WorkTraceProps) {
const t = useT();
const rail = useRail();
const blocks = useMemo(() => withoutQuestionPolls(rawBlocks), [rawBlocks]);
@@ -683,12 +693,14 @@ function WorkTraceBody({ blocks: rawBlocks, status, startedMs, durationMs, error
// whenever the trace has no reply between it and the work); a finished
// conversation turn closes with a quiet line under its reply instead.
const statusOnRail = !fold && (!conversation || live);
- const petState = companion && working ? livePetState(blocks) : null;
+ const petState = (companion || face) && working ? livePetState(blocks) : null;
const statusLine = <div role="status" aria-live="polite" data-trace-status={outcome}
className={cn("flex min-w-0 flex-wrap items-start gap-x-3 text-xs leading-6 text-muted-foreground",
statusOnRail ? "py-1" : "pb-2 pt-1", failed && "text-destructive", pending && "text-foreground")}>
{petState
- ? <span aria-hidden className="trace-node trace-node-pet" data-trace-pet={petState}><PetMark size={32} state={petState} /></span>
+ ? <span aria-hidden className="trace-node trace-node-pet" data-trace-pet={petState}>{face
+ ? <PetStageContext.Provider value={petState}>{face}</PetStageContext.Provider>
+ : <PetMark size={32} state={petState} />}</span>
: null}
<span className="inline-flex min-w-0 flex-1 flex-wrap items-center gap-x-2">
<span><Live on={working}>{outcomeLabel}</Live></span>
@@ -705,7 +717,7 @@ function WorkTraceBody({ blocks: rawBlocks, status, startedMs, durationMs, error
renderLive: (block) => block.kind === "tool" ? <TraceTool block={block} status={status} onDecide={onDecide} />
: block.kind === "reasoning" ? <ReasoningTrace block={block} turnLive={live} compact={conversation} />
: replyNode(block, conversation, renderText),
- liveOnRail: (block) => block.kind === "reasoning" || (block.kind === "tool" && !block.question),
+ liveOnRail: (block) => block.kind === "reasoning" || (block.kind === "tool" && !hasPersonCard(block)),
renderNarration: (text, id) => renderText ? renderText(text, id)
: <div className="prose prose-sm max-w-none text-foreground dark:prose-invert [overflow-wrap:anywhere]"><ChatMarkdown text={text} /></div>,
renderDetails: (block) => <ToolDetails block={block} />,
@@ -739,7 +751,7 @@ function WorkTraceBody({ blocks: rawBlocks, status, startedMs, durationMs, error
return <div className={cn("min-w-0", conversation && "w-full max-w-[44rem] self-start", className)} data-testid="work-trace" data-look="rail" data-state={status} {...(conversation ? { "data-conversation": "" } : {})}>
<ConversationWorkFold durationMs={durationMs} expandInner={false}
failureCount={waiting.filter((block) => block.isError).length} attention={waiting.length ? <Rail items={waiting.map((block) => ({
- key: block.callId, rail: !block.question, node: <TraceTool block={block} status={status} onDecide={onDecide} />,
+ key: block.callId, rail: !hasPersonCard(block), node: <TraceTool block={block} status={status} onDecide={onDecide} />,
}))} /> : undefined}>
<Rail items={workItems} />
</ConversationWorkFold>
@@ -781,7 +793,7 @@ function WorkTraceBody({ blocks: rawBlocks, status, startedMs, durationMs, error
</div>;
}
-export function TurnTrace({ turn, ...props }: { turn: TurnItem; onDecide?: Decide; renderText?: (text: string, id: string) => ReactNode; conversation?: boolean; look?: TraceLook; companion?: boolean; extras?: WorkTraceProps["extras"] }) {
+export function TurnTrace({ turn, ...props }: { turn: TurnItem; onDecide?: Decide; renderText?: (text: string, id: string) => ReactNode; conversation?: boolean; look?: TraceLook; companion?: boolean; face?: ReactNode; extras?: WorkTraceProps["extras"] }) {
const t = useT();
const tokens = outputTokens(turn.usage ?? turn.liveUsage);
const answered = turn.blocks.some(block => block.kind === "text" && block.text.trim());
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/botStage/BotStage.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/botStage/BotStage.test.tsx
new file mode 100644
index 0000000000..c2f3a7ef66
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/botStage/BotStage.test.tsx
@@ -0,0 +1,128 @@
+import { useContext } from "react";
+import { act, cleanup, render, screen } from "@testing-library/react";
+import { PetStageContext } from "@/components/pets/petStage";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { BotStage, useDoneFlourish } from "./BotStage";
+import { DONE_MS, MIN_SCENE_MS, PET_STATE, SCENES, THINK_ROTATE_MS, sceneFor } from "./scenes";
+import type { Presence } from "../messengerPresence";
+
+afterEach(cleanup);
+
+describe("sceneFor", () => {
+ it("offers many thinking scenes and one for every presence", () => {
+ expect(SCENES.thinking.length).toBeGreaterThanOrEqual(10);
+ for (const presence of Object.keys(SCENES) as Presence[]) expect(SCENES[presence].length).toBeGreaterThan(0);
+ });
+
+ it("is stable per turn and varies across turns", () => {
+ expect(sceneFor("thinking", "turn-a")).toBe(sceneFor("thinking", "turn-a"));
+ const seen = new Set(Array.from({ length: 40 }, (_, i) => sceneFor("thinking", `turn-${i}`)));
+ expect(seen.size).toBeGreaterThanOrEqual(6);
+ });
+
+ it("never repeats a scene from one rotation step to the next", () => {
+ for (let step = 0; step < 20; step++) {
+ expect(sceneFor("thinking", "t", step)).not.toBe(sceneFor("thinking", "t", step + 1));
+ }
+ });
+});
+
+describe("BotStage", () => {
+ beforeEach(() => vi.useFakeTimers());
+ afterEach(() => vi.useRealTimers());
+
+ const face = <span data-testid="face" />;
+ const scene = () => screen.getByTestId("messenger-presence").dataset.presence;
+ const sceneName = () => document.querySelector(".bot-stage")?.getAttribute("data-scene");
+
+ it("holds a scene long enough that a quick call does not flicker", () => {
+ const { rerender } = render(<BotStage presence="thinking" seed="t" avatar={face} color="#7ab6ef" />);
+ rerender(<BotStage presence="searching" seed="t" avatar={face} color="#7ab6ef" />);
+ rerender(<BotStage presence="thinking" seed="t" avatar={face} color="#7ab6ef" />);
+ act(() => { vi.advanceTimersByTime(MIN_SCENE_MS + 10); });
+ expect(SCENES.thinking).toContain(sceneName());
+ rerender(<BotStage presence="writing" seed="t" avatar={face} color="#7ab6ef" />);
+ expect(SCENES.thinking).toContain(sceneName());
+ act(() => { vi.advanceTimersByTime(MIN_SCENE_MS + 10); });
+ expect(SCENES.writing).toContain(sceneName());
+ expect(scene()).toBe("writing");
+ });
+
+ it("rotates a long think through different scenes", () => {
+ render(<BotStage presence="thinking" seed="t" avatar={face} color="#7ab6ef" />);
+ const first = sceneName();
+ act(() => { vi.advanceTimersByTime(THINK_ROTATE_MS + 10); });
+ expect(sceneName()).not.toBe(first);
+ expect(SCENES.thinking).toContain(sceneName());
+ });
+
+ it("crossfades a scene change instead of cutting to an empty frame", () => {
+ render(<BotStage presence="thinking" seed="t" avatar={face} color="#7ab6ef" />);
+ const first = sceneName()!;
+ act(() => { vi.advanceTimersByTime(THINK_ROTATE_MS + 10); });
+ const next = sceneName()!;
+ const faces = () => Array.from(document.querySelectorAll(".bs-face")).map((el) => [el.getAttribute("data-scene"), el.classList.contains("is-out")]);
+ const words = () => Array.from(document.querySelectorAll(".bs-word")).map((el) => el.classList.contains("is-out"));
+ expect(faces()).toEqual([[first, true], [next, false]]);
+ expect(words()).toEqual([true, false]);
+ act(() => { vi.advanceTimersByTime(600); });
+ expect(faces()).toEqual([[next, false]]);
+ expect(words()).toEqual([false]);
+ });
+
+ it("asks a face that wears a pet to play the scene's pet row", () => {
+ function Row() { return <span data-testid="row">{useContext(PetStageContext)}</span>; }
+ render(<BotStage presence="searching" seed="t" avatar={<Row />} color="#7ab6ef" />);
+ expect(screen.getByTestId("row").textContent).toBe("searching");
+ expect(PET_STATE[sceneName() as keyof typeof PET_STATE]).toBe("searching");
+ expect(PET_STATE.typing).toBe("talking");
+ expect(PET_STATE.done).toBe("success");
+ });
+
+ it("plays the agent's own face", () => {
+ render(<BotStage presence="typing" seed="t" avatar={face} color="#7ab6ef" />);
+ expect(screen.getByTestId("face")).toBeTruthy();
+ });
+});
+
+describe("useDoneFlourish", () => {
+ beforeEach(() => vi.useFakeTimers());
+ afterEach(() => vi.useRealTimers());
+
+ function Probe({ running, done }: { running: boolean; done: boolean }) {
+ return <span data-testid="flourish">{String(useDoneFlourish(running, done))}</span>;
+ }
+
+ it("cheers once when a live turn finishes, then clears", () => {
+ const { rerender } = render(<Probe running done={false} />);
+ rerender(<Probe running={false} done />);
+ expect(screen.getByTestId("flourish").textContent).toBe("true");
+ act(() => { vi.advanceTimersByTime(DONE_MS + 10); });
+ expect(screen.getByTestId("flourish").textContent).toBe("false");
+ });
+
+ it("stays quiet for a turn that was already finished when it loaded", () => {
+ render(<Probe running={false} done />);
+ expect(screen.getByTestId("flourish").textContent).toBe("false");
+ });
+});
+
+describe("expressive faces", () => {
+ it("give each eye an open, happy and shut shape the stage can swap", async () => {
+ const { AgentSymbol } = await import("@/components/society/AgentSymbol");
+ const { container } = render(<AgentSymbol shape="circle" color="#79c7c4" size={28} expressive />);
+ for (const side of ["l", "r"]) {
+ const eye = container.querySelector(`.ae-${side}`);
+ expect(eye?.querySelector(".ae-open")).not.toBeNull();
+ expect(eye?.querySelector(".ae-happy")).not.toBeNull();
+ expect(eye?.querySelector(".ae-shut")).not.toBeNull();
+ }
+ });
+
+ it("leave an ordinary face exactly as it was", async () => {
+ const { AgentSymbol } = await import("@/components/society/AgentSymbol");
+ const { container } = render(<AgentSymbol shape="circle" color="#79c7c4" size={28} />);
+ expect(container.querySelector(".ae-eye")).toBeNull();
+ expect(container.querySelectorAll(".agent-symbol-lids ellipse")).toHaveLength(2);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/botStage/BotStage.tsx b/jarvis/ui/web/frontend/src/components/agentchat/botStage/BotStage.tsx
new file mode 100644
index 0000000000..6889e4be41
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/botStage/BotStage.tsx
@@ -0,0 +1,202 @@
+import { useEffect, useRef, useState, type CSSProperties, type ReactNode } from "react";
+import { useT } from "@/i18n";
+import type { Presence } from "../messengerPresence";
+import { DONE_MS, MIN_SCENE_MS, PET_STATE, THINK_ROTATE_MS, sceneFor, type Scene } from "./scenes";
+import { PetStageContext } from "@/components/pets/petStage";
+import { bubbleTint } from "@/lib/bubbleTint";
+import "./botStage.css";
+
+/**
+ * The agent at work, the way a messenger shows someone typing: its own face
+ * and a word ("Thinking…", "Typing…"). The face carries the scene — its eyes
+ * look up and around, squint, widen at an idea, wink, smile, read line by
+ * line, turn to dashes while it types — and a tool scene adds one tiny prop
+ * beside it (a quill writing, a magnifier, a gear). The face must be drawn
+ * with `expressive` eyes (AgentSwatch) for the expressions to play; an agent
+ * that wears a pet plays the scene's pet row instead (PET_STATE).
+ *
+ * A scene never flashes: it plays at least MIN_SCENE_MS before the next one
+ * takes over, so a call that lasts 80 ms does not flicker. A long think
+ * rotates through the thinking expressions every THINK_ROTATE_MS. Changing
+ * scene never cuts: the outgoing face, prop and word stay mounted, still
+ * playing their scene, and fade off over the incoming ones (Crossfade).
+ */
+export function BotStage({ presence, seed, avatar, color }: {
+ presence: Presence;
+ /** Stable per turn: picks which scene of a family this turn plays. */
+ seed: string;
+ /** The agent's face (AgentSwatch with `expressive` eyes). */
+ avatar: ReactNode;
+ /** The agent's colour, for the accents of the props. */
+ color: string;
+}) {
+ const t = useT();
+ const scene = useSceneDirector(presence, seed);
+ const label = t(`bot_stage.${scene}`);
+ // The tick on the finished turn's badge takes the ink that reads on the agent's colour.
+ const ink = bubbleTint(color)?.color ?? "#ffffff";
+ return <div className="bot-stage" data-scene={scene} style={{ "--bot": color, "--bot-ink": ink } as CSSProperties}
+ role="status" aria-label={label} data-testid="messenger-presence" data-presence={presence}>
+ <span className="bs-set" aria-hidden>
+ <Crossfade id={scene} className="bs-face">
+ {/* A face that wears a pet plays the scene's pet row instead of moving its eyes. */}
+ <span className="bs-body"><PetStageContext.Provider value={PET_STATE[scene]}>{avatar}</PetStageContext.Provider></span>
+ {OVER[scene] && <svg className="bs-over" viewBox="0 0 28 28" width="28" height="28">{OVER[scene]}</svg>}
+ </Crossfade>
+ </span>
+ <span className="bs-props" data-empty={PROP[scene] ? undefined : ""} aria-hidden>
+ <Crossfade id={scene}>
+ {PROP[scene] && <svg className="bs-prop" viewBox="0 0 20 20" width="20" height="20">{PROP[scene]}</svg>}
+ </Crossfade>
+ </span>
+ <span className="bs-label"><Crossfade id={scene} className="bs-word">{label}</Crossfade></span>
+ </div>;
+}
+
+/** How long an outgoing layer stays mounted; the CSS fades it within `--bs-swap`. */
+const SWAP_MS = 480;
+
+interface Layer { id: string; node: ReactNode }
+
+/**
+ * Renders `children` as the layer for `id`, and keeps the layers of the last
+ * ids mounted for SWAP_MS after `id` changes, marked `is-out`, so the CSS can
+ * fade them over the incoming one. The outgoing layer keeps its element (one
+ * keyed list), so its animations play on instead of restarting. The incoming
+ * layer is marked `is-in` only after the first change: the stage's own entry
+ * already covers its first scene.
+ */
+function Crossfade({ id, className, children }: { id: string; className?: string; children: ReactNode }) {
+ const [current, setCurrent] = useState(id);
+ const [leaving, setLeaving] = useState<Layer[]>([]);
+ const [swapped, setSwapped] = useState(false);
+ const shown = useRef(children);
+ if (current !== id) {
+ // Adjusting state while rendering: the outgoing layer must be in this very
+ // frame, or it would vanish for one paint before the fade begins.
+ setCurrent(id);
+ setSwapped(true);
+ setLeaving((gone) => [...gone.filter((layer) => layer.id !== id && layer.id !== current), { id: current, node: shown.current }].slice(-2));
+ }
+ useEffect(() => { shown.current = children; });
+ useEffect(() => {
+ if (leaving.length === 0) return;
+ const timer = window.setTimeout(() => setLeaving([]), SWAP_MS);
+ return () => window.clearTimeout(timer);
+ }, [leaving]);
+ const layers = [...leaving.filter((layer) => layer.id !== id).map((layer) => ({ ...layer, out: true })), { id, node: children, out: false }];
+ return <>{layers.map((layer) => <span key={layer.id} data-scene={layer.id} aria-hidden={layer.out || undefined}
+ className={["bs-scene", className, layer.out ? "is-out" : swapped ? "is-in" : ""].filter(Boolean).join(" ")}>{layer.node}</span>)}</>;
+}
+
+/** Holds each scene long enough to read, and rotates a long think. */
+function useSceneDirector(presence: Presence, seed: string): Scene {
+ const latest = useRef(presence);
+ latest.current = presence;
+ const [shown, setShown] = useState({ presence, step: 0, phase: 0, at: Date.now() });
+ useEffect(() => {
+ if (presence === shown.presence) return;
+ const wait = Math.max(0, MIN_SCENE_MS - (Date.now() - shown.at));
+ const id = window.setTimeout(() => setShown((s) => s.presence === latest.current
+ ? s : { presence: latest.current, step: 0, phase: s.phase + 1, at: Date.now() }), wait);
+ return () => window.clearTimeout(id);
+ }, [presence, shown]);
+ useEffect(() => {
+ if (shown.presence !== "thinking" && shown.presence !== "working") return;
+ const id = window.setInterval(() => setShown((s) => ({ ...s, step: s.step + 1, at: Date.now() })), THINK_ROTATE_MS);
+ return () => window.clearInterval(id);
+ }, [shown.presence, shown.phase]);
+ return sceneFor(shown.presence, `${seed}:${shown.phase}`, shown.step);
+}
+
+/** Keeps a finished turn's flourish on stage for DONE_MS after it ends live. */
+export function useDoneFlourish(running: boolean, finishedWell: boolean): boolean {
+ const was = useRef(running);
+ const [show, setShow] = useState(false);
+ useEffect(() => {
+ if (was.current && !running && finishedWell) {
+ setShow(true);
+ const id = window.setTimeout(() => setShow(false), DONE_MS);
+ was.current = running;
+ return () => window.clearTimeout(id);
+ }
+ was.current = running;
+ return undefined;
+ }, [running, finishedWell]);
+ return show;
+}
+
+/** "Typing" only while the words are still arriving; a pause is thinking again. */
+export function useTypingPresence(presence: Presence | null, textLength: number): Presence | null {
+ const [grewAt, setGrewAt] = useState(0);
+ const [, tick] = useState(0);
+ const last = useRef(textLength);
+ useEffect(() => {
+ if (textLength !== last.current) {
+ last.current = textLength;
+ setGrewAt(Date.now());
+ }
+ }, [textLength]);
+ const fresh = Date.now() - grewAt < 1200;
+ useEffect(() => {
+ if (presence !== "typing" || !fresh) return;
+ const id = window.setTimeout(() => tick((n) => n + 1), 1250);
+ return () => window.clearTimeout(id);
+ }, [presence, fresh, grewAt]);
+ return presence === "typing" && !fresh ? "thinking" : presence;
+}
+
+const FEATHER = "M12.67 19a2 2 0 0 0 1.416-.588l6.154-6.172a6 6 0 0 0-8.49-8.49L5.586 9.914A2 2 0 0 0 5 11.328V18a1 1 0 0 0 1 1z M16 8 2 22 M17.5 15H9";
+const POINTER = "M4.037 4.688a.495.495 0 0 1 .651-.651l16 6.5a.5.5 0 0 1-.063.947l-6.124 1.58a2 2 0 0 0-1.438 1.435l-1.579 6.126a.5.5 0 0 1-.947.063z";
+const GEAR = "M10 2.2l1.2 1.9 2.2-.5.5 2.2 1.9 1.2-1.2 1.9 1.2 1.9-1.9 1.2-.5 2.2-2.2-.5L10 17.8l-1.2-1.9-2.2.5-.5-2.2-1.9-1.2 1.2-1.9-1.2-1.9 1.9-1.2.5-2.2 2.2.5z";
+
+/** One tiny prop beside the face, in a 20 x 20 box: the tool, never a scene of its own. */
+const PROP: Partial<Record<Scene, ReactNode>> = {
+ note: <>
+ <path className="bs-ink" d="M2 16.5c2-2 3.5 1.5 5.5-.5s3.5 1.5 5.5-.5" />
+ <g className="bs-quill"><path className="bs-line" d={FEATHER} transform="scale(.62)" /></g>
+ </>,
+ jot: <>
+ <rect className="bs-paper" x="3.5" y="2.5" width="13" height="15" rx="2" />
+ <path className="bs-jot bs-j1" d="M6.5 7H13.5" /><path className="bs-jot bs-j2" d="M6.5 10.5H12.5" /><path className="bs-jot bs-j3" d="M6.5 14H10.5" />
+ </>,
+ scan: <g className="bs-lens"><circle cx="8.5" cy="8.5" r="4.6" /><path d="M12 12l4.5 4.5" /></g>,
+ flip: <>
+ <rect className="bs-paper" x="5" y="3" width="11" height="14" rx="1.6" />
+ <g className="bs-page"><rect x="5" y="3" width="11" height="14" rx="1.6" /><path d="M7.5 7h6M7.5 10h5M7.5 13h6" /></g>
+ </>,
+ browser: <>
+ <rect className="bs-paper" x="1.5" y="3" width="17" height="13" rx="2" />
+ <path className="bs-line" d="M1.5 6.5h17" />
+ <rect className="bs-button" x="9" y="10.5" width="7" height="3" rx="1.5" />
+ <g className="bs-pointer"><path className="bs-pointer-shape" d={POINTER} transform="scale(.42)" /></g>
+ </>,
+ setup: <g className="bs-gear"><path d={GEAR} /><circle cx="10" cy="10" r="2.3" /></g>,
+ tune: <>
+ <path className="bs-track" d="M2 7H18M2 13H18" />
+ <circle className="bs-knob bs-knob1" cx="0" cy="7" r="2.2" /><circle className="bs-knob bs-knob2" cx="0" cy="13" r="2.2" />
+ </>,
+ command: <>
+ <path className="bs-prompt" d="M2.5 6.5l3 3-3 3" />
+ <rect className="bs-typed" x="7.5" y="8.4" width="9" height="2.2" rx="1.1" />
+ <rect className="bs-caret" x="0" y="7.6" width="1.8" height="3.8" rx=".4" />
+ </>,
+ build: <>
+ <rect className="bs-paper" x="8" y="13" width="10" height="5" rx="1" />
+ <g className="bs-hammer"><path d="M3 8.5H13" /><rect x="11" y="5" width="4" height="7" rx="1" /></g>
+ <g className="bs-spark"><path d="M15.5 10.5l2-2M17 12.5l2.4-.6" /></g>
+ </>,
+};
+
+/** Drawn over the face (28 x 28, the face's own box): what happens on or above its head. */
+const OVER: Partial<Record<Scene, ReactNode>> = {
+ juggle: <>
+ <circle className="bs-ball bs-b1" cx="0" cy="0" r="2.2" />
+ <circle className="bs-ball bs-b2" cx="0" cy="0" r="2.2" />
+ </>,
+ recall: <>
+ <circle className="bs-puff bs-p1" cx="24" cy="5.5" r="1" />
+ <circle className="bs-puff bs-p2" cx="26.5" cy="2" r="1.6" />
+ </>,
+ done: <g className="bs-check"><circle cx="23.5" cy="23.5" r="5" /><path d="M21.2 23.6l1.6 1.6 3-3.2" /></g>,
+};
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/botStage/botStage.css b/jarvis/ui/web/frontend/src/components/agentchat/botStage/botStage.css
new file mode 100644
index 0000000000..5da6088e6d
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/botStage/botStage.css
@@ -0,0 +1,563 @@
+/*
+ * The bot stage: the agent's own face and a word, the way a messenger shows
+ * someone typing. The face carries every scene through its expressive eyes
+ * (AgentSymbol `expressive`): `.ae-eye` groups move and scale, `.ae-open`,
+ * `.ae-happy` and `.ae-shut` swap the eye's shape. Eye units are the symbol's
+ * own viewBox units (a face is ~40 units wide), so a 2px translate is a
+ * glance, not a jump. Tool scenes add one tiny prop in a 20 x 20 box beside
+ * the face. Reduced motion keeps the face still and every prop at rest.
+ *
+ * A scene change never cuts: the face, the prop and the word each stack the
+ * outgoing scene over the incoming one (`.bs-scene`, `.is-out`) and fade it
+ * away, so the eyes ease into their next pose and the word slides over.
+ * Scene rules therefore hang on the layer's own `.bs-scene[data-scene]`,
+ * never on the stage: the outgoing face keeps playing its own scene.
+ */
+
+.bot-stage {
+ --bot: hsl(var(--muted-foreground));
+ --bs-swap: 0.42s;
+ display: flex;
+ align-items: center;
+ gap: 8px;
+ min-height: 36px;
+ padding: 2px 0;
+ color: hsl(var(--muted-foreground));
+}
+
+.bs-set {
+ display: inline-grid;
+ flex: none;
+ width: 28px;
+ height: 28px;
+ animation: bs-enter 0.3s cubic-bezier(0.2, 0.8, 0.2, 1) both;
+}
+@keyframes bs-enter {
+ from { opacity: 0; transform: scale(0.85); }
+ to { opacity: 1; transform: none; }
+}
+/* The incoming face sits still underneath; the outgoing one fades off on top. */
+.bs-face {
+ grid-area: 1 / 1;
+ position: relative;
+ width: 28px;
+ height: 28px;
+}
+.bs-face.is-out {
+ z-index: 1;
+ animation: bs-fade-out var(--bs-swap) ease-in-out both;
+}
+@keyframes bs-fade-out { to { opacity: 0; } }
+.bs-body {
+ display: block;
+ width: 28px;
+ height: 28px;
+ transform-origin: 50% 90%;
+ animation: bs-breathe 2.8s ease-in-out infinite;
+}
+.bs-body > * { display: block; width: 28px !important; height: 28px !important; }
+.bs-body svg { width: 28px; height: 28px; }
+
+.bs-over {
+ position: absolute;
+ inset: 0;
+ overflow: visible;
+ pointer-events: none;
+}
+/* The prop's slot opens and closes, so the word glides aside instead of jumping. */
+.bs-props {
+ display: inline-grid;
+ grid-template-columns: 20px;
+ flex: none;
+ width: 20px;
+ height: 20px;
+ margin-left: -2px;
+ transition: width var(--bs-swap) cubic-bezier(0.2, 0.8, 0.2, 1), margin-left var(--bs-swap) cubic-bezier(0.2, 0.8, 0.2, 1);
+}
+.bs-props[data-empty] { width: 0; margin-left: -8px; }
+.bs-props > .bs-scene { grid-area: 1 / 1; display: block; width: 20px; height: 20px; }
+.bs-prop { display: block; overflow: visible; }
+.bs-props > .bs-scene:not(.is-out) .bs-prop { animation: bs-enter 0.3s 0.14s cubic-bezier(0.2, 0.8, 0.2, 1) both; }
+.bs-props > .is-out { animation: bs-prop-out var(--bs-swap) ease-in both; }
+@keyframes bs-prop-out { to { opacity: 0; transform: scale(0.85); } }
+.bs-prop *, .bs-over * { transform-box: view-box; }
+
+@keyframes bs-breathe {
+ 0%, 100% { transform: translateY(0) scale(1, 1); }
+ 50% { transform: translateY(-0.6px) scale(1.015, 0.985); }
+}
+
+/* Every scene blinks now and then, unless its eyes are busy being shut. */
+.bot-stage .agent-symbol-lids {
+ transform-box: fill-box;
+ transform-origin: center;
+ animation: bs-blink 4.6s ease-in-out infinite;
+}
+@keyframes bs-blink {
+ 0%, 91%, 100% { transform: scaleY(1); }
+ 94% { transform: scaleY(0.1); }
+ 97% { transform: scaleY(1); }
+}
+.bot-stage :is(.ae-eye, .ae-open, .ae-happy, .ae-shut) { transition: opacity 0.2s; }
+
+/* ---- thinking: ponder — up to the left, a blink, up to the right ---------- */
+
+.bs-scene[data-scene="ponder"] .ae-eye { animation: bs-ponder 3.6s ease-in-out infinite; }
+.bs-scene[data-scene="ponder"] .bs-body { animation: bs-tilt 3.6s ease-in-out infinite; }
+@keyframes bs-ponder {
+ 0%, 8%, 100% { transform: translate(0, 0); }
+ 22%, 44% { transform: translate(-2px, -2px); }
+ 60%, 86% { transform: translate(2px, -1.8px); }
+}
+@keyframes bs-tilt {
+ 0%, 100% { transform: rotate(0); }
+ 30% { transform: rotate(-5deg); }
+ 70% { transform: rotate(5deg); }
+}
+
+/* ---- thinking: glance — quick looks to either side ------------------------ */
+
+.bs-scene[data-scene="glance"] .ae-eye { animation: bs-glance 2.6s cubic-bezier(0.7, 0, 0.3, 1) infinite; }
+.bs-scene[data-scene="glance"] .bs-body { animation: bs-glance-body 2.6s cubic-bezier(0.7, 0, 0.3, 1) infinite; }
+@keyframes bs-glance {
+ 0%, 12%, 100% { transform: translate(0, 0); }
+ 18%, 40% { transform: translate(2.6px, 0.2px); }
+ 48%, 70% { transform: translate(-2.6px, 0.2px); }
+ 78%, 92% { transform: translate(0, -0.6px); }
+}
+@keyframes bs-glance-body {
+ 0%, 12%, 78%, 100% { transform: rotate(0); }
+ 18%, 40% { transform: rotate(3deg); }
+ 48%, 70% { transform: rotate(-3deg); }
+}
+
+/* ---- thinking: squint — eyes narrow in concentration, then relax ---------- */
+
+.bs-scene[data-scene="squint"] .ae-open { animation: bs-squint 3.2s ease-in-out infinite; }
+.bs-scene[data-scene="squint"] .ae-eye { animation: bs-squint-eye 3.2s ease-in-out infinite; }
+.bs-scene[data-scene="squint"] .bs-body { animation: bs-squint-body 3.2s ease-in-out infinite; }
+@keyframes bs-squint {
+ 0%, 14%, 86%, 100% { transform: scaleY(1); }
+ 26%, 72% { transform: scaleY(0.32) scaleX(1.15); }
+}
+@keyframes bs-squint-eye {
+ 0%, 14%, 86%, 100% { transform: translate(0, 0); }
+ 26%, 72% { transform: translate(0.8px, 0.6px); }
+}
+@keyframes bs-squint-body {
+ 0%, 14%, 86%, 100% { transform: rotate(0); }
+ 26%, 72% { transform: rotate(-6deg) translateY(0.4px); }
+}
+
+/* ---- thinking: spark — eyes go up, then pop wide at an idea --------------- */
+
+.bs-scene[data-scene="spark"] .ae-eye { animation: bs-idea-eye 3s ease-in-out infinite; }
+.bs-scene[data-scene="spark"] .bs-body { animation: bs-hop 3s ease-in-out infinite; }
+@keyframes bs-idea-eye {
+ 0%, 100% { transform: translate(0, 0) scale(1); }
+ 30% { transform: translate(-0.6px, -1.8px) scale(0.9); }
+ 42% { transform: translate(0, -1px) scale(1.5); }
+ 52%, 78% { transform: translate(0, -0.8px) scale(1.3); }
+ 90% { transform: translate(0, 0) scale(1); }
+}
+@keyframes bs-hop {
+ 0%, 36%, 100% { transform: translateY(0) scale(1, 1); }
+ 40% { transform: translateY(0) scale(1.08, 0.9); }
+ 47% { transform: translateY(-4px) scale(0.95, 1.06); }
+ 55% { transform: translateY(0) scale(1.05, 0.94); }
+ 62% { transform: translateY(0) scale(1, 1); }
+}
+
+/* ---- thinking: wink — one eye shuts while the head tips ------------------- */
+
+.bs-scene[data-scene="wink"] .ae-r .ae-open { animation: bs-wink-open 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="wink"] .ae-r .ae-shut { animation: bs-wink-shut 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="wink"] .ae-l { animation: bs-wink-other 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="wink"] .bs-body { animation: bs-wink-body 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="wink"] .agent-symbol-lids { animation: none; }
+@keyframes bs-wink-open { 0%, 30%, 70%, 100% { opacity: 1; } 36%, 64% { opacity: 0; } }
+@keyframes bs-wink-shut { 0%, 30%, 70%, 100% { opacity: 0; } 36%, 64% { opacity: 1; } }
+@keyframes bs-wink-other {
+ 0%, 26%, 74%, 100% { transform: translate(0, 0) scale(1); }
+ 36%, 64% { transform: translate(0.4px, -0.8px) scale(1.15); }
+}
+@keyframes bs-wink-body {
+ 0%, 26%, 74%, 100% { transform: rotate(0); }
+ 36%, 64% { transform: rotate(8deg); }
+}
+
+/* ---- thinking: roll — eyes circle round, turning it over ------------------ */
+
+.bs-scene[data-scene="roll"] .ae-eye { animation: bs-roll 2.4s linear infinite; }
+.bs-scene[data-scene="roll"] .bs-body { animation: bs-roll-body 2.4s ease-in-out infinite; }
+@keyframes bs-roll {
+ 0%, 100% { transform: translate(2.2px, 0); }
+ 12.5% { transform: translate(1.55px, 1.55px); }
+ 25% { transform: translate(0, 2.2px); }
+ 37.5% { transform: translate(-1.55px, 1.55px); }
+ 50% { transform: translate(-2.2px, 0); }
+ 62.5% { transform: translate(-1.55px, -1.55px); }
+ 75% { transform: translate(0, -2.2px); }
+ 87.5% { transform: translate(1.55px, -1.55px); }
+}
+@keyframes bs-roll-body {
+ 0%, 100% { transform: rotate(2deg); }
+ 50% { transform: rotate(-2deg); }
+}
+
+/* ---- thinking: drowsy — heavy-lidded mulling, then a perk-up -------------- */
+
+.bs-scene[data-scene="drowsy"] .ae-open { animation: bs-drowsy 3.4s ease-in-out infinite; }
+.bs-scene[data-scene="drowsy"] .ae-eye { animation: bs-drowsy-eye 3.4s ease-in-out infinite; }
+.bs-scene[data-scene="drowsy"] .bs-body { animation: bs-drowsy-body 3.4s ease-in-out infinite; }
+.bs-scene[data-scene="drowsy"] .agent-symbol-lids { animation: none; }
+@keyframes bs-drowsy {
+ 0%, 8%, 100% { transform: scaleY(1); }
+ 30%, 66% { transform: scaleY(0.45); }
+ 74% { transform: scaleY(1.3); }
+ 84% { transform: scaleY(1); }
+}
+@keyframes bs-drowsy-eye {
+ 0%, 8%, 100% { transform: translate(0, 0); }
+ 30%, 66% { transform: translate(-0.6px, 1.2px); }
+ 74% { transform: translate(0, -1px); }
+}
+@keyframes bs-drowsy-body {
+ 0%, 8%, 100% { transform: rotate(0) translateY(0); }
+ 30%, 66% { transform: rotate(-4deg) translateY(0.8px); }
+ 74% { transform: rotate(1deg) translateY(-1.5px); }
+}
+
+/* ---- thinking: beam — a happy squint, it is going well -------------------- */
+
+.bs-scene[data-scene="beam"] .ae-open { animation: bs-beam-open 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="beam"] .ae-happy { animation: bs-beam-happy 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="beam"] .bs-body { animation: bs-beam-body 2.8s ease-in-out infinite; }
+.bs-scene[data-scene="beam"] .agent-symbol-lids { animation: none; }
+@keyframes bs-beam-open { 0%, 24%, 78%, 100% { opacity: 1; } 30%, 72% { opacity: 0; } }
+@keyframes bs-beam-happy { 0%, 24%, 78%, 100% { opacity: 0; } 30%, 72% { opacity: 1; } }
+@keyframes bs-beam-body {
+ 0%, 26%, 76%, 100% { transform: translateY(0) rotate(0); }
+ 36% { transform: translateY(-1.5px) rotate(-3deg); }
+ 46% { transform: translateY(0) rotate(0); }
+ 56% { transform: translateY(-1.5px) rotate(3deg); }
+ 66% { transform: translateY(0) rotate(0); }
+}
+
+/* ---- thinking: skyread — eyes up, reading something only it can see ------- */
+
+.bs-scene[data-scene="skyread"] .ae-eye { animation: bs-skyread 3s ease-in-out infinite; }
+.bs-scene[data-scene="skyread"] .bs-body { animation: bs-tilt 3s ease-in-out infinite; }
+@keyframes bs-skyread {
+ 0%, 100% { transform: translate(-2px, -1.9px); }
+ 42% { transform: translate(2px, -1.9px); }
+ 50% { transform: translate(-2px, -1.3px); }
+ 92% { transform: translate(2px, -1.3px); }
+}
+
+/* ---- thinking: doubt — one eye wide, one narrowed, double-checking -------- */
+
+.bs-scene[data-scene="doubt"] .ae-l { animation: bs-doubt-l 3s ease-in-out infinite; }
+.bs-scene[data-scene="doubt"] .ae-r { animation: bs-doubt-r 3s ease-in-out infinite; }
+.bs-scene[data-scene="doubt"] .bs-body { animation: bs-doubt-body 3s ease-in-out infinite; }
+@keyframes bs-doubt-l {
+ 0%, 18%, 84%, 100% { transform: translate(0, 0) scale(1); }
+ 30%, 72% { transform: translate(1.2px, -0.6px) scale(1.25); }
+}
+@keyframes bs-doubt-r {
+ 0%, 18%, 84%, 100% { transform: translate(0, 0) scale(1, 1); }
+ 30%, 72% { transform: translate(1.2px, 0.4px) scale(1.05, 0.5); }
+}
+@keyframes bs-doubt-body {
+ 0%, 18%, 84%, 100% { transform: rotate(0); }
+ 30%, 72% { transform: rotate(-7deg); }
+}
+
+/* ---- typing — eyes become dashes looking down at the keys ----------------- */
+
+.bs-scene[data-scene="typing"] .ae-open { opacity: 0; }
+.bs-scene[data-scene="typing"] .ae-shut { opacity: 1; }
+.bs-scene[data-scene="typing"] .agent-symbol-lids { animation: none; }
+.bs-scene[data-scene="typing"] .ae-eye { animation: bs-typing-eye 1.4s ease-in-out infinite; }
+.bs-scene[data-scene="typing"] .bs-body { animation: bs-typing-body 0.7s ease-in-out infinite alternate; }
+@keyframes bs-typing-eye {
+ 0%, 100% { transform: translate(1.2px, 1.6px) rotate(14deg); }
+ 50% { transform: translate(1.8px, 1.4px) rotate(4deg); }
+}
+@keyframes bs-typing-body {
+ from { transform: translateY(0) rotate(1deg); }
+ to { transform: translateY(-1px) rotate(-1deg); }
+}
+
+/* ---- reading the person's message — line by line ------------------------- */
+
+.bs-scene[data-scene="reading"] .ae-eye { animation: bs-reading 2.4s ease-in-out infinite; }
+.bs-scene[data-scene="reading"] .ae-open { transform: scale(0.85); }
+@keyframes bs-reading {
+ 0%, 100% { transform: translate(-1.8px, -1.2px); }
+ 28% { transform: translate(2px, -1.2px); }
+ 34% { transform: translate(-1.8px, 0); }
+ 62% { transform: translate(2px, 0); }
+ 68% { transform: translate(-1.8px, 1.2px); }
+ 94% { transform: translate(1.6px, 1.2px); }
+}
+
+/* ---- props: the tiny tool beside the face -------------------------------- */
+
+.bs-line, .bs-ink, .bs-jot, .bs-track, .bs-prompt {
+ fill: none;
+ stroke: currentColor;
+ stroke-width: 1.4;
+ stroke-linecap: round;
+ stroke-linejoin: round;
+}
+.bs-paper { fill: hsl(var(--chat-bubble-in)); stroke: currentColor; stroke-opacity: 0.5; stroke-width: 1; }
+
+/* note: a quill writing a line of ink; the eyes follow it down */
+.bs-scene[data-scene="note"] .ae-eye { animation: bs-note-eye 2s ease-in-out infinite; }
+.bs-scene[data-scene="note"] .bs-body { transform: rotate(5deg); animation: none; }
+@keyframes bs-note-eye {
+ 0%, 100% { transform: translate(1.2px, 1.8px); }
+ 70% { transform: translate(2.4px, 1.8px); }
+}
+.bs-ink { stroke: var(--bot); stroke-dasharray: 18; stroke-dashoffset: 18; animation: bs-ink 2s linear infinite; }
+@keyframes bs-ink {
+ 0% { stroke-dashoffset: 18; opacity: 1; }
+ 70%, 88% { stroke-dashoffset: 0; opacity: 1; }
+ 100% { stroke-dashoffset: 0; opacity: 0; }
+}
+.bs-quill { color: hsl(var(--foreground)); transform: translate(1px, 3px); animation: bs-quill 2s linear infinite; }
+.bs-quill .bs-line { stroke-width: 2.2; }
+@keyframes bs-quill {
+ 0% { transform: translate(1px, 3px) rotate(0); }
+ 18% { transform: translate(4px, 1.6px) rotate(-8deg); }
+ 36% { transform: translate(7px, 3.2px) rotate(6deg); }
+ 54% { transform: translate(10px, 1.6px) rotate(-8deg); }
+ 70% { transform: translate(12.5px, 3px) rotate(0); }
+ 88% { transform: translate(13px, -1px) rotate(10deg); }
+ 100% { transform: translate(1px, 3px) rotate(0); }
+}
+
+/* jot: lines appear on a notepad, eyes down */
+.bs-scene[data-scene="jot"] .ae-eye { transform: translate(1.6px, 1.8px); }
+.bs-scene[data-scene="jot"] .bs-body { animation: bs-nod 1.2s ease-in-out infinite; }
+@keyframes bs-nod {
+ 0%, 100% { transform: rotate(3deg); }
+ 50% { transform: rotate(6deg) translateY(0.5px); }
+}
+.bs-jot { stroke-opacity: 0.85; stroke-dasharray: 8; stroke-dashoffset: 8; }
+.bs-j1 { animation: bs-jot1 2.4s linear infinite; }
+.bs-j2 { animation: bs-jot2 2.4s linear infinite; }
+.bs-j3 { animation: bs-jot3 2.4s linear infinite; stroke: var(--bot); stroke-opacity: 1; }
+@keyframes bs-jot1 { 0% { stroke-dashoffset: 8; opacity: 1; } 22%, 90% { stroke-dashoffset: 0; opacity: 1; } 100% { opacity: 0; } }
+@keyframes bs-jot2 { 0%, 28% { stroke-dashoffset: 8; opacity: 1; } 50%, 90% { stroke-dashoffset: 0; opacity: 1; } 100% { opacity: 0; } }
+@keyframes bs-jot3 { 0%, 56% { stroke-dashoffset: 8; opacity: 1; } 74%, 90% { stroke-dashoffset: 0; opacity: 1; } 100% { opacity: 0; } }
+
+/* scan: a magnifier circles; the eyes dart after it */
+.bs-scene[data-scene="scan"] .ae-eye { animation: bs-dart 1.6s ease-in-out infinite; }
+@keyframes bs-dart {
+ 0%, 100% { transform: translate(1.2px, -0.8px); }
+ 25% { transform: translate(2.6px, 0.6px); }
+ 50% { transform: translate(1.2px, 1.6px); }
+ 75% { transform: translate(-0.2px, 0.4px); }
+}
+.bs-lens circle { fill: color-mix(in srgb, var(--bot) 22%, transparent); stroke: currentColor; stroke-width: 1.6; }
+.bs-lens path { fill: none; stroke: currentColor; stroke-width: 2.2; stroke-linecap: round; }
+.bs-lens { color: hsl(var(--foreground)); animation: bs-lens 1.6s ease-in-out infinite; }
+@keyframes bs-lens {
+ 0%, 100% { transform: translate(1.5px, -1.5px); }
+ 25% { transform: translate(2.5px, 1px); }
+ 50% { transform: translate(0, 2px); }
+ 75% { transform: translate(-1px, -0.5px); }
+}
+
+/* flip: a page turns; the eyes step down the page */
+.bs-scene[data-scene="flip"] .ae-eye { animation: bs-flip-eye 1.8s ease-in-out infinite; }
+@keyframes bs-flip-eye {
+ 0%, 100% { transform: translate(1.6px, -1px); }
+ 33% { transform: translate(1.6px, 0.4px); }
+ 66% { transform: translate(1.6px, 1.6px); }
+}
+.bs-page rect { fill: hsl(var(--chat-bubble-in)); stroke: currentColor; stroke-opacity: 0.7; stroke-width: 1; }
+.bs-page path { fill: none; stroke: currentColor; stroke-opacity: 0.7; stroke-width: 1.1; stroke-linecap: round; }
+.bs-page { transform-origin: 5px 10px; animation: bs-page 1.8s cubic-bezier(0.45, 0, 0.3, 1) infinite; }
+@keyframes bs-page {
+ 0%, 30% { transform: scaleX(1); opacity: 1; }
+ 58% { transform: scaleX(0.05); opacity: 1; }
+ 80% { transform: scaleX(-0.9); opacity: 0; }
+ 81% { transform: scaleX(1); opacity: 0; }
+ 100% { transform: scaleX(1); opacity: 1; }
+}
+
+/* recall: eyes up and away, thought puffs rising */
+.bs-scene[data-scene="recall"] .ae-eye { animation: bs-recall 3s ease-in-out infinite; }
+.bs-scene[data-scene="recall"] .bs-body { animation: bs-tilt 3s ease-in-out infinite; }
+@keyframes bs-recall {
+ 0%, 100% { transform: translate(1.6px, -2px); }
+ 45% { transform: translate(-1.8px, -2px); }
+ 60% { transform: translate(-1.8px, -2px) scale(1.15); }
+}
+.bs-puff { fill: hsl(var(--chat-bubble-in)); stroke: currentColor; stroke-opacity: 0.6; stroke-width: 0.8; animation: bs-puff 1.8s ease-in-out infinite; }
+.bs-p2 { animation-delay: 0.3s; fill: color-mix(in srgb, var(--bot) 50%, hsl(var(--chat-bubble-in))); }
+@keyframes bs-puff { 0%, 100% { opacity: 0.35; } 50% { opacity: 1; } }
+
+/* browser: a pointer clicks a button; the eyes follow it */
+.bs-scene[data-scene="browser"] .ae-eye { animation: bs-browser-eye 2.4s ease-in-out infinite; }
+@keyframes bs-browser-eye {
+ 0%, 100% { transform: translate(1px, -0.4px); }
+ 50%, 80% { transform: translate(2.4px, 1.2px); }
+}
+.bs-scene[data-scene="browser"] .bs-line { stroke-width: 1; stroke-opacity: 0.5; }
+.bs-button { fill: transparent; stroke: currentColor; stroke-opacity: 0.6; stroke-width: 1; animation: bs-button 2.4s ease-out infinite; }
+@keyframes bs-button { 0%, 54% { fill: transparent; } 58%, 86% { fill: var(--bot); stroke: var(--bot); } 100% { fill: transparent; } }
+.bs-pointer-shape { fill: hsl(var(--foreground)); stroke: hsl(var(--background)); stroke-width: 2.4; stroke-linejoin: round; }
+.bs-pointer { transform: translate(4px, 9px); animation: bs-pointer 2.4s cubic-bezier(0.5, 0, 0.3, 1) infinite; }
+@keyframes bs-pointer {
+ 0%, 8% { transform: translate(4px, 9px) scale(1); }
+ 46% { transform: translate(11px, 11px) scale(1); }
+ 54% { transform: translate(11px, 11px) scale(0.75); }
+ 60%, 88% { transform: translate(11px, 11px) scale(1); }
+ 100% { transform: translate(4px, 9px) scale(1); }
+}
+
+/* setup: a gear turns; the eyes narrow on it */
+.bs-scene[data-scene="setup"] .ae-open { transform: scaleY(0.55); }
+.bs-scene[data-scene="setup"] .ae-eye { transform: translate(1.8px, 0.6px); }
+.bs-scene[data-scene="setup"] .bs-body { animation: bs-effort 0.7s ease-in-out infinite; }
+@keyframes bs-effort {
+ 0%, 100% { transform: rotate(-2deg); }
+ 50% { transform: rotate(3deg) translateY(-0.6px); }
+}
+.bs-gear path { fill: color-mix(in srgb, var(--bot) 70%, transparent); stroke: currentColor; stroke-width: 0.8; stroke-linejoin: round; }
+.bs-gear circle { fill: hsl(var(--background)); stroke: currentColor; stroke-width: 0.8; }
+.bs-gear { transform-origin: 10px 10px; animation: bs-spin 3s linear infinite; }
+@keyframes bs-spin { to { transform: rotate(360deg); } }
+
+/* tune: two knobs slide; the eyes follow the lower one */
+.bs-scene[data-scene="tune"] .ae-eye { animation: bs-tune-eye 2.6s ease-in-out infinite; }
+@keyframes bs-tune-eye {
+ 0%, 100% { transform: translate(1px, 0.6px); }
+ 50% { transform: translate(2.6px, 0.6px); }
+}
+.bs-track { stroke-opacity: 0.4; stroke-width: 1.6; }
+.bs-knob { fill: hsl(var(--background)); stroke: currentColor; stroke-width: 1.2; }
+.bs-knob2 { fill: var(--bot); stroke: var(--bot); }
+.bs-knob1 { transform: translateX(14px); animation: bs-knob1 2.6s ease-in-out infinite; }
+.bs-knob2 { transform: translateX(5px); animation: bs-knob2 2.6s ease-in-out infinite; }
+@keyframes bs-knob1 { 0%, 100% { transform: translateX(14px); } 50% { transform: translateX(6px); } }
+@keyframes bs-knob2 { 0%, 100% { transform: translateX(5px); } 50% { transform: translateX(15px); } }
+
+/* command: eyes as dashes on the prompt, a line typing in */
+.bs-scene[data-scene="command"] .ae-open { opacity: 0; }
+.bs-scene[data-scene="command"] .ae-shut { opacity: 1; }
+.bs-scene[data-scene="command"] .ae-eye { transform: translate(1.8px, 1px) rotate(6deg); }
+.bs-scene[data-scene="command"] .agent-symbol-lids { animation: none; }
+.bs-scene[data-scene="command"] .bs-body { animation: bs-typing-body 0.36s ease-in-out infinite alternate; }
+.bs-prompt { stroke: var(--bot); stroke-width: 1.6; }
+.bs-typed { fill: currentColor; fill-opacity: 0.7; transform-origin: 7.5px 0; transform: scaleX(0); animation: bs-typed 2s steps(8) infinite; }
+@keyframes bs-typed { 0% { transform: scaleX(0); opacity: 1; } 70%, 90% { transform: scaleX(1); opacity: 1; } 100% { transform: scaleX(1); opacity: 0; } }
+.bs-caret { fill: hsl(var(--foreground)); transform: translate(8px, 0); animation: bs-caret 2s steps(8) infinite, bs-caret-blink 0.7s steps(1) infinite; }
+@keyframes bs-caret { 0% { transform: translate(8px, 0); } 70%, 100% { transform: translate(17px, 0); } }
+@keyframes bs-caret-blink { 0%, 49% { opacity: 1; } 50%, 100% { opacity: 0.2; } }
+
+/* juggle: two balls arc over the head; the eyes follow them */
+.bs-scene[data-scene="juggle"] .ae-eye { animation: bs-juggle-eye 1.2s ease-in-out infinite; }
+.bs-scene[data-scene="juggle"] .bs-body { animation: bs-juggle-body 1.2s ease-in-out infinite; }
+@keyframes bs-juggle-eye {
+ 0%, 100% { transform: translate(-1.6px, -1.6px); }
+ 50% { transform: translate(1.8px, -1.6px); }
+}
+@keyframes bs-juggle-body {
+ 0%, 100% { transform: rotate(-3deg); }
+ 50% { transform: rotate(3deg); }
+}
+.bs-ball { fill: var(--bot); stroke: hsl(var(--background)); stroke-width: 0.8; transform: translate(1px, 12px); animation: bs-ball 1.2s infinite; }
+.bs-b2 { animation-delay: -0.6s; fill: hsl(var(--foreground)); }
+@keyframes bs-ball {
+ 0% { transform: translate(1px, 12px); animation-timing-function: cubic-bezier(0.2, 0.6, 0.4, 1); }
+ 50% { transform: translate(14px, -7px); animation-timing-function: cubic-bezier(0.6, 0, 0.8, 0.4); }
+ 100% { transform: translate(27px, 12px); }
+}
+
+/* build: a hammer taps; the eyes narrow on the nail */
+.bs-scene[data-scene="build"] .ae-open { transform: scaleY(0.6); }
+.bs-scene[data-scene="build"] .ae-eye { transform: translate(2px, 1.4px); }
+.bs-scene[data-scene="build"] .bs-body { animation: bs-strike 0.9s ease-in-out infinite; }
+@keyframes bs-strike {
+ 0%, 40%, 100% { transform: rotate(-2deg); }
+ 52% { transform: rotate(4deg) translateY(0.6px); }
+}
+.bs-hammer path { fill: none; stroke: currentColor; stroke-width: 1.8; stroke-linecap: round; }
+.bs-hammer rect { fill: currentColor; }
+.bs-hammer { color: hsl(var(--foreground)); transform-origin: 3px 8.5px; transform: rotate(-40deg); animation: bs-hammer 0.9s infinite; }
+@keyframes bs-hammer {
+ 0% { transform: rotate(-40deg); animation-timing-function: ease-in-out; }
+ 38% { transform: rotate(-48deg); animation-timing-function: cubic-bezier(0.6, 0, 1, 0.6); }
+ 52% { transform: rotate(18deg); }
+ 60% { transform: rotate(12deg); animation-timing-function: ease-out; }
+ 100% { transform: rotate(-40deg); }
+}
+.bs-spark path { fill: none; stroke: var(--bot); stroke-width: 1.3; stroke-linecap: round; }
+.bs-spark { opacity: 0; animation: bs-spark 0.9s ease-out infinite; }
+@keyframes bs-spark { 0%, 50% { opacity: 0; } 56% { opacity: 1; } 76%, 100% { opacity: 0; } }
+
+/* ---- done — a smile, a hop and a tick ------------------------------------- */
+
+.bs-scene[data-scene="done"] .ae-open { opacity: 0; }
+.bs-scene[data-scene="done"] .ae-happy { opacity: 1; }
+.bs-scene[data-scene="done"] .agent-symbol-lids { animation: none; }
+.bs-scene[data-scene="done"] .bs-body { animation: bs-cheer 0.9s cubic-bezier(0.3, 0.7, 0.3, 1) both; }
+@keyframes bs-cheer {
+ 0% { transform: translateY(0) scale(1, 1); }
+ 18% { transform: translateY(0) scale(1.12, 0.86); }
+ 42% { transform: translateY(-6px) scale(0.94, 1.08); }
+ 66% { transform: translateY(0) scale(1.06, 0.92); }
+ 82%, 100% { transform: translateY(0) scale(1, 1); }
+}
+.bs-check circle { fill: var(--bot); stroke: hsl(var(--background)); stroke-width: 1.2; }
+.bs-check path { fill: none; stroke: var(--bot-ink); stroke-width: 1.5; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 8; stroke-dashoffset: 8; animation: bs-check-draw 0.4s ease-out 0.35s both; }
+.bs-check { transform-origin: 23.5px 23.5px; animation: bs-check-pop 0.5s cubic-bezier(0.3, 1.5, 0.5, 1) 0.15s both; }
+@keyframes bs-check-pop { from { transform: scale(0); } to { transform: scale(1); } }
+@keyframes bs-check-draw { to { stroke-dashoffset: 0; } }
+
+/* ---- the word ------------------------------------------------------------ */
+
+.bs-label {
+ display: inline-grid;
+ font-size: 13.5px;
+ line-height: 20px;
+ color: hsl(var(--muted-foreground));
+ white-space: nowrap;
+ animation: bs-label-in 0.35s ease-out both, bs-label-pulse 2s ease-in-out 0.35s infinite;
+}
+/* The old word drifts up and out while the new one rises into its place. */
+.bs-word { grid-area: 1 / 1; }
+.bs-word.is-in { animation: bs-word-in 0.36s 0.12s cubic-bezier(0.2, 0.8, 0.2, 1) both; }
+.bs-word.is-out { animation: bs-word-out 0.22s ease-in both; }
+@keyframes bs-word-in {
+ from { opacity: 0; transform: translateY(6px); filter: blur(2px); }
+ to { opacity: 1; transform: none; filter: none; }
+}
+@keyframes bs-word-out {
+ to { opacity: 0; transform: translateY(-6px); filter: blur(2px); }
+}
+@keyframes bs-label-in {
+ from { opacity: 0; transform: translateX(-3px); }
+ to { opacity: 1; transform: none; }
+}
+@keyframes bs-label-pulse {
+ 0%, 100% { color: hsl(var(--muted-foreground)); }
+ 50% { color: hsl(var(--foreground)); }
+}
+.bot-stage[data-scene="done"] .bs-label { animation: bs-label-in 0.35s ease-out both; color: hsl(var(--foreground)); }
+
+/* ---- reduced motion: a still face, props at rest ------------------------- */
+
+@media (prefers-reduced-motion: reduce) {
+ .bot-stage *, .bot-stage *::before, .bot-stage *::after { animation: none !important; }
+ .bs-ink, .bs-jot, .bs-check path { stroke-dashoffset: 0; }
+ .bs-typed { transform: scaleX(1); }
+ .bs-spark { opacity: 0; }
+ .bot-stage .is-out { visibility: hidden; }
+ .bs-props { transition: none; }
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/botStage/scenes.ts b/jarvis/ui/web/frontend/src/components/agentchat/botStage/scenes.ts
new file mode 100644
index 0000000000..9c74fddd97
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/botStage/scenes.ts
@@ -0,0 +1,73 @@
+/**
+ * Which expression the bot plays while it works.
+ *
+ * A turn's presence ("thinking", "searching", …) maps to a family of scenes.
+ * The face carries every scene — its eyes look around, squint, widen, wink,
+ * smile — and a tool scene adds at most one tiny prop beside it. The scene
+ * inside a family is picked from the turn id and the phase, so two turns
+ * rarely look alike, and a long think rotates through the family instead of
+ * looping one expression. Pure: the stage only draws what this returns.
+ */
+
+import type { PetState } from "@/lib/petStates";
+import type { Presence } from "../messengerPresence";
+
+export type Scene =
+ // thinking: the face alone
+ | "ponder" | "glance" | "squint" | "spark" | "wink" | "roll" | "drowsy" | "beam" | "skyread" | "doubt"
+ // work: the face plus one tiny prop
+ | "note" | "jot" | "scan" | "flip" | "recall" | "browser" | "setup" | "tune" | "command" | "juggle" | "build"
+ // conversation
+ | "typing" | "reading" | "done";
+
+export const SCENES: Record<Presence, readonly Scene[]> = {
+ thinking: ["ponder", "glance", "squint", "spark", "wink", "roll", "drowsy", "beam", "skyread", "doubt"],
+ writing: ["note", "jot"],
+ searching: ["scan", "flip"],
+ recall: ["recall"],
+ browsing: ["browser"],
+ setting_up: ["setup", "tune"],
+ command: ["command"],
+ working: ["juggle", "build"],
+ typing: ["typing"],
+ reading: ["reading"],
+ done: ["done"],
+};
+
+/**
+ * The row a pet plays for each scene when the agent wears a pet instead of a
+ * shape: it has no eyes for the stage to move, so its own animation carries
+ * the moment. A pet without a row borrows the nearest one (STATE_FALLBACKS).
+ */
+export const PET_STATE: Record<Scene, PetState> = {
+ ponder: "thinking", glance: "thinking", squint: "thinking", spark: "thinking", wink: "thinking",
+ roll: "thinking", drowsy: "thinking", beam: "thinking", skyread: "thinking", doubt: "thinking",
+ note: "working", jot: "working", scan: "searching", flip: "searching", recall: "thinking",
+ browser: "searching", setup: "working", tune: "working", command: "working", juggle: "working", build: "working",
+ typing: "talking", reading: "listening", done: "success",
+};
+
+/** How long one thinking expression plays before the next one takes over. */
+export const THINK_ROTATE_MS = 4200;
+/** No scene is cut short before this: a call that lasts 80 ms must not flash a scene. */
+export const MIN_SCENE_MS = 1400;
+/** How long the finished turn's smile stays before the stage clears. */
+export const DONE_MS = 1700;
+
+function hash(text: string): number {
+ let h = 2166136261;
+ for (let i = 0; i < text.length; i++) h = Math.imul(h ^ text.charCodeAt(i), 16777619) >>> 0;
+ return h;
+}
+
+/**
+ * The scene for a presence: deterministic per turn and phase. `step` advances
+ * through the family (a long think rotates); consecutive steps never repeat
+ * a scene when the family has more than one.
+ */
+export function sceneFor(presence: Presence, seed: string, step = 0): Scene {
+ const family = SCENES[presence];
+ if (family.length === 1) return family[0];
+ const start = hash(`${seed}:${presence}`) % family.length;
+ return family[(start + step) % family.length];
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/chatAttachments.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/chatAttachments.test.tsx
index cf0f5db3bd..3e1bb01d61 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/chatAttachments.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/chatAttachments.test.tsx
@@ -293,6 +293,23 @@ describe("the timeline's receipt for a message that carried files", () => {
});
});
+describe("the thumbnail strip", () => {
+ it("shows the picture alone, without a read receipt", () => {
+ render(
+ <ChatAttachmentStrip
+ look="thumbnail"
+ attachments={[{ name: "shot.png", kind: "image", described_by: "none", detail: "", note: "" } as unknown as ChatAttachment]}
+ analyzing={0}
+ onRemove={() => {}}
+ previews={{ "shot.png": "blob:shot" }}
+ />,
+ );
+ const card = screen.getByTestId("chat-attachment-shot.png");
+ expect(card.querySelector("img")?.getAttribute("src")).toBe("blob:shot");
+ expect(card.textContent).not.toContain("not described");
+ });
+});
+
describe("the app-wide file drop guard", () => {
function Guarded() {
useFileDropGuard();
@@ -340,20 +357,3 @@ describe("the app-wide file drop guard", () => {
expect(event.defaultPrevented).toBe(false);
});
});
-
-describe("the thumbnail strip", () => {
- it("shows the picture alone, without a read receipt", () => {
- render(
- <ChatAttachmentStrip
- look="thumbnail"
- attachments={[{ name: "shot.png", kind: "image", described_by: "none", detail: "", note: "" } as unknown as ChatAttachment]}
- analyzing={0}
- onRemove={() => {}}
- previews={{ "shot.png": "blob:shot" }}
- />,
- );
- const card = screen.getByTestId("chat-attachment-shot.png");
- expect(card.querySelector("img")?.getAttribute("src")).toBe("blob:shot");
- expect(card.textContent).not.toContain("not described");
- });
-});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/composerDrafts.ts b/jarvis/ui/web/frontend/src/components/agentchat/composerDrafts.ts
index 198ba5f430..ece09fbdb1 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/composerDrafts.ts
+++ b/jarvis/ui/web/frontend/src/components/agentchat/composerDrafts.ts
@@ -13,6 +13,42 @@ const createDraftStore = () => createStore<{ drafts: Map<string | null, MessageD
// Keep unsent text in memory, owned by its chat store, across view unmounts.
// Separate stores also isolate the front page from IDE panes and other agents.
const stores = new WeakMap<AgentChatStoreHook, ReturnType<typeof createDraftStore>>();
+export interface ComposerDraftTarget { sessionId: string | null; superseded?: boolean }
+const blankTargets = new WeakMap<AgentChatStoreHook, ComposerDraftTarget>();
+
+/** Capture before sending so a late failure restores the chat that was created. */
+export function captureComposerDraftTarget(chat: AgentChatStoreHook, sessionId: string | null): ComposerDraftTarget {
+ if (sessionId !== null) return { sessionId };
+ let target = blankTargets.get(chat);
+ if (!target || target.sessionId !== null) {
+ target = { sessionId: null };
+ blankTargets.set(chat, target);
+ }
+ return target;
+}
+
+export function resetBlankComposerDraftTarget(chat: AgentChatStoreHook): void {
+ const previous = blankTargets.get(chat);
+ if (previous?.sessionId === null) previous.superseded = true;
+ writeComposerDraft(chat, null, EMPTY_DRAFT);
+ blankTargets.set(chat, { sessionId: null });
+}
+
+/** Transfer unsent text/chips before opening the assigned session. */
+export function bindComposerDraftTarget(chat: AgentChatStoreHook, target: ComposerDraftTarget, sessionId: string): boolean {
+ const currentBlank = target.sessionId === null && blankTargets.get(chat) === target;
+ target.sessionId = sessionId;
+ target.superseded = false;
+ if (!currentBlank) return false;
+ const source = readComposerDraft(chat, null);
+ const destination = readComposerDraft(chat, sessionId);
+ writeComposerDraft(chat, sessionId, {
+ text: [source.text, destination.text].filter(Boolean).join("\n"),
+ choices: [...source.choices, ...destination.choices.filter((row) => !source.choices.some((sent) => sent.id === row.id))],
+ });
+ writeComposerDraft(chat, null, EMPTY_DRAFT);
+ return true;
+}
export function composerDraftsFor(chat: AgentChatStoreHook) {
let store = stores.get(chat);
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/messengerPresence.ts b/jarvis/ui/web/frontend/src/components/agentchat/messengerPresence.ts
new file mode 100644
index 0000000000..8b25fe2ed2
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentchat/messengerPresence.ts
@@ -0,0 +1,127 @@
+/**
+ * The pure model behind an agent chat's messenger look (MessengerTurn).
+ *
+ * An agent chat shows no reasoning and no work log. While a turn runs, ONE
+ * presence line says what the agent is doing in a word or two ("Thinking",
+ * "Writing something down"), and a finished call that changed something the
+ * person cares about — a routine, a rule, a saved skill — leaves a quiet
+ * centred notice between the messages. Everything else the agent did stays
+ * out of the chat; its own messages say what it means.
+ */
+
+import { waitsOnCard, type ToolBlock, type TurnBlock, type TurnItem } from "./reduce";
+
+export type Presence =
+ | "thinking"
+ | "typing"
+ | "writing"
+ | "searching"
+ | "recall"
+ | "browsing"
+ | "setting_up"
+ | "command"
+ | "working"
+ /** The person's message is in, the turn has not started yet. */
+ | "reading"
+ /** The turn just finished: a short flourish before the stage clears. */
+ | "done";
+
+/** The bare operation of a call, without its transport prefix (`mcp__jarvis__…`, `jarvis/…`). */
+export function bareToolName(name: string): string {
+ return (name.split(/__|\//).pop() ?? name).replace(/^(?:functions|tools)\./, "").toLowerCase();
+}
+
+function inputOf(block: ToolBlock): Record<string, unknown> {
+ return block.input && typeof block.input === "object" ? (block.input as Record<string, unknown>) : {};
+}
+
+function payloadOf(block: ToolBlock): Record<string, unknown> {
+ const payload = inputOf(block).payload;
+ return payload && typeof payload === "object" ? (payload as Record<string, unknown>) : {};
+}
+
+const PROPOSE = "society_propose_change";
+
+/** What kind of work a call is, as the presence line names it. */
+export function callPresence(block: ToolBlock): Presence {
+ const name = bareToolName(block.name);
+ if (name === PROPOSE) {
+ const kind = String(inputOf(block).kind ?? "");
+ return kind === "rule" || kind === "skill" ? "writing" : "setting_up";
+ }
+ if (/(wiki_?note|remember|memory_?(write|update|save)|^(write|edit|multiedit|applypatch|apply_patch|str_replace|writefile|editfile|createfile)$)/.test(name)) {
+ return "writing";
+ }
+ if (/(^browser|browser_|navigate|click|screenshot|computer)/.test(name)) return "browsing";
+ if (/(recall|memory_?(read|search|get)|^memory$)/.test(name)) return "recall";
+ if (/^(bash|powershell|shell|run_?shell(_command)?|exec(_command)?|run_?command|terminal)$/.test(name)) return "command";
+ if (/(search|^grep$|^rg$|^glob$|^read|^ls$|^list|fetch|lookup|^find)/.test(name)) return "searching";
+ if (/(credential|routine|install|connect|plugin|connector|configure|setup)/.test(name)) return "setting_up";
+ return "working";
+}
+
+function waitsForPerson(block: ToolBlock): boolean {
+ return waitsOnCard(block) || Boolean(block.approval && block.approval.decision === null);
+}
+
+/**
+ * What the running turn is doing right now, or null when nothing should say
+ * so: the turn is over, or a card is waiting for the person (the card says it).
+ */
+export function presenceOf(turn: TurnItem): Presence | null {
+ if (turn.status !== "running") return null;
+ const blocks: TurnBlock[] = turn.blocks;
+ if (blocks.some((block) => block.kind === "tool" && waitsForPerson(block))) return null;
+ for (let i = blocks.length - 1; i >= 0; i--) {
+ const block = blocks[i];
+ if (block.kind === "tool") return block.output === null ? callPresence(block) : "thinking";
+ if (block.kind === "text") return block.text.trim() ? "typing" : "thinking";
+ // A thought is never shown; it only means the agent is thinking.
+ return "thinking";
+ }
+ return "thinking";
+}
+
+export type NoticeVerb = "created" | "updated" | "paused" | "resumed" | "deleted";
+
+export interface ActionNotice {
+ /** i18n key under `messenger_turn.` naming what happened ("Created: Routine"). */
+ key: string;
+ /** The thing it happened to ("Daily contributor check"), when the call names it. */
+ subject: string;
+ icon: "routine" | "rule" | "skill" | "settings" | "note";
+}
+
+const ROUTINE_VERB: Record<string, NoticeVerb> = {
+ create: "created", update: "updated", pause: "paused", resume: "resumed", delete: "deleted",
+};
+
+/**
+ * The quiet notice a FINISHED call leaves in the chat, or null when the call
+ * changed nothing worth a line of its own. A call that failed, or a proposal
+ * still waiting on its card, leaves none: the agent's message and the card
+ * speak for those.
+ */
+export function actionNoticeOf(block: ToolBlock): ActionNotice | null {
+ if (block.output === null || block.isError) return null;
+ if (/"status"\s*:\s*"pending"/.test(block.output)) return null;
+ const name = bareToolName(block.name);
+ if (name === PROPOSE) {
+ const kind = String(inputOf(block).kind ?? "");
+ const payload = payloadOf(block);
+ if (kind === "routine") {
+ const verb = ROUTINE_VERB[String(payload.operation ?? "create")] ?? "updated";
+ return { key: `routine_${verb}`, subject: String(payload.title ?? ""), icon: "routine" };
+ }
+ if (kind === "rule") return { key: "rule_saved", subject: "", icon: "rule" };
+ if (kind === "skill") return { key: "skill_saved", subject: String(payload.name ?? ""), icon: "skill" };
+ if (kind === "focus" || kind === "approval_rule") return { key: "settings_updated", subject: "", icon: "settings" };
+ // Identity changes update the agent profile without a chat notice.
+ return null;
+ }
+ if (/(wiki_?note|^remember$)/.test(name)) {
+ const input = inputOf(block);
+ return { key: "note_saved", subject: String(input.title ?? ""), icon: "note" };
+ }
+ return null;
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/reduce.test.ts b/jarvis/ui/web/frontend/src/components/agentchat/reduce.test.ts
index 4ce88a0478..09b2df3166 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/reduce.test.ts
+++ b/jarvis/ui/web/frontend/src/components/agentchat/reduce.test.ts
@@ -6,6 +6,8 @@ import {
reduceEvent,
reduceEvents,
runningTurn,
+ settleApproval,
+ type ToolBlock,
type TurnItem,
type UserItem,
} from "./reduce";
@@ -76,6 +78,45 @@ describe("agent-chat reduce", () => {
expect(turn.blocks[0]).toMatchObject({ kind: "tool", approval: { approvalId: "a1", decision: "deny" } });
});
+ it("lets the same call asking again replace its older card", () => {
+ // 2026-10-06: an app restart re-asked the call under a new id; the old
+ // card stayed first in line and its buttons answered 404 for ever.
+ const tl = reduceEvents(EMPTY_TIMELINE, [
+ ev("turn_started", { turn_id: "t9" }),
+ ev("tool_call", { turn_id: "t9", call_id: "c9", name: "Bash", input: { command: "ls" } }),
+ ev("approval_required", { turn_id: "t9", approval_id: "old", call_id: "c9", name: "Bash", input: { command: "ls" }, summary: "ls" }),
+ ev("approval_required", { turn_id: "t9", approval_id: "other", call_id: "c10", name: "Bash", input: {}, summary: "pwd" }),
+ ev("approval_required", { turn_id: "t9", approval_id: "new", call_id: "c9", name: "Bash", input: { command: "ls" }, summary: "ls" }),
+ ]);
+ expect(tl.pendingApprovals.map((a) => a.approvalId)).toEqual(["other", "new"]);
+ const row = (tl.items[0] as TurnItem).blocks.find((b): b is ToolBlock => b.kind === "tool" && b.callId === "c9");
+ expect(row?.approval).toMatchObject({ approvalId: "new", decision: null });
+ });
+
+ it("stops offering approval buttons once the turn ended", () => {
+ const tl = reduceEvents(EMPTY_TIMELINE, [
+ ev("turn_started", { turn_id: "t10" }),
+ ev("approval_required", { turn_id: "t10", approval_id: "a10", call_id: "c10", name: "Bash", input: {}, summary: "ls" }),
+ ev("turn_finished", { turn_id: "t10", status: "error", error: "Jarvis restarted while this turn was running." }),
+ ]);
+ expect(tl.pendingApprovals).toHaveLength(0);
+ expect((tl.items[0] as TurnItem).blocks[0]).toMatchObject({ kind: "tool", approval: { approvalId: "a10", decision: "expired" } });
+ });
+
+ it("settles a card the backend answered before its event arrives", () => {
+ let tl = reduceEvents(EMPTY_TIMELINE, [
+ ev("turn_started", { turn_id: "t11" }),
+ ev("approval_required", { turn_id: "t11", approval_id: "a11", call_id: "c11", name: "Bash", input: {}, summary: "ls" }),
+ ]);
+ tl = settleApproval(tl, "a11", "allow");
+ expect(tl.pendingApprovals).toHaveLength(0);
+ expect((tl.items[0] as TurnItem).blocks[0]).toMatchObject({ approval: { approvalId: "a11", decision: "allow" } });
+ // The event that follows changes nothing; an unknown id is a no-op.
+ const after = reduceEvent(tl, ev("approval_resolved", { turn_id: "t11", approval_id: "a11", decision: "allow" }));
+ expect(after.items).toBe(tl.items);
+ expect(settleApproval(after, "nope", "expired")).toBe(after);
+ });
+
it("returns the same object when an event changes nothing", () => {
const tl = reduceEvents(EMPTY_TIMELINE, [ev("turn_started", { turn_id: "t3" })]);
const same = reduceEvent(tl, ev("text_delta", { turn_id: "t3", message_id: "m", text: "" }, false));
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/reduce.ts b/jarvis/ui/web/frontend/src/components/agentchat/reduce.ts
index 88daf716c0..2e6f759a18 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/reduce.ts
+++ b/jarvis/ui/web/frontend/src/components/agentchat/reduce.ts
@@ -106,6 +106,41 @@ export function isQuestionTool(name: string): boolean {
return name === QUESTION_TOOL || name.endsWith(`__${QUESTION_TOOL}`) || name === CLAUDE_QUESTION_TOOL;
}
+/**
+ * An agent's secure credential field (jarvis/agent_chat/credential_requests.py).
+ * The person pastes a secret that goes straight to the agent's vault; no event
+ * ever carries the value. `status` is `null` until the person saves or declines.
+ * Older event logs can also contain `timeout` or `cancelled` resolutions.
+ */
+export interface CredentialState {
+ requestId: string;
+ asker: string;
+ env: string;
+ label: string;
+ description: string;
+ placeholder: string;
+ replace: boolean;
+ expiresMs: number | null;
+ status: string | null;
+}
+
+/** The tool an agent asks for a credential with — bare or behind an MCP prefix. */
+export const CREDENTIAL_TOOL = "society_request_credential";
+
+export function isCredentialTool(name: string): boolean {
+ return name === CREDENTIAL_TOOL || name.endsWith(`__${CREDENTIAL_TOOL}`);
+}
+
+/** The call shows a card the person answers: a question or a credential field. */
+export function hasPersonCard(block: ToolBlock): boolean {
+ return Boolean(block.question || block.credential);
+}
+
+/** The call's card still waits for the person. */
+export function waitsOnCard(block: ToolBlock): boolean {
+ return Boolean((block.question && !block.question.closed) || (block.credential && block.credential.status === null));
+}
+
/**
* A coding agent's plan card (jarvis/agent_chat/turn_prompts.py): the turn
* finished in plan mode. `decision` is `build`, `keep`, `superseded` (the
@@ -127,6 +162,8 @@ export interface ToolBlock {
approval: ApprovalState | null;
/** The question card this call shows, when the call is an agent's question. */
question?: QuestionState;
+ /** The secure credential field this call shows, when the agent asked for a secret. */
+ credential?: CredentialState;
/** When the call was made; a result without its own duration is timed from here. */
startedMs: number;
/** The sub-agent this call spawned, when the runner reported one (`subagent_*` events). */
@@ -446,6 +483,57 @@ function settleWaitingCards(items: TimelineItem[]): TimelineItem[] {
return changed ? next : items;
}
+function updateCredential(
+ tl: Timeline,
+ turnId: string,
+ requestId: string,
+ fn: (c: CredentialState) => CredentialState,
+): Timeline {
+ return updateTurn(tl, turnId, (turn) => {
+ const i = turn.blocks.findIndex((b) => b.kind === "tool" && b.credential?.requestId === requestId);
+ if (i < 0) return turn;
+ const block = turn.blocks[i] as ToolBlock;
+ return { ...turn, blocks: replaceAt(turn.blocks, i, { ...block, credential: fn(block.credential!) }) };
+ });
+}
+
+/** Attach a credential field to its tool row: the open request call, else a row of its own. */
+function withCredential(turn: TurnItem, credential: CredentialState, tsMs: number): TurnItem {
+ let index = -1;
+ for (let i = turn.blocks.length - 1; i >= 0; i -= 1) {
+ const b = turn.blocks[i];
+ if (b.kind !== "tool") continue;
+ if (b.credential?.requestId === credential.requestId) return turn;
+ if (isCredentialTool(b.name) && !b.credential) {
+ index = i;
+ break;
+ }
+ }
+ if (index >= 0) {
+ const block = turn.blocks[index] as ToolBlock;
+ return { ...turn, blocks: replaceAt(turn.blocks, index, { ...block, credential }) };
+ }
+ const closed = closeLiveReasoning(turn, tsMs);
+ return {
+ ...closed,
+ blocks: [
+ ...closed.blocks,
+ {
+ kind: "tool",
+ callId: `cred-${credential.requestId}`,
+ name: CREDENTIAL_TOOL,
+ input: null,
+ output: null,
+ isError: false,
+ durationMs: null,
+ approval: null,
+ credential,
+ startedMs: tsMs,
+ },
+ ],
+ };
+}
+
function withQuestion(turn: TurnItem, question: QuestionState, tsMs: number): TurnItem {
let index = -1;
for (let i = turn.blocks.length - 1; i >= 0; i -= 1) {
@@ -541,6 +629,46 @@ function mapToolDeep(
return null;
}
+/** A row whose approval was never answered, closed as `expired` — its sub-agent's rows too. */
+function expireOpenApproval(block: TurnBlock): TurnBlock {
+ if (block.kind !== "tool") return block;
+ const open = block.approval !== null && block.approval.decision === null;
+ const inner = block.subagent?.blocks.map(expireOpenApproval);
+ const innerChanged = inner !== undefined && inner.some((b, i) => b !== block.subagent?.blocks[i]);
+ if (!open && !innerChanged) return block;
+ return {
+ ...block,
+ approval: open && block.approval ? { ...block.approval, decision: "expired" } : block.approval,
+ ...(innerChanged && block.subagent && inner ? { subagent: { ...block.subagent, blocks: inner } } : {}),
+ };
+}
+
+/**
+ * Record `decision` on approval `approvalId`: its row stops asking and its
+ * card leaves the composer. The `approval_resolved` event lands here, and so
+ * does a decision the backend already confirmed (or reported expired) before
+ * that event arrived — the card must not wait on the socket to go away.
+ */
+export function settleApproval(tl: Timeline, approvalId: string, decision: string, turnId = ""): Timeline {
+ const owner = turnId || tl.pendingApprovals.find((a) => a.approvalId === approvalId)?.turnId || "";
+ const settle = (turn: TurnItem): TurnItem => {
+ const blocks = mapToolDeep(
+ turn.blocks,
+ (b) => b.approval?.approvalId === approvalId,
+ (b) => (b.approval && b.approval.decision === null ? { ...b, approval: { ...b.approval, decision } } : b),
+ );
+ return blocks && blocks !== turn.blocks ? { ...turn, blocks } : turn;
+ };
+ const withTurn = owner
+ ? updateTurn(tl, owner, settle)
+ : (() => {
+ const items = tl.items.map((item) => (item.type === "turn" ? settle(item) : item));
+ return items.some((item, i) => item !== tl.items[i]) ? { ...tl, items } : tl;
+ })();
+ const pendingApprovals = withTurn.pendingApprovals.filter((a) => a.approvalId !== approvalId);
+ return pendingApprovals.length === withTurn.pendingApprovals.length ? withTurn : { ...withTurn, pendingApprovals };
+}
+
/** Update the sub-agent `agentId` spawned; a spawn call the stream never announced gets a row of its own. */
function updateSubagent(
turn: TurnItem,
@@ -860,6 +988,7 @@ export function reduceEvent(tl: Timeline, ev: AgentChatEvent): Timeline {
(b) => b.kind === "tool" && b.callId === callId,
(ex) => ({
...(ex?.question ? { question: ex.question } : {}),
+ ...(ex?.credential ? { credential: ex.credential } : {}),
...(ex?.subagent ? { subagent: ex.subagent } : {}),
kind: "tool",
callId,
@@ -979,31 +1108,18 @@ export function reduceEvent(tl: Timeline, ev: AgentChatEvent): Timeline {
}),
);
});
+ // The same call asking again (its CLI outlived an app restart) replaces
+ // the older card: nothing waits on that one any more.
+ const replaced = (a: PendingApproval) =>
+ a.approvalId === approvalId || (callId !== "" && a.turnId === turnId && a.callId === callId);
return {
...withTurn,
- pendingApprovals: [
- ...withTurn.pendingApprovals.filter((a) => a.approvalId !== approvalId),
- pending,
- ],
+ pendingApprovals: [...withTurn.pendingApprovals.filter((a) => !replaced(a)), pending],
};
}
- case "approval_resolved": {
- const approvalId = str(p.approval_id);
- const decision = str(p.decision);
- const withTurn = updateTurn(base, turnId, (turn) => {
- const blocks = mapToolDeep(
- turn.blocks,
- (b) => b.approval?.approvalId === approvalId,
- (b) => ({ ...b, approval: b.approval ? { ...b.approval, decision } : null }),
- );
- return blocks && blocks !== turn.blocks ? { ...turn, blocks } : turn;
- });
- return {
- ...withTurn,
- pendingApprovals: withTurn.pendingApprovals.filter((a) => a.approvalId !== approvalId),
- };
- }
+ case "approval_resolved":
+ return settleApproval(base, str(p.approval_id), str(p.decision), turnId);
case "question_required": {
const questionId = str(p.question_id);
@@ -1046,6 +1162,28 @@ export function reduceEvent(tl: Timeline, ev: AgentChatEvent): Timeline {
}));
}
+ case "credential_required": {
+ const requestId = str(p.request_id);
+ const env = str(p.env);
+ if (!requestId || !env) return base;
+ const credential: CredentialState = {
+ requestId,
+ asker: str(p.asker),
+ env,
+ label: str(p.label) || env,
+ description: str(p.description),
+ placeholder: str(p.placeholder),
+ replace: Boolean(p.replace),
+ expiresMs: num(p.expires_ms),
+ status: null,
+ };
+ return updateTurn(base, turnId, (turn) => withCredential(turn, credential, ev.ts_ms));
+ }
+
+ case "credential_resolved": {
+ return updateCredential(base, turnId, str(p.request_id), (c) => ({ ...c, status: str(p.status, "cancelled") }));
+ }
+
case "turn_finished": {
// Older event logs can contain a stale reader's timeout after the real
// process completed. Only a terminal event settles a turn, and it does
@@ -1074,8 +1212,10 @@ export function reduceEvent(tl: Timeline, ev: AgentChatEvent): Timeline {
answers: b.question.answers.map((a) => a ?? { text: "", optionIndex: null, source: "closed" }),
},
}
+ // Credential entry belongs to the person, not the turn's lifetime.
: b,
- ),
+ // An approval nobody answered stops offering buttons nothing listens to.
+ ).map(expireOpenApproval),
durationMs: num(p.duration_ms) ?? Math.max(0, ev.ts_ms - turn.startedMs),
usage:
p.usage && typeof p.usage === "object" && Object.keys(p.usage as object).length > 0
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/traceEntries.ts b/jarvis/ui/web/frontend/src/components/agentchat/traceEntries.ts
index 68d0508be0..d39779bfa9 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/traceEntries.ts
+++ b/jarvis/ui/web/frontend/src/components/agentchat/traceEntries.ts
@@ -25,7 +25,7 @@ import { fill } from "@/i18n";
import { cliVendor } from "@/lib/cliVendors";
import { resolveToolBrand } from "@/lib/toolBrand";
import { describeToolStep } from "@/lib/toolStepLabel";
-import { isQuestionTool, type ReasoningBlock, type TextBlock, type ToolBlock, type TurnBlock, type TurnStatus } from "./reduce";
+import { isCredentialTool, isQuestionTool, type ReasoningBlock, type TextBlock, type ToolBlock, type TurnBlock, type TurnStatus } from "./reduce";
import { toolDiff } from "./toolDiff";
import { traceToolIdentity, traceToolName } from "./traceActivity";
@@ -147,6 +147,7 @@ export function shortPath(path: string): string {
/** Plumbing calls draw nothing: loading a tool's schema, polling a question card. */
function isPlumbing(block: ToolBlock): boolean {
if (isQuestionTool(block.name) && !block.question) return true;
+ if (isCredentialTool(block.name) && !block.credential && !block.isError) return true;
if (block.name.replace(/^mcp__[^_]+__/, "") === "ToolSearch") {
return firstString(inputRecord(block), ["query"]).startsWith("select:");
}
@@ -155,7 +156,7 @@ function isPlumbing(block: ToolBlock): boolean {
/** Asks the person for something: drawn by the caller as its card. */
function needsPerson(block: ToolBlock): boolean {
- return Boolean(block.question) || Boolean(block.approval && block.approval.decision === null);
+ return Boolean(block.question || block.credential) || Boolean(block.approval && block.approval.decision === null);
}
function stripCapMarker(text: string): string {
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.draft.test.tsx b/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.draft.test.tsx
index 196c96607a..03d5bf4b69 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.draft.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.draft.test.tsx
@@ -1,7 +1,7 @@
import { act, cleanup, renderHook, waitFor } from "@testing-library/react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
-import { forgetHeldFiles, useChatAttachments } from "@/components/agentchat/useChatAttachments";
+import { discardHeldFiles, forgetHeldFiles, moveHeldFiles, restoreHeldFiles, useChatAttachments } from "@/components/agentchat/useChatAttachments";
/**
* Files held for an unsent message survive the composer unmounting, the way
@@ -86,4 +86,51 @@ describe("held files across unmounts", () => {
const second = renderHook(() => useChatAttachments(TARGET, () => {}));
expect(second.result.current.attachments).toEqual([]);
});
+
+ it("moves files to an assigned session and restores a failed submission there", async () => {
+ const blank = renderHook(() => useChatAttachments(TARGET, () => {}, { owner, key: "" }));
+ await act(async () => blank.result.current.attachFiles([png()]));
+ await waitFor(() => expect(blank.result.current.attachments).toHaveLength(1));
+ act(() => moveHeldFiles(owner, "", "created"));
+ const assigned = renderHook(() => useChatAttachments({ ...TARGET, sessionId: "created" }, () => {}, { owner, key: "created" }));
+ expect(blank.result.current.attachments).toEqual([]);
+ expect(assigned.result.current.attachments).toEqual([SHOT]);
+ expect(URL.revokeObjectURL).not.toHaveBeenCalled();
+ let sent!: ReturnType<typeof assigned.result.current.take>;
+ act(() => { sent = assigned.result.current.take(); });
+ expect(assigned.result.current.attachments).toEqual([]);
+ act(() => restoreHeldFiles(owner, "created", sent));
+ expect(assigned.result.current.attachments).toEqual([SHOT]);
+ expect(assigned.result.current.previews).toEqual({ "shot.png": "blob:shot" });
+ });
+
+ it("routes an upload completing after session creation to the assigned draft", async () => {
+ let uploaded!: (response: Response) => void;
+ vi.stubGlobal("fetch", vi.fn(() => new Promise<Response>((resolve) => { uploaded = resolve; })));
+ const blank = renderHook(() => useChatAttachments(TARGET, () => {}, { owner, key: "" }));
+ act(() => blank.result.current.attachFiles([png()]));
+ act(() => moveHeldFiles(owner, "", "created"));
+ const assigned = renderHook(() => useChatAttachments({ ...TARGET, sessionId: "created" }, () => {}, { owner, key: "created" }));
+ await act(async () => uploaded(new Response(JSON.stringify({ attachments: [SHOT], cwd: "C:\\work" }))));
+ await waitFor(() => expect(assigned.result.current.attachments).toEqual([SHOT]));
+ expect(blank.result.current.attachments).toEqual([]);
+ expect(URL.revokeObjectURL).not.toHaveBeenCalled();
+ const freshBlank = renderHook(() => useChatAttachments(TARGET, () => {}, { owner, key: "" }));
+ expect(freshBlank.result.current.attachments).toEqual([]);
+ });
+
+ it("keeps an abandoned upload out of an explicitly new blank chat", async () => {
+ let uploaded!: (response: Response) => void;
+ vi.stubGlobal("fetch", vi.fn(() => new Promise<Response>((resolve) => { uploaded = resolve; })));
+ const blank = renderHook(() => useChatAttachments(TARGET, () => {}, { owner, key: "" }));
+ act(() => blank.result.current.attachFiles([png()]));
+ act(() => discardHeldFiles(owner, ""));
+ await act(async () => uploaded(new Response(JSON.stringify({ attachments: [SHOT], cwd: "" }))));
+ expect(blank.result.current.attachments).toEqual([]);
+ expect(URL.revokeObjectURL).toHaveBeenCalledWith("blob:shot");
+ // The same mounted composer can still attach to the newly requested chat.
+ act(() => blank.result.current.attachFiles([png()]));
+ await act(async () => uploaded(new Response(JSON.stringify({ attachments: [SHOT], cwd: "" }))));
+ await waitFor(() => expect(blank.result.current.attachments).toEqual([SHOT]));
+ });
});
diff --git a/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.ts b/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.ts
index 66c79f199e..64c06bdc58 100644
--- a/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.ts
+++ b/jarvis/ui/web/frontend/src/components/agentchat/useChatAttachments.ts
@@ -77,8 +77,24 @@ const NOTHING_HELD: HeldFiles = { attachments: [], previews: {} };
* so two chats with the same session key never share files.
*/
const parked = new WeakMap<object, Map<string, HeldFiles>>();
+type AttachmentTarget = { key: string; discarded?: boolean };
+const attachmentTargets = new WeakMap<object, Map<string, AttachmentTarget>>();
const listeners = new Set<() => void>();
+function attachmentTarget(owner: object, key: string): AttachmentTarget {
+ let targets = attachmentTargets.get(owner);
+ if (!targets) {
+ targets = new Map();
+ attachmentTargets.set(owner, targets);
+ }
+ let target = targets.get(key);
+ if (!target) {
+ target = { key };
+ targets.set(key, target);
+ }
+ return target;
+}
+
function heldFor(owner: object, key: string): HeldFiles {
return parked.get(owner)?.get(key) ?? NOTHING_HELD;
}
@@ -102,8 +118,42 @@ export function releaseHeldFiles(files: HeldFiles): void {
Object.values(files.previews).forEach(revokePreview);
}
+/** Restore a submission to its assigned chat, including after session creation. */
+export function restoreHeldFiles(owner: object, key: string, files: HeldFiles): void {
+ updateHeld(owner, key, (current) => ({
+ attachments: [
+ ...files.attachments,
+ ...current.attachments.filter((item) => !files.attachments.some((back) => back.name === item.name)),
+ ],
+ previews: { ...files.previews, ...current.previews },
+ }));
+}
+
+/** Session creation gives the blank draft a stable key without losing new files. */
+export function moveHeldFiles(owner: object, from: string, to: string): void {
+ if (from === to) return;
+ // Uploads already in flight follow this draft to its assigned session.
+ // A later blank chat gets a fresh target instead of inheriting this alias.
+ attachmentTarget(owner, from).key = to;
+ attachmentTargets.get(owner)?.delete(from);
+ const held = heldFor(owner, from);
+ updateHeld(owner, from, () => NOTHING_HELD);
+ restoreHeldFiles(owner, to, held);
+}
+
+/** Explicit New Chat discards the previous blank draft, including late uploads. */
+export function discardHeldFiles(owner: object, key: string): void {
+ attachmentTarget(owner, key).discarded = true;
+ attachmentTargets.get(owner)?.delete(key);
+ updateHeld(owner, key, (current) => {
+ releaseHeldFiles(current);
+ return NOTHING_HELD;
+ });
+}
+
/** Let go of every draft's held files for one owner, pictures included. */
export function forgetHeldFiles(owner: object): void {
+ attachmentTargets.delete(owner);
const drafts = parked.get(owner);
if (!drafts) return;
drafts.forEach((held) => Object.values(held.previews).forEach(revokePreview));
@@ -162,6 +212,7 @@ export function useChatAttachments(
const attach = useCallback(
async (payload: PaneDropPayload) => {
if (isEmptyPayload(payload)) return;
+ const uploadTarget = attachmentTarget(owner, key);
setAnalyzing((n) => n + 1);
try {
const { attachments: found, cwd: folder } = await attachChatFilesIn({
@@ -177,6 +228,10 @@ export function useChatAttachments(
return;
}
const pictures = matchPreviews(found, payload.files);
+ if (uploadTarget.discarded) {
+ Object.values(pictures).forEach(revokePreview);
+ return;
+ }
if (folder) {
for (const item of found) {
if (!pictures[item.name] && attachmentMedia(item)) {
@@ -186,7 +241,7 @@ export function useChatAttachments(
}
// Written to the draft this drop was made in, even when the person
// switched away while the file was still being read.
- updateHeld(owner, key, (current) => {
+ updateHeld(owner, uploadTarget.key, (current) => {
const nextPreviews = { ...current.previews };
for (const [name, url] of Object.entries(pictures)) {
if (nextPreviews[name]) revokePreview(nextPreviews[name]);
@@ -268,13 +323,7 @@ export function useChatAttachments(
}, [owner, key]);
const restore = useCallback((files: HeldFiles) => {
- updateHeld(owner, key, (current) => ({
- attachments: [
- ...files.attachments,
- ...current.attachments.filter((item) => !files.attachments.some((back) => back.name === item.name)),
- ],
- previews: { ...files.previews, ...current.previews },
- }));
+ restoreHeldFiles(owner, key, files);
}, [owner, key]);
return {
diff --git a/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/BankedResets.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/BankedResets.test.tsx
new file mode 100644
index 0000000000..bb5368a90f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/BankedResets.test.tsx
@@ -0,0 +1,116 @@
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { consumeAccountReset, fetchAccountResets, ResetRequestError, type AccountResets } from "@/lib/agentResetsApi";
+import { openExternalUrl } from "@/lib/openExternal";
+import { BankedResets } from "./BankedResets";
+
+vi.mock("@/lib/agentResetsApi", async () => ({
+ ...await vi.importActual<typeof import("@/lib/agentResetsApi")>("@/lib/agentResetsApi"),
+ consumeAccountReset: vi.fn(), fetchAccountResets: vi.fn(),
+}));
+vi.mock("@/lib/openExternal", () => ({ openExternalUrl: vi.fn(async () => true) }));
+
+const inventory: AccountResets = {
+ account_id: "codex:test", status: "ok", available_count: 3,
+ credits: [{ id: "credit-1", title: "Full reset", description: "A saved reset", expires_at: "2099-11-01T12:00:00Z", redeemable: true }],
+ can_redeem: true, account_key: "a".repeat(64), manage_url: "https://chatgpt.com/codex/settings/usage", as_of: 1, usage: null,
+};
+const onUsage = vi.fn();
+const mount = () => render(<BankedResets accountId="codex:test" accountName="[email protected]" revision={0} onUsage={onUsage} />);
+beforeEach(() => {
+ vi.clearAllMocks();
+ sessionStorage.clear();
+ vi.mocked(fetchAccountResets).mockResolvedValue(inventory);
+ vi.mocked(consumeAccountReset).mockResolvedValue({ outcome: "reset", resets: { ...inventory, available_count: 0, credits: [], can_redeem: false } });
+});
+afterEach(cleanup);
+
+it("shows the authoritative count, expiry and selected account before sending", async () => {
+ mount();
+ await screen.findByText("3 available");
+ expect(screen.getByText(/Expires/).textContent).toContain("2099");
+ fireEvent.click(screen.getByRole("button", { name: "Use reset" }));
+ expect(screen.getByRole("group").textContent).toContain("[email protected]");
+ expect(consumeAccountReset).not.toHaveBeenCalled();
+ fireEvent.click(screen.getByRole("button", { name: "Cancel" }));
+ expect(consumeAccountReset).not.toHaveBeenCalled();
+});
+
+it("consumes once after confirmation and updates inventory", async () => {
+ mount();
+ await screen.findByText("3 available");
+ fireEvent.click(screen.getByRole("button", { name: "Use reset" }));
+ const confirm = screen.getByRole("button", { name: "Confirm and use reset" });
+ fireEvent.click(confirm);
+ fireEvent.click(confirm);
+ await screen.findByText("0 available");
+ expect(consumeAccountReset).toHaveBeenCalledTimes(1);
+ expect(consumeAccountReset).toHaveBeenCalledWith("codex:test", expect.objectContaining({ credit_id: "credit-1", account_key: inventory.account_key }));
+ expect(sessionStorage.getItem("subscription-reset:codex:test")).toBeNull();
+});
+
+it("retains the same attempt across lost replies and panel remounts without auto-retry", async () => {
+ vi.mocked(consumeAccountReset).mockRejectedValueOnce(new Error("offline"));
+ const view = mount();
+ await screen.findByText("3 available");
+ fireEvent.click(screen.getByRole("button", { name: "Use reset" }));
+ fireEvent.click(screen.getByRole("button", { name: "Confirm and use reset" }));
+ await screen.findByRole("alert");
+ const attempt = vi.mocked(consumeAccountReset).mock.calls[0][1];
+ view.unmount();
+ mount();
+ await screen.findByText("3 available");
+ expect(consumeAccountReset).toHaveBeenCalledTimes(1);
+ fireEvent.click(screen.getByRole("button", { name: "Check / retry same request" }));
+ await screen.findByText("0 available");
+ expect(vi.mocked(consumeAccountReset).mock.calls[1][1]).toEqual(attempt);
+});
+
+it("clears a rejected account identity without silently using the replacement account", async () => {
+ vi.mocked(consumeAccountReset).mockRejectedValueOnce(new ResetRequestError("account_changed"));
+ mount();
+ await screen.findByText("3 available");
+ fireEvent.click(screen.getByRole("button", { name: "Use reset" }));
+ fireEvent.click(screen.getByRole("button", { name: "Confirm and use reset" }));
+ await screen.findByRole("alert");
+ expect(sessionStorage.getItem("subscription-reset:codex:test")).toBeNull();
+ expect(screen.queryByRole("button", { name: "Use reset" })).toBeNull();
+});
+
+it("explains count-only inventory and lets the service select the next reset", async () => {
+ vi.mocked(fetchAccountResets).mockResolvedValue({ ...inventory, credits: null });
+ mount();
+ await screen.findByText(/reports a count only/);
+ fireEvent.click(screen.getByRole("button", { name: "Use reset" }));
+ fireEvent.click(screen.getByRole("button", { name: "Confirm and use reset" }));
+ await waitFor(() => expect(consumeAccountReset).toHaveBeenCalledWith("codex:test", expect.objectContaining({ credit_id: null })));
+});
+
+it("does not offer expired reset details", async () => {
+ vi.mocked(fetchAccountResets).mockResolvedValue({ ...inventory, credits: [{ ...inventory.credits![0], expires_at: "2000-01-01T00:00:00Z" }] });
+ mount();
+ await screen.findByText("3 available");
+ expect((screen.getByRole("button", { name: "Use reset" }) as HTMLButtonElement).disabled).toBe(true);
+});
+
+it("offers a browser handoff without pretending to know the provider's reset count", async () => {
+ vi.mocked(fetchAccountResets).mockResolvedValue({ ...inventory, status: "browser", available_count: null, credits: null, can_redeem: false, manage_url: "https://claude.ai/settings/usage" });
+ mount();
+ await screen.findByText(/manages resets on its usage page/);
+ expect(screen.queryByText("0 available")).toBeNull();
+ expect(screen.queryByRole("button", { name: "Use reset" })).toBeNull();
+ fireEvent.click(screen.getByRole("button", { name: "Open provider usage" }));
+ expect(openExternalUrl).toHaveBeenCalledWith("https://claude.ai/settings/usage");
+ expect(screen.getByText(/In the browser/).textContent).toContain("[email protected]");
+});
+
+it("keeps the account identity that was presented for confirmation across refreshes", async () => {
+ mount();
+ await screen.findByText("3 available");
+ fireEvent.click(screen.getByRole("button", { name: "Use reset" }));
+ vi.mocked(fetchAccountResets).mockResolvedValue({ ...inventory, account_key: "b".repeat(64), available_count: 4 });
+ fireEvent.click(screen.getByRole("button", { name: "Refresh resets" }));
+ await screen.findByText("4 available");
+ fireEvent.click(screen.getByRole("button", { name: "Confirm and use reset" }));
+ await waitFor(() => expect(consumeAccountReset).toHaveBeenCalledWith("codex:test", expect.objectContaining({ account_key: inventory.account_key })));
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/BankedResets.tsx b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/BankedResets.tsx
new file mode 100644
index 0000000000..0e0884807e
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/BankedResets.tsx
@@ -0,0 +1,152 @@
+import { useCallback, useEffect, useRef, useState } from "react";
+import { ExternalLink, Loader2, RefreshCw } from "lucide-react";
+import { fill, useT, useUiLanguage } from "@/i18n";
+import type { AccountUsage } from "@/lib/agentAccountsApi";
+import { consumeAccountReset, fetchAccountResets, ResetRequestError, type AccountResets, type ResetAttempt, type ResetCredit } from "@/lib/agentResetsApi";
+import { openExternalUrl } from "@/lib/openExternal";
+
+const BASE = "ide_side_panel.subscriptions.bank";
+const BUTTON = "rounded border border-border px-2 py-1 text-[11px] text-foreground hover:bg-secondary focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-50";
+
+function savedAttempt(accountId: string): ResetAttempt | null {
+ try {
+ const data = JSON.parse(sessionStorage.getItem(`subscription-reset:${accountId}`) || "null");
+ return data && typeof data.idempotency_key === "string" && typeof data.account_key === "string"
+ && (data.credit_id === null || typeof data.credit_id === "string") ? data : null;
+ } catch {
+ // An unavailable or corrupt storage entry must never trigger a redemption.
+ return null;
+ }
+}
+
+export function BankedResets({ accountId, accountName, revision, onUsage }: {
+ accountId: string; accountName: string; revision: number;
+ onUsage: (usage: AccountUsage) => void;
+}) {
+ const t = useT();
+ const lang = useUiLanguage();
+ const [snapshot, setSnapshot] = useState<AccountResets | null>(null);
+ const [loading, setLoading] = useState(true);
+ const [busy, setBusy] = useState(false);
+ const [error, setError] = useState("");
+ const [outcome, setOutcome] = useState("");
+ const [confirm, setConfirm] = useState<{ credit: ResetCredit | null; accountKey: string; accountName: string } | null>(null);
+ const [pending, setPending] = useState(() => savedAttempt(accountId));
+ const submitting = useRef(false);
+ const generation = useRef(0);
+ const load = useCallback(async () => {
+ if (submitting.current) return;
+ const id = ++generation.current;
+ setLoading(true);
+ try {
+ const next = await fetchAccountResets(accountId);
+ if (id !== generation.current) return;
+ setSnapshot(next);
+ setError("");
+ if (next.usage) onUsage(next.usage);
+ } catch {
+ if (id === generation.current) { setSnapshot(null); setError("unavailable"); }
+ } finally {
+ if (id === generation.current) setLoading(false);
+ }
+ }, [accountId, onUsage]);
+
+ useEffect(() => {
+ void load();
+ return () => { generation.current += 1; };
+ }, [load, revision]);
+
+ const redeem = async () => {
+ if (submitting.current) return;
+ const attempt = pending ?? (confirm ? {
+ idempotency_key: crypto.randomUUID(), credit_id: confirm.credit?.id ?? null, account_key: confirm.accountKey,
+ } : null);
+ if (!attempt) return;
+ // Save BEFORE sending. A closed panel, reload or lost reply must retain
+ // the same provider idempotency key; there is no automatic retry.
+ try { sessionStorage.setItem(`subscription-reset:${accountId}`, JSON.stringify(attempt)); }
+ catch { setError("storage_failed"); return; }
+ submitting.current = true;
+ setBusy(true);
+ setPending(attempt);
+ setError("");
+ setOutcome("");
+ generation.current += 1;
+ try {
+ const result = await consumeAccountReset(accountId, attempt);
+ sessionStorage.removeItem(`subscription-reset:${accountId}`);
+ setPending(null);
+ setConfirm(null);
+ setSnapshot(result.resets);
+ setOutcome(result.outcome);
+ if (result.resets.usage) onUsage(result.resets.usage);
+ } catch (err) {
+ if (err instanceof ResetRequestError && ["account_changed", "signed_out", "unsupported"].includes(err.code)) {
+ sessionStorage.removeItem(`subscription-reset:${accountId}`);
+ setPending(null);
+ setConfirm(null);
+ setSnapshot(null);
+ setError(err.code);
+ } else setError("uncertain");
+ } finally {
+ submitting.current = false;
+ setBusy(false);
+ setLoading(false);
+ }
+ };
+
+ const eligible = (credit: ResetCredit) => credit.redeemable
+ && (!credit.expires_at || Date.parse(credit.expires_at) > Date.now());
+ const useButton = (credit: ResetCredit | null) => (
+ <button type="button" className={BUTTON} disabled={busy || loading || !!pending || !!confirm || !snapshot?.can_redeem || (!!credit && !eligible(credit))}
+ onClick={() => { if (snapshot?.account_key) setConfirm({ credit, accountKey: snapshot.account_key, accountName }); setOutcome(""); }}>
+ {t(`${BASE}.use`)}
+ </button>
+ );
+ return (
+ <section aria-label={t(`${BASE}.title`)} className="mt-3 space-y-2 border-t border-border/60 pt-2 text-[11px]">
+ <div className="flex items-center justify-between gap-2">
+ <h4 className="font-medium text-foreground">{t(`${BASE}.title`)}</h4>
+ <button type="button" className={BUTTON} aria-label={t(`${BASE}.refresh`)} disabled={loading || busy} onClick={() => void load()}>
+ {loading ? <Loader2 className="h-3 w-3 animate-spin" /> : <RefreshCw className="h-3 w-3" />}
+ </button>
+ </div>
+ {loading && <p className="text-muted-foreground">{t(`${BASE}.loading`)}</p>}
+ {!loading && snapshot?.status === "ok" && <>
+ <p className="text-muted-foreground">{fill(t(`${BASE}.count`), { count: snapshot.available_count ?? 0 })}</p>
+ {snapshot.credits === null && snapshot.can_redeem && <div className="space-y-1">
+ <p className="text-muted-foreground">{t(`${BASE}.count_only`)}</p>{useButton(null)}
+ </div>}
+ {snapshot.credits?.map((credit) => <div key={credit.id} className="space-y-1 rounded-md bg-muted/50 p-2">
+ <p className="break-words font-medium text-foreground">{credit.title || t(`${BASE}.reset_name`)}</p>
+ {credit.description && <p className="break-words text-muted-foreground">{credit.description}</p>}
+ <p className="text-muted-foreground">{credit.expires_at
+ ? fill(t(`${BASE}.expires`), { when: new Intl.DateTimeFormat(lang, { dateStyle: "medium", timeStyle: "short" }).format(new Date(credit.expires_at)) })
+ : t(`${BASE}.expiry_unknown`)}</p>
+ {useButton(credit)}
+ </div>)}
+ </>}
+ {!loading && snapshot && snapshot.status !== "ok" && <p className="text-muted-foreground">{t(`${BASE}.${snapshot.status}`)}</p>}
+ {confirm && !pending && <div role="group" aria-label={t(`${BASE}.confirm_title`)} className="space-y-2 rounded-md border border-border p-2">
+ <p className="break-words text-foreground">{fill(t(`${BASE}.confirm_text`), { account: confirm.accountName, reset: confirm.credit?.title || t(`${BASE}.reset_name`) })}</p>
+ <p className="text-muted-foreground">{t(`${BASE}.irreversible`)}</p>
+ <div className="flex flex-wrap gap-2">
+ <button type="button" className={BUTTON} disabled={busy} onClick={() => void redeem()}>{t(`${BASE}.confirm`)}</button>
+ <button type="button" className={BUTTON} disabled={busy} onClick={() => setConfirm(null)}>{t(`${BASE}.cancel`)}</button>
+ </div>
+ </div>}
+ {pending && <div className="space-y-2">
+ <p className="text-muted-foreground">{t(`${BASE}.pending`)}</p>
+ <button type="button" className={BUTTON} disabled={busy} onClick={() => void redeem()}>{t(`${BASE}.${busy ? "using" : "retry"}`)}</button>
+ </div>}
+ {error && <p role="alert" className="text-destructive">{t(`${BASE}.${error}`)}</p>}
+ {outcome && <p role="status" className="text-foreground">{t(`${BASE}.${outcome}`)}</p>}
+ {snapshot?.manage_url && <div className="space-y-1">
+ <button type="button" className={BUTTON} onClick={async () => {
+ if (!await openExternalUrl(snapshot.manage_url!)) setError("open_failed");
+ }}><ExternalLink className="mr-1 inline h-3 w-3" />{t(`${BASE}.manage`)}</button>
+ <p className="text-muted-foreground">{fill(t(`${BASE}.browser_account`), { account: accountName })}</p>
+ </div>}
+ </section>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.test.tsx
index cab7adb6c4..d80f4988a0 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.test.tsx
@@ -5,6 +5,10 @@ import { useEventStore } from "@/store/events";
import { SubscriptionsTab } from "./SubscriptionsTab";
import { groupSubscriptions, resetText, windowLength } from "./subscriptionsModel";
+vi.mock("@/lib/agentResetsApi", () => ({ fetchAccountResets: vi.fn(async () => ({
+ status: "unsupported", credits: null, available_count: null, can_redeem: false, usage: null,
+})), consumeAccountReset: vi.fn(), ResetRequestError: class extends Error {} }));
+
vi.mock("@/lib/agentAccountsApi", async () => {
const actual = await vi.importActual<typeof import("@/lib/agentAccountsApi")>("@/lib/agentAccountsApi");
return { ...actual, fetchAgentAccounts: vi.fn(), fetchAgentUsage: vi.fn() };
diff --git a/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.tsx b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.tsx
index 9702d6c251..e7f8629c80 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/sidePanel/subscriptions/SubscriptionsTab.tsx
@@ -1,4 +1,4 @@
-import { useCallback, useEffect, useId, useState } from "react";
+import { useCallback, useEffect, useId, useRef, useState } from "react";
import { AlertTriangle, ChevronRight, Loader2, RefreshCw } from "lucide-react";
import { fill, useT, useUiLanguage } from "@/i18n";
import {
@@ -12,6 +12,7 @@ import { cn } from "@/lib/utils";
import { ProviderLogo } from "@/components/providers/ProviderLogo";
import { QuickTooltip } from "@/components/ui/tooltip";
import { useEventStore } from "@/store/events";
+import { BankedResets } from "./BankedResets";
import {
groupSubscriptions,
planName,
@@ -52,6 +53,7 @@ function useSubscriptions() {
const [error, setError] = useState("");
const [loading, setLoading] = useState(true);
const [force, setForce] = useState(0);
+ const identities = useRef(new Map<string, string>());
useEffect(() => {
let alive = true;
let timer: number | undefined;
@@ -65,10 +67,18 @@ function useSubscriptions() {
refresh = false;
if (nextUsage) pollMs = Math.max(MIN_POLL_MS, nextUsage.ttl_seconds * 1000);
if (alive) {
+ const nextIdentities = new Map(nextAccounts.platforms.flatMap((group) => group.accounts.map((account) =>
+ [account.id, JSON.stringify([account.email, account.config_dir, account.mode])] as const)));
+ const previousIdentities = identities.current;
+ identities.current = nextIdentities;
setAccounts(nextAccounts);
// A backend without the usage route answers null: the rows still
// show, just without numbers.
- setUsage(nextUsage?.accounts ?? []);
+ setUsage((previous) => (nextUsage?.accounts ?? []).map((reading) => {
+ const newer = previous.find((item) => item.account_id === reading.account_id);
+ return newer && previousIdentities.get(reading.account_id) === nextIdentities.get(reading.account_id)
+ && (newer.as_of ?? 0) > (reading.as_of ?? 0) ? newer : reading;
+ }));
setError("");
}
} catch (err) {
@@ -86,7 +96,10 @@ function useSubscriptions() {
};
}, [force]);
const refresh = useCallback(() => setForce((value) => value + 1), []);
- return { accounts, usage, error, loading, refresh };
+ const updateUsage = useCallback((reading: AccountUsage) => {
+ setUsage((previous) => [...previous.filter((item) => item.account_id !== reading.account_id), reading]);
+ }, []);
+ return { accounts, usage, error, loading, refresh, updateUsage, revision: force };
}
/** Wall-clock minutes, so reset countdowns stay true between polls. */
@@ -191,7 +204,9 @@ function UsageDetails({ usage, now }: { usage: AccountUsage | null; now: number
);
}
-function SubscriptionLine({ row, now }: { row: SubscriptionRow; now: number }) {
+interface ResetProps { revision: number; onUsage: (usage: AccountUsage) => void }
+
+function SubscriptionLine({ row, now, revision, onUsage }: { row: SubscriptionRow; now: number } & ResetProps) {
const t = useT();
const [open, setOpen] = useState(false);
const detailsId = useId();
@@ -251,13 +266,14 @@ function SubscriptionLine({ row, now }: { row: SubscriptionRow; now: number }) {
{open && (
<div id={detailsId} data-testid="subscription-details" className="pb-2 pl-7 pr-2 pt-1">
<UsageDetails usage={row.usage} now={now} />
+ <BankedResets key={`${account.id}:${account.email ?? ""}`} accountId={account.id} accountName={title} revision={revision} onUsage={onUsage} />
</div>
)}
</li>
);
}
-function SubscriptionBlock({ group, now }: { group: SubscriptionGroup; now: number }) {
+function SubscriptionBlock({ group, now, revision, onUsage }: { group: SubscriptionGroup; now: number } & ResetProps) {
return (
<section data-testid="subscription-group" data-platform={group.platform} className="px-2 py-2">
<header className="flex items-center gap-2 px-2 pb-1">
@@ -267,7 +283,7 @@ function SubscriptionBlock({ group, now }: { group: SubscriptionGroup; now: numb
</header>
<ul className="space-y-0.5">
{group.rows.map((row) => (
- <SubscriptionLine key={row.account.id} row={row} now={now} />
+ <SubscriptionLine key={row.account.id} row={row} now={now} revision={revision} onUsage={onUsage} />
))}
</ul>
</section>
@@ -281,7 +297,7 @@ function SubscriptionBlock({ group, now }: { group: SubscriptionGroup; now: numb
*/
export function SubscriptionsTab() {
const t = useT();
- const { accounts, usage, error, loading, refresh } = useSubscriptions();
+ const { accounts, usage, error, loading, refresh, updateUsage, revision } = useSubscriptions();
const now = useMinuteClock();
const groups = groupSubscriptions(accounts, usage);
return (
@@ -325,7 +341,7 @@ export function SubscriptionsTab() {
)}
<div className="divide-y divide-border/60">
{groups.map((group) => (
- <SubscriptionBlock key={group.platform} group={group} now={now} />
+ <SubscriptionBlock key={group.platform} group={group} now={now} revision={revision} onUsage={updateUsage} />
))}
</div>
</div>
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadComposer.tsx b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadComposer.tsx
index 020c688436..b74f65d3f0 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadComposer.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadComposer.tsx
@@ -4,12 +4,12 @@ import { ChatAttachmentStrip } from "@/components/agentchat/ChatAttachmentStrip"
import { ChatMarkdown } from "@/components/agentchat/ChatMarkdown";
import { ComposerTypeahead } from "@/components/agentchat/ComposerTypeahead";
import { DictationButton } from "@/components/agentchat/DictationButton";
-import { runningTurn, type QuestionState, type Timeline, type ToolBlock, type TurnItem, type UserItem } from "@/components/agentchat/reduce";
+import { runningTurn, type PendingApproval, type QuestionState, type Timeline, type ToolBlock, type TurnItem, type UserItem } from "@/components/agentchat/reduce";
import { releaseHeldFiles, useChatAttachments, type HeldFiles } from "@/components/agentchat/useChatAttachments";
import { useComposerDictation } from "@/components/agentchat/useComposerDictation";
import { useComposerTypeahead } from "@/components/agentchat/useComposerTypeahead";
import { useT } from "@/i18n";
-import type { ChatAttachment, PlanDecision } from "@/lib/agentChatApi";
+import type { AgentChatSession, ApprovalDecision, ChatAttachment, PlanDecision } from "@/lib/agentChatApi";
import { joinProviderOptions, type ComposerDraft, type ProviderOption } from "@/store/agentChat";
import { cn } from "@/lib/utils";
import { effortLadder, snapEffort } from "@/lib/effortLadder";
@@ -44,7 +44,6 @@ interface SentMessage {
afterSeq: number;
}
let lastSent: SentMessage | null = null;
-
/** How long after sending Escape still takes a message back once the agent started thinking. */
export const RECALL_WINDOW_MS = 15_000;
@@ -107,33 +106,60 @@ const PLAN_PROSE = cn(
"[&>div>:first-child]:mt-0 [&>div>:last-child]:mb-0",
);
-function ApprovalPanel({ timeline, onDecide }: { timeline: Timeline; onDecide: (id: string, decision: "allow" | "allow_always" | "deny") => void }) {
+/** The oldest approval the agent waits on, shown at the top of the composer. */
+function ApprovalPanel({ timeline }: { timeline: Timeline }) {
const pending = timeline.pendingApprovals[0];
if (!pending) return null;
+ // Keyed by the card: a new card starts with fresh buttons and no old error.
+ return <ApprovalCard key={pending.approvalId} pending={pending} count={timeline.pendingApprovals.length} />;
+}
+
+function ApprovalCard({ pending, count }: { pending: PendingApproval; count: number }) {
+ const [sending, setSending] = useState<ApprovalDecision | null>(null);
+ const [error, setError] = useState("");
const input = pending.input && typeof pending.input === "object" ? pending.input as Record<string, unknown> : {};
// Claude Code's finished plan asks here too, and the plan itself is what gets approved.
const isPlan = pending.name === "ExitPlanMode";
const plan = isPlan ? String(input.plan ?? "").trim() : "";
const detail = isPlan ? "" : String(input.command ?? input.file_path ?? input.path ?? input.url ?? "");
- return <div data-testid="thread-approval" className="border-b border-border px-4 py-3">
+ // One answer per card: a second click while the first is on its way would
+ // land on a card that is already closed.
+ const decide = async (decision: ApprovalDecision) => {
+ if (sending) return;
+ setSending(decision);
+ setError("");
+ try {
+ await useThreadChatStore.getState().decide(pending.approvalId, decision);
+ } catch (err) {
+ setError(err instanceof Error ? err.message : String(err));
+ setSending(null);
+ }
+ };
+ const spinner = (decision: ApprovalDecision) => sending === decision && <Loader2 aria-hidden className="h-3.5 w-3.5 animate-spin" />;
+ return <div data-testid="thread-approval" aria-busy={sending !== null} className="border-b border-border px-4 py-3">
<div className="flex items-start gap-2.5">
{isPlan
? <ListChecks aria-hidden className="mt-0.5 h-4 w-4 shrink-0 text-accent" />
: <ShieldAlert aria-hidden className="mt-0.5 h-4 w-4 shrink-0 text-warning" />}
<div className="min-w-0 flex-1">
- <p className="text-sm font-medium text-foreground-strong">{isPlan ? "Plan ready" : "Approval needed"}{timeline.pendingApprovals.length > 1 ? ` (${timeline.pendingApprovals.length})` : ""}</p>
+ <p className="flex items-baseline gap-2 text-sm font-medium text-foreground-strong">
+ <span className="min-w-0 flex-1">{isPlan ? "Plan ready" : "Approval needed"}</span>
+ {count > 1 && <span data-testid="thread-approval-count" className="shrink-0 text-xs font-normal tabular-nums text-muted-foreground">1 of {count}</span>}
+ </p>
<p className="mt-0.5 text-sm text-muted-foreground">{isPlan ? "Build it, or keep planning and say what to change." : pending.summary || pending.name}</p>
{plan && <div data-testid="thread-approval-plan" className={cn("mt-2 max-h-72 overflow-auto rounded-md bg-secondary px-3 py-2 scrollbar-jarvis", PLAN_PROSE)}><ChatMarkdown text={plan} /></div>}
{detail && <pre className="mt-2 max-h-32 overflow-auto whitespace-pre-wrap rounded-md bg-secondary px-2.5 py-1.5 font-mono text-xs text-foreground scrollbar-jarvis">{detail}</pre>}
+ {error && <p role="alert" className="mt-2 text-xs text-destructive">{error}</p>}
</div>
</div>
<div className="mt-3 flex flex-wrap justify-end gap-2">
- <button type="button" onClick={() => onDecide(pending.approvalId, "deny")}
- className="rounded-lg px-3 py-1.5 text-sm text-muted-foreground hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">{isPlan ? "Keep planning" : "Decline"}</button>
- {!isPlan && <button type="button" onClick={() => onDecide(pending.approvalId, "allow_always")}
- className="rounded-lg border border-border px-3 py-1.5 text-sm text-foreground hover:bg-secondary focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">Always allow</button>}
- <button type="button" data-testid="thread-approve" onClick={() => onDecide(pending.approvalId, "allow")}
- className="rounded-lg bg-accent px-3 py-1.5 text-sm font-medium text-accent-foreground hover:opacity-90 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">{isPlan ? "Build it" : "Approve"}</button>
+ <button type="button" disabled={sending !== null} onClick={() => void decide("deny")}
+ className="flex items-center gap-1.5 rounded-lg px-3 py-1.5 text-sm text-muted-foreground hover:bg-secondary hover:text-foreground disabled:opacity-50 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">{spinner("deny")}{isPlan ? "Keep planning" : "Decline"}</button>
+ {!isPlan && <button type="button" disabled={sending !== null} onClick={() => void decide("allow_always")}
+ title="Approve, and stop asking for this in this thread"
+ className="flex items-center gap-1.5 rounded-lg border border-border px-3 py-1.5 text-sm text-foreground hover:bg-secondary disabled:opacity-50 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">{spinner("allow_always")}Always allow</button>}
+ <button type="button" data-testid="thread-approve" disabled={sending !== null} onClick={() => void decide("allow")}
+ className="flex items-center gap-1.5 rounded-lg bg-accent px-3 py-1.5 text-sm font-medium text-accent-foreground hover:opacity-90 disabled:opacity-50 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">{spinner("allow")}{isPlan ? "Build it" : "Approve"}</button>
</div>
</div>;
}
@@ -252,6 +278,7 @@ function QuestionPanel({ timeline }: { timeline: Timeline }) {
export function ThreadComposer({
threadKey,
prepareDraft,
+ onSessionCreated,
placeholder = "Ask for changes, send follow-ups, or attach images",
autoFocusNonce,
strip,
@@ -260,6 +287,7 @@ export function ThreadComposer({
/** Which thread the box is typing for — the session id, or `draft:<project>`. */
threadKey: string;
prepareDraft: () => Promise<string | null>;
+ onSessionCreated?: (session: AgentChatSession) => void;
placeholder?: string;
autoFocusNonce: number;
/** The strip that hangs under the card — where the agent works and on which branch. */
@@ -391,9 +419,9 @@ export function ThreadComposer({
setStarting(false);
}
}
- await useThreadChatStore.getState().send(text, attachments);
+ await useThreadChatStore.getState().send(text, attachments, [], onSessionCreated);
return !useThreadChatStore.getState().lastError;
- }, [prepareDraft]);
+ }, [prepareDraft, onSessionCreated]);
// A queued message goes out the moment the agent is free again.
useEffect(() => {
@@ -454,7 +482,7 @@ export function ThreadComposer({
lastSent = null;
setPaused(true);
if (activeSessionId) useRecalledMessages.getState().hide(activeSessionId, asked.id);
- void useThreadChatStore.getState().cancel();
+ void useThreadChatStore.getState().cancel("thread-recall");
const current = drafts.get(threadKey) ?? "";
setValue(current.trim() ? `${text}\n${current}` : text);
files.restore(held);
@@ -491,10 +519,6 @@ export function ThreadComposer({
rememberSeat(useThreadChatStore.getState().draft);
};
- const decide = (approvalId: string, decision: "allow" | "allow_always" | "deny") => {
- void useThreadChatStore.getState().decide(approvalId, decision);
- };
-
const canSend = (value.trim().length > 0 || files.attachments.length > 0 || dictation.dictating) && files.analyzing === 0 && !starting;
const error = problem || lastError || "";
@@ -517,7 +541,7 @@ export function ThreadComposer({
"relative z-10 overflow-hidden rounded-3xl border border-border bg-card shadow-[0_8px_24px_-12px_rgb(var(--scrim-rgb)/0.5)] transition-colors focus-within:border-border-strong",
files.dragging && "border-accent ring-2 ring-accent/40",
)}>
- <ApprovalPanel timeline={timeline} onDecide={decide} />
+ <ApprovalPanel timeline={timeline} />
<QuestionPanel timeline={timeline} />
{!running && <PlanPanel timeline={timeline} provider={provider} />}
{(files.attachments.length > 0 || files.analyzing > 0) && <div className="px-3 pt-3 sm:px-4">
@@ -549,7 +573,7 @@ export function ThreadComposer({
startLabel={t("chats_view.dictation_start")} stopLabel={t("chats_view.dictation_stop")} shape="round" />
{running && !canSend
? <button type="button" aria-label="Stop the agent" title="Stop" data-testid="thread-stop"
- onClick={() => { setPaused(true); void useThreadChatStore.getState().cancel(); }}
+ onClick={() => { setPaused(true); void useThreadChatStore.getState().cancel("thread-stop-button"); }}
className="flex h-8 w-8 items-center justify-center rounded-full bg-foreground text-background transition-transform duration-150 hover:scale-105 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
<Square className="h-3 w-3 fill-current" />
</button>
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPickers.tsx b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPickers.tsx
index 200ccad7ee..3bf60c43ca 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPickers.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPickers.tsx
@@ -5,7 +5,7 @@ import { AgentMark } from "@/components/agentic/AgentMark";
import { isUnguardedPermissionMode, permissionModeIcon } from "@/components/agentchat/permissionIcons";
import { Combobox, type ComboboxGroup, type ComboboxOption } from "@/components/ui/combobox";
import { useT } from "@/i18n";
-import type { CuratedModel } from "@/lib/agentChatApi";
+import { isApiRunner, type CuratedModel } from "@/lib/agentChatApi";
import { offeredModels, useSavedHiddenModels } from "@/lib/agentProviderPrefs";
import { effortLadder } from "@/lib/effortLadder";
import { rankModels } from "@/lib/modelRanking";
@@ -103,6 +103,8 @@ export function AgentModelPicker({
// The rail on the panel's left: one mark per coding agent, plus favourites.
const sections = useMemo<BrainSection[]>(() => providers.map((provider) => ({
id: provider.id,
+ family: provider.family,
+ access: provider.keyless ? "local" : isApiRunner(provider.runner) ? "api" : "subscription",
label: agentName(provider),
icon: <AgentMark agent={provider.agent ?? ""} label={provider.label} logoUrl={provider.logoUrl} variant="plain" size="sm" />,
muted: !provider.connected || (lockedProvider !== null && provider.id !== lockedProvider),
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPrompts.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPrompts.test.tsx
index 1df2698c75..a18c1b7eef 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPrompts.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadPrompts.test.tsx
@@ -1,6 +1,6 @@
import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
-import type { AgentChatCatalog, AgentChatEvent, AgentChatSession } from "@/lib/agentChatApi";
+import { AgentChatApiError, type AgentChatCatalog, type AgentChatEvent, type AgentChatSession } from "@/lib/agentChatApi";
import type { IdeProject } from "@/lib/agenticIdeApi";
import { useIdeProjectsStore } from "@/store/ideProjects";
import { useIdeThreadsStore } from "@/store/ideThreads";
@@ -149,3 +149,51 @@ describe("a coding agent's questions and plans in a thread", () => {
expect(api.resolveAgentChatApproval).toHaveBeenCalledWith("s1", "a1", "allow");
});
});
+
+describe("a thread's approval card", () => {
+ function ask(approvalId: string, callId: string, command: string): [string, Record<string, unknown>] {
+ return ["approval_required", { turn_id: "t1", approval_id: approvalId, call_id: callId, name: "Bash", input: { command }, summary: command }];
+ }
+
+ it("closes as soon as the backend took the answer, and stays closed for one click", async () => {
+ await openThread();
+ feed([ask("a1", "c1", "ls")]);
+ let finish: () => void = () => undefined;
+ api.resolveAgentChatApproval.mockImplementation(() => new Promise<void>((resolve) => { finish = resolve; }));
+ fireEvent.click(screen.getByTestId("thread-approve"));
+ // On its way: every button holds still, so a second click cannot answer twice.
+ for (const button of screen.getByTestId("thread-approval").querySelectorAll("button")) {
+ expect((button as HTMLButtonElement).disabled).toBe(true);
+ }
+ fireEvent.click(screen.getByTestId("thread-approve"));
+ await act(async () => { finish(); });
+ expect(api.resolveAgentChatApproval).toHaveBeenCalledTimes(1);
+ // Gone before the socket's approval_resolved arrived.
+ expect(screen.queryByTestId("thread-approval")).toBeNull();
+ });
+
+ it("drops a card the backend says expired and shows the live one behind it", async () => {
+ await openThread();
+ // 2026-10-06: the card from before a restart sat on top of the live one.
+ feed([ask("old", "c1", "ls"), ask("live", "c2", "git log")]);
+ expect(screen.getByTestId("thread-approval-count").textContent).toBe("1 of 2");
+ api.resolveAgentChatApproval.mockRejectedValueOnce(new AgentChatApiError("expired", 410));
+ await act(async () => { fireEvent.click(screen.getByTestId("thread-approve")); });
+ const card = screen.getByTestId("thread-approval");
+ expect(card.textContent).toContain("git log");
+ expect(card.querySelector("[role=alert]")).toBeNull();
+ expect(screen.queryByTestId("thread-approval-count")).toBeNull();
+ await act(async () => { fireEvent.click(screen.getByTestId("thread-approve")); });
+ expect(api.resolveAgentChatApproval).toHaveBeenLastCalledWith("s1", "live", "allow");
+ });
+
+ it("keeps the card and says why when the answer did not go through", async () => {
+ await openThread();
+ feed([ask("a1", "c1", "ls")]);
+ api.resolveAgentChatApproval.mockRejectedValueOnce(new AgentChatApiError("server error", 500));
+ await act(async () => { fireEvent.click(screen.getByTestId("thread-approve")); });
+ const card = screen.getByTestId("thread-approval");
+ expect(card.querySelector("[role=alert]")?.textContent).toBe("server error");
+ expect((screen.getByTestId("thread-approve") as HTMLButtonElement).disabled).toBe(false);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadRecall.test.tsx b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadRecall.test.tsx
index 2e763ae643..c47809aa10 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadRecall.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadRecall.test.tsx
@@ -143,7 +143,7 @@ describe("Escape takes back a message the agent has not started on", () => {
await act(async () => { fireEvent.keyDown(window, { key: "Escape" }); });
- expect(api.cancelAgentChatTurn).toHaveBeenCalledWith("s1");
+ expect(api.cancelAgentChatTurn).toHaveBeenCalledWith("s1", "thread-recall");
expect(box.value).toBe("fix the login");
expect(screen.getByTestId("chat-attachment-shot.png")).toBeTruthy();
});
@@ -165,7 +165,7 @@ describe("Escape takes back a message the agent has not started on", () => {
await act(async () => { fireEvent.keyDown(window, { key: "Escape" }); });
- expect(api.cancelAgentChatTurn).toHaveBeenCalledWith("s1");
+ expect(api.cancelAgentChatTurn).toHaveBeenCalledWith("s1", "thread-recall");
expect(box.value).toBe("fix the login");
expect(screen.getByTestId("chat-attachment-shot.png")).toBeTruthy();
});
@@ -206,7 +206,7 @@ describe("Escape takes back a message the agent has not started on", () => {
await act(async () => { fireEvent.keyDown(box, { key: "Escape" }); });
- expect(api.cancelAgentChatTurn).toHaveBeenCalledWith("s1");
+ expect(api.cancelAgentChatTurn).toHaveBeenCalledWith("s1", "thread-recall");
expect(box.value).toBe(typed);
expect(screen.getAllByTestId("thread-user-message")).toHaveLength(1);
});
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadTimeline.tsx b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadTimeline.tsx
index 500de4c29a..a8fcd2f279 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadTimeline.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadTimeline.tsx
@@ -1,8 +1,9 @@
import { createContext, memo, useCallback, useContext, useEffect, useLayoutEffect, useMemo, useRef, useState, type ReactNode } from "react";
import { ArrowDown, Bot, Brain, Check, ChevronDown, ChevronRight, Copy, Diff, FileText, Hammer, MessageCircleQuestion, ShieldAlert, X } from "lucide-react";
import { ChatMarkdown } from "@/components/agentchat/ChatMarkdown";
+import { CredentialCard } from "@/components/agentchat/CredentialCard";
import { QuestionCard } from "@/components/agentchat/QuestionCard";
-import type { TextBlock, TimelineItem, ToolBlock, TurnBlock, TurnItem, UserItem } from "@/components/agentchat/reduce";
+import { waitsOnCard, type TextBlock, type TimelineItem, type ToolBlock, type TurnBlock, type TurnItem, type UserItem } from "@/components/agentchat/reduce";
import { toolDiff, type DiffFile } from "@/components/agentchat/toolDiff";
import { CallMark, StretchMark } from "@/components/agentchat/TraceTimeline";
import { plural, readableOutput, traceDuration, type Call } from "@/components/agentchat/traceEntries";
@@ -30,7 +31,7 @@ import { listSubagents } from "./subagents";
* MIT License, Copyright (c) 2026 T3 Tools Inc. — third_party/t3code/LICENSE.
*/
-const PROSE = cn(
+export const PROSE = cn(
"py-1 prose prose-neutral max-w-none text-base leading-6 text-foreground dark:prose-invert dark:text-foreground [overflow-wrap:anywhere]",
"[&>div>:first-child]:mt-0 [&>div>:last-child]:mb-0",
"prose-p:my-2 prose-p:text-foreground prose-li:text-foreground prose-strong:text-foreground-strong",
@@ -263,7 +264,7 @@ function ThinkingRow({ grouped = false }: { grouped?: boolean }) {
* A picture in the agent's words — an image it viewed or made — reads as a
* small preview in the thread; a click opens it full size.
*/
-const THREAD_MEDIA = "[&_[data-kind=image]_img]:max-h-48 [&_[data-kind=image]_img]:max-w-sm [&_[data-kind=image]_img]:border [&_[data-kind=image]_img]:border-border";
+export const THREAD_MEDIA = "[&_[data-kind=image]_img]:max-h-48 [&_[data-kind=image]_img]:max-w-sm [&_[data-kind=image]_img]:border [&_[data-kind=image]_img]:border-border";
function StepRow({ item, running, grouped, onOpenChange }: { item: WorkItem; running: boolean; grouped?: boolean; onOpenChange?: OpenChange }) {
if (item.kind === "call") return <CallRow call={item.call} stamp={item.startedMs} grouped={grouped} onOpenChange={onOpenChange} />;
@@ -545,7 +546,7 @@ export type Decide = (id: string, decision: ApprovalDecision) => void | Promise<
/** A call still waiting for the person: an open question or an undecided approval. */
function waitsForPerson(block: ToolBlock): boolean {
- return Boolean(block.question && !block.question.closed) || Boolean(block.approval && block.approval.decision === null);
+ return waitsOnCard(block) || Boolean(block.approval && block.approval.decision === null);
}
function pendingLabel(block: ToolBlock, t: (key: string) => string): { icon: ReactNode; text: string } | null {
@@ -560,7 +561,7 @@ function pendingLabel(block: ToolBlock, t: (key: string) => string): { icon: Rea
* An approval answered in the turn itself: the waiting line, what the agent
* asks for, and the three answers under it.
*/
-function ApprovalPrompt({ block, onDecide }: { block: ToolBlock; onDecide?: Decide }) {
+export function ApprovalPrompt({ block, onDecide }: { block: ToolBlock; onDecide?: Decide }) {
const t = useT();
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
@@ -643,8 +644,12 @@ export const ThreadTurn = memo(function ThreadTurn({ turn, prompts = "composer",
const running = turn.status === "running";
const inline = prompts === "inline";
const pictures = useMemo(() => callPictures(turn.blocks), [turn.blocks]);
+ // Keep credential fields mounted while the surrounding work folds at turn end.
+ // The pasted value lives only in the field, so a remount would discard it.
+ const credentials = turn.blocks.filter((block): block is ToolBlock => block.kind === "tool" && Boolean(block.credential));
const rows = useMemo(
- () => buildThreadRows(turn.blocks, { t, lang, status: turn.status }).filter((row) => !(row.kind === "text" && pictures.moved.has(row.block.id))),
+ () => buildThreadRows(turn.blocks, { t, lang, status: turn.status }).filter((row) =>
+ !(row.kind === "text" && pictures.moved.has(row.block.id)) && !(row.kind === "pending" && row.block.credential)),
[turn.blocks, t, lang, turn.status, pictures],
);
const answer = answerText(turn);
@@ -672,6 +677,8 @@ export const ThreadTurn = memo(function ThreadTurn({ turn, prompts = "composer",
if (row.kind === "thought") return <ThoughtView key={row.id} text={row.text} />;
if (row.kind === "text") return <div key={row.id} className={cn(PROSE, THREAD_MEDIA)}><ChatMarkdown text={row.text} /></div>;
if (row.kind === "agent") return <SubagentCard key={row.id} block={row.block} turn={turn} />;
+ // A secret is pasted where it is asked for, never into a composer.
+ if (row.block.credential) return <CredentialCard key={row.id} credential={row.block.credential} />;
if (inline) {
if (row.block.question) return <QuestionCard key={row.id} question={row.block.question} />;
if (row.block.approval && row.block.approval.decision === null) return <ApprovalPrompt key={row.id} block={row.block} onDecide={onDecide} />;
@@ -700,6 +707,7 @@ export const ThreadTurn = memo(function ThreadTurn({ turn, prompts = "composer",
</div>}
{folded && agents.length > 0 && <div className="space-y-1.5 pt-1" data-testid="thread-turn-subagents">{agents.map(renderRow)}</div>}
{folded && waiting.map(renderRow)}
+ {credentials.map((block) => <CredentialCard key={block.credential!.requestId} credential={block.credential!} />)}
{shown.slice(0, extraAt).map(renderRow)}
{!folded && extraNodes}
{shown.slice(extraAt).map(renderRow)}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadView.tsx b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadView.tsx
index 6a8cb54136..82ec82e45d 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadView.tsx
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/ThreadView.tsx
@@ -1,6 +1,7 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { Check, ChevronDown, FolderPlus } from "lucide-react";
import { prepareGit } from "@/lib/gitApi";
+import type { AgentChatSession } from "@/lib/agentChatApi";
import { useEventStore } from "@/store/events";
import { useIdeProjectsStore } from "@/store/ideProjects";
import { useIdeThreadsStore } from "@/store/ideThreads";
@@ -60,6 +61,10 @@ export function ThreadView({ onScreen }: { onScreen: boolean }) {
?? fallbackProject;
const folder = session?.cwd || project?.path || "";
const isDraft = selection.sessionId === null;
+ const originatingProject = project?.id;
+ const rememberCreated = useCallback((created: AgentChatSession) => {
+ if (originatingProject) useIdeThreadsStore.getState().rememberProject(created.session_id, originatingProject);
+ }, [originatingProject]);
// The terminal drawer and its caption toggle open shells in this folder.
useEffect(() => { useThreadTerminalsStore.getState().setFolder(folder); }, [folder]);
@@ -253,7 +258,7 @@ export function ThreadView({ onScreen }: { onScreen: boolean }) {
</div>
</ThreadPopover>
<div className="mt-7 w-full">
- <ThreadComposer threadKey={threadKey} prepareDraft={prepareDraft} autoFocusNonce={focusNonce + (onScreen ? 1 : 0)} strip={strip} onScreen={onScreen} />
+ <ThreadComposer threadKey={threadKey} prepareDraft={prepareDraft} onSessionCreated={rememberCreated} autoFocusNonce={focusNonce + (onScreen ? 1 : 0)} strip={strip} onScreen={onScreen} />
</div>
</div>
: <OpenSubagent.Provider value={setOpenAgent}>
@@ -264,7 +269,7 @@ export function ThreadView({ onScreen }: { onScreen: boolean }) {
: <ThreadTimeline items={items} sessionId={selection.sessionId} bottomInset={composerHeight} folder={folder} />}
<div ref={composerBox} className="pointer-events-none absolute inset-x-0 bottom-0 bg-gradient-to-t from-background from-70% to-transparent px-5 pb-4 pt-6">
<div className="pointer-events-auto">
- <ThreadComposer threadKey={threadKey} prepareDraft={prepareDraft} autoFocusNonce={focusNonce} strip={strip} onScreen={onScreen} />
+ <ThreadComposer threadKey={threadKey} prepareDraft={prepareDraft} onSessionCreated={rememberCreated} autoFocusNonce={focusNonce} strip={strip} onScreen={onScreen} />
</div>
</div>
</OpenSubagent.Provider>}
diff --git a/jarvis/ui/web/frontend/src/components/agentic/threads/subagents.ts b/jarvis/ui/web/frontend/src/components/agentic/threads/subagents.ts
index e68e5007c5..b77ab9eef2 100644
--- a/jarvis/ui/web/frontend/src/components/agentic/threads/subagents.ts
+++ b/jarvis/ui/web/frontend/src/components/agentic/threads/subagents.ts
@@ -80,6 +80,7 @@ function waitsForPerson(blocks: TurnBlock[]): boolean {
return blocks.some((b) => b.kind === "tool" && (
(b.approval !== null && b.approval.decision === null)
|| (b.question !== undefined && !b.question.closed)
+ || (b.credential !== undefined && b.credential.status === null)
));
}
diff --git a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.test.tsx b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.test.tsx
index 931e2421bc..e64c703d5a 100644
--- a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.test.tsx
@@ -1,35 +1,51 @@
import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
-import { afterEach, describe, expect, it, vi } from "vitest";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { AppshotRecordingPanel } from "./AppshotRecordingPanel";
-import type { AppshotSettings } from "@/lib/appshotApi";
+import { loadLocaleChunk } from "@/i18n";
+import type { AppshotLibraryItem, AppshotSettings } from "@/lib/appshotApi";
const settings = { enabled: true, recording_hotkey: "ctrl+shift+9" } as AppshotSettings;
const ready = { available: true, detail: "", permission_required: false };
function response(body: unknown, status = 200) {
return { ok: status < 400, status, json: async () => body } as Response;
}
+function video(id: string): AppshotLibraryItem {
+ return {
+ id: `recording_${id}`, variant: "original", path: `/videos/${id}.mp4`, mime: "video/mp4",
+ width: 1920, height: 1080, label: "Screen recording", app_name: "", trigger: "recording",
+ taken_at: 100, edited_at: 0, has_edit: false, duration_s: 5,
+ };
+}
+const SCREENSHOT = { ...video("x"), id: "shot", mime: "image/png", path: "/shot.png" };
+beforeEach(async () => { await loadLocaleChunk("appshot_editor"); });
afterEach(() => { cleanup(); vi.unstubAllGlobals(); });
describe("AppShot recording controls", () => {
it("starts selection, keeps a stop control after the master is disabled, and offers the saved video", async () => {
+ const onSaved = vi.fn();
let phase = "idle";
const calls: string[] = [];
vi.stubGlobal("fetch", vi.fn(async (url: string, init?: RequestInit) => {
calls.push(`${init?.method ?? "GET"} ${url}`);
+ if (url === "/api/appshot/library") {
+ return response({ items: phase === "saved" ? [video("abc")] : [], max_entries: 500 });
+ }
if (url.endsWith("/start")) phase = "recording";
if (url.endsWith("/stop")) phase = "saved";
- return response({ phase, id: "abc", message: "", capability: ready });
+ return response({ phase, id: "abc", message: "", capability: ready,
+ recent: phase === "saved" ? [{ id: "abc", created_at: 100 }] : [] });
}));
- const { rerender } = render(<AppshotRecordingPanel settings={settings} saving={false} onShortcut={async () => {}} />);
+ const { rerender } = render(<AppshotRecordingPanel settings={settings} saving={false} onSaved={onSaved} />);
const button = await screen.findByTestId("appshots-recording-control");
await waitFor(() => expect(button.hasAttribute("disabled")).toBe(false));
fireEvent.click(button);
await screen.findByRole("button", { name: "Stop and save" });
- rerender(<AppshotRecordingPanel settings={{ ...settings, enabled: false }} saving={false} onShortcut={async () => {}} />);
+ rerender(<AppshotRecordingPanel settings={{ ...settings, enabled: false }} saving={false} onSaved={onSaved} />);
expect(button.hasAttribute("disabled")).toBe(false);
fireEvent.click(button);
- const link = await screen.findByRole("link", { name: "Download video" });
- expect(link.getAttribute("href")).toBe("/api/appshot/recording/abc/video");
+ const tile = await screen.findByTestId("appshot-library-tile");
+ expect(tile.getAttribute("data-media")).toBe("video");
+ await waitFor(() => expect(onSaved).toHaveBeenCalledExactlyOnceWith("abc"));
expect(calls.filter((call) => call.startsWith("POST"))).toEqual([
"POST /api/appshot/recording/start", "POST /api/appshot/recording/stop",
]);
@@ -40,10 +56,11 @@ describe("AppShot recording controls", () => {
available: false, detail: "Permission is required", permission_required: true,
} }));
vi.stubGlobal("fetch", fetcher);
- render(<AppshotRecordingPanel settings={settings} saving={false} onShortcut={async () => {}} />);
+ render(<AppshotRecordingPanel settings={settings} saving={false} />);
const allow = await screen.findByRole("button", { name: "Allow screen recording" });
expect(screen.getByTestId("appshots-recording-control").hasAttribute("disabled")).toBe(true);
- expect(fetcher.mock.calls).toHaveLength(1);
+ // Only reads so far (status and the video tiles); no permission request.
+ expect(fetcher.mock.calls.some((call) => String((call as unknown[])[0]).includes("/permissions/"))).toBe(false);
fireEvent.click(allow);
await waitFor(() => expect(fetcher).toHaveBeenCalledWith(
"/api/permissions/screen_recording/request?dry_run=false", {
@@ -57,7 +74,7 @@ describe("AppShot recording controls", () => {
vi.stubGlobal("fetch", vi.fn(async (url: string) => url.endsWith("/start")
? response({ detail: "No screen" }, 409)
: response({ phase: "idle", id: "", message: "", capability: ready })));
- render(<AppshotRecordingPanel settings={settings} saving={false} onShortcut={async () => {}} />);
+ render(<AppshotRecordingPanel settings={settings} saving={false} />);
const button = screen.getByTestId("appshots-recording-control");
await waitFor(() => expect(button.hasAttribute("disabled")).toBe(false));
fireEvent.click(button);
@@ -65,27 +82,17 @@ describe("AppShot recording controls", () => {
expect(screen.queryByRole("button", { name: "Stop and save" })).toBeNull();
});
- it("shows what to do while the recording shortcut is being captured", async () => {
- const onShortcut = vi.fn(async () => {});
- vi.stubGlobal("fetch", vi.fn(async () => response({ phase: "idle", id: "", message: "", capability: ready })));
- render(<AppshotRecordingPanel settings={{ ...settings, recording_hotkey: "" }} saving={false} onShortcut={onShortcut} />);
- fireEvent.click(await screen.findByTestId("appshots-recording-hotkey-change"));
- expect(await screen.findByTestId("appshots-recording-hotkey-status")).toHaveProperty(
- "textContent",
- expect.stringMatching(/Hold the keys/),
- );
- fireEvent.keyDown(window, { code: "ControlLeft", key: "Control", ctrlKey: true });
- fireEvent.keyDown(window, { code: "Digit9", key: "9", ctrlKey: true });
- fireEvent.keyUp(window, { code: "Digit9", key: "9", ctrlKey: false });
- await waitFor(() => expect(onShortcut).toHaveBeenCalledWith("ctrl+9"));
- });
-
- it("keeps older recordings downloadable after reopening the page", async () => {
- vi.stubGlobal("fetch", vi.fn(async () => response({ phase: "idle", id: "", message: "",
- capability: ready, recent: [{ id: "older", created_at: 100 }],
- })));
- render(<AppshotRecordingPanel settings={settings} saving={false} onShortcut={async () => {}} />);
- const link = await screen.findByRole("link");
- expect(link.getAttribute("href")).toBe("/api/appshot/recording/older/video");
+ it("shows kept recordings as playable tiles after reopening the page", async () => {
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => url === "/api/appshot/library"
+ ? response({ items: [video("older"), SCREENSHOT], max_entries: 500 })
+ : response({ phase: "idle", id: "", message: "", capability: ready,
+ recent: [{ id: "older", created_at: 100 }] })));
+ render(<AppshotRecordingPanel settings={settings} saving={false} />);
+ const tiles = await screen.findAllByTestId("appshot-library-tile");
+ expect(tiles).toHaveLength(1);
+ fireEvent.click(tiles[0].querySelector("button")!);
+ const player = await screen.findByTestId("appshot-library-player");
+ expect(player.getAttribute("src")).toContain("/library/recording_older/image");
+ expect(screen.getByRole("link", { name: "Download video" }).getAttribute("download")).toMatch(/\.mp4$/);
});
});
diff --git a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.tsx b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.tsx
index c548be463e..d431c6003b 100644
--- a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.tsx
+++ b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPanel.tsx
@@ -1,17 +1,21 @@
-import { useCallback, useEffect, useState } from "react";
-import { Download, Loader2, Square, Video } from "lucide-react";
+import { useEffect, useState } from "react";
+import { Loader2, Square, Video } from "lucide-react";
import { Button } from "@/components/ui/button";
import { useT } from "@/i18n";
+import { AppshotLibrary } from "@/views/AppshotLibrary";
import {
- appshotRecordingUrl, controlAppshotRecording, fetchAppshotRecording,
- requestRecordingPermission, type AppshotRecording, type AppshotSettings,
+ controlAppshotRecording, fetchAppshotRecording,
+ requestRecordingPermission, type AppshotRecording, type AppshotSettings, type AppshotSettingsPatch,
} from "@/lib/appshotApi";
-import { AppshotShortcutField } from "@/views/AppshotShortcutField";
+import { AppshotRecordingSettings } from "./AppshotRecordingSettings";
-export function AppshotRecordingPanel({ settings, saving, onShortcut }: {
+/** Start, stop, quality and the kept videos as playable tiles. Its shortcut
+ * lives with the other appshot shortcuts at the top of the page. */
+export function AppshotRecordingPanel({ settings, saving, onSaved, onSettings }: {
settings: AppshotSettings;
saving: boolean;
- onShortcut: (shortcut: string) => Promise<void>;
+ onSaved?: (id: string) => void;
+ onSettings?: (patch: AppshotSettingsPatch) => Promise<void>;
}) {
const t = useT();
const [recording, setRecording] = useState<AppshotRecording | null>(null);
@@ -19,10 +23,11 @@ export function AppshotRecordingPanel({ settings, saving, onShortcut }: {
const [error, setError] = useState("");
const [loadError, setLoadError] = useState("");
const [revision, setRevision] = useState(0);
- const [shortcutNote, setShortcutNote] = useState<string | null>(null);
- const onShortcutStatus = useCallback((text: string | null) => {
- setShortcutNote((current) => (current === text ? current : text));
- }, []);
+ const [settingsDirty, setSettingsDirty] = useState(false);
+
+ useEffect(() => {
+ if (recording?.phase === "saved" && recording.id) onSaved?.(recording.id);
+ }, [recording?.phase, recording?.id, onSaved]);
useEffect(() => {
const controller = new AbortController();
@@ -76,26 +81,14 @@ export function AppshotRecordingPanel({ settings, saving, onShortcut }: {
<p className="mt-1 text-sm text-muted-foreground">{t("appshots.recording_hint")}</p>
</div>
<Button type="button" variant={active ? "secondary" : "default"}
- disabled={busy || recording?.phase === "stopping" || (!active && (!settings.enabled || !recording?.capability?.available))}
+ disabled={busy || recording?.phase === "stopping" || (!active && (saving || settingsDirty || !settings.enabled || !recording?.capability?.available))}
onClick={() => void control()} data-testid="appshots-recording-control">
{busy || recording?.phase === "stopping" ? <Loader2 className="animate-spin" aria-hidden /> : active ? <Square aria-hidden /> : <Video aria-hidden />}
{active ? t(stop ? "appshots.recording_stop" : "appshots.recording_cancel") : t("appshots.recording_start")}
</Button>
</div>
- <div className="mt-4 flex flex-wrap items-center justify-between gap-3 border-t border-border pt-4">
- <p className="text-sm text-foreground">{t("appshots.recording_shortcut")}</p>
- <AppshotShortcutField value={settings.recording_hotkey ?? ""} disabled={saving || Boolean(active)}
- isMac={/Mac/i.test(navigator.platform || "")} testId="appshots-recording-hotkey"
- label={t("appshots.recording_shortcut")} onSave={onShortcut} onStatus={onShortcutStatus} />
- </div>
- {shortcutNote && (
- <p className="mt-2 text-sm text-muted-foreground" data-testid="appshots-recording-hotkey-status">
- {shortcutNote}
- </p>
- )}
- {settings.recording_shortcut?.detail && settings.recording_hotkey && !settings.recording_shortcut.armed && (
- <p className="mt-2 text-sm text-muted-foreground">{settings.recording_shortcut.detail}</p>
- )}
+ {onSettings && <AppshotRecordingSettings settings={settings} recording={recording}
+ disabled={saving || busy || Boolean(active)} onSave={onSettings} onDirty={setSettingsDirty} />}
{recording?.capability?.detail && <p className="mt-3 text-sm text-muted-foreground">{recording.capability.detail}</p>}
{recording?.capability?.permission_required && (
<Button type="button" variant="secondary" className="mt-2" disabled={busy} onClick={async () => {
@@ -109,27 +102,19 @@ export function AppshotRecordingPanel({ settings, saving, onShortcut }: {
{recording && !["idle", "error"].includes(recording.phase) ? t(`appshots.recording_${recording.phase}`) : ""}
{recording?.phase === "recording" || recording?.phase === "saved" ? ` · ${time}` : ""}
</p>
- {(error || loadError || recording?.message) && <p role="alert" className="mt-2 text-sm text-destructive">{error || loadError || recording?.message}</p>}
- {recording?.phase === "saved" && (
- <div className="mt-3">
- <video key={recording.id} src={appshotRecordingUrl(recording.id)} controls preload="metadata"
- className="max-h-80 w-full rounded-lg bg-secondary" aria-label={t("appshots.recording_title")} />
- <a href={appshotRecordingUrl(recording.id)} download className="mt-3 inline-flex items-center gap-2 text-sm text-foreground underline underline-offset-4">
- <Download className="h-4 w-4" aria-hidden />{t("appshots.recording_download")}
- </a>
- </div>
- )}
- {Boolean(recording?.recent?.length) && (
- <ul className="mt-4 space-y-2 border-t border-border pt-3">
- {recording?.recent?.filter((video) => recording.phase !== "saved" || video.id !== recording.id).map((video) => (
- <li key={video.id}>
- <a href={appshotRecordingUrl(video.id)} download className="text-sm text-foreground underline underline-offset-4">
- {t("appshots.recording_download")} · {new Date(video.created_at * 1000).toLocaleString()}
- </a>
- </li>
- ))}
- </ul>
+ {recording?.width && recording?.height && recording?.fps && (
+ <p className="mt-1 text-xs text-muted-foreground">
+ {recording.display_name ? `${recording.display_name} · ` : ""}
+ {recording.width} × {recording.height} · {recording.fps} FPS · {recording.bitrate_mbps} Mbps
+ </p>
)}
+ {Boolean(recording?.dropped_frames) && <p className="mt-1 text-xs text-muted-foreground">
+ {t("appshots.recording_dropped_hint")}
+ </p>}
+ {(error || loadError || recording?.message) && <p role="alert" className="mt-2 text-sm text-destructive">{error || loadError || recording?.message}</p>}
+ {/* A new or removed video changes the recorder's list; the tiles reload then. */}
+ <AppshotLibrary enabled={undefined} recordingsOnly
+ refreshKey={(recording?.recent ?? []).map((video) => video.id).join(",")} />
</section>
);
}
diff --git a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPlayer.tsx b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPlayer.tsx
new file mode 100644
index 0000000000..893e0347a6
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingPlayer.tsx
@@ -0,0 +1,41 @@
+import * as Dialog from "@radix-ui/react-dialog";
+import { Download, X } from "lucide-react";
+
+import { useT } from "@/i18n";
+import { appshotLibraryFileName, appshotLibraryImageUrl, type AppshotLibraryItem } from "@/lib/appshotApi";
+
+export function AppshotRecordingPlayer({ item, onClose }: {
+ item: AppshotLibraryItem | null;
+ onClose: () => void;
+}) {
+ const t = useT();
+ return (
+ <Dialog.Root open={item !== null} onOpenChange={(open) => { if (!open) onClose(); }}>
+ {item && (
+ <Dialog.Portal>
+ <Dialog.Overlay className="fixed inset-0 z-[90] bg-scrim/60" />
+ <Dialog.Content
+ className="fixed left-1/2 top-1/2 z-[91] flex max-h-[90vh] w-[min(90vw,64rem)] -translate-x-1/2 -translate-y-1/2 flex-col gap-4 rounded-xl border border-border bg-card p-5 text-foreground shadow-2xl"
+ aria-describedby={undefined}
+ >
+ <Dialog.Title className="pr-9 text-base font-medium">
+ {t("appshots.recording_title")}
+ </Dialog.Title>
+ <Dialog.Close className="absolute right-4 top-4 rounded-md p-1 text-muted-foreground hover:text-foreground focus-visible:ring-2 focus-visible:ring-border-strong"
+ aria-label={t("appshot_editor.close")}>
+ <X className="h-5 w-5" aria-hidden />
+ </Dialog.Close>
+ <video key={item.id} src={appshotLibraryImageUrl(item)} controls preload="metadata"
+ poster={appshotLibraryImageUrl(item, true)}
+ className="min-h-0 w-full flex-1 rounded-lg bg-secondary object-contain"
+ aria-label={t("appshots.recording_title")} data-testid="appshot-library-player" />
+ <a href={appshotLibraryImageUrl(item)} download={appshotLibraryFileName(item)}
+ className="inline-flex items-center gap-2 self-start text-sm underline underline-offset-4">
+ <Download className="h-4 w-4" aria-hidden />{t("appshots.recording_download")}
+ </a>
+ </Dialog.Content>
+ </Dialog.Portal>
+ )}
+ </Dialog.Root>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingSettings.test.tsx b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingSettings.test.tsx
new file mode 100644
index 0000000000..e1a3da9f89
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingSettings.test.tsx
@@ -0,0 +1,55 @@
+import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { AppshotRecordingPanel } from "./AppshotRecordingPanel";
+import type { AppshotSettings } from "@/lib/appshotApi";
+
+const settings = { enabled: true, recording_hotkey: "ctrl+shift+9" } as AppshotSettings;
+function setup(audio = true, phase = "idle") {
+ vi.stubGlobal("fetch", vi.fn(async () => ({ ok: true, status: 200, json: async () => ({
+ phase, id: "", message: "", capability: { available: true, permission_required: false,
+ detail: "", system_audio: { available: audio, detail: "" } },
+ }) })));
+ const save = vi.fn(async () => {});
+ render(<AppshotRecordingPanel settings={settings} saving={false}
+ onSettings={save} />);
+ return save;
+}
+afterEach(() => { cleanup(); vi.unstubAllGlobals(); });
+
+describe("Recording quality preferences", () => {
+ it("defaults to Full HD at 60 FPS and saves custom bitrate and audio together", async () => {
+ const save = setup();
+ const start = screen.getByTestId("appshots-recording-control");
+ await waitFor(() => expect(start.hasAttribute("disabled")).toBe(false));
+ expect(screen.getByLabelText("Resolution").textContent).toContain("1080p");
+ expect(screen.getByLabelText("Frame rate").textContent).toContain("60");
+ expect((screen.getByLabelText("Video bitrate (Mbps)") as HTMLInputElement).value).toBe("12");
+ fireEvent.click(screen.getByLabelText("Resolution"));
+ fireEvent.click(screen.getByRole("option", { name: /Native/i }));
+ fireEvent.click(screen.getByLabelText("Frame rate"));
+ fireEvent.click(screen.getByRole("option", { name: "120 FPS" }));
+ fireEvent.change(screen.getByLabelText("Video bitrate (Mbps)"), { target: { value: "37" } });
+ fireEvent.click(screen.getByLabelText("Include system audio"));
+ await waitFor(() => expect(start.hasAttribute("disabled")).toBe(true));
+ fireEvent.click(screen.getByRole("button", { name: "Save recording settings" }));
+ expect(save).toHaveBeenCalledWith({ recording_resolution: "native", recording_fps: 120,
+ recording_bitrate_mbps: 37, recording_system_audio: true });
+ });
+
+ it("does not submit invalid bitrates or enable unavailable system audio", async () => {
+ const save = setup(false);
+ await waitFor(() => expect(screen.getByTestId("appshots-recording-control").hasAttribute("disabled")).toBe(false));
+ expect(screen.getByLabelText("Include system audio").hasAttribute("disabled")).toBe(true);
+ fireEvent.change(screen.getByLabelText("Video bitrate (Mbps)"), { target: { value: "101" } });
+ expect(screen.getByRole("button", { name: "Save recording settings" }).hasAttribute("disabled")).toBe(true);
+ expect(save).not.toHaveBeenCalled();
+ });
+
+ it("locks preferences while capturing and keeps Stop available", async () => {
+ setup(true, "recording");
+ const stop = await screen.findByRole("button", { name: "Stop and save" });
+ expect(stop.hasAttribute("disabled")).toBe(false);
+ expect((screen.getByLabelText("Resolution").closest("fieldset") as HTMLFieldSetElement).disabled).toBe(true);
+ expect(screen.getByLabelText("Include system audio").hasAttribute("disabled")).toBe(true);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingSettings.tsx b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingSettings.tsx
new file mode 100644
index 0000000000..0f1a314af8
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/appshot/AppshotRecordingSettings.tsx
@@ -0,0 +1,85 @@
+import { useEffect, useId, useState } from "react";
+import { Button } from "@/components/ui/button";
+import { BrandedSelect } from "@/components/ui/select";
+import { useT } from "@/i18n";
+import type { AppshotRecording, AppshotSettings, AppshotSettingsPatch } from "@/lib/appshotApi";
+
+const fieldClass = "mt-1 h-10 w-full rounded-md border border-border bg-background px-3 text-sm text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-50";
+
+export function AppshotRecordingSettings({ settings, recording, disabled, onSave, onDirty }: {
+ settings: AppshotSettings;
+ recording: AppshotRecording | null;
+ disabled: boolean;
+ onSave: (patch: AppshotSettingsPatch) => Promise<void>;
+ onDirty: (dirty: boolean) => void;
+}) {
+ const t = useT();
+ const id = useId();
+ const resolution = settings.recording_resolution ?? "1080p";
+ const fps = settings.recording_fps ?? 60;
+ const bitrate = settings.recording_bitrate_mbps ?? 12;
+ const audio = settings.recording_system_audio ?? false;
+ const [draft, setDraft] = useState({ resolution, fps, bitrate: String(bitrate), audio });
+ useEffect(() => {
+ setDraft({ resolution, fps, bitrate: String(bitrate), audio });
+ }, [resolution, fps, bitrate, audio]);
+ const dirty = draft.resolution !== resolution || draft.fps !== fps
+ || draft.bitrate !== String(bitrate) || draft.audio !== audio;
+ useEffect(() => { onDirty(dirty); }, [dirty, onDirty]);
+ const validBitrate = /^\d+$/.test(draft.bitrate) && Number(draft.bitrate) >= 1 && Number(draft.bitrate) <= 100;
+ const audioCapability = recording?.capability?.system_audio;
+
+ return (
+ <form className="mt-4 space-y-3 border-t border-border pt-4" onSubmit={(event) => {
+ event.preventDefault();
+ if (!validBitrate || disabled || !dirty) return;
+ void onSave({ recording_resolution: draft.resolution, recording_fps: draft.fps,
+ recording_bitrate_mbps: Number(draft.bitrate), recording_system_audio: draft.audio });
+ }}>
+ <fieldset disabled={disabled} className="grid gap-3 sm:grid-cols-3">
+ <legend className="sr-only">{t("appshots.recording_quality")}</legend>
+ <label htmlFor={`${id}-resolution`} className="text-sm text-foreground">
+ {t("appshots.recording_resolution")}
+ <BrandedSelect id={`${id}-resolution`} className={fieldClass} value={draft.resolution}
+ ariaLabel={t("appshots.recording_resolution")} disabled={disabled}
+ onValueChange={(value) => setDraft({ ...draft, resolution: value as typeof resolution })}
+ options={[
+ { value: "720p", label: "720p · HD" },
+ { value: "1080p", label: "1080p · Full HD" },
+ { value: "1440p", label: "1440p · QHD" },
+ { value: "2160p", label: "2160p · 4K" },
+ { value: "native", label: t("appshots.recording_native") },
+ ]} />
+ </label>
+ <label htmlFor={`${id}-fps`} className="text-sm text-foreground">
+ {t("appshots.recording_fps")}
+ <BrandedSelect id={`${id}-fps`} className={fieldClass} value={String(draft.fps)}
+ ariaLabel={t("appshots.recording_fps")} disabled={disabled}
+ onValueChange={(value) => setDraft({ ...draft, fps: Number(value) as typeof fps })}
+ options={[30, 60, 120].map((value) => ({ value: String(value), label: `${value} FPS` }))} />
+ </label>
+ <label htmlFor={`${id}-bitrate`} className="text-sm text-foreground">
+ {t("appshots.recording_bitrate")}
+ <input id={`${id}-bitrate`} className={fieldClass} type="number" min={1} max={100} step={1}
+ value={draft.bitrate} aria-invalid={!validBitrate} aria-describedby={`${id}-bitrate-help`}
+ onChange={(event) => setDraft({ ...draft, bitrate: event.target.value })} />
+ </label>
+ </fieldset>
+ <p id={`${id}-bitrate-help`} className={`text-xs ${validBitrate ? "text-muted-foreground" : "text-destructive"}`}>
+ {t("appshots.recording_bitrate_hint")}
+ </p>
+ <label className="flex items-center gap-2 text-sm text-foreground">
+ <input type="checkbox" checked={draft.audio} className="h-4 w-4 accent-primary"
+ disabled={disabled || (!draft.audio && !audioCapability?.available)}
+ onChange={(event) => setDraft({ ...draft, audio: event.target.checked })} />
+ {t("appshots.recording_system_audio")}
+ </label>
+ <p className="text-xs text-muted-foreground">{t("appshots.recording_audio_hint")}</p>
+ {audioCapability?.detail && <p className="text-xs text-muted-foreground">{audioCapability.detail}</p>}
+ <p className="text-xs text-muted-foreground">{t("appshots.recording_monitor_hint")}</p>
+ {dirty && <Button type="submit" variant="secondary" disabled={disabled || !validBitrate}>
+ {t("appshots.recording_apply")}
+ </Button>}
+ </form>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/components/icons/sectionIcons.tsx b/jarvis/ui/web/frontend/src/components/icons/sectionIcons.tsx
index dbcbff3bd2..f8b46c77c1 100644
--- a/jarvis/ui/web/frontend/src/components/icons/sectionIcons.tsx
+++ b/jarvis/ui/web/frontend/src/components/icons/sectionIcons.tsx
@@ -332,17 +332,10 @@ export const SocialsIcon = defineIcon(
// ── System ────────────────────────────────────────────────────────────────
-/** Two stacked server units. */
+/** A cloud: the machines that live somewhere other than this one. */
export const ComputersIcon = defineIcon(
"ComputersIcon",
- <>
- <rect x="3.5" y="4" width="17" height="7" rx="2.25" />
- <rect x="3.5" y="13" width="17" height="7" rx="2.25" />
- <path d="M7.5 7.5h.01" />
- <path d="M7.5 16.5h.01" />
- <path d="M12.5 7.5h4" />
- <path d="M12.5 16.5h4" />
- </>,
+ <path d="M17.5 19H9a7 7 0 1 1 6.71-9h1.79a4.5 4.5 0 1 1 0 9Z" />,
);
/** A gear: the general settings. */
diff --git a/jarvis/ui/web/frontend/src/components/pets/petStage.ts b/jarvis/ui/web/frontend/src/components/pets/petStage.ts
new file mode 100644
index 0000000000..b989786a84
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/pets/petStage.ts
@@ -0,0 +1,10 @@
+import { createContext } from "react";
+import type { PetState } from "@/lib/petStates";
+
+/**
+ * The pet row a stage asks every pet face inside it to play. The agent
+ * chat's bot stage sets it per scene (searching sniffs, typing talks), so an
+ * agent that wears a pet acts out the same moment its shape would. Null
+ * outside a stage: the face picks its own row.
+ */
+export const PetStageContext = createContext<PetState | null>(null);
diff --git a/jarvis/ui/web/frontend/src/components/providers/ModelAccessSwitch.tsx b/jarvis/ui/web/frontend/src/components/providers/ModelAccessSwitch.tsx
new file mode 100644
index 0000000000..8b8437978a
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/providers/ModelAccessSwitch.tsx
@@ -0,0 +1,44 @@
+import { useT } from "@/i18n";
+import type { ModelAccess } from "@/lib/modelAccess";
+import { cn } from "@/lib/utils";
+
+/** A separate, labelled choice of how the selected provider is accessed. */
+export function ModelAccessSwitch({ options, value, onChange, disabled = false }: {
+ options: { kind: ModelAccess; disabled?: boolean }[];
+ value: ModelAccess;
+ onChange: (kind: ModelAccess) => void;
+ disabled?: boolean;
+}) {
+ const t = useT();
+ if (options.length < 2) return null;
+ return <div className="shrink-0 border-b border-border px-3 py-2" data-model-access-switch
+ onKeyDown={(event) => {
+ // The surrounding model list must not consume the radio group's keys.
+ if (event.key !== "Escape") event.stopPropagation();
+ }}>
+ <div className="mb-1.5 text-[11px] font-medium text-muted-foreground">{t("agent_chat.model_access")}</div>
+ <div role="radiogroup" aria-label={t("agent_chat.model_access")} className="flex gap-1 rounded-md bg-secondary/60 p-0.5">
+ {options.map((option) => <button key={option.kind} type="button" role="radio"
+ aria-checked={option.kind === value} disabled={disabled || option.disabled}
+ tabIndex={option.kind === value ? 0 : -1}
+ onClick={() => onChange(option.kind)}
+ onKeyDown={(event) => {
+ if (!["ArrowLeft", "ArrowRight", "ArrowUp", "ArrowDown", "Home", "End"].includes(event.key)) return;
+ event.preventDefault();
+ const available = options.filter((entry) => !entry.disabled);
+ const index = available.findIndex((entry) => entry.kind === option.kind);
+ const next = event.key === "Home" ? 0 : event.key === "End" ? available.length - 1
+ : (index + (["ArrowLeft", "ArrowUp"].includes(event.key) ? -1 : 1) + available.length) % available.length;
+ const choice = available[next];
+ if (choice) {
+ onChange(choice.kind);
+ event.currentTarget.parentElement?.querySelector<HTMLButtonElement>(`[data-access="${choice.kind}"]`)?.focus();
+ }
+ }} data-access={option.kind}
+ className={cn("min-w-0 flex-1 rounded px-2 py-1 text-xs transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-45",
+ option.kind === value ? "bg-popover font-medium text-popover-foreground shadow-sm" : "text-muted-foreground hover:text-foreground")}>
+ {t(`agent_chat.access_${option.kind}`)}
+ </button>)}
+ </div>
+ </div>;
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/AgentSwatch.test.tsx b/jarvis/ui/web/frontend/src/components/society/AgentSwatch.test.tsx
index 83741f473b..f7917046da 100644
--- a/jarvis/ui/web/frontend/src/components/society/AgentSwatch.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/AgentSwatch.test.tsx
@@ -60,6 +60,6 @@ describe("agent vector identities", () => {
const first = identities.map(symbolAppearance);
expect(identities.map(symbolAppearance)).toEqual(first);
expect(new Set(first.map(a => a.shape)).size).toBe(7);
- expect(new Set(first.map(a => a.color)).size).toBe(8);
+ expect(new Set(first.map(a => a.color)).size).toBe(11);
});
});
diff --git a/jarvis/ui/web/frontend/src/components/society/AgentSwatch.tsx b/jarvis/ui/web/frontend/src/components/society/AgentSwatch.tsx
index b1d3db4725..9bc9191868 100644
--- a/jarvis/ui/web/frontend/src/components/society/AgentSwatch.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/AgentSwatch.tsx
@@ -1,27 +1,47 @@
import { GigiAvatar } from "./GigiAvatar";
import { cn } from "@/lib/utils";
-import { AgentSymbol, SymbolThinkingDots } from "./AgentSymbol";
+import { SymbolThinkingDots } from "./AgentSymbol";
import { resolveCompanion } from "./companion/appearance";
+import { CompanionMark } from "./companion/CompanionMark";
import type { SocietyAgent } from "./data";
type SwatchAgent = Pick<SocietyAgent, "figure" | "palette" | "name"> &
Partial<Pick<SocietyAgent, "agentId" | "tier" | "state">>;
+/** The gold of the lead's mark (GigiAvatar's lens rim). */
+export const LEAD_COLOR = "#ffcd61";
+
+/**
+ * The one colour a person knows an agent by: the lead's gold, or the
+ * companion colour its symbol wears everywhere else.
+ */
+export function agentColor(agent: SwatchAgent): string {
+ if (isLead(agent)) return LEAD_COLOR;
+ return resolveCompanion(agent.agentId || agent.name, agent.figure?.companion).color;
+}
+
+function isLead(agent: SwatchAgent): boolean {
+ // Older internal-message participants carry the figure but not the tier.
+ return agent.tier === "lead" || (
+ !agent.tier && agent.figure?.archetype === "spirit" && agent.figure.base === "gigi"
+ );
+}
+
/** Lightweight vector identity shared by roster, profile and message surfaces. */
export function AgentSwatch({
agent,
size = 36,
className,
+ expressive = false,
}: {
agent: SwatchAgent;
size?: number;
className?: string;
+ /** Eyes a stage can animate between expressions (look, squint, wink, smile). */
+ expressive?: boolean;
}) {
- // Older internal-message participants carry the figure but not the tier.
- const isJarvis = agent.tier === "lead" || (
- !agent.tier && agent.figure?.archetype === "spirit" && agent.figure.base === "gigi"
- );
+ const isJarvis = isLead(agent);
// Real roster identities survive renames. Name-only historical participants
// still get a deterministic symbol without fetching a roster or a 3D model.
const appearance = resolveCompanion(agent.agentId || agent.name, agent.figure?.companion);
@@ -40,7 +60,7 @@ export function AgentSwatch({
{thinking && <svg aria-hidden viewBox="0 0 40 44" className="pointer-events-none absolute inset-0 h-full w-full"><SymbolThinkingDots color="#ffcf45" /></svg>}
</>
) : (
- <AgentSymbol {...appearance} size={size} thinking={thinking} />
+ <CompanionMark appearance={appearance} size={size} thinking={thinking} expressive={expressive} />
)}
</span>
);
diff --git a/jarvis/ui/web/frontend/src/components/society/AgentSymbol.tsx b/jarvis/ui/web/frontend/src/components/society/AgentSymbol.tsx
index 5781b404a9..1ee4215d99 100644
--- a/jarvis/ui/web/frontend/src/components/society/AgentSymbol.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/AgentSymbol.tsx
@@ -142,7 +142,21 @@ function AccessoryItemShapes({ item, shape, color, regions, uid }: { item: Acces
}
/** Plain ink eyes share one resting gaze; only a working agent moves. */
-export function AgentSymbol({ shape, color, size, eyes = "lines", thinking = false, accessories }: { shape: SymbolShape; color: string; size: number; eyes?: "dots" | "lines"; thinking?: boolean; accessories?: AccessoryChoice }) {
+/**
+ * One eye that can change its expression: the open ellipse it always has,
+ * plus a happy arc and a closed dash that stay hidden until a stage
+ * (components/agentchat/botStage) animates between them. Each eye is its own
+ * group, so the two can look, squint or wink independently.
+ */
+function ExpressiveEye({ side, cx, cy, rx, ry }: { side: "l" | "r"; cx: number; cy: number; rx: number; ry: number }) {
+ return <g className={`ae-eye ae-${side}`}>
+ <ellipse className="ae-open" cx={cx} cy={cy} rx={rx} ry={ry} />
+ <path className="ae-happy" d={`M${cx - 2.6} ${cy + 1}Q${cx} ${cy - 2.9} ${cx + 2.6} ${cy + 1}`} />
+ <path className="ae-shut" d={`M${cx - 2.4} ${cy}H${cx + 2.4}`} />
+ </g>;
+}
+
+export function AgentSymbol({ shape, color, size, eyes = "lines", thinking = false, accessories, expressive = false }: { shape: SymbolShape; color: string; size: number; eyes?: "dots" | "lines"; thinking?: boolean; accessories?: AccessoryChoice; expressive?: boolean }) {
const eyeY = shape === "cloud" || shape === "triangle" ? 23 : shape === "drop" ? 25 : 17.2;
const ink = companionEyeColors(color);
const maskId = `agent-body-${useId().replace(/[^a-zA-Z0-9_-]/g, "")}`;
@@ -152,7 +166,7 @@ export function AgentSymbol({ shape, color, size, eyes = "lines", thinking = fal
const faceTransform = `translate(2 -0.6) rotate(-14 20 ${eyeY})`;
const onFace = worn.filter(item => FACE_SLOTS.has(item.slot));
return (
- <svg aria-hidden focusable="false" data-agent-symbol={shape} data-thinking={thinking ? "true" : undefined} width={size} height={size} style={{ width: size, height: size, flexShrink: 0 }} viewBox={viewBox} className="society-agent-symbol block">
+ <svg aria-hidden focusable="false" data-agent-symbol={shape} data-expressive={expressive ? "true" : undefined} data-thinking={thinking ? "true" : undefined} width={size} height={size} style={{ width: size, height: size, flexShrink: 0 }} viewBox={viewBox} className="society-agent-symbol block">
<g className="agent-symbol-character">
{draw(worn.filter(item => item.slot === "back"))}
{worn.flatMap(item => item.parts.map((part, index) => {
@@ -175,8 +189,14 @@ export function AgentSymbol({ shape, color, size, eyes = "lines", thinking = fal
<g className="agent-symbol-gaze">
<g data-agent-eyes fill={ink.eye} transform={faceTransform}>
<g className="agent-symbol-lids">
- <ellipse cx={15.4} cy={eyeY} rx={eyes === "lines" ? 1.45 : 2} ry={eyes === "lines" ? 3.1 : 2.3} />
- <ellipse cx={24.6} cy={eyeY} rx={eyes === "lines" ? 1.45 : 2} ry={eyes === "lines" ? 3.1 : 2.3} />
+ {expressive ? <>
+ {/* A little larger than the resting eyes: at chat size the expression must read. */}
+ <ExpressiveEye side="l" cx={15.2} cy={eyeY} rx={eyes === "lines" ? 1.8 : 2.4} ry={eyes === "lines" ? 3.5 : 2.7} />
+ <ExpressiveEye side="r" cx={24.8} cy={eyeY} rx={eyes === "lines" ? 1.8 : 2.4} ry={eyes === "lines" ? 3.5 : 2.7} />
+ </> : <>
+ <ellipse cx={15.4} cy={eyeY} rx={eyes === "lines" ? 1.45 : 2} ry={eyes === "lines" ? 3.1 : 2.3} />
+ <ellipse cx={24.6} cy={eyeY} rx={eyes === "lines" ? 1.45 : 2} ry={eyes === "lines" ? 3.1 : 2.3} />
+ </>}
</g>
</g>
{onFace.length > 0 && <g transform={faceTransform}>{draw(onFace)}</g>}
diff --git a/jarvis/ui/web/frontend/src/components/society/RuntimeBadge.test.tsx b/jarvis/ui/web/frontend/src/components/society/RuntimeBadge.test.tsx
new file mode 100644
index 0000000000..9d84ab1f53
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/RuntimeBadge.test.tsx
@@ -0,0 +1,21 @@
+import { cleanup, render, screen } from "@testing-library/react";
+import { afterEach, describe, expect, test, vi } from "vitest";
+import { RuntimeBadge } from "./RuntimeBadge";
+
+vi.mock("@/i18n", () => ({ useT: () => (key: string) => (key === "society.runtime.runs_on" ? "Runs on {0}" : key) }));
+
+afterEach(cleanup);
+
+describe("RuntimeBadge", () => {
+ test.each(["hermes", "openclaw"] as const)("names the %s runtime with its logo", (runtime) => {
+ render(<RuntimeBadge runtime={runtime} />);
+ const badge = screen.getByRole("img", { name: `Runs on society.runtime.${runtime}` });
+ expect(badge.querySelector(`[data-testid="provider-logo-${runtime}"]`)).toBeTruthy();
+ });
+
+ test("an agent without a runtime runs on Jarvis", () => {
+ render(<RuntimeBadge runtime={undefined} />);
+ expect(screen.getByTestId("runtime-badge-jarvis")).toBeTruthy();
+ expect(screen.getByTestId("runtime-mark-jarvis")).toBeTruthy();
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/society/RuntimeBadge.tsx b/jarvis/ui/web/frontend/src/components/society/RuntimeBadge.tsx
new file mode 100644
index 0000000000..8078645144
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/RuntimeBadge.tsx
@@ -0,0 +1,38 @@
+/**
+ * The runtime an agent runs on (Jarvis, Hermes or OpenClaw), as a mark: inline
+ * beside a word (`RuntimeMark`) or as a small round badge pinned to the corner
+ * of an agent's face (`RuntimeBadge`), so a roster row says which loop runs it.
+ */
+import { ProviderLogo } from "@/components/providers/ProviderLogo";
+import { useT } from "@/i18n";
+import type { AgentRuntime } from "@/lib/societyApi";
+import { cn } from "@/lib/utils";
+
+export function RuntimeMark({ runtime, label }: { runtime: AgentRuntime; label: string }) {
+ // The built-in runtime carries the Jarvis app logo, not the user's pet:
+ // here it names a product beside Hermes and OpenClaw, not the companion.
+ return runtime === "jarvis"
+ ? <img src="/jarvis-gigi-256.png" alt="" aria-hidden="true" width={16} height={16} className="size-4 shrink-0 rounded-[4px]" data-testid="runtime-mark-jarvis" />
+ : <ProviderLogo providerId={runtime} label={label} size="sm" />;
+}
+
+/** The runtime's mark in a round chip; the parent positions it over the avatar. */
+export function RuntimeBadge({ runtime, className }: { runtime: AgentRuntime | undefined; className?: string }) {
+ const t = useT();
+ const resolved = runtime ?? "jarvis";
+ const label = t("society.runtime.runs_on").replace("{0}", t(`society.runtime.${resolved}`));
+ return (
+ <span
+ role="img"
+ aria-label={label}
+ title={label}
+ data-testid={`runtime-badge-${resolved}`}
+ className={cn(
+ "pointer-events-auto inline-flex h-[18px] w-[18px] items-center justify-center overflow-hidden rounded-full bg-secondary ring-2 ring-sidebar [&_img]:size-3.5",
+ className,
+ )}
+ >
+ <RuntimeMark runtime={resolved} label={t(`society.runtime.${resolved}`)} />
+ </span>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/agentSymbol.css b/jarvis/ui/web/frontend/src/components/society/agentSymbol.css
index 0596d8e921..964670a6fd 100644
--- a/jarvis/ui/web/frontend/src/components/society/agentSymbol.css
+++ b/jarvis/ui/web/frontend/src/components/society/agentSymbol.css
@@ -88,3 +88,19 @@
.society-agent-symbol .agent-acc-twinkle,
.society-agent-symbol .agent-acc-twinkle-late { opacity: 0; }
}
+
+/* Expressive eyes (AgentSymbol `expressive`): the open ellipse shows; the
+ happy arc and the closed dash wait for a stage to animate between them. */
+.society-agent-symbol .ae-eye,
+.society-agent-symbol .ae-open {
+ transform-box: fill-box;
+ transform-origin: center;
+}
+.society-agent-symbol .ae-happy,
+.society-agent-symbol .ae-shut {
+ fill: none;
+ stroke: #101014;
+ stroke-width: 1.5;
+ stroke-linecap: round;
+ opacity: 0;
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/card/AgentCardOverlay.tsx b/jarvis/ui/web/frontend/src/components/society/card/AgentCardOverlay.tsx
index 570c0b6017..afe11f206c 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/AgentCardOverlay.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/AgentCardOverlay.tsx
@@ -18,6 +18,7 @@ import { AgentChatPanel } from "../chat/AgentChatPanel";
import { RosterRail } from "../roster/RosterRail";
import type { SocietyChatGroup } from "@/lib/societyChatGroups";
import { OptionsRail } from "./OptionsRail";
+import { AgentChatLayout } from "./AgentChatLayout";
export interface AgentCardOverlayProps {
agent: SocietyAgent | null;
@@ -90,7 +91,7 @@ export function AgentCardOverlay({
</button>
</header>
)}
- <div className="grid min-h-0 flex-1 grid-cols-[minmax(240px,300px)_minmax(0,1fr)]">
+ <div className="agent-card-columns grid min-h-0 flex-1 grid-cols-[minmax(240px,300px)_minmax(0,1fr)]">
<RosterRail
agents={roster}
groups={groups}
@@ -111,29 +112,28 @@ export function AgentCardOverlay({
meet exactly at the rounded corner, and on a fractional grid
seam that joint rasterizes as a small step. The gray caption
above already separates by ground, so one border is enough. */}
- <div className="grid min-h-0 grid-cols-[minmax(0,1fr)_minmax(280px,320px)] overflow-hidden rounded-tl-[12px] border-l border-border bg-background">
+ <AgentChatLayout options={<OptionsRail agent={agent} onRetired={onClose} sample={sample} />}>
<section
- className="flex min-h-0 flex-col"
+ className="flex min-h-0 min-w-0 flex-col"
aria-label={t("society.card.chat")}
data-testid="agent-card-chat"
>
<AgentChatPanel key={agent.agentId} agent={agent} roster={roster} />
</section>
- <OptionsRail agent={agent} onRetired={onClose} sample={sample} />
- </div>
+ </AgentChatLayout>
</div>
</>
) : null}
</>
);
if (embedded) {
- return <div data-testid="agent-card" className="flex min-h-0 flex-1 flex-col overflow-hidden bg-card">{content}</div>;
+ return <div data-testid="agent-card" className="agent-card-workspace flex min-h-0 flex-1 flex-col overflow-hidden bg-card">{content}</div>;
}
return (
<Dialog.Root open={agent !== null} onOpenChange={(next) => (next ? undefined : onClose())}>
<Dialog.Portal>
<Dialog.Overlay className="fixed inset-0 z-40 bg-scrim/60 backdrop-blur-sm" />
- <Dialog.Content data-testid="agent-card" className="fixed inset-0 z-50 flex flex-col overflow-hidden bg-card focus:outline-none">
+ <Dialog.Content data-testid="agent-card" className="agent-card-workspace fixed inset-0 z-50 flex flex-col overflow-hidden bg-card focus:outline-none">
{content}
</Dialog.Content>
</Dialog.Portal>
diff --git a/jarvis/ui/web/frontend/src/components/society/card/AgentChatLayout.tsx b/jarvis/ui/web/frontend/src/components/society/card/AgentChatLayout.tsx
new file mode 100644
index 0000000000..6c2d18e1fa
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/card/AgentChatLayout.tsx
@@ -0,0 +1,40 @@
+import { useId, useState, type ReactNode } from "react";
+import { PanelRightClose, PanelRightOpen } from "lucide-react";
+
+import { useLocaleChunk, useT } from "@/i18n";
+import "./agentChatLayout.css";
+
+/** Keep both panes mounted: folding Options must not reset a browser or draft. */
+export function AgentChatLayout({ children, options }: { children: ReactNode; options: ReactNode }) {
+ const t = useT();
+ useLocaleChunk("society");
+ const [open, setOpen] = useState(true);
+ const panelId = useId();
+ const label = t(open ? "society.card.options_collapse" : "society.card.options_expand");
+ const Icon = open ? PanelRightClose : PanelRightOpen;
+
+ return (
+ <div className="agent-chat-layout flex min-h-0 min-w-0 flex-col overflow-hidden rounded-tl-[12px] border-l border-border bg-background">
+ <div className="flex shrink-0 justify-end border-b border-border px-2 py-1">
+ <button
+ type="button"
+ onClick={() => setOpen((value) => !value)}
+ title={label}
+ aria-label={label}
+ aria-expanded={open}
+ aria-controls={panelId}
+ className="inline-flex h-8 shrink-0 items-center justify-center gap-2 rounded-md px-2 text-sm font-medium text-muted-foreground hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring"
+ >
+ <Icon aria-hidden className="h-4 w-4" />
+ {t("society.card.options")}
+ </button>
+ </div>
+ <div className="agent-chat-layout-panes" data-options-open={open}>
+ {children}
+ <div id={panelId} hidden={!open} className="agent-chat-layout-options">
+ {options}
+ </div>
+ </div>
+ </div>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/card/AgentRoutinesList.test.tsx b/jarvis/ui/web/frontend/src/components/society/card/AgentRoutinesList.test.tsx
index f917063c4e..9365a93807 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/AgentRoutinesList.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/AgentRoutinesList.test.tsx
@@ -95,7 +95,7 @@ describe("AgentRoutinesList", () => {
let respond!: (response: Response) => void;
fetchMock.mockImplementation(() => new Promise<Response>((resolve) => { respond = resolve; }));
mount();
- expect(screen.getByText("Loading…")).toBeTruthy();
+ expect(await screen.findByText("Loading…")).toBeTruthy();
expect(screen.queryByText("No routines yet.")).toBeNull();
respond(json({ routines: [] }));
await screen.findByText("No routines yet.");
diff --git a/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.test.tsx b/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.test.tsx
index bd9a76a759..d9a6673628 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.test.tsx
@@ -7,6 +7,7 @@ import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/re
import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
import { OptionsRail } from "@/components/society/card/OptionsRail";
+import { AgentChatLayout } from "./AgentChatLayout";
import type { SocietyAgent } from "@/components/society/data";
function agent(over: Partial<SocietyAgent> = {}): SocietyAgent {
@@ -49,7 +50,9 @@ function mount(row: SocietyAgent, onRetired = vi.fn()) {
client,
...render(
<QueryClientProvider client={client}>
- <OptionsRail agent={row} onRetired={onRetired} />
+ <AgentChatLayout options={<OptionsRail agent={row} onRetired={onRetired} />}>
+ <section aria-label="Chat"><textarea aria-label="Chat draft" /></section>
+ </AgentChatLayout>
</QueryClientProvider>,
),
};
@@ -69,6 +72,10 @@ describe("OptionsRail", () => {
beforeEach(() => {
fetchMock = vi.fn(async (url: string) => {
const path = String(url);
+ if (path === "/api/tasks/gm1") {
+ return json({ id: "gm1", title: "Inbox sweep", state: "scheduled", steps: [],
+ spec: { action: { kind: "agent", prompt: "Read the inbox." } } });
+ }
if (path.includes("/browser/status")) {
return json({ installed: false, phase: "idle", percent: 0, running: false });
}
@@ -131,6 +138,51 @@ describe("OptionsRail", () => {
expect(fetchMock.mock.calls.filter((c) => (c[1] as RequestInit | undefined)?.method === "DELETE")).toHaveLength(0);
});
+ test("collapses the entire rail with a reachable toggle and preserves browser and routine drafts", async () => {
+ mount(agent());
+ await screen.findByText("Inbox sweep");
+ const toggle = await screen.findByRole("button", { name: "Collapse Options" });
+ const panel = document.getElementById(toggle.getAttribute("aria-controls")!)!;
+ const browser = screen.getByTestId("agent-browser-preview");
+ const canvas = browser.querySelector("canvas");
+ fireEvent.click(screen.getByTestId("agent-routines-add"));
+ const title = screen.getByRole("textbox", { name: "Title" }) as HTMLInputElement;
+ fireEvent.change(title, { target: { value: "Unsaved routine" } });
+ const chat = screen.getByRole("textbox", { name: "Chat draft" }) as HTMLTextAreaElement;
+ fireEvent.change(chat, { target: { value: "Unsent message" } });
+
+ toggle.focus();
+ fireEvent.click(toggle);
+ expect(toggle.getAttribute("aria-expanded")).toBe("false");
+ expect(panel.hidden).toBe(true);
+ expect(screen.queryByRole("complementary", { name: "Options" })).toBeNull();
+ expect(document.activeElement).toBe(toggle);
+ expect(screen.getByRole("textbox", { name: "Chat draft" })).toBe(chat);
+ expect(chat.value).toBe("Unsent message");
+
+ fireEvent.click(screen.getByRole("button", { name: "Expand Options" }));
+ expect(panel.hidden).toBe(false);
+ expect(toggle.getAttribute("aria-expanded")).toBe("true");
+ expect(screen.getByTestId("agent-browser-preview")).toBe(browser);
+ expect(browser.querySelector("canvas")).toBe(canvas);
+ expect(screen.getByRole("textbox", { name: "Title" })).toBe(title);
+ expect(title.value).toBe("Unsaved routine");
+ expect(fetchMock.mock.calls.some(([, init]) => {
+ const method = (init as RequestInit | undefined)?.method ?? "GET";
+ return method !== "GET";
+ })).toBe(false);
+ });
+
+ test("keeps the selected routine detail open when Options is folded", async () => {
+ mount(agent());
+ fireEvent.click(await screen.findByText("Inbox sweep"));
+ const detail = await screen.findByTestId("agent-routine-detail");
+ const toggle = await screen.findByRole("button", { name: "Collapse Options" });
+ fireEvent.click(toggle);
+ fireEvent.click(screen.getByRole("button", { name: "Expand Options" }));
+ expect(screen.getByTestId("agent-routine-detail")).toBe(detail);
+ });
+
test("switching agent swaps the caption and the routines list", async () => {
const { rerender, client } = mount(agent());
await waitFor(() => expect(screen.getByTestId("agent-routines").textContent).toContain("Inbox sweep"));
diff --git a/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.tsx b/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.tsx
index 95d24056be..18cd261e76 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/OptionsRail.tsx
@@ -87,7 +87,7 @@ export function OptionsRail({ agent, onRetired, sample = false }: OptionsRailPro
) : null}
</div>
</header>
- <div className="flex min-h-0 flex-1 flex-col gap-3 overflow-hidden px-3 pb-3 pt-2">
+ <div className="flex min-h-0 flex-1 flex-col gap-3 overflow-y-auto px-3 pb-3 pt-2">
<div className="flex shrink-0 gap-2">
<button type="button" onClick={() => setAppearanceOpen(true)} data-testid="edit-agent-appearance" className="min-w-0 flex-1 rounded-lg border border-border px-3 py-2 text-left text-sm font-medium text-foreground hover:bg-secondary">{t("society.companion.appearance")}</button>
{/* The lead is every install's own Jarvis and sample rows are not real agents: neither is shareable. */}
@@ -108,7 +108,7 @@ export function OptionsRail({ agent, onRetired, sample = false }: OptionsRailPro
onDetailOpenChange={setRoutineOpen}
sampleRoutines={sample ? agent.routines : undefined}
variant="rail"
- className="min-h-0 flex-1"
+ className="min-h-48 flex-1"
/>
</div>
{appearanceOpen && <Suspense fallback={null}><AgentAppearanceDialog key={agent.agentId} agent={agent} sample={sample} onClose={() => setAppearanceOpen(false)} /></Suspense>}
diff --git a/jarvis/ui/web/frontend/src/components/society/card/RoutineChat.tsx b/jarvis/ui/web/frontend/src/components/society/card/RoutineChat.tsx
index 94118b1bd5..262618f143 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/RoutineChat.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/RoutineChat.tsx
@@ -2,7 +2,7 @@
* The selected routine run or agent conversation occupies the agent's chat
* lane and loads only on navigation.
*/
-import { useEffect, useMemo, useState } from "react";
+import { useEffect, useMemo, useState, type ReactNode } from "react";
import { ArrowLeft, Send, Square } from "lucide-react";
import { AgentTimeline } from "@/components/agentchat/AgentTimeline";
import { ChatMarkdown } from "@/components/agentchat/ChatMarkdown";
@@ -18,7 +18,11 @@ function displayItems(items: TimelineItem[]): TimelineItem[] {
return items.map((item) => item.type === "user" ? { ...item, text: routineTask(item.text) ?? item.text } : item);
}
-export default function RoutineChat({ target, onClose }: { target: RoutineChatTarget; onClose: () => void }) {
+export default function RoutineChat({ target, onClose, face }: {
+ target: RoutineChatTarget; onClose: () => void;
+ /** The agent's face for the live line of a running trace. */
+ face?: ReactNode;
+}) {
const t = useT();
return <section className="flex h-full min-h-0 flex-1 flex-col bg-background p-4 text-foreground sm:p-6" data-testid="routine-chat" data-session-id={target.sessionId}>
<header className="flex shrink-0 flex-wrap items-center gap-x-4 gap-y-2 border-b border-border pb-3">
@@ -28,7 +32,7 @@ export default function RoutineChat({ target, onClose }: { target: RoutineChatTa
</div>
<span className="text-[11px] text-muted-foreground">{t(target.kind === "conversation" ? "society.conversations.badge" : "society.routine_detail.background_chat")}</span>
</header>
- {target.legacy ? <HistoricalExecution target={target} /> : <LiveExecution sessionId={target.sessionId} conversation={target.kind === "conversation"} />}
+ {target.legacy ? <HistoricalExecution target={target} /> : <LiveExecution sessionId={target.sessionId} conversation={target.kind === "conversation"} face={face} />}
</section>;
}
@@ -65,7 +69,7 @@ function HistoricalExecution({ target }: { target: RoutineChatTarget }) {
</>;
}
-function LiveExecution({ sessionId, conversation = false }: { sessionId: string; conversation?: boolean }) {
+function LiveExecution({ sessionId, conversation = false, face }: { sessionId: string; conversation?: boolean; face?: ReactNode }) {
const t = useT();
const label = (key: string) => t(`society.routine_detail.${key}`);
const followUp = conversation ? t("society.conversations.follow_up") : label("follow_up");
@@ -87,7 +91,7 @@ function LiveExecution({ sessionId, conversation = false }: { sessionId: string;
return <AgentChatStoreProvider store={store}>
<div ref={scroll.rootRef} className="min-h-0 flex-1 overflow-y-auto py-4"><div ref={scroll.contentRef} className="space-y-3">
{!ready && !error && <p role="status" className="text-sm">{t("tasks_view.loading_details")}</p>}
- <AgentTimeline items={visibleItems} assistantName={session?.title.split(" · ")[0] ?? ""} providerLabel={(id) => id} onDecide={store.getState().decide} />
+ <AgentTimeline items={visibleItems} assistantName={session?.title.split(" · ")[0] ?? ""} providerLabel={(id) => id} onDecide={store.getState().decide} traceFace={face} />
</div></div>
{error && <p role="alert" className="text-sm text-destructive">{error}</p>}
<form className="mt-3 flex shrink-0 items-end gap-2" onSubmit={(event) => {
diff --git a/jarvis/ui/web/frontend/src/components/society/card/RuntimePicker.tsx b/jarvis/ui/web/frontend/src/components/society/card/RuntimePicker.tsx
index dfca4e7981..d6f100f2a2 100644
--- a/jarvis/ui/web/frontend/src/components/society/card/RuntimePicker.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/card/RuntimePicker.tsx
@@ -10,7 +10,7 @@
import { useCallback, useEffect, useId, useRef, useState, type KeyboardEvent } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { Loader2 } from "lucide-react";
-import { ProviderLogo } from "@/components/providers/ProviderLogo";
+import { RuntimeMark } from "@/components/society/RuntimeBadge";
import { useT } from "@/i18n";
import {
ensureAgentRuntime,
@@ -52,14 +52,6 @@ function setupState(status: AgentRuntimeStatus | undefined): SetupState {
return status.job?.state === "failed" ? "failed" : "pending";
}
-function RuntimeMark({ runtime, label }: { runtime: AgentRuntime; label: string }) {
- // The built-in runtime carries the Jarvis app logo, not the user's pet:
- // here it names a product beside Hermes and OpenClaw, not the companion.
- return runtime === "jarvis"
- ? <img src="/jarvis-gigi-256.png" alt="" aria-hidden="true" width={16} height={16} className="size-4 shrink-0 rounded-[4px]" data-testid="runtime-mark-jarvis" />
- : <ProviderLogo providerId={runtime} label={label} size="sm" />;
-}
-
/** Ask the backend to set a runtime up; refreshes the status either way. */
function useEnsureRuntime() {
const client = useQueryClient();
diff --git a/jarvis/ui/web/frontend/src/components/society/card/agentChatLayout.css b/jarvis/ui/web/frontend/src/components/society/card/agentChatLayout.css
new file mode 100644
index 0000000000..f078169f8f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/card/agentChatLayout.css
@@ -0,0 +1,51 @@
+.agent-card-workspace {
+ container: agent-card / inline-size;
+}
+
+@container agent-card (max-width: 760px) {
+ .agent-card-columns {
+ grid-template-columns: minmax(160px, 200px) minmax(0, 1fr);
+ }
+}
+
+@container agent-card (max-width: 480px) {
+ .agent-card-columns {
+ grid-template-columns: minmax(0, 1fr);
+ grid-template-rows: minmax(0, 140px) minmax(0, 1fr);
+ }
+}
+
+.agent-chat-layout {
+ container: agent-chat-layout / inline-size;
+}
+
+.agent-chat-layout-panes {
+ display: grid;
+ flex: 1;
+ min-width: 0;
+ min-height: 0;
+ grid-template-columns: minmax(0, 1fr);
+ grid-template-rows: minmax(0, 1fr);
+}
+
+.agent-chat-layout-panes[data-options-open="true"] {
+ grid-template-rows: minmax(0, 1fr) minmax(0, 1fr);
+}
+
+.agent-chat-layout-options {
+ min-width: 0;
+ min-height: 0;
+ border-top: 1px solid hsl(var(--border));
+}
+
+/* Use the available pane width, including when the app navigation is open. */
+@container agent-chat-layout (min-width: 680px) {
+ .agent-chat-layout-panes[data-options-open="true"] {
+ grid-template-columns: minmax(0, 1fr) minmax(280px, 320px);
+ grid-template-rows: minmax(0, 1fr);
+ }
+
+ .agent-chat-layout-options {
+ border-top: 0;
+ }
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.switch.test.tsx b/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.switch.test.tsx
index 65b50f4131..8dd24382e0 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.switch.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.switch.test.tsx
@@ -23,6 +23,7 @@ import * as cardData from "../cardData";
const attachmentState = vi.hoisted(() => ({
current: [] as ChatAttachment[],
clear: vi.fn(),
+ restore: vi.fn(),
}));
vi.mock("@/i18n", () => ({ useT: () => (key: string) => key, fill: (text: string) => text }));
@@ -43,12 +44,18 @@ vi.mock("@/components/agentchat/DictationButton", () => ({ DictationButton: () =
vi.mock("@/components/agentchat/useChatAttachments", async (importOriginal) => ({
// The pure helpers (attachmentMedia, ...) stay real; only the hook is scripted.
...(await importOriginal<typeof import("@/components/agentchat/useChatAttachments")>()),
+ restoreHeldFiles: (_owner: object, _key: string, files: { attachments: ChatAttachment[] }) => {
+ attachmentState.current = files.attachments;
+ attachmentState.restore(files);
+ },
useChatAttachments: () => ({
attachments: attachmentState.current,
analyzing: 0,
dragging: false,
dragHandlers: {},
clear: attachmentState.clear,
+ take: () => { const attachments = attachmentState.current; attachmentState.clear(); return { attachments, previews: {} }; },
+ restore: (files: { attachments: ChatAttachment[] }) => { attachmentState.current = files.attachments; attachmentState.restore(files); },
remove() {},
attachFiles() {},
}),
@@ -119,6 +126,7 @@ const gmail = agent({ agentId: "gmail-agent", name: "Gmail Agent" });
beforeEach(() => {
attachmentState.current = [];
attachmentState.clear.mockClear();
+ attachmentState.restore.mockClear();
useRoutineNavigation.getState().close();
seq = 0;
useTranscriptViewStore.setState({ boundaries: {} });
@@ -279,7 +287,7 @@ it.each(["specialist", "lead"] as const)("/clear empties only the %s view and ke
expect(screen.queryByText("PREVIOUS_CONTEXT_TO_KEEP")).toBeNull();
});
-it.each(["specialist", "lead"] as const)("turns send into stop while the %s is reasoning, then back when the turn ends", async (tier) => {
+it.each(["specialist", "lead"] as const)("keeps follow-up sending available while the %s is reasoning", async (tier) => {
const current = tier === "lead" ? agent({ agentId: "jarvis", name: "Jarvis", tier }) : gmail;
const store = tier === "lead" ? useAgentChatStore : useSocietyChatStore;
store.getState().disconnect();
@@ -287,9 +295,9 @@ it.each(["specialist", "lead"] as const)("turns send into stop while the %s is r
store.setState({ activeSessionId: current.chatSessionId, timeline: reasoningTurn(), busy: false });
setJarvisCardMode("chat");
render(<AgentChatPanel agent={current} roster={[current]} />);
- expect(screen.getByTestId("composer-stop")).toBeTruthy();
- expect(screen.queryByTestId("composer-send")).toBeNull();
- fireEvent.click(screen.getByTestId("composer-stop"));
+ expect(screen.getByRole("button", { name: "society.chat.stop" })).toBeTruthy();
+ expect(screen.getByTestId("composer-send")).toBeTruthy();
+ fireEvent.click(screen.getByRole("button", { name: "society.chat.stop" }));
await waitFor(() => expect(vi.mocked(fetch).mock.calls.some(([url, options]) =>
String(url).includes("/cancel") && options?.method === "POST",
)).toBe(true));
@@ -299,7 +307,7 @@ it.each(["specialist", "lead"] as const)("turns send into stop while the %s is r
]),
}));
expect(screen.getByTestId("composer-send")).toBeTruthy();
- expect(screen.queryByTestId("composer-stop")).toBeNull();
+ expect(screen.queryByTestId("composer-cancel")).toBeNull();
});
it("routine-check opens the clicked execution instead of folding its instruction", () => {
@@ -318,8 +326,8 @@ it("shows stop while the send request is in flight, before the turn stream start
useSocietyChatStore.getState().openSession(gmail.chatSessionId!);
useSocietyChatStore.setState({ activeSessionId: gmail.chatSessionId, timeline: EMPTY_TIMELINE, busy: true });
render(<AgentChatPanel agent={gmail} roster={[gmail]} />);
- expect(screen.getByTestId("composer-stop")).toBeTruthy();
- expect(screen.queryByTestId("composer-send")).toBeNull();
+ expect(screen.getByRole("button", { name: "society.chat.stop" })).toBeTruthy();
+ expect(screen.getByTestId("composer-send")).toBeTruthy();
});
const screenshot: ChatAttachment = {
@@ -360,7 +368,7 @@ it("retains the exact draft and attachment when the message request fails", asyn
fireEvent.click(screen.getByTestId("composer-send"));
await waitFor(() => expect(useSocietyChatStore.getState().lastError).toBeTruthy());
await waitFor(() => expect(screen.getByTestId("composer-chip-field").textContent).toBe(" Please inspect this "));
- expect(attachmentState.clear).not.toHaveBeenCalled();
+ expect(attachmentState.restore).toHaveBeenCalledWith({ attachments: [screenshot], previews: {} });
expect(screen.getByText("screen.png")).toBeTruthy();
});
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.tsx b/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.tsx
index b87e7ee327..d6cc1e6a10 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/AgentChatPanel.tsx
@@ -8,7 +8,7 @@ import {
assignmentOf,
routineTask,
} from "./ChatActivity";
-import { MemoryUpdateNotice } from "./MemoryUpdateNotice";
+import { MemoryNoticeViewer, MemoryUpdateNotice, memoryNoticePath, memoryNoticeTitle } from "./MemoryUpdateNotice";
import { foldMemoryNotices } from "./memoryNotices";
import { foldRepeatedThreadStatus } from "@/components/agentic/threads/openCodingThread";
import { mergeOutgoingMessages, useOutgoingMessages } from "@/components/agentchat/useOutgoingMessages";
@@ -45,10 +45,11 @@ import { AgentChatStoreProvider, useAgentChat, useAgentChatApi } from "@/compone
import { ChatAttachmentStrip } from "@/components/agentchat/ChatAttachmentStrip";
import { ScrollToEndButton } from "@/components/ui/scroll-to-end-button";
import { useStickToBottom } from "@/hooks/useStickToBottom";
-import { ComposerChipField, type ComposerChipFieldHandle } from "@/components/agentchat/ComposerChipField";
+import { ComposerChipField, type ComposerChipFieldHandle, type ComposerDraft } from "@/components/agentchat/ComposerChipField";
+import { captureComposerDraftTarget, readComposerDraft, useComposerDraft, writeComposerDraft } from "@/components/agentchat/composerDrafts";
import { MessageWithChips } from "@/components/agentchat/ToolChoiceChips";
import { choiceToken } from "@/components/agentchat/composerChips";
-import { useChatAttachments } from "@/components/agentchat/useChatAttachments";
+import { releaseHeldFiles, restoreHeldFiles, useChatAttachments } from "@/components/agentchat/useChatAttachments";
import { DictationButton } from "@/components/agentchat/DictationButton";
import { useComposerDictation } from "@/components/agentchat/useComposerDictation";
import { useEventStore } from "@/store/events";
@@ -60,8 +61,7 @@ import {
type TurnItem,
type UserItem,
} from "@/components/agentchat/reduce";
-import { ThreadTurn } from "@/components/agentic/threads/ThreadTimeline";
-import { formatTokens, outputTokens } from "@/components/agentchat/toolView";
+import { AwaitingTurn, MessengerTurn, type MemoryLink } from "@/components/agentchat/MessengerTurn";
import { VoiceStage } from "@/components/home/VoiceStage";
import { ProviderLogo } from "@/components/providers/ProviderLogo";
import { useT } from "@/i18n";
@@ -71,7 +71,8 @@ import { createAgentChatStore, useAgentChatStore, type AgentChatStoreHook } from
import type { AgentChatSurface, ApprovalDecision } from "@/lib/agentChatApi";
import { offeredModels, useSavedHiddenModels } from "@/lib/agentProviderPrefs";
-import { AgentSwatch } from "../AgentSwatch";
+import { AgentSwatch, agentColor } from "../AgentSwatch";
+import { bubbleTint, type BubbleTint } from "@/lib/bubbleTint";
import {
useResolveProposal,
useSocietyCapabilities,
@@ -702,11 +703,16 @@ export function Transcript({
// watches the content's own size too, so growth follows; scrolled up, the
// reader keeps their place and gets a button back.
const { rootRef, contentRef, atEnd, jumpToEnd, follow } = useStickToBottom();
+ // The person's bubbles wear the colour of the agent they talk to.
+ const tint = useMemo(() => bubbleTint(agentColor(agent)), [agent]);
// `items` itself, not its length: a reasoning trace or tool row grows
// the same turn in place, so the length does not change. Pin in this
// layout pass — waiting for ResizeObserver is one frame too late, and
// that frame is when overflow anchoring would unstick the view.
useLayoutEffect(follow, [follow, items]);
+ // A message of the person's with no turn after it yet: the agent is reading it.
+ const lastItem = items[items.length - 1];
+ const awaiting = lastItem?.type === "user" && lastItem.origin !== "control" && !assignmentOf(lastItem.text) ? lastItem : null;
if (items.length === 0) {
return (
@@ -735,9 +741,9 @@ export function Transcript({
) : item.type === "user" && assignmentOf(item.text) ? (
<DelegationActivity {...assignmentOf(item.text)!} roster={roster} />
) : item.type === "user" ? (
- <UserBubble item={item} agentId={agent.agentId} sessionId={sessionId ?? agent.chatSessionId ?? undefined} />
+ <UserBubble item={item} agentId={agent.agentId} sessionId={sessionId ?? agent.chatSessionId ?? undefined} tint={tint} />
) : item.type === "turn" ? (
- <TurnBubble item={item} memory={memoryByTurn.get(item.id)} onDecide={onDecide} />
+ <TurnBubble item={item} agent={agent} memory={memoryByTurn.get(item.id)} onDecide={onDecide} />
) : item.type === "notice" ? (
item.kind === "proposal" ? (
<ProposalCard item={item} />
@@ -750,6 +756,8 @@ export function Transcript({
</div>
);
})}
+ {awaiting ? <AwaitingTurn since={awaiting.tsMs} seed={awaiting.id}
+ avatar={<AgentSwatch agent={{ ...agent, state: undefined }} size={28} expressive />} color={agentColor(agent)} /> : null}
</div>
</div>
{!atEnd && <ScrollToEndButton onClick={jumpToEnd} testId="society-scroll-end" className="top-auto bottom-3" />}
@@ -1014,7 +1022,12 @@ function visibleUserText(text: string): string {
.trimEnd();
}
-export function UserBubble({ item, agentId, sessionId }: { item: UserItem; agentId?: string; sessionId?: string }) {
+/**
+ * The person's message: a bubble on the right in the agent's colour, with ink
+ * chosen for that colour (`bubbleTint`). Without an agent colour (a meeting
+ * of several agents) it wears the app's accent.
+ */
+export function UserBubble({ item, agentId, sessionId, tint }: { item: UserItem; agentId?: string; sessionId?: string; tint?: BubbleTint | null }) {
const t = useT();
const choices = messageChoices(item);
const text = visibleUserText(item.text);
@@ -1027,7 +1040,11 @@ export function UserBubble({ item, agentId, sessionId }: { item: UserItem; agent
}) : undefined} />;
return (
<div className="flex min-w-0 max-w-[min(85%,42rem)] flex-col items-end gap-1 self-end">
- <div className="min-w-0 rounded-2xl rounded-br-md bg-secondary px-4 py-2.5 text-sm leading-relaxed text-foreground [overflow-wrap:anywhere]">
+ <div
+ data-testid="user-bubble"
+ className={cn("jarvis-chat-tinted min-w-0 rounded-[20px] px-4 py-2.5 text-base leading-6 [overflow-wrap:anywhere]", !tint && "jarvis-chat-out")}
+ style={tint ? { background: tint.background, color: tint.color } : undefined}
+ >
<MessageWithChips text={text} choices={choices} />
</div>
{item.attachments.length > 0 ? (
@@ -1045,27 +1062,33 @@ export function UserBubble({ item, agentId, sessionId }: { item: UserItem; agent
function TurnBubble({
item,
+ agent,
memory,
onDecide,
}: {
item: TurnItem;
+ agent: SocietyAgent;
memory?: NoticeItem[];
onDecide: (approvalId: string, decision: ApprovalDecision) => Promise<void>;
}) {
const t = useT();
- const extras = useMemo(
- () => memory?.map((notice) => ({ key: notice.id, node: <MemoryUpdateNotice item={notice} inTrace /> })),
- [memory],
+ const [openId, setOpenId] = useState<string | null>(null);
+ const memoryLinks = useMemo<MemoryLink[] | undefined>(
+ () => memory?.flatMap((notice) => {
+ const path = memoryNoticePath(notice);
+ return path ? [{ key: notice.id, title: memoryNoticeTitle(t, path), onOpen: () => setOpenId(notice.id) }] : [];
+ }),
+ [memory, t],
);
- // What the turn spent: output tokens only (BUG-173), and the cost when billed.
- const tokens = outputTokens(item.usage ?? item.liveUsage);
- const spent = [
- tokens !== null && tokens > 0 ? `${formatTokens(tokens)} ${t("agent_chat.tokens")}` : "",
- item.costUsd !== null && item.costUsd > 0 ? `$${item.costUsd.toFixed(4)}` : "",
- ].filter(Boolean).join(" · ");
- // An agent's turn reads exactly like a coding thread's turn in the Agentic
- // IDE; only its questions and approvals are answered in place.
- return <ThreadTurn turn={item} prompts="inline" onDecide={onDecide} extras={extras} receipt={spent || undefined} />;
+ const opened = openId ? memory?.find((notice) => notice.id === openId) : undefined;
+ // An agent's turn reads like a messenger: its messages, what it changed,
+ // and while it works its face with a word for what it is doing.
+ // The face stays a face: the effect beside it says what it is doing, so
+ // the swatch's own "working" dots would only repeat it.
+ return <>
+ <MessengerTurn turn={item} avatar={<AgentSwatch agent={{ ...agent, state: undefined }} size={28} expressive />} color={agentColor(agent)} onDecide={onDecide} memoryLinks={memoryLinks} />
+ {opened && <MemoryNoticeViewer item={opened} onClose={() => setOpenId(null)} />}
+ </>;
}
/**
@@ -1120,19 +1143,27 @@ export function Composer({ agent, mentionable, busy, sessionId, cwd, provider, s
const t = useT();
const chatStore = useAgentChatApi();
const [modelSaving, setModelSaving] = useState(false);
- const [value, setValue] = useState("");
+ const messageDraft = useComposerDraft(chatStore, sessionId);
+ const value = messageDraft.text;
+ const setValue = (text: string) => writeComposerDraft(chatStore, sessionId, {
+ ...readComposerDraft(chatStore, sessionId), text,
+ });
const [plusOpen, setPlusOpen] = useState(false);
const [problem, setProblem] = useState<string | null>(null);
const [mention, setMention] = useState<{ query: string; start: number } | null>(null);
const [activeIndex, setActiveIndex] = useState(0);
- const [selectedTools, setSelectedTools] = useState<MentionItem[]>([]);
- useEffect(() => { setSelectedTools([]); }, [sessionId, agent.agentId]);
const fieldRef = useRef<ComposerChipFieldHandle>(null);
const composerRef = useRef<HTMLDivElement>(null);
const fileInput = useRef<HTMLInputElement>(null);
- const attachments = useChatAttachments({ sessionId, cwd, provider, surface }, (message) => setProblem(message));
- const attachmentsRef = useRef(attachments.attachments);
- attachmentsRef.current = attachments.attachments;
+ const attachments = useChatAttachments({ sessionId, cwd, provider, surface }, (message) => setProblem(message),
+ { owner: chatStore, key: sessionId ?? "" });
+ useLayoutEffect(() => {
+ const field = fieldRef.current;
+ const current = field?.getDraft();
+ if (current && (current.text !== messageDraft.text || JSON.stringify(current.choices) !== JSON.stringify(messageDraft.choices))) {
+ field?.hydrate(messageDraft.text, messageDraft.choices);
+ }
+ }, [messageDraft, sessionId]);
const commands = useChatCommands({ value, agentId: agent.agentId, onClear,
attachments: attachments.attachments, attachmentsBusy: attachments.analyzing > 0, onAttachmentsSent: attachments.clear,
setValue: (next) => { setValue(next); fieldRef.current?.setText(next); },
@@ -1145,12 +1176,12 @@ export function Composer({ agent, mentionable, busy, sessionId, cwd, provider, s
}, () => void submit());
const timeline = useAgentChat((s) => s.timeline);
const sending = useAgentChat((s) => s.busy);
- // `busy` on this composer also covers "session not open yet". Stop is only
- // for a live turn: the HTTP send, or the stream after it (reasoning, tools).
+ // Track HTTP admission and running work independently of composer readiness.
const live = runningTurn(timeline) !== null || sending;
- // A created agent's one chat never refuses its person: a message written
- // while it works waits and starts as its next turn (MASTERPLAN §2.10).
- const canQueue = surface === "society" && agent.tier !== "lead" && !sending && runningTurn(timeline) !== null;
+ // Admission is serialized by the store; execution waits in the backend.
+ // A pending HTTP request or credential card does not lock this composer.
+ const canQueue = surface === "jarvis" || surface === "society";
+ const sessionReady = surface !== "society" || Boolean(sessionId && chatStore.getState().activeSessionId === sessionId);
// "@" completes teammates AND the capability catalog — plugins, MCP
// servers, CLIs, skills, Jarvis tools — on every agent card, including
@@ -1191,8 +1222,8 @@ export function Composer({ agent, mentionable, busy, sessionId, cwd, provider, s
if (activeIndex >= matches.length) setActiveIndex(Math.max(0, matches.length - 1));
}, [matches.length, activeIndex]);
- const onDraftChange = (draft: { text: string; caret: number }) => {
- setValue(draft.text);
+ const onDraftChange = (draft: ComposerDraft) => {
+ writeComposerDraft(chatStore, sessionId, { text: draft.text, choices: draft.choices });
setMention(mentionToken(draft.text, draft.caret));
};
@@ -1211,21 +1242,18 @@ export function Composer({ agent, mentionable, busy, sessionId, cwd, provider, s
const row = mentionChoice(item);
const next = `${before}${choiceToken(row)} ${after}`;
field?.hydrate(next, [...(draft?.choices ?? []), row]);
- setSelectedTools((items) => (items.some((row) => row.key === item.key) ? items : [...items, item]));
}
setMention(null);
};
const submit = async () => {
+ if (await commands.execute(fieldRef.current?.getDraft().text.trim() ?? value.trim())) return;
const draft = fieldRef.current?.getDraft();
const fullDraft = draft?.text ?? value;
const draftText = fullDraft.trim();
const submittedFolder = codingFolder;
- const selected = selectedTools;
- const sentAttachments = attachments.attachments;
const text = draftText;
- if (await commands.execute(text)) return;
- if ((!text && attachments.attachments.length === 0) || (busy || live) && !commands.canSteer && !canQueue || modelSaving || attachments.analyzing > 0) return;
+ if ((!text && attachments.attachments.length === 0) || !sessionReady || (busy || live) && !commands.canSteer && !canQueue || modelSaving || attachments.analyzing > 0) return;
const chosenIds = new Set((draft?.choices ?? []).map((row) => row.id));
const chosen = [...chosenIds].map((id) => catalog.find((item) => item.key === id));
if (chosen.some((item) => !item || !item.connected)) {
@@ -1258,25 +1286,34 @@ export function Composer({ agent, mentionable, busy, sessionId, cwd, provider, s
if (codingHint) lines.push(codingHint);
const hint = lines.join("\n");
setProblem(null);
+ const held = attachments.take();
+ if (!text && held.attachments.length === 0) return;
+ const draftTarget = captureComposerDraftTarget(chatStore, sessionId);
+ const sentDraft = { text: fullDraft, choices: draft?.choices ?? messageDraft.choices };
+ writeComposerDraft(chatStore, sessionId, { text: "", choices: [] });
+ fieldRef.current?.clear();
+ setMention(null);
+ setCodingFolder("");
try {
- const result = await onSend(hint ? `${text}\n\n${hint}` : text, sentAttachments);
- if (result === "stale" || sessionId && chatStore.getState().activeSessionId !== sessionId) return;
+ const result = await onSend(hint ? `${text}\n\n${hint}` : text, held.attachments);
const sendError = chatStore.getState().lastError;
if (result === "failed" || result === undefined && sendError) throw new Error(sendError ?? t("common.error_generic"));
- if ((fieldRef.current?.getDraft().text ?? value) === fullDraft) {
- setValue("");
- fieldRef.current?.clear();
- setMention(null);
- setSelectedTools([]);
- }
- if (attachmentsRef.current === sentAttachments) attachments.clear();
- else sentAttachments.forEach((file) => attachments.remove(file.name));
+ releaseHeldFiles(held);
} catch (err) {
- // The input and files stay in place until the server accepts them.
- if ((fieldRef.current?.getDraft().text ?? value) === fullDraft) {
- setSelectedTools(selected);
- setCodingFolder(submittedFolder);
+ if (draftTarget.superseded) {
+ releaseHeldFiles(held);
+ useEventStore.getState().pushToast("error", t("society.chat.message_not_sent").replace("{0}", sentDraft.text));
+ return;
}
+ // Restore only this submission's draft, even after changing agents.
+ // Preserve anything typed while the HTTP request was pending.
+ const current = readComposerDraft(chatStore, draftTarget.sessionId);
+ writeComposerDraft(chatStore, draftTarget.sessionId, {
+ text: [sentDraft.text, current.text].filter(Boolean).join("\n"),
+ choices: [...sentDraft.choices, ...current.choices.filter((row) => !sentDraft.choices.some((sent) => sent.id === row.id))],
+ });
+ restoreHeldFiles(chatStore, draftTarget.sessionId ?? "", held);
+ setCodingFolder(submittedFolder);
setProblem(err instanceof Error ? err.message : String(err));
}
};
@@ -1430,27 +1467,21 @@ export function Composer({ agent, mentionable, busy, sessionId, cwd, provider, s
stopLabel={t("society.chat.stop_recording")}
shape="round"
/>
- {live && !commands.isCommand && !((commands.canSteer || canQueue) && value.trim()) ? (
- <button
- type="button"
- onClick={() => void onCancel()}
- aria-label={t("society.chat.stop")}
- title={t("society.chat.stop")}
- data-testid="composer-stop"
- className="flex h-8 w-8 items-center justify-center rounded-full bg-foreground text-background transition-colors hover:bg-foreground/90"
- >
- <Square className="h-3.5 w-3.5" aria-hidden />
- </button>
- ) : (
+ {(busy || live) && <button type="button" onClick={() => void onCancel()}
+ aria-label={t("society.chat.stop")} data-testid="composer-cancel"
+ className="flex h-8 w-8 items-center justify-center rounded-full bg-foreground text-background hover:bg-foreground/90">
+ <Square className="h-3.5 w-3.5" aria-hidden />
+ </button>}
+ {(
<button
type="button"
// While recording, Send ends the dictation and sends once the
// words land, so it is live before the box holds any text.
onClick={() => (dictation.dictating ? dictation.stopAndSend() : void submit())}
- disabled={modelSaving || attachments.analyzing > 0 || (!value.trim() && selectedTools.length === 0 && attachments.attachments.length === 0 && !dictation.dictating)}
+ disabled={!sessionReady || modelSaving || attachments.analyzing > 0 || (!value.trim() && messageDraft.choices.length === 0 && attachments.attachments.length === 0 && !dictation.dictating)}
aria-label={t("society.chat.send")}
data-testid="composer-send"
- className="flex h-8 w-8 items-center justify-center rounded-full bg-primary text-primary-foreground disabled:opacity-40"
+ className="flex h-8 w-8 items-center justify-center rounded-full bg-[hsl(var(--chat-send-background))] text-[hsl(var(--chat-send-foreground))] disabled:[&_svg]:opacity-40"
>
<Send className="h-3.5 w-3.5" aria-hidden />
</button>
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.test.tsx b/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.test.tsx
index 2d6119ccae..55107721a8 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.test.tsx
@@ -11,6 +11,7 @@ import type { SocietyAgentRow } from "@/lib/societyApi";
import { AgentChatStoreProvider } from "@/components/agentchat/AgentChatStoreContext";
import { createAgentChatStore, type AgentChatStoreHook } from "@/store/agentChat";
import { clearModelMenuSnapshot, MODEL_MENU_SNAPSHOT_KEY, writeModelMenuSnapshot } from "./modelMenuSnapshot";
+import { MODEL_ACCESS_KEY } from "@/lib/modelAccess";
const provider = (id: string, overrides: Partial<AgentChatProvider> = {}): AgentChatProvider => ({
id, label: id, family: id, runner: "brain", models_source: "curated",
@@ -25,6 +26,10 @@ let failSave: boolean;
let extraProviders: AgentChatProvider[];
let liveModels: Record<string, { id: string; label: string }[]>;
let providerRows: unknown[];
+let accessWays: Record<string, string[]>;
+let supportedProviders: string[];
+let runtimeGate: Promise<void> | undefined;
+let failedModelProvider: string;
let catalogCalls: number;
let saving: Mock<(saving: boolean) => void>;
let catalogGate: Promise<void> | undefined;
@@ -32,6 +37,7 @@ let connectionsGate: Promise<void> | undefined;
let accountsGate: Promise<void> | undefined;
beforeEach(async () => {
+ localStorage.removeItem(MODEL_ACCESS_KEY);
clearModelMenuSnapshot();
await loadLocaleChunk("society");
row = {
@@ -41,12 +47,19 @@ beforeEach(async () => {
} as unknown as SocietyAgentRow;
posts = []; failSave = false; saving = vi.fn<(saving: boolean) => void>();
extraProviders = []; liveModels = {}; providerRows = []; catalogCalls = 0;
+ accessWays = {};
+ supportedProviders = []; runtimeGate = undefined;
+ failedModelProvider = "";
catalogGate = undefined; connectionsGate = undefined; accountsGate = undefined;
vi.stubGlobal("fetch", vi.fn(async (url: string, init?: RequestInit) => {
if (url.includes("/catalog")) await catalogGate;
if (url.endsWith("/status")) await connectionsGate;
if (url.endsWith("/providers")) await accountsGate;
const response = (data: unknown, ok = true) => ({ ok, status: ok ? 200 : 503, json: async () => data }) as Response;
+ if (url === "/api/agent-runtimes") {
+ await runtimeGate;
+ return response({ runtimes: [], supported_providers: supportedProviders, access: accessWays });
+ }
if (url.endsWith("/model")) {
const choice = JSON.parse(String(init?.body));
posts.push(choice);
@@ -56,7 +69,10 @@ beforeEach(async () => {
}
if (url.includes("/catalog")) { catalogCalls++; return response({ providers: [provider("openai"), provider("gemini"), provider("offline"), provider("ollama", { keyless: true, models_source: "live" }), ...extraProviders] }); }
if (url.endsWith("/status")) return response({ mapping: [{ jarvis: "openai", key_set: true }, { jarvis: "gemini", key_set: true }, { jarvis: "openrouter", key_set: true }, { jarvis: "offline", key_set: false }] });
- if (url.endsWith("/models")) return response({ models: liveModels[url.split("/").at(-2)!] ?? [] });
+ if (url.endsWith("/models")) {
+ const id = url.split("/").at(-2)!;
+ return response({ models: liveModels[id] ?? [] }, id !== failedModelProvider);
+ }
if (url.endsWith("/providers")) return response({ providers: providerRows });
if (url.endsWith("/agents")) return response({ agents: [row] });
throw new Error(`Unexpected request: ${url}`);
@@ -64,6 +80,34 @@ beforeEach(async () => {
});
afterEach(() => { cleanup(); vi.unstubAllGlobals(); });
+test.each(["hermes", "openclaw"] as const)("%s never offers unapproved providers while runtime metadata loads or is empty", async (runtime) => {
+ row = { ...row, runtime };
+ let release!: () => void;
+ runtimeGate = new Promise<void>((resolve) => { release = resolve; });
+ mount();
+ fireEvent.click(screen.getByRole("button", { name: "Model" }));
+ expect(screen.queryByTitle("openai-small")).toBeNull();
+ await act(async () => { release(); await runtimeGate; });
+ await screen.findByText("No matching models. Connect a provider or refresh the list.");
+ expect(screen.queryByTitle("openai-small")).toBeNull();
+ supportedProviders = ["gemini"];
+ fireEvent.click(screen.getByRole("button", { name: "Refresh models" }));
+ await screen.findByTitle("gemini-small");
+ expect(screen.queryByTitle("openai-small")).toBeNull();
+ expect(posts).toEqual([]);
+});
+
+test("a failed OpenRouter catalog reports its error without hiding another provider", async () => {
+ extraProviders = [provider("openrouter", { models_source: "live" })];
+ failedModelProvider = "openrouter";
+ mount(); await open(); show("openrouter");
+ await screen.findByRole("alert");
+ show("gemini");
+ expect(screen.getByTitle("gemini-small")).toBeTruthy();
+ expect(screen.queryByRole("alert")).toBeNull();
+ expect(posts).toEqual([]);
+});
+
test("a first open after page reload needs no request when a snapshot is available", async () => {
writeModelMenuSnapshot({ version: 1, savedAt: Date.now(),
catalog: { providers: [provider("openai")], default_cwd: "", shell: "" },
@@ -193,9 +237,9 @@ test("catalog loading starts when the card mounts, before the picker is clicked"
expect(catalogCalls).toBe(1);
});
-async function open() {
+async function open(expected = "openai-large") {
fireEvent.click(screen.getByRole("button", { name: "Model" }));
- await screen.findByTitle("openai-large");
+ await screen.findByTitle(expected);
return screen.getByRole("textbox", { name: "Search models" });
}
@@ -389,6 +433,46 @@ test.each([
await waitFor(() => expect(posts[0]).toEqual({ provider: id, model: "", effort: "", account_id: "" }));
});
+test("one logo covers OpenAI access, defaults to subscription and retains an explicit API choice", async () => {
+ extraProviders = [provider("openai-codex", { family: "openai", runner: "codex-cli", cli_installed: true })];
+ providerRows = [{ id: "openai-codex", subscription: true, accounts: [{ id: "work", label: "Work", connected: true }] }];
+ mount(); await open("openai-codex-small");
+ expect(screen.queryByTestId("agent-model-rail-openai-codex")).toBeNull();
+ expect(screen.getByTestId("agent-model-rail-openai")).toBeTruthy();
+ expect(screen.getByRole("radio", { name: "Subscription" }).getAttribute("aria-checked")).toBe("true");
+ expect(screen.getByTitle("openai-codex-small")).toBeTruthy();
+ fireEvent.click(screen.getByRole("radio", { name: "API key" }));
+ expect(screen.getByTitle("openai-small")).toBeTruthy();
+ expect(posts).toEqual([]);
+ fireEvent.keyDown(screen.getByRole("radio", { name: "API key" }), { key: "Escape" });
+ await open();
+ expect(screen.getByRole("radio", { name: "API key" }).getAttribute("aria-checked")).toBe("true");
+ show("gemini"); show("openai");
+ expect(screen.getByRole("radio", { name: "API key" }).getAttribute("aria-checked")).toBe("true");
+});
+
+test("a dual Claude row saves the exact access and never carries a subscription account onto its API", async () => {
+ row = { ...row, provider: "claude-api", model: "claude-sonnet-5", account_id: "work" };
+ accessWays = { "claude-api": ["api", "subscription"] };
+ extraProviders = [provider("claude-api", { family: "claude", runner: "claude-cli", cli_installed: true,
+ curated_models: [{ id: "opusplan", label: "Opus Plan" }, { id: "claude-sonnet-5", label: "Sonnet" }] })];
+ providerRows = [{ id: "claude-api", subscription: true, accounts: [{ id: "work", label: "Work", connected: true }] }];
+ mount(); await open("claude-sonnet-5");
+ await screen.findByRole("radio", { name: "Subscription" });
+ fireEvent.click(screen.getByRole("button", { name: /Older models/ }));
+ expect(screen.getByTitle("opusplan")).toBeTruthy();
+ fireEvent.click(screen.getByRole("radio", { name: "API key" }));
+ expect(screen.queryByTitle("opusplan")).toBeNull();
+ expect(screen.queryByRole("button", { name: /^Account:/ })).toBeNull();
+ fireEvent.click(screen.getByTitle("claude-sonnet-5"));
+ await waitFor(() => expect(posts[0]).toMatchObject({ provider: "claude-api", model: "claude-sonnet-5", account_id: "api-key" }));
+ await open("claude-sonnet-5");
+ fireEvent.click(screen.getByRole("radio", { name: "Subscription" }));
+ fireEvent.click(screen.getByRole("button", { name: /Older models/ }));
+ fireEvent.click(screen.getByTitle("opusplan"));
+ await waitFor(() => expect(posts[1]).toMatchObject({ provider: "claude-api", model: "opusplan", account_id: "subscription" }));
+});
+
test("rail marks move with Alt+Arrow and keep their place on reopen", async () => {
localStorage.removeItem("jarvis.chat.providerOrder");
mount(); const input = await open();
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.tsx b/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.tsx
index 4a1b063485..223914c372 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/AgentModelPicker.tsx
@@ -17,8 +17,12 @@ import { blockedReasonKey } from "../create/seatChoice";
import { useModelMenuData } from "./useModelMenuData";
import { RuntimeStatusRow, useAgentRuntimes } from "../card/RuntimePicker";
+import { providerChoices, API_KEY_ACCOUNT, SUBSCRIPTION_ACCOUNT } from "../create/seatChoice";
+import { modelAccessFamily, preferredModelAccess, useModelAccess } from "@/lib/modelAccess";
+import { ModelAccessSwitch } from "@/components/providers/ModelAccessSwitch";
type Submenu = { provider: string; model?: CuratedModel; anchor: DOMRect };
+type PickerSeat = BrainSeat & { key: string; family: string; accessAccount: string };
const menuRow = "flex w-full items-center gap-2 px-3 py-1.5 text-left text-[13px] text-popover-foreground hover:bg-secondary focus-visible:bg-secondary focus-visible:outline-none disabled:opacity-45";
const FAVORITES = "favorites";
@@ -43,6 +47,7 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
const [expanded, setExpanded] = useState<Record<string, boolean>>({});
// Rows are prepared while closed, so start on the agent's own provider.
const [section, setSection] = useState(agent.provider);
+ const [access, rememberAccess] = useModelAccess();
const [favorites, toggleFavorite] = useFavoriteModels();
const [position, setPosition] = useState<CSSProperties>({});
const trigger = useRef<HTMLButtonElement>(null);
@@ -58,10 +63,14 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
const chatCatalog = useAgentChat((state) => state.surface === "society" ? state.catalog : null);
const chatConnections = useAgentChat((state) => state.connections);
- const { options, providers, live, loading, refreshing, failed, refresh: refreshData } = useModelMenuData(chatCatalog, chatConnections, { [agent.provider]: agent.accountId ?? "", ...accounts });
+ const { options, providers, live, modelErrors, loading: catalogLoading, refreshing: catalogRefreshing, failed: catalogFailed, refresh: refreshData } = useModelMenuData(chatCatalog, chatConnections,
+ { [agent.provider]: agent.accountId ?? "", ...Object.fromEntries(Object.entries(accounts).map(([key, account]) => [key.slice(0, key.lastIndexOf(":")), account])) });
const defaultModelLabel = t("agent_chat.model_default");
const runtimes = useAgentRuntimes();
const external = Boolean(agent.runtime && agent.runtime !== "jarvis");
+ const loading = catalogLoading || (external && runtimes.isPending);
+ const refreshing = catalogRefreshing || (external && runtimes.isFetching);
+ const failed = catalogFailed || (external && runtimes.isError);
const supportedKey = (Array.isArray(runtimes.data?.supported_providers) ? runtimes.data.supported_providers : []).join(",");
const gatewayKey = (Array.isArray(runtimes.data?.subscription_providers) ? runtimes.data.subscription_providers : []).join(",");
const loginKey = (Array.isArray(runtimes.data?.login_providers) ? runtimes.data.login_providers : []).join(",");
@@ -69,20 +78,29 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
const all = modelSeats(options, providers ?? [], live, defaultModelLabel);
// Hermes / OpenClaw run on an API key, a local model or a subscription
// Jarvis' model gateway serves; never a subscription CLI's own loop.
- if (!external || !supportedKey) return all;
+ if (!external) return all;
const list = (key: string) => key.split(",").filter(Boolean);
return runtimeSeats(all, list(supportedKey), list(gatewayKey), list(loginKey));
}, [options, providers, live, defaultModelLabel, external, supportedKey, gatewayKey, loginKey]);
- const currentAccount = (seat: BrainSeat) => accounts[seat.provider.id] ?? (agent.provider === seat.provider.id ? agent.accountId ?? "" : "");
- // A seat the provider refuses right now (Claude's login with Extra Usage off)
- // stays visible with its reason; its models cannot be picked.
- const blockedFor = (seat: BrainSeat) => external ? seatBlocked(seat, runtimes.data?.access_blocked, currentAccount(seat)) : "";
+ const accessKey = JSON.stringify(runtimes.data?.access ?? {});
+ const blockedKey = JSON.stringify(external ? runtimes.data?.access_blocked ?? {} : {});
+ const choices = useMemo(() => providerChoices(seats, JSON.parse(accessKey), external, JSON.parse(blockedKey)).sort((a, b) =>
+ Math.min(...a.options.map((option) => modelGroupOrder(option.seat))) - Math.min(...b.options.map((option) => modelGroupOrder(option.seat)))), [seats, accessKey, blockedKey, external]);
+ const accessSeats = useMemo<PickerSeat[]>(() => choices.flatMap((choice) => choice.options.map((option) => ({
+ ...option.seat, key: `${option.seat.provider.id}:${option.kind}`, family: choice.id, accessAccount: option.accountId,
+ }))), [choices]);
+ const currentAccount = (seat: PickerSeat) => {
+ if (seat.kind !== "subscription") return seat.accessAccount || (agent.provider === seat.provider.id ? agent.accountId ?? "" : "");
+ const stored = agent.provider === seat.provider.id ? agent.accountId ?? "" : "";
+ return accounts[seat.key] ?? (stored && stored !== API_KEY_ACCOUNT && stored !== SUBSCRIPTION_ACCOUNT ? stored : seat.accessAccount);
+ };
+ const blockedFor = (seat: PickerSeat) => external ? seatBlocked(seat, runtimes.data?.access_blocked, currentAccount(seat)) : "";
const preferredEffort = (seat: BrainSeat, model: CuratedModel) => modelEffort(seat, model.id, seat.provider.id === agent.provider ? agent.effort : seat.provider.default_effort);
// useT returns a new function each render; memoize by its actual labels.
- const titleKey = JSON.stringify(seats.map((seat) => providerTitle(seat, t)));
+ const titleKey = JSON.stringify(accessSeats.map((seat) => providerTitle(seat, t)));
const groups = useMemo(() => {
const titles = JSON.parse(titleKey) as string[];
- return seats.map((seat, index) => {
+ return accessSeats.map((seat, index) => {
const matched = seat.provider.curated_models.filter((model) => matchesModel(seat, model, search, titles[index]));
// Newest of each model line first; earlier versions fold away. Catalogs
// with their own fold (OpenCode, OpenRouter) keep their order.
@@ -90,23 +108,26 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
return { seat, title: titles[index], models: [...ranked.current, ...ranked.older], older: new Set(ranked.older) };
}).filter((group) => group.models.length > 0).sort((a, b) =>
modelGroupOrder(a.seat) - modelGroupOrder(b.seat) || a.title.localeCompare(b.title));
- }, [seats, search, titleKey]);
- const sideSeat = seats.find((seat) => seat.provider.id === submenu?.provider);
+ }, [accessSeats, search, titleKey]);
+ const sideSeat = accessSeats.find((seat) => seat.key === submenu?.provider);
const browsing = !search.trim();
- const activeSection = section === FAVORITES || groups.some((group) => group.seat.provider.id === section) ? section : groups[0]?.seat.provider.id ?? "";
- const showFavorites = browsing && activeSection === FAVORITES;
- const shownGroups = browsing ? groups.filter((group) => group.seat.provider.id === activeSection) : groups;
+ const activeChoice = choices.find((choice) => choice.id === section || choice.options.some((option) => option.seat.provider.id === section)) ?? choices[0];
+ const activeOption = activeChoice ? preferredModelAccess(activeChoice.options, access[activeChoice.id]) : undefined;
+ const showFavorites = browsing && section === FAVORITES;
+ const shownGroups = browsing ? groups.filter((group) => group.seat.family === activeChoice?.id && group.seat.kind === activeOption?.kind) : groups;
const [providerOrder, moveProvider] = useProviderOrder();
- const railSeats = orderBy(groups.map((group) => group.seat), (seat) => seat.provider.id, providerOrder);
- const railIds = railSeats.map((seat) => seat.provider.id);
+ const railChoices = orderBy(choices, (choice) => choice.id, providerOrder.map(modelAccessFamily));
+ const railIds = railChoices.map((choice) => choice.id);
const starred = useMemo(() => favorites.flatMap((value) => {
- for (const seat of seats) {
+ for (const seat of accessSeats) {
+ const sameProvider = accessSeats.filter((entry) => entry.provider.id === seat.provider.id);
+ if (preferredModelAccess(sameProvider, access[seat.family]) !== seat) continue;
const model = seat.provider.curated_models.find((entry) => brainValue(seat.provider.id, entry.id) === value);
if (model) return [{ seat, model }];
}
return [];
- }), [favorites, seats]);
- const pickSection = (id: string) => { setSubmenu(null); setSection(id); input.current?.focus({ preventScroll: true }); };
+ }), [favorites, accessSeats, access]);
+ const pickSection = (id: string) => { setSubmenu(null); setSearch(""); setSection(id); input.current?.focus({ preventScroll: true }); };
function close() {
if (inFlight.current) return;
@@ -183,11 +204,12 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
return () => document.removeEventListener("pointerdown", outside);
}, [open]);
- async function save(seat: BrainSeat, model: CuratedModel, effort = preferredEffort(seat, model)) {
- if (busy || inFlight.current) return;
+ async function save(seat: PickerSeat, model: CuratedModel, effort = preferredEffort(seat, model)) {
+ if (busy || inFlight.current || blockedFor(seat)) return;
inFlight.current = true; setSaving(true); onSavingChange(true); setError(null);
try {
await update(agent.agentId, { provider: seat.provider.id, model: model.id, effort, account_id: currentAccount(seat) });
+ rememberAccess(seat.family, seat.kind);
setOpen(false); setSubmenu(null); trigger.current?.focus();
} catch (err) {
setError(err instanceof RuntimeProviderUnsupported
@@ -200,7 +222,7 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
async function refresh() {
setSubmenu(null);
- await refreshData();
+ await Promise.all([refreshData(), runtimes.refetch()]);
}
function moveFocus(event: KeyboardEvent, container: HTMLElement | null) {
@@ -226,8 +248,9 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
} : {};
/** One model row; `owner` adds the provider's name under it (the starred tab mixes providers). */
- const row = (seat: BrainSeat, model: CuratedModel, owner = false) => {
- const selected = agent.provider === seat.provider.id && agent.model === model.id && (agent.accountId ?? "") === currentAccount(seat);
+ const row = (seat: PickerSeat, model: CuratedModel, owner = false) => {
+ const selected = agent.provider === seat.provider.id && agent.model === model.id
+ && (agent.accountId ?? "") === currentAccount(seat);
const effort = preferredEffort(seat, model);
const value = brainValue(seat.provider.id, model.id);
const isFavorite = favorites.includes(value);
@@ -237,7 +260,7 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
onKeyDown={(event) => {
if (event.key === "ArrowRight" && effortsFor(seat, model.id).length) {
event.preventDefault(); event.stopPropagation();
- setSubmenu({ provider: seat.provider.id, model, anchor: event.currentTarget.getBoundingClientRect() });
+ setSubmenu({ provider: seat.key, model, anchor: event.currentTarget.getBoundingClientRect() });
}
}}
onClick={() => void save(seat, model)} className={cn(menuRow, "min-w-0 flex-1")} title={model.id}>
@@ -256,7 +279,7 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
<Star className={cn("h-3 w-3", isFavorite && "fill-current")} aria-hidden />
</button>
{effortsFor(seat, model.id).length ? <button type="button" disabled={busy || saving} aria-label={`${t("society.chat.effort")}: ${model.label}`} aria-haspopup="menu"
- onClick={(event) => setSubmenu({ provider: seat.provider.id, model, anchor: event.currentTarget.getBoundingClientRect() })}
+ onClick={(event) => setSubmenu({ provider: seat.key, model, anchor: event.currentTarget.getBoundingClientRect() })}
className="mr-1 rounded p-1.5 text-muted-foreground opacity-60 hover:bg-secondary hover:opacity-100 focus-visible:opacity-100 focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring">
<ChevronRight className="h-3 w-3" aria-hidden />
</button> : null}
@@ -277,17 +300,18 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
{host && (open || !loading) ? createPortal(<>
<div ref={panel} id={menuId} aria-hidden={!open} style={{ ...position, visibility: open ? "visible" : "hidden", contain: "layout paint style" }} className="fixed z-[80] flex overflow-hidden rounded-md border border-border bg-popover text-popover-foreground shadow-float"
onKeyDown={(event) => moveFocus(event, panel.current)}>
- {!loading && !failed && railSeats.length ? <div role="toolbar" aria-orientation="vertical" aria-label={t("agent_chat.pick_provider")} data-testid="agent-model-rail"
+ {!loading && !failed && railChoices.length ? <div role="toolbar" aria-orientation="vertical" aria-label={t("agent_chat.pick_provider")} data-testid="agent-model-rail"
className="scrollbar-jarvis flex w-12 shrink-0 flex-col gap-1 overflow-y-auto border-r border-border bg-sheen/[0.03] p-1.5">
<RailButton active={showFavorites} label={t("agent_chat.favorites")} onSelect={() => pickSection(FAVORITES)} testId="agent-model-rail-favorites">
<Star className="h-4 w-4 fill-current" aria-hidden />
</RailButton>
<span className="mx-1 my-0.5 border-b border-border" aria-hidden />
- {railSeats.map((seat, index) => <RailButton key={seat.provider.id} active={browsing && activeSection === seat.provider.id} label={providerTitle(seat, t)}
- reorder={{ id: seat.provider.id, onMove: (dragged, target) => moveProvider(railIds, dragged, target),
- onStep: (step) => { const target = railIds[index + step]; if (target) moveProvider(railIds, seat.provider.id, target); } }}
- onSelect={() => pickSection(seat.provider.id)} testId={`agent-model-rail-${seat.provider.id}`}>
- <span className="inline-flex scale-[1.3]"><ProviderLogo providerId={seat.provider.id} label={seat.provider.label} size="sm" /></span>
+ {railChoices.map((choice, index) => <RailButton key={choice.id} active={browsing && !showFavorites && activeChoice?.id === choice.id}
+ label={choice.options.length > 1 ? choice.label : providerTitle(choice.options[0].seat, t)}
+ reorder={{ id: choice.id, onMove: (dragged, target) => moveProvider(railIds, dragged, target),
+ onStep: (step) => { const target = railIds[index + step]; if (target) moveProvider(railIds, choice.id, target); } }}
+ onSelect={() => pickSection(choice.id)} testId={`agent-model-rail-${choice.options.length === 1 ? choice.options[0].seat.provider.id : choice.id}`}>
+ <span className="inline-flex scale-[1.3]"><ProviderLogo providerId={choice.logo} label={choice.label} size="sm" /></span>
</RailButton>)}
</div> : null}
<div className="flex min-h-0 min-w-0 flex-1 flex-col">
@@ -303,27 +327,32 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
className="min-w-0 flex-1 bg-transparent text-sm outline-none placeholder:text-muted-foreground" />
{refreshing ? <Loader2 className="h-3.5 w-3.5 animate-spin text-muted-foreground" aria-label={t("society.chat.models_loading")} /> : null}
</div>
+ {browsing && !showFavorites && activeChoice && activeOption ? <ModelAccessSwitch
+ options={activeChoice.options} value={activeOption.kind} disabled={busy || saving}
+ onChange={(kind) => { setSubmenu(null); rememberAccess(activeChoice.id, kind); }} /> : null}
<div role="menu" aria-label={t("society.chat.model")} aria-busy={saving} className="min-h-0 overflow-y-auto overscroll-contain py-1" onScroll={() => setSubmenu(null)}>
{loading ? <p role="status" className="px-3 py-3 text-xs text-muted-foreground">{t("society.create.catalog_loading")}</p> : null}
{failed ? <p role="alert" className="px-3 py-3 text-xs text-destructive">{t("society.chat.model_load_failed")}</p> : null}
+ {browsing && !showFavorites && activeOption && modelErrors.includes(activeOption.seat.provider.id)
+ ? <p role="alert" className="px-3 py-3 text-xs text-destructive">{t("society.chat.model_load_failed")}</p> : null}
{!loading && !failed && groups.length === 0 ? <p className="px-3 py-3 text-xs text-muted-foreground">{t(refreshing ? "society.chat.models_loading" : "society.chat.model_no_matches")}</p> : null}
{!loading && !failed && showFavorites ? (starred.length
? starred.map(({ seat, model }) => row(seat, model, true))
: <p className="px-3 py-3 text-xs text-muted-foreground">{t("agent_chat.favorites_empty")}</p>) : null}
{!loading && !failed && !showFavorites ? shownGroups.map(({ seat, title, models, older }) => {
- const isExpanded = expanded[seat.provider.id] ?? false;
+ const isExpanded = expanded[seat.key] ?? false;
const shown = visibleModels(seat, models, isExpanded, search);
const foldable = collapsibleModels(seat) && !search.trim();
const isRouter = seat.provider.family === "openrouter";
- const choicesId = `${menuId}-${seat.provider.id}-models`;
- const toggle = () => { setSubmenu(null); setExpanded((previous) => ({ ...previous, [seat.provider.id]: !isExpanded })); };
- const olderKey = `older:${seat.provider.id}`;
+ const choicesId = `${menuId}-${seat.key}-models`;
+ const toggle = () => { setSubmenu(null); setExpanded((previous) => ({ ...previous, [seat.key]: !isExpanded })); };
+ const olderKey = `older:${seat.key}`;
const folded = search.trim() ? [] : shown.filter((model) => older.has(model));
const lineup = folded.length ? shown.filter((model) => !older.has(model)) : shown;
const olderOpen = expanded[olderKey] ?? folded.some((model) => agent.provider === seat.provider.id && agent.model === model.id);
const toggleOlder = () => { setSubmenu(null); setExpanded((previous) => ({ ...previous, [olderKey]: !olderOpen })); };
const toRow = (model: CuratedModel) => row(seat, model);
- return <div key={seat.provider.id} role="group" aria-label={title}>
+ return <div key={seat.key} role="group" aria-label={title}>
<div className="sticky top-0 z-10 flex items-center gap-1 bg-popover px-3 pb-1 pt-3">
{foldable && isRouter ? <button type="button" data-menu-choice disabled={busy || saving}
onClick={toggle} aria-label={title} aria-expanded={isExpanded} aria-controls={choicesId}
@@ -332,8 +361,8 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
<span className="truncate">{title}</span><span className="ml-auto">{models.length}</span>
</button> : <span className="min-w-0 flex-1 truncate text-[11px] font-semibold uppercase tracking-wide text-muted-foreground" title={title}>{title}</span>}
{seat.accounts.length ? <button type="button" disabled={busy || saving} data-menu-choice
- aria-label={`${t("society.chat.model_account")}: ${title}`} aria-haspopup="menu" aria-expanded={submenu?.provider === seat.provider.id && !submenu.model}
- onClick={(event) => setSubmenu({ provider: seat.provider.id, anchor: event.currentTarget.getBoundingClientRect() })}
+ aria-label={`${t("society.chat.model_account")}: ${title}`} aria-haspopup="menu" aria-expanded={submenu?.provider === seat.key && !submenu.model}
+ onClick={(event) => setSubmenu({ provider: seat.key, anchor: event.currentTarget.getBoundingClientRect() })}
className="flex max-w-[140px] items-center gap-1 rounded px-1 py-0.5 text-[10px] text-muted-foreground hover:bg-secondary focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring">
<Users className="h-3 w-3 shrink-0" aria-hidden /><span className="truncate">{seat.accounts.find((account) => account.id === currentAccount(seat))?.label ?? t("society.chat.model_active_account")}</span><ChevronDown className="h-2.5 w-2.5 shrink-0" aria-hidden />
</button> : null}
@@ -375,9 +404,9 @@ export function AgentModelPicker({ agent, busy, onSavingChange }: {
aria-checked={effort === preferredEffort(sideSeat, submenu.model!)} disabled={busy || saving}
onClick={() => void save(sideSeat, submenu.model!, effort)} className={menuRow}>
{effortLabel(effort, t)}{effort === preferredEffort(sideSeat, submenu.model!) ? <Check className="ml-auto h-3.5 w-3.5" aria-hidden /> : null}
- </button>) : [{ id: "", label: t("society.chat.model_active_account") }, ...sideSeat.accounts].map((account) => <button key={account.id} type="button" role="menuitemradio" data-menu-choice
+ </button>) : [{ id: sideSeat.accessAccount, label: t("society.chat.model_active_account") }, ...sideSeat.accounts].map((account) => <button key={account.id} type="button" role="menuitemradio" data-menu-choice
aria-checked={account.id === currentAccount(sideSeat)} disabled={busy || saving}
- onClick={() => { setAccounts((previous) => ({ ...previous, [sideSeat.provider.id]: account.id })); setSubmenu(null); input.current?.focus(); }} className={menuRow}>
+ onClick={() => { setAccounts((previous) => ({ ...previous, [sideSeat.key]: account.id || sideSeat.accessAccount })); setSubmenu(null); input.current?.focus(); }} className={menuRow}>
<span className="truncate">{account.label}</span>{account.id === currentAccount(sideSeat) ? <Check className="ml-auto h-3.5 w-3.5" aria-hidden /> : null}
</button>)}
</div> : null}
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/ComposerFollowups.test.tsx b/jarvis/ui/web/frontend/src/components/society/chat/ComposerFollowups.test.tsx
new file mode 100644
index 0000000000..573f2d5724
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/chat/ComposerFollowups.test.tsx
@@ -0,0 +1,139 @@
+import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { AgentChatStoreProvider } from "@/components/agentchat/AgentChatStoreContext";
+import { readComposerDraft } from "@/components/agentchat/composerDrafts";
+import { createAgentChatStore } from "@/store/agentChat";
+import type { SocietyAgent } from "../data";
+import { Composer } from "./AgentChatPanel";
+
+vi.mock("../data", async (load) => ({
+ ...await load<typeof import("../data")>(),
+ useSocietyCapabilities: () => ({ data: [], isLoading: false }),
+}));
+vi.mock("./AgentModelPicker", () => ({ AgentModelPicker: () => null }));
+
+beforeEach(() => {
+ vi.stubGlobal("fetch", vi.fn(async () => new Response(JSON.stringify({ commands: [] }))));
+});
+afterEach(() => { cleanup(); vi.unstubAllGlobals(); });
+
+function write(text: string) {
+ const box = screen.getByRole("textbox");
+ box.textContent = text;
+ fireEvent.input(box);
+ return box;
+}
+
+function mount(surface: "jarvis" | "society" = "society") {
+ const store = createAgentChatStore(surface);
+ store.setState({ activeSessionId: "chat-a", busy: true });
+ const cancel = vi.fn().mockResolvedValue(undefined);
+ const send = vi.fn().mockResolvedValue("sent");
+ const tree = () => <AgentChatStoreProvider store={store}>
+ <Composer agent={{ agentId: "george", name: "George", tier: surface === "jarvis" ? "lead" : "worker" } as SocietyAgent}
+ mentionable={[]} busy sessionId="chat-a" cwd="" provider="local" surface={surface}
+ onSend={send} onCancel={cancel} />
+ </AgentChatStoreProvider>;
+ return { store, send, cancel, tree, view: render(tree()) };
+}
+
+it.each(["jarvis", "society"] as const)("keeps only %s Send during startup and accepts rapid followups", async (surface) => {
+ const { send, cancel } = mount(surface);
+ let accept!: (value: string) => void;
+ send.mockImplementationOnce(() => new Promise<string>((resolve) => { accept = resolve; }));
+ write("first followup");
+ expect(screen.getByTestId("composer-send").hasAttribute("disabled")).toBe(false);
+ expect(screen.queryByTestId("composer-stop")).toBeNull();
+ fireEvent.click(screen.getByTestId("composer-send"));
+ await waitFor(() => expect(send).toHaveBeenCalledTimes(1));
+ expect(screen.getByRole("textbox").textContent).toBe("");
+ const box = write("second followup");
+ fireEvent.keyDown(box, { key: "Enter" });
+ await waitFor(() => expect(send).toHaveBeenCalledTimes(2));
+ await act(async () => accept("sent"));
+ expect(send.mock.calls.map((args) => args[0])).toEqual(["first followup", "second followup"]);
+ expect(cancel).not.toHaveBeenCalled();
+});
+
+it("does not send the same draft twice on a rapid repeated submit", async () => {
+ const { send } = mount();
+ const box = write("once");
+ fireEvent.keyDown(box, { key: "Enter" });
+ fireEvent.keyDown(box, { key: "Enter" });
+ await waitFor(() => expect(send).toHaveBeenCalledTimes(1));
+});
+
+it("restores a failed send beside newer typing and keeps that draft across unmounts", async () => {
+ const { send, store, view, tree } = mount();
+ let fail!: (value: string) => void;
+ send.mockImplementationOnce(() => new Promise<string>((resolve) => { fail = resolve; }));
+ write("failed message");
+ fireEvent.click(screen.getByTestId("composer-send"));
+ await waitFor(() => expect(send).toHaveBeenCalledTimes(1));
+ write("newer draft");
+ view.unmount();
+ store.setState({ activeSessionId: "chat-b" });
+ await act(async () => fail("failed"));
+ expect(readComposerDraft(store, "chat-a").text).toBe("failed message\nnewer draft");
+ expect(readComposerDraft(store, "chat-b").text).toBe("");
+ store.setState({ activeSessionId: "chat-a" });
+ render(tree());
+ expect(screen.getByRole("textbox").textContent).toBe("failed message\nnewer draft");
+});
+
+it("sends an attachment-only followup while busy and restores the file on HTTP failure", async () => {
+ const attachment = { name: "notes.txt", reference: "notes.txt", kind: "text", detail: "notes", described_by: "parser", note: "" };
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => new Response(JSON.stringify(
+ String(url).includes("attachments") ? { attachments: [attachment], cwd: "" } : { commands: [] },
+ ))));
+ const { send, view } = mount();
+ send.mockResolvedValueOnce("failed");
+ const input = view.container.querySelector<HTMLInputElement>('input[type="file"]')!;
+ fireEvent.change(input, { target: { files: [new File(["notes"], "notes.txt", { type: "text/plain" })] } });
+ await waitFor(() => expect(screen.getByTestId("composer-send").hasAttribute("disabled")).toBe(false));
+ fireEvent.click(screen.getByTestId("composer-send"));
+ await waitFor(() => expect(send).toHaveBeenCalledWith("", [attachment]));
+ await waitFor(() => expect(screen.getByTestId("composer-send").hasAttribute("disabled")).toBe(false));
+ fireEvent.click(screen.getByTestId("composer-send"));
+ await waitFor(() => expect(send).toHaveBeenCalledTimes(2));
+ expect(send.mock.calls[1]).toEqual(["", [attachment]]);
+});
+
+it("keeps newly typed text through session creation and restores a rejected first send into that session", async () => {
+ const store = createAgentChatStore("jarvis");
+ vi.stubGlobal("WebSocket", undefined);
+ let open!: (response: Response) => void;
+ let rejectMessage!: (response: Response) => void;
+ vi.stubGlobal("fetch", vi.fn(async (input: string, init?: RequestInit) => {
+ const url = String(input);
+ if (url === "/api/agent-chat/sessions" && init?.method === "POST") {
+ return new Promise<Response>((resolve) => { open = resolve; });
+ }
+ if (url.endsWith("/messages")) {
+ return new Promise<Response>((resolve) => { rejectMessage = resolve; });
+ }
+ return new Response(JSON.stringify({ commands: [], sessions: [] }));
+ }));
+ function StartingChat() {
+ const sid = store((state) => state.activeSessionId);
+ const busy = store((state) => state.busy);
+ return <AgentChatStoreProvider store={store}>
+ <Composer agent={{ agentId: "jarvis", name: "Jarvis", tier: "lead" } as SocietyAgent}
+ mentionable={[]} busy={busy} sessionId={sid} cwd="" provider="local"
+ onSend={store.getState().send} onCancel={async () => {}} />
+ </AgentChatStoreProvider>;
+ }
+ render(<StartingChat />);
+ write("first message");
+ fireEvent.click(screen.getByTestId("composer-send"));
+ await waitFor(() => expect(open).toBeTypeOf("function"));
+ write("second draft");
+ await act(async () => open(new Response(JSON.stringify({ session_id: "created", surface: "jarvis" }))));
+ await waitFor(() => expect(store.getState().activeSessionId).toBe("created"));
+ expect(screen.getByRole("textbox").textContent).toBe("second draft");
+ await waitFor(() => expect(rejectMessage).toBeTypeOf("function"));
+ await act(async () => rejectMessage(new Response(JSON.stringify({ detail: "Please retry" }), { status: 503 })));
+ await waitFor(() => expect(screen.getByRole("textbox").textContent).toBe("first message\nsecond draft"));
+ expect(readComposerDraft(store, null).text).toBe("");
+ expect(readComposerDraft(store, "created").text).toBe("first message\nsecond draft");
+});
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/MeetingChat.tsx b/jarvis/ui/web/frontend/src/components/society/chat/MeetingChat.tsx
index addd3a0830..72bd42af12 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/MeetingChat.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/MeetingChat.tsx
@@ -115,12 +115,14 @@ export function MeetingChat({ group, roster }: { group: SocietyChatGroup; roster
<UserBubble key={message.id}
item={{ type: "user", id: message.id, text: message.text, attachments: [], tsMs: 0 }} />
) : (
- <article key={message.id} className="flex min-w-0 flex-col gap-1.5 self-start">
- <header className="flex items-center gap-2">
+ <article key={message.id} className="flex min-w-0 max-w-[min(85%,42rem)] flex-col gap-1.5 self-start">
+ <header className="flex items-center gap-2 px-1">
{agentOf(message.speaker) ? <AgentSwatch agent={agentOf(message.speaker)!} size={22} /> : null}
<span className="text-xs font-medium text-foreground">{nameOf(message.speaker)}</span>
</header>
- <Prose text={message.text} />
+ <div className="jarvis-chat-in min-w-0 self-start rounded-[20px] px-4 py-2.5">
+ <Prose text={message.text} />
+ </div>
</article>
))}
{running ? (
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/MemoryUpdateNotice.tsx b/jarvis/ui/web/frontend/src/components/society/chat/MemoryUpdateNotice.tsx
index 09c1057163..87a1989e67 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/MemoryUpdateNotice.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/MemoryUpdateNotice.tsx
@@ -1,38 +1,44 @@
import { lazy, Suspense, useState } from "react";
-import { ChevronRight, FilePenLine, FileText } from "lucide-react";
+import { NotedLine } from "@/components/agentchat/MessengerTurn";
import type { NoticeItem } from "@/components/agentchat/reduce";
import { useT } from "@/i18n";
const MemoryFileViewer = lazy(() => import("./MemoryFileViewer").then(module => ({ default: module.MemoryFileViewer })));
/**
- * The "Memory updated" receipt. Inside a turn (`inTrace`) it is one quiet
- * step of the turn's work, drawn like a work-log row of a thread; standalone
- * (no turn before it) it keeps the centred chip.
+ * The agent's own memory file a "Memory updated" receipt points at, or null
+ * when the path leaves the agent's folder. The receipt draws no row of its
+ * own: the turn's "Noted" quill line opens it.
*/
-export function MemoryUpdateNotice({ item, inTrace = false }: { item: NoticeItem; inTrace?: boolean }) {
- const t = useT();
- const [open, setOpen] = useState(false);
+export function memoryNoticePath(item: NoticeItem): string | null {
const path = typeof item.data.path === "string" ? item.data.path : "";
+ if (!path.startsWith(`society/${item.agentId}/`) || path.split("/").includes("..")) return null;
+ return path;
+}
+
+/** "Memory updated · USER.md": what a "Noted" line opens, as its tooltip. */
+export function memoryNoticeTitle(t: ReturnType<typeof useT>, path: string): string {
+ return `${t("society.chat.memory_updated")} · ${path.split("/").at(-1)}`;
+}
+
+/** The memory file of a receipt, with the change it made when the receipt carries one. */
+export function MemoryNoticeViewer({ item, onClose }: { item: NoticeItem; onClose: () => void }) {
+ const path = memoryNoticePath(item);
+ if (!path) return null;
const before = typeof item.data.before === "string" ? item.data.before : undefined;
const after = typeof item.data.after === "string" ? item.data.after : undefined;
const patch = Array.isArray(item.data.markdown_diff) && item.data.markdown_diff.every(line => typeof line === "string") ? item.data.markdown_diff as string[] : undefined;
- if (!path.startsWith(`society/${item.agentId}/`) || path.split("/").includes("..")) return null;
- const label = `${t("society.chat.memory_updated")} · ${path.split("/").at(-1)}`;
+ return <Suspense fallback={null}><MemoryFileViewer path={path} before={before} after={after} patch={patch} onClose={onClose} /></Suspense>;
+}
+
+/** A memory receipt with no turn before it: a lone "Noted" quill line that opens the file. */
+export function MemoryUpdateNotice({ item }: { item: NoticeItem }) {
+ const t = useT();
+ const [open, setOpen] = useState(false);
+ const path = memoryNoticePath(item);
+ if (!path) return null;
return <>
- {inTrace ? (
- <button type="button" onClick={() => setOpen(true)} data-trace-memory={item.id} className="group/row flex min-h-6 w-full min-w-0 items-center gap-2.5 rounded-md py-1.5 text-left text-base leading-6 text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-ring">
- <span aria-hidden className="flex h-6 w-4 shrink-0 items-center justify-center"><FilePenLine className="h-4 w-4" strokeWidth={1.75} /></span>
- <span className="min-w-0 truncate transition-colors group-hover/row:text-foreground">{label}</span>
- <ChevronRight aria-hidden className="h-3.5 w-3.5 shrink-0 opacity-0 transition duration-200 group-hover/row:opacity-70 group-focus-visible/row:opacity-70" />
- </button>
- ) : (
- <button type="button" onClick={() => setOpen(true)} className="mx-auto my-3 flex max-w-[min(100%,36rem)] select-none items-center gap-2 rounded-lg bg-secondary px-3 py-2 text-left text-xs text-foreground hover:bg-muted focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
- <FileText size={14} aria-hidden className="shrink-0" />
- <span className="truncate">{label}</span>
- <ChevronRight size={14} aria-hidden className="shrink-0 text-muted-foreground" />
- </button>
- )}
- {open && <Suspense fallback={null}><MemoryFileViewer path={path} before={before} after={after} patch={patch} onClose={() => setOpen(false)} /></Suspense>}
+ <NotedLine link={{ key: item.id, title: memoryNoticeTitle(t, path), onOpen: () => setOpen(true) }} />
+ {open && <MemoryNoticeViewer item={item} onClose={() => setOpen(false)} />}
</>;
}
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/RoutineChatHost.tsx b/jarvis/ui/web/frontend/src/components/society/chat/RoutineChatHost.tsx
index 8e60eb50d2..40eacd20e8 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/RoutineChatHost.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/RoutineChatHost.tsx
@@ -1,6 +1,8 @@
-import { lazy, Suspense, useEffect, type ReactNode } from "react";
+import { lazy, Suspense, useEffect, type ComponentProps, type ReactNode } from "react";
import { useT } from "@/i18n";
import { useRoutineNavigation } from "./routineNavigation";
+import { AgentSwatch } from "../AgentSwatch";
+import { useSocietyAgent } from "../data";
const RoutineChat = lazy(() => import("../card/RoutineChat"));
@@ -16,6 +18,12 @@ export function RoutineChatHost({ agentId, onOpen, children }: {
if (useRoutineNavigation.getState().target?.agentId === agentId) close();
}, [close, agentId]);
return selected ? <Suspense fallback={<p role="status" className="p-4 text-sm">{t("tasks_view.loading_details")}</p>}>
- <RoutineChat key={`${selected.sessionId}:${selected.legacy?.messageId ?? selected.timestamp}`} target={selected} onClose={close} />
+ <RoutineChatWithFace key={`${selected.sessionId}:${selected.legacy?.messageId ?? selected.timestamp}`} target={selected} onClose={close} agentId={agentId} />
</Suspense> : children;
}
+
+/** The routine chat with its agent's face (or worn pet) on the live line of a running trace. */
+function RoutineChatWithFace({ agentId, ...props }: { agentId: string } & ComponentProps<typeof RoutineChat>) {
+ const { agent } = useSocietyAgent(agentId);
+ return <RoutineChat {...props} face={agent ? <AgentSwatch agent={{ ...agent, state: "working" }} size={28} /> : undefined} />;
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/mentionChoices.test.tsx b/jarvis/ui/web/frontend/src/components/society/chat/mentionChoices.test.tsx
index 3145dd2b8f..0652739e6b 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/mentionChoices.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/mentionChoices.test.tsx
@@ -1,5 +1,7 @@
import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
-import { afterEach, expect, it, vi } from "vitest";
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { useAgentChatStore } from "@/store/agentChat";
+import { composerDraftsFor } from "@/components/agentchat/composerDrafts";
import { Composer, UserBubble } from "./AgentChatPanel";
import { mentionChoice, pinnedMessageChoices, withoutChoiceTokens } from "./mentionChoices";
import { buildMentionCatalog, mentionsInText } from "./mentionItems";
@@ -55,18 +57,25 @@ vi.mock("@/components/agentchat/useComposerDictation", () => ({
useComposerDictation: () => ({ dictating: false, stop() {}, toggle() {} }),
}));
vi.mock("@/components/agentchat/DictationButton", () => ({ DictationButton: () => null }));
-vi.mock("@/components/agentchat/useChatAttachments", () => ({
+vi.mock("@/components/agentchat/useChatAttachments", async (original) => ({
+ ...(await original<typeof import("@/components/agentchat/useChatAttachments")>()),
useChatAttachments: () => ({
attachments: [],
analyzing: 0,
dragging: false,
dragHandlers: {},
clear() {},
+ take: () => ({ attachments: [], previews: {} }),
+ restore() {},
remove() {},
attachFiles() {},
}),
}));
afterEach(cleanup);
+beforeEach(() => {
+ useAgentChatStore.setState({ activeSessionId: "society:test", busy: false, lastError: null });
+ composerDraftsFor(useAgentChatStore).setState({ drafts: new Map() });
+});
it("uses Add to select a real branded chip and preserves the existing pin protocol", async () => {
const send = vi.fn(async (_text: string) => {});
@@ -154,6 +163,7 @@ const teammates = [
] as SocietyAgent[];
async function sendDraft(text: string, surface: "society" | "jarvis" = "society") {
+ useAgentChatStore.setState({ activeSessionId: surface === "society" ? "society:gmail-agent" : "jarvis-chat" });
const send = vi.fn(async (_text: string) => {});
render(
<Composer
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.test.ts b/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.test.ts
index e23cf742a7..b85778e6a1 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.test.ts
+++ b/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.test.ts
@@ -53,6 +53,17 @@ test("shared observers do not serialize and write the same data twice", () => {
expect(write).toHaveBeenCalledTimes(1);
});
+test("reload preserves disabled providers and the model visibility saved in API Keys", () => {
+ const source = snapshot();
+ source.catalog.providers[0].enabled = false;
+ source.catalog.providers[0].hidden_models = ["small"];
+ writeModelMenuSnapshot(source);
+ const raw = localStorage.getItem(MODEL_MENU_SNAPSHOT_KEY)!;
+ clearModelMenuSnapshot();
+ localStorage.setItem(MODEL_MENU_SNAPSHOT_KEY, raw);
+ expect(readModelMenuSnapshot()?.catalog.providers[0]).toMatchObject({ enabled: false, hidden_models: ["small"] });
+});
+
test("fresh connection metadata does not make an old live model list fresh", () => {
const source = snapshot();
source.liveUpdatedAt = { example: Date.now() - 11 * 60_000 };
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.ts b/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.ts
index 780475e0dc..49c06b8da8 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.ts
+++ b/jarvis/ui/web/frontend/src/components/society/chat/modelMenuSnapshot.ts
@@ -18,6 +18,7 @@ const snapshotSchema = z.object({
effort_levels: z.array(text), default_effort: text,
permission_modes: z.array(z.object({ id: text, label: text, description: text })),
default_permission_mode: text, cli_installed: z.boolean().nullable(), agent: text.optional(),
+ enabled: z.boolean().optional(), hidden_models: z.array(text).max(20_000).optional(),
})).max(100),
}),
connections: z.array(z.object({
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.accounts.test.tsx b/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.accounts.test.tsx
index 551b28d440..d731694b02 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.accounts.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.accounts.test.tsx
@@ -8,6 +8,28 @@ import { clearModelMenuSnapshot } from "./modelMenuSnapshot";
afterEach(() => { cleanup(); clearModelMenuSnapshot(); vi.unstubAllGlobals(); });
+test.each(["api-key", "subscription"])("%s access is not sent as a subscription login id", async (account) => {
+ clearModelMenuSnapshot();
+ const calls: string[] = [];
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => {
+ calls.push(url);
+ if (url.includes("/catalog")) return new Response(JSON.stringify({ providers: [{
+ id: "claude-api", family: "claude", label: "Claude", runner: "claude-cli", cli_installed: true,
+ curated_models: [{ id: "claude-sonnet-5", label: "Sonnet" }], models_source: "curated",
+ keyless: false, default_model: "", effort_levels: [], default_effort: "", permission_modes: [], default_permission_mode: "ask",
+ }] }));
+ if (url.includes("/status")) return new Response(JSON.stringify({ mapping: [{ jarvis: "claude-api", key_set: true }] }));
+ return new Response(JSON.stringify({ providers: [] }));
+ }));
+ const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
+ const wrapper = ({ children }: { children: ReactNode }) => <QueryClientProvider client={client}>{children}</QueryClientProvider>;
+ const { result } = renderHook(() => useModelMenuData(null, [], { "claude-api": account }), { wrapper });
+ await waitFor(() => expect(result.current.loading).toBe(false));
+ expect(result.current.options[0].curated_models[0].id).toBe("claude-sonnet-5");
+ expect(calls.filter((url) => url.includes("account_id"))).toEqual([]);
+ client.clear();
+});
+
test("changing a pinned account hides the previous models until that account replies", async () => {
clearModelMenuSnapshot();
let release!: () => void;
diff --git a/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.ts b/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.ts
index 910500362d..c57bfdd572 100644
--- a/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.ts
+++ b/jarvis/ui/web/frontend/src/components/society/chat/useModelMenuData.ts
@@ -5,17 +5,23 @@ import { fetchAgentChatCatalog, fetchAgentConnections, fetchProviderModels, type
import { fetchSocietyProviders } from "@/lib/societyApi";
import { joinProviderOptions } from "@/store/agentChat";
import { readModelMenuSnapshot, writeModelMenuSnapshot } from "./modelMenuSnapshot";
+import { API_KEY_ACCOUNT, SUBSCRIPTION_ACCOUNT } from "../create/seatChoice";
+import { useSavedHiddenModels } from "@/lib/agentProviderPrefs";
const EMPTY_CONNECTIONS: AgentConnectionRow[] = [];
const QUERY_POLICY = { staleTime: 60_000, gcTime: 30 * 60_000, refetchOnWindowFocus: false };
+const isLogin = (account: string) => Boolean(account) && account !== API_KEY_ACCOUNT && account !== SUBSCRIPTION_ACCOUNT;
/** Prepare once at the Agents view; the card reuses the same query instances. */
-export function useModelMenuData(chatCatalog: AgentChatCatalog | null = null, chatConnections = EMPTY_CONNECTIONS, accountIds: Record<string, string> = {}) {
+export function useModelMenuData(chatCatalog: AgentChatCatalog | null = null, chatConnections = EMPTY_CONNECTIONS,
+ accountIds: Record<string, string> = {}, behavior: { refreshOnMount?: boolean } = {}) {
const client = useQueryClient();
+ const savedHidden = useSavedHiddenModels((state) => state.hidden);
+ const refetchOnMount = behavior.refreshOnMount ? "always" as const : true;
const previousChatCatalog = useRef(chatCatalog);
const [saved] = useState(readModelMenuSnapshot);
const catalog = useQuery({ queryKey: ["agent-chat", "catalog", "society"], queryFn: () => fetchAgentChatCatalog("society"),
- initialData: saved?.catalog, initialDataUpdatedAt: saved?.savedAt, ...QUERY_POLICY });
+ initialData: saved?.catalog, initialDataUpdatedAt: saved?.savedAt, ...QUERY_POLICY, refetchOnMount });
const connections = useQuery({ queryKey: ["agent-chat", "connections"], queryFn: fetchAgentConnections,
initialData: saved?.connections, initialDataUpdatedAt: saved?.savedAt, placeholderData: chatCatalog ? chatConnections : undefined, ...QUERY_POLICY });
const providers = useQuery({ queryKey: ["society", "providers"], queryFn: fetchSocietyProviders,
@@ -26,7 +32,7 @@ export function useModelMenuData(chatCatalog: AgentChatCatalog | null = null, ch
const availableCatalog = catalog.data ?? fallbackCatalog;
const availableConnections = connections.data ?? (chatCatalog ? chatConnections : undefined);
const cliProviders = new Set((availableCatalog?.providers ?? []).filter((provider) => provider.runner.endsWith("-cli")).map((provider) => provider.id));
- const scopedIds = [...new Set(Object.entries(accountIds).filter(([provider, account]) => cliProviders.has(provider) && account).map(([, account]) => account))].sort();
+ const scopedIds = [...new Set(Object.entries(accountIds).filter(([provider, account]) => cliProviders.has(provider) && isLogin(account)).map(([, account]) => account))].sort();
const scopeKey = JSON.stringify(scopedIds);
const accountKey = JSON.stringify(accountIds);
const combineScoped = useCallback((queries: UseQueryResult<AgentChatCatalog, Error>[]) => ({
@@ -39,31 +45,34 @@ export function useModelMenuData(chatCatalog: AgentChatCatalog | null = null, ch
const scopedCatalogs = useQueries({ queries: scopedIds.map((accountId) => ({
queryKey: ["agent-chat", "catalog", "society", accountId],
queryFn: () => fetchAgentChatCatalog("society", { accountId }),
- ...QUERY_POLICY,
+ ...QUERY_POLICY, refetchOnMount,
})), combine: combineScoped });
const options = useMemo(() => {
const seats = JSON.parse(accountKey) as Record<string, string>;
// Never display a different login's models while a pinned seat is loading.
const rows = (availableCatalog?.providers ?? []).map((provider) => {
const accountId = seats[provider.id];
- if (!accountId || !provider.runner.endsWith("-cli")) return provider;
+ if (!isLogin(accountId) || !provider.runner.endsWith("-cli")) return provider;
return scopedCatalogs.data[accountId]?.providers.find((row) => row.id === provider.id)
?? { ...provider, curated_models: [] };
});
- return joinProviderOptions(rows, availableConnections ?? []);
- }, [availableCatalog, availableConnections, accountKey, scopedCatalogs.data]);
- const liveProviders = useMemo(() => options.filter((option) => option.connected && option.models_source === "live"), [options]);
+ // A settings save is newer than any in-flight or cached catalog response.
+ return joinProviderOptions(rows, availableConnections ?? []).map((option) => savedHidden
+ ? { ...option, hidden_models: savedHidden[option.id] ?? [] } : option);
+ }, [availableCatalog, availableConnections, accountKey, scopedCatalogs.data, savedHidden]);
+ const liveProviders = useMemo(() => options.filter((option) => option.connected && option.enabled !== false && option.models_source === "live"), [options]);
const combine = useCallback((queries: UseQueryResult<CuratedModel[], Error>[]) => ({
live: Object.fromEntries(queries.flatMap((query, index) => query.data ? [[liveProviders[index].id, query.data]] : [])) as Record<string, CuratedModel[]>,
updatedAt: Object.fromEntries(queries.map((query, index) => [liveProviders[index].id, query.dataUpdatedAt])),
fetching: queries.some((query) => query.isFetching),
+ errors: queries.flatMap((query, index) => query.isError ? [liveProviders[index].id] : []),
refresh: () => Promise.all(queries.map((query) => query.refetch())),
}), [liveProviders]);
const models = useQueries({ queries: liveProviders.map((provider) => ({
queryKey: ["society", "model-menu", "live", provider.id],
- queryFn: async (): Promise<CuratedModel[]> => (await fetchProviderModels(provider.id)).map((model) => ({ id: model.id, label: model.label ?? model.name ?? model.id })),
+ queryFn: async (): Promise<CuratedModel[]> => (await fetchProviderModels(provider.id, { strict: true })).map((model) => ({ id: model.id, label: model.label ?? model.name ?? model.id })),
initialData: saved?.live[provider.id], initialDataUpdatedAt: saved?.liveUpdatedAt?.[provider.id] ?? saved?.savedAt,
- enabled: Boolean(availableConnections), ...QUERY_POLICY, staleTime: 10 * 60_000, retry: false,
+ enabled: Boolean(availableConnections), ...QUERY_POLICY, staleTime: 10 * 60_000, retry: false, refetchOnMount,
})), combine });
useEffect(() => {
@@ -87,10 +96,11 @@ export function useModelMenuData(chatCatalog: AgentChatCatalog | null = null, ch
}, [catalog.data, availableConnections, providers.data, models.live, models.updatedAt, catalog.dataUpdatedAt, connections.dataUpdatedAt, providers.dataUpdatedAt]);
return {
- options, live: models.live, providers: providers.data,
+ options, live: models.live, providers: providers.data, modelErrors: models.errors,
loading: !availableCatalog || !availableConnections || scopedCatalogs.pending,
refreshing: scopedCatalogs.fetching || catalog.isFetching || connections.isFetching || providers.isFetching || models.fetching,
failed: scopedCatalogs.failed || (!availableCatalog && catalog.isError) || (!availableConnections && connections.isError),
+ refreshFailed: scopedCatalogs.failed || catalog.isError || connections.isError || providers.isError,
refresh: () => Promise.all([catalog.refetch(), connections.refetch(), providers.refetch(), models.refresh(), scopedCatalogs.refresh()]),
};
}
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/AgentAppearanceDialog.test.tsx b/jarvis/ui/web/frontend/src/components/society/companion/AgentAppearanceDialog.test.tsx
index d04c632740..b6bbe51ad7 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/AgentAppearanceDialog.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/companion/AgentAppearanceDialog.test.tsx
@@ -10,6 +10,10 @@ vi.mock("./CompanionPreview", () => ({ CompanionPreview: () => <div data-testid=
vi.mock("../figures/AgentFigureViewer", () => ({ AgentFigureViewer: () => <div data-testid="character-preview" /> }));
afterEach(() => { cleanup(); vi.unstubAllGlobals(); });
+const PETS = { active: "gigi", scale: 1, bubble: true, strip_always: false, visible: true, pets: ["gigi", "cocoa"].map(id => ({
+ id, name: id, description: "", builtin: true, frame_size: 48, animations: { idle: { row: 0, frames: 1, fps: 1, loop: true } }, sheet_url: `/api/pets/${id}/sheet.png`,
+})) };
+
function setup(fail = false, legacy = false) {
const original = SAMPLE_ROSTER.find(a => a.tier !== "lead")!;
const figure = legacy ? { contract: 1 as const, archetype: "biped" as const, base: "rogue", parts: {} } : original.figure!;
@@ -18,6 +22,7 @@ function setup(fail = false, legacy = false) {
// The look picker reads agent levels; an unreachable level system leaves every look open.
const fetcher = vi.fn(async (url: RequestInfo | URL) => String(url).startsWith("/api/progression")
? new Response("{}", { status: 404 })
+ : String(url).startsWith("/api/pets") ? new Response(JSON.stringify(PETS), { status: 200 })
: new Response(JSON.stringify({ agent }), { status: fail ? 500 : 200 }));
const saves = () => fetcher.mock.calls.filter(([url]) => String(url).startsWith("/api/society/agents/"));
vi.stubGlobal("fetch", fetcher);
@@ -79,3 +84,21 @@ it("saves a chosen hairstyle on the recipe without dropping its colours", async
expect(body.avatar.hairStyle).toBe("beanie");
expect(body.avatar.palette).toEqual(agent.figure?.palette);
});
+
+it("lets an agent wear a pet instead of its shape, and take the shape back", async () => {
+ const { saves } = setup();
+ fireEvent.click(screen.getByRole("button", { name: "society.companion.look_pet" }));
+ fireEvent.click(await screen.findByRole("button", { name: "cocoa" }));
+ expect(screen.queryByRole("button", { name: "society.companion.shapes.cloud" })).toBeNull();
+ // A pet brings its own colours: no colour picker beside it.
+ expect(screen.queryByLabelText("society.companion.custom_color")).toBeNull();
+ fireEvent.click(screen.getByRole("button", { name: "society.card.save" }));
+ await waitFor(() => expect(saves()).toHaveLength(1));
+ const worn = JSON.parse((saves()[0] as unknown as [string, RequestInit])[1].body as string);
+ expect(worn.avatar.companion).toMatchObject({ pet: "cocoa", shape: "circle" });
+ fireEvent.click(screen.getByRole("button", { name: "society.companion.look_shape" }));
+ fireEvent.click(screen.getByRole("button", { name: "society.card.save" }));
+ await waitFor(() => expect(saves()).toHaveLength(2));
+ const bare = JSON.parse((saves()[1] as unknown as [string, RequestInit])[1].body as string);
+ expect("pet" in bare.avatar.companion).toBe(false);
+});
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/AgentFollower.tsx b/jarvis/ui/web/frontend/src/components/society/companion/AgentFollower.tsx
index dfdbc53217..33c21caaa2 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/AgentFollower.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/companion/AgentFollower.tsx
@@ -9,6 +9,9 @@ import accessoryModels from "@/assets/society/companions/accessories.glb";
import { ACCESSORY_CATALOG, resolveFill, slotDepthM, wornAccessories } from "./accessories";
import { companionEyeColors, type CompanionAppearance } from "./appearance";
import { advancePetTrail, createPetTrail, petDisplayPosition, recordOwner, type PetTrail, type TrailPoint } from "./trail";
+import { useWornPet } from "./companionPetStore";
+import { companionFlies, type CompanionPet } from "./petCompanions";
+import { PetModel, VoxelPet, type PetDrive } from "./PetModel";
class PetBoundary extends Component<{ children: ReactNode }, { failed: boolean }> {
state = { failed: false };
@@ -17,8 +20,44 @@ class PetBoundary extends Component<{ children: ReactNode }, { failed: boolean }
render() { return this.state.failed ? null : this.props.children; }
}
+/** A flying pet worn by an agent hovers this high beside its owner, metres. */
+export const WORN_PET_LIFT_M = 0.32;
+
+/**
+ * An agent's companion in 3D: its shape, or the pet it wears instead
+ * (`companion.pet`) with that pet's own model and gait. While the pets
+ * answer is still on its way a worn pet draws nothing, so no shape flashes
+ * first; a pet this machine does not have falls back to the shape.
+ */
+export function CompanionModel({ appearance, lead = false, drive, paused = false }: {
+ appearance: CompanionAppearance; lead?: boolean;
+ /** How fast the owner moves, for a worn pet's gait; a still pet without it. */
+ drive?: { current: PetDrive }; paused?: boolean;
+}) {
+ const pet = useWornPet(lead ? undefined : appearance.pet);
+ if (pet === undefined) return null;
+ if (pet) return <WornPet pet={pet} drive={drive} paused={paused} />;
+ return <ShapeModel appearance={appearance} lead={lead} />;
+}
+
+function WornPet({ pet, drive, paused }: { pet: CompanionPet; drive?: { current: PetDrive }; paused: boolean }) {
+ const reduced = useReducedMotion() ?? false;
+ const still = useRef<PetDrive>({ speed: 0, mood: "idle" });
+ const lift = companionFlies(pet) ? WORN_PET_LIFT_M : 0;
+ return <group position={[0, lift, 0]}>
+ {pet.kind === "model" ? <PetModel pet={pet} drive={drive ?? still} reduced={reduced} paused={paused} />
+ : pet.kind === "voxel" ? <VoxelPet pet={pet} reduced={reduced} paused={paused} />
+ // Gigi worn by an agent: the lead's own hover model at companion size.
+ : <ShapeModel appearance={GIGI_LOOK} lead />}
+ </group>;
+}
+
+const GIGI_LOOK: CompanionAppearance = {
+ shape: "circle", color: "#ffcd61", eyes: "dots", enabled: true, accessories: {}, sizeM: 0.5, followDistanceM: 1,
+};
+
/** A cached authored mesh, with instance-owned materials and no extra canvas. */
-export function CompanionModel({ appearance, lead = false }: { appearance: CompanionAppearance; lead?: boolean }) {
+function ShapeModel({ appearance, lead = false }: { appearance: CompanionAppearance; lead?: boolean }) {
const { scene } = useGLTF(lead ? gigiModel : companionModels);
const instance = useMemo(() => {
const original = lead ? scene : scene.getObjectByName(appearance.shape);
@@ -161,6 +200,9 @@ export function AgentFollower({ owner, appearance, paused, lead = false, waypoin
const phase = useRef(0);
const reduced = useReducedMotion() ?? false;
const { invalidate } = useThree();
+ // A worn pet walks, waddles or flies on its own rig: it gets the owner's pace, not the shape's bob.
+ const worn = useWornPet(lead ? undefined : appearance.pet);
+ const drive = useRef<PetDrive>({ speed: 0, mood: "idle" });
useEffect(() => { motion.current = null; invalidate(); }, [appearance.enabled, invalidate]);
useFrame((_, delta) => {
if (!root.current || !owner.current) return;
@@ -178,13 +220,14 @@ export function AgentFollower({ owner, appearance, paused, lead = false, waypoin
if (waypoint) waypoint.current = null;
advancePetTrail(trail, appearance.followDistanceM, delta);
if (!reduced) phase.current += Math.min(delta, 0.1) * trail.speed * 9;
- const bob = reduced || trail.speed < 0.02 ? 0 : Math.abs(Math.sin(phase.current)) * 0.035;
+ drive.current.speed = trail.speed;
+ const bob = worn || reduced || trail.speed < 0.02 ? 0 : Math.abs(Math.sin(phase.current)) * 0.035;
const display = petDisplayPosition(trail.position, trail.yaw, appearance.sizeM, clear);
root.current.position.set(display[0], display[1] + 0.04 + bob, display[2]);
root.current.rotation.y = trail.points.length ? trail.yaw : owner.current.rotation.y;
if (trail.speed > 0.005) invalidate();
});
return <group ref={root} visible={false}>
- {appearance.enabled && <PetBoundary key={lead ? "gigi" : appearance.shape}><Suspense fallback={null}><CompanionModel appearance={appearance} lead={lead} /></Suspense></PetBoundary>}
+ {appearance.enabled && <PetBoundary key={lead ? "gigi" : appearance.pet ?? appearance.shape}><Suspense fallback={null}><CompanionModel appearance={appearance} lead={lead} drive={drive} paused={paused} /></Suspense></PetBoundary>}
</group>;
}
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/CompanionEditor.tsx b/jarvis/ui/web/frontend/src/components/society/companion/CompanionEditor.tsx
index f6e427631a..f3ac534520 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/CompanionEditor.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/companion/CompanionEditor.tsx
@@ -3,8 +3,11 @@ import { Switch } from "@/components/ui/switch";
import { AgentSymbol } from "../AgentSymbol";
import { AccessoryPicker } from "./AccessoryPicker";
import { COMPANION_COLORS, COMPANION_SHAPES, type CompanionAppearance } from "./appearance";
+import { CompanionMark } from "./CompanionMark";
+import { PetSprite } from "@/components/pets/PetSprite";
+import { usePets } from "@/hooks/usePets";
import gigiMark from "@/assets/gigi-companion-avatar.png";
-import { lazy, Suspense } from "react";
+import { lazy, Suspense, useEffect, useState } from "react";
const CompanionPreview = lazy(() => import("./CompanionPreview").then(m => ({ default: m.CompanionPreview })));
export function CompanionEditor({ value, onChange, disabled = false, lead = false, preview3d = true, agentId }: {
@@ -16,25 +19,49 @@ export function CompanionEditor({ value, onChange, disabled = false, lead = fals
}) {
const t = useT();
const update = (patch: Partial<CompanionAppearance>) => onChange({ ...value, ...patch });
+ const petsQuery = usePets();
+ const pets = petsQuery.data?.pets ?? [];
+ // The look is a shape or a pet; "Pet" stays chosen even before a pet exists to pick.
+ const [kind, setKind] = useState<"shape" | "pet">(value.pet ? "pet" : "shape");
+ const choose = (next: "shape" | "pet") => {
+ setKind(next);
+ if (next === "shape" && value.pet) update({ pet: undefined });
+ if (next === "pet" && !value.pet && pets[0]) update({ pet: pets[0].id });
+ };
+ // "Pet" chosen before the pets arrived: wear the first one as soon as they do.
+ const firstPet = pets[0]?.id;
+ useEffect(() => {
+ if (kind === "pet" && !value.pet && firstPet) onChange({ ...value, pet: firstPet });
+ }, [kind, value, firstPet, onChange]);
return <fieldset disabled={disabled} className="flex min-w-0 flex-col gap-5 p-4" data-testid="companion-editor">
<div className="flex items-center gap-5 rounded-xl bg-secondary/50 p-5">
- {lead ? <img src={gigiMark} width={88} height={88} alt="" /> : <AgentSymbol {...value} size={88} />}
+ {lead ? <img src={gigiMark} width={88} height={88} alt="" /> : <CompanionMark appearance={value} size={88} />}
<div><h3 className="font-medium">{t("society.companion.title")}</h3><p className="mt-1 text-sm text-muted-foreground">{t("society.companion.hint")}</p></div>
</div>
{preview3d && <Suspense fallback={null}><CompanionPreview appearance={value} lead={lead} /></Suspense>}
- {!lead && <><div><span className="mb-2 block text-sm font-medium">{t("society.companion.shape")}</span>
+ {!lead && <div className="flex items-center justify-between gap-3 text-sm"><span>{t("society.companion.look")}</span>
+ <div className="flex gap-1.5" role="group" aria-label={t("society.companion.look")}>
+ {(["shape", "pet"] as const).map(option => <button key={option} type="button" aria-pressed={kind === option} onClick={() => choose(option)}
+ className={`rounded-full border px-2.5 py-1 text-xs ${kind === option ? "border-transparent bg-primary text-primary-foreground" : "border-border bg-background text-foreground hover:bg-secondary"}`}>
+ {t(`society.companion.look_${option}`)}
+ </button>)}
+ </div>
+ </div>}
+ {!lead && kind === "pet" && <div data-testid="companion-pet-picker"><span className="mb-1 block text-sm font-medium">{t("society.companion.pet")}</span>
+ <p className="mb-2 text-xs text-muted-foreground">{pets.length || petsQuery.isLoading ? t("society.companion.pet_hint") : t("society.companion.no_pets")}</p>
+ <div className="flex flex-wrap gap-2">{pets.map(pet => <button key={pet.id} type="button" title={pet.name}
+ aria-label={pet.name} aria-pressed={value.pet === pet.id} onClick={() => update({ pet: pet.id })}
+ className={`grid h-14 w-14 place-items-center rounded-lg border focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring ${value.pet === pet.id ? "border-foreground bg-secondary" : "border-border hover:bg-secondary"}`}>
+ <PetSprite pet={pet} state={value.pet === pet.id ? "success" : "idle"} px={44} />
+ </button>)}</div>
+ </div>}
+ {!lead && kind === "shape" && <><div><span className="mb-2 block text-sm font-medium">{t("society.companion.shape")}</span>
<div className="flex flex-wrap gap-2">{COMPANION_SHAPES.map(shape => <button key={shape} type="button"
aria-label={t(`society.companion.shapes.${shape}`)} aria-pressed={value.shape === shape}
onClick={() => update({ shape })} className={`grid h-12 w-12 place-items-center rounded-lg border focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring ${value.shape === shape ? "border-foreground bg-secondary" : "border-border hover:bg-secondary"}`}>
<AgentSymbol shape={shape} color={value.color} eyes={value.eyes} accessories={value.accessories} size={36} />
</button>)}</div>
</div>
- <div><span className="mb-2 block text-sm font-medium">{t("society.companion.color")}</span>
- <div className="flex flex-wrap items-center gap-2">{COMPANION_COLORS.map(color => <button key={color} type="button" aria-label={`${t("society.companion.color")} ${color}`} aria-pressed={value.color === color}
- onClick={() => update({ color })} className={`h-8 w-8 rounded-full border-2 ${value.color === color ? "border-foreground ring-2 ring-background" : "border-transparent"}`} style={{ background: color }} />)}
- <input type="color" aria-label={t("society.companion.custom_color")} value={value.color} onChange={e => update({ color: e.target.value })} className="h-8 w-10 rounded border border-border bg-background" />
- </div>
- </div>
<div className="flex items-center justify-between gap-3 text-sm"><span>{t("society.companion.eyes")}</span>
{/* Plain toggles: a popup menu inside a modal dialog can strand its pointer lock. */}
<div className="flex gap-1.5" role="group" aria-label={t("society.companion.eyes")}>
@@ -46,6 +73,14 @@ export function CompanionEditor({ value, onChange, disabled = false, lead = fals
</div>
<AccessoryPicker value={value} onChange={accessories => update({ accessories })} agentId={agentId} />
</>}
+ {/* A shape is painted in the agent's colour; a pet brings its own colours. */}
+ {!lead && kind === "shape" && <div><span className="mb-2 block text-sm font-medium">{t("society.companion.color")}</span>
+ {/* Two rows across the full width: the swatches scale with the dialog instead of leaving a gap on the right. */}
+ <div className="grid w-full grid-cols-12 items-center gap-2">{COMPANION_COLORS.map(color => <button key={color} type="button" aria-label={`${t("society.companion.color")} ${color}`} aria-pressed={value.color === color}
+ onClick={() => update({ color })} className={`aspect-square w-full rounded-full border border-border ring-offset-2 ring-offset-popover focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring ${value.color === color ? "ring-2 ring-foreground" : ""}`} style={{ background: color }} />)}
+ <input type="color" aria-label={t("society.companion.custom_color")} value={value.color} onChange={e => update({ color: e.target.value })} className="aspect-square h-auto w-full rounded border border-border bg-background" />
+ </div>
+ </div>}
<label className="flex items-center justify-between gap-3 text-sm">{t("society.companion.visible")}
<Switch checked={value.enabled} onCheckedChange={enabled => update({ enabled })} aria-label={t("society.companion.visible")} />
</label>
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/CompanionMark.test.tsx b/jarvis/ui/web/frontend/src/components/society/companion/CompanionMark.test.tsx
new file mode 100644
index 0000000000..3ce33404e3
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/companion/CompanionMark.test.tsx
@@ -0,0 +1,65 @@
+import { cleanup, render, screen } from "@testing-library/react";
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { afterEach, describe, expect, it } from "vitest";
+
+import { PetStageContext } from "@/components/pets/petStage";
+import { petKeys } from "@/hooks/usePets";
+import type { Pet, PetsState } from "@/lib/petsApi";
+import { CompanionMark } from "./CompanionMark";
+import { defaultCompanion, type CompanionAppearance } from "./appearance";
+
+function pet(id: string): Pet {
+ return {
+ id, name: id, description: "", builtin: true, frame_size: 48,
+ animations: { idle: { row: 0, frames: 1, fps: 1, loop: true }, searching: { row: 8, frames: 1, fps: 1, loop: true } },
+ sheet_url: `/api/pets/${id}/sheet.png`,
+ };
+}
+
+const PETS: PetsState = { active: "gigi", scale: 1, bubble: true, strip_always: false, visible: true, pets: [pet("gigi"), pet("cocoa")] };
+const look = (patch: Partial<CompanionAppearance> = {}): CompanionAppearance => ({ ...defaultCompanion("agent"), shape: "drop", ...patch });
+
+/** A client already holding the pets answer (or none yet), so nothing is fetched. */
+function renderWith(ui: React.ReactElement, state: PetsState | null = PETS) {
+ const client = new QueryClient({ defaultOptions: { queries: { retry: false, enabled: state !== null } } });
+ if (state) client.setQueryData(petKeys.all, state);
+ return render(<QueryClientProvider client={client}>{ui}</QueryClientProvider>);
+}
+
+afterEach(cleanup);
+
+describe("CompanionMark", () => {
+ it("draws the shape for an agent that wears no pet", () => {
+ const { container } = renderWith(<CompanionMark appearance={look()} size={30} />);
+ expect(container.querySelector("svg")).not.toBeNull();
+ expect(screen.queryByTestId("pet-sprite")).toBeNull();
+ });
+
+ it("draws the worn pet instead of the shape, idle at rest and working while the agent works", () => {
+ const { container, rerender } = renderWith(<CompanionMark appearance={look({ pet: "cocoa" })} size={30} />);
+ expect(container.querySelector("[data-agent-pet='cocoa']")).not.toBeNull();
+ expect(screen.getByTestId("pet-sprite").dataset.state).toBe("idle");
+ const client = new QueryClient();
+ client.setQueryData(petKeys.all, PETS);
+ rerender(<QueryClientProvider client={client}><CompanionMark appearance={look({ pet: "cocoa" })} size={30} thinking /></QueryClientProvider>);
+ expect(screen.getByTestId("pet-sprite").dataset.state).toBe("working");
+ });
+
+ it("plays the row a surrounding stage asks for", () => {
+ renderWith(<PetStageContext.Provider value="searching"><CompanionMark appearance={look({ pet: "cocoa" })} size={30} /></PetStageContext.Provider>);
+ expect(screen.getByTestId("pet-sprite").dataset.state).toBe("searching");
+ });
+
+ it("keeps an empty box while the pets load, so no shape flashes first", () => {
+ const { container } = renderWith(<CompanionMark appearance={look({ pet: "cocoa" })} size={30} />, null);
+ expect(screen.queryByTestId("pet-sprite")).toBeNull();
+ expect(container.querySelector("svg")).toBeNull();
+ expect((container.firstElementChild as HTMLElement).style.width).toBe("30px");
+ });
+
+ it("falls back to the shape for a pet this machine does not have", () => {
+ const { container } = renderWith(<CompanionMark appearance={look({ pet: "u0123456789abcdef" })} size={30} />);
+ expect(screen.queryByTestId("pet-sprite")).toBeNull();
+ expect(container.querySelector("svg")).not.toBeNull();
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/CompanionMark.tsx b/jarvis/ui/web/frontend/src/components/society/companion/CompanionMark.tsx
new file mode 100644
index 0000000000..6f2640e0cb
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/companion/CompanionMark.tsx
@@ -0,0 +1,49 @@
+import { useContext } from "react";
+import { QueryClientContext } from "@tanstack/react-query";
+
+import { PetSprite } from "@/components/pets/PetSprite";
+import { PetStageContext } from "@/components/pets/petStage";
+import { usePetById } from "@/hooks/usePets";
+import type { PetState } from "@/lib/petStates";
+import { AgentSymbol } from "../AgentSymbol";
+import type { CompanionAppearance } from "./appearance";
+
+type MarkProps = {
+ appearance: CompanionAppearance;
+ size: number;
+ /** The agent is at work: the shape shows thinking dots, a pet plays its working row. */
+ thinking?: boolean;
+ /** Eyes a stage can animate (shapes only; a pet acts through its own rows). */
+ expressive?: boolean;
+ /** Fixes the pet's row; otherwise a surrounding stage or `thinking` decides. */
+ state?: PetState;
+};
+
+/**
+ * An agent's face in 2D: its shape symbol, or the pet it wears instead
+ * (`companion.pet`, chosen in the look dialog from My Pets). Every surface
+ * that draws an agent goes through here, so choosing a pet changes the
+ * roster, the chats, the bot stage and the level hall at once.
+ *
+ * A pet face reads the shared pets answer. Until it arrives the box stays
+ * empty at its final size, so no shape flashes before the pet; a pet that
+ * does not exist here (deleted, or drawn on another machine) shows the shape.
+ */
+export function CompanionMark(props: MarkProps) {
+ const { appearance, size, thinking = false, expressive = false } = props;
+ const shape = <AgentSymbol {...appearance} size={size} thinking={thinking} expressive={expressive} />;
+ // Outside a query client (an isolated render, a bare preview) there is no catalog to read.
+ const client = useContext(QueryClientContext);
+ if (!appearance.pet || !client) return shape;
+ return <PetFace {...props} petId={appearance.pet} fallback={shape} />;
+}
+
+function PetFace({ petId, size, thinking = false, state, fallback }: MarkProps & { petId: string; fallback: JSX.Element }) {
+ const pet = usePetById(petId);
+ const staged = useContext(PetStageContext);
+ if (pet === undefined) return <span aria-hidden className="inline-block shrink-0" style={{ width: size, height: size }} />;
+ if (pet === null) return fallback;
+ return <span className="agent-pet-face inline-flex shrink-0" data-agent-pet={pet.id}>
+ <PetSprite pet={pet} state={state ?? staged ?? (thinking ? "working" : "idle")} px={size} />
+ </span>;
+}
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/CompanionPreview.tsx b/jarvis/ui/web/frontend/src/components/society/companion/CompanionPreview.tsx
index 40831fa375..f8efd3c515 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/CompanionPreview.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/companion/CompanionPreview.tsx
@@ -1,11 +1,14 @@
-import { Component, Suspense, useRef, type ReactNode } from "react";
-import { Canvas } from "@react-three/fiber";
-import { OrbitControls } from "@react-three/drei";
+import { Component, Suspense, useEffect, useRef, type ReactNode } from "react";
+import { Canvas, useThree } from "@react-three/fiber";
+import { ContactShadows, Environment, Lightformer, OrbitControls } from "@react-three/drei";
+import { useReducedMotion } from "framer-motion";
import { useCanvasAwake } from "@/hooks/useCanvasAwake";
import { useWebglSurface } from "@/hooks/useWebglSurface";
import { useWebglSupported } from "@/lib/graphDimension";
-import { CompanionModel } from "./AgentFollower";
+import { CompanionModel, WORN_PET_LIFT_M } from "./AgentFollower";
import type { CompanionAppearance } from "./appearance";
+import { useSyncCompanionPet, useWornPet } from "./companionPetStore";
+import { companionFlies } from "./petCompanions";
class PreviewBoundary extends Component<{ children: ReactNode }, { failed: boolean }> {
state = { failed: false };
@@ -14,16 +17,92 @@ class PreviewBoundary extends Component<{ children: ReactNode }, { failed: boole
render() { return this.state.failed ? null : this.props.children; }
}
+/** Vertical field of view of the preview camera, degrees. */
+const FOV = 30;
+/** How much taller than the body the frame is, so it never touches the edges. */
+const MARGIN = 1.55;
+
+/**
+ * Where the camera looks and how far back it stands, so the body fills the
+ * frame whatever its size: a 0.5 m shape, a 0.28 m blob or a cat that walks
+ * on the floor. Every companion knows its own height, so no bounds have to
+ * be measured after the model loads.
+ */
+export function previewFraming(heightM: number, liftM: number): { target: [number, number, number]; position: [number, number, number] } {
+ const centre = liftM + heightM / 2;
+ const distance = (heightM * MARGIN) / 2 / Math.tan((FOV * Math.PI) / 360);
+ // A three-quarter view, slightly from above, the way a figure sits on a desk.
+ return { target: [0, centre, 0], position: [distance * 0.5, centre + distance * 0.24, distance * 0.84] };
+}
+
+/**
+ * A soft studio around the body: an environment lit only by local light
+ * panels (no image is downloaded), a warm key that casts the shadow, a cool
+ * rim that separates the silhouette from the dark dialog, and a contact
+ * shadow that sets it on the ground.
+ */
+function Studio() {
+ return <>
+ <Environment resolution={256} frames={1}>
+ <Lightformer form="rect" intensity={2.2} position={[2.5, 3, 2.5]} scale={[3, 3, 1]} target={[0, 0, 0]} />
+ <Lightformer form="rect" intensity={0.9} color="#cfe0ff" position={[-3, 1.5, 1]} scale={[2, 3, 1]} target={[0, 0, 0]} />
+ <Lightformer form="ring" intensity={1.4} position={[0, 2, -3]} scale={2} target={[0, 0, 0]} />
+ </Environment>
+ <ambientLight intensity={0.25} />
+ <directionalLight position={[1.6, 2.6, 1.8]} intensity={1.7} color="#fff4e6" castShadow
+ shadow-mapSize={[1024, 1024]} shadow-bias={-0.0004} />
+ <directionalLight position={[-1.8, 1.4, -2.2]} intensity={1.1} color="#b9d4ff" />
+ </>;
+}
+
+type Framing = ReturnType<typeof previewFraming>;
+
+/**
+ * Moves the camera when the body changes. The canvas itself stays: building
+ * a new one per pet would drop the old WebGL context, which the surface
+ * reads as a lost GPU and stops drawing.
+ */
+function Frame({ framing }: { framing: Framing }) {
+ const camera = useThree((state) => state.camera);
+ const controls = useThree((state) => state.controls) as { target: { set: (x: number, y: number, z: number) => void }; update: () => void } | null;
+ const invalidate = useThree((state) => state.invalidate);
+ const [px, py, pz] = framing.position;
+ const [tx, ty, tz] = framing.target;
+ useEffect(() => {
+ camera.position.set(px, py, pz);
+ camera.lookAt(tx, ty, tz);
+ controls?.target.set(tx, ty, tz);
+ controls?.update();
+ invalidate();
+ }, [camera, controls, invalidate, px, py, pz, tx, ty, tz]);
+ return null;
+}
+
export function CompanionPreview({ appearance, lead = false }: { appearance: CompanionAppearance; lead?: boolean }) {
const host = useRef<HTMLDivElement>(null);
const { generation } = useWebglSurface(host);
const awake = useCanvasAwake(host);
const supported = useWebglSupported();
- return <div ref={host} className="h-48 w-full touch-none" data-testid="companion-preview">
- {supported && awake && <PreviewBoundary key={generation}><Canvas frameloop="demand" dpr={[1, 1.5]} camera={{ position: [0.75, 0.65, 1.8], fov: 38 }} gl={{ alpha: true, antialias: true }}>
- <ambientLight intensity={1.6} /><directionalLight position={[2, 3, 4]} intensity={2.6} />
- <Suspense fallback={null}><CompanionModel appearance={appearance} lead={lead} /></Suspense>
- <OrbitControls target={[0, appearance.sizeM / 2, 0]} enablePan={false} enableZoom={false} minPolarAngle={0.25} maxPolarAngle={Math.PI * 0.65} />
- </Canvas></PreviewBoundary>}
+ const reduced = useReducedMotion() ?? false;
+ // A worn pet's model comes from the pets catalog, which the canvas cannot query itself.
+ useSyncCompanionPet();
+ const pet = useWornPet(lead ? undefined : appearance.pet);
+ const framing = pet
+ ? previewFraming(pet.heightM, companionFlies(pet) ? WORN_PET_LIFT_M : 0)
+ : previewFraming(appearance.sizeM, 0);
+ // A pet breathes, blinks and turns on the table; a still preview only draws when asked.
+ const live = !!pet && !reduced;
+ return <div ref={host} className="h-56 w-full touch-none" data-testid="companion-preview">
+ {supported && awake && <PreviewBoundary key={generation}>
+ <Canvas frameloop={live ? "always" : "demand"} shadows dpr={[1, 2]}
+ camera={{ position: framing.position, fov: FOV, near: 0.02, far: 20 }} gl={{ alpha: true, antialias: true }}>
+ <Studio />
+ <Frame framing={framing} />
+ <Suspense fallback={null}><CompanionModel appearance={appearance} lead={lead} /></Suspense>
+ <ContactShadows position={[0, 0.001, 0]} opacity={0.5} scale={Math.max(1, framing.target[1] * 6)} blur={2.6} far={1} resolution={512} frames={live ? Infinity : 1} />
+ <OrbitControls makeDefault target={framing.target} enablePan={false} enableZoom={false} autoRotate={live} autoRotateSpeed={0.9}
+ minPolarAngle={0.25} maxPolarAngle={Math.PI * 0.55} />
+ </Canvas>
+ </PreviewBoundary>}
</div>;
}
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/PetModel.tsx b/jarvis/ui/web/frontend/src/components/society/companion/PetModel.tsx
index 42bf4a1b7d..b1a5ff686c 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/PetModel.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/companion/PetModel.tsx
@@ -6,10 +6,12 @@
*/
import { useEffect, useMemo, useRef, useState } from "react";
import { useGLTF } from "@react-three/drei";
-import { useFrame } from "@react-three/fiber";
+import { useFrame, useThree } from "@react-three/fiber";
import {
- Box3, BoxGeometry, Color, InstancedMesh, Matrix4, MeshStandardMaterial, Object3D, Quaternion, Vector3, type Group,
+ Box3, BoxGeometry, Color, InstancedMesh, Matrix4, MeshStandardMaterial, Object3D, PMREMGenerator, Quaternion, Vector3,
+ type Group, type Material, type Mesh, type Texture, type WebGLRenderer,
} from "three";
+import { RoomEnvironment } from "three/examples/jsm/environments/RoomEnvironment.js";
import type { CompanionPet } from "./petCompanions";
import { createRigState, rigPose, stepRig, type PartMotion, type PetMood } from "./petRig";
import { voxelFrames, type VoxelFrames } from "./voxelPet";
@@ -32,6 +34,47 @@ export function petModelUrl(id: string): string | null {
return null;
}
+/** One soft studio reflection per renderer, built locally (no image is downloaded). */
+const STUDIO = new WeakMap<WebGLRenderer, Texture>();
+
+function studioReflection(gl: WebGLRenderer): Texture {
+ let texture = STUDIO.get(gl);
+ if (!texture) {
+ const pmrem = new PMREMGenerator(gl);
+ const room = new RoomEnvironment();
+ texture = pmrem.fromScene(room, 0.04).texture;
+ room.dispose();
+ pmrem.dispose();
+ STUDIO.set(gl, texture);
+ }
+ return texture;
+}
+
+/**
+ * The pet's own materials, so a reflection given here never leaks into the
+ * cached model another canvas shares. Where the scene has no environment
+ * (the Verse), the vinyl gets a soft studio reflection: without one a
+ * glossy toy reads as flat colour under the room's direct lights.
+ */
+function ownMaterials(model: Object3D, reflection: Texture | null): Material[] {
+ const owned: Material[] = [];
+ model.traverse((object) => {
+ const mesh = object as Mesh;
+ if (!mesh.isMesh) return;
+ const swap = (material: Material) => {
+ const own = material.clone();
+ if (reflection && own instanceof MeshStandardMaterial && !own.envMap) {
+ own.envMap = reflection;
+ own.envMapIntensity = 0.7;
+ }
+ owned.push(own);
+ return own;
+ };
+ mesh.material = Array.isArray(mesh.material) ? mesh.material.map(swap) : swap(mesh.material);
+ });
+ return owned;
+}
+
interface RiggedPart {
object: Object3D;
position: Vector3;
@@ -53,8 +96,11 @@ export function PetModel({ pet, drive, reduced, paused }: {
const url = petModelUrl(pet.id);
if (!url) throw new Error(`No 3D model for pet ${pet.id}`);
const { scene } = useGLTF(url);
+ const gl = useThree((state) => state.gl);
+ const lit = useThree((state) => !!state.scene.environment);
const instance = useMemo(() => {
const model = scene.clone(true);
+ const materials = ownMaterials(model, lit ? null : studioReflection(gl));
const parts = new Map<string, RiggedPart>();
model.traverse((object) => {
if ((object as { isMesh?: boolean }).isMesh) object.castShadow = true;
@@ -79,8 +125,9 @@ export function PetModel({ pet, drive, reduced, paused }: {
groundOffset.set(-groundOffset.x, 0, -groundOffset.z);
}
}
- return { model, parts, groundOffset };
- }, [scene, pet.id]);
+ return { model, parts, groundOffset, materials };
+ }, [scene, pet.id, gl, lit]);
+ useEffect(() => () => { instance.materials.forEach((material) => material.dispose()); }, [instance]);
const rig = useRef(createRigState());
const pose = useMemo(() => new Map<string, PartMotion>(), []);
useEffect(() => { pose.clear(); }, [pet.id, pose]);
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/appearance.ts b/jarvis/ui/web/frontend/src/components/society/companion/appearance.ts
index f988671454..465cf17dfb 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/appearance.ts
+++ b/jarvis/ui/web/frontend/src/components/society/companion/appearance.ts
@@ -3,14 +3,26 @@ import { ACCESSORY_SLOTS, type AccessoryChoice } from "./accessories";
export const COMPANION_SHAPES = ["circle", "squircle", "pill", "triangle", "hexagon", "cloud", "drop"] as const;
export type SymbolShape = typeof COMPANION_SHAPES[number];
-export const COMPANION_COLORS = ["#8b5cf6", "#c5dfd4", "#f2a65a", "#7ab6ef", "#ed91aa", "#b7cb78", "#bba7ed", "#79c7c4"] as const;
+export const COMPANION_COLORS = [
+ "#ffffff", "#875b36", "#dd2233", "#ff6801", "#ff9700", "#029858",
+ "#00a591", "#1174da", "#804ee1", "#df2a87", "#777777",
+ "#e8c89a", "#ff7a6b", "#ffd23f", "#9ccc3a", "#3cc4e8", "#b59cf0",
+ "#222222", "#8c1c3a", "#6b7a1f", "#0f5c63", "#1f3a8a", "#f49ac1",
+] as const;
+/** Default looks hash over the first colours only, so a longer palette never recolours an agent. */
+const DEFAULT_COLOR_COUNT = 11;
export const COMPANION_SIZE_M = 0.5;
export const COMPANION_FOLLOW_DISTANCE_M = 1;
+/** `PetId` in `jarvis/society/companion.py` (built-in slug or a drawn pet's `u<hex>`). */
+export const PET_ID = /^[a-z0-9][a-z0-9-]{0,31}$/;
export const companionSchema = z.object({
shape: z.enum(COMPANION_SHAPES), color: z.string().regex(/^#[0-9a-fA-F]{6}$/),
eyes: z.enum(["dots", "lines"]).default("lines"), enabled: z.boolean().default(true),
// One item id per slot. Ids stay open so a newer catalog never invalidates a saved look.
accessories: z.record(z.enum(ACCESSORY_SLOTS), z.string().regex(/^[a-z0-9_]{1,40}$/)).default({}),
+ // A pet from My Pets worn instead of the shape. Open like accessory ids: a pet
+ // missing on this machine shows the shape, it never invalidates the look.
+ pet: z.string().regex(PET_ID).optional(),
// Accept earlier saved slider values, but every presentation uses one scale.
sizeM: z.number().finite().min(0.25).max(0.8).default(COMPANION_SIZE_M).transform(() => COMPANION_SIZE_M),
followDistanceM: z.number().finite().min(0.5).max(2).default(COMPANION_FOLLOW_DISTANCE_M).transform(() => COMPANION_FOLLOW_DISTANCE_M),
@@ -26,7 +38,7 @@ function identityHash(identity: string): number {
export function defaultCompanion(identity: string): CompanionAppearance {
return {
shape: COMPANION_SHAPES[identityHash(`shape:${identity}`) % COMPANION_SHAPES.length]!,
- color: COMPANION_COLORS[identityHash(`color:${identity}`) % COMPANION_COLORS.length]!,
+ color: COMPANION_COLORS[identityHash(`color:${identity}`) % DEFAULT_COLOR_COUNT]!,
eyes: "lines", enabled: true, accessories: {}, sizeM: COMPANION_SIZE_M, followDistanceM: COMPANION_FOLLOW_DISTANCE_M,
};
}
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.test.ts b/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.test.ts
new file mode 100644
index 0000000000..42bc2e82f5
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.test.ts
@@ -0,0 +1,30 @@
+import { afterEach, describe, expect, it } from "vitest";
+import { useCompanionPet } from "./companionPetStore";
+import { BUILTIN_COMPANIONS, GIGI_COMPANION, type CompanionPet } from "./petCompanions";
+
+const cocoa: CompanionPet = { id: "cocoa", ...BUILTIN_COMPANIONS.cocoa! };
+const worn = (id: string | undefined) => {
+ const { catalog } = useCompanionPet.getState();
+ return !id ? null : catalog === null ? undefined : catalog[id] ?? null;
+};
+
+afterEach(() => useCompanionPet.setState({ pet: GIGI_COMPANION, catalog: null }));
+
+describe("the shared pet catalog an agent's companion reads", () => {
+ it("knows nothing before the pets answer, then every pet by id", () => {
+ expect(worn("cocoa")).toBeUndefined();
+ useCompanionPet.getState().setCatalog({ cocoa, gigi: GIGI_COMPANION });
+ expect(worn("cocoa")).toBe(cocoa);
+ expect(worn("u0123456789abcdef")).toBeNull();
+ expect(worn(undefined)).toBeNull();
+ });
+
+ it("keeps an unchanged pet's object on a refetch, so its model never reloads", () => {
+ useCompanionPet.getState().setCatalog({ cocoa });
+ const before = useCompanionPet.getState().catalog;
+ useCompanionPet.getState().setCatalog({ cocoa: { ...cocoa } });
+ expect(useCompanionPet.getState().catalog).toBe(before);
+ useCompanionPet.getState().setCatalog({ cocoa: { ...cocoa }, gigi: GIGI_COMPANION });
+ expect(useCompanionPet.getState().catalog?.cocoa).toBe(cocoa);
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.ts b/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.ts
index 68530cd15e..2e43023071 100644
--- a/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.ts
+++ b/jarvis/ui/web/frontend/src/components/society/companion/companionPetStore.ts
@@ -1,18 +1,24 @@
/**
- * The companion the Jarvis Verse draws for Jarvis, shared by every place in
- * the scene that shows it. The office stage feeds it from the pets query
- * (outside the WebGL canvas); the flyer and the lead walker read it inside.
- * The object only changes when the pet does, so a refetch never reloads a
- * model or a sprite sheet.
+ * The pets the Jarvis Verse draws, shared by every place in the scene that
+ * shows one: Jarvis's own companion (the pet chosen in My Pets) and the pets
+ * agents wear instead of their shape (`companion.pet`). The stages feed it
+ * from the pets query outside the WebGL canvas; the flyer, the lead walker
+ * and every agent follower read it inside. An entry only changes when its
+ * pet does, so a refetch never reloads a model or a sprite sheet.
*/
import { useEffect } from "react";
+import { useQuery } from "@tanstack/react-query";
import { create } from "zustand";
-import { useActivePet } from "@/hooks/usePets";
+import { pickActivePet, petKeys } from "@/hooks/usePets";
+import { fetchPets } from "@/lib/petsApi";
import { companionFor, GIGI_COMPANION, type CompanionPet } from "./petCompanions";
interface CompanionPetState {
pet: CompanionPet;
+ /** Every pet on this machine by id, as a companion; null until the pets answer arrives. */
+ catalog: Readonly<Record<string, CompanionPet>> | null;
setPet: (pet: CompanionPet) => void;
+ setCatalog: (catalog: Record<string, CompanionPet>) => void;
}
export function sameCompanion(a: CompanionPet, b: CompanionPet): boolean {
@@ -22,13 +28,47 @@ export function sameCompanion(a: CompanionPet, b: CompanionPet): boolean {
export const useCompanionPet = create<CompanionPetState>((set, get) => ({
pet: GIGI_COMPANION,
+ catalog: null,
setPet: (pet) => { if (!sameCompanion(get().pet, pet)) set({ pet }); },
+ setCatalog: (next) => {
+ const current = get().catalog;
+ // Keep each unchanged entry's object, so a worn pet's model never reloads on a refetch.
+ const merged: Record<string, CompanionPet> = {};
+ let changed = !current || Object.keys(current).length !== Object.keys(next).length;
+ for (const [id, pet] of Object.entries(next)) {
+ const kept = current?.[id];
+ merged[id] = kept && sameCompanion(kept, pet) ? kept : pet;
+ if (merged[id] !== kept) changed = true;
+ }
+ if (changed) set({ catalog: merged });
+ },
}));
-/** Keeps the shared companion in step with the pet chosen in My Pets. */
+/**
+ * The companion an agent wears instead of its shape: undefined while the
+ * pets answer is still on its way (draw nothing yet), null when it wears no
+ * pet or this machine has no such pet (draw the shape).
+ */
+export function useWornPet(petId: string | undefined): CompanionPet | null | undefined {
+ return useCompanionPet((s) => !petId ? null : s.catalog === null ? undefined : s.catalog[petId] ?? null);
+}
+
+/** Keeps the shared companions in step with My Pets: Jarvis's own, and every pet an agent may wear. */
export function useSyncCompanionPet(): void {
- const active = useActivePet();
+ const { data } = useQuery({ queryKey: petKeys.all, queryFn: fetchPets, staleTime: Infinity });
const setPet = useCompanionPet((s) => s.setPet);
+ const setCatalog = useCompanionPet((s) => s.setCatalog);
// Until the pets answer arrives the last known companion stays (Gigi on a cold start).
- useEffect(() => { if (active) setPet(companionFor(active)); }, [active, setPet]);
+ useEffect(() => {
+ if (!data) return;
+ const active = pickActivePet(data);
+ if (active) setPet(companionFor(active));
+ const catalog: Record<string, CompanionPet> = {};
+ for (const pet of Array.isArray(data.pets) ? data.pets : []) {
+ // A drawn pet without a usable sheet would come back as Gigi; leave it out so the agent keeps its shape.
+ const companion = companionFor(pet);
+ if (companion.id === pet.id) catalog[pet.id] = companion;
+ }
+ setCatalog(catalog);
+ }, [data, setPet, setCatalog]);
}
diff --git a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.catalog.test.tsx b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.catalog.test.tsx
new file mode 100644
index 0000000000..1d03637c89
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.catalog.test.tsx
@@ -0,0 +1,158 @@
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, beforeEach, expect, test, vi } from "vitest";
+import { loadLocaleChunk } from "@/i18n";
+import type { AgentChatProvider } from "@/lib/agentChatApi";
+import { useSavedHiddenModels } from "@/lib/agentProviderPrefs";
+import { MODEL_ACCESS_KEY } from "@/lib/modelAccess";
+import { clearModelMenuSnapshot, writeModelMenuSnapshot } from "../chat/modelMenuSnapshot";
+import { CreateAgentDialogHost } from "./CreateAgentDialog";
+import { useCreateAgentDialog } from "./createAgentStore";
+
+vi.mock("../companion/CompanionEditor", () => ({ CompanionEditor: () => <div>Companion preview</div> }));
+const { createAgent } = vi.hoisted(() => ({ createAgent: vi.fn() }));
+vi.mock("../data", () => ({ useCreateSocietyAgent: () => createAgent, AgentNameTaken: class extends Error {} }));
+
+const fallback: AgentChatProvider = {
+ id: "openai-codex", label: "OpenAI Codex", family: "openai", runner: "codex-cli", cli_installed: true,
+ models_source: "curated", default_model: "", keyless: false, native_resume: true,
+ effort_levels: ["medium"], default_effort: "medium", permission_modes: [], default_permission_mode: "ask",
+ hidden_models: ["gpt-5.6-sol"], enabled: true,
+ curated_models: [{ id: "gpt-5.6-sol", label: "GPT-5.6-Sol" }, { id: "gpt-5.2", label: "GPT-5.2" }],
+};
+const current = { ...fallback, curated_models: [
+ { id: "gpt-6.1-sol", label: "GPT-6.1-Sol" }, { id: "gpt-6-astra", label: "GPT-6-Astra" },
+ { id: "gpt-6-luna", label: "GPT-6-Luna" }, { id: "gpt-5.6-sol", label: "GPT-5.6-Sol" },
+] };
+const connections = [{ jarvis: "openai-codex", key_set: true, is_active_brain: false }];
+const providers = [{ id: "openai-codex", label: "Codex", family: "openai", runner: "codex-cli", subscription: true,
+ keyless: false, platform: null, accounts: [] }];
+const catalog = (row: AgentChatProvider) => ({ providers: [row], default_cwd: "", shell: "" });
+let client: QueryClient;
+
+beforeEach(async () => {
+ createAgent.mockReset(); createAgent.mockResolvedValue({ agentId: "created" });
+ clearModelMenuSnapshot();
+ localStorage.removeItem(MODEL_ACCESS_KEY);
+ useSavedHiddenModels.setState({ hidden: null });
+ await loadLocaleChunk("society");
+ client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
+});
+afterEach(() => {
+ cleanup(); client.clear(); clearModelMenuSnapshot(); vi.unstubAllGlobals();
+ useSavedHiddenModels.setState({ hidden: null });
+ act(() => useCreateAgentDialog.getState().cancel());
+});
+
+function openDialog() {
+ render(<QueryClientProvider client={client}><CreateAgentDialogHost /></QueryClientProvider>);
+ act(() => { void useCreateAgentDialog.getState().request().catch(() => undefined); });
+}
+
+test("opening replaces a fresh cached GPT-5.2 fallback with the enabled live subscription models", async () => {
+ writeModelMenuSnapshot({ version: 1, savedAt: Date.now(), catalog: catalog(fallback), connections, providers, live: {} });
+ let release!: () => void;
+ const pending = new Promise<void>((resolve) => { release = resolve; });
+ const requests: string[] = [];
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => {
+ requests.push(url);
+ if (url.includes("/catalog")) { await pending; return new Response(JSON.stringify(catalog(current))); }
+ if (url.includes("/agent-runtimes")) return new Response(JSON.stringify({ runtimes: [], supported_providers: [] }));
+ if (url.includes("/status")) return new Response(JSON.stringify({ mapping: connections }));
+ if (url.endsWith("/providers")) return new Response(JSON.stringify({ providers }));
+ throw new Error(`Unexpected request: ${url}`);
+ }));
+ openDialog();
+ expect(screen.getByTestId("create-agent-model").textContent).toContain("GPT-5.2");
+ expect(screen.getByRole("status").textContent).toContain("Loading");
+ await act(async () => { release(); await pending; });
+ await waitFor(() => expect(screen.getByTestId("create-agent-model").textContent).toContain("GPT-6.1-Sol"));
+ fireEvent.click(screen.getByTestId("create-agent-model"));
+ expect(screen.getByRole("option", { name: "GPT-6-Astra" })).toBeTruthy();
+ expect(screen.getByRole("option", { name: "GPT-6-Luna" })).toBeTruthy();
+ expect(screen.queryByRole("option", { name: "GPT-5.2" })).toBeNull();
+ expect(screen.queryByRole("option", { name: "GPT-5.6-Sol" })).toBeNull();
+ expect(requests.filter((url) => url.includes("/catalog"))).toHaveLength(1);
+ // A just-saved visibility change wins even if a cached catalog still has the old list.
+ act(() => useSavedHiddenModels.setState({ hidden: { "openai-codex": ["gpt-6-astra", "gpt-5.6-sol"] } }));
+ expect(screen.queryByRole("option", { name: "GPT-6-Astra" })).toBeNull();
+ expect(screen.getByRole("option", { name: "GPT-6-Luna" })).toBeTruthy();
+});
+
+test("a failed refresh is visible and the refresh button recovers the current catalog", async () => {
+ client.setQueryData(["agent-chat", "catalog", "society"], catalog(fallback));
+ client.setQueryData(["agent-chat", "connections"], connections);
+ client.setQueryData(["society", "providers"], providers);
+ client.setQueryData(["agent-runtimes"], { runtimes: [], supported_providers: [] });
+ let fail = true;
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => {
+ if (url.includes("/catalog")) return new Response(JSON.stringify(fail ? {} : catalog(current)), { status: fail ? 503 : 200 });
+ if (url.includes("/status")) return new Response(JSON.stringify({ mapping: connections }));
+ if (url.endsWith("/providers")) return new Response(JSON.stringify({ providers }));
+ throw new Error(`Unexpected request: ${url}`);
+ }));
+ openDialog();
+ expect(await screen.findByRole("alert")).toBeTruthy();
+ fail = false;
+ fireEvent.click(screen.getByRole("button", { name: "Refresh models" }));
+ await waitFor(() => expect(screen.getByTestId("create-agent-model").textContent).toContain("GPT-6.1-Sol"));
+ expect(screen.queryByRole("alert")).toBeNull();
+});
+
+test.each([
+ ["claude-api", "claude", "claude-cli"], ["grok-build", "xai", "grok-cli"], ["openai-codex", "openai", "codex-cli"],
+])("%s loads and submits the selected account's models, never the active account's list", async (id, family, runner) => {
+ const row = { ...fallback, id, family, runner, hidden_models: [], curated_models: [{ id: "active-model", label: "Active model" }] };
+ const accounts = ["first", "second"].map((accountId) => ({ id: accountId, label: accountId, connected: true, mode: "subscription" }));
+ let release!: () => void;
+ const pending = new Promise<void>((resolve) => { release = resolve; });
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => {
+ if (url.includes("/catalog")) {
+ const account = new URL(url, "http://localhost").searchParams.get("account_id");
+ if (account === "second") await pending;
+ return new Response(JSON.stringify(catalog(account ? { ...row, curated_models: [{ id: `${account}-only`, label: `${account} only` }] } : row)));
+ }
+ if (url.includes("/agent-runtimes")) return new Response(JSON.stringify({ runtimes: [], supported_providers: [] }));
+ if (url.includes("/status")) return new Response(JSON.stringify({ mapping: [{ jarvis: id, key_set: true }] }));
+ if (url.endsWith("/providers")) return new Response(JSON.stringify({ providers: [{ ...providers[0], id, family, runner, accounts }] }));
+ throw new Error(`Unexpected request: ${url}`);
+ }));
+ openDialog();
+ await screen.findByTestId("create-agent-account");
+ await waitFor(() => expect(screen.getByTestId("create-agent-model").textContent).toContain("Active model"));
+ fireEvent.click(screen.getByTestId("create-agent-account"));
+ fireEvent.click(screen.getByRole("option", { name: "second" }));
+ expect((screen.getByTestId("create-agent-submit") as HTMLButtonElement).disabled).toBe(true);
+ expect(screen.getByTestId("create-agent-model").textContent).not.toContain("Active model");
+ await act(async () => { release(); await pending; });
+ await waitFor(() => expect(screen.getByTestId("create-agent-model").textContent).toContain("second only"));
+ fireEvent.click(screen.getByTestId("create-agent-submit"));
+ await waitFor(() => expect(createAgent).toHaveBeenCalledWith(expect.objectContaining({ provider: id, model: "second-only", accountId: "second" })));
+});
+
+test.each(["claude-api", "gemini", "grok", "openrouter", "nvidia"])("%s distinguishes a failed model catalog from an empty one and recovers without inventing a default", async (id) => {
+ const row = { ...fallback, id, family: id, runner: "brain", cli_installed: null, models_source: "live" as const,
+ hidden_models: [], curated_models: [{ id: "stale", label: "Stale fallback" }] };
+ let phase: "error" | "empty" | "ready" = "error";
+ vi.stubGlobal("fetch", vi.fn(async (url: string) => {
+ if (url.includes("/catalog")) return new Response(JSON.stringify(catalog(row)));
+ if (url.includes("/agent-runtimes")) return new Response(JSON.stringify({ runtimes: [], supported_providers: [] }));
+ if (url.includes("/status")) return new Response(JSON.stringify({ mapping: [{ jarvis: id, key_set: true, api_key_set: true }] }));
+ if (url.endsWith("/providers")) return new Response(JSON.stringify({ providers: [{ ...providers[0], id, subscription: false, accounts: [] }] }));
+ if (url.endsWith("/models")) return new Response(JSON.stringify({ models: phase === "ready" ? [{ id: "live-model", label: "Live model" }] : [] }), { status: phase === "error" ? 401 : 200 });
+ throw new Error(`Unexpected request: ${url}`);
+ }));
+ openDialog();
+ await screen.findByRole("alert");
+ expect((screen.getByTestId("create-agent-submit") as HTMLButtonElement).disabled).toBe(true);
+ phase = "empty";
+ fireEvent.click(screen.getByRole("button", { name: "Refresh models" }));
+ await waitFor(() => expect(screen.queryByRole("alert")).toBeNull());
+ expect(screen.getByTestId("create-agent-model").textContent).not.toContain("Stale fallback");
+ expect((screen.getByTestId("create-agent-submit") as HTMLButtonElement).disabled).toBe(true);
+ phase = "ready";
+ fireEvent.click(screen.getByRole("button", { name: "Refresh models" }));
+ await waitFor(() => expect(screen.getByTestId("create-agent-model").textContent).toContain("Live model"));
+ expect((screen.getByTestId("create-agent-submit") as HTMLButtonElement).disabled).toBe(false);
+ expect(createAgent).not.toHaveBeenCalled();
+});
diff --git a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.test.tsx b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.test.tsx
index f5de6f70dc..795903975d 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.test.tsx
@@ -5,6 +5,7 @@ import type { AgentRuntimesResponse } from "@/lib/agentRuntimesApi";
import type { ProviderOption } from "@/store/agentChat";
import { CreateAgentDialogHost } from "./CreateAgentDialog";
import { isCreateCancelled, useCreateAgentDialog } from "./createAgentStore";
+import { MODEL_ACCESS_KEY } from "@/lib/modelAccess";
vi.mock("@/i18n", () => ({ useT: () => (key: string) => key }));
vi.mock("../companion/CompanionEditor", () => ({
@@ -68,6 +69,7 @@ function mount() {
}
beforeEach(() => {
+ localStorage.removeItem(MODEL_ACCESS_KEY);
runtimes.value = READY;
menu.options = [OPENAI, OLLAMA];
menu.live = { ollama: [{ id: "qwen3:8b", label: "qwen3:8b" }] };
@@ -80,6 +82,17 @@ afterEach(() => {
});
describe("CreateAgentDialog", () => {
+ test("uses the access preference saved by the other model pickers", async () => {
+ localStorage.setItem(MODEL_ACCESS_KEY, JSON.stringify({ claude: "api" }));
+ runtimes.value = { ...READY, access: { "claude-api": ["api", "subscription"] } };
+ menu.options = [CLAUDE];
+ mount();
+ act(() => { useCreateAgentDialog.getState().request().catch(() => undefined); });
+ const api = await screen.findByRole("radio", { name: "society.create.kind_api" });
+ await waitFor(() => expect(api.getAttribute("aria-checked")).toBe("true"));
+ fireEvent.click(screen.getByRole("radio", { name: "society.create.kind_subscription" }));
+ expect(JSON.parse(localStorage.getItem(MODEL_ACCESS_KEY)!)).toEqual({ claude: "subscription" });
+ });
test("creates a named Hermes agent with a provider and a companion", async () => {
const agent = { agentId: "agent-1", name: "Scout" };
createAgent.mockResolvedValue(agent);
diff --git a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx
index 1b375087f0..d45393dbad 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/create/CreateAgentDialog.tsx
@@ -9,7 +9,7 @@
*/
import { useEffect, useMemo, useState, Suspense, lazy } from "react";
import * as Dialog from "@radix-ui/react-dialog";
-import { X } from "lucide-react";
+import { Loader2, RefreshCw, X } from "lucide-react";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { BrandedSelect } from "@/components/ui/select";
@@ -26,6 +26,7 @@ import { accountChoice, accountHint } from "./brainPicker";
import { useCreateAgentDialog } from "./createAgentStore";
import { AccessChoice } from "./AccessChoice";
import { accessModels, defaultModel, pickableOption, providerChoices, type AccessBlocked, type AccessOption } from "./seatChoice";
+import { useModelAccess, type ModelAccess } from "@/lib/modelAccess";
const CompanionEditor = lazy(() =>
import("../companion/CompanionEditor").then((m) => ({ default: m.CompanionEditor })),
@@ -53,16 +54,20 @@ function CreateAgentDialog() {
const { finish, cancel } = useCreateAgentDialog.getState();
const createAgent = useCreateSocietyAgent();
const runtimes = useAgentRuntimes();
- const menu = useModelMenuData();
const [name, setName] = useState("");
const [runtime, setRuntime] = useState<AgentRuntime>("jarvis");
const [providerId, setProviderId] = useState("");
- const [kind, setKind] = useState("");
+ const [access, rememberAccess] = useModelAccess();
const [model, setModel] = useState("");
const [account, setAccount] = useState("");
+ const [accountProvider, setAccountProvider] = useState("");
const [companion, setCompanion] = useState<CompanionAppearance>(randomCompanion);
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
+ // A stored catalog may be a startup fallback. Revalidate on this user gesture,
+ // and scope subscription models to the login that will actually run the agent.
+ const menu = useModelMenuData(null, undefined,
+ account && accountProvider ? { [accountProvider]: account } : {}, { refreshOnMount: true });
const external = runtime !== "jarvis";
const defaultModelLabel = t("agent_chat.model_default");
@@ -85,12 +90,12 @@ function CreateAgentDialog() {
const chosen = choices.find((choice) => choice.id === providerId)
?? choices.find((choice) => pickableOption(choice)) ?? choices[0] ?? null;
// A refused access is listed with its reason but never picked.
- const option: AccessOption | null = pickableOption(chosen, kind);
+ const option: AccessOption | null = pickableOption(chosen, chosen ? access[chosen.id] : undefined);
const models = accessModels(option);
const accounts = accountChoice(option?.seat ?? null);
const modelValue = models.some((entry) => entry.id === model) ? model : defaultModel(option);
- const accountValue = accounts.some((entry) => entry.id === account) ? account : "";
- const seatKey = `${chosen?.id ?? ""}|${option?.kind ?? ""}`;
+ const accountValue = option?.seat.provider.id === accountProvider && accounts.some((entry) => entry.id === account) ? account : "";
+ const seatKey = `${runtime}|${chosen?.id ?? ""}|${option?.kind ?? ""}|${accountValue}`;
useEffect(() => {
// Another provider or access: its own default model, not the last one's id.
setModel("");
@@ -100,7 +105,10 @@ function CreateAgentDialog() {
// first turn waits for the setup (agent_runtimes.manager). A Jarvis agent
// with nothing connected yet still starts: it takes the last chat seat.
const needsProvider = external && !option;
- const canCreate = !saving && !needsProvider;
+ const accountBlocked = Boolean(account && accountProvider === option?.seat.provider.id
+ && (accountValue !== account || menu.loading || menu.failed));
+ const selectedModelsFailed = Boolean(option && menu.modelErrors?.includes(option.seat.provider.id));
+ const canCreate = !saving && !needsProvider && !accountBlocked && !selectedModelsFailed && (!option || models.length > 0);
async function submit() {
if (!canCreate) return;
@@ -167,7 +175,7 @@ function CreateAgentDialog() {
<div className="flex flex-col gap-1.5 text-sm">
<span className="font-medium">{t("society.create_agent.type_label")}</span>
- <RuntimeChoice value={runtime} onChange={setRuntime} disabled={saving} />
+ <RuntimeChoice value={runtime} onChange={(next) => { setRuntime(next); setAccount(""); }} disabled={saving} />
<p className="text-xs text-muted-foreground">{t("society.create_agent.type_fixed")}</p>
</div>
@@ -176,7 +184,7 @@ function CreateAgentDialog() {
{chosen ? (
<BrandedSelect
value={chosen.id}
- onValueChange={(next) => { setProviderId(next); setKind(""); setAccount(""); }}
+ onValueChange={(next) => { setProviderId(next); setAccount(""); }}
ariaLabel={t("society.create_agent.provider_label")}
disabled={saving}
testId="create-agent-provider"
@@ -204,7 +212,7 @@ function CreateAgentDialog() {
value={option?.kind ?? ""}
hint={option ? t(option.extraUsage ? "society.create_agent.access_extra_usage" : `society.create_agent.access_hint_${option.kind}`) : ""}
disabled={saving}
- onChange={(next) => { setKind(next); setAccount(""); }}
+ onChange={(next) => { rememberAccess(chosen.id, next as ModelAccess); setAccount(""); }}
/>
) : null}
@@ -213,7 +221,7 @@ function CreateAgentDialog() {
<span className="font-medium">{t("society.create_agent.account_label")}</span>
<BrandedSelect
value={accountValue}
- onValueChange={setAccount}
+ onValueChange={(next) => { setAccountProvider(option.seat.provider.id); setAccount(next); setModel(""); }}
ariaLabel={t("society.create_agent.account_label")}
disabled={saving}
testId="create-agent-account"
@@ -227,20 +235,30 @@ function CreateAgentDialog() {
{chosen && option ? (
<div className="flex flex-col gap-1.5 text-sm">
- <span className="font-medium">{t("society.create_agent.model_label")}</span>
+ <div className="flex items-center justify-between gap-2">
+ <span className="font-medium">{t("society.create_agent.model_label")}</span>
+ <button type="button" disabled={saving || menu.refreshing} onClick={() => void Promise.all([menu.refresh(), runtimes.refetch()])}
+ aria-label={t("society.chat.model_refresh")} title={t("society.chat.model_refresh")}
+ className="rounded p-1 text-muted-foreground hover:bg-secondary focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-50">
+ {menu.refreshing ? <Loader2 className="h-3.5 w-3.5 animate-spin" aria-hidden /> : <RefreshCw className="h-3.5 w-3.5" aria-hidden />}
+ </button>
+ </div>
<BrandedSelect
value={modelValue}
onValueChange={setModel}
ariaLabel={t("society.create_agent.model_label")}
- disabled={saving}
+ placeholder={t("agent_chat.no_models")}
+ disabled={saving || accountBlocked || models.length === 0}
testId="create-agent-model"
searchPlaceholder={models.length >= SEARCH_FROM ? t("society.create_agent.model_search") : undefined}
- options={(models.length ? models : [{ id: "", label: defaultModelLabel }]).map((entry) => ({
+ options={models.map((entry) => ({
value: entry.id,
label: entry.label || entry.id || defaultModelLabel,
}))}
/>
<p className="text-xs text-muted-foreground">{t("society.create_agent.model_hint")}</p>
+ {menu.refreshing ? <p role="status" className="text-xs text-muted-foreground">{t("society.chat.models_loading")}</p> : null}
+ {menu.refreshFailed || selectedModelsFailed ? <p role="alert" className="text-xs text-destructive">{t("society.chat.model_load_failed")}</p> : null}
</div>
) : null}
diff --git a/jarvis/ui/web/frontend/src/components/society/create/brainPicker.test.ts b/jarvis/ui/web/frontend/src/components/society/create/brainPicker.test.ts
index ea69f47393..66c956e352 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/brainPicker.test.ts
+++ b/jarvis/ui/web/frontend/src/components/society/create/brainPicker.test.ts
@@ -46,6 +46,18 @@ function societyRow(over: Partial<SocietyProviderRow>): SocietyProviderRow {
}
describe("brainSeats", () => {
+ it.each(["claude-api", "gemini", "grok"])("%s uses its actual API transport even when account metadata describes a subscription", (id) => {
+ const [seat] = brainSeats([option({ id, runner: "brain" })],
+ [societyRow({ id, subscription: true, accounts: [account("subscription-login")] })]);
+ expect(seat.kind).toBe("api");
+ expect(seat.accounts).toEqual([]);
+ });
+
+ it.each(["claude-api", "gemini", "grok", "openrouter", "nvidia"])("%s does not resurrect fallback models after a successful empty live catalog", (id) => {
+ const [seat] = brainSeats([option({ id, curated_models: [{ id: "retired", label: "Retired" }] })], [], { [id]: [] });
+ expect(seat.provider.curated_models).toEqual([]);
+ });
+
it("lists only connected rows, subscriptions first, then keys, then local", () => {
const seats = brainSeats(
[
diff --git a/jarvis/ui/web/frontend/src/components/society/create/brainPicker.ts b/jarvis/ui/web/frontend/src/components/society/create/brainPicker.ts
index dc0f846910..d3dc6a936c 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/brainPicker.ts
+++ b/jarvis/ui/web/frontend/src/components/society/create/brainPicker.ts
@@ -35,6 +35,9 @@ export interface BrainSeat {
const KIND_ORDER: Record<BrainKind, number> = { subscription: 0, api: 1, local: 2 };
function kindOf(option: ProviderOption, row: SocietyProviderRow | undefined): BrainKind {
+ // The runtime catalog resolves dual providers to the actual transport.
+ // An account-reader row can still describe their subscription capability.
+ if (isApiRunner(option.runner)) return option.keyless ? "local" : "api";
if (row ? row.subscription : !isApiRunner(option.runner)) return "subscription";
return option.keyless ? "local" : "api";
}
@@ -68,7 +71,7 @@ export function brainSeats(
.filter((o) => !o.keyless || (liveModels[o.id]?.length ?? 0) > 0)
.flatMap((option) => {
const live = liveModels[option.id];
- const listed = live?.length ? live : option.curated_models;
+ const listed = live ?? option.curated_models;
// Models hidden on the API Keys page stay out of the agents' pickers; a
// provider whose every model is hidden is not offered at all.
const hidden = new Set(option.hidden_models ?? []);
diff --git a/jarvis/ui/web/frontend/src/components/society/create/seatChoice.test.ts b/jarvis/ui/web/frontend/src/components/society/create/seatChoice.test.ts
index ba95a498f5..6e3027a160 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/seatChoice.test.ts
+++ b/jarvis/ui/web/frontend/src/components/society/create/seatChoice.test.ts
@@ -54,6 +54,26 @@ const openai = seat({});
const ollama = seat({ id: "ollama", label: "Ollama", family: "ollama", keyless: true, curated_models: models("qwen3:8b") }, "local");
describe("providerChoices", () => {
+ it("respects an explicit empty access list and a missing required CLI", () => {
+ expect(providerChoices([claude], { "claude-api": [] })).toEqual([]);
+ const missing = { ...claude, provider: { ...claude.provider, cli_installed: false } };
+ expect(providerChoices([missing], { "claude-api": ["subscription"] })).toEqual([]);
+ });
+
+ it.each([
+ ["grok-build", "grok", "xai", "xai", "grok-cli"],
+ ["antigravity", "gemini", "antigravity", "gemini", "agy-cli"],
+ ])("groups %s with its API brand while retaining each access's own models", (subscription, api, family, brand, runner) => {
+ const [choice] = providerChoices([
+ seat({ id: api, family: brand, curated_models: models("api-model-1") }),
+ seat({ id: subscription, family, runner, cli_installed: true, curated_models: models("subscription-model-2") }, "subscription"),
+ ]);
+ expect(choice.id).toBe(brand);
+ expect(choice.options.map((option) => [option.kind, option.seat.provider.id, accessModels(option)[0].id])).toEqual([
+ ["subscription", subscription, "subscription-model-2"], ["api", api, "api-model-1"],
+ ]);
+ });
+
it("folds a brand's subscription and API rows into one provider", () => {
const [choice] = providerChoices([codex, openai]);
expect(choice.label).toBe("OpenAI");
diff --git a/jarvis/ui/web/frontend/src/components/society/create/seatChoice.ts b/jarvis/ui/web/frontend/src/components/society/create/seatChoice.ts
index 5a9710c80c..45a193edcf 100644
--- a/jarvis/ui/web/frontend/src/components/society/create/seatChoice.ts
+++ b/jarvis/ui/web/frontend/src/components/society/create/seatChoice.ts
@@ -23,6 +23,7 @@
*/
import type { CuratedModel } from "@/lib/agentChatApi";
import { rankModels } from "@/lib/modelRanking";
+import { modelAccessFamily } from "@/lib/modelAccess";
import { apiModels } from "../chat/modelChoices";
import type { BrainKind, BrainSeat } from "./brainPicker";
@@ -62,9 +63,6 @@ export interface ProviderChoice {
const KIND_ORDER: Record<BrainKind, number> = { subscription: 0, api: 1, local: 2 };
-/** Catalog families that are one brand to a person (Gemini's CLI is Antigravity). */
-const SAME_BRAND: Record<string, string> = { antigravity: "gemini" };
-
/** A brand's own name and logo where its rows name a product instead. */
const BRANDS: Record<string, { label: string; logo: string }> = {
claude: { label: "Anthropic Claude", logo: "claude-api" },
@@ -75,7 +73,7 @@ const BRANDS: Record<string, { label: string; logo: string }> = {
function brandOf(seat: BrainSeat): string {
const family = seat.provider.family || seat.provider.id;
- return SAME_BRAND[family] ?? family;
+ return modelAccessFamily(family);
}
function optionsFor(
@@ -107,7 +105,7 @@ function optionsFor(
for (const option of options) {
if (blocked[option.kind]) option.blocked = blocked[option.kind];
}
- return options.length ? options : [single];
+ return options;
}
/** Refusal codes with their own explanation (`provider_errors.py`). */
@@ -163,6 +161,7 @@ export function providerChoices(
const best = (choice: ProviderChoice) => Math.min(
...choice.options.map((option) => (option.blocked ? 10 : 0) + KIND_ORDER[option.kind]));
return [...byBrand.values()]
+ .filter((choice) => choice.options.length > 0)
.map((choice) => ({ ...choice, options: [...choice.options].sort((a, b) => KIND_ORDER[a.kind] - KIND_ORDER[b.kind]) }))
.sort((a, b) => best(a) - best(b) || a.label.localeCompare(b.label));
}
diff --git a/jarvis/ui/web/frontend/src/components/society/mars/MarsWorldStage.tsx b/jarvis/ui/web/frontend/src/components/society/mars/MarsWorldStage.tsx
index 25df4fcc3e..77a654c8dc 100644
--- a/jarvis/ui/web/frontend/src/components/society/mars/MarsWorldStage.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/mars/MarsWorldStage.tsx
@@ -23,6 +23,7 @@ import { useAgentFollowTarget } from "./useAgentFollowTarget";
import { bindFollowEscape } from "./followInput";
import { ownsTextOrUi } from "./input";
import "./mars.css";
+import { useSyncCompanionPet } from "../companion/companionPetStore";
export interface MarsWorldStageProps {
topRight?: ReactNode;
@@ -51,6 +52,8 @@ export function MarsWorldStage({ topRight, onOpenLedger, onSelectAgent, stationP
const ready = useLocaleChunk("society");
const hostRef = useRef<HTMLDivElement>(null);
const awake = useCanvasAwake(hostRef);
+ // Agents that wear a pet need the pets inside the canvas.
+ useSyncCompanionPet();
const [initial] = useState(readViewPreferences);
const [mode, setMode] = useState<CameraMode>(initial.mode);
const [followAgentId, setFollowAgentId] = useState(initial.followAgentId);
diff --git a/jarvis/ui/web/frontend/src/components/society/office/OfficeAgents.groundAnchor.test.tsx b/jarvis/ui/web/frontend/src/components/society/office/OfficeAgents.groundAnchor.test.tsx
index d586b7d360..9dcab799c8 100644
--- a/jarvis/ui/web/frontend/src/components/society/office/OfficeAgents.groundAnchor.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/office/OfficeAgents.groundAnchor.test.tsx
@@ -75,7 +75,11 @@ async function mount(reduced: boolean) {
const current = createRoot(canvas);
root = current;
let state!: RootState;
+ const scene = new THREE.Scene();
+ // Check real geometry and animation without GPU reflection baking.
+ scene.environment = new THREE.Texture();
current.configure({
+ scene,
gl: fakeRenderer(canvas) as unknown as NonNullable<Parameters<typeof current.configure>[0]>["gl"],
frameloop: "never", size: { width: 100, height: 100, top: 0, left: 0 },
onCreated: (value) => { state = value; },
diff --git a/jarvis/ui/web/frontend/src/components/society/progression/LevelHud.tsx b/jarvis/ui/web/frontend/src/components/society/progression/LevelHud.tsx
index 709d85820f..29384633c4 100644
--- a/jarvis/ui/web/frontend/src/components/society/progression/LevelHud.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/progression/LevelHud.tsx
@@ -156,8 +156,9 @@ function LevelToast({ toast, name, look, onClose, label }: { toast: Celebration;
{looks.map((id) => {
return (
<span key={id} className="level-toast-look">
- <AgentSymbol shape={look.shape} color={look.color} eyes={look.eyes} size={30}
- accessories={{ ...look.accessories, ...lookSlot(id) }} />
+ {/* Looks dress the shape; an agent that wears a pet shows only the words. */}
+ {!look.pet && <AgentSymbol shape={look.shape} color={look.color} eyes={look.eyes} size={30}
+ accessories={{ ...look.accessories, ...lookSlot(id) }} />}
<span>{label("society.level.new_look").replace("{0}", label(`society.companion.items.${id}`))}</span>
</span>
);
diff --git a/jarvis/ui/web/frontend/src/components/society/progression/hall/LevelHallScreen.tsx b/jarvis/ui/web/frontend/src/components/society/progression/hall/LevelHallScreen.tsx
index c4eb1aed44..580f6152fc 100644
--- a/jarvis/ui/web/frontend/src/components/society/progression/hall/LevelHallScreen.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/progression/hall/LevelHallScreen.tsx
@@ -37,6 +37,7 @@ import { RanksTab } from "./RanksTab";
import { RewardsTab } from "./RewardsTab";
import { StudioTab } from "./StudioTab";
import { AgentSymbol } from "../../AgentSymbol";
+import { CompanionMark } from "../../companion/CompanionMark";
import { accessoryItem } from "../../companion/accessories";
import { resolveCompanion } from "../../companion/appearance";
import "./hall.css";
@@ -213,15 +214,15 @@ function TeamTab({ agents }: { agents: readonly SocietyAgent[] }) {
return (
<li key={agent.agentId} data-podium={i < 3 ? i + 1 : undefined}>
<span className="hall-rank">{i + 1}</span>
- <AgentSymbol shape={look.shape} color={look.color} eyes={look.eyes} accessories={look.accessories} size={34} />
+ <CompanionMark appearance={look} size={34} />
<LevelChip kind="agent" level={level} />
<span className="hall-rank-body">
<span><strong>{agent.name}</strong> <em>{t(`society.level.title.${rankOf(subject?.title)}`)}</em></span>
<XpBar fraction={levelFraction(subject)} kind="agent" label={t("society.level.agent_xp_label").replace("{0}", agent.name)} />
{next && nextItem && (
<span className="hall-rank-next">
- <AgentSymbol shape={look.shape} color={look.color} eyes={look.eyes} size={22}
- accessories={{ ...look.accessories, [nextItem.slot]: next[0] }} />
+ {!look.pet && <AgentSymbol shape={look.shape} color={look.color} eyes={look.eyes} size={22}
+ accessories={{ ...look.accessories, [nextItem.slot]: next[0] }} />}
{t("society.hall.next_look").replace("{0}", t(`society.companion.items.${next[0]}`)).replace("{1}", String(next[1]))}
</span>
)}
diff --git a/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx b/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx
index 9e9f69d67b..d313571675 100644
--- a/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/roster/RosterRail.tsx
@@ -33,6 +33,7 @@ import { useEventStore } from "@/store/events";
import type { SocietyChatGroup } from "@/lib/societyChatGroups";
import { AgentSwatch } from "../AgentSwatch";
+import { RuntimeBadge } from "../RuntimeBadge";
import type { AgentRunState, SocietyAgent } from "../data";
import { AgentRosterActions } from "./AgentRosterActions";
import { GroupRosterActions } from "./GroupRosterActions";
@@ -630,7 +631,10 @@ export function RosterRail({
)}
>
<button type="button" onClick={() => openProfile(agent.agentId)} aria-label={t("society.profile_card.open").replace("{0}", agent.name)} className="shrink-0 rounded-full focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
- <AgentSwatch agent={agent} size={48} />
+ <span className="relative flex">
+ <AgentSwatch agent={agent} size={48} />
+ {agent.tier !== "lead" ? <RuntimeBadge runtime={agent.runtime} className="absolute bottom-0.5 right-0.5" /> : null}
+ </span>
</button>
<button type="button" onClick={() => openAgent(agent.agentId)} aria-current={agent.agentId === activeAgentId ? "true" : undefined} className="flex min-w-0 flex-1 select-none items-center gap-2.5 rounded-md py-2 pl-2.5 text-left focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
<span className="min-w-0 flex-1">
diff --git a/jarvis/ui/web/frontend/src/components/society/world/WorldStage.tsx b/jarvis/ui/web/frontend/src/components/society/world/WorldStage.tsx
index 61b746e493..9c7ed55410 100644
--- a/jarvis/ui/web/frontend/src/components/society/world/WorldStage.tsx
+++ b/jarvis/ui/web/frontend/src/components/society/world/WorldStage.tsx
@@ -63,6 +63,7 @@ function asHub(place: PlaceId): KitPlace | null {
import { CAMERA_FAR_M, cameraOffset } from "./worldCamera";
import { SKY } from "./worldPalette";
import { useWorldSettings } from "./worldSettings";
+import { useSyncCompanionPet } from "../companion/companionPetStore";
export interface WorldStageProps {
/** Where the fallback sends someone whose window cannot draw 3D. */
@@ -87,6 +88,8 @@ export function WorldStage({ onOpenAgents, onSelectAgent, onSelectPlace }: World
const ready = useLocaleChunk("society");
const hostRef = useRef<HTMLDivElement>(null);
const { generation } = useWebglSurface(hostRef);
+ // Agents that wear a pet need the pets inside the canvas.
+ useSyncCompanionPet();
const awake = useCanvasAwake(hostRef);
const reduced = useReducedMotion() ?? false;
const webgl = useWebglSupported();
diff --git a/jarvis/ui/web/frontend/src/components/ui/combobox.tsx b/jarvis/ui/web/frontend/src/components/ui/combobox.tsx
index f319aebd75..7b6e003cfe 100644
--- a/jarvis/ui/web/frontend/src/components/ui/combobox.tsx
+++ b/jarvis/ui/web/frontend/src/components/ui/combobox.tsx
@@ -125,6 +125,8 @@ export interface ComboboxProps {
browseGroups?: ComboboxGroup[];
/** A column drawn left of the search and the list (a rail of sections). */
aside?: ReactNode;
+ /** Controls above the browsed rows; hidden while searching across all groups. */
+ browseHeader?: ReactNode;
/** Told whenever the panel opens or closes. */
onOpenChange?: (open: boolean) => void;
/** The panel's narrowest width; defaults to 288 px. */
@@ -268,6 +270,7 @@ export function Combobox({
triggerPrefix,
browseGroups,
aside,
+ browseHeader,
onOpenChange,
panelMinWidth = MIN_PANEL_WIDTH,
}: ComboboxProps) {
@@ -691,6 +694,7 @@ export function Combobox({
</div>
)}
+ {!query.trim() && browseHeader}
<div
ref={attachList}
id={listId}
diff --git a/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.test.tsx b/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.test.tsx
index ea13f55900..39b05edbc5 100644
--- a/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.test.tsx
+++ b/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.test.tsx
@@ -16,6 +16,7 @@ import {
import { BrowserRealtimeControl, waveformPhase } from "./BrowserRealtimeControl";
const fakes = vi.hoisted(() => ({
+ prepareAudio: vi.fn(() => vi.fn()),
native: false,
mode: "realtime",
available: true,
@@ -36,6 +37,8 @@ const fakes = vi.hoisted(() => ({
options: null as null | { requiresWebRtcOffer?: boolean; browserAudio?: boolean },
}));
+vi.mock("@/lib/realtimeAudioPreparation", () => ({ registerRealtimeAudioPreparation: fakes.prepareAudio }));
+
vi.mock("@/hooks/useCapabilities", () => ({
useCapabilities: () => ({ data: { native_file_actions: fakes.native, platform: "linux" } }),
}));
@@ -73,6 +76,7 @@ vi.mock("@/lib/realtimeAudio", () => ({
describe("BrowserRealtimeControl", () => {
beforeEach(() => {
+ fakes.prepareAudio.mockClear();
fakes.native = false;
fakes.mode = "realtime";
fakes.available = true;
@@ -96,6 +100,31 @@ describe("BrowserRealtimeControl", () => {
});
});
+ it("prepares only the enabled voice owner's local audio and disposes it when disabled", () => {
+ fakes.native = true;
+ fakes.browserAudio = true;
+ (window as unknown as { pywebview?: unknown }).pywebview = { api: {} };
+ const view = render(<BrowserRealtimeControl controlOnly />);
+ expect(fakes.prepareAudio).toHaveBeenCalledOnce();
+ const dispose = fakes.prepareAudio.mock.results[0].value;
+ fakes.mode = "pipeline";
+ view.rerender(<BrowserRealtimeControl controlOnly />);
+ expect(dispose).toHaveBeenCalledOnce();
+ expect(fakes.connect).not.toHaveBeenCalled();
+ });
+
+ it("does not prepare in a detached non-owner desktop view or unavailable mode", () => {
+ fakes.native = true;
+ fakes.browserAudio = true;
+ (window as unknown as { pywebview?: unknown }).pywebview = { api: {} };
+ useEventStore.setState({ solo: true, activeSection: "settings" });
+ const view = render(<BrowserRealtimeControl controlOnly />);
+ expect(fakes.prepareAudio).not.toHaveBeenCalled();
+ fakes.available = false;
+ view.rerender(<BrowserRealtimeControl />);
+ expect(fakes.prepareAudio).not.toHaveBeenCalled();
+ });
+
it("offers a browser recovery link when the hidden media host cannot start", async () => {
fakes.browserAudio = true;
fakes.connect.mockRejectedValueOnce(new Error("Microphone unavailable"));
@@ -107,6 +136,159 @@ describe("BrowserRealtimeControl", () => {
expect(await screen.findByRole("link", { name: "live.open_browser" })).toBeTruthy();
});
+ describe("correlated native startup failure", () => {
+ const firstRequest = "1a468a93-a7c3-43f6-a95e-7b1a6b49c28b";
+ const secondRequest = "4fd86de5-0c5b-4351-8871-fb43ab708ac7";
+ let fetchSpy: ReturnType<typeof vi.fn>;
+ const request = (id: string, action: string, requestId?: string) => act(() => useEventStore.getState().pushEvent({
+ id, name: "BrowserVoiceRequested", ts: Date.now(),
+ payload: { action, ...(requestId ? { request_id: requestId } : {}) },
+ }));
+ beforeEach(() => {
+ fakes.browserAudio = true;
+ fetchSpy = vi.fn(async () => new Response("{}", { status: 200 }));
+ vi.stubGlobal("fetch", fetchSpy);
+ });
+ afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); });
+
+ it.each([200, 404])("reports only the current request once and preserves its error (HTTP %s)", async status => {
+ fetchSpy.mockResolvedValueOnce(new Response("{}", { status }));
+ fakes.connect.mockRejectedValueOnce(new Error("Local audio device did not become ready"));
+ render(<BrowserRealtimeControl controlOnly />);
+ request("native-failure", "start", firstRequest);
+ await waitFor(() => expect(fetchSpy).toHaveBeenCalledOnce());
+ expect(fetchSpy.mock.calls[0]).toEqual(["/api/voice/startup-failed", {
+ method: "POST", headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ request_id: firstRequest }),
+ }]);
+ expect(screen.getByRole("alert").textContent).toContain("sidebar.realtime_error");
+ act(() => fakes.callbacks?.onStatus?.("disconnected", { reason: "late close" }));
+ expect(fetchSpy).toHaveBeenCalledOnce();
+ });
+
+ it("preserves request correlation while a hidden browser waits for visibility", async () => {
+ const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("hidden");
+ fakes.connect.mockRejectedValueOnce(new Error("Device failed"));
+ render(<BrowserRealtimeControl controlOnly />);
+ request("hidden-failure", "start", firstRequest);
+ expect(fakes.connect).not.toHaveBeenCalled();
+ visibility.mockReturnValue("visible");
+ act(() => document.dispatchEvent(new Event("visibilitychange")));
+ await waitFor(() => expect(fetchSpy).toHaveBeenCalledOnce());
+ expect(JSON.parse(fetchSpy.mock.calls[0][1].body)).toEqual({ request_id: firstRequest });
+ });
+
+ it("ignores an old rejected attempt and its stop after a newer call starts", async () => {
+ let rejectOld!: (error: Error) => void;
+ fakes.connect.mockImplementationOnce(() => new Promise<undefined>((_resolve, reject) => { rejectOld = reject; }));
+ render(<BrowserRealtimeControl controlOnly />);
+ request("old-start", "start", firstRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledTimes(1));
+ request("old-stop", "stop", firstRequest);
+ await waitFor(() => expect(fakes.disconnect).toHaveBeenCalledTimes(1));
+ request("new-start", "start", secondRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledTimes(2));
+ await act(async () => { rejectOld(new Error("Late old device failure")); });
+ request("late-old-stop", "stop", firstRequest);
+ expect(fetchSpy).not.toHaveBeenCalled();
+ expect(fakes.disconnect).toHaveBeenCalledTimes(1);
+ expect(screen.queryByRole("alert")).toBeNull();
+ request("new-stop", "stop", secondRequest);
+ await waitFor(() => expect(fakes.disconnect).toHaveBeenCalledTimes(2));
+ });
+
+ it.each(["resolve", "reject"])("replaces a pending native request without an old stop and ignores its late %s", async settle => {
+ let resolveOld!: () => void, rejectOld!: (error: Error) => void, rejectNew!: (error: Error) => void;
+ fakes.connect.mockImplementationOnce(() => new Promise<undefined>((resolve, reject) => {
+ resolveOld = () => resolve(undefined);
+ rejectOld = reject;
+ }));
+ fakes.connect.mockImplementationOnce(() => new Promise<undefined>((_resolve, reject) => { rejectNew = reject; }));
+ render(<BrowserRealtimeControl controlOnly />);
+ request("pending-old-start", "start", firstRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledTimes(1));
+ const oldCallbacks = fakes.callbacks;
+ request("replacement-start", "start", secondRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledTimes(2));
+ expect(fakes.disconnect).toHaveBeenCalledTimes(1);
+ await act(async () => {
+ if (settle === "resolve") resolveOld(); else rejectOld(new Error("Old attempt failed"));
+ });
+ act(() => oldCallbacks?.onStatus?.("provider_error", { error: "Late old callback" }));
+ expect(fetchSpy).not.toHaveBeenCalled();
+ expect(fakes.disconnect).toHaveBeenCalledTimes(1);
+ expect(screen.queryByRole("alert")).toBeNull();
+ await act(async () => { rejectNew(new Error("New attempt failed")); });
+ await waitFor(() => expect(fetchSpy).toHaveBeenCalledOnce());
+ expect(JSON.parse(fetchSpy.mock.calls[0][1].body)).toEqual({ request_id: secondRequest });
+ });
+
+ it("retires a superseded connecting request while hidden and starts its replacement on visibility", async () => {
+ const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
+ let finishOld!: () => void;
+ fakes.connect.mockImplementationOnce(() => new Promise<undefined>(resolve => { finishOld = () => resolve(undefined); }));
+ render(<BrowserRealtimeControl controlOnly />);
+ request("visible-old", "start", firstRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledOnce());
+ visibility.mockReturnValue("hidden");
+ request("hidden-replacement", "start", secondRequest);
+ await waitFor(() => expect(fakes.disconnect).toHaveBeenCalledOnce());
+ expect(fakes.connect).toHaveBeenCalledOnce();
+ visibility.mockReturnValue("visible");
+ act(() => document.dispatchEvent(new Event("visibilitychange")));
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledTimes(2));
+ await act(async () => { finishOld(); });
+ expect(fetchSpy).not.toHaveBeenCalled();
+ expect(fakes.disconnect).toHaveBeenCalledOnce();
+ request("replacement-stop", "stop", secondRequest);
+ await waitFor(() => expect(fakes.disconnect).toHaveBeenCalledTimes(2));
+ });
+
+ it("keeps an established native call through reconnecting and preserves its original stop correlation", async () => {
+ render(<BrowserRealtimeControl controlOnly />);
+ request("established-start", "start", firstRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledOnce());
+ act(() => fakes.callbacks?.onStatus?.("audio_ready", {}));
+ act(() => fakes.callbacks?.onStatus?.("reconnecting", {}));
+ request("new-during-reconnect", "start", secondRequest);
+ await act(async () => undefined);
+ expect(fakes.disconnect).not.toHaveBeenCalled();
+ expect(fakes.connect).toHaveBeenCalledOnce();
+ expect(fetchSpy).not.toHaveBeenCalled();
+ expect(useEventStore.getState().voiceState).toBe("connecting");
+ request("unrelated-stop", "stop", secondRequest);
+ expect(fakes.disconnect).not.toHaveBeenCalled();
+ request("original-stop", "stop", firstRequest);
+ await waitFor(() => expect(fakes.disconnect).toHaveBeenCalledOnce());
+ });
+
+ it("does not acknowledge an uncorrelated older backend request", async () => {
+ fakes.connect.mockRejectedValueOnce(new Error("Device failed"));
+ render(<BrowserRealtimeControl controlOnly />);
+ request("legacy-start", "start");
+ await screen.findByRole("alert");
+ expect(fetchSpy).not.toHaveBeenCalled();
+ });
+
+ it("does not acknowledge a browser-held user call", async () => {
+ fakes.connect.mockRejectedValueOnce(new Error("Device failed"));
+ render(<BrowserRealtimeControl controlOnly />);
+ act(() => { expect(startBrowserVoiceCall()).toBe(true); });
+ await waitFor(() => expect(fakes.disconnect).toHaveBeenCalled());
+ expect(fetchSpy).not.toHaveBeenCalled();
+ expect(browserVoiceCallLive()).toBe(false);
+ });
+
+ it("does not acknowledge a provider failure after native startup succeeded", async () => {
+ render(<BrowserRealtimeControl controlOnly />);
+ request("connected-start", "start", firstRequest);
+ await waitFor(() => expect(fakes.connect).toHaveBeenCalledOnce());
+ act(() => fakes.callbacks?.onStatus?.("audio_ready", {}));
+ act(() => fakes.callbacks?.onStatus?.("provider_error", { error: "Provider disconnected" }));
+ expect(fetchSpy).not.toHaveBeenCalled();
+ });
+ });
+
it("parks a wake start while hidden and fires it when the tab returns", async () => {
fakes.browserAudio = true;
const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("hidden");
diff --git a/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.tsx b/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.tsx
index 1857e7621d..ae0b0229a9 100644
--- a/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.tsx
+++ b/jarvis/ui/web/frontend/src/components/voice/BrowserRealtimeControl.tsx
@@ -20,6 +20,7 @@ import {
} from "@/lib/realtimeAudio";
import { useEventStore, type VoiceState } from "@/store/events";
import { setReloadHold } from "@/lib/reloadHold";
+import { registerRealtimeAudioPreparation } from "@/lib/realtimeAudioPreparation";
import { cn } from "@/lib/utils";
import {
clearVoiceInputLevel,
@@ -81,6 +82,25 @@ async function reportHostMicrophoneDenied(): Promise<void> {
}
}
+function startupRequestId(payload: unknown): string | null {
+ const id = (payload as { request_id?: unknown } | null)?.request_id;
+ return typeof id === "string" && id.length > 0 && id.length <= 128 ? id : null;
+}
+
+async function reportNativeStartupFailure(requestId: string): Promise<void> {
+ try {
+ const response = await fetch("/api/voice/startup-failed", {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ request_id: requestId }),
+ });
+ // An updated frontend can still be talking to the preceding backend.
+ if (!response.ok && response.status !== 404) {
+ console.warn(`Voice startup failure acknowledgement failed: HTTP ${response.status}`);
+ }
+ } catch (error) { console.warn("Voice startup failure acknowledgement failed", error); }
+}
+
/** Browser-owned microphone control for remote/headless installations.
*
* The desktop shell already owns the physical microphone through
@@ -135,7 +155,9 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
// A wake that lands while the tab is hidden must not be consumed: the
// desktop is already waiting for this call, and dropping the request
// leaves it waiting out the full handshake budget for nothing.
- const pendingStart = useRef<{ id: string; ts: number } | null>(null);
+ const pendingStart = useRef<{ id: string; ts: number; requestId: string | null } | null>(null);
+ const nativeRequestRef = useRef<string | null>(null);
+ const initialNativeStartPending = useRef(false);
const connectionGenerationRef = useRef(0);
// A progress/preamble surface line is not the end of the turn. After the
// browser finishes speaking it, tts_end must restore thinking — not
@@ -156,6 +178,12 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
const callSurface = visible || localCall;
const supportIssue = visible ? browserRealtimeSupportIssue() : null;
+ useEffect(() => {
+ if (visible && realtimeAvailable && !supportIssue && (wakeOwner || !controlOnly)) {
+ return registerRealtimeAudioPreparation();
+ }
+ }, [visible, realtimeAvailable, supportIssue, wakeOwner, controlOnly]);
+
const supportMessage = useCallback(
(issue: BrowserRealtimeSupportIssue) =>
t(
@@ -170,6 +198,9 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
const stop = useCallback(async () => {
connectionGenerationRef.current += 1;
+ nativeRequestRef.current = null;
+ initialNativeStartPending.current = false;
+ pendingStart.current = null;
const client = clientRef.current;
clientRef.current = null;
if (localCallRef.current) {
@@ -191,7 +222,7 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
await client?.disconnect();
}, [setVoice]);
- const start = useCallback(async (options?: { fromGesture?: boolean; local?: boolean }) => {
+ const start = useCallback(async (options?: { fromGesture?: boolean; local?: boolean; requestId?: string | null }) => {
// A local start does not wait for realtime: on a host with no speech
// pipeline the server answers with a realtime session or the classic
// chain, whichever it can build.
@@ -199,6 +230,10 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
if ((!local && !realtimeAvailable) || clientRef.current || state === "connecting") return;
const generation = connectionGenerationRef.current + 1;
connectionGenerationRef.current = generation;
+ const nativeRequestId = local ? null : options?.requestId ?? null;
+ nativeRequestRef.current = nativeRequestId;
+ initialNativeStartPending.current = nativeRequestId !== null;
+ pendingStart.current = null;
if (local) {
localCallRef.current = true;
setLocalCall(true);
@@ -230,8 +265,16 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
void stop();
};
let client: RealtimeAudioClient;
+ let connected = false;
const isCurrent = () =>
connectionGenerationRef.current === generation && clientRef.current === client;
+ const acknowledgeStartupFailure = () => {
+ if (!connected && isCurrent() && nativeRequestId && nativeRequestRef.current === nativeRequestId) {
+ nativeRequestRef.current = null;
+ initialNativeStartPending.current = false;
+ void reportNativeStartupFailure(nativeRequestId);
+ }
+ };
client = new RealtimeAudioClient(
{
onTranscript: (text, isFinal, role) => {
@@ -271,6 +314,8 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
(typeof payload.error === "string" ? payload.error.trim() : "") ||
(typeof payload.reason === "string" ? payload.reason.trim() : "");
if (status === "audio_ready") {
+ connected = true;
+ initialNativeStartPending.current = false;
setState("connected");
const provider =
typeof payload.provider === "string" ? payload.provider : "";
@@ -354,6 +399,7 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
} else if (status === "audio_closed") {
void stop();
} else if (status === "provider_error" || status === "disconnected") {
+ acknowledgeStartupFailure();
if (local) {
endLocalCall(backendDetail || startFailed);
return;
@@ -379,9 +425,12 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
try {
await client.connect();
if (!isCurrent()) return;
+ connected = true;
+ initialNativeStartPending.current = false;
setState("connected");
} catch (cause) {
if (!isCurrent()) return;
+ acknowledgeStartupFailure();
clientRef.current = null;
void client.disconnect();
clearVoiceInputLevel("browser");
@@ -451,7 +500,7 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
}
pendingStart.current = null;
handledRequest.current = pending.id;
- void start();
+ void start({ requestId: pending.requestId });
};
document.addEventListener("visibilitychange", onVisible);
return () => document.removeEventListener("visibilitychange", onVisible);
@@ -464,25 +513,38 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
const event = events.find(e => e.name === "BrowserVoiceRequested");
if (!event || event.id === handledRequest.current || Date.now() - event.ts > 45_000) return;
const action = (event.payload as { action?: string })?.action;
+ const requestId = startupRequestId(event.payload);
if (action === "start") {
if (!realtimeAvailable) return;
+ if (requestId && nativeRequestRef.current && requestId !== nativeRequestRef.current &&
+ initialNativeStartPending.current && clientRef.current) {
+ // The native owner can replace its pending request without emitting an
+ // obsolete stop. Retire this unfinished generation, but leave the new
+ // event unhandled so the idle render starts its matching attempt.
+ // Established calls and browser-held calls are never replaced here.
+ void stop();
+ return;
+ }
if (canStartInBackground || document.visibilityState === "visible") {
handledRequest.current = event.id;
pendingStart.current = null;
- void start();
+ void start({ requestId });
} else {
// Parked, not handled: firing when the tab returns keeps a
// background wake from dying silently on the desktop side.
- pendingStart.current = { id: event.id, ts: event.ts };
+ pendingStart.current = { id: event.id, ts: event.ts, requestId };
}
return;
}
if (action === "stop") {
handledRequest.current = event.id;
+ // A retired native wake must never end a newer wake or a browser-held
+ // call. Legacy backends retain their existing uncorrelated stop behavior.
+ if (requestId && requestId !== nativeRequestRef.current && requestId !== pendingStart.current?.requestId) return;
pendingStart.current = null;
void stop();
}
- }, [events, browserAudio, wakeOwner, canStartInBackground, realtimeAvailable, start, stop]);
+ }, [events, browserAudio, wakeOwner, canStartInBackground, realtimeAvailable, start, stop, state]);
useEffect(() => {
// This surface owns BOTH directions while it is live: it holds the
@@ -508,6 +570,9 @@ export function BrowserRealtimeControl({ controlOnly = false }: { controlOnly?:
useEffect(
() => () => {
connectionGenerationRef.current += 1;
+ nativeRequestRef.current = null;
+ initialNativeStartPending.current = false;
+ pendingStart.current = null;
const client = clientRef.current;
clientRef.current = null;
setBrowserVoiceInputOwnership(false);
diff --git a/jarvis/ui/web/frontend/src/hooks/useBackgroundAgents.ts b/jarvis/ui/web/frontend/src/hooks/useBackgroundAgents.ts
index 66779a1ac3..cf68ff4a5f 100644
--- a/jarvis/ui/web/frontend/src/hooks/useBackgroundAgents.ts
+++ b/jarvis/ui/web/frontend/src/hooks/useBackgroundAgents.ts
@@ -6,6 +6,9 @@ import { useCallback, useEffect, useState } from "react";
*/
export interface BackgroundAgentsConfig {
keep_agents_running: boolean;
+ persistent_server: boolean;
+ server_url: string;
+ running_as_server: boolean;
background_only_at_login: boolean;
autostart_enabled: boolean;
/** This page is served by the windowless background service itself. */
@@ -17,7 +20,8 @@ export interface BackgroundAgentsConfig {
}
export type BackgroundAgentsPatch = Partial<
- Pick<BackgroundAgentsConfig, "keep_agents_running" | "background_only_at_login">
+ Pick<BackgroundAgentsConfig,
+ "keep_agents_running" | "background_only_at_login" | "persistent_server" | "server_url">
>;
/** Loads the background-agents settings and exposes save(). Mirrors useAutostart. */
@@ -50,7 +54,14 @@ export function useBackgroundAgents() {
body: JSON.stringify(patch),
});
const body = await res.json().catch(() => ({}));
- if (!res.ok) throw new Error(body.detail ?? `HTTP ${res.status}`);
+ if (!res.ok) {
+ const detail = body.detail;
+ const message = typeof detail === "string" ? detail
+ : Array.isArray(detail)
+ ? detail.map((item: { msg?: string }) => item.msg ?? "Invalid setting").join("; ")
+ : `HTTP ${res.status}`;
+ throw new Error(message);
+ }
setConfig(body as BackgroundAgentsConfig);
return body as BackgroundAgentsConfig;
}, []);
diff --git a/jarvis/ui/web/frontend/src/hooks/usePets.ts b/jarvis/ui/web/frontend/src/hooks/usePets.ts
index e667165461..6163616b10 100644
--- a/jarvis/ui/web/frontend/src/hooks/usePets.ts
+++ b/jarvis/ui/web/frontend/src/hooks/usePets.ts
@@ -54,6 +54,20 @@ export function useActivePet(): Pet | null {
return useMemo(() => pickActivePet(data), [data]);
}
+/**
+ * One pet by id, for an agent that wears it: undefined while the shared
+ * answer loads, null when this machine has no such pet (or the answer
+ * failed), so the caller can fall back to the agent's shape.
+ */
+export function usePetById(id: string | undefined): Pet | null | undefined {
+ const { data, isError } = useQuery({ queryKey: petKeys.all, queryFn: fetchPets, staleTime: Infinity, enabled: !!id });
+ return useMemo(() => {
+ if (!id) return null;
+ if (!data) return isError ? null : undefined;
+ return (Array.isArray(data.pets) ? data.pets : []).find((pet) => pet.id === id) ?? null;
+ }, [data, id, isError]);
+}
+
function useInvalidatePets() {
const qc = useQueryClient();
return () => qc.invalidateQueries({ queryKey: petKeys.all });
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/de.json b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/de.json
index 902438fb9e..94af0a1c94 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/de.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/de.json
@@ -92,27 +92,32 @@
"arrow_tapered": "Spitz zulaufend",
"arrow_classic": "Klassisch",
"arrow_double": "Zwei Spitzen",
- "library_title": "Deine Appshots",
- "library_subtitle": "Jeder Appshot, den du machst, und jede Bearbeitung, die du speicherst. Zieh einen in einen Chat, ein Terminal oder eine andere App.",
- "library_off": "Der Verlauf ist aus, neue Appshots werden nicht aufbewahrt. Schalte oben „Appshot-Verlauf behalten“ ein.",
- "library_count": "{0} Bilder",
+ "library_play": "Abspielen",
+ "library_video": "Video",
+ "library_filter_recordings": "Videos",
+ "library_empty_recordings": "Noch keine Bildschirmaufnahmen. Beende eine Aufnahme, dann erscheint sie hier.",
+ "library_delete_recording_hint": "Diese Bildschirmaufnahme löschen",
+ "library_title": "Deine Aufnahmen",
+ "library_subtitle": "Deine Screenshots, gespeicherten Bearbeitungen und Bildschirmaufnahmen an einem Ort. Klicke auf ein Bild zum Bearbeiten oder ein Video zum Abspielen.",
+ "library_off": "Der Verlauf ist aus, neue Screenshots werden nicht gespeichert. Bildschirmaufnahmen werden weiterhin hier gespeichert.",
+ "library_count": "{0} Einträge",
"library_filter_all": "Alle",
"library_filter_edited": "Bearbeitet",
"library_edited_badge": "Bearbeitet",
- "library_empty": "Noch keine Appshots. Mach einen mit dem Kürzel, dann erscheint er hier.",
+ "library_empty": "Noch keine Aufnahmen. Erstelle einen Screenshot oder beende eine Bildschirmaufnahme, damit sie hier erscheint.",
"library_empty_edited": "Noch keine bearbeiteten Appshots. Öffne einen, markiere etwas und speichere.",
"clipboard_label": "In die Zwischenablage kopieren",
"clipboard_hint": "Jeder Appshot per Tastenkürzel oder Knopf landet auch in der Zwischenablage und lässt sich sofort mit Strg+V einfügen (Cmd+V auf dem Mac).",
"library_label": "Appshot-Verlauf behalten",
- "library_hint": "Bewahrt jeden Appshot und jede Bearbeitung auf diesem Computer auf, die neuesten 500. Aus: Neues wird nicht aufbewahrt.",
+ "library_hint": "Behält die neuesten 500 Screenshots und ihre Bearbeitungen auf diesem Computer. Bildschirmaufnahmen bleiben gespeichert, bis du sie löschst.",
"library_edit": "Bearbeiten",
"library_delete": "Löschen",
"library_delete_edit_hint": "Nur diese bearbeitete Fassung löschen",
"library_delete_original_hint": "Diesen Appshot samt Bearbeitung löschen",
"library_delete_all": "Alle löschen",
- "library_delete_all_confirm": "Wirklich alle löschen?",
- "library_deleted_all": "Appshot-Verlauf gelöscht",
- "library_drag_hint": "Zieh ein Bild in einen Chat oder ein Terminal, um es anzuhängen. Klick es an, um es zu bearbeiten.",
+ "library_delete_all_confirm": "Alle Bilder und Videos löschen?",
+ "library_deleted_all": "Aufnahmeverlauf gelöscht",
+ "library_drag_hint": "Ziehe eine Aufnahme in einen Chat oder ein Terminal. Klicke auf ein Bild zum Bearbeiten oder ein Video zum Abspielen.",
"library_drag_out": "In eine andere App ziehen",
"library_show_more": "Mehr anzeigen",
"library_show_less": "Weniger anzeigen",
@@ -121,6 +126,22 @@
"keep_all": "Nie",
"keep_option": "Die neuesten {0} behalten",
"library_load_failed": "Die Appshots konnten nicht geladen werden: {0}",
- "library_open_failed": "Dieser Appshot konnte nicht geöffnet werden: {0}"
+ "library_open_failed": "Dieser Appshot konnte nicht geöffnet werden: {0}",
+ "video_title": "Video",
+ "video_loading": "Video wird geladen…",
+ "video_gone": "Dieses Video ist nicht mehr da.",
+ "video_play": "Abspielen",
+ "video_pause": "Pause",
+ "video_to_start": "Zum Anfang",
+ "video_to_end": "Zum Ende",
+ "video_speed": "Tempo",
+ "video_timeline": "Zeitleiste: ziehen zum Spulen, Ränder ziehen zum Kürzen",
+ "video_trim_start": "Anfang kürzen",
+ "video_trim_end": "Ende kürzen",
+ "video_trim_reset": "Kürzen zurücksetzen",
+ "video_clip_length": "Clip {0}",
+ "video_save_hint": "Video in Downloads speichern",
+ "video_saving": "Wird gespeichert…",
+ "video_shortcuts": "Leertaste abspielen · ←/→ spulen · I/O Anfang und Ende setzen"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/en.json b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/en.json
index 01e9b0a850..9b74e2e26b 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/en.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/en.json
@@ -92,27 +92,32 @@
"arrow_tapered": "Tapered",
"arrow_classic": "Classic",
"arrow_double": "Two heads",
- "library_title": "Your appshots",
- "library_subtitle": "Every appshot you take and every edit you save. Drag one into a chat, a terminal or another app.",
- "library_off": "History is off, so new appshots are not kept. Turn on “Keep appshot history” above.",
- "library_count": "{0} pictures",
+ "library_play": "Play",
+ "library_video": "Video",
+ "library_filter_recordings": "Recordings",
+ "library_empty_recordings": "No screen recordings yet. Finish a recording and it will appear here.",
+ "library_delete_recording_hint": "Delete this screen recording",
+ "library_title": "Your captures",
+ "library_subtitle": "Your screenshots, saved edits and screen recordings in one place. Click a picture to edit it or a video to play it.",
+ "library_off": "History is off, so new screenshots are not kept. Screen recordings are still saved here.",
+ "library_count": "{0} items",
"library_filter_all": "All",
"library_filter_edited": "Edited",
"library_edited_badge": "Edited",
- "library_empty": "No appshots yet. Take one with the shortcut and it shows up here.",
+ "library_empty": "No captures yet. Take a screenshot or finish a screen recording to see it here.",
"library_empty_edited": "No edited appshots yet. Open one, mark it up and save.",
"clipboard_label": "Copy to clipboard",
"clipboard_hint": "Every appshot you take by shortcut or button also goes on the clipboard, ready to paste with Ctrl+V (Cmd+V on a Mac).",
"library_label": "Keep appshot history",
- "library_hint": "Keeps every appshot and edit on this computer, the newest 500. Off: nothing new is kept.",
+ "library_hint": "Keeps the newest 500 screenshots and their edits on this computer. Screen recordings are always saved until you delete them.",
"library_edit": "Edit",
"library_delete": "Delete",
"library_delete_edit_hint": "Delete only this edited version",
"library_delete_original_hint": "Delete this appshot and its edit",
"library_delete_all": "Delete all",
- "library_delete_all_confirm": "Really delete all?",
- "library_deleted_all": "Appshot history deleted",
- "library_drag_hint": "Drag a picture into a chat or a terminal to attach it. Click it to edit.",
+ "library_delete_all_confirm": "Delete all pictures and videos?",
+ "library_deleted_all": "Capture history deleted",
+ "library_drag_hint": "Drag a capture into a chat or terminal to attach it. Click a picture to edit it or a video to play it.",
"library_drag_out": "Drag into another app",
"library_show_more": "Show more",
"library_show_less": "Show less",
@@ -121,6 +126,22 @@
"keep_all": "Never",
"keep_option": "Keep the newest {0}",
"library_load_failed": "The appshots could not be loaded: {0}",
- "library_open_failed": "This appshot could not be opened: {0}"
+ "library_open_failed": "This appshot could not be opened: {0}",
+ "video_title": "Video",
+ "video_loading": "Loading the video…",
+ "video_gone": "This video is no longer available.",
+ "video_play": "Play",
+ "video_pause": "Pause",
+ "video_to_start": "Back to start",
+ "video_to_end": "To the end",
+ "video_speed": "Speed",
+ "video_timeline": "Timeline: drag to scrub, drag the edges to trim",
+ "video_trim_start": "Trim start",
+ "video_trim_end": "Trim end",
+ "video_trim_reset": "Reset trim",
+ "video_clip_length": "Clip {0}",
+ "video_save_hint": "Save the video to Downloads",
+ "video_saving": "Saving…",
+ "video_shortcuts": "Space play · ←/→ seek · I/O set start and end"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/es.json b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/es.json
index 87eb727ea5..e8b490fdd8 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/es.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/es.json
@@ -92,27 +92,32 @@
"arrow_tapered": "Afilada",
"arrow_classic": "Clásica",
"arrow_double": "Dos puntas",
- "library_title": "Tus appshots",
- "library_subtitle": "Cada appshot que haces y cada edición que guardas. Arrastra uno a un chat, una terminal u otra app.",
- "library_off": "El historial está desactivado, así que los nuevos appshots no se guardan. Activa “Guardar historial de appshots” arriba.",
- "library_count": "{0} imágenes",
+ "library_play": "Reproducir",
+ "library_video": "Vídeo",
+ "library_filter_recordings": "Grabaciones",
+ "library_empty_recordings": "Aún no hay grabaciones de pantalla. Termina una grabación para verla aquí.",
+ "library_delete_recording_hint": "Eliminar esta grabación de pantalla",
+ "library_title": "Tus capturas",
+ "library_subtitle": "Tus capturas de pantalla, ediciones guardadas y grabaciones en un solo lugar. Pulsa una imagen para editarla o un vídeo para reproducirlo.",
+ "library_off": "El historial está desactivado: no se guardan nuevas capturas de pantalla. Las grabaciones se siguen guardando aquí.",
+ "library_count": "{0} elementos",
"library_filter_all": "Todos",
"library_filter_edited": "Editados",
"library_edited_badge": "Editado",
- "library_empty": "Aún no hay appshots. Haz uno con el atajo y aparecerá aquí.",
+ "library_empty": "Aún no hay capturas. Haz una captura de pantalla o termina una grabación para verla aquí.",
"library_empty_edited": "Aún no hay appshots editados. Abre uno, márcalo y guárdalo.",
"clipboard_label": "Copiar al portapapeles",
"clipboard_hint": "Cada appshot que tomes con un atajo o un botón también va al portapapeles, listo para pegar con Ctrl+V (Cmd+V en un Mac).",
"library_label": "Guardar historial de appshots",
- "library_hint": "Guarda cada appshot y cada edición en este equipo, los 500 más recientes. Desactivado: no se guarda nada nuevo.",
+ "library_hint": "Guarda las 500 capturas de pantalla más recientes y sus ediciones en este equipo. Las grabaciones se guardan hasta que las elimines.",
"library_edit": "Editar",
"library_delete": "Eliminar",
"library_delete_edit_hint": "Eliminar solo esta versión editada",
"library_delete_original_hint": "Eliminar este appshot y su edición",
"library_delete_all": "Eliminar todo",
- "library_delete_all_confirm": "¿Eliminar todo de verdad?",
- "library_deleted_all": "Historial de appshots eliminado",
- "library_drag_hint": "Arrastra una imagen a un chat o una terminal para adjuntarla. Haz clic para editarla.",
+ "library_delete_all_confirm": "¿Eliminar todas las imágenes y vídeos?",
+ "library_deleted_all": "Historial de capturas eliminado",
+ "library_drag_hint": "Arrastra una captura a un chat o terminal para adjuntarla. Pulsa una imagen para editarla o un vídeo para reproducirlo.",
"library_drag_out": "Arrastrar a otra app",
"library_show_more": "Mostrar más",
"library_show_less": "Mostrar menos",
@@ -121,6 +126,22 @@
"keep_all": "Nunca",
"keep_option": "Conservar las {0} más recientes",
"library_load_failed": "No se pudieron cargar los appshots: {0}",
- "library_open_failed": "No se pudo abrir este appshot: {0}"
+ "library_open_failed": "No se pudo abrir este appshot: {0}",
+ "video_title": "Vídeo",
+ "video_loading": "Cargando el vídeo…",
+ "video_gone": "Este vídeo ya no está disponible.",
+ "video_play": "Reproducir",
+ "video_pause": "Pausa",
+ "video_to_start": "Al inicio",
+ "video_to_end": "Al final",
+ "video_speed": "Velocidad",
+ "video_timeline": "Línea de tiempo: arrastra para buscar, arrastra los bordes para recortar",
+ "video_trim_start": "Recortar inicio",
+ "video_trim_end": "Recortar final",
+ "video_trim_reset": "Restablecer recorte",
+ "video_clip_length": "Clip {0}",
+ "video_save_hint": "Guardar el vídeo en Descargas",
+ "video_saving": "Guardando…",
+ "video_shortcuts": "Espacio reproducir · ←/→ buscar · I/O fijar inicio y final"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/zh.json b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/zh.json
index 85fd111545..b15ecbb34b 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/zh.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/appshot_editor/zh.json
@@ -92,6 +92,11 @@
"arrow_tapered": "渐细",
"arrow_classic": "经典",
"arrow_double": "双向",
+ "library_play": "播放",
+ "library_video": "视频",
+ "library_filter_recordings": "录制",
+ "library_empty_recordings": "还没有屏幕录制。完成一次录制后会显示在这里。",
+ "library_delete_recording_hint": "删除这段屏幕录制",
"library_title": "你的捕获",
"library_subtitle": "你的截图、已保存的编辑和屏幕录制都在这里。点击图片可编辑,点击视频可播放。",
"library_off": "历史记录已关闭,新截图不会保留。屏幕录制仍会保存在这里。",
@@ -121,6 +126,22 @@
"keep_all": "从不",
"keep_option": "保留最新的 {0} 个",
"library_load_failed": "无法加载截图:{0}",
- "library_open_failed": "无法打开这张截图:{0}"
+ "library_open_failed": "无法打开这张截图:{0}",
+ "video_title": "视频",
+ "video_loading": "正在加载视频…",
+ "video_gone": "这段视频已不可用。",
+ "video_play": "播放",
+ "video_pause": "暂停",
+ "video_to_start": "回到开头",
+ "video_to_end": "跳到结尾",
+ "video_speed": "速度",
+ "video_timeline": "时间轴:拖动可预览,拖动两端可剪辑",
+ "video_trim_start": "剪辑起点",
+ "video_trim_end": "剪辑终点",
+ "video_trim_reset": "重置剪辑",
+ "video_clip_length": "片段 {0}",
+ "video_save_hint": "将视频保存到“下载”文件夹",
+ "video_saving": "正在保存…",
+ "video_shortcuts": "空格 播放 · ←/→ 跳转 · I/O 设置起点和终点"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/computers/de.json b/jarvis/ui/web/frontend/src/i18n/locales/computers/de.json
index 9bdab19fb6..b271fdaa4e 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/computers/de.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/computers/de.json
@@ -363,6 +363,18 @@
"cx_key_only_body": "Führe diese Zeile einmal auf dem Server aus, in der Web-Konsole deines Hosting-Panels oder in einer SSH-Sitzung. Sie trägt den Schlüssel von {name} ein. Dann auf „Erneut versuchen“ drücken.",
"cx_key_only_line": "Zeile für den Server",
"cx_key_only_agent": "Oder lass das einen Coding-Agenten auf dem Server erledigen",
- "cx_try_again": "Erneut versuchen"
+ "cx_try_again": "Erneut versuchen",
+ "cx_ssh_host": "SSH-Host",
+ "cx_ssh_placeholder": "Host oder IP-Adresse eingeben",
+ "cx_invalid_port": "Gib einen Port zwischen 1 und 65535 ein.",
+ "cx_additional_options": "Weitere Möglichkeiten zum Hinzufügen",
+ "cx_connection_hint": "Wähle, wie du diesen Computer verbinden möchtest.",
+ "cx_connection_type": "Verbindungsart",
+ "cx_connection_server": "Server-Adresse",
+ "cx_connection_ssh": "SSH",
+ "cx_server_host": "Host",
+ "cx_pairing_code": "Pairing-Code",
+ "cx_pairing_hint": "Füge eine vollständige Pairing-URL ein, um beide Felder automatisch auszufüllen.",
+ "cx_pairing_unavailable": "Das Verbinden per Pairing-Code ist noch nicht verfügbar. Über SSH kannst du dich bereits verbinden."
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/computers/en.json b/jarvis/ui/web/frontend/src/i18n/locales/computers/en.json
index 879185dbe7..334066dd99 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/computers/en.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/computers/en.json
@@ -363,6 +363,18 @@
"cx_key_only_body": "Run this line once on the server, in your hosting panel's web console or any SSH session. It adds {name}'s key. Then press Try again.",
"cx_key_only_line": "Line for the server",
"cx_key_only_agent": "Or let a coding agent on the server do it",
- "cx_try_again": "Try again"
+ "cx_try_again": "Try again",
+ "cx_ssh_host": "SSH host",
+ "cx_ssh_placeholder": "Enter a host or IP address",
+ "cx_invalid_port": "Enter a port between 1 and 65535.",
+ "cx_additional_options": "More ways to add a computer",
+ "cx_connection_hint": "Choose how to connect this computer.",
+ "cx_connection_type": "Connection type",
+ "cx_connection_server": "Server address",
+ "cx_connection_ssh": "SSH",
+ "cx_server_host": "Host",
+ "cx_pairing_code": "Pairing code",
+ "cx_pairing_hint": "Paste a full pairing URL here to fill both fields automatically.",
+ "cx_pairing_unavailable": "Connecting with a pairing code is not available yet. Use SSH to connect now."
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/computers/es.json b/jarvis/ui/web/frontend/src/i18n/locales/computers/es.json
index 6faa32003e..ef8fc999f4 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/computers/es.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/computers/es.json
@@ -363,6 +363,18 @@
"cx_key_only_body": "Ejecuta esta línea una vez en el servidor, en la consola web de tu proveedor o en cualquier sesión SSH. Añade la clave de {name}. Luego pulsa Reintentar.",
"cx_key_only_line": "Línea para el servidor",
"cx_key_only_agent": "O deja que lo haga un agente de código en el servidor",
- "cx_try_again": "Reintentar"
+ "cx_try_again": "Reintentar",
+ "cx_ssh_host": "Host SSH",
+ "cx_ssh_placeholder": "Introduce un host o una dirección IP",
+ "cx_invalid_port": "Introduce un puerto entre 1 y 65535.",
+ "cx_additional_options": "Más formas de añadir un ordenador",
+ "cx_connection_hint": "Elige cómo conectar este ordenador.",
+ "cx_connection_type": "Tipo de conexión",
+ "cx_connection_server": "Dirección del servidor",
+ "cx_connection_ssh": "SSH",
+ "cx_server_host": "Host",
+ "cx_pairing_code": "Código de emparejamiento",
+ "cx_pairing_hint": "Pega una URL de emparejamiento completa para rellenar ambos campos automáticamente.",
+ "cx_pairing_unavailable": "La conexión mediante código de emparejamiento aún no está disponible. Usa SSH para conectarte ahora."
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/computers/zh.json b/jarvis/ui/web/frontend/src/i18n/locales/computers/zh.json
index 78cec89152..b0dd2fe8b9 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/computers/zh.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/computers/zh.json
@@ -363,6 +363,18 @@
"cx_key_only_body": "在服务器上运行一次这一行命令,可以在托管面板的网页控制台或任意 SSH 会话中运行。它会添加 {name} 的密钥。然后点击“重试”。",
"cx_key_only_line": "在服务器上运行的命令",
"cx_key_only_agent": "或者让服务器上的编程智能体来完成",
- "cx_try_again": "重试"
+ "cx_try_again": "重试",
+ "cx_ssh_host": "SSH 主机",
+ "cx_ssh_placeholder": "输入主机名或 IP 地址",
+ "cx_invalid_port": "请输入 1 到 65535 之间的端口。",
+ "cx_additional_options": "添加计算机的其他方式",
+ "cx_connection_hint": "选择连接这台计算机的方式。",
+ "cx_connection_type": "连接方式",
+ "cx_connection_server": "服务器地址",
+ "cx_connection_ssh": "SSH",
+ "cx_server_host": "主机",
+ "cx_pairing_code": "配对码",
+ "cx_pairing_hint": "粘贴完整的配对链接,即可自动填写两个字段。",
+ "cx_pairing_unavailable": "暂不支持通过配对码连接。现在可以使用 SSH 连接。"
}
}
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/de.json b/jarvis/ui/web/frontend/src/i18n/locales/de.json
index 7ae9683b11..5c9013b55f 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/de.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/de.json
@@ -893,6 +893,10 @@
"surface_jarvis_hint": "{name} — derselbe Assistent, den das Mikrofon erreicht: dein Gedächtnis, dein Wiki, deine Skills und Werkzeuge. Die Auswahl rechts ist das, worauf {name} in diesem Chat läuft.",
"surface_agent_hint": "Ein Coding-Agent, der an den Dateien in diesem Ordner arbeitet",
"pick_provider": "Anbieter",
+ "model_access": "Zugangsart",
+ "access_subscription": "Subscription",
+ "access_api": "API-Key",
+ "access_local": "Lokal",
"pick_model": "Modell",
"pick_effort": "Denkaufwand",
"pick_permission": "Berechtigungen",
@@ -3001,6 +3005,13 @@
"instant_start_declined_toast": "Berechtigung abgelehnt — {name} startet weiterhin über die Verknüpfung (evtl. verzögert)."
},
"background_agents": {
+ "server_title": "Unabhängiger Agentenserver",
+ "server_backend_update": "Starte die Desktop-App einmal neu, um die Unterstützung für den Agentenserver zu laden.",
+ "server_description": "Jarvis, Hermes und OpenClaw laufen auf einem Server weiter, auch wenn alle Clients geschlossen sind. Gilt ab dem nächsten Desktop-Start.",
+ "server_active": "Dieser unabhängige Server hält die Agenten am Laufen. Das Schließen dieses Clients beendet ihn nicht.",
+ "server_url_label": "Serveradresse (leer lassen für diesen Computer)",
+ "server_save": "Adresse speichern",
+ "server_hint": "Entfernte Server verwenden HTTPS und ihre eigene Anmeldung. Der Client nutzt Sprache über den Browser; lokales Aktivierungswort, Overlays und Gerätesteuerung benötigen den integrierten Desktop oder ein verfügbares Gerät.",
"title": "Agenten nach dem Schließen weiterlaufen lassen",
"description": "Wenn du {name} schließt, hält ein kleiner Hintergrund-Dienst deine Routinen und Chat-Kanäle (Telegram, Discord) am Laufen. Öffnest du die App, übernimmt sie wieder. Er startet nur, wenn es etwas weiterzuführen gibt.",
"keeps_now": "Beim Schließen laufen weiter: {what}.",
@@ -6201,6 +6212,30 @@
"hint": "Wähle Tools für die nächste Nachricht. Berechtigungen gelten weiterhin.",
"done": "Fertig"
},
+ "credential_card": {
+ "aria": "Sicheres Feld für Zugangsdaten",
+ "fallback_agent": "deinem Agenten",
+ "subtitle": "Wird sicher gespeichert und steht {agent} als Umgebungsvariable {env} zur Verfügung.",
+ "placeholder": "{label} hier einfügen",
+ "save": "Sicher speichern",
+ "saving": "Wird gespeichert…",
+ "decline": "Abbrechen",
+ "footer": "Sicher hinterlegt, wird deinem Agenten nie angezeigt",
+ "stays_open": "Dieses Feld bleibt offen, bis du speicherst oder abbrichst. Du kannst weiter chatten.",
+ "replace_note": "{agent} meldet, dass der gespeicherte Wert nicht mehr funktioniert.",
+ "saved": "{label} gespeichert · verfügbar als {env}",
+ "declined": "{label} nicht angegeben",
+ "timeout": "{label} wurde nicht rechtzeitig eingegeben",
+ "cancelled": "{label} nicht angegeben, der Durchlauf wurde beendet",
+ "invalid": "Dieser Wert kann nicht gespeichert werden. Füge die Zugangsdaten als eine einzige Zeile ein.",
+ "expired": "Diese Anfrage ist nicht mehr verfügbar. Bitte den Agenten um ein neues sicheres Eingabefeld.",
+ "invalid_token": "Der Dienst hat diesen Token abgelehnt. Prüfe ihn und versuche es erneut.",
+ "network_error": "Der Dienst ist nicht erreichbar. Prüfe deine Verbindung und versuche es erneut.",
+ "validation_timeout": "Die Anfrage hat zu lange gedauert. Das Feld bleibt offen; versuche es erneut.",
+ "storage_failed": "Der Token konnte nicht sicher gespeichert werden. Bitte versuche es erneut.",
+ "request_busy": "Dieser Token wird bereits geprüft. Warte kurz und versuche es dann erneut.",
+ "failed": "Die Zugangsdaten konnten nicht gespeichert werden."
+ },
"question_card": {
"aria": "Rückfrage des Agenten",
"title_one": "{agent} hat eine Frage",
@@ -6220,6 +6255,43 @@
"own_words": "deine eigene Antwort",
"unanswered": "Nicht beantwortet – der Durchlauf wurde beendet."
},
+ "bot_stage": {
+ "ponder": "Denkt nach…",
+ "glance": "Schaut sich um…",
+ "squint": "Konzentriert sich…",
+ "spark": "Hat eine Idee…",
+ "wink": "Heckt etwas aus…",
+ "roll": "Lässt es kreisen…",
+ "drowsy": "Grübelt…",
+ "beam": "Kommt voran…",
+ "skyread": "Überlegt…",
+ "doubt": "Prüft das nochmal…",
+ "note": "Schreibt sich etwas auf…",
+ "jot": "Macht sich eine Notiz…",
+ "scan": "Schlägt nach…",
+ "flip": "Blättert durch…",
+ "recall": "Erinnert sich…",
+ "browser": "Ist im Browser…",
+ "setup": "Richtet es ein…",
+ "tune": "Stellt es ein…",
+ "command": "Lässt einen Befehl laufen…",
+ "juggle": "Kümmert sich…",
+ "build": "Baut daran…",
+ "typing": "Tippt…",
+ "reading": "Liest deine Nachricht…",
+ "done": "Fertig"
+ },
+ "messenger_turn": {
+ "routine_created": "Erstellt: Routine",
+ "routine_updated": "Aktualisiert: Routine",
+ "routine_paused": "Pausiert: Routine",
+ "routine_resumed": "Fortgesetzt: Routine",
+ "routine_deleted": "Gelöscht: Routine",
+ "rule_saved": "Gemerkt: neue Regel",
+ "skill_saved": "Gespeichert: Skill",
+ "settings_updated": "Aktualisiert: Einstellungen",
+ "note_saved": "Notiert"
+ },
"thread_turn": {
"worked": "Gearbeitet",
"worked_for": "{duration} gearbeitet",
@@ -6609,6 +6681,39 @@
"office": "Jarvis Verse"
},
"subscriptions": {
+ "bank": {
+ "account_changed": "Das Login dieses Kontos hat sich geändert. Aktualisiere die Anzeige und bestätige das gewünschte Konto erneut.",
+ "signed_out": "Melde dich bei diesem Abo an und aktualisiere die Anzeige.",
+ "title": "Angesparte Resets",
+ "refresh": "Resets aktualisieren",
+ "loading": "Verfügbare Resets werden geprüft…",
+ "count": "{count} verfügbar",
+ "use": "Reset einsetzen",
+ "reset": "Reset eingesetzt. Verfügbare Nutzungsdaten wurden aktualisiert.",
+ "expires": "Läuft am {when} ab",
+ "expiry_unknown": "Kein Ablaufdatum gemeldet",
+ "count_only": "Der Anbieter meldet nur die Anzahl und wählt den nächsten verfügbaren Reset.",
+ "browser": "Dieser Anbieter verwaltet Resets auf seiner Nutzungsseite. Die Verfügbarkeit lässt sich hier nicht auslesen.",
+ "unsupported": "Dieser Anbieter stellt über das verbundene Abo keine Reset-Schnittstelle bereit.",
+ "unavailable": "Resets konnten nicht geladen werden. Aktualisiere die Anzeige oder CLI, oder öffne die Nutzungsseite des Anbieters.",
+ "confirm_title": "Reset bestätigen",
+ "confirm_text": "„{reset}“ für {account} einsetzen?",
+ "irreversible": "Dabei wird ein angesparter Reset unwiderruflich verbraucht. Berechtigte Nutzungslimits werden sofort zurückgesetzt; die nächsten Reset-Zeiten können sich ändern.",
+ "confirm": "Bestätigen und Reset einsetzen",
+ "cancel": "Abbrechen",
+ "pending": "Eine Reset-Anfrage ist offen. Prüfe dieselbe Anfrage, bevor du eine weitere startest.",
+ "retry": "Dieselbe Anfrage prüfen / wiederholen",
+ "using": "Reset wird eingesetzt…",
+ "uncertain": "Keine bestätigte Antwort. Die Wiederholung dieser Anfrage verbraucht keinen zweiten Reset.",
+ "storage_failed": "Die Anfrage konnte nicht sicher gespeichert werden. Es wurde kein Reset angefordert.",
+ "alreadyRedeemed": "Diese Anfrage hat den Reset bereits eingesetzt. Es wurde kein zweiter Reset verbraucht.",
+ "nothingToReset": "Kein berechtigtes Nutzungslimit muss zurückgesetzt werden. Es wurde nichts verbraucht.",
+ "noCredit": "Für dieses Konto sind keine angesparten Resets verfügbar.",
+ "manage": "Nutzungsseite öffnen",
+ "browser_account": "Prüfe im Browser, dass du als {account} angemeldet bist.",
+ "open_failed": "Der Browser konnte nicht geöffnet werden.",
+ "reset_name": "Nutzungsreset"
+ },
"hint": "Angemeldete Abos, nach Tool",
"refresh": "Verbrauch aktualisieren",
"empty": "Noch kein Abo angemeldet. Füge eines in den Workspace-Einstellungen hinzu.",
@@ -6944,7 +7049,18 @@
},
"appshots": {
"recording_title": "Bildschirmaufnahme",
- "recording_hint": "Wähle einen Bereich oder den gesamten Bildschirm. Das Video wird ohne Ton lokal gespeichert und nicht an den Assistenten gesendet. Dasselbe Tastenkürzel stoppt die Aufnahme.",
+ "recording_hint": "Wähle einen Bereich oder den gesamten Bildschirm. Das Video wird mit optionalem Systemton lokal gespeichert. Dasselbe Tastenkürzel stoppt die Aufnahme.",
+ "recording_quality": "Aufnahmequalität",
+ "recording_resolution": "Auflösung",
+ "recording_native": "Original · Monitor-Maximum",
+ "recording_fps": "Bildrate",
+ "recording_bitrate": "Video-Bitrate (Mbit/s)",
+ "recording_bitrate_hint": "1–100 Mbit/s Zielbitrate. Höhere Werte vergrößern die Datei; einfache Szenen können weniger benötigen.",
+ "recording_system_audio": "Systemton aufnehmen",
+ "recording_audio_hint": "Nimmt das Standard-Ausgabegerät auf, einschließlich Apps auf anderen Monitoren. Dein Mikrofon wird nicht aufgenommen. Bei einem Gerätewechsel ist eine neue Aufnahme nötig.",
+ "recording_monitor_hint": "Wähle den Bereich auf dem gewünschten Monitor. Dessen Auflösung und Bildrate begrenzen die Aufnahme; kleinere Bereiche werden nicht vergrößert. Bei unbekannter Bildwiederholrate gilt maximal 60 FPS. Die tatsächlichen Werte erscheinen beim Aufnahmestart.",
+ "recording_apply": "Aufnahmeeinstellungen speichern",
+ "recording_dropped_hint": "Unter Last wurden einzelne Bilder ausgelassen. Versuche eine niedrigere Bildrate oder Auflösung.",
"recording_start": "Aufnahmebereich wählen",
"recording_stop": "Stoppen und speichern",
"recording_cancel": "Abbrechen",
@@ -6972,6 +7088,9 @@
"region_shortcut_label": "Kürzel für einen Bereich",
"region_shortcut_hint": "Drücke {0} und ziehe dann über den Teil des Bildschirms, den du zeigen willst.",
"region_shortcut_hint_off": "Kein Kürzel. Nutze unten „Bereich wählen“.",
+ "recording_shortcut_label": "Kürzel für eine Videoaufnahme",
+ "recording_shortcut_hint": "Drücke {0} und wähle einen Bereich: Die Aufnahme startet. Noch einmal {0} stoppt und speichert sie.",
+ "recording_shortcut_hint_off": "Kein Kürzel. Nutze unten „Aufnahmebereich wählen“.",
"shortcut_change": "Ändern",
"shortcut_cancel": "Abbrechen",
"shortcut_clear": "Dieses Kürzel ausschalten",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/en.json b/jarvis/ui/web/frontend/src/i18n/locales/en.json
index 78eac3582a..ea84707d27 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/en.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/en.json
@@ -893,6 +893,10 @@
"surface_jarvis_hint": "{name} — the same assistant the microphone reaches: your memory, your wiki, your skills and tools. The picks to the right are what {name} runs on for this chat.",
"surface_agent_hint": "A coding agent working on the files in this folder",
"pick_provider": "Provider",
+ "model_access": "Access",
+ "access_subscription": "Subscription",
+ "access_api": "API key",
+ "access_local": "Local",
"pick_model": "Model",
"pick_effort": "Reasoning effort",
"pick_permission": "Permissions",
@@ -3001,6 +3005,13 @@
"instant_start_declined_toast": "Permission was declined — {name} still starts via the shortcut (may be delayed)."
},
"background_agents": {
+ "server_title": "Independent agent server",
+ "server_backend_update": "Restart the desktop app once to load agent server support.",
+ "server_description": "Run Jarvis, Hermes and OpenClaw on a server that stays running when every client closes. Applies to the next desktop launch.",
+ "server_active": "This independent server keeps agents running. Closing this client does not stop it.",
+ "server_url_label": "Server address (leave empty to run on this computer)",
+ "server_save": "Save address",
+ "server_hint": "Remote servers use HTTPS and their own sign-in. The client uses browser voice; local wake word, overlays and device control require the integrated desktop or an available device.",
"title": "Keep agents running after closing",
"description": "When you close {name}, a small background service keeps your routines and chat channels (Telegram, Discord) running. Opening the app takes over again. It only starts when there is something to keep.",
"keeps_now": "Closing now keeps: {what}.",
@@ -6201,6 +6212,30 @@
"hint": "Pick tools for your next message. Permissions still apply.",
"done": "Done"
},
+ "credential_card": {
+ "aria": "Secure credential field",
+ "fallback_agent": "your agent",
+ "subtitle": "Stored securely and available to {agent} as the environment variable {env}.",
+ "placeholder": "Paste {label} here",
+ "save": "Save securely",
+ "saving": "Saving…",
+ "decline": "Cancel",
+ "footer": "Stored securely, never shown to your agent",
+ "stays_open": "This field stays open until you save or cancel. You can keep chatting.",
+ "replace_note": "{agent} says the stored value no longer works.",
+ "saved": "{label} saved · available as {env}",
+ "declined": "{label} not provided",
+ "timeout": "{label} was not entered in time",
+ "cancelled": "{label} not provided, the turn ended",
+ "invalid": "This value cannot be stored. Paste the credential as a single line.",
+ "expired": "This request is no longer available. Ask the agent for a new secure field.",
+ "invalid_token": "The service rejected this token. Check that it is correct and try again.",
+ "network_error": "Could not reach the service. Check your connection and try again.",
+ "validation_timeout": "The request timed out. The field remains open; please try again.",
+ "storage_failed": "The token could not be saved securely. Please try again.",
+ "request_busy": "This token is already being checked. Please wait, then try again.",
+ "failed": "The credential could not be stored."
+ },
"question_card": {
"aria": "Question from the agent",
"title_one": "{agent} has a question",
@@ -6220,6 +6255,43 @@
"own_words": "your own answer",
"unanswered": "Not answered — the turn ended."
},
+ "bot_stage": {
+ "ponder": "Thinking…",
+ "glance": "Looking around…",
+ "squint": "Focusing…",
+ "spark": "Got an idea…",
+ "wink": "Cooking something up…",
+ "roll": "Turning it over…",
+ "drowsy": "Mulling it over…",
+ "beam": "Getting there…",
+ "skyread": "Considering…",
+ "doubt": "Double-checking…",
+ "note": "Writing something down…",
+ "jot": "Making a note…",
+ "scan": "Looking it up…",
+ "flip": "Leafing through…",
+ "recall": "Remembering…",
+ "browser": "In the browser…",
+ "setup": "Setting it up…",
+ "tune": "Adjusting it…",
+ "command": "Running a command…",
+ "juggle": "On it…",
+ "build": "Building it…",
+ "typing": "Typing…",
+ "reading": "Reading your message…",
+ "done": "Done"
+ },
+ "messenger_turn": {
+ "routine_created": "Created: Routine",
+ "routine_updated": "Updated: Routine",
+ "routine_paused": "Paused: Routine",
+ "routine_resumed": "Resumed: Routine",
+ "routine_deleted": "Deleted: Routine",
+ "rule_saved": "Noted: new rule",
+ "skill_saved": "Saved: Skill",
+ "settings_updated": "Updated: Settings",
+ "note_saved": "Noted"
+ },
"thread_turn": {
"worked": "Worked",
"worked_for": "Worked for {duration}",
@@ -6609,6 +6681,39 @@
"office": "Jarvis Verse"
},
"subscriptions": {
+ "bank": {
+ "account_changed": "This account login changed. Refresh and confirm the intended account again.",
+ "signed_out": "Sign in to this subscription, then refresh.",
+ "title": "Banked resets",
+ "refresh": "Refresh resets",
+ "loading": "Checking available resets…",
+ "count": "{count} available",
+ "use": "Use reset",
+ "reset": "Reset applied. Usage has been refreshed where available.",
+ "expires": "Expires {when}",
+ "expiry_unknown": "No expiry reported",
+ "count_only": "The provider reports a count only. It will choose the next available reset.",
+ "browser": "This provider manages resets on its usage page. Availability cannot be read here.",
+ "unsupported": "This provider does not expose banked resets through the connected subscription.",
+ "unavailable": "Resets could not be read. Refresh, update the CLI, or check the provider’s usage page.",
+ "confirm_title": "Confirm reset",
+ "confirm_text": "Use “{reset}” for {account}?",
+ "irreversible": "This spends one saved reset and cannot be undone. Eligible usage limits reset immediately; their next reset times may change.",
+ "confirm": "Confirm and use reset",
+ "cancel": "Cancel",
+ "pending": "A reset request is pending. Check the same request before starting another.",
+ "retry": "Check / retry same request",
+ "using": "Using reset…",
+ "uncertain": "No confirmed response. Retrying this request cannot spend a second reset.",
+ "storage_failed": "The request could not be saved safely. No reset was sent.",
+ "alreadyRedeemed": "This request already applied the reset. No second reset was spent.",
+ "nothingToReset": "No eligible usage limit needs a reset. Nothing was spent.",
+ "noCredit": "There are no banked resets available for this account.",
+ "manage": "Open provider usage",
+ "browser_account": "In the browser, check that you are signed in as {account}.",
+ "open_failed": "The browser could not be opened.",
+ "reset_name": "Usage reset"
+ },
"hint": "Signed-in subscriptions, by tool",
"refresh": "Refresh usage",
"empty": "No subscription is signed in yet. Add one in the workspace settings.",
@@ -6944,7 +7049,18 @@
},
"appshots": {
"recording_title": "Screen recording",
- "recording_hint": "Select an area or the entire screen. Records video without audio and saves it locally; nothing is sent to the assistant. Use the same shortcut to stop.",
+ "recording_hint": "Select an area or the entire screen. Saves the video locally with optional system audio. Use the same shortcut to stop.",
+ "recording_quality": "Recording quality",
+ "recording_resolution": "Resolution",
+ "recording_native": "Native · monitor maximum",
+ "recording_fps": "Frame rate",
+ "recording_bitrate": "Video bitrate (Mbps)",
+ "recording_bitrate_hint": "1–100 Mbps target bitrate. Higher values increase file size; simple scenes may use less.",
+ "recording_system_audio": "Include system audio",
+ "recording_audio_hint": "Records the default output device, including apps on other monitors. Your microphone is not recorded. Changing devices requires a new recording.",
+ "recording_monitor_hint": "Choose the area on the monitor you want to record. Resolution and FPS are capped by that monitor; smaller areas are not enlarged. Unknown refresh rates are capped at 60 FPS. The actual values appear when recording starts.",
+ "recording_apply": "Save recording settings",
+ "recording_dropped_hint": "Some frames were skipped under load. Try a lower frame rate or resolution.",
"recording_start": "Select recording area",
"recording_stop": "Stop and save",
"recording_cancel": "Cancel",
@@ -6972,6 +7088,9 @@
"region_shortcut_label": "Area shortcut",
"region_shortcut_hint": "Press {0}, then drag over the part of the screen you want to show.",
"region_shortcut_hint_off": "No shortcut. Use \"Select area\" below instead.",
+ "recording_shortcut_label": "Video recording shortcut",
+ "recording_shortcut_hint": "Press {0} and choose an area to start recording. Press {0} again to stop and save.",
+ "recording_shortcut_hint_off": "No shortcut. Use “Select recording area” below.",
"shortcut_change": "Change",
"shortcut_cancel": "Cancel",
"shortcut_clear": "Turn this shortcut off",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/es.json b/jarvis/ui/web/frontend/src/i18n/locales/es.json
index 77d82a4669..c0b344bc33 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/es.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/es.json
@@ -893,6 +893,10 @@
"surface_jarvis_hint": "{name}: el mismo asistente que atiende el micrófono — tu memoria, tu wiki, tus skills y herramientas. Lo que elijas a la derecha es sobre lo que corre {name} en este chat.",
"surface_agent_hint": "Un agente de código que trabaja en los archivos de esta carpeta",
"pick_provider": "Proveedor",
+ "model_access": "Acceso",
+ "access_subscription": "Suscripción",
+ "access_api": "Clave API",
+ "access_local": "Local",
"pick_model": "Modelo",
"pick_effort": "Esfuerzo de razonamiento",
"pick_permission": "Permisos",
@@ -3001,6 +3005,13 @@
"instant_start_declined_toast": "Se denegó el permiso: {name} sigue iniciándose mediante el acceso directo (puede retrasarse)."
},
"background_agents": {
+ "server_title": "Servidor de agentes independiente",
+ "server_backend_update": "Reinicia la aplicación de escritorio una vez para cargar la compatibilidad con el servidor de agentes.",
+ "server_description": "Ejecuta Jarvis, Hermes y OpenClaw en un servidor que sigue activo al cerrar todos los clientes. Se aplica al próximo inicio de la aplicación de escritorio.",
+ "server_active": "Este servidor independiente mantiene los agentes en marcha. Cerrar este cliente no detiene el servidor.",
+ "server_url_label": "Dirección del servidor (déjala vacía para usar este equipo)",
+ "server_save": "Guardar dirección",
+ "server_hint": "Los servidores remotos usan HTTPS y su propio inicio de sesión. El cliente ofrece voz mediante el navegador; la palabra de activación local, las superposiciones y el control del dispositivo requieren el escritorio integrado o un dispositivo disponible.",
"title": "Mantener los agentes activos al cerrar",
"description": "Al cerrar {name}, un pequeño servicio en segundo plano mantiene activas tus rutinas y canales de chat (Telegram, Discord). Al abrir la app, esta vuelve a tomar el control. Solo se inicia cuando hay algo que mantener.",
"keeps_now": "Al cerrar ahora se mantiene: {what}.",
@@ -6201,6 +6212,30 @@
"hint": "Elige herramientas para el próximo mensaje. Se siguen aplicando los permisos.",
"done": "Listo"
},
+ "credential_card": {
+ "aria": "Campo seguro para credenciales",
+ "fallback_agent": "tu agente",
+ "subtitle": "Se guarda de forma segura y {agent} lo recibe como la variable de entorno {env}.",
+ "placeholder": "Pega {label} aquí",
+ "save": "Guardar de forma segura",
+ "saving": "Guardando…",
+ "decline": "Cancelar",
+ "footer": "Guardado de forma segura, tu agente nunca lo ve",
+ "stays_open": "Este campo permanece abierto hasta que guardes o canceles. Puedes seguir chateando.",
+ "replace_note": "{agent} indica que el valor guardado ya no funciona.",
+ "saved": "{label} guardado · disponible como {env}",
+ "declined": "{label} no proporcionado",
+ "timeout": "{label} no se introdujo a tiempo",
+ "cancelled": "{label} no proporcionado, el turno terminó",
+ "invalid": "Este valor no se puede guardar. Pega la credencial en una sola línea.",
+ "expired": "Esta solicitud ya no está disponible. Pide al agente un nuevo campo seguro.",
+ "invalid_token": "El servicio rechazó este token. Compruébalo e inténtalo de nuevo.",
+ "network_error": "No se pudo conectar con el servicio. Comprueba tu conexión e inténtalo de nuevo.",
+ "validation_timeout": "La solicitud tardó demasiado. El campo sigue abierto; inténtalo de nuevo.",
+ "storage_failed": "No se pudo guardar el token de forma segura. Inténtalo de nuevo.",
+ "request_busy": "Este token ya se está comprobando. Espera un momento e inténtalo de nuevo.",
+ "failed": "No se pudo guardar la credencial."
+ },
"question_card": {
"aria": "Pregunta del agente",
"title_one": "{agent} tiene una pregunta",
@@ -6220,6 +6255,43 @@
"own_words": "tu propia respuesta",
"unanswered": "Sin responder: el turno terminó."
},
+ "bot_stage": {
+ "ponder": "Pensando…",
+ "glance": "Mirando alrededor…",
+ "squint": "Concentrándose…",
+ "spark": "Tiene una idea…",
+ "wink": "Tramando algo…",
+ "roll": "Dándole vueltas…",
+ "drowsy": "Meditándolo…",
+ "beam": "Va por buen camino…",
+ "skyread": "Considerándolo…",
+ "doubt": "Revisándolo…",
+ "note": "Tomando nota…",
+ "jot": "Apuntándolo…",
+ "scan": "Buscando…",
+ "flip": "Hojeando…",
+ "recall": "Recordando…",
+ "browser": "En el navegador…",
+ "setup": "Configurándolo…",
+ "tune": "Ajustándolo…",
+ "command": "Ejecutando un comando…",
+ "juggle": "En ello…",
+ "build": "Construyéndolo…",
+ "typing": "Escribiendo…",
+ "reading": "Leyendo tu mensaje…",
+ "done": "Listo"
+ },
+ "messenger_turn": {
+ "routine_created": "Creada: rutina",
+ "routine_updated": "Actualizada: rutina",
+ "routine_paused": "Pausada: rutina",
+ "routine_resumed": "Reanudada: rutina",
+ "routine_deleted": "Eliminada: rutina",
+ "rule_saved": "Anotado: nueva regla",
+ "skill_saved": "Guardado: skill",
+ "settings_updated": "Actualizado: ajustes",
+ "note_saved": "Anotado"
+ },
"thread_turn": {
"worked": "Trabajó",
"worked_for": "Trabajó durante {duration}",
@@ -6609,6 +6681,39 @@
"office": "Jarvis Verse"
},
"subscriptions": {
+ "bank": {
+ "account_changed": "La cuenta conectada ha cambiado. Actualiza y confirma de nuevo la cuenta deseada.",
+ "signed_out": "Inicia sesión en esta suscripción y actualiza.",
+ "title": "Reinicios guardados",
+ "refresh": "Actualizar reinicios",
+ "loading": "Consultando reinicios disponibles…",
+ "count": "{count} disponibles",
+ "use": "Usar reinicio",
+ "reset": "Reinicio aplicado. Se actualizaron los datos de uso disponibles.",
+ "expires": "Caduca {when}",
+ "expiry_unknown": "Sin fecha de caducidad indicada",
+ "count_only": "El proveedor solo indica la cantidad y elegirá el próximo reinicio disponible.",
+ "browser": "Este proveedor gestiona los reinicios en su página de uso. Aquí no se puede consultar la disponibilidad.",
+ "unsupported": "Este proveedor no ofrece reinicios guardados mediante la suscripción conectada.",
+ "unavailable": "No se pudieron consultar los reinicios. Actualiza la vista o la CLI, o abre la página de uso del proveedor.",
+ "confirm_title": "Confirmar reinicio",
+ "confirm_text": "¿Usar «{reset}» para {account}?",
+ "irreversible": "Esto consume un reinicio guardado y no se puede deshacer. Los límites aptos se restablecen de inmediato; las próximas fechas de reinicio pueden cambiar.",
+ "confirm": "Confirmar y usar reinicio",
+ "cancel": "Cancelar",
+ "pending": "Hay una solicitud pendiente. Comprueba la misma solicitud antes de iniciar otra.",
+ "retry": "Comprobar / repetir la misma solicitud",
+ "using": "Aplicando reinicio…",
+ "uncertain": "No hay respuesta confirmada. Repetir esta solicitud no consume otro reinicio.",
+ "storage_failed": "No se pudo guardar la solicitud. No se envió ningún reinicio.",
+ "alreadyRedeemed": "Esta solicitud ya aplicó el reinicio. No se consumió otro.",
+ "nothingToReset": "Ningún límite apto necesita reiniciarse. No se consumió nada.",
+ "noCredit": "No hay reinicios guardados disponibles para esta cuenta.",
+ "manage": "Abrir uso del proveedor",
+ "browser_account": "Comprueba que has iniciado sesión como {account} en el navegador.",
+ "open_failed": "No se pudo abrir el navegador.",
+ "reset_name": "Reinicio de uso"
+ },
"hint": "Suscripciones con sesión iniciada, por herramienta",
"refresh": "Actualizar uso",
"empty": "Aún no hay ninguna suscripción con sesión iniciada. Añade una en los ajustes del espacio de trabajo.",
@@ -6944,7 +7049,18 @@
},
"appshots": {
"recording_title": "Grabación de pantalla",
- "recording_hint": "Selecciona un área o la pantalla completa. El vídeo se guarda localmente sin audio y no se envía al asistente. El mismo atajo detiene la grabación.",
+ "recording_hint": "Selecciona un área o la pantalla completa. El vídeo se guarda localmente con audio del sistema opcional. El mismo atajo detiene la grabación.",
+ "recording_quality": "Calidad de grabación",
+ "recording_resolution": "Resolución",
+ "recording_native": "Original · máximo del monitor",
+ "recording_fps": "Fotogramas por segundo",
+ "recording_bitrate": "Tasa de bits del vídeo (Mbps)",
+ "recording_bitrate_hint": "Tasa objetivo de 1–100 Mbps. Los valores altos aumentan el tamaño; las escenas simples pueden necesitar menos.",
+ "recording_system_audio": "Incluir audio del sistema",
+ "recording_audio_hint": "Graba el dispositivo de salida predeterminado, incluidas las apps de otros monitores. No graba el micrófono. Cambiar de dispositivo requiere una nueva grabación.",
+ "recording_monitor_hint": "Elige el área en el monitor que quieras grabar. Su resolución y frecuencia limitan la grabación; las áreas pequeñas no se amplían. Si se desconoce la frecuencia, se limita a 60 FPS. Los valores reales aparecen al comenzar.",
+ "recording_apply": "Guardar ajustes de grabación",
+ "recording_dropped_hint": "Se omitieron fotogramas por la carga del equipo. Prueba con menos FPS o una resolución menor.",
"recording_start": "Seleccionar área",
"recording_stop": "Detener y guardar",
"recording_cancel": "Cancelar",
@@ -6972,6 +7088,9 @@
"region_shortcut_label": "Atajo para una zona",
"region_shortcut_hint": "Pulsa {0} y arrastra sobre la parte de la pantalla que quieres mostrar.",
"region_shortcut_hint_off": "Sin atajo. Usa «Elegir zona» más abajo.",
+ "recording_shortcut_label": "Atajo para grabar vídeo",
+ "recording_shortcut_hint": "Pulsa {0} y elige una zona para empezar a grabar. Pulsa {0} otra vez para detener y guardar.",
+ "recording_shortcut_hint_off": "Sin atajo. Usa «Seleccionar área» abajo.",
"shortcut_change": "Cambiar",
"shortcut_cancel": "Cancelar",
"shortcut_clear": "Desactivar este atajo",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/society/de.json b/jarvis/ui/web/frontend/src/i18n/locales/society/de.json
index 523d5184ee..d8db162707 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/society/de.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/society/de.json
@@ -1,5 +1,9 @@
{
"society": {
+ "card": {
+ "options_collapse": "Optionen einklappen",
+ "options_expand": "Optionen ausklappen"
+ },
"browser_profiles": {
"profile_unavailable": "Profil getrennt — bitte ein Profil auswählen",
"chrome_offline": "Chrome nicht verbunden",
@@ -887,6 +891,12 @@
"dots": "Runde Augen",
"lines": "Strichaugen",
"visible": "Auf der Map folgen",
+ "look": "Aussehen",
+ "look_shape": "Form",
+ "look_pet": "Pet",
+ "pet": "Pet",
+ "pet_hint": "Der Agent trägt dieses Pet überall: in Chats, bei der Arbeit und als Begleiter in der Verse.",
+ "no_pets": "Noch keine Pets. Lege eines unter „Meine Pets“ an.",
"shapes": {
"circle": "Kreis",
"squircle": "Weiches Quadrat",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/society/en.json b/jarvis/ui/web/frontend/src/i18n/locales/society/en.json
index 78fb619e37..2acf633599 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/society/en.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/society/en.json
@@ -1,5 +1,9 @@
{
"society": {
+ "card": {
+ "options_collapse": "Collapse Options",
+ "options_expand": "Expand Options"
+ },
"browser_profiles": {
"profile_unavailable": "Profile disconnected — choose a profile",
"chrome_offline": "Chrome disconnected",
@@ -887,6 +891,12 @@
"dots": "Round eyes",
"lines": "Line eyes",
"visible": "Follow on the map",
+ "look": "Look",
+ "look_shape": "Shape",
+ "look_pet": "Pet",
+ "pet": "Pet",
+ "pet_hint": "The agent wears this pet everywhere: in chats, while it works and as its companion in the Verse.",
+ "no_pets": "No pets yet. Add one under My Pets.",
"shapes": {
"circle": "Circle",
"squircle": "Soft square",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/society/es.json b/jarvis/ui/web/frontend/src/i18n/locales/society/es.json
index 0846d92145..6a87dd1b21 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/society/es.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/society/es.json
@@ -1,5 +1,9 @@
{
"society": {
+ "card": {
+ "options_collapse": "Contraer opciones",
+ "options_expand": "Expandir opciones"
+ },
"browser_profiles": {
"profile_unavailable": "Perfil desconectado — elige un perfil",
"chrome_offline": "Chrome desconectado",
@@ -887,6 +891,12 @@
"dots": "Ojos redondos",
"lines": "Ojos de línea",
"visible": "Seguir en el mapa",
+ "look": "Aspecto",
+ "look_shape": "Forma",
+ "look_pet": "Mascota",
+ "pet": "Mascota",
+ "pet_hint": "El agente lleva esta mascota en todas partes: en los chats, mientras trabaja y como compañero en el Verse.",
+ "no_pets": "Aún no hay mascotas. Añade una en Mis mascotas.",
"shapes": {
"circle": "Círculo",
"squircle": "Cuadrado suave",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/society/zh.json b/jarvis/ui/web/frontend/src/i18n/locales/society/zh.json
index 8fd5ac9a3d..0ee8d4c6f3 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/society/zh.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/society/zh.json
@@ -1,5 +1,9 @@
{
"society": {
+ "card": {
+ "options_collapse": "收起选项",
+ "options_expand": "展开选项"
+ },
"browser_profiles": {
"profile_unavailable": "配置文件已断开 — 请选择一个配置文件",
"chrome_offline": "Chrome 已断开",
@@ -887,6 +891,12 @@
"dots": "圆眼",
"lines": "线条眼",
"visible": "在地图上跟随",
+ "look": "外观",
+ "look_shape": "形状",
+ "look_pet": "宠物",
+ "pet": "宠物",
+ "pet_hint": "这个智能体在所有地方都会以这只宠物出现:聊天中、工作时,以及在 Verse 中作为伙伴。",
+ "no_pets": "还没有宠物。请在“我的宠物”中添加。",
"shapes": {
"circle": "圆形",
"squircle": "圆角方形",
diff --git a/jarvis/ui/web/frontend/src/i18n/locales/zh.json b/jarvis/ui/web/frontend/src/i18n/locales/zh.json
index 8bf8958a27..06ab0592d4 100644
--- a/jarvis/ui/web/frontend/src/i18n/locales/zh.json
+++ b/jarvis/ui/web/frontend/src/i18n/locales/zh.json
@@ -893,6 +893,10 @@
"surface_jarvis_hint": "{name}——和麦克风那头是同一个助手:你的记忆、知识库、技能和工具。右侧的选项就是 {name} 在本次对话中使用的配置。",
"surface_agent_hint": "处理此文件夹中文件的编程智能体",
"pick_provider": "服务商",
+ "model_access": "接入方式",
+ "access_subscription": "订阅",
+ "access_api": "API 密钥",
+ "access_local": "本地",
"pick_model": "模型",
"pick_effort": "推理强度",
"pick_permission": "权限",
@@ -3003,6 +3007,13 @@
"instant_start_declined_toast": "权限被拒绝,{name}仍会通过快捷方式启动(可能会延迟)。"
},
"background_agents": {
+ "server_title": "独立代理服务器",
+ "server_backend_update": "重启一次桌面应用,以加载代理服务器支持。",
+ "server_description": "让 Jarvis、Hermes 和 OpenClaw 在服务器上持续运行,即使所有客户端均已关闭。下次启动桌面应用时生效。",
+ "server_active": "此独立服务器会让代理持续运行。关闭此客户端不会停止服务器。",
+ "server_url_label": "服务器地址(留空则在本机运行)",
+ "server_save": "保存地址",
+ "server_hint": "远程服务器使用 HTTPS 并需单独登录。客户端通过浏览器提供语音功能;本地唤醒词、悬浮界面和设备控制仍需要集成桌面应用或可用设备。",
"title": "关闭后保持智能体运行",
"description": "关闭{name}后,一个小型后台服务会让你的例程和聊天渠道(Telegram、Discord)继续运行。打开应用后会重新接管。只有在有需要保持运行的内容时它才会启动。",
"keeps_now": "现在关闭会保留:{what}。",
@@ -6207,6 +6218,30 @@
"hint": "为下一条消息选择工具。权限规则仍然适用。",
"done": "完成"
},
+ "credential_card": {
+ "aria": "安全凭据输入框",
+ "fallback_agent": "你的智能体",
+ "subtitle": "安全保存,并以环境变量 {env} 的形式提供给{agent}。",
+ "placeholder": "在此粘贴{label}",
+ "save": "安全保存",
+ "saving": "正在保存…",
+ "decline": "取消",
+ "footer": "已安全保存,绝不会显示给你的智能体",
+ "stays_open": "此输入框会保持打开,直到你保存或取消。你可以继续聊天。",
+ "replace_note": "{agent}表示已保存的值不再有效。",
+ "saved": "{label}已保存 · 可通过 {env} 使用",
+ "declined": "未提供{label}",
+ "timeout": "{label}未及时输入",
+ "cancelled": "未提供{label},本轮已结束",
+ "invalid": "无法保存此值。请将凭据粘贴为单行。",
+ "expired": "此请求已不可用。请让智能体提供新的安全输入框。",
+ "invalid_token": "服务拒绝了此令牌。请检查后重试。",
+ "network_error": "无法连接到服务。请检查网络连接后重试。",
+ "validation_timeout": "请求超时。输入框会保持打开,请重试。",
+ "storage_failed": "无法安全保存令牌。请重试。",
+ "request_busy": "正在检查此令牌。请稍等后重试。",
+ "failed": "无法保存凭据。"
+ },
"question_card": {
"aria": "来自智能体的问题",
"title_one": "{agent} 有一个问题",
@@ -6226,6 +6261,43 @@
"own_words": "你自己的回答",
"unanswered": "未回答——本轮已结束。"
},
+ "bot_stage": {
+ "ponder": "正在思考…",
+ "glance": "四处看看…",
+ "squint": "正在专注…",
+ "spark": "有个主意…",
+ "wink": "在酝酿什么…",
+ "roll": "正在琢磨…",
+ "drowsy": "正在反复思量…",
+ "beam": "快好了…",
+ "skyread": "正在斟酌…",
+ "doubt": "正在复查…",
+ "note": "正在记下…",
+ "jot": "正在做笔记…",
+ "scan": "正在查找…",
+ "flip": "正在翻阅…",
+ "recall": "正在回忆…",
+ "browser": "正在浏览器中…",
+ "setup": "正在设置…",
+ "tune": "正在调整…",
+ "command": "正在运行命令…",
+ "juggle": "正在处理…",
+ "build": "正在搭建…",
+ "typing": "正在输入…",
+ "reading": "正在阅读你的消息…",
+ "done": "完成"
+ },
+ "messenger_turn": {
+ "routine_created": "已创建:例程",
+ "routine_updated": "已更新:例程",
+ "routine_paused": "已暂停:例程",
+ "routine_resumed": "已恢复:例程",
+ "routine_deleted": "已删除:例程",
+ "rule_saved": "已记下:新规则",
+ "skill_saved": "已保存:技能",
+ "settings_updated": "已更新:设置",
+ "note_saved": "已记下"
+ },
"thread_turn": {
"worked": "已工作",
"worked_for": "已工作 {duration}",
@@ -6615,6 +6687,39 @@
"office": "Jarvis Verse"
},
"subscriptions": {
+ "bank": {
+ "account_changed": "此账户的登录已更改。请刷新并重新确认目标账户。",
+ "signed_out": "请登录此订阅后刷新。",
+ "title": "已存重置次数",
+ "refresh": "刷新重置次数",
+ "loading": "正在检查可用重置…",
+ "count": "{count} 次可用",
+ "use": "使用重置",
+ "reset": "已使用重置,并刷新了可用的用量数据。",
+ "expires": "到期时间:{when}",
+ "expiry_unknown": "未提供到期时间",
+ "count_only": "提供商仅提供次数,将选择下一个可用重置。",
+ "browser": "此提供商在用量页面管理重置,此处无法读取可用次数。",
+ "unsupported": "此提供商未通过已连接的订阅提供重置接口。",
+ "unavailable": "无法读取重置。请刷新、更新 CLI 或查看提供商的用量页面。",
+ "confirm_title": "确认重置",
+ "confirm_text": "为 {account} 使用“{reset}”?",
+ "irreversible": "此操作会消耗一次已存重置且无法撤销。符合条件的用量限制将立即重置,下次重置时间可能变化。",
+ "confirm": "确认并使用重置",
+ "cancel": "取消",
+ "pending": "有一项待确认的重置请求。开始新请求前,请检查原请求。",
+ "retry": "检查/重试原请求",
+ "using": "正在使用重置…",
+ "uncertain": "未收到确认响应。重试此请求不会消耗第二次重置。",
+ "storage_failed": "无法安全保存请求。未发送重置请求。",
+ "alreadyRedeemed": "此请求已使用过重置,未消耗第二次重置。",
+ "nothingToReset": "没有符合条件的限制需要重置,未消耗重置次数。",
+ "noCredit": "此账户没有可用的已存重置。",
+ "manage": "打开提供商用量页面",
+ "browser_account": "请确认浏览器登录的账户为 {account}。",
+ "open_failed": "无法打开浏览器。",
+ "reset_name": "用量重置"
+ },
"hint": "已登录的订阅,按工具分组",
"refresh": "刷新用量",
"empty": "还没有登录任何订阅。请在工作区设置中添加。",
@@ -6951,6 +7056,17 @@
"appshots": {
"recording_title": "屏幕录制",
"recording_hint": "选择一个区域或整个屏幕。视频保存在本地,可选择包含系统音频。使用同一个快捷键停止。",
+ "recording_quality": "录制质量",
+ "recording_resolution": "分辨率",
+ "recording_native": "原生 · 显示器最高",
+ "recording_fps": "帧率",
+ "recording_bitrate": "视频码率(Mbps)",
+ "recording_bitrate_hint": "目标码率 1–100 Mbps。数值越高,文件越大;简单画面可能用得更少。",
+ "recording_system_audio": "包含系统音频",
+ "recording_audio_hint": "录制默认输出设备,包括其他显示器上的应用。不会录制你的麦克风。更换设备需要重新开始录制。",
+ "recording_monitor_hint": "在要录制的显示器上选择区域。分辨率和 FPS 以该显示器为上限;较小的区域不会被放大。刷新率未知时上限为 60 FPS。实际数值会在开始录制时显示。",
+ "recording_apply": "保存录制设置",
+ "recording_dropped_hint": "负载过高时跳过了部分帧。请尝试降低帧率或分辨率。",
"recording_start": "选择录制区域",
"recording_stop": "停止并保存",
"recording_cancel": "取消",
@@ -6978,6 +7094,9 @@
"region_shortcut_label": "区域快捷键",
"region_shortcut_hint": "按 {0},然后拖选你想展示的屏幕部分。",
"region_shortcut_hint_off": "没有快捷键。请改用下方的“选择区域”。",
+ "recording_shortcut_label": "视频录制快捷键",
+ "recording_shortcut_hint": "按 {0} 并选择一个区域开始录制。再按一次 {0} 停止并保存。",
+ "recording_shortcut_hint_off": "没有快捷键。请使用下方的“选择录制区域”。",
"shortcut_change": "更改",
"shortcut_cancel": "取消",
"shortcut_clear": "关闭此快捷键",
diff --git a/jarvis/ui/web/frontend/src/index.css b/jarvis/ui/web/frontend/src/index.css
index cccd900132..b3448ca02d 100644
--- a/jarvis/ui/web/frontend/src/index.css
+++ b/jarvis/ui/web/frontend/src/index.css
@@ -578,6 +578,10 @@ label,
Memory = the wiki) carry their own gradient pair. */
--chat-bubble-out: 212 92% 45%;
--chat-bubble-out-foreground: 0 0% 100%;
+ /* An agent chat's incoming bubble: a neutral grey lift that sits
+ beside any agent colour the outgoing bubble wears. */
+ --chat-bubble-in: 0 0% 93%; /* #EDEDED */
+ --chat-bubble-in-foreground: 0 0% 9%;
--soul-from: 347 83% 52%;
--soul-to: 14 92% 56%;
--memory-from: 224 86% 52%;
@@ -606,6 +610,9 @@ label,
carried by weight and by --foreground-strong instead. 14.3:1 on paper. */
--primary: 0 0% 9%;
--primary-foreground: 0 0% 100%;
+ /* The chat Send control keeps its white fill in both appearances. */
+ --chat-send-background: 0 0% 100%;
+ --chat-send-foreground: 0 0% 9%;
--accent: 212 92% 45%; /* #096CDC — the SIGNAL: links, focus, active, selected, info */
--accent-foreground: 0 0% 100%;
--ring: 212 92% 45%;
@@ -777,6 +784,8 @@ label,
--user-bubble-foreground: 0 0% 98%;
--chat-bubble-out: 212 100% 56%;
--chat-bubble-out-foreground: 0 0% 100%;
+ --chat-bubble-in: 0 0% 15%; /* #262626 — see :root */
+ --chat-bubble-in-foreground: 0 0% 95%;
--soul-from: 347 86% 56%;
--soul-to: 14 94% 58%;
--memory-from: 226 92% 60%;
@@ -1065,6 +1074,23 @@ body {
color: inherit;
}
+ /* An agent chat's bubbles: the agent's own reply in neutral grey; the
+ person's bubble is tinted inline with the agent's colour (bubbleTint). */
+ .jarvis-chat-in {
+ background: hsl(var(--chat-bubble-in));
+ color: hsl(var(--chat-bubble-in-foreground));
+ }
+ .jarvis-chat-tinted a {
+ color: inherit;
+ }
+ /* A capability chip in a tinted bubble takes the bubble's ink: a brand
+ hue on an arbitrary agent colour cannot promise contrast. */
+ .jarvis-chat-tinted .tool-identity,
+ .dark .jarvis-chat-tinted .tool-identity {
+ --tool-ink: currentColor;
+ --tool-wash: color-mix(in srgb, currentColor 16%, transparent);
+ }
+
/* The assistant card's vivid tiles: a gradient pair plus a soft light
bloom in the top corner, so the tile reads as an object, not a swatch. */
.jarvis-tile-soul,
diff --git a/jarvis/ui/web/frontend/src/lib/agentChatApi.ts b/jarvis/ui/web/frontend/src/lib/agentChatApi.ts
index 7ebc2af89a..cbd20f2932 100644
--- a/jarvis/ui/web/frontend/src/lib/agentChatApi.ts
+++ b/jarvis/ui/web/frontend/src/lib/agentChatApi.ts
@@ -231,6 +231,7 @@ export class AgentChatApiError extends Error {
constructor(
message: string,
readonly status: number,
+ readonly code: string = "",
) {
super(message);
this.name = "AgentChatApiError";
@@ -240,13 +241,19 @@ export class AgentChatApiError extends Error {
async function json<T>(res: Response, what: string): Promise<T> {
if (!res.ok) {
let detail = what;
+ let code = "";
try {
const body = (await res.json()) as { detail?: unknown };
if (typeof body.detail === "string") detail = body.detail;
+ else if (body.detail && typeof body.detail === "object") {
+ const structured = body.detail as { code?: unknown; message?: unknown };
+ if (typeof structured.code === "string") code = structured.code;
+ if (typeof structured.message === "string") detail = structured.message;
+ }
} catch {
/* no JSON body — keep the generic label */
}
- throw new AgentChatApiError(detail, res.status);
+ throw new AgentChatApiError(detail, res.status, code);
}
return (await res.json()) as T;
}
@@ -347,8 +354,9 @@ export interface LiveModel {
}
/** The provider's live model list (brain catalog route); [] when the route has none. */
-export async function fetchProviderModels(providerId: string): Promise<LiveModel[]> {
+export async function fetchProviderModels(providerId: string, { strict = false }: { strict?: boolean } = {}): Promise<LiveModel[]> {
const res = await fetch(`/api/providers/${encodeURIComponent(providerId)}/models`);
+ if (!res.ok && strict) throw new Error(`Model catalog request failed: HTTP ${res.status}`);
if (!res.ok) return [];
const data = (await res.json()) as { models?: unknown } | unknown[];
const raw = Array.isArray(data) ? data : Array.isArray(data.models) ? data.models : [];
@@ -544,10 +552,12 @@ export function attachmentFileUrl(cwd: string, reference: string): string {
return `/api/agent-chat/attachments/file?${query.toString()}`;
}
-export async function cancelAgentChatTurn(sessionId: string): Promise<void> {
+/** `via` names what asked for the stop, so the backend log can say who ended a turn. */
+export async function cancelAgentChatTurn(sessionId: string, via = ""): Promise<void> {
await json(
await fetch(`/api/agent-chat/sessions/${encodeURIComponent(sessionId)}/cancel`, {
method: "POST",
+ ...(via ? { headers: { "X-Jarvis-Stop-Via": via } } : {}),
}),
"cancel-failed",
);
@@ -609,6 +619,47 @@ export async function skipAgentChatQuestion(sessionId: string, questionId: strin
);
}
+function credentialUrl(sessionId: string, requestId: string): string {
+ return `/api/agent-chat/sessions/${encodeURIComponent(sessionId)}/credentials/${encodeURIComponent(requestId)}`;
+}
+
+export interface CredentialResult {
+ ok: boolean;
+ request_id: string;
+ status: "saved" | "declined";
+}
+
+/** Bound a submission, never the time the person has to enter a credential. */
+async function credentialRequest(url: string, init: RequestInit): Promise<CredentialResult> {
+ const controller = new AbortController();
+ const timeout = setTimeout(() => controller.abort(), 30_000);
+ try {
+ return await json<CredentialResult>(await fetch(url, { ...init, signal: controller.signal }), "credential-failed");
+ } catch (error) {
+ if (controller.signal.aborted) throw new AgentChatApiError("credential-timeout", 408, "validation_timeout");
+ throw error;
+ } finally {
+ clearTimeout(timeout);
+ }
+}
+
+/**
+ * Save the secret an agent asked for. The value goes to the agent's vault and
+ * nowhere else; the response never echoes it.
+ */
+export async function submitAgentChatCredential(sessionId: string, requestId: string, value: string): Promise<CredentialResult> {
+ return credentialRequest(credentialUrl(sessionId, requestId), {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ value }),
+ });
+}
+
+/** Close an agent's credential field without providing the secret. */
+export async function declineAgentChatCredential(sessionId: string, requestId: string): Promise<CredentialResult> {
+ return credentialRequest(`${credentialUrl(sessionId, requestId)}/decline`, { method: "POST" });
+}
+
/** A coding agent's plan card: `build` switches to building and sends the go-ahead. */
export type PlanDecision = "build" | "keep";
diff --git a/jarvis/ui/web/frontend/src/lib/agentResetsApi.test.ts b/jarvis/ui/web/frontend/src/lib/agentResetsApi.test.ts
new file mode 100644
index 0000000000..2c55fc30f5
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/agentResetsApi.test.ts
@@ -0,0 +1,21 @@
+import { afterEach, expect, it, vi } from "vitest";
+import { consumeAccountReset } from "./agentResetsApi";
+
+afterEach(() => vi.unstubAllGlobals());
+const attempt = { idempotency_key: "one-attempt", credit_id: "one-credit", account_key: "a".repeat(64) };
+
+it("never accepts an unknown redemption outcome as success", async () => {
+ const fetcher = vi.fn(async () => Response.json({ outcome: "futureState", resets: {} }));
+ vi.stubGlobal("fetch", fetcher);
+ await expect(consumeAccountReset("seat", attempt)).rejects.toThrow("uncertain");
+ expect(fetcher).toHaveBeenCalledTimes(1);
+});
+
+it("sends one confirmed request and retains a safe failure classification", async () => {
+ const fetcher = vi.fn(async () => Response.json({ detail: { code: "account_changed" } }, { status: 409 }));
+ vi.stubGlobal("fetch", fetcher);
+ await expect(consumeAccountReset("codex:one", attempt)).rejects.toThrow("account_changed");
+ expect(fetcher).toHaveBeenCalledWith("/api/agent-accounts/codex%3Aone/resets/consume", expect.objectContaining({
+ method: "POST", body: JSON.stringify({ ...attempt, confirmed: true }),
+ }));
+});
diff --git a/jarvis/ui/web/frontend/src/lib/agentResetsApi.ts b/jarvis/ui/web/frontend/src/lib/agentResetsApi.ts
new file mode 100644
index 0000000000..c832cde588
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/agentResetsApi.ts
@@ -0,0 +1,61 @@
+import type { AccountUsage } from "./agentAccountsApi";
+
+export interface ResetCredit {
+ id: string;
+ title: string | null;
+ description: string | null;
+ expires_at: string | null;
+ redeemable: boolean;
+}
+
+export interface AccountResets {
+ account_id: string;
+ status: "ok" | "browser" | "unsupported" | "unavailable";
+ available_count: number | null;
+ credits: ResetCredit[] | null;
+ can_redeem: boolean;
+ account_key: string | null;
+ manage_url: string | null;
+ as_of: number;
+ usage: AccountUsage | null;
+}
+
+export interface ResetAttempt {
+ idempotency_key: string;
+ credit_id: string | null;
+ account_key: string;
+}
+
+export interface ResetResult {
+ outcome: "reset" | "alreadyRedeemed" | "nothingToReset" | "noCredit";
+ resets: AccountResets;
+}
+
+export class ResetRequestError extends Error {
+ constructor(public code: string) { super(code); }
+}
+
+async function answer<T>(response: Response): Promise<T> {
+ if (!response.ok) {
+ const body = await response.json().catch(() => null);
+ const code = body?.detail?.code;
+ throw new ResetRequestError(typeof code === "string" ? code : "uncertain");
+ }
+ return response.json() as Promise<T>;
+}
+
+export async function fetchAccountResets(accountId: string): Promise<AccountResets> {
+ return answer(await fetch(`/api/agent-accounts/${encodeURIComponent(accountId)}/resets`, { cache: "no-store" }));
+}
+
+export async function consumeAccountReset(accountId: string, attempt: ResetAttempt): Promise<ResetResult> {
+ const result = await answer<ResetResult>(await fetch(`/api/agent-accounts/${encodeURIComponent(accountId)}/resets/consume`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ ...attempt, confirmed: true }),
+ }));
+ if (!["reset", "alreadyRedeemed", "nothingToReset", "noCredit"].includes(result.outcome) || !result.resets) {
+ throw new ResetRequestError("uncertain");
+ }
+ return result;
+}
diff --git a/jarvis/ui/web/frontend/src/lib/appshotApi.ts b/jarvis/ui/web/frontend/src/lib/appshotApi.ts
index 28b0c5cfbc..d928953f0a 100644
--- a/jarvis/ui/web/frontend/src/lib/appshotApi.ts
+++ b/jarvis/ui/web/frontend/src/lib/appshotApi.ts
@@ -22,6 +22,10 @@ export interface AppshotSettings {
/** Shortcut for an area appshot: drag a rectangle, that part is captured. */
region_hotkey: string;
recording_hotkey?: string;
+ recording_resolution?: "720p" | "1080p" | "1440p" | "2160p" | "native";
+ recording_fps?: 30 | 60 | 120;
+ recording_bitrate_mbps?: number;
+ recording_system_audio?: boolean;
recording_shortcut?: AppshotShortcutStatus;
target: AppshotTarget;
sound: boolean;
@@ -63,6 +67,7 @@ export type AppshotSettingsPatch = Partial<
AppshotSettings,
"enabled" | "hotkey" | "region_hotkey" | "recording_hotkey" | "target" | "sound" | "effect" | "card_seconds" | "library"
| "copy_to_clipboard" | "keep_newest"
+ | "recording_resolution" | "recording_fps" | "recording_bitrate_mbps" | "recording_system_audio"
>
>;
@@ -202,6 +207,8 @@ export interface AppshotLibraryItem {
edited_at: number;
/** An original that also has an edited version. */
has_edit: boolean;
+ /** Finalized recording duration; absent for screenshots and older backends. */
+ duration_s?: number;
}
export function fetchAppshotLibrary(): Promise<{ items: AppshotLibraryItem[]; max_entries: number }> {
@@ -256,7 +263,16 @@ export interface AppshotRecording {
duration_s?: number;
width?: number;
height?: number;
- capability?: { available: boolean; detail: string; permission_required: boolean };
+ fps?: number;
+ bitrate_mbps?: number;
+ system_audio?: boolean;
+ display_name?: string;
+ refresh_hz?: number;
+ dropped_frames?: number;
+ capability?: {
+ available: boolean; detail: string; permission_required: boolean;
+ system_audio?: { available: boolean; detail: string };
+ };
recent?: { id: string; created_at: number }[];
}
@@ -278,3 +294,38 @@ export function requestRecordingPermission(): Promise<unknown> {
body: JSON.stringify({ feature: "appshot" }),
});
}
+
+// -- video editor: play, trim and speed up a finished recording ---------------
+
+/** Speeds the video editor offers; mirrored by `SPEEDS` in `jarvis.appshot.video_edit`. */
+export const RECORDING_SPEEDS = [0.5, 1, 1.5, 2] as const;
+
+/** Write the trim and speed as a new recording; returns its id (the same id when unchanged). */
+export function exportAppshotRecording(
+ id: string,
+ clip: { start_s: number; end_s: number; speed: number },
+): Promise<{ id: string }> {
+ return request(`/api/appshot/recording/${encodeURIComponent(id)}/export`, {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify(clip),
+ });
+}
+
+/** Copy a recording into Downloads on the machine running Jarvis (desktop only). */
+export function saveAppshotRecording(id: string): Promise<{ path: string; filename: string }> {
+ return request(`/api/appshot/recording/${encodeURIComponent(id)}/save`, { method: "POST" });
+}
+
+/** Open the video editor window on a recording; `false` where no window can open. */
+export async function openAppshotRecordingEditor(id: string): Promise<boolean> {
+ try {
+ const body = await request<{ window?: unknown }>(
+ `/api/appshot/recording/${encodeURIComponent(id)}/open-editor`,
+ { method: "POST" },
+ );
+ return body?.window === true;
+ } catch {
+ return false;
+ }
+}
diff --git a/jarvis/ui/web/frontend/src/lib/bubbleTint.test.ts b/jarvis/ui/web/frontend/src/lib/bubbleTint.test.ts
new file mode 100644
index 0000000000..3790b8e1dc
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/bubbleTint.test.ts
@@ -0,0 +1,45 @@
+import { describe, expect, it } from "vitest";
+import { COMPANION_COLORS } from "@/components/society/companion/appearance";
+import { DARK_INK, LIGHT_INK, bubbleTint, contrast } from "./bubbleTint";
+
+describe("bubbleTint", () => {
+ it("keeps every companion colour readable", () => {
+ for (const color of COMPANION_COLORS) {
+ const tint = bubbleTint(color)!;
+ expect(contrast(tint.background, tint.color), color).toBeGreaterThanOrEqual(4.5);
+ }
+ });
+
+ it("meets AA on any colour a person can pick", () => {
+ for (let r = 0; r < 256; r += 17) {
+ for (let g = 0; g < 256; g += 17) {
+ for (let b = 0; b < 256; b += 17) {
+ const hex = `#${[r, g, b].map((c) => c.toString(16).padStart(2, "0")).join("")}`;
+ const tint = bubbleTint(hex)!;
+ expect(contrast(tint.background, tint.color), hex).toBeGreaterThanOrEqual(4.5);
+ }
+ }
+ }
+ });
+
+ it("puts dark ink on a light colour and keeps the colour", () => {
+ expect(bubbleTint("#c5dfd4")).toEqual({ background: "#c5dfd4", color: DARK_INK });
+ });
+
+ it("puts white ink on a dark colour and keeps the colour", () => {
+ expect(bubbleTint("#1d4ed8")).toEqual({ background: "#1d4ed8", color: LIGHT_INK });
+ });
+
+ it("darkens a mid-tone just enough for white ink, keeping its hue", () => {
+ const tint = bubbleTint("#8b5cf6")!;
+ expect(tint.color).toBe(LIGHT_INK);
+ expect(tint.background).not.toBe("#8b5cf6");
+ expect(contrast(tint.background, LIGHT_INK)).toBeGreaterThanOrEqual(4.5);
+ });
+
+ it("returns null for anything that is not a colour", () => {
+ expect(bubbleTint("")).toBeNull();
+ expect(bubbleTint(undefined)).toBeNull();
+ expect(bubbleTint("blue")).toBeNull();
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/lib/bubbleTint.ts b/jarvis/ui/web/frontend/src/lib/bubbleTint.ts
new file mode 100644
index 0000000000..6976028e3f
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/bubbleTint.ts
@@ -0,0 +1,67 @@
+/**
+ * The fill and ink of a message bubble tinted with an agent's colour.
+ *
+ * An agent's colour is any hex a person picked, so the ink is chosen from the
+ * colour itself instead of a fixed white: a light colour (most of the
+ * companion palette) keeps its fill and gets dark ink, a dark one keeps its
+ * fill and gets white ink. A mid-tone that reaches neither threshold is
+ * darkened step by step, keeping its hue, until white ink reads on it. Every
+ * result meets WCAG AA (4.5:1) for body text.
+ */
+
+export interface BubbleTint {
+ background: string;
+ color: string;
+}
+
+/** Near-black ink for light bubbles; softer than #000 on a saturated fill. */
+export const DARK_INK = "#111114";
+export const LIGHT_INK = "#ffffff";
+
+/** WCAG AA for body text. */
+const AA = 4.5;
+/** Dark ink only where it reads clearly better than white could after darkening. */
+const DARK_INK_MIN = 7;
+
+function parseHex(hex: string): [number, number, number] | null {
+ const match = /^#?([0-9a-f]{6})$/i.exec(hex.trim());
+ if (!match) return null;
+ const value = parseInt(match[1], 16);
+ return [(value >> 16) & 255, (value >> 8) & 255, value & 255];
+}
+
+function toHex([r, g, b]: [number, number, number]): string {
+ return `#${[r, g, b].map((c) => Math.round(c).toString(16).padStart(2, "0")).join("")}`;
+}
+
+function channel(c: number): number {
+ const s = c / 255;
+ return s <= 0.04045 ? s / 12.92 : ((s + 0.055) / 1.055) ** 2.4;
+}
+
+export function luminance(rgb: [number, number, number]): number {
+ return 0.2126 * channel(rgb[0]) + 0.7152 * channel(rgb[1]) + 0.0722 * channel(rgb[2]);
+}
+
+export function contrast(a: string, b: string): number {
+ const ra = parseHex(a);
+ const rb = parseHex(b);
+ if (!ra || !rb) return 1;
+ const la = luminance(ra);
+ const lb = luminance(rb);
+ return (Math.max(la, lb) + 0.05) / (Math.min(la, lb) + 0.05);
+}
+
+/** Null for anything that is not a #rrggbb colour; the caller keeps its default look. */
+export function bubbleTint(hex: string | null | undefined): BubbleTint | null {
+ const rgb = hex ? parseHex(hex) : null;
+ if (!rgb) return null;
+ const base = toHex(rgb);
+ if (contrast(base, LIGHT_INK) >= AA) return { background: base, color: LIGHT_INK };
+ if (contrast(base, DARK_INK) >= DARK_INK_MIN) return { background: base, color: DARK_INK };
+ for (let factor = 0.95; factor > 0; factor -= 0.05) {
+ const darker = toHex([rgb[0] * factor, rgb[1] * factor, rgb[2] * factor]);
+ if (contrast(darker, LIGHT_INK) >= AA) return { background: darker, color: LIGHT_INK };
+ }
+ return { background: "#000000", color: LIGHT_INK };
+}
diff --git a/jarvis/ui/web/frontend/src/lib/modelAccess.test.tsx b/jarvis/ui/web/frontend/src/lib/modelAccess.test.tsx
new file mode 100644
index 0000000000..3767a83e54
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/modelAccess.test.tsx
@@ -0,0 +1,46 @@
+import { act, cleanup, renderHook } from "@testing-library/react";
+import { afterEach, beforeEach, expect, test, vi } from "vitest";
+import { MODEL_ACCESS_KEY, preferredModelAccess, useModelAccess } from "./modelAccess";
+
+beforeEach(() => localStorage.clear());
+afterEach(() => { cleanup(); vi.restoreAllMocks(); localStorage.clear(); });
+
+test("starts on a subscription, honors explicit choices and only offers available access", () => {
+ const api = { kind: "api" as const };
+ const subscription = { kind: "subscription" as const };
+ expect(preferredModelAccess([api, subscription])).toBe(subscription);
+ expect(preferredModelAccess([api, subscription], "api")).toBe(api);
+ expect(preferredModelAccess([api], "subscription")).toBe(api);
+ expect(preferredModelAccess([api, { ...subscription, disabled: true }])).toBe(api);
+ expect(preferredModelAccess([])).toBeUndefined();
+});
+
+test("shares explicit choices across pickers and remounts without changing other providers", () => {
+ const first = renderHook(useModelAccess);
+ const second = renderHook(useModelAccess);
+ act(() => first.result.current[1]("openai", "api"));
+ act(() => first.result.current[1]("antigravity", "subscription"));
+ expect(second.result.current[0]).toEqual({ openai: "api", gemini: "subscription" });
+ first.unmount(); second.unmount();
+ expect(renderHook(useModelAccess).result.current[0]).toEqual({ openai: "api", gemini: "subscription" });
+});
+
+test("reloads cross-window changes and clearing preferences", () => {
+ const state = renderHook(useModelAccess);
+ act(() => {
+ localStorage.setItem(MODEL_ACCESS_KEY, JSON.stringify({ openai: "api", invalid: "paid" }));
+ window.dispatchEvent(new StorageEvent("storage", { key: MODEL_ACCESS_KEY }));
+ });
+ expect(state.result.current[0]).toEqual({ openai: "api" });
+ act(() => { localStorage.clear(); window.dispatchEvent(new StorageEvent("storage", { key: null })); });
+ expect(state.result.current[0]).toEqual({});
+});
+
+test("ignores corrupt storage and retains an explicit choice when writing is blocked", () => {
+ localStorage.setItem(MODEL_ACCESS_KEY, "broken");
+ const state = renderHook(useModelAccess);
+ expect(state.result.current[0]).toEqual({});
+ vi.spyOn(Storage.prototype, "setItem").mockImplementation(() => { throw new Error("blocked"); });
+ act(() => state.result.current[1]("openai", "api"));
+ expect(state.result.current[0]).toEqual({ openai: "api" });
+});
diff --git a/jarvis/ui/web/frontend/src/lib/modelAccess.ts b/jarvis/ui/web/frontend/src/lib/modelAccess.ts
new file mode 100644
index 0000000000..64de3ebd02
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/modelAccess.ts
@@ -0,0 +1,75 @@
+import { useCallback, useSyncExternalStore } from "react";
+
+export type ModelAccess = "subscription" | "api" | "local";
+type Preferences = Partial<Record<string, ModelAccess>>;
+
+export const MODEL_ACCESS_KEY = "jarvis.modelAccess.v1";
+const listeners = new Set<() => void>();
+let cached: { raw: string | null; value: Preferences } = { raw: null, value: {} };
+const EMPTY: Preferences = {};
+
+/** Presentation families shared by every model picker. Routing keeps the catalog id. */
+export function modelAccessFamily(family: string): string {
+ return family === "antigravity" ? "gemini" : family;
+}
+
+function read(): Preferences {
+ let raw: string | null;
+ try {
+ raw = window.localStorage.getItem(MODEL_ACCESS_KEY);
+ } catch {
+ // Storage unavailable: explicit choices still last for this window.
+ return cached.value;
+ }
+ if (raw === cached.raw) return cached.value;
+ let value: Preferences = {};
+ try {
+ const parsed: unknown = JSON.parse(raw ?? "{}");
+ if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
+ value = Object.fromEntries(Object.entries(parsed).filter(([, kind]) =>
+ kind === "subscription" || kind === "api" || kind === "local"));
+ }
+ } catch {
+ // Corrupt preferences restore subscription-first browsing.
+ }
+ cached = { raw, value };
+ return value;
+}
+
+function subscribe(listener: () => void) {
+ listeners.add(listener);
+ const storage = (event: StorageEvent) => {
+ if (event.key === MODEL_ACCESS_KEY || event.key === null) listener();
+ };
+ window.addEventListener("storage", storage);
+ return () => { listeners.delete(listener); window.removeEventListener("storage", storage); };
+}
+
+/** A browsing preference, never permission to switch an existing agent or bill a key. */
+export function useModelAccess(): [Preferences, (family: string, kind: ModelAccess) => void] {
+ const preferences = useSyncExternalStore(subscribe, read, () => EMPTY);
+ const remember = useCallback((family: string, kind: ModelAccess) => {
+ const value = { ...read(), [modelAccessFamily(family)]: kind };
+ let raw = cached.raw;
+ try {
+ const serialized = JSON.stringify(value);
+ window.localStorage.setItem(MODEL_ACCESS_KEY, serialized);
+ raw = serialized;
+ } catch {
+ // The UI remains usable when the browser refuses persistence.
+ }
+ cached = { raw, value };
+ listeners.forEach((listener) => listener());
+ }, []);
+ return [preferences, remember];
+}
+
+/** Use only offered access. An absent saved choice is retained for reconnection. */
+export function preferredModelAccess<T extends { kind: ModelAccess; disabled?: boolean }>(
+ options: T[], preferred?: ModelAccess,
+): T | undefined {
+ const available = options.filter((option) => !option.disabled);
+ return available.find((option) => option.kind === preferred)
+ ?? available.find((option) => option.kind === "subscription")
+ ?? available[0] ?? options[0];
+}
diff --git a/jarvis/ui/web/frontend/src/lib/pcm-worklet.startup.test.ts b/jarvis/ui/web/frontend/src/lib/pcm-worklet.startup.test.ts
new file mode 100644
index 0000000000..ac63daf46b
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/pcm-worklet.startup.test.ts
@@ -0,0 +1,35 @@
+import { afterEach, expect, it, vi } from "vitest";
+
+afterEach(() => vi.unstubAllGlobals());
+
+it("acknowledges the actual render quantum and catchup without sending pre-activation audio", async () => {
+ type Processor = {
+ port: { onmessage: (e: { data: unknown }) => void; postMessage: ReturnType<typeof vi.fn> };
+ process: (inputs: Float32Array[][], outputs: Float32Array[][]) => void;
+ };
+ const processors = new Map<string, new () => Processor>();
+ vi.stubGlobal("sampleRate", 48000);
+ vi.stubGlobal("currentTime", 0);
+ vi.stubGlobal("registerProcessor", (name: string, ctor: unknown) => processors.set(name, ctor as never));
+ vi.stubGlobal("AudioWorkletProcessor", class { port = { onmessage: null, postMessage: vi.fn() }; });
+ await import("./pcm-worklet");
+ const capture = new (processors.get("pcm-startup")!)();
+ const block = new Float32Array(128).fill(0.02), output = new Float32Array(128);
+ for (let i = 0; i < 750; i++) {
+ capture.process([[block]], [[output]]);
+ expect(output.every(value => value === 0)).toBe(true);
+ }
+ expect(capture.port.postMessage).not.toHaveBeenCalled();
+ capture.port.onmessage({ data: { type: "start" } });
+ expect(capture.port.postMessage).not.toHaveBeenCalled();
+ capture.process([[block]], [[output]]);
+ expect(capture.port.postMessage).toHaveBeenCalledWith({ type: "input_started", pending_ms: expect.any(Number) });
+ expect(output.some(value => value > 0)).toBe(true);
+ expect(capture.port.postMessage).not.toHaveBeenCalledWith({ type: "input_caught_up" });
+ for (let i = 0; i < 2_000; i++) capture.process([[block]], [[output]]);
+ expect(capture.port.postMessage).toHaveBeenCalledWith({ type: "input_caught_up" });
+ expect(capture.port.postMessage).toHaveBeenCalledTimes(2);
+ capture.port.onmessage({ data: { type: "suspend" } });
+ capture.process([[block]], [[output]]);
+ expect(output.every(value => value === 0)).toBe(true);
+});
diff --git a/jarvis/ui/web/frontend/src/lib/pcm-worklet.ts b/jarvis/ui/web/frontend/src/lib/pcm-worklet.ts
index b36f23f87b..e2bbe1d100 100644
--- a/jarvis/ui/web/frontend/src/lib/pcm-worklet.ts
+++ b/jarvis/ui/web/frontend/src/lib/pcm-worklet.ts
@@ -54,13 +54,16 @@ class PcmCapture extends AudioWorkletProcessor {
class StartupCapture extends AudioWorkletProcessor {
private queue = new StartupAudioQueue(sampleRate);
private failed = false;
+ private started = false;
+ private reportedStart = false;
+ private reportedCatchup = false;
constructor() {
super();
this.port.onmessage = (event: MessageEvent) => {
try {
- if (event.data.type === "start") this.queue.start();
- else if (event.data.type === "suspend") this.queue.suspend();
+ if (event.data.type === "start") { this.queue.start(); this.started = true; }
+ else if (event.data.type === "suspend") { this.queue.suspend(); this.started = false; }
else if (event.data.type === "resume") this.queue.resume();
else if (event.data.type === "prefix") this.queue.prepend(event.data.samples);
} catch (error) { this.fail(error); }
@@ -77,7 +80,17 @@ class StartupCapture extends AudioWorkletProcessor {
const output = outputs[0]?.[0];
if (!output) return true;
if (this.failed) { output.fill(0); return true; }
- try { this.queue.process(inputs[0]?.[0] ?? new Float32Array(output.length), output); }
+ try {
+ this.queue.process(inputs[0]?.[0] ?? new Float32Array(output.length), output);
+ if (this.started && !this.reportedStart) {
+ this.reportedStart = true;
+ this.port.postMessage({ type: "input_started", pending_ms: this.queue.pendingMs });
+ }
+ if (this.started && !this.reportedCatchup && this.queue.pendingMs <= output.length / sampleRate * 1000) {
+ this.reportedCatchup = true;
+ this.port.postMessage({ type: "input_caught_up" });
+ }
+ }
catch (error) { this.fail(error); }
return true;
}
diff --git a/jarvis/ui/web/frontend/src/lib/realtimeAudio.test.ts b/jarvis/ui/web/frontend/src/lib/realtimeAudio.test.ts
index 8c61f1405f..4b4b56fee1 100644
--- a/jarvis/ui/web/frontend/src/lib/realtimeAudio.test.ts
+++ b/jarvis/ui/web/frontend/src/lib/realtimeAudio.test.ts
@@ -12,6 +12,8 @@ import {
} from "./realtimeAudio";
import { TimedPcmQueue } from "./playbackTimeline";
import { readTimedSpeechPlayback } from "./speechPlayback";
+import { registerRealtimeAudioPreparation } from "./realtimeAudioPreparation";
+import { resetConnectBudgetForTests } from "./connectBudget";
class FakePort {
onmessage: ((event: MessageEvent) => void) | null = null;
@@ -41,6 +43,7 @@ class FakeAudioContext {
destination = {} as AudioDestinationNode;
audioWorklet = { addModule: vi.fn(async () => undefined) };
resume = vi.fn(async () => undefined);
+ suspend = vi.fn(async () => { this.state = "suspended"; });
close = vi.fn(async () => undefined);
createMediaStreamSource = vi.fn(() => new FakeAudioNode());
createGain = vi.fn(() => Object.assign(new FakeAudioNode(), { gain: {
@@ -370,8 +373,13 @@ describe("realtime audio client", () => {
expect(FakePeerConnection.instances[0].addTrack.mock.calls[0][0]).toBe("buffered-track");
expect(FakeAudioNode.instances.some(node => node.name === "pcm-startup")).toBe(false);
expect(FakeWebSocket.instances).toHaveLength(0);
+ const beforeMic = (client as unknown as { startupMarks: Record<string, number> }).startupMarks;
+ expect(beforeMic).toHaveProperty("offer_ready");
+ expect(beforeMic).not.toHaveProperty("microphone_ready");
release({ getTracks: () => [track], getAudioTracks: () => [track] } as unknown as MediaStream);
await vi.waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1));
+ expect(beforeMic).toHaveProperty("microphone_ready");
+ expect(beforeMic).toHaveProperty("worklet_ready");
const socket = FakeWebSocket.instances[0];
socket.open();
socket.receive({ type: "audio_ready", requires_webrtc_answer: true, webrtc_answer_sdp: "answer" });
@@ -575,6 +583,9 @@ describe("realtime audio client", () => {
});
beforeEach(() => {
+ // Each case owns its clock. Fake-timer advances must not strand the next
+ // real-clock case behind a token refill timestamp from the future.
+ resetConnectBudgetForTests();
FakeAudioContext.voices = [];
vi.stubGlobal("window", {
location: { protocol: "https:", host: "app.example", hostname: "app.example" },
@@ -590,7 +601,205 @@ describe("realtime audio client", () => {
wsFakes.mintWsTicket.mockClear();
});
- afterEach(() => vi.unstubAllGlobals());
+ afterEach(() => { resetConnectBudgetForTests(); vi.unstubAllGlobals(); });
+
+ it("closes prepared local audio after microphone denial without opening a billed socket", async () => {
+ installVoiceBrowserFakes();
+ vi.stubGlobal("Audio", class { play = async () => undefined; pause = () => undefined; });
+ const events = new EventTarget();
+ Object.assign(window, {
+ addEventListener: events.addEventListener.bind(events),
+ removeEventListener: events.removeEventListener.bind(events),
+ });
+ vi.useFakeTimers();
+ Object.assign(window, { setTimeout: globalThis.setTimeout, clearTimeout: globalThis.clearTimeout });
+ const dispose = registerRealtimeAudioPreparation();
+ try {
+ await vi.advanceTimersByTimeAsync(501);
+ const context = FakeAudioContext.instances.at(-1)!;
+ expect(context.suspend).toHaveBeenCalledOnce();
+ vi.mocked(navigator.mediaDevices.getUserMedia).mockRejectedValueOnce(new DOMException("Denied", "NotAllowedError"));
+ const client = new RealtimeAudioClient({}, { browserAudio: true, requiresWebRtcOffer: true });
+ // jsdom's DOMException belongs to a different Error realm; the client's
+ // existing error normalization keeps its denial detail in the message.
+ await expect(client.connect()).rejects.toThrow(/Denied/);
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ expect(context.close).toHaveBeenCalledOnce();
+ expect(FakePeerConnection.instances[0].close).toHaveBeenCalledOnce();
+ } finally {
+ dispose();
+ vi.useRealTimers();
+ }
+ });
+
+ it.each(["resume", "worklet"])("bounds a hung local %s after permission and ignores late completion", async stalled => {
+ const { track } = installVoiceBrowserFakes();
+ vi.useFakeTimers();
+ Object.assign(window, { setTimeout: globalThis.setTimeout, clearTimeout: globalThis.clearTimeout });
+ let finish!: () => void;
+ const pending = new Promise<undefined>(resolve => { finish = () => resolve(undefined); });
+ vi.stubGlobal("AudioContext", class extends FakeAudioContext {
+ constructor() {
+ super();
+ if (stalled === "resume") this.resume = vi.fn(() => pending);
+ else this.audioWorklet.addModule = vi.fn(() => pending);
+ }
+ });
+ const client = new RealtimeAudioClient({}, { requiresWebRtcOffer: true });
+ const outcome = client.connect().catch(error => error);
+ try {
+ await vi.advanceTimersByTimeAsync(9_999);
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ expect(track.stop).not.toHaveBeenCalled();
+ await vi.advanceTimersByTimeAsync(1);
+ expect(await outcome).toMatchObject({ message: "Local voice audio device did not become ready within 10 seconds" });
+ expect(track.stop).toHaveBeenCalledOnce();
+ expect(FakeAudioContext.instances.at(-1)!.close).toHaveBeenCalledOnce();
+ expect(FakePeerConnection.instances[0].close).toHaveBeenCalledOnce();
+ finish();
+ await vi.advanceTimersByTimeAsync(100);
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ expect((client as unknown as { startupMarks: Record<string, number> }).startupMarks).not.toHaveProperty("worklet_ready");
+ } finally {
+ await client.disconnect();
+ vi.useRealTimers();
+ }
+ });
+
+ it.each(["missing", "prompt", "unsupported"])("does not time out microphone permission (%s) and cancels a late grant", async permission => {
+ const { track } = installVoiceBrowserFakes();
+ if (permission !== "missing") Object.assign(navigator, { permissions: {
+ query: permission === "prompt"
+ ? vi.fn(async () => ({ state: "prompt" }))
+ : vi.fn(async () => { throw new TypeError("Unsupported microphone descriptor"); }),
+ } });
+ vi.useFakeTimers();
+ Object.assign(window, { setTimeout: globalThis.setTimeout, clearTimeout: globalThis.clearTimeout });
+ let grant!: (stream: MediaStream) => void;
+ vi.mocked(navigator.mediaDevices.getUserMedia).mockImplementation(() => new Promise(resolve => { grant = resolve; }));
+ const client = new RealtimeAudioClient({}, { requiresWebRtcOffer: true });
+ let settled = false;
+ const outcome = client.connect().catch(error => { settled = true; return error; });
+ try {
+ await vi.advanceTimersByTimeAsync(30_000);
+ expect(settled).toBe(false); // Choosing permission consumes no device deadline.
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ await client.disconnect();
+ expect(await outcome).toMatchObject({ message: "Voice start cancelled" });
+ expect(FakePeerConnection.instances[0].close).toHaveBeenCalledOnce();
+ grant({ getTracks: () => [track], getAudioTracks: () => [track] } as unknown as MediaStream);
+ await vi.advanceTimersByTimeAsync(100);
+ expect(track.stop).toHaveBeenCalledOnce();
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ expect(FakeAudioNode.instances.some(node => node.name === "pcm-capture")).toBe(false);
+ } finally { vi.useRealTimers(); }
+ });
+
+ it("bounds microphone hardware opening when permission is already granted", async () => {
+ const { track } = installVoiceBrowserFakes();
+ Object.assign(navigator, { permissions: { query: vi.fn(async () => ({ state: "granted" })) } });
+ vi.useFakeTimers();
+ Object.assign(window, { setTimeout: globalThis.setTimeout, clearTimeout: globalThis.clearTimeout });
+ let grant!: (stream: MediaStream) => void;
+ vi.mocked(navigator.mediaDevices.getUserMedia).mockImplementation(() => new Promise(resolve => { grant = resolve; }));
+ const client = new RealtimeAudioClient({}, { requiresWebRtcOffer: true });
+ const outcome = client.connect().catch(error => error);
+ try {
+ await vi.advanceTimersByTimeAsync(10_000);
+ expect(await outcome).toMatchObject({ message: "Local voice audio device did not become ready within 10 seconds" });
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ expect(FakePeerConnection.instances[0].close).toHaveBeenCalledOnce();
+ grant({ getTracks: () => [track], getAudioTracks: () => [track] } as unknown as MediaStream);
+ await vi.advanceTimersByTimeAsync(0);
+ expect(track.stop).toHaveBeenCalledOnce();
+ } finally { await client.disconnect(); vi.useRealTimers(); }
+ });
+
+ it.each(["timeout", "stop"])("settles %s even when native audio close never settles", async action => {
+ const { track } = installVoiceBrowserFakes();
+ vi.useFakeTimers();
+ Object.assign(window, { setTimeout: globalThis.setTimeout, clearTimeout: globalThis.clearTimeout });
+ let finishResume!: () => void, finishClose!: () => void;
+ const resume = new Promise<undefined>(resolve => { finishResume = () => resolve(undefined); });
+ const close = new Promise<undefined>(resolve => { finishClose = () => resolve(undefined); });
+ vi.stubGlobal("AudioContext", class extends FakeAudioContext {
+ resume = vi.fn(() => resume);
+ close = vi.fn(() => close);
+ });
+ const client = new RealtimeAudioClient({}, { requiresWebRtcOffer: true });
+ let failure: Error | undefined;
+ const outcome = client.connect().catch((error: Error) => { failure = error; });
+ try {
+ await vi.advanceTimersByTimeAsync(0);
+ if (action === "timeout") await vi.advanceTimersByTimeAsync(10_000);
+ else {
+ let stopped = false;
+ void client.disconnect().then(() => { stopped = true; });
+ await vi.advanceTimersByTimeAsync(0);
+ expect(stopped).toBe(true);
+ }
+ expect(failure?.message).toMatch(action === "timeout" ? /within 10 seconds/ : /cancelled/);
+ await outcome;
+ expect(track.stop).toHaveBeenCalledOnce();
+ expect(FakeAudioContext.instances.at(-1)!.close).toHaveBeenCalledOnce();
+ expect(FakePeerConnection.instances[0].close).toHaveBeenCalledOnce();
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ // The same client can establish a fresh call while an old context is
+ // retiring. Late old worklet/close promises must not erase its socket.
+ vi.stubGlobal("AudioContext", FakeAudioContext);
+ const next = client.connect();
+ await vi.advanceTimersByTimeAsync(200);
+ const socket = FakeWebSocket.instances[0];
+ expect(socket).toBeDefined();
+ socket.open();
+ socket.receive({ type: "audio_ready", webrtc_answer_sdp: "new-answer" });
+ await next;
+ finishResume(); finishClose();
+ await vi.advanceTimersByTimeAsync(0);
+ expect(socket.close).not.toHaveBeenCalled();
+ await client.disconnect();
+ } finally {
+ finishResume(); finishClose();
+ await client.disconnect();
+ vi.useRealTimers();
+ }
+ });
+
+ it("separates input release request, rendered input and audible provider PCM timing", async () => {
+ installVoiceBrowserFakes();
+ vi.stubGlobal("Audio", class { play = async () => undefined; pause = () => undefined; });
+ const client = new RealtimeAudioClient({}, { browserAudio: true, requiresWebRtcOffer: true, webRtcStartEventRequired: false });
+ const connecting = client.connect();
+ await vi.waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1));
+ const socket = FakeWebSocket.instances[0];
+ socket.open();
+ socket.receive({ type: "audio_ready", webrtc_answer_sdp: "answer", output_transport: "timed_pcm" });
+ await connecting;
+ const timings = () => Object.assign({}, ...socket.sent.filter(item => typeof item === "string")
+ .map(item => JSON.parse(String(item))).filter(item => item.type === "startup_timing").map(item => item.marks_ms));
+ expect(timings()).toHaveProperty("input_released");
+ expect(timings()).not.toHaveProperty("input_started");
+ const input = FakeAudioNode.instances.find(node => node.name === "pcm-startup")!;
+ input.port.onmessage!({ data: { type: "input_started" } } as MessageEvent);
+ expect(timings()).toHaveProperty("input_started");
+ expect(timings()).not.toHaveProperty("input_caught_up");
+ input.port.onmessage!({ data: { type: "input_caught_up" } } as MessageEvent);
+ socket.receiveBinary(new Int16Array([0, 0]).buffer);
+ expect(timings()).toHaveProperty("first_output_packet");
+ expect(timings()).not.toHaveProperty("first_output_audio");
+ const output = FakeAudioNode.instances.find(node => node.name === "pcm-playback")!;
+ output.port.onmessage!({ data: { type: "level", rms: 0.01 } } as MessageEvent);
+ expect(timings()).toHaveProperty("first_output_audio");
+ const batches = socket.sent.filter(item => typeof item === "string").map(item => JSON.parse(String(item)))
+ .filter(item => item.type === "startup_timing").map(item => item.marks_ms);
+ expect(batches.every(marks => Object.keys(marks).length <= 8)).toBe(true);
+ // The preceding frontend emitted maps larger than old backend allowlists,
+ // which dropped all telemetry after enough newly introduced phases.
+ expect(Object.keys(timings()).length).toBeGreaterThan(11);
+ expect(Object.assign({}, ...batches.filter(marks => Object.keys(marks).length <= 11))).toHaveProperty("input_released");
+ socket.receive({ type: "audio_closed" });
+ await client.disconnect();
+ });
it("drops muted PCM, rejects stale state and preserves mute through readiness", async () => {
const { track } = installVoiceBrowserFakes();
@@ -946,6 +1155,86 @@ describe("realtime audio client", () => {
await client.disconnect();
});
+ it("dispatches local startup before a 350 ms native peer stall without bypassing microphone permission", async () => {
+ const { track } = installVoiceBrowserFakes();
+ vi.stubGlobal("Audio", class { play = async () => undefined; pause = () => undefined; });
+ let elapsed = 0, microphoneRequestedAt = -1;
+ const clock = vi.spyOn(performance, "now").mockImplementation(() => elapsed);
+ let releaseCapture!: (stream: MediaStream) => void;
+ vi.mocked(navigator.mediaDevices.getUserMedia).mockImplementation(() => {
+ microphoneRequestedAt = elapsed;
+ return new Promise<MediaStream>(resolve => { releaseCapture = resolve; });
+ });
+ vi.stubGlobal("RTCPeerConnection", class extends FakePeerConnection {
+ constructor() {
+ expect(FakeAudioContext.instances.at(-1)!.resume).toHaveBeenCalledOnce();
+ expect(wsFakes.mintWsTicket).toHaveBeenCalledOnce();
+ elapsed += 350;
+ super();
+ }
+ });
+ const client = new RealtimeAudioClient({}, { browserAudio: true, requiresWebRtcOffer: true });
+ const connecting = client.connect();
+ try {
+ expect(microphoneRequestedAt).toBe(0);
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ releaseCapture({ getTracks: () => [track], getAudioTracks: () => [track] } as unknown as MediaStream);
+ await vi.waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1));
+ const socket = FakeWebSocket.instances[0];
+ socket.open();
+ const marks = Object.assign({}, ...socket.sent.map(frame => JSON.parse(String(frame)))
+ .filter(frame => frame.type === "startup_timing").map(frame => frame.marks_ms));
+ expect(marks.local_requests_dispatched).toBe(0);
+ expect(marks.peer_created).toBe(350);
+ socket.receive({ type: "audio_ready", webrtc_answer_sdp: "answer" });
+ await connecting;
+ socket.close();
+ } finally {
+ await client.disconnect();
+ clock.mockRestore();
+ }
+ });
+
+ it("releases a failed start's late microphone grant while a replacement start owns its own capture", async () => {
+ const { track } = installVoiceBrowserFakes();
+ vi.stubGlobal("Audio", class { play = async () => undefined; pause = () => undefined; });
+ const replacementTrack = { stop: vi.fn() };
+ const grants: Array<(stream: MediaStream) => void> = [];
+ vi.mocked(navigator.mediaDevices.getUserMedia).mockImplementation(
+ () => new Promise<MediaStream>(resolve => { grants.push(resolve); }),
+ );
+ let failDestination = true;
+ vi.stubGlobal("AudioContext", class extends FakeAudioContext {
+ constructor() {
+ super();
+ if (failDestination) {
+ failDestination = false;
+ this.createMediaStreamDestination.mockImplementation(() => { throw new Error("Audio destination unavailable"); });
+ }
+ }
+ });
+ const client = new RealtimeAudioClient({}, { browserAudio: true, requiresWebRtcOffer: true });
+ await expect(client.connect()).rejects.toThrow("Audio destination unavailable");
+ expect(FakeAudioContext.instances.at(-1)!.close).toHaveBeenCalledOnce();
+ const replacement = client.connect();
+ try {
+ expect(grants).toHaveLength(2);
+ expect(FakeWebSocket.instances).toHaveLength(0);
+ grants[0]({ getTracks: () => [track], getAudioTracks: () => [track] } as unknown as MediaStream);
+ await vi.waitFor(() => expect(track.stop).toHaveBeenCalledOnce());
+ expect(replacementTrack.stop).not.toHaveBeenCalled();
+ grants[1]({ getTracks: () => [replacementTrack], getAudioTracks: () => [replacementTrack] } as unknown as MediaStream);
+ await vi.waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1));
+ const socket = FakeWebSocket.instances[0];
+ socket.open();
+ socket.receive({ type: "audio_ready", webrtc_answer_sdp: "answer" });
+ await replacement;
+ expect(replacementTrack.stop).not.toHaveBeenCalled();
+ socket.close();
+ } finally { await client.disconnect(); }
+ expect(replacementTrack.stop).toHaveBeenCalledOnce();
+ });
+
it("fails closed before opening a socket when subscription WebRTC is missing", async () => {
const { track } = installVoiceBrowserFakes();
vi.stubGlobal("RTCPeerConnection", undefined);
diff --git a/jarvis/ui/web/frontend/src/lib/realtimeAudio.ts b/jarvis/ui/web/frontend/src/lib/realtimeAudio.ts
index 6e0eab084a..9ed35c55e3 100644
--- a/jarvis/ui/web/frontend/src/lib/realtimeAudio.ts
+++ b/jarvis/ui/web/frontend/src/lib/realtimeAudio.ts
@@ -9,7 +9,7 @@ import { mintWsTicket } from "./ws";
import { beginSpeechPlayback, clearSpeechPlayback, setTimedSpeechSession, spokenWordEnd, updateSpeechPlayback } from "./speechPlayback";
import { audibleContextTime, DevicePlaybackTimeline, type AudioInterval, type RenderedAudioInterval } from "./playbackTimeline";
import { TimedAudioFrame, TimedTextFrame, TimedSpeechTracker } from "./timedSpeech";
-import pcmWorkletUrl from "./pcm-worklet.ts?worker&url";
+import { acquireRealtimeAudio, releaseRealtimeAudio, type PreparedRealtimeAudio } from "./realtimeAudioPreparation";
export function buildAudioSocketUrl(ticket?: string | null): string {
const proto = window.location.protocol === "https:" ? "wss" : "ws";
@@ -62,6 +62,49 @@ const OUTPUT_TAP_TTL_MS = 600;
/** Historical fixed budget for one realtime start attempt. */
const DEFAULT_START_BUDGET_MS = 20_000;
+// This starts only AFTER microphone permission is granted. A suspended audio
+// device or stuck worklet must not leave the call connecting indefinitely.
+const LOCAL_AUDIO_READY_TIMEOUT_MS = 10_000;
+
+function awaitLocalStartup<T>(operation: Promise<T>, signal: AbortSignal, timeoutMs?: number): Promise<T> {
+ return new Promise((resolve, reject) => {
+ let settled = false;
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const finish = (error?: unknown, value?: T) => {
+ if (settled) return;
+ settled = true;
+ if (timer !== undefined) clearTimeout(timer);
+ signal.removeEventListener("abort", cancel);
+ if (error !== undefined) reject(error); else resolve(value as T);
+ };
+ const cancel = () => finish(new Error("Voice start cancelled"));
+ signal.addEventListener("abort", cancel, { once: true });
+ if (timeoutMs !== undefined) {
+ timer = setTimeout(() => finish(new Error("Local voice audio device did not become ready within 10 seconds")), timeoutMs);
+ }
+ void operation.then(value => finish(undefined, value), error => finish(error));
+ if (signal.aborted) cancel();
+ });
+}
+
+function awaitMicrophone(operation: Promise<MediaStream>, signal: AbortSignal): Promise<MediaStream> {
+ const opened = awaitLocalStartup(operation, signal);
+ if (typeof navigator.permissions?.query !== "function") return opened;
+ try {
+ // Read-only and off the critical path. When permission is already granted,
+ // getUserMedia is waiting for hardware, not for a person to answer a prompt.
+ const permission = navigator.permissions.query({ name: "microphone" as PermissionName }).then(
+ status => status.state === "granted"
+ ? awaitLocalStartup(opened, signal, LOCAL_AUDIO_READY_TIMEOUT_MS)
+ : opened,
+ () => opened, // Engines without microphone-query support retain normal permission behavior.
+ );
+ return Promise.race([opened, permission]);
+ } catch {
+ // Older permission implementations may reject this descriptor synchronously.
+ return opened;
+ }
+}
/**
* Consecutive quiet analyser frames before the WebRTC voice tap reports
@@ -106,7 +149,10 @@ export class RealtimeWebRtcTransport {
private finishStartup: ((error?: Error) => void) | null = null;
private outputMuted = true;
private outputVolume = 1;
- constructor(private onRemoteStream?: (stream: MediaStream) => void) {}
+ constructor(
+ private onRemoteStream?: (stream: MediaStream) => void,
+ private onStartupPhase?: (phase: string) => void,
+ ) {}
async createOffer(stream?: MediaStream, startEventRequired = true, startBudgetMs = 25_000): Promise<string | null> {
this.close();
@@ -114,6 +160,7 @@ export class RealtimeWebRtcTransport {
const peer = new RTCPeerConnection();
this.peer = peer;
+ this.onStartupPhase?.("peer_created");
if (stream) {
for (const track of stream.getAudioTracks()) peer.addTrack(track, stream);
this.player = new Audio();
@@ -169,7 +216,9 @@ export class RealtimeWebRtcTransport {
void this.started.catch(() => undefined);
}
// Only the backend sideband executes tools or continues Responses work.
- const offer = await peer.createOffer();
+ const offerPending = peer.createOffer();
+ this.onStartupPhase?.("offer_requested");
+ const offer = await offerPending;
await peer.setLocalDescription(offer);
await waitForIceGathering(peer);
if (this.peer !== peer) return null;
@@ -455,6 +504,7 @@ export class StreamingPcm16Resampler {
export class RealtimeAudioClient {
private ws: WebSocket | null = null;
private ctx: AudioContext | null = null;
+ private preparedAudio: PreparedRealtimeAudio | null = null;
private captureNode: AudioWorkletNode | null = null;
private captureSink: GainNode | null = null;
private playbackNode: AudioWorkletNode | null = null;
@@ -466,6 +516,7 @@ export class RealtimeAudioClient {
private receivedPrefix = false;
private playbackResampler: StreamingPcm16Resampler | null = null;
private connecting: Promise<void> | null = null;
+ private localStartupAbort: AbortController | null = null;
private ready = false;
private intentionalClose = false;
private inputMeter = new LevelMeter();
@@ -568,7 +619,10 @@ export class RealtimeAudioClient {
private cb: RealtimeCallbacks = {},
private options: RealtimeAudioOptions = {},
) {
- this.webRtcTransport = new RealtimeWebRtcTransport(stream => this.observeRemoteAudio(stream));
+ this.webRtcTransport = new RealtimeWebRtcTransport(
+ stream => this.observeRemoteAudio(stream),
+ phase => this.markStartup(phase),
+ );
}
private remoteMeter: AudioWorkletNode | null = null;
@@ -584,7 +638,13 @@ export class RealtimeAudioClient {
this.startupMarks[phase] = Math.round(performance.now() - this.startupAt);
console.info(`Voice startup phase=${phase} elapsed_ms=${this.startupMarks[phase]}`);
if (this.ws?.readyState === WebSocket.OPEN) {
- this.ws.send(JSON.stringify({ type: "startup_timing", marks_ms: this.startupMarks }));
+ // A new frontend can load before its backend restarts. Older servers
+ // bound each message to their smaller phase allowlist; flush early
+ // milestones in bounded batches, then send only the newly reached one.
+ const marks = phase === "socket_open" ? Object.entries(this.startupMarks) : [[phase, this.startupMarks[phase]]];
+ for (let start = 0; start < marks.length; start += 8) {
+ this.ws.send(JSON.stringify({ type: "startup_timing", marks_ms: Object.fromEntries(marks.slice(start, start + 8)) }));
+ }
}
}
@@ -687,6 +747,11 @@ export class RealtimeAudioClient {
}
private async open(): Promise<void> {
+ const localStartupAbort = new AbortController();
+ this.localStartupAbort = localStartupAbort;
+ const markLocalStartup = (phase: string) => {
+ if (!localStartupAbort.signal.aborted) this.markStartup(phase);
+ };
// A reused client starts a new source timeline. Keep the local render
// generation monotonic so callbacks from the old worklet stay obsolete.
this.timedOutput = false;
@@ -707,37 +772,23 @@ export class RealtimeAudioClient {
try {
const supportIssue = browserRealtimeSupportIssue();
if (supportIssue) throw new RealtimeAudioSupportError(supportIssue);
- this.ctx = new AudioContext({ latencyHint: "interactive" });
+ this.preparedAudio = acquireRealtimeAudio();
+ this.ctx = this.preparedAudio.context;
+ this.markStartup(this.preparedAudio.prepared ? "context_prepared" : "context_created");
if (!this.ctx.audioWorklet) {
throw new RealtimeAudioSupportError("audio_worklet_unavailable");
}
- if (this.options.browserAudio && this.options.requiresWebRtcOffer) {
- // This destination has no source yet: negotiate a silent track while
- // the microphone opens. The startup worklet later owns its only input.
- this.rtcInput = this.ctx.createMediaStreamDestination();
- this.rtcInput.channelCount = 1;
- }
- // Start ICE on a silent destination while microphone/worklet setup runs.
- // No raw microphone track enters the peer: pcm-startup is the sole
- // source and stays silent until both media and control are ready.
- const webRtcOffer: Promise<{
- sdp: string | null;
- error: unknown | null;
- }> | null = this.options.requiresWebRtcOffer
- ? this.webRtcTransport.createOffer(this.rtcInput?.stream, this.options.webRtcStartEventRequired, this.options.startBudgetMs).then(
- (sdp) => ({ sdp, error: null }),
- (error: unknown) => ({ sdp: null, error }),
- )
- : null;
// The worklet load, the microphone open, the one-time WS ticket and
// the shared connect-budget turn are independent: acquiring them
// concurrently keeps three disk/network waits off the wake path
// instead of stacked end to end. The silent WebRTC destination also
// allows offer generation to overlap permission and device setup.
const setupStartedAt = performance.now();
- const workletReady = this.ctx.audioWorklet
- .addModule(pcmWorkletUrl)
- .then(() => this.ctx?.resume());
+ const workletReady = Promise.all([
+ this.preparedAudio.loaded.then(() => markLocalStartup("worklet_loaded")),
+ this.ctx.resume().then(() => markLocalStartup("context_running")),
+ ])
+ .then(() => markLocalStartup("worklet_ready"));
// Observed again below, but a rejected worklet must not become an
// unhandled rejection while a permission prompt holds getUserMedia.
void workletReady.catch(() => undefined);
@@ -749,11 +800,47 @@ export class RealtimeAudioClient {
autoGainControl: true,
},
});
+ // getUserMedia has no cancellation API. Release a grant that arrives
+ // after Stop without ever attaching it to a graph or provider.
+ void micReady.then(stream => {
+ if (localStartupAbort.signal.aborted) stream.getTracks().forEach(track => track.stop());
+ }, () => undefined); // The awaited operation below reports permission errors.
const ticketReady = mintWsTicket();
- const turnReady = new Promise<void>((resolve) => requestConnect(resolve));
- this.stream = await micReady;
+ const turnReady = new Promise<void>((resolve) => {
+ const cancelTurn = requestConnect(resolve);
+ localStartupAbort.signal.addEventListener("abort", cancelTurn, { once: true });
+ });
+ this.markStartup("local_requests_dispatched");
+ // Native destination/peer construction can block the main thread. Start
+ // the independent device and network requests before entering that work.
+ // Promise callbacks can only run afterward; their timestamps alone do
+ // not measure the underlying module/device operation's duration.
+ if (this.options.browserAudio && this.options.requiresWebRtcOffer) {
+ this.markStartup("rtc_destination_begin");
+ // This destination has no source yet: negotiate a silent track while
+ // the microphone opens. The startup worklet later owns its only input.
+ this.rtcInput = this.ctx.createMediaStreamDestination();
+ this.rtcInput.channelCount = 1;
+ this.markStartup("rtc_destination_ready");
+ }
+ // No raw microphone track enters the peer: pcm-startup is the sole
+ // source and stays silent until both media and control are ready.
+ const webRtcOffer: Promise<{
+ sdp: string | null;
+ error: unknown | null;
+ }> | null = this.options.requiresWebRtcOffer
+ ? this.webRtcTransport.createOffer(this.rtcInput?.stream, this.options.webRtcStartEventRequired, this.options.startBudgetMs).then(
+ (sdp) => {
+ if (sdp) markLocalStartup("offer_ready");
+ return { sdp, error: null };
+ },
+ (error: unknown) => ({ sdp: null, error }),
+ )
+ : null;
+ this.stream = await awaitMicrophone(micReady, localStartupAbort.signal);
if (this.intentionalClose) throw new Error("Voice start cancelled");
- await workletReady;
+ this.markStartup("microphone_ready");
+ await awaitLocalStartup(workletReady, localStartupAbort.signal, LOCAL_AUDIO_READY_TIMEOUT_MS);
if (this.intentionalClose) throw new Error("Voice start cancelled");
const setupMs = Math.round(performance.now() - setupStartedAt);
const source = this.ctx.createMediaStreamSource(this.stream);
@@ -796,6 +883,8 @@ export class RealtimeAudioClient {
source.connect(this.startupNode);
this.startupNode.connect(this.rtcInput!);
this.startupNode.port.onmessage = (event: MessageEvent) => {
+ if (event.data?.type === "input_started") this.markStartup("input_started");
+ else if (event.data?.type === "input_caught_up") this.markStartup("input_caught_up");
if (event.data?.type !== "error" || this.intentionalClose) return;
this.cb.onStatus?.("provider_error", { error: "Voice startup audio could not be retained. Please start again." });
void this.disconnect();
@@ -808,7 +897,7 @@ export class RealtimeAudioClient {
// alone. A subscription session cannot be opened or billed correctly
// without its WebRTC peer, so this path fails closed.
if (this.options.requiresWebRtcOffer) {
- const result = await webRtcOffer;
+ const result = await awaitLocalStartup(Promise.resolve(webRtcOffer), localStartupAbort.signal);
if (result?.error) {
this.webRtcTransport.close();
this.webRtcOfferSdp = null;
@@ -821,7 +910,6 @@ export class RealtimeAudioClient {
this.webRtcTransport.close();
throw new Error("Subscription Realtime requires WebRTC support");
}
- this.markStartup("offer_ready");
}
// Proactive one-time ticket: WebKit engines do not attach the HttpOnly
@@ -831,14 +919,15 @@ export class RealtimeAudioClient {
// The ticket was minted concurrently with the microphone setup above,
// and the budget turn was queued there too, so both are (nearly) free
// by now instead of two more serial waits on the wake path.
- const ticket = await ticketReady;
- await turnReady;
+ const ticket = await awaitLocalStartup(ticketReady, localStartupAbort.signal);
+ await awaitLocalStartup(turnReady, localStartupAbort.signal);
console.info(`Voice start setup took ${setupMs} ms (mic/worklet/ticket).`);
if (this.intentionalClose) throw new Error("Voice start cancelled");
this.ws = new WebSocket(buildAudioSocketUrl(ticket));
this.ws.binaryType = "arraybuffer";
await this.waitUntilReady(this.ws);
} catch (error) {
+ localStartupAbort.abort();
await this.teardown(false);
throw error instanceof Error ? error : new Error(String(error));
}
@@ -1039,6 +1128,7 @@ export class RealtimeAudioClient {
});
return;
} else if (type === "audio_ready") {
+ this.markStartup("control_ready");
this.selectOutputTransport(message.output_transport, message.session_id);
// The local control channel can already relay levels while RTP
// finishes its handshake. Native and browser bars share this input.
@@ -1204,6 +1294,7 @@ export class RealtimeAudioClient {
}
private handleAudio(pcm: ArrayBuffer, timing?: AudioInterval): void {
+ if (pcm.byteLength) this.markStartup("first_output_packet");
if (this.outputMuted) return;
// Timed sessions mute the peer's RTP element before negotiation. Only
// this worklet owns their audible output and its source-time mapping.
@@ -1248,6 +1339,7 @@ export class RealtimeAudioClient {
async disconnect(): Promise<void> {
this.intentionalClose = true;
+ this.localStartupAbort?.abort();
this.applyOutputState(true, this.outputVolume, undefined);
if (this.options.browserAudio && this.ready && !this.serverClosed && this.ws?.readyState === WebSocket.OPEN) {
this.ready = false;
@@ -1266,6 +1358,8 @@ export class RealtimeAudioClient {
}
private async teardown(sendStop: boolean): Promise<void> {
+ this.localStartupAbort?.abort();
+ this.localStartupAbort = null;
this.stopListeningCue?.();
this.stopListeningCue = null;
if (this.remoteMeter) this.remoteMeter.port.onmessage = null;
@@ -1306,11 +1400,8 @@ export class RealtimeAudioClient {
this.captureSink?.disconnect();
this.playbackNode?.disconnect();
this.stream?.getTracks().forEach((track) => track.stop());
- try {
- await this.ctx?.close();
- } catch {
- // Closing an already closed AudioContext is harmless.
- }
+ const preparedAudio = this.preparedAudio;
+ this.preparedAudio = null;
this.captureNode = null;
this.captureSink = null;
this.playbackNode = null;
@@ -1325,5 +1416,10 @@ export class RealtimeAudioClient {
this.cb.onInputLevel?.(0);
// null, not 0: playback is gone, so nothing may keep drawing a voice.
this.cb.onOutputLevel?.(null);
+ // Native close can itself stall after device failure. Tracks, graph and
+ // peer are already gone; retirement owns only this old context and never
+ // delays Stop/start failure or mutates a later call. The pool prevents
+ // idle replacement preparation until native cleanup actually settles.
+ if (preparedAudio) void releaseRealtimeAudio(preparedAudio);
}
}
diff --git a/jarvis/ui/web/frontend/src/lib/realtimeAudioPreparation.test.ts b/jarvis/ui/web/frontend/src/lib/realtimeAudioPreparation.test.ts
new file mode 100644
index 0000000000..6a49dc371d
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/realtimeAudioPreparation.test.ts
@@ -0,0 +1,148 @@
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import {
+ acquireRealtimeAudio,
+ registerRealtimeAudioPreparation,
+ releaseRealtimeAudio,
+ type PreparedRealtimeAudio,
+} from "./realtimeAudioPreparation";
+
+class LocalAudioContext {
+ static instances: LocalAudioContext[] = [];
+ static moduleDelayMs = 0;
+ state = "running";
+ constructor() { LocalAudioContext.instances.push(this); }
+ audioWorklet = { addModule: vi.fn(() => new Promise<void>(resolve => {
+ setTimeout(resolve, LocalAudioContext.moduleDelayMs);
+ })) };
+ suspend = vi.fn(async () => { this.state = "suspended"; });
+ resume = vi.fn(async () => { this.state = "running"; });
+ close = vi.fn(async () => { this.state = "closed"; });
+}
+
+const disposers: Array<() => void> = [];
+const leases: PreparedRealtimeAudio[] = [];
+function register() { const dispose = registerRealtimeAudioPreparation(); disposers.push(dispose); return dispose; }
+function acquire() { const entry = acquireRealtimeAudio(); leases.push(entry); return entry; }
+
+beforeEach(() => {
+ vi.useFakeTimers();
+ LocalAudioContext.instances = [];
+ LocalAudioContext.moduleDelayMs = 0;
+ vi.stubGlobal("AudioContext", LocalAudioContext);
+ vi.stubGlobal("AudioWorkletNode", vi.fn());
+ vi.stubGlobal("RTCPeerConnection", vi.fn());
+ vi.stubGlobal("fetch", vi.fn());
+ vi.stubGlobal("navigator", { mediaDevices: { getUserMedia: vi.fn() } });
+});
+
+afterEach(async () => {
+ disposers.splice(0).forEach(dispose => dispose());
+ await Promise.all(leases.splice(0).map(releaseRealtimeAudio));
+ vi.useRealTimers();
+ vi.unstubAllGlobals();
+});
+
+it("prepares a single suspended worklet without microphone, peer or provider requests", async () => {
+ register(); register();
+ expect(LocalAudioContext.instances).toHaveLength(0);
+ await vi.advanceTimersByTimeAsync(501);
+ expect(LocalAudioContext.instances).toHaveLength(1);
+ const context = LocalAudioContext.instances[0];
+ expect(context.state).toBe("suspended");
+ expect(context.resume).not.toHaveBeenCalled();
+ expect(context.audioWorklet.addModule).toHaveBeenCalledOnce();
+ expect(navigator.mediaDevices.getUserMedia).not.toHaveBeenCalled();
+ expect(RTCPeerConnection).not.toHaveBeenCalled();
+ expect(fetch).not.toHaveBeenCalled();
+ expect(acquire().context).toBe(context);
+});
+
+it("moves a simulated two-second worklet load entirely before activation", async () => {
+ LocalAudioContext.moduleDelayMs = 2_000;
+ const coldAt = Date.now();
+ const cold = acquire();
+ await vi.advanceTimersByTimeAsync(2_000);
+ await cold.loaded;
+ expect(Date.now() - coldAt).toBe(2_000);
+ await releaseRealtimeAudio(cold);
+ register();
+ await vi.advanceTimersByTimeAsync(2_501);
+ const warmAt = Date.now();
+ const warm = acquire();
+ await warm.loaded;
+ expect(warm.prepared).toBe(true);
+ expect(Date.now() - warmAt).toBe(0);
+ expect(LocalAudioContext.instances).toHaveLength(2);
+});
+
+it("cancels StrictMode's first effect and never replenishes during a call", async () => {
+ const dispose = register();
+ dispose();
+ register();
+ await vi.advanceTimersByTimeAsync(501);
+ const entry = acquire();
+ register();
+ await vi.advanceTimersByTimeAsync(5_000);
+ expect(LocalAudioContext.instances).toHaveLength(1);
+ await releaseRealtimeAudio(entry);
+ await vi.advanceTimersByTimeAsync(501);
+ expect(LocalAudioContext.instances).toHaveLength(2);
+ expect(LocalAudioContext.instances[0].close).toHaveBeenCalledOnce();
+});
+
+it("transfers an unfinished preparation exclusively without duplicate module loads", async () => {
+ LocalAudioContext.moduleDelayMs = 2_000;
+ register();
+ await vi.advanceTimersByTimeAsync(501);
+ const first = acquire();
+ const second = acquire();
+ expect(first.context).not.toBe(second.context);
+ expect(first.prepared).toBe(true);
+ expect(second.prepared).toBe(false);
+ await vi.advanceTimersByTimeAsync(2_000);
+ await Promise.all([first.loaded, second.loaded]);
+ expect(LocalAudioContext.instances.every(context => context.audioWorklet.addModule.mock.calls.length === 1)).toBe(true);
+});
+
+it("disposes idle contexts on disable and pagehide and prepares again after pageshow", async () => {
+ const dispose = register();
+ await vi.advanceTimersByTimeAsync(501);
+ window.dispatchEvent(new Event("pagehide"));
+ expect(LocalAudioContext.instances[0].close).toHaveBeenCalledOnce();
+ await vi.advanceTimersByTimeAsync(5_000);
+ expect(LocalAudioContext.instances).toHaveLength(1);
+ window.dispatchEvent(new Event("pageshow"));
+ await vi.advanceTimersByTimeAsync(501);
+ expect(LocalAudioContext.instances).toHaveLength(2);
+ dispose();
+ expect(LocalAudioContext.instances[1].close).toHaveBeenCalledOnce();
+});
+
+it("keeps an acquired context alive until its call releases it after unmount", async () => {
+ const dispose = register();
+ await vi.advanceTimersByTimeAsync(501);
+ const entry = acquire();
+ dispose();
+ expect(LocalAudioContext.instances[0].close).not.toHaveBeenCalled();
+ await releaseRealtimeAudio(entry);
+ await releaseRealtimeAudio(entry);
+ expect(LocalAudioContext.instances[0].close).toHaveBeenCalledOnce();
+ await vi.advanceTimersByTimeAsync(5_000);
+ expect(LocalAudioContext.instances).toHaveLength(1);
+});
+
+it("does not prepare a replacement while native context retirement is stalled", async () => {
+ register();
+ await vi.advanceTimersByTimeAsync(501);
+ const entry = acquire();
+ let finishClose!: () => void;
+ LocalAudioContext.instances[0].close = vi.fn(() => new Promise<void>(resolve => { finishClose = resolve; }));
+ const retiring = releaseRealtimeAudio(entry);
+ register();
+ await vi.advanceTimersByTimeAsync(10_000);
+ expect(LocalAudioContext.instances).toHaveLength(1);
+ finishClose();
+ await retiring;
+ await vi.advanceTimersByTimeAsync(501);
+ expect(LocalAudioContext.instances).toHaveLength(2);
+});
diff --git a/jarvis/ui/web/frontend/src/lib/realtimeAudioPreparation.ts b/jarvis/ui/web/frontend/src/lib/realtimeAudioPreparation.ts
new file mode 100644
index 0000000000..753ef0a0e1
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/lib/realtimeAudioPreparation.ts
@@ -0,0 +1,119 @@
+import pcmWorkletUrl from "./pcm-worklet.ts?worker&url";
+
+export type PreparedRealtimeAudio = {
+ context: AudioContext;
+ loaded: Promise<void>;
+ prepared: boolean;
+};
+
+// Only local code and a suspended output context are prepared. There is no
+// microphone, audio graph, peer, ticket or provider session until activation.
+let idle: PreparedRealtimeAudio | null = null;
+const borrowed = new Set<PreparedRealtimeAudio>();
+const retiring = new Set<PreparedRealtimeAudio>();
+const owners = new Set<symbol>();
+let cancelScheduled: (() => void) | null = null;
+let pageHidden = false;
+
+async function close(entry: PreparedRealtimeAudio): Promise<void> {
+ try { await entry.context.close(); }
+ catch (error) { console.warn("Prepared voice audio could not close", error); }
+}
+
+function create(prepared: boolean): PreparedRealtimeAudio {
+ const context = new AudioContext({ latencyHint: "interactive" });
+ try {
+ // Suspend immediately, before loading local DSP code. Never resume an
+ // idle context; it must neither render nor hold the output device open.
+ const suspended = prepared ? context.suspend() : Promise.resolve();
+ const loaded = Promise.all([suspended, context.audioWorklet.addModule(pcmWorkletUrl)])
+ .then(() => undefined);
+ return { context, loaded, prepared };
+ } catch (error) {
+ void context.close().catch(closeError => console.warn("Voice audio cleanup failed", closeError));
+ throw error;
+ }
+}
+
+function schedule(): void {
+ if (!owners.size || borrowed.size || retiring.size || idle || cancelScheduled || pageHidden ||
+ typeof AudioContext !== "function" || typeof AudioWorkletNode !== "function") return;
+ const prepare = () => {
+ cancelScheduled = null;
+ if (!owners.size || borrowed.size || retiring.size || idle || pageHidden) return;
+ try {
+ const entry = create(true);
+ idle = entry;
+ void entry.loaded.catch(error => {
+ console.warn("Local voice audio preparation failed", error);
+ if (idle === entry) { idle = null; void close(entry); }
+ });
+ } catch (error) { console.warn("Local voice audio preparation failed", error); }
+ };
+ if (typeof window.requestIdleCallback === "function") {
+ const handle = window.requestIdleCallback(prepare, { timeout: 2_000 });
+ cancelScheduled = () => window.cancelIdleCallback(handle);
+ } else {
+ const handle = window.setTimeout(prepare, 500);
+ cancelScheduled = () => window.clearTimeout(handle);
+ }
+}
+
+function discardIdle(): void {
+ cancelScheduled?.();
+ cancelScheduled = null;
+ const entry = idle;
+ idle = null;
+ if (entry) void close(entry);
+}
+
+function onPageHide(): void { pageHidden = true; discardIdle(); }
+function onPageShow(): void { pageHidden = false; schedule(); }
+
+/** The selected voice surface owns preparation, including its disposal. */
+export function registerRealtimeAudioPreparation(): () => void {
+ const owner = Symbol();
+ if (!owners.size) {
+ pageHidden = false;
+ window.addEventListener("pagehide", onPageHide);
+ window.addEventListener("pageshow", onPageShow);
+ }
+ owners.add(owner);
+ schedule();
+ return () => {
+ owners.delete(owner);
+ if (!owners.size) {
+ discardIdle();
+ window.removeEventListener("pagehide", onPageHide);
+ window.removeEventListener("pageshow", onPageShow);
+ }
+ };
+}
+
+/** Exclusive transfer: two calls never share one audio context or graph. */
+export function acquireRealtimeAudio(): PreparedRealtimeAudio {
+ cancelScheduled?.();
+ cancelScheduled = null;
+ if (idle?.context.state === "closed") idle = null;
+ const entry = idle ?? create(false);
+ idle = null;
+ borrowed.add(entry);
+ return entry;
+}
+
+export async function releaseRealtimeAudio(entry: PreparedRealtimeAudio): Promise<void> {
+ if (!borrowed.delete(entry)) return;
+ retiring.add(entry);
+ try { await close(entry); }
+ finally { retiring.delete(entry); }
+ // No session nodes, tracks or captured frames survive into preparation for
+ // the next call. Its fresh graph is constructed only after another start.
+ schedule();
+}
+
+if (import.meta.hot) import.meta.hot.dispose(() => {
+ owners.clear();
+ discardIdle();
+ window.removeEventListener("pagehide", onPageHide);
+ window.removeEventListener("pageshow", onPageShow);
+});
diff --git a/jarvis/ui/web/frontend/src/lib/startupAudio.test.ts b/jarvis/ui/web/frontend/src/lib/startupAudio.test.ts
index 28869914a9..07ae00fb6e 100644
--- a/jarvis/ui/web/frontend/src/lib/startupAudio.test.ts
+++ b/jarvis/ui/web/frontend/src/lib/startupAudio.test.ts
@@ -18,6 +18,81 @@ it("discards muted audio and resumes retention without releasing it early", () =
});
describe("startup RTP audio", () => {
+ it.each([16000, 24000, 44100, 48000])("bounds a four-second silent opening to 100 ms and catches up within 300 ms at %s Hz", rate => {
+ const queue = new StartupAudioQueue(rate);
+ const block = 128;
+ const silence = new Float32Array(block), output = new Float32Array(block);
+ for (let n = 0; n < rate * 4; n += block) queue.process(silence, output);
+ expect(queue.pendingMs).toBeGreaterThanOrEqual(4000);
+ queue.start();
+ expect(queue.pendingMs).toBeCloseTo(100, 3);
+ queue.start(); // Repeated readiness must not alter the retained opening.
+ expect(queue.pendingMs).toBeCloseTo(100, 3);
+ for (let n = 0; n < rate * 0.3; n += block) queue.process(silence, output);
+ expect(queue.pendingMs).toBeLessThan(10);
+ const fresh = Float32Array.from({ length: block }, (_, i) => Math.sin(i / 7) * 0.0005);
+ queue.process(fresh, output);
+ expect(output).toEqual(fresh);
+ });
+
+ it("compacts across the native/browser boundary without removing the quiet onset or spoken pauses", () => {
+ const rate = 16000, queue = new StartupAudioQueue(rate);
+ const quiet = Float32Array.from([1e-30, -1e-30, 0.0005, -0.0005]);
+ const pause = new Float32Array(rate);
+ queue.prepend(new Float32Array(rate));
+ queue.process(new Float32Array(rate), new Float32Array(128));
+ queue.process(quiet, new Float32Array(128));
+ queue.process(pause, new Float32Array(128));
+ queue.process(quiet, new Float32Array(128));
+ queue.start();
+ const expected = new Float32Array(rate / 10 + quiet.length * 2 + pause.length);
+ expected.set(quiet, rate / 10);
+ expected.set(quiet, rate / 10 + quiet.length + pause.length);
+ expect(queue.pendingMs).toBeCloseTo(expected.length / rate * 1000, 3);
+ // Drain in one block to inspect compaction independently of the existing
+ // pitch-preserving replay. Every remaining sample must be bit-exact.
+ const output = new Float32Array(expected.length);
+ queue.process(new Float32Array(), output);
+ expect(output).toEqual(expected);
+ });
+
+ it("retains browser-leading silence when the native prefix already contains speech", () => {
+ const rate = 16000, queue = new StartupAudioQueue(rate);
+ const quiet = Float32Array.from([1e-30, -1e-30]);
+ queue.prepend(quiet);
+ queue.process(new Float32Array(rate * 4), new Float32Array(128));
+ queue.start();
+ const expected = new Float32Array(quiet.length + rate * 4);
+ expected.set(quiet);
+ expect(queue.pendingMs).toBeCloseTo(expected.length / rate * 1000, 3);
+ const output = new Float32Array(expected.length);
+ queue.process(new Float32Array(), output);
+ expect(output).toEqual(expected);
+ });
+
+ it("leaves a short quiet opening unchanged and forgets compaction state after mute or cancellation", () => {
+ const rate = 16000, queue = new StartupAudioQueue(rate);
+ const shortOpening = new Float32Array(rate / 20).fill(0.0005);
+ queue.process(shortOpening, new Float32Array(128));
+ queue.start();
+ expect(queue.pendingMs).toBe(50);
+ const output = new Float32Array(shortOpening.length);
+ queue.process(new Float32Array(), output);
+ expect(output).toEqual(shortOpening);
+ queue.suspend();
+ queue.process(new Float32Array(rate).fill(0.7), output);
+ queue.resume();
+ queue.process(new Float32Array(rate * 4), output);
+ queue.start();
+ expect(queue.pendingMs).toBe(100);
+ queue.suspend();
+ queue.resume();
+ queue.process(shortOpening, output);
+ queue.start();
+ queue.process(new Float32Array(), output);
+ expect(output).toEqual(shortOpening);
+ });
+
it.each([16000, 24000, 44100, 48000])("pays down a two-second backlog despite constant room noise at %s Hz", rate => {
const queue = new StartupAudioQueue(rate);
const block = 128;
diff --git a/jarvis/ui/web/frontend/src/lib/startupAudio.ts b/jarvis/ui/web/frontend/src/lib/startupAudio.ts
index 213744d9e2..751d1a027d 100644
--- a/jarvis/ui/web/frontend/src/lib/startupAudio.ts
+++ b/jarvis/ui/web/frontend/src/lib/startupAudio.ts
@@ -55,6 +55,11 @@ export class StartupAudioQueue {
// backlog does: waiting for exactly zero microphone samples stranded it
// indefinitely in rooms with even very faint background noise.
this.catchingUp = this.samples > this.rate * 0.1;
+ // Check the combined native/browser opening only now: browser-leading
+ // zeros can be a spoken pause if the native prefix already contains speech.
+ // Keep the original catch-up decision so a compacted 100 ms reserve does
+ // not become a permanent offset behind the live microphone.
+ this.compactLeadingSilence();
}
suspend(): void {
@@ -117,6 +122,22 @@ export class StartupAudioQueue {
}
}
+ private compactLeadingSilence(): void {
+ let leading = 0;
+ opening: for (let frameIndex = this.head; frameIndex < this.frames.length; frameIndex++) {
+ const frame = this.frames[frameIndex];
+ for (let i = frameIndex === this.head ? this.offset : 0; i < frame.length; i++) {
+ if (frame[i] !== 0) break opening;
+ leading++;
+ }
+ }
+ // Only exact digital silence before the first nonzero sample is removed.
+ // Quiet speech and every pause after it retain their original samples.
+ // The short reserve keeps an onset separate from the transport boundary.
+ const removable = leading - Math.round(this.rate * 0.1);
+ if (removable > 0) this.consume(removable);
+ }
+
private consume(count: number): void {
this.samples -= count;
while (count) {
diff --git a/jarvis/ui/web/frontend/src/store/agentChat.test.ts b/jarvis/ui/web/frontend/src/store/agentChat.test.ts
index e3cdcbab9d..9326b3d0be 100644
--- a/jarvis/ui/web/frontend/src/store/agentChat.test.ts
+++ b/jarvis/ui/web/frontend/src/store/agentChat.test.ts
@@ -4,6 +4,7 @@ import type { AgentChatSession } from "@/lib/agentChatApi";
import { providerKind } from "@/components/agentchat/AgentComposer";
import { EMPTY_TIMELINE, type Timeline, type UserItem } from "@/components/agentchat/reduce";
import { createAgentChatStore, draftKey } from "@/store/agentChat";
+import { captureComposerDraftTarget, readComposerDraft, writeComposerDraft } from "@/components/agentchat/composerDrafts";
/**
* The store knows its surface: the front page's store speaks for the typed
@@ -337,6 +338,123 @@ function timelineWith(text: string): Timeline {
return { ...EMPTY_TIMELINE, items: [item], lastSeq: 1 };
}
+describe("followup admission", () => {
+ beforeEach(() => {
+ window.localStorage.clear();
+ vi.stubGlobal("WebSocket", FakeSocket);
+ });
+ afterEach(() => vi.unstubAllGlobals());
+
+ it("shares session creation and preserves rapid message order until every request settles", async () => {
+ let open!: (response: Response) => void;
+ let finishFirst!: (response: Response) => void;
+ const messages: string[] = [];
+ let creations = 0;
+ vi.stubGlobal("fetch", vi.fn(async (input: string, init?: RequestInit) => {
+ const url = String(input);
+ if (url === "/api/agent-chat/sessions" && init?.method === "POST") {
+ creations += 1;
+ return new Promise<Response>((resolve) => { open = resolve; });
+ }
+ if (url.endsWith("/messages")) {
+ messages.push(JSON.parse(String(init?.body)).text);
+ if (messages.length === 1) return new Promise<Response>((resolve) => { finishFirst = resolve; });
+ return new Response(JSON.stringify({ queued: true, queue_id: "q2" }));
+ }
+ return new Response(JSON.stringify({ sessions: [] }));
+ }));
+ const store = createAgentChatStore("jarvis");
+ const first = store.getState().send("first");
+ const second = store.getState().send("second");
+ await vi.waitFor(() => expect(creations).toBe(1));
+ const target = captureComposerDraftTarget(store, null);
+ const unsent = { text: "still typing while the chat opens", choices: [] };
+ writeComposerDraft(store, null, unsent);
+ open(new Response(JSON.stringify(session("created", "jarvis"))));
+ await vi.waitFor(() => expect(messages).toEqual(["first"]));
+ expect(readComposerDraft(store, "created")).toEqual(unsent);
+ expect(readComposerDraft(store, null).text).toBe("");
+ expect(target.sessionId).toBe("created");
+ const third = store.getState().send("third");
+ expect(store.getState().busy).toBe(true);
+ finishFirst(new Response(JSON.stringify({ turn_id: "t1" })));
+ expect(await Promise.all([first, second, third])).toEqual(["sent", "sent", "sent"]);
+ expect(messages).toEqual(["first", "second", "third"]);
+ expect(creations).toBe(1);
+ expect(store.getState().busy).toBe(false);
+ expect(readComposerDraft(store, "created")).toEqual(unsent);
+ });
+
+ it("continues queued submissions after one HTTP failure without changing the selected chat", async () => {
+ let failFirst!: (response: Response) => void;
+ const messages: { url: string; text: string }[] = [];
+ vi.stubGlobal("fetch", vi.fn(async (input: string, init?: RequestInit) => {
+ const url = String(input);
+ if (url.endsWith("/messages")) {
+ messages.push({ url, text: JSON.parse(String(init?.body)).text });
+ if (messages.length === 1) return new Promise<Response>((resolve) => { failFirst = resolve; });
+ return new Response(JSON.stringify({ queued: true, queue_id: "q" }));
+ }
+ return new Response(JSON.stringify({ sessions: [] }));
+ }));
+ const store = createAgentChatStore("society");
+ store.setState({ activeSessionId: "society:a" });
+ const first = store.getState().send("first");
+ const second = store.getState().send("second");
+ await vi.waitFor(() => expect(messages).toHaveLength(1));
+ store.setState({ activeSessionId: "society:b", busy: false, lastError: null });
+ failFirst(new Response(JSON.stringify({ detail: "temporarily unavailable" }), { status: 503 }));
+ expect(await first).toBe("failed");
+ expect(await second).toBe("stale");
+ expect(messages.map((row) => row.text)).toEqual(["first", "second"]);
+ expect(messages.every((row) => row.url.includes("society%3Aa"))).toBe(true);
+ expect(store.getState()).toMatchObject({ activeSessionId: "society:b", busy: false, lastError: null });
+ });
+
+ it("does not reopen a new chat after the person switched away during creation", async () => {
+ let open!: (response: Response) => void;
+ const messages: string[] = [];
+ vi.stubGlobal("fetch", vi.fn(async (input: string, init?: RequestInit) => {
+ const url = String(input);
+ if (url === "/api/agent-chat/sessions" && init?.method === "POST") {
+ return new Promise<Response>((resolve) => { open = resolve; });
+ }
+ if (url.endsWith("/messages")) messages.push(url);
+ return new Response(JSON.stringify({ sessions: [], turn_id: "t" }));
+ }));
+ const store = createAgentChatStore("jarvis");
+ const sending = store.getState().send("for the original chat");
+ await vi.waitFor(() => expect(open).toBeTypeOf("function"));
+ store.setState({ activeSessionId: "elsewhere", busy: false });
+ open(new Response(JSON.stringify(session("created", "jarvis"))));
+ expect(await sending).toBe("stale");
+ expect(messages).toEqual(["/api/agent-chat/sessions/created/messages"]);
+ expect(store.getState().activeSessionId).toBe("elsewhere");
+ });
+
+ it("marks a replaced blank draft so a rejected creation cannot restore into the new chat", async () => {
+ let rejectCreation!: (response: Response) => void;
+ vi.stubGlobal("fetch", vi.fn(async (input: string, init?: RequestInit) => {
+ if (String(input) === "/api/agent-chat/sessions" && init?.method === "POST") {
+ return new Promise<Response>((resolve) => { rejectCreation = resolve; });
+ }
+ return new Response(JSON.stringify({ sessions: [], providers: [] }));
+ }));
+ const store = createAgentChatStore("jarvis");
+ const original = captureComposerDraftTarget(store, null);
+ const sending = store.getState().send("original send");
+ await vi.waitFor(() => expect(rejectCreation).toBeTypeOf("function"));
+ store.getState().newChat();
+ writeComposerDraft(store, null, { text: "deliberate new draft", choices: [] });
+ rejectCreation(new Response(JSON.stringify({ detail: "unavailable" }), { status: 503 }));
+ expect(await sending).toBe("failed");
+ expect(original.superseded).toBe(true);
+ expect(captureComposerDraftTarget(store, null)).not.toBe(original);
+ expect(readComposerDraft(store, null).text).toBe("deliberate new draft");
+ expect(store.getState()).toMatchObject({ activeSessionId: null, lastError: null });
+ });
+});
+
describe("agent-chat store session switch", () => {
beforeEach(() => {
window.localStorage.clear();
diff --git a/jarvis/ui/web/frontend/src/store/agentChat.ts b/jarvis/ui/web/frontend/src/store/agentChat.ts
index a609a28273..f0894c760b 100644
--- a/jarvis/ui/web/frontend/src/store/agentChat.ts
+++ b/jarvis/ui/web/frontend/src/store/agentChat.ts
@@ -1,6 +1,9 @@
-import { create } from "zustand";
+import { create, type StoreApi, type UseBoundStore } from "zustand";
+import { bindComposerDraftTarget, captureComposerDraftTarget, resetBlankComposerDraftTarget, type ComposerDraftTarget } from "@/components/agentchat/composerDrafts";
+import { discardHeldFiles, moveHeldFiles } from "@/components/agentchat/useChatAttachments";
import {
+ AgentChatApiError,
agentChatSocketUrl,
cancelAgentChatTurn,
createAgentChatSession,
@@ -34,7 +37,7 @@ import {
type PatchSessionInput,
type ProviderHealth,
} from "@/lib/agentChatApi";
-import { EMPTY_TIMELINE, reduceEvent, reduceEvents, type Timeline } from "@/components/agentchat/reduce";
+import { EMPTY_TIMELINE, reduceEvent, reduceEvents, settleApproval, type Timeline } from "@/components/agentchat/reduce";
import { jitteredDelay, requestConnect } from "../lib/connectBudget";
import { reuseHistoryRows } from "@/lib/historyRequests";
@@ -175,8 +178,14 @@ export interface AgentChatStore {
openSession: (sessionId: string) => void;
removeSession: (sessionId: string) => Promise<void>;
/** Send the sentence, with whatever files the composer is holding for it. */
- send: (text: string, attachments?: ChatAttachment[], toolChoices?: string[]) => Promise<void | "sent" | "failed" | "stale">;
- cancel: () => Promise<void>;
+ send: (text: string, attachments?: ChatAttachment[], toolChoices?: string[], onSessionCreated?: (session: AgentChatSession) => void) => Promise<void | "sent" | "failed" | "stale">;
+ /** Stop the running turn; `via` names the control that asked, for the log. */
+ cancel: (via?: string) => Promise<void>;
+ /**
+ * Answer an approval card in the active session. The card closes as soon as
+ * the backend took the answer, or said nothing waits on it any more; any
+ * other failure throws so the card can say why.
+ */
decide: (approvalId: string, decision: ApprovalDecision) => Promise<void>;
/** Answer question `index` of an agent's card in the active session; throws so the card can say why. */
answerQuestion: (questionId: string, index: number, answer: QuestionAnswerInput) => Promise<void>;
@@ -330,7 +339,7 @@ function onLadder(value: string, ladder: readonly string[], fallback: string): s
/** How many left-behind transcripts the store keeps for an instant switch-back. */
const TIMELINE_CACHE_LIMIT = 24;
-export function createAgentChatStore(surface: AgentChatSurface, draftNamespace = "") {
+export function createAgentChatStore(surface: AgentChatSurface, draftNamespace = ""): UseBoundStore<StoreApi<AgentChatStore>> {
const DRAFT_KEY = draftKey(surface) + (draftNamespace ? `:${draftNamespace}` : "");
/**
@@ -350,6 +359,10 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
let catalogRequest = 0;
const patchQueues = new Map<string, Promise<void>>();
const patchRevisions = new Map<string, number>();
+ type SendLane = { tail: Promise<unknown>; pending: number; sessionId: string | null; draftTarget?: ComposerDraftTarget };
+ const sendLanes = new Map<string, SendLane>();
+ const emptySendLane = (): SendLane => ({ tail: Promise.resolve(), pending: 0, sessionId: null });
+ let newChatLane = emptySendLane();
let selectionRevision = 0;
let selectionWrite: Promise<unknown> = Promise.resolve();
let socket: WebSocket | null = null;
@@ -397,7 +410,7 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
}
}
- return create<AgentChatStore>((set, get) => {
+ const chatStore = create<AgentChatStore>((set, get) => {
function connect(sessionId: string, afterSeq: number): void {
if (typeof WebSocket === "undefined") return;
closeSocket();
@@ -735,11 +748,15 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
newChat: (opts) => {
closeSocket();
+ newChatLane = emptySendLane();
+ resetBlankComposerDraftTarget(chatStore);
+ discardHeldFiles(chatStore, "");
++catalogRequest;
set({
...(surface === "jarvis" ? { draft: readDraft(DRAFT_KEY) } : {}),
activeSessionId: null,
activeSession: null,
+ busy: false,
timeline: EMPTY_TIMELINE,
socketState: "idle",
lastError: null,
@@ -762,7 +779,7 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
activeSessionId: sessionId,
activeSession: prev.sessions.find((s) => s.session_id === sessionId) ?? null,
timeline: timelineCache.get(sessionId) ?? EMPTY_TIMELINE,
- busy: false,
+ busy: (sendLanes.get(sessionId)?.pending ?? 0) > 0,
lastError: null,
});
bindVoice(sessionId);
@@ -784,17 +801,29 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
void get().loadSessions();
},
- send: async (text, attachments = [], toolChoices = []) => {
+ send: async (text, attachments = [], toolChoices = [], onSessionCreated) => {
const content = text.trim();
// A message may be files alone — dropping a screenshot and pressing
// Enter is a complete gesture — but never nothing at all.
if (!content && attachments.length === 0) return "failed";
const st = get();
+ const lane = st.activeSessionId
+ ? sendLanes.get(st.activeSessionId) ?? { ...emptySendLane(), sessionId: st.activeSessionId }
+ : newChatLane;
+ lane.draftTarget ??= captureComposerDraftTarget(chatStore, st.activeSessionId);
+ if (lane.sessionId) sendLanes.set(lane.sessionId, lane);
+ const isVisible = () => lane.sessionId
+ ? get().activeSessionId === lane.sessionId
+ : get().activeSessionId === null && newChatLane === lane;
+ const previous = lane.tail;
+ lane.pending += 1;
set({ busy: true, lastError: null });
- let sid = st.activeSessionId;
+ // Serialize transport admission, not agent execution. A second send
+ // shares an unfinished session creation and reaches the server in order.
+ const submission = previous.catch(() => undefined).then(async () => {
try {
await selectionWrite;
- if (!sid) {
+ if (!lane.sessionId) {
const d = st.draft;
const session = await createAgentChatSession({
provider: d.provider,
@@ -805,35 +834,47 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
permission_mode: d.permissionMode,
surface,
});
- sid = session.session_id;
- set({
- activeSessionId: sid,
- activeSession: session,
- timeline: EMPTY_TIMELINE,
- sessions: [session, ...get().sessions],
- });
- bindVoice(sid);
- connect(sid, 0);
+ const visible = isVisible();
+ if (bindComposerDraftTarget(chatStore, lane.draftTarget!, session.session_id)) {
+ moveHeldFiles(chatStore, "", session.session_id);
+ }
+ lane.sessionId = session.session_id;
+ sendLanes.set(session.session_id, lane);
+ set({ sessions: [session, ...get().sessions] });
+ onSessionCreated?.(session);
+ if (visible) {
+ set({ activeSessionId: session.session_id, activeSession: session, timeline: EMPTY_TIMELINE });
+ bindVoice(session.session_id);
+ connect(session.session_id, 0);
+ }
}
+ const sid = lane.sessionId;
await sendAgentChatMessage(sid, content, attachments, toolChoices);
if (surface === "jarvis") writeDraft(DRAFT_KEY, st.draft);
if (get().activeSessionId === sid) void get().loadSessions();
- return get().activeSessionId === sid ? "sent" : "stale";
+ return isVisible() ? "sent" as const : "stale" as const;
} catch (err) {
- if (get().activeSessionId === sid) set({ lastError: errorText(err) });
- return get().activeSessionId === sid ? "failed" : "stale";
+ if (isVisible()) set({ lastError: errorText(err) });
+ return "failed" as const;
} finally {
// A switch away from this session already cleared `busy`; do not
// unlock (or error) the chat that is on screen now.
- if (get().activeSessionId === sid) set({ busy: false });
+ lane.pending -= 1;
+ if (isVisible()) set({ busy: lane.pending > 0 });
+ if (lane.pending === 0 && lane.sessionId && sendLanes.get(lane.sessionId) === lane) {
+ sendLanes.delete(lane.sessionId);
+ }
}
+ });
+ lane.tail = submission;
+ return submission;
},
- cancel: async () => {
+ cancel: async (via) => {
const sid = get().activeSessionId;
if (!sid) return;
try {
- await cancelAgentChatTurn(sid);
+ await cancelAgentChatTurn(sid, via);
} catch (err) {
set({ lastError: errorText(err) });
}
@@ -842,11 +883,18 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
decide: async (approvalId, decision) => {
const sid = get().activeSessionId;
if (!sid) return;
+ let outcome: string = decision;
try {
await resolveAgentChatApproval(sid, approvalId, decision);
} catch (err) {
- set({ lastError: errorText(err) });
+ // 410: the card outlived what it asked for (an app restart) and is
+ // closed now; 404: the backend never knew it. Either way nothing
+ // waits on it, and the next card the agent opened takes its place.
+ const gone = err instanceof AgentChatApiError && (err.status === 404 || err.status === 410);
+ if (!gone) throw err;
+ outcome = "expired";
}
+ if (get().activeSessionId === sid) set({ timeline: settleApproval(get().timeline, approvalId, outcome) });
},
answerQuestion: async (questionId, index, answer) => {
@@ -891,6 +939,7 @@ export function createAgentChatStore(surface: AgentChatSurface, draftNamespace =
},
};
});
+ return chatStore;
}
/** The front page's store — every existing call site reads this one. */
diff --git a/jarvis/ui/web/frontend/src/views/AppshotEditor.tsx b/jarvis/ui/web/frontend/src/views/AppshotEditor.tsx
index bb95ed9023..ff6e543084 100644
--- a/jarvis/ui/web/frontend/src/views/AppshotEditor.tsx
+++ b/jarvis/ui/web/frontend/src/views/AppshotEditor.tsx
@@ -1786,7 +1786,9 @@ function ImportPanel({
useEffect(() => {
let alive = true;
fetchAppshotLibrary()
- .then((body) => alive && setItems(Array.isArray(body.items) ? body.items : []))
+ .then((body) => alive && setItems(
+ Array.isArray(body.items) ? body.items.filter((item) => item.mime.startsWith("image/")) : [],
+ ))
.catch(() => {
// The gallery is optional here: a file still works without it.
if (alive) {
diff --git a/jarvis/ui/web/frontend/src/views/AppshotEditorWindow.tsx b/jarvis/ui/web/frontend/src/views/AppshotEditorWindow.tsx
index 809e3e3dfe..4e3b85a1c4 100644
--- a/jarvis/ui/web/frontend/src/views/AppshotEditorWindow.tsx
+++ b/jarvis/ui/web/frontend/src/views/AppshotEditorWindow.tsx
@@ -3,10 +3,13 @@ import { useCallback, useEffect, useRef, useState } from "react";
import { ToastLayer } from "@/components/ToastLayer";
import { loadLocaleChunk } from "@/i18n";
import { AppshotEditor, type EditorExit } from "@/views/AppshotEditor";
+import { AppshotVideoEditor } from "@/views/AppshotVideoEditor";
/**
* The appshot editor as its own desktop window (`?view=appshot-editor&solo=1
- * &appshot=<id>`), opened by a click on the corner card.
+ * &appshot=<id>`), opened by a click on the corner card. Pointed at a screen
+ * recording (`&recording=<id>`, or `window.__jarvisOpenRecording`) the same
+ * window is the video editor instead.
* Nothing of the app is loaded around it.
*
* When it closes the appshot goes back into the screen corner
@@ -26,6 +29,12 @@ export function appshotIdFromUrl(search: string): string {
return /^[0-9a-f]{8,64}$/.test(id) ? id : "";
}
+/** The screen recording this window plays and trims, from its URL; empty when none. */
+export function recordingIdFromUrl(search: string): string {
+ const id = new URLSearchParams(search).get("recording") ?? "";
+ return /^[0-9a-f]{32}$/.test(id) ? id : "";
+}
+
/** Is this document the editor window rather than the app? */
export function isAppshotEditorWindow(search: string): boolean {
return new URLSearchParams(search).get("view") === APPSHOT_EDITOR_VIEW;
@@ -38,10 +47,11 @@ export function isAppshotEditorWindow(search: string): boolean {
* reload of that address (the blank-window watchdog, a rebuilt bundle) would
* otherwise come back as the empty gray shell. The address is the backup.
*/
-function rememberEditorTarget(id: string): void {
+function rememberEditorTarget(kind: "appshot" | "recording", id: string): void {
const url = new URL(window.location.href);
url.searchParams.delete("appshot");
- if (id) url.searchParams.set("appshot", id);
+ url.searchParams.delete("recording");
+ if (id) url.searchParams.set(kind, id);
const next = `${url.pathname}${url.search}${url.hash}`;
const current = `${window.location.pathname}${window.location.search}${window.location.hash}`;
if (next !== current) window.history.replaceState(null, "", next);
@@ -71,6 +81,8 @@ declare global {
interface Window {
/** The desktop shell points the warm (hidden) editor window at an appshot. */
__jarvisOpenAppshot?: (id: string) => boolean;
+ /** The same window, pointed at a screen recording: it plays and trims it. */
+ __jarvisOpenRecording?: (id: string) => boolean;
}
}
@@ -79,6 +91,9 @@ export function AppshotEditorWindow({ deps = browserDeps }: { deps?: EditorWindo
// appshot and be pointed at one later (window.__jarvisOpenAppshot), so the
// editor opens at once instead of booting a fresh window each time.
const [appshotId, setAppshotId] = useState(() => appshotIdFromUrl(window.location.search));
+ const [recordingId, setRecordingId] = useState(() =>
+ appshotIdFromUrl(window.location.search) ? "" : recordingIdFromUrl(window.location.search),
+ );
const [session, setSession] = useState(0);
const closing = useRef(false);
@@ -89,13 +104,25 @@ export function AppshotEditorWindow({ deps = browserDeps }: { deps?: EditorWindo
const valid = appshotIdFromUrl(`?appshot=${encodeURIComponent(id)}`);
if (!valid) return false;
closing.current = false;
- rememberEditorTarget(valid);
+ rememberEditorTarget("appshot", valid);
+ setRecordingId("");
setAppshotId(valid);
setSession((n) => n + 1);
return true;
};
+ window.__jarvisOpenRecording = (id: string) => {
+ const valid = recordingIdFromUrl(`?recording=${encodeURIComponent(id)}`);
+ if (!valid) return false;
+ closing.current = false;
+ rememberEditorTarget("recording", valid);
+ setAppshotId("");
+ setRecordingId(valid);
+ setSession((n) => n + 1);
+ return true;
+ };
return () => {
delete window.__jarvisOpenAppshot;
+ delete window.__jarvisOpenRecording;
};
}, []);
@@ -110,16 +137,29 @@ export function AppshotEditorWindow({ deps = browserDeps }: { deps?: EditorWindo
.catch(() => undefined)
.finally(() => {
setAppshotId("");
- rememberEditorTarget("");
+ rememberEditorTarget("appshot", "");
void deps.closeWindow().catch(() => undefined);
});
}, [appshotId, deps]);
+ // A video has no corner card to return to: closing just hides the window
+ // and forgets the video, so a hidden window never keeps playing or flashes it.
+ const closeVideo = useCallback(() => {
+ if (closing.current) return;
+ closing.current = true;
+ setRecordingId("");
+ rememberEditorTarget("recording", "");
+ void deps.closeWindow().catch(() => undefined);
+ }, [deps]);
+
return (
<div className="fixed inset-0 overflow-hidden bg-popover" data-testid="appshot-editor-window">
{appshotId && (
<AppshotEditor key={session} appshotId={appshotId} onClose={close} variant="window" />
)}
+ {!appshotId && recordingId && (
+ <AppshotVideoEditor key={session} recordingId={recordingId} onClose={closeVideo} />
+ )}
<ToastLayer />
</div>
);
diff --git a/jarvis/ui/web/frontend/src/views/AppshotLibrary.test.tsx b/jarvis/ui/web/frontend/src/views/AppshotLibrary.test.tsx
index 0470ee6bd6..e497f67f3c 100644
--- a/jarvis/ui/web/frontend/src/views/AppshotLibrary.test.tsx
+++ b/jarvis/ui/web/frontend/src/views/AppshotLibrary.test.tsx
@@ -30,6 +30,11 @@ const EDITED: AppshotLibraryItem = {
has_edit: false,
};
const OTHER: AppshotLibraryItem = { ...ORIGINAL, id: "b2", has_edit: false, app_name: "Browser" };
+const RECORDING: AppshotLibraryItem = {
+ ...ORIGINAL, id: `recording_${"a".repeat(32)}`, mime: "video/mp4", has_edit: false,
+ path: `C:\\Jarvis\\appshot-recordings\\${"a".repeat(32)}.mp4`,
+ app_name: "", label: "Screen recording", duration_s: 61,
+};
function json(body: unknown, status = 200): Response {
return { ok: status < 400, status, json: async () => body } as Response;
@@ -73,6 +78,10 @@ describe("AppshotLibrary", () => {
return json({ ok: true });
}
if (url === "/api/appshot/library/b2/open") return json({ id: "b2", window: false });
+ if (url === `/api/appshot/library/${RECORDING.id}?variant=original` && init?.method === "DELETE") {
+ items = items.filter((item) => item !== RECORDING);
+ return json({ ok: true });
+ }
return json({}, 404);
});
vi.stubGlobal("fetch", fetchMock);
@@ -171,4 +180,56 @@ describe("AppshotLibrary", () => {
render(<AppshotLibrary enabled={false} refreshKey="" />);
await screen.findByText(/History is off/);
});
+
+ it("filters recordings, shows duration and plays a video without sending it to the image editor", async () => {
+ items = [RECORDING, ...items];
+ render(<AppshotLibrary enabled refreshKey="" />);
+ await waitFor(() => expect(screen.getAllByTestId("appshot-library-tile")).toHaveLength(4));
+ expect(screen.getByText("1:01")).toBeDefined();
+ fireEvent.click(screen.getByTestId("appshot-library-filter-recordings"));
+ const tiles = screen.getAllByTestId("appshot-library-tile");
+ expect(tiles).toHaveLength(1);
+ expect(tiles[0].getAttribute("data-media")).toBe("video");
+ fireEvent.click(tiles[0].querySelector("button")!);
+ const player = await screen.findByTestId("appshot-library-player");
+ expect(player.getAttribute("src")).toContain(`/library/${RECORDING.id}/image`);
+ expect(player.hasAttribute("controls")).toBe(true);
+ expect(screen.getByRole("link", { name: "Download video" }).getAttribute("download")).toMatch(/\.mp4$/);
+ expect(useAppshotEditor.getState().openId).toBeNull();
+ expect(fetchMock.mock.calls.some(([url]) => url.endsWith("/open"))).toBe(false);
+ fireEvent.click(screen.getByRole("button", { name: "Close" }));
+ await waitFor(() => expect(screen.queryByTestId("appshot-library-player")).toBeNull());
+ });
+
+ it("shows only recordings, folded to two rows, without filters or delete-all", async () => {
+ const more = Array.from({ length: 14 }, (_, n) => ({
+ ...RECORDING, id: `recording_${String(n).padStart(32, "0")}`,
+ }));
+ items = [...more, ...items];
+ render(<AppshotLibrary enabled={undefined} refreshKey="" recordingsOnly />);
+ await waitFor(() => expect(screen.getAllByTestId("appshot-library-tile")).toHaveLength(12));
+ expect(screen.getByTestId("appshot-recordings")).toBeDefined();
+ for (const tile of screen.getAllByTestId("appshot-library-tile")) {
+ expect(tile.getAttribute("data-media")).toBe("video");
+ }
+ expect(screen.queryByTestId("appshot-library-filter-all")).toBeNull();
+ expect(screen.queryByTestId("appshot-library-clear")).toBeNull();
+ fireEvent.click(screen.getByTestId("appshot-library-more"));
+ expect(screen.getAllByTestId("appshot-library-tile")).toHaveLength(14);
+ });
+
+ it("drags a recording as an MP4 and deletes only that video", async () => {
+ items = [RECORDING, ORIGINAL];
+ render(<AppshotLibrary enabled refreshKey="" />);
+ const [tile] = await screen.findAllByTestId("appshot-library-tile");
+ const transfer = new FakeDataTransfer();
+ fireEvent.dragStart(tile, { dataTransfer: transfer });
+ expect(transfer.getData("DownloadURL")).toMatch(/^video\/mp4:.*\.mp4:http/);
+ expect(transfer.getData(WORKSPACE_PATH_TYPE)).toMatch(/^[a-f0-9]{64}$/);
+ expect(extractPaneDrop(transfer as unknown as DataTransfer).paths).toEqual([RECORDING.path]);
+ fireEvent.dragEnd(tile);
+ fireEvent.click(screen.getAllByTestId("appshot-library-delete")[0]);
+ await waitFor(() => expect(screen.getAllByTestId("appshot-library-tile")).toHaveLength(1));
+ expect(screen.getAllByTestId("appshot-library-tile")[0].getAttribute("data-media")).toBe("image");
+ });
});
diff --git a/jarvis/ui/web/frontend/src/views/AppshotLibrary.tsx b/jarvis/ui/web/frontend/src/views/AppshotLibrary.tsx
index 3219117867..6a8aa2741b 100644
--- a/jarvis/ui/web/frontend/src/views/AppshotLibrary.tsx
+++ b/jarvis/ui/web/frontend/src/views/AppshotLibrary.tsx
@@ -1,7 +1,8 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
-import { GripVertical, Loader2, PenLine, Trash2 } from "lucide-react";
+import { GripVertical, Loader2, PenLine, Play, Trash2, Video } from "lucide-react";
import { setWorkspaceEntryDrag } from "@/components/agentic/explorerDrag";
+import { AppshotRecordingPlayer } from "@/components/appshot/AppshotRecordingPlayer";
import { Button } from "@/components/ui/button";
import { QuickTooltip } from "@/components/ui/tooltip";
import { useLocaleChunk, useT } from "@/i18n";
@@ -35,7 +36,7 @@ import { useEventStore } from "@/store/events";
* LIFT_DIALOG_CSS) and comes back the moment the drag ends.
*/
-type Filter = "all" | "edited";
+type Filter = "all" | "edited" | "recordings";
/** Rows shown while the gallery is folded; "Show more" opens the rest. */
const ROWS = 2;
@@ -63,6 +64,12 @@ function useGridColumns(grid: HTMLUListElement | null): number {
}
const tileKey = (item: AppshotLibraryItem) => `${item.id}:${item.variant}`;
+const isRecording = (item: AppshotLibraryItem) => item.mime.startsWith("video/");
+
+function duration(seconds: number): string {
+ const whole = Math.max(0, Math.floor(seconds));
+ return `${Math.floor(whole / 60)}:${String(whole % 60).padStart(2, "0")}`;
+}
/**
* While a tile is dragged: the Settings dialog and its dim fade out and let
@@ -109,7 +116,8 @@ function Tile({
}) {
const t = useT();
const imageRef = useRef<HTMLImageElement | null>(null);
- const where = item.app_name || item.label || t("appshots.preview_front_window");
+ const video = isRecording(item);
+ const where = video ? t("appshots.recording_title") : item.app_name || item.label || t("appshots.preview_front_window");
const edited = item.variant === "edited";
const nativeDrag = canNativeDrag();
@@ -138,14 +146,15 @@ function Tile({
onDragEnd={() => liftDialog(false)}
data-testid="appshot-library-tile"
data-variant={item.variant}
+ data-media={video ? "video" : "image"}
className="group relative flex cursor-grab flex-col overflow-hidden rounded-lg border border-border bg-background active:cursor-grabbing"
>
<button
type="button"
onClick={onOpen}
disabled={busy}
- aria-label={`${t("appshot_editor.library_edit")}: ${where}`}
- className="flex aspect-[16/10] w-full items-center justify-center overflow-hidden bg-secondary/60 p-2 focus:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-border-strong"
+ aria-label={`${t(video ? "appshot_editor.library_play" : "appshot_editor.library_edit")}: ${where}`}
+ className="relative flex aspect-[16/10] w-full items-center justify-center overflow-hidden bg-secondary/60 p-2 focus:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-border-strong"
>
<img
ref={imageRef}
@@ -155,7 +164,18 @@ function Tile({
draggable={false}
className="max-h-full max-w-full rounded-sm object-contain shadow-sm ring-1 ring-border transition-opacity group-hover:opacity-90"
/>
+ {video && (
+ <span className="absolute rounded-full border border-border bg-card/90 p-3 text-foreground shadow-sm">
+ <Play className="h-5 w-5" aria-hidden />
+ </span>
+ )}
</button>
+ {video && (
+ <span className="pointer-events-none absolute bottom-[4.25rem] right-3 inline-flex items-center gap-1 rounded-md bg-card/95 px-2 py-0.5 text-xs text-foreground">
+ <Video className="h-3 w-3" aria-hidden />
+ {item.duration_s ? duration(item.duration_s) : t("appshot_editor.library_video")}
+ </span>
+ )}
{edited && (
<span className="pointer-events-none absolute left-2 top-2 inline-flex items-center gap-1 rounded-full bg-accent px-2 py-0.5 text-xs font-medium text-accent-foreground shadow-sm">
<PenLine className="h-3 w-3" aria-hidden />
@@ -184,7 +204,7 @@ function Tile({
)}
<QuickTooltip
content={t(
- edited
+ video ? "appshot_editor.library_delete_recording_hint" : edited
? "appshot_editor.library_delete_edit_hint"
: "appshot_editor.library_delete_original_hint",
)}
@@ -205,7 +225,7 @@ function Tile({
<div className="min-w-0 px-3 py-2">
<p className="truncate text-sm font-medium text-foreground">{where}</p>
<p className="truncate text-xs text-muted-foreground">
- {[when(item), `${item.width} × ${item.height}`].join(" · ")}
+ {[when(item), item.width > 0 && item.height > 0 ? `${item.width} × ${item.height}` : ""].filter(Boolean).join(" · ")}
</p>
</div>
</li>
@@ -215,11 +235,18 @@ function Tile({
export function AppshotLibrary({
enabled,
refreshKey,
+ recordingsOnly = false,
}: {
/** `[appshot].library`; `undefined` while the settings load or on an older backend. */
enabled: boolean | undefined;
/** Changes whenever a new appshot was taken or an edit replaced one. */
refreshKey: string;
+ /**
+ * Only the screen recordings, as tiles inside the recording panel: same
+ * posters, player and two-row fold as the gallery, without its filters and
+ * without "Delete all" (which would take every appshot with it).
+ */
+ recordingsOnly?: boolean;
}) {
const t = useT();
// The gallery's strings live in the editor's locale chunk.
@@ -228,7 +255,7 @@ export function AppshotLibrary({
const openEditor = useAppshotEditor((s) => s.open);
const [items, setItems] = useState<AppshotLibraryItem[] | null>(null);
const [error, setError] = useState("");
- const [filter, setFilter] = useState<Filter>("all");
+ const [filter, setFilter] = useState<Filter>(recordingsOnly ? "recordings" : "all");
// Folded: exactly ROWS full rows. Unfolded: `limit` tiles, PAGE more per click.
const [expanded, setExpanded] = useState(false);
const [limit, setLimit] = useState(PAGE);
@@ -237,6 +264,7 @@ export function AppshotLibrary({
const [busyKey, setBusyKey] = useState("");
const [confirmClear, setConfirmClear] = useState(false);
const [clearing, setClearing] = useState(false);
+ const [playing, setPlaying] = useState<AppshotLibraryItem | null>(null);
const confirmTimer = useRef<number | null>(null);
const load = useCallback(async () => {
@@ -289,13 +317,20 @@ export function AppshotLibrary({
() => (items ?? []).filter((item) => item.variant === "edited").length,
[items],
);
+ const recordingCount = useMemo(() => (items ?? []).filter(isRecording).length, [items]);
const shown = useMemo(
- () => (items ?? []).filter((item) => filter === "all" || item.variant === "edited"),
+ () => (items ?? []).filter((item) => filter === "all" || (
+ filter === "recordings" ? isRecording(item) : item.variant === "edited"
+ )),
[items, filter],
);
const open = useCallback(
async (item: AppshotLibraryItem) => {
+ if (isRecording(item)) {
+ setPlaying(item);
+ return;
+ }
setBusyKey(tileKey(item));
try {
const result = await openAppshotLibraryItem(item);
@@ -354,73 +389,91 @@ export function AppshotLibrary({
const folded = columns * ROWS;
const visible = expanded ? Math.max(limit, folded) : folded;
+ const titleId = recordingsOnly ? "appshot-recordings-title" : "appshot-library-title";
+
return (
<section
- data-testid="appshot-library"
- className="mt-5 rounded-xl border border-border bg-card p-5"
- aria-labelledby="appshot-library-title"
+ data-testid={recordingsOnly ? "appshot-recordings" : "appshot-library"}
+ className={
+ recordingsOnly
+ ? "mt-4 border-t border-border pt-4"
+ : "mt-5 rounded-xl border border-border bg-card p-5"
+ }
+ aria-labelledby={titleId}
>
<style>{LIFT_DIALOG_CSS}</style>
- <div className="flex flex-wrap items-start justify-between gap-x-6 gap-y-3">
- <div className="min-w-[min(100%,16rem)] flex-1">
- <p id="appshot-library-title" className="text-base font-medium text-foreground">
- {t("appshot_editor.library_title")}
- {total > 0 && (
- <span className="ml-2 text-sm font-normal text-muted-foreground">
- {t("appshot_editor.library_count").replace("{0}", String(total))}
- </span>
- )}
- </p>
- <p className="mt-0.5 text-sm text-muted-foreground">
- {enabled === false
- ? t("appshot_editor.library_off")
- : t("appshot_editor.library_subtitle")}
- </p>
- </div>
- {total > 0 && (
- <div className="ml-auto flex shrink-0 items-center gap-2">
- <div role="tablist" className="flex items-center rounded-lg bg-secondary p-0.5">
- {(["all", "edited"] as const).map((value) => (
- <button
- key={value}
- type="button"
- role="tab"
- aria-selected={filter === value}
- data-testid={`appshot-library-filter-${value}`}
- onClick={() => {
- setFilter(value);
- setExpanded(false);
- setLimit(PAGE);
- }}
- className={cn(
- "h-7 rounded-md px-3 text-sm transition-colors focus:outline-none focus-visible:ring-2 focus-visible:ring-border-strong",
- filter === value
- ? "bg-card text-foreground shadow-sm"
- : "text-muted-foreground hover:text-foreground",
- )}
- >
- {value === "all"
- ? t("appshot_editor.library_filter_all")
- : `${t("appshot_editor.library_filter_edited")} ${editedCount}`}
- </button>
- ))}
- </div>
- <Button
- type="button"
- variant={confirmClear ? "destructive" : "ghost"}
- size="sm"
- disabled={clearing}
- onClick={() => void clearAll()}
- data-testid="appshot-library-clear"
- >
- {clearing ? <Loader2 className="animate-spin" aria-hidden /> : <Trash2 aria-hidden />}
- {confirmClear
- ? t("appshot_editor.library_delete_all_confirm")
- : t("appshot_editor.library_delete_all")}
- </Button>
+ {recordingsOnly ? (
+ <p id={titleId} className="text-sm font-medium text-foreground">
+ {t("appshot_editor.library_filter_recordings")}
+ {shown.length > 0 && (
+ <span className="ml-2 font-normal text-muted-foreground">
+ {t("appshot_editor.library_count").replace("{0}", String(shown.length))}
+ </span>
+ )}
+ </p>
+ ) : (
+ <div className="flex flex-wrap items-start justify-between gap-x-6 gap-y-3">
+ <div className="min-w-[min(100%,16rem)] flex-1">
+ <p id={titleId} className="text-base font-medium text-foreground">
+ {t("appshot_editor.library_title")}
+ {total > 0 && (
+ <span className="ml-2 text-sm font-normal text-muted-foreground">
+ {t("appshot_editor.library_count").replace("{0}", String(total))}
+ </span>
+ )}
+ </p>
+ <p className="mt-0.5 text-sm text-muted-foreground">
+ {enabled === false
+ ? t("appshot_editor.library_off")
+ : t("appshot_editor.library_subtitle")}
+ </p>
</div>
- )}
- </div>
+ {total > 0 && (
+ <div className="ml-auto flex max-w-full flex-wrap items-center gap-2">
+ <div role="tablist" className="flex items-center rounded-lg bg-secondary p-0.5">
+ {(["all", "edited", "recordings"] as const).map((value) => (
+ <button
+ key={value}
+ type="button"
+ role="tab"
+ aria-selected={filter === value}
+ data-testid={`appshot-library-filter-${value}`}
+ onClick={() => {
+ setFilter(value);
+ setExpanded(false);
+ setLimit(PAGE);
+ }}
+ className={cn(
+ "h-7 rounded-md px-3 text-sm transition-colors focus:outline-none focus-visible:ring-2 focus-visible:ring-border-strong",
+ filter === value
+ ? "bg-card text-foreground shadow-sm"
+ : "text-muted-foreground hover:text-foreground",
+ )}
+ >
+ {value === "all"
+ ? t("appshot_editor.library_filter_all")
+ : value === "edited" ? `${t("appshot_editor.library_filter_edited")} ${editedCount}`
+ : `${t("appshot_editor.library_filter_recordings")} ${recordingCount}`}
+ </button>
+ ))}
+ </div>
+ <Button
+ type="button"
+ variant={confirmClear ? "destructive" : "ghost"}
+ size="sm"
+ disabled={clearing}
+ onClick={() => void clearAll()}
+ data-testid="appshot-library-clear"
+ >
+ {clearing ? <Loader2 className="animate-spin" aria-hidden /> : <Trash2 aria-hidden />}
+ {confirmClear
+ ? t("appshot_editor.library_delete_all_confirm")
+ : t("appshot_editor.library_delete_all")}
+ </Button>
+ </div>
+ )}
+ </div>
+ )}
{error && (
<p className="mt-3 text-sm text-destructive" role="alert">
@@ -435,7 +488,7 @@ export function AppshotLibrary({
) : empty ? (
!error && (
<p className="mt-4 rounded-lg border border-dashed border-border px-4 py-8 text-center text-sm text-muted-foreground">
- {filter === "edited"
+ {filter === "recordings" ? t("appshot_editor.library_empty_recordings") : filter === "edited"
? t("appshot_editor.library_empty_edited")
: t("appshot_editor.library_empty")}
</p>
@@ -492,6 +545,7 @@ export function AppshotLibrary({
</div>
</>
)}
+ <AppshotRecordingPlayer item={playing} onClose={() => setPlaying(null)} />
</section>
);
}
diff --git a/jarvis/ui/web/frontend/src/views/AppshotVideoEditor.test.tsx b/jarvis/ui/web/frontend/src/views/AppshotVideoEditor.test.tsx
new file mode 100644
index 0000000000..5fea20bf98
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/AppshotVideoEditor.test.tsx
@@ -0,0 +1,131 @@
+import { act, cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+import { useEventStore } from "@/store/events";
+import { AppshotEditorWindow, recordingIdFromUrl, type EditorWindowDeps } from "@/views/AppshotEditorWindow";
+import { AppshotVideoEditor, formatVideoTime, moveTrimEdge } from "@/views/AppshotVideoEditor";
+
+/**
+ * The video editor: a played recording opens in the editor window, plays
+ * and trims there, and Save writes the chosen part at the chosen speed.
+ */
+
+const RECORDING = "a".repeat(32);
+
+function json(body: unknown, status = 200): Response {
+ return { ok: status < 400, status, json: async () => body } as Response;
+}
+
+/** The desktop shell's capabilities decide between a native save and a browser download. */
+async function capabilitiesLoaded() {
+ // The capability check waits out the boot stagger (4 s) first; then it is cached.
+ await waitFor(() => expect(screen.getByTestId("appshot-video-editor").dataset.native).toBe("true"), {
+ timeout: 6000,
+ });
+}
+
+function loadVideo(duration = 20) {
+ const video = screen.getByTestId("appshot-video") as HTMLVideoElement;
+ Object.defineProperty(video, "duration", { configurable: true, value: duration });
+ Object.defineProperty(video, "videoWidth", { configurable: true, value: 1280 });
+ Object.defineProperty(video, "videoHeight", { configurable: true, value: 720 });
+ fireEvent(video, new Event("loadedmetadata"));
+ return video;
+}
+
+describe("the video editor", () => {
+ beforeEach(() => {
+ useEventStore.setState({ events: [], toasts: [], assistantName: "George" });
+ vi.stubGlobal("fetch", vi.fn(async () => json({ native_file_actions: true, platform: "win32" })));
+ });
+
+ afterEach(() => {
+ cleanup();
+ vi.unstubAllGlobals();
+ window.history.replaceState(null, "", "/");
+ });
+
+ it("formats times and keeps trims inside the video", () => {
+ expect(formatVideoTime(0)).toBe("0:00.0");
+ expect(formatVideoTime(75.36)).toBe("1:15.3");
+ expect(formatVideoTime(3725, false)).toBe("1:02:05");
+ const trim = { start: 0, end: 10 };
+ expect(moveTrimEdge(trim, "start", 9.9, 10)).toEqual({ start: 9.5, end: 10 });
+ expect(moveTrimEdge(trim, "end", -3, 10)).toEqual({ start: 0, end: 0.5 });
+ expect(moveTrimEdge(trim, "end", 42, 10)).toEqual({ start: 0, end: 10 });
+ });
+
+ it("only accepts a recording id from the window URL", () => {
+ expect(recordingIdFromUrl(`?recording=${RECORDING}`)).toBe(RECORDING);
+ expect(recordingIdFromUrl("?recording=../../etc")).toBe("");
+ });
+
+ it("opens on a recording named by the shell and closes without a corner card", async () => {
+ window.history.replaceState(null, "", "/?view=appshot-editor&solo=1");
+ const calls: string[] = [];
+ const deps: EditorWindowDeps = {
+ returnCard: async () => void calls.push("card"),
+ closeWindow: async () => void calls.push("close"),
+ };
+ render(<AppshotEditorWindow deps={deps} />);
+ act(() => {
+ expect(window.__jarvisOpenRecording?.(RECORDING)).toBe(true);
+ });
+ expect(await screen.findByTestId("appshot-video-editor")).toBeTruthy();
+ fireEvent.click(screen.getByTestId("appshot-video-close"));
+ await waitFor(() => expect(calls).toEqual(["close"]));
+ expect(screen.queryByTestId("appshot-video-editor")).toBeNull();
+ });
+
+ it("saves the trimmed part at the chosen speed as a new recording", { timeout: 10_000 }, async () => {
+ const requests: Array<[string, string | undefined]> = [];
+ vi.stubGlobal(
+ "fetch",
+ vi.fn(async (url: string, init?: RequestInit) => {
+ requests.push([url, init?.body as string | undefined]);
+ if (url.endsWith("/export")) return json({ id: "b".repeat(32) });
+ if (url.endsWith("/save")) return json({ path: "C:/Downloads/clip.mp4", filename: "clip.mp4" });
+ return json({ native_file_actions: true, platform: "win32" });
+ }),
+ );
+ render(<AppshotVideoEditor recordingId={RECORDING} onClose={() => undefined} />);
+ const video = loadVideo(20);
+ video.currentTime = 4;
+ fireEvent.keyDown(window, { key: "i" });
+ video.currentTime = 12;
+ fireEvent.keyDown(window, { key: "o" });
+ fireEvent.click(await screen.findByRole("button", { name: /^2×$/ }));
+ expect(video.playbackRate).toBe(2);
+ await waitFor(() => expect(screen.getByTestId("appshot-video-clip-length").textContent).toContain("0:04"));
+
+ await capabilitiesLoaded();
+ fireEvent.keyDown(window, { key: "s", ctrlKey: true });
+ await waitFor(() => expect(requests.some(([url]) => url.endsWith(`/${"b".repeat(32)}/save`))).toBe(true));
+ const exported = requests.find(([url]) => url.endsWith(`/${RECORDING}/export`));
+ expect(JSON.parse(exported?.[1] ?? "{}")).toEqual({ start_s: 4, end_s: 12, speed: 2 });
+ });
+
+ it("saves an unchanged video without writing a copy first", { timeout: 10_000 }, async () => {
+ const urls: string[] = [];
+ vi.stubGlobal(
+ "fetch",
+ vi.fn(async (url: string) => {
+ urls.push(url);
+ if (url.endsWith("/save")) return json({ path: "C:/Downloads/v.mp4", filename: "v.mp4" });
+ return json({ native_file_actions: true, platform: "win32" });
+ }),
+ );
+ render(<AppshotVideoEditor recordingId={RECORDING} onClose={() => undefined} />);
+ loadVideo(8);
+ await capabilitiesLoaded();
+ fireEvent.click(screen.getByTestId("appshot-video-save"));
+ await waitFor(() => expect(urls.some((url) => url.endsWith(`/${RECORDING}/save`))).toBe(true));
+ expect(urls.some((url) => url.endsWith("/export"))).toBe(false);
+ });
+
+ it("says so when the video is gone", async () => {
+ render(<AppshotVideoEditor recordingId={RECORDING} onClose={() => undefined} />);
+ fireEvent(screen.getByTestId("appshot-video"), new Event("error"));
+ expect(await screen.findByTestId("appshot-video-failed")).toBeTruthy();
+ });
+});
diff --git a/jarvis/ui/web/frontend/src/views/AppshotVideoEditor.tsx b/jarvis/ui/web/frontend/src/views/AppshotVideoEditor.tsx
new file mode 100644
index 0000000000..7e3414054c
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/AppshotVideoEditor.tsx
@@ -0,0 +1,580 @@
+import { useCallback, useEffect, useMemo, useRef, useState, type PointerEvent as ReactPointerEvent } from "react";
+import { Film, Loader2, Pause, Play, RotateCcw, SkipBack, SkipForward, X } from "lucide-react";
+
+import { Button } from "@/components/ui/button";
+import { QuickTooltip } from "@/components/ui/tooltip";
+import { useCapabilities } from "@/hooks/useCapabilities";
+import { fill, useLocaleChunk, useT } from "@/i18n";
+import {
+ RECORDING_SPEEDS,
+ appshotRecordingUrl,
+ exportAppshotRecording,
+ saveAppshotRecording,
+} from "@/lib/appshotApi";
+import { cn } from "@/lib/utils";
+import { useEventStore } from "@/store/events";
+
+/**
+ * The video editor: plays a finished screen recording and trims it.
+ *
+ * It opens in the appshot editor's window when a recording card is played.
+ * Playback, scrubbing and trimming happen on the original file in the page;
+ * only Save writes something — the chosen part at the chosen speed becomes a
+ * new recording on the backend (`/recording/{id}/export`), the original stays.
+ */
+
+/** Shortest clip the trim handles allow, in seconds. */
+const MIN_CLIP_S = 0.5;
+/** Frames in the timeline's film strip at most. */
+const MAX_THUMBS = 14;
+const FRAME_S = 1 / 30;
+
+export interface Trim {
+ start: number;
+ end: number;
+}
+
+/** `m:ss.t` under an hour, `h:mm:ss` beyond. */
+export function formatVideoTime(seconds: number, precise = true): string {
+ const s = Math.max(0, seconds);
+ const hours = Math.floor(s / 3600);
+ const minutes = Math.floor((s % 3600) / 60);
+ const whole = Math.floor(s % 60);
+ if (hours) return `${hours}:${String(minutes).padStart(2, "0")}:${String(whole).padStart(2, "0")}`;
+ const tenth = precise ? `.${Math.floor((s * 10) % 10)}` : "";
+ return `${minutes}:${String(whole).padStart(2, "0")}${tenth}`;
+}
+
+/** Move one trim edge, keeping the clip at least `MIN_CLIP_S` long and inside the video. */
+export function moveTrimEdge(trim: Trim, edge: "start" | "end", at: number, duration: number): Trim {
+ const min = Math.min(MIN_CLIP_S, duration);
+ if (edge === "start") return { ...trim, start: Math.max(0, Math.min(at, trim.end - min)) };
+ return { ...trim, end: Math.min(duration, Math.max(at, trim.start + min)) };
+}
+
+type Drag = { kind: "scrub" } | { kind: "start" | "end" };
+
+export function AppshotVideoEditor({ recordingId, onClose }: { recordingId: string; onClose: () => void }) {
+ const t = useT();
+ const ready = useLocaleChunk("appshot_editor");
+ const label = (key: string) => (ready ? t(`appshot_editor.${key}`) : "");
+ const caps = useCapabilities();
+ const native = caps.data?.native_file_actions ?? false;
+ const pushToast = useEventStore((s) => s.pushToast);
+
+ const src = appshotRecordingUrl(recordingId);
+ const videoRef = useRef<HTMLVideoElement | null>(null);
+ const trackRef = useRef<HTMLDivElement | null>(null);
+ const [load, setLoad] = useState<"loading" | "ready" | "failed">("loading");
+ const [duration, setDuration] = useState(0);
+ const [size, setSize] = useState<[number, number]>([0, 0]);
+ const [time, setTime] = useState(0);
+ const [playing, setPlaying] = useState(false);
+ const [speed, setSpeed] = useState<number>(1);
+ const [trim, setTrim] = useState<Trim>({ start: 0, end: 0 });
+ const trimRef = useRef(trim);
+ trimRef.current = trim;
+ const [drag, setDrag] = useState<Drag | null>(null);
+ const [saving, setSaving] = useState(false);
+ const thumbs = useFilmStrip(src, load === "ready" ? duration : 0, size);
+
+ const trimmed = duration > 0 && (trim.start > 0.05 || trim.end < duration - 0.05);
+
+ // Smooth playhead: follow the video every frame while it plays, and stop
+ // at the end of the trimmed part.
+ useEffect(() => {
+ if (!playing) return;
+ let frame = 0;
+ const step = () => {
+ const video = videoRef.current;
+ if (!video) return;
+ const { end } = trimRef.current;
+ if (video.currentTime >= end - 0.01) {
+ video.pause();
+ video.currentTime = end;
+ setTime(end);
+ return;
+ }
+ setTime(video.currentTime);
+ frame = requestAnimationFrame(step);
+ };
+ frame = requestAnimationFrame(step);
+ return () => cancelAnimationFrame(frame);
+ }, [playing]);
+
+ useEffect(() => {
+ if (videoRef.current) videoRef.current.playbackRate = speed;
+ }, [speed]);
+
+ const seek = useCallback((at: number) => {
+ const video = videoRef.current;
+ if (!video || !duration) return;
+ const clamped = Math.max(0, Math.min(at, duration));
+ video.currentTime = clamped;
+ setTime(clamped);
+ }, [duration]);
+
+ const toggle = useCallback(() => {
+ const video = videoRef.current;
+ if (!video || load !== "ready") return;
+ if (!video.paused) {
+ video.pause();
+ return;
+ }
+ const { start, end } = trimRef.current;
+ // Play inside the trimmed part; at its end, start over.
+ if (video.currentTime >= end - 0.05 || video.currentTime < start) video.currentTime = start;
+ void video.play().catch(() => undefined);
+ }, [load]);
+
+ const save = useCallback(async () => {
+ if (load !== "ready" || saving) return;
+ setSaving(true);
+ try {
+ const unchanged = !trimmed && speed === 1;
+ const { id } = unchanged
+ ? { id: recordingId }
+ : await exportAppshotRecording(recordingId, { start_s: trim.start, end_s: trim.end, speed });
+ if (native) {
+ const { path, filename } = await saveAppshotRecording(id);
+ pushToast("success", fill(t("appshot_editor.saved_to"), { 0: path }), { filePath: path, filename });
+ } else {
+ // A browser: the file comes down through the browser's own download.
+ const link = document.createElement("a");
+ link.href = appshotRecordingUrl(id);
+ link.download = `recording-${id.slice(0, 6)}.mp4`;
+ link.click();
+ }
+ } catch (error) {
+ pushToast("error", fill(t("appshot_editor.save_failed"), { 0: (error as Error).message }));
+ } finally {
+ setSaving(false);
+ }
+ }, [load, native, pushToast, recordingId, saving, speed, t, trim, trimmed]);
+
+ // Keyboard: Space/K play, arrows seek (Shift = 5 s), , and . step a frame,
+ // I and O set the trim at the playhead, Home/End jump, Ctrl+S saves, Esc closes.
+ useEffect(() => {
+ const onKey = (event: KeyboardEvent) => {
+ const target = event.target;
+ if (target instanceof Element && target.closest("input, textarea, [contenteditable]")) return;
+ const video = videoRef.current;
+ const now = video?.currentTime ?? time;
+ const key = event.key;
+ if ((event.ctrlKey || event.metaKey) && key.toLowerCase() === "s") {
+ event.preventDefault();
+ void save();
+ return;
+ }
+ if (event.ctrlKey || event.metaKey || event.altKey) return;
+ const handled = (() => {
+ switch (key) {
+ case " ":
+ case "k":
+ case "K":
+ toggle();
+ return true;
+ case "ArrowLeft":
+ seek(now - (event.shiftKey ? 5 : 1));
+ return true;
+ case "ArrowRight":
+ seek(now + (event.shiftKey ? 5 : 1));
+ return true;
+ case ",":
+ video?.pause();
+ seek(now - FRAME_S);
+ return true;
+ case ".":
+ video?.pause();
+ seek(now + FRAME_S);
+ return true;
+ case "Home":
+ seek(trimRef.current.start);
+ return true;
+ case "End":
+ seek(trimRef.current.end);
+ return true;
+ case "i":
+ case "I":
+ setTrim((current) => moveTrimEdge(current, "start", now, duration));
+ return true;
+ case "o":
+ case "O":
+ setTrim((current) => moveTrimEdge(current, "end", now, duration));
+ return true;
+ case "Escape":
+ onClose();
+ return true;
+ default:
+ return false;
+ }
+ })();
+ if (handled) event.preventDefault();
+ };
+ window.addEventListener("keydown", onKey);
+ return () => window.removeEventListener("keydown", onKey);
+ }, [duration, onClose, save, seek, time, toggle]);
+
+ // -- timeline gestures -------------------------------------------------------
+ const timeAt = useCallback((clientX: number) => {
+ const track = trackRef.current;
+ if (!track || !duration) return 0;
+ const box = track.getBoundingClientRect();
+ return Math.max(0, Math.min(1, (clientX - box.left) / box.width)) * duration;
+ }, [duration]);
+
+ const onTrackDown = (event: ReactPointerEvent<HTMLDivElement>, kind: Drag["kind"]) => {
+ if (event.button !== 0 || load !== "ready") return;
+ event.preventDefault();
+ event.stopPropagation();
+ event.currentTarget.setPointerCapture?.(event.pointerId);
+ setDrag({ kind } as Drag);
+ const at = timeAt(event.clientX);
+ if (kind === "scrub") seek(at);
+ };
+
+ const onTrackMove = (event: ReactPointerEvent<HTMLDivElement>) => {
+ if (!drag) return;
+ const at = timeAt(event.clientX);
+ if (drag.kind === "scrub") {
+ seek(at);
+ return;
+ }
+ const next = moveTrimEdge(trimRef.current, drag.kind, at, duration);
+ setTrim(next);
+ // Show the frame at the edge being dragged.
+ seek(drag.kind === "start" ? next.start : next.end);
+ };
+
+ const onTrackUp = () => setDrag(null);
+
+ const pct = (seconds: number) => (duration ? `${(seconds / duration) * 100}%` : "0%");
+ const clipLength = Math.max(0, trim.end - trim.start) / speed;
+
+ return (
+ <div className="relative flex h-full w-full flex-col overflow-hidden bg-popover text-popover-foreground" data-testid="appshot-video-editor" data-native={native}>
+ {/* Top bar: title and size · free space to move the window · Save, close. */}
+ <div className="flex h-12 shrink-0 items-center gap-2 border-b border-border px-3">
+ <div className="flex min-w-0 items-center gap-2 pl-1">
+ <Film className="h-4 w-4 shrink-0 text-muted-foreground" aria-hidden />
+ <span className="truncate text-[13px] font-semibold">{label("video_title")}</span>
+ {size[0] > 0 && (
+ <span className="rounded-md bg-foreground/[0.06] px-1.5 py-0.5 text-[11px] font-medium tabular-nums text-muted-foreground">
+ {size[0]} × {size[1]}
+ </span>
+ )}
+ </div>
+ <div className="pywebview-drag-region h-full min-w-4 flex-1" aria-hidden />
+ <div className="flex shrink-0 items-center gap-1.5">
+ <QuickTooltip content={`${label("video_save_hint")} (Ctrl+S)`} side="bottom">
+ <button
+ type="button"
+ onClick={() => void save()}
+ disabled={load !== "ready" || saving}
+ data-testid="appshot-video-save"
+ className="flex h-7 items-center gap-1.5 rounded-lg bg-accent px-3.5 text-[13px] font-medium text-accent-foreground transition-colors hover:bg-accent/90 disabled:opacity-50"
+ >
+ {saving && <Loader2 className="h-3.5 w-3.5 animate-spin" aria-hidden />}
+ {saving ? label("video_saving") : label("save")}
+ </button>
+ </QuickTooltip>
+ <div className="mx-0.5 h-5 w-px shrink-0 bg-border" aria-hidden />
+ <QuickTooltip content={`${label("close")} (Esc)`} side="bottom">
+ <button
+ type="button"
+ onClick={onClose}
+ aria-label={label("close")}
+ data-testid="appshot-video-close"
+ className="flex h-7 w-7 shrink-0 items-center justify-center rounded-lg text-muted-foreground transition-colors hover:bg-destructive hover:text-destructive-foreground"
+ >
+ <X className="h-4 w-4" aria-hidden />
+ </button>
+ </QuickTooltip>
+ </div>
+ </div>
+
+ {/* The video: a click plays or pauses it. */}
+ <div className="relative flex min-h-0 flex-1 items-center justify-center bg-background/70 p-5">
+ {load === "failed" ? (
+ <div className="flex max-w-sm flex-col items-center gap-3 text-center" data-testid="appshot-video-failed">
+ <p className="text-base text-muted-foreground">{label("video_gone")}</p>
+ <Button type="button" variant="secondary" size="sm" onClick={onClose}>{label("close")}</Button>
+ </div>
+ ) : (
+ <>
+ <video
+ ref={videoRef}
+ src={src}
+ preload="auto"
+ playsInline
+ onClick={toggle}
+ onLoadedMetadata={(event) => {
+ const video = event.currentTarget;
+ const length = Number.isFinite(video.duration) ? video.duration : 0;
+ setDuration(length);
+ setSize([video.videoWidth, video.videoHeight]);
+ setTrim({ start: 0, end: length });
+ video.playbackRate = speed;
+ setLoad("ready");
+ }}
+ onError={() => setLoad("failed")}
+ onPlay={() => setPlaying(true)}
+ onPause={() => setPlaying(false)}
+ onSeeked={(event) => setTime(event.currentTarget.currentTime)}
+ className={cn(
+ "max-h-full max-w-full cursor-pointer rounded-xl bg-black shadow-2xl ring-1 ring-border transition-opacity duration-200",
+ load === "ready" ? "opacity-100" : "opacity-0",
+ )}
+ data-testid="appshot-video"
+ />
+ {load === "loading" && (
+ <div className="absolute inset-0 flex items-center justify-center gap-2 text-sm text-muted-foreground">
+ <Loader2 className="h-4 w-4 animate-spin" aria-hidden />
+ {label("video_loading")}
+ </div>
+ )}
+ {load === "ready" && (
+ <button
+ type="button"
+ onClick={toggle}
+ aria-label={label("video_play")}
+ tabIndex={-1}
+ className={cn(
+ "absolute left-1/2 top-1/2 flex h-16 w-16 -translate-x-1/2 -translate-y-1/2 items-center justify-center rounded-full bg-black/45 text-white shadow-xl ring-1 ring-white/20 backdrop-blur-md transition-all duration-200 hover:scale-105 hover:bg-black/60",
+ playing ? "pointer-events-none scale-90 opacity-0" : "opacity-100",
+ )}
+ >
+ <Play className="ml-1 h-7 w-7 fill-current" aria-hidden />
+ </button>
+ )}
+ </>
+ )}
+ </div>
+
+ {/* Timeline and transport. */}
+ <div className="shrink-0 space-y-3 border-t border-border px-4 pb-3 pt-3.5">
+ <div
+ ref={trackRef}
+ role="slider"
+ aria-label={label("video_timeline")}
+ aria-valuemin={0}
+ aria-valuemax={Math.round(duration * 10) / 10}
+ aria-valuenow={Math.round(time * 10) / 10}
+ tabIndex={-1}
+ onPointerDown={(event) => onTrackDown(event, "scrub")}
+ onPointerMove={onTrackMove}
+ onPointerUp={onTrackUp}
+ onPointerCancel={onTrackUp}
+ className={cn(
+ "relative h-14 select-none overflow-hidden rounded-xl border border-border bg-foreground/[0.05]",
+ load === "ready" ? "cursor-pointer" : "pointer-events-none opacity-60",
+ )}
+ data-testid="appshot-video-timeline"
+ >
+ {/* Film strip. */}
+ <div className="absolute inset-0 flex">
+ {thumbs.map((thumb, index) => (
+ <div key={index} className="h-full min-w-0 flex-1 overflow-hidden border-r border-black/20 last:border-r-0">
+ <img src={thumb} alt="" draggable={false} className="h-full w-full object-cover animate-in fade-in-0 duration-300" />
+ </div>
+ ))}
+ </div>
+ {/* Everything outside the trimmed part is dimmed. */}
+ <div className="pointer-events-none absolute inset-y-0 left-0 bg-black/55" style={{ width: pct(trim.start) }} />
+ <div className="pointer-events-none absolute inset-y-0 right-0 bg-black/55" style={{ width: `calc(100% - ${pct(trim.end)})` }} />
+ {/* The trimmed part with its two handles. */}
+ <div
+ className="pointer-events-none absolute inset-y-0 rounded-[10px] border-2 border-accent"
+ style={{ left: pct(trim.start), width: `calc(${pct(trim.end)} - ${pct(trim.start)})` }}
+ />
+ {(["start", "end"] as const).map((edge) => (
+ <div
+ key={edge}
+ role="separator"
+ aria-label={`${label(edge === "start" ? "video_trim_start" : "video_trim_end")} (${edge === "start" ? "I" : "O"})`}
+ onPointerDown={(event) => onTrackDown(event, edge)}
+ onPointerMove={onTrackMove}
+ onPointerUp={onTrackUp}
+ onPointerCancel={onTrackUp}
+ className={cn(
+ "absolute inset-y-0 z-10 flex w-3.5 cursor-ew-resize items-center justify-center bg-accent transition-[filter] hover:brightness-110",
+ edge === "start" ? "rounded-l-[10px]" : "-translate-x-full rounded-r-[10px]",
+ )}
+ style={{ left: pct(edge === "start" ? trim.start : trim.end) }}
+ data-testid={`appshot-video-trim-${edge}`}
+ >
+ <span className="h-5 w-[3px] rounded-full bg-accent-foreground/80" aria-hidden />
+ </div>
+ ))}
+ {/* Playhead. */}
+ <div
+ className={cn("pointer-events-none absolute inset-y-0 z-20 w-0", drag?.kind === "scrub" ? "" : "transition-[left] duration-75 ease-linear")}
+ style={{ left: pct(time) }}
+ data-testid="appshot-video-playhead"
+ >
+ <div className="absolute inset-y-0 -left-px w-0.5 bg-white shadow-[0_0_0_1px_rgba(0,0,0,0.35)]" />
+ <div className="absolute -left-[5px] -top-0.5 h-2.5 w-2.5 rounded-full bg-white shadow-[0_0_0_1px_rgba(0,0,0,0.35)]" />
+ </div>
+ </div>
+
+ <div className="grid grid-cols-[1fr_auto_1fr] items-center gap-3">
+ <div className="flex min-w-0 items-baseline gap-1.5 text-[13px] tabular-nums">
+ <span className="font-semibold text-foreground">{formatVideoTime(time)}</span>
+ <span className="text-muted-foreground">/ {formatVideoTime(duration)}</span>
+ </div>
+
+ <div className="flex items-center gap-1.5">
+ <QuickTooltip content={`${label("video_to_start")} (Home)`} side="top">
+ <button
+ type="button"
+ onClick={() => seek(trim.start)}
+ disabled={load !== "ready"}
+ aria-label={label("video_to_start")}
+ className="flex h-8 w-8 items-center justify-center rounded-lg text-muted-foreground transition-colors hover:bg-foreground/10 hover:text-foreground disabled:opacity-40"
+ >
+ <SkipBack className="h-4 w-4 fill-current" aria-hidden />
+ </button>
+ </QuickTooltip>
+ <QuickTooltip content={`${label(playing ? "video_pause" : "video_play")} (Space)`} side="top">
+ <button
+ type="button"
+ onClick={toggle}
+ disabled={load !== "ready"}
+ aria-label={label(playing ? "video_pause" : "video_play")}
+ data-testid="appshot-video-play"
+ className="flex h-10 w-10 items-center justify-center rounded-full bg-foreground text-background shadow-md transition-transform hover:scale-105 active:scale-95 disabled:opacity-40"
+ >
+ {playing ? <Pause className="h-[18px] w-[18px] fill-current" aria-hidden /> : <Play className="ml-0.5 h-[18px] w-[18px] fill-current" aria-hidden />}
+ </button>
+ </QuickTooltip>
+ <QuickTooltip content={`${label("video_to_end")} (End)`} side="top">
+ <button
+ type="button"
+ onClick={() => seek(trim.end)}
+ disabled={load !== "ready"}
+ aria-label={label("video_to_end")}
+ className="flex h-8 w-8 items-center justify-center rounded-lg text-muted-foreground transition-colors hover:bg-foreground/10 hover:text-foreground disabled:opacity-40"
+ >
+ <SkipForward className="h-4 w-4 fill-current" aria-hidden />
+ </button>
+ </QuickTooltip>
+ </div>
+
+ <div className="flex min-w-0 items-center justify-end gap-2">
+ {trimmed && (
+ <QuickTooltip content={label("video_trim_reset")} side="top">
+ <button
+ type="button"
+ onClick={() => setTrim({ start: 0, end: duration })}
+ aria-label={label("video_trim_reset")}
+ data-testid="appshot-video-trim-reset"
+ className="flex h-7 w-7 items-center justify-center rounded-lg text-muted-foreground transition-colors hover:bg-foreground/10 hover:text-foreground"
+ >
+ <RotateCcw className="h-3.5 w-3.5" aria-hidden />
+ </button>
+ </QuickTooltip>
+ )}
+ {(trimmed || speed !== 1) && (
+ <span className="hidden whitespace-nowrap text-[12px] tabular-nums text-muted-foreground sm:inline" data-testid="appshot-video-clip-length">
+ {fill(label("video_clip_length"), { 0: formatVideoTime(clipLength, false) })}
+ </span>
+ )}
+ <div
+ className="flex items-center gap-0.5 rounded-xl border border-border bg-foreground/[0.04] p-0.5"
+ role="group"
+ aria-label={label("video_speed")}
+ data-testid="appshot-video-speed"
+ >
+ {RECORDING_SPEEDS.map((value) => (
+ <button
+ key={value}
+ type="button"
+ onClick={() => setSpeed(value)}
+ aria-pressed={speed === value}
+ className={cn(
+ "h-6 rounded-lg px-2 text-[12px] font-medium tabular-nums transition-colors",
+ speed === value ? "bg-accent text-accent-foreground shadow-sm" : "text-muted-foreground hover:bg-foreground/10 hover:text-foreground",
+ )}
+ >
+ {value}×
+ </button>
+ ))}
+ </div>
+ </div>
+ </div>
+ <p className="text-center text-[11px] text-muted-foreground/80">{label("video_shortcuts")}</p>
+ </div>
+ </div>
+ );
+}
+
+/**
+ * Small frames along the video for the timeline, taken by a second, hidden
+ * video element so playback is never disturbed. Empty until the first frame
+ * is ready; frames appear one by one.
+ */
+function useFilmStrip(src: string, duration: number, size: [number, number]): string[] {
+ const [thumbs, setThumbs] = useState<string[]>([]);
+ const count = useMemo(() => {
+ if (!duration || !size[0] || !size[1]) return 0;
+ // About one frame per 80 px of a typical timeline, never more than MAX_THUMBS.
+ return Math.max(4, Math.min(MAX_THUMBS, Math.round(duration / 2) + 4));
+ }, [duration, size]);
+
+ useEffect(() => {
+ setThumbs([]);
+ if (!count) return;
+ let cancelled = false;
+ const video = document.createElement("video");
+ video.muted = true;
+ video.preload = "auto";
+ video.src = src;
+ const canvas = document.createElement("canvas");
+ const height = 96;
+ canvas.height = height;
+ canvas.width = Math.max(1, Math.round((height * size[0]) / size[1]));
+ const ctx = canvas.getContext("2d");
+ const grab = (at: number) =>
+ new Promise<string>((resolve, reject) => {
+ const done = () => {
+ video.removeEventListener("seeked", done);
+ video.removeEventListener("error", fail);
+ try {
+ ctx?.drawImage(video, 0, 0, canvas.width, canvas.height);
+ resolve(canvas.toDataURL("image/jpeg", 0.72));
+ } catch (error) {
+ reject(error);
+ }
+ };
+ const fail = () => reject(new Error("seek failed"));
+ video.addEventListener("seeked", done);
+ video.addEventListener("error", fail);
+ video.currentTime = at;
+ });
+ void (async () => {
+ try {
+ await new Promise<void>((resolve, reject) => {
+ if (video.readyState >= 1) resolve();
+ video.addEventListener("loadedmetadata", () => resolve(), { once: true });
+ video.addEventListener("error", () => reject(new Error("load failed")), { once: true });
+ });
+ for (let i = 0; i < count && !cancelled; i += 1) {
+ const frame = await grab(((i + 0.5) * duration) / count);
+ if (!cancelled) setThumbs((current) => [...current, frame]);
+ }
+ } catch {
+ // The strip is decoration: the timeline still works without it.
+ }
+ })();
+ return () => {
+ cancelled = true;
+ video.removeAttribute("src");
+ video.load();
+ };
+ }, [count, duration, size, src]);
+
+ // Keep the strip's slots stable while frames arrive.
+ return thumbs.length ? [...thumbs, ...Array<string>(Math.max(0, count - thumbs.length)).fill("")].map((thumb) => thumb || TRANSPARENT) : [];
+}
+
+const TRANSPARENT = "data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==";
diff --git a/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx b/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx
index a4d45182b9..ab0d43f69f 100644
--- a/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx
+++ b/jarvis/ui/web/frontend/src/views/AppshotsView.test.tsx
@@ -274,6 +274,32 @@ describe("AppshotsView shortcut recorder", () => {
await waitFor(() => expect(puts()).toEqual([{ region_hotkey: "shift+shift" }]));
});
+ it("offers the video recording shortcut beside the picture shortcuts", async () => {
+ const withRecording = {
+ ...SETTINGS,
+ recording_hotkey: "",
+ recording_shortcut: { hotkey: "", armed: false, detail: "" },
+ };
+ fetchMock.mockImplementation(async (url: string, init?: RequestInit) => {
+ if (url === "/api/appshot/settings" && init?.method === "PUT") {
+ return json({ ...withRecording, ...JSON.parse(String(init.body)) });
+ }
+ if (url === "/api/appshot/settings") return json(withRecording);
+ return json({ phase: "idle", id: "", message: "" });
+ });
+ render(<AppshotsView />);
+ const field = await screen.findByTestId("appshots-recording-hotkey");
+ const region = screen.getByTestId("appshots-region-hotkey");
+ // Same settings list, right after the area shortcut.
+ expect(region.compareDocumentPosition(field) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
+ expect(screen.getByText("Video recording shortcut")).toBeTruthy();
+ fireEvent.click(screen.getByTestId("appshots-recording-hotkey-change"));
+ fireEvent.keyDown(window, { code: "ControlLeft", key: "Control", ctrlKey: true });
+ fireEvent.keyDown(window, { code: "Digit9", key: "9", ctrlKey: true });
+ fireEvent.keyUp(window, { code: "Digit9", key: "9", ctrlKey: false });
+ await waitFor(() => expect(puts()).toEqual([{ recording_hotkey: "ctrl+9" }]));
+ });
+
it("records an ordinary combo for the window shortcut", async () => {
render(<AppshotsView />);
fireEvent.click(await screen.findByTestId("appshots-hotkey-change"));
diff --git a/jarvis/ui/web/frontend/src/views/AppshotsView.tsx b/jarvis/ui/web/frontend/src/views/AppshotsView.tsx
index f67fe31ddf..4d31eeed81 100644
--- a/jarvis/ui/web/frontend/src/views/AppshotsView.tsx
+++ b/jarvis/ui/web/frontend/src/views/AppshotsView.tsx
@@ -220,15 +220,17 @@ export function AppshotsView() {
const [saving, setSaving] = useState(false);
const [countdown, setCountdown] = useState<number | null>(null);
const [picking, setPicking] = useState(false);
+ const [savedRecording, setSavedRecording] = useState("");
const openEditor = useAppshotEditor((s) => s.open);
// Bumped by AppshotEditorHost when an edit replaced the held picture.
const revision = useAppshotEditor((s) => s.revision);
// While a shortcut field records (or refuses a gesture), its row says so
// in place of the description.
- const [fieldStatus, setFieldStatus] = useState<{ window: string | null; region: string | null }>({
- window: null,
- region: null,
- });
+ const [fieldStatus, setFieldStatus] = useState<{
+ window: string | null;
+ region: string | null;
+ recording: string | null;
+ }>({ window: null, region: null, recording: null });
const windowStatus = useCallback(
(text: string | null) =>
setFieldStatus((s) => (s.window === text ? s : { ...s, window: text })),
@@ -239,6 +241,11 @@ export function AppshotsView() {
setFieldStatus((s) => (s.region === text ? s : { ...s, region: text })),
[],
);
+ const recordingStatus = useCallback(
+ (text: string | null) =>
+ setFieldStatus((s) => (s.recording === text ? s : { ...s, recording: text })),
+ [],
+ );
const countdownTimer = useRef<number | null>(null);
useEffect(() => {
@@ -415,6 +422,22 @@ export function AppshotsView() {
);
})();
+ // Screen recording sits beside the two picture shortcuts, not below the
+ // library: it is the shortcut people look for first when they want a video.
+ const recordingSupported = typeof settings?.recording_hotkey === "string";
+
+ const recordingShortcutHint = (() => {
+ if (!settings || !recordingSupported) return "";
+ if (!settings.recording_hotkey) return t("appshots.recording_shortcut_hint_off");
+ const detail = settings.recording_shortcut?.detail;
+ if (settings.enabled && !settings.recording_shortcut?.armed && detail) {
+ return t("appshots.shortcut_unavailable").replace("{0}", detail);
+ }
+ return t("appshots.recording_shortcut_hint")
+ .split("{0}")
+ .join(formatAppshotHotkey(settings.recording_hotkey, IS_MAC));
+ })();
+
const targetHint =
settings?.target === "message"
? t("appshots.target_message_hint")
@@ -499,6 +522,24 @@ export function AppshotsView() {
}
/>
)}
+ {recordingSupported && (
+ <Row
+ label={t("appshots.recording_shortcut_label")}
+ hint={fieldStatus.recording ?? recordingShortcutHint}
+ control={
+ <AppshotShortcutField
+ value={settings.recording_hotkey ?? ""}
+ isMac={IS_MAC}
+ disabled={disabled || saving}
+ testId="appshots-recording-hotkey"
+ label={t("appshots.recording_shortcut_label")}
+ className="w-44"
+ onSave={(recordingHotkey) => save({ recording_hotkey: recordingHotkey })}
+ onStatus={recordingStatus}
+ />
+ }
+ />
+ )}
<Row
label={t("appshots.target_label")}
hint={targetHint}
@@ -678,12 +719,13 @@ export function AppshotsView() {
<AppshotLibrary
enabled={settings?.library}
- refreshKey={`${lastAppshotEvent}:${revision}`}
+ refreshKey={`${lastAppshotEvent}:${revision}:${savedRecording}`}
/>
{settings && typeof settings.recording_hotkey === "string" && (
<AppshotRecordingPanel settings={settings} saving={saving}
- onShortcut={(recording_hotkey) => save({ recording_hotkey })} />
+ onSettings={save}
+ onSaved={setSavedRecording} />
)}
<p className="mt-5 text-sm text-muted-foreground">{t("appshots.voice_hint")}</p>
</div>
diff --git a/jarvis/ui/web/frontend/src/views/ComputersView.test.tsx b/jarvis/ui/web/frontend/src/views/ComputersView.test.tsx
index 43cc0de073..ec9d568b64 100644
--- a/jarvis/ui/web/frontend/src/views/ComputersView.test.tsx
+++ b/jarvis/ui/web/frontend/src/views/ComputersView.test.tsx
@@ -5,7 +5,7 @@
*/
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
-import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { ComputersView } from "@/views/ComputersView";
import { loadLocaleChunk, setUiLanguage } from "@/i18n";
@@ -304,6 +304,7 @@ describe("ComputersView", () => {
renderView();
fireEvent.click(await screen.findByTestId("computers-add-first"));
+ fireEvent.click(await screen.findByTestId("cx-additional-options"));
fireEvent.click(await screen.findByTestId("cx-account-hostinger"));
expect(await screen.findByTestId("wz-token")).toBeTruthy();
});
@@ -371,6 +372,62 @@ describe("ComputersView", () => {
expect(screen.getByTestId("cx-auto")).toBeTruthy();
});
+ it("switches connection forms without losing drafts or sending pairing data to SSH", async () => {
+ const calls = installFetch({
+ "GET /api/computers": () => ({ computers: [] }),
+ "GET /api/computers/providers": () => ({ providers: PROVIDERS }),
+ "GET /api/computers/identity": () => IDENTITY,
+ });
+ renderView();
+ fireEvent.click(await screen.findByTestId("computers-add-first"));
+ fireEvent.change(await screen.findByTestId("cx-address"), { target: { value: "192.0.2.10" } });
+ fireEvent.click(screen.getByRole("radio", { name: "Server address" }));
+ expect(screen.queryByRole("textbox", { name: "SSH host" })).toBeNull();
+ expect(screen.queryByTestId("cx-additional-options")).toBeNull();
+ fireEvent.paste(screen.getByRole("textbox", { name: "Host" }), {
+ clipboardData: { getData: () => "https://backend.example.com/pair?code=ExampleCode" },
+ });
+ expect((screen.getByTestId("cx-server-host") as HTMLInputElement).value).toBe("https://backend.example.com");
+ expect((screen.getByTestId("cx-pairing-code") as HTMLInputElement).value).toBe("ExampleCode");
+ expect((within(screen.getByTestId("cx-server-form")).getByRole("button", { name: "Add computer" }) as HTMLButtonElement).disabled).toBe(true);
+ fireEvent.submit(screen.getByTestId("cx-server-form"));
+ expect(calls.some((call) => call.method === "POST")).toBe(false);
+ fireEvent.click(screen.getByRole("radio", { name: "SSH" }));
+ expect((screen.getByRole("textbox", { name: "SSH host" }) as HTMLInputElement).value).toBe("192.0.2.10");
+ expect(screen.queryByRole("textbox", { name: "Pairing code" })).toBeNull();
+ fireEvent.click(screen.getByRole("radio", { name: "Server address" }));
+ expect((screen.getByRole("textbox", { name: "Pairing code" }) as HTMLInputElement).value).toBe("ExampleCode");
+ });
+
+ it("validates the visible SSH port and sends explicit login overrides", async () => {
+ const calls = installFetch({
+ "GET /api/computers": () => ({ computers: [] }),
+ "GET /api/computers/providers": () => ({ providers: PROVIDERS }),
+ "GET /api/computers/identity": () => IDENTITY,
+ "POST /api/computers/test": () => ({
+ ok: false, kind: "unreachable", message: null, facts: null,
+ host_fingerprint: null, latency_ms: null,
+ }),
+ });
+ renderView();
+ fireEvent.click(await screen.findByTestId("computers-add-first"));
+ fireEvent.change(await screen.findByLabelText("SSH host"), { target: { value: "ssh [email protected]:2222" } });
+ fireEvent.change(screen.getByTestId("cx-username"), { target: { value: "deploy" } });
+ const port = screen.getByTestId("cx-port");
+ for (const value of ["0", "65536", "1.5", "-1"]) {
+ fireEvent.change(port, { target: { value } });
+ expect((screen.getByTestId("cx-connect") as HTMLButtonElement).disabled).toBe(true);
+ expect(screen.getByRole("alert").textContent).toContain("between 1 and 65535");
+ }
+ expect(calls.some((call) => call.method === "POST")).toBe(false);
+ fireEvent.change(port, { target: { value: "2200" } });
+ fireEvent.click(screen.getByTestId("cx-connect"));
+ await screen.findByTestId("cx-edit");
+ expect(calls.find((call) => call.url === "/api/computers/test")?.body).toMatchObject({
+ host: "192.0.2.10", username: "deploy", port: 2200, auth: "auto",
+ });
+ });
+
it("lists a machine, opens it and runs a console command", async () => {
const calls = installFetch({
"GET /api/computers": () => ({ computers: [VPS] }),
diff --git a/jarvis/ui/web/frontend/src/views/ComputersView.tsx b/jarvis/ui/web/frontend/src/views/ComputersView.tsx
index 90fa6e2ee5..ad464a28fc 100644
--- a/jarvis/ui/web/frontend/src/views/ComputersView.tsx
+++ b/jarvis/ui/web/frontend/src/views/ComputersView.tsx
@@ -2,25 +2,28 @@
* Computers — the servers and virtual machines {name} can work on besides
* this one. A tab of the Settings hub (System group).
*
- * The page is a quiet table of machines with their state and live load; a
- * row opens the machine's own page (Overview · Console · Access · Agents).
- * Adding one is a guided four-step wizard. Below the table sit the two
- * settings that belong to all machines at once: keeping IDE work going when
- * this PC closes, and {name}'s own SSH key.
+ * One centred column. A small heading names the page; the card under it
+ * holds what belongs to this PC and every machine at once (keeping IDE work
+ * going when this PC closes, {name}'s own SSH key). Below, "Your computers"
+ * lists each machine as one row (glyph, name, one subtitle line, status dot);
+ * a row opens the machine's own page (Overview · Console · Access · Agents),
+ * and "Add computer" opens the guided dialog.
*/
import { useMemo, useState } from "react";
-import { Fingerprint, Loader2, Plus, RefreshCw, Server } from "lucide-react";
+import { Check, Copy, Loader2, Plus, RefreshCw } from "lucide-react";
import { Panel } from "@/components/extensions/primitives";
-import { PageHeader } from "@/components/layout/PageHeader";
+import { ComputersIcon } from "@/components/icons/sectionIcons";
import { ScrollArea } from "@/components/ui/scroll-area";
import { Button } from "@/components/ui/button";
import { useCheckAll, useComputers, useIdentity } from "@/hooks/useComputers";
import { useLocaleChunk, useT } from "@/i18n";
+import { robustCopy } from "@/lib/clipboard";
import type { Computer } from "@/lib/computersApi";
import { ComputerDetail, type DetailTab } from "@/views/computers/ComputerDetail";
-import { ComputerRow, ComputerTableHead } from "@/views/computers/ComputerRow";
+import { ComputerRow } from "@/views/computers/ComputerRow";
import { KeepWorking } from "@/views/computers/KeepWorking";
-import { CopyField, needsAttention } from "@/views/computers/parts";
+import { needsAttention } from "@/views/computers/parts";
+import { Empty, Group, Row, Section, sectionActionCls } from "@/views/computers/surface";
import { ConnectDialog } from "@/views/computers/ConnectDialog";
/** Keyframes the provisioning bar uses; scoped by name, shipped with the view. */
@@ -29,49 +32,39 @@ const KEYFRAMES = `@keyframes computers-indeterminate {
100% { transform: translateX(300%); }
}`;
-function EmptyState({ onAdd }: { onAdd: () => void }) {
- const t = useT();
- return (
- <div
- className="flex flex-col items-center rounded-xl border border-dashed border-border px-6 py-16 text-center"
- data-testid="computers-welcome"
- >
- <span className="flex h-12 w-12 items-center justify-center rounded-full bg-secondary text-muted-foreground">
- <Server className="h-5 w-5" aria-hidden />
- </span>
- <h2 className="mt-4 text-lg font-semibold text-foreground-strong">{t("computers.empty_title")}</h2>
- <p className="mt-1.5 max-w-md text-base text-muted-foreground">{t("computers.empty_body")}</p>
- <Button className="mt-6" onClick={onAdd} data-testid="computers-add-first">
- <Plus />
- {t("computers.add")}
- </Button>
- <p className="mt-4 text-xs text-muted-foreground">{t("computers.empty_foot")}</p>
- </div>
- );
-}
-
function IdentityRow() {
const t = useT();
const identity = useIdentity();
+ const [copied, setCopied] = useState(false);
if (!identity.data) return null;
+ const { algorithm, fingerprint, public_key } = identity.data;
return (
- <Panel className="p-5">
- <div className="flex items-start gap-3">
- <Fingerprint className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" aria-hidden />
- <div className="min-w-0 flex-1">
- <div className="text-base font-semibold text-foreground-strong">{t("computers.identity_title")}</div>
- <p className="mt-0.5 text-sm text-muted-foreground">{t("computers.identity_body")}</p>
- <div className="mt-4">
- <CopyField
- label={`${identity.data.algorithm} · ${identity.data.fingerprint}`}
- value={identity.data.public_key}
- copyLabel={t("computers.copy")}
- copiedLabel={t("computers.copied")}
- />
- </div>
- </div>
- </div>
- </Panel>
+ <Row
+ title={t("computers.identity_title")}
+ description={t("computers.identity_body")}
+ status={
+ <span className="block truncate font-mono" title={public_key}>
+ {algorithm} · {fingerprint}
+ </span>
+ }
+ control={
+ <Button
+ variant="outline"
+ size="sm"
+ aria-label={t("computers.copy")}
+ onClick={() => {
+ void robustCopy(public_key).then((ok) => {
+ if (!ok) return;
+ setCopied(true);
+ window.setTimeout(() => setCopied(false), 1600);
+ });
+ }}
+ >
+ {copied ? <Check className="text-success" /> : <Copy />}
+ {copied ? t("computers.copied") : t("computers.copy")}
+ </Button>
+ }
+ />
);
}
@@ -87,6 +80,12 @@ export function ComputersView() {
const current = open ? rows.find((c) => c.id === open.id) ?? null : null;
const attention = rows.filter(needsAttention).length;
const online = rows.filter((c) => c.health.status === "online").length;
+ const summary = [
+ t("computers.list_summary").replace("{online}", String(online)).replace("{total}", String(rows.length)),
+ attention > 0 ? t("computers.list_attention").replace("{count}", String(attention)) : null,
+ ]
+ .filter(Boolean)
+ .join(" · ");
const openComputer = (computer: Computer, tab: DetailTab) => {
setAdding(false);
@@ -97,95 +96,115 @@ export function ComputersView() {
<div className="flex h-full min-h-0 w-full flex-col" data-testid="computers-view">
<style>{KEYFRAMES}</style>
<ScrollArea className="min-h-0 flex-1">
- <div className="flex w-full flex-col gap-6 px-8 pb-10">
- {!current && (
- <PageHeader
- icon={<Server />}
- title={t("computers.title")}
- description={t("computers.subtitle")}
- actions={
- rows.length > 0 ? (
+ {current && open ? (
+ <div className="flex w-full flex-col px-8 pb-10 pt-6">
+ <ComputerDetail
+ key={current.id}
+ computer={current}
+ initialTab={open.tab}
+ onBack={() => setOpen(null)}
+ />
+ </div>
+ ) : (
+ <div className="mx-auto flex w-full max-w-4xl flex-col gap-8 px-6 pb-20 pt-10 sm:px-10">
+ <div className="space-y-2.5">
+ <header
+ className="flex min-h-7 items-center justify-between gap-4 px-3 sm:px-4"
+ data-testid="section-header"
+ >
+ <h1
+ className="flex min-w-0 items-center gap-2 text-base font-medium text-foreground-strong"
+ title={t("computers.subtitle")}
+ >
+ <ComputersIcon className="h-4 w-4 shrink-0 text-muted-foreground" aria-hidden />
+ <span className="truncate">{t("computers.title")}</span>
+ </h1>
+ {rows.length > 0 && (
+ <span className="truncate text-sm text-muted-foreground">{summary}</span>
+ )}
+ </header>
+
+ {computers.isLoading && (
+ <div className="flex items-center gap-2 px-4 py-10 text-sm text-muted-foreground">
+ <Loader2 className="h-4 w-4 animate-spin" /> {t("computers.loading")}
+ </div>
+ )}
+ {computers.isError && (
+ <Panel className="p-5 text-sm text-destructive">{t("computers.load_failed")}</Panel>
+ )}
+ {computers.isSuccess && (
+ <Group>
+ {rows.length > 0 && <KeepWorking computers={rows} />}
+ <IdentityRow />
+ </Group>
+ )}
+ </div>
+
+ {computers.isSuccess && (
+ <Section
+ title={t("computers.list_title")}
+ action={
<>
+ {rows.length > 0 && (
+ <button
+ type="button"
+ className={sectionActionCls}
+ onClick={() => checkAll.mutate()}
+ disabled={checkAll.isPending}
+ data-testid="computers-check-all"
+ >
+ {checkAll.isPending ? <Loader2 className="animate-spin" /> : <RefreshCw />}
+ {t("computers.check_all")}
+ </button>
+ )}
+ <button
+ type="button"
+ className={sectionActionCls}
+ onClick={() => setAdding(true)}
+ data-testid="computers-add"
+ >
+ <Plus />
+ {t("computers.add")}
+ </button>
+ </>
+ }
+ >
+ {rows.length === 0 ? (
+ <Empty
+ testId="computers-welcome"
+ icon={<ComputersIcon />}
+ title={t("computers.empty_title")}
+ description={t("computers.empty_body")}
+ >
<Button
variant="outline"
- onClick={() => checkAll.mutate()}
- disabled={checkAll.isPending}
- data-testid="computers-check-all"
+ size="sm"
+ className="mt-5"
+ onClick={() => setAdding(true)}
+ data-testid="computers-add-first"
>
- {checkAll.isPending ? <Loader2 className="animate-spin" /> : <RefreshCw />}
- {t("computers.check_all")}
- </Button>
- <Button onClick={() => setAdding(true)} data-testid="computers-add">
<Plus />
{t("computers.add")}
</Button>
- </>
- ) : null
- }
- />
- )}
-
- {computers.isLoading && (
- <div className="flex items-center gap-2 py-10 text-sm text-muted-foreground">
- <Loader2 className="h-4 w-4 animate-spin" /> {t("computers.loading")}
- </div>
- )}
- {computers.isError && (
- <Panel className="p-5 text-sm text-destructive">{t("computers.load_failed")}</Panel>
- )}
-
- {current && open && (
- <div className="pt-6">
- <ComputerDetail
- key={current.id}
- computer={current}
- initialTab={open.tab}
- onBack={() => setOpen(null)}
- />
- </div>
- )}
-
- {!current && computers.isSuccess && rows.length === 0 && <EmptyState onAdd={() => setAdding(true)} />}
-
- {!current && rows.length > 0 && (
- <>
- <section aria-labelledby="computers-list-title">
- <div className="mb-3 flex items-baseline justify-between gap-3">
- <h2 id="computers-list-title" className="text-base font-semibold text-foreground-strong">
- {t("computers.list_title")}
- </h2>
- <span className="text-sm text-muted-foreground">
- {t("computers.list_summary")
- .replace("{online}", String(online))
- .replace("{total}", String(rows.length))}
- {attention > 0 && ` · ${t("computers.list_attention").replace("{count}", String(attention))}`}
- </span>
- </div>
- <div className="overflow-hidden rounded-lg border border-border bg-card" role="table">
- <ComputerTableHead />
- <ul className="divide-y divide-border" data-testid="computers-list" role="rowgroup">
+ <p className="mt-4 text-xs text-foreground-faint">{t("computers.empty_foot")}</p>
+ </Empty>
+ ) : (
+ <ul className="divide-y divide-border" data-testid="computers-list">
{rows.map((computer) => (
- <ComputerRow
- key={computer.id}
- computer={computer}
- checking={checkAll.isPending}
- onOpen={() => setOpen({ id: computer.id, tab: "overview" })}
- />
+ <li key={computer.id}>
+ <ComputerRow
+ computer={computer}
+ checking={checkAll.isPending}
+ onOpen={() => setOpen({ id: computer.id, tab: "overview" })}
+ />
+ </li>
))}
</ul>
- </div>
- </section>
-
- <section aria-labelledby="computers-settings-title" className="space-y-4">
- <h2 id="computers-settings-title" className="text-base font-semibold text-foreground-strong">
- {t("computers.settings_title")}
- </h2>
- <KeepWorking computers={rows} />
- <IdentityRow />
- </section>
- </>
- )}
- </div>
+ )}
+ </Section>
+ )}
+ </div>
+ )}
</ScrollArea>
{adding && <ConnectDialog onClose={() => setAdding(false)} onOpen={openComputer} />}
diff --git a/jarvis/ui/web/frontend/src/views/MarketplaceView.tsx b/jarvis/ui/web/frontend/src/views/MarketplaceView.tsx
index a4616589d9..360034e251 100644
--- a/jarvis/ui/web/frontend/src/views/MarketplaceView.tsx
+++ b/jarvis/ui/web/frontend/src/views/MarketplaceView.tsx
@@ -36,7 +36,7 @@ import { openExternalUrl } from "@/lib/openExternal";
import { bundledPluginLogo } from "@/lib/pluginLogos";
import { useEventStore, type SectionId } from "@/store/events";
import { installAgentTemplate, readFileText, type AgentTemplateWire } from "@/lib/agentShare";
-import { AgentSymbol } from "@/components/society/AgentSymbol";
+import { CompanionMark } from "@/components/society/companion/CompanionMark";
import { resolveCompanion } from "@/components/society/companion/appearance";
import {
MARKETPLACE_SUBMIT_URL,
@@ -206,7 +206,7 @@ function AgentTile({ entry }: { entry: Entry }) {
const appearance = resolveCompanion(entry.agent?.name || entry.name, entry.agent?.avatar?.companion);
return (
<div className="grid h-10 w-10 shrink-0 place-items-center rounded-lg bg-secondary">
- <AgentSymbol {...appearance} size={30} />
+ <CompanionMark appearance={appearance} size={30} />
</div>
);
}
diff --git a/jarvis/ui/web/frontend/src/views/computers/ComputerRow.tsx b/jarvis/ui/web/frontend/src/views/computers/ComputerRow.tsx
index 11877780d9..63f0f32d6f 100644
--- a/jarvis/ui/web/frontend/src/views/computers/ComputerRow.tsx
+++ b/jarvis/ui/web/frontend/src/views/computers/ComputerRow.tsx
@@ -1,48 +1,30 @@
/**
- * The computer list as a quiet table: one row per machine with its provider,
- * state, address, system and live load. The whole row opens the detail page.
+ * Portions adapted from pingdotgg/t3code @ 12069ee (apps/web EnvironmentRow),
+ * MIT License, Copyright (c) 2026 T3 Tools Inc. Full text:
+ * third_party/t3code/LICENSE.
+ *
+ * One machine in the grouped list: its glyph, its name and ONE subtitle line
+ * (how it is reached, its state, its system), then any tight vital, the
+ * status dot and a chevron. The whole row opens the detail page.
*/
import { ChevronRight, Loader2 } from "lucide-react";
+import { ComputersIcon } from "@/components/icons/sectionIcons";
import { ProviderLogo } from "@/components/providers/ProviderLogo";
import { useT } from "@/i18n";
import { cn } from "@/lib/utils";
import type { Computer } from "@/lib/computersApi";
-import { Meter, StatusLight, formatMemory, loadPct, statusTone } from "./parts";
+import { formatMemory, loadPct, statusTone } from "./parts";
+import { StatusDot } from "./surface";
export function statusLabel(computer: Computer, t: (k: string) => string): string {
return t(`computers.status_${computer.health.status}`);
}
-/** Shared column template so the header and every row line up. */
-export const TABLE_COLUMNS =
- "md:grid-cols-[minmax(0,1.6fr)_minmax(0,0.9fr)_minmax(0,1fr)_minmax(0,1.2fr)_minmax(0,1.3fr)_20px]";
+/** Providers without a brand mark of their own wear the section's cloud. */
+const CLOUD_MARK_PROVIDERS = new Set(["generic", "home_server", "strato"]);
-export function ComputerTableHead() {
- const t = useT();
- const cells = [
- t("computers.col_computer"),
- t("computers.col_status"),
- t("computers.col_address"),
- t("computers.col_system"),
- t("computers.col_load"),
- ];
- return (
- <div
- role="row"
- className={cn(
- "hidden gap-x-6 border-b border-border px-5 py-2.5 text-xs font-medium uppercase tracking-wide text-foreground-faint md:grid",
- TABLE_COLUMNS,
- )}
- >
- {cells.map((cell) => (
- <span key={cell} role="columnheader">
- {cell}
- </span>
- ))}
- <span aria-hidden />
- </div>
- );
-}
+/** A vital earns a place in the row only once it gets tight. */
+const TIGHT_PCT = 75;
export function ComputerRow({
computer,
@@ -56,60 +38,67 @@ export function ComputerRow({
const t = useT();
const { facts, health } = computer;
const online = health.status === "online";
- const system = [facts?.os_name, facts?.cpu_count ? `${facts.cpu_count} ${t("computers.unit_cpu")}` : null, formatMemory(facts?.mem_total_mb)]
+ const tone = statusTone(health.status);
+ const busy = checking || computer.busy;
+ const login = `${computer.username}@${computer.host === "0.0.0.0" ? "…" : computer.host}${computer.port !== 22 ? `:${computer.port}` : ""}`;
+ const status = [statusLabel(computer, t), online && health.latency_ms !== null ? `${health.latency_ms} ms` : null]
+ .filter(Boolean)
+ .join(" ");
+ const system = [
+ facts?.os_name,
+ facts?.cpu_count ? `${facts.cpu_count} ${t("computers.unit_cpu")}` : null,
+ formatMemory(facts?.mem_total_mb),
+ ]
.filter(Boolean)
.join(" · ");
- const providerName =
- computer.provider_name ||
- (computer.provider === "generic" ? t("computers.provider_generic") : t(`computers.provider_${computer.provider}`));
+ const tight = online
+ ? [
+ { label: t("computers.meter_cpu"), pct: loadPct(computer) },
+ { label: t("computers.meter_memory"), pct: health.mem_used_pct },
+ { label: t("computers.meter_disk"), pct: health.disk_used_pct },
+ ].filter((v): v is { label: string; pct: number } => v.pct !== null && v.pct >= TIGHT_PCT)
+ : [];
return (
- <li role="row">
- <button
- type="button"
- onClick={onOpen}
- data-testid={`computer-row-${computer.id}`}
- className={cn(
- "group grid w-full grid-cols-1 items-center gap-x-6 gap-y-2 px-5 py-3.5 text-left transition-colors",
- TABLE_COLUMNS,
- "hover:bg-secondary/50 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-ring",
- )}
- >
- <span className="flex min-w-0 items-center gap-3" role="cell">
- <ProviderLogo providerId={computer.provider} label={computer.name} />
- <span className="min-w-0">
- <span className="block truncate text-base font-medium text-foreground-strong">{computer.name}</span>
- <span className="block truncate text-xs text-muted-foreground">{providerName}</span>
- </span>
- </span>
- <span className="inline-flex min-w-0 items-center gap-2 text-sm text-foreground-secondary" role="cell">
- {checking || computer.busy ? (
- <Loader2 className="h-3 w-3 shrink-0 animate-spin text-muted-foreground" aria-hidden />
- ) : (
- <StatusLight tone={statusTone(health.status)} />
- )}
- <span className="truncate">{statusLabel(computer, t)}</span>
- {online && health.latency_ms !== null && (
- <span className="shrink-0 tabular-nums text-xs text-foreground-faint">{health.latency_ms} ms</span>
- )}
- </span>
- <span className="truncate font-mono text-xs text-muted-foreground" role="cell">
- {computer.username}@{computer.host === "0.0.0.0" ? "…" : computer.host}
- {computer.port !== 22 ? `:${computer.port}` : ""}
- </span>
- <span className="truncate text-sm text-muted-foreground" role="cell">
- {system || "—"}
- </span>
- <span className="grid min-w-0 grid-cols-3 gap-3" role="cell">
- <Meter compact label={t("computers.meter_cpu")} pct={online ? loadPct(computer) : null} />
- <Meter compact label={t("computers.meter_memory")} pct={online ? health.mem_used_pct : null} />
- <Meter compact label={t("computers.meter_disk")} pct={online ? health.disk_used_pct : null} />
+ <button
+ type="button"
+ onClick={onOpen}
+ data-testid={`computer-row-${computer.id}`}
+ className="group grid w-full grid-cols-[auto_minmax(0,1fr)_auto] items-center gap-x-3 px-3 py-2.5 text-left transition-colors hover:bg-secondary/50 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-inset focus-visible:ring-ring sm:px-4"
+ >
+ {CLOUD_MARK_PROVIDERS.has(computer.provider) ? (
+ <ComputersIcon aria-hidden className="h-4 w-4 text-muted-foreground" />
+ ) : (
+ <ProviderLogo providerId={computer.provider} label={computer.name} size="sm" />
+ )}
+ <span className="min-w-0">
+ <span className="block truncate text-base font-medium text-foreground">{computer.name}</span>
+ <span className="block truncate text-xs text-muted-foreground">
+ <span className="font-mono">SSH {login}</span>
+ <span aria-hidden> · </span>
+ <span className={cn(tone === "error" && "text-destructive", tone === "warn" && "text-warning")}>{status}</span>
+ {system && <span> · {system}</span>}
</span>
+ </span>
+ <span className="flex shrink-0 items-center gap-2.5">
+ {tight.map((v) => (
+ <span
+ key={v.label}
+ className={cn("hidden text-xs tabular-nums sm:inline", v.pct >= 90 ? "text-destructive" : "text-warning")}
+ >
+ {v.label} {Math.round(Math.min(100, v.pct))} %
+ </span>
+ ))}
+ {busy ? (
+ <Loader2 className="h-3 w-3 animate-spin text-muted-foreground" aria-hidden />
+ ) : (
+ <StatusDot tone={tone} label={status} />
+ )}
<ChevronRight
aria-hidden
- className="hidden h-4 w-4 text-foreground-faint transition-transform group-hover:translate-x-0.5 group-hover:text-foreground md:block"
+ className="h-4 w-4 text-foreground-faint transition-transform group-hover:translate-x-0.5 group-hover:text-foreground"
/>
- </button>
- </li>
+ </span>
+ </button>
);
}
diff --git a/jarvis/ui/web/frontend/src/views/computers/ConnectDialog.tsx b/jarvis/ui/web/frontend/src/views/computers/ConnectDialog.tsx
index d6bb3624a9..130777f646 100644
--- a/jarvis/ui/web/frontend/src/views/computers/ConnectDialog.tsx
+++ b/jarvis/ui/web/frontend/src/views/computers/ConnectDialog.tsx
@@ -1,5 +1,5 @@
/**
- * "Connect a computer" — one screen, and usually one field: the address.
+ * Connect a computer with explicit SSH host, username and port fields.
*
* The address box understands a bare IP, user@host, a whole ssh command, a
* block with a private key, or the one line our setup instructions end with,
@@ -29,10 +29,12 @@ import {
ChevronDown,
Copy,
FileKey,
+ Globe,
KeyRound,
Lock,
MonitorSmartphone,
Plug,
+ Plus,
SquareTerminal,
X,
} from "lucide-react";
@@ -50,6 +52,7 @@ import { CopyField, Field, inputClass } from "./parts";
import { ApiImportStep } from "./wizard/ApiImportStep";
import { LocalVmStep } from "./wizard/LocalVmStep";
import { errorText } from "./wizard/shared";
+import { ServerAddressForm } from "./ServerAddressForm";
type Screen = { kind: "form" } | { kind: "check" } | { kind: "account"; provider: ProviderInfo } | { kind: "vm" };
type Method = "auto" | "password" | "ssh_key" | "agent";
@@ -122,6 +125,9 @@ export function ConnectDialog({
const catalog = useProviderCatalog();
const identity = useIdentity();
const [screen, setScreen] = useState<Screen>({ kind: "form" });
+ const [connectionType, setConnectionType] = useState<"server" | "ssh">("ssh");
+ const connecting = useRef(false);
+ const [additionalOptions, setAdditionalOptions] = useState(false);
const assistantName = useEventStore((s) => s.assistantName) || "Jarvis";
const [pasted, setPasted] = useState("");
const [method, setMethod] = useState<Method>("auto");
@@ -175,10 +181,12 @@ export function ConnectDialog({
? "private_key"
: "key";
const effectiveUser = username.trim() || detected.user || "root";
- const effectivePort = Number(port) || detected.port || 22;
+ const effectivePort = port.trim() ? Number(port) : detected.port ?? 22;
+ const validPort = Number.isInteger(effectivePort) && effectivePort >= 1 && effectivePort <= 65535;
const keyText = pastedKey ?? privateKey;
const ready =
Boolean(detected.host) &&
+ validPort &&
(login === "key" ||
login === "auto" ||
(login === "password" ? password.length > 0 : keyText.trim().length > 0));
@@ -215,7 +223,17 @@ export function ConnectDialog({
}
async function connect() {
- if (!ready || !detected.host) return;
+ if (connectionType !== "ssh" || !ready || !detected.host || connecting.current) return;
+ connecting.current = true;
+ try {
+ await connectSsh();
+ } finally {
+ connecting.current = false;
+ }
+ }
+
+ async function connectSsh() {
+ if (!detected.host) return;
const input: AddServerInput = {
name: name.trim() || detected.host,
host: detected.host,
@@ -320,7 +338,7 @@ export function ConnectDialog({
aria-modal="true"
aria-label={t("computers.cx_title")}
data-testid="computers-connect-dialog"
- className="flex max-h-[90vh] w-full max-w-lg flex-col overflow-hidden rounded-xl border border-border bg-popover shadow-float"
+ className="flex max-h-[90vh] w-full max-w-2xl flex-col overflow-hidden rounded-xl border border-border bg-popover shadow-float"
>
<header className="flex items-start gap-3 border-b border-border px-6 py-5">
{screen.kind !== "form" && !(screen.kind === "check" && check?.computer) && (
@@ -336,7 +354,29 @@ export function ConnectDialog({
<div className="min-w-0 flex-1">
<h2 className="text-lg font-semibold text-foreground-strong">{title}</h2>
{screen.kind === "form" && (
- <p className="mt-0.5 text-sm text-muted-foreground">{t("computers.cx_subtitle")}</p>
+ <>
+ <p className="mt-0.5 text-sm text-muted-foreground">{t("computers.cx_connection_hint")}</p>
+ <fieldset className="mt-4 inline-flex max-w-full gap-1 rounded-xl border border-border bg-card p-1">
+ <legend className="sr-only">{t("computers.cx_connection_type")}</legend>
+ {(["server", "ssh"] as const).map((type) => (
+ <label key={type} className="relative min-w-0 cursor-pointer">
+ <input
+ type="radio"
+ name="computer-connection-type"
+ value={type}
+ checked={connectionType === type}
+ onChange={() => setConnectionType(type)}
+ aria-controls={`cx-${type}-panel`}
+ className="peer sr-only"
+ />
+ <span className="flex items-center justify-center gap-2 rounded-lg px-4 py-2 text-sm font-medium text-muted-foreground transition-colors hover:text-foreground peer-checked:bg-secondary peer-checked:text-foreground-strong peer-checked:shadow-sm peer-focus-visible:ring-2 peer-focus-visible:ring-ring">
+ {type === "server" ? <Globe className="h-4 w-4 shrink-0" aria-hidden /> : <SquareTerminal className="h-4 w-4 shrink-0" aria-hidden />}
+ {t(`computers.cx_connection_${type}`)}
+ </span>
+ </label>
+ ))}
+ </fieldset>
+ </>
)}
{screen.kind === "check" && check?.computer && (
<p className="mt-0.5 text-sm text-muted-foreground">{t("computers.cx_verified_body")}</p>
@@ -383,8 +423,12 @@ export function ConnectDialog({
/>
)}
+ {screen.kind === "form" && <div id="cx-server-panel" hidden={connectionType !== "server"}><ServerAddressForm /></div>}
+
{screen.kind === "form" && (
<form
+ id="cx-ssh-panel"
+ hidden={connectionType !== "ssh"}
className="space-y-5"
onSubmit={(e) => {
e.preventDefault();
@@ -392,9 +436,9 @@ export function ConnectDialog({
}}
>
<div>
- <Field label={t("computers.cx_paste")} hint={t("computers.cx_paste_hint")}>
+ <Field label={t("computers.cx_ssh_host")}>
<textarea
- className={cn(inputClass, "h-auto min-h-[40px] resize-none py-2 font-mono text-sm")}
+ className={cn(inputClass, "h-auto min-h-12 resize-none rounded-xl px-4 py-3")}
rows={pasted.includes("\n") ? 4 : 1}
value={pasted}
onChange={(e) => setPasted(e.target.value)}
@@ -404,14 +448,14 @@ export function ConnectDialog({
void connect();
}
}}
- placeholder="203.0.113.10"
+ placeholder={t("computers.cx_ssh_placeholder")}
autoFocus
spellCheck={false}
autoComplete="off"
data-testid="cx-address"
/>
</Field>
- {detected.host && (
+ {detected.host && (detected.user || detected.port || pastedKey || detected.fromSetupPrompt) && (
<div className="mt-2 flex flex-wrap items-center gap-1.5 text-xs" data-testid="cx-detected">
<span className="text-muted-foreground">{t("computers.cx_detected")}</span>
<span className="rounded bg-secondary px-1.5 py-0.5 font-mono text-foreground-secondary">
@@ -428,6 +472,44 @@ export function ConnectDialog({
)}
</div>
+ <div className="grid grid-cols-[minmax(0,1fr)_7rem] gap-4 sm:grid-cols-[minmax(0,1fr)_9rem]">
+ <Field label={t("computers.field_username")}>
+ <input
+ className={cn(inputClass, "h-12 rounded-xl px-4")}
+ value={username}
+ onChange={(e) => setUsername(e.target.value)}
+ placeholder={detected.user || "root"}
+ autoComplete="off"
+ spellCheck={false}
+ data-testid="cx-username"
+ />
+ </Field>
+ <Field label={t("computers.field_port")}>
+ <input
+ className={cn(inputClass, "h-12 rounded-xl px-4")}
+ value={port}
+ type="number"
+ min={1}
+ max={65535}
+ step={1}
+ inputMode="numeric"
+ onChange={(e) => setPort(e.target.value)}
+ placeholder={String(detected.port ?? 22)}
+ aria-invalid={!validPort}
+ aria-describedby={!validPort ? "cx-port-error" : undefined}
+ data-testid="cx-port"
+ />
+ </Field>
+ </div>
+ {!validPort && <p id="cx-port-error" role="alert" className="text-sm text-destructive">{t("computers.cx_invalid_port")}</p>}
+
+ {method === "auto" && !missing && (
+ <Button type="submit" variant="outline" className="h-12 w-full rounded-xl text-base" disabled={!ready} data-testid="cx-connect">
+ <Plus />
+ {t("computers.add")}
+ </Button>
+ )}
+
{method === "auto" && (
<div className="space-y-3" data-testid="cx-auto">
{missing === "key_only" ? (
@@ -453,12 +535,7 @@ export function ConnectDialog({
{t("computers.cx_key_only_agent")}
</button>
</div>
- ) : (
- <p className="flex items-start gap-2 text-sm text-muted-foreground">
- <KeyRound className="mt-0.5 h-4 w-4 shrink-0" aria-hidden />
- {t("computers.cx_auto_hint")}
- </p>
- )}
+ ) : null}
<button
type="button"
onClick={() => chooseMethod("password")}
@@ -697,7 +774,7 @@ export function ConnectDialog({
{t("computers.cx_more")}
</button>
{more && (
- <div className="mt-3 grid gap-3 sm:grid-cols-[minmax(0,1fr)_110px_80px]">
+ <div className="mt-3">
<Field label={t("computers.field_name")}>
<input
className={inputClass}
@@ -706,41 +783,34 @@ export function ConnectDialog({
placeholder={detected.host || t("computers.field_name_placeholder")}
/>
</Field>
- <Field label={t("computers.field_username")}>
- <input
- className={cn(inputClass, "font-mono")}
- value={username}
- onChange={(e) => setUsername(e.target.value)}
- placeholder={detected.user || "root"}
- spellCheck={false}
- />
- </Field>
- <Field label={t("computers.field_port")}>
- <input
- className={cn(inputClass, "font-mono")}
- value={port}
- inputMode="numeric"
- onChange={(e) => setPort(e.target.value.replace(/[^0-9]/g, ""))}
- placeholder={String(detected.port ?? 22)}
- />
- </Field>
</div>
)}
</div>
- <div>
+ {(method !== "auto" || missing) && <div>
<Button type="submit" className="h-10 w-full" disabled={!ready} data-testid="cx-connect">
<Plug />
{missing === "key_only" && method === "auto" ? t("computers.cx_try_again") : t("computers.cx_connect")}
</Button>
<p className="mt-2 text-center text-xs text-muted-foreground">{t("computers.cx_connect_hint")}</p>
- </div>
+ </div>}
</form>
)}
</div>
- {screen.kind === "form" && (
+ {screen.kind === "form" && connectionType === "ssh" && (
<footer className="space-y-3 border-t border-border bg-secondary/30 px-6 py-4">
+ <button
+ type="button"
+ onClick={() => setAdditionalOptions((value) => !value)}
+ aria-expanded={additionalOptions}
+ className="inline-flex items-center gap-1.5 text-sm text-muted-foreground hover:text-foreground"
+ data-testid="cx-additional-options"
+ >
+ <ChevronDown className={cn("h-4 w-4 transition-transform", additionalOptions && "rotate-180")} aria-hidden />
+ {t("computers.cx_additional_options")}
+ </button>
+ {additionalOptions && <>
{accounts.length > 0 && (
<div>
<div className="text-sm font-medium text-foreground-secondary">{t("computers.cx_accounts_title")}</div>
@@ -770,6 +840,7 @@ export function ConnectDialog({
<MonitorSmartphone className="h-3.5 w-3.5" aria-hidden />
{t("computers.cx_vm_link")}
</button>
+ </>}
</footer>
)}
</div>
diff --git a/jarvis/ui/web/frontend/src/views/computers/KeepWorking.tsx b/jarvis/ui/web/frontend/src/views/computers/KeepWorking.tsx
index 81e983064b..cf1bc69ba6 100644
--- a/jarvis/ui/web/frontend/src/views/computers/KeepWorking.tsx
+++ b/jarvis/ui/web/frontend/src/views/computers/KeepWorking.tsx
@@ -7,13 +7,15 @@
* app's connection.
*/
import { useEffect, useState } from "react";
-import { Loader2, MoonStar } from "lucide-react";
-import { Panel } from "@/components/extensions/primitives";
+import { Loader2 } from "lucide-react";
import { useT } from "@/i18n";
-import { cn } from "@/lib/utils";
import type { Computer } from "@/lib/computersApi";
+import { SettingsSelect } from "@/views/settings/SettingsLayout";
+import { Row } from "./surface";
const ENDPOINT = "/api/agentic-ide/offload-on-quit";
+/** The select's value for "no computer"; a computer id never takes this shape. */
+const OFF = "__off__";
export function KeepWorking({ computers }: { computers: Computer[] }) {
const t = useT();
@@ -62,50 +64,35 @@ export function KeepWorking({ computers }: { computers: Computer[] }) {
const isWindows = (computer: Computer) => computer.facts?.os_id === "windows";
const hasWindows = computers.some(isWindows);
const options = [
- { id: null as string | null, label: t("computers.keep_off") },
+ { value: OFF, label: t("computers.keep_off") },
...computers
.filter((computer) => computer.health.status !== "provisioning" && !isWindows(computer))
- .map((computer) => ({ id: computer.id as string | null, label: computer.name })),
+ .map((computer) => ({ value: computer.id, label: computer.name })),
];
return (
- <Panel className="p-5">
- <div className="flex items-start gap-3">
- <MoonStar className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" aria-hidden />
- <div className="min-w-0 flex-1">
- <div className="flex items-center gap-2">
- <span className="text-base font-semibold text-foreground-strong">{t("computers.keep_title")}</span>
- {saving && <Loader2 className="h-3.5 w-3.5 animate-spin text-muted-foreground" aria-hidden />}
- </div>
- <p className="mt-0.5 text-sm text-muted-foreground">{t("computers.keep_body")}</p>
- <div role="radiogroup" aria-label={t("computers.keep_title")} className="mt-4 flex flex-wrap gap-1.5" data-testid="computers-keep-working">
- {options.map((option) => {
- const active = loaded && option.id === value;
- return (
- <button
- key={option.id ?? "off"}
- type="button"
- role="radio"
- aria-checked={active}
- disabled={!loaded || saving}
- onClick={() => void choose(option.id)}
- className={cn(
- "h-8 rounded-md border px-3 text-sm font-medium transition-colors",
- "focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-60",
- active
- ? "border-accent bg-accent-soft text-foreground-strong"
- : "border-border text-muted-foreground hover:border-border-strong hover:text-foreground",
- )}
- >
- {option.label}
- </button>
- );
- })}
- </div>
- {hasWindows && <p className="mt-2 text-xs text-muted-foreground">{t("computers.keep_windows_note")}</p>}
- {error && <p role="alert" className="mt-2 text-sm text-destructive">{error}</p>}
- </div>
- </div>
- </Panel>
+ <Row
+ title={
+ <span className="inline-flex items-center gap-2">
+ {t("computers.keep_title")}
+ {saving && <Loader2 className="h-3.5 w-3.5 animate-spin text-muted-foreground" aria-hidden />}
+ </span>
+ }
+ description={t("computers.keep_body")}
+ status={hasWindows ? t("computers.keep_windows_note") : undefined}
+ control={
+ <SettingsSelect
+ value={value ?? OFF}
+ options={options}
+ onValueChange={(next) => void choose(next === OFF ? null : next)}
+ ariaLabel={t("computers.keep_title")}
+ disabled={!loaded || saving}
+ testId="computers-keep-working"
+ className="h-8 min-w-[12rem] justify-between border border-border bg-secondary/40"
+ />
+ }
+ >
+ {error && <p role="alert" className="pb-1 text-sm text-destructive">{error}</p>}
+ </Row>
);
}
diff --git a/jarvis/ui/web/frontend/src/views/computers/ServerAddressForm.tsx b/jarvis/ui/web/frontend/src/views/computers/ServerAddressForm.tsx
new file mode 100644
index 0000000000..7170f70e7b
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/computers/ServerAddressForm.tsx
@@ -0,0 +1,69 @@
+import { useState } from "react";
+import { Plus } from "lucide-react";
+import { Button } from "@/components/ui/button";
+import { useT } from "@/i18n";
+import { cn } from "@/lib/utils";
+import { Field, inputClass } from "./parts";
+
+/** Collect pairing details without sending them to the unrelated SSH API. */
+export function ServerAddressForm() {
+ const t = useT();
+ const [host, setHost] = useState("");
+ const [code, setCode] = useState("");
+
+ function readPairingUrl(value: string): boolean {
+ try {
+ const url = new URL(value.trim());
+ if (!["http:", "https:"].includes(url.protocol) || url.username || url.password) return false;
+ const pairingCode = url.searchParams.get("code") ?? url.searchParams.get("pairing_code");
+ if (!pairingCode?.trim()) return false;
+ setHost(url.origin);
+ setCode(pairingCode);
+ return true;
+ } catch {
+ // Plain host names are valid field input, but are not pairing URLs.
+ return false;
+ }
+ }
+
+ return (
+ <form className="space-y-5" onSubmit={(event) => event.preventDefault()} data-testid="cx-server-form">
+ <div className="grid gap-4 sm:grid-cols-[minmax(0,1fr)_10rem]">
+ <Field label={t("computers.cx_server_host")}>
+ <input
+ className={cn(inputClass, "h-12 rounded-xl px-4")}
+ value={host}
+ onChange={(event) => setHost(event.target.value)}
+ onBlur={() => readPairingUrl(host)}
+ onPaste={(event) => {
+ if (readPairingUrl(event.clipboardData.getData("text"))) event.preventDefault();
+ }}
+ placeholder="backend.example.com"
+ autoComplete="off"
+ spellCheck={false}
+ aria-describedby="cx-pairing-hint"
+ data-testid="cx-server-host"
+ />
+ </Field>
+ <Field label={t("computers.cx_pairing_code")}>
+ <input
+ className={cn(inputClass, "h-12 rounded-xl px-4")}
+ value={code}
+ onChange={(event) => setCode(event.target.value)}
+ placeholder="PAIRCODE"
+ autoComplete="off"
+ autoCapitalize="none"
+ spellCheck={false}
+ data-testid="cx-pairing-code"
+ />
+ </Field>
+ </div>
+ <p id="cx-pairing-hint" className="text-sm text-muted-foreground">{t("computers.cx_pairing_hint")}</p>
+ <Button type="submit" variant="outline" className="h-12 w-full rounded-xl text-base" disabled aria-describedby="cx-pairing-unavailable">
+ <Plus />
+ {t("computers.add")}
+ </Button>
+ <p id="cx-pairing-unavailable" className="text-sm text-muted-foreground">{t("computers.cx_pairing_unavailable")}</p>
+ </form>
+ );
+}
diff --git a/jarvis/ui/web/frontend/src/views/computers/parts.tsx b/jarvis/ui/web/frontend/src/views/computers/parts.tsx
index 7f360f9381..a8d8563f23 100644
--- a/jarvis/ui/web/frontend/src/views/computers/parts.tsx
+++ b/jarvis/ui/web/frontend/src/views/computers/parts.tsx
@@ -222,3 +222,4 @@ export function Field({
export const inputClass =
"h-9 w-full rounded-md border border-border-strong bg-input px-3 text-base text-foreground placeholder:text-foreground-faint transition-colors focus:border-accent focus:outline-none focus:ring-2 focus:ring-ring";
+
diff --git a/jarvis/ui/web/frontend/src/views/computers/surface.tsx b/jarvis/ui/web/frontend/src/views/computers/surface.tsx
new file mode 100644
index 0000000000..df63774b3e
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/computers/surface.tsx
@@ -0,0 +1,147 @@
+/**
+ * Portions adapted from pingdotgg/t3code @ 12069ee (apps/web settingsLayout,
+ * SettingsGroup, EnvironmentRow, ConnectionStatusDot, ui/empty), MIT License,
+ * Copyright (c) 2026 T3 Tools Inc. Full text: third_party/t3code/LICENSE.
+ *
+ * The surface of the Computers page: a muted section heading with an action
+ * on its right, one grouped card whose rows are split by hairlines, a setting
+ * row (title, one short line, control), a machine row (glyph, name, one
+ * subtitle line, controls), the status dot and the empty state with its
+ * fanned icon tiles. Theme tokens only, so light and dark need no second path.
+ */
+import type { ReactNode } from "react";
+import { cn } from "@/lib/utils";
+import type { Tone } from "./parts";
+
+/** A muted heading over one grouped card. Explanatory copy belongs to the rows. */
+export function Section({
+ title,
+ action,
+ children,
+ className,
+ testId,
+}: {
+ title: string;
+ /** Right of the heading: a small text action such as "Add computer". */
+ action?: ReactNode;
+ children: ReactNode;
+ className?: string;
+ testId?: string;
+}) {
+ return (
+ <section className={cn("space-y-2.5", className)} data-testid={testId}>
+ <div className="flex min-h-7 items-center justify-between gap-4 px-3 sm:px-4">
+ <h2 className="text-base font-normal text-foreground-secondary">{title}</h2>
+ {action && <div className="flex min-h-7 items-center justify-end gap-1">{action}</div>}
+ </div>
+ <Group>{children}</Group>
+ </section>
+ );
+}
+
+/** The grouped card: rounded, a soft rim, hairlines between its rows. */
+export function Group({ children, className }: { children: ReactNode; className?: string }) {
+ return (
+ <div
+ className={cn(
+ "overflow-hidden rounded-xl border border-border bg-card/60 text-foreground",
+ "[&>*+*]:border-t [&>*+*]:border-border",
+ className,
+ )}
+ >
+ {children}
+ </div>
+ );
+}
+
+/** One setting: title and a short line on the left, its control on the right. */
+export function Row({
+ title,
+ description,
+ status,
+ control,
+ children,
+ testId,
+}: {
+ title: ReactNode;
+ description?: ReactNode;
+ /** A quiet extra line under the description: a note, a fingerprint. */
+ status?: ReactNode;
+ control?: ReactNode;
+ children?: ReactNode;
+ testId?: string;
+}) {
+ return (
+ <div data-testid={testId} className={cn("px-3 sm:px-4", children ? "pb-2 pt-3" : "py-3")}>
+ <div className="flex flex-col gap-3 sm:grid sm:grid-cols-[minmax(0,1fr)_auto] sm:items-center sm:gap-8">
+ <div className="min-w-0 space-y-1">
+ <h3 className="flex min-h-5 items-center gap-1.5 text-base font-medium text-foreground-strong">{title}</h3>
+ {description && <p className="max-w-xl text-sm leading-normal text-muted-foreground">{description}</p>}
+ {status && <div className="pt-0.5 text-xs text-foreground-faint">{status}</div>}
+ </div>
+ {control && <div className="flex min-w-0 shrink-0 items-center gap-2 sm:justify-end">{control}</div>}
+ </div>
+ {children}
+ </div>
+ );
+}
+
+const DOT: Record<Tone, string> = {
+ ok: "bg-success",
+ busy: "bg-warning",
+ warn: "bg-warning",
+ error: "bg-destructive",
+ off: "bg-foreground-faint",
+};
+
+/** The connection dot: a halo pings only while the state is in motion. */
+export function StatusDot({ tone, label }: { tone: Tone; label?: string }) {
+ return (
+ <span className="relative flex h-3 w-3 shrink-0 items-center justify-center" title={label} aria-label={label}>
+ {tone === "busy" && (
+ <span className="absolute inline-flex h-full w-full rounded-full bg-warning/60 motion-safe:animate-ping" />
+ )}
+ <span className={cn("relative inline-flex h-2 w-2 rounded-full", DOT[tone])} />
+ </span>
+ );
+}
+
+const MEDIA_TILE =
+ "flex h-9 w-9 items-center justify-center rounded-md border border-border bg-card text-foreground shadow-sm [&_svg]:h-[18px] [&_svg]:w-[18px]";
+
+/** A centred empty state: three fanned icon tiles, a title, one or two lines. */
+export function Empty({
+ icon,
+ title,
+ description,
+ children,
+ testId,
+}: {
+ icon: ReactNode;
+ title: string;
+ description?: ReactNode;
+ children?: ReactNode;
+ testId?: string;
+}) {
+ return (
+ <div
+ data-testid={testId}
+ className="flex min-h-[13rem] min-w-0 flex-col items-center justify-center gap-0 text-balance px-6 py-10 text-center"
+ >
+ <div className="relative mb-6" aria-hidden>
+ <div className={cn(MEDIA_TILE, "absolute bottom-px origin-bottom-left -translate-x-0.5 -rotate-[10deg] scale-[0.84] shadow-none")} />
+ <div className={cn(MEDIA_TILE, "absolute bottom-px origin-bottom-right translate-x-0.5 rotate-[10deg] scale-[0.84] shadow-none")} />
+ <div className={cn(MEDIA_TILE, "relative")}>{icon}</div>
+ </div>
+ <div className="flex max-w-sm flex-col items-center">
+ <div className="text-lg font-semibold text-foreground-strong">{title}</div>
+ {description && <div className="mt-1 text-sm text-muted-foreground">{description}</div>}
+ </div>
+ {children}
+ </div>
+ );
+}
+
+/** Small ghost text action for a section heading: icon plus a short label. */
+export const sectionActionCls =
+ "inline-flex h-7 items-center gap-1.5 rounded-md px-2 text-sm font-medium text-muted-foreground transition-colors hover:bg-secondary hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring disabled:opacity-50 [&>svg]:h-3.5 [&>svg]:w-3.5";
diff --git a/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.server.test.tsx b/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.server.test.tsx
new file mode 100644
index 0000000000..b3f5cd7a30
--- /dev/null
+++ b/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.server.test.tsx
@@ -0,0 +1,60 @@
+import { fireEvent, render, screen, waitFor } from "@testing-library/react";
+import { afterEach, expect, it, vi } from "vitest";
+
+const restart = vi.hoisted(() => vi.fn());
+vi.mock("@/hooks/useRestartApp", () => ({
+ useRestartApp: () => ({ restart, restarting: false, buttonLabel: "Restart application" }),
+}));
+
+vi.mock("@/hooks/useTheme", () => ({
+ useTheme: () => ({ preference: "dark", setPreference: vi.fn() }),
+}));
+vi.mock("@/hooks/useAutostart", () => ({
+ useAutostart: () => ({ config: null, loading: true, save: vi.fn() }),
+}));
+
+import { AppSettingsGroup } from "./AppSettingsGroup";
+
+afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); });
+
+it("enables an independent server and saves its destination without stopping live work", async () => {
+ let config = {
+ supported: true, keep_agents_running: true, persistent_server: false, server_url: "",
+ running_as_server: false, autostart_enabled: false, running_as_service: false,
+ background_only_at_login: false, work: { routines: 1, running: 1, channels: [] },
+ };
+ const requests: Array<{ path: string; method: string; body: unknown }> = [];
+ vi.stubGlobal("fetch", vi.fn(async (path: string, options?: RequestInit) => {
+ const body = options?.body ? JSON.parse(String(options.body)) : null;
+ requests.push({ path, method: options?.method ?? "GET", body });
+ if (options?.method === "PUT") config = { ...config, ...body };
+ return { ok: true, json: async () => config };
+ }));
+ render(<AppSettingsGroup />);
+ const toggle = await screen.findByRole("switch", { name: "Independent agent server" });
+ await waitFor(() => expect(toggle.getAttribute("disabled")).toBeNull());
+ fireEvent.click(toggle);
+ const field = await screen.findByLabelText("Server address (leave empty to run on this computer)");
+ fireEvent.change(field, { target: { value: "https://jarvis.example.com" } });
+ fireEvent.click(screen.getByRole("button", { name: "Save address" }));
+ await waitFor(() => expect(config.server_url).toBe("https://jarvis.example.com"));
+ expect(config.persistent_server).toBe(true);
+ expect(requests.filter((r) => r.method === "PUT")).toEqual([
+ { path: "/api/settings/background", method: "PUT", body: { persistent_server: true } },
+ { path: "/api/settings/background", method: "PUT", body: { server_url: "https://jarvis.example.com" } },
+ ]);
+ expect(requests.every((r) => !r.path.includes("stop") && !r.path.includes("restart"))).toBe(true);
+});
+
+it("offers a real restart instead of silently saving to a backend without server support", async () => {
+ const fetcher = vi.fn(async () => ({ ok: true, json: async () => ({
+ supported: true, keep_agents_running: true, work: { routines: 0, channels: [] },
+ }) }));
+ vi.stubGlobal("fetch", fetcher);
+ render(<AppSettingsGroup />);
+ const button = await screen.findByRole("button", { name: "Restart application" });
+ expect(screen.queryByRole("switch", { name: "Independent agent server" })).toBeNull();
+ fireEvent.click(button);
+ expect(restart).toHaveBeenCalledOnce();
+ expect(fetcher.mock.calls.every((args) => args.length < 2)).toBe(true);
+});
diff --git a/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.tsx b/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.tsx
index d23bd2cd34..ad100e292f 100644
--- a/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.tsx
+++ b/jarvis/ui/web/frontend/src/views/settings/AppSettingsGroup.tsx
@@ -2,9 +2,11 @@ import { useState, type ReactNode } from "react";
import { Monitor, Moon, Sun, Zap } from "lucide-react";
import { SETUP_REPLAY_EVENT } from "@/components/onboarding/tourEvents";
import { Button } from "@/components/ui/button";
+import { Input } from "@/components/ui/input";
import { Switch } from "@/components/ui/switch";
import { useAutostart } from "@/hooks/useAutostart";
import { useBackgroundAgents, type BackgroundAgentsPatch } from "@/hooks/useBackgroundAgents";
+import { useRestartApp } from "@/hooks/useRestartApp";
import { useTheme, type ThemePreference } from "@/hooks/useTheme";
import { useEventStore } from "@/store/events";
import { fill, useLocaleChunk, useT } from "@/i18n";
@@ -255,6 +257,8 @@ function BackgroundAgentsRows() {
const { config, loading, error, save } = useBackgroundAgents();
const pushToast = useEventStore((s) => s.pushToast);
const [saving, setSaving] = useState(false);
+ const [serverDraft, setServerDraft] = useState<string | null>(null);
+ const restartApp = useRestartApp();
if (config && !config.supported) return null;
@@ -280,7 +284,7 @@ function BackgroundAgentsRows() {
return (
<>
- <SettingsRow
+ {!config?.persistent_server && <SettingsRow
title={t("settings_view.background_agents.title")}
description={t("settings_view.background_agents.description")}
control={
@@ -298,7 +302,9 @@ function BackgroundAgentsRows() {
{config?.running_as_service && (
<p className="text-sm text-foreground">
- {t("settings_view.background_agents.service_active")}
+ {t(config.running_as_server
+ ? "settings_view.background_agents.server_active"
+ : "settings_view.background_agents.service_active")}
</p>
)}
@@ -311,7 +317,7 @@ function BackgroundAgentsRows() {
)}
</>
)}
- </SettingsRow>
+ </SettingsRow>}
<SettingsRow
title={t("settings_view.background_agents.login_title")}
description={t("settings_view.background_agents.login_description")}
@@ -324,6 +330,58 @@ function BackgroundAgentsRows() {
/>
}
/>
+ <SettingsRow
+ title={t("settings_view.background_agents.server_title")}
+ description={t(config && typeof config.persistent_server !== "boolean"
+ ? "settings_view.background_agents.server_backend_update"
+ : "settings_view.background_agents.server_description")}
+ control={
+ config && typeof config.persistent_server !== "boolean" ? (
+ <Button variant="outline" disabled={restartApp.restarting}
+ onClick={() => void restartApp.restart()}>
+ {restartApp.buttonLabel}
+ </Button>
+ ) : <Switch
+ checked={config?.persistent_server ?? false}
+ disabled={loading || saving}
+ aria-label={t("settings_view.background_agents.server_title")}
+ onCheckedChange={(next) => void apply({ persistent_server: next })}
+ />
+ }
+ >
+ {config?.running_as_server && (
+ <p className="text-sm text-foreground">
+ {t("settings_view.background_agents.server_active")}
+ </p>
+ )}
+ {config?.persistent_server && (
+ <form className="space-y-2" onSubmit={(event) => {
+ event.preventDefault();
+ void apply({ server_url: (serverDraft ?? config.server_url).trim() });
+ }}>
+ <label className="text-sm text-muted-foreground" htmlFor="agent-server-url">
+ {t("settings_view.background_agents.server_url_label")}
+ </label>
+ <div className="flex flex-wrap items-center gap-2">
+ <Input
+ id="agent-server-url"
+ type="url"
+ autoComplete="off"
+ value={serverDraft ?? config.server_url ?? ""}
+ placeholder="https://jarvis.example.com"
+ className="min-w-56 flex-1"
+ onChange={(event) => setServerDraft(event.target.value)}
+ />
+ <Button type="submit" variant="outline" disabled={saving}>
+ {t("settings_view.background_agents.server_save")}
+ </Button>
+ </div>
+ <p className="text-sm text-muted-foreground">
+ {t("settings_view.background_agents.server_hint")}
+ </p>
+ </form>
+ )}
+ </SettingsRow>
</>
);
}
diff --git a/jarvis/ui/web/launcher.py b/jarvis/ui/web/launcher.py
index 80abb94f57..e244faa79e 100644
--- a/jarvis/ui/web/launcher.py
+++ b/jarvis/ui/web/launcher.py
@@ -131,6 +131,14 @@ def _parse_args(argv: list[str]) -> argparse.Namespace:
dest="after_pid",
help=argparse.SUPPRESS,
)
+ p.add_argument(
+ "--persistent-server", action="store_true",
+ help="Run the independent agent server; clients never take it over or stop it.",
+ )
+ p.add_argument(
+ "--connect", nargs="?", const="local", default=None, metavar="URL",
+ help="Open a client of an HTTPS server, or the independent local agent server.",
+ )
p.add_argument(
"--instance",
default=None,
@@ -611,7 +619,7 @@ async def _run_headless(args) -> int:
if _service_lock is None:
_headless_lock = _claim_headless_primary_lock(args, port=_port)
else:
- _write_service_sidecar(_port)
+ _write_service_sidecar(_port, persistent=getattr(args, "persistent_server", False))
_lx_mark("lock")
from jarvis.brain.factory import build_default_brain
@@ -914,6 +922,16 @@ async def _run_headless(args) -> int:
asyncio.create_task(_autostart_mcps())
stop_event = asyncio.Event()
+ server.app.state.agent_server_mode = bool(getattr(args, "persistent_server", False))
+ stop_loop = asyncio.get_running_loop()
+ server.app.state.agent_server_stop = lambda: stop_loop.call_soon_threadsafe(stop_event.set)
+ agent_boot = None
+ if server.app.state.agent_server_mode:
+ from jarvis.ui.web.agent_server import start_agents
+
+ agent_boot = asyncio.create_task(
+ start_agents(server.app.state), name="agent-server-restore",
+ )
def _stop(*_):
stop_event.set()
@@ -928,7 +946,7 @@ async def _run_headless(args) -> int:
_service_watch = None
if _service_lock is not None:
_service_watch, _service_tray = _start_background_service_duties(
- server.app.state, stop_event
+ server.app.state, stop_event, persistent=getattr(args, "persistent_server", False)
)
# Show the actual bind host (JARVIS_BIND_HOST may be 0.0.0.0 on a VPS);
@@ -944,6 +962,9 @@ async def _run_headless(args) -> int:
# A wedged teardown must not keep a windowless process holding the
# lock the reopening desktop is waiting for.
_arm_service_exit_backstop(SERVICE_SHUTDOWN_BACKSTOP_S)
+ if agent_boot is not None:
+ agent_boot.cancel()
+ await asyncio.gather(agent_boot, return_exceptions=True)
if _service_watch is not None:
_service_watch.cancel()
if _service_tray is not None:
@@ -1062,7 +1083,7 @@ def _prepare_background_service(args):
if not wait_for_pid_exit(int(after), timeout=bg.PARENT_EXIT_WAIT_S):
logger.warning("background: desktop pid={} never exited — service not started", after)
return None
- if bg.handover_requested():
+ if not getattr(args, "persistent_server", False) and bg.handover_requested():
logger.info("background: the desktop app is opening again — service not started")
return None
from jarvis.ui.desktop_app import SingleInstanceError, acquire_single_instance_lock
@@ -1076,21 +1097,21 @@ def _prepare_background_service(args):
# Findable the instant it holds the lock: a desktop reopened during this
# boot must ask for a hand-back, never read the holder as a stuck app.
try:
- bg.write_marker(0)
+ bg.write_marker(0, persistent=getattr(args, "persistent_server", False))
except OSError as exc:
logger.warning("background: service marker not written: {}", exc)
logger.info("background: agent service starting (pid={})", os.getpid())
return lock
-def _write_service_sidecar(port: int) -> None:
+def _write_service_sidecar(port: int, *, persistent: bool = False) -> None:
"""The service's marker plus the instance sidecar a desktop launch reads."""
from loguru import logger
from jarvis.core import background_service as bg
try:
- bg.write_marker(port)
+ bg.write_marker(port, persistent=persistent)
except OSError as exc:
logger.warning("background: service marker not written: {}", exc)
try:
@@ -1101,14 +1122,15 @@ def _write_service_sidecar(port: int) -> None:
logger.opt(exception=True).warning("background: instance sidecar not written")
-def _start_background_service_duties(state, stop_event):
+def _start_background_service_duties(state, stop_event, *, persistent: bool = False):
"""Start the handover/idle watch and the tray. Returns ``(task, tray)``."""
from loguru import logger
from jarvis.ui.background_tray import BackgroundTray, open_desktop_app
loop = asyncio.get_running_loop()
- task = loop.create_task(
+ # Future requests must remain possible even with no clients or scheduled work.
+ task = None if persistent else loop.create_task(
_watch_background_service(state, stop_event), name="background-service-watch"
)
@@ -1855,7 +1877,10 @@ def _recover_from_already_running(
def _take_over_from_background_service(*, expected_pid: int | None = None):
"""The lock, handed back by a running background service — else ``None``."""
try:
- from jarvis.core.background_service import service_pid, take_over_from_service
+ from jarvis.core.background_service import read_marker, service_pid, take_over_from_service
+
+ if (read_marker() or {}).get("persistent"):
+ return None
from jarvis.ui import desktop_app as _desktop_app
def find_service():
@@ -1884,6 +1909,21 @@ def _run_desktop(cfg, use_lock: bool) -> int:
``main()`` has already established that pywebview is importable
(``_missing_window_toolkit``), so the window layer can be built here.
"""
+ from jarvis.core.background_service import read_marker, service_pid
+
+ section = getattr(cfg, "background", None)
+ independent = bool(getattr(section, "persistent_server", False))
+ server_url = str(getattr(section, "server_url", "") or "")
+ existing_server = bool((read_marker() or {}).get("persistent") and service_pid())
+ if independent or server_url or existing_server:
+ from jarvis.ui.server_client import run
+
+ try:
+ return run(server_url or None, port=cfg.ui.admin_api_port)
+ except (RuntimeError, ValueError) as exc:
+ _report_startup_failure(str(exc))
+ return 4
+
# AUMID must be set before pywebview creates the window (taskbar icon).
_ensure_windows_app_identity()
from jarvis.ui.desktop_app import (
@@ -2279,6 +2319,8 @@ def _main(argv: list[str] | None = None) -> int:
)
args = _parse_args(_raw_argv)
+ if args.persistent_server:
+ args.background_service = True
if args.background_service:
args.headless = True
@@ -2397,6 +2439,22 @@ def _main(argv: list[str] | None = None) -> int:
_exit_background_service(code)
return code
+ # A client needs the same toolkit, privilege and branded-launch guards as
+ # the integrated window. It must only skip backend construction, not those
+ # device-side prerequisites (an elevated window rejects normal text input).
+ if args.connect is not None:
+ from jarvis.ui.server_client import run
+
+ _ensure_windows_app_identity()
+ try:
+ return run(
+ None if args.connect == "local" else args.connect,
+ port=args.port or _fast_admin_port(),
+ )
+ except (RuntimeError, ValueError) as exc:
+ _report_startup_failure(str(exc))
+ return 4
+
# Desktop boot: CLASSIC path (proven + GUI-safe). The serve-first bootstrap
# + static-shell + boot-splash (the black-screen fix) live in
# ``DesktopApp._run_backend``, so this path still opens with the real UI
diff --git a/jarvis/ui/web/runtime_gateway_routes.py b/jarvis/ui/web/runtime_gateway_routes.py
index f092320254..609c4372dc 100644
--- a/jarvis/ui/web/runtime_gateway_routes.py
+++ b/jarvis/ui/web/runtime_gateway_routes.py
@@ -10,6 +10,7 @@ OpenAPI schema.
from __future__ import annotations
+import math
from typing import Any
from fastapi import APIRouter, HTTPException, Request
@@ -50,6 +51,7 @@ async def runtime_gateway_chat(request: Request) -> Any:
"""One model call for a Hermes / OpenClaw agent, on its Jarvis provider."""
grant = _grant(request)
try:
+ lease = gateway.capture_turn(grant)
if grant.provider == gateway.SUBSCRIPTION_PROVIDER:
raise gateway.GatewayError("This agent's subscription answers on /responses.")
body = await _body(request)
@@ -59,13 +61,13 @@ async def runtime_gateway_chat(request: Request) -> Any:
temperature_given = gateway.temperature_given(body)
if body.get("stream") is True:
chunks = await gateway.open_chat_stream(
- grant, model, brain_request, temperature_given=temperature_given
+ grant, model, brain_request, temperature_given=temperature_given, lease=lease
)
return StreamingResponse(
chunks, media_type="text/event-stream", headers={"Cache-Control": "no-cache"}
)
return await gateway.complete_chat(
- grant, model, brain_request, temperature_given=temperature_given
+ grant, model, brain_request, temperature_given=temperature_given, lease=lease
)
except gateway.GatewayError as exc:
return _error(exc)
@@ -76,19 +78,20 @@ async def runtime_gateway_responses(request: Request) -> Any:
"""One model turn for a Hermes / OpenClaw agent, on its ChatGPT subscription."""
grant = _grant(request)
try:
+ lease = gateway.capture_turn(grant)
if grant.provider != gateway.SUBSCRIPTION_PROVIDER:
raise gateway.GatewayError("This agent's provider answers on /chat/completions.")
body = await _body(request)
args = gateway.request_args(body)
gateway.check_model(grant, args["model"])
if body.get("stream") is True:
- events = await gateway.open_response_stream(grant, args)
+ events = await gateway.open_response_stream(grant, args, lease=lease)
return StreamingResponse(
events,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache"},
)
- return await gateway.complete_response(grant, args)
+ return await gateway.complete_response(grant, args, lease=lease)
except gateway.GatewayError as exc:
return _error(exc)
diff --git a/jarvis/ui/web/server.py b/jarvis/ui/web/server.py
index 3e1b161526..620494b44f 100644
--- a/jarvis/ui/web/server.py
+++ b/jarvis/ui/web/server.py
@@ -756,6 +756,9 @@ class WebServer:
app.state.agent_chat = None
app.state.agent_chat_factory = self._build_agent_chat_service
app.include_router(agent_chat_router)
+ from .agent_server_routes import router as agent_server_router
+
+ app.include_router(agent_server_router)
# Agent society (jarvis/society) — the roster, the typed board, the
# scheduler and the mission bridge. Built on the first /api/society
# call from the factory (store = data/society.db); nothing opens on
@@ -4008,6 +4011,18 @@ class WebServer:
async def stop(self) -> None:
from jarvis.core import runtime_refs
+ appshot_task = getattr(self, "_appshot_shortcut_task", None)
+ if appshot_task is not None:
+ appshot_task.cancel()
+ await asyncio.gather(appshot_task, return_exceptions=True)
+ self._appshot_shortcut_task = None
+ try:
+ from jarvis.appshot.hotkey import stop_appshot_shortcut
+
+ await stop_appshot_shortcut()
+ except Exception:
+ logger.opt(exception=True).warning("Appshot helpers failed to stop")
+
if runtime_refs.get_web_app() is self.app:
runtime_refs.set_web_app(None)
# A cancelled setup waiter cannot stop its worker thread. Join it
diff --git a/jarvis/ui/web/settings_routes.py b/jarvis/ui/web/settings_routes.py
index c1b7a198cc..42cefbd33d 100644
--- a/jarvis/ui/web/settings_routes.py
+++ b/jarvis/ui/web/settings_routes.py
@@ -2335,6 +2335,16 @@ async def put_autostart(body: AutostartBody, request: Request) -> dict[str, obje
class BackgroundBody(BaseModel):
keep_agents_running: bool | None = Field(default=None)
background_only_at_login: bool | None = Field(default=None)
+ persistent_server: bool | None = None
+ server_url: str | None = Field(default=None, max_length=2048)
+
+ @model_validator(mode="after")
+ def check_server(self) -> BackgroundBody:
+ if self.server_url and self.server_url.strip():
+ from jarvis.core.server_endpoint import server_endpoint
+
+ self.server_url = server_endpoint(self.server_url)
+ return self
def _background_payload(request: Request) -> dict[str, object]:
@@ -2345,15 +2355,23 @@ def _background_payload(request: Request) -> dict[str, object]:
cfg = _config(request)
autostart_cfg = getattr(cfg, "autostart", None)
+ background_cfg = getattr(cfg, "background", None)
work = work_from_state(request.app.state)
return {
"keep_agents_running": keep_running_enabled(cfg),
+ "persistent_server": bool(getattr(background_cfg, "persistent_server", False)),
+ "server_url": str(getattr(background_cfg, "server_url", "") or ""),
+ "running_as_server": bool(getattr(request.app.state, "agent_server_mode", False)),
"background_only_at_login": bool(getattr(autostart_cfg, "background_only", False)),
"autostart_enabled": bool(getattr(autostart_cfg, "enabled", True)),
# This process IS the windowless service (a browser on its port).
- "running_as_service": SERVICE_FLAG in _sys.argv,
+ "running_as_service": SERVICE_FLAG in _sys.argv or bool(
+ getattr(request.app.state, "agent_server_mode", False)
+ ),
# Only the default app hands off; a dev instance stops with its window.
- "supported": current_instance().owns_ambient_duties,
+ "supported": current_instance().owns_ambient_duties or bool(
+ getattr(request.app.state, "agent_server_mode", False)
+ ),
"work": {
"routines": work.routines,
"running": work.running,
@@ -2372,6 +2390,22 @@ async def put_background(body: BackgroundBody, request: Request) -> dict[str, ob
from jarvis.core import config_writer
cfg = _config(request)
+ if body.persistent_server is not None or body.server_url is not None:
+ section = getattr(cfg, "background", None)
+ persistent = (
+ body.persistent_server if body.persistent_server is not None
+ else bool(getattr(section, "persistent_server", False))
+ )
+ origin = (
+ body.server_url if body.server_url is not None
+ else str(getattr(section, "server_url", "") or "")
+ )
+ await asyncio.to_thread(
+ config_writer.set_agent_server, persistent=persistent, server_url=origin,
+ )
+ if section is not None:
+ section.persistent_server = persistent
+ section.server_url = origin
if body.keep_agents_running is not None:
value = bool(body.keep_agents_running)
config_writer.set_background_keep_running(value)
@@ -2390,6 +2424,10 @@ async def put_background(body: BackgroundBody, request: Request) -> dict[str, ob
autostart_cfg.background_only = value # type: ignore[attr-defined]
except Exception as exc: # noqa: BLE001 — same as above
log.debug("in-memory autostart.background_only update skipped: %s", exc)
+ if (
+ body.background_only_at_login is not None or body.persistent_server is not None
+ or body.server_url is not None
+ ):
enabled, _caps, manager, spec = _autostart_components(request)
if enabled:
# The login entry carries the mode in its command line; refresh it
diff --git a/jarvis/ui/web/society_routes.py b/jarvis/ui/web/society_routes.py
index 8bb8963568..710ec3249a 100644
--- a/jarvis/ui/web/society_routes.py
+++ b/jarvis/ui/web/society_routes.py
@@ -93,7 +93,11 @@ def _validated_chat_runner(
account_id: str | None = None,
) -> str:
"""Reject a runner change that cannot honor the effective chat approval."""
- from jarvis.agent_chat.permissions import normalize_permission, society_mode_supported
+ from jarvis.agent_chat.permissions import (
+ default_permission,
+ normalize_permission,
+ society_mode_supported,
+ )
from jarvis.agent_chat.service import resolve_runner
chosen_runtime = str(agent.runtime)
@@ -111,6 +115,9 @@ def _validated_chat_runner(
mode = approval_mode if approval_mode is not None else (
str(agent.approval_mode) if agent.approval_mode is not None else ""
)
+ if not mode and agent.agent_id == rt.lead_id:
+ # The lead uses the app's chat policy; NULL is not a legacy Ask choice.
+ mode = default_permission("jarvis")
if mode and not society_mode_supported(runner, mode):
raise HTTPException(422, "This runner cannot provide an actionable approval for that mode.")
ceiling = str(
@@ -541,8 +548,11 @@ async def agent_conversations(agent_id: str, request: Request) -> dict[str, Any]
row["running"] = svc.is_running(sid)
# An approval card or a question waits for the person in this chat.
questions = getattr(svc, "pending_questions", None)
+ credentials = getattr(svc, "pending_credential_requests", None)
row["waiting"] = bool(
- svc.pending_approvals(sid) or (questions(sid) if callable(questions) else [])
+ svc.pending_approvals(sid)
+ or (questions(sid) if callable(questions) else [])
+ or (credentials(sid) if callable(credentials) else [])
)
row["owner_id"] = owner_id
row["counterpart_id"] = other_id
@@ -564,6 +574,42 @@ async def archive_agent(agent_id: str, request: Request) -> dict[str, Any]:
return {"agent": archived.to_dict()}
+ "/agents/{agent_id}/credentials",
+ summary="List the credentials an agent stored (names and labels, never values)",
+)
+async def list_agent_credentials(agent_id: str, request: Request) -> dict[str, Any]:
+ from jarvis.society.credentials import vault_for
+
+ rt = await _runtime(request)
+ agent = await rt.roster.resolve(agent_id)
+ if agent is None:
+ raise HTTPException(404, {"reason": str(FailureReason.TARGET_UNKNOWN)})
+ rows = await asyncio.to_thread(vault_for(rt.data_dir).list, agent.agent_id)
+ return {"agent_id": agent.agent_id, "credentials": [row.to_dict() for row in rows]}
+
+
+ "/agents/{agent_id}/credentials/{env}",
+ summary="Delete one of an agent's stored credentials",
+ openapi_extra={"x-jarvis-dangerous": True},
+)
+async def delete_agent_credential(agent_id: str, env: str, request: Request) -> dict[str, Any]:
+ from jarvis.society.credentials import CredentialError, vault_for
+
+ rt = await _runtime(request)
+ agent = await rt.roster.resolve(agent_id)
+ if agent is None:
+ raise HTTPException(404, {"reason": str(FailureReason.TARGET_UNKNOWN)})
+ try:
+ deleted = await asyncio.to_thread(vault_for(rt.data_dir).delete, agent.agent_id, env)
+ except CredentialError as exc:
+ raise HTTPException(500, str(exc)) from exc
+ if not deleted:
+ raise HTTPException(404, "no such credential")
+ return {"ok": True, "env": env}
+
+
async def _valid_group_members(rt: SocietyRuntime, members: list[str]) -> list[str]:
if len(set(members)) != len(members):
raise HTTPException(422, "group members must be unique")
@@ -1900,9 +1946,8 @@ async def install_template(template: dict[str, Any], request: Request) -> dict[s
"""Create a NEW agent from a template on this install's own model.
Shared by the import route below and the marketplace's install-by-name.
- The agent asks before it acts where its runner can (``approval_mode=ask``):
- its instructions were written by somebody else, so the person sees its
- first moves before trusting it with more.
+ Use the normal creation defaults, including Bypass permissions. Explicit
+ template approval rules and any creator restrictions still apply.
"""
from jarvis.society.agent_template import TemplateError, create_fields
@@ -1914,15 +1959,8 @@ async def install_template(template: dict[str, Any], request: Request) -> dict[s
wanted = fields.pop("name")
name = await _free_agent_name(rt, wanted)
scope = template.get("knowledge_scope", "shared")
- body = CreateAgentBody(name=name, approval_mode="ask", **fields)
- try:
- created = await create_agent(body, request)
- except HTTPException as exc:
- if exc.status_code != 422:
- raise
- # This runner cannot hold an approval prompt: the app's default applies.
- body = CreateAgentBody(name=name, **fields)
- created = await create_agent(body, request)
+ body = CreateAgentBody(name=name, **fields)
+ created = await create_agent(body, request)
agent_row = created["agent"]
if scope == "own" and agent_row.get("knowledge_scope") != "own":
try:
diff --git a/jarvis/ui/web/voice_call_routes.py b/jarvis/ui/web/voice_call_routes.py
index 1f27484d1d..c84d84a2a3 100644
--- a/jarvis/ui/web/voice_call_routes.py
+++ b/jarvis/ui/web/voice_call_routes.py
@@ -25,12 +25,31 @@ import logging
from typing import Any
from fastapi import APIRouter, HTTPException, Request
+from pydantic import BaseModel, ConfigDict, Field
log = logging.getLogger(__name__)
router = APIRouter(prefix="/api/voice", tags=["voice"])
+class BrowserStartupFailure(BaseModel):
+ """A bounded correlation token; no microphone data or failure text."""
+
+ model_config = ConfigDict(extra="forbid")
+ request_id: str = Field(
+ min_length=36, max_length=36,
+ pattern=r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$",
+ )
+
+
[email protected]("/startup-failed")
+async def voice_startup_failed(body: BrowserStartupFailure) -> dict[str, bool]:
+ """Release a pending wake request when browser audio could not start."""
+ from jarvis.live.runtime import browser_startup_failed
+
+ return {"acknowledged": browser_startup_failed(body.request_id)}
+
+
def _pipeline() -> Any:
"""The live SpeechPipeline, or ``None`` (headless / voice disabled)."""
try:
diff --git a/jarvis/workspace/agents.py b/jarvis/workspace/agents.py
index 2bb3784b77..9fe6af4a24 100644
--- a/jarvis/workspace/agents.py
+++ b/jarvis/workspace/agents.py
@@ -740,11 +740,13 @@ _KIMI_PICKS = LaunchPicks(
)
#: Cursor CLI takes ``--model`` and ``--mode plan|ask``; Agent mode is the
-#: default (no extra flag). Effort is the model's own, not a CLI flag.
+#: default; --force lets its tools act without permission prompts.
+#: Effort is the model's own, not a CLI flag.
_CURSOR_PICKS = LaunchPicks(
provider="cursor",
model_args=("--model", picks_value),
permission_args=(
+ ("auto", ("--force",)),
("plan", ("--mode", "plan")),
("ask", ("--mode", "ask")),
),
@@ -1800,7 +1802,13 @@ def build_agent_argv(name: str) -> tuple[str, ...] | None:
return None
if agent.launch_command is None:
return plain_terminal_argv()
- return _build_pty_argv(agent.launch_command)
+ from jarvis.workspace.launch_picks import launch_argv
+
+ # Only catalog-owned permission defaults enter this shell command;
+ # request-supplied model, effort and permission strings are not accepted.
+ defaults = launch_argv(name)
+ command = " ".join((agent.launch_command, *defaults))
+ return _build_pty_argv(command)
def build_install_argv(name: str) -> tuple[str, ...] | None:
diff --git a/jarvis/workspace/launch_picks.py b/jarvis/workspace/launch_picks.py
index 4f8fccb63f..9352c17726 100644
--- a/jarvis/workspace/launch_picks.py
+++ b/jarvis/workspace/launch_picks.py
@@ -163,13 +163,14 @@ def default_effort(agent: str) -> str:
def default_permission(agent: str) -> str:
- """The stance a fresh pane opens in when nobody picks one.
+ """Use the shared autonomous default when the CLI can express it."""
+ picks = picks_for(agent)
+ if picks is not None and (key := _ladder_key(picks)):
+ from jarvis.agent_chat.permissions import default_permission as chat_default
- The CLI's OWN default — an empty string — rather than a mode this app
- chose for it: a pane is a live TUI the person is sitting in front of, so
- the binary's own out-of-the-box behaviour is the honest starting point and
- every other stance is something they asked for.
- """
+ mode = chat_default(key)
+ if picks.permission_argv(mode) is not None:
+ return mode
return ""
@@ -187,18 +188,20 @@ def normalize_effort(agent: str, level: str | None) -> str:
def normalize_permission(agent: str, mode: str | None) -> str:
- """Keep ``mode`` when this CLI knows the word, else drop it.
+ """Use the autonomous default for an omitted mode; validate explicit picks.
- No folding onto a neighbouring stance here, unlike the chat runner's
- ladder: a pane's permission stance is the one thing where landing NEAR
- what was asked for is worse than landing on the CLI's own default, which
- at least asks before it acts.
+ Reject unknown explicit choices so a later normalization cannot mistake
+ a discarded choice for an omitted one and widen it to Bypass.
"""
picks = picks_for(agent)
wanted = (mode or "").strip()
- if picks is None or not wanted:
+ if picks is None:
return ""
- return wanted if picks.permission_argv(wanted) is not None else ""
+ if not wanted:
+ return default_permission(agent)
+ if picks.permission_argv(wanted) is None:
+ raise ValueError(f"Unsupported permission mode {wanted!r} for {agent}")
+ return wanted
def normalize_model(agent: str, model: str | None) -> str:
diff --git a/packaging/pyinstaller_hooks/hook-soundcard.py b/packaging/pyinstaller_hooks/hook-soundcard.py
new file mode 100644
index 0000000000..2372415136
--- /dev/null
+++ b/packaging/pyinstaller_hooks/hook-soundcard.py
@@ -0,0 +1,5 @@
+"""SoundCard loads its native API declarations from bundled C header files."""
+
+from PyInstaller.utils.hooks import collect_data_files
+
+datas = collect_data_files("soundcard", includes=["*.h"])
diff --git a/pyproject.toml b/pyproject.toml
index 4ee3f131ad..3a1e42b5c3 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -363,9 +363,11 @@ desktop = [
# so the dep is unsatisfiable there for every CPython. The indicator's
# probe already degrades to a logged no-op where PySide6 is absent.
"pyside6-essentials>=6.7; sys_platform != 'linux' or platform_machine != 'aarch64'",
- # Wayland screen recording uses Qt's user-approved ScreenCast portal.
- "pyside6-addons>=6.7; sys_platform == 'linux' and platform_machine != 'aarch64'",
+ # Streaming desktop capture; Wayland uses the user-approved ScreenCast portal.
+ "pyside6-addons>=6.7; sys_platform != 'linux' or platform_machine != 'aarch64'",
"av>=14; sys_platform != 'win32' or platform_machine != 'ARM64'",
+ # Opt-in screen-recording output loopback; never imported during base boot.
+ "SoundCard>=0.4.6; sys_platform == 'win32' or sys_platform == 'linux'",
# Drag-drop ONTO the floating overlay (bar/mascot): the cross-platform tkdnd
# Tcl extension, with bundled binaries for Windows / macOS / Linux. Registered
# onto the live Tk root in place (jarvis/overlay/drop_target.py). All-platform
diff --git a/scripts/bench_live_readiness.py b/scripts/bench_live_readiness.py
new file mode 100644
index 0000000000..a9f8ac1cde
--- /dev/null
+++ b/scripts/bench_live_readiness.py
@@ -0,0 +1,144 @@
+"""Offline comparison of allocation/TLS overlap and the actual audio-release frame.
+
+Run from the checkout to measure; the script may live in another checkout.
+All provider peers and credentials are fakes. No microphone or network opens.
+Delays are controlled experiments, not claims about provider latency.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import statistics
+import sys
+import tempfile
+import time
+from pathlib import Path
+from types import SimpleNamespace
+
+sys.path.insert(0, str(Path.cwd()))
+
+import httpx # noqa: E402
+import pytest # noqa: E402
+
+from jarvis.core.bus import EventBus # noqa: E402
+from jarvis.core.events import RealtimeSessionReady, SystemStateChanged # noqa: E402
+from jarvis.live import session as host # noqa: E402
+from jarvis.live.config import LiveConfig # noqa: E402
+from jarvis.plugins.realtime import _live_transport as transport # noqa: E402
+from tests.fakes.fake_subscription_live_wire import FakeSubscriptionLiveWire # noqa: E402
+
+
+async def measure(samples: int) -> dict:
+ started = time.perf_counter()
+ await transport.websocket_options()
+ cold_tls_ms = (time.perf_counter() - started) * 1000
+ started = time.perf_counter()
+ await transport.websocket_options()
+ warm_tls_ms = (time.perf_counter() - started) * 1000
+ results: dict[str, list[float]] = {"transport_ms": [], "audio_release_ms": []}
+ for _ in range(samples):
+ wire = FakeSubscriptionLiveWire()
+
+ async def response(request, _wire=wire):
+ await asyncio.sleep(0.3)
+ return _wire._respond(request)
+
+ def client(**kwargs):
+ return httpx.AsyncClient(transport=httpx.MockTransport(response), **kwargs)
+
+ async def options():
+ await asyncio.sleep(0.2)
+ return {} # Fake connector: no TLS or network handshake.
+
+ async def connect(*args, _wire=wire, **kwargs):
+ await asyncio.sleep(0.35)
+ return await _wire.connect(*args, **kwargs)
+
+ provider = wire.provider()
+ provider._http_client_factory = client
+ provider._websocket_connect = connect
+ with pytest.MonkeyPatch.context() as patch:
+ patch.setattr(transport, "websocket_options", options)
+ started = time.perf_counter()
+ connection = await provider.open_session(wire.config())
+ results["transport_ms"].append((time.perf_counter() - started) * 1000)
+ await connection.close()
+
+ bus = EventBus()
+ ready_at = 0.0
+ wire = FakeSubscriptionLiveWire()
+ provider = wire.provider()
+ provider.requires_close_ack = False
+
+ async def observe(event):
+ if isinstance(event, RealtimeSessionReady) or (
+ isinstance(event, SystemStateChanged) and event.new_state == "LISTENING"
+ ):
+ await asyncio.sleep(0.25)
+
+ async def send(message):
+ nonlocal ready_at
+ if message.get("type") == "audio_ready":
+ ready_at = time.perf_counter()
+
+ async def take_input(self, message):
+ return None # No physical audio device in an offline comparison.
+
+ async def immediate_options():
+ return {}
+
+ bus.subscribe(RealtimeSessionReady, observe)
+ bus.subscribe(SystemStateChanged, observe)
+ with tempfile.TemporaryDirectory(prefix="jarvis-readiness-") as directory:
+ with pytest.MonkeyPatch.context() as patch:
+ patch.setattr(transport, "websocket_options", immediate_options)
+ patch.setattr(host, "get_supervisor_tool_gateway", lambda: SimpleNamespace())
+ patch.setattr(host, "user_data_dir", lambda: Path(directory))
+ patch.setattr(host, "_identity", lambda config: "")
+ patch.setattr(host.LiveTools, "declarations", lambda self, **kw: [])
+ patch.setattr(host.LiveVoiceSession, "_take_startup_input", take_input)
+ patch.setattr(host.LiveVoiceSession, "_adopt_desktop_session", lambda self: None)
+ session = host.LiveVoiceSession(
+ session_id="offline-readiness", send_binary=send, send_json=send,
+ providers=[provider], bus=bus,
+ config=SimpleNamespace(
+ brain=SimpleNamespace(reply_language="en"),
+ live=LiveConfig(configured=True, backend_model="test",
+ model="gpt-live-1-codex", voice="cove"),
+ ),
+ )
+ started = time.perf_counter()
+ try:
+ await session.handle_control({
+ "type": "audio_start", "sample_rate": 48000,
+ "webrtc_offer_sdp": wire.config().offer_sdp,
+ })
+ assert ready_at, "No audio-release frame was sent"
+ results["audio_release_ms"].append((ready_at - started) * 1000)
+ finally:
+ await session.end()
+ return {
+ "samples": samples,
+ "network_requests": 0,
+ "physical_audio_verified": False,
+ "measured_local_tls_ms": {"cold": round(cold_tls_ms, 2), "warm": round(warm_tls_ms, 2)},
+ "controlled_delays_ms": {
+ "allocation": 300, "local_tls": 200, "attachment": 350,
+ "readiness_observer": 250, "indicator_observer": 250,
+ },
+ "median_ms": {key: round(statistics.median(values), 2)
+ for key, values in results.items()},
+ "measurements_ms": {key: [round(value, 2) for value in values]
+ for key, values in results.items()},
+ }
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--samples", type=int, default=5)
+ args = parser.parse_args()
+ if not 1 <= args.samples <= 30:
+ parser.error("samples must be between 1 and 30")
+ print(json.dumps(asyncio.run(measure(args.samples)), indent=2))
diff --git a/scripts/config-soll.json b/scripts/config-soll.json
index 56ebd9de92..d7c896b2d9 100644
--- a/scripts/config-soll.json
+++ b/scripts/config-soll.json
@@ -56,7 +56,7 @@
"deep_model": "claude-opus-4-8"
},
"brain.worker": {
- "provider": "claude-api",
+ "provider": "openai-codex",
"model": "",
"fallback_provider": "gemini",
"fallback_model": "gemini-3.1-pro-preview",
@@ -64,7 +64,7 @@
"fallback_model_2": "gemini-3-pro-preview"
},
"stt": {
- "provider": "faster-whisper",
+ "provider": "groq-api",
"model": "whisper-large-v3",
"bias_prompt": "Jarvis, OpenClaw, Codex, Gemini, Grok, Claude, Anthropic, OpenAI, OpenRouter, Whisper, Subagent, Sub-Agent, Mission-Manager, Critic-Loop, Critic, Worker, Router-Brain, Router, Kontrollierer, Voice-Pipeline, Wake-Word, Silero, VAD, TTS, STT, Telemetry, Transkript, Token, Provider, Plugin, Sprechblase, Orb, Mascot, Quip, Browser, Terminal, Repository, Branch, Commit, Push, Pull-Request, Issue, Merge, Refactor, Deployment, Screenshot, Debuggen, Workflow, Microservice, Container, Docker, Kubernetes, FastAPI, React, TypeScript, Python, JSON, Markdown, Logfile, Konfiguration, Endpoint, Webhook, API-Key, Service-Account, Cloud, Google Cloud, Vertex AI, BigQuery, GitHub, Twilio, ElevenLabs, OAuth, Credential Manager, Frontend, Backend, Skript, Daemon, Hangup, Auflegen, Denkpause, submitted, distanziert, erkennt",
"language": "auto",
diff --git a/scripts/spikes/agent_runtimes_e2e.py b/scripts/spikes/agent_runtimes_e2e.py
index 322b2c25ac..127f450d0e 100644
--- a/scripts/spikes/agent_runtimes_e2e.py
+++ b/scripts/spikes/agent_runtimes_e2e.py
@@ -53,6 +53,8 @@ class _IO:
async def _run(name: str, turn: RuntimeTurn, text: str) -> tuple[AcpTurn, _IO]:
+ from jarvis.agent_runtimes import gateway
+
runtime = driver(name)
launch = await runtime.launch(turn)
acp = AcpTurn(
@@ -66,7 +68,12 @@ async def _run(name: str, turn: RuntimeTurn, text: str) -> tuple[AcpTurn, _IO]:
)
tree = make_process_tree("runtime-e2e")
proc = None
+ failure = None
+ token = turn.route.api_key or ""
try:
+ if gateway.verify(token) is not None:
+ # Match the chat runner's ownership of each model-gateway turn.
+ failure = gateway.watch_failure(token, effort=turn.effort or "")
with (turn.workspace.parent / "runtime-stderr.log").open("ab") as stderr:
proc = await asyncio.create_subprocess_exec(
*launch.argv,
@@ -101,6 +108,8 @@ async def _run(name: str, turn: RuntimeTurn, text: str) -> tuple[AcpTurn, _IO]:
await asyncio.wait_for(proc.wait(), timeout=30)
return acp, io
finally:
+ if failure is not None:
+ gateway.unwatch_failure(token, failure)
tree.close()
try:
if proc is not None:
diff --git a/tests/contract/fixtures/agent-companion.json b/tests/contract/fixtures/agent-companion.json
index 480dabb9ff..b560f9c8b1 100644
--- a/tests/contract/fixtures/agent-companion.json
+++ b/tests/contract/fixtures/agent-companion.json
@@ -3,7 +3,9 @@
{"shape":"circle","color":"#8b5cf6"},
{"shape":"cloud","color":"#112233","eyes":"lines","enabled":false,"sizeM":0.25,"followDistanceM":2},
{"shape":"drop","color":"#FFAA00","eyes":"dots","enabled":true,"sizeM":0.8,"followDistanceM":0.5},
- {"shape":"hexagon","color":"#7ab6ef","accessories":{"head":"crown","mouth":"cigar","outfit":"suit","held":"future_item_2"}}
+ {"shape":"hexagon","color":"#7ab6ef","accessories":{"head":"crown","mouth":"cigar","outfit":"suit","held":"future_item_2"}},
+ {"shape":"drop","color":"#79c7c4","pet":"cocoa"},
+ {"shape":"circle","color":"#8b5cf6","pet":"u0123456789abcdef"}
],
"invalid": [
{"shape":"unknown","color":"#123456"},
@@ -20,6 +22,10 @@
{"shape":"circle","color":"#123456","accessories":{"head":"Crown"}},
{"shape":"circle","color":"#123456","accessories":{"head":"../../etc"}},
{"shape":"circle","color":"#123456","accessories":{"head":7}},
- {"shape":"circle","color":"#123456","accessories":["crown"]}
+ {"shape":"circle","color":"#123456","accessories":["crown"]},
+ {"shape":"circle","color":"#123456","pet":"Cocoa"},
+ {"shape":"circle","color":"#123456","pet":"../pets/cocoa"},
+ {"shape":"circle","color":"#123456","pet":""},
+ {"shape":"circle","color":"#123456","pet":7}
]
}
diff --git a/tests/contract/native_window_probe.py b/tests/contract/native_window_probe.py
index 99f3242ccc..7d058aa452 100644
--- a/tests/contract/native_window_probe.py
+++ b/tests/contract/native_window_probe.py
@@ -7,10 +7,89 @@ import ctypes
import faulthandler
import json
import sys
+import tempfile
from ctypes import wintypes
from pathlib import Path
+async def capture_styles() -> None:
+ """Report capture support for individual window styles on a CI desktop."""
+ root = Path(__file__).resolve().parents[2]
+ sys.path.insert(0, str(root / "jarvis" / "society" / "browser"))
+ from native_window import NativeWindow
+ from playwright.async_api import async_playwright
+
+ original = NativeWindow._hide_from_desktop
+ original_park = NativeWindow.park
+ variants = [
+ ("unchanged", 0, 0, None, False),
+ ("background-only", 0, 0, None, True),
+ ("no-activate", 0x08000000, 0x40000, None, False),
+ ("no-activate-appwindow", 0x08040000, 0, None, False),
+ ("no-appwindow", 0, 0x40000, None, False),
+ ("layered-opaque", 0x80000, 0, 255, False),
+ ("layered-minimum", 0x80000, 0, 1, False),
+ ("parked-minimum", 0x08080000, 0x40080, 1, False),
+ ("positioned-minimum", 0x08080000, 0x40080, 1, True),
+ ("parked-click-through", 0x08080020, 0x40080, 1, False),
+ ("layered-zero", 0x80000, 0, 0, False),
+ ]
+ try:
+ async with async_playwright() as pw:
+ for label, add, remove, alpha, reposition in variants:
+ def apply_style(self, hwnd, add=add, remove=remove, alpha=alpha):
+ if not self._owned(hwnd):
+ raise RuntimeError("Probe window ownership changed")
+ u = self.user32
+ u.GetWindowLongW.argtypes = [wintypes.HWND, ctypes.c_int]
+ u.GetWindowLongW.restype = ctypes.c_long
+ u.SetWindowLongW.argtypes = [wintypes.HWND, ctypes.c_int, ctypes.c_long]
+ u.SetWindowLongW.restype = ctypes.c_long
+ style = u.GetWindowLongW(hwnd, -20)
+ wanted = (style | add) & ~remove
+ if wanted != style:
+ u.SetWindowLongW(hwnd, -20, wanted)
+ if alpha is not None:
+ u.SetLayeredWindowAttributes.argtypes = [wintypes.HWND, wintypes.DWORD, wintypes.BYTE, wintypes.DWORD]
+ u.SetLayeredWindowAttributes.restype = wintypes.BOOL
+ if not u.SetLayeredWindowAttributes(hwnd, 0, alpha, 2):
+ raise RuntimeError("Probe opacity was refused")
+
+ NativeWindow._hide_from_desktop = apply_style
+ def styles_only(self):
+ self._hide_from_desktop(self.hwnd)
+ self._parked = True
+
+ NativeWindow.park = original_park if reposition else styles_only
+ with tempfile.TemporaryDirectory(prefix="jarvis-capture-style-") as profile:
+ context = await pw.chromium.launch_persistent_context(
+ profile, executable_path=sys.argv[1], headless=False, no_viewport=True,
+ )
+ native = None
+ try:
+ page = context.pages[0]
+ await page.set_content('<h1 style="background:#f00">Capture fixture</h1>')
+ cdp = await context.browser.new_browser_cdp_session()
+ processes = await cdp.send("SystemInfo.getProcessInfo")
+ pid = next(p["id"] for p in processes["processInfo"] if p["type"] == "browser")
+ native = NativeWindow(int(pid))
+ ready = await asyncio.to_thread(native.ready.wait, 5)
+ await asyncio.sleep(0.2)
+ result = {"ready": ready, "failed": native.failed}
+ if ready and not native.failed:
+ result["frame"] = bool(native.frame())
+ print(json.dumps({"capture_style": label, **result}), flush=True)
+ except Exception as exc:
+ print(json.dumps({"capture_style": label, "error": str(exc)}), flush=True)
+ finally:
+ if native:
+ native.close()
+ await context.close()
+ finally:
+ NativeWindow._hide_from_desktop = original
+ NativeWindow.park = original_park
+
+
async def main() -> None:
root = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(root / "jarvis" / "society" / "browser"))
@@ -70,7 +149,11 @@ async def main() -> None:
await asyncio.sleep(0.3)
value = await page.get_by_label("Name").input_value()
assert value == "Typed through the preview", value
- assert native.input_hwnd != native.hwnd
+ # Chromium may route the page's keyboard input through its main
+ # HWND on ARM. Check ownership and the real input value above.
+ input_pid = wintypes.DWORD()
+ u.GetWindowThreadProcessId(native.input_hwnd, ctypes.byref(input_pid))
+ assert input_pid.value == int(pid)
assert badge
# Focus the address bar through Chrome's own location command, not
# a click at a guessed toolbar position: on a slow runner the URL
@@ -90,4 +173,4 @@ async def main() -> None:
if __name__ == "__main__":
faulthandler.enable()
- asyncio.run(main())
+ asyncio.run(capture_styles() if "--capture-styles" in sys.argv else main())
diff --git a/tests/contract/test_agent_companion.py b/tests/contract/test_agent_companion.py
index 6c67e04940..b4adf267fc 100644
--- a/tests/contract/test_agent_companion.py
+++ b/tests/contract/test_agent_companion.py
@@ -10,6 +10,7 @@ from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import ValidationError
+from jarvis.society.agent_template import public_avatar
from jarvis.society.companion import AccessorySlot, CompanionAppearance, validate_avatar_companion
from jarvis.society.runtime import SocietyRuntime
from jarvis.ui.web.society_routes import router
@@ -48,6 +49,20 @@ def test_worn_accessories_survive_and_bare_looks_stay_unchanged():
assert "accessories" not in bare
+def test_a_worn_pet_survives_and_a_look_without_one_stores_no_pet_key():
+ worn = validate_avatar_companion({"companion": {"shape": "drop", "color": "#79c7c4", "pet": "cocoa"}})
+ assert worn["companion"]["pet"] == "cocoa"
+ bare = validate_avatar_companion({"companion": {"shape": "drop", "color": "#79c7c4", "pet": None}})
+ assert "pet" not in bare["companion"]
+
+
+def test_a_shared_template_keeps_a_builtin_pet_and_drops_a_drawn_one():
+ builtin = public_avatar({"companion": {"shape": "drop", "color": "#79c7c4", "pet": "cocoa"}})
+ assert builtin["companion"]["pet"] == "cocoa"
+ drawn = public_avatar({"companion": {"shape": "drop", "color": "#79c7c4", "pet": "u0123456789abcdef"}})
+ assert drawn["companion"] == {"shape": "drop", "color": "#79c7c4"}
+
+
def test_legacy_and_imported_characters_survive_companion_validation():
old = {
"contract": 1,
diff --git a/tests/contract/test_agent_mcp_harmony.py b/tests/contract/test_agent_mcp_harmony.py
index 9cd1577224..6645e508ee 100644
--- a/tests/contract/test_agent_mcp_harmony.py
+++ b/tests/contract/test_agent_mcp_harmony.py
@@ -81,6 +81,12 @@ WITHHELD: Final[dict[str, str]] = {
"DELETE /api/society/agents/{agent_id}": (
"archiving a teammate is destructive and irreversible from outside"
),
+ "GET /api/society/agents/{agent_id}/credentials": (
+ "an agent's stored credentials are the owner's business, managed in the app only"
+ ),
+ "DELETE /api/society/agents/{agent_id}/credentials/{env}": (
+ "deleting a stored credential breaks the agent's work and stays with the owner"
+ ),
"POST /api/society/agents/{agent_id}/kill": (
"kill_switch stops the whole house; killing ONE agent's runs needs the "
"context of watching them, which a remote client does not have"
diff --git a/tests/contract/test_browser_live.py b/tests/contract/test_browser_live.py
index 9c72b534d9..9281f6488c 100644
--- a/tests/contract/test_browser_live.py
+++ b/tests/contract/test_browser_live.py
@@ -704,6 +704,46 @@ async def test_finished_chat_releases_real_browser_for_the_next_task(live, monke
await live.close()
+def _owned_windows(root_pid):
+ """Visible top-level windows of a worker's Chrome: title, class, hang and styles."""
+ if os.name != "nt":
+ return []
+ import ctypes
+ from ctypes import wintypes
+
+ import psutil
+
+ try:
+ pids = {child.pid for child in psutil.Process(root_pid).children(recursive=True)}
+ except psutil.Error as exc:
+ return [f"process tree unavailable: {exc}"]
+ user32 = ctypes.windll.user32
+ user32.GetWindowLongW.restype = ctypes.c_long
+ found = []
+
+ @ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.HWND, wintypes.LPARAM)
+ def visit(hwnd, _):
+ pid = wintypes.DWORD()
+ user32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid))
+ if pid.value in pids and user32.IsWindowVisible(hwnd):
+ title = ctypes.create_unicode_buffer(256)
+ name = ctypes.create_unicode_buffer(128)
+ user32.GetWindowTextW(hwnd, title, 256)
+ user32.GetClassNameW(hwnd, name, 128)
+ alpha, flags = wintypes.BYTE(), wintypes.DWORD()
+ user32.GetLayeredWindowAttributes(hwnd, None, ctypes.byref(alpha), ctypes.byref(flags))
+ found.append({
+ "class": name.value, "title": title.value,
+ "hung": bool(user32.IsHungAppWindow(hwnd)),
+ "exstyle": hex(user32.GetWindowLongW(hwnd, -20) & 0xFFFFFFFF),
+ "alpha": alpha.value, "layered_flags": flags.value,
+ })
+ return True
+
+ user32.EnumWindows(visit, 0)
+ return found
+
+
async def test_login_profile_survives_restart_and_stays_with_its_agent(live, site):
first = SimpleNamespace(
agent_id="signed-in", model="", browser_allowed_domains=["http*://127.0.0.1"]
@@ -718,8 +758,12 @@ async def test_login_profile_survives_restart_and_stays_with_its_agent(live, sit
session, _ = await live.subscribe(agent)
await _control(live, session, "viewer", "takeover", {"enabled": True})
await _control(live, session, "viewer", "navigate", {"url": site + path})
- async with asyncio.timeout(15):
- assert await asyncio.to_thread(PageHandler.wait_for_path, path)
+ if not await asyncio.to_thread(PageHandler.wait_for_path, path):
+ windows = await asyncio.to_thread(_owned_windows, session.proc.pid)
+ raise AssertionError(
+ f"{path} was never requested; requests={PageHandler.page_requests}; "
+ f"chrome_windows={windows}; worker_stderr_end={session.stderr_tail[-3000:]}"
+ )
await _control(live, session, "viewer", "takeover", {"enabled": False})
return session
diff --git a/tests/contract/test_bypass_permissions.py b/tests/contract/test_bypass_permissions.py
new file mode 100644
index 0000000000..51c03d9c58
--- /dev/null
+++ b/tests/contract/test_bypass_permissions.py
@@ -0,0 +1,176 @@
+"""A fresh installation runs agents and permitted tools without permission prompts."""
+
+import asyncio
+import tomllib
+from pathlib import Path
+from uuid import uuid4
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.agent_chat.approval_bridge import ChatApprovalBridge, ChatGrant
+from jarvis.agent_chat.permissions import default_permission
+from jarvis.core.bus import EventBus
+from jarvis.core.config import JarvisConfig, SafetyConfig
+from jarvis.core.events import ActionApproved
+from jarvis.safety import ApprovalWorkflow, RiskTierEvaluator, ToolExecutor
+from jarvis.society.runtime import SocietyRuntime
+from jarvis.tasks.approval_bridge import TaskAutoApprover
+from jarvis.ui.web.society_routes import router
+from jarvis.workspace import launch_picks
+from tests.fakes.fake_permission_tool import FakePermissionTool
+
+
+def test_example_config_and_fresh_config_share_the_bypass_default():
+ example = tomllib.loads(Path("jarvis.toml.example").read_text(encoding="utf-8"))
+ assert SafetyConfig.model_validate(example["safety"]).approval_mode == "bypass"
+ assert JarvisConfig().safety.approval_mode == "bypass"
+
+
+ ("runner", "mode"),
+ [
+ ("claude-cli", "bypassPermissions"), ("glm-cli", "bypassPermissions"),
+ ("codex-cli", "full-access"), ("agy-cli", "skip-permissions"),
+ ("grok-cli", "bypassPermissions"), ("opencode-cli", "auto"),
+ ("kimi-cli", "auto"), ("dsh-cli", "auto"), ("cursor-cli", "auto"),
+ ("api", "auto"), ("jarvis", "bypass"), ("society", "bypass"),
+ ],
+)
+def test_every_chat_runner_defaults_to_autonomous_execution(runner, mode):
+ assert default_permission(runner) == mode
+
+
+ ("agent", "flag"),
+ [
+ ("claude", "bypassPermissions"), ("codex", "--dangerously-bypass-approvals-and-sandbox"),
+ ("antigravity", "--dangerously-skip-permissions"), ("grok-build", "bypassPermissions"),
+ ("glm", "bypassPermissions"),
+ ("opencode", "--auto"), ("kimi", "--auto"), ("cursor", "--force"),
+ ],
+)
+def test_native_pane_launches_apply_the_default_without_a_ui_pick(agent, flag):
+ assert launch_picks.default_permission(agent)
+ assert flag in launch_picks.launch_argv(agent)
+
+
[email protected]("creation", ["manual", "delegated", "template"])
+def test_nine_new_agents_share_the_fresh_install_default(tmp_path: Path, creation):
+ runtime = SocietyRuntime(tmp_path, seed_starter_team=False)
+ app = FastAPI()
+ app.include_router(router)
+ app.state.society_factory = lambda: runtime
+ with TestClient(app) as client:
+ for number in range(9):
+ name = f"Helper {number}"
+ if creation == "template":
+ response = client.post("/api/society/templates/install", json={"template": {
+ "schema": 1, "name": name, "instructions": "Maintain the task list.",
+ }})
+ else:
+ headers = (
+ {"X-Jarvis-Chat-Session": "society:jarvis"} if creation == "delegated" else {}
+ )
+ response = client.post("/api/society/agents", json={"name": name}, headers=headers)
+ assert response.status_code == 200, response.text
+ agent = response.json()["agent"]
+ assert agent["approval_mode"] == "bypass"
+ assert agent["permission_ceiling"] == "ask"
+ assert agent["grant_mode"] == "all"
+ agents = client.get("/api/society/agents").json()["agents"]
+ assert len([a for a in agents if a["agent_id"] != "jarvis"]) == 9
+
+
+def test_the_leads_ceiling_can_be_updated_under_the_app_policy(tmp_path: Path):
+ runtime = SocietyRuntime(tmp_path, seed_starter_team=False)
+ app = FastAPI()
+ app.include_router(router)
+ app.state.society_factory = lambda: runtime
+ with TestClient(app) as client:
+ for ceiling in ("monitor", "ask"):
+ response = client.patch(
+ "/api/society/agents/jarvis", json={"permission_ceiling": ceiling}
+ )
+ assert response.status_code == 200, response.text
+ assert response.json()["agent"]["permission_ceiling"] == ceiling
+ response = client.post(
+ "/api/society/agents", json={"name": "Delegated helper"},
+ headers={"X-Jarvis-Chat-Session": "society:jarvis"},
+ )
+ assert response.status_code == 200, response.text
+ assert response.json()["agent"]["permission_ceiling"] == "ask"
+ assert response.json()["agent"]["approval_mode"] == "bypass"
+
+
[email protected]("snapshot", [
+ {}, {"voice_confirm": True}, {"approval_surface": "unattended"},
+ {"approval_surface": "interactive", "mission_id": "mission", "worker_id": "worker"},
+])
+async def test_fresh_config_executes_with_an_immediate_audited_approval(snapshot):
+ bus = EventBus()
+ approvals = []
+
+ async def record(event):
+ approvals.append(event.approved_by)
+
+ bus.subscribe(ActionApproved, record)
+ executor = ToolExecutor(bus, RiskTierEvaluator(JarvisConfig().safety), ApprovalWorkflow(bus))
+ tool = FakePermissionTool()
+ result = await asyncio.wait_for(executor.execute(tool, {}, config_snapshot=snapshot), 1)
+ assert result.success and len(tool.calls) == 1
+ assert tool.calls[0].approved_by == "bypass"
+ assert approvals == ["bypass"]
+
+
[email protected]("mode", ["ask", "bypass"])
+async def test_a_chats_explicit_stance_owns_its_approval(mode):
+ bus = EventBus()
+ workflow = ApprovalWorkflow(bus)
+ bridge = ChatApprovalBridge(bus)
+ cards = []
+
+ async def ask(*args):
+ cards.append(args)
+ return "allow"
+
+ bridge.arm("agent-chat:test", ChatGrant("test", "turn", mode, set(), ask))
+ executor = ToolExecutor(bus, RiskTierEvaluator(SafetyConfig()), workflow)
+ result = await asyncio.wait_for(executor.execute(FakePermissionTool(), {}, config_snapshot={
+ "approval_ref": "agent-chat:test", "approval_surface": "interactive",
+ }), 1)
+ assert result.success
+ assert len(cards) == (1 if mode == "ask" else 0)
+
+
+async def test_bypass_preserves_a_scheduled_tasks_read_only_grant():
+ bus = EventBus()
+ workflow = ApprovalWorkflow(bus)
+ approver = TaskAutoApprover(bus)
+ trace = uuid4()
+ approver.arm(trace, [], approved_by="scheduled-task:readonly")
+ tool = FakePermissionTool()
+ executor = ToolExecutor(bus, RiskTierEvaluator(SafetyConfig()), workflow)
+ result = await executor.execute(tool, {}, trace_id=trace)
+ assert not result.success and not tool.calls
+ assert "no write grant" in result.error
+
+
[email protected]("snapshot,tier", [({"chat_read_only": True}, "ask"), ({}, "block")])
+async def test_bypass_does_not_execute_a_blocked_action(snapshot, tier):
+ bus = EventBus()
+ executor = ToolExecutor(bus, RiskTierEvaluator(SafetyConfig()), ApprovalWorkflow(bus))
+ tool = FakePermissionTool(tier)
+ result = await executor.execute(tool, {}, config_snapshot=snapshot)
+ assert not result.success and not tool.calls
+
+
+async def test_tool_arguments_cannot_change_an_explicit_ask_policy():
+ bus = EventBus()
+ executor = ToolExecutor(
+ bus, RiskTierEvaluator(SafetyConfig(approval_mode="ask")), ApprovalWorkflow(bus)
+ )
+ tool = FakePermissionTool()
+ result = await executor.execute(tool, {"approval_mode": "bypass"})
+ assert not result.success and not tool.calls
diff --git a/tests/contract/test_calendar_routines.py b/tests/contract/test_calendar_routines.py
index aed0f58736..24bb8ea591 100644
--- a/tests/contract/test_calendar_routines.py
+++ b/tests/contract/test_calendar_routines.py
@@ -263,6 +263,11 @@ async def test_http_timezone_propagates_to_spawned_turn_without_leaking():
captured = []
class Service:
+ store = SimpleNamespace(get_session=lambda _sid: SimpleNamespace(surface="society"))
+
+ def is_running(self, _sid):
+ return False
+
async def send(self, *args, **kwargs):
async def turn():
await asyncio.sleep(0)
diff --git a/tests/contract/test_internal_messages.py b/tests/contract/test_internal_messages.py
index 1e399ba30a..1fc261b9c7 100644
--- a/tests/contract/test_internal_messages.py
+++ b/tests/contract/test_internal_messages.py
@@ -71,6 +71,16 @@ async def drain(rt):
await asyncio.sleep(0.01)
+async def drain_until(rt, predicate):
+ """Drain until ``predicate`` holds: a cancelled turn hands over on its own
+ schedule, which a slow CI runner stretches past a fixed number of drains."""
+ async with asyncio.timeout(10):
+ while True:
+ await drain(rt)
+ if await predicate():
+ return
+
+
async def test_lead_message_is_an_internal_chat_receipt(world):
rt, svc, seen = world
tool = LeadMessageAgentTool(runtime_resolver=lambda: rt)
@@ -101,12 +111,20 @@ async def test_busy_recipient_keeps_fifo_receipts_then_resumes_once(world):
receipt = svc.store.incoming_message("society:scout", second.event_id)
assert receipt["status"] == "queued"
await svc.cancel("society:scout")
- await drain(rt)
- assert await rt.store.delivery_status(second.event_id) == "delivered"
+
+ async def second_delivered():
+ return await rt.store.delivery_status(second.event_id) == "delivered"
+
+ await drain_until(rt, second_delivered)
assert await rt.store.delivery_status(third.event_id) == "queued"
await rt.scheduler.on_envelope(first)
await svc.cancel("society:scout")
- await drain(rt)
+
+ async def three_turns():
+ events = svc.store.list_events("society:scout")
+ return len([e for e in events if e["kind"] == "turn_started"]) >= 3
+
+ await drain_until(rt, three_turns)
events = svc.store.list_events("society:scout")
assert [e["payload"]["text"] for e in events if e["kind"] == "agent_message"] == [
"first",
diff --git a/tests/contract/test_live_computer.py b/tests/contract/test_live_computer.py
index 068c92db0d..cc3fbcdf8a 100644
--- a/tests/contract/test_live_computer.py
+++ b/tests/contract/test_live_computer.py
@@ -113,12 +113,28 @@ def test_voice_catalog_drops_the_second_model_vehicles(monkeypatch):
def test_backend_instructions_hand_the_screen_to_the_thinking_model():
from jarvis.live.config import LiveConfig
+ config = LiveConfig(configured=True, backend_model="thinking-model")
+ computer = [{"type": "function", "name": "computer", "parameters": tool_schema()}]
+ instructions = config.session_config(language="en", tools=computer)["delegation"][
+ "responses"
+ ]["instructions"]
+ assert "computer tool" in instructions
+ assert "no separate computer-use agent" in instructions
+ assert "Use computer for operating the desktop." in instructions
+
+
+def test_instructions_never_name_a_computer_tool_that_is_switched_off():
+ """With Computer Use off the voice catalog has no computer tool; its rules
+ would send the model to a tool it cannot call."""
+ from jarvis.live.config import LiveConfig
+
config = LiveConfig(configured=True, backend_model="thinking-model")
instructions = config.session_config(language="en", tools=[])["delegation"]["responses"][
"instructions"
]
- assert "computer tool" in instructions
- assert "no separate computer-use agent" in instructions
+ assert "computer tool" not in instructions
+ assert "Use computer for operating" not in instructions
+ assert "call take_appshot for a fresh capture" in instructions
@pytest.mark.parametrize("auth_mode", ["api_key", "chatgpt_subscription"])
diff --git a/tests/contract/test_live_media_feedback.py b/tests/contract/test_live_media_feedback.py
index 47aa8a2769..70f4a07a8a 100644
--- a/tests/contract/test_live_media_feedback.py
+++ b/tests/contract/test_live_media_feedback.py
@@ -75,7 +75,8 @@ async def test_native_microphone_meter_remains_visible_while_connecting(presenta
await bus.publish(SystemStateChanged(new_state="LISTENING", previous="IDLE"))
await bus.publish(SystemStateChanged(new_state="CONNECTING", previous="LISTENING"))
mic_level.publish(0.65)
- assert surface.mode == "listen"
+ # The loading loop shows the handshake; the meter inside it keeps moving.
+ assert surface.mode == "connect"
assert surface.level == pytest.approx(0.65)
diff --git a/tests/contract/test_society_one_click_create.py b/tests/contract/test_society_one_click_create.py
index b96af465aa..f95a6d2ab0 100644
--- a/tests/contract/test_society_one_click_create.py
+++ b/tests/contract/test_society_one_click_create.py
@@ -41,7 +41,7 @@ async def roster(tmp_path: Path):
async def test_unnamed_create_gets_placeholder_name_and_random_id(roster: Roster):
agent, created = await roster.create()
assert created is True
- assert agent.name == "New Bot"
+ assert agent.name == "New Agent"
assert re.fullmatch(r"agent-[0-9a-f]{8}", agent.agent_id)
assert agent.session_id == f"society:{agent.agent_id}"
assert is_fresh(agent)
@@ -52,14 +52,14 @@ async def test_unnamed_creates_never_adopt_each_other(roster: Roster):
second, created = await roster.create()
assert created is True
assert first.agent_id != second.agent_id
- assert (first.name, second.name) == ("New Bot", "New Bot 2")
+ assert (first.name, second.name) == ("New Agent", "New Agent 2")
async def test_concurrent_unnamed_creates_have_distinct_names_and_chats(roster: Roster):
results = await asyncio.gather(*(roster.create() for _ in range(3)))
assert all(created for _, created in results)
agents = [agent for agent, _ in results]
- assert {agent.name for agent in agents} == {"New Bot", "New Bot 2", "New Bot 3"}
+ assert {agent.name for agent in agents} == {"New Agent", "New Agent 2", "New Agent 3"}
assert len({agent.session_id for agent in agents}) == 3
@@ -73,17 +73,17 @@ async def test_rename_keeps_the_random_id(roster: Roster):
assert renamed.name == "Mail Desk"
assert not is_fresh(renamed)
next_agent, created = await roster.create()
- assert created and next_agent.name == "New Bot"
+ assert created and next_agent.name == "New Agent"
assert next_agent.session_id != renamed.session_id
async def test_placeholder_numbering_respects_existing_names_case_insensitively(roster: Roster):
- for name in ("new bot", "NEW BOT 2"):
+ for name in ("new agent", "NEW AGENT 2"):
await roster.create(name=name, title="taken")
agent, _ = await roster.create()
- assert agent.name == "New Bot 3"
- assert (await roster.resolve("new bot")).title == "taken"
- assert (await roster.resolve("NEW BOT 2")).title == "taken"
+ assert agent.name == "New Agent 3"
+ assert (await roster.resolve("new agent")).title == "taken"
+ assert (await roster.resolve("NEW AGENT 2")).title == "taken"
async def test_unnamed_lead_is_refused(roster: Roster):
@@ -135,12 +135,12 @@ def test_post_without_a_name_creates_a_fresh_agent(tmp_path: Path):
body = response.json()
assert body["created"] is True
agent = body["agent"]
- assert agent["name"] == "New Bot"
+ assert agent["name"] == "New Agent"
assert agent["agent_id"].startswith("agent-")
assert agent["avatar"]["companion"]["shape"] in COMPANION_SHAPES
again = client.post("/api/society/agents", json={"name": ""}).json()
assert again["created"] is True
- assert again["agent"]["name"] == "New Bot 2"
+ assert again["agent"]["name"] == "New Agent 2"
assert again["agent"]["agent_id"] != agent["agent_id"]
finally:
client.portal.call(runtime.close)
diff --git a/tests/fakes/fake_acp_agent.py b/tests/fakes/fake_acp_agent.py
index 8407687c9a..cb9a0b6404 100644
--- a/tests/fakes/fake_acp_agent.py
+++ b/tests/fakes/fake_acp_agent.py
@@ -99,6 +99,16 @@ def _read() -> dict[str, Any] | None:
def _prompt(rid: Any, session_id: str, text: str, store: dict[str, list[str]]) -> None:
store.setdefault(session_id, []).append(text)
_save_store(store)
+ if "EOF_PARTIAL" in text:
+ _text(session_id, "partial only")
+ raise SystemExit(0)
+ if "WAIT_CANCEL" in text:
+ _text(session_id, "waiting")
+ while frame := _read():
+ if frame.get("method") == "session/cancel":
+ _send({"jsonrpc": "2.0", "id": rid, "result": {"stopReason": "cancelled"}})
+ return
+ return
if endpoint := os.environ.get("FAKE_ACP_GATEWAY"):
url = urlsplit(endpoint)
assert url.scheme == "http" and url.hostname == "127.0.0.1"
@@ -232,7 +242,7 @@ def _prompt(rid: Any, session_id: str, text: str, store: dict[str, list[str]]) -
"jsonrpc": "2.0",
"id": rid,
"result": {
- "stopReason": "max_tokens" if "MAXTOK" in text else "end_turn",
+ "stopReason": "max_tokens" if ("TRUNCATE" in text or "MAXTOK" in text) else "end_turn",
"usage": {"inputTokens": 11, "outputTokens": 7, "totalTokens": 18},
},
}
diff --git a/tests/fakes/fake_anthropic_wire.py b/tests/fakes/fake_anthropic_wire.py
new file mode 100644
index 0000000000..594a9b677d
--- /dev/null
+++ b/tests/fakes/fake_anthropic_wire.py
@@ -0,0 +1,66 @@
+"""Recorded Anthropic SSE shapes served locally through an SDK transport."""
+
+import json
+
+from anthropic import _base_client
+
+# Fresh SDK installs use httpx2; the pinned older SDK still uses httpx.
+# Keep the request, response and mock transport in the SDK's own HTTP family.
+_HTTP = getattr(_base_client, "httpx2", None) or _base_client.httpx
+
+
+class AnthropicWire:
+ def __init__(self) -> None:
+ self.requests: list[_HTTP.Request] = []
+ self.intermediate_usage = False
+ self.tool_arguments: str | None = None
+ self.thinking = False
+
+ def client(self):
+ return _HTTP.AsyncClient(transport=_HTTP.MockTransport(self.handle))
+
+ def handle(self, request: _HTTP.Request) -> _HTTP.Response:
+ self.requests.append(request)
+ events = [
+ {"type": "message_start", "message": {
+ "id": "msg_test", "type": "message", "role": "assistant",
+ "model": "claude-sonnet-4-6", "content": [], "stop_reason": None,
+ "stop_sequence": None, "usage": {"input_tokens": 100, "output_tokens": 1,
+ "cache_read_input_tokens": 20, "cache_creation_input_tokens": 5},
+ }},
+ {"type": "content_block_start", "index": 0,
+ "content_block": {"type": "text", "text": ""}},
+ {"type": "content_block_delta", "index": 0,
+ "delta": {"type": "text_delta", "text": "OK"}},
+ {"type": "content_block_stop", "index": 0},
+ {"type": "message_delta", "delta": {
+ "stop_reason": "end_turn", "stop_sequence": None,
+ }, "usage": {"output_tokens": 2}},
+ {"type": "message_stop"},
+ ]
+ if self.tool_arguments is not None:
+ events[1]["content_block"] = {
+ "type": "tool_use", "id": "call_test", "name": "read_file", "input": {},
+ }
+ events[2]["delta"] = {"type": "input_json_delta", "partial_json": self.tool_arguments}
+ events[4]["delta"]["stop_reason"] = "tool_use"
+ if self.intermediate_usage:
+ events.insert(4, {"type": "message_delta", "delta": {
+ "stop_reason": None, "stop_sequence": None,
+ }, "usage": {"output_tokens": 1}})
+ if self.thinking:
+ for event in events:
+ if "index" in event:
+ event["index"] += 1
+ events[1:1] = [
+ {"type": "content_block_start", "index": 0, "content_block": {
+ "type": "thinking", "thinking": "", "signature": "",
+ }},
+ {"type": "content_block_delta", "index": 0,
+ "delta": {"type": "thinking_delta", "thinking": "private test reasoning"}},
+ {"type": "content_block_delta", "index": 0,
+ "delta": {"type": "signature_delta", "signature": "opaque-test-signature"}},
+ {"type": "content_block_stop", "index": 0},
+ ]
+ body = "".join(f"event: {e['type']}\ndata: {json.dumps(e)}\n\n" for e in events)
+ return _HTTP.Response(200, text=body, headers={"content-type": "text/event-stream"})
diff --git a/tests/fakes/fake_disconnect_runner.py b/tests/fakes/fake_disconnect_runner.py
new file mode 100644
index 0000000000..ffff7d222f
--- /dev/null
+++ b/tests/fakes/fake_disconnect_runner.py
@@ -0,0 +1,31 @@
+"""A runner that waits on the real server approval before recording one effect."""
+
+from __future__ import annotations
+
+from jarvis.agent_chat.events import make_event
+
+
+class ApprovalRunner:
+ def __init__(self):
+ self.starts = []
+ self.effects = []
+
+ async def __call__(self, handle, prompt, runner="api", **kwargs):
+ self.starts.append(runner)
+ await handle.emit(make_event("tool_call", {
+ "turn_id": handle.turn_id, "call_id": "effect-1", "name": "Write", "input": {},
+ }))
+ answer = await handle.request_approval("effect-1", "Write", {}, "Write one result")
+ if answer == "allow":
+ self.effects.append(handle.turn_id)
+ await handle.emit(make_event("tool_result", {
+ "turn_id": handle.turn_id, "call_id": "effect-1",
+ "output": "Recorded" if answer == "allow" else "Denied", "is_error": answer != "allow",
+ }))
+ await handle.emit(make_event("assistant_text", {
+ "turn_id": handle.turn_id, "message_id": "result", "text": "The result is saved.",
+ }))
+ await handle.emit(make_event("turn_finished", {
+ "turn_id": handle.turn_id, "status": "done", "duration_ms": 1, "usage": {},
+ }))
+ return None
diff --git a/tests/fakes/fake_native_browser.py b/tests/fakes/fake_native_browser.py
index 744f0b1471..ffb9ede54c 100644
--- a/tests/fakes/fake_native_browser.py
+++ b/tests/fakes/fake_native_browser.py
@@ -128,6 +128,8 @@ class NativeDesktop:
self.co_initialized = self.co_uninitialized = 0
self.hit = 1
self.positions = []
+ self.styles = {}
+ self.alpha = {}
self.children = {}
self.focus = 0
self.user32 = SimpleNamespace(
@@ -148,6 +150,10 @@ class NativeDesktop:
GetGUIThreadInfo=NativeCall(self.gui_info),
SendMessageTimeoutW=NativeCall(self.send),
SetWindowPos=NativeCall(self.position),
+ GetWindowLongW=NativeCall(lambda hwnd, index: self.styles.get(hwnd, 0)),
+ SetWindowLongW=NativeCall(self.set_style),
+ SetLayeredWindowAttributes=NativeCall(self.set_alpha),
+ GetLayeredWindowAttributes=NativeCall(self.get_alpha),
MonitorFromWindow=NativeCall(lambda *args: 1),
GetMonitorInfoW=NativeCall(self.monitor_info),
PeekMessageW=NativeCall(lambda *args: 1),
@@ -194,6 +200,22 @@ class NativeDesktop:
self.positions.append(args)
return True
+ def set_style(self, hwnd, index, style):
+ previous = self.styles.get(hwnd, 0)
+ self.styles[hwnd] = style
+ return previous
+
+ def set_alpha(self, hwnd, color, alpha, flags):
+ self.alpha[hwnd] = alpha
+ return True
+
+ def get_alpha(self, hwnd, color, alpha, flags):
+ if hwnd not in self.alpha:
+ return False
+ alpha._obj.value = self.alpha[hwnd]
+ flags._obj.value = 2
+ return True
+
def gui_info(self, _thread, pointer):
pointer._obj.hwndFocus = self.focus
return True
@@ -232,6 +254,7 @@ class NativeDesktop:
from jarvis.society.browser.native_window import NativeWindow
native = NativeWindow.__new__(NativeWindow)
+ native._parked = False
kernel = SimpleNamespace(GetCurrentThreadId=NativeCall(lambda: 77))
native.ctypes = SimpleNamespace(
**{name: getattr(ctypes, name) for name in (
@@ -252,6 +275,7 @@ class NativeDesktop:
native._capture_factory = Capture
native._captures, native._images = {}, {}
native._capture_keys = {}
+ native._restarted_popups = set()
native._popups, native._regions, native._hooks = [], [], []
native._revision, native._encoded_revision, native._geometry_id = 0, -1, 0
native._windows_dirty = threading.Event()
diff --git a/tests/fakes/fake_openai_tool_wire.py b/tests/fakes/fake_openai_tool_wire.py
new file mode 100644
index 0000000000..420b03c61a
--- /dev/null
+++ b/tests/fakes/fake_openai_tool_wire.py
@@ -0,0 +1,24 @@
+"""OpenAI SDK stream with controllable tool JSON and termination reason."""
+
+import json
+
+import httpx
+
+
+class OpenAIToolWire:
+ def __init__(self, arguments: str, finish: str) -> None:
+ self.arguments = arguments
+ self.finish = finish
+
+ def handle(self, request: httpx.Request) -> httpx.Response:
+ deltas = [
+ ({"tool_calls": [{"index": 0, "id": "call_1", "type": "function",
+ "function": {"name": "read_file", "arguments": self.arguments}}]}, None),
+ ({}, self.finish),
+ ]
+ events = [{"id": "chatcmpl-test", "object": "chat.completion.chunk", "created": 1,
+ "model": "test", "choices": [{"index": 0, "delta": delta,
+ "finish_reason": finish}]}
+ for delta, finish in deltas]
+ body = "".join(f"data: {json.dumps(event)}\n\n" for event in events) + "data: [DONE]\n\n"
+ return httpx.Response(200, text=body, headers={"content-type": "text/event-stream"})
diff --git a/tests/fakes/fake_permission_tool.py b/tests/fakes/fake_permission_tool.py
new file mode 100644
index 0000000000..d7bb5dccca
--- /dev/null
+++ b/tests/fakes/fake_permission_tool.py
@@ -0,0 +1,19 @@
+"""A side-effect-free tool for approval-policy contracts."""
+
+from typing import Any
+
+from jarvis.core.protocols import ExecutionContext, ToolResult
+
+
+class FakePermissionTool:
+ name = "fake_write"
+ description = "Record a permitted call without performing an external action."
+ schema = {"type": "object", "properties": {}}
+
+ def __init__(self, risk_tier: str = "ask") -> None:
+ self.risk_tier = risk_tier
+ self.calls: list[ExecutionContext] = []
+
+ async def execute(self, args: dict[str, Any], ctx: ExecutionContext) -> ToolResult:
+ self.calls.append(ctx)
+ return ToolResult(success=True, output={"calls": len(self.calls)})
diff --git a/tests/fakes/fake_reset_rpc.py b/tests/fakes/fake_reset_rpc.py
new file mode 100644
index 0000000000..715f2084cb
--- /dev/null
+++ b/tests/fakes/fake_reset_rpc.py
@@ -0,0 +1,41 @@
+"""Deterministic account/reset app-server; no provider or model traffic."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+class FakeResetRpc:
+ def __init__(self, payload: dict[str, Any], outcome: Any = "reset") -> None:
+ self.payload = payload
+ self.outcome = outcome
+ self.account = {"type": "chatgpt", "email": "[email protected]"}
+ self.calls: list[tuple[str, dict]] = []
+ self.closed = False
+ self.failure: str | None = None
+ self.options: tuple[Any, ...] = ()
+
+ def factory(self, *options: Any) -> FakeResetRpc:
+ self.options = options
+ return self
+
+ async def __aenter__(self) -> FakeResetRpc:
+ return self
+
+ async def __aexit__(self, *_: Any) -> None:
+ self.closed = True
+
+ async def send(self, payload: dict) -> None:
+ self.calls.append((payload["method"], {}))
+
+ async def request(self, method: str, params: dict) -> dict:
+ self.calls.append((method, params))
+ if method == self.failure:
+ raise ConnectionError("private provider details must not escape")
+ if method == "account/read":
+ return {"account": self.account}
+ if method == "account/rateLimits/read":
+ return self.payload
+ if method == "account/rateLimitResetCredit/consume":
+ return {"outcome": self.outcome}
+ return {}
diff --git a/tests/fakes/fake_secret_store.py b/tests/fakes/fake_secret_store.py
new file mode 100644
index 0000000000..7d13b0daa5
--- /dev/null
+++ b/tests/fakes/fake_secret_store.py
@@ -0,0 +1,26 @@
+"""A dict-backed credential store, so tests never touch the real OS keyring."""
+
+from __future__ import annotations
+
+
+class FakeSecretStore:
+ """The ``SecretStore`` shape of ``jarvis.society.credentials``."""
+
+ def __init__(self, *, refuse: bool = False) -> None:
+ self.values: dict[str, str] = {}
+ self.refuse = refuse
+ self.reads = 0
+
+ def get(self, slot: str) -> str | None:
+ self.reads += 1
+ return self.values.get(slot)
+
+ def set(self, slot: str, value: str) -> bool:
+ if self.refuse:
+ return False
+ self.values[slot] = value
+ return True
+
+ def delete(self, slot: str) -> bool:
+ self.values.pop(slot, None)
+ return True
diff --git a/tests/fakes/fake_subscription_session.py b/tests/fakes/fake_subscription_session.py
index bddeeee830..89ffd79da0 100644
--- a/tests/fakes/fake_subscription_session.py
+++ b/tests/fakes/fake_subscription_session.py
@@ -62,6 +62,72 @@ class AppshotSubscriptionGateway(SubscriptionGateway):
return ToolResult(True, output)
+class PausingAgentGateway(SubscriptionGateway):
+ """An agent start remains in flight while another instruction arrives."""
+
+ def __init__(self, *, uncertain=False):
+ super().__init__()
+ self.started = asyncio.Event()
+ self.release = asyncio.Event()
+ self.requests = []
+ self.uncertain = uncertain
+
+ def catalog(self):
+ return tuple(
+ SupervisorToolDescriptor(name, name, {"type": "object"}, "monitor")
+ for name in ("start-agent", "message-agent")
+ )
+
+ async def execute(self, name, args, request):
+ self.calls.append((name, args, operation_model.get()))
+ self.requests.append(request)
+ if name == "start-agent":
+ self.started.set()
+ await self.release.wait()
+ if self.uncertain:
+ raise OSError("Receipt lost after starting the agent")
+ return ToolResult(True, {"agent_id": "agent-alpha", "verified": True})
+
+
+class ApprovalSubscriptionGateway(SubscriptionGateway):
+ def __init__(self):
+ super().__init__()
+ self.confirmed = []
+
+ def catalog(self):
+ return (SupervisorToolDescriptor(
+ "inspect-state", "Approved action fixture", {"type": "object"}, "ask",
+ is_action_tool=True,
+ ),)
+
+ async def execute(self, name, args, request):
+ from jarvis.safety.tool_executor import VOICE_CONFIRM_SENTINEL
+
+ self.calls.append((name, args, request))
+ return ToolResult(False, {}, VOICE_CONFIRM_SENTINEL)
+
+ async def execute_confirmed(self, trace, request):
+ self.confirmed.append((trace, request))
+ return ToolResult(True, {"verified": True})
+
+
+class SafeMessageGateway(SubscriptionGateway):
+ def catalog(self):
+ return (
+ SupervisorToolDescriptor(
+ "message_agent", "Message a teammate", {"type": "object"}, "safe",
+ is_action_tool=True,
+ ),
+ SupervisorToolDescriptor(
+ "workspace-orchestrate", "Inspect or change a workspace", {"type": "object"},
+ "monitor", is_action_tool=True,
+ describe_args=lambda args: {
+ "level": "read" if args.get("action") == "inspect" else "modify",
+ },
+ ),
+ )
+
+
class SubscriptionConnection:
session_id = "rtc_fake"
answer_sdp = "answer"
@@ -113,6 +179,29 @@ class ScriptedSubscriptionReasoning:
self.closed = True
+class PausingSubscriptionReasoning(ScriptedSubscriptionReasoning):
+ """Pause selected rounds before completion, including cancellation races."""
+
+ def __init__(self, rounds, *, paused=(0,)):
+ super().__init__(rounds)
+ self.entered = [asyncio.Event() for _ in rounds]
+ self.release = {index: asyncio.Event() for index in paused}
+ self.cancelled = []
+
+ async def stream(self, **request):
+ index = len(self.requests)
+ self.requests.append(copy.deepcopy(request))
+ self.entered[index].set()
+ try:
+ if index in self.release:
+ await self.release[index].wait()
+ for event in self.rounds[index]:
+ yield event
+ except asyncio.CancelledError:
+ self.cancelled.append(index)
+ raise
+
+
def subscription_provider():
return SimpleNamespace(
name="openai-live-subscription",
diff --git a/tests/missions/test_mission_tool_auto_approver.py b/tests/missions/test_mission_tool_auto_approver.py
index b38345f63a..908d271263 100644
--- a/tests/missions/test_mission_tool_auto_approver.py
+++ b/tests/missions/test_mission_tool_auto_approver.py
@@ -235,11 +235,11 @@ async def test_armed_mission_call_executes_without_human() -> None:
assert tool.approved_by == "mission-grant:m-e2e"
-async def test_unarmed_mission_call_times_out_without_running() -> None:
+async def test_unarmed_mission_in_ask_mode_times_out_without_running() -> None:
bus = EventBus()
executor = ToolExecutor(
bus,
- RiskTierEvaluator(SafetyConfig()),
+ RiskTierEvaluator(SafetyConfig(approval_mode="ask")),
ApprovalWorkflow(bus, timeout_s=0.05),
default_timeout_s=0.05,
)
diff --git a/tests/missions/test_worker_tool_broker.py b/tests/missions/test_worker_tool_broker.py
index dd9c74e9a5..9d635d7d0e 100644
--- a/tests/missions/test_worker_tool_broker.py
+++ b/tests/missions/test_worker_tool_broker.py
@@ -228,7 +228,7 @@ async def test_ask_tier_waits_then_resumes_the_exact_call_after_approval() -> No
approval = ApprovalWorkflow(bus, timeout_s=1.0)
executor = ToolExecutor(
bus,
- RiskTierEvaluator(SafetyConfig()),
+ RiskTierEvaluator(SafetyConfig(approval_mode="ask")),
approval,
default_timeout_s=1.0,
)
@@ -284,7 +284,7 @@ async def test_denied_ask_tier_never_executes_the_tool() -> None:
approval = ApprovalWorkflow(bus, timeout_s=1.0)
executor = ToolExecutor(
bus,
- RiskTierEvaluator(SafetyConfig()),
+ RiskTierEvaluator(SafetyConfig(approval_mode="ask")),
approval,
default_timeout_s=1.0,
)
diff --git a/tests/unit/agent_chat/test_agent_chat_ladders.py b/tests/unit/agent_chat/test_agent_chat_ladders.py
index e838dd581c..2ae838f909 100644
--- a/tests/unit/agent_chat/test_agent_chat_ladders.py
+++ b/tests/unit/agent_chat/test_agent_chat_ladders.py
@@ -119,7 +119,7 @@ def test_every_runner_has_a_ladder_with_its_default_on_it():
("agy-cli", "default", "accept-edits"), # ask is never folded onto plan
("agy-cli", "bypassPermissions", "skip-permissions"),
("api", "acceptEdits", "accept-edits"),
- ("api", "", "ask"),
+ ("api", "", "auto"),
("api", "read-only", "plan"),
# The Jarvis ladder's words, folded onto every seat's own spelling.
("claude-cli", "bypass", "bypassPermissions"),
@@ -132,7 +132,7 @@ def test_every_runner_has_a_ladder_with_its_default_on_it():
("agy-cli", "ask", "accept-edits"),
("grok-cli", "accept-edits", "acceptEdits"),
("grok-cli", "bypass", "bypassPermissions"),
- ("jarvis", "", "ask"),
+ ("jarvis", "", "bypass"),
("jarvis", "acceptEdits", "accept-edits"),
("jarvis", "bypassPermissions", "bypass"),
("jarvis", "plan", "plan"),
diff --git a/tests/unit/agent_chat/test_agent_chat_service.py b/tests/unit/agent_chat/test_agent_chat_service.py
index 4e127945ef..f34116f656 100644
--- a/tests/unit/agent_chat/test_agent_chat_service.py
+++ b/tests/unit/agent_chat/test_agent_chat_service.py
@@ -107,10 +107,10 @@ async def test_api_stream_preserves_complete_replies_under_the_short_default(
store.close()
-async def _drain(q: asyncio.Queue, until_kind: str, timeout: float = 5.0) -> list[dict[str, Any]]:
+async def _drain(q: asyncio.Queue, until_kind: str, budget_s: float = 5.0) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
loop = asyncio.get_running_loop()
- deadline = loop.time() + timeout
+ deadline = loop.time() + budget_s
while True:
remaining = deadline - loop.time()
if remaining <= 0:
@@ -144,7 +144,7 @@ def test_turn_streams_text_runs_a_tool_and_asks_for_approval(tmp_path: Path, scr
store = AgentChatStore(":memory:")
svc = AgentChatService(store, assistant_name=lambda: "Testo")
session = svc.create_session(
- provider="fakeprov", model="m", effort="high", cwd=str(tmp_path)
+ provider="fakeprov", model="m", effort="high", cwd=str(tmp_path), permission_mode="ask"
)
q = svc.subscribe(session.session_id)
turn_id = await svc.send(session.session_id, "please write hello.txt")
@@ -194,7 +194,7 @@ def test_deny_feeds_a_denied_result_and_allow_always_flips_the_mode(tmp_path: Pa
async def scenario() -> None:
svc = AgentChatService(AgentChatStore(":memory:"))
- session = svc.create_session(provider="fakeprov", cwd=str(tmp_path))
+ session = svc.create_session(provider="fakeprov", cwd=str(tmp_path), permission_mode="ask")
q = svc.subscribe(session.session_id)
await svc.send(session.session_id, "run things")
ev = (await _drain(q, "approval_required"))[-1]["payload"]
@@ -224,7 +224,7 @@ def test_cancel_ends_the_turn(tmp_path: Path, scripted):
async def scenario() -> None:
svc = AgentChatService(AgentChatStore(":memory:"))
- session = svc.create_session(provider="fakeprov", cwd=str(tmp_path))
+ session = svc.create_session(provider="fakeprov", cwd=str(tmp_path), permission_mode="ask")
q = svc.subscribe(session.session_id)
turn_id = await svc.send(session.session_id, "go")
await _drain(q, "approval_required")
@@ -787,7 +787,7 @@ def test_allow_always_on_a_kit_that_handles_it_does_not_flip_the_mode(
async def scenario() -> None:
svc = AgentChatService(AgentChatStore(":memory:"))
- session = svc.create_session(provider="fakeprov", cwd=str(tmp_path))
+ session = svc.create_session(provider="fakeprov", cwd=str(tmp_path), permission_mode="ask")
q = svc.subscribe(session.session_id)
await svc.send(session.session_id, "run things")
ev = (await _drain(q, "approval_required"))[-1]["payload"]
@@ -846,14 +846,24 @@ def test_a_new_process_closes_turns_a_restart_left_open(tmp_path: Path) -> None:
"""A turn whose runner died with the old process ends as failed, not Working for ever."""
db = tmp_path / "agent_chat.db"
store = AgentChatStore(db)
- open_chat = store.create_session(provider="claude-api", model="m", effort="medium", cwd=str(tmp_path))
- done_chat = store.create_session(provider="claude-api", model="m", effort="medium", cwd=str(tmp_path))
+ open_chat = store.create_session(
+ provider="claude-api", model="m", effort="medium", cwd=str(tmp_path)
+ )
+ done_chat = store.create_session(
+ provider="claude-api", model="m", effort="medium", cwd=str(tmp_path)
+ )
for sid, ts in ((open_chat.session_id, 1_000), (done_chat.session_id, 1_000)):
- store.append_event(sid, {"kind": "turn_started", "ts_ms": ts, "payload": {"turn_id": f"t-{sid}"}})
- store.append_event(sid, {"kind": "tool_call", "ts_ms": ts + 500, "payload": {"turn_id": f"t-{sid}", "call_id": "c"}})
+ store.append_event(sid, {
+ "kind": "turn_started", "ts_ms": ts, "payload": {"turn_id": f"t-{sid}"},
+ })
+ store.append_event(sid, {
+ "kind": "tool_call", "ts_ms": ts + 500,
+ "payload": {"turn_id": f"t-{sid}", "call_id": "c"},
+ })
store.append_event(
done_chat.session_id,
- {"kind": "turn_finished", "ts_ms": 2_000, "payload": {"turn_id": f"t-{done_chat.session_id}", "status": "done"}},
+ {"kind": "turn_finished", "ts_ms": 2_000,
+ "payload": {"turn_id": f"t-{done_chat.session_id}", "status": "done"}},
)
AgentChatService(AgentChatStore(db))
diff --git a/tests/unit/agent_chat/test_default_permissions.py b/tests/unit/agent_chat/test_default_permissions.py
new file mode 100644
index 0000000000..95f8e0826e
--- /dev/null
+++ b/tests/unit/agent_chat/test_default_permissions.py
@@ -0,0 +1,28 @@
+"""New Jarvis chats run autonomously while explicit permission choices persist."""
+
+from pathlib import Path
+
+import pytest
+
+from jarvis.agent_chat.service import AgentChatService
+from jarvis.agent_chat.store import AgentChatStore
+
+
[email protected]("provider", ["openai", "gemini", "claude-api"])
[email protected]("mode", ["", "ask", "accept-edits", "plan"])
+def test_jarvis_default_and_explicit_modes_survive_reopening(tmp_path: Path, provider, mode):
+ path = tmp_path / "chat.db"
+ store = AgentChatStore(path)
+ try:
+ service = AgentChatService(store)
+ session = service.create_session(
+ provider=provider, surface="jarvis", cwd=str(tmp_path), permission_mode=mode
+ )
+ assert session.permission_mode == (mode or "bypass")
+ finally:
+ store.close()
+ reopened = AgentChatStore(path)
+ try:
+ assert reopened.get_session(session.session_id).permission_mode == (mode or "bypass")
+ finally:
+ reopened.close()
diff --git a/tests/unit/agent_chat/test_queue_recovery_routes.py b/tests/unit/agent_chat/test_queue_recovery_routes.py
new file mode 100644
index 0000000000..85269fff91
--- /dev/null
+++ b/tests/unit/agent_chat/test_queue_recovery_routes.py
@@ -0,0 +1,60 @@
+"""Opening a manual chat reconciles restart-orphaned queue notices."""
+
+from __future__ import annotations
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.agent_chat.events import make_event
+from jarvis.agent_chat.service import AgentChatService
+from jarvis.agent_chat.store import AgentChatStore
+from jarvis.ui.web.agent_chat_routes import router
+
+
[email protected]("transport", ["http", "websocket"])
+def test_jarvis_restart_marks_old_waiting_message_failed_before_snapshot(transport):
+ store = AgentChatStore(":memory:")
+ session = store.create_session(
+ provider="fakeprov", model="m", effort="low", cwd=".", surface="jarvis",
+ )
+ sid = session.session_id
+ store.append_event(sid, make_event("notice", {
+ "kind": "message_queued", "queue_id": "orphaned", "text": "Please check this too.",
+ }))
+ # A tool-rich running turn used to push the queued notice outside tail=400.
+ for index in range(405):
+ store.append_event(sid, make_event("tool_result", {
+ "turn_id": "old-turn", "call_id": f"tool-{index}", "output": "Done.",
+ }))
+ fresh = AgentChatService(store)
+ app = FastAPI()
+ app.include_router(router)
+ app.state.agent_chat = fresh
+ try:
+ with TestClient(app) as client:
+ if transport == "http":
+ response = client.get(f"/api/agent-chat/sessions/{sid}")
+ assert response.status_code == 200
+ events = response.json()["events"]
+ else:
+ with client.websocket_connect(f"/api/agent-chat/sessions/{sid}/ws") as ws:
+ snapshot = ws.receive_json()
+ assert snapshot["type"] == "snapshot"
+ events = snapshot["events"]
+ closed = [
+ event["payload"] for event in events
+ if event["kind"] == "notice"
+ and event["payload"].get("kind") == "message_dequeued"
+ ]
+ assert closed == [{
+ "kind": "message_dequeued", "queue_id": "orphaned", "status": "failed",
+ "text": "Please check this too.",
+ }]
+ client.get(f"/api/agent-chat/sessions/{sid}")
+ assert sum(
+ event["payload"].get("kind") == "message_dequeued"
+ for event in store.queue_notice_events(sid)
+ ) == 1
+ finally:
+ store.close()
diff --git a/tests/unit/agent_chat/test_response_style.py b/tests/unit/agent_chat/test_response_style.py
index 06d45a351f..9c62081379 100644
--- a/tests/unit/agent_chat/test_response_style.py
+++ b/tests/unit/agent_chat/test_response_style.py
@@ -23,13 +23,36 @@ def test_api_and_main_chat_receive_the_same_reporting_policy(tmp_path: Path) ->
def test_fresh_and_resumed_defaults_allow_detail_and_preserve_essential_information() -> None:
assert CONVERSATIONAL_TURN_REMINDER in CONVERSATIONAL_RESPONSE_STYLE
for policy in (CONVERSATIONAL_RESPONSE_STYLE, CONVERSATIONAL_TURN_REMINDER):
- assert "one to three short sentences in one paragraph" in policy
+ assert "one or two short paragraphs" in policy
+ assert "someone who is not technical understands it" in policy
assert "user asks for detail or the task needs" in policy
assert "not a hard length limit" in policy
assert "blockers, uncertainty and essential questions" in policy
assert "Follow explicit user preferences" in policy
+def test_reply_style_keeps_the_person_oriented_without_a_persona() -> None:
+ # What is happening, what is done, what the person does next — in plain
+ # words, with concrete numbers and click paths instead of theory.
+ for phrase in (
+ "what is happening now, what is done, and what they can do next",
+ "Open with the state in the first words",
+ "a short yes plus what you are doing now",
+ "real numbers, times and names",
+ "exact click path",
+ "say sorry once",
+ "secure field, never into the chat",
+ "No headings, bold labels or bullet walls",
+ "your first message comes before the first tool call",
+ "no reasoning and no tool log, only your messages",
+ "Two messages in a row are fine",
+ ):
+ assert phrase in CONVERSATIONAL_RESPONSE_STYLE, phrase
+ # A general style: no persona, name or flirt register leaks in.
+ for leak in ("Jenny", "flirt", "Discord"):
+ assert leak not in CONVERSATIONAL_RESPONSE_STYLE
+
+
async def test_cli_without_brain_layers_still_receives_the_shared_default(monkeypatch) -> None:
monkeypatch.setattr(jarvis_harness, "_brain", lambda: None)
prompt = await jarvis_harness.identity_prompt(user_text="Hello", history=[], resume=None)
diff --git a/tests/unit/agent_chat/test_secret_requests.py b/tests/unit/agent_chat/test_secret_requests.py
new file mode 100644
index 0000000000..302bd0393b
--- /dev/null
+++ b/tests/unit/agent_chat/test_secret_requests.py
@@ -0,0 +1,43 @@
+"""The plain-text secret request check behind the credential guard."""
+
+from __future__ import annotations
+
+import pytest
+
+from jarvis.agent_chat.secret_requests import asks_for_secret, claims_stored_secret
+
+ASKS = [
+ # Replies seen from real agents before the guard existed.
+ "Bitte melde dich im Discord-Tab meines Browsers an; Passwörter oder Bot-Tokens gehören nicht in den Chat.", # noqa: E501 i18n-allow
+ "Bitte gib den Token ausschließlich in ein dafür vorgesehenes sicheres Eingabefeld ein, niemals hier im Chat.", # noqa: E501 i18n-allow
+ "Please paste your Discord bot token here so I can connect the bot.",
+ "I need your OpenAI API key to finish the app.",
+ "Do you have a GitHub personal access token?",
+ "Schick mir bitte deinen API-Schlüssel.", # i18n-allow
+ "Por favor, envíame la contraseña de la base de datos.", # i18n-allow
+ "请提供你的 API 密钥。", # i18n-allow
+]
+
+QUIET = [
+ "The bot reads DISCORD_BOT_TOKEN from the environment and is running.",
+ "Ich habe die App gebaut; sie liest den Schlüssel aus OPENAI_API_KEY.", # i18n-allow
+ "Done: the summary is in report.md.",
+ "A bot token identifies your Discord application.",
+ "Der Discord-Bot-Token ist gespeichert und Discord hat ihn akzeptiert.", # i18n-allow
+]
+
+
[email protected]("text", ASKS)
+def test_a_plain_text_request_for_a_secret_is_spotted(text: str) -> None:
+ assert asks_for_secret(text)
+
+
[email protected]("text", QUIET)
+def test_using_or_explaining_a_secret_is_not_a_request(text: str) -> None:
+ assert not asks_for_secret(text)
+
+
+def test_a_claim_that_a_secret_is_stored_is_spotted() -> None:
+ assert claims_stored_secret("Der Bot-Token ist bereits sicher gespeichert.") # i18n-allow
+ assert claims_stored_secret("Your API key is already stored.")
+ assert not claims_stored_secret("The bot is online.")
diff --git a/tests/unit/agent_chat/test_send_queue.py b/tests/unit/agent_chat/test_send_queue.py
index a29ea9b8cd..c12be4c3f5 100644
--- a/tests/unit/agent_chat/test_send_queue.py
+++ b/tests/unit/agent_chat/test_send_queue.py
@@ -25,7 +25,7 @@ class BusyChat:
self.fail_next = False
self.store = SimpleNamespace(
get_session=lambda sid: SimpleNamespace(session_id=sid, surface=self.surface),
- list_events=lambda sid, tail=None: [
+ queue_notice_events=lambda sid: [
{"kind": "notice", "payload": payload} for s, payload in self.notices if s == sid
],
)
@@ -119,8 +119,7 @@ async def test_a_new_message_never_overtakes_waiting_ones():
async def test_other_chats_still_refuse_while_busy():
for surface, session_id in (
- ("jarvis", "chat-1"),
- ("society", "society:jarvis"),
+ ("agent", "coding-1"),
("society", "society:scout:routine:t1:r1"),
("society", "society:scout:with:jarvis"),
):
@@ -154,10 +153,9 @@ async def test_a_message_that_fails_to_start_is_reported_and_the_next_runs():
assert "refused" not in failed[0]["text"]
-async def test_a_message_behind_a_long_running_turn_still_runs(monkeypatch):
+async def test_a_message_behind_a_long_running_turn_still_runs():
# A Hermes agent's first turn may set its runtime up for many minutes; the
# message behind it waits for that turn, not for a fixed idle timer.
- monkeypatch.setattr(send_queue, "MAX_WAIT_S", 0.03)
chat = BusyChat()
chat.busy.add("society:scout")
queue_id = (await send_or_queue(chat, "society:scout", "after the setup"))[1]
@@ -170,29 +168,91 @@ async def test_a_message_behind_a_long_running_turn_still_runs(monkeypatch):
for _, p in chat.notices)
-async def test_a_message_the_idle_chat_cannot_take_is_reported(monkeypatch):
- monkeypatch.setattr(send_queue, "MAX_WAIT_S", 0.05)
+async def test_waiting_for_a_credential_does_not_expire_accepted_messages():
chat = BusyChat()
chat.busy.add("society:scout")
- queue_id = (await send_or_queue(chat, "society:scout", "too late"))[1]
+ queue_id = (await send_or_queue(chat, "society:scout", "after the credential"))[1]
+ await asyncio.sleep(0.08)
+ assert chat.sent == []
+ assert not any(p.get("status") == "failed" for _, p in chat.notices)
chat.busy.clear()
- chat.refusing.add("society:scout") # the agent works in another chat
- await _until(lambda: any(p.get("status") == "failed" for _, p in chat.notices))
+ await _until(lambda: any(
+ p.get("queue_id") == queue_id and p.get("status") == "sent"
+ for _, p in chat.notices
+ ))
+
+
+ "surface, session_id", [("jarvis", "chat-1"), ("society", "society:jarvis")]
+)
+async def test_lead_chat_queues_without_calling_send_on_the_running_goal(surface, session_id):
+ class SteeringChat(BusyChat):
+ async def send(self, sid, text, attachments=None, *, tool_choices=None):
+ assert not self.is_running(sid), "A followup must not cancel a live goal"
+ return await super().send(sid, text, attachments, tool_choices=tool_choices)
+
+ chat = SteeringChat(surface)
+ chat.busy.add(session_id)
+ _, queue_id = await send_or_queue(chat, session_id, "followup")
+ assert queue_id and not chat.sent
+ chat.busy.clear()
+ await _until(lambda: chat.sent)
+ assert chat.sent == [(session_id, "followup")]
+
+
+async def test_followups_do_not_overtake_a_first_send_during_async_setup():
+ started, release = asyncio.Event(), asyncio.Event()
+
+ class PreparingChat(BusyChat):
+ async def send(self, sid, text, attachments=None, *, tool_choices=None):
+ if text == "first":
+ started.set()
+ await release.wait()
+ raise SessionBusy(sid) # A teammate temporarily owns the shared seat.
+ return await super().send(sid, text, attachments, tool_choices=tool_choices)
+
+ chat = PreparingChat()
+ first = asyncio.create_task(send_or_queue(chat, "society:scout", "first"))
+ await started.wait()
+ await send_or_queue(chat, "society:scout", "second")
+ await send_or_queue(chat, "society:scout", "third")
+ await asyncio.sleep(0.04)
assert chat.sent == []
- failed = [p for _, p in chat.notices if p.get("status") == "failed"]
- assert failed[0]["queue_id"] == queue_id
+ release.set()
+ await first
+ # Replace the setup-only barrier with a free seat after all admissions.
+ chat.send = BusyChat.send.__get__(chat)
+ await _until(lambda: len(chat.sent) == 3)
+ assert [text for _, text in chat.sent] == ["first", "second", "third"]
-async def test_a_message_that_waits_too_long_is_reported(monkeypatch):
- monkeypatch.setattr(send_queue, "MAX_TOTAL_WAIT_S", 0.05)
- chat = BusyChat()
+async def test_queued_messages_keep_attachments_and_their_own_timezone():
+ from jarvis.tasks.context import client_timezone
+
+ delivered = []
+
+ class ContextChat(BusyChat):
+ async def send(self, sid, text, attachments=None, *, tool_choices=None):
+ turn = await super().send(sid, text, attachments, tool_choices=tool_choices)
+ delivered.append((attachments, tool_choices, client_timezone.get()))
+ return turn
+
+ chat = ContextChat()
chat.busy.add("society:scout")
- queue_id = (await send_or_queue(chat, "society:scout", "too late"))[1]
- await _until(lambda: any(p.get("status") == "failed" for _, p in chat.notices))
- assert chat.sent == []
- failed = [p for _, p in chat.notices if p.get("status") == "failed"]
- assert failed[0]["queue_id"] == queue_id
- assert "society:scout" not in send_queue._queues(chat)
+ for timezone in ("Europe/Berlin", "America/New_York"):
+ token = client_timezone.set(timezone)
+ try:
+ await send_or_queue(
+ chat, "society:scout", timezone, [{"name": timezone}], tool_choices=[timezone]
+ )
+ finally:
+ client_timezone.reset(token)
+ chat.busy.clear()
+ await _until(lambda: len(delivered) == 2)
+ assert delivered == [
+ ([{"name": zone}], [zone], zone)
+ for zone in ("Europe/Berlin", "America/New_York")
+ ]
async def test_opening_the_chat_closes_waiting_notices_a_restart_orphaned():
@@ -210,3 +270,100 @@ async def test_opening_the_chat_closes_waiting_notices_a_restart_orphaned():
closed = chat.notices[-1][1]
assert closed["queue_id"] == "old" and closed["status"] == "failed"
assert await send_queue.close_orphans(chat, "society:scout") == 0
+
+
+async def test_orphan_recovery_keeps_a_message_admitted_during_history_read(monkeypatch):
+ chat = BusyChat(surface="jarvis")
+ chat.busy.add("main-chat")
+ original_read = chat.store.queue_notice_events
+ admitted = []
+ loop = asyncio.get_running_loop()
+
+ def read_with_admission(sid):
+ if not admitted:
+ admitted.append(asyncio.run_coroutine_threadsafe(
+ send_or_queue(chat, sid, "arrived during recovery"), loop,
+ ).result(timeout=2))
+ return original_read(sid)
+
+ monkeypatch.setattr(chat.store, "queue_notice_events", read_with_admission)
+ assert await send_queue.close_orphans(chat, "main-chat") == 0
+ assert len(admitted) == 1 and admitted[0][1]
+ assert not any(payload.get("kind") == "message_dequeued" for _, payload in chat.notices)
+ chat._society_send_queues["main-chat"].drain.cancel()
+
+
+async def test_orphan_recovery_does_not_fail_a_message_sent_during_history_read(monkeypatch):
+ chat = BusyChat(surface="jarvis")
+ chat.busy.add("main-chat")
+ original_read = chat.store.queue_notice_events
+ loop = asyncio.get_running_loop()
+ reads = []
+
+ async def finish_message():
+ chat.busy.clear()
+ await _until(lambda: bool(chat.sent))
+ await _until(lambda: any(
+ payload.get("status") == "sent" for _, payload in chat.notices
+ ))
+
+ def read_while_message_completes(sid):
+ reads.append(sid)
+ if len(reads) == 1:
+ asyncio.run_coroutine_threadsafe(
+ send_or_queue(chat, sid, "arrived and finished during recovery"), loop,
+ ).result(timeout=2)
+ stale_snapshot = original_read(sid)
+ asyncio.run_coroutine_threadsafe(finish_message(), loop).result(timeout=2)
+ return stale_snapshot
+ return original_read(sid)
+
+ monkeypatch.setattr(chat.store, "queue_notice_events", read_while_message_completes)
+ assert await send_queue.close_orphans(chat, "main-chat") == 0
+ assert len(reads) == 2
+ assert chat.sent == [("main-chat", "arrived and finished during recovery")]
+ assert [
+ payload.get("status") for _, payload in chat.notices
+ if payload.get("kind") == "message_dequeued"
+ ] == ["sent"]
+
+
+async def test_orphan_recovery_retries_at_most_once_during_continuous_admissions(monkeypatch):
+ chat = BusyChat(surface="jarvis")
+ chat.notices.append(
+ ("main-chat", {"kind": "message_queued", "queue_id": "old", "text": "lost"}),
+ )
+ original_read = chat.store.queue_notice_events
+ reads = []
+
+ def unstable_read(sid):
+ reads.append(sid)
+ chat._society_send_generation = getattr(chat, "_society_send_generation", 0) + 1
+ return original_read(sid)
+
+ monkeypatch.setattr(chat.store, "queue_notice_events", unstable_read)
+ assert await send_queue.close_orphans(chat, "main-chat") == 0
+ assert len(reads) == 2
+ assert not any(payload.get("kind") == "message_dequeued" for _, payload in chat.notices)
+
+
+async def test_concurrent_reopens_publish_one_orphan_receipt(monkeypatch):
+ chat = BusyChat(surface="jarvis")
+ chat.notices.append(
+ ("main-chat", {"kind": "message_queued", "queue_id": "old", "text": "lost"}),
+ )
+ original_post = chat.post_notice
+
+ async def slow_post(sid, payload):
+ await asyncio.sleep(0.01)
+ await original_post(sid, payload)
+
+ monkeypatch.setattr(chat, "post_notice", slow_post)
+ results = await asyncio.gather(
+ send_queue.close_orphans(chat, "main-chat"),
+ send_queue.close_orphans(chat, "main-chat"),
+ )
+ assert sum(results) == 1
+ assert sum(
+ payload.get("kind") == "message_dequeued" for _, payload in chat.notices
+ ) == 1
diff --git a/tests/unit/agent_chat/test_server_disconnect.py b/tests/unit/agent_chat/test_server_disconnect.py
new file mode 100644
index 0000000000..7c32290469
--- /dev/null
+++ b/tests/unit/agent_chat/test_server_disconnect.py
@@ -0,0 +1,88 @@
+"""All runtime dispatches retain approvals, effects and results across UI disconnects."""
+
+from __future__ import annotations
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.agent_chat import service as service_module
+from jarvis.agent_chat.service import AgentChatService
+from jarvis.agent_chat.store import AgentChatStore
+from jarvis.ui.web.agent_chat_routes import router
+from tests.fakes.fake_disconnect_runner import ApprovalRunner
+
+
[email protected](("runtime", "runner_name"), [
+ ("", "brain"), ("hermes", "hermes-cli"), ("openclaw", "openclaw-cli"),
+])
+def test_client_disconnect_reconnect_and_completed_restart(
+ tmp_path, monkeypatch, runtime, runner_name,
+):
+ database = tmp_path / "chat.db"
+ store = AgentChatStore(database)
+ service = AgentChatService(store)
+ session = store.create_session(
+ session_id="society:server-test", provider="openai", model="fake-model", effort="",
+ surface="society", runtime=runtime, permission_mode="ask", cwd=str(tmp_path),
+ )
+
+ async def bind(*args, **kwargs):
+ return session
+
+ fake = ApprovalRunner()
+ monkeypatch.setattr(service, "bind_society_session", bind)
+ monkeypatch.setattr(service_module, "run_cli_turn", fake)
+
+ async def brain(handle, prompt, **kwargs):
+ return await fake(handle, prompt, "brain")
+
+ monkeypatch.setattr(service_module, "run_brain_turn", brain)
+ app = FastAPI()
+ app.include_router(router)
+ app.state.agent_chat = service
+ path = f"/api/agent-chat/sessions/{session.session_id}"
+ with TestClient(app) as client:
+ with client.websocket_connect(path + "/ws") as ws:
+ ws.receive_json()
+ response = client.post(path + "/messages", json={"text": "Write one result"})
+ assert response.status_code == 202, response.text
+ while True:
+ frame = ws.receive_json()
+ event = frame.get("event", {})
+ if event.get("kind") == "approval_required":
+ approval = event["payload"]["approval_id"]
+ seq = event["seq"]
+ break
+ assert event.get("kind") != "turn_finished", frame
+
+ # No UI subscriber remains. Disconnecting must not grant or cancel anything.
+ assert service.is_running(session.session_id)
+ assert fake.effects == []
+ assert fake.starts == [runner_name]
+ with client.websocket_connect(path + f"/ws?after={seq}") as ws:
+ snapshot = ws.receive_json()
+ assert snapshot["session"]["running"]
+ assert approval in snapshot["session"]["pending_approvals"]
+ assert all(event["seq"] > seq for event in snapshot["events"])
+ response = client.post(path + f"/approvals/{approval}", json={"decision": "allow"})
+ assert response.status_code == 200, response.text
+ while True:
+ event = ws.receive_json().get("event", {})
+ if event.get("kind") == "turn_finished":
+ assert event["payload"]["status"] == "done"
+ break
+ with client.websocket_connect(path + "/ws") as ws:
+ events = ws.receive_json()["events"]
+ assert sum(e["kind"] == "assistant_text" for e in events) == 1
+ assert sum(e["kind"] == "turn_finished" for e in events) == 1
+ assert len(fake.effects) == 1
+ assert fake.starts == [runner_name]
+ store.close()
+ reopened = AgentChatStore(database)
+ try:
+ events = reopened.list_events(session.session_id)
+ assert sum(e["kind"] == "tool_result" for e in events) == 1
+ assert sum(e["kind"] == "turn_finished" for e in events) == 1
+ finally:
+ reopened.close()
diff --git a/tests/unit/agent_chat/test_society_memory_refresh.py b/tests/unit/agent_chat/test_society_memory_refresh.py
index c698112efd..bdb7f6d99e 100644
--- a/tests/unit/agent_chat/test_society_memory_refresh.py
+++ b/tests/unit/agent_chat/test_society_memory_refresh.py
@@ -36,3 +36,15 @@ def test_cleared_memory_explicitly_replaces_previous_snapshot():
output = society_memory_refresh("## You are Scout\n## Your memory\nNothing remembered yet.")
assert "Nothing remembered yet." in output
assert "An absent section has no current entries" in output
+
+
+def test_resumed_cli_gets_the_credential_rule_and_the_current_stored_list():
+ text = (
+ "## You are Finn\nDiscord moderator\n\n## Standing instructions\nModerate.\n"
+ "\nClient timezone for this turn: Europe/Berlin."
+ "\nStored credentials: none. Ask with society_request_credential when a task needs one."
+ )
+ identity = Identity(session_id="society:finn", text=text, compact=text)
+ prompt = _with_identity("Use a token, please.", identity, "vendor-session")
+ assert "Credentials are mandatory tool work" in prompt
+ assert "Stored credentials: none." in prompt
diff --git a/tests/unit/agent_chat/test_turn_completion.py b/tests/unit/agent_chat/test_turn_completion.py
index ae4f6cf588..e10cd1fda7 100644
--- a/tests/unit/agent_chat/test_turn_completion.py
+++ b/tests/unit/agent_chat/test_turn_completion.py
@@ -313,3 +313,78 @@ async def test_service_continues_to_artifact_after_question_without_another_user
await svc.cancel(session.session_id)
await rt.close()
svc.store.close()
+
+
+# ------------------------------------------------------------------ credential guard
+
+
+def stored(names):
+ async def fake(_session_id):
+ return names
+
+ return fake
+
+
+async def finish_with(gate, *events):
+ for event in events:
+ await gate.emit(event)
+ await gate.emit(make_event("turn_finished", {"status": "done", "usage": {}}))
+
+
+async def test_a_secret_asked_for_in_the_reply_is_steered_to_the_secure_field_once(
+ card_world, monkeypatch
+):
+ from jarvis.agent_chat import turn_completion
+
+ monkeypatch.setattr(turn_completion, "_stored_credentials", stored([]))
+ _, _, gate = card_world
+ ask = make_event(
+ "assistant_text", {"text": "Please paste your Discord bot token here so I can connect."}
+ )
+ await finish_with(gate, ask)
+ prompt = await gate.next_prompt()
+ assert "society_request_credential" in prompt and "Stored credentials right now: none" in prompt
+ assert gate.request in prompt
+ await finish_with(gate, ask)
+ assert await gate.next_prompt() is None
+
+
+async def test_a_turn_that_opened_the_field_is_left_alone(card_world, monkeypatch):
+ from jarvis.agent_chat import turn_completion
+
+ monkeypatch.setattr(turn_completion, "_stored_credentials", stored([]))
+ _, _, gate = card_world
+ call = make_event(
+ "tool_call",
+ {"call_id": "c", "name": "mcp__jarvis__society_request_credential", "input": {}},
+ )
+ await finish_with(
+ gate, call, make_event("assistant_text", {"text": "Please enter the token in the field."})
+ )
+ assert await gate.next_prompt() is None
+
+
+async def test_a_claimed_token_that_is_not_stored_is_corrected(card_world, monkeypatch):
+ from jarvis.agent_chat import turn_completion
+
+ _, _, gate = card_world
+ claim = make_event(
+ "assistant_text", {"text": "Der Bot-Token ist bereits sicher gespeichert."} # i18n-allow
+ )
+ monkeypatch.setattr(turn_completion, "_stored_credentials", stored(["DISCORD_BOT_TOKEN"]))
+ await finish_with(gate, claim)
+ assert await gate.next_prompt() is None # it really is stored
+ gate.credential_corrected = False
+ monkeypatch.setattr(turn_completion, "_stored_credentials", stored([]))
+ assert "society_request_credential" in await gate.next_prompt()
+
+
+async def test_chats_without_the_credential_tool_are_never_steered(card_world, monkeypatch):
+ from jarvis.agent_chat import turn_completion
+
+ monkeypatch.setattr(turn_completion, "_stored_credentials", stored(None))
+ _, _, gate = card_world
+ await finish_with(gate, make_event("assistant_text", {"text": "I need your API key."}))
+ prompt = await gate.next_prompt()
+ assert prompt is None or "society_request_credential" not in prompt
+ assert not gate.credential_corrected
diff --git a/tests/unit/agent_chat/test_turn_host.py b/tests/unit/agent_chat/test_turn_host.py
index d10df36e87..dff0344a22 100644
--- a/tests/unit/agent_chat/test_turn_host.py
+++ b/tests/unit/agent_chat/test_turn_host.py
@@ -668,9 +668,22 @@ def test_concurrent_first_requests_build_only_one_chat_service() -> None:
assert all(svc is state.agent_chat for svc in services)
-async def test_reattaching_twice_hands_over_the_old_reader_without_killing_the_cli(tmp_path):
+async def test_reattaching_twice_hands_over_the_old_reader_without_killing_the_cli(tmp_path, monkeypatch):
host, task, port = await _start_host(tmp_path)
client = await _client(port)
+ attach = host._attach
+ attachments = 0
+
+ async def interleaved(peer, rid, turn):
+ nonlocal attachments
+ attachments += 1
+ if attachments == 2:
+ # A live frame can reach the replacement before the host handles
+ # its attach request. Replay must still start at the first line.
+ await peer.send({"ev": "line", "id": turn.turn_id, "seq": 2, "d": "two"})
+ await attach(peer, rid, turn)
+
+ monkeypatch.setattr(host, "_attach", interleaved)
try:
spawned = await client.spawn(
[sys.executable, "-c", _CHILD],
@@ -707,6 +720,36 @@ async def test_reattaching_twice_hands_over_the_old_reader_without_killing_the_c
await _stop(host, task)
+async def test_failed_attach_can_retry_without_leaving_a_dead_reader(tmp_path, monkeypatch):
+ host, task, port = await _start_host(tmp_path)
+ client = await _client(port)
+ try:
+ spawned = await client.spawn(
+ [sys.executable, "-c", _CHILD], cwd=str(tmp_path), env=None,
+ stdin="hi\n", keep_stdin=True, meta={"turn_id": "retry-attach", "keep_stdin": True},
+ )
+ request = client._request
+
+ async def refused(*_args, **_kwargs):
+ raise ConnectionError("attach reply unavailable")
+
+ monkeypatch.setattr(client, "_request", refused)
+ with pytest.raises(ConnectionError, match="attach reply unavailable"):
+ await client.attach(spawned.host_id)
+ assert client.live() == []
+ monkeypatch.setattr(client, "_request", request)
+ resumed = await client.attach(spawned.host_id)
+ assert resumed is not None
+ assert await resumed.stdout.readline() == b"got hi\n"
+ assert resumed.stdin is not None
+ resumed.stdin.write(b"go\n")
+ assert await asyncio.wait_for(resumed.wait(), 3) == 3
+ resumed.release()
+ finally:
+ client.detach()
+ await _stop(host, task)
+
+
async def test_duplicate_recovery_publishes_one_success_and_no_late_timeout(tmp_path):
from jarvis.agent_chat.turn_host_client import TurnHandedOver
diff --git a/tests/unit/agent_chat/test_turn_prompts.py b/tests/unit/agent_chat/test_turn_prompts.py
index 85d14a6691..c1b1e06595 100644
--- a/tests/unit/agent_chat/test_turn_prompts.py
+++ b/tests/unit/agent_chat/test_turn_prompts.py
@@ -191,13 +191,13 @@ def test_no_card_for_a_failed_turn_or_plain_text(tmp_path: Path) -> None:
@pytest.mark.parametrize(
("provider", "runner", "build"),
[
- ("openai-codex", "codex-cli", "auto"),
- ("antigravity", "agy-cli", "accept-edits"),
- ("grok-build", "grok-cli", "acceptEdits"),
- ("opencode", "opencode-cli", "default"),
+ ("openai-codex", "codex-cli", "full-access"),
+ ("antigravity", "agy-cli", "skip-permissions"),
+ ("grok-build", "grok-cli", "bypassPermissions"),
+ ("opencode", "opencode-cli", "auto"),
("kimi", "kimi-cli", "auto"),
("cursor", "cursor-cli", "auto"),
- ("claude-api", "claude-cli", "acceptEdits"),
+ ("claude-api", "claude-cli", "bypassPermissions"),
],
)
def test_a_finished_plan_gets_a_build_card(
@@ -455,10 +455,10 @@ def test_an_approved_claude_plan_leaves_plan_mode(
(body,) = answers
assert body["behavior"] == "allow"
assert body["updatedPermissions"] == [
- {"type": "setMode", "mode": "acceptEdits", "destination": "session"}
+ {"type": "setMode", "mode": "bypassPermissions", "destination": "session"}
]
- assert chat.updates == [{"permission_mode": "acceptEdits"}]
- assert {"permission_mode": "acceptEdits"} in [
+ assert chat.updates == [{"permission_mode": "bypassPermissions"}]
+ assert {"permission_mode": "bypassPermissions"} in [
e["payload"] for e in events if e["kind"] == "session_updated"
]
diff --git a/tests/unit/agent_runtimes/test_acp_runner.py b/tests/unit/agent_runtimes/test_acp_runner.py
index 2b2eeac15a..a9b942d601 100644
--- a/tests/unit/agent_runtimes/test_acp_runner.py
+++ b/tests/unit/agent_runtimes/test_acp_runner.py
@@ -54,6 +54,7 @@ def _turn(
report_session: str | None = None,
argv: list[str] | None = None,
released: list[int] | None = None,
+ cancel_on_delta: bool = False,
) -> tuple[str | None, list[dict[str, Any]], list[tuple[str, str]], list[str | None]]:
resumes: list[str | None] = []
@@ -86,6 +87,8 @@ def _turn(
async def emit(event: dict[str, Any]) -> None:
events.append(event)
+ if cancel_on_delta and event["kind"] == "text_delta":
+ handle.cancel.set()
async def ask(call_id: str, name: str, args: dict[str, Any], summary: str) -> str:
asked.append((call_id, name))
@@ -115,6 +118,28 @@ def test_the_runtime_runners_count_as_cli_seats():
assert rc.supports_cli_runner("openclaw-cli")
+def test_acp_eof_without_prompt_result_is_incomplete(monkeypatch, tmp_path):
+ _, events, _, _ = _turn(monkeypatch, tmp_path, "EOF_PARTIAL")
+ assert _finished(events)["status"] == "error"
+ assert "terminal result" in _finished(events)["error"]
+
+
+def test_acp_token_exhaustion_keeps_partial_text_but_reports_error(monkeypatch, tmp_path):
+ _, events, _, _ = _turn(monkeypatch, tmp_path, "TRUNCATE")
+ assert _texts(events) == ["echo: TRUNCATE"]
+ assert _finished(events)["status"] == "error"
+ assert "max_tokens" in _finished(events)["error"]
+
+
+def test_stop_sends_upstream_acp_cancel_before_terminating_bridge(monkeypatch, tmp_path):
+ log = tmp_path / "frames.jsonl"
+ monkeypatch.setenv("FAKE_ACP_LOG", str(log))
+ _, events, _, _ = _turn(monkeypatch, tmp_path, "WAIT_CANCEL", cancel_on_delta=True)
+ frames = [json.loads(line) for line in log.read_text(encoding="utf-8").splitlines()]
+ assert any(frame.get("method") == "session/cancel" for frame in frames)
+ assert _finished(events)["status"] == "cancelled"
+
+
def test_a_fresh_turn_streams_text_reasoning_and_usage(monkeypatch, tmp_path):
vendor, events, asked, _ = _turn(monkeypatch, tmp_path, "hello")
assert vendor # the ACP session id is kept for the next turn
@@ -239,6 +264,7 @@ def model_gateway(monkeypatch):
if state["limited"]:
raise Limit("private provider body")
yield BrainDelta(content="I review contributor pull requests for security issues.")
+ yield BrainDelta(finish_reason="stop")
monkeypatch.setattr(gateway, "_deltas", deltas)
gateway.reset()
@@ -277,6 +303,7 @@ def test_gateway_failure_ends_the_real_acp_runner_and_recovery_keeps_the_questio
endpoint, state = model_gateway
released = []
+ invalidated = []
class Driver:
label = "Fake runtime"
@@ -299,6 +326,7 @@ def test_gateway_failure_ends_the_real_acp_runner_and_recovery_keeps_the_questio
cwd=tmp_path,
acp_resume=turn.resume,
release=lambda: released.append(1),
+ invalidate=lambda: invalidated.append(1),
)
async def agent(_agent_id):
@@ -349,6 +377,7 @@ def test_gateway_failure_ends_the_real_acp_runner_and_recovery_keeps_the_questio
assert "/continue" not in _finished(events)["error"]
assert _texts(events) == []
assert len(state["calls"]) == 1 and released == [1]
+ assert invalidated == [1]
assert not gateway._FAILURES
history = json.loads((tmp_path / "gateway-store.json").read_text(encoding="utf-8"))
assert any("Was ist deine Aufgabe" in prompt for turns in history.values() for prompt in turns)
@@ -360,6 +389,7 @@ def test_gateway_failure_ends_the_real_acp_runner_and_recovery_keeps_the_questio
assert _finished(recovered)["status"] == "done"
assert _texts(recovered) == ["I review contributor pull requests for security issues."]
assert len(state["calls"]) == 2 and released == [1, 1]
+ assert invalidated == [1] # Healthy recovery keeps its persistent gateway.
assert not gateway._FAILURES
@@ -502,9 +532,9 @@ def test_a_runtime_that_exits_without_answering_is_an_error(monkeypatch, tmp_pat
assert "3" in (finished["error"] or "")
-def test_a_cut_off_answer_stands_and_says_so(monkeypatch, tmp_path):
+def test_a_cut_off_answer_is_retained_and_reported_incomplete(monkeypatch, tmp_path):
events, _ = _run_live(monkeypatch, tmp_path, "MAXTOK")
- assert _finished(events)["status"] == "done"
+ assert _finished(events)["status"] == "error"
assert _texts(events) == ["echo: MAXTOK"]
notices = [e["payload"] for e in events if e["kind"] == "notice"]
assert [n["stop_reason"] for n in notices] == ["max_tokens"]
@@ -532,4 +562,4 @@ def test_a_detached_child_of_the_runtime_is_reaped_with_the_turn(monkeypatch, tm
deadline = time.monotonic() + 5
while running() and time.monotonic() < deadline:
time.sleep(0.1)
- assert not running()
\ No newline at end of file
+ assert not running()
diff --git a/tests/unit/agent_runtimes/test_full_model_capacity.py b/tests/unit/agent_runtimes/test_full_model_capacity.py
index fa49df7e17..a06e50cb69 100644
--- a/tests/unit/agent_runtimes/test_full_model_capacity.py
+++ b/tests/unit/agent_runtimes/test_full_model_capacity.py
@@ -69,7 +69,7 @@ async def test_ollama_really_allocates_the_advertised_context(monkeypatch):
monkeypatch.setattr(brain, "_resolve_root", lambda: "http://local.invalid")
seen = []
- async def profile(root, model, options):
+ async def profile(root, model, options, *, headers=None):
seen.append(options)
return "prepared-profile"
diff --git a/tests/unit/agent_runtimes/test_provider_availability.py b/tests/unit/agent_runtimes/test_provider_availability.py
new file mode 100644
index 0000000000..9ec963891c
--- /dev/null
+++ b/tests/unit/agent_runtimes/test_provider_availability.py
@@ -0,0 +1,113 @@
+"""The picker must not offer a login that runtime startup already rejects."""
+
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.agent_runtimes import model_map, provider_errors
+from jarvis.core import config
+
+
+def claude_only(monkeypatch):
+ monkeypatch.setattr(model_map, "_ENDPOINTS", {"claude-api": model_map._ENDPOINTS["claude-api"]})
+ monkeypatch.setattr(model_map, "claude_login_token", lambda: "fixture-login")
+ monkeypatch.setattr(config, "get_jarvis_agent_secret", lambda provider: None)
+ monkeypatch.setattr(
+ config, "resolve_provider_endpoint", lambda *args, **kwargs: SimpleNamespace(
+ credential=None, base_url="https://api.anthropic.com", via_proxy=False,
+ ),
+ )
+ monkeypatch.setattr(provider_errors, "_REPORTS", {})
+
+
+ {"is_enabled": False}, {"is_enabled": True, "spend_limit_reached": True},
+])
+def test_blocked_login_is_absent_from_every_picker_list(claude_only, monkeypatch, extra):
+ calls = []
+
+ def usage(token):
+ calls.append(token)
+ return {"extra_usage": extra}
+
+ monkeypatch.setattr(provider_errors, "_claude_usage", usage)
+ assert model_map.usable_providers(None) == []
+ assert model_map.login_providers() == []
+ assert model_map.access_choices() == {}
+ assert calls == ["fixture-login"]
+ with pytest.raises(model_map.RouteUnavailable, match="Extra Usage"):
+ model_map.route_for(None, "claude-api", "fixture-model", account_id="subscription")
+
+
+def test_blocked_login_does_not_hide_a_separate_api_key(claude_only, monkeypatch):
+ monkeypatch.setattr(config, "get_jarvis_agent_secret", lambda provider: "fixture-api-key")
+ monkeypatch.setattr(provider_errors, "_claude_usage", lambda token: {
+ "extra_usage": {"is_enabled": False},
+ })
+ assert model_map.usable_providers(None) == ["claude-api"]
+ assert model_map.login_providers() == []
+ assert model_map.access_choices() == {"claude-api": ["api"]}
+ with pytest.raises(model_map.RouteUnavailable):
+ model_map.route_for(None, "claude-api", "fixture-model", account_id="subscription")
+
+
[email protected]("usage", [None, {}, {"extra_usage": {"is_enabled": True}}])
+def test_unknown_or_enabled_usage_does_not_invent_a_block(claude_only, monkeypatch, usage):
+ monkeypatch.setattr(provider_errors, "_claude_usage", lambda token: usage)
+ assert model_map.usable_providers(None) == ["claude-api"]
+ assert model_map.login_providers() == ["claude-api"]
+ assert model_map.access_choices() == {"claude-api": ["subscription"]}
+
+
+def test_missing_login_never_requests_usage(claude_only, monkeypatch):
+ monkeypatch.setattr(model_map, "claude_login_token", lambda: None)
+
+ def unexpected(token):
+ pytest.fail("A disconnected account must not request usage")
+
+ monkeypatch.setattr(provider_errors, "_claude_usage", unexpected)
+ assert model_map.usable_providers(None) == []
+ assert model_map.login_providers() == []
+ assert model_map.access_choices() == {}
+
+
+def test_usage_outage_is_cached_briefly_and_recovers(claude_only, monkeypatch):
+ import time
+
+ now = [100.0]
+ calls = []
+
+ def unavailable(token):
+ calls.append(token)
+ return None
+
+ monkeypatch.setattr(time, "monotonic", lambda: now[0])
+ monkeypatch.setattr(provider_errors, "_claude_usage", unavailable)
+ assert model_map.usable_providers(None) == ["claude-api"]
+ assert model_map.login_providers() == ["claude-api"]
+ assert model_map.access_choices() == {"claude-api": ["subscription"]}
+ assert calls == ["fixture-login"]
+
+ now[0] += 11.0
+ monkeypatch.setattr(provider_errors, "_claude_usage", lambda token: {
+ "extra_usage": {"is_enabled": False},
+ })
+ assert model_map.usable_providers(None) == []
+
+
+def test_picker_recovers_after_extra_usage_is_enabled(claude_only, monkeypatch):
+ import time
+
+ now = [100.0]
+ monkeypatch.setattr(time, "monotonic", lambda: now[0])
+ monkeypatch.setattr(provider_errors, "_claude_usage", lambda token: {
+ "extra_usage": {"is_enabled": False},
+ })
+ assert model_map.usable_providers(None) == []
+ now[0] += 121.0
+ monkeypatch.setattr(provider_errors, "_claude_usage", lambda token: {
+ "extra_usage": {"is_enabled": True},
+ })
+ assert model_map.usable_providers(None) == ["claude-api"]
+ assert model_map.access_choices() == {"claude-api": ["subscription"]}
diff --git a/tests/unit/agent_runtimes/test_provider_transport_regressions.py b/tests/unit/agent_runtimes/test_provider_transport_regressions.py
new file mode 100644
index 0000000000..1db0c3fbec
--- /dev/null
+++ b/tests/unit/agent_runtimes/test_provider_transport_regressions.py
@@ -0,0 +1,450 @@
+"""Provider boundaries exercised with SDKs and local, scripted transports."""
+
+import json
+
+import httpx
+import pytest
+from anthropic import AsyncAnthropic
+
+from jarvis.agent_runtimes import gateway
+from jarvis.core.protocols import BrainDelta, BrainMessage, BrainRequest
+from jarvis.plugins.brain._anthropic_base import stream_complete
+from tests.fakes.fake_anthropic_wire import AnthropicWire
+
+
+def isolated_gateway():
+ gateway.reset()
+ yield
+ gateway.reset()
+
+
+async def test_claude_tool_history_uses_the_declared_names_without_mutating_history():
+ wire = AnthropicWire()
+ blocks = [
+ {"type": "tool_use", "id": "one", "name": "github/search", "input": {}},
+ {"type": "tool_use", "id": "two", "name": "github_search", "input": {}},
+ ]
+ request = BrainRequest(messages=(
+ BrainMessage("user", "Search"), BrainMessage("assistant", blocks),
+ BrainMessage("tool", "first", tool_call_id="one"),
+ BrainMessage("tool", "second", tool_call_id="two"),
+ ), tools=({"name": "github/search"}, {"name": "github_search"}))
+ async with AsyncAnthropic(api_key="test-only", http_client=wire.client()) as client:
+ assert [d async for d in stream_complete(client, "claude-sonnet-4-6", request)]
+ payload = json.loads(wire.requests[0].content)
+ declared = [tool["name"] for tool in payload["tools"]]
+ history = [block["name"] for block in payload["messages"][1]["content"]]
+ assert history == declared
+ assert blocks[0]["name"] == "github/search"
+
+
+async def test_claude_agent_timeout_reaches_the_sdks_own_http_transport():
+ from jarvis.plugins.brain import _agent_profile
+
+ wire = AnthropicWire()
+ token = _agent_profile.PROFILE.set(_agent_profile.AgentRequestProfile(read_timeout_s=301.0))
+ try:
+ async with AsyncAnthropic(api_key="test-only", http_client=wire.client()) as client:
+ result = [delta async for delta in stream_complete(
+ client, "claude-sonnet-4-6", BrainRequest(messages=(BrainMessage("user", "Hi"),)),
+ )]
+ assert any(delta.content == "OK" for delta in result)
+ assert wire.requests[0].extensions["timeout"] == {
+ "connect": 5.0, "read": 301.0, "write": 60.0, "pool": 30.0,
+ }
+ finally:
+ _agent_profile.PROFILE.reset(token)
+
+
[email protected]("intermediate_usage", [False, True])
+async def test_claude_cache_preserves_oauth_header_and_stream_usage(
+ monkeypatch, intermediate_usage,
+):
+ monkeypatch.setenv("JARVIS_ANTHROPIC_PROMPT_CACHE", "1")
+ wire = AnthropicWire()
+ wire.intermediate_usage = intermediate_usage
+ async with AsyncAnthropic(api_key="", auth_token="test-only", # noqa: S106 - local fake
+ default_headers={"anthropic-beta": "oauth-2025-04-20"},
+ http_client=wire.client(),
+ ) as client:
+ deltas = [d async for d in stream_complete(client, "claude-sonnet-4-6",
+ BrainRequest(messages=(BrainMessage("user", "Hi"),), system="Be brief."))]
+ assert set(wire.requests[0].headers["anthropic-beta"].split(",")) == {
+ "oauth-2025-04-20", "extended-cache-ttl-2025-04-11",
+ }
+ usage = {key: sum((d.usage or {}).get(key, 0) for d in deltas) for key in (
+ "input_tokens", "output_tokens", "cache_hit_tokens", "cache_write_tokens",
+ )}
+ assert usage == {"input_tokens": 100, "output_tokens": 2,
+ "cache_hit_tokens": 20, "cache_write_tokens": 5}
+
+
[email protected]("provider", ["ollama", "claude-api", "openai", "gemini",
+ "grok", "openrouter", "nvidia", "local-openai"])
[email protected]("streamed", [False, True])
[email protected]("partial", [False, True])
+async def test_gateway_never_marks_unfinished_provider_streams_complete(
+ monkeypatch, isolated_gateway, provider, streamed, partial,
+):
+ async def deltas(*args):
+ if partial:
+ yield BrainDelta(content="unfinished")
+ monkeypatch.setattr(gateway, "_deltas", deltas)
+ token = gateway.grant_token("agent", provider)
+ signal = gateway.watch_failure(token)
+ grant = gateway.verify(token)
+ request = BrainRequest(messages=(BrainMessage("user", "Hello"),))
+ if streamed and partial:
+ stream = await gateway.open_chat_stream(grant, "model", request)
+ chunks = b"".join([chunk async for chunk in stream])
+ assert b'"error"' in chunks
+ assert b"[DONE]" not in chunks
+ else:
+ with pytest.raises(gateway.GatewayError) as failed:
+ if streamed:
+ await gateway.open_chat_stream(grant, "model", request)
+ else:
+ await gateway.complete_chat(grant, "model", request)
+ assert failed.value.code == "incomplete"
+ assert signal.done()
+
+
+async def test_subscription_pin_cannot_fall_back_to_api_key_after_login_expires(
+ monkeypatch, isolated_gateway,
+):
+ from jarvis.agent_chat import runner_api
+ from jarvis.agent_chat.catalog import SUBSCRIPTION_ACCOUNT
+ from jarvis.agent_runtimes import model_map
+ from jarvis.core import config
+
+ monkeypatch.setattr(model_map, "claude_login_token", lambda: None)
+ monkeypatch.setattr(config, "get_jarvis_agent_secret", lambda provider: "test-api-key")
+ called = []
+ monkeypatch.setattr(runner_api, "build_brain", lambda *args: called.append(args))
+ grant = gateway.Grant("agent", "claude-api", SUBSCRIPTION_ACCOUNT)
+ with pytest.raises(Exception, match="Claude Code login"):
+ await anext(gateway._deltas(grant, "claude-sonnet-4-6", BrainRequest(messages=())))
+ assert called == []
+
+
[email protected]("arguments", ['{"path":', '[]', 'null'])
+async def test_claude_never_emits_malformed_executable_tools(arguments):
+ wire = AnthropicWire()
+ wire.tool_arguments = arguments
+ received = []
+ async with AsyncAnthropic(api_key="test-only", http_client=wire.client()) as client:
+ with pytest.raises(ValueError, match="tool arguments"):
+ async for delta in stream_complete(client, "claude-sonnet-4-6",
+ BrainRequest(messages=(BrainMessage("user", "Read"),))):
+ received.append(delta)
+ assert not any(d.tool_call for d in received)
+
+
+def test_standard_tool_reply_restores_name_required_by_gemini():
+ _, request = gateway.chat_request({"model": "gemini-test", "messages": [
+ {"role": "assistant", "tool_calls": [{"id": "c1", "type": "function",
+ "function": {"name": "read_file", "arguments": "{}"}}]},
+ {"role": "tool", "tool_call_id": "c1", "content": "contents"},
+ ]})
+ assert request.messages[1].name == "read_file"
+
+
[email protected]("state", ["idle", "released", "failed", "cancelled"])
+async def test_persistent_runtime_token_cannot_infer_outside_live_turn(
+ monkeypatch, isolated_gateway, state,
+):
+ from jarvis.agent_chat import runner_api
+
+ token = gateway.grant_token("agent", "openai", require_active_turn=True)
+ if state != "idle":
+ signal = gateway.watch_failure(token)
+ if state == "released":
+ gateway.unwatch_failure(token, signal)
+ elif state == "failed":
+ signal.set_result("provider failure")
+ else:
+ signal.cancel()
+ calls = []
+ monkeypatch.setattr(runner_api, "build_brain", lambda *args: calls.append(args))
+ with pytest.raises(gateway.GatewayError, match="no active turn"):
+ await anext(gateway._deltas(gateway.verify(token), "model", BrainRequest(messages=())))
+ assert not calls
+
+
+async def test_selected_effort_reaches_api_brain(monkeypatch, isolated_gateway):
+ from jarvis.agent_chat import runner_api
+ from jarvis.agent_runtimes import model_map
+ from jarvis.core import config
+
+ requests = []
+
+ class Brain:
+ async def complete(self, request):
+ requests.append(request)
+ yield BrainDelta(content="OK", finish_reason="stop")
+
+ monkeypatch.setattr(config, "get_jarvis_agent_secret", lambda provider: None)
+ monkeypatch.setattr(model_map, "login_route", lambda *args: (False, None))
+ monkeypatch.setattr(runner_api, "build_brain", lambda *args: Brain())
+ token = gateway.grant_token("agent", "ollama", require_active_turn=True)
+ gateway.watch_failure(token, effort="high")
+ assert [d async for d in gateway._deltas(gateway.verify(token), "model",
+ BrainRequest(messages=(), reasoning_effort="low"))]
+ assert requests[0].reasoning_effort == "high"
+
+
+def test_openclaw_workshop_cannot_start_background_model_turns(tmp_path):
+ from jarvis.agent_runtimes.openclaw import OpenClawRuntime
+ from tests.unit.agent_runtimes.test_runtime_configs import _turn
+
+ config = OpenClawRuntime().config_for(
+ _turn(tmp_path), port=12345, token="test-only", # noqa: S106 - local fake
+ )
+ assert config["skills"]["workshop"]["autonomous"]["mode"] == "off"
+
+
[email protected]("arguments,finish", [('{"path":', "tool_calls"),
+ ('[]', "tool_calls"), ('null', "tool_calls"), ('{"path":"a"}', "length")])
+async def test_openai_compatible_streams_never_emit_broken_tool_calls(arguments, finish):
+ from openai import AsyncOpenAI
+
+ from jarvis.plugins.brain._openai_base import stream_complete as stream_openai
+ from tests.fakes.fake_openai_tool_wire import OpenAIToolWire
+
+ wire = OpenAIToolWire(arguments, finish)
+ received = []
+ async with AsyncOpenAI(api_key="test-only", http_client=httpx.AsyncClient(
+ transport=httpx.MockTransport(wire.handle),
+ )) as client:
+ if finish == "length":
+ received = [delta async for delta in stream_openai(client, "test", BrainRequest(
+ messages=(BrainMessage("user", "Read"),),
+ ))]
+ assert any(delta.finish_reason == "length" for delta in received)
+ else:
+ with pytest.raises(ValueError, match="tool"):
+ async for delta in stream_openai(client, "test", BrainRequest(messages=(
+ BrainMessage("user", "Read"),))):
+ received.append(delta)
+ assert not any(delta.tool_call for delta in received)
+
+
+async def test_ollama_native_calls_forward_resolved_endpoint_auth(monkeypatch, tmp_path):
+ from jarvis.brain import ollama_profiles
+ from jarvis.brain.model_catalog import ModelCatalog
+ from jarvis.core import config
+ from jarvis.plugins.brain.ollama import OllamaBrain
+
+ seen = []
+ models = [{"name": "test:latest", "size": 100}]
+ root = "http://ollama.test/p/ollama"
+
+ def respond(request):
+ seen.append(request)
+ assert request.headers.get("Authorization") == "Bearer test-proxy-token"
+ if request.url.path.endswith("/api/tags"):
+ return httpx.Response(200, json={"models": models})
+ if request.url.path.endswith("/api/show"):
+ return httpx.Response(200, json={"capabilities": ["completion", "tools"],
+ "model_info": {"test.context_length": 8192}})
+ return httpx.Response(200, json={"status": "success", "done": True})
+
+ real_client = httpx.AsyncClient
+ monkeypatch.setattr(httpx, "AsyncClient", lambda **kwargs: real_client(
+ **{**kwargs, "transport": httpx.MockTransport(respond)}))
+ monkeypatch.setattr(config, "resolve_provider_endpoint", lambda *args, **kwargs:
+ config.ResolvedEndpoint(base_url=root, credential="test-proxy-token", via_proxy=True))
+ monkeypatch.setattr(config, "get_provider_secret", lambda provider: None)
+ brain = OllamaBrain()
+ assert await brain._resolve_model(need_tools=True) == "test:latest"
+ monkeypatch.setattr(brain, "_model_options", lambda model:
+ config.OllamaModelOptions(num_ctx=4096, keep_alive="1m"))
+ ollama_profiles.reset_process_memo()
+ await brain._apply_model_options("test:latest", BrainRequest(messages=()))
+ catalog = ModelCatalog(cache_path=tmp_path / "catalog.json")
+ rows = await catalog._fetch_raw("ollama")
+ assert rows and rows[0].context_length == 8192
+ assert {request.url.path.rsplit("/", 1)[-1] for request in seen} >= {
+ "tags", "show", "create", "generate",
+ }
+
+
+async def test_claude_signed_tool_continuation_survives_openai_runtime(
+ monkeypatch, isolated_gateway,
+):
+ from jarvis.agent_chat import runner_api
+ from jarvis.agent_runtimes import anthropic_history, model_map
+ from jarvis.core import config
+ from jarvis.plugins.brain.claude_api import ClaudeAPIBrain
+
+ wire = AnthropicWire()
+ wire.tool_arguments = '{"path":"a"}'
+ wire.thinking = True
+ monkeypatch.setattr(config, "get_jarvis_agent_secret", lambda provider: None)
+ monkeypatch.setattr(model_map, "login_route", lambda *args: (False, None))
+ grant = gateway.Grant("agent", "claude-api")
+ body = {"model": "claude-sonnet-4-6", "messages": [{"role": "user", "content": "Read a"}]}
+ async with AsyncAnthropic(api_key="test-only", http_client=wire.client()) as client:
+ brain = ClaudeAPIBrain(model=body["model"])
+ brain._client = client
+ monkeypatch.setattr(runner_api, "build_brain", lambda *args: brain)
+ model, request = gateway.chat_request(body)
+ first = await gateway.complete_chat(grant, model, request)
+ message = first["choices"][0]["message"]
+ assert "signature" not in json.dumps(message)
+ assert "private test reasoning" not in json.dumps(message)
+ body["messages"].extend([message, {
+ "role": "tool", "tool_call_id": message["tool_calls"][0]["id"], "content": "contents",
+ }])
+ _, continuation = gateway.chat_request(body)
+ await gateway.complete_chat(grant, model, continuation)
+ assistant = json.loads(wire.requests[1].content)["messages"][1]["content"]
+ assert assistant[0] == {"type": "thinking", "thinking": "private test reasoning",
+ "signature": "opaque-test-signature"}
+ assert assistant[1]["type"] == "tool_use"
+ # Opaque signatures never move to another agent/account or changed prefix.
+ other = anthropic_history.restore(gateway.Grant("other", "claude-api"), model, continuation)
+ assert other == continuation
+ from dataclasses import replace
+
+ changed = replace(continuation, system="different instructions")
+ assert anthropic_history.restore(grant, model, changed) == changed
+ changed_tools = replace(continuation, tools=({"name": "changed", "input_schema": {}},))
+ assert anthropic_history.restore(grant, model, changed_tools) == changed_tools
+
+
[email protected]("provider_reason,wire_reason", [
+ ("FinishReason.MAX_TOKENS", "length"), ("SAFETY", "content_filter"), ("end_turn", "stop"),
+])
+def test_provider_finish_reasons_keep_their_meaning(provider_reason, wire_reason):
+ assert gateway._finish(provider_reason) == wire_reason
+
+
[email protected]("partial", [False, True])
+async def test_responses_stream_requires_terminal_event(monkeypatch, isolated_gateway, partial):
+ from types import SimpleNamespace
+
+ async def events(**kwargs):
+ if partial:
+ yield {"type": "response.created", "response": {"id": "test"}}
+
+ monkeypatch.setattr(gateway, "_client", lambda account: SimpleNamespace(stream=events))
+ token = gateway.grant_token("agent", "openai-codex", require_active_turn=True)
+ signal = gateway.watch_failure(token)
+ if partial:
+ stream = await gateway.open_response_stream(gateway.verify(token), {"model": "test"})
+ assert b"response.failed" in b"".join([part async for part in stream])
+ else:
+ with pytest.raises(gateway.GatewayError, match="before completion"):
+ await gateway.open_response_stream(gateway.verify(token), {"model": "test"})
+ assert signal.done()
+
+
+async def test_selected_effort_reaches_subscription(monkeypatch, isolated_gateway):
+ from tests.unit.agent_runtimes.test_runtime_gateway import FakeSubscription
+
+ client = FakeSubscription()
+ monkeypatch.setattr(gateway, "_client", lambda account: client)
+ token = gateway.grant_token("agent", "openai-codex", require_active_turn=True)
+ gateway.watch_failure(token, effort="low")
+ await gateway.complete_response(
+ gateway.verify(token), {"model": "test", "reasoning_effort": "high"},
+ )
+ assert client.calls[0]["reasoning_effort"] == "low"
+
+
+async def test_poisoned_openclaw_gateway_is_reaped_before_reuse(monkeypatch, tmp_path):
+ import hashlib
+ from types import SimpleNamespace
+
+ from jarvis.agent_runtimes import openclaw
+ from tests.unit.agent_runtimes.test_runtime_configs import _turn
+
+ runtime = openclaw.OpenClawRuntime()
+ old = openclaw._Gateway("agent", tmp_path, 12345,
+ hashlib.sha256(b"[]").hexdigest(), SimpleNamespace(returncode=None), None, None,
+ None, None,
+ poisoned=True)
+ runtime._gateways["agent"] = old
+ actions = []
+
+ async def stop(current):
+ actions.append("stop")
+ assert current is old
+ current.proc.returncode = 0
+
+ async def start(turn, key, home, port, token, env_hash, launcher, env):
+ actions.append("start")
+ assert old.proc.returncode == 0
+ return openclaw._Gateway(key, home, port, env_hash,
+ SimpleNamespace(returncode=None), None, None, None, None)
+
+ monkeypatch.setattr(runtime, "_stop_gateway", stop)
+ monkeypatch.setattr(runtime, "_start_gateway", start)
+ monkeypatch.setattr(runtime, "_ensure_reaper", lambda: None)
+ monkeypatch.setattr(openclaw, "write_json_if_changed", lambda *args: False)
+ replacement = await runtime._ensure_gateway(_turn(tmp_path), "agent", tmp_path,
+ "test-only", ["fake"], {})
+ assert actions == ["stop", "start"] and replacement is not old
+
+
[email protected]("streamed", [False, True])
+async def test_waiting_request_cannot_acquire_replacement_turn(
+ monkeypatch, isolated_gateway, streamed,
+):
+ import asyncio
+ import threading
+
+ from jarvis.agent_runtimes.model_limits import ModelLimits
+
+ entered, resume = threading.Event(), threading.Event()
+ calls = []
+
+ def limits(*args):
+ entered.set()
+ assert resume.wait(5)
+ return ModelLimits(context_window=8192)
+
+ async def deltas(*args):
+ calls.append(args)
+ yield BrainDelta(content="must not run", finish_reason="stop")
+
+ monkeypatch.setattr(gateway, "model_limits", limits)
+ monkeypatch.setattr(gateway, "_deltas", deltas)
+ token = gateway.grant_token("agent", "ollama", require_active_turn=True)
+ old = gateway.watch_failure(token)
+ complete = gateway.open_chat_stream if streamed else gateway.complete_chat
+ task = asyncio.create_task(complete(gateway.verify(token), "model", BrainRequest(messages=())))
+ try:
+ assert await asyncio.to_thread(entered.wait, 5)
+ gateway.unwatch_failure(token, old)
+ current = gateway.watch_failure(token)
+ finally:
+ resume.set()
+ with pytest.raises(gateway.GatewayError, match="no active turn"):
+ await task
+ assert not calls and not current.done()
+
+
+def test_claude_evicted_ancestor_cannot_restore_dependent_signature(isolated_gateway):
+ from dataclasses import replace
+
+ from jarvis.agent_runtimes import anthropic_history as history
+
+ grant = gateway.Grant("agent", "claude-api")
+ base = BrainRequest(messages=(BrainMessage("user", "Read"),))
+ first = {"type": "tool_use", "id": "first", "name": "read", "input": {}}
+ thought = {"type": "thinking", "thinking": "private", "signature": "signed"}
+ history.remember(grant, "model", base,
+ {"id": "first", "_anthropic_blocks": [thought, first]}, provider_request=base)
+ next_request = replace(base, messages=(*base.messages, BrainMessage("assistant", [first]),
+ BrainMessage("tool", "ok", tool_call_id="first")))
+ restored = history.restore(grant, "model", next_request)
+ second = {**first, "id": "second"}
+ history.remember(grant, "model", next_request,
+ {"id": "second", "_anthropic_blocks": [thought, second]}, provider_request=restored)
+ history._HISTORY.pop((grant, "model", "first"))
+ final = replace(next_request, messages=(*next_request.messages,
+ BrainMessage("assistant", [second])))
+ assert history.restore(grant, "model", final) == final
diff --git a/tests/unit/agent_runtimes/test_runtime_configs.py b/tests/unit/agent_runtimes/test_runtime_configs.py
index e015538c71..53c525671c 100644
--- a/tests/unit/agent_runtimes/test_runtime_configs.py
+++ b/tests/unit/agent_runtimes/test_runtime_configs.py
@@ -35,10 +35,12 @@ _GATEWAY = "http://127.0.0.1:47821/api/runtime-gateway/v1"
def gateway_up(monkeypatch):
"""Jarvis' gateway is listening, and this machine's Agents-tier keys stay out."""
import jarvis.core.config as config
- from jarvis.agent_runtimes import gateway
+ from jarvis.agent_runtimes import gateway, provider_errors
monkeypatch.setattr(gateway, "base_url", lambda: _GATEWAY)
monkeypatch.setattr(config, "get_jarvis_agent_secret", lambda provider: None)
+ monkeypatch.setattr(provider_errors, "_claude_usage", lambda token: None)
+ monkeypatch.setattr(provider_errors, "_REPORTS", {})
gateway.reset()
yield gateway
gateway.reset()
@@ -126,6 +128,39 @@ def test_hermes_discovers_connected_tools_with_explicit_persona_guidance(tmp_pat
assert "Search before concluding a capability is missing" in persona
+def test_hermes_profiles_share_installation_but_keep_agent_state_separate(tmp_path, monkeypatch):
+ from jarvis.agent_runtimes import base, hermes
+
+ monkeypatch.setattr(base, "runtimes_root", lambda: tmp_path)
+ monkeypatch.setattr(base, "legacy_runtimes_root", lambda: tmp_path / "legacy")
+ monkeypatch.setattr(hermes, "hermes_root", lambda: tmp_path / "hermes-home")
+ first = hermes.profile_home("first")
+ second = hermes.profile_home("second")
+ routine = hermes.profile_home(base.home_key("first", "society:first:routine:1"))
+ assert first.parent == second.parent == routine.parent == tmp_path / "hermes-home" / "profiles"
+ assert len({first, second, routine}) == 3
+ (first / "state.db").write_bytes(b"private conversation")
+ assert not (second / "state.db").exists()
+ assert not (routine / "state.db").exists()
+ assert hermes.profile_home("first") == first
+ assert hermes.profile_home("profiles") == first.parent / "profiles"
+
+
+def test_existing_runtime_homes_keep_their_native_sessions(tmp_path, monkeypatch):
+ from jarvis.agent_runtimes import base, hermes
+
+ monkeypatch.setattr(base, "runtimes_root", lambda: tmp_path)
+ monkeypatch.setattr(base, "legacy_runtimes_root", lambda: tmp_path / "legacy")
+ monkeypatch.setattr(hermes, "hermes_root", lambda: tmp_path / "hermes-home")
+ legacy = tmp_path / "legacy" / "hermes" / "existing"
+ legacy.mkdir(parents=True)
+ (legacy / "state.db").write_bytes(b"existing session")
+ home = hermes.profile_home("existing")
+ assert home == tmp_path / "hermes-home" / "profiles" / "existing"
+ assert (home / "state.db").read_bytes() == b"existing session"
+ assert base.agent_home("openclaw", "first") == tmp_path / "openclaw" / "first"
+
+
def test_hermes_always_asks_jarvis_and_never_the_guardian(tmp_path):
"""The stance is answered over ACP, so the file never depends on it."""
runtime = HermesRuntime()
@@ -310,7 +345,9 @@ def test_every_provider_runs_through_jarvis_gateway(gateway_up, monkeypatch):
# The runtime gets Jarvis' address and a token for this agent, never the key.
assert route.base_url == _GATEWAY and route.transport == "chat_completions"
assert route.api_key != _SECRET
- assert gateway.verify(route.api_key or "") == gateway.Grant("agent-1", "openai")
+ assert gateway.verify(route.api_key or "") == gateway.Grant(
+ "agent-1", "openai", require_active_turn=True,
+ )
assert route.env() == {KEY_ENV_VAR: route.api_key}
@@ -322,7 +359,28 @@ def test_the_chatgpt_subscription_speaks_responses(gateway_up, monkeypatch):
monkeypatch.setattr(gateway, "subscription_ready", lambda account_id="": True)
route = route_for(_cfg(), "openai-codex", "gpt-5.5", agent_id="agent-1", account_id="acct")
assert route.transport == "responses" and route.base_url == _GATEWAY
- assert gateway.verify(route.api_key or "") == gateway.Grant("agent-1", "openai-codex", "acct")
+ assert gateway.verify(route.api_key or "") == gateway.Grant(
+ "agent-1", "openai-codex", "acct", require_active_turn=True,
+ )
+
+
+def test_chat_and_routine_routes_have_stable_separate_failure_scopes(gateway_up):
+ with override_provider_secrets({"openai": _SECRET}):
+ routes = [route_for(_cfg(), "openai", "m", agent_id="a", session_id=sid)
+ for sid in ("society:a", "society:a:routine:r1", "society:a:routine:r2")]
+ assert routes[0].api_key != routes[1].api_key
+ assert routes[1].api_key == routes[2].api_key
+
+
+def test_a_rate_limited_route_refuses_before_starting_a_runtime(gateway_up):
+ gateway = gateway_up
+ grant = gateway.Grant("a", "openai")
+ gateway._report_failure(
+ grant, "m", gateway.GatewayError("limited", status=429, retry_after=60), None,
+ )
+ with override_provider_secrets({"openai": _SECRET}):
+ with pytest.raises(RouteUnavailable, match="HTTP 429"):
+ route_for(_cfg(), "openai", "m", agent_id="a", session_id="society:a")
def test_chat_and_routine_routes_have_stable_separate_failure_scopes(gateway_up):
@@ -374,7 +432,9 @@ def test_claude_without_a_key_runs_on_the_claude_code_login(gateway_up, monkeypa
# The runtime only ever holds Jarvis' gateway token, never the login.
assert route.transport == "chat_completions" and route.base_url == _GATEWAY
assert route.api_key != live
- assert gateway_up.verify(route.api_key or "") == gateway_up.Grant("agent-1", "claude-api", "")
+ assert gateway_up.verify(route.api_key or "") == gateway_up.Grant(
+ "agent-1", "claude-api", "", require_active_turn=True,
+ )
# Every other provider keeps needing its own key.
assert model_map.login_token_for("openai") is None
@@ -395,7 +455,9 @@ def test_an_agent_can_pin_claude_to_the_key_or_the_login(gateway_up, monkeypatch
_cfg(), "claude-api", "claude-sonnet-5", agent_id="a", account_id=SUBSCRIPTION_ACCOUNT
)
grant = gateway_up.verify(route.api_key or "")
- assert grant == gateway_up.Grant("a", "claude-api", SUBSCRIPTION_ACCOUNT)
+ assert grant == gateway_up.Grant(
+ "a", "claude-api", SUBSCRIPTION_ACCOUNT, require_active_turn=True,
+ )
with override_provider_secrets({"claude-api": None}):
assert model_map.access_choices() == {"claude-api": ["subscription"]}
assert model_map.login_token_for("claude-api", API_KEY_ACCOUNT) is None
@@ -537,4 +599,4 @@ def test_capabilities_come_from_the_catalog_never_the_name(monkeypatch):
assert runner_acp.model_capabilities("openrouter", "a/thinks-sees") == (True, True)
assert runner_acp.model_capabilities("ollama", "plain") == (False, False)
# A name that "sounds" capable but has no catalog entry stays unknown.
- assert runner_acp.model_capabilities("openai", "gpt-vision-reasoner") == (None, None)
\ No newline at end of file
+ assert runner_acp.model_capabilities("openai", "gpt-vision-reasoner") == (None, None)
diff --git a/tests/unit/agent_runtimes/test_smoke_lifecycle.py b/tests/unit/agent_runtimes/test_smoke_lifecycle.py
index a86d9b94ed..4d1f36812e 100644
--- a/tests/unit/agent_runtimes/test_smoke_lifecycle.py
+++ b/tests/unit/agent_runtimes/test_smoke_lifecycle.py
@@ -70,3 +70,34 @@ async def test_success_releases_slot(monkeypatch, tmp_path):
assert result.status == "done"
assert result.result_text == "echo: hello"
assert released == [True]
+
+
[email protected]("missing", [False, True])
+async def test_probe_owns_its_gateway_turn_and_releases_it(monkeypatch, tmp_path, missing):
+ from dataclasses import replace
+
+ from jarvis.agent_runtimes import gateway
+
+ turn, released = setup_probe(monkeypatch, tmp_path, missing=missing)
+ token = gateway.grant_token("smoke-owned-turn", "ollama", require_active_turn=True)
+ turn.route = replace(turn.route, api_key=token)
+ grant = gateway.verify(token)
+ create = smoke.asyncio.create_subprocess_exec
+
+ async def capture(*args, **kwargs):
+ assert gateway.capture_turn(grant) is not None
+ return await create(*args, **kwargs)
+
+ monkeypatch.setattr(smoke.asyncio, "create_subprocess_exec", capture)
+ try:
+ if missing:
+ with pytest.raises(OSError):
+ await smoke._run("fake", turn, "hello")
+ else:
+ result, _io = await smoke._run("fake", turn, "hello")
+ assert result.status == "done"
+ assert released == [True]
+ with pytest.raises(gateway.GatewayError, match="no active turn"):
+ gateway.capture_turn(grant)
+ finally:
+ gateway.revoke_agent("smoke-owned-turn")
diff --git a/tests/unit/agentic_ide/test_pane_picks.py b/tests/unit/agentic_ide/test_pane_picks.py
index 91ed681e35..f2df1bb091 100644
--- a/tests/unit/agentic_ide/test_pane_picks.py
+++ b/tests/unit/agentic_ide/test_pane_picks.py
@@ -68,16 +68,16 @@ async def test_the_picks_reach_the_command_line(
assert argv[argv.index("--permission-mode") + 1] == "acceptEdits"
-async def test_a_pane_with_no_picks_is_launched_exactly_as_before(
+async def test_a_pane_with_no_picks_launches_with_bypass_permissions(
registry: Registry, fake_pty: FakePtyManager, tmp_path: Path
) -> None:
- """Nobody who never opens the picker may notice this feature exists."""
+ """A fresh pane applies the project default without opening a picker."""
await registry.start(str(tmp_path), [{"agent": "claude"}])
await _attach(registry, "T1")
argv = _argv(fake_pty)
assert "--model" not in argv
assert "--effort" not in argv
- assert "--permission-mode" not in argv
+ assert argv[argv.index("--permission-mode") + 1] == "bypassPermissions"
async def test_the_wizard_opens_each_pane_on_its_own_picks(
diff --git a/tests/unit/agentic_ide/test_restart_pane.py b/tests/unit/agentic_ide/test_restart_pane.py
new file mode 100644
index 0000000000..fa2b0c0947
--- /dev/null
+++ b/tests/unit/agentic_ide/test_restart_pane.py
@@ -0,0 +1,247 @@
+"""Restarting a stopped coding pane on request, under the same IDs.
+
+Live 2026-10-07: a Codex pane that had stopped could not be brought back by
+voice. Every send was refused "not running; nothing was sent" and no action
+started it again, although the app's own start path continues the agent's
+conversation. Restart is that path, run on request: the same pane, its own
+conversation, never a second process, never a running turn cut short.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+from uuid import uuid4
+
+import pytest
+
+from jarvis.agentic_ide import session as ide
+from jarvis.agentic_ide.orchestration import WorkspaceOrchestrator
+from jarvis.brain.workspace_tool import WorkspaceOrchestrationTool
+from jarvis.live.state import LiveLedger
+from tests.fakes.fake_pty_manager import FakePtyManager
+
+
+async def _noop(_text: str) -> None:
+ return None
+
+
+async def _noop_exit(_code: int) -> None:
+ return None
+
+
+def fake_pty() -> FakePtyManager:
+ return FakePtyManager()
+
+
+def registry(fake_pty: FakePtyManager, monkeypatch: pytest.MonkeyPatch) -> ide.Registry:
+ monkeypatch.setattr(ide, "agent_argv", lambda name: (f"/usr/bin/{name}",))
+ return ide.Registry(pty_manager=fake_pty)
+
+
+async def _stopped_pane(
+ registry: ide.Registry, fake_pty: FakePtyManager, folder: Path, existing_conversation,
+) -> ide.Terminal:
+ """A Claude pane that worked, then whose agent exited with an error."""
+ await registry.start(str(folder), [{"agent": "claude", "name": "T1"}])
+ await registry.attach("T1", 80, 24, _noop, _noop_exit)
+ term = registry.session.find("T1")
+ existing_conversation(term.resume.id)
+ await fake_pty.die(term.pty_id, 1)
+ assert term.status == "exited"
+ return term
+
+
+async def test_restart_brings_a_stopped_pane_back_on_its_own_conversation(
+ registry, fake_pty, tmp_path, existing_conversation,
+):
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ identity, conversation = "pane:" + term.history_id, term.resume.id
+
+ restarted, started = await registry.restart_terminal(identity, registry.session.id)
+
+ assert started is True and restarted is term
+ assert term.status == "live" and term.pty_id
+ assert "pane:" + term.history_id == identity # the stable target survives
+ assert fake_pty.spawns[-1]["argv"][-2:] == ("--resume", conversation)
+ assert term.resumed is True
+ # Nothing was typed into the restarted agent.
+ assert not [data for pid, data in fake_pty.writes if pid == term.pty_id]
+
+
+async def test_restart_never_touches_a_running_agent(registry, fake_pty, tmp_path):
+ await registry.start(str(tmp_path), [{"agent": "claude", "name": "T1"}])
+ await registry.attach("T1", 80, 24, _noop, _noop_exit)
+ term = registry.session.find("T1")
+ running = term.pty_id
+
+ _, started = await registry.restart_terminal("pane:" + term.history_id)
+
+ assert started is False
+ assert term.pty_id == running and len(fake_pty.spawns) == 1
+ assert running not in fake_pty.closed
+
+
+async def test_concurrent_restarts_start_one_process(
+ registry, fake_pty, tmp_path, existing_conversation,
+):
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ before = len(fake_pty.spawns)
+
+ outcomes = await asyncio.gather(
+ *(registry.restart_terminal("pane:" + term.history_id) for _ in range(3))
+ )
+
+ assert sorted(started for _, started in outcomes) == [False, False, True]
+ assert len(fake_pty.spawns) == before + 1
+
+
+async def test_a_restart_that_cannot_start_reports_why(
+ registry, fake_pty, tmp_path, existing_conversation,
+):
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ fake_pty.spawn_error = "no PTY backend"
+
+ with pytest.raises(ide.SessionError, match="no PTY backend"):
+ await registry.restart_terminal("pane:" + term.history_id)
+ assert term.status == "error" and not term.pty_id
+
+
+async def test_a_pane_that_is_moving_is_left_as_it_was(
+ registry, fake_pty, tmp_path, existing_conversation,
+):
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ term.placing = "Copying the folder…"
+
+ with pytest.raises(ide.SessionNotReady):
+ await registry.restart_terminal("pane:" + term.history_id)
+ assert term.status == "exited"
+
+
+async def test_a_shell_pane_is_not_restarted(registry, tmp_path):
+ await registry.start(str(tmp_path), [{"agent": "shell", "name": "T1"}])
+ term = registry.session.find("T1")
+
+ with pytest.raises(ide.SessionError, match="not a coding agent"):
+ await registry.restart_terminal("pane:" + term.history_id)
+
+
+# ------------------------------------------------------------ orchestration
+class _Sessions:
+ """The coding-session gateway: every send it gets is typed and accepted."""
+
+ def __init__(self) -> None:
+ self.typed: list[str] = []
+
+ async def run(self, args):
+ if args["action"] == "send":
+ self.typed.append(args["prompt"])
+ return {"delivery": "accepted", "submitted": True, "completed": False}
+
+
+def orchestrator(registry, tmp_path):
+ ledger = LiveLedger(tmp_path / "receipts.db")
+ sessions = _Sessions()
+ yield WorkspaceOrchestrator(registry, sessions, ledger), sessions
+ ledger.close()
+
+
+def _target(registry: ide.Registry, term: ide.Terminal) -> dict[str, str]:
+ workspace = registry.session
+ workspace.project_id = workspace.project_id or "project"
+ return {
+ "project_id": workspace.project_id,
+ "workspace_id": workspace.id,
+ "terminal_id": "pane:" + term.history_id,
+ }
+
+
+async def test_a_refused_send_is_delivered_after_restart_under_the_same_request(
+ registry, fake_pty, tmp_path, existing_conversation, orchestrator,
+):
+ work, sessions = orchestrator
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ target, request_id = _target(registry, term), uuid4().hex
+ task = {"action": "send", **target, "request_id": request_id, "prompt": "Fix the build"}
+
+ refused = await work.run(task)
+ assert refused["status"] == "not_accepted" and refused["stopped"] is True
+ assert refused["input_written"] is False and refused["restartable"] is True
+ assert sessions.typed == []
+
+ restarted = await work.run({"action": "restart", **target})
+ assert restarted["status"] == "restarted"
+ assert restarted["target"] == target and restarted["conversation"] == "continued"
+ assert restarted["input_written"] is False
+
+ delivered = await work.run(task)
+ assert delivered["status"] == "accepted"
+ assert sessions.typed == ["Fix the build"]
+ # The accepted receipt is now the answer for this request: no second copy.
+ again = await work.run(task)
+ assert again["status"] == "accepted" and sessions.typed == ["Fix the build"]
+
+
+async def test_restart_of_a_running_agent_is_a_receipted_no_op(
+ registry, fake_pty, tmp_path, orchestrator,
+):
+ work, _ = orchestrator
+ await registry.start(str(tmp_path), [{"agent": "claude", "name": "T1"}])
+ await registry.attach("T1", 80, 24, _noop, _noop_exit)
+ term = registry.session.find("T1")
+
+ receipt = await work.run({"action": "restart", **_target(registry, term)})
+
+ assert receipt["status"] == "already_running"
+ assert len(fake_pty.spawns) == 1
+
+
+async def test_restart_by_spoken_call_sign(
+ registry, fake_pty, tmp_path, existing_conversation, orchestrator, monkeypatch,
+):
+ work, _ = orchestrator
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ target = _target(registry, term)
+ graph = {
+ "projects": [{
+ "id": target["project_id"], "name": "Project", "path": str(tmp_path),
+ "workspaces": [{
+ "id": target["workspace_id"], "name": "Main", "status": "open",
+ "agents": [{
+ "id": target["terminal_id"], "name": "T1", "agent": "claude",
+ "status": term.status, "activity": "exited", "accepts_tasks": True,
+ }],
+ }],
+ }],
+ "active_workspace_id": target["workspace_id"],
+ }
+ monkeypatch.setattr(work, "graph", lambda: graph)
+
+ receipt = await work.run({"action": "restart", "agent": "T1"})
+
+ assert receipt["status"] == "restarted" and receipt["target"] == target
+
+
+async def test_a_pane_that_cannot_start_gets_a_refusal_receipt(
+ registry, fake_pty, tmp_path, existing_conversation, orchestrator,
+):
+ work, _ = orchestrator
+ term = await _stopped_pane(registry, fake_pty, tmp_path, existing_conversation)
+ fake_pty.spawn_error = "no PTY backend"
+
+ receipt = await work.run({"action": "restart", **_target(registry, term)})
+
+ assert receipt["status"] == "not_accepted" and "no PTY backend" in receipt["reason"]
+
+
+def test_restart_is_a_logged_action_in_the_tool_contract():
+ tool = WorkspaceOrchestrationTool(gateway=None)
+ assert "restart" in tool.schema["properties"]["action"]["enum"]
+ args = {"action": "restart", "terminal_id": "pane:abc"}
+ # Monitor: runs without a spoken confirmation and is recorded; never "safe".
+ assert tool.risk_tier_for_args(args) == "monitor"
+ assert tool.read_only_for_args(args) is False
+ assert tool.describe_args(args)["level"] == "modify"
diff --git a/tests/unit/agentic_ide/test_send_gate.py b/tests/unit/agentic_ide/test_send_gate.py
index 2f7b0b49cc..1cd20f87a0 100644
--- a/tests/unit/agentic_ide/test_send_gate.py
+++ b/tests/unit/agentic_ide/test_send_gate.py
@@ -159,6 +159,36 @@ async def test_an_interrupted_turn_accepts_the_next_instruction(pane):
assert sent == ["Updated brief"]
+def codex_event(moment: float, kind: str, **payload) -> dict:
+ return {"timestamp": at(moment), "type": "event_msg", "payload": {"type": kind, **payload}}
+
+
+async def test_a_failed_codex_turn_accepts_the_next_instruction(pane, monkeypatch):
+ """Live 2026-10-07: a Codex turn that ended in an error left the pane
+ "not running (failed); nothing was sent" for every later task, although
+ the CLI sat at its prompt."""
+ registry, workspace, term, record, sent = pane
+ term.agent = "codex"
+ monkeypatch.setattr(task_state.agent_transcript, "_codex_file", lambda *_: record)
+ now = time.time()
+ submitted(term, now - 300)
+ write(record, codex_event(now - 299, "task_started"),
+ codex_event(now - 200, "task_complete", error="stream disconnected"))
+ await task_state.probe(term)
+ assert activity.read_activity(term) == "failed" # the badge still says so
+ await send(registry, workspace, term)
+ assert sent == ["Updated brief"]
+
+
[email protected]("status", ["exited", "error"])
+async def test_a_stopped_agent_is_refused_with_the_way_back(pane, status):
+ registry, workspace, term, record, sent = pane
+ term.status = status
+ with pytest.raises(SessionError, match="not running.*Restart it"):
+ await send(registry, workspace, term)
+ assert sent == []
+
+
@pytest.mark.parametrize("last", ["tool_use", "question"])
async def test_a_running_turn_or_open_question_still_refuses(pane, last):
registry, workspace, term, record, sent = pane
diff --git a/tests/unit/app_actions/test_app_actions.py b/tests/unit/app_actions/test_app_actions.py
index efa277cb78..fe207e1e1f 100644
--- a/tests/unit/app_actions/test_app_actions.py
+++ b/tests/unit/app_actions/test_app_actions.py
@@ -100,6 +100,21 @@ def test_remote_commands_ask_and_the_brain_switch_is_never_offered() -> None:
assert ("POST", "/api/brain/switch") not in by_route
+def test_coding_panes_are_reached_only_through_the_addressed_workspace_tool() -> None:
+ """The pane prompt and fan-out routes pick a pane by call-sign in the front
+ workspace, with no delivery receipt; voice removed the tools wrapping them
+ for that reason, and run-app-action must not hand them back."""
+ from jarvis.ui.web.agentic_ide_routes import router as ide_router
+
+ application = FastAPI()
+ application.include_router(ide_router)
+ by_route = {(e.method, e.path) for e in build_catalog(application.openapi()).values()}
+ assert ("POST", "/api/agentic-ide/terminals/{name}/prompt") not in by_route
+ assert ("POST", "/api/agentic-ide/fanout") not in by_route
+ # Other pane actions stay offered under the person's policy.
+ assert ("POST", "/api/agentic-ide/terminals/{name}/interrupt") in by_route
+
+
def test_default_tiers_read_safe_change_monitor_delete_ask(app: FastAPI) -> None:
catalog = build_catalog(app.openapi())
tiers = {(e.method, e.path): default_tier(e) for e in catalog.values()}
@@ -145,6 +160,25 @@ async def test_a_blocked_action_never_runs(app: FastAPI) -> None:
assert history.recent(1)[0]["outcome"] == "blocked"
+def test_a_read_is_described_as_a_read_and_a_change_as_a_change(app: FastAPI) -> None:
+ """Live voice keys replay and supersession on this: a GET read as a change
+ was answered from an earlier call's result instead of being read again."""
+ tool = RunAppActionTool(runtime=_Runtime(app))
+ read = {"action_id": _id(app, "/api/skills", "GET")}
+ change = {"action_id": _id(app, "/api/workspace/panes/{pane_id}/rename", "POST")}
+ delete = {"action_id": _id(app, "/api/skills/{name}", "DELETE")}
+ assert tool.describe_args(read)["level"] == "read"
+ for args in (change, delete, {"action_id": "nope"}):
+ assert tool.describe_args(args)["level"] == "modify"
+ # A read-only chat still refuses every app action, reads included.
+ from jarvis.core.tool_read_only import allows_read
+
+ assert allows_read(tool, read) is False
+ # The person's mode still decides whether a read runs at all.
+ set_mode(read["action_id"], "block")
+ assert tool.risk_tier_for_args(read) == "block"
+
+
async def test_query_parameters_and_unknown_ids(app: FastAPI) -> None:
tool = RunAppActionTool(runtime=_Runtime(app))
listed = await tool.execute(
diff --git a/tests/unit/appshot/conftest.py b/tests/unit/appshot/conftest.py
index 33ee2e6a2b..0ff947632e 100644
--- a/tests/unit/appshot/conftest.py
+++ b/tests/unit/appshot/conftest.py
@@ -6,9 +6,10 @@ import pytest
@pytest.fixture(autouse=True)
-def _isolated_library(tmp_path):
- from jarvis.appshot import library
+def _isolated_library(tmp_path, monkeypatch):
+ from jarvis.appshot import library, recording
library.set_root(tmp_path / "appshot-library")
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path / "recordings")
yield
library.set_root(None)
diff --git a/tests/unit/appshot/test_media_library.py b/tests/unit/appshot/test_media_library.py
new file mode 100644
index 0000000000..854688da92
--- /dev/null
+++ b/tests/unit/appshot/test_media_library.py
@@ -0,0 +1,138 @@
+"""Durable recordings join the gallery without copying or exposing partial files."""
+
+from __future__ import annotations
+
+import io
+import os
+from pathlib import Path
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from PIL import Image
+
+from jarvis.appshot import library, media_library, recording
+from jarvis.appshot.store import Appshot
+
+ID = "a" * 32
+
+
+def client():
+ from jarvis.ui.web.appshot_routes import router
+
+ app = FastAPI()
+ app.include_router(router)
+ return TestClient(app)
+
+
+def video(recording_id: str = ID, *, valid: bool = False) -> Path:
+ path = recording.recording_dir() / f"{recording_id}.mp4"
+ path.parent.mkdir(parents=True, exist_ok=True)
+ if not valid:
+ path.write_bytes(b"fake finalized recording")
+ return path
+ av = pytest.importorskip("av")
+ with av.open(str(path), "w", format="mp4") as writer:
+ stream = writer.add_stream("mpeg4", rate=10)
+ stream.width, stream.height, stream.pix_fmt = 160, 90, "yuv420p"
+ for index in range(10):
+ frame = av.VideoFrame.from_image(Image.new("RGB", (160, 90), (20, 80, 140)))
+ frame.pts = index
+ for packet in stream.encode(frame):
+ writer.mux(packet)
+ for packet in stream.encode():
+ writer.mux(packet)
+ return path
+
+
+def screenshot(shot_id: str = "image") -> None:
+ output = io.BytesIO()
+ Image.new("RGB", (20, 10)).save(output, "PNG")
+ library.save(Appshot(
+ id=shot_id, image=output.getvalue(), mime="image/png", width=20, height=10,
+ label="active window", app_name="Editor", note="", ui_text="", trigger="hotkey",
+ taken_at=100.0,
+ ))
+
+
+def test_existing_recordings_appear_without_live_session_or_image_directory(client):
+ path = video()
+ assert not library.library_root().exists()
+ body = client.get("/api/appshot/library").json()
+ [item] = body["items"]
+ assert item["id"] == f"recording_{ID}"
+ assert item["mime"] == "video/mp4" and item["path"] == str(path)
+ # Process-local caches are optional: reopening still finds the durable file.
+ media_library._metadata.cache_clear()
+ assert client.get("/api/appshot/library").json()["items"] == body["items"]
+
+
+def test_gallery_includes_more_than_the_recent_ten_videos_and_sorts_with_images(client):
+ screenshot()
+ for index in range(12):
+ path = video(f"{index:032x}")
+ os.utime(path, (200 + index, 200 + index))
+ items = client.get("/api/appshot/library").json()["items"]
+ assert len(items) == 13
+ assert items[0]["id"] == f"recording_{11:032x}"
+ assert items[-1]["id"] == "image"
+
+
+def test_video_metadata_poster_and_seekable_playback(client):
+ path = video(valid=True)
+ [item] = client.get("/api/appshot/library").json()["items"]
+ assert (item["width"], item["height"]) == (160, 90)
+ assert item["duration_s"] == pytest.approx(1, abs=0.15)
+ url = f"/api/appshot/library/recording_{ID}/image"
+ poster = client.get(f"{url}?thumb=1")
+ assert poster.headers["content-type"] == "image/jpeg"
+ assert Image.open(io.BytesIO(poster.content)).size == (160, 90)
+ part = client.get(url, headers={"Range": "bytes=0-15"})
+ assert part.status_code == 206 and part.content == path.read_bytes()[:16]
+ assert part.headers["content-type"] == "video/mp4"
+ assert part.headers["cache-control"] == "no-store"
+ assert client.get(f"{url}?variant=edited").status_code == 404
+ assert client.post(f"/api/appshot/library/recording_{ID}/open").status_code == 404
+
+
+def test_bad_video_uses_a_small_poster_instead_of_returning_video_bytes(client):
+ video()
+ poster = client.get(f"/api/appshot/library/recording_{ID}/image?thumb=1")
+ assert poster.headers["content-type"] == "image/svg+xml"
+ assert poster.content.startswith(b"<svg") and len(poster.content) < 1000
+
+
+def test_delete_video_never_deletes_a_screenshot_with_the_same_recorder_id(client):
+ path = video()
+ screenshot(ID)
+ response = client.delete(f"/api/appshot/library/recording_{ID}")
+ assert response.status_code == 200 and not path.exists()
+ assert library.get_item(ID, "original") is not None
+ assert client.delete(f"/api/appshot/library/recording_{ID}").status_code == 404
+
+
+def test_clear_removes_finalized_captures_but_preserves_partial_and_foreign_files(client):
+ screenshot()
+ path = video()
+ partial = path.with_suffix(".partial")
+ partial.write_bytes(b"still recording")
+ foreign = video("not-a-recorder-id")
+ assert len(client.get("/api/appshot/library").json()["items"]) == 2
+ body = client.delete("/api/appshot/library").json()
+ assert body == {"ok": True, "removed": 2}
+ assert not path.exists() and partial.exists() and foreign.exists()
+
+
+def test_recording_symlinks_are_neither_listed_nor_deleted(client, tmp_path):
+ target = tmp_path / "private.mp4"
+ target.write_bytes(b"outside the recorder")
+ link = recording.recording_dir() / f"{ID}.mp4"
+ link.parent.mkdir(parents=True, exist_ok=True)
+ try:
+ link.symlink_to(target)
+ except OSError:
+ pytest.skip("The host does not allow creating symlinks")
+ assert client.get("/api/appshot/library").json()["items"] == []
+ assert client.delete(f"/api/appshot/library/recording_{ID}").status_code == 404
+ assert target.read_bytes() == b"outside the recorder" and link.is_symlink()
diff --git a/tests/unit/appshot/test_picker_host.py b/tests/unit/appshot/test_picker_host.py
new file mode 100644
index 0000000000..46dde966d4
--- /dev/null
+++ b/tests/unit/appshot/test_picker_host.py
@@ -0,0 +1,194 @@
+"""Warm picker lifecycle without a GUI, hooks, pixels or child processes."""
+
+from __future__ import annotations
+
+import asyncio
+import queue
+import subprocess
+import threading
+
+import pytest
+
+from jarvis.appshot import picker as wire
+from jarvis.appshot import picker_host, region
+
+
+class Output:
+ def __init__(self) -> None:
+ self.lines: queue.Queue[str | None] = queue.Queue()
+ self.closed = False
+
+ def emit(self, payload: dict) -> None:
+ self.lines.put(wire.encode(payload))
+
+ def __iter__(self):
+ while (line := self.lines.get()) is not None:
+ yield line
+
+ def close(self) -> None:
+ self.closed = True
+
+
+class Process:
+ def __init__(self, *, finish: bool = True, standby: bool = True) -> None:
+ self.stdout = Output()
+ self.stdin = self
+ self.returncode = None
+ self.commands: list[dict] = []
+ self.finish = finish
+ self.closed = False
+ self.started = threading.Event()
+ if standby:
+ self.stdout.emit({"event": wire.EVENT_STANDBY})
+
+ def write(self, line: str) -> None:
+ command = wire.decode(line)
+ self.commands.append(command)
+ if command["cmd"] == wire.CMD_START:
+ self.started.set()
+ self.stdout.emit({"event": wire.EVENT_READY})
+ if self.finish:
+ self.stdout.emit({"event": wire.EVENT_SELECTION, "cancelled": True})
+ elif command["cmd"] == wire.CMD_QUIT:
+ self.exit(0)
+
+ def flush(self) -> None:
+ pass # In-memory pipe writes are immediate.
+
+ def exit(self, code: int) -> None:
+ self.returncode = code
+ self.stdout.lines.put(None)
+
+ def poll(self):
+ return self.returncode
+
+ def wait(self, timeout=None):
+ if self.returncode is None:
+ raise subprocess.TimeoutExpired("picker", timeout)
+ return self.returncode
+
+ def kill(self) -> None:
+ self.exit(-9)
+
+ def close(self) -> None:
+ self.closed = True
+
+
+def host(monkeypatch):
+ children: list[Process] = []
+
+ def spawn(*, resident):
+ assert resident
+ proc = Process()
+ children.append(proc)
+ return proc
+
+ async def escape(_proc):
+ await asyncio.Future()
+
+ monkeypatch.setattr(region, "_spawn", spawn)
+ monkeypatch.setattr(region, "_escape_cancels", escape)
+ monkeypatch.setattr(region, "picker_capability", lambda: (True, ""))
+ monkeypatch.setattr(region, "snap_layout", lambda: {"monitors": [], "windows": []})
+ return picker_host.PickerHost(), children
+
+
+async def test_standby_never_requests_pixels_and_repeated_picks_reuse_process(host):
+ owner, children = host
+ try:
+ await owner.prewarm()
+ assert children[0].commands == []
+ for language in ("en", "de", "es"):
+ payload, code, timed_out = await owner.pick(1, language)
+ assert payload["cancelled"] and code is None and not timed_out
+ assert children[0].commands[-1]["language"] == language
+ assert len(children) == 1
+ finally:
+ await owner.close()
+ assert children[0].closed and children[0].stdout.closed
+ assert children[0].returncode == 0
+
+
+async def test_each_pick_refreshes_window_layout(host, monkeypatch):
+ owner, children = host
+ layouts = iter(([1], [2]))
+ monkeypatch.setattr(region, "snap_layout", lambda: {"windows": next(layouts)})
+ try:
+ await owner.pick(1, "en")
+ await owner.pick(1, "en")
+ assert [c["windows"] for c in children[0].commands] == [[1], [2]]
+ finally:
+ await owner.close()
+
+
+async def test_dead_idle_process_is_reaped_and_replaced(host):
+ owner, children = host
+ try:
+ await owner.prewarm()
+ children[0].exit(7)
+ await owner.pick(1, "en")
+ assert len(children) == 2 and children[0].closed
+ finally:
+ await owner.close()
+
+
+async def test_timeout_reaps_reader_before_next_request(host):
+ owner, children = host
+ try:
+ await owner.prewarm()
+ children[0].finish = False
+ payload, _, timed_out = await owner.pick(0.02, "en")
+ assert payload is None and timed_out and children[0].closed
+ await owner.pick(1, "en")
+ assert len(children) == 2
+ finally:
+ await owner.close()
+
+
+async def test_shutdown_cancels_visible_selection_and_joins_reader(host):
+ owner, children = host
+ await owner.prewarm()
+ children[0].finish = False
+ task = asyncio.create_task(owner.pick(30, "en"))
+ assert await asyncio.to_thread(children[0].started.wait, 1)
+ await asyncio.wait_for(owner.close(), 1)
+ assert task.cancelled() and children[0].closed
+
+
+async def test_cancel_during_spawn_reaps_late_process(host, monkeypatch):
+ owner, _ = host
+ spawned = threading.Event()
+ release = threading.Event()
+ child = Process()
+
+ def spawn(**_kwargs):
+ spawned.set()
+ assert release.wait(2)
+ return child
+
+ monkeypatch.setattr(region, "_spawn", spawn)
+ warm = asyncio.create_task(owner.prewarm())
+ assert await asyncio.to_thread(spawned.wait, 1)
+ warm.cancel()
+ release.set()
+ with pytest.raises(asyncio.CancelledError):
+ await warm
+ assert child.returncode == 0 and child.closed
+
+
+async def test_startup_timeout_closes_pipe_and_joins_reader(host, monkeypatch):
+ owner, _ = host
+ child = Process(standby=False)
+ monkeypatch.setattr(region, "_spawn", lambda **_kwargs: child)
+ monkeypatch.setattr(picker_host, "_START_TIMEOUT_S", 0.02)
+ with pytest.raises(TimeoutError):
+ await owner.prewarm()
+ assert child.returncode == 0 and child.stdout.closed
+
+
+async def test_no_display_never_spawns(host, monkeypatch):
+ owner, children = host
+ monkeypatch.setattr(region, "picker_capability", lambda: (False, "no display"))
+ await owner.prewarm()
+ assert children == []
diff --git a/tests/unit/appshot/test_picker_standby.py b/tests/unit/appshot/test_picker_standby.py
new file mode 100644
index 0000000000..9e00bf8fd3
--- /dev/null
+++ b/tests/unit/appshot/test_picker_standby.py
@@ -0,0 +1,47 @@
+"""Renderer standby frees pixel references without constructing a GUI."""
+
+from types import SimpleNamespace
+
+import pytest
+
+pytest.importorskip("PySide6.QtWidgets")
+
+from jarvis.appshot.picker import renderer # noqa: E402
+
+
+def test_release_capture_drops_frozen_pixels_and_blur_cache():
+ window = SimpleNamespace(
+ _frozen=object(), _patches=object(), _markup=object(), _scale=lambda: 1
+ )
+ renderer._SelectWindow.release_capture(window)
+ assert window._frozen is None
+ assert window._patches._frozen is None
+ assert window._patches._cache == {}
+ assert window._markup.sel is None
+
+
+def test_resident_finish_releases_every_screen_without_quitting(monkeypatch):
+ calls = []
+ window = SimpleNamespace(
+ hide=lambda: calls.append("hide"),
+ release_capture=lambda: calls.append("release"),
+ close=lambda: calls.append("close"),
+ deleteLater=lambda: calls.append("delete"),
+ )
+ owner = SimpleNamespace(
+ _app=SimpleNamespace(
+ processEvents=lambda: calls.append("flush"), quit=lambda: calls.append("quit")
+ ),
+ _windows=[window], _done=False, _resident=True, _started=True,
+ _focused=window, marking=window,
+ )
+ monkeypatch.setattr(renderer, "_emit", lambda payload: calls.append(payload))
+ renderer.Picker.finish(owner, None, None)
+ assert calls == [
+ "hide", "flush", "release", "close", "delete",
+ {"event": "selection", "cancelled": True},
+ ]
+ assert owner._windows == [] and owner._focused is None and owner.marking is None
+ assert not owner._started
+ renderer.Picker.finish(owner, None, None)
+ assert len(calls) == 6, "duplicate finish cannot emit a second selection"
diff --git a/tests/unit/appshot/test_recording_colour_tags.py b/tests/unit/appshot/test_recording_colour_tags.py
new file mode 100644
index 0000000000..dcb4e2af94
--- /dev/null
+++ b/tests/unit/appshot/test_recording_colour_tags.py
@@ -0,0 +1,97 @@
+"""SDR recordings say which matrix their encoder used, so players decode exact colours."""
+
+from __future__ import annotations
+
+from fractions import Fraction
+
+import numpy as np
+import pytest
+
+from jarvis.appshot import hdr_recording
+
+
+def _decode_601(y: float, u: float, v: float) -> np.ndarray:
+ kr, kb = 0.299, 0.114
+ luma, cb, cr = (y - 16) / 219, (u - 128) / 224, (v - 128) / 224
+ r = luma + 2 * (1 - kr) * cr
+ b = luma + 2 * (1 - kb) * cb
+ g = (luma - kr * r - kb * b) / (1 - kr - kb)
+ return np.clip(np.array([r, g, b]) * 255, 0, 255)
+
+
+def test_libx264_recording_is_tagged_with_the_matrix_it_was_converted_with(tmp_path) -> None:
+ av = pytest.importorskip("av")
+ if "libx264" not in av.codecs_available:
+ pytest.skip("libx264 missing from this PyAV build")
+ from jarvis.appshot import recording_codec
+
+ colours = [(255, 0, 0), (0, 255, 0), (40, 90, 200)]
+ image = np.zeros((64, 64 * len(colours), 4), np.uint8)
+ image[..., 3] = 255
+ for index, colour in enumerate(colours):
+ image[:, index * 64 : (index + 1) * 64, :3] = colour
+ candidates = [c for c in recording_codec._SDR_CANDIDATES if c[0] == "libx264"]
+ path = tmp_path / "sdr.mp4"
+ container, stream = recording_codec._open_first(
+ av, path, (image.shape[1], image.shape[0]), 30, 4_000_000, candidates,
+ recording_codec._tag_srgb,
+ )
+ for index in range(3):
+ frame = av.VideoFrame.from_ndarray(image, format="rgba").reformat(format=stream.pix_fmt)
+ frame.pts, frame.time_base = index * 2000, Fraction(1, 60000)
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode(None):
+ container.mux(packet)
+ container.close()
+ with av.open(str(path)) as source:
+ decoded = next(source.decode(video=0))
+ context = source.streams.video[0].codec_context
+ assert (context.color_primaries, context.color_trc, context.colorspace) == (1, 13, 5)
+ y = np.frombuffer(decoded.planes[0], np.uint8).reshape(decoded.height, -1)
+ u = np.frombuffer(decoded.planes[1], np.uint8).reshape(decoded.height // 2, -1)
+ v = np.frombuffer(decoded.planes[2], np.uint8).reshape(decoded.height // 2, -1)
+ for index, colour in enumerate(colours):
+ x = index * 64 + 32
+ rgb = _decode_601(float(y[32, x]), float(u[16, x // 2]), float(v[16, x // 2]))
+ assert np.abs(rgb - colour).max() <= 3
+
+
+def test_hdr_video_round_trips_through_a_software_encoder(tmp_path) -> None:
+ av = pytest.importorskip("av")
+ if "libx265" not in av.codecs_available and "libsvtav1" not in av.codecs_available:
+ pytest.skip("no software 10-bit encoder in this PyAV build")
+ from jarvis.appshot.recording_codec import open_video
+
+ path = tmp_path / "hdr.mp4"
+ container, stream = open_video(av, path, (64, 64), 30, 4_000_000, hdr=True)
+ y = np.full((64, 64), (64 + 500) << 6, np.uint16)
+ uv = np.full((32, 32, 2), 512 << 6, np.uint16)
+ for index in range(5):
+ frame = hdr_recording.p010_frame(av, stream.pix_fmt, y, uv)
+ frame.pts, frame.time_base = index * 2000, Fraction(1, 60000)
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode(None):
+ container.mux(packet)
+ container.close()
+ with av.open(str(path)) as source:
+ decoded = next(source.decode(video=0))
+ context = source.streams.video[0].codec_context
+ assert (context.color_primaries, context.color_trc, context.colorspace) == (9, 16, 9)
+ assert decoded.format.name in ("yuv420p10le", "p010le")
+
+
+def test_clip_export_keeps_hdr_depth_and_every_colour_tag() -> None:
+ from types import SimpleNamespace
+
+ from jarvis.appshot import video_edit
+
+ av = SimpleNamespace(codecs_available={"libsvtav1", "libx264"})
+ hdr = SimpleNamespace(color_primaries=9, color_trc=16, colorspace=9, color_range=1)
+ sdr = SimpleNamespace(color_primaries=1, color_trc=13, colorspace=5, color_range=1)
+ assert video_edit._clip_encoder(av, hdr)[:2] == ("libsvtav1", "yuv420p10le")
+ assert video_edit._clip_encoder(av, sdr)[:2] == ("libx264", "yuv420p")
+ target = SimpleNamespace()
+ video_edit._copy_colour_tags(hdr, target)
+ assert (target.color_primaries, target.color_trc, target.colorspace) == (9, 16, 9)
diff --git a/tests/unit/appshot/test_recording_frame.py b/tests/unit/appshot/test_recording_frame.py
new file mode 100644
index 0000000000..709bcbfb78
--- /dev/null
+++ b/tests/unit/appshot/test_recording_frame.py
@@ -0,0 +1,136 @@
+"""Packed capture buffers are copied with stride and released before they can change."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from importlib import import_module
+
+import pytest
+
+
+def qt():
+ gui = pytest.importorskip("PySide6.QtGui")
+ try:
+ multimedia = import_module("PySide6.QtMultimedia")
+ except ImportError as exc:
+ pytest.skip(f"QtMultimedia's native runtime is unavailable: {exc}")
+ core = pytest.importorskip("PySide6.QtCore")
+ return SimpleNamespace(gui=gui, media=multimedia, core=core)
+
+
+def frame_for(qt, colors, *, padding=8, valid=True, mapping=True, short=False):
+ """A backend-owned BGRA buffer that is deliberately overwritten on unmap."""
+ height, width = len(colors), len(colors[0])
+ stride = width * 4 + padding
+ pixels = bytearray([91] * (stride * height))
+ for y, row in enumerate(colors):
+ for x, (r, g, b, a) in enumerate(row):
+ pixels[y * stride + x * 4:y * stride + x * 4 + 4] = bytes((b, g, r, a))
+
+ class Frame:
+ unmapped = False
+ rotation_degrees = 0
+ mirror = False
+
+ def isValid(self):
+ return valid
+
+ def pixelFormat(self):
+ return qt.media.QVideoFrameFormat.PixelFormat.Format_BGRA8888
+
+ def map(self, mode):
+ assert mode == qt.media.QVideoFrame.MapMode.ReadOnly
+ return mapping
+
+ def width(self):
+ return width
+
+ def height(self):
+ return height
+
+ def bytesPerLine(self, plane):
+ assert plane == 0
+ return stride
+
+ def mappedBytes(self, plane):
+ return len(pixels) - int(short)
+
+ def bits(self, plane):
+ return memoryview(pixels)
+
+ def unmap(self):
+ self.unmapped = True
+ pixels[:] = b"\0" * len(pixels)
+
+ def surfaceFormat(self):
+ return qt.media.QVideoFrameFormat(qt.core.QSize(width, height), self.pixelFormat())
+
+ def rotation(self):
+ return SimpleNamespace(value=self.rotation_degrees)
+
+ def mirrored(self):
+ return self.mirror
+
+ def toImage(self):
+ raise AssertionError("Packed desktop RGB must not enter GPU conversion")
+
+ return Frame()
+
+
+def test_rgb_stride_colors_and_lifetime_survive_backend_reuse(qt):
+ from jarvis.appshot.recording_frame import owned_capture_image
+
+ colors = [[(211, 17, 39, 255), (12, 207, 36, 255), (23, 43, 227, 255)],
+ [(15, 18, 21, 255), (231, 225, 220, 255), (17, 65, 95, 255)]]
+ frame = frame_for(qt, colors)
+ image = owned_capture_image(frame)
+ assert frame.unmapped
+ assert (image.width(), image.height()) == (3, 2)
+ for y, row in enumerate(colors):
+ for x, color in enumerate(row):
+ assert image.pixelColor(x, y).getRgb() == color
+
+
[email protected]("alpha,empty", [(0, True), (255, False)])
+def test_uninitialized_transparent_buffers_are_skipped_but_black_screens_are_kept(qt, alpha, empty):
+ from jarvis.appshot.recording_frame import owned_capture_image
+
+ frame = frame_for(qt, [[(0, 0, 0, alpha)] * 3] * 2)
+ image = owned_capture_image(frame)
+ assert image.isNull() == empty
+ assert frame.unmapped
+
+
[email protected]("kind", ["invalid", "unmappable", "short"])
+def test_unusable_buffers_do_not_leak_a_mapping_or_return_pixels(qt, kind):
+ from jarvis.appshot.recording_frame import owned_capture_image
+
+ frame = frame_for(qt, [[(10, 20, 30, 255)] * 3] * 2,
+ valid=kind != "invalid", mapping=kind != "unmappable", short=kind == "short")
+ assert owned_capture_image(frame).isNull()
+ assert frame.unmapped == (kind == "short")
+
+
+def test_rotated_and_mirrored_rgb_frames_keep_presentation_orientation(qt):
+ from jarvis.appshot.recording_frame import owned_capture_image
+
+ frame = frame_for(qt, [[(255, 0, 0, 255), (0, 255, 0, 255)],
+ [(0, 0, 255, 255), (255, 255, 255, 255)],
+ [(10, 20, 30, 255), (40, 50, 60, 255)]])
+ frame.rotation_degrees = 90
+ frame.mirror = True
+ image = owned_capture_image(frame)
+ assert (image.width(), image.height()) == (3, 2)
+ assert image.pixelColor(0, 0).getRgb() == (255, 0, 0, 255)
+
+
+def test_desktop_rejects_uninitialized_alpha_without_rejecting_portal_transparency(qt):
+ from jarvis.appshot.recording_frame import owned_capture_image
+
+ colors = [[(0, 43, 68, 43)] * 3] * 2
+ desktop = frame_for(qt, colors)
+ assert owned_capture_image(desktop, opaque_desktop=True).isNull()
+ assert desktop.unmapped
+ portal = frame_for(qt, colors)
+ assert not owned_capture_image(portal).isNull()
diff --git a/tests/unit/appshot/test_recording_overlay.py b/tests/unit/appshot/test_recording_overlay.py
new file mode 100644
index 0000000000..82c9bef811
--- /dev/null
+++ b/tests/unit/appshot/test_recording_overlay.py
@@ -0,0 +1,157 @@
+"""Recording boundary placement, saved-video actions and the shared card handoff."""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.appshot.recording_geometry import elapsed_label, selected_rect, toolbar_rect
+from jarvis.cu.indicator import protocol
+
+
+def test_selection_and_toolbar_follow_a_negative_origin_monitor():
+ screen = (-1920, 100, 1920, 1080)
+ selection = selected_rect(screen, (0.25, 0.2, 0.5, 0.4))
+ assert selection == (-1440, 316, 960, 432)
+ bar = toolbar_rect(selection, screen, (240, 44))
+ assert bar == (-1080, 758, 240, 44)
+
+
+def test_full_screen_controls_remain_inside_the_work_area():
+ bar = toolbar_rect((0, 0, 3840, 2160), (0, 0, 3840, 2100), (260, 44))
+ assert 0 <= bar[0] and bar[0] + bar[2] <= 3840
+ assert 0 <= bar[1] and bar[1] + bar[3] <= 2100
+
+
+def test_controls_use_space_above_a_selection_at_the_bottom_edge():
+ assert toolbar_rect((100, 800, 500, 280), (0, 0, 1920, 1080), (240, 44))[1] == 746
+
+
[email protected]("seconds,text", [(0, "00:00"), (61.9, "01:01"), (3601, "01:00:01")])
+def test_recording_clock_is_elapsed_time(seconds, text):
+ assert elapsed_label(seconds) == text
+
+
+def test_video_poster_keeps_the_first_frame_when_later_frames_change():
+ qt = pytest.importorskip("PySide6.QtGui")
+ from jarvis.appshot.video_encoder import FirstFramePoster
+
+ poster = FirstFramePoster()
+ first = qt.QImage(1280, 720, qt.QImage.Format.Format_RGB32)
+ first.fill(qt.QColor("red"))
+ poster.observe(first)
+ first.fill(qt.QColor("blue"))
+ poster.observe(first)
+ assert poster.image.width() == 560
+ assert poster.image.pixelColor(100, 100).red() == 255
+ assert poster.image.pixelColor(100, 100).blue() == 0
+
+
+async def test_video_preview_uses_the_shared_sidecar_without_overwriting_a_screenshot(monkeypatch):
+ from jarvis.cu.indicator.controller import CUIndicatorController
+
+ controller = CUIndicatorController(SimpleNamespace())
+ controller._card_image_b64 = "pending-screenshot"
+ messages = []
+ monkeypatch.setattr(controller, "_border_capability", lambda: (True, ""))
+ monkeypatch.setattr(
+ controller, "_spawn_sidecar", lambda: setattr(controller, "_proc", object())
+ )
+ monkeypatch.setattr(controller, "_schedule_idle_quit", lambda: None)
+ monkeypatch.setattr("jarvis.cu.indicator.capture_guard.register_hook", lambda _hook: None)
+
+ def send(command, timeout, **payload):
+ messages.append((command, payload))
+ return True
+
+ monkeypatch.setattr(controller, "_send_and_wait", send)
+ assert await controller.show_recording(
+ recording_id="a" * 32,
+ video_path="video.mp4",
+ thumb_b64="jpeg",
+ monitor=[0, 0, 1920, 1080],
+ rect=[0.1, 0.2, 0.4, 0.5],
+ screen_name="display",
+ duration_s=12,
+ rest_ms=6000,
+ labels={},
+ )
+ assert messages[0][0] == protocol.CMD_RECORDING
+ assert messages[0][1]["id"] == "recording:" + "a" * 32
+ assert messages[0][1]["rect"] == [0.1, 0.2, 0.4, 0.5]
+ assert controller._card_image_b64 == "pending-screenshot"
+
+
+def test_saving_a_video_never_overwrites_an_existing_export(monkeypatch, tmp_path):
+ from jarvis.appshot import recording, recording_cards
+
+ recording_id = "a" * 32
+ library = tmp_path / "library"
+ library.mkdir()
+ (library / f"{recording_id}.mp4").write_bytes(b"video-content")
+ monkeypatch.setattr(recording, "recording_dir", lambda: library)
+ monkeypatch.setattr(recording_cards.time, "strftime", lambda _fmt: "stamp")
+ target = tmp_path / "Downloads"
+ first = recording_cards.save_recording(recording_id, target)
+ second = recording_cards.save_recording(recording_id, target)
+ assert first != second
+ assert first.read_bytes() == second.read_bytes() == b"video-content"
+ assert recording_cards.save_recording("../outside", target) is None
+
+
+def test_failed_video_export_removes_only_its_incomplete_copy(monkeypatch, tmp_path):
+ from jarvis.appshot import recording, recording_cards
+
+ (tmp_path / f"{'a' * 32}.mp4").write_bytes(b"video")
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path)
+
+ def failed_copy(source, target, **kwargs):
+ target.write(b"partial")
+ raise OSError("disk full")
+
+ monkeypatch.setattr(recording_cards.shutil, "copyfileobj", failed_copy)
+ exports = tmp_path / "exports"
+ with pytest.raises(OSError, match="disk full"):
+ recording_cards.save_recording("a" * 32, exports)
+ assert not list(exports.iterdir())
+ assert recording.recording_file("a" * 32).read_bytes() == b"video"
+
+
+async def test_video_click_never_opens_the_screenshot_editor(monkeypatch):
+ from jarvis.appshot import recording_cards
+ from jarvis.cu.indicator.controller import CUIndicatorController
+
+ opened = []
+ monkeypatch.setattr(recording_cards, "open_recording", lambda key: opened.append(key))
+ controller = CUIndicatorController(SimpleNamespace())
+ await controller._recording_action("open", "recording:" + "b" * 32)
+ assert opened == ["b" * 32]
+
+
+async def test_recording_card_events_reach_the_video_action_handler(monkeypatch):
+ from jarvis.cu.indicator.controller import CUIndicatorController
+
+ controller = CUIndicatorController(SimpleNamespace())
+ calls = []
+
+ async def action(name, key):
+ calls.append((name, key))
+
+ monkeypatch.setattr(controller, "_recording_action", action)
+ for event in (protocol.EVENT_RECORDING_OPEN, protocol.EVENT_RECORDING_SAVE):
+ payload = protocol.decode_event(protocol.encode_event(event, id="recording:" + "a" * 32))
+ controller._handle_sidecar_event(payload)
+ await asyncio.sleep(0)
+ assert calls == [("open", "recording:" + "a" * 32), ("save", "recording:" + "a" * 32)]
+
+
+async def test_missing_preview_does_not_invalidate_a_saved_video(monkeypatch, tmp_path):
+ from jarvis.appshot import recording, recording_cards
+
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path)
+ (tmp_path / f"{'c' * 32}.mp4").write_bytes(b"video")
+ monkeypatch.setattr("jarvis.cu.indicator.controller.get_indicator_controller", lambda: None)
+ assert not await recording_cards.show_recording_preview("c" * 32, {"preview": True})
+ assert recording.recording_file("c" * 32).read_bytes() == b"video"
diff --git a/tests/unit/appshot/test_recording_quality.py b/tests/unit/appshot/test_recording_quality.py
new file mode 100644
index 0000000000..fdbc02b874
--- /dev/null
+++ b/tests/unit/appshot/test_recording_quality.py
@@ -0,0 +1,358 @@
+"""Recording preferences reach real MP4 encoding and respect each display's limits."""
+
+from __future__ import annotations
+
+import queue
+import threading
+
+import pytest
+from pydantic import ValidationError
+
+from jarvis.appshot.recording_options import RecordingOptions, effective_fps, output_size
+
+
[email protected]("source,resolution,expected", [
+ ((3840, 2160), "1080p", (1920, 1080)),
+ ((3840, 2160), "1440p", (2560, 1440)),
+ ((7680, 4320), "native", (7680, 4320)),
+ ((1920, 1080), "2160p", (1920, 1080)),
+ ((800, 600), "1080p", (800, 600)),
+ ((3440, 1440), "1080p", (1920, 802)),
+ ((2160, 3840), "1080p", (1080, 1920)),
+ ((501, 299), "native", (502, 300)),
+])
+def test_resolution_fits_without_upscaling_or_stretching(source, resolution, expected):
+ assert output_size(*source, resolution) == expected
+
+
[email protected]("requested,hz,expected", [
+ (120, 59.94, 60), (120, 119.88, 120), (120, 144, 120), (60, 75, 60),
+ (30, 144, 30), (120, 0, 60), (120, None, 60), (120, float("nan"), 60),
+ (60, 24, 24), (120, 100, 60),
+])
+def test_frame_rate_uses_selected_monitor(requested, hz, expected):
+ assert effective_fps(requested, hz) == expected
+
+
+def test_config_api_and_worker_share_defaults_and_validation(tmp_path):
+ import json
+ import tomllib
+ from pathlib import Path
+
+ from jarvis.core.config import AppshotConfig
+ from jarvis.core.config_writer import set_appshot_settings
+ from jarvis.ui.web.appshot_routes import SettingsPatch
+
+ expected = {f"recording_{key}": value for key, value in RecordingOptions().model_dump().items()}
+ block = AppshotConfig()
+ assert {key: getattr(block, key) for key in expected} == expected
+ assert SettingsPatch(**expected).model_dump(exclude_none=True) == expected
+ path = tmp_path / "jarvis.toml"
+ path.write_text("[appshot]\nsound = true\n", encoding="utf-8")
+ changed = dict(recording_fps=120, recording_resolution="native", recording_bitrate_mbps=48,
+ recording_system_audio=True)
+ set_appshot_settings(changed, path=path)
+ stored = tomllib.loads(path.read_text(encoding="utf-8"))["appshot"]
+ assert stored == {"sound": True, **changed}
+ assert RecordingOptions.from_config(AppshotConfig(**stored)).fps == 120
+ # Every locale exposes the same real preference labels.
+ root = Path(__file__).resolve().parents[3] / "jarvis/ui/web/frontend/src/i18n/locales"
+ keys = ["recording_resolution", "recording_fps", "recording_bitrate", "recording_system_audio"]
+ for locale in ("en", "de", "es"):
+ labels = json.loads((root / f"{locale}.json").read_text(encoding="utf-8"))["appshots"]
+ assert all(labels[key] for key in keys)
+
+
[email protected]("field,value", [
+ ("fps", 0), ("fps", 240), ("resolution", "8k"), ("bitrate_mbps", 0),
+ ("bitrate_mbps", 101), ("bitrate_mbps", 1.5), ("bitrate_mbps", True),
+])
+def test_invalid_quality_is_rejected_at_all_boundaries(field, value):
+ from jarvis.core.config import AppshotConfig
+ from jarvis.ui.web.appshot_routes import SettingsPatch
+
+ for cls, values in ((RecordingOptions, {field: value}),
+ (AppshotConfig, {f"recording_{field}": value}),
+ (SettingsPatch, {f"recording_{field}": value})):
+ with pytest.raises(ValidationError):
+ cls(**values)
+
+
+def _encoder(tmp_path, *, fps=60, size=(64, 48), resolution="native", audio=False):
+ pytest.importorskip("av")
+ qt = pytest.importorskip("PySide6.QtGui")
+ from jarvis.appshot.video_encoder import VideoEncoder
+
+ encoder = VideoEncoder(tmp_path / "capture.mp4", options=RecordingOptions(
+ resolution=resolution, fps=fps, bitrate_mbps=7, system_audio=audio))
+ # Deterministic producer: the test fills a short clip before the encoder runs.
+ encoder.frames = queue.Queue()
+ image = qt.QImage(*size, qt.QImage.Format.Format_RGBA8888)
+ image.fill(qt.QColor("red"))
+ for frame in range(fps):
+ encoder.submit(image, frame / fps)
+ encoder.stop(1.0)
+ return encoder
+
+
[email protected]("fps", [30, 60, 120])
+def test_real_mp4_has_requested_frame_rate_and_duration(tmp_path, fps):
+ av = pytest.importorskip("av")
+ encoder = _encoder(tmp_path, fps=fps)
+ encoder._run()
+ assert encoder.error == ""
+ with av.open(str(encoder.path)) as source:
+ video = source.streams.video[0]
+ assert video.average_rate == fps
+ assert float(video.duration * video.time_base) == pytest.approx(1.0, abs=1 / fps)
+ assert len(list(source.decode(video))) == fps
+ assert not source.streams.audio
+
+
+def test_real_mp4_downscales_the_pixels(tmp_path):
+ av = pytest.importorskip("av")
+ encoder = _encoder(tmp_path, fps=30, size=(2560, 1440), resolution="1080p")
+ encoder._run()
+ assert encoder.error == ""
+ with av.open(str(encoder.path)) as source:
+ frame = next(source.decode(video=0))
+ assert (frame.width, frame.height) == (1920, 1080)
+
+
[email protected]("speed", [0.5, 1.0, 2.0])
[email protected]("offset", [0, 0.2])
+def test_real_mp4_audio_survives_trim_and_speed_changes(tmp_path, monkeypatch, speed, offset):
+ av = pytest.importorskip("av")
+ np = pytest.importorskip("numpy")
+ from jarvis.appshot import recording_audio
+
+ class Audio:
+ def __init__(self, epoch):
+ self.ready = threading.Event()
+ self.stopping = threading.Event()
+ self.error = ""
+ self.chunks = queue.Queue()
+ count = round((1 - offset) * 48000)
+ wave = (np.sin(np.arange(count) * 440 * 2 * np.pi / 48000) * 0.1).astype("float32")
+ self.chunks.put((np.stack([wave, wave]), round(offset * 48000)))
+
+ def start(self):
+ self.ready.set()
+
+ def stop(self):
+ self.stopping.set()
+
+ monkeypatch.setattr(recording_audio, "SystemAudioCapture", Audio)
+ encoder = _encoder(tmp_path, audio=True)
+ encoder._run()
+ assert encoder.error == ""
+ with av.open(str(encoder.path)) as source:
+ audio = source.streams.audio[0]
+ assert audio.codec_context.sample_rate == 48000
+ assert audio.codec_context.channels == 2
+ assert float(audio.duration * audio.time_base) == pytest.approx(1 - offset, abs=0.05)
+ data = np.concatenate([frame.to_ndarray() for frame in source.decode(audio)], axis=1)
+ assert np.max(np.abs(data)) > 0.05
+
+ from jarvis.appshot import recording, video_edit
+
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path)
+ monkeypatch.setattr(video_edit, "recording_dir", lambda: tmp_path)
+ monkeypatch.setattr(video_edit, "_clips", {})
+ original = tmp_path / ("a" * 32 + ".mp4")
+ encoder.path.rename(original)
+ clip = video_edit.export_clip("a" * 32, 0.1, 0.9, speed)
+ with av.open(str(tmp_path / f"{clip}.mp4")) as source:
+ sound = source.streams.audio[0]
+ picture = source.streams.video[0]
+ audio_duration = float(sound.duration * sound.time_base)
+ video_duration = float(picture.duration * picture.time_base)
+ assert video_duration == pytest.approx(0.8 / speed, abs=0.04)
+ assert audio_duration == pytest.approx(video_duration, abs=0.08)
+ assert picture.base_rate == min(60, round(60 * speed))
+ assert any(np.max(np.abs(frame.to_ndarray())) > 0.05 for frame in source.decode(sound))
+
+
+def test_audio_unavailable_never_falls_back_to_microphone(monkeypatch):
+ from jarvis.appshot import recording_audio
+
+ monkeypatch.setattr(recording_audio.sys, "platform", "darwin")
+ assert not recording_audio.capability()["available"]
+
+
+def test_duplicate_timestamps_do_not_stretch_video(tmp_path):
+ encoder = _encoder(tmp_path)
+ image = encoder.frames.queue[0][0]
+ encoder.frames.put((image, 59 / 60))
+ encoder._run()
+ assert encoder.error == ""
+ assert encoder.dropped_frames == 0 # A redundant early tick is not a lost source frame.
+
+
+def test_capture_gaps_are_reported_without_speeding_up_playback(tmp_path):
+ av = pytest.importorskip("av")
+ encoder = _encoder(tmp_path)
+ image = encoder.frames.queue[0][0]
+ encoder.frames = queue.Queue()
+ encoder.frames.put((image, 0.0))
+ encoder.frames.put((image, 0.5))
+ encoder._run()
+ assert encoder.error == ""
+ assert encoder.dropped_frames == 57
+ with av.open(str(encoder.path)) as source:
+ video = source.streams.video[0]
+ assert float(video.duration * video.time_base) == pytest.approx(1, abs=0.03)
+
+
+def test_hardware_encoder_failure_closes_candidate_before_software_fallback(tmp_path):
+ from types import SimpleNamespace
+
+ from jarvis.appshot.recording_codec import open_video
+
+ attempts = []
+
+ class Container:
+ closed = False
+
+ def add_stream(self, codec, **kwargs):
+ self.codec = codec
+ def open_codec():
+ if codec == "h264_nvenc":
+ raise RuntimeError("Driver unavailable")
+ return SimpleNamespace(codec_context=SimpleNamespace(open=open_codec))
+
+ def close(self):
+ self.closed = True
+
+ def open_container(*args, **kwargs):
+ container = Container()
+ attempts.append(container)
+ return container
+
+ av = SimpleNamespace(codecs_available={"h264_nvenc", "libx264"}, open=open_container)
+ container, stream = open_video(av, tmp_path / "test.partial", (1920, 1080), 60, 37000000)
+ assert len(attempts) == 2 and attempts[0].closed
+ assert container.codec == "libx264" and not container.closed
+ assert stream.bit_rate == 37000000 and stream.pix_fmt == "yuv420p"
+
+
+def test_subframe_clock_jitter_keeps_frames_and_editor_rate(tmp_path, monkeypatch):
+ av = pytest.importorskip("av")
+ from jarvis.appshot import recording, video_edit
+
+ encoder = _encoder(tmp_path)
+ image = encoder.frames.queue[0][0]
+ encoder.frames = queue.Queue()
+ # Two real consecutive frames can arrive inside one nominal frame slot.
+ # Coarse 1/60 timestamps used to discard them and reduce apparent FPS.
+ for index in range(60):
+ at = index / 60 + (0.005 if index % 2 else -0.005)
+ encoder.frames.put((image, max(0, at)))
+ encoder._run()
+ assert not encoder.error
+ with av.open(str(encoder.path)) as source:
+ assert len(list(source.decode(video=0))) >= 60
+ encoder.path.rename(tmp_path / ("b" * 32 + ".mp4"))
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path)
+ monkeypatch.setattr(video_edit, "recording_dir", lambda: tmp_path)
+ monkeypatch.setattr(video_edit, "_clips", {})
+ result = video_edit.export_clip("b" * 32, 0.1, 0.9, 1)
+ with av.open(str(tmp_path / f"{result}.mp4")) as source:
+ assert source.streams.video[0].codec_context.framerate == 60
+
+
+async def test_saved_display_details_do_not_collide_with_preview_geometry():
+ import json
+ from types import SimpleNamespace
+
+ from jarvis.appshot.recording import RecordingService
+
+ class Events:
+ async def __aiter__(self):
+ for phase in ("recording", "saved"):
+ yield json.dumps({
+ "phase": phase, "width": 1920, "height": 1080, "fps": 60,
+ "bitrate_mbps": 12, "system_audio": True, "display_name": "Display 2",
+ "monitor": [-1920, 0, 1920, 1080], "refresh_hz": 60,
+ }).encode("utf-8")
+
+ async def wait():
+ return 0
+
+ service = RecordingService()
+ await service._read(SimpleNamespace(stdout=Events(), stdin=None, wait=wait, returncode=0))
+ state = service.status()
+ assert state["phase"] == "saved"
+ assert state["display_name"] == "Display 2" and state["fps"] == 60
+ assert state["system_audio"] and "monitor" not in state
+
+
[email protected]("codec", ["libx264", "mpeg4"])
+def test_software_only_install_can_encode_without_a_gpu(tmp_path, monkeypatch, codec):
+ av = pytest.importorskip("av")
+ if codec not in av.codecs_available:
+ pytest.skip(f"{codec} is not part of this PyAV build")
+ monkeypatch.setattr(av, "codecs_available", {codec})
+ encoder = _encoder(tmp_path, fps=120)
+ encoder._run()
+ assert not encoder.error
+ with av.open(str(encoder.path)) as source:
+ video = source.streams.video[0]
+ assert video.average_rate == 120
+ assert len(list(source.decode(video))) == 120
+
+
+def test_requested_audio_failure_never_reports_a_silent_success(tmp_path, monkeypatch):
+ from jarvis.appshot import recording_audio
+
+ class UnavailableAudio:
+ def __init__(self, epoch):
+ self.error = "The output device is unavailable."
+ self.ready = threading.Event()
+ self.stopping = threading.Event()
+
+ def start(self):
+ self.ready.set()
+
+ def stop(self):
+ self.stopping.set()
+
+ monkeypatch.setattr(recording_audio, "SystemAudioCapture", UnavailableAudio)
+ encoder = _encoder(tmp_path, audio=True)
+ encoder._run()
+ assert encoder.done.is_set()
+ assert "output device" in encoder.error
+ assert not encoder.path.exists() and not encoder.partial.exists()
+
+
+def test_delayed_first_frame_does_not_create_a_black_lead_in(tmp_path):
+ av = pytest.importorskip("av")
+ encoder = _encoder(tmp_path)
+ image = encoder.frames.queue[0][0]
+ encoder.frames = queue.Queue()
+ encoder.frames.put((image, 0.2))
+ encoder.frames.put((image, 0.3))
+ encoder._run()
+ assert not encoder.error
+ with av.open(str(encoder.path)) as source:
+ stream = source.streams.video[0]
+ assert stream.start_time == 0
+ assert next(source.decode(stream)).pts == 0
+
+
+def test_clip_rate_snaps_jittered_average_to_the_capture_rate():
+ from fractions import Fraction
+ from types import SimpleNamespace
+
+ from jarvis.appshot.video_edit import _nominal_rate
+
+ def stream(declared, average):
+ return SimpleNamespace(
+ codec_context=SimpleNamespace(framerate=declared), average_rate=average)
+
+ # VideoToolbox writes no timing info; jitter skews the average to 59.7 FPS.
+ assert _nominal_rate(stream(None, Fraction(4000, 67))) == 60
+ assert _nominal_rate(stream(Fraction(30), Fraction(4000, 67))) == 30
+ assert _nominal_rate(stream(None, None)) == 60
+ # A rate far from every capture rate is kept as measured.
+ assert _nominal_rate(stream(None, Fraction(37))) == 37
diff --git a/tests/unit/appshot/test_recording_warm.py b/tests/unit/appshot/test_recording_warm.py
new file mode 100644
index 0000000000..aa6dfd653a
--- /dev/null
+++ b/tests/unit/appshot/test_recording_warm.py
@@ -0,0 +1,222 @@
+"""Warm recording lifecycle, shutdown, and startup ownership regression coverage."""
+
+import asyncio
+import json
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.appshot import recording
+
+
+class FakeInput:
+ def __init__(self, process):
+ self.process = process
+ self.closed = False
+
+ def write(self, data):
+ self.process.commands.append(data)
+ if data == b"stop\n":
+ self.process.finish("saved")
+ else:
+ assert json.loads(data)["cmd"] == "start"
+
+ async def drain(self):
+ pass
+
+ def close(self):
+ self.closed = True
+ if self.process.returncode is None:
+ self.process.finish("cancelled")
+
+
+class FakeProcess:
+ def __init__(self):
+ self.stdout = asyncio.StreamReader()
+ self.stdin = FakeInput(self)
+ self.returncode = None
+ self.exited = asyncio.Event()
+ self.commands = []
+ self.stdout.feed_data(b'{"phase":"ready"}\n')
+
+ def finish(self, phase, message=""):
+ self.stdout.feed_data((json.dumps({"phase": phase, "message": message}) + "\n").encode())
+ self.stdout.feed_eof()
+ self.returncode = 0
+ self.exited.set()
+
+ async def wait(self):
+ await self.exited.wait()
+ return self.returncode
+
+ def kill(self):
+ self.finish("error")
+ self.returncode = -9
+
+
+def setup_service(monkeypatch, tmp_path):
+ from jarvis.core import config
+
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path)
+ monkeypatch.setattr(recording, "capability", lambda: {"available": True})
+ monkeypatch.setattr(
+ config,
+ "load_config",
+ lambda: SimpleNamespace(
+ screen_context=SimpleNamespace(enabled=True),
+ ui=SimpleNamespace(language="en"),
+ ),
+ )
+ processes = []
+
+ async def spawn(*args, **kwargs):
+ assert "jarvis.appshot.recording_worker" in args
+ assert "creationflags" in kwargs
+ process = FakeProcess()
+ processes.append(process)
+ return process
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)
+ return recording.RecordingService(), processes
+
+
+async def test_standby_takes_no_pixels_or_file_and_start_reuses_it(setup_service, tmp_path):
+ service, processes = setup_service
+ await service.set_warm(True)
+ await service._warm_task
+ assert service.status()["phase"] == "idle"
+ assert len(processes) == 1 and not processes[0].commands
+ assert not list(tmp_path.iterdir())
+ first = processes[0]
+ try:
+ started = await service.start()
+ assert service._process is first and len(processes) == 1
+ command = json.loads(first.commands[0])
+ assert command["output"].endswith(started["id"] + ".mp4")
+ assert command["language"] == "en"
+ await service.stop()
+ await service._warm_task
+ assert len(processes) == 2 and not processes[1].commands
+ assert service.status()["phase"] == "saved"
+ finally:
+ await service.close()
+ assert all(process.stdin.closed and process.returncode is not None for process in processes)
+
+
+async def test_disabling_warmup_preserves_active_recording_and_prevents_replacement(setup_service):
+ service, processes = setup_service
+ await service.set_warm(True)
+ await service._warm_task
+ await service.start()
+ await service.set_warm(False)
+ assert processes[0].returncode is None
+ assert len(processes[0].commands) == 1
+ await service.stop()
+ assert len(processes) == 1 and service._warm_task is None
+
+
+async def test_dead_standby_is_replaced_on_next_press(setup_service):
+ service, processes = setup_service
+ await service.set_warm(True)
+ await service._warm_task
+ processes[0].finish("error")
+ try:
+ await service.start()
+ assert service._process is processes[1]
+ finally:
+ await service.close()
+
+
+async def test_standby_does_not_bypass_permission_checks(setup_service, monkeypatch):
+ service, processes = setup_service
+ await service.set_warm(True)
+ await service._warm_task
+ monkeypatch.setattr(
+ recording, "capability", lambda: {"available": False, "detail": "Permission revoked"}
+ )
+ try:
+ with pytest.raises(ValueError, match="Permission revoked"):
+ await service.start()
+ assert not processes[0].commands
+ finally:
+ await service.close()
+
+
+async def test_headless_prewarm_does_not_spawn(setup_service, monkeypatch):
+ service, processes = setup_service
+ monkeypatch.setattr(recording, "capability", lambda: {"available": False})
+ await service.set_warm(True)
+ await service._warm_task
+ await service.close()
+ assert not processes
+
+
+async def test_slow_warmup_and_start_share_one_child(setup_service, monkeypatch):
+ service, processes = setup_service
+ original = asyncio.create_subprocess_exec
+ entered, proceed = asyncio.Event(), asyncio.Event()
+
+ async def spawn(*args, **kwargs):
+ entered.set()
+ await proceed.wait()
+ return await original(*args, **kwargs)
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)
+ await service.set_warm(True)
+ await entered.wait()
+ start = asyncio.create_task(service.start())
+ proceed.set()
+ try:
+ await start
+ await service._warm_task
+ assert len(processes) == 1
+ finally:
+ await service.close()
+
+
+async def test_cancelled_warmup_reaps_a_late_spawn(setup_service, monkeypatch):
+ service, processes = setup_service
+ original = asyncio.create_subprocess_exec
+ entered, proceed = asyncio.Event(), asyncio.Event()
+
+ async def spawn(*args, **kwargs):
+ entered.set()
+ await proceed.wait()
+ return await original(*args, **kwargs)
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)
+ await service.set_warm(True)
+ await entered.wait()
+ close = asyncio.create_task(service.close())
+ await asyncio.sleep(0)
+ proceed.set()
+ await close
+ assert len(processes) == 1 and processes[0].stdin.closed
+ assert processes[0].returncode is not None
+
+
+async def test_standby_timeout_reaps_worker_and_allows_retry(setup_service, monkeypatch):
+ from jarvis.appshot import recording_runtime
+
+ service, processes = setup_service
+ original = asyncio.create_subprocess_exec
+
+ async def spawn(*args, **kwargs):
+ process = await original(*args, **kwargs)
+ if len(processes) == 1:
+ await process.stdout.readline() # Simulate a worker that never announces readiness.
+ return process
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)
+ monkeypatch.setattr(recording_runtime, "_READY_TIMEOUT_S", 0.02)
+ await service.set_warm(True)
+ await service._warm_task
+ assert processes[0].stdin.closed
+ try:
+ await service.start()
+ assert service._process is processes[1]
+ finally:
+ await service.close()
+
+
diff --git a/tests/unit/appshot/test_shortcut_prewarm.py b/tests/unit/appshot/test_shortcut_prewarm.py
new file mode 100644
index 0000000000..8087499eed
--- /dev/null
+++ b/tests/unit/appshot/test_shortcut_prewarm.py
@@ -0,0 +1,43 @@
+"""Only the ambient owner prepares Appshot surfaces, outside shortcut arming."""
+
+import asyncio
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.appshot.hotkey import AppshotShortcut
+
+
[email protected](("enabled", "owns"), [(True, True), (False, True), (True, False)])
+async def test_prewarm_is_deferred_and_scoped_to_enabled_primary_instance(
+ monkeypatch, enabled, owns
+):
+ config = SimpleNamespace(
+ screen_context=SimpleNamespace(enabled=enabled),
+ appshot=SimpleNamespace(hotkey="", region_hotkey="", recording_hotkey="", effect=True),
+ )
+ monkeypatch.setattr("jarvis.core.config.load_config", lambda: config)
+ monkeypatch.setattr(
+ "jarvis.core.instance.current_instance", lambda: SimpleNamespace(owns_ambient_duties=owns)
+ )
+ monkeypatch.setattr("jarvis.cu.indicator.controller.get_indicator_controller", lambda: None)
+ entered = asyncio.Event()
+ stopped = asyncio.Event()
+
+ async def warm(_effect):
+ entered.set()
+ try:
+ await asyncio.Future()
+ finally:
+ stopped.set()
+
+ owner = AppshotShortcut(object())
+ monkeypatch.setattr(owner, "_warm_surfaces", warm)
+ await asyncio.wait_for(owner.reload(), 1)
+ if enabled and owns:
+ await asyncio.wait_for(entered.wait(), 1)
+ assert owner._warm_task is not None and not owner._warm_task.done()
+ else:
+ assert not entered.is_set() and owner._warm_task is None
+ await owner.stop()
+ assert stopped.is_set() == (enabled and owns)
diff --git a/tests/unit/appshot/test_video_editor.py b/tests/unit/appshot/test_video_editor.py
new file mode 100644
index 0000000000..2e1be4882e
--- /dev/null
+++ b/tests/unit/appshot/test_video_editor.py
@@ -0,0 +1,120 @@
+"""The video editor: a played recording opens in the app's own editor window,
+and its trim and speed are written as a new recording beside the original."""
+
+from __future__ import annotations
+
+from fractions import Fraction
+
+import pytest
+
+SOURCE = "a" * 32
+
+
+def library(monkeypatch, tmp_path):
+ from jarvis.appshot import recording
+
+ monkeypatch.setattr(recording, "recording_dir", lambda: tmp_path)
+ monkeypatch.setattr("jarvis.appshot.video_edit.recording_dir", lambda: tmp_path)
+ monkeypatch.setattr("jarvis.appshot.video_edit._clips", {})
+ return tmp_path
+
+
+def _write_video(path, seconds: float = 3.0, fps: int = 30) -> None:
+ av = pytest.importorskip("av")
+ np = pytest.importorskip("numpy")
+ with av.open(str(path), "w", format="mp4") as container:
+ codec = "libx264" if "libx264" in av.codecs_available else "mpeg4"
+ stream = container.add_stream(codec, rate=fps)
+ stream.width, stream.height, stream.pix_fmt = 64, 48, "yuv420p"
+ for index in range(round(seconds * fps)):
+ pixels = np.full((48, 64, 3), index % 255, dtype=np.uint8)
+ frame = av.VideoFrame.from_ndarray(pixels, format="rgb24")
+ frame.pts, frame.time_base = index, Fraction(1, fps)
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode(None):
+ container.mux(packet)
+
+
+async def test_playing_a_recording_opens_the_video_editor_window(monkeypatch):
+ from jarvis.appshot import editor_window, recording_cards
+
+ queries, played = [], []
+ editor_window.register_window_opener(lambda query: queries.append(query) or {"ok": True})
+ monkeypatch.setattr(recording_cards, "open_recording", lambda key: played.append(key))
+ try:
+ assert await recording_cards.run_recording_card_action("open", "b" * 32) == ""
+ finally:
+ editor_window.register_window_opener(None)
+ assert queries == ["recording=" + "b" * 32]
+ assert played == []
+
+
+async def test_without_a_window_the_system_player_still_plays_it(monkeypatch):
+ from jarvis.appshot import editor_window, recording_cards
+
+ editor_window.register_window_opener(None)
+ played = []
+ monkeypatch.setattr(recording_cards, "open_recording", lambda key: played.append(key) or True)
+ assert await recording_cards.run_recording_card_action("open", "b" * 32) == ""
+ assert played == ["b" * 32]
+
+
+async def test_only_recording_ids_reach_the_window_url():
+ from jarvis.appshot import editor_window
+
+ queries = []
+ editor_window.register_window_opener(lambda query: queries.append(query) or {"ok": True})
+ try:
+ assert not await editor_window.open_recording_window("../../etc/passwd")
+ assert not await editor_window.open_recording_window("abc")
+ finally:
+ editor_window.register_window_opener(None)
+ assert queries == []
+
+
+def test_a_trimmed_faster_clip_is_a_new_recording(library):
+ pytest.importorskip("av")
+ from jarvis.appshot.recording import recording_file
+ from jarvis.appshot.video_edit import export_clip, probe_duration
+
+ _write_video(library / f"{SOURCE}.mp4")
+ original = (library / f"{SOURCE}.mp4").read_bytes()
+ clip = export_clip(SOURCE, 0.5, 2.5, 2.0)
+ assert clip != SOURCE and recording_file(clip) is not None
+ assert probe_duration(clip) == pytest.approx(1.0, abs=0.1)
+ # The original is never touched, and the same edit is not encoded twice.
+ assert (library / f"{SOURCE}.mp4").read_bytes() == original
+ assert export_clip(SOURCE, 0.5, 2.5, 2.0) == clip
+ assert not list(library.glob("*.partial"))
+
+
+def test_slow_motion_stretches_the_clip(library):
+ pytest.importorskip("av")
+ from jarvis.appshot.video_edit import export_clip, probe_duration
+
+ _write_video(library / f"{SOURCE}.mp4")
+ assert probe_duration(export_clip(SOURCE, 1.0, 2.0, 0.5)) == pytest.approx(2.0, abs=0.1)
+
+
+def test_an_unchanged_edit_keeps_the_original(library):
+ pytest.importorskip("av")
+ from jarvis.appshot.video_edit import export_clip, probe_duration
+
+ _write_video(library / f"{SOURCE}.mp4")
+ assert export_clip(SOURCE, 0.0, probe_duration(SOURCE), 1.0) == SOURCE
+ assert len(list(library.glob("*.mp4"))) == 1
+
+
+def test_bad_edits_are_refused_with_a_clear_reason(library):
+ pytest.importorskip("av")
+ from jarvis.appshot.video_edit import ClipError, export_clip
+
+ _write_video(library / f"{SOURCE}.mp4")
+ with pytest.raises(ClipError, match="speed"):
+ export_clip(SOURCE, 0.0, 1.0, 3.0)
+ with pytest.raises(ClipError, match="too short"):
+ export_clip(SOURCE, 1.0, 1.1, 1.0)
+ with pytest.raises(FileNotFoundError):
+ export_clip("c" * 32, 0.0, 1.0, 1.0)
diff --git a/tests/unit/audio/test_duck_controller.py b/tests/unit/audio/test_duck_controller.py
index d8ad67f5e3..7898e138ed 100644
--- a/tests/unit/audio/test_duck_controller.py
+++ b/tests/unit/audio/test_duck_controller.py
@@ -1,10 +1,16 @@
"""AudioDuckController: session-boundary mute/restore with a fake ducker."""
from __future__ import annotations
+import asyncio
+import threading
from types import SimpleNamespace
+import pytest
+
from jarvis.audio.ducking.controller import AudioDuckController
from jarvis.audio.ducking.protocol import DuckPermissionReport
+from jarvis.core.bus import EventBus
+from jarvis.core.events import VoiceSessionEnded, VoiceSessionStarted
class FakeDucker:
@@ -34,11 +40,17 @@ def _cfg(enabled=True, delay=0):
)
+async def _settle(controller):
+ while controller._tasks:
+ await asyncio.gather(*controller._tasks)
+
+
async def test_mutes_on_start_restores_on_end_when_enabled():
d, bus = FakeDucker(), FakeBus()
c = AudioDuckController(bus=bus, cfg=_cfg(enabled=True), ducker=d)
c.attach()
await bus.subs["VoiceSessionStarted"](object())
+ await _settle(c)
assert d.muted_calls == 1 and c._muted == [111, 222]
await bus.subs["VoiceSessionEnded"](object())
assert d.restored == [[111, 222]] and c._muted == []
@@ -57,6 +69,7 @@ async def test_set_enabled_false_midsession_restores():
c = AudioDuckController(bus=bus, cfg=_cfg(enabled=True), ducker=d)
c.attach()
await bus.subs["VoiceSessionStarted"](object())
+ await _settle(c)
await c.set_enabled(False)
assert d.restored == [[111, 222]] and c._muted == []
assert c._cfg.ducking.enabled is False
@@ -68,6 +81,7 @@ async def test_double_start_does_not_remute():
c.attach()
await bus.subs["VoiceSessionStarted"](object())
await bus.subs["VoiceSessionStarted"](object())
+ await _settle(c)
assert d.muted_calls == 1 # already muted → no second sweep
@@ -80,16 +94,16 @@ async def test_restore_idempotent_when_nothing_muted():
async def test_new_session_during_restore_delay_keeps_music_muted():
- import asyncio
-
d, bus = FakeDucker(), FakeBus()
c = AudioDuckController(bus=bus, cfg=_cfg(enabled=True, delay=50), ducker=d)
c.attach()
await bus.subs["VoiceSessionStarted"](object()) # session 1 mutes [111, 222]
+ await _settle(c)
end_task = asyncio.create_task(bus.subs["VoiceSessionEnded"](object()))
await asyncio.sleep(0.01) # let _on_end capture the generation + enter the sleep
await bus.subs["VoiceSessionStarted"](object()) # session 2 bumps the generation
await end_task # session 1's delayed restore wakes → generation changed → SKIP
+ await _settle(c)
assert d.restored == [] # music stayed muted for the still-active session 2
assert c._muted == [111, 222]
@@ -99,6 +113,7 @@ async def test_restore_sync_unmutes_on_shutdown():
c = AudioDuckController(bus=bus, cfg=_cfg(enabled=True), ducker=d)
c.attach()
await bus.subs["VoiceSessionStarted"](object())
+ await _settle(c)
c.restore_sync() # synchronous (shutdown path)
assert d.restored == [[111, 222]] and c._muted == []
c.restore_sync() # idempotent
@@ -153,5 +168,139 @@ async def test_a_session_never_asks_the_backend_to_prewarm():
c = AudioDuckController(bus=bus, cfg=_cfg(enabled=True), ducker=d)
c.attach()
await bus.subs["VoiceSessionStarted"](object())
+ await _settle(c)
await bus.subs["VoiceSessionEnded"](object())
assert d.prewarms == 0 and d.muted_calls == 1
+
+
+class DelayedDucker(FakeDucker):
+ def __init__(self):
+ super().__init__()
+ self.entered = threading.Event()
+ self.release = threading.Event()
+ self.restored_event = threading.Event()
+
+ def mute_others(self, *, own_pid, never):
+ self.entered.set()
+ assert self.release.wait(5), "test did not release the mute worker"
+ return super().mute_others(own_pid=own_pid, never=never)
+
+ def restore(self, pids):
+ super().restore(pids)
+ self.restored_event.set()
+
+
+async def test_slow_mute_does_not_hold_up_the_lifecycle_publisher():
+ d, bus = DelayedDucker(), EventBus()
+ c = AudioDuckController(bus=bus, cfg=_cfg(), ducker=d)
+ c.attach()
+ published = asyncio.create_task(bus.publish(VoiceSessionStarted()))
+ try:
+ assert await asyncio.to_thread(d.entered.wait, 2)
+ await asyncio.wait_for(asyncio.shield(published), 0.5)
+ assert not d.release.is_set() and d.muted_calls == 0
+ finally:
+ d.release.set()
+ await published
+ await _settle(c)
+ await c.restore()
+
+
[email protected]("cleanup", ["hangup", "toggle", "restore"])
+async def test_late_mute_is_restored_after_session_cleanup(cleanup):
+ d = DelayedDucker()
+ c = AudioDuckController(bus=FakeBus(), cfg=_cfg(), ducker=d)
+ await c._on_start(object())
+ try:
+ assert await asyncio.to_thread(d.entered.wait, 2)
+ if cleanup == "hangup":
+ ending = asyncio.create_task(c._on_end(object()))
+ elif cleanup == "toggle":
+ ending = asyncio.create_task(c.set_enabled(False))
+ else:
+ ending = asyncio.create_task(c.restore())
+ await asyncio.sleep(0)
+ assert not ending.done()
+ finally:
+ d.release.set()
+ await ending
+ await _settle(c)
+ assert d.muted_calls == 1 and d.restored == [[111, 222]]
+ assert c._muted == []
+
+
+async def test_cancelled_hangup_waiter_keeps_late_mute_cleanup_owned():
+ d = DelayedDucker()
+ c = AudioDuckController(bus=FakeBus(), cfg=_cfg(), ducker=d)
+ await c._on_start(object())
+ try:
+ assert await asyncio.to_thread(d.entered.wait, 2)
+ ending = asyncio.create_task(c._on_end(object()))
+ await asyncio.sleep(0)
+ ending.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await ending
+ finally:
+ d.release.set()
+ await _settle(c)
+ assert d.restored == [[111, 222]] and c._muted == []
+
+
[email protected]("cancel_waiter", [False, True])
+async def test_shutdown_bounds_native_wait_and_restores_the_late_mute(cancel_waiter, caplog):
+ d = DelayedDucker()
+ c = AudioDuckController(bus=FakeBus(), cfg=_cfg(), ducker=d)
+ await c._on_start(object())
+ try:
+ assert await asyncio.to_thread(d.entered.wait, 2)
+ if cancel_waiter:
+ (muting,) = c._tasks
+ muting.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await muting
+ shutdown = asyncio.create_task(asyncio.to_thread(c.restore_sync))
+ await asyncio.wait_for(asyncio.shield(shutdown), 1)
+ assert c._closed.is_set() and not d.release.is_set()
+ assert d.restored == []
+ assert "shutdown cleanup is pending" in caplog.text
+ finally:
+ d.release.set()
+ await shutdown
+ assert await asyncio.to_thread(d.restored_event.wait, 2)
+ await _settle(c)
+ assert d.restored == [[111, 222]] and c._muted == []
+ c.restore_sync()
+ assert d.restored == [[111, 222]]
+ await c._on_start(object())
+ await _settle(c)
+ assert d.muted_calls == 1
+
+
+async def test_shutdown_before_queued_mute_prevents_a_later_native_effect():
+ d = FakeDucker()
+ c = AudioDuckController(bus=FakeBus(), cfg=_cfg(), ducker=d)
+ await c._on_start(object())
+ c.restore_sync()
+ await _settle(c)
+ assert d.muted_calls == 0 and d.restored == []
+
+
+async def test_old_restore_waiting_for_mute_does_not_unmute_a_new_session():
+ d = DelayedDucker()
+ c = AudioDuckController(bus=FakeBus(), cfg=_cfg(), ducker=d)
+ await c._on_start(object())
+ try:
+ assert await asyncio.to_thread(d.entered.wait, 2)
+ ending = asyncio.create_task(c._on_end(object()))
+ await asyncio.sleep(0)
+ await asyncio.sleep(0)
+ assert not c._duck_requested
+ await c._on_start(object())
+ finally:
+ d.release.set()
+ await ending
+ await _settle(c)
+ assert d.muted_calls == 1 and d.restored == []
+ assert c._muted == [111, 222]
+ await c._on_end(VoiceSessionEnded())
+ assert d.restored == [[111, 222]]
diff --git a/tests/unit/audio/test_ducking_permissions.py b/tests/unit/audio/test_ducking_permissions.py
index 2c1e7dbef4..3ec5c1c7c9 100644
--- a/tests/unit/audio/test_ducking_permissions.py
+++ b/tests/unit/audio/test_ducking_permissions.py
@@ -475,6 +475,7 @@ async def test_a_report_that_hangs_never_holds_the_controller_session_start_up(m
try:
await asyncio.wait_for(controller._on_start(object()), 5)
+ await asyncio.wait_for(asyncio.gather(*controller._tasks), 5)
assert entered.is_set() # the report really was out while the session started
assert controller._muted == [2] # the other player still ducked
assert not controller._lock.locked()
@@ -553,6 +554,7 @@ async def test_the_controller_lock_is_free_while_the_dialog_is_open(monkeypatch)
# The user starts and ends a voice session while the dialog is still open:
# neither handler waits for it, and neither asks anything.
await asyncio.wait_for(bus.subs["VoiceSessionStarted"](object()), 5)
+ await asyncio.wait_for(asyncio.gather(*controller._tasks), 5)
assert controller._muted == [] # no grant yet: nothing was scripted
await asyncio.wait_for(bus.subs["VoiceSessionEnded"](object()), 5)
assert not ask.done() and tcc.requests("automation") == []
@@ -564,6 +566,7 @@ async def test_the_controller_lock_is_free_while_the_dialog_is_open(monkeypatch)
# The ducker's own state lock-free too: a later session ducks with the new grant.
get_permission_service().invalidate()
await asyncio.wait_for(bus.subs["VoiceSessionStarted"](object()), 5)
+ await asyncio.wait_for(asyncio.gather(*controller._tasks), 5)
assert controller._muted == [1]
diff --git a/tests/unit/brain/test_claude_provider_status.py b/tests/unit/brain/test_claude_provider_status.py
new file mode 100644
index 0000000000..0274c871be
--- /dev/null
+++ b/tests/unit/brain/test_claude_provider_status.py
@@ -0,0 +1,60 @@
+"""The provider catalog and Claude connection card agree without model calls."""
+
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis import claude_auth
+from jarvis.brain import app_control
+from jarvis.ui.web.provider_spec import get_spec
+
+
+ ("connected", "key", "api_key_present"),
+ [(True, None, False), (False, None, False),
+ (True, "fixture-api-key", True), (False, "sk-ant-oat-fixture", False)],
+)
+def test_claude_catalog_uses_connection_status(monkeypatch, connected, key, api_key_present):
+ calls = []
+
+ class AuthService:
+ def __init__(self, binary_path, **kwargs):
+ calls.append((binary_path, kwargs))
+
+ def status(self):
+ return SimpleNamespace(connected=connected)
+
+ monkeypatch.setattr(claude_auth, "ClaudeAuthService", AuthService)
+ monkeypatch.setattr(app_control.cfg_mod, "get_jarvis_agent_secret", lambda provider: key)
+ assert app_control.is_credential_present(get_spec("claude-cli"), "fixture-claude") is connected
+ assert calls == [("fixture-claude", {"api_key_present": api_key_present})]
+
+
+def test_claude_catalog_survives_failed_status_without_leaking_secrets(monkeypatch, caplog):
+ def unavailable(*args, **kwargs):
+ raise OSError("private-credential-detail")
+
+ monkeypatch.setattr(app_control.cfg_mod, "get_jarvis_agent_secret", lambda provider: None)
+ monkeypatch.setattr(claude_auth, "ClaudeAuthService", unavailable)
+ with caplog.at_level("DEBUG", logger=app_control.__name__):
+ assert not app_control.is_credential_present(get_spec("claude-cli"))
+ assert "OSError" in caplog.text
+ assert "private-credential-detail" not in caplog.text
+
+
+def test_unreadable_keyring_does_not_hide_native_login(monkeypatch, caplog):
+ def unavailable(provider):
+ raise OSError("private-keyring-detail")
+
+ class AuthService:
+ def __init__(self, binary_path, *, api_key_present):
+ assert not api_key_present
+
+ def status(self):
+ return SimpleNamespace(connected=True)
+
+ monkeypatch.setattr(app_control.cfg_mod, "get_jarvis_agent_secret", unavailable)
+ monkeypatch.setattr(claude_auth, "ClaudeAuthService", AuthService)
+ with caplog.at_level("DEBUG", logger=app_control.__name__):
+ assert app_control.is_credential_present(get_spec("claude-cli"))
+ assert "private-keyring-detail" not in caplog.text
diff --git a/tests/unit/brain/test_model_catalog_local.py b/tests/unit/brain/test_model_catalog_local.py
index 5ca9008a44..21584bcb7d 100644
--- a/tests/unit/brain/test_model_catalog_local.py
+++ b/tests/unit/brain/test_model_catalog_local.py
@@ -267,7 +267,9 @@ class _FakeOllamaClient(_FakeClient):
self.show_caps = show_caps
self.shown: list[str] = []
- async def post(self, url: str, json: dict[str, Any] | None = None) -> _FakeResponse:
+ async def post(
+ self, url: str, json: dict[str, Any] | None = None, headers=None,
+ ) -> _FakeResponse:
name = str((json or {}).get("model") or "")
self.shown.append(name)
assert url.endswith("/api/show")
@@ -319,7 +321,7 @@ async def test_ollama_catalog_survives_a_server_without_api_show(tmp_path, monke
(capable), exactly as before this enrichment existed."""
class _NoShow(_FakeOllamaClient):
- async def post(self, url, json=None): # type: ignore[override]
+ async def post(self, url, json=None, headers=None): # type: ignore[override]
raise RuntimeError("api/show unavailable")
_local_env(monkeypatch)
diff --git a/tests/unit/brain/test_ollama_profiles.py b/tests/unit/brain/test_ollama_profiles.py
index 5587205946..c5425ce216 100644
--- a/tests/unit/brain/test_ollama_profiles.py
+++ b/tests/unit/brain/test_ollama_profiles.py
@@ -2,7 +2,7 @@
The server is a fake behind ``httpx.MockTransport`` (a real httpx feature, no
``unittest.mock``): it records every call so the tests can pin how many
-round-trips a turn costs, which is the whole point of the idempotency memo.
+round-trips a turn costs and whether other callers' profiles survive.
"""
from __future__ import annotations
@@ -110,28 +110,43 @@ async def test_ensure_profile_creates_once_and_is_idempotent() -> None:
assert create == [
{"model": alias, "from": "qwen3.5:9b", "parameters": {"num_ctx": 16384}, "stream": False}
]
- # Second call: the process memo answers, no HTTP at all.
+ # Recheck existence without recreating the profile.
before = len(server.calls)
assert await ensure_profile(ROOT, "qwen3.5:9b", opts, transport=server.transport) == alias
- assert len(server.calls) == before
+ assert server.calls[before:] == [("GET", "/api/tags", None)]
# A fresh process finds the alias on the server and skips the create.
profiles.reset_process_memo()
assert await ensure_profile(ROOT, "qwen3.5:9b", opts, transport=server.transport) == alias
assert server.paths("POST").count("/api/create") == 1
-async def test_ensure_profile_deletes_the_stale_alias_when_the_hash_changes() -> None:
+async def test_ensure_profile_preserves_other_callers_option_sets() -> None:
old = profile_name("qwen3.5:9b", OllamaModelOptions(num_ctx=8192))
other = profile_name("gemma4:12b", OllamaModelOptions(num_ctx=8192))
server = FakeOllamaServer(["qwen3.5:9b", f"{old}:latest", f"{other}:latest"])
new_opts = OllamaModelOptions(num_ctx=16384)
alias = await ensure_profile(ROOT, "qwen3.5:9b", new_opts, transport=server.transport)
deleted = [b["model"] for m, p, b in server.calls if p == "/api/delete"]
- assert deleted == [old] # the other base's alias is left alone
+ assert deleted == []
+ assert f"{old}:latest" in server.models
assert f"{alias}:latest" in server.models
assert f"{other}:latest" in server.models
+async def test_two_agents_can_alternate_context_profiles_and_recover_removal() -> None:
+ server = FakeOllamaServer(["qwen3.5:9b"])
+ small = OllamaModelOptions(num_ctx=4096)
+ large = OllamaModelOptions(num_ctx=8192)
+ first = await ensure_profile(ROOT, "qwen3.5:9b", small, transport=server.transport)
+ second = await ensure_profile(ROOT, "qwen3.5:9b", large, transport=server.transport)
+ assert await ensure_profile(ROOT, "qwen3.5:9b", small, transport=server.transport) == first
+ assert {f"{first}:latest", f"{second}:latest"}.issubset(server.models)
+ server.models.remove(f"{first}:latest")
+ assert await ensure_profile(ROOT, "qwen3.5:9b", small, transport=server.transport) == first
+ assert f"{first}:latest" in server.models
+ assert "/api/delete" not in server.paths()
+
+
async def test_ensure_profile_without_bakeable_knobs_returns_the_base() -> None:
server = FakeOllamaServer(["qwen3.5:9b"])
opts = OllamaModelOptions(temperature=0.2, keep_alive="30m", think=False)
diff --git a/tests/unit/core/test_agent_server.py b/tests/unit/core/test_agent_server.py
new file mode 100644
index 0000000000..e2c195cd39
--- /dev/null
+++ b/tests/unit/core/test_agent_server.py
@@ -0,0 +1,197 @@
+"""Independent server ownership, credential-free endpoints and atomic settings."""
+
+from __future__ import annotations
+
+import asyncio
+import tomllib
+from types import SimpleNamespace
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.core import background_service as bg
+from jarvis.core.config import BackgroundConfig
+from jarvis.core.server_endpoint import server_endpoint
+
+
+ "https://jarvis.example.com", "http://127.0.0.1:47821", "http://[::1]:47821/",
+])
+def test_valid_endpoint(url):
+ assert server_endpoint(url) == url.rstrip("/")
+
+
+ "http://jarvis.example.com", "https://user:[email protected]", "https://example.com?key=x",
+ "file:///etc/passwd", "https://example.com/#secret", "https://example.com/api",
+ "http://localhost:0", "http://localhost:99999", "//example.com",
+])
+def test_endpoint_rejects_insecure_or_credential_bearing_address(url):
+ with pytest.raises(ValueError):
+ server_endpoint(url)
+
+
+def test_setting_is_atomic_and_preserves_the_other_background_options(tmp_path):
+ from jarvis.core.config_writer import set_agent_server
+
+ path = tmp_path / "jarvis.toml"
+ path.write_text("[background]\nkeep_agents_running = false\n", encoding="utf8")
+ set_agent_server(persistent=True, server_url="https://jarvis.example.com/", path=path)
+ config = BackgroundConfig(**tomllib.loads(path.read_text(encoding="utf8"))["background"])
+ assert config.persistent_server
+ assert config.server_url == "https://jarvis.example.com"
+ assert not config.keep_agents_running
+ before = path.read_bytes()
+ with pytest.raises(ValueError):
+ set_agent_server(persistent=False, server_url="http://public.example.com", path=path)
+ assert path.read_bytes() == before
+
+
+def test_server_spawn_keeps_independent_mode_and_instance(monkeypatch):
+ from jarvis.core.instance import resolve_instance
+
+ monkeypatch.setattr(
+ "jarvis.core.instance.current_instance",
+ lambda: resolve_instance({"JARVIS_INSTANCE": "dev"}),
+ )
+ command = bg.service_command(after_pid=None, persistent=True, frozen=False, executable="python")
+ assert "--persistent-server" in command
+ assert command[command.index("--instance") + 1] == "dev"
+ assert "--after-pid" not in command
+
+
+async def test_persistent_server_has_no_idle_or_handover_watch(monkeypatch):
+ from jarvis.ui import background_tray
+ from jarvis.ui.web import launcher
+
+ class Tray:
+ def __init__(self, **kwargs):
+ self.stop_owner = kwargs["on_stop"]
+
+ def start(self):
+ return True
+
+ monkeypatch.setattr(background_tray, "BackgroundTray", Tray)
+ stop = asyncio.Event()
+ task, tray = launcher._start_background_service_duties(
+ SimpleNamespace(), stop, persistent=True,
+ )
+ assert task is None
+ assert not stop.is_set()
+ tray.stop_owner()
+ await asyncio.sleep(0)
+ assert stop.is_set()
+
+
+def test_reopening_never_takes_over_a_persistent_server(monkeypatch):
+ from jarvis.ui.web import launcher
+
+ monkeypatch.setattr(bg, "read_marker", lambda: {"persistent": True})
+
+ def forbidden(*args, **kwargs):
+ pytest.fail("A client must not request a server handover")
+
+ monkeypatch.setattr(bg, "take_over_from_service", forbidden)
+ assert launcher._take_over_from_background_service() is None
+
+
+def test_explicit_stop_is_separate_from_client_connections():
+ from jarvis.ui.web.agent_server_routes import router
+
+ app = FastAPI()
+ app.include_router(router)
+ stops = []
+ app.state.agent_server_mode = True
+ app.state.agent_server_ready = True
+ app.state.agent_server_stop = lambda: stops.append("stopped")
+ for _ in range(2):
+ with TestClient(app) as client:
+ assert client.get("/api/agent-server/status").json()["ready"]
+ assert stops == []
+ with TestClient(app) as client:
+ assert client.post("/api/agent-server/stop").json() == {"stopping": True}
+ assert stops == ["stopped"]
+ assert app.openapi()["paths"]["/api/agent-server/stop"]["post"]["x-jarvis-dangerous"]
+
+
+async def test_agent_services_start_without_a_client(monkeypatch):
+ from jarvis.ui.web import agent_chat_routes
+ from jarvis.ui.web.agent_server import start_agents
+
+ restored = []
+
+ class Runtime:
+ async def ensure_started(self):
+ restored.append("society")
+
+ service = object()
+ monkeypatch.setattr(agent_chat_routes, "_service_from_state", lambda state: service)
+ runtime = Runtime()
+ state = SimpleNamespace(society=None, society_factory=lambda: runtime)
+ await start_agents(state)
+ assert state.agent_server_ready and state.society is runtime
+ assert restored == ["society"]
+
+
+def test_concurrent_clients_launch_only_one_server(tmp_path, monkeypatch):
+ from concurrent.futures import ThreadPoolExecutor
+
+ from jarvis.ui import desktop_app, server_client
+
+ marker = {}
+ starts = []
+ monkeypatch.setattr(bg, "marker_path", lambda: tmp_path / "service.json")
+ monkeypatch.setattr(bg, "read_marker", lambda: marker)
+ monkeypatch.setattr(bg, "service_pid", lambda: 42 if marker else None)
+ monkeypatch.setattr(desktop_app, "_read_meta", lambda: {})
+ monkeypatch.setattr(server_client, "wait_for_server", lambda url: None)
+
+ def spawn(**kwargs):
+ starts.append(kwargs)
+ marker.update(persistent=True, port=47999)
+ return True
+
+ with ThreadPoolExecutor(max_workers=2) as pool:
+ jobs = [pool.submit(server_client.local_server, port=47999, spawn=spawn) for _ in range(2)]
+ assert [job.result() for job in jobs] == ["http://127.0.0.1:47999"] * 2
+ assert starts == [{"after_pid": None, "persistent": True, "port": 47999}]
+
+
+def test_local_client_refuses_to_interrupt_integrated_owner(tmp_path, monkeypatch):
+ from jarvis.ui import desktop_app, server_client
+
+ monkeypatch.setattr(bg, "marker_path", lambda: tmp_path / "service.json")
+ monkeypatch.setattr(bg, "read_marker", lambda: None)
+ monkeypatch.setattr(bg, "service_pid", lambda: None)
+ monkeypatch.setattr(desktop_app, "_read_meta", lambda: {"pid": 42})
+ monkeypatch.setattr(desktop_app, "_pid_alive", lambda pid: True)
+ with pytest.raises(RuntimeError, match="integrated desktop"):
+ server_client.local_server(port=47999, spawn=lambda **kw: pytest.fail("must not spawn"))
+
+
+def test_remote_readiness_never_reads_or_sends_local_credentials(monkeypatch):
+ from jarvis.core import control_key
+ from jarvis.ui.server_client import wait_for_server
+
+ monkeypatch.setattr(control_key, "get_control_key", lambda: pytest.fail("must not read key"))
+ with pytest.raises(ValueError, match="loopback"):
+ wait_for_server("https://jarvis.example.com", timeout_s=0.1)
+
+
+def test_direct_persistent_entry_is_recognized(monkeypatch):
+ import psutil
+
+ monkeypatch.setattr(psutil, "Process", lambda pid: SimpleNamespace(
+ cmdline=lambda: ["python", "-m", "jarvis", "--persistent-server"],
+ ))
+ assert bg._is_service_process(42)
+
+
+def test_enabling_server_mode_hands_off_even_without_scheduled_work(monkeypatch):
+ started = []
+ monkeypatch.setattr(bg, "clear_handover_request", lambda: None)
+ monkeypatch.setattr(bg, "spawn_service", lambda **kw: started.append(kw) or True)
+ cfg = SimpleNamespace(background=BackgroundConfig(persistent_server=True))
+ assert bg.hand_off_on_quit(SimpleNamespace(), cfg, pid=42)
+ assert started == [{"after_pid": 42, "persistent": True}]
diff --git a/tests/unit/cu/indicator/test_appshot_warm.py b/tests/unit/cu/indicator/test_appshot_warm.py
new file mode 100644
index 0000000000..a6f0221389
--- /dev/null
+++ b/tests/unit/cu/indicator/test_appshot_warm.py
@@ -0,0 +1,58 @@
+"""Idle shutter preparation preserves other users of the shared sidecar."""
+
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.cu.indicator import protocol
+from jarvis.cu.indicator.controller import CUIndicatorController
+
+
+def warm_controller(monkeypatch):
+ ctl = CUIndicatorController(object())
+ calls = []
+ monkeypatch.setattr(ctl, "_border_capability", lambda: (True, ""))
+
+ def spawn():
+ calls.append("spawn")
+ ctl._proc = SimpleNamespace(poll=lambda: None)
+
+ async def quit_sidecar():
+ calls.append("quit")
+ ctl._proc = None
+
+ monkeypatch.setattr(ctl, "_spawn_sidecar", spawn)
+ monkeypatch.setattr(ctl, "_send_and_wait", lambda cmd, *_a, **_k: calls.append(cmd) or True)
+ monkeypatch.setattr(ctl, "_quit_sidecar", quit_sidecar)
+ return ctl, calls
+
+
+async def test_warm_runtime_survives_card_close_and_is_released_on_shutdown(warm_controller):
+ ctl, calls = warm_controller
+ await ctl.warm_for_appshots(True)
+ assert calls == ["spawn", protocol.CMD_HIDE]
+ await ctl._idle_quit()
+ assert "quit" not in calls
+ await ctl.close()
+ assert calls[-1] == "quit"
+
+
+async def test_prewarm_never_hides_an_existing_mission_border(warm_controller):
+ ctl, calls = warm_controller
+ ctl._proc = SimpleNamespace(poll=lambda: None)
+ ctl._active = 1
+ await ctl.warm_for_appshots(True)
+ assert calls == []
+ await ctl.warm_for_appshots(False)
+ await ctl._idle_quit_task
+ assert calls == []
+ await ctl.close()
+
+
+async def test_disabling_appshots_releases_idle_runtime(warm_controller):
+ ctl, calls = warm_controller
+ await ctl.warm_for_appshots(True)
+ await ctl.warm_for_appshots(False)
+ await ctl._idle_quit_task
+ assert calls[-1] == "quit"
diff --git a/tests/unit/live/test_browser_startup_timing.py b/tests/unit/live/test_browser_startup_timing.py
new file mode 100644
index 0000000000..83e28a4296
--- /dev/null
+++ b/tests/unit/live/test_browser_startup_timing.py
@@ -0,0 +1,26 @@
+"""Browser startup milestones must survive the backend's telemetry allowlist."""
+
+import re
+from pathlib import Path
+
+from jarvis.live.timing import BROWSER_PHASES, StartupTimings
+
+
+def test_frontend_startup_milestones_are_accepted_without_clock_conversion():
+ source = (
+ Path(__file__).resolve().parents[3]
+ / "jarvis/ui/web/frontend/src/lib/realtimeAudio.ts"
+ ).read_text(encoding="utf-8")
+ phases = set(re.findall(
+ r'\b(?:markStartup|markLocalStartup|onStartupPhase)(?:\?\.)?\(\s*"([^"]+)"',
+ source,
+ ))
+ assert phases
+ assert phases <= BROWSER_PHASES
+ timing = StartupTimings("browser-startup-test")
+ expected = {phase: float(index * 10) for index, phase in enumerate(sorted(phases))}
+ # Match the browser's bounded compatibility batches.
+ pairs = list(expected.items())
+ for offset in range(0, len(pairs), 8):
+ timing.browser(dict(pairs[offset:offset + 8]))
+ assert timing.browser_marks == expected
diff --git a/tests/unit/live/test_followup_tools.py b/tests/unit/live/test_followup_tools.py
new file mode 100644
index 0000000000..47bfb2ba9b
--- /dev/null
+++ b/tests/unit/live/test_followup_tools.py
@@ -0,0 +1,69 @@
+"""Steering reuses effects without treating low-risk sends as reads or consent."""
+
+import json
+
+import pytest
+
+from jarvis.live.state import LiveLedger
+from jarvis.live.tools import LiveTools
+from tests.fakes.fake_subscription_session import SafeMessageGateway
+
+
+def runtime(tmp_path):
+ ledger = LiveLedger(tmp_path / "receipts.db")
+ tools = LiveTools(SafeMessageGateway(), ledger, "call", language="en", backend_model="")
+ yield tools
+ ledger.close()
+
+
+async def test_internal_message_is_an_effect_even_when_permission_tier_is_safe(runtime):
+ runtime.revision = 2
+ args = {"name": "message_agent", "arguments_json": '{"target":"Alpha","message":"Hi"}'}
+ assert runtime.steering_replay_key("call_tool", args) is not None
+ result = await runtime.execute("obsolete", "call_tool", args, 1)
+ assert result["status"] == "superseded"
+ assert not runtime.gateway.calls
+
+
+def test_replay_identity_uses_canonical_tool_target_and_payload(runtime):
+ runtime.declarations()
+ alias = next(alias for alias, name in runtime._names.items() if name == "message_agent")
+ args = {"target": "Alpha", "message": "Run tests"}
+ wrapped = {"name": "message_agent", "arguments_json": json.dumps(args)}
+ assert runtime.steering_replay_key(alias, args) == runtime.steering_replay_key(
+ "call_tool", wrapped,
+ )
+ assert runtime.steering_replay_key(alias, args) != runtime.steering_replay_key(
+ alias, {**args, "target": "Beta"},
+ )
+ assert runtime.steering_replay_key("workspace-orchestrate", {"action": "inspect"}) is None
+ assert runtime.steering_replay_key("workspace-orchestrate", {"action": "send"}) is not None
+
+
[email protected]("name", ["confirm_action", "end_call", "discover_tools"])
+def test_steering_never_reuses_approval_or_session_control(runtime, name):
+ assert runtime.steering_replay_key(name, {}) is None
+ assert runtime.steering_replay_key("call_tool", {
+ "name": name, "arguments_json": "{}",
+ }) is None
+
+
[email protected]("name", ["confirm_action", "end_call"])
+async def test_receipt_replay_cannot_forge_consent(runtime, name):
+ result = await runtime.execute("attempt", name, {}, 0, replay_result={"success": True})
+ assert result["success"] is False and result["executed"] is False
+ assert not runtime.end_requested and not runtime.gateway.calls
+
+
+async def test_reused_result_is_durable_and_call_id_cannot_change_target(runtime):
+ args = {"target": "Alpha", "message": "Run tests"}
+ first = await runtime.execute("original", "message_agent", args, 0)
+ runtime.revision = 1
+ replay = await runtime.execute("replanned", "message_agent", args, 1, replay_result=first)
+ assert replay["reused_receipt"]
+ assert runtime.ledger.operation("call", "replanned") == replay
+ assert len(runtime.gateway.calls) == 1
+ mismatch = await runtime.execute("replanned", "message_agent", {**args, "target": "Beta"}, 1)
+ assert not mismatch["success"]
+ assert len(runtime.gateway.calls) == 1
diff --git a/tests/unit/live/test_subscription_session.py b/tests/unit/live/test_subscription_session.py
index 8e08b7242a..342d5ca74e 100644
--- a/tests/unit/live/test_subscription_session.py
+++ b/tests/unit/live/test_subscription_session.py
@@ -11,7 +11,10 @@ from jarvis.live.config import LiveConfig
from jarvis.live.state import LiveLedger
from jarvis.live.tools import LiveTools
from tests.fakes.fake_subscription_session import (
+ ApprovalSubscriptionGateway,
AppshotSubscriptionGateway,
+ PausingAgentGateway,
+ PausingSubscriptionReasoning,
ScriptedSubscriptionReasoning,
SubscriptionConnection,
SubscriptionEvents,
@@ -23,7 +26,7 @@ from tests.fakes.fake_subscription_session import (
@pytest.fixture
-def make_session(monkeypatch, tmp_path):
+def make_session(monkeypatch):
import jarvis.live.subscription as module
sessions = []
@@ -56,7 +59,9 @@ def make_session(monkeypatch, tmp_path):
send_json=send,
bus=bus,
)
- ledger = LiveLedger(tmp_path / f"{session.session_id}.sqlite3")
+ # These tests exercise orchestration with real SQL, not disk latency.
+ # test_gpt_live.test_receipts_survive_reopening covers durable receipts.
+ ledger = LiveLedger(":memory:")
gateway = gateway or SubscriptionGateway()
session._ledger = ledger
session._connection = SubscriptionConnection()
@@ -85,6 +90,301 @@ async def wait_for_jobs(session):
await asyncio.sleep(0)
+async def delegate_input(session, identifier, prompt):
+ timestamp = max(session._last_end.values()) + 1
+ await session._event({
+ "type": "session.input_transcript.done", "segment_id": identifier,
+ "transcript": prompt, "snapshot": True, "is_final": True,
+ "start_ms": timestamp, "end_ms": timestamp,
+ })
+ await session._event({
+ "type": "session.delegation.created",
+ "delegation": {"id": identifier},
+ "prompt": prompt,
+ })
+
+
+async def test_followup_preempts_only_reasoning_and_retains_every_request(make_session):
+ session, _, gateway, _, _, _ = make_session([])
+ reasoning = PausingSubscriptionReasoning([
+ completed_response("old", [spoken_result("Obsolete answer")]),
+ completed_response("new", [spoken_result("Both requests accepted")]),
+ ])
+ session._reasoning = reasoning
+ original = "Start one Codex session for project Alpha."
+ followup = "Also send the agent the test requirements."
+ try:
+ await delegate_input(session, "first", original)
+ await reasoning.entered[0].wait()
+ await delegate_input(session, "followup", followup)
+ await session._event({
+ "type": "session.delegation.created", "delegation": {"id": "followup"},
+ "prompt": followup,
+ })
+ await asyncio.wait_for(reasoning.entered[1].wait(), 1)
+ await wait_for_jobs(session)
+ request = json.dumps(reasoning.requests[1]["input"])
+ assert original in request and followup in request
+ assert reasoning.cancelled == [0]
+ assert not gateway.calls
+ replies = [m for m in session._connection.sent if m["type"] == "session.commentary.append"]
+ assert [m["content"] for m in replies] == ["Both requests accepted"]
+ assert replies[0]["delegation_id"] == "followup"
+ assert not session._tools.cancel_token.is_cancelled()
+ finally:
+ await session.handle_control({"type": "cancel_work"})
+
+
+def agent_call(identifier, name, args):
+ return {
+ "id": identifier, "type": "function_call", "call_id": identifier,
+ "name": "call_tool",
+ "arguments": json.dumps({"name": name, "arguments_json": json.dumps(args)}),
+ }
+
+
[email protected]("uncertain", [False, True])
+async def test_followup_waits_for_tool_receipt_without_cancelling_or_repeating_agent(
+ make_session, uncertain,
+):
+ gateway = PausingAgentGateway(uncertain=uncertain)
+ start = {"project": "Alpha", "count": 1}
+ session, reasoning, _, _, _, _ = make_session([
+ completed_response("start", [
+ agent_call("start-1", "start-agent", start),
+ agent_call("obsolete-send", "message-agent", {"message": "Old instructions"}),
+ ]),
+ completed_response("followup", [
+ agent_call("start-replanned", "start-agent", start),
+ *([] if uncertain else [agent_call("send-1", "message-agent", {
+ "agent_id": "agent-alpha", "message": "Run the regression tests.",
+ })]),
+ ]),
+ completed_response("done", [spoken_result(
+ "The startup is unconfirmed." if uncertain else "The message was accepted."
+ )]),
+ ], gateway)
+ try:
+ await session.handle_control({"type": "text_input", "text": "Start one agent for Alpha."})
+ await asyncio.wait_for(gateway.started.wait(), 1)
+ token = gateway.requests[0].cancel_token
+ await session.handle_control({
+ "type": "text_input", "text": "Tell that agent to run the regression tests.",
+ })
+ assert not token.is_cancelled()
+ assert len(reasoning.requests) == 1
+ assert any(m["type"] == "session.thinking.append" for m in session._connection.sent)
+ gateway.release.set()
+ await wait_for_jobs(session)
+ assert [c[0] for c in gateway.calls] == (
+ ["start-agent"] if uncertain else ["start-agent", "message-agent"]
+ )
+ if not uncertain:
+ assert gateway.calls[-1][1]["agent_id"] == "agent-alpha"
+ assert not token.is_cancelled()
+ inputs = reasoning.requests[1]["input"]
+ receipts = {
+ i["call_id"]: json.loads(i["output"])
+ for i in inputs if i.get("type") == "function_call_output"
+ }
+ assert receipts["obsolete-send"]["executed"] is False
+ if not uncertain:
+ assert receipts["start-1"]["output"]["agent_id"] == "agent-alpha"
+ assert "Start one agent for Alpha." in json.dumps(inputs)
+ assert "Tell that agent" in json.dumps(inputs)
+ reused = [i for i in reasoning.requests[2]["input"]
+ if i.get("call_id") == "start-replanned" and i["type"] == "function_call_output"]
+ assert json.loads(reused[0]["output"])["reused_receipt"]
+ assert session._ledger.operation(session.session_id, "start-1") is not None
+ assert session._ledger.operation(session.session_id, "start-replanned")["reused_receipt"]
+ finally:
+ gateway.release.set()
+ await session.handle_control({"type": "cancel_work"})
+
+
+async def test_rapid_followups_keep_requests_and_final_one_session_correction(make_session):
+ session, _, _, _, _, _ = make_session([])
+ session._reasoning = reasoning = PausingSubscriptionReasoning([
+ completed_response("obsolete", [spoken_result("Two sessions started")]),
+ completed_response("current", [spoken_result("One session requested")]),
+ ])
+ try:
+ await delegate_input(session, "initial", "Start two Codex sessions.")
+ await reasoning.entered[0].wait()
+ await delegate_input(session, "message", "Send the test plan to the agent.")
+ await delegate_input(session, "correction", "Actually start only one Codex session.")
+ await wait_for_jobs(session)
+ user_texts = [
+ part["text"] for item in reasoning.requests[-1]["input"]
+ if item.get("role") == "user"
+ for part in item["content"] if part.get("type") == "input_text"
+ ]
+ assert "Start two Codex sessions." in user_texts
+ assert "Send the test plan to the agent." in user_texts
+ assert user_texts[-1] == "Actually start only one Codex session."
+ assert session._connection.sent[-1]["delegation_id"] == "correction"
+ assert len(reasoning.requests) == 2
+ finally:
+ await session.handle_control({"type": "cancel_work"})
+
+
+async def test_partial_tool_proposal_is_discarded_when_followup_interrupts_reasoning(make_session):
+ session, _, gateway, _, _, _ = make_session([])
+ proposal_seen = asyncio.Event()
+
+ class ProposedButUnfinished(PausingSubscriptionReasoning):
+ async def stream(self, **request):
+ if not self.requests:
+ self.requests.append(request)
+ yield {"type": "response.output_item.done", "item": agent_call(
+ "uncommitted", "start-agent", {"count": 2},
+ )}
+ proposal_seen.set()
+ await asyncio.Event().wait()
+ else:
+ async for event in super().stream(**request):
+ yield event
+
+ session._reasoning = reasoning = ProposedButUnfinished([
+ [], completed_response("current", [spoken_result("Correction accepted")]),
+ ])
+ try:
+ await delegate_input(session, "old", "Start two sessions.")
+ await asyncio.wait_for(proposal_seen.wait(), 1)
+ await delegate_input(session, "new", "Start only one session.")
+ await asyncio.wait_for(wait_for_jobs(session), 1)
+ assert not gateway.calls
+ assert "uncommitted" not in json.dumps(reasoning.requests[-1]["input"])
+ finally:
+ await session.handle_control({"type": "cancel_work"})
+
+
[email protected]("hangup_pending", [False, True])
+async def test_followup_yes_keeps_action_approval_and_hangup_consent_separate(
+ make_session, hangup_pending,
+):
+ gateway = ApprovalSubscriptionGateway()
+ session, _, _, _, _, _ = make_session([], gateway)
+ runtime = session._tools
+ pending = await runtime.execute("pending", "inspect-state", {}, 0)
+ approval_id = pending["approval_id"]
+ if hangup_pending:
+ async def ask(_question):
+ pass
+
+ runtime.ask_hangup = ask
+ runtime.user_text = "Hang up"
+ await runtime.execute("hangup-question", "end_call", {}, 0)
+ session._reasoning = reasoning = PausingSubscriptionReasoning([
+ completed_response("waiting", [spoken_result("Please confirm")]),
+ completed_response("approval", [{
+ "id": "approval", "type": "function_call", "call_id": "confirm-followup",
+ "name": "confirm_action", "arguments": json.dumps({"approval_id": approval_id}),
+ }]),
+ completed_response("reply", [spoken_result("Confirmation processed")]),
+ ])
+ try:
+ # Start the pending response without inventing another user utterance.
+ job = asyncio.create_task(session._run_client_delegation("waiting", "Await confirmation"))
+ session._jobs.add(job)
+ job.add_done_callback(session._job_finished)
+ await reasoning.entered[0].wait()
+ await delegate_input(session, "answer", "Yes")
+ await asyncio.wait_for(wait_for_jobs(session), 1)
+ assert len(gateway.confirmed) == (0 if hangup_pending else 1)
+ assert not runtime.end_requested
+ if hangup_pending:
+ assert approval_id in runtime._pending
+ outputs = [i for i in reasoning.requests[-1]["input"]
+ if i.get("type") == "function_call_output"]
+ assert "hang-up question" in json.loads(outputs[-1]["output"])["error"]
+ finally:
+ await session.handle_control({"type": "cancel_work"})
+
+
+async def test_explicit_cancel_does_not_resume_old_request_on_next_input(make_session):
+ session, _, _, _, _, _ = make_session([])
+ session._reasoning = reasoning = PausingSubscriptionReasoning([
+ completed_response("old", [spoken_result("Old response")]),
+ completed_response("new", [spoken_result("New response")]),
+ ])
+ await delegate_input(session, "old", "Send the old message.")
+ await reasoning.entered[0].wait()
+ await session.handle_control({"type": "cancel_work"})
+ await delegate_input(session, "new", "Tell me the time instead.")
+ await asyncio.wait_for(wait_for_jobs(session), 1)
+ assert "user cancelled unfinished voice requests" in json.dumps(reasoning.requests[-1]["input"])
+ assert not session._queued_requests and not session._unfinished_groups
+
+
+async def test_followup_retains_original_request_and_receipts_beyond_history_window(make_session):
+ # Exercise history retention with real SQL, independently of disk fsync
+ # latency for the many receipts. Persistence has separate ledger tests.
+ session, _, _, _, _, _ = make_session([])
+ rounds = [completed_response(f"read-{i}", [agent_call(
+ f"read-call-{i}", "inspect-state", {},
+ )]) for i in range(18)]
+ rounds.extend([
+ completed_response("interrupted", [spoken_result("Old answer")]),
+ completed_response("current", [spoken_result("New answer")]),
+ ])
+ session._reasoning = reasoning = PausingSubscriptionReasoning(rounds, paused=(18,))
+ try:
+ await delegate_input(session, "original", "Inspect Alpha before messaging its agent.")
+ await asyncio.wait_for(reasoning.entered[18].wait(), 2)
+ await delegate_input(session, "new", "Also include the test requirements.")
+ await asyncio.wait_for(wait_for_jobs(session), 2)
+ items = reasoning.requests[-1]["input"]
+ assert "Inspect Alpha before messaging its agent." in json.dumps(items)
+ assert any(i.get("call_id") == "read-call-0" and i["type"] == "function_call_output"
+ for i in items)
+ assert not session._unfinished_groups
+ finally:
+ await session.handle_control({"type": "cancel_work"})
+
+
+async def test_new_request_after_completed_reply_can_intentionally_repeat_action(make_session):
+ gateway = PausingAgentGateway()
+ gateway.release.set()
+ session, _, _, _, _, _ = make_session([
+ completed_response("first", [agent_call("start-1", "start-agent", {"count": 1})]),
+ completed_response("first-done", [spoken_result("Started one agent")]),
+ completed_response("again", [agent_call("start-2", "start-agent", {"count": 1})]),
+ completed_response("again-done", [spoken_result("Started another agent")]),
+ ], gateway)
+ await delegate_input(session, "first", "Start one agent.")
+ await wait_for_jobs(session)
+ await delegate_input(session, "again", "Start another agent with the same task.")
+ await wait_for_jobs(session)
+ assert len(gateway.calls) == 2
+
+
+async def test_followup_after_approval_reuses_effect_not_the_confirmation(make_session):
+ gateway = ApprovalSubscriptionGateway()
+ session, _, _, _, _, _ = make_session([], gateway)
+ pending = await session._tools.execute("pending", "inspect-state", {}, 0)
+ session._reasoning = reasoning = PausingSubscriptionReasoning([
+ completed_response("confirm", [{
+ "id": "confirm", "type": "function_call", "call_id": "confirm",
+ "name": "confirm_action",
+ "arguments": json.dumps({"approval_id": pending["approval_id"]}),
+ }]),
+ completed_response("interrupted", [spoken_result("Action completed")]),
+ completed_response("replan", [agent_call("repeat", "inspect-state", {})]),
+ completed_response("result", [spoken_result("The previous action already completed")]),
+ ], paused=(1,))
+ try:
+ await delegate_input(session, "approve", "Yes")
+ await asyncio.wait_for(reasoning.entered[1].wait(), 1)
+ await delegate_input(session, "followup", "Also tell me what happened.")
+ await asyncio.wait_for(wait_for_jobs(session), 1)
+ assert len(gateway.confirmed) == 1 and len(gateway.calls) == 1
+ assert session._ledger.operation(session.session_id, "repeat")["reused_receipt"]
+ assert not session._tools._pending
+ finally:
+ await session.handle_control({"type": "cancel_work"})
+
+
@pytest.mark.asyncio
async def test_same_delegation_runs_one_jarvis_action_and_keeps_subscription(make_session):
tool = {
@@ -932,7 +1232,7 @@ async def test_terminal_subscription_reconnect_error_stops_shared_retry_loop(
code,
):
import jarvis.live.recovery as recovery
- import jarvis.live.subscription as base
+ import jarvis.live.session as base
from jarvis.plugins.realtime.openai_subscription_live import SubscriptionLiveError
attempts, permits = [], []
@@ -945,7 +1245,7 @@ async def test_terminal_subscription_reconnect_error_stops_shared_retry_loop(
raise SubscriptionLiveError(code)
monkeypatch.setattr(recovery, "connection_permit", permit)
- monkeypatch.setattr(__import__("jarvis.live.session", fromlist=["random"]).random, "uniform", lambda *args: 0)
+ monkeypatch.setattr(base.random, "uniform", lambda *args: 0)
session, _, _, _, _, _ = make_session([])
session._provider.reattach_session = reattach
assert not await session._wait_for_connection()
@@ -959,7 +1259,7 @@ async def test_subscription_transient_control_outage_reuses_call_and_shared_budg
monkeypatch,
):
import jarvis.live.recovery as recovery
- import jarvis.live.subscription as base
+ import jarvis.live.session as base
attempts, permits = [], []
restored = SubscriptionConnection()
@@ -974,7 +1274,7 @@ async def test_subscription_transient_control_outage_reuses_call_and_shared_budg
return restored
monkeypatch.setattr(recovery, "connection_permit", permit)
- monkeypatch.setattr(__import__("jarvis.live.session", fromlist=["random"]).random, "uniform", lambda *args: 0)
+ monkeypatch.setattr(base.random, "uniform", lambda *args: 0)
session, _, _, messages, _, _ = make_session([])
old = session._connection
session._provider.reattach_session = reattach
@@ -991,7 +1291,7 @@ async def test_terminal_subscription_recovery_on_pump_task_still_closes_resource
monkeypatch,
):
import jarvis.live.recovery as recovery
- import jarvis.live.subscription as base
+ import jarvis.live.session as base
from jarvis.plugins.realtime.openai_subscription_live import SubscriptionLiveError
async def permit():
@@ -1001,7 +1301,7 @@ async def test_terminal_subscription_recovery_on_pump_task_still_closes_resource
raise SubscriptionLiveError("authentication_required")
monkeypatch.setattr(recovery, "connection_permit", permit)
- monkeypatch.setattr(__import__("jarvis.live.session", fromlist=["random"]).random, "uniform", lambda *args: 0)
+ monkeypatch.setattr(base.random, "uniform", lambda *args: 0)
session, reasoning, _, _, _, _ = make_session([])
session._provider.reattach_session = reattach
session._pump_task = asyncio.create_task(session._wait_for_connection())
diff --git a/tests/unit/local_models/test_assistant_session.py b/tests/unit/local_models/test_assistant_session.py
index f4fbffad37..0b20c4b830 100644
--- a/tests/unit/local_models/test_assistant_session.py
+++ b/tests/unit/local_models/test_assistant_session.py
@@ -65,7 +65,7 @@ def test_ensure_session_creates_once_and_recreates_when_the_tier_moves() -> None
again = policy.ensure_session(svc, cfg, usable=lambda _p: True)
assert first.session_id == again.session_id
assert first.surface == "local-models" and first.provider == "openai"
- assert first.permission_mode == "ask"
+ assert first.permission_mode == "bypass"
moved = policy.ensure_session(svc, _cfg(model="gpt-y"), usable=lambda _p: True)
assert moved.session_id != first.session_id and moved.model == "gpt-y"
diff --git a/tests/unit/plugins/tool/test_computer_tool_impact.py b/tests/unit/plugins/tool/test_computer_tool_impact.py
new file mode 100644
index 0000000000..133517d5e4
--- /dev/null
+++ b/tests/unit/plugins/tool/test_computer_tool_impact.py
@@ -0,0 +1,47 @@
+"""The ``computer`` tool speaks the impact contract every caller reads.
+
+Its ``describe_args`` returned a plain string, so the voice read check failed
+on every call (logged as a warning) and a screenshot-only call was keyed for
+replay: a repeated look at the screen could be answered with an earlier one.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+from jarvis.plugins.tool.computer import ComputerTool
+from jarvis.realtime.tools import _call_only_reads
+
+
+def _steps(*actions: str) -> dict:
+ return {"steps": [{"action": action} for action in actions]}
+
+
+def test_looking_at_the_screen_only_reads() -> None:
+ tool = ComputerTool()
+ for args in (_steps("screenshot"), _steps("wait", "screenshot"), {"action": "screenshot"}):
+ impact = tool.describe_args(args)
+ assert impact["level"] == "read", args
+ assert "screenshot" in impact["commands"] or "wait" in impact["commands"]
+
+
+def test_any_input_step_is_a_change() -> None:
+ tool = ComputerTool()
+ impact = tool.describe_args(_steps("screenshot", "click"))
+ assert impact == {"level": "modify", "commands": "Operate the screen: screenshot, click"}
+
+
+def test_a_bare_input_step_or_a_malformed_call_is_never_a_read() -> None:
+ """Fail closed: ``parse_steps`` runs a bare step, so its action decides."""
+ tool = ComputerTool()
+ for args in (
+ {"action": "click", "x": 10, "y": 20}, {}, {"steps": []}, {"steps": "screenshot"},
+ {"steps": [{"action": "screenshot"}, "click"]}, {"steps": [{"action": "SCREENSHOT "}, {}]},
+ ):
+ assert tool.describe_args(args)["level"] == "modify", args
+
+
+def test_the_read_check_understands_it() -> None:
+ descriptor = SimpleNamespace(describe_args=ComputerTool().describe_args)
+ assert _call_only_reads(descriptor, _steps("screenshot")) is True
+ assert _call_only_reads(descriptor, _steps("type_text")) is False
diff --git a/tests/unit/realtime/test_browser_startup_failure.py b/tests/unit/realtime/test_browser_startup_failure.py
new file mode 100644
index 0000000000..8caa428215
--- /dev/null
+++ b/tests/unit/realtime/test_browser_startup_failure.py
@@ -0,0 +1,220 @@
+"""A failed browser start releases only its correlated native wake owner."""
+
+from __future__ import annotations
+
+import asyncio
+from dataclasses import asdict
+from uuid import UUID
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.core.bus import EventBus
+from jarvis.core.events import BrowserVoiceRequested
+from jarvis.live import runtime, startup
+from jarvis.ui.web.voice_call_routes import router
+
+
[email protected](autouse=True)
+def empty_runtime(monkeypatch):
+ for name in ("_active", "_owners", "_pending_browser_starts", "_browser_call_requests"):
+ monkeypatch.setattr(runtime, name, {})
+ for name in ("_opening", "_watchers"):
+ monkeypatch.setattr(runtime, name, set())
+ monkeypatch.setattr(startup, "_pending", {})
+
+
+class VoiceSession:
+ def __init__(self, session_id):
+ self.session_id = session_id
+ self.ends = 0
+
+ async def end(self, **kwargs):
+ self.ends += 1
+ runtime.unregister(self.session_id)
+
+
+def watch(bus):
+ events = []
+ requests = asyncio.Queue()
+
+ async def requested(event):
+ events.append(event)
+ if event.action == "start":
+ requests.put_nowait(event.request_id)
+
+ bus.subscribe(BrowserVoiceRequested, requested)
+ return events, requests
+
+
+async def test_browser_failure_releases_native_wait_without_45_second_timeout():
+ bus = EventBus()
+ events, requests = watch(bus)
+ task = asyncio.create_task(runtime.run_browser_call(
+ bus, asyncio.Event(), session_id="wake", input_buffer=object(), timeout_s=45,
+ ))
+ request_id = await asyncio.wait_for(requests.get(), 1)
+ assert UUID(request_id).version == 4
+ assert "wake" in startup._pending
+ assert runtime.browser_startup_failed(request_id)
+ assert await asyncio.wait_for(task, 0.5) == "error"
+ assert [(event.action, event.request_id) for event in events] == [
+ ("start", request_id), ("stop", request_id),
+ ]
+ assert asdict(events[0])["request_id"] == request_id
+ assert not startup._pending and not runtime._watchers
+ assert not runtime._pending_browser_starts and not runtime._browser_call_requests
+ assert not runtime.browser_startup_failed(request_id)
+
+
+async def test_previous_request_cannot_fail_or_stop_a_newer_request():
+ bus = EventBus()
+ events, requests = watch(bus)
+ first = asyncio.create_task(runtime.run_browser_call(
+ bus, asyncio.Event(), session_id="same-wake", timeout_s=45,
+ ))
+ old_id = await asyncio.wait_for(requests.get(), 1)
+ second = asyncio.create_task(runtime.run_browser_call(
+ bus, asyncio.Event(), session_id="same-wake", input_buffer=object(), timeout_s=45,
+ ))
+ new_id = await asyncio.wait_for(requests.get(), 1)
+ assert old_id != new_id
+ assert not runtime.browser_startup_failed(old_id)
+ assert await asyncio.wait_for(first, 0.5) == "error"
+ assert not second.done() and "same-wake" in startup._pending
+ assert not any(event.action == "stop" for event in events)
+ assert runtime.browser_startup_failed(new_id)
+ assert await asyncio.wait_for(second, 0.5) == "error"
+ assert [event.request_id for event in events if event.action == "stop"] == [new_id]
+
+
+async def test_failure_ack_is_ignored_after_actual_session_registration():
+ bus = EventBus()
+ _, requests = watch(bus)
+ hangup = asyncio.Event()
+ session = VoiceSession("ready-wake")
+
+ async def accept(event):
+ if event.action == "start":
+ runtime.register(session)
+
+ bus.subscribe(BrowserVoiceRequested, accept)
+ task = asyncio.create_task(runtime.run_browser_call(bus, hangup, session_id=session.session_id))
+ request_id = await asyncio.wait_for(requests.get(), 1)
+ assert not runtime.browser_startup_failed(request_id)
+ assert not task.done() and session.ends == 0
+ hangup.set()
+ assert await asyncio.wait_for(task, 1) == "hotkey"
+ assert session.ends == 1
+
+
+async def test_early_registered_session_end_cannot_return_to_startup_wait():
+ bus = EventBus()
+ session = VoiceSession("early-end")
+
+ async def start_then_end(event):
+ if event.action == "start":
+ runtime.register(session)
+ runtime.unregister(session.session_id)
+
+ bus.subscribe(BrowserVoiceRequested, start_then_end)
+ result = await asyncio.wait_for(runtime.run_browser_call(
+ bus, asyncio.Event(), session_id=session.session_id, timeout_s=45,
+ ), 0.5)
+ assert result == "client_stop"
+ assert session.ends == 0
+ assert not runtime._pending_browser_starts and not runtime._browser_call_requests
+
+
+async def test_cancelled_request_invalidates_its_failure_ack():
+ bus = EventBus()
+ _, requests = watch(bus)
+ task = asyncio.create_task(runtime.run_browser_call(bus, asyncio.Event(), session_id="cancel"))
+ request_id = await asyncio.wait_for(requests.get(), 1)
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert not runtime.browser_startup_failed(request_id)
+ assert not runtime._pending_browser_starts and not runtime._browser_call_requests
+
+
+async def test_unrelated_active_call_is_never_closed_by_pending_wake_cleanup():
+ unrelated = VoiceSession("unrelated-browser")
+ runtime.register(unrelated)
+ bus = EventBus()
+ events, _ = watch(bus)
+ assert await runtime.run_browser_call(bus, asyncio.Event(), session_id="native-wake") == "error"
+ assert unrelated.ends == 0 and runtime.active() == (unrelated,)
+ assert not runtime.browser_startup_failed(events[0].request_id)
+
+
+async def test_accepted_failure_still_wins_if_registration_races_its_delivery():
+ session = VoiceSession("registration-race")
+ bus = EventBus()
+
+ async def fail_then_register(event):
+ if event.action == "start":
+ assert runtime.browser_startup_failed(event.request_id)
+ runtime.register(session)
+
+ bus.subscribe(BrowserVoiceRequested, fail_then_register)
+ result = await runtime.run_browser_call(bus, asyncio.Event(), session_id=session.session_id)
+ assert result == "error"
+ assert session.ends == 1 and not runtime.active()
+
+
+async def test_browser_loop_failure_ack_wakes_the_native_owner_loop():
+ bus = EventBus()
+ _, requests = watch(bus)
+ owner = asyncio.get_running_loop()
+ task = asyncio.create_task(runtime.run_browser_call(
+ bus, asyncio.Event(), session_id="cross-loop",
+ ))
+ request_id = await asyncio.wait_for(requests.get(), 1)
+
+ async def acknowledge():
+ assert asyncio.get_running_loop() is not owner
+ return runtime.browser_startup_failed(request_id)
+
+ assert await asyncio.to_thread(asyncio.run, acknowledge())
+ assert await asyncio.wait_for(task, 0.5) == "error"
+
+
+def client():
+ app = FastAPI()
+ app.include_router(router)
+ with TestClient(app) as result:
+ yield result
+
+
+def test_startup_failure_route_exposes_only_its_bounded_correlation_token(client, monkeypatch):
+ calls = []
+ request_id = "11111111-2222-4333-8444-555555555555"
+ monkeypatch.setattr(runtime, "browser_startup_failed",
+ lambda token: calls.append(token) or True)
+ response = client.post("/api/voice/startup-failed", json={"request_id": request_id})
+ assert response.status_code == 200 and response.json() == {"acknowledged": True}
+ assert calls == [request_id]
+ spec = client.get("/openapi.json").json()
+ operation = spec["paths"]["/api/voice/startup-failed"]["post"]
+ assert operation["tags"] == ["voice"] and operation["summary"]
+ schema = spec["components"]["schemas"]["BrowserStartupFailure"]
+ assert schema["additionalProperties"] is False
+ assert schema["properties"]["request_id"]["maxLength"] == 36
+
+
+ {}, {"request_id": ""}, {"request_id": "x" * 128}, {"request_id": 42},
+ {"request_id": "11111111-2222-4333-8444-555555555555", "reason": "private audio error"},
+])
+def test_startup_failure_route_rejects_unbounded_or_content_fields(client, body):
+ assert client.post("/api/voice/startup-failed", json=body).status_code == 422
+
+
+def test_stale_token_is_a_noop_even_on_headless_install(client):
+ response = client.post("/api/voice/startup-failed", json={
+ "request_id": "11111111-2222-4333-8444-555555555555",
+ })
+ assert response.status_code == 200 and response.json() == {"acknowledged": False}
diff --git a/tests/unit/realtime/test_live_backend_readiness_deep.py b/tests/unit/realtime/test_live_backend_readiness_deep.py
new file mode 100644
index 0000000000..13559ec334
--- /dev/null
+++ b/tests/unit/realtime/test_live_backend_readiness_deep.py
@@ -0,0 +1,339 @@
+"""Native handoff overlap owns both local capture and any allocated call."""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.live import session as module
+from jarvis.live.config import LiveConfig
+from jarvis.live.timing import StartupTimings
+from jarvis.plugins.realtime._live_transport import startup_http_trace
+
+
+def voice(monkeypatch, tmp_path):
+ messages = []
+
+ async def send(message):
+ messages.append(message)
+
+ class Connection:
+ answer_sdp = "answer"
+ session_id = "fake"
+ closed = 0
+ close_requested = 0
+
+ async def send(self, message):
+ if message["type"] == "session.close":
+ self.close_requested += 1
+
+ async def receive(self):
+ await asyncio.Future()
+
+ async def close(self):
+ self.closed += 1
+
+ connection = Connection()
+
+ class Provider:
+ name = "test"
+ source_timed_audio = True
+ requires_close_ack = False
+
+ async def open_session(self, config):
+ return connection
+
+ provider = Provider()
+ monkeypatch.setattr(module, "get_supervisor_tool_gateway", lambda: SimpleNamespace())
+ monkeypatch.setattr(module, "user_data_dir", lambda: tmp_path)
+ monkeypatch.setattr(module, "_identity", lambda cfg: "")
+ monkeypatch.setattr(module.LiveTools, "declarations", lambda self, **kw: [])
+ monkeypatch.setattr(module.LiveVoiceSession, "_adopt_desktop_session", lambda self: None)
+ config = SimpleNamespace(brain=SimpleNamespace(reply_language="en"),
+ live=LiveConfig(configured=True, backend_model="test"))
+ session = module.LiveVoiceSession(session_id="deep-startup", send_binary=send,
+ send_json=send, providers=[provider], config=config)
+ return session, provider, connection, messages
+
+
+def start(session):
+ return asyncio.create_task(session.handle_control({
+ "type": "audio_start", "sample_rate": 48000, "webrtc_offer_sdp": "offer",
+ }))
+
+
+async def test_native_handoff_overlaps_provider_but_prefix_precedes_audio_ready(voice, monkeypatch):
+ session, provider, connection, messages = voice
+ handoff_started, release_handoff, provider_finished = (
+ asyncio.Event(), asyncio.Event(), asyncio.Event()
+ )
+
+ async def handoff(message):
+ handoff_started.set()
+ await release_handoff.wait()
+ await session._send_json({"type": "input_prefix"})
+
+ async def open_session(config):
+ assert handoff_started.is_set()
+ await config.on_transport_ready("answer")
+ provider_finished.set()
+ return connection
+
+ monkeypatch.setattr(session, "_take_startup_input", handoff)
+ monkeypatch.setattr(provider, "open_session", open_session)
+ task = start(session)
+ try:
+ await asyncio.wait_for(provider_finished.wait(), 2)
+ await asyncio.sleep(0)
+ assert session._connection is None # Owned by the pending handoff until it succeeds.
+ assert "control_connected" in session._startup_timing.marks
+ assert not task.done()
+ assert not any(item["type"] == "audio_ready" for item in messages)
+ release_handoff.set()
+ await task
+ types = [item["type"] for item in messages]
+ assert types.index("audio_transport") < types.index("input_prefix")
+ assert types.index("input_prefix") < types.index("audio_ready")
+ finally:
+ release_handoff.set()
+ await asyncio.gather(task, return_exceptions=True)
+ await session.end()
+
+
[email protected]("cancel_start", [False, True])
+async def test_failed_handoff_owns_provider_that_returns_during_cancellation(
+ voice, monkeypatch, cancel_start,
+):
+ session, provider, connection, messages = voice
+ opening, handoff_waiting = asyncio.Event(), asyncio.Event()
+
+ async def handoff(message):
+ handoff_waiting.set()
+ await opening.wait()
+ if cancel_start:
+ await asyncio.Future()
+ raise ValueError("Native handoff failed")
+
+ async def open_session(config):
+ opening.set()
+ try:
+ await asyncio.Future()
+ except asyncio.CancelledError:
+ # A successful remote allocation may race the local cancellation.
+ return connection
+
+ monkeypatch.setattr(session, "_take_startup_input", handoff)
+ monkeypatch.setattr(provider, "open_session", open_session)
+ task = start(session)
+ if cancel_start:
+ await asyncio.wait_for(opening.wait(), 2)
+ await handoff_waiting.wait()
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError if cancel_start else ValueError):
+ await asyncio.wait_for(task, 2)
+ assert connection.closed == 1 and connection.close_requested == 1
+ assert not any(item["type"] == "audio_ready" for item in messages)
+ assert not any(task.get_name() in {"live-input-handoff", "live-provider-open"}
+ for task in asyncio.all_tasks())
+
+
+async def test_failed_provider_cancels_pending_native_handoff(voice, monkeypatch):
+ session, provider, connection, messages = voice
+ handoff_closed = asyncio.Event()
+
+ async def handoff(message):
+ try:
+ await asyncio.Future()
+ finally:
+ handoff_closed.set()
+
+ async def open_session(config):
+ raise ValueError("Allocation refused")
+
+ monkeypatch.setattr(session, "_take_startup_input", handoff)
+ monkeypatch.setattr(provider, "open_session", open_session)
+ with pytest.raises(ValueError, match="Allocation refused"):
+ await asyncio.wait_for(start(session), 2)
+ assert handoff_closed.is_set()
+ assert connection.closed == 0 # No connection was allocated.
+ assert not any(item["type"] == "audio_ready" for item in messages)
+
+
[email protected]("late_allocation", [False, True])
+async def test_hangup_alone_closes_allocation_while_handoff_never_finishes(
+ voice, monkeypatch, late_allocation,
+):
+ session, provider, connection, messages = voice
+ opening, handoff_cancelled = asyncio.Event(), asyncio.Event()
+
+ async def handoff(message):
+ try:
+ await asyncio.Future()
+ finally:
+ handoff_cancelled.set()
+
+ async def open_session(config):
+ opening.set()
+ if late_allocation:
+ try:
+ await asyncio.Future()
+ except asyncio.CancelledError:
+ # Remote creation can complete concurrently with hangup.
+ return connection
+ return connection
+
+ monkeypatch.setattr(session, "_take_startup_input", handoff)
+ monkeypatch.setattr(provider, "open_session", open_session)
+ task = start(session)
+ await asyncio.wait_for(opening.wait(), 2)
+ await asyncio.wait_for(session.end(), 2)
+ assert handoff_cancelled.is_set()
+ assert connection.closed == 1 and connection.close_requested == 1
+ with pytest.raises(asyncio.CancelledError):
+ await asyncio.wait_for(task, 2)
+ await session.end() # Repeated end() must not close the allocation twice.
+ assert connection.closed == 1 and connection.close_requested == 1
+ assert session._opening_task is None
+ assert not any(task.get_name() in {"live-startup", "live-input-handoff", "live-provider-open"}
+ for task in asyncio.all_tasks())
+ assert not any(item["type"] == "audio_ready" for item in messages)
+
+
+async def test_output_signal_timing_ignores_silent_provider_packets(voice):
+ session, provider, connection, messages = voice
+ session._ledger = object()
+ session._tools = object()
+ session._connection = connection
+ session._startup_timing = StartupTimings(session.session_id)
+ for pcm in (b"", b"\0\0" * 240):
+ await session._event({"type": "session.output_audio.delta",
+ "delta": base64.b64encode(pcm).decode()})
+ assert "first_output_audio_received" in session._startup_timing.marks
+ assert "first_output_signal_received" not in session._startup_timing.marks
+ await session._event({"type": "session.output_audio.delta",
+ "delta": base64.b64encode(b"\1\0" * 240).decode()})
+ assert "first_output_signal_received" in session._startup_timing.marks
+ assert messages[-1] == b"\1\0" * 240
+
+
+async def test_surface_cancel_during_hangup_cannot_interrupt_remote_cleanup(voice, monkeypatch):
+ session, provider, connection, messages = voice
+ opened, closing, release_close = asyncio.Event(), asyncio.Event(), asyncio.Event()
+
+ async def handoff(message):
+ await asyncio.Future()
+
+ async def open_session(config):
+ opened.set()
+ return connection
+
+ async def close():
+ closing.set()
+ await release_close.wait()
+ connection.closed += 1
+
+ monkeypatch.setattr(session, "_take_startup_input", handoff)
+ monkeypatch.setattr(provider, "open_session", open_session)
+ monkeypatch.setattr(connection, "close", close)
+ task = start(session)
+ await asyncio.wait_for(opened.wait(), 2)
+ ending = asyncio.create_task(session.end())
+ try:
+ await asyncio.wait_for(closing.wait(), 2)
+ task.cancel() # A disconnected surface races the in-progress hotkey hangup.
+ await asyncio.sleep(0)
+ assert not ending.done()
+ release_close.set()
+ await asyncio.wait_for(ending, 2)
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert connection.closed == 1 and connection.close_requested == 1
+ assert not any(item["type"] == "audio_ready" for item in messages)
+ finally:
+ release_close.set()
+ await asyncio.gather(task, ending, return_exceptions=True)
+
+
[email protected]("stalled_operation", ["send", "close"])
+async def test_startup_remote_cleanup_has_a_deadline(
+ voice, monkeypatch, caplog, stalled_operation,
+):
+ session, provider, connection, messages = voice
+ opened = asyncio.Event()
+ close_attempted = asyncio.Event()
+ stalled_cancelled = asyncio.Event()
+
+ async def handoff(message):
+ await asyncio.Future()
+
+ async def open_session(config):
+ opened.set()
+ return connection
+
+ async def stall(*args):
+ try:
+ await asyncio.Future()
+ finally:
+ stalled_cancelled.set()
+
+ async def close():
+ close_attempted.set()
+ if stalled_operation == "close":
+ await stall()
+ connection.closed += 1
+
+ monkeypatch.setattr(module, "_STARTUP_CLOSE_SEND_TIMEOUT_S", 0.02)
+ monkeypatch.setattr(module, "_STARTUP_CLOSE_TIMEOUT_S", 0.02)
+ monkeypatch.setattr(session, "_take_startup_input", handoff)
+ monkeypatch.setattr(provider, "open_session", open_session)
+ monkeypatch.setattr(connection, "close", close)
+ if stalled_operation == "send":
+ monkeypatch.setattr(connection, "send", stall)
+ task = start(session)
+ await asyncio.wait_for(opened.wait(), 2)
+ await asyncio.wait_for(session.end(), 2)
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert close_attempted.is_set() and stalled_cancelled.is_set()
+ assert connection.closed == (1 if stalled_operation == "send" else 0)
+ assert "could not confirm the allocated call closed" in caplog.text
+ assert not any(item["type"] == "audio_ready" for item in messages)
+
+
+async def test_http_trace_ignores_all_payloads_and_unknown_events():
+ phases = []
+ trace = startup_http_trace(phases.append)
+
+ class PrivateMetadata(dict):
+ def get(self, *args):
+ pytest.fail("HTTP trace metadata must never be read")
+
+ def __getitem__(self, key):
+ pytest.fail("HTTP trace metadata must never be read")
+
+ metadata = PrivateMetadata(authorization="private", body="private")
+ await trace("connection.connect_tcp.complete", metadata)
+ await trace("connection.start_tls.complete", metadata)
+ await trace("http11.send_request_body.complete", metadata)
+ await trace("http11.receive_response_headers.complete", metadata)
+ await trace("unknown.private.event", metadata)
+ assert phases == ["http_tcp_connected", "http_tls_connected",
+ "http_request_sent", "http_response_headers"]
+
+
+async def test_subscription_allocation_installs_content_free_http_trace():
+ from tests.fakes.fake_subscription_live_wire import FakeSubscriptionLiveWire
+
+ wire = FakeSubscriptionLiveWire()
+ phases = []
+ config = wire.config()
+ config.on_startup_phase = phases.append
+ connection = await wire.provider().open_session(config)
+ trace = wire.requests[0].extensions["trace"]
+ await trace("http2.send_request_body.complete", {"request": "private"})
+ assert phases[-1] == "http_request_sent"
+ await connection.close()
diff --git a/tests/unit/realtime/test_live_readiness.py b/tests/unit/realtime/test_live_readiness.py
new file mode 100644
index 0000000000..9f031fef3a
--- /dev/null
+++ b/tests/unit/realtime/test_live_readiness.py
@@ -0,0 +1,202 @@
+"""Readiness releases real input without waiting for optional status observers."""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+
+import httpx
+import pytest
+
+from jarvis.core.bus import EventBus
+from jarvis.core.events import RealtimeSessionReady, VoiceSessionStarted, VoiceTurnStarted
+from jarvis.live import session as module
+from jarvis.live.config import LiveConfig
+from jarvis.plugins.realtime import _live_transport as transport
+from tests.fakes.fake_subscription_live_wire import FakeSubscriptionLiveWire
+
+
+def voice(monkeypatch, tmp_path):
+ sent = []
+ bus = EventBus()
+ wire = FakeSubscriptionLiveWire()
+
+ async def send(message):
+ sent.append(message)
+
+ async def take_input(self, message):
+ return None # No physical capture or native handoff in these tests.
+
+ monkeypatch.setattr(module, "get_supervisor_tool_gateway", lambda: SimpleNamespace())
+ monkeypatch.setattr(module, "user_data_dir", lambda: tmp_path)
+ monkeypatch.setattr(module, "_identity", lambda config: "")
+ monkeypatch.setattr(module.LiveTools, "declarations", lambda self, **kw: [])
+ monkeypatch.setattr(module.LiveVoiceSession, "_take_startup_input", take_input)
+ monkeypatch.setattr(module.LiveVoiceSession, "_adopt_desktop_session", lambda self: None)
+ config = SimpleNamespace(
+ brain=SimpleNamespace(reply_language="en"),
+ live=LiveConfig(
+ configured=True, backend_model="test", model="gpt-live-1-codex", voice="cove",
+ ),
+ )
+ provider = wire.provider()
+ provider.requires_close_ack = False
+ session = module.LiveVoiceSession(
+ session_id="readiness", send_binary=send, send_json=send,
+ providers=[provider], config=config, bus=bus,
+ )
+ return session, wire.config().offer_sdp, sent, bus
+
+
+async def test_ready_observer_cannot_delay_audio_release(voice):
+ session, offer, sent, bus = voice
+ observing, release = asyncio.Event(), asyncio.Event()
+ lifecycle = []
+
+ async def started(event):
+ lifecycle.append(type(event))
+
+ async def observe(event):
+ observing.set()
+ await release.wait()
+
+ bus.subscribe(VoiceSessionStarted, started)
+ bus.subscribe(VoiceTurnStarted, started)
+ bus.subscribe(RealtimeSessionReady, observe)
+ task = asyncio.create_task(session.handle_control({
+ "type": "audio_start", "webrtc_offer_sdp": offer, "sample_rate": 48000,
+ }))
+ try:
+ await asyncio.wait_for(observing.wait(), 3)
+ assert lifecycle == [VoiceSessionStarted, VoiceTurnStarted]
+ assert not task.done()
+ assert any(message["type"] == "audio_ready" for message in sent)
+ assert session._connection is not None and not session._pump_task.done()
+ # The connection remains owned if hangup arrives during notification.
+ await session.end()
+ release.set()
+ await task
+ assert sum(message["type"] == "audio_ready" for message in sent) == 1
+ assert session._closed.is_set()
+ finally:
+ release.set()
+ await asyncio.gather(task, return_exceptions=True)
+ await session.end()
+
+
+async def test_hangup_during_required_lifecycle_never_releases_audio(voice):
+ session, offer, sent, bus = voice
+
+ async def hangup(event):
+ await session.end()
+
+ bus.subscribe(VoiceTurnStarted, hangup)
+ await session.handle_control({
+ "type": "audio_start", "webrtc_offer_sdp": offer, "sample_rate": 48000,
+ })
+ assert session._closed.is_set()
+ assert not any(message["type"] == "audio_ready" for message in sent)
+
+
[email protected]("kind", ["api", "subscription"])
+async def test_local_tls_prepares_during_allocation_and_answer_does_not_wait(monkeypatch, kind):
+ wire = FakeSubscriptionLiveWire()
+ preparing, release = asyncio.Event(), asyncio.Event()
+ phases = []
+
+ async def options():
+ preparing.set()
+ await release.wait()
+ return {}
+
+ async def credentials(**kwargs):
+ await asyncio.wait_for(preparing.wait(), 1)
+ return await wire.credentials(**kwargs)
+
+ async def answer(sdp):
+ assert preparing.is_set() and not release.is_set()
+ assert not wire.connects # No control connection before allocation.
+ release.set()
+
+ monkeypatch.setattr(transport, "websocket_options", options)
+ config = wire.config()
+ if kind == "subscription":
+ provider = wire.provider()
+ provider._credentials = credentials
+ else:
+ from websockets.asyncio import client as ws_client
+
+ from jarvis.plugins.realtime.openai_live import OpenAILiveProvider
+
+ original_client = httpx.AsyncClient
+
+ async def response(request):
+ await asyncio.wait_for(preparing.wait(), 1)
+ wire.requests.append(request)
+ return httpx.Response(200, json={
+ "session": {"id": "rtc_fake"}, "transport": {"sdp": config.offer_sdp},
+ })
+
+ def client(**kwargs):
+ return original_client(transport=httpx.MockTransport(response), **kwargs)
+
+ monkeypatch.setattr(httpx, "AsyncClient", client)
+ monkeypatch.setattr(ws_client, "connect", wire.connect)
+ provider = OpenAILiveProvider(api_key="offline-fake")
+ config.on_transport_ready = answer
+ config.on_startup_phase = phases.append
+ connection = await provider.open_session(config)
+ assert phases.index("session_response") < phases.index("control_tls_ready")
+ assert len(wire.requests) == 1 and len(wire.connects) == 1
+ await connection.close()
+
+
+async def test_tls_failure_after_allocation_retires_subscription(monkeypatch):
+ wire = FakeSubscriptionLiveWire()
+ allocated = asyncio.Event()
+ attempts = 0
+
+ async def options():
+ nonlocal attempts
+ attempts += 1
+ if attempts == 1:
+ await allocated.wait()
+ raise RuntimeError("Invalid local trust configuration")
+ return {}
+
+ async def answer(sdp):
+ allocated.set()
+
+ monkeypatch.setattr(transport, "websocket_options", options)
+ config = wire.config()
+ config.on_transport_ready = answer
+ with pytest.raises(RuntimeError, match="temporarily unavailable"):
+ await wire.provider().open_session(config)
+ assert len(wire.requests) == 1
+ assert wire.socket.sent == [{"type": "session.close"}]
+ assert wire.socket.closed == 1
+
+
+async def test_cancelled_local_tls_preparation_is_reaped(monkeypatch):
+ preparing, stopped = asyncio.Event(), asyncio.Event()
+
+ async def options():
+ preparing.set()
+ try:
+ await asyncio.Future()
+ finally:
+ stopped.set()
+
+ monkeypatch.setattr(transport, "websocket_options", options)
+
+ async def prepare():
+ async with transport.preparing_websocket_options():
+ await asyncio.Future()
+
+ task = asyncio.create_task(prepare())
+ await asyncio.wait_for(preparing.wait(), 1)
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert stopped.is_set()
diff --git a/tests/unit/realtime/test_live_startup_latency.py b/tests/unit/realtime/test_live_startup_latency.py
index 7dadb9f9e7..b8edbff822 100644
--- a/tests/unit/realtime/test_live_startup_latency.py
+++ b/tests/unit/realtime/test_live_startup_latency.py
@@ -261,7 +261,7 @@ async def test_late_connection_closes_even_if_hangup_send_fails(monkeypatch, tmp
from jarvis.live import session as module
from jarvis.live.config import LiveConfig
- opening, release = asyncio.Event(), asyncio.Event()
+ opening = asyncio.Event()
closes = []
async def send(message):
@@ -278,7 +278,11 @@ async def test_late_connection_closes_even_if_hangup_send_fails(monkeypatch, tmp
async def open_session(self, config):
opening.set()
- await release.wait()
+ try:
+ await asyncio.Future()
+ except asyncio.CancelledError:
+ # Remote allocation may complete concurrently with hangup.
+ pass
return SimpleNamespace(answer_sdp="answer", send=failed_send, close=close)
monkeypatch.setattr(module, "get_supervisor_tool_gateway", lambda: SimpleNamespace())
@@ -294,8 +298,7 @@ async def test_late_connection_closes_even_if_hangup_send_fails(monkeypatch, tmp
}))
await asyncio.wait_for(opening.wait(), 2)
await session.end()
- release.set()
- with pytest.raises(RuntimeError, match="control disconnected"):
+ with pytest.raises(asyncio.CancelledError):
await task
assert closes == [True]
diff --git a/tests/unit/realtime/test_live_transport_preparation.py b/tests/unit/realtime/test_live_transport_preparation.py
index 4cb3a1d60a..56fc6ac909 100644
--- a/tests/unit/realtime/test_live_transport_preparation.py
+++ b/tests/unit/realtime/test_live_transport_preparation.py
@@ -190,7 +190,7 @@ async def test_subscription_prepares_client_during_credentials_and_reports_phase
try:
connection = await provider.open_session(config)
assert phases == ["credentials_ready", "http_client_ready", "session_response",
- "control_tls_ready"]
+ "control_tls_ready", "control_connect_started"]
context = wire.connects[0][1]["ssl"]
assert context.check_hostname and context.verify_mode == ssl.CERT_REQUIRED
assert wire.auth_refreshes == [False] and len(wire.requests) == 1
diff --git a/tests/unit/realtime/test_live_websocket_startup.py b/tests/unit/realtime/test_live_websocket_startup.py
new file mode 100644
index 0000000000..a14069c565
--- /dev/null
+++ b/tests/unit/realtime/test_live_websocket_startup.py
@@ -0,0 +1,128 @@
+"""Content-free WebSocket transport and upgrade milestones use real lifecycle hooks."""
+
+from __future__ import annotations
+
+import asyncio
+from http import HTTPStatus
+from types import SimpleNamespace
+
+import pytest
+
+from jarvis.plugins.realtime._live_transport import startup_websocket_options
+from tests.fakes.fake_subscription_live_wire import (
+ FakeSubscriptionLiveWire,
+ FakeSubscriptionSocket,
+)
+
+
+async def test_real_socket_separates_transport_from_delayed_upgrade():
+ from websockets.asyncio.client import connect
+ from websockets.asyncio.server import serve
+
+ phases = []
+ request_received = asyncio.Event()
+ release_upgrade = asyncio.Event()
+
+ async def process_request(connection, request):
+ request_received.set()
+ await release_upgrade.wait()
+
+ async def handler(socket):
+ await socket.wait_closed()
+
+ async with serve(handler, "127.0.0.1", 0, process_request=process_request) as server:
+ port = server.sockets[0].getsockname()[1]
+
+ async def open_socket():
+ return await connect(
+ f"ws://127.0.0.1:{port}/private-path",
+ additional_headers={"Authorization": "private-header"},
+ **startup_websocket_options(phases.append),
+ )
+
+ pending = asyncio.create_task(open_socket())
+ try:
+ await asyncio.wait_for(request_received.wait(), 2)
+ assert phases == ["control_transport_connected", "control_handshake_started"]
+ assert not pending.done()
+ release_upgrade.set()
+ socket = await asyncio.wait_for(pending, 2)
+ await socket.close()
+ assert phases == [
+ "control_transport_connected", "control_handshake_started",
+ "control_handshake_complete",
+ ]
+ finally:
+ release_upgrade.set()
+ if not pending.done():
+ pending.cancel()
+ results = await asyncio.gather(pending, return_exceptions=True)
+ if not isinstance(results[0], BaseException):
+ await results[0].close()
+
+
+async def test_rejected_upgrade_never_reports_handshake_complete():
+ from websockets.asyncio.client import connect
+ from websockets.asyncio.server import serve
+ from websockets.exceptions import InvalidStatus
+
+ phases = []
+
+ def process_request(connection, request):
+ return connection.respond(HTTPStatus.UNAUTHORIZED, "private-response")
+
+ async def handler(socket):
+ pytest.fail("Rejected clients must never reach the connection handler")
+
+ async with serve(handler, "127.0.0.1", 0, process_request=process_request) as server:
+ port = server.sockets[0].getsockname()[1]
+ with pytest.raises(InvalidStatus):
+ await connect(
+ f"ws://127.0.0.1:{port}/private-path",
+ **startup_websocket_options(phases.append),
+ )
+ assert phases == ["control_transport_connected", "control_handshake_started"]
+
+
+async def test_subscription_injected_connector_receives_startup_hook():
+ from websockets.asyncio.client import ClientConnection
+
+ wire = FakeSubscriptionLiveWire()
+ phases = []
+ config = wire.config()
+ config.on_startup_phase = phases.append
+ connection = await wire.provider().open_session(config)
+ try:
+ assert len(wire.connects) == 1
+ assert "control_connect_started" in phases
+ assert issubclass(wire.connects[0][1]["create_connection"], ClientConnection)
+ # A fake connector did not complete a transport or a handshake.
+ assert "control_transport_connected" not in phases
+ assert "control_handshake_complete" not in phases
+ finally:
+ await connection.close()
+
+
+async def test_public_live_injected_connector_receives_startup_hook(monkeypatch):
+ from websockets.asyncio import client
+
+ from jarvis.plugins.realtime.openai_live import OpenAILiveProvider
+
+ phases, connects = [], []
+ socket = FakeSubscriptionSocket()
+
+ async def connect(url, **kwargs):
+ connects.append(kwargs)
+ return socket
+
+ monkeypatch.setattr(client, "connect", connect)
+ config = SimpleNamespace(session={}, offer_sdp="", on_startup_phase=phases.append)
+ connection = await OpenAILiveProvider(api_key="fake-key").open_session(config)
+ try:
+ assert len(connects) == 1
+ assert "control_connect_started" in phases
+ assert issubclass(connects[0]["create_connection"], client.ClientConnection)
+ assert "control_handshake_complete" not in phases
+ assert socket.sent[0]["type"] == "session.start"
+ finally:
+ await connection.close()
diff --git a/tests/unit/safety/test_explicit_intent.py b/tests/unit/safety/test_explicit_intent.py
index 9ef0ed501e..c082147354 100644
--- a/tests/unit/safety/test_explicit_intent.py
+++ b/tests/unit/safety/test_explicit_intent.py
@@ -80,10 +80,12 @@ def test_windows_literal_target_and_mismatch():
def test_shell_parsing_cannot_change_the_authorized_operation_or_target():
- target = r"/tmp/keep\me"
+ target = r"/tmp/keep\me" # noqa: S108 -- parser input, never opened
assert not command_confirms_destruction(f"rm {target}", f"delete '{target}'", windows=False)
assert command_confirms_destruction(f"rm '{target}'", f"delete '{target}'", windows=False)
- assert not command_confirms_destruction("rm\n/tmp/command", "delete /tmp/command", windows=False)
+ assert not command_confirms_destruction(
+ "rm\n/tmp/command", "delete /tmp/command", windows=False
+ )
@pytest.mark.parametrize("authorized", [False, True])
@@ -100,7 +102,9 @@ async def test_real_executor_leaves_protected_file_and_deletes_authorized_target
f"delete '{operand}'" if authorized else "List this directory. Do not delete any files."
)
bus = EventBus()
- executor = ToolExecutor(bus, RiskTierEvaluator(SafetyConfig()), ApprovalWorkflow(bus))
+ executor = ToolExecutor(
+ bus, RiskTierEvaluator(SafetyConfig(approval_mode="ask")), ApprovalWorkflow(bus)
+ )
result = await executor.execute(
RunShellTool(),
{"command": command},
diff --git a/tests/unit/safety/test_tool_executor_plausibility.py b/tests/unit/safety/test_tool_executor_plausibility.py
index 68bc051ad4..78638123c6 100644
--- a/tests/unit/safety/test_tool_executor_plausibility.py
+++ b/tests/unit/safety/test_tool_executor_plausibility.py
@@ -70,7 +70,7 @@ def _executor_with_plausibility(
auto_approval: bool = True,
) -> tuple[ToolExecutor, _AutoApproval]:
bus = EventBus()
- safety = safety or SafetyConfig()
+ safety = safety or SafetyConfig(approval_mode="ask")
evaluator = RiskTierEvaluator(safety)
approval = _AutoApproval(bus, approve=auto_approval)
executor = ToolExecutor(
diff --git a/tests/unit/safety/test_tool_executor_unattended.py b/tests/unit/safety/test_tool_executor_unattended.py
index e8a14593db..6d2db95838 100644
--- a/tests/unit/safety/test_tool_executor_unattended.py
+++ b/tests/unit/safety/test_tool_executor_unattended.py
@@ -94,7 +94,7 @@ def _executor(
approval = _WaitSpy(bus, timeout_s=timeout_s)
executor = ToolExecutor(
bus,
- RiskTierEvaluator(safety or SafetyConfig()),
+ RiskTierEvaluator(safety or SafetyConfig(approval_mode="ask")),
approval,
default_timeout_s=timeout_s,
)
diff --git a/tests/unit/safety/test_tool_executor_voice_confirm.py b/tests/unit/safety/test_tool_executor_voice_confirm.py
index f88a89f4c4..250a3f9d95 100644
--- a/tests/unit/safety/test_tool_executor_voice_confirm.py
+++ b/tests/unit/safety/test_tool_executor_voice_confirm.py
@@ -62,7 +62,7 @@ class _BlockingApproval(ApprovalWorkflow):
def _executor() -> tuple[ToolExecutor, _BlockingApproval, EventBus]:
bus = EventBus()
- evaluator = RiskTierEvaluator(SafetyConfig())
+ evaluator = RiskTierEvaluator(SafetyConfig(approval_mode="ask"))
approval = _BlockingApproval(bus)
executor = ToolExecutor(bus=bus, evaluator=evaluator, approval=approval)
return executor, approval, bus
diff --git a/tests/unit/society/test_agent_credentials.py b/tests/unit/society/test_agent_credentials.py
new file mode 100644
index 0000000000..2818d698fe
--- /dev/null
+++ b/tests/unit/society/test_agent_credentials.py
@@ -0,0 +1,465 @@
+"""Credentials an agent asks for: a secure field, a vault, and a value the agent never sees."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+from uuid import uuid4
+
+import pytest
+
+from jarvis.agent_chat.credential_requests import (
+ MAX_REQUESTS_PER_TURN,
+ CredentialSpec,
+ TooManyCredentialRequests,
+)
+from jarvis.agent_chat.service import AgentChatService, _Running
+from jarvis.agent_chat.store import AgentChatStore
+from jarvis.society import credentials
+from jarvis.society.credential_tool import CREDENTIAL_TOOL_NAME, RequestCredentialTool
+from jarvis.society.credentials import CredentialError, CredentialVault, validate_env_name
+from jarvis.society.shell import ShellResult
+from tests.fakes.fake_secret_store import FakeSecretStore
+
+SECRET = "ghp_" + "s3cr3t" * 6
+SID = "society:ada"
+
+
+def vault(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> CredentialVault:
+ store = CredentialVault(tmp_path / "credentials", secrets=FakeSecretStore())
+ monkeypatch.setattr(credentials, "current_vault", lambda: store)
+ return store
+
+
+# ------------------------------------------------------------------ vault
+
+
+def test_env_names_are_conventional_and_never_system_variables() -> None:
+ assert validate_env_name(" GITHUB_TOKEN ") == "GITHUB_TOKEN"
+ for bad in ("github_token", "1TOKEN", "A", "PATH", "SYSTEMROOT", "JARVIS_CONTROL_API_KEY", ""):
+ with pytest.raises(CredentialError):
+ validate_env_name(bad)
+
+
+def test_the_index_holds_names_and_the_store_holds_values(vault: CredentialVault) -> None:
+ info = vault.store("ada", "GITHUB_TOKEN", f" {SECRET} ", label="GitHub token")
+ assert info.env == "GITHUB_TOKEN" and info.label == "GitHub token"
+ assert vault.has("ada", "GITHUB_TOKEN") and not vault.has("bob", "GITHUB_TOKEN")
+ assert [row.env for row in vault.list("ada")] == ["GITHUB_TOKEN"]
+ assert vault.values("ada") == {"GITHUB_TOKEN": SECRET}
+ index = next((vault._root).glob("*.json")).read_text(encoding="utf-8")
+ assert SECRET not in index and "GitHub token" in index
+ assert vault.delete("ada", "GITHUB_TOKEN")
+ assert not vault.delete("ada", "GITHUB_TOKEN")
+ assert vault.values("ada") == {} and vault.list("ada") == []
+
+
+def test_bad_values_are_refused(tmp_path: Path) -> None:
+ vault = CredentialVault(tmp_path, secrets=FakeSecretStore())
+ for bad in ("", " ", "two\nlines", "x" * 20_000):
+ with pytest.raises(CredentialError):
+ vault.store("ada", "API_KEY", bad)
+ refusing = CredentialVault(tmp_path, secrets=FakeSecretStore(refuse=True))
+ with pytest.raises(CredentialError):
+ refusing.store("ada", "API_KEY", SECRET)
+ assert refusing.list("ada") == []
+
+
+def test_encoded_forms_and_nested_secrets_are_masked(vault: CredentialVault) -> None:
+ import base64
+ from urllib.parse import quote
+
+ vault.store("ada", "GITHUB_TOKEN", SECRET)
+ vault.store("ada", "GITHUB_PREFIX", SECRET[:12]) # a shorter secret inside the longer one
+ printed = " ".join([
+ SECRET,
+ base64.b64encode(SECRET.encode()).decode(),
+ base64.b64encode((SECRET + "\n").encode()).decode(),
+ quote(SECRET + "/x", safe=""),
+ ])
+ masked = vault.redact("ada", printed)
+ assert SECRET[:12] not in masked and "s3cr3t" not in masked
+ assert masked.startswith("[credential GITHUB_TOKEN] ")
+
+
+def test_a_vanished_store_entry_is_not_available(tmp_path: Path) -> None:
+ secrets = FakeSecretStore()
+ vault = CredentialVault(tmp_path, secrets=secrets)
+ vault.store("ada", "GITHUB_TOKEN", SECRET)
+ secrets.values.clear() # the OS credential store lost it
+ fresh = CredentialVault(tmp_path, secrets=secrets)
+ assert [row.env for row in fresh.list("ada")] == ["GITHUB_TOKEN"]
+ assert not fresh.has("ada", "GITHUB_TOKEN")
+
+
+def test_a_save_during_a_load_is_never_lost(tmp_path: Path) -> None:
+ class SlowStore(FakeSecretStore):
+ armed = False
+
+ def get(self, slot: str) -> str | None:
+ if self.armed:
+ self.armed = False
+ # Another caller saves while this load reads the store.
+ vault.store("ada", "DISCORD_BOT_TOKEN", "d" * 30)
+ return super().get(slot)
+
+ secrets = SlowStore()
+ CredentialVault(tmp_path, secrets=secrets).store("ada", "GITHUB_TOKEN", SECRET)
+ vault = CredentialVault(tmp_path, secrets=secrets)
+ secrets.armed = True
+ assert vault.values("ada") == {"GITHUB_TOKEN": SECRET, "DISCORD_BOT_TOKEN": "d" * 30}
+
+
+def test_values_are_masked_everywhere_in_a_payload(vault: CredentialVault) -> None:
+ vault.store("ada", "GITHUB_TOKEN", SECRET)
+ vault.store("ada", "PIN_CODE", "1234") # too short to mask without eating numbers
+ payload = {"output": f"token={SECRET}", "list": [SECRET, 1234], "keep": "1234"}
+ redacted = vault.redact_payload("ada", payload)
+ assert redacted == {
+ "output": "token=[credential GITHUB_TOKEN]",
+ "list": ["[credential GITHUB_TOKEN]", 1234],
+ "keep": "1234",
+ }
+ clean = {"output": "nothing secret"}
+ assert vault.redact_payload("ada", clean) is clean
+ assert vault.redact_payload("bob", payload) is payload
+
+
+def test_values_load_from_the_store_once(tmp_path: Path) -> None:
+ secrets = FakeSecretStore()
+ CredentialVault(tmp_path, secrets=secrets).store("ada", "GITHUB_TOKEN", SECRET)
+ secrets.reads = 0
+ fresh = CredentialVault(tmp_path, secrets=secrets) # a new process
+ assert not fresh.is_loaded("ada")
+ assert fresh.values("ada") == {"GITHUB_TOKEN": SECRET}
+ fresh.values("ada")
+ assert secrets.reads == 1 and fresh.is_loaded("ada")
+
+
+# ------------------------------------------------------------------ the card
+
+
+async def _service_with_turn(tmp_path: Path, session_id: str = SID):
+ svc = AgentChatService(AgentChatStore(":memory:"), assistant_name=lambda: "Testo")
+ session = svc.store.create_session(
+ provider="fakeprov",
+ model="m",
+ effort="high",
+ cwd=str(tmp_path),
+ session_id=session_id,
+ surface="society",
+ )
+ run = _Running("turn-1", asyncio.Event())
+ run.task = asyncio.create_task(asyncio.sleep(3600))
+ svc._running[session.session_id] = run
+ return svc, session.session_id, run.task
+
+
+async def _next(q: asyncio.Queue, kind: str) -> dict[str, Any]:
+ while True:
+ ev = await asyncio.wait_for(q.get(), timeout=5)
+ if ev["kind"] == kind:
+ return ev["payload"]
+
+
+def _logged(svc: AgentChatService, sid: str) -> str:
+ return json.dumps(svc.store.list_events(sid), ensure_ascii=False)
+
+
+_SPEC = CredentialSpec.build(
+ "GITHUB_TOKEN", "GitHub token", "To open the pull request.", "ghp_…"
+)
+
+
+async def test_the_value_goes_to_save_and_never_into_the_chat(tmp_path: Path, vault) -> None:
+ svc, sid, task = await _service_with_turn(tmp_path)
+ q = svc.subscribe(sid)
+ saved: list[str] = []
+ rid = await svc.open_credential_request(sid, _SPEC, saved.append, asker="Ada")
+ card = await _next(q, "credential_required")
+ assert card["request_id"] == rid and card["env"] == "GITHUB_TOKEN"
+ assert card["asker"] == "Ada" and card["placeholder"] == "ghp_…"
+ assert svc.pending_credential_requests(sid) == [rid]
+ assert not await svc.submit_credential("other", rid, SECRET)
+ assert await svc.submit_credential(sid, rid, SECRET)
+ assert saved == [SECRET]
+ resolved = await _next(q, "credential_resolved")
+ assert resolved == {
+ "turn_id": "turn-1", "request_id": rid, "env": "GITHUB_TOKEN", "status": "saved"
+ }
+ assert await svc.wait_credential_request(sid, rid, 1) == "saved"
+ assert await svc.submit_credential(sid, rid, SECRET) # a lost success receipt is retryable
+ assert saved == [SECRET]
+ assert SECRET not in _logged(svc, sid)
+ task.cancel()
+
+
+async def test_a_refused_value_keeps_the_field_open(tmp_path: Path) -> None:
+ svc, sid, task = await _service_with_turn(tmp_path)
+
+ def save(value: str) -> None:
+ raise CredentialError("the credential is empty")
+
+ rid = await svc.open_credential_request(sid, _SPEC, save)
+ with pytest.raises(ValueError):
+ await svc.submit_credential(sid, rid, " ")
+ assert svc.pending_credential_requests(sid) == [rid]
+ task.cancel()
+
+
+async def test_only_explicit_decline_closes_an_unsaved_field(tmp_path: Path) -> None:
+ svc, sid, task = await _service_with_turn(tmp_path)
+ q = svc.subscribe(sid)
+ declined = await svc.open_credential_request(sid, _SPEC, lambda _v: None)
+ assert await svc.decline_credential(sid, declined)
+ assert await svc.wait_credential_request(sid, declined, 1) == "declined"
+ late = await svc.open_credential_request(sid, _SPEC, lambda _v: None, timeout_s=0.05)
+ assert await svc.wait_credential_request(sid, late, 0.1) is None
+ assert (await _next(q, "credential_resolved"))["status"] == "declined"
+ assert svc.pending_credential_requests(sid) == [late]
+ task.cancel()
+ svc2, sid2, task2 = await _service_with_turn(tmp_path, "society:bob")
+ open_rid = await svc2.open_credential_request(sid2, _SPEC, lambda _v: None)
+ svc2._cancel_questions(sid2) # what a finished or stopped turn runs
+ assert svc2.pending_credential_requests(sid2) == [open_rid]
+ assert await svc2.wait_credential_request(sid2, open_rid, 0) is None
+ assert await svc2.submit_credential(sid2, open_rid, SECRET)
+ task2.cancel()
+
+
+async def test_a_turn_runs_out_of_fields_and_needs_a_turn(tmp_path: Path) -> None:
+ svc, sid, task = await _service_with_turn(tmp_path)
+ for index in range(MAX_REQUESTS_PER_TURN):
+ await svc.open_credential_request(
+ sid, CredentialSpec.build(f"TOKEN_{index}", "Token"), lambda _v: None,
+ )
+ with pytest.raises(TooManyCredentialRequests):
+ await svc.open_credential_request(sid, _SPEC, lambda _v: None)
+ task.cancel()
+ idle = AgentChatService(AgentChatStore(":memory:"))
+ with pytest.raises(RuntimeError):
+ await idle.open_credential_request("society:ada", _SPEC, lambda _v: None)
+
+
+async def test_a_printed_value_is_masked_in_the_chat_log(tmp_path: Path, vault) -> None:
+ vault.store("ada", "GITHUB_TOKEN", SECRET)
+ svc, sid, task = await _service_with_turn(tmp_path)
+ from jarvis.agent_chat.events import make_event
+
+ await svc._emit(
+ sid,
+ make_event(
+ "tool_result",
+ {"turn_id": "turn-1", "call_id": "c1", "output": f"echo -> {SECRET}"},
+ ),
+ )
+ logged = _logged(svc, sid)
+ assert SECRET not in logged and "[credential GITHUB_TOKEN]" in logged
+ task.cancel()
+
+
+# ------------------------------------------------------------------ the tool
+
+
+def _tool(session_id: str, service: Any) -> RequestCredentialTool:
+ runtime = SimpleNamespace(
+ chat_service=lambda: service, cached_agent=lambda _id: SimpleNamespace(name="Ada")
+ )
+ return RequestCredentialTool(runtime, "ada", session_id=session_id)
+
+
+_ARGS = {
+ "env": "GITHUB_TOKEN",
+ "label": "GitHub token",
+ "description": "To push the fix. Create it under Settings > Developer settings.",
+}
+
+
+async def test_the_tool_asks_and_the_agent_only_learns_the_name(tmp_path: Path, vault) -> None:
+ svc, sid, task = await _service_with_turn(tmp_path)
+ q = svc.subscribe(sid)
+ first = await asyncio.wait_for(_tool(sid, svc).execute(dict(_ARGS), None), timeout=1)
+ assert first.success and first.output["status"] == "waiting"
+ card = await _next(q, "credential_required")
+ assert await svc.submit_credential(sid, card["request_id"], SECRET)
+ result = await _tool(sid, svc).execute({"wait_for": card["request_id"]}, None)
+ assert result.success and result.output["status"] == "saved"
+ assert "$GITHUB_TOKEN" in result.output["note"] and "society_shell" in result.output["note"]
+ assert SECRET not in json.dumps(result.output) + (result.error or "")
+ assert vault.values("ada") == {"GITHUB_TOKEN": SECRET}
+ assert [row.label for row in vault.list("ada")] == ["GitHub token"]
+ again = await _tool(sid, svc).execute(dict(_ARGS), None)
+ assert again.success and again.output["status"] == "available" # no second card
+ task.cancel()
+
+
+async def test_checking_a_pending_field_returns_without_waiting(tmp_path: Path, vault) -> None:
+ svc, sid, task = await _service_with_turn(tmp_path)
+ tool = _tool(sid, svc)
+ first = await asyncio.wait_for(tool.execute(dict(_ARGS), None), timeout=1)
+ assert first.success and first.output["status"] == "waiting"
+ rid = first.output["request_id"]
+ pending = await asyncio.wait_for(tool.execute({"wait_for": rid}, None), timeout=1)
+ assert pending.output["status"] == "waiting"
+ assert svc.pending_credential_requests(sid) == [rid]
+ assert await svc.decline_credential(sid, rid)
+ second = await tool.execute({"wait_for": rid}, None)
+ assert not second.success and second.output["status"] == "declined"
+ assert "not set" in (second.error or "")
+ gone = await tool.execute({"wait_for": "nope"}, None)
+ assert not gone.success
+ task.cancel()
+
+
+async def test_followup_finishes_before_the_token_is_saved(
+ tmp_path: Path, vault, monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ from jarvis.agent_chat import service as service_module
+ from jarvis.agent_chat.events import make_event
+ from jarvis.agent_chat.send_queue import send_or_queue
+
+ svc = AgentChatService(AgentChatStore(":memory:"), assistant_name=lambda: "Testo")
+ session = svc.store.create_session(
+ provider="fakeprov", model="m", effort="", cwd=str(tmp_path), session_id=SID,
+ surface="society",
+ )
+ results = []
+ replies = []
+
+ async def bind(_sid, **_kwargs):
+ return session
+
+ async def run(handle, prompt):
+ if not results:
+ results.append(await _tool(SID, svc).execute(dict(_ARGS), None))
+ else:
+ replies.append(prompt)
+ assert svc.pending_credential_requests(SID) == [results[0].output["request_id"]]
+ await handle.emit(make_event("assistant_text", {
+ "turn_id": handle.turn_id, "text": "It allows me to open the pull request.",
+ }))
+ await handle.emit(make_event("turn_finished", {
+ "turn_id": handle.turn_id, "status": "done",
+ }))
+
+ monkeypatch.setattr(svc, "bind_society_session", bind)
+ monkeypatch.setattr(service_module, "resolve_runner", lambda *a, **k: "api")
+ monkeypatch.setattr(service_module, "supports_api_runner", lambda _provider: True)
+ monkeypatch.setattr(service_module, "run_api_turn", run)
+ try:
+ await svc.send(SID, "Open the pull request")
+ await asyncio.wait_for(svc.wait_turn(SID), timeout=2)
+ assert results[0].output["status"] == "waiting"
+ rid = results[0].output["request_id"]
+ turn_id, queue_id = await send_or_queue(svc, SID, "What do you need it for?")
+ assert turn_id and not queue_id
+ await asyncio.wait_for(svc.wait_turn(SID), timeout=2)
+ assert replies == ["What do you need it for?"]
+ assert not vault.has("ada", "GITHUB_TOKEN")
+ assert await svc.submit_credential(SID, rid, SECRET)
+ assert vault.values("ada") == {"GITHUB_TOKEN": SECRET}
+ assert (await _tool(SID, svc).execute({"wait_for": rid}, None)).output["status"] == "saved"
+ assert SECRET not in _logged(svc, SID)
+ finally:
+ await svc.cancel(SID)
+
+
+async def test_routines_and_bad_requests_never_open_a_field(vault) -> None:
+ opened: list[Any] = []
+ service = SimpleNamespace(open_credential_request=lambda *a, **k: opened.append(a))
+ routine = await _tool("society:ada:routine:t1:run1", service).execute(dict(_ARGS), None)
+ assert not routine.success and routine.output["status"] == "unattended"
+ bad = await _tool(SID, service).execute({**_ARGS, "env": "path"}, None)
+ assert not bad.success and "invalid request" in (bad.error or "")
+ nameless = await _tool(SID, service).execute({**_ARGS, "label": " "}, None)
+ assert not nameless.success
+ unexplained = await _tool(SID, service).execute({**_ARGS, "description": " "}, None)
+ assert not unexplained.success and "explain why" in unexplained.error
+ assert opened == []
+
+
+def test_the_tool_is_offered_outside_routines(monkeypatch: pytest.MonkeyPatch) -> None:
+ from jarvis.society import surface
+
+ browser = SimpleNamespace(is_installed=lambda: False, live=SimpleNamespace(model_resolver=None))
+ rt = SimpleNamespace(browser=browser, chat_service=lambda: None)
+ monkeypatch.setattr(surface, "current_runtime", lambda: rt)
+ canonical = SimpleNamespace(session_id="society:ada", cwd="", permission_mode="")
+ routine = SimpleNamespace(
+ session_id="society:ada:routine:t1:run1", cwd="", permission_mode="bypass"
+ )
+ assert CREDENTIAL_TOOL_NAME in surface.society_tools(None, None, canonical)
+ assert CREDENTIAL_TOOL_NAME not in surface.society_tools(None, None, routine)
+
+
+# ------------------------------------------------------------------ where the value is used
+
+
+class _EnvBackend:
+ name = "local"
+ accepts_env = True
+
+ def __init__(self) -> None:
+ self.extra_env: dict[str, str] | None = None
+
+ async def run(self, command: str, *, cwd: Path, timeout_s: float, extra_env=None):
+ self.extra_env = dict(extra_env or {})
+ return ShellResult(output=f"Bearer {SECRET}", exit_code=0, seconds=0.01)
+
+
+async def test_society_shell_sets_the_variable_and_masks_its_output(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ from jarvis.society.agent_tools import ShellTool
+ from jarvis.society.runtime import SocietyRuntime
+
+ rt = SocietyRuntime(tmp_path / "society", seed_starter_team=False)
+ await rt.ensure_started()
+ try:
+ await rt.roster.create(name="Scout")
+ store = CredentialVault(tmp_path / "creds", secrets=FakeSecretStore())
+ store.store("scout", "GITHUB_TOKEN", SECRET)
+ monkeypatch.setattr(credentials, "current_vault", lambda: store)
+ backend = _EnvBackend()
+ tool = ShellTool(rt, "scout", workspace=tmp_path / "ws", backend=backend)
+ ctx = SimpleNamespace(trace_id=uuid4(), user_utterance="", config={}, memory_read=None)
+ command = 'curl -H "Authorization: Bearer $GITHUB_TOKEN" https://api.github.com/user'
+ result = await tool.execute({"command": command}, ctx)
+ assert result.success, result.error
+ assert backend.extra_env == {"GITHUB_TOKEN": SECRET}
+ assert result.output["output"] == "Bearer [credential GITHUB_TOKEN]"
+ finally:
+ await rt.close()
+
+
+async def test_the_briefing_names_stored_credentials_and_says_when_there_are_none(
+ tmp_path: Path, vault, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ from jarvis.society.surface import _credential_line
+
+ monkeypatch.setattr(credentials, "vault_for", lambda _data_dir: vault)
+ rt = SimpleNamespace(data_dir=tmp_path)
+ # A resumed CLI conversation remembers a token it once saved; the briefing
+ # must say it is gone, or the agent reports a deleted credential as stored.
+ assert "Stored credentials: none" in await _credential_line(rt, "ada")
+ vault.store("ada", "DISCORD_BOT_TOKEN", SECRET, label="Discord bot token")
+ line = await _credential_line(rt, "ada")
+ assert "DISCORD_BOT_TOKEN (Discord bot token)" in line and SECRET not in line
+
+
+async def test_a_question_card_never_asks_for_the_secret_itself() -> None:
+ from jarvis.society.ask_tool import AskUserTool
+
+ rt = SimpleNamespace(chat_service=lambda: None)
+ tool = AskUserTool(rt, "ada", session_id=SID)
+ asking = {"question": "Please paste your Discord bot token.", "options": ["Done", "Later"]}
+ result = await tool.execute({"questions": [asking]}, None)
+ assert not result.success and "society_request_credential" in (result.error or "")
+ # Choosing HOW to connect is a real decision, not a secret.
+ choice = {"question": "Connect the bot with a token or a browser login?", "options": ["Token", "Browser"]} # noqa: E501
+ result = await tool.execute({"questions": [choice]}, None)
+ assert "society_request_credential" not in (result.error or "")
diff --git a/tests/unit/society/test_credential_submission.py b/tests/unit/society/test_credential_submission.py
new file mode 100644
index 0000000000..0095e79a1c
--- /dev/null
+++ b/tests/unit/society/test_credential_submission.py
@@ -0,0 +1,270 @@
+"""Credential validation, safe failures and recovery with fake storage and HTTP."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+
+import httpx
+import pytest
+from fastapi import FastAPI
+
+from jarvis.agent_chat.credential_requests import CredentialRequestBusy
+from jarvis.agent_chat.events import make_event
+from jarvis.agent_chat.service import AgentChatService
+from jarvis.society import credentials
+from jarvis.society.credentials import CredentialError, CredentialVault
+from jarvis.ui.web.agent_chat_routes import router
+from tests.fakes.fake_secret_store import FakeSecretStore
+from tests.unit.society.test_agent_credentials import _SPEC, SECRET, _service_with_turn
+
+
+def _discord(monkeypatch: pytest.MonkeyPatch, handler):
+ client_type = httpx.AsyncClient
+ seen = []
+
+ def respond(request):
+ seen.append(request)
+ return handler(request)
+
+ monkeypatch.setattr(
+ httpx, "AsyncClient",
+ lambda **kwargs: client_type(transport=httpx.MockTransport(respond), **kwargs),
+ )
+ return seen
+
+
[email protected]("status,body,code", [
+ (401, {"message": SECRET}, "invalid_token"),
+ (403, {"message": SECRET}, "invalid_token"),
+ (429, {"message": SECRET}, "network_error"),
+ (503, {"message": SECRET}, "network_error"),
+ (302, {}, "network_error"),
+ (200, {"id": "123", "bot": False}, "invalid_token"),
+ (200, {"bot": True}, "invalid_token"),
+])
+async def test_discord_failure_does_not_store_or_echo_the_value(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, status, body, code,
+) -> None:
+ vault = CredentialVault(tmp_path, secrets=FakeSecretStore())
+ monkeypatch.setattr(credentials, "current_vault", lambda: vault)
+ seen = _discord(monkeypatch, lambda _req: httpx.Response(
+ status, json=body, headers={"Location": "https://untrusted.invalid/token"},
+ ))
+ with pytest.raises(CredentialError) as caught:
+ await credentials.save_requested_credential("ada", "DISCORD_BOT_TOKEN", SECRET)
+ assert caught.value.code == code
+ assert SECRET not in str(caught.value)
+ assert vault.values("ada") == {} and vault.list("ada") == []
+ assert len(seen) == 1 # No redirect or automatic retry with the secret.
+ assert str(seen[0].url) == "https://discord.com/api/v10/users/@me"
+
+
+async def test_discord_success_checks_bot_identity_then_persists(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ secrets = FakeSecretStore()
+ vault = CredentialVault(tmp_path, secrets=secrets)
+ monkeypatch.setattr(credentials, "current_vault", lambda: vault)
+ seen = _discord(monkeypatch, lambda _req: httpx.Response(200, json={"id": "123", "bot": True}))
+ await credentials.save_requested_credential("ada", "DISCORD_BOT_TOKEN", f" {SECRET} ")
+ assert seen[0].headers["authorization"] == f"Bot {SECRET}"
+ assert CredentialVault(tmp_path, secrets=secrets).values("ada") == {"DISCORD_BOT_TOKEN": SECRET}
+ index = await asyncio.to_thread(lambda: (tmp_path / "ada.json").read_text(encoding="utf-8"))
+ assert SECRET not in index
+
+
[email protected]("failure,code", [
+ (httpx.ConnectError, "network_error"),
+ (httpx.ReadTimeout, "validation_timeout"),
+])
+async def test_discord_transport_errors_are_safe_and_retryable(
+ monkeypatch: pytest.MonkeyPatch, failure, code,
+) -> None:
+ def respond(request):
+ raise failure(SECRET, request=request)
+
+ _discord(monkeypatch, respond)
+ with pytest.raises(CredentialError) as caught:
+ await credentials.validate_token("DISCORD_BOT_TOKEN", SECRET)
+ assert caught.value.code == code and SECRET not in str(caught.value)
+
+
+async def test_arbitrary_credentials_never_select_a_validation_host(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ def never(_request):
+ raise AssertionError("An unknown service must not receive the secret")
+
+ seen = _discord(monkeypatch, never)
+ await credentials.validate_token("CUSTOM_API_KEY", SECRET, label="https://untrusted.invalid")
+ assert seen == []
+
+
+def test_storage_backend_errors_and_failed_index_commit_restore_previous_value(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ secrets = FakeSecretStore()
+ vault = CredentialVault(tmp_path, secrets=secrets)
+ vault.store("ada", "API_KEY", "previous-safe-value")
+
+ def failed_index(*_args):
+ raise OSError(f"disk failure containing {SECRET}")
+
+ monkeypatch.setattr(vault, "_write_index", failed_index)
+ with pytest.raises(CredentialError) as caught:
+ vault.store("ada", "API_KEY", SECRET)
+ assert caught.value.code == "storage_failed" and SECRET not in str(caught.value)
+ assert vault.values("ada") == {"API_KEY": "previous-safe-value"}
+ assert CredentialVault(tmp_path, secrets=secrets).values("ada") == vault.values("ada")
+
+
+def test_a_store_that_claims_success_without_saving_is_not_accepted(tmp_path: Path) -> None:
+ class LosingStore(FakeSecretStore):
+ def set(self, slot: str, value: str) -> bool:
+ return True
+
+ vault = CredentialVault(tmp_path, secrets=LosingStore())
+ with pytest.raises(CredentialError, match="could not be stored"):
+ vault.store("ada", "API_KEY", SECRET)
+ assert vault.values("ada") == {} and vault.list("ada") == []
+
+
+async def test_pending_field_survives_restart_and_saved_retry_does_not_write_again(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ vault = CredentialVault(tmp_path / "vault", secrets=FakeSecretStore())
+ monkeypatch.setattr(credentials, "current_vault", lambda: vault)
+ first, sid, turn = await _service_with_turn(tmp_path)
+ rid = await first.open_credential_request(sid, _SPEC, lambda _value: None)
+ turn.cancel()
+ first._cancel_questions(sid)
+ fresh = AgentChatService(first.store)
+ assert await fresh.submit_credential(sid, rid, SECRET)
+ assert vault.values("ada") == {"GITHUB_TOKEN": SECRET}
+ restarted = AgentChatService(first.store)
+ assert await restarted.submit_credential(sid, rid, "different-value")
+ assert vault.values("ada") == {"GITHUB_TOKEN": SECRET}
+ assert not await restarted.submit_credential("society:bob", rid, SECRET)
+ assert SECRET not in json.dumps(first.store.list_events(sid))
+
+
[email protected]("status", ["cancelled", "declined", "timeout"])
+async def test_historical_closed_requests_are_never_reopened(
+ tmp_path: Path, status: str,
+) -> None:
+ first, sid, turn = await _service_with_turn(tmp_path)
+ rid = await first.open_credential_request(sid, _SPEC, lambda _value: None)
+ await first._emit(sid, make_event("credential_resolved", {"request_id": rid, "status": status}))
+ fresh = AgentChatService(first.store)
+ assert not await fresh.submit_credential(sid, rid, SECRET)
+ assert fresh.pending_credential_requests(sid) == []
+ turn.cancel()
+
+
+async def test_disconnected_submit_finishes_once_and_cannot_race_decline(tmp_path: Path) -> None:
+ svc, sid, turn = await _service_with_turn(tmp_path)
+ entered, release = asyncio.Event(), asyncio.Event()
+ saved = []
+
+ async def save(value):
+ entered.set()
+ await release.wait()
+ saved.append(value)
+
+ rid = await svc.open_credential_request(sid, _SPEC, save)
+ caller = asyncio.create_task(svc.submit_credential(sid, rid, SECRET))
+ await entered.wait()
+ caller.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await caller
+ with pytest.raises(CredentialRequestBusy):
+ await svc.decline_credential(sid, rid)
+ with pytest.raises(CredentialRequestBusy):
+ await svc.submit_credential(sid, rid, SECRET)
+ svc._cancel_questions(sid)
+ release.set()
+ assert await svc.wait_credential_request(sid, rid, 1) == "saved"
+ assert await svc.submit_credential(sid, rid, SECRET)
+ assert saved == [SECRET]
+ turn.cancel()
+
+
+async def test_repeated_requests_reuse_the_existing_field(tmp_path: Path) -> None:
+ svc, sid, turn = await _service_with_turn(tmp_path)
+ rid = await svc.open_credential_request(sid, _SPEC, lambda _value: None)
+ svc._cancel_questions(sid)
+ assert await svc.open_credential_request(sid, _SPEC, lambda _value: None) == rid
+ assert svc.pending_credential_requests(sid) == [rid]
+ turn.cancel()
+
+
[email protected]("failure,code,status", [
+ (CredentialError("The token was rejected."), "invalid_token", 400),
+ (CredentialError("Network unavailable.", code="network_error"), "network_error", 503),
+ (CredentialError("Timed out.", code="validation_timeout"), "validation_timeout", 504),
+ (OSError(SECRET), "storage_failed", 503),
+])
+async def test_route_failures_keep_field_open_and_success_has_receipt(
+ tmp_path: Path, caplog, failure, code, status,
+) -> None:
+ svc, sid, turn = await _service_with_turn(tmp_path)
+ failing = True
+ saved = []
+
+ def save(value):
+ if failing:
+ raise failure
+ saved.append(value)
+
+ rid = await svc.open_credential_request(sid, _SPEC, save)
+ app = FastAPI()
+ app.state.agent_chat = svc
+ app.include_router(router)
+ path = f"/api/agent-chat/sessions/{sid}/credentials/{rid}"
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=app), base_url="http://test",
+ ) as client:
+ failed = await client.post(path, json={"value": SECRET})
+ assert failed.status_code == status
+ assert failed.json()["detail"]["code"] == code
+ assert SECRET not in failed.text + caplog.text
+ assert svc.pending_credential_requests(sid) == [rid]
+ failing = False
+ success = await client.post(path, json={"value": SECRET})
+ assert success.json() == {"ok": True, "request_id": rid, "status": "saved"}
+ assert saved == [SECRET]
+ turn.cancel()
+
+
+async def test_non_string_value_is_not_echoed_in_validation_error(tmp_path: Path) -> None:
+ svc, sid, turn = await _service_with_turn(tmp_path)
+ rid = await svc.open_credential_request(sid, _SPEC, lambda _value: None)
+ app = FastAPI()
+ app.state.agent_chat = svc
+ app.include_router(router)
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=app), base_url="http://test",
+ ) as client:
+ response = await client.post(
+ f"/api/agent-chat/sessions/{sid}/credentials/{rid}", json={"value": {"secret": SECRET}},
+ )
+ assert response.status_code == 400 and SECRET not in response.text
+ assert svc.pending_credential_requests(sid) == [rid]
+ turn.cancel()
+
+
+async def test_failed_save_after_legacy_timeout_keeps_field_open(tmp_path: Path) -> None:
+ svc, sid, turn = await _service_with_turn(tmp_path)
+
+ async def save(_value):
+ await asyncio.sleep(0.02)
+ raise CredentialError("The token was rejected.")
+
+ rid = await svc.open_credential_request(sid, _SPEC, save, timeout_s=0.001)
+ with pytest.raises(CredentialError):
+ await svc.submit_credential(sid, rid, SECRET)
+ assert svc.pending_credential_requests(sid) == [rid]
+ assert not any(e["kind"] == "credential_resolved" for e in svc.store.list_events(sid))
+ turn.cancel()
diff --git a/tests/unit/society/test_identity_proposal.py b/tests/unit/society/test_identity_proposal.py
index cc7d9de51b..42517d979a 100644
--- a/tests/unit/society/test_identity_proposal.py
+++ b/tests/unit/society/test_identity_proposal.py
@@ -52,7 +52,7 @@ async def test_fresh_agent_applies_its_first_identity_from_the_user_turn(world):
rt, svc = world
fresh, _ = await rt.roster.create()
placeholder = fresh.name
- assert placeholder == "New Bot"
+ assert placeholder == "New Agent"
tool = ProposeChangeTool(rt, fresh.agent_id, session_id=fresh.session_id)
token = _user_turn(fresh.session_id, "You handle my Gmail inbox.")
try:
@@ -152,6 +152,109 @@ async def test_a_long_first_role_goes_to_a_card(world): # noqa: F811
assert is_fresh(await rt.roster.get(fresh.agent_id))
+async def _lead(rt):
+ from jarvis.society.roster import LEAD_AGENT_ID
+
+ lead = await rt.roster.get(LEAD_AGENT_ID)
+ if lead is None:
+ lead, _ = await rt.roster.create(name="Jarvis", tier="lead", title="Lead")
+ return lead
+
+
+async def test_naming_a_new_agent_in_its_own_chat_leaves_jarvis_alone(world): # noqa: F811
+ rt, svc = world
+ lead = await _lead(rt)
+ fresh, _ = await rt.roster.create()
+ request = "Your name is Miro. Just a Discord mod in every single thread."
+ tool = ProposeChangeTool(rt, fresh.agent_id, session_id=fresh.session_id)
+ token = _user_turn(fresh.session_id, request)
+ try:
+ result = await tool.execute(
+ {
+ "kind": "identity",
+ "mode": "apply",
+ "request_quote": "Just a Discord mod in every single thread.",
+ "payload": {"name": "Miro", "title": "Discord moderator"},
+ },
+ None,
+ )
+ finally:
+ current_chat_turn.reset(token)
+ assert result.success, result.error
+ assert (await rt.roster.get(fresh.agent_id)).name == "Miro"
+ unchanged = await rt.roster.get(lead.agent_id)
+ assert (unchanged.name, unchanged.title) == (lead.name, lead.title)
+ outcome = _identity_notices(svc, "proposal_resolved")[-1]
+ assert outcome["agent_id"] == fresh.agent_id
+ assert outcome["previous"]["name"] == fresh.name # what Undo restores
+
+
+async def test_jarvis_itself_is_never_renamed_by_an_identity_change(world): # noqa: F811
+ rt, _svc = world
+ lead = await _lead(rt)
+ tool = ProposeChangeTool(rt, lead.agent_id, session_id=lead.session_id)
+ token = _user_turn(lead.session_id, "Set up a Discord mod called Miro.")
+ try:
+ result = await tool.execute(
+ {
+ "kind": "identity",
+ "mode": "apply",
+ "request_quote": "Set up a Discord mod called Miro.",
+ "payload": {"name": "Miro"},
+ },
+ None,
+ )
+ finally:
+ current_chat_turn.reset(token)
+ assert not result.success
+ assert (await rt.roster.get(lead.agent_id)).name == lead.name
+
+
+async def test_a_request_from_another_chat_never_renames_an_agent(world): # noqa: F811
+ rt, _svc = world
+ lead = await _lead(rt)
+ fresh, _ = await rt.roster.create()
+ tool = ProposeChangeTool(rt, fresh.agent_id, session_id=fresh.session_id)
+ # The person is talking to Jarvis, not to the new agent.
+ token = _user_turn(lead.session_id, "Make the new agent Miro.")
+ try:
+ result = await tool.execute(
+ {
+ "kind": "identity",
+ "mode": "apply",
+ "request_quote": "Make the new agent Miro.",
+ "payload": {"name": "Miro"},
+ },
+ None,
+ )
+ finally:
+ current_chat_turn.reset(token)
+ assert not result.success
+ assert (await rt.roster.get(fresh.agent_id)).name == fresh.name
+
+
+async def test_an_explicit_rename_of_an_established_agent_applies_at_once(world): # noqa: F811
+ rt, svc = world
+ tool = ProposeChangeTool(rt, "mailbox", session_id="society:mailbox")
+ token = _user_turn("society:mailbox", "Call yourself Inbox from now on.")
+ try:
+ result = await tool.execute(
+ {
+ "kind": "identity",
+ "mode": "apply",
+ "request_quote": "Call yourself Inbox from now on.",
+ "payload": {"name": "Inbox"},
+ },
+ None,
+ )
+ finally:
+ current_chat_turn.reset(token)
+ assert result.success, result.error
+ assert (await rt.roster.get("mailbox")).name == "Inbox"
+ outcome = _identity_notices(svc, "proposal_resolved")[-1]
+ assert outcome["status"] == "applied" and outcome["previous"]["name"] == "Mailbox"
+
+
async def test_undo_data_includes_the_approval_rules(world): # noqa: F811
rt, _svc = world
item = await proposals.propose(
diff --git a/tests/unit/society/test_native_browser_window.py b/tests/unit/society/test_native_browser_window.py
index 07ad1823a2..8af4993675 100644
--- a/tests/unit/society/test_native_browser_window.py
+++ b/tests/unit/society/test_native_browser_window.py
@@ -298,6 +298,97 @@ def test_parking_preserves_monitor_geometry_and_does_not_activate(scene):
args = desktop.positions[-1]
assert args[:2] == (1, 1) # Owned Chrome, HWND_BOTTOM.
assert args[-1] & 0x13 == 0x13 # No move, no size, no activation.
+ assert desktop.alpha[1] == 0
+ assert desktop.styles[1] & 0x00080020 == 0x00080020
+ assert not desktop.styles[1] & 0x08040080
+
+
+def test_parking_hides_later_owned_popups_but_never_other_windows(scene):
+ window, desktop, _codec = scene
+ desktop.styles[1] = 0x40000
+ window.park()
+ assert not desktop.styles[1] & 0x40000
+ popup(window, desktop)
+ desktop.add(3, owner=0)
+ desktop.add(4, pid=900)
+ window._sync_windows()
+ assert desktop.alpha == {1: 0, 2: 0}
+ assert window.click_target(45, 15) == (2, 5, 5)
+
+
+def test_parked_windows_remain_eligible_for_capture_creation(scene):
+ window, desktop, _codec = scene
+ factory = window._capture_factory
+
+ def captureable(**kwargs):
+ # WGC rejects a no-activate window unless it is an explicit app window.
+ hwnd = kwargs["window_hwnd"]
+ style = desktop.styles.get(hwnd, 0)
+ assert not style & 0x80
+ assert not style & 0x08000000 or style & 0x40000
+ return factory(**kwargs)
+
+ window._capture_factory = captureable
+ desktop.styles[1] = 0x40080
+ window.park()
+ assert window._stop_capture(1)
+ window._start_capture(1)
+ desktop.styles[2] = 0x80
+ popup(window, desktop)
+ assert desktop.alpha == {1: 0, 2: 0}
+ assert all(desktop.styles[hwnd] & 0x80000 for hwnd in (1, 2))
+
+
+def test_failed_capture_creation_restores_desktop_transparency(scene):
+ window, desktop, _codec = scene
+ window.park()
+ assert window._stop_capture(1)
+
+ def refused(**_kwargs):
+ raise RuntimeError("capture item unavailable")
+
+ window._capture_factory = refused
+ with pytest.raises(RuntimeError, match="capture item unavailable"):
+ window._start_capture(1)
+ assert desktop.alpha[1] == 0
+ assert desktop.styles[1] & 0x80000
+ assert 1 not in window._captures and 1 not in window._capture_keys
+
+
+def test_parking_preserves_captured_pixels_and_manual_input(scene):
+ window, desktop, _codec = scene
+ window.park()
+ window._captures[1][0].arrive(Pixels(100, 80, 7))
+ frame = window.frame()
+ assert frame["bytes"] == b"owned-window-image"
+ window.input("text", {"text": "test"})
+ assert [row[2] for row in desktop.messages if row[1] == 0x102] == list(map(ord, "test"))
+
+
+def test_parking_fails_visibly_if_windows_cannot_hide_it(scene):
+ window, desktop, _codec = scene
+ desktop.user32.SetLayeredWindowAttributes = NativeCall(lambda *args: False)
+ with pytest.raises(RuntimeError, match="transparent"):
+ window.park()
+
+
+def test_repeated_window_events_do_not_rewrite_desktop_opacity(scene):
+ window, desktop, _codec = scene
+ window.park()
+ popup(window, desktop)
+ desktop.user32.SetLayeredWindowAttributes = NativeCall(
+ lambda *args: pytest.fail("Already hidden windows must not trigger new style events")
+ )
+ window.park()
+ window._sync_windows()
+
+
+def test_recycled_window_is_not_hidden(scene):
+ window, desktop, _codec = scene
+ desktop.windows[1]["pid"] = 900
+ with pytest.raises(RuntimeError, match="owned Chrome window"):
+ window.park()
+ assert desktop.styles == {} and desktop.alpha == {}
def test_old_offscreen_parking_position_is_recovered_without_activation(scene):
diff --git a/tests/unit/society/test_native_popup_recovery.py b/tests/unit/society/test_native_popup_recovery.py
new file mode 100644
index 0000000000..f01773657f
--- /dev/null
+++ b/tests/unit/society/test_native_popup_recovery.py
@@ -0,0 +1,159 @@
+"""A stopped popup stream cannot leave the entire browser preview unusable."""
+
+import asyncio
+import sys
+from types import SimpleNamespace
+
+import pytest
+import pytest_asyncio
+
+from jarvis.society.browser import native_window
+from tests.fakes.fake_native_browser import (
+ BlockingStopControl,
+ ImageCodec,
+ NativeDesktop,
+ Pixels,
+)
+
+pytestmark = pytest.mark.asyncio
+
+
+@pytest_asyncio.fixture
+async def scene(monkeypatch):
+ codec, desktop = ImageCodec(), NativeDesktop()
+ monkeypatch.setattr(native_window, "os", SimpleNamespace(name="nt"))
+ monkeypatch.setitem(sys.modules, "cv2", codec)
+ window = desktop.instance()
+ window._loop = asyncio.get_running_loop()
+ window._events = asyncio.Queue(maxsize=1)
+ window._event_task = asyncio.create_task(window._consume_window_events())
+ desktop.add(2)
+ window._sync_windows()
+ try:
+ yield window, desktop, codec
+ finally:
+ window.close()
+ await asyncio.sleep(0)
+ await asyncio.gather(window._event_task, return_exceptions=True)
+
+
+async def deliver_events():
+ # First deliver the native thread's scheduled callback, then its consumer.
+ await asyncio.sleep(0)
+ await asyncio.sleep(0)
+
+
+async def test_capture_stops_before_first_frame_then_recovers_without_replaying_input(scene):
+ window, desktop, codec = scene
+ old, control = window._captures[2]
+ old.events["on_closed"]()
+ await deliver_events()
+ with pytest.raises(RuntimeError, match="not visible"):
+ window.input("click", {"x": 10, "y": 15})
+ replacement = window._captures[2][0]
+ assert replacement is not old
+ assert (control.stopped, control.waited) == (1, 1)
+ assert desktop.messages == []
+
+ replacement.arrive(Pixels(60, 50, 7))
+ frame = window.frame()
+ assert codec.canvas.rows[10][40] == 7
+ window.input("click", {"x": 45, "y": 15, "geometry_id": frame["geometry_id"]})
+ assert desktop.messages[-1][0] == 2
+
+
+async def test_stopped_capture_drops_old_pixels_and_ignores_late_callbacks(scene):
+ window, _desktop, codec = scene
+ old = window._captures[2][0]
+ old.arrive(Pixels(60, 50, 2))
+ window.frame()
+ old.events["on_closed"]()
+ assert 2 not in window._images
+ await deliver_events()
+ window.frame()
+ assert codec.canvas.rows[10][40] == 1
+ replacement = window._captures[2][0]
+ replacement.arrive(Pixels(60, 50, 7))
+ old.arrive(Pixels(60, 50, 9))
+ old.events["on_closed"]()
+ await deliver_events()
+ window.frame()
+ assert window._captures[2][0] is replacement
+ assert codec.canvas.rows[10][40] == 7
+
+
+async def test_disappeared_popup_does_not_restart_and_main_remains_clickable(scene):
+ window, desktop, _codec = scene
+ old, control = window._captures[2]
+ desktop.windows[2]["visible"] = False
+ old.events["on_closed"]()
+ await deliver_events()
+ window.input("click", {"x": 10, "y": 15})
+ assert 2 not in window._captures
+ assert (control.stopped, control.waited) == (1, 1)
+ assert desktop.messages[-1][0] == 1
+
+
+async def test_repeated_capture_failure_is_bounded_and_does_not_send_input(scene):
+ window, desktop, _codec = scene
+ window._captures[2][0].events["on_closed"]()
+ await deliver_events()
+ window.frame()
+ replacement, control = window._captures[2]
+ replacement.events["on_closed"]()
+ await deliver_events()
+ with pytest.raises(RuntimeError, match="repeatedly stopped"):
+ window.input("click", {"x": 45, "y": 15})
+ assert 2 not in window._captures
+ assert window.failed
+ assert (control.stopped, control.waited) == (1, 1)
+ assert desktop.messages == []
+
+
+async def test_recovery_startup_failure_is_reported_instead_of_freezing(scene):
+ window, _desktop, _codec = scene
+
+ def unavailable_capture(**_options):
+ raise RuntimeError("Capture device unavailable")
+
+ window._capture_factory = unavailable_capture
+ window._captures[2][0].events["on_closed"]()
+ await deliver_events()
+ with pytest.raises(RuntimeError, match="could not restart"):
+ window.frame()
+ assert window.failed
+
+
+async def test_recovery_budget_resets_when_popup_closes_and_reopens(scene):
+ window, desktop, _codec = scene
+ window._captures[2][0].events["on_closed"]()
+ await deliver_events()
+ window.frame()
+ desktop.windows[2]["visible"] = False
+ window._sync_windows()
+ assert 2 not in window._restarted_popups
+ desktop.windows[2]["visible"] = True
+ window._sync_windows()
+ old = window._captures[2][0]
+ old.events["on_closed"]()
+ await deliver_events()
+ window.frame()
+ assert window._captures[2][0] is not old
+ assert not window.failed
+
+
+async def test_recovery_does_not_start_another_capture_while_cleanup_is_stuck(scene):
+ window, _desktop, _codec = scene
+ old = window._captures[2][0]
+ control = BlockingStopControl()
+ window._captures[2] = (old, control)
+ old.events["on_closed"]()
+ await deliver_events()
+ try:
+ with pytest.raises(RuntimeError, match="could not be released"):
+ window.frame()
+ assert control.entered.is_set()
+ assert 2 not in window._captures
+ assert window.failed
+ finally:
+ control.release.set()
diff --git a/tests/unit/society/test_role_naming.py b/tests/unit/society/test_role_naming.py
new file mode 100644
index 0000000000..e4b92fb4cc
--- /dev/null
+++ b/tests/unit/society/test_role_naming.py
@@ -0,0 +1,155 @@
+"""Placeholder agents acquire a role name in their first meaningful exchanges."""
+
+from dataclasses import replace
+
+import pytest
+
+from jarvis.agent_chat.jarvis_harness import COMPACT_MAX_CHARS, Identity
+from jarvis.agent_chat.runner_cli import _with_identity
+from jarvis.core.chat_turn import ChatTurn
+from jarvis.core.protocols import current_chat_turn
+from jarvis.society.agent_tools import ProposeChangeTool
+from jarvis.society.roster import has_placeholder_name
+from jarvis.society.surface import AGENT_NAMING_GUIDANCE, build_briefing
+from tests.unit.society.test_proposals_apply import world # noqa: F401
+
+
+async def _choose(rt, agent, name, *, direct=True, payload=None, session_id=None):
+ sid = session_id or agent.session_id
+ token = current_chat_turn.set(ChatTurn(sid, "t3", "Moderate my Discord.", direct, "trace"))
+ try:
+ return await ProposeChangeTool(rt, agent.agent_id, session_id=sid).execute(
+ {"kind": "identity", "payload": payload or {"name": name}},
+ None,
+ )
+ finally:
+ current_chat_turn.reset(token)
+
+
[email protected]("name", ["New Agent", "New Agent 2", "New Bot", "New Bot 12"])
+async def test_placeholder_survives_role_storage_and_can_be_named_without_confirmation(world, name): # noqa: F811, E501
+ rt, svc = world
+ agent, _ = await rt.roster.create()
+ agent = await rt.roster.update(
+ agent.agent_id,
+ {
+ "name": name,
+ "description": "Moderate my Discord and draft Gmail reports.",
+ },
+ )
+ assert has_placeholder_name(agent)
+ assert AGENT_NAMING_GUIDANCE in build_briefing(agent, [], [])
+
+ result = await _choose(rt, agent, "Discord-Mod")
+ assert result.success, result.error
+ assert result.output["applied"]
+ updated = await rt.roster.get(agent.agent_id)
+ assert updated.name == "Discord-Mod"
+ assert updated.session_id == agent.session_id
+ assert (updated.title, updated.description) == (agent.title, agent.description)
+ assert updated.focus == agent.focus
+ assert updated.approval_rules == agent.approval_rules
+ assert (await rt.store.get_agent_row(agent.agent_id))["name"] == "Discord-Mod"
+ assert not has_placeholder_name(updated)
+ assert AGENT_NAMING_GUIDANCE not in build_briefing(updated, [], [])
+ notice = svc.notices[-1][1]
+ assert notice["agent_name"] == "Discord-Mod"
+ assert notice["previous"]["name"] == name
+ assert not any(p.get("kind") == "proposal" for _, p in svc.notices)
+
+
[email protected]("role_turn", [1, 2, 3])
+async def test_first_three_turns_keep_naming_guidance_until_identity_is_saved(world, role_turn): # noqa: F811, E501
+ rt, _ = world
+ agent, _ = await rt.roster.create()
+ for turn in range(1, role_turn + 1):
+ text = build_briefing(agent, [], [])
+ identity = Identity(session_id=agent.session_id, text=text, compact=text)
+ prompt = _with_identity(
+ "Moderate my Discord." if turn == role_turn else "Hello.",
+ identity,
+ "vendor-session" if turn > 1 else None,
+ )
+ assert AGENT_NAMING_GUIDANCE in prompt
+ assert "society_propose_change" in prompt
+ assert (await rt.roster.get(agent.agent_id)).name == "New Agent"
+ result = await _choose(rt, agent, "Discord-Mod")
+ assert result.success
+ updated = await rt.roster.get(agent.agent_id)
+ text = build_briefing(updated, [], [])
+ prompt = _with_identity(
+ "Another task.", Identity(agent.session_id, text, text), "vendor-session"
+ )
+ assert "## You are Discord-Mod" in prompt
+ assert "## Choosing your name" not in prompt
+ assert "earlier placeholder instructions no longer apply" in prompt
+
+
[email protected]("direct,foreign", [(False, False), (True, True)])
+async def test_first_name_requires_the_agents_own_direct_user_chat(world, direct, foreign): # noqa: F811, E501
+ rt, _ = world
+ agent, _ = await rt.roster.create(description="Moderate Discord.")
+ result = await _choose(
+ rt, agent, "Discord-Mod", direct=direct, session_id="society:mailbox" if foreign else None
+ )
+ assert result.success and result.output["status"] == "pending"
+ assert (await rt.roster.get(agent.agent_id)).name == "New Agent"
+
+
+async def test_naming_an_existing_role_does_not_auto_apply_changed_instructions(world): # noqa: F811
+ rt, _ = world
+ agent, _ = await rt.roster.create(description="Moderate Discord.")
+ result = await _choose(
+ rt,
+ agent,
+ "Discord-Mod",
+ payload={
+ "name": "Discord-Mod",
+ "description": "Also send Gmail messages.",
+ },
+ )
+ assert result.output["status"] == "pending"
+ assert (await rt.roster.get(agent.agent_id)).description == agent.description
+
+
[email protected]("name", ["Scout", "New Agent", "New Bot 2"])
+async def test_explicit_names_are_not_treated_as_automatic_placeholders(world, name): # noqa: F811
+ rt, _ = world
+ agent, _ = await rt.roster.create(name=name)
+ assert not has_placeholder_name(agent)
+ assert AGENT_NAMING_GUIDANCE not in build_briefing(agent, [], [])
+ result = await _choose(rt, agent, "Discord-Mod")
+ assert result.output["status"] == "pending"
+ assert (await rt.roster.get(agent.agent_id)).name == name
+
+
+async def test_chosen_name_is_not_replaced_on_later_role_tasks(world): # noqa: F811
+ rt, _ = world
+ agent, _ = await rt.roster.create()
+ assert (await _choose(rt, agent, "Discord-Mod")).success
+ agent = await rt.roster.get(agent.agent_id)
+ result = await _choose(rt, agent, "Researcher")
+ assert result.output["status"] == "pending"
+ assert (await rt.roster.get(agent.agent_id)).name == "Discord-Mod"
+
+
+async def test_taken_role_name_can_be_retried_without_losing_chat(world): # noqa: F811
+ rt, _ = world
+ agent, _ = await rt.roster.create(description="Handle email.")
+ assert not (await _choose(rt, agent, "Mailbox")).success
+ assert (await rt.roster.get(agent.agent_id)).name == "New Agent"
+ assert (await _choose(rt, agent, "Mail Assistant")).success
+ assert (await rt.roster.get(agent.agent_id)).session_id == agent.session_id
+
+
+async def test_compact_resume_keeps_name_guidance_ahead_of_long_memory(world): # noqa: F811
+ rt, _ = world
+ agent, _ = await rt.roster.create(description="Research sources.")
+ text = build_briefing(agent, [], [], memory="## Your memory\n" + "A fact.\n" * 3000)
+ from jarvis.agent_chat.jarvis_harness import society_memory_refresh
+
+ refreshed = society_memory_refresh(text, compact=True)
+ assert AGENT_NAMING_GUIDANCE in refreshed
+ assert len(refreshed) < COMPACT_MAX_CHARS
+ assert "New Agent" in refreshed
+ assert not has_placeholder_name(replace(agent, tier="lead"))
diff --git a/tests/unit/society/test_surface.py b/tests/unit/society/test_surface.py
index 41327490bd..541f5619e9 100644
--- a/tests/unit/society/test_surface.py
+++ b/tests/unit/society/test_surface.py
@@ -14,7 +14,11 @@ from jarvis.agent_chat.store import AgentChatStore
from jarvis.core.bus import EventBus
from jarvis.core.config import SafetyConfig
from jarvis.core.protocols import ToolResult
-from jarvis.core.response_style import KEEP_GOING_ON_TOOL_FAILURE, TASK_EXECUTION_GUIDANCE
+from jarvis.core.response_style import (
+ CREDENTIAL_GUIDANCE,
+ KEEP_GOING_ON_TOOL_FAILURE,
+ TASK_EXECUTION_GUIDANCE,
+)
from jarvis.safety.approval import ApprovalWorkflow
from jarvis.safety.risk_tier import RiskTierEvaluator
from jarvis.safety.tool_executor import ToolExecutor
@@ -95,6 +99,7 @@ async def test_task_execution_contract_survives_long_memory_and_compact_transpor
compact = compact_identity(briefing + "\n\n" + SYSTEM_PREAMBLE)
assert TASK_EXECUTION_GUIDANCE in compact
assert KEEP_GOING_ON_TOOL_FAILURE in compact
+ assert CREDENTIAL_GUIDANCE in compact
assert len(compact) <= 16000
@@ -168,6 +173,7 @@ async def test_tools_and_filter_follow_the_roster_row(rt: SocietyRuntime, tmp_pa
"society_routines",
"society_invoke_routine",
"society_ask_user",
+ "society_request_credential",
"society_share_template",
*FOLDER,
}
@@ -190,6 +196,7 @@ async def test_tools_and_filter_follow_the_roster_row(rt: SocietyRuntime, tmp_pa
"society_routines",
"society_invoke_routine",
"society_ask_user",
+ "society_request_credential",
"society_share_template",
*FOLDER,
}
diff --git a/tests/unit/society/test_template_approval_defaults.py b/tests/unit/society/test_template_approval_defaults.py
new file mode 100644
index 0000000000..647019baa3
--- /dev/null
+++ b/tests/unit/society/test_template_approval_defaults.py
@@ -0,0 +1,37 @@
+"""Imported agents use the same approval default as manually created agents."""
+
+from pathlib import Path
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis.society.runtime import SocietyRuntime
+from jarvis.ui.web.society_routes import router
+
+
[email protected]("wrapped", [False, True])
+def test_import_uses_bypass_and_preserves_explicit_rules(tmp_path: Path, wrapped: bool):
+ runtime = SocietyRuntime(tmp_path, seed_starter_team=False)
+ app = FastAPI()
+ app.include_router(router)
+ app.state.society_factory = lambda: runtime
+ template = {
+ "schema": 1,
+ "name": "Imported helper",
+ "instructions": "Maintain the task list.",
+ "require_approval": ["plugin:gmail:send"],
+ "denies": ["core:shell"],
+ }
+ payload = {"kind": "agent", "agent": template} if wrapped else template
+ with TestClient(app) as client:
+ manual = client.post("/api/society/agents", json={"name": "Manual helper"})
+ assert manual.status_code == 200, manual.text
+ imported = client.post("/api/society/templates/install", json={"template": payload})
+ assert imported.status_code == 200, imported.text
+ agent = imported.json()["agent"]
+ assert agent["approval_mode"] == manual.json()["agent"]["approval_mode"] == "bypass"
+ assert agent["approval_rules"]["require_approval"] == ["plugin:gmail:send"]
+ assert agent["denies"] == ["core:shell"]
+ saved = client.get(f"/api/society/agents/{agent['agent_id']}").json()["agent"]
+ assert saved["approval_mode"] == "bypass"
diff --git a/tests/unit/test_agent_resets.py b/tests/unit/test_agent_resets.py
new file mode 100644
index 0000000000..a00b95e10d
--- /dev/null
+++ b/tests/unit/test_agent_resets.py
@@ -0,0 +1,234 @@
+"""Reset receipts, account boundaries and unknown-versus-empty inventory."""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from jarvis import agent_resets
+from jarvis.agent_accounts import AgentAccount
+from tests.fakes.fake_reset_rpc import FakeResetRpc
+
+
+def account(tmp_path: Path) -> AgentAccount:
+ (tmp_path / "auth.json").write_text(json.dumps({"tokens": {"account_id": "seat-1"}}))
+ return AgentAccount("codex:test", "codex", "Test seat", tmp_path)
+
+
+def payload(count=2, details=None):
+ return {
+ "rateLimitResetCredits": {"availableCount": count, "credits": details},
+ "rateLimits": {
+ "limitId": "codex",
+ "planType": "pro",
+ "secondary": {
+ "usedPercent": 81,
+ "windowDurationMins": 10080,
+ "resetsAt": 1900000000,
+ },
+ },
+ }
+
+
+def rpc(monkeypatch):
+ from jarvis.agent_chat import native_control, runner_cli
+
+ fake = FakeResetRpc(payload())
+ monkeypatch.setattr(native_control, "GoalRpc", fake.factory)
+ monkeypatch.setattr(runner_cli, "codex_argv_prefix", lambda: ["fake-codex"])
+ return fake
+
+
[email protected]("count,rows,can_redeem", [(0, [], False), (2, None, True), (2, [], False)])
+def test_authoritative_count_and_missing_details(account, count, rows, can_redeem):
+ result = agent_resets._snapshot(account, payload(count, rows), "key")
+ assert result["available_count"] == count
+ assert result["credits"] == rows
+ assert result["can_redeem"] is can_redeem
+ assert result["usage"]["windows"][0]["kind"] == "weekly"
+
+
[email protected]("summary", [None, {}, {"availableCount": True}, {"availableCount": -1}])
+def test_missing_or_invalid_inventory_is_not_zero(account, summary):
+ result = agent_resets._snapshot(account, {"rateLimitResetCredits": summary}, "key")
+ assert result["status"] == "unavailable"
+ assert result["available_count"] is None
+ assert not result["can_redeem"]
+
+
+ "status,kind,expiry,enabled",
+ [
+ ("available", "codexRateLimits", None, True),
+ ("redeemed", "codexRateLimits", None, False),
+ ("available", "unknown", None, False),
+ ("available", "codexRateLimits", 1, False),
+ ("available", "codexRateLimits", "bad", False),
+ ],
+)
+def test_only_known_available_unexpired_details_can_be_used(account, status, kind, expiry, enabled):
+ result = agent_resets._snapshot(
+ account,
+ payload(
+ 5,
+ [
+ {
+ "id": "credit-1",
+ "status": status,
+ "resetType": kind,
+ "expiresAt": expiry,
+ "title": {"malformed": True},
+ }
+ ],
+ ),
+ "key",
+ )
+ assert result["available_count"] == 5 # Details may be capped.
+ assert result["can_redeem"] is enabled
+ assert result["credits"][0]["title"] is None
+
+
+async def test_read_uses_selected_seat_and_never_starts_a_turn(account, rpc, monkeypatch):
+ monkeypatch.setenv("OPENAI_API_KEY", "must-not-inherit")
+ monkeypatch.setenv("CODEX_HOME", "wrong-seat")
+ result = await agent_resets.read_resets(account)
+ assert result["status"] == "ok"
+ assert rpc.options[1]["CODEX_HOME"] == str(account.config_dir)
+ assert "OPENAI_API_KEY" not in rpc.options[1]
+ assert 'cli_auth_credentials_store="file"' in rpc.options[0]
+ assert [method for method, _ in rpc.calls] == [
+ "initialize",
+ "initialized",
+ "account/read",
+ "account/rateLimits/read",
+ ]
+ assert rpc.closed
+
+
[email protected]("outcome", ["reset", "alreadyRedeemed", "nothingToReset", "noCredit"])
+async def test_receipt_and_same_key_replay_without_inventory_precheck(account, rpc, outcome):
+ rpc.outcome = outcome
+ key = agent_resets._identity(account, rpc.account)
+ for _ in range(2):
+ result = await agent_resets.consume_reset(
+ account, attempt="same-uuid", credit_id="credit-1", account_key=key
+ )
+ assert result["outcome"] == outcome
+ redemptions = [params for method, params in rpc.calls if method.endswith("/consume")]
+ assert redemptions == [{"idempotencyKey": "same-uuid", "creditId": "credit-1"}] * 2
+ assert rpc.calls.index(("account/rateLimits/read", {})) > rpc.calls.index(
+ ("account/rateLimitResetCredit/consume", redemptions[0])
+ )
+
+
+async def test_changed_login_cannot_consume(account, rpc):
+ with pytest.raises(agent_resets.ResetError) as caught:
+ await agent_resets.consume_reset(account, attempt="uuid", credit_id=None, account_key="old")
+ assert caught.value.code == "account_changed"
+ assert not any(method.endswith("/consume") for method, _ in rpc.calls)
+ assert rpc.closed
+
+
+async def test_refresh_failure_preserves_confirmed_receipt(account, rpc):
+ rpc.failure = "account/rateLimits/read"
+ result = await agent_resets.consume_reset(
+ account,
+ attempt="uuid",
+ credit_id=None,
+ account_key=agent_resets._identity(account, rpc.account),
+ )
+ assert result["outcome"] == "reset"
+ assert result["resets"]["status"] == "unavailable"
+
+
+async def test_lost_response_never_retries_and_does_not_leak_error(account, rpc):
+ rpc.failure = "account/rateLimitResetCredit/consume"
+ with pytest.raises(agent_resets.ResetError) as caught:
+ await agent_resets.consume_reset(
+ account,
+ attempt="uuid",
+ credit_id=None,
+ account_key=agent_resets._identity(account, rpc.account),
+ )
+ assert "private" not in str(caught.value)
+ assert len([m for m, _ in rpc.calls if m.endswith("/consume")]) == 1
+
+
+ "provider,status",
+ [("claude", "browser"), ("grok-build", "unsupported"), ("future", "unsupported")],
+)
+async def test_unsupported_providers_do_not_invent_inventory(account, rpc, provider, status):
+ other = AgentAccount(provider, provider, "Test", account.config_dir)
+ result = await agent_resets.read_resets(other)
+ assert result["status"] == status
+ assert result["available_count"] is None
+ assert rpc.calls == []
+ with pytest.raises(agent_resets.ResetError):
+ await agent_resets.consume_reset(other, attempt="uuid", credit_id=None, account_key="key")
+
+
+def test_routes_validate_confirmation_and_account_and_mark_cli_dangerous(account, rpc, monkeypatch):
+ from jarvis.ui.web import agent_accounts_routes as routes
+
+ monkeypatch.setattr(
+ routes.agent_accounts, "resolve", lambda value: account if value == account.id else None
+ )
+ app = FastAPI()
+ app.include_router(routes.router)
+ client = TestClient(app)
+ url = f"/api/agent-accounts/{account.id}/resets"
+ inventory = client.get(url).json()
+ request = {
+ "confirmed": False,
+ "idempotency_key": "f453a2b8-8ad7-449f-b7c9-27a1f3483934",
+ "account_key": inventory["account_key"],
+ }
+ assert client.post(url + "/consume", json=request).status_code == 422
+ assert client.post(url + "/consume", json={**request, "confirmed": "true"}).status_code == 422
+ assert client.get("/api/agent-accounts/missing/resets").status_code == 404
+ assert not any(m.endswith("/consume") for m, _ in rpc.calls)
+ response = client.post(url + "/consume", json={**request, "confirmed": True})
+ assert response.status_code == 200
+ assert response.json()["outcome"] == "reset"
+ schema = app.openapi()["paths"]["/api/agent-accounts/{account_id}/resets/consume"]["post"]
+ assert schema["x-jarvis-dangerous"] is True
+
+
+def test_frontend_reset_contract_matches_serialized_inventory_and_request(account):
+ from jarvis.ui.web.agent_accounts_routes import ResetConsumeRequest
+
+ source = (
+ (
+ Path(__file__).resolve().parents[2]
+ / "jarvis/ui/web/frontend/src/lib/agentResetsApi.ts"
+ )
+ .resolve()
+ .read_text(encoding="utf-8")
+ )
+
+ def fields(name):
+ body = re.search(rf"export interface {name} \{{(.*?)\n\}}", source, re.S).group(1)
+ return set(re.findall(r"^ (\w+):", body, re.M))
+
+ assert fields("AccountResets") == set(agent_resets._empty(account, "unsupported"))
+ assert fields("ResetAttempt") == set(ResetConsumeRequest.model_fields) - {"confirmed"}
+ snapshot = agent_resets._snapshot(
+ account,
+ payload(1, [{"id": "one", "status": "available", "resetType": "codexRateLimits"}]),
+ "key",
+ )
+ assert fields("ResetCredit") == set(snapshot["credits"][0])
diff --git a/tests/unit/ui/jarvisbar/test_mode_parity.py b/tests/unit/ui/jarvisbar/test_mode_parity.py
index 347acd49f9..1f3abc5290 100644
--- a/tests/unit/ui/jarvisbar/test_mode_parity.py
+++ b/tests/unit/ui/jarvisbar/test_mode_parity.py
@@ -146,7 +146,7 @@ def test_the_null_surface_swallows_every_mode(mode: str) -> None:
#: Every look ``renderer.render`` knows how to draw. Derived from the mode
#: vocabulary so a new mode either resolves to an existing look or brings its
#: own — it can never resolve to a look nothing draws.
-_RENDERABLE_LOOKS = ("idle", "speak", "think") + modes.NOTICE_MODES
+_RENDERABLE_LOOKS = ("idle", "speak", "think") + modes.NOTICE_MODES + modes.CONNECT_MODES
@pytest.mark.parametrize("mode", modes.MODES)
diff --git a/tests/unit/ui/test_connect_indicator.py b/tests/unit/ui/test_connect_indicator.py
new file mode 100644
index 0000000000..2e43b0e3e6
--- /dev/null
+++ b/tests/unit/ui/test_connect_indicator.py
@@ -0,0 +1,322 @@
+"""The "connecting" look: from the wake word until the realtime provider takes the call.
+
+The supervisor has a ``CONNECTING`` state for the realtime handshake, but every
+desktop surface used to ignore it and kept showing the listening look while
+the provider had not accepted a single frame. These pin the whole chain: the
+bridge paints ``connect`` and greets on connecting, the shared strip (Jarvis
+Bar and pet) runs a loading loop and then a one-shot "connected" flourish that
+settles into the listening look, the pet figure is busy meanwhile, the voice
+orb circles a comet, and the connecting call can still be hung up.
+"""
+
+from __future__ import annotations
+
+import math
+import sys
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from jarvis.core.events import SystemStateChanged, VoiceSessionStarted
+from jarvis.ui.jarvisbar import interaction, modes, renderer
+from jarvis.ui.pets.state_machine import PetStateMachine
+
+_REPO_ROOT = Path(__file__).resolve().parents[3]
+if str(_REPO_ROOT) in sys.path:
+ sys.path.remove(str(_REPO_ROOT))
+sys.path.insert(0, str(_REPO_ROOT))
+sys.modules.pop("ui", None)
+
+try: # noqa: SIM105 — intentional try-import for the discovery quirk
+ from ui.orb import controls # type: ignore[import-not-found]
+ from ui.orb.bus_bridge import OrbBusBridge # type: ignore[import-not-found]
+ from ui.orb.voice_orb import VoiceOrbRenderer # type: ignore[import-not-found]
+except ModuleNotFoundError: # pragma: no cover
+ pytest.skip("ui.orb not available on the pytest PYTHONPATH", allow_module_level=True)
+
+KEY = (255, 0, 255)
+
+
+class _FakeBus:
+ def subscribe(self, *_args, **_kwargs) -> None:
+ pass
+
+
+class _FakeOrb:
+ def __init__(self) -> None:
+ self.calls: list[tuple[str, object]] = []
+
+ def show(self, mode: str = "listen") -> None:
+ self.calls.append(("show", mode))
+
+ def hide(self) -> None:
+ self.calls.append(("hide", None))
+
+ def set_level(self, level: float) -> None:
+ self.calls.append(("set_level", level))
+
+ def play_animation(self, name: str) -> None:
+ self.calls.append(("play_animation", name))
+
+ def stop_animation(self, name: str) -> None:
+ self.calls.append(("stop_animation", name))
+
+ def show_listening_transcript(self, text: str = "", duration_ms: int = 30000) -> None:
+ self.calls.append(("show_listening_transcript", text))
+
+ def hide_comment(self) -> None:
+ self.calls.append(("hide_comment", None))
+
+
+# -- vocabulary ---------------------------------------------------------------
+
+
+def test_connect_is_a_call_mode_but_not_a_plain_voice_mode() -> None:
+ assert "connect" in modes.MODES
+ assert "connect" in modes.ACTIVE_VOICE_MODES
+ assert "connect" not in modes.VOICE_MODES
+ assert "connect" not in modes.DICTATION_MODES
+
+
+def test_a_connecting_call_can_be_hung_up_from_the_talk_control_and_the_phone() -> None:
+ width = controls.pet_strip_size()[0]
+ for action in ("orb", "call"):
+ layout = controls.pet_strip_layout()
+ if action == "call":
+ x = layout.call[0]
+ else:
+ x = next((x0 + x1) / 2 for name, x0, x1 in layout.slots if name == "orb")
+ assert interaction.resolve_click(x, width, "connect") == "hangup"
+
+
+# -- bridge -------------------------------------------------------------------
+
+
+async def test_connecting_state_paints_the_loading_loop() -> None:
+ orb = _FakeOrb()
+ bridge = OrbBusBridge(bus=_FakeBus(), orb=orb, idle_animations_enabled=False) # type: ignore[arg-type]
+ await bridge._on_session_started(VoiceSessionStarted(session_id="s1")) # noqa: SLF001
+ orb.calls.clear()
+
+ await bridge._on_state(SystemStateChanged(new_state="CONNECTING", previous="LISTENING")) # noqa: SLF001
+
+ assert ("show", "connect") in orb.calls
+ assert bridge._current_voice_mode() == "connect" # noqa: SLF001
+
+
+async def test_connected_call_returns_to_listening_with_a_greeting() -> None:
+ orb = _FakeOrb()
+ bridge = OrbBusBridge(bus=_FakeBus(), orb=orb, idle_animations_enabled=False) # type: ignore[arg-type]
+ await bridge._on_session_started(VoiceSessionStarted(session_id="s1")) # noqa: SLF001
+ await bridge._on_state(SystemStateChanged(new_state="CONNECTING", previous="LISTENING")) # noqa: SLF001
+ orb.calls.clear()
+
+ await bridge._on_state(SystemStateChanged(new_state="LISTENING", previous="CONNECTING")) # noqa: SLF001
+
+ assert ("show", "listen") in orb.calls
+ assert ("play_animation", "wave") in orb.calls
+
+
+async def test_a_failed_handshake_never_leaves_the_loading_loop_behind() -> None:
+ orb = _FakeOrb()
+ bridge = OrbBusBridge( # type: ignore[arg-type]
+ bus=_FakeBus(), orb=orb, idle_animations_enabled=False, hide_on_idle=False
+ )
+ await bridge._on_session_started(VoiceSessionStarted(session_id="s1")) # noqa: SLF001
+ await bridge._on_state(SystemStateChanged(new_state="CONNECTING", previous="LISTENING")) # noqa: SLF001
+ orb.calls.clear()
+
+ await bridge._on_state(SystemStateChanged(new_state="IDLE", previous="CONNECTING")) # noqa: SLF001
+
+ assert orb.calls[-1] == ("show", "idle") or ("show", "idle") in orb.calls
+
+
+# -- the shared strip (Jarvis Bar + pet) --------------------------------------
+
+
+def test_no_audio_signal_can_repaint_the_loading_loop() -> None:
+ """The opening words are buffered, not heard: a live level stays a loop."""
+ for playback in (False, True):
+ for seconds in (0.0, 99.0):
+ look = renderer.visual_mode("connect", seconds, hold_s=0.4, playback_active=playback)
+ assert look == "connect"
+
+
+def _arc_spread(stroke: controls.MorphStroke) -> float:
+ """Distance from the talk control's centre, averaged over a stroke."""
+ return sum((x * x + y * y) ** 0.5 for x, y in stroke.points) / len(stroke.points)
+
+
+def test_the_strokes_themselves_bend_into_the_loop() -> None:
+ """No extra ring: the three strokes become the loading loop."""
+ assert controls.morph_strokes(controls.PetStripState(motion="voice")) is None
+ first = controls.morph_strokes(controls.PetStripState(motion="connect", phase=0))
+ assert first is None # the very first step is still the plain row
+ bent = controls.morph_strokes(
+ controls.PetStripState(motion="connect", phase=controls.PET_MORPH_STEPS)
+ )
+ assert bent is not None and len(bent) == controls.PET_INDICATOR_BARS
+ for stroke in bent:
+ # Every point of a fully bent stroke lies on the loop's ring.
+ for x, y in stroke.points:
+ assert (x * x + y * y) ** 0.5 == pytest.approx(controls.PET_LOOP_R, abs=1e-6)
+ half = controls.morph_strokes(
+ controls.PetStripState(motion="connect", phase=controls.PET_MORPH_STEPS // 2)
+ )
+ assert half is not None
+ assert 0.0 < controls.morph_amount(
+ controls.PetStripState(motion="connect", phase=controls.PET_MORPH_STEPS // 2)
+ ) < 1.0
+
+
+def test_the_bent_loop_spins_and_repeats_every_third_of_a_turn() -> None:
+ frames = {
+ controls.render_pet_strip(
+ controls.PetStripState(
+ jarvis_bar=True,
+ active=True,
+ motion="connect",
+ phase=controls.PET_MORPH_STEPS,
+ spin=step,
+ )
+ ).tobytes()
+ for step in (0, 3, 6, 9)
+ }
+ assert len(frames) == 4
+ period = controls.PET_SPIN_PERIOD_S
+ assert controls.spin_step(0.15) == controls.spin_step(0.15 + period)
+
+
+def test_a_live_voice_lengthens_the_loop_arcs_without_closing_the_ring() -> None:
+ quiet = controls.morph_strokes(
+ controls.PetStripState(motion="connect", phase=controls.PET_MORPH_STEPS, level=0)
+ )
+ loud = controls.morph_strokes(
+ controls.PetStripState(
+ motion="connect", phase=controls.PET_MORPH_STEPS, level=controls.PET_LEVEL_STEPS
+ )
+ )
+ assert quiet is not None and loud is not None
+
+ def span(stroke: controls.MorphStroke) -> float:
+ """The arc's angular length in degrees, from its centreline."""
+ pts = stroke.points
+ length = sum(
+ math.dist(pts[k], pts[k + 1]) for k in range(len(pts) - 1)
+ )
+ return math.degrees(length / controls.PET_LOOP_R)
+
+ assert span(loud[0]) > span(quiet[0])
+ assert span(loud[0]) < 115.0 # three arcs still read as a loop with gaps
+
+
+def test_the_connected_flourish_flashes_green_and_ends_as_the_plain_row() -> None:
+ mid = controls.morph_strokes(
+ controls.PetStripState(
+ motion="connected", phase=controls.PET_CONNECTED_PHASES // 2, spin=5
+ )
+ )
+ assert mid is not None and max(s.green for s in mid) > 0.5
+ last = controls.PetStripState(
+ motion="connected", phase=controls.PET_CONNECTED_PHASES - 1, spin=5
+ )
+ # The last step is exactly the resting row, so nothing jumps after it.
+ assert controls.morph_strokes(last) is None
+ rest = controls.PetStripState(motion="voice", level=0)
+ assert controls.indicator_bars(last) == controls.indicator_bars(rest)
+
+
+def test_the_flourish_unbends_from_where_the_loop_stopped_spinning() -> None:
+ """The arcs turn on from their handover angle; they never snap back."""
+ for spin in (0, 4, 11):
+ state = controls.PetStripState(motion="connected", phase=0, spin=spin)
+ before = controls.morph_strokes(
+ controls.PetStripState(motion="connect", phase=controls.PET_MORPH_STEPS, spin=spin)
+ )
+ after = controls.morph_strokes(state)
+ assert before is not None and after is not None
+ for a, b in zip(before, after, strict=True):
+ assert _arc_spread(b) == pytest.approx(_arc_spread(a), abs=0.03)
+
+
+def test_the_timeline_bends_spins_and_flourishes_on_the_surface_clock() -> None:
+ tl = controls.ConnectTimeline()
+ assert tl.look("listen", 0.0) is None
+ tl.note_mode("listen", "connect", 1.0)
+ early = tl.look("connect", 1.05)
+ assert early is not None and early.motion == "connect"
+ assert early.phase < controls.PET_MORPH_STEPS
+ later = tl.look("connect", 2.0)
+ assert later is not None and later.phase == controls.PET_MORPH_STEPS
+ tl.note_mode("connect", "listen", 2.0)
+ flourish = tl.look("listen", 2.1)
+ assert flourish is not None and flourish.motion == "connected"
+ assert flourish.spin == controls.spin_step(1.0)
+ assert tl.look("listen", 2.01 + controls.PET_CONNECTED_S) is None
+
+
+def test_hanging_up_while_connecting_plays_no_flourish() -> None:
+ tl = controls.ConnectTimeline()
+ tl.note_mode("listen", "connect", 0.0)
+ tl.note_mode("connect", "idle", 0.5)
+ assert tl.look("idle", 0.6) is None
+ tl.note_mode("idle", "listen", 0.7)
+ assert tl.look("listen", 0.8) is None
+
+
+def test_the_bar_renders_the_flourish_then_the_plain_listening_look() -> None:
+ bar = renderer.JarvisBarRenderer()
+ tl = controls.ConnectTimeline()
+ tl.note_mode("listen", "connect", 0.0)
+ tl.note_mode("connect", "listen", 1.0)
+ flourish = bar.render(
+ 0.0, "listen", 0.0, surface_mode="listen", connect_look=tl.look("listen", 1.3)
+ )
+ plain = bar.render(0.0, "listen", 0.0, surface_mode="listen")
+ assert flourish.tobytes() != plain.tobytes()
+ late = bar.render(0.0, "listen", 0.0, surface_mode="listen", connect_look=tl.look("listen", 9))
+ assert late.tobytes() == plain.tobytes()
+
+
+# -- pet figure + voice orb ---------------------------------------------------
+
+
+def test_the_pet_is_busy_while_connecting_and_listens_once_connected() -> None:
+ machine = PetStateMachine(clock=lambda: 0.0)
+ machine.on_mode("connect")
+ assert machine.state() == "thinking"
+ machine.on_mode("listen")
+ assert machine.state() == "listening"
+
+
+def _outside_sphere_pixels(frame: np.ndarray, radius_share: float = 0.8) -> int:
+ size = frame.shape[0]
+ yy, xx = np.mgrid[0:size, 0:size]
+ r = np.hypot(xx - size / 2 + 0.5, yy - size / 2 + 0.5) / (size / 2)
+ ring = (r > radius_share) & (r < 0.98)
+ return int((np.any(frame != KEY, axis=-1) & ring).sum())
+
+
+def test_the_voice_orb_circles_a_comet_while_connecting() -> None:
+ connecting = VoiceOrbRenderer(size=160, color_key=KEY)
+ idle = VoiceOrbRenderer(size=160, color_key=KEY)
+ for step in range(12):
+ a = np.asarray(connecting.render(step * 0.05, "connect", None))
+ b = np.asarray(idle.render(step * 0.05, "idle", None))
+ assert _outside_sphere_pixels(a) > _outside_sphere_pixels(b) + 40
+
+
+def test_the_voice_orb_sends_one_ring_out_when_the_call_connects() -> None:
+ orb = VoiceOrbRenderer(size=160, color_key=KEY)
+ t = 0.0
+ for _ in range(10):
+ t += 0.05
+ orb.render(t, "connect", None)
+ t += 0.05
+ orb.render(t, "listen", 0.0)
+ assert orb._connected_age < controls.PET_CONNECTED_S # noqa: SLF001
+ for _ in range(30):
+ t += 0.05
+ orb.render(t, "listen", 0.0)
+ assert orb._connected_age == float("inf") # noqa: SLF001
diff --git a/tests/unit/ui/test_pet_right_click.py b/tests/unit/ui/test_pet_right_click.py
new file mode 100644
index 0000000000..830f3ff938
--- /dev/null
+++ b/tests/unit/ui/test_pet_right_click.py
@@ -0,0 +1,114 @@
+"""The pet's right-click hides the whole surface until its global shortcut."""
+
+from __future__ import annotations
+
+import pytest
+
+from jarvis.core.bus import EventBus
+from jarvis.core.events import PetVisibilityToggleRequested
+from jarvis.ui.jarvisbar import host
+from jarvis.ui.jarvisbar.modes import MODES
+from jarvis.ui.jarvisbar.subprocess_overlay import SubprocessMascotOverlay
+from ui.orb.bus_bridge import OrbBusBridge
+from ui.orb.overlay import OrbOverlay, PetControlStrip
+
+
+class _Window:
+ def __init__(self) -> None:
+ self.visible = True
+
+ def withdraw(self) -> None:
+ self.visible = False
+
+ hide = withdraw
+
+ def deiconify(self) -> None:
+ self.visible = True
+
+ def winfo_ismapped(self) -> bool:
+ return self.visible
+
+ def lift(self) -> None:
+ pass
+
+
+def pet(monkeypatch):
+ surface = OrbOverlay(style="pet", pet_id="ember")
+ surface._mac_transparent = False
+ surface._root = _Window()
+ surface._controls = _Window()
+ surface._notices = _Window()
+ surface._thought = _Window()
+ monkeypatch.setattr(surface, "_enqueue_ui", lambda fn: fn())
+ monkeypatch.setattr(surface, "_ensure_frame_loop", lambda: None)
+ monkeypatch.setattr(surface, "_reassert_topmost", lambda **kwargs: None)
+ monkeypatch.setattr(surface, "_sync_controls_visibility", lambda: None)
+ return surface
+
+
+def test_right_click_hides_figure_strip_and_cards_without_opening_window(pet) -> None:
+ opened = []
+ pet.set_on_show_window(lambda: opened.append(True))
+ pet._on_right_click()
+ assert pet.pet_user_hidden
+ assert not opened
+ assert all(not window.visible for window in (
+ pet._root, pet._controls, pet._notices, pet._thought,
+ ))
+
+
+async def test_voice_updates_stay_hidden_until_existing_hotkey_event(pet) -> None:
+ bus = EventBus()
+ bridge = OrbBusBridge(bus, pet)
+ bridge.attach()
+ pet._on_right_click()
+ for mode in MODES:
+ pet.show(mode)
+ assert not pet._root.visible
+ assert pet._mode == mode
+ await bus.publish(PetVisibilityToggleRequested(source="hotkey"))
+ assert pet._root.visible
+ assert not pet.pet_user_hidden
+ await bus.publish(PetVisibilityToggleRequested(source="hotkey"))
+ assert not pet._root.visible
+ assert pet.pet_user_hidden
+
+
+def test_strip_right_click_uses_the_same_pet_dismissal(pet) -> None:
+ strip = PetControlStrip.__new__(PetControlStrip)
+ strip._on_context = pet._on_right_click
+ strip._on_context_event(None)
+ assert pet.pet_user_hidden
+ assert not pet._root.visible
+
+
+def test_host_reports_right_click_and_restores_hidden_pet_after_respawn(pet, monkeypatch) -> None:
+ proxy = SubprocessMascotOverlay(style="pet")
+ monkeypatch.setattr(host, "emit", lambda event, **kw: proxy._dispatch_event(
+ {"event": event, **kw},
+ ))
+ monkeypatch.setattr(proxy, "_send", lambda msg: host.dispatch(pet, msg))
+ host._wire_surface_events(pet)
+ pet._on_right_click()
+ assert proxy.pet_user_hidden
+ replay = []
+ monkeypatch.setattr(proxy, "_send", replay.append)
+ proxy._reapply_desired_state()
+ assert {"op": "set_visible", "visible": False} in replay
+ monkeypatch.setattr(proxy, "_send", lambda msg: host.dispatch(pet, msg))
+ proxy.toggle_visible()
+ assert not proxy.pet_user_hidden
+ assert not pet.pet_user_hidden
+ assert pet._root.visible
+
+
+def test_right_click_is_safe_without_window_or_with_failed_observer() -> None:
+ pet = OrbOverlay(style="pet")
+
+ def fail(visible):
+ raise RuntimeError("observer unavailable")
+
+ pet.set_on_visibility_changed(fail)
+ pet._on_right_click()
+ assert pet.pet_user_hidden
diff --git a/tests/unit/ui/web/test_registry_bootstrap_shutdown.py b/tests/unit/ui/web/test_registry_bootstrap_shutdown.py
index 999b1a62e5..5f821280a1 100644
--- a/tests/unit/ui/web/test_registry_bootstrap_shutdown.py
+++ b/tests/unit/ui/web/test_registry_bootstrap_shutdown.py
@@ -37,7 +37,9 @@ async def test_server_joins_slow_registry_bootstraps_before_registry_stop():
tasks = [server._cli_bootstrap_task, server._plugin_bootstrap_task]
try:
await asyncio.wait_for(asyncio.gather(*(event.wait() for event in started)), 1)
- await asyncio.wait_for(server.stop(), 2)
+ # A hang guard, not a speed budget: stop() also releases every other
+ # helper, which a loaded Windows runner can stretch past two seconds.
+ await asyncio.wait_for(server.stop(), 15)
assert all(task.done() for task in tasks)
assert server._cli_bootstrap_task is None
assert server._plugin_bootstrap_task is None
diff --git a/tests/unit/workspace/test_agents.py b/tests/unit/workspace/test_agents.py
index 7eecfc29ee..d1be968cf2 100644
--- a/tests/unit/workspace/test_agents.py
+++ b/tests/unit/workspace/test_agents.py
@@ -180,6 +180,7 @@ def test_build_agent_argv_wraps_command_in_a_shell() -> None:
assert argv is not None
# the agent command appears in the argv, wrapped by a shell
assert any("claude" in part for part in argv)
+ assert any("--permission-mode bypassPermissions" in part for part in argv)
assert len(argv) >= 2 # shell + at least one flag/command
assert build_agent_argv("nope") is None
diff --git a/tests/unit/workspace/test_launch_picks.py b/tests/unit/workspace/test_launch_picks.py
index db639f8acf..eb735d6248 100644
--- a/tests/unit/workspace/test_launch_picks.py
+++ b/tests/unit/workspace/test_launch_picks.py
@@ -73,7 +73,9 @@ def test_codex_effort_is_a_config_override_not_a_flag() -> None:
literal string, so quotes here would end up inside the setting.
"""
argv = launch_picks.launch_argv("codex", effort="xhigh")
- assert argv == ("-c", "model_reasoning_effort=xhigh")
+ assert argv == (
+ "-c", "model_reasoning_effort=xhigh", "--dangerously-bypass-approvals-and-sandbox"
+ )
def test_codex_permission_words_are_sandbox_flags() -> None:
@@ -88,7 +90,9 @@ def test_codex_permission_words_are_sandbox_flags() -> None:
def test_a_model_the_cli_does_not_offer_is_dropped() -> None:
"""A stale pick costs the CLI's own default, never a pane that will not start."""
- assert launch_picks.launch_argv("codex", model="claude-opus-5") == ()
+ assert launch_picks.launch_argv("codex", model="claude-opus-5") == (
+ "--dangerously-bypass-approvals-and-sandbox",
+ )
assert launch_picks.normalize_model("codex", "claude-opus-5") == ""
@@ -96,7 +100,7 @@ def test_a_flag_cannot_be_smuggled_in_as_a_model() -> None:
"""These strings come from a browser; an argv reads a stray flag as an order."""
for hostile in ("--dangerously-skip-permissions", "-c evil=1", "a b", "$(whoami)", "x;y"):
assert launch_picks.normalize_model("opencode", hostile) == "", hostile
- assert launch_picks.launch_argv("opencode", model=hostile) == (), hostile
+ assert launch_picks.launch_argv("opencode", model=hostile) == ("--auto",), hostile
def test_a_cli_with_no_published_list_takes_the_users_own_id() -> None:
@@ -104,13 +108,15 @@ def test_a_cli_with_no_published_list_takes_the_users_own_id() -> None:
assert launch_picks.launch_argv("opencode", model="anthropic/claude-opus-5") == (
"--model",
"anthropic/claude-opus-5",
+ "--auto",
)
-def test_an_unknown_permission_mode_is_dropped_never_folded() -> None:
- """Landing near a stance is worse than landing on the CLI's ask-first default."""
- assert launch_picks.normalize_permission("claude", "full-access") == ""
- assert launch_picks.launch_argv("claude", permission_mode="full-access") == ()
+def test_an_unknown_permission_mode_is_rejected_never_widened() -> None:
+ with pytest.raises(ValueError, match="Unsupported permission mode"):
+ launch_picks.normalize_permission("claude", "full-access")
+ with pytest.raises(ValueError, match="Unsupported permission mode"):
+ launch_picks.launch_argv("claude", permission_mode="full-access")
def test_an_effort_off_the_ladder_snaps_down_never_up() -> None:
@@ -170,8 +176,7 @@ def test_offered_answers_in_the_composers_own_shape() -> None:
"default_permission_mode",
}
assert offered["default_effort"] in offered["effort_levels"]
- # A pane opens in the CLI's own stance until somebody picks another.
- assert offered["default_permission_mode"] == ""
+ assert offered["default_permission_mode"] == "bypassPermissions"
def test_a_live_list_replaces_the_curated_one() -> None:
diff --git a/ui/orb/bus_bridge.py b/ui/orb/bus_bridge.py
index d66f6a6d5e..0ad76483c8 100644
--- a/ui/orb/bus_bridge.py
+++ b/ui/orb/bus_bridge.py
@@ -1588,7 +1588,7 @@ class OrbBusBridge:
# ``log.debug`` behind it: the shortcut did nothing, visibly or
# otherwise, until the app was restarted. Suppressed states must not
# resurrect the mirror any more than they resurrect the mascot.
- if state == "LISTENING":
+ if state in ("CONNECTING", "LISTENING"):
# An authoritative supervisor edge can arrive after a late bridge
# attach even if VoiceSessionStarted itself was missed.
self._voice_session_active = True
@@ -1630,9 +1630,18 @@ class OrbBusBridge:
# stands it down when the call ended in the meantime.
return
- if state == "LISTENING":
+ if state == "CONNECTING":
+ # The call is accepted and the realtime transport is negotiating:
+ # the loading loop runs until the provider takes the session. The
+ # opening words are buffered meanwhile, so the transcript bubble
+ # stays as it is; the LISTENING that follows opens a fresh one.
+ self._orb.show(mode="connect")
+ self._cancel_idle_scheduler()
+ elif state == "LISTENING":
self._orb.show(mode="listen")
- if prev_state in ("IDLE", "ERROR", "PAUSED"):
+ if prev_state in ("IDLE", "ERROR", "PAUSED", "CONNECTING"):
+ # After CONNECTING this is the "connected" greeting: the bar
+ # and the pet strip play their flourish, the pet nods.
self._orb.play_animation("wave")
# The pulsing listen-mode already signals "I'm hearing you"
# visually. The bubble starts empty and fills with the live
@@ -2156,6 +2165,8 @@ class OrbBusBridge:
``renderer.visual_mode``), not the state label.
"""
state = self._last_state
+ if state == "CONNECTING":
+ return "connect"
if state in ("LISTENING", "WAITING_FOR_COMPLETION"):
return "listen"
if state in ("THINKING", "SPEAKING"):
diff --git a/ui/orb/controls.py b/ui/orb/controls.py
index 603ffa277c..0e1d73c7aa 100644
--- a/ui/orb/controls.py
+++ b/ui/orb/controls.py
@@ -38,6 +38,8 @@ from dataclasses import dataclass
from PIL import Image, ImageDraw
+from jarvis.ui.jarvisbar.modes import ACTIVE_VOICE_MODES, CONNECT_MODES
+
# --- Palette (the app's dark theme, flattened) ------------------------------
#: Resting disc: ``bg-background/85`` over the desktop reads as a near-black
#: pebble; solid here because a layered Tk window has no partial alpha.
@@ -431,11 +433,45 @@ PET_THINK_BASE_V = 0.21
PET_THINK_PEAK_V = 0.93
PET_THINK_DIM = 0.5
+#: Connecting: the three strokes themselves BECOME the loading loop. Each one
+#: bends into an arc of one ring and the three arcs spin round the talk
+#: control. The bend plays over ``PET_MORPH_STEPS`` steps of
+#: ``PET_MORPH_S`` seconds; the spin repeats every third of a turn (the arcs
+#: are interchangeable), in ``PET_SPIN_STEPS`` steps of ``PET_SPIN_PERIOD_S``.
+PET_MORPH_STEPS = 8
+PET_MORPH_S = 0.34
+PET_SPIN_STEPS = 12
+PET_SPIN_PERIOD_S = 0.4
+#: The ring the arcs lie on (centreline radius, share of the pill's height),
+#: each arc's span in degrees, how far it breathes per third of a turn, and
+#: how much a live voice lengthens it (the opening words are being captured).
+PET_LOOP_R = 0.27
+PET_ARC_SPAN_DEG = 62.0
+PET_ARC_BREATH_DEG = 12.0
+PET_ARC_VOICE_DEG = 18.0
+#: Where each stroke's arc sits once bent (clockwise from 12 o'clock): the
+#: left stroke curls to 9 o'clock, the others follow a third of a turn apart.
+PET_ARC_HOME_DEG: tuple[float, ...] = (270.0, 30.0, 150.0)
+#: Points per stroke centreline; enough for a round arc after the downscale.
+PET_STROKE_POINTS = 10
+#: Connected: once the provider takes the call the arcs turn on to the next
+#: rest position, unbend and slide back into the row, flashing green on the
+#: way, and the strokes pop up once and settle - a one-shot of
+#: ``PET_CONNECTED_PHASES`` steps over ``PET_CONNECTED_S`` seconds.
+PET_CONNECTED_PHASES = 14
+PET_CONNECTED_S = 0.72
+PET_CONNECTED_GREEN = (134, 239, 172)
+#: The strokes' glow in the silent voice look (``indicator_bars`` at level 0).
+PET_VOICE_SILENT_GLOW = 0.62
+
#: What the indicator is showing. ``rest``: nothing running. ``voice``: the
#: microphone or Jarvis's voice is live and the strokes follow its level.
#: ``think``: work is in flight with no signal to measure, so a highlight
-#: travels — motion, never a fake level.
-PET_MOTIONS: tuple[str, ...] = ("rest", "voice", "think")
+#: travels — motion, never a fake level. ``connect``: the call is accepted
+#: and the realtime transport is still negotiating (the strokes bend into
+#: the loading loop).
+#: ``connected``: the one-shot flourish when that negotiation succeeded.
+PET_MOTIONS: tuple[str, ...] = ("rest", "voice", "think", "connect", "connected")
#: Fill of the pen disc and the pill: a blue-black only a breath away from a
#: dark desktop, so the controls read as glyphs floating in a faint shadow
@@ -626,11 +662,15 @@ class PetStripState:
#: A conversation is running — the talk control and the phone hang up
#: instead of starting one.
active: bool = False
- #: The voice level step (``quantize_level``); drawn only in ``voice``.
+ #: The voice level step (``quantize_level``); drawn in ``voice`` and
+ #: ``connect`` (where it lengthens the loop's arcs).
level: int = 0
#: What the indicator shows (``PET_MOTIONS``) and its animation step.
motion: str = "rest"
phase: int = 0
+ #: ``connect``: the spin step (``PET_SPIN_STEPS`` per third of a turn).
+ #: ``connected``: the spin step the loop had when the call connected.
+ spin: int = 0
#: Which control the pointer is over, if any.
hovered: str | None = None
#: The user switched notifications off with the bell (this run only).
@@ -923,9 +963,80 @@ def indicator_phase(motion: str, t: float) -> int:
if motion == "think":
cycle = (t % PET_THINK_PERIOD_S) / PET_THINK_PERIOD_S
return int(cycle * PET_THINK_PHASES) % PET_THINK_PHASES
+ if motion == "connect":
+ # ``t`` is the time since connecting began: the bend, then done.
+ return min(PET_MORPH_STEPS, int(max(0.0, t) / PET_MORPH_S * PET_MORPH_STEPS))
+ if motion == "connected":
+ # ``t`` is the time since the call connected; the flourish plays once.
+ step = int(max(0.0, t) / PET_CONNECTED_S * PET_CONNECTED_PHASES)
+ return min(PET_CONNECTED_PHASES - 1, step)
return 0
+def spin_step(t: float) -> int:
+ """The loading loop's spin step ``t`` seconds after connecting began."""
+ cycle = (max(0.0, t) % PET_SPIN_PERIOD_S) / PET_SPIN_PERIOD_S
+ return int(cycle * PET_SPIN_STEPS) % PET_SPIN_STEPS
+
+
+@dataclass(frozen=True)
+class ConnectLook:
+ """What the talk control shows for a connecting or just-connected call."""
+
+ motion: str
+ phase: int
+ spin: int
+
+
+class ConnectTimeline:
+ """The clock behind the loading loop and its "connected" flourish.
+
+ One per surface (the Tk bar, the Qt bar, the pet strip). The surface
+ reports every coarse-mode change with :meth:`note_mode` and asks
+ :meth:`look` when it paints; times are the surface's own clock.
+ """
+
+ def __init__(self) -> None:
+ self._connect_t0: float | None = None
+ self._connected_t0: float | None = None
+ self._spin_from = 0
+
+ def note_mode(self, previous: str, new: str, now: float) -> None:
+ if new in CONNECT_MODES:
+ if previous not in CONNECT_MODES or self._connect_t0 is None:
+ self._connect_t0 = now
+ self._connected_t0 = None
+ return
+ if new not in ACTIVE_VOICE_MODES:
+ # Hung up, or the call ended: no flourish for a call that is gone.
+ self._connect_t0 = None
+ self._connected_t0 = None
+ return
+ if previous in CONNECT_MODES and self._connect_t0 is not None:
+ self._spin_from = spin_step(now - self._connect_t0)
+ self._connected_t0 = now
+ self._connect_t0 = None
+
+ def look(self, mode: str, now: float) -> ConnectLook | None:
+ """The connect look due at ``now``, or ``None`` for the plain look."""
+ if mode in CONNECT_MODES:
+ if self._connect_t0 is None:
+ self._connect_t0 = now
+ elapsed = now - self._connect_t0
+ return ConnectLook(
+ "connect", indicator_phase("connect", elapsed), spin_step(elapsed)
+ )
+ if self._connected_t0 is None or mode not in ACTIVE_VOICE_MODES:
+ return None
+ elapsed = now - self._connected_t0
+ if not 0.0 <= elapsed < PET_CONNECTED_S:
+ self._connected_t0 = None
+ return None
+ return ConnectLook(
+ "connected", indicator_phase("connected", elapsed), self._spin_from
+ )
+
+
def _sweep_gain(index: int, phase: int) -> float:
"""Brightness of stroke ``index`` under the thinking highlight.
@@ -951,7 +1062,8 @@ def indicator_bars(state: PetStripState) -> list[tuple[float, float]]:
out = []
for i in range(PET_INDICATOR_BARS):
wobble = 0.55 + 0.45 * (0.5 + 0.5 * math.sin(angle + i * 2.1))
- out.append((lo + (hi - lo) * level * wobble, 0.62 + 0.38 * level))
+ glow = PET_VOICE_SILENT_GLOW + (1.0 - PET_VOICE_SILENT_GLOW) * level
+ out.append((lo + (hi - lo) * level * wobble, glow))
return out
if state.motion == "think":
out = []
@@ -960,9 +1072,104 @@ def indicator_bars(state: PetStripState) -> list[tuple[float, float]]:
v = PET_THINK_BASE_V + (PET_THINK_PEAK_V - PET_THINK_BASE_V) * g
out.append((lo + (hi - lo) * v, PET_THINK_DIM + (1.0 - PET_THINK_DIM) * g))
return out
+ if state.motion in ("connect", "connected"):
+ # Straight (the first bend step, the flourish's last): the silent
+ # voice look, so the loop leaves and rejoins the listening row exactly.
+ return [(h, PET_VOICE_SILENT_GLOW) for h in PET_INDICATOR_REST_H]
return [(h, PET_INDICATOR_REST_GLOW) for h in PET_INDICATOR_REST_H]
+def _ease(x: float) -> float:
+ x = max(0.0, min(1.0, x))
+ return x * x * (3.0 - 2.0 * x)
+
+
+@dataclass(frozen=True)
+class MorphStroke:
+ """One stroke on its way between the row and the loop.
+
+ ``points`` is the centreline from top (leading end) to bottom, in shares
+ of the pill's height relative to the talk control's centre; ``glow`` per
+ point (0 night .. 1 sky, above 1 toward cloud white); ``green`` 0..1 is
+ the "connected" flash.
+ """
+
+ points: tuple[tuple[float, float], ...]
+ glow: tuple[float, ...]
+ green: float
+
+
+def morph_amount(state: PetStripState) -> float:
+ """How far the strokes are bent into the loop (0 row .. 1 ring)."""
+ if state.motion == "connect":
+ return _ease(state.phase / PET_MORPH_STEPS)
+ if state.motion == "connected":
+ p = (state.phase + 1) / PET_CONNECTED_PHASES
+ return 1.0 - _ease((p - 0.12) / 0.58)
+ return 0.0
+
+
+def morph_strokes(state: PetStripState) -> list[MorphStroke] | None:
+ """The three strokes bent toward the loading loop, or ``None`` when straight.
+
+ Pure in the state, so every frame is testable and cacheable. Each stroke's
+ centreline is blended point by point between its straight form in the row
+ and an arc of the ring; the arcs turn with ``spin``.
+ """
+ if state.motion not in ("connect", "connected"):
+ return None
+ m = morph_amount(state)
+ level = max(0.0, min(1.0, state.level / PET_LEVEL_STEPS))
+ third = 120.0
+ if state.motion == "connect":
+ cycle = (state.spin % PET_SPIN_STEPS) / PET_SPIN_STEPS
+ spin = third * cycle
+ breath = math.sin(2.0 * math.pi * cycle)
+ span = PET_ARC_SPAN_DEG + PET_ARC_BREATH_DEG * breath + PET_ARC_VOICE_DEG * level
+ green = 0.0
+ pop = 0.0
+ relabel = 0
+ else:
+ p = (state.phase + 1) / PET_CONNECTED_PHASES
+ start = third * (state.spin % PET_SPIN_STEPS) / PET_SPIN_STEPS
+ rest = (third - start) % third
+ # Turn on to the next rest position while unbending, never backwards.
+ spin = start + rest * _ease(p / 0.6)
+ span = PET_ARC_SPAN_DEG
+ green = 0.0 if p >= 1.0 else math.sin(math.pi * min(1.0, p / 0.85)) ** 1.5 * 0.9
+ pop = 0.0 if p >= 1.0 else math.sin(math.pi * _ease((p - 0.45) / 0.55))
+ # A third of a turn lands every arc on its neighbour's home, so it
+ # straightens into that neighbour's slot in the row.
+ relabel = 1 if rest > 1e-6 else 0
+ if m <= 1e-3 and green <= 1e-3 and pop <= 1e-3:
+ return None
+ n = PET_STROKE_POINTS
+ half_w = PET_INDICATOR_HALF_W
+ grow = PET_INDICATOR_MAX_H * 0.62 - PET_INDICATOR_MIN_H
+ out = []
+ for i in range(PET_INDICATOR_BARS):
+ slot = (i + relabel) % PET_INDICATOR_BARS
+ x = (slot - (PET_INDICATOR_BARS - 1) / 2.0) * PET_INDICATOR_SPACING
+ shape = (0.62, 1.0, 0.62)[slot % 3]
+ straight_h = PET_INDICATOR_REST_H[slot] + grow * shape * pop
+ half_len = max(0.0, straight_h / 2.0 - half_w)
+ centre = PET_ARC_HOME_DEG[i % len(PET_ARC_HOME_DEG)] + spin
+ points = []
+ glow = []
+ for k in range(n):
+ u = k / (n - 1)
+ sx, sy = x, -half_len + 2.0 * half_len * u
+ a = math.radians(centre + span / 2.0 - span * u - 90.0)
+ ax, ay = PET_LOOP_R * math.cos(a), PET_LOOP_R * math.sin(a)
+ points.append((sx + (ax - sx) * m, sy + (ay - sy) * m))
+ # Along the arc the head is bright and the tail fades to night.
+ arc_glow = 1.25 - 0.85 * u
+ rest_glow = PET_VOICE_SILENT_GLOW + 0.3 * pop
+ glow.append(rest_glow + (arc_glow - rest_glow) * m)
+ out.append(MorphStroke(tuple(points), tuple(glow), green))
+ return out
+
+
def _cloud_cover(u: float, v: float, w: float, h: float, puffs) -> float:
"""How white the sky is at ``(u, v)`` inside a stroke (0 sky, 1 cloud).
@@ -1035,8 +1242,13 @@ def _draw_indicator(
Drawn on the supersampled pill layer, so the clouds keep their feathered
edges after the downscale. The frame is RGB (the colour key needs it), so
"dim" is a mix toward a night blue, which keeps a resting stroke reading
- as sky instead of turning grey.
+ as sky instead of turning grey. While a call connects the same strokes
+ bend into the loading loop (``morph_strokes``).
"""
+ strokes = morph_strokes(state)
+ if strokes is not None:
+ _draw_morph_strokes(ImageDraw.Draw(layer), cx, cy, pill_h, strokes)
+ return
width = max(2, int(round(2.0 * pill_h * PET_INDICATOR_HALF_W)))
tallest = max(width, int(round(pill_h * PET_INDICATOR_MAX_H)))
step = pill_h * PET_INDICATOR_SPACING
@@ -1054,6 +1266,36 @@ def _draw_indicator(
layer.paste(stroke, (x0, y0), _stroke_mask(width, h))
+def _morph_colour(glow: float, green: float) -> _Rgb:
+ base = _lerp(PET_INDICATOR_NIGHT, PET_INDICATOR_SKY, min(1.0, glow))
+ if glow > 1.0:
+ base = _lerp(base, PET_INDICATOR_CLOUD, min(1.0, (glow - 1.0) / 0.3) * 0.8)
+ return _lerp(base, PET_CONNECTED_GREEN, green)
+
+
+def _draw_morph_strokes(
+ d: ImageDraw.ImageDraw, cx: float, cy: float, pill_h: float, strokes: list[MorphStroke]
+) -> None:
+ """Strokes bent toward the loop, as thick round-capped polylines.
+
+ Drawn on the supersampled pill layer; the downscale smooths the curve.
+ The frame has no alpha, so the tail fades by its colour sinking toward
+ night blue, never by transparency.
+ """
+ width = max(2, int(round(2.0 * pill_h * PET_INDICATOR_HALF_W)))
+ r = width / 2.0
+ for stroke in strokes:
+ pts = [(cx + x * pill_h, cy + y * pill_h) for x, y in stroke.points]
+ cols = [_morph_colour(g, stroke.green) for g in stroke.glow]
+ # Tail first, so the bright head is painted on top.
+ for k in range(len(pts) - 1, 0, -1):
+ d.line([pts[k], pts[k - 1]], fill=_lerp(cols[k], cols[k - 1], 0.5), width=width)
+ x, y = pts[k]
+ d.ellipse((x - r, y - r, x + r, y + r), fill=cols[k])
+ x, y = pts[0]
+ d.ellipse((x - r, y - r, x + r, y + r), fill=cols[0])
+
+
def _render_call_disc(state: PetStripState, diameter: int, scale: float) -> Image.Image:
"""The phone disc: the bell disc's look, green or red under the pointer."""
size = diameter * _SS
@@ -1111,7 +1353,8 @@ def _render_pill(state: PetStripState, layout: PetStripLayout, scale: float) ->
cx = ((sx0 + sx1) / 2.0 - x0) * _SS
if action == "orb":
# The pet and the Jarvis Bar share one talk control: the three
- # strokes that rest, follow the voice and run while Jarvis thinks.
+ # strokes that rest, follow the voice, run while Jarvis thinks
+ # and bend into the loading loop while a call connects.
_draw_indicator(layer, cx, cy, h_ss, state)
else:
_draw_glyph(action, d, cx, cy, box, stroke, state)
diff --git a/ui/orb/overlay.py b/ui/orb/overlay.py
index 1387def9fa..fab465dd1f 100644
--- a/ui/orb/overlay.py
+++ b/ui/orb/overlay.py
@@ -67,7 +67,7 @@ from PIL import Image, ImageDraw, ImageFilter, ImageTk
from jarvis.core.config import DEFAULT_CONFIG_FILE as JARVIS_TOML_PATH
from jarvis.core.win32_dpi import ensure_dpi_awareness as _ensure_dpi_awareness
-from jarvis.ui.jarvisbar.modes import MODES
+from jarvis.ui.jarvisbar.modes import ACTIVE_VOICE_MODES, MODES
from jarvis.ui.overlay_styles import LEGACY_STYLE_ALIASES, ORB_STYLES, PERSISTENT_ORB_STYLES
from jarvis.ui.pets.states import ACTION_STATES, DEFAULT_PET_ID, NO_PET_ID, ONE_SHOT_STATES
from ui.orb import controls as orb_controls
@@ -2118,6 +2118,7 @@ class PetControlStrip(OrbControlRow):
notify_off: bool | None = None,
ring: int | None = None,
call_ring: int | None = None,
+ spin: int | None = None,
) -> None:
"""Update what the strip says, repainting only on a real change."""
_ = can_attach # the pet strip has no attach control
@@ -2129,6 +2130,7 @@ class PetControlStrip(OrbControlRow):
level=current.level if level is None else int(level),
motion=current.motion if motion is None else str(motion),
phase=current.phase if phase is None else int(phase),
+ spin=current.spin if spin is None else int(spin),
hovered=current.hovered,
notify_off=current.notify_off if notify_off is None else bool(notify_off),
ring=current.ring if ring is None else int(ring),
@@ -2216,6 +2218,9 @@ FRAME_INTERVAL_MS = 16
#: ticks at least this often, so the strokes follow the voice even when the
#: figure is slow.
PET_PULSE_INTERVAL_MS = 100
+#: The loading loop and the "connected" flourish turn faster than a level can
+#: change, so the strip ticks at this rate while either is on screen.
+PET_CONNECT_INTERVAL_MS = 40
#: A level older than this no longer moves the strip's strokes (the voice stopped).
PET_LEVEL_FRESH_S = 0.25
#: Overlay modes and what the strip's indicator shows for them. Dictation
@@ -2276,10 +2281,11 @@ class OrbOverlay:
# stays decoupled from the bus — OrbBusBridge injects a
# callable that publishes ``VoiceMuteToggleRequested``.
self._mute_toggle_callback: Callable[[], None] | None = None
- # Right-click on the orb raises the main desktop window. OrbBusBridge
+ # Right-click on non-pet orbs raises the main window. OrbBusBridge
# injects a callable that publishes ``ShowWindowRequested``; the orb
# itself stays bus-agnostic (same contract as the mute toggle).
self._on_show_window: Callable[[], None] | None = None
+ self._on_visibility_changed: Callable[[bool], None] | None = None
# The control row's actions. Each is optional: wired, the callback
# decides (the macOS host forwards it to the parent process, which is
# the only place a SpeechPipeline exists); unwired, the row falls back
@@ -2347,6 +2353,9 @@ class OrbOverlay:
self._photo: ImageTk.PhotoImage | None = None
self._image_id: int | None = None
self._mode: str = "idle"
+ # The strip's clock of the strokes bending into the loading loop while
+ # a call connects, and of their one-shot back out (time.monotonic()).
+ self._connect_timeline = orb_controls.ConnectTimeline()
self._ext_level: float | None = None
self._ext_level_at: float = -math.inf
self._t0: float = 0.0
@@ -2988,7 +2997,7 @@ class OrbOverlay:
return
row.set_on_action(self._on_control_action)
row.set_state(
- active=self._mode in ("listen", "think", "speak"),
+ active=self._mode in ACTIVE_VOICE_MODES,
speaker_muted=self._speaker_muted,
)
if isinstance(row, PetControlStrip):
@@ -3011,7 +3020,7 @@ class OrbOverlay:
"""
if self._pet_strip_always or self._pet_id == NO_PET_ID:
return True
- if self._mode in PET_VOICE_MODES or self._mode in PET_THINK_MODES:
+ if self._mode in ACTIVE_VOICE_MODES + PET_VOICE_MODES + PET_THINK_MODES:
return True
notices = self._notices
if notices is not None and notices.count:
@@ -3031,7 +3040,7 @@ class OrbOverlay:
# A live conversation keeps the controls out: hanging up is the one
# thing the user reaches for mid-call, and hunting for a row that hides
# itself is worse than a row that stays.
- if self._mode in ("listen", "think", "speak"):
+ if self._mode in ACTIVE_VOICE_MODES:
return
row.hide_after_grace()
@@ -3040,7 +3049,7 @@ class OrbOverlay:
row = self._controls
if row is None:
return
- active = self._mode in ("listen", "think", "speak")
+ active = self._mode in ACTIVE_VOICE_MODES
row.set_state(active=active)
if self._style == "pet":
# The pet alone at rest; its strip joins it while it is useful.
@@ -3098,7 +3107,7 @@ class OrbOverlay:
def _do_call(self) -> None:
"""The phone disc: call Jarvis (ringing as it dials), or hang up."""
- if self._mode not in ("listen", "think", "speak"):
+ if self._mode not in ACTIVE_VOICE_MODES:
self._ring_phone()
self._do_talk_or_hangup()
@@ -3204,7 +3213,7 @@ class OrbOverlay:
def _do_talk_or_hangup(self) -> None:
"""The mic disc: start a conversation, or end the running one."""
- if self._mode in ("listen", "think", "speak"):
+ if self._mode in ACTIVE_VOICE_MODES:
self._do_hangup()
return
callback = self._on_talk
@@ -3232,7 +3241,7 @@ class OrbOverlay:
def _do_close(self) -> None:
"""X: end the conversation and put the orb away, like the in-app X."""
- if self._mode in ("listen", "think", "speak"):
+ if self._mode in ACTIVE_VOICE_MODES:
self._do_hangup()
row = self._controls
if row is not None:
@@ -3303,19 +3312,19 @@ class OrbOverlay:
self._on_speaker_toggle = callback
def set_on_show_window(self, callback: Callable[[], None] | None) -> None:
- """Inject the right-click → raise-main-window action.
+ """Inject the main-window action used by non-pet orb right-clicks.
- Fired on a right-click of the orb. Same bus-agnostic contract as
+ Same bus-agnostic contract as
``set_on_mute_toggle``: OrbBusBridge passes a callable that publishes
``ShowWindowRequested``. Pass ``None`` to detach.
"""
self._on_show_window = callback
def _on_right_click(self, _event: tk.Event | None = None) -> None:
- """Right-click → raise the main desktop window via the injected
- callback. Replaces the old Reset/Mute context menu (spec 2026-06-02):
- "Reset position" now lives on middle-click, mute stays on the
- double-double-click gesture. No callback wired → safe no-op."""
+ """Dismiss the pet; other orb styles raise the main desktop window."""
+ if self._style == "pet":
+ self.set_visible(False)
+ return
callback = self._on_show_window
if callback is None:
return
@@ -4020,6 +4029,20 @@ class OrbOverlay:
self._user_hidden = not bool(visible)
self._note_activity()
self._enqueue_ui(self._apply_user_visibility)
+ self._notify_visibility_changed()
+
+ def set_on_visibility_changed(self, callback: Callable[[bool], None] | None) -> None:
+ """Report pet dismissal to the companion host's parent process."""
+ self._on_visibility_changed = callback
+
+ def _notify_visibility_changed(self) -> None:
+ callback = self._on_visibility_changed
+ if self._style != "pet" or callback is None:
+ return
+ try:
+ callback(not self._user_hidden)
+ except Exception: # noqa: BLE001 — observers must not break the Tk loop
+ logging.getLogger("jarvis.orb").debug("pet visibility callback failed", exc_info=True)
def toggle_visible(self) -> None:
"""The pet shortcut: hide the pet, or show it and bring it to the front."""
@@ -4152,9 +4175,11 @@ class OrbOverlay:
if not self._user_hidden and self._window_mapped():
self._user_hidden = True
self._apply_user_visibility()
+ self._notify_visibility_changed()
return
self._user_hidden = False
self._reveal_pet(raise_to_front=True)
+ self._notify_visibility_changed()
def _reveal_pet(self, *, raise_to_front: bool) -> None:
"""Map the pet window, its strip, and restart the frame loop."""
@@ -4425,6 +4450,7 @@ class OrbOverlay:
if mode not in MODES:
raise ValueError(f"Unknown mode: {mode!r} (allowed: {', '.join(MODES)})")
changed = mode != self._mode
+ self._connect_timeline.note_mode(self._mode, mode, time.monotonic())
self._mode = mode
renderer = getattr(self, "_renderer", None)
if isinstance(renderer, PetRenderer):
@@ -4579,7 +4605,13 @@ class OrbOverlay:
self._frame_key = key
delay_ms = int(renderer.next_frame_delay_ms(t))
if self._pump_strip_level():
- delay_ms = min(delay_ms, PET_PULSE_INTERVAL_MS)
+ row = self._controls
+ connecting = isinstance(row, PetControlStrip) and row.state.motion in (
+ "connect",
+ "connected",
+ )
+ pulse_ms = PET_CONNECT_INTERVAL_MS if connecting else PET_PULSE_INTERVAL_MS
+ delay_ms = min(delay_ms, pulse_ms)
return delay_ms
def _show_photo(self, photo: Any) -> None:
@@ -4592,26 +4624,32 @@ class OrbOverlay:
"""Drive the strip's three strokes. True while they move.
Listening or talking: the strokes follow the live audio level.
- Thinking: a highlight travels across them. At rest they stand still
- and the strip is not repainted at all.
+ Thinking: a highlight travels across them. Connecting: the loading
+ strokes bend into a spinning loop, then straighten once connected. At
+ rest they stand still and the strip is not repainted at all.
"""
row = self._controls
if not isinstance(row, PetControlStrip):
return False
now = time.monotonic()
- if self._mode in PET_VOICE_MODES:
+ look = self._connect_timeline.look(self._mode, now)
+ phase = spin = 0
+ if look is not None:
+ # Connecting: the strokes bend into the loading loop; connected:
+ # the one-shot back out of it.
+ motion, phase, spin = look.motion, look.phase, look.spin
+ elif self._mode in PET_VOICE_MODES:
motion = "voice"
elif self._mode in PET_THINK_MODES:
motion = "think"
else:
motion = "rest"
fresh = now - self._ext_level_at <= PET_LEVEL_FRESH_S
- level = orb_controls.quantize_level(self._ext_level) if motion == "voice" and fresh else 0
- row.set_state(
- level=level,
- motion=motion,
- phase=orb_controls.indicator_phase(motion, now),
- )
+ metered = motion in ("voice", "connect")
+ level = orb_controls.quantize_level(self._ext_level) if metered and fresh else 0
+ if look is None:
+ phase = orb_controls.indicator_phase(motion, now)
+ row.set_state(level=level, motion=motion, phase=phase, spin=spin)
return motion != "rest"
def _arm_frame(self, delay_ms: int) -> None:
diff --git a/ui/orb/voice_orb.py b/ui/orb/voice_orb.py
index 8c35aa25ad..3e1c2df9e7 100644
--- a/ui/orb/voice_orb.py
+++ b/ui/orb/voice_orb.py
@@ -117,6 +117,10 @@ class Motion:
#: shimmering as though it were still hearing something.
#: * ``notice`` — something the user asked for did not happen. Quiet and dim,
#: clearly awake but clearly not listening.
+#: * ``connect`` — the call is accepted and the realtime transport negotiates.
+#: The field gathers and slowly turns while a comet circles the sphere (the
+#: loading loop, ``_spinner_layer``); when the call connects, one bright ring
+#: leaves the sphere before the listening weather takes over.
MOTIONS: dict[str, Motion] = {
"idle": Motion(0.025, 0.008, 0.004, 0.14, 0.8, 0.86, 0.0, 0.28),
"listen": Motion(0.11, 0.035, 0.016, 0.72, 1.02, 0.98, 0.0, 0.34),
@@ -129,15 +133,17 @@ MOTIONS: dict[str, Motion] = {
"dictate": Motion(0.11, 0.035, 0.016, 0.72, 1.02, 0.98, 0.0, 0.34),
"dictate_transcribing": Motion(0.95, -0.44, 0.010, 0.34, 1.85, 1.08, 0.0, 1.45),
"notice": Motion(0.012, 0.0, 0.003, 0.12, 0.7, 0.72, 0.0, 0.28),
+ "connect": Motion(0.42, 0.18, 0.022, 0.55, 1.25, 0.9, 0.0, 0.6),
}
-#: Modes that react to a live level. ``speak`` is here because the bridge
+#: Modes that react to a live level. ``connect`` is here because the opening
+#: words are captured while the transport negotiates. ``speak`` is here because the bridge
#: forwards the REAL TTS loudness through ``set_level`` while Jarvis talks
#: (``OrbBridge._note_tts_level``); using it means the orb swells on the actual
#: voice instead of a synthetic cadence that only looks like speech. Every
#: other mode ignores ``ext_level`` outright, so a stale sample cannot animate
#: the orb while nothing is being heard.
-_LEVEL_REACTIVE = frozenset({"listen", "dictate", "speak"})
+_LEVEL_REACTIVE = frozenset({"listen", "dictate", "speak", "connect"})
#: Floor under the dictation level: a quiet moment while recording must still
#: read as "I am listening", never as a dead orb.
@@ -176,6 +182,22 @@ _WAVE_PERIOD_S = 1.35
_WAVE_TRAVEL_S = 1.05
#: Wave ring thickness, as a fraction of the half-window.
_WAVE_WIDTH = 0.055
+# --- Connecting loop ----------------------------------------------------------
+#: One turn of the comet around the sphere, in seconds.
+_SPIN_PERIOD_S = 1.15
+#: The comet's tail, as a share of the full circle.
+_SPIN_TAIL = 0.42
+#: Gap between the sphere's edge and the comet's path, and the path's
+#: thickness, as fractions of the half-window.
+_SPIN_GAP = 0.08
+_SPIN_WIDTH = 0.075
+#: The "connected" ring: how long it travels outward and its thickness.
+_CONNECTED_S = 0.7
+_CONNECTED_WIDTH = 0.07
+#: Modes that count as a live call after ``connect`` (the ring plays on the
+#: change to one of them).
+_CALL_MODES = frozenset({"listen", "think", "speak"})
+
#: Ordered-dither cell. 8x8 is fine enough that the pattern reads as texture
#: rather than as a grid, and cheap enough to tile across a 216² window.
_DITHER_N = 8
@@ -326,6 +348,9 @@ class VoiceOrbRenderer:
self._speaking_age = math.inf
self._live_impact = 0.0
self._live_input = 0.0
+ #: Seconds since the call connected while the "connected" ring plays;
+ #: ``inf`` when it is not playing.
+ self._connected_age = math.inf
self._field_due_t = -math.inf
self._texture: Image.Image | None = None
self._masks: dict[int, Image.Image] = {}
@@ -351,6 +376,10 @@ class VoiceOrbRenderer:
self._last_t = t
if mode != self._active_mode:
+ if self._active_mode == "connect" and mode in _CALL_MODES:
+ self._connected_age = 0.0
+ elif mode == "connect" or mode not in _CALL_MODES:
+ self._connected_age = math.inf
self._active_mode = mode
self._speaking_age = 0.0 if mode == "speak" else math.inf
@@ -371,6 +400,10 @@ class VoiceOrbRenderer:
self._activity_elapsed += dt
if math.isfinite(self._speaking_age):
self._speaking_age += dt
+ if math.isfinite(self._connected_age):
+ self._connected_age += dt
+ if self._connected_age >= _CONNECTED_S:
+ self._connected_age = math.inf
# Speech choreography. When the host forwards a REAL output level the
# sphere follows that; the synthetic cadence is only the fallback for a
@@ -416,7 +449,13 @@ class VoiceOrbRenderer:
energy = max(energy, 0.42 + 0.30 * math.sin(self._aura_elapsed * 3.1))
elif mode == "speak":
energy = max(energy, self._live_impact * 6.0)
- return self._compose(self._texture, radius, min(1.0, max(0.0, energy)))
+ elif mode == "connect":
+ # A low, steady glow under the comet: awake, getting through.
+ energy = max(energy, 0.22 + 0.08 * math.sin(self._aura_elapsed * 2.4))
+ frame = self._compose(self._texture, radius, min(1.0, max(0.0, energy)))
+ if mode == "connect" or math.isfinite(self._connected_age):
+ self._draw_connect_ring(frame, radius, mode == "connect")
+ return frame
# Compatibility no-ops: the overlay drives mascot-only expressions through
# these. Answering them here (instead of leaving them to fail) keeps the
@@ -609,6 +648,46 @@ class VoiceOrbRenderer:
layer[visible, 2] = (LOWER[2] + (MID[2] - LOWER[2]) * amount).astype(np.uint8)
return layer
+ def _draw_connect_ring(self, frame: Image.Image, radius: float, connecting: bool) -> None:
+ """The loading comet around the sphere, or the "connected" ring leaving it.
+
+ Painted straight into the colour-keyed frame with the aura's density
+ trick: the tail thins by dropping pixels against the fixed dither
+ field, never by alpha the window cannot carry. Only pixels outside the
+ sphere are touched, so the field itself stays untouched.
+ """
+ half = max(1.0, self._size / 2.0)
+ inner = radius / half
+ field = self._pixel_radius
+ if connecting:
+ path_r = min(_AURA_REACH - _SPIN_WIDTH, inner + _SPIN_GAP)
+ band = np.clip(1.0 - np.abs(field - path_r) / _SPIN_WIDTH, 0.0, 1.0)
+ # Clockwise from 12 o'clock: 0 at the head, growing along the tail.
+ angle = (self._pixel_angle + math.pi / 2.0) / (2.0 * math.pi)
+ head = (self._aura_elapsed / _SPIN_PERIOD_S) % 1.0
+ behind = (head - angle) % 1.0
+ tail = np.clip(1.0 - behind / _SPIN_TAIL, 0.0, 1.0)
+ strength = band * tail**1.2 * 1.6
+ bright = CREST
+ dim = LOWER
+ else:
+ travel = min(1.0, self._connected_age / _CONNECTED_S)
+ ring_r = inner + (_AURA_REACH - inner) * (0.15 + 0.85 * travel)
+ band = np.clip(1.0 - np.abs(field - ring_r) / _CONNECTED_WIDTH, 0.0, 1.0)
+ strength = band * (1.0 - travel) * 1.3
+ bright = CLOUD
+ dim = MID
+ visible = (field > inner) & (strength > self._dither_field)
+ if not visible.any():
+ return
+ pixels = np.asarray(frame).copy()
+ amount = np.clip(strength[visible], 0.0, 1.0)[:, None]
+ tone = np.array(dim, dtype=np.float32) + (
+ np.array(bright, dtype=np.float32) - np.array(dim, dtype=np.float32)
+ ) * amount
+ pixels[visible] = np.clip(np.round(tone), 0, 255).astype(np.uint8)
+ frame.paste(Image.fromarray(pixels, "RGB"))
+
def _compose(self, texture: Image.Image, radius: float, energy: float = 0.0) -> Image.Image:
diameter = max(2, int(round(radius * 2.0)))
aura = self._aura_layer(radius, energy)
diff --git a/uv.lock b/uv.lock
index b7aa332599..acb805cc77 100644
--- a/uv.lock
+++ b/uv.lock
@@ -2907,14 +2907,14 @@ desktop = [
{ name = "pyautogui" },
{ name = "pycaw", marker = "sys_platform == 'win32'" },
{ name = "pynput", marker = "sys_platform == 'darwin' or sys_platform == 'win32'" },
- { name = "pyside6-addons", marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
- { name = "pyside6-essentials", version = "6.11.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'linux'" },
- { name = "pyside6-essentials", version = "6.11.2", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
+ { name = "pyside6-addons", marker = "platform_machine != 'aarch64' or sys_platform != 'linux'" },
+ { name = "pyside6-essentials", marker = "platform_machine != 'aarch64' or sys_platform != 'linux'" },
{ name = "pystray" },
{ name = "pywebview" },
{ name = "pywin32", marker = "sys_platform == 'win32'" },
{ name = "pywinauto", marker = "sys_platform == 'win32'" },
{ name = "pywinpty", marker = "sys_platform == 'win32'" },
+ { name = "soundcard", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
{ name = "tkinterdnd2" },
{ name = "win10toast-click", marker = "sys_platform == 'win32'" },
{ name = "winrt-runtime", marker = "sys_platform == 'win32'" },
@@ -2945,9 +2945,8 @@ dev = [
{ name = "pycaw", marker = "sys_platform == 'win32'" },
{ name = "pyinstaller" },
{ name = "pynput", marker = "sys_platform == 'darwin' or sys_platform == 'win32'" },
- { name = "pyside6-addons", marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
- { name = "pyside6-essentials", version = "6.11.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'linux'" },
- { name = "pyside6-essentials", version = "6.11.2", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
+ { name = "pyside6-addons", marker = "platform_machine != 'aarch64' or sys_platform != 'linux'" },
+ { name = "pyside6-essentials", marker = "platform_machine != 'aarch64' or sys_platform != 'linux'" },
{ name = "pystray" },
{ name = "pytest" },
{ name = "pytest-asyncio" },
@@ -2958,6 +2957,7 @@ dev = [
{ name = "pywinpty", marker = "sys_platform == 'win32'" },
{ name = "ruff" },
{ name = "sherpa-onnx", marker = "platform_machine != 'ARM64' or sys_platform != 'win32'" },
+ { name = "soundcard", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
{ name = "tkinterdnd2" },
{ name = "webrtcvad-wheels", marker = "(python_full_version < '3.14' and platform_machine != 'ARM64') or (python_full_version < '3.14' and sys_platform != 'win32')" },
{ name = "win10toast-click", marker = "sys_platform == 'win32'" },
@@ -2982,15 +2982,15 @@ full = [
{ name = "pyobjc-framework-avfoundation", marker = "sys_platform == 'darwin'" },
{ name = "pyobjc-framework-quartz", marker = "sys_platform == 'darwin'" },
{ name = "pyobjc-framework-screencapturekit", marker = "sys_platform == 'darwin'" },
- { name = "pyside6-addons", marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
- { name = "pyside6-essentials", version = "6.11.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'linux'" },
- { name = "pyside6-essentials", version = "6.11.2", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
+ { name = "pyside6-addons", marker = "platform_machine != 'aarch64' or sys_platform != 'linux'" },
+ { name = "pyside6-essentials", marker = "platform_machine != 'aarch64' or sys_platform != 'linux'" },
{ name = "pystray" },
{ name = "pywebview" },
{ name = "pywin32", marker = "sys_platform == 'win32'" },
{ name = "pywinauto", marker = "sys_platform == 'win32'" },
{ name = "pywinpty", marker = "sys_platform == 'win32'" },
{ name = "sherpa-onnx", marker = "platform_machine != 'ARM64' or sys_platform != 'win32'" },
+ { name = "soundcard", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
{ name = "tkinterdnd2" },
{ name = "twilio" },
{ name = "webrtcvad-wheels", marker = "(python_full_version < '3.14' and platform_machine != 'ARM64') or (python_full_version < '3.14' and sys_platform != 'win32')" },
@@ -3098,7 +3098,7 @@ requires-dist = [
{ name = "pyobjc-framework-quartz", marker = "sys_platform == 'darwin' and extra == 'desktop-macos'", specifier = ">=10" },
{ name = "pyobjc-framework-screencapturekit", marker = "sys_platform == 'darwin' and extra == 'desktop-macos'", specifier = ">=10" },
{ name = "pypdf", specifier = ">=6.16.1" },
- { name = "pyside6-addons", marker = "platform_machine != 'aarch64' and sys_platform == 'linux' and extra == 'desktop'", specifier = ">=6.7" },
+ { name = "pyside6-addons", marker = "(platform_machine != 'aarch64' and extra == 'desktop') or (sys_platform != 'linux' and extra == 'desktop')", specifier = ">=6.7" },
{ name = "pyside6-essentials", marker = "(platform_machine != 'aarch64' and extra == 'desktop') or (sys_platform != 'linux' and extra == 'desktop')", specifier = ">=6.7" },
{ name = "pystray", marker = "extra == 'desktop'", specifier = ">=0.19" },
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=8.3" },
@@ -3116,6 +3116,7 @@ requires-dist = [
{ name = "scipy", specifier = ">=1.14" },
{ name = "send2trash", specifier = ">=1.8.3" },
{ name = "sherpa-onnx", marker = "(platform_machine != 'ARM64' and extra == 'local-voice') or (sys_platform != 'win32' and extra == 'local-voice')", specifier = ">=1.13.3" },
+ { name = "soundcard", marker = "(sys_platform == 'linux' and extra == 'desktop') or (sys_platform == 'win32' and extra == 'desktop')", specifier = ">=0.4.6" },
{ name = "sounddevice", specifier = ">=0.5" },
{ name = "structlog", specifier = ">=24.4" },
{ name = "tenacity", specifier = ">=9.0" },
@@ -4052,66 +4053,30 @@ name = "pyside6-addons"
version = "6.11.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "pyside6-essentials", version = "6.11.2", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'linux'" },
- { name = "shiboken6", version = "6.11.2", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'linux'" },
+ { name = "pyside6-essentials" },
+ { name = "shiboken6" },
]
wheels = [
+ { url = "https://files.pythonhosted.org/packages/e3/91/339fe9f7c94163815004ccd6b5c58b926b2dfb4e51c3b5563d412eab151f/pyside6_addons-6.11.2-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:354c574a839f7d0751960ba03f33bec87c95e0f2796dca3287f54d03abe97dac", size = 332047075, upload-time = "2026-08-18T06:35:33.037Z" },
{ url = "https://files.pythonhosted.org/packages/28/cd/ce470e20f68901f4dfa5f742d6a3a58ba865684a232e4b87d5a1a2179cc0/pyside6_addons-6.11.2-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:a8b00956925fbdeffc7052cf933506391289f35ac4f3f71a0a167ec195cd47b4", size = 175062576, upload-time = "2026-08-18T06:35:45.583Z" },
{ url = "https://files.pythonhosted.org/packages/11/bd/f725a8984703612c72cd17d0871d047132bacaccc13fc0d288a5e8a37dad/pyside6_addons-6.11.2-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:a2ca3c73e5f060d4aca49451abd9fba68dd0e1c66ff40cea3a1fe0415def579a", size = 170827254, upload-time = "2026-08-18T06:35:56.675Z" },
-]
-
-[[package]]
-name = "pyside6-essentials"
-version = "6.11.1"
-source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "(python_full_version >= '3.14' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
- "python_full_version >= '3.14' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version >= '3.14' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.13.*' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version == '3.13.*' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version >= '3.14' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "python_full_version == '3.13.*' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "(python_full_version == '3.13.*' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version == '3.13.*' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
- "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "python_full_version < '3.12' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.12.*' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version == '3.12.*' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version < '3.12' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version < '3.12' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.12.*' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "python_full_version < '3.12' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "(python_full_version == '3.12.*' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version == '3.12.*' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
- "(python_full_version < '3.12' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version < '3.12' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
-]
-dependencies = [
- { name = "shiboken6", version = "6.11.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'linux'" },
-]
-wheels = [
- { url = "https://files.pythonhosted.org/packages/b3/da/10d9197e7370eb4fed8df5fc547b7548dec88e5c5949e2d450db4ae96feb/pyside6_essentials-6.11.1-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:228de53c2bc26b07e5021fbe3614fc44ca08e4dab9999af08c2b389d2c239957", size = 110352945, upload-time = "2026-05-13T09:43:08.006Z" },
- { url = "https://files.pythonhosted.org/packages/64/0e/b663ecc96ca57b5c91b83b6615d6b174380b0faf30338125c26e053d6aa7/pyside6_essentials-6.11.1-cp310-abi3-win_amd64.whl", hash = "sha256:63311bd48e32c584599ab04b9ef7c324082374cd2c9fa533f978fb893bb47e40", size = 77549267, upload-time = "2026-05-13T09:43:44.92Z" },
- { url = "https://files.pythonhosted.org/packages/f1/12/eb6723faf5cb7fa581145da1c15f40d641b96e080f0491af2f1859fdeedb/pyside6_essentials-6.11.1-cp310-abi3-win_arm64.whl", hash = "sha256:11253ea52aabecefe9febddbbe78b43a824129e3af1cec98431028fba7fa954f", size = 57964512, upload-time = "2026-05-13T09:43:52.968Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/c1/08bf6594101b42fc88834009f1dac59a983c5b6805c62c5081f077d09b4a/pyside6_addons-6.11.2-cp310-abi3-win_amd64.whl", hash = "sha256:f449ea4431da20e7b86752cca8d166f93434516fe417f981c27e5f8e1b554407", size = 168208836, upload-time = "2026-08-18T06:36:07.295Z" },
+ { url = "https://files.pythonhosted.org/packages/26/60/e4b7f09343e9f29e565f488fb44c4f1fe260db481ba2c08ad8a1a7d48216/pyside6_addons-6.11.2-cp310-abi3-win_arm64.whl", hash = "sha256:d5606af369484b862b4d5741cfc179e19d2b8819c3e04d210b8188e0c9f85988", size = 35807245, upload-time = "2026-08-18T06:36:12.485Z" },
]
[[package]]
name = "pyside6-essentials"
version = "6.11.2"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version < '3.12' and platform_machine == 'x86_64' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version < '3.12' and platform_machine != 'x86_64' and sys_platform == 'linux'",
-]
dependencies = [
- { name = "shiboken6", version = "6.11.2", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'linux'" },
+ { name = "shiboken6" },
]
wheels = [
+ { url = "https://files.pythonhosted.org/packages/6f/16/0b7ecf89ebada82ed0430be33809ff325761ece83104f8635b1bd101fcd0/pyside6_essentials-6.11.2-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:77795c145202e65a78d88f7cd409d186e3ba23d159bdb3ba2dcd159ae5e5f0d9", size = 110800753, upload-time = "2026-08-18T06:36:33.674Z" },
{ url = "https://files.pythonhosted.org/packages/0e/1a/84a84bbc75835ee88abdcb66eb875b8e1a21834af9b833bc1893f2c1bfaf/pyside6_essentials-6.11.2-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:aaf9f25f0f324874085fa5b26a610318db8a8e243cf85bb3e5400595191c7778", size = 80108947, upload-time = "2026-08-18T06:36:40.637Z" },
{ url = "https://files.pythonhosted.org/packages/60/4c/d92b08a2151d095b9a3725fa0e71b215ec686d40bbd5d8b702d21a9b5eaa/pyside6_essentials-6.11.2-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:d3ec6e1885c46e57f16364f38ea8040ffc5dc0b18341058f608fede3ba930567", size = 79575497, upload-time = "2026-08-18T06:36:46.97Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/09/d2662684aee4d3f027ff9b1e965f22301bc28ffc5b7f167658508e4475dc/pyside6_essentials-6.11.2-cp310-abi3-win_amd64.whl", hash = "sha256:c8a29def77032773a30879f7f24415b5395ad08592d147c170824ef4c735dfc1", size = 76913043, upload-time = "2026-08-18T06:36:54.038Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/87/4dd2a6468cea600da1fcf9c7f02bc248eeaa98cdc75658e48e669d95326a/pyside6_essentials-6.11.2-cp310-abi3-win_arm64.whl", hash = "sha256:fadd75c5c20800d64dd0a586ea8cb337c5e630aa2246df0e5105738afa46e02c", size = 58125520, upload-time = "2026-08-18T06:36:59.073Z" },
]
[[package]]
@@ -4886,52 +4851,16 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/95/b0/c3d59ac76f3db873e41bd0cb4fc30b352a278da3289217985aaae3650211/sherpa_onnx_core-1.13.4-py3-none-win_amd64.whl", hash = "sha256:0a6949cf0fd83adb9fbcfdf5c27b8907a57f7b48626db703c7f6037be9b61764", size = 16450053, upload-time = "2026-07-07T12:03:05.196Z" },
]
-[[package]]
-name = "shiboken6"
-version = "6.11.1"
-source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "(python_full_version >= '3.14' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
- "python_full_version >= '3.14' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version >= '3.14' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.13.*' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version == '3.13.*' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version >= '3.14' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "python_full_version == '3.13.*' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "(python_full_version == '3.13.*' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version == '3.13.*' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
- "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "python_full_version < '3.12' and platform_machine == 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.12.*' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version == '3.12.*' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version < '3.12' and platform_machine == 'arm64' and sys_platform == 'darwin'",
- "python_full_version < '3.12' and platform_machine != 'arm64' and platform_machine != 'x86_64' and sys_platform == 'darwin'",
- "python_full_version == '3.12.*' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "python_full_version < '3.12' and platform_machine == 'ARM64' and sys_platform == 'win32'",
- "(python_full_version == '3.12.*' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version == '3.12.*' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
- "(python_full_version < '3.12' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version < '3.12' and sys_platform != 'darwin' and sys_platform != 'linux' and sys_platform != 'win32')",
-]
-wheels = [
- { url = "https://files.pythonhosted.org/packages/17/f3/f2b63df0251e7cd3172ea28e32ede52739de9566bcefcd0178681538ac81/shiboken6-6.11.1-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:1a16867f103ef1c662a5f09dfed03273a9f81688b174555162c58e83650a3f02", size = 476874, upload-time = "2026-05-13T09:47:01.091Z" },
- { url = "https://files.pythonhosted.org/packages/52/b5/3f6fb2ee65b534193fb4ef713dd619dc31dadff5d12c16979a7699ad58be/shiboken6-6.11.1-cp310-abi3-win_amd64.whl", hash = "sha256:c2c6863aa80ec18c0f82cea3417837b279cdc60024ac17123461dc9042577df7", size = 1223647, upload-time = "2026-05-13T09:47:05.924Z" },
- { url = "https://files.pythonhosted.org/packages/98/d1/f15ca0e1666faae02c945f48e745ea35f8fcd8243b176109b4e2c4251f47/shiboken6-6.11.1-cp310-abi3-win_arm64.whl", hash = "sha256:7c8d9af17db4495d4fa5b1c393f218311c4855546b9dfa6a0bd21bcd66b55e9d", size = 1784170, upload-time = "2026-05-13T09:47:07.617Z" },
-]
-
[[package]]
name = "shiboken6"
version = "6.11.2"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version < '3.12' and platform_machine == 'x86_64' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version < '3.12' and platform_machine != 'x86_64' and sys_platform == 'linux'",
-]
wheels = [
+ { url = "https://files.pythonhosted.org/packages/47/44/11bf71c36e71936ab4e923e5dd23599dab527c4488b01860b0f689ce9947/shiboken6-6.11.2-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:53659683b1f7a08e9f87eff9b1065f1ceb7110cd7a4bc09fdf5efe43d286604d", size = 476674, upload-time = "2026-08-18T06:37:11.526Z" },
{ url = "https://files.pythonhosted.org/packages/a5/9a/f41bea4316d0c1bec4800842dcb4bd8fd3e29d6a79c496bc32e1aec42583/shiboken6-6.11.2-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:7a7a0a72a9ed26c9bf77d42246b1c736486befb8f31aa2fb29957ea4cdd1c1c2", size = 272435, upload-time = "2026-08-18T06:37:13.397Z" },
{ url = "https://files.pythonhosted.org/packages/56/05/79ed0697515d4fc7435459e4ed4c4a3b5621a55b7c91d0f3f2cc3d617aa0/shiboken6-6.11.2-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:4bbbd6fa4d7cff5ec5e12bc4c10e1d845fa30c89457ecb13cc64f7bddb77f6f9", size = 270668, upload-time = "2026-08-18T06:37:15.025Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/6b/fee1b4814317e4664e318644c9c007676c65a71e5b2b9d3ac4121a94dc0a/shiboken6-6.11.2-cp310-abi3-win_amd64.whl", hash = "sha256:6ab0eba1c904455df621f9a6df3ca2bb896bab8670572d2bc4e37804ae91f19a", size = 1226578, upload-time = "2026-08-18T06:37:18.002Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/b5/80c4e5aafeb8ac777cf5a1becfe87ac7197c35051df00b5f7819787e09d5/shiboken6-6.11.2-cp310-abi3-win_arm64.whl", hash = "sha256:97c49432488df958f0308736f3d15b6915d8826fc380af17bb4b9ec5c4a5e81b", size = 1784669, upload-time = "2026-08-18T06:37:21.046Z" },
]
[[package]]
@@ -4970,6 +4899,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" },
]
+[[package]]
+name = "soundcard"
+version = "0.4.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cffi", marker = "sys_platform != 'darwin'" },
+ { name = "numpy", version = "2.1.3", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.14' and platform_machine != 'ARM64' and sys_platform == 'win32') or (python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'win32')" },
+ { name = "numpy", version = "2.3.5", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.14' and sys_platform == 'win32') or (python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'win32') or (platform_machine == 'ARM64' and sys_platform == 'win32')" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f5/ef/85322a77399b19b23bd741ad57d86677687cdf9c9d9bc01997bc6bf9d30d/soundcard-0.4.6.tar.gz", hash = "sha256:9b46d64a26f97cd7d88bcfc385c97c069f97c5ec3404e4e7c23771598a8cf47b", size = 40830, upload-time = "2026-04-12T13:46:24.743Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4c/be/eb381e700a61b4ce899e14850d7ae78f9600fd2f8b58b4c52b8714e81917/soundcard-0.4.6-py3-none-any.whl", hash = "sha256:ab44b8c1033ac6ec49c5d618ce096dab9b60c7d7a427c92bef15856e5ebb3e09", size = 43939, upload-time = "2026-04-12T13:46:23.224Z" },
+]
+
[[package]]
name = "sounddevice"
version = "0.5.5"