Personal AI, On Personal Devices
Python
Revert "Merge pull request #1119 from AzeftheGoat187/azefthegoat187-jarvis-dashboard"
Elliot Slusky9eefe90d27b4
diff --git a/docs/architecture/design-principles.md b/docs/architecture/design-principles.mdindex 5d6e1412..54b09712 100644--- a/docs/architecture/design-principles.md+++ b/docs/architecture/design-principles.md@@ -94,7 +94,6 @@ OpenJarvis is designed to work **entirely without network access**. All core fun| Traces | SQLite-based, fully local || Tools | Calculator, Think, FileRead all local || Configuration | TOML file on disk |-| Hand tracking | Optional MediaPipe model and WASM assets run locally after installation |Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cloud` backend, but they are:@@ -102,14 +101,6 @@ Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cl- Only activated if API keys are set as environment variables- Never required for any core functionality-Offline operation still requires the local API server, inference runtime, and model to-be installed and available. The desktop app and PWA can cache their interface assets,-but API inference requests are not cached or replayed offline. Speech recognition is-offline only when a local speech backend and its model are configured; remote speech-providers require network access and may receive audio. Hand tracking is opt-in and-uses the bundled model locally after the app has been installed or its PWA assets-cached.-```python# This works without any network connectionfrom openjarvis import Jarvisdiff --git a/docs/user-guide/chat-simple.md b/docs/user-guide/chat-simple.mdindex c70bfb66..4bab5538 100644--- a/docs/user-guide/chat-simple.md+++ b/docs/user-guide/chat-simple.md@@ -110,60 +110,6 @@ jarvis serveThe server is OpenAI-compatible, so any client that works with the OpenAI API can point to `http://localhost:8000/v1`.-## Connecting Colibri--OpenJarvis can use a Colibri instance that exposes OpenAI-compatible Chat-Completions and model-list endpoints. In desktop **Settings → Inference source**,-choose **Custom OpenAI-compatible server**, then click **Use Colibri preset**.-Configure Colibri separately to bind to loopback on port `8001` (Jarvis uses port-`8000` by default). Enter the exact model ID reported by Colibri, add its API key-only if Colibri requires one, save, and restart OpenJarvis. The preset only sets-OpenJarvis's connection target; it does not install or start Colibri. To find the-model ID, request `http://127.0.0.1:8001/v1/models` (for example,-`curl http://127.0.0.1:8001/v1/models`) and copy the exact `id` value. Enter the-base URL as either `http://127.0.0.1:8001` or `http://127.0.0.1:8001/v1`; the-desktop normalizes either form so requests use one `/v1` prefix. Optional API-keys are stored in desktop secure storage and sent as bearer authorization.-Keep Colibri bound to loopback for local use; the preset does not change its bind-address or expose it to the network.--This connects Colibri for inference; OpenJarvis remains responsible for running-its own configured tools when the selected agent requests them. Colibri's optional-MCP server manages Colibri's lifecycle and is not itself a tool executor. Colibri-serves one active model at a time, so multi-model routing remains an OpenJarvis-capability and is not provided by this connection.--Ollama is already a supported local inference source and its installed models are-listed in Settings. Maya AI is not integrated: the reviewed assistant script does-not expose a supported service or MCP tool interface and contains direct OS-actions. Do not run or wrap that script as an OpenJarvis tool.--## Voice, camera, and offline use--Microphone transcription requires the Speech-to-Text setting and a configured-speech backend. Offline transcription requires a local backend and model; a remote-provider can send audio off-device. Camera hand tracking is optional in-**Settings → Hand tracking → Open hand tracking in chat**, or the **Hand tracking**-button above the chat input. Opening the panel does not start capture. It detects-landmarks and basic gestures locally, and stops when stopped, when the panel closes,-when you leave chat, or when the app/tab is hidden. Returning does not restart the-camera. The first camera action briefly requests permission to reveal-camera names, then immediately stops that temporary stream. Select the intended-device (for example, **Integrated Webcam**) from the camera list before pressing-**Start camera**. Open palm can optionally focus the chat input; gestures never-submit messages or call tools. It uses the bundled MediaPipe Tasks Vision runtime-and [Hand Landmarker model](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker).-The browser's online indicator is only a connectivity hint, not proof that a-particular server is reachable.--Local inference and cached interface assets do not make the assistant universally-autonomous. No financial or business-venture integration is configured here, and-OpenJarvis cannot guarantee legal compliance or revenue. It does not autonomously-retrain model weights; its existing memory, traces, and learning policies are-separate capabilities. MCP tools should be narrowly allowlisted, with integration--level approval and audit controls for financial transactions, contracts, public-communications, and other irreversible actions.-## Using the Desktop App1. Start the backend: `jarvis serve` (or `./scripts/quickstart.sh`)diff --git a/docs/user-guide/mcp-external-servers.md b/docs/user-guide/mcp-external-servers.mdindex 1ddbbe89..99a54616 100644--- a/docs/user-guide/mcp-external-servers.md+++ b/docs/user-guide/mcp-external-servers.md@@ -18,11 +18,6 @@ If a server is unreachable or returns an error, OpenJarvis logs a warning and coExternal MCP servers are configured in `config.toml` under `[tools.mcp]`. The`servers` field accepts either inline JSON or a path to a JSON file:-The default config path is `~/.openjarvis/config.toml` (on Windows,-`%USERPROFILE%\.openjarvis\config.toml`). Run `jarvis config path` to confirm-the active path. The dashboard does not currently provide an MCP connection-editor; configure the backend's file and restart the OpenJarvis backend.-```toml[tools.mcp]enabled = true@@ -74,7 +69,6 @@ Each server object supports the following fields:| `url` | string | No* | URL for Streamable HTTP transport. || `command` | string | No* | Command to launch a stdio-based MCP server. || `args` | list of strings| No | Arguments passed to the stdio command. |-| `token` | string | No | Bearer token sent as the HTTP `Authorization` header. || `include_tools` | list of strings| No | Whitelist of tool names to import. Only these tools are loaded. || `exclude_tools` | list of strings| No | Blacklist of tool names to skip. All other tools are loaded. |@@ -82,38 +76,6 @@ Each server object supports the following fields:When both `include_tools` and `exclude_tools` are specified, the whitelist is applied first, then the blacklist filters the result.-### Authentication and provider endpoints--OpenJarvis supports a bearer token for Streamable HTTP MCP servers, but does-not implement an interactive OAuth login flow or arbitrary HTTP header-configuration. The `token` value is stored in the local config (or MCP JSON-file) as plain text: do not commit or share that file, and restrict access to-it. Never paste credentials into chat.--Some MCP providers require browser-based OAuth rather than a static bearer-token. Those servers are not directly confirmed to work with OpenJarvis until-their authentication flow has been tested. A third-party stdio OAuth bridge-such as [`mcp-remote`](https://github.com/punkpeye/mcp-remote) may be-structurally usable through the `command`/`args` transport, but OpenJarvis has-not validated that bridge or its OAuth flow on Windows; it also runs-third-party code and stores its own OAuth state locally. Treat it as an-unverified option, not built-in OpenJarvis support.--URLs in the GitHub MCP Registry API (for example,-`api.mcp.github.com/oss/v0.1/servers/.../versions/latest`) return registry-metadata; they are not MCP service endpoints. Obtain the runtime endpoint and-supported authentication method from the provider's account dashboard or-official connection guide. Zapier, Vibe Prospecting, Tallyfy, and Hermoso have-appeared in the registry, but their account-specific endpoints and auth have-not been tested with OpenJarvis.--Use `include_tools` as an allowlist of exact names returned by that server's-`tools/list`, and grant only the provider-side permissions needed for those-tools. Do not enable write, outreach, publishing, advertising, payment, or-other spend-capable actions until you have reviewed them and set explicit-approval and budget limits. Tool filtering in OpenJarvis is not a substitute-for limiting permissions in the provider account.-## Examples### Home Assistant via Streamable HTTPdiff --git a/frontend/package-lock.json b/frontend/package-lock.jsonindex 80afcea6..07ff1ddf 100644--- a/frontend/package-lock.json+++ b/frontend/package-lock.json@@ -10,7 +10,6 @@"dependencies": {"@base-ui/react": "^1.8.0","@fontsource-variable/geist": "^5.3.0",- "@mediapipe/tasks-vision": "^1.1.0","@tailwindcss/vite": "^4.3.3","@tauri-apps/api": "^2.12.1","@tauri-apps/plugin-autostart": "^2",@@ -46,7 +45,6 @@"@types/react": "^19.3.0","@types/react-dom": "^19.3.0","@vitejs/plugin-react": "^6.1.2",- "happy-dom": "^20.14.6","typescript": "~7.0.2","vite": "^8.3.4","vite-plugin-pwa": "^2.0.0",@@ -1788,12 +1786,6 @@"@jridgewell/sourcemap-codec": "^1.4.14"}},- "node_modules/@mediapipe/tasks-vision": {- "version": "1.1.0",- "resolved": "https://registry.npmjs.org/@mediapipe/tasks-vision/-/tasks-vision-1.1.0.tgz",- "integrity": "sha512-ZJqh0wMKOINorfSffvDAxzyO1//c+FiG3IkzOdNDnLqvBB3DbMfj0bB/tys2SNf9S914sgoIbIpF4HTOi336cg==",- "license": "Apache-2.0"- },"node_modules/@oxc-project/types": {"version": "0.153.0","resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.153.0.tgz",@@ -3379,16 +3371,6 @@"integrity": "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==","license": "MIT"},- "node_modules/@types/node": {- "version": "26.6.5",- "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.5.tgz",- "integrity": "sha512-uMzsv9stDdofIZBx4UGh4Z9ZgXnGC9seRDscilBcTHAeOFk67MZg/k9bWVfO+bk4qZmG+l52hRnLwUiH2LnJTg==",- "devOptional": true,- "license": "MIT",- "dependencies": {- "undici-types": "~8.9.0"- }- },"node_modules/@types/react": {"version": "19.3.0","resolved": "https://registry.npmjs.org/@types/react/-/react-19.3.0.tgz",@@ -3434,23 +3416,6 @@"integrity": "sha512-zFDAD+tlpf2r4asuHEj0XH6pY6i0g5NeAHPn+15wk3BV6JA69eERFXC1gyGThDkVa1zCyKr5jox1+2LbV/AMLg==","license": "MIT"},- "node_modules/@types/whatwg-mimetype": {- "version": "3.0.2",- "resolved": "https://registry.npmjs.org/@types/whatwg-mimetype/-/whatwg-mimetype-3.0.2.tgz",- "integrity": "sha512-c2AKvDT8ToxLIOUlN51gTiHXflsfIFisS4pO7pDPoKouJCESkhZnEy623gwP9laCy5lnLDAw1vAzu2vM2YLOrA==",- "dev": true,- "license": "MIT"- },- "node_modules/@types/ws": {- "version": "8.18.2",- "resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.2.tgz",- "integrity": "sha512-67MQl+fpWKVTT1NYdnmo3U4sc/xPo/zQBncVnI74qmQa0z/b+1g6iYqNmGCPbxO+zz2aklb08a0oHfegiVd0/w==",- "dev": true,- "license": "MIT",- "dependencies": {- "@types/node": "*"- }- },"node_modules/@typescript/typescript-aix-ppc64": {"version": "7.0.2","resolved": "https://registry.npmjs.org/@typescript/typescript-aix-ppc64/-/typescript-aix-ppc64-7.0.2.tgz",@@ -4136,19 +4101,6 @@"devOptional": true,"license": "MIT"},- "node_modules/buffer-image-size": {- "version": "0.6.4",- "resolved": "https://registry.npmjs.org/buffer-image-size/-/buffer-image-size-0.6.4.tgz",- "integrity": "sha512-nEh+kZOPY1w+gcCMobZ6ETUp9WfibndnosbpwB1iJk/8Gt5ZF2bhS6+B6bPYz424KtwsR6Rflc3tCz1/ghX2dQ==",- "dev": true,- "license": "MIT",- "dependencies": {- "@types/node": "*"- },- "engines": {- "node": ">=4.0"- }- },"node_modules/call-bind": {"version": "1.0.9","resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.9.tgz",@@ -5388,38 +5340,6 @@"integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==","license": "ISC"},- "node_modules/happy-dom": {- "version": "20.14.6",- "resolved": "https://registry.npmjs.org/happy-dom/-/happy-dom-20.14.6.tgz",- "integrity": "sha512-5DDsf4QmEIlRL6NkCv/fxSjYHHkZceL/IaYQdJc2h3G7DRTEai++VTjbfL0LB9geW2VrA3+vrjF3Efx+OTtX0Q==",- "dev": true,- "license": "MIT",- "dependencies": {- "@types/node": ">=20.0.0",- "@types/whatwg-mimetype": "^3.0.2",- "@types/ws": "^8.18.1",- "buffer-image-size": "^0.6.4",- "entities": "^7.0.1",- "whatwg-mimetype": "^3.0.0",- "ws": "^8.21.0"- },- "engines": {- "node": ">=20.0.0"- }- },- "node_modules/happy-dom/node_modules/entities": {- "version": "7.0.1",- "resolved": "https://registry.npmjs.org/entities/-/entities-7.0.1.tgz",- "integrity": "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==",- "dev": true,- "license": "BSD-2-Clause",- "engines": {- "node": ">=0.12"- },- "funding": {- "url": "https://github.com/fb55/entities?sponsor=1"- }- },"node_modules/has-bigints": {"version": "1.1.0","resolved": "https://registry.npmjs.org/has-bigints/-/has-bigints-1.1.0.tgz",@@ -9197,13 +9117,6 @@"url": "https://github.com/sponsors/ljharb"}},- "node_modules/undici-types": {- "version": "8.9.0",- "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz",- "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==",- "devOptional": true,- "license": "MIT"- },"node_modules/unicode-canonical-property-names-ecmascript": {"version": "2.0.1","resolved": "https://registry.npmjs.org/unicode-canonical-property-names-ecmascript/-/unicode-canonical-property-names-ecmascript-2.0.1.tgz",@@ -9986,16 +9899,6 @@"integrity": "sha512-rLcLXA2sx6+9dE88NHFubwTtGxpK4yYBLj6qHPdFoCaLr0cXGb4efOqtKLlm4loGA4OEKHIQKMKzZkKyOh5ctw==","license": "Apache-2.0"},- "node_modules/whatwg-mimetype": {- "version": "3.0.0",- "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-3.0.0.tgz",- "integrity": "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q==",- "dev": true,- "license": "MIT",- "engines": {- "node": ">=12"- }- },"node_modules/which-boxed-primitive": {"version": "1.1.1","resolved": "https://registry.npmjs.org/which-boxed-primitive/-/which-boxed-primitive-1.1.1.tgz",@@ -10379,28 +10282,6 @@"workbox-core": "7.4.1"}},- "node_modules/ws": {- "version": "8.22.0",- "resolved": "https://registry.npmjs.org/ws/-/ws-8.22.0.tgz",- "integrity": "sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==",- "dev": true,- "license": "MIT",- "engines": {- "node": ">=10.0.0"- },- "peerDependencies": {- "bufferutil": "^4.0.1",- "utf-8-validate": ">=5.0.2"- },- "peerDependenciesMeta": {- "bufferutil": {- "optional": true- },- "utf-8-validate": {- "optional": true- }- }- },"node_modules/yallist": {"version": "3.1.1","resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz",diff --git a/frontend/package.json b/frontend/package.jsonindex 5b5f19e1..0a942806 100644--- a/frontend/package.json+++ b/frontend/package.json@@ -19,7 +19,6 @@"dependencies": {"@base-ui/react": "^1.8.0","@fontsource-variable/geist": "^5.3.0",- "@mediapipe/tasks-vision": "^1.1.0","@tailwindcss/vite": "^4.3.3","@tauri-apps/api": "^2.12.1","@tauri-apps/plugin-autostart": "^2",@@ -55,7 +54,6 @@"@types/react": "^19.3.0","@types/react-dom": "^19.3.0","@vitejs/plugin-react": "^6.1.2",- "happy-dom": "^20.14.6","typescript": "~7.0.2","vite": "^8.3.4","vite-plugin-pwa": "^2.0.0",diff --git a/frontend/public/models/hand_landmarker.task b/frontend/public/models/hand_landmarker.taskdeleted file mode 100644index 0d53faf3..00000000Binary files a/frontend/public/models/hand_landmarker.task and /dev/null differdiff --git a/frontend/src-tauri/Entitlements.plist b/frontend/src-tauri/Entitlements.plistindex 724656aa..da5ee242 100644--- a/frontend/src-tauri/Entitlements.plist+++ b/frontend/src-tauri/Entitlements.plist@@ -8,8 +8,6 @@<true/><key>com.apple.security.network.server</key><true/>- <key>com.apple.security.device.camera</key>- <true/><key>com.apple.security.files.user-selected.read-write</key><true/><key>com.apple.security.cs.allow-jit</key>diff --git a/frontend/src-tauri/Info.plist b/frontend/src-tauri/Info.plistindex a4faca0f..794e7a46 100644--- a/frontend/src-tauri/Info.plist+++ b/frontend/src-tauri/Info.plist@@ -2,8 +2,6 @@<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict>- <key>NSCameraUsageDescription</key>- <string>Use your selected camera for optional on-device hand tracking in chat. Frames are not recorded or sent to a server.</string><key>NSAppTransportSecurity</key><dict><!-- The desktop API URL is user-configurable, so it cannot be representeddiff --git a/frontend/src-tauri/tauri.conf.json b/frontend/src-tauri/tauri.conf.jsonindex 262c5c0d..d68eb5ac 100644--- a/frontend/src-tauri/tauri.conf.json+++ b/frontend/src-tauri/tauri.conf.json@@ -24,7 +24,7 @@}],"security": {- "csp": "default-src 'self'; script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; connect-src 'self' http: https: ws: wss:; img-src 'self' data: blob:; media-src 'self' blob:"+ "csp": "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; connect-src 'self' http: https: ws: wss:; img-src 'self' data: blob:; media-src 'self' blob:"}},"bundle": {diff --git a/frontend/src/components/Chat/ChatArea.tsx b/frontend/src/components/Chat/ChatArea.tsxindex 5e1c8536..ed886b0e 100644--- a/frontend/src/components/Chat/ChatArea.tsx+++ b/frontend/src/components/Chat/ChatArea.tsx@@ -2,7 +2,6 @@ import { useRef, useEffect, useState, useCallback } from 'react';import { useNavigate } from 'react-router';import { MessageBubble } from './MessageBubble';import { InputArea } from './InputArea';-import { ChatHandTracking } from './ChatHandTracking';import { StreamingDots } from './StreamingDots';import { useAppStore } from '../../lib/store';import { shouldAutoplayFinishedReply, useTtsStore } from '../../lib/tts';@@ -256,7 +255,6 @@ export function ChatArea() {</div>)}</div>- <ChatHandTracking /><InputArea /></div>);diff --git a/frontend/src/components/Chat/ChatHandTracking.tsx b/frontend/src/components/Chat/ChatHandTracking.tsxdeleted file mode 100644index e3b3da0b..00000000--- a/frontend/src/components/Chat/ChatHandTracking.tsx+++ /dev/null@@ -1,41 +0,0 @@-import { Hand } from 'lucide-react';-import { useSearchParams } from 'react-router';-import { HandTrackingPanel } from '../HandTrackingPanel';--/** Keep the opt-in camera and its focus target on the same route. */-export function ChatHandTracking() {- const [searchParams, setSearchParams] = useSearchParams();- const open = searchParams.get('handTracking') === '1';-- const togglePanel = () => {- const next = new URLSearchParams(searchParams);- if (open) next.delete('handTracking');- else next.set('handTracking', '1');- setSearchParams(next, { replace: true });- };-- return (- <div className="shrink-0 px-4 max-w-[var(--chat-max-width)] w-full mx-auto">- <button- type="button"- aria-expanded={open}- aria-controls="chat-hand-tracking"- onClick={togglePanel}- className="flex items-center gap-1.5 text-xs py-2 cursor-pointer"- style={{ color: 'var(--color-text-secondary)' }}- >- <Hand size={14} />- {open ? 'Close hand tracking (stops camera)' : 'Hand tracking'}- </button>- {open && (- <div id="chat-hand-tracking" className="max-h-[45vh] overflow-y-auto pb-3">- <HandTrackingPanel />- <p className="text-xs mt-2" style={{ color: 'var(--color-text-tertiary)' }}>- Camera access starts only when you choose a camera. Closing this panel,- leaving chat, or hiding the app stops it; returning does not restart it.- </p>- </div>- )}- </div>- );-}diff --git a/frontend/src/components/Chat/InputArea.tsx b/frontend/src/components/Chat/InputArea.tsxindex 5c552738..79a0849e 100644--- a/frontend/src/components/Chat/InputArea.tsx+++ b/frontend/src/components/Chat/InputArea.tsx@@ -162,12 +162,6 @@ export function InputArea() {el.style.height = Math.min(el.scrollHeight, 200) + 'px';}, [input]);- useEffect(() => {- const focusInput = () => textareaRef.current?.focus();- window.addEventListener('openjarvis:focus-chat-input', focusInput);- return () => window.removeEventListener('openjarvis:focus-chat-input', focusInput);- }, []);-const stopStreaming = useCallback(() => {abortRef.current?.abort();if (timerRef.current) {diff --git a/frontend/src/components/HandTrackingPanel.desktop.test.ts b/frontend/src/components/HandTrackingPanel.desktop.test.tsdeleted file mode 100644index 1e851adc..00000000--- a/frontend/src/components/HandTrackingPanel.desktop.test.ts+++ /dev/null@@ -1,22 +0,0 @@-import { describe, expect, it } from 'vitest';-import nativeConfig from '../../src-tauri/tauri.conf.json?raw';-import infoPlist from '../../src-tauri/Info.plist?raw';-import entitlements from '../../src-tauri/Entitlements.plist?raw';--describe('packaged desktop hand tracking prerequisites', () => {- it('allows local WASM compilation without permitting JavaScript eval or remote scripts', () => {- const config = JSON.parse(nativeConfig);- const directives = config.app.security.csp.split(';').map((part: string) => part.trim().split(/\s+/));- const scriptSources = directives.find(([name]: string[]) => name === 'script-src')?.slice(1);- expect(scriptSources).toContain("'wasm-unsafe-eval'");- expect(scriptSources).toContain("'self'");- expect(scriptSources).not.toContain("'unsafe-eval'");- expect(scriptSources).not.toContain('*');- expect(scriptSources).not.toContain('https:');- });-- it('declares camera use for macOS permission prompts and hardened runtime', () => {- expect(infoPlist).toMatch(/<key>NSCameraUsageDescription<\/key>\s*<string>[^<]+<\/string>/);- expect(entitlements).toMatch(/<key>com\.apple\.security\.device\.camera<\/key>\s*<true\s*\/>/);- });-});diff --git a/frontend/src/components/HandTrackingPanel.test.ts b/frontend/src/components/HandTrackingPanel.test.tsdeleted file mode 100644index 271cf567..00000000--- a/frontend/src/components/HandTrackingPanel.test.ts+++ /dev/null@@ -1,56 +0,0 @@-import { describe, expect, it } from 'vitest';-import type { NormalizedLandmark } from '@mediapipe/tasks-vision';-import { classifyHandGesture, preferredCameraDevice } from './HandTrackingPanel';--function landmarks(- extendedFingers: number[],- thumbExtended = false,-): NormalizedLandmark[] {- const points = Array.from({ length: 21 }, (_, index) => ({- x: index === 4 && thumbExtended ? 0.8 : index === 3 && thumbExtended ? 0.6 : 0.5,- y: 0.7,- z: 0,- visibility: 1,- }));- points[0].x = 0.5;- points[0].y = 0.8;-- const fingerTips = [8, 12, 16, 20];- const fingerPips = [6, 10, 14, 18];- for (let index = 0; index < 4; index += 1) {- points[fingerTips[index]].y = extendedFingers.includes(index) ? 0.2 : 0.8;- points[fingerPips[index]].y = 0.5;- }- return points;-}--describe('hand gesture classification', () => {- it('recognizes basic hand poses from local landmarks', () => {- expect(classifyHandGesture(landmarks([0, 1, 2, 3], true))).toBe('Open palm');- expect(classifyHandGesture(landmarks([]))).toBe('Fist');- expect(classifyHandGesture(landmarks([0, 1]))).toBe('Peace sign');- expect(classifyHandGesture(landmarks([0]))).toBe('Pointing');- });-- describe('camera device selection', () => {- it('prefers the built-in webcam over a phone camera when both are available', () => {- const cameras = [- { deviceId: 'phone', label: 'Phone Link Camera' },- { deviceId: 'webcam', label: 'Integrated Webcam' },- ];-- expect(preferredCameraDevice(cameras)).toEqual(cameras[1]);- });-- it('uses the first available camera when no built-in camera is listed', () => {- const cameras = [{ deviceId: 'phone', label: 'Phone Link Camera' }];-- expect(preferredCameraDevice(cameras)).toEqual(cameras[0]);- expect(preferredCameraDevice([])).toBeUndefined();- });- });-- it('avoids classifying incomplete landmark sets as a specific gesture', () => {- expect(classifyHandGesture(landmarks([0]).slice(0, 10))).toBe('Hand detected');- });-});diff --git a/frontend/src/components/HandTrackingPanel.tsx b/frontend/src/components/HandTrackingPanel.tsxdeleted file mode 100644index 6d885ab8..00000000--- a/frontend/src/components/HandTrackingPanel.tsx+++ /dev/null@@ -1,424 +0,0 @@-import { useCallback, useEffect, useRef, useState } from 'react';-import {- FilesetResolver,- HandLandmarker,- type NormalizedLandmark,-} from '@mediapipe/tasks-vision';-import simdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_internal.js?url';-import simdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_internal.wasm?url';-import noSimdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.js?url';-import noSimdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.wasm?url';-import { Camera, Hand, StopCircle } from 'lucide-react';--const HAND_CONNECTIONS = [- [0, 1], [1, 2], [2, 3], [3, 4],- [0, 5], [5, 6], [6, 7], [7, 8],- [5, 9], [9, 10], [10, 11], [11, 12],- [9, 13], [13, 14], [14, 15], [15, 16],- [13, 17], [17, 18], [18, 19], [19, 20], [0, 17],-];--export type HandGesture =- | 'Open palm'- | 'Fist'- | 'Peace sign'- | 'Pointing'- | 'Hand detected';--export interface CameraDevice {- deviceId: string;- label: string;-}--export function preferredCameraDevice(devices: CameraDevice[]): CameraDevice | undefined {- return devices.find((device) => /integrated|built[\s-]?in|internal|webcam/i.test(device.label))- ?? devices[0];-}--export function classifyHandGesture(points: NormalizedLandmark[]): HandGesture {- if (points.length < 21) return 'Hand detected';-- const extended = [- points[8].y < points[6].y - 0.02,- points[12].y < points[10].y - 0.02,- points[16].y < points[14].y - 0.02,- points[20].y < points[18].y - 0.02,- ];- const distanceFromWrist = (index: number) =>- Math.hypot(points[index].x - points[0].x, points[index].y - points[0].y);- const thumbExtended = distanceFromWrist(4) > distanceFromWrist(3) * 1.1;- const extendedCount = extended.filter(Boolean).length;-- if (extendedCount === 4 && thumbExtended) return 'Open palm';- if (extendedCount === 0 && !thumbExtended) return 'Fist';- if (extended[0] && extended[1] && !extended[2] && !extended[3]) return 'Peace sign';- if (extended[0] && !extended[1] && !extended[2] && !extended[3]) return 'Pointing';- return 'Hand detected';-}--function localVisionFileset() {- return FilesetResolver.isSimdSupported().then(async (simdSupported) => {- const fileset = await FilesetResolver.forVisionTasks('/');- fileset.wasmLoaderPath = simdSupported ? simdLoaderUrl : noSimdLoaderUrl;- fileset.wasmBinaryPath = simdSupported ? simdWasmUrl : noSimdWasmUrl;- return fileset;- });-}--type TrackerState = 'idle' | 'choosing' | 'starting' | 'active' | 'error';--export function HandTrackingPanel() {- const [state, setState] = useState<TrackerState>('idle');- const [error, setError] = useState('');- const [landmarks, setLandmarks] = useState<NormalizedLandmark[][]>([]);- const [gesture, setGesture] = useState('No hand detected');- const [cameras, setCameras] = useState<CameraDevice[]>([]);- const [selectedCameraId, setSelectedCameraId] = useState('');- const [focusOnOpenPalm, setFocusOnOpenPalm] = useState(- () => localStorage.getItem('openjarvis-hand-focus') === 'true',- );- const focusOnOpenPalmRef = useRef(focusOnOpenPalm);- const videoRef = useRef<HTMLVideoElement>(null);- const detectorRef = useRef<HandLandmarker | null>(null);- const streamRef = useRef<MediaStream | null>(null);- const frameRef = useRef<number | null>(null);- const generationRef = useRef(0);- const previousGestureRef = useRef<string>('');-- const releaseTracking = useCallback(() => {- generationRef.current += 1;- if (frameRef.current !== null) cancelAnimationFrame(frameRef.current);- frameRef.current = null;- const detector = detectorRef.current;- detectorRef.current = null;- // Camera cleanup must still run if the WASM runtime failed during shutdown.- try { detector?.close(); } catch { /* Already failed/closed. */ }- streamRef.current?.getTracks().forEach((track) => track.stop());- streamRef.current = null;- if (videoRef.current) videoRef.current.srcObject = null;- }, []);-- const stopTracking = useCallback(() => {- releaseTracking();- setLandmarks([]);- setGesture('No hand detected');- previousGestureRef.current = '';- setState('idle');- setError('');- }, [releaseTracking]);-- const chooseCamera = useCallback(async () => {- if (!navigator.mediaDevices?.getUserMedia) {- setError('Camera access is not available in this browser or context.');- setState('error');- return;- }-- const generation = ++generationRef.current;- setError('');- setState('choosing');- try {- const permissionStream = await navigator.mediaDevices.getUserMedia({- video: true,- audio: false,- });- if (generationRef.current !== generation) {- permissionStream.getTracks().forEach((track) => track.stop());- return;- }-- // Permission reveals device labels. Stop this temporary stream before- // enumerating or opening the user-selected camera.- permissionStream.getTracks().forEach((track) => track.stop());- const devices = (await navigator.mediaDevices.enumerateDevices())- .filter((device) => device.kind === 'videoinput')- .map(({ deviceId, label }, index) => ({- deviceId,- label: label || `Camera ${index + 1}`,- }));- if (generationRef.current !== generation) return;- if (devices.length === 0) throw new Error('No cameras were found.');-- setCameras(devices);- setSelectedCameraId(preferredCameraDevice(devices)?.deviceId ?? devices[0].deviceId);- setState('idle');- } catch (cause) {- if (generationRef.current !== generation) return;- setError(- cause instanceof DOMException && cause.name === 'NotAllowedError'- ? 'Camera permission was denied. Allow camera access in your browser settings, then try again.'- : cause instanceof Error- ? cause.message- : 'Could not list available cameras.',- );- setState('error');- }- }, []);-- const startTracking = useCallback(async () => {- if (!navigator.mediaDevices?.getUserMedia) {- setError('Camera access is not available in this browser or context.');- setState('error');- return;- }-- const generation = ++generationRef.current;- setError('');- setState('starting');-- try {- const stream = await navigator.mediaDevices.getUserMedia({- video: { deviceId: { exact: selectedCameraId } },- audio: false,- });- if (generationRef.current !== generation) {- stream.getTracks().forEach((track) => track.stop());- return;- }- streamRef.current = stream;- const video = videoRef.current;- if (!video) throw new Error('Camera preview is unavailable.');- video.srcObject = stream;- await video.play();- if (generationRef.current !== generation) return;-- const fileset = await localVisionFileset();- if (generationRef.current !== generation) return;- const detector = await HandLandmarker.createFromOptions(fileset, {- baseOptions: {- modelAssetPath: new URL('/models/hand_landmarker.task', window.location.origin).href,- },- runningMode: 'VIDEO',- numHands: 2,- minHandDetectionConfidence: 0.55,- minTrackingConfidence: 0.55,- });-- if (generationRef.current !== generation) {- detector.close();- stream.getTracks().forEach((track) => track.stop());- return;- }-- detectorRef.current = detector;- setState('active');- let lastVideoTime = -1;- let lastUiUpdate = 0;-- const detectFrame = () => {- if (generationRef.current !== generation) return;- const currentDetector = detectorRef.current;- if (- currentDetector &&- video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA &&- video.currentTime !== lastVideoTime- ) {- lastVideoTime = video.currentTime;- let result;- try {- result = currentDetector.detectForVideo(video, performance.now());- } catch {- stopTracking();- setError('Hand tracking stopped after a camera or detector error. Start the camera to try again.');- setState('error');- return;- }- const detectedHands = result.landmarks;- const currentGesture = detectedHands.length- ? classifyHandGesture(detectedHands[0])- : 'No hand detected';-- if (performance.now() - lastUiUpdate > 80) {- lastUiUpdate = performance.now();- setLandmarks(detectedHands);- setGesture(currentGesture);- }-- if (- focusOnOpenPalmRef.current &&- currentGesture === 'Open palm' &&- previousGestureRef.current !== 'Open palm'- ) {- window.dispatchEvent(new Event('openjarvis:focus-chat-input'));- }- previousGestureRef.current = currentGesture;- }- frameRef.current = requestAnimationFrame(detectFrame);- };- frameRef.current = requestAnimationFrame(detectFrame);- } catch (cause) {- if (generationRef.current !== generation) return;- releaseTracking();- setError(- cause instanceof DOMException && cause.name === 'NotAllowedError'- ? 'Camera permission was denied. Allow camera access in system settings, then try again.'- : cause instanceof Error- ? cause.message- : 'Could not start local hand tracking.',- );- setState('error');- }- }, [selectedCameraId, releaseTracking, stopTracking]);-- useEffect(() => {- const onVisibilityChange = () => {- if (document.hidden) stopTracking();- };- document.addEventListener('visibilitychange', onVisibilityChange);- window.addEventListener('pagehide', stopTracking);- return () => {- document.removeEventListener('visibilitychange', onVisibilityChange);- window.removeEventListener('pagehide', stopTracking);- releaseTracking();- };- }, [releaseTracking, stopTracking]);-- const updateFocusOption = (enabled: boolean) => {- setFocusOnOpenPalm(enabled);- focusOnOpenPalmRef.current = enabled;- localStorage.setItem('openjarvis-hand-focus', String(enabled));- };-- return (- <div className="pt-3">- <div className="flex items-start justify-between gap-4">- <div className="min-w-0">- <div className="text-sm" style={{ color: 'var(--color-text)' }}>Camera hand tracking</div>- <p className="text-xs mt-1" style={{ color: 'var(--color-text-tertiary)' }}>- Optional, on-device landmark and basic gesture detection. Camera frames stay in this app and are not recorded.- </p>- </div>- <button- type="button"- onClick={() => {- if (state === 'active' || state === 'starting' || state === 'choosing') {- stopTracking();- } else if (cameras.length === 0) {- void chooseCamera();- } else {- void startTracking();- }- }}- aria-pressed={state === 'active' || state === 'starting'}- className="flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium cursor-pointer shrink-0"- style={{- background: state === 'active' ? 'var(--color-error)' : 'var(--color-bg-secondary)',- border: '1px solid var(--color-border)',- color: state === 'active' ? 'white' : 'var(--color-text)',- }}- >- {state === 'choosing'- ? <><StopCircle size={13} /> Cancel camera access</>- : state === 'starting'- ? <><StopCircle size={13} /> Stop camera</>- : state === 'active'- ? <><StopCircle size={13} /> Stop camera</>- : cameras.length === 0- ? <><Camera size={13} /> Choose camera</>- : <><Camera size={13} /> Start camera</>}- </button>- </div>-- {cameras.length > 0 && state !== 'active' && (- <div className="flex items-center gap-3 mt-3">- <label htmlFor="openjarvis-camera-device" className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>- Camera device- </label>- <select- id="openjarvis-camera-device"- aria-label="Camera device"- value={selectedCameraId}- onChange={(event) => setSelectedCameraId(event.target.value)}- disabled={state === 'starting'}- className="text-xs px-2 py-1 rounded-lg outline-none max-w-full"- style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}- >- {cameras.map((camera) => (- <option key={camera.deviceId} value={camera.deviceId}>{camera.label}</option>- ))}- </select>- <button- type="button"- onClick={() => void chooseCamera()}- disabled={state === 'choosing' || state === 'starting'}- className="text-xs underline cursor-pointer"- style={{ color: 'var(--color-accent)' }}- >- Refresh- </button>- </div>- )}-- {/* Keep the ref available even when camera permission resolves before React- commits the starting state. An empty video never starts capture. */}- <div className={state === 'active' || state === 'starting' ? 'flex flex-col sm:flex-row items-start gap-4 mt-4' : 'hidden'}>- <div className="relative w-full sm:w-64 aspect-video overflow-hidden rounded-lg bg-black">- <video- ref={videoRef}- muted- playsInline- aria-label="Live camera preview for hand tracking"- className="absolute inset-0 w-full h-full object-cover"- style={{ transform: 'scaleX(-1)' }}- />- <svg- aria-hidden="true"- viewBox="0 0 1 1"- preserveAspectRatio="none"- className="absolute inset-0 w-full h-full"- style={{ transform: 'scaleX(-1)' }}- >- {landmarks.map((hand, handIndex) => (- <g key={handIndex}>- {HAND_CONNECTIONS.map(([start, end]) => (- <line- key={`${start}-${end}`}- x1={hand[start].x}- y1={hand[start].y}- x2={hand[end].x}- y2={hand[end].y}- stroke="#34d399"- strokeWidth="0.009"- />- ))}- {hand.map((point, pointIndex) => (- <circle key={pointIndex} cx={point.x} cy={point.y} r="0.014" fill="#fbbf24" />- ))}- </g>- ))}- </svg>- </div>- <div aria-live="polite" className="flex items-center gap-2 text-xs" style={{ color: 'var(--color-text-secondary)' }}>- <Hand size={15} style={{ color: landmarks.length ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />- {state === 'starting'- ? 'Starting local hand tracker…'- : landmarks.length- ? `${landmarks.length} hand${landmarks.length === 1 ? '' : 's'} — ${gesture}`- : 'No hand detected'}- </div>- </div>-- {state === 'error' && (- <p role="alert" className="text-xs mt-3" style={{ color: 'var(--color-error)' }}>{error}</p>- )}- {cameras.length > 0 && state !== 'active' && (- <p className="text-xs mt-2" style={{ color: 'var(--color-text-tertiary)' }}>- Choose the webcam by name before starting. The camera used to reveal device names was stopped immediately.- </p>- )}-- <label className="flex items-center gap-2 mt-3 text-xs cursor-pointer" style={{ color: 'var(--color-text-secondary)' }}>- <input- type="checkbox"- checked={focusOnOpenPalm}- onChange={(event) => updateFocusOption(event.target.checked)}- className="accent-[var(--color-accent)]"- />- Focus chat input when an open palm is detected- </label>- <p className="text-xs mt-1 pl-5" style={{ color: 'var(--color-text-tertiary)' }}>- Optional and off by default. Gestures never send messages or trigger tools.- </p>- </div>- );-}diff --git a/frontend/src/pages/ChatPage.hand-tracking.test.tsx b/frontend/src/pages/ChatPage.hand-tracking.test.tsxdeleted file mode 100644index bf70686c..00000000--- a/frontend/src/pages/ChatPage.hand-tracking.test.tsx+++ /dev/null@@ -1,247 +0,0 @@-// @vitest-environment happy-dom-// @vitest-environment-options { "url": "http://localhost:5173" }-import { act } from 'react';-import { createRoot, type Root } from 'react-dom/client';-import { createMemoryRouter, RouterProvider } from 'react-router';-import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';-import { ChatPage } from './ChatPage';-import { useAppStore } from '../lib/store';--const vision = vi.hoisted(() => ({- create: vi.fn(),- detect: vi.fn(),- close: vi.fn(),-}));-const send = vi.hoisted(() => vi.fn());--vi.mock('@mediapipe/tasks-vision', () => ({- FilesetResolver: {- isSimdSupported: vi.fn().mockResolvedValue(true),- forVisionTasks: vi.fn().mockResolvedValue({}),- },- HandLandmarker: { createFromOptions: vision.create },-}));-vi.mock('../lib/connectors-api', () => ({ listConnectors: vi.fn().mockResolvedValue([]) }));-vi.mock('../lib/sse', () => ({ streamChat: send, streamResearch: send }));-vi.mock('../components/Chat/MessageBubble', () => ({ MessageBubble: () => null }));-vi.mock('../hooks/useSpeech', () => ({- useSpeech: () => ({ state: 'idle', error: null, available: false }),-}));--function openPalm() {- const points = Array.from({ length: 21 }, () => ({ x: 0.5, y: 0.7, z: 0 }));- points[0].y = 0.8;- points[3].x = 0.6;- points[4].x = 0.8;- for (const tip of [8, 12, 16, 20]) points[tip].y = 0.2;- for (const pip of [6, 10, 14, 18]) points[pip].y = 0.5;- return points;-}--function mockStream() {- const stop = vi.fn();- const stream = new MediaStream();- Object.defineProperty(stream, 'getTracks', { value: () => [{ stop }] });- return { stop, stream };-}--let root: Root;-let container: HTMLDivElement;-let router: ReturnType<typeof createMemoryRouter>;-let frames: Map<number, FrameRequestCallback>;-let camera: ReturnType<typeof mockStream>;-let permission: ReturnType<typeof mockStream>;-let getUserMedia: ReturnType<typeof vi.fn>;-let hidden: boolean;-let videoTime: number;--beforeEach(() => {- (globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;- localStorage.clear();- vi.clearAllMocks();- useAppStore.setState({- selectedModel: 'test-model', modelLoading: false, systemPanelOpen: false,- deepResearch: false, activeId: null, messages: [],- streamState: { ...useAppStore.getState().streamState, isStreaming: false },- });- hidden = false;- videoTime = 0;- vi.spyOn(document, 'hidden', 'get').mockImplementation(() => hidden);- vi.spyOn(HTMLMediaElement.prototype, 'play').mockResolvedValue();- // happy-dom exposes media elements but does not define their readiness constants.- Object.defineProperty(HTMLMediaElement, 'HAVE_CURRENT_DATA', { configurable: true, value: 2 });- vi.spyOn(HTMLMediaElement.prototype, 'readyState', 'get').mockReturnValue(2);- vi.spyOn(HTMLMediaElement.prototype, 'currentTime', 'get').mockImplementation(() => videoTime);- frames = new Map();- let id = 0;- vi.stubGlobal('requestAnimationFrame', vi.fn((callback: FrameRequestCallback) => {- frames.set(++id, callback);- return id;- }));- vi.stubGlobal('cancelAnimationFrame', vi.fn((frame: number) => frames.delete(frame)));- // Catch accidental backend/model downloads rather than making real requests.- vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new Error('Unexpected network request'))));- permission = mockStream();- camera = mockStream();- getUserMedia = vi.fn().mockResolvedValueOnce(permission.stream).mockResolvedValue(camera.stream);- Object.defineProperty(navigator, 'mediaDevices', {- configurable: true,- value: {- getUserMedia,- enumerateDevices: vi.fn().mockResolvedValue([- { kind: 'videoinput', deviceId: 'phone', label: 'Phone camera' },- { kind: 'videoinput', deviceId: 'webcam', label: 'Integrated Webcam' },- ]),- },- });- vision.create.mockResolvedValue({ detectForVideo: vision.detect, close: vision.close });- vision.detect.mockReturnValue({ landmarks: [openPalm()] });- container = document.createElement('div');- document.body.append(container);- root = createRoot(container);-});--afterEach(async () => {- await act(async () => root?.unmount());- router?.dispose();- container?.remove();- vi.restoreAllMocks();- vi.unstubAllGlobals();-});--async function mount(path = '/?handTracking=1') {- router = createMemoryRouter([- { path: '/', element: <ChatPage /> },- { path: '/settings', element: <div>Settings</div> },- ], { initialEntries: [path] });- await act(async () => root.render(<RouterProvider router={router} />));-}--function button(text: string) {- const result = [...container.querySelectorAll('button')].find((node) => node.textContent?.trim() === text);- expect(result, `Button: ${text}`).toBeDefined();- return result!;-}--async function click(text: string) {- await act(async () => button(text).click());-}--async function start() {- await click('Choose camera');- expect(permission.stop).toHaveBeenCalledOnce();- await click('Start camera');- expect(container.querySelector('[role="alert"]')?.textContent).toBeUndefined();-}--async function frame() {- const [id, callback] = frames.entries().next().value!;- frames.delete(id);- videoTime += 1;- await act(async () => callback(performance.now()));-}--describe('chat hand tracking route and camera lifecycle', () => {- it('opens without permission and focuses the real chat input only after opt-in, without sending', async () => {- await mount('/');- expect(container.querySelector('video')).toBeNull();- await click('Hand tracking');- expect(getUserMedia).not.toHaveBeenCalled();- const focusOption = container.querySelector<HTMLInputElement>('input[type="checkbox"]')!;- expect(focusOption.checked).toBe(false);- await start();- button('Stop camera').focus();- await frame();- expect(document.activeElement).toBe(button('Stop camera'));- await act(async () => focusOption.click());- vision.detect.mockReturnValueOnce({ landmarks: [] });- await frame();- await frame();- expect(document.activeElement).toBe(container.querySelector('textarea'));- expect(getUserMedia).toHaveBeenNthCalledWith(1, { video: true, audio: false });- expect(getUserMedia).toHaveBeenNthCalledWith(2, { video: { deviceId: { exact: 'webcam' } }, audio: false });- expect(send).not.toHaveBeenCalled();- expect(useAppStore.getState().messages).toEqual([]);- await click('Close hand tracking (stops camera)');- expect(camera.stop).toHaveBeenCalledOnce();- expect(vision.close).toHaveBeenCalledOnce();- expect(frames.size).toBe(0);- await click('Hand tracking');- expect(getUserMedia).toHaveBeenCalledTimes(2);- });-- it('stops on route changes and does not resume on returning to chat', async () => {- await mount();- await start();- await act(async () => { await router.navigate('/settings'); });- expect(camera.stop).toHaveBeenCalledOnce();- expect(vision.close).toHaveBeenCalledOnce();- expect(frames.size).toBe(0);- await act(async () => { await router.navigate('/?handTracking=1'); });- expect(getUserMedia).toHaveBeenCalledTimes(2);- expect(container.querySelector('video')?.srcObject).toBeFalsy();- });-- it.each(['hidden', 'pagehide', 'stop'])('stops capture on %s and requires another start', async (reason) => {- await mount();- await start();- await act(async () => {- if (reason === 'hidden') {- hidden = true;- document.dispatchEvent(new Event('visibilitychange'));- } else if (reason === 'pagehide') window.dispatchEvent(new Event('pagehide'));- else button('Stop camera').click();- });- expect(camera.stop).toHaveBeenCalledOnce();- expect(vision.close).toHaveBeenCalledOnce();- expect(frames.size).toBe(0);- await act(async () => {- hidden = false;- document.dispatchEvent(new Event('visibilitychange'));- });- expect(getUserMedia).toHaveBeenCalledTimes(2);- expect(button('Start camera')).toBeDefined();- });-- it('closes a permission stream that arrives after leaving chat', async () => {- let resolvePermission!: (stream: MediaStream) => void;- getUserMedia.mockReset().mockReturnValue(new Promise((resolve) => { resolvePermission = resolve; }));- await mount();- await click('Choose camera');- await act(async () => { await router.navigate('/settings'); });- await act(async () => resolvePermission(permission.stream));- expect(permission.stop).toHaveBeenCalledOnce();- expect(vision.create).not.toHaveBeenCalled();- });-- it('stops the camera immediately and closes a detector that finishes loading after cancellation', async () => {- let resolveDetector!: (detector: unknown) => void;- vision.create.mockReturnValueOnce(new Promise((resolve) => { resolveDetector = resolve; }));- await mount();- await start();- await click('Close hand tracking (stops camera)');- expect(camera.stop).toHaveBeenCalledOnce();- await act(async () => resolveDetector({ close: vision.close }));- expect(vision.close).toHaveBeenCalledOnce();- expect(frames.size).toBe(0);- });-- it('reports permission denial without loading the detector', async () => {- getUserMedia.mockReset().mockRejectedValue(new DOMException('Denied', 'NotAllowedError'));- await mount();- await click('Choose camera');- expect(container.querySelector('[role="alert"]')?.textContent).toContain('Camera permission was denied');- expect(vision.create).not.toHaveBeenCalled();- });-- it('releases the camera if inference or detector cleanup fails', async () => {- vision.detect.mockImplementationOnce(() => { throw new Error('WASM failed'); });- vision.close.mockImplementationOnce(() => { throw new Error('Already failed'); });- await mount();- await start();- await frame();- expect(camera.stop).toHaveBeenCalledOnce();- expect(frames.size).toBe(0);- expect(container.querySelector('[role="alert"]')?.textContent).toContain('Hand tracking stopped');- });-});diff --git a/frontend/src/pages/SettingsPage.cloud-providers.test.tsx b/frontend/src/pages/SettingsPage.cloud-providers.test.tsxindex fdf2b400..2e35fe7f 100644--- a/frontend/src/pages/SettingsPage.cloud-providers.test.tsx+++ b/frontend/src/pages/SettingsPage.cloud-providers.test.tsx@@ -1,5 +1,4 @@import { renderToStaticMarkup } from 'react-dom/server';-import { MemoryRouter } from 'react-router';import { describe, expect, it, vi } from 'vitest';describe('desktop cloud provider settings', () => {@@ -7,7 +6,7 @@ describe('desktop cloud provider settings', () => {vi.stubGlobal('localStorage', { getItem: () => null });vi.stubGlobal('window', { __TAURI_INTERNALS__: {} });const { SettingsPage } = await import('./SettingsPage');- const html = renderToStaticMarkup(<MemoryRouter><SettingsPage /></MemoryRouter>);+ const html = renderToStaticMarkup(<SettingsPage />);expect(html).toContain('Atlas Cloud');const atlasKeyInput = html.match(/<input[^>]*placeholder="Atlas Cloud API key"[^>]*>/)?.[0];diff --git a/frontend/src/pages/SettingsPage.tsx b/frontend/src/pages/SettingsPage.tsxindex db6bda27..6b8bdc1d 100644--- a/frontend/src/pages/SettingsPage.tsx+++ b/frontend/src/pages/SettingsPage.tsx@@ -1,5 +1,4 @@import { useState, useEffect, useCallback, useRef } from 'react';-import { Link } from 'react-router';import {Palette,Globe,@@ -18,8 +17,6 @@ import {Search,Brain,RefreshCw,- Server,- Wifi,} from 'lucide-react';import { useAppStore, type ThemeMode } from '../lib/store';import {@@ -232,36 +229,6 @@ function SettingRow({ label, description, children }: { label: string; descripti);}-export function NetworkStatusRow() {- const [online, setOnline] = useState<boolean | null>(- () => typeof navigator === 'undefined' ? null : navigator.onLine,- );-- useEffect(() => {- const updateOnline = () => setOnline(navigator.onLine);- window.addEventListener('online', updateOnline);- window.addEventListener('offline', updateOnline);- return () => {- window.removeEventListener('online', updateOnline);- window.removeEventListener('offline', updateOnline);- };- }, []);-- return (- <SettingRow- label="Browser network"- description="Browser signal only; it does not verify internet or cloud-provider reachability."- >- <div className="flex items-center gap-2">- <Wifi size={14} style={{ color: online ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />- <span className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>- {online === null ? 'Unknown' : online ? 'Online' : 'Offline'}- </span>- </div>- </SettingRow>- );-}-type MemoryStatus =| { kind: 'loading' }| { kind: 'ready'; stats: MemoryStats }@@ -373,7 +340,6 @@ export function SettingsPage() {const [customEngine, setCustomEngine] = useState('lmstudio');const [customKey, setCustomKey] = useState('');const [srcMsg, setSrcMsg] = useState('');- const inferenceConfigurable = isTauri();useEffect(() => {getInferenceSource().then((s) => {@@ -554,7 +520,6 @@ export function SettingsPage() {{/* Connection */}<Section title="Connection">- <NetworkStatusRow /><SettingRow label="Server status" description={serverInfo ? `${serverInfo.engine} / ${serverInfo.model}` : 'Not connected'}><div className="flex items-center gap-2"><span@@ -599,14 +564,10 @@ export function SettingsPage() {{/* Inference source */}<Section title="Inference source">- <p className="text-xs mb-2" style={{ color: 'var(--color-text-tertiary)' }}>- Ollama is supported for local inference. The desktop app can connect to Colibri through its OpenAI-compatible API; OpenJarvis agents provide multi-step tool use and reasoning, while Colibri serves one model at a time. Maya AI is not wired in: its reviewed assistant script has no supported service or plugin API, so it cannot be safely managed here.- </p>- <SettingRow label="Source" description={inferenceConfigurable ? 'Where the app runs models. Applies after restart.' : 'The browser uses the backend’s configured model source; change inference sources in the desktop app.'}>+ <SettingRow label="Source" description="Where the app runs models. Applies after restart."><selectvalue={srcKind}onChange={(e) => { setSrcKind(e.target.value as InferenceSource['kind']); setSrcMsg(''); }}- disabled={!inferenceConfigurable}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>@@ -616,39 +577,21 @@ export function SettingsPage() {</SettingRow>{srcKind === 'custom' && (<>- <SettingRow label="Colibri preset" description="Sets the desktop app's LM Studio-compatible adapter target; configure Colibri separately and enter its exact model ID below.">- <button- type="button"- onClick={() => {- setSrcKind('custom');- setCustomEngine('lmstudio');- setCustomHost('http://127.0.0.1:8001');- setCustomModel('');- setSrcMsg('Colibri preset selected. Enter a model ID exposed by Colibri, then save and restart.');- }}- disabled={!inferenceConfigurable}- title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}- className="text-xs px-3 py-1.5 rounded-lg cursor-pointer"- style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}- >- <span className="flex items-center gap-1.5"><Server size={13} /> Use Colibri preset</span>- </button>- </SettingRow>- <SettingRow label="Server URL" description={customEngine === 'lmstudio' ? 'Base URL. For Colibri, configure loopback:8001 to avoid Jarvis’s default port 8000.' : 'OpenAI-compatible server base URL'}>- <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1" disabled={!inferenceConfigurable}+ <SettingRow label="Server URL" description="e.g. LM Studio: http://localhost:1234/v1">+ <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1"className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} /></SettingRow>- <SettingRow label="Model" description={customEngine === 'lmstudio' ? 'Enter the exact model id from your server’s GET /v1/models response.' : 'Model id served by your endpoint'}>- <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct" disabled={!inferenceConfigurable}+ <SettingRow label="Model" description="Model id served by your endpoint">+ <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct"className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} /></SettingRow>- <SettingRow label="Server type" description={customEngine === 'lmstudio' ? 'LM Studio adapter; also compatible with Colibri’s OpenAI-style /v1 API.' : 'OpenAI-compatible inference backend'}>- <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }} disabled={!inferenceConfigurable}+ <SettingRow label="Server type" description="OpenAI-compatible engine">+ <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>- <option value="lmstudio">LM Studio / Colibri-compatible</option>+ <option value="lmstudio">LM Studio</option><option value="vllm">vLLM</option><option value="sglang">SGLang</option><option value="llamacpp">llama.cpp</option>@@ -656,7 +599,7 @@ export function SettingsPage() {</select></SettingRow><SettingRow label="API key (optional)" description="Only if your server requires one">- <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none" disabled={!inferenceConfigurable}+ <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none"className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} /></SettingRow>@@ -664,10 +607,8 @@ export function SettingsPage() {)}<SettingRow label="" description={srcMsg}><button onClick={saveSource}- disabled={!inferenceConfigurable}- title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}className="text-sm px-3 py-1.5 rounded-lg outline-none cursor-pointer"- style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}>+ style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>Save inference source</button></SettingRow>@@ -935,22 +876,6 @@ export function SettingsPage() {See the <a href="https://open-jarvis.github.io/OpenJarvis/user-guide/tools/" target="_blank" rel="noopener noreferrer" style={{ color: 'var(--color-accent)' }}>documentation</a> for details.</div>)}- <p className="text-xs mt-3" style={{ color: 'var(--color-text-tertiary)' }}>- Offline speech requires a configured local backend and downloaded model. Remote speech providers may send audio off-device.- </p>- </Section>-- {/* Hand tracking */}- <Section title="Hand tracking">- <SettingRow label="Camera gestures" description="Open camera controls alongside the chat input. Camera access requires an explicit action and stops when you close the panel or leave chat.">- <Link- to="/?handTracking=1"- className="text-xs underline"- style={{ color: 'var(--color-accent)' }}- >- Open hand tracking in chat- </Link>- </SettingRow></Section>{/* Data */}diff --git a/frontend/vite.config.ts b/frontend/vite.config.tsindex 57ca3f31..360565cd 100644--- a/frontend/vite.config.ts+++ b/frontend/vite.config.ts@@ -37,8 +37,7 @@ export default defineConfig({],},workbox: {- globPatterns: ['**/*.{js,css,html,ico,png,svg,wasm,task}'],- maximumFileSizeToCacheInBytes: 16 * 1024 * 1024,+ globPatterns: ['**/*.{js,css,html,ico,png,svg}'],navigateFallbackDenylist: [/^\/v1\//, /^\/health/, /^\/dashboard/, /^\/api\//],},}),diff --git a/tests/engine/test_openai_compat_api_key.py b/tests/engine/test_openai_compat_api_key.pyindex 65a9d229..dc541358 100644--- a/tests/engine/test_openai_compat_api_key.py+++ b/tests/engine/test_openai_compat_api_key.py@@ -2,15 +2,12 @@from __future__ import annotations-import json-import httpximport pytestimport respxfrom openjarvis.core.types import Message, Rolefrom openjarvis.engine.openai_compat_engines import (- LMStudioEngine,OpenAICompatEngine,VLLMEngine,normalize_openai_base_url,@@ -82,33 +79,6 @@ class TestAuthorizationHeader:engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")assert engine._api_key == "sk-explicit"- def test_colibri_compatible_lmstudio_engine_uses_model_and_bearer_key(self) -> None:- model_id = "colibri/qwen3-8b"- engine = LMStudioEngine(host="http://127.0.0.1:8001", api_key="coli-test")- with respx.mock:- route = respx.post("http://127.0.0.1:8001/v1/chat/completions").mock(- return_value=httpx.Response(200, json=_CHAT_RESPONSE)- )- engine.generate([Message(role=Role.USER, content="hi")], model=model_id)- request = route.calls.last.request- assert request.headers["Authorization"] == "Bearer coli-test"- assert request.url.path == "/v1/chat/completions"- assert json.loads(request.content)["model"] == model_id-- def test_lmstudio_model_list_uses_single_v1_prefix_and_optional_auth(self) -> None:- host = normalize_openai_base_url("http://127.0.0.1:8001/v1")- engine = LMStudioEngine(host=host, api_key="coli-test")- with respx.mock:- route = respx.get("http://127.0.0.1:8001/v1/models").mock(- return_value=httpx.Response(- 200, json={"data": [{"id": "colibri/qwen3-8b"}]}- )- )- assert engine.list_models() == ["colibri/qwen3-8b"]- request = route.calls.last.request- assert request.url.path == "/v1/models"- assert request.headers["Authorization"] == "Bearer coli-test"-class TestNormalizeOpenAIBaseUrl:@pytest.mark.parametrize(