Skip to content

Personal AI, On Personal Devices

Python

Revert "Merge pull request #1119 from AzeftheGoat187/azefthegoat187-jarvis-dashboard"

Elliot Slusky9eefe90d27b4
diff --git a/docs/architecture/design-principles.md b/docs/architecture/design-principles.md
index 5d6e1412..54b09712 100644
--- a/docs/architecture/design-principles.md
+++ b/docs/architecture/design-principles.md
@@ -94,7 +94,6 @@ OpenJarvis is designed to work **entirely without network access**. All core fun
| Traces | SQLite-based, fully local |
| Tools | Calculator, Think, FileRead all local |
| Configuration | TOML file on disk |
-| Hand tracking | Optional MediaPipe model and WASM assets run locally after installation |
Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cloud` backend, but they are:
@@ -102,14 +101,6 @@ Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cl
- Only activated if API keys are set as environment variables
- Never required for any core functionality
-Offline operation still requires the local API server, inference runtime, and model to
-be installed and available. The desktop app and PWA can cache their interface assets,
-but API inference requests are not cached or replayed offline. Speech recognition is
-offline only when a local speech backend and its model are configured; remote speech
-providers require network access and may receive audio. Hand tracking is opt-in and
-uses the bundled model locally after the app has been installed or its PWA assets
-cached.
-
```python
# This works without any network connection
from openjarvis import Jarvis
diff --git a/docs/user-guide/chat-simple.md b/docs/user-guide/chat-simple.md
index c70bfb66..4bab5538 100644
--- a/docs/user-guide/chat-simple.md
+++ b/docs/user-guide/chat-simple.md
@@ -110,60 +110,6 @@ jarvis serve
The server is OpenAI-compatible, so any client that works with the OpenAI API can point to `http://localhost:8000/v1`.
-## Connecting Colibri
-
-OpenJarvis can use a Colibri instance that exposes OpenAI-compatible Chat
-Completions and model-list endpoints. In desktop **Settings → Inference source**,
-choose **Custom OpenAI-compatible server**, then click **Use Colibri preset**.
-Configure Colibri separately to bind to loopback on port `8001` (Jarvis uses port
-`8000` by default). Enter the exact model ID reported by Colibri, add its API key
-only if Colibri requires one, save, and restart OpenJarvis. The preset only sets
-OpenJarvis's connection target; it does not install or start Colibri. To find the
-model ID, request `http://127.0.0.1:8001/v1/models` (for example,
-`curl http://127.0.0.1:8001/v1/models`) and copy the exact `id` value. Enter the
-base URL as either `http://127.0.0.1:8001` or `http://127.0.0.1:8001/v1`; the
-desktop normalizes either form so requests use one `/v1` prefix. Optional API
-keys are stored in desktop secure storage and sent as bearer authorization.
-Keep Colibri bound to loopback for local use; the preset does not change its bind
-address or expose it to the network.
-
-This connects Colibri for inference; OpenJarvis remains responsible for running
-its own configured tools when the selected agent requests them. Colibri's optional
-MCP server manages Colibri's lifecycle and is not itself a tool executor. Colibri
-serves one active model at a time, so multi-model routing remains an OpenJarvis
-capability and is not provided by this connection.
-
-Ollama is already a supported local inference source and its installed models are
-listed in Settings. Maya AI is not integrated: the reviewed assistant script does
-not expose a supported service or MCP tool interface and contains direct OS
-actions. Do not run or wrap that script as an OpenJarvis tool.
-
-## Voice, camera, and offline use
-
-Microphone transcription requires the Speech-to-Text setting and a configured
-speech backend. Offline transcription requires a local backend and model; a remote
-provider can send audio off-device. Camera hand tracking is optional in
-**Settings → Hand tracking → Open hand tracking in chat**, or the **Hand tracking**
-button above the chat input. Opening the panel does not start capture. It detects
-landmarks and basic gestures locally, and stops when stopped, when the panel closes,
-when you leave chat, or when the app/tab is hidden. Returning does not restart the
-camera. The first camera action briefly requests permission to reveal
-camera names, then immediately stops that temporary stream. Select the intended
-device (for example, **Integrated Webcam**) from the camera list before pressing
-**Start camera**. Open palm can optionally focus the chat input; gestures never
-submit messages or call tools. It uses the bundled MediaPipe Tasks Vision runtime
-and [Hand Landmarker model](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker).
-The browser's online indicator is only a connectivity hint, not proof that a
-particular server is reachable.
-
-Local inference and cached interface assets do not make the assistant universally
-autonomous. No financial or business-venture integration is configured here, and
-OpenJarvis cannot guarantee legal compliance or revenue. It does not autonomously
-retrain model weights; its existing memory, traces, and learning policies are
-separate capabilities. MCP tools should be narrowly allowlisted, with integration-
-level approval and audit controls for financial transactions, contracts, public
-communications, and other irreversible actions.
-
## Using the Desktop App
1. Start the backend: `jarvis serve` (or `./scripts/quickstart.sh`)
diff --git a/docs/user-guide/mcp-external-servers.md b/docs/user-guide/mcp-external-servers.md
index 1ddbbe89..99a54616 100644
--- a/docs/user-guide/mcp-external-servers.md
+++ b/docs/user-guide/mcp-external-servers.md
@@ -18,11 +18,6 @@ If a server is unreachable or returns an error, OpenJarvis logs a warning and co
External MCP servers are configured in `config.toml` under `[tools.mcp]`. The
`servers` field accepts either inline JSON or a path to a JSON file:
-The default config path is `~/.openjarvis/config.toml` (on Windows,
-`%USERPROFILE%\.openjarvis\config.toml`). Run `jarvis config path` to confirm
-the active path. The dashboard does not currently provide an MCP connection
-editor; configure the backend's file and restart the OpenJarvis backend.
-
```toml
[tools.mcp]
enabled = true
@@ -74,7 +69,6 @@ Each server object supports the following fields:
| `url` | string | No* | URL for Streamable HTTP transport. |
| `command` | string | No* | Command to launch a stdio-based MCP server. |
| `args` | list of strings| No | Arguments passed to the stdio command. |
-| `token` | string | No | Bearer token sent as the HTTP `Authorization` header. |
| `include_tools` | list of strings| No | Whitelist of tool names to import. Only these tools are loaded. |
| `exclude_tools` | list of strings| No | Blacklist of tool names to skip. All other tools are loaded. |
@@ -82,38 +76,6 @@ Each server object supports the following fields:
When both `include_tools` and `exclude_tools` are specified, the whitelist is applied first, then the blacklist filters the result.
-### Authentication and provider endpoints
-
-OpenJarvis supports a bearer token for Streamable HTTP MCP servers, but does
-not implement an interactive OAuth login flow or arbitrary HTTP header
-configuration. The `token` value is stored in the local config (or MCP JSON
-file) as plain text: do not commit or share that file, and restrict access to
-it. Never paste credentials into chat.
-
-Some MCP providers require browser-based OAuth rather than a static bearer
-token. Those servers are not directly confirmed to work with OpenJarvis until
-their authentication flow has been tested. A third-party stdio OAuth bridge
-such as [`mcp-remote`](https://github.com/punkpeye/mcp-remote) may be
-structurally usable through the `command`/`args` transport, but OpenJarvis has
-not validated that bridge or its OAuth flow on Windows; it also runs
-third-party code and stores its own OAuth state locally. Treat it as an
-unverified option, not built-in OpenJarvis support.
-
-URLs in the GitHub MCP Registry API (for example,
-`api.mcp.github.com/oss/v0.1/servers/.../versions/latest`) return registry
-metadata; they are not MCP service endpoints. Obtain the runtime endpoint and
-supported authentication method from the provider's account dashboard or
-official connection guide. Zapier, Vibe Prospecting, Tallyfy, and Hermoso have
-appeared in the registry, but their account-specific endpoints and auth have
-not been tested with OpenJarvis.
-
-Use `include_tools` as an allowlist of exact names returned by that server's
-`tools/list`, and grant only the provider-side permissions needed for those
-tools. Do not enable write, outreach, publishing, advertising, payment, or
-other spend-capable actions until you have reviewed them and set explicit
-approval and budget limits. Tool filtering in OpenJarvis is not a substitute
-for limiting permissions in the provider account.
-
## Examples
### Home Assistant via Streamable HTTP
diff --git a/frontend/package-lock.json b/frontend/package-lock.json
index 80afcea6..07ff1ddf 100644
--- a/frontend/package-lock.json
+++ b/frontend/package-lock.json
@@ -10,7 +10,6 @@
"dependencies": {
"@base-ui/react": "^1.8.0",
"@fontsource-variable/geist": "^5.3.0",
- "@mediapipe/tasks-vision": "^1.1.0",
"@tailwindcss/vite": "^4.3.3",
"@tauri-apps/api": "^2.12.1",
"@tauri-apps/plugin-autostart": "^2",
@@ -46,7 +45,6 @@
"@types/react": "^19.3.0",
"@types/react-dom": "^19.3.0",
"@vitejs/plugin-react": "^6.1.2",
- "happy-dom": "^20.14.6",
"typescript": "~7.0.2",
"vite": "^8.3.4",
"vite-plugin-pwa": "^2.0.0",
@@ -1788,12 +1786,6 @@
"@jridgewell/sourcemap-codec": "^1.4.14"
}
},
- "node_modules/@mediapipe/tasks-vision": {
- "version": "1.1.0",
- "resolved": "https://registry.npmjs.org/@mediapipe/tasks-vision/-/tasks-vision-1.1.0.tgz",
- "integrity": "sha512-ZJqh0wMKOINorfSffvDAxzyO1//c+FiG3IkzOdNDnLqvBB3DbMfj0bB/tys2SNf9S914sgoIbIpF4HTOi336cg==",
- "license": "Apache-2.0"
- },
"node_modules/@oxc-project/types": {
"version": "0.153.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.153.0.tgz",
@@ -3379,16 +3371,6 @@
"integrity": "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==",
"license": "MIT"
},
- "node_modules/@types/node": {
- "version": "26.6.5",
- "resolved": "https://registry.npmjs.org/@types/node/-/node-26.6.5.tgz",
- "integrity": "sha512-uMzsv9stDdofIZBx4UGh4Z9ZgXnGC9seRDscilBcTHAeOFk67MZg/k9bWVfO+bk4qZmG+l52hRnLwUiH2LnJTg==",
- "devOptional": true,
- "license": "MIT",
- "dependencies": {
- "undici-types": "~8.9.0"
- }
- },
"node_modules/@types/react": {
"version": "19.3.0",
"resolved": "https://registry.npmjs.org/@types/react/-/react-19.3.0.tgz",
@@ -3434,23 +3416,6 @@
"integrity": "sha512-zFDAD+tlpf2r4asuHEj0XH6pY6i0g5NeAHPn+15wk3BV6JA69eERFXC1gyGThDkVa1zCyKr5jox1+2LbV/AMLg==",
"license": "MIT"
},
- "node_modules/@types/whatwg-mimetype": {
- "version": "3.0.2",
- "resolved": "https://registry.npmjs.org/@types/whatwg-mimetype/-/whatwg-mimetype-3.0.2.tgz",
- "integrity": "sha512-c2AKvDT8ToxLIOUlN51gTiHXflsfIFisS4pO7pDPoKouJCESkhZnEy623gwP9laCy5lnLDAw1vAzu2vM2YLOrA==",
- "dev": true,
- "license": "MIT"
- },
- "node_modules/@types/ws": {
- "version": "8.18.2",
- "resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.2.tgz",
- "integrity": "sha512-67MQl+fpWKVTT1NYdnmo3U4sc/xPo/zQBncVnI74qmQa0z/b+1g6iYqNmGCPbxO+zz2aklb08a0oHfegiVd0/w==",
- "dev": true,
- "license": "MIT",
- "dependencies": {
- "@types/node": "*"
- }
- },
"node_modules/@typescript/typescript-aix-ppc64": {
"version": "7.0.2",
"resolved": "https://registry.npmjs.org/@typescript/typescript-aix-ppc64/-/typescript-aix-ppc64-7.0.2.tgz",
@@ -4136,19 +4101,6 @@
"devOptional": true,
"license": "MIT"
},
- "node_modules/buffer-image-size": {
- "version": "0.6.4",
- "resolved": "https://registry.npmjs.org/buffer-image-size/-/buffer-image-size-0.6.4.tgz",
- "integrity": "sha512-nEh+kZOPY1w+gcCMobZ6ETUp9WfibndnosbpwB1iJk/8Gt5ZF2bhS6+B6bPYz424KtwsR6Rflc3tCz1/ghX2dQ==",
- "dev": true,
- "license": "MIT",
- "dependencies": {
- "@types/node": "*"
- },
- "engines": {
- "node": ">=4.0"
- }
- },
"node_modules/call-bind": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.9.tgz",
@@ -5388,38 +5340,6 @@
"integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==",
"license": "ISC"
},
- "node_modules/happy-dom": {
- "version": "20.14.6",
- "resolved": "https://registry.npmjs.org/happy-dom/-/happy-dom-20.14.6.tgz",
- "integrity": "sha512-5DDsf4QmEIlRL6NkCv/fxSjYHHkZceL/IaYQdJc2h3G7DRTEai++VTjbfL0LB9geW2VrA3+vrjF3Efx+OTtX0Q==",
- "dev": true,
- "license": "MIT",
- "dependencies": {
- "@types/node": ">=20.0.0",
- "@types/whatwg-mimetype": "^3.0.2",
- "@types/ws": "^8.18.1",
- "buffer-image-size": "^0.6.4",
- "entities": "^7.0.1",
- "whatwg-mimetype": "^3.0.0",
- "ws": "^8.21.0"
- },
- "engines": {
- "node": ">=20.0.0"
- }
- },
- "node_modules/happy-dom/node_modules/entities": {
- "version": "7.0.1",
- "resolved": "https://registry.npmjs.org/entities/-/entities-7.0.1.tgz",
- "integrity": "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==",
- "dev": true,
- "license": "BSD-2-Clause",
- "engines": {
- "node": ">=0.12"
- },
- "funding": {
- "url": "https://github.com/fb55/entities?sponsor=1"
- }
- },
"node_modules/has-bigints": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/has-bigints/-/has-bigints-1.1.0.tgz",
@@ -9197,13 +9117,6 @@
"url": "https://github.com/sponsors/ljharb"
}
},
- "node_modules/undici-types": {
- "version": "8.9.0",
- "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.9.0.tgz",
- "integrity": "sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==",
- "devOptional": true,
- "license": "MIT"
- },
"node_modules/unicode-canonical-property-names-ecmascript": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/unicode-canonical-property-names-ecmascript/-/unicode-canonical-property-names-ecmascript-2.0.1.tgz",
@@ -9986,16 +9899,6 @@
"integrity": "sha512-rLcLXA2sx6+9dE88NHFubwTtGxpK4yYBLj6qHPdFoCaLr0cXGb4efOqtKLlm4loGA4OEKHIQKMKzZkKyOh5ctw==",
"license": "Apache-2.0"
},
- "node_modules/whatwg-mimetype": {
- "version": "3.0.0",
- "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-3.0.0.tgz",
- "integrity": "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q==",
- "dev": true,
- "license": "MIT",
- "engines": {
- "node": ">=12"
- }
- },
"node_modules/which-boxed-primitive": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/which-boxed-primitive/-/which-boxed-primitive-1.1.1.tgz",
@@ -10379,28 +10282,6 @@
"workbox-core": "7.4.1"
}
},
- "node_modules/ws": {
- "version": "8.22.0",
- "resolved": "https://registry.npmjs.org/ws/-/ws-8.22.0.tgz",
- "integrity": "sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==",
- "dev": true,
- "license": "MIT",
- "engines": {
- "node": ">=10.0.0"
- },
- "peerDependencies": {
- "bufferutil": "^4.0.1",
- "utf-8-validate": ">=5.0.2"
- },
- "peerDependenciesMeta": {
- "bufferutil": {
- "optional": true
- },
- "utf-8-validate": {
- "optional": true
- }
- }
- },
"node_modules/yallist": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz",
diff --git a/frontend/package.json b/frontend/package.json
index 5b5f19e1..0a942806 100644
--- a/frontend/package.json
+++ b/frontend/package.json
@@ -19,7 +19,6 @@
"dependencies": {
"@base-ui/react": "^1.8.0",
"@fontsource-variable/geist": "^5.3.0",
- "@mediapipe/tasks-vision": "^1.1.0",
"@tailwindcss/vite": "^4.3.3",
"@tauri-apps/api": "^2.12.1",
"@tauri-apps/plugin-autostart": "^2",
@@ -55,7 +54,6 @@
"@types/react": "^19.3.0",
"@types/react-dom": "^19.3.0",
"@vitejs/plugin-react": "^6.1.2",
- "happy-dom": "^20.14.6",
"typescript": "~7.0.2",
"vite": "^8.3.4",
"vite-plugin-pwa": "^2.0.0",
diff --git a/frontend/public/models/hand_landmarker.task b/frontend/public/models/hand_landmarker.task
deleted file mode 100644
index 0d53faf3..00000000
Binary files a/frontend/public/models/hand_landmarker.task and /dev/null differ
diff --git a/frontend/src-tauri/Entitlements.plist b/frontend/src-tauri/Entitlements.plist
index 724656aa..da5ee242 100644
--- a/frontend/src-tauri/Entitlements.plist
+++ b/frontend/src-tauri/Entitlements.plist
@@ -8,8 +8,6 @@
<true/>
<key>com.apple.security.network.server</key>
<true/>
- <key>com.apple.security.device.camera</key>
- <true/>
<key>com.apple.security.files.user-selected.read-write</key>
<true/>
<key>com.apple.security.cs.allow-jit</key>
diff --git a/frontend/src-tauri/Info.plist b/frontend/src-tauri/Info.plist
index a4faca0f..794e7a46 100644
--- a/frontend/src-tauri/Info.plist
+++ b/frontend/src-tauri/Info.plist
@@ -2,8 +2,6 @@
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
- <key>NSCameraUsageDescription</key>
- <string>Use your selected camera for optional on-device hand tracking in chat. Frames are not recorded or sent to a server.</string>
<key>NSAppTransportSecurity</key>
<dict>
<!-- The desktop API URL is user-configurable, so it cannot be represented
diff --git a/frontend/src-tauri/tauri.conf.json b/frontend/src-tauri/tauri.conf.json
index 262c5c0d..d68eb5ac 100644
--- a/frontend/src-tauri/tauri.conf.json
+++ b/frontend/src-tauri/tauri.conf.json
@@ -24,7 +24,7 @@
}
],
"security": {
- "csp": "default-src 'self'; script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; connect-src 'self' http: https: ws: wss:; img-src 'self' data: blob:; media-src 'self' blob:"
+ "csp": "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; connect-src 'self' http: https: ws: wss:; img-src 'self' data: blob:; media-src 'self' blob:"
}
},
"bundle": {
diff --git a/frontend/src/components/Chat/ChatArea.tsx b/frontend/src/components/Chat/ChatArea.tsx
index 5e1c8536..ed886b0e 100644
--- a/frontend/src/components/Chat/ChatArea.tsx
+++ b/frontend/src/components/Chat/ChatArea.tsx
@@ -2,7 +2,6 @@ import { useRef, useEffect, useState, useCallback } from 'react';
import { useNavigate } from 'react-router';
import { MessageBubble } from './MessageBubble';
import { InputArea } from './InputArea';
-import { ChatHandTracking } from './ChatHandTracking';
import { StreamingDots } from './StreamingDots';
import { useAppStore } from '../../lib/store';
import { shouldAutoplayFinishedReply, useTtsStore } from '../../lib/tts';
@@ -256,7 +255,6 @@ export function ChatArea() {
</div>
)}
</div>
- <ChatHandTracking />
<InputArea />
</div>
);
diff --git a/frontend/src/components/Chat/ChatHandTracking.tsx b/frontend/src/components/Chat/ChatHandTracking.tsx
deleted file mode 100644
index e3b3da0b..00000000
--- a/frontend/src/components/Chat/ChatHandTracking.tsx
+++ /dev/null
@@ -1,41 +0,0 @@
-import { Hand } from 'lucide-react';
-import { useSearchParams } from 'react-router';
-import { HandTrackingPanel } from '../HandTrackingPanel';
-
-/** Keep the opt-in camera and its focus target on the same route. */
-export function ChatHandTracking() {
- const [searchParams, setSearchParams] = useSearchParams();
- const open = searchParams.get('handTracking') === '1';
-
- const togglePanel = () => {
- const next = new URLSearchParams(searchParams);
- if (open) next.delete('handTracking');
- else next.set('handTracking', '1');
- setSearchParams(next, { replace: true });
- };
-
- return (
- <div className="shrink-0 px-4 max-w-[var(--chat-max-width)] w-full mx-auto">
- <button
- type="button"
- aria-expanded={open}
- aria-controls="chat-hand-tracking"
- onClick={togglePanel}
- className="flex items-center gap-1.5 text-xs py-2 cursor-pointer"
- style={{ color: 'var(--color-text-secondary)' }}
- >
- <Hand size={14} />
- {open ? 'Close hand tracking (stops camera)' : 'Hand tracking'}
- </button>
- {open && (
- <div id="chat-hand-tracking" className="max-h-[45vh] overflow-y-auto pb-3">
- <HandTrackingPanel />
- <p className="text-xs mt-2" style={{ color: 'var(--color-text-tertiary)' }}>
- Camera access starts only when you choose a camera. Closing this panel,
- leaving chat, or hiding the app stops it; returning does not restart it.
- </p>
- </div>
- )}
- </div>
- );
-}
diff --git a/frontend/src/components/Chat/InputArea.tsx b/frontend/src/components/Chat/InputArea.tsx
index 5c552738..79a0849e 100644
--- a/frontend/src/components/Chat/InputArea.tsx
+++ b/frontend/src/components/Chat/InputArea.tsx
@@ -162,12 +162,6 @@ export function InputArea() {
el.style.height = Math.min(el.scrollHeight, 200) + 'px';
}, [input]);
- useEffect(() => {
- const focusInput = () => textareaRef.current?.focus();
- window.addEventListener('openjarvis:focus-chat-input', focusInput);
- return () => window.removeEventListener('openjarvis:focus-chat-input', focusInput);
- }, []);
-
const stopStreaming = useCallback(() => {
abortRef.current?.abort();
if (timerRef.current) {
diff --git a/frontend/src/components/HandTrackingPanel.desktop.test.ts b/frontend/src/components/HandTrackingPanel.desktop.test.ts
deleted file mode 100644
index 1e851adc..00000000
--- a/frontend/src/components/HandTrackingPanel.desktop.test.ts
+++ /dev/null
@@ -1,22 +0,0 @@
-import { describe, expect, it } from 'vitest';
-import nativeConfig from '../../src-tauri/tauri.conf.json?raw';
-import infoPlist from '../../src-tauri/Info.plist?raw';
-import entitlements from '../../src-tauri/Entitlements.plist?raw';
-
-describe('packaged desktop hand tracking prerequisites', () => {
- it('allows local WASM compilation without permitting JavaScript eval or remote scripts', () => {
- const config = JSON.parse(nativeConfig);
- const directives = config.app.security.csp.split(';').map((part: string) => part.trim().split(/\s+/));
- const scriptSources = directives.find(([name]: string[]) => name === 'script-src')?.slice(1);
- expect(scriptSources).toContain("'wasm-unsafe-eval'");
- expect(scriptSources).toContain("'self'");
- expect(scriptSources).not.toContain("'unsafe-eval'");
- expect(scriptSources).not.toContain('*');
- expect(scriptSources).not.toContain('https:');
- });
-
- it('declares camera use for macOS permission prompts and hardened runtime', () => {
- expect(infoPlist).toMatch(/<key>NSCameraUsageDescription<\/key>\s*<string>[^<]+<\/string>/);
- expect(entitlements).toMatch(/<key>com\.apple\.security\.device\.camera<\/key>\s*<true\s*\/>/);
- });
-});
diff --git a/frontend/src/components/HandTrackingPanel.test.ts b/frontend/src/components/HandTrackingPanel.test.ts
deleted file mode 100644
index 271cf567..00000000
--- a/frontend/src/components/HandTrackingPanel.test.ts
+++ /dev/null
@@ -1,56 +0,0 @@
-import { describe, expect, it } from 'vitest';
-import type { NormalizedLandmark } from '@mediapipe/tasks-vision';
-import { classifyHandGesture, preferredCameraDevice } from './HandTrackingPanel';
-
-function landmarks(
- extendedFingers: number[],
- thumbExtended = false,
-): NormalizedLandmark[] {
- const points = Array.from({ length: 21 }, (_, index) => ({
- x: index === 4 && thumbExtended ? 0.8 : index === 3 && thumbExtended ? 0.6 : 0.5,
- y: 0.7,
- z: 0,
- visibility: 1,
- }));
- points[0].x = 0.5;
- points[0].y = 0.8;
-
- const fingerTips = [8, 12, 16, 20];
- const fingerPips = [6, 10, 14, 18];
- for (let index = 0; index < 4; index += 1) {
- points[fingerTips[index]].y = extendedFingers.includes(index) ? 0.2 : 0.8;
- points[fingerPips[index]].y = 0.5;
- }
- return points;
-}
-
-describe('hand gesture classification', () => {
- it('recognizes basic hand poses from local landmarks', () => {
- expect(classifyHandGesture(landmarks([0, 1, 2, 3], true))).toBe('Open palm');
- expect(classifyHandGesture(landmarks([]))).toBe('Fist');
- expect(classifyHandGesture(landmarks([0, 1]))).toBe('Peace sign');
- expect(classifyHandGesture(landmarks([0]))).toBe('Pointing');
- });
-
- describe('camera device selection', () => {
- it('prefers the built-in webcam over a phone camera when both are available', () => {
- const cameras = [
- { deviceId: 'phone', label: 'Phone Link Camera' },
- { deviceId: 'webcam', label: 'Integrated Webcam' },
- ];
-
- expect(preferredCameraDevice(cameras)).toEqual(cameras[1]);
- });
-
- it('uses the first available camera when no built-in camera is listed', () => {
- const cameras = [{ deviceId: 'phone', label: 'Phone Link Camera' }];
-
- expect(preferredCameraDevice(cameras)).toEqual(cameras[0]);
- expect(preferredCameraDevice([])).toBeUndefined();
- });
- });
-
- it('avoids classifying incomplete landmark sets as a specific gesture', () => {
- expect(classifyHandGesture(landmarks([0]).slice(0, 10))).toBe('Hand detected');
- });
-});
diff --git a/frontend/src/components/HandTrackingPanel.tsx b/frontend/src/components/HandTrackingPanel.tsx
deleted file mode 100644
index 6d885ab8..00000000
--- a/frontend/src/components/HandTrackingPanel.tsx
+++ /dev/null
@@ -1,424 +0,0 @@
-import { useCallback, useEffect, useRef, useState } from 'react';
-import {
- FilesetResolver,
- HandLandmarker,
- type NormalizedLandmark,
-} from '@mediapipe/tasks-vision';
-import simdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_internal.js?url';
-import simdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_internal.wasm?url';
-import noSimdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.js?url';
-import noSimdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.wasm?url';
-import { Camera, Hand, StopCircle } from 'lucide-react';
-
-const HAND_CONNECTIONS = [
- [0, 1], [1, 2], [2, 3], [3, 4],
- [0, 5], [5, 6], [6, 7], [7, 8],
- [5, 9], [9, 10], [10, 11], [11, 12],
- [9, 13], [13, 14], [14, 15], [15, 16],
- [13, 17], [17, 18], [18, 19], [19, 20], [0, 17],
-];
-
-export type HandGesture =
- | 'Open palm'
- | 'Fist'
- | 'Peace sign'
- | 'Pointing'
- | 'Hand detected';
-
-export interface CameraDevice {
- deviceId: string;
- label: string;
-}
-
-export function preferredCameraDevice(devices: CameraDevice[]): CameraDevice | undefined {
- return devices.find((device) => /integrated|built[\s-]?in|internal|webcam/i.test(device.label))
- ?? devices[0];
-}
-
-export function classifyHandGesture(points: NormalizedLandmark[]): HandGesture {
- if (points.length < 21) return 'Hand detected';
-
- const extended = [
- points[8].y < points[6].y - 0.02,
- points[12].y < points[10].y - 0.02,
- points[16].y < points[14].y - 0.02,
- points[20].y < points[18].y - 0.02,
- ];
- const distanceFromWrist = (index: number) =>
- Math.hypot(points[index].x - points[0].x, points[index].y - points[0].y);
- const thumbExtended = distanceFromWrist(4) > distanceFromWrist(3) * 1.1;
- const extendedCount = extended.filter(Boolean).length;
-
- if (extendedCount === 4 && thumbExtended) return 'Open palm';
- if (extendedCount === 0 && !thumbExtended) return 'Fist';
- if (extended[0] && extended[1] && !extended[2] && !extended[3]) return 'Peace sign';
- if (extended[0] && !extended[1] && !extended[2] && !extended[3]) return 'Pointing';
- return 'Hand detected';
-}
-
-function localVisionFileset() {
- return FilesetResolver.isSimdSupported().then(async (simdSupported) => {
- const fileset = await FilesetResolver.forVisionTasks('/');
- fileset.wasmLoaderPath = simdSupported ? simdLoaderUrl : noSimdLoaderUrl;
- fileset.wasmBinaryPath = simdSupported ? simdWasmUrl : noSimdWasmUrl;
- return fileset;
- });
-}
-
-type TrackerState = 'idle' | 'choosing' | 'starting' | 'active' | 'error';
-
-export function HandTrackingPanel() {
- const [state, setState] = useState<TrackerState>('idle');
- const [error, setError] = useState('');
- const [landmarks, setLandmarks] = useState<NormalizedLandmark[][]>([]);
- const [gesture, setGesture] = useState('No hand detected');
- const [cameras, setCameras] = useState<CameraDevice[]>([]);
- const [selectedCameraId, setSelectedCameraId] = useState('');
- const [focusOnOpenPalm, setFocusOnOpenPalm] = useState(
- () => localStorage.getItem('openjarvis-hand-focus') === 'true',
- );
- const focusOnOpenPalmRef = useRef(focusOnOpenPalm);
- const videoRef = useRef<HTMLVideoElement>(null);
- const detectorRef = useRef<HandLandmarker | null>(null);
- const streamRef = useRef<MediaStream | null>(null);
- const frameRef = useRef<number | null>(null);
- const generationRef = useRef(0);
- const previousGestureRef = useRef<string>('');
-
- const releaseTracking = useCallback(() => {
- generationRef.current += 1;
- if (frameRef.current !== null) cancelAnimationFrame(frameRef.current);
- frameRef.current = null;
- const detector = detectorRef.current;
- detectorRef.current = null;
- // Camera cleanup must still run if the WASM runtime failed during shutdown.
- try { detector?.close(); } catch { /* Already failed/closed. */ }
- streamRef.current?.getTracks().forEach((track) => track.stop());
- streamRef.current = null;
- if (videoRef.current) videoRef.current.srcObject = null;
- }, []);
-
- const stopTracking = useCallback(() => {
- releaseTracking();
- setLandmarks([]);
- setGesture('No hand detected');
- previousGestureRef.current = '';
- setState('idle');
- setError('');
- }, [releaseTracking]);
-
- const chooseCamera = useCallback(async () => {
- if (!navigator.mediaDevices?.getUserMedia) {
- setError('Camera access is not available in this browser or context.');
- setState('error');
- return;
- }
-
- const generation = ++generationRef.current;
- setError('');
- setState('choosing');
- try {
- const permissionStream = await navigator.mediaDevices.getUserMedia({
- video: true,
- audio: false,
- });
- if (generationRef.current !== generation) {
- permissionStream.getTracks().forEach((track) => track.stop());
- return;
- }
-
- // Permission reveals device labels. Stop this temporary stream before
- // enumerating or opening the user-selected camera.
- permissionStream.getTracks().forEach((track) => track.stop());
- const devices = (await navigator.mediaDevices.enumerateDevices())
- .filter((device) => device.kind === 'videoinput')
- .map(({ deviceId, label }, index) => ({
- deviceId,
- label: label || `Camera ${index + 1}`,
- }));
- if (generationRef.current !== generation) return;
- if (devices.length === 0) throw new Error('No cameras were found.');
-
- setCameras(devices);
- setSelectedCameraId(preferredCameraDevice(devices)?.deviceId ?? devices[0].deviceId);
- setState('idle');
- } catch (cause) {
- if (generationRef.current !== generation) return;
- setError(
- cause instanceof DOMException && cause.name === 'NotAllowedError'
- ? 'Camera permission was denied. Allow camera access in your browser settings, then try again.'
- : cause instanceof Error
- ? cause.message
- : 'Could not list available cameras.',
- );
- setState('error');
- }
- }, []);
-
- const startTracking = useCallback(async () => {
- if (!navigator.mediaDevices?.getUserMedia) {
- setError('Camera access is not available in this browser or context.');
- setState('error');
- return;
- }
-
- const generation = ++generationRef.current;
- setError('');
- setState('starting');
-
- try {
- const stream = await navigator.mediaDevices.getUserMedia({
- video: { deviceId: { exact: selectedCameraId } },
- audio: false,
- });
- if (generationRef.current !== generation) {
- stream.getTracks().forEach((track) => track.stop());
- return;
- }
- streamRef.current = stream;
- const video = videoRef.current;
- if (!video) throw new Error('Camera preview is unavailable.');
- video.srcObject = stream;
- await video.play();
- if (generationRef.current !== generation) return;
-
- const fileset = await localVisionFileset();
- if (generationRef.current !== generation) return;
- const detector = await HandLandmarker.createFromOptions(fileset, {
- baseOptions: {
- modelAssetPath: new URL('/models/hand_landmarker.task', window.location.origin).href,
- },
- runningMode: 'VIDEO',
- numHands: 2,
- minHandDetectionConfidence: 0.55,
- minTrackingConfidence: 0.55,
- });
-
- if (generationRef.current !== generation) {
- detector.close();
- stream.getTracks().forEach((track) => track.stop());
- return;
- }
-
- detectorRef.current = detector;
- setState('active');
- let lastVideoTime = -1;
- let lastUiUpdate = 0;
-
- const detectFrame = () => {
- if (generationRef.current !== generation) return;
- const currentDetector = detectorRef.current;
- if (
- currentDetector &&
- video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA &&
- video.currentTime !== lastVideoTime
- ) {
- lastVideoTime = video.currentTime;
- let result;
- try {
- result = currentDetector.detectForVideo(video, performance.now());
- } catch {
- stopTracking();
- setError('Hand tracking stopped after a camera or detector error. Start the camera to try again.');
- setState('error');
- return;
- }
- const detectedHands = result.landmarks;
- const currentGesture = detectedHands.length
- ? classifyHandGesture(detectedHands[0])
- : 'No hand detected';
-
- if (performance.now() - lastUiUpdate > 80) {
- lastUiUpdate = performance.now();
- setLandmarks(detectedHands);
- setGesture(currentGesture);
- }
-
- if (
- focusOnOpenPalmRef.current &&
- currentGesture === 'Open palm' &&
- previousGestureRef.current !== 'Open palm'
- ) {
- window.dispatchEvent(new Event('openjarvis:focus-chat-input'));
- }
- previousGestureRef.current = currentGesture;
- }
- frameRef.current = requestAnimationFrame(detectFrame);
- };
- frameRef.current = requestAnimationFrame(detectFrame);
- } catch (cause) {
- if (generationRef.current !== generation) return;
- releaseTracking();
- setError(
- cause instanceof DOMException && cause.name === 'NotAllowedError'
- ? 'Camera permission was denied. Allow camera access in system settings, then try again.'
- : cause instanceof Error
- ? cause.message
- : 'Could not start local hand tracking.',
- );
- setState('error');
- }
- }, [selectedCameraId, releaseTracking, stopTracking]);
-
- useEffect(() => {
- const onVisibilityChange = () => {
- if (document.hidden) stopTracking();
- };
- document.addEventListener('visibilitychange', onVisibilityChange);
- window.addEventListener('pagehide', stopTracking);
- return () => {
- document.removeEventListener('visibilitychange', onVisibilityChange);
- window.removeEventListener('pagehide', stopTracking);
- releaseTracking();
- };
- }, [releaseTracking, stopTracking]);
-
- const updateFocusOption = (enabled: boolean) => {
- setFocusOnOpenPalm(enabled);
- focusOnOpenPalmRef.current = enabled;
- localStorage.setItem('openjarvis-hand-focus', String(enabled));
- };
-
- return (
- <div className="pt-3">
- <div className="flex items-start justify-between gap-4">
- <div className="min-w-0">
- <div className="text-sm" style={{ color: 'var(--color-text)' }}>Camera hand tracking</div>
- <p className="text-xs mt-1" style={{ color: 'var(--color-text-tertiary)' }}>
- Optional, on-device landmark and basic gesture detection. Camera frames stay in this app and are not recorded.
- </p>
- </div>
- <button
- type="button"
- onClick={() => {
- if (state === 'active' || state === 'starting' || state === 'choosing') {
- stopTracking();
- } else if (cameras.length === 0) {
- void chooseCamera();
- } else {
- void startTracking();
- }
- }}
- aria-pressed={state === 'active' || state === 'starting'}
- className="flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium cursor-pointer shrink-0"
- style={{
- background: state === 'active' ? 'var(--color-error)' : 'var(--color-bg-secondary)',
- border: '1px solid var(--color-border)',
- color: state === 'active' ? 'white' : 'var(--color-text)',
- }}
- >
- {state === 'choosing'
- ? <><StopCircle size={13} /> Cancel camera access</>
- : state === 'starting'
- ? <><StopCircle size={13} /> Stop camera</>
- : state === 'active'
- ? <><StopCircle size={13} /> Stop camera</>
- : cameras.length === 0
- ? <><Camera size={13} /> Choose camera</>
- : <><Camera size={13} /> Start camera</>}
- </button>
- </div>
-
- {cameras.length > 0 && state !== 'active' && (
- <div className="flex items-center gap-3 mt-3">
- <label htmlFor="openjarvis-camera-device" className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>
- Camera device
- </label>
- <select
- id="openjarvis-camera-device"
- aria-label="Camera device"
- value={selectedCameraId}
- onChange={(event) => setSelectedCameraId(event.target.value)}
- disabled={state === 'starting'}
- className="text-xs px-2 py-1 rounded-lg outline-none max-w-full"
- style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}
- >
- {cameras.map((camera) => (
- <option key={camera.deviceId} value={camera.deviceId}>{camera.label}</option>
- ))}
- </select>
- <button
- type="button"
- onClick={() => void chooseCamera()}
- disabled={state === 'choosing' || state === 'starting'}
- className="text-xs underline cursor-pointer"
- style={{ color: 'var(--color-accent)' }}
- >
- Refresh
- </button>
- </div>
- )}
-
- {/* Keep the ref available even when camera permission resolves before React
- commits the starting state. An empty video never starts capture. */}
- <div className={state === 'active' || state === 'starting' ? 'flex flex-col sm:flex-row items-start gap-4 mt-4' : 'hidden'}>
- <div className="relative w-full sm:w-64 aspect-video overflow-hidden rounded-lg bg-black">
- <video
- ref={videoRef}
- muted
- playsInline
- aria-label="Live camera preview for hand tracking"
- className="absolute inset-0 w-full h-full object-cover"
- style={{ transform: 'scaleX(-1)' }}
- />
- <svg
- aria-hidden="true"
- viewBox="0 0 1 1"
- preserveAspectRatio="none"
- className="absolute inset-0 w-full h-full"
- style={{ transform: 'scaleX(-1)' }}
- >
- {landmarks.map((hand, handIndex) => (
- <g key={handIndex}>
- {HAND_CONNECTIONS.map(([start, end]) => (
- <line
- key={`${start}-${end}`}
- x1={hand[start].x}
- y1={hand[start].y}
- x2={hand[end].x}
- y2={hand[end].y}
- stroke="#34d399"
- strokeWidth="0.009"
- />
- ))}
- {hand.map((point, pointIndex) => (
- <circle key={pointIndex} cx={point.x} cy={point.y} r="0.014" fill="#fbbf24" />
- ))}
- </g>
- ))}
- </svg>
- </div>
- <div aria-live="polite" className="flex items-center gap-2 text-xs" style={{ color: 'var(--color-text-secondary)' }}>
- <Hand size={15} style={{ color: landmarks.length ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />
- {state === 'starting'
- ? 'Starting local hand tracker…'
- : landmarks.length
- ? `${landmarks.length} hand${landmarks.length === 1 ? '' : 's'} — ${gesture}`
- : 'No hand detected'}
- </div>
- </div>
-
- {state === 'error' && (
- <p role="alert" className="text-xs mt-3" style={{ color: 'var(--color-error)' }}>{error}</p>
- )}
- {cameras.length > 0 && state !== 'active' && (
- <p className="text-xs mt-2" style={{ color: 'var(--color-text-tertiary)' }}>
- Choose the webcam by name before starting. The camera used to reveal device names was stopped immediately.
- </p>
- )}
-
- <label className="flex items-center gap-2 mt-3 text-xs cursor-pointer" style={{ color: 'var(--color-text-secondary)' }}>
- <input
- type="checkbox"
- checked={focusOnOpenPalm}
- onChange={(event) => updateFocusOption(event.target.checked)}
- className="accent-[var(--color-accent)]"
- />
- Focus chat input when an open palm is detected
- </label>
- <p className="text-xs mt-1 pl-5" style={{ color: 'var(--color-text-tertiary)' }}>
- Optional and off by default. Gestures never send messages or trigger tools.
- </p>
- </div>
- );
-}
diff --git a/frontend/src/pages/ChatPage.hand-tracking.test.tsx b/frontend/src/pages/ChatPage.hand-tracking.test.tsx
deleted file mode 100644
index bf70686c..00000000
--- a/frontend/src/pages/ChatPage.hand-tracking.test.tsx
+++ /dev/null
@@ -1,247 +0,0 @@
-// @vitest-environment happy-dom
-// @vitest-environment-options { "url": "http://localhost:5173" }
-import { act } from 'react';
-import { createRoot, type Root } from 'react-dom/client';
-import { createMemoryRouter, RouterProvider } from 'react-router';
-import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
-import { ChatPage } from './ChatPage';
-import { useAppStore } from '../lib/store';
-
-const vision = vi.hoisted(() => ({
- create: vi.fn(),
- detect: vi.fn(),
- close: vi.fn(),
-}));
-const send = vi.hoisted(() => vi.fn());
-
-vi.mock('@mediapipe/tasks-vision', () => ({
- FilesetResolver: {
- isSimdSupported: vi.fn().mockResolvedValue(true),
- forVisionTasks: vi.fn().mockResolvedValue({}),
- },
- HandLandmarker: { createFromOptions: vision.create },
-}));
-vi.mock('../lib/connectors-api', () => ({ listConnectors: vi.fn().mockResolvedValue([]) }));
-vi.mock('../lib/sse', () => ({ streamChat: send, streamResearch: send }));
-vi.mock('../components/Chat/MessageBubble', () => ({ MessageBubble: () => null }));
-vi.mock('../hooks/useSpeech', () => ({
- useSpeech: () => ({ state: 'idle', error: null, available: false }),
-}));
-
-function openPalm() {
- const points = Array.from({ length: 21 }, () => ({ x: 0.5, y: 0.7, z: 0 }));
- points[0].y = 0.8;
- points[3].x = 0.6;
- points[4].x = 0.8;
- for (const tip of [8, 12, 16, 20]) points[tip].y = 0.2;
- for (const pip of [6, 10, 14, 18]) points[pip].y = 0.5;
- return points;
-}
-
-function mockStream() {
- const stop = vi.fn();
- const stream = new MediaStream();
- Object.defineProperty(stream, 'getTracks', { value: () => [{ stop }] });
- return { stop, stream };
-}
-
-let root: Root;
-let container: HTMLDivElement;
-let router: ReturnType<typeof createMemoryRouter>;
-let frames: Map<number, FrameRequestCallback>;
-let camera: ReturnType<typeof mockStream>;
-let permission: ReturnType<typeof mockStream>;
-let getUserMedia: ReturnType<typeof vi.fn>;
-let hidden: boolean;
-let videoTime: number;
-
-beforeEach(() => {
- (globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
- localStorage.clear();
- vi.clearAllMocks();
- useAppStore.setState({
- selectedModel: 'test-model', modelLoading: false, systemPanelOpen: false,
- deepResearch: false, activeId: null, messages: [],
- streamState: { ...useAppStore.getState().streamState, isStreaming: false },
- });
- hidden = false;
- videoTime = 0;
- vi.spyOn(document, 'hidden', 'get').mockImplementation(() => hidden);
- vi.spyOn(HTMLMediaElement.prototype, 'play').mockResolvedValue();
- // happy-dom exposes media elements but does not define their readiness constants.
- Object.defineProperty(HTMLMediaElement, 'HAVE_CURRENT_DATA', { configurable: true, value: 2 });
- vi.spyOn(HTMLMediaElement.prototype, 'readyState', 'get').mockReturnValue(2);
- vi.spyOn(HTMLMediaElement.prototype, 'currentTime', 'get').mockImplementation(() => videoTime);
- frames = new Map();
- let id = 0;
- vi.stubGlobal('requestAnimationFrame', vi.fn((callback: FrameRequestCallback) => {
- frames.set(++id, callback);
- return id;
- }));
- vi.stubGlobal('cancelAnimationFrame', vi.fn((frame: number) => frames.delete(frame)));
- // Catch accidental backend/model downloads rather than making real requests.
- vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new Error('Unexpected network request'))));
- permission = mockStream();
- camera = mockStream();
- getUserMedia = vi.fn().mockResolvedValueOnce(permission.stream).mockResolvedValue(camera.stream);
- Object.defineProperty(navigator, 'mediaDevices', {
- configurable: true,
- value: {
- getUserMedia,
- enumerateDevices: vi.fn().mockResolvedValue([
- { kind: 'videoinput', deviceId: 'phone', label: 'Phone camera' },
- { kind: 'videoinput', deviceId: 'webcam', label: 'Integrated Webcam' },
- ]),
- },
- });
- vision.create.mockResolvedValue({ detectForVideo: vision.detect, close: vision.close });
- vision.detect.mockReturnValue({ landmarks: [openPalm()] });
- container = document.createElement('div');
- document.body.append(container);
- root = createRoot(container);
-});
-
-afterEach(async () => {
- await act(async () => root?.unmount());
- router?.dispose();
- container?.remove();
- vi.restoreAllMocks();
- vi.unstubAllGlobals();
-});
-
-async function mount(path = '/?handTracking=1') {
- router = createMemoryRouter([
- { path: '/', element: <ChatPage /> },
- { path: '/settings', element: <div>Settings</div> },
- ], { initialEntries: [path] });
- await act(async () => root.render(<RouterProvider router={router} />));
-}
-
-function button(text: string) {
- const result = [...container.querySelectorAll('button')].find((node) => node.textContent?.trim() === text);
- expect(result, `Button: ${text}`).toBeDefined();
- return result!;
-}
-
-async function click(text: string) {
- await act(async () => button(text).click());
-}
-
-async function start() {
- await click('Choose camera');
- expect(permission.stop).toHaveBeenCalledOnce();
- await click('Start camera');
- expect(container.querySelector('[role="alert"]')?.textContent).toBeUndefined();
-}
-
-async function frame() {
- const [id, callback] = frames.entries().next().value!;
- frames.delete(id);
- videoTime += 1;
- await act(async () => callback(performance.now()));
-}
-
-describe('chat hand tracking route and camera lifecycle', () => {
- it('opens without permission and focuses the real chat input only after opt-in, without sending', async () => {
- await mount('/');
- expect(container.querySelector('video')).toBeNull();
- await click('Hand tracking');
- expect(getUserMedia).not.toHaveBeenCalled();
- const focusOption = container.querySelector<HTMLInputElement>('input[type="checkbox"]')!;
- expect(focusOption.checked).toBe(false);
- await start();
- button('Stop camera').focus();
- await frame();
- expect(document.activeElement).toBe(button('Stop camera'));
- await act(async () => focusOption.click());
- vision.detect.mockReturnValueOnce({ landmarks: [] });
- await frame();
- await frame();
- expect(document.activeElement).toBe(container.querySelector('textarea'));
- expect(getUserMedia).toHaveBeenNthCalledWith(1, { video: true, audio: false });
- expect(getUserMedia).toHaveBeenNthCalledWith(2, { video: { deviceId: { exact: 'webcam' } }, audio: false });
- expect(send).not.toHaveBeenCalled();
- expect(useAppStore.getState().messages).toEqual([]);
- await click('Close hand tracking (stops camera)');
- expect(camera.stop).toHaveBeenCalledOnce();
- expect(vision.close).toHaveBeenCalledOnce();
- expect(frames.size).toBe(0);
- await click('Hand tracking');
- expect(getUserMedia).toHaveBeenCalledTimes(2);
- });
-
- it('stops on route changes and does not resume on returning to chat', async () => {
- await mount();
- await start();
- await act(async () => { await router.navigate('/settings'); });
- expect(camera.stop).toHaveBeenCalledOnce();
- expect(vision.close).toHaveBeenCalledOnce();
- expect(frames.size).toBe(0);
- await act(async () => { await router.navigate('/?handTracking=1'); });
- expect(getUserMedia).toHaveBeenCalledTimes(2);
- expect(container.querySelector('video')?.srcObject).toBeFalsy();
- });
-
- it.each(['hidden', 'pagehide', 'stop'])('stops capture on %s and requires another start', async (reason) => {
- await mount();
- await start();
- await act(async () => {
- if (reason === 'hidden') {
- hidden = true;
- document.dispatchEvent(new Event('visibilitychange'));
- } else if (reason === 'pagehide') window.dispatchEvent(new Event('pagehide'));
- else button('Stop camera').click();
- });
- expect(camera.stop).toHaveBeenCalledOnce();
- expect(vision.close).toHaveBeenCalledOnce();
- expect(frames.size).toBe(0);
- await act(async () => {
- hidden = false;
- document.dispatchEvent(new Event('visibilitychange'));
- });
- expect(getUserMedia).toHaveBeenCalledTimes(2);
- expect(button('Start camera')).toBeDefined();
- });
-
- it('closes a permission stream that arrives after leaving chat', async () => {
- let resolvePermission!: (stream: MediaStream) => void;
- getUserMedia.mockReset().mockReturnValue(new Promise((resolve) => { resolvePermission = resolve; }));
- await mount();
- await click('Choose camera');
- await act(async () => { await router.navigate('/settings'); });
- await act(async () => resolvePermission(permission.stream));
- expect(permission.stop).toHaveBeenCalledOnce();
- expect(vision.create).not.toHaveBeenCalled();
- });
-
- it('stops the camera immediately and closes a detector that finishes loading after cancellation', async () => {
- let resolveDetector!: (detector: unknown) => void;
- vision.create.mockReturnValueOnce(new Promise((resolve) => { resolveDetector = resolve; }));
- await mount();
- await start();
- await click('Close hand tracking (stops camera)');
- expect(camera.stop).toHaveBeenCalledOnce();
- await act(async () => resolveDetector({ close: vision.close }));
- expect(vision.close).toHaveBeenCalledOnce();
- expect(frames.size).toBe(0);
- });
-
- it('reports permission denial without loading the detector', async () => {
- getUserMedia.mockReset().mockRejectedValue(new DOMException('Denied', 'NotAllowedError'));
- await mount();
- await click('Choose camera');
- expect(container.querySelector('[role="alert"]')?.textContent).toContain('Camera permission was denied');
- expect(vision.create).not.toHaveBeenCalled();
- });
-
- it('releases the camera if inference or detector cleanup fails', async () => {
- vision.detect.mockImplementationOnce(() => { throw new Error('WASM failed'); });
- vision.close.mockImplementationOnce(() => { throw new Error('Already failed'); });
- await mount();
- await start();
- await frame();
- expect(camera.stop).toHaveBeenCalledOnce();
- expect(frames.size).toBe(0);
- expect(container.querySelector('[role="alert"]')?.textContent).toContain('Hand tracking stopped');
- });
-});
diff --git a/frontend/src/pages/SettingsPage.cloud-providers.test.tsx b/frontend/src/pages/SettingsPage.cloud-providers.test.tsx
index fdf2b400..2e35fe7f 100644
--- a/frontend/src/pages/SettingsPage.cloud-providers.test.tsx
+++ b/frontend/src/pages/SettingsPage.cloud-providers.test.tsx
@@ -1,5 +1,4 @@
import { renderToStaticMarkup } from 'react-dom/server';
-import { MemoryRouter } from 'react-router';
import { describe, expect, it, vi } from 'vitest';
describe('desktop cloud provider settings', () => {
@@ -7,7 +6,7 @@ describe('desktop cloud provider settings', () => {
vi.stubGlobal('localStorage', { getItem: () => null });
vi.stubGlobal('window', { __TAURI_INTERNALS__: {} });
const { SettingsPage } = await import('./SettingsPage');
- const html = renderToStaticMarkup(<MemoryRouter><SettingsPage /></MemoryRouter>);
+ const html = renderToStaticMarkup(<SettingsPage />);
expect(html).toContain('Atlas Cloud');
const atlasKeyInput = html.match(/<input[^>]*placeholder="Atlas Cloud API key"[^>]*>/)?.[0];
diff --git a/frontend/src/pages/SettingsPage.tsx b/frontend/src/pages/SettingsPage.tsx
index db6bda27..6b8bdc1d 100644
--- a/frontend/src/pages/SettingsPage.tsx
+++ b/frontend/src/pages/SettingsPage.tsx
@@ -1,5 +1,4 @@
import { useState, useEffect, useCallback, useRef } from 'react';
-import { Link } from 'react-router';
import {
Palette,
Globe,
@@ -18,8 +17,6 @@ import {
Search,
Brain,
RefreshCw,
- Server,
- Wifi,
} from 'lucide-react';
import { useAppStore, type ThemeMode } from '../lib/store';
import {
@@ -232,36 +229,6 @@ function SettingRow({ label, description, children }: { label: string; descripti
);
}
-export function NetworkStatusRow() {
- const [online, setOnline] = useState<boolean | null>(
- () => typeof navigator === 'undefined' ? null : navigator.onLine,
- );
-
- useEffect(() => {
- const updateOnline = () => setOnline(navigator.onLine);
- window.addEventListener('online', updateOnline);
- window.addEventListener('offline', updateOnline);
- return () => {
- window.removeEventListener('online', updateOnline);
- window.removeEventListener('offline', updateOnline);
- };
- }, []);
-
- return (
- <SettingRow
- label="Browser network"
- description="Browser signal only; it does not verify internet or cloud-provider reachability."
- >
- <div className="flex items-center gap-2">
- <Wifi size={14} style={{ color: online ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />
- <span className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>
- {online === null ? 'Unknown' : online ? 'Online' : 'Offline'}
- </span>
- </div>
- </SettingRow>
- );
-}
-
type MemoryStatus =
| { kind: 'loading' }
| { kind: 'ready'; stats: MemoryStats }
@@ -373,7 +340,6 @@ export function SettingsPage() {
const [customEngine, setCustomEngine] = useState('lmstudio');
const [customKey, setCustomKey] = useState('');
const [srcMsg, setSrcMsg] = useState('');
- const inferenceConfigurable = isTauri();
useEffect(() => {
getInferenceSource().then((s) => {
@@ -554,7 +520,6 @@ export function SettingsPage() {
{/* Connection */}
<Section title="Connection">
- <NetworkStatusRow />
<SettingRow label="Server status" description={serverInfo ? `${serverInfo.engine} / ${serverInfo.model}` : 'Not connected'}>
<div className="flex items-center gap-2">
<span
@@ -599,14 +564,10 @@ export function SettingsPage() {
{/* Inference source */}
<Section title="Inference source">
- <p className="text-xs mb-2" style={{ color: 'var(--color-text-tertiary)' }}>
- Ollama is supported for local inference. The desktop app can connect to Colibri through its OpenAI-compatible API; OpenJarvis agents provide multi-step tool use and reasoning, while Colibri serves one model at a time. Maya AI is not wired in: its reviewed assistant script has no supported service or plugin API, so it cannot be safely managed here.
- </p>
- <SettingRow label="Source" description={inferenceConfigurable ? 'Where the app runs models. Applies after restart.' : 'The browser uses the backend’s configured model source; change inference sources in the desktop app.'}>
+ <SettingRow label="Source" description="Where the app runs models. Applies after restart.">
<select
value={srcKind}
onChange={(e) => { setSrcKind(e.target.value as InferenceSource['kind']); setSrcMsg(''); }}
- disabled={!inferenceConfigurable}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}
>
@@ -616,39 +577,21 @@ export function SettingsPage() {
</SettingRow>
{srcKind === 'custom' && (
<>
- <SettingRow label="Colibri preset" description="Sets the desktop app's LM Studio-compatible adapter target; configure Colibri separately and enter its exact model ID below.">
- <button
- type="button"
- onClick={() => {
- setSrcKind('custom');
- setCustomEngine('lmstudio');
- setCustomHost('http://127.0.0.1:8001');
- setCustomModel('');
- setSrcMsg('Colibri preset selected. Enter a model ID exposed by Colibri, then save and restart.');
- }}
- disabled={!inferenceConfigurable}
- title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}
- className="text-xs px-3 py-1.5 rounded-lg cursor-pointer"
- style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}
- >
- <span className="flex items-center gap-1.5"><Server size={13} /> Use Colibri preset</span>
- </button>
- </SettingRow>
- <SettingRow label="Server URL" description={customEngine === 'lmstudio' ? 'Base URL. For Colibri, configure loopback:8001 to avoid Jarvis’s default port 8000.' : 'OpenAI-compatible server base URL'}>
- <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1" disabled={!inferenceConfigurable}
+ <SettingRow label="Server URL" description="e.g. LM Studio: http://localhost:1234/v1">
+ <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1"
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} />
</SettingRow>
- <SettingRow label="Model" description={customEngine === 'lmstudio' ? 'Enter the exact model id from your server’s GET /v1/models response.' : 'Model id served by your endpoint'}>
- <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct" disabled={!inferenceConfigurable}
+ <SettingRow label="Model" description="Model id served by your endpoint">
+ <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct"
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} />
</SettingRow>
- <SettingRow label="Server type" description={customEngine === 'lmstudio' ? 'LM Studio adapter; also compatible with Colibri’s OpenAI-style /v1 API.' : 'OpenAI-compatible inference backend'}>
- <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }} disabled={!inferenceConfigurable}
+ <SettingRow label="Server type" description="OpenAI-compatible engine">
+ <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>
- <option value="lmstudio">LM Studio / Colibri-compatible</option>
+ <option value="lmstudio">LM Studio</option>
<option value="vllm">vLLM</option>
<option value="sglang">SGLang</option>
<option value="llamacpp">llama.cpp</option>
@@ -656,7 +599,7 @@ export function SettingsPage() {
</select>
</SettingRow>
<SettingRow label="API key (optional)" description="Only if your server requires one">
- <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none" disabled={!inferenceConfigurable}
+ <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none"
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} />
</SettingRow>
@@ -664,10 +607,8 @@ export function SettingsPage() {
)}
<SettingRow label="" description={srcMsg}>
<button onClick={saveSource}
- disabled={!inferenceConfigurable}
- title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}
className="text-sm px-3 py-1.5 rounded-lg outline-none cursor-pointer"
- style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}>
+ style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>
Save inference source
</button>
</SettingRow>
@@ -935,22 +876,6 @@ export function SettingsPage() {
See the <a href="https://open-jarvis.github.io/OpenJarvis/user-guide/tools/" target="_blank" rel="noopener noreferrer" style={{ color: 'var(--color-accent)' }}>documentation</a> for details.
</div>
)}
- <p className="text-xs mt-3" style={{ color: 'var(--color-text-tertiary)' }}>
- Offline speech requires a configured local backend and downloaded model. Remote speech providers may send audio off-device.
- </p>
- </Section>
-
- {/* Hand tracking */}
- <Section title="Hand tracking">
- <SettingRow label="Camera gestures" description="Open camera controls alongside the chat input. Camera access requires an explicit action and stops when you close the panel or leave chat.">
- <Link
- to="/?handTracking=1"
- className="text-xs underline"
- style={{ color: 'var(--color-accent)' }}
- >
- Open hand tracking in chat
- </Link>
- </SettingRow>
</Section>
{/* Data */}
diff --git a/frontend/vite.config.ts b/frontend/vite.config.ts
index 57ca3f31..360565cd 100644
--- a/frontend/vite.config.ts
+++ b/frontend/vite.config.ts
@@ -37,8 +37,7 @@ export default defineConfig({
],
},
workbox: {
- globPatterns: ['**/*.{js,css,html,ico,png,svg,wasm,task}'],
- maximumFileSizeToCacheInBytes: 16 * 1024 * 1024,
+ globPatterns: ['**/*.{js,css,html,ico,png,svg}'],
navigateFallbackDenylist: [/^\/v1\//, /^\/health/, /^\/dashboard/, /^\/api\//],
},
}),
diff --git a/tests/engine/test_openai_compat_api_key.py b/tests/engine/test_openai_compat_api_key.py
index 65a9d229..dc541358 100644
--- a/tests/engine/test_openai_compat_api_key.py
+++ b/tests/engine/test_openai_compat_api_key.py
@@ -2,15 +2,12 @@
from __future__ import annotations
-import json
-
import httpx
import pytest
import respx
from openjarvis.core.types import Message, Role
from openjarvis.engine.openai_compat_engines import (
- LMStudioEngine,
OpenAICompatEngine,
VLLMEngine,
normalize_openai_base_url,
@@ -82,33 +79,6 @@ class TestAuthorizationHeader:
engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")
assert engine._api_key == "sk-explicit"
- def test_colibri_compatible_lmstudio_engine_uses_model_and_bearer_key(self) -> None:
- model_id = "colibri/qwen3-8b"
- engine = LMStudioEngine(host="http://127.0.0.1:8001", api_key="coli-test")
- with respx.mock:
- route = respx.post("http://127.0.0.1:8001/v1/chat/completions").mock(
- return_value=httpx.Response(200, json=_CHAT_RESPONSE)
- )
- engine.generate([Message(role=Role.USER, content="hi")], model=model_id)
- request = route.calls.last.request
- assert request.headers["Authorization"] == "Bearer coli-test"
- assert request.url.path == "/v1/chat/completions"
- assert json.loads(request.content)["model"] == model_id
-
- def test_lmstudio_model_list_uses_single_v1_prefix_and_optional_auth(self) -> None:
- host = normalize_openai_base_url("http://127.0.0.1:8001/v1")
- engine = LMStudioEngine(host=host, api_key="coli-test")
- with respx.mock:
- route = respx.get("http://127.0.0.1:8001/v1/models").mock(
- return_value=httpx.Response(
- 200, json={"data": [{"id": "colibri/qwen3-8b"}]}
- )
- )
- assert engine.list_models() == ["colibri/qwen3-8b"]
- request = route.calls.last.request
- assert request.url.path == "/v1/models"
- assert request.headers["Authorization"] == "Bearer coli-test"
-
class TestNormalizeOpenAIBaseUrl:
@pytest.mark.parametrize(