Skip to content

Personal AI, On Personal Devices

Python

feat: add offline hand tracking and Colibri setup

AzeftheGoat18747688b37d2c5
diff --git a/docs/architecture/design-principles.md b/docs/architecture/design-principles.md
index 54b09712..5d6e1412 100644
--- a/docs/architecture/design-principles.md
+++ b/docs/architecture/design-principles.md
@@ -94,6 +94,7 @@ OpenJarvis is designed to work **entirely without network access**. All core fun
| Traces | SQLite-based, fully local |
| Tools | Calculator, Think, FileRead all local |
| Configuration | TOML file on disk |
+| Hand tracking | Optional MediaPipe model and WASM assets run locally after installation |
Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cloud` backend, but they are:
@@ -101,6 +102,14 @@ Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cl
- Only activated if API keys are set as environment variables
- Never required for any core functionality
+Offline operation still requires the local API server, inference runtime, and model to
+be installed and available. The desktop app and PWA can cache their interface assets,
+but API inference requests are not cached or replayed offline. Speech recognition is
+offline only when a local speech backend and its model are configured; remote speech
+providers require network access and may receive audio. Hand tracking is opt-in and
+uses the bundled model locally after the app has been installed or its PWA assets
+cached.
+
```python
# This works without any network connection
from openjarvis import Jarvis
diff --git a/docs/user-guide/chat-simple.md b/docs/user-guide/chat-simple.md
index 4bab5538..7871d176 100644
--- a/docs/user-guide/chat-simple.md
+++ b/docs/user-guide/chat-simple.md
@@ -110,6 +110,58 @@ jarvis serve
The server is OpenAI-compatible, so any client that works with the OpenAI API can point to `http://localhost:8000/v1`.
+## Connecting Colibri
+
+OpenJarvis can use a Colibri instance that exposes OpenAI-compatible Chat
+Completions and model-list endpoints. In desktop **Settings → Inference source**,
+choose **Custom OpenAI-compatible server**, then click **Use Colibri preset**.
+Configure Colibri separately to bind to loopback on port `8001` (Jarvis uses port
+`8000` by default). Enter the exact model ID reported by Colibri, add its API key
+only if Colibri requires one, save, and restart OpenJarvis. The preset only sets
+OpenJarvis's connection target; it does not install or start Colibri. To find the
+model ID, request `http://127.0.0.1:8001/v1/models` (for example,
+`curl http://127.0.0.1:8001/v1/models`) and copy the exact `id` value. Enter the
+base URL as either `http://127.0.0.1:8001` or `http://127.0.0.1:8001/v1`; the
+desktop normalizes either form so requests use one `/v1` prefix. Optional API
+keys are stored in desktop secure storage and sent as bearer authorization.
+Keep Colibri bound to loopback for local use; the preset does not change its bind
+address or expose it to the network.
+
+This connects Colibri for inference; OpenJarvis remains responsible for running
+its own configured tools when the selected agent requests them. Colibri's optional
+MCP server manages Colibri's lifecycle and is not itself a tool executor. Colibri
+serves one active model at a time, so multi-model routing remains an OpenJarvis
+capability and is not provided by this connection.
+
+Ollama is already a supported local inference source and its installed models are
+listed in Settings. Maya AI is not integrated: the reviewed assistant script does
+not expose a supported service or MCP tool interface and contains direct OS
+actions. Do not run or wrap that script as an OpenJarvis tool.
+
+## Voice, camera, and offline use
+
+Microphone transcription requires the Speech-to-Text setting and a configured
+speech backend. Offline transcription requires a local backend and model; a remote
+provider can send audio off-device. Camera hand tracking is optional in
+**Settings → Hand tracking**: it asks for camera permission only when started,
+detects landmarks and basic gestures locally, and stops when stopped or when the
+settings page closes. The first camera action briefly requests permission to reveal
+camera names, then immediately stops that temporary stream. Select the intended
+device (for example, **Integrated Webcam**) from the camera list before pressing
+**Start camera**. Open palm can optionally focus the chat input; gestures never
+submit messages or call tools. It uses the bundled MediaPipe Tasks Vision runtime
+and [Hand Landmarker model](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker).
+The browser's online indicator is only a connectivity hint, not proof that a
+particular server is reachable.
+
+Local inference and cached interface assets do not make the assistant universally
+autonomous. No financial or business-venture integration is configured here, and
+OpenJarvis cannot guarantee legal compliance or revenue. It does not autonomously
+retrain model weights; its existing memory, traces, and learning policies are
+separate capabilities. MCP tools should be narrowly allowlisted, with integration-
+level approval and audit controls for financial transactions, contracts, public
+communications, and other irreversible actions.
+
## Using the Desktop App
1. Start the backend: `jarvis serve` (or `./scripts/quickstart.sh`)
diff --git a/frontend/package-lock.json b/frontend/package-lock.json
index ea28b996..86b19b21 100644
--- a/frontend/package-lock.json
+++ b/frontend/package-lock.json
@@ -10,6 +10,7 @@
"dependencies": {
"@base-ui/react": "^1.8.0",
"@fontsource-variable/geist": "^5.3.0",
+ "@mediapipe/tasks-vision": "^1.1.0",
"@tailwindcss/vite": "^4.3.3",
"@tauri-apps/api": "^2.12.0",
"@tauri-apps/plugin-autostart": "^2",
@@ -1987,6 +1988,12 @@
"@jridgewell/sourcemap-codec": "^1.4.14"
}
},
+ "node_modules/@mediapipe/tasks-vision": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@mediapipe/tasks-vision/-/tasks-vision-1.1.0.tgz",
+ "integrity": "sha512-ZJqh0wMKOINorfSffvDAxzyO1//c+FiG3IkzOdNDnLqvBB3DbMfj0bB/tys2SNf9S914sgoIbIpF4HTOi336cg==",
+ "license": "Apache-2.0"
+ },
"node_modules/@modelcontextprotocol/sdk": {
"version": "1.28.0",
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.28.0.tgz",
diff --git a/frontend/package.json b/frontend/package.json
index 07cd449d..7f8826cc 100644
--- a/frontend/package.json
+++ b/frontend/package.json
@@ -19,6 +19,7 @@
"dependencies": {
"@base-ui/react": "^1.8.0",
"@fontsource-variable/geist": "^5.3.0",
+ "@mediapipe/tasks-vision": "^1.1.0",
"@tailwindcss/vite": "^4.3.3",
"@tauri-apps/api": "^2.12.0",
"@tauri-apps/plugin-autostart": "^2",
diff --git a/frontend/public/models/hand_landmarker.task b/frontend/public/models/hand_landmarker.task
new file mode 100644
index 00000000..0d53faf3
Binary files /dev/null and b/frontend/public/models/hand_landmarker.task differ
diff --git a/frontend/src/components/Chat/InputArea.tsx b/frontend/src/components/Chat/InputArea.tsx
index 79a0849e..5c552738 100644
--- a/frontend/src/components/Chat/InputArea.tsx
+++ b/frontend/src/components/Chat/InputArea.tsx
@@ -162,6 +162,12 @@ export function InputArea() {
el.style.height = Math.min(el.scrollHeight, 200) + 'px';
}, [input]);
+ useEffect(() => {
+ const focusInput = () => textareaRef.current?.focus();
+ window.addEventListener('openjarvis:focus-chat-input', focusInput);
+ return () => window.removeEventListener('openjarvis:focus-chat-input', focusInput);
+ }, []);
+
const stopStreaming = useCallback(() => {
abortRef.current?.abort();
if (timerRef.current) {
diff --git a/frontend/src/components/HandTrackingPanel.test.ts b/frontend/src/components/HandTrackingPanel.test.ts
new file mode 100644
index 00000000..271cf567
--- /dev/null
+++ b/frontend/src/components/HandTrackingPanel.test.ts
@@ -0,0 +1,56 @@
+import { describe, expect, it } from 'vitest';
+import type { NormalizedLandmark } from '@mediapipe/tasks-vision';
+import { classifyHandGesture, preferredCameraDevice } from './HandTrackingPanel';
+
+function landmarks(
+ extendedFingers: number[],
+ thumbExtended = false,
+): NormalizedLandmark[] {
+ const points = Array.from({ length: 21 }, (_, index) => ({
+ x: index === 4 && thumbExtended ? 0.8 : index === 3 && thumbExtended ? 0.6 : 0.5,
+ y: 0.7,
+ z: 0,
+ visibility: 1,
+ }));
+ points[0].x = 0.5;
+ points[0].y = 0.8;
+
+ const fingerTips = [8, 12, 16, 20];
+ const fingerPips = [6, 10, 14, 18];
+ for (let index = 0; index < 4; index += 1) {
+ points[fingerTips[index]].y = extendedFingers.includes(index) ? 0.2 : 0.8;
+ points[fingerPips[index]].y = 0.5;
+ }
+ return points;
+}
+
+describe('hand gesture classification', () => {
+ it('recognizes basic hand poses from local landmarks', () => {
+ expect(classifyHandGesture(landmarks([0, 1, 2, 3], true))).toBe('Open palm');
+ expect(classifyHandGesture(landmarks([]))).toBe('Fist');
+ expect(classifyHandGesture(landmarks([0, 1]))).toBe('Peace sign');
+ expect(classifyHandGesture(landmarks([0]))).toBe('Pointing');
+ });
+
+ describe('camera device selection', () => {
+ it('prefers the built-in webcam over a phone camera when both are available', () => {
+ const cameras = [
+ { deviceId: 'phone', label: 'Phone Link Camera' },
+ { deviceId: 'webcam', label: 'Integrated Webcam' },
+ ];
+
+ expect(preferredCameraDevice(cameras)).toEqual(cameras[1]);
+ });
+
+ it('uses the first available camera when no built-in camera is listed', () => {
+ const cameras = [{ deviceId: 'phone', label: 'Phone Link Camera' }];
+
+ expect(preferredCameraDevice(cameras)).toEqual(cameras[0]);
+ expect(preferredCameraDevice([])).toBeUndefined();
+ });
+ });
+
+ it('avoids classifying incomplete landmark sets as a specific gesture', () => {
+ expect(classifyHandGesture(landmarks([0]).slice(0, 10))).toBe('Hand detected');
+ });
+});
diff --git a/frontend/src/components/HandTrackingPanel.tsx b/frontend/src/components/HandTrackingPanel.tsx
new file mode 100644
index 00000000..00e082bf
--- /dev/null
+++ b/frontend/src/components/HandTrackingPanel.tsx
@@ -0,0 +1,406 @@
+import { useCallback, useEffect, useRef, useState } from 'react';
+import {
+ FilesetResolver,
+ HandLandmarker,
+ type NormalizedLandmark,
+} from '@mediapipe/tasks-vision';
+import simdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_internal.js?url';
+import simdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_internal.wasm?url';
+import noSimdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.js?url';
+import noSimdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.wasm?url';
+import { Camera, Hand, LoaderCircle, StopCircle } from 'lucide-react';
+
+const HAND_CONNECTIONS = [
+ [0, 1], [1, 2], [2, 3], [3, 4],
+ [0, 5], [5, 6], [6, 7], [7, 8],
+ [5, 9], [9, 10], [10, 11], [11, 12],
+ [9, 13], [13, 14], [14, 15], [15, 16],
+ [13, 17], [17, 18], [18, 19], [19, 20], [0, 17],
+];
+
+export type HandGesture =
+ | 'Open palm'
+ | 'Fist'
+ | 'Peace sign'
+ | 'Pointing'
+ | 'Hand detected';
+
+export interface CameraDevice {
+ deviceId: string;
+ label: string;
+}
+
+export function preferredCameraDevice(devices: CameraDevice[]): CameraDevice | undefined {
+ return devices.find((device) => /integrated|built[\s-]?in|internal|webcam/i.test(device.label))
+ ?? devices[0];
+}
+
+export function classifyHandGesture(points: NormalizedLandmark[]): HandGesture {
+ if (points.length < 21) return 'Hand detected';
+
+ const extended = [
+ points[8].y < points[6].y - 0.02,
+ points[12].y < points[10].y - 0.02,
+ points[16].y < points[14].y - 0.02,
+ points[20].y < points[18].y - 0.02,
+ ];
+ const distanceFromWrist = (index: number) =>
+ Math.hypot(points[index].x - points[0].x, points[index].y - points[0].y);
+ const thumbExtended = distanceFromWrist(4) > distanceFromWrist(3) * 1.1;
+ const extendedCount = extended.filter(Boolean).length;
+
+ if (extendedCount === 4 && thumbExtended) return 'Open palm';
+ if (extendedCount === 0 && !thumbExtended) return 'Fist';
+ if (extended[0] && extended[1] && !extended[2] && !extended[3]) return 'Peace sign';
+ if (extended[0] && !extended[1] && !extended[2] && !extended[3]) return 'Pointing';
+ return 'Hand detected';
+}
+
+function localVisionFileset() {
+ return FilesetResolver.isSimdSupported().then(async (simdSupported) => {
+ const fileset = await FilesetResolver.forVisionTasks('/');
+ fileset.wasmLoaderPath = simdSupported ? simdLoaderUrl : noSimdLoaderUrl;
+ fileset.wasmBinaryPath = simdSupported ? simdWasmUrl : noSimdWasmUrl;
+ return fileset;
+ });
+}
+
+type TrackerState = 'idle' | 'choosing' | 'starting' | 'active' | 'error';
+
+export function HandTrackingPanel() {
+ const [state, setState] = useState<TrackerState>('idle');
+ const [error, setError] = useState('');
+ const [landmarks, setLandmarks] = useState<NormalizedLandmark[][]>([]);
+ const [gesture, setGesture] = useState('No hand detected');
+ const [cameras, setCameras] = useState<CameraDevice[]>([]);
+ const [selectedCameraId, setSelectedCameraId] = useState('');
+ const [focusOnOpenPalm, setFocusOnOpenPalm] = useState(
+ () => localStorage.getItem('openjarvis-hand-focus') === 'true',
+ );
+ const focusOnOpenPalmRef = useRef(focusOnOpenPalm);
+ const videoRef = useRef<HTMLVideoElement>(null);
+ const detectorRef = useRef<HandLandmarker | null>(null);
+ const streamRef = useRef<MediaStream | null>(null);
+ const frameRef = useRef<number | null>(null);
+ const generationRef = useRef(0);
+ const previousGestureRef = useRef<string>('');
+
+ const stopTracking = useCallback(() => {
+ generationRef.current += 1;
+ if (frameRef.current !== null) cancelAnimationFrame(frameRef.current);
+ frameRef.current = null;
+ detectorRef.current?.close();
+ detectorRef.current = null;
+ streamRef.current?.getTracks().forEach((track) => track.stop());
+ streamRef.current = null;
+ if (videoRef.current) videoRef.current.srcObject = null;
+ setLandmarks([]);
+ setGesture('No hand detected');
+ previousGestureRef.current = '';
+ setState('idle');
+ setError('');
+ }, []);
+
+ const chooseCamera = useCallback(async () => {
+ if (!navigator.mediaDevices?.getUserMedia) {
+ setError('Camera access is not available in this browser or context.');
+ setState('error');
+ return;
+ }
+
+ const generation = ++generationRef.current;
+ setError('');
+ setState('choosing');
+ try {
+ const permissionStream = await navigator.mediaDevices.getUserMedia({
+ video: true,
+ audio: false,
+ });
+ if (generationRef.current !== generation) {
+ permissionStream.getTracks().forEach((track) => track.stop());
+ return;
+ }
+
+ // Permission reveals device labels. Stop this temporary stream before
+ // enumerating or opening the user-selected camera.
+ permissionStream.getTracks().forEach((track) => track.stop());
+ const devices = (await navigator.mediaDevices.enumerateDevices())
+ .filter((device) => device.kind === 'videoinput')
+ .map(({ deviceId, label }, index) => ({
+ deviceId,
+ label: label || `Camera ${index + 1}`,
+ }));
+ if (generationRef.current !== generation) return;
+ if (devices.length === 0) throw new Error('No cameras were found.');
+
+ setCameras(devices);
+ setSelectedCameraId(preferredCameraDevice(devices)?.deviceId ?? devices[0].deviceId);
+ setState('idle');
+ } catch (cause) {
+ if (generationRef.current !== generation) return;
+ setError(
+ cause instanceof DOMException && cause.name === 'NotAllowedError'
+ ? 'Camera permission was denied. Allow camera access in your browser settings, then try again.'
+ : cause instanceof Error
+ ? cause.message
+ : 'Could not list available cameras.',
+ );
+ setState('error');
+ }
+ }, []);
+
+ const startTracking = useCallback(async () => {
+ if (!navigator.mediaDevices?.getUserMedia) {
+ setError('Camera access is not available in this browser or context.');
+ setState('error');
+ return;
+ }
+
+ const generation = ++generationRef.current;
+ setError('');
+ setState('starting');
+
+ try {
+ const stream = await navigator.mediaDevices.getUserMedia({
+ video: { deviceId: { exact: selectedCameraId } },
+ audio: false,
+ });
+ if (generationRef.current !== generation) {
+ stream.getTracks().forEach((track) => track.stop());
+ return;
+ }
+ streamRef.current = stream;
+ const video = videoRef.current;
+ if (!video) throw new Error('Camera preview is unavailable.');
+ video.srcObject = stream;
+ await video.play();
+
+ const fileset = await localVisionFileset();
+ const detector = await HandLandmarker.createFromOptions(fileset, {
+ baseOptions: {
+ modelAssetPath: new URL('/models/hand_landmarker.task', window.location.origin).href,
+ },
+ runningMode: 'VIDEO',
+ numHands: 2,
+ minHandDetectionConfidence: 0.55,
+ minTrackingConfidence: 0.55,
+ });
+
+ if (generationRef.current !== generation) {
+ detector.close();
+ stream.getTracks().forEach((track) => track.stop());
+ return;
+ }
+
+ detectorRef.current = detector;
+ setState('active');
+ let lastVideoTime = -1;
+ let lastUiUpdate = 0;
+
+ const detectFrame = () => {
+ if (generationRef.current !== generation) return;
+ const currentDetector = detectorRef.current;
+ if (
+ currentDetector &&
+ video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA &&
+ video.currentTime !== lastVideoTime
+ ) {
+ lastVideoTime = video.currentTime;
+ const result = currentDetector.detectForVideo(video, performance.now());
+ const detectedHands = result.landmarks;
+ const currentGesture = detectedHands.length
+ ? classifyHandGesture(detectedHands[0])
+ : 'No hand detected';
+
+ if (performance.now() - lastUiUpdate > 80) {
+ lastUiUpdate = performance.now();
+ setLandmarks(detectedHands);
+ setGesture(currentGesture);
+ }
+
+ if (
+ focusOnOpenPalmRef.current &&
+ currentGesture === 'Open palm' &&
+ previousGestureRef.current !== 'Open palm'
+ ) {
+ window.dispatchEvent(new Event('openjarvis:focus-chat-input'));
+ }
+ previousGestureRef.current = currentGesture;
+ }
+ frameRef.current = requestAnimationFrame(detectFrame);
+ };
+ frameRef.current = requestAnimationFrame(detectFrame);
+ } catch (cause) {
+ if (generationRef.current !== generation) return;
+ streamRef.current?.getTracks().forEach((track) => track.stop());
+ streamRef.current = null;
+ if (videoRef.current) videoRef.current.srcObject = null;
+ detectorRef.current?.close();
+ detectorRef.current = null;
+ setError(
+ cause instanceof DOMException && cause.name === 'NotAllowedError'
+ ? 'Camera permission was denied. Allow camera access in system settings, then try again.'
+ : cause instanceof Error
+ ? cause.message
+ : 'Could not start local hand tracking.',
+ );
+ setState('error');
+ }
+ }, [selectedCameraId]);
+
+ useEffect(() => () => {
+ generationRef.current += 1;
+ if (frameRef.current !== null) cancelAnimationFrame(frameRef.current);
+ detectorRef.current?.close();
+ streamRef.current?.getTracks().forEach((track) => track.stop());
+ }, []);
+
+ const updateFocusOption = (enabled: boolean) => {
+ setFocusOnOpenPalm(enabled);
+ focusOnOpenPalmRef.current = enabled;
+ localStorage.setItem('openjarvis-hand-focus', String(enabled));
+ };
+
+ return (
+ <div className="pt-3">
+ <div className="flex items-start justify-between gap-4">
+ <div className="min-w-0">
+ <div className="text-sm" style={{ color: 'var(--color-text)' }}>Camera hand tracking</div>
+ <p className="text-xs mt-1" style={{ color: 'var(--color-text-tertiary)' }}>
+ Optional, on-device landmark and basic gesture detection. Camera frames stay in this app and are not recorded.
+ </p>
+ </div>
+ <button
+ type="button"
+ onClick={() => {
+ if (state === 'active' || state === 'starting' || state === 'choosing') {
+ stopTracking();
+ } else if (cameras.length === 0) {
+ void chooseCamera();
+ } else {
+ void startTracking();
+ }
+ }}
+ aria-pressed={state === 'active' || state === 'starting'}
+ className="flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium cursor-pointer shrink-0"
+ style={{
+ background: state === 'active' ? 'var(--color-error)' : 'var(--color-bg-secondary)',
+ border: '1px solid var(--color-border)',
+ color: state === 'active' ? 'white' : 'var(--color-text)',
+ }}
+ >
+ {state === 'choosing'
+ ? <><StopCircle size={13} /> Cancel camera access</>
+ : state === 'starting'
+ ? <><StopCircle size={13} /> Stop camera</>
+ : state === 'active'
+ ? <><StopCircle size={13} /> Stop camera</>
+ : cameras.length === 0
+ ? <><Camera size={13} /> Choose camera</>
+ : <><Camera size={13} /> Start camera</>}
+ </button>
+ </div>
+
+ {cameras.length > 0 && state !== 'active' && (
+ <div className="flex items-center gap-3 mt-3">
+ <label htmlFor="openjarvis-camera-device" className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>
+ Camera device
+ </label>
+ <select
+ id="openjarvis-camera-device"
+ aria-label="Camera device"
+ value={selectedCameraId}
+ onChange={(event) => setSelectedCameraId(event.target.value)}
+ disabled={state === 'starting'}
+ className="text-xs px-2 py-1 rounded-lg outline-none max-w-full"
+ style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}
+ >
+ {cameras.map((camera) => (
+ <option key={camera.deviceId} value={camera.deviceId}>{camera.label}</option>
+ ))}
+ </select>
+ <button
+ type="button"
+ onClick={() => void chooseCamera()}
+ disabled={state === 'choosing' || state === 'starting'}
+ className="text-xs underline cursor-pointer"
+ style={{ color: 'var(--color-accent)' }}
+ >
+ Refresh
+ </button>
+ </div>
+ )}
+
+ {(state === 'active' || state === 'starting') && (
+ <div className="flex flex-col sm:flex-row items-start gap-4 mt-4">
+ <div className="relative w-full sm:w-64 aspect-video overflow-hidden rounded-lg bg-black">
+ <video
+ ref={videoRef}
+ muted
+ playsInline
+ aria-label="Live camera preview for hand tracking"
+ className="absolute inset-0 w-full h-full object-cover"
+ style={{ transform: 'scaleX(-1)' }}
+ />
+ <svg
+ aria-hidden="true"
+ viewBox="0 0 1 1"
+ preserveAspectRatio="none"
+ className="absolute inset-0 w-full h-full"
+ style={{ transform: 'scaleX(-1)' }}
+ >
+ {landmarks.map((hand, handIndex) => (
+ <g key={handIndex}>
+ {HAND_CONNECTIONS.map(([start, end]) => (
+ <line
+ key={`${start}-${end}`}
+ x1={hand[start].x}
+ y1={hand[start].y}
+ x2={hand[end].x}
+ y2={hand[end].y}
+ stroke="#34d399"
+ strokeWidth="0.009"
+ />
+ ))}
+ {hand.map((point, pointIndex) => (
+ <circle key={pointIndex} cx={point.x} cy={point.y} r="0.014" fill="#fbbf24" />
+ ))}
+ </g>
+ ))}
+ </svg>
+ </div>
+ <div aria-live="polite" className="flex items-center gap-2 text-xs" style={{ color: 'var(--color-text-secondary)' }}>
+ <Hand size={15} style={{ color: landmarks.length ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />
+ {state === 'starting'
+ ? 'Starting local hand tracker…'
+ : landmarks.length
+ ? `${landmarks.length} hand${landmarks.length === 1 ? '' : 's'} — ${gesture}`
+ : 'No hand detected'}
+ </div>
+ </div>
+ )}
+
+ {state === 'error' && (
+ <p role="alert" className="text-xs mt-3" style={{ color: 'var(--color-error)' }}>{error}</p>
+ )}
+ {cameras.length > 0 && state !== 'active' && (
+ <p className="text-xs mt-2" style={{ color: 'var(--color-text-tertiary)' }}>
+ Choose the webcam by name before starting. The camera used to reveal device names was stopped immediately.
+ </p>
+ )}
+
+ <label className="flex items-center gap-2 mt-3 text-xs cursor-pointer" style={{ color: 'var(--color-text-secondary)' }}>
+ <input
+ type="checkbox"
+ checked={focusOnOpenPalm}
+ onChange={(event) => updateFocusOption(event.target.checked)}
+ className="accent-[var(--color-accent)]"
+ />
+ Focus chat input when an open palm is detected
+ </label>
+ <p className="text-xs mt-1 pl-5" style={{ color: 'var(--color-text-tertiary)' }}>
+ Optional and off by default. Gestures never send messages or trigger tools.
+ </p>
+ </div>
+ );
+}
diff --git a/frontend/src/pages/SettingsPage.tsx b/frontend/src/pages/SettingsPage.tsx
index 6b8bdc1d..db57947d 100644
--- a/frontend/src/pages/SettingsPage.tsx
+++ b/frontend/src/pages/SettingsPage.tsx
@@ -17,8 +17,11 @@ import {
Search,
Brain,
RefreshCw,
+ Server,
+ Wifi,
} from 'lucide-react';
import { useAppStore, type ThemeMode } from '../lib/store';
+import { HandTrackingPanel } from '../components/HandTrackingPanel';
import {
checkHealth,
fetchSpeechHealth,
@@ -229,6 +232,36 @@ function SettingRow({ label, description, children }: { label: string; descripti
);
}
+export function NetworkStatusRow() {
+ const [online, setOnline] = useState<boolean | null>(
+ () => typeof navigator === 'undefined' ? null : navigator.onLine,
+ );
+
+ useEffect(() => {
+ const updateOnline = () => setOnline(navigator.onLine);
+ window.addEventListener('online', updateOnline);
+ window.addEventListener('offline', updateOnline);
+ return () => {
+ window.removeEventListener('online', updateOnline);
+ window.removeEventListener('offline', updateOnline);
+ };
+ }, []);
+
+ return (
+ <SettingRow
+ label="Browser network"
+ description="Browser signal only; it does not verify internet or cloud-provider reachability."
+ >
+ <div className="flex items-center gap-2">
+ <Wifi size={14} style={{ color: online ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />
+ <span className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>
+ {online === null ? 'Unknown' : online ? 'Online' : 'Offline'}
+ </span>
+ </div>
+ </SettingRow>
+ );
+}
+
type MemoryStatus =
| { kind: 'loading' }
| { kind: 'ready'; stats: MemoryStats }
@@ -340,6 +373,7 @@ export function SettingsPage() {
const [customEngine, setCustomEngine] = useState('lmstudio');
const [customKey, setCustomKey] = useState('');
const [srcMsg, setSrcMsg] = useState('');
+ const inferenceConfigurable = isTauri();
useEffect(() => {
getInferenceSource().then((s) => {
@@ -520,6 +554,7 @@ export function SettingsPage() {
{/* Connection */}
<Section title="Connection">
+ <NetworkStatusRow />
<SettingRow label="Server status" description={serverInfo ? `${serverInfo.engine} / ${serverInfo.model}` : 'Not connected'}>
<div className="flex items-center gap-2">
<span
@@ -564,10 +599,14 @@ export function SettingsPage() {
{/* Inference source */}
<Section title="Inference source">
- <SettingRow label="Source" description="Where the app runs models. Applies after restart.">
+ <p className="text-xs mb-2" style={{ color: 'var(--color-text-tertiary)' }}>
+ Ollama is supported for local inference. The desktop app can connect to Colibri through its OpenAI-compatible API; OpenJarvis agents provide multi-step tool use and reasoning, while Colibri serves one model at a time. Maya AI is not wired in: its reviewed assistant script has no supported service or plugin API, so it cannot be safely managed here.
+ </p>
+ <SettingRow label="Source" description={inferenceConfigurable ? 'Where the app runs models. Applies after restart.' : 'The browser uses the backend’s configured model source; change inference sources in the desktop app.'}>
<select
value={srcKind}
onChange={(e) => { setSrcKind(e.target.value as InferenceSource['kind']); setSrcMsg(''); }}
+ disabled={!inferenceConfigurable}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}
>
@@ -577,21 +616,39 @@ export function SettingsPage() {
</SettingRow>
{srcKind === 'custom' && (
<>
- <SettingRow label="Server URL" description="e.g. LM Studio: http://localhost:1234/v1">
- <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1"
+ <SettingRow label="Colibri preset" description="Sets the desktop app's LM Studio-compatible adapter target; configure Colibri separately and enter its exact model ID below.">
+ <button
+ type="button"
+ onClick={() => {
+ setSrcKind('custom');
+ setCustomEngine('lmstudio');
+ setCustomHost('http://127.0.0.1:8001');
+ setCustomModel('');
+ setSrcMsg('Colibri preset selected. Enter a model ID exposed by Colibri, then save and restart.');
+ }}
+ disabled={!inferenceConfigurable}
+ title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}
+ className="text-xs px-3 py-1.5 rounded-lg cursor-pointer"
+ style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}
+ >
+ <span className="flex items-center gap-1.5"><Server size={13} /> Use Colibri preset</span>
+ </button>
+ </SettingRow>
+ <SettingRow label="Server URL" description={customEngine === 'lmstudio' ? 'Base URL. For Colibri, configure loopback:8001 to avoid Jarvis’s default port 8000.' : 'OpenAI-compatible server base URL'}>
+ <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1" disabled={!inferenceConfigurable}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} />
</SettingRow>
- <SettingRow label="Model" description="Model id served by your endpoint">
- <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct"
+ <SettingRow label="Model" description={customEngine === 'lmstudio' ? 'Enter the exact model id from your server’s GET /v1/models response.' : 'Model id served by your endpoint'}>
+ <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct" disabled={!inferenceConfigurable}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} />
</SettingRow>
- <SettingRow label="Server type" description="OpenAI-compatible engine">
- <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }}
+ <SettingRow label="Server type" description={customEngine === 'lmstudio' ? 'LM Studio adapter; also compatible with Colibri’s OpenAI-style /v1 API.' : 'OpenAI-compatible inference backend'}>
+ <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }} disabled={!inferenceConfigurable}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>
- <option value="lmstudio">LM Studio</option>
+ <option value="lmstudio">LM Studio / Colibri-compatible</option>
<option value="vllm">vLLM</option>
<option value="sglang">SGLang</option>
<option value="llamacpp">llama.cpp</option>
@@ -599,7 +656,7 @@ export function SettingsPage() {
</select>
</SettingRow>
<SettingRow label="API key (optional)" description="Only if your server requires one">
- <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none"
+ <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none" disabled={!inferenceConfigurable}
className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"
style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} />
</SettingRow>
@@ -607,8 +664,10 @@ export function SettingsPage() {
)}
<SettingRow label="" description={srcMsg}>
<button onClick={saveSource}
+ disabled={!inferenceConfigurable}
+ title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}
className="text-sm px-3 py-1.5 rounded-lg outline-none cursor-pointer"
- style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>
+ style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}>
Save inference source
</button>
</SettingRow>
@@ -876,6 +935,14 @@ export function SettingsPage() {
See the <a href="https://open-jarvis.github.io/OpenJarvis/user-guide/tools/" target="_blank" rel="noopener noreferrer" style={{ color: 'var(--color-accent)' }}>documentation</a> for details.
</div>
)}
+ <p className="text-xs mt-3" style={{ color: 'var(--color-text-tertiary)' }}>
+ Offline speech requires a configured local backend and downloaded model. Remote speech providers may send audio off-device.
+ </p>
+ </Section>
+
+ {/* Hand tracking */}
+ <Section title="Hand tracking">
+ <HandTrackingPanel />
</Section>
{/* Data */}
diff --git a/frontend/vite.config.ts b/frontend/vite.config.ts
index 360565cd..57ca3f31 100644
--- a/frontend/vite.config.ts
+++ b/frontend/vite.config.ts
@@ -37,7 +37,8 @@ export default defineConfig({
],
},
workbox: {
- globPatterns: ['**/*.{js,css,html,ico,png,svg}'],
+ globPatterns: ['**/*.{js,css,html,ico,png,svg,wasm,task}'],
+ maximumFileSizeToCacheInBytes: 16 * 1024 * 1024,
navigateFallbackDenylist: [/^\/v1\//, /^\/health/, /^\/dashboard/, /^\/api\//],
},
}),
diff --git a/tests/engine/test_openai_compat_api_key.py b/tests/engine/test_openai_compat_api_key.py
index dc541358..65a9d229 100644
--- a/tests/engine/test_openai_compat_api_key.py
+++ b/tests/engine/test_openai_compat_api_key.py
@@ -2,12 +2,15 @@
from __future__ import annotations
+import json
+
import httpx
import pytest
import respx
from openjarvis.core.types import Message, Role
from openjarvis.engine.openai_compat_engines import (
+ LMStudioEngine,
OpenAICompatEngine,
VLLMEngine,
normalize_openai_base_url,
@@ -79,6 +82,33 @@ class TestAuthorizationHeader:
engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")
assert engine._api_key == "sk-explicit"
+ def test_colibri_compatible_lmstudio_engine_uses_model_and_bearer_key(self) -> None:
+ model_id = "colibri/qwen3-8b"
+ engine = LMStudioEngine(host="http://127.0.0.1:8001", api_key="coli-test")
+ with respx.mock:
+ route = respx.post("http://127.0.0.1:8001/v1/chat/completions").mock(
+ return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+ )
+ engine.generate([Message(role=Role.USER, content="hi")], model=model_id)
+ request = route.calls.last.request
+ assert request.headers["Authorization"] == "Bearer coli-test"
+ assert request.url.path == "/v1/chat/completions"
+ assert json.loads(request.content)["model"] == model_id
+
+ def test_lmstudio_model_list_uses_single_v1_prefix_and_optional_auth(self) -> None:
+ host = normalize_openai_base_url("http://127.0.0.1:8001/v1")
+ engine = LMStudioEngine(host=host, api_key="coli-test")
+ with respx.mock:
+ route = respx.get("http://127.0.0.1:8001/v1/models").mock(
+ return_value=httpx.Response(
+ 200, json={"data": [{"id": "colibri/qwen3-8b"}]}
+ )
+ )
+ assert engine.list_models() == ["colibri/qwen3-8b"]
+ request = route.calls.last.request
+ assert request.url.path == "/v1/models"
+ assert request.headers["Authorization"] == "Bearer coli-test"
+
class TestNormalizeOpenAIBaseUrl:
@pytest.mark.parametrize(