Personal AI, On Personal Devices
Python
feat: add offline hand tracking and Colibri setup
AzeftheGoat18747688b37d2c5
diff --git a/docs/architecture/design-principles.md b/docs/architecture/design-principles.mdindex 54b09712..5d6e1412 100644--- a/docs/architecture/design-principles.md+++ b/docs/architecture/design-principles.md@@ -94,6 +94,7 @@ OpenJarvis is designed to work **entirely without network access**. All core fun| Traces | SQLite-based, fully local || Tools | Calculator, Think, FileRead all local || Configuration | TOML file on disk |+| Hand tracking | Optional MediaPipe model and WASM assets run locally after installation |Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cloud` backend, but they are:@@ -101,6 +102,14 @@ Cloud engines (OpenAI, Anthropic, Google) are available through the optional `cl- Only activated if API keys are set as environment variables- Never required for any core functionality+Offline operation still requires the local API server, inference runtime, and model to+be installed and available. The desktop app and PWA can cache their interface assets,+but API inference requests are not cached or replayed offline. Speech recognition is+offline only when a local speech backend and its model are configured; remote speech+providers require network access and may receive audio. Hand tracking is opt-in and+uses the bundled model locally after the app has been installed or its PWA assets+cached.+```python# This works without any network connectionfrom openjarvis import Jarvisdiff --git a/docs/user-guide/chat-simple.md b/docs/user-guide/chat-simple.mdindex 4bab5538..7871d176 100644--- a/docs/user-guide/chat-simple.md+++ b/docs/user-guide/chat-simple.md@@ -110,6 +110,58 @@ jarvis serveThe server is OpenAI-compatible, so any client that works with the OpenAI API can point to `http://localhost:8000/v1`.+## Connecting Colibri++OpenJarvis can use a Colibri instance that exposes OpenAI-compatible Chat+Completions and model-list endpoints. In desktop **Settings → Inference source**,+choose **Custom OpenAI-compatible server**, then click **Use Colibri preset**.+Configure Colibri separately to bind to loopback on port `8001` (Jarvis uses port+`8000` by default). Enter the exact model ID reported by Colibri, add its API key+only if Colibri requires one, save, and restart OpenJarvis. The preset only sets+OpenJarvis's connection target; it does not install or start Colibri. To find the+model ID, request `http://127.0.0.1:8001/v1/models` (for example,+`curl http://127.0.0.1:8001/v1/models`) and copy the exact `id` value. Enter the+base URL as either `http://127.0.0.1:8001` or `http://127.0.0.1:8001/v1`; the+desktop normalizes either form so requests use one `/v1` prefix. Optional API+keys are stored in desktop secure storage and sent as bearer authorization.+Keep Colibri bound to loopback for local use; the preset does not change its bind+address or expose it to the network.++This connects Colibri for inference; OpenJarvis remains responsible for running+its own configured tools when the selected agent requests them. Colibri's optional+MCP server manages Colibri's lifecycle and is not itself a tool executor. Colibri+serves one active model at a time, so multi-model routing remains an OpenJarvis+capability and is not provided by this connection.++Ollama is already a supported local inference source and its installed models are+listed in Settings. Maya AI is not integrated: the reviewed assistant script does+not expose a supported service or MCP tool interface and contains direct OS+actions. Do not run or wrap that script as an OpenJarvis tool.++## Voice, camera, and offline use++Microphone transcription requires the Speech-to-Text setting and a configured+speech backend. Offline transcription requires a local backend and model; a remote+provider can send audio off-device. Camera hand tracking is optional in+**Settings → Hand tracking**: it asks for camera permission only when started,+detects landmarks and basic gestures locally, and stops when stopped or when the+settings page closes. The first camera action briefly requests permission to reveal+camera names, then immediately stops that temporary stream. Select the intended+device (for example, **Integrated Webcam**) from the camera list before pressing+**Start camera**. Open palm can optionally focus the chat input; gestures never+submit messages or call tools. It uses the bundled MediaPipe Tasks Vision runtime+and [Hand Landmarker model](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker).+The browser's online indicator is only a connectivity hint, not proof that a+particular server is reachable.++Local inference and cached interface assets do not make the assistant universally+autonomous. No financial or business-venture integration is configured here, and+OpenJarvis cannot guarantee legal compliance or revenue. It does not autonomously+retrain model weights; its existing memory, traces, and learning policies are+separate capabilities. MCP tools should be narrowly allowlisted, with integration-+level approval and audit controls for financial transactions, contracts, public+communications, and other irreversible actions.+## Using the Desktop App1. Start the backend: `jarvis serve` (or `./scripts/quickstart.sh`)diff --git a/frontend/package-lock.json b/frontend/package-lock.jsonindex ea28b996..86b19b21 100644--- a/frontend/package-lock.json+++ b/frontend/package-lock.json@@ -10,6 +10,7 @@"dependencies": {"@base-ui/react": "^1.8.0","@fontsource-variable/geist": "^5.3.0",+ "@mediapipe/tasks-vision": "^1.1.0","@tailwindcss/vite": "^4.3.3","@tauri-apps/api": "^2.12.0","@tauri-apps/plugin-autostart": "^2",@@ -1987,6 +1988,12 @@"@jridgewell/sourcemap-codec": "^1.4.14"}},+ "node_modules/@mediapipe/tasks-vision": {+ "version": "1.1.0",+ "resolved": "https://registry.npmjs.org/@mediapipe/tasks-vision/-/tasks-vision-1.1.0.tgz",+ "integrity": "sha512-ZJqh0wMKOINorfSffvDAxzyO1//c+FiG3IkzOdNDnLqvBB3DbMfj0bB/tys2SNf9S914sgoIbIpF4HTOi336cg==",+ "license": "Apache-2.0"+ },"node_modules/@modelcontextprotocol/sdk": {"version": "1.28.0","resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.28.0.tgz",diff --git a/frontend/package.json b/frontend/package.jsonindex 07cd449d..7f8826cc 100644--- a/frontend/package.json+++ b/frontend/package.json@@ -19,6 +19,7 @@"dependencies": {"@base-ui/react": "^1.8.0","@fontsource-variable/geist": "^5.3.0",+ "@mediapipe/tasks-vision": "^1.1.0","@tailwindcss/vite": "^4.3.3","@tauri-apps/api": "^2.12.0","@tauri-apps/plugin-autostart": "^2",diff --git a/frontend/public/models/hand_landmarker.task b/frontend/public/models/hand_landmarker.tasknew file mode 100644index 00000000..0d53faf3Binary files /dev/null and b/frontend/public/models/hand_landmarker.task differdiff --git a/frontend/src/components/Chat/InputArea.tsx b/frontend/src/components/Chat/InputArea.tsxindex 79a0849e..5c552738 100644--- a/frontend/src/components/Chat/InputArea.tsx+++ b/frontend/src/components/Chat/InputArea.tsx@@ -162,6 +162,12 @@ export function InputArea() {el.style.height = Math.min(el.scrollHeight, 200) + 'px';}, [input]);+ useEffect(() => {+ const focusInput = () => textareaRef.current?.focus();+ window.addEventListener('openjarvis:focus-chat-input', focusInput);+ return () => window.removeEventListener('openjarvis:focus-chat-input', focusInput);+ }, []);+const stopStreaming = useCallback(() => {abortRef.current?.abort();if (timerRef.current) {diff --git a/frontend/src/components/HandTrackingPanel.test.ts b/frontend/src/components/HandTrackingPanel.test.tsnew file mode 100644index 00000000..271cf567--- /dev/null+++ b/frontend/src/components/HandTrackingPanel.test.ts@@ -0,0 +1,56 @@+import { describe, expect, it } from 'vitest';+import type { NormalizedLandmark } from '@mediapipe/tasks-vision';+import { classifyHandGesture, preferredCameraDevice } from './HandTrackingPanel';++function landmarks(+ extendedFingers: number[],+ thumbExtended = false,+): NormalizedLandmark[] {+ const points = Array.from({ length: 21 }, (_, index) => ({+ x: index === 4 && thumbExtended ? 0.8 : index === 3 && thumbExtended ? 0.6 : 0.5,+ y: 0.7,+ z: 0,+ visibility: 1,+ }));+ points[0].x = 0.5;+ points[0].y = 0.8;++ const fingerTips = [8, 12, 16, 20];+ const fingerPips = [6, 10, 14, 18];+ for (let index = 0; index < 4; index += 1) {+ points[fingerTips[index]].y = extendedFingers.includes(index) ? 0.2 : 0.8;+ points[fingerPips[index]].y = 0.5;+ }+ return points;+}++describe('hand gesture classification', () => {+ it('recognizes basic hand poses from local landmarks', () => {+ expect(classifyHandGesture(landmarks([0, 1, 2, 3], true))).toBe('Open palm');+ expect(classifyHandGesture(landmarks([]))).toBe('Fist');+ expect(classifyHandGesture(landmarks([0, 1]))).toBe('Peace sign');+ expect(classifyHandGesture(landmarks([0]))).toBe('Pointing');+ });++ describe('camera device selection', () => {+ it('prefers the built-in webcam over a phone camera when both are available', () => {+ const cameras = [+ { deviceId: 'phone', label: 'Phone Link Camera' },+ { deviceId: 'webcam', label: 'Integrated Webcam' },+ ];++ expect(preferredCameraDevice(cameras)).toEqual(cameras[1]);+ });++ it('uses the first available camera when no built-in camera is listed', () => {+ const cameras = [{ deviceId: 'phone', label: 'Phone Link Camera' }];++ expect(preferredCameraDevice(cameras)).toEqual(cameras[0]);+ expect(preferredCameraDevice([])).toBeUndefined();+ });+ });++ it('avoids classifying incomplete landmark sets as a specific gesture', () => {+ expect(classifyHandGesture(landmarks([0]).slice(0, 10))).toBe('Hand detected');+ });+});diff --git a/frontend/src/components/HandTrackingPanel.tsx b/frontend/src/components/HandTrackingPanel.tsxnew file mode 100644index 00000000..00e082bf--- /dev/null+++ b/frontend/src/components/HandTrackingPanel.tsx@@ -0,0 +1,406 @@+import { useCallback, useEffect, useRef, useState } from 'react';+import {+ FilesetResolver,+ HandLandmarker,+ type NormalizedLandmark,+} from '@mediapipe/tasks-vision';+import simdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_internal.js?url';+import simdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_internal.wasm?url';+import noSimdLoaderUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.js?url';+import noSimdWasmUrl from '@mediapipe/tasks-vision/vision_wasm_nosimd_internal.wasm?url';+import { Camera, Hand, LoaderCircle, StopCircle } from 'lucide-react';++const HAND_CONNECTIONS = [+ [0, 1], [1, 2], [2, 3], [3, 4],+ [0, 5], [5, 6], [6, 7], [7, 8],+ [5, 9], [9, 10], [10, 11], [11, 12],+ [9, 13], [13, 14], [14, 15], [15, 16],+ [13, 17], [17, 18], [18, 19], [19, 20], [0, 17],+];++export type HandGesture =+ | 'Open palm'+ | 'Fist'+ | 'Peace sign'+ | 'Pointing'+ | 'Hand detected';++export interface CameraDevice {+ deviceId: string;+ label: string;+}++export function preferredCameraDevice(devices: CameraDevice[]): CameraDevice | undefined {+ return devices.find((device) => /integrated|built[\s-]?in|internal|webcam/i.test(device.label))+ ?? devices[0];+}++export function classifyHandGesture(points: NormalizedLandmark[]): HandGesture {+ if (points.length < 21) return 'Hand detected';++ const extended = [+ points[8].y < points[6].y - 0.02,+ points[12].y < points[10].y - 0.02,+ points[16].y < points[14].y - 0.02,+ points[20].y < points[18].y - 0.02,+ ];+ const distanceFromWrist = (index: number) =>+ Math.hypot(points[index].x - points[0].x, points[index].y - points[0].y);+ const thumbExtended = distanceFromWrist(4) > distanceFromWrist(3) * 1.1;+ const extendedCount = extended.filter(Boolean).length;++ if (extendedCount === 4 && thumbExtended) return 'Open palm';+ if (extendedCount === 0 && !thumbExtended) return 'Fist';+ if (extended[0] && extended[1] && !extended[2] && !extended[3]) return 'Peace sign';+ if (extended[0] && !extended[1] && !extended[2] && !extended[3]) return 'Pointing';+ return 'Hand detected';+}++function localVisionFileset() {+ return FilesetResolver.isSimdSupported().then(async (simdSupported) => {+ const fileset = await FilesetResolver.forVisionTasks('/');+ fileset.wasmLoaderPath = simdSupported ? simdLoaderUrl : noSimdLoaderUrl;+ fileset.wasmBinaryPath = simdSupported ? simdWasmUrl : noSimdWasmUrl;+ return fileset;+ });+}++type TrackerState = 'idle' | 'choosing' | 'starting' | 'active' | 'error';++export function HandTrackingPanel() {+ const [state, setState] = useState<TrackerState>('idle');+ const [error, setError] = useState('');+ const [landmarks, setLandmarks] = useState<NormalizedLandmark[][]>([]);+ const [gesture, setGesture] = useState('No hand detected');+ const [cameras, setCameras] = useState<CameraDevice[]>([]);+ const [selectedCameraId, setSelectedCameraId] = useState('');+ const [focusOnOpenPalm, setFocusOnOpenPalm] = useState(+ () => localStorage.getItem('openjarvis-hand-focus') === 'true',+ );+ const focusOnOpenPalmRef = useRef(focusOnOpenPalm);+ const videoRef = useRef<HTMLVideoElement>(null);+ const detectorRef = useRef<HandLandmarker | null>(null);+ const streamRef = useRef<MediaStream | null>(null);+ const frameRef = useRef<number | null>(null);+ const generationRef = useRef(0);+ const previousGestureRef = useRef<string>('');++ const stopTracking = useCallback(() => {+ generationRef.current += 1;+ if (frameRef.current !== null) cancelAnimationFrame(frameRef.current);+ frameRef.current = null;+ detectorRef.current?.close();+ detectorRef.current = null;+ streamRef.current?.getTracks().forEach((track) => track.stop());+ streamRef.current = null;+ if (videoRef.current) videoRef.current.srcObject = null;+ setLandmarks([]);+ setGesture('No hand detected');+ previousGestureRef.current = '';+ setState('idle');+ setError('');+ }, []);++ const chooseCamera = useCallback(async () => {+ if (!navigator.mediaDevices?.getUserMedia) {+ setError('Camera access is not available in this browser or context.');+ setState('error');+ return;+ }++ const generation = ++generationRef.current;+ setError('');+ setState('choosing');+ try {+ const permissionStream = await navigator.mediaDevices.getUserMedia({+ video: true,+ audio: false,+ });+ if (generationRef.current !== generation) {+ permissionStream.getTracks().forEach((track) => track.stop());+ return;+ }++ // Permission reveals device labels. Stop this temporary stream before+ // enumerating or opening the user-selected camera.+ permissionStream.getTracks().forEach((track) => track.stop());+ const devices = (await navigator.mediaDevices.enumerateDevices())+ .filter((device) => device.kind === 'videoinput')+ .map(({ deviceId, label }, index) => ({+ deviceId,+ label: label || `Camera ${index + 1}`,+ }));+ if (generationRef.current !== generation) return;+ if (devices.length === 0) throw new Error('No cameras were found.');++ setCameras(devices);+ setSelectedCameraId(preferredCameraDevice(devices)?.deviceId ?? devices[0].deviceId);+ setState('idle');+ } catch (cause) {+ if (generationRef.current !== generation) return;+ setError(+ cause instanceof DOMException && cause.name === 'NotAllowedError'+ ? 'Camera permission was denied. Allow camera access in your browser settings, then try again.'+ : cause instanceof Error+ ? cause.message+ : 'Could not list available cameras.',+ );+ setState('error');+ }+ }, []);++ const startTracking = useCallback(async () => {+ if (!navigator.mediaDevices?.getUserMedia) {+ setError('Camera access is not available in this browser or context.');+ setState('error');+ return;+ }++ const generation = ++generationRef.current;+ setError('');+ setState('starting');++ try {+ const stream = await navigator.mediaDevices.getUserMedia({+ video: { deviceId: { exact: selectedCameraId } },+ audio: false,+ });+ if (generationRef.current !== generation) {+ stream.getTracks().forEach((track) => track.stop());+ return;+ }+ streamRef.current = stream;+ const video = videoRef.current;+ if (!video) throw new Error('Camera preview is unavailable.');+ video.srcObject = stream;+ await video.play();++ const fileset = await localVisionFileset();+ const detector = await HandLandmarker.createFromOptions(fileset, {+ baseOptions: {+ modelAssetPath: new URL('/models/hand_landmarker.task', window.location.origin).href,+ },+ runningMode: 'VIDEO',+ numHands: 2,+ minHandDetectionConfidence: 0.55,+ minTrackingConfidence: 0.55,+ });++ if (generationRef.current !== generation) {+ detector.close();+ stream.getTracks().forEach((track) => track.stop());+ return;+ }++ detectorRef.current = detector;+ setState('active');+ let lastVideoTime = -1;+ let lastUiUpdate = 0;++ const detectFrame = () => {+ if (generationRef.current !== generation) return;+ const currentDetector = detectorRef.current;+ if (+ currentDetector &&+ video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA &&+ video.currentTime !== lastVideoTime+ ) {+ lastVideoTime = video.currentTime;+ const result = currentDetector.detectForVideo(video, performance.now());+ const detectedHands = result.landmarks;+ const currentGesture = detectedHands.length+ ? classifyHandGesture(detectedHands[0])+ : 'No hand detected';++ if (performance.now() - lastUiUpdate > 80) {+ lastUiUpdate = performance.now();+ setLandmarks(detectedHands);+ setGesture(currentGesture);+ }++ if (+ focusOnOpenPalmRef.current &&+ currentGesture === 'Open palm' &&+ previousGestureRef.current !== 'Open palm'+ ) {+ window.dispatchEvent(new Event('openjarvis:focus-chat-input'));+ }+ previousGestureRef.current = currentGesture;+ }+ frameRef.current = requestAnimationFrame(detectFrame);+ };+ frameRef.current = requestAnimationFrame(detectFrame);+ } catch (cause) {+ if (generationRef.current !== generation) return;+ streamRef.current?.getTracks().forEach((track) => track.stop());+ streamRef.current = null;+ if (videoRef.current) videoRef.current.srcObject = null;+ detectorRef.current?.close();+ detectorRef.current = null;+ setError(+ cause instanceof DOMException && cause.name === 'NotAllowedError'+ ? 'Camera permission was denied. Allow camera access in system settings, then try again.'+ : cause instanceof Error+ ? cause.message+ : 'Could not start local hand tracking.',+ );+ setState('error');+ }+ }, [selectedCameraId]);++ useEffect(() => () => {+ generationRef.current += 1;+ if (frameRef.current !== null) cancelAnimationFrame(frameRef.current);+ detectorRef.current?.close();+ streamRef.current?.getTracks().forEach((track) => track.stop());+ }, []);++ const updateFocusOption = (enabled: boolean) => {+ setFocusOnOpenPalm(enabled);+ focusOnOpenPalmRef.current = enabled;+ localStorage.setItem('openjarvis-hand-focus', String(enabled));+ };++ return (+ <div className="pt-3">+ <div className="flex items-start justify-between gap-4">+ <div className="min-w-0">+ <div className="text-sm" style={{ color: 'var(--color-text)' }}>Camera hand tracking</div>+ <p className="text-xs mt-1" style={{ color: 'var(--color-text-tertiary)' }}>+ Optional, on-device landmark and basic gesture detection. Camera frames stay in this app and are not recorded.+ </p>+ </div>+ <button+ type="button"+ onClick={() => {+ if (state === 'active' || state === 'starting' || state === 'choosing') {+ stopTracking();+ } else if (cameras.length === 0) {+ void chooseCamera();+ } else {+ void startTracking();+ }+ }}+ aria-pressed={state === 'active' || state === 'starting'}+ className="flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium cursor-pointer shrink-0"+ style={{+ background: state === 'active' ? 'var(--color-error)' : 'var(--color-bg-secondary)',+ border: '1px solid var(--color-border)',+ color: state === 'active' ? 'white' : 'var(--color-text)',+ }}+ >+ {state === 'choosing'+ ? <><StopCircle size={13} /> Cancel camera access</>+ : state === 'starting'+ ? <><StopCircle size={13} /> Stop camera</>+ : state === 'active'+ ? <><StopCircle size={13} /> Stop camera</>+ : cameras.length === 0+ ? <><Camera size={13} /> Choose camera</>+ : <><Camera size={13} /> Start camera</>}+ </button>+ </div>++ {cameras.length > 0 && state !== 'active' && (+ <div className="flex items-center gap-3 mt-3">+ <label htmlFor="openjarvis-camera-device" className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>+ Camera device+ </label>+ <select+ id="openjarvis-camera-device"+ aria-label="Camera device"+ value={selectedCameraId}+ onChange={(event) => setSelectedCameraId(event.target.value)}+ disabled={state === 'starting'}+ className="text-xs px-2 py-1 rounded-lg outline-none max-w-full"+ style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}+ >+ {cameras.map((camera) => (+ <option key={camera.deviceId} value={camera.deviceId}>{camera.label}</option>+ ))}+ </select>+ <button+ type="button"+ onClick={() => void chooseCamera()}+ disabled={state === 'choosing' || state === 'starting'}+ className="text-xs underline cursor-pointer"+ style={{ color: 'var(--color-accent)' }}+ >+ Refresh+ </button>+ </div>+ )}++ {(state === 'active' || state === 'starting') && (+ <div className="flex flex-col sm:flex-row items-start gap-4 mt-4">+ <div className="relative w-full sm:w-64 aspect-video overflow-hidden rounded-lg bg-black">+ <video+ ref={videoRef}+ muted+ playsInline+ aria-label="Live camera preview for hand tracking"+ className="absolute inset-0 w-full h-full object-cover"+ style={{ transform: 'scaleX(-1)' }}+ />+ <svg+ aria-hidden="true"+ viewBox="0 0 1 1"+ preserveAspectRatio="none"+ className="absolute inset-0 w-full h-full"+ style={{ transform: 'scaleX(-1)' }}+ >+ {landmarks.map((hand, handIndex) => (+ <g key={handIndex}>+ {HAND_CONNECTIONS.map(([start, end]) => (+ <line+ key={`${start}-${end}`}+ x1={hand[start].x}+ y1={hand[start].y}+ x2={hand[end].x}+ y2={hand[end].y}+ stroke="#34d399"+ strokeWidth="0.009"+ />+ ))}+ {hand.map((point, pointIndex) => (+ <circle key={pointIndex} cx={point.x} cy={point.y} r="0.014" fill="#fbbf24" />+ ))}+ </g>+ ))}+ </svg>+ </div>+ <div aria-live="polite" className="flex items-center gap-2 text-xs" style={{ color: 'var(--color-text-secondary)' }}>+ <Hand size={15} style={{ color: landmarks.length ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />+ {state === 'starting'+ ? 'Starting local hand tracker…'+ : landmarks.length+ ? `${landmarks.length} hand${landmarks.length === 1 ? '' : 's'} — ${gesture}`+ : 'No hand detected'}+ </div>+ </div>+ )}++ {state === 'error' && (+ <p role="alert" className="text-xs mt-3" style={{ color: 'var(--color-error)' }}>{error}</p>+ )}+ {cameras.length > 0 && state !== 'active' && (+ <p className="text-xs mt-2" style={{ color: 'var(--color-text-tertiary)' }}>+ Choose the webcam by name before starting. The camera used to reveal device names was stopped immediately.+ </p>+ )}++ <label className="flex items-center gap-2 mt-3 text-xs cursor-pointer" style={{ color: 'var(--color-text-secondary)' }}>+ <input+ type="checkbox"+ checked={focusOnOpenPalm}+ onChange={(event) => updateFocusOption(event.target.checked)}+ className="accent-[var(--color-accent)]"+ />+ Focus chat input when an open palm is detected+ </label>+ <p className="text-xs mt-1 pl-5" style={{ color: 'var(--color-text-tertiary)' }}>+ Optional and off by default. Gestures never send messages or trigger tools.+ </p>+ </div>+ );+}diff --git a/frontend/src/pages/SettingsPage.tsx b/frontend/src/pages/SettingsPage.tsxindex 6b8bdc1d..db57947d 100644--- a/frontend/src/pages/SettingsPage.tsx+++ b/frontend/src/pages/SettingsPage.tsx@@ -17,8 +17,11 @@ import {Search,Brain,RefreshCw,+ Server,+ Wifi,} from 'lucide-react';import { useAppStore, type ThemeMode } from '../lib/store';+import { HandTrackingPanel } from '../components/HandTrackingPanel';import {checkHealth,fetchSpeechHealth,@@ -229,6 +232,36 @@ function SettingRow({ label, description, children }: { label: string; descripti);}+export function NetworkStatusRow() {+ const [online, setOnline] = useState<boolean | null>(+ () => typeof navigator === 'undefined' ? null : navigator.onLine,+ );++ useEffect(() => {+ const updateOnline = () => setOnline(navigator.onLine);+ window.addEventListener('online', updateOnline);+ window.addEventListener('offline', updateOnline);+ return () => {+ window.removeEventListener('online', updateOnline);+ window.removeEventListener('offline', updateOnline);+ };+ }, []);++ return (+ <SettingRow+ label="Browser network"+ description="Browser signal only; it does not verify internet or cloud-provider reachability."+ >+ <div className="flex items-center gap-2">+ <Wifi size={14} style={{ color: online ? 'var(--color-success)' : 'var(--color-text-tertiary)' }} />+ <span className="text-xs" style={{ color: 'var(--color-text-secondary)' }}>+ {online === null ? 'Unknown' : online ? 'Online' : 'Offline'}+ </span>+ </div>+ </SettingRow>+ );+}+type MemoryStatus =| { kind: 'loading' }| { kind: 'ready'; stats: MemoryStats }@@ -340,6 +373,7 @@ export function SettingsPage() {const [customEngine, setCustomEngine] = useState('lmstudio');const [customKey, setCustomKey] = useState('');const [srcMsg, setSrcMsg] = useState('');+ const inferenceConfigurable = isTauri();useEffect(() => {getInferenceSource().then((s) => {@@ -520,6 +554,7 @@ export function SettingsPage() {{/* Connection */}<Section title="Connection">+ <NetworkStatusRow /><SettingRow label="Server status" description={serverInfo ? `${serverInfo.engine} / ${serverInfo.model}` : 'Not connected'}><div className="flex items-center gap-2"><span@@ -564,10 +599,14 @@ export function SettingsPage() {{/* Inference source */}<Section title="Inference source">- <SettingRow label="Source" description="Where the app runs models. Applies after restart.">+ <p className="text-xs mb-2" style={{ color: 'var(--color-text-tertiary)' }}>+ Ollama is supported for local inference. The desktop app can connect to Colibri through its OpenAI-compatible API; OpenJarvis agents provide multi-step tool use and reasoning, while Colibri serves one model at a time. Maya AI is not wired in: its reviewed assistant script has no supported service or plugin API, so it cannot be safely managed here.+ </p>+ <SettingRow label="Source" description={inferenceConfigurable ? 'Where the app runs models. Applies after restart.' : 'The browser uses the backend’s configured model source; change inference sources in the desktop app.'}><selectvalue={srcKind}onChange={(e) => { setSrcKind(e.target.value as InferenceSource['kind']); setSrcMsg(''); }}+ disabled={!inferenceConfigurable}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>@@ -577,21 +616,39 @@ export function SettingsPage() {</SettingRow>{srcKind === 'custom' && (<>- <SettingRow label="Server URL" description="e.g. LM Studio: http://localhost:1234/v1">- <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1"+ <SettingRow label="Colibri preset" description="Sets the desktop app's LM Studio-compatible adapter target; configure Colibri separately and enter its exact model ID below.">+ <button+ type="button"+ onClick={() => {+ setSrcKind('custom');+ setCustomEngine('lmstudio');+ setCustomHost('http://127.0.0.1:8001');+ setCustomModel('');+ setSrcMsg('Colibri preset selected. Enter a model ID exposed by Colibri, then save and restart.');+ }}+ disabled={!inferenceConfigurable}+ title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}+ className="text-xs px-3 py-1.5 rounded-lg cursor-pointer"+ style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}+ >+ <span className="flex items-center gap-1.5"><Server size={13} /> Use Colibri preset</span>+ </button>+ </SettingRow>+ <SettingRow label="Server URL" description={customEngine === 'lmstudio' ? 'Base URL. For Colibri, configure loopback:8001 to avoid Jarvis’s default port 8000.' : 'OpenAI-compatible server base URL'}>+ <input type="text" value={customHost} onChange={(e) => { setCustomHost(e.target.value); setSrcMsg(''); }} placeholder="http://localhost:1234/v1" disabled={!inferenceConfigurable}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} /></SettingRow>- <SettingRow label="Model" description="Model id served by your endpoint">- <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct"+ <SettingRow label="Model" description={customEngine === 'lmstudio' ? 'Enter the exact model id from your server’s GET /v1/models response.' : 'Model id served by your endpoint'}>+ <input type="text" value={customModel} onChange={(e) => { setCustomModel(e.target.value); setSrcMsg(''); }} placeholder="qwen2.5-7b-instruct" disabled={!inferenceConfigurable}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} /></SettingRow>- <SettingRow label="Server type" description="OpenAI-compatible engine">- <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }}+ <SettingRow label="Server type" description={customEngine === 'lmstudio' ? 'LM Studio adapter; also compatible with Colibri’s OpenAI-style /v1 API.' : 'OpenAI-compatible inference backend'}>+ <select value={customEngine} onChange={(e) => { setCustomEngine(e.target.value); setSrcMsg(''); }} disabled={!inferenceConfigurable}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>- <option value="lmstudio">LM Studio</option>+ <option value="lmstudio">LM Studio / Colibri-compatible</option><option value="vllm">vLLM</option><option value="sglang">SGLang</option><option value="llamacpp">llama.cpp</option>@@ -599,7 +656,7 @@ export function SettingsPage() {</select></SettingRow><SettingRow label="API key (optional)" description="Only if your server requires one">- <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none"+ <input type="password" value={customKey} onChange={(e) => { setCustomKey(e.target.value); setSrcMsg(''); }} placeholder="leave blank if none" disabled={!inferenceConfigurable}className="text-sm px-3 py-1.5 rounded-lg outline-none w-56"style={{ background: 'var(--color-bg-secondary)', color: 'var(--color-text)', border: '1px solid var(--color-border)' }} /></SettingRow>@@ -607,8 +664,10 @@ export function SettingsPage() {)}<SettingRow label="" description={srcMsg}><button onClick={saveSource}+ disabled={!inferenceConfigurable}+ title={inferenceConfigurable ? undefined : 'Inference sources can only be changed in the desktop app.'}className="text-sm px-3 py-1.5 rounded-lg outline-none cursor-pointer"- style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)' }}>+ style={{ background: 'var(--color-accent, var(--color-bg-tertiary))', color: 'var(--color-text)', border: '1px solid var(--color-border)', opacity: inferenceConfigurable ? 1 : 0.5, cursor: inferenceConfigurable ? 'pointer' : 'not-allowed' }}>Save inference source</button></SettingRow>@@ -876,6 +935,14 @@ export function SettingsPage() {See the <a href="https://open-jarvis.github.io/OpenJarvis/user-guide/tools/" target="_blank" rel="noopener noreferrer" style={{ color: 'var(--color-accent)' }}>documentation</a> for details.</div>)}+ <p className="text-xs mt-3" style={{ color: 'var(--color-text-tertiary)' }}>+ Offline speech requires a configured local backend and downloaded model. Remote speech providers may send audio off-device.+ </p>+ </Section>++ {/* Hand tracking */}+ <Section title="Hand tracking">+ <HandTrackingPanel /></Section>{/* Data */}diff --git a/frontend/vite.config.ts b/frontend/vite.config.tsindex 360565cd..57ca3f31 100644--- a/frontend/vite.config.ts+++ b/frontend/vite.config.ts@@ -37,7 +37,8 @@ export default defineConfig({],},workbox: {- globPatterns: ['**/*.{js,css,html,ico,png,svg}'],+ globPatterns: ['**/*.{js,css,html,ico,png,svg,wasm,task}'],+ maximumFileSizeToCacheInBytes: 16 * 1024 * 1024,navigateFallbackDenylist: [/^\/v1\//, /^\/health/, /^\/dashboard/, /^\/api\//],},}),diff --git a/tests/engine/test_openai_compat_api_key.py b/tests/engine/test_openai_compat_api_key.pyindex dc541358..65a9d229 100644--- a/tests/engine/test_openai_compat_api_key.py+++ b/tests/engine/test_openai_compat_api_key.py@@ -2,12 +2,15 @@from __future__ import annotations+import json+import httpximport pytestimport respxfrom openjarvis.core.types import Message, Rolefrom openjarvis.engine.openai_compat_engines import (+ LMStudioEngine,OpenAICompatEngine,VLLMEngine,normalize_openai_base_url,@@ -79,6 +82,33 @@ class TestAuthorizationHeader:engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")assert engine._api_key == "sk-explicit"+ def test_colibri_compatible_lmstudio_engine_uses_model_and_bearer_key(self) -> None:+ model_id = "colibri/qwen3-8b"+ engine = LMStudioEngine(host="http://127.0.0.1:8001", api_key="coli-test")+ with respx.mock:+ route = respx.post("http://127.0.0.1:8001/v1/chat/completions").mock(+ return_value=httpx.Response(200, json=_CHAT_RESPONSE)+ )+ engine.generate([Message(role=Role.USER, content="hi")], model=model_id)+ request = route.calls.last.request+ assert request.headers["Authorization"] == "Bearer coli-test"+ assert request.url.path == "/v1/chat/completions"+ assert json.loads(request.content)["model"] == model_id++ def test_lmstudio_model_list_uses_single_v1_prefix_and_optional_auth(self) -> None:+ host = normalize_openai_base_url("http://127.0.0.1:8001/v1")+ engine = LMStudioEngine(host=host, api_key="coli-test")+ with respx.mock:+ route = respx.get("http://127.0.0.1:8001/v1/models").mock(+ return_value=httpx.Response(+ 200, json={"data": [{"id": "colibri/qwen3-8b"}]}+ )+ )+ assert engine.list_models() == ["colibri/qwen3-8b"]+ request = route.calls.last.request+ assert request.url.path == "/v1/models"+ assert request.headers["Authorization"] == "Bearer coli-test"+class TestNormalizeOpenAIBaseUrl:@pytest.mark.parametrize(