A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.
TypeScript
feat(voice): talk to the assistant and hear its replies
Alan Buscagliafd7c25f91ea7
diff --git a/apps/desktop/src-tauri/capabilities/default.json b/apps/desktop/src-tauri/capabilities/default.jsonindex 7c49a7f..d05779c 100644--- a/apps/desktop/src-tauri/capabilities/default.json+++ b/apps/desktop/src-tauri/capabilities/default.json@@ -29,12 +29,7 @@"identifier": "voice","description": "Voice input (S30.1): the panel's mic button records and transcribes natively; the Dot never records.","windows": ["panel"],- "permissions": [- "allow-voice-status",- "allow-voice-start",- "allow-voice-stop",- "allow-voice-cancel"- ]+ "permissions": ["allow-voice-status", "allow-voice-start", "allow-voice-stop", "allow-voice-cancel"]},{"identifier": "glow",diff --git a/docs/design.md b/docs/design.mdindex 036865f..b0d7d02 100644--- a/docs/design.md+++ b/docs/design.md@@ -179,6 +179,16 @@ The assistant can see the screen and operate apps through a built-in MCP serverWhy enforcement lives in the app: everything inside the engine, and the daemon's WebSocket, is within the agent's reach (it has a shell, and with the daemon's access key it could answer its own approval cards, L49). The session grant, panic stop, blocklist, rate limit, and risky-action confirmations therefore run in the helper, behind native dialogs the agent cannot answer, and the in-engine guard adds nothing for these tools. macOS also attributes Accessibility and Screen Recording to the app that holds them, Gentle Dot.app. The key in `mcp.json` only reaches the helper, which still asks the user before anything happens.+### Voice (S30)++The user can talk to the assistant: a mic button in the composer, and spoken replies.++- Where speech becomes text: "Sign in with ChatGPT" cannot transcribe (L88), so voice uses an OpenAI **API key** when one is connected, and the desktop app's own recognizer otherwise. On the desktop the key is preferred when present: the panel records with `MediaRecorder` and sends the audio to the daemon. Without a key it uses the Tauri commands `voice_status`, `voice_start {locale?}`, `voice_stop` → `{text}`, and `voice_cancel`, plus the events `voice://partial {text}`, `voice://level {level}`, and `voice://error {message}` (§9). The web only uses the daemon, and shows the mic only when a key is connected and the page is a secure context (HTTPS or localhost).+- Protocol: `voice_transcribe {requestId, mime, data}` (base64, at most `MAX_VOICE_AUDIO` = 960,000 characters, so the frame stays under the 1 MiB WebSocket limit) → `voice_transcript {requestId, text}`. `voice_speak {requestId, text ≤ 4096}` → `voice_speech {requestId, mime, data}`. A failure answers `voice_unavailable {requestId, reason}`, with the reason in plain words. `ready` and `auth_providers` carry `voice: {transcribe, speak}`, so voice turns on or off as soon as Accounts changes.+- Daemon (`packages/daemon/src/voice.ts`): it reads the key through Pi's ModelRuntime (`AuthRuntime.openAIApiKey`). It returns a key only when the stored `openai` credential is an API key (`checkAuth` type `api_key`), never the ChatGPT token. Transcription is `POST /v1/audio/transcriptions` (multipart, `gpt-4o-mini-transcribe`). Speech is `POST /v1/audio/speech` (`gpt-4o-mini-tts`, voice `alloy`, mp3). The key goes only into the `Authorization` header. Logs carry only the status code, and replies never carry the key or OpenAI's error bodies. 401, 403, 429, other statuses, and network failures each map to their own reason.+- Mic button (S30.4): tap to start and tap to send. While recording, the composer shows the elapsed time, a level meter, the live partial text (desktop recognizer), and ✕. Esc also cancels, and it is handled before the panel's own Esc. Recordings stop on their own at 60 s. The transcript is sent as the user's message and stays in the chat. When voice cannot work, the button is disabled and its tooltip says why (permission, insecure page, no recorder); on the web without a key it is hidden.+- Spoken replies (S30.3): only the reply to a spoken message is read aloud, once the assistant is idle and the reply did not end in an error or Stop. Markdown, code, and links are stripped first. With `voice.speak` the daemon synthesizes it and the UI plays the mp3. Otherwise `speechSynthesis` reads it with a voice in the reply's language (a Spanish/English guess, then the browser language). A speaker toggle in the header mutes it, and the choice is kept in `localStorage` (`gentle-dot-voice-muted`). Starting to talk, or Stop (shown while speaking), cuts the speech.+## 5. White-label layer- **Identity**: `--append-system-prompt` with `identity.md`: the assistant is "Gentle Dot", a personal assistant; it must not mention Gentle Shell, Pi, el Gentleman, ODD, or Engram by name, and describes them as "my workflow" and "my memory" when needed. It keeps every behavior.diff --git a/packages/daemon/src/auth.ts b/packages/daemon/src/auth.tsindex f953290..47c80b0 100644--- a/packages/daemon/src/auth.ts+++ b/packages/daemon/src/auth.ts@@ -45,6 +45,11 @@ export interface AuthRuntime {options?: { getDeviceId?: () => string },): Promise<unknown>;logout(providerId: string): Promise<void>;+ /**+ * The connected OpenAI API key, for voice (S30.2). Never the "Sign in with ChatGPT" token:+ * OpenAI documents that sign-in does not support transcription.+ */+ openAIApiKey?(): Promise<string | undefined>;}/** The assistant's own engine home: never the user's Gentle Shell or Pi home unless explicitly overridden. */@@ -71,6 +76,11 @@ export async function createModelAuthRuntime(agentHome: string, cwd: string): PrgetDeviceId: () => settings.getOrCreateDeviceId(),}),logout: (providerId) => runtime.logout(providerId),+ openAIApiKey: async () => {+ // Checked first: resolving a sign-in could refresh its token over the network.+ if ((await runtime.checkAuth("openai"))?.type !== "api_key") return undefined;+ return (await runtime.getAuth("openai"))?.auth.apiKey || undefined;+ },};}@@ -191,6 +201,12 @@ export class AuthManager {return undefined;}+ /** The connected OpenAI API key, read fresh; only for the daemon's own requests to OpenAI. */+ async openAIApiKey(): Promise<string | undefined> {+ const runtime = await this.options.runtime();+ return runtime.openAIApiKey?.();+ }+/** True when at least one provider has a credential. */async hasAccount(): Promise<boolean> {const runtime = await this.options.runtime();diff --git a/packages/daemon/src/bridge.ts b/packages/daemon/src/bridge.tsindex a91cecb..599589e 100644--- a/packages/daemon/src/bridge.ts+++ b/packages/daemon/src/bridge.ts@@ -9,6 +9,7 @@ import {parseQueue,type RoleRoute,type ServerPayload,+ type VoiceCapability,} from "@gentle-dot/protocol";import type { AuthManager } from "./auth.ts";import type { ConnectorManager, ConnectorRefusal } from "./connectors.ts";@@ -30,6 +31,7 @@ import {} from "./profiles.ts";import { DEFAULT_ROTATION, type RotationLimits, SessionRotator } from "./rotation.ts";import type { AgentRecord, AgentSupervisor, SupervisorEvent } from "./supervisor.ts";+import type { VoiceService } from "./voice.ts";import { isBlockedInput, presentText, shouldShowToast } from "./white-label.ts";export interface BridgeClient {@@ -42,6 +44,8 @@ export interface BridgeOptions {auth?: AuthManager;profiles?: ProfileStore;connectors?: ConnectorManager;+ /** Speech to text and text to speech with an OpenAI API key (S30.2). */+ voice?: VoiceService;/** How long switching conversations waits for a running answer to stop. Default 10 s. */stopTimeoutMs?: number;/** Optional parts of the app; by default one continuous chat. */@@ -143,6 +147,7 @@ export class DotBridge {...this.conversationRef(),...(this.supervisor.model ? { model: this.supervisor.model } : {}),features: this.features,+ ...(this.options.voice ? { voice: this.options.voice.capability() } : {}),});if (this.queued()) this.deliver(client, { type: "queue", ...this.queue });for (const { ask } of this.asks.values()) this.deliver(client, { type: "ask", ask });@@ -322,6 +327,37 @@ export class DotBridge {});}return;+ case "voice_transcribe":+ case "voice_speak": {+ const { requestId } = message;+ const voice = this.options.voice;+ if (!voice) {+ this.deliver(client, {+ type: "voice_unavailable",+ requestId,+ reason: "Voice is not available here.",+ });+ return;+ }+ if (message.type === "voice_transcribe") {+ const result = await voice.transcribe(message.mime, message.data);+ this.deliver(+ client,+ result.ok+ ? { type: "voice_transcript", requestId, text: result.text }+ : { type: "voice_unavailable", requestId, reason: result.reason },+ );+ return;+ }+ const result = await voice.speak(message.text);+ this.deliver(+ client,+ result.ok+ ? { type: "voice_speech", requestId, mime: result.mime, data: result.data }+ : { type: "voice_unavailable", requestId, reason: result.reason },+ );+ return;+ }case "auth_logout": {const refused = await this.requireAuth().logout(message.providerId);if (refused) this.deliver(client, { type: "error", ...refused });@@ -474,7 +510,18 @@ export class DotBridge {private async sendProviders(client: BridgeClient, open = false): Promise<void> {const providers = await this.requireAuth().providers();- this.deliver(client, { type: "auth_providers", providers, ...(open ? { open: true } : {}) });+ this.deliver(client, {+ type: "auth_providers",+ providers,+ ...(open ? { open: true } : {}),+ ...(await this.voiceCapability()),+ });+ }++ /** The voice capability, checked now, for the accounts list. */+ private async voiceCapability(): Promise<{ voice?: VoiceCapability }> {+ const voice = this.options.voice;+ return voice ? { voice: await voice.refresh() } : {};}private requireConnectors(): ConnectorManager {@@ -640,7 +687,9 @@ export class DotBridge {private afterCredentialsChange(): void {void this.requireAuth().providers()- .then((providers) => this.broadcast({ type: "auth_providers", providers }))+ .then(async (providers) =>+ this.broadcast({ type: "auth_providers", providers, ...(await this.voiceCapability()) }),+ ).catch((error: Error) => this.log(`could not list providers: ${error.message}`));if (this.supervisor.busy) {this.restartPending = true;diff --git a/packages/daemon/src/daemon.ts b/packages/daemon/src/daemon.tsindex f5d60da..2ae1bdd 100644--- a/packages/daemon/src/daemon.ts+++ b/packages/daemon/src/daemon.ts@@ -19,6 +19,7 @@ import { ensureMemoryProject, ensurePrivateDir, isolatedAgentEnv, privateMemoryimport { defaultImportPath, ProfileStore } from "./profiles.ts";import { type RotationLimits, rotationLimits } from "./rotation.ts";import { AgentSupervisor } from "./supervisor.ts";+import { VoiceService } from "./voice.ts";import { identityArgs } from "./white-label.ts";export interface DaemonOptions {@@ -41,6 +42,8 @@ export interface DaemonOptions {profilesImportPath?: string;/** Creates the sign-in runtime; defaults to Pi's ModelRuntime on the agent home. */authRuntime?: () => Promise<AuthRuntime>;+ /** The fetch for voice requests to OpenAI; tests pass a fake one. */+ voiceFetch?: typeof fetch;/** The engine's command line for connector sign-in; default `GENTLE_DOT_MCP_CLI` (JSON array) or the bundled one. */connectorCli?: McpCli;/** The home folder "Import my MCP servers" reads other apps' configs from; default `GENTLE_DOT_IMPORT_HOME` or the user's. */@@ -177,6 +180,11 @@ export async function startDaemon(options: DaemonOptions): Promise<DotDaemon> {: {}),log,});+ const voice = new VoiceService({+ apiKey: () => auth.openAIApiKey(),+ ...(options.voiceFetch ? { fetch: options.voiceFetch } : {}),+ log,+ });const historyPage = options.historyPage ?? Number(process.env.GENTLE_DOT_HISTORY_PAGE);const bridge = new DotBridge(supervisor, {dataDir: options.dataDir,@@ -184,6 +192,7 @@ export async function startDaemon(options: DaemonOptions): Promise<DotDaemon> {auth,profiles,connectors,+ voice,features: { conversations: options.conversations ?? process.env.GENTLE_DOT_CONVERSATIONS === "1" },rotation: options.rotation ?? rotationLimits(process.env),...(Number.isInteger(historyPage) && historyPage > 0 ? { historyPage } : {}),diff --git a/packages/daemon/src/voice.ts b/packages/daemon/src/voice.tsnew file mode 100644index 0000000..54c4ffb--- /dev/null+++ b/packages/daemon/src/voice.ts@@ -0,0 +1,161 @@+import { MAX_VOICE_AUDIO, MAX_VOICE_TEXT, type VoiceCapability } from "@gentle-dot/protocol";++export const TRANSCRIBE_URL = "https://api.openai.com/v1/audio/transcriptions";+export const SPEECH_URL = "https://api.openai.com/v1/audio/speech";+const TRANSCRIBE_MODEL = "gpt-4o-mini-transcribe";+const SPEECH_MODEL = "gpt-4o-mini-tts";+const SPEECH_VOICE = "alloy";+const TIMEOUT_MS = 60_000;++export type VoiceResult<T> = ({ ok: true } & T) | { ok: false; reason: string };++export interface VoiceServiceOptions {+ /** The connected OpenAI API key, read on each request; never a ChatGPT sign-in token. */+ apiKey: () => Promise<string | undefined>;+ fetch?: typeof fetch;+ log?: (line: string) => void;+ timeoutMs?: number;+}++const REASONS = {+ noKey: "Voice through the assistant needs an OpenAI API key. Connect one in Accounts.",+ tooLong: "That recording is too long. Keep it under a minute.",+ empty: "I didn't catch anything. Try again.",+ noText: "There is nothing to read aloud.",+ unauthorized: "OpenAI did not accept the API key. Check it in Accounts.",+ forbidden: "This OpenAI key is not allowed to use voice. Check its project and permissions.",+ limited: "OpenAI is limiting requests or the account is out of credit. Try again later.",+ failed: "OpenAI could not handle the voice request. Try again.",+ offline: "Could not reach OpenAI. Check the internet connection.",+} as const;++/** File extensions OpenAI recognizes, by audio type. */+const EXTENSIONS: Record<string, string> = {+ "audio/webm": "webm",+ "audio/ogg": "ogg",+ "audio/mp4": "mp4",+ "audio/m4a": "m4a",+ "audio/x-m4a": "m4a",+ "audio/aac": "m4a",+ "audio/mpeg": "mp3",+ "audio/mp3": "mp3",+ "audio/wav": "wav",+ "audio/x-wav": "wav",+ "audio/flac": "flac",+};++/**+ * Speech to text and text to speech with the user's OpenAI API key (S30.2).+ * The key stays here: it goes only into the `Authorization` header of requests+ * to OpenAI, and neither logs nor replies carry it or OpenAI's error bodies.+ */+export class VoiceService {+ private current: VoiceCapability = { transcribe: false, speak: false };+ private readonly options: VoiceServiceOptions;++ constructor(options: VoiceServiceOptions) {+ this.options = options;+ }++ /** The capability as last checked. */+ capability(): VoiceCapability {+ return this.current;+ }++ /** Checks whether a key is connected now. */+ async refresh(): Promise<VoiceCapability> {+ const on = (await this.key()) !== undefined;+ this.current = { transcribe: on, speak: on };+ return this.current;+ }++ async transcribe(mime: string, data: string): Promise<VoiceResult<{ text: string }>> {+ if (data.length > MAX_VOICE_AUDIO) return { ok: false, reason: REASONS.tooLong };+ const type = mime.split(";")[0]?.trim().toLowerCase() ?? "";+ const form = new FormData();+ form.append("model", TRANSCRIBE_MODEL);+ form.append("response_format", "json");+ form.append(+ "file",+ new File([Buffer.from(data, "base64")], `recording.${EXTENSIONS[type] ?? "webm"}`, { type }),+ );+ const response = await this.post("transcription", TRANSCRIBE_URL, form);+ if (!response.ok) return response;+ try {+ const { text } = (await response.value.json()) as { text?: unknown };+ const transcript = typeof text === "string" ? text.trim() : "";+ return transcript ? { ok: true, text: transcript } : { ok: false, reason: REASONS.empty };+ } catch {+ this.log("transcription answer was not readable");+ return { ok: false, reason: REASONS.failed };+ }+ }++ async speak(text: string): Promise<VoiceResult<{ mime: string; data: string }>> {+ if (text.trim() === "" || text.length > MAX_VOICE_TEXT) return { ok: false, reason: REASONS.noText };+ const body = JSON.stringify({+ model: SPEECH_MODEL,+ voice: SPEECH_VOICE,+ input: text,+ response_format: "mp3",+ });+ const response = await this.post("speech", SPEECH_URL, body, "application/json");+ if (!response.ok) return response;+ try {+ const audio = Buffer.from(await response.value.arrayBuffer());+ return { ok: true, mime: "audio/mpeg", data: audio.toString("base64") };+ } catch {+ this.log("speech answer was not readable");+ return { ok: false, reason: REASONS.failed };+ }+ }++ private async post(+ what: string,+ url: string,+ body: FormData | string,+ contentType?: string,+ ): Promise<VoiceResult<{ value: Response }>> {+ const key = await this.key();+ if (!key) return { ok: false, reason: REASONS.noKey };+ const headers: Record<string, string> = { Authorization: `Bearer ${key}` };+ if (contentType) headers["Content-Type"] = contentType;+ let response: Response;+ try {+ response = await (this.options.fetch ?? fetch)(url, {+ method: "POST",+ headers,+ body,+ signal: AbortSignal.timeout(this.options.timeoutMs ?? TIMEOUT_MS),+ });+ } catch {+ // The error may quote the request; only the fact that it failed is logged.+ this.log(`${what} failed: OpenAI not reachable`);+ return { ok: false, reason: REASONS.offline };+ }+ if (response.ok) return { ok: true, value: response };+ this.log(`${what} failed: OpenAI answered ${response.status}`);+ await response.body?.cancel().catch(() => {});+ return { ok: false, reason: reasonFor(response.status) };+ }++ private async key(): Promise<string | undefined> {+ try {+ return (await this.options.apiKey()) || undefined;+ } catch {+ this.log("could not read the OpenAI key");+ return undefined;+ }+ }++ private log(line: string): void {+ this.options.log?.(`voice: ${line}`);+ }+}++function reasonFor(status: number): string {+ if (status === 401) return REASONS.unauthorized;+ if (status === 403) return REASONS.forbidden;+ if (status === 429) return REASONS.limited;+ return REASONS.failed;+}diff --git a/packages/daemon/test/voice.test.ts b/packages/daemon/test/voice.test.tsnew file mode 100644index 0000000..3be002e--- /dev/null+++ b/packages/daemon/test/voice.test.ts@@ -0,0 +1,298 @@+import { writeFileSync } from "node:fs";+import { join } from "node:path";+import { MAX_VOICE_AUDIO, MAX_VOICE_TEXT, type ServerMessage } from "@gentle-dot/protocol";+import { afterEach, describe, expect, it } from "vitest";+import WebSocket from "ws";+import { createModelAuthRuntime } from "../src/auth.ts";+import { type DotDaemon, startDaemon } from "../src/daemon.ts";+import { SPEECH_URL, TRANSCRIBE_URL, VoiceService } from "../src/voice.ts";+import { fakeAuthRuntime } from "./fake-auth-runtime.ts";+import { FAKE_AGENT, tempDir, waitFor } from "./helpers.ts";++const KEY = "sk-voice-test-key-not-real-1234";+const AUDIO = Buffer.from("fake opus bytes").toString("base64");++type Call = { url: string; init: RequestInit };++/** A fetch that records each call and answers with `reply`. */+function fakeFetch(reply: (call: Call) => Response | Promise<Response>) {+ const calls: Call[] = [];+ const fetch = async (url: string | URL | Request, init: RequestInit = {}) => {+ const call = { url: String(url), init };+ calls.push(call);+ return reply(call);+ };+ return { fetch: fetch as typeof globalThis.fetch, calls };+}++/** `key: null` means no key is connected. */+function service(reply: (call: Call) => Response | Promise<Response>, key: string | null = KEY) {+ const logs: string[] = [];+ const fake = fakeFetch(reply);+ const voice = new VoiceService({+ apiKey: async () => key ?? undefined,+ fetch: fake.fetch,+ log: (line) => logs.push(line),+ });+ return { voice, logs, calls: fake.calls };+}++const header = (call: Call | undefined, name: string) => new Headers(call?.init.headers).get(name);++describe("VoiceService capability", () => {+ it("is off until an OpenAI API key is found, and on for both directions once it is", async () => {+ const { voice } = service(() => new Response());+ expect(voice.capability()).toEqual({ transcribe: false, speak: false });+ expect(await voice.refresh()).toEqual({ transcribe: true, speak: true });+ expect(voice.capability()).toEqual({ transcribe: true, speak: true });+ });++ it("stays off without a key, and when reading the key fails", async () => {+ expect(await service(() => new Response(), null).voice.refresh()).toEqual({+ transcribe: false,+ speak: false,+ });+ const failing = new VoiceService({+ apiKey: async () => {+ throw new Error("locked");+ },+ fetch: fakeFetch(() => new Response()).fetch,+ });+ expect(await failing.refresh()).toEqual({ transcribe: false, speak: false });+ });+});++describe("VoiceService.transcribe", () => {+ it("posts the recording as multipart to OpenAI with the transcribe model and the key in the header", async () => {+ const { voice, calls } = service(() => Response.json({ text: "hello from voice" }));+ expect(await voice.transcribe("audio/webm;codecs=opus", AUDIO)).toEqual({+ ok: true,+ text: "hello from voice",+ });+ const call = calls[0];+ expect(call?.url).toBe(TRANSCRIBE_URL);+ expect(TRANSCRIBE_URL).toBe("https://api.openai.com/v1/audio/transcriptions");+ expect(call?.init.method).toBe("POST");+ expect(header(call, "authorization")).toBe(`Bearer ${KEY}`);+ const body = call?.init.body as FormData;+ expect(body).toBeInstanceOf(FormData);+ expect(body.get("model")).toBe("gpt-4o-mini-transcribe");+ expect(body.get("response_format")).toBe("json");+ const file = body.get("file") as File;+ expect(file.name).toBe("recording.webm");+ expect(file.type).toBe("audio/webm");+ expect(Buffer.from(await file.arrayBuffer()).toString()).toBe("fake opus bytes");+ });++ it("names the file after the audio type so OpenAI can read it", async () => {+ const { voice, calls } = service(() => Response.json({ text: "ok" }));+ await voice.transcribe("audio/mp4", AUDIO);+ const body = calls[0]?.init.body as FormData;+ expect((body.get("file") as File).name).toBe("recording.mp4");+ });++ it("says nothing was heard when the transcript is empty", async () => {+ const { voice } = service(() => Response.json({ text: " " }));+ expect(await voice.transcribe("audio/webm", AUDIO)).toEqual({+ ok: false,+ reason: "I didn't catch anything. Try again.",+ });+ });++ it("refuses a recording over the size cap without calling OpenAI", async () => {+ const { voice, calls } = service(() => Response.json({ text: "never" }));+ const result = await voice.transcribe("audio/webm", "A".repeat(MAX_VOICE_AUDIO + 4));+ expect(result).toEqual({ ok: false, reason: "That recording is too long. Keep it under a minute." });+ expect(calls).toHaveLength(0);+ });++ it("explains how to turn voice on when no API key is connected, without calling OpenAI", async () => {+ const { voice, calls } = service(() => Response.json({ text: "never" }), null);+ expect(await voice.transcribe("audio/webm", AUDIO)).toEqual({+ ok: false,+ reason: "Voice through the assistant needs an OpenAI API key. Connect one in Accounts.",+ });+ expect(calls).toHaveLength(0);+ });+});++describe("VoiceService.speak", () => {+ it("asks OpenAI for mp3 speech with the speech model and a default voice, and returns it in base64", async () => {+ const mp3 = Buffer.from("ID3 fake mp3");+ const { voice, calls } = service(() => new Response(mp3, { headers: { "content-type": "audio/mpeg" } }));+ expect(await voice.speak("Hi! Here is the answer.")).toEqual({+ ok: true,+ mime: "audio/mpeg",+ data: mp3.toString("base64"),+ });+ const call = calls[0];+ expect(call?.url).toBe(SPEECH_URL);+ expect(SPEECH_URL).toBe("https://api.openai.com/v1/audio/speech");+ expect(call?.init.method).toBe("POST");+ expect(header(call, "authorization")).toBe(`Bearer ${KEY}`);+ expect(header(call, "content-type")).toBe("application/json");+ expect(JSON.parse(String(call?.init.body))).toEqual({+ model: "gpt-4o-mini-tts",+ voice: "alloy",+ input: "Hi! Here is the answer.",+ response_format: "mp3",+ });+ });++ it("refuses empty text and text over the speech limit without calling OpenAI", async () => {+ const { voice, calls } = service(() => new Response("x"));+ expect((await voice.speak(" ")).ok).toBe(false);+ expect((await voice.speak("a".repeat(MAX_VOICE_TEXT + 1))).ok).toBe(false);+ expect(calls).toHaveLength(0);+ });+});++describe("VoiceService errors", () => {+ const cases: [string, () => Response | Promise<Response>, string][] = [+ [+ "401",+ () => new Response(`{"error":{"message":"Incorrect API key provided: ${KEY}"}}`, { status: 401 }),+ "OpenAI did not accept the API key. Check it in Accounts.",+ ],+ [+ "403",+ () => new Response(`{"error":"forbidden for ${KEY}"}`, { status: 403 }),+ "This OpenAI key is not allowed to use voice. Check its project and permissions.",+ ],+ [+ "429",+ () => new Response(`{"error":"quota for ${KEY}"}`, { status: 429 }),+ "OpenAI is limiting requests or the account is out of credit. Try again later.",+ ],+ [+ "500",+ () => new Response(`oops ${KEY}`, { status: 500 }),+ "OpenAI could not handle the voice request. Try again.",+ ],+ [+ "network",+ () => Promise.reject(new TypeError(`fetch failed for ${KEY}`)),+ "Could not reach OpenAI. Check the internet connection.",+ ],+ ];++ for (const [name, reply, reason] of cases) {+ it(`maps ${name} to a readable reason and never shows or logs the key`, async () => {+ const { voice, logs } = service(reply);+ const results = [await voice.transcribe("audio/webm", AUDIO), await voice.speak("hello")];+ for (const result of results) expect(result).toEqual({ ok: false, reason });+ expect(JSON.stringify(results)).not.toContain(KEY);+ expect(logs.length).toBeGreaterThan(0);+ expect(logs.join("\n")).not.toContain(KEY);+ });+ }+});++const daemons: DotDaemon[] = [];+afterEach(async () => {+ await Promise.all(daemons.splice(0).map((d) => d.close()));+});++async function startWith(apiKey: string | undefined, reply: (call: Call) => Response | Promise<Response>) {+ const auth = fakeAuthRuntime();+ auth.runtime.openAIApiKey = async () => apiKey;+ const fake = fakeFetch(reply);+ const logs: string[] = [];+ const dataDir = tempDir();+ const d = await startDaemon({+ port: 0,+ host: "127.0.0.1",+ dataDir,+ workspace: dataDir,+ uiDir: dataDir,+ agentCommand: process.execPath,+ agentArgs: [FAKE_AGENT],+ backoffMs: [50],+ authRuntime: async () => auth.runtime,+ voiceFetch: fake.fetch,+ log: (line) => logs.push(line),+ });+ daemons.push(d);+ const ws = new WebSocket(`ws://127.0.0.1:${d.port}/ws`);+ const messages: ServerMessage[] = [];+ ws.on("message", (data) => messages.push(JSON.parse(String(data)) as ServerMessage));+ await new Promise((resolve) => ws.on("open", resolve));+ ws.send(JSON.stringify({ type: "hello", token: d.token, protocol: 1 }));+ const send = (m: object) => ws.send(JSON.stringify(m));+ const find = <T extends ServerMessage["type"]>(type: T) =>+ waitFor(() => messages.find((m): m is Extract<ServerMessage, { type: T }> => m.type === type));+ return { messages, send, find, logs, calls: fake.calls };+}++describe("voice through the daemon", () => {+ it("reports voice in the accounts list and transcribes a recording with the key kept in the daemon", async () => {+ const { send, find, messages, logs, calls } = await startWith(KEY, () =>+ Response.json({ text: "what's on my calendar" }),+ );+ await find("ready");+ send({ type: "auth_list" });+ expect((await find("auth_providers")).voice).toEqual({ transcribe: true, speak: true });+ send({ type: "voice_transcribe", requestId: "v1", mime: "audio/webm", data: AUDIO });+ expect(await find("voice_transcript")).toMatchObject({ requestId: "v1", text: "what's on my calendar" });+ expect(header(calls[0], "authorization")).toBe(`Bearer ${KEY}`);+ expect(JSON.stringify(messages)).not.toContain(KEY);+ expect(logs.join("\n")).not.toContain(KEY);+ });++ it("speaks a reply and returns the audio to the window that asked", async () => {+ const { send, find } = await startWith(KEY, () => new Response(Buffer.from("mp3")));+ await find("ready");+ send({ type: "voice_speak", requestId: "s1", text: "Done." });+ expect(await find("voice_speech")).toMatchObject({+ requestId: "s1",+ mime: "audio/mpeg",+ data: Buffer.from("mp3").toString("base64"),+ });+ });++ it("has voice off with only a ChatGPT sign-in, and says why a request did not work", async () => {+ const { send, find, calls } = await startWith(undefined, () => Response.json({ text: "never" }));+ expect((await find("ready")).voice).toEqual({ transcribe: false, speak: false });+ send({ type: "voice_transcribe", requestId: "v2", mime: "audio/webm", data: AUDIO });+ expect(await find("voice_unavailable")).toMatchObject({+ requestId: "v2",+ reason: "Voice through the assistant needs an OpenAI API key. Connect one in Accounts.",+ });+ expect(calls).toHaveLength(0);+ });+});++describe("createModelAuthRuntime openAIApiKey (offline)", () => {+ const withoutEnvKey = async (run: () => Promise<void>) => {+ const saved = process.env.OPENAI_API_KEY;+ delete process.env.OPENAI_API_KEY;+ try {+ await run();+ } finally {+ if (saved !== undefined) process.env.OPENAI_API_KEY = saved;+ }+ };++ it("returns a stored OpenAI API key", () =>+ withoutEnvKey(async () => {+ const home = tempDir();+ const runtime = await createModelAuthRuntime(home, home);+ await runtime.login("openai", "api_key", { prompt: async () => KEY, notify: () => {} });+ expect(await (await createModelAuthRuntime(home, home)).openAIApiKey?.()).toBe(KEY);+ }));++ it("never returns the ChatGPT sign-in token, which cannot transcribe", () =>+ withoutEnvKey(async () => {+ const home = tempDir();+ const oauth = {+ type: "oauth",+ access: "chatgpt-access-token",+ refresh: "r",+ expires: Date.now() + 3.6e6,+ };+ writeFileSync(join(home, "auth.json"), JSON.stringify({ openai: oauth }), { mode: 0o600 });+ const runtime = await createModelAuthRuntime(home, home);+ expect(runtime.getProviderAuthStatus("openai").configured).toBe(true);+ expect(await runtime.openAIApiKey?.()).toBeUndefined();+ }));+});diff --git a/packages/protocol/src/index.ts b/packages/protocol/src/index.tsindex 92f268b..a944980 100644--- a/packages/protocol/src/index.ts+++ b/packages/protocol/src/index.ts@@ -44,6 +44,22 @@ export interface Features {conversations: boolean;}+/** What talking to the assistant can use through the daemon (an OpenAI API key, S30.2). */+export interface VoiceCapability {+ /** The daemon can turn a recording into text. */+ transcribe: boolean;+ /** The daemon can read a reply aloud. */+ speak: boolean;+}++/**+ * Longest base64 recording in `voice_transcribe`: with the rest of the frame it stays under the+ * daemon's 1 MiB WebSocket limit (about 700 KB of audio, over a minute of compressed speech).+ */+export const MAX_VOICE_AUDIO = 960_000;+/** Longest text `voice_speak` reads aloud (OpenAI's speech input limit). */+export const MAX_VOICE_TEXT = 4096;+export interface ConversationSummary {id: string;title: string;@@ -279,10 +295,20 @@ export type ClientMessage =| { type: "connector_import"; ids: string[] }/** The desktop app's computer-control helper (macOS), at `http://127.0.0.1:<port>/mcp` with its key. */| { type: "computer_register"; url: string; token: string }- | { type: "computer_unregister" };+ | { type: "computer_unregister" }+ /** A recording to turn into text: its audio type and the bytes in base64. */+ | { type: "voice_transcribe"; requestId: string; mime: string; data: string }+ | { type: "voice_speak"; requestId: string; text: string };export type ServerPayload =- | { type: "ready"; agentState: AgentState; conversationId?: string; model?: string; features?: Features }+ | {+ type: "ready";+ agentState: AgentState;+ conversationId?: string;+ model?: string;+ features?: Features;+ voice?: VoiceCapability;+ }| { type: "agent_state"; state: AgentState }| { type: "user_message"; messageId: string; text: string }| { type: "message_delta"; messageId: string; delta: string }@@ -311,7 +337,8 @@ export type ServerPayload =| { type: "interrupted" }| ({ type: "queue" } & MessageQueue)| { type: "error"; code: string; message: string }- | { type: "auth_providers"; providers: AuthProvider[]; open?: boolean }+ /** `voice` changes with the accounts: an OpenAI API key turns it on. */+ | { type: "auth_providers"; providers: AuthProvider[]; open?: boolean; voice?: VoiceCapability }| { type: "auth_prompt"; prompt: AuthPrompt }| { type: "auth_event"; flowId: string; event: AuthEvent }| { type: "auth_done"; flowId: string; providerId: string; ok: boolean; message?: string }@@ -332,7 +359,12 @@ export type ServerPayload =| { type: "connector_draft"; draft: ConnectorDraft }| { type: "connector_draft_resolved"; draftId: string; approved: boolean }| { type: "connector_imports"; found: ImportCandidate[] }- | { type: "connector_imported"; names: string[] };+ | { type: "connector_imported"; names: string[] }+ | { type: "voice_transcript"; requestId: string; text: string }+ /** Speech for a `voice_speak` request, in base64. */+ | { type: "voice_speech"; requestId: string; mime: string; data: string }+ /** A voice request did not work; `reason` is plain words for the user. */+ | { type: "voice_unavailable"; requestId: string; reason: string };/** Every daemon message carries a per-connection, monotonic `seq`. */export type ServerMessage = ServerPayload & { seq: number };@@ -345,6 +377,11 @@ const isConnectorId = (value: unknown): value is string => isString(value) && COconst isShortString = (value: unknown): value is string => isString(value) && value.length <= 200;const MAX_IMPORTS = 200;const COMPUTER_URL = /^http:\/\/127\.0\.0\.1:([1-9][0-9]{0,4})\/mcp$/;+const VOICE_MIME =+ /^audio\/(webm|ogg|mp4|mpeg|mp3|wav|x-wav|m4a|x-m4a|aac|flac)(;\s*codecs="?[A-Za-z0-9.,-]+"?)?$/;+const BASE64 = /^[A-Za-z0-9+/]+={0,2}$/;+const isAudio = (value: unknown): value is string =>+ isString(value) && value.length > 0 && value.length <= MAX_VOICE_AUDIO && BASE64.test(value);/** The helper's key goes into an `Authorization` header, so only header-safe characters. */const COMPUTER_TOKEN = /^[A-Za-z0-9._~+/=-]{16,512}$/;@@ -460,6 +497,17 @@ export function parseClientMessage(raw: string): ClientMessage | undefined {return isComputerUrl(m.url) && isString(m.token) && COMPUTER_TOKEN.test(m.token)? { type: "computer_register", url: m.url, token: m.token }: undefined;+ case "voice_transcribe":+ return isShortString(m.requestId) && isString(m.mime) && VOICE_MIME.test(m.mime) && isAudio(m.data)+ ? { type: "voice_transcribe", requestId: m.requestId, mime: m.mime, data: m.data }+ : undefined;+ case "voice_speak":+ return isShortString(m.requestId) &&+ isString(m.text) &&+ m.text.trim() !== "" &&+ m.text.length <= MAX_VOICE_TEXT+ ? { type: "voice_speak", requestId: m.requestId, text: m.text }+ : undefined;case "computer_unregister":case "connectors_list":case "connectors_scan":diff --git a/packages/protocol/test/voice.test.ts b/packages/protocol/test/voice.test.tsnew file mode 100644index 0000000..5f7bd08--- /dev/null+++ b/packages/protocol/test/voice.test.ts@@ -0,0 +1,59 @@+import { describe, expect, it } from "vitest";+import { MAX_VOICE_AUDIO, MAX_VOICE_TEXT, parseClientMessage } from "../src/index.ts";++const parse = (value: object) => parseClientMessage(JSON.stringify(value));+const AUDIO = Buffer.from("fake opus bytes").toString("base64");++describe("voice messages", () => {+ it("accepts a recording to transcribe: request id, audio type, and base64 data", () => {+ expect(+ parse({ type: "voice_transcribe", requestId: "v1", mime: "audio/webm;codecs=opus", data: AUDIO }),+ ).toEqual({ type: "voice_transcribe", requestId: "v1", mime: "audio/webm;codecs=opus", data: AUDIO });+ expect(+ parse({ type: "voice_transcribe", requestId: "v2", mime: "audio/mp4", data: AUDIO, extra: 1 }),+ ).toEqual({+ type: "voice_transcribe",+ requestId: "v2",+ mime: "audio/mp4",+ data: AUDIO,+ });+ });++ it("refuses recordings that are not audio, not base64, empty, or missing a request id", () => {+ expect(+ parse({ type: "voice_transcribe", requestId: "v", mime: "text/html", data: AUDIO }),+ ).toBeUndefined();+ expect(+ parse({ type: "voice_transcribe", requestId: "v", mime: "audio/webm", data: "not base64!" }),+ ).toBeUndefined();+ expect(parse({ type: "voice_transcribe", requestId: "v", mime: "audio/webm", data: "" })).toBeUndefined();+ expect(parse({ type: "voice_transcribe", mime: "audio/webm", data: AUDIO })).toBeUndefined();+ expect(+ parse({ type: "voice_transcribe", requestId: "x".repeat(201), mime: "audio/webm", data: AUDIO }),+ ).toBeUndefined();+ });++ it("caps the recording so the whole frame stays under the 1 MiB WebSocket limit", () => {+ expect(MAX_VOICE_AUDIO).toBeLessThan(1024 * 1024 - 1024);+ const atCap = "A".repeat(MAX_VOICE_AUDIO);+ expect(parse({ type: "voice_transcribe", requestId: "v", mime: "audio/webm", data: atCap })?.type).toBe(+ "voice_transcribe",+ );+ expect(+ parse({ type: "voice_transcribe", requestId: "v", mime: "audio/webm", data: `${atCap}AAAA` }),+ ).toBeUndefined();+ });++ it("accepts text to speak, up to the speech limit, and refuses empty or longer text", () => {+ expect(parse({ type: "voice_speak", requestId: "s1", text: "Hello there" })).toEqual({+ type: "voice_speak",+ requestId: "s1",+ text: "Hello there",+ });+ expect(parse({ type: "voice_speak", requestId: "s1", text: " " })).toBeUndefined();+ expect(+ parse({ type: "voice_speak", requestId: "s1", text: "a".repeat(MAX_VOICE_TEXT + 1) }),+ ).toBeUndefined();+ expect(parse({ type: "voice_speak", text: "hi" })).toBeUndefined();+ });+});diff --git a/packages/ui/src/components/ChatSurface.tsx b/packages/ui/src/components/ChatSurface.tsxindex 876df0e..55ba286 100644--- a/packages/ui/src/components/ChatSurface.tsx+++ b/packages/ui/src/components/ChatSurface.tsx@@ -3,6 +3,7 @@ import { useState } from "react";import { openUrl as openExternal } from "../desktop.ts";import { activeTitle, type DotAction, type DotState, isBusy, needsAccount } from "../store.ts";import type { PanelWindowControls } from "../usePanelWindow.ts";+import { useVoice, type VoiceControls } from "../useVoice.ts";import { AccountsPanel } from "./AccountsPanel.tsx";import { AskCard } from "./AskCard.tsx";import { Composer } from "./Composer.tsx";@@ -21,6 +22,8 @@ import {PlusIcon,ProfilesIcon,RoseGlyph,+ SpeakerIcon,+ SpeakerOffIcon,} from "./icons.tsx";import { MessageList } from "./MessageList.tsx";import { ProfilesPanel } from "./ProfilesPanel.tsx";@@ -81,6 +84,7 @@ export function ChatSurface({dispatch({ type: "connectors", open: true });};const [showConversations, setShowConversations] = useState(false);+ const voice = useVoice(state, send);/** A switch that waits for the user to confirm stopping the running answer. */const [pendingSwitch, setPendingSwitch] = useState<SwitchMessage | undefined>(undefined);const switchTo = (message: SwitchMessage) => {@@ -148,6 +152,7 @@ export function ChatSurface({><ConnectorsIcon /></button>+ {voice.mic ? <MuteButton voice={voice} /> : null}{conversations ? (<buttontype="button"@@ -321,11 +326,29 @@ export function ChatSurface({send={send}focusKey={focusKey}onStartTyping={closeOptions}+ voice={voice}/></div>);}+/** Spoken replies on or off (S30.3); the choice is remembered. */+function MuteButton({ voice }: { voice: VoiceControls }) {+ const label = voice.muted ? "Unmute spoken replies" : "Mute spoken replies";+ return (+ <button+ type="button"+ className="icon"+ aria-label={label}+ title={label}+ aria-pressed={voice.muted}+ onClick={() => voice.setMuted(!voice.muted)}+ >+ {voice.muted ? <SpeakerOffIcon /> : <SpeakerIcon />}+ </button>+ );+}+function RoseButton({ controls }: { controls: PanelWindowControls }) {const label = controls.roseHidden ? "Show the rose" : "Hide the rose";return (diff --git a/packages/ui/src/components/Composer.tsx b/packages/ui/src/components/Composer.tsxindex 6bc6442..2bce319 100644--- a/packages/ui/src/components/Composer.tsx+++ b/packages/ui/src/components/Composer.tsx@@ -1,5 +1,6 @@import { type KeyboardEvent, useEffect, useRef, useState } from "react";-import { SendIcon } from "./icons.tsx";+import type { VoiceControls } from "../useVoice.ts";+import { MicIcon, SendIcon } from "./icons.tsx";import { newRequestId, type Send } from "./types.ts";interface ComposerProps {@@ -10,11 +11,15 @@ interface ComposerProps {focusKey?: number;/** Called when the text box goes from empty to having text. */onStartTyping?: () => void;+ /** Talking to the assistant (S30.4); without it there is no mic. */+ voice?: VoiceControls;}-export function Composer({ busy, disabled, send, focusKey, onStartTyping }: ComposerProps) {+export function Composer({ busy, disabled, send, focusKey, onStartTyping, voice }: ComposerProps) {const [text, setText] = useState("");const box = useRef<HTMLTextAreaElement>(null);+ const recording = voice?.phase === "recording";+ const voiceStatus = voice?.phase === "transcribing" ? "Transcribing…" : voice?.note;useEffect(() => {if (focusKey !== undefined) box.current?.focus();@@ -24,6 +29,7 @@ export function Composer({ busy, disabled, send, focusKey, onStartTyping }: Compconst value = text.trim();if (!value || disabled) return;send({ type: "send", text: value, requestId: newRequestId() });+ voice?.typed();setText("");}@@ -35,40 +41,97 @@ export function Composer({ busy, disabled, send, focusKey, onStartTyping }: Comp}return (- <form- className="composer"- onSubmit={(event) => {- event.preventDefault();- submit();- }}- >- <textarea- ref={box}- aria-label="Message"- placeholder={busy ? "Add to what I'm doing…" : "Ask me anything…"}- rows={1}- value={text}- disabled={disabled}- onChange={(event) => {- if (text === "" && event.target.value !== "") onStartTyping?.();- setText(event.target.value);- }}- onKeyDown={onKeyDown}- />- {busy ? (- <button type="button" className="composer-stop" onClick={() => send({ type: "abort" })}>- Stop- </button>+ <>+ {voiceStatus ? (+ <p className="voice-status" role="status" aria-label="Voice">+ {voiceStatus}+ </p>) : null}- <button- type="submit"- className="composer-send"- disabled={disabled || text.trim() === ""}- aria-label="Send"- title="Send"+ <form+ className="composer"+ onSubmit={(event) => {+ event.preventDefault();+ submit();+ }}>- <SendIcon />- </button>- </form>+ {recording ? (+ <div className="voice-recording">+ <span className="voice-time" role="timer" aria-label="Recording time">+ {formatElapsed(voice.elapsed)}+ </span>+ <meter+ className="voice-meter"+ aria-label="Microphone level"+ min={0}+ max={1}+ value={voice.level}+ />+ <span className="voice-partial">{voice.partial || "Listening…"}</span>+ <button+ type="button"+ className="voice-cancel"+ aria-label="Cancel recording"+ title="Cancel (Esc)"+ onClick={voice.cancel}+ >+ ✕+ </button>+ </div>+ ) : (+ <textarea+ ref={box}+ aria-label="Message"+ placeholder={busy ? "Add to what I'm doing…" : "Ask me anything…"}+ rows={1}+ value={text}+ disabled={disabled}+ onChange={(event) => {+ if (text === "" && event.target.value !== "") onStartTyping?.();+ setText(event.target.value);+ }}+ onKeyDown={onKeyDown}+ />+ )}+ {busy || voice?.speaking ? (+ <button+ type="button"+ className="composer-stop"+ onClick={() => {+ if (busy) send({ type: "abort" });+ voice?.stopSpeaking();+ }}+ >+ Stop+ </button>+ ) : null}+ {voice?.mic ? (+ <button+ type="button"+ className={`composer-mic${recording ? " composer-mic-on" : ""}`}+ aria-label={recording ? "Stop and send" : "Talk"}+ title={voice.mic.available ? "Tap to talk, tap again to send" : voice.mic.reason}+ disabled={disabled || !voice.mic.available || voice.phase === "transcribing"}+ onClick={recording ? voice.stop : voice.start}+ >+ <MicIcon />+ </button>+ ) : null}+ {recording ? null : (+ <button+ type="submit"+ className="composer-send"+ disabled={disabled || text.trim() === ""}+ aria-label="Send"+ title="Send"+ >+ <SendIcon />+ </button>+ )}+ </form>+ </>);}++function formatElapsed(seconds: number): string {+ return `${Math.floor(seconds / 60)}:${String(seconds % 60).padStart(2, "0")}`;+}diff --git a/packages/ui/src/components/icons.tsx b/packages/ui/src/components/icons.tsxindex 7333632..c3a6903 100644--- a/packages/ui/src/components/icons.tsx+++ b/packages/ui/src/components/icons.tsx@@ -90,6 +90,39 @@ export function SendIcon() {);}+/** A microphone: talk to the assistant. */+export function MicIcon() {+ return (+ <Icon>+ <rect x="9" y="3" width="6" height="11" rx="3" />+ <path d="M5 11a7 7 0 0 0 14 0" />+ <path d="M12 18v3" />+ </Icon>+ );+}++/** A speaker: spoken replies are on. */+export function SpeakerIcon() {+ return (+ <Icon>+ <path d="M4 9v6h4l5 4V5L8 9Z" />+ <path d="M16.5 8.5a5 5 0 0 1 0 7" />+ <path d="M19 6a8.5 8.5 0 0 1 0 12" />+ </Icon>+ );+}++/** A crossed speaker: spoken replies are muted. */+export function SpeakerOffIcon() {+ return (+ <Icon>+ <path d="M4 9v6h4l5 4V5L8 9Z" />+ <path d="m17 9 5 6" />+ <path d="m22 9-5 6" />+ </Icon>+ );+}+/** The rose (the app's favicon drawing): who the user is talking to. */export function RoseGlyph() {return (diff --git a/packages/ui/src/speech.ts b/packages/ui/src/speech.tsnew file mode 100644index 0000000..eabdec2--- /dev/null+++ b/packages/ui/src/speech.ts@@ -0,0 +1,40 @@+/** Turns a Markdown reply into plain text worth reading aloud: no code, links, or markup. */+export function speakableText(markdown: string): string {+ return (+ markdown+ .replace(/```[\s\S]*?(```|$)/g, " ")+ .replace(/`[^`\n]*`/g, "")+ .replace(/!\[[^\]]*\]\([^)]*\)/g, "")+ .replace(/\[([^\]]*)\]\([^)]*\)/g, "$1")+ .replace(/https?:\/\/\S+/g, "")+ .replace(/<[^>]+>/g, "")+ .replace(/^\s{0,3}(#{1,6}|>|[-*+]|\d+[.)])\s+/gm, "")+ .replace(/[*_~|]/g, "")+ .replace(/^\s*-{3,}\s*$/gm, "")+ .replace(/\s+/g, " ")+ // A space left where something was removed, before punctuation.+ .replace(/\s+([.,;:!?])/g, "$1")+ .trim()+ );+}++const SPANISH = /[ñ¿¡áéíóú]|\b(que|qué|el|los|las|para|por|una|con|está|hola|gracias)\b/gi;+const ENGLISH = /\b(the|and|you|is|are|to|of|with|this|that|hello|thanks)\b/gi;++/** A best guess of the reply's language (`es` or `en`), or undefined when unclear. */+export function guessLanguage(text: string): string | undefined {+ const spanish = text.match(SPANISH)?.length ?? 0;+ const english = text.match(ENGLISH)?.length ?? 0;+ if (spanish === 0 && english === 0) return undefined;+ return spanish > english ? "es" : "en";+}++/** A system voice for the language, preferring the user's own region. */+export function pickVoice<V extends { lang: string }>(+ voices: readonly V[],+ language: string,+ preferred = "",+): V | undefined {+ const matching = voices.filter((v) => v.lang.toLowerCase().startsWith(language));+ return matching.find((v) => v.lang.toLowerCase() === preferred.toLowerCase()) ?? matching[0];+}diff --git a/packages/ui/src/store.ts b/packages/ui/src/store.tsindex 394a4c2..d13d89e 100644--- a/packages/ui/src/store.ts+++ b/packages/ui/src/store.ts@@ -15,6 +15,7 @@ import type {Profile,ProfileRole,ServerMessage,+ VoiceCapability,} from "@gentle-dot/protocol";import { MAX_CONNECTOR_DRAFTS } from "@gentle-dot/protocol";@@ -60,6 +61,12 @@ export interface ConnectorsState {imports?: ImportCandidate[];}+/** The daemon's answer to a voice request, matched by `requestId`. */+export type VoiceReply = Extract<+ ServerMessage,+ { type: "voice_transcript" | "voice_speech" | "voice_unavailable" }+>;+export type ConnectionStatus = "connecting" | "open" | "closed" | "unauthorized";export interface ChatMessage {@@ -98,6 +105,10 @@ export interface DotState {auth: AuthState;profiles: ProfilesState;connectors: ConnectorsState;+ /** What the daemon can do with voice (an OpenAI API key). */+ voice: VoiceCapability;+ /** The latest answers to voice requests, newest last. */+ voiceReplies: VoiceReply[];}export type DotAction =@@ -125,8 +136,12 @@ export const initialState: DotState = {auth: { open: false },profiles: { open: false, roles: [], models: [], importable: false },connectors: { open: false },+ voice: { transcribe: false, speak: false },+ voiceReplies: [],};+const VOICE_REPLIES = 4;+let nextNotice = 0;export function reduce(state: DotState, action: DotAction): DotState {@@ -181,6 +196,7 @@ function reduceServer(state: DotState, message: ServerMessage): DotState {conversationId: message.conversationId ?? state.conversationId,model: message.model ?? state.model,features: message.features ?? initialState.features,+ voice: message.voice ?? initialState.voice,// The daemon sends the drafts still waiting right after ready.connectors: { ...state.connectors, drafts: [] },};@@ -251,7 +267,12 @@ function reduceServer(state: DotState, message: ServerMessage): DotState {return {...closeConnectors(closeProfiles(state, message.open === true), message.open === true),auth: { ...state.auth, providers: message.providers, open: message.open ? true : state.auth.open },+ voice: message.voice ?? state.voice,};+ case "voice_transcript":+ case "voice_speech":+ case "voice_unavailable":+ return { ...state, voiceReplies: [...state.voiceReplies, message].slice(-VOICE_REPLIES) };case "connectors": {const updated = { ...state, connectors: { ...state.connectors, list: message.connectors } };return message.open ? openConnectors(updated, true) : updated;diff --git a/packages/ui/src/styles.css b/packages/ui/src/styles.cssindex 0025eb2..ceee92d 100644--- a/packages/ui/src/styles.css+++ b/packages/ui/src/styles.css@@ -1183,6 +1183,89 @@ html[data-effects="none"] .chat-panel {border-radius: var(--radius-pill);}+/* ---------- Voice (S30.4) ---------- */++.composer-mic {+ display: grid;+ place-items: center;+ width: 32px;+ height: 32px;+ padding: 0;+ border-radius: 50%;+}++.composer-mic-on {+ background: var(--accent);+ border-color: var(--accent);+ color: var(--accent-ink);+ box-shadow: 0 0 0 4px var(--accent-soft);+}++.voice-recording {+ flex: 1;+ display: flex;+ align-items: center;+ gap: 10px;+ min-height: 32px;+ min-width: 0;+}++.voice-time {+ font-family: var(--font-mono);+ font-variant-numeric: tabular-nums;+ color: var(--accent);+}++.voice-meter {+ appearance: none;+ width: 56px;+ height: 6px;+ flex: none;+ border: none;+ border-radius: var(--radius-pill);+ background: var(--surface-2);+ overflow: hidden;+}++.voice-meter::-webkit-meter-bar {+ height: 6px;+ border: none;+ border-radius: var(--radius-pill);+ background: var(--surface-2);+}++.voice-meter::-webkit-meter-optimum-value {+ background: var(--accent);+}++.voice-meter::-moz-meter-bar {+ background: var(--accent);+}++.voice-partial {+ flex: 1;+ min-width: 0;+ overflow: hidden;+ text-overflow: ellipsis;+ white-space: nowrap;+ color: var(--muted);+}++.voice-cancel {+ display: grid;+ place-items: center;+ width: 28px;+ height: 28px;+ padding: 0;+ border-radius: 50%;+}++.voice-status {+ margin: 0 16px 6px;+ font-size: 13px;+ color: var(--muted);+}+/* ---------- Accounts ---------- */.onboarding {diff --git a/packages/ui/src/useVoice.ts b/packages/ui/src/useVoice.tsnew file mode 100644index 0000000..fd369d1--- /dev/null+++ b/packages/ui/src/useVoice.ts@@ -0,0 +1,451 @@+import { MAX_VOICE_AUDIO, MAX_VOICE_TEXT } from "@gentle-dot/protocol";+import { useCallback, useEffect, useRef, useState } from "react";+import { newRequestId, type Send } from "./components/types.ts";+import { inDesktop } from "./desktop.ts";+import { guessLanguage, pickVoice, speakableText } from "./speech.ts";+import { type DotState, isBusy } from "./store.ts";+import {+ cancelNativeVoice,+ type NativeVoiceStatus,+ nativeVoiceStatus,+ onNativeVoice,+ startNativeVoice,+ startRefusal,+ stopNativeVoice,+} from "./voice-native.ts";++export type VoicePhase = "idle" | "recording" | "transcribing";++/** The mic button: usable, or disabled with the reason. */+export type MicState = { available: true } | { available: false; reason: string };++/** Talking to the assistant (S30.2–S30.4), as the composer and the header use it. */+export interface VoiceControls {+ /** Undefined when the mic is not offered at all (the web without an OpenAI key). */+ mic?: MicState;+ phase: VoicePhase;+ /** Seconds since recording started. */+ elapsed: number;+ /** Microphone level, 0 to 1. */+ level: number;+ /** Live text from the desktop recognizer. */+ partial: string;+ /** Why the last attempt did not work. */+ note?: string;+ speaking: boolean;+ muted: boolean;+ start(): void;+ stop(): void;+ cancel(): void;+ stopSpeaking(): void;+ setMuted(muted: boolean): void;+ /** The user sent a typed message: its reply is not read aloud. */+ typed(): void;+}++/** Recordings stop on their own so they fit the daemon's WebSocket limit (S30.2). */+const MAX_RECORDING_MS = 60_000;+const MUTED_KEY = "gentle-dot-voice-muted";+const RECORDING_TYPES = ["audio/webm;codecs=opus", "audio/webm", "audio/mp4", "audio/ogg;codecs=opus"];+const REASONS = {+ insecure: "Voice needs a secure page (HTTPS or localhost).",+ noRecorder: "This browser cannot record audio.",+ denied: "Microphone access was denied. Allow it to talk to the assistant.",+ tooLong: "That recording is too long. Keep it under a minute.",+ empty: "I didn't catch anything. Try again.",+} as const;++type Source = { kind: "daemon" | "native" } | { kind: "none"; reason?: string };++/**+ * Where speech becomes text: the daemon (an OpenAI key) when it can and the page may record,+ * else the desktop recognizer. The web offers only the daemon, over HTTPS or localhost.+ */+function chooseSource(desktop: boolean, transcribe: boolean, native: NativeVoiceStatus | undefined): Source {+ const secure = desktop || window.isSecureContext === true;+ const recorder = typeof MediaRecorder !== "undefined" && navigator.mediaDevices?.getUserMedia !== undefined;+ if (transcribe && secure && recorder) return { kind: "daemon" };+ if (desktop) {+ if (!native) return { kind: "none" };+ return native.available+ ? { kind: "native" }+ : { kind: "none", reason: native.reason ?? REASONS.noRecorder };+ }+ if (!transcribe) return { kind: "none" };+ return { kind: "none", reason: secure ? REASONS.noRecorder : REASONS.insecure };+}++interface Recording {+ kind: "daemon" | "native";+ cancelled: boolean;+ recorder?: MediaRecorder;+ ticker?: ReturnType<typeof setInterval>;+ limit?: ReturnType<typeof setTimeout>;+ stopMeter?: () => void;+}++/** Stops the elapsed-time ticker, the length limit, and the level meter. */+function stopTimers(entry: Recording): void {+ clearInterval(entry.ticker);+ clearTimeout(entry.limit);+ entry.stopMeter?.();+}++export function useVoice(state: DotState, send: Send): VoiceControls {+ const desktop = inDesktop();+ const [native, setNative] = useState<NativeVoiceStatus | undefined>(undefined);+ const [phase, setPhaseState] = useState<VoicePhase>("idle");+ const [elapsed, setElapsed] = useState(0);+ const [level, setLevel] = useState(0);+ const [partial, setPartial] = useState("");+ const [note, setNote] = useState<string | undefined>(undefined);+ const [speaking, setSpeaking] = useState(false);+ const [muted, setMutedState] = useState(() => readMuted());++ const phaseNow = useRef<VoicePhase>("idle");+ const recording = useRef<Recording | undefined>(undefined);+ const starting = useRef(false);+ const pendingTranscript = useRef<string | undefined>(undefined);+ const pendingSpeech = useRef<{ requestId: string; text: string } | undefined>(undefined);+ const audio = useRef<HTMLAudioElement | undefined>(undefined);+ /** A reply to a spoken message is awaited: the ids of the messages shown before it. */+ const voiceTurn = useRef<Set<string> | undefined>(undefined);+ const latest = useRef(state);+ latest.current = state;+ const mutedNow = useRef(muted);+ mutedNow.current = muted;++ const source = chooseSource(desktop, state.voice.transcribe, native);++ const setPhase = useCallback((next: VoicePhase) => {+ phaseNow.current = next;+ setPhaseState(next);+ }, []);++ const finish = useCallback(+ (reason?: string) => {+ if (recording.current) stopTimers(recording.current);+ recording.current = undefined;+ pendingTranscript.current = undefined;+ setPhase("idle");+ setElapsed(0);+ setLevel(0);+ setPartial("");+ setNote(reason);+ },+ [setPhase],+ );++ const stopSpeaking = useCallback(() => {+ pendingSpeech.current = undefined;+ audio.current?.pause();+ audio.current = undefined;+ globalThis.speechSynthesis?.cancel();+ setSpeaking(false);+ }, []);++ const speakWithSystem = useCallback((text: string) => {+ const synth = globalThis.speechSynthesis;+ if (!synth || typeof SpeechSynthesisUtterance === "undefined") return;+ const utterance = new SpeechSynthesisUtterance(text);+ const language = guessLanguage(text) ?? navigator.language.slice(0, 2).toLowerCase();+ utterance.lang = language;+ const voice = pickVoice(synth.getVoices(), language, navigator.language);+ if (voice) utterance.voice = voice;+ utterance.onend = () => setSpeaking(false);+ synth.speak(utterance);+ setSpeaking(true);+ }, []);++ const speak = useCallback(+ (reply: string) => {+ const text = speakableText(reply).slice(0, MAX_VOICE_TEXT);+ if (!text || mutedNow.current) return;+ if (!latest.current.voice.speak) {+ speakWithSystem(text);+ return;+ }+ const requestId = newRequestId();+ pendingSpeech.current = { requestId, text };+ send({ type: "voice_speak", requestId, text });+ },+ [send, speakWithSystem],+ );++ const sendTranscript = useCallback(+ (transcript: string) => {+ const text = transcript.trim();+ if (!text) {+ finish(REASONS.empty);+ return;+ }+ voiceTurn.current = new Set(latest.current.messages.map((m) => m.id));+ send({ type: "send", text, requestId: newRequestId() });+ finish();+ },+ [send, finish],+ );++ const begin = useCallback(+ (entry: Recording) => {+ const started = Date.now();+ entry.ticker = setInterval(() => setElapsed(Math.floor((Date.now() - started) / 1000)), 250);+ recording.current = entry;+ setNote(undefined);+ setPartial("");+ setElapsed(0);+ setPhase("recording");+ },+ [setPhase],+ );++ const upload = useCallback(+ async (blob: Blob, mime: string) => {+ setPhase("transcribing");+ const data = await toBase64(blob);+ if (data.length > MAX_VOICE_AUDIO) {+ finish(REASONS.tooLong);+ return;+ }+ const requestId = newRequestId();+ pendingTranscript.current = requestId;+ send({ type: "voice_transcribe", requestId, mime: mime.replace(/\s+/g, ""), data });+ },+ [send, finish, setPhase],+ );++ const stop = useCallback(() => {+ const current = recording.current;+ if (!current || phaseNow.current !== "recording") return;+ stopTimers(current);+ if (current.kind === "daemon") {+ current.recorder?.stop();+ return;+ }+ setPhase("transcribing");+ stopNativeVoice().then(sendTranscript, (error: unknown) => finish(startRefusal(error)));+ }, [finish, sendTranscript, setPhase]);++ const startDaemon = useCallback(async () => {+ let stream: MediaStream;+ try {+ stream = await navigator.mediaDevices.getUserMedia({ audio: true });+ } catch {+ finish(REASONS.denied);+ return;+ }+ const type = RECORDING_TYPES.find((t) => MediaRecorder.isTypeSupported?.(t));+ const recorder = new MediaRecorder(stream, type ? { mimeType: type } : undefined);+ const entry: Recording = { kind: "daemon", cancelled: false, recorder };+ const chunks: Blob[] = [];+ recorder.ondataavailable = (event) => {+ if (event.data.size > 0) chunks.push(event.data);+ };+ recorder.onstop = () => {+ for (const track of stream.getTracks()) track.stop();+ if (entry.cancelled) return;+ const mime = recorder.mimeType || type || "audio/webm";+ void upload(new Blob(chunks, { type: mime }), mime);+ };+ recorder.start();+ entry.stopMeter = startMeter(stream, setLevel);+ begin(entry);+ entry.limit = setTimeout(stop, MAX_RECORDING_MS);+ }, [begin, finish, stop, upload]);++ const startNative = useCallback(async () => {+ try {+ await startNativeVoice(navigator.language);+ } catch (error) {+ finish(startRefusal(error));+ return;+ }+ begin({ kind: "native", cancelled: false });+ }, [begin, finish]);++ const start = useCallback(() => {+ if (phaseNow.current !== "idle" || starting.current) return;+ stopSpeaking();+ if (source.kind === "none") return;+ starting.current = true;+ const run = source.kind === "daemon" ? startDaemon() : startNative();+ void run.finally(() => {+ starting.current = false;+ });+ }, [source.kind, startDaemon, startNative, stopSpeaking]);++ const cancel = useCallback(() => {+ const current = recording.current;+ if (!current) return;+ current.cancelled = true;+ if (current.kind === "daemon") current.recorder?.stop();+ else void cancelNativeVoice().catch(() => {});+ finish();+ }, [finish]);++ const setMuted = useCallback(+ (next: boolean) => {+ setMutedState(next);+ try {+ if (next) localStorage.setItem(MUTED_KEY, "1");+ else localStorage.removeItem(MUTED_KEY);+ } catch {+ // Private browsing: the choice lasts for this window only.+ }+ if (next) stopSpeaking();+ },+ [stopSpeaking],+ );++ const typed = useCallback(() => {+ voiceTurn.current = undefined;+ }, []);++ // The desktop recognizer's status and live events.+ useEffect(() => {+ if (!desktop) return;+ let live = true;+ void nativeVoiceStatus().then((status) => {+ if (live) setNative(status);+ });+ const off = onNativeVoice({+ partial: (text) => {+ if (phaseNow.current === "recording") setPartial(text);+ },+ level: (value) => {+ if (phaseNow.current === "recording") setLevel(value);+ },+ error: (message) => {+ if (recording.current?.kind === "native") finish(message);+ },+ }).catch(() => () => {});+ return () => {+ live = false;+ void off.then((stopListening) => stopListening());+ };+ }, [desktop, finish]);++ // Answers to voice requests.+ useEffect(() => {+ for (const reply of state.voiceReplies) {+ if (reply.requestId === pendingTranscript.current) {+ pendingTranscript.current = undefined;+ if (reply.type === "voice_transcript") sendTranscript(reply.text);+ else if (reply.type === "voice_unavailable") finish(reply.reason);+ }+ const speech = pendingSpeech.current;+ if (speech && reply.requestId === speech.requestId) {+ pendingSpeech.current = undefined;+ if (reply.type === "voice_speech") {+ const player = new Audio(`data:${reply.mime};base64,${reply.data}`);+ audio.current = player;+ player.onended = () => setSpeaking(false);+ setSpeaking(true);+ void player.play().catch(() => setSpeaking(false));+ } else if (reply.type === "voice_unavailable") {+ speakWithSystem(speech.text);+ }+ }+ }+ }, [state.voiceReplies, sendTranscript, finish, speakWithSystem]);++ // Read the reply to a spoken message aloud once the assistant finished it.+ useEffect(() => {+ const before = voiceTurn.current;+ if (!before || isBusy(state.agentState)) return;+ const reply = state.messages.findLast((m) => m.role === "assistant" && !before.has(m.id));+ if (!reply || reply.streaming) return;+ voiceTurn.current = undefined;+ if (!reply.note) speak(reply.text);+ }, [state.messages, state.agentState, speak]);++ // Esc cancels a recording before the panel sees it (it would hide the panel).+ useEffect(() => {+ if (phase !== "recording") return;+ const onKey = (event: KeyboardEvent) => {+ if (event.key !== "Escape") return;+ event.preventDefault();+ cancel();+ };+ window.addEventListener("keydown", onKey, true);+ return () => window.removeEventListener("keydown", onKey, true);+ }, [phase, cancel]);++ // Leaving the chat drops a recording and silences the voice.+ const cancelNow = useRef(cancel);+ cancelNow.current = cancel;+ const stopSpeakingNow = useRef(stopSpeaking);+ stopSpeakingNow.current = stopSpeaking;+ useEffect(+ () => () => {+ cancelNow.current();+ stopSpeakingNow.current();+ },+ [],+ );++ const mic: MicState | undefined =+ source.kind === "none"+ ? source.reason+ ? { available: false, reason: source.reason }+ : undefined+ : { available: true };++ return {+ ...(mic ? { mic } : {}),+ phase,+ elapsed,+ level,+ partial,+ ...(note ? { note } : {}),+ speaking,+ muted,+ start,+ stop,+ cancel,+ stopSpeaking,+ setMuted,+ typed,+ };+}++function readMuted(): boolean {+ try {+ return localStorage.getItem(MUTED_KEY) === "1";+ } catch {+ return false;+ }+}++async function toBase64(blob: Blob): Promise<string> {+ const bytes = new Uint8Array(await blob.arrayBuffer());+ let binary = "";+ for (let i = 0; i < bytes.length; i += 0x8000)+ binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000));+ return btoa(binary);+}++/** The microphone level from the recording's stream, where the browser can measure it. */+function startMeter(stream: MediaStream, onLevel: (level: number) => void): () => void {+ const Context = globalThis.AudioContext;+ if (!Context) return () => {};+ try {+ const context = new Context();+ const analyser = context.createAnalyser();+ analyser.fftSize = 512;+ context.createMediaStreamSource(stream).connect(analyser);+ const samples = new Uint8Array(analyser.fftSize);+ const timer = setInterval(() => {+ analyser.getByteTimeDomainData(samples);+ let sum = 0;+ for (const sample of samples) sum += ((sample - 128) / 128) ** 2;+ onLevel(Math.min(1, Math.sqrt(sum / samples.length) * 3));+ }, 100);+ return () => {+ clearInterval(timer);+ void context.close();+ };+ } catch {+ return () => {};+ }+}diff --git a/packages/ui/src/voice-native.ts b/packages/ui/src/voice-native.tsnew file mode 100644index 0000000..a40c0cf--- /dev/null+++ b/packages/ui/src/voice-native.ts@@ -0,0 +1,73 @@+import { invoke } from "@tauri-apps/api/core";++/**+ * The desktop app's speech recognizer (S30.1, contract in L88): the commands+ * `voice_status`, `voice_start`, `voice_stop`, `voice_cancel`, and the events+ * `voice://partial`, `voice://level`, `voice://error`. Desktop only.+ */+export interface NativeVoiceStatus {+ available: boolean;+ reason?: string;+}++const UNAVAILABLE = "Voice is not available on this computer.";++export async function nativeVoiceStatus(): Promise<NativeVoiceStatus> {+ try {+ const status = await invoke<Partial<NativeVoiceStatus> | undefined>("voice_status");+ if (status?.available === true) return { available: true };+ return { available: false, reason: typeof status?.reason === "string" ? status.reason : UNAVAILABLE };+ } catch {+ return { available: false, reason: UNAVAILABLE };+ }+}++/** Starts listening; rejects with the app's reason (for example a denied permission). */+export function startNativeVoice(locale?: string): Promise<void> {+ return invoke<void>("voice_start", locale ? { locale } : {});+}++/** Stops listening and resolves with the final text. */+export async function stopNativeVoice(): Promise<string> {+ const result = await invoke<{ text?: unknown } | undefined>("voice_stop");+ return typeof result?.text === "string" ? result.text : "";+}++export function cancelNativeVoice(): Promise<void> {+ return invoke<void>("voice_cancel");+}++export interface NativeVoiceHandlers {+ partial: (text: string) => void;+ level: (level: number) => void;+ error: (message: string) => void;+}++/** Listens to the recognizer's events; resolves with a function that stops listening. */+export async function onNativeVoice(handlers: NativeVoiceHandlers): Promise<() => void> {+ const { listen } = await import("@tauri-apps/api/event");+ const stops = await Promise.all([+ listen<{ text?: unknown }>("voice://partial", (event) => {+ if (typeof event.payload?.text === "string") handlers.partial(event.payload.text);+ }),+ listen<{ level?: unknown }>("voice://level", (event) => {+ const level = event.payload?.level;+ if (typeof level === "number" && Number.isFinite(level))+ handlers.level(Math.min(1, Math.max(0, level)));+ }),+ listen<{ message?: unknown }>("voice://error", (event) => {+ const message = event.payload?.message;+ handlers.error(typeof message === "string" && message ? message : "Listening stopped.");+ }),+ ]);+ return () => {+ for (const stop of stops) stop();+ };+}++/** Why `voice_start` refused, in the app's words when it gave some. */+export function startRefusal(error: unknown): string {+ if (typeof error === "string" && error) return error;+ if (error instanceof Error && error.message) return error.message;+ return "Could not start listening.";+}diff --git a/packages/ui/test/voice.test.tsx b/packages/ui/test/voice.test.tsxnew file mode 100644index 0000000..4296140--- /dev/null+++ b/packages/ui/test/voice.test.tsx@@ -0,0 +1,432 @@+import type { ClientMessage, ServerMessage, ServerPayload } from "@gentle-dot/protocol";+import { act, fireEvent, render, screen, waitFor } from "@testing-library/react";+import userEvent from "@testing-library/user-event";+import { useReducer } from "react";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import { ChatSurface } from "../src/components/ChatSurface.tsx";+import { type DotState, initialState, reduce } from "../src/store.ts";++/** Talking to the assistant (S30.2–S30.4) through the chat, with the desktop and browser mocked. */+const h = vi.hoisted(() => ({+ tauri: false,+ invoke: vi.fn(),+ listeners: new Map<string, (event: { payload: unknown }) => void>(),+ server: (_payload: ServerPayload) => {},+}));++vi.mock("@tauri-apps/api/core", () => ({+ isTauri: () => h.tauri,+ invoke: (command: string, args?: unknown) => h.invoke(command, args),+}));++vi.mock("@tauri-apps/api/event", () => ({+ listen: async (name: string, handler: (event: { payload: unknown }) => void) => {+ h.listeners.set(name, handler);+ return () => h.listeners.delete(name);+ },+}));++class FakeRecorder {+ static instances: FakeRecorder[] = [];+ static isTypeSupported = (type: string) => type === "audio/webm;codecs=opus";+ state: "inactive" | "recording" = "inactive";+ mimeType: string;+ ondataavailable: ((event: { data: Blob }) => void) | null = null;+ onstop: (() => void) | null = null;+ constructor(_stream: unknown, options?: { mimeType?: string }) {+ this.mimeType = options?.mimeType ?? "";+ FakeRecorder.instances.push(this);+ }+ start() {+ this.state = "recording";+ }+ stop() {+ this.state = "inactive";+ this.ondataavailable?.({ data: new Blob(["fake audio"], { type: this.mimeType }) });+ this.onstop?.();+ }+}++const track = { stop: vi.fn() };+const getUserMedia = vi.fn(async () => ({ getTracks: () => [track] }));++class FakeUtterance {+ lang = "";+ voice: unknown = null;+ onend: (() => void) | null = null;+ readonly text: string;+ constructor(text: string) {+ this.text = text;+ }+}+const synth = {+ speaking: false,+ speak: vi.fn((u: FakeUtterance) => {+ synth.speaking = true;+ synth.current = u;+ }),+ cancel: vi.fn(() => {+ synth.speaking = false;+ }),+ getVoices: () => [+ { name: "Samantha", lang: "en-US" },+ { name: "Paulina", lang: "es-MX" },+ ],+ current: undefined as FakeUtterance | undefined,+};++class FakeAudio {+ static instances: FakeAudio[] = [];+ onended: (() => void) | null = null;+ play = vi.fn(async () => {});+ pause = vi.fn();+ readonly src: string;+ constructor(src: string) {+ this.src = src;+ FakeAudio.instances.push(this);+ }+}++let seq = 0;+const sent: ClientMessage[] = [];+const sentOf = <T extends ClientMessage["type"]>(type: T) =>+ sent.filter((m): m is Extract<ClientMessage, { type: T }> => m.type === type);++function Harness({ initial }: { initial: DotState }) {+ const [state, dispatch] = useReducer(reduce, initial);+ h.server = (payload) => dispatch({ type: "server", message: { ...payload, seq: ++seq } as ServerMessage });+ return (+ <ChatSurface+ variant="panel"+ state={state}+ send={(m) => sent.push(m)}+ dismiss={() => {}}+ dispatch={dispatch}+ />+ );+}++const ready: DotState = { ...initialState, connection: "open", agentState: "idle" };+const withVoice = (transcribe: boolean, speak = transcribe): DotState => ({+ ...ready,+ voice: { transcribe, speak },+});+const server = (...payloads: ServerPayload[]) =>+ act(() => {+ for (const payload of payloads) h.server(payload);+ });+const talk = () => screen.getByRole("button", { name: "Talk" });++function setSecure(secure: boolean) {+ Object.defineProperty(window, "isSecureContext", { value: secure, configurable: true });+}++beforeEach(() => {+ h.tauri = false;+ h.listeners.clear();+ h.invoke = vi.fn(async () => undefined);+ sent.length = 0;+ FakeRecorder.instances = [];+ FakeAudio.instances = [];+ synth.speaking = false;+ synth.current = undefined;+ synth.speak.mockClear();+ synth.cancel.mockClear();+ getUserMedia.mockClear();+ track.stop.mockClear();+ localStorage.clear();+ setSecure(true);+ vi.stubGlobal("MediaRecorder", FakeRecorder);+ vi.stubGlobal("SpeechSynthesisUtterance", FakeUtterance);+ vi.stubGlobal("speechSynthesis", synth);+ vi.stubGlobal("Audio", FakeAudio);+ Object.defineProperty(navigator, "mediaDevices", { value: { getUserMedia }, configurable: true });+});++afterEach(() => {+ vi.useRealTimers();+ vi.unstubAllGlobals();+});++/** A voice turn through the daemon: record, stop, transcript, and the reply's messages. */+async function voiceTurn(transcript: string) {+ await userEvent.click(talk());+ await userEvent.click(await screen.findByRole("button", { name: "Stop and send" }));+ const request = await waitFor(() => {+ const found = sentOf("voice_transcribe").at(-1);+ if (!found) throw new Error("no voice_transcribe yet");+ return found;+ });+ server({ type: "voice_transcript", requestId: request.requestId, text: transcript });+ return request;+}++describe("the mic in the browser", () => {+ it("is hidden without an OpenAI key, and so is the mute toggle", () => {+ render(<Harness initial={withVoice(false)} />);+ expect(screen.queryByRole("button", { name: "Talk" })).not.toBeInTheDocument();+ expect(screen.queryByRole("button", { name: /spoken replies/ })).not.toBeInTheDocument();+ });++ it("is disabled with the reason on a page that is not secure", () => {+ setSecure(false);+ render(<Harness initial={withVoice(true)} />);+ expect(talk()).toBeDisabled();+ expect(talk()).toHaveAccessibleDescription("Voice needs a secure page (HTTPS or localhost).");+ });++ it("appears once the accounts list reports voice", () => {+ render(<Harness initial={withVoice(false)} />);+ server({ type: "auth_providers", providers: [], voice: { transcribe: true, speak: true } });+ expect(talk()).toBeEnabled();+ });+});++describe("talking through the daemon", () => {+ it("goes idle → recording → transcribing → sent, and the transcript stays as the message", async () => {+ render(<Harness initial={withVoice(true)} />);+ await userEvent.click(talk());+ expect(getUserMedia).toHaveBeenCalledWith({ audio: true });+ expect(await screen.findByRole("timer")).toHaveTextContent("0:00");+ expect(screen.getByRole("meter", { name: "Microphone level" })).toBeInTheDocument();+ expect(screen.queryByRole("textbox", { name: "Message" })).not.toBeInTheDocument();++ await userEvent.click(screen.getByRole("button", { name: "Stop and send" }));+ const request = await waitFor(() => {+ const found = sentOf("voice_transcribe")[0];+ if (!found) throw new Error("not sent");+ return found;+ });+ expect(request.mime).toBe("audio/webm;codecs=opus");+ expect(request.data).toBe(btoa("fake audio"));+ expect(track.stop).toHaveBeenCalled();+ expect(screen.getByRole("status", { name: "Voice" })).toHaveTextContent("Transcribing…");++ server({ type: "voice_transcript", requestId: request.requestId, text: "remind me at five" });+ expect(sentOf("send")).toEqual([+ { type: "send", text: "remind me at five", requestId: expect.any(String) },+ ]);+ server({ type: "user_message", messageId: "u1", text: "remind me at five" });+ expect(screen.getByText("remind me at five")).toBeInTheDocument();+ expect(screen.getByRole("textbox", { name: "Message" })).toBeInTheDocument();+ });++ it("cancels with ✕ or Esc without sending anything, and Esc does not reach the panel", async () => {+ render(<Harness initial={withVoice(true)} />);+ await userEvent.click(talk());+ await userEvent.click(await screen.findByRole("button", { name: "Cancel recording" }));+ expect(talk()).toBeEnabled();++ await userEvent.click(talk());+ await screen.findByRole("timer");+ const panelSaw = vi.fn();+ window.addEventListener("keydown", panelSaw);+ fireEvent.keyDown(document.body, { key: "Escape" });+ window.removeEventListener("keydown", panelSaw);+ expect(panelSaw.mock.calls[0]?.[0].defaultPrevented).toBe(true);+ await waitFor(() => expect(talk()).toBeEnabled());+ expect(sentOf("voice_transcribe")).toEqual([]);+ expect(sentOf("send")).toEqual([]);+ });++ it("stops and sends on its own after about 60 seconds", async () => {+ vi.useFakeTimers({ shouldAdvanceTime: true });+ render(<Harness initial={withVoice(true)} />);+ fireEvent.click(talk());+ await screen.findByRole("timer");+ await act(async () => {+ vi.advanceTimersByTime(60_000);+ });+ await waitFor(() => expect(sentOf("voice_transcribe")).toHaveLength(1));+ });++ it("shows why when the daemon cannot transcribe, and sends nothing", async () => {+ render(<Harness initial={withVoice(true)} />);+ await userEvent.click(talk());+ await userEvent.click(await screen.findByRole("button", { name: "Stop and send" }));+ const request = await waitFor(() => {+ const found = sentOf("voice_transcribe")[0];+ if (!found) throw new Error("not sent");+ return found;+ });+ server({+ type: "voice_unavailable",+ requestId: request.requestId,+ reason: "OpenAI did not accept the API key. Check it in Accounts.",+ });+ expect(screen.getByRole("status", { name: "Voice" })).toHaveTextContent(+ "OpenAI did not accept the API key. Check it in Accounts.",+ );+ expect(sentOf("send")).toEqual([]);+ });+});++describe("talking on the desktop", () => {+ function desktop(status: unknown, stopText = "hola mundo") {+ h.tauri = true;+ h.invoke = vi.fn(async (command: string) => {+ if (command === "voice_status") return status;+ if (command === "voice_stop") return { text: stopText };+ return undefined;+ });+ }+ const invoked = (command: string) => h.invoke.mock.calls.filter(([c]) => c === command);+ const emit = (name: string, payload: unknown) => act(() => h.listeners.get(name)?.({ payload }));++ it("uses the native recognizer without a key: live partial text, level, then the final text is sent", async () => {+ desktop({ available: true });+ render(<Harness initial={withVoice(false)} />);+ await waitFor(() => expect(talk()).toBeEnabled());+ await userEvent.click(talk());+ expect(invoked("voice_start")[0]?.[1]).toEqual({ locale: navigator.language });+ expect(getUserMedia).not.toHaveBeenCalled();+ await screen.findByRole("timer");+ emit("voice://partial", { text: "hola" });+ emit("voice://level", { level: 0.5 });+ expect(screen.getByText("hola")).toBeInTheDocument();+ expect((screen.getByRole("meter", { name: "Microphone level" }) as HTMLMeterElement).value).toBe(0.5);+ await userEvent.click(screen.getByRole("button", { name: "Stop and send" }));+ await waitFor(() =>+ expect(sentOf("send")).toEqual([{ type: "send", text: "hola mundo", requestId: expect.any(String) }]),+ );+ expect(invoked("voice_stop")).toHaveLength(1);+ });++ it("cancels the native recognizer", async () => {+ desktop({ available: true });+ render(<Harness initial={withVoice(false)} />);+ await waitFor(() => expect(talk()).toBeEnabled());+ await userEvent.click(talk());+ await userEvent.click(await screen.findByRole("button", { name: "Cancel recording" }));+ expect(invoked("voice_cancel")).toHaveLength(1);+ expect(sentOf("send")).toEqual([]);+ });++ it("shows the native error and goes back to idle", async () => {+ desktop({ available: true });+ render(<Harness initial={withVoice(false)} />);+ await waitFor(() => expect(talk()).toBeEnabled());+ await userEvent.click(talk());+ await screen.findByRole("timer");+ emit("voice://error", { message: "Speech recognition stopped." });+ expect(screen.getByRole("status", { name: "Voice" })).toHaveTextContent("Speech recognition stopped.");+ expect(talk()).toBeEnabled();+ });++ it("disables the mic with the reason the app gives", async () => {+ desktop({ available: false, reason: "Microphone access is off. Turn it on in System Settings." });+ render(<Harness initial={withVoice(false)} />);+ await waitFor(() =>+ expect(talk()).toHaveAccessibleDescription("Microphone access is off. Turn it on in System Settings."),+ );+ expect(talk()).toBeDisabled();+ });++ it("prefers the daemon when an OpenAI key is connected", async () => {+ desktop({ available: true });+ render(<Harness initial={withVoice(true)} />);+ await userEvent.click(talk());+ expect(getUserMedia).toHaveBeenCalled();+ expect(invoked("voice_start")).toHaveLength(0);+ });+});++describe("spoken replies", () => {+ const reply = (id: string, text: string) =>+ server(+ { type: "agent_state", state: "thinking" },+ { type: "message_done", messageId: id, text },+ { type: "agent_state", state: "idle" },+ );++ it("reads a reply to a spoken message aloud with a system voice, without markdown or code", async () => {+ render(<Harness initial={withVoice(true, false)} />);+ await voiceTurn("hi");+ server({ type: "user_message", messageId: "u1", text: "hi" });+ reply("a1", "**Hello** there!\n\n```js\nconsole.log(1)\n```\nSee [the docs](https://x.dev) and `npm i`.");+ await waitFor(() => expect(synth.speak).toHaveBeenCalledTimes(1));+ expect(synth.current?.text).toBe("Hello there! See the docs and.");+ expect(synth.current?.lang).toBe("en");+ });++ it("picks a voice in the reply's language", async () => {+ render(<Harness initial={withVoice(true, false)} />);+ await voiceTurn("hola");+ reply("a1", "¡Hola! ¿Qué necesitás que haga por vos hoy?");+ await waitFor(() => expect(synth.speak).toHaveBeenCalled());+ expect(synth.current?.lang).toBe("es");+ expect(synth.current?.voice).toMatchObject({ name: "Paulina" });+ });++ it("does not read replies to typed messages", async () => {+ render(<Harness initial={withVoice(true, false)} />);+ await userEvent.type(screen.getByRole("textbox", { name: "Message" }), "typed{Enter}");+ reply("a1", "A typed answer.");+ await new Promise((r) => setTimeout(r, 20));+ expect(synth.speak).not.toHaveBeenCalled();+ });++ it("uses the daemon's voice when it can speak, and plays the audio it returns", async () => {+ render(<Harness initial={withVoice(true, true)} />);+ await voiceTurn("hi");+ reply("a1", "# Done\n- one item");+ const request = await waitFor(() => {+ const found = sentOf("voice_speak")[0];+ if (!found) throw new Error("not asked");+ return found;+ });+ expect(request.text).toBe("Done one item");+ server({ type: "voice_speech", requestId: request.requestId, mime: "audio/mpeg", data: "QUJD" });+ await waitFor(() => expect(FakeAudio.instances[0]?.play).toHaveBeenCalled());+ expect(FakeAudio.instances[0]?.src).toBe("data:audio/mpeg;base64,QUJD");+ expect(synth.speak).not.toHaveBeenCalled();+ });++ it("stays silent while muted, and remembers the choice", async () => {+ const { unmount } = render(<Harness initial={withVoice(true, false)} />);+ await userEvent.click(screen.getByRole("button", { name: "Mute spoken replies" }));+ expect(screen.getByRole("button", { name: "Unmute spoken replies" })).toHaveAttribute(+ "aria-pressed",+ "true",+ );+ await voiceTurn("hi");+ reply("a1", "Quiet answer.");+ await new Promise((r) => setTimeout(r, 20));+ expect(synth.speak).not.toHaveBeenCalled();+ unmount();+ render(<Harness initial={withVoice(true, false)} />);+ expect(screen.getByRole("button", { name: "Unmute spoken replies" })).toBeInTheDocument();+ });++ it("cuts the speech when the user presses Stop or starts to talk", async () => {+ render(<Harness initial={withVoice(true, false)} />);+ await voiceTurn("hi");+ reply("a1", "A long answer to read.");+ await waitFor(() => expect(synth.speak).toHaveBeenCalled());+ await userEvent.click(screen.getByRole("button", { name: "Stop" }));+ expect(synth.cancel).toHaveBeenCalled();+ expect(sentOf("abort")).toEqual([]);++ await voiceTurn("again");+ reply("a2", "Another answer.");+ await waitFor(() => expect(synth.speak).toHaveBeenCalledTimes(2));+ synth.cancel.mockClear();+ await userEvent.click(talk());+ expect(synth.cancel).toHaveBeenCalled();+ });+});++describe("the composer with the mic present", () => {+ it("still sends typed messages, steers while busy, and stops the run as before", async () => {+ render(<Harness initial={withVoice(true)} />);+ const box = screen.getByRole("textbox", { name: "Message" });+ await userEvent.type(box, "first{Enter}");+ expect(sentOf("send")).toEqual([{ type: "send", text: "first", requestId: expect.any(String) }]);+ server({ type: "agent_state", state: "working" });+ expect(box).toHaveAttribute("placeholder", "Add to what I'm doing…");+ await userEvent.type(box, "more{Enter}");+ expect(sentOf("send").map((m) => m.text)).toEqual(["first", "more"]);+ await userEvent.click(screen.getByRole("button", { name: "Stop" }));+ expect(sentOf("abort")).toEqual([{ type: "abort" }]);+ });+});