Cursor plugin specification and official plugins
TypeScript
feat(grok-voice): add Grok Voice plugin (add-voice, add-dictation, add-read-aloud, debug-voice) (#337)
Eric Zakariasson2b8ae2ee306f
diff --git a/.cursor-plugin/marketplace.json b/.cursor-plugin/marketplace.jsonindex 7eb2de2f..d7e5a268 100644--- a/.cursor-plugin/marketplace.json+++ b/.cursor-plugin/marketplace.json@@ -73,6 +73,11 @@"source": "pstack","description": "if you want to go fast, go deep first. pstack helps you write less, but higher quality code. rigorous agent workflows you can parallelize with confidence."},+ {+ "name": "grok-voice",+ "source": "grok-voice",+ "description": "Add Grok voice to an app: realtime speech-to-speech, speech-to-text dictation, text-to-speech read-aloud, and a log-driven fix loop for voice sessions."+ },{"name": "advisor","source": "advisor",diff --git a/README.md b/README.mdindex 5408d433..758e4e97 100644--- a/README.md+++ b/README.md@@ -7,19 +7,20 @@ Official Cursor plugins for popular developer tools, frameworks, and SaaS produc| `name` | Plugin | Author | Category | `description` (from marketplace) ||:-------|:-------|:-------|:---------|:-------------------------------------|| `teaching` | [Teaching](teaching/) | Cursor | Utilities | Skill mapping, practice plans, and learning retrospectives. |-| `continual-learning` | [Continual Learning](continual-learning/) | Cursor | Developer Tools | Incremental transcript-driven memory updates for AGENTS.md using high-signal bullet points only. |-| `cursor-team-kit` | [Cursor Team Kit](cursor-team-kit/) | Cursor | Developer Tools | Internal team workflows for CI, code review, shipping, local automation, and verification. |+| `continual-learning` | [Continual Learning](continual-learning/) | Eric Zakariasson | Developer Tools | Incremental transcript-driven memory updates for AGENTS.md using high-signal bullet points only. |+| `cursor-team-kit` | [Cursor Team Kit](cursor-team-kit/) | Eric Zakariasson | Developer Tools | Internal team workflows for CI, code review, shipping, local automation, and verification. || `thermos` | [Thermos](thermos/) | Cursor | Developer Tools | Thermo-nuclear branch review: deep security/correctness audits, harsh code-quality rubrics, parallel subagents, thermos orchestration, and optional merge-ready PR flows. || `create-plugin` | [Create Plugin](create-plugin/) | Cursor | Developer Tools | Scaffold and validate new agent plugins. || `ralph-loop` | [Ralph Loop](ralph-loop/) | Cursor | Developer Tools | Iterative self-referential AI loops using the Ralph Wiggum technique. || `agent-compatibility` | [Agent Compatibility](agent-compatibility/) | Cursor | Developer Tools | CLI-backed repo compatibility scans plus agents that audit startup, validation, and docs against reality. |-| `cli-for-agent` | [CLI for Agents](cli-for-agent/) | Cursor | Developer Tools | Patterns for designing CLIs that coding agents can run reliably: flags, help with examples, pipelines, errors, idempotency, dry-run. |+| `cli-for-agent` | [CLI for Agents](cli-for-agent/) | Eric Zakariasson | Developer Tools | Patterns for designing CLIs that coding agents can run reliably: flags, help with examples, pipelines, errors, idempotency, dry-run. || `pr-review-canvas` | [PR Review Canvas](pr-review-canvas/) | Cursor | Developer Tools | Render PR diffs as review canvases grouped by importance. || `docs-canvas` | [Docs Canvas](docs-canvas/) | Cursor | Developer Tools | Render documentation as a navigable canvas. || `cursor-sdk` | [Cursor SDK](cursor-sdk/) | Cursor | Developer Tools | Build apps, scripts, and automations with the TypeScript SDK. || `orchestrate` | [Orchestrate](orchestrate/) | Cursor | Developer Tools | Fan large tasks out across parallel cloud agents with planners, workers, verifiers, and structured handoffs. || `pstack` | [pstack](pstack/) | Lauren Tan | Developer Tools | if you want to go fast, go deep first. pstack helps you write less, but higher quality code. rigorous agent workflows you can parallelize with confidence. || `advisor` | [Advisor](advisor/) | Cursor | Developer Tools | Consult a stronger model before major decisions, when stuck, and before declaring done. |+| `grok-voice` | [Grok Voice](grok-voice/) | Eric Zakariasson | Developer Tools | Add Grok voice to an app: realtime speech-to-speech, speech-to-text dictation, text-to-speech read-aloud, and a log-driven fix loop for voice sessions. || `gmail` | [Gmail](third_party/gmail/) | Cursor | Productivity | Search, read, draft, and manage email. || `google-drive` | [Google Drive](third_party/google-drive/) | Cursor | Productivity | Search, read, create, and share files. || `google-calendar` | [Google Calendar](third_party/google-calendar/) | Cursor | Productivity | Search events and schedule meetings. |diff --git a/cli-for-agent/.cursor-plugin/plugin.json b/cli-for-agent/.cursor-plugin/plugin.jsonindex c0d3a06a..a03b6e8d 100644--- a/cli-for-agent/.cursor-plugin/plugin.json+++ b/cli-for-agent/.cursor-plugin/plugin.json@@ -4,8 +4,7 @@"version": "1.0.0","description": "Patterns for designing CLIs that coding agents can run reliably: non-interactive flags, layered help with examples, pipelines, actionable errors, idempotency, and dry-run.","author": {- "name": "Cursor",- "email": "[email protected]"+ "name": "Eric Zakariasson"},"homepage": "https://github.com/cursor/plugins/tree/main/cli-for-agent","repository": "https://github.com/cursor/plugins",diff --git a/continual-learning/.cursor-plugin/plugin.json b/continual-learning/.cursor-plugin/plugin.jsonindex 8382ba9a..3255db70 100644--- a/continual-learning/.cursor-plugin/plugin.json+++ b/continual-learning/.cursor-plugin/plugin.json@@ -4,8 +4,7 @@"version": "1.0.0","description": "Incrementally learns durable user preferences and workspace facts from transcript changes and keeps AGENTS.md up to date with plain bullet points.","author": {- "name": "Cursor",- "email": "[email protected]"+ "name": "Eric Zakariasson"},"homepage": "https://github.com/cursor/plugins","repository": "https://github.com/cursor/plugins",diff --git a/cursor-team-kit/.cursor-plugin/plugin.json b/cursor-team-kit/.cursor-plugin/plugin.jsonindex b642937d..23a67962 100644--- a/cursor-team-kit/.cursor-plugin/plugin.json+++ b/cursor-team-kit/.cursor-plugin/plugin.json@@ -4,8 +4,7 @@"version": "1.2.0","description": "Internal engineering team workflows for CI, code review, shipping, control-cli, control-ui, verify-this, test reliability, code cleanup, and work summaries. Designed to work without requiring third-party service integrations.","author": {- "name": "Cursor",- "email": "[email protected]"+ "name": "Eric Zakariasson"},"homepage": "https://github.com/cursor/plugins","repository": "https://github.com/cursor/plugins",diff --git a/grok-voice/.cursor-plugin/plugin.json b/grok-voice/.cursor-plugin/plugin.jsonnew file mode 100644index 00000000..fff49d04--- /dev/null+++ b/grok-voice/.cursor-plugin/plugin.json@@ -0,0 +1,32 @@+{+ "name": "grok-voice",+ "displayName": "Grok Voice",+ "version": "0.1.0",+ "description": "Add Grok voice to an app: realtime speech-to-speech, speech-to-text dictation, text-to-speech read-aloud, and a log-driven fix loop for voice sessions.",+ "author": {+ "name": "Eric Zakariasson"+ },+ "homepage": "https://github.com/cursor/plugins/tree/main/grok-voice",+ "repository": "https://github.com/cursor/plugins",+ "license": "MIT",+ "category": "developer-tools",+ "keywords": [+ "grok",+ "xai",+ "voice",+ "realtime",+ "speech-to-speech",+ "speech-to-text",+ "text-to-speech",+ "dictation",+ "tts",+ "stt"+ ],+ "tags": [+ "voice",+ "grok",+ "ai",+ "developer-tools"+ ],+ "skills": "./skills/"+}diff --git a/grok-voice/LICENSE b/grok-voice/LICENSEnew file mode 100644index 00000000..ca2bba77--- /dev/null+++ b/grok-voice/LICENSE@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2026 Cursor++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/grok-voice/README.md b/grok-voice/README.mdnew file mode 100644index 00000000..ae939b8a--- /dev/null+++ b/grok-voice/README.md@@ -0,0 +1,16 @@+# Grok Voice++Cursor plugin with four skills for building voice features on Grok: realtime speech-to-speech, speech-to-text dictation, text-to-speech read-aloud, and a debug loop for voice sessions.++## What it includes++- `add-voice`: build speech-to-speech into an app, or replace an STT-LLM-TTS cascade / OpenAI Realtime with it. Wires the realtime session, safe auth, and the app mic; adds a waveform Voice Mode button to the composer.+- `add-dictation`: speech-to-text. Batch for a mic button or recorded audio (files, uploads, URLs) with word timestamps, diarization, and subtitles; streaming through a relay for live text.+- `add-read-aloud`: text-to-speech. Batch MP3 for a speaker button on replies; streaming PCM through a relay so audio starts before the reply finishes.+- `debug-voice`: proposes a plan, installs a dev-only log pipeline (client logger → local NDJSON) in the app's own language and conventions, then runs the fix loop: match the user's report to log signatures, fix one thing, re-test.++Icon convention across the skills: waveform = voice mode, microphone = dictation, speaker = read aloud.++## When to use it++Use when a user asks to add voice, dictation, or read-aloud to an app with Grok, runs `/add-voice`, `/add-dictation`, `/add-read-aloud`, or `/debug-voice`, or reports that voice mode misbehaves and needs logs to diagnose it.diff --git a/grok-voice/skills/add-dictation/SKILL.md b/grok-voice/skills/add-dictation/SKILL.mdnew file mode 100644index 00000000..6d42d06f--- /dev/null+++ b/grok-voice/skills/add-dictation/SKILL.md@@ -0,0 +1,164 @@+---+name: add-dictation+description: >-+ Use when the user runs /add-dictation or wants speech turned into text with+ Grok speech-to-text: a mic button that dictates into the composer, live+ captions, or transcribing recorded audio (files, uploads, URLs) with word+ timestamps, diarization, subtitles, meeting notes. STT, transcribe,+ transcription. For a voice agent that talks back use /add-voice.+---++# Add Dictation++Add Grok Speech to Text to an existing app: a mic button that dictates into the composer, live captions, or transcripts of recorded audio. Run on `/add-dictation`, typed **Dictate**, or clear “transcribe” intent. Cursor has no mic; wire the **app**, not the IDE.++## Docs++- https://docs.x.ai/developers/model-capabilities/audio/speech-to-text+- Pricing (cite docs only): https://docs.x.ai/developers/pricing++## Pick the path++| Need | Path |+| --- | --- |+| Tap, speak, tap, text appears. Uploaded files. URLs. | **Batch** `POST https://api.x.ai/v1/stt` (default) |+| Text appears while speaking: captions, long dictation, push-to-talk | **Streaming** `wss://api.x.ai/v1/stt` through a backend relay |++Batch is the default for a composer mic button: one request, no socket, the key never leaves the server. Go streaming only when the UX needs interim text.++## Auth++- Bearer `XAI_API_KEY`, server side only. The STT docs document no ephemeral-token flow, and browsers cannot set WebSocket headers, so browser streaming goes through your backend relay. Do not invent a token flow.+- Never put the key in a client bundle. Do not paste keys in chat.++## Steps++1. **Map the app**+ - Composer or input component, where the text should land (insert at cursor vs replace), server framework, package manager.+ - The **microphone icon belongs to dictation**. If `/add-voice` is installed, its waveform primary button stays as is; add the mic as a secondary ghost button beside it.+ - Existing mic capture? If `/add-voice` ran, its PCM capture can feed streaming STT; pass its rate as `sample_rate`. 16 kHz is the model’s native rate; other supported rates (8000, 16000, 22050, 24000, 44100, 48000) are resampled server side.++2. **Batch path (default)**+ - Client: `MediaRecorder` → `Blob` → `POST` to your own route. The endpoint auto-detects containers (WAV, MP3, OGG, Opus, FLAC, AAC, MP4, M4A, MKV, WebM), so send whatever `MediaRecorder` produces.+ - Server: forward as `multipart/form-data`. Option fields first, **`file` last**; fields after `file` may be ignored. `file` or `url`, max 500 MB.++```ts+// server (any runtime with fetch + FormData)+export async function transcribe(blob: Blob, filename: string) {+ const form = new FormData();+ form.append("format", "true"); // written-form numbers/currency; requires language+ form.append("language", "en");+ // form.append("keyterm", "Acme"); // repeat per term, ≤100 terms × 50 chars+ form.append("file", blob, filename); // last+ const res = await fetch("https://api.x.ai/v1/stt", {+ method: "POST",+ headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },+ body: form,+ });+ if (!res.ok) throw new Error(`STT ${res.status}`); // 400 bad input, 413 >500 MB, 429 back off, 502 url fetch failed, 503 retry+ return (await res.json()) as {+ text: string; language: string; duration: number;+ words?: { text: string; start: number; end: number; speaker?: number }[];+ channels?: { index: number; text: string; words: unknown[] }[];+ };+}+```++```ts+// client+const mime = MediaRecorder.isTypeSupported("audio/webm;codecs=opus") ? "audio/webm;codecs=opus" : "audio/mp4";+const rec = new MediaRecorder(stream, { mimeType: mime });+const parts: BlobPart[] = [];+rec.ondataavailable = (e) => parts.push(e.data);+rec.onstop = async () => {+ const fd = new FormData();+ fd.append("file", new Blob(parts, { type: mime }), "dictation");+ const { text } = await (await fetch("/api/dictation", { method: "POST", body: fd })).json();+ insertAtCursor(text);+};+rec.start(); // second tap: rec.stop()+```++3. **Streaming path**+ - Relay: server holds the key, upgrades the browser socket, forwards binary frames and client control messages up, JSON events down. Build the query string server side.++```ts+import { WebSocketServer, WebSocket } from "ws";++new WebSocketServer({ port: 8788 }).on("connection", (client) => {+ const q = new URLSearchParams({ sample_rate: "16000", encoding: "pcm", interim_results: "true", language: "en" });+ const up = new WebSocket(`wss://api.x.ai/v1/stt?${q}`, { headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` } });+ up.on("message", (d) => client.send(d.toString())); // transcript.* and error events+ client.on("message", (d, isBinary) => up.readyState === WebSocket.OPEN && up.send(d, { binary: isBinary })); // audio + finalize/audio.done+ const end = () => { client.close(); up.close(); };+ up.on("close", end); up.on("error", end); client.on("close", end);+});+```++ - Browser capture: PCM16 little-endian, mono, 16 kHz, **100 ms frames = 3,200 bytes**, raw binary, no base64. Wait for `transcript.created` before sending. `MediaRecorder` output is a container, not raw frames; do not stream it.++```ts+const ws = new WebSocket(relayUrl); ws.binaryType = "arraybuffer";+const ctx = new AudioContext({ sampleRate: 16000 }); // if ctx.sampleRate !== 16000, downsample in the worklet+await ctx.audioWorklet.addModule("/pcm16-worklet.js"); // Float32 → Int16LE, posts one 3,200-byte frame per 100 ms+const node = new AudioWorkletNode(ctx, "pcm16");+ctx.createMediaStreamSource(stream).connect(node);+let ready = false;+node.port.onmessage = (e) => ready && ws.readyState === WebSocket.OPEN && ws.send(e.data);++let committed = "", locked = "", live = "";+ws.addEventListener("message", (e) => {+ const ev = JSON.parse(e.data);+ if (ev.type === "transcript.created") ready = true;+ else if (ev.type === "transcript.partial") {+ if (ev.speech_final) { committed += ev.text + " "; locked = ""; live = ""; } // complete stitched utterance+ else if (ev.is_final) { locked += ev.text + " "; live = ""; } // chunk final: text will not change+ else live = ev.text; // interim: may change+ render(committed + locked + live);+ } else if (ev.type === "transcript.done") ws.close(); // after audio.done+ else if (ev.type === "error") showError(ev.message); // most errors close the socket+});+// stop: ws.send(JSON.stringify({ type: "audio.done" }))+// push-to-talk release: ws.send(JSON.stringify({ type: "Finalize" })) then keep streaming (docs show both `finalize` and `Finalize`; the examples use `Finalize`)+```++4. **Options** (query params for streaming, form fields for batch)++| Want | Set |+| --- | --- |+| Text while speaking | `interim_results=true` |+| “one hundred dollars” → `$100` | streaming: `language=en`; batch: `format=true` + `language=en` |+| Product names, jargon | `keyterm=` repeated |+| Not cut off mid-sentence while dictating numbers | `smart_turn=0.7&smart_turn_timeout=3000` |+| Faster or slower end of utterance | `endpointing=` ms, default 400 |+| Who said what (meetings) | `diarize=true` → `words[].speaker` |+| Agent and customer on separate channels | `multichannel=true&channels=2` (PCM only, not Opus) |+| Keep “um”, “uh” | `filler_words=true` (removed by default) |+| Low bandwidth or mobile | `encoding=opus`, exactly one raw Opus packet per frame, omit `sample_rate` |+| Raw audio to batch | `audio_format=pcm|mulaw|alaw` + `sample_rate` |+| Quiet or telephony audio | lower `vad_threshold` (streaming default 0.08, batch 0.5) |++5. **Python twin (only if the server is Python)**++```python+import os, requests+r = requests.post(+ "https://api.x.ai/v1/stt",+ headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},+ data=[("format", "true"), ("language", "en")],+ files={"file": ("dictation.webm", blob, "audio/webm")}, # requests sends data fields before files+)+r.raise_for_status(); text = r.json()["text"]+# streaming: websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}); await ws.send(pcm_bytes)+```++6. **Smoke**+ - Batch: `curl -X POST https://api.x.ai/v1/stt -H "Authorization: Bearer $XAI_API_KEY" -F language=en -F [email protected]` → 200 with `text`. Same call with `-F format=true` and no `language` → 400.+ - Streaming: dictate two sentences with a pause between them. Expect interim text, then a final; no duplicated or vanished words at the utterance boundary (if words vanish, the stitched `speech_final` text did not include the chunk finals: append instead of replacing `locked`). `audio.done` → `transcript.done`, socket closes.+ - Search the client bundle for `XAI_API_KEY`; it must not be there.+ - Debug from logs with `/debug-voice`; swap its hook points to `transcript.*` events.++## Out of scope++- Speech that talks back (`/add-voice`), speaking text (`/add-read-aloud`)+- Inventing an STT token flow, endpoints, or event names not in the docsdiff --git a/grok-voice/skills/add-read-aloud/SKILL.md b/grok-voice/skills/add-read-aloud/SKILL.mdnew file mode 100644index 00000000..1fdb96d9--- /dev/null+++ b/grok-voice/skills/add-read-aloud/SKILL.md@@ -0,0 +1,183 @@+---+name: add-read-aloud+description: >-+ Use when the user runs /add-read-aloud or wants the app to speak text with+ Grok text-to-speech: read-aloud button on assistant replies, auto-speak, TTS,+ voice output, narration, IVR prompts, speech tags, voice_id. For a two-way+ voice agent use /add-voice. For speech-to-text use /add-dictation.+---++# Add Read Aloud++Add Grok Text to Speech to an existing app: a speaker button on assistant replies, auto-speak, or narration of any text. Run on `/add-read-aloud`, typed **Read aloud**, or clear “speak this” / “TTS” intent. Cursor has no speaker; wire the **app**, not the IDE.++## Docs++- https://docs.x.ai/developers/model-capabilities/audio/text-to-speech+- API reference: https://docs.x.ai/developers/rest-api-reference/inference/voice+- Custom voices: https://docs.x.ai/developers/model-capabilities/audio/custom-voices+- Pricing (cite docs only): https://docs.x.ai/developers/pricing++## Pick the path++| Need | Path |+| --- | --- |+| Tap speaker, hear the finished reply. Narrate a page. Generate a file. | **Batch** `POST https://api.x.ai/v1/tts` (default) |+| Audio starts while the LLM is still streaming; barge-in; texts over 15,000 chars | **Streaming** `wss://api.x.ai/v1/tts` through a backend relay |++Batch is the default for a read-aloud button: one request, one MP3, cacheable, the key never leaves the server. Go streaming only when the UX needs audio before the text is complete. `POST /v1/tts` has no documented streaming flag; do not invent one.++## Auth++- Bearer `XAI_API_KEY`, server side only. The TTS docs document no ephemeral-token flow, and browsers cannot set WebSocket headers, so browser streaming goes through your backend relay.+- Never put the key in a client bundle. Do not paste keys in chat.++## Steps++1. **Map the app**+ - Where assistant messages render, where per-message actions live (copy, regenerate), how the reply stream ends, server framework, package manager.+ - The **speaker icon belongs to read aloud**. Waveform is voice mode (`/add-voice`), microphone is dictation (`/add-dictation`). Put a ghost speaker button in the message action row; loading shows a spinner, playing shows a stop square. One utterance at a time: starting a new one stops the current one.+ - Align the action row to the reply’s **text edge**, not the button’s box: an icon button centres its glyph, so if the assistant bubble has no padding pull the row left by that inset (e.g. `-ml-[5px]` for a 12–14 px icon in a 24 px button). Measure in the browser; `getBoundingClientRect` on the `<p>` and the `<svg>` should share a left edge.+ - Render the button only once the reply has **finished streaming**; on a live message it would speak a partial reply.+ - With many messages on screen, keep player state (active message id, `loading | playing`, last error) in one shared store (`useSyncExternalStore`, a signal, whatever the app uses) so every button reflects it and errors can surface in the app's existing status area. A per-button `let current` is not enough.+ - Auto-speak: opt-in toggle, off by default, and only after a user gesture on the page (autoplay policy). Never auto-speak on load.+ - If `/add-voice` is installed, its `AudioContext` and PCM player can play streaming TTS; do not add a second audio graph.++2. **Prepare the text**+ - Speak prose, not markup. Strip markdown: headings → text, `**bold**` → text, links → link text, inline code → the code, fenced blocks → `[pause] Code block omitted.`, tables → one sentence per row or omit. Keep punctuation; it drives pacing.+ - Neutralise speech tags that arrive inside the reply (`[laugh]`, `<whisper>`…) so the model’s text cannot steer delivery. Strip only the **documented tag names** (list in step 5), not every bracket: `[1]` citations and `[note]` must survive.+ - Batch limit is **15,000 characters per request**. Split longer text on paragraph, then sentence, then word boundaries and play the parts in order; fetch part N+1 while N plays or there is a silent gap at every boundary. Or use streaming.+ - Cache by `hash(text + voice_id + language + speed)`; the same reply is often replayed.++3. **Batch path (default)**+ - Server: your route takes `{ text, voice_id?, language? }`, validates the shape of each (`voice_id` `^[a-z0-9-]{1,64}$`, `language` BCP-47 or `auto`), forwards JSON, streams the body back with the upstream `Content-Type` and `Cache-Control: no-store`. Map upstream 404 to “unknown voice” so the client gets a readable error. Default output is MP3 at 24 kHz / 128 kbps, playable everywhere in the browser.+ - `language`: default to `"auto"` for a chat app, where replies follow the user’s language; pin `"en"` etc. only for fixed-language products.++```ts+// server (any runtime with fetch)+export async function speak(text: string, voice_id = "eve", language = "auto") {+ if (!text.trim() || text.length > 15_000) throw new Error("TTS text must be 1–15,000 chars");+ const res = await fetch("https://api.x.ai/v1/tts", {+ method: "POST",+ headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}`, "Content-Type": "application/json" },+ body: JSON.stringify({+ text,+ voice_id,+ language, // required: `auto` or BCP-47 (`en`, `pt-BR`); omitting it → 422+ // output_format: { codec: "mp3", sample_rate: 24000, bit_rate: 128000 }, // default+ // speed: 1.0, // 0.7–1.5+ // text_normalization: true, // "$5" → "five dollars"+ // replace: { nginx: "/ˈɛndʒɪn ˈɛks/" },+ }),+ });+ if (!res.ok) throw new Error(`TTS ${res.status}`); // 400 bad text/format, 401 key, 404 unknown voice_id, 422 missing required field (e.g. language), 429/500/503 back off and retry+ return new Response(res.body, { headers: { "Content-Type": res.headers.get("content-type") ?? "audio/mpeg" } });+}+```++```ts+// client+let current: HTMLAudioElement | null = null;+async function readAloud(text: string, voiceId = "eve") {+ current?.pause(); current = null; // one utterance at a time+ const res = await fetch("/api/tts", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text, voice_id: voiceId }) });+ if (!res.ok) throw new Error("TTS request failed");+ const url = URL.createObjectURL(await res.blob());+ const audio = new Audio(url);+ audio.addEventListener("ended", () => URL.revokeObjectURL(url)); // avoid blob leaks+ current = audio;+ await audio.play(); // call from the click handler’s promise chain+}+function stop() { current?.pause(); current = null; }+```++ - Safari: `audio.duration` is `Infinity` on blob URLs. If you need a progress bar, decode with `AudioContext.decodeAudioData(buf.slice(0))` or request `with_timestamps: true` and read `duration` from the JSON envelope (audio is then base64 in `audio`).+ - Safari suspends an `AudioContext` created outside a gesture for good. Create it synchronously in the click handler, before any `await`.++4. **Streaming path**+ - Relay: server holds the key, upgrades the browser socket, builds the query string, forwards client JSON up and server JSON down. Request `codec=pcm` for the browser: raw PCM16 chunks can be scheduled as they arrive, while MP3 chunks cannot be decoded piecemeal without `MediaSource`.++```ts+import { WebSocketServer, WebSocket } from "ws";++new WebSocketServer({ port: 8789 }).on("connection", (client) => {+ const q = new URLSearchParams({ language: "en", voice: "eve", codec: "pcm", sample_rate: "24000" /* optimize_streaming_latency: "1" */ });+ const up = new WebSocket(`wss://api.x.ai/v1/tts?${q}`, { headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` } });+ up.on("message", (d) => client.send(d.toString())); // audio.delta, audio.done, audio.clear, session.updated, error+ client.on("message", (d) => up.readyState === WebSocket.OPEN && up.send(d.toString())); // text.delta, text.done, text.clear, session.update+ const end = () => { client.close(); up.close(); };+ up.on("close", end); up.on("error", end); client.on("close", end);+});+```++ - Client: one socket per chat session (it stays open across utterances; 50 concurrent sessions per team, permit TTL 600 s, so reconnect on close). Forward LLM tokens as `text.delta` (each ≤ 15,000 chars), send `text.done` when the reply finishes, `text.clear` on stop or when a new reply starts; drop queued audio on `audio.clear`.++```ts+const ws = new WebSocket(relayUrl);+const ctx = new AudioContext({ sampleRate: 24000 }); // create in the click handler; resume if suspended+let playhead = 0, sources: AudioBufferSourceNode[] = [];++ws.addEventListener("message", (e) => {+ const ev = JSON.parse(e.data);+ if (ev.type === "audio.delta") {+ const bytes = Uint8Array.from(atob(ev.delta), (c) => c.charCodeAt(0));+ const pcm = new Int16Array(bytes.buffer, 0, bytes.byteLength >> 1);+ const buf = ctx.createBuffer(1, pcm.length, 24000);+ const ch = buf.getChannelData(0);+ for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;+ const src = ctx.createBufferSource(); src.buffer = buf; src.connect(ctx.destination);+ playhead = Math.max(playhead, ctx.currentTime + 0.15); // ~150 ms lead so chunks butt together+ src.start(playhead); playhead += buf.duration; sources.push(src);+ } else if (ev.type === "audio.done") { /* utterance finished; socket stays open */ }+ else if (ev.type === "audio.clear") { sources.forEach((s) => s.stop()); sources = []; playhead = 0; }+ else if (ev.type === "error") showError(ev.message);+});+// on each LLM token: ws.send(JSON.stringify({ type: "text.delta", delta: token }))+// on reply finished: ws.send(JSON.stringify({ type: "text.done" }))+// on stop / barge-in: ws.send(JSON.stringify({ type: "text.clear" })) → wait for audio.clear before the next text.delta+```++ - Words split across `text.delta` boundaries are fine; matching and synthesis run across deltas.++5. **Options** (JSON fields for batch, query params for streaming)++| Want | Set |+| --- | --- |+| Different voice | `voice_id` (batch) / `voice` (streaming). Built-ins from `GET /v1/tts/voices`: `eve` (default), `ara`, `rex`, `leo`, `luna`, `atlas`, `aurora`, `orion`, … 28 total, all multilingual, case-insensitive. Custom voice: 8-char id from the console or `GET /v1/custom-voices` |+| Non-English or mixed | `language`: `en`, `ar-EG`, `ar-SA`, `ar-AE`, `bn`, `zh`, `fr`, `de`, `hi`, `id`, `it`, `ja`, `ko`, `pt-BR`, `pt-PT`, `ru`, `es-MX`, `es-ES`, `tr`, `vi`, or `auto` |+| Faster or slower | `speed` 0.7–1.5 |+| “$5”, “Dr.”, “3/4” spoken as words | `text_normalization: true` |+| Brand names, acronyms, jargon | `replace: { "Acme Mobile": "Acme Mobull", "nginx": "/ˈɛndʒɪn ˈɛks/" }`; ≤200 entries, keys ≤100 chars (letters, digits, apostrophes, spaces), values ≤128; whole-word, case-insensitive, longest match wins. Streaming: `session.update { replace }` before the first `text.delta` |+| Expressive delivery | Inline `[pause]`, `[long-pause]`, `[laugh]`, `[chuckle]`, `[giggle]`, `[cry]`, `[sigh]`, `[breath]`, `[inhale]`, `[exhale]`, `[tsk]`, `[tongue-click]`, `[lip-smack]`, `[hum-tune]`. Wrapping `<whisper>`, `<soft>`, `<loud>`, `<emphasis>`, `<build-intensity>`, `<decrease-intensity>`, `<slow>`, `<fast>`, `<higher-pitch>`, `<lower-pitch>`, `<singing>`, `<sing-song>` around whole phrases |+| Captions, karaoke, lip-sync | `with_timestamps: true` → JSON `{ audio (base64), content_type, duration, audio_timestamps: { graph_chars[], graph_times[][start,end] } }`; step through `graph_chars` in order, never slice input by index |+| First audio sooner (streaming) | `optimize_streaming_latency=1` (docs also list `2`; API reference lists `0`/`1`) |+| Telephony / IVR | `output_format: { codec: "mulaw" \| "alaw", sample_rate: 8000 }`; not playable in browsers |+| Editing, post-production | `codec: "wav"`, `sample_rate: 44100` or `48000` |+| Smaller files | `codec: "mp3"`, `bit_rate: 64000` |++6. **Python twin (only if the server is Python)**++```python+import os, requests+r = requests.post(+ "https://api.x.ai/v1/tts",+ headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},+ json={"text": text, "voice_id": "eve", "language": "en"},+)+r.raise_for_status(); audio_bytes = r.content # audio/mpeg+# streaming: websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}); send {"type":"text.delta",...}, {"type":"text.done"}; read audio.delta / audio.done+```++7. **Smoke**+ - Batch: `curl -X POST https://api.x.ai/v1/tts -H "Authorization: Bearer $XAI_API_KEY" -H "Content-Type: application/json" -d '{"text":"Hello from read aloud.","voice_id":"eve","language":"en"}' --output /tmp/hello.mp3` → 200 `audio/mpeg` (MP3, 24 kHz, 128 kbps, mono), plays. Omit `language` → 422 (observed; the docs’ table only lists 400). `voice_id: "nope"` → 404.+ - In the app: no speaker while a reply streams; it appears when the reply ends. Click → spinner → stop square → back to speaker when audio ends; click again mid-play → stops at once; click a second reply while the first plays → first stops, second plays. Safari: first play works from the click, `URL.revokeObjectURL` fires on `ended`.+ - Text prep: a reply with a fenced block, a table, a `[1]` citation and a stray `[laugh]` → spoken as “Code block omitted”, one sentence per row, the citation intact, the tag gone. Unit-test this; it is pure.+ - Streaming: send a two-sentence reply token by token; audio should start before `text.done`. Send `text.clear` mid-utterance → `audio.clear`, playback stops with nothing stale. Second utterance on the same socket → fresh `audio.delta`s, no bleed from the first.+ - Search the client bundle for `XAI_API_KEY`; it must not be there. Against a running dev server, fetch `/`, collect the `/_next/static/chunks/*.js` (or equivalent) URLs it references, and grep each; do not rely on a production build you have not made.+ - Debug from logs with `/debug-voice`; swap its hook points to `audio.delta` (byte counts), `audio.done`, `audio.clear`, `error`.++## Out of scope++- A voice agent that listens and answers (`/add-voice`), speech to text (`/add-dictation`)+- Voice cloning beyond passing an existing custom `voice_id`+- Inventing a TTS token flow, a `stream` flag on `POST /v1/tts`, or event names not in the docsdiff --git a/grok-voice/skills/add-voice/SKILL.md b/grok-voice/skills/add-voice/SKILL.mdnew file mode 100644index 00000000..aee71871--- /dev/null+++ b/grok-voice/skills/add-voice/SKILL.md@@ -0,0 +1,123 @@+---+name: add-voice+description: >-+ Use when the user runs /add-voice, types Voice Mode, or asks to add Grok+ realtime voice to an app, including replacing an STT-LLM-TTS cascade or+ OpenAI Realtime. Wire speech-to-speech, safe auth, and app mic. Composer: waveform button,+ mic icon reserved for dictation. For mic-to-text only use /add-dictation; to speak text replies use+ /add-read-aloud. To add debug logging and fix from logs use /debug-voice.+---++# Add Voice++Add Grok Speech to Speech to an existing app. Run on `/add-voice`, typed **Voice Mode**, or clear “add Grok voice” intent.++## Goal++Working duplex path: user-app mic in, audio out, `wss://api.x.ai/v1/realtime?model=grok-voice-latest`, safe auth. Cursor has no native mic; wire the **app** (or a sample client), not the IDE.++## Protocol first++Language-agnostic event loop. **TypeScript samples default.** Short Python twins only where the client API differs (e.g. `ws` vs `websockets`).++## Docs++- https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech+- https://docs.x.ai/developers/model-capabilities/audio/ephemeral-tokens+- Pricing (cite docs only): https://docs.x.ai/developers/pricing (~$0.08/min STS + $0.004/text item; max session 120 min)++## Steps++1. **Map the app**+ - Stack: none, OpenAI Realtime, STT→LLM→TTS cascade, TTS/STT only.+ - Client: web / Node / iOS / Android / server.+ - If a cascade or OpenAI Realtime exists: replace it with the single duplex loop below (URL, model, voice, event diffs); keep standalone `/v1/stt` or `/v1/tts` only if the product still needs one-shot listen or speak outside the agent.++2. **Auth**+ - Server: Bearer `XAI_API_KEY`.+ - Browser/mobile: backend `POST https://api.x.ai/v1/realtime/client_secrets`, client uses ephemeral token (Bearer or browser `sec-websocket-protocol`: `xai-client-secret.<token>`).+ - Never put a long-lived key in client bundles. Do not paste keys in chat.++3. **Connect + session**+ - URL: `wss://api.x.ai/v1/realtime?model=grok-voice-latest`+ - On open: `session.update` with `voice` (default `eve`), `instructions`, `turn_detection: { type: "server_vad" }` (or `null` for push-to-talk), PCM 24 kHz unless the app already standardizes elsewhere.+ - Set `audio.input.transcription.model: "grok-transcribe"` or no user transcript arrives (`conversation.item.input_audio_transcription.updated` is cumulative, not a delta).+ - Tools if needed: `web_search`, `x_search`, `file_search`, `mcp`, custom `function`.++4. **Audio I/O (app-side)**+ - One `AudioContext` per session for capture and playback, created inside the user gesture (autoplay policy). Ask for 24 kHz; if the browser gives another rate, resample before sending.+ - Mic → AudioWorklet in ~100 ms chunks → `input_audio_buffer.append` (or binary transport). Start WS and mic in parallel; buffer early audio, flush on open.+ - Play `response.output_audio.delta` immediately; schedule with a ~150 ms lead so chunks butt together. On `input_audio_buffer.speech_started`, stop everything queued (barge-in).+ - Transcript rows: create the user row on `input_audio_buffer.committed` (`item_id`), fill it on `…transcription.updated`; assistant text from `response.output_audio_transcript.delta` / `.done`, close the turn on `response.done`.+ - On function tools: `function_call_output`, finish playback, then `response.create`.++5. **Composer UI convention**+ - One primary button, right side of the composer. Empty composer → **waveform** icon (stroked, e.g. Phosphor `WaveformIcon weight="bold"`; never the `fill` weight, which renders as a blob at 16 px), starts voice mode. Any text present → classic **send** arrow; in voice mode that text goes into the live session (`conversation.item.create` + `response.create`). Text reply streaming → stop square.+ - While voice is live the same button shows an **animated waveform** (4 bars, ~3 px wide, 2 px gap, ~16 px tall, min scale 0.4 so they stay legible in a 28 px button) and ends the session on click. Phase drives the animation: listening slow, speaking fast, connecting/thinking slower and slightly dimmed (opacity ≥ 0.75). Honor `prefers-reduced-motion`. No X button, no pulsing ring.+ - Status lives **in the composer, not around it**: the placeholder reads `Connecting…` / `Listening…` / `Thinking…` / `Speaking…`, plus an `sr-only` `role="status"`. No separate status row.+ - The **microphone icon is reserved for dictation** (`/add-dictation`). Never use it for voice mode.++6. **TS skeleton (default)**++```ts+const url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest";+// Node: pass Authorization header. Browser: use xai-client-secret.<token> protocol.+const ws = new WebSocket(url /* , { headers: { Authorization: `Bearer ${token}` } } */);++ws.addEventListener("open", () => {+ ws.send(JSON.stringify({+ type: "session.update",+ session: {+ voice: "eve",+ instructions: "You are a helpful voice agent.",+ turn_detection: { type: "server_vad" },+ },+ }));+});++ws.addEventListener("message", (ev) => {+ const event = JSON.parse(String(ev.data));+ if (event.type === "response.output_audio.delta") {+ // decode base64 PCM and play+ }+});+```++7. **Python twin (only if the app is Python)**++```python+import json, os, websockets++url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest"+headers = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}++async with websockets.connect(url, additional_headers=headers) as ws:+ await ws.send(json.dumps({+ "type": "session.update",+ "session": {+ "voice": "eve",+ "instructions": "You are a helpful voice agent.",+ "turn_detection": {"type": "server_vad"},+ },+ }))+ async for raw in ws:+ event = json.loads(raw)+ if event.get("type") == "response.output_audio.delta":+ pass # decode and play+```++8. **Instrument (before the first human test)**+ - Run `/debug-voice`: it proposes a plan, then installs a dev-only log sink (`POST /api/voice/log` → `.voice-logs/<sessionId>.ndjson`, gitignored), a client logger with audio reduced to byte counts, and the session id in the UI, in the app's own language.+ - The same skill carries the fix loop and the symptom → log signature → fix table.++9. **Smoke**+ - Text turn via `conversation.item.create` + `response.create`; confirm audio or transcript events.+ - Confirm no long-lived key in client (grep the built client bundle for the env name and `client_secrets`).+ - Hand the app to the user with **headphones**. On speakers the mic hears the reply and the model answers itself; that is echo, not a bug in the loop.+ - Iterate with `/debug-voice`.++## Out of scope++- Speech-to-text only (`/add-dictation`), speaking text (`/add-read-aloud`)+- Image generation and text-only inference+- Invented endpoints, events, or CLI flagsdiff --git a/grok-voice/skills/debug-voice/SKILL.md b/grok-voice/skills/debug-voice/SKILL.mdnew file mode 100644index 00000000..c7ee7570--- /dev/null+++ b/grok-voice/skills/debug-voice/SKILL.md@@ -0,0 +1,198 @@+---+name: debug-voice+description: >-+ Use when the user runs /debug-voice, says voice mode has flaws, asks to see or+ capture what happened in a Grok realtime voice session, or a voice integration+ has no debug logging yet. Proposes a plan, then installs a dev-only log+ pipeline (client logger → local NDJSON) in the app's own language and+ conventions, then runs the fix loop: match the user's report to log+ signatures, fix one thing, re-test.+---++# Debug Voice++Make a voice session readable after the fact, then fix from evidence. The agent cannot hear the app; the log is its ears, the user is its judge. No audio, no tokens, never in prod.++Works for any stack. The pipeline is a small contract (below); implement it in whatever the app already uses.++## Workflow++1. **Map** the app (read only).+2. **Plan**: write the change list, show it, **stop**. Nothing is edited until the user aligns.+3. **Install** the agreed pieces in the app's language, framework, and conventions.+4. **Verify** the sink, hand the app to the user.+5. **Fix loop** from the log.++## 1. Map++Find, and note the paths:++- Voice client: where realtime events are received and sent, mic capture, audio playback, token fetch.+- Server: framework, how routes are declared, where shared server code lives, how env is read, what "production" means here.+- Conventions: language(s), module system, formatter, where scripts or tasks live (`package.json`, `Makefile`, `pyproject`, `justfile`), `.gitignore`.+- Client kind: browser, mobile, desktop, CLI. A non-browser client still POSTs the same JSON; a single-process app can skip HTTP and append to the file directly.+- Where audio deltas are handled. They must be counted, never logged.++## 2. Plan, then stop++Fill this in with real paths and the app's language, post it, and wait for a yes or a trimmed list. Do not edit files before that.++```markdown+## Debug voice: plan++Add+- <path>: client logger (batch, redact, flush) in <language>+- <path>: dev-only sink `POST /api/voice/log` → `.voice-logs/<sessionId>.ndjson`+- <path> (optional): summary command `<cmd>`; otherwise read the NDJSON with jq++Modify+- <voice client file>: hook points start, token, mic, env, ws.*, client/server events,+ audio.in (2 s windows), audio.out.first, audio.out, play.stop, stop+- <token route>: append `server.token { ok, status, ms, upstream }` (never the token)+- <UI file>: session id in the voice status line and in voice error messages+- .gitignore: `/.voice-logs`+- <scripts file>: a `voice:logs` task (only if the summary command is wanted)++Logged: event names and non-audio fields, timings, byte counts, mic RMS.+Never: tokens, API keys, raw audio, strings over 400 chars.+Off in production unless `VOICE_LOG=1`.++Reply "go", or strike lines you do not want.+```++## 3. Install: the contract++Match the app. Same language as the surrounding code, same route style, same formatter. Write the pieces from the contract below; do not introduce a second language or toolchain for logging.++**Session id**: 8 lowercase hex chars from a UUID. The sink accepts `^[a-z0-9]{4,64}$`; it becomes a file name.++**Entry** (one JSON object per line):++| Field | Client | Server |+| --- | --- | --- |+| `t` | ms since the logger started | absent; the reader aligns by `ts` |+| `ts` | epoch ms | epoch ms |+| `kind` | `start`, `server`, `client`, `error`, `audio.in`, … | `server.token`, … |+| `src` | absent | `"server"` |+| rest | the hook's fields, redacted | the hook's fields |++**Redaction, applied client side before buffering**: on audio event types (`response.output_audio.delta`, `response.audio.delta`, `input_audio_buffer.append`) replace `delta` / `audio` with `bytes` = decoded base64 length; strings over 400 chars cut to 400 + `…[N chars]`; objects deeper than 4 → `"[depth]"`; arrays over 50 items truncated.++**Client logger**: buffer entries; flush as `POST <sink> {"sessionId","entries":[…]}` every 1 s or at 200 entries; on stop flush with keepalive (or the platform's "survive navigation" equivalent); swallow every transport error, logging must never throw into the voice path. Also mirror entries to the console in dev.++**Sink**: `POST /api/voice/log`, JSON body. `404` unless dev or `VOICE_LOG=1`. `400` if `sessionId` fails the regex or `entries` is not an array. Append at most 500 entries per request, drop any line over 16,000 chars, to `.voice-logs/<sessionId>.ndjson`, creating the directory. Reply `204`.++Pseudocode for any server:++```+handle POST /api/voice/log:+ if production and VOICE_LOG != "1": return 404+ body = parse json or return 400+ if not regex(body.sessionId) or not list(body.entries): return 400+ mkdir .voice-logs; append join(json(e) for e in body.entries[:500] if len < 16000) to .voice-logs/{sessionId}.ndjson+ return 204+```++Pseudocode for the client logger:++```+logger(sessionId, sink):+ buffer = []; started = now()+ log(kind, data): buffer.push({ ...redact(data), t: now() - started, ts: epoch_ms(), kind }); schedule flush (1 s timer, or immediately at 200 entries)+ server(event, extra): log("server", { ...redact_event(event), ...extra }) # never per audio delta+ client(event): log("client", redact_event(event)) # never per audio chunk+ error(where, err, extra): log("error", { where, name, message, ...extra })+ flush(final=false): POST sink {"sessionId","entries": buffer}; buffer = []; ignore all errors; keepalive when final+ close(): flush(final=true)+```++## Hook points++`kind` and fields; the shape is the same in every language.++| When | `kind` and fields |+| --- | --- |+| Session start | `start { url, target_rate }` |+| Token fetched / failed | `token.ok { ms }` / `error { where: "token", name, message, ms }` |+| Mic granted / denied | `mic.ok { ms, label, settings }` / `error { where: "mic", … }` |+| Audio graph ready | `env { ua, mic_rate, mic_state, play_rate, play_state, capture_frames, target_rate }` |+| Socket | `ws.connecting`, `ws.open { ms }`, `ws.error`, `ws.close { code, reason, wasClean, by }` |+| Every event sent, except audio chunks | `client { …redacted event }` |+| Every event received, except audio deltas | `server { …redacted event, phase }` |+| Phase change (dedupe) | `phase { phase }` |+| Mic chunks, aggregated per 2 s | `audio.in { chunks, bytes, rms_max, rms_avg, pending, mic_state, phase }` |+| Pre-open buffer sent on open | `audio.flush { chunks }` |+| First audio delta of a response | `audio.out.first { response_id, bytes, since_response_created_ms, since_speech_stopped_ms, play_state }` |+| `response.done` | `audio.out { response_id, status, deltas, bytes, audio_ms, wall_ms, max_gap_ms, queued_ms, underruns, drain_ms_max }` |+| Barge-in stop | `play.stop { reason, dropped_ms }` |+| User stop | `stop { by: "client", phase }`, then flush |+| Token route (server) | `server.token { ok, status, ms, upstream }` |++In the message handler: if the event is an audio delta, count bytes and gaps and play it; otherwise log it as `server` with the current phase, then run the existing handling. Keep `speechStoppedT` from `input_audio_buffer.speech_stopped` and `createdT` from `response.created`; report both distances on the first delta. In the player, count an underrun when the next scheduled time is already in the past mid-response, track the largest drain, reset on `response.created`.++UI: show the id in the voice status line (`Listening · session ab12cd34`) and append `(voice session <id>)` to voice errors, so the user can name the run.++## 4. Verify++```bash+curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \+ -H 'Content-Type: application/json' \+ -d '{"sessionId":"smoke001","entries":[{"t":0,"ts":0,"kind":"start"}]}' # 204+curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \+ -H 'Content-Type: application/json' -d '{"sessionId":"../x","entries":[]}' # 400+cat .voice-logs/smoke001.ndjson && rm .voice-logs/smoke001.ndjson+```++Then hand off: the user tests in the real app and reports what they said, what they heard, when it went wrong, and the session id.++## Reading a log++Any of these; none needs the app's toolchain:++```bash+f=.voice-logs/<id>.ndjson+jq -r 'select(.kind|IN("start","token.ok","mic.ok","env","ws.open","ws.close","server.token","stop","error")) | "\(.t // .ts)ms \(.kind) \(.type // "") \(.message // "")"' $f # milestones and errors+jq -r 'select(.kind=="server") | .type' $f | sort | uniq -c | sort -rn # server event counts+jq -c 'select(.kind|IN("audio.out.first","audio.out"))' $f # per-turn latency, gaps, underruns+jq -c 'select(.kind=="audio.in")' $f # mic windows, rms+```++Or read the file; it is one object per line in time order. Server entries have only `ts`; align them to the client clock with the `start` entry's `ts`. If the user wants a summary command, write one in the app's language that prints: milestones, server event counts, one line per `audio.out` turn with its `audio.out.first` latencies, mic window totals, errors and closes, and the last 25 entries.++## 5. Fix loop++1. **Instrument** (steps 1–4) if the app has no `.voice-logs` pipeline yet.+2. **User tests** in the real app and describes the run.+3. **Read the log** around the failing `t`.+4. **Match** symptom → signature → fix (table below). No matching signature: add logging first, re-test, then fix.+5. **Fix one thing**, re-test, confirm the signature is gone in a fresh log.+6. **Write it down**: append a confirmed row under "Confirmed from sessions". If the fix changes how voice should be built, update `/add-voice` too.++## Symptom → log signature → fix (starter rows)++| Symptom (user) | Signature (log) | Fix |+| --- | --- | --- |+| Silence, but transcript appears | `env.play_state` or `audio.out.first.play_state` = `suspended` | Create and resume the playback audio context inside the user gesture; one context per session, not per turn |+| Assistant interrupts itself | `speech_started` with `phase=speaking`; `audio.in.rms_max` rises only during playback | Echo. Confirm with headphones (if it stops, it is echo). Keep echo cancellation on, lower speaker volume, or gate mic sends while `speaking` |+| Choppy, stuttering | `audio.out.underruns` > 0, `drain_ms_max` high, `max_gap_ms` far above chunk length | Schedule a small lead (150–250 ms) before the first chunk plays; do not rebuild the audio context per turn |+| Crackle, wrong pitch or speed | `session.update` rate ≠ buffer rate; odd `bytes` | One rate everywhere (`audio.input/output.format.rate`, player buffer); even-byte alignment |+| Never connects, or closes at once | `ws.close` before `session.updated`; `server.token.ok=false` | Mint a token per click (300 s), protocol `xai-client-secret.<token>`, `model` in URL; read `server.token.status` and `upstream` |+| Mic does nothing | `audio.in.rms_max` ≈ 0 in every window; `mic.ok.label` unexpected | Wrong device or OS permission; check `mic.ok.settings`, `label`, `mic_state` |+| No user transcript | no `conversation.item.input_audio_transcription.updated` | Set `audio.input.transcription.model: "grok-transcribe"` in `session.update` |+| Slow first word | `audio.out.first.since_speech_stopped_ms` high | Try `reasoning.effort: "none"`; shorter `instructions`; check `token.ok.ms` and `ws.open.ms` for connect cost |+| User text appears after the reply | `...transcription.updated` `t` > `response.created` `t` | Create the user row on `input_audio_buffer.committed` (`item_id`), fill it on `updated` |+| First words cut off | `ws.open.ms` large and `audio.flush.chunks` at the buffer cap | Start mic before the socket, buffer early audio, raise the pre-open cap |++## Confirmed from sessions++Append after a fix is verified in a fresh log. Format: `YYYY-MM-DD · symptom · signature · fix · file(s)`.++## Rules++- Plan first; no edits before the user aligns on the change list.+- Never tokens, keys, or raw audio. Audio becomes byte counts.+- Dev only. The sink returns 404 in production unless `VOICE_LOG=1`. `.voice-logs/` is gitignored.+- Logging never throws into the voice path.+- Aggregate audio; never log per chunk.+- One change per re-test so the log tells you which fix worked.+- Do not invent xAI event names; confirm in https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech before adding a signature.