Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(voice): ship a runnable example that exercises the whole voice loop

nirholas3510872eedee
diff --git a/packages/voice/examples/README.md b/packages/voice/examples/README.md
new file mode 100644
index 000000000..9bb8b01ad
--- /dev/null
+++ b/packages/voice/examples/README.md
@@ -0,0 +1,51 @@
+# Examples: @three-ws/voice
+
+| File | What it shows | Run |
+|---|---|---|
+| [`voice-loop.mjs`](voice-loop.mjs) | The whole loop against the live endpoints: probe both lanes, `speak()` text into a WAV, `lipsync()` that WAV into ARKit visemes, `transcribe()` it back to text, then `say()` for the one-round-trip path. | `node examples/voice-loop.mjs` |
+
+Run it from the package directory:
+
+```bash
+cd packages/voice
+node examples/voice-loop.mjs
+```
+
+No key, no wallet, nothing to install: all three lanes lead with free NVIDIA NIM
+models (Magpie TTS, Audio2Face-3D, Riva ASR). Expected output:
+
+```
+asr lane: configured=true encodings=wav/pcm/flac/ogg
+lipsync lane: configured=true model=audio2face-3d 30 fps arkit
+voices: 11 (nova, alloy, ash, ballad, …)
+
+speak("The quick brown fox jumps over the lazy dog.")
+ 208940 bytes of audio/wav from magpie-tts-multilingual (voice Magpie-Multilingual.EN-US.Aria)
+
+lipsync(clip)
+ 72 frames @ 30 fps over 2.37s
+ 55 blendshapes; JawOpen peaks at 0.506 (the mouth actually opens)
+
+transcribe(clip)
+ "The quick brown fox jumps over the lazy dog."
+ 9 word timings, 2.37s of audio, model riva-asr
+
+say("Welcome back.")
+ 77868 bytes of audio/wav (Magpie-Multilingual.EN-US.Aria)
+ + 27 aligned face frames @ 30 fps
+```
+
+Byte counts and the JawOpen peak move run to run (TTS is not deterministic); the
+shape does not.
+
+Two knobs:
+
+```bash
+TEXT="give the avatar different words" node examples/voice-loop.mjs
+SAVE_WAV=out.wav node examples/voice-loop.mjs # also write the synthesized clip
+```
+
+If a lane's provider key is not set on the deployment you point at, the probe
+reports `configured=false` and the call raises a typed `ThreeWsError` with code
+`not_configured` rather than crashing, which is the signal to fall back to
+in-browser recognition or amplitude lipsync.
diff --git a/packages/voice/examples/voice-loop.mjs b/packages/voice/examples/voice-loop.mjs
new file mode 100644
index 000000000..330260508
--- /dev/null
+++ b/packages/voice/examples/voice-loop.mjs
@@ -0,0 +1,62 @@
+// The whole voice loop against the live three.ws endpoints, in one run:
+//
+// speak() text → a voiced clip (NVIDIA Magpie TTS)
+// lipsync() that clip → ARKit visemes (NVIDIA Audio2Face-3D)
+// transcribe() that clip → text again (NVIDIA Riva ASR)
+// say() text → voice + face in one round trip
+//
+// Run: node examples/voice-loop.mjs
+// TEXT="say this instead" node examples/voice-loop.mjs
+// SAVE_WAV=out.wav node examples/voice-loop.mjs (also write the clip)
+//
+// Free and key-free: all three lanes lead with NVIDIA NIM. Nothing is paid and
+// nothing is stored server-side. Installed consumers import '@three-ws/voice'
+// instead of ../src/index.js.
+
+import { writeFile } from 'node:fs/promises';
+
+import { speak, lipsync, transcribe, say, voices, asrInfo, lipsyncInfo } from '../src/index.js';
+
+const TEXT = process.env.TEXT || 'The quick brown fox jumps over the lazy dog.';
+
+// 0. Probe both lanes first. A UI does exactly this to decide between the server
+// lane and its in-browser fallback.
+const [asr, face] = await Promise.all([asrInfo(), lipsyncInfo()]);
+console.log(`asr lane: configured=${asr.configured} encodings=${asr.encodings.join('/')}`);
+console.log(`lipsync lane: configured=${face.configured} model=${face.model} ${face.fps} fps ${face.blendshapeFormat}`);
+
+const catalog = await voices();
+console.log(`voices: ${catalog.voices.length} (${catalog.voices.slice(0, 4).map((v) => v.id).join(', ')}, …)`);
+
+// 1. Text → speech. WAV so the same bytes can drive both A2F and Riva below.
+console.log(`\nspeak("${TEXT}")`);
+const clip = await speak(TEXT, { voice: 'nova', format: 'wav' });
+console.log(` ${clip.blob.size} bytes of ${clip.contentType} from ${clip.model} (voice ${clip.voice})`);
+if (process.env.SAVE_WAV) {
+ await writeFile(process.env.SAVE_WAV, Buffer.from(await clip.blob.arrayBuffer()));
+ console.log(` written to ${process.env.SAVE_WAV}`);
+}
+
+// 2. Speech → face. The track is time-coded in seconds from clip start, so the
+// browser plays `clip` and samples `frames` by the audio element's currentTime.
+console.log('\nlipsync(clip)');
+const track = await lipsync(clip.blob, { format: 'wav' });
+const jaw = track.blendShapeNames.findIndex((n) => /jawopen/i.test(n));
+const jawMax = Math.max(...track.frames.map((f) => f.w[jaw] ?? 0));
+console.log(` ${track.frameCount} frames @ ${track.fps} fps over ${track.durationSec.toFixed(2)}s`);
+console.log(` ${track.blendShapeNames.length} blendshapes; JawOpen peaks at ${jawMax.toFixed(3)} (the mouth actually opens)`);
+
+// 3. Speech → text, closing the loop on the exact clip we synthesized.
+console.log('\ntranscribe(clip)');
+const heard = await transcribe(clip.blob, { format: 'wav', words: true });
+console.log(` "${heard.text}"`);
+console.log(` ${heard.words.length} word timings, ${heard.durationSec.toFixed(2)}s of audio, model ${heard.model}`);
+
+// 4. One round trip instead of two: the server synthesizes and animates that
+// exact audio, so the track and the clip are aligned by construction.
+console.log('\nsay("Welcome back.")');
+const { audio, animation } = await say('Welcome back.', { voice: 'nova' });
+console.log(` ${audio.blob.size} bytes of ${audio.contentType} (${audio.voiceName})`);
+console.log(` + ${animation.frameCount} aligned face frames @ ${animation.fps} fps`);
+
+console.log('\nHeard, spoke, and moved a face. No key, no payment.');