Explainer videos and product demos made by your AI agent. Free and open source: a local voice (Kokoro), word timing (Whisper) and a canvas renderer turn a script into a narrated MP4.
JavaScript ai-agentsanimationcanvasclaude-codecodexexplainer-videoffmpegjavascriptkokoroproduct-demotext-to-speechvideo-generationwhisper
Narrate with ElevenLabs or your own recording (#7)
Vincent Schmalbachd818e7d6dc59
diff --git a/AGENTS.md b/AGENTS.mdindex 888c921..8271283 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -6,11 +6,13 @@ the words and the pictures. explainroo makes the video.explainroo turns two files into an MP4 video with narration. `script.md`holds what the voice says. `scenes.js` draws what the viewer sees. Every-drawing can appear on a word the voice says. A voice model (Kokoro) reads the-script on this computer. Whisper listens to the recording and writes down-when each word is spoken. Chrome draws the frames in the background.+drawing can appear on a word the voice says. A voice model (Kokoro) reads+the script on this computer, or the narration comes from ElevenLabs or your+own recording. Whisper listens to the narration and writes down when each+word is spoken. Chrome draws the frames in the background.explainroo adds music and sound effects, and ffmpeg makes the MP4. You need-no API key unless the user wants AI images.+no API key for local voices or recordings. ElevenLabs and AI images need+keys.The full documentation for people is at https://www.explainroo.com/docs/.@@ -45,6 +47,7 @@ When you ask, put everything you need in one message. Typical reasons to ask:- The topic is unclear, or you need a fact you cannot check.- The user wants AI images and there is no OpenRouter key (see Images).+- The user wants ElevenLabs and there is no ElevenLabs key (see Voices).- The user wants something explainroo cannot do, like video shot with acamera.@@ -80,7 +83,8 @@ When you ask, put everything you need in one message. Typical reasons to ask:The command prints how long each scene is and how many words the speechcheck confirmed. If a word is not confirmed, the voice probably said itwrong. Fix it with `{shown|spoken}` in the script and run the command- again. Only the scenes you changed are made again.+ again. For a recording, make script.md say what was actually said.+ Only the scenes you changed are made again.5. **Plan the pictures.** For each scene, decide what is on screen when eachmarker or important word is spoken. Decide where each thing sits and what@@ -224,6 +228,77 @@ Narration for this scene. [#marker] A marker names a moment you can animate on.picture of the real screen is better than three boxes with general words.- **A clear last frame** that shows the main point.+## Voices++Kokoro runs on this computer and needs no key. Pick a voice with+`explainroo voices`. A string in `video.json` keeps using Kokoro:++```json+"voice": "af_heart"+```++`{ "provider": "kokoro", "id": "af_heart" }` means the same thing.++ElevenLabs offers premium and cloned voices. It costs money on the user's+account and receives the script text. If the user wants it and there is no+key, ask them for an API key from elevenlabs.io. Set `ELEVENLABS_API_KEY` in+the environment, or save `ELEVENLABS_API_KEY=...` in a `.env` file in the+project folder or the explainroo folder. The environment takes priority,+then the project file. Never print the key, and never put it in a script,+a commit or a log. Keep `.env` out of git.++```bash+node bin/explainroo.js voices videos/<name> --elevenlabs+node bin/explainroo.js voices videos/<name> --elevenlabs --json+```++The list shows the names, ids and labels available to that key. Use an id+in `video.json`:++```json+"voice": { "provider": "elevenlabs", "id": "JBFqnCBsd6RMkjVDRZzb", "model": "eleven_multilingual_v2" }+```++`model` is optional and defaults to `eleven_multilingual_v2`. The voice+command reports the characters used. Each request is saved in+`build/voice/elevenlabs-usage.jsonl`, without the key or script text.+ElevenLabs uses `speed` times `pace`, limited to 0.7 through 1.2.+The voice command tells you when it uses that limit.++You can also record the narration yourself. Use a quiet room and read+script.md as written. Put one take for the whole video in `assets/`:++```json+"voice": { "provider": "recording", "src": "assets/narration.m4a" }+```++Or put one file per scene in a folder:++```json+"voice": { "provider": "recording", "src": "assets/narration/" }+```++Each scene with narration needs `<scene-id>.<ext>`, for example `intro.wav`.+Supported formats are wav, mp3, m4a, aac, flac, ogg and opus. Files must stay+inside `assets/`, with no `..` in the path.++One take is split into scenes using the words in script.md. Each cut keeps+about 0.15 seconds around the speech. Pauses within a scene stay as recorded.+The scene's `lead` and `hold` add the gaps between scenes. Recordings are+made mono and their volume is brought close to Kokoro's. Their sample rate+is kept up to 48 kHz. Changing `pace` or an explicit `speed` changes their+tempo without changing pitch.++`[pause]`, `sentenceGap` and `paragraphGap` do not add silence to recordings.+Markers `[#name]`, captions, cues and `{shown|spoken}` still work. If the+speech check reports a mismatch, make script.md say what was actually said.+Replacing a scene recording makes that scene again. Replacing a whole take+makes all its scenes again.++ElevenLabs narration and recordings must be in English for now. The word+timing model understands English only. Run `voice`, `check` and `render`+as usual with any of the three sources.+## Imagesexplainroo can make illustrations with AI image models through OpenRouter.@@ -457,7 +532,7 @@ seconds are real seconds of the scene, like `s.t` and `s.cue()`. Sofixed time like `at: 4` does not move with the pace. That is one more reasonto use cues. 1 is the normal pace. 1.15 to 1.3 suits social media and viewerswho know the topic. Above 1.4 the video gets hard to follow. `speed` changes-only the voice.+only the voice. Its default is 0.9 for Kokoro and 1 for the other sources.## The watermark@@ -488,8 +563,8 @@ videos/<name>/| `theme` | `paper` | a built-in look, or a relative path to a `.json` look file (see Custom looks) || `size` | `16:9` | a platform (`youtube`, `shorts`, `tiktok`, `reels`, `vertical`, `instagram`, `linkedin`, `square`), `16:9`, `9:16`, `1:1`, `4:5` or `WIDTHxHEIGHT` || `fps` | 30 | 24, 25, 30, 50 or 60 |-| `voice` | `af_heart` | see `explainroo voices` |-| `speed` | 0.9 | voice speed only, 0.6 to 1.6 |+| `voice` | `af_heart` | a Kokoro id or a provider object for Kokoro, ElevenLabs or a recording; see Voices |+| `speed` | 0.9 for Kokoro, 1 otherwise | voice speed only, 0.6 to 1.6; ElevenLabs limits speed times pace to 0.7 through 1.2 || `pace` | 1 | speed of the whole video (voice, pauses, animations), 0.7 to 1.6 || `music` | `true` | `true` (the look's style), `warm`, `upbeat`, `calm`, `tech`, `playful`, `{ "style", "volume" }` or `false` || `sfx` | `true` | sound effects: `true`, `"minimal"` or `false` |@@ -497,7 +572,7 @@ videos/<name>/| `cover` | `"auto"` | full-size JPEG at the end of the first scene, or a still spec: `12.5`, `"intro"`, `"[email protected]"`, `"intro@end"` || `transition` | `"auto"` | the look's default, or `fade`, `slide`, `wipe`, `zoom`, `brush`, `cut` || `lead`, `hold`, `end` | 0.35, 0.7, 1.4 | seconds before the voice, after it, and extra at the very end |-| `sentenceGap`, `paragraphGap` | 0.3, 0.55 | pauses in the narration |+| `sentenceGap`, `paragraphGap` | 0.3, 0.55 | pauses in generated narration; recordings keep their own pauses || `loudness` | -14 | target loudness in LUFS || `boil` | 0 | redraws per second of hand-drawn lines; 0 keeps them still || `watermark` | `"explainroo.com"` | small text in a corner, or `false` |@@ -574,7 +649,8 @@ when `ink` or `muted` has too little contrast against `bg`.| `verify [project]` | checks the finished file: loudness, black frames, silence, clear voice || `image [project] <name> "<prompt>"` | makes an illustration with OpenRouter || `images [project]` | lists the images and what they cost |-| `voices`, `say "text"` | lists the 28 voices, or makes a sample |+| `voices`, `say "text"` | lists the 28 Kokoro voices, or makes a sample |+| `voices [project] --elevenlabs` | lists the names, ids and labels available to the key (`--json` also works) || `themes`, `formats`, `icons <word>` | lists the looks and the sizes, searches the 1,854 icons || `doctor` | checks the setup (`--fetch` downloads the speech models) |diff --git a/src/cli.js b/src/cli.jsindex 17cf500..5ca8436 100644--- a/src/cli.js+++ b/src/cli.js@@ -13,6 +13,7 @@ import { writeWav } from './wav.js';import { ffmpegVersion } from './ffmpeg.js';import { findChrome } from './browser.js';import { generateImage, imageLog, imageSpend, IMAGE_MODELS } from './images.js';+import { elevenVoices, elevenUsage, elevenSpeed } from './elevenlabs.js';const VERSION = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf8')).version;@@ -22,7 +23,7 @@ Usage: explainroo <command> [project] [options]Make a videoinit <dir> create a project (--theme, --size, --pace, --voice, --title)- voice [project] generate narration and word timings (cached)+ voice [project] make the narration (Kokoro, ElevenLabs or a recording) and word timingspreview [project] live preview in your browser, reloads on save (--port)still [project] [t…] PNG stills: "12.5", "scene", "[email protected]" (default: end of every scene)sheet [project] contact sheet of the whole video (--scene id, --every s)@@ -37,7 +38,8 @@ Images (optional, needs an OpenRouter API key)images [project] list generated images and what they costReference- voices list voices; "explainroo say 'text' --voice am_michael" to hear one+ voices [project] list Kokoro voices (--elevenlabs lists voices available to your key)+ say "text" hear a Kokoro sample (--voice am_michael, --speed 1, --out sample.wav)themes list looksformats list sizes for YouTube, Shorts, TikTok, Reels, Instagram, LinkedInicons <word…> search the 1,800+ built-in icons@@ -61,7 +63,7 @@ function parseArgs(argv) {}return { pos, flags };}-const BOOL = new Set(['json', 'draft', 'force', 'fetch', 'help', 'version', 'quiet', 'open', 'no-style']);+const BOOL = new Set(['json', 'draft', 'force', 'fetch', 'help', 'version', 'quiet', 'open', 'no-style', 'elevenlabs']);function num(v, name) {if (v === undefined) return undefined;@@ -95,12 +97,22 @@ export async function main(argv) {case 'init': return init(pos[0], flags, print);case 'voice': {const project = loadProject(pos[0]);+ const before = elevenUsage(project).length;const { voices, timeline } = await prepare(project, { log, force: !!flags.force });+ const characters = elevenUsage(project).slice(before).reduce((n, e) => n + e.characters, 0);const rows = project.script.scenes.map((s) => {const v = voices[s.id];- return { scene: s.id, voice: v.duration, confirmed: v.matchRate, unconfirmed: v.unmatched, marks: v.marks };+ return { scene: s.id, voice: v.duration, confirmed: v.matchRate, unconfirmed: v.unmatched, marks: v.marks,+ ...(project.config.voice.provider === 'recording' && v.matchRate < 0.85 ? { message: 'The recording does not match script.md here. Make script.md say what was actually said.' } : {}) };});- print(rows.map((r) => `${r.scene.padEnd(18)} ${r.voice.toFixed(1).padStart(5)}s ${Math.round(r.confirmed * 100)}% confirmed${r.unconfirmed.length ? ' unconfirmed: ' + r.unconfirmed.slice(0, 6).join(', ') : ''}`).join('\n') + `\ntotal ${timeline.duration.toFixed(1)}s`, { scenes: rows, duration: timeline.duration });+ const mismatch = project.config.voice.provider === 'recording' && rows.some((r) => r.confirmed < 0.85)+ ? '\nThe recording does not match script.md in those scenes. Make script.md say what was actually said.' : '';+ const isEleven = project.config.voice.provider === 'elevenlabs';+ const wanted = project.config.speed * project.config.pace;+ const speed = elevenSpeed(wanted);+ const speedNote = isEleven && speed !== wanted ? `\nElevenLabs speed ${wanted.toFixed(2)} is outside 0.7 to 1.2; using ${speed}.` : '';+ const usage = isEleven ? `\nElevenLabs: ${characters} characters used` : '';+ print(rows.map((r) => `${r.scene.padEnd(18)} ${r.voice.toFixed(1).padStart(5)}s ${Math.round(r.confirmed * 100)}% confirmed${r.unconfirmed.length ? ' unconfirmed: ' + r.unconfirmed.slice(0, 6).join(', ') : ''}`).join('\n') + `\ntotal ${timeline.duration.toFixed(1)}s` + mismatch + usage + speedNote, { scenes: rows, duration: timeline.duration, ...(isEleven ? { characters, speed, ...(speed !== wanted ? { requested_speed: wanted } : {}) } : {}) });return 0;}case 'preview': return await preview(pos[0], flags, log);@@ -168,7 +180,12 @@ export async function main(argv) {return 0;}case 'voices': {- print(Object.entries(VOICES).map(([k, v]) => `${k.padEnd(12)} ${v}`).join('\n'), { voices: VOICES });+ if (flags.elevenlabs) {+ const voices = await elevenVoices(path.resolve(pos[0] || '.'));+ print(voices.map((v) => `${v.name} ${v.id} ${Object.entries(v.labels).map(([k, x]) => `${k}: ${x}`).join(', ')}`).join('\n'), { voices });+ } else {+ print(Object.entries(VOICES).map(([k, v]) => `${k.padEnd(12)} ${v}`).join('\n') + '\nUse voices --elevenlabs for ElevenLabs, or set voice to { "provider": "recording", "src": "assets/narration.wav" } in video.json.', { voices: VOICES });+ }return 0;}case 'say': return await say(pos.join(' '), flags, log, print);diff --git a/src/elevenlabs.js b/src/elevenlabs.jsnew file mode 100644index 0000000..76dfaf3--- /dev/null+++ b/src/elevenlabs.js@@ -0,0 +1,90 @@+import fs from 'node:fs';+import path from 'node:path';+import { setTimeout as sleep } from 'node:timers/promises';+import { findEnv } from './env.js';+import { ProjectError } from './project.js';++export function elevenSpeed(speed) {+ return Math.max(0.7, Math.min(1.2, speed));+}++function apiKey(dir) {+ const key = findEnv('ELEVENLABS_API_KEY', dir);+ if (!key) throw new ProjectError('ElevenLabs needs an API key. Ask the user for one from elevenlabs.io, then set ELEVENLABS_API_KEY or put ELEVENLABS_API_KEY=... in a .env file in the project or explainroo folder.');+ return key;+}++async function request(url, key, options = {}) {+ for (let attempt = 0; attempt < 3; attempt++) {+ let res;+ try {+ res = await fetch(url, { ...options, signal: AbortSignal.timeout(120000), headers: { 'xi-api-key': key, 'Content-Type': 'application/json' } });+ } catch {+ throw new ProjectError('ElevenLabs did not answer. Check your connection and try again.');+ }+ if (res.ok) return res;+ const body = await res.json().catch(() => null);+ const status = body?.detail?.status;+ if (status === 'quota_exceeded' || res.status === 402) throw new ProjectError('the ElevenLabs account has no characters left. Ask the user to check their plan.');+ if (status === 'voice_not_found' || res.status === 404) throw new ProjectError('ElevenLabs could not find that voice id. Run "explainroo voices --elevenlabs" and check voice.id.');+ if (res.status === 401 || res.status === 403) throw new ProjectError('ElevenLabs rejected the API key. Ask the user to check the key and its permissions.');+ if ((res.status === 429 || res.status >= 500) && attempt < 2) {+ await sleep(250 * 2 ** attempt);+ continue;+ }+ throw new ProjectError(res.status === 429 || res.status >= 500+ ? 'ElevenLabs is busy. Try again in a moment.'+ : 'ElevenLabs could not make the voice. Check voice.id and voice.model in video.json.');+ }+}++export function decodePCM(buffer) {+ const bytes = Buffer.from(buffer);+ if (!bytes.length || bytes.length % 2) throw new ProjectError('ElevenLabs returned unreadable audio. Try the voice command again.');+ const samples = new Float32Array(bytes.length / 2);+ for (let i = 0; i < samples.length; i++) samples[i] = bytes.readInt16LE(i * 2) / 32768;+ return samples;+}++export function elevenUsage(project) {+ const file = path.join(project.paths.voice, 'elevenlabs-usage.jsonl');+ return fs.existsSync(file) ? fs.readFileSync(file, 'utf8').split('\n').filter(Boolean).map((l) => JSON.parse(l)) : [];+}++export async function elevenSpeech(project, scene, text, { previous, next } = {}) {+ const key = apiKey(project.dir);+ const { voice, speed, pace } = project.config;+ const res = await request(`https://api.elevenlabs.io/v1/text-to-speech/${encodeURIComponent(voice.id)}?output_format=pcm_24000`, key, {+ method: 'POST',+ // Without voice_settings ElevenLabs uses the voice's own saved settings.+ body: JSON.stringify({ text, model_id: voice.model, ...(elevenSpeed(speed * pace) !== 1 ? { voice_settings: { speed: elevenSpeed(speed * pace) } } : {}),+ ...(previous && voice.model !== 'eleven_v3' ? { previous_text: previous } : {}), ...(next && voice.model !== 'eleven_v3' ? { next_text: next } : {}) }),+ });+ const entry = { time: new Date().toISOString(), scene, characters: [...text].length, model: voice.model, voice_id: voice.id };+ fs.mkdirSync(project.paths.voice, { recursive: true });+ fs.appendFileSync(path.join(project.paths.voice, 'elevenlabs-usage.jsonl'), JSON.stringify(entry) + '\n');+ let buffer;+ try { buffer = await res.arrayBuffer(); }+ catch { throw new ProjectError('ElevenLabs audio download failed. Check your connection and try again.'); }+ return { audio: decodePCM(buffer), sampling_rate: 24000 };+}++export async function elevenVoices(dir) {+ const key = apiKey(dir);+ const voices = [];+ let token;+ do {+ const url = new URL('https://api.elevenlabs.io/v2/voices');+ url.searchParams.set('page_size', '100');+ if (token) url.searchParams.set('next_page_token', token);+ let data;+ try { data = await (await request(url, key)).json(); }+ catch (e) {+ if (e instanceof ProjectError) throw e;+ throw new ProjectError('ElevenLabs returned an unreadable voice list. Try again.');+ }+ voices.push(...data.voices.map((v) => ({ name: v.name, id: v.voice_id, labels: v.labels || {} })));+ token = data.has_more ? data.next_page_token : null;+ } while (token);+ return voices;+}diff --git a/src/env.js b/src/env.jsnew file mode 100644index 0000000..3e61b94--- /dev/null+++ b/src/env.js@@ -0,0 +1,16 @@+import fs from 'node:fs';+import path from 'node:path';+import { ROOT } from './server.js';++export function readEnvFile(file, name) {+ if (!fs.existsSync(file)) return null;+ for (const line of fs.readFileSync(file, 'utf8').split('\n')) {+ const m = /^\s*(?:export\s+)?([A-Z][A-Z0-9_]*)\s*=\s*(.*?)\s*$/.exec(line);+ if (m && m[1] === name) return m[2].replace(/^["']|["']$/g, '').trim() || null;+ }+ return null;+}++export function findEnv(name, projectDir, root = ROOT) {+ return process.env[name] || readEnvFile(path.join(projectDir, '.env'), name) || readEnvFile(path.join(root, '.env'), name);+}diff --git a/src/ffmpeg.js b/src/ffmpeg.jsindex 6223936..418515d 100644--- a/src/ffmpeg.js+++ b/src/ffmpeg.js@@ -69,13 +69,18 @@ export function frameEncoder({ fps, out, draft }) {}// Two-pass loudness normalization to `target` LUFS, peak-limited.-export async function normalizeLoudness(input, output, target = -14) {+export async function normalizeLoudness(input, output, target = -14, sampleRate = 48000) {const base = `I=${target}:TP=-1.5:LRA=11`;const pass1 = await ffmpeg(['-i', input, '-af', `loudnorm=${base}:print_format=json`, '-f', 'null', '-']);const json = pass1.stderr.slice(pass1.stderr.lastIndexOf('{'), pass1.stderr.lastIndexOf('}') + 1);const m = JSON.parse(json);+ // Silence and very short clips have no measurable integrated loudness.+ if (!Number.isFinite(Number(m.input_i))) {+ await ffmpeg(['-i', input, '-ar', String(sampleRate), '-c:a', 'pcm_s16le', output]);+ return { inputI: Number(m.input_i), inputTP: Number(m.input_tp) };+ }const measured = `measured_I=${m.input_i}:measured_TP=${m.input_tp}:measured_LRA=${m.input_lra}:measured_thresh=${m.input_thresh}:offset=${m.target_offset}`;- await ffmpeg(['-i', input, '-af', `loudnorm=${base}:${measured}:linear=true:print_format=summary`, '-ar', '48000', '-c:a', 'pcm_s16le', output]);+ await ffmpeg(['-i', input, '-af', `loudnorm=${base}:${measured}:linear=true:print_format=summary`, '-ar', String(sampleRate), '-c:a', 'pcm_s16le', output]);return { inputI: Number(m.input_i), inputTP: Number(m.input_tp) };}diff --git a/src/images.js b/src/images.jsindex 0885578..1575a87 100644--- a/src/images.js+++ b/src/images.js@@ -4,7 +4,7 @@import fs from 'node:fs';import path from 'node:path';import crypto from 'node:crypto';-import { ROOT } from './server.js';+import { findEnv } from './env.js';import { ProjectError } from './project.js';export const IMAGE_MODELS = {@@ -17,18 +17,8 @@ const ASPECTS = ['16:9', '9:16', '1:1', '4:3', '3:4', '3:2', '2:3'];export const DEFAULT_STYLE ='Flat illustration for a friendly how-to video, soft colors, simple shapes, clear subject in the middle, plain light background, no text, no letters, no numbers, no logos, no watermark.';-function readEnvFile(file) {- if (!fs.existsSync(file)) return null;- for (const line of fs.readFileSync(file, 'utf8').split('\n')) {- const m = /^\s*(?:export\s+)?OPENROUTER_API_KEY\s*=\s*(.*)\s*$/.exec(line);- if (m) return m[1].replace(/^["']|["']$/g, '').trim() || null;- }- return null;-}-export function findKey(projectDir) {- if (process.env.OPENROUTER_API_KEY) return process.env.OPENROUTER_API_KEY;- return readEnvFile(path.join(projectDir, '.env')) || readEnvFile(path.join(ROOT, '.env'));+ return findEnv('OPENROUTER_API_KEY', projectDir);}export function resolveModel(name) {diff --git a/src/project.js b/src/project.jsindex 3b64a9b..e8fabbd 100644--- a/src/project.js+++ b/src/project.js@@ -162,10 +162,31 @@ export function normalizeConfig(raw, dir = '.') {const unknown = Object.keys(raw).filter((k) => !(k in DEFAULTS) && !k.startsWith('_'));if (unknown.length) fail(`video.json has unknown setting(s): ${unknown.join(', ')}. Known: ${Object.keys(DEFAULTS).join(', ')}`);if (!THEMES.includes(cfg.theme) && !lookPath(cfg.theme)) fail(`theme must be one of ${THEMES.join(', ')} or a relative path to a .json look file, not "${cfg.theme}"`);- if (!VOICES[cfg.voice]) fail(`voice "${cfg.voice}" does not exist. Run "explainroo voices" for the list.`);+ if (typeof cfg.voice === 'object') {+ const v = cfg.voice;+ if (!v || Array.isArray(v)) fail('voice must be a Kokoro id or an object with provider');+ const keys = { kokoro: ['provider', 'id'], elevenlabs: ['provider', 'id', 'model'], recording: ['provider', 'src'] };+ if (typeof v.provider !== 'string' || !Object.hasOwn(keys, v.provider)) fail('voice.provider must be kokoro, elevenlabs or recording');+ const extra = Object.keys(v).filter((k) => !keys[v.provider].includes(k));+ if (extra.length) fail(`voice has unknown setting(s): ${extra.join(', ')}. Known: ${keys[v.provider].join(', ')}`);+ if (v.provider === 'recording') {+ if (!(typeof v.src === 'string' && /^assets\/.+/.test(v.src) && !v.src.split('/').includes('..') && !v.src.includes('\\'))) {+ fail('voice.src must be a file or folder inside assets/, with no ..');+ }+ } else if (!(typeof v.id === 'string' && v.id.trim())) fail('voice.id is required');+ if (v.provider === 'kokoro') cfg.voice = v.id;+ if (v.provider === 'elevenlabs') {+ if (v.model !== undefined && !(typeof v.model === 'string' && v.model.trim())) fail('voice.model must be an ElevenLabs model id');+ cfg.voice = { ...v, model: v.model ?? 'eleven_multilingual_v2' };+ }+ }+ if (typeof cfg.voice === 'string') {+ if (!Object.hasOwn(VOICES, cfg.voice)) fail(`voice "${cfg.voice}" does not exist. Run "explainroo voices" for the list.`);+ } else if (!cfg.voice || typeof cfg.voice !== 'object') fail('voice must be a Kokoro id or an object with provider');+ if (typeof cfg.voice !== 'string' && raw.speed === undefined) cfg.speed = 1;if (!(cfg.speed >= 0.6 && cfg.speed <= 1.6)) fail('speed must be between 0.6 and 1.6');if (!(cfg.pace >= 0.7 && cfg.pace <= 1.6)) fail('pace must be between 0.7 and 1.6 (1 is normal, 1.3 is 30% faster)');- if (cfg.speed * cfg.pace > 1.8) fail(`speed ${cfg.speed} times pace ${cfg.pace} makes the voice too fast; keep speed x pace at 1.8 or below`);+ if (typeof cfg.voice === 'string' && cfg.speed * cfg.pace > 1.8) fail(`speed ${cfg.speed} times pace ${cfg.pace} makes the voice too fast; keep speed x pace at 1.8 or below`);if (![24, 25, 30, 50, 60].includes(cfg.fps)) fail('fps must be 24, 25, 30, 50 or 60');if (!TRANSITIONS.includes(cfg.transition)) fail(`transition must be one of ${TRANSITIONS.join(', ')}`);if (![true, false, 'minimal'].includes(cfg.sfx)) fail('sfx must be true, false or "minimal"');diff --git a/src/qa.js b/src/qa.jsindex 79dd873..29aa331 100644--- a/src/qa.js+++ b/src/qa.js@@ -17,7 +17,7 @@ export async function check(project, { log = () => {}, step = 0.25, viewWidth =const v = voices[sc.id];const suspicious = v ? v.unmatched.filter((w) => w.replace(/[^\p{L}\p{N}]/gu, '').length > 3) : [];if (v && v.duration && (v.matchRate < 0.9 || suspicious.length)) {- issues.push({ level: 'warn', scene: sc.id, t: null, message: `the voice check could not confirm ${suspicious.length ? suspicious.slice(0, 8).map((w) => `"${w}"`).join(', ') : `${Math.round((1 - v.matchRate) * 100)}% of the words`}. It heard: "${v.transcript.slice(0, 200)}". If a word is mispronounced, write it as {shown|spoken} in script.md.` });+ issues.push({ level: 'warn', scene: sc.id, t: null, message: `the voice check could not confirm ${suspicious.length ? suspicious.slice(0, 8).map((w) => `"${w}"`).join(', ') : `${Math.round((1 - v.matchRate) * 100)}% of the words`}. It heard: "${v.transcript.slice(0, 200)}". ${project.config.voice.provider === 'recording' ? 'The recording does not match script.md here. Make script.md say what was actually said.' : 'If a word is mispronounced, write it as {shown|spoken} in script.md.'}` });}if (v && v.duration > 40) issues.push({ level: 'hint', scene: sc.id, t: null, message: `the narration runs ${v.duration.toFixed(0)}s; long scenes are easier to follow when split` });}diff --git a/src/recordings.js b/src/recordings.jsnew file mode 100644index 0000000..61da60d--- /dev/null+++ b/src/recordings.js@@ -0,0 +1,89 @@+import fs from 'node:fs';+import path from 'node:path';+import crypto from 'node:crypto';+import { ProjectError } from './project.js';+import { alignWords } from './align.js';+import { ffmpeg, probe, normalizeLoudness } from './ffmpeg.js';+import { readWav, writeWav } from './wav.js';++const AUDIO_FILE = /\.(wav|mp3|m4a|aac|flac|ogg|opus)$/i;+// Four Kokoro scenes from how-dns-works measured -22.0 to -22.5 LUFS.+const RECORDING_LUFS = -22.3;++export function recordingFiles(project) {+ const src = project.config.voice.src;+ const file = path.join(project.dir, src);+ const assets = path.join(fs.realpathSync(project.dir), 'assets');+ const inside = (p) => fs.realpathSync(p).startsWith(assets + path.sep);+ if (!fs.existsSync(file)) throw new ProjectError(`${src} is missing. Put the recording in assets/ and check voice.src.`);+ if (!inside(file)) throw new ProjectError('recordings must stay inside assets/, including symlinks');+ const folder = fs.statSync(file).isDirectory();+ if (!folder && !AUDIO_FILE.test(file)) throw new ProjectError('recordings must be wav, mp3, m4a, aac, flac, ogg or opus files');+ const files = {};+ const hashes = new Map();+ const missing = [];+ const names = folder ? fs.readdirSync(file).sort() : [];+ for (const scene of project.script.scenes) {+ if (!scene.units.some((u) => u.type === 'word')) continue;+ const name = names.find((n) => path.parse(n).name === scene.id && AUDIO_FILE.test(n) && fs.existsSync(path.join(file, n)) && fs.statSync(path.join(file, n)).isFile());+ if (folder && !name) { missing.push(scene.id); continue; }+ const p = folder ? path.join(file, name) : file;+ if (!inside(p)) throw new ProjectError('recordings must stay inside assets/, including symlinks');+ if (!hashes.has(p)) hashes.set(p, crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex'));+ files[scene.id] = { file: p, hash: hashes.get(p) };+ }+ if (missing.length) throw new ProjectError(`missing scene recordings in ${src}: ${missing.join(', ')}. Add <scene-id>.<ext> for each scene with narration.`);+ return { folder, files };+}++function tempo(speed) {+ const filters = [];+ while (speed < 0.5) { filters.push('atempo=0.5'); speed /= 0.5; }+ while (speed > 2) { filters.push('atempo=2'); speed /= 2; }+ if (speed !== 1) filters.push(`atempo=${speed}`);+ return filters;+}++export async function decodeRecording(file, build, speed) {+ const info = await probe(file);+ const stream = info.streams.find((s) => s.codec_type === 'audio');+ if (!stream) throw new ProjectError('the recording has no audio. Choose an audio file in voice.src.');+ const sampleRate = Math.min(48000, Number(stream.sample_rate));+ const dir = fs.mkdtempSync(path.join(build, 'recording-'));+ try {+ const wav = path.join(dir, 'decoded.wav');+ const filters = tempo(speed);+ await ffmpeg(['-loglevel', 'error', '-i', file, '-vn', '-ac', '1', '-ar', String(sampleRate),+ ...(filters.length ? ['-af', filters.join(',')] : []), '-c:a', 'pcm_s16le', wav]);+ return readWav(wav);+ } finally {+ fs.rmSync(dir, { recursive: true, force: true });+ }+}++export async function saveRecording(file, samples, sampleRate) {+ const dir = fs.mkdtempSync(path.join(path.dirname(file), 'recording-'));+ try {+ const input = path.join(dir, 'input.wav');+ writeWav(input, samples, sampleRate);+ await normalizeLoudness(input, file, RECORDING_LUFS, sampleRate);+ } finally {+ fs.rmSync(dir, { recursive: true, force: true });+ }+}++export function recordingCuts(scenes, heard, duration) {+ const spoken = scenes.map((scene) => ({ scene, words: scene.units.filter((u) => u.type === 'word') })).filter((s) => s.words.length);+ const aligned = alignWords(spoken.flatMap((s) => s.words), heard, duration).words;+ let offset = 0;+ const spans = spoken.map(({ scene, words }) => {+ const times = aligned.slice(offset, offset + words.length);+ offset += words.length;+ return { scene, words: times, first: times[0].start, last: times.at(-1).end };+ });+ return spans.map((s, i) => {+ const left = i ? (spans[i - 1].last + s.first) / 2 : 0;+ const right = i + 1 < spans.length ? (s.last + spans[i + 1].first) / 2 : duration;+ return { scene: s.scene, words: s.words, start: Math.max(0, left, s.first - 0.15), end: Math.min(duration, right, s.last + 0.15) };+ });+}diff --git a/src/voice.js b/src/voice.jsindex 4a204f2..1f54bc3 100644--- a/src/voice.js+++ b/src/voice.js@@ -8,10 +8,12 @@ import { speechChunks, resolveMarks } from './script.js';import { alignWords } from './align.js';import { loadTTS, transcribeWords, TTS_MODEL, ASR_MODEL, TTS_SAMPLE_RATE, ttsDtype } from './models.js';import { writeWav, readWav, resample, trimSilence } from './wav.js';+import { elevenSpeech } from './elevenlabs.js';+import { recordingFiles, decodeRecording, saveRecording, recordingCuts } from './recordings.js';const PIPELINE = 7;-function sceneHash(scene, chunks, config) {+export function sceneHash(scene, chunks, config, recording = null) {return crypto.createHash('sha256').update(JSON.stringify({@@ -21,10 +23,12 @@ function sceneHash(scene, chunks, config) {dtype: ttsDtype(),voice: config.voice,speed: config.speed * config.pace,- chunks: chunks.map((c) => [c.gapBefore, c.text]),+ chunks: recording ? scene.spoken : chunks.map((c) => [c.gapBefore, c.text]),// Markers and the shown text of each word are saved with the timings,// so moving a [#marker] or changing {shown|spoken} makes the scene again.- layout: scene.units.map((u) => (u.type === 'mark' ? `#${u.name}` : u.type === 'word' ? u.display : '')).join('|'),+ layout: (recording ? scene.units.filter((u) => u.type !== 'pause') : scene.units).map((u) => (u.type === 'mark' ? `#${u.name}` : u.type === 'word' ? u.display : '')).join('|'),+ // Added last and only for recordings, so Kokoro scenes keep their cache.+ ...(recording ? { recording } : {}),})).digest('hex').slice(0, 16);@@ -48,11 +52,17 @@ export function readVoice(project, id) {export async function synthesize(project, { force = false, log = () => {}, only = null } = {}) {fs.mkdirSync(project.paths.voice, { recursive: true });const { config } = project;+ const provider = config.voice.provider || 'kokoro';+ const recordings = provider === 'recording' ? recordingFiles(project) : null;const result = {};const todo = [];for (const scene of project.script.scenes) {const chunks = speechChunks(scene.units, { sentenceGap: config.sentenceGap, paragraphGap: config.paragraphGap, pace: config.pace });- const hash = sceneHash(scene, chunks, config);+ const recording = recordings?.files[scene.id];+ const hash = sceneHash(scene, chunks, config, recording ? {+ content: recording.hash,+ scenes: recordings.folder ? null : project.script.scenes.map((s) => s.spoken),+ } : null);const cached = readVoice(project, scene.id);const hasAudio = !chunks.length || fs.existsSync(voicePaths(project, scene.id).wav);if (!force && cached && cached.hash === hash && hasAudio) {@@ -66,13 +76,27 @@ export async function synthesize(project, { force = false, log = () => {}, onlyif (!todo.length) return result;const needsSpeech = todo.some((t) => t.chunks.length);- const tts = needsSpeech ? await loadTTS({ log }) : null;+ const tts = needsSpeech && provider === 'kokoro' ? await loadTTS({ log }) : null;+ const wanted = config.speed * config.pace;+ let take = null;+ let cuts = [];+ let heardTake = null;+ if (needsSpeech && recordings && !recordings.folder) {+ log('transcribing the recording and finding scene boundaries');+ take = await decodeRecording(Object.values(recordings.files)[0].file, project.paths.build, wanted);+ heardTake = await transcribeWords(resample(take.samples, take.sampleRate, 16000), { log });+ cuts = recordingCuts(project.script.scenes, heardTake.words, take.samples.length / take.sampleRate);+ }for (const { scene, chunks, hash } of todo) {const t0 = Date.now();- const info = await synthesizeScene(project, scene, chunks, hash, tts, log);+ const info = recordings && chunks.length+ ? await recordingScene(project, scene, hash, recordings.files[scene.id], take, cuts.find((c) => c.scene.id === scene.id), heardTake, log)+ : await synthesizeScene(project, scene, chunks, hash, tts, log);result[scene.id] = info;if (chunks.length) {- const warn = info.matchRate < 0.85 ? `, check pronunciation (${Math.round(info.matchRate * 100)}% of words confirmed)` : '';+ const warn = info.matchRate < 0.85 ? (recordings+ ? `, recording does not match script.md here (${Math.round(info.matchRate * 100)}% of words confirmed). Make script.md say what was actually said.`+ : `, check pronunciation (${Math.round(info.matchRate * 100)}% of words confirmed)`) : '';log(`voice ${scene.id}: ${info.duration.toFixed(1)}s in ${((Date.now() - t0) / 1000).toFixed(1)}s${warn}`);}}@@ -105,7 +129,9 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {pieces.push(new Float32Array(Math.round(gap * sr)));cursor += Math.round(gap * sr);}- const audio = await tts.generate(chunk.text, { voice: config.voice, speed: config.speed * config.pace });+ const audio = config.voice.provider === 'elevenlabs'+ ? await elevenSpeech(project, scene.id, chunk.text, { previous: chunks[ci - 1]?.text, next: chunks[ci + 1]?.text })+ : await tts.generate(chunk.text, { voice: config.voice, speed: config.speed * config.pace });const samples = trimSilence(audio.audio, audio.sampling_rate || sr);const start = cursor / sr;const dur = samples.length / sr;@@ -154,6 +180,49 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {return info;}+async function recordingScene(project, scene, hash, source, take, cut, heardTake, log) {+ const audio = take || await decodeRecording(source.file, project.paths.build, project.config.speed * project.config.pace);+ const sr = audio.sampleRate;+ let from = cut ? Math.floor(cut.start * sr) : 0;+ let to = cut ? Math.ceil(cut.end * sr) : audio.samples.length;+ let samples = audio.samples.subarray(from, to);+ if (cut) {+ // Whisper can include the silence between scenes in a word's timestamp.+ const trimmed = trimSilence(samples, sr, { keep: 0.15 });+ if (trimmed.length) {+ from += (trimmed.byteOffset - samples.byteOffset) / 4;+ samples = trimmed;+ to = from + samples.length;+ }+ }+ const duration = samples.length / sr;+ const local = (w) => {+ const start = Math.max(0, Math.min(duration, w.start - from / sr));+ return { ...w, start, end: Math.min(duration, Math.max(start + 0.05, w.end - from / sr)) };+ };+ const heard = heardTake ? {+ words: heardTake.words.filter((w) => w.start >= cut.start && w.start < cut.end).map(local),+ } : await transcribeWords(resample(samples, sr, 16000), { log });+ const units = scene.units.filter((u) => u.type === 'word');+ const aligned = alignWords(units, heard.words, duration);+ const times = cut ? cut.words.map(local) : aligned.words;+ const wordTimes = [];+ let wi = 0;+ scene.units.forEach((u, i) => { if (u.type === 'word') wordTimes[i] = times[wi++]; });+ const files = voicePaths(project, scene.id);+ await saveRecording(files.wav, samples, sr);+ const info = {+ hash, id: scene.id, duration: round(duration),+ words: units.map((u, i) => ({ text: u.display, spoken: u.spoken, start: round(times[i].start), end: round(times[i].end), matched: aligned.words[i].matched })),+ marks: Object.fromEntries(Object.entries(resolveMarks(scene.units, wordTimes)).map(([k, v]) => [k, round(v)])),+ chunks: [{ start: 0, end: round(duration), text: scene.spoken }],+ transcript: heard.words.map((w) => w.text).join(' '),+ matchRate: round(aligned.matchRate), unmatched: units.filter((u, i) => !aligned.words[i].matched).map((u) => u.display),+ };+ fs.writeFileSync(files.json, JSON.stringify(info, null, 2));+ return info;+}+function round(v) {return Math.round(v * 1000) / 1000;}diff --git a/test/voice-providers.test.js b/test/voice-providers.test.jsnew file mode 100644index 0000000..7edc202--- /dev/null+++ b/test/voice-providers.test.js@@ -0,0 +1,338 @@+import test from 'node:test';+import assert from 'node:assert/strict';+import fs from 'node:fs';+import os from 'node:os';+import path from 'node:path';+import { normalizeConfig, loadProject, ProjectError } from '../src/project.js';+import { parseScript, speechChunks } from '../src/script.js';+import { findEnv, readEnvFile } from '../src/env.js';+import { findKey } from '../src/images.js';+import { elevenSpeech, elevenSpeed, decodePCM, elevenVoices, elevenUsage } from '../src/elevenlabs.js';+import { recordingCuts, recordingFiles, decodeRecording, saveRecording } from '../src/recordings.js';+import { sceneHash } from '../src/voice.js';+import { readWav, writeWav } from '../src/wav.js';+import { loudness } from '../src/ffmpeg.js';+import { main } from '../src/cli.js';++function temp(t) {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-voices-'));+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));+ return dir;+}++function project(t, voice = { provider: 'elevenlabs', id: 'test-voice' }) {+ const dir = temp(t);+ fs.mkdirSync(path.join(dir, 'assets'));+ fs.writeFileSync(path.join(dir, 'video.json'), JSON.stringify({ voice }));+ fs.writeFileSync(path.join(dir, 'script.md'), '# Test\n## first\nHello.\n## silent\n## second\nGoodbye.');+ fs.writeFileSync(path.join(dir, 'scenes.js'), 'export default { first() {}, silent() {}, second() {} };');+ return loadProject(dir);+}++function setEnv(t, name, value) {+ const previous = process.env[name];+ process.env[name] = value;+ t.after(() => {+ if (previous === undefined) delete process.env[name];+ else process.env[name] = previous;+ });+}++function mockAPI(t, fn) {+ t.mock.method(globalThis, 'fetch', fn);+ setEnv(t, 'ELEVENLABS_API_KEY', 'secret-test-key');+}++test('voice config accepts strings and all providers, with provider speed defaults', () => {+ assert.equal(normalizeConfig({}).voice, 'af_heart');+ assert.equal(normalizeConfig({ voice: 'am_michael' }).speed, 0.9);+ assert.equal(normalizeConfig({ voice: { provider: 'kokoro', id: 'af_heart' } }).voice, 'af_heart');+ assert.equal(normalizeConfig({ voice: { provider: 'kokoro', id: 'af_heart' } }).speed, 0.9);+ const cfg = normalizeConfig({ voice: { provider: 'elevenlabs', id: 'voice' }, pace: 1.6 });+ assert.deepEqual(cfg.voice, { provider: 'elevenlabs', id: 'voice', model: 'eleven_multilingual_v2' });+ assert.equal(cfg.speed, 1);+ assert.equal(normalizeConfig({ voice: { provider: 'elevenlabs', id: 'voice', model: 'eleven_flash_v2_5' }, speed: 0.8 }).speed, 0.8);+ for (const src of ['assets/narration.wav', 'assets/narration/', 'assets/take.m4a']) {+ assert.equal(normalizeConfig({ voice: { provider: 'recording', src } }).speed, 1);+ assert.equal(normalizeConfig({ voice: { provider: 'recording', src }, speed: 1.2 }).speed, 1.2);+ }+ assert.throws(() => normalizeConfig({ speed: 1.6, pace: 1.6 }), /too fast/);+});++test('voice config rejects bad providers, missing ids, unknown keys and unsafe paths', () => {+ for (const voice of [null, [], 1, {}, { provider: 'bad' }, { provider: 'toString' }, { provider: ['kokoro'], id: 'af_heart' },+ { provider: 'kokoro' }, { provider: 'kokoro', id: 'bad' }, { provider: 'kokoro', id: 'toString' },+ { provider: 'kokoro', id: 'af_heart', model: 'x' }, { provider: 'elevenlabs' },+ { provider: 'elevenlabs', id: '' }, { provider: 'elevenlabs', id: 'v', model: 4 },+ { provider: 'elevenlabs', id: 'v', key: 'secret' }, { provider: 'recording' },+ { provider: 'recording', src: '../take.wav' }, { provider: 'recording', src: 'assets/../take.wav' },+ { provider: 'recording', src: 'assets/..\\take.wav' }, { provider: 'recording', src: 'assets/a.wav', id: 'x' }]) {+ assert.throws(() => normalizeConfig({ voice }), ProjectError);+ }+});++test('shared env reader keeps environment, project, root precedence and quoted exports', (t) => {+ const dir = temp(t);+ const root = temp(t);+ const name = 'EXPLAINROO_TEST_KEY';+ const file = path.join(dir, '.env');+ setEnv(t, name, 'environment');+ fs.writeFileSync(path.join(root, '.env'), `${name}=root\n`);+ fs.writeFileSync(file, `UNRELATED=x\n export ${name} = "project" \nOPENROUTER_API_KEY='image-key'\n`);+ assert.equal(readEnvFile(file, name), 'project');+ assert.equal(findEnv(name, dir, root), 'environment');+ process.env[name] = '';+ assert.equal(findEnv(name, dir, root), 'project');+ setEnv(t, 'OPENROUTER_API_KEY', '');+ assert.equal(findKey(dir), 'image-key');+ fs.writeFileSync(file, `${name}=''\n`);+ assert.equal(findEnv(name, dir, root), 'root');+ fs.rmSync(file);+ assert.equal(readEnvFile(file, name), null);+});++test('ElevenLabs requests PCM, neighbouring text, speed, and logs usage without secrets', async (t) => {+ const p = project(t);+ p.config.speed = 1.1;+ p.config.pace = 1.3;+ const pcm = Buffer.alloc(6);+ pcm.writeInt16LE(-32768, 0);+ pcm.writeInt16LE(0, 2);+ pcm.writeInt16LE(16384, 4);+ mockAPI(t, async (url, opts) => {+ assert.equal(url, 'https://api.elevenlabs.io/v1/text-to-speech/test-voice?output_format=pcm_24000');+ assert.equal(opts.method, 'POST');+ assert.equal(opts.headers['xi-api-key'], 'secret-test-key');+ assert.deepEqual(JSON.parse(opts.body), { text: 'Hello.', model_id: 'eleven_multilingual_v2', voice_settings: { speed: 1.2 }, previous_text: 'Before.', next_text: 'After.' });+ return new Response(pcm);+ });+ const audio = await elevenSpeech(p, 'first', 'Hello.', { previous: 'Before.', next: 'After.' });+ assert.equal(audio.sampling_rate, 24000);+ assert.deepEqual([...audio.audio], [-1, 0, 0.5]);+ const usage = elevenUsage(p);+ assert.equal(usage.length, 1);+ assert.deepEqual(Object.keys(usage[0]), ['time', 'scene', 'characters', 'model', 'voice_id']);+ assert.equal(usage[0].scene, 'first');+ assert.equal(usage[0].characters, 6);+ assert.equal(usage[0].voice_id, 'test-voice');+ assert.equal(usage[0].model, 'eleven_multilingual_v2');+ assert.ok(Number.isFinite(Date.parse(usage[0].time)));+ assert.ok(!JSON.stringify(usage).includes('secret-test-key'));+ assert.throws(() => decodePCM(Buffer.alloc(1)), /unreadable audio/);+ assert.throws(() => decodePCM(Buffer.alloc(0)), /unreadable audio/);+ assert.equal(elevenSpeed(0.42), 0.7);+ assert.equal(elevenSpeed(2.56), 1.2);+ assert.equal(elevenSpeed(1), 1);+});++test('ElevenLabs keeps the saved voice settings at normal speed', async (t) => {+ const p = project(t);+ p.config.speed = 1;+ p.config.pace = 1;+ mockAPI(t, async (url, opts) => {+ assert.equal(JSON.parse(opts.body).voice_settings, undefined);+ return new Response(Buffer.alloc(2));+ });+ await elevenSpeech(p, 'first', 'Hello.');+});++test('ElevenLabs omits unsupported continuity for v3 and reports unreadable downloads safely', async (t) => {+ const p = project(t, { provider: 'elevenlabs', id: 'test-voice', model: 'eleven_v3' });+ mockAPI(t, async (url, opts) => {+ const body = JSON.parse(opts.body);+ assert.equal(body.model_id, 'eleven_v3');+ assert.equal(body.previous_text, undefined);+ assert.equal(body.next_text, undefined);+ return new Response(Buffer.alloc(2));+ });+ await elevenSpeech(p, 'first', 'Hello.', { previous: 'Before.', next: 'After.' });+ globalThis.fetch = async () => ({ ok: true, arrayBuffer: async () => { throw new Error('secret-test-key'); } });+ await assert.rejects(() => elevenSpeech(p, 'first', 'Hello.'), (e) => {+ assert.match(e.message, /download failed/);+ assert.ok(!e.message.includes('secret-test-key'));+ return true;+ });+ assert.equal(elevenUsage(p).length, 2);+});++test('ElevenLabs gives friendly errors without echoing the key or server body', async (t) => {+ const p = project(t);+ mockAPI(t, async () => { throw new Error('secret-test-key'); });+ const error = async (pattern) => assert.rejects(() => elevenSpeech(p, 'first', 'Hello.'), (e) => {+ assert.match(e.message, pattern);+ assert.ok(!e.message.includes('secret-test-key'));+ return true;+ });+ await error(/connection/);+ for (const [http, status, pattern] of [[401, 'invalid_api_key', /rejected/], [403, '', /permissions/],+ [404, 'voice_not_found', /voice id/], [400, 'voice_not_found', /voice id/], [401, 'quota_exceeded', /no characters/],+ [402, '', /no characters/], [422, '', /voice.model/]]) {+ globalThis.fetch = async () => new Response(JSON.stringify({ detail: { status, message: 'secret-test-key' } }), { status: http });+ await error(pattern);+ }+ assert.deepEqual(elevenUsage(p), []);+ process.env.ELEVENLABS_API_KEY = '';+ await error(/Ask the user.*elevenlabs.io.*\.env/);+});++test('ElevenLabs retries 429 and server failures, but stops after three attempts', async (t) => {+ const p = project(t);+ let calls = 0;+ mockAPI(t, async () => {+ calls++;+ return calls < 3 ? new Response('{}', { status: calls === 1 ? 429 : 503 }) : new Response(Buffer.alloc(2));+ });+ await elevenSpeech(p, 'first', 'Hello.');+ assert.equal(calls, 3);+ assert.equal(elevenUsage(p).length, 1);+ calls = 0;+ globalThis.fetch = async () => { calls++; return new Response('secret-test-key', { status: 500 }); };+ await assert.rejects(() => elevenSpeech(p, 'first', 'Hello.'), /busy/);+ assert.equal(calls, 3);+ assert.equal(elevenUsage(p).length, 1);+});++test('ElevenLabs voice list paginates and CLI prints names, ids and labels with JSON support', async (t) => {+ const dir = temp(t);+ let calls = 0;+ mockAPI(t, async (url) => {+ calls++;+ assert.equal(url.pathname, '/v2/voices');+ assert.equal(url.searchParams.get('page_size'), '100');+ const page2 = url.searchParams.has('next_page_token');+ if (page2) assert.equal(url.searchParams.get('next_page_token'), 'next');+ return Response.json({ voices: [{ name: page2 ? 'Bob' : 'Alice', voice_id: page2 ? 'b' : 'a', labels: { accent: 'American' } }], has_more: !page2, next_page_token: page2 ? null : 'next' });+ });+ assert.deepEqual(await elevenVoices(dir), [{ name: 'Alice', id: 'a', labels: { accent: 'American' } }, { name: 'Bob', id: 'b', labels: { accent: 'American' } }]);+ assert.equal(calls, 2);+ let output = '';+ t.mock.method(process.stdout, 'write', (s) => { output += s; return true; });+ assert.equal(await main(['voices', '--elevenlabs', dir, '--json']), 0);+ assert.equal(JSON.parse(output).voices[1].id, 'b');+ output = '';+ assert.equal(await main(['voices', dir, '--elevenlabs']), 0);+ assert.match(output, /Alice a accent: American/);+ output = '';+ await main(['voices']);+ assert.match(output, /af_heart/);+ assert.match(output, /voices --elevenlabs.*recording/);+});++test('cached CLI voice output reports recording mismatches and ElevenLabs speed limits', async (t) => {+ const p = project(t, { provider: 'recording', src: 'assets/take.wav' });+ writeWav(path.join(p.dir, 'assets/take.wav'), new Float32Array(24000), 24000);+ const recordings = recordingFiles(p);+ fs.mkdirSync(p.paths.voice, { recursive: true });+ const cache = (recording) => {+ for (const scene of p.script.scenes) {+ const chunks = speechChunks(scene.units, p.config);+ const hash = sceneHash(scene, chunks, p.config, recording && chunks.length+ ? { content: recordings.files[scene.id].hash, scenes: p.script.scenes.map((s) => s.spoken) } : null);+ const info = { hash, id: scene.id, duration: chunks.length ? 1 : 0, words: [], marks: {}, chunks: [], transcript: 'Different words.', matchRate: chunks.length ? 0 : 1, unmatched: chunks.length ? [scene.spoken] : [] };+ fs.writeFileSync(path.join(p.paths.voice, scene.id + '.json'), JSON.stringify(info));+ if (chunks.length) writeWav(path.join(p.paths.voice, scene.id + '.wav'), new Float32Array(24000), 24000);+ }+ };+ cache(true);+ let output = '';+ t.mock.method(process.stdout, 'write', (s) => { output += s; return true; });+ assert.equal(await main(['voice', p.dir, '--json']), 0);+ assert.match(JSON.parse(output).scenes[0].message, /recording does not match.*actually said/);+ fs.writeFileSync(path.join(p.dir, 'video.json'), JSON.stringify({ voice: { provider: 'elevenlabs', id: 'v' }, pace: 1.3 }));+ p.config = loadProject(p.dir).config;+ cache(false);+ output = '';+ assert.equal(await main(['voice', p.dir, '--json']), 0);+ assert.equal(JSON.parse(output).speed, 1.2);+ assert.equal(JSON.parse(output).requested_speed, 1.3);+ assert.equal(JSON.parse(output).characters, 0);+ output = '';+ assert.equal(await main(['voice', p.dir]), 0);+ assert.equal((output.match(/outside 0.7 to 1.2/g) || []).length, 1);+});++test('single-take cuts retain internal pauses and margins, skip silent scenes and avoid overlap', () => {+ const scenes = parseScript('## first\n[#hello] Hello [pause 9] {world|world}.\n## silent\n## second\nGoodbye now.').scenes;+ const heard = [{ text: 'Hello', start: 1, end: 1.5 }, { text: 'world', start: 3, end: 3.5 },+ { text: 'Goodbye', start: 5, end: 5.5 }, { text: 'now', start: 6, end: 6.5 }];+ const cuts = recordingCuts(scenes, heard, 8);+ assert.equal(cuts.length, 2);+ assert.equal(cuts[0].scene.id, 'first');+ assert.equal(cuts[0].start, 0.85);+ assert.equal(cuts[0].end, 3.65);+ assert.deepEqual(cuts[0].words, [{ start: 1, end: 1.5, matched: true }, { start: 3, end: 3.5, matched: true }]);+ assert.equal(cuts[1].start, 4.85);+ assert.equal(cuts[1].end, 6.65);+ heard[2].start = 3.6;+ const close = recordingCuts(scenes, heard, 6.5);+ assert.equal(close[0].end, 3.55);+ assert.equal(close[1].start, 3.55);+ assert.equal(close[1].end, 6.5);+ assert.deepEqual(recordingCuts(scenes.filter((s) => s.id === 'silent'), [], 8), []);+});++test('recording folders report all missing scenes, validate files and stay inside assets', (t) => {+ const p = project(t, { provider: 'recording', src: 'assets/narration/' });+ const folder = path.join(p.dir, 'assets/narration');+ fs.mkdirSync(folder);+ assert.throws(() => recordingFiles(p), /missing scene recordings.*first, second/);+ fs.writeFileSync(path.join(folder, 'first.mp3'), 'first');+ fs.writeFileSync(path.join(folder, 'second.M4A'), 'second');+ const files = recordingFiles(p);+ assert.equal(files.folder, true);+ assert.deepEqual(Object.keys(files.files), ['first', 'second']);+ fs.writeFileSync(path.join(folder, 'first.mp3'), 'replaced');+ const changed = recordingFiles(p);+ assert.notEqual(changed.files.first.hash, files.files.first.hash);+ assert.equal(changed.files.second.hash, files.files.second.hash);+ fs.rmSync(path.join(folder, 'first.mp3'));+ const outside = path.join(temp(t), 'outside.wav');+ fs.writeFileSync(outside, 'outside');+ fs.symlinkSync(outside, path.join(folder, 'first.wav'));+ assert.throws(() => recordingFiles(p), /including symlinks/);+ p.config.voice.src = 'assets/no.wav';+ assert.throws(() => recordingFiles(p), /missing/);+ p.config.voice.src = 'assets/bad.txt';+ fs.writeFileSync(path.join(p.dir, 'assets/bad.txt'), 'not audio');+ assert.throws(() => recordingFiles(p), /wav, mp3/);+});++test('scene cache includes audio settings and recording contents, ignores recording gaps', () => {+ const scene = parseScript('## first\n[#a] Hello. Goodbye.').scenes[0];+ const cfg = normalizeConfig({});+ const chunks = speechChunks(scene.units);+ const hash = sceneHash(scene, chunks, cfg);+ assert.notEqual(hash, sceneHash(scene, chunks, { ...cfg, voice: 'am_michael' }));+ assert.notEqual(hash, sceneHash(scene, chunks, { ...cfg, speed: 1 }));+ const eleven = normalizeConfig({ voice: { provider: 'elevenlabs', id: 'v' } });+ assert.notEqual(sceneHash(scene, chunks, eleven), sceneHash(scene, chunks, { ...eleven, voice: { ...eleven.voice, model: 'another' } }));+ const recording = normalizeConfig({ voice: { provider: 'recording', src: 'assets/take.wav' } });+ const recHash = sceneHash(scene, chunks, recording, { content: 'abc' });+ assert.notEqual(recHash, sceneHash(scene, chunks, recording, { content: 'def' }));+ assert.equal(recHash, sceneHash(scene, speechChunks(scene.units, { sentenceGap: 9 }), recording, { content: 'abc' }));+ assert.notEqual(recHash, sceneHash(scene, chunks, { ...recording, pace: 1.2 }, { content: 'abc' }));+ const paused = parseScript('## first\n[#a] Hello. [pause 9] Goodbye.').scenes[0];+ assert.equal(recHash, sceneHash(paused, speechChunks(paused.units), recording, { content: 'abc' }));+});++test('recordings preserve sample rate up to 48 kHz, apply tempo and normalize loudness', async (t) => {+ const dir = temp(t);+ const file = path.join(dir, 'tone.wav');+ const sr = 24000;+ const samples = Float32Array.from({ length: sr * 4 }, (_, i) => 0.02 * Math.sin(2 * Math.PI * 440 * i / sr));+ writeWav(file, samples, sr);+ const decoded = await decodeRecording(file, dir, 1.25);+ assert.equal(decoded.sampleRate, sr);+ assert.ok(Math.abs(decoded.samples.length / sr - 3.2) < 0.05);+ const out = path.join(dir, 'out.wav');+ await saveRecording(out, decoded.samples, decoded.sampleRate);+ assert.equal(readWav(out).sampleRate, sr);+ assert.ok(Math.abs((await loudness(out)).integrated - -22.3) < 0.5);+ const high = path.join(dir, 'high.wav');+ writeWav(high, new Float32Array(96000), 96000);+ assert.equal((await decodeRecording(high, dir, 1)).sampleRate, 48000);+ const silent = path.join(dir, 'silent.wav');+ await saveRecording(silent, new Float32Array(sr), sr);+ assert.equal(readWav(silent).samples.length, sr);+ assert.deepEqual(fs.readdirSync(dir).filter((n) => n.startsWith('recording-')), []);+});