Skip to content

Explainer videos and product demos made by your AI agent. Free and open source: a local voice (Kokoro), word timing (Whisper) and a canvas renderer turn a script into a narrated MP4.

Narrate with ElevenLabs or your own recording (#7)

Vincent Schmalbachd818e7d6dc59
diff --git a/AGENTS.md b/AGENTS.md
index 888c921..8271283 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -6,11 +6,13 @@ the words and the pictures. explainroo makes the video.
explainroo turns two files into an MP4 video with narration. `script.md`
holds what the voice says. `scenes.js` draws what the viewer sees. Every
-drawing can appear on a word the voice says. A voice model (Kokoro) reads the
-script on this computer. Whisper listens to the recording and writes down
-when each word is spoken. Chrome draws the frames in the background.
+drawing can appear on a word the voice says. A voice model (Kokoro) reads
+the script on this computer, or the narration comes from ElevenLabs or your
+own recording. Whisper listens to the narration and writes down when each
+word is spoken. Chrome draws the frames in the background.
explainroo adds music and sound effects, and ffmpeg makes the MP4. You need
-no API key unless the user wants AI images.
+no API key for local voices or recordings. ElevenLabs and AI images need
+keys.
The full documentation for people is at https://www.explainroo.com/docs/.
@@ -45,6 +47,7 @@ When you ask, put everything you need in one message. Typical reasons to ask:
- The topic is unclear, or you need a fact you cannot check.
- The user wants AI images and there is no OpenRouter key (see Images).
+- The user wants ElevenLabs and there is no ElevenLabs key (see Voices).
- The user wants something explainroo cannot do, like video shot with a
camera.
@@ -80,7 +83,8 @@ When you ask, put everything you need in one message. Typical reasons to ask:
The command prints how long each scene is and how many words the speech
check confirmed. If a word is not confirmed, the voice probably said it
wrong. Fix it with `{shown|spoken}` in the script and run the command
- again. Only the scenes you changed are made again.
+ again. For a recording, make script.md say what was actually said.
+ Only the scenes you changed are made again.
5. **Plan the pictures.** For each scene, decide what is on screen when each
marker or important word is spoken. Decide where each thing sits and what
@@ -224,6 +228,77 @@ Narration for this scene. [#marker] A marker names a moment you can animate on.
picture of the real screen is better than three boxes with general words.
- **A clear last frame** that shows the main point.
+## Voices
+
+Kokoro runs on this computer and needs no key. Pick a voice with
+`explainroo voices`. A string in `video.json` keeps using Kokoro:
+
+```json
+"voice": "af_heart"
+```
+
+`{ "provider": "kokoro", "id": "af_heart" }` means the same thing.
+
+ElevenLabs offers premium and cloned voices. It costs money on the user's
+account and receives the script text. If the user wants it and there is no
+key, ask them for an API key from elevenlabs.io. Set `ELEVENLABS_API_KEY` in
+the environment, or save `ELEVENLABS_API_KEY=...` in a `.env` file in the
+project folder or the explainroo folder. The environment takes priority,
+then the project file. Never print the key, and never put it in a script,
+a commit or a log. Keep `.env` out of git.
+
+```bash
+node bin/explainroo.js voices videos/<name> --elevenlabs
+node bin/explainroo.js voices videos/<name> --elevenlabs --json
+```
+
+The list shows the names, ids and labels available to that key. Use an id
+in `video.json`:
+
+```json
+"voice": { "provider": "elevenlabs", "id": "JBFqnCBsd6RMkjVDRZzb", "model": "eleven_multilingual_v2" }
+```
+
+`model` is optional and defaults to `eleven_multilingual_v2`. The voice
+command reports the characters used. Each request is saved in
+`build/voice/elevenlabs-usage.jsonl`, without the key or script text.
+ElevenLabs uses `speed` times `pace`, limited to 0.7 through 1.2.
+The voice command tells you when it uses that limit.
+
+You can also record the narration yourself. Use a quiet room and read
+script.md as written. Put one take for the whole video in `assets/`:
+
+```json
+"voice": { "provider": "recording", "src": "assets/narration.m4a" }
+```
+
+Or put one file per scene in a folder:
+
+```json
+"voice": { "provider": "recording", "src": "assets/narration/" }
+```
+
+Each scene with narration needs `<scene-id>.<ext>`, for example `intro.wav`.
+Supported formats are wav, mp3, m4a, aac, flac, ogg and opus. Files must stay
+inside `assets/`, with no `..` in the path.
+
+One take is split into scenes using the words in script.md. Each cut keeps
+about 0.15 seconds around the speech. Pauses within a scene stay as recorded.
+The scene's `lead` and `hold` add the gaps between scenes. Recordings are
+made mono and their volume is brought close to Kokoro's. Their sample rate
+is kept up to 48 kHz. Changing `pace` or an explicit `speed` changes their
+tempo without changing pitch.
+
+`[pause]`, `sentenceGap` and `paragraphGap` do not add silence to recordings.
+Markers `[#name]`, captions, cues and `{shown|spoken}` still work. If the
+speech check reports a mismatch, make script.md say what was actually said.
+Replacing a scene recording makes that scene again. Replacing a whole take
+makes all its scenes again.
+
+ElevenLabs narration and recordings must be in English for now. The word
+timing model understands English only. Run `voice`, `check` and `render`
+as usual with any of the three sources.
+
## Images
explainroo can make illustrations with AI image models through OpenRouter.
@@ -457,7 +532,7 @@ seconds are real seconds of the scene, like `s.t` and `s.cue()`. So
fixed time like `at: 4` does not move with the pace. That is one more reason
to use cues. 1 is the normal pace. 1.15 to 1.3 suits social media and viewers
who know the topic. Above 1.4 the video gets hard to follow. `speed` changes
-only the voice.
+only the voice. Its default is 0.9 for Kokoro and 1 for the other sources.
## The watermark
@@ -488,8 +563,8 @@ videos/<name>/
| `theme` | `paper` | a built-in look, or a relative path to a `.json` look file (see Custom looks) |
| `size` | `16:9` | a platform (`youtube`, `shorts`, `tiktok`, `reels`, `vertical`, `instagram`, `linkedin`, `square`), `16:9`, `9:16`, `1:1`, `4:5` or `WIDTHxHEIGHT` |
| `fps` | 30 | 24, 25, 30, 50 or 60 |
-| `voice` | `af_heart` | see `explainroo voices` |
-| `speed` | 0.9 | voice speed only, 0.6 to 1.6 |
+| `voice` | `af_heart` | a Kokoro id or a provider object for Kokoro, ElevenLabs or a recording; see Voices |
+| `speed` | 0.9 for Kokoro, 1 otherwise | voice speed only, 0.6 to 1.6; ElevenLabs limits speed times pace to 0.7 through 1.2 |
| `pace` | 1 | speed of the whole video (voice, pauses, animations), 0.7 to 1.6 |
| `music` | `true` | `true` (the look's style), `warm`, `upbeat`, `calm`, `tech`, `playful`, `{ "style", "volume" }` or `false` |
| `sfx` | `true` | sound effects: `true`, `"minimal"` or `false` |
@@ -497,7 +572,7 @@ videos/<name>/
| `cover` | `"auto"` | full-size JPEG at the end of the first scene, or a still spec: `12.5`, `"intro"`, `"[email protected]"`, `"intro@end"` |
| `transition` | `"auto"` | the look's default, or `fade`, `slide`, `wipe`, `zoom`, `brush`, `cut` |
| `lead`, `hold`, `end` | 0.35, 0.7, 1.4 | seconds before the voice, after it, and extra at the very end |
-| `sentenceGap`, `paragraphGap` | 0.3, 0.55 | pauses in the narration |
+| `sentenceGap`, `paragraphGap` | 0.3, 0.55 | pauses in generated narration; recordings keep their own pauses |
| `loudness` | -14 | target loudness in LUFS |
| `boil` | 0 | redraws per second of hand-drawn lines; 0 keeps them still |
| `watermark` | `"explainroo.com"` | small text in a corner, or `false` |
@@ -574,7 +649,8 @@ when `ink` or `muted` has too little contrast against `bg`.
| `verify [project]` | checks the finished file: loudness, black frames, silence, clear voice |
| `image [project] <name> "<prompt>"` | makes an illustration with OpenRouter |
| `images [project]` | lists the images and what they cost |
-| `voices`, `say "text"` | lists the 28 voices, or makes a sample |
+| `voices`, `say "text"` | lists the 28 Kokoro voices, or makes a sample |
+| `voices [project] --elevenlabs` | lists the names, ids and labels available to the key (`--json` also works) |
| `themes`, `formats`, `icons <word>` | lists the looks and the sizes, searches the 1,854 icons |
| `doctor` | checks the setup (`--fetch` downloads the speech models) |
diff --git a/src/cli.js b/src/cli.js
index 17cf500..5ca8436 100644
--- a/src/cli.js
+++ b/src/cli.js
@@ -13,6 +13,7 @@ import { writeWav } from './wav.js';
import { ffmpegVersion } from './ffmpeg.js';
import { findChrome } from './browser.js';
import { generateImage, imageLog, imageSpend, IMAGE_MODELS } from './images.js';
+import { elevenVoices, elevenUsage, elevenSpeed } from './elevenlabs.js';
const VERSION = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf8')).version;
@@ -22,7 +23,7 @@ Usage: explainroo <command> [project] [options]
Make a video
init <dir> create a project (--theme, --size, --pace, --voice, --title)
- voice [project] generate narration and word timings (cached)
+ voice [project] make the narration (Kokoro, ElevenLabs or a recording) and word timings
preview [project] live preview in your browser, reloads on save (--port)
still [project] [t…] PNG stills: "12.5", "scene", "[email protected]" (default: end of every scene)
sheet [project] contact sheet of the whole video (--scene id, --every s)
@@ -37,7 +38,8 @@ Images (optional, needs an OpenRouter API key)
images [project] list generated images and what they cost
Reference
- voices list voices; "explainroo say 'text' --voice am_michael" to hear one
+ voices [project] list Kokoro voices (--elevenlabs lists voices available to your key)
+ say "text" hear a Kokoro sample (--voice am_michael, --speed 1, --out sample.wav)
themes list looks
formats list sizes for YouTube, Shorts, TikTok, Reels, Instagram, LinkedIn
icons <word…> search the 1,800+ built-in icons
@@ -61,7 +63,7 @@ function parseArgs(argv) {
}
return { pos, flags };
}
-const BOOL = new Set(['json', 'draft', 'force', 'fetch', 'help', 'version', 'quiet', 'open', 'no-style']);
+const BOOL = new Set(['json', 'draft', 'force', 'fetch', 'help', 'version', 'quiet', 'open', 'no-style', 'elevenlabs']);
function num(v, name) {
if (v === undefined) return undefined;
@@ -95,12 +97,22 @@ export async function main(argv) {
case 'init': return init(pos[0], flags, print);
case 'voice': {
const project = loadProject(pos[0]);
+ const before = elevenUsage(project).length;
const { voices, timeline } = await prepare(project, { log, force: !!flags.force });
+ const characters = elevenUsage(project).slice(before).reduce((n, e) => n + e.characters, 0);
const rows = project.script.scenes.map((s) => {
const v = voices[s.id];
- return { scene: s.id, voice: v.duration, confirmed: v.matchRate, unconfirmed: v.unmatched, marks: v.marks };
+ return { scene: s.id, voice: v.duration, confirmed: v.matchRate, unconfirmed: v.unmatched, marks: v.marks,
+ ...(project.config.voice.provider === 'recording' && v.matchRate < 0.85 ? { message: 'The recording does not match script.md here. Make script.md say what was actually said.' } : {}) };
});
- print(rows.map((r) => `${r.scene.padEnd(18)} ${r.voice.toFixed(1).padStart(5)}s ${Math.round(r.confirmed * 100)}% confirmed${r.unconfirmed.length ? ' unconfirmed: ' + r.unconfirmed.slice(0, 6).join(', ') : ''}`).join('\n') + `\ntotal ${timeline.duration.toFixed(1)}s`, { scenes: rows, duration: timeline.duration });
+ const mismatch = project.config.voice.provider === 'recording' && rows.some((r) => r.confirmed < 0.85)
+ ? '\nThe recording does not match script.md in those scenes. Make script.md say what was actually said.' : '';
+ const isEleven = project.config.voice.provider === 'elevenlabs';
+ const wanted = project.config.speed * project.config.pace;
+ const speed = elevenSpeed(wanted);
+ const speedNote = isEleven && speed !== wanted ? `\nElevenLabs speed ${wanted.toFixed(2)} is outside 0.7 to 1.2; using ${speed}.` : '';
+ const usage = isEleven ? `\nElevenLabs: ${characters} characters used` : '';
+ print(rows.map((r) => `${r.scene.padEnd(18)} ${r.voice.toFixed(1).padStart(5)}s ${Math.round(r.confirmed * 100)}% confirmed${r.unconfirmed.length ? ' unconfirmed: ' + r.unconfirmed.slice(0, 6).join(', ') : ''}`).join('\n') + `\ntotal ${timeline.duration.toFixed(1)}s` + mismatch + usage + speedNote, { scenes: rows, duration: timeline.duration, ...(isEleven ? { characters, speed, ...(speed !== wanted ? { requested_speed: wanted } : {}) } : {}) });
return 0;
}
case 'preview': return await preview(pos[0], flags, log);
@@ -168,7 +180,12 @@ export async function main(argv) {
return 0;
}
case 'voices': {
- print(Object.entries(VOICES).map(([k, v]) => `${k.padEnd(12)} ${v}`).join('\n'), { voices: VOICES });
+ if (flags.elevenlabs) {
+ const voices = await elevenVoices(path.resolve(pos[0] || '.'));
+ print(voices.map((v) => `${v.name} ${v.id} ${Object.entries(v.labels).map(([k, x]) => `${k}: ${x}`).join(', ')}`).join('\n'), { voices });
+ } else {
+ print(Object.entries(VOICES).map(([k, v]) => `${k.padEnd(12)} ${v}`).join('\n') + '\nUse voices --elevenlabs for ElevenLabs, or set voice to { "provider": "recording", "src": "assets/narration.wav" } in video.json.', { voices: VOICES });
+ }
return 0;
}
case 'say': return await say(pos.join(' '), flags, log, print);
diff --git a/src/elevenlabs.js b/src/elevenlabs.js
new file mode 100644
index 0000000..76dfaf3
--- /dev/null
+++ b/src/elevenlabs.js
@@ -0,0 +1,90 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import { setTimeout as sleep } from 'node:timers/promises';
+import { findEnv } from './env.js';
+import { ProjectError } from './project.js';
+
+export function elevenSpeed(speed) {
+ return Math.max(0.7, Math.min(1.2, speed));
+}
+
+function apiKey(dir) {
+ const key = findEnv('ELEVENLABS_API_KEY', dir);
+ if (!key) throw new ProjectError('ElevenLabs needs an API key. Ask the user for one from elevenlabs.io, then set ELEVENLABS_API_KEY or put ELEVENLABS_API_KEY=... in a .env file in the project or explainroo folder.');
+ return key;
+}
+
+async function request(url, key, options = {}) {
+ for (let attempt = 0; attempt < 3; attempt++) {
+ let res;
+ try {
+ res = await fetch(url, { ...options, signal: AbortSignal.timeout(120000), headers: { 'xi-api-key': key, 'Content-Type': 'application/json' } });
+ } catch {
+ throw new ProjectError('ElevenLabs did not answer. Check your connection and try again.');
+ }
+ if (res.ok) return res;
+ const body = await res.json().catch(() => null);
+ const status = body?.detail?.status;
+ if (status === 'quota_exceeded' || res.status === 402) throw new ProjectError('the ElevenLabs account has no characters left. Ask the user to check their plan.');
+ if (status === 'voice_not_found' || res.status === 404) throw new ProjectError('ElevenLabs could not find that voice id. Run "explainroo voices --elevenlabs" and check voice.id.');
+ if (res.status === 401 || res.status === 403) throw new ProjectError('ElevenLabs rejected the API key. Ask the user to check the key and its permissions.');
+ if ((res.status === 429 || res.status >= 500) && attempt < 2) {
+ await sleep(250 * 2 ** attempt);
+ continue;
+ }
+ throw new ProjectError(res.status === 429 || res.status >= 500
+ ? 'ElevenLabs is busy. Try again in a moment.'
+ : 'ElevenLabs could not make the voice. Check voice.id and voice.model in video.json.');
+ }
+}
+
+export function decodePCM(buffer) {
+ const bytes = Buffer.from(buffer);
+ if (!bytes.length || bytes.length % 2) throw new ProjectError('ElevenLabs returned unreadable audio. Try the voice command again.');
+ const samples = new Float32Array(bytes.length / 2);
+ for (let i = 0; i < samples.length; i++) samples[i] = bytes.readInt16LE(i * 2) / 32768;
+ return samples;
+}
+
+export function elevenUsage(project) {
+ const file = path.join(project.paths.voice, 'elevenlabs-usage.jsonl');
+ return fs.existsSync(file) ? fs.readFileSync(file, 'utf8').split('\n').filter(Boolean).map((l) => JSON.parse(l)) : [];
+}
+
+export async function elevenSpeech(project, scene, text, { previous, next } = {}) {
+ const key = apiKey(project.dir);
+ const { voice, speed, pace } = project.config;
+ const res = await request(`https://api.elevenlabs.io/v1/text-to-speech/${encodeURIComponent(voice.id)}?output_format=pcm_24000`, key, {
+ method: 'POST',
+ // Without voice_settings ElevenLabs uses the voice's own saved settings.
+ body: JSON.stringify({ text, model_id: voice.model, ...(elevenSpeed(speed * pace) !== 1 ? { voice_settings: { speed: elevenSpeed(speed * pace) } } : {}),
+ ...(previous && voice.model !== 'eleven_v3' ? { previous_text: previous } : {}), ...(next && voice.model !== 'eleven_v3' ? { next_text: next } : {}) }),
+ });
+ const entry = { time: new Date().toISOString(), scene, characters: [...text].length, model: voice.model, voice_id: voice.id };
+ fs.mkdirSync(project.paths.voice, { recursive: true });
+ fs.appendFileSync(path.join(project.paths.voice, 'elevenlabs-usage.jsonl'), JSON.stringify(entry) + '\n');
+ let buffer;
+ try { buffer = await res.arrayBuffer(); }
+ catch { throw new ProjectError('ElevenLabs audio download failed. Check your connection and try again.'); }
+ return { audio: decodePCM(buffer), sampling_rate: 24000 };
+}
+
+export async function elevenVoices(dir) {
+ const key = apiKey(dir);
+ const voices = [];
+ let token;
+ do {
+ const url = new URL('https://api.elevenlabs.io/v2/voices');
+ url.searchParams.set('page_size', '100');
+ if (token) url.searchParams.set('next_page_token', token);
+ let data;
+ try { data = await (await request(url, key)).json(); }
+ catch (e) {
+ if (e instanceof ProjectError) throw e;
+ throw new ProjectError('ElevenLabs returned an unreadable voice list. Try again.');
+ }
+ voices.push(...data.voices.map((v) => ({ name: v.name, id: v.voice_id, labels: v.labels || {} })));
+ token = data.has_more ? data.next_page_token : null;
+ } while (token);
+ return voices;
+}
diff --git a/src/env.js b/src/env.js
new file mode 100644
index 0000000..3e61b94
--- /dev/null
+++ b/src/env.js
@@ -0,0 +1,16 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import { ROOT } from './server.js';
+
+export function readEnvFile(file, name) {
+ if (!fs.existsSync(file)) return null;
+ for (const line of fs.readFileSync(file, 'utf8').split('\n')) {
+ const m = /^\s*(?:export\s+)?([A-Z][A-Z0-9_]*)\s*=\s*(.*?)\s*$/.exec(line);
+ if (m && m[1] === name) return m[2].replace(/^["']|["']$/g, '').trim() || null;
+ }
+ return null;
+}
+
+export function findEnv(name, projectDir, root = ROOT) {
+ return process.env[name] || readEnvFile(path.join(projectDir, '.env'), name) || readEnvFile(path.join(root, '.env'), name);
+}
diff --git a/src/ffmpeg.js b/src/ffmpeg.js
index 6223936..418515d 100644
--- a/src/ffmpeg.js
+++ b/src/ffmpeg.js
@@ -69,13 +69,18 @@ export function frameEncoder({ fps, out, draft }) {
}
// Two-pass loudness normalization to `target` LUFS, peak-limited.
-export async function normalizeLoudness(input, output, target = -14) {
+export async function normalizeLoudness(input, output, target = -14, sampleRate = 48000) {
const base = `I=${target}:TP=-1.5:LRA=11`;
const pass1 = await ffmpeg(['-i', input, '-af', `loudnorm=${base}:print_format=json`, '-f', 'null', '-']);
const json = pass1.stderr.slice(pass1.stderr.lastIndexOf('{'), pass1.stderr.lastIndexOf('}') + 1);
const m = JSON.parse(json);
+ // Silence and very short clips have no measurable integrated loudness.
+ if (!Number.isFinite(Number(m.input_i))) {
+ await ffmpeg(['-i', input, '-ar', String(sampleRate), '-c:a', 'pcm_s16le', output]);
+ return { inputI: Number(m.input_i), inputTP: Number(m.input_tp) };
+ }
const measured = `measured_I=${m.input_i}:measured_TP=${m.input_tp}:measured_LRA=${m.input_lra}:measured_thresh=${m.input_thresh}:offset=${m.target_offset}`;
- await ffmpeg(['-i', input, '-af', `loudnorm=${base}:${measured}:linear=true:print_format=summary`, '-ar', '48000', '-c:a', 'pcm_s16le', output]);
+ await ffmpeg(['-i', input, '-af', `loudnorm=${base}:${measured}:linear=true:print_format=summary`, '-ar', String(sampleRate), '-c:a', 'pcm_s16le', output]);
return { inputI: Number(m.input_i), inputTP: Number(m.input_tp) };
}
diff --git a/src/images.js b/src/images.js
index 0885578..1575a87 100644
--- a/src/images.js
+++ b/src/images.js
@@ -4,7 +4,7 @@
import fs from 'node:fs';
import path from 'node:path';
import crypto from 'node:crypto';
-import { ROOT } from './server.js';
+import { findEnv } from './env.js';
import { ProjectError } from './project.js';
export const IMAGE_MODELS = {
@@ -17,18 +17,8 @@ const ASPECTS = ['16:9', '9:16', '1:1', '4:3', '3:4', '3:2', '2:3'];
export const DEFAULT_STYLE =
'Flat illustration for a friendly how-to video, soft colors, simple shapes, clear subject in the middle, plain light background, no text, no letters, no numbers, no logos, no watermark.';
-function readEnvFile(file) {
- if (!fs.existsSync(file)) return null;
- for (const line of fs.readFileSync(file, 'utf8').split('\n')) {
- const m = /^\s*(?:export\s+)?OPENROUTER_API_KEY\s*=\s*(.*)\s*$/.exec(line);
- if (m) return m[1].replace(/^["']|["']$/g, '').trim() || null;
- }
- return null;
-}
-
export function findKey(projectDir) {
- if (process.env.OPENROUTER_API_KEY) return process.env.OPENROUTER_API_KEY;
- return readEnvFile(path.join(projectDir, '.env')) || readEnvFile(path.join(ROOT, '.env'));
+ return findEnv('OPENROUTER_API_KEY', projectDir);
}
export function resolveModel(name) {
diff --git a/src/project.js b/src/project.js
index 3b64a9b..e8fabbd 100644
--- a/src/project.js
+++ b/src/project.js
@@ -162,10 +162,31 @@ export function normalizeConfig(raw, dir = '.') {
const unknown = Object.keys(raw).filter((k) => !(k in DEFAULTS) && !k.startsWith('_'));
if (unknown.length) fail(`video.json has unknown setting(s): ${unknown.join(', ')}. Known: ${Object.keys(DEFAULTS).join(', ')}`);
if (!THEMES.includes(cfg.theme) && !lookPath(cfg.theme)) fail(`theme must be one of ${THEMES.join(', ')} or a relative path to a .json look file, not "${cfg.theme}"`);
- if (!VOICES[cfg.voice]) fail(`voice "${cfg.voice}" does not exist. Run "explainroo voices" for the list.`);
+ if (typeof cfg.voice === 'object') {
+ const v = cfg.voice;
+ if (!v || Array.isArray(v)) fail('voice must be a Kokoro id or an object with provider');
+ const keys = { kokoro: ['provider', 'id'], elevenlabs: ['provider', 'id', 'model'], recording: ['provider', 'src'] };
+ if (typeof v.provider !== 'string' || !Object.hasOwn(keys, v.provider)) fail('voice.provider must be kokoro, elevenlabs or recording');
+ const extra = Object.keys(v).filter((k) => !keys[v.provider].includes(k));
+ if (extra.length) fail(`voice has unknown setting(s): ${extra.join(', ')}. Known: ${keys[v.provider].join(', ')}`);
+ if (v.provider === 'recording') {
+ if (!(typeof v.src === 'string' && /^assets\/.+/.test(v.src) && !v.src.split('/').includes('..') && !v.src.includes('\\'))) {
+ fail('voice.src must be a file or folder inside assets/, with no ..');
+ }
+ } else if (!(typeof v.id === 'string' && v.id.trim())) fail('voice.id is required');
+ if (v.provider === 'kokoro') cfg.voice = v.id;
+ if (v.provider === 'elevenlabs') {
+ if (v.model !== undefined && !(typeof v.model === 'string' && v.model.trim())) fail('voice.model must be an ElevenLabs model id');
+ cfg.voice = { ...v, model: v.model ?? 'eleven_multilingual_v2' };
+ }
+ }
+ if (typeof cfg.voice === 'string') {
+ if (!Object.hasOwn(VOICES, cfg.voice)) fail(`voice "${cfg.voice}" does not exist. Run "explainroo voices" for the list.`);
+ } else if (!cfg.voice || typeof cfg.voice !== 'object') fail('voice must be a Kokoro id or an object with provider');
+ if (typeof cfg.voice !== 'string' && raw.speed === undefined) cfg.speed = 1;
if (!(cfg.speed >= 0.6 && cfg.speed <= 1.6)) fail('speed must be between 0.6 and 1.6');
if (!(cfg.pace >= 0.7 && cfg.pace <= 1.6)) fail('pace must be between 0.7 and 1.6 (1 is normal, 1.3 is 30% faster)');
- if (cfg.speed * cfg.pace > 1.8) fail(`speed ${cfg.speed} times pace ${cfg.pace} makes the voice too fast; keep speed x pace at 1.8 or below`);
+ if (typeof cfg.voice === 'string' && cfg.speed * cfg.pace > 1.8) fail(`speed ${cfg.speed} times pace ${cfg.pace} makes the voice too fast; keep speed x pace at 1.8 or below`);
if (![24, 25, 30, 50, 60].includes(cfg.fps)) fail('fps must be 24, 25, 30, 50 or 60');
if (!TRANSITIONS.includes(cfg.transition)) fail(`transition must be one of ${TRANSITIONS.join(', ')}`);
if (![true, false, 'minimal'].includes(cfg.sfx)) fail('sfx must be true, false or "minimal"');
diff --git a/src/qa.js b/src/qa.js
index 79dd873..29aa331 100644
--- a/src/qa.js
+++ b/src/qa.js
@@ -17,7 +17,7 @@ export async function check(project, { log = () => {}, step = 0.25, viewWidth =
const v = voices[sc.id];
const suspicious = v ? v.unmatched.filter((w) => w.replace(/[^\p{L}\p{N}]/gu, '').length > 3) : [];
if (v && v.duration && (v.matchRate < 0.9 || suspicious.length)) {
- issues.push({ level: 'warn', scene: sc.id, t: null, message: `the voice check could not confirm ${suspicious.length ? suspicious.slice(0, 8).map((w) => `"${w}"`).join(', ') : `${Math.round((1 - v.matchRate) * 100)}% of the words`}. It heard: "${v.transcript.slice(0, 200)}". If a word is mispronounced, write it as {shown|spoken} in script.md.` });
+ issues.push({ level: 'warn', scene: sc.id, t: null, message: `the voice check could not confirm ${suspicious.length ? suspicious.slice(0, 8).map((w) => `"${w}"`).join(', ') : `${Math.round((1 - v.matchRate) * 100)}% of the words`}. It heard: "${v.transcript.slice(0, 200)}". ${project.config.voice.provider === 'recording' ? 'The recording does not match script.md here. Make script.md say what was actually said.' : 'If a word is mispronounced, write it as {shown|spoken} in script.md.'}` });
}
if (v && v.duration > 40) issues.push({ level: 'hint', scene: sc.id, t: null, message: `the narration runs ${v.duration.toFixed(0)}s; long scenes are easier to follow when split` });
}
diff --git a/src/recordings.js b/src/recordings.js
new file mode 100644
index 0000000..61da60d
--- /dev/null
+++ b/src/recordings.js
@@ -0,0 +1,89 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import crypto from 'node:crypto';
+import { ProjectError } from './project.js';
+import { alignWords } from './align.js';
+import { ffmpeg, probe, normalizeLoudness } from './ffmpeg.js';
+import { readWav, writeWav } from './wav.js';
+
+const AUDIO_FILE = /\.(wav|mp3|m4a|aac|flac|ogg|opus)$/i;
+// Four Kokoro scenes from how-dns-works measured -22.0 to -22.5 LUFS.
+const RECORDING_LUFS = -22.3;
+
+export function recordingFiles(project) {
+ const src = project.config.voice.src;
+ const file = path.join(project.dir, src);
+ const assets = path.join(fs.realpathSync(project.dir), 'assets');
+ const inside = (p) => fs.realpathSync(p).startsWith(assets + path.sep);
+ if (!fs.existsSync(file)) throw new ProjectError(`${src} is missing. Put the recording in assets/ and check voice.src.`);
+ if (!inside(file)) throw new ProjectError('recordings must stay inside assets/, including symlinks');
+ const folder = fs.statSync(file).isDirectory();
+ if (!folder && !AUDIO_FILE.test(file)) throw new ProjectError('recordings must be wav, mp3, m4a, aac, flac, ogg or opus files');
+ const files = {};
+ const hashes = new Map();
+ const missing = [];
+ const names = folder ? fs.readdirSync(file).sort() : [];
+ for (const scene of project.script.scenes) {
+ if (!scene.units.some((u) => u.type === 'word')) continue;
+ const name = names.find((n) => path.parse(n).name === scene.id && AUDIO_FILE.test(n) && fs.existsSync(path.join(file, n)) && fs.statSync(path.join(file, n)).isFile());
+ if (folder && !name) { missing.push(scene.id); continue; }
+ const p = folder ? path.join(file, name) : file;
+ if (!inside(p)) throw new ProjectError('recordings must stay inside assets/, including symlinks');
+ if (!hashes.has(p)) hashes.set(p, crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex'));
+ files[scene.id] = { file: p, hash: hashes.get(p) };
+ }
+ if (missing.length) throw new ProjectError(`missing scene recordings in ${src}: ${missing.join(', ')}. Add <scene-id>.<ext> for each scene with narration.`);
+ return { folder, files };
+}
+
+function tempo(speed) {
+ const filters = [];
+ while (speed < 0.5) { filters.push('atempo=0.5'); speed /= 0.5; }
+ while (speed > 2) { filters.push('atempo=2'); speed /= 2; }
+ if (speed !== 1) filters.push(`atempo=${speed}`);
+ return filters;
+}
+
+export async function decodeRecording(file, build, speed) {
+ const info = await probe(file);
+ const stream = info.streams.find((s) => s.codec_type === 'audio');
+ if (!stream) throw new ProjectError('the recording has no audio. Choose an audio file in voice.src.');
+ const sampleRate = Math.min(48000, Number(stream.sample_rate));
+ const dir = fs.mkdtempSync(path.join(build, 'recording-'));
+ try {
+ const wav = path.join(dir, 'decoded.wav');
+ const filters = tempo(speed);
+ await ffmpeg(['-loglevel', 'error', '-i', file, '-vn', '-ac', '1', '-ar', String(sampleRate),
+ ...(filters.length ? ['-af', filters.join(',')] : []), '-c:a', 'pcm_s16le', wav]);
+ return readWav(wav);
+ } finally {
+ fs.rmSync(dir, { recursive: true, force: true });
+ }
+}
+
+export async function saveRecording(file, samples, sampleRate) {
+ const dir = fs.mkdtempSync(path.join(path.dirname(file), 'recording-'));
+ try {
+ const input = path.join(dir, 'input.wav');
+ writeWav(input, samples, sampleRate);
+ await normalizeLoudness(input, file, RECORDING_LUFS, sampleRate);
+ } finally {
+ fs.rmSync(dir, { recursive: true, force: true });
+ }
+}
+
+export function recordingCuts(scenes, heard, duration) {
+ const spoken = scenes.map((scene) => ({ scene, words: scene.units.filter((u) => u.type === 'word') })).filter((s) => s.words.length);
+ const aligned = alignWords(spoken.flatMap((s) => s.words), heard, duration).words;
+ let offset = 0;
+ const spans = spoken.map(({ scene, words }) => {
+ const times = aligned.slice(offset, offset + words.length);
+ offset += words.length;
+ return { scene, words: times, first: times[0].start, last: times.at(-1).end };
+ });
+ return spans.map((s, i) => {
+ const left = i ? (spans[i - 1].last + s.first) / 2 : 0;
+ const right = i + 1 < spans.length ? (s.last + spans[i + 1].first) / 2 : duration;
+ return { scene: s.scene, words: s.words, start: Math.max(0, left, s.first - 0.15), end: Math.min(duration, right, s.last + 0.15) };
+ });
+}
diff --git a/src/voice.js b/src/voice.js
index 4a204f2..1f54bc3 100644
--- a/src/voice.js
+++ b/src/voice.js
@@ -8,10 +8,12 @@ import { speechChunks, resolveMarks } from './script.js';
import { alignWords } from './align.js';
import { loadTTS, transcribeWords, TTS_MODEL, ASR_MODEL, TTS_SAMPLE_RATE, ttsDtype } from './models.js';
import { writeWav, readWav, resample, trimSilence } from './wav.js';
+import { elevenSpeech } from './elevenlabs.js';
+import { recordingFiles, decodeRecording, saveRecording, recordingCuts } from './recordings.js';
const PIPELINE = 7;
-function sceneHash(scene, chunks, config) {
+export function sceneHash(scene, chunks, config, recording = null) {
return crypto
.createHash('sha256')
.update(JSON.stringify({
@@ -21,10 +23,12 @@ function sceneHash(scene, chunks, config) {
dtype: ttsDtype(),
voice: config.voice,
speed: config.speed * config.pace,
- chunks: chunks.map((c) => [c.gapBefore, c.text]),
+ chunks: recording ? scene.spoken : chunks.map((c) => [c.gapBefore, c.text]),
// Markers and the shown text of each word are saved with the timings,
// so moving a [#marker] or changing {shown|spoken} makes the scene again.
- layout: scene.units.map((u) => (u.type === 'mark' ? `#${u.name}` : u.type === 'word' ? u.display : '')).join('|'),
+ layout: (recording ? scene.units.filter((u) => u.type !== 'pause') : scene.units).map((u) => (u.type === 'mark' ? `#${u.name}` : u.type === 'word' ? u.display : '')).join('|'),
+ // Added last and only for recordings, so Kokoro scenes keep their cache.
+ ...(recording ? { recording } : {}),
}))
.digest('hex')
.slice(0, 16);
@@ -48,11 +52,17 @@ export function readVoice(project, id) {
export async function synthesize(project, { force = false, log = () => {}, only = null } = {}) {
fs.mkdirSync(project.paths.voice, { recursive: true });
const { config } = project;
+ const provider = config.voice.provider || 'kokoro';
+ const recordings = provider === 'recording' ? recordingFiles(project) : null;
const result = {};
const todo = [];
for (const scene of project.script.scenes) {
const chunks = speechChunks(scene.units, { sentenceGap: config.sentenceGap, paragraphGap: config.paragraphGap, pace: config.pace });
- const hash = sceneHash(scene, chunks, config);
+ const recording = recordings?.files[scene.id];
+ const hash = sceneHash(scene, chunks, config, recording ? {
+ content: recording.hash,
+ scenes: recordings.folder ? null : project.script.scenes.map((s) => s.spoken),
+ } : null);
const cached = readVoice(project, scene.id);
const hasAudio = !chunks.length || fs.existsSync(voicePaths(project, scene.id).wav);
if (!force && cached && cached.hash === hash && hasAudio) {
@@ -66,13 +76,27 @@ export async function synthesize(project, { force = false, log = () => {}, only
if (!todo.length) return result;
const needsSpeech = todo.some((t) => t.chunks.length);
- const tts = needsSpeech ? await loadTTS({ log }) : null;
+ const tts = needsSpeech && provider === 'kokoro' ? await loadTTS({ log }) : null;
+ const wanted = config.speed * config.pace;
+ let take = null;
+ let cuts = [];
+ let heardTake = null;
+ if (needsSpeech && recordings && !recordings.folder) {
+ log('transcribing the recording and finding scene boundaries');
+ take = await decodeRecording(Object.values(recordings.files)[0].file, project.paths.build, wanted);
+ heardTake = await transcribeWords(resample(take.samples, take.sampleRate, 16000), { log });
+ cuts = recordingCuts(project.script.scenes, heardTake.words, take.samples.length / take.sampleRate);
+ }
for (const { scene, chunks, hash } of todo) {
const t0 = Date.now();
- const info = await synthesizeScene(project, scene, chunks, hash, tts, log);
+ const info = recordings && chunks.length
+ ? await recordingScene(project, scene, hash, recordings.files[scene.id], take, cuts.find((c) => c.scene.id === scene.id), heardTake, log)
+ : await synthesizeScene(project, scene, chunks, hash, tts, log);
result[scene.id] = info;
if (chunks.length) {
- const warn = info.matchRate < 0.85 ? `, check pronunciation (${Math.round(info.matchRate * 100)}% of words confirmed)` : '';
+ const warn = info.matchRate < 0.85 ? (recordings
+ ? `, recording does not match script.md here (${Math.round(info.matchRate * 100)}% of words confirmed). Make script.md say what was actually said.`
+ : `, check pronunciation (${Math.round(info.matchRate * 100)}% of words confirmed)`) : '';
log(`voice ${scene.id}: ${info.duration.toFixed(1)}s in ${((Date.now() - t0) / 1000).toFixed(1)}s${warn}`);
}
}
@@ -105,7 +129,9 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {
pieces.push(new Float32Array(Math.round(gap * sr)));
cursor += Math.round(gap * sr);
}
- const audio = await tts.generate(chunk.text, { voice: config.voice, speed: config.speed * config.pace });
+ const audio = config.voice.provider === 'elevenlabs'
+ ? await elevenSpeech(project, scene.id, chunk.text, { previous: chunks[ci - 1]?.text, next: chunks[ci + 1]?.text })
+ : await tts.generate(chunk.text, { voice: config.voice, speed: config.speed * config.pace });
const samples = trimSilence(audio.audio, audio.sampling_rate || sr);
const start = cursor / sr;
const dur = samples.length / sr;
@@ -154,6 +180,49 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {
return info;
}
+async function recordingScene(project, scene, hash, source, take, cut, heardTake, log) {
+ const audio = take || await decodeRecording(source.file, project.paths.build, project.config.speed * project.config.pace);
+ const sr = audio.sampleRate;
+ let from = cut ? Math.floor(cut.start * sr) : 0;
+ let to = cut ? Math.ceil(cut.end * sr) : audio.samples.length;
+ let samples = audio.samples.subarray(from, to);
+ if (cut) {
+ // Whisper can include the silence between scenes in a word's timestamp.
+ const trimmed = trimSilence(samples, sr, { keep: 0.15 });
+ if (trimmed.length) {
+ from += (trimmed.byteOffset - samples.byteOffset) / 4;
+ samples = trimmed;
+ to = from + samples.length;
+ }
+ }
+ const duration = samples.length / sr;
+ const local = (w) => {
+ const start = Math.max(0, Math.min(duration, w.start - from / sr));
+ return { ...w, start, end: Math.min(duration, Math.max(start + 0.05, w.end - from / sr)) };
+ };
+ const heard = heardTake ? {
+ words: heardTake.words.filter((w) => w.start >= cut.start && w.start < cut.end).map(local),
+ } : await transcribeWords(resample(samples, sr, 16000), { log });
+ const units = scene.units.filter((u) => u.type === 'word');
+ const aligned = alignWords(units, heard.words, duration);
+ const times = cut ? cut.words.map(local) : aligned.words;
+ const wordTimes = [];
+ let wi = 0;
+ scene.units.forEach((u, i) => { if (u.type === 'word') wordTimes[i] = times[wi++]; });
+ const files = voicePaths(project, scene.id);
+ await saveRecording(files.wav, samples, sr);
+ const info = {
+ hash, id: scene.id, duration: round(duration),
+ words: units.map((u, i) => ({ text: u.display, spoken: u.spoken, start: round(times[i].start), end: round(times[i].end), matched: aligned.words[i].matched })),
+ marks: Object.fromEntries(Object.entries(resolveMarks(scene.units, wordTimes)).map(([k, v]) => [k, round(v)])),
+ chunks: [{ start: 0, end: round(duration), text: scene.spoken }],
+ transcript: heard.words.map((w) => w.text).join(' '),
+ matchRate: round(aligned.matchRate), unmatched: units.filter((u, i) => !aligned.words[i].matched).map((u) => u.display),
+ };
+ fs.writeFileSync(files.json, JSON.stringify(info, null, 2));
+ return info;
+}
+
function round(v) {
return Math.round(v * 1000) / 1000;
}
diff --git a/test/voice-providers.test.js b/test/voice-providers.test.js
new file mode 100644
index 0000000..7edc202
--- /dev/null
+++ b/test/voice-providers.test.js
@@ -0,0 +1,338 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { normalizeConfig, loadProject, ProjectError } from '../src/project.js';
+import { parseScript, speechChunks } from '../src/script.js';
+import { findEnv, readEnvFile } from '../src/env.js';
+import { findKey } from '../src/images.js';
+import { elevenSpeech, elevenSpeed, decodePCM, elevenVoices, elevenUsage } from '../src/elevenlabs.js';
+import { recordingCuts, recordingFiles, decodeRecording, saveRecording } from '../src/recordings.js';
+import { sceneHash } from '../src/voice.js';
+import { readWav, writeWav } from '../src/wav.js';
+import { loudness } from '../src/ffmpeg.js';
+import { main } from '../src/cli.js';
+
+function temp(t) {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-voices-'));
+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
+ return dir;
+}
+
+function project(t, voice = { provider: 'elevenlabs', id: 'test-voice' }) {
+ const dir = temp(t);
+ fs.mkdirSync(path.join(dir, 'assets'));
+ fs.writeFileSync(path.join(dir, 'video.json'), JSON.stringify({ voice }));
+ fs.writeFileSync(path.join(dir, 'script.md'), '# Test\n## first\nHello.\n## silent\n## second\nGoodbye.');
+ fs.writeFileSync(path.join(dir, 'scenes.js'), 'export default { first() {}, silent() {}, second() {} };');
+ return loadProject(dir);
+}
+
+function setEnv(t, name, value) {
+ const previous = process.env[name];
+ process.env[name] = value;
+ t.after(() => {
+ if (previous === undefined) delete process.env[name];
+ else process.env[name] = previous;
+ });
+}
+
+function mockAPI(t, fn) {
+ t.mock.method(globalThis, 'fetch', fn);
+ setEnv(t, 'ELEVENLABS_API_KEY', 'secret-test-key');
+}
+
+test('voice config accepts strings and all providers, with provider speed defaults', () => {
+ assert.equal(normalizeConfig({}).voice, 'af_heart');
+ assert.equal(normalizeConfig({ voice: 'am_michael' }).speed, 0.9);
+ assert.equal(normalizeConfig({ voice: { provider: 'kokoro', id: 'af_heart' } }).voice, 'af_heart');
+ assert.equal(normalizeConfig({ voice: { provider: 'kokoro', id: 'af_heart' } }).speed, 0.9);
+ const cfg = normalizeConfig({ voice: { provider: 'elevenlabs', id: 'voice' }, pace: 1.6 });
+ assert.deepEqual(cfg.voice, { provider: 'elevenlabs', id: 'voice', model: 'eleven_multilingual_v2' });
+ assert.equal(cfg.speed, 1);
+ assert.equal(normalizeConfig({ voice: { provider: 'elevenlabs', id: 'voice', model: 'eleven_flash_v2_5' }, speed: 0.8 }).speed, 0.8);
+ for (const src of ['assets/narration.wav', 'assets/narration/', 'assets/take.m4a']) {
+ assert.equal(normalizeConfig({ voice: { provider: 'recording', src } }).speed, 1);
+ assert.equal(normalizeConfig({ voice: { provider: 'recording', src }, speed: 1.2 }).speed, 1.2);
+ }
+ assert.throws(() => normalizeConfig({ speed: 1.6, pace: 1.6 }), /too fast/);
+});
+
+test('voice config rejects bad providers, missing ids, unknown keys and unsafe paths', () => {
+ for (const voice of [null, [], 1, {}, { provider: 'bad' }, { provider: 'toString' }, { provider: ['kokoro'], id: 'af_heart' },
+ { provider: 'kokoro' }, { provider: 'kokoro', id: 'bad' }, { provider: 'kokoro', id: 'toString' },
+ { provider: 'kokoro', id: 'af_heart', model: 'x' }, { provider: 'elevenlabs' },
+ { provider: 'elevenlabs', id: '' }, { provider: 'elevenlabs', id: 'v', model: 4 },
+ { provider: 'elevenlabs', id: 'v', key: 'secret' }, { provider: 'recording' },
+ { provider: 'recording', src: '../take.wav' }, { provider: 'recording', src: 'assets/../take.wav' },
+ { provider: 'recording', src: 'assets/..\\take.wav' }, { provider: 'recording', src: 'assets/a.wav', id: 'x' }]) {
+ assert.throws(() => normalizeConfig({ voice }), ProjectError);
+ }
+});
+
+test('shared env reader keeps environment, project, root precedence and quoted exports', (t) => {
+ const dir = temp(t);
+ const root = temp(t);
+ const name = 'EXPLAINROO_TEST_KEY';
+ const file = path.join(dir, '.env');
+ setEnv(t, name, 'environment');
+ fs.writeFileSync(path.join(root, '.env'), `${name}=root\n`);
+ fs.writeFileSync(file, `UNRELATED=x\n export ${name} = "project" \nOPENROUTER_API_KEY='image-key'\n`);
+ assert.equal(readEnvFile(file, name), 'project');
+ assert.equal(findEnv(name, dir, root), 'environment');
+ process.env[name] = '';
+ assert.equal(findEnv(name, dir, root), 'project');
+ setEnv(t, 'OPENROUTER_API_KEY', '');
+ assert.equal(findKey(dir), 'image-key');
+ fs.writeFileSync(file, `${name}=''\n`);
+ assert.equal(findEnv(name, dir, root), 'root');
+ fs.rmSync(file);
+ assert.equal(readEnvFile(file, name), null);
+});
+
+test('ElevenLabs requests PCM, neighbouring text, speed, and logs usage without secrets', async (t) => {
+ const p = project(t);
+ p.config.speed = 1.1;
+ p.config.pace = 1.3;
+ const pcm = Buffer.alloc(6);
+ pcm.writeInt16LE(-32768, 0);
+ pcm.writeInt16LE(0, 2);
+ pcm.writeInt16LE(16384, 4);
+ mockAPI(t, async (url, opts) => {
+ assert.equal(url, 'https://api.elevenlabs.io/v1/text-to-speech/test-voice?output_format=pcm_24000');
+ assert.equal(opts.method, 'POST');
+ assert.equal(opts.headers['xi-api-key'], 'secret-test-key');
+ assert.deepEqual(JSON.parse(opts.body), { text: 'Hello.', model_id: 'eleven_multilingual_v2', voice_settings: { speed: 1.2 }, previous_text: 'Before.', next_text: 'After.' });
+ return new Response(pcm);
+ });
+ const audio = await elevenSpeech(p, 'first', 'Hello.', { previous: 'Before.', next: 'After.' });
+ assert.equal(audio.sampling_rate, 24000);
+ assert.deepEqual([...audio.audio], [-1, 0, 0.5]);
+ const usage = elevenUsage(p);
+ assert.equal(usage.length, 1);
+ assert.deepEqual(Object.keys(usage[0]), ['time', 'scene', 'characters', 'model', 'voice_id']);
+ assert.equal(usage[0].scene, 'first');
+ assert.equal(usage[0].characters, 6);
+ assert.equal(usage[0].voice_id, 'test-voice');
+ assert.equal(usage[0].model, 'eleven_multilingual_v2');
+ assert.ok(Number.isFinite(Date.parse(usage[0].time)));
+ assert.ok(!JSON.stringify(usage).includes('secret-test-key'));
+ assert.throws(() => decodePCM(Buffer.alloc(1)), /unreadable audio/);
+ assert.throws(() => decodePCM(Buffer.alloc(0)), /unreadable audio/);
+ assert.equal(elevenSpeed(0.42), 0.7);
+ assert.equal(elevenSpeed(2.56), 1.2);
+ assert.equal(elevenSpeed(1), 1);
+});
+
+test('ElevenLabs keeps the saved voice settings at normal speed', async (t) => {
+ const p = project(t);
+ p.config.speed = 1;
+ p.config.pace = 1;
+ mockAPI(t, async (url, opts) => {
+ assert.equal(JSON.parse(opts.body).voice_settings, undefined);
+ return new Response(Buffer.alloc(2));
+ });
+ await elevenSpeech(p, 'first', 'Hello.');
+});
+
+test('ElevenLabs omits unsupported continuity for v3 and reports unreadable downloads safely', async (t) => {
+ const p = project(t, { provider: 'elevenlabs', id: 'test-voice', model: 'eleven_v3' });
+ mockAPI(t, async (url, opts) => {
+ const body = JSON.parse(opts.body);
+ assert.equal(body.model_id, 'eleven_v3');
+ assert.equal(body.previous_text, undefined);
+ assert.equal(body.next_text, undefined);
+ return new Response(Buffer.alloc(2));
+ });
+ await elevenSpeech(p, 'first', 'Hello.', { previous: 'Before.', next: 'After.' });
+ globalThis.fetch = async () => ({ ok: true, arrayBuffer: async () => { throw new Error('secret-test-key'); } });
+ await assert.rejects(() => elevenSpeech(p, 'first', 'Hello.'), (e) => {
+ assert.match(e.message, /download failed/);
+ assert.ok(!e.message.includes('secret-test-key'));
+ return true;
+ });
+ assert.equal(elevenUsage(p).length, 2);
+});
+
+test('ElevenLabs gives friendly errors without echoing the key or server body', async (t) => {
+ const p = project(t);
+ mockAPI(t, async () => { throw new Error('secret-test-key'); });
+ const error = async (pattern) => assert.rejects(() => elevenSpeech(p, 'first', 'Hello.'), (e) => {
+ assert.match(e.message, pattern);
+ assert.ok(!e.message.includes('secret-test-key'));
+ return true;
+ });
+ await error(/connection/);
+ for (const [http, status, pattern] of [[401, 'invalid_api_key', /rejected/], [403, '', /permissions/],
+ [404, 'voice_not_found', /voice id/], [400, 'voice_not_found', /voice id/], [401, 'quota_exceeded', /no characters/],
+ [402, '', /no characters/], [422, '', /voice.model/]]) {
+ globalThis.fetch = async () => new Response(JSON.stringify({ detail: { status, message: 'secret-test-key' } }), { status: http });
+ await error(pattern);
+ }
+ assert.deepEqual(elevenUsage(p), []);
+ process.env.ELEVENLABS_API_KEY = '';
+ await error(/Ask the user.*elevenlabs.io.*\.env/);
+});
+
+test('ElevenLabs retries 429 and server failures, but stops after three attempts', async (t) => {
+ const p = project(t);
+ let calls = 0;
+ mockAPI(t, async () => {
+ calls++;
+ return calls < 3 ? new Response('{}', { status: calls === 1 ? 429 : 503 }) : new Response(Buffer.alloc(2));
+ });
+ await elevenSpeech(p, 'first', 'Hello.');
+ assert.equal(calls, 3);
+ assert.equal(elevenUsage(p).length, 1);
+ calls = 0;
+ globalThis.fetch = async () => { calls++; return new Response('secret-test-key', { status: 500 }); };
+ await assert.rejects(() => elevenSpeech(p, 'first', 'Hello.'), /busy/);
+ assert.equal(calls, 3);
+ assert.equal(elevenUsage(p).length, 1);
+});
+
+test('ElevenLabs voice list paginates and CLI prints names, ids and labels with JSON support', async (t) => {
+ const dir = temp(t);
+ let calls = 0;
+ mockAPI(t, async (url) => {
+ calls++;
+ assert.equal(url.pathname, '/v2/voices');
+ assert.equal(url.searchParams.get('page_size'), '100');
+ const page2 = url.searchParams.has('next_page_token');
+ if (page2) assert.equal(url.searchParams.get('next_page_token'), 'next');
+ return Response.json({ voices: [{ name: page2 ? 'Bob' : 'Alice', voice_id: page2 ? 'b' : 'a', labels: { accent: 'American' } }], has_more: !page2, next_page_token: page2 ? null : 'next' });
+ });
+ assert.deepEqual(await elevenVoices(dir), [{ name: 'Alice', id: 'a', labels: { accent: 'American' } }, { name: 'Bob', id: 'b', labels: { accent: 'American' } }]);
+ assert.equal(calls, 2);
+ let output = '';
+ t.mock.method(process.stdout, 'write', (s) => { output += s; return true; });
+ assert.equal(await main(['voices', '--elevenlabs', dir, '--json']), 0);
+ assert.equal(JSON.parse(output).voices[1].id, 'b');
+ output = '';
+ assert.equal(await main(['voices', dir, '--elevenlabs']), 0);
+ assert.match(output, /Alice a accent: American/);
+ output = '';
+ await main(['voices']);
+ assert.match(output, /af_heart/);
+ assert.match(output, /voices --elevenlabs.*recording/);
+});
+
+test('cached CLI voice output reports recording mismatches and ElevenLabs speed limits', async (t) => {
+ const p = project(t, { provider: 'recording', src: 'assets/take.wav' });
+ writeWav(path.join(p.dir, 'assets/take.wav'), new Float32Array(24000), 24000);
+ const recordings = recordingFiles(p);
+ fs.mkdirSync(p.paths.voice, { recursive: true });
+ const cache = (recording) => {
+ for (const scene of p.script.scenes) {
+ const chunks = speechChunks(scene.units, p.config);
+ const hash = sceneHash(scene, chunks, p.config, recording && chunks.length
+ ? { content: recordings.files[scene.id].hash, scenes: p.script.scenes.map((s) => s.spoken) } : null);
+ const info = { hash, id: scene.id, duration: chunks.length ? 1 : 0, words: [], marks: {}, chunks: [], transcript: 'Different words.', matchRate: chunks.length ? 0 : 1, unmatched: chunks.length ? [scene.spoken] : [] };
+ fs.writeFileSync(path.join(p.paths.voice, scene.id + '.json'), JSON.stringify(info));
+ if (chunks.length) writeWav(path.join(p.paths.voice, scene.id + '.wav'), new Float32Array(24000), 24000);
+ }
+ };
+ cache(true);
+ let output = '';
+ t.mock.method(process.stdout, 'write', (s) => { output += s; return true; });
+ assert.equal(await main(['voice', p.dir, '--json']), 0);
+ assert.match(JSON.parse(output).scenes[0].message, /recording does not match.*actually said/);
+ fs.writeFileSync(path.join(p.dir, 'video.json'), JSON.stringify({ voice: { provider: 'elevenlabs', id: 'v' }, pace: 1.3 }));
+ p.config = loadProject(p.dir).config;
+ cache(false);
+ output = '';
+ assert.equal(await main(['voice', p.dir, '--json']), 0);
+ assert.equal(JSON.parse(output).speed, 1.2);
+ assert.equal(JSON.parse(output).requested_speed, 1.3);
+ assert.equal(JSON.parse(output).characters, 0);
+ output = '';
+ assert.equal(await main(['voice', p.dir]), 0);
+ assert.equal((output.match(/outside 0.7 to 1.2/g) || []).length, 1);
+});
+
+test('single-take cuts retain internal pauses and margins, skip silent scenes and avoid overlap', () => {
+ const scenes = parseScript('## first\n[#hello] Hello [pause 9] {world|world}.\n## silent\n## second\nGoodbye now.').scenes;
+ const heard = [{ text: 'Hello', start: 1, end: 1.5 }, { text: 'world', start: 3, end: 3.5 },
+ { text: 'Goodbye', start: 5, end: 5.5 }, { text: 'now', start: 6, end: 6.5 }];
+ const cuts = recordingCuts(scenes, heard, 8);
+ assert.equal(cuts.length, 2);
+ assert.equal(cuts[0].scene.id, 'first');
+ assert.equal(cuts[0].start, 0.85);
+ assert.equal(cuts[0].end, 3.65);
+ assert.deepEqual(cuts[0].words, [{ start: 1, end: 1.5, matched: true }, { start: 3, end: 3.5, matched: true }]);
+ assert.equal(cuts[1].start, 4.85);
+ assert.equal(cuts[1].end, 6.65);
+ heard[2].start = 3.6;
+ const close = recordingCuts(scenes, heard, 6.5);
+ assert.equal(close[0].end, 3.55);
+ assert.equal(close[1].start, 3.55);
+ assert.equal(close[1].end, 6.5);
+ assert.deepEqual(recordingCuts(scenes.filter((s) => s.id === 'silent'), [], 8), []);
+});
+
+test('recording folders report all missing scenes, validate files and stay inside assets', (t) => {
+ const p = project(t, { provider: 'recording', src: 'assets/narration/' });
+ const folder = path.join(p.dir, 'assets/narration');
+ fs.mkdirSync(folder);
+ assert.throws(() => recordingFiles(p), /missing scene recordings.*first, second/);
+ fs.writeFileSync(path.join(folder, 'first.mp3'), 'first');
+ fs.writeFileSync(path.join(folder, 'second.M4A'), 'second');
+ const files = recordingFiles(p);
+ assert.equal(files.folder, true);
+ assert.deepEqual(Object.keys(files.files), ['first', 'second']);
+ fs.writeFileSync(path.join(folder, 'first.mp3'), 'replaced');
+ const changed = recordingFiles(p);
+ assert.notEqual(changed.files.first.hash, files.files.first.hash);
+ assert.equal(changed.files.second.hash, files.files.second.hash);
+ fs.rmSync(path.join(folder, 'first.mp3'));
+ const outside = path.join(temp(t), 'outside.wav');
+ fs.writeFileSync(outside, 'outside');
+ fs.symlinkSync(outside, path.join(folder, 'first.wav'));
+ assert.throws(() => recordingFiles(p), /including symlinks/);
+ p.config.voice.src = 'assets/no.wav';
+ assert.throws(() => recordingFiles(p), /missing/);
+ p.config.voice.src = 'assets/bad.txt';
+ fs.writeFileSync(path.join(p.dir, 'assets/bad.txt'), 'not audio');
+ assert.throws(() => recordingFiles(p), /wav, mp3/);
+});
+
+test('scene cache includes audio settings and recording contents, ignores recording gaps', () => {
+ const scene = parseScript('## first\n[#a] Hello. Goodbye.').scenes[0];
+ const cfg = normalizeConfig({});
+ const chunks = speechChunks(scene.units);
+ const hash = sceneHash(scene, chunks, cfg);
+ assert.notEqual(hash, sceneHash(scene, chunks, { ...cfg, voice: 'am_michael' }));
+ assert.notEqual(hash, sceneHash(scene, chunks, { ...cfg, speed: 1 }));
+ const eleven = normalizeConfig({ voice: { provider: 'elevenlabs', id: 'v' } });
+ assert.notEqual(sceneHash(scene, chunks, eleven), sceneHash(scene, chunks, { ...eleven, voice: { ...eleven.voice, model: 'another' } }));
+ const recording = normalizeConfig({ voice: { provider: 'recording', src: 'assets/take.wav' } });
+ const recHash = sceneHash(scene, chunks, recording, { content: 'abc' });
+ assert.notEqual(recHash, sceneHash(scene, chunks, recording, { content: 'def' }));
+ assert.equal(recHash, sceneHash(scene, speechChunks(scene.units, { sentenceGap: 9 }), recording, { content: 'abc' }));
+ assert.notEqual(recHash, sceneHash(scene, chunks, { ...recording, pace: 1.2 }, { content: 'abc' }));
+ const paused = parseScript('## first\n[#a] Hello. [pause 9] Goodbye.').scenes[0];
+ assert.equal(recHash, sceneHash(paused, speechChunks(paused.units), recording, { content: 'abc' }));
+});
+
+test('recordings preserve sample rate up to 48 kHz, apply tempo and normalize loudness', async (t) => {
+ const dir = temp(t);
+ const file = path.join(dir, 'tone.wav');
+ const sr = 24000;
+ const samples = Float32Array.from({ length: sr * 4 }, (_, i) => 0.02 * Math.sin(2 * Math.PI * 440 * i / sr));
+ writeWav(file, samples, sr);
+ const decoded = await decodeRecording(file, dir, 1.25);
+ assert.equal(decoded.sampleRate, sr);
+ assert.ok(Math.abs(decoded.samples.length / sr - 3.2) < 0.05);
+ const out = path.join(dir, 'out.wav');
+ await saveRecording(out, decoded.samples, decoded.sampleRate);
+ assert.equal(readWav(out).sampleRate, sr);
+ assert.ok(Math.abs((await loudness(out)).integrated - -22.3) < 0.5);
+ const high = path.join(dir, 'high.wav');
+ writeWav(high, new Float32Array(96000), 96000);
+ assert.equal((await decodeRecording(high, dir, 1)).sampleRate, 48000);
+ const silent = path.join(dir, 'silent.wav');
+ await saveRecording(silent, new Float32Array(sr), sr);
+ assert.equal(readWav(silent).samples.length, sr);
+ assert.deepEqual(fs.readdirSync(dir).filter((n) => n.startsWith('recording-')), []);
+});