Skip to content

Explainer videos and product demos made by your AI agent. Free and open source: a local voice (Kokoro), word timing (Whisper) and a canvas renderer turn a script into a narrated MP4.

Narrate in Spanish, French, Italian, Portuguese and Hindi (#14)

Vincent Schmalbach23f2c0283e67
diff --git a/AGENTS.md b/AGENTS.md
index 1d8fed1..5b973f7 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -34,10 +34,11 @@ npm install
node bin/explainroo.js doctor --fetch
```
-`doctor` checks Node, ffmpeg and Chrome. It also downloads the speech models,
-about 400 MB, the first time only. If the user ran `npm link`, you can write
-`explainroo` instead of `node bin/explainroo.js` in the commands below. Every
-command takes the project folder first and accepts `--json`.
+`doctor` checks Node, ffmpeg, Chrome and espeak-ng. It also downloads the
+English speech models, about 400 MB, the first time only. Other languages
+fetch their speech check model on first use. If the user ran `npm link`, you
+can write `explainroo` instead of `node bin/explainroo.js` in the commands
+below. Every command takes the project folder first and accepts `--json`.
### When to ask the user
@@ -295,9 +296,58 @@ speech check reports a mismatch, make script.md say what was actually said.
Replacing a scene recording makes that scene again. Replacing a whole take
makes all its scenes again.
-ElevenLabs narration and recordings must be in English for now. The word
-timing model understands English only. Run `voice`, `check` and `render`
-as usual with any of the three sources.
+ElevenLabs narration and recordings work in every supported language.
+Run `voice`, `check` and `render` as usual with any of the three sources.
+
+## Videos in other languages
+
+Set `language` in `video.json` to `en`, `es`, `fr`, `it`, `pt-br` or `hi`.
+English is the default. `pt-br` means Brazilian Portuguese. You can also
+start with `explainroo init videos/my-video --language es`.
+
+When `voice` is left out, the language picks its Kokoro voice:
+
+| Language | Default voice |
+|---|---|
+| English (`en`) | `af_heart` |
+| Spanish (`es`) | `ef_dora` |
+| French (`fr`) | `ff_siwis` |
+| Italian (`it`) | `if_sara` |
+| Brazilian Portuguese (`pt-br`) | `pf_dora` |
+| Hindi (`hi`) | `hf_alpha` |
+
+Run `explainroo voices --language es` to see voices for one language.
+A Kokoro voice must match the video's language. If a project already has
+`voice` set, change it too, or remove it to use the default.
+
+Write the narration in that language as a person would say it. Do not
+translate English word for word. The writing rules above apply in spirit.
+Cue words and `[#markers]` work the same. `at: 'qué'` matches "Qué".
+Hindi sentences can end with `।`. Captions and subtitle files keep the
+written text. The looks include a Hindi font, so no system font is needed.
+
+Kokoro needs espeak-ng once for these five languages. Install it on Linux
+with `sudo apt install espeak-ng`, or on macOS with `brew install espeak-ng`.
+On Windows, use the installer from the
+[espeak-ng releases page](https://github.com/espeak-ng/espeak-ng/releases).
+If the program is in another place, set `EXPLAINROO_ESPEAK` to its binary.
+English, ElevenLabs and recordings do not need it.
+
+The first speech check in another language downloads multilingual Whisper
+small. `doctor --fetch` still fetches only the English models. `verify`
+uses the video's language to check the finished sound too.
+
+If a symbol, short form or word sounds wrong, use `{shown|spoken}` in
+`script.md` and run `voice` again. Put the pronunciation in the same
+language. The speech check accepts digits for numbers. It does not try to
+match spelled numbers to digits outside English. Write the digits, or use
+an override whose spoken text has digits when that helps the check.
+Japanese and Mandarin are not supported.
+
+In Hindi, the speech check confirms fewer words than in the other languages,
+because the speech model knows Hindi less well. A few unconfirmed words are
+normal there. Fix a word when it stays unconfirmed in several scenes, or
+when the user hears it wrong.
## Images
@@ -575,7 +625,7 @@ project.
```
videos/<name>/
- video.json look, size, voice, music, captions, watermark
+ video.json look, size, language, voice, music, captions, watermark
script.md the narration, one "## scene-id" per scene
scenes.js one drawing function per scene
assets/ images: screenshots, logos, generated illustrations
@@ -591,7 +641,8 @@ videos/<name>/
| `theme` | `paper` | a built-in look, or a relative path to a `.json` look file (see Custom looks) |
| `size` | `16:9` | a platform (`youtube`, `shorts`, `tiktok`, `reels`, `vertical`, `instagram`, `linkedin`, `square`), `16:9`, `9:16`, `1:1`, `4:5` or `WIDTHxHEIGHT` |
| `fps` | 30 | 24, 25, 30, 50 or 60 |
-| `voice` | `af_heart` | a Kokoro id or a provider object for Kokoro, ElevenLabs or a recording; see Voices |
+| `language` | `en` | narration language: `en`, `es`, `fr`, `it`, `pt-br` or `hi`; see Videos in other languages |
+| `voice` | the language's voice | a Kokoro id or a provider object for Kokoro, ElevenLabs or a recording; English uses `af_heart`; see Voices |
| `speed` | 0.9 for Kokoro, 1 otherwise | voice speed only, 0.6 to 1.6; ElevenLabs limits speed times pace to 0.7 through 1.2 |
| `pace` | 1 | speed of the whole video (voice, pauses, animations), 0.7 to 1.6 |
| `music` | `true` | `true` (the look's style), `warm`, `upbeat`, `calm`, `tech`, `playful`, `{ "style", "volume" }` or `false` |
@@ -667,7 +718,7 @@ when `ink` or `muted` has too little contrast against `bg`.
| Command | What it does |
|---|---|
-| `init <dir>` | creates a project (`--theme`, `--size`, `--pace`, `--voice`, `--title`) |
+| `init <dir>` | creates a project (`--theme`, `--size`, `--pace`, `--language`, `--voice`, `--title`) |
| `voice [project]` | makes the narration and the word times, saved per scene so only changed scenes are made again |
| `preview [project]` | a live preview in the browser that reloads when you save |
| `still [project] [times]` | PNG pictures at `12.5`, `scene`, `[email protected]` or `scene@end` (`--transparent` keeps alpha) |
@@ -678,10 +729,10 @@ when `ink` or `muted` has too little contrast against `bg`.
| `capture [project] <url>` | saves screenshots, wording, colors, fonts and logos from a live site (`--mobile`, `--json`) |
| `image [project] <name> "<prompt>"` | makes an illustration with OpenRouter |
| `images [project]` | lists the images and what they cost |
-| `voices`, `say "text"` | lists the 28 Kokoro voices, or makes a sample |
+| `voices`, `say "text"` | lists Kokoro voices with their languages (`--language es` filters voices), or makes a sample in the chosen voice's language |
| `voices [project] --elevenlabs` | lists the names, ids and labels available to the key (`--json` also works) |
| `themes`, `formats`, `icons <word>` | lists the looks and the sizes, searches the 1,854 icons |
-| `doctor` | checks the setup (`--fetch` downloads the speech models) |
+| `doctor` | checks Node, ffmpeg, Chrome, espeak-ng and models (`--fetch` downloads only English models); missing espeak-ng is allowed |
The extra files share the video's name, also with `--out`: `video.srt`,
`video.vtt`, `video.cover.jpg` and `video.chapters.txt`. A GIF has no sound,
diff --git a/engine/captions.js b/engine/captions.js
index ae71278..6ed71af 100644
--- a/engine/captions.js
+++ b/engine/captions.js
@@ -21,7 +21,7 @@ export function buildPhrases(timeline, maxChars) {
if (!cur) cur = { words: [], start: g.start };
cur.words.push(g);
cur.end = g.end;
- if (/[.!?;:]["'’)]*$/.test(g.text) || (/,$/.test(g.text) && cur.words.length >= 4)) flush();
+ if (/[.!?;:।]["'’)]*$/.test(g.text) || (/,$/.test(g.text) && cur.words.length >= 4)) flush();
});
flush();
}
diff --git a/engine/runtime.js b/engine/runtime.js
index 0a84f3b..5b276f1 100644
--- a/engine/runtime.js
+++ b/engine/runtime.js
@@ -26,6 +26,7 @@ async function loadFonts(theme, config) {
// The UI kit (s.ui) draws app screens in Inter and headlines in Instrument Serif.
wanted.add('Inter');
wanted.add('Instrument Serif');
+ wanted.add('Noto Sans Devanagari');
const faces = [];
const add = (family, url, o) => {
const face = new FontFace(family, `url(${url})`, { weight: String(o.weight ?? 400), style: o.style ?? 'normal', unicodeRange: o.unicodeRange, display: 'block' });
diff --git a/engine/themes.js b/engine/themes.js
index a98aca8..72532d4 100644
--- a/engine/themes.js
+++ b/engine/themes.js
@@ -199,7 +199,7 @@ export function fontString(theme, role = 'body', size = 48, weight) {
const f = theme.fonts[role] || theme.fonts.body;
const w = weight ?? f.weight;
const fallback = role === 'mono' ? 'monospace' : 'sans-serif';
- return `${w} ${Math.round(size * 100) / 100}px "${f.family}", ${fallback}`;
+ return `${w} ${Math.round(size * 100) / 100}px "${f.family}", "Noto Sans Devanagari", ${fallback}`;
}
// Background textures are rendered once and reused on every frame.
diff --git a/engine/ui.js b/engine/ui.js
index ddeb25f..0094420 100644
--- a/engine/ui.js
+++ b/engine/ui.js
@@ -121,7 +121,7 @@ export class UI {
}
fontString(size, weight = 400, family = this.font, italic = false) {
- return `${italic ? 'italic ' : ''}${weight} ${size}px "${family}", Inter, sans-serif`;
+ return `${italic ? 'italic ' : ''}${weight} ${size}px "${family}", Inter, "Noto Sans Devanagari", sans-serif`;
}
_family(f) {
diff --git a/engine/util.js b/engine/util.js
index a0d4eca..2f9525a 100644
--- a/engine/util.js
+++ b/engine/util.js
@@ -197,6 +197,6 @@ export function normWord(w) {
.normalize('NFKD')
.replace(/[̀-ͯ]/g, '')
.replace(/['’]/g, '')
- .replace(/[^a-z0-9%$€£+#@.]/g, '')
+ .replace(/[^\p{L}\p{M}\p{N}%$€£+#@.]/gu, '')
.replace(/^\.+|\.+$/g, '');
}
diff --git a/fonts/NotoSansDevanagari-400.ttf b/fonts/NotoSansDevanagari-400.ttf
new file mode 100644
index 0000000..cb4957a
Binary files /dev/null and b/fonts/NotoSansDevanagari-400.ttf differ
diff --git a/fonts/NotoSansDevanagari-700.ttf b/fonts/NotoSansDevanagari-700.ttf
new file mode 100644
index 0000000..669f369
Binary files /dev/null and b/fonts/NotoSansDevanagari-700.ttf differ
diff --git a/fonts/fonts.json b/fonts/fonts.json
index 88c62c6..21b46f6 100644
--- a/fonts/fonts.json
+++ b/fonts/fonts.json
@@ -198,5 +198,19 @@
"subset": "latin",
"file": "InstrumentSerif-400-italic-latin.woff2",
"unicodeRange": "U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD"
+ },
+ {
+ "family": "Noto Sans Devanagari",
+ "weight": 400,
+ "subset": "devanagari",
+ "file": "NotoSansDevanagari-400.ttf",
+ "unicodeRange": "U+0900-097F,U+1CD0-1CFF,U+200C-200D,U+20B9,U+25CC,U+A8E0-A8FF"
+ },
+ {
+ "family": "Noto Sans Devanagari",
+ "weight": 700,
+ "subset": "devanagari",
+ "file": "NotoSansDevanagari-700.ttf",
+ "unicodeRange": "U+0900-097F,U+1CD0-1CFF,U+200C-200D,U+20B9,U+25CC,U+A8E0-A8FF"
}
]
diff --git a/fonts/licenses/NotoSansDevanagari-OFL.txt b/fonts/licenses/NotoSansDevanagari-OFL.txt
new file mode 100644
index 0000000..63426f0
--- /dev/null
+++ b/fonts/licenses/NotoSansDevanagari-OFL.txt
@@ -0,0 +1,118 @@
+Copyright 2010, 2012-2020 Google Inc.
+Copyright 2015-2020 Google LLC.
+
+SIL OPEN FONT LICENSE Version 1.1
+
+PREAMBLE
+
+The goals of the Open Font License (OFL) are to stimulate
+worldwide development of collaborative font projects,
+to support the font creation efforts
+of academic and linguistic communities,
+and to provide a free and open framework
+in which fonts may be shared and improved
+in partnership with others.
+
+The OFL allows the licensed fonts to be used, studied,
+modified and redistributed freely
+as long as they are not sold by themselves.
+The fonts, including any derivative works,
+can be bundled, embedded,
+redistributed and/or sold with any software
+provided that any reserved names are not used
+by derivative works.
+The fonts and derivatives, however,
+cannot be released under any other type of license.
+The requirement for fonts to remain under this license
+does not apply to any document
+created using the fonts or their derivatives.
+
+DEFINITIONS
+
+"Font Software" refers to the set of files
+released by the Copyright Holder(s) under this license
+and clearly marked as such.
+This may include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such
+after the copyright statement(s).
+
+"Original Version" refers to the collection
+of Font Software components
+as distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative
+made by adding to, deleting, or substituting --
+in part or in whole --
+any of the components of the Original Version,
+by changing formats
+or by porting the Font Software to a new environment.
+
+"Author" refers to any designer, engineer, programmer,
+technical writer or other person
+who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+
+Permission is hereby granted, free of charge,
+to any person obtaining a copy of the Font Software,
+to use, study, copy, merge, embed, modify, redistribute,
+and sell modified and unmodified copies of the Font Software,
+subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions,
+may be sold by itself.
+
+2) Original or Modified Versions of the Font Software
+may be bundled, redistributed and/or sold with any software,
+provided that each copy contains the above copyright notice
+and this license.
+These can be included either as stand-alone text files,
+human-readable headers or in the appropriate
+machine-readable metadata fields within text or binary files
+as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use
+the Reserved Font Name(s)
+unless explicit written permission is granted
+by the corresponding Copyright Holder.
+This restriction only applies to the primary font name
+as presented to the users.
+
+4) The name(s) of the Copyright Holder(s)
+or the Author(s) of the Font Software
+shall not be used to promote,
+endorse or advertise any Modified Version,
+except to acknowledge the contribution(s)
+of the Copyright Holder(s) and the Author(s)
+or with their explicit written permission.
+
+5) The Font Software,
+modified or unmodified, in part or in whole,
+must be distributed entirely under this license,
+and must not be distributed under any other license.
+The requirement for fonts to remain under this license
+does not apply to any document
+created using the Font Software.
+
+TERMINATION
+
+This license becomes null and void
+if any of the above conditions are not met.
+
+DISCLAIMER
+
+THE FONT SOFTWARE IS PROVIDED "AS IS",
+WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO
+ANY WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT.
+IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE
+FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT,
+INCIDENTAL, OR CONSEQUENTIAL DAMAGES,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF THE USE OR INABILITY TO USE
+THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.
diff --git a/scripts/fetch-fonts.mjs b/scripts/fetch-fonts.mjs
index 0c80bf8..0ea8b12 100644
--- a/scripts/fetch-fonts.mjs
+++ b/scripts/fetch-fonts.mjs
@@ -48,11 +48,18 @@ for (const [family, weights] of FAMILIES) {
console.log(`${file} ${(buf.length / 1024).toFixed(1)} KB`);
}
}
+for (const [weight, style] of [[400, 'Regular'], [700, 'Bold']]) {
+ const file = `NotoSansDevanagari-${weight}.ttf`;
+ const res = await fetch(`https://raw.githubusercontent.com/notofonts/noto-fonts/main/hinted/ttf/NotoSansDevanagari/NotoSansDevanagari-${style}.ttf`);
+ if (!res.ok) throw new Error(`no ${file} (${res.status})`);
+ fs.writeFileSync(path.join(dir, file), Buffer.from(await res.arrayBuffer()));
+ manifest.push({ family: 'Noto Sans Devanagari', weight, subset: 'devanagari', file, unicodeRange: 'U+0900-097F,U+1CD0-1CFF,U+200C-200D,U+20B9,U+25CC,U+A8E0-A8FF' });
+}
fs.writeFileSync(path.join(dir, 'fonts.json'), JSON.stringify(manifest, null, 2) + '\n');
// SIL Open Font License texts, one per family.
fs.mkdirSync(path.join(dir, 'licenses'), { recursive: true });
-for (const [family] of FAMILIES) {
+for (const [family] of [...FAMILIES, ['Noto Sans Devanagari']]) {
const slug = family.toLowerCase().replace(/\s+/g, '');
const res = await fetch(`https://raw.githubusercontent.com/google/fonts/main/ofl/${slug}/OFL.txt`);
if (!res.ok) throw new Error(`no OFL.txt for ${family} (${res.status})`);
diff --git a/src/align.js b/src/align.js
index e6cadd2..e27b6c2 100644
--- a/src/align.js
+++ b/src/align.js
@@ -35,7 +35,12 @@ export function intWords(n) {
// Whisper writes "$1,050" and "5%" where the voice said "one thousand and
// fifty dollars" and "five percent", and "example.com" or "1.1.1.1" where it
// said words joined by "dot". Both sides are expanded the same way.
-export function expand(word) {
+export function expand(word, language = 'en') {
+ if (language !== 'en') {
+ const n = normWord(word).replace(/[०-९]/g, d => String(d.charCodeAt(0) - 0x0966));
+ const digits = /^[\p{N}%$€£+.]+$/u.test(n) ? n.replace(/[^\p{N}]/gu, '') : '';
+ return n ? [digits || n] : [];
+ }
const raw = String(word).trim().toLowerCase();
// Whisper splits "1.1.1.1" into "1", ".1", ".1", ".1": a leading dot is a spoken "dot".
const lead = /^\.[a-z0-9]/i.test(raw) ? ['dot'] : [];
@@ -90,14 +95,16 @@ export function similar(a, b) {
// words: [{ spoken }] in order. heard: [{ text, start, end }] from Whisper.
// Returns per-word { start, end, matched } and the share of spoken tokens matched.
-export function alignWords(words, heard, duration) {
+export function alignWords(words, heard, duration, language = 'en') {
+ const partsOf = word => expand(word, language);
+ const matches = (a, b) => language !== 'en' && (/\p{N}/u.test(a) || /\p{N}/u.test(b)) ? a === b : similar(a, b);
const tokens = [];
words.forEach((w, wi) => {
- const parts = String(w.spoken).split(/\s+/).flatMap(expand);
+ const parts = String(w.spoken).split(/\s+/).flatMap(partsOf);
(parts.length ? parts : ['']).forEach((p, k) => {
// "and" inside a spoken number ("one thousand and fifty") is optional.
const prev = tokens[tokens.length - 1];
- const optional = p === 'and' && prev && isNumberToken(prev.norm);
+ const optional = language === 'en' && p === 'and' && prev && isNumberToken(prev.norm);
tokens.push({ wi, norm: p, first: k === 0, last: k === parts.length - 1 || !parts.length, optional });
});
});
@@ -117,7 +124,7 @@ export function alignWords(words, heard, duration) {
}
const H = [];
for (const h of joined) {
- const parts = expand(h.text);
+ const parts = partsOf(h.text);
const step = (h.end - h.start) / Math.max(1, parts.length);
parts.forEach((p, k) => H.push({ text: p, norm: p, start: h.start + k * step, end: h.start + (k + 1) * step }));
}
@@ -144,16 +151,16 @@ export function alignWords(words, heard, duration) {
if (i < T && c + (tokens[i].optional ? 0 : 1) < cost[i + 1][j]) { cost[i + 1][j] = c + (tokens[i].optional ? 0 : 1); move[i + 1][j] = 2; }
if (j < N && c + 1 < cost[i][j + 1]) { cost[i][j + 1] = c + 1; move[i][j + 1] = 3; }
if (i < T && j < N) {
- const ok = similar(tokens[i].norm, H[j].norm);
+ const ok = matches(tokens[i].norm, H[j].norm);
const v = c + (ok ? 0 : 1.6);
if (v < cost[i + 1][j + 1]) { cost[i + 1][j + 1] = v; move[i + 1][j + 1] = ok ? 1 : 4; }
}
if (i < T && j < N && tokens[i].first && groupEnd[i] > i) {
const e = groupEnd[i];
const joined = tokens.slice(i, e + 1).map((t) => t.norm).join('');
- if (similar(joined, H[j].norm) && c < cost[e + 1][j + 1]) { cost[e + 1][j + 1] = c; move[e + 1][j + 1] = 5; }
+ if (matches(joined, H[j].norm) && c < cost[e + 1][j + 1]) { cost[e + 1][j + 1] = c; move[e + 1][j + 1] = 5; }
}
- if (i < T && j + 1 < N && similar(tokens[i].norm, H[j].norm + H[j + 1].norm) && c < cost[i + 1][j + 2]) {
+ if (i < T && j + 1 < N && matches(tokens[i].norm, H[j].norm + H[j + 1].norm) && c < cost[i + 1][j + 2]) {
cost[i + 1][j + 2] = c;
move[i + 1][j + 2] = 6;
}
diff --git a/src/cli.js b/src/cli.js
index 4ae5102..e25721d 100644
--- a/src/cli.js
+++ b/src/cli.js
@@ -8,7 +8,8 @@ import { prepare, makeState } from './pipeline.js';
import { render } from './render.js';
import { check, stills, sheet, verify } from './qa.js';
import { startServer, ROOT } from './server.js';
-import { VOICES, loadTTS, loadASR, cacheRoot, ttsDtype, TTS_SAMPLE_RATE } from './models.js';
+import { VOICES, LANGUAGES, voiceLanguage, loadTTS, loadASR, cacheRoot, ttsDtype, TTS_SAMPLE_RATE, MULTILINGUAL_ASR_MODEL } from './models.js';
+import { espeakVersion, phonemize, generateSpeech } from './phonemes.js';
import { writeWav } from './wav.js';
import { ffmpegVersion } from './ffmpeg.js';
import { findChrome } from './browser.js';
@@ -23,7 +24,7 @@ const HELP = `explainroo ${VERSION}: narrated explainer videos from code
Usage: explainroo <command> [project] [options]
Make a video
- init <dir> create a project (--theme, --size, --pace, --voice, --title)
+ init <dir> create a project (--theme, --size, --pace, --language es, --voice, --title)
voice [project] make the narration (Kokoro, ElevenLabs or a recording) and word timings
preview [project] live preview in your browser, reloads on save (--port)
still [project] [t…] PNG stills: "12.5", "scene", "[email protected]" (--transparent, default: end of every scene)
@@ -42,12 +43,12 @@ Images (optional, needs an OpenRouter API key)
images [project] list generated images and what they cost
Reference
- voices [project] list Kokoro voices (--elevenlabs lists the voices of your ElevenLabs key)
+ voices [project] list Kokoro voices (--language es for one language, --elevenlabs for your ElevenLabs key)
say "text" hear a Kokoro voice (--voice am_michael, --speed 1, --out sample.wav)
themes list looks
formats list sizes for YouTube, Shorts, TikTok, Reels, Instagram, LinkedIn
icons <word…> search the 1,800+ built-in icons
- doctor check ffmpeg, Chrome and the speech models (--fetch downloads them)
+ doctor check ffmpeg, Chrome, espeak-ng and models (--fetch downloads English models)
Options: --json for machine-readable output. The project defaults to the current folder.
Docs: AGENTS.md (for coding agents) and https://www.explainroo.com/docs/`;
@@ -191,11 +192,14 @@ export async function main(argv) {
return 0;
}
case 'voices': {
+ const language = textFlag(flags, 'language');
+ if (language !== undefined) normalizeConfig({ language });
if (flags.elevenlabs) {
const voices = await elevenVoices(path.resolve(pos[0] || '.'));
print(voices.map((v) => `${v.name} ${v.id} ${Object.entries(v.labels).map(([k, x]) => `${k}: ${x}`).join(', ')}`).join('\n'), { voices });
} else {
- print(Object.entries(VOICES).map(([k, v]) => `${k.padEnd(12)} ${v}`).join('\n') + '\nUse voices --elevenlabs for ElevenLabs, or set voice to { "provider": "recording", "src": "assets/narration.wav" } in video.json.', { voices: VOICES });
+ const voices = Object.fromEntries(Object.entries(VOICES).filter(([id]) => language === undefined || voiceLanguage(id) === language));
+ print(Object.entries(voices).map(([k, v]) => `${k.padEnd(12)} ${voiceLanguage(k).padEnd(6)} ${v}`).join('\n') + '\nUse voices --elevenlabs for ElevenLabs, or set voice to { "provider": "recording", "src": "assets/narration.wav" } in video.json.', { voices, languages: Object.fromEntries(Object.keys(voices).map(id => [id, voiceLanguage(id)])) });
}
return 0;
}
@@ -250,18 +254,21 @@ function textFlag(flags, name) {
}
function init(dir, flags, print) {
- if (!dir) throw new ProjectError('usage: explainroo init <dir> [--theme paper] [--size youtube|tiktok|linkedin|...] [--pace 1.2] [--voice af_heart] [--title "..."]');
+ if (!dir) throw new ProjectError('usage: explainroo init <dir> [--theme paper] [--size youtube|tiktok|linkedin|...] [--pace 1.2] [--language es] [--voice af_heart] [--title "..."]');
const target = path.resolve(dir);
if (fs.existsSync(path.join(target, 'video.json'))) throw new ProjectError(`${target} already has a video.json`);
const theme = textFlag(flags, 'theme') || 'paper';
if (!THEMES.includes(theme)) throw new ProjectError(`theme must be one of ${THEMES.join(', ')}`);
- const voice = textFlag(flags, 'voice') || 'af_heart';
+ const language = textFlag(flags, 'language') ?? 'en';
+ normalizeConfig({ language });
+ const voice = textFlag(flags, 'voice') || LANGUAGES[language];
if (!VOICES[voice]) throw new ProjectError(`voice "${voice}" does not exist. Run "explainroo voices".`);
const title = textFlag(flags, 'title') || path.basename(target).replace(/[-_]+/g, ' ').replace(/^\w/, (c) => c.toUpperCase());
const tpl = path.join(ROOT, 'templates', 'starter');
const size = textFlag(flags, 'size') || '16:9';
resolveSize(size);
const config = { title, theme, size, voice, music: true, captions: 'auto' };
+ if (flags.language !== undefined) config.language = language;
if (flags.pace !== undefined) config.pace = num(flags.pace, 'pace');
normalizeConfig(config);
// Create the folder only after every setting checks out, so a bad flag
@@ -329,7 +336,9 @@ async function say(text, flags, log, print) {
const voice = textFlag(flags, 'voice') || 'af_heart';
if (!VOICES[voice]) throw new ProjectError(`voice "${voice}" does not exist. Run "explainroo voices".`);
const tts = await loadTTS({ log });
- const audio = await tts.generate(text, { voice, speed: num(flags.speed, 'speed') ?? 1 });
+ const language = voiceLanguage(voice);
+ const phonemes = language === 'en' ? undefined : (await phonemize([text], language))[0];
+ const audio = await generateSpeech(tts, text, { language, voice, speed: num(flags.speed, 'speed') ?? 1, phonemes });
const out = path.resolve(textFlag(flags, 'out') || `${voice}.wav`);
writeWav(out, audio.audio, audio.sampling_rate || TTS_SAMPLE_RATE);
print(out, { file: out, voice });
@@ -373,6 +382,8 @@ async function doctor(flags, log, print) {
add('node', nodeOk, `v${process.versions.node}${nodeOk ? '' : ' (need 20.11 or newer)'}`);
const ff = ffmpegVersion();
add('ffmpeg', !!ff, ff || 'not found; install ffmpeg or set EXPLAINROO_FFMPEG');
+ const espeak = espeakVersion();
+ add('espeak-ng', true, espeak || 'not found; only needed for other languages');
try {
add('chrome', true, findChrome());
} catch (e) {
@@ -387,6 +398,7 @@ async function doctor(flags, log, print) {
}
add('voice model', has('Kokoro-82M-v1.0-ONNX'), has('Kokoro-82M-v1.0-ONNX') ? `Kokoro 82M (${ttsDtype()}) in ${models}` : 'not downloaded yet; "explainroo doctor --fetch" or the first voice run fetches it');
add('speech check model', has('whisper-base.en_timestamped'), has('whisper-base.en_timestamped') ? 'Whisper base.en' : 'not downloaded yet; fetched on first use');
+ add('language model', has(MULTILINGUAL_ASR_MODEL.split('/')[1]), 'Whisper small for es, fr, it, pt-br and hi; downloaded on first use, not by --fetch');
const ok = rows.every((r) => r.ok || r.name.includes('model'));
print(rows.map((r) => `${r.ok ? 'ok ' : 'FAIL'} ${r.name.padEnd(20)} ${r.detail}`).join('\n'), { ok, checks: rows });
return ok ? 0 : 1;
diff --git a/src/licenses/misaki-Apache-2.0.txt b/src/licenses/misaki-Apache-2.0.txt
new file mode 100644
index 0000000..261eeb9
--- /dev/null
+++ b/src/licenses/misaki-Apache-2.0.txt
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/src/models.js b/src/models.js
index bd9cbf6..edd1d38 100644
--- a/src/models.js
+++ b/src/models.js
@@ -6,8 +6,21 @@ import path from 'node:path';
export const TTS_MODEL = 'onnx-community/Kokoro-82M-v1.0-ONNX';
export const ASR_MODEL = 'onnx-community/whisper-base.en_timestamped';
+export const MULTILINGUAL_ASR_MODEL = 'onnx-community/whisper-small_timestamped';
export const TTS_SAMPLE_RATE = 24000;
+export const LANGUAGES = {
+ en: 'af_heart', es: 'ef_dora', fr: 'ff_siwis', it: 'if_sara', 'pt-br': 'pf_dora', hi: 'hf_alpha',
+};
+
+export function voiceLanguage(id) {
+ return { a: 'en', b: 'en', e: 'es', f: 'fr', i: 'it', p: 'pt-br', h: 'hi' }[id[0]];
+}
+
+export function asrModel(language = 'en') {
+ return language === 'en' ? ASR_MODEL : MULTILINGUAL_ASR_MODEL;
+}
+
export function cacheRoot() {
return process.env.EXPLAINROO_CACHE || path.join(os.homedir(), '.cache', 'explainroo');
}
@@ -62,31 +75,34 @@ export function loadTTS({ log } = {}) {
return ttsPromise;
}
-let asrPromise = null;
-export function loadASR({ log } = {}) {
- if (!asrPromise) {
- asrPromise = (async () => {
+const asrPromises = new Map();
+export function loadASR({ log, language = 'en' } = {}) {
+ const model = asrModel(language);
+ if (!asrPromises.has(model)) {
+ asrPromises.set(model, (async () => {
const { pipeline } = await lib();
- return pipeline('automatic-speech-recognition', ASR_MODEL, {
+ return pipeline('automatic-speech-recognition', model, {
dtype: 'q8',
progress_callback: progressLogger('speech check model', log),
});
- })();
+ })());
}
- return asrPromise;
+ return asrPromises.get(model);
}
// samples: Float32Array at 16 kHz. Returns [{ text, start, end }].
-export async function transcribeWords(samples, { log } = {}) {
- const asr = await loadASR({ log });
- const r = await asr(samples, { return_timestamps: 'word', chunk_length_s: 30, stride_length_s: 5 });
+export async function transcribeWords(samples, { log, language = 'en' } = {}) {
+ const asr = await loadASR({ log, language });
+ const r = await asr(samples, { return_timestamps: 'word', chunk_length_s: 30, stride_length_s: 5,
+ ...(language === 'en' ? {} : { language: language === 'pt-br' ? 'pt' : language, task: 'transcribe' }),
+ });
return {
text: (r.text || '').trim(),
words: (r.chunks || []).map((c) => ({ text: c.text.trim(), start: c.timestamp[0] ?? 0, end: c.timestamp[1] ?? c.timestamp[0] ?? 0 })),
};
}
-// Kokoro v1.0 English voices.
+// Kokoro v1.0 voices for the supported languages.
export const VOICES = {
af_heart: 'American English, female (default)',
af_bella: 'American English, female',
@@ -116,4 +132,17 @@ export const VOICES = {
bm_fable: 'British English, male',
bm_lewis: 'British English, male',
bm_daniel: 'British English, male',
+ ef_dora: 'Spanish, female (default)',
+ em_alex: 'Spanish, male',
+ em_santa: 'Spanish, male',
+ ff_siwis: 'French, female (default)',
+ if_sara: 'Italian, female (default)',
+ im_nicola: 'Italian, male',
+ pf_dora: 'Brazilian Portuguese, female (default)',
+ pm_alex: 'Brazilian Portuguese, male',
+ pm_santa: 'Brazilian Portuguese, male',
+ hf_alpha: 'Hindi, female (default)',
+ hf_beta: 'Hindi, female',
+ hm_omega: 'Hindi, male',
+ hm_psi: 'Hindi, male',
};
diff --git a/src/phonemes.js b/src/phonemes.js
new file mode 100644
index 0000000..1526070
--- /dev/null
+++ b/src/phonemes.js
@@ -0,0 +1,68 @@
+import { spawn, spawnSync } from 'node:child_process';
+
+const ESPEAK_LANGUAGES = { es: 'es', fr: 'fr-fr', it: 'it', 'pt-br': 'pt-br', hi: 'hi' };
+// Adapted from misaki's EspeakG2P (Apache-2.0, see licenses/).
+// https://github.com/hexgrad/misaki/blob/main/misaki/espeak.py
+// Kokoro v1.0 keeps nasal vowels, unlike misaki's v2.0 mapping.
+const E2M = {
+ 'a^ɪ': 'I', 'a^ʊ': 'W', 'd^z': 'ʣ', 'd^ʒ': 'ʤ', 'e^ɪ': 'A',
+ 'o^ʊ': 'O', 'ə^ʊ': 'Q', 's^s': 'S', 't^s': 'ʦ', 't^ʃ': 'ʧ', 'ɔ^ɪ': 'Y',
+};
+
+export function mapPhonemes(ipa) {
+ let ps = ipa.trim();
+ for (const [old, replacement] of Object.entries(E2M).sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0)) ps = ps.replaceAll(old, replacement);
+ return ps.replaceAll('^', '').replaceAll('-', '').replaceAll('«', '(').replaceAll('»', ')');
+}
+
+function binary() {
+ return process.env.EXPLAINROO_ESPEAK || 'espeak-ng';
+}
+
+const INSTALL = 'espeak-ng is needed for this language. Linux: sudo apt install espeak-ng. macOS: brew install espeak-ng. Windows: use the installer at https://github.com/espeak-ng/espeak-ng/releases. Or set EXPLAINROO_ESPEAK to its binary.';
+
+export function espeakVersion() {
+ const r = spawnSync(binary(), ['--version'], { encoding: 'utf8', timeout: 5000 });
+ return !r.error && r.status === 0 ? r.stdout.trim().split('\n')[0] : null;
+}
+
+export async function phonemize(texts, language) {
+ const lang = ESPEAK_LANGUAGES[language];
+ if (!lang) throw new Error('espeak-ng supports es, fr, it, pt-br and hi here.');
+ const input = [];
+ const parts = texts.map(text => {
+ // Preserve punctuation outside espeak, as misaki's phonemizer backend does.
+ const prepared = text.replaceAll('«', '“').replaceAll('»', '”').replaceAll('(', '«').replaceAll(')', '»');
+ return prepared.split(/([;:,.!?¡¿—…"«»“”{}\[\]।]+|\s*\n\s*)/u).map((part, i) => {
+ if (i % 2 || !part.trim()) return { text: part };
+ const index = input.length;
+ input.push(part.trim());
+ return { index, before: /^\s*/.exec(part)[0], after: /\s*$/.exec(part)[0] };
+ });
+ });
+ if (!input.length) return parts.map(p => mapPhonemes(p.map(x => x.text).join('')));
+ const output = await new Promise((resolve, reject) => {
+ // Without --stdin, espeak processes each input line separately.
+ const child = spawn(binary(), ['-q', '-b', '1', '-v', lang, '--ipa=1', '--tie=^'], { stdio: ['pipe', 'pipe', 'pipe'] });
+ let stdout = '';
+ let stderr = '';
+ child.stdout.setEncoding('utf8').on('data', data => { stdout += data; });
+ child.stderr.setEncoding('utf8').on('data', data => { stderr += data; });
+ child.on('error', e => reject(new Error(['ENOENT', 'EACCES'].includes(e.code) ? INSTALL : 'espeak-ng could not start. Check EXPLAINROO_ESPEAK.')));
+ child.stdin.on('error', () => {});
+ child.on('close', code => {
+ if (code !== 0) return reject(new Error('espeak-ng could not read the text. Check its installation and EXPLAINROO_ESPEAK.'));
+ const lines = stdout.trimEnd().split(/\r?\n/);
+ if (lines.length !== input.length || lines.some(line => !line.trim()) || stderr.trim()) return reject(new Error('espeak-ng returned incomplete phonemes. Check its installation and EXPLAINROO_ESPEAK.'));
+ resolve(lines.map(line => line.replace(/\([a-z-]+\)/g, '').trim()));
+ });
+ child.stdin.end(input.join('\n') + '\n');
+ });
+ return parts.map(p => mapPhonemes(p.map(x => x.index === undefined ? x.text : x.before + output[x.index] + x.after).join('')));
+}
+
+export function generateSpeech(tts, text, { language = 'en', voice, speed, phonemes } = {}) {
+ if (language === 'en') return tts.generate(text, { voice, speed });
+ const { input_ids } = tts.tokenizer(phonemes, { truncation: true });
+ return tts.generate_from_ids(input_ids, { voice, speed });
+}
diff --git a/src/project.js b/src/project.js
index e8fabbd..61c5342 100644
--- a/src/project.js
+++ b/src/project.js
@@ -6,7 +6,7 @@
import fs from 'node:fs';
import path from 'node:path';
import { parseScript } from './script.js';
-import { VOICES } from './models.js';
+import { VOICES, LANGUAGES, voiceLanguage } from './models.js';
import { THEMES as LOOKS } from '../engine/themes.js';
import { parseColor } from '../engine/util.js';
@@ -36,6 +36,7 @@ export const DEFAULTS = {
theme: 'paper',
size: '16:9',
fps: 30,
+ language: 'en',
voice: 'af_heart',
speed: 0.9,
pace: 1,
@@ -159,6 +160,8 @@ export function resolveSize(size) {
export function normalizeConfig(raw, dir = '.') {
const cfg = { ...DEFAULTS, ...raw };
+ if (typeof cfg.language !== 'string' || !Object.hasOwn(LANGUAGES, cfg.language)) fail(`language must be one of ${Object.keys(LANGUAGES).join(', ')}`);
+ if (raw.voice === undefined) cfg.voice = LANGUAGES[cfg.language];
const unknown = Object.keys(raw).filter((k) => !(k in DEFAULTS) && !k.startsWith('_'));
if (unknown.length) fail(`video.json has unknown setting(s): ${unknown.join(', ')}. Known: ${Object.keys(DEFAULTS).join(', ')}`);
if (!THEMES.includes(cfg.theme) && !lookPath(cfg.theme)) fail(`theme must be one of ${THEMES.join(', ')} or a relative path to a .json look file, not "${cfg.theme}"`);
@@ -182,6 +185,7 @@ export function normalizeConfig(raw, dir = '.') {
}
if (typeof cfg.voice === 'string') {
if (!Object.hasOwn(VOICES, cfg.voice)) fail(`voice "${cfg.voice}" does not exist. Run "explainroo voices" for the list.`);
+ if (voiceLanguage(cfg.voice) !== cfg.language) fail(`voice "${cfg.voice}" does not speak ${cfg.language}. Use ${Object.keys(VOICES).filter(id => voiceLanguage(id) === cfg.language).join(', ')}.`);
} else if (!cfg.voice || typeof cfg.voice !== 'object') fail('voice must be a Kokoro id or an object with provider');
if (typeof cfg.voice !== 'string' && raw.speed === undefined) cfg.speed = 1;
if (!(cfg.speed >= 0.6 && cfg.speed <= 1.6)) fail('speed must be between 0.6 and 1.6');
diff --git a/src/qa.js b/src/qa.js
index 3bee7ef..f4f28a6 100644
--- a/src/qa.js
+++ b/src/qa.js
@@ -15,7 +15,7 @@ export async function check(project, { log = () => {}, step = 0.25, viewWidth =
const issues = chapterHints(buildChapters(timeline, project.script.title || project.config.title));
for (const sc of project.script.scenes) {
const v = voices[sc.id];
- const suspicious = v ? v.unmatched.filter((w) => w.replace(/[^\p{L}\p{N}]/gu, '').length > 3) : [];
+ const suspicious = v ? v.unmatched.filter((w) => w.replace(/[^\p{L}\p{M}\p{N}]/gu, '').length > 3) : [];
if (v && v.duration && (v.matchRate < 0.9 || suspicious.length)) {
issues.push({ level: 'warn', scene: sc.id, t: null, message: `the voice check could not confirm ${suspicious.length ? suspicious.slice(0, 8).map((w) => `"${w}"`).join(', ') : `${Math.round((1 - v.matchRate) * 100)}% of the words`}. It heard: "${v.transcript.slice(0, 200)}". ${project.config.voice.provider === 'recording' ? 'The recording does not match script.md here. Make script.md say what was actually said.' : 'If a word is mispronounced, write it as {shown|spoken} in script.md.'}` });
}
@@ -148,9 +148,9 @@ export async function verify(project, { file = null, log = () => {} } = {}) {
if (!words.length) continue;
const from = Math.max(0, Math.floor(t.start * 16000));
const to = Math.min(samples.length, Math.ceil((t.start + t.dur) * 16000));
- const heard = await transcribeWords(samples.subarray(from, to), { log });
+ const heard = await transcribeWords(samples.subarray(from, to), { log, language: project.config.language });
texts.push(heard.text);
- const rate = alignWords(words, heard.words, t.dur).matchRate;
+ const rate = alignWords(words, heard.words, t.dur, project.config.language).matchRate;
scenes.push({ id: t.id, matchRate: Math.round(rate * 1000) / 1000 });
total += words.length;
matched += rate * words.length;
@@ -159,9 +159,9 @@ export async function verify(project, { file = null, log = () => {} } = {}) {
const rate = total ? matched / total : 1;
speech = { matchRate: Math.round(rate * 1000) / 1000, transcript: texts.join(' '), scenes };
} else {
- const heard = await transcribeWords(samples, { log });
+ const heard = await transcribeWords(samples, { log, language: project.config.language });
const script = project.script.scenes.flatMap(spokenWords);
- speech = { matchRate: Math.round(alignWords(script, heard.words, duration).matchRate * 1000) / 1000, transcript: heard.text };
+ speech = { matchRate: Math.round(alignWords(script, heard.words, duration, project.config.language).matchRate * 1000) / 1000, transcript: heard.text };
}
if (speech.matchRate < 0.85) issues.push({ level: 'warn', message: `only ${Math.round(speech.matchRate * 100)}% of the script was understood in the final mix; lower music.volume or check pronunciation` });
fs.rmSync(wav, { force: true });
diff --git a/src/recordings.js b/src/recordings.js
index 61da60d..0dcd978 100644
--- a/src/recordings.js
+++ b/src/recordings.js
@@ -72,9 +72,9 @@ export async function saveRecording(file, samples, sampleRate) {
}
}
-export function recordingCuts(scenes, heard, duration) {
+export function recordingCuts(scenes, heard, duration, language = 'en') {
const spoken = scenes.map((scene) => ({ scene, words: scene.units.filter((u) => u.type === 'word') })).filter((s) => s.words.length);
- const aligned = alignWords(spoken.flatMap((s) => s.words), heard, duration).words;
+ const aligned = alignWords(spoken.flatMap((s) => s.words), heard, duration, language).words;
let offset = 0;
const spans = spoken.map(({ scene, words }) => {
const times = aligned.slice(offset, offset + words.length);
diff --git a/src/script.js b/src/script.js
index ff067f8..53b35ed 100644
--- a/src/script.js
+++ b/src/script.js
@@ -22,7 +22,7 @@ export function normWord(w) {
.normalize('NFKD')
.replace(/[̀-ͯ]/g, '')
.replace(/['’]/g, '')
- .replace(/[^a-z0-9%$€£+#@.]/g, '')
+ .replace(/[^\p{L}\p{M}\p{N}%$€£+#@.]/gu, '')
.replace(/^\.+|\.+$/g, '');
}
@@ -61,7 +61,7 @@ export function tokenizeNarration(text, line = 0) {
const n = text.length;
const pushWord = (display, spoken) => {
const prev = units[units.length - 1];
- if (!pendingSpace && prev && prev.type === 'word' && /^[.,;:!?)\]"'’…]+$/.test(display)) {
+ if (!pendingSpace && prev && prev.type === 'word' && /^[.,;:!?।)\]"'’…]+$/.test(display)) {
prev.display += display;
prev.spoken += spoken;
return;
@@ -223,7 +223,7 @@ export function speechChunks(units, { sentenceGap = 0.3, paragraphGap = 0.55, pa
}
cur.text += (cur.words.length ? ' ' : '') + u.spoken;
cur.words.push({ unitIndex: index, display: u.display, spoken: u.spoken });
- if (/[.!?…]["'’)\]]*$/.test(u.spoken)) {
+ if (/[.!?…।]["'’)\]]*$/.test(u.spoken)) {
close();
gapBefore = sentenceGap / pace;
}
diff --git a/src/subtitles.js b/src/subtitles.js
index cfdc881..3a3ee72 100644
--- a/src/subtitles.js
+++ b/src/subtitles.js
@@ -1,6 +1,6 @@
// Subtitle cues use the shown words and their times from the voice timeline.
-const sentenceEnd = /[.!?…]["'’”)\]]*$/;
-const punctuation = /[,;:.!?…]["'’”)\]]*$/;
+const sentenceEnd = /[.!?…।]["'’”)\]]*$/;
+const punctuation = /[,;:.!?…।]["'’”)\]]*$/;
function lines(words) {
const text = words.map((w) => w.text).join(' ');
diff --git a/src/voice.js b/src/voice.js
index 1f54bc3..f50c017 100644
--- a/src/voice.js
+++ b/src/voice.js
@@ -6,7 +6,8 @@ import path from 'node:path';
import crypto from 'node:crypto';
import { speechChunks, resolveMarks } from './script.js';
import { alignWords } from './align.js';
-import { loadTTS, transcribeWords, TTS_MODEL, ASR_MODEL, TTS_SAMPLE_RATE, ttsDtype } from './models.js';
+import { loadTTS, transcribeWords, TTS_MODEL, TTS_SAMPLE_RATE, ttsDtype, asrModel } from './models.js';
+import { phonemize, generateSpeech } from './phonemes.js';
import { writeWav, readWav, resample, trimSilence } from './wav.js';
import { elevenSpeech } from './elevenlabs.js';
import { recordingFiles, decodeRecording, saveRecording, recordingCuts } from './recordings.js';
@@ -19,7 +20,7 @@ export function sceneHash(scene, chunks, config, recording = null) {
.update(JSON.stringify({
PIPELINE,
TTS_MODEL,
- ASR_MODEL,
+ ASR_MODEL: asrModel(config.language),
dtype: ttsDtype(),
voice: config.voice,
speed: config.speed * config.pace,
@@ -29,6 +30,7 @@ export function sceneHash(scene, chunks, config, recording = null) {
layout: (recording ? scene.units.filter((u) => u.type !== 'pause') : scene.units).map((u) => (u.type === 'mark' ? `#${u.name}` : u.type === 'word' ? u.display : '')).join('|'),
// Added last and only for recordings, so Kokoro scenes keep their cache.
...(recording ? { recording } : {}),
+ ...(config.language && config.language !== 'en' ? { language: config.language } : {}),
}))
.digest('hex')
.slice(0, 16);
@@ -84,8 +86,8 @@ export async function synthesize(project, { force = false, log = () => {}, only
if (needsSpeech && recordings && !recordings.folder) {
log('transcribing the recording and finding scene boundaries');
take = await decodeRecording(Object.values(recordings.files)[0].file, project.paths.build, wanted);
- heardTake = await transcribeWords(resample(take.samples, take.sampleRate, 16000), { log });
- cuts = recordingCuts(project.script.scenes, heardTake.words, take.samples.length / take.sampleRate);
+ heardTake = await transcribeWords(resample(take.samples, take.sampleRate, 16000), { log, language: config.language });
+ cuts = recordingCuts(project.script.scenes, heardTake.words, take.samples.length / take.sampleRate, config.language);
}
for (const { scene, chunks, hash } of todo) {
const t0 = Date.now();
@@ -122,6 +124,8 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {
let matchedTokens = 0;
let totalTokens = 0;
+ const phonemes = config.voice.provider !== 'elevenlabs' && config.language !== 'en'
+ ? await phonemize(chunks.map(c => c.text), config.language) : null;
for (let ci = 0; ci < chunks.length; ci++) {
const chunk = chunks[ci];
const gap = ci === 0 ? 0 : chunk.gapBefore;
@@ -131,16 +135,16 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {
}
const audio = config.voice.provider === 'elevenlabs'
? await elevenSpeech(project, scene.id, chunk.text, { previous: chunks[ci - 1]?.text, next: chunks[ci + 1]?.text })
- : await tts.generate(chunk.text, { voice: config.voice, speed: config.speed * config.pace });
+ : await generateSpeech(tts, chunk.text, { language: config.language, voice: config.voice, speed: config.speed * config.pace, phonemes: phonemes?.[ci] });
const samples = trimSilence(audio.audio, audio.sampling_rate || sr);
const start = cursor / sr;
const dur = samples.length / sr;
pieces.push(samples);
cursor += samples.length;
- const heard = await transcribeWords(resample(samples, sr, 16000), { log });
+ const heard = await transcribeWords(resample(samples, sr, 16000), { log, language: config.language });
transcripts.push(heard.text);
- const aligned = alignWords(chunk.words, heard.words, dur);
+ const aligned = alignWords(chunk.words, heard.words, dur, config.language);
chunk.words.forEach((w, k) => {
const a = aligned.words[k];
wordTimes[w.unitIndex] = { start: start + a.start, end: start + a.end, matched: a.matched };
@@ -181,6 +185,7 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {
}
async function recordingScene(project, scene, hash, source, take, cut, heardTake, log) {
+ const { config } = project;
const audio = take || await decodeRecording(source.file, project.paths.build, project.config.speed * project.config.pace);
const sr = audio.sampleRate;
let from = cut ? Math.floor(cut.start * sr) : 0;
@@ -202,9 +207,9 @@ async function recordingScene(project, scene, hash, source, take, cut, heardTake
};
const heard = heardTake ? {
words: heardTake.words.filter((w) => w.start >= cut.start && w.start < cut.end).map(local),
- } : await transcribeWords(resample(samples, sr, 16000), { log });
+ } : await transcribeWords(resample(samples, sr, 16000), { log, language: config.language });
const units = scene.units.filter((u) => u.type === 'word');
- const aligned = alignWords(units, heard.words, duration);
+ const aligned = alignWords(units, heard.words, duration, config.language);
const times = cut ? cut.words.map(local) : aligned.words;
const wordTimes = [];
let wi = 0;
diff --git a/test/languages-browser.test.js b/test/languages-browser.test.js
new file mode 100644
index 0000000..2889004
--- /dev/null
+++ b/test/languages-browser.test.js
@@ -0,0 +1,78 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { findChrome } from '../src/browser.js';
+import { loadProject, THEMES } from '../src/project.js';
+import { buildTimeline } from '../src/timeline.js';
+import { withEngine } from '../src/pipeline.js';
+
+let chrome;
+try { chrome = findChrome(); } catch {}
+
+const sentences = [
+ '¿Qué pasó? ¡El niño llegó al café!',
+ 'La sœur prépare une leçon en français.',
+ 'La città è bella e il caffè è pronto.',
+ 'Não há açúcar, mas os pães são bons.',
+ 'हिन्दी में शब्दों और तस्वीरों से जानकारी समझाएँ।',
+];
+
+test('every look uses bundled fonts for all five languages, including Devanagari in captions and UI', { skip: !chrome }, async t => {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-language-fonts-'));
+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
+ fs.writeFileSync(path.join(dir, 'script.md'), '## demo {min=3}');
+ fs.writeFileSync(path.join(dir, 'scenes.js'), `export default { demo(s) { s.title('Qué', { at: 0 }); } };`);
+ for (const theme of THEMES) {
+ fs.writeFileSync(path.join(dir, 'video.json'), JSON.stringify({ theme, captions: true, sfx: false }));
+ const project = loadProject(dir);
+ await withEngine(project, buildTimeline(project, {}), {}, async page => {
+ const cdp = await page.context().newCDPSession(page);
+ await cdp.send('DOM.enable');
+ await cdp.send('CSS.enable');
+ const { root } = await cdp.send('DOM.getDocument');
+ await page.evaluate(async ([sentences, theme]) => {
+ const { getTheme, fontString } = await import('/engine/themes.js');
+ const { UI } = await import('/engine/ui.js');
+ const fonts = ['display', 'body', 'hand', 'mono'].map(role => fontString(getTheme(theme), role, 48));
+ fonts.push(UI.prototype.fontString.call({ font: 'Inter' }, 48));
+ let i = 0;
+ for (const font of fonts) for (const sentence of sentences) {
+ const span = document.createElement('div');
+ span.id = 'glyph' + i++;
+ span.style.font = font;
+ span.textContent = sentence;
+ document.body.append(span);
+ }
+ await new Promise(resolve => requestAnimationFrame(resolve));
+ }, [sentences, theme]);
+ for (let i = 0; i < 25; i++) {
+ const { nodeId } = await cdp.send('DOM.querySelector', { nodeId: root.nodeId, selector: '#glyph' + i });
+ const { fonts } = await cdp.send('CSS.getPlatformFontsForNode', { nodeId });
+ assert.ok(fonts.length, `${theme} sentence ${i}`);
+ assert.ok(fonts.every(f => f.isCustomFont), `${theme} sentence ${i}: ${JSON.stringify(fonts)}`);
+ if (i % 5 === 4) assert.ok(fonts.some(f => f.familyName === 'Noto Sans Devanagari'), `${theme} Hindi fallback`);
+ }
+ });
+ }
+});
+
+test('accented and Hindi cue words resolve to their spoken times in the page', { skip: !chrome }, async t => {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-language-cues-'));
+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
+ fs.writeFileSync(path.join(dir, 'video.json'), '{}');
+ fs.writeFileSync(path.join(dir, 'script.md'), '## demo\nQué cœur pão città फ़ाइलें');
+ fs.writeFileSync(path.join(dir, 'scenes.js'), `export default { demo(s) {
+ s.text('Qué', { y: 200, at: 'qué' }); s.text('cœur', { y: 300, at: 'CŒUR' });
+ s.text('pão', { y: 400, at: 'PÃO' }); s.text('città', { y: 500, at: 'CITTÀ' });
+ s.text('फ़ाइलें', { y: 600, at: 'फ़ाइलें' });
+ } };`);
+ const project = loadProject(dir);
+ const words = project.script.scenes[0].units.map((u, i) => ({ text: u.display, start: i * 0.5, end: (i + 1) * 0.5 }));
+ const timeline = buildTimeline(project, { demo: { duration: 2.5, words, marks: {} } });
+ await withEngine(project, timeline, {}, async page => {
+ const result = await page.evaluate(() => window.explainroo.check());
+ assert.deepEqual(result.filter(i => i.level === 'error'), []);
+ });
+});
diff --git a/test/languages.test.js b/test/languages.test.js
new file mode 100644
index 0000000..081bc2a
--- /dev/null
+++ b/test/languages.test.js
@@ -0,0 +1,133 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import crypto from 'node:crypto';
+import { normWord, parseScript, speechChunks, tokenizeNarration } from '../src/script.js';
+import { normWord as browserNorm } from '../engine/util.js';
+import { alignWords, expand } from '../src/align.js';
+import { normalizeConfig, ProjectError } from '../src/project.js';
+import { VOICES, LANGUAGES, voiceLanguage, asrModel, ASR_MODEL, MULTILINGUAL_ASR_MODEL, TTS_MODEL, ttsDtype } from '../src/models.js';
+import { sceneHash } from '../src/voice.js';
+import { recordingCuts } from '../src/recordings.js';
+import { subtitleCues, toSrt, toVtt } from '../src/subtitles.js';
+import { buildPhrases } from '../engine/captions.js';
+import { main } from '../src/cli.js';
+
+test('word normalization keeps letters, marks and numbers across scripts, with the existing accent rules', () => {
+ for (const [word, expected] of [
+ ['Qué', 'que'], ['¿NIÑO?', 'nino'], ['français', 'francais'], ['cœur', 'cœur'],
+ ['ação', 'acao'], ['pão', 'pao'], ['città', 'citta'], ['हिन्दी।', 'हिन्दी'],
+ ['फ़ाइलें', 'फ़ाइलें'], ['१२३', '१२३'], ['किंतु', 'किंतु'], ['देवनागरी', 'देवनागरी'],
+ ["Visitor’s", 'visitors'], ['example.com.', 'example.com'], ['$1,050.50', '$1050.50'], ['API+#@', 'api+#@'],
+ ]) {
+ assert.equal(normWord(word), expected, word);
+ assert.equal(browserNorm(word), expected, word);
+ assert.equal(normWord(word.normalize('NFD')), expected, word);
+ }
+});
+
+test('sentence boundaries include danda and Spanish question and exclamation marks', () => {
+ const words = tokenizeNarration('¿Qué pasa? ¡Todo bien! \"यह अच्छा है।\" फिर चलें।');
+ const chunks = speechChunks(words);
+ assert.deepEqual(chunks.map(c => c.text), ['¿Qué pasa?', '¡Todo bien!', '\"यह अच्छा है।\"', 'फिर चलें।']);
+ assert.deepEqual(chunks.map(c => c.gapBefore), [0, 0.3, 0.3, 0.3]);
+ assert.equal(tokenizeNarration('एक {दो|दो}।').length, 2);
+});
+
+test('each language picks its default and rejects every voice from a different language', () => {
+ assert.equal(normalizeConfig({}).language, 'en');
+ for (const [language, id] of Object.entries(LANGUAGES)) {
+ assert.equal(normalizeConfig({ language }).voice, id);
+ assert.equal(normalizeConfig({ language, voice: { provider: 'kokoro', id } }).voice, id);
+ for (const voice of Object.keys(VOICES)) {
+ if (voiceLanguage(voice) === language) assert.equal(normalizeConfig({ language, voice }).voice, voice);
+ else assert.throws(() => normalizeConfig({ language, voice }), e => e instanceof ProjectError && e.message.includes(id));
+ }
+ for (const voice of [{ provider: 'elevenlabs', id: 'sample' }, { provider: 'recording', src: 'assets/take.wav' }]) {
+ assert.equal(normalizeConfig({ language, voice }).language, language);
+ }
+ assert.equal(asrModel(language), language === 'en' ? ASR_MODEL : MULTILINGUAL_ASR_MODEL);
+ }
+ for (const language of ['ja', 'zh', 'pt', '', 'EN', 'toString', null, true, ['es']]) assert.throws(() => normalizeConfig({ language }), /language must be one of/);
+});
+
+test('non-English numbers stay digits, including Devanagari, and are never fuzzy matched or spelled out', () => {
+ for (const language of ['es', 'fr', 'it', 'pt-br', 'hi']) {
+ assert.deepEqual(expand('1,050', language), ['1050']);
+ assert.deepEqual(expand('€1.050', language), ['1050']);
+ assert.deepEqual(expand('१२%', language), ['12']);
+ assert.deepEqual(expand('twenty', language), ['twenty']);
+ const r = alignWords([{ spoken: '१२३' }, { spoken: '1050' }], [{ text: '123', start: 0, end: 0.5 }, { text: '1,050', start: 0.5, end: 1 }], 1, language);
+ assert.equal(r.matchRate, 1);
+ assert.equal(alignWords([{ spoken: '1234' }], [{ text: '1235', start: 0, end: 1 }], 1, language).matchRate, 0);
+ assert.equal(alignWords([{ spoken: 'twenty' }], [{ text: '20', start: 0, end: 1 }], 1, language).matchRate, 0);
+ }
+ assert.deepEqual(expand('twenty'), ['20']);
+});
+
+test('Hindi recording cuts use the same language rules as per-scene alignment', () => {
+ const scenes = parseScript('## first\n१२३\n## second\nफ़ाइलें').scenes;
+ const cuts = recordingCuts(scenes, [{ text: '123', start: 0.2, end: 0.5 }, { text: 'फ़ाइलें', start: 1, end: 1.5 }], 2, 'hi');
+ assert.ok(cuts.every(c => c.words[0].matched));
+});
+
+test('English cache hashes keep the previous format; other languages are part of the cache', () => {
+ const scene = parseScript('## first\n[#a] Hello. Goodbye.').scenes[0];
+ const chunks = speechChunks(scene.units);
+ const config = normalizeConfig({});
+ const previous = crypto.createHash('sha256').update(JSON.stringify({
+ PIPELINE: 7, TTS_MODEL, ASR_MODEL, dtype: ttsDtype(), voice: config.voice, speed: config.speed * config.pace,
+ chunks: chunks.map(c => [c.gapBefore, c.text]),
+ layout: scene.units.map(u => u.type === 'mark' ? `#${u.name}` : u.type === 'word' ? u.display : '').join('|'),
+ })).digest('hex').slice(0, 16);
+ assert.equal(sceneHash(scene, chunks, config), previous);
+ assert.equal(sceneHash(scene, chunks, { ...config, language: undefined }), previous);
+ const voice = { provider: 'recording', src: 'assets/take.wav' };
+ assert.notEqual(sceneHash(scene, chunks, normalizeConfig({ language: 'fr', voice })), sceneHash(scene, chunks, normalizeConfig({ language: 'es', voice })));
+});
+
+test('captions and subtitle files keep native text and split at danda', () => {
+ for (const text of ['¿Qué pasó?', 'Le cœur écoute.', 'La città è bella.', 'Não há açúcar.', 'यह अच्छा है। फिर चलें।']) {
+ const words = text.split(' ').map((text, i) => ({ text, start: i * 0.2, end: (i + 1) * 0.2 }));
+ const timeline = { scenes: [{ start: 0, dur: 10, words }] };
+ const cues = subtitleCues(timeline);
+ assert.equal(cues.map(c => c.text.replaceAll('\n', ' ')).join(' '), text);
+ assert.ok(toSrt(cues).includes(text.split(' ')[0]));
+ assert.ok(toVtt(cues).includes(text.split(' ')[0]));
+ if (text.includes('।')) {
+ assert.equal(cues.length, 2);
+ assert.equal(buildPhrases(timeline, 80).length, 2);
+ }
+ }
+});
+
+test('init stores the language and default voice; voices filters and doctor allows missing espeak', async t => {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-language-cli-'));
+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
+ let output = '';
+ let errors = '';
+ t.mock.method(process.stdout, 'write', s => { output += s; return true; });
+ t.mock.method(process.stderr, 'write', s => { errors += s; return true; });
+ assert.equal(await main(['init', path.join(dir, 'video'), '--language', 'es', '--json']), 0);
+ const config = JSON.parse(fs.readFileSync(path.join(dir, 'video/video.json')));
+ assert.equal(config.language, 'es');
+ assert.equal(config.voice, 'ef_dora');
+ for (const flags of [['--language', 'ja'], ['--language'], ['--language', 'es', '--voice', 'af_heart']]) {
+ assert.equal(await main(['init', path.join(dir, 'bad'), ...flags]), 1);
+ assert.equal(fs.existsSync(path.join(dir, 'bad')), false);
+ }
+ assert.match(errors, /--language needs a value/);
+ output = '';
+ assert.equal(await main(['voices', '--language', 'es', '--json']), 0);
+ const list = JSON.parse(output);
+ assert.deepEqual(Object.keys(list.voices), ['ef_dora', 'em_alex', 'em_santa']);
+ assert.equal(list.languages.ef_dora, 'es');
+ const saved = process.env.EXPLAINROO_ESPEAK;
+ process.env.EXPLAINROO_ESPEAK = path.join(dir, 'missing');
+ t.after(() => { if (saved === undefined) delete process.env.EXPLAINROO_ESPEAK; else process.env.EXPLAINROO_ESPEAK = saved; });
+ output = '';
+ assert.equal(await main(['doctor', '--json']), 0);
+ assert.match(JSON.parse(output).checks.find(c => c.name === 'espeak-ng').detail, /only needed for other languages/);
+});
diff --git a/test/phonemes.test.js b/test/phonemes.test.js
new file mode 100644
index 0000000..5d6fde6
--- /dev/null
+++ b/test/phonemes.test.js
@@ -0,0 +1,90 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { mapPhonemes, phonemize, espeakVersion, generateSpeech } from '../src/phonemes.js';
+
+function fake(t, source) {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-espeak-'));
+ const file = path.join(dir, 'fake espeak');
+ fs.writeFileSync(file, '#!/usr/bin/env node\n' + source, { mode: 0o755 });
+ const previous = process.env.EXPLAINROO_ESPEAK;
+ process.env.EXPLAINROO_ESPEAK = file;
+ t.after(() => {
+ if (previous === undefined) delete process.env.EXPLAINROO_ESPEAK;
+ else process.env.EXPLAINROO_ESPEAK = previous;
+ fs.rmSync(dir, { recursive: true, force: true });
+ });
+ return file;
+}
+
+test('maps every misaki v1.0 tie sequence and leaves language-specific IPA intact', () => {
+ assert.equal(mapPhonemes(' a^ɪ a^ʊ d^z d^ʒ e^ɪ o^ʊ ə^ʊ s^s t^s t^ʃ ɔ^ɪ '), 'I W ʣ ʤ A O Q S ʦ ʧ Y');
+ assert.equal(mapPhonemes('«ʧˈit-t^sːa» r x ɬ ʲ œ̃ ɔ̃ ɑ̃ ɛ̃ ɐ̃ ʊ̃ n̩'), '(ʧˈitʦːa) r x ɬ ʲ œ̃ ɔ̃ ɑ̃ ɛ̃ ɐ̃ ʊ̃ n̩');
+ assert.equal(mapPhonemes('कˈʌcʰcʰi'), 'कˈʌcʰcʰi');
+});
+
+test('batches text through stdin, preserves punctuation and removes language flags', async t => {
+ fake(t, `
+ if (process.argv.includes('--version')) { console.log('eSpeak NG 1.51'); process.exit(0); }
+ const assert = require('node:assert/strict');
+ assert.deepEqual(process.argv.slice(2), ['-q', '-b', '1', '-v', 'fr-fr', '--ipa=1', '--tie=^']);
+ let text = '';
+ process.stdin.on('data', d => text += d);
+ process.stdin.on('end', () => {
+ assert.equal(text, ['bonjour', 'merci', 'שלום $HOME \`echo bad\`', 'encore'].join('\\n') + '\\n');
+ process.stdout.write('(en) t^ʃa\\nd^ʒa\\ne^ɪ\\no^ʊ\\n');
+ });
+ `);
+ assert.equal(espeakVersion(), 'eSpeak NG 1.51');
+ const result = await phonemize(['«bonjour», (merci)!', 'שלום $HOME `echo bad`\nencore'], 'fr');
+ assert.deepEqual(result, ['“ʧa”, (ʤa)!', 'A\nO']);
+});
+
+test('punctuation-only text needs no program', async t => {
+ const file = fake(t, 'process.exit(1);');
+ fs.rmSync(file);
+ assert.deepEqual(await phonemize(['¿?!', '(…)'], 'es'), ['¿?!', '(…)']);
+ assert.equal(espeakVersion(), null);
+});
+
+test('missing and non-executable programs give install instructions for every platform', async t => {
+ const file = fake(t, '');
+ for (const missing of [false, true]) {
+ if (missing) fs.rmSync(file);
+ else fs.chmodSync(file, 0o644);
+ await assert.rejects(phonemize(['Hola'], 'es'), e => {
+ assert.match(e.message, /sudo apt install espeak-ng/);
+ assert.match(e.message, /brew install espeak-ng/);
+ assert.match(e.message, /espeak-ng\/espeak-ng\/releases/);
+ assert.match(e.message, /EXPLAINROO_ESPEAK/);
+ return true;
+ });
+ }
+});
+
+test('failed and incomplete output gives a friendly error', async t => {
+ const file = fake(t, 'process.stderr.write("failure"); process.exit(2);');
+ await assert.rejects(phonemize(['Hola'], 'es'), /could not read the text/);
+ fs.writeFileSync(file, '#!/usr/bin/env node\nconsole.log("one");');
+ await assert.rejects(phonemize(['Hola', 'Adiós'], 'es'), /incomplete phonemes/);
+ await assert.rejects(phonemize(['こんにちは'], 'ja'), /supports es, fr, it, pt-br and hi/);
+});
+
+test('English keeps generate; other languages tokenize mapped phonemes and call generate_from_ids', async () => {
+ const calls = [];
+ const ids = { dims: [1, 4] };
+ const tts = {
+ generate(text, options) { calls.push(['english', text, options]); return 'audio'; },
+ tokenizer(text, options) { calls.push(['tokens', text, options]); return { input_ids: ids }; },
+ generate_from_ids(input, options) { calls.push(['ids', input, options]); return 'audio'; },
+ };
+ assert.equal(await generateSpeech(tts, 'Hello.', { voice: 'af_heart', speed: 0.9 }), 'audio');
+ assert.equal(await generateSpeech(tts, 'Bonjour.', { language: 'fr', voice: 'ff_siwis', speed: 1, phonemes: 'bɔ̃ʒˈuʁ.' }), 'audio');
+ assert.deepEqual(calls, [
+ ['english', 'Hello.', { voice: 'af_heart', speed: 0.9 }],
+ ['tokens', 'bɔ̃ʒˈuʁ.', { truncation: true }],
+ ['ids', ids, { voice: 'ff_siwis', speed: 1 }],
+ ]);
+});