Explainer videos and product demos made by your AI agent. Free and open source: a local voice (Kokoro), word timing (Whisper) and a canvas renderer turn a script into a narrated MP4.
JavaScript ai-agentsanimationcanvasclaude-codecodexexplainer-videoffmpegjavascriptkokoroproduct-demotext-to-speechvideo-generationwhisper
Add Japanese and Mandarin Chinese narration (#26)
Vincent Schmalbacha4208379623a
diff --git a/AGENTS.md b/AGENTS.mdindex 84a0de5..c23fe3b 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -34,9 +34,10 @@ npm installnode bin/explainroo.js doctor --fetch```-`doctor` checks Node, ffmpeg, Chrome and espeak-ng. It also downloads the-English speech models, about 400 MB, the first time only. Other languages-fetch their speech check model on first use. If the user ran `npm link`, you+`doctor` checks Node, ffmpeg and Chrome, and the extra programs some+languages need. It also downloads the English speech models, about 400 MB,+the first time only. Other languages fetch their speech check model on first+use. If the user ran `npm link`, youcan write `explainroo` instead of `node bin/explainroo.js` in the commandsbelow. Commands that work on a video take the project folder first, and everycommand accepts `--json`.@@ -303,9 +304,10 @@ Run `voice`, `check` and `render` as usual with any of the three sources.## Videos in other languages-Set `language` in `video.json` to `en`, `es`, `fr`, `it`, `pt-br` or `hi`.-English is the default. `pt-br` means Brazilian Portuguese. You can also-start with `explainroo init videos/my-video --language es`.+Set `language` in `video.json` to `en`, `es`, `fr`, `it`, `pt-br`, `hi`,+`ja` or `zh`. English is the default. `pt-br` means Brazilian Portuguese and+`zh` Mandarin Chinese. You can also start with+`explainroo init videos/my-video --language es`.When `voice` is left out, the language picks its Kokoro voice:@@ -317,6 +319,8 @@ When `voice` is left out, the language picks its Kokoro voice:| Italian (`it`) | `if_sara` || Brazilian Portuguese (`pt-br`) | `pf_dora` || Hindi (`hi`) | `hf_alpha` |+| Japanese (`ja`) | `jf_alpha` |+| Mandarin Chinese (`zh`) | `zf_xiaobei` |Run `explainroo voices --language es` to see voices for one language.A Kokoro voice must match the video's language. If a project already has@@ -328,12 +332,54 @@ Cue words and `[#markers]` work the same. `at: 'qué'` matches "Qué".Hindi sentences can end with `।`. Captions and subtitle files keep thewritten text. The looks include a Hindi font, so no system font is needed.-Kokoro needs espeak-ng once for these five languages. Install it on Linux+Spanish, French, Italian, Portuguese and Hindi need the espeak-ng program.+Install it on Linuxwith `sudo apt install espeak-ng`, or on macOS with `brew install espeak-ng`.On Windows, use the installer from the[espeak-ng releases page](https://github.com/espeak-ng/espeak-ng/releases).If the program is in another place, set `EXPLAINROO_ESPEAK` to its binary.-English, ElevenLabs and recordings do not need it.+English, Japanese, Chinese, ElevenLabs and recordings do not need it.++### Japanese and Chinese++Japanese and Chinese need Python 3 with misaki, the pronunciation library+Kokoro's voices were trained with. Install what the language needs:++```bash+python3 -m pip install "misaki[ja]" # Japanese+python3 -m unidic download # Japanese dictionary, about 526 MB+python3 -m pip install "misaki[zh]" # Chinese+```++On Windows, write `python` instead of `python3`. If misaki is in a virtual+environment, set `EXPLAINROO_PYTHON` to that environment's Python.+ElevenLabs and recordings need neither.++Write the narration as usual, without spaces between words. explainroo+finds the words itself, so cue words and markers work: `at: '解析'` matches+the word 解析, and a phrase like `at: '名前の解析'` works too. A cue must+match whole words. Put a `[#marker]` where you need a moment inside a word.+Sentences end with `。`, `!` or `?`.++Latin words such as API or JavaScript are read in English. For a Japanese+reading, write it out: `{API|エーピーアイ}`.++Whisper often writes a word with other characters that sound the same, so+the speech check compares sounds. Japanese is compared by its kana reading,+so 世紀 heard as 正規 still counts. Chinese is compared by pinyin without+tones, and Traditional and Simplified characters count as the same. So do+not reword a sentence only because Whisper chose other characters. Captions+and subtitles join words without spaces, with up to 18 characters per+subtitle line, and never break inside a word. Keep the shown text of an+override short.++The looks' fonts have no Japanese or Chinese characters. The first `voice`,+`still` or `render` downloads Noto Sans JP or Noto Sans SC, regular and bold,+about 33 MB, and checks the files. They are saved in+`~/.cache/explainroo/fonts/` (or `$EXPLAINROO_CACHE/fonts/`) and used in+every look, in captions and in app screens. After that first download,+`EXPLAINROO_OFFLINE=1` works. A web player made with `export` keeps only+the characters the video shows, so its fonts stay small.The first speech check in another language downloads multilingual Whispersmall. `doctor --fetch` still fetches only the English models. `verify`@@ -344,7 +390,6 @@ If a symbol, short form or word sounds wrong, use `{shown|spoken}` inlanguage. The speech check accepts digits for numbers. It does not try tomatch spelled numbers to digits outside English. Write the digits, or usean override whose spoken text has digits when that helps the check.-Japanese and Mandarin are not supported.In Hindi, the speech check confirms fewer words than in the other languages,because the speech model knows Hindi less well. A few unconfirmed words are@@ -678,7 +723,7 @@ videos/<name>/| `theme` | `paper` | a built-in look, or a relative path to a `.json` look file (see Custom looks) || `size` | `16:9` | a platform (`youtube`, `shorts`, `tiktok`, `reels`, `vertical`, `instagram`, `linkedin`, `square`), `16:9`, `9:16`, `1:1`, `4:5` or `WIDTHxHEIGHT` || `fps` | 30 | 24, 25, 30, 50 or 60 |-| `language` | `en` | narration language: `en`, `es`, `fr`, `it`, `pt-br` or `hi`; see Videos in other languages |+| `language` | `en` | narration language: `en`, `es`, `fr`, `it`, `pt-br`, `hi`, `ja` or `zh`; see Videos in other languages || `voice` | the language's voice | a Kokoro id or a provider object for Kokoro, ElevenLabs or a recording; English uses `af_heart`; see Voices || `speed` | 0.9 for Kokoro, 1 otherwise | voice speed only, 0.6 to 1.6; ElevenLabs limits speed times pace to 0.7 through 1.2 || `pace` | 1 | speed of the whole video (voice, pauses, animations), 0.7 to 1.6 |@@ -768,10 +813,10 @@ when `ink` or `muted` has too little contrast against `bg`.| `capture [project] <url>` | saves screenshots, wording, colors, fonts and logos from a live site (`--mobile`, `--json`) || `image [project] <name> "<prompt>"` | makes an illustration with OpenRouter || `images [project]` | lists the images and what they cost |-| `voices`, `say "text"` | lists Kokoro voices with their languages (`--language es` filters voices), or makes a sample in the chosen voice's language |+| `voices`, `say "text"` | lists Kokoro voices with their languages (`--language es`, `ja` or `zh` filters voices), or makes a sample in the chosen voice's language || `voices [project] --elevenlabs` | lists the names, ids and labels available to the key (`--json` also works) || `themes`, `formats`, `icons <word>` | lists the looks and the sizes, searches the 1,854 icons |-| `doctor` | checks Node, ffmpeg, Chrome, espeak-ng and models (`--fetch` downloads only English models); missing espeak-ng is allowed |+| `doctor` | checks Node, ffmpeg, Chrome, the models, and espeak-ng and Python with misaki for other languages (`--fetch` downloads only English models) |The extra files share the video's name, also with `--out`: `video.srt`,`video.vtt`, `video.cover.jpg` and `video.chapters.txt`. A GIF has no sound,diff --git a/README.md b/README.mdindex 5d97164..bc44e24 100644--- a/README.md+++ b/README.md@@ -53,8 +53,8 @@ explainroo does the rest:- **Voice.** [Kokoro](https://huggingface.co/hexgrad/Kokoro-82M), an openvoice model, reads the script aloud in English, Spanish, French, Italian,- Brazilian Portuguese or Hindi. It has 41 voices and needs no account or API- key. You can also use an [ElevenLabs](https://elevenlabs.io) voice or+ Brazilian Portuguese, Hindi, Japanese or Mandarin Chinese. It has 54 voices+ and needs no account or API key. You can also use an [ElevenLabs](https://elevenlabs.io) voice orrecord the narration yourself.- **Timing.** [Whisper](https://github.com/openai/whisper) listens to thenarration and notes when each word is spoken.@@ -165,7 +165,9 @@ node bin/explainroo.js doctor --fetchFor narration in Spanish, French, Italian, Portuguese or Hindi, also install[espeak-ng](https://github.com/espeak-ng/espeak-ng), for example with-`sudo apt install espeak-ng` or `brew install espeak-ng`.+`sudo apt install espeak-ng` or `brew install espeak-ng`. Japanese and+Chinese need Python with [misaki](https://github.com/hexgrad/misaki) instead,+as described in [AGENTS.md](AGENTS.md#japanese-and-chinese).Then start your agent in the `explainroo` folder and ask for a video, forexample "Make a 60 second video about how HTTPS keeps a password secret." The@@ -192,5 +194,9 @@ icons. [MathJax](https://www.mathjax.org) typesets the formulas.[ffmpeg](https://ffmpeg.org) makes the video file. For other languages,explainroo runs [espeak-ng](https://github.com/espeak-ng/espeak-ng) as aseparate program, with phoneme rules from Kokoro's-[misaki](https://github.com/hexgrad/misaki). The fonts are under the SIL Open-Font License.+[misaki](https://github.com/hexgrad/misaki), and misaki itself for Japanese+and Chinese. [OpenCC](https://github.com/nk2028/opencc-js) and+[pinyin-pro](https://github.com/zh-lx/pinyin-pro) help the speech check+compare Chinese, and [HarfBuzz](https://github.com/harfbuzz/harfbuzzjs) keeps+web player fonts small. The fonts, including Noto Sans CJK for Japanese and+Chinese, are under the SIL Open Font License.diff --git a/engine/captions.js b/engine/captions.jsindex 96c85dd..b635580 100644--- a/engine/captions.js+++ b/engine/captions.js@@ -4,20 +4,23 @@ import { fontString } from './themes.js';import { clamp, ease } from './util.js';export function buildPhrases(timeline, maxChars) {+ const cjk = ['ja', 'zh'].includes(timeline.language);+ const sentenceEnd = cjk ? /[.!?;:।。!?]["'’)」』)]*$/ : /[.!?;:।]["'’)]*$/;+ if (cjk) maxChars = Math.min(maxChars, 18);const phrases = [];for (const sc of timeline.scenes) {// A sentence (or a run of words between long pauses) is split into as few// phrases as fit, balanced in length, so no phrase is one lonely word.let run = [];const flush = () => {- for (const words of balancePhrases(run, maxChars)) phrases.push({ words, start: words[0].start, end: words[words.length - 1].end });+ for (const words of balancePhrases(run, maxChars, cjk ? '' : ' ')) phrases.push({ words, ...(cjk ? { separator: '' } : {}), start: words[0].start, end: words[words.length - 1].end });run = [];};for (const w of sc.words) {const g = { text: w.text, start: sc.start + w.start, end: sc.start + w.end };if (run.length && g.start - run[run.length - 1].end > 0.55) flush();run.push(g);- if (/[.!?;:।]["'’)]*$/.test(g.text)) flush();+ if (sentenceEnd.test(g.text)) flush();}if (run.length) flush();}@@ -30,10 +33,11 @@ export function buildPhrases(timeline, maxChars) {// Splits words into the fewest phrases of at most maxChars characters, with// lengths as even as possible and breaks after commas where they fit.-export function balancePhrases(words, maxChars) {+export function balancePhrases(words, maxChars, separator = ' ') {if (!words.length) return [];- const len = (a, b) => words.slice(a, b).reduce((n, w) => n + w.text.length + 1, -1);+ const len = (a, b) => words.slice(a, b).reduce((n, w) => n + w.text.length + separator.length, -separator.length);if (len(0, words.length) <= maxChars) return [words];+ const punctuation = separator === '' ? /[,;:–—、,;:]["'’)」』)]*$/ : /[,;:–—]["'’)]*$/;const n = words.length;const best = [{ parts: 0, cost: 0, from: -1 }];for (let b = 1; b <= n; b++) {@@ -42,7 +46,7 @@ export function balancePhrases(words, maxChars) {const l = len(a, b);if (l > maxChars && b - a > 1) break;if (!best[a]) continue;- const comma = b < n && /[,;:–—]["'’)]*$/.test(words[b - 1].text) ? 150 : 0;+ const comma = b < n && punctuation.test(words[b - 1].text) ? 150 : 0;const c = { parts: best[a].parts + 1, cost: best[a].cost + (maxChars - l) ** 2 - comma, from: a };if (!best[b] || c.parts < best[b].parts || (c.parts === best[b].parts && c.cost < best[b].cost)) best[b] = c;}@@ -64,7 +68,7 @@ export function drawCaptions(ctx, phrases, T, theme, W, H, area = null) {ctx.font = font;ctx.textBaseline = 'middle';ctx.textAlign = 'left';- const space = ctx.measureText(' ').width;+ const space = ctx.measureText(ph.separator ?? ' ').width;const widths = ph.words.map((w) => ctx.measureText(w.text).width);const maxLine = area ? area.maxWidth - size * 1.1 : W * (vertical ? 0.8 : 0.7);const lines = [[]];diff --git a/engine/runtime.js b/engine/runtime.jsindex 0a636b2..ba1f3fa 100644--- a/engine/runtime.js+++ b/engine/runtime.js@@ -36,6 +36,10 @@ async function loadFonts(theme, config, root = '/') {}));};for (const f of manifest) if (root === './' || wanted.has(f.family)) add(f.family, `${root}fonts/${f.file}`, f);+ if (root !== './' && ['ja', 'zh'].includes(config.language)) {+ const family = config.language === 'ja' ? 'Noto Sans JP' : 'Noto Sans SC';+ for (const weight of [400, 700]) add(family, `/language-fonts/${config.language}-${weight}.otf`, { weight });+ }// A video's own fonts, from "fonts" in video.json.for (const f of config.fonts || []) add(f.family, root + 'project/' + f.src.split('/').map(encodeURIComponent).join('/'), f);await Promise.all(faces);diff --git a/engine/stage.js b/engine/stage.jsindex b59b45f..0c21c7d 100644--- a/engine/stage.js+++ b/engine/stage.js@@ -81,7 +81,8 @@ export class Stage {}_cue(word, n) {- const want = String(word).split(/\s+/).map(normWord).filter(Boolean);+ const cjk = ['ja', 'zh'].includes(this.engine.config.language);+ const want = (cjk ? [String(word)] : String(word).split(/\s+/)).map(normWord).filter(Boolean);if (!want.length) throw new SceneError(`cue("${word}") is empty`);const ws = this.words;if (!ws.length) throw new SceneError(`cue("${word}"): scene "${this.id}" has no narration`);@@ -90,8 +91,15 @@ export class Stage {let found = null;for (let i = 0; i + want.length <= norms.length; i++) {let ok = true;- for (let k = 0; k < want.length; k++) if (norms[i + k] !== want[k]) { ok = false; break; }- if (ok && ++seen === n && !found) found = { start: ws[i].start, end: ws[i + want.length - 1].end };+ let end = i + want.length;+ if (cjk) {+ let phrase = norms[i];+ while (phrase.length < want[0].length && end < norms.length) phrase += norms[end++];+ ok = phrase === want[0];+ } else {+ for (let k = 0; k < want.length; k++) if (norms[i + k] !== want[k]) { ok = false; break; }+ }+ if (ok && ++seen === n && !found) found = { start: ws[i].start, end: ws[end - 1].end };}if (found) {// A word spoken more than once silently means its first time; tell the check.diff --git a/engine/themes.js b/engine/themes.jsindex dc9322f..a490a79 100644--- a/engine/themes.js+++ b/engine/themes.js@@ -231,7 +231,7 @@ export function fontString(theme, role = 'body', size = 48, weight) {const f = theme.fonts[role] || theme.fonts.body;const w = weight ?? f.weight;const fallback = role === 'mono' ? 'monospace' : 'sans-serif';- return `${w} ${Math.round(size * 100) / 100}px "${f.family}", "Noto Sans Devanagari", ${fallback}`;+ return `${w} ${Math.round(size * 100) / 100}px "${f.family}", "Noto Sans Devanagari", "Noto Sans JP", "Noto Sans SC", ${fallback}`;}// Background textures are rendered once and reused on every frame.diff --git a/engine/ui.js b/engine/ui.jsindex 0094420..586fd48 100644--- a/engine/ui.js+++ b/engine/ui.js@@ -121,7 +121,7 @@ export class UI {}fontString(size, weight = 400, family = this.font, italic = false) {- return `${italic ? 'italic ' : ''}${weight} ${size}px "${family}", Inter, "Noto Sans Devanagari", sans-serif`;+ return `${italic ? 'italic ' : ''}${weight} ${size}px "${family}", Inter, "Noto Sans Devanagari", "Noto Sans JP", "Noto Sans SC", sans-serif`;}_family(f) {diff --git a/package-lock.json b/package-lock.jsonindex 67f3cab..d84c537 100644--- a/package-lock.json+++ b/package-lock.json@@ -10,8 +10,12 @@"license": "MIT","dependencies": {"@huggingface/transformers": "3.8.1",+ "harfbuzzjs": "1.6.2","kokoro-js": "1.2.1","mathjax": "4.1.3",+ "opencc-js": "1.4.2",+ "phonemizer": "1.2.1",+ "pinyin-pro": "3.29.4","playwright-core": "1.63.0","roughjs": "4.6.6"},@@ -804,6 +808,12 @@"integrity": "sha512-3GKBOn+m2LX9iq+JC1064cSFprJY4jL1jCXTcpnfER5HYE2l/4EfWSGzkPa/ZDBmYI0ZOEj5VHV/eKnPGkHuOg==","license": "MIT"},+ "node_modules/harfbuzzjs": {+ "version": "1.6.2",+ "resolved": "https://registry.npmjs.org/harfbuzzjs/-/harfbuzzjs-1.6.2.tgz",+ "integrity": "sha512-95c1vWuzoHjM19d5fQgPbz1wcv1pTa0UM9dZrdPgSMUEVuSwuzeVPkq6UhHHw3jLOkE9AYzzTP8XQkRBpPYpEg==",+ "license": "MIT"+ },"node_modules/has-property-descriptors": {"version": "1.0.2","resolved": "https://registry.npmjs.org/has-property-descriptors/-/has-property-descriptors-1.0.2.tgz",@@ -939,6 +949,12 @@"integrity": "sha512-vDJMkfCfb0b1A836rgHj+ORuZf4B4+cc2bASQtpeoJLueuFc5DuYwjIZUBrSvx/fO5IrLjLz+oTrB3pcGlhovQ==","license": "MIT"},+ "node_modules/opencc-js": {+ "version": "1.4.2",+ "resolved": "https://registry.npmjs.org/opencc-js/-/opencc-js-1.4.2.tgz",+ "integrity": "sha512-EdcspgdROiCCZDM+jZR6Pa670xW+EW2QhctPBQ8gRjg6lrx4IHEl+2s+kt5izk/TkxVRgdWuFgaGkaEdIUfdaw==",+ "license": "MIT AND Apache-2.0"+ },"node_modules/path-data-parser": {"version": "0.1.0","resolved": "https://registry.npmjs.org/path-data-parser/-/path-data-parser-0.1.0.tgz",@@ -951,6 +967,12 @@"integrity": "sha512-v0KJ4mi2T4Q7eJQ0W15Xd4G9k4kICSXE8bpDeJ8jisL4RyJhNWsweKTOi88QXFc4r4LZlz5jVL5lCHhkpdT71A==","license": "Apache-2.0"},+ "node_modules/pinyin-pro": {+ "version": "3.29.4",+ "resolved": "https://registry.npmjs.org/pinyin-pro/-/pinyin-pro-3.29.4.tgz",+ "integrity": "sha512-SPXpDT2cHEy+d26V1RXYMlVzXN42hotFAak1fzyWPi4o2dKXb61UqD4pzxDJHwk6gbv8vQ6EfErd+hYX0Qhzug==",+ "license": "MIT"+ },"node_modules/platform": {"version": "1.3.6","resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz",diff --git a/package.json b/package.jsonindex 6a77847..ba1491b 100644--- a/package.json+++ b/package.json@@ -49,8 +49,12 @@},"dependencies": {"@huggingface/transformers": "3.8.1",+ "harfbuzzjs": "1.6.2","kokoro-js": "1.2.1","mathjax": "4.1.3",+ "opencc-js": "1.4.2",+ "phonemizer": "1.2.1",+ "pinyin-pro": "3.29.4","playwright-core": "1.63.0","roughjs": "4.6.6"},diff --git a/src/align.js b/src/align.jsindex 52a87be..910d64b 100644--- a/src/align.js+++ b/src/align.js@@ -1,6 +1,76 @@// Aligns the words we asked the voice model to say with the words Whisper// heard, so every script word gets a start and end time.+import { createRequire } from 'node:module';import { normWord } from './script.js';+import { japaneseReadings } from './phonemes.js';++const require = createRequire(import.meta.url);++// Whisper often writes Mandarin in Traditional characters, even when the+// script is Simplified. Both sides are compared in Simplified.+let toSimplified;+function simplified(text) {+ toSimplified ??= require('opencc-js/t2cn').Converter({ from: 'tw', to: 'cn' });+ return toSimplified(text);+}++// Whisper also often writes another character with the same sound, like 直+// for 值. Mandarin characters match when they share a reading without tones.+const readings = new Map();+function soundsAlike(a, b) {+ const as = Array.from(a);+ const bs = Array.from(b);+ if (as.length !== bs.length) return false;+ const { pinyin } = require('pinyin-pro');+ const of = c => {+ if (!readings.has(c)) readings.set(c, /\p{Script=Han}/u.test(c) ? pinyin(c, { multiple: true, toneType: 'none', type: 'array' }) : [c]);+ return readings.get(c);+ };+ return as.every((c, i) => c === bs[i] || of(c).some(r => of(bs[i]).includes(r)));+}++const katakana = text => text.replace(/[\u3041-\u3096]/g, c => String.fromCharCode(c.charCodeAt(0) + 0x60));++// Gives each piece of text the reading of its characters. A reading that+// covers several characters is shared out between them.+export function spreadReadings(pieces, tokens) {+ const kana = Array.from(pieces.join(''), () => '');+ for (const [start, end, reading] of tokens) {+ const r = Array.from(katakana(reading));+ const n = end - start;+ for (let k = 0; k < n && start + k < kana.length; k++) kana[start + k] = r.slice(Math.round(k * r.length / n), Math.round((k + 1) * r.length / n)).join('');+ }+ let at = 0;+ return pieces.map(p => {+ const length = Array.from(p).length;+ at += length;+ return kana.slice(at - length, at).join('');+ });+}++// Japanese has many words that sound the same, and Whisper often writes a+// word in kana that the script writes in kanji. When Python with misaki is+// there, both sides are compared by their reading.+export async function readAlike(words, heard, language) {+ if (language !== 'ja' || !words.length || !heard.length) return { words, heard };+ try {+ const spoken = words.map(w => String(w.spoken));+ const pieces = heard.map(h => String(h.text).trim());+ const shown = words.map((w, i) => w.display !== undefined && String(w.display) !== spoken[i] ? String(w.display) : null);+ const extra = shown.filter(s => s !== null);+ const [a, b, ...c] = await japaneseReadings([spoken.join(''), pieces.join(''), ...extra]);+ const spokenKana = spreadReadings(spoken, a);+ const heardKana = spreadReadings(pieces, b);+ let k = 0;+ const shownKana = shown.map(s => s === null ? null : spreadReadings([s], c[k++])[0]);+ return {+ words: words.map((w, i) => ({ ...w, spoken: spokenKana[i] || w.spoken, display: shown[i] === null ? undefined : shownKana[i] || w.display })),+ heard: heard.map((h, i) => ({ ...h, text: heardKana[i] || h.text })),+ };+ } catch {+ return { words, heard };+ }+}const NUMBER_WORDS = {zero: '0', one: '1', two: '2', three: '3', four: '4', five: '5', six: '6', seven: '7', eight: '8', nine: '9',@@ -124,8 +194,11 @@ export function similar(a, b) {// A word also counts as heard when Whisper writes its shown form, so// {SQL|sequel} or {async|a sink} is confirmed by "SQL" or "async".export function alignWords(words, heard, duration, language = 'en') {- const partsOf = word => expand(word, language);- const matches = (a, b) => language !== 'en' && (/\p{N}/u.test(a) || /\p{N}/u.test(b)) ? a === b : similar(a, b);+ const cjk = ['ja', 'zh'].includes(language);+ const chars = word => normWord(word).normalize('NFC');+ const partsOf = word => cjk ? Array.from(language === 'zh' ? simplified(chars(word)) : chars(word)) : expand(word, language);+ const same = (a, b) => a === b || (language === 'zh' && soundsAlike(a, b));+ const matches = (a, b) => cjk ? same(a, b) : language !== 'en' && (/\p{N}/u.test(a) || /\p{N}/u.test(b)) ? a === b : similar(a, b);const tokens = [];words.forEach((w, wi) => {const parts = String(w.spoken).split(/\s+/).flatMap(partsOf);@@ -168,6 +241,7 @@ export function alignWords(words, heard, duration, language = 'en') {const INF = 1e9;const cost = Array.from({ length: T + 1 }, () => new Float64Array(N + 1).fill(INF));const move = Array.from({ length: T + 1 }, () => new Int8Array(N + 1));+ const shownLength = cjk ? Array.from({ length: T + 1 }, () => new Int32Array(N + 1)) : null;cost[0][0] = 0;const groupEnd = new Int32Array(T);for (let i = 0; i < T; i++) {@@ -192,6 +266,15 @@ export function alignWords(words, heard, duration, language = 'en') {const ok = (groupEnd[i] > i && matches(joined, H[j].norm)) || (tokens[i].shown && matches(tokens[i].shown, H[j].norm));if (ok && c < cost[e + 1][j + 1]) { cost[e + 1][j + 1] = c; move[e + 1][j + 1] = 5; }}+ if (cjk && i < T && tokens[i].first && tokens[i].shown) {+ const length = Array.from(tokens[i].shown).length;+ const e = groupEnd[i];+ if (j + length <= N && same(tokens[i].shown, H.slice(j, j + length).map(h => h.norm).join('')) && c < cost[e + 1][j + length]) {+ cost[e + 1][j + length] = c;+ move[e + 1][j + length] = 7;+ shownLength[e + 1][j + length] = length;+ }+ }if (i < T && j + 1 < N && matches(tokens[i].norm, H[j].norm + H[j + 1].norm) && c < cost[i + 1][j + 2]) {cost[i + 1][j + 2] = c;move[i + 1][j + 2] = 6;@@ -224,6 +307,14 @@ export function alignWords(words, heard, duration, language = 'en') {matched += i - s;i = s;j--;+ } else if (m === 7) {+ const length = shownLength[i][j];+ let s = i - 1;+ while (s > 0 && tokens[s - 1].wi === tokens[i - 1].wi) s--;+ add(tokens[i - 1].wi, H[j - length].start, H[j - 1].end, i - s);+ matched += i - s;+ i = s;+ j -= length;} else if (m === 6) { add(tokens[i - 1].wi, H[j - 2].start, H[j - 1].end); matched++; i--; j -= 2; }else break;}diff --git a/src/cli.js b/src/cli.jsindex 271a5e1..a76bdba 100644--- a/src/cli.js+++ b/src/cli.js@@ -10,7 +10,7 @@ import { exportPlayer } from './export.js';import { check, stills, sheet, verify } from './qa.js';import { startServer, ROOT } from './server.js';import { VOICES, LANGUAGES, voiceLanguage, loadTTS, loadASR, cacheRoot, ttsDtype, TTS_SAMPLE_RATE, MULTILINGUAL_ASR_MODEL } from './models.js';-import { espeakVersion, phonemize, generateSpeech } from './phonemes.js';+import { espeakVersion, misakiAvailable, phonemize, generateSpeech } from './phonemes.js';import { writeWav } from './wav.js';import { ffmpegVersion } from './ffmpeg.js';import { findChrome } from './browser.js';@@ -25,7 +25,7 @@ const HELP = `explainroo ${VERSION}: narrated explainer videos from codeUsage: explainroo <command> [project] [options]Make a video- init <dir> create a project (--theme, --size, --pace, --language es, --voice, --title)+ init <dir> create a project (--theme, --size, --pace, --language en|es|fr|it|pt-br|hi|ja|zh, --voice, --title)voice [project] make the narration (Kokoro, ElevenLabs or a recording) and word timingspreview [project] live preview in your browser, reloads on save (--port, --size)still [project] [t…] PNG stills: "12.5", "scene", "[email protected]" (default: end of every scene; --size, --transparent)@@ -50,7 +50,7 @@ Referencethemes list looksformats list sizes for YouTube, Shorts, TikTok, Reels, Instagram, LinkedInicons <word…> search the 1,800+ built-in icons- doctor check ffmpeg, Chrome, espeak-ng and models (--fetch downloads English models)+ doctor check ffmpeg, Chrome, espeak-ng, misaki and models (--fetch downloads English models)Options: --size <size> overrides video.json for preview, still, sheet, check and render.Size outputs get a suffix, like video-tiktok.mp4. --out still wins.@@ -395,7 +395,10 @@ async function doctor(flags, log, print) {const ff = ffmpegVersion();add('ffmpeg', !!ff, ff || 'not found; install ffmpeg or set EXPLAINROO_FFMPEG');const espeak = espeakVersion();- add('espeak-ng', true, espeak || 'not found; only needed for other languages');+ add('espeak-ng', true, espeak || 'not found; only needed for Spanish, French, Italian, Portuguese and Hindi');+ for (const [language, name] of [['ja', 'Japanese'], ['zh', 'Chinese']]) {+ add(`misaki ${language}`, true, misakiAvailable(language) ? `Python with misaki for ${name}` : `not found; only needed for ${name}`);+ }try {add('chrome', true, findChrome());} catch (e) {@@ -404,13 +407,14 @@ async function doctor(flags, log, print) {const models = path.join(cacheRoot(), 'models', 'onnx-community');const has = (n) => fs.existsSync(path.join(models, n));if (flags.fetch) {- log('downloading speech models (about 400 MB the first time)');+ log('downloading English speech models (about 400 MB the first time)');+ log('other languages fetch their speech check model on first use, and Japanese and Chinese their fonts');await loadTTS({ log });await loadASR({ log });}add('voice model', has('Kokoro-82M-v1.0-ONNX'), has('Kokoro-82M-v1.0-ONNX') ? `Kokoro 82M (${ttsDtype()}) in ${models}` : 'not downloaded yet; "explainroo doctor --fetch" or the first voice run fetches it');add('speech check model', has('whisper-base.en_timestamped'), has('whisper-base.en_timestamped') ? 'Whisper base.en' : 'not downloaded yet; fetched on first use');- add('language model', has(MULTILINGUAL_ASR_MODEL.split('/')[1]), 'Whisper small for es, fr, it, pt-br and hi; downloaded on first use, not by --fetch');+ add('language model', has(MULTILINGUAL_ASR_MODEL.split('/')[1]), 'Whisper small for languages other than English; downloaded on first use, not by --fetch');const ok = rows.every((r) => r.ok || r.name.includes('model'));print(rows.map((r) => `${r.ok ? 'ok ' : 'FAIL'} ${r.name.padEnd(20)} ${r.detail}`).join('\n'), { ok, checks: rows });return ok ? 0 : 1;diff --git a/src/export.js b/src/export.jsindex 09edea1..ad56fa1 100644--- a/src/export.js+++ b/src/export.js@@ -2,6 +2,8 @@import fs from 'node:fs';import path from 'node:path';import { fileURLToPath } from 'node:url';+import { cacheRoot } from './models.js';+import { languageFonts, subsetFont } from './fonts.js';import { prepare, makeState } from './pipeline.js';import { startServer, projectFile, ROOT } from './server.js';import { launch, openEngine } from './browser.js';@@ -44,13 +46,22 @@ export function copyPlayer(project, timeline, used, out) {if (/\.svg$/i.test(rel) && /(?:href\s*=\s*['"]|url\(\s*['"]?)(?:\/|[a-z]+:)/i.test(fs.readFileSync(file, 'utf8'))) throw new Error(`image "${rel}" uses an outside file. Use a self-contained image.`);copy(file, `project/${rel}`);}- const fonts = JSON.parse(fs.readFileSync(path.join(ROOT, 'fonts/fonts.json'), 'utf8')).filter(f => used.fonts.includes(f.family));+ const text = (used.text || '') + timeline.scenes.flatMap(s => s.words.map(w => w.text)).join('');+ // Every font string names the Hindi fallback, but only Hindi text needs it.+ const fonts = JSON.parse(fs.readFileSync(path.join(ROOT, 'fonts/fonts.json'), 'utf8'))+ .filter(f => used.fonts.includes(f.family) && (f.family !== 'Noto Sans Devanagari' || /\p{Script=Devanagari}/u.test(text)));for (const f of fonts) copy(path.join(ROOT, 'fonts', f.file), `fonts/${f.file}`);for (const family of new Set(fonts.map(f => f.family))) {const name = family.replaceAll(' ', '') + '-OFL.txt';copy(path.join(ROOT, 'fonts/licenses', name), `fonts/licenses/${name}`);}fs.mkdirSync(path.join(out, 'fonts'), { recursive: true });+ // Japanese and Chinese fonts keep only the characters the video draws.+ for (const f of languageFonts(project.config.language)) {+ fs.writeFileSync(path.join(out, 'fonts', f.file), subsetFont(fs.readFileSync(f.path), text));+ fonts.push({ family: f.family, weight: f.weight, file: f.file });+ }+ if (languageFonts(project.config.language).length) copy(path.join(cacheRoot(), 'fonts', 'NotoSansCJK-OFL.txt'), 'fonts/licenses/NotoSansCJK-OFL.txt');fs.writeFileSync(path.join(out, 'fonts/fonts.json'), JSON.stringify(fonts));// Audio synthesis and upload code run during export, never during playback.const modules = ['runtime', 'stage', 'themes', 'pen', 'transitions', 'captions', 'util', 'layout', 'clips', 'ui', 'readability', 'pathdata', 'text', 'code'];@@ -91,7 +102,7 @@ export async function exportPlayer(project, { out, log = () => {} } = {}) {fs.mkdirSync(path.dirname(out), { recursive: true });const temp = fs.mkdtempSync(path.join(path.dirname(out), '.player-'));try {- server = await startServer({ projectDir: project.dir, getState: () => state });+ server = await startServer({ projectDir: project.dir, getState: () => state, log });browser = await launch();const context = await browser.newContext();let outside;@@ -124,6 +135,12 @@ export async function exportPlayer(project, { out, log = () => {} } = {}) {}const proto = CanvasRenderingContext2D.prototype;const font = Object.getOwnPropertyDescriptor(proto, 'font');+ const chars = new Set();+ const drawing = ['fillText', 'strokeText', 'measureText'].map(name => [name, proto[name]]);+ for (const [name, original] of drawing) proto[name] = function (text, ...rest) {+ for (const c of String(text)) chars.add(c);+ return original.call(this, text, ...rest);+ };Object.defineProperty(proto, 'font', { ...font, set(value) {for (const m of value.matchAll(/["']([^"']+)["']/g)) fonts.add(m[1]);font.set.call(this, value);@@ -143,8 +160,11 @@ export async function exportPlayer(project, { out, log = () => {} } = {}) {await upload('build/player-mix.wav', audio.encodeWav(sound.buffer));await upload('build/player-poster.jpg', await new Promise(resolve => engine.canvas.toBlob(resolve, 'image/jpeg', 0.9)));if ([...engine.mathCache.values()].some(m => m?.error)) throw new Error('a formula could not be typeset. Run check and fix it before exporting.');- return { assets: [...assets], fonts: [...fonts], math: Object.fromEntries(engine.mathCache) };- } finally { Object.defineProperty(proto, 'font', font); }+ return { assets: [...assets], fonts: [...fonts], text: [...chars].join(''), math: Object.fromEntries(engine.mathCache) };+ } finally {+ Object.defineProperty(proto, 'font', font);+ for (const [name, original] of drawing) proto[name] = original;+ }}, cover);if (outside) throw new Error('the project uses an outside URL. Use local files before exporting.');used.modules = [...projectModules];diff --git a/src/fonts.js b/src/fonts.jsnew file mode 100644index 0000000..800ebc3--- /dev/null+++ b/src/fonts.js@@ -0,0 +1,86 @@+import fs from 'node:fs';+import path from 'node:path';+import crypto from 'node:crypto';+import { createRequire } from 'node:module';+import { cacheRoot } from './models.js';++const SOURCE = 'https://raw.githubusercontent.com/googlefonts/noto-cjk/f8d157532fbfaeda587e826d4cd5b21a49186f7c/Sans';+const FONTS = {+ ja: { family: 'Noto Sans JP', dir: 'Japanese', name: 'NotoSansCJKjp', hashes: ['68a3fc98800b2a27b371f2fb79991daf3633bd89309d4ffaa6946fd587f375b5', 'e53dcb0dcb2922e45d01aae1ebd2f382bb81d4229b18b6b883bd170678af1f76'] },+ zh: { family: 'Noto Sans SC', dir: 'SimplifiedChinese', name: 'NotoSansCJKsc', hashes: ['2c76254f6fc379fddfce0a7e84fb5385bb135d3e399294f6eeb6680d0365b74b', 'b5f0d1a190a7f9b43c310a8850630af12553df32c4c050543f9059732d9b4c0a'] },+};++export function languageFonts(language) {+ if (!Object.hasOwn(FONTS, language)) return [];+ const f = FONTS[language];+ return [400, 700].map((weight, i) => ({+ family: f.family, weight, file: `${language}-${weight}.otf`,+ path: path.join(cacheRoot(), 'fonts', `${language}-${weight}.otf`),+ url: `${SOURCE}/OTF/${f.dir}/${f.name}-${i ? 'Bold' : 'Regular'}.otf`,+ hash: f.hashes[i],+ }));+}++function valid(data, hash) {+ return data.subarray(0, 4).toString() === 'OTTO' && crypto.createHash('sha256').update(data).digest('hex') === hash;+}++export async function ensureFonts(language, { log = () => {} } = {}) {+ const fonts = languageFonts(language);+ if (!fonts.length) return;+ for (const f of fonts) {+ if (fs.existsSync(f.path) && valid(fs.readFileSync(f.path), f.hash)) continue;+ if (process.env.EXPLAINROO_OFFLINE === '1') throw new Error(`${f.family} is missing or damaged. Run this command once without EXPLAINROO_OFFLINE to download the fonts.`);+ log(`downloading ${f.family} ${f.weight === 700 ? 'Bold' : 'Regular'} (about 17 MB, first use only)`);+ try {+ const response = await fetch(f.url);+ if (!response.ok) throw new Error();+ const data = Buffer.from(await response.arrayBuffer());+ if (!valid(data, f.hash)) throw new Error();+ fs.mkdirSync(path.dirname(f.path), { recursive: true });+ const temp = `${f.path}.${crypto.randomBytes(8).toString('hex')}.part`;+ try {+ fs.writeFileSync(temp, data);+ fs.renameSync(temp, f.path);+ } finally { fs.rmSync(temp, { force: true }); }+ } catch {+ throw new Error(`${f.family} could not be downloaded or checked. Check your connection and run the command again.`);+ }+ }+ const license = path.join(cacheRoot(), 'fonts', 'NotoSansCJK-OFL.txt');+ if (!fs.existsSync(license)) {+ if (process.env.EXPLAINROO_OFFLINE === '1') throw new Error('the Noto font license is missing. Run once without EXPLAINROO_OFFLINE to download it.');+ const response = await fetch(`${SOURCE}/LICENSE`);+ if (!response.ok) throw new Error('the Noto font license could not be downloaded. Check your connection and try again.');+ const data = Buffer.from(await response.arrayBuffer());+ if (crypto.createHash('sha256').update(data).digest('hex') !== '6a73f9541c2de74158c0e7cf6b0a58ef774f5a780bf191f2d7ec9cc53efe2bf2') throw new Error('the Noto font license could not be checked. Try again.');+ fs.writeFileSync(license, data);+ }+}++let hb;++// Keeps only the characters in text, so an exported player loads a few+// kilobytes of font instead of the full 16 MB file.+export function subsetFont(data, text) {+ hb ??= new WebAssembly.Instance(new WebAssembly.Module(fs.readFileSync(createRequire(import.meta.url).resolve('harfbuzzjs/dist/harfbuzz-subset.wasm')))).exports;+ const pointer = hb.malloc(data.byteLength);+ new Uint8Array(hb.memory.buffer).set(data, pointer);+ const blob = hb.hb_blob_create(pointer, data.byteLength, 2, 0, 0);+ const face = hb.hb_face_create(blob, 0);+ hb.hb_blob_destroy(blob);+ const input = hb.hb_subset_input_create_or_fail();+ const unicodes = hb.hb_subset_input_unicode_set(input);+ for (const c of new Set(text)) hb.hb_set_add(unicodes, c.codePointAt(0));+ const result = hb.hb_subset_or_fail(face, input);+ hb.hb_subset_input_destroy(input);+ hb.hb_face_destroy(face);+ hb.free(pointer);+ if (!result) throw new Error('a font could not be reduced to the characters of the video.');+ const out = hb.hb_face_reference_blob(result);+ const start = hb.hb_blob_get_data(out, 0);+ const bytes = Buffer.from(new Uint8Array(hb.memory.buffer).slice(start, start + hb.hb_blob_get_length(out)));+ hb.hb_blob_destroy(out);+ hb.hb_face_destroy(result);+ return bytes;+}diff --git a/src/g2p.py b/src/g2p.pynew file mode 100644index 0000000..dbf21b8--- /dev/null+++ b/src/g2p.py@@ -0,0 +1,46 @@+import contextlib+import json+import sys++sys.stdin.reconfigure(encoding='utf-8')+sys.stdout.reconfigure(encoding='utf-8')+sys.stderr.reconfigure(encoding='utf-8')++mode = sys.argv[1]+++def readings(tagger, text):+ # [start, end, katakana reading] for each word, in characters.+ out, pos = [], 0+ for word in tagger(text):+ start = text.find(word.surface, pos)+ if start < 0:+ continue+ pos = start + len(word.surface)+ pron = getattr(word.feature, 'pron', None)+ out.append([start, pos, pron if pron and pron != '*' else word.surface])+ return out+++try:+ with contextlib.redirect_stdout(sys.stderr):+ # Mandarin 1.1 uses a different phoneme alphabet from Kokoro v1.0.+ if mode == 'ja':+ from misaki.ja import JAG2P+ g2p = JAG2P()+ run = lambda text: g2p(text)[0]+ elif mode == 'zh':+ from misaki.zh import ZHG2P+ g2p = ZHG2P(version=None)+ run = lambda text: g2p(text)[0]+ else:+ # The speech check compares Japanese by sound, with the UniDic+ # dictionary misaki uses.+ import fugashi+ tagger = fugashi.Tagger()+ run = lambda text: readings(tagger, text)+ for line in sys.stdin:+ print(json.dumps(run(json.loads(line)), ensure_ascii=False), file=sys.__stdout__, flush=True)+except Exception as error:+ print(f'{type(error).__name__}: {error}'.splitlines()[0], file=sys.stderr)+ sys.exit(1)diff --git a/src/models.js b/src/models.jsindex edd1d38..e0df955 100644--- a/src/models.js+++ b/src/models.js@@ -10,11 +10,11 @@ export const MULTILINGUAL_ASR_MODEL = 'onnx-community/whisper-small_timestamped'export const TTS_SAMPLE_RATE = 24000;export const LANGUAGES = {- en: 'af_heart', es: 'ef_dora', fr: 'ff_siwis', it: 'if_sara', 'pt-br': 'pf_dora', hi: 'hf_alpha',+ en: 'af_heart', es: 'ef_dora', fr: 'ff_siwis', it: 'if_sara', 'pt-br': 'pf_dora', hi: 'hf_alpha', ja: 'jf_alpha', zh: 'zf_xiaobei',};export function voiceLanguage(id) {- return { a: 'en', b: 'en', e: 'es', f: 'fr', i: 'it', p: 'pt-br', h: 'hi' }[id[0]];+ return { a: 'en', b: 'en', e: 'es', f: 'fr', i: 'it', p: 'pt-br', h: 'hi', j: 'ja', z: 'zh' }[id[0]];}export function asrModel(language = 'en') {@@ -145,4 +145,17 @@ export const VOICES = {hf_beta: 'Hindi, female',hm_omega: 'Hindi, male',hm_psi: 'Hindi, male',+ jf_alpha: 'Japanese, female (default)',+ jf_gongitsune: 'Japanese, female',+ jf_nezumi: 'Japanese, female',+ jf_tebukuro: 'Japanese, female',+ jm_kumo: 'Japanese, male',+ zf_xiaobei: 'Mandarin Chinese, female (default)',+ zf_xiaoni: 'Mandarin Chinese, female',+ zf_xiaoxiao: 'Mandarin Chinese, female',+ zf_xiaoyi: 'Mandarin Chinese, female',+ zm_yunjian: 'Mandarin Chinese, male',+ zm_yunxi: 'Mandarin Chinese, male',+ zm_yunxia: 'Mandarin Chinese, male',+ zm_yunyang: 'Mandarin Chinese, male',};diff --git a/src/phonemes.js b/src/phonemes.jsindex 1526070..d4e0a3b 100644--- a/src/phonemes.js+++ b/src/phonemes.js@@ -1,4 +1,5 @@import { spawn, spawnSync } from 'node:child_process';+import { fileURLToPath } from 'node:url';const ESPEAK_LANGUAGES = { es: 'es', fr: 'fr-fr', it: 'it', 'pt-br': 'pt-br', hi: 'hi' };// Adapted from misaki's EspeakG2P (Apache-2.0, see licenses/).@@ -27,6 +28,7 @@ export function espeakVersion() {}export async function phonemize(texts, language) {+ if (['ja', 'zh'].includes(language)) return misakiPhonemes(texts, language);const lang = ESPEAK_LANGUAGES[language];if (!lang) throw new Error('espeak-ng supports es, fr, it, pt-br and hi here.');const input = [];@@ -66,3 +68,88 @@ export function generateSpeech(tts, text, { language = 'en', voice, speed, phoneconst { input_ids } = tts.tokenizer(phonemes, { truncation: true });return tts.generate_from_ids(input_ids, { voice, speed });}++function python() {+ return process.env.EXPLAINROO_PYTHON || (process.platform === 'win32' ? 'python' : 'python3');+}++function misakiInstall(language) {+ return `Python with misaki is needed for ${language === 'ja' ? 'Japanese' : 'Chinese'}. Run: ${python()} -m pip install "misaki[${language}]".${language === 'ja' ? ` Then run: ${python()} -m unidic download.` : ''} Or set EXPLAINROO_PYTHON to a Python with these packages.`;+}++export function misakiAvailable(language) {+ const r = spawnSync(python(), [fileURLToPath(new URL('./g2p.py', import.meta.url)), language], { input: JSON.stringify(language === 'ja' ? '日本語' : '中文') + '\n', encoding: 'utf8', timeout: 15000 });+ if (r.error || r.status !== 0) return false;+ try {+ const phonemes = JSON.parse(r.stdout);+ return typeof phonemes === 'string' && !!phonemes.trim();+ } catch { return false; }+}++// misaki leaves Latin words like "API" or "JavaScript" as letters, which the+// Japanese and Chinese voices garble. They are read as English instead, with+// the espeak-ng build and fixes kokoro-js uses for English.+const LATIN = /[A-Za-z]+(?:['’][A-Za-z]+)*(?:[\s-]+[A-Za-z]+(?:['’][A-Za-z]+)*)*/g;++async function englishPhonemes(text) {+ const { phonemize: espeakWasm } = await import('phonemizer');+ return (await espeakWasm(text, 'en-us')).join(' ').replace(/ʲ/g, 'j').replace(/r/g, 'ɹ').replace(/x/g, 'k').replace(/ɬ/g, 'l').trim();+}++async function misakiPhonemes(texts, language) {+ const parts = texts.map(text => {+ const list = [];+ let last = 0;+ for (const m of text.matchAll(LATIN)) {+ if (m.index > last) list.push({ text: text.slice(last, m.index) });+ list.push({ text: m[0], latin: true });+ last = m.index + m[0].length;+ }+ if (last < text.length) list.push({ text: text.slice(last) });+ return list;+ });+ const native = parts.flat().filter(p => !p.latin && p.text.trim());+ const phonemes = await runMisaki(native.map(p => p.text), language, line => typeof line === 'string');+ native.forEach((p, i) => { p.phonemes = phonemes[i]; });+ for (const p of parts.flat()) if (p.latin) p.phonemes = await englishPhonemes(p.text);+ return parts.map(list => {+ const joined = list.map(p => p.phonemes?.trim()).filter(Boolean).join(' ');+ if (!joined) throw new Error('misaki returned incomplete phonemes. Check its installation and EXPLAINROO_PYTHON.');+ return joined;+ });+}++// Katakana readings of Japanese text, from the dictionary misaki installs:+// [start, end, reading] for each word, counted in characters.+export function japaneseReadings(texts) {+ return runMisaki(texts, 'ja-readings', line => Array.isArray(line) && line.every(t => Array.isArray(t) && t.length === 3));+}++async function runMisaki(texts, mode, valid) {+ const language = mode.slice(0, 2);+ if (!texts.length) return [];+ return new Promise((resolve, reject) => {+ const child = spawn(python(), [fileURLToPath(new URL('./g2p.py', import.meta.url)), mode], { stdio: ['pipe', 'pipe', 'pipe'] });+ let stdout = '';+ let stderr = '';+ child.stdout.setEncoding('utf8').on('data', data => { stdout += data; });+ child.stderr.setEncoding('utf8').on('data', data => { stderr += data; });+ child.stdin.on('error', () => {});+ child.on('error', () => reject(new Error(misakiInstall(language))));+ child.on('close', code => {+ if (code !== 0) {+ const said = stderr.trim().split(/\r?\n/).at(-1);+ return reject(new Error(misakiInstall(language) + (said ? ` Python said: ${said}` : '')));+ }+ try {+ // misaki drops punctuation in Japanese, so a piece may come back empty.+ const lines = stdout.trimEnd().split(/\r?\n/).map(line => JSON.parse(line));+ if (lines.length !== texts.length || !lines.every(valid)) throw new Error();+ resolve(lines);+ } catch {+ reject(new Error('misaki returned incomplete phonemes. Check its installation and EXPLAINROO_PYTHON.'));+ }+ });+ child.stdin.end(texts.map(text => JSON.stringify(text)).join('\n') + '\n');+ });+}diff --git a/src/pipeline.js b/src/pipeline.jsindex 5eb869b..a22218b 100644--- a/src/pipeline.js+++ b/src/pipeline.js@@ -3,6 +3,7 @@import fs from 'node:fs';import path from 'node:path';import crypto from 'node:crypto';+import { ensureFonts } from './fonts.js';import { synthesize } from './voice.js';import { buildTimeline } from './timeline.js';import { listAssets, loadProject } from './project.js';@@ -10,6 +11,7 @@ import { startServer } from './server.js';import { launch, openEngine } from './browser.js';export async function prepare(project, { log = () => {}, force = false } = {}) {+ await ensureFonts(project.config.language, { log });const voices = await synthesize(project, { log, force });const timeline = buildTimeline(project, voices);fs.mkdirSync(project.paths.build, { recursive: true });@@ -41,7 +43,7 @@ export function makeState(project, timeline) {// Runs fn(page, ctx) with the engine loaded at `scale`, then cleans up.export async function withEngine(project, timeline, { scale = 1, pages = 1, transparent = false, log = () => {} } = {}, fn) {const state = makeState(project, timeline);- const server = await startServer({ projectDir: project.dir, getState: () => state });+ const server = await startServer({ projectDir: project.dir, getState: () => state, log });let browser = null;try {browser = await launch();diff --git a/src/project.js b/src/project.jsindex 6bd74f8..4595ec3 100644--- a/src/project.js+++ b/src/project.js@@ -297,7 +297,7 @@ export function loadProject(arg, { size } = {}) {}const scriptPath = path.join(dir, 'script.md');if (!fs.existsSync(scriptPath)) fail(`${scriptPath} is missing`);- const script = parseScript(fs.readFileSync(scriptPath, 'utf8'));+ const script = parseScript(fs.readFileSync(scriptPath, 'utf8'), { language: config.language });if (!config.title) config.title = script.title || path.basename(dir);const scenesPath = path.join(dir, 'scenes.js');if (!fs.existsSync(scenesPath)) fail(`${scenesPath} is missing`);diff --git a/src/qa.js b/src/qa.jsindex 557955e..c46d420 100644--- a/src/qa.js+++ b/src/qa.js@@ -6,7 +6,7 @@ import { prepare, withEngine } from './pipeline.js';import { probe, loudness, ffmpeg } from './ffmpeg.js';import { transcribeWords } from './models.js';import { readWav } from './wav.js';-import { alignWords } from './align.js';+import { alignWords, readAlike } from './align.js';import { buildChapters, chapterHints } from './chapters.js';import { outputPath } from './project.js';@@ -151,7 +151,8 @@ export async function verify(project, { file = null, log = () => {} } = {}) {const to = Math.min(samples.length, Math.ceil((t.start + t.dur) * 16000));const heard = await transcribeWords(samples.subarray(from, to), { log, language: project.config.language });texts.push(heard.text);- const rate = alignWords(words, heard.words, t.dur, project.config.language).matchRate;+ const alike = await readAlike(words, heard.words, project.config.language);+ const rate = alignWords(alike.words, alike.heard, t.dur, project.config.language).matchRate;scenes.push({ id: t.id, matchRate: Math.round(rate * 1000) / 1000 });total += words.length;matched += rate * words.length;@@ -162,7 +163,8 @@ export async function verify(project, { file = null, log = () => {} } = {}) {} else {const heard = await transcribeWords(samples, { log, language: project.config.language });const script = project.script.scenes.flatMap(spokenWords);- speech = { matchRate: Math.round(alignWords(script, heard.words, duration, project.config.language).matchRate * 1000) / 1000, transcript: heard.text };+ const alike = await readAlike(script, heard.words, project.config.language);+ speech = { matchRate: Math.round(alignWords(alike.words, alike.heard, duration, project.config.language).matchRate * 1000) / 1000, transcript: heard.text };}if (speech.matchRate < 0.85) issues.push({ level: 'warn', message: `only ${Math.round(speech.matchRate * 100)}% of the script was understood in the final mix; lower music.volume or check pronunciation` });fs.rmSync(wav, { force: true });diff --git a/src/script.js b/src/script.jsindex 53b35ed..2f7507e 100644--- a/src/script.js+++ b/src/script.js@@ -54,14 +54,16 @@ function parseAttrs(src, line) {// Splits one scene's narration into units. A unit is a word (or an override// group) with separate display and spoken text, a marker, or a pause.-export function tokenizeNarration(text, line = 0) {+export function tokenizeNarration(text, line = 0, language = 'en') {const units = [];+ const segmenter = ['ja', 'zh'].includes(language) ? new Intl.Segmenter(language, { granularity: 'word' }) : null;+ const punctuation = segmenter ? /^[.,;:!?।。!?、,;:)」』)\]"'’”…]+$/ : /^[.,;:!?।)\]"'’…]+$/;let i = 0;let pendingSpace = false;const n = text.length;const pushWord = (display, spoken) => {const prev = units[units.length - 1];- if (!pendingSpace && prev && prev.type === 'word' && /^[.,;:!?।)\]"'’…]+$/.test(display)) {+ if (!pendingSpace && prev && prev.type === 'word' && punctuation.test(display)) {prev.display += display;prev.spoken += spoken;return;@@ -98,7 +100,7 @@ export function tokenizeNarration(text, line = 0) {throw new ScriptError('pronunciation overrides look like {shown|spoken}, for example {SQL|sequel}', line);}let j = end + 1;- while (j < n && !/\s/.test(text[j]) && text[j] !== '[' && text[j] !== '{') j++;+ while (j < n && !/\s/.test(text[j]) && text[j] !== '[' && text[j] !== '{' && (!segmenter || punctuation.test(text[j]))) j++;const trail = text.slice(end + 1, j);pushWord(inner.slice(0, bar).trim() + trail, inner.slice(bar + 1).trim() + trail);i = j;@@ -107,13 +109,21 @@ export function tokenizeNarration(text, line = 0) {let j = i;while (j < n && !/\s/.test(text[j]) && !text.startsWith('[#', j) && !text.startsWith('[pause', j) && text[j] !== '{') j++;const word = text.slice(i, j);- pushWord(word, word);+ if (segmenter) {+ let prefix = '';+ for (const part of segmenter.segment(word)) {+ if (!part.isWordLike && (/[「『(“‘(]/u.test(part.segment) || !units.some(u => u.type === 'word'))) { prefix += part.segment; continue; }+ pushWord(prefix + part.segment, prefix + part.segment);+ prefix = '';+ }+ if (prefix) pushWord(prefix, prefix);+ } else pushWord(word, word);i = j;}return units;}-export function parseScript(source) {+export function parseScript(source, { language = 'en' } = {}) {const lines = String(source).replace(/\r\n?/g, '\n').split('\n');let title = null;const scenes = [];@@ -183,7 +193,7 @@ export function parseScript(source) {const units = [];s._paras.forEach((p, pi) => {if (pi > 0) units.push({ type: 'pause', sec: null, paragraph: true });- units.push(...tokenizeNarration(p.text, p.line));+ units.push(...tokenizeNarration(p.text, p.line, language));});const words = units.filter((u) => u.type === 'word');return {@@ -202,7 +212,9 @@ export function parseScript(source) {// the text between explicit pauses) and is sent to the TTS model on its own,// which keeps every request well under the model's context limit and gives// exact control over the silence between sentences.-export function speechChunks(units, { sentenceGap = 0.3, paragraphGap = 0.55, pace = 1 } = {}) {+export function speechChunks(units, { sentenceGap = 0.3, paragraphGap = 0.55, pace = 1, language = 'en' } = {}) {+ const cjk = ['ja', 'zh'].includes(language);+ const sentenceEnd = cjk ? /[.!?…।。!?]["'’)\]」』)]*$/ : /[.!?…।]["'’)\]]*$/;const chunks = [];let cur = null;let gapBefore = 0;@@ -221,9 +233,9 @@ export function speechChunks(units, { sentenceGap = 0.3, paragraphGap = 0.55, pacur = { gapBefore, words: [], text: '' };gapBefore = 0;}- cur.text += (cur.words.length ? ' ' : '') + u.spoken;+ cur.text += (cur.words.length && !cjk ? ' ' : '') + u.spoken;cur.words.push({ unitIndex: index, display: u.display, spoken: u.spoken });- if (/[.!?…।]["'’)\]]*$/.test(u.spoken)) {+ if (sentenceEnd.test(u.spoken)) {close();gapBefore = sentenceGap / pace;}diff --git a/src/server.js b/src/server.jsindex 413d554..a588a29 100644--- a/src/server.js+++ b/src/server.js@@ -5,6 +5,7 @@ import fs from 'node:fs';import path from 'node:path';import crypto from 'node:crypto';import { fileURLToPath } from 'node:url';+import { ensureFonts, languageFonts } from './fonts.js';import { extractClip, VIDEO_FILE } from './clips.js';import { typesetMath, mathHash, MAX_TEX_LENGTH } from './math.js';@@ -149,7 +150,10 @@ function isSymlink(p) {}}-export function startServer({ projectDir, getState, port = 0 }) {+export async function startServer({ projectDir, getState, port = 0, log = () => {} }) {+ const { config } = await getState();+ await ensureFonts(config?.language, { log });+ const fonts = new Map(['ja', 'zh'].flatMap(languageFonts).map(f => [`/language-fonts/${f.file}`, f.path]));const roughPath = resolveModule('roughjs/bundled/rough.esm.js');const clips = new Map();const server = http.createServer(async (req, res) => {@@ -255,6 +259,7 @@ export function startServer({ projectDir, getState, port = 0 }) {let file = null;if (p === '/' || p === '/studio.html') file = path.join(ROOT, 'engine', 'studio.html');else if (p === '/frame.html') file = path.join(ROOT, 'engine', 'frame.html');+ else if (fonts.has(p)) file = fonts.get(p);else if (p === '/vendor/rough.js') file = roughPath;else if (p.startsWith('/engine/')) file = safeJoin(path.join(ROOT, 'engine'), p.slice(8));else if (p.startsWith('/fonts/')) file = safeJoin(path.join(ROOT, 'fonts'), p.slice(7));diff --git a/src/subtitles.js b/src/subtitles.jsindex 3a3ee72..e63fd57 100644--- a/src/subtitles.js+++ b/src/subtitles.js@@ -2,16 +2,19 @@const sentenceEnd = /[.!?…।]["'’”)\]]*$/;const punctuation = /[,;:.!?…।]["'’”)\]]*$/;-function lines(words) {- const text = words.map((w) => w.text).join(' ');- if (text.length <= 42) return text;+function lines(words, language = 'en') {+ const cjk = ['ja', 'zh'].includes(language);+ const limit = cjk ? 18 : 42;+ const separator = cjk ? '' : ' ';+ const text = words.map((w) => w.text).join(separator);+ if (text.length <= limit) return text;let best = null;for (let i = 1; i < words.length; i++) {- const a = words.slice(0, i).map((w) => w.text).join(' ');- const b = words.slice(i).map((w) => w.text).join(' ');- if (a.length > 42 || b.length > 42) continue;+ const a = words.slice(0, i).map((w) => w.text).join(separator);+ const b = words.slice(i).map((w) => w.text).join(separator);+ if (a.length > limit || b.length > limit) continue;// Break after punctuation, unless that leaves a word or two alone on a line.- const atPunctuation = punctuation.test(words[i - 1].text) && Math.min(a.length, b.length) >= 12;+ const atPunctuation = (cjk ? /[,;:.!?…।。!?、,;:]["'’”)\]」』)]*$/ : punctuation).test(words[i - 1].text) && Math.min(a.length, b.length) >= (cjk ? 5 : 12);const score = (atPunctuation ? 30 : 0) - Math.abs(a.length - b.length);if (!best || score > best.score) best = { score, text: `${a}\n${b}` };}@@ -19,18 +22,23 @@ function lines(words) {}export function subtitleCues(timeline, { from = 0, to = Infinity } = {}) {+ const language = timeline.language || 'en';+ const cjk = ['ja', 'zh'].includes(language);+ const limit = cjk ? 18 : 42;+ const endSentence = cjk ? /[.!?…।。!?]["'’”)\]」』)]*$/ : sentenceEnd;+ const atPunctuation = cjk ? /[,;:.!?…।。!?、,;:]["'’”)\]」』)]*$/ : punctuation;const cues = [];for (const scene of timeline.scenes) {const words = scene.words || [];let i = 0;while (i < words.length) {- if (words[i].text.length > 42) throw new Error(`subtitle word "${words[i].text}" is longer than 42 characters. Shorten its shown text in script.md.`);+ if (words[i].text.length > limit) throw new Error(`subtitle word "${words[i].text}" is longer than ${limit} characters. Shorten its shown text in script.md.`);let j = i + 1;while (j < words.length) {- if (sentenceEnd.test(words[j - 1].text) || words[j].start - words[j - 1].end > 0.6 + 1e-9) break;- if (words[j].end + 0.25 - words[i].start > 6 || lines(words.slice(i, j + 1)) === null) {+ if (endSentence.test(words[j - 1].text) || words[j].start - words[j - 1].end > 0.6 + 1e-9) break;+ if (words[j].end + 0.25 - words[i].start > 6 || lines(words.slice(i, j + 1), language) === null) {for (let k = j - 1; k > i; k--) {- if (punctuation.test(words[k - 1].text)) { j = k; break; }+ if (atPunctuation.test(words[k - 1].text)) { j = k; break; }}break;}@@ -38,7 +46,7 @@ export function subtitleCues(timeline, { from = 0, to = Infinity } = {}) {}const start = scene.start + words[i].start;const end = Math.min(scene.start + words[j - 1].end + 0.25, start + 6, scene.start + (scene.dur ?? Infinity));- cues.push({ start, end, text: lines(words.slice(i, j)) });+ cues.push({ start, end, text: lines(words.slice(i, j), language) });i = j;}}diff --git a/src/timeline.js b/src/timeline.jsindex a192fd5..02905d3 100644--- a/src/timeline.js+++ b/src/timeline.js@@ -42,6 +42,7 @@ export function buildTimeline(project, voices) {});return {fps,+ ...(['ja', 'zh'].includes(config.language) ? { language: config.language } : {}),width: config.width,height: config.height,duration: snap(t),diff --git a/src/voice.js b/src/voice.jsindex f50c017..8d0f77c 100644--- a/src/voice.js+++ b/src/voice.js@@ -4,8 +4,8 @@import fs from 'node:fs';import path from 'node:path';import crypto from 'node:crypto';-import { speechChunks, resolveMarks } from './script.js';-import { alignWords } from './align.js';+import { speechChunks, resolveMarks, normWord } from './script.js';+import { alignWords, readAlike } from './align.js';import { loadTTS, transcribeWords, TTS_MODEL, TTS_SAMPLE_RATE, ttsDtype, asrModel } from './models.js';import { phonemize, generateSpeech } from './phonemes.js';import { writeWav, readWav, resample, trimSilence } from './wav.js';@@ -59,7 +59,7 @@ export async function synthesize(project, { force = false, log = () => {}, onlyconst result = {};const todo = [];for (const scene of project.script.scenes) {- const chunks = speechChunks(scene.units, { sentenceGap: config.sentenceGap, paragraphGap: config.paragraphGap, pace: config.pace });+ const chunks = speechChunks(scene.units, { sentenceGap: config.sentenceGap, paragraphGap: config.paragraphGap, pace: config.pace, language: config.language });const recording = recordings?.files[scene.id];const hash = sceneHash(scene, chunks, config, recording ? {content: recording.hash,@@ -144,13 +144,14 @@ async function synthesizeScene(project, scene, chunks, hash, tts, log) {const heard = await transcribeWords(resample(samples, sr, 16000), { log, language: config.language });transcripts.push(heard.text);- const aligned = alignWords(chunk.words, heard.words, dur, config.language);+ const alike = await readAlike(chunk.words, heard.words, config.language);+ const aligned = alignWords(alike.words, alike.heard, dur, config.language);chunk.words.forEach((w, k) => {const a = aligned.words[k];wordTimes[w.unitIndex] = { start: start + a.start, end: start + a.end, matched: a.matched };if (!a.matched) unmatched.push(w.display);});- const tokens = chunk.words.reduce((n, w) => n + w.spoken.split(/\s+/).length, 0);+ const tokens = chunk.words.reduce((n, w) => n + (['ja', 'zh'].includes(config.language) ? Array.from(normWord(w.spoken).normalize('NFC')).length : w.spoken.split(/\s+/).length), 0);totalTokens += tokens;matchedTokens += aligned.matchRate * tokens;chunkInfo.push({ start, end: start + dur, text: chunk.text });@@ -209,7 +210,8 @@ async function recordingScene(project, scene, hash, source, take, cut, heardTakewords: heardTake.words.filter((w) => w.start >= cut.start && w.start < cut.end).map(local),} : await transcribeWords(resample(samples, sr, 16000), { log, language: config.language });const units = scene.units.filter((u) => u.type === 'word');- const aligned = alignWords(units, heard.words, duration, config.language);+ const alike = await readAlike(units, heard.words, config.language);+ const aligned = alignWords(alike.words, alike.heard, duration, config.language);const times = cut ? cut.words.map(local) : aligned.words;const wordTimes = [];let wi = 0;diff --git a/test/cjk-browser.test.js b/test/cjk-browser.test.jsnew file mode 100644index 0000000..074ab11--- /dev/null+++ b/test/cjk-browser.test.js@@ -0,0 +1,101 @@+import test from 'node:test';+import assert from 'node:assert/strict';+import fs from 'node:fs';+import os from 'node:os';+import path from 'node:path';+import { copyPlayer } from '../src/export.js';+import { cacheRoot } from '../src/models.js';+import { languageFonts } from '../src/fonts.js';+import { findChrome } from '../src/browser.js';+import { loadProject, THEMES } from '../src/project.js';+import { buildTimeline } from '../src/timeline.js';+import { withEngine } from '../src/pipeline.js';++let chrome;+try { chrome = findChrome(); } catch {}++for (const language of ['ja', 'zh']) {+ const cached = languageFonts(language).every(f => fs.existsSync(f.path));+ test(`${language} uses cached Noto glyphs in every look and UI, with word and phrase cues`, { skip: !chrome || !cached }, async t => {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-cjk-browser-'));+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));+ fs.writeFileSync(path.join(dir, 'video.json'), JSON.stringify({ language, captions: true, sfx: false }));+ fs.writeFileSync(path.join(dir, 'script.md'), '## demo\n解析住所');+ fs.writeFileSync(path.join(dir, 'scenes.js'), `export default { demo(s) { s.text('解析', { at: '解析' }); } };`);+ const project = loadProject(dir);+ const timeline = buildTimeline(project, { demo: { duration: 2, words: [{ text: '解析', start: 0, end: 1 }, { text: '住所', start: 1, end: 2 }], marks: {} } });+ const out = path.join(dir, 'player');+ const sentence = language === 'ja' ? '日本語の解析が始まります。' : '中文解析开始了。';+ copyPlayer(project, timeline, { assets: [], fonts: [], math: {}, text: sentence }, out);+ const manifest = JSON.parse(fs.readFileSync(path.join(out, 'fonts/fonts.json')));+ assert.deepEqual(manifest.map(f => f.weight), [400, 700]);+ const subsets = languageFonts(language).map(f => fs.readFileSync(path.join(out, 'fonts', f.file)));+ for (const data of subsets) {+ assert.equal(data.subarray(0, 4).toString(), 'OTTO');+ assert.ok(data.length < 100_000, `${data.length} bytes`);+ }+ assert.deepEqual(fs.readFileSync(path.join(out, 'fonts/licenses/NotoSansCJK-OFL.txt')), fs.readFileSync(path.join(cacheRoot(), 'fonts', 'NotoSansCJK-OFL.txt')));+ await withEngine(project, timeline, {}, async (page, { server }) => {+ const result = await page.evaluate(async ([language, themes, sentence, subsets]) => {+ const { getTheme, fontString } = await import('/engine/themes.js');+ const { UI } = await import('/engine/ui.js');+ const { Stage } = await import('/engine/stage.js');+ let i = 0;+ // The exported player's reduced fonts still draw every character of the video.+ for (const [k, data] of subsets.entries()) {+ const face = new FontFace('Exported', Uint8Array.from(atob(data), c => c.charCodeAt(0)), { weight: String(400 + k * 300) });+ document.fonts.add(await face.load());+ for (const text of [sentence, '解析住所']) {+ const span = document.createElement('div');+ span.id = 'glyph' + i++;+ span.style.font = `${400 + k * 300} 48px Exported, serif`;+ span.textContent = text;+ document.body.append(span);+ }+ }+ for (const theme of themes) {+ const fonts = ['display', 'body', 'hand', 'mono'].map(role => fontString(getTheme(theme), role, 48));+ fonts.push(UI.prototype.fontString.call({ font: 'Inter' }, 48));+ for (const font of fonts) {+ const span = document.createElement('div');+ span.id = 'glyph' + i++;+ span.style.font = font;+ span.textContent = sentence;+ document.body.append(span);+ }+ }+ await new Promise(resolve => requestAnimationFrame(resolve));+ const s = Object.create(Stage.prototype);+ s.engine = { config: { language } };+ s.words = [{ text: '解析', start: 0.35, end: 1.35 }, { text: '住所', start: 1.35, end: 2.35 }];+ assertCue(s.cue('解析'), 0.35);+ assertCue(s.cueEnd('解析住所'), 2.35);+ s.words.push({ text: '名前の解析', start: 2.5, end: 3 });+ assertCue(s.cue('名前の解析'), 2.5);+ let rejected = false;+ try { s.cue('名前'); } catch { rejected = true; }+ if (!rejected) throw new Error('cue broke inside a word');+ function assertCue(actual, expected) { if (actual !== expected) throw new Error(`cue ${actual} != ${expected}`); }+ return { count: i, issues: window.explainroo.check() };+ }, [language, THEMES, sentence, subsets.map(data => data.toString('base64'))]);+ assert.deepEqual(result.issues.filter(i => i.level === 'error'), []);+ const cdp = await page.context().newCDPSession(page);+ await cdp.send('DOM.enable');+ await cdp.send('CSS.enable');+ const { root } = await cdp.send('DOM.getDocument');+ for (let i = 0; i < result.count; i++) {+ const { nodeId } = await cdp.send('DOM.querySelector', { nodeId: root.nodeId, selector: '#glyph' + i });+ const { fonts } = await cdp.send('CSS.getPlatformFontsForNode', { nodeId });+ assert.ok(fonts.every(f => f.isCustomFont), JSON.stringify(fonts));+ assert.ok(fonts.some(f => f.familyName === (language === 'ja' ? 'Noto Sans CJK JP' : 'Noto Sans CJK SC')), JSON.stringify(fonts));+ }+ assert.equal((await fetch(server.url + '/language-fonts/' + language + '-400.otf')).status, 200);+ const other = language === 'ja' ? 'zh' : 'ja';+ if (languageFonts(other).every(f => fs.existsSync(f.path))) {+ project.config.language = other;+ assert.equal((await fetch(server.url + '/language-fonts/' + other + '-400.otf')).status, 200);+ }+ for (const file of ['NotoSansCJK-OFL.txt', 'ja-400.otf.part', '%2e%2e%2fmodels%2fconfig.json']) assert.equal((await fetch(server.url + '/language-fonts/' + file)).status, 404);+ });+ });+}diff --git a/test/cjk.test.js b/test/cjk.test.jsnew file mode 100644index 0000000..33ed7ca--- /dev/null+++ b/test/cjk.test.js@@ -0,0 +1,210 @@+import test from 'node:test';+import assert from 'node:assert/strict';+import fs from 'node:fs';+import os from 'node:os';+import path from 'node:path';+import { tokenizeNarration, parseScript, speechChunks, normWord, resolveMarks } from '../src/script.js';+import { alignWords, readAlike, spreadReadings } from '../src/align.js';+import { subtitleCues, toSrt, toVtt } from '../src/subtitles.js';+import { buildPhrases, drawCaptions } from '../engine/captions.js';+import { getTheme } from '../engine/themes.js';+import { phonemize, misakiAvailable } from '../src/phonemes.js';+import { languageFonts, ensureFonts } from '../src/fonts.js';+import { cacheRoot } from '../src/models.js';+import { normalizeConfig } from '../src/project.js';+import { buildTimeline } from '../src/timeline.js';++function env(t, key, value) {+ const previous = process.env[key];+ process.env[key] = value;+ t.after(() => { if (previous === undefined) delete process.env[key]; else process.env[key] = previous; });+}++const native = { ja: '住所を解析します。次の画面を表示します!終わりです?', zh: '浏览器解析地址。下一步显示网页!完成了吗?' };++for (const [language, text] of Object.entries(native)) {+ test(`${language} segments words, keeps punctuation and preserves markers and overrides`, () => {+ const units = tokenizeNarration('[#first]' + text + '[pause 0.4][#last]{住所|じゅうしょ}です。', 0, language);+ const words = units.filter(u => u.type === 'word');+ assert.equal(words.map(w => w.display).join(''), text + '住所です。');+ assert.ok(words.some(w => w.display === '解析'));+ assert.deepEqual(words.find(w => w.display === '住所' && w.spoken !== w.display), { type: 'word', display: '住所', spoken: 'じゅうしょ', space: true });+ assert.equal(normWord('解析。'), '解析');+ assert.deepEqual(tokenizeNarration('次に「解析」を行う。', 0, language).filter(u => u.type === 'word').map(u => u.display).join(''), '次に「解析」を行う。');+ assert.ok(tokenizeNarration('次に「解析」を行う。', 0, language).filter(u => u.type === 'word').every(u => normWord(u.display)));+ assert.deepEqual(tokenizeNarration('{住所|じゅうしょ}。次。', 0, language).filter(u => u.type === 'word').map(u => u.display), ['住所。', '次。']);+ assert.deepEqual(speechChunks(tokenizeNarration('{住所|じゅうしょ}。「解析」。', 0, language), { language }).map(c => c.text), ['じゅうしょ。', '「解析」。']);+ assert.equal(normWord('が'), normWord('が'.normalize('NFD')));+ const times = units.map((u, i) => u.type === 'word' ? { start: i, end: i + 0.5 } : null);+ assert.equal(resolveMarks(units, times).first, 1);+ assert.equal(resolveMarks(units, times).last, units.findIndex(u => u.spoken === 'じゅうしょ'));+ const chunks = speechChunks(units, { language });+ assert.deepEqual(chunks.map(c => c.text), [...text.match(/[^。!?]+[。!?]/g), 'じゅうしょです。']);+ assert.equal(chunks.at(-1).gapBefore, 0.7);+ assert.equal(parseScript('## intro\n' + text, { language }).scenes[0].text, text);+ for (const ending of ['。', '!', '?', '.', '!', '?']) {+ assert.equal(speechChunks(tokenizeNarration(`解析${ending}解析${ending}`, 0, language), { language }).length, 2);+ }+ });++ test(`${language} alignment ignores Whisper word boundaries and times each word's characters`, () => {+ const words = [{ spoken: '解析' }, { spoken: '地址。' }];+ const whole = alignWords(words, [{ text: '解析地址', start: 0.2, end: 1 }], 1.2, language);+ const single = alignWords(words, [...'解析地址'].map((text, i) => ({ text, start: 0.2 + i * 0.2, end: 0.4 + i * 0.2 })), 1.2, language);+ assert.equal(whole.matchRate, 1);+ assert.deepEqual(whole.words, single.words);+ assert.ok(Math.abs(whole.words[0].start - 0.2) < 1e-9);+ assert.ok(Math.abs(whole.words[0].end - 0.6) < 1e-9);+ const shown = alignWords([{ spoken: 'じゅうしょ', display: '住所' }, { spoken: 'です。' }], [{ text: '住所です', start: 0, end: 1 }], 1, language);+ assert.equal(shown.matchRate, 1);+ assert.ok(shown.words.every(w => w.matched));+ const missing = alignWords([{ spoken: '解析' }, { spoken: '地' }, { spoken: '址' }], [{ text: '解析址', start: 0, end: 0.9 }], 1, language);+ assert.equal(missing.matchRate, 0.75);+ assert.equal(missing.words[1].matched, false);+ assert.ok(missing.words.every(w => w.start <= w.end));+ assert.equal(alignWords([{ spoken: 'かな' }], [{ text: 'かな'.normalize('NFD'), start: 0, end: 1 }], 1, language).matchRate, 1);+ });++ test(`${language} captions and subtitles join without spaces and limit lines at word boundaries`, () => {+ const text = native[language].repeat(3);+ const words = tokenizeNarration(text, 0, language).filter(u => u.type === 'word').map((u, i) => ({ text: u.display, start: i * 0.2, end: (i + 1) * 0.2 }));+ const timeline = buildTimeline({ config: normalizeConfig({ language, captions: true }), script: parseScript('## intro\n' + text, { language }) }, { intro: { duration: words.length * 0.2, words, marks: {} } });+ const cues = subtitleCues(timeline);+ assert.equal(cues.map(c => c.text.replaceAll('\n', '')).join(''), text);+ assert.ok(cues.every(c => c.text.split('\n').every(line => line.length <= 18)));+ assert.ok(toSrt(cues).includes('解析'));+ assert.ok(toVtt(cues).includes('解析'));+ const phrases = buildPhrases(timeline, 46);+ assert.equal(phrases.map(p => p.words.map(w => w.text).join(p.separator)).join(''), text);+ assert.ok(phrases.every(p => p.words.map(w => w.text).join('').length <= 18));+ const drawn = [];+ const ctx = new Proxy({ measureText: text => ({ width: text.length * 44 }), fillText: (text, x) => drawn.push({ text, x }) }, { get: (o, key) => o[key] ?? (() => {}) });+ drawCaptions(ctx, phrases, phrases[0].start + 0.1, getTheme('clean'), 1920, 1080);+ assert.equal(drawn[1].x - drawn[0].x, drawn[0].text.length * 44);+ const long = { language, scenes: [{ start: 0, words: Array.from({ length: 30 }, (_, i) => ({ text: '解析', start: i * 0.1, end: (i + 1) * 0.1 })) }] };+ assert.ok(subtitleCues(long).every(c => c.text.split('\n').every(line => line.length <= 18 && line.length % 2 === 0)));+ assert.throws(() => subtitleCues({ language, scenes: [{ words: [{ text: '長'.repeat(19) }] }] }), /longer than 18/);+ });+}++test('Mandarin alignment reads Traditional characters as Simplified, Japanese keeps them', () => {+ const rate = language => alignWords([{ spoken: '浏览器' }, { spoken: '会询问网址。' }], [{ text: '瀏覽器會詢問網址', start: 0, end: 1.6 }], 1.6, language).matchRate;+ assert.equal(rate('zh'), 1);+ assert.equal(rate('ja'), 0.25);+});++test('Mandarin alignment accepts a character with the same sound, not a different one', () => {+ const words = [{ spoken: '哈希值' }, { spoken: '和' }, { spoken: '盐。' }];+ const heard = text => [{ text, start: 0, end: 1 }];+ // Whisper writes 直 for 值 and 研 for 盐; all are read zhi and yan.+ assert.equal(alignWords(words, heard('哈希直和研'), 1, 'zh').matchRate, 1);+ assert.equal(alignWords(words, heard('哈希職和研'), 1, 'zh').matchRate, 1);+ const wrong = alignWords(words, heard('哈希值和鸡'), 1, 'zh');+ assert.equal(wrong.matchRate, 0.8);+ assert.equal(wrong.words[2].matched, false);+ assert.equal(alignWords(words, heard('哈希直和研'), 1, 'ja').matchRate, 0.6);+ const shown = alignWords([{ spoken: '哈希', display: '哈希值' }], heard('哈希直'), 1, 'zh');+ assert.equal(shown.matchRate, 1);+});++test('Japanese readings are shared out over the characters they cover', () => {+ assert.deepEqual(spreadReadings(['うるう', '年', 'には'], [[0, 4, 'ウルードシ'], [4, 5, 'ニ'], [5, 6, 'ワ']]), ['ウルード', 'シ', 'ニワ']);+ assert.deepEqual(spreadReadings(['ウルードシ', '、'], [[0, 5, 'うるーどし'], [5, 6, '、']]), ['ウルードシ', '、']);+ assert.deepEqual(spreadReadings(['今日', 'は'], [[0, 2, 'キョー']]), ['キョー', '']);+});++test('the Japanese speech check compares readings, and falls back to characters without Python', { skip: process.platform === 'win32' }, async t => {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-readings-'));+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));+ const file = path.join(dir, 'python');+ env(t, 'EXPLAINROO_PYTHON', file);+ // A tiny dictionary: 世紀 and 正規 are both read セーキ.+ fs.writeFileSync(file, `#!/usr/bin/env node+ const assert = require('node:assert/strict');+ assert.equal(process.argv[3], 'ja-readings');+ const words = { 世紀: 'セーキ', 正規: 'セーキ', の: 'ノ', 年: 'トシ', とし: 'トシ' };+ let input = '';+ process.stdin.on('data', d => input += d);+ process.stdin.on('end', () => {+ for (const text of input.trimEnd().split('\\n').map(JSON.parse)) {+ const out = [];+ for (let i = 0; i < text.length;) {+ const w = Object.keys(words).find(w => text.startsWith(w, i));+ if (w) { out.push([i, i + w.length, words[w]]); i += w.length; } else i++;+ }+ console.log(JSON.stringify(out));+ }+ });+ `, { mode: 0o755 });+ const words = [{ spoken: '世紀', display: '世紀' }, { spoken: 'の', display: 'の' }, { spoken: '年。', display: '年。' }];+ const heard = [{ text: '正規の', start: 0, end: 0.6 }, { text: 'とし', start: 0.6, end: 1 }];+ const alike = await readAlike(words, heard, 'ja');+ assert.deepEqual(alike.words.map(w => w.spoken), ['セーキ', 'ノ', 'トシ']);+ assert.deepEqual(alike.heard.map(h => h.text), ['セーキノ', 'トシ']);+ assert.equal(alignWords(alike.words, alike.heard, 1, 'ja').matchRate, 1);+ assert.deepEqual(await readAlike(words, heard, 'zh'), { words, heard });+ fs.rmSync(file);+ assert.deepEqual(await readAlike(words, heard, 'ja'), { words, heard });+ assert.ok(alignWords(words, heard, 1, 'ja').matchRate < 0.7);+});++test('Python batches JSON lines without shell expansion and accepts misaki phonemes unchanged', { skip: process.platform === 'win32' }, async t => {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-python-'));+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));+ const file = path.join(dir, 'fake python');+ env(t, 'EXPLAINROO_PYTHON', file);+ fs.writeFileSync(file, `#!/usr/bin/env node+ const assert = require('node:assert/strict');+ assert.equal(process.argv[2].endsWith('/g2p.py'), true);+ assert.ok(['ja', 'zh'].includes(process.argv[3]));+ let input = '';+ process.stdin.on('data', d => input += d);+ process.stdin.on('end', () => {+ const texts = input.trimEnd().split('\\n').map(JSON.parse);+ process.stdout.write(texts.map(t => JSON.stringify(t + '↗')).join('\\n') + '\\n');+ });+ `, { mode: 0o755 });+ for (const language of ['ja', 'zh']) {+ const texts = ['解析。\n次の行', '$( ) `*` "中文" $@'];+ assert.deepEqual(await phonemize(texts, language), texts.map(t => t + '↗'));+ assert.equal(misakiAvailable(language), true);+ // Latin words are read as English between the misaki pieces.+ assert.deepEqual(await phonemize(['我用JavaScript写代码。', 'API。'], language), ['我用↗ dʒˈɑːvə skɹˈɪpt 写代码。↗', 'ˌeɪpˌiːˈaɪ 。↗']);+ }+ fs.writeFileSync(file, '#!/usr/bin/env node\nconsole.log("not JSON");');+ await assert.rejects(phonemize(['解析'], 'ja'), /incomplete phonemes/);+ assert.equal(misakiAvailable('ja'), false);+ fs.writeFileSync(file, '#!/usr/bin/env node\nconsole.log(JSON.stringify("one"));');+ await assert.rejects(phonemize(['解析', '次'], 'ja'), /incomplete phonemes/);+ fs.writeFileSync(file, '#!/usr/bin/env node\nconsole.error("ModuleNotFoundError: No module named \'misaki\'"); process.exit(1);');+ await assert.rejects(phonemize(['解析'], 'zh'), /pip install "misaki\[zh\]".*Python said: ModuleNotFoundError: No module named 'misaki'/);+ fs.chmodSync(file, 0o644);+ await assert.rejects(phonemize(['解析'], 'ja'), /EXPLAINROO_PYTHON/);+ fs.rmSync(file);+ assert.equal(misakiAvailable('ja'), false);+ await assert.rejects(phonemize(['解析'], 'ja'), /-m unidic download/);+ assert.deepEqual(await phonemize([], 'ja'), []);+});++test('font paths follow EXPLAINROO_CACHE, offline mode refuses missing or damaged fonts', async t => {+ assert.equal(languageFonts('ja')[0].path, path.join(cacheRoot(), 'fonts', 'ja-400.otf'));+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-font-cache-'));+ t.after(() => fs.rmSync(dir, { recursive: true, force: true }));+ env(t, 'EXPLAINROO_CACHE', dir);+ env(t, 'EXPLAINROO_OFFLINE', '1');+ assert.deepEqual(languageFonts('en'), []);+ for (const language of ['ja', 'zh']) {+ const fonts = languageFonts(language);+ assert.deepEqual(fonts.map(f => f.weight), [400, 700]);+ assert.equal(fonts[0].path, path.join(dir, 'fonts', `${language}-400.otf`));+ await assert.rejects(ensureFonts(language), /without EXPLAINROO_OFFLINE/);+ fs.mkdirSync(path.join(dir, 'fonts'), { recursive: true });+ fs.writeFileSync(fonts[0].path, 'OTTOcorrupt');+ await assert.rejects(ensureFonts(language), /missing or damaged/);+ }+ await ensureFonts('en');+ process.env.EXPLAINROO_OFFLINE = '0';+ t.mock.method(globalThis, 'fetch', async () => ({ ok: true, arrayBuffer: async () => Buffer.from('OTTOcorrupt') }));+ await assert.rejects(ensureFonts('ja'), /could not be downloaded or checked/);+ assert.equal(fs.readFileSync(languageFonts('ja')[0].path, 'utf8'), 'OTTOcorrupt');+});diff --git a/test/export.test.js b/test/export.test.jsindex 6c10688..92d5b15 100644--- a/test/export.test.js+++ b/test/export.test.js@@ -53,6 +53,17 @@ test('player copies only used assets, font families and local modules, with publassert.ok(!html.includes('http:') && !html.includes('https:'));});+test('player ships the Hindi fallback font only when the video draws Hindi', t => {+ const { dir, project, timeline } = fixture(t);+ const families = text => {+ const out = path.join(dir, 'player-' + text.length);+ copyPlayer(project, timeline, { assets: [], fonts: ['Inter', 'Noto Sans Devanagari'], math: {}, text }, out);+ return [...new Set(JSON.parse(fs.readFileSync(path.join(out, 'fonts/fonts.json'), 'utf8')).map(f => f.family))];+ };+ assert.deepEqual(families('Hello'), ['Inter']);+ assert.deepEqual(families('नमस्ते'), ['Inter', 'Noto Sans Devanagari']);+});+test('export follows local imports, refusing private files, outside symlinks and remote modules', t => {const { dir, project, timeline } = fixture(t);assert.deepEqual([...localModules(dir).keys()], ['scenes.js', 'lib/draw.mjs', 'helper.js']);diff --git a/test/languages.test.js b/test/languages.test.jsindex 081bc2a..20d1fc4 100644--- a/test/languages.test.js+++ b/test/languages.test.js@@ -50,7 +50,7 @@ test('each language picks its default and rejects every voice from a different l}assert.equal(asrModel(language), language === 'en' ? ASR_MODEL : MULTILINGUAL_ASR_MODEL);}- for (const language of ['ja', 'zh', 'pt', '', 'EN', 'toString', null, true, ['es']]) assert.throws(() => normalizeConfig({ language }), /language must be one of/);+ for (const language of ['ko', 'pt', '', 'EN', 'toString', null, true, ['es']]) assert.throws(() => normalizeConfig({ language }), /language must be one of/);});test('non-English numbers stay digits, including Devanagari, and are never fuzzy matched or spelled out', () => {@@ -103,7 +103,7 @@ test('captions and subtitle files keep native text and split at danda', () => {}});-test('init stores the language and default voice; voices filters and doctor allows missing espeak', async t => {+test('init stores the language and default voice; voices filters and doctor allows missing espeak-ng and misaki', async t => {const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'explainroo-language-cli-'));t.after(() => fs.rmSync(dir, { recursive: true, force: true }));let output = '';@@ -114,7 +114,7 @@ test('init stores the language and default voice; voices filters and doctor alloconst config = JSON.parse(fs.readFileSync(path.join(dir, 'video/video.json')));assert.equal(config.language, 'es');assert.equal(config.voice, 'ef_dora');- for (const flags of [['--language', 'ja'], ['--language'], ['--language', 'es', '--voice', 'af_heart']]) {+ for (const flags of [['--language', 'ko'], ['--language'], ['--language', 'es', '--voice', 'af_heart']]) {assert.equal(await main(['init', path.join(dir, 'bad'), ...flags]), 1);assert.equal(fs.existsSync(path.join(dir, 'bad')), false);}@@ -129,5 +129,12 @@ test('init stores the language and default voice; voices filters and doctor allot.after(() => { if (saved === undefined) delete process.env.EXPLAINROO_ESPEAK; else process.env.EXPLAINROO_ESPEAK = saved; });output = '';assert.equal(await main(['doctor', '--json']), 0);- assert.match(JSON.parse(output).checks.find(c => c.name === 'espeak-ng').detail, /only needed for other languages/);+ assert.match(JSON.parse(output).checks.find(c => c.name === 'espeak-ng').detail, /only needed for Spanish, French, Italian, Portuguese and Hindi/);+ const python = process.env.EXPLAINROO_PYTHON;+ process.env.EXPLAINROO_PYTHON = path.join(dir, 'missing');+ t.after(() => { if (python === undefined) delete process.env.EXPLAINROO_PYTHON; else process.env.EXPLAINROO_PYTHON = python; });+ output = '';+ assert.equal(await main(['doctor', '--json']), 0);+ const misaki = JSON.parse(output).checks.filter(c => c.name.startsWith('misaki'));+ assert.deepEqual(misaki.map(c => [c.name, c.ok, c.detail]), [['misaki ja', true, 'not found; only needed for Japanese'], ['misaki zh', true, 'not found; only needed for Chinese']]);});diff --git a/test/phonemes.test.js b/test/phonemes.test.jsindex a905e76..bf44530 100644--- a/test/phonemes.test.js+++ b/test/phonemes.test.js@@ -71,7 +71,7 @@ test('failed and incomplete output gives a friendly error', posix, async t => {await assert.rejects(phonemize(['Hola'], 'es'), /could not read the text/);fs.writeFileSync(file, '#!/usr/bin/env node\nconsole.log("one");');await assert.rejects(phonemize(['Hola', 'Adiós'], 'es'), /incomplete phonemes/);- await assert.rejects(phonemize(['こんにちは'], 'ja'), /supports es, fr, it, pt-br and hi/);+ await assert.rejects(phonemize(['안녕'], 'ko'), /supports es, fr, it, pt-br and hi/);});test('English keeps generate; other languages tokenize mapped phonemes and call generate_from_ids', async () => {