An AI skill that turns books into animated, narrated, interactive web experiences
JavaScript ai-agentsai-educationanimationclaudeclaude-codeclaude-skillsedtecheducationinteractive-textbookscormtext-to-video
feat(skill): Kokoro as a local narration engine
yzr083d560cb479
diff --git a/.claude/skills/papermorph/SKILL.md b/.claude/skills/papermorph/SKILL.mdindex 6d956a9..e09044c 100644--- a/.claude/skills/papermorph/SKILL.md+++ b/.claude/skills/papermorph/SKILL.md@@ -32,7 +32,7 @@ site/<book>/ the static book (only this is delivered)chNN/index.html, audio/<lang>/ a lesson, its beat MP3s + timings.js```-`SKILL` means this skill's absolute folder; `<book>` is the book slug, `<lang>` its primary language code. Existing books keep their paths. Needs: `uv`, `ffmpeg`/`ffprobe`, Edge TTS network access and Playwright browsers (`uv run --with playwright playwright install chromium firefox`). Preview: `python3 SKILL/scripts/serve.py 8765 site` (no browser caching).+`SKILL` means this skill's absolute folder; `<book>` is the book slug, `<lang>` its primary language code. Existing books keep their paths. Needs: `uv`, `ffmpeg`/`ffprobe`, Edge TTS network access (or Kokoro, local: `uv run --python 3.12 --with kokoro`) and Playwright browsers (`uv run --with playwright playwright install chromium firefox`). Preview: `python3 SKILL/scripts/serve.py 8765 site` (no browser caching).## Pipeline@@ -47,7 +47,7 @@ Start with intake whenever `books/<book>/style.json` does not exist yet; reuse a## Chapter loop1. **Read** this chapter's `pages/chNN/text.md`; render pages with `split_pages.py … --only chNN` when diagrams or missing text need images. Choose the ideas, examples and sections to teach.-2. **Beats and narration**: list beats in `chapters/chNN.md` with what each says and which word triggers which action ([authoring.md](references/authoring.md)). Write `content/<book>/chNN/narration.<lang>.json`, then `uv run --with edge-tts SKILL/scripts/tts.py content/<book>/chNN/narration.<lang>.json site/<book>/chNN/audio/<lang>`.+2. **Beats and narration**: list beats in `chapters/chNN.md` with what each says and which word triggers which action ([authoring.md](references/authoring.md)). Write `content/<book>/chNN/narration.<lang>.json`, then `uv run --with edge-tts SKILL/scripts/tts.py content/<book>/chNN/narration.<lang>.json site/<book>/chNN/audio/<lang>` (for `"engine": "kokoro"`: `uv run --python 3.12 --with kokoro …`).3. **Storyboard, layout first**: in the real page, place everything for one to three key moments per beat, real elements at their real sizes, no motion yet. Capture them into one sheet with `strip.py` and give each frame one line: what it says and where the eye goes. That sheet is the storyboard. Note what the chapter needs beyond the library: draw it, or add it to the image requests.4. **Motion**: add the motion, the camera and the questions; API and replay invariants are in [engine.md](references/engine.md).5. **Delivery pass**: watch your output in motion and judge it as a viewer would, against the storyboard; then run the checks in [review.md](references/review.md). Correct what you find, recheck, deliver; report unresolved problems.diff --git a/.claude/skills/papermorph/references/authoring.md b/.claude/skills/papermorph/references/authoring.mdindex 397bb44..da1abdb 100644--- a/.claude/skills/papermorph/references/authoring.md+++ b/.claude/skills/papermorph/references/authoring.md@@ -37,6 +37,7 @@ Before writing code, list the beats. For each lesson beat decide what the eye sh- Match the audience: plain, warm, never childish. Explain why, not only how.- Question beats get one short line ("Quick check. Find the mean."); the card holds the question.- `tts.py` writes `<beat>.mp3` and `timings.js` (duration, marks, captions) and caches each finished beat beside the narration JSON: moving `[[marks]]` reuses the cache, and after a failure the same command resumes rather than starting over.+- Engine: `"engine": "edge"` (default, online) or `"kokoro"` (local, Apache 2.0; English word timings; voices such as `am_michael`, `af_heart`, `bm_george`), set at the top of the narration file with `voice` and `rate`. Keep one engine and voice per book.## Questionsdiff --git a/.claude/skills/papermorph/references/intake.md b/.claude/skills/papermorph/references/intake.mdindex bb3b492..a75c3f0 100644--- a/.claude/skills/papermorph/references/intake.md+++ b/.claude/skills/papermorph/references/intake.md@@ -51,7 +51,7 @@ Start from [the template](../assets/templates/style.json). Blocks are grouped by| `schema`, `tier`, `book` | schema version; `1`, `2` or `"custom"`; title, slug, language, readers, pilot chapter | all || `words` | the user's own words about the look, verbatim, with later feedback that changed the bar | all || `look` | palette, what each colour means, text and maths fonts, notation and rounding | all |-| `voice` | TTS engine (`edge`), voice, rate | all |+| `voice` | TTS engine (`edge`, or `kokoro` when Edge is unavailable or local audio is wanted), voice, rate | all || `pacing` | minutes per chapter, where checks and practice go | all || `engine` | engine source and version used to build the book | all || `scenes` | art style, cast, how storyboards are reviewed | tier 2 |diff --git a/.claude/skills/papermorph/scripts/tts.py b/.claude/skills/papermorph/scripts/tts.pyindex d62684c..57fa3b1 100644--- a/.claude/skills/papermorph/scripts/tts.py+++ b/.claude/skills/papermorph/scripts/tts.py@@ -1,9 +1,13 @@-"""Generate narration audio and word-mark timings with Edge TTS.+"""Generate narration audio and word-mark timings with Edge TTS or Kokoro.Usage (from the project root; SKILL is this skill's folder):uv run --with edge-tts $SKILL/scripts/tts.py content/ch01/narration.en.json site/ch01/audio/en+ uv run --python 3.12 --with kokoro $SKILL/scripts/tts.py ... # narration with "engine": "kokoro"-Needs network access and ffprobe (from ffmpeg) for clip durations.+The narration file names the engine ("edge", the default, or "kokoro"), the voice and the rate. Edge needs network+access. Kokoro (Apache 2.0) runs locally: its first run downloads the model (about 330 MB) and an English dictionary;+voices start with the accent's letter (am_michael, af_heart: American; bm_george, bf_emma: British) and its word+timings cover English. Both need ffmpeg/ffprobe.Narration text may contain [[mark]] tags placed before a word. The outputtimings.js records each clip's duration and the time (seconds) at which the@@ -31,8 +35,6 @@ import sysimport tempfilefrom pathlib import Path-import edge_tts-from edge_tts.exceptions import NoAudioReceivedMARK = re.compile(r"\[\[(\w+)\]\]")MAX_ATTEMPTS = 5 # requests per beat when Edge returns no audio@@ -41,6 +43,7 @@ PROBE_TEXT = "1, 2, 3." # checks the voice after a beat fails; any voice readsasync def synth(text, voice, rate, mp3_path):+ import edge_ttswords = [] # (seconds, word)comm = edge_tts.Communicate(text, voice, rate=rate, boundary="WordBoundary")with open(mp3_path, "wb") as f:@@ -54,6 +57,7 @@ async def synth(text, voice, rate, mp3_path):async def synth_with_retries(beat, text, voice, rate, mp3_path):"""synth(), requested again after NoAudioReceived; each attempt rewrites mp3_path from the start."""+ from edge_tts.exceptions import NoAudioReceivedfor attempt in range(1, MAX_ATTEMPTS + 1):try:return await synth(text, voice, rate, mp3_path)@@ -68,6 +72,7 @@ async def synth_with_retries(beat, text, voice, rate, mp3_path):async def explain_no_audio(beat, voice, rate, out_dir):"""The message for a beat that ran out of attempts: whether the voice or this beat's text failed."""+ from edge_tts.exceptions import NoAudioReceivedwith tempfile.NamedTemporaryFile(dir=out_dir, prefix=".probe-", suffix=".mp3", delete=False) as f:probe_path = Path(f.name)try:@@ -85,6 +90,39 @@ async def explain_no_audio(beat, voice, rate, out_dir):"Rerun later, or choose another voice for the whole book (uv run --with edge-tts edge-tts --list-voices).")+async def edge_beat(beat, text, voice, rate, mp3_path, out_dir):+ from edge_tts.exceptions import NoAudioReceived+ try:+ return await synth_with_retries(beat, text, voice, rate, mp3_path)+ except NoAudioReceived:+ raise SystemExit(await explain_no_audio(beat, voice, rate, out_dir)) from None+++KOKORO = {} # one pipeline per accent, loaded once per run+++def kokoro_beat(text, voice, rate, mp3_path):+ """Synthesize locally with Kokoro; word start times come from its predicted durations."""+ import numpy as np+ from kokoro import KPipeline+ lang = voice[0]+ if lang not in KOKORO:+ KOKORO[lang] = KPipeline(lang_code=lang, repo_id="hexgrad/Kokoro-82M")+ speed = 1 + float(rate.rstrip("%")) / 100 if rate else 1+ words, chunks, offset = [], [], 0.0+ for r in KOKORO[lang](text, voice=voice, speed=speed):+ audio = np.asarray(r.audio, dtype=np.float32)+ for tk in r.tokens or []:+ if tk.start_ts is not None and any(c.isalnum() for c in tk.text):+ words.append((round(offset + tk.start_ts, 3), tk.text))+ chunks.append(audio)+ offset += len(audio) / 24000+ pcm = (np.clip(np.concatenate(chunks), -1, 1) * 32767).astype("<i2").tobytes()+ subprocess.run(["ffmpeg", "-v", "error", "-y", "-f", "s16le", "-ar", "24000", "-ac", "1", "-i", "-",+ "-b:a", "64k", str(mp3_path)], input=pcm, check=True)+ return words++def split_marks(raw):"""Return clean text and {mark: char position in clean text}."""marks, clean, pos = {}, [], 0@@ -157,7 +195,9 @@ async def main(src, out_dir):out_dir.mkdir(parents=True, exist_ok=True)cache_path = Path(src).with_suffix(".timings.json") # build cache, kept out of the sitecache = json.loads(cache_path.read_text(encoding="utf-8-sig")) if cache_path.exists() else {}- voice, rate = spec["voice"], spec["rate"]+ voice, rate, engine = spec["voice"], spec["rate"], spec.get("engine", "edge")+ if engine not in ("edge", "kokoro"):+ raise SystemExit(f"unknown engine {engine!r}: use edge or kokoro")result = {}public = lambda v: {k: v[k] for k in ("dur", "marks", "cues")}live = {b: public(v) for b, v in cache.items()@@ -167,11 +207,12 @@ async def main(src, out_dir):mp3 = out_dir / f"{beat}.mp3"old = cache.get(beat)clean, marks = split_marks(raw)- key = hashlib.sha1(f"{voice}|{rate}|{clean}".encode()).hexdigest()+ tag = "" if engine == "edge" else f"{engine}|" # Edge keys stay as they were, so its audio is kept+ key = hashlib.sha1(f"{tag}{voice}|{rate}|{clean}".encode()).hexdigest()# Existing caches included [[marks]] in the key; reuse an unchanged old beat once,# then save its spoken-text key so future mark edits do not regenerate audio.legacy_key = hashlib.sha1(f"{voice}|{rate}|{raw}".encode()).hexdigest()- reusable = old and old.get("key") in (key, legacy_key) and "words" in old and mp3.exists() and mp3.stat().st_size > 0+ reusable = old and old.get("key") in (key, legacy_key if engine == "edge" else key) and "words" in old and mp3.exists() and mp3.stat().st_size > 0tmp = Noneif reusable:words = old["words"]@@ -182,10 +223,8 @@ async def main(src, out_dir):clip = tmptry:if not reusable:- try:- words = await synth_with_retries(beat, clean, voice, rate, clip)- except NoAudioReceived:- raise SystemExit(await explain_no_audio(beat, voice, rate, out_dir)) from None+ words = (kokoro_beat(clean, voice, rate, clip) if engine == "kokoro"+ else await edge_beat(beat, clean, voice, rate, clip, out_dir))entry = {"key": key, "dur": duration(clip), "words": words,"marks": mark_times(clean, marks, words), "cues": caption_cues(clean, words)}if tmp: