An AI skill that turns books into animated, narrated, interactive web experiences
JavaScript ai-agentsai-educationanimationclaudeclaude-codeclaude-skillsedtecheducationinteractive-textbookscormtext-to-video
feat(skill): ElevenLabs word cache, pause-aware word times, long-word mouths, export video option
yzr0240b16dc3b7
diff --git a/.claude/skills/papermorph/references/core/sound.md b/.claude/skills/papermorph/references/core/sound.mdindex 54fe739..dda47cb 100644--- a/.claude/skills/papermorph/references/core/sound.md+++ b/.claude/skills/papermorph/references/core/sound.md@@ -35,6 +35,7 @@ Voices carry every book and everything else sits under them: in early cuts music- It prints the cost before paying (`--dry` stops there; about $0.07 a minute, quote it) and caches paid clips (a rerun costs nothing); `--only beat,beat` makes a fresh, paid take of those beats alone.- A sped-up voice (`--tempo 1.1`, or `"tempo"` in the narration file) is sped before levelling and alignment, and stays sped on a rerun. `--max-pause 0.3` shortens its sentence pauses (0.4 to 1.0 s) before the marks are placed.- v4 audio tags such as `[cackles]` are acted, not read, and left out of the captions and of `lint.py`'s comparison of narration and page.+ - Every word's heard time also goes in the `.timings.json` beside the narration, as `tts.py` writes it, so a picture book's `mouths.py` and `join.py --words` work on its clips (a cast of ElevenLabs voices: *The Little Horse Crosses the River*); those times need no Edge correction, and both scripts know it.- Its word alignment (local faster-whisper) assumes words split by spaces, so a Chinese or Japanese film needs `tts.py` for now. The transcript writes spoken numbers as digits; `eleven.py` matches them back and prints the marks a rerun moved, but check marks on numbers: a missed one lands late (up to 0.75 s).- **Several voices.** List them under `voices` in the narration file and write each line as `{"speaker": name, "text": …}`; a line without a speaker takes the file's own `voice`, and a speaker missing from `voices` stops the run, naming it. Both scripts do this. Voices need pitches apart (a dog at 113 Hz under a 110 Hz narrator read as the same person).- Let the user pick a voice by ear from a few short samples in the book's own words.diff --git a/.claude/skills/papermorph/references/picture-books/stories.md b/.claude/skills/papermorph/references/picture-books/stories.mdindex f9761b5..9a86bc3 100644--- a/.claude/skills/papermorph/references/picture-books/stories.md+++ b/.claude/skills/papermorph/references/picture-books/stories.md@@ -92,7 +92,7 @@ Everybody in the shot stays alive and stays there: extras blink and glance aboutVoices, levels, effects and music work as in [core/sound.md](../core/sound.md); a picture book adds a cast that speaks on screen and one clip per scene.-- **Mouths** follow the words: `mouths.py` turns each line's word timings into mouth shapes (a Chinese or Japanese line opens and shuts once per character), and `lips(q, scene, pre, line…)` picks the frame's shape. Edge's word times run about a tenth of a second ahead of the sound; `mouths.py` and `join.py` move mouths and captions that much later and shut the mouth where the clip's sound stops, so a lip flap that leads or trails the voice is worth reporting.+- **Mouths** follow the words: `mouths.py` turns each line's word timings into mouth shapes (a Chinese or Japanese line opens and shuts once per character), and `lips(q, scene, pre, line…)` picks the frame's shape. Edge's word times run about a tenth of a second ahead of the sound (`eleven.py`'s are heard times and stay put); `mouths.py` and `join.py` move Edge's mouths and captions that much later and shut the mouth where the clip's sound stops, so a lip flap that leads or trails the voice is worth reporting.- **One clip per scene.** `join.py` joins a scene's lines into one clip with short pauses, and inserts silence where the action needs room (`line@T+S`: S seconds of silence T seconds into the line, after a word's sound has ended rather than where the next word begins) or after the last line (`name:S`, a stare or a tag). Gaps for a held stare, a fly landing, a class turning round belong here, not in rushed animation. `--words` writes every word's time in the scene clip beside the narration, for scene authors to time actions against (the beat clock adds the scene's `pre`).- **One script for the whole sound**, `tools/<book>/audio.sh`, rerun after any change: lines replaced by hand, `mouths.py`, level fixes, `join.py` with its holes, caption edits. Join arguments typed only in a terminal have had to be read back out of `shots.js`.- **Levels.** `level.py` runs right after `tts.py` and before `join.py`, at its default −18 LUFS ([core/sound.md](../core/sound.md#levels)). The line the joke hangs on must not be the quietest; one that must sit above the rest (a quiet old man's punchline) gets its own target (`level.py o1.mp3 --lufs -15`) and stays out of the others' list.diff --git a/.claude/skills/papermorph/scripts/eleven.py b/.claude/skills/papermorph/scripts/eleven.pyindex 6943a0f..d17ce73 100644--- a/.claude/skills/papermorph/scripts/eleven.py+++ b/.claude/skills/papermorph/scripts/eleven.py@@ -18,7 +18,10 @@ language, from the file name narration.<lang>.json; the first run downloads thematched to the heard words in order; each [[mark]] is the start of the word after it, and there is one caption cue persentence and per clause after a colon ("Mr." and "e.g." do not end one, as in tts.py). Whisper writes spoken numbers indigits ("a hundred and twenty" heard as "120"): a number's first word is matched to such a heard number, so a mark on it-lands on the number's start. A rerun prints the marks that moved more than 0.1 s against the old timings.js. Audio is decoded with ffmpeg, not PyAV (faster-whisper's PyAV version clashes). Needs ffmpeg.+lands on the number's start. A rerun prints the marks that moved more than 0.1 s against the old timings.js. Every word's heard time also goes+in the .timings.json beside the narration, as tts.py writes it, so a picture book's mouths.py and join.py --words work+on these clips too. A word Whisper starts inside a pause (it often does, up to 0.7 s early) is moved to where the+sound comes back, and the run prints it. Audio is decoded with ffmpeg, not PyAV (faster-whisper's PyAV version clashes). Needs ffmpeg.v4 audio tags such as [cackles] or [whispers] act: they are performed, not read aloud, and left out of captions.Price: about $0.00008 per character (about $0.65 for a ten-minute film). Before paying, the run prints the characters@@ -50,6 +53,7 @@ import argparse, atexit, hashlib, json, pathlib, re, subprocess, sys, urllib.errfrom paint import keyfrom ledger import appendfrom tts import ABBREV, split_marks, mark_times, duration, write_atomic+from silence import silencesimport level as lvTAG = re.compile(r"(?<!\[)\[[^\[\]]+\](?!\])") # a v4 audio tag, not a [[mark]]@@ -147,6 +151,8 @@ print(f"{sum(b[6] for b in beats)} beats to synthesize, {todo} characters, aboutif a.dry:sys.exit()timings, spent = {}, 0+wc_path = pathlib.Path(a.narration).with_suffix(".timings.json") # every word's heard time, for mouths.py and join.py+wcache = json.loads(wc_path.read_text(encoding="utf-8")) if wc_path.exists() else {}def charge():@@ -196,6 +202,24 @@ for beat, voice, sub, clean, marks, cached, fresh in beats:if d is not None and (k is None or d < k): k, run = d, heard[d][1]times.append(None if k is None else heard[k][1]); j = j if k is None else k + 1dur = duration(mp3)+ # Whisper starts a word that follows a pause inside the pause (up to 0.7 s early, the mouth moving in silence):+ # a word heard inside a silence starts where the sound comes back+ gaps = [(a, b) for a, b in silences(mp3) if b < dur - .05]+ for i, v in enumerate(times):+ g = next(((a, b) for a, b in gaps if v is not None and a - .02 <= v < b - .02), None)+ if g:+ print(f"{beat}: {toks[i]!r} moved from {v:.2f} to the end of the pause, {g[1]:.2f}")+ times[i] = round(g[1], 3)+ # ... or puts the word after a pause before it ("river? Isn't it" heard as Isn'[email protected], pause 0.78-1.21, [email protected]):+ # when no word starts where a pause of 0.3 s or more ends and the next word comes later than 0.2 s after it,+ # the last word before the pause is the one spoken there+ for g0, g1 in gaps: # (not a, b: `a` holds the arguments)+ if g1 - g0 < .3 or any(v is not None and abs(v - g1) < .03 for v in times): continue+ k = max((q for q, v in enumerate(times) if v is not None and v < g0), default=None)+ n = next((v for v in times[k + 1:] if v is not None), None) if k is not None else None+ if k is not None and k > 0 and n is not None and n > g1 + .2:+ print(f"{beat}: {toks[k]!r} moved from {times[k]:.2f} past the pause to {g1:.2f}")+ times[k] = round(g1, 3)for i, v in enumerate(times): # words Whisper merged or split: halfway between their known neighboursif v is None:prev = next((times[q] for q in range(i - 1, -1, -1) if times[q] is not None), 0.0)@@ -204,7 +228,7 @@ for beat, voice, sub, clean, marks, cached, fresh in beats:words = list(zip(times, toks))# caption cues: one per sentence or clause after a colon (long documentary sentences), at its first wordcues, i, sents = [], 0, []- for sent in re.split(r"(?<=[.?!:])\s+", re.sub(r"\s+", " ", TAG.sub(" ", clean)).strip()): # a tag parts words+ for sent in re.split(r"(?<=[.?!:])(?<!\.\.\.)\s+", re.sub(r"\s+", " ", TAG.sub(" ", clean)).strip()): # a tag parts words; "I... I can't" stays one lineif sents and ABBREV.search(sents[-1]): sents[-1] += " " + sent # "Mr. Lee", "e.g. a lot" go onelse: sents.append(sent)for sent in sents:@@ -213,10 +237,12 @@ for beat, voice, sub, clean, marks, cached, fresh in beats:if cues: cues[0][0] = min(cues[0][0], .1)timings[beat] = {"dur": dur, "marks": mark_times(clean, marks, words), "cues": [[0, sub]] if sub else cues}if sub: timings[beat]["forced"] = True+ wcache[beat] = {"dur": dur, "words": [[t, w] for t, w in words], "lag": 0} # heard times: no Edge lead to correctmoved = [f"{k} {old[beat]['marks'][k]} -> {v}" for k, v in timings[beat]["marks"].items()if k in old.get(beat, {}).get("marks", {}) and abs(old[beat]["marks"][k] - v) > .1]if moved: print(f"{beat}: marks moved: {', '.join(moved)}")print(f"{beat}: {dur} s, {len(marks)} marks")write_atomic(tf, "window.TIMINGS = " + json.dumps(timings, indent=1, ensure_ascii=False) + ";\n")write_atomic(tk, json.dumps(takes, indent=1))+write_atomic(wc_path, json.dumps(wcache, indent=1, ensure_ascii=False))print(f"{len(timings)} beats in {tf}; synthesized {spent} characters, about ${spent * PRICE:.2f}")diff --git a/.claude/skills/papermorph/scripts/export.py b/.claude/skills/papermorph/scripts/export.pyindex 18eeb3c..a439d86 100644--- a/.claude/skills/papermorph/scripts/export.py+++ b/.claude/skills/papermorph/scripts/export.py@@ -11,7 +11,9 @@ of their scene, as a scene change stops them) and the BGM tracks with their croschecks and practice) are left out, and then the video ends on a card saying they are in the interactive lesson(--link adds its address); in a film with interactive moments (CHAPTER.film, e.g. the reader fires a rifle) they stay in,each holding its last picture for its `hold` seconds (1.5 by default, the music up at 70 % as the page plays it while-it waits) before the next beat plays as if the reader had acted. A point-and-read beat (tapWords) holds while it shows+it waits) before the next beat plays as if the reader had acted. A beat with `video: false` in its fourth slot is left+out, and `video: S` ends it S seconds into its clock, sound and captions cut there, with no wait: a game that only+works on the page (a listen-and-find stop) stays out of the video. A point-and-read beat (tapWords) holds while it showsand says each word in turn, 2 s apart (its `hold`, if set, still wins); a word clip that is missing is named andleft silent. With --link and no question beats left out, the endcard shows the title and the link only. A mix that would clip is turned down as a whole (the run prints by how much).@@ -122,14 +124,18 @@ async def main(a):if errs:sys.exit(f'page errors: {errs}')info = await pg.evaluate("""({lang: CHAPTER.language || 'en', film: !!CHAPTER.film,- beats: BEATS.map(b => ({id: b.id, ask: !!b.ask, pre: b.pre || 0, hold: b.hold ?? null})),+ beats: BEATS.map(b => ({id: b.id, ask: !!b.ask, pre: b.pre || 0, hold: b.hold ?? null, video: b.video ?? null})),bgm: typeof BGM === 'undefined' ? [] : BGM.map(m => ({src: m.a.src, beats: m.beats, gain: m.gain, fade: m.fade ?? .8})),upInPre: window.MUSIC_UP_IN_PRE || false})""")# a film's question beats (CHAPTER.film: the reader acts, e.g. fires) stay in the video: their picture holds `hold` s# (BEATS[i].hold, 1.5 by default) where the page waits for the reader, then the next beat plays as if they had acted- beats = [(i, x['id']) for i, x in enumerate(info['beats']) if not x['ask'] or info['film']]+ beats = [(i, x['id']) for i, x in enumerate(info['beats']) if (not x['ask'] or info['film']) and x['video'] is not False]+ # `video` in a beat's fourth slot: false leaves it out of the video; a number (seconds on its clock) ends it+ # there, its sound and captions cut, with no wait (a point-and-read game that only works on the page)+ skipped = [x['id'] for x in info['beats'] if x['video'] is False]+ if skipped: print(f'left out of the video (video: false): {", ".join(skipped)}')info['sfx'] = []- asks = len(beats) < len(info['beats'])+ asks = not info['film'] and any(x['ask'] for x in info['beats']) # a lesson's questions, left out: the end card says where they arebox = await pg.locator('#stage').bounding_box()clip = {k: box[k] for k in ('x', 'y', 'width', 'height')}audio_dir = Path(tempfile.mkdtemp(prefix='export-'))@@ -142,7 +148,9 @@ async def main(a):await open_beat(pg, base, i)end, T = await pg.evaluate('[P.end, TIMINGS[BEATS[P.i].id]]')if T.get('silent'): silent.add(bid) # { dur } in the beat's fourth slot: no clip, its sound is silence- if info['beats'][i]['ask']: # from P.end the page waits+ cut = info['beats'][i]['video']+ if isinstance(cut, (int, float)) and not isinstance(cut, bool): end = min(end, cut)+ elif info['beats'][i]['ask']: # from P.end the page waitswaits[bid], hold = end, info['beats'][i]['hold']if demo := await pg.evaluate('P.tapDemo ? [P.tapDemo.clips, P.tapDemo.step] : null'):words[bid] = demo@@ -160,7 +168,7 @@ async def main(a):[bid, count, a.fps]))for k, c in enumerate(T['cues']): # [t, text, end?]: the page shows it at pre + tstop = c[2] if len(c) > 2 else T['cues'][k + 1][0] if k + 1 < len(T['cues']) else T['dur']- cues.append((offset + pre + c[0], offset + pre + stop, c[1]))+ if pre + c[0] < end: cues.append((offset + pre + c[0], offset + pre + min(stop, end - pre), c[1]))offset += endtotal = offset + end_strack = np.zeros((round(total * RATE) + 1, 2), np.float32)@@ -269,6 +277,7 @@ async def main(a):await x.close()with open(out / f'{a.chapter}.srt', 'w', encoding='utf-8') as f:for k, (t0, t1, line) in enumerate(cues, 1):+ if k < len(cues): t1 = min(t1, cues[k][0]) # a cue ends where the next begins: video sites stack overlapsf.write(f'{k}\n{srt_time(t0)} --> {srt_time(t1)}\n{line}\n\n')if errs or enc.returncode:part.unlink(missing_ok=True)diff --git a/.claude/skills/papermorph/scripts/join.py b/.claude/skills/papermorph/scripts/join.pyindex 9fec78b..256c4d8 100644--- a/.claude/skills/papermorph/scripts/join.py+++ b/.claude/skills/papermorph/scripts/join.py@@ -77,7 +77,7 @@ for arg in args:mine = [w for w in said if t <= w < lim]stop = min([last] + ([nxt] + [h for h, _ in holes[b] if t < h < nxt] if nxt is not None else [])+ [x for x in ps if t + .15 < x < lim and mine and x > max(mine) + .2])- cues.append([round(shift(b, t) + at + LAG, 3), c, round(shift(b, stop - 1e-3) + at + .3, 3)] + [1] * bool(T[b].get('forced')))+ cues.append([round(shift(b, t) + at + (WB.get(b) or {}).get('lag', LAG), 3), c, round(shift(b, stop - 1e-3) + at + .3, 3)] + [1] * bool(T[b].get('forced')))marks.update({f'{b}.{k}': round(shift(b, v) + at, 3) for k, v in T[b]['marks'].items()})at += T[b]['dur'] + sum(sec for _, sec in holes[b]) + GAPdur = float(subprocess.run(['ffprobe', '-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', str(d / f'{shot}.mp3')], capture_output=True, text=True).stdout)@@ -96,6 +96,6 @@ if words: # every word where it is heard in its scene clipfor b, p in v['parts'].items():hs = ' '.join(f'[hole {round(p + sh(b, at - 1e-6), 2)}…{round(p + sh(b, at - 1e-6) + sec, 2)}]' for at, sec in H.get(b, []))ws = W.get(b, {}).get('words', [])- lines.append(f" {b} @{p} {hs} | " + (' '.join(f'{w}@{round(p + sh(b, t) + LAG, 2)}' for t, w in ws) or '(silence)'))+ lines.append(f" {b} @{p} {hs} | " + (' '.join(f'{w}@{round(p + sh(b, t) + W.get(b, {}).get("lag", LAG), 2)}' for t, w in ws) or '(silence)'))words.with_suffix('.words.txt').write_text('\n'.join(lines) + '\n', encoding='utf-8')print({k: (v['dur'], v['parts']) for k, v in out.items()})diff --git a/.claude/skills/papermorph/scripts/mouths.py b/.claude/skills/papermorph/scripts/mouths.pyindex 3e5153a..f145342 100644--- a/.claude/skills/papermorph/scripts/mouths.py+++ b/.claude/skills/papermorph/scripts/mouths.py@@ -3,14 +3,15 @@python3 mouths.py content/<book>/chNN/narration.en.json site/<book>/chNN/audio/en-Reads the .timings.json cache that tts.py wrote next to the narration file and writes AUDIO_DIR/mouths.js+Reads the .timings.json cache that tts.py (or eleven.py) wrote next to the narration file and writes AUDIO_DIR/mouths.js(window.MOUTHS = {beat: [[seconds, shape], ...]}). Shapes follow the cartoon set Rhubarb uses: A closed (m b p),B teeth/most consonants, C open (e i), D wide (a), E round (o), F pucker (oo w u), G f/v, H l, X rest. Each word'sletters become shapes spread over the word's time; a pause between words closes the mouth. Letters are read asEnglish spelling; for another language the rules below are a rough guide only, and a word with no Latin letters(Chinese, Japanese) opens and closes once per character. Edge's word times run about 0.12 s ahead of the sound, so-every cue is moved that much later; when the clip AUDIO_DIR/<beat>.mp3 exists, the mouth shuts where its sound-stops. Run it after tts.py (and before join.py, which moves the cues of clips it joins). Needs ffmpeg.+every cue is moved that much later (eleven.py's times are heard ones and stay where they are); when the clip AUDIO_DIR/<beat>.mp3 exists, the mouth shuts where its sound+stops, and a word lasts until the next word or the clip's next silence (so a drawn-out "anything" keeps the mouth+moving). Run it after tts.py (and before join.py, which moves the cues of clips it joins). Needs ffmpeg."""import json, re, subprocess, sys, pathlibif len(sys.argv) < 3 or sys.argv[1] in ('-h', '--help'):@@ -30,20 +31,23 @@ def shapes(word):out = [x for j, x in enumerate(s) if j == 0 or x != s[j - 1]]return out or ['B']LAG = .12 # Edge's word times run ahead of the sound (join.py has the same)-from silence import speech_end # the mouth shuts where the sound stops, not .55 s after the last word starts+from silence import speech_end, silences # the mouth shuts where the sound stops, not .55 s after the last word startsres = {}for beat, e in cache.items():- words, dur, cues = e['words'], e['dur'], []+ words, dur, cues, lag = e['words'], e['dur'], [], e.get('lag', LAG) # eleven.py's heard times need no lag+ f = out / f'{beat}.mp3'+ gaps = [(a - lag, b - lag) for a, b in silences(f)] if f.exists() else [] # in word-time (a slow "anything" runs on)for j, (t, w) in enumerate(words):- end = min(words[j + 1][0] if j + 1 < len(words) else dur - .1, t + .55)+ nxt = words[j + 1][0] if j + 1 < len(words) else dur - .1+ hush = next((a for a, b in gaps if a > t + .05), None) # a word lasts until the next word or the next silence+ end = min(nxt, hush) if hush is not None else min(nxt, t + .55)sh = shapes(w); step = (end - t) / len(sh)cues += [[round(t + k * step, 3), x] for k, x in enumerate(sh)]if j + 1 < len(words) and words[j + 1][0] - end > .12: cues.append([round(end, 3), 'X']) # a pause closes the mouthcues.append([round(min(dur, (words[-1][0] + .55) if words else 0), 3), 'X'])- f = out / f'{beat}.mp3'if f.exists():- se = speech_end(f) - LAG + .05 # (LAG is added below)+ se = speech_end(f) - lag + .05 # (lag is added below)cues = [c for c in cues if c[0] < se] + [[round(se, 3), 'X']] if se < cues[-1][0] else cues- res[beat] = [[round(t + LAG, 3), k] for t, k in cues]+ res[beat] = [[round(t + lag, 3), k] for t, k in cues](out / 'mouths.js').write_text('window.MOUTHS = ' + json.dumps(res, separators=(',', ':')) + ';\n')print(len(res), 'beats;', next(iter(res.items()), None))diff --git a/.claude/skills/papermorph/scripts/silence.py b/.claude/skills/papermorph/scripts/silence.pyindex 6eb7f9f..8628dbb 100644--- a/.claude/skills/papermorph/scripts/silence.py+++ b/.claude/skills/papermorph/scripts/silence.py@@ -21,3 +21,9 @@ def speech_end(f):def pauses(f):"""The start of every pause of 0.45 s or more inside a clip."""return _detect(f, .45)[0]+++def silences(f, d=.15):+ """Every silence of d seconds or more inside a clip, as (start, end) pairs (a trailing one without an end is left out)."""+ starts, ends = _detect(f, d)+ return list(zip(starts, ends))