Skip to content

An AI skill that turns books into animated, narrated, interactive web experiences

feat(skill): ElevenLabs word cache, pause-aware word times, long-word mouths, export video option

yzr0240b16dc3b7
diff --git a/.claude/skills/papermorph/references/core/sound.md b/.claude/skills/papermorph/references/core/sound.md
index 54fe739..dda47cb 100644
--- a/.claude/skills/papermorph/references/core/sound.md
+++ b/.claude/skills/papermorph/references/core/sound.md
@@ -35,6 +35,7 @@ Voices carry every book and everything else sits under them: in early cuts music
- It prints the cost before paying (`--dry` stops there; about $0.07 a minute, quote it) and caches paid clips (a rerun costs nothing); `--only beat,beat` makes a fresh, paid take of those beats alone.
- A sped-up voice (`--tempo 1.1`, or `"tempo"` in the narration file) is sped before levelling and alignment, and stays sped on a rerun. `--max-pause 0.3` shortens its sentence pauses (0.4 to 1.0 s) before the marks are placed.
- v4 audio tags such as `[cackles]` are acted, not read, and left out of the captions and of `lint.py`'s comparison of narration and page.
+ - Every word's heard time also goes in the `.timings.json` beside the narration, as `tts.py` writes it, so a picture book's `mouths.py` and `join.py --words` work on its clips (a cast of ElevenLabs voices: *The Little Horse Crosses the River*); those times need no Edge correction, and both scripts know it.
- Its word alignment (local faster-whisper) assumes words split by spaces, so a Chinese or Japanese film needs `tts.py` for now. The transcript writes spoken numbers as digits; `eleven.py` matches them back and prints the marks a rerun moved, but check marks on numbers: a missed one lands late (up to 0.75 s).
- **Several voices.** List them under `voices` in the narration file and write each line as `{"speaker": name, "text": …}`; a line without a speaker takes the file's own `voice`, and a speaker missing from `voices` stops the run, naming it. Both scripts do this. Voices need pitches apart (a dog at 113 Hz under a 110 Hz narrator read as the same person).
- Let the user pick a voice by ear from a few short samples in the book's own words.
diff --git a/.claude/skills/papermorph/references/picture-books/stories.md b/.claude/skills/papermorph/references/picture-books/stories.md
index f9761b5..9a86bc3 100644
--- a/.claude/skills/papermorph/references/picture-books/stories.md
+++ b/.claude/skills/papermorph/references/picture-books/stories.md
@@ -92,7 +92,7 @@ Everybody in the shot stays alive and stays there: extras blink and glance about
Voices, levels, effects and music work as in [core/sound.md](../core/sound.md); a picture book adds a cast that speaks on screen and one clip per scene.
-- **Mouths** follow the words: `mouths.py` turns each line's word timings into mouth shapes (a Chinese or Japanese line opens and shuts once per character), and `lips(q, scene, pre, line…)` picks the frame's shape. Edge's word times run about a tenth of a second ahead of the sound; `mouths.py` and `join.py` move mouths and captions that much later and shut the mouth where the clip's sound stops, so a lip flap that leads or trails the voice is worth reporting.
+- **Mouths** follow the words: `mouths.py` turns each line's word timings into mouth shapes (a Chinese or Japanese line opens and shuts once per character), and `lips(q, scene, pre, line…)` picks the frame's shape. Edge's word times run about a tenth of a second ahead of the sound (`eleven.py`'s are heard times and stay put); `mouths.py` and `join.py` move Edge's mouths and captions that much later and shut the mouth where the clip's sound stops, so a lip flap that leads or trails the voice is worth reporting.
- **One clip per scene.** `join.py` joins a scene's lines into one clip with short pauses, and inserts silence where the action needs room (`line@T+S`: S seconds of silence T seconds into the line, after a word's sound has ended rather than where the next word begins) or after the last line (`name:S`, a stare or a tag). Gaps for a held stare, a fly landing, a class turning round belong here, not in rushed animation. `--words` writes every word's time in the scene clip beside the narration, for scene authors to time actions against (the beat clock adds the scene's `pre`).
- **One script for the whole sound**, `tools/<book>/audio.sh`, rerun after any change: lines replaced by hand, `mouths.py`, level fixes, `join.py` with its holes, caption edits. Join arguments typed only in a terminal have had to be read back out of `shots.js`.
- **Levels.** `level.py` runs right after `tts.py` and before `join.py`, at its default −18 LUFS ([core/sound.md](../core/sound.md#levels)). The line the joke hangs on must not be the quietest; one that must sit above the rest (a quiet old man's punchline) gets its own target (`level.py o1.mp3 --lufs -15`) and stays out of the others' list.
diff --git a/.claude/skills/papermorph/scripts/eleven.py b/.claude/skills/papermorph/scripts/eleven.py
index 6943a0f..d17ce73 100644
--- a/.claude/skills/papermorph/scripts/eleven.py
+++ b/.claude/skills/papermorph/scripts/eleven.py
@@ -18,7 +18,10 @@ language, from the file name narration.<lang>.json; the first run downloads the
matched to the heard words in order; each [[mark]] is the start of the word after it, and there is one caption cue per
sentence and per clause after a colon ("Mr." and "e.g." do not end one, as in tts.py). Whisper writes spoken numbers in
digits ("a hundred and twenty" heard as "120"): a number's first word is matched to such a heard number, so a mark on it
-lands on the number's start. A rerun prints the marks that moved more than 0.1 s against the old timings.js. Audio is decoded with ffmpeg, not PyAV (faster-whisper's PyAV version clashes). Needs ffmpeg.
+lands on the number's start. A rerun prints the marks that moved more than 0.1 s against the old timings.js. Every word's heard time also goes
+in the .timings.json beside the narration, as tts.py writes it, so a picture book's mouths.py and join.py --words work
+on these clips too. A word Whisper starts inside a pause (it often does, up to 0.7 s early) is moved to where the
+sound comes back, and the run prints it. Audio is decoded with ffmpeg, not PyAV (faster-whisper's PyAV version clashes). Needs ffmpeg.
v4 audio tags such as [cackles] or [whispers] act: they are performed, not read aloud, and left out of captions.
Price: about $0.00008 per character (about $0.65 for a ten-minute film). Before paying, the run prints the characters
@@ -50,6 +53,7 @@ import argparse, atexit, hashlib, json, pathlib, re, subprocess, sys, urllib.err
from paint import key
from ledger import append
from tts import ABBREV, split_marks, mark_times, duration, write_atomic
+from silence import silences
import level as lv
TAG = re.compile(r"(?<!\[)\[[^\[\]]+\](?!\])") # a v4 audio tag, not a [[mark]]
@@ -147,6 +151,8 @@ print(f"{sum(b[6] for b in beats)} beats to synthesize, {todo} characters, about
if a.dry:
sys.exit()
timings, spent = {}, 0
+wc_path = pathlib.Path(a.narration).with_suffix(".timings.json") # every word's heard time, for mouths.py and join.py
+wcache = json.loads(wc_path.read_text(encoding="utf-8")) if wc_path.exists() else {}
def charge():
@@ -196,6 +202,24 @@ for beat, voice, sub, clean, marks, cached, fresh in beats:
if d is not None and (k is None or d < k): k, run = d, heard[d][1]
times.append(None if k is None else heard[k][1]); j = j if k is None else k + 1
dur = duration(mp3)
+ # Whisper starts a word that follows a pause inside the pause (up to 0.7 s early, the mouth moving in silence):
+ # a word heard inside a silence starts where the sound comes back
+ gaps = [(a, b) for a, b in silences(mp3) if b < dur - .05]
+ for i, v in enumerate(times):
+ g = next(((a, b) for a, b in gaps if v is not None and a - .02 <= v < b - .02), None)
+ if g:
+ print(f"{beat}: {toks[i]!r} moved from {v:.2f} to the end of the pause, {g[1]:.2f}")
+ times[i] = round(g[1], 3)
+ # ... or puts the word after a pause before it ("river? Isn't it" heard as Isn'[email protected], pause 0.78-1.21, [email protected]):
+ # when no word starts where a pause of 0.3 s or more ends and the next word comes later than 0.2 s after it,
+ # the last word before the pause is the one spoken there
+ for g0, g1 in gaps: # (not a, b: `a` holds the arguments)
+ if g1 - g0 < .3 or any(v is not None and abs(v - g1) < .03 for v in times): continue
+ k = max((q for q, v in enumerate(times) if v is not None and v < g0), default=None)
+ n = next((v for v in times[k + 1:] if v is not None), None) if k is not None else None
+ if k is not None and k > 0 and n is not None and n > g1 + .2:
+ print(f"{beat}: {toks[k]!r} moved from {times[k]:.2f} past the pause to {g1:.2f}")
+ times[k] = round(g1, 3)
for i, v in enumerate(times): # words Whisper merged or split: halfway between their known neighbours
if v is None:
prev = next((times[q] for q in range(i - 1, -1, -1) if times[q] is not None), 0.0)
@@ -204,7 +228,7 @@ for beat, voice, sub, clean, marks, cached, fresh in beats:
words = list(zip(times, toks))
# caption cues: one per sentence or clause after a colon (long documentary sentences), at its first word
cues, i, sents = [], 0, []
- for sent in re.split(r"(?<=[.?!:])\s+", re.sub(r"\s+", " ", TAG.sub(" ", clean)).strip()): # a tag parts words
+ for sent in re.split(r"(?<=[.?!:])(?<!\.\.\.)\s+", re.sub(r"\s+", " ", TAG.sub(" ", clean)).strip()): # a tag parts words; "I... I can't" stays one line
if sents and ABBREV.search(sents[-1]): sents[-1] += " " + sent # "Mr. Lee", "e.g. a lot" go on
else: sents.append(sent)
for sent in sents:
@@ -213,10 +237,12 @@ for beat, voice, sub, clean, marks, cached, fresh in beats:
if cues: cues[0][0] = min(cues[0][0], .1)
timings[beat] = {"dur": dur, "marks": mark_times(clean, marks, words), "cues": [[0, sub]] if sub else cues}
if sub: timings[beat]["forced"] = True
+ wcache[beat] = {"dur": dur, "words": [[t, w] for t, w in words], "lag": 0} # heard times: no Edge lead to correct
moved = [f"{k} {old[beat]['marks'][k]} -> {v}" for k, v in timings[beat]["marks"].items()
if k in old.get(beat, {}).get("marks", {}) and abs(old[beat]["marks"][k] - v) > .1]
if moved: print(f"{beat}: marks moved: {', '.join(moved)}")
print(f"{beat}: {dur} s, {len(marks)} marks")
write_atomic(tf, "window.TIMINGS = " + json.dumps(timings, indent=1, ensure_ascii=False) + ";\n")
write_atomic(tk, json.dumps(takes, indent=1))
+write_atomic(wc_path, json.dumps(wcache, indent=1, ensure_ascii=False))
print(f"{len(timings)} beats in {tf}; synthesized {spent} characters, about ${spent * PRICE:.2f}")
diff --git a/.claude/skills/papermorph/scripts/export.py b/.claude/skills/papermorph/scripts/export.py
index 18eeb3c..a439d86 100644
--- a/.claude/skills/papermorph/scripts/export.py
+++ b/.claude/skills/papermorph/scripts/export.py
@@ -11,7 +11,9 @@ of their scene, as a scene change stops them) and the BGM tracks with their cros
checks and practice) are left out, and then the video ends on a card saying they are in the interactive lesson
(--link adds its address); in a film with interactive moments (CHAPTER.film, e.g. the reader fires a rifle) they stay in,
each holding its last picture for its `hold` seconds (1.5 by default, the music up at 70 % as the page plays it while
-it waits) before the next beat plays as if the reader had acted. A point-and-read beat (tapWords) holds while it shows
+it waits) before the next beat plays as if the reader had acted. A beat with `video: false` in its fourth slot is left
+out, and `video: S` ends it S seconds into its clock, sound and captions cut there, with no wait: a game that only
+works on the page (a listen-and-find stop) stays out of the video. A point-and-read beat (tapWords) holds while it shows
and says each word in turn, 2 s apart (its `hold`, if set, still wins); a word clip that is missing is named and
left silent. With --link and no question beats left out, the end
card shows the title and the link only. A mix that would clip is turned down as a whole (the run prints by how much).
@@ -122,14 +124,18 @@ async def main(a):
if errs:
sys.exit(f'page errors: {errs}')
info = await pg.evaluate("""({lang: CHAPTER.language || 'en', film: !!CHAPTER.film,
- beats: BEATS.map(b => ({id: b.id, ask: !!b.ask, pre: b.pre || 0, hold: b.hold ?? null})),
+ beats: BEATS.map(b => ({id: b.id, ask: !!b.ask, pre: b.pre || 0, hold: b.hold ?? null, video: b.video ?? null})),
bgm: typeof BGM === 'undefined' ? [] : BGM.map(m => ({src: m.a.src, beats: m.beats, gain: m.gain, fade: m.fade ?? .8})),
upInPre: window.MUSIC_UP_IN_PRE || false})""")
# a film's question beats (CHAPTER.film: the reader acts, e.g. fires) stay in the video: their picture holds `hold` s
# (BEATS[i].hold, 1.5 by default) where the page waits for the reader, then the next beat plays as if they had acted
- beats = [(i, x['id']) for i, x in enumerate(info['beats']) if not x['ask'] or info['film']]
+ beats = [(i, x['id']) for i, x in enumerate(info['beats']) if (not x['ask'] or info['film']) and x['video'] is not False]
+ # `video` in a beat's fourth slot: false leaves it out of the video; a number (seconds on its clock) ends it
+ # there, its sound and captions cut, with no wait (a point-and-read game that only works on the page)
+ skipped = [x['id'] for x in info['beats'] if x['video'] is False]
+ if skipped: print(f'left out of the video (video: false): {", ".join(skipped)}')
info['sfx'] = []
- asks = len(beats) < len(info['beats'])
+ asks = not info['film'] and any(x['ask'] for x in info['beats']) # a lesson's questions, left out: the end card says where they are
box = await pg.locator('#stage').bounding_box()
clip = {k: box[k] for k in ('x', 'y', 'width', 'height')}
audio_dir = Path(tempfile.mkdtemp(prefix='export-'))
@@ -142,7 +148,9 @@ async def main(a):
await open_beat(pg, base, i)
end, T = await pg.evaluate('[P.end, TIMINGS[BEATS[P.i].id]]')
if T.get('silent'): silent.add(bid) # { dur } in the beat's fourth slot: no clip, its sound is silence
- if info['beats'][i]['ask']: # from P.end the page waits
+ cut = info['beats'][i]['video']
+ if isinstance(cut, (int, float)) and not isinstance(cut, bool): end = min(end, cut)
+ elif info['beats'][i]['ask']: # from P.end the page waits
waits[bid], hold = end, info['beats'][i]['hold']
if demo := await pg.evaluate('P.tapDemo ? [P.tapDemo.clips, P.tapDemo.step] : null'):
words[bid] = demo
@@ -160,7 +168,7 @@ async def main(a):
[bid, count, a.fps]))
for k, c in enumerate(T['cues']): # [t, text, end?]: the page shows it at pre + t
stop = c[2] if len(c) > 2 else T['cues'][k + 1][0] if k + 1 < len(T['cues']) else T['dur']
- cues.append((offset + pre + c[0], offset + pre + stop, c[1]))
+ if pre + c[0] < end: cues.append((offset + pre + c[0], offset + pre + min(stop, end - pre), c[1]))
offset += end
total = offset + end_s
track = np.zeros((round(total * RATE) + 1, 2), np.float32)
@@ -269,6 +277,7 @@ async def main(a):
await x.close()
with open(out / f'{a.chapter}.srt', 'w', encoding='utf-8') as f:
for k, (t0, t1, line) in enumerate(cues, 1):
+ if k < len(cues): t1 = min(t1, cues[k][0]) # a cue ends where the next begins: video sites stack overlaps
f.write(f'{k}\n{srt_time(t0)} --> {srt_time(t1)}\n{line}\n\n')
if errs or enc.returncode:
part.unlink(missing_ok=True)
diff --git a/.claude/skills/papermorph/scripts/join.py b/.claude/skills/papermorph/scripts/join.py
index 9fec78b..256c4d8 100644
--- a/.claude/skills/papermorph/scripts/join.py
+++ b/.claude/skills/papermorph/scripts/join.py
@@ -77,7 +77,7 @@ for arg in args:
mine = [w for w in said if t <= w < lim]
stop = min([last] + ([nxt] + [h for h, _ in holes[b] if t < h < nxt] if nxt is not None else [])
+ [x for x in ps if t + .15 < x < lim and mine and x > max(mine) + .2])
- cues.append([round(shift(b, t) + at + LAG, 3), c, round(shift(b, stop - 1e-3) + at + .3, 3)] + [1] * bool(T[b].get('forced')))
+ cues.append([round(shift(b, t) + at + (WB.get(b) or {}).get('lag', LAG), 3), c, round(shift(b, stop - 1e-3) + at + .3, 3)] + [1] * bool(T[b].get('forced')))
marks.update({f'{b}.{k}': round(shift(b, v) + at, 3) for k, v in T[b]['marks'].items()})
at += T[b]['dur'] + sum(sec for _, sec in holes[b]) + GAP
dur = float(subprocess.run(['ffprobe', '-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', str(d / f'{shot}.mp3')], capture_output=True, text=True).stdout)
@@ -96,6 +96,6 @@ if words: # every word where it is heard in its scene clip
for b, p in v['parts'].items():
hs = ' '.join(f'[hole {round(p + sh(b, at - 1e-6), 2)}…{round(p + sh(b, at - 1e-6) + sec, 2)}]' for at, sec in H.get(b, []))
ws = W.get(b, {}).get('words', [])
- lines.append(f" {b} @{p} {hs} | " + (' '.join(f'{w}@{round(p + sh(b, t) + LAG, 2)}' for t, w in ws) or '(silence)'))
+ lines.append(f" {b} @{p} {hs} | " + (' '.join(f'{w}@{round(p + sh(b, t) + W.get(b, {}).get("lag", LAG), 2)}' for t, w in ws) or '(silence)'))
words.with_suffix('.words.txt').write_text('\n'.join(lines) + '\n', encoding='utf-8')
print({k: (v['dur'], v['parts']) for k, v in out.items()})
diff --git a/.claude/skills/papermorph/scripts/mouths.py b/.claude/skills/papermorph/scripts/mouths.py
index 3e5153a..f145342 100644
--- a/.claude/skills/papermorph/scripts/mouths.py
+++ b/.claude/skills/papermorph/scripts/mouths.py
@@ -3,14 +3,15 @@
python3 mouths.py content/<book>/chNN/narration.en.json site/<book>/chNN/audio/en
-Reads the .timings.json cache that tts.py wrote next to the narration file and writes AUDIO_DIR/mouths.js
+Reads the .timings.json cache that tts.py (or eleven.py) wrote next to the narration file and writes AUDIO_DIR/mouths.js
(window.MOUTHS = {beat: [[seconds, shape], ...]}). Shapes follow the cartoon set Rhubarb uses: A closed (m b p),
B teeth/most consonants, C open (e i), D wide (a), E round (o), F pucker (oo w u), G f/v, H l, X rest. Each word's
letters become shapes spread over the word's time; a pause between words closes the mouth. Letters are read as
English spelling; for another language the rules below are a rough guide only, and a word with no Latin letters
(Chinese, Japanese) opens and closes once per character. Edge's word times run about 0.12 s ahead of the sound, so
-every cue is moved that much later; when the clip AUDIO_DIR/<beat>.mp3 exists, the mouth shuts where its sound
-stops. Run it after tts.py (and before join.py, which moves the cues of clips it joins). Needs ffmpeg.
+every cue is moved that much later (eleven.py's times are heard ones and stay where they are); when the clip AUDIO_DIR/<beat>.mp3 exists, the mouth shuts where its sound
+stops, and a word lasts until the next word or the clip's next silence (so a drawn-out "anything" keeps the mouth
+moving). Run it after tts.py (and before join.py, which moves the cues of clips it joins). Needs ffmpeg.
"""
import json, re, subprocess, sys, pathlib
if len(sys.argv) < 3 or sys.argv[1] in ('-h', '--help'):
@@ -30,20 +31,23 @@ def shapes(word):
out = [x for j, x in enumerate(s) if j == 0 or x != s[j - 1]]
return out or ['B']
LAG = .12 # Edge's word times run ahead of the sound (join.py has the same)
-from silence import speech_end # the mouth shuts where the sound stops, not .55 s after the last word starts
+from silence import speech_end, silences # the mouth shuts where the sound stops, not .55 s after the last word starts
res = {}
for beat, e in cache.items():
- words, dur, cues = e['words'], e['dur'], []
+ words, dur, cues, lag = e['words'], e['dur'], [], e.get('lag', LAG) # eleven.py's heard times need no lag
+ f = out / f'{beat}.mp3'
+ gaps = [(a - lag, b - lag) for a, b in silences(f)] if f.exists() else [] # in word-time (a slow "anything" runs on)
for j, (t, w) in enumerate(words):
- end = min(words[j + 1][0] if j + 1 < len(words) else dur - .1, t + .55)
+ nxt = words[j + 1][0] if j + 1 < len(words) else dur - .1
+ hush = next((a for a, b in gaps if a > t + .05), None) # a word lasts until the next word or the next silence
+ end = min(nxt, hush) if hush is not None else min(nxt, t + .55)
sh = shapes(w); step = (end - t) / len(sh)
cues += [[round(t + k * step, 3), x] for k, x in enumerate(sh)]
if j + 1 < len(words) and words[j + 1][0] - end > .12: cues.append([round(end, 3), 'X']) # a pause closes the mouth
cues.append([round(min(dur, (words[-1][0] + .55) if words else 0), 3), 'X'])
- f = out / f'{beat}.mp3'
if f.exists():
- se = speech_end(f) - LAG + .05 # (LAG is added below)
+ se = speech_end(f) - lag + .05 # (lag is added below)
cues = [c for c in cues if c[0] < se] + [[round(se, 3), 'X']] if se < cues[-1][0] else cues
- res[beat] = [[round(t + LAG, 3), k] for t, k in cues]
+ res[beat] = [[round(t + lag, 3), k] for t, k in cues]
(out / 'mouths.js').write_text('window.MOUTHS = ' + json.dumps(res, separators=(',', ':')) + ';\n')
print(len(res), 'beats;', next(iter(res.items()), None))
diff --git a/.claude/skills/papermorph/scripts/silence.py b/.claude/skills/papermorph/scripts/silence.py
index 6eb7f9f..8628dbb 100644
--- a/.claude/skills/papermorph/scripts/silence.py
+++ b/.claude/skills/papermorph/scripts/silence.py
@@ -21,3 +21,9 @@ def speech_end(f):
def pauses(f):
"""The start of every pause of 0.45 s or more inside a clip."""
return _detect(f, .45)[0]
+
+
+def silences(f, d=.15):
+ """Every silence of d seconds or more inside a clip, as (start, end) pairs (a trailing one without an end is left out)."""
+ starts, ends = _detect(f, d)
+ return list(zip(starts, ends))