Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

release(v4.12.0): the talking one shot — J-cut speech handoff, a lip-sync gate that only judges speakers, a join without audio drift, two-speaker staging

mrbizarro41d6a3e94d87
diff --git a/README.md b/README.md
index 23f2d3f..4f37d02 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@
</p>
-> **Current release: v4.11.1.** Time estimates now follow your chip (an M4 Pro or M1 Max renders two to four times slower than the M4 Max the numbers were measured on; an M5 Max or M3 Ultra much faster). Control asks for a clip before it queues, a saved HiDream pick falls back to Auto instead of failing, training says no on Macs under 24 GB, and the image pre-flight frees the idle video helper before refusing a 32 GB Mac. Includes 4.11.0: **One Shot** as its own mode, LTX one shots that hold together, the planner placing a One Shot inside a film. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).
+> **Current release: v4.12.0.** A One Shot with a talking character now keeps the voice on the mouth across the joins: each part hands off on a frame where the character is still mid-word, the last word carries over the next part, every spoken part is measured for lip-sync and retaken up to twice, and the joined clip no longer drifts its sound ahead of the picture. Dialogue between two characters has a recipe that works (one face on screen at a time, a shot that carries a line frames only the speaker), written up in `docs/LTX_DIALOGUE.md`. Includes 4.11.x. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).
## Overview
diff --git a/VERSION b/VERSION
index d782fca..815588e 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-4.11.1
+4.12.0
diff --git a/docs/API.md b/docs/API.md
index 8cbf5a2..df5fc3a 100644
--- a/docs/API.md
+++ b/docs/API.md
@@ -159,6 +159,8 @@ Two optional fields, both default `on`:
|---|---|
| `take_light_lock` | the continuity sentence (`take_light_lock()` — "it is still night…") is NOT appended to the beats; `take.light_lock` is `""` and a blank beat holds with the hold sentence alone |
| `take_retake` | a part whose light drifts past `TAKE_DRIFT_MAX` is kept as rendered instead of being retaken once with a fresh seed; `take.retake` is `false` |
+| `take_camera` | optional, free text, one move for the WHOLE shot ("a slow, steady clockwise arc around the table at eye level, wide to close"). `take_camera_lock()` puts it first in part 1's prompt ("Camera: …; one steady move at one speed, never stops, never changes direction") and, in every later part, a sentence that the camera is *already* moving that way and continues in the same direction at the same speed — the handoff anchors the picture between parts, not the motion, and the seam was visible as a camera that changed direction or stopped and restarted at every join. Left blank, later parts still get the generic continuity sentence. Recorded as `take.camera` |
+| `take_handoff` | `last` (default) or `speech`. With `speech` the runner cuts every part but the last on a TALKING frame: `take_handoff_points()` finds where the line ends (audio RMS), stops the picture `TAKE_TALKING_BACK` = 0.2 s before it (the mouth still mid-word — the only anchor measured to carry the voice onto the next part's mouth: +0.54 / +0.47 against −0.15…−0.01 on a closed mouth), keeps the sound to the end of the line plus its 0.4 s decay, and mixes that tail (`part{k}_tail.wav`, `amix normalize=0`) over the head of the next part (`part{k+1}_lead.mp4`) — a J-cut, so the last word is heard whole. Write each part's first beat to OPEN IN SILENCE so the tail has room. The silent tail after each line is dropped, so the take runs shorter than its nominal length; the last part is not trimmed. Every spoken part is scored by `take_lipsync_score()` and retaken up to `TAKE_LIPSYNC_RETAKES` = 2 times with fresh seeds (ids `-pNl`, `-pNl2`) when under `TAKE_LIPSYNC_MIN` = 0.20; the best clip is kept and its score recorded in `job["take_lipsync"]`. Recorded as `take.handoff` |
`params.take` on the job: `{seconds, beats, parts, frames, part_frames,
beats_per_part, engine, beat_prompts, light_lock, retake}` — `frames` is the
diff --git a/docs/LTX_DIALOGUE.md b/docs/LTX_DIALOGUE.md
new file mode 100644
index 0000000..39f5d47
--- /dev/null
+++ b/docs/LTX_DIALOGUE.md
@@ -0,0 +1,153 @@
+# Two characters talking in LTX — what the model wants (research, 2026-09-07)
+
+Eight renders of "two aliens at dinner" in one day taught the panel a lot the hard
+way. Before rendering a ninth, this is what Lightricks' own guides, the LTX-2 model
+card, the community workflows and the open issues say about dialogue, multiple
+characters and continuity — and the recipe that follows from it.
+
+## 1. Lip-sync with several faces in frame is an open problem, not a prompt bug
+
+* Lightricks' ComfyUI issue #395 ("How to control who is speaking… when there are
+ multiple characters in the image?") describes exactly what the owner heard:
+ "Character A or B starts lip syncing randomly to parts of the dialogue… very
+ often they both appear to talk the same words simultaneously." It is **open,
+ with no maintainer answer**.
+* The community's working answers are all the same shape: **one speaking face per
+ generation** (render one character at a time and composite), masks with
+ external audio, or pre-recorded dialogue. "Insisting" in the prompt who talks
+ ("the woman in the red dress is talking, the man is silently listening") helps
+ sometimes and fails often — which is what our v2/v3 measured.
+* The audio guide's own advice: **one speaker per line, short lines**, the
+ speaker and delivery written before the quote ("she says in a calm, seasoned
+ voice: '…'"), ~9 s of speech at a natural pace, the word "lip sync" in the
+ prompt, a well-lit front-facing face, and "let the face arrive first" — a line
+ that starts on frame one gives the sync nothing to lock onto.
+
+**So:** a conversation is written so that *only the speaker's face is on screen
+while a line plays*. The panel's One Shot staging rule (a beat names only who
+should be in the picture) is the same idea; it is just the wrong container.
+
+## 2. The right container is a multi-shot generation, not a one shot
+
+LTX-2.5's headline feature is **native multi-shot**: several connected shots with
+explicit cuts inside ONE prompt and ONE generation, with the model keeping
+"character identity, environment, lighting and voice" across the cuts because it
+reasons about the whole sequence at once. Lightricks' 2.5 prompt guide:
+
+* "Use a single continuous take when you want unbroken camera motion, intimate
+ performance, or dialogue that must stay lip-synced in one framing. … When a
+ scene involves dialogue, multiple beats, or precise timing, write it in a
+ screenplay style."
+* "Prefer 2–4 shots in one generation." At every cut: **name the edit in prose**
+ ("a hard cut to…"), **re-establish the new shot** (scale, angle, who is in
+ frame, light), **re-identify returning characters with the same descriptors,
+ never a pronoun**, and **say what the sound does** ("the room tone continues
+ across the cut", "the dialogue drops; only wind remains").
+* Lock what must persist in words: "Preserve her face, red coat, blue-magenta
+ lighting… across every shot."
+
+A two-person conversation is therefore **shot / reverse shot inside one
+generation**: a medium shot of A saying its line, *a hard cut to* a medium shot of
+B at the same table in the same light answering, the room tone continuing. One
+face per shot, so nothing lip-syncs in unison; one generation, so it is one
+table, one pair, one voice each. The "two different tables, the character
+changes in the middle" of the long-table take was the cost of stitching three
+separate generations of a scene the model was never allowed to see whole.
+
+## 3. Continuity across generations, when a scene must outlast one
+
+* Community extension workflows (RuneXX, Kijai, the IAMCCS extension nodes) feed
+ the **last ~3 seconds (73 frames)** of the previous clip as context — not one
+ frame — colour-match the new segment to the old, blend the audio, keep segments
+ 10–20 s, and expect drift after ~60 s. "Incorrect slices can result in weird
+ seam timing, sudden resets, or continuity breaks."
+* The panel's One Shot hands off ONE frame. That is why every join stalls and
+ restarts (measured this morning) and why a scene drifts across parts. The
+ vendored `RetakePipeline.extend_from_video` conditions on a source clip; fed a
+ 73-frame tail, it is the community method. The earlier "windows chain" that
+ went to mush used a 121-frame tail on Q4 — a different experiment.
+* The vendored fork also carries **Prompt Relay** (`--segment TEXT [LEN]`): each
+ segment's words are gated to a slice of the timeline inside one generation.
+ It cannot stop a visible mouth from moving, but it can keep speaker B's line
+ from bleeding into speaker A's seconds. The panel does not expose it yet.
+
+## 4. The recipe for the aliens (and any two-hander)
+
+One generation per exchange, 10 s (241 frames) at High 720p on this Mac, written
+as a screenplay-style multi-shot prompt:
+
+```
+Night, a candle-lit dining room. A medium shot of a tall thin alien in a deep
+violet robe, pale grey-green skin, seated at a small round table, warm
+candlelight on its face, alone in frame. It looks up and says slowly, in a deep
+resonant voice: "The humans split the atom before they learned to share bread."
+A hard cut to a medium shot, same table, same candlelight, of a short round
+alien in a plain grey robe, darker olive skin, wide head, alone in frame; the
+room tone continues across the cut. It answers quickly, in a thin high voice:
+"Every species we ever watched said that about its neighbours." Preserve both
+aliens' faces, robes and the candlelit room across every shot. Lip sync.
+```
+
+Rules baked into it: one speaker per shot and per line; the speaker described
+before the quote, every time, with the same words; the cut named; the sound
+across the cut stated; the "preserve" sentence; "lip sync"; the face arrives
+before the line. A 30 s conversation is three such generations, joined on cuts
+(a cut hides a seam; a one shot cannot), each re-describing both aliens with the
+same descriptors so the model re-creates the same pair. If a scene must be one
+unbroken take longer than 10 s, extend with a 73-frame tail as context, not a
+frame (§3) — that path is unbuilt in the panel today.
+
+## 4b. Two faults measured on a 90 s monologue take, and their fixes
+
+* **Voice over a closed mouth** at the end of parts and on the first frame: 10–14-word
+ lines in 5 s beats. Speech runs ~2.5 words/s, so the audio overran the beat and was
+ cut at the part boundary. Fix: ≤7 words per beat, the silence after the line written,
+ a silent look before the first line (the panel's own duration rule; §4 already
+ assumed it).
+* **"Cuts blended with an overexposure effect"**: an image-to-video part drifts in
+ exposure inside itself (mean luma 137 → 178 across ten seconds) and the next part
+ re-levels one frame after its anchored first frame (178 → 157), so a boundary colour
+ match sees nothing to fix. Fix: `scripts/join_smooth.py --hold-exposure` pins every
+ frame's mean luma to the take's first second (smoothed over 12 frames) — flat at 159
+ across the same 30 s that swung 126–175 — plus the seam colour fade and a 120 ms audio
+ crossfade. Valid only when the light is constant by contract, which a one shot's light
+ lock makes true; a scene that is meant to darken must not use it.
+
+## 4c. A talking character across parts: hand off on a talking frame, gate the sync
+
+* A continuation part anchored on a **closed, silent mouth** will not lip-sync the
+ next line (measured: mouth-opening-vs-voice r −0.15 / +0.20; the community's
+ extension note says the same). Anchored on a frame from **mid-word** it syncs
+ (+0.47). One Shot's `take_handoff=speech` trims each part where its line ends
+ (`take_speech_end`, 0.4 s pad) and anchors the next part there; each part's
+ first beat opens with a silent look so the sync has a moment to lock.
+* Even a first part is a roll: identical settings gave +0.41 and −0.27 on two
+ lines. So every spoken part is **measured** (`take_lipsync_score`, the same
+ measure as `scripts/lipsync_check.py`) and **retaken once** under 0.20 with a
+ fresh seed; the better clip is kept.
+* One real line (≤18 words) per 10 s part, never a line per 5 s beat; no
+ cut/edit/shot/scene words in the dialogue; a trained face renders on Q8 Pro.
+
+## 5. Proof (rendered 2026-09-07, 10 s, High 720p, seed 4242)
+
+The §4 prompt, submitted as one ordinary text-to-video job (no take, no
+handoff): 0–5 s the tall alien alone at the candle-lit table saying its line;
+a clean hard cut at 5 s; 5–10 s the short alien alone at the same table
+answering, brightly lit, expression readable. One face per shot, one room, one
+pair — the first render of the day where nothing could speak in unison and
+nothing changed tables. `mlx_outputs/ltx_multishot_dialogue_aliens_10s_high720p.mp4`.
+
+## Sources
+
+* Lightricks, *LTX-2.5 prompt guide* — https://ltx.io/blog/ltx-2-5-prompt-guide
+* Lightricks, *Prompting guide for LTX-2* — https://ltx.io/blog/prompting-guide-for-ltx-2
+* Lightricks/LTX-2 README and the LTX-2 model card (structure, "frame count 8k+1")
+* Lightricks/ComfyUI-LTXVideo issue #395 (multi-character lip-sync, open) —
+ https://github.com/Lightricks/ComfyUI-LTXVideo/issues/395
+* RuneXX/LTX-2.3-Workflows discussions #116 (dual character) and #131 (multi-extend:
+ 3 s of context frames, colour match, 20 s groups, ~60 s before drift)
+* Kijai/LTXV2_comfy discussion #36 (extend any video: 73-frame minimum overlap)
+* fal.ai, *How to use LTX-2.5* (multi-shot: name the edit, re-establish, re-identify,
+ state the sound; ≤3 shots; let the face arrive before the line)
+* ltx23.org, *LTX 2.5 release guide* (native multi-shot, "Preserve … across every shot")
+* ltx.io/blog *How to add AI voiceovers & dialogue* (one speaker per line, short lines)
diff --git a/docs/PROMPTING.md b/docs/PROMPTING.md
index 32acf94..0e57c68 100644
--- a/docs/PROMPTING.md
+++ b/docs/PROMPTING.md
@@ -10,7 +10,9 @@ Copy this whole page into your own assistant (ChatGPT, Claude, Codex) and ask it
- **Settle before the turn.** The beat before a scene change or a reveal should slow down; the model needs a moment to land before it can pivot.
- **Repeat the invariants every beat.** The subject's look, the camera position, the palette, the sound. The engine only remembers the last frame; the words carry the rest. Phosphene prepends your style block to every beat for you.
- **Something new every second.** A near miss, an object entering, a light change. A beat that describes a mood and no event reads as a still.
+- **One short line per beat, and write the silence.** Speech runs about 2.5 words a second, so a 5-second beat holds one line of at most seven words, with a breath before it and the silence after it written out ("then he is silent, his mouth settles closed, and only the wind is heard"). A 12-word line spills past the beat and past the part boundary as voice over a closed mouth (measured 2026-09-07). Open the take with a silent look before the first line — a line on frame one gives the lip sync nothing to lock onto.
- **Lock the light.** Say the time of day and the weather once, in the first beat, and never imply another hour later ("neon comes on", "dawn breaks"). Phosphene appends a continuity sentence to every beat and measures each part; a part whose light drifts is retaken once on its own.
+- **One camera move for the whole shot, written once, in the Camera line.** Direction and speed: "a slow, steady clockwise arc around the table at eye level, wide to close". The engine carries the picture across the join between parts, never the motion — a part that starts from a still picks its own move, and the seam shows as a camera that changes direction or stops and restarts. Phosphene puts your Camera line first in every part ("the camera is already moving — … — and continues in the same direction at the same steady speed"). Keep the beats' own camera words consistent with it: no "the camera settles", "swings round" or "pulls back" halfway unless that IS the move for the whole shot. One slow, constant move reads as one shot; three moves read as three.
- **Sound is part of the shot.** Say what we hear: footsteps, a band from a doorway, rain on an awning. Without sound cues the audio is near-silent ambience.
## Start from a still
@@ -22,6 +24,11 @@ Copy this whole page into your own assistant (ChatGPT, Claude, Codex) and ask it
- No "fast forward", "time-lapse" or "montage" in a One Shot: the model cuts.
- No two subjects at once unless one of them is the point of the beat.
+- **Two characters talking: read [`docs/LTX_DIALOGUE.md`](LTX_DIALOGUE.md) first.** The model's own answer is a multi-shot generation (shot / reverse shot inside ONE prompt, one speaking face per shot, the cut named, the sound stated, the characters re-identified with the same words), not a one shot stitched from parts.
+- **The two-speaker staging that works (rendered 2026-09-07):** a long table with one speaker at each end and ONE slow lateral track along it. Speaker A alone in frame for its lines (a beat that names only A), a silent beat of empty table as A slides out (names nobody), speaker B entering alone at the far end for its answer (the first beat that names B). Never both in frame while either speaks. A profile two-shot, an orbit, or "over B's shoulder" all put two mouths in frame or add a body.
+- **A beat names only who should be in the picture.** Whoever a beat's text names is composed into that beat's frame — "only A is in the frame" does not remove a B the same text describes, and "over B's shoulder, B's back to the camera" adds a third body rather than turning B around (both measured, 2026-09-07). To have one face on screen, write a beat that mentions one character and nothing about the other; introduce the second one in the beat where it enters.
+- **Two speakers must be impossible to confuse, and only one speaks per beat.** Two look-alikes (the two aliens of the first dinner take) get their lines swapped, answer themselves, or say a line in unison — nothing in the picture tells the engine whose mouth a line belongs to. Give each speaker a different build, a different colour to wear and one vocal trait ("the tall one, deep and slow"; "the short one, thin and quick") in the first beat, repeat the pair every beat, write ONE line per beat, and say that the other one listens with its mouth closed.
+- **A character scene holds ONE human face.** A trained character's LoRA paints every face in the frame: the second person at the table comes out as the character too, whoever the prompt says they are, and two characters' LoRAs together give both people the stronger one's face (measured 2026-09-07, Q8 Pro). Put a robot, an animal, a puppet or an off-screen voice across the table — not a second person.
- No text on screen; no lettering on signs unless you accept invented words.
## Time and quality
diff --git a/docs/STATE.md b/docs/STATE.md
index 73402de..491a190 100644
--- a/docs/STATE.md
+++ b/docs/STATE.md
@@ -1,5 +1,184 @@
# Phosphene — project state, history, open work
+> **🎙️ 2026-09-07, evening — the handoff is a J-cut, and audio-first is out.**
+> Two questions from the owner. (1) *Can he keep talking through a cut?* Yes,
+> when the next part is anchored on a frame where his mouth is mid-word:
+> a head-of-part + continuation joined on such a frame scores +0.54 on the
+> mouth-vs-voice meter and reads as one uninterrupted line. It is necessary,
+> not sufficient: repeats on a second source part gave +0.15 (pause-end
+> anchor, mouth opening) and −0.09 (mid-phrase word gap), and first parts at
+> identical settings gave +0.41 / +0.25 / +0.15 / −0.15. (2) *Generate the
+> voice first and drive the picture from it?* No: A2V on Q8 Pro with the face
+> and voice models, fed a clean 10 s line of his own voice, anchored on his
+> frame and free, came out as voiceover both times (r −0.06; the mouth barely
+> moves). Shipped from that (`e8695b9`): `take_handoff=speech` now cuts the
+> PICTURE `TAKE_TALKING_BACK` = 0.2 s before the line ends and the SOUND at the
+> line end + pad, saving the tail as `part{k}_tail.wav` and mixing it
+> (`amix normalize=0`) over the head of the next part (`part{k+1}_lead.mp4`)
+> — the last word is heard whole and the next part is anchored on a talking
+> frame; the last part keeps its written silence; the lip-sync gate allows
+> `TAKE_LIPSYNC_RETAKES` = 2 fresh-seed retakes (`-pNl`, `-pNl2`) and records
+> the kept clip's score. `take_handoff_points()`; `test_take.py` 31. A2V's
+> input contract, learned the hard way: the audio must be exactly the clip's
+> length (241 f ↔ 10.0417 s) or the helper dies in `broadcast_shapes`.
+
+> **🎙️ 2026-09-07 — a talking one shot: what it took to make the mouth carry the voice.**
+> On the Bizarro monologue the owner heard every continuation part as
+> "a voiceover instead of him talking". Measured, in order: (1) 10–14-word
+> lines in 5 s beats overran the beat and the audio was cut at the part
+> boundary → one real line per 10 s part, silence written after it;
+> (2) the i2v continuation is anchored on the previous part's LAST frame,
+> after a line a closed silent mouth, and on that frame the model will not
+> lip-sync (mouth-opening-vs-voice r −0.15 / +0.20) while on a frame from
+> mid-word it does (+0.47) → `take_handoff=speech`: `take_speech_end()` finds
+> where the line stops (audio RMS, 0.4 s pad — 0.12 clipped the last
+> syllable), the part is trimmed there and the next part anchors on that
+> talking frame; (3) even a first part at identical settings came out +0.41 on
+> one line and −0.27 on the next → `take_lipsync_score()` (cv2 cascade face
+> box, dark-pixel mouth opening vs audio RMS, lags ±6; calibrated on the
+> owner's ear: fine +0.29/+0.39/+0.41, voiceover +0.17/−0.13/+0.19) gates
+> every spoken part and `TAKE_LIPSYNC_MIN = 0.20` retakes it once with a fresh
+> seed, keeping the better; `job["take_lipsync"]` records each. Also: the
+> character renders on Q8 Pro (the graded recipe; High 720p repainted the
+> face — "the quality is shit"), dialogue must not contain cut/edit/shot/scene
+> (the model rendered dissolves while he said "no cuts"), and exposure drift
+> inside i2v parts (137→178) is held flat by `scripts/join_smooth.py
+> --hold-exposure`. Tests: `test_take.py` 28. The owner's next question — generate
+> the voice first and drive the picture from it (A2V) — is queued as two
+> experiments behind the take. Camera plan for a one shot now changes INSIDE
+> parts (push-in, a sudden tilt, an orbit), never at a join.
+
+> **📚 2026-09-07 — research: how LTX wants two characters to talk (`docs/LTX_DIALOGUE.md`).**
+> The long-table one shot read as "two different tables, the character
+> changes in the middle" — three separate generations of a scene the model
+> never saw whole. Lightricks' own guides say dialogue-heavy scenes are
+> written screenplay-style as a MULTI-SHOT prompt inside one generation (2.5
+> native multi-shot): name the cut, re-establish the shot, re-identify each
+> character with the same words, state what the sound does across the cut,
+> add "Preserve … across every shot"; one speaker per line, short, the
+> speaker and delivery before the quote, "lip sync", the face before the
+> line. Multi-face lip-sync is an OPEN Lightricks issue (#395) — the field's
+> fixes are all one speaking face per generation. Continuity across
+> generations in the field uses ~3 s (73 frames) of context, colour matching
+> and audio blending, not one frame — the One Shot's handoff is the source of
+> its seams and its drift. A 10 s High-720p multi-shot proof of the recipe
+> (A's line, a hard cut, B's answer, same table) RENDERED and holds: one face
+> per shot, one room, one pair, a clean cut at 5 s —
+> `mlx_outputs/ltx_multishot_dialogue_aliens_10s_high720p.mp4`. Next: the
+> planner should write dialogue scenes as multi-shot prompts (docs/LTX_DIALOGUE.md §4).
+
+> **🎭 2026-09-07 — two speakers without a shared mouth: the staging that works, measured.**
+> The owner heard both aliens speak in unison even after they were made
+> unmistakable (look, colour, voice, one line per beat, "the other keeps its
+> mouth closed"): LTX lip-syncs EVERY mouth in frame to whatever voice plays,
+> and words do not hold a mouth still. Eight renders in a day found the rule
+> and the staging. Rule: whoever a beat names is in its picture — "only A is
+> in the frame" does not remove a B the same beat describes (v5), and
+> "over B's shoulder, back to camera" adds a THIRD body (v4). Staging (v8,
+> High 720p, `mlx_outputs/ltx_oneshot_aliens_debate_one_face_at_a_time_30s_
+> high720p.mp4`): a long table, one speaker at each end, one lateral track —
+> A alone in frame for its lines, a silent empty-table beat as A slides out,
+> B entering alone at the far end for its answer; never two mouths on screen.
+> Composition was iterated with six 5 s Balanced probes (two seeds × three
+> wordings, ~2 min each) after five 15-minute High-720p part-1 attempts —
+> the cheap loop the guide already prescribed. The dialogue became an
+> argument (the atom before bread / prayers to a flown-through sky; "every
+> species said that about its neighbours… wait for the songs"). The joins
+> still carry the speed step (slow speech parts, a fast empty-table glide) —
+> the structural seam of last-frame handoff, unchanged by this. Planner block,
+> `_sb_take_concept` and docs/PROMPTING.md carry the rule and the staging.
+
+> **🗣️ 2026-09-07 — two speakers: the lines landed on the wrong mouth.** On the
+> aliens take the owner heard the right alien answer itself and both speak a
+> line in unison: two identical speakers give the joint audio nothing to bind
+> a line to. The rule is now in the planner's One Shot block, in
+> `_sb_take_concept` and in docs/PROMPTING.md: unmistakable look AND voice
+> for each (build, the colour they wear, one vocal trait), ONE speaker per
+> beat, the other listening with its mouth closed (`test_take.py`). The take
+> was re-rendered that way (tall in violet, deep and slow; short in grey, thin
+> and quick) — `mlx_outputs/ltx_oneshot_aliens_at_dinner_v3_two_voices_30s_
+> high720p.mp4` — and the ear test is the owner's. `scripts/speaker_check.py`
+> (per-beat face motion per side during speech) reproduced the fault on the
+> first clip but measures heads, not mouths: a flag for listening, not a grade.
+
+> **👽 2026-09-07 — the aliens-at-dinner one shot, and what the H3 extender repo teaches.**
+> The Bizarro-and-robot clip was judged terrible (dark Q8 Pro pass, the robot
+> never held its cup, goggles and voice bled across characters, too much
+> hand business for an engine that is bad at physics). The replacement is
+> two plain characters and dialogue that carries the scene: two aliens at a
+> candle-lit table discussing humans in detached wonder, 30 s, **High 720p**
+> (Q8 two-stage, 1280×704 native, no upscale, no LoRAs), one slow push-in
+> as the Camera line. v1 came out too dark and too wide and was stopped after
+> part 1; v2 lit the faces and framed them medium. Its joins measure clean —
+> the camera is near-static (0.02 px/frame) on both sides, so there is nothing
+> to stop and restart — and the push-in accumulates across all three parts.
+> `mlx_outputs/ltx_oneshot_aliens_at_dinner_30s_high720p.mp4`.
+> **The owner's repo pointer** (pmhaidn/ComfyUI-Minimax-H3-Extender, GPLv3):
+> it pins the LAST 5 or 22 frames of the previous clip as a head Guide in the
+> next generation (plus tail audio), trims them, and polishes the join with a
+> one-frame optical-flow warp and a 24-frame colour seam-fade. The polish is
+> re-implemented in `scripts/join_smooth.py` (4.6 s for 30 s of 720p; proven
+> on both clips, changes nothing measurable on a multi-second stall/burst —
+> as its code predicts). The cure is the motion handoff: for H3 it is engine
+> work in the hailuo-mlx codex (the runner has first-frame chaining only);
+> for LTX the untested variant is `retake.py`'s `extend_from_video` with a
+> SHORT source (the last 9–25 frames), trimmed — the 121-frame windows chain
+> that went to mush is not the same experiment. `scripts/seam_motion.py` now
+> calls a join with (near-)static camera on both sides clean.
+
+> **📐 2026-09-07 — the seam is structural: measured before and after the camera lock.**
+> `seam_motion.py` (scratch; mean dense optical flow one second before vs one
+> second after each join) on the old 40 s bar one shot: 10 s speed ×2.3
+> (stop/restart), 20 s 86° direction change, 30 s 149° reversal — exactly
+> what the owner sees. On the new 30 s Bizarro-and-robot one shot rendered
+> WITH the camera lock in every part: 10 s ×6.3 and 96°, 20 s ×24. The lock
+> changed nothing at the joins, because the cause is not the words: every
+> 10 s LTX part decays to a standstill in its last ~3 s (0.1 → 0.02 px/frame)
+> and every i2v part lurches from rest in its first ~2 s (0.35). A still frame
+> cannot be told to start at speed. What could: hand off from a frame that is
+> still moving (overlapping parts, trimming the dead tail — at the cost of
+> beat content), slow the burst in post, or continuation with real motion
+> context at the engine level. The metric itself is worth keeping — a
+> `take_seam` beside `take_drift`, logged per join — but a retake would not
+> fix a systematic decay, so it is a report, not a trigger.
+
+> **🧪 2026-09-07 — measured: one identity LoRA paints every face in the frame.**
+> The owner asked for a viral one shot — Bizarro and a Trump-like figure at one
+> table. Three Q8 Pro renders at one seed, graded by eye: `bizarrotrn` +
+> `eltrumpo` fused together → both men have Bizarro's face; `bizarrotrn` alone
+> with the other man described in text → a second Bizarro; `eltrumpo` alone →
+> the faces stay distinct only because that adapter is weak. Weight-level
+> fusion has no spatial control in the MLX port, so a character scene holds
+> ONE human face (docs/PROMPTING.md says so now); the example became Bizarro
+> and a white robot with a visor. Not a prompt problem — do not spend Q8
+> minutes on two-face LoRA scenes.
+
+> **🎥 2026-09-07 (dev) — One Shot: the camera crosses the join.** The owner
+> can see every seam in a one shot: "either the camera suddenly changes
+> direction or it suddenly stops and starts again… if we are doing a pan left
+> or going behind his head, in the next shot we need to continue in the same
+> direction and speed." The handoff anchors the PICTURE (last frame) between
+> parts, never the MOTION — each i2v part starts from a still and picks its
+> own move. Now `take_camera` (optional, one move for the whole shot, a text
+> field in the One Shot panel) is carried into every part by
+> `take_camera_lock()`: part 1 opens on "Camera: …; one steady move at one
+> speed, never stops, never changes direction", every later part on "the same
+> continuous shot, no cut: the camera is already moving — … — and continues in
+> the same direction at the same steady speed; it does not stop, pause, restart
+> or change direction" (the first window's prompt on H3). Left blank, later
+> parts still get the generic sentence — the seam is there whether or not the
+> move was named. Sidecar `take.camera`, Load Params restores it, API.md row,
+> `test_take.py` 25. Also recorded: LTX examples render on the Q8 tier only —
+> the 40 s bar one shot at Q4 was "pretty decent, but the quality is really
+> shitty on the faces" — and the two-character experiment (bizarrotrn +
+> eltrumpo at one table, Q8 Pro, camera lock) is the first render on this code.
+
+> **🚀 2026-09-08 — v4.12.0 ships: the talking one shot.** `take_handoff=speech` as a
+> J-cut (picture cut on a talking frame, the last word mixed over the next part),
+> the lip-sync gate with two retakes that only judges parts meant to speak, the
+> join without audio drift, the two-speaker staging rules in the planner, and
+> `docs/LTX_DIALOGUE.md`. Carries 4.11.x.
+
> **🚀 2026-09-07 — v4.11.1 ships: the fleet fixes** (chip-scaled estimates,
> Control gate, HiDream fallback, training floor, image pre-flight frees the
> helper). Carries 4.11.0.
diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.py
index 0abd7f7..7189278 100644
--- a/mlx_ltx_panel.py
+++ b/mlx_ltx_panel.py
@@ -15924,6 +15924,170 @@ def take_light_lock(prompt: str) -> str:
f"of day, weather or season.")
+def take_camera_lock(camera: str, continuing: bool) -> str:
+ """One sentence that holds the CAMERA across a join.
+
+ The handoff anchors the picture (the previous part's last frame) but not
+ the motion: the next part starts from a still and picks its own move, so
+ every functional one shot showed the seam — the camera changing direction,
+ or stopping and starting again, exactly at the join (owner, 2026-09-07:
+ "if we are doing a pan left or going behind his head, in the next shot we
+ need to continue in the same direction and speed"). `camera` is the one
+ move the user wrote for the whole shot (may be empty); `continuing` is
+ True for every part after the first. With no camera written, the first
+ part gets nothing and every later part still gets the generic lock — the
+ seam is there whether or not the user named the move.
+ """
+ cam = (camera or "").strip().rstrip(".")
+ if continuing:
+ move = (f"the camera is already moving — {cam} — and continues" if cam
+ else "the camera continues the movement of the previous moment")
+ return (f"The same continuous shot, no cut: {move} in the same direction at the "
+ f"same steady speed; it does not stop, pause, restart or change direction.")
+ if not cam:
+ return ""
+ return (f"Camera: {cam}. One steady, continuous move at one speed for the whole "
+ f"shot; the camera never stops and never changes direction.")
+
+
+def take_speech_end(path, thresh: float = 0.02, pad: float = 0.4) -> float | None:
+ """Seconds into `path` where speech last stops, plus `pad`; None when the clip
+ carries no speech.
+
+ Why this exists (2026-09-07, measured): a continuation part is anchored on
+ the previous part's LAST frame, and after a spoken line that frame is a
+ closed, silent mouth — the model then will not lip-sync the next line at
+ all (voice plays as voiceover; mouth-vs-voice correlation 0.20 against 0.33
+ for a first part). Anchored on a frame where the mouth is still speaking,
+ the next part syncs (0.33). So the runner can hand off from the frame where
+ the line ends instead of the frame where the part ends. Audio RMS over 40 ms
+ blocks; the threshold is the larger of `thresh` and a quarter of the peak so
+ a quiet mix still registers. `pad` keeps the last syllable's decay: 0.12 s
+ clipped it audibly (owner: "it got cut in the middle").
+ """
+ try:
+ raw = subprocess.run([str(FFMPEG), "-loglevel", "error", "-i", str(path), "-vn", "-ac", "1",
+ "-ar", "16000", "-f", "s16le", "-"], capture_output=True, check=True, timeout=120).stdout
+ except Exception: # noqa: BLE001
+ return None
+ import array, math
+ a = array.array("h"); a.frombytes(raw[: len(raw) - len(raw) % 2])
+ blk = 640
+ if len(a) < blk:
+ return None
+ rms = [math.sqrt(sum(x * x for x in a[i:i + blk]) / blk) / 32768.0 for i in range(0, len(a) - blk + 1, blk)]
+ t = max(thresh, 0.25 * max(rms))
+ last = None
+ for i, v in enumerate(rms):
+ if v >= t:
+ last = i
+ if last is None:
+ return None
+ return (last + 1) * blk / 16000.0 + pad
+
+
+TAKE_LIPSYNC_MIN = 0.20 # mouth-opening vs voice correlation under which a spoken part is retaken
+TAKE_LIPSYNC_RETAKES = 2 # how many fresh-seed retakes a failing part gets before the best clip is kept
+TAKE_TALKING_BACK = 0.2 # seconds before the line ends where the handoff frame is taken (mouth still mid-word)
+
+
+def take_handoff_points(path, back: float = TAKE_TALKING_BACK) -> tuple[float, float] | None:
+ """(picture_cut, sound_cut) for a speech handoff, or None when the clip carries no speech.
+
+ The picture is cut `back` seconds BEFORE the line ends, on a frame where the
+ mouth is still mid-word — the next part is anchored on that frame, and only a
+ talking anchor gives it a chance to lip-sync (measured 2026-09-07: mid-word
+ anchor +0.54 and +0.47, pause-end anchor +0.15, closed mouth −0.15 to −0.01).
+ The sound is cut where the line really ends (plus the decay pad), so the last
+ word is heard whole: the runner mixes that tail over the next part's opening
+ silence — a J-cut — instead of clipping the syllable (owner: "it got cut in
+ the middle").
+ """
+ end = take_speech_end(path)
+ if end is None:
+ return None
+ pad = take_speech_end.__defaults__[1]
+ cut = max(0.0, end - pad - back)
+ return (round(cut, 3), round(end, 3))
+
+
+def take_expects_speech(params: dict, beats: list[str] | None = None) -> bool:
+ """Whether a take part is expected to carry a spoken line — the only case the
+ lip-sync gate may judge. A character job speaks with its own voice; a plain
+ job speaks when a beat quotes a line. An aerial flight has neither, and the
+ cascade face box still finds a "face" in a cityscape: on 2026-09-08 the gate
+ scored an EX ALTO flight +0.15 and retook a 47-minute part for lip-sync on a
+ shot with no mouth in it."""
+ if (params.get("character_id") or "").strip():
+ return True
+ text = " ".join([str(params.get("prompt") or "")] + [str(b) for b in (beats or [])])
+ return bool(re.search(r'[“"][^”"]{2,}[”"]', text))
+
+
+def take_lipsync_score(path) -> float | None:
+ """How well the mouth follows the voice in one clip, or None when there is
+ no face or no speech to judge (or cv2 is not importable).
+
+ The face is found once a second with OpenCV's frontal cascade; inside the
+ lower third of the face box, the fraction of pixels darker than the box's
+ own median minus 40 is how open the mouth is. That series and the audio
+ RMS are correlated over the speaking frames at lags −6..+6 frames; the
+ best value is returned. Calibrated 2026-09-07 against the owner's ear on
+ the Bizarro one shots: parts judged fine +0.29/+0.39/+0.41, parts judged
+ voiceover +0.17/−0.13/+0.19. A first part rendered at the same settings
+ came out +0.41 and −0.27 on two lines, so a part's sync is not something
+ the prompt can guarantee — it has to be measured, and retaken.
+ """
+ try:
+ import cv2 # noqa: F401
+ import numpy as np
+ except Exception: # noqa: BLE001
+ return None
+ try:
+ casc = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
+ cap = cv2.VideoCapture(str(path)); fps = cap.get(cv2.CAP_PROP_FPS) or 24.0
+ opens = []; box = None; i = 0
+ while True:
+ ok, fr = cap.read()
+ if not ok:
+ break
+ g = cv2.cvtColor(fr, cv2.COLOR_BGR2GRAY)
+ if box is None or i % int(fps) == 0:
+ f = casc.detectMultiScale(g, 1.1, 5, minSize=(80, 80))
+ if len(f):
+ box = max(f, key=lambda b: b[2] * b[3])
+ if box is None:
+ opens.append(float("nan")); i += 1; continue
+ x, y, w, h = box
+ roi = g[y + int(h * 0.62):y + h, x + int(w * 0.25):x + int(w * 0.75)]
+ opens.append(float((roi < (np.median(roi) - 40)).mean())); i += 1
+ n = len(opens)
+ if n < 48:
+ return None
+ o = np.array(opens); o = np.where(np.isnan(o), np.nanmean(o), o)
+ if not np.isfinite(o).all():
+ return None
+ raw = subprocess.run([str(FFMPEG), "-loglevel", "error", "-i", str(path), "-vn", "-ac", "1", "-ar", "16000",
+ "-f", "s16le", "-"], capture_output=True, check=True, timeout=120).stdout
+ a = np.frombuffer(raw, np.int16).astype(np.float32) / 32768.0; spf = int(16000 / fps)
+ rms = np.array([np.sqrt(np.mean(a[j * spf:(j + 1) * spf] ** 2) + 1e-12) if len(a[j * spf:(j + 1) * spf]) else 0.0
+ for j in range(n)])
+ k = np.ones(3) / 3; m = np.convolve(o, k, "same"); r = np.convolve(rms, k, "same")
+ sp = r > max(0.02, float(np.percentile(rms, 55)))
+ if sp.sum() < 24:
+ return None
+ best = None
+ for lag in range(-6, 7):
+ x_ = m[lag:][sp[:n - lag]] if lag >= 0 else m[:n + lag][sp[-lag:]]
+ y_ = r[:n - lag][sp[:n - lag]] if lag >= 0 else r[-lag:][sp[-lag:]]
+ if len(x_) >= 24 and x_.std() > 1e-6 and y_.std() > 1e-6:
+ c = float(np.corrcoef(x_, y_)[0, 1])
+ best = c if best is None else max(best, c)
+ return best
+ except Exception: # noqa: BLE001
+ return None
+
+
def take_frame_light(path, at_sec: float) -> tuple[float, float] | None:
"""(mean luma 0..1, warmth = mean R − mean B, −1..1) of one frame."""
try:
@@ -16711,7 +16875,25 @@ def _sb_take_concept(concept: str, seconds: int) -> str:
f"Name the sound in every beat. State the time of day and the weather "
f"ONCE, in the first beat, and keep them for the whole take; never let a "
f"beat imply a different hour (no 'dawn breaks', no 'neon comes on' unless "
- f"it is already dark) — the light must not change between beats.")
+ f"it is already dark) — the light must not change between beats. "
+ f"When two characters speak: give each an unmistakable look AND voice in "
+ f"the first beat (build, the colour they wear, one vocal trait each — deep "
+ f"and slow, thin and quick), let ONE of them speak per beat, and say that "
+ f"the other listens with its mouth closed; two look-alike speakers get "
+ f"their lines swapped or spoken in unison. A beat names ONLY who should "
+ f"be in the picture — whoever it names is composed into its frame, and "
+ f"'only A is in the frame' does not remove a B the same beat describes — "
+ f"so to show one speaker at a time, mention one character and nothing "
+ f"about the other, and introduce the second one in the beat where it "
+ f"enters. A spoken beat holds ONE line of at most seven words — speech "
+ f"runs about two and a half words a second and a longer line spills "
+ f"past the beat as voice over a closed mouth — with the silence after it "
+ f"WRITTEN ('then he is silent, his mouth settles closed, and only the "
+ f"wind is heard'), and the first line arrives after a silent look, never "
+ f"on the first frame. The staging that works: a long table with one speaker at each "
+ f"end and one slow lateral track along it — A alone in frame for its "
+ f"lines, a silent beat of empty table as A slides out, B entering alone "
+ f"at the far end for its answer.")
def _sb_plan_thread(board_id: str, brief: dict, previous: dict | None) -> None:
@@ -20677,6 +20859,11 @@ def make_job(form: dict[str, list[str]] | dict[str, str], *,
for b in take_beats(f("beats", ""), _take["beats"])]
_take["light_lock"] = _lock
_take["retake"] = f("take_retake", "on").strip().lower() not in ("off", "0", "false", "no")
+ # The camera, once, for the whole shot (take_camera_lock).
+ _take["camera"] = f("take_camera", "").strip()[:240]
+ # Where a part hands off: its last frame ("last", the default) or the
+ # frame where its speech ends ("speech") — see take_speech_end.
+ _take["handoff"] = ("speech" if f("take_handoff", "last").strip().lower() == "speech" else "last")
if _take["engine"] == "h3":
# A take is 15 s parts whatever the form said; without a quality
# the tier composer fell back to draft_3s and every part became
@@ -23977,6 +24164,7 @@ def run_take_job_inner(job: dict) -> None:
take_dir.mkdir(parents=True, exist_ok=True)
ff = str(FFMPEG)
outs: list[str] = []
+ tail_wav: str | None = None # speech handoff: the previous part's last word, mixed over this part's head
last_png: str | None = None
# The continuity lock: "" when the user switched it off (then blank beats
# hold with TAKE_HOLD alone and a retake does not double anything).
@@ -24003,6 +24191,13 @@ def run_take_job_inner(job: dict) -> None:
"label": f"{label or 'one shot'} · part {k + 1} of {n_parts}",
"open_when_done": False,
})
+ # THE CAMERA CROSSES THE JOIN. The lead sentence goes first in the
+ # part's prompt (the first window's, on H3): the move is already in
+ # progress at this part's first frame, same direction, same speed.
+ lead = take_camera_lock(take.get("camera") or "", continuing=(k > 0))
+ chain = list(chain)
+ if lead:
+ chain[0] = (lead + " " + (chain[0] or "")).strip()
if last_png:
cp["mode"] = "i2v"
cp["image"] = last_png
@@ -24081,6 +24276,46 @@ def run_take_job_inner(job: dict) -> None:
push(f"[take] part {k + 1}: the light drifted (luma {drift.get('luma_first')} → "
f"{drift.get('luma_last')}) — retakes are off, keeping it")
job.setdefault("take_drift", []).append(drift)
+ # LIP-SYNC GATE: a spoken part whose mouth does not follow its voice is
+ # retaken once with a fresh seed, and the better-scoring clip is kept.
+ # Measured 2026-09-07: identical settings gave +0.41 on one line and
+ # −0.27 on the next, so this is not something the words can secure.
+ ls = take_lipsync_score(out) if take_expects_speech(p, chain) else None
+ if ls is not None:
+ push(f"[take] part {k + 1}: lip-sync {ls:+.2f}" + ("" if ls >= TAKE_LIPSYNC_MIN else
+ f" — under {TAKE_LIPSYNC_MIN:.2f}, the voice is not on his mouth"))
+ attempt = 0
+ while (ls is not None and ls < TAKE_LIPSYNC_MIN and retake_allowed
+ and attempt < TAKE_LIPSYNC_RETAKES and not job.get("cancel_requested")):
+ attempt += 1
+ push(f"[take] part {k + 1}: retaking it for lip-sync with a fresh seed "
+ f"({attempt} of {TAKE_LIPSYNC_RETAKES})")
+ ls_params = _child_params(k, idxs, chain)
+ try:
+ ls_params["seed"] = str(int(ls_params.get("seed") or 0) + 211 * attempt)
+ except (TypeError, ValueError):
+ ls_params["seed"] = "-1"
+ ls_retry = {"id": f"{job['id']}-p{k + 1}l" + (str(attempt) if attempt > 1 else ""),
+ "params": ls_params, "status": "running",
+ "created_at": job.get("created_at"), "started_ts": time.time()}
+ render_part(ls_retry)
+ out3 = ls_retry.get("output_path")
+ ls2 = take_lipsync_score(out3) if out3 and Path(out3).is_file() else None
+ if ls2 is not None and ls2 > ls:
+ push(f"[take] part {k + 1}: the retake syncs better ({ls2:+.2f} vs {ls:+.2f}) — using it")
+ try:
+ set_hidden(str(out), True)
+ except Exception: # noqa: BLE001
+ pass
+ out, ls = out3, ls2
+ else:
+ push(f"[take] part {k + 1}: the retake did not sync better ({ls2}) — keeping the earlier one")
+ if out3:
+ try:
+ set_hidden(str(out3), True)
+ except Exception: # noqa: BLE001
+ pass
+ job.setdefault("take_lipsync", []).append(ls)
outs.append(out)
# Parts are working files: kept, hidden from the gallery. The one shot
# is the output.
@@ -24088,6 +24323,45 @@ def run_take_job_inner(job: dict) -> None:
set_hidden(str(out), True)
except Exception: # noqa: BLE001
pass
+ if take.get("handoff") == "speech" and tail_wav and Path(tail_wav).is_file():
+ # The previous part's last word finishes over this part's opening
+ # silence (a J-cut): its sound tail is mixed onto the head of this
+ # clip, at level, before this clip is cut or anchored.
+ led = str(take_dir / f"part{k + 1}_lead.mp4")
+ subprocess.run([ff, "-loglevel", "error", "-y", "-i", out, "-i", tail_wav,
+ "-filter_complex", "[1:a]apad[t];[0:a][t]amix=inputs=2:duration=first:normalize=0[a]",
+ "-map", "0:v", "-map", "[a]", "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", led],
+ check=True)
+ outs[-1] = led
+ out = led
+ tail_wav = None
+ if take.get("handoff") == "speech" and k + 1 < n_parts:
+ # Hand off on a TALKING frame, not where the part ends: the picture
+ # is cut a moment before the line ends (the mouth still mid-word,
+ # the only anchor measured to carry the voice onto the next part's
+ # mouth), the sound runs to the end of the line, and the silent tail
+ # after it is dropped. The last part keeps its written silence.
+ pts = take_handoff_points(out)
+ if pts is not None:
+ cut, end = pts
+ try:
+ dur = float(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
+ "-of", "csv=p=0", out], capture_output=True, text=True, timeout=30).stdout.strip())
+ except Exception: # noqa: BLE001
+ dur = 0.0
+ if dur and 0.5 < cut < end <= dur:
+ trimmed = str(take_dir / f"part{k + 1}_speech.mp4")
+ subprocess.run([ff, "-loglevel", "error", "-y", "-i", out, "-t", f"{cut:.3f}",
+ "-c:v", "libx264", "-preset", "medium", "-crf", "17", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", "-b:a", "192k", trimmed], check=True)
+ tail = str(take_dir / f"part{k + 1}_tail.wav")
+ subprocess.run([ff, "-loglevel", "error", "-y", "-ss", f"{cut:.3f}", "-to", f"{end:.3f}",
+ "-i", out, "-vn", "-c:a", "pcm_s16le", tail], check=True)
+ push(f"[take] part {k + 1}: hands off at {cut:.1f} s on a talking frame; the last word "
+ f"finishes over the next part ({dur - end:.1f} s of silent tail dropped)")
+ outs[-1] = trimmed
+ out = trimmed
+ tail_wav = tail
last_png = str(take_dir / f"part{k + 1}_last.png")
subprocess.run([ff, "-loglevel", "error", "-y", "-sseof", "-0.05", "-i", out,
"-frames:v", "1", "-update", "1", last_png], check=True)
@@ -24115,7 +24389,9 @@ def run_take_job_inner(job: dict) -> None:
"beats_per_part": take.get("beats_per_part")
or (TAKE_H3_BEATS_PER_PART if engine == "h3" else TAKE_LTX_BEATS_PER_PART),
"part_frames": take.get("part_frames"),
- "light_lock": lock, "retake": retake_allowed},
+ "light_lock": lock, "retake": retake_allowed,
+ "camera": take.get("camera") or "",
+ "handoff": take.get("handoff") or "last"},
"temporal_mode": "native", "long_mode": "native", "window_prompts": [],
})
if engine == "h3":
diff --git a/scripts/join_smooth.py b/scripts/join_smooth.py
new file mode 100644
index 0000000..824dfff
--- /dev/null
+++ b/scripts/join_smooth.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""Join one-shot parts with the two seam fixes from the H3 extender (pmhaidn,
+GPLv3 — re-implemented, not copied): a colour match at the boundary that
+cosine-fades back to the part's own colours over 24 frames, and a one-frame
+optical-flow warp that removes the first-frame hesitation and balances the
+velocity across the cut. Frames are never dropped; audio is concatenated.
+
+usage: join_smooth.py [--hold-exposure] <out.mp4> <part1.mp4> <part2.mp4> [...]
+"""
+import subprocess, sys, json, os, tempfile
+import numpy as np, cv2
+
+def _ffmpeg() -> str:
+ """The same ladder the panel climbs: LTX_FFMPEG, then PATH, then the Pinokio
+ build, then Homebrew. Never a hardcoded home directory — this script ships."""
+ import os, shutil
+ from pathlib import Path
+ for c in (os.environ.get("LTX_FFMPEG"), shutil.which("ffmpeg"),
+ str(Path.home() / "pinokio/bin/ffmpeg-env/bin/ffmpeg"),
+ "/opt/homebrew/bin/ffmpeg", "/usr/local/bin/ffmpeg"):
+ if c and Path(c).exists():
+ return c
+ return "ffmpeg"
+
+
+FF = _ffmpeg()
+FP = os.path.join(os.path.dirname(FF), "ffprobe") if os.path.dirname(FF) else "ffprobe"
+
+def probe(p):
+ j = json.loads(subprocess.run([FP, "-v", "error", "-select_streams", "v:0", "-show_entries",
+ "stream=width,height,r_frame_rate", "-of", "json", p],
+ capture_output=True, text=True, check=True).stdout)["streams"][0]
+ num, den = j["r_frame_rate"].split("/"); return int(j["width"]), int(j["height"]), float(num) / float(den)
+
+def frames(p, w, h):
+ raw = subprocess.run([FF, "-v", "error", "-i", p, "-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
+ capture_output=True, check=True).stdout
+ return np.frombuffer(raw, np.uint8).reshape(-1, h, w, 3).copy()
+
+def color_seam_fade(b, a_last, span=24):
+ """Match b's first frame to a's last frame (per-channel mean/std), fade the correction out over `span` frames."""
+ bf = b[:span].astype(np.float32) / 255.0; s = a_last.astype(np.float32) / 255.0; t = bf[0]
+ # Exposure/colour OFFSET only (per-channel mean), not a std rescale: on a high-key frame
+ # (a white salt flat) rescaling the spread clips the highlights and the mean never lands.
+ corr = bf.copy()
+ for c in range(3):
+ corr[..., c] = bf[..., c] + (s[..., c].mean() - t[..., c].mean())
+ corr = np.clip(corr, 0, 1)
+ n = bf.shape[0]; ramp = 0.5 * (1.0 + np.cos(np.linspace(0.0, np.pi, n)))
+ out = ramp[:, None, None, None] * corr + (1 - ramp[:, None, None, None]) * bf
+ b[:span] = (out * 255.0 + 0.5).astype(np.uint8); return b
+
+def gray(x): return (x[..., 0] * 0.299 + x[..., 1] * 0.587 + x[..., 2] * 0.114).astype(np.uint8)
+
+def seam_warp(a, b):
+ """The extender's one-frame fix: advance b[0] 20% toward b[1] (kills the first-frame hesitation),
+ then warp a[-1] and b[0] 15% toward each other (balances velocity across the cut)."""
+ h, w = a.shape[1:3]; x, y = np.meshgrid(np.arange(w, dtype=np.float32), np.arange(h, dtype=np.float32))
+ b0, b1 = b[0].astype(np.float32), b[1].astype(np.float32)
+ fl = cv2.calcOpticalFlowFarneback(gray(b[0]), gray(b[1]), None, 0.5, 3, 15, 3, 5, 1.2, 0)
+ adv = 0.20
+ w0 = cv2.remap(b0, x - fl[..., 0] * adv, y - fl[..., 1] * adv, cv2.INTER_LINEAR)
+ w1 = cv2.remap(b1, x + fl[..., 0] * (1 - adv), y + fl[..., 1] * (1 - adv), cv2.INTER_LINEAR)
+ b0 = (1 - adv) * w0 + adv * w1
+ al = a[-1].astype(np.float32)
+ fs = cv2.calcOpticalFlowFarneback(gray(a[-1]), gray(b0.astype(np.uint8)), None, 0.5, 3, 15, 3, 5, 1.2, 0)
+ k = 0.15
+ wa = cv2.remap(al, x - fs[..., 0] * k, y - fs[..., 1] * k, cv2.INTER_LINEAR)
+ wb = cv2.remap(b0, x + fs[..., 0] * k, y + fs[..., 1] * k, cv2.INTER_LINEAR)
+ a[-1] = np.clip(0.85 * al + 0.15 * wb, 0, 255).astype(np.uint8)
+ b[0] = np.clip(0.85 * b0 + 0.15 * wa, 0, 255).astype(np.uint8)
+ return a, b
+
+def hold_exposure(seqs, smooth=12):
+ """Pin the mean luma of every frame to the level of the take's first second.
+
+ Measured on the Bizarro one shot: part 2 brightened 137 -> 178 across its ten
+ seconds and part 3 re-levelled to 157 one frame after its anchored first frame,
+ so the join read as an exposure pop and the part as a slow overexposure. The
+ scene's light is constant by contract (the light lock), so a per-frame offset
+ toward one target, smoothed over `smooth` frames, is the honest correction.
+ """
+ means = np.concatenate([q.reshape(q.shape[0], -1).mean(axis=1) for q in seqs]).astype(np.float32)
+ target = float(np.median(means[:24]))
+ delta = target - means
+ k = np.ones(smooth, np.float32) / smooth
+ delta = np.convolve(np.pad(delta, (smooth // 2, smooth - smooth // 2 - 1), mode="edge"), k, mode="valid")
+ i = 0
+ for q in seqs:
+ n = q.shape[0]
+ d = delta[i:i + n].reshape(n, 1, 1, 1)
+ q[:] = np.clip(q.astype(np.float32) + d, 0, 255).astype(np.uint8)
+ i += n
+ return seqs
+
+
+def main():
+ hold = "--hold-exposure" in sys.argv
+ args = [a for a in sys.argv[1:] if a != "--hold-exposure"]
+ out, parts = args[0], args[1:]
+ w, h, fps = probe(parts[0])
+ seqs = [frames(p, w, h) for p in parts]
+ if hold:
+ seqs = hold_exposure(seqs)
+ for i in range(1, len(seqs)):
+ seqs[i] = color_seam_fade(seqs[i], seqs[i - 1][-1])
+ seqs[i - 1], seqs[i] = seam_warp(seqs[i - 1], seqs[i])
+ with tempfile.TemporaryDirectory() as td:
+ # audio: a butt-join with a 15 ms fade out / fade in at every seam. NOT a crossfade:
+ # acrossfade overlaps the parts and shortens the sound by its length at every seam
+ # while every video frame is kept, so the voice ran 0.12 s earlier per join —
+ # 0.6 s ahead of the mouth by the sixth part of a take (owner, 2026-09-07:
+ # "at some point the lip sync doesn't work anymore"). The fades kill the click
+ # and cost no time; the sound stays on the frame it was rendered with.
+ aud = os.path.join(td, "a.m4a"); ins = []; fc = ""
+ for i, q in enumerate(parts): ins += ["-i", q]
+ if len(parts) == 1:
+ subprocess.run([FF, "-v", "error", "-y", "-i", parts[0], "-vn", "-c:a", "aac", "-b:a", "192k", aud], check=True)
+ else:
+ durs = [float(subprocess.run([FP, "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", q],
+ capture_output=True, text=True).stdout.strip()) for q in parts]
+ for i, d in enumerate(durs):
+ fin = f"afade=t=in:st=0:d=0.015," if i > 0 else ""
+ fout = f"afade=t=out:st={max(0.0, d - 0.015):.3f}:d=0.015" if i < len(parts) - 1 else "anull"
+ fc += f"[{i}:a]{fin}{fout}[f{i}];"
+ fc += "".join(f"[f{i}]" for i in range(len(parts))) + f"concat=n={len(parts)}:v=0:a=1[aout]"
+ subprocess.run([FF, "-v", "error", "-y", *ins, "-filter_complex", fc, "-map", "[aout]",
+ "-c:a", "aac", "-b:a", "192k", aud], check=True)
+ enc = subprocess.Popen([FF, "-v", "error", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24", "-s", f"{w}x{h}", "-r", f"{fps}", "-i", "-",
+ "-i", aud, "-map", "0:v", "-map", "1:a", "-c:v", "libx264", "-preset", "medium", "-crf", "17",
+ "-pix_fmt", "yuv420p", "-c:a", "copy", "-movflags", "+faststart", "-shortest", out],
+ stdin=subprocess.PIPE, stderr=subprocess.PIPE)
+ try:
+ for s in seqs: enc.stdin.write(s.tobytes())
+ enc.stdin.close()
+ except BrokenPipeError:
+ pass
+ err = enc.stderr.read().decode(errors="ignore"); enc.wait()
+ if enc.returncode != 0:
+ raise SystemExit(f"encoder failed ({enc.returncode}): {err[-800:]}")
+ print("joined", len(parts), "parts ->", out, "| seams smoothed:", len(parts) - 1)
+
+if __name__ == "__main__": main()
diff --git a/scripts/lipsync_check.py b/scripts/lipsync_check.py
new file mode 100644
index 0000000..e81de07
--- /dev/null
+++ b/scripts/lipsync_check.py
@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+"""lipsync_check.py <mp4> [...] — lip-sync by mouth OPENING, not motion.
+Face found once per second with OpenCV's frontal cascade (full resolution); inside
+the lower third of the face box, the fraction of pixels darker than the box's own
+median minus 40 = how open the mouth is. That series, smoothed over 3 frames, is
+correlated with the audio RMS over the frames where speech is present, at lags
+-6..+6. A voice that the mouth follows: positive r near lag 0.
+
+Calibration (2026-09-07, the owner's ear as ground truth, Bizarro one shots):
+ parts judged fine r = +0.29, +0.39, +0.41
+ parts judged voiceover r = +0.17, -0.13, +0.19
+ anchor on a mid-word frame +0.47 · anchor on a closed mouth -0.15
+Gate: r >= 0.25 passes; below 0.20 fails; between, listen. The optical-flow
+variant that preceded this scored a fine part and a voiceover part the same."""
+import sys, subprocess, numpy as np, cv2
+def _ffmpeg() -> str:
+ """The same ladder the panel climbs: LTX_FFMPEG, then PATH, then the Pinokio
+ build, then Homebrew. Never a hardcoded home directory — this script ships."""
+ import os, shutil
+ from pathlib import Path
+ for c in (os.environ.get("LTX_FFMPEG"), shutil.which("ffmpeg"),
+ str(Path.home() / "pinokio/bin/ffmpeg-env/bin/ffmpeg"),
+ "/opt/homebrew/bin/ffmpeg", "/usr/local/bin/ffmpeg"):
+ if c and Path(c).exists():
+ return c
+ return "ffmpeg"
+
+
+FF = _ffmpeg()
+casc=cv2.CascadeClassifier(cv2.data.haarcascades+"haarcascade_frontalface_default.xml")
+def score(p):
+ cap=cv2.VideoCapture(p); fps=cap.get(cv2.CAP_PROP_FPS); opens=[]; box=None; i=0
+ while True:
+ ok,fr=cap.read()
+ if not ok: break
+ g=cv2.cvtColor(fr,cv2.COLOR_BGR2GRAY)
+ if box is None or i%int(fps)==0:
+ f=casc.detectMultiScale(g,1.1,5,minSize=(80,80))
+ if len(f): box=max(f,key=lambda b:b[2]*b[3])
+ if box is None: opens.append(np.nan); i+=1; continue
+ x,y,w,h=box; roi=g[y+int(h*0.62):y+h, x+int(w*0.25):x+int(w*0.75)]
+ thr=np.median(roi)-40; opens.append(float((roi<thr).mean())); i+=1
+ n=len(opens); o=np.array(opens); o=np.where(np.isnan(o),np.nanmean(o),o)
+ pcm=subprocess.run([FF,"-v","error","-i",p,"-vn","-ac","1","-ar","16000","-f","s16le","-"],capture_output=True,check=True).stdout
+ a=np.frombuffer(pcm,np.int16).astype(np.float32)/32768; spf=int(16000/fps)
+ rms=np.array([np.sqrt(np.mean(a[j*spf:(j+1)*spf]**2)+1e-12) for j in range(n)])
+ k=np.ones(3)/3; m=np.convolve(o,k,"same"); r=np.convolve(rms,k,"same"); sp=r>max(0.02,np.percentile(rms,55))
+ if sp.sum()<24: return None
+ best=(0,-9)
+ for lag in range(-6,7):
+ x=m[lag:][sp[:n-lag]] if lag>=0 else m[:n+lag][sp[-lag:]]; y=r[:n-lag][sp[:n-lag]] if lag>=0 else r[-lag:][sp[-lag:]]
+ if len(x)>=24 and x.std()>1e-6: c=float(np.corrcoef(x,y)[0,1]); best=max(best,(lag,c),key=lambda t:t[1])
+ return best, float(o[sp].mean()/max(o[~sp].mean(),1e-6))
+for p in sys.argv[1:]:
+ s=score(p)
+ v = "no face/speech" if s is None else f"best r={s[0][1]:+.2f} @ {s[0][0]:+d} open(speech)/open(quiet)={s[1]:.2f} -> " + ("SYNC" if s[0][1] >= 0.25 else ("VOICEOVER" if s[0][1] < 0.20 else "listen"))
+ print(f"{p.split('/')[-1]:45s} {v}")
diff --git a/scripts/seam_motion.py b/scripts/seam_motion.py
new file mode 100644
index 0000000..aa6a866
--- /dev/null
+++ b/scripts/seam_motion.py
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+"""Camera motion across the joins of a one shot.
+
+Dense optical flow (Farneback) on a 320-px-wide grayscale copy, averaged over
+the frame = a global motion vector per frame (camera move + subject motion;
+on a two-shot at a table the camera dominates). For each join time t we
+compare the mean vector over [t-1.0, t-0.1] s with [t+0.1, t+1.0] s:
+direction change in degrees and speed ratio after/before. A clean join is a
+small angle and a ratio near 1; the seams the owner sees are a big angle
+(direction change) or a ratio far from 1 (stop / restart). Also prints the
+motion magnitude per half-second so a stall anywhere shows.
+
+usage: seam_motion.py <mp4> <join_sec> [<join_sec> ...]
+"""
+import sys, math
+import cv2, numpy as np
+
+path = sys.argv[1]; joins = [float(x) for x in sys.argv[2:]]
+cap = cv2.VideoCapture(path)
+fps = cap.get(cv2.CAP_PROP_FPS) or 24.0
+n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+prev = None; vecs = [] # per-frame (dx, dy) in px/frame at 320-px width
+while True:
+ ok, fr = cap.read()
+ if not ok: break
+ h, w = fr.shape[:2]; sc = 320.0 / w
+ g = cv2.cvtColor(cv2.resize(fr, (320, int(h * sc))), cv2.COLOR_BGR2GRAY)
+ if prev is not None:
+ flow = cv2.calcOpticalFlowFarneback(prev, g, None, 0.5, 3, 15, 3, 5, 1.2, 0)
+ vecs.append((float(flow[..., 0].mean()), float(flow[..., 1].mean())))
+ else:
+ vecs.append((0.0, 0.0))
+ prev = g
+vecs = np.array(vecs); t = np.arange(len(vecs)) / fps
+mag = np.hypot(vecs[:, 0], vecs[:, 1])
+print(f"{path.split('/')[-1]}: {len(vecs)} frames @ {fps:.0f} fps, {len(vecs)/fps:.1f} s")
+print("motion magnitude per 0.5 s (px/frame at 320 px wide):")
+row = []
+for k in range(int(len(vecs) / fps * 2)):
+ m = mag[(t >= k / 2) & (t < (k + 1) / 2)]
+ row.append(f"{m.mean():.2f}" if len(m) else "-")
+for i in range(0, len(row), 20):
+ print(f" {i/2:5.1f}s " + " ".join(row[i:i + 20]))
+def win(a, b):
+ sel = (t >= a) & (t < b); v = vecs[sel].mean(axis=0) if sel.any() else np.zeros(2)
+ return v, float(np.hypot(*v))
+for j in joins:
+ vb, mb = win(j - 1.0, j - 0.1); va, ma = win(j + 0.1, j + 1.0)
+ ang = 0.0
+ if mb > 0.02 and ma > 0.02:
+ ang = math.degrees(math.acos(max(-1, min(1, float(np.dot(vb, va)) / (mb * ma)))))
+ ratio = (ma / mb) if mb > 1e-6 else float("inf")
+ STATIC = 0.05 # px/frame at 320 px wide: below this the camera is standing still
+ if mb < STATIC and ma < STATIC:
+ verdict = "clean (static both sides)"
+ elif mb < STATIC or ma < STATIC:
+ verdict = "stop/restart" # moving on one side, still on the other
+ else:
+ verdict = "clean" if (ang < 35 and 0.6 <= ratio <= 1.7) else ("direction change" if ang >= 35 else "stop/restart")
+ print(f"join @ {j:5.1f}s: before ({vb[0]:+.2f},{vb[1]:+.2f}) |{mb:.2f}| after ({va[0]:+.2f},{va[1]:+.2f}) |{ma:.2f}| "
+ f"angle {ang:5.1f}° speed x{ratio:.2f} -> {verdict}")
diff --git a/scripts/speaker_check.py b/scripts/speaker_check.py
new file mode 100644
index 0000000..16673ee
--- /dev/null
+++ b/scripts/speaker_check.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Who is talking, per beat, for a two-shot with one character on each side.
+
+Face motion = mean dense optical flow magnitude inside the upper-middle band
+of the LEFT half and of the RIGHT half of the frame (where the two heads sit
+in a seated two-shot); speech = audio RMS. Per 5-second beat: the two face
+motions during the frames where speech is present, their ratio, and a
+verdict — "left", "right", or "both" (ratio under 1.4 = the two faces move
+alike while someone speaks: unison, or the wrong mouth). Crude, but it turns
+"sometimes they both talk" into a table.
+
+usage: speaker_check.py <mp4> [beat_seconds=5]
+
+Calibration (2026-09-07): on the first aliens take it reproduced what the
+owner heard (beat 3 "right" when the script said left; beats 4-6 "both").
+It is a heuristic: it measures the whole head, not the mouth, so a nod or
+the push-in on the larger head reads as speech. Use it to flag a beat for
+listening, not to grade a clip on its own.
+"""
+import sys, subprocess, json
+import numpy as np, cv2
+
+def _ffmpeg() -> str:
+ """The same ladder the panel climbs: LTX_FFMPEG, then PATH, then the Pinokio
+ build, then Homebrew. Never a hardcoded home directory — this script ships."""
+ import os, shutil
+ from pathlib import Path
+ for c in (os.environ.get("LTX_FFMPEG"), shutil.which("ffmpeg"),
+ str(Path.home() / "pinokio/bin/ffmpeg-env/bin/ffmpeg"),
+ "/opt/homebrew/bin/ffmpeg", "/usr/local/bin/ffmpeg"):
+ if c and Path(c).exists():
+ return c
+ return "ffmpeg"
+
+
+FF = _ffmpeg()
+path = sys.argv[1]; beat = float(sys.argv[2]) if len(sys.argv) > 2 else 5.0
+cap = cv2.VideoCapture(path); fps = cap.get(cv2.CAP_PROP_FPS) or 24.0
+prev = None; left = []; right = []
+while True:
+ ok, fr = cap.read()
+ if not ok: break
+ h, w = fr.shape[:2]; sc = 320.0 / w
+ g = cv2.cvtColor(cv2.resize(fr, (320, int(h * sc))), cv2.COLOR_BGR2GRAY)
+ if prev is not None:
+ fl = cv2.calcOpticalFlowFarneback(prev, g, None, 0.5, 3, 15, 3, 5, 1.2, 0)
+ mag = np.hypot(fl[..., 0], fl[..., 1]); H, W = mag.shape
+ band = mag[int(H * 0.15):int(H * 0.65)] # heads, not the table
+ left.append(float(band[:, :W // 2].mean())); right.append(float(band[:, W // 2:].mean()))
+ else:
+ left.append(0.0); right.append(0.0)
+ prev = g
+left = np.array(left); right = np.array(right); n = len(left); t = np.arange(n) / fps
+# speech energy: mono 16 kHz PCM, RMS per video frame
+pcm = subprocess.run([FF, "-v", "error", "-i", path, "-vn", "-ac", "1", "-ar", "16000", "-f", "s16le", "-"],
+ capture_output=True, check=True).stdout
+a = np.frombuffer(pcm, np.int16).astype(np.float32) / 32768.0
+spf = int(16000 / fps); rms = np.array([np.sqrt(np.mean(a[i * spf:(i + 1) * spf] ** 2) + 1e-12) for i in range(n)])
+thr = max(0.02, np.percentile(rms, 60)) # "speech present" = louder than the quiet 60 %
+print(f"{path.split('/')[-1]}: {n} frames, speech threshold rms {thr:.3f}")
+print(f"{'beat':>4} {'span':>11} {'speech%':>7} {'L-face':>7} {'R-face':>7} {'ratio':>6} verdict")
+for k in range(int(np.ceil(n / fps / beat))):
+ sel = (t >= k * beat) & (t < (k + 1) * beat); sp = sel & (rms > thr)
+ if sp.sum() < 3:
+ print(f"{k + 1:>4} {k * beat:5.1f}-{(k + 1) * beat:5.1f} {'-':>7} (no speech)"); continue
+ L, R = left[sp].mean(), right[sp].mean(); ratio = max(L, R) / max(min(L, R), 1e-6)
+ who = "both" if ratio < 1.4 else ("left" if L > R else "right")
+ print(f"{k + 1:>4} {k * beat:5.1f}-{(k + 1) * beat:5.1f} {100 * sp.sum() / sel.sum():6.0f}% {L:7.3f} {R:7.3f} {ratio:6.2f} {who}")
diff --git a/storyboard_planner.py b/storyboard_planner.py
index 6aa5107..756d270 100644
--- a/storyboard_planner.py
+++ b/storyboard_planner.py
@@ -270,6 +270,11 @@ L6 DIALOGUE lives inside <d>[English] ...</d> and nowhere else. The speaker, th
and their delivery are described OUTSIDE the tag. Immediately after the tag, stop the
mouth: "his jaw ceases speaking motion and his mouth settles closed". One or two short
sentences per 5 s. If a shot has no dialogue, it has no <d> at all.
+ A SHOT THAT CARRIES A LINE FRAMES ONLY THE SPEAKER. The engine moves every mouth in
+ frame to whatever voice plays, so a listener in the same frame mouths the line too or
+ speaks it in unison. Give the listener its own shot (a reaction, a cutaway); never two
+ faces in frame while a line plays, and never ask for the listener's back - that adds a
+ third body.
L7 MUSIC IS INSTRUMENTATION, not mood. "Sparse piano at a slow tempo, joined by low
strings that swell and cut out" - never "epic", "emotional", "uplifting". "N/A" is the
correct value when there is no score, and is the right answer most of the time.
@@ -457,6 +462,22 @@ How to write one. The shot object keeps every key above and adds exactly two:
in the first beat, and never change them - no 'dawn breaks', no lights
coming on. Every beat starts exactly where the one before it ends: no
new angle, no jump in time.
+ When TWO characters speak: give each an unmistakable look AND voice in
+ the first beat (build, robe or jacket colour, one vocal trait each - deep
+ and slow, thin and quick), let ONE of them speak per beat, and say that
+ the other listens with its mouth closed. Two look-alike speakers make
+ the engine give a line to the wrong mouth, or to both at once.
+ A beat names ONLY who should be in the picture: whoever a beat names
+ is composed into its frame, "only A is in the frame" does not remove a
+ B the same beat describes, and asking for B's back adds a third body.
+ To show one speaker at a time, write a beat that mentions one
+ character and nothing about the other, and introduce the second one
+ in the beat where it enters.
+ The staging that works for two speakers: a long table with one at
+ each end and one slow lateral track along it - speaker A alone in
+ frame for its lines, a silent beat of empty table as A slides out,
+ speaker B entering alone at the far end for its answer. Never both
+ in frame while either speaks.
"description" is the whole take in one paragraph; "duration_s" equals "take_seconds".
Every other shot in the film has NEITHER key.
diff --git a/test_storyboard_planner.py b/test_storyboard_planner.py
index 46dc892..cec2d12 100644
--- a/test_storyboard_planner.py
+++ b/test_storyboard_planner.py
@@ -1883,3 +1883,13 @@ class TestGeographyParser(unittest.TestCase):
if __name__ == "__main__":
unittest.main(verbosity=2)
+
+
+def test_planner_rules_frame_only_the_speaker_while_a_line_plays():
+ # Measured 2026-09-07: two faces in frame during a line -> both mouths move (unison), or
+ # the wrong mouth; "the listener's back to the camera" added a third body.
+ text = P.RULES if hasattr(P, "RULES") else "\n".join(
+ v for k, v in vars(P).items() if isinstance(v, str) and "DIALOGUE lives inside" in v)
+ assert "A SHOT THAT CARRIES A LINE FRAMES ONLY THE SPEAKER" in text
+ assert "never two" in text and "faces in frame while a line plays" in text
+ assert "never ask for the listener's back" in text
diff --git a/test_take.py b/test_take.py
index fd38100..b9afc93 100644
--- a/test_take.py
+++ b/test_take.py
@@ -161,7 +161,10 @@ def test_take_runner_chains_parts_and_joins(tmp_path, monkeypatch):
p.run_take_job_inner(j)
assert len(seen) == 2
assert seen[0]["mode"] == "t2v" and [c.split(" Continuity")[0] for c in seen[0]["h3_chain_prompts"]] == ["b1", "b2", "b3"]
- assert seen[1]["mode"] == "i2v" and [c.split(" Continuity")[0] for c in seen[1]["h3_chain_prompts"]] == ["b4", "b5", "b6"]
+ # part 2's first window opens on the camera lock (the join), then its beat
+ assert seen[1]["mode"] == "i2v"
+ assert seen[1]["h3_chain_prompts"][0].startswith(p.take_camera_lock("", True))
+ assert [c.split(" Continuity")[0].split(" ")[-1] for c in seen[1]["h3_chain_prompts"]] == ["b4", "b5", "b6"]
assert Path(seen[1]["image"]).is_file() # the last frame of part 1
assert seen[1]["take"] is None # a part is not a take
final = Path(j["output_path"])
@@ -220,10 +223,13 @@ def test_ltx_take_runner_chains_parts_by_last_frame_handoff(tmp_path, monkeypatc
for k in (1, 2, 3, 4):
assert seen[k]["mode"] == "i2v" and seen[k]["i2v_reference_mode"] == "anchor"
assert Path(seen[k]["image"]).is_file() and seen[k]["image"].endswith(f"part{k}_last.png")
- assert seen[1]["prompt"].split(" Continuity")[0] == "b3"
+ # every later part opens on the camera lock, then its own beats
+ assert seen[1]["prompt"].startswith(p.take_camera_lock("", True))
+ assert seen[1]["prompt"].split(" Continuity")[0].endswith(" b3")
assert p.TAKE_HOLD in seen[1]["prompt"] # the blank 4th beat holds
assert seen[4]["frames"] == 121 # the one-beat tail
- assert seen[4]["prompt"].split(" Continuity")[0] == "b9"
+ assert seen[4]["prompt"].startswith(p.take_camera_lock("", True))
+ assert seen[4]["prompt"].split(" Continuity")[0].endswith(" b9")
for c in seen: # never the windows chain
assert c["long_mode"] == "native" and c["temporal_mode"] == "native"
assert c["window_prompts"] == [] and c["h3_chain_prompts"] == []
@@ -335,6 +341,13 @@ def test_take_shot_is_priced_by_parts_and_validates():
def test_take_concept_asks_for_beats_that_never_cut():
c = p._sb_take_concept("a hen skateboards through the city", 60)
assert "12 beats" in c and "never cuts" in c and "a hen skateboards" in c
+ # two speakers: unmistakable, one per beat, the other silent (the aliens' unison lines)
+ assert "ONE of them speak per beat" in c and "mouth closed" in c and "unmistakable look AND voice" in c
+ # whoever a beat names is in its picture (the long-table takes, 2026-09-07)
+ assert "names ONLY who should be in the picture" in c and "introduce the second one in the beat where it enters" in c
+ assert "long table with one speaker at each end" in c and "B entering alone" in c
+ # the word budget and the written silence (voice over a closed mouth, 2026-09-07)
+ assert "at most seven words" in c and "silence after it" in c and "silent look" in c
# ---- continuity: the light lock and the drift retake ---------------------------
@@ -392,3 +405,223 @@ def test_take_runner_retakes_a_drifting_part_once_and_keeps_the_steadier(tmp_pat
assert side["take"]["parts"][0].endswith("-p1r.mp4") # the steadier retake is part 1
assert str(out_dir / f"{j['id']}-p1.mp4") in hidden # the drifting one is hidden
assert j["take_drift"][0]["drifted"] is True
+
+
+def test_camera_lock_holds_direction_and_speed_across_the_join():
+ # No camera written: part 1 gets nothing, every later part the generic lock.
+ assert p.take_camera_lock("", False) == ""
+ generic = p.take_camera_lock("", True)
+ assert generic.startswith("The same continuous shot, no cut:")
+ assert "continues the movement of the previous moment" in generic
+ assert "same direction at the same steady speed" in generic and "change direction" in generic
+ # A camera written once is carried into every part, first and later.
+ first = p.take_camera_lock("a slow clockwise arc around the table.", False)
+ assert first.startswith("Camera: a slow clockwise arc around the table.")
+ assert "never stops and never changes direction" in first
+ later = p.take_camera_lock("a slow clockwise arc around the table", True)
+ assert "already moving — a slow clockwise arc around the table — and continues" in later
+
+
+def test_ltx_take_parts_open_on_the_camera_lock(tmp_path, monkeypatch):
+ out_dir = tmp_path / "out"; out_dir.mkdir()
+ monkeypatch.setattr(p, "OUTPUT", out_dir)
+ monkeypatch.setattr(p, "STATE_DIR", tmp_path / "state")
+ monkeypatch.setattr(p, "set_hidden", lambda *a: None)
+ monkeypatch.setattr(p, "take_drift", lambda *a, **k: {"ok": True, "delta": 0.0, "drifted": False})
+ seen = []
+
+ def fake_ltx(child):
+ seen.append(dict(child["params"]))
+ out = out_dir / f"{child['id']}.mp4"; _tiny_clip(out, "red")
+ (out_dir / f"{child['id']}.mp4.json").write_text(json.dumps({"image": child["params"].get("image")}))
+ child["output_path"] = str(out)
+ monkeypatch.setattr(p, "run_job_inner", fake_ltx)
+ cam = "a slow, steady clockwise arc around the table at eye level"
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "two men at a table", "take_seconds": "30",
+ "beats": json.dumps(["b1", "b2", "b3", "b4", "b5", "b6"]), "take_camera": cam})
+ assert j["params"]["take"]["camera"] == cam
+ p.run_take_job_inner(j)
+ assert seen[0]["prompt"].startswith(f"Camera: {cam}.") # part 1: the move, named once
+ assert " b1" in seen[0]["prompt"] and " b2" in seen[0]["prompt"]
+ for c in seen[1:]: # every later part: already moving
+ assert c["prompt"].startswith("The same continuous shot, no cut: the camera is already moving — " + cam)
+ side = json.loads(Path(j["output_path"]).with_suffix(".mp4.json").read_text())
+ assert side["take"]["camera"] == cam
+
+
+def test_h3_take_first_window_of_each_part_carries_the_camera_lock(tmp_path, monkeypatch):
+ out_dir = tmp_path / "out"; out_dir.mkdir()
+ monkeypatch.setattr(p, "OUTPUT", out_dir)
+ monkeypatch.setattr(p, "STATE_DIR", tmp_path / "state")
+ monkeypatch.setattr(p, "set_hidden", lambda *a: None)
+ monkeypatch.setattr(p, "take_drift", lambda *a, **k: {"ok": True, "delta": 0.0, "drifted": False})
+ seen = []
+
+ def fake_h3(child):
+ seen.append(dict(child["params"]))
+ out = out_dir / f"{child['id']}.mp4"; _tiny_clip(out, "red")
+ (out_dir / f"{child['id']}.mp4.json").write_text("{}")
+ child["output_path"] = str(out)
+ monkeypatch.setattr(p, "run_h3_job_inner", fake_h3)
+ j = p.make_job({"mode": "t2v", "engine": "h3", "prompt": "x", "take_seconds": "30",
+ "beats": json.dumps(["b1", "b2", "b3", "b4", "b5", "b6"])})
+ p.run_take_job_inner(j)
+ assert len(seen) == 2
+ assert not seen[0]["h3_chain_prompts"][0].startswith("The same continuous shot") # no camera: part 1 as written
+ assert seen[1]["h3_chain_prompts"][0].startswith("The same continuous shot, no cut: the camera continues")
+ assert seen[1]["h3_chain_prompts"][1].split(" Continuity")[0] == "b5" # later windows untouched
+
+
+def test_speech_end_finds_where_the_line_stops(tmp_path):
+ # 3 s clip: a 1 kHz tone for the first 1.4 s, then silence. Speech ends at ~1.4 s (+ pad).
+ out = tmp_path / "tone.mp4"
+ subprocess.run([str(p.FFMPEG), "-loglevel", "error", "-y",
+ "-f", "lavfi", "-i", "color=c=gray:s=64x64:r=24:d=3",
+ "-f", "lavfi", "-i", "sine=frequency=1000:duration=1.4",
+ "-filter_complex", "[1:a]apad=whole_dur=3[a]", "-map", "0:v", "-map", "[a]",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-shortest", str(out)], check=True)
+ end = p.take_speech_end(out)
+ assert end is not None and 1.6 <= end <= 2.1, end
+ silent = tmp_path / "silent.mp4"
+ subprocess.run([str(p.FFMPEG), "-loglevel", "error", "-y", "-f", "lavfi", "-i", "color=c=gray:s=64x64:r=24:d=2",
+ "-f", "lavfi", "-i", "anullsrc=r=16000:cl=mono", "-t", "2", "-c:v", "libx264", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", "-shortest", str(silent)], check=True)
+ assert p.take_speech_end(silent) is None
+
+
+def test_make_job_records_the_handoff_mode():
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "x", "take_seconds": "30", "take_handoff": "speech"})
+ assert j["params"]["take"]["handoff"] == "speech"
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "x", "take_seconds": "30"})
+ assert j["params"]["take"]["handoff"] == "last"
+
+
+def test_take_runner_retakes_a_part_whose_mouth_does_not_follow_its_voice(tmp_path, monkeypatch):
+ out_dir = tmp_path / "out"; out_dir.mkdir()
+ monkeypatch.setattr(p, "OUTPUT", out_dir)
+ monkeypatch.setattr(p, "STATE_DIR", tmp_path / "state")
+ monkeypatch.setattr(p, "set_hidden", lambda *a: None)
+ monkeypatch.setattr(p, "take_drift", lambda *a, **k: {"ok": True, "delta": 0.0, "drifted": False})
+ scores = iter([-0.2, 0.4, 0.5, 0.5]) # part 1 fails then its retake passes; parts 2, 3 pass
+ monkeypatch.setattr(p, "take_lipsync_score", lambda path: next(scores))
+ seen = []
+
+ def fake_ltx(child):
+ seen.append((child["id"], child["params"].get("seed")))
+ out = out_dir / f"{child['id']}.mp4"; _tiny_clip(out, "red")
+ (out_dir / f"{child['id']}.mp4.json").write_text("{}")
+ child["output_path"] = str(out)
+ monkeypatch.setattr(p, "run_job_inner", fake_ltx)
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": 'He says: "Hello."', "take_seconds": "30", "seed": "7"})
+ p.run_take_job_inner(j)
+ ids = [i for i, _ in seen]
+ assert ids == [f"{j['id']}-p1", f"{j['id']}-p1l", f"{j['id']}-p2", f"{j['id']}-p3"]
+ assert seen[1][1] == str(7 + 211) # a fresh seed for the retake
+ side = json.loads(Path(j["output_path"]).with_suffix(".mp4.json").read_text())
+ assert side["take"]["parts"][0].endswith("-p1l.mp4") # the better-syncing clip is the one kept
+ assert j["take_lipsync"] == [0.4, 0.5, 0.5] # the score of the clip that was kept
+
+
+def _tone_clip(path: Path, seconds: float = 3.0, tone: float = 1.4) -> None:
+ """`seconds` of grey picture with a 1 kHz tone for the first `tone` seconds, then silence."""
+ subprocess.run([str(p.FFMPEG), "-loglevel", "error", "-y",
+ "-f", "lavfi", "-i", f"color=c=gray:s=64x64:r=24:d={seconds}",
+ "-f", "lavfi", "-i", f"sine=frequency=1000:duration={tone}",
+ "-filter_complex", f"[1:a]apad=whole_dur={seconds}[a]", "-map", "0:v", "-map", "[a]",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-shortest", str(path)], check=True)
+
+
+def _dur(path) -> float:
+ return float(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0",
+ str(path)], capture_output=True, text=True).stdout.strip())
+
+
+def test_handoff_points_cut_the_picture_before_the_line_ends_and_the_sound_after(tmp_path):
+ out = tmp_path / "tone.mp4"; _tone_clip(out)
+ cut, end = p.take_handoff_points(out)
+ assert 1.6 <= end <= 2.1, end # the line ends at ~1.4 s + the decay pad
+ assert abs((end - cut) - (0.4 + p.TAKE_TALKING_BACK)) < 0.05 # the picture is cut on a talking frame
+ assert cut < 1.4 < end # ...while the tone is still sounding
+ silent = tmp_path / "silent.mp4"
+ subprocess.run([str(p.FFMPEG), "-loglevel", "error", "-y", "-f", "lavfi", "-i", "color=c=gray:s=64x64:r=24:d=2",
+ "-f", "lavfi", "-i", "anullsrc=r=16000:cl=mono", "-t", "2", "-c:v", "libx264", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", "-shortest", str(silent)], check=True)
+ assert p.take_handoff_points(silent) is None
+
+
+def test_ltx_take_speech_handoff_is_a_j_cut(tmp_path, monkeypatch):
+ """Every part but the last is cut on a talking frame; its last word is carried over the next part."""
+ out_dir = tmp_path / "out"; out_dir.mkdir()
+ monkeypatch.setattr(p, "OUTPUT", out_dir)
+ monkeypatch.setattr(p, "STATE_DIR", tmp_path / "state")
+ monkeypatch.setattr(p, "set_hidden", lambda *a: None)
+ monkeypatch.setattr(p, "take_drift", lambda *a, **k: {"ok": True, "delta": 0.0, "drifted": False})
+ monkeypatch.setattr(p, "take_lipsync_score", lambda path: 0.5)
+
+ def fake_ltx(child):
+ out = out_dir / f"{child['id']}.mp4"; _tone_clip(out)
+ (out_dir / f"{child['id']}.mp4.json").write_text("{}")
+ child["output_path"] = str(out)
+ monkeypatch.setattr(p, "run_job_inner", fake_ltx)
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "x", "take_seconds": "30", "take_handoff": "speech"})
+ p.run_take_job_inner(j)
+ take_dir = tmp_path / "state" / "take" / j["id"]
+ side = json.loads(Path(j["output_path"]).with_suffix(".mp4.json").read_text())
+ parts = side["take"]["parts"]
+ assert [Path(x).name for x in parts] == ["part1_speech.mp4", "part2_speech.mp4", "part3_lead.mp4"]
+ cut, end = p.take_handoff_points(out_dir / f"{j['id']}-p1.mp4")
+ assert abs(_dur(parts[0]) - cut) < 0.1 # picture stops on the talking frame
+ assert abs(_dur(take_dir / "part1_tail.wav") - (end - cut)) < 0.1 # the rest of the word travels on
+ assert abs(_dur(parts[2]) - 3.0) < 0.1 # the last part keeps its silence
+ assert _dur(j["output_path"]) < 3 * 3.0 # the silent tails of parts 1–2 are gone
+
+
+def test_take_runner_gives_a_failing_part_two_retakes_and_keeps_the_best(tmp_path, monkeypatch):
+ out_dir = tmp_path / "out"; out_dir.mkdir()
+ monkeypatch.setattr(p, "OUTPUT", out_dir)
+ monkeypatch.setattr(p, "STATE_DIR", tmp_path / "state")
+ monkeypatch.setattr(p, "set_hidden", lambda *a: None)
+ monkeypatch.setattr(p, "take_drift", lambda *a, **k: {"ok": True, "delta": 0.0, "drifted": False})
+ scores = iter([-0.2, 0.1, 0.05, 0.5, 0.5]) # part 1 fails twice more; the middle attempt is the best
+ monkeypatch.setattr(p, "take_lipsync_score", lambda path: next(scores))
+ seen = []
+
+ def fake_ltx(child):
+ seen.append((child["id"], child["params"].get("seed")))
+ out = out_dir / f"{child['id']}.mp4"; _tiny_clip(out, "red")
+ (out_dir / f"{child['id']}.mp4.json").write_text("{}")
+ child["output_path"] = str(out)
+ monkeypatch.setattr(p, "run_job_inner", fake_ltx)
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": 'He says: "Hello."', "take_seconds": "30", "seed": "7"})
+ p.run_take_job_inner(j)
+ assert [i for i, _ in seen] == [f"{j['id']}-p1", f"{j['id']}-p1l", f"{j['id']}-p1l2", f"{j['id']}-p2", f"{j['id']}-p3"]
+ assert [s for _, s in seen][:3] == ["7", str(7 + 211), str(7 + 422)]
+ side = json.loads(Path(j["output_path"]).with_suffix(".mp4.json").read_text())
+ assert side["take"]["parts"][0].endswith("-p1l.mp4") # 0.1 beat −0.2 and 0.05
+ assert j["take_lipsync"] == [0.1, 0.5, 0.5]
+
+
+def test_lipsync_gate_only_judges_parts_that_speak(tmp_path, monkeypatch):
+ """A flight with no character and no quoted line is never scored or retaken for lip-sync."""
+ assert p.take_expects_speech({"character_id": "bizarrotrn", "prompt": "x"}) is True
+ assert p.take_expects_speech({"prompt": 'He says: "Hello."'}) is True
+ assert p.take_expects_speech({"prompt": "aerial flight"}, ['He looks up and says, deadpan: "No."']) is True
+ assert p.take_expects_speech({"prompt": "dr0nesh0t aerial over the city"}, ["The roof slides away beneath."]) is False
+ out_dir = tmp_path / "out"; out_dir.mkdir()
+ monkeypatch.setattr(p, "OUTPUT", out_dir)
+ monkeypatch.setattr(p, "STATE_DIR", tmp_path / "state")
+ monkeypatch.setattr(p, "set_hidden", lambda *a: None)
+ monkeypatch.setattr(p, "take_drift", lambda *a, **k: {"ok": True, "delta": 0.0, "drifted": False})
+ monkeypatch.setattr(p, "take_lipsync_score", lambda path: -0.9) # would fail every part if consulted
+ seen = []
+
+ def fake_ltx(child):
+ seen.append(child["id"])
+ out = out_dir / f"{child['id']}.mp4"; _tiny_clip(out, "red")
+ (out_dir / f"{child['id']}.mp4.json").write_text("{}")
+ child["output_path"] = str(out)
+ monkeypatch.setattr(p, "run_job_inner", fake_ltx)
+ j = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "aerial flight over the city", "take_seconds": "30", "seed": "7"})
+ p.run_take_job_inner(j)
+ assert seen == [f"{j['id']}-p1", f"{j['id']}-p2", f"{j['id']}-p3"] # no -p1l: nothing was retaken
+ assert j["take_lipsync"] == [None, None, None]
diff --git a/webapp/index.html b/webapp/index.html
index e64bbcd..77cd484 100644
--- a/webapp/index.html
+++ b/webapp/index.html
@@ -1153,6 +1153,19 @@
oninput="beatsInput()"
placeholder="she pushes off down the sunlit avenue&#10;a van sweeps past, pigeons burst up ahead&#10;she drops off the kerb into the road&#10;…"></textarea>
</div>
+ <!-- The camera, once, for the whole shot. The handoff anchors the
+ picture between parts but not the motion; this line is carried
+ into every part's prompt so the move continues in the same
+ direction at the same speed across the join (take_camera_lock).
+ Left blank, later parts still get the generic continuity line. -->
+ <div class="oneshot-camera" id="oneshotCamera">
+ <div class="qs-label h3-axis-label">
+ <span class="qs-name">Camera</span>
+ <span class="qs-meta">optional · one move for the whole shot, its direction and speed — every part continues it</span>
+ </div>
+ <input type="text" name="take_camera" id="take_camera" class="sb-input" maxlength="240" autocomplete="off" spellcheck="false"
+ placeholder="a slow, steady clockwise arc around the table at eye level, wide to close">
+ </div>
<!-- Start frame (optional). The SAME hidden `image` field as
Image mode, through the same /upload path (pickerUploadFile),
so when it is set the hidden #mode becomes i2v and the shot
diff --git a/webapp/js/queue.js b/webapp/js/queue.js
index 86dd441..8390aec 100644
--- a/webapp/js/queue.js
+++ b/webapp/js/queue.js
@@ -3633,6 +3633,9 @@ async function loadParams() {
if (typeof setTakeRetake === 'function') {
setTakeRetake(p.take.retake === false ? 'off' : 'on');
}
+ // The camera line, as written once for the whole shot (take.camera).
+ const _camEl = document.getElementById('take_camera');
+ if (_camEl) _camEl.value = p.take.camera || '';
} else if (typeof setTakeSeconds === 'function') setTakeSeconds(0);
if (p.upscale) setUpscale(p.upscale);
if (p.upscale_method) setUpscaleMethod(p.upscale_method);
diff --git a/webapp/style/panel.css b/webapp/style/panel.css
index 6ecb333..b6ff3cf 100644
--- a/webapp/style/panel.css
+++ b/webapp/style/panel.css
@@ -2908,6 +2908,8 @@
.sb-auto { margin: 4px 0 8px; font-size: var(--fs-xs); }
.sb-track-row { display: flex; gap: 8px; align-items: center; }
.sb-track-row .sb-input { flex: 1 1 auto; min-width: 0; }
+ /* One Shot: the camera line, one move for the whole shot (take_camera_lock). */
+ .oneshot-camera .sb-input { width: 100%; box-sizing: border-box; font-size: var(--fs-sm); }
.sb-track-bars { flex: 0 0 auto; width: auto; min-width: 150px; }
#sbLengthRow.is-standing-down { opacity: 0.45; pointer-events: none; }
/* `.cz-control` sets a display, which beats the UA's [hidden]; say it