Skip to content

World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.

video-compose: catch TTS punctuation leaks in final review

calesthio0efed7427c7c
diff --git a/skills/core/hyperframes.md b/skills/core/hyperframes.md
index b56378e2..3902eae1 100644
--- a/skills/core/hyperframes.md
+++ b/skills/core/hyperframes.md
@@ -288,54 +288,76 @@ Same pattern as Remotion.
Things the upstream docs don't warn about but will cost you a 60-minute
render to discover. Fix at author time, not at render time.
-### Video elements need explicit size, in BOTH HTML attrs and `!important` CSS
+### Full-frame background video: source-resolution trap (NOT a framework bug)
+
+Symptom: background video renders as a small centered box with black
+around it, even though you've told HyperFrames the clip is 1920×1080.
+Six renders of debugging revealed this is **almost always a source
+quality issue, not a HyperFrames framework bug.** Fix the input, not
+the CSS.
+
+Root cause (observed on Pexels + Pixabay stock): many free stock
+clips ship at 640×360 or 960×540 even when you ask for the "large"
+size tier. If your pre-transform does
+`scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:...`
+you'll produce a 1920×1080 file whose visible content is a centered
+640×360 rectangle with black padding. HyperFrames plays this clip
+faithfully — the small-video-in-black-frame you see is a correctly
+rendered letterboxed input.
+
+Diagnostic first: before blaming CSS, run
+`ffprobe -v error -select_streams v -show_entries stream=width,height`
+on the source clip and on your pre-transformed clip. If the pre-
+transform already shows a 640×360 active region inside 1920×1080
+padding, the problem is upstream.
+
+Fix the pre-transform to scale-to-COVER and crop, not scale-to-FIT
+and pad:
-HyperFrames runtime applies inline `style="width:...px; height:...px"` to
-every `<video>` element based on the video's **intrinsic** dimensions.
-That inline style beats any class-selector CSS you write, so even a
-`video.bg-video { width: 100%; height: 100% }` rule gets silently ignored
-and the video renders at whatever size the decoded stream reports —
-usually a small centered box inside an otherwise-black clip.
-
-The bug is invisible in some scenarios:
-- Low `filter: brightness(0.25-0.35)` makes the small video box look
- mostly black.
-- Fullscreen typography layered on top hides the problem.
-- `object-fit: cover` on a class selector does NOT fix it because the
- inline width/height wins first.
+```
+-vf "scale=1920:1080:force_original_aspect_ratio=increase,\
+ crop=1920:1080,unsharp=3:3:0.6"
+```
-It becomes obvious the moment you go footage-forward (brightness > 0.5,
-lower-third typography) — the small centered video box appears against
-a black frame that was supposed to be full-bleed b-roll.
+This upscales the small clip to at least cover 1920×1080, crops the
+overflow, and sharpens (unsharp) to offset perceived softness. The
+output is full-bleed and HyperFrames will render it edge-to-edge.
-**Correct pattern** for a full-frame background video:
+The canonical wrapper-div pattern (`patterns.md`) is still the right
+HTML shape for backgrounds:
```html
-<video
- class="clip bg-video"
- data-start="0" data-duration="8" data-track-index="0"
- src="assets/hero.mp4"
- muted playsinline preload="metadata"
- width="1920" height="1080"
-></video>
+<div style="position:absolute;top:0;left:0;width:1920px;height:1080px;overflow:hidden;">
+ <video
+ data-start="0" data-duration="60" data-track-index="0"
+ src="..." muted playsinline
+ style="width:100%;height:100%;object-fit:cover;"
+ ></video>
+</div>
```
-```css
-video.bg-video {
- position: absolute !important; inset: 0 !important;
- width: 100% !important; height: 100% !important;
- object-fit: cover !important;
- /* filter optional */
-}
-```
-
-Both the explicit `width="1920" height="1080"` HTML attrs AND the
-`!important` CSS are required. Either one alone isn't enough.
-
-Alternative: wrap the `<video>` in a `<div class="clip">` and put
-`data-start`/`data-duration` on the wrapper. Let the video fill its div
-via `position: absolute; inset: 0` without the framework's inline-style
-interference. Slightly more DOM but more robust.
+Use it because `overflow: hidden` on the wrapper crops gracefully when
+the video's aspect differs slightly from 16:9, and because
+`object-fit: cover` on the inner `<video>` handles the (rare) case
+where source aspect isn't 16:9. It is **not** a workaround for a
+framework layout bug — the framework will size `<video class="clip">`
+correctly too if the source file has content filling the full frame.
+
+Apply visual treatments (`filter`, `border-radius`, etc.) on the
+wrapper or on a scoped selector like `.bg-slot video { filter: ... }`.
+
+Invisible-vs-obvious failure mode: this trap hides when you stack
+dim filters (`brightness(0.25–0.35)`) or full-bleed typography on top —
+the letterbox reads as "moody darkness." It becomes obvious the moment
+you go footage-forward (brightness > 0.5, lower-third typography). If
+you're planning footage-forward from the start, probe your source
+resolutions in the asset stage.
+
+When you truly can't source HD clips, switch pipelines: use FFmpeg
+hybrid-compositing (scale-to-cover + crop + unsharp in an external
+b-roll reel, then overlay HyperFrames typography via chromakey). That's
+the pattern `projects/quantum-willow-multiverse/` settled on after six
+HyperFrames renders with 640×360 Pexels sources.
### Always preview-scrub footage-forward scenes before render
diff --git a/tests/tools/test_hyperframes_compose.py b/tests/tools/test_hyperframes_compose.py
index b461eb94..35655880 100644
--- a/tests/tools/test_hyperframes_compose.py
+++ b/tests/tools/test_hyperframes_compose.py
@@ -627,6 +627,171 @@ def test_decision_log_accepts_render_runtime_selection_with_both_options():
jsonschema.validate(log, schema)
+def test_transcript_comparison_catches_literal_punctuation_leak(tmp_path):
+ """Regression: Chirp3-HD (and some other TTS engines) read literal `...`
+ as the word 'dot' in audio output. This failure is invisible to
+ volume-based audio spotchecks but ships audio that literally says
+ 'dot dot dot' twelve times. The transcript_comparison check catches
+ this automatically before the video is marked pass."""
+ import json
+
+ # Real-world example: user's script had `...` everywhere for dramatic
+ # pause, Chirp read them all as "dot", transcript contains "dot dot dot"
+ # phrases the script never had.
+ script_text = (
+ "A computer just did in five minutes what would take every machine on Earth, "
+ "running since the Big Bang, ten septillion years to finish. "
+ "We may have gotten help from parallel universes."
+ )
+ transcript_data = {
+ "word_timestamps": [
+ {"word": "A", "start": 0.0, "end": 0.1},
+ {"word": "computer", "start": 0.1, "end": 0.5},
+ {"word": "just", "start": 0.5, "end": 0.7},
+ {"word": "did", "start": 0.7, "end": 0.9},
+ {"word": "in", "start": 0.9, "end": 1.0},
+ {"word": "five", "start": 1.0, "end": 1.3},
+ {"word": "minutes", "start": 1.3, "end": 1.7},
+ {"word": "dot", "start": 1.7, "end": 1.9}, # leak!
+ {"word": "dot", "start": 1.9, "end": 2.1}, # leak!
+ {"word": "dot", "start": 2.1, "end": 2.3}, # leak!
+ {"word": "what", "start": 2.5, "end": 2.8},
+ {"word": "would", "start": 2.8, "end": 3.0},
+ {"word": "take", "start": 3.0, "end": 3.3},
+ {"word": "every", "start": 3.3, "end": 3.6},
+ {"word": "machine", "start": 3.6, "end": 4.0},
+ {"word": "on", "start": 4.0, "end": 4.2},
+ {"word": "Earth", "start": 4.2, "end": 4.6},
+ {"word": "running", "start": 4.7, "end": 5.1},
+ {"word": "since", "start": 5.1, "end": 5.4},
+ {"word": "the", "start": 5.4, "end": 5.5},
+ {"word": "Big", "start": 5.5, "end": 5.8},
+ {"word": "Bang", "start": 5.8, "end": 6.2},
+ {"word": "ten", "start": 6.2, "end": 6.5},
+ {"word": "septillion", "start": 6.5, "end": 7.3},
+ {"word": "years", "start": 7.3, "end": 7.7},
+ {"word": "to", "start": 7.7, "end": 7.9},
+ {"word": "finish", "start": 7.9, "end": 8.3},
+ {"word": "dot", "start": 8.3, "end": 8.5}, # another leak
+ {"word": "We", "start": 9.0, "end": 9.2},
+ {"word": "may", "start": 9.2, "end": 9.4},
+ {"word": "have", "start": 9.4, "end": 9.6},
+ {"word": "gotten", "start": 9.6, "end": 9.9},
+ {"word": "help", "start": 9.9, "end": 10.3},
+ {"word": "from", "start": 10.3, "end": 10.5},
+ {"word": "parallel", "start": 10.5, "end": 11.0},
+ {"word": "universes", "start": 11.0, "end": 11.7},
+ ]
+ }
+ transcript_path = tmp_path / "transcript.json"
+ transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")
+
+ result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)
+
+ # Must catch the punctuation leak
+ assert result["spurious_punctuation_words"], (
+ "transcript_comparison failed to detect the 'dot' leak from literal ... punctuation."
+ )
+ leak_counts = {
+ entry["word"]: entry["count"]
+ for entry in result["spurious_punctuation_words"]
+ }
+ assert leak_counts.get("dot") == 4, f"Expected 4 'dot' leaks, got {leak_counts}"
+
+ # Must produce a CRITICAL-severity issue message
+ issue_text = " ".join(result["issues"]).lower()
+ assert "tts punctuation leak" in issue_text
+ assert "not in the script" in issue_text
+
+ # Must NOT mark the transcript as matching
+ assert result["transcript_matches_script"] is False
+
+
+def test_transcript_comparison_passes_clean_audio(tmp_path):
+ """Clean audio with no punctuation leaks must NOT trigger a false
+ positive."""
+ import json
+
+ script_text = "The quick brown fox jumps over the lazy dog."
+ transcript_data = {
+ "word_timestamps": [
+ {"word": "The", "start": 0.0, "end": 0.1},
+ {"word": "quick", "start": 0.1, "end": 0.4},
+ {"word": "brown", "start": 0.4, "end": 0.7},
+ {"word": "fox", "start": 0.7, "end": 1.0},
+ {"word": "jumps", "start": 1.0, "end": 1.3},
+ {"word": "over", "start": 1.3, "end": 1.6},
+ {"word": "the", "start": 1.6, "end": 1.7},
+ {"word": "lazy", "start": 1.7, "end": 2.0},
+ {"word": "dog", "start": 2.0, "end": 2.3},
+ ]
+ }
+ transcript_path = tmp_path / "transcript.json"
+ transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")
+
+ result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)
+ assert result["spurious_punctuation_words"] == []
+ assert result["transcript_matches_script"] is True
+ assert result["word_accuracy"] >= 0.9
+ # issues may still have informational content but no CRITICAL TTS leak
+ assert not any("tts punctuation leak" in i.lower() for i in result["issues"])
+
+
+def test_transcript_comparison_graceful_when_inputs_missing(tmp_path):
+ """When transcript or script is unavailable, the check should NOT
+ crash — it should record the skip in issues so the silence is visible."""
+ # No transcript
+ result = VideoCompose._compare_transcript_to_script(None, "some script text")
+ assert any("not provided" in i for i in result["issues"])
+
+ # No script
+ dummy = tmp_path / "t.json"
+ dummy.write_text('{"word_timestamps": []}', encoding="utf-8")
+ result = VideoCompose._compare_transcript_to_script(dummy, "")
+ assert any("not provided" in i for i in result["issues"])
+
+ # Transcript file missing
+ result = VideoCompose._compare_transcript_to_script(tmp_path / "nonexistent.json", "script")
+ assert any("not provided" in i for i in result["issues"])
+
+
+def test_run_final_review_includes_transcript_comparison_section(tmp_path):
+ """Regression: the `transcript_comparison` section must ALWAYS appear in
+ the final_review output — even when the caller doesn't provide a
+ transcript. A missing section = silent governance failure."""
+ import subprocess
+
+ # Build a minimal MP4 so _run_final_review can probe it.
+ mp4 = tmp_path / "out.mp4"
+ subprocess.run(
+ [
+ "ffmpeg", "-y",
+ "-f", "lavfi", "-i", "color=c=#000000:s=320x240:d=2",
+ "-f", "lavfi", "-i", "sine=frequency=440:duration=2",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", "-shortest", str(mp4),
+ ],
+ capture_output=True, check=True, timeout=30,
+ )
+
+ review = VideoCompose()._run_final_review(
+ mp4,
+ edit_decisions={
+ "version": "1.0",
+ "renderer_family": "animation-first",
+ "render_runtime": "hyperframes",
+ "cuts": [{"id": "c1", "source": "x", "in_seconds": 0, "out_seconds": 2}],
+ },
+ )
+ assert "transcript_comparison" in review["checks"], (
+ "final_review must always include a transcript_comparison section. "
+ "When the caller doesn't provide a transcript, the section should "
+ "still appear with a 'skipped' issue entry — not be omitted."
+ )
+ tc = review["checks"]["transcript_comparison"]
+ assert any("not provided" in i for i in tc["issues"])
+
+
def test_hyperframes_root_composition_has_data_start_and_duration(tmp_path):
"""Regression: the generated root composition was missing data-start
and data-duration, violating the HyperFrames contract (SKILL.md table)."""
diff --git a/tools/video/video_compose.py b/tools/video/video_compose.py
index ea22a6bf..90d99596 100644
--- a/tools/video/video_compose.py
+++ b/tools/video/video_compose.py
@@ -105,6 +105,32 @@ class VideoCompose(BaseTool):
"edit_decisions.metadata.proposal_render_runtime."
),
},
+ "narration_transcript_path": {
+ "type": "string",
+ "description": (
+ "Path to a word-level transcript JSON (from `transcriber` "
+ "tool output). Optional but STRONGLY recommended: when "
+ "combined with script_path/script_text, final_review "
+ "runs transcript_comparison and catches TTS failures "
+ "like 'Chirp3-HD reads ... as the word dot'. Without "
+ "it, content-level audio bugs ship silently."
+ ),
+ },
+ "script_path": {
+ "type": "string",
+ "description": (
+ "Path to the source narration script (plain text). "
+ "Used by transcript_comparison to diff against the "
+ "transcribed audio. Provide this OR script_text."
+ ),
+ },
+ "script_text": {
+ "type": "string",
+ "description": (
+ "Inline source narration script. Used by "
+ "transcript_comparison when a file path is unavailable."
+ ),
+ },
"subtitle_path": {"type": "string"},
"subtitle_style": {
"type": "object",
@@ -1043,7 +1069,13 @@ class VideoCompose(BaseTool):
# --- Post-render: mandatory final self-review ---
if render_result.success and output_path.exists():
final_review = self._run_final_review(
- output_path, edit_decisions, inputs.get("proposal_packet")
+ output_path,
+ edit_decisions,
+ inputs.get("proposal_packet"),
+ narration_transcript_path=inputs.get("narration_transcript_path"),
+ script_text=inputs.get("script_text") or self._read_text_file(
+ inputs.get("script_path")
+ ),
)
# Attach final_review to the ToolResult data so the compose-director
@@ -1160,7 +1192,13 @@ class VideoCompose(BaseTool):
# Post-render: mandatory final self-review (identical contract to the Remotion path).
if output_path.exists():
final_review = self._run_final_review(
- output_path, edit_decisions, inputs.get("proposal_packet")
+ output_path,
+ edit_decisions,
+ inputs.get("proposal_packet"),
+ narration_transcript_path=inputs.get("narration_transcript_path"),
+ script_text=inputs.get("script_text") or self._read_text_file(
+ inputs.get("script_path")
+ ),
)
if render_result.data is None:
render_result.data = {}
@@ -1214,7 +1252,13 @@ class VideoCompose(BaseTool):
if render_result.success and output_path.exists():
final_review = self._run_final_review(
- output_path, edit_decisions, inputs.get("proposal_packet")
+ output_path,
+ edit_decisions,
+ inputs.get("proposal_packet"),
+ narration_transcript_path=inputs.get("narration_transcript_path"),
+ script_text=inputs.get("script_text") or self._read_text_file(
+ inputs.get("script_path")
+ ),
)
if render_result.data is None:
render_result.data = {}
@@ -1353,11 +1397,156 @@ class VideoCompose(BaseTool):
# Final self-review — mandatory post-render inspection
# ------------------------------------------------------------------
+ # Punctuation/SSML-leak words that should NEVER appear in rendered audio.
+ # When a TTS engine reads a literal "..." as the word "dot", or a "—" as
+ # "hyphen", those leak into the transcript. Catching these in the final
+ # review is the difference between catching a bad voice render in-tool
+ # vs. shipping a video that says "dot dot dot" twelve times. CRITICAL.
+ _TTS_PUNCTUATION_LEAK_WORDS = {
+ "dot", "dots", "ellipsis", "period", "periods",
+ "comma", "commas", "semicolon", "colon",
+ "dash", "hyphen", "emdash", "endash",
+ "parenthesis", "bracket", "brace",
+ "asterisk", "slash", "backslash",
+ "exclamation", "question mark",
+ }
+
+ @staticmethod
+ def _read_text_file(path: str | Path | None) -> str | None:
+ """Read a small text file if given a path; None-safe and exception-safe."""
+ if not path:
+ return None
+ try:
+ return Path(path).read_text(encoding="utf-8")
+ except Exception:
+ return None
+
+ @classmethod
+ def _tokenize(cls, text: str) -> list[str]:
+ """Split text into comparable word tokens (lowercased, punctuation
+ stripped, numeric-word-aware). Empty tokens dropped."""
+ import re
+
+ # Preserve hyphenated words as single tokens ("many-worlds" -> "many-worlds").
+ # Drop everything except letters, digits, hyphens, apostrophes.
+ cleaned = re.sub(r"[^A-Za-z0-9\-' ]+", " ", text.lower())
+ return [t for t in cleaned.split() if t and t != "-"]
+
+ @classmethod
+ def _compare_transcript_to_script(
+ cls,
+ transcript_path: Path,
+ script_text: str,
+ ) -> dict[str, Any]:
+ """Compare a word-level transcript against the source script.
+
+ Purpose: catch TTS failures that look fine on audio-volume/duration
+ checks but produce garbage content. The canonical example is
+ Chirp3-HD reading ellipses ("...") literally as the word "dot" — our
+ volume check says "narration present, not clipped" and the video
+ ships. This check diffs the actual transcribed audio against what
+ was supposed to be said, and flags:
+
+ - Spurious punctuation-leak words ("dot", "comma", "hyphen", etc.)
+ that appear in audio but not script → CRITICAL
+ - Overall word-accuracy ratio against script → SUGGESTION if < 0.9
+
+ Returns the transcript_comparison section of final_review, or a
+ placeholder with an issue describing why the check couldn't run
+ (missing transcript, missing script) so the review never goes
+ silently quiet on this contract.
+ """
+ result: dict[str, Any] = {
+ "transcript_matches_script": False,
+ "word_accuracy": None,
+ "script_word_count": 0,
+ "transcript_word_count": 0,
+ "spurious_punctuation_words": [],
+ "issues": [],
+ }
+
+ if not transcript_path or not Path(transcript_path).is_file():
+ result["issues"].append(
+ "transcript_comparison skipped: narration_transcript not provided"
+ )
+ return result
+ if not script_text:
+ result["issues"].append(
+ "transcript_comparison skipped: script_text not provided"
+ )
+ return result
+
+ try:
+ transcript_data = json.loads(Path(transcript_path).read_text(encoding="utf-8"))
+ except Exception as e:
+ result["issues"].append(f"transcript_comparison could not parse transcript: {e}")
+ return result
+
+ transcript_words = [
+ w.get("word", "").strip() for w in transcript_data.get("word_timestamps", [])
+ ]
+ transcript_tokens = cls._tokenize(" ".join(transcript_words))
+ script_tokens = cls._tokenize(script_text)
+
+ result["script_word_count"] = len(script_tokens)
+ result["transcript_word_count"] = len(transcript_tokens)
+
+ if not script_tokens or not transcript_tokens:
+ result["issues"].append(
+ f"transcript_comparison: empty token set "
+ f"(script={len(script_tokens)}, transcript={len(transcript_tokens)})"
+ )
+ return result
+
+ # --- Punctuation-leak detection (TTS reading literal punctuation) ---
+ script_set = set(script_tokens)
+ leak_occurrences: dict[str, int] = {}
+ for token in transcript_tokens:
+ if token in cls._TTS_PUNCTUATION_LEAK_WORDS and token not in script_set:
+ leak_occurrences[token] = leak_occurrences.get(token, 0) + 1
+
+ if leak_occurrences:
+ formatted = ", ".join(
+ f"{w!r}×{n}" for w, n in sorted(leak_occurrences.items(), key=lambda x: -x[1])
+ )
+ result["spurious_punctuation_words"] = [
+ {"word": w, "count": n} for w, n in leak_occurrences.items()
+ ]
+ result["issues"].append(
+ f"TTS punctuation leak: transcript contains {formatted} — "
+ f"these words are NOT in the script, which means the voice "
+ f"engine is reading literal punctuation aloud. Rewrite the "
+ f"script to eliminate the corresponding characters (ellipses, "
+ f"em-dashes, etc.) and regenerate narration."
+ )
+
+ # --- Word accuracy via set overlap (cheap & ordering-insensitive) ---
+ # We don't penalize small word-order differences or minor TTS
+ # hallucinations; we just want to know "did 90%+ of the script's
+ # content make it into the audio." Using set overlap on the script
+ # side is robust to transcription noise.
+ matched = sum(1 for t in script_tokens if t in set(transcript_tokens))
+ accuracy = matched / max(1, len(script_tokens))
+ result["word_accuracy"] = round(accuracy, 3)
+ result["transcript_matches_script"] = accuracy >= 0.9 and not leak_occurrences
+
+ if accuracy < 0.9:
+ result["issues"].append(
+ f"Low transcript-to-script match: only {accuracy:.0%} of script "
+ f"words appear in the transcribed audio ({matched}/"
+ f"{len(script_tokens)}). Narration may be truncated, mispronounced, "
+ f"or the wrong script was used."
+ )
+
+ return result
+
def _run_final_review(
self,
output_path: Path,
edit_decisions: dict[str, Any] | None = None,
proposal_packet: dict[str, Any] | None = None,
+ narration_transcript_path: str | Path | None = None,
+ script_text: str | None = None,
) -> dict[str, Any]:
"""Run post-render self-review and produce a final_review artifact.
@@ -1706,12 +1895,24 @@ class VideoCompose(BaseTool):
issues.extend(subtitle_check.get("issues", []))
- # --- 6. Determine overall status ---
+ # --- 6. Transcript-vs-script comparison ---
+ # Catches content-level TTS failures (the classic "Chirp reads `...`
+ # as the word 'dot'" trap) that volume-based audio checks miss.
+ # Only runs when caller provides both the transcript and script; when
+ # skipped, issues list records that so the silence is visible.
+ transcript_comparison = self._compare_transcript_to_script(
+ Path(narration_transcript_path) if narration_transcript_path else None,
+ script_text,
+ )
+ issues.extend(transcript_comparison.get("issues", []))
+
+ # --- 7. Determine overall status ---
critical_issues = [
i for i in issues
if any(kw in i.lower() for kw in [
"silent downgrade", "delivery promise violation",
"effectively silent", "ffprobe failed", "suspiciously short",
+ "tts punctuation leak", # reading literal punctuation aloud
])
]
@@ -1739,6 +1940,7 @@ class VideoCompose(BaseTool):
"audio_spotcheck": audio_spotcheck,
"promise_preservation": promise_preservation,
"subtitle_check": subtitle_check,
+ "transcript_comparison": transcript_comparison,
},
"issues_found": issues,
"recommended_action": recommended_action,