World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
Python agentagentic-aiaiclaudecopilotcursorelevenlabsffmpegfluximage-generationopen-sourceopenaipythonremotionstable-diffusiontext-to-speechtext-to-videovideo-generationvideo-production
video-compose: catch TTS punctuation leaks in final review
calesthio0efed7427c7c
diff --git a/skills/core/hyperframes.md b/skills/core/hyperframes.mdindex b56378e2..3902eae1 100644--- a/skills/core/hyperframes.md+++ b/skills/core/hyperframes.md@@ -288,54 +288,76 @@ Same pattern as Remotion.Things the upstream docs don't warn about but will cost you a 60-minuterender to discover. Fix at author time, not at render time.-### Video elements need explicit size, in BOTH HTML attrs and `!important` CSS+### Full-frame background video: source-resolution trap (NOT a framework bug)++Symptom: background video renders as a small centered box with black+around it, even though you've told HyperFrames the clip is 1920×1080.+Six renders of debugging revealed this is **almost always a source+quality issue, not a HyperFrames framework bug.** Fix the input, not+the CSS.++Root cause (observed on Pexels + Pixabay stock): many free stock+clips ship at 640×360 or 960×540 even when you ask for the "large"+size tier. If your pre-transform does+`scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:...`+you'll produce a 1920×1080 file whose visible content is a centered+640×360 rectangle with black padding. HyperFrames plays this clip+faithfully — the small-video-in-black-frame you see is a correctly+rendered letterboxed input.++Diagnostic first: before blaming CSS, run+`ffprobe -v error -select_streams v -show_entries stream=width,height`+on the source clip and on your pre-transformed clip. If the pre-+transform already shows a 640×360 active region inside 1920×1080+padding, the problem is upstream.++Fix the pre-transform to scale-to-COVER and crop, not scale-to-FIT+and pad:-HyperFrames runtime applies inline `style="width:...px; height:...px"` to-every `<video>` element based on the video's **intrinsic** dimensions.-That inline style beats any class-selector CSS you write, so even a-`video.bg-video { width: 100%; height: 100% }` rule gets silently ignored-and the video renders at whatever size the decoded stream reports —-usually a small centered box inside an otherwise-black clip.--The bug is invisible in some scenarios:-- Low `filter: brightness(0.25-0.35)` makes the small video box look- mostly black.-- Fullscreen typography layered on top hides the problem.-- `object-fit: cover` on a class selector does NOT fix it because the- inline width/height wins first.+```+-vf "scale=1920:1080:force_original_aspect_ratio=increase,\+ crop=1920:1080,unsharp=3:3:0.6"+```-It becomes obvious the moment you go footage-forward (brightness > 0.5,-lower-third typography) — the small centered video box appears against-a black frame that was supposed to be full-bleed b-roll.+This upscales the small clip to at least cover 1920×1080, crops the+overflow, and sharpens (unsharp) to offset perceived softness. The+output is full-bleed and HyperFrames will render it edge-to-edge.-**Correct pattern** for a full-frame background video:+The canonical wrapper-div pattern (`patterns.md`) is still the right+HTML shape for backgrounds:```html-<video- class="clip bg-video"- data-start="0" data-duration="8" data-track-index="0"- src="assets/hero.mp4"- muted playsinline preload="metadata"- width="1920" height="1080"-></video>+<div style="position:absolute;top:0;left:0;width:1920px;height:1080px;overflow:hidden;">+ <video+ data-start="0" data-duration="60" data-track-index="0"+ src="..." muted playsinline+ style="width:100%;height:100%;object-fit:cover;"+ ></video>+</div>```-```css-video.bg-video {- position: absolute !important; inset: 0 !important;- width: 100% !important; height: 100% !important;- object-fit: cover !important;- /* filter optional */-}-```--Both the explicit `width="1920" height="1080"` HTML attrs AND the-`!important` CSS are required. Either one alone isn't enough.--Alternative: wrap the `<video>` in a `<div class="clip">` and put-`data-start`/`data-duration` on the wrapper. Let the video fill its div-via `position: absolute; inset: 0` without the framework's inline-style-interference. Slightly more DOM but more robust.+Use it because `overflow: hidden` on the wrapper crops gracefully when+the video's aspect differs slightly from 16:9, and because+`object-fit: cover` on the inner `<video>` handles the (rare) case+where source aspect isn't 16:9. It is **not** a workaround for a+framework layout bug — the framework will size `<video class="clip">`+correctly too if the source file has content filling the full frame.++Apply visual treatments (`filter`, `border-radius`, etc.) on the+wrapper or on a scoped selector like `.bg-slot video { filter: ... }`.++Invisible-vs-obvious failure mode: this trap hides when you stack+dim filters (`brightness(0.25–0.35)`) or full-bleed typography on top —+the letterbox reads as "moody darkness." It becomes obvious the moment+you go footage-forward (brightness > 0.5, lower-third typography). If+you're planning footage-forward from the start, probe your source+resolutions in the asset stage.++When you truly can't source HD clips, switch pipelines: use FFmpeg+hybrid-compositing (scale-to-cover + crop + unsharp in an external+b-roll reel, then overlay HyperFrames typography via chromakey). That's+the pattern `projects/quantum-willow-multiverse/` settled on after six+HyperFrames renders with 640×360 Pexels sources.### Always preview-scrub footage-forward scenes before renderdiff --git a/tests/tools/test_hyperframes_compose.py b/tests/tools/test_hyperframes_compose.pyindex b461eb94..35655880 100644--- a/tests/tools/test_hyperframes_compose.py+++ b/tests/tools/test_hyperframes_compose.py@@ -627,6 +627,171 @@ def test_decision_log_accepts_render_runtime_selection_with_both_options():jsonschema.validate(log, schema)+def test_transcript_comparison_catches_literal_punctuation_leak(tmp_path):+ """Regression: Chirp3-HD (and some other TTS engines) read literal `...`+ as the word 'dot' in audio output. This failure is invisible to+ volume-based audio spotchecks but ships audio that literally says+ 'dot dot dot' twelve times. The transcript_comparison check catches+ this automatically before the video is marked pass."""+ import json++ # Real-world example: user's script had `...` everywhere for dramatic+ # pause, Chirp read them all as "dot", transcript contains "dot dot dot"+ # phrases the script never had.+ script_text = (+ "A computer just did in five minutes what would take every machine on Earth, "+ "running since the Big Bang, ten septillion years to finish. "+ "We may have gotten help from parallel universes."+ )+ transcript_data = {+ "word_timestamps": [+ {"word": "A", "start": 0.0, "end": 0.1},+ {"word": "computer", "start": 0.1, "end": 0.5},+ {"word": "just", "start": 0.5, "end": 0.7},+ {"word": "did", "start": 0.7, "end": 0.9},+ {"word": "in", "start": 0.9, "end": 1.0},+ {"word": "five", "start": 1.0, "end": 1.3},+ {"word": "minutes", "start": 1.3, "end": 1.7},+ {"word": "dot", "start": 1.7, "end": 1.9}, # leak!+ {"word": "dot", "start": 1.9, "end": 2.1}, # leak!+ {"word": "dot", "start": 2.1, "end": 2.3}, # leak!+ {"word": "what", "start": 2.5, "end": 2.8},+ {"word": "would", "start": 2.8, "end": 3.0},+ {"word": "take", "start": 3.0, "end": 3.3},+ {"word": "every", "start": 3.3, "end": 3.6},+ {"word": "machine", "start": 3.6, "end": 4.0},+ {"word": "on", "start": 4.0, "end": 4.2},+ {"word": "Earth", "start": 4.2, "end": 4.6},+ {"word": "running", "start": 4.7, "end": 5.1},+ {"word": "since", "start": 5.1, "end": 5.4},+ {"word": "the", "start": 5.4, "end": 5.5},+ {"word": "Big", "start": 5.5, "end": 5.8},+ {"word": "Bang", "start": 5.8, "end": 6.2},+ {"word": "ten", "start": 6.2, "end": 6.5},+ {"word": "septillion", "start": 6.5, "end": 7.3},+ {"word": "years", "start": 7.3, "end": 7.7},+ {"word": "to", "start": 7.7, "end": 7.9},+ {"word": "finish", "start": 7.9, "end": 8.3},+ {"word": "dot", "start": 8.3, "end": 8.5}, # another leak+ {"word": "We", "start": 9.0, "end": 9.2},+ {"word": "may", "start": 9.2, "end": 9.4},+ {"word": "have", "start": 9.4, "end": 9.6},+ {"word": "gotten", "start": 9.6, "end": 9.9},+ {"word": "help", "start": 9.9, "end": 10.3},+ {"word": "from", "start": 10.3, "end": 10.5},+ {"word": "parallel", "start": 10.5, "end": 11.0},+ {"word": "universes", "start": 11.0, "end": 11.7},+ ]+ }+ transcript_path = tmp_path / "transcript.json"+ transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")++ result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)++ # Must catch the punctuation leak+ assert result["spurious_punctuation_words"], (+ "transcript_comparison failed to detect the 'dot' leak from literal ... punctuation."+ )+ leak_counts = {+ entry["word"]: entry["count"]+ for entry in result["spurious_punctuation_words"]+ }+ assert leak_counts.get("dot") == 4, f"Expected 4 'dot' leaks, got {leak_counts}"++ # Must produce a CRITICAL-severity issue message+ issue_text = " ".join(result["issues"]).lower()+ assert "tts punctuation leak" in issue_text+ assert "not in the script" in issue_text++ # Must NOT mark the transcript as matching+ assert result["transcript_matches_script"] is False+++def test_transcript_comparison_passes_clean_audio(tmp_path):+ """Clean audio with no punctuation leaks must NOT trigger a false+ positive."""+ import json++ script_text = "The quick brown fox jumps over the lazy dog."+ transcript_data = {+ "word_timestamps": [+ {"word": "The", "start": 0.0, "end": 0.1},+ {"word": "quick", "start": 0.1, "end": 0.4},+ {"word": "brown", "start": 0.4, "end": 0.7},+ {"word": "fox", "start": 0.7, "end": 1.0},+ {"word": "jumps", "start": 1.0, "end": 1.3},+ {"word": "over", "start": 1.3, "end": 1.6},+ {"word": "the", "start": 1.6, "end": 1.7},+ {"word": "lazy", "start": 1.7, "end": 2.0},+ {"word": "dog", "start": 2.0, "end": 2.3},+ ]+ }+ transcript_path = tmp_path / "transcript.json"+ transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")++ result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)+ assert result["spurious_punctuation_words"] == []+ assert result["transcript_matches_script"] is True+ assert result["word_accuracy"] >= 0.9+ # issues may still have informational content but no CRITICAL TTS leak+ assert not any("tts punctuation leak" in i.lower() for i in result["issues"])+++def test_transcript_comparison_graceful_when_inputs_missing(tmp_path):+ """When transcript or script is unavailable, the check should NOT+ crash — it should record the skip in issues so the silence is visible."""+ # No transcript+ result = VideoCompose._compare_transcript_to_script(None, "some script text")+ assert any("not provided" in i for i in result["issues"])++ # No script+ dummy = tmp_path / "t.json"+ dummy.write_text('{"word_timestamps": []}', encoding="utf-8")+ result = VideoCompose._compare_transcript_to_script(dummy, "")+ assert any("not provided" in i for i in result["issues"])++ # Transcript file missing+ result = VideoCompose._compare_transcript_to_script(tmp_path / "nonexistent.json", "script")+ assert any("not provided" in i for i in result["issues"])+++def test_run_final_review_includes_transcript_comparison_section(tmp_path):+ """Regression: the `transcript_comparison` section must ALWAYS appear in+ the final_review output — even when the caller doesn't provide a+ transcript. A missing section = silent governance failure."""+ import subprocess++ # Build a minimal MP4 so _run_final_review can probe it.+ mp4 = tmp_path / "out.mp4"+ subprocess.run(+ [+ "ffmpeg", "-y",+ "-f", "lavfi", "-i", "color=c=#000000:s=320x240:d=2",+ "-f", "lavfi", "-i", "sine=frequency=440:duration=2",+ "-c:v", "libx264", "-pix_fmt", "yuv420p",+ "-c:a", "aac", "-shortest", str(mp4),+ ],+ capture_output=True, check=True, timeout=30,+ )++ review = VideoCompose()._run_final_review(+ mp4,+ edit_decisions={+ "version": "1.0",+ "renderer_family": "animation-first",+ "render_runtime": "hyperframes",+ "cuts": [{"id": "c1", "source": "x", "in_seconds": 0, "out_seconds": 2}],+ },+ )+ assert "transcript_comparison" in review["checks"], (+ "final_review must always include a transcript_comparison section. "+ "When the caller doesn't provide a transcript, the section should "+ "still appear with a 'skipped' issue entry — not be omitted."+ )+ tc = review["checks"]["transcript_comparison"]+ assert any("not provided" in i for i in tc["issues"])++def test_hyperframes_root_composition_has_data_start_and_duration(tmp_path):"""Regression: the generated root composition was missing data-startand data-duration, violating the HyperFrames contract (SKILL.md table)."""diff --git a/tools/video/video_compose.py b/tools/video/video_compose.pyindex ea22a6bf..90d99596 100644--- a/tools/video/video_compose.py+++ b/tools/video/video_compose.py@@ -105,6 +105,32 @@ class VideoCompose(BaseTool):"edit_decisions.metadata.proposal_render_runtime."),},+ "narration_transcript_path": {+ "type": "string",+ "description": (+ "Path to a word-level transcript JSON (from `transcriber` "+ "tool output). Optional but STRONGLY recommended: when "+ "combined with script_path/script_text, final_review "+ "runs transcript_comparison and catches TTS failures "+ "like 'Chirp3-HD reads ... as the word dot'. Without "+ "it, content-level audio bugs ship silently."+ ),+ },+ "script_path": {+ "type": "string",+ "description": (+ "Path to the source narration script (plain text). "+ "Used by transcript_comparison to diff against the "+ "transcribed audio. Provide this OR script_text."+ ),+ },+ "script_text": {+ "type": "string",+ "description": (+ "Inline source narration script. Used by "+ "transcript_comparison when a file path is unavailable."+ ),+ },"subtitle_path": {"type": "string"},"subtitle_style": {"type": "object",@@ -1043,7 +1069,13 @@ class VideoCompose(BaseTool):# --- Post-render: mandatory final self-review ---if render_result.success and output_path.exists():final_review = self._run_final_review(- output_path, edit_decisions, inputs.get("proposal_packet")+ output_path,+ edit_decisions,+ inputs.get("proposal_packet"),+ narration_transcript_path=inputs.get("narration_transcript_path"),+ script_text=inputs.get("script_text") or self._read_text_file(+ inputs.get("script_path")+ ),)# Attach final_review to the ToolResult data so the compose-director@@ -1160,7 +1192,13 @@ class VideoCompose(BaseTool):# Post-render: mandatory final self-review (identical contract to the Remotion path).if output_path.exists():final_review = self._run_final_review(- output_path, edit_decisions, inputs.get("proposal_packet")+ output_path,+ edit_decisions,+ inputs.get("proposal_packet"),+ narration_transcript_path=inputs.get("narration_transcript_path"),+ script_text=inputs.get("script_text") or self._read_text_file(+ inputs.get("script_path")+ ),)if render_result.data is None:render_result.data = {}@@ -1214,7 +1252,13 @@ class VideoCompose(BaseTool):if render_result.success and output_path.exists():final_review = self._run_final_review(- output_path, edit_decisions, inputs.get("proposal_packet")+ output_path,+ edit_decisions,+ inputs.get("proposal_packet"),+ narration_transcript_path=inputs.get("narration_transcript_path"),+ script_text=inputs.get("script_text") or self._read_text_file(+ inputs.get("script_path")+ ),)if render_result.data is None:render_result.data = {}@@ -1353,11 +1397,156 @@ class VideoCompose(BaseTool):# Final self-review — mandatory post-render inspection# ------------------------------------------------------------------+ # Punctuation/SSML-leak words that should NEVER appear in rendered audio.+ # When a TTS engine reads a literal "..." as the word "dot", or a "—" as+ # "hyphen", those leak into the transcript. Catching these in the final+ # review is the difference between catching a bad voice render in-tool+ # vs. shipping a video that says "dot dot dot" twelve times. CRITICAL.+ _TTS_PUNCTUATION_LEAK_WORDS = {+ "dot", "dots", "ellipsis", "period", "periods",+ "comma", "commas", "semicolon", "colon",+ "dash", "hyphen", "emdash", "endash",+ "parenthesis", "bracket", "brace",+ "asterisk", "slash", "backslash",+ "exclamation", "question mark",+ }++ @staticmethod+ def _read_text_file(path: str | Path | None) -> str | None:+ """Read a small text file if given a path; None-safe and exception-safe."""+ if not path:+ return None+ try:+ return Path(path).read_text(encoding="utf-8")+ except Exception:+ return None++ @classmethod+ def _tokenize(cls, text: str) -> list[str]:+ """Split text into comparable word tokens (lowercased, punctuation+ stripped, numeric-word-aware). Empty tokens dropped."""+ import re++ # Preserve hyphenated words as single tokens ("many-worlds" -> "many-worlds").+ # Drop everything except letters, digits, hyphens, apostrophes.+ cleaned = re.sub(r"[^A-Za-z0-9\-' ]+", " ", text.lower())+ return [t for t in cleaned.split() if t and t != "-"]++ @classmethod+ def _compare_transcript_to_script(+ cls,+ transcript_path: Path,+ script_text: str,+ ) -> dict[str, Any]:+ """Compare a word-level transcript against the source script.++ Purpose: catch TTS failures that look fine on audio-volume/duration+ checks but produce garbage content. The canonical example is+ Chirp3-HD reading ellipses ("...") literally as the word "dot" — our+ volume check says "narration present, not clipped" and the video+ ships. This check diffs the actual transcribed audio against what+ was supposed to be said, and flags:++ - Spurious punctuation-leak words ("dot", "comma", "hyphen", etc.)+ that appear in audio but not script → CRITICAL+ - Overall word-accuracy ratio against script → SUGGESTION if < 0.9++ Returns the transcript_comparison section of final_review, or a+ placeholder with an issue describing why the check couldn't run+ (missing transcript, missing script) so the review never goes+ silently quiet on this contract.+ """+ result: dict[str, Any] = {+ "transcript_matches_script": False,+ "word_accuracy": None,+ "script_word_count": 0,+ "transcript_word_count": 0,+ "spurious_punctuation_words": [],+ "issues": [],+ }++ if not transcript_path or not Path(transcript_path).is_file():+ result["issues"].append(+ "transcript_comparison skipped: narration_transcript not provided"+ )+ return result+ if not script_text:+ result["issues"].append(+ "transcript_comparison skipped: script_text not provided"+ )+ return result++ try:+ transcript_data = json.loads(Path(transcript_path).read_text(encoding="utf-8"))+ except Exception as e:+ result["issues"].append(f"transcript_comparison could not parse transcript: {e}")+ return result++ transcript_words = [+ w.get("word", "").strip() for w in transcript_data.get("word_timestamps", [])+ ]+ transcript_tokens = cls._tokenize(" ".join(transcript_words))+ script_tokens = cls._tokenize(script_text)++ result["script_word_count"] = len(script_tokens)+ result["transcript_word_count"] = len(transcript_tokens)++ if not script_tokens or not transcript_tokens:+ result["issues"].append(+ f"transcript_comparison: empty token set "+ f"(script={len(script_tokens)}, transcript={len(transcript_tokens)})"+ )+ return result++ # --- Punctuation-leak detection (TTS reading literal punctuation) ---+ script_set = set(script_tokens)+ leak_occurrences: dict[str, int] = {}+ for token in transcript_tokens:+ if token in cls._TTS_PUNCTUATION_LEAK_WORDS and token not in script_set:+ leak_occurrences[token] = leak_occurrences.get(token, 0) + 1++ if leak_occurrences:+ formatted = ", ".join(+ f"{w!r}×{n}" for w, n in sorted(leak_occurrences.items(), key=lambda x: -x[1])+ )+ result["spurious_punctuation_words"] = [+ {"word": w, "count": n} for w, n in leak_occurrences.items()+ ]+ result["issues"].append(+ f"TTS punctuation leak: transcript contains {formatted} — "+ f"these words are NOT in the script, which means the voice "+ f"engine is reading literal punctuation aloud. Rewrite the "+ f"script to eliminate the corresponding characters (ellipses, "+ f"em-dashes, etc.) and regenerate narration."+ )++ # --- Word accuracy via set overlap (cheap & ordering-insensitive) ---+ # We don't penalize small word-order differences or minor TTS+ # hallucinations; we just want to know "did 90%+ of the script's+ # content make it into the audio." Using set overlap on the script+ # side is robust to transcription noise.+ matched = sum(1 for t in script_tokens if t in set(transcript_tokens))+ accuracy = matched / max(1, len(script_tokens))+ result["word_accuracy"] = round(accuracy, 3)+ result["transcript_matches_script"] = accuracy >= 0.9 and not leak_occurrences++ if accuracy < 0.9:+ result["issues"].append(+ f"Low transcript-to-script match: only {accuracy:.0%} of script "+ f"words appear in the transcribed audio ({matched}/"+ f"{len(script_tokens)}). Narration may be truncated, mispronounced, "+ f"or the wrong script was used."+ )++ return result+def _run_final_review(self,output_path: Path,edit_decisions: dict[str, Any] | None = None,proposal_packet: dict[str, Any] | None = None,+ narration_transcript_path: str | Path | None = None,+ script_text: str | None = None,) -> dict[str, Any]:"""Run post-render self-review and produce a final_review artifact.@@ -1706,12 +1895,24 @@ class VideoCompose(BaseTool):issues.extend(subtitle_check.get("issues", []))- # --- 6. Determine overall status ---+ # --- 6. Transcript-vs-script comparison ---+ # Catches content-level TTS failures (the classic "Chirp reads `...`+ # as the word 'dot'" trap) that volume-based audio checks miss.+ # Only runs when caller provides both the transcript and script; when+ # skipped, issues list records that so the silence is visible.+ transcript_comparison = self._compare_transcript_to_script(+ Path(narration_transcript_path) if narration_transcript_path else None,+ script_text,+ )+ issues.extend(transcript_comparison.get("issues", []))++ # --- 7. Determine overall status ---critical_issues = [i for i in issuesif any(kw in i.lower() for kw in ["silent downgrade", "delivery promise violation","effectively silent", "ffprobe failed", "suspiciously short",+ "tts punctuation leak", # reading literal punctuation aloud])]@@ -1739,6 +1940,7 @@ class VideoCompose(BaseTool):"audio_spotcheck": audio_spotcheck,"promise_preservation": promise_preservation,"subtitle_check": subtitle_check,+ "transcript_comparison": transcript_comparison,},"issues_found": issues,"recommended_action": recommended_action,