Open-source AI podcast clipper. Generate vertical clips with face tracking and burned-in captions. CLI, MCP server, web app.
podcli → Horizontal (16:9) clips + multi-format repurposing
Goal: render every detected moment to more than one format (vertical 9:16 today, horizontal 16:9 next, square later) so podcli becomes the one place to repurpose a podcast, not just a vertical-clip button. The AI half is the real product: horizontal needs a different viral-moment profile than vertical, not just a wider crop.
North star
one detected moment
├─ 9:16 short → Shorts / Reels / TikTok (hook-first, 20-45s) ← today
└─ 16:9 clip → YouTube in-feed / X / LinkedIn (arc-first, 60-300s) ← this plan
↓ both performance streams feed the YouTube learning loop
→ the scorer learns which format wins per moment type
The one decision everything hangs on
Format is a render-time property. A moment is format-neutral.
A moment is a scored time-range (start, end, segments, speakers, content_type) with no aspect ratio. It carries per-format scores (vertical_score, horizontal_score) because virality scoring differs by format, then renders to one or more formats. Keeping format at render time (not on the moment) is what makes "detect once, fan out to N formats" coherent and preserves the learning loop. Every other repurposing output (audiogram, quote card, show notes, X thread) is later "just another renderer off the same moment + transcript + brand KB."
Vertical vs horizontal are different products
| Vertical 9:16 (have) | Horizontal 16:9 (this plan) | |
|---|---|---|
| Winning moment | one punchy line, hook in 3s | narrative arc, debate/tension, payoff |
| Length | 20-45s | 60-300s |
| Framing | active-speaker crop / split-screen | wide two-shot, reaction faces are an asset |
| Reframe cost | high (face tracking) | low (near-passthrough, reframe=false) |
| Platform | Shorts, Reels, TikTok | YouTube in-feed, X, LinkedIn desktop |
So the work splits in two: selection (a second scoring profile + a dialogue-tension signal) and framing (which gets simpler, since 16:9-from-16:9 is scale/pad, not crop).
Verified coupling inventory
An adversarial audit (7 parallel readers + 8 refutation passes + synthesis) found the real surface is 31 distinct aspect-ratio coupling sites (56 raw literals, ~66 edit points including format threading + duration caps), not the 8 originally mapped. The audit refuted 4 design assumptions. Those corrections are baked into the phases below:
- Captions are NOT relative.
HormoziCaptions/SubtleCaptions/KaraokeCaptionsread onlyfpsfromuseVideoConfig();BrandedCaptionsalso readsheight(only for a face-avoid nudge), neverwidth. All font sizes/margins/logo box/side insets are absolute pixels inremotion/src/types.tsSTYLES tuned for 1080x1920. Changing composition dims alone leaves captions ~1.8x oversized floating mid-frame. Caption geometry is the biggest blind spot and is net-new plumbing across 4 layers (Python cmd →render.mjsCLI flag →inputProps→CaptionedClipprop), plus the ASS fallback (captions_burn.py,caption_renderer.pyPlayRes,caption_styles.pymargins). - The MCP "two schemas" was wrong. The handler
inputSchemaobjects (create-clip.handler.ts:33,batch-clips.handler.ts) are dead code (server.tool()uses only.name/.description). The real gates are (a) the inline Zod shape insrc/server.ts(MCP, strips unknown keys), and (b)src/ui/web-server.ts, the primary path, which manually destructuresreq.bodyand builds the Python payload with an explicit allowlist (styledClipswhitelist ~`2671,/api/create-clip,/api/batch-clips). Aformat` param must be added at ~10 hops or it silently reverts to vertical with no error. fit_to_frameis NOT redundant withcrop_to_vertical's center branch. That branch blur-pillarboxes wider-than-target sources and crops top/bottom off narrower ones; only pixel-exact 16:9 passes through. Real-world near-16:9 inputs (1920x1088, 2.39:1, 4:3 inserts) are mishandled. Keepfit_to_frameas a distinct scale+pad/letterbox path.- Render canvas dims are already dynamic.
render.mjs:170-238ffprobes the cropped clip and overrides Remotion composition width/height atrenderMediatime, so a 1920x1080 source auto-produces a 1920x1080 overlay (DaVinci ProRes overlay too).Root.tsx:38-39literals govern only Remotion Studio preview. So horizontal render is dimension-correct for free; it is caption-layout-broken until profile work lands.
Two hard caps that block horizontal from functioning
backend/services/clip_generator.py:711:if duration > MAX_CLIP_DURATION: raise ValueError(45s). A 60-300s clip crashes here. Must becomespec.dur_max-aware.src/ui/web-server.ts: 180s HTTP duration cap on/api/create-clipand/api/batch-clips. Rejects horizontal at the boundary before Python.
Deferred / do-not-touch
- Thumbnails are a separate ~4-file / ~12-literal 9:16 track (
thumbnail_ai.py,thumbnail_generator.py,thumbnail_html.py,ThumbnailTemplate.tsx). Not part of clip FormatSpec. Horizontal clips get portrait thumbnails until Phase 5. - Latent-safe at 9:16, only matters for a non-vertical reframe format (square):
face_analysis.py:171and thecrop_to_vertical:170call into_detect_local_speaker_reframe_plan(which omitstarget_ratio, solocal_reframe.py:188stays hardcoded 9/16). Fix only when square/reframe lands. - Native/dist:
formats.pyauto-propagates intocli/internal/backend/files/(go:embed) via the sync step; contributors must rungo generatebefore a native build. Any Phase-5 CSS variant must land insrc+dist/ui+dist/studiocopies.
Phases
Phase 1: Parameterize rendering (SAFE, no behavior change) ✅ DONE
Verified byte-identical for the vertical default (341/341 backend tests pass; smoke-checked that duration aliases hold 20/45/20/35, affected importers load, and crop_to_vertical's target_dims default is (1080,1920)). Two guardrails were mandatory and applied:
- Guardrail A: keep duration constants as aliases.
cli.py:70,398,clip_generator.py:31,claude_suggest.py:21, andDEFAULT_PRESETitself importMIN/MAX/TARGET_CLIP_DURATIONby name; deleting them ImportError-crashes all four. They now derive fromFORMATS["vertical"]but keep the exact module-level names. - Guardrail B: change only the crop-path literal. Only
video_processor.py:103was touched. Thumbnails, caption PlayRes,clip_studio.py,render.mjsfallback, andRoot.tsxwere left alone because they resolve to vertical anyway, and rewriting them adds risk with no behavior benefit.
Edits shipped:
- NEW
backend/services/formats.py:FormatSpec(frozen dataclass) +FORMATS{vertical, horizontal, square} +get_format(). Imports nothing frompresets(avoids cycle;presetsimports from it). backend/presets.py: duration constants now aliasFORMATS["vertical"].{dur_min,dur_max,target_min,target_max}; added"format": "vertical"toDEFAULT_PRESET(inert; the{**DEFAULT_PRESET, **saved}merge back-fills old presets).backend/services/video_processor.py:crop_to_vertical(...)gainedtarget_dims: tuple = (1080, 1920); line 103 is nowtarget_w, target_h = target_dims.
Phase 2: Render horizontal (still defaults vertical) ✅ DONE
Verified end to end: a 1280x720 source rendered format="horizontal" → 1920x1080 via chose=fit-letterbox (face analysis skipped), and format="vertical" → 1080x1920 unchanged. 341 Python + 47 TS tests pass, tsc clean.
- Render fork in
clip_generator.pystep 2:spec.reframe ? crop_to_vertical(..., target_dims=spec.dims) : fit_to_frame(..., spec.dims). Newfit_to_frameinvideo_processor.py=scale=…:force_original_aspect_ratio=decrease+ centeredpad+setsar=1(letterbox, collapses to plain scale for exact-ratio), skips all face analysis.spec = get_format(format)resolved once and reused for the dur cap, progress label, and fork. (Kept thecrop_to_verticalname for now; rename deferred.) - Caps relaxed:
clip_generator.pyusesspec.dur_max(vertical still 45); the two web-server 180s HTTP caps use a format-keyed ceiling (horizontal→300, else 180, notspec.dur_max, which would tighten vertical). formatthreaded end to end, defaulting vertical at every hop:generate_clipsignature + result dict;main.pycreate/batch; 6cli.pygenerate_clipcall sites +--formatflag +_selection_signaturecache key;server.tscreate/batch Zod + primary clip literal + export-selected/clip-numbers maps +findDuplicate/record;web-server.tscreate-clip destructure/validation/payload/history/recipe + MCP-export resolver +styledClipswhitelist + batch per-clip cap +createBatchHistoryRecorder;models/index.tstypes;clips-history.tsfindDuplicate(backward-compatible, missing→vertical, so a horizontal re-clip of a vertical range is no longer a false duplicate).Root.tsx/render.mjsuntouched (ffprobe already sizes the canvas). Output is dimension-correct; captions still use vertical geometry until Phase 3 (conscious deferral, not a silent bug).
Phase 3: Correct captions per format (the real blind spot)
- Add a
caption_profilecarrying per-format geometry (fontSize, margins, logo box, side insets). Wire a--format/--caption-profileflag throughrender.mjs→inputProps→CaptionedClipProps, and add alower_thirdprofile totypes.tsSTYLES + the 4 Remotion caption components (make absolute pixels profile-driven). - Mirror in the ASS fallback (default path,
allow_ass_fallback=True): thread real dims intocaptions_burn.py:27,caption_renderer.pyPlayRes (17,501),caption_styles.py:73margins; wire up the already-parameterized-but-dead_calibrate_libass_y.
Phase 4: Iterate the AI for horizontal viral moments (the product)
- Second scoring profile. Keep
standalone/hook/relevance/quotabilityasvertical_score; addarc/tension/depth/payoffashorizontal_scoreinclaude_suggest.py:_build_prompt. New KB file.podcli/knowledge/04b-longform-creation-guide.mdloaded alongside04-shorts-creation-guide.md. Moment schema (suggest-clips.handler.ts) gains both scores (keepscorealias). Detect once, rank twice. - Dialogue-tension signal (highest-leverage new AI): a function in
audio_analyzer.pynext tocompute_energy_scorescombining speaker-turn frequency (from diarization labels) + sustained energy → catches debate/back-and-forth that vertical spike-scoring misses. Feedshorizontal_score. - Make the LLM duration window format-aware (
claude_suggest.py:241-311,433currently bake 20-45s into the prompt).
Phase 5: UI + learning loop + more renderers
- ContentStudio format selector + "render both"; 16:9 preview variants in
styles.css(+distcopies) andEpisodeWorkspace.jsx:77(PROD_TO_PCThardcodes 1920). - Feed both format performance streams into the YouTube learning loop ([studio-phase-1] roadmap) → learn format × moment-type winners → back into scoring.
- Horizontal thumbnails (16:9 / 1280x720) as a deliverable. Then the cheap renderers off the same spine: audiograms, quote cards (reuse
thumbnail_ai.py), show notes / X threads (LLM over the transcript).
Sequencing guarantee
Each phase is independently mergeable and leaves main green. Phase 1 already is (behavior-identical). Phases 2-3 ship behind the default-vertical flag (no existing user sees a change until they pick horizontal). Phase 4 is where the product value lands.