Open-source AI podcast clipper. Generate vertical clips with face tracking and burned-in captions. CLI, MCP server, web app.
podcli → Generalized moment detection (multi-signal saliency engine)
Goal: one detector that works across content types. The podcast clip selector and a new "auto-cut the funny/interesting moments" highlighter (party videos, vlogs, action, streams) become two profiles of one engine, not two features. Anchor use case that surfaced the need: a folder of party clips, "cut me the funny bits", any language, no labels. Podcast selection is preserved as the default profile.
North star
one long video (any content, any language)
→ extract N signal channels (transcript, energy, laughter, prosody, motion, face)
→ per-video-normalize each into a 0..1 saliency curve on a common time grid
→ per-PROFILE weighted fusion → one interestingness curve
→ peak-pick (NMS + prominence) → reaction-expand → snap to natural boundaries
→ candidate moments → (render to formats via the existing fan-out)
The one decision everything hangs on
Content profile is orthogonal to output format. FormatSpec (9:16 / 16:9 / 1:1) is a render-time aspect-ratio decision. A ContentProfile (podcast / party / action / auto) is a detection-time decision about which signals matter and how candidates are generated. A moment is detected under a profile, then rendered to one or more formats. Do not overload FormatSpec.score_key for this, mirror its pattern in a separate ContentProfile spec.
The existing transcript-LLM selector is one channel in this engine (the transcript_semantic channel), not the whole thing. That is the reframe: today the LLM is selection; tomorrow it is the dominant channel for the podcast profile and near-zero for party.
Why this generalizes (research-backed)
Multi-signal highlight detection is a solved shape: compute a per-modality saliency curve, normalize per-video (never global), fuse (weighted sum with soft-OR bias, audio weighted heaviest for human-centric video), peak-pick, and for reaction-driven content extend backwards from the reaction to the cause. FunnyNet-W: a funny moment is "an n-second clip followed by laughter," audio carries >50% of the decision, optimal look-back ≈ 8s. Cai/Lu/Cai built laughter+applause+cheer detection for home videos at ~93% recall. All of it is unsupervised, no labeled data, only pretrained off-the-shelf models. See the audio-model and fusion research threads (this session) for citations.
Profiles as concrete weight vectors
Channels each emit a per-video-normalized 0..1 curve; a profile is a weight vector + candidate-source + peak params.
| Channel | podcast | party | action | Source status |
|---|---|---|---|---|
transcript_semantic (LLM dialogue/quotability) |
0.5 | 0.0 | 0.1 | REUSE claude_suggest.py |
prosody (pitch/energy/rate spikes) |
0.2 | 0.2 | 0.2 | NEW (numpy, no librosa) |
audio_event (laughter/cheer/applause/scream) |
0.15 | 0.4 | 0.3 | NEW (YAMNet-ONNX, no torch) |
energy (RMS, peak-weighted) |
0.1 | 0.2 | 0.2 | REUSE audio_analyzer.py |
motion (optical flow + scene-cut density) |
0.0 | 0.1 | 0.2 | PARTIAL (cuts exist; flow NEW) |
face_reaction (mouth activity / smile proxy) |
0.05 | 0.1 | 0.0 | REUSE face_analysis.py (crude) |
| candidate source | llm |
saliency |
saliency |
n/a |
Weights are starting points, tuned later. auto = a cheap first pass picks the profile: clean speech + 1-2 stable faces → podcast; sparse speech + high motion + crowd/laughter audio → party; sustained high motion → action.
Two candidate-generation modes (the key to not regressing podcast)
llm(podcast): the existingclaude_suggestflow generates candidates over the whole transcript, exactly as today. The fused signal score blends in at the ranking seam. When party channels have weight 0, output is byte-for-byte the current behavior. This is the safety property.saliency(party / action): the fused curve drives candidate generation directly:find_peaks→ reaction-expand → boundary-snap → candidate windows. Works with no usable transcript. The LLM is optional here, used only to title/caption the already-chosen windows.
Architecture: new modules
backend/services/audio_events.py(NEW, no-torch)- YAMNet exported to a self-contained ONNX graph (raw 16 kHz waveform in, log-mel baked into the graph → no librosa needed). ~15 MB, Apache-2.0. AudioSet classes
Laughter,Cheering,Applause,Screamingare native outputs (+ Giggle/Belly-laugh/Whoop children). - Reuse
transcription_whispercpp._extract_wav(-ar 16000 -ac 1) for audio; do not importspeaker_detection(drags in torch). - Returns per-window (≈1 Hz, matching energy) class probabilities → one 0..1 curve per trigger class.
- Runtime:
onnxruntime. First test whethercv2.dnn(already loads YuNet.onnx) can run yamnet.onnx, if yes, zero new deps. - Ship
backend/models/yamnet.onnx+ a dev-onlyscripts/export_yamnet.py(uses tf2onnx; never shipped or imported at runtime).
- YAMNet exported to a self-contained ONNX graph (raw 16 kHz waveform in, log-mel baked into the graph → no librosa needed). ~15 MB, Apache-2.0. AudioSet classes
backend/services/saliency.py(NEW)normalize_per_video(curve): robust z-score via numpy median/MAD, or percentile-rank. Never global.fuse(channels, profile): weighted sum with soft-OR bias.pick_peaks(curve, profile): numpy reimplementation offind_peaks(~40 lines: height floormean + kσ, min-distance NMS by descending height, prominence filter, min-width). Avoids adding scipy. Keep peaks above the top ~10-15th percentile of the video's own prominence distribution (adaptive count).reaction_expand(peak, events): if the peak's driver is anaudio_eventclass, move anchor to reaction onset, extend ~8 s back, keep 1-3 s of payoff.snap_boundaries(window): snap start/end to nearest {Whisper sentence-end, silence gap >300 ms, scene cut, motion-quiet local min} within ±500 ms.
backend/services/profiles.py(NEW):ContentProfiledataclass (name,channel_weights,candidate_source, peak params, reaction-expand params, boundary sources) mirroringFormatSpec;get_profile(name);auto_detect(signals).motionchannel (Phase 3):cv2.calcOpticalFlowFarnebackon sampled frames (OpenCV already present) + reuselocal_reframe.count_scene_cuts/clip_generator._detect_scene_cuts.prosodychannel (Phase 2/3): numpy F0 (autocorrelation) + short-term energy + speech-rate vs speaker baseline. No librosa.
Dependency deltas (no-torch native path)
- ADD
onnxruntimetorequirements-runtime.txt: the only new hard dep. Settled by spike: OpenCV-DNN cannot run the YAMNet graph (rejects the dynamic-shape mel front-end:dynamic 'zero' shapes are not supported), so the zero-new-dep route is out.onnxruntimestill pulls in no torch/TF. - NO librosa (self-contained waveform-in graph, confirmed), NO torch/TF at runtime.
scipyis present in the dev venv (sofind_peaksworks there) but is not inrequirements-runtime.txt: either add it or keep the numpy peak-pick reimpl for the hermetic runtime. Decide at Phase 2.- Optical flow + face already covered by
opencv-python-headless. yamnet.onnxweight (~16 MB) committed tobackend/models/; no export step is needed because a self-contained waveform-in export already exists at HFzeropointnine/yamnet-onnx(inputwaveform[dynamic], outputs[frames,521]scores +[frames,1024]embeddings +[frames,64]mel). Verify its license before committing; keep a dev-onlyscripts/export_yamnet.py(tf2onnx) as the reproducible fallback.
Spike results (validated this session)
Ran the self-contained YAMNet ONNX on real podcli audio (a 38 s Deeptech Decoded clip, extracted with podcli's exact -ar 16000 -ac 1):
- No-torch confirmed:
torch/tensorflowabsent from the process before and after inference (onnxruntime only). - Correct classification: Speech 0.774 (dominant, correct for a talking clip), Silence 0.186, and Laughter/Cheering/Applause/Screaming all exactly 0.000, which is a correct negative control (serious monologue, no laughter). Mel front-end baked in the graph → featurization provably correct, no code to own.
- Negligible cost: 98 ms for 38 s of audio (~380× realtime); a 3 h video ≈ 28 s of audio-event analysis.
- Positive reaction detection confirmed on podcast content. Swept 32 real clips: the detector ranked the one comedic show + two chuckle moments above the dry technical majority (which scored exactly 0.00). Zooming in, laughter fires as a sharp 1-2 frame spike anti-correlated with Speech (e.g. laugh=0.23 while speech drops 1.00→0.11 at 13.5s, then back to speech), the exact reaction signature the 8s-backward-expansion relies on. Absolute values are modest (0.23-0.25 for brief chuckles in pre-cut clips) but that is a non-issue: a spike on a 0.00 baseline is trivially caught by prominence-based peak-picking, which is why per-video normalization (not absolute thresholds) is mandated above.
- Still worth a check (needs real party footage): amplitude/robustness on noisy handheld party audio with music and overlapping speech. YAMNet's known soft spot. If weak, AST ONNX (
onnx-community, pre-exported, mAP 0.485) is the drop-in fallback; needs a 128-bin log-mel numpy front-end.
Integration seams (from the code map)
- Detect-once hub:
backend/cli.py~658-720 (and MCP twinbackend/main.py:handle_suggest_clips~358). Energy is already computed once here (get_energy_profile) andface_mapextracted. All new channels compute here; fusion produces the saliency curve. - Scoring merge:
claude_suggest.suggest_with_claudenormalization (999) +687). Blend the LLM score in as the_select_top_by_score(transcript_semanticchannel; key the weights off the activeContentProfile. - Transcript-packer co-location:
transcript_packer.pack_transcript(~274) already flags top-RMS moments for the in-chat MCP agent. Add laughter/event/motion flags so the conversational path also sees them.
Contract-change tax
A new profile param threads the same ~12 hops the format field did: suggest-clips.handler.ts inputSchema, src/models/index.ts types, src/server.ts Zod (dual-declaration), src/ui/web-server.ts styledClips whitelist, python-executor stdin, main.py params, and cli.py's generate_clip call sites, or it silently reverts to the default. Follow the format precedent exactly.
Phasing
- Phase 0: profile scaffolding, zero behavior change. [DONE, PR #43 / Phase 2 branch]
ContentProfileabstraction added (profiles.py);profileparam threaded through the CLI (--profile, config, selection signature). Python-side defaultpodcastreproduces current selection. TS-side threading (MCP/webprofileparam) still open: see below. - Phase 1: audio-event channel (the isolated valuable core). [DONE, PR #43] YAMNet-ONNX laughter/cheer/applause/scream (
audio_events.py) computed in the detect-once hub; surfaced in the packed transcript and CLI heuristic. Podcast output unchanged unless the channel is weighted. Validated on 32 real clips. - Phase 2: fusion engine + saliency candidate source + party profile (audio-only). [DONE, this branch]
saliency.py: per-video robust-z normalization, weighted fusion, numpy peak-pick (NMS), reaction dilation, reaction-first candidate generation, 8 s backward expansion, quiet-point boundary snap. Wired to--profile party|actioninpodcli process. Deterministic on clean input; reactions detected with correct run-up windows. Gate remaining: tune thresholds/weights on real party footage (synthetic podcast-clip concat is not representative). - Phase 3: visual channels + action profile + multi-file pooling. Optical flow (OpenCV) + face-reaction channels; action profile; pool peaks across a folder of clips and rank globally ("best 15 bits from tonight" across 80 phone videos). Gate: catches a silent visual gag; folder-level ranking works.
- Phase 4 (optional): highlight reel renderer. Ordering, pacing, optional music-bed ducking, transitions: a thin renderer atop the detected moments, reusing the clip-render stack.
Guardrails
- Per-video normalization is mandatory: party clips vary wildly in level/room/camera; global normalization is wrong.
- Podcast profile must reproduce current behavior when party channels are weight 0. Ship behind the profile param; default is safe. This is the non-regression contract.
find_peaksdistanceparam is the NMS / min-gap. It deletes lower peaks within the gap automatically. Adaptive clip count = top percentile of the video's own prominence distribution, not an absolute N.- YAMNet ONNX must never import torch/TF at runtime. The export step is dev/CI only; the native runtime installs
onnxruntime(or reuses OpenCV-DNN) and feeds raw 16 kHz samples. - Reaction-expansion only fires for saliency candidates whose trigger is an audio-event class: never rewrite LLM-chosen podcast windows.