Skip to content

Open-source AI podcast clipper. Generate vertical clips with face tracking and burned-in captions. CLI, MCP server, web app.

Harden clips, transcription and multicam; add opening hooks and Omnilingual (#265)

Nika Siradze6426d14d1be3
diff --git a/.claude/commands/auto.md b/.claude/commands/auto.md
index 87aa8f5..dc373e7 100644
--- a/.claude/commands/auto.md
+++ b/.claude/commands/auto.md
@@ -1,6 +1,6 @@
---
description: One-verb pipeline — drop a video, confirm strategy, render clips
-allowed-tools: Read, Bash, mcp__podcli__transcribe_podcast, mcp__podcli__transcribe_start, mcp__podcli__transcribe_status, mcp__podcli__get_ui_state, mcp__podcli__set_video, mcp__podcli__suggest_clips, mcp__podcli__batch_create_clips, mcp__podcli__knowledge_base, mcp__podcli__clip_history
+allowed-tools: Read, Bash, mcp__podcli__transcribe_podcast, mcp__podcli__transcribe_start, mcp__podcli__job_status, mcp__podcli__get_ui_state, mcp__podcli__set_video, mcp__podcli__suggest_clips, mcp__podcli__batch_create_clips, mcp__podcli__knowledge_base, mcp__podcli__clip_history, mcp__podcli__record_decisions
argument-hint: [video-path-or-episode-slug] [optional: count e.g. "5 clips"]
triggers:
- auto
@@ -21,7 +21,7 @@ This command orchestrates the existing MCP tools on top of the compact packed tr
1. **Read, don't watch.** Reason about clips from the packed markdown view — not raw segments, not frame dumps.
2. **Strategy first, render after.** Propose the cut list and WAIT for user confirmation before calling `batch_create_clips`.
-3. **Knowledge base is context, not template.** If `` exists, read it for brand voice and format preferences. If not, infer from the content itself.
+3. **Knowledge base is context, not template.** If `.podcli/knowledge/` exists, read it for brand voice and format preferences. If not, infer from the content itself.
4. **Never silently render.** Every clip that ships must appear in the proposal the user approved.
5. **Every clip carries its own context.** A stranger who never heard the episode has to follow it from the first second. If the moment is an answer, the question comes with it.
@@ -46,14 +46,16 @@ This command orchestrates the existing MCP tools on top of the compact packed tr
- Call `transcribe_start(file_path)` → returns `{job_id, cached, estimate}` immediately.
- If `cached: true`, skip to step 3.
- Otherwise emit a short status to the user: _"Transcription started — estimated {estimate}. I'll check progress every 30s."_
- - Loop: call `transcribe_status(job_id, wait_seconds: 30)`. Between calls, emit ONE terse line to the user like `"Progress: 47% — pyannote diarization"`. Keep it to one line per poll — no repeat prose. Exit the loop when `done: true`.
+ - Loop: call `job_status(job_id, wait_seconds: 30)`. Between calls, emit ONE terse line to the user like `"Progress: 47%, pyannote diarization"`. Keep it to one line per poll, no repeat prose. Exit the loop when `done: true`.
- If `status: "error"`, stop and report the error.
3. Read the packed transcript: `get_ui_state(include_transcript: true)`. This returns a compact phrase-grouped view with speakers, silence gaps, and energy peaks.
- **If the header says speakers: 0**, stop and tell the user before going further. Without speaker labels you cannot tell a question from an answer, so the whole question-with-the-answer rule below is inert and the picks will be worse. Offer to re-transcribe with `transcribe_start(file_path, enable_diarization: true)`. Only continue without it if the user says to.
-4. If `` exists, read `01-brand-identity.md`, `02-voice-and-tone.md`, and `04-shorts-creation-guide.md` for show context. Skip silently if missing — `/auto` works on any content.
+4. If `.podcli/knowledge/` exists, read `01-brand-identity.md`, `02-voice-and-tone.md`, and `04-shorts-creation-guide.md` for show context. Skip silently if missing: `/auto` works on any content.
5. Call `clip_history` to see what's already been shipped for this episode. Avoid duplicates in the proposal.
-**Fallback**: if `transcribe_start` returns an error about the Web UI not running, tell the user and offer either (a) run `npm run ui` in another terminal then retry, or (b) fall back to the synchronous `transcribe_podcast` (no live progress, works silently).
+**Fallback**: if `transcribe_start` returns an error about the Web UI not running, tell the user and offer either (a) start the Web UI in another terminal then retry (`podcli studio` for a launcher install, `npm run ui` in a source checkout), or (b) fall back to the synchronous `transcribe_podcast` (no live progress, works silently).
+
+6. **Ask once, reuse the answer.** `get_ui_state` lists this episode's unanswered decisions under `OPEN QUESTIONS` (clip count, duration range, captions, language, thumbnails, delivery target). Ask whichever are relevant to this run, batched, not one dialog box per field, then call `record_decisions(video_path, ...)` with the answers. On every later run against this same video, those fields are already answered and won't appear in `OPEN QUESTIONS` again.
### Phase 2 — Topic Map (silent)
@@ -86,6 +88,7 @@ Work inside one topic at a time. Set boundaries by meaning, not by the clock.
- The question has to be inside the clip. `context_line` is a note for the editor, not a fix: nothing burns it into the video yet, so a clip that relies on it still ships with no setup.
- If the question rambles past roughly 8 seconds, use `segments` to keep the asked part and cut the rambling, or drop the moment.
- Never open on a word pointing back before the cut: "that", "it", "they", "yeah", "so", "exactly", "right", "which is why". Widen the start until the reference is inside the clip.
+- If the sharpest line sits mid-clip, you may pass it as `hook` (`{start, end, mode}`, 1-15 seconds) so it plays first. It must be a line actually spoken inside the clip, never invented text. `repeat` replays it in place; `move` lifts it out.
**end_second**
diff --git a/.claude/commands/bootstrap-knowledge.md b/.claude/commands/bootstrap-knowledge.md
index caad6f3..a1a2695 100644
--- a/.claude/commands/bootstrap-knowledge.md
+++ b/.claude/commands/bootstrap-knowledge.md
@@ -18,7 +18,7 @@ triggers:
## Before starting
-1. If `` has no files, run `podcli knowledge init` first so all 14 templates exist.
+1. If `.podcli/knowledge/` has no files, run `podcli knowledge init` first so all 14 templates exist.
2. Ask for whichever of these the user has not provided:
- Channel or podcast URL (YouTube channel, Spotify show, RSS feed)
- Or a few sentences about the show if nothing is published yet
diff --git a/.claude/commands/generate-titles.md b/.claude/commands/generate-titles.md
index 39224bc..e91f2d3 100644
--- a/.claude/commands/generate-titles.md
+++ b/.claude/commands/generate-titles.md
@@ -1,6 +1,6 @@
---
description: Generate 8 verified title options for a clip, moment, or episode
-allowed-tools: Read
+allowed-tools: Read, mcp__podcli__knowledge_base
argument-hint: [clip-transcript-or-moment-brief]
triggers:
- titles for
diff --git a/.claude/commands/plan-episode.md b/.claude/commands/plan-episode.md
index b90f3e1..77d030c 100644
--- a/.claude/commands/plan-episode.md
+++ b/.claude/commands/plan-episode.md
@@ -1,6 +1,6 @@
---
description: Design questions, story arc, and moment map BEFORE recording an episode
-allowed-tools: Read, Write
+allowed-tools: Read, Write, mcp__podcli__knowledge_base
argument-hint: [guest-name-and-company]
triggers:
- plan episode
diff --git a/.claude/commands/plan-thumbnails.md b/.claude/commands/plan-thumbnails.md
index a665161..6008e93 100644
--- a/.claude/commands/plan-thumbnails.md
+++ b/.claude/commands/plan-thumbnails.md
@@ -66,6 +66,7 @@ What is the single most compelling image or concept?
- Guest photo requirements
- Background suggestion
- Special visual elements
+- **Layout:** `single` (one face) or `pair` (two people). Propose `pair` for interview clips where the exchange is the point: a question and its answer, a disagreement, a reaction. podcli takes both faces from the clip itself, guest on the left and host on the right. Set it with `manage_thumbnail_config` (`set_layout`), the "Two people" toggle on the clip page, or `podcli thumbnails --layout pair`. Add `--swap` to flip sides, or `--left-image` and `--right-image` to name the people. It falls back to `single` when podcli cannot tell two people apart, so keep a single-face brief ready.
### Step 4: Quality Check
- [ ] Readable at phone screen size
@@ -88,6 +89,7 @@ What is the single most compelling image or concept?
**Shorts (9:16):**
- Text: "[LINE 1] / [LINE 2 — accent]"
+- Layout: [single / pair: guest left, host right]
- Visual: [action shot / dramatic imagery / B-roll]
- Text position: Lower third, centered
diff --git a/.claude/commands/process-transcript.md b/.claude/commands/process-transcript.md
index bf609fb..7426bf5 100644
--- a/.claude/commands/process-transcript.md
+++ b/.claude/commands/process-transcript.md
@@ -1,6 +1,6 @@
---
description: Extract, score, and classify the best moments from a raw podcast transcript
-allowed-tools: Read, Write
+allowed-tools: Read, Write, mcp__podcli__knowledge_base
argument-hint: [transcript-file-or-paste]
triggers:
- transcript
@@ -96,6 +96,8 @@ Before scoring, fix each flagged moment's edges and state its payoff.
**Then run the standalone check.** Name what the viewer must already know. If it is anything other than nothing, the start moves back until the clip covers it. If it cannot, drop the moment.
+**Optionally mark an opening hook.** If the sharpest line sits mid-clip, it can play first as a `hook` (1-15 seconds, `repeat` replays it in place, `move` lifts it out). It must be a line actually spoken inside the clip, quoted verbatim with its timestamps. Never invent hook text.
+
### Phase 4: Score Each Moment
For every flagged moment, score on four dimensions (1-5 each):
@@ -182,6 +184,7 @@ Format: comma-separated, under 500 characters.
**Payoff:** [What the viewer walks away with. One sentence, second person.]
**Needs:** [nothing | what the viewer must already know]
**Setup line:** [The question this answers, in one line, or omit when the clip carries its own setup]
+**Opening hook:** [Optional. "Verbatim line" XX:XX-XX:XX, repeat or move. Omit when the clip opens strong on its own]
**Why it works:** [One sentence explaining the appeal]
diff --git a/.claude/commands/produce-shorts.md b/.claude/commands/produce-shorts.md
index cb1f1aa..682f929 100644
--- a/.claude/commands/produce-shorts.md
+++ b/.claude/commands/produce-shorts.md
@@ -1,6 +1,6 @@
---
description: Full pipeline from transcript to publish-ready content package
-allowed-tools: Read, Write, Edit, Task
+allowed-tools: Read, Write, Edit, Task, mcp__podcli__knowledge_base, mcp__podcli__get_ui_state, mcp__podcli__record_decisions
argument-hint: [transcript-file-or-episode-number]
triggers:
- process episode
@@ -29,6 +29,8 @@ Read the full knowledge base with the `knowledge_base` MCP tool:
- `07-thumbnail-guide.md` — visual specs
- `13-learnings.md` — past retro patterns (what worked, what didn't)
+If a video is already set (`get_ui_state`), check its `OPEN QUESTIONS` block for unanswered episode decisions (clip count, duration range, captions, language, thumbnails, delivery target). Ask whichever are relevant to this run, batched into one NEEDS_INPUT prompt rather than one per field, then call `record_decisions(video_path, ...)` so the next run against this video doesn't ask again.
+
---
## Inputs
@@ -52,6 +54,8 @@ Each phase calls the corresponding skill's logic. Each phase reports its own Com
Extract guest info, flag 15-20 moments, anchor each one (boundaries by meaning, question pulled in or carried as a setup line, payoff written before any title), score them, select top moments, classify by content type, check for duplicates.
+A moment may open with a `hook`: a 1-15 second line spoken inside the clip, played first. Quote it from the transcript. Never invent it.
+
### Phase 2: Title Development
*Runs `/generate-titles` logic per moment*
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index aa82145..dabe620 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -87,6 +87,18 @@ jobs:
- run: node scripts/gen-docs-manifest.mjs
- run: node scripts/check-docs-drift.mjs
+ release-hygiene:
+ name: Release hygiene check
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ persist-credentials: false
+ - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+ with:
+ node-version: "20"
+ - run: node scripts/release-hygiene.mjs
+
python:
name: Python tests
runs-on: ${{ matrix.os }}
@@ -102,6 +114,17 @@ jobs:
with:
python-version: "3.12"
cache: "pip"
+ - name: Install ffmpeg
+ # Without this, every test that shells out to ffmpeg/ffprobe skips.
+ # About 22 tests were silently not running on any OS.
+ shell: bash
+ run: |
+ case "${{ runner.os }}" in
+ Linux) sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg ;;
+ macOS) brew install --quiet ffmpeg ;;
+ Windows) choco install ffmpeg -y --no-progress ;;
+ esac
+ ffmpeg -version | head -1
- name: Install minimal test deps
# Skip whisper/torch — tests don't exercise them and they balloon CI time.
run: |
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..c7b0d65
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,17 @@
+# AGENTS.md
+
+This file exists for coding agents (OpenAI Codex, opencode, Aider, Cursor Agent,
+and others) that read `AGENTS.md` by convention.
+
+- `CLAUDE.md` is the primary instruction document: project layout, the MCP tool
+ table, the knowledge base, and the quality gate all live there and are not
+ repeated here.
+- `AGENTS.podstack.md` covers cross-tool PodStack usage: how each host runs the
+ content-production commands (`/plan-episode`, `/process-transcript`,
+ `/generate-titles`, and the rest), and where each host installs them.
+- `.claude/commands/*.md` are the PodStack command sources. Claude Code reads
+ them directly from that path; `podcli auto` (and the other PodStack
+ commands) also installs them as Codex skills under `~/.codex/skills/` when
+ the `codex` CLI is present.
+
+Start with `CLAUDE.md`, then `AGENTS.podstack.md` for command-by-command detail.
diff --git a/AGENTS.podstack.md b/AGENTS.podstack.md
index a1bae00..c21a55e 100644
--- a/AGENTS.podstack.md
+++ b/AGENTS.podstack.md
@@ -8,7 +8,7 @@ PodStack turns your AI tool into a podcast content team: Episode Architect, Cont
## How to use
-Each skill below is a self-contained instruction file in `commands/` (or `.claude/commands/`, `.codex/prompts/`, `.cursor/rules/`, `.opencode/commands/`, depending on which host installed it).
+Each skill below is a self-contained instruction file in `commands/` (or `.claude/commands/`, `~/.codex/skills/<name>/SKILL.md`, `.cursor/rules/`, `.opencode/commands/`, depending on which host installed it).
**To run a skill:** ask your agent to "run the [skill-name] skill" or invoke its slash command (`/[skill-name]`) where supported. The agent opens the corresponding file and follows it step by step.
@@ -30,7 +30,7 @@ Each skill below is a self-contained instruction file in `commands/` (or `.claud
- **role:** Episode Architect
- **description:** Design questions, story arc, and moment map BEFORE recording
-- **allowed-tools:** Read, Write
+- **allowed-tools:** Read, Write, mcp__podcli__knowledge_base
- **triggers:** plan episode, upcoming recording, guest prep, prepare for interview
- **outputs:** episode plan written to `episodes/ep[XX]-[guest]-plan.md`
- **next:** record → `/process-transcript`
@@ -39,7 +39,7 @@ Each skill below is a self-contained instruction file in `commands/` (or `.claud
- **role:** Content Analyst
- **description:** Extract, score, classify best moments from a raw transcript
-- **allowed-tools:** Read, Write
+- **allowed-tools:** Read, Write, mcp__podcli__knowledge_base
- **triggers:** transcript, process transcript, extract moments, podcast transcript
- **outputs:** moment brief with timestamps, scores, titles, thumbnails, descriptions
- **next:** `/generate-titles` or `/produce-shorts`
@@ -48,7 +48,7 @@ Each skill below is a self-contained instruction file in `commands/` (or `.claud
- **role:** Title Writer
- **description:** Generate 8 verified title options for a clip or moment
-- **allowed-tools:** Read
+- **allowed-tools:** Read, mcp__podcli__knowledge_base
- **triggers:** titles for, title options, write titles, generate titles
- **outputs:** 8 titles + 2 top picks with rationale
@@ -80,7 +80,7 @@ Each skill below is a self-contained instruction file in `commands/` (or `.claud
- **role:** Producer (master orchestrator)
- **description:** Full pipeline from transcript to publish-ready content package
-- **allowed-tools:** Read, Write, Edit, Task
+- **allowed-tools:** Read, Write, Edit, Task, mcp__podcli__knowledge_base
- **triggers:** process episode, produce shorts, full pipeline, prep episode, make content package
- **outputs:** complete content package in `episodes/ep[XX]-[guest]-content-package.md`
- **orchestrates:** process-transcript → generate-titles → generate-descriptions → plan-thumbnails → review-content
@@ -114,16 +114,16 @@ Skill files read the 14 knowledge files at `.podcli/knowledge/`; the full file t
PodStack ships one source-of-truth (`commands/`) and installs to the right location for each tool:
-| Host | Install location | Primary doc |
-|------|-----------------|-------------|
-| Claude Code | `.claude/commands/*.md` | `CLAUDE.md` |
-| OpenAI Codex | `.codex/prompts/*.md` | `AGENTS.podstack.md` (this file) |
-| Cursor | `.cursor/rules/*.mdc` | `AGENTS.podstack.md` |
-| opencode | `.opencode/commands/*.md` | `AGENTS.podstack.md` |
-| Generic | `commands/*.md` | `AGENTS.podstack.md` |
+| Host | Install location | Installed by | Primary doc |
+|------|-----------------|--------------|-------------|
+| Claude Code | `.claude/commands/*.md` (per project) | `podcli auto` / any PodStack command | `CLAUDE.md` |
+| OpenAI Codex | `~/.codex/skills/<name>/SKILL.md` (global) | `podcli auto` / any PodStack command, when the `codex` CLI is on PATH | `AGENTS.podstack.md` (this file) |
+| Cursor | `.cursor/rules/*.mdc` | not automated yet, copy by hand | `AGENTS.podstack.md` |
+| opencode | `.opencode/commands/*.md` | not automated yet, copy by hand | `AGENTS.podstack.md` |
+| Generic | `commands/*.md` | not automated yet, copy by hand | `AGENTS.podstack.md` |
-These command files ship with podcli; place the set for your tool (left column) in
-its command dir. See `README.md` for per-host usage examples.
+Claude and Codex installs are automatic and kept in sync on upgrade; see `README.md`
+for per-host usage examples and manual steps for the other hosts.
---
diff --git a/CLAUDE.md b/CLAUDE.md
index a6332ba..644a417 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -31,7 +31,7 @@ Both share the same knowledge base at `.podcli/knowledge/`.
## MCP tools (podcli engine)
-All 27 tools registered by the MCP server.
+All 30 tools registered by the MCP server.
**Transcription and input**
@@ -43,6 +43,8 @@ All 27 tools registered by the MCP server.
| `set_video` | Set the working video without transcribing |
| `import_transcript` | Import an external transcript with word-level timestamps, skips Whisper |
| `parse_transcript` | Parse a speaker-labeled plain text transcript into word-level timestamps |
+| `compare_transcription_engines` | Transcribe one sample with two engines and write a side-by-side disagreement report |
+| `mine_channel` | List a YouTube channel's uploads or mine one video's existing captions, without downloading the video |
**Clip workflow**
@@ -56,6 +58,7 @@ All 27 tools registered by the MCP server.
| `batch_create_clips` | Render multiple clips in one batch |
| `manage_reel` | Build a highlights reel: detect once, edit moments, rebuild without re-detecting |
| `analyze_energy` | Analyze audio energy levels to find high-energy moments |
+| `record_decisions` | Record per-episode decisions (clip count, duration range, captions, language, thumbnails, delivery target) so later runs never re-ask |
**Full-episode editing**
diff --git a/README.md b/README.md
index 0dce7df..1776c7a 100644
--- a/README.md
+++ b/README.md
@@ -97,7 +97,7 @@ Clips land in `podcli-clips/` in the directory you ran it from, so each show kee
**Shipping it**
-- 27 MCP tools, so an agent can transcribe, score, render, and publish through conversation
+- 30 MCP tools, so an agent can transcribe, score, render, and publish through conversation
- YouTube publishing plus performance analytics to see which clips landed
- DaVinci Resolve export as FCPXML when you want to finish by hand
- Presets, clip history with duplicate detection, and a transcript cache
@@ -108,7 +108,7 @@ If you are weighing podcli against the cloud clippers, this is the difference:
- Runs locally. Transcription and rendering happen on your machine by default, so episodes stay there. Only the optional cloud engine (AssemblyAI) and publishing to YouTube send anything out.
- Free and open source under AGPL-3.0. Exports are unlimited, full quality, and watermark-free.
-- Agent-native. 27 MCP tools let Claude Code or Codex drive the whole flow, transcription through publishing.
+- Agent-native. 30 MCP tools let Claude Code or Codex drive the whole flow, transcription through publishing.
- A knowledge base keeps titles, captions, and descriptions in your show's voice, and stops the engine from resuggesting moments you already published.
- DaVinci Resolve handoff. Export any clip as FCPXML when you want to finish the edit yourself.
@@ -117,7 +117,7 @@ If you are weighing podcli against the cloud clippers, this is the difference:
podcli is an [MCP](https://modelcontextprotocol.io) server, so an agent can transcribe, suggest clips, and render them through conversation.
```bash
-podcli mcp install # registers it with Claude Code
+podcli mcp install # registers it with Claude Code and Codex, whichever CLI is on PATH
```
Claude Desktop and Codex setup is in the [MCP docs](https://podcli.com/docs/mcp-server).
diff --git a/backend/cli.py b/backend/cli.py
index 2d05929..c7ea09c 100644
--- a/backend/cli.py
+++ b/backend/cli.py
@@ -124,16 +124,20 @@ def _json_object_arg(raw: str | None, name: str):
return parsed
-def _cached_face_map(video_path: str):
- """Face maps are keyed by video content, not by transcript, so an imported
- transcript can still borrow the map from an earlier run on the same file."""
+def _cached_transcript(video_path: str) -> dict:
+ """The cached transcript for this video, or {} when there is none."""
try:
from services.transcript_packer import load_cached_transcript_for_video
- cached = load_cached_transcript_for_video(video_path)
+ return load_cached_transcript_for_video(video_path) or {}
except Exception:
- return None
- return (cached or {}).get("face_map")
+ return {}
+
+
+def _cached_face_map(video_path: str):
+ """Face maps are keyed by video content, not by transcript, so an imported
+ transcript can still borrow the map from an earlier run on the same file."""
+ return _cached_transcript(video_path).get("face_map")
def _suggestions_session_path(cache_hash: str) -> str:
@@ -557,6 +561,8 @@ def cmd_studio(args):
cmd += ["--progress-color", args.progress_color]
if getattr(args, "cards", None):
cmd += ["--cards", args.cards]
+ if getattr(args, "hook", None):
+ cmd += ["--hook", args.hook]
if getattr(args, "brand", None):
cmd += ["--brand", args.brand]
if getattr(args, "style", None):
@@ -798,7 +804,7 @@ def _open_multicam_session(target: str, people):
session = mc.new_session(folder=target, people=people, progress_callback=report)
done()
else:
- session = mc.MulticamSession.load(target)
+ session = mc.open_session(target)
# Reopening keeps the saved names; --people on a re-run renames them.
if people and people != [p.name for p in session.people]:
session = mc.rename_people(session, people)
@@ -994,10 +1000,12 @@ def _run_multicam(args, mc, target: str):
def _render_multicam(args, mc, session):
report, done = _multicam_progress("Rendering")
- outputs = mc.render_session(session, stems=not args.no_stems, progress_callback=report)
+ outputs = mc.render_session(session, stems=not args.no_stems, validate=args.validate, progress_callback=report)
done()
for path in [outputs["video"], *(outputs.get("stems") or [])]:
print(f" ✓ {path}")
+ for warning in (outputs.get("validation") or {}).get("warnings") or []:
+ print(f" ! {warning}", file=sys.stderr)
def _pull_multicam(args, mc, target: str):
@@ -1296,7 +1304,12 @@ def cmd_process(args):
if skip_transcript:
# Reuse an existing transcript so highlight boundaries snap to whole sentences;
# only skip transcription outright when there is none (true no-dialogue footage).
- cached = load_cached_transcript_for_video(video_path)
+ cached = load_cached_transcript_for_video(
+ video_path,
+ engine=os.environ.get("PODCLI_ENGINE"),
+ model=config.get("whisper_model", "base"),
+ language=config.get("language"),
+ )
if cached and not config.get("no_cache", False):
words = cached["words"]
segments = cached["segments"]
@@ -1337,8 +1350,16 @@ def cmd_process(args):
else:
print(" No cached face map, crop falls back to per-clip face tracking")
elif not skip_transcript:
- # Check cache first
- cached = load_cached_transcript_for_video(video_path)
+ # Check cache first, with the same (engine, model, language) the
+ # transcribe call below would run with, so a hit here is guaranteed
+ # to be the combo this invocation actually asked for, not a
+ # different one that happens to share the file.
+ cached = load_cached_transcript_for_video(
+ video_path,
+ engine=os.environ.get("PODCLI_ENGINE"),
+ model=config.get("whisper_model", "base"),
+ language=config.get("language"),
+ )
if cached and not config.get("no_cache", False):
print(" [1/4] Loaded from cache (instant)")
words = cached["words"]
@@ -1398,8 +1419,16 @@ def cmd_process(args):
segments = result["segments"]
print(f" Done: {len(segments)} segments, {len(words)} words")
- # Save to cache for next run
- save_cached_transcript_for_video(video_path, result)
+ # Save to cache for next run, under the same key the read above
+ # checked. result["engine"] is what actually ran, which can
+ # differ from the env var on a fallback (see transcribe_file).
+ save_cached_transcript_for_video(
+ video_path,
+ result,
+ engine=result.get("engine") or os.environ.get("PODCLI_ENGINE"),
+ model=config.get("whisper_model", "base"),
+ language=config.get("language"),
+ )
# Apply word corrections (Whisper misheard proper nouns, brand names)
from services.corrections import apply_corrections
@@ -1668,7 +1697,7 @@ def cmd_process(args):
_thumb_photo = None
if _thumb_enabled:
try:
- from services.thumbnail_ai import generate_variations as _tv, thumbnail_to_video_frame as _ttv
+ from services.thumbnail_ai import render_variations as _tv, thumbnail_to_video_frame as _ttv
_thumb_gen = _tv
_thumb_to_video = _ttv
_thumb_logo = config.get("logo_path") or None
@@ -1718,6 +1747,7 @@ def cmd_process(args):
motion=config.get("motion"),
bookend_fade=config.get("bookend_fade", 0.0),
keep_segments=clip.get("segments"),
+ hook=_clip_hook(clip),
face_map=face_map,
allow_ass_fallback=config.get("allow_ass_fallback", False),
use_ass_captions=config.get("use_ass_captions", False),
@@ -1741,7 +1771,7 @@ def cmd_process(args):
output_path=os.path.join(clip_thumb_dir, "_lead_frame.jpg"),
start_second=result.get("start_second", clip.get("start_second", 0)),
)
- thumb_paths = _thumb_gen(
+ rendered_thumbs = _thumb_gen(
title=clip.get("title", f"Clip {i+1}"),
output_dir=clip_thumb_dir,
photo_path=lead_frame or _thumb_photo,
@@ -1750,7 +1780,14 @@ def cmd_process(args):
end_second=result.get("end_second", clip.get("end_second")),
logo_path=_thumb_logo,
config=_thumb_style,
+ grounding=_clip_grounding(clip),
+ face_map=face_map,
+ segments=segments,
)
+ thumb_paths = rendered_thumbs["paths"]
+ _pair = rendered_thumbs["pair"]
+ if _pair and _pair["layout"] != "pair":
+ print(f" ℹ {_pair['reason']}")
if thumb_paths and _thumb_placement == "off":
print(f" + {len(thumb_paths)} thumbnail(s) in "
f"{os.path.basename(clip_thumb_dir)}/")
@@ -1945,6 +1982,7 @@ def cmd_process(args):
outro_path=config.get("outro_path") or None,
intro_path=config.get("intro_path") or None,
keep_segments=clip.get("segments"),
+ hook=_clip_hook(clip),
face_map=face_map,
allow_ass_fallback=config.get("allow_ass_fallback", False),
use_ass_captions=config.get("use_ass_captions", False),
@@ -2196,6 +2234,24 @@ def _review_clips(clips: list, segments: list, energy_scores: list | None, confi
print(f" No additional suggestions found.")
+def _clip_hook(clip: dict) -> dict | None:
+ """The clip's opening hook when it still fits the clip, else None.
+
+ Review can move a clip's edges after the hook was proposed. A hook left
+ outside the body would fail the whole render, so it is dropped with a note
+ and the clip renders without it.
+ """
+ from services.opening_hook import validate_hook
+
+ if not clip.get("hook"):
+ return None
+ try:
+ return validate_hook(clip["hook"], clip["start_second"], clip["end_second"], clip.get("segments"))
+ except ValueError as e:
+ print(f" Opening hook dropped: {e}")
+ return None
+
+
def _filter_duplicate_clip_suggestions(candidates: list, existing: list, overlap_threshold: float = 5.0) -> list:
"""Drop suggestions that significantly overlap already-selected clips."""
filtered = []
@@ -2272,6 +2328,7 @@ def _post_render_loop(
outro_path=config.get("outro_path") or None,
intro_path=config.get("intro_path") or None,
keep_segments=clip.get("segments"),
+ hook=_clip_hook(clip),
face_map=face_map,
allow_ass_fallback=config.get("allow_ass_fallback", False),
use_ass_captions=config.get("use_ass_captions", False),
@@ -2459,6 +2516,7 @@ def _post_render_loop(
outro_path=config.get("outro_path") or None,
intro_path=config.get("intro_path") or None,
keep_segments=f_clip.get("segments"),
+ hook=_clip_hook(f_clip),
face_map=face_map,
allow_ass_fallback=config.get("allow_ass_fallback", False),
use_ass_captions=config.get("use_ass_captions", False),
@@ -2516,6 +2574,7 @@ def _post_render_loop(
outro_path=config.get("outro_path") or None,
intro_path=config.get("intro_path") or None,
keep_segments=nc.get("segments"),
+ hook=_clip_hook(nc),
face_map=face_map,
allow_ass_fallback=config.get("allow_ass_fallback", False),
use_ass_captions=config.get("use_ass_captions", False),
@@ -3130,12 +3189,30 @@ def cmd_thumbnail_config(args):
raise ValueError(f"unknown thumbnail-config action: {action}")
+def _clip_grounding(clip: dict) -> dict | None:
+ """A suggestion's payoff, question and opening line, for grounding thumbnail copy."""
+ grounding = {k: clip.get(k) for k in ("payoff", "context_line", "preview_text")}
+ return grounding if any(grounding.values()) else None
+
+
+def _grounding_from_args(args) -> dict | None:
+ """Collect the clip's payoff/question/opening-line CLI flags into the dict
+ thumbnail_ai expects, or None if the caller passed none of them (e.g. a
+ bare title with no clip behind it, as in the standalone thumbnail studio)."""
+ grounding = {
+ "payoff": getattr(args, "payoff", None),
+ "context_line": getattr(args, "context_line", None),
+ "preview_text": getattr(args, "preview_text", None),
+ }
+ return grounding if any(grounding.values()) else None
+
+
def cmd_thumbnail_options(args):
"""Emit candidate headline text pairs and face frames for the thumbnail picker."""
from services.thumbnail_ai import generate_headline_variations, extract_candidate_frames
os.makedirs(args.output, exist_ok=True)
- texts = generate_headline_variations(args.title, args.texts) or []
+ texts = generate_headline_variations(args.title, args.texts, grounding=_grounding_from_args(args)) or []
frames = []
if args.video:
frames = extract_candidate_frames(
@@ -3180,34 +3257,80 @@ def _check_frame(path):
sys.exit(1)
+def _pair_from_args(args, output_dir: str) -> dict | None:
+ """The two-person panels the thumbnail flags ask for, or None for one face.
+
+ The template's own layout applies when --layout is not given. Speaker
+ turns and the face map come from the cached transcript of --video.
+ """
+ from services.thumbnail_ai import _load_brand_config, resolve_pair
+
+ video = getattr(args, "video", None)
+ cached = _cached_transcript(video) if video else {}
+ try:
+ return resolve_pair(
+ _load_brand_config(), output_dir, video,
+ getattr(args, "start", None), getattr(args, "end", None),
+ layout=getattr(args, "layout", None),
+ left_image=getattr(args, "left_image", None),
+ right_image=getattr(args, "right_image", None),
+ swap=bool(getattr(args, "swap", False)),
+ face_map=cached.get("face_map"), segments=cached.get("segments"),
+ )
+ except ValueError as err:
+ print(f"thumbnail failed: {err}", file=sys.stderr)
+ sys.exit(1)
+
+
+def _layout_report(pair: dict | None) -> dict:
+ """What a thumbnail result says about its layout: which people, from which source seconds."""
+ if not pair:
+ return {"layout": "single"}
+ if pair["layout"] != "pair":
+ return {"layout": "single", "note": pair["reason"]}
+ return {"layout": "pair", "roles": pair["roles"], "swapped": pair["swapped"], "people": pair["people"]}
+
+
def cmd_thumbnail_render(args):
"""Render one final thumbnail from a chosen frame + headline.
Empty line1/line2 let the AI write the text; a chosen frame is used as-is.
+ With the pair layout the two people come from --video between --start and
+ --end, or from --left-image and --right-image, and the frame is the
+ fallback when two people cannot be told apart.
"""
from services.thumbnail_ai import generate_thumbnail_with_template
from services.asset_store import resolve_logo
- _check_frame(args.frame)
- frame_info = json.loads(args.frame_info) if args.frame_info else None
+ pair = _pair_from_args(args, os.path.dirname(os.path.abspath(args.output)))
+ people = pair["people"] if pair and pair["layout"] == "pair" else None
+ if not people:
+ if not args.frame:
+ reason = f" {pair['reason']}" if pair else ""
+ print(f"thumbnail render failed: no frame to fall back on.{reason}", file=sys.stderr)
+ sys.exit(1)
+ _check_frame(args.frame)
+ frame_info = json.loads(args.frame_info) if args.frame_info and not people else None
out = generate_thumbnail_with_template(
title=args.title,
- frame_path=args.frame,
+ frame_path=None if people else args.frame,
output_path=args.output,
logo_path=resolve_logo(args.logo) if args.logo else None,
frame_info=frame_info,
line1_override=args.line1 or None,
line2_override=args.line2 or None,
+ grounding=_grounding_from_args(args),
+ people=people,
)
if not out:
print("thumbnail render failed", file=sys.stderr)
sys.exit(1)
- print(json.dumps({"path": out}))
+ print(json.dumps({"path": out, **_layout_report(pair)}))
def cmd_thumbnails(args):
"""Generate thumbnail variations for a title."""
- from services.thumbnail_ai import generate_variations
+ from services.thumbnail_ai import render_variations
from services.asset_store import resolve as resolve_asset, resolve_logo
accent = "\033[38;2;212;135;74m"
@@ -3251,23 +3374,41 @@ def cmd_thumbnails(args):
print(f"\n {bold}Generating {args.variations} thumbnail variations...{reset}")
print(f" Title: {accent}{args.title}{reset}")
- paths = generate_variations(
- title=args.title,
- output_dir=args.output,
- photo_path=photo,
- video_path=video,
- start_second=getattr(args, "start", None),
- end_second=getattr(args, "end", None),
- logo_path=logo,
- config={"variations": args.variations},
- line1=getattr(args, "line1", None),
- line2=getattr(args, "line2", None),
- )
+ cached = _cached_transcript(video) if video else {}
+ try:
+ rendered = render_variations(
+ title=args.title,
+ output_dir=args.output,
+ photo_path=photo,
+ video_path=video,
+ start_second=getattr(args, "start", None),
+ end_second=getattr(args, "end", None),
+ logo_path=logo,
+ config={"variations": args.variations},
+ line1=getattr(args, "line1", None),
+ line2=getattr(args, "line2", None),
+ layout=getattr(args, "layout", None),
+ left_image=getattr(args, "left_image", None),
+ right_image=getattr(args, "right_image", None),
+ swap=bool(getattr(args, "swap", False)),
+ face_map=cached.get("face_map"),
+ segments=cached.get("segments"),
+ )
+ except ValueError as err:
+ print(f" {red}✗{reset} {err}", file=sys.stderr)
+ sys.exit(1)
+ paths = rendered["paths"]
+ report = _layout_report(rendered["pair"])
if as_json:
- print(json.dumps({"paths": paths}))
+ print(json.dumps({"paths": paths, **report}))
return
+ if report.get("note"):
+ print(f" {gray}{report['note']}{reset}")
+ for person in report.get("people", []):
+ when = f" at {person['source_time']:.1f}s" if person.get("source_time") is not None else ""
+ print(f" {gray}{person['side'].capitalize()}: {person.get('role') or 'person'}{when}{reset}")
for p in paths:
print(f" {green}✓{reset} {p}")
print(f"\n {gray}Open the folder to preview and pick the best one.{reset}\n")
@@ -4088,6 +4229,40 @@ def cmd_cache(args):
print(f" {gray}Run {accent}podcli cache clear{reset} {gray}to delete all{reset}\n")
+def cmd_compare_engines(args):
+ """Transcribe the same sample window with two engines and report where they disagree."""
+ from services.engine_comparison import compare_engines
+
+ accent = "\033[38;2;212;135;74m"
+ gray = "\033[38;5;245m"
+ reset = "\033[0m"
+
+ if not os.path.exists(args.video):
+ print(f"podcli: file not found: {args.video}", file=sys.stderr)
+ sys.exit(1)
+
+ print(f"Comparing {accent}{args.engine_a}{reset} vs {accent}{args.engine_b}{reset} "
+ f"on {args.duration:.0f}s starting at {args.start:.0f}s...")
+
+ report = compare_engines(
+ args.video,
+ args.engine_a,
+ args.engine_b,
+ start_seconds=args.start,
+ duration_seconds=args.duration,
+ window_seconds=args.window,
+ model_size=args.model_size,
+ language=args.language,
+ output_dir=args.output,
+ )
+
+ print(f"\nOverall disagreement: {accent}{report['overall_disagreement']:.2f}{reset} "
+ f"(0 = identical output, 1 = completely different; not accuracy against a transcript)")
+ if report["both_empty_window_count"]:
+ print(f"{gray}{report['both_empty_window_count']} window(s) had no words from either engine{reset}")
+ print(f"{gray}Wrote {report['json_path']} and {report['html_path']}{reset}")
+
+
def cmd_info(args):
"""Show system info."""
from services.encoder import get_encoder_info
@@ -4820,6 +4995,15 @@ def _first_run_setup() -> bool:
return True
+def _add_pair_args(parser) -> None:
+ parser.add_argument("--layout", choices=["single", "pair"],
+ help="single: one face. pair: the guest left and the host right, from the clip's "
+ "footage. Defaults to the template's layout")
+ parser.add_argument("--left-image", dest="left_image", help="Image of the person on the left (pair layout)")
+ parser.add_argument("--right-image", dest="right_image", help="Image of the person on the right (pair layout)")
+ parser.add_argument("--swap", action="store_true", help="Swap the two people's sides (pair layout)")
+
+
def main():
parser = argparse.ArgumentParser(
prog="podcli",
@@ -4856,7 +5040,7 @@ def main():
help="Render on podcli.com instead of this machine (needs `podcli login`)")
proc.add_argument("--template-id",
help="Cut in a saved cloud template, by id (with --cloud)")
- proc.add_argument("--engine", choices=["whisper-py", "whispercpp", "assemblyai"], help="Transcription engine (default: whisper-py; whispercpp is local; assemblyai uses ASSEMBLYAI_API_KEY)")
+ proc.add_argument("--engine", choices=["whisper-py", "whispercpp", "assemblyai", "omnilingual"], help="Transcription engine (default: whisper-py; whispercpp is local; assemblyai uses ASSEMBLYAI_API_KEY)")
proc.add_argument("--language", help="Language of the recording (e.g. es, pt-BR, ka). Auto-detect if omitted.")
proc.add_argument("--assemblyai-api-key", help="AssemblyAI API key for --engine assemblyai. Prefer ASSEMBLYAI_API_KEY; command-line secrets can appear in process listings.")
proc.add_argument("--fast", action="store_true", help="Draft mode: tiny Whisper, heuristic selection, center crop, low quality")
@@ -4996,6 +5180,10 @@ def main():
"File > Export > Timeline > FCP 7 XML)")
mc_p.add_argument("--no-render", action="store_true", dest="no_render", help="Skip the MP4 render (fast, export only)")
mc_p.add_argument("--no-stems", action="store_true", dest="no_stems", help="Skip the per-person WAV files")
+ mc_p.add_argument("--validate", choices=["sample", "full"], default="sample",
+ help="How hard to check the rendered episode decodes cleanly: 'sample' (default) checks the "
+ "first and last 10s plus a few points in between; 'full' decodes the whole thing, which "
+ "costs minutes per hour of 1080p")
mc_p.add_argument("--resync", action="store_true",
help="Sync every file again, including offsets you set by hand")
mc_p.add_argument("-y", "--yes", action="store_true", help="Don't stop to review guessed roles")
@@ -5011,7 +5199,7 @@ def main():
mc_p.add_argument("--transcript", action="store_true",
help="Transcribe the episode with each word credited to whoever's mic was speaking")
mc_p.add_argument("--model", default="base", help="Whisper model for --transcript (default base)")
- mc_p.add_argument("--engine", choices=["whisper-py", "whispercpp", "assemblyai"],
+ mc_p.add_argument("--engine", choices=["whisper-py", "whispercpp", "assemblyai", "omnilingual"],
help="Transcription engine for --transcript (default: the one podcli is set up with)")
mc_p.add_argument("--activity", action="store_true", help="Report who speaks when (talk time, or spans with --json)")
mc_p.add_argument("--cloud", action="store_true",
@@ -5028,7 +5216,7 @@ def main():
studio.add_argument("--end", type=float, help="Fragment end (seconds)")
studio.add_argument("--paragraph", help="Find the fragment by matching this text in the transcript")
studio.add_argument("--language", help="Transcription language (e.g. es). Auto-detect if omitted.")
- studio.add_argument("--engine", choices=["whisper-py", "whispercpp", "assemblyai"], help="Transcription engine")
+ studio.add_argument("--engine", choices=["whisper-py", "whispercpp", "assemblyai", "omnilingual"], help="Transcription engine")
studio.add_argument("--transcript", help="Word timings JSON for this video ({words:[...]} or a list); skips transcription")
studio.add_argument("--assemblyai-api-key", help="AssemblyAI API key for --engine assemblyai. Prefer ASSEMBLYAI_API_KEY; command-line secrets can appear in process listings.")
studio.add_argument("--caption-style", choices=["hormozi", "karaoke", "subtle", "branded", "outline"], default="hormozi")
@@ -5056,6 +5244,10 @@ def main():
help="Draw how much of the clip is left along the bottom edge")
studio.add_argument("--progress-color", dest="progress_color")
studio.add_argument("--cards", help="On-screen cards as JSON, each with kind/start/end")
+ studio.add_argument("--hook", help='Opening hook as JSON, on the source clock: '
+ '{"start":41.2,"end":45.8,"mode":"repeat"}. '
+ "Plays a 1-15 s passage from inside the fragment first; "
+ '"move" lifts it out of the body')
studio.add_argument("--brand", help="Show colours as JSON: "
'{"accent":"#4C9DF5","ink":"#FFFFFF","surface":"#0A0D14"}')
studio.add_argument("--style", help='Visual theme as JSON: '
@@ -5176,6 +5368,7 @@ def main():
thumb.add_argument("--line1", help="Explicit first thumbnail line (skips AI rewrite)")
thumb.add_argument("--line2", help="Explicit second thumbnail line")
thumb.add_argument("--json", action="store_true", help="Emit JSON {paths:[...]} to stdout")
+ _add_pair_args(thumb)
# ── thumbnail-config ──
tcfg = sub.add_parser("thumbnail-config", help="Show, export, import, or reset the thumbnail template")
@@ -5196,16 +5389,27 @@ def main():
topt.add_argument("--end", type=float, help="Frame window end (seconds)")
topt.add_argument("--texts", type=int, default=6, help="Number of headline options")
topt.add_argument("--frames", type=int, default=6, help="Number of frame options")
+ topt.add_argument("--payoff", help="Clip's payoff line, so headline copy is grounded in it rather than the title alone")
+ topt.add_argument("--context-line", dest="context_line", help="The question this clip answers, if any")
+ topt.add_argument("--preview-text", dest="preview_text", help="Clip's verbatim opening line")
# ── thumbnail-render (one final thumbnail from a chosen frame + headline) ──
trnd = sub.add_parser("thumbnail-render", help="Render one thumbnail PNG from a chosen frame + headline")
trnd.add_argument("title", help="Clip/episode title")
- trnd.add_argument("--frame", required=True, help="Background frame image path")
+ trnd.add_argument("--frame", help="Background frame image path. Optional with the pair layout, "
+ "where it is the fallback when two people cannot be told apart")
+ trnd.add_argument("--video", help="Source video the pair layout takes both people from")
+ trnd.add_argument("--start", type=float, help="Clip start in --video (seconds)")
+ trnd.add_argument("--end", type=float, help="Clip end in --video (seconds)")
+ _add_pair_args(trnd)
trnd.add_argument("-o", "--output", required=True, help="Destination PNG path")
trnd.add_argument("--line1", help="Headline line 1 (empty = AI writes it)")
trnd.add_argument("--line2", help="Headline line 2 (empty = AI writes it)")
trnd.add_argument("--frame-info", dest="frame_info", help="JSON face metadata for the frame")
trnd.add_argument("--logo", help="Logo (asset name or path)")
+ trnd.add_argument("--payoff", help="Clip's payoff line, so headline copy is grounded in it rather than the title alone")
+ trnd.add_argument("--context-line", dest="context_line", help="The question this clip answers, if any")
+ trnd.add_argument("--preview-text", dest="preview_text", help="Clip's verbatim opening line")
# ── swap-thumbnail ──
st = sub.add_parser("swap-thumbnail", help="Regenerate thumbnail on an existing clip")
@@ -5324,6 +5528,21 @@ def main():
# ── info ──
sub.add_parser("info", help="Show system info (encoder, etc.)")
+ # ── compare-engines ──
+ cmp_p = sub.add_parser(
+ "compare-engines",
+ help="Transcribe the same sample window with two engines and report where they disagree",
+ )
+ cmp_p.add_argument("video", help="Path to podcast video/audio file")
+ cmp_p.add_argument("engine_a", choices=["whisper-py", "whispercpp", "assemblyai", "omnilingual"])
+ cmp_p.add_argument("engine_b", choices=["whisper-py", "whispercpp", "assemblyai", "omnilingual"])
+ cmp_p.add_argument("--start", type=float, default=0.0, help="Sample start, seconds into the source (default: 0)")
+ cmp_p.add_argument("--duration", type=float, default=120.0, help="Sample length in seconds (default: 120)")
+ cmp_p.add_argument("--window", type=float, default=20.0, help="Report window size in seconds (default: 20)")
+ cmp_p.add_argument("--model-size", default="base", help="Model size for engines that take one (default: base)")
+ cmp_p.add_argument("--language", help="ISO language code. Auto-detect if omitted.")
+ cmp_p.add_argument("-o", "--output", default="./engine-comparison", help="Output directory for comparison.json/.html")
+
init_thumb = sub.add_parser(
"init-thumbnail",
help="Scaffold .podcli/thumbnail-config.json so podcli generates thumbnails for you",
@@ -5404,6 +5623,8 @@ def main():
cmd_cache(args)
elif args.command == "info":
cmd_info(args)
+ elif args.command == "compare-engines":
+ cmd_compare_engines(args)
elif args.command == "init-thumbnail":
cmd_init_thumbnail(args)
elif args.command in ("ui", "webui"):
diff --git a/backend/clip_studio.py b/backend/clip_studio.py
index 02948df..c4926db 100644
--- a/backend/clip_studio.py
+++ b/backend/clip_studio.py
@@ -196,6 +196,23 @@ def _json_file(raw, name):
return None
+def _hook_arg(raw, start, end):
+ """The --hook object checked against the fragment, or nothing.
+
+ Checked as soon as the fragment's range is known, so a bad hook stops the
+ run before the face scan and the render. It stops rather than drops,
+ because someone asked for it.
+ """
+ hook = _json_object_arg(raw, "--hook")
+ if hook is None:
+ return None
+ from services.opening_hook import validate_hook
+ try:
+ return validate_hook(hook, start, end)
+ except ValueError as exc:
+ raise SystemExit(f"Error: --hook {exc}")
+
+
# The crops that cannot place a frame without knowing where the faces are.
_WANTS_FACES = ("face", "speaker", "speaker-hardcut")
@@ -249,7 +266,7 @@ def _render_fragment(video, start, end, words, style, crop, title, out_dir, fmt=
logo=None, name_card=None, motion=None, caption_position="auto",
caption_scale=1.0, logo_position="top-left", logo_scale=1.0,
topic=None, progress=None, cards=None, brand=None, theme=None, font_family=None,
- captions=True, face_map=None, crop_keyframes=None):
+ captions=True, face_map=None, crop_keyframes=None, hook=None):
"""Render the fragment with face-crop + captions via the existing engine."""
from services.clip_generator import generate_clip
print(f" [fragment] rendering {start:.1f}s–{end:.1f}s ({style}, crop={crop}, {fmt})", flush=True)
@@ -263,7 +280,7 @@ def _render_fragment(video, start, end, words, style, crop, title, out_dir, fmt=
transcript_words=words, title=title, output_dir=out_dir,
logo_path=logo, name_card=name_card, motion=motion,
topic=topic, progress=progress, cards=cards, brand=brand, theme=theme, font_family=font_family,
- captions=captions,
+ captions=captions, hook=hook,
clean_fillers=True, allow_ass_fallback=True,
progress_callback=lambda p, m: print(f" {p}% {m}", flush=True),
)
@@ -388,6 +405,9 @@ def main():
help="Path to hand-placed crop positions as JSON. Used by --crop manual.")
ap.add_argument("--cards", default=None,
help="On-screen cards as JSON, each with kind/start/end")
+ ap.add_argument("--hook", default=None,
+ help='Opening hook as JSON on the source clock: {"start","end","mode"}. '
+ 'mode is "repeat" or "move"')
ap.add_argument("--brand", default=None,
help='Show colours as JSON: {"accent":"#4C9DF5","ink":"#FFF","surface":"#000"}')
ap.add_argument("--style", default=None,
@@ -448,6 +468,8 @@ def main():
else:
raise SystemExit("Provide either --start/--end or --paragraph")
+ hook = _hook_arg(args.hook, start, end)
+
# The canvas every part is rendered and stitched on. One lookup, so the
# fragment, the bookends and the concat cannot disagree about the shape.
from services.formats import get_format
@@ -511,6 +533,7 @@ def main():
captions=not args.no_captions,
face_map=face_map,
crop_keyframes=_json_file(args.crop_keyframes, "--crop-keyframes"),
+ hook=hook,
)
platforms = [p.strip() for p in platforms_str.split(",") if p.strip()]
diff --git a/backend/main.py b/backend/main.py
index a124f3b..3b76216 100644
--- a/backend/main.py
+++ b/backend/main.py
@@ -67,15 +67,29 @@ def handle_ping(task_id: str, params: dict):
emit_result(task_id, "success", data={"message": "pong", "version": VERSION})
+def handle_resolve_transcribe_engine(task_id: str, params: dict):
+ """Predict transcribe_file's engine resolution without transcribing, so
+ callers can build the right cache key before deciding whether to run it."""
+ from services.transcription import resolve_engine_info
+
+ emit_result(
+ task_id,
+ "success",
+ data=resolve_engine_info(params.get("engine"), params.get("model_size", "base")),
+ )
+
+
def handle_transcribe(task_id: str, params: dict):
"""Transcribe a podcast video/audio file with speaker detection."""
from services.transcription import transcribe_file
from services.corrections import apply_corrections
- from services.transcript_packer import compute_cache_hash, engine_cache_suffix, write_packed
+ from services.transcript_packer import compute_cache_hash, cache_key_suffix, write_packed
emit_progress(task_id, "transcribing", 0, "Starting transcription...")
file_path = params["file_path"]
engine = params.get("engine")
+ model_size = params.get("model_size", "base")
+ language = params.get("language")
previous_engine = os.environ.get("PODCLI_ENGINE")
previous_assemblyai_key = os.environ.get("ASSEMBLYAI_API_KEY")
if engine:
@@ -83,14 +97,26 @@ def handle_transcribe(task_id: str, params: dict):
if params.get("assemblyai_api_key"):
os.environ["ASSEMBLYAI_API_KEY"] = params["assemblyai_api_key"]
+ start_seconds = params.get("start_seconds")
+ duration_seconds = params.get("duration_seconds")
+ # Matches src/handlers/transcribe.handler.ts and web-server.ts exactly:
+ # a sample is a *positive* window, not merely a present key. A caller
+ # that explicitly passes null (dropped by JSON on the TS side, but
+ # still visible here as None) or duration_seconds: 0 must get the full
+ # transcription path, the same as not passing the field at all.
+ is_sample = (duration_seconds or 0) > 0 or (start_seconds or 0) > 0
+
# One shared 16 kHz mono wav feeds transcription, energy and reactions
- # instead of decoding the source three times.
+ # instead of decoding the source three times. Skipped for a sample run:
+ # transcribe_file extracts its own trimmed window, and decoding the full
+ # source here would undo the whole point of a quick sample.
shared_wav = None
- try:
- from services.audio_extract import extract_wav_16k_mono
- shared_wav = extract_wav_16k_mono(file_path)
- except Exception:
- shared_wav = None
+ if not is_sample:
+ try:
+ from services.audio_extract import extract_wav_16k_mono
+ shared_wav = extract_wav_16k_mono(file_path)
+ except Exception:
+ shared_wav = None
try:
result = transcribe_file(
@@ -100,46 +126,56 @@ def handle_transcribe(task_id: str, params: dict):
language=params.get("language"),
enable_diarization=params.get("enable_diarization", True),
num_speakers=params.get("num_speakers"),
+ start_seconds=start_seconds,
+ duration_seconds=duration_seconds,
progress_callback=lambda pct, msg: emit_progress(task_id, "transcribing", pct, msg),
wav_path=shared_wav,
)
# Apply word corrections (Whisper misheard proper nouns)
apply_corrections(result.get("words", []), result.get("segments", []))
- energy_data = None
- try:
- from services.audio_analyzer import extract_audio_energy
- energy_data = extract_audio_energy(file_path, wav_path=shared_wav)
- except Exception:
- pass # energy is a nice-to-have
+ # A sample is a throwaway language/quality check on a slice of the
+ # source. Energy/event signals and the packed view are keyed by the
+ # full file and meant to describe the whole episode, so skip them
+ # rather than caching partial (or source-wide-but-wrongly-expensive)
+ # data under those keys.
+ if not is_sample:
+ energy_data = None
+ try:
+ from services.audio_analyzer import extract_audio_energy
+ energy_data = extract_audio_energy(file_path, wav_path=shared_wav)
+ except Exception:
+ pass # energy is a nice-to-have
- events_data = None
- try:
- from services.audio_events import extract_audio_events
- events_data = extract_audio_events(file_path, wav_path=shared_wav)
- except Exception:
- pass # reactions are a nice-to-have
+ events_data = None
+ try:
+ from services.audio_events import extract_audio_events
+ events_data = extract_audio_events(file_path, wav_path=shared_wav)
+ except Exception:
+ pass # reactions are a nice-to-have
- # Cached so clip suggestion reuses these instead of decoding the source again.
- from services.signal_cache import save_signals
- save_signals(file_path, energy_data=energy_data, events_data=events_data)
+ # Cached so clip suggestion reuses these instead of decoding the source again.
+ from services.signal_cache import save_signals
+ save_signals(file_path, energy_data=energy_data, events_data=events_data)
- # Auto-pack: emit compact LLM-readable markdown alongside raw JSON.
- # Pulls energy data so the packed view includes peak moments for clip reasoning.
- try:
- cache_hash = compute_cache_hash(file_path) + engine_cache_suffix(result.get("engine") or engine)
- packed_path, packed_md = write_packed(
- result,
- cache_hash,
- source_label=os.path.basename(file_path),
- energy_data=energy_data,
- events_data=events_data,
- )
- result["packed_path"] = packed_path
- result["packed_size_bytes"] = len(packed_md.encode("utf-8"))
- except Exception as e:
- # Non-fatal — transcription result is still useful without the packed view
- emit_progress(task_id, "packing", 99, f"Packer skipped: {e}")
+ # Auto-pack: emit compact LLM-readable markdown alongside raw JSON.
+ # Pulls energy data so the packed view includes peak moments for clip reasoning.
+ try:
+ cache_hash = compute_cache_hash(file_path) + cache_key_suffix(
+ engine=result.get("engine") or engine, model=model_size, language=language
+ )
+ packed_path, packed_md = write_packed(
+ result,
+ cache_hash,
+ source_label=os.path.basename(file_path),
+ energy_data=energy_data,
+ events_data=events_data,
+ )
+ result["packed_path"] = packed_path
+ result["packed_size_bytes"] = len(packed_md.encode("utf-8"))
+ except Exception as e:
+ # Non-fatal: transcription result is still useful without the packed view
+ emit_progress(task_id, "packing", 99, f"Packer skipped: {e}")
finally:
if previous_engine is None:
os.environ.pop("PODCLI_ENGINE", None)
@@ -192,11 +228,13 @@ def handle_create_clip(task_id: str, params: dict):
clean_fillers=params.get("clean_fillers", True),
face_map=params.get("face_map"),
keep_segments=params.get("keep_segments"),
+ hook=params.get("hook"),
trim_opening=params.get("trim_opening"),
preserve_timing=params.get("preserve_timing", False),
allow_ass_fallback=params.get("allow_ass_fallback", False),
use_ass_captions=params.get("use_ass_captions", False),
keep_caption_overlay=params.get("keep_caption_overlay", False),
+ write_clean_variant=params.get("write_clean_variant", False),
progress_callback=lambda pct, msg: emit_progress(task_id, "processing", pct, msg),
)
emit_result(task_id, "success", data=result)
@@ -258,11 +296,15 @@ def handle_batch_clips(task_id: str, params: dict):
clean_fillers=params.get("clean_fillers", True),
face_map=params.get("face_map"),
keep_segments=clip.get("keep_segments"),
+ hook=clip.get("hook"),
allow_ass_fallback=clip.get("allow_ass_fallback", params.get("allow_ass_fallback", False)),
use_ass_captions=clip.get("use_ass_captions", params.get("use_ass_captions", False)),
keep_caption_overlay=clip.get(
"keep_caption_overlay", params.get("keep_caption_overlay", False)
),
+ write_clean_variant=clip.get(
+ "write_clean_variant", params.get("write_clean_variant", False)
+ ),
progress_callback=lambda pct, msg, _i=i: emit_progress(
task_id, "batch", int((_i / total) * 100 + pct / total), msg
),
@@ -331,13 +373,16 @@ def handle_parse_transcript(task_id: str, params: dict):
raw_text = params.get("raw_text", "")
total_duration = params.get("total_duration")
time_adjust = params.get("time_adjust", 0.0)
+ language = params.get("language")
if not raw_text:
emit_result(task_id, "error", error="raw_text is required")
return
emit_progress(task_id, "parsing", 50, "Parsing transcript...")
- result = detect_and_parse(raw_text, total_duration=total_duration, time_adjust=time_adjust)
+ result = detect_and_parse(
+ raw_text, total_duration=total_duration, time_adjust=time_adjust, language=language
+ )
if "error" in result:
emit_result(task_id, "error", error=result["error"])
@@ -943,6 +988,40 @@ def handle_analyze_silence(task_id: str, params: dict):
emit_result(task_id, "error", error=str(e))
+def handle_compare_engines(task_id: str, params: dict):
+ """Transcribe the same sample window with two engines and report where
+ they disagree. See services/engine_comparison.py for the scoring."""
+ import time as _time
+ from config.paths import paths
+ from services.engine_comparison import compare_engines
+
+ file_path = params.get("file_path", "")
+ if not file_path or not os.path.exists(file_path):
+ emit_result(task_id, "error", error=f"File not found: {file_path}")
+ return
+
+ output_dir = params.get("output_dir") or os.path.join(
+ paths["output"], "engine-comparisons", f"{int(_time.time())}"
+ )
+ try:
+ emit_progress(task_id, "comparing", 10, f"Transcribing with {params.get('engine_a')}...")
+ report = compare_engines(
+ file_path,
+ params.get("engine_a", "whispercpp"),
+ params.get("engine_b", "whisper-py"),
+ start_seconds=params.get("start_seconds", 0.0) or 0.0,
+ duration_seconds=params.get("duration_seconds", 120.0) or 120.0,
+ window_seconds=params.get("window_seconds", 20.0) or 20.0,
+ model_size=params.get("model_size", "base"),
+ language=params.get("language"),
+ output_dir=output_dir,
+ )
+ emit_progress(task_id, "comparing", 100, "Comparison complete")
+ emit_result(task_id, "success", data=report)
+ except (FileNotFoundError, RuntimeError, ValueError) as e:
+ emit_result(task_id, "error", error=str(e))
+
+
def handle_render_silence_removed(task_id: str, params: dict):
"""Render the approved local cut plan and remap transcript timestamps."""
from config.paths import paths
@@ -1006,7 +1085,7 @@ def handle_manage_multicam(task_id: str, params: dict):
emit_result(task_id, "success", data={"deleted": True, "session_id": session_id})
return
- session = mc.MulticamSession.load(session_id)
+ session = mc.open_session(session_id)
if action == "render" and any(k in params for k in MULTICAM_MAP_KEYS if k != "look"):
# A mapping change can drop the cut, and a render needs one: map, then plan, then render.
raise ValueError("render takes only look and stems. Change the mapping with 'map', then 'plan' again.")
@@ -1036,7 +1115,10 @@ def handle_manage_multicam(task_id: str, params: dict):
stems = params.get("stems", True)
if not isinstance(stems, bool):
raise ValueError("stems is true or false")
- mc.render_session(session, stems=stems, progress_callback=progress("rendering"))
+ validate = params.get("validate", "sample")
+ if validate not in ("sample", "full"):
+ raise ValueError("validate must be 'sample' or 'full'")
+ mc.render_session(session, stems=stems, validate=validate, progress_callback=progress("rendering"))
elif action == "export":
data["export_path"] = mc.export_xml(session, params.get("format", "premiere"), review=bool(params.get("review")))
elif action == "import_timeline":
@@ -1098,7 +1180,9 @@ def handle_run_integration_tool(task_id: str, params: dict):
TASK_HANDLERS = {
"ping": handle_ping,
+ "resolve_transcribe_engine": handle_resolve_transcribe_engine,
"transcribe": handle_transcribe,
+ "compare_engines": handle_compare_engines,
"parse_transcript": handle_parse_transcript,
"create_clip": handle_create_clip,
"batch_clips": handle_batch_clips,
diff --git a/backend/requirements-runtime.txt b/backend/requirements-runtime.txt
index 8cd3252..de10161 100644
--- a/backend/requirements-runtime.txt
+++ b/backend/requirements-runtime.txt
@@ -7,6 +7,9 @@ numpy>=2.5.2
# Audio-event detection (YAMNet laughter/reaction channel) runs on ONNX Runtime —
# no torch/TF, so it stays on the hermetic native path.
onnxruntime>=1.28.0
+# --engine omnilingual: Meta's Omnilingual ASR CTC model via sherpa-onnx, for
+# languages whisper.cpp handles poorly. No torch, stays on the hermetic path.
+sherpa-onnx>=1.13.8
Pillow>=12.3.0
questionary>=2.1.1
python-dotenv>=1.2.2
diff --git a/backend/requirements.txt b/backend/requirements.txt
index 546df29..96edede 100644
--- a/backend/requirements.txt
+++ b/backend/requirements.txt
@@ -15,6 +15,10 @@ numpy>=2.5.2
# Audio-event detection (laughter/reaction channel via YAMNet ONNX — no torch/TF)
onnxruntime>=1.28.0
+# --engine omnilingual: Meta's Omnilingual ASR CTC model via sherpa-onnx, for
+# languages whisper.cpp handles poorly (e.g. Georgian). No torch/TF.
+sherpa-onnx>=1.13.8
+
# Thumbnails
Pillow>=12.3.0
diff --git a/backend/services/audio_extract.py b/backend/services/audio_extract.py
index f9a6883..d96a2ce 100644
--- a/backend/services/audio_extract.py
+++ b/backend/services/audio_extract.py
@@ -17,18 +17,29 @@ def extract_wav_16k_mono(
media_path: str,
wav_path: Optional[str] = None,
timeout: int = 1800,
+ start_seconds: Optional[float] = None,
+ duration_seconds: Optional[float] = None,
) -> str:
"""Extract audio as 16 kHz mono 16-bit PCM WAV. Returns the wav path.
When wav_path is None a temp file is created; the caller owns cleanup.
+ start_seconds/duration_seconds trim the output to a window of the source,
+ used for sample-mode transcription (test a language on a short clip
+ instead of the full episode).
"""
owns_wav = wav_path is None
if owns_wav:
fd, wav_path = tempfile.mkstemp(prefix="podcli_audio_", suffix=".wav")
os.close(fd)
- cmd = [
- "ffmpeg", "-y", "-loglevel", "error",
- "-i", media_path,
+ cmd = ["ffmpeg", "-y", "-loglevel", "error"]
+ if start_seconds:
+ # Before -i: fast (keyframe-seek) trim. Precision to the frame
+ # doesn't matter for a language-check sample.
+ cmd += ["-ss", str(start_seconds)]
+ cmd += ["-i", media_path]
+ if duration_seconds:
+ cmd += ["-t", str(duration_seconds)]
+ cmd += [
"-vn",
"-acodec", "pcm_s16le",
"-ar", "16000",
diff --git a/backend/services/caption_renderer.py b/backend/services/caption_renderer.py
index 8f44d3a..f8bb5d4 100644
--- a/backend/services/caption_renderer.py
+++ b/backend/services/caption_renderer.py
@@ -12,6 +12,27 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from config.caption_styles import get_style
from utils.timing_utils import seconds_to_ass
+from utils.text import safe_upper
+
+
+def _sanitize_ass_text(text: str) -> str:
+ """Neutralize characters that mean something to the ASS/libass parser
+ when they show up in transcribed word text instead of our own markup.
+
+ A literal backslash isn't escapable in the plain-text part of a
+ Dialogue line: libass still reads \\N, \\n and \\h out of it (that's how
+ an uppercase transform can turn a stray "\\n" into a forced \\N
+ linebreak), and a literal "{" opens a new override block early, letting
+ anything after it (including a following "}") be read as ASS tags
+ instead of rendered as text. There's no escape sequence for any of
+ these in plain text, so swap in a full-width look-alike glyph that
+ renders the same character shape without being special to the parser.
+ """
+ return (
+ text.replace("\\", "\")
+ .replace("{", "{")
+ .replace("}", "}")
+ )
def generate_ass_header(style: dict, play_res_x: int = 1080, play_res_y: int = 1920) -> str:
@@ -207,7 +228,7 @@ def _render_hormozi(words: list[dict], style: dict, offset: float) -> str:
parts = []
for w in chunk:
duration_cs = int((w["end"] - w["start"]) * 100)
- text = w["word"].upper() if uppercase else w["word"]
+ text = _sanitize_ass_text(safe_upper(w["word"]) if uppercase else w["word"])
parts.append(f"{{\\kf{duration_cs}}}{text}")
# \c = active (filled) color, \2c = inactive (unfilled) color
@@ -246,7 +267,7 @@ def _render_karaoke(words: list[dict], style: dict, offset: float) -> str:
parts = []
for w in sentence:
duration_cs = int((w["end"] - w["start"]) * 100)
- text = w["word"]
+ text = _sanitize_ass_text(w["word"])
parts.append(f"{{\\kf{duration_cs}}}{text}")
line_text = " ".join(parts)
@@ -279,7 +300,7 @@ def _render_subtle(words: list[dict], style: dict, offset: float) -> str:
line_start = max(0, line_words[0]["start"] - offset)
line_end = _hold_through_gap(lines, idx, max(0, line_words[-1]["end"] - offset), offset)
- line_text = " ".join(w["word"] for w in line_words)
+ line_text = " ".join(_sanitize_ass_text(w["word"]) for w in line_words)
start_ts = seconds_to_ass(line_start)
end_ts = seconds_to_ass(line_end)
@@ -544,9 +565,9 @@ def _render_branded(words: list[dict], style: dict, offset: float) -> str:
# Normalize casing
normalized = []
for j, w in enumerate(chunk):
- text = _normalize_case(w["word"])
+ text = _sanitize_ass_text(_normalize_case(w["word"]))
if j == 0:
- text = text[0].upper() + text[1:] if len(text) > 1 else text.upper()
+ text = safe_upper(text[0]) + text[1:] if len(text) > 1 else safe_upper(text)
normalized.append(text)
# Measure word widths for pill positioning.
diff --git a/backend/services/claude_suggest.py b/backend/services/claude_suggest.py
index 6585658..280907b 100644
--- a/backend/services/claude_suggest.py
+++ b/backend/services/claude_suggest.py
@@ -23,6 +23,7 @@ from services.audio_events import compute_event_scores, dominant_reaction
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from presets import DEFAULT_PRESET, MIN_CLIP_DURATION, MAX_CLIP_DURATION, TARGET_CLIP_DURATION_MIN, TARGET_CLIP_DURATION_MAX
from services.formats import get_format
+from services.opening_hook import HOOK_MODES, validate_hook
from utils.text import clean_title
from services import ai_provider, podcli_cloud
from services.ai_cli import (
@@ -126,7 +127,7 @@ KB_FILES = [
# Bump when the selection rules in _build_prompt change. Folded into the
# suggestion cache key for the same reason the knowledge base is: replaying old
# picks under new rules teaches people the rules do nothing.
-PROMPT_VERSION = 5
+PROMPT_VERSION = 6
def kb_signature() -> str:
@@ -245,6 +246,26 @@ def _total_score(clip: dict) -> float:
MAX_REACTION_ANCHORS = 40
+def _suggested_hook(clip: dict, start: float, end: float, segments: list[dict]) -> Optional[dict]:
+ """The opening hook the model proposed for this clip, or None.
+
+ A hook that is malformed or falls outside the clip it opens is dropped
+ rather than failing the clip: the moment is still worth rendering without it.
+ """
+ raw = clip.get("hook")
+ if not isinstance(raw, dict) or raw.get("mode") not in HOOK_MODES:
+ return None
+ try:
+ hook = {
+ "start": round(float(raw.get("start")), 1),
+ "end": round(float(raw.get("end")), 1),
+ "mode": raw["mode"],
+ }
+ return validate_hook(hook, start, end, segments)
+ except (TypeError, ValueError):
+ return None
+
+
def _format_reaction_anchors(reaction_times: list[float] | None) -> str:
"""Prompt block listing detected laughter/reaction peaks as anchor moments."""
if not reaction_times:
@@ -401,6 +422,12 @@ SEGMENTS RULES:
- "start_second" / "end_second" = outer bounds (first segment start, last segment end)
- Example: speaker makes great point (10s), rambles (8s), delivers punchline (12s) → 2 segments, 22s total
+HOOK (optional):
+- If the sharpest line sits mid-clip, add "hook": {{"start": X, "end": Y, "mode": "repeat"}} so it plays first.
+- It must be 1-15 seconds of words actually spoken inside the clip's segments. Never invent text.
+- "repeat" plays it again where it belongs. "move" lifts it out of the body.
+- Leave "hook" out when the clip already opens on its strongest line.
+
Rules:
- Final clip duration (sum of segments) MUST be {b.dur_min}-{b.dur_max} seconds (target {b.target_min}-{b.target_max}s)
- Each segment must start and end on COMPLETE SENTENCES — never mid-thought
@@ -576,6 +603,8 @@ RULES:
- In "needs", name what the viewer must already know, or "nothing". If it is anything
else, move start_second back until the clip covers it. "context_line" is a note for
the editor, not a fix: nothing burns it into the video.
+- Optional "hook": {{"start": X, "end": Y, "mode": "repeat" or "move"}}, 1-15 seconds of
+ words spoken inside the clip, played first. Leave it out when the clip opens strong.
Return this JSON:
{{
@@ -633,11 +662,14 @@ Transcript:
if not bounds.keeps(kept_duration):
continue
+ clip_start = keep_segments[0]["start"] if keep_segments else start_sec
+ clip_end = keep_segments[-1]["end"] if keep_segments else end_sec
found.append({
"title": clean_title(c.get("title", "Untitled")),
- "start_second": keep_segments[0]["start"] if keep_segments else start_sec,
- "end_second": keep_segments[-1]["end"] if keep_segments else end_sec,
+ "start_second": clip_start,
+ "end_second": clip_end,
"segments": keep_segments,
+ "hook": _suggested_hook(c, clip_start, clip_end, keep_segments),
"duration": round(kept_duration),
"score": total,
"content_type": c.get("content_type", "unknown"),
@@ -839,11 +871,14 @@ def suggest_with_claude(
if not bounds.keeps(kept_duration):
continue
+ clip_start = keep_segments[0]["start"] if keep_segments else start_sec
+ clip_end = keep_segments[-1]["end"] if keep_segments else end_sec
normalized.append({
"title": clean_title(c.get("title", "Untitled")),
- "start_second": keep_segments[0]["start"] if keep_segments else start_sec,
- "end_second": keep_segments[-1]["end"] if keep_segments else end_sec,
+ "start_second": clip_start,
+ "end_second": clip_end,
"segments": keep_segments,
+ "hook": _suggested_hook(c, clip_start, clip_end, keep_segments),
"duration": round(kept_duration),
"score": total,
"content_type": c.get("content_type", "unknown"),
diff --git a/backend/services/clip_generator.py b/backend/services/clip_generator.py
index 314f390..a1cfc83 100644
--- a/backend/services/clip_generator.py
+++ b/backend/services/clip_generator.py
@@ -32,6 +32,15 @@ from services.video_processor import (
normalize_audio,
concat_outro,
)
+from services.video_cut import probe_has_audio_stream, verify_full_decode
+from services.glyph_coverage import check_caption_font_coverage
+from services.subtitle_export import write_sidecars
+from services.opening_hook import (
+ validate_hook,
+ snap_hook_to_words,
+ order_with_hook,
+ keyframes_to_playback,
+)
from config.caption_styles import get_style
from services.formats import get_format
@@ -131,6 +140,35 @@ def _get_media_duration(path: str) -> float:
return 0.0
+def _bound_range_to_source(
+ start_second: float,
+ end_second: float,
+ keep_segments: Optional[list[dict]],
+ source_duration: float,
+) -> tuple[float, Optional[list[dict]]]:
+ """Clamp the requested range (and any keep_segments) to the source's
+ actual duration, so a too-long end_second can't silently truncate the
+ render instead of failing. source_duration <= 0 means ffprobe couldn't
+ read it; skip bounding rather than clamp against an unknown length.
+ """
+ if source_duration <= 0:
+ return end_second, keep_segments
+ if start_second >= source_duration:
+ raise ValueError(
+ f"start_second ({start_second}s) is at or past the end of the "
+ f"source video ({source_duration:.2f}s)."
+ )
+ end_second = min(end_second, source_duration)
+ if keep_segments:
+ bounded_segments = []
+ for seg in keep_segments:
+ if seg["start"] >= source_duration:
+ continue
+ bounded_segments.append({**seg, "end": min(seg["end"], source_duration)})
+ keep_segments = bounded_segments or None
+ return end_second, keep_segments
+
+
def _detect_scene_cuts(path: str, threshold: float = 0.22, max_cuts: int = 32) -> list[float]:
"""
Detect hard visual changes likely to feel like jump cuts.
@@ -859,11 +897,13 @@ def generate_clip(
bookend_fade: float = 0.0,
clean_fillers: bool = True,
keep_segments: list[dict] = None,
+ hook: Optional[dict] = None,
trim_opening: Optional[bool] = None,
preserve_timing: bool = False,
allow_ass_fallback: bool = False,
use_ass_captions: bool = False,
keep_caption_overlay: bool = False,
+ write_clean_variant: bool = False,
topic: Optional[dict] = None,
progress: Optional[dict] = None,
cards: Optional[list] = None,
@@ -871,6 +911,7 @@ def generate_clip(
font_family: Optional[str] = None,
theme: Optional[dict] = None,
captions: bool = True,
+ write_subtitles: Optional[bool] = None,
progress_callback: Optional[Callable[[int, str], None]] = None,
) -> dict:
"""
@@ -887,6 +928,13 @@ def generate_clip(
title: Clip title (used in filename)
output_dir: Where to save the final clip (defaults to temp)
logo_path: Path to logo image (PNG). Used with "branded" style.
+ hook: Optional {"start", "end", "mode"} on the source clock: a passage
+ from inside the body played first. "repeat" plays it again in
+ place, "move" removes it from the body.
+ captions: Whether to burn captions into the video.
+ write_subtitles: Whether to write .srt/.vtt sidecars. Defaults to
+ following `captions` (no burned captions, no sidecars either);
+ pass True to request them for a clean (caption-free) export.
progress_callback: Optional (percent, message) callback
Returns:
@@ -912,6 +960,25 @@ def generate_clip(
if end_second <= start_second:
raise ValueError("end_second must be greater than start_second")
+ # A video-only source (e.g. a silent screen recording) never gets an
+ # audio stream from any of the steps below, so requiring one would
+ # reject every render of it as "broken" instead of a legitimately
+ # silent clip.
+ source_has_audio = probe_has_audio_stream(video_path)
+
+ # ffmpeg's -ss/-t cut silently stops at EOF when end_second runs past the
+ # source, so without this the render "succeeds" with a shorter clip than
+ # requested while duration/end_second in the result still say the planned
+ # (too-long) window. Bound every range to what the source actually has.
+ source_duration = _get_media_duration(video_path)
+ end_second, keep_segments = _bound_range_to_source(
+ start_second, end_second, keep_segments, source_duration
+ )
+
+ # Checked against the requested body, before any tightening moves it.
+ hook = validate_hook(hook, start_second, end_second, keep_segments)
+ requested_start = start_second
+
spec = get_format(format)
# An episode that was never filmed takes the other road entirely. Branching
@@ -920,6 +987,11 @@ def generate_clip(
# back is the same dict describing the same window.
from services.audiogram import is_audio_only
if is_audio_only(video_path):
+ if hook:
+ raise ValueError(
+ "An opening hook needs a video source. Audiogram clips play one "
+ "continuous range. Drop the hook to render this audio-only clip."
+ )
from services.audiogram import render_audiogram
return render_audiogram(
audio_path=video_path,
@@ -1025,11 +1097,22 @@ def generate_clip(
keep_segments = None
duration = end_second - start_second
+ # The body is final here. The hook is cut as given (widened only to whole
+ # words) and put ahead of it, so every later step reads play_ranges, in
+ # playback order, never re-sorted. None means one continuous range.
+ play_ranges = keep_segments if keep_segments and len(keep_segments) > 1 else None
+ if hook:
+ hook = snap_hook_to_words(hook, transcript_words)
+ body = keep_segments or [{"start": start_second, "end": end_second}]
+ play_ranges = order_with_hook(body, hook)
+ duration = sum(r["end"] - r["start"] for r in play_ranges)
+
length_warning = None
# A clip that asks for captions and renders without them used to be
# indistinguishable from a clip that never wanted them. Two shipped clips
# went out silent that way.
caption_warning = None
+ glyph_warning = None
if duration > spec.dur_max:
length_warning = f"{duration:.0f}s, over the {spec.dur_max}s {spec.name} target"
print(f" {length_warning}", file=sys.stderr, flush=True)
@@ -1048,40 +1131,54 @@ def generate_clip(
# Step 1: Cut the segment(s) from the source video
if progress_callback:
- n_segs = len(keep_segments) if keep_segments else 1
+ n_segs = len(play_ranges) if play_ranges else 1
msg = f"Cutting {n_segs} segment{'s' if n_segs > 1 else ''} (1/{total_steps})"
progress_callback(10, msg)
segment_path = os.path.join(work_dir, "segment.mp4")
- with timed("render", "cut", segments=len(keep_segments) if keep_segments else 1):
- if keep_segments and len(keep_segments) > 1:
- cut_multi_segment(video_path, segment_path, keep_segments)
+ part_durations: Optional[list[float]] = None
+ with timed("render", "cut", segments=len(play_ranges) if play_ranges else 1):
+ if play_ranges:
+ _, part_durations = cut_multi_segment(video_path, segment_path, play_ranges)
else:
cut_segment(video_path, segment_path, start_second, end_second)
- # Remap transcript words for multi-segment clips.
+ # Remap transcript words for multi-segment clips, range by range in
+ # playback order, so a repeated hook's words appear twice.
# Needed before crop (speaker detection) and captions.
- if keep_segments and len(keep_segments) > 1 and transcript_words:
- remapped_words = []
+ remapped_words: list[dict] = []
+ if play_ranges and transcript_words:
cumulative_t = 0.0
- for seg in keep_segments:
+ for i, seg in enumerate(play_ranges):
seg_words = [
w for w in transcript_words
if w["end"] > seg["start"] and w["start"] < seg["end"]
]
- seg_duration = seg["end"] - seg["start"]
+ # Each part is encoded separately before the concat, and an
+ # encoder snaps a cut to whole frames, so its real duration is
+ # typically a few ms off the requested end - start. Advancing
+ # cumulative_t by the planned length instead of the probed
+ # one drifts captions further out of sync with every segment
+ # concatenated in. Fall back to the planned length only if
+ # the part couldn't be probed.
+ requested_duration = seg["end"] - seg["start"]
+ actual_duration = (
+ part_durations[i]
+ if part_durations and part_durations[i] > 0
+ else requested_duration
+ )
for w in seg_words:
# Clamp to segment bounds to avoid negative/overflow timestamps
# for words that straddle a segment boundary
- remapped_start = max(0, cumulative_t + (w["start"] - seg["start"]))
- remapped_end = min(cumulative_t + seg_duration, cumulative_t + (w["end"] - seg["start"]))
+ remapped_start = max(cumulative_t, cumulative_t + (w["start"] - seg["start"]))
+ remapped_end = min(cumulative_t + actual_duration, cumulative_t + (w["end"] - seg["start"]))
if remapped_end > remapped_start:
remapped_words.append({
**w,
"start": round(remapped_start, 3),
"end": round(remapped_end, 3),
})
- cumulative_t += seg_duration
+ cumulative_t += actual_duration
crop_words = remapped_words
crop_clip_start = 0
caption_time_offset = 0
@@ -1098,6 +1195,13 @@ def generate_clip(
if progress_callback:
progress_callback(30, f"Resizing for {spec.name} format (2/{total_steps})")
+ playback_keyframes = keyframes_to_playback(
+ crop_keyframes,
+ requested_start,
+ play_ranges or [{"start": start_second, "end": end_second}],
+ part_durations,
+ ) if crop_keyframes else crop_keyframes
+
cropped_path = os.path.join(work_dir, "cropped.mp4")
with timed("render", "crop", strategy=crop_strategy if spec.reframe else "fit"):
if spec.reframe:
@@ -1107,7 +1211,7 @@ def generate_clip(
transcript_words=crop_words,
clip_start=crop_clip_start,
face_map=face_map,
- crop_keyframes=crop_keyframes,
+ crop_keyframes=playback_keyframes,
target_dims=spec.dims,
)
else:
@@ -1140,6 +1244,9 @@ def generate_clip(
logo_path or topic or progress or cards or theme
or (name_card and name_card.get("title"))
)
+ # Default so the sidecar/clean-variant step below has something to
+ # check even when neither branch below runs (no transcript, no overlay).
+ clip_words = []
if not captions and wants_overlay:
print(" drawing the overlay without captions", flush=True)
@@ -1154,7 +1261,7 @@ def generate_clip(
if progress_callback:
progress_callback(50, f"Adding {caption_style} captions (3/{total_steps})")
- if keep_segments and len(keep_segments) > 1:
+ if play_ranges:
clip_words = remapped_words
else:
clip_words = [
@@ -1180,6 +1287,15 @@ def generate_clip(
)
print(f" {caption_warning}", file=sys.stderr, flush=True)
+ if clip_words and captions:
+ caption_text = " ".join(w.get("word", "") for w in clip_words)
+ glyph_warning = check_caption_font_coverage(
+ caption_text, style_config["font_name"], bool(style_config["bold"]),
+ use_ass=use_ass_captions,
+ )
+ if glyph_warning:
+ print(f" {glyph_warning}", file=sys.stderr, flush=True)
+
if (clip_words and captions) or wants_overlay:
if progress_callback:
progress_callback(65, f"Rendering captions into video (3/{total_steps})")
@@ -1310,14 +1426,17 @@ def generate_clip(
final_video_path = with_intro_path
intro_offset = max(0.0, _get_media_duration(with_intro_path) - clip_duration)
+ outro_offset = 0.0
if outro_path and os.path.exists(outro_path):
if progress_callback:
progress_callback(85, f"Adding outro ({total_steps}/{total_steps})")
+ pre_outro_duration = _get_media_duration(final_video_path)
with_outro_path = os.path.join(work_dir, "with_outro.mp4")
concat_outro(final_video_path, outro_path, with_outro_path,
crossfade_duration=bookend_fade)
final_video_path = with_outro_path
+ outro_offset = max(0.0, _get_media_duration(with_outro_path) - pre_outro_duration)
# Step 6: Move to output
if progress_callback:
@@ -1341,17 +1460,107 @@ def generate_clip(
reframe=spec.reframe,
crop_strategy=crop_strategy,
crop_keyframes=crop_keyframes,
- keep_segments=keep_segments,
+ keep_segments=play_ranges,
)
if max_autofix_passes > 0:
if progress_callback:
progress_callback(97, "Quality gate: checking transitions...")
+ # The cut from the hook into the body is deliberate. Smoothing
+ # it would blur the first beat the hook exists to land.
+ hook_cut = [intro_offset + (part_durations[0] if part_durations else 0.0)] if hook else []
_auto_fix_transition_jumps(
final_path,
max_passes=max_autofix_passes,
- designed=_designed_cuts(cards, offset=intro_offset),
+ designed=_designed_cuts(cards, offset=intro_offset) + hook_cut,
)
+ # A 0-exit ffmpeg run can still have written a short, silent, or
+ # corrupt file (source ran out under the requested range, a crop/
+ # caption/normalize pass dropped the audio track, a concat mismatch).
+ # Catch that here instead of returning a "successful" result that
+ # describes a different clip than what landed on disk.
+ expected_duration = duration + intro_offset + outro_offset
+ actual_duration = _get_media_duration(final_path)
+ duration_tolerance = max(0.75, 0.05 * expected_duration)
+ if actual_duration <= 0 or abs(actual_duration - expected_duration) > duration_tolerance:
+ os.remove(final_path)
+ raise RuntimeError(
+ f"Render produced a {actual_duration:.2f}s file but expected "
+ f"~{expected_duration:.2f}s (body {duration:.2f}s"
+ f"{f' + intro {intro_offset:.2f}s' if intro_offset else ''}"
+ f"{f' + outro {outro_offset:.2f}s' if outro_offset else ''}). "
+ f"The source video likely ended before the requested range, or "
+ f"the render failed partway through."
+ )
+ if source_has_audio and not probe_has_audio_stream(final_path):
+ os.remove(final_path)
+ raise RuntimeError(
+ "Render completed but the output has no audio stream. "
+ "Re-check the source video and caption/crop pipeline."
+ )
+ decode_error = verify_full_decode(final_path)
+ if decode_error:
+ os.remove(final_path)
+ raise RuntimeError(f"Render produced an undecodable output: {decode_error}")
+
+ # Sidecar subtitles: the same words already burned into the video,
+ # on the same playback clock as the delivered file (clip-local time,
+ # shifted past whatever intro got prepended). clip_words is populated
+ # for overlay-only renders too (logo/cards with captions off), so
+ # gate on whether captions were actually requested rather than just
+ # "were there words": otherwise a captions-off render still gets an
+ # .srt/.vtt describing dialogue nothing on screen shows.
+ want_subtitles = write_subtitles if write_subtitles is not None else captions
+ output_base, _ = os.path.splitext(final_path)
+ sidecar_paths: dict = {}
+ # A re-render claims the same output path as last time
+ # (_reserve_output_path), so a sidecar this pass doesn't produce can
+ # be a leftover from a previous render of this clip, now describing
+ # dialogue or a style that no longer matches. Clear it before
+ # (maybe) writing a fresh one.
+ for stale_ext in (".srt", ".vtt"):
+ stale_sidecar = f"{output_base}{stale_ext}"
+ if os.path.exists(stale_sidecar):
+ os.remove(stale_sidecar)
+ if want_subtitles:
+ retimed_words = [
+ {
+ "word": w.get("word", ""),
+ "start": round(max(0.0, w["start"] - caption_time_offset + intro_offset), 3),
+ "end": round(max(0.0, w["end"] - caption_time_offset + intro_offset), 3),
+ }
+ for w in clip_words
+ ]
+ sidecar_paths = write_sidecars(retimed_words, output_base)
+
+ # Optional clean variant: the same audio, loudness, and intro/outro,
+ # minus burned captions, built from the cropped (pre-caption)
+ # source with the identical normalize_audio/concat_outro calls the
+ # main render used, so the two files only differ in the overlay.
+ clean_output_path = None
+ stale_clean_path = f"{output_base}_clean.mp4"
+ if os.path.exists(stale_clean_path):
+ os.remove(stale_clean_path)
+ if write_clean_variant and os.path.exists(cropped_path):
+ clean_normalized_path = os.path.join(work_dir, "clean_normalized.mp4")
+ normalize_audio(cropped_path, clean_normalized_path)
+ clean_video_path = clean_normalized_path
+
+ if intro_path and os.path.exists(intro_path):
+ clean_with_intro_path = os.path.join(work_dir, "clean_with_intro.mp4")
+ concat_outro(intro_scaled, clean_video_path, clean_with_intro_path,
+ crossfade_duration=bookend_fade)
+ clean_video_path = clean_with_intro_path
+
+ if outro_path and os.path.exists(outro_path):
+ clean_with_outro_path = os.path.join(work_dir, "clean_with_outro.mp4")
+ concat_outro(clean_video_path, outro_path, clean_with_outro_path,
+ crossfade_duration=bookend_fade)
+ clean_video_path = clean_with_outro_path
+
+ clean_output_path = f"{output_base}_clean.mp4"
+ shutil.copy2(clean_video_path, clean_output_path)
+
# Get file size
file_size = os.path.getsize(final_path)
file_size_mb = round(file_size / (1024 * 1024), 2)
@@ -1359,9 +1568,21 @@ def generate_clip(
if progress_callback:
progress_callback(100, "Clip complete!")
+ # "duration" has always meant the content's own length: what
+ # clip_history and the learnings it feeds record, not however long
+ # the delivered file plays including intro/outro. Derive it from the
+ # probed actual_duration (not the planned `duration` window) so a
+ # source that ran out early still gets caught by the check above and
+ # reported as shorter, instead of silently reporting the plan.
+ content_duration = max(0.0, actual_duration - intro_offset - outro_offset)
+
out = {
"output_path": final_path,
- "duration": round(duration, 2),
+ "duration": round(content_duration, 2),
+ # The probed duration of the file actually on disk, including any
+ # intro/outro, for callers that need the delivered file's full
+ # playback length rather than just its content.
+ "output_duration": round(actual_duration, 2),
"file_size_mb": file_size_mb,
"title": title,
"start_second": start_second,
@@ -1369,8 +1590,13 @@ def generate_clip(
"caption_style": caption_style,
"crop_strategy": crop_strategy,
"format": spec.name,
+ **sidecar_paths,
}
- warnings = [w for w in (caption_warning, length_warning) if w]
+ if clean_output_path:
+ out["clean_output_path"] = clean_output_path
+ if hook:
+ out["hook"] = hook
+ warnings = [w for w in (caption_warning, length_warning, glyph_warning) if w]
if warnings:
out["warning"] = "; ".join(warnings)
if keep_caption_overlay and caption_overlay_path and os.path.exists(caption_overlay_path):
diff --git a/backend/services/clips_history.py b/backend/services/clips_history.py
index 4a5f8d9..7878a1d 100644
--- a/backend/services/clips_history.py
+++ b/backend/services/clips_history.py
@@ -8,8 +8,8 @@ file; this module is the Python writer used by the CLI.
Entry shape (all fields beyond the core render record are optional):
id, source_video, start_second, end_second, caption_style, crop_strategy,
logo_path?, title, output_path, file_size_mb, duration, created_at,
- content_type?, transcript_slice?, youtube_video_id?, metrics?,
- generated_titles?, description?, tags?, hashtags?
+ content_type?, transcript_slice?, payoff?, context_line?, preview_text?,
+ youtube_video_id?, metrics?, generated_titles?, description?, tags?, hashtags?
metrics? = {views?, retention?, ctr?, impressions?, fetched_at?} (Phase 2)
diff --git a/backend/services/corrections.py b/backend/services/corrections.py
index 959c2fc..f1e5a31 100644
--- a/backend/services/corrections.py
+++ b/backend/services/corrections.py
@@ -73,6 +73,91 @@ def _replace_match(match: re.Match, corrections: dict[str, str]) -> str:
return replacement if replacement is not None else matched
+def _strip_for_match(text: str) -> str:
+ return text.strip(".,!?;:\"'()-")
+
+
+_TRAILING_PUNCT_RE = re.compile(r"[.,!?;:\"'()\-]+$")
+
+
+def _trailing_punct(text: str) -> str:
+ """Trailing punctuation only (not leading), e.g. 'AI.' -> '.'."""
+ match = _TRAILING_PUNCT_RE.search(text)
+ return match.group(0) if match else ""
+
+
+def _merge_multiword_corrections(words: list[dict], corrections: dict[str, str]) -> list[dict]:
+ """
+ Merge consecutive words matching a multi-word correction key (e.g.
+ "open AI" -> "OpenAI") into the corrected word(s).
+
+ Segment text gets multi-word corrections for free from the regex pass
+ below, but captions are burned from individual words. Left split,
+ "open" and "AI" render as two separate caption words instead of the
+ fix. The merged word(s) span from the start of the first matched word
+ to the end of the last one, so caption timing stays continuous.
+ """
+ multiword = {k: v for k, v in corrections.items() if len(k.split()) > 1}
+ if not multiword or not words:
+ return words
+
+ # Longest key first so a 3-word phrase wins over a 2-word prefix of it.
+ key_tokens = sorted(
+ ((key.split(), value) for key, value in multiword.items()),
+ key=lambda kv: len(kv[0]),
+ reverse=True,
+ )
+
+ merged: list[dict] = []
+ i = 0
+ n = len(words)
+ while i < n:
+ match = None
+ for tokens, replacement in key_tokens:
+ span = len(tokens)
+ if i + span > n:
+ continue
+ candidate = words[i : i + span]
+ candidate_norm = [_strip_for_match(w.get("word", "")).lower() for w in candidate]
+ if candidate_norm == [t.lower() for t in tokens]:
+ match = (candidate, replacement)
+ break
+ if match is None:
+ merged.append(words[i])
+ i += 1
+ continue
+
+ candidate, replacement = match
+ first, last = candidate[0], candidate[-1]
+ repl_words = replacement.split() or [replacement]
+ span_start = first["start"]
+ span_end = last["end"]
+ span_dur = max(0.0, span_end - span_start)
+ per = span_dur / len(repl_words)
+
+ # The matched words' punctuation and other fields (speaker,
+ # confidence, ...) would otherwise vanish behind the correction:
+ # "open AI." becomes "OpenAI" with no period and a dropped speaker.
+ # Carry the last word's trailing punctuation onto the final merged
+ # word, and the first word's other fields onto every merged word.
+ trailing_punct = _trailing_punct(last.get("word", ""))
+ confidences = [
+ w["confidence"] for w in candidate if isinstance(w.get("confidence"), (int, float))
+ ]
+ extra_fields = {k: v for k, v in first.items() if k not in ("word", "start", "end")}
+ if confidences:
+ extra_fields["confidence"] = min(confidences)
+
+ for idx, rw in enumerate(repl_words):
+ w_start = span_start + per * idx
+ w_end = span_end if idx == len(repl_words) - 1 else span_start + per * (idx + 1)
+ word_text = rw + trailing_punct if idx == len(repl_words) - 1 else rw
+ merged.append({**extra_fields, "word": word_text, "start": w_start, "end": w_end})
+ i += len(candidate)
+
+ return merged
+
+
def apply_corrections(
words: list[dict],
segments: list[dict],
@@ -81,7 +166,11 @@ def apply_corrections(
Apply corrections to transcript words and segments in-place.
Modifies the 'word' field in each word dict and the 'text' field
- in each segment dict. Returns the same lists (mutated).
+ in each segment dict. Multi-word corrections can change the number of
+ words (several words merge into the correction's word(s)), so the
+ `words` list itself is replaced in-place via slice assignment, so
+ callers that hold a reference to the original list still see the
+ update. Returns the same lists (mutated).
"""
corrections = _load_corrections()
if not corrections:
@@ -93,6 +182,12 @@ def apply_corrections(
replacer = lambda m: _replace_match(m, corrections)
+ # Merge multi-word corrections first so captions (built from words) read
+ # the fix the same way the segment text already does.
+ merged_words = _merge_multiword_corrections(words, corrections)
+ if merged_words is not words:
+ words[:] = merged_words
+
# Fix individual words (strip punctuation for matching, preserve it in output)
for w in words:
word_text = w.get("word", "")
diff --git a/backend/services/engine_comparison.py b/backend/services/engine_comparison.py
new file mode 100644
index 0000000..8429d40
--- /dev/null
+++ b/backend/services/engine_comparison.py
@@ -0,0 +1,299 @@
+"""Compare two transcription engines over the same window of a file.
+
+Transcribes a sample range twice, once per engine, and reports where the two
+transcripts disagree, windowed every 20s by default. This measures
+disagreement between the two outputs, not accuracy against a ground truth.
+Neither engine is assumed correct.
+"""
+
+from __future__ import annotations
+
+import html
+import json
+import os
+import shutil
+import unicodedata
+from typing import Any, Optional
+
+# Unicode categories kept when normalizing a word for comparison: letters (L*),
+# marks (M*, e.g. combining diacritics), numbers (N*). Punctuation and symbols
+# are dropped so "hello," and "hello" agree.
+_KEEP_CATEGORY_PREFIXES = ("L", "M", "N")
+
+
+def normalize_word(word: str) -> str:
+ """NFKC-normalize, casefold, and strip everything but letters/marks/numbers."""
+ normalized = unicodedata.normalize("NFKC", word or "").casefold()
+ return "".join(ch for ch in normalized if unicodedata.category(ch)[0] in _KEEP_CATEGORY_PREFIXES)
+
+
+def normalize_words(words: list[str]) -> list[str]:
+ return [w for w in (normalize_word(w) for w in words) if w]
+
+
+def word_levenshtein(a: list[str], b: list[str]) -> int:
+ """Levenshtein distance over word sequences (insert/delete/substitute, cost 1 each)."""
+ if a == b:
+ return 0
+ if not a:
+ return len(b)
+ if not b:
+ return len(a)
+ prev = list(range(len(b) + 1))
+ for i, wa in enumerate(a, 1):
+ curr = [i] + [0] * len(b)
+ for j, wb in enumerate(b, 1):
+ cost = 0 if wa == wb else 1
+ curr[j] = min(
+ prev[j] + 1, # delete from a
+ curr[j - 1] + 1, # insert into a
+ prev[j - 1] + cost, # substitute
+ )
+ prev = curr
+ return prev[-1]
+
+
+def disagreement_ratio(text_a: str, text_b: str) -> float:
+ """Levenshtein over normalized words, divided by the longer word count.
+ 0.0 = identical (after normalization), 1.0 = completely different.
+ Two empty texts disagree 0.0 (nothing to disagree about), so callers should
+ flag that case separately rather than reading it as agreement."""
+ words_a = normalize_words(text_a.split())
+ words_b = normalize_words(text_b.split())
+ denom = max(len(words_a), len(words_b))
+ if denom == 0:
+ return 0.0
+ return word_levenshtein(words_a, words_b) / denom
+
+
+def _words_in_window(words: list[dict], window_start: float, window_end: float) -> str:
+ """Join words whose midpoint falls in [window_start, window_end). Matches
+ the midpoint-ownership rule used elsewhere so a word isn't double-counted
+ in two adjacent windows."""
+ picked = []
+ for w in words:
+ try:
+ start, end = float(w.get("start", 0.0)), float(w.get("end", 0.0))
+ except (TypeError, ValueError):
+ continue
+ midpoint = (start + end) / 2.0
+ if window_start <= midpoint < window_end:
+ text = str(w.get("word", "")).strip()
+ if text:
+ picked.append(text)
+ return " ".join(picked)
+
+
+def build_windows(
+ words_a: list[dict],
+ words_b: list[dict],
+ total_duration: float,
+ window_seconds: float = 20.0,
+) -> list[dict]:
+ windows = []
+ n_windows = max(1, int(total_duration // window_seconds) + (1 if total_duration % window_seconds else 0))
+ for i in range(n_windows):
+ start = i * window_seconds
+ end = min(start + window_seconds, total_duration)
+ text_a = _words_in_window(words_a, start, end)
+ text_b = _words_in_window(words_b, start, end)
+ both_empty = not text_a and not text_b
+ windows.append({
+ "start": round(start, 3),
+ "end": round(end, 3),
+ "text_a": text_a,
+ "text_b": text_b,
+ "both_empty": both_empty,
+ "disagreement": 0.0 if both_empty else round(disagreement_ratio(text_a, text_b), 4),
+ })
+ return windows
+
+
+def compare_engines(
+ file_path: str,
+ engine_a: str,
+ engine_b: str,
+ *,
+ start_seconds: float = 0.0,
+ duration_seconds: Optional[float] = None,
+ window_seconds: float = 20.0,
+ model_size: str = "base",
+ language: Optional[str] = None,
+ output_dir: Optional[str] = None,
+ transcribe_fn=None,
+) -> dict:
+ """Transcribe the same [start_seconds, start_seconds + duration_seconds)
+ window with engine_a and engine_b, and report per-window disagreement.
+
+ transcribe_fn defaults to services.transcription.transcribe_file; tests
+ inject a fake to avoid running real engines.
+ """
+ if transcribe_fn is None:
+ from services.transcription import transcribe_file as transcribe_fn
+
+ result_a = transcribe_fn(
+ file_path, model_size=model_size, engine=engine_a, language=language,
+ enable_diarization=False, start_seconds=start_seconds, duration_seconds=duration_seconds,
+ )
+ result_b = transcribe_fn(
+ file_path, model_size=model_size, engine=engine_b, language=language,
+ enable_diarization=False, start_seconds=start_seconds, duration_seconds=duration_seconds,
+ )
+
+ duration = max(
+ float(result_a.get("duration", 0.0) or 0.0),
+ float(result_b.get("duration", 0.0) or 0.0),
+ duration_seconds or 0.0,
+ )
+ windows = build_windows(
+ result_a.get("words") or [], result_b.get("words") or [], duration, window_seconds
+ )
+ scored = [w["disagreement"] for w in windows if not w["both_empty"]]
+ overall_disagreement = round(sum(scored) / len(scored), 4) if scored else 0.0
+ flagged_empty = sum(1 for w in windows if w["both_empty"])
+
+ report = {
+ "file_path": file_path,
+ "engine_a": engine_a,
+ "engine_b": engine_b,
+ "start_seconds": start_seconds,
+ "duration_seconds": duration,
+ "window_seconds": window_seconds,
+ "windows": windows,
+ "overall_disagreement": overall_disagreement,
+ "both_empty_window_count": flagged_empty,
+ "note": "disagreement between the two engines' output, not accuracy against a transcript",
+ }
+
+ if output_dir:
+ os.makedirs(output_dir, exist_ok=True)
+ json_path = os.path.join(output_dir, "comparison.json")
+ with open(json_path, "w", encoding="utf-8") as f:
+ json.dump(report, f, indent=2, ensure_ascii=False)
+ report["json_path"] = json_path
+
+ # A fresh extraction of the same window, purely for the report's
+ # <audio> player, independent of whatever transcribe_fn did
+ # internally (its own sample wav is already cleaned up by the time
+ # we get a result back).
+ audio_rel = None
+ try:
+ from services.audio_extract import extract_wav_16k_mono
+
+ sample_wav = extract_wav_16k_mono(
+ file_path, start_seconds=start_seconds, duration_seconds=duration_seconds
+ )
+ try:
+ audio_rel = "sample.wav"
+ shutil.copyfile(sample_wav, os.path.join(output_dir, audio_rel))
+ finally:
+ try:
+ os.unlink(sample_wav)
+ except OSError:
+ pass
+ except Exception:
+ audio_rel = None
+
+ html_path = os.path.join(output_dir, "comparison.html")
+ with open(html_path, "w", encoding="utf-8") as f:
+ f.write(render_html(report, audio_rel))
+ report["html_path"] = html_path
+
+ return report
+
+
+def _json_for_script_tag(data: Any) -> str:
+ """json.dumps, safe to embed inside a <script> tag.
+
+ Replacing only a literal "</script" (case-sensitively) left "</SCRIPT>"
+ or "</ScRiPt>" in ASR text able to close the tag early and inject HTML.
+ Escaping every "<" closes that regardless of case, and also neutralizes
+ "<!--". U+2028/U+2029 are escaped too: valid in a JSON string, but
+ treated as line terminators by some JS engines even inside a string
+ literal, which can truncate the script.
+ """
+ encoded = json.dumps(data, ensure_ascii=False)
+ return (
+ encoded.replace("<", "\\u003c")
+ .replace("
", "\\u2028")
+ .replace("
", "\\u2029")
+ )
+
+
+def render_html(report: dict, audio_rel: Optional[str]) -> str:
+ engine_a = html.escape(str(report["engine_a"]))
+ engine_b = html.escape(str(report["engine_b"]))
+ file_label = html.escape(os.path.basename(str(report["file_path"])))
+ overall = report["overall_disagreement"]
+ windows_json = _json_for_script_tag(report["windows"])
+ audio_html = (
+ f'<audio id="player" controls preload="metadata" src="{html.escape(audio_rel)}"></audio>'
+ if audio_rel
+ else "<p class=\"no-audio\">No sample audio saved alongside this report.</p>"
+ )
+ return f"""<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Engine comparison: {engine_a} vs {engine_b} ({file_label})</title>
+<style>
+ body {{ font-family: -apple-system, system-ui, sans-serif; margin: 2rem; color: #1a1a1a; background: #fafafa; }}
+ h1 {{ font-size: 1.25rem; }}
+ .note {{ color: #666; font-size: 0.9rem; margin-bottom: 1.5rem; }}
+ table {{ border-collapse: collapse; width: 100%; }}
+ th, td {{ border: 1px solid #ddd; padding: 0.5rem 0.75rem; text-align: left; vertical-align: top; }}
+ th {{ background: #f0f0f0; }}
+ tr.both-empty {{ background: #fff8e1; }}
+ .disagreement {{ font-variant-numeric: tabular-nums; }}
+ .disagreement.high {{ color: #b00020; font-weight: 600; }}
+ .disagreement.low {{ color: #2e7d32; }}
+ button.seek {{ cursor: pointer; }}
+ audio {{ width: 100%; margin-bottom: 1.5rem; }}
+ .no-audio {{ color: #999; font-style: italic; }}
+</style>
+</head>
+<body>
+<h1>Engine comparison: {engine_a} vs {engine_b}</h1>
+<p class="note">File: {file_label}. Overall disagreement: <span class="disagreement">{overall}</span>
+(word-level edit distance between the two engines' output, normalized; not accuracy against either transcript).</p>
+{audio_html}
+<table>
+ <thead><tr><th>Window</th><th>{engine_a}</th><th>{engine_b}</th><th>Disagreement</th></tr></thead>
+ <tbody id="rows"></tbody>
+</table>
+<script>
+const windows = {windows_json};
+const rows = document.getElementById("rows");
+const player = document.getElementById("player");
+function esc(s) {{
+ const d = document.createElement("div");
+ d.textContent = s;
+ return d.innerHTML;
+}}
+function fmtTime(s) {{
+ const m = Math.floor(s / 60);
+ const sec = Math.floor(s % 60).toString().padStart(2, "0");
+ return m + ":" + sec;
+}}
+for (const w of windows) {{
+ const tr = document.createElement("tr");
+ if (w.both_empty) tr.className = "both-empty";
+ const level = w.disagreement >= 0.5 ? "high" : w.disagreement <= 0.1 ? "low" : "";
+ tr.innerHTML =
+ '<td><button class="seek" data-t="' + w.start + '">' + fmtTime(w.start) + '</button>' +
+ ' - ' + fmtTime(w.end) + (w.both_empty ? ' <em>(both empty)</em>' : '') + '</td>' +
+ '<td>' + esc(w.text_a) + '</td>' +
+ '<td>' + esc(w.text_b) + '</td>' +
+ '<td class="disagreement ' + level + '">' + w.disagreement.toFixed(2) + '</td>';
+ rows.appendChild(tr);
+}}
+rows.addEventListener("click", (e) => {{
+ const btn = e.target.closest("button.seek");
+ if (!btn || !player) return;
+ player.currentTime = parseFloat(btn.dataset.t);
+ player.play();
+}});
+</script>
+</body>
+</html>
+"""
diff --git a/backend/services/engines.py b/backend/services/engines.py
index 1fd49b8..a8de58b 100644
--- a/backend/services/engines.py
+++ b/backend/services/engines.py
@@ -4,8 +4,14 @@ def normalize_engine(name: str | None) -> str:
return "whispercpp"
if value in ("assemblyai", "assembly-ai", "aai"):
return "assemblyai"
+ if value in ("omnilingual", "omni", "omnilingual-asr"):
+ return "omnilingual"
return "whisper-py"
def is_assemblyai_engine(name: str | None) -> bool:
return normalize_engine(name) == "assemblyai"
+
+
+def is_omnilingual_engine(name: str | None) -> bool:
+ return normalize_engine(name) == "omnilingual"
diff --git a/backend/services/glyph_coverage.py b/backend/services/glyph_coverage.py
new file mode 100644
index 0000000..2d592dc
--- /dev/null
+++ b/backend/services/glyph_coverage.py
@@ -0,0 +1,166 @@
+"""Caption font glyph coverage: a pre-render check, not a render step.
+
+Before burning captions, verify the font that will actually draw them has
+glyphs for every character in the clip's text. A font silently missing a
+glyph doesn't fail the render; it draws a tofu box (or nothing), which only
+shows up when someone watches the clip. This surfaces that as a warning
+instead, naming the characters, without blocking the render.
+
+Two render paths, two coverage sources:
+ - ASS (libass/ffmpeg): the font is resolved by family name via fontconfig,
+ the same way caption_renderer.py's pixel-width measurer resolves it
+ (fc-match). fontconfig already ships wherever ASS rendering does, so
+ reading a font's charset via `fc-query` avoids adding fontTools/Pillow
+ glyph-table parsing as a new dependency.
+ - Remotion: fonts are bundled directly (see remotion/src/Root.tsx), not
+ resolved from the system, so there's no font *file* to query at render
+ time the way fc-query queries one. REMOTION_BUNDLED_SUBSETS mirrors
+ exactly which Unicode subsets Root.tsx loads via FontFace; keep the two
+ in sync if that import list changes.
+"""
+
+import os
+from typing import Optional
+
+from utils.proc import run as proc_run
+
+# Keep in sync with the FontFace loads in remotion/src/Root.tsx. Each bundled
+# @fontsource package ships one or more of these subsets; this lists only
+# the subsets actually imported there, not everything the packages offer.
+REMOTION_BUNDLED_SUBSETS = frozenset({"latin", "latin-ext", "georgian"})
+
+# Coarse Unicode block -> fontsource-style subset name. Enough to catch a
+# script with no bundled Remotion font at all (which falls through to an
+# unverified system "sans-serif"), without needing full OpenType script
+# tagging.
+_BLOCK_SUBSETS: tuple[tuple[int, int, str], ...] = (
+ (0x0000, 0x007F, "latin"),
+ (0x0080, 0x024F, "latin-ext"),
+ (0x0370, 0x03FF, "greek"),
+ (0x0400, 0x04FF, "cyrillic"),
+ (0x0500, 0x052F, "cyrillic-ext"),
+ (0x0590, 0x05FF, "hebrew"),
+ (0x0600, 0x06FF, "arabic"),
+ (0x10A0, 0x10FF, "georgian"),
+ (0x1C90, 0x1CBF, "georgian"),
+ (0x2D00, 0x2D2F, "georgian"),
+ (0x4E00, 0x9FFF, "cjk"),
+ (0x3040, 0x30FF, "japanese"),
+ (0xAC00, 0xD7AF, "korean"),
+)
+
+
+def _subset_for(codepoint: int) -> Optional[str]:
+ for lo, hi, subset in _BLOCK_SUBSETS:
+ if lo <= codepoint <= hi:
+ return subset
+ return None
+
+
+def _unique_chars_outside_coverage(text: str, is_covered) -> list[str]:
+ seen: set[str] = set()
+ missing: list[str] = []
+ for ch in text:
+ if ch.isspace() or ch in seen:
+ continue
+ seen.add(ch)
+ if not is_covered(ch):
+ missing.append(ch)
+ return missing
+
+
+def _format_warning(missing: list[str], where: str) -> str:
+ sample = "".join(missing[:20])
+ tail = " and more" if len(missing) > 20 else ""
+ return (
+ f"Caption font may be missing glyphs for: {sample}{tail} "
+ f"({len(missing)} unique character(s), {where})."
+ )
+
+
+def resolve_ass_font_path(font_name: str, bold: bool = False) -> Optional[str]:
+ """The exact font file fc-match/libass resolves for an ASS style, the
+ same resolution caption_renderer.py's pixel-width measurer uses.
+ """
+ style = "Bold" if bold else "Regular"
+ try:
+ result = proc_run(
+ ["fc-match", f"{font_name}:{style}", "--format=%{file}"],
+ timeout=3, check=False,
+ )
+ except Exception:
+ return None
+ if result.returncode != 0:
+ return None
+ path = (result.stdout or "").strip()
+ return path if path and os.path.exists(path) else None
+
+
+def _font_charset_ranges(font_path: str) -> Optional[list[tuple[int, int]]]:
+ """Parse `fc-query --format=%{charset}` into (lo, hi) codepoint ranges."""
+ try:
+ result = proc_run(
+ ["fc-query", "--format=%{charset}", font_path],
+ timeout=5, check=False,
+ )
+ except Exception:
+ return None
+ if result.returncode != 0:
+ return None
+ ranges: list[tuple[int, int]] = []
+ for token in (result.stdout or "").split():
+ parts = token.split("-", 1)
+ try:
+ lo = int(parts[0], 16)
+ hi = int(parts[1], 16) if len(parts) > 1 else lo
+ except ValueError:
+ continue
+ ranges.append((lo, hi))
+ return ranges
+
+
+def check_ass_font_coverage(text: str, font_name: str, bold: bool = False) -> Optional[str]:
+ """Warning naming characters the resolved ASS font has no glyph for, or
+ None if it covers everything (or coverage couldn't be determined; this
+ never fails the render, so an undetermined font is treated as fine).
+ """
+ if not text:
+ return None
+ font_path = resolve_ass_font_path(font_name, bold)
+ if not font_path:
+ return None
+ ranges = _font_charset_ranges(font_path)
+ if ranges is None:
+ return None
+ missing = _unique_chars_outside_coverage(
+ text, lambda ch: any(lo <= ord(ch) <= hi for lo, hi in ranges)
+ )
+ if not missing:
+ return None
+ return _format_warning(missing, f"font '{font_name}'")
+
+
+def check_remotion_font_coverage(text: str) -> Optional[str]:
+ """Warning naming characters whose script has no bundled Remotion font
+ at all, so the render would fall back to an unverified system font.
+ """
+ if not text:
+ return None
+ missing = _unique_chars_outside_coverage(
+ text,
+ lambda ch: (lambda s: s is None or s in REMOTION_BUNDLED_SUBSETS)(
+ _subset_for(ord(ch))
+ ),
+ )
+ if not missing:
+ return None
+ return _format_warning(missing, "no bundled Remotion font covers this script")
+
+
+def check_caption_font_coverage(
+ text: str, font_name: str, bold: bool, use_ass: bool
+) -> Optional[str]:
+ """Dispatch to whichever render path will actually draw the captions."""
+ if use_ass:
+ return check_ass_font_coverage(text, font_name, bold)
+ return check_remotion_font_coverage(text)
diff --git a/backend/services/integrations/_shared/multicam_xml.py b/backend/services/integrations/_shared/multicam_xml.py
index f9dd6ce..2fc6992 100644
--- a/backend/services/integrations/_shared/multicam_xml.py
+++ b/backend/services/integrations/_shared/multicam_xml.py
@@ -8,8 +8,11 @@ audio track.
Both follow the same render plan as the MP4, so removed stretches drop out of
picture and mics at the same frames. Clock drift can't be expressed as a clip
-property in either format, so a drifting mic is split at every shot and each
-piece gets its own in-point, which keeps it within a millisecond of picture.
+property in either format, so a drifting file is split into pieces short
+enough that its clock slips at most half a frame across each, and each piece's
+in-point is mapped at its middle: drift adds at most a quarter of a frame at a
+piece's ends. xmeml addresses source media in whole frames, so its in-points
+also round to the nearest frame; FCPXML keeps mics sample-accurate.
A review timeline keeps the whole episode instead: every camera on its own
track under the cut, and each removed stretch left in place, split out on
@@ -42,17 +45,39 @@ def _text(parent: ET.Element, tag: str, value) -> ET.Element:
return el
+def _total_audio_channels(source) -> int:
+ """Every channel in the file, across all of its audio streams.
+
+ A camera that carries one mono stream per mic exposes each as its own
+ audio_stream_channels entry; an editor addresses them as consecutive
+ track/channel numbers across the whole file, not per stream.
+ """
+ return sum(source.audio_stream_channels) or max(1, source.audio_channels)
+
+
+def _stream_channel_offset(source) -> int:
+ """How many channels sit before the chosen audio_stream_index in the file.
+
+ Lets a track/channel index built from `channel` alone (0-based within the
+ selected stream) land on the right stream instead of always the first.
+ """
+ return sum(source.audio_stream_channels[:source.audio_stream_index])
+
+
def _pieces(source, plan, fps: float) -> list[tuple[int, int, float, dict | None]]:
"""(output_frame_in, output_frame_out, source_seconds_at_in, removal) for one file across the episode.
Follows the render plan, so removed stretches drop out of the mics exactly
where they drop out of the picture. Pieces that continue each other on the
timeline are joined unless one of them is a removal; a drifting file is left
- split at every shot, so each piece gets its own in-point and stays within a
- millisecond of picture.
+ split at every shot and again wherever its clock would slip past half a
+ frame, and each piece's in-point is mapped at its middle.
"""
+ from services.multicam import drift_parts
+
pieces: list[list] = []
- joinable = abs(source.speed - 1.0) <= 1e-6
+ drift = abs(source.speed - 1.0)
+ joinable = drift <= 1e-6
last_end = None
for p in plan:
a = p.tl_start
@@ -68,11 +93,42 @@ def _pieces(source, plan, fps: float) -> list[tuple[int, int, float, dict | None
and last_end is not None and abs(last_end - lo) < 1e-3):
pieces[-1][1] = f1
else:
- pieces.append([f0, f1, source.source_time(p.tl_start + (f0 - p.out_frame) / fps), p.removal])
+ parts = drift_parts((f1 - f0) / fps, drift, fps)
+ edges = [f0 + round((f1 - f0) * k / parts) for k in range(parts + 1)]
+ for g0, g1 in zip(edges, edges[1:]):
+ if g1 > g0:
+ tl = p.tl_start + (g0 - p.out_frame) / fps
+ pieces.append([g0, g1, source.source_in(tl, (g1 - g0) / fps), p.removal])
last_end = hi
return [tuple(x) for x in pieces]
+def _fit_scale(source, width: int, height: int) -> float | None:
+ """Percent scale that fits a camera inside the sequence frame, or None when it already matches.
+
+ Premiere places a clip at 100% of its own pixels, so a 4K camera on an HD
+ sequence would show only its middle quarter.
+ """
+ if source.kind != "video" or not source.width or not source.height or (source.width, source.height) == (width, height):
+ return None
+ return min(width / source.width, height / source.height) * 100
+
+
+def _basic_motion(item: ET.Element, scale: float) -> None:
+ effect = ET.SubElement(ET.SubElement(item, "filter"), "effect")
+ _text(effect, "name", "Basic Motion")
+ _text(effect, "effectid", "basic")
+ _text(effect, "effectcategory", "motion")
+ _text(effect, "effecttype", "motion")
+ _text(effect, "mediatype", "video")
+ param = ET.SubElement(effect, "parameter", {"authoringApp": "PremierePro"})
+ _text(param, "parameterid", "scale")
+ _text(param, "name", "Scale")
+ _text(param, "valuemin", 0)
+ _text(param, "valuemax", 1000)
+ _text(param, "value", f"{scale:.2f}")
+
+
def _label(session, source, channel: int, person: str) -> str:
names = {p.id: p.name for p in session.people}
base = names.get(person, "Room")
@@ -164,7 +220,7 @@ def write_xmeml(session, out_path: Path, *, width: int, height: int, fps: float,
_text(sc, "height", source.height)
if source.has_audio:
a = ET.SubElement(fm, "audio")
- _text(a, "channelcount", max(1, source.audio_channels))
+ _text(a, "channelcount", _total_audio_channels(source))
def clipitem(track: ET.Element, source, f0: int, f1: int, src_seconds: float,
*, name: str, audio_channel: int | None = None, removal: dict | None = None) -> None:
@@ -183,7 +239,9 @@ def write_xmeml(session, out_path: Path, *, width: int, height: int, fps: float,
if audio_channel is not None:
st = ET.SubElement(item, "sourcetrack")
_text(st, "mediatype", "audio")
- _text(st, "trackindex", max(1, audio_channel + 1))
+ _text(st, "trackindex", _stream_channel_offset(source) + max(0, audio_channel) + 1)
+ elif (scale := _fit_scale(source, width, height)) is not None:
+ _basic_motion(item, scale)
if removal is not None:
labels = ET.SubElement(item, "labels")
_text(labels, "label2", "Mango")
@@ -197,7 +255,7 @@ def write_xmeml(session, out_path: Path, *, width: int, height: int, fps: float,
if not p.source_id:
continue
cam = session.source(p.source_id)
- clipitem(vtrack, cam, p.out_frame, p.out_frame + p.frames, cam.source_time(p.tl_start),
+ clipitem(vtrack, cam, p.out_frame, p.out_frame + p.frames, cam.source_in(p.tl_start, p.frames / fps),
name=os.path.basename(cam.path), removal=p.removal)
audio = ET.SubElement(media, "audio")
@@ -266,7 +324,7 @@ def write_fcpxml(session, out_path: Path, *, width: int, height: int, fps: float
format_id=formats[key],
has_video=True,
has_audio=source.has_audio,
- audio_channels=source.audio_channels,
+ audio_channels=_total_audio_channels(source),
start=src_t(source, 0.0),
))
else:
@@ -278,7 +336,7 @@ def write_fcpxml(session, out_path: Path, *, width: int, height: int, fps: float
"hasVideo": "0",
"hasAudio": "1",
"audioSources": "1",
- "audioChannels": str(max(1, source.audio_channels)),
+ "audioChannels": str(_total_audio_channels(source)),
"audioRate": "48000",
})
ET.SubElement(asset, "media-rep", {"kind": "original-media", "src": fx.file_uri(Path(source.path))})
@@ -295,9 +353,14 @@ def write_fcpxml(session, out_path: Path, *, width: int, height: int, fps: float
# lanes, so every clip is positioned against the same zero.
gap = ET.SubElement(spine, "gap", {"name": "Episode", "offset": "0s", "start": "0s", "duration": t(total)})
+ def conform(clip: ET.Element, source) -> None:
+ # FCPXML's own fit conform does what Premiere's Basic Motion scale does there.
+ if _fit_scale(source, width, height) is not None:
+ ET.SubElement(clip, "adjust-conform", {"type": "fit"})
+
for lane, cam in enumerate(angles, start=1):
for f0, f1, src_seconds, removal in _pieces(cam, plan, fps):
- ET.SubElement(gap, "asset-clip", {
+ angle = ET.SubElement(gap, "asset-clip", {
"ref": asset_ids[cam.id],
"lane": str(lane),
"offset": t(f0),
@@ -306,11 +369,12 @@ def write_fcpxml(session, out_path: Path, *, width: int, height: int, fps: float
"name": _removed_name(os.path.basename(cam.path), removal),
"srcEnable": "video",
})
+ conform(angle, cam)
for p in plan:
if not p.source_id:
continue
cam = session.source(p.source_id)
- start = src_t(cam, cam.source_time(p.tl_start))
+ start = src_t(cam, cam.source_in(p.tl_start, p.frames / fps))
clip = ET.SubElement(gap, "asset-clip", {
"ref": asset_ids[p.source_id],
"lane": str(len(angles) + 1),
@@ -320,6 +384,7 @@ def write_fcpxml(session, out_path: Path, *, width: int, height: int, fps: float
"name": _removed_name(os.path.basename(cam.path), p.removal),
"srcEnable": "video",
})
+ conform(clip, cam)
if p.removal is not None:
ET.SubElement(clip, "marker", {
"start": start, "duration": t(p.frames), "value": p.removal.get("reason") or "Remove",
@@ -337,7 +402,10 @@ def write_fcpxml(session, out_path: Path, *, width: int, height: int, fps: float
"audioRole": "dialogue",
"srcEnable": "audio",
})
- if channel >= 0:
- ET.SubElement(clip, "audio-channel-source", {"srcCh": str(channel + 1), "role": "dialogue"})
+ offset = _stream_channel_offset(source)
+ if channel >= 0 or offset:
+ ET.SubElement(clip, "audio-channel-source", {
+ "srcCh": str(offset + max(0, channel) + 1), "role": "dialogue",
+ })
fx.write_fcpxml(out_path, resources, library)
diff --git a/backend/services/integrations/youtube/learnings.py b/backend/services/integrations/youtube/learnings.py
index 049746e..75a97e5 100644
--- a/backend/services/integrations/youtube/learnings.py
+++ b/backend/services/integrations/youtube/learnings.py
@@ -8,10 +8,15 @@ actually performed.
from __future__ import annotations
import os
+import statistics
from collections import defaultdict
from datetime import datetime, timezone
from typing import Any, Callable, Optional
+# Below this, one viral or one flop clip swings the bucket's number enough
+# that it reads as a trend. Hide it until there's enough data to trust.
+MIN_BUCKET_SIZE = 4
+
from config.paths import paths
from services.clips_history import load_clips_history
@@ -34,18 +39,24 @@ def _has_perf(c: dict) -> bool:
def _agg(clips: list[dict], key: Callable[[dict], Any]) -> list[dict]:
+ """Group clips by `key` and report the median (not mean) of each metric,
+ since one viral or one flop clip would otherwise swing a small bucket's
+ average far past what's typical. Buckets under MIN_BUCKET_SIZE are
+ dropped: too little data to call it a trend."""
groups: dict[str, list[dict]] = defaultdict(list)
for c in clips:
groups[key(c) or "—"].append(c)
out = []
for k, cs in groups.items():
+ if len(cs) < MIN_BUCKET_SIZE:
+ continue
vals = lambda f: [c["metrics"][f] for c in cs if (c.get("metrics") or {}).get(f) is not None]
ret, ctr, views = vals("retention"), vals("ctr"), vals("views")
out.append({
"key": k, "n": len(cs),
- "ret": round(sum(ret) / len(ret), 1) if ret else None,
- "ctr": round(sum(ctr) / len(ctr), 1) if ctr else None,
- "views": int(sum(views) / len(views)) if views else None,
+ "ret": round(statistics.median(ret), 1) if ret else None,
+ "ctr": round(statistics.median(ctr), 1) if ctr else None,
+ "views": int(statistics.median(views)) if views else None,
})
return out
diff --git a/backend/services/multicam.py b/backend/services/multicam.py
index 4994033..2273805 100644
--- a/backend/services/multicam.py
+++ b/backend/services/multicam.py
@@ -24,6 +24,7 @@ import uuid
import wave
from concurrent.futures import ThreadPoolExecutor
from dataclasses import asdict, dataclass, field
+from fractions import Fraction
from pathlib import Path
from typing import Callable, Optional
@@ -112,11 +113,18 @@ class Source:
height: int = 0
fps: float = 0.0
timecode: float = 0.0 # embedded start timecode in seconds; editors address media from here
+ file_size: int = 0 # size at probe time; a mismatch on reopen means the file changed underneath us
+ file_mtime_ns: int = 0 # mtime at probe time, nanosecond resolution
+ fps_warning: str = "" # set when the container's frame rate looks variable or had to be guessed
+ audio_stream_count: int = 1 # separate audio streams in the container (not channels within one stream)
+ audio_stream_index: int = 0 # which audio stream to use, for MXF-style cameras with one mono stream per mic
+ audio_stream_channels: list[int] = field(default_factory=list) # channel count per audio stream, by index
role: str = "ignore" # "camera" | "mic" | "ignore"
# camera: a person id or "wide"; mic: a person id, or "" for a shared room mic
person: str = ""
# mic only: one person id per channel when a recorder puts two people on L/R
channel_people: list[str] = field(default_factory=list)
+ input_lut: str = "" # camera only: absolute path to a 3D .cube LUT applied before the look
guessed: bool = True
offset: Optional[float] = None # timeline seconds where source time 0 sits
speed: float = 1.0 # timeline seconds per source second (clock drift)
@@ -144,6 +152,15 @@ class Source:
def source_time(self, timeline_seconds: float) -> float:
return (timeline_seconds - self.timeline_start()) / self.speed
+ def source_in(self, timeline_seconds: float, span: float) -> float:
+ """Source second to start reading a `span`-second piece at, played at real speed.
+
+ A drifting clock slips |speed - 1| * span across a piece; pinning the
+ piece's middle instead of its start halves the worst slip.
+ """
+ middle = timeline_seconds + span / 2
+ return self.source_time(middle) - span / 2
+
@dataclass
class MulticamSession:
@@ -275,16 +292,68 @@ def scan_folder(folder: str) -> list[str]:
return found
-def _timecode_seconds(info: dict, fps: float, sample_rate: int) -> float:
- """Embedded start timecode (pro cameras) or BWF time reference (field recorders), in seconds."""
+_TIMECODE_RE = re.compile(r"^(\d{2})([:;.])(\d{2})([:;.])(\d{2})([:;.])(\d{2})$")
+
+
+def _ntsc_frame_duration(fps: float, r_frame_rate: str = "") -> tuple[float, int]:
+ """Exact frame duration and the nominal (rounded) frame rate for an NTSC-pulldown fps.
+
+ 29.97 is really 30000/1001 fps, so each frame lasts 1001/30000 s, not 1/30 s. The
+ same pulldown ratio applies to 59.94 and 23.976. Integer rates (25, 24, 30 exactly)
+ have no pulldown and use a plain 1/fps duration.
+
+ avg_frame_rate (what `fps` usually comes from) is measured, not declared, and a
+ steady 25 fps camera can read 24.98 by measurement noise alone, which a bare
+ closeness check mistakes for pulldown. r_frame_rate is the stream's exact time
+ base; every real pulldown rate reduces to a fraction with denominator 1001, so
+ when it's available that decides, not the noisy float.
+ """
+ if r_frame_rate:
+ try:
+ exact = Fraction(str(r_frame_rate))
+ except (ValueError, ZeroDivisionError):
+ exact = None
+ if exact and exact > 0:
+ rounded = round(float(exact))
+ if rounded <= 0:
+ return 0.0, 0
+ if exact.denominator == 1001:
+ return 1001.0 / (rounded * 1000.0), rounded
+ return 1.0 / rounded, rounded
+ rounded = round(fps)
+ if rounded <= 0:
+ return 0.0, 0
+ if abs(fps - rounded) > 1e-3:
+ return 1001.0 / (rounded * 1000.0), rounded
+ return 1.0 / rounded, rounded
+
+
+def _timecode_seconds(info: dict, fps: float, sample_rate: int, r_frame_rate: str = "") -> float:
+ """Embedded start timecode (pro cameras) or BWF time reference (field recorders), in seconds.
+
+ Honors drop-frame timecode (separator ';' before the frame field): drop-frame
+ counters skip frame numbers :00 and :01 at the start of every minute except every
+ tenth, so the raw H:M:S:F reading overstates elapsed time unless those skipped
+ counts are added back before converting to seconds.
+ """
tags = [info.get("format", {}).get("tags") or {}] + [s.get("tags") or {} for s in info.get("streams", [])]
for t in tags:
tc = t.get("timecode") or t.get("TIMECODE")
if tc and fps > 0:
- parts = re.split(r"[:;.]", tc)
- if len(parts) == 4 and all(p.isdigit() for p in parts):
- h, m, sec, frames = (int(p) for p in parts)
- return h * 3600 + m * 60 + sec + frames / round(fps)
+ m = _TIMECODE_RE.match(str(tc).strip())
+ if m:
+ h, m1, mi, m2, sec, m3, frames = m.groups()
+ h, mi, sec, frames = int(h), int(mi), int(sec), int(frames)
+ drop_frame = m3 == ";"
+ frame_duration, fps_round = _ntsc_frame_duration(fps, r_frame_rate)
+ if fps_round <= 0:
+ continue
+ total_frames = fps_round * 3600 * h + fps_round * 60 * mi + fps_round * sec + frames
+ if drop_frame:
+ drop_per_min = 2 if fps_round == 30 else (4 if fps_round == 60 else 0)
+ total_minutes = 60 * h + mi
+ total_frames -= drop_per_min * (total_minutes - total_minutes // 10)
+ return total_frames * frame_duration
for t in tags:
ref = t.get("time_reference")
if ref and str(ref).isdigit() and sample_rate > 0:
@@ -300,22 +369,37 @@ def probe_source(path: str) -> Source:
and not (s.get("disposition") or {}).get("attached_pic")),
None,
)
- audio = next((s for s in streams if s.get("codec_type") == "audio"), None)
+ audio_streams = [s for s in streams if s.get("codec_type") == "audio"]
+ audio = audio_streams[0] if audio_streams else None
durations = [float(info.get("format", {}).get("duration") or 0)]
durations += [float(s.get("duration") or 0) for s in streams]
duration = max(durations)
if duration <= 0:
raise RuntimeError(f"Could not read the duration of {os.path.basename(path)}")
fps = 0.0
+ fps_warning = ""
if video:
- num, _, den = str(video.get("avg_frame_rate") or video.get("r_frame_rate") or "0/1").partition("/")
- try:
- fps = float(num) / float(den or 1) if float(den or 1) else 0.0
- except ValueError:
- fps = 0.0
+ def rate(field: str) -> float:
+ num, _, den = str(video.get(field) or "0/1").partition("/")
+ try:
+ return float(num) / float(den or 1) if float(den or 1) else 0.0
+ except ValueError:
+ return 0.0
+
+ avg_fps, r_fps = rate("avg_frame_rate"), rate("r_frame_rate")
+ fps = avg_fps or r_fps
+ # r_frame_rate is the stream's time base, a ceiling on how fast frames
+ # could appear; avg_frame_rate is what actually played out. A material
+ # gap between them means some frames held longer than others, so a shot
+ # cut can't assume a fixed grid: the frame at a given timestamp drifts.
+ if avg_fps and r_fps and abs(avg_fps - r_fps) / r_fps > 0.01:
+ fps_warning = (f"Variable frame rate detected ({avg_fps:.3g} fps average, "
+ f"{r_fps:.3g} fps max): cuts may drift by a frame or more.")
if not 1 <= fps <= 240:
+ fps_warning = f"Could not read a usable frame rate ({fps or 'none'}); assuming 30 fps."
fps = 30.0
sample_rate = int(audio.get("sample_rate") or 0) if audio else 0
+ stat = os.stat(path)
return Source(
# Built from name and size so the same recording gets the same id
# on another machine and a saved edit or cut list still points at it.
@@ -328,10 +412,101 @@ def probe_source(path: str) -> Source:
width=int(video.get("width") or 0) if video else 0,
height=int(video.get("height") or 0) if video else 0,
fps=round(fps, 3),
- timecode=round(_timecode_seconds(info, fps, sample_rate), 6),
+ timecode=round(_timecode_seconds(info, fps, sample_rate, video.get("r_frame_rate") if video else ""), 6),
+ file_size=stat.st_size,
+ file_mtime_ns=stat.st_mtime_ns,
+ fps_warning=fps_warning,
+ audio_stream_count=len(audio_streams),
+ audio_stream_channels=[int(a.get("channels") or 0) for a in audio_streams],
)
+def _file_identity(s: Source) -> str:
+ """Short fingerprint of the file state a source was last probed against.
+
+ Used to key derived caches (extracted sync audio, activity) so that a
+ file silently changing underneath its source (replaced, re-exported,
+ re-encoded) can't serve stale cached work keyed only on the source id,
+ which is built from basename and size and so does not change.
+ """
+ return hashlib.sha1(f"{s.file_size}:{s.file_mtime_ns}".encode()).hexdigest()[:8]
+
+
+def _source_changed_on_disk(s: Source) -> bool:
+ """True if the file at `s.path` no longer matches what was probed.
+
+ A session saved before file identity was tracked has file_size and
+ file_mtime_ns both at their dataclass default of 0. That is not evidence
+ the file shrank to nothing; it means we never recorded an identity for
+ it. Treat that pair as unknown rather than as a mismatch, so an old
+ session doesn't look "changed" on every single source the first time it
+ reopens under the new code.
+ """
+ if s.file_size == 0 and s.file_mtime_ns == 0:
+ return False
+ try:
+ stat = os.stat(s.path)
+ except OSError:
+ return False
+ return stat.st_size != s.file_size or stat.st_mtime_ns != s.file_mtime_ns
+
+
+def backfill_file_identity(session: "MulticamSession") -> bool:
+ """Stamp file_size/file_mtime_ns on sources saved before identity tracking existed.
+
+ Stat-only, and never resets sync, cuts, or range: an old session without
+ this fingerprint hasn't necessarily changed, it just predates the field.
+ """
+ changed = False
+ for s in session.sources:
+ if s.virtual or not s.path or (s.file_size or s.file_mtime_ns):
+ continue
+ try:
+ stat = os.stat(s.path)
+ except OSError:
+ continue
+ s.file_size, s.file_mtime_ns = stat.st_size, stat.st_mtime_ns
+ changed = True
+ return changed
+
+
+def refresh_stale_sources(session: "MulticamSession") -> bool:
+ """Re-probe any source whose file changed since it was last probed.
+
+ A saved session reopens by matching the set of file paths alone, so a
+ camera file swapped out for a re-export with the same name silently kept
+ its old sync, offset and caches. Re-probe it, reset everything derived
+ from its old bytes, and let the normal flows (sync, activity, previews)
+ regenerate against the new file.
+ """
+ affected_in_use = False
+ for s in session.sources:
+ if s.virtual or not s.path or not _source_changed_on_disk(s):
+ continue
+ try:
+ fresh = probe_source(s.path)
+ except Exception:
+ continue
+ s.duration, s.has_audio, s.audio_channels = fresh.duration, fresh.has_audio, fresh.audio_channels
+ s.width, s.height, s.fps, s.timecode = fresh.width, fresh.height, fresh.fps, fresh.timecode
+ s.file_size, s.file_mtime_ns = fresh.file_size, fresh.file_mtime_ns
+ s.fps_warning = fresh.fps_warning
+ s.audio_stream_count = fresh.audio_stream_count
+ s.audio_stream_channels = fresh.audio_stream_channels
+ s.audio_stream_index = min(s.audio_stream_index, max(0, fresh.audio_stream_count - 1))
+ s.offset, s.speed, s.sync = None, 1.0, {
+ "status": "failed",
+ "message": "This file changed on disk since it was last synced. Sync again.",
+ }
+ affected_in_use = affected_in_use or _in_use(session, s)
+ if affected_in_use:
+ session.activity_key = ""
+ session.cuts = []
+ session.range_start = session.range_end = None
+ place_views(session)
+ return affected_in_use
+
+
_WIDE = re.compile(r"(^|[^a-z])(wide|master|main|both|all|group|ws|two.?shot|2.?shot|overview)([^a-z]|$)")
_MIX = re.compile(r"(^|[^a-z])(mix|lr|stereo|master.?mix|program)([^a-z]|$)")
_TRACK = re.compile(r"(?:tr|track|ch|channel|in|input|mic)[ _-]?0*(\d{1,2})(?!\d)")
@@ -601,6 +776,22 @@ def collect_files(folder: str = "", files: Optional[list] = None) -> list[str]:
return found
+def open_session(session_id: str) -> MulticamSession:
+ """Load a session by id, catching up on file-identity bookkeeping once.
+
+ `new_session` already does this for the folder-reopen path. Everything
+ that instead jumps straight to a known session id (the MCP tool, the CLI,
+ the podcli cloud resolver) went through plain `MulticamSession.load` and
+ so never noticed a camera file that changed or was replaced on disk.
+ """
+ session = MulticamSession.load(session_id)
+ backfilled = backfill_file_identity(session)
+ refreshed = refresh_stale_sources(session)
+ if backfilled or refreshed:
+ session.save()
+ return session
+
+
def find_session(found: list[str]) -> Optional[MulticamSession]:
"""The saved edit for exactly these files, including ones that couldn't be read."""
wanted = set(found)
@@ -614,6 +805,54 @@ def find_session(found: list[str]) -> Optional[MulticamSession]:
return None
+def session_for_render(video_path: str) -> Optional[MulticamSession]:
+ """The saved edit whose rendered episode is this file, if any."""
+ target = os.path.realpath(video_path)
+ for p in sorted(_sessions_dir().glob("*.json"), key=lambda x: x.stat().st_mtime, reverse=True):
+ try:
+ session = MulticamSession.load(p.stem)
+ except (ValueError, OSError, TypeError, json.JSONDecodeError):
+ continue
+ video = session.outputs.get("video")
+ if video and os.path.realpath(video) == target:
+ return session
+ return None
+
+
+def render_to_timeline(session: MulticamSession, seconds: float) -> Optional[float]:
+ """The timeline second shown at this second of the rendered episode, or None past its end.
+
+ The render starts at the first cut and leaves the removed stretches out.
+ """
+ if not session.cuts or seconds < 0:
+ return None
+ left = seconds
+ for a, b in kept_segments(session):
+ if left < b - a:
+ return a + left
+ left -= b - a
+ return None
+
+
+def person_still(session: MulticamSession, person_id: str, timeline_seconds: float, out: Path,
+ width: int = 1280) -> Optional[dict]:
+ """A still from this person's own camera at a timeline second.
+
+ Returns {"path", "camera", "camera_time"}, or None when no camera of theirs
+ is rolling then. A split screen is skipped: it shows everyone at once.
+ """
+ for cam in session.cameras():
+ if cam.person != person_id or cam.members:
+ continue
+ src = session.source(cam.parent) if cam.parent else cam
+ if not (src.synced and src.timeline_start() <= timeline_seconds < src.timeline_end()):
+ continue
+ _still(session, cam, timeline_seconds, out, width=width)
+ return {"path": str(out), "camera": src.path,
+ "camera_time": round(src.source_time(timeline_seconds), 3)}
+ return None
+
+
def rename_people(session: MulticamSession, names: list[str]) -> MulticamSession:
"""Rename people in order, keeping ids so every file mapped to them stays mapped."""
ids = [p.id for p in session.people]
@@ -636,7 +875,15 @@ def new_session(
# caller makes after, under the same lock as any other.
existing = find_session(found)
if existing:
- _emit(progress_callback, 100, f"Reopened the edit for these {len(found)} files")
+ backfilled = backfill_file_identity(existing)
+ refreshed = refresh_stale_sources(existing)
+ if backfilled or refreshed:
+ existing.save()
+ if refreshed:
+ _emit(progress_callback, 100, f"Reopened the edit for these {len(found)} files; "
+ "one or more changed on disk and need syncing again")
+ else:
+ _emit(progress_callback, 100, f"Reopened the edit for these {len(found)} files")
return existing
named = [(str(p.get("name", "")).strip(), p.get("role")) if isinstance(p, dict) else (str(p or "").strip(), None)
for p in (people or [])]
@@ -754,6 +1001,19 @@ def needs_sync(session: MulticamSession) -> bool:
return any(_in_use(session, s) and not s.virtual and not s.synced for s in session.sources)
+def sync_basis_signature(session: MulticamSession) -> str:
+ """Fingerprint of where every in-use source currently sits on the timeline.
+
+ A cut or removal made in an external editor (the podcli cloud editor)
+ references timeline seconds measured against this placement. If sync,
+ a nudge, or a re-map moves anything after the edit was sent out, applying
+ that cut back silently lands on the wrong footage: this lets callers
+ detect that before it happens.
+ """
+ feeds = sorted((s.id, s.offset, s.speed) for s in session.sources if _in_use(session, s) and not s.virtual)
+ return hashlib.sha1(json.dumps(feeds, default=str).encode()).hexdigest()[:16]
+
+
def _in_use(session: MulticamSession, s: Source) -> bool:
"""Mapped to something, or the recording a tile in use is cut from (even when the whole frame is ignored)."""
return s.role != "ignore" or any(v.parent == s.id and v.role != "ignore" for v in session.sources)
@@ -828,8 +1088,28 @@ def update_mapping(session: MulticamSession, params: dict) -> MulticamSession:
if "channel_people" in edit:
chans = [c if c in valid else "" for c in (edit["channel_people"] or [])]
s.channel_people = chans[: max(0, s.audio_channels)] if any(chans) else []
- if (("offset" in edit and edit["offset"] is not None) or "nudge" in edit) and s.virtual:
+ if "audio_stream_index" in edit:
+ idx = edit["audio_stream_index"]
+ if not isinstance(idx, int) or not 0 <= idx < max(1, s.audio_stream_count):
+ raise ValueError(f"{os.path.basename(s.path)} has {s.audio_stream_count} audio "
+ f"stream(s); audio_stream_index must be between 0 and {s.audio_stream_count - 1}")
+ s.audio_stream_index = idx
+ if idx < len(s.audio_stream_channels):
+ s.audio_channels = s.audio_stream_channels[idx]
+ s.channel_people = []
+ if "input_lut" in edit:
+ if s.kind != "video" or s.virtual:
+ raise ValueError("Only a camera file takes an input LUT; tiles and split screens use their files' LUTs")
+ s.input_lut = _check_cube(edit["input_lut"]) if edit["input_lut"] else ""
+ if (("offset" in edit and edit["offset"] is not None) or "nudge" in edit or "anchors" in edit) and s.virtual:
raise ValueError("A tile or split screen moves with the files it comes from. Move those instead.")
+ if "anchors" in edit:
+ pairs, fit = _anchor_fit(s, edit["anchors"])
+ s.offset, s.speed = fit.offset, fit.speed
+ s.sync = {
+ "status": "manual", "method": "manual", "anchors": pairs,
+ "residual_ms": round(fit.residual_ms, 1), "drift_ppm": round((fit.speed - 1.0) * 1e6, 1),
+ }
if "offset" in edit and edit["offset"] is not None:
s.offset = _number(edit["offset"], "offset", -MAX_SECONDS, MAX_SECONDS)
s.sync = {**s.sync, "status": "manual"}
@@ -882,9 +1162,77 @@ def update_mapping(session: MulticamSession, params: dict) -> MulticamSession:
return session
+MAX_LUT_BYTES = 64 * 1024 * 1024
+
+
+def _check_cube(path) -> str:
+ """The path of a well-formed 3D .cube LUT, or a ValueError saying what's wrong with it."""
+ if not isinstance(path, str) or not os.path.isabs(os.path.expanduser(path)):
+ raise ValueError("input_lut must be an absolute path to a .cube file")
+ p = Path(path).expanduser()
+ if p.suffix.lower() != ".cube" or not p.is_file():
+ raise ValueError(f"{path} isn't a .cube file")
+ if p.stat().st_size > MAX_LUT_BYTES:
+ raise ValueError(f"{p.name} is too big to be a LUT")
+ size, entries = 0, 0
+ for line in p.read_text(encoding="utf-8", errors="replace").splitlines():
+ words = line.split()
+ if not words or words[0].startswith("#"):
+ continue
+ key = words[0].upper()
+ if key == "LUT_1D_SIZE":
+ raise ValueError(f"{p.name} is a 1D LUT. A camera needs a 3D LUT (LUT_3D_SIZE).")
+ if key == "LUT_3D_SIZE":
+ size = int(_number(words[1] if len(words) > 1 else "", "LUT_3D_SIZE", 2, 256))
+ elif key[0].isdigit() or key[0] in "+-.":
+ if len(words) != 3:
+ raise ValueError(f"{p.name} has a LUT entry without exactly three values: {line.strip()[:40]}")
+ for w in words:
+ _number(w, f"a value in {p.name}", -1e6, 1e6)
+ entries += 1
+ if not size:
+ raise ValueError(f"{p.name} has no LUT_3D_SIZE, so it isn't a 3D LUT")
+ if entries != size ** 3:
+ raise ValueError(f"{p.name} declares a {size}-point cube ({size ** 3} entries) but holds {entries}")
+ return str(p)
+
+
+MAX_ANCHORS = 100
+
+
+def _anchor_fit(s: Source, anchors) -> tuple[list[dict], sig.ClockFit]:
+ """Offset and speed through hand-picked {timeline, source} second pairs, by least squares.
+
+ One pair sets the offset alone. More fit drift too, within the bound an
+ audio sync accepts. Every pair counts: unlike sync checkpoints, nobody
+ picks an anchor by accident, so none is dropped as an outlier.
+ """
+ name = os.path.basename(s.path)
+ if not isinstance(anchors, list) or not 1 <= len(anchors) <= MAX_ANCHORS:
+ raise ValueError(f"anchors must be a list of 1 to {MAX_ANCHORS} {{timeline, source}} pairs in seconds")
+ pairs = []
+ for a in anchors:
+ if not isinstance(a, dict) or not {"timeline", "source"} <= a.keys():
+ raise ValueError("Each anchor needs timeline and source, in seconds")
+ pairs.append({"timeline": _number(a["timeline"], "anchor timeline", -MAX_SECONDS, MAX_SECONDS),
+ "source": _number(a["source"], f"anchor source second in {name}", 0, s.duration)})
+ pairs.sort(key=lambda p: p["source"])
+ for a, b in zip(pairs, pairs[1:]):
+ if b["source"] - a["source"] < 1e-3 or b["timeline"] <= a["timeline"]:
+ raise ValueError("Anchors must move forward together: a later second in the file needs a later "
+ "second on the timeline, and no two anchors may share a file second")
+ fit = sig.fit_clock([(p["source"], p["timeline"], 1.0) for p in pairs], max_residual=math.inf)
+ if fit is None or fit.speed_fallback:
+ first, last = pairs[0], pairs[-1]
+ ppm = ((last["timeline"] - first["timeline"]) / (last["source"] - first["source"]) - 1.0) * 1e6
+ raise ValueError(f"Those anchors put {name}'s clock {ppm:+.0f} ppm off the timeline; real drift stays within "
+ f"{sig.MAX_DRIFT * 1e6:.0f} ppm. Check each pair.")
+ return pairs, fit
+
+
def _cut_inputs(session: MulticamSession) -> str:
return json.dumps([
- [(s.id, s.role, s.person, s.channel_people, s.offset) for s in session.sources],
+ [(s.id, s.role, s.person, s.channel_people, s.audio_stream_index, s.offset, s.speed) for s in session.sources],
[(p.id, p.role) for p in session.people], session.range_start, session.range_end,
session.cut_settings, session.speaker_map,
], default=str)
@@ -895,7 +1243,11 @@ def _cut_inputs(session: MulticamSession) -> str:
# ---------------------------------------------------------------------------
def _extract(source: Source, out: Path, channel: int = -1, rate: int = SYNC_RATE) -> Path:
- if out.exists() and out.stat().st_mtime >= os.path.getmtime(source.path):
+ # Keyed on the probed file identity, not a live mtime check: a replaced
+ # file can land within the same mtime second, and the id in `out`'s name
+ # (basename:size) doesn't change either, so neither alone is reliable.
+ out = out.with_name(f"{out.stem}-{_file_identity(source)}{out.suffix}")
+ if out.exists():
return out
tmp = out.with_suffix(".tmp.wav")
pick = "pan=mono|c0=c0" if source.audio_channels < 2 else "pan=mono|c0=0.5*c0+0.5*c1"
@@ -905,7 +1257,7 @@ def _extract(source: Source, out: Path, channel: int = -1, rate: int = SYNC_RATE
# clock the render and editors use; audio often starts 20-100 ms after video.
proc_run([
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
- "-i", source.path, "-vn", "-map", "0:a:0",
+ "-i", source.path, "-vn", "-map", f"0:a:{source.audio_stream_index}",
"-af", f"aresample=async=1:first_pts=0,{pick}",
"-ar", str(rate), "-acodec", "pcm_s16le", str(tmp),
], timeout=3600, check=True)
@@ -981,13 +1333,47 @@ def _match(ref: np.ndarray, ref_env: np.ndarray, src: np.ndarray) -> tuple[Optio
"status": "rough", "score": round(match.score, 1),
"message": "Synced to within 10 ms; fine alignment found no clear speech.",
}
- return fit, {
- "status": "ok",
+ overlap_seconds = min(len(ref), len(src)) / SYNC_RATE
+ reasons = _sync_review_reasons(fit, match, overlap_seconds)
+ report = {
+ "status": "review" if reasons else "ok",
"score": round(match.score, 1),
"checkpoints": fit.checkpoints,
"residual_ms": round(fit.residual_ms, 1),
+ "residual_all_ms": round(fit.residual_all_ms, 1),
"drift_ppm": round((fit.speed - 1.0) * 1e6, 1),
}
+ if reasons:
+ report["reasons"] = reasons
+ report["message"] = "Sync may be off: " + "; ".join(reasons) + ". Check it before rendering."
+ return fit, report
+
+
+def _sync_review_reasons(fit: sig.ClockFit, match: sig.CoarseMatch, overlap_seconds: float) -> list[str]:
+ """Why a fit that otherwise looks like a clean sync should get a second look.
+
+ A fit can report a tidy inlier residual while still being wrong: outliers
+ get dropped before the residual is measured, drift can be forced back to
+ speed 1 when it looked implausible, or there just weren't enough
+ checkpoints to trust a long overlap's drift estimate.
+ """
+ reasons = []
+ # residual_all_ms includes the checkpoints the fit already rejected as
+ # outliers, so one bad checkpoint correctly thrown out would otherwise
+ # flag an inlier fit that's actually clean. Judge the fit on the
+ # residual it was actually built from; residual_all_ms is still reported.
+ if fit.residual_ms > 30.0:
+ reasons.append(f"residual {fit.residual_ms:.0f} ms")
+ if fit.speed_fallback:
+ reasons.append("drift fit was implausible, so speed was forced back to 1.0")
+ dropped = fit.total_checkpoints - fit.checkpoints
+ if fit.total_checkpoints and dropped / fit.total_checkpoints > 0.4:
+ reasons.append(f"dropped {dropped} of {fit.total_checkpoints} checkpoints as outliers")
+ if overlap_seconds > 600.0 and fit.checkpoints < 3:
+ reasons.append("fewer than 3 checkpoints over a 10+ minute overlap")
+ if match.peak_ratio > 0.5:
+ reasons.append("weak correlation peak")
+ return reasons
def sync_session(
@@ -1100,7 +1486,8 @@ def sync_session(
# ---------------------------------------------------------------------------
def _activity_key(session: MulticamSession) -> str:
- feeds = [(s.id, ch, pid, s.offset, s.speed) for s, ch, pid in session.person_mics()]
+ feeds = [(s.id, _file_identity(s), s.audio_stream_index, ch, pid, s.offset, s.speed)
+ for s, ch, pid in session.person_mics()]
blob = json.dumps([feeds, session.speaker_map, session.person_ids()], sort_keys=True, default=str)
return hashlib.sha1(blob.encode()).hexdigest()[:16]
@@ -1310,11 +1697,13 @@ def activity(session: MulticamSession) -> dict:
# Previews
# ---------------------------------------------------------------------------
-def _picture(session: MulticamSession, cam: Source, tl: float, width: int, height: int) -> tuple[list[str], str]:
- """ffmpeg inputs and a filter graph drawing `cam` at timeline second tl into [pic], width x height.
+def _picture(session: MulticamSession, cam: Source, tl: float, width: int, height: int,
+ span: float = 0.0) -> tuple[list[str], str]:
+ """ffmpeg inputs and a filter graph drawing `cam` from timeline second tl into [pic], width x height.
A real camera is fitted to the frame; a pane is cropped out of its call
- recording first; a split screen fills one equal column per person.
+ recording first; a split screen fills one equal column per person. `span`
+ is how long the piece plays, so its middle lands in sync on a drifting file.
"""
fit = f"scale={width}:{height}:force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,setsar=1"
if cam.members:
@@ -1326,10 +1715,12 @@ def _picture(session: MulticamSession, cam: Source, tl: float, width: int, heigh
rolling = next((c for c in session.cameras() if not c.virtual and c.synced and c.person == person
and c.timeline_start() <= tl < c.timeline_end()), None)
if rolling:
- args += ["-ss", f"{max(0.0, rolling.source_time(tl)):.4f}", "-i", rolling.path]
+ args += ["-ss", f"{max(0.0, rolling.source_in(tl, span)):.4f}", "-i", rolling.path]
else:
args += ["-f", "lavfi", "-i", f"color=c=black:s={pane}x{height}"]
- chains.append(f"[{i}:v:0]scale={pane}:{height}:force_original_aspect_ratio=increase,crop={pane}:{height},setsar=1[p{i}]")
+ lut = _lut_step(rolling) if rolling else ""
+ chains.append(f"[{i}:v:0]{lut}scale={pane}:{height}:force_original_aspect_ratio=increase,"
+ f"crop={pane}:{height},setsar=1[p{i}]")
joined = "".join(f"[p{i}]" for i in range(n))
return args, ";".join(chains) + f";{joined}hstack=inputs={n},pad={width}:{height}:(ow-iw)/2:(oh-ih)/2[pic]"
src = session.source(cam.parent) if cam.parent else cam
@@ -1337,7 +1728,39 @@ def _picture(session: MulticamSession, cam: Source, tl: float, width: int, heigh
if cam.crop:
x, y, w, h = cam.crop
crop = f"crop=iw*{w:.4f}:ih*{h:.4f}:iw*{x:.4f}:ih*{y:.4f},"
- return ["-ss", f"{max(0.0, src.source_time(tl)):.4f}", "-i", src.path], f"[0:v:0]{crop}{fit}[pic]"
+ return ["-ss", f"{max(0.0, src.source_in(tl, span)):.4f}", "-i", src.path], f"[0:v:0]{crop}{_lut_step(src)}{fit}[pic]"
+
+
+def _filter_path(path: str) -> str:
+ """A file path escaped for an option value inside an ffmpeg filter graph.
+
+ Two parsers read it: the graph splits on , ; [ ] and the filter splits its
+ options on :, and each strips one level of backslashes.
+ """
+ option = path.replace("\\", "\\\\").replace(":", "\\:").replace("'", "\\'")
+ return "".join(f"\\{ch}" if ch in "\\'[],;" else ch for ch in option)
+
+
+def _lut_step(s: Source) -> str:
+ """The file's input LUT as a filter step ending in a comma, or ''.
+
+ It runs before scaling and padding, so it maps the camera's own pixels and never tints the bars.
+ """
+ return f"lut3d=file={_filter_path(s.input_lut)}," if s.input_lut else ""
+
+
+def _check_luts_exist(session: MulticamSession) -> None:
+ """Fail now, naming the camera, rather than deep inside an ffmpeg worker later.
+
+ A LUT set at map time can get moved or deleted on disk by the time a
+ render or a preview actually reads it; without this, ffmpeg's own error
+ surfaces from inside a shot or a still with no indication of which
+ camera (or which of several in a split) it was for.
+ """
+ missing = [s for s in session.sources if s.input_lut and not os.path.isfile(s.input_lut)]
+ if missing:
+ names = ", ".join(f"{source_label(session, s)} ({s.input_lut})" for s in missing)
+ raise ValueError(f"The LUT set for {names} is missing. Pick it again or clear it.")
def _still(session: MulticamSession, cam: Source, tl: float, out: Path, look: str = "none", width: int = 480) -> Path:
@@ -1350,8 +1773,23 @@ def _still(session: MulticamSession, cam: Source, tl: float, out: Path, look: st
return out
+def _still_basis(session: MulticamSession, s: Source) -> str:
+ """Short fingerprint of the offset, speed and input LUT of every file a still reads.
+
+ Stills are cached to disk by filename. The source's mapping from timeline
+ time to source time (`source_time`) depends on offset and speed, so a
+ cache key that only captures the timeline moment `at` goes stale the
+ instant a nudge or re-sync changes that mapping: the filename looks the
+ same but would now decode a different source frame. A new LUT recolors it.
+ """
+ files = _picture_files(session, s)
+ raw = json.dumps([(f.offset, f.speed, _fingerprint(f.input_lut) if f.input_lut else None) for f in files])
+ return hashlib.sha1(raw.encode()).hexdigest()[:8]
+
+
def previews(session: MulticamSession, *, looks: bool = False, at: Optional[float] = None) -> dict:
- """One still per camera, and optionally one still per look from the wide camera."""
+ """One still per camera, and optionally one still per look per camera, each through its input LUT."""
+ _check_luts_exist(session)
work = _work_dir(session.session_id)
frames: dict = {"cameras": {}, "looks": {}}
@@ -1364,19 +1802,34 @@ def previews(session: MulticamSession, *, looks: bool = False, at: Optional[floa
if s.kind != "video":
continue
t = moment(s)
- out = work / f"frame-{s.id}-{int(t * 10)}.jpg"
+ out = work / f"frame-{s.id}-{int(t * 10)}-{_still_basis(session, s)}.jpg"
if not out.exists():
+ # A nudge or re-sync changes the basis (and often the moment), so the
+ # old filename never matches again; delete it rather than let every
+ # edit leave one more still behind.
+ for stale in work.glob(f"frame-{s.id}-*.jpg"):
+ if stale != out:
+ stale.unlink(missing_ok=True)
_still(session, s, t, out)
frames["cameras"][s.id] = str(out)
if looks:
- cams = session.cameras() or [s for s in session.sources if s.kind == "video"]
- if cams:
- cam = next((c for c in cams if c.person == "wide"), cams[0])
+ # Cameras rarely match out of the box, so each one shows every look on its own picture.
+ candidates = session.cameras() or [s for s in session.sources if s.kind == "video"]
+ # Once a cut exists, a camera that never actually appears in it (an
+ # alternate angle left mapped but unused) doesn't need four look
+ # renders; before any cut exists there's nothing to filter by yet.
+ used = {c["source_id"] for c in session.cuts}
+ for cam in [c for c in candidates if not used or c.id in used]:
+ t = moment(cam)
+ frames["looks"][cam.id] = {}
for name in LOOKS:
- out = work / f"look-{cam.id}-{name}.jpg"
+ out = work / f"look-{cam.id}-{name}-{int(t * 10)}-{_still_basis(session, cam)}.jpg"
if not out.exists():
- _still(session, cam, moment(cam), out, look=name, width=640)
- frames["looks"][name] = str(out)
+ for stale in work.glob(f"look-{cam.id}-{name}-*.jpg"):
+ if stale != out:
+ stale.unlink(missing_ok=True)
+ _still(session, cam, t, out, look=name, width=640)
+ frames["looks"][cam.id][name] = str(out)
return frames
@@ -1488,6 +1941,29 @@ def transcript(session: MulticamSession, *, model_size: str = "base", engine: Op
_SENTENCE_END = (".", "?", "!")
+_GEORGIAN_RANGES = ((0x10A0, 0x10FF), (0x1C90, 0x1CBF))
+
+
+def _looks_like_sentence_opener(text: str) -> bool:
+ """True if `text` could start a new sentence, by case or by having none.
+
+ `str.isupper()` alone misses caseless scripts: CJK, Arabic, Thai, and
+ Hebrew letters are never upper or lower (`ch.upper() == ch.lower()`
+ catches those). Georgian is a special case: Unicode still carries a
+ Mtavruli uppercase mapping for it, so `str.isupper()`/`islower()` report
+ it as cased, but real Georgian text is written only in the lowercase
+ Mkhedruli form and never uses that case distinction, so every opener was
+ silently dropped. Treat Georgian letters as potential openers too.
+ """
+ ch = text[:1]
+ if not ch.isalpha():
+ return False
+ if ch.isupper() or ch.upper() == ch.lower():
+ return True
+ cp = ord(ch)
+ return any(lo <= cp <= hi for lo, hi in _GEORGIAN_RANGES)
+
+
def _settle_turn_edges(words: list[dict], margins: list[float], clear: float = 6.0) -> None:
"""Fix credits that loose word timestamps get wrong around a turn change.
@@ -1508,7 +1984,7 @@ def _settle_turn_edges(words: list[dict], margins: list[float], clear: float = 6
for i in range(1, len(words) - 1):
prev, word, nxt = words[i - 1], words[i], words[i + 1]
if (margins[i] < clear and word["person"] == prev["person"] != nxt["person"]
- and prev["text"].endswith(_SENTENCE_END) and word["text"][:1].isupper()):
+ and prev["text"].endswith(_SENTENCE_END) and _looks_like_sentence_opener(word["text"])):
word["person"] = nxt["person"]
# Moving a sentence opener can leave the word after it stranded; rejoin it too.
rejoin_strays()
@@ -1559,32 +2035,153 @@ def _output_dir(session: MulticamSession) -> Path:
return d
-def _render_shot(session: MulticamSession, cam: Optional[Source], out: Path, tl_start: float, frames: int,
- width: int, height: int, fps: float, look: str) -> None:
- """Encode exactly `frames` frames starting at timeline second tl_start.
+# Bump when the shot pipeline changes in a way its cached files can't show,
+# so a rerun re-encodes instead of reusing shots built the old way.
+RENDER_VERSION = 1
+SHOT_ENCODE = ("-c:v", "libx264", "-crf", "18", "-preset", "fast", "-pix_fmt", "yuv420p",
+ "-video_track_timescale", "90000")
+
+
+def _shot_command(session: MulticamSession, cam: Optional[Source], tl_start: float, frames: int,
+ width: int, height: int, fps: float, look: str) -> list[str]:
+ """The ffmpeg command, minus its output path, that encodes exactly `frames` frames from timeline second tl_start.
cam None renders black, which keeps picture and sound aligned across a
stretch no camera covered.
"""
- common = [
- "-frames:v", str(frames), "-an",
- "-c:v", "libx264", "-crf", "18", "-preset", "fast", "-pix_fmt", "yuv420p",
- "-video_track_timescale", "90000", str(out),
- ]
+ common = ["-frames:v", str(frames), "-an", *SHOT_ENCODE]
if cam is None:
- proc_run([
- "ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
- "-f", "lavfi", "-i", f"color=c=black:s={width}x{height}:r={fps:.6f}", *common,
- ], timeout=600, check=True)
- return
- args, graph = _picture(session, cam, tl_start, width, height)
+ return ["ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", f"color=c=black:s={width}x{height}:r={fps:.6f}", *common]
+ args, graph = _picture(session, cam, tl_start, width, height, frames / fps)
# A camera that stops a few frames early would shorten the shot and slip
# every later shot against the audio; holding its last frame prevents that.
tail = [f"fps={fps:.6f}", "tpad=stop_mode=clone:stop=-1", *([LOOKS[look]] if LOOKS.get(look) else [])]
- proc_run([
- "ffmpeg", "-y", "-hide_banner", "-loglevel", "error", *args,
- "-filter_complex", f"{graph};[pic]{','.join(tail)}[v]", "-map", "[v]", *common,
- ], timeout=3600, check=True)
+ return ["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", *args,
+ "-filter_complex", f"{graph};[pic]{','.join(tail)}[v]", "-map", "[v]", *common]
+
+
+def _picture_files(session: MulticamSession, cam: Source) -> list[Source]:
+ """The recordings a camera's picture is read from: itself, its call recording, or each split member's files."""
+ if cam.parent:
+ return [session.source(cam.parent)]
+ if cam.members:
+ return [c for c in session.cameras() if not c.virtual and c.person in cam.members]
+ return [cam]
+
+
+def _shot_key(session: MulticamSession, cam: Optional[Source], command: list[str]) -> str:
+ """Names a cached shot by everything that decides its pixels.
+
+ The command carries the seek point, frame range, crop, look and encoder
+ settings; the files it reads are fingerprinted live so a re-export under
+ the same name can't serve an old shot.
+ """
+ files = [(_fingerprint(f.path), f.offset, f.speed, _fingerprint(f.input_lut) if f.input_lut else None)
+ for f in (_picture_files(session, cam) if cam else [])]
+ blob = json.dumps([RENDER_VERSION, command, files], default=str)
+ return hashlib.sha1(blob.encode()).hexdigest()[:20]
+
+
+def _fingerprint(path: str) -> tuple:
+ """A file's path, size and mtime as they are on disk right now."""
+ try:
+ st = os.stat(path)
+ return path, st.st_size, st.st_mtime_ns
+ except OSError:
+ return path, None, None
+
+
+def _probe_value(path: Path, *args: str) -> str:
+ res = proc_run(["ffprobe", "-v", "error", *args, "-of", "csv=p=0", str(path)], timeout=1800, check=False)
+ lines = res.stdout.strip().splitlines() if res.returncode == 0 else []
+ return lines[0].strip().rstrip(",") if lines else ""
+
+
+def _frame_count(path: Path) -> int:
+ """Video frames actually in a file, counted packet by packet rather than trusting the header."""
+ value = _probe_value(path, "-select_streams", "v:0", "-count_packets", "-show_entries", "stream=nb_read_packets")
+ return int(value) if value.isdigit() else -1
+
+
+def _media_duration(path: Path, stream: str = "") -> float:
+ value = _probe_value(path, *(["-select_streams", stream, "-show_entries", "stream=duration"] if stream
+ else ["-show_entries", "format=duration"]))
+ try:
+ return float(value)
+ except ValueError:
+ return 0.0
+
+
+LOUDNESS_TARGET = -16.0
+TRUE_PEAK_CEILING = -1.5
+
+
+def _loudness(path: Path) -> tuple[Optional[float], Optional[float]]:
+ """Integrated loudness (LUFS) and true peak (dBTP) of a file's first audio stream, by EBU R128."""
+ res = proc_run([
+ "ffmpeg", "-hide_banner", "-nostats", "-i", str(path), "-map", "0:a:0",
+ "-af", "ebur128=peak=true", "-f", "null", "-",
+ ], timeout=3600, check=False)
+ summary = res.stderr[res.stderr.rfind("Summary:"):] if "Summary:" in res.stderr else ""
+ lufs = re.search(r"I:\s+(-?[\d.]+|-inf) LUFS", summary)
+ peak = re.search(r"True peak:\s+Peak:\s+(-?[\d.]+|-inf) dBFS", summary)
+
+ def number(m) -> Optional[float]:
+ return float(m.group(1)) if m and m.group(1) != "-inf" else None
+
+ return number(lufs), number(peak)
+
+
+def _decode_errors(path: Path, *, full: bool = False) -> str:
+ """Whatever ffmpeg complains about while decoding the episode, or '' when clean.
+
+ A full decode is exhaustive but costs minutes per hour of 1080p on every
+ single render, almost all of it spent re-confirming frames nothing
+ touched. By default this instead decodes the first and last 10 s plus
+ three points spread through the middle: enough to catch what a render
+ is actually at risk of (a bad concat seam between shots, a broken mux,
+ a codec fault at the point it happened) without paying for the whole
+ file. Pass full=True (render_session(..., validate="full")) for the
+ exhaustive check when that trade-off isn't acceptable.
+ """
+ if full:
+ res = proc_run(["ffmpeg", "-hide_banner", "-v", "error", "-i", str(path), "-f", "null", "-"],
+ timeout=7200, check=False)
+ return res.stderr.strip() or (f"ffmpeg exited with {res.returncode}" if res.returncode else "")
+ duration = _media_duration(path) or 0.0
+ spans = [0.0]
+ if duration > 10.0:
+ spans.append(max(0.0, duration - 10.0))
+ for frac in (0.25, 0.5, 0.75):
+ t = duration * frac
+ if all(abs(t - s) > 10.0 for s in spans):
+ spans.append(t)
+ errors = []
+ for t in spans:
+ res = proc_run(["ffmpeg", "-hide_banner", "-v", "error", "-ss", f"{t:.3f}", "-i", str(path),
+ "-t", "10", "-f", "null", "-"], timeout=600, check=False)
+ err = res.stderr.strip() or (f"ffmpeg exited with {res.returncode}" if res.returncode else "")
+ if err:
+ errors.append(err)
+ return "; ".join(errors)
+
+
+def _ran_out(session: MulticamSession, cam: Source, piece: "Piece", fps: float, ends: dict[str, float]) -> list[str]:
+ """Warnings for files that stop before a piece does, so tpad holds their last frame."""
+ out = []
+ span = piece.frames / fps
+ files = _picture_files(session, cam)
+ if cam.members:
+ files = [f for f in files if f.timeline_start() <= piece.tl_start < f.timeline_end()]
+ for f in files:
+ if f.path not in ends:
+ ends[f.path] = _media_duration(Path(f.path), "v:0") or f.duration
+ short = f.source_in(piece.tl_start, span) + span - ends[f.path]
+ if short > 0.5 / fps:
+ out.append(f"{os.path.basename(f.path)} ran out {short:.2f} s before the end of the shot at "
+ f"{piece.tl_start:.1f} s; its last frame is held.")
+ return out
def _audio_inputs(session: MulticamSession) -> list[tuple[Source, int]]:
@@ -1594,13 +2191,29 @@ def _audio_inputs(session: MulticamSession) -> list[tuple[Source, int]]:
rooms = [s for s in session.sources if s.role == "mic" and s.synced]
if rooms:
return [(max(rooms, key=lambda s: s.duration), -1)]
+ # The reference is only chosen by the first sync, so a fresh edit has no mix yet.
+ if not session.reference_id:
+ return []
return [(session.source(session.reference_id), -1)]
+MIX_RATE = 48000
+# asetrate only takes whole rates, so at 48 kHz the nearest one can be 10 ppm
+# off the drift; labelling the audio 1000 times faster first lets one resample
+# step land within 0.01 ppm, and the resample itself stays near 1:1.
+DRIFT_LABEL_SCALE = 1000
+
+
+def _drift_steps(speed: float) -> list[str]:
+ """Filter steps stretching MIX_RATE audio by `speed` through a resample, keeping its length exact."""
+ fast = MIX_RATE * DRIFT_LABEL_SCALE
+ return [f"asetrate={fast}", f"aresample={round(fast * speed)}", f"asetrate={MIX_RATE}"]
+
+
def _aligned_input(s: Source, channel: int, start: float, duration: float) -> tuple[list[str], list[str]]:
- """ffmpeg input args and mono filter steps that place a source on [start, start + duration]."""
+ """ffmpeg input args and mono MIX_RATE filter steps that place a source on [start, start + duration]."""
src_start = s.source_time(start)
- steps = ["aresample=async=1:first_pts=0"]
+ steps = [f"aresample={MIX_RATE}:async=1:first_pts=0"]
if channel >= 0:
steps.append(f"pan=mono|c0=c{channel}")
elif s.audio_channels >= 2:
@@ -1608,11 +2221,11 @@ def _aligned_input(s: Source, channel: int, start: float, duration: float) -> tu
else:
steps.append("pan=mono|c0=c0")
if abs(s.speed - 1.0) > 1e-7:
- steps.append(f"atempo={1.0 / s.speed:.8f}")
+ steps += _drift_steps(s.speed)
if src_start < 0:
- steps.append(f"adelay={int(round(-src_start * 1000))}")
- steps.append(f"apad,atrim=0:{duration:.4f}")
- return ["-ss", f"{max(0.0, src_start):.4f}", "-i", s.path], steps
+ steps.append(f"adelay={round(-src_start * s.speed * MIX_RATE)}S")
+ steps.append(f"apad,atrim=0:{duration:.6f}")
+ return ["-ss", f"{max(0.0, src_start):.6f}", "-i", s.path], steps
def _write_mix(session: MulticamSession, out: Path, start: float, duration: float, *,
@@ -1624,7 +2237,7 @@ def _write_mix(session: MulticamSession, out: Path, start: float, duration: floa
for i, (s, ch) in enumerate(feeds if feeds is not None else _audio_inputs(session)):
inp, steps = _aligned_input(s, ch, start, duration)
args += inp
- chains.append(f"[{i}:a:0]{','.join([*steps, *per_feed])}[a{i}]")
+ chains.append(f"[{i}:a:{s.audio_stream_index}]{','.join([*steps, *per_feed])}[a{i}]")
n = len(chains)
mix = "".join(f"[a{i}]" for i in range(n)) + f"amix=inputs={n}:normalize=0," if n > 1 else "[a0]"
proc_run([
@@ -1635,24 +2248,73 @@ def _write_mix(session: MulticamSession, out: Path, start: float, duration: floa
def _mix_key(session: MulticamSession) -> str:
"""Changes whenever which mics are mixed, or where they sit, changes."""
- feeds = [(s.id, ch, s.offset, s.speed, os.path.getmtime(s.path)) for s, ch in _audio_inputs(session)]
+ feeds = [(s.id, s.audio_stream_index, ch, s.offset, s.speed, os.path.getmtime(s.path))
+ for s, ch in _audio_inputs(session)]
return hashlib.sha1(json.dumps(feeds, default=str).encode()).hexdigest()[:12]
-def _render_audio(session: MulticamSession, out: Path, start: float, duration: float) -> None:
- _write_mix(
- session, out, start, duration,
- per_feed=("dynaudnorm=f=250:g=15:p=0.9",),
- after="loudnorm=I=-16:TP=-1.5:LRA=11,aformat=channel_layouts=stereo",
- encode=("-ar", "48000", "-c:a", "pcm_s16le"),
- )
+# Rumble and handling noise sit under 70 Hz; a gentle high-pass takes them
+# out before leveling would lift them along with the voice.
+VOICE_CHAIN = ("highpass=f=70:poles=2", "dynaudnorm=f=250:g=15:p=0.9")
+
+
+def _normalization(path: Path) -> tuple[float, bool]:
+ """The linear gain (dB) that brings a mix to LOUDNESS_TARGET, and whether peaks then need limiting."""
+ from services.audio_normalize import _parse_loudnorm_stats
+
+ res = proc_run([
+ "ffmpeg", "-hide_banner", "-nostats", "-i", str(path),
+ "-af", f"loudnorm=I={LOUDNESS_TARGET}:TP={TRUE_PEAK_CEILING}:LRA=11:print_format=json", "-f", "null", "-",
+ ], timeout=3600, check=False)
+ stats = _parse_loudnorm_stats(res.stderr)
+ if not stats:
+ return 0.0, False
+ gain = LOUDNESS_TARGET - float(stats["input_i"])
+ return gain, float(stats["input_tp"]) + gain > TRUE_PEAK_CEILING
+
+
+def _render_audio(session: MulticamSession, out: Path, start: float, duration: float,
+ splice: Optional[list[tuple[float, float]]] = None) -> float:
+ """The episode mix: every mic high-passed and leveled, then one measured gain to LOUDNESS_TARGET.
+
+ Returns that gain in dB so the stems can carry it too. Only peaks the gain
+ would push past TRUE_PEAK_CEILING meet a limiter, and only on the mix.
+ """
+ premix = out.with_name(f"{out.stem}.premix.wav")
+ # Float keeps a sum of hot mics from clipping before the gain brings it down.
+ _write_mix(session, premix, start, duration, per_feed=VOICE_CHAIN, encode=("-c:a", "pcm_f32le"))
+ if splice:
+ # Measure loudness on what actually airs: a retake or a dead patch
+ # removed from the episode shouldn't pull the target gain around,
+ # whether it's silence (would ask for too much gain) or loud
+ # cross-talk no one hears in the final cut (would ask for too little).
+ kept = out.with_name(f"{out.stem}.premix-kept.wav")
+ _splice_audio(premix, kept, splice, ["-c:a", "pcm_f32le"])
+ premix.unlink(missing_ok=True)
+ premix = kept
+ gain, limit = _normalization(premix)
+ steps = [f"volume={gain:.3f}dB"]
+ if limit:
+ # alimiter reads sample peaks; half a dB under the ceiling leaves room for the peaks between samples.
+ ceiling = 10 ** ((TRUE_PEAK_CEILING - 0.5) / 20)
+ steps.append(f"alimiter=limit={ceiling:.4f}:level=0:latency=1")
+ proc_run([
+ "ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", str(premix),
+ "-af", ",".join([*steps, "aformat=channel_layouts=stereo"]), "-ar", str(MIX_RATE), "-c:a", "pcm_s16le", str(out),
+ ], timeout=3600, check=True)
+ premix.unlink(missing_ok=True)
+ return gain
def _render_stems(
session: MulticamSession, out_dir: Path, start: float, duration: float,
- splice: Optional[list[tuple[float, float]]] = None,
+ splice: Optional[list[tuple[float, float]]] = None, gain: float = 0.0,
) -> list[str]:
- """One WAV per person; a person recorded across several files (recorder splits) gets them mixed in."""
+ """One WAV per person, processed like the mix and carrying its gain, so the stems sum to it.
+
+ A person recorded across several files (recorder splits) gets them mixed
+ in. Float samples keep a stem the mix's limiter caught from clipping here.
+ """
stems = []
for pid in session.person_ids():
feeds = [(s, ch) for s, ch, who in session.person_mics() if who == pid and s.synced]
@@ -1662,18 +2324,20 @@ def _render_stems(
for i, (s, ch) in enumerate(feeds):
inp, steps = _aligned_input(s, ch, start, duration)
args += inp
- chains.append(f"[{i}:a:0]{','.join(steps)}[a{i}]")
+ chains.append(f"[{i}:a:{s.audio_stream_index}]{','.join([*steps, *VOICE_CHAIN])}[a{i}]")
mix = "".join(f"[a{i}]" for i in range(len(feeds)))
- graph = ";".join(chains) + (f";{mix}amix=inputs={len(feeds)}:normalize=0[out]" if len(feeds) > 1 else ";[a0]anull[out]")
+ level = f"volume={gain:.3f}dB"
+ graph = ";".join(chains) + (f";{mix}amix=inputs={len(feeds)}:normalize=0,{level}[out]" if len(feeds) > 1
+ else f";[a0]{level}[out]")
# Person ids are unique slugs, so two people never share a file name.
out = out_dir / f"{pid}.wav"
full = out.with_name(f"{pid}.full.wav") if splice else out
proc_run([
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error", *args,
- "-filter_complex", graph, "-map", "[out]", "-ar", "48000", "-c:a", "pcm_s24le", str(full),
+ "-filter_complex", graph, "-map", "[out]", "-ar", str(MIX_RATE), "-c:a", "pcm_f32le", str(full),
], timeout=3600, check=True)
if splice:
- _splice_audio(full, out, splice, ["-c:a", "pcm_s24le"])
+ _splice_audio(full, out, splice, ["-c:a", "pcm_f32le"])
full.unlink(missing_ok=True)
stems.append(str(out))
return stems
@@ -1710,6 +2374,22 @@ def kept_segments(session: MulticamSession) -> list[tuple[float, float]]:
return [(a, b) for a, b, r in _segments(session) if r is None]
+def drift_parts(seconds: float, drift: float, fps: float) -> int:
+ """How many pieces a stretch splits into so a clock off by `drift` (|speed - 1|) slips at most half a frame in each.
+
+ One frame of slack covers pieces landing a frame longer on the output grid.
+ """
+ return max(1, math.ceil(drift * (seconds + 1 / fps) * 2 * fps - 1e-9))
+
+
+def _camera_drift(session: MulticamSession, source_id: str) -> float:
+ """|speed - 1| of the worst-drifting file a camera draws its picture from."""
+ cam = next((s for s in session.sources if s.id == source_id), None)
+ if cam is None:
+ return 0.0
+ return max((abs(f.speed - 1.0) for f in _picture_files(session, cam)), default=0.0)
+
+
def render_plan(session: MulticamSession, fps: float, *, review: bool = False) -> list[Piece]:
"""Every shot, split at removals, laid end to end on one output frame grid.
@@ -1717,9 +2397,13 @@ def render_plan(session: MulticamSession, fps: float, *, review: bool = False) -
rounding drift against the audio; the MP4, the stems, and both editor
timelines are all built from this list so they agree frame for frame.
A review plan keeps the removed stretches in place, tagged with their removal.
+ A shot on a drifting camera is split into back-to-back pieces on that
+ camera, each short enough that the clock slips at most half a frame across
+ it; the cut list itself is unchanged.
"""
pieces: list[Piece] = []
out_t, out_f = 0.0, 0
+ drift = {c["source_id"]: _camera_drift(session, c["source_id"]) for c in session.cuts}
def emit(source_id: Optional[str], a: float, b: float, removal: Optional[dict]) -> None:
nonlocal out_t, out_f
@@ -1743,7 +2427,9 @@ def render_plan(session: MulticamSession, fps: float, *, review: bool = False) -
emit(None, t, c["start"], removal)
t = c["start"]
e = min(c["end"], b)
- emit(c["source_id"], t, e, removal)
+ parts = drift_parts(e - t, drift[c["source_id"]], fps)
+ for k in range(parts):
+ emit(c["source_id"], t + (e - t) * k / parts, t + (e - t) * (k + 1) / parts, removal)
t = e
if t < b:
emit(None, t, b, removal)
@@ -1818,9 +2504,14 @@ def set_removals(session: MulticamSession, removals: list) -> MulticamSession:
def _render_key(session: MulticamSession, stems: bool) -> str:
"""Everything the MP4 depends on, so an unchanged edit isn't rendered twice."""
used = {c["source_id"] for c in session.cuts} | {s.id for s, _ in _audio_inputs(session)}
- files = [(s.id, s.offset, s.speed, s.channel_people, s.person, os.path.getmtime(s.path))
+ files = [(s.id, s.offset, s.speed, s.channel_people, s.audio_stream_index, s.person, os.path.getmtime(s.path),
+ _fingerprint(s.input_lut) if s.input_lut else None)
for s in session.sources if s.id in used and os.path.exists(s.path)]
- blob = json.dumps([session.cuts, session.removals, session.look, stems, files], sort_keys=True, default=str)
+ # A tile or split screen takes its picture, and its LUT, from other files.
+ files += [(f.id, _fingerprint(f.input_lut)) for c in session.cameras() if c.id in used and c.virtual
+ for f in _picture_files(session, c) if f.input_lut]
+ audio = [VOICE_CHAIN, LOUDNESS_TARGET, TRUE_PEAK_CEILING]
+ blob = json.dumps([session.cuts, session.removals, session.look, stems, files, audio], sort_keys=True, default=str)
return hashlib.sha1(blob.encode()).hexdigest()[:16]
@@ -1828,13 +2519,19 @@ def render_session(
session: MulticamSession,
*,
stems: bool = True,
+ validate: str = "sample",
progress_callback: ProgressCallback = None,
) -> dict:
+ if validate not in ("sample", "full"):
+ raise ValueError("validate must be 'sample' or 'full'")
if not session.cuts:
raise ValueError("Plan the cuts before rendering")
+ _check_luts_exist(session)
key = _render_key(session, stems)
video = session.outputs.get("video")
- if session.outputs.get("render_key") == key and video and os.path.exists(video):
+ # A validate="full" request is for checking an existing render, not for
+ # getting a cached answer back unexamined, so it always re-earns its check.
+ if validate != "full" and session.outputs.get("render_key") == key and video and os.path.exists(video):
_emit(progress_callback, 100, "Already rendered with these settings")
return session.outputs
width, height, fps = _output_format(session)
@@ -1850,62 +2547,150 @@ def render_session(
splice = [(a - start, b - start) for a, b in kept_segments(session)] if session.removals else None
done = [0, 0]
lock = threading.Lock()
+ shots_dir = _work_dir(session.session_id) / "shots"
+ warnings: list[str] = []
+ video_ends: dict[str, float] = {}
def run(item):
i, piece = item
- chunk = work / f"shot-{i:05d}.mp4"
cam = session.source(piece.source_id) if piece.source_id else None
- _render_shot(session, cam, chunk, piece.tl_start, piece.frames, width, height, fps, session.look)
+ command = _shot_command(session, cam, piece.tl_start, piece.frames, width, height, fps, session.look)
+ chunk = shots_dir / _shot_key(session, cam, command) / "shot.mp4"
+ if not (chunk.exists() and _frame_count(chunk) == piece.frames):
+ chunk.parent.mkdir(parents=True, exist_ok=True)
+ tmp = chunk.with_name(f"shot.{i}.tmp.mp4")
+ proc_run([*command, str(tmp)], timeout=600 if cam is None else 3600, check=True)
+ got = _frame_count(tmp)
+ if abs(got - piece.frames) > 1:
+ raise RuntimeError(f"Shot {i + 1} came out {got} frames long instead of {piece.frames}.")
+ os.replace(tmp, chunk)
+ if got != piece.frames:
+ with lock:
+ warnings.append(f"Shot {i + 1} came out {got} frames long instead of {piece.frames}.")
+ held = _ran_out(session, cam, piece, fps, video_ends) if cam is not None else []
with lock:
+ warnings.extend(held)
done[0] += piece.frames
done[1] += 1
- _emit(progress_callback, 3 + 80 * done[0] / total_frames, f"Cut {done[1]} of {len(jobs)} shots")
+ _emit(progress_callback, 3 + 78 * done[0] / total_frames, f"Cut {done[1]} of {len(jobs)} shots")
return chunk
workers = max(1, min(4, (os.cpu_count() or 2) // 2))
with ThreadPoolExecutor(max_workers=workers) as pool:
chunks = list(pool.map(run, enumerate(jobs)))
- _emit(progress_callback, 84, "Mixing microphones")
+ _emit(progress_callback, 82, "Mixing microphones")
audio = work / "audio.wav"
- _render_audio(session, audio, start, end - start)
- if splice:
- _splice_audio(audio, work / "kept.wav", splice, ["-c:a", "pcm_s16le"])
- audio = work / "kept.wav"
+ gain = _render_audio(session, audio, start, end - start, splice)
+ short = duration - _media_duration(audio)
+ if short >= 1 / fps:
+ warnings.append(f"The mixed audio is {short:.2f} s shorter than the picture; the end plays silent.")
- _emit(progress_callback, 92, "Joining shots")
+ _emit(progress_callback, 88, "Joining shots")
listing = work / "shots.txt"
listing.write_text("".join(f"file '{c.as_posix()}'\n" for c in chunks), encoding="utf-8")
partial = work / "episode.mp4"
+ # Capped at the plan's length rather than -shortest, which would quietly
+ # cut the picture to fit a short mix instead of letting the check above see it.
proc_run([
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
"-f", "concat", "-safe", "0", "-i", str(listing), "-i", str(audio),
- "-map", "0:v:0", "-map", "1:a:0", "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest",
- "-movflags", "+faststart", str(partial),
+ "-map", "0:v:0", "-map", "1:a:0", "-c:v", "copy", "-c:a", "aac", "-b:a", "192k",
+ "-t", f"{duration:.6f}", "-movflags", "+faststart", str(partial),
], timeout=3600, check=True)
- video = out_dir / "episode.mp4"
- # shutil.move falls back to copy+delete when work and output sit on different volumes.
- shutil.move(str(partial), str(video))
+
+ _emit(progress_callback, 91, "Checking the episode")
+ validation = _validate(partial, total_frames, fps, warnings, full=validate == "full")
stem_paths = []
if stems:
_emit(progress_callback, 96, "Writing separate mic tracks")
- stem_paths = _render_stems(session, out_dir, start, end - start, splice)
+ # Built into `work`, not `out_dir`: if this fails, nothing at the
+ # canonical output path changes, and the video built above is
+ # discarded along with it instead of being published alone.
+ stem_paths = _render_stems(session, work, start, end - start, splice, gain)
+
+ # Every piece rendered and staged beside its final name, so publishing
+ # is a run of same-volume renames that takes milliseconds. Paths stay
+ # stable across renders; a crash inside that window can still pair a
+ # new video with old stems, which the next render overwrites.
+ video = out_dir / "episode.mp4"
+ tmp_video = video.with_name(video.name + ".publishing")
+ # shutil.move falls back to copy+delete when work and output sit on different volumes.
+ shutil.move(str(partial), str(tmp_video))
+ pending_stems = []
+ for stem in stem_paths:
+ dest = out_dir / Path(stem).name
+ tmp_stem = dest.with_name(dest.name + ".publishing")
+ shutil.move(stem, str(tmp_stem))
+ pending_stems.append((tmp_stem, dest))
+
+ os.replace(str(tmp_video), str(video))
+ for tmp_stem, dest in pending_stems:
+ os.replace(str(tmp_stem), str(dest))
+ # A person dropped from the edit would otherwise keep last render's stem.
+ current = {str(dest) for _, dest in pending_stems}
+ for old in session.outputs.get("stems") or []:
+ if old not in current and Path(old).parent == out_dir and os.path.exists(old):
+ os.remove(old)
session.outputs = {
**session.outputs,
"video": str(video),
- "stems": stem_paths,
+ "stems": [str(dest) for _, dest in pending_stems],
"duration": round(duration, 3),
"render_key": key,
+ "validation": validation,
}
session.save()
- _emit(progress_callback, 100, "Episode ready")
+ # Shots this edit no longer uses would pile up across re-cuts; keep only this render's.
+ used = {c.parent for c in chunks}
+ for d in shots_dir.iterdir() if shots_dir.exists() else []:
+ if d not in used:
+ shutil.rmtree(d, ignore_errors=True)
+ _emit(progress_callback, 100, "Episode ready" if not validation["warnings"]
+ else f"Episode ready with {len(validation['warnings'])} warning(s)")
return session.outputs
finally:
shutil.rmtree(work, ignore_errors=True)
+def _validate(video: Path, frames_expected: int, fps: float, warnings: list[str], *, full: bool = False) -> dict:
+ """Check the muxed episode against its plan; raises on a broken file, warns on everything else.
+
+ A decode error or a picture more than a frame off the plan means the file
+ can't be trusted at all. Loudness off target or a held frame still makes
+ a usable episode, so those are reported, not fatal.
+ """
+ errors = _decode_errors(video, full=full)
+ if errors:
+ raise RuntimeError(f"The rendered episode doesn't decode cleanly: {errors[:400]}")
+ frames_actual = _frame_count(video)
+ if abs(frames_actual - frames_expected) > 1:
+ raise RuntimeError(f"The rendered episode has {frames_actual} frames; the cut needs {frames_expected}.")
+ warnings = list(warnings)
+ if frames_actual != frames_expected:
+ warnings.append(f"The episode has {frames_actual} frames; the cut planned {frames_expected}.")
+ duration = _media_duration(video)
+ if abs(duration - frames_expected / fps) > 1.5 / fps:
+ warnings.append(f"The episode runs {duration:.3f} s; the cut planned {frames_expected / fps:.3f} s.")
+ lufs, true_peak = _loudness(video)
+ if lufs is None:
+ warnings.append("The episode's audio is silent.")
+ elif abs(lufs - LOUDNESS_TARGET) > 1.0:
+ warnings.append(f"Loudness is {lufs:.1f} LUFS; the target is {LOUDNESS_TARGET:.0f} LUFS.")
+ if true_peak is not None and true_peak > TRUE_PEAK_CEILING + 0.5:
+ warnings.append(f"True peak is {true_peak:.1f} dBTP, above the {TRUE_PEAK_CEILING} dBTP ceiling.")
+ return {
+ "frames_expected": frames_expected,
+ "frames_actual": frames_actual,
+ "duration": round(duration, 3),
+ "lufs": lufs,
+ "true_peak": true_peak,
+ "warnings": warnings,
+ }
+
+
def export_xml(session: MulticamSession, fmt: str, *, review: bool = False) -> str:
from services.integrations._shared import multicam_xml
@@ -1925,11 +2710,33 @@ def export_xml(session: MulticamSession, fmt: str, *, review: bool = False) -> s
else:
out = out_dir / f"episode{tag}.fcpxml"
multicam_xml.write_fcpxml(session, out, width=width, height=height, fps=fps, review=review)
- session.outputs = {**session.outputs, f"{fmt}_review" if review else fmt: str(out)}
+ handoff = _write_color_handoff(session, out_dir)
+ session.outputs = {**session.outputs, f"{fmt}_review" if review else fmt: str(out), "color_handoff": str(handoff)}
session.save()
return str(out)
+def _write_color_handoff(session: MulticamSession, out_dir: Path) -> Path:
+ """Which LUT each camera file needs, beside the exported timeline.
+
+ Neither xmeml nor FCPXML carries a LUT an editor applies reliably, so the
+ colorist gets this list instead of a half-applied grade.
+ """
+ out = out_dir / "color_handoff.json"
+ names = {p.id: p.name for p in session.people}
+ cameras = [{
+ "source_id": c.id,
+ "file": c.path,
+ "person": names.get(c.person, c.person),
+ "input_lut": c.input_lut or None,
+ } for c in session.cameras() if c.synced and not c.virtual]
+ tmp = out.with_suffix(".json.tmp")
+ tmp.write_text(json.dumps({"session": session.name, "look": session.look, "cameras": cameras}, indent=2),
+ encoding="utf-8")
+ os.replace(tmp, out)
+ return out
+
+
MAX_TIMELINE_BYTES = 200 * 1024 * 1024
diff --git a/backend/services/multicam_cloud.py b/backend/services/multicam_cloud.py
index 422ce20..ed0a1ab 100644
--- a/backend/services/multicam_cloud.py
+++ b/backend/services/multicam_cloud.py
@@ -80,6 +80,11 @@ def push(session: mc.MulticamSession, *, model_size: str = "base", engine: Optio
_emit(progress_callback, 35, "Making previews")
session = mc.build_preview(session)
activity = mc.activity(session)
+ # Freeze the basis against what's actually being sent, not whatever is on
+ # disk once uploads finish: edits keep landing while proxies encode, and
+ # the cloud editor's cuts must be checked against the placement that
+ # cuts and removals were actually made on, not a later one.
+ basis = mc.sync_basis_signature(session)
data = mc.payload(session)
preview = data.pop("preview", None) or {}
for key in ("outputs", "stats"):
@@ -110,7 +115,7 @@ def push(session: mc.MulticamSession, *, model_size: str = "base", engine: Optio
_request("POST", f"/v1/multicam/{opened['id']}/hybrid-complete", {"files": stored})
latest = mc.MulticamSession.load(session.session_id)
- latest.cloud = {"id": opened["id"], "url": editor_url(opened["id"])}
+ latest.cloud = {"id": opened["id"], "url": editor_url(opened["id"]), "basis": basis}
latest.save()
_emit(progress_callback, 100, "In the cloud editor")
return latest
@@ -119,11 +124,11 @@ def push(session: mc.MulticamSession, *, model_size: str = "base", engine: Optio
def resolve(target: str) -> mc.MulticamSession:
"""A local session id, or the cloud edit id a local session was sent as."""
if re.fullmatch(r"[a-f0-9]{6,32}", target or ""):
- return mc.MulticamSession.load(target)
+ return mc.open_session(target)
for item in mc.list_sessions():
session = mc.MulticamSession.load(item["session_id"])
if (session.cloud or {}).get("id") == target:
- return session
+ return mc.open_session(session.session_id)
raise MulticamCloudError(
f"No multicam edit on this computer was sent to the cloud as {target}. "
"Pull it where it was prepared, with the camera files")
@@ -134,6 +139,14 @@ def pull(session: mc.MulticamSession) -> mc.MulticamSession:
edit_id = (session.cloud or {}).get("id")
if not edit_id:
raise MulticamCloudError("this edit was never sent to podcli cloud. Send it with --cloud first")
+ pushed_basis = (session.cloud or {}).get("basis")
+ current_basis = mc.sync_basis_signature(session)
+ if pushed_basis and pushed_basis != current_basis:
+ raise MulticamCloudError(
+ "sources moved on the timeline (re-synced, nudged, or re-mapped) since this edit was sent to the "
+ "podcli cloud editor. Its cuts and removals were made against the old positions and would land on "
+ "the wrong footage now. Send it again with --cloud before pulling."
+ )
edit = _request("GET", f"/v1/multicam/{edit_id}/edit")
state = edit.get("state") or {}
@@ -148,5 +161,8 @@ def pull(session: mc.MulticamSession) -> mc.MulticamSession:
if cuts:
session = mc.set_cuts(session, cuts)
if isinstance(edit.get("removals"), list):
- session = mc.set_removals(session, [{"start": r["start"], "end": r["end"]} for r in edit["removals"]])
+ session = mc.set_removals(session, [
+ {"start": r["start"], "end": r["end"], **({"reason": r["reason"]} if r.get("reason") else {})}
+ for r in edit["removals"]
+ ])
return session
diff --git a/backend/services/multicam_signal.py b/backend/services/multicam_signal.py
index 22a47f9..b3881f5 100644
--- a/backend/services/multicam_signal.py
+++ b/backend/services/multicam_signal.py
@@ -116,8 +116,15 @@ def gcc_phat(ref: np.ndarray, src: np.ndarray, sample_rate: int, max_shift_secon
class ClockFit:
offset: float
speed: float
- residual_ms: float
- checkpoints: int
+ residual_ms: float # worst residual among the surviving (inlier) checkpoints
+ checkpoints: int # surviving checkpoint count
+ total_checkpoints: int = 0 # checkpoints offered to the fit, before outliers were dropped
+ residual_all_ms: float = 0.0 # worst residual against every checkpoint, outliers included
+ speed_fallback: bool = False # drift looked implausible, so speed was forced back to 1.0
+
+
+# Real clock drift between recorders stays far under this (|speed - 1|, so 1000 ppm).
+MAX_DRIFT = 1e-3
def fit_clock(points: list[tuple[float, float, float]], *, max_residual: float = 0.02) -> Optional[ClockFit]:
@@ -125,12 +132,13 @@ def fit_clock(points: list[tuple[float, float, float]], *, max_residual: float =
Checkpoints further than max_residual seconds from a Theil-Sen line are
dropped before the weighted least-squares fit.
- A single point yields speed 1. Implausible speeds (over 1000 ppm) fall back
+ A single point yields speed 1. Implausible speeds (over MAX_DRIFT) fall back
to the median offset at speed 1, since real clock drift is far smaller.
"""
if not points:
return None
- pts = np.array(points, dtype=np.float64)
+ all_pts = np.array(points, dtype=np.float64)
+ total = len(all_pts)
def solve(p: np.ndarray) -> tuple[float, float]:
if len(p) < 2 or np.ptp(p[:, 0]) < 1.0:
@@ -142,25 +150,32 @@ def fit_clock(points: list[tuple[float, float, float]], *, max_residual: float =
# Theil-Sen seeds the outlier test; least squares alone lets one bad
# checkpoint drag the line far enough that good points look bad too.
slopes = [
- (pts[j, 1] - pts[i, 1]) / (pts[j, 0] - pts[i, 0])
- for i in range(len(pts)) for j in range(i + 1, len(pts))
- if pts[j, 0] - pts[i, 0] >= 1.0
+ (all_pts[j, 1] - all_pts[i, 1]) / (all_pts[j, 0] - all_pts[i, 0])
+ for i in range(total) for j in range(i + 1, total)
+ if all_pts[j, 0] - all_pts[i, 0] >= 1.0
]
speed = float(np.median(slopes)) if slopes else 1.0
- offset = float(np.median(pts[:, 1] - speed * pts[:, 0]))
- keep = np.abs(pts[:, 1] - (offset + speed * pts[:, 0])) <= max_residual
- if keep.any():
- pts = pts[keep]
+ offset = float(np.median(all_pts[:, 1] - speed * all_pts[:, 0]))
+ keep = np.abs(all_pts[:, 1] - (offset + speed * all_pts[:, 0])) <= max_residual
+ pts = all_pts[keep] if keep.any() else all_pts
offset, speed = solve(pts)
residual = np.abs(pts[:, 1] - (offset + speed * pts[:, 0]))
- if abs(speed - 1.0) > 1e-3:
+ speed_fallback = abs(speed - 1.0) > MAX_DRIFT
+ if speed_fallback:
offset, speed = float(np.median(pts[:, 1] - pts[:, 0])), 1.0
residual = np.abs(pts[:, 1] - (offset + pts[:, 0]))
+ # Residual over every checkpoint offered to the fit, outliers included: a
+ # fit that dropped its way to a clean-looking inlier residual can still be
+ # wrong if most of the checkpoints it threw out disagreed with the line.
+ residual_all = np.abs(all_pts[:, 1] - (offset + speed * all_pts[:, 0]))
return ClockFit(
offset=offset,
speed=speed,
residual_ms=float(residual.max()) * 1000.0,
checkpoints=len(pts),
+ total_checkpoints=total,
+ residual_all_ms=float(residual_all.max()) * 1000.0,
+ speed_fallback=speed_fallback,
)
@@ -296,7 +311,8 @@ def plan_cuts(
longer one opens wide for guest_delay seconds before going to the guest,
as a call recording cuts between the split screen and the guest. Split
camera files (part 1, part 2) are handled because every decision checks
- which camera actually covers the moment.
+ which camera actually covers the moment. Once the shots are settled, a
+ cut made where someone starts to speak moves LEAD_IN seconds earlier.
"""
if not cameras or range_end <= range_start:
return []
@@ -389,6 +405,7 @@ def plan_cuts(
broken.append({**shot, "start": cursor, "end": shot["end"]})
shots = broken
+ _lead_in(shots, labels, by_id, range_start, min_shot)
return [
{"start": round(s["start"], 3), "end": round(s["end"], 3), "source_id": s["source_id"]}
for s in shots
@@ -396,6 +413,35 @@ def plan_cuts(
]
+# A cut made where someone starts to speak lands this much before their first
+# word, so the picture is already on them when the sound arrives.
+LEAD_IN = 0.12
+
+
+def _lead_in(shots: list[dict], labels: np.ndarray, by_id: dict[str, Camera], range_start: float,
+ min_shot: float) -> None:
+ """Move each cut that a speech onset caused up to LEAD_IN seconds earlier, in place.
+
+ The cut only moves back through silence (or the incoming speaker's own
+ sound), never across anyone else's words. It never leaves the outgoing
+ shot shorter than min_shot, and never reaches before the incoming camera
+ started rolling.
+ """
+ first = int(range_start / FRAME_SECONDS)
+ steps = int(round(LEAD_IN / FRAME_SECONDS))
+ for prev, shot in zip(shots, shots[1:]):
+ f = first + int(round((shot["start"] - range_start) / FRAME_SECONDS))
+ if not 0 < f < len(labels) or labels[f] == SILENT or labels[f] == labels[f - 1]:
+ continue
+ earliest = max(prev["start"] + min_shot, by_id[shot["source_id"]].start)
+ k = f
+ while (f - k < steps and k > 0 and labels[k - 1] in (SILENT, labels[f])
+ and range_start + (k - 1 - first) * FRAME_SECONDS >= earliest - 1e-9):
+ k -= 1
+ if k < f:
+ prev["end"] = shot["start"] = range_start + (k - first) * FRAME_SECONDS
+
+
# A guest's full-frame shot runs this far past their last word, then the
# split returns before the next speaker, as the measured remote edits do.
SOLO_TAIL = 1.0
diff --git a/backend/services/opening_hook.py b/backend/services/opening_hook.py
new file mode 100644
index 0000000..2884d2f
--- /dev/null
+++ b/backend/services/opening_hook.py
@@ -0,0 +1,143 @@
+"""Opening hook: a short spoken passage from inside a clip, played first.
+
+A clip's body plays its keep ranges in source order. A hook puts one range
+from inside that body ahead of it, so the playback order stops being
+monotonic on the source clock. "repeat" plays the passage again where it
+belongs; "move" lifts it out of the body.
+"""
+
+from typing import Optional
+
+MIN_HOOK_SECONDS = 1.0
+MAX_HOOK_SECONDS = 15.0
+HOOK_MODES = ("repeat", "move")
+
+# Agents and the studio round timestamps independently. A hook that starts on
+# the clip's own start_second must not fail containment by a float hair.
+_EDGE_TOLERANCE = 0.01
+
+# A move hook can leave a sliver of body between its edge and a segment edge.
+# A part that short is a frame or two of encoder padding, not content.
+_MIN_REMNANT_SECONDS = 0.1
+
+
+def _is_number(value) -> bool:
+ return isinstance(value, (int, float)) and not isinstance(value, bool)
+
+
+def _covering_spans(ranges: list[dict]) -> list[list[float]]:
+ spans: list[list[float]] = []
+ for r in sorted(ranges, key=lambda r: r["start"]):
+ if spans and r["start"] <= spans[-1][1] + _EDGE_TOLERANCE:
+ spans[-1][1] = max(spans[-1][1], r["end"])
+ else:
+ spans.append([r["start"], r["end"]])
+ return spans
+
+
+def validate_hook(
+ hook: Optional[dict],
+ start_second: float,
+ end_second: float,
+ segments: Optional[list[dict]] = None,
+) -> Optional[dict]:
+ """Check a hook against the clip it opens. Returns it normalized, or None.
+
+ The body is the clip's segments when it has any, else its whole range.
+ Raises ValueError naming what is wrong.
+ """
+ if hook is None:
+ return None
+ if not isinstance(hook, dict):
+ raise ValueError("hook must be an object with start, end and mode")
+ start, end, mode = hook.get("start"), hook.get("end"), hook.get("mode")
+ if not _is_number(start) or not _is_number(end):
+ raise ValueError("hook start and end must be numbers of seconds")
+ if mode not in HOOK_MODES:
+ raise ValueError('hook mode must be "repeat" or "move"')
+ if end <= start:
+ raise ValueError("hook end must be greater than hook start")
+ length = end - start
+ if length < MIN_HOOK_SECONDS or length > MAX_HOOK_SECONDS:
+ raise ValueError(
+ f"hook runs {length:.1f}s. It must run between "
+ f"{MIN_HOOK_SECONDS:.0f} and {MAX_HOOK_SECONDS:.0f} seconds."
+ )
+ body = [s for s in (segments or []) if s["end"] > s["start"]]
+ if not body:
+ body = [{"start": start_second, "end": end_second}]
+ if not any(
+ a - _EDGE_TOLERANCE <= start and end <= b + _EDGE_TOLERANCE
+ for a, b in _covering_spans(body)
+ ):
+ raise ValueError(
+ f"hook {start:.2f}-{end:.2f}s is not inside the clip body. "
+ "Pick a passage the clip already plays."
+ )
+ return {"start": float(start), "end": float(end), "mode": mode}
+
+
+def snap_hook_to_words(hook: dict, words: Optional[list[dict]]) -> dict:
+ """Widen a hook edge that cuts through a word so the whole word plays."""
+ start, end = hook["start"], hook["end"]
+ for w in words or []:
+ if w["start"] < start < w["end"]:
+ start = w["start"]
+ if w["start"] < end < w["end"]:
+ end = w["end"]
+ return {**hook, "start": round(start, 3), "end": round(end, 3)}
+
+
+def order_with_hook(body: list[dict], hook: dict) -> list[dict]:
+ """The ranges to cut, in the order they play: the hook, then the body.
+
+ repeat keeps the body whole. move removes the hook's span from it.
+ """
+ head = {"start": hook["start"], "end": hook["end"]}
+ if hook["mode"] == "repeat":
+ return [head] + [{"start": s["start"], "end": s["end"]} for s in body]
+ rest = []
+ for s in body:
+ for a, b in (
+ (s["start"], min(s["end"], hook["start"])),
+ (max(s["start"], hook["end"]), s["end"]),
+ ):
+ if b - a >= _MIN_REMNANT_SECONDS:
+ rest.append({"start": a, "end": b})
+ if not rest:
+ raise ValueError(
+ "A move hook cannot take the whole clip. Use repeat, or widen the clip."
+ )
+ return [head] + rest
+
+
+def keyframes_to_playback(
+ keyframes: list[dict],
+ origin: float,
+ ranges: list[dict],
+ part_durations: Optional[list[float]] = None,
+) -> list[dict]:
+ """Re-express manual crop keyframes on the clip's playback clock.
+
+ Keyframes arrive relative to origin (the requested start_second) on the
+ source clock. Ranges play in list order, so one source instant can play
+ zero, one or two times. A keyframe holds until the next one, so each range
+ opens on whichever keyframe was in force at its first source instant.
+ """
+ src = sorted((origin + float(k["t"]), k["x_pct"]) for k in keyframes)
+ if not src:
+ return []
+ out = []
+ offset = 0.0
+ for i, r in enumerate(ranges):
+ held = src[0][1]
+ for t, x in src:
+ if t <= r["start"]:
+ held = x
+ out.append({"t": round(offset, 3), "x_pct": held})
+ for t, x in src:
+ if r["start"] < t < r["end"]:
+ out.append({"t": round(offset + t - r["start"], 3), "x_pct": x})
+ probed = part_durations[i] if part_durations and i < len(part_durations) else 0.0
+ offset += probed if probed > 0 else r["end"] - r["start"]
+ return out
diff --git a/backend/services/signal_cache.py b/backend/services/signal_cache.py
index 791cb27..831e6b6 100644
--- a/backend/services/signal_cache.py
+++ b/backend/services/signal_cache.py
@@ -15,9 +15,13 @@ from typing import Any, Optional
from config.paths import paths
from services.transcript_packer import compute_cache_hash
+# Bump when the energy or reaction analyzer's math changes, so old cached
+# profiles (keyed only on video content) don't get served under a new algorithm.
+SIGNAL_CACHE_VERSION = 2
+
def _signals_path(video_path: str) -> str:
- return os.path.join(paths["cache"], "signals", f"{compute_cache_hash(video_path)}.json")
+ return os.path.join(paths["cache"], "signals", f"{compute_cache_hash(video_path)}-v{SIGNAL_CACHE_VERSION}.json")
def load_signals(video_path: str) -> dict[str, Any]:
diff --git a/backend/services/silence_removal.py b/backend/services/silence_removal.py
index 9fcc111..96824ac 100644
--- a/backend/services/silence_removal.py
+++ b/backend/services/silence_removal.py
@@ -327,7 +327,46 @@ def analyze_silence(
return plan
-def _map_range(start: float, end: float, keep_segments: list[dict]) -> Optional[tuple[float, float]]:
+def _map_point(t: float, keep_segments: list[dict]) -> Optional[tuple[float, float]]:
+ """Map a single instant through keep_segments. None if t fell in a
+ removed range; otherwise its output position as a zero-length range."""
+ cursor = 0.0
+ for segment in keep_segments:
+ if segment["start"] <= t <= segment["end"]:
+ mapped = cursor + (t - segment["start"])
+ return mapped, mapped
+ cursor += segment["end"] - segment["start"]
+ return None
+
+
+def _map_range(
+ start: float,
+ end: float,
+ keep_segments: list[dict],
+ *,
+ assign_by_midpoint: bool = False,
+) -> Optional[tuple[float, float]]:
+ # A zero (or inverted) duration item has no overlap for the interval test
+ # below to find. overlap_end > overlap_start never holds when they're
+ # equal, so it always mapped to None and got dropped. Map it as a point
+ # instead: keep it if it falls inside a kept range, drop it if not.
+ if end <= start:
+ return _map_point(start, keep_segments)
+ if assign_by_midpoint:
+ # A word that straddles a cut (part of it sits in the removed gap
+ # between two kept ranges) belongs wholly to whichever side holds its
+ # midpoint, not to a stitched span across the cut. Stitching would
+ # silently absorb the cut into the word's own duration.
+ midpoint = (start + end) / 2.0
+ cursor = 0.0
+ for segment in keep_segments:
+ seg_start, seg_end = segment["start"], segment["end"]
+ if seg_start <= midpoint <= seg_end:
+ overlap_start = max(start, seg_start)
+ overlap_end = min(end, seg_end)
+ return cursor + overlap_start - seg_start, cursor + overlap_end - seg_start
+ cursor += seg_end - seg_start
+ return None
output_cursor = 0.0
mapped_parts: list[tuple[float, float]] = []
for segment in keep_segments:
@@ -344,7 +383,12 @@ def _map_range(start: float, end: float, keep_segments: list[dict]) -> Optional[
return mapped_parts[0][0], mapped_parts[-1][1]
-def remap_timed_items(items: list[dict], keep_segments: list[dict]) -> list[dict]:
+def remap_timed_items(
+ items: list[dict],
+ keep_segments: list[dict],
+ *,
+ assign_by_midpoint: bool = False,
+) -> list[dict]:
remapped: list[dict] = []
for item in items:
try:
@@ -352,16 +396,27 @@ def remap_timed_items(items: list[dict], keep_segments: list[dict]) -> list[dict
end = float(item["end"])
except (KeyError, TypeError, ValueError):
continue
- mapped = _map_range(start, end, keep_segments)
- if not mapped or mapped[1] - mapped[0] < 0.01:
+ mapped = _map_range(start, end, keep_segments, assign_by_midpoint=assign_by_midpoint)
+ if mapped is None:
+ continue
+ mapped_start, mapped_end = mapped
+ # The 10ms floor only makes sense for an interval a cut clipped down to
+ # near-nothing. Whisper emits many 10ms words (and float subtraction
+ # puts some at 9.99ms), so an item that kept its whole length stays,
+ # as does a zero-duration point from _map_point.
+ mapped_length = mapped_end - mapped_start
+ clipped = (end - start) - mapped_length > 1e-6
+ if clipped and mapped_length < 0.01:
continue
- remapped.append({**item, "start": round(mapped[0], 3), "end": round(mapped[1], 3)})
+ remapped.append({**item, "start": round(mapped_start, 3), "end": round(mapped_end, 3)})
return remapped
def remap_transcript(transcript: dict, keep_segments: list[dict]) -> dict:
remapped = dict(transcript or {})
- remapped["words"] = remap_timed_items(list(remapped.get("words") or []), keep_segments)
+ remapped["words"] = remap_timed_items(
+ list(remapped.get("words") or []), keep_segments, assign_by_midpoint=True
+ )
remapped["segments"] = remap_timed_items(list(remapped.get("segments") or []), keep_segments)
remapped["duration"] = round(sum(s["end"] - s["start"] for s in keep_segments), 3)
remapped["silence_removed"] = True
diff --git a/backend/services/subtitle_export.py b/backend/services/subtitle_export.py
new file mode 100644
index 0000000..c0e3378
--- /dev/null
+++ b/backend/services/subtitle_export.py
@@ -0,0 +1,98 @@
+"""SRT/VTT sidecar generation from a rendered clip's word list.
+
+Every rendered clip's captions are burned from a word list already retimed
+to the clip's own playback clock (0 at the first frame of the exported
+file). These sidecars are the same words and the same clock, written out as
+plain subtitle files instead of pixels, useful for platforms that take an
+uploaded subtitle track, for accessibility, and for anyone editing the clip
+further downstream.
+"""
+
+from typing import Optional
+
+
+def _timestamp(seconds: float, vtt: bool) -> str:
+ """HH:MM:SS,mmm (SRT) or HH:MM:SS.mmm (VTT).
+
+ Rounds to whole milliseconds once, up front, then derives h/m/s/ms by
+ integer division, so a value like 1.9996 carries into 2.000 instead of
+ rounding seconds and milliseconds separately and emitting "01,1000".
+ """
+ total_ms = round(max(0.0, seconds) * 1000)
+ h, rem = divmod(total_ms, 3_600_000)
+ m, rem = divmod(rem, 60_000)
+ s, ms = divmod(rem, 1000)
+ sep = "." if vtt else ","
+ return f"{h:02d}:{m:02d}:{s:02d}{sep}{ms:03d}"
+
+
+def _group_words(words: list[dict], words_per_line: int) -> list[dict]:
+ groups = []
+ chunk: list[dict] = []
+ for w in words:
+ text = (w.get("word") or "").strip()
+ if not text:
+ continue
+ chunk.append(w)
+ if len(chunk) >= words_per_line:
+ groups.append(chunk)
+ chunk = []
+ if chunk:
+ groups.append(chunk)
+ return [
+ {
+ "text": " ".join(w["word"].strip() for w in g),
+ "start": g[0]["start"],
+ "end": g[-1]["end"],
+ }
+ for g in groups
+ ]
+
+
+def words_to_srt(words: list[dict], words_per_line: int = 8) -> Optional[str]:
+ """SRT text from retimed words, or None if there's nothing to write."""
+ groups = _group_words(words, words_per_line)
+ if not groups:
+ return None
+ blocks = []
+ for i, g in enumerate(groups, 1):
+ blocks.append(
+ f"{i}\n{_timestamp(g['start'], False)} --> {_timestamp(g['end'], False)}\n{g['text']}\n"
+ )
+ return "\n".join(blocks)
+
+
+def words_to_vtt(words: list[dict], words_per_line: int = 8) -> Optional[str]:
+ """WEBVTT text from retimed words, or None if there's nothing to write."""
+ groups = _group_words(words, words_per_line)
+ if not groups:
+ return None
+ blocks = ["WEBVTT\n"]
+ for i, g in enumerate(groups, 1):
+ blocks.append(
+ f"{i}\n{_timestamp(g['start'], True)} --> {_timestamp(g['end'], True)}\n{g['text']}\n"
+ )
+ return "\n".join(blocks)
+
+
+def write_sidecars(words: list[dict], output_base: str, words_per_line: int = 8) -> dict:
+ """Write {output_base}.srt and {output_base}.vtt. Returns the paths
+ actually written (a key is omitted rather than pointing at an empty file
+ when there are no words to caption).
+ """
+ paths: dict[str, str] = {}
+ srt = words_to_srt(words, words_per_line)
+ if srt is not None:
+ srt_path = f"{output_base}.srt"
+ with open(srt_path, "w", encoding="utf-8") as f:
+ f.write(srt)
+ paths["srt_path"] = srt_path
+
+ vtt = words_to_vtt(words, words_per_line)
+ if vtt is not None:
+ vtt_path = f"{output_base}.vtt"
+ with open(vtt_path, "w", encoding="utf-8") as f:
+ f.write(vtt)
+ paths["vtt_path"] = vtt_path
+
+ return paths
diff --git a/backend/services/thumbnail_ai.py b/backend/services/thumbnail_ai.py
index 4b10ff0..c88998b 100644
--- a/backend/services/thumbnail_ai.py
+++ b/backend/services/thumbnail_ai.py
@@ -29,7 +29,9 @@ def _load_brand_config() -> dict:
"height": 1920,
"accent_color": "#00CED1",
"bg_color": "#0D0D0D",
- "font_family": "'Inter', 'Helvetica Neue', 'Arial', sans-serif",
+ # Keep this fallback stack in sync with thumbnail_html.py's defaults and
+ # remotion/src/types.ts FONT. 'Inter' alone has no Georgian glyphs.
+ "font_family": "'Inter', 'Helvetica Neue', 'Arial', 'Noto Sans Georgian', sans-serif",
"enabled": True,
"variations": 3,
}
@@ -176,6 +178,37 @@ def _face_expression_quality(frame, x1: int, y1: int, x2: int, y2: int) -> float
return max(0.15, min(1.0, quality))
+# Below either floor a face is blurred, blinking or turned away. Kept only
+# when nothing better exists, so a clip still gets a picture.
+EXPRESSION_FLOOR = 0.25
+SHARPNESS_FLOOR = 12.0
+
+
+def face_portrait_score(frame, x1: int, y1: int, x2: int, y2: int, confidence: float) -> tuple[float, float, float]:
+ """How well one detected face would read on a thumbnail.
+
+ Returns (score, sharpness, expression quality). Favors a confident,
+ sharp, front-facing face at a natural portrait size over an extreme
+ close-up or a speck in the background.
+ """
+ h, w = frame.shape[:2]
+ face_area_pct = (x2 - x1) * (y2 - y1) / max(1, w * h) * 100
+ sharpness = _frame_sharpness(frame, (x1, y1, x2, y2))
+ expression_quality = _face_expression_quality(frame, x1, y1, x2, y2)
+
+ if face_area_pct > 35:
+ size_factor = 0.4
+ elif face_area_pct > 25:
+ size_factor = 0.7
+ elif face_area_pct < 3:
+ size_factor = 0.5
+ else:
+ size_factor = 1.0
+
+ score = (confidence ** 2) * (sharpness ** 0.5) * size_factor * expression_quality
+ return score, sharpness, expression_quality
+
+
def extract_candidate_frames(
video_path: str,
output_dir: str,
@@ -238,8 +271,6 @@ def extract_candidate_frames(
sample_count = max(count * 10, 40)
candidates = []
best_rejected = None
- expression_floor = 0.25
- sharpness_floor = 12.0
for i in range(sample_count):
t = start_t + i * (end_t - start_t) / sample_count
@@ -270,25 +301,14 @@ def extract_candidate_frames(
if not is_portrait and 0.4 * w < face_cx < 0.6 * w and fw < w * 0.15:
continue
- face_area_pct = (fw * fh) / (w * h) * 100
- sharpness = _frame_sharpness(frame, (x1, y1, x2, y2))
- expression_quality = _face_expression_quality(frame, x1, y1, x2, y2)
+ face_score, sharpness, expression_quality = face_portrait_score(frame, x1, y1, x2, y2, conf)
mid_t = (start_t + end_t) / 2
span = max(0.1, end_t - start_t)
distance_from_mid = abs(t - mid_t) / (span / 2)
position_boost = 1.0 - 0.4 * min(1.0, distance_from_mid)
- if face_area_pct > 35:
- size_factor = 0.4
- elif face_area_pct > 25:
- size_factor = 0.7
- elif face_area_pct < 3:
- size_factor = 0.5
- else:
- size_factor = 1.0
-
- score = (conf ** 2) * (sharpness ** 0.5) * size_factor * expression_quality * position_boost
+ score = face_score * position_boost
entry = {
"frame": frame.copy(),
@@ -302,7 +322,7 @@ def extract_candidate_frames(
"score": score,
}
- if sharpness < sharpness_floor or expression_quality < expression_floor:
+ if sharpness < SHARPNESS_FLOOR or expression_quality < EXPRESSION_FLOOR:
if best_rejected is None or score > best_rejected["score"]:
best_rejected = entry
continue
@@ -512,6 +532,27 @@ def _thumbnail_kb_context() -> str:
return f"\nBRAND KNOWLEDGE BASE (follow its thumbnail text rules, voice, and banned words):\n{kb}\n"
+def _grounding_context(grounding: Optional[dict]) -> str:
+ """Format the clip's payoff/question/opening line for the headline prompt.
+
+ Without this, headline copy is rewritten from the title alone, a short
+ label that drifts from what the clip actually says. The clip's own
+ payoff and verbatim opening line keep the copy honest to the content.
+ """
+ if not grounding:
+ return ""
+ lines = []
+ if grounding.get("payoff"):
+ lines.append(f"Payoff (what the clip delivers): {grounding['payoff']}")
+ if grounding.get("context_line"):
+ lines.append(f"Question this clip answers: {grounding['context_line']}")
+ if grounding.get("preview_text"):
+ lines.append(f"Clip's verbatim opening line: {grounding['preview_text']}")
+ if not lines:
+ return ""
+ return "\nCLIP CONTENT (ground the headline in this, not just the title):\n" + "\n".join(lines) + "\n"
+
+
def ask_claude_for_layout(
title: str,
frame_path: str,
@@ -519,6 +560,7 @@ def ask_claude_for_layout(
logo_path: Optional[str] = None,
config: Optional[dict] = None,
variation: int = 0,
+ grounding: Optional[dict] = None,
) -> Optional[dict]:
"""
Ask an available AI CLI to generate layout values for the thumbnail.
@@ -539,7 +581,7 @@ def ask_claude_for_layout(
prompt = f"""You are a thumbnail layout engine. Given a title and face position, return CSS values for a YouTube Shorts thumbnail (1080x1920).
TITLE: "{title}"
-{_thumbnail_kb_context()}
+{_grounding_context(grounding)}{_thumbnail_kb_context()}
PHOTO INFO:
{face_ctx}
@@ -554,7 +596,7 @@ Return ONLY valid JSON with these fields:
}}
RULES:
-- Rewrite the title into thumbnail copy, not a literal transcript sentence.
+- Rewrite the title into thumbnail copy, not a literal transcript sentence. If clip content is given above, the copy must match what the clip actually delivers.
- Split it into 2 impactful lines. Line 1 = setup, Line 2 = payoff.
- Keep the combined copy to 4-8 words total whenever possible.
- Keep each line short enough to fit comfortably on a Shorts thumbnail. Hard max: 24 characters per line.
@@ -569,7 +611,9 @@ RULES:
return layout if isinstance(layout, dict) else None
-def generate_headline_variations(title: str, n: int, config: Optional[dict] = None) -> list[tuple[str, str]]:
+def generate_headline_variations(
+ title: str, n: int, config: Optional[dict] = None, grounding: Optional[dict] = None
+) -> list[tuple[str, str]]:
"""Write n DISTINCT 2-line thumbnail headlines in a single AI call.
One call (rather than n independent layout calls) is both cheaper and the
@@ -580,12 +624,12 @@ def generate_headline_variations(title: str, n: int, config: Optional[dict] = No
prompt = f"""You are a thumbnail copywriter. Write {n} DISTINCT headline options for a YouTube Shorts thumbnail.
TITLE: "{title}"
-{_thumbnail_kb_context()}
+{_grounding_context(grounding)}{_thumbnail_kb_context()}
Return ONLY a JSON array of exactly {n} objects, each with "line1" and "line2":
[{{"line1": "FIRST LINE", "line2": "SECOND LINE"}}, ...]
RULES:
-- Rewrite the title into punchy thumbnail copy, not a literal transcript sentence.
+- Rewrite the title into punchy thumbnail copy, not a literal transcript sentence. If clip content is given above, the copy must match what the clip actually delivers.
- Line 1 = setup, Line 2 = payoff. 4-8 words total. Hard max 24 characters per line.
- Drop filler words; keep numbers, nouns, and the strongest claim. No slashes.
- Every option must read DIFFERENTLY — vary the hook, the emphasis, or which idea leads. Do not repeat the same phrasing across options."""
@@ -610,6 +654,8 @@ def generate_thumbnail_with_template(
variation: int = 0,
line1_override: Optional[str] = None,
line2_override: Optional[str] = None,
+ grounding: Optional[dict] = None,
+ people: Optional[list[dict]] = None,
) -> Optional[str]:
"""
Template + AI layout. Claude decides all dynamic values per frame.
@@ -617,6 +663,7 @@ def generate_thumbnail_with_template(
When line1_override is given, the two lines are used verbatim and the
Claude rewrite is skipped — the caller controls the exact split.
+ `people` carries the two panels of the pair layout from thumbnail_pair.
"""
from services.thumbnail_html import generate_thumbnail, _load_config, _prepare_thumbnail_lines
@@ -624,7 +671,9 @@ def generate_thumbnail_with_template(
if config:
cfg.update(config)
- layout = None if line1_override is not None else ask_claude_for_layout(title, frame_path, frame_info, logo_path, cfg, variation=variation)
+ layout = None if line1_override is not None else ask_claude_for_layout(
+ title, frame_path, frame_info, logo_path, cfg, variation=variation, grounding=grounding
+ )
if line1_override is not None:
line1, line2 = _prepare_thumbnail_lines(
@@ -656,10 +705,50 @@ def generate_thumbnail_with_template(
config=cfg,
variation=variation,
face_info=frame_info,
+ people=people,
+ )
+
+
+def resolve_pair(
+ cfg: dict,
+ output_dir: str,
+ video_path: Optional[str] = None,
+ start_second: Optional[float] = None,
+ end_second: Optional[float] = None,
+ layout: Optional[str] = None,
+ left_image: Optional[str] = None,
+ right_image: Optional[str] = None,
+ swap: bool = False,
+ face_map: Optional[dict] = None,
+ segments: Optional[list[dict]] = None,
+) -> Optional[dict]:
+ """The two-person panels when the pair layout is asked for, else None.
+
+ Asked for by `layout`, by the template's own layout, or by naming images
+ for both sides. The answer is thumbnail_pair's result, which says why when
+ it falls back to one face.
+ """
+ from services.thumbnail_html import _load_config, pair_panel_size
+ from services.thumbnail_pair import pick_pair
+
+ wanted = layout or cfg.get("layout") or "single"
+ if wanted != "pair" and not (left_image or right_image):
+ return None
+ sized = {**_load_config(), **cfg}
+ return pick_pair(
+ video_path, start_second, end_second,
+ os.path.join(output_dir, "_pair"), pair_panel_size(sized),
+ left_image=left_image, right_image=right_image, swap=swap,
+ face_map=face_map, segments=segments,
)
-def generate_variations(
+def generate_variations(*args, **kwargs) -> list[str]:
+ """Generate thumbnail variations and return their paths. See render_variations."""
+ return render_variations(*args, **kwargs)["paths"]
+
+
+def render_variations(
title: str,
output_dir: str,
photo_path: Optional[str] = None,
@@ -670,15 +759,24 @@ def generate_variations(
config: Optional[dict] = None,
line1: Optional[str] = None,
line2: Optional[str] = None,
-) -> list[str]:
+ grounding: Optional[dict] = None,
+ layout: Optional[str] = None,
+ left_image: Optional[str] = None,
+ right_image: Optional[str] = None,
+ swap: bool = False,
+ face_map: Optional[dict] = None,
+ segments: Optional[list[dict]] = None,
+) -> dict:
"""
Generate thumbnail variations using AI.
- 1. Extract candidate frames from video (if no photo provided)
+ 1. Extract candidate frames from video (if no photo provided), or two
+ people's panels for the pair layout
2. For each variation, Claude generates the HTML layout
3. Playwright renders to PNG
Falls back to template-based generation if Claude is unavailable.
+ Returns {"paths": [...], "pair": thumbnail_pair's result or None}.
"""
cfg = _load_brand_config()
if config:
@@ -687,9 +785,19 @@ def generate_variations(
os.makedirs(output_dir, exist_ok=True)
n = cfg.get("variations", 3)
+ pair = resolve_pair(
+ cfg, output_dir, video_path, start_second, end_second, layout,
+ left_image, right_image, swap, face_map, segments,
+ )
+ people = pair["people"] if pair and pair["layout"] == "pair" else None
+ if pair and not people:
+ log_event("thumbnail-ai", "pair layout fell back to one face", reason=pair["reason"])
+
# Get frames
frames = []
- if photo_path and os.path.exists(photo_path):
+ if people:
+ frames = [{"path": None}]
+ elif photo_path and os.path.exists(photo_path):
frames = [{"path": photo_path}]
elif video_path:
frames_dir = os.path.join(output_dir, "_frames")
@@ -702,7 +810,7 @@ def generate_variations(
end_second=end_second,
)
- if not frames and video_path:
+ if not frames and video_path and not people:
# Fallback: simpler face extraction with less aggressive filters
try:
from services.thumbnail_generator import extract_face_frame
@@ -724,7 +832,7 @@ def generate_variations(
# the variations differ in wording — not just in frame/styling.
headlines: list[tuple[str, str]] = []
if line1 is None:
- headlines = generate_headline_variations(title, n, config=cfg)
+ headlines = generate_headline_variations(title, n, config=cfg, grounding=grounding)
paths = []
for i in range(n):
@@ -746,12 +854,14 @@ def generate_variations(
variation=i,
line1_override=v_line1,
line2_override=v_line2,
+ grounding=grounding,
+ people=people,
)
if result:
paths.append(result)
- return paths
+ return {"paths": paths, "pair": pair}
def thumbnail_to_video_frame(
diff --git a/backend/services/thumbnail_html.py b/backend/services/thumbnail_html.py
index 81f6f11..c8de9a7 100644
--- a/backend/services/thumbnail_html.py
+++ b/backend/services/thumbnail_html.py
@@ -16,6 +16,7 @@ import shutil
import subprocess
from config.paths import paths
from utils.proc import run as proc_run, ProcError
+from utils.text import safe_upper
import sys
import tempfile
from typing import Optional
@@ -88,6 +89,14 @@ def _load_config() -> dict:
# Photo
"photo_brightness": 0.85,
+ # Layout: "single" is one face behind the headline. "pair" puts two
+ # people from the clip side by side, the guest left and the host right,
+ # over the top of the canvas, with the headline in the clear space below.
+ "layout": "single",
+ "pair_photo_height": "70%",
+ "pair_box_y": "85%",
+ "pair_divider_width": 4,
+
# Gradients
"gradient_top_height": "25%",
"gradient_top_start_color": "rgba(0,0,0,0.5)",
@@ -102,9 +111,14 @@ def _load_config() -> dict:
"logo_margin": "50px",
"logo_opacity": 0.35,
- # Font
- "font_family": "'Inter', 'Helvetica Neue', 'Arial', sans-serif",
- "font_import_url": "https://fonts.googleapis.com/css2?family=Inter:ital,wght@0,400;0,500;0,600;0,700;1,400;1,500;1,600;1,700&display=swap",
+ # Font. 'Inter' has no Georgian glyphs, so a Georgian-language show's
+ # headline would fall through to the browser's generic sans-serif with
+ # no brand voice at all. Noto Sans Georgian is the same family the
+ # Remotion caption renderer bundles for the same reason (see
+ # remotion/src/types.ts FONT), kept consistent so a thumbnail and its
+ # clip's captions render the same typeface for non-Latin text.
+ "font_family": "'Inter', 'Helvetica Neue', 'Arial', 'Noto Sans Georgian', sans-serif",
+ "font_import_url": "https://fonts.googleapis.com/css2?family=Inter:ital,wght@0,400;0,500;0,600;0,700;1,400;1,500;1,600;1,700&family=Noto+Sans+Georgian:wght@400;700&display=swap",
# Variations
"variations": 3,
@@ -400,6 +414,80 @@ def _layer_html(layers) -> str:
return "\n ".join(drawn)
+def pair_panel_size(config: dict) -> tuple[int, int]:
+ """Pixel size of one person's panel in the pair layout."""
+ height_pct = float(str(config.get("pair_photo_height", "70%")).rstrip("%") or 70)
+ return int(config["width"]) // 2, int(int(config["height"]) * height_pct / 100)
+
+
+def _single_photo_layer(cfg: dict, photo_path: str) -> tuple[str, str]:
+ """CSS and markup for one photo filling the canvas."""
+ w, h = cfg["width"], cfg["height"]
+ # as_uri() rather than an f-string: a '#' or '?' anywhere in the path is
+ # a fragment or a query to the browser, so the image silently drops and
+ # the card renders as bg_color with a broken-image glyph.
+ photo_uri = Path(photo_path).absolute().as_uri()
+ css = f"""
+ .photo {{
+ position: absolute; top: 0; left: 0; width: {w}px; height: {h}px;
+ overflow: hidden;
+ z-index: 1;
+ }}
+ .photo img {{
+ width: 100%; height: 100%;
+ object-fit: cover;
+ object-position: {cfg.get("photo_object_position", "center center")};
+ filter: brightness({cfg.get("photo_brightness", 0.85)});
+ }}
+ .photo-vignette {{
+ position: absolute; top: 0; left: 0; width: 100%; height: 100%;
+ z-index: 1;
+ }}
+ """
+ return css, f'<div class="photo"><img src="{photo_uri}" /></div><div class="photo-vignette"></div>'
+
+
+def _pair_photo_layer(cfg: dict, people: list[dict]) -> tuple[str, str]:
+ """CSS and markup for two portrait panels side by side, split by an accent rule."""
+ panel_h = cfg.get("pair_photo_height", "70%")
+ accent = cfg.get("frame_border_color") or cfg["accent_color"]
+ css = f"""
+ .pair {{
+ position: absolute; top: 0; width: 50%; height: {panel_h};
+ overflow: hidden;
+ z-index: 1;
+ }}
+ .pair-left {{ left: 0; }}
+ .pair-right {{ right: 0; }}
+ .pair img {{
+ width: 100%; height: 100%;
+ object-fit: cover;
+ object-position: center top;
+ filter: brightness({cfg.get("photo_brightness", 0.85)});
+ }}
+ .pair-divider {{
+ position: absolute; top: 0; left: 50%; height: {panel_h};
+ width: {cfg.get("pair_divider_width", 4)}px; transform: translateX(-50%);
+ background: {accent};
+ z-index: 3;
+ }}
+ """
+ panels = "".join(
+ f'<div class="pair pair-{p["side"]}"><img src="{Path(p["path"]).absolute().as_uri()}" /></div>'
+ for p in people
+ )
+ return css, panels + '<div class="pair-divider"></div>'
+
+
+def _usable_pair(people: Optional[list[dict]]) -> bool:
+ return bool(
+ people
+ and len(people) == 2
+ and {p.get("side") for p in people} == {"left", "right"}
+ and all(p.get("path") and os.path.exists(p["path"]) for p in people)
+ )
+
+
def _build_html(
line1: str,
line2: str,
@@ -408,12 +496,15 @@ def _build_html(
config: Optional[dict] = None,
variation: int = 0,
face_info: Optional[dict] = None,
+ people: Optional[list[dict]] = None,
) -> str:
"""Build the HTML for a single thumbnail — all values from config.
Args:
face_info: Dict with face_y_pct, face_h_pct etc. from frame extraction.
Used to auto-position the text box below the face.
+ people: Two portrait panels from thumbnail_pair, each with a side and
+ a path. When given, they replace the single photo.
"""
cfg = _load_config()
if config:
@@ -429,13 +520,16 @@ def _build_html(
# Clean any stray slashes from title split
line1 = line1.strip().strip("/").strip()
line2 = line2.strip().strip("/").strip()
- l1 = line1.upper() if cfg.get("line1_uppercase", True) else line1
- l2 = line2.upper() if cfg.get("line2_uppercase", True) else line2
+ l1 = safe_upper(line1) if cfg.get("line1_uppercase", True) else line1
+ l2 = safe_upper(line2) if cfg.get("line2_uppercase", True) else line2
- has_photo = photo_path and os.path.exists(str(photo_path))
+ pair = _usable_pair(people)
+ has_photo = pair or (photo_path and os.path.exists(str(photo_path)))
# Auto-position text box below face if we have face data
- if face_info and has_photo:
+ if pair:
+ default_y = cfg.get("pair_box_y", "85%")
+ elif face_info and has_photo:
face_y = face_info.get("face_y_pct", 50)
face_h = face_info.get("face_h_pct", 20)
# Bottom edge of face as percentage
@@ -456,32 +550,11 @@ def _build_html(
elif variation == 2:
box_y = f"calc({default_y} + {offset_down})"
- # Photo CSS
- photo_css = ""
- photo_uri = ""
- if has_photo:
- # as_uri() rather than an f-string: a '#' or '?' anywhere in the path is
- # a fragment or a query to the browser, so the image silently drops and
- # the card renders as bg_color with a broken-image glyph.
- photo_uri = Path(photo_path).absolute().as_uri()
- brightness = cfg.get("photo_brightness", 0.85)
- photo_css = f"""
- .photo {{
- position: absolute; top: 0; left: 0; width: {w}px; height: {h}px;
- overflow: hidden;
- z-index: 1;
- }}
- .photo img {{
- width: 100%; height: 100%;
- object-fit: cover;
- object-position: {cfg.get("photo_object_position", "center center")};
- filter: brightness({brightness});
- }}
- .photo-vignette {{
- position: absolute; top: 0; left: 0; width: 100%; height: 100%;
- z-index: 1;
- }}
- """
+ photo_css, photo_html = "", ""
+ if pair:
+ photo_css, photo_html = _pair_photo_layer(cfg, people)
+ elif has_photo:
+ photo_css, photo_html = _single_photo_layer(cfg, photo_path)
# Logo
logo_html = ""
@@ -643,7 +716,10 @@ body {{
font-weight: {l1_weight};
letter-spacing: {l1_spacing};
color: {l1_color};
- text-transform: uppercase;
+ /* Casing is already resolved in Python via safe_upper(), which skips
+ caseless scripts (e.g. Georgian) that CSS text-transform would
+ otherwise still remap to a different alphabet (Mkhedruli -> Mtavruli). */
+ text-transform: none;
text-align: center;
line-height: {l1_lh};
margin-bottom: {l1_mb};
@@ -663,7 +739,8 @@ body {{
font-weight: {l2_weight};
font-style: {l2_style};
letter-spacing: {l2_spacing};
- text-transform: uppercase;
+ /* See .line1 above: casing is resolved in Python, not here. */
+ text-transform: none;
line-height: {l2_lh};
background: {hl_color};
color: {l2_text_color};
@@ -679,7 +756,7 @@ body {{
</head>
<body>
<div class="frame"></div>
- {f'<div class="photo"><img src="{photo_uri}" /></div><div class="photo-vignette"></div>' if has_photo else ''}
+ {photo_html}
<div class="gradient-top"></div>
<div class="gradient-bottom"></div>
<div class="text-box">
@@ -701,13 +778,14 @@ def generate_thumbnail(
config: Optional[dict] = None,
variation: int = 0,
face_info: Optional[dict] = None,
+ people: Optional[list[dict]] = None,
) -> str:
"""Generate a single thumbnail via HTML + headless browser screenshot."""
cfg = _load_config()
if config:
cfg.update(config)
- html = _build_html(line1, line2, photo_path, logo_path, cfg, variation, face_info=face_info)
+ html = _build_html(line1, line2, photo_path, logo_path, cfg, variation, face_info=face_info, people=people)
# Write HTML to temp file
tmp_html = tempfile.NamedTemporaryFile(suffix=".html", delete=False, mode="w")
diff --git a/backend/services/thumbnail_pair.py b/backend/services/thumbnail_pair.py
new file mode 100644
index 0000000..bab1a74
--- /dev/null
+++ b/backend/services/thumbnail_pair.py
@@ -0,0 +1,327 @@
+"""
+Two people side by side on a thumbnail, each taken from the clip's own footage.
+
+An interview clip is a conversation, and a thumbnail with one face hides half
+of it. This picks one sharp, face-forward frame per person from inside the
+clip and cuts each into a portrait panel: the guest on the left and the host
+on the right, unless asked otherwise.
+
+Who is who comes from what podcli already knows, in this order:
+ 1. images the caller names for each side;
+ 2. a multicam edit whose render is this video, where every person has a
+ camera of their own and a role;
+ 3. two people seated in one shot, or two panes of a call recording, found
+ the same way the face map finds seats.
+When none of these tells two people apart, the result says so and the caller
+draws the single-face layout. A panel is never filled with an invented or
+repeated person.
+"""
+
+import os
+import tempfile
+from pathlib import Path
+from typing import Iterator, Optional
+
+from services.face_track_helpers import seats_from_frames
+from services.thumbnail_ai import (
+ EXPRESSION_FLOOR,
+ SHARPNESS_FLOOR,
+ _tile_bounds_for_face,
+ face_portrait_score,
+)
+
+SAMPLES = 36
+
+# A face belongs to the nearer seat only when it sits this close to it, as a
+# share of the frame width. Further out it is someone walking through.
+_SEAT_REACH = 0.15
+
+# The face's share of the panel width, and where its centre sits down the panel.
+_FACE_WIDTH_SHARE = 0.42
+_FACE_HEIGHT_AT = 0.38
+
+
+def _single(reason: str) -> dict:
+ return {"layout": "single", "people": [], "reason": reason}
+
+
+def _sample_frames(video_path: str, start: float, end: float, count: int) -> Iterator[tuple[float, object]]:
+ """(source second, BGR frame) spread across the clip, clear of its edges."""
+ import cv2
+
+ cap = cv2.VideoCapture(video_path)
+ if not cap.isOpened():
+ return
+ try:
+ margin = min(0.6, (end - start) * 0.08)
+ a, b = start + margin, end - margin
+ if b <= a:
+ a, b = start, end
+ for i in range(count):
+ t = a + (i + 0.5) * (b - a) / count
+ cap.set(cv2.CAP_PROP_POS_MSEC, t * 1000)
+ ok, frame = cap.read()
+ if ok:
+ yield round(t, 2), frame
+ finally:
+ cap.release()
+
+
+_detectors: dict = {}
+
+
+def _find_faces(frame) -> list[dict]:
+ """Faces in one frame as [{cx, cy, fw, fh, confidence}], or [] without a detector."""
+ from services.face_detector import create_detector, detect_faces
+
+ h, w = frame.shape[:2]
+ if (w, h) not in _detectors:
+ _detectors[(w, h)] = create_detector(w, h)
+ detector = _detectors[(w, h)]
+ return detect_faces(detector, frame, w, h) if detector else []
+
+
+def _candidate(frame, face: dict, t: Optional[float]) -> dict:
+ h, w = frame.shape[:2]
+ x1, y1 = max(0, face["cx"] - face["fw"] // 2), max(0, face["cy"] - face["fh"] // 2)
+ x2, y2 = min(w, face["cx"] + face["fw"] // 2), min(h, face["cy"] + face["fh"] // 2)
+ score, sharpness, expression = face_portrait_score(frame, x1, y1, x2, y2, face["confidence"])
+ return {
+ "frame": frame, "face": face, "time": t, "score": score,
+ "clean": sharpness >= SHARPNESS_FLOOR and expression >= EXPRESSION_FLOOR,
+ }
+
+
+def _best(candidates: list[dict]) -> Optional[dict]:
+ """The highest-scoring face, preferring one that is neither blurred nor turned away."""
+ clean = [c for c in candidates if c["clean"]]
+ pool = clean or candidates
+ return max(pool, key=lambda c: c["score"]) if pool else None
+
+
+def _panel(frame, face: dict, size: tuple[int, int]):
+ """Cut a portrait of one face to the panel's shape. Returns (image, face position in %)."""
+ import cv2
+
+ pw, ph = size
+ ratio = pw / ph
+ h, w = frame.shape[:2]
+ crop_w = min(w, max(int(face["fw"] / _FACE_WIDTH_SHARE), 2))
+ crop_h = int(crop_w / ratio)
+ if crop_h > h:
+ crop_h = h
+ crop_w = int(crop_h * ratio)
+
+ # A call recording's panes are tiles. Staying inside this face's tile keeps
+ # the next person's pane, and the seam between them, out of the portrait.
+ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+ tile_left, tile_right = _tile_bounds_for_face(gray, face["cx"])
+ if tile_right - tile_left < crop_w:
+ crop_w = tile_right - tile_left
+ crop_h = int(crop_w / ratio)
+ x = max(tile_left, min(face["cx"] - crop_w // 2, tile_right - crop_w))
+ y = max(0, min(face["cy"] - int(crop_h * _FACE_HEIGHT_AT), h - crop_h))
+
+ cut = frame[y:y + crop_h, x:x + crop_w]
+ interp = cv2.INTER_AREA if crop_w > pw else cv2.INTER_LANCZOS4
+ image = cv2.resize(cut, (pw, ph), interpolation=interp)
+ return image, {
+ "face_x_pct": round((face["cx"] - x) / crop_w * 100, 1),
+ "face_y_pct": round((face["cy"] - y) / crop_h * 100, 1),
+ "face_w_pct": round(face["fw"] / crop_w * 100, 1),
+ "face_h_pct": round(face["fh"] / crop_h * 100, 1),
+ }
+
+
+def _write_panel(pick: dict, size: tuple[int, int], path: str) -> dict:
+ import cv2
+
+ image, where = _panel(pick["frame"], pick["face"], size)
+ if not cv2.imwrite(path, image, [cv2.IMWRITE_JPEG_QUALITY, 92]):
+ raise OSError(f"could not write {path}")
+ return {"path": path, **where}
+
+
+def _from_images(left: str, right: str, size: tuple[int, int], out_dir: str) -> list[dict]:
+ import cv2
+
+ people = []
+ for n, (side, src) in enumerate((("left", left), ("right", right)), start=1):
+ frame = cv2.imread(src)
+ if frame is None:
+ raise ValueError(f"cannot read the {side} image: {src}")
+ faces = _find_faces(frame)
+ h, w = frame.shape[:2]
+ # A portrait without a detectable face is still the person the caller
+ # chose; it is framed from its centre instead.
+ face = max(faces, key=lambda f: f["fw"]) if faces else {
+ "cx": w // 2, "cy": h // 3, "fw": int(w * _FACE_WIDTH_SHARE), "fh": int(w * _FACE_WIDTH_SHARE),
+ }
+ panel = _write_panel({"frame": frame, "face": face}, size, os.path.join(out_dir, f"image_{n}.jpg"))
+ people.append({"side": side, "role": None, "from": "image", "image": src, "source_time": None, **panel})
+ return people
+
+
+def _talk_seconds(segments: Optional[list[dict]], start: float, end: float) -> dict[str, float]:
+ talk: dict[str, float] = {}
+ for s in segments or []:
+ sp = s.get("speaker")
+ overlap = min(end, s.get("end", 0)) - max(start, s.get("start", 0))
+ if sp and overlap > 0:
+ talk[sp] = talk.get(sp, 0.0) + overlap
+ return talk
+
+
+def _seat_roles(face_map: Optional[dict], segments: Optional[list[dict]], start: float, end: float) -> Optional[list[str]]:
+ """["guest", "host"] or the reverse for the left and right seats, or None when unknown.
+
+ The face map ties each diarized speaker to a seat. Of the two people
+ talking in the clip, the one who talks more is taken as the guest: an
+ interview clip is built around the answer.
+ """
+ clusters = (face_map or {}).get("clusters") or []
+ mapping = (face_map or {}).get("speaker_mappings") or {}
+ if len(clusters) != 2 or not mapping:
+ return None
+ talk = sorted(_talk_seconds(segments, start, end).items(), key=lambda kv: kv[1], reverse=True)
+ if len(talk) < 2:
+ return None
+ guest_seat, host_seat = mapping.get(talk[0][0]), mapping.get(talk[1][0])
+ if guest_seat not in (0, 1) or host_seat not in (0, 1) or guest_seat == host_seat:
+ return None
+ return ["guest", "host"] if guest_seat == 0 else ["host", "guest"]
+
+
+def _from_seats(video_path, start, end, size, out_dir, face_map, segments, samples) -> tuple[Optional[list[dict]], str]:
+ frames = list(_sample_frames(video_path, start, end, samples))
+ if not frames:
+ return None, "podcli could not read frames from this clip."
+ found = [(t, frame, _find_faces(frame)) for t, frame in frames]
+ width = frames[0][1].shape[1]
+ seats = seats_from_frames([[f["cx"] for f in faces] for _, _, faces in found], width)
+ if seats is None:
+ return None, "Only one person is on screen in this clip."
+
+ by_seat: list[list[dict]] = [[], []]
+ for t, frame, faces in found:
+ for face in faces:
+ gaps = [abs(face["cx"] - seat) for seat in seats]
+ nearest = gaps.index(min(gaps))
+ if gaps[nearest] <= width * _SEAT_REACH:
+ by_seat[nearest].append(_candidate(frame, face, t))
+ picks = [_best(c) for c in by_seat]
+ if not all(picks):
+ return None, "One of the two people never shows a clear face in this clip."
+
+ roles = _seat_roles(face_map, segments, start, end) or [None, None]
+ people = []
+ for i, (side, pick) in enumerate(zip(("left", "right"), picks)):
+ panel = _write_panel(pick, size, os.path.join(out_dir, f"seat_{side}.jpg"))
+ people.append({"side": side, "role": roles[i], "from": "seats", "source_time": pick["time"], **panel})
+ return people, ""
+
+
+def _from_multicam(video_path, start, end, size, out_dir, samples) -> tuple[Optional[list[dict]], str]:
+ import cv2
+ from services import multicam
+
+ session = multicam.session_for_render(video_path)
+ if session is None:
+ return None, ""
+ count = max(4, samples // 4)
+ times = [start + (i + 0.5) * (end - start) / count for i in range(count)]
+ people = []
+ with tempfile.TemporaryDirectory(prefix="podcli_pair_") as stills:
+ for person in session.people:
+ candidates = []
+ for i, t in enumerate(times):
+ tl = multicam.render_to_timeline(session, t)
+ if tl is None:
+ continue
+ still = multicam.person_still(session, person.id, tl, Path(stills) / f"{person.id}_{i}.jpg")
+ frame = cv2.imread(still["path"]) if still else None
+ if frame is None:
+ continue
+ faces = _find_faces(frame)
+ if faces:
+ pick = _candidate(frame, max(faces, key=lambda f: f["fw"]), round(t, 2))
+ candidates.append({**pick, "still": still})
+ best = _best(candidates)
+ if best:
+ people.append((person, best))
+ if len(people) < 2:
+ return None, "The multicam edit behind this video shows fewer than two clear faces in this clip."
+
+ guests = [p for p in people if p[0].role == "guest"]
+ hosts = [p for p in people if p[0].role != "guest"]
+ chosen = [guests[0], hosts[0]] if guests and hosts else people[:2]
+ out = []
+ for i, (person, pick) in enumerate(chosen):
+ side = ("left", "right")[i]
+ panel = _write_panel(pick, size, os.path.join(out_dir, f"camera_{side}.jpg"))
+ out.append({
+ "side": side, "role": person.role, "name": person.name, "from": "multicam",
+ "source_time": pick["time"], "camera": pick["still"]["camera"],
+ "camera_time": pick["still"]["camera_time"], **panel,
+ })
+ return out, ""
+
+
+def _arrange(people: list[dict], swap: bool) -> list[dict]:
+ """Guest left and host right when the roles are known; footage order when not. Then swap."""
+ ordered = list(people)
+ if [p.get("role") for p in ordered] == ["host", "guest"]:
+ ordered.reverse()
+ if swap:
+ ordered.reverse()
+ return [{**p, "side": side} for p, side in zip(ordered, ("left", "right"))]
+
+
+def pick_pair(
+ video_path: Optional[str],
+ start_second: Optional[float],
+ end_second: Optional[float],
+ output_dir: str,
+ panel_size: tuple[int, int],
+ *,
+ left_image: Optional[str] = None,
+ right_image: Optional[str] = None,
+ swap: bool = False,
+ face_map: Optional[dict] = None,
+ segments: Optional[list[dict]] = None,
+ samples: int = SAMPLES,
+) -> dict:
+ """Two portrait panels for the pair layout, or the reason there are none.
+
+ Returns {"layout": "pair", "people": [left, right], "roles": ...} where
+ each person records the panel path, the source second its face came from,
+ and how podcli knew who it was. Without two distinct people it returns
+ {"layout": "single", "people": [], "reason": ...}.
+ """
+ os.makedirs(output_dir, exist_ok=True)
+
+ if left_image or right_image:
+ if not (left_image and right_image):
+ raise ValueError("pass both a left and a right image, or neither")
+ for path in (left_image, right_image):
+ if not os.path.exists(path):
+ raise ValueError(f"image not found: {path}")
+ people = _arrange(_from_images(left_image, right_image, panel_size, output_dir), swap)
+ return {"layout": "pair", "people": people, "roles": "chosen", "swapped": swap}
+
+ if not video_path or not os.path.exists(video_path):
+ return _single("the two-person layout needs the clip's source video")
+ if start_second is None or end_second is None or end_second <= start_second:
+ return _single("the two-person layout needs the clip's start and end")
+
+ people, camera_reason = _from_multicam(video_path, start_second, end_second, panel_size, output_dir, samples)
+ roles = "multicam"
+ if people is None:
+ people, seat_reason = _from_seats(
+ video_path, start_second, end_second, panel_size, output_dir, face_map, segments, samples,
+ )
+ roles = "talk_time" if people and people[0]["role"] else "footage_order"
+ if people is None:
+ reasons = " ".join(r for r in (camera_reason, seat_reason) if r)
+ return _single(f"{reasons} The thumbnail uses the single-face layout.")
+ return {"layout": "pair", "people": _arrange(people, swap), "roles": roles, "swapped": swap}
diff --git a/backend/services/transcribe_runs.py b/backend/services/transcribe_runs.py
new file mode 100644
index 0000000..f5ad740
--- /dev/null
+++ b/backend/services/transcribe_runs.py
@@ -0,0 +1,85 @@
+"""Per-run receipts for long transcriptions, keyed by {file fingerprint,
+engine, model, language}, so a restarted job can resume instead of starting
+over: re-uploading a file to AssemblyAI or re-decoding chunks already done.
+
+A receipt is just a JSON file written atomically (temp + rename) under a
+directory named for the run's key. Nothing here assumes what's inside a
+receipt; each engine decides its own shape (e.g. AssemblyAI stores
+transcript_id, the omnilingual chunker stores a chunk's words/segments).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+from typing import Any, Optional
+
+_SAFE_KEY = re.compile(r"[^a-zA-Z0-9_.-]+")
+
+
+def fingerprint_file(file_path: str) -> str:
+ """Matches transcript_packer.compute_cache_hash: sha256 of the first
+ 10MB + size, 16 hex chars. Same file -> same fingerprint regardless of
+ which engine/run reads it."""
+ size = os.path.getsize(file_path)
+ h = hashlib.sha256()
+ remaining = 10 * 1024 * 1024
+ with open(file_path, "rb") as f:
+ while remaining > 0:
+ chunk = f.read(min(1 << 20, remaining))
+ if not chunk:
+ break
+ h.update(chunk)
+ remaining -= len(chunk)
+ h.update(f"size:{size}".encode())
+ return h.hexdigest()[:16]
+
+
+def run_key(file_path: str, engine: str, model_size: Optional[str], language: Optional[str]) -> str:
+ fp = fingerprint_file(file_path)
+ model_part = _SAFE_KEY.sub("-", model_size or "default")
+ lang_part = _SAFE_KEY.sub("-", (language or "auto").lower())
+ engine_part = _SAFE_KEY.sub("-", engine)
+ return f"{fp}-{engine_part}-{model_part}-{lang_part}"
+
+
+def run_dir(cache_root: str, file_path: str, engine: str, model_size: Optional[str], language: Optional[str]) -> str:
+ return os.path.join(cache_root, "transcribe_runs", run_key(file_path, engine, model_size, language))
+
+
+def read_receipt(directory: str, name: str) -> Optional[dict]:
+ path = os.path.join(directory, name)
+ if not os.path.exists(path):
+ return None
+ try:
+ with open(path, encoding="utf-8") as f:
+ return json.load(f)
+ except (OSError, json.JSONDecodeError):
+ return None
+
+
+def write_receipt(directory: str, name: str, data: Any) -> None:
+ os.makedirs(directory, exist_ok=True)
+ path = os.path.join(directory, name)
+ tmp_path = f"{path}.{os.getpid()}.tmp"
+ with open(tmp_path, "w", encoding="utf-8") as f:
+ json.dump(data, f)
+ os.replace(tmp_path, path)
+
+
+def clear_run(directory: str) -> None:
+ """Remove every receipt for a run (e.g. once the final transcript is
+ assembled, or to force a clean restart)."""
+ if not os.path.isdir(directory):
+ return
+ for name in os.listdir(directory):
+ try:
+ os.unlink(os.path.join(directory, name))
+ except OSError:
+ pass
+ try:
+ os.rmdir(directory)
+ except OSError:
+ pass
diff --git a/backend/services/transcript_packer.py b/backend/services/transcript_packer.py
index 726704c..31a8f35 100644
--- a/backend/services/transcript_packer.py
+++ b/backend/services/transcript_packer.py
@@ -68,11 +68,12 @@ def legacy_md5_cache_path(video_path: str) -> str:
return os.path.join(_legacy_cache_dir(), hashlib.md5(raw.encode()).hexdigest() + ".json")
-def _engine_cache_suffix() -> str:
- """Namespace the cache by engine so a whisper.cpp run doesn't reuse a
- whisper-py transcript (their timings/word splits differ). whisper-py keeps
- the bare filename, which the TS transcript cache also writes."""
- return engine_cache_suffix(os.environ.get("PODCLI_ENGINE", "whisper-py"))
+def _effective_engine(engine: str | None) -> str | None:
+ """None means "not told": fall back to the process-wide engine (the
+ same default every pre-existing caller relied on), not to whisper-py,
+ so an unset PODCLI_ENGINE=whispercpp request still lands in its own
+ namespace."""
+ return engine if engine is not None else os.environ.get("PODCLI_ENGINE", "whisper-py")
def engine_cache_suffix(engine: str | None) -> str:
@@ -81,38 +82,129 @@ def engine_cache_suffix(engine: str | None) -> str:
return "-whispercpp"
if engine == "assemblyai":
return "-assemblyai"
+ if engine == "omnilingual":
+ return "-omnilingual"
return ""
-def transcript_json_path(cache_hash: str) -> str:
- return os.path.join(_transcripts_cache_dir(), f"{cache_hash}{_engine_cache_suffix()}.json")
+_LANGUAGE_RE = re.compile(r"[^a-z0-9-]+")
-def load_cached_transcript_for_video(video_path: str) -> dict[str, Any] | None:
+def sanitize_language(language: str | None) -> str:
+ """Matches src/services/transcript-cache.ts sanitizeLanguage exactly:
+ lowercase, then strip anything outside [a-z0-9-]. The tag can come from
+ caller-supplied input, so without this a value carrying a path separator
+ would land in the cache filename unescaped."""
+ value = (language or "").strip().lower()
+ return _LANGUAGE_RE.sub("", value)
+
+
+def cache_key_suffix(engine: str | None = None, model: str | None = None, language: str | None = None) -> str:
+ """Port of src/services/transcript-cache.ts keySuffix: engine + model +
+ language. base model and auto/empty language contribute no suffix, so a
+ cache written before model/language were tracked still reads back under
+ the same key. Every Python reader/writer of the transcript cache has to
+ produce this exact suffix for a given (engine, model, language), or a
+ transcript one side writes is invisible to the other."""
+ engine_part = engine_cache_suffix(_effective_engine(engine))
+ model_norm = (model or "").strip().lower()
+ model_part = f"-m{model_norm}" if model_norm and model_norm != "base" else ""
+ language_norm = sanitize_language(language)
+ language_part = f"-l{language_norm}" if language_norm and language_norm != "auto" else ""
+ return f"{engine_part}{model_part}{language_part}"
+
+
+def transcript_json_path(
+ cache_hash: str,
+ engine: str | None = None,
+ model: str | None = None,
+ language: str | None = None,
+) -> str:
+ return os.path.join(
+ _transcripts_cache_dir(), f"{cache_hash}{cache_key_suffix(engine, model, language)}.json"
+ )
+
+
+# A suffix can only continue with "-m<model>" and/or "-l<language>" (see
+# cache_key_suffix). No other engine's suffix starts with "m" or "l", so
+# anchoring on this is enough to never cross into a different engine's files.
+_CACHE_SUFFIX_TAIL_RE = r"(?:-m[a-z0-9]+)?(?:-l[a-z0-9-]+)?\.json$"
+
+
+def find_cached_transcript_path(cache_hash: str, engine: str | None = None) -> str | None:
+ """Locate a cached transcript for this file hash when the caller does not
+ know which model/language produced it, e.g. a reel or face-map lookup
+ that only has a video path, not the transcribe request that created the
+ cache. Deterministic rule: prefer the engine-only key (base model, auto
+ language, what every pre-model/language cache, and most base-model
+ runs, write); otherwise the most recently modified matching file for
+ this hash+engine, i.e. the transcript this file most recently produced.
+ """
+ cache_dir = _transcripts_cache_dir()
+ engine_suffix = engine_cache_suffix(_effective_engine(engine))
+ exact = os.path.join(cache_dir, f"{cache_hash}{engine_suffix}.json")
+ if os.path.exists(exact):
+ return exact
+ if not os.path.isdir(cache_dir):
+ return None
+ pattern = re.compile(rf"^{re.escape(cache_hash)}{re.escape(engine_suffix)}{_CACHE_SUFFIX_TAIL_RE}")
+ candidates = [
+ os.path.join(cache_dir, name) for name in os.listdir(cache_dir) if pattern.match(name)
+ ]
+ if not candidates:
+ return None
+ return max(candidates, key=os.path.getmtime)
+
+
+def load_cached_transcript_for_video(
+ video_path: str,
+ engine: str | None = None,
+ model: str | None = None,
+ language: str | None = None,
+) -> dict[str, Any] | None:
cache_hash = compute_cache_hash(video_path)
- canonical = transcript_json_path(cache_hash)
- if os.path.exists(canonical):
- with open(canonical, encoding="utf-8") as f:
+ if model is not None or language is not None:
+ # The caller knows exactly what it's asking for: read that key only,
+ # never silently substitute a different model/language's transcript.
+ canonical = transcript_json_path(cache_hash, engine=engine, model=model, language=language)
+ if os.path.exists(canonical):
+ with open(canonical, encoding="utf-8") as f:
+ return json.load(f)
+ return None
+
+ found = find_cached_transcript_path(cache_hash, engine=engine)
+ if found:
+ with open(found, encoding="utf-8") as f:
return json.load(f)
# The legacy md5 cache predates the engine split and only ever held
# whisper-py output, so don't let a whisper.cpp run adopt it under its own
# namespace.
- if _engine_cache_suffix() == "":
+ if engine_cache_suffix(_effective_engine(engine)) == "":
legacy = legacy_md5_cache_path(video_path)
if os.path.exists(legacy):
with open(legacy, encoding="utf-8") as f:
data = json.load(f)
- save_cached_transcript_for_video(video_path, data)
+ save_cached_transcript_for_video(video_path, data, engine=engine)
return data
return None
-def save_cached_transcript_for_video(video_path: str, data: dict[str, Any]) -> str:
+def save_cached_transcript_for_video(
+ video_path: str,
+ data: dict[str, Any],
+ engine: str | None = None,
+ model: str | None = None,
+ language: str | None = None,
+) -> str:
cache_hash = compute_cache_hash(video_path)
- path = transcript_json_path(cache_hash)
+ path = transcript_json_path(cache_hash, engine=engine, model=model, language=language)
os.makedirs(_transcripts_cache_dir(), exist_ok=True)
- with open(path, "w", encoding="utf-8") as f:
+ # Write-then-rename: a reader never observes a half-written cache file,
+ # matching src/services/transcript-cache.ts's set().
+ tmp_path = f"{path}.{os.getpid()}.tmp"
+ with open(tmp_path, "w", encoding="utf-8") as f:
json.dump(data, f)
+ os.replace(tmp_path, path)
return path
diff --git a/backend/services/transcript_parser.py b/backend/services/transcript_parser.py
index e34fa4c..07b6960 100644
--- a/backend/services/transcript_parser.py
+++ b/backend/services/transcript_parser.py
@@ -14,6 +14,16 @@ import json
from typing import List, Dict, Any, Optional
+def _clean_caption_text(text: str) -> str:
+ """Strip markup that isn't spoken text: HTML-like tags and ASS/SSA
+ override blocks such as {\\an8} that some SRT/VTT exports carry over.
+ Left in place, they get imported as literal words.
+ """
+ text = re.sub(r'<[^>]+>', '', text)
+ text = re.sub(r'\{[^}]*\}', '', text)
+ return re.sub(r'\s+', ' ', text).strip()
+
+
def parse_srt_timestamp(ts: str) -> float:
"""Convert SRT timestamp HH:MM:SS,mmm to seconds."""
ts = ts.strip().replace(",", ".")
@@ -34,7 +44,12 @@ def parse_vtt_timestamp(ts: str) -> float:
return 0.0
-def parse_srt(text: str, total_duration: Optional[float] = None, time_adjust: float = 0.0) -> Dict[str, Any]:
+def parse_srt(
+ text: str,
+ total_duration: Optional[float] = None,
+ time_adjust: float = 0.0,
+ language: Optional[str] = None,
+) -> Dict[str, Any]:
"""
Parse an SRT subtitle file into structured data with word-level timestamps.
@@ -71,8 +86,7 @@ def parse_srt(text: str, total_duration: Optional[float] = None, time_adjust: fl
# End of block
text_content = " ".join(current_text_lines).strip()
# Strip HTML-like tags
- text_content = re.sub(r'<[^>]+>', '', text_content)
- text_content = re.sub(r'\s+', ' ', text_content).strip()
+ text_content = _clean_caption_text(text_content)
if text_content:
blocks.append({
"start": current_start,
@@ -86,8 +100,7 @@ def parse_srt(text: str, total_duration: Optional[float] = None, time_adjust: fl
# Handle last block if file doesn't end with blank line
if state == "text" and current_text_lines:
text_content = " ".join(current_text_lines).strip()
- text_content = re.sub(r'<[^>]+>', '', text_content)
- text_content = re.sub(r'\s+', ' ', text_content).strip()
+ text_content = _clean_caption_text(text_content)
if text_content:
blocks.append({
"start": current_start,
@@ -98,10 +111,15 @@ def parse_srt(text: str, total_duration: Optional[float] = None, time_adjust: fl
if not blocks:
return {"error": "No subtitle blocks found in SRT"}
- return _blocks_to_result(blocks, total_duration, time_adjust, fmt="srt")
+ return _blocks_to_result(blocks, total_duration, time_adjust, fmt="srt", language=language)
-def parse_vtt(text: str, total_duration: Optional[float] = None, time_adjust: float = 0.0) -> Dict[str, Any]:
+def parse_vtt(
+ text: str,
+ total_duration: Optional[float] = None,
+ time_adjust: float = 0.0,
+ language: Optional[str] = None,
+) -> Dict[str, Any]:
"""
Parse a WebVTT subtitle file into structured data with word-level timestamps.
@@ -138,8 +156,7 @@ def parse_vtt(text: str, total_duration: Optional[float] = None, time_adjust: fl
# Save previous block
if in_text and current_text_lines:
text_content = " ".join(current_text_lines).strip()
- text_content = re.sub(r'<[^>]+>', '', text_content)
- text_content = re.sub(r'\s+', ' ', text_content).strip()
+ text_content = _clean_caption_text(text_content)
if text_content:
blocks.append({
"start": current_start,
@@ -156,8 +173,7 @@ def parse_vtt(text: str, total_duration: Optional[float] = None, time_adjust: fl
if in_text:
if line_stripped == "":
text_content = " ".join(current_text_lines).strip()
- text_content = re.sub(r'<[^>]+>', '', text_content)
- text_content = re.sub(r'\s+', ' ', text_content).strip()
+ text_content = _clean_caption_text(text_content)
if text_content:
blocks.append({
"start": current_start,
@@ -173,8 +189,7 @@ def parse_vtt(text: str, total_duration: Optional[float] = None, time_adjust: fl
# Handle last block
if in_text and current_text_lines:
text_content = " ".join(current_text_lines).strip()
- text_content = re.sub(r'<[^>]+>', '', text_content)
- text_content = re.sub(r'\s+', ' ', text_content).strip()
+ text_content = _clean_caption_text(text_content)
if text_content:
blocks.append({
"start": current_start,
@@ -185,10 +200,16 @@ def parse_vtt(text: str, total_duration: Optional[float] = None, time_adjust: fl
if not blocks:
return {"error": "No subtitle blocks found in VTT"}
- return _blocks_to_result(blocks, total_duration, time_adjust, fmt="vtt")
+ return _blocks_to_result(blocks, total_duration, time_adjust, fmt="vtt", language=language)
-def _blocks_to_result(blocks: List[Dict], total_duration: Optional[float], time_adjust: float, fmt: str) -> Dict[str, Any]:
+def _blocks_to_result(
+ blocks: List[Dict],
+ total_duration: Optional[float],
+ time_adjust: float,
+ fmt: str,
+ language: Optional[str] = None,
+) -> Dict[str, Any]:
"""Convert parsed subtitle blocks into the standard output format."""
all_words = []
segments = []
@@ -227,7 +248,10 @@ def _blocks_to_result(blocks: List[Dict], total_duration: Optional[float], time_
"words": all_words,
"segments": segments,
"duration": round(duration, 2),
- "language": "en",
+ # These formats carry no language info of their own; caller-supplied
+ # language wins, "und" (undetermined) otherwise. Previously this
+ # always claimed "en", which is simply wrong for anything else.
+ "language": language or "und",
"speakers": [],
"speaker_segments": [],
"imported": True,
@@ -235,7 +259,12 @@ def _blocks_to_result(blocks: List[Dict], total_duration: Optional[float], time_
}
-def detect_and_parse(text: str, total_duration: Optional[float] = None, time_adjust: float = 0.0) -> Dict[str, Any]:
+def detect_and_parse(
+ text: str,
+ total_duration: Optional[float] = None,
+ time_adjust: float = 0.0,
+ language: Optional[str] = None,
+) -> Dict[str, Any]:
"""
Auto-detect transcript format and parse accordingly.
@@ -249,12 +278,12 @@ def detect_and_parse(text: str, total_duration: Optional[float] = None, time_adj
# VTT detection
if stripped.startswith("WEBVTT"):
- return parse_vtt(text, total_duration=total_duration, time_adjust=time_adjust)
+ return parse_vtt(text, total_duration=total_duration, time_adjust=time_adjust, language=language)
# SRT detection: first line is a digit, second line contains -->
lines = stripped.split("\n", 3)
if len(lines) >= 2 and re.match(r'^\d+$', lines[0].strip()) and '-->' in lines[1]:
- return parse_srt(text, total_duration=total_duration, time_adjust=time_adjust)
+ return parse_srt(text, total_duration=total_duration, time_adjust=time_adjust, language=language)
# JSON detection
if stripped.startswith("{") or stripped.startswith("["):
@@ -276,7 +305,9 @@ def detect_and_parse(text: str, total_duration: Optional[float] = None, time_adj
"words": words,
"segments": segments_list,
"duration": round(duration, 2),
- "language": "en",
+ "language": (data.get("language") if isinstance(data, dict) else None)
+ or language
+ or "und",
"speakers": [],
"speaker_segments": [],
"imported": True,
@@ -286,7 +317,9 @@ def detect_and_parse(text: str, total_duration: Optional[float] = None, time_adj
pass # Fall through to speaker format
# Default: speaker format
- return parse_speaker_transcript(text, total_duration=total_duration, time_adjust=time_adjust)
+ return parse_speaker_transcript(
+ text, total_duration=total_duration, time_adjust=time_adjust, language=language
+ )
def parse_timestamp(ts: str) -> float:
@@ -299,7 +332,12 @@ def parse_timestamp(ts: str) -> float:
return 0.0
-def parse_speaker_transcript(raw_text: str, total_duration: Optional[float] = None, time_adjust: float = 0.0) -> Dict[str, Any]:
+def parse_speaker_transcript(
+ raw_text: str,
+ total_duration: Optional[float] = None,
+ time_adjust: float = 0.0,
+ language: Optional[str] = None,
+) -> Dict[str, Any]:
"""
Parse a speaker-labeled transcript into structured data with word-level timestamps.
@@ -309,6 +347,8 @@ def parse_speaker_transcript(raw_text: str, total_duration: Optional[float] = No
raw_text: The raw transcript text
total_duration: Total duration of the podcast in seconds
time_adjust: Seconds to add/subtract from all timestamps (e.g., -1.0 to shift 1s earlier)
+ language: Language of the transcript, if known. Defaults to "und" (undetermined):
+ this format carries no language info of its own to detect it from.
"""
lines = raw_text.strip().split("\n")
@@ -434,10 +474,17 @@ def parse_speaker_transcript(raw_text: str, total_duration: Optional[float] = No
"words": all_words,
"segments": segments,
"duration": round(duration, 2),
- "language": "en",
+ "language": language or "und",
"speakers": speakers_list,
+ # words/segments above apply time_adjust and clamp to 0; this has to
+ # match or a speaker's segment boundaries drift out of sync with the
+ # words and segments supposedly inside them.
"speaker_segments": [
- {"speaker": b["speaker"], "start": b["start"], "end": b["end"]}
+ {
+ "speaker": b["speaker"],
+ "start": round(max(0, b["start"] + time_adjust), 3),
+ "end": round(max(0, b["end"] + time_adjust), 3),
+ }
for b in blocks
],
"imported": True,
diff --git a/backend/services/transcription.py b/backend/services/transcription.py
index e6859e9..fc13cb8 100644
--- a/backend/services/transcription.py
+++ b/backend/services/transcription.py
@@ -8,7 +8,10 @@ Produces word-level timestamps with speaker labels by:
"""
import json
+import functools
+import hashlib
import http.client
+import importlib.util
import os
import shutil
import sys
@@ -19,7 +22,7 @@ import urllib.parse
import urllib.request
from typing import Optional, Callable
-from services.engines import is_assemblyai_engine, normalize_engine
+from services.engines import is_assemblyai_engine, is_omnilingual_engine, normalize_engine
def _managed_home() -> str:
@@ -48,9 +51,16 @@ def _whispercpp_cli() -> Optional[str]:
return hermetic if os.path.exists(hermetic) else None
+
+# "large" alone doesn't name a real ggml file (upstream ships v1/v2/v3/v3-turbo
+# builds); provisioning always fetches large-v3, so resolve the same way here.
+_WHISPERCPP_MODEL_ALIASES = {"large": "large-v3"}
+
+
def _whispercpp_model(model_size: str) -> str:
+ resolved = _WHISPERCPP_MODEL_ALIASES.get(model_size, model_size)
return os.environ.get("PODCLI_WHISPERCPP_MODEL") or os.path.join(
- _managed_home(), "models", f"ggml-{model_size}.bin"
+ _managed_home(), "models", f"ggml-{resolved}.bin"
)
@@ -58,6 +68,33 @@ def _whispercpp_ready(model_size: str) -> bool:
return _whispercpp_cli() is not None and os.path.exists(_whispercpp_model(model_size))
+def _omnilingual_model() -> str:
+ return os.environ.get("PODCLI_OMNILINGUAL_MODEL") or os.path.join(
+ _managed_home(), "models", "omnilingual", "model.int8.onnx"
+ )
+
+
+def _omnilingual_tokens() -> str:
+ return os.environ.get("PODCLI_OMNILINGUAL_TOKENS") or os.path.join(
+ _managed_home(), "models", "omnilingual", "tokens.txt"
+ )
+
+
+def _omnilingual_ready() -> bool:
+ return os.path.exists(_omnilingual_model()) and os.path.exists(_omnilingual_tokens())
+
+
+def _omnilingual_model_fingerprint(model_path: str) -> str:
+ """Cheap stand-in for hashing the whole model file: size+mtime changes
+ whenever the file is replaced (re-provisioned, a different quantization
+ swapped in), without reading potentially hundreds of MB on every run."""
+ try:
+ st = os.stat(model_path)
+ return f"{st.st_size}-{int(st.st_mtime)}"
+ except OSError:
+ return "unknown"
+
+
def _whisper_threads() -> int:
raw = os.environ.get("PODCLI_WHISPER_THREADS", "").strip()
try:
@@ -97,6 +134,62 @@ def _transcribe_with_whispercpp(file_path, model_size, language, progress_callba
return result
+def _transcribe_with_omnilingual(file_path, progress_callback, wav_path=None):
+ from config.paths import paths
+ from services import transcribe_runs
+ from services import transcription_omnilingual as omni
+
+ if progress_callback:
+ progress_callback(10, "Transcribing with omnilingual...")
+
+ model = _omnilingual_model()
+ tokens = _omnilingual_tokens()
+ # Fetch only into podcli's own model folder; a custom model path is the
+ # caller's to provide, and must never trigger a 1 GB download beside it.
+ managed_dir = os.path.join(_managed_home(), "models", "omnilingual")
+ if os.path.dirname(model) == managed_dir and os.path.dirname(tokens) == managed_dir:
+ omni.ensure_model(managed_dir, progress_callback)
+ if not os.path.exists(model) or not os.path.exists(tokens):
+ raise FileNotFoundError(
+ f"omnilingual model not found: {model}. "
+ "Set PODCLI_OMNILINGUAL_MODEL/PODCLI_OMNILINGUAL_TOKENS or run provisioning."
+ )
+
+ # Resumable: a crash partway through a long file loses at most the
+ # window that was decoding, not the whole run. A rerun with the same
+ # file/model/language skips every window that already has a receipt.
+ #
+ # The key has to change whenever a resumed window's receipt would no
+ # longer match what a fresh decode produces: a different model file (so
+ # fold in its size+mtime, cheap to stat vs. hashing the whole model), or
+ # a change to the window/context constants a receipt's timestamps are
+ # only valid under (resuming under new constants would silently splice
+ # old-geometry windows into a new-geometry transcript).
+ model_fp = _omnilingual_model_fingerprint(model)
+ run_dir = transcribe_runs.run_dir(
+ paths["cache"], file_path, "omnilingual",
+ model_size=f"int8-{model_fp}-w{omni.WINDOW_SECONDS}-c{omni.CONTEXT_SECONDS}",
+ language="und",
+ )
+
+ def window_progress(pct, msg):
+ if progress_callback:
+ progress_callback(10 + int(pct * 0.4), msg)
+
+ result = omni.transcribe_file(
+ file_path,
+ model_path=model,
+ tokens_path=tokens,
+ wav_path=wav_path,
+ threads=_whisper_threads(),
+ run_dir=run_dir,
+ progress_callback=window_progress,
+ )
+ if progress_callback:
+ progress_callback(50, "Transcription complete")
+ return result
+
+
def _assemblyai_base_url() -> str:
region = os.environ.get("ASSEMBLYAI_REGION", "").strip().lower()
if region == "eu":
@@ -104,6 +197,16 @@ def _assemblyai_base_url() -> str:
return "https://api.assemblyai.com/v2"
+class AssemblyAIHTTPError(RuntimeError):
+ """Carries the HTTP status so callers can tell a bad/expired resource
+ (4xx, e.g. the transcript a resumed receipt points at no longer exists)
+ from a transient server error, without re-parsing the message string."""
+
+ def __init__(self, message: str, status: int):
+ super().__init__(message)
+ self.status = status
+
+
def _assemblyai_json_request(method: str, url: str, api_key: str, payload: Optional[dict], timeout: int) -> dict:
body = None if payload is None else json.dumps(payload).encode("utf-8")
headers = {"Authorization": api_key}
@@ -117,8 +220,9 @@ def _assemblyai_json_request(method: str, url: str, api_key: str, payload: Optio
return json.loads(res.read().decode("utf-8"))
except urllib.error.HTTPError as e:
detail = e.read().decode("utf-8", errors="replace")
- last_error = RuntimeError(
- f"AssemblyAI request failed: method={method} url={url} status={e.code} body={detail}"
+ last_error = AssemblyAIHTTPError(
+ f"AssemblyAI request failed: method={method} url={url} status={e.code} body={detail}",
+ status=e.code,
)
if e.code not in (408, 409, 425, 429) and e.code < 500:
raise last_error from e
@@ -288,55 +392,102 @@ def _assemblyai_result(data: dict) -> dict:
def _transcribe_with_assemblyai(file_path, language, enable_diarization, num_speakers, progress_callback):
+ from config.paths import paths
+ from services import transcribe_runs
+
api_key = os.environ.get("ASSEMBLYAI_API_KEY", "").strip()
if not api_key:
raise RuntimeError("ASSEMBLYAI_API_KEY is required when PODCLI_ENGINE=assemblyai")
base_url = _assemblyai_base_url()
- if progress_callback:
- progress_callback(10, "Uploading media to AssemblyAI...")
- upload_url = _assemblyai_upload(file_path, api_key, base_url)
-
- payload = {
- "audio_url": upload_url,
- "punctuate": True,
- "format_text": True,
- "speaker_labels": bool(enable_diarization),
- }
- if language:
- payload["language_code"] = language
- else:
- payload["language_detection"] = True
- if num_speakers:
- payload["speakers_expected"] = num_speakers
+ region = os.environ.get("ASSEMBLYAI_REGION", "").strip().lower() or "us"
+ key_hash = hashlib.sha256(api_key.encode("utf-8")).hexdigest()[:8]
+ # A restarted job (crash, kill, machine reboot) must resume by polling
+ # the transcript it already started, not re-upload the file and pay for
+ # a second AssemblyAI job. The receipt is keyed by exactly what affects
+ # the AssemblyAI request (language, diarization, speaker count, region,
+ # which account) so a different request for the same file never adopts
+ # someone else's job: a region or key mismatch would otherwise poll a
+ # transcript ID that belongs to a different account/API surface.
+ directory = transcribe_runs.run_dir(
+ paths["cache"], file_path, "assemblyai",
+ model_size=f"diar{int(bool(enable_diarization))}-spk{num_speakers or 0}-{region}-{key_hash}",
+ language=language,
+ )
- if progress_callback:
- progress_callback(20, "Starting AssemblyAI transcript...")
- started = _assemblyai_json_request("POST", f"{base_url}/transcript", api_key, payload, 60)
- transcript_id = started.get("id")
- if not transcript_id:
- raise RuntimeError(f"AssemblyAI transcript response missing id: body={started}")
-
- url = f"{base_url}/transcript/{transcript_id}"
- for _ in range(720):
- data = _assemblyai_json_request("GET", url, api_key, None, 60)
- status = data.get("status")
- if status == "completed":
+ # A resumed transcript_id that 4xxs on the resume GET (e.g. AssemblyAI
+ # deleted it after its retention window, or it belonged to a key that's
+ # since been rotated) is permanently bad: retrying it forever would
+ # never succeed. Clear the receipt and fall through to a fresh upload,
+ # once.
+ for resume_attempt in (True, False):
+ receipt = transcribe_runs.read_receipt(directory, "assemblyai.json") if resume_attempt else None
+ transcript_id = receipt.get("transcript_id") if receipt else None
+ resumed = transcript_id is not None
+
+ if not transcript_id:
if progress_callback:
- progress_callback(50, "AssemblyAI transcription complete")
- return _assemblyai_result(data)
- if status == "error":
- raise RuntimeError(
- f"AssemblyAI transcript failed: transcript_id={transcript_id} error={data.get('error')}"
- )
- if status not in ("queued", "processing"):
- raise RuntimeError(
- f"AssemblyAI transcript returned unknown status: transcript_id={transcript_id} status={status} body={data}"
- )
- if progress_callback:
- progress_callback(30, f"AssemblyAI transcript {status}...")
- time.sleep(5)
- raise TimeoutError(f"AssemblyAI transcript timed out: transcript_id={transcript_id}")
+ progress_callback(10, "Uploading media to AssemblyAI...")
+ upload_url = _assemblyai_upload(file_path, api_key, base_url)
+
+ payload = {
+ "audio_url": upload_url,
+ "punctuate": True,
+ "format_text": True,
+ "speaker_labels": bool(enable_diarization),
+ }
+ if language:
+ payload["language_code"] = language
+ else:
+ payload["language_detection"] = True
+ if num_speakers:
+ payload["speakers_expected"] = num_speakers
+
+ if progress_callback:
+ progress_callback(20, "Starting AssemblyAI transcript...")
+ started = _assemblyai_json_request("POST", f"{base_url}/transcript", api_key, payload, 60)
+ transcript_id = started.get("id")
+ if not transcript_id:
+ raise RuntimeError(f"AssemblyAI transcript response missing id: body={started}")
+ # Written before the poll loop starts: if the process dies mid-poll,
+ # the next run finds this and resumes instead of re-uploading.
+ transcribe_runs.write_receipt(directory, "assemblyai.json", {"transcript_id": transcript_id})
+ elif progress_callback:
+ progress_callback(20, f"Resuming AssemblyAI transcript {transcript_id}...")
+
+ url = f"{base_url}/transcript/{transcript_id}"
+ try:
+ for _ in range(720):
+ data = _assemblyai_json_request("GET", url, api_key, None, 60)
+ status = data.get("status")
+ if status == "completed":
+ if progress_callback:
+ progress_callback(50, "AssemblyAI transcription complete")
+ transcribe_runs.clear_run(directory)
+ return _assemblyai_result(data)
+ if status == "error":
+ transcribe_runs.clear_run(directory)
+ raise RuntimeError(
+ f"AssemblyAI transcript failed: transcript_id={transcript_id} error={data.get('error')}"
+ )
+ if status not in ("queued", "processing"):
+ raise RuntimeError(
+ f"AssemblyAI transcript returned unknown status: transcript_id={transcript_id} status={status} body={data}"
+ )
+ if progress_callback:
+ progress_callback(30, f"AssemblyAI transcript {status}...")
+ time.sleep(5)
+ raise TimeoutError(f"AssemblyAI transcript timed out: transcript_id={transcript_id}")
+ except AssemblyAIHTTPError as e:
+ if resumed and 400 <= e.status < 500:
+ transcribe_runs.clear_run(directory)
+ if progress_callback:
+ progress_callback(
+ 10, f"Saved AssemblyAI transcript {transcript_id} is gone (status {e.status}); re-uploading..."
+ )
+ continue
+ raise
+ raise RuntimeError("AssemblyAI resume retry exhausted") # unreachable: loop always returns or raises
def _attach_speakers_and_faces(
@@ -456,6 +607,79 @@ def _attach_speakers_and_faces(
return base
[email protected]_cache(maxsize=1)
+def _whisper_py_available() -> bool:
+ """Cheap availability check for the openai-whisper package.
+
+ `import whisper` doesn't just find the module, it executes it, which
+ transitively imports torch and costs hundreds of ms to seconds.
+ resolve_engine_info runs this on every single transcribe request (it's
+ a cache-key prediction, not an actual transcription), so it uses
+ find_spec, which only locates the module and never runs its code, and
+ memoizes the result, since whether the package is installed can't
+ change within a process's lifetime.
+ """
+ try:
+ return importlib.util.find_spec("whisper") is not None
+ except (ImportError, ValueError):
+ # find_spec itself can raise if "whisper" is already in sys.modules
+ # under a module object missing __spec__ (a malformed stand-in, not
+ # a real install state). Fall back to treating that as available
+ # rather than crashing a cache-key prediction over it.
+ return True
+
+
+def _resolve_whisper_py_fallback(requested: Optional[str], model_size: str, probe_model: bool):
+ """Single source of truth for "should an unset whisper-py request fall
+ back to whispercpp instead": the one decision resolve_engine_info and
+ transcribe_file both have to make the same way, or a cache key built
+ from one's answer misses the transcript the other actually wrote.
+
+ probe_model=False (resolve_engine_info, a cache-key prediction made on
+ every request, so it can't afford to load model weights, or even
+ import whisper for real, see _whisper_py_available) only checks
+ whether the package is installed. probe_model=True (transcribe_file,
+ about to actually transcribe) does the real import and loads the
+ model, catching the one case the cheap check can't: the package is
+ present but load_model fails. That gap means a cache lookup built from
+ resolve_engine_info's prediction can still miss once in that rarer
+ case, before the result is written under the engine transcribe_file
+ actually ran with.
+
+ Returns (fall_back_to_whispercpp, loaded_model_or_None, error_or_None).
+ """
+ if not probe_model:
+ if _whisper_py_available():
+ return False, None, None
+ if not requested and _whispercpp_ready(model_size):
+ return True, None, None
+ return False, None, None
+
+ try:
+ import whisper
+
+ model = whisper.load_model(model_size)
+ return False, model, None
+ except Exception as e:
+ if not requested and _whispercpp_ready(model_size):
+ return True, None, None
+ return False, None, e
+
+
+def resolve_engine_info(requested: Optional[str], model_size: str = "base") -> dict:
+ """Predict which engine transcribe_file will actually use, without doing
+ any transcription work. Callers cache transcripts by engine; the cache
+ key has to match what transcribe_file resolves to, not the raw request,
+ or an unset engine writes under one key and reads under another.
+ """
+ requested = requested if requested is not None else os.environ.get("PODCLI_ENGINE", "")
+ engine = normalize_engine(requested)
+ if engine != "whisper-py":
+ return {"engine": engine, "model_size": model_size}
+ fall_back, _model, _error = _resolve_whisper_py_fallback(requested, model_size, probe_model=False)
+ return {"engine": "whispercpp" if fall_back else "whisper-py", "model_size": model_size}
+
+
def transcribe_file(
file_path: str,
model_size: str = "base",
@@ -465,6 +689,8 @@ def transcribe_file(
num_speakers: Optional[int] = None,
progress_callback: Optional[Callable[[int, str], None]] = None,
wav_path: Optional[str] = None,
+ start_seconds: Optional[float] = None,
+ duration_seconds: Optional[float] = None,
) -> dict:
"""
Transcribe a video/audio file with word-level timestamps and speaker detection.
@@ -472,6 +698,15 @@ def transcribe_file(
wav_path: optional pre-extracted 16 kHz mono WAV shared across analysis
stages — used by whisper.cpp and diarization instead of re-decoding.
+ start_seconds/duration_seconds: sample mode, transcribe only a window of
+ the source (e.g. to test a language on 40s before committing to a full
+ run) instead of the whole file. The result is marked complete: False and
+ its timestamps are relative to the sample window, not the source;
+ sample_offset_seconds carries where in the source the window started.
+ Diarization and face analysis are skipped: a throwaway sample isn't
+ worth the extra passes, and both would need frame/audio access to the
+ original file that the trimmed clip doesn't carry.
+
Returns:
{
"transcript": str,
@@ -486,35 +721,105 @@ def transcribe_file(
if not os.path.exists(file_path):
raise FileNotFoundError(f"File not found: {file_path}")
+ # Matches backend/main.py's handle_transcribe and both TS transcribe
+ # entry points exactly: a sample is a positive window, not merely a
+ # present key, so start_seconds=0/duration_seconds=0 or an explicit
+ # null both mean "no sample" rather than "sample from t=0".
+ is_sample = (duration_seconds or 0) > 0 or (start_seconds or 0) > 0
+ if is_sample:
+ from services.audio_extract import extract_wav_16k_mono
+
+ sample_wav = extract_wav_16k_mono(
+ file_path,
+ start_seconds=start_seconds or 0.0,
+ duration_seconds=duration_seconds,
+ )
+ try:
+ result = _transcribe_file_inner(
+ sample_wav,
+ model_size=model_size,
+ engine=engine,
+ language=language,
+ enable_diarization=False,
+ num_speakers=num_speakers,
+ progress_callback=progress_callback,
+ wav_path=sample_wav,
+ )
+ finally:
+ try:
+ os.unlink(sample_wav)
+ except OSError:
+ pass
+ result["complete"] = False
+ result["sample_offset_seconds"] = start_seconds or 0.0
+ return result
+
+ return _transcribe_file_inner(
+ file_path,
+ model_size=model_size,
+ engine=engine,
+ language=language,
+ enable_diarization=enable_diarization,
+ num_speakers=num_speakers,
+ progress_callback=progress_callback,
+ wav_path=wav_path,
+ )
+
+
+def _transcribe_file_inner(
+ file_path: str,
+ model_size: str = "base",
+ engine: Optional[str] = None,
+ language: Optional[str] = None,
+ enable_diarization: bool = True,
+ num_speakers: Optional[int] = None,
+ progress_callback: Optional[Callable[[int, str], None]] = None,
+ wav_path: Optional[str] = None,
+) -> dict:
+
requested = engine if engine is not None else os.environ.get("PODCLI_ENGINE", "")
engine = normalize_engine(requested)
use_cpp = engine == "whispercpp"
use_assemblyai = is_assemblyai_engine(engine)
+ use_omnilingual = is_omnilingual_engine(engine)
if use_assemblyai:
base = _transcribe_with_assemblyai(
file_path, language, enable_diarization, num_speakers, progress_callback
)
+ # AssemblyAI does its own diarization (speaker_labels in the request),
+ # not the pyannote path _attach_speakers_and_faces runs below. Record
+ # against the flag that actually drove that request, not the False
+ # passed to attach (which only controls the pyannote attempt here).
+ base["diarization_attempted"] = bool(enable_diarization)
return _attach_speakers_and_faces(file_path, base, False, num_speakers, progress_callback)
+ if use_omnilingual:
+ base = _transcribe_with_omnilingual(file_path, progress_callback, wav_path=wav_path)
+ base["engine"] = "omnilingual"
+ # Same no-torch constraint as whisper.cpp: skip diarization, keep face analysis.
+ # Diarization is never possible on this engine, so it's never
+ # "attempted" regardless of what the caller asked for: a cache
+ # entry from this engine must never look like a retriable miss.
+ base["diarization_attempted"] = False
+ return _attach_speakers_and_faces(
+ file_path, base, False, num_speakers, progress_callback, wav_path=wav_path
+ )
+
# Native installs ship whisper.cpp, not openai-whisper. Fall back to it
# automatically — whether whisper is missing OR a broken install fails to
# load/run — unless the user explicitly asked for the whisper-py engine.
if not use_cpp:
if progress_callback:
progress_callback(5, "Loading Whisper model...")
- try:
- import whisper
-
- model = whisper.load_model(model_size)
- except Exception as e:
- if not requested and _whispercpp_ready(model_size):
- use_cpp = True
- else:
- raise RuntimeError(
- "The whisper-py engine needs the full source install (openai-whisper + torch). "
- "This native install ships whisper.cpp — rerun with --engine whispercpp."
- ) from e
+ fall_back, model, error = _resolve_whisper_py_fallback(requested, model_size, probe_model=True)
+ if fall_back:
+ use_cpp = True
+ elif error is not None:
+ raise RuntimeError(
+ "The whisper-py engine needs the full source install (openai-whisper + torch). "
+ "This native install ships whisper.cpp. Rerun with --engine whispercpp."
+ ) from error
if use_cpp:
base = _transcribe_with_whispercpp(
@@ -523,6 +828,8 @@ def transcribe_file(
base["engine"] = "whispercpp"
# whisper.cpp is the no-torch path: importing torch for diarization can
# hard-crash native runtimes. Skip diarization, keep face analysis (OpenCV).
+ # Never possible on this engine, see the omnilingual branch above.
+ base["diarization_attempted"] = False
return _attach_speakers_and_faces(
file_path, base, False, num_speakers, progress_callback
)
@@ -609,6 +916,7 @@ def transcribe_file(
"duration": duration,
"language": detected_lang,
"engine": "whisper-py",
+ "diarization_attempted": bool(enable_diarization),
}
return _attach_speakers_and_faces(
file_path, base, enable_diarization, num_speakers, progress_callback,
diff --git a/backend/services/transcription_omnilingual.py b/backend/services/transcription_omnilingual.py
new file mode 100644
index 0000000..664c805
--- /dev/null
+++ b/backend/services/transcription_omnilingual.py
@@ -0,0 +1,339 @@
+"""Omnilingual ASR (sherpa-onnx CTC) adapter behind the transcribe_file contract.
+
+Meta's Omnilingual ASR CTC model covers ~1600 languages, including ones
+whisper.cpp handles poorly. It's character-level CTC: tokens.txt has no
+SentencePiece "continuation vs. word-start" convention. Token id 4 is a
+literal space character, emitted by the model itself as a word boundary.
+Grouping tokens into words means splitting the token stream on that space
+token, not on a leading-marker prefix like the whisper.cpp adapter does.
+
+The model takes no language conditioning: there's no -l equivalent. It
+decodes everything through the same multilingual weights.
+"""
+
+import hashlib
+import json
+import os
+import shutil
+import urllib.request
+import uuid
+import wave
+from typing import Optional
+
+# Mirrors the pins in cli/internal/provision/provision.go so the studio and
+# MCP paths can fetch the model on first use, not only the launcher.
+MODEL_REVISION = "5243e02858d1428b8fbeeb5f7f1cc6fccf4d9433"
+MODEL_REPO = "csukuangfj2/sherpa-onnx-omnilingual-asr-1600-languages-1B-ctc-v2-int8-2026-02-05"
+MODEL_FILES = {
+ "model.int8.onnx": ("8af72da192fc2c8567c328d4f8059bdf47f182a0369077893c295ef39740c637", 1032239439),
+ "tokens.txt": ("7d99997ef207ff14c2cfe825f2aa037528ea250113cc3c6392bfe49326884ba6", 90630),
+ "LICENSE": ("a70a523bafbb595c2844104feb313d204904dac91c3d186c05f22a10a71c7a94", 581),
+}
+
+# Decode window: 20s cores tiled across the file, each padded with 1s of
+# context on each side (clipped to file bounds) so CTC isn't starved of
+# context at a window edge. A word's midpoint decides which core window
+# claims it, so the context padding doesn't duplicate words across windows.
+WINDOW_SECONDS = 20.0
+CONTEXT_SECONDS = 1.0
+
+# Mirrors transcript_packer.SILENCE_SPLIT_SEC: the pause length that splits
+# a run of words into separate segments.
+SEGMENT_PAUSE_SECONDS = 0.5
+
+# whisper-cli's trailing-token stretch over silence doesn't apply to a CTC
+# model with no language-model smoothing, but a token's timestamp from
+# sherpa-onnx is its emission time, not its end; pad the last token of a
+# word by the model's frame stride (10ms at this model's subsampling) so a
+# one-token word doesn't collapse to zero length.
+WORD_END_PAD_SECONDS = 0.02
+
+_WORD_BOUNDARY = " "
+
+
+def _sha256(path: str) -> str:
+ digest = hashlib.sha256()
+ with open(path, "rb") as source:
+ for block in iter(lambda: source.read(1024 * 1024), b""):
+ digest.update(block)
+ return digest.hexdigest()
+
+
+def ensure_model(model_dir: str, progress_callback=None) -> None:
+ """Download any missing or outdated pinned model file into model_dir.
+
+ Files only land through a hash-checked rename, so an existing file of the
+ pinned size is trusted without rehashing 1 GB on every run. A size
+ mismatch means an older model (the 300M one) and gets replaced.
+ """
+ os.makedirs(model_dir, exist_ok=True)
+ for name, (sha, size) in MODEL_FILES.items():
+ dest = os.path.join(model_dir, name)
+ if os.path.exists(dest) and os.path.getsize(dest) == size:
+ continue
+ if progress_callback:
+ progress_callback(5, f"Downloading the omnilingual model, {size / 1e9:.1f} GB on first use ({name})")
+ url = f"https://huggingface.co/{MODEL_REPO}/resolve/{MODEL_REVISION}/{name}"
+ tmp = os.path.join(model_dir, f".{name}.{uuid.uuid4().hex}.download")
+ try:
+ request = urllib.request.Request(url, headers={"User-Agent": "podcli"})
+ with urllib.request.urlopen(request, timeout=60) as response, open(tmp, "wb") as target:
+ shutil.copyfileobj(response, target, 4 * 1024 * 1024)
+ if _sha256(tmp) != sha:
+ raise RuntimeError(f"omnilingual {name} checksum mismatch; download again")
+ os.replace(tmp, dest)
+ finally:
+ if os.path.exists(tmp):
+ os.unlink(tmp)
+
+
+def _read_wav_mono16(wav_path: str):
+ """Returns (samples: np.float32 in [-1, 1], sample_rate, duration_seconds)."""
+ import numpy as np
+
+ with wave.open(wav_path, "rb") as w:
+ sr, width, n = w.getframerate(), w.getsampwidth(), w.getnframes()
+ raw = w.readframes(n)
+ if width != 2:
+ raise ValueError(f"omnilingual adapter needs 16-bit PCM wav, got {width * 8}-bit: {wav_path}")
+ samples = np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0
+ duration = len(samples) / sr if sr else 0.0
+ return samples, sr, duration
+
+
+def _load_recognizer(model_path: str, tokens_path: str, threads: int):
+ import sherpa_onnx
+
+ if not hasattr(sherpa_onnx.OfflineRecognizer, "from_omnilingual_asr_ctc"):
+ raise RuntimeError(
+ "This sherpa-onnx version has no OfflineRecognizer.from_omnilingual_asr_ctc; "
+ "upgrade sherpa-onnx (pip install -U sherpa-onnx) to use --engine omnilingual."
+ )
+ return sherpa_onnx.OfflineRecognizer.from_omnilingual_asr_ctc(
+ model=model_path, tokens=tokens_path, num_threads=threads, provider="cpu",
+ )
+
+
+def _validate_tokens_and_timestamps(tokens: list[str], timestamps: list[float], window_label: str) -> None:
+ if len(tokens) != len(timestamps):
+ raise ValueError(
+ f"omnilingual decode produced {len(tokens)} tokens but {len(timestamps)} timestamps ({window_label})"
+ )
+ prev = -1.0
+ for i, t in enumerate(timestamps):
+ if t != t or t < 0: # t != t is the NaN check
+ raise ValueError(f"omnilingual decode produced an invalid timestamp {t!r} at token {i} ({window_label})")
+ if t < prev:
+ raise ValueError(f"omnilingual decode produced out-of-order timestamps at token {i} ({window_label})")
+ prev = t
+
+
+def _tokens_to_words(tokens: list[str], timestamps: list[float], decode_end: float) -> list[dict]:
+ """Split a flat token stream on the word-boundary space token, into
+ {word, start, end} with absolute-within-decode-window times."""
+ words: list[dict] = []
+ cur_tokens: list[str] = []
+ cur_times: list[float] = []
+
+ def flush():
+ if not cur_tokens:
+ return
+ text = "".join(cur_tokens).strip()
+ if not text:
+ return
+ start = cur_times[0]
+ end = min(cur_times[-1] + WORD_END_PAD_SECONDS, decode_end)
+ if end <= start:
+ end = start + WORD_END_PAD_SECONDS
+ words.append({"word": text, "start": start, "end": end})
+
+ for tok, ts in zip(tokens, timestamps):
+ if tok == _WORD_BOUNDARY:
+ flush()
+ cur_tokens, cur_times = [], []
+ continue
+ cur_tokens.append(tok)
+ cur_times.append(ts)
+ flush()
+ return words
+
+
+def _drop_seam_duplicates(prev_words: list[dict], candidates: list[dict], boundary: float) -> list[dict]:
+ """Drop a candidate word that the previous window already claimed near
+ their shared boundary. Each window owns a disjoint [core_start,
+ core_end) by its own midpoint estimate (see transcribe_file), but the
+ 1s of context padding on each side means the same audio gets decoded
+ twice, by two separate recognizer calls, and the acoustic model can give
+ the same word a slightly different timestamp each time, so it can pass
+ both windows' own membership test and get counted twice. Only words
+ near the boundary are ever compared, so a legitimate stutter/repeat
+ elsewhere in the transcript is never touched.
+ """
+ kept = []
+ for w in candidates:
+ near_boundary = abs(w["start"] - boundary) <= CONTEXT_SECONDS or abs(w["end"] - boundary) <= CONTEXT_SECONDS
+ is_duplicate = near_boundary and any(
+ pw["word"] == w["word"] and abs(pw["start"] - w["start"]) <= CONTEXT_SECONDS
+ for pw in prev_words
+ if abs(pw["end"] - boundary) <= CONTEXT_SECONDS or abs(pw["start"] - boundary) <= CONTEXT_SECONDS
+ )
+ if not is_duplicate:
+ kept.append(w)
+ return kept
+
+
+def _group_into_segments(words: list[dict]) -> list[dict]:
+ """Group words into segments on a pause >= SEGMENT_PAUSE_SECONDS, the
+ same boundary transcript_packer uses to split phrases for the packed
+ markdown view."""
+ segments: list[dict] = []
+ current: Optional[dict] = None
+ for w in words:
+ if current is not None and w["start"] - current["end"] >= SEGMENT_PAUSE_SECONDS:
+ segments.append(current)
+ current = None
+ if current is None:
+ current = {"text": w["word"], "start": w["start"], "end": w["end"]}
+ else:
+ current["text"] += " " + w["word"]
+ current["end"] = w["end"]
+ if current is not None:
+ segments.append(current)
+ return [
+ {"id": i, "start": round(s["start"], 3), "end": round(s["end"], 3), "text": s["text"], "speaker": None}
+ for i, s in enumerate(segments)
+ ]
+
+
+def transcribe_file(
+ file_path: str,
+ model_path: str,
+ tokens_path: str,
+ threads: int = 4,
+ wav_path: Optional[str] = None,
+ run_dir: Optional[str] = None,
+ progress_callback=None,
+ **_ignored,
+) -> dict:
+ """run_dir, if given, makes this resumable: each core window's words are
+ written to run_dir/window-<i>.json as soon as they're decoded, and a
+ rerun with the same run_dir skips any window that already has one. The
+ final transcript is only assembled (and the receipts cleared) once every
+ window has a receipt. A crash mid-file loses at most the window that
+ was decoding, not the whole run."""
+ if not os.path.exists(file_path):
+ raise FileNotFoundError(file_path)
+ if not os.path.exists(model_path):
+ raise FileNotFoundError(f"omnilingual model not found: {model_path}")
+ if not os.path.exists(tokens_path):
+ raise FileNotFoundError(f"omnilingual tokens not found: {tokens_path}")
+
+ owns_wav = False
+ if not (wav_path and os.path.exists(wav_path)):
+ from services.audio_extract import extract_wav_16k_mono
+
+ wav_path = extract_wav_16k_mono(file_path)
+ owns_wav = True
+
+ from services import transcribe_runs
+
+ try:
+ samples, sr, duration = _read_wav_mono16(wav_path)
+ recognizer = None # lazily loaded: a fully-resumed run never needs it
+
+ all_words: list[dict] = []
+ n_windows = max(1, int(duration // WINDOW_SECONDS) + (1 if duration % WINDOW_SECONDS else 0))
+ for i in range(n_windows):
+ core_start = i * WINDOW_SECONDS
+ core_end = min(core_start + WINDOW_SECONDS, duration)
+
+ receipt_name = f"window-{i}.json"
+ cached = transcribe_runs.read_receipt(run_dir, receipt_name) if run_dir else None
+ if cached is not None:
+ all_words.extend(cached.get("words") or [])
+ if progress_callback:
+ progress_callback(int(100 * (i + 1) / n_windows), f"Window {i + 1}/{n_windows} (resumed)")
+ continue
+
+ decode_start = max(0.0, core_start - CONTEXT_SECONDS)
+ decode_end = min(duration, core_end + CONTEXT_SECONDS)
+
+ lo, hi = int(decode_start * sr), int(decode_end * sr)
+ chunk = samples[lo:hi]
+ if chunk.size == 0:
+ continue
+
+ if recognizer is None:
+ recognizer = _load_recognizer(model_path, tokens_path, threads)
+
+ stream = recognizer.create_stream()
+ stream.accept_waveform(sr, chunk)
+ recognizer.decode_stream(stream)
+ result = stream.result
+
+ tokens = list(result.tokens)
+ timestamps = [decode_start + float(t) for t in result.timestamps]
+ _validate_tokens_and_timestamps(tokens, timestamps, window_label=f"window {i} [{decode_start:.1f}, {decode_end:.1f})")
+
+ words = _tokens_to_words(tokens, timestamps, decode_end)
+ window_words = [
+ w for w in words
+ if core_start <= (w["start"] + w["end"]) / 2.0 < core_end
+ or (i == n_windows - 1 and (w["start"] + w["end"]) / 2.0 == core_end)
+ ]
+ if i > 0:
+ # Ownership-by-midpoint alone isn't enough at a seam: the
+ # context overlap means the same word gets decoded twice,
+ # once in each neighbor's own context, and the acoustic
+ # model doesn't always timestamp it identically both times.
+ # A word whose own-window estimate lands it just past the
+ # boundary can duplicate one the previous window already
+ # claimed under its own (slightly different) estimate.
+ window_words = _drop_seam_duplicates(all_words, window_words, core_start)
+ all_words.extend(window_words)
+ if run_dir:
+ transcribe_runs.write_receipt(run_dir, receipt_name, {"words": window_words})
+ if progress_callback:
+ progress_callback(int(100 * (i + 1) / n_windows), f"Window {i + 1}/{n_windows}")
+
+ if run_dir:
+ transcribe_runs.clear_run(run_dir)
+
+ all_words.sort(key=lambda w: w["start"])
+ words_out = [
+ {"word": w["word"], "start": round(w["start"], 3), "end": round(w["end"], 3), "speaker": None}
+ for w in all_words
+ ]
+ segments = _group_into_segments(all_words)
+
+ return {
+ "transcript": " ".join(w["word"] for w in words_out).strip(),
+ "segments": segments,
+ "words": words_out,
+ "duration": round(duration, 3),
+ # The model takes no language conditioning: it decodes every
+ # language through the same weights. "und" reflects that nothing
+ # was detected or requested, not that detection failed.
+ "language": "und",
+ }
+ finally:
+ if owns_wav and wav_path and os.path.exists(wav_path):
+ try:
+ os.unlink(wav_path)
+ except OSError:
+ pass
+
+
+if __name__ == "__main__":
+ import sys
+
+ media, model, tokens = sys.argv[1], sys.argv[2], sys.argv[3]
+ out = sys.argv[4] if len(sys.argv) > 4 else None
+ result = transcribe_file(media, model, tokens)
+ payload = json.dumps(result, indent=2, ensure_ascii=False)
+ if out:
+ with open(out, "w", encoding="utf-8") as f:
+ f.write(payload)
+ print(f"{len(result['words'])} words, {len(result['segments'])} segments -> {out}")
+ else:
+ print(payload)
diff --git a/backend/services/transcription_whispercpp.py b/backend/services/transcription_whispercpp.py
index 6d3548c..7c3b025 100644
--- a/backend/services/transcription_whispercpp.py
+++ b/backend/services/transcription_whispercpp.py
@@ -86,6 +86,27 @@ def _tokens_to_words(tokens: list[dict]) -> list[dict]:
return words
+def _frame_rms(samples, hop: int, frame: int, nf: int):
+ """Per-frame RMS via a running sum of squares, O(n) instead of the
+ O(nf * frame) index-matrix this replaces (see _voiced_intervals).
+
+ The cumulative sum has to be float64. float32 has ~7 decimal digits of
+ precision; on a long file the running total grows into the 1e13-1e15
+ range, where a single float32 ULP is already bigger than a whole
+ frame's sum of squares, so window_sums (a difference of two large
+ nearly-equal floats) comes out wrong for frames late in the file. A
+ short test clip never sums high enough to show this: the regression
+ only appears on real multi-hour episodes.
+ """
+ import numpy as np
+
+ sq = samples.astype(np.float64) ** 2
+ csum = np.concatenate(([0.0], np.cumsum(sq, dtype=np.float64)))
+ starts = np.arange(nf) * hop
+ window_sums = csum[starts + frame] - csum[starts]
+ return np.sqrt(np.maximum(window_sums, 0.0) / frame)
+
+
def _voiced_intervals(wav_path: str, bridge: float = 0.3, thresh_ratio: float = 0.07):
import wave
@@ -106,8 +127,7 @@ def _voiced_intervals(wav_path: str, bridge: float = 0.3, thresh_ratio: float =
if len(samples) < frame:
return []
nf = 1 + (len(samples) - frame) // hop
- idx = np.arange(nf)[:, None] * hop + np.arange(frame)[None, :]
- rms = np.sqrt((samples[idx] ** 2).mean(axis=1))
+ rms = _frame_rms(samples, hop, frame, nf)
peak = float(rms.max())
if peak <= 0:
return []
@@ -164,7 +184,7 @@ def transcribe_file(
model_path: str,
whisper_cli: str = "whisper-cli",
ffmpeg: str = "ffmpeg",
- language: Optional[str] = "en",
+ language: Optional[str] = None,
dtw_model: Optional[str] = None,
threads: int = 4,
vad: bool = False,
@@ -196,8 +216,9 @@ def transcribe_file(
# systematic early bias (silence-removal remapping). Off by default;
# the energy-snap below addresses the same defect without the bias.
cmd += ["--vad", "--vad-model", vad_model]
- if language:
- cmd += ["-l", language]
+ # An unset language must not fall through to whisper-cli's own "en"
+ # default; "auto" makes it run language detection instead.
+ cmd += ["-l", language or "auto"]
subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True, timeout=7200)
with open(out_base + ".json", encoding="utf-8") as f:
@@ -224,7 +245,15 @@ def transcribe_file(
"segments": segments,
"words": words,
"duration": segments[-1]["end"] if segments else 0.0,
- "language": (data.get("params") or {}).get("language") or language or "en",
+ # whisper-cli reports the language it actually detected under
+ # "result"; "params" only echoes back what we passed in ("auto"
+ # when unset), so it can't label the output.
+ "language": (
+ (data.get("result") or {}).get("language")
+ or (data.get("params") or {}).get("language")
+ or language
+ or "en"
+ ),
}
finally:
shutil.rmtree(tmpdir, ignore_errors=True)
diff --git a/backend/services/video_cut.py b/backend/services/video_cut.py
index cbe72d9..e2211a5 100644
--- a/backend/services/video_cut.py
+++ b/backend/services/video_cut.py
@@ -49,33 +49,57 @@ def cut_segment(
return output_path
+def probe_duration(path: str) -> float:
+ """Read media duration in seconds (best effort, 0.0 if unreadable)."""
+ try:
+ cmd = [
+ "ffprobe", "-v", "error",
+ "-show_entries", "format=duration",
+ "-of", "default=nk=1:nw=1",
+ path,
+ ]
+ result = proc_run(cmd, timeout=10, check=False)
+ if result.returncode != 0:
+ return 0.0
+ return float((result.stdout or "0").strip() or 0.0)
+ except Exception:
+ return 0.0
+
+
def cut_multi_segment(
input_path: str,
output_path: str,
segments: list[dict],
-) -> str:
+) -> tuple[str, list[float]]:
"""Cut multiple time ranges and concatenate them seamlessly.
segments: [{"start": 10.5, "end": 25.0}, {"start": 30.2, "end": 45.0}]
- Each segment is cut individually with frame-accurate encoding,
- then concatenated with stream copy (matching codecs means no
- re-encode needed).
+ Each segment is cut individually with frame-accurate encoding, then
+ concatenated with stream copy (matching codecs means no re-encode
+ needed). Returns (output_path, part_durations): the probed duration of
+ each encoded part, not the requested one. An encoder snaps a cut to
+ whole frames, so the actual part is typically a few milliseconds off
+ the requested end - start; captions timed from the requested length
+ instead of the probed one drift further with every cut concatenated in.
"""
if len(segments) == 1:
- return cut_segment(
+ out = cut_segment(
input_path, output_path, segments[0]["start"], segments[0]["end"]
)
+ return out, [probe_duration(out)]
work_dir = os.path.dirname(output_path) or "."
part_paths: list[str] = []
concat_file = os.path.join(work_dir, "_concat_parts.txt")
try:
+ part_durations: list[float] = []
for i, seg in enumerate(segments):
part_path = os.path.join(work_dir, f"_part_{i}.mp4")
cut_segment(input_path, part_path, seg["start"], seg["end"])
part_paths.append(part_path)
+ part_durations.append(probe_duration(part_path))
with open(concat_file, "w", encoding="utf-8") as f:
for p in part_paths:
@@ -93,7 +117,7 @@ def cut_multi_segment(
if result.returncode != 0:
raise RuntimeError(f"FFmpeg concat failed: {result.stderr[-500:]}")
- return output_path
+ return output_path, part_durations
finally:
for p in part_paths:
@@ -101,3 +125,47 @@ def cut_multi_segment(
os.remove(p)
if os.path.exists(concat_file):
os.remove(concat_file)
+
+
+def probe_has_audio_stream(path: str) -> bool:
+ """True if ffprobe finds at least one audio stream in the file."""
+ cmd = [
+ "ffprobe", "-v", "error",
+ "-select_streams", "a",
+ "-show_entries", "stream=codec_type",
+ "-of", "csv=p=0",
+ path,
+ ]
+ result = proc_run(cmd, timeout=FFMPEG_TIMEOUT, check=False)
+ if result.returncode != 0:
+ return False
+ return "audio" in (result.stdout or "")
+
+
+def verify_full_decode(path: str, max_error_lines: int = 3) -> str | None:
+ """Decode the whole file and return ffmpeg's error output, or None if clean.
+
+ An ffmpeg render that exits 0 can still have written a truncated or
+ corrupt file: a moov atom cut short, a partial frame at the tail, a
+ stream copy/concat mismatch. Those only surface on a full decode, which
+ is what this runs: the same check as `ffmpeg -v error -i x -f null -`
+ from the command line. -threads auto keeps that decode from running
+ single-threaded on a multi-core box.
+
+ A nonzero exit always fails. Otherwise, `-v error` also logs a handful
+ of lines ffmpeg can emit on an otherwise-fine file (e.g. a non-monotonic
+ DTS warning from a concat/re-encode), a single clip used to get deleted
+ for one such line even though it decoded and played fine. Only treat it
+ as a real decode failure once more than a few such lines show up.
+ """
+ cmd = ["ffmpeg", "-v", "error", "-threads", "auto", "-i", path, "-f", "null", "-"]
+ result = proc_run(cmd, timeout=FFMPEG_TIMEOUT, check=False)
+ stderr = (result.stderr or "").strip()
+ if result.returncode != 0:
+ return stderr[-1000:] if stderr else f"ffmpeg exited {result.returncode} decoding the output"
+ if not stderr:
+ return None
+ error_lines = [line for line in stderr.splitlines() if line.strip()]
+ if len(error_lines) > max_error_lines:
+ return stderr[-1000:]
+ return None
diff --git a/backend/utils/text.py b/backend/utils/text.py
index 1f734ba..9d481c0 100644
--- a/backend/utils/text.py
+++ b/backend/utils/text.py
@@ -3,6 +3,31 @@
TITLE_MAX = 55
FILENAME_MAX = 50
+# Scripts with no case distinction whose letters Unicode nonetheless assigns
+# an uppercase mapping for display styling (Georgian Mkhedruli -> Mtavruli).
+# str.upper() applies that mapping uncritically, so a caption or thumbnail
+# style that uppercases for emphasis silently switches alphabets for these
+# scripts instead of just emphasizing them.
+_CASELESS_SCRIPT_RANGES = (
+ (0x10A0, 0x10FF), # Georgian (Mkhedruli, Asomtavruli)
+ (0x1C90, 0x1CBF), # Georgian Extended (Mtavruli)
+ (0x2D00, 0x2D2F), # Georgian Supplement
+)
+
+
+def _is_caseless_script_char(ch: str) -> bool:
+ cp = ord(ch)
+ return any(lo <= cp <= hi for lo, hi in _CASELESS_SCRIPT_RANGES)
+
+
+def safe_upper(text: str) -> str:
+ """Uppercase text, except for scripts where Unicode's uppercase mapping
+ would change the alphabet rather than just the case (see above).
+ """
+ if not text or not any(_is_caseless_script_char(c) for c in text):
+ return text.upper() if text else text
+ return "".join(c if _is_caseless_script_char(c) else c.upper() for c in text)
+
_TRAILING_PUNCT = " ,;:-–—"
# Windows refuses these as a filename stem whatever the extension, so "CON.fcpxml"
diff --git a/cli/doctor_test.go b/cli/doctor_test.go
new file mode 100644
index 0000000..70d29b6
--- /dev/null
+++ b/cli/doctor_test.go
@@ -0,0 +1,161 @@
+package main
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+
+ "podcli/internal/provision"
+)
+
+func TestFirstLine(t *testing.T) {
+ if got := firstLine("a\nb\nc"); got != "a" {
+ t.Fatalf("got %q", got)
+ }
+ if got := firstLine(" padded \n"); got != "padded" {
+ t.Fatalf("got %q", got)
+ }
+ if got := firstLine(""); got != "" {
+ t.Fatalf("got %q", got)
+ }
+}
+
+// go is guaranteed present wherever these tests run (it built the test
+// binary), so it stands in for "a real hermetic/PATH tool that works".
+func TestRunCheckSucceedsForARealBinary(t *testing.T) {
+ c := runCheck("go", "", "go", "version")
+ if !c.OK {
+ t.Fatalf("expected ok, got %+v", c)
+ }
+ if c.Detail == "" {
+ t.Fatal("expected a detail message")
+ }
+}
+
+func TestRunCheckFailsWhenBinaryIsMissing(t *testing.T) {
+ c := runCheck("nope", "", "podcli-definitely-not-a-real-binary-xyz")
+ if c.OK {
+ t.Fatalf("expected failure, got %+v", c)
+ }
+}
+
+func TestRunCheckFailsWhenHermeticPathDoesNotRun(t *testing.T) {
+ // A hermetic path that resolves but can't actually run (e.g. corrupted
+ // install) must fail the check, not just report "found".
+ dir := t.TempDir()
+ bogus := filepath.Join(dir, "not-executable")
+ if err := os.WriteFile(bogus, []byte("not a real binary"), 0o644); err != nil {
+ t.Fatalf("seed bogus binary: %v", err)
+ }
+ c := runCheck("bogus", bogus, "bogus")
+ if c.OK {
+ t.Fatalf("expected failure, got %+v", c)
+ }
+}
+
+func TestModelChecksSkipsUnprovisionedModels(t *testing.T) {
+ home := t.TempDir()
+ old, hadOld := os.LookupEnv("PODCLI_HOME")
+ os.Setenv("PODCLI_HOME", home)
+ t.Cleanup(func() {
+ if hadOld {
+ os.Setenv("PODCLI_HOME", old)
+ } else {
+ os.Unsetenv("PODCLI_HOME")
+ }
+ })
+
+ checks := modelChecks()
+ if len(checks) != 0 {
+ t.Fatalf("expected no checks for an install with no models downloaded, got %+v", checks)
+ }
+}
+
+func TestModelChecksFlagsAHashMismatch(t *testing.T) {
+ home := t.TempDir()
+ old, hadOld := os.LookupEnv("PODCLI_HOME")
+ os.Setenv("PODCLI_HOME", home)
+ t.Cleanup(func() {
+ if hadOld {
+ os.Setenv("PODCLI_HOME", old)
+ } else {
+ os.Unsetenv("PODCLI_HOME")
+ }
+ })
+
+ modelPath := provision.ModelPath("base")
+ if err := os.MkdirAll(filepath.Dir(modelPath), 0o755); err != nil {
+ t.Fatalf("mkdir models dir: %v", err)
+ }
+ if err := os.WriteFile(modelPath, []byte("not the real model bytes"), 0o644); err != nil {
+ t.Fatalf("seed fake model: %v", err)
+ }
+
+ checks := modelChecks()
+ found := false
+ for _, c := range checks {
+ if c.Name == "model base" {
+ found = true
+ if c.OK {
+ t.Fatalf("expected a hash mismatch to fail, got %+v", c)
+ }
+ }
+ }
+ if !found {
+ t.Fatalf("expected a check for the present-but-wrong base model, got %+v", checks)
+ }
+}
+
+func TestLevelRegistrationChecksWarnsWhenAtLeastOneAgentIsRegistered(t *testing.T) {
+ // Claude registered, Codex not: Codex's unregistered folder is only a
+ // warning, since the MCP tools already work through Claude.
+ checks := levelRegistrationChecks([]doctorCheck{
+ {Name: "mcp registration (Claude)", OK: true},
+ {Name: "mcp registration (Codex)", OK: false},
+ })
+ if checks[0].Level != levelOK {
+ t.Fatalf("expected the registered agent to be level ok, got %+v", checks[0])
+ }
+ if checks[1].Level != levelWarn {
+ t.Fatalf("expected the unregistered agent to be a warning, not a failure, got %+v", checks[1])
+ }
+}
+
+func TestLevelRegistrationChecksFailsWhenNoAgentIsRegisteredAnywhere(t *testing.T) {
+ // Both agents detected, neither registered: nothing would actually work,
+ // so this escalates to a real failure instead of a quiet warning.
+ checks := levelRegistrationChecks([]doctorCheck{
+ {Name: "mcp registration (Claude)", OK: false},
+ {Name: "mcp registration (Codex)", OK: false},
+ })
+ for _, c := range checks {
+ if c.Level != levelFail {
+ t.Fatalf("expected every check to fail when none are registered, got %+v", c)
+ }
+ }
+}
+
+func TestLevelRegistrationChecksIsANoOpOnNoDetectedAgents(t *testing.T) {
+ if checks := levelRegistrationChecks(nil); len(checks) != 0 {
+ t.Fatalf("expected no checks when no agent CLI was detected, got %+v", checks)
+ }
+}
+
+func TestChecksAllOKIgnoresWarnLevelChecks(t *testing.T) {
+ checks := []doctorCheck{
+ {Name: "ffmpeg", OK: true, Level: levelOK},
+ {Name: "mcp registration (Codex)", OK: false, Level: levelWarn},
+ }
+ if !checksAllOK(checks) {
+ t.Fatal("a warn-level check must not flip doctor's overall result to failing")
+ }
+}
+
+func TestChecksAllOKCountsFailLevelChecks(t *testing.T) {
+ checks := []doctorCheck{
+ {Name: "ffmpeg", OK: false, Level: levelFail},
+ }
+ if checksAllOK(checks) {
+ t.Fatal("a fail-level check must flip doctor's overall result to failing")
+ }
+}
diff --git a/cli/internal/podstack/podstack.go b/cli/internal/podstack/podstack.go
index 6a05b37..16e008d 100644
--- a/cli/internal/podstack/podstack.go
+++ b/cli/internal/podstack/podstack.go
@@ -6,7 +6,10 @@
package podstack
import (
+ "crypto/sha256"
"embed"
+ "encoding/hex"
+ "encoding/json"
"fmt"
"io/fs"
"os"
@@ -75,28 +78,271 @@ func warnEmptyKnowledge() {
fmt.Fprintf(os.Stderr, " %sKnowledge base is empty.%s Run %spodcli knowledge init%s first, or %s/bootstrap-knowledge%s in your agent, so the workflow knows your show.\n", colYellow, colReset, colAccent, colReset, colAccent, colReset)
}
+// manifestName is the install-tracking file, hidden among the slash commands
+// it describes. It is not itself a command (no .md suffix), so Names() and
+// IsCommand() never see it.
+const manifestName = ".podcli-manifest.json"
+
+// installManifest records the sha256 of the content podcli last wrote for
+// each command file, so a later install can tell "podcli wrote this and
+// nothing has touched it since" (safe to overwrite with the new version)
+// apart from "the user edited this" (leave it alone).
+type installManifest struct {
+ Files map[string]string `json:"files"`
+}
+
+func readManifest(dest string) installManifest {
+ m := installManifest{Files: map[string]string{}}
+ data, err := os.ReadFile(filepath.Join(dest, manifestName))
+ if err != nil {
+ return m
+ }
+ _ = json.Unmarshal(data, &m)
+ if m.Files == nil {
+ m.Files = map[string]string{}
+ }
+ return m
+}
+
+func writeManifest(dest string, m installManifest) error {
+ data, err := json.MarshalIndent(m, "", " ")
+ if err != nil {
+ return err
+ }
+ return os.WriteFile(filepath.Join(dest, manifestName), data, 0o644)
+}
+
+func sha256Hex(data []byte) string {
+ sum := sha256.Sum256(data)
+ return hex.EncodeToString(sum[:])
+}
+
+// installReport summarizes what installCommands did, for callers that want
+// to tell the user about files it left alone.
+type installReport struct {
+ Installed []string // written for the first time
+ Updated []string // podcli-owned, unmodified by the user, refreshed to the new version
+ UserModified []string // changed since podcli last wrote them, left alone
+}
+
+func (r installReport) hasUpdates() bool {
+ return len(r.Installed) > 0 || len(r.Updated) > 0
+}
+
// installCommands writes the embedded slash-command files into
-// <project>/.claude/commands so the agent can resolve /<cmd>. Existing files are
-// left untouched, so a user's local edits win.
-func installCommands(project string) error {
+// <project>/.claude/commands so the agent can resolve /<cmd>. A manifest
+// tracks the hash of what podcli wrote for each file: on a later run (e.g.
+// after `podcli update`), a file whose on-disk hash still matches that
+// record gets refreshed to the new embedded version; a file the user edited
+// is left untouched and reported instead of silently overwritten. A file
+// that predates the manifest (no record at all) is treated the same way:
+// left alone. Its current content becomes the new baseline, so podcli
+// never clobbers an install from before this tracking existed.
+func installCommands(project string) (installReport, error) {
+ var report installReport
dest := filepath.Join(project, ".claude", "commands")
if err := os.MkdirAll(dest, 0o755); err != nil {
- return err
+ return report, err
}
- return fs.WalkDir(commands, "commands", func(p string, d fs.DirEntry, err error) error {
+ manifest := readManifest(dest)
+ changed := false
+
+ err := fs.WalkDir(commands, "commands", func(p string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
- target := filepath.Join(dest, filepath.Base(p))
- if _, err := os.Stat(target); err == nil {
+ name := filepath.Base(p)
+ target := filepath.Join(dest, name)
+ embedded, err := commands.ReadFile(p)
+ if err != nil {
+ return err
+ }
+ embeddedHash := sha256Hex(embedded)
+
+ current, statErr := os.ReadFile(target)
+ if statErr != nil {
+ // Never installed here before.
+ if err := os.WriteFile(target, embedded, 0o644); err != nil {
+ return err
+ }
+ manifest.Files[name] = embeddedHash
+ changed = true
+ report.Installed = append(report.Installed, name)
+ return nil
+ }
+
+ lastInstalledHash, tracked := manifest.Files[name]
+ currentHash := sha256Hex(current)
+
+ if !tracked {
+ // Predates the manifest. Only adopt it as the baseline if it's
+ // actually the stock file (hash matches what's embedded). That's
+ // the "installed before tracking existed" case. If it doesn't
+ // match, it's a user edit (or something else entirely); leave it
+ // untracked rather than recording its current hash as a baseline,
+ // which would make the next run believe it's unmodified and
+ // overwrite it once the embedded version changes.
+ if currentHash == embeddedHash {
+ manifest.Files[name] = currentHash
+ changed = true
+ return nil
+ }
+ report.UserModified = append(report.UserModified, name)
+ return nil
+ }
+
+ if currentHash != lastInstalledHash {
+ // The user changed it since podcli last wrote it.
+ report.UserModified = append(report.UserModified, name)
return nil
}
- data, err := commands.ReadFile(p)
+
+ if currentHash == embeddedHash {
+ return nil // already current
+ }
+
+ if err := os.WriteFile(target, embedded, 0o644); err != nil {
+ return err
+ }
+ manifest.Files[name] = embeddedHash
+ changed = true
+ report.Updated = append(report.Updated, name)
+ return nil
+ })
+ if err != nil {
+ return report, err
+ }
+ if changed {
+ if err := writeManifest(dest, manifest); err != nil {
+ return report, err
+ }
+ }
+ return report, nil
+}
+
+// codexSkillsDir is where the installed Codex CLI reads skills from
+// (confirmed against a local `codex` install: ~/.codex/skills/<name>/SKILL.md,
+// one directory per skill). It is global, not per-project, unlike
+// .claude/commands.
+func codexSkillsDir() (string, error) {
+ home, err := os.UserHomeDir()
+ if err != nil {
+ return "", err
+ }
+ return filepath.Join(home, ".codex", "skills"), nil
+}
+
+// frontmatterDescription pulls the `description:` field out of a command
+// file's YAML frontmatter without a YAML dependency. The format here is a
+// fixed, simple `key: value` list.
+func frontmatterDescription(raw string) string {
+ lines := strings.Split(raw, "\n")
+ inFrontmatter := false
+ for i, line := range lines {
+ if i == 0 && strings.TrimSpace(line) == "---" {
+ inFrontmatter = true
+ continue
+ }
+ if !inFrontmatter {
+ return ""
+ }
+ if strings.TrimSpace(line) == "---" {
+ return ""
+ }
+ if rest, ok := strings.CutPrefix(line, "description:"); ok {
+ return strings.TrimSpace(rest)
+ }
+ }
+ return ""
+}
+
+// codexSkillContent rewraps a PodStack command file as a Codex SKILL.md:
+// same body, frontmatter translated from this project's `description:` /
+// `argument-hint:` shape to the `name:` / `description:` shape codex reads.
+func codexSkillContent(name, raw string) string {
+ desc := frontmatterDescription(raw)
+ if desc == "" {
+ desc = "PodStack command: " + name
+ }
+ body := raw
+ if end := strings.Index(raw, "\n---\n"); strings.HasPrefix(raw, "---\n") && end != -1 {
+ body = strings.TrimPrefix(raw[end+len("\n---\n"):], "\n")
+ }
+ return fmt.Sprintf("---\nname: %s\ndescription: %s\n---\n\n%s", name, desc, body)
+}
+
+// installCodexSkills writes every PodStack command as a Codex skill under
+// ~/.codex/skills/<name>/SKILL.md. Skills are global (not per-project like
+// .claude/commands), so this always targets the user's home directory
+// regardless of which project podcli was run from. Like installCommands, a
+// manifest tracks what podcli last wrote so an upgrade can refresh
+// unmodified skills and leave user edits alone.
+func installCodexSkills() error {
+ skillsDir, err := codexSkillsDir()
+ if err != nil {
+ return err
+ }
+ if err := os.MkdirAll(skillsDir, 0o755); err != nil {
+ return err
+ }
+ manifest := readManifest(skillsDir)
+ changed := false
+
+ err = fs.WalkDir(commands, "commands", func(p string, d fs.DirEntry, err error) error {
+ if err != nil || d.IsDir() {
+ return err
+ }
+ name := strings.TrimSuffix(filepath.Base(p), ".md")
+ raw, err := commands.ReadFile(p)
if err != nil {
return err
}
- return os.WriteFile(target, data, 0o644)
+ content := []byte(codexSkillContent(name, string(raw)))
+ contentHash := sha256Hex(content)
+
+ skillDir := filepath.Join(skillsDir, name)
+ target := filepath.Join(skillDir, "SKILL.md")
+ current, statErr := os.ReadFile(target)
+ if statErr != nil {
+ if err := os.MkdirAll(skillDir, 0o755); err != nil {
+ return err
+ }
+ if err := os.WriteFile(target, content, 0o644); err != nil {
+ return err
+ }
+ manifest.Files[name] = contentHash
+ changed = true
+ return nil
+ }
+
+ lastInstalledHash, tracked := manifest.Files[name]
+ currentHash := sha256Hex(current)
+ if !tracked {
+ // Same reasoning as installCommands: only adopt an untracked file
+ // as the baseline if it's actually the stock skill content.
+ if currentHash == contentHash {
+ manifest.Files[name] = currentHash
+ changed = true
+ }
+ return nil
+ }
+ if currentHash != lastInstalledHash || currentHash == contentHash {
+ return nil // user-modified, or already current
+ }
+ if err := os.WriteFile(target, content, 0o644); err != nil {
+ return err
+ }
+ manifest.Files[name] = contentHash
+ changed = true
+ return nil
})
+ if err != nil {
+ return err
+ }
+ if changed {
+ return writeManifest(skillsDir, manifest)
+ }
+ return nil
}
// Run launches the agent for cmd with the remaining args as the slash-command
@@ -137,9 +383,13 @@ func Run(cmd string, args []string) int {
if err != nil {
project = "."
}
- if err := installCommands(project); err != nil {
+ report, err := installCommands(project)
+ if err != nil {
fmt.Fprintf(os.Stderr, " %swarning:%s could not install slash commands: %v\n", colYellow, colReset, err)
}
+ if len(report.UserModified) > 0 {
+ fmt.Fprintf(os.Stderr, " %sKept your edits%s to: %s\n", colDim, colReset, strings.Join(report.UserModified, ", "))
+ }
warnEmptyKnowledge()
prompt := "/" + cmd
@@ -151,6 +401,12 @@ func Run(cmd string, args []string) int {
claudeBin, _ := exec.LookPath("claude")
codexBin, _ := exec.LookPath("codex")
+ if codexBin != "" {
+ if err := installCodexSkills(); err != nil {
+ fmt.Fprintf(os.Stderr, " %swarning:%s could not install Codex skills: %v\n", colYellow, colReset, err)
+ }
+ }
+
if engine == "codex" && codexBin == "" {
fmt.Fprintf(os.Stderr, "\n %sCodex not found in PATH.%s\n Install it, then run:\n %scodex --cd %q %q%s\n\n", colBold, colReset, colAccent, project, codexPrompt, colReset)
return 1
@@ -160,12 +416,14 @@ func Run(cmd string, args []string) int {
return 1
}
+ // Fall back to Codex only when Claude isn't installed at all. Claude
+ // exiting nonzero (the user cancelled, a tool failed mid-run, etc.) is
+ // not a reason to silently relaunch the whole workflow under a
+ // different agent. It previously was, which could run the same
+ // destructive command twice under two different engines.
if engine != "codex" && claudeBin != "" {
fmt.Fprintf(os.Stderr, "\n %s▶%s Launching Claude Code with: %s%s%s\n %scwd: %s%s\n\n", colGreen, colReset, colAccent, prompt, colReset, colDim, project, colReset)
- if code := runIn(project, claudeBin, prompt); code == 0 || engine == "claude" || codexBin == "" {
- return code
- }
- fmt.Fprintf(os.Stderr, "\n %s⚠%s Claude exited nonzero; trying Codex...\n", colYellow, colReset)
+ return runIn(project, claudeBin, prompt)
}
if codexBin == "" {
diff --git a/cli/internal/podstack/podstack_test.go b/cli/internal/podstack/podstack_test.go
new file mode 100644
index 0000000..50ac8ed
--- /dev/null
+++ b/cli/internal/podstack/podstack_test.go
@@ -0,0 +1,367 @@
+package podstack
+
+import (
+ "os"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "testing"
+)
+
+// withHome points os.UserHomeDir() (and HOME/USERPROFILE) at a temp dir for
+// the duration of the test, so installCodexSkills never touches the real
+// ~/.codex/skills.
+func withHome(t *testing.T) string {
+ t.Helper()
+ home := t.TempDir()
+ envVar := "HOME"
+ if runtime.GOOS == "windows" {
+ envVar = "USERPROFILE"
+ }
+ old, hadOld := os.LookupEnv(envVar)
+ if err := os.Setenv(envVar, home); err != nil {
+ t.Fatalf("setenv: %v", err)
+ }
+ t.Cleanup(func() {
+ if hadOld {
+ os.Setenv(envVar, old)
+ } else {
+ os.Unsetenv(envVar)
+ }
+ })
+ return home
+}
+
+func TestInstallCommandsWritesEveryFileOnFirstRun(t *testing.T) {
+ project := t.TempDir()
+
+ report, err := installCommands(project)
+ if err != nil {
+ t.Fatalf("installCommands: %v", err)
+ }
+ if len(report.Installed) == 0 {
+ t.Fatal("expected files to be reported as installed")
+ }
+ if len(report.Updated) != 0 || len(report.UserModified) != 0 {
+ t.Fatalf("first run should only install, got updated=%v userModified=%v", report.Updated, report.UserModified)
+ }
+
+ dest := filepath.Join(project, ".claude", "commands")
+ if _, err := os.Stat(filepath.Join(dest, "auto.md")); err != nil {
+ t.Fatalf("auto.md not written: %v", err)
+ }
+ if _, err := os.Stat(filepath.Join(dest, manifestName)); err != nil {
+ t.Fatalf("manifest not written: %v", err)
+ }
+}
+
+func TestInstallCommandsIsIdempotentWhenNothingChanged(t *testing.T) {
+ project := t.TempDir()
+ if _, err := installCommands(project); err != nil {
+ t.Fatalf("first install: %v", err)
+ }
+
+ report, err := installCommands(project)
+ if err != nil {
+ t.Fatalf("second install: %v", err)
+ }
+ if report.hasUpdates() || len(report.UserModified) != 0 {
+ t.Fatalf("expected no-op on second run, got %+v", report)
+ }
+}
+
+func TestInstallCommandsLeavesUserEditsAloneAndReportsThem(t *testing.T) {
+ project := t.TempDir()
+ if _, err := installCommands(project); err != nil {
+ t.Fatalf("first install: %v", err)
+ }
+ dest := filepath.Join(project, ".claude", "commands")
+ target := filepath.Join(dest, "auto.md")
+
+ installed, err := os.ReadFile(target)
+ if err != nil {
+ t.Fatalf("read installed file: %v", err)
+ }
+ edited := append(append([]byte{}, installed...), []byte("\n<!-- user edit -->\n")...)
+ if err := os.WriteFile(target, edited, 0o644); err != nil {
+ t.Fatalf("simulate user edit: %v", err)
+ }
+
+ report, err := installCommands(project)
+ if err != nil {
+ t.Fatalf("second install: %v", err)
+ }
+ found := false
+ for _, f := range report.UserModified {
+ if f == "auto.md" {
+ found = true
+ }
+ }
+ if !found {
+ t.Fatalf("expected auto.md to be reported as user-modified, got %+v", report)
+ }
+
+ after, err := os.ReadFile(target)
+ if err != nil {
+ t.Fatalf("read after install: %v", err)
+ }
+ if string(after) != string(edited) {
+ t.Fatal("user-modified file was overwritten, but it should have been left alone")
+ }
+}
+
+func TestInstallCommandsUpdatesFilesTheUserDidNotTouch(t *testing.T) {
+ project := t.TempDir()
+ if _, err := installCommands(project); err != nil {
+ t.Fatalf("first install: %v", err)
+ }
+ dest := filepath.Join(project, ".claude", "commands")
+ target := filepath.Join(dest, "auto.md")
+
+ // Simulate an older podcli version: rewrite both the file and its
+ // manifest entry to some other content, consistently, as if that's what
+ // an earlier install wrote. The file is still untouched by the user
+ // relative to that record, so the next install should treat it as safe
+ // to refresh to the current embedded content.
+ stale := []byte("# an older auto.md\n")
+ if err := os.WriteFile(target, stale, 0o644); err != nil {
+ t.Fatalf("simulate stale install: %v", err)
+ }
+ manifest := readManifest(dest)
+ manifest.Files["auto.md"] = sha256Hex(stale)
+ if err := writeManifest(dest, manifest); err != nil {
+ t.Fatalf("rewrite manifest: %v", err)
+ }
+
+ report, err := installCommands(project)
+ if err != nil {
+ t.Fatalf("second install: %v", err)
+ }
+ found := false
+ for _, f := range report.Updated {
+ if f == "auto.md" {
+ found = true
+ }
+ }
+ if !found {
+ t.Fatalf("expected auto.md to be reported as updated, got %+v", report)
+ }
+ if len(report.UserModified) != 0 {
+ t.Fatalf("unmodified file should not be reported as user-modified, got %v", report.UserModified)
+ }
+
+ after, err := os.ReadFile(target)
+ if err != nil {
+ t.Fatalf("read after install: %v", err)
+ }
+ if string(after) == string(stale) {
+ t.Fatal("stale file was not refreshed to the current embedded content")
+ }
+}
+
+func TestInstallCommandsAdoptsPreManifestFilesWithoutOverwriting(t *testing.T) {
+ project := t.TempDir()
+ dest := filepath.Join(project, ".claude", "commands")
+ if err := os.MkdirAll(dest, 0o755); err != nil {
+ t.Fatalf("mkdir: %v", err)
+ }
+ // A file that predates manifest tracking: some arbitrary content, no
+ // manifest entry for it at all.
+ legacy := filepath.Join(dest, "auto.md")
+ if err := os.WriteFile(legacy, []byte("# a pre-existing, possibly hand-edited auto.md\n"), 0o644); err != nil {
+ t.Fatalf("seed legacy file: %v", err)
+ }
+
+ report, err := installCommands(project)
+ if err != nil {
+ t.Fatalf("installCommands: %v", err)
+ }
+ for _, f := range report.Installed {
+ if f == "auto.md" {
+ t.Fatal("pre-manifest file must not be reported as freshly installed")
+ }
+ }
+ for _, f := range report.Updated {
+ if f == "auto.md" {
+ t.Fatal("pre-manifest file must not be overwritten on first sight")
+ }
+ }
+
+ content, err := os.ReadFile(legacy)
+ if err != nil {
+ t.Fatalf("read: %v", err)
+ }
+ if string(content) != "# a pre-existing, possibly hand-edited auto.md\n" {
+ t.Fatal("pre-manifest file content was changed, but it should have been adopted as-is")
+ }
+
+ // The legacy content here is not the stock embedded file, so it must not
+ // be adopted as a baseline. Doing so is what let the second run in
+ // TestInstallCommandsNeverOverwritesAPreManifestUserEditOnASecondRun
+ // clobber a genuine user edit.
+ manifest := readManifest(dest)
+ if manifest.Files["auto.md"] != "" {
+ t.Fatal("a pre-manifest file that doesn't match the stock content must not get a baseline hash recorded")
+ }
+ found := false
+ for _, f := range report.UserModified {
+ if f == "auto.md" {
+ found = true
+ }
+ }
+ if !found {
+ t.Fatalf("expected the non-stock pre-manifest file to be reported as user-modified, got %+v", report)
+ }
+}
+
+// Regression test for the bug where a pre-manifest, user-edited file was
+// adopted with its *current* hash as the baseline on first sight. The next
+// run then saw "unmodified relative to baseline" and overwrote it with the
+// embedded version, destroying the user's edit instead of leaving it alone.
+func TestInstallCommandsNeverOverwritesAPreManifestUserEditOnASecondRun(t *testing.T) {
+ project := t.TempDir()
+ dest := filepath.Join(project, ".claude", "commands")
+ if err := os.MkdirAll(dest, 0o755); err != nil {
+ t.Fatalf("mkdir: %v", err)
+ }
+ legacy := filepath.Join(dest, "auto.md")
+ edited := []byte("# a pre-existing, hand-edited auto.md\n")
+ if err := os.WriteFile(legacy, edited, 0o644); err != nil {
+ t.Fatalf("seed legacy file: %v", err)
+ }
+
+ if _, err := installCommands(project); err != nil {
+ t.Fatalf("first install: %v", err)
+ }
+ report, err := installCommands(project)
+ if err != nil {
+ t.Fatalf("second install: %v", err)
+ }
+ for _, f := range report.Updated {
+ if f == "auto.md" {
+ t.Fatal("a pre-manifest user edit must never be adopted as a baseline and then overwritten")
+ }
+ }
+
+ after, err := os.ReadFile(legacy)
+ if err != nil {
+ t.Fatalf("read after second install: %v", err)
+ }
+ if string(after) != string(edited) {
+ t.Fatal("pre-manifest user edit was overwritten on the second install run")
+ }
+}
+
+func TestFrontmatterDescriptionExtractsTheDescriptionField(t *testing.T) {
+ raw := "---\ndescription: Full pipeline from transcript to publish-ready package\nallowed-tools: Read\n---\n\n# body\n"
+ if got := frontmatterDescription(raw); got != "Full pipeline from transcript to publish-ready package" {
+ t.Fatalf("got %q", got)
+ }
+}
+
+func TestFrontmatterDescriptionIsEmptyWithoutFrontmatter(t *testing.T) {
+ if got := frontmatterDescription("# just a body\n"); got != "" {
+ t.Fatalf("got %q, want empty", got)
+ }
+}
+
+func TestCodexSkillContentTranslatesFrontmatterAndKeepsBody(t *testing.T) {
+ raw := "---\ndescription: does a thing\nallowed-tools: Read\n---\n\n# /auto\n\nbody text\n"
+ got := codexSkillContent("auto", raw)
+ if !strings.HasPrefix(got, "---\nname: auto\ndescription: does a thing\n---\n\n") {
+ t.Fatalf("unexpected frontmatter translation: %q", got)
+ }
+ if !strings.Contains(got, "# /auto\n\nbody text\n") {
+ t.Fatal("body was dropped or altered")
+ }
+}
+
+func TestInstallCodexSkillsWritesOneSkillDirPerCommand(t *testing.T) {
+ withHome(t)
+
+ if err := installCodexSkills(); err != nil {
+ t.Fatalf("installCodexSkills: %v", err)
+ }
+
+ skillsDir, err := codexSkillsDir()
+ if err != nil {
+ t.Fatalf("codexSkillsDir: %v", err)
+ }
+ for _, name := range Names() {
+ skillFile := filepath.Join(skillsDir, name, "SKILL.md")
+ data, err := os.ReadFile(skillFile)
+ if err != nil {
+ t.Fatalf("%s: %v", skillFile, err)
+ }
+ if !strings.HasPrefix(string(data), "---\nname: "+name+"\n") {
+ t.Fatalf("%s: missing expected frontmatter, got: %q", skillFile, string(data)[:min(60, len(data))])
+ }
+ }
+}
+
+func TestInstallCodexSkillsLeavesUserEditsAlone(t *testing.T) {
+ withHome(t)
+ if err := installCodexSkills(); err != nil {
+ t.Fatalf("first install: %v", err)
+ }
+ skillsDir, err := codexSkillsDir()
+ if err != nil {
+ t.Fatalf("codexSkillsDir: %v", err)
+ }
+ target := filepath.Join(skillsDir, "auto", "SKILL.md")
+ installed, err := os.ReadFile(target)
+ if err != nil {
+ t.Fatalf("read installed skill: %v", err)
+ }
+ edited := append(append([]byte{}, installed...), []byte("\n<!-- edited -->\n")...)
+ if err := os.WriteFile(target, edited, 0o644); err != nil {
+ t.Fatalf("simulate user edit: %v", err)
+ }
+
+ if err := installCodexSkills(); err != nil {
+ t.Fatalf("second install: %v", err)
+ }
+ after, err := os.ReadFile(target)
+ if err != nil {
+ t.Fatalf("read after reinstall: %v", err)
+ }
+ if string(after) != string(edited) {
+ t.Fatal("user-edited skill file was overwritten")
+ }
+}
+
+// Same regression as TestInstallCommandsNeverOverwritesAPreManifestUserEditOnASecondRun,
+// for the Codex skills installer: a skill file that predates manifest
+// tracking, and doesn't match the stock content, must never be adopted as a
+// baseline and then overwritten on a later install.
+func TestInstallCodexSkillsNeverOverwritesAPreManifestUserEditOnASecondRun(t *testing.T) {
+ withHome(t)
+ skillsDir, err := codexSkillsDir()
+ if err != nil {
+ t.Fatalf("codexSkillsDir: %v", err)
+ }
+ skillDir := filepath.Join(skillsDir, "auto")
+ if err := os.MkdirAll(skillDir, 0o755); err != nil {
+ t.Fatalf("mkdir: %v", err)
+ }
+ target := filepath.Join(skillDir, "SKILL.md")
+ edited := []byte("---\nname: auto\ndescription: hand-edited before manifest tracking existed\n---\n\nbody\n")
+ if err := os.WriteFile(target, edited, 0o644); err != nil {
+ t.Fatalf("seed legacy skill file: %v", err)
+ }
+
+ if err := installCodexSkills(); err != nil {
+ t.Fatalf("first install: %v", err)
+ }
+ if err := installCodexSkills(); err != nil {
+ t.Fatalf("second install: %v", err)
+ }
+
+ after, err := os.ReadFile(target)
+ if err != nil {
+ t.Fatalf("read after second install: %v", err)
+ }
+ if string(after) != string(edited) {
+ t.Fatal("pre-manifest user edit was overwritten on the second install run")
+ }
+}
diff --git a/cli/internal/provision/download_test.go b/cli/internal/provision/download_test.go
index c5a7a99..8d875f6 100644
--- a/cli/internal/provision/download_test.go
+++ b/cli/internal/provision/download_test.go
@@ -81,6 +81,23 @@ func TestAllModelsPinned(t *testing.T) {
if len(m.SHA256) != 64 {
t.Errorf("model %s has no pinned sha256", size)
}
+ if strings.Contains(m.URL, "/resolve/main/") {
+ t.Errorf("model %s uses a mutable 'main' URL: %s", size, m.URL)
+ }
+ }
+}
+
+func TestOmnilingualFilesPinned(t *testing.T) {
+ for name, m := range omnilingualFiles {
+ if len(m.SHA256) != 64 {
+ t.Errorf("omnilingual file %s has no pinned sha256", name)
+ }
+ if strings.Contains(m.URL, "/resolve/main/") {
+ t.Errorf("omnilingual file %s uses a mutable 'main' URL: %s", name, m.URL)
+ }
+ if !strings.Contains(m.URL, omnilingualRevision) {
+ t.Errorf("omnilingual file %s URL doesn't carry the pinned revision: %s", name, m.URL)
+ }
}
}
diff --git a/cli/internal/provision/provision.go b/cli/internal/provision/provision.go
index 2af7c71..143fd95 100644
--- a/cli/internal/provision/provision.go
+++ b/cli/internal/provision/provision.go
@@ -35,24 +35,82 @@ type artifactState struct {
Files map[string]string `json:"files"`
}
+// whisperCppRevision pins every ggml-*.bin download to this commit of
+// ggerganov/whisper.cpp so an upstream re-upload (or a force-push that
+// reshuffles "main") can't silently swap the bytes behind a pinned hash.
+const whisperCppRevision = "5359861c739e955e79d9a303bcbc70fb988958b1"
+
+func whisperCppURL(file string) string {
+ return "https://huggingface.co/ggerganov/whisper.cpp/resolve/" + whisperCppRevision + "/" + file
+}
+
var models = map[string]model{
"base": {
- URL: "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.bin",
+ URL: whisperCppURL("ggml-base.bin"),
SHA256: "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe",
},
+ "tiny": {
+ URL: whisperCppURL("ggml-tiny.bin"),
+ SHA256: "be07e048e1e599ad46341c8d2a135645097a538221678b7acdd1b1919c6e1b21",
+ },
"tiny.en": {
- URL: "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin",
+ URL: whisperCppURL("ggml-tiny.en.bin"),
SHA256: "921e4cf8686fdd993dcd081a5da5b6c365bfde1162e72b08d75ac75289920b1f",
},
"small": {
- URL: "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin",
+ URL: whisperCppURL("ggml-small.bin"),
SHA256: "1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b",
},
+ "medium": {
+ URL: whisperCppURL("ggml-medium.bin"),
+ SHA256: "6c14d5adee5f86394037b4e4e8b59f1673b6cee10e3cf0b11bbdbee79c156208",
+ },
+ "large-v3": {
+ URL: whisperCppURL("ggml-large-v3.bin"),
+ SHA256: "64d182b440b98d5203c4f9bd541544d84c605196c4f7b845dfa11fb23594d1e2",
+ },
+}
+
+// modelAliases maps a schema-level size name to the models key that actually
+// has a pinned download. "large" alone is ambiguous upstream (v1/v2/v3); we
+// always provision large-v3, the current best-accuracy build.
+var modelAliases = map[string]string{
+ "large": "large-v3",
}
-const vadURL = "https://huggingface.co/ggml-org/whisper-vad/resolve/main/ggml-silero-v5.1.2.bin"
+// whisperVADRevision pins the Silero VAD download the same way.
+const whisperVADRevision = "9ffd54a1e1ee413ddf265af9913beaf518d1639b"
+
+const vadURL = "https://huggingface.co/ggml-org/whisper-vad/resolve/" + whisperVADRevision + "/ggml-silero-v5.1.2.bin"
const vadSHA = "29940d98d42b91fbd05ce489f3ecf7c72f0a42f027e4875919a28fb4c04ea2cf"
+// omnilingualRevision pins the "omnilingual" engine's model download
+// (sherpa-onnx CTC, ~1600 languages, including ones whisper.cpp handles
+// poorly) the same way the ggml downloads are pinned. The 1B v2 model is
+// used because the 300M one takes no language hint and wrote accented
+// English in Arabic script on real podcast audio.
+const omnilingualRevision = "5243e02858d1428b8fbeeb5f7f1cc6fccf4d9433"
+
+func omnilingualURL(file string) string {
+ return "https://huggingface.co/csukuangfj2/sherpa-onnx-omnilingual-asr-1600-languages-1B-ctc-v2-int8-2026-02-05/resolve/" +
+ omnilingualRevision + "/" + file
+}
+
+var omnilingualFiles = map[string]model{
+ "model.int8.onnx": {
+ URL: omnilingualURL("model.int8.onnx"),
+ SHA256: "8af72da192fc2c8567c328d4f8059bdf47f182a0369077893c295ef39740c637",
+ },
+ "tokens.txt": {
+ URL: omnilingualURL("tokens.txt"),
+ SHA256: "7d99997ef207ff14c2cfe825f2aa037528ea250113cc3c6392bfe49326884ba6",
+ },
+ "LICENSE": {
+ URL: omnilingualURL("LICENSE"),
+ SHA256: "a70a523bafbb595c2844104feb313d204904dac91c3d186c05f22a10a71c7a94",
+ },
+}
+
func ModelPath(size string) string {
return filepath.Join(paths.ModelsDir(), "ggml-"+size+".bin")
}
@@ -61,6 +119,58 @@ func VADModelPath() string {
return filepath.Join(paths.ModelsDir(), "ggml-silero-v5.1.2.bin")
}
+// KnownModelSizes lists the whisper.cpp model sizes podcli knows how to
+// provision, for callers (doctor) that need to check every one that's
+// actually present rather than assuming a single size.
+func KnownModelSizes() []string {
+ sizes := make([]string, 0, len(models))
+ for size := range models {
+ sizes = append(sizes, size)
+ }
+ return sizes
+}
+
+// ModelSHA256 returns the pinned hash for a whisper.cpp model size.
+func ModelSHA256(size string) (string, bool) {
+ m, ok := models[size]
+ if !ok {
+ return "", false
+ }
+ return m.SHA256, true
+}
+
+// VADModelSHA256 returns the pinned hash for the VAD model.
+func VADModelSHA256() string {
+ return vadSHA
+}
+
+// OmnilingualFileHashes returns the pinned hash of each omnilingual model file
+// by filename.
+func OmnilingualFileHashes() map[string]string {
+ hashes := make(map[string]string, len(omnilingualFiles))
+ for name, m := range omnilingualFiles {
+ hashes[name] = m.SHA256
+ }
+ return hashes
+}
+
+func OmnilingualDir() string {
+ return filepath.Join(paths.ModelsDir(), "omnilingual")
+}
+
+// EnsureOmnilingualModel downloads (or verifies) the omnilingual ASR model
+// and its tokens file, returning their paths.
+func EnsureOmnilingualModel() (modelPath, tokensPath string, err error) {
+ dir := OmnilingualDir()
+ for name, m := range omnilingualFiles {
+ dest := filepath.Join(dir, name)
+ if err := download(m.URL, dest, m.SHA256, "omnilingual-"+name); err != nil {
+ return "", "", err
+ }
+ }
+ return filepath.Join(dir, "model.int8.onnx"), filepath.Join(dir, "tokens.txt"), nil
+}
+
func have(p string) bool {
if fi, err := os.Stat(p); err == nil && fi.Size() > 0 {
return true
@@ -69,12 +179,16 @@ func have(p string) bool {
}
func EnsureModel(size string) (string, error) {
- dest := ModelPath(size)
- m, ok := models[size]
+ resolved := size
+ if alias, ok := modelAliases[size]; ok {
+ resolved = alias
+ }
+ dest := ModelPath(resolved)
+ m, ok := models[resolved]
if !ok {
- return "", fmt.Errorf("unknown model size %q (known: base, tiny.en, small)", size)
+ return "", fmt.Errorf("unknown model size %q (known: base, tiny, tiny.en, small, medium, large)", size)
}
- if err := download(m.URL, dest, m.SHA256, "ggml-"+size); err != nil {
+ if err := download(m.URL, dest, m.SHA256, "ggml-"+resolved); err != nil {
return "", err
}
return dest, nil
diff --git a/cli/main.go b/cli/main.go
index b615dfb..9e4b275 100644
--- a/cli/main.go
+++ b/cli/main.go
@@ -4,11 +4,13 @@ package main
import (
"bytes"
+ "encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
+ "sort"
"strings"
"podcli/internal/backend"
@@ -33,7 +35,7 @@ func main() {
case "version", "--version", "-v":
fmt.Printf("podcli %s\n", Version)
case "doctor":
- doctor()
+ os.Exit(doctor(args[1:]))
case "update":
os.Exit(update.Run(Version))
case "uninstall":
@@ -182,7 +184,8 @@ func runEngine(args []string) int {
if wantsStudio(args) {
refreshStudioBundles()
}
- if transcribeEngine(args) == "whispercpp" {
+ switch transcribeEngine(args) {
+ case "whispercpp":
model, err := provision.EnsureModel(transcribeModel(args))
if err != nil {
fmt.Fprintln(os.Stderr, "podcli: provisioning model:", err)
@@ -190,6 +193,15 @@ func runEngine(args []string) int {
}
os.Setenv("PODCLI_ENGINE", "whispercpp")
os.Setenv("PODCLI_WHISPERCPP_MODEL", model)
+ case "omnilingual":
+ model, tokens, err := provision.EnsureOmnilingualModel()
+ if err != nil {
+ fmt.Fprintln(os.Stderr, "podcli: provisioning omnilingual model:", err)
+ return 1
+ }
+ os.Setenv("PODCLI_ENGINE", "omnilingual")
+ os.Setenv("PODCLI_OMNILINGUAL_MODEL", model)
+ os.Setenv("PODCLI_OMNILINGUAL_TOKENS", tokens)
}
code, err := engine.Run(args)
if err != nil {
@@ -200,12 +212,37 @@ func runEngine(args []string) int {
}
func transcribeModel(args []string) string {
+ fast := false
for _, arg := range args {
if arg == "--fast" {
- return "tiny.en"
+ fast = true
+ break
+ }
+ }
+ if !fast {
+ return "base"
+ }
+ // tiny.en is English-only; unset language runs auto-detection, which
+ // needs the multilingual tiny model same as any non-English request.
+ lang := strings.ToLower(transcribeLanguage(args))
+ if lang == "en" || lang == "english" {
+ return "tiny.en"
+ }
+ return "tiny"
+}
+
+// transcribeLanguage extracts --language/--language=<value> the same way
+// transcribeEngine extracts --engine.
+func transcribeLanguage(args []string) string {
+ lang := ""
+ for i, a := range args {
+ if a == "--language" && i+1 < len(args) {
+ lang = args[i+1]
+ } else if strings.HasPrefix(a, "--language=") {
+ lang = strings.TrimPrefix(a, "--language=")
}
}
- return "base"
+ return lang
}
func configCmd(args []string) int {
@@ -375,14 +412,24 @@ func setup(args []string) int {
}
if engine.MCPServer() != "" {
if mcpRegisteredToSelf() {
- fmt.Printf(" mcp: already registered\n")
+ fmt.Printf(" mcp: already registered with Claude Code\n")
} else if _, err := exec.LookPath("claude"); err != nil {
// Claude MCP registration is optional; Codex users do not need this.
} else if err := registerMCPServer(); err != nil {
- fmt.Fprintf(os.Stderr, " mcp: not registered (%v) - run `podcli mcp install`\n", err)
+ fmt.Fprintf(os.Stderr, " mcp: not registered with Claude Code (%v) - run `podcli mcp install`\n", err)
} else {
fmt.Printf(" mcp: registered with Claude Code\n")
}
+
+ if codexMCPRegisteredToSelf() {
+ fmt.Printf(" mcp: already registered with Codex\n")
+ } else if _, err := exec.LookPath("codex"); err != nil {
+ // Codex MCP registration is optional; Claude users do not need this.
+ } else if err := registerCodexMCPServer(); err != nil {
+ fmt.Fprintf(os.Stderr, " mcp: not registered with Codex (%v)\n", err)
+ } else {
+ fmt.Printf(" mcp: registered with Codex\n")
+ }
}
fmt.Println("Done.")
return 0
@@ -419,14 +466,72 @@ func mcpRegisteredToSelf() bool {
return err == nil && strings.Contains(string(out), self)
}
+// registerCodexMCPServer points Codex at this binary's `mcp` command. Unlike
+// Claude's `mcp add`, `codex mcp add` overwrites an existing entry by name,
+// so no remove-first step is needed to stay idempotent.
+func registerCodexMCPServer() error {
+ codex, err := exec.LookPath("codex")
+ if err != nil {
+ return fmt.Errorf("Codex CLI not found on PATH")
+ }
+ self, err := os.Executable()
+ if err != nil {
+ return err
+ }
+ if out, err := exec.Command(codex, "mcp", "add", "podcli", "--", self, "mcp").CombinedOutput(); err != nil {
+ return fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+ }
+ return nil
+}
+
+func codexMCPRegisteredToSelf() bool {
+ codex, err := exec.LookPath("codex")
+ if err != nil {
+ return false
+ }
+ self, err := os.Executable()
+ if err != nil {
+ return false
+ }
+ out, err := exec.Command(codex, "mcp", "get", "podcli").CombinedOutput()
+ return err == nil && strings.Contains(string(out), self)
+}
+
func mcpInstall() int {
- if err := registerMCPServer(); err != nil {
- self, _ := os.Executable()
- fmt.Fprintf(os.Stderr, "podcli: %v\n", err)
- fmt.Fprintf(os.Stderr, "Register manually: claude mcp add podcli -- %s mcp\n", self)
+ self, _ := os.Executable()
+ _, claudeErr := exec.LookPath("claude")
+ _, codexErr := exec.LookPath("codex")
+ registeredAny := false
+ failedAny := false
+
+ if claudeErr == nil {
+ if err := registerMCPServer(); err != nil {
+ fmt.Fprintf(os.Stderr, "podcli: %v\n", err)
+ fmt.Fprintf(os.Stderr, "Register manually: claude mcp add podcli -- %s mcp\n", self)
+ failedAny = true
+ } else {
+ fmt.Println("Registered podcli MCP server with Claude Code.")
+ registeredAny = true
+ }
+ }
+ if codexErr == nil {
+ if err := registerCodexMCPServer(); err != nil {
+ fmt.Fprintf(os.Stderr, "podcli: %v\n", err)
+ fmt.Fprintf(os.Stderr, "Register manually: codex mcp add podcli -- %s mcp\n", self)
+ failedAny = true
+ } else {
+ fmt.Println("Registered podcli MCP server with Codex.")
+ registeredAny = true
+ }
+ }
+
+ if !registeredAny && !failedAny {
+ fmt.Fprintln(os.Stderr, "podcli: neither Claude Code nor Codex CLI found on PATH")
+ return 1
+ }
+ if failedAny {
return 1
}
- fmt.Println("Registered podcli MCP server with Claude Code.")
return 0
}
@@ -677,17 +782,252 @@ func backendStamp(root string) string {
}
}
-func doctor() {
+// Check severity. "warn" surfaces a problem without counting toward
+// doctor's exit code. Used for MCP registration, which is a per-folder
+// (Claude) or optional (Codex-only users) setting, not evidence anything is
+// broken.
+const (
+ levelOK = "ok"
+ levelWarn = "warn"
+ levelFail = "fail"
+)
+
+// doctorCheck is one pass/fail probe doctor actually runs, as opposed to the
+// path/engine-resolution report above it, which only states what was found.
+type doctorCheck struct {
+ Name string `json:"name"`
+ OK bool `json:"ok"`
+ Level string `json:"level"` // "ok", "warn", or "fail" - only "fail" counts toward doctor's exit code
+ Detail string `json:"detail"`
+}
+
+type doctorReport struct {
+ Version string `json:"version"`
+ Paths map[string]string `json:"paths"`
+ Checks []doctorCheck `json:"checks"`
+ OK bool `json:"ok"`
+}
+
+func firstLine(s string) string {
+ s = strings.TrimSpace(s)
+ if i := strings.IndexByte(s, '\n'); i != -1 {
+ s = s[:i]
+ }
+ return s
+}
+
+// runCheck resolves a hermetic binary (falling back to PATH), then actually
+// runs it. A binary that resolves but fails to run (missing shared lib, bad
+// install) is exactly the failure mode `podcli doctor` otherwise can't see.
+func runCheck(name, hermetic, pathFallback string, args ...string) doctorCheck {
+ bin := hermetic
+ source := "hermetic"
+ if bin == "" {
+ if p, err := exec.LookPath(pathFallback); err == nil {
+ bin = p
+ source = "PATH"
+ }
+ }
+ if bin == "" {
+ return doctorCheck{Name: name, OK: false, Level: levelFail, Detail: "not found (hermetic or PATH)"}
+ }
+ out, err := exec.Command(bin, args...).CombinedOutput()
+ if err != nil {
+ return doctorCheck{Name: name, OK: false, Level: levelFail, Detail: fmt.Sprintf("%s (%s): %v: %s", bin, source, err, firstLine(string(out)))}
+ }
+ return doctorCheck{Name: name, OK: true, Level: levelOK, Detail: fmt.Sprintf("%s (%s): %s", bin, source, firstLine(string(out)))}
+}
+
+func pythonBackendCheck() doctorCheck {
+ root, ok := engine.BackendRoot()
+ if !ok {
+ return doctorCheck{Name: "python backend", OK: false, Level: levelFail, Detail: "backend not found (set PODCLI_BACKEND or run inside the repo)"}
+ }
+ // A representative sample, not every module: enough to catch "the
+ // interpreter can't even import the backend's own services" without
+ // reimplementing the whole import graph here.
+ modules := []string{"services.ai_cli", "services.multicam", "services.caption_renderer", "services.transcription"}
+ script := fmt.Sprintf("import sys; sys.path.insert(0, %q); import %s", root, strings.Join(modules, ", "))
+ out, err := exec.Command(engine.Python(), "-c", script).CombinedOutput()
+ if err != nil {
+ return doctorCheck{Name: "python backend", OK: false, Level: levelFail, Detail: fmt.Sprintf("%s: %v: %s", engine.Python(), err, firstLine(string(out)))}
+ }
+ return doctorCheck{Name: "python backend", OK: true, Level: levelOK, Detail: fmt.Sprintf("%s imports %s", engine.Python(), strings.Join(modules, ", "))}
+}
+
+// modelHashCheck reports ok=false when the file is absent: an unprovisioned
+// model is a valid state, not a failure.
+func modelHashCheck(name, p, want string) (doctorCheck, bool) {
+ if !fileExists(p) {
+ return doctorCheck{}, false
+ }
+ got, err := provision.Sha256File(p)
+ if err != nil {
+ return doctorCheck{Name: name, OK: false, Level: levelFail, Detail: fmt.Sprintf("could not hash %s: %v", p, err)}, true
+ }
+ if got != want {
+ return doctorCheck{Name: name, OK: false, Level: levelFail, Detail: fmt.Sprintf("%s hash mismatch: got %s, want %s", p, got, want)}, true
+ }
+ return doctorCheck{Name: name, OK: true, Level: levelOK, Detail: p}, true
+}
+
+func modelChecks() []doctorCheck {
+ var checks []doctorCheck
+ sizes := provision.KnownModelSizes()
+ sort.Strings(sizes)
+ for _, size := range sizes {
+ want, _ := provision.ModelSHA256(size)
+ if c, ok := modelHashCheck("model "+size, provision.ModelPath(size), want); ok {
+ checks = append(checks, c)
+ }
+ }
+ if c, ok := modelHashCheck("model vad", provision.VADModelPath(), provision.VADModelSHA256()); ok {
+ checks = append(checks, c)
+ }
+ files := provision.OmnilingualFileHashes()
+ names := make([]string, 0, len(files))
+ for name := range files {
+ names = append(names, name)
+ }
+ sort.Strings(names)
+ for _, name := range names {
+ p := filepath.Join(provision.OmnilingualDir(), name)
+ if c, ok := modelHashCheck("model omnilingual "+name, p, files[name]); ok {
+ checks = append(checks, c)
+ }
+ }
+ return checks
+}
+
+// mcpRegistrationChecks reports registration status for whichever agent
+// CLIs are on PATH. Registration is per-folder for Claude and optional for
+// Codex-only users, so an unregistered folder is normal, not broken: these
+// checks are warnings and don't count toward doctor's exit code, unless
+// none of the detected agents are registered anywhere this can see, in
+// which case nothing would actually work and that's a real failure.
+func mcpRegistrationChecks() []doctorCheck {
+ var checks []doctorCheck
+ if _, err := exec.LookPath("claude"); err == nil {
+ ok := mcpRegisteredToSelf()
+ checks = append(checks, doctorCheck{
+ Name: "mcp registration (Claude)",
+ OK: ok,
+ Detail: pick(ok, "registered", "not registered for this folder - run `podcli mcp install` here"),
+ })
+ }
+ if _, err := exec.LookPath("codex"); err == nil {
+ ok := codexMCPRegisteredToSelf()
+ checks = append(checks, doctorCheck{
+ Name: "mcp registration (Codex)",
+ OK: ok,
+ Detail: pick(ok, "registered", "not registered - run `podcli mcp install`"),
+ })
+ }
+ return levelRegistrationChecks(checks)
+}
+
+func pick(cond bool, ifTrue, ifFalse string) string {
+ if cond {
+ return ifTrue
+ }
+ return ifFalse
+}
+
+// levelRegistrationChecks turns the OK/not-OK result of each detected
+// agent's registration check into a severity: registered is "ok"; an
+// unregistered agent is just "warn" as long as at least one detected agent
+// is registered somewhere, since the MCP tools work through that one. If
+// none of the detected agents are registered anywhere, nothing would
+// actually work, which escalates every one of them to "fail".
+func levelRegistrationChecks(checks []doctorCheck) []doctorCheck {
+ anyRegistered := false
+ for _, c := range checks {
+ if c.OK {
+ anyRegistered = true
+ break
+ }
+ }
+ for i := range checks {
+ switch {
+ case checks[i].OK:
+ checks[i].Level = levelOK
+ case anyRegistered:
+ checks[i].Level = levelWarn
+ default:
+ checks[i].Level = levelFail
+ }
+ }
+ return checks
+}
+
+func runDoctorChecks() []doctorCheck {
+ var checks []doctorCheck
+ checks = append(checks, runCheck("ffmpeg", engine.FFmpeg(), "ffmpeg", "-version"))
+ checks = append(checks, runCheck("ffprobe", engine.FFprobe(), "ffprobe", "-version"))
+ checks = append(checks, runCheck("whisper-cli", engine.WhisperCLI(), "whisper-cli", "--help"))
+ checks = append(checks, runCheck("node", engine.Node(), "node", "--version"))
+ checks = append(checks, pythonBackendCheck())
+ checks = append(checks, modelChecks()...)
+ checks = append(checks, mcpRegistrationChecks()...)
+ return checks
+}
+
+// checksAllOK is doctor's exit-code rule: only a "fail" level counts.
+// "warn" (currently just MCP registration) surfaces in the output without
+// turning an otherwise-healthy install into a reported failure.
+func checksAllOK(checks []doctorCheck) bool {
+ for _, c := range checks {
+ if c.Level == levelFail {
+ return false
+ }
+ }
+ return true
+}
+
+func doctor(args []string) int {
+ asJSON := false
+ for _, a := range args {
+ if a == "--json" {
+ asJSON = true
+ }
+ }
+
+ pathInfo := map[string]string{
+ "home": paths.Home(),
+ "runtime": paths.RuntimeDir(),
+ "models": paths.ModelsDir(),
+ }
+ if out := os.Getenv("PODCLI_OUTPUT"); out != "" {
+ pathInfo["clips"] = out
+ } else if cwd, err := os.Getwd(); err == nil {
+ pathInfo["clips"] = filepath.Join(cwd, "podcli-clips")
+ }
+
+ checks := runDoctorChecks()
+ allOK := checksAllOK(checks)
+
+ if asJSON {
+ report := doctorReport{Version: Version, Paths: pathInfo, Checks: checks, OK: allOK}
+ data, err := json.MarshalIndent(report, "", " ")
+ if err != nil {
+ fmt.Fprintln(os.Stderr, "podcli: doctor:", err)
+ return 1
+ }
+ fmt.Println(string(data))
+ if !allOK {
+ return 1
+ }
+ return 0
+ }
+
fmt.Printf("podcli %s\n\n", Version)
fmt.Println("Paths")
- fmt.Printf(" home: %s\n", paths.Home())
- fmt.Printf(" runtime: %s\n", paths.RuntimeDir())
- fmt.Printf(" models: %s\n", paths.ModelsDir())
+ fmt.Printf(" home: %s\n", pathInfo["home"])
+ fmt.Printf(" runtime: %s\n", pathInfo["runtime"])
+ fmt.Printf(" models: %s\n", pathInfo["models"])
fmt.Printf(" presets/knowledge/assets/history/cache: %s (global - follow you everywhere)\n", paths.Home())
- if out := os.Getenv("PODCLI_OUTPUT"); out != "" {
- fmt.Printf(" clips: %s (PODCLI_OUTPUT)\n", out)
- } else if cwd, err := os.Getwd(); err == nil {
- fmt.Printf(" clips: %s (rendered into your working directory)\n", filepath.Join(cwd, "podcli-clips"))
+ if clips, ok := pathInfo["clips"]; ok {
+ fmt.Printf(" clips: %s\n", clips)
}
fmt.Println("\nEngine resolution")
if root, ok := engine.BackendRoot(); ok {
@@ -703,24 +1043,6 @@ func doctor() {
fmt.Printf(" backend: NOT FOUND (set PODCLI_BACKEND or run inside the repo)\n")
}
fmt.Printf(" python: %s\n", engine.Python())
- if ff := engine.FFmpeg(); ff != "" {
- fmt.Printf(" ffmpeg: %s (hermetic)\n", ff)
- } else {
- fmt.Printf(" ffmpeg: PATH fallback (not yet hermetic)\n")
- }
- if fp := engine.FFprobe(); fp != "" {
- fmt.Printf(" ffprobe: %s (hermetic)\n", fp)
- }
- if wc := engine.WhisperCLI(); wc != "" {
- fmt.Printf(" whisper: %s (hermetic)\n", wc)
- } else {
- fmt.Printf(" whisper: PATH fallback (install whisper-cli, or provisioned once hosted)\n")
- }
- if nd := engine.Node(); nd != "" {
- fmt.Printf(" node: %s (hermetic)\n", nd)
- } else {
- fmt.Printf(" node: PATH fallback (Web UI uses system Node, or run `podcli setup`)\n")
- }
if ss := engine.StudioServer(); ss != "" {
fmt.Printf(" studio: %s\n", ss)
} else {
@@ -736,16 +1058,25 @@ func doctor() {
} else {
fmt.Printf(" remotion: not provisioned (captions/thumbnails need a published release)\n")
}
- fmt.Println("\nModels")
- fmt.Printf(" base: %s\n", presence(provision.ModelPath("base")))
- fmt.Printf(" vad: %s\n", presence(provision.VADModelPath()))
-}
-func presence(p string) string {
- if fi, err := os.Stat(p); err == nil && fi.Size() > 0 {
- return fmt.Sprintf("%s (%s)", p, humanBytes(fi.Size()))
+ fmt.Println("\nChecks")
+ for _, c := range checks {
+ mark := "OK "
+ switch c.Level {
+ case levelWarn:
+ mark = "WARN"
+ case levelFail:
+ mark = "FAIL"
+ }
+ fmt.Printf(" [%s] %-28s %s\n", mark, c.Name, c.Detail)
}
- return "not provisioned - run `podcli setup`"
+
+ if !allOK {
+ fmt.Println("\npodcli doctor found problems above.")
+ return 1
+ }
+ fmt.Println("\nAll checks passed.")
+ return 0
}
func fileExists(p string) bool {
@@ -753,17 +1084,6 @@ func fileExists(p string) bool {
return err == nil
}
-func humanBytes(n int64) string {
- switch {
- case n >= 1<<20:
- return fmt.Sprintf("%d MB", n>>20)
- case n >= 1<<10:
- return fmt.Sprintf("%d KB", n>>10)
- default:
- return fmt.Sprintf("%d B", n)
- }
-}
-
func printHelp() {
fmt.Printf(`podcli %s - AI podcast clip generator
diff --git a/cli/main_test.go b/cli/main_test.go
index c98265b..f549aca 100644
--- a/cli/main_test.go
+++ b/cli/main_test.go
@@ -9,8 +9,17 @@ func TestTranscribeModel(t *testing.T) {
if got := transcribeModel([]string{"process", "episode.mp4"}); got != "base" {
t.Fatalf("default model = %q, want base", got)
}
- if got := transcribeModel([]string{"process", "episode.mp4", "--fast"}); got != "tiny.en" {
- t.Fatalf("fast model = %q, want tiny.en", got)
+ if got := transcribeModel([]string{"process", "episode.mp4", "--fast"}); got != "tiny" {
+ t.Fatalf("fast model with unset language = %q, want tiny (multilingual, no language conditioning)", got)
+ }
+ if got := transcribeModel([]string{"process", "episode.mp4", "--fast", "--language", "en"}); got != "tiny.en" {
+ t.Fatalf("fast english model = %q, want tiny.en", got)
+ }
+ if got := transcribeModel([]string{"process", "episode.mp4", "--fast", "--language", "ka"}); got != "tiny" {
+ t.Fatalf("fast non-english model = %q, want tiny", got)
+ }
+ if got := transcribeModel([]string{"process", "episode.mp4", "--fast", "--language=ka"}); got != "tiny" {
+ t.Fatalf("fast non-english model (= form) = %q, want tiny", got)
}
}
@@ -28,3 +37,18 @@ func TestTranscribeEngineAssemblyAI(t *testing.T) {
t.Fatalf("engine = %q, want assemblyai", got)
}
}
+
+func TestTranscribeEngineOmnilingual(t *testing.T) {
+ old, ok := os.LookupEnv("PODCLI_ENGINE")
+ t.Cleanup(func() {
+ if ok {
+ os.Setenv("PODCLI_ENGINE", old)
+ } else {
+ os.Unsetenv("PODCLI_ENGINE")
+ }
+ })
+ os.Unsetenv("PODCLI_ENGINE")
+ if got := transcribeEngine([]string{"process", "episode.mp4", "--engine", "omnilingual"}); got != "omnilingual" {
+ t.Fatalf("engine = %q, want omnilingual", got)
+ }
+}
diff --git a/docs/clips-schema.md b/docs/clips-schema.md
index 5c991a0..7ad2a75 100644
--- a/docs/clips-schema.md
+++ b/docs/clips-schema.md
@@ -26,6 +26,9 @@ The TS type is `ClipHistoryEntry` in `src/models/index.ts`. Keep both languages
| `created_at` | string (ISO) | TS render | |
| `content_type` | string? | TS render | Carried from the suggestion (`guest_story`, `hot_take`, …). Undefined when no suggestion was used. |
| `transcript_slice` | string? | TS render | Plain text the clip says. The session transcript is overwritten, so this is the **only durable** copy — never reconstruct it later. |
+| `payoff` | string? | TS render | Carried from the suggestion. The clip's payoff line; grounds thumbnail headline copy. |
+| `context_line` | string? | TS render | Carried from the suggestion. The question the clip answers, if any. |
+| `preview_text` | string? | TS render | Carried from the suggestion. The clip's verbatim opening line. |
| `thumbnail_config` | object? | TS render / edit | Per-clip thumbnail settings (`line1`, `line2`, `image_path`, `timestamp`, `preview_path`, `variations`). |
| `youtube_video_id` | string? | Phase 2 | Set when a published video is linked. |
| `metrics` | object? | Phase 2 | `{ views?, retention?, ctr?, impressions?, fetched_at? }`. retention/ctr are 0–100. |
diff --git a/docs/limitations.md b/docs/limitations.md
new file mode 100644
index 0000000..d640dd4
--- /dev/null
+++ b/docs/limitations.md
@@ -0,0 +1,88 @@
+# Limitations
+
+What each stage can and cannot do today. This is a plain statement of current
+behavior, not a roadmap.
+
+## Multicam sync
+
+- Sync is audio-only: coarse cross-correlation of onset envelopes, then a
+ fine GCC-PHAT match with linear clock-drift correction
+ (`backend/services/multicam_signal.py`). There is no timecode-based sync;
+ embedded timecode is read as session metadata only
+ (`backend/services/multicam.py:278-282`).
+- A camera with no audio track cannot be synced automatically. Its offset
+ must be set by hand (`backend/services/multicam.py:1082`).
+- Remote call recordings (Zoom, Riverside, one file per person) are assumed
+ to start together, since there's no shared audio to cross-correlate
+ against.
+
+## Auto-cut to speaker
+
+- Speaker assignment comes from per-person microphone levels, or from
+ imported diarization turns. There is no video-based speaker detection
+ (face tracking, lip sync).
+- Short interjections are suppressed so a quick "mm-hm" doesn't steal the
+ cut, which means genuine quick back-and-forth exchanges can read as
+ slower-paced than they were.
+- Call-recording layouts (one file per person as a split screen, one
+ gallery recording split into a tile per camera) render to MP4 fine but
+ cannot export to Premiere XML or FCPXML yet (`src/server.ts`,
+ `manage_multicam` tool description).
+
+## Transcription
+
+- Four engines: `whisper-py` (OpenAI Whisper, default), `whispercpp`
+ (local, no network), `omnilingual` (local, ~1600 languages) and
+ `assemblyai` (`backend/services/engines.py`).
+- `omnilingual` runs Meta's Omnilingual ASR 1B CTC model on the CPU. The
+ first run downloads 1 GB. It returns lowercase text with no punctuation,
+ so captions and sentence splits come from pauses alone. It takes no
+ language hint; test a 40 s sample before a full run, and compare it with
+ `compare_transcription_engines`.
+- Speaker diarization runs on the `whisper-py` and `assemblyai` paths.
+ `whispercpp` and `omnilingual` never produce speaker labels. Every word comes back with
+ `speaker: null` (`backend/services/transcription_whispercpp.py:67,215`).
+- Diarization needs `HF_TOKEN` set (see `docs/configuration.md`). Without
+ it, or if diarization fails for any other reason, transcription still
+ completes. It just comes back with no speaker labels, degrading
+ silently rather than failing the job.
+- Language support follows whichever engine is in use; podcli does not add
+ or restrict language coverage beyond what Whisper, whisper.cpp,
+ Omnilingual ASR, or AssemblyAI support natively.
+
+## Captions
+
+- Four built-in styles: `hormozi`, `karaoke`, `subtle`, `branded`
+ (`backend/services/caption_renderer.py`).
+- Burn-in (rendered into the video via ASS/libass) is the primary path.
+ Soft `.srt` export also exists, and `.srt`/`.vtt` transcripts can be
+ imported directly.
+- Non-Latin scripts (Georgian, Russian, etc.) render through font-fallback
+ stacks, not through the primary brand font. The brand font only draws
+ Latin glyphs. Expect a visibly different typeface on non-Latin captions
+ until a matching family is bundled for the style in use.
+
+## Thumbnails
+
+- Layout and copy are generated together: an LLM writes the HTML/CSS, and
+ Playwright renders it (`backend/services/thumbnail_ai.py`).
+- Split-screen/multicam source frames are detected and cropped to the half
+ containing the speaking face.
+- The `pair` layout places two people side by side, each cut from their own
+ frame of the clip. It tells people apart only by seat (two faces in one
+ shot or two call panes) or by a multicam edit whose render is the clip's
+ source. A single camera that cuts between people falls back to one face.
+- Without a multicam edit, the guest is the speaker who talks more in the
+ clip. Without diarized speakers mapped to seats, sides follow the footage.
+ `--swap` or explicit images fix either case.
+- Headline copy is written to match the clip's theme, not quoted verbatim
+ from the transcript. There's no automated check that the headline's
+ claim matches what's actually said in the clip.
+
+## Exports
+
+- Supported targets: DaVinci Resolve FCPXML, Premiere XML/FCPXML (via
+ `manage_multicam`), and the podcli cloud editor.
+- The cloud editor requires a podcli account and an active Pro plan.
+- Call-recording-derived layouts (split screen, gallery tiles) cannot
+ export to Premiere or FCPXML. See Auto-cut above.
diff --git a/docs/thumbnail-config.example.json b/docs/thumbnail-config.example.json
index 27318e7..f0ed77b 100644
--- a/docs/thumbnail-config.example.json
+++ b/docs/thumbnail-config.example.json
@@ -60,6 +60,12 @@
"gradient_bottom_end_color": "#0D0D0D",
"gradient_bottom_fade_point": "70%",
+ "_comment_layout": "single draws one face behind the headline. pair draws two people from the clip side by side, the guest on the left and the host on the right, over the top pair_photo_height of the canvas, with the headline centred at pair_box_y. pair falls back to single when podcli cannot tell two people apart in the clip.",
+ "layout": "single",
+ "pair_photo_height": "70%",
+ "pair_box_y": "85%",
+ "pair_divider_width": 4,
+
"_comment_logo": "Position one of: top-left, top-right, top-center, bottom-center, none. Logo image lives under .podcli/assets/ (register via manage_assets).",
"logo_position": "bottom-center",
"logo_height": "50px",
diff --git a/remotion/src/Audiogram.tsx b/remotion/src/Audiogram.tsx
index 1749a82..1c909f3 100644
--- a/remotion/src/Audiogram.tsx
+++ b/remotion/src/Audiogram.tsx
@@ -5,6 +5,7 @@ import { KaraokeCaptions } from "./components/KaraokeCaptions";
import { SubtleCaptions } from "./components/SubtleCaptions";
import { BrandedCaptions } from "./components/BrandedCaptions";
import type { Word, CaptionStyle } from "./types";
+import { safeUpper } from "./text";
/**
* What an episode that was never filmed looks like.
@@ -94,10 +95,12 @@ export const Audiogram: React.FC<AudiogramProps> = ({
fontWeight: 700,
fontSize: Math.round(height * 0.022),
letterSpacing: "0.08em",
- textTransform: "uppercase",
+ // Resolved below via safeUpper. CSS text-transform would still
+ // remap caseless scripts like Georgian to a different alphabet.
+ textTransform: "none",
}}
>
- {title}
+ {safeUpper(title)}
</div>
)}
diff --git a/remotion/src/components/HormoziCaptions.tsx b/remotion/src/components/HormoziCaptions.tsx
index b52df0c..b36206f 100644
--- a/remotion/src/components/HormoziCaptions.tsx
+++ b/remotion/src/components/HormoziCaptions.tsx
@@ -8,6 +8,7 @@ import { captionScale } from "../types";
import { buildChunks, activeChunkAt } from "../chunks";
import { MOTION, motionAt } from "../motion";
import type { Motion } from "../motion";
+import { safeUpper } from "../text";
interface Props {
words: Word[];
@@ -73,7 +74,7 @@ export const HormoziCaptions: React.FC<Props> = ({
{activeChunk.words.map((word, i) => {
const isActive = currentTime >= word.start && currentTime < word.end;
const text = style.uppercase
- ? word.word.toUpperCase()
+ ? safeUpper(word.word)
: word.word;
// The word being spoken still wins: the sweep is what this style is.
// Emphasis colours it for the rest of the chunk, which is the part
diff --git a/remotion/src/components/Scene.tsx b/remotion/src/components/Scene.tsx
index a5e9340..a370732 100644
--- a/remotion/src/components/Scene.tsx
+++ b/remotion/src/components/Scene.tsx
@@ -10,6 +10,7 @@ import {
sizeOf,
} from "../scene";
import type { Align, Block, Gap, Layout, Size, Tone } from "../scene";
+import { safeUpper } from "../text";
/**
* Blocks, drawn.
@@ -92,13 +93,17 @@ const Piece: React.FC<{ block: Block; paint: Paint }> = ({ block, paint }) => {
fontWeight: WEIGHT[size],
lineHeight: LINE_HEIGHT[size],
letterSpacing: TRACKING[size] * unit,
- textTransform: block.caps ? "uppercase" : undefined,
+ // Resolved per-run below via safeUpper. CSS text-transform would
+ // still remap caseless scripts like Georgian to a different alphabet.
+ textTransform: "none",
color: toneOf(brand, accent, block.tone, size === "xs" ? "muted" : "ink"),
textAlign: block.align ?? "start",
}}
>
{emphasisRuns(block.text, block.emphasis).map((run, i) => (
- <span key={i} style={run.mark ? { color: accent } : undefined}>{run.text}</span>
+ <span key={i} style={run.mark ? { color: accent } : undefined}>
+ {block.caps ? safeUpper(run.text) : run.text}
+ </span>
))}
</div>
);
diff --git a/remotion/src/components/TopicChip.tsx b/remotion/src/components/TopicChip.tsx
index 2606a00..226cb05 100644
--- a/remotion/src/components/TopicChip.tsx
+++ b/remotion/src/components/TopicChip.tsx
@@ -2,6 +2,7 @@ import React from "react";
import { useVideoConfig } from "remotion";
import { captionScale, FONT, LOGO_EDGE, LOGO_INSET } from "../types";
import type { LogoPosition } from "../types";
+import { safeUpper } from "../text";
export interface TopicChipProps {
/** What the clip is about, in two or three words. */
@@ -62,7 +63,9 @@ export const TopicChip: React.FC<TopicChipProps> = ({
fontSize: 30 * s,
fontWeight: 700,
letterSpacing: 3 * s,
- textTransform: "uppercase",
+ // Resolved below via safeUpper, not here. CSS text-transform would
+ // still remap caseless scripts like Georgian to a different alphabet.
+ textTransform: "none",
color,
background,
...(padded
@@ -70,7 +73,7 @@ export const TopicChip: React.FC<TopicChipProps> = ({
: { textShadow: "0 2px 12px rgba(0,0,0,0.8)" }),
}}
>
- {label}
+ {safeUpper(label)}
</div>
);
};
diff --git a/remotion/src/text.test.ts b/remotion/src/text.test.ts
new file mode 100644
index 0000000..e4951e5
--- /dev/null
+++ b/remotion/src/text.test.ts
@@ -0,0 +1,24 @@
+import { describe, expect, it } from "vitest";
+import { safeUpper } from "./text";
+
+describe("safeUpper", () => {
+ it("uppercases plain ASCII text", () => {
+ expect(safeUpper("hello world")).toBe("HELLO WORLD");
+ });
+
+ it("leaves Georgian Mkhedruli text unchanged instead of switching to Mtavruli", () => {
+ const georgian = "მიშა";
+ expect(safeUpper(georgian)).toBe(georgian);
+ // Sanity check the bug this guards against: toUpperCase() alone does
+ // remap Mkhedruli to Mtavruli.
+ expect(georgian.toUpperCase()).not.toBe(georgian);
+ });
+
+ it("uppercases the Latin parts of a mixed-script string and leaves Georgian alone", () => {
+ expect(safeUpper("hello მიშა")).toBe("HELLO მიშა");
+ });
+
+ it("passes through empty strings", () => {
+ expect(safeUpper("")).toBe("");
+ });
+});
diff --git a/remotion/src/text.ts b/remotion/src/text.ts
new file mode 100644
index 0000000..8dae0a8
--- /dev/null
+++ b/remotion/src/text.ts
@@ -0,0 +1,34 @@
+// Scripts with no case distinction whose letters Unicode nonetheless assigns
+// an uppercase mapping for display styling (Georgian Mkhedruli -> Mtavruli).
+// String.prototype.toUpperCase() applies that mapping uncritically, so a
+// caption style that uppercases for emphasis silently switches alphabets for
+// these scripts instead of just emphasizing them.
+const CASELESS_SCRIPT_RANGES: Array<[number, number]> = [
+ [0x10a0, 0x10ff], // Georgian (Mkhedruli, Asomtavruli)
+ [0x1c90, 0x1cbf], // Georgian Extended (Mtavruli)
+ [0x2d00, 0x2d2f], // Georgian Supplement
+];
+
+function isCaselessScriptChar(ch: string): boolean {
+ const cp = ch.codePointAt(0) ?? 0;
+ return CASELESS_SCRIPT_RANGES.some(([lo, hi]) => cp >= lo && cp <= hi);
+}
+
+/**
+ * Uppercase text, except for scripts where Unicode's uppercase mapping
+ * would change the alphabet rather than just the case (see above).
+ */
+export function safeUpper(text: string): string {
+ if (!text) return text;
+ let hasCaseless = false;
+ for (const ch of text) {
+ if (isCaselessScriptChar(ch)) {
+ hasCaseless = true;
+ break;
+ }
+ }
+ if (!hasCaseless) return text.toUpperCase();
+ return Array.from(text)
+ .map((ch) => (isCaselessScriptChar(ch) ? ch : ch.toUpperCase()))
+ .join("");
+}
diff --git a/scripts/gen-docs-manifest.mjs b/scripts/gen-docs-manifest.mjs
index 2405e0c..9ff419c 100644
--- a/scripts/gen-docs-manifest.mjs
+++ b/scripts/gen-docs-manifest.mjs
@@ -25,14 +25,17 @@ function collectToolNames(sources) {
}
}
for (const src of sources) {
- for (const m of src.matchAll(/server\.tool\(\s*(?:"([^"]+)"|(\w+)\.name)/g)) {
+ // server.registerTool (the non-deprecated API, needed for a strict
+ // input schema that server.tool's raw-shape shorthand can't express)
+ // takes its name the same way: as the first positional argument.
+ for (const m of src.matchAll(/server\.(?:registerTool|tool)\(\s*(?:"([^"]+)"|(\w+)\.name)/g)) {
registrations += 1;
if (m[1]) names.push(m[1]);
else if (defs.has(m[2])) names.push(defs.get(m[2]));
}
}
if (names.length !== registrations) {
- console.error(`resolved ${names.length} tool names for ${registrations} server.tool( registrations`);
+ console.error(`resolved ${names.length} tool names for ${registrations} server.tool(/server.registerTool( registrations`);
process.exit(1);
}
return names.sort();
diff --git a/scripts/release-hygiene.mjs b/scripts/release-hygiene.mjs
new file mode 100644
index 0000000..6d84fe8
--- /dev/null
+++ b/scripts/release-hygiene.mjs
@@ -0,0 +1,136 @@
+// Catches what a build step can't: things that should never reach a commit.
+// Runs over `git ls-files` (tracked files only) so untracked local scratch
+// (data/, podcli-clips/, .env) never trips it.
+import { execSync } from "child_process";
+import { readFileSync, statSync } from "fs";
+import { join, dirname, extname } from "path";
+import { fileURLToPath, pathToFileURL } from "url";
+
+const root = join(dirname(fileURLToPath(import.meta.url)), "..");
+
+// Real source recordings/renders have no business in git history. They
+// belong in data/ or podcli-clips/, both gitignored. Icons, logos, and other
+// still-image UI assets are handled by the size check below instead, since
+// those are legitimately tracked.
+const MEDIA_EXTENSIONS = new Set([
+ ".mp4", ".mov", ".mkv", ".avi", ".webm", ".wmv", ".flv",
+ ".mp3", ".wav", ".flac", ".m4a", ".aac", ".ogg",
+]);
+
+// Files over 1MB that are legitimately tracked. Keep this list short, add
+// to it only when the file truly belongs in history (a model weight, a brand
+// asset), not as a workaround for a one-off mistake.
+const LARGE_FILE_ALLOWLIST = new Set([
+ "backend/models/yamnet.onnx", // audio-energy model analyze_energy depends on
+ "public/promo.gif", // README hero asset
+ "remotion/public/style/detail-kraft.png", // riso pack texture
+]);
+
+const MAX_SIZE_BYTES = 1024 * 1024;
+
+const SECRET_PATTERNS = [
+ { name: "AWS access key", re: /\bAKIA[0-9A-Z]{16}\b/ },
+ { name: "private key block", re: /-----BEGIN (?:RSA |EC |OPENSSH |DSA )?PRIVATE KEY-----/ },
+ { name: "GitHub token", re: /\bghp_[A-Za-z0-9]{36}\b/ },
+ { name: "Slack token", re: /\bxox[baprs]-[A-Za-z0-9-]{10,}\b/ },
+ { name: "OpenAI-style secret key", re: /\bsk-[A-Za-z0-9]{32,}\b/ },
+];
+
+// .env.example documents the shape of the file without real values, keep it.
+const ENV_FILE_RE = /(^|\/)\.env(\..+)?$/;
+const ENV_ALLOWLIST = new Set([".env.example"]);
+
+// cwd is parameterized (default: this repo's root) so the test suite can
+// point it at a throwaway repo instead of git ls-files-ing the real one.
+export function trackedFiles(cwd = root) {
+ // Plain `git ls-files` C-quotes any path with a non-ASCII byte (e.g.
+ // "\346\226\207.mp4") instead of printing it raw, so a filename like that
+ // never matches a real path on disk and every check below silently skips
+ // it. `-z` prints paths NUL-separated with no quoting at all.
+ return execSync("git ls-files -z", { cwd, encoding: "utf8" })
+ .split("\0")
+ .filter(Boolean);
+}
+
+export function checkMediaFiles(files) {
+ return files
+ .filter((f) => MEDIA_EXTENSIONS.has(extname(f).toLowerCase()))
+ .map((f) => `tracked media file: ${f}`);
+}
+
+function checkLargeFiles(files) {
+ const errors = [];
+ for (const f of files) {
+ if (LARGE_FILE_ALLOWLIST.has(f)) continue;
+ let size;
+ try {
+ size = statSync(join(root, f)).size;
+ } catch {
+ continue; // deleted-but-staged path, or a submodule gitlink; not our concern here
+ }
+ if (size > MAX_SIZE_BYTES) {
+ errors.push(`file over 1MB, not on the allowlist: ${f} (${(size / 1024 / 1024).toFixed(2)}MB)`);
+ }
+ }
+ return errors;
+}
+
+export function checkEnvFiles(files) {
+ return files
+ .filter((f) => ENV_FILE_RE.test(f) && !ENV_ALLOWLIST.has(f))
+ .map((f) => `tracked .env file: ${f}`);
+}
+
+// Binary files (images, the onnx model) throw on readFileSync(..., "utf8")
+// often enough that it's simpler to just skip anything non-text-shaped.
+const TEXT_EXTENSIONS = new Set([
+ ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs", ".py", ".go", ".sh",
+ ".md", ".json", ".yml", ".yaml", ".toml", ".txt", ".html", ".css",
+]);
+
+export function matchSecretPatterns(content) {
+ return SECRET_PATTERNS.filter(({ re }) => re.test(content)).map(({ name }) => name);
+}
+
+// Known fixture files that intentionally contain fake secrets shaped like
+// the real thing, to test matchSecretPatterns itself. Everything else still
+// gets scanned. A real secret pasted into some other test is still a leak.
+const SECRET_SCAN_ALLOWLIST = new Set(["scripts/release-hygiene.test.mjs"]);
+
+function checkSecretPatterns(files) {
+ const errors = [];
+ for (const f of files) {
+ if (SECRET_SCAN_ALLOWLIST.has(f)) continue;
+ if (!TEXT_EXTENSIONS.has(extname(f).toLowerCase())) continue;
+ let content;
+ try {
+ content = readFileSync(join(root, f), "utf8");
+ } catch {
+ continue;
+ }
+ for (const name of matchSecretPatterns(content)) {
+ errors.push(`possible ${name} in ${f}`);
+ }
+ }
+ return errors;
+}
+
+// Guarded so src/release-hygiene.test.ts can import the pure check functions
+// above without running the whole CLI (and without a stray process.exit).
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+ const files = trackedFiles();
+ const errors = [
+ ...checkMediaFiles(files),
+ ...checkLargeFiles(files),
+ ...checkEnvFiles(files),
+ ...checkSecretPatterns(files),
+ ];
+
+ if (errors.length) {
+ console.error("Release hygiene check failed:");
+ for (const e of errors) console.error(" -", e);
+ process.exit(1);
+ }
+
+ console.log(`Release hygiene check passed: ${files.length} tracked files checked.`);
+}
diff --git a/scripts/release-hygiene.test.mjs b/scripts/release-hygiene.test.mjs
new file mode 100644
index 0000000..24dcadd
--- /dev/null
+++ b/scripts/release-hygiene.test.mjs
@@ -0,0 +1,82 @@
+import { describe, it, expect } from "vitest";
+import { execSync } from "child_process";
+import { mkdtempSync, writeFileSync, rmSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+import { checkMediaFiles, checkEnvFiles, matchSecretPatterns, trackedFiles } from "./release-hygiene.mjs";
+
+describe("release-hygiene: tracked media files", () => {
+ it("flags audio/video extensions", () => {
+ expect(checkMediaFiles(["episode.mp4", "raw.wav"])).toHaveLength(2);
+ });
+
+ it("leaves still-image assets alone (handled by the size check instead)", () => {
+ expect(checkMediaFiles(["logo.png", "icon.svg"])).toEqual([]);
+ });
+});
+
+describe("release-hygiene: tracked .env files", () => {
+ it("flags .env and dotted variants", () => {
+ expect(checkEnvFiles([".env", ".env.local", "backend/.env"])).toHaveLength(3);
+ });
+
+ it("allows .env.example", () => {
+ expect(checkEnvFiles([".env.example"])).toEqual([]);
+ });
+});
+
+describe("release-hygiene: trackedFiles", () => {
+ function initRepo() {
+ const dir = mkdtempSync(join(tmpdir(), "release-hygiene-"));
+ execSync("git init -q", { cwd: dir });
+ execSync('git config user.email "[email protected]"', { cwd: dir });
+ execSync('git config user.name "Test"', { cwd: dir });
+ return dir;
+ }
+
+ it("lists a file whose name has a non-ASCII character", () => {
+ const dir = initRepo();
+ try {
+ const name = "文字.txt"; // CJK characters, non-ASCII bytes in the path
+ writeFileSync(join(dir, name), "content");
+ execSync("git add -A", { cwd: dir });
+ execSync('git commit -q -m "add file"', { cwd: dir });
+
+ const files = trackedFiles(dir);
+ expect(files).toContain(name);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ it("lists plain-ASCII filenames alongside the non-ASCII one", () => {
+ const dir = initRepo();
+ try {
+ writeFileSync(join(dir, "école.txt"), "content"); // accented e
+ writeFileSync(join(dir, "plain.txt"), "content");
+ execSync("git add -A", { cwd: dir });
+ execSync('git commit -q -m "add files"', { cwd: dir });
+
+ const files = trackedFiles(dir);
+ expect(files).toEqual(expect.arrayContaining(["école.txt", "plain.txt"]));
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+});
+
+describe("release-hygiene: secret patterns", () => {
+ it("flags an AWS access key", () => {
+ expect(matchSecretPatterns("key = AKIAABCDEFGHIJKLMNOP")).toContain("AWS access key");
+ });
+
+ it("flags a private key block", () => {
+ expect(matchSecretPatterns("-----BEGIN RSA PRIVATE KEY-----\nabc\n-----END RSA PRIVATE KEY-----")).toContain(
+ "private key block",
+ );
+ });
+
+ it("leaves ordinary source text alone", () => {
+ expect(matchSecretPatterns("const apiKey = process.env.API_KEY;")).toEqual([]);
+ });
+});
diff --git a/src/config/paths.ts b/src/config/paths.ts
index 4c46f7a..ce4e98e 100644
--- a/src/config/paths.ts
+++ b/src/config/paths.ts
@@ -81,6 +81,7 @@ export const paths = {
clipsHistory: join(home, "history", "clips.json"),
knowledge: join(home, "knowledge"),
uiState: join(home, "ui-state.json"),
+ episodeDecisions: join(home, "episode-decisions.json"),
corrections: join(home, "corrections.json"),
thumbnailConfig: join(home, "thumbnail-config.json"),
integrations: join(home, "integrations.json"),
diff --git a/src/create-clip-offline-identity.test.ts b/src/create-clip-offline-identity.test.ts
new file mode 100644
index 0000000..4052bb7
--- /dev/null
+++ b/src/create-clip-offline-identity.test.ts
@@ -0,0 +1,84 @@
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// create_clip/batch_create_clips fill transcript_words from session state
+// when the caller omits it (e.g. clip_number-only calls). handleCreateClip
+// and handleBatchClips only run the "transcript belongs to a different
+// video" guard when transcript_words arrives null, so filling it from state
+// in server.ts has to run that guard itself. Otherwise a render against a
+// video that was swapped in after the transcript was generated goes through
+// unchecked. These tests never need the studio server: the check runs
+// before server.ts even tries to reach it.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-offline-identity-"));
+const savedHome = process.env.PODCLI_HOME;
+const savedData = process.env.PODCLI_DATA;
+const savedPort = process.env.PODCLI_PORT;
+
+let createServer: typeof import("./server.js").createServer;
+let videoPath: string;
+let uiStatePath: string;
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ process.env.PODCLI_PORT = "18425"; // force readUIState to fall back to disk
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+ uiStatePath = join(process.env.PODCLI_HOME, "ui-state.json");
+
+ ({ createServer } = await import("./server.js"));
+ videoPath = join(tmp, "episode.mp4");
+ writeFileSync(videoPath, "fake video bytes");
+});
+
+afterAll(() => {
+ if (savedHome === undefined) delete process.env.PODCLI_HOME;
+ else process.env.PODCLI_HOME = savedHome;
+ if (savedData === undefined) delete process.env.PODCLI_DATA;
+ else process.env.PODCLI_DATA = savedData;
+ if (savedPort === undefined) delete process.env.PODCLI_PORT;
+ else process.env.PODCLI_PORT = savedPort;
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function getHandler(name: string) {
+ const server = createServer() as unknown as {
+ _registeredTools: Record<string, { handler: (args: unknown, extra: unknown) => Promise<{ content: { text: string }[]; isError?: boolean }> }>;
+ };
+ return server._registeredTools[name].handler;
+}
+
+function writeMismatchedState() {
+ writeFileSync(
+ uiStatePath,
+ JSON.stringify({
+ videoPath,
+ filePath: videoPath,
+ suggestions: [{ start_second: 10, end_second: 20, title: "a moment" }],
+ deselectedIndices: [],
+ transcript: { words: [{ word: "hi", start: 0, end: 1 }] },
+ // Deliberately wrong size so it never matches the real file on disk.
+ transcriptVideoIdentity: { path: videoPath, size: 999999, mtimeMs: 1 },
+ }),
+ );
+}
+
+describe("create_clip offline identity guard", () => {
+ it("rejects clip_number-only calls when the session transcript doesn't match the video", async () => {
+ writeMismatchedState();
+ const handler = getHandler("create_clip");
+ const result = await handler({ clip_number: 1 }, {});
+ expect(result.content[0].text).toContain("belongs to a different video");
+ });
+});
+
+describe("batch_create_clips offline identity guard", () => {
+ it("rejects export_selected calls when the session transcript doesn't match the video", async () => {
+ writeMismatchedState();
+ const handler = getHandler("batch_create_clips");
+ const result = await handler({ export_selected: true }, {});
+ expect(result.content[0].text).toContain("belongs to a different video");
+ });
+});
diff --git a/src/handlers/batch-clips.handler.ts b/src/handlers/batch-clips.handler.ts
index 8b178f8..1c191e4 100644
--- a/src/handlers/batch-clips.handler.ts
+++ b/src/handlers/batch-clips.handler.ts
@@ -5,6 +5,9 @@ import { ClipsHistory } from "../services/clips-history.js";
import { paths } from "../config/paths.js";
import { webServerUrl } from "../config/server.js";
import { validateClipRange } from "../utils/clip-validation.js";
+import { findContentType } from "../utils/transcript.js";
+import { validateHook } from "../utils/clip-hook.js";
+import { transcriptVideoMismatch } from "../utils/video-identity.js";
import { childLogger } from "../utils/logger.js";
import type {
BatchClipsInput,
@@ -83,6 +86,12 @@ export const batchClipsToolDef = {
allow_ass_fallback: {
type: "boolean",
},
+ hook: {
+ type: ["object", "null"],
+ description:
+ "Opening hook: { start, end, mode: repeat|move }, a 1-15s passage from inside the clip played first. " +
+ "Null renders without one.",
+ },
},
required: ["start_second", "end_second"],
},
@@ -105,6 +114,12 @@ export const batchClipsToolDef = {
"Keep ProRes 4444 alpha caption overlays for DaVinci Resolve export. Default: false.",
default: false,
},
+ write_clean_variant: {
+ type: "boolean",
+ description:
+ "Also render a clean (no burned captions) variant per clip, with the same audio, loudness, and intro/outro. Returns clean_output_path per clip. Default: false.",
+ default: false,
+ },
transcript_words: {
type: "array",
description:
@@ -146,6 +161,16 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
return JSON.stringify({ error: "video_path is required (no video in session state)" });
}
+ // When the caller relies on the session transcript (rather than passing
+ // transcript_words explicitly), refuse to render against a video that was
+ // swapped in after that transcript was generated.
+ if (input.transcript_words == null && transcript) {
+ const mismatch = transcriptVideoMismatch(state?.transcriptVideoIdentity, videoPath);
+ if (mismatch) {
+ return JSON.stringify({ error: mismatch });
+ }
+ }
+
// Auto-resolve transcript words
const transcriptWords = input.transcript_words ?? transcript?.words ?? [];
@@ -165,8 +190,10 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
format: batchFormat,
allow_ass_fallback: input.allow_ass_fallback === true,
keep_caption_overlay: input.keep_caption_overlay === true,
+ write_clean_variant: input.write_clean_variant === true,
logo_path: settings.logoPath || null,
...(s.segments && s.segments.length > 0 && { keep_segments: s.segments }),
+ ...(s.hook && { hook: s.hook }),
});
if (input.export_selected) {
@@ -211,6 +238,9 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
if (input.keep_caption_overlay === true) {
clips = clips.map((c) => ({ ...c, keep_caption_overlay: c.keep_caption_overlay ?? true }));
}
+ if (input.write_clean_variant === true) {
+ clips = clips.map((c) => ({ ...c, write_clean_variant: c.write_clean_variant ?? true }));
+ }
for (let i = 0; i < clips.length; i++) {
const rangeError = validateClipRange(
@@ -221,6 +251,15 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
if (rangeError) {
return JSON.stringify({ error: `Clip ${i + 1}: ${rangeError}` });
}
+ const hookError = validateHook(
+ clips[i].hook,
+ clips[i].start_second,
+ clips[i].end_second,
+ clips[i].keep_segments,
+ );
+ if (hookError) {
+ return JSON.stringify({ error: `Clip ${i + 1}: ${hookError}` });
+ }
}
// Async path — route through Web UI so caller can poll job_status for
@@ -241,6 +280,7 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
outro_path: settings.outroPath || null,
intro_path: settings.introPath || null,
keep_caption_overlay: input.keep_caption_overlay === true,
+ write_clean_variant: input.write_clean_variant === true,
}),
});
if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
@@ -270,6 +310,7 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
clean_fillers: cleanFillers,
allow_ass_fallback: input.allow_ass_fallback === true,
keep_caption_overlay: input.keep_caption_overlay === true,
+ write_clean_variant: input.write_clean_variant === true,
output_dir: paths.output,
logo_path: settings.logoPath || null,
outro_path: settings.outroPath || null,
@@ -287,6 +328,8 @@ export async function handleBatchClips(input: BatchClipsInput): Promise<string>
defaultCaptionStyle: settings.captionStyle || "hormozi",
defaultCropStrategy: settings.cropStrategy || "speaker",
defaultFormat: batchFormat,
+ contentTypeFor: (s, e) => findContentType(suggestions, s, e),
+ suggestions,
});
await history.persistBatchRecipes(data.results, recorded, {
transcriptWords: transcriptWords,
diff --git a/src/handlers/compare-engines.handler.test.ts b/src/handlers/compare-engines.handler.test.ts
new file mode 100644
index 0000000..2e1a001
--- /dev/null
+++ b/src/handlers/compare-engines.handler.test.ts
@@ -0,0 +1,71 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+
+const executeMock = vi.fn();
+
+vi.mock("../services/python-executor.js", () => ({
+ PythonExecutor: class {
+ execute = executeMock;
+ },
+}));
+
+const { handleCompareEngines } = await import("./compare-engines.handler.js");
+
+describe("handleCompareEngines", () => {
+ beforeEach(() => {
+ executeMock.mockReset();
+ });
+
+ it("passes all params through to the compare_engines task", async () => {
+ executeMock.mockResolvedValue({
+ data: { overall_disagreement: 0.2, both_empty_window_count: 1, json_path: "/x/comparison.json", html_path: "/x/comparison.html" },
+ });
+
+ await handleCompareEngines({
+ file_path: "/video.mp4",
+ engine_a: "whispercpp",
+ engine_b: "whisper-py",
+ start_seconds: 10,
+ duration_seconds: 60,
+ window_seconds: 20,
+ model_size: "base",
+ language: "ka",
+ output_dir: "/tmp/cmp",
+ });
+
+ expect(executeMock).toHaveBeenCalledWith("compare_engines", {
+ file_path: "/video.mp4",
+ engine_a: "whispercpp",
+ engine_b: "whisper-py",
+ start_seconds: 10,
+ duration_seconds: 60,
+ window_seconds: 20,
+ model_size: "base",
+ language: "ka",
+ output_dir: "/tmp/cmp",
+ });
+ });
+
+ it("returns a summary labeled as disagreement, not accuracy", async () => {
+ executeMock.mockResolvedValue({
+ data: { overall_disagreement: 0.35, both_empty_window_count: 0, json_path: "/x/comparison.json", html_path: "/x/comparison.html" },
+ });
+
+ const text = await handleCompareEngines({
+ file_path: "/video.mp4",
+ engine_a: "whispercpp",
+ engine_b: "whisper-py",
+ });
+ const parsed = JSON.parse(text);
+
+ expect(parsed.overall_disagreement).toBe(0.35);
+ expect(parsed.note).toMatch(/disagreement/i);
+ expect(parsed.note).toContain("not accuracy against a transcript");
+ });
+
+ it("throws when the backend returns no data", async () => {
+ executeMock.mockResolvedValue({ data: undefined });
+ await expect(
+ handleCompareEngines({ file_path: "/video.mp4", engine_a: "whispercpp", engine_b: "whisper-py" }),
+ ).rejects.toThrow(/no data/i);
+ });
+});
diff --git a/src/handlers/compare-engines.handler.ts b/src/handlers/compare-engines.handler.ts
new file mode 100644
index 0000000..560c2a3
--- /dev/null
+++ b/src/handlers/compare-engines.handler.ts
@@ -0,0 +1,98 @@
+import { PythonExecutor } from "../services/python-executor.js";
+
+const executor = new PythonExecutor();
+
+export interface CompareEnginesInput {
+ file_path: string;
+ engine_a: string;
+ engine_b: string;
+ start_seconds?: number;
+ duration_seconds?: number;
+ window_seconds?: number;
+ model_size?: string;
+ language?: string;
+ output_dir?: string;
+}
+
+export const compareEnginesToolDef = {
+ name: "compare_transcription_engines",
+ description:
+ "Transcribe the same sample window of a file with two engines and report where their output " +
+ "disagrees, in 20s windows by default. This measures disagreement between the two engines' " +
+ "output, not accuracy against a ground-truth transcript. Neither engine is assumed correct. " +
+ "Writes comparison.json and a self-contained comparison.html (with a sample audio player and " +
+ "per-window seek buttons) to output_dir.",
+ inputSchema: {
+ type: "object" as const,
+ properties: {
+ file_path: { type: "string", description: "Absolute path to the podcast file" },
+ engine_a: {
+ type: "string",
+ enum: ["whisper-py", "whispercpp", "assemblyai", "omnilingual"],
+ description: "First engine to compare",
+ },
+ engine_b: {
+ type: "string",
+ enum: ["whisper-py", "whispercpp", "assemblyai", "omnilingual"],
+ description: "Second engine to compare",
+ },
+ start_seconds: {
+ type: "number",
+ description: "Sample start, seconds into the source. Default: 0.",
+ },
+ duration_seconds: {
+ type: "number",
+ description: "Sample length in seconds. Default: 120.",
+ },
+ window_seconds: {
+ type: "number",
+ description: "Report window size in seconds. Default: 20.",
+ },
+ model_size: {
+ type: "string",
+ enum: ["tiny", "base", "small", "medium", "large"],
+ description: "Model size for engines that take one. Default: base.",
+ },
+ language: {
+ type: "string",
+ description: "ISO language code. Leave empty for auto-detect.",
+ },
+ output_dir: {
+ type: "string",
+ description: "Where to write comparison.json/.html. Defaults under the podcli output directory.",
+ },
+ },
+ required: ["file_path", "engine_a", "engine_b"],
+ },
+};
+
+export async function handleCompareEngines(input: CompareEnginesInput): Promise<string> {
+ const result = await executor.execute<{
+ overall_disagreement: number;
+ both_empty_window_count: number;
+ json_path?: string;
+ html_path?: string;
+ }>("compare_engines", {
+ file_path: input.file_path,
+ engine_a: input.engine_a,
+ engine_b: input.engine_b,
+ start_seconds: input.start_seconds,
+ duration_seconds: input.duration_seconds,
+ window_seconds: input.window_seconds,
+ model_size: input.model_size,
+ language: input.language,
+ output_dir: input.output_dir,
+ });
+
+ if (!result.data) {
+ throw new Error("Engine comparison returned no data");
+ }
+
+ return JSON.stringify({
+ overall_disagreement: result.data.overall_disagreement,
+ both_empty_window_count: result.data.both_empty_window_count,
+ json_path: result.data.json_path,
+ html_path: result.data.html_path,
+ note: "disagreement between the two engines' output, not accuracy against a transcript",
+ });
+}
diff --git a/src/handlers/create-clip.handler.ts b/src/handlers/create-clip.handler.ts
index b68a2c1..c1d2420 100644
--- a/src/handlers/create-clip.handler.ts
+++ b/src/handlers/create-clip.handler.ts
@@ -6,6 +6,8 @@ import type { ClipResult, CreateClipInput, SuggestedClip, UIState } from "../mod
import { childLogger } from "../utils/logger.js";
import { sliceTranscript } from "../utils/transcript.js";
import { validateClipRange } from "../utils/clip-validation.js";
+import { validateHook } from "../utils/clip-hook.js";
+import { transcriptVideoMismatch } from "../utils/video-identity.js";
const log = childLogger("create-clip");
const executor = new PythonExecutor();
@@ -135,6 +137,18 @@ export const createClipToolDef = {
"Keep ProRes 4444 alpha caption overlay for DaVinci Resolve. Returns caption_overlay_path and cropped_source_path.",
default: false,
},
+ write_clean_variant: {
+ type: "boolean",
+ description:
+ "Also render a second file with the same audio, loudness, and intro/outro but no burned captions. Returns clean_output_path.",
+ default: false,
+ },
+ hook: {
+ type: ["object", "null"],
+ description:
+ "Opening hook: { start, end, mode: repeat|move }, a 1-15s passage from inside the clip played first. " +
+ "Auto-loaded from clip_number if omitted; null renders without one.",
+ },
},
required: [],
},
@@ -179,6 +193,19 @@ export async function handleCreateClip(input: CreateClipInput): Promise<string>
// Pull multi-cut segments from suggestion (if available)
const keepSegments = suggestion?.segments ?? null;
+ const hook = input.hook !== undefined ? input.hook : suggestion?.hook ?? null;
+
+ // When the caller relies on the session transcript (rather than passing
+ // transcript_words explicitly), refuse to render against a video that was
+ // swapped in after that transcript was generated. set_video clears the
+ // transcript itself, but older sessions or a stale on-disk state file can
+ // still carry a mismatched one.
+ if (input.transcript_words == null && transcript) {
+ const mismatch = transcriptVideoMismatch(state?.transcriptVideoIdentity, videoPath);
+ if (mismatch) {
+ return JSON.stringify({ error: mismatch });
+ }
+ }
// Validate required fields
if (!videoPath) {
@@ -193,6 +220,10 @@ export async function handleCreateClip(input: CreateClipInput): Promise<string>
if (rangeError) {
return JSON.stringify({ error: rangeError });
}
+ const hookError = validateHook(hook, startSecond, endSecond, keepSegments);
+ if (hookError) {
+ return JSON.stringify({ error: hookError });
+ }
const result = await executor.execute<ClipResult>("create_clip", {
video_path: videoPath,
@@ -207,12 +238,14 @@ export async function handleCreateClip(input: CreateClipInput): Promise<string>
clean_fillers: input.clean_fillers ?? settings.cleanFillers ?? true,
allow_ass_fallback: input.allow_ass_fallback === true,
keep_caption_overlay: input.keep_caption_overlay === true,
+ write_clean_variant: input.write_clean_variant === true,
logo_path: logoPath,
outro_path: outroPath,
intro_path: introPath,
...(input.name_card ? { name_card: input.name_card } : {}),
...(input.motion ? { motion: input.motion } : {}),
...(keepSegments && { keep_segments: keepSegments }),
+ ...(hook && { hook }),
});
if (!result.data) {
@@ -226,7 +259,14 @@ export async function handleCreateClip(input: CreateClipInput): Promise<string>
duration: data.duration,
file_size_mb: data.file_size_mb,
content_type: suggestion?.content_type,
+ payoff: suggestion?.payoff,
+ context_line: suggestion?.context_line,
+ preview_text: suggestion?.preview_text,
transcript_slice: sliceTranscript(transcriptWords, startSecond, endSecond),
+ ...(data.srt_path && { srt_path: data.srt_path }),
+ ...(data.vtt_path && { vtt_path: data.vtt_path }),
+ ...(data.clean_output_path && { clean_output_path: data.clean_output_path }),
+ ...(data.hook && { hook: data.hook }),
message: `Clip created successfully! ${data.duration}s, ${data.file_size_mb}MB`,
});
}
diff --git a/src/handlers/mine-channel.handler.test.ts b/src/handlers/mine-channel.handler.test.ts
new file mode 100644
index 0000000..e7a074d
--- /dev/null
+++ b/src/handlers/mine-channel.handler.test.ts
@@ -0,0 +1,163 @@
+import { describe, it, expect } from "vitest";
+import { listChannelUploads, mineVideoCaptions, type MineChannelDeps } from "./mine-channel.handler.js";
+
+// Fixture data standing in for yt-dlp's real output shape, so these tests
+// never touch the network or spawn a real subprocess.
+
+const FLAT_PLAYLIST_STDOUT = [
+ JSON.stringify({ id: "vid1", title: "Episode 1", duration: 3600, upload_date: "20240101", url: "https://youtu.be/vid1" }),
+ JSON.stringify({ id: "vid2", title: "Episode 2", duration: 4200, upload_date: "20240108" }),
+].join("\n");
+
+const VIDEO_INFO_WITH_ORIGINAL_TRACK = JSON.stringify({
+ id: "vid1",
+ title: "Episode 1",
+ duration: 3600,
+ subtitles: {},
+ automatic_captions: {
+ ka: [{ ext: "vtt", url: "https://captions.example/ka-translated.vtt" }],
+ "ka-orig": [{ ext: "vtt", url: "https://captions.example/ka-orig.vtt" }],
+ },
+});
+
+const VIDEO_INFO_NO_CAPTIONS = JSON.stringify({ id: "vid3", title: "No captions here", duration: 100 });
+
+const FIXTURE_VTT = [
+ "WEBVTT",
+ "",
+ "00:00:00.000 --> 00:00:02.000",
+ "Hello and welcome",
+ "",
+ "00:00:02.000 --> 00:00:04.000",
+ "to the show",
+ "",
+].join("\n");
+
+function depsReturning(opts: {
+ stdout: string;
+ code?: number;
+ stderr?: string;
+ minedIds?: string[];
+ captionText?: string;
+}): MineChannelDeps {
+ return {
+ runYtDlp: async () => ({ stdout: opts.stdout, stderr: opts.stderr ?? "", code: opts.code ?? 0 }),
+ fetchText: async () => opts.captionText ?? "",
+ minedVideoIds: async () => new Set(opts.minedIds ?? []),
+ };
+}
+
+describe("listChannelUploads", () => {
+ it("normalizes a bare channel root to the /videos tab before calling yt-dlp", async () => {
+ const seenArgs: string[][] = [];
+ const deps: MineChannelDeps = {
+ runYtDlp: async (args) => {
+ seenArgs.push(args);
+ return { stdout: FLAT_PLAYLIST_STDOUT, stderr: "", code: 0 };
+ },
+ fetchText: async () => "",
+ minedVideoIds: async () => new Set(),
+ };
+ await listChannelUploads(deps, { channel_url: "https://www.youtube.com/@deeptechdecodedai" });
+ expect(seenArgs[0].at(-1)).toBe("https://www.youtube.com/@deeptechdecodedai/videos");
+ });
+
+ it("drops tab/sub-playlist entries that carry no video id of their own", async () => {
+ const stdout = [
+ JSON.stringify({ title: "Shorts", _type: "playlist" }),
+ JSON.stringify({ id: "vid1", title: "Episode 1", duration: 3600 }),
+ ].join("\n");
+ const uploads = await listChannelUploads(depsReturning({ stdout }), {
+ channel_url: "https://www.youtube.com/@example/videos",
+ });
+ expect(uploads).toHaveLength(1);
+ expect(uploads[0].video_id).toBe("vid1");
+ });
+
+ it("parses one JSON object per line into channel uploads", async () => {
+ const uploads = await listChannelUploads(depsReturning({ stdout: FLAT_PLAYLIST_STDOUT }), {
+ channel_url: "https://www.youtube.com/@example/videos",
+ });
+ expect(uploads).toHaveLength(2);
+ expect(uploads[0]).toMatchObject({ video_id: "vid1", title: "Episode 1", duration: 3600 });
+ });
+
+ it("falls back to a watch url when yt-dlp doesn't print one", async () => {
+ const uploads = await listChannelUploads(depsReturning({ stdout: FLAT_PLAYLIST_STDOUT }), {
+ channel_url: "https://www.youtube.com/@example/videos",
+ });
+ expect(uploads[1].url).toBe("https://www.youtube.com/watch?v=vid2");
+ });
+
+ it("flags uploads that already have clips in history", async () => {
+ const uploads = await listChannelUploads(depsReturning({ stdout: FLAT_PLAYLIST_STDOUT, minedIds: ["vid1"] }), {
+ channel_url: "https://www.youtube.com/@example/videos",
+ });
+ expect(uploads.find((u) => u.video_id === "vid1")?.already_mined).toBe(true);
+ expect(uploads.find((u) => u.video_id === "vid2")?.already_mined).toBe(false);
+ });
+
+ it("throws with yt-dlp's stderr when the listing fails", async () => {
+ await expect(
+ listChannelUploads(depsReturning({ stdout: "", code: 1, stderr: "channel not found" }), {
+ channel_url: "https://www.youtube.com/@nope",
+ }),
+ ).rejects.toThrow("channel not found");
+ });
+});
+
+describe("mineVideoCaptions", () => {
+ it("prefers the original-language track over the auto-translated one", async () => {
+ const result = await mineVideoCaptions(
+ depsReturning({ stdout: VIDEO_INFO_WITH_ORIGINAL_TRACK, captionText: FIXTURE_VTT }),
+ { video_url: "https://youtu.be/vid1" },
+ );
+ expect(result.track).toEqual({ lang: "ka-orig", kind: "auto" });
+ expect(result.transcript?.language).toBe("ka");
+ });
+
+ it("converts fetched caption cues into a word-level transcript", async () => {
+ const result = await mineVideoCaptions(
+ depsReturning({ stdout: VIDEO_INFO_WITH_ORIGINAL_TRACK, captionText: FIXTURE_VTT }),
+ { video_url: "https://youtu.be/vid1" },
+ );
+ expect(result.transcript?.words.map((w) => w.word)).toEqual(["Hello", "and", "welcome", "to", "the", "show"]);
+ expect(result.transcript?.text).toBe("Hello and welcome to the show");
+ });
+
+ it("reports no transcript when the video has no captions at all", async () => {
+ const result = await mineVideoCaptions(depsReturning({ stdout: VIDEO_INFO_NO_CAPTIONS }), {
+ video_url: "https://youtu.be/vid3",
+ });
+ expect(result.transcript).toBeNull();
+ expect(result.track).toBeNull();
+ expect(result.message).toMatch(/no captions/i);
+ });
+
+ it("flags a video that's already been mined into clip history", async () => {
+ const result = await mineVideoCaptions(
+ depsReturning({ stdout: VIDEO_INFO_WITH_ORIGINAL_TRACK, captionText: FIXTURE_VTT, minedIds: ["vid1"] }),
+ { video_url: "https://youtu.be/vid1" },
+ );
+ expect(result.already_mined).toBe(true);
+ expect(result.message).toMatch(/already/i);
+ });
+
+ it("throws with yt-dlp's stderr when fetching video info fails", async () => {
+ await expect(
+ mineVideoCaptions(depsReturning({ stdout: "", code: 1, stderr: "video unavailable" }), {
+ video_url: "https://youtu.be/gone",
+ }),
+ ).rejects.toThrow("video unavailable");
+ });
+
+ it("throws when yt-dlp resolves the url to a different video than requested", async () => {
+ // VIDEO_INFO_WITH_ORIGINAL_TRACK is for vid1; asking with a url naming vid2
+ // (e.g. a playlist/mix redirect) must not be returned as if it were vid2's captions.
+ await expect(
+ mineVideoCaptions(depsReturning({ stdout: VIDEO_INFO_WITH_ORIGINAL_TRACK, captionText: FIXTURE_VTT }), {
+ video_url: "https://www.youtube.com/watch?v=vid2",
+ }),
+ ).rejects.toThrow(/resolved .* to video vid1, not the requested vid2/);
+ });
+});
diff --git a/src/handlers/mine-channel.handler.ts b/src/handlers/mine-channel.handler.ts
new file mode 100644
index 0000000..e448065
--- /dev/null
+++ b/src/handlers/mine-channel.handler.ts
@@ -0,0 +1,232 @@
+import { spawn } from "child_process";
+import { paths, pythonEnv } from "../config/paths.js";
+import { ClipsHistory } from "../services/clips-history.js";
+import {
+ buildChannelListArgs,
+ buildVideoInfoArgs,
+ extractYouTubeVideoId,
+ isCookieBrowser,
+ normalizeChannelUrl,
+} from "../utils/ytdlp-args.js";
+import { selectBestCaptionTrack, parseVtt, parseJson3, cuesToWords, type CaptionTrackRef } from "../utils/captions.js";
+import type { WordTimestamp } from "../models/index.js";
+
+const history = new ClipsHistory();
+
+export const mineChannelToolDef = {
+ name: "mine_channel",
+ description:
+ "Mine a YouTube channel's back catalog for clip-worthy moments without downloading any video. " +
+ "action='list' lists a channel's uploads (title, duration, upload date), flagging ones already mined " +
+ "into clip_history so you don't re-suggest from them. action='mine' fetches one video's existing " +
+ "captions (never the video itself), preferring the original-language track over an auto-translated one, " +
+ "and converts them to podcli's word-level transcript format. Feed the result straight into the " +
+ "suggest_clips flow the same way import_transcript's output is used. Never downloads a full video; " +
+ "that only happens if you separately choose to render a clip from one.",
+};
+
+export interface MineChannelInput {
+ action: "list" | "mine";
+ channel_url?: string;
+ video_url?: string;
+ limit?: number;
+ cookies_from_browser?: string;
+}
+
+interface RunResult {
+ stdout: string;
+ stderr: string;
+ code: number;
+}
+
+async function runYtDlp(args: string[]): Promise<RunResult> {
+ return new Promise((resolve, reject) => {
+ const proc = spawn(paths.pythonPath, args, { env: pythonEnv() });
+ let stdout = "";
+ let stderr = "";
+ proc.stdout.on("data", (d) => (stdout += d.toString()));
+ proc.stderr.on("data", (d) => (stderr += d.toString()));
+ proc.on("error", reject);
+ proc.on("close", (code) => resolve({ stdout, stderr, code: code ?? 1 }));
+ });
+}
+
+async function fetchText(url: string): Promise<string> {
+ const res = await fetch(url);
+ if (!res.ok) throw new Error(`HTTP ${res.status} fetching caption track`);
+ return res.text();
+}
+
+export interface ChannelUpload {
+ video_id: string;
+ title: string;
+ duration?: number;
+ upload_date?: string;
+ url: string;
+ already_mined: boolean;
+}
+
+export interface MinedVideoResult {
+ video_id: string;
+ title: string;
+ duration?: number;
+ url: string;
+ already_mined: boolean;
+ track: { lang: string; kind: "manual" | "auto" } | null;
+ transcript: {
+ words: WordTimestamp[];
+ segments: Array<{ text: string; start: number; end: number }>;
+ duration?: number;
+ language: string;
+ text: string;
+ } | null;
+ message: string;
+}
+
+// Injectable I/O so the orchestration (parsing, track selection, dedup) is
+// testable against fixture data with zero network or subprocess calls.
+export interface MineChannelDeps {
+ runYtDlp: (args: string[]) => Promise<RunResult>;
+ fetchText: (url: string) => Promise<string>;
+ minedVideoIds: () => Promise<Set<string>>;
+}
+
+export async function listChannelUploads(
+ deps: MineChannelDeps,
+ input: { channel_url: string; limit?: number; cookies_from_browser?: string },
+): Promise<ChannelUpload[]> {
+ const args = buildChannelListArgs({
+ channelUrl: normalizeChannelUrl(input.channel_url),
+ limit: input.limit,
+ cookiesFromBrowser: isCookieBrowser(input.cookies_from_browser) ? input.cookies_from_browser : undefined,
+ });
+ const { stdout, stderr, code } = await deps.runYtDlp(args);
+ if (code !== 0) {
+ throw new Error(stderr.trim() || "yt-dlp failed to list the channel's uploads");
+ }
+ const minedIds = await deps.minedVideoIds();
+ return stdout
+ .trim()
+ .split("\n")
+ .filter(Boolean)
+ .map((line) => JSON.parse(line) as { id?: string; title: string; duration?: number; upload_date?: string; url?: string })
+ // A tab or sub-playlist entry (e.g. "Shorts", "Live") has no video id of
+ // its own: only real uploads do.
+ .filter((u): u is { id: string; title: string; duration?: number; upload_date?: string; url?: string } => Boolean(u.id))
+ .map((u) => ({
+ video_id: u.id,
+ title: u.title,
+ duration: u.duration,
+ upload_date: u.upload_date,
+ url: u.url ?? `https://www.youtube.com/watch?v=${u.id}`,
+ already_mined: minedIds.has(u.id),
+ }));
+}
+
+export async function mineVideoCaptions(
+ deps: MineChannelDeps,
+ input: { video_url: string; cookies_from_browser?: string },
+): Promise<MinedVideoResult> {
+ const args = buildVideoInfoArgs({
+ videoUrl: input.video_url,
+ cookiesFromBrowser: isCookieBrowser(input.cookies_from_browser) ? input.cookies_from_browser : undefined,
+ });
+ const { stdout, stderr, code } = await deps.runYtDlp(args);
+ if (code !== 0) {
+ throw new Error(stderr.trim() || "yt-dlp failed to fetch video info");
+ }
+ // --dump-json prints exactly one JSON object; take the last non-empty line
+ // defensively in case a warning slipped through on an earlier line.
+ const jsonLine = stdout.trim().split("\n").filter(Boolean).at(-1);
+ if (!jsonLine) throw new Error("yt-dlp returned no video info");
+ const info = JSON.parse(jsonLine) as {
+ id: string;
+ title: string;
+ duration?: number;
+ subtitles?: Record<string, CaptionTrackRef[]>;
+ automatic_captions?: Record<string, CaptionTrackRef[]>;
+ };
+
+ // --no-playlist stops a playlist/mix url from resolving to its first entry,
+ // but a url that already names one video (e.g. a stale redirect) can still
+ // resolve to a different id than the one asked for. Catch that here rather
+ // than silently returning the wrong video's captions.
+ const requestedId = extractYouTubeVideoId(input.video_url);
+ if (requestedId && requestedId !== info.id) {
+ throw new Error(
+ `yt-dlp resolved ${input.video_url} to video ${info.id}, not the requested ${requestedId}`,
+ );
+ }
+
+ const minedIds = await deps.minedVideoIds();
+ const alreadyMined = minedIds.has(info.id);
+
+ const track = selectBestCaptionTrack(info.subtitles, info.automatic_captions);
+ if (!track) {
+ return {
+ video_id: info.id,
+ title: info.title,
+ duration: info.duration,
+ url: input.video_url,
+ already_mined: alreadyMined,
+ track: null,
+ transcript: null,
+ message: "No captions available for this video, nothing to mine.",
+ };
+ }
+
+ const captionText = await deps.fetchText(track.url);
+ const { words, segments } =
+ track.ext === "json3"
+ ? parseJson3(captionText)
+ : (() => {
+ const cues = parseVtt(captionText);
+ return { words: cuesToWords(cues), segments: cues.map((c) => ({ text: c.text, start: c.start, end: c.end })) };
+ })();
+ const language = track.lang.replace(/-orig$/, "");
+
+ return {
+ video_id: info.id,
+ title: info.title,
+ duration: info.duration,
+ url: input.video_url,
+ already_mined: alreadyMined,
+ track: { lang: track.lang, kind: track.kind },
+ transcript: {
+ words,
+ segments,
+ duration: info.duration,
+ language,
+ text: words.map((w) => w.word).join(" "),
+ },
+ message: alreadyMined
+ ? "This video already has clips in history. Review before re-mining to avoid duplicates."
+ : `Mined ${words.length} words from the ${track.kind} ${track.lang} track. ` +
+ "Use this transcript the same way import_transcript's output is used, then suggest_clips.",
+ };
+}
+
+const defaultDeps: MineChannelDeps = {
+ runYtDlp,
+ fetchText,
+ minedVideoIds: () => history.minedYouTubeVideoIds(),
+};
+
+export async function handleMineChannel(input: MineChannelInput): Promise<string> {
+ if (input.action === "list") {
+ if (!input.channel_url) throw new Error("channel_url is required for action=list");
+ const uploads = await listChannelUploads(defaultDeps, {
+ channel_url: input.channel_url,
+ limit: input.limit,
+ cookies_from_browser: input.cookies_from_browser,
+ });
+ return JSON.stringify({ uploads, count: uploads.length });
+ }
+
+ if (!input.video_url) throw new Error("video_url is required for action=mine");
+ const result = await mineVideoCaptions(defaultDeps, {
+ video_url: input.video_url,
+ cookies_from_browser: input.cookies_from_browser,
+ });
+ return JSON.stringify(result);
+}
diff --git a/src/handlers/suggest-clips.handler.test.ts b/src/handlers/suggest-clips.handler.test.ts
new file mode 100644
index 0000000..36e2a2e
--- /dev/null
+++ b/src/handlers/suggest-clips.handler.test.ts
@@ -0,0 +1,50 @@
+import { describe, it, expect } from "vitest";
+import { z } from "zod";
+import { handleSuggestClips, hookSchema, suggestClipsInputShape } from "./suggest-clips.handler.js";
+
+const base = {
+ title: "Why the seed round cost them pricing",
+ start_second: 100,
+ end_second: 130,
+ payoff: "You learn why raising early cost the founders control of pricing.",
+ standalone: "nothing",
+ reasoning: "Concrete stakes and a clear turn.",
+ preview_text: "We raised a seed round before we had a single paying customer.",
+};
+
+describe("suggest_clips hook", () => {
+ it("accepts a hook in the tool schema and rejects an unknown mode", () => {
+ const schema = z.object(suggestClipsInputShape);
+ expect(
+ schema.safeParse({ suggestions: [{ ...base, hook: { start: 110, end: 113, mode: "move" } }] }).success,
+ ).toBe(true);
+ expect(hookSchema.safeParse({ start: 110, end: 113, mode: "loop" }).success).toBe(false);
+ expect(hookSchema.nullable().safeParse(null).success).toBe(true);
+ });
+
+ it("stores a valid hook and counts it in the duration", async () => {
+ const out = JSON.parse(
+ await handleSuggestClips({
+ suggestions: [{ ...base, hook: { start: 110, end: 113, mode: "repeat" } }],
+ }),
+ );
+ expect(out.clips[0].hook).toEqual({ start: 110, end: 113, mode: "repeat" });
+ expect(out.clips[0].duration).toBe(33);
+ });
+
+ it("rejects a hook outside the clip", async () => {
+ await expect(
+ handleSuggestClips({
+ suggestions: [{ ...base, hook: { start: 90, end: 95, mode: "repeat" } }],
+ }),
+ ).rejects.toThrow(/not inside the clip body/);
+ });
+
+ it("rejects a hook longer than 15 seconds", async () => {
+ await expect(
+ handleSuggestClips({
+ suggestions: [{ ...base, hook: { start: 105, end: 125, mode: "move" } }],
+ }),
+ ).rejects.toThrow(/between 1 and 15/);
+ });
+});
diff --git a/src/handlers/suggest-clips.handler.ts b/src/handlers/suggest-clips.handler.ts
index f028160..8bb08e0 100644
--- a/src/handlers/suggest-clips.handler.ts
+++ b/src/handlers/suggest-clips.handler.ts
@@ -11,6 +11,23 @@ import {
validateSuggestionContext,
validateSuggestionRange,
} from "../utils/clip-validation.js";
+import { playbackDuration, validateHook } from "../utils/clip-hook.js";
+
+/** Shared by suggest_clips, modify_clip, create_clip and batch_create_clips. */
+export const hookSchema = z
+ .object({
+ start: z.number().describe("Hook start in seconds, on the source clock"),
+ end: z.number().describe("Hook end in seconds, on the source clock"),
+ mode: z
+ .enum(["repeat", "move"])
+ .describe(
+ "repeat: the passage plays first and again in place. move: it plays first and is cut from its place.",
+ ),
+ })
+ .describe(
+ "Opening hook: a 1-15s passage from inside the clip, played before the clip. " +
+ "It must be a line actually spoken inside the clip's range (or its segments), never invented text.",
+ );
const suggestionSchema = z.object({
title: z.string().describe("Short catchy title for the clip"),
@@ -28,6 +45,7 @@ const suggestionSchema = z.object({
"Multi-cut keep-ranges within the clip. Use to cut out filler/tangents " +
"in the middle. Omit for a single continuous clip.",
),
+ hook: hookSchema.optional(),
payoff: z
.string()
.describe(
@@ -114,6 +132,7 @@ export async function handleSuggestClips(input: SuggestClipsInput): Promise<stri
const errors = [
validateSuggestionRange(s.start_second, s.end_second),
validateSuggestionContext(s),
+ validateHook(s.hook, s.start_second, s.end_second, s.segments),
].filter((e): e is string => e !== null);
for (const error of errors) {
problems.push(`Suggestion ${i + 1} ("${s.title}"): ${error}`);
@@ -128,11 +147,9 @@ export async function handleSuggestClips(input: SuggestClipsInput): Promise<stri
// Validate and enrich suggestions
const enriched = suggestions.map((s, i) => {
- // Compute duration from segments if available, otherwise from start/end
+ // Kept segments (or the whole range), plus whatever the hook adds.
const segments = s.segments?.filter((seg) => seg.end > seg.start) || [];
- const keptDuration = segments.length > 0
- ? segments.reduce((sum, seg) => sum + (seg.end - seg.start), 0)
- : s.end_second - s.start_second;
+ const keptDuration = playbackDuration(s.start_second, s.end_second, segments, s.hook);
return {
clip_number: i + 1,
@@ -142,6 +159,7 @@ export async function handleSuggestClips(input: SuggestClipsInput): Promise<stri
end_second: s.end_second,
segments: segments.length > 0 ? segments : [{ start: s.start_second, end: s.end_second }],
duration: Math.round(keptDuration * 10) / 10,
+ ...(s.hook && { hook: s.hook }),
payoff: s.payoff,
standalone: s.standalone,
context_line: s.context_line || "",
diff --git a/src/handlers/transcribe.handler.test.ts b/src/handlers/transcribe.handler.test.ts
new file mode 100644
index 0000000..4633025
--- /dev/null
+++ b/src/handlers/transcribe.handler.test.ts
@@ -0,0 +1,124 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+
+const executeMock = vi.fn();
+const cacheGetMock = vi.fn();
+const cacheSetMock = vi.fn();
+const cacheGetPackedMock = vi.fn();
+const cacheGetHashForEngineMock = vi.fn();
+
+vi.mock("../services/python-executor.js", () => ({
+ PythonExecutor: class {
+ execute = executeMock;
+ },
+}));
+
+vi.mock("../services/transcript-cache.js", () => ({
+ TranscriptCache: class {
+ get = cacheGetMock;
+ set = cacheSetMock;
+ getPackedMarkdown = cacheGetPackedMock;
+ getFileHashForEngine = cacheGetHashForEngineMock;
+ },
+ hasSpeakerLabels: () => true,
+ needsDiarizationRetry: () => false,
+}));
+
+vi.mock("../services/engine-resolve.js", () => ({
+ resolveTranscribeEngine: vi.fn().mockResolvedValue("whispercpp"),
+ engineCanDiarize: () => false,
+}));
+
+const { handleTranscribe } = await import("./transcribe.handler.js");
+
+describe("handleTranscribe, sample mode", () => {
+ beforeEach(() => {
+ executeMock.mockReset();
+ cacheGetMock.mockReset();
+ cacheSetMock.mockReset();
+ cacheGetPackedMock.mockReset();
+ });
+
+ it("never reads the main cache when start_seconds/duration_seconds are set", async () => {
+ executeMock.mockResolvedValue({
+ data: { transcript: "hi", segments: [], words: [], duration: 40, language: "ka", engine: "whispercpp" },
+ });
+
+ const result = await handleTranscribe({
+ file_path: "/video.mp4",
+ start_seconds: 120,
+ duration_seconds: 40,
+ });
+
+ expect(cacheGetMock).not.toHaveBeenCalled();
+ const parsed = JSON.parse(result);
+ expect(parsed.cached).toBe(false);
+ expect(parsed.packed_ready).toBe(false);
+ });
+
+ it("never writes the main cache for a sample result", async () => {
+ executeMock.mockResolvedValue({
+ data: {
+ transcript: "hi",
+ segments: [],
+ words: [],
+ duration: 40,
+ language: "ka",
+ engine: "whispercpp",
+ complete: false,
+ sample_offset_seconds: 120,
+ },
+ });
+
+ const result = await handleTranscribe({
+ file_path: "/video.mp4",
+ start_seconds: 120,
+ duration_seconds: 40,
+ });
+
+ expect(cacheSetMock).not.toHaveBeenCalled();
+ const parsed = JSON.parse(result);
+ expect(parsed.complete).toBe(false);
+ expect(parsed.sample_offset_seconds).toBe(120);
+ });
+
+ it("passes start_seconds/duration_seconds through to the transcribe task", async () => {
+ executeMock.mockResolvedValue({
+ data: { transcript: "hi", segments: [], words: [], duration: 40, language: "ka", engine: "whispercpp" },
+ });
+
+ await handleTranscribe({ file_path: "/video.mp4", start_seconds: 10, duration_seconds: 30 });
+
+ expect(executeMock).toHaveBeenCalledWith(
+ "transcribe",
+ expect.objectContaining({ start_seconds: 10, duration_seconds: 30 }),
+ );
+ });
+
+ it("treats duration_seconds: 0 as a normal (non-sample) request", async () => {
+ cacheGetMock.mockResolvedValue(null);
+ executeMock.mockResolvedValue({
+ data: { transcript: "hi", segments: [], words: [], duration: 400, language: "en", engine: "whispercpp" },
+ });
+ cacheGetPackedMock.mockResolvedValue("# packed");
+
+ const result = await handleTranscribe({ file_path: "/video.mp4", duration_seconds: 0 });
+
+ expect(cacheGetMock).toHaveBeenCalled();
+ expect(cacheSetMock).toHaveBeenCalled();
+ const parsed = JSON.parse(result);
+ expect(parsed.complete).toBeUndefined();
+ });
+
+ it("still uses the main cache for a normal (non-sample) request", async () => {
+ cacheGetMock.mockResolvedValue(null);
+ executeMock.mockResolvedValue({
+ data: { transcript: "hi", segments: [], words: [], duration: 400, language: "en", engine: "whispercpp" },
+ });
+ cacheGetPackedMock.mockResolvedValue("# packed");
+
+ await handleTranscribe({ file_path: "/video.mp4" });
+
+ expect(cacheGetMock).toHaveBeenCalled();
+ expect(cacheSetMock).toHaveBeenCalled();
+ });
+});
diff --git a/src/handlers/transcribe.handler.ts b/src/handlers/transcribe.handler.ts
index 4ec89d2..f2ce804 100644
--- a/src/handlers/transcribe.handler.ts
+++ b/src/handlers/transcribe.handler.ts
@@ -1,6 +1,7 @@
import { basename } from "path";
import { PythonExecutor } from "../services/python-executor.js";
-import { TranscriptCache, hasSpeakerLabels } from "../services/transcript-cache.js";
+import { TranscriptCache, needsDiarizationRetry } from "../services/transcript-cache.js";
+import { resolveTranscribeEngine, engineCanDiarize } from "../services/engine-resolve.js";
import { webServerUrl } from "../config/server.js";
import type { TranscriptResult } from "../models/index.js";
@@ -10,10 +11,12 @@ const cache = new TranscriptCache();
export interface TranscribeInput {
file_path: string;
model_size?: "tiny" | "base" | "small" | "medium" | "large";
- engine?: "whisper-py" | "whispercpp" | "assemblyai";
+ engine?: "whisper-py" | "whispercpp" | "assemblyai" | "omnilingual";
language?: string;
enable_diarization?: boolean;
num_speakers?: number;
+ start_seconds?: number;
+ duration_seconds?: number;
}
export const transcribeToolDef = {
@@ -51,7 +54,7 @@ export const transcribeToolDef = {
},
engine: {
type: "string",
- enum: ["whisper-py", "whispercpp", "assemblyai"],
+ enum: ["whisper-py", "whispercpp", "assemblyai", "omnilingual"],
description: "Transcription engine. Default: whisper-py.",
},
enable_diarization: {
@@ -68,6 +71,18 @@ export const transcribeToolDef = {
"Exact number of speakers if known (e.g. 2 for a two-person podcast). " +
"Leave empty to auto-detect (2-5 speakers).",
},
+ start_seconds: {
+ type: "number",
+ description:
+ "Sample mode: only transcribe a window starting here (seconds into the source), " +
+ "instead of the whole file, e.g. to test a language on 40s before committing to " +
+ "a full run. Pair with duration_seconds. The result is marked complete: false and " +
+ "is not written to the main transcript cache.",
+ },
+ duration_seconds: {
+ type: "number",
+ description: "Sample mode window length in seconds. Defaults start_seconds to 0 if omitted.",
+ },
},
required: ["file_path"],
},
@@ -80,26 +95,43 @@ export async function handleTranscribe(input: TranscribeInput): Promise<string>
const language = input.language;
const enableDiarization = input.enable_diarization !== false;
const numSpeakers = input.num_speakers;
+ const startSeconds = input.start_seconds;
+ const durationSeconds = input.duration_seconds;
+ // A sample is a positive window, not merely a present key. An explicit
+ // null or a duration_seconds: 0 both mean "no sample", matching
+ // backend/services/transcription.py and backend/main.py's handle_transcribe.
+ const isSample = (durationSeconds ?? 0) > 0 || (startSeconds ?? 0) > 0;
+
+ // Resolve before reading the cache: an unset engine is written under
+ // whatever transcribe_file actually ran (e.g. "whispercpp" on a native
+ // install), so reading with the raw unset request always misses.
+ const resolvedEngine = await resolveTranscribeEngine(executor, engine, modelSize);
+ const cacheKey = { engine: resolvedEngine, model: modelSize, language };
- // Check cache first. A cached transcript without speakers cannot answer a
- // request for them, so serving it makes re-transcribing look like a no-op.
- const cachedRaw = await cache.get(filePath, engine);
- const cached =
- cachedRaw && enableDiarization && !hasSpeakerLabels(cachedRaw) ? null : cachedRaw;
+ // A sample is a throwaway check on a slice of the file. It must never
+ // serve (or pollute) the main transcript cache, which is keyed by the
+ // whole file and assumed complete.
+ const cachedRaw = isSample ? null : await cache.get(filePath, cacheKey);
+ const cached = needsDiarizationRetry(cachedRaw, enableDiarization, engineCanDiarize(resolvedEngine))
+ ? null
+ : cachedRaw;
if (cached) {
- const packedEngine = cached.engine ?? engine;
+ // Same combo cache.get(filePath, cacheKey) just matched: the packed
+ // view has to be keyed identically, or a backfill here would write it
+ // under a key the next read for this same request won't find.
+ const packedKey = { engine: cached.engine ?? resolvedEngine, model: modelSize, language };
// Backfill packed view if this cache predates auto-packing.
- let packed = await cache.getPackedMarkdown(filePath, packedEngine);
+ let packed = await cache.getPackedMarkdown(filePath, packedKey);
if (!packed) {
try {
- const cacheHash = await cache.getFileHashForEngine(filePath, packedEngine);
+ const cacheHash = await cache.getFileHashForEngine(filePath, packedKey);
await executor.execute("pack_transcript", {
transcript: cached,
cache_hash: cacheHash,
source_label: basename(filePath),
file_path: filePath,
});
- packed = await cache.getPackedMarkdown(filePath, packedEngine);
+ packed = await cache.getPackedMarkdown(filePath, packedKey);
} catch {
// Non-fatal — caller still gets metadata
}
@@ -115,17 +147,27 @@ export async function handleTranscribe(input: TranscribeInput): Promise<string>
language,
enable_diarization: enableDiarization,
num_speakers: numSpeakers,
+ start_seconds: startSeconds,
+ duration_seconds: durationSeconds,
});
if (!result.data) {
throw new Error("Transcription returned no data");
}
const data = result.data;
- const resolvedEngine = data.engine;
+ const actualEngine = data.engine ?? resolvedEngine;
- // Cache the raw result
- await cache.set(filePath, data, resolvedEngine);
- const packed = await cache.getPackedMarkdown(filePath, resolvedEngine);
+ if (isSample) {
+ // Not cached and not packed: it's a slice of the file, not the whole
+ // transcript the cache/packed-view keys assume.
+ return JSON.stringify({ cached: false, packed_ready: false, ...formatResult(data) });
+ }
+
+ // Cache the raw result under what it actually ran with, not the prediction
+ // above: resolveTranscribeEngine can't see a model-load failure that only
+ // shows up once transcribe_file tries it for real.
+ await cache.set(filePath, data, { engine: actualEngine, model: modelSize, language });
+ const packed = await cache.getPackedMarkdown(filePath, { engine: actualEngine, model: modelSize, language });
return JSON.stringify({ cached: false, packed_ready: !!packed, ...formatResult(data) });
}
@@ -140,9 +182,15 @@ function formatResult(data: TranscriptResult) {
return {
duration: data.duration,
language: data.language,
+ // Exposed so callers that need to re-read the cache (e.g. the UI state
+ // push in server.ts) key it the same way this handler just wrote it.
+ engine: data.engine,
word_count: (data.words ?? []).length,
segment_count: (data.segments ?? []).length,
speakers: data.speakers ?? { num_speakers: 0, speakers: {} },
+ ...(data.complete === false
+ ? { complete: false, sample_offset_seconds: data.sample_offset_seconds ?? 0 }
+ : {}),
next_step: "Read the transcript via get_ui_state(include_transcript: true), then suggest_clips.",
};
}
@@ -173,10 +221,20 @@ export const transcribeStartToolDef = {
language: { type: "string" },
engine: {
type: "string",
- enum: ["whisper-py", "whispercpp", "assemblyai"],
+ enum: ["whisper-py", "whispercpp", "assemblyai", "omnilingual"],
},
enable_diarization: { type: "boolean", default: true },
num_speakers: { type: "number" },
+ start_seconds: {
+ type: "number",
+ description:
+ "Sample mode: only transcribe a window starting here (seconds into the source). " +
+ "Pair with duration_seconds. Not written to the main transcript cache.",
+ },
+ duration_seconds: {
+ type: "number",
+ description: "Sample mode window length in seconds. Defaults start_seconds to 0 if omitted.",
+ },
},
required: ["file_path"],
},
@@ -194,6 +252,8 @@ export async function handleTranscribeStart(input: TranscribeInput): Promise<str
language: input.language,
enable_diarization: input.enable_diarization !== false,
num_speakers: input.num_speakers,
+ start_seconds: input.start_seconds,
+ duration_seconds: input.duration_seconds,
}),
});
if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
diff --git a/src/models/index.ts b/src/models/index.ts
index e57668a..8e82688 100644
--- a/src/models/index.ts
+++ b/src/models/index.ts
@@ -1,8 +1,14 @@
+import type { ClipHook } from "../utils/clip-hook.js";
+import type { ThumbnailLayout, ThumbnailPerson } from "../utils/thumbnail-layout.js";
+
+export type { ClipHook, HookMode } from "../utils/clip-hook.js";
+export type { ThumbnailLayout, ThumbnailPerson } from "../utils/thumbnail-layout.js";
+
// === Task Communication Models ===
export interface TaskRequest {
task_id: string;
- task_type: "transcribe" | "parse_transcript" | "create_clip" | "batch_clips" | "analyze_energy" | "detect_highlights" | "manage_reel" | "pack_transcript" | "detect_encoder" | "presets" | "ping" | "suggest_clips" | "find_moment" | "generate_content" | "generate_custom" | "corrections" | "manage_integrations" | "run_integration_tool" | "manage_config" | "manage_env" | "ai_cli_status" | "ai_provider_status" | "analyze_silence" | "render_silence_removed" | "manage_multicam";
+ task_type: "transcribe" | "resolve_transcribe_engine" | "compare_engines" | "parse_transcript" | "create_clip" | "batch_clips" | "analyze_energy" | "detect_highlights" | "manage_reel" | "pack_transcript" | "detect_encoder" | "presets" | "ping" | "suggest_clips" | "find_moment" | "generate_content" | "generate_custom" | "corrections" | "manage_integrations" | "run_integration_tool" | "manage_config" | "manage_env" | "ai_cli_status" | "ai_provider_status" | "analyze_silence" | "render_silence_removed" | "manage_multicam";
params: Record<string, unknown>;
}
@@ -66,6 +72,16 @@ export interface TranscriptResult {
speakers: SpeakerSummary;
speaker_segments: SpeakerSegment[];
engine?: string;
+ // Present only for a sample-mode transcription (start_seconds/duration_seconds):
+ // complete is always false, and timestamps are relative to the sample window,
+ // offset from the source by sample_offset_seconds.
+ complete?: boolean;
+ sample_offset_seconds?: number;
+ // Whether diarization was actually tried for this result (set by every
+ // branch of backend/services/transcription.py). False for engines that
+ // never diarize (whispercpp, omnilingual) regardless of what was
+ // requested, see needsDiarizationRetry in services/transcript-cache.ts.
+ diarization_attempted?: boolean;
}
// === Clip Models ===
@@ -91,6 +107,13 @@ export interface ClipResult {
format?: Format;
caption_overlay_path?: string;
cropped_source_path?: string;
+ /** Sidecar subtitles, retimed to the exported file's own playback clock. */
+ srt_path?: string;
+ vtt_path?: string;
+ /** Same audio/loudness/intro/outro, without burned captions. */
+ clean_output_path?: string;
+ /** The hook as rendered, edges widened to whole words. */
+ hook?: ClipHook;
}
export interface SuggestedClip {
@@ -105,11 +128,17 @@ export interface SuggestedClip {
reasoning: string;
preview_text: string;
segments?: Array<{ start: number; end: number }>;
+ /** Spoken passage from inside the clip, played before it. */
+ hook?: ClipHook;
suggested_caption_style?: string;
timestamp_display?: string;
content_type?: string;
score?: number;
rank?: number;
+ /** Fingerprint of render-relevant fields, stamped when the clip is selected for export. */
+ selectionHash?: string;
+ /** True when the clip was edited after selectionHash was stamped, the selection may be stale. */
+ changedSinceSelection?: boolean;
}
export interface UIState {
@@ -117,9 +146,14 @@ export interface UIState {
filePath?: string;
activeExportJobId?: string | null;
transcript?: TranscriptResult | null;
+ /** Identity (path + size + mtime) of the video the transcript was generated from. */
+ transcriptVideoIdentity?: { path: string; size: number; mtimeMs: number } | null;
rawTranscriptText?: string;
silenceOriginal?: { videoPath: string; transcript: TranscriptResult } | null;
silencePlan?: Record<string, unknown> | null;
+ /** True when videoPath didn't exist at last check (e.g. an external drive
+ * is unmounted). The session is kept, not wiped, while this is set. */
+ videoMissing?: boolean;
suggestions?: SuggestedClip[];
deselectedIndices?: number[];
settings?: {
@@ -142,6 +176,32 @@ export interface UIState {
lastUpdated?: number;
}
+/**
+ * Decisions the user has already answered for one episode, so /auto and
+ * /produce-shorts ask each question once and reuse the answer on every later
+ * run against the same video, including after a Web UI restart, since this
+ * is keyed by video identity and stored independently of ui-state.json.
+ */
+export interface EpisodeDecisions {
+ videoPath: string;
+ fileSize: number;
+ clipCount?: number;
+ clipDurationRange?: { min?: number; max?: number };
+ captionStyle?: string;
+ captionsEnabled?: boolean;
+ language?: string;
+ thumbnailsWanted?: boolean;
+ deliveryTarget?: string;
+ notes?: string;
+ updatedAt: number;
+}
+
+/** One decision from EpisodeDecisions that has no answer yet. */
+export interface OpenQuestion {
+ field: keyof EpisodeDecisions;
+ question: string;
+}
+
/** Who is speaking, shown as a lower third for the first seconds of a clip. */
export interface NameCard {
title: string;
@@ -186,6 +246,9 @@ export interface CreateClipInput {
clean_fillers?: boolean;
allow_ass_fallback?: boolean;
keep_caption_overlay?: boolean;
+ write_clean_variant?: boolean;
+ /** Overrides the suggestion's hook; null renders without one. */
+ hook?: ClipHook | null;
}
export interface BatchClipSpec {
@@ -199,7 +262,9 @@ export interface BatchClipSpec {
intro_path?: string | null;
allow_ass_fallback?: boolean;
keep_caption_overlay?: boolean;
+ write_clean_variant?: boolean;
keep_segments?: Array<{ start: number; end: number }>;
+ hook?: ClipHook | null;
}
export interface BatchClipsInput {
@@ -212,6 +277,7 @@ export interface BatchClipsInput {
clean_fillers?: boolean;
allow_ass_fallback?: boolean;
keep_caption_overlay?: boolean;
+ write_clean_variant?: boolean;
/**
* When true, POST to the Web UI's /api/batch-clips and return a job_id
* immediately so the caller can poll job_status and emit live progress.
@@ -238,6 +304,9 @@ export interface BatchClipsResult {
title?: string;
file_size_mb?: number;
duration?: number;
+ srt_path?: string;
+ vtt_path?: string;
+ clean_output_path?: string;
error?: string;
}>;
}
@@ -288,6 +357,8 @@ export interface ClipThumbnailConfig {
preview_path?: string; // chosen thumbnail PNG
variations?: string[]; // all generated thumbnail PNGs to pick from
card_seconds?: number; // duration of the thumbnail card baked into the clip start
+ layout?: ThumbnailLayout; // the layout the chosen thumbnail was drawn with
+ people?: ThumbnailPerson[]; // pair layout only: who sits on each side
}
export interface ClipHistoryEntry {
@@ -308,6 +379,12 @@ export interface ClipHistoryEntry {
created_at: string;
content_type?: string;
transcript_slice?: string;
+ // Carried from the suggestion at render time: the payoff, the question it
+ // answers, and its verbatim opening line. Thumbnail generation grounds
+ // headline copy in these instead of the title alone.
+ payoff?: string;
+ context_line?: string;
+ preview_text?: string;
logo_backup_path?: string;
logo_position?: string;
keep_segments?: Array<{ start: number; end: number }>;
diff --git a/src/podstack-commands.test.ts b/src/podstack-commands.test.ts
new file mode 100644
index 0000000..b93da5f
--- /dev/null
+++ b/src/podstack-commands.test.ts
@@ -0,0 +1,64 @@
+import { describe, it, expect } from "vitest";
+import { readFileSync, readdirSync } from "fs";
+import { dirname, join, resolve } from "path";
+import { fileURLToPath } from "url";
+import { createServer } from "./server.js";
+
+// Catches the class of bug fixed alongside this test: a PodStack command
+// frontmatter or body references an MCP tool name that doesn't match what
+// the server actually registers (e.g. a renamed tool, or a typo).
+const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
+const commandsDir = join(repoRoot, ".claude", "commands");
+
+function registeredToolNames(): Set<string> {
+ const server = createServer() as unknown as { _registeredTools: Record<string, unknown> };
+ return new Set(Object.keys(server._registeredTools));
+}
+
+function parseFrontmatter(source: string): { allowedTools: string[]; body: string } {
+ const match = source.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/);
+ if (!match) return { allowedTools: [], body: source };
+ const [, frontmatter, body] = match;
+ const line = frontmatter.split("\n").find((l) => l.startsWith("allowed-tools:"));
+ const allowedTools = line
+ ? line
+ .slice("allowed-tools:".length)
+ .split(",")
+ .map((t) => t.trim())
+ .filter(Boolean)
+ : [];
+ return { allowedTools, body };
+}
+
+const commandFiles = readdirSync(commandsDir).filter((f) => f.endsWith(".md"));
+
+describe("PodStack command frontmatter matches registered MCP tools", () => {
+ const toolNames = registeredToolNames();
+
+ it("the server registers at least one tool", () => {
+ expect(toolNames.size).toBeGreaterThan(0);
+ });
+
+ for (const file of commandFiles) {
+ it(`${file}: every mcp__podcli__* in allowed-tools is a registered tool`, () => {
+ const source = readFileSync(join(commandsDir, file), "utf-8");
+ const { allowedTools } = parseFrontmatter(source);
+ const mcpTools = allowedTools.filter((t) => t.startsWith("mcp__podcli__"));
+ const unknown = mcpTools
+ .map((t) => t.slice("mcp__podcli__".length))
+ .filter((name) => !toolNames.has(name));
+ expect(unknown, `${file} allows unknown tools: ${unknown.join(", ")}`).toEqual([]);
+ });
+
+ it(`${file}: every backtick tool-call reference in the body names a registered tool`, () => {
+ const source = readFileSync(join(commandsDir, file), "utf-8");
+ const { body } = parseFrontmatter(source);
+ // Only call-shaped references, e.g. `set_video(file_path)` or
+ // `job_status(job_id, wait_seconds: 30)`. Bare backticked words like
+ // `payoff` or `async_mode` are field names, not tool calls.
+ const calls = Array.from(body.matchAll(/`([a-z][a-z0-9_]*)\(/g)).map((m) => m[1]);
+ const unknown = calls.filter((name) => !toolNames.has(name));
+ expect(unknown, `${file} calls unknown tool(s): ${unknown.join(", ")}`).toEqual([]);
+ });
+ }
+});
diff --git a/src/record-decisions.test.ts b/src/record-decisions.test.ts
new file mode 100644
index 0000000..7a3a20d
--- /dev/null
+++ b/src/record-decisions.test.ts
@@ -0,0 +1,145 @@
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// record_decisions is the MCP way to answer the per-episode open questions
+// (clip count, caption style, language, ...) once instead of every run.
+// These tests exercise the registered tool end to end against a real
+// EpisodeState file, not a mock, so a refactor that breaks the wiring
+// between the tool and the service shows up here.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-record-decisions-"));
+const savedHome = process.env.PODCLI_HOME;
+const savedData = process.env.PODCLI_DATA;
+const savedPort = process.env.PODCLI_PORT;
+
+let createServer: typeof import("./server.js").createServer;
+let recordDecisionsInputSchema: typeof import("./server.js").recordDecisionsInputSchema;
+let videoPath: string;
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ // Force readUIState to fall back to the on-disk ui-state.json instead of
+ // reaching a real studio server that happens to be running on the default
+ // port during local development.
+ process.env.PODCLI_PORT = "18423";
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+ ({ createServer, recordDecisionsInputSchema } = await import("./server.js"));
+ videoPath = join(tmp, "episode.mp4");
+ writeFileSync(videoPath, "fake video bytes");
+});
+
+afterAll(() => {
+ if (savedHome === undefined) delete process.env.PODCLI_HOME;
+ else process.env.PODCLI_HOME = savedHome;
+ if (savedPort === undefined) delete process.env.PODCLI_PORT;
+ else process.env.PODCLI_PORT = savedPort;
+ if (savedData === undefined) delete process.env.PODCLI_DATA;
+ else process.env.PODCLI_DATA = savedData;
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function getHandler(name: string) {
+ const server = createServer() as unknown as {
+ _registeredTools: Record<string, { handler: (args: unknown, extra: unknown) => Promise<{ content: { text: string }[] }> }>;
+ };
+ return server._registeredTools[name].handler;
+}
+
+describe("record_decisions", () => {
+ it("records a decision and reports what's still open", async () => {
+ const handler = getHandler("record_decisions");
+ const result = await handler({ video_path: videoPath, clip_count: 5 }, {});
+ const text = result.content[0].text;
+ expect(text).toContain("clipCount");
+ expect(text).toContain("Still open");
+ });
+
+ it("reports every decision answered once all fields are recorded", async () => {
+ const handler = getHandler("record_decisions");
+ await handler(
+ {
+ video_path: videoPath,
+ clip_count: 5,
+ clip_duration_min: 20,
+ clip_duration_max: 45,
+ caption_style: "hormozi",
+ captions_enabled: true,
+ language: "en",
+ thumbnails_wanted: true,
+ delivery_target: "youtube_shorts",
+ },
+ {},
+ );
+ const result = await handler({ video_path: videoPath, notes: "final note" }, {});
+ expect(result.content[0].text).toContain("All episode decisions answered.");
+ });
+
+ it("rejects a call with no fields to record", async () => {
+ const handler = getHandler("record_decisions");
+ const result = await handler({ video_path: videoPath }, {});
+ expect(result.content[0].text).toContain("No decisions provided");
+ });
+
+ it("keys decisions on the pre-silence-removal video, not its derivative", async () => {
+ // A silence-removal pass rewrites the working video to a new path with a
+ // different size, which would otherwise orphan decisions already recorded
+ // against the original (path+size+mtime no longer matches).
+ const originalPath = join(tmp, "original.mp4");
+ const derivativePath = join(tmp, "original.silence-removed.mp4");
+ writeFileSync(originalPath, "original bytes");
+ writeFileSync(derivativePath, "a shorter derivative");
+
+ writeFileSync(
+ join(process.env.PODCLI_HOME!, "ui-state.json"),
+ JSON.stringify({
+ videoPath: derivativePath,
+ silenceOriginal: { videoPath: originalPath, transcript: null },
+ }),
+ );
+
+ const handler = getHandler("record_decisions");
+ await handler({ video_path: derivativePath, clip_count: 7 }, {});
+
+ const { EpisodeState } = await import("./services/episode-state.js");
+ const state = new EpisodeState();
+ expect((await state.get(originalPath))?.clipCount).toBe(7);
+ expect(await state.get(derivativePath)).toBeNull();
+ });
+});
+
+describe("recordDecisionsInputSchema", () => {
+ // An unrecognized key like captions (instead of captions_enabled) must be
+ // a schema error naming the valid fields, not silently stripped and
+ // recorded as nothing, which is what z.object's default behavior does.
+ it("rejects an unknown field and names the valid fields in the error", () => {
+ const result = recordDecisionsInputSchema.safeParse({ video_path: videoPath, captions: true });
+ expect(result.success).toBe(false);
+ if (!result.success) {
+ const message = result.error.issues[0].message;
+ expect(message).toContain("captions");
+ expect(message).toContain("captions_enabled");
+ expect(message).toContain("clip_count");
+ }
+ });
+
+ it("accepts a call with only recognized fields", () => {
+ const result = recordDecisionsInputSchema.safeParse({ video_path: videoPath, clip_count: 5 });
+ expect(result.success).toBe(true);
+ });
+});
+
+describe("record_decisions duration range", () => {
+ it("stores only the side given, never a 0 for the missing one", async () => {
+ const { EpisodeState } = await import("./services/episode-state.js");
+ const fresh = join(tmp, "fresh-episode.mp4");
+ writeFileSync(fresh, "another fake video");
+ const handler = getHandler("record_decisions");
+ await handler({ video_path: fresh, clip_duration_min: 30 }, {});
+ expect((await new EpisodeState().get(fresh))?.clipDurationRange).toEqual({ min: 30 });
+ await handler({ video_path: fresh, clip_duration_max: 60 }, {});
+ expect((await new EpisodeState().get(fresh))?.clipDurationRange).toEqual({ min: 30, max: 60 });
+ });
+});
diff --git a/src/server.ts b/src/server.ts
index 2419bd8..713c6ba 100644
--- a/src/server.ts
+++ b/src/server.ts
@@ -1,6 +1,6 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
-import { readFileSync, writeFileSync } from "fs";
+import { existsSync, readFileSync, writeFileSync } from "fs";
import {
transcribeToolDef,
@@ -10,10 +10,15 @@ import {
jobStatusToolDef,
handleJobStatus,
} from "./handlers/transcribe.handler.js";
+import {
+ compareEnginesToolDef,
+ handleCompareEngines,
+} from "./handlers/compare-engines.handler.js";
import {
suggestClipsToolDef,
suggestClipsInputShape,
handleSuggestClips,
+ hookSchema,
} from "./handlers/suggest-clips.handler.js";
import {
createClipToolDef,
@@ -24,17 +29,21 @@ import {
handleBatchClips,
} from "./handlers/batch-clips.handler.js";
import { registerIntegrationMcpTools } from "./handlers/integrations.handler.js";
+import { mineChannelToolDef, handleMineChannel } from "./handlers/mine-channel.handler.js";
+import { isHttpUrl } from "./utils/ytdlp-args.js";
import { FileManager } from "./services/file-manager.js";
import { KnowledgeBase } from "./services/knowledge-base.js";
import { AssetManager, inferType } from "./services/asset-manager.js";
import { ClipsHistory } from "./services/clips-history.js";
import { TranscriptCache } from "./services/transcript-cache.js";
+import { EpisodeState } from "./services/episode-state.js";
import { paths } from "./config/paths.js";
import { webServerUrl } from "./config/server.js";
import { childLogger } from "./utils/logger.js";
import { mcpError } from "./utils/errors.js";
-import { podcliVersion } from "./version.js";
-import type { Format, SuggestedClip, UIState, WordTimestamp } from "./models/index.js";
+import { transcriptVideoMismatch } from "./utils/video-identity.js";
+import { podcliVersion, studioStartCommand } from "./version.js";
+import type { ClipHook, Format, SuggestedClip, UIState, WordTimestamp } from "./models/index.js";
const log = childLogger("server");
@@ -75,6 +84,7 @@ async function uiPing(body: Record<string, unknown>): Promise<void> {
const kb = new KnowledgeBase();
const assets = new AssetManager();
const history = new ClipsHistory();
+const episodeState = new EpisodeState();
/** Prepend knowledge base file listing to a tool result. */
async function withKnowledge(result: string): Promise<string> {
@@ -97,6 +107,33 @@ function withNextStep(result: string, nextStep: string): string {
// long episode; cap what goes into a tool response.
const TRANSCRIPT_FALLBACK_CAP = 50_000;
+// Every other tool's schema uses z.object's default behavior, which strips
+// an unknown key silently, so a caller passing captions: true instead of
+// captions_enabled gets no error and nothing recorded. record_decisions is
+// strict on purpose: it's meant to be the durable record of a decision, so a
+// typo'd key should fail loudly rather than quietly not stick.
+const recordDecisionsInputShape = {
+ video_path: z.string().describe("The episode's source video path (same path used with set_video/transcribe_podcast)"),
+ clip_count: z.number().optional().describe("How many clips to produce"),
+ clip_duration_min: z.number().optional().describe("Minimum target clip duration in seconds"),
+ clip_duration_max: z.number().optional().describe("Maximum target clip duration in seconds"),
+ caption_style: z.enum(["hormozi", "karaoke", "subtle", "branded"]).optional(),
+ captions_enabled: z.boolean().optional().describe("Whether clips should have captions burned in at all"),
+ language: z.string().optional().describe("The episode's spoken language"),
+ thumbnails_wanted: z.boolean().optional(),
+ delivery_target: z.string().optional().describe("Where clips are headed, e.g. youtube_shorts, tiktok, instagram, export_only"),
+ notes: z.string().optional().describe("Free-form notes that don't fit another field"),
+};
+
+export const recordDecisionsInputSchema = z.strictObject(recordDecisionsInputShape, {
+ // Returning undefined keeps zod's own message for every other issue.
+ error: (issue) =>
+ issue.code === "unrecognized_keys"
+ ? `Unknown field(s): ${issue.keys.join(", ")}. Valid fields: ` +
+ Object.keys(recordDecisionsInputShape).join(", ")
+ : undefined,
+});
+
function capTranscriptText(text: string): string {
if (text.length <= TRANSCRIPT_FALLBACK_CAP) return text;
return (
@@ -165,17 +202,56 @@ interface ImportTranscriptResult extends ApiError {
};
}
+/**
+ * Read session state straight off disk (paths.uiState) instead of the Web
+ * UI's HTTP API. Used when the Web UI isn't running. The state file is the
+ * same JSON the UI persists on every change, so the agent isn't blind just
+ * because nothing is listening on webServerUrl.
+ */
+function readUIStateFromDisk(): ServerUIState | null {
+ try {
+ if (!existsSync(paths.uiState)) return null;
+ const raw = JSON.parse(readFileSync(paths.uiState, "utf-8")) as UIState;
+ const words = raw.transcript?.words;
+ return {
+ ...raw,
+ transcriptWordCount: Array.isArray(words) ? words.length : 0,
+ };
+ } catch (err) {
+ log.debug("readUIStateFromDisk failed", {
+ err: err instanceof Error ? err.message : String(err),
+ });
+ return null;
+ }
+}
+
async function readUIState(): Promise<ServerUIState | null> {
try {
const res = await fetch(`${webServerUrl}/api/ui-state`);
- if (!res.ok) return null;
+ if (!res.ok) return readUIStateFromDisk();
return (await res.json()) as ServerUIState;
} catch (err) {
- log.debug("readUIState failed (UI likely not running)", {
+ log.debug("readUIState via Web UI failed, falling back to disk", {
err: err instanceof Error ? err.message : String(err),
});
- return null;
+ return readUIStateFromDisk();
+ }
+}
+
+/**
+ * Resolve the path episode decisions should be keyed on. A silence-removal
+ * pass rewrites the working video to a new path with a different size, which
+ * would otherwise orphan decisions already recorded against the original.
+ * When videoPath is the UI's current (derivative) video and a silenceOriginal
+ * is on record, key on the original instead so decisions carry over.
+ */
+async function resolveEpisodeKeyPath(videoPath: string): Promise<string> {
+ const state = await readUIState();
+ const original = state?.silenceOriginal?.videoPath;
+ if (original && (state?.videoPath === videoPath || state?.filePath === videoPath)) {
+ return original;
}
+ return videoPath;
}
/** Generate workflow guidance based on current state. */
@@ -190,7 +266,7 @@ async function getWorkflowGuidance(): Promise<string> {
"4. Suggest clips: analyze the transcript yourself, then call suggest_clips with your picks.\n" +
" Each pick needs a payoff and a standalone check, and an answer needs its question.\n" +
"5. Export: use batch_create_clips(export_selected: true) or create_clip(clip_number: N)\n\n" +
- "Note: The Web UI is not running. Start it with: npm run ui"
+ `Note: The Web UI is not running. Start it with: ${studioStartCommand()}`
);
}
@@ -250,6 +326,21 @@ async function getWorkflowGuidance(): Promise<string> {
);
}
+ const videoPath = state.videoPath || state.filePath;
+ if (videoPath) {
+ // A silence-removal pass rewrites the video to a new path with a different
+ // size, which would otherwise orphan decisions already recorded against the
+ // original. Key on the original so they carry over to its derivative.
+ const episodeKeyPath = state.silenceOriginal?.videoPath ?? videoPath;
+ const openQuestions = await episodeState.openQuestions(episodeKeyPath).catch(() => []);
+ if (openQuestions.length > 0) {
+ lines.push(
+ "\nOPEN QUESTIONS for this episode. Ask once, then call record_decisions so these never come up again:\n" +
+ openQuestions.map((q) => ` → ${q.question}`).join("\n"),
+ );
+ }
+ }
+
return lines.join("\n");
}
@@ -273,7 +364,7 @@ export function createServer(): McpServer {
.default("base")
.describe("Whisper model size"),
engine: z
- .enum(["whisper-py", "whispercpp", "assemblyai"])
+ .enum(["whisper-py", "whispercpp", "assemblyai", "omnilingual"])
.optional()
.describe("Transcription engine"),
language: z.string().optional().describe("ISO language code"),
@@ -292,6 +383,18 @@ export function createServer(): McpServer {
.describe(
"Exact number of speakers if known (e.g. 2). Auto-detects if omitted.",
),
+ start_seconds: z
+ .number()
+ .optional()
+ .describe(
+ "Sample mode: only transcribe a window starting here (seconds into the source), " +
+ "instead of the whole file, e.g. to test a language on 40s before committing to " +
+ "a full run. Pair with duration_seconds. Not written to the main transcript cache.",
+ ),
+ duration_seconds: z
+ .number()
+ .optional()
+ .describe("Sample mode window length in seconds. Defaults start_seconds to 0 if omitted."),
},
async ({
file_path,
@@ -300,6 +403,8 @@ export function createServer(): McpServer {
language,
enable_diarization,
num_speakers,
+ start_seconds,
+ duration_seconds,
}) => {
try {
const result = await handleTranscribe({
@@ -309,13 +414,29 @@ export function createServer(): McpServer {
language,
enable_diarization,
num_speakers,
+ start_seconds,
+ duration_seconds,
});
// Push FULL transcript (words[] + segments[]) to Web UI state from the
// on-disk cache — NOT the trimmed MCP response. Without words in UI
// state, downstream batch_create_clips can't burn captions.
+ // Skipped for a sample: it's never written to this cache, and it's a
+ // slice of the file, not something that belongs in the UI session.
+ const isSample = start_seconds !== undefined || duration_seconds !== undefined;
try {
- const cached = await transcriptCache.get(file_path, engine);
+ // handleTranscribe's result carries the engine it actually resolved
+ // to and cached under; the request-time `engine` can be unset while
+ // the write landed under "whispercpp", so reading with it misses.
+ const resolvedEngine =
+ (JSON.parse(result) as { engine?: string }).engine ?? engine;
+ const cached = isSample
+ ? null
+ : await transcriptCache.get(file_path, {
+ engine: resolvedEngine,
+ model: model_size,
+ language,
+ });
if (cached) {
await uiPing({
videoPath: file_path,
@@ -355,7 +476,7 @@ export function createServer(): McpServer {
.optional()
.default("base"),
language: z.string().optional(),
- engine: z.enum(["whisper-py", "whispercpp", "assemblyai"]).optional(),
+ engine: z.enum(["whisper-py", "whispercpp", "assemblyai", "omnilingual"]).optional(),
enable_diarization: z.boolean().optional().default(true),
num_speakers: z.number().optional(),
},
@@ -530,6 +651,19 @@ export function createServer(): McpServer {
.describe(
"Keep ProRes 4444 alpha caption overlay beside the render (for DaVinci Resolve export). Returns caption_overlay_path and cropped_source_path.",
),
+ write_clean_variant: z
+ .boolean()
+ .optional()
+ .default(false)
+ .describe(
+ "Also render a second file with the same audio, loudness, and intro/outro but no burned captions. Returns clean_output_path.",
+ ),
+ hook: hookSchema
+ .nullable()
+ .optional()
+ .describe(
+ "Opening hook for this render. Auto-loaded from clip_number if omitted; null renders without one.",
+ ),
},
async (params) => {
try {
@@ -541,6 +675,7 @@ export function createServer(): McpServer {
// Resolve clip_number from UI state BEFORE routing
let keepSegments: Array<{ start: number; end: number }> | null = null;
+ let hook: ClipHook | null = params.hook ?? null;
if (
params.clip_number != null &&
(params.start_second == null || params.end_second == null)
@@ -578,7 +713,20 @@ export function createServer(): McpServer {
}
if (!params.transcript_words) {
const transcript = uiState?.transcript;
- if (transcript?.words) params.transcript_words = transcript.words;
+ if (transcript?.words) {
+ // handleCreateClip only runs this guard when transcript_words
+ // arrives null, so filling it from state here has to run the
+ // check itself or an offline render against a swapped video
+ // would silently skip it.
+ const mismatch = transcriptVideoMismatch(
+ uiState?.transcriptVideoIdentity,
+ params.video_path as string,
+ );
+ if (mismatch) {
+ return { content: [{ type: "text" as const, text: mismatch }], isError: true };
+ }
+ params.transcript_words = transcript.words;
+ }
}
// Pull multi-cut segments from suggestion
const segs = suggestion.segments as
@@ -587,6 +735,7 @@ export function createServer(): McpServer {
if (segs && segs.length > 0) {
keepSegments = segs;
}
+ if (params.hook === undefined && suggestion.hook) hook = suggestion.hook;
}
// Check for duplicates
@@ -628,7 +777,9 @@ export function createServer(): McpServer {
format: params.format || "vertical",
allow_ass_fallback: params.allow_ass_fallback === true,
keep_caption_overlay: params.keep_caption_overlay === true,
+ write_clean_variant: params.write_clean_variant === true,
...(keepSegments && { segments: keepSegments }),
+ hook,
},
],
transcript_words: params.transcript_words,
@@ -638,6 +789,7 @@ export function createServer(): McpServer {
clean_fillers: params.clean_fillers,
}),
keep_caption_overlay: params.keep_caption_overlay === true,
+ write_clean_variant: params.write_clean_variant === true,
}),
});
if (webRes.ok) {
@@ -682,7 +834,7 @@ export function createServer(): McpServer {
// Notify UI that export is starting
await uiPing({ phase: "exporting" });
- finalResult = await handleCreateClip(params);
+ finalResult = await handleCreateClip({ ...params, hook });
const parsed = JSON.parse(finalResult);
// Record to history
@@ -700,6 +852,9 @@ export function createServer(): McpServer {
duration: parsed.duration,
content_type: parsed.content_type,
transcript_slice: parsed.transcript_slice,
+ payoff: parsed.payoff,
+ context_line: parsed.context_line,
+ preview_text: parsed.preview_text,
});
const uiStateForRecipe = await readUIState().catch(() => null);
const recipeSettings = uiStateForRecipe?.settings ?? {};
@@ -710,6 +865,7 @@ export function createServer(): McpServer {
introPath: recipeSettings.introPath || null,
cleanFillers: params.clean_fillers ?? recipeSettings.cleanFillers ?? true,
keepSegments: keepSegments ?? undefined,
+ hook,
});
// Notify UI that export is done
@@ -756,12 +912,20 @@ export function createServer(): McpServer {
format: z.enum(["vertical", "horizontal", "square"]).optional(),
allow_ass_fallback: z.boolean().optional(),
keep_caption_overlay: z.boolean().optional(),
+ write_clean_variant: z.boolean().optional(),
+ hook: hookSchema.nullable().optional(),
}),
)
.optional()
.describe(
"Array of clips to create. Auto-loaded from suggestions if omitted.",
),
+ write_clean_variant: z
+ .boolean()
+ .optional()
+ .describe(
+ "Also render a clean (no burned captions) variant per clip, batch-level default; per-clip overrides. Returns clean_output_path.",
+ ),
keep_caption_overlay: z
.boolean()
.optional()
@@ -823,7 +987,20 @@ export function createServer(): McpServer {
"") as string;
if (!resolvedTranscriptWords) {
const transcript = uiState?.transcript;
- if (transcript?.words) resolvedTranscriptWords = transcript.words;
+ if (transcript?.words) {
+ // handleBatchClips only runs this guard when transcript_words
+ // arrives null, so filling it from state here has to run the
+ // check itself or an offline render against a swapped video
+ // would silently skip it.
+ const mismatch = transcriptVideoMismatch(
+ uiState?.transcriptVideoIdentity,
+ resolvedVideoPath as string,
+ );
+ if (mismatch) {
+ return { content: [{ type: "text" as const, text: mismatch }], isError: true };
+ }
+ resolvedTranscriptWords = transcript.words;
+ }
}
if (params.export_selected) {
@@ -842,6 +1019,7 @@ export function createServer(): McpServer {
allow_ass_fallback: false,
...(s.segments &&
s.segments.length > 0 && { keep_segments: s.segments }),
+ hook: s.hook ?? null,
})) as any;
} else if (params.clip_numbers) {
resolvedClips = (params.clip_numbers as number[])
@@ -861,6 +1039,7 @@ export function createServer(): McpServer {
allow_ass_fallback: false,
...(s.segments &&
s.segments.length > 0 && { keep_segments: s.segments }),
+ hook: s.hook ?? null,
};
}) as any;
}
@@ -882,6 +1061,7 @@ export function createServer(): McpServer {
clean_fillers: params.clean_fillers,
}),
keep_caption_overlay: params.keep_caption_overlay === true,
+ write_clean_variant: params.write_clean_variant === true,
}),
});
if (webRes.ok) {
@@ -1358,13 +1538,22 @@ export function createServer(): McpServer {
},
async ({ include_transcript }) => {
try {
- const res = await fetch(`${webServerUrl}/api/ui-state`);
- if (!res.ok) throw new Error(`HTTP ${res.status}`);
- const state = (await res.json()) as ServerUIState;
+ const state = await readUIState();
+ if (!state) {
+ const guidance = await getWorkflowGuidance();
+ return { content: [{ type: "text" as const, text: guidance }] };
+ }
const lines: string[] = [];
lines.push(`Phase: ${state.phase}`);
lines.push(`Video: ${state.videoPath || state.filePath || "(none)"}`);
+ if (state.videoMissing) {
+ lines.push(
+ "WARNING: the video file above was not found on disk at last check (e.g. an external " +
+ "drive may be unmounted). The session (transcript, suggestions) is kept, but create_clip/" +
+ "batch_create_clips will fail until the file is reachable again.",
+ );
+ }
lines.push(
`Settings: caption=${state.settings?.captionStyle}, crop=${state.settings?.cropStrategy}, logo=${state.settings?.logoPath || "none"}`,
);
@@ -1393,7 +1582,11 @@ export function createServer(): McpServer {
? `${Math.round(end - start)}s`
: "?";
const style = clip.suggested_caption_style || "hormozi";
- const tag = deselected.includes(i) ? " [DESELECTED]" : "";
+ const tag = deselected.includes(i)
+ ? " [DESELECTED]"
+ : clip.changedSinceSelection
+ ? " [CHANGED SINCE SELECTION, re-confirm before exporting]"
+ : "";
lines.push(
` #${num}: "${title}" (${start}s–${end}s, ${duration}) [${style}]${tag}`,
);
@@ -1452,17 +1645,6 @@ export function createServer(): McpServer {
return { content: [{ type: "text" as const, text: lines.join("\n") }] };
} catch (err: unknown) {
const msg = err instanceof Error ? err.message : String(err);
- if (msg.includes("ECONNREFUSED") || msg.includes("fetch failed")) {
- const guidance = await getWorkflowGuidance();
- return {
- content: [
- {
- type: "text" as const,
- text: `Web UI is not running. Start with: npm run ui\n\n${guidance}`,
- },
- ],
- };
- }
return {
content: [
{ type: "text" as const, text: `Error reading UI state: ${msg}` },
@@ -1478,7 +1660,7 @@ export function createServer(): McpServer {
// =============================================
server.tool(
"modify_clip",
- "Adjust a suggested clip before exporting. Change timing, title, or caption style. " +
+ "Adjust a suggested clip before exporting. Change timing, title, caption style, or opening hook. " +
"Use action='delete' to remove a clip entirely. Reference clips by clip_number (from get_ui_state).",
{
clip_number: z
@@ -1511,6 +1693,10 @@ export function createServer(): McpServer {
suggested_caption_style: z
.enum(["hormozi", "karaoke", "subtle", "branded"])
.optional(),
+ hook: hookSchema
+ .nullable()
+ .optional()
+ .describe("Set the opening hook, or pass null to clear it."),
})
.optional()
.describe(
@@ -1524,7 +1710,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "No updates provided. Specify at least one field: title, start_second, end_second, payoff, standalone, context_line, reasoning, preview_text, or suggested_caption_style.",
+ text: "No updates provided. Specify at least one field: title, start_second, end_second, payoff, standalone, context_line, reasoning, preview_text, suggested_caption_style, or hook.",
},
],
};
@@ -1576,7 +1762,10 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: `Updated clip #${data.index + 1}: "${data.clip.title}" (${data.clip.start_second}s–${data.clip.end_second}s, ${data.clip.duration}s)`,
+ text: `Updated clip #${data.index + 1}: "${data.clip.title}" (${data.clip.start_second}s-${data.clip.end_second}s, ${data.clip.duration}s` +
+ (data.clip.hook
+ ? `, opens with a ${data.clip.hook.mode} hook ${data.clip.hook.start}s-${data.clip.hook.end}s)`
+ : ")"),
},
],
};
@@ -1587,7 +1776,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -1670,7 +1859,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -1762,7 +1951,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -1775,6 +1964,69 @@ export function createServer(): McpServer {
},
);
+ // =============================================
+ // Tool: record_decisions
+ // =============================================
+ server.registerTool(
+ "record_decisions",
+ {
+ description:
+ "Record per-episode workflow decisions (clip count, clip duration range, caption style, captions on/off, " +
+ "language, whether thumbnails are wanted, delivery target, free-form notes) so later runs against the same " +
+ "video never ask the same question twice. Keyed by the video's path + file size, not by session, so these " +
+ "answers survive a new episode overwriting ui-state.json. Pass only the fields you have an answer for; " +
+ "existing answers are preserved unless explicitly overwritten. get_ui_state lists any fields still unanswered " +
+ "as open questions.",
+ inputSchema: recordDecisionsInputSchema,
+ },
+ async ({ video_path, clip_count, clip_duration_min, clip_duration_max, caption_style, captions_enabled, language, thumbnails_wanted, delivery_target, notes }) => {
+ try {
+ const keyPath = await resolveEpisodeKeyPath(video_path);
+ const decisions: Record<string, unknown> = {};
+ if (clip_count !== undefined) decisions.clipCount = clip_count;
+ if (clip_duration_min !== undefined || clip_duration_max !== undefined) {
+ const existing = await episodeState.get(keyPath);
+ // A side nobody gave stays absent; 0 would read as a real limit.
+ const min = clip_duration_min ?? existing?.clipDurationRange?.min;
+ const max = clip_duration_max ?? existing?.clipDurationRange?.max;
+ decisions.clipDurationRange = {
+ ...(min !== undefined && { min }),
+ ...(max !== undefined && { max }),
+ };
+ }
+ if (caption_style !== undefined) decisions.captionStyle = caption_style;
+ if (captions_enabled !== undefined) decisions.captionsEnabled = captions_enabled;
+ if (language !== undefined) decisions.language = language;
+ if (thumbnails_wanted !== undefined) decisions.thumbnailsWanted = thumbnails_wanted;
+ if (delivery_target !== undefined) decisions.deliveryTarget = delivery_target;
+ if (notes !== undefined) decisions.notes = notes;
+
+ if (Object.keys(decisions).length === 0) {
+ return {
+ content: [{ type: "text" as const, text: "No decisions provided. Pass at least one field to record." }],
+ };
+ }
+
+ const recorded = await episodeState.record(keyPath, decisions);
+ const remaining = await episodeState.openQuestions(keyPath);
+ const lines = [
+ `Recorded: ${Object.keys(decisions).join(", ")}`,
+ remaining.length > 0
+ ? `Still open: ${remaining.map((q) => q.question).join(" / ")}`
+ : "All episode decisions answered.",
+ ];
+ void recorded;
+ return { content: [{ type: "text" as const, text: lines.join("\n") }] };
+ } catch (err: unknown) {
+ const msg = err instanceof Error ? err.message : String(err);
+ return {
+ content: [{ type: "text" as const, text: `Error recording decisions: ${msg}` }],
+ isError: true,
+ };
+ }
+ },
+ );
+
// =============================================
// Tool: list_outputs
// =============================================
@@ -1816,7 +2068,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -1953,7 +2205,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -1971,17 +2223,35 @@ export function createServer(): McpServer {
// =============================================
server.tool(
"manage_thumbnail_config",
- "Show, export, import, or reset the thumbnail template (colors, fonts, frame, box, layout) podcli uses to generate thumbnails. 'show' returns the effective config; 'export' writes it to a file path; 'import' replaces it from a file path; 'reset' reverts to the generic default.",
+ "Show, export, import, or reset the thumbnail template (colors, fonts, frame, box, layout) podcli uses to generate thumbnails. 'show' returns the effective config; 'export' writes it to a file path; 'import' replaces it from a file path; 'reset' reverts to the generic default; 'set_layout' picks the layout: 'single' (one face) or 'pair' (two people from the clip, guest left and host right, for interview clips; falls back to one face when podcli cannot tell two people apart).",
{
- action: z.enum(["show", "export", "import", "reset"]).describe("Config action"),
+ action: z.enum(["show", "export", "import", "reset", "set_layout"]).describe("Config action"),
path: z.string().optional().describe("File path for export (destination) or import (source)"),
+ layout: z
+ .enum(["single", "pair"])
+ .optional()
+ .describe("For set_layout. single: one face behind the headline. pair: the guest left and the host right, both from the clip's own footage"),
},
- async ({ action, path: filePath }) => {
+ async ({ action, path: filePath, layout }) => {
try {
if ((action === "export" || action === "import") && !filePath) {
return mcpError(`'path' is required for action '${action}'.`);
}
+ if (action === "set_layout" && !layout) {
+ return mcpError("'layout' is required for action 'set_layout': 'single' or 'pair'.");
+ }
const base = `${webServerUrl}/api/thumbnail-config`;
+ if (action === "set_layout") {
+ const current = await fetch(base);
+ if (!current.ok) throw new Error(`HTTP ${current.status}`);
+ const res = await fetch(base, {
+ method: "PUT",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ ...(await current.json()), layout }),
+ });
+ if (!res.ok) throw new Error(`HTTP ${res.status}`);
+ return { content: [{ type: "text" as const, text: `Thumbnail layout set to ${layout}.` }] };
+ }
if (action === "show") {
const res = await fetch(base);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
@@ -2005,7 +2275,7 @@ export function createServer(): McpServer {
} catch (err: unknown) {
const msg = err instanceof Error ? err.message : String(err);
if (msg.includes("ECONNREFUSED") || msg.includes("fetch failed")) {
- return { content: [{ type: "text" as const, text: "Web UI is not running. Start with: npm run ui" }] };
+ return { content: [{ type: "text" as const, text: `Web UI is not running. Start with: ${studioStartCommand()}` }] };
}
return mcpError(msg);
}
@@ -2105,7 +2375,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -2207,7 +2477,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -2231,9 +2501,10 @@ export function createServer(): McpServer {
"'new' returns the session with guessed roles; calling it again on the same files reopens that edit. " +
"'sync', 'plan' and 'render' start a background job and return job_id: poll job_status, then call 'show'. " +
"Mapping fields (people, sources, range_start, range_end, cut_settings, speaker_map, look, removals) apply on 'map', 'sync' and 'plan'; 'render' takes only look and stems. " +
- "Changing who is in a file or where it sits clears the cut, so run 'plan' again. 'render' skips work when nothing changed. " +
+ "Changing who is in a file or where it sits clears the cut, so run 'plan' again. 'render' skips work when nothing changed and reuses shots it already encoded. " +
+ "Every render checks the finished file (frame count, a full decode, loudness and true peak) and reports outputs.validation; only a broken decode or a picture more than a frame off the cut fails it, the rest are warnings. " +
"Other actions: 'list', 'cut' (index, source_id: swap one shot's camera), 'set_cuts' (cuts: replace the whole cut with back-to-back shots), " +
- "'activity' (who speaks when, as spans per person), 'previews' (still frames per camera, looks: true adds color-look stills), " +
+ "'activity' (who speaks when, as spans per person), 'previews' (still frames per camera, looks: true adds every color look on every camera), " +
"'preview' (background job: playback proxies, a mic mix and stills, for a browser editor such as podcli cloud), 'delete'. " +
"'cloud' (background job, needs podcli login and Pro: sends previews and the edit to the podcli cloud editor; camera files stay here) and " +
"'pull' (background job: renders the cut made in that editor from the files here). " +
@@ -2260,8 +2531,19 @@ export function createServer(): McpServer {
role: z.enum(["camera", "mic", "ignore"]).optional(),
person: z.string().optional().describe("Camera: a person id or 'wide'. Mic: a person id, or '' for a shared room mic"),
channel_people: z.array(z.string()).optional().describe("Mic: one person id per channel when a recorder puts two people on L/R"),
+ input_lut: z.string().optional().describe("Camera: absolute path to a 3D .cube LUT (log to Rec.709, say) applied before the look in renders and stills; '' clears it. Exports list it in color_handoff.json beside the timeline"),
+ audio_stream_index: z.number().int().min(0).optional().describe("Which audio stream in the container to use, for cameras (often MXF) that carry one mono stream per mic instead of packing channels into a single stream"),
offset: z.number().optional().describe("Timeline seconds where this file starts, to override sync"),
nudge: z.number().optional().describe("Seconds to shift the synced offset by"),
+ anchors: z
+ .array(z.object({ timeline: z.number(), source: z.number() }))
+ .min(1)
+ .optional()
+ .describe(
+ "Manual sync for a file that can't sync by sound (a camera with no audio, or a fit that's off): " +
+ "pairs of the same moment as {timeline, source} seconds. One pair sets the offset; two or more also fit clock drift by least squares. " +
+ "Pairs must move forward together, and drift must stay within 1000 ppm. The fit reports its residual and survives later syncs unless force is true",
+ ),
}),
)
.optional()
@@ -2315,18 +2597,22 @@ export function createServer(): McpServer {
isError: true,
};
}
+ const validation = (data.outputs as { validation?: { warnings?: string[] } } | undefined)?.validation;
+ const warnings = validation?.warnings?.length
+ ? `\n\n[Render warnings]\n${validation.warnings.map((w) => ` - ${w}`).join("\n")}`
+ : "";
const text = typeof data.job_id === "string"
? withNextStep(
JSON.stringify(data, null, 2),
`Poll job_status("${data.job_id}", wait_seconds: 30) until done, then manage_multicam(action: "show", session_id) to read the result.`,
)
- : JSON.stringify(data, null, 2);
+ : JSON.stringify(data, null, 2) + warnings;
return { content: [{ type: "text" as const, text }] };
} catch (err: unknown) {
const msg = err instanceof Error ? err.message : String(err);
if (msg.includes("ECONNREFUSED") || msg.includes("fetch failed")) {
return {
- content: [{ type: "text" as const, text: "Web UI is not running. Start with: npm run ui" }],
+ content: [{ type: "text" as const, text: `Web UI is not running. Start with: ${studioStartCommand()}` }],
};
}
return mcpError(err);
@@ -2386,7 +2672,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -2487,7 +2773,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -2520,8 +2806,15 @@ export function createServer(): McpServer {
.optional()
.default(0)
.describe("Offset in seconds to add to all timestamps"),
+ language: z
+ .string()
+ .optional()
+ .describe(
+ "ISO language code of the transcript (e.g. 'ka'). This format has no language " +
+ "info of its own, so omitting it labels the result 'und' rather than guessing.",
+ ),
},
- async ({ file_path, raw_text, total_duration, time_adjust }) => {
+ async ({ file_path, raw_text, total_duration, time_adjust, language }) => {
try {
const res = await fetch(`${webServerUrl}/api/parse-transcript`, {
method: "POST",
@@ -2531,6 +2824,7 @@ export function createServer(): McpServer {
raw_text,
total_duration,
time_adjust,
+ language,
}),
});
if (!res.ok) {
@@ -2575,7 +2869,7 @@ export function createServer(): McpServer {
content: [
{
type: "text" as const,
- text: "Web UI is not running. Start with: npm run ui",
+ text: `Web UI is not running. Start with: ${studioStartCommand()}`,
},
],
};
@@ -2588,6 +2882,73 @@ export function createServer(): McpServer {
},
);
+ // =============================================
+ // Tool: mine_channel
+ // =============================================
+ server.tool(
+ mineChannelToolDef.name,
+ mineChannelToolDef.description,
+ {
+ action: z.enum(["list", "mine"]).describe("'list' = a channel's uploads, 'mine' = one video's captions"),
+ channel_url: z
+ .string()
+ .optional()
+ .refine((v) => v === undefined || isHttpUrl(v), { message: "channel_url must be an http(s) URL" })
+ .describe("Channel or uploads URL (required for action=list)"),
+ video_url: z
+ .string()
+ .optional()
+ .refine((v) => v === undefined || isHttpUrl(v), { message: "video_url must be an http(s) URL" })
+ .describe("Video URL (required for action=mine)"),
+ limit: z.number().optional().describe("Max uploads to list (action=list)"),
+ cookies_from_browser: z
+ .string()
+ .optional()
+ .describe("Browser to read cookies from for members-only or unlisted content"),
+ },
+ async (input) => {
+ try {
+ const text = await handleMineChannel(input);
+ return { content: [{ type: "text" as const, text }] };
+ } catch (err: unknown) {
+ return mcpError(err);
+ }
+ },
+ );
+
+ // =============================================
+ // Tool: compare_transcription_engines
+ // =============================================
+ server.tool(
+ compareEnginesToolDef.name,
+ compareEnginesToolDef.description,
+ {
+ file_path: z.string().describe("Absolute path to the podcast file"),
+ engine_a: z.enum(["whisper-py", "whispercpp", "assemblyai", "omnilingual"]).describe("First engine to compare"),
+ engine_b: z.enum(["whisper-py", "whispercpp", "assemblyai", "omnilingual"]).describe("Second engine to compare"),
+ start_seconds: z.number().optional().describe("Sample start, seconds into the source. Default: 0."),
+ duration_seconds: z.number().optional().describe("Sample length in seconds. Default: 120."),
+ window_seconds: z.number().optional().describe("Report window size in seconds. Default: 20."),
+ model_size: z
+ .enum(["tiny", "base", "small", "medium", "large"])
+ .optional()
+ .describe("Model size for engines that take one. Default: base."),
+ language: z.string().optional().describe("ISO language code. Leave empty for auto-detect."),
+ output_dir: z
+ .string()
+ .optional()
+ .describe("Where to write comparison.json/.html. Defaults under the podcli output directory."),
+ },
+ async (input) => {
+ try {
+ const text = await handleCompareEngines(input);
+ return { content: [{ type: "text" as const, text }] };
+ } catch (err: unknown) {
+ return mcpError(err);
+ }
+ },
+ );
+
registerIntegrationMcpTools(server);
// =============================================
diff --git a/src/services/clips-history.test.ts b/src/services/clips-history.test.ts
index 93d6deb..a553df1 100644
--- a/src/services/clips-history.test.ts
+++ b/src/services/clips-history.test.ts
@@ -79,6 +79,36 @@ describe("ClipsHistory", () => {
expect(dup).toBeNull();
});
+ it("minedYouTubeVideoIds reads the id out of the yt-dlp output template", async () => {
+ await history.record({
+ source_video: "/videos/Some Episode Title [aBc123_XYZ].mp4",
+ output_path: makeFakeOutput("d.mp4"),
+ start_second: 0,
+ end_second: 10,
+ caption_style: "karaoke",
+ crop_strategy: "smart",
+ title: "clip",
+ } as any);
+
+ const ids = await history.minedYouTubeVideoIds();
+ expect(ids.has("aBc123_XYZ")).toBe(true);
+ });
+
+ it("minedYouTubeVideoIds is empty when no source_video carries a bracketed id", async () => {
+ await history.record({
+ source_video: "/videos/local-recording.mp4",
+ output_path: makeFakeOutput("e.mp4"),
+ start_second: 0,
+ end_second: 10,
+ caption_style: "karaoke",
+ crop_strategy: "smart",
+ title: "clip",
+ } as any);
+
+ const ids = await history.minedYouTubeVideoIds();
+ expect(ids.size).toBe(0);
+ });
+
it("recordBatchResults skips failed/output-less rows and applies defaults", async () => {
const ok = makeFakeOutput("batch-ok.mp4");
const recorded = await history.recordBatchResults(
diff --git a/src/services/clips-history.ts b/src/services/clips-history.ts
index 4ba255a..bd4d2c8 100644
--- a/src/services/clips-history.ts
+++ b/src/services/clips-history.ts
@@ -4,9 +4,9 @@ import { basename, join } from "path";
import { v4 as uuidv4 } from "uuid";
import { paths } from "../config/paths.js";
import { writeFileAtomic } from "../utils/atomic-file.js";
-import { sliceTranscript, sliceWords } from "../utils/transcript.js";
+import { sliceTranscript, sliceWords, findGroundingText } from "../utils/transcript.js";
import { isDemoMode, demoClips } from "../ui/demo-fixtures.js";
-import type { BatchClipsResult, ClipHistoryEntry, Format, WordTimestamp } from "../models/index.js";
+import type { BatchClipsResult, ClipHistoryEntry, ClipHook, Format, WordTimestamp } from "../models/index.js";
type BatchResultRow = BatchClipsResult["results"][number];
@@ -17,12 +17,20 @@ interface BatchRecordContext {
defaultCropStrategy?: string;
defaultFormat?: Format;
contentTypeFor?: (start: number, end: number) => string | undefined;
+ suggestions?: Array<{
+ start_second: number;
+ end_second: number;
+ payoff?: string;
+ context_line?: string;
+ preview_text?: string;
+ }> | null;
}
export interface BatchClipSpec {
start_second: number;
end_second: number;
keep_segments?: Array<{ start: number; end: number }>;
+ hook?: ClipHook | null;
}
export interface BatchRecipeContext {
@@ -184,6 +192,7 @@ export class ClipsHistory {
duration: r.duration || 0,
content_type: ctx.contentTypeFor?.(start, end),
transcript_slice: sliceTranscript(ctx.transcriptWords, start, end),
+ ...findGroundingText(ctx.suggestions, start, end),
}),
);
}
@@ -210,6 +219,7 @@ export class ClipsHistory {
introPath: ctx.introPath,
cleanFillers: ctx.cleanFillers,
keepSegments: spec?.keep_segments,
+ hook: spec?.hook,
});
}
}
@@ -223,6 +233,7 @@ export class ClipsHistory {
introPath?: string | null;
cleanFillers?: boolean;
keepSegments?: Array<{ start: number; end: number }>;
+ hook?: ClipHook | null;
},
): Promise<void> {
const words = sliceWords(ctx.transcriptWords ?? [], rec.start_second, rec.end_second);
@@ -237,6 +248,7 @@ export class ClipsHistory {
clean_fillers: ctx.cleanFillers ?? false,
transcript_words: words,
...(ctx.keepSegments?.length && { keep_segments: ctx.keepSegments }),
+ ...(ctx.hook && { hook: ctx.hook }),
});
if (ctx.keepSegments?.length) {
await this.update(rec.id, { keep_segments: ctx.keepSegments });
@@ -272,6 +284,19 @@ export class ClipsHistory {
);
}
+ // mine_channel's yt-dlp download writes "<title> [<video_id>].<ext>" (see
+ // ytdlp-args.ts's output template), so the id survives in source_video even
+ // though nothing else in the schema tracks where a video came from.
+ async minedYouTubeVideoIds(): Promise<Set<string>> {
+ const entries = await this.load();
+ const ids = new Set<string>();
+ for (const e of entries) {
+ const m = basename(e.source_video || "").match(/\[([A-Za-z0-9_-]{6,})\]\.[^.]+$/);
+ if (m) ids.add(m[1]);
+ }
+ return ids;
+ }
+
async list(limit = 50): Promise<ClipHistoryEntry[]> {
const entries = await this.load();
return entries.slice(-limit).reverse();
diff --git a/src/services/engine-resolve.test.ts b/src/services/engine-resolve.test.ts
new file mode 100644
index 0000000..378729c
--- /dev/null
+++ b/src/services/engine-resolve.test.ts
@@ -0,0 +1,53 @@
+import { describe, it, expect, vi } from "vitest";
+import type { PythonExecutor } from "./python-executor.js";
+import { resolveTranscribeEngine, engineCanDiarize } from "./engine-resolve.js";
+
+function fakeExecutor(response: unknown): PythonExecutor {
+ return { execute: vi.fn().mockResolvedValue(response) } as unknown as PythonExecutor;
+}
+
+describe("resolveTranscribeEngine", () => {
+ it("returns what the backend predicts it will resolve to", async () => {
+ const executor = fakeExecutor({ data: { engine: "whispercpp" } });
+ const engine = await resolveTranscribeEngine(executor, undefined, "base");
+ expect(engine).toBe("whispercpp");
+ expect(executor.execute).toHaveBeenCalledWith("resolve_transcribe_engine", {
+ engine: undefined,
+ model_size: "base",
+ });
+ });
+
+ it("passes an explicit engine through for the backend to normalize", async () => {
+ const executor = fakeExecutor({ data: { engine: "assemblyai" } });
+ const engine = await resolveTranscribeEngine(executor, "assemblyai", "base");
+ expect(engine).toBe("assemblyai");
+ });
+
+ it("falls back to the raw request if resolution itself fails", async () => {
+ const executor = { execute: vi.fn().mockRejectedValue(new Error("boom")) } as unknown as PythonExecutor;
+ const engine = await resolveTranscribeEngine(executor, "whisper-py", "base");
+ expect(engine).toBe("whisper-py");
+ });
+
+ it("falls back to whisper-py when both the request and the resolution are empty", async () => {
+ const executor = { execute: vi.fn().mockRejectedValue(new Error("boom")) } as unknown as PythonExecutor;
+ const engine = await resolveTranscribeEngine(executor, undefined, "base");
+ expect(engine).toBe("whisper-py");
+ });
+});
+
+describe("engineCanDiarize", () => {
+ it("is false for whisper.cpp and omnilingual, which never diarize", () => {
+ expect(engineCanDiarize("whispercpp")).toBe(false);
+ expect(engineCanDiarize("omnilingual")).toBe(false);
+ });
+
+ it("is true for whisper-py and assemblyai", () => {
+ expect(engineCanDiarize("whisper-py")).toBe(true);
+ expect(engineCanDiarize("assemblyai")).toBe(true);
+ });
+
+ it("is true (the safe default) for an unresolved engine", () => {
+ expect(engineCanDiarize(undefined)).toBe(true);
+ });
+});
diff --git a/src/services/engine-resolve.ts b/src/services/engine-resolve.ts
new file mode 100644
index 0000000..57fad83
--- /dev/null
+++ b/src/services/engine-resolve.ts
@@ -0,0 +1,42 @@
+import { PythonExecutor } from "./python-executor.js";
+
+/**
+ * Predicts which engine backend/services/transcription.py's transcribe_file
+ * will actually use for an unset/auto request, without transcribing anything.
+ *
+ * The transcript cache is keyed by engine. Reading the cache with the raw
+ * request (e.g. undefined, meaning "whisper-py unless this install can't run
+ * it") instead of what transcribe_file resolves to always misses on a native
+ * install, because the write afterward lands under "whispercpp", the key
+ * the read never looked at.
+ */
+/**
+ * Whether the resolved engine can ever produce speaker labels. whisper.cpp
+ * and omnilingual skip diarization unconditionally (see
+ * backend/services/transcription.py's whispercpp/omnilingual branches). A
+ * cache entry from either engine will never gain speaker labels no matter
+ * how many times it's re-transcribed, so treating a missing-labels cache as
+ * a retriable miss for them would re-transcribe on every single call.
+ */
+export function engineCanDiarize(engine: string | undefined): boolean {
+ return engine !== "whispercpp" && engine !== "omnilingual";
+}
+
+export async function resolveTranscribeEngine(
+ executor: PythonExecutor,
+ engine: string | undefined,
+ modelSize: string
+): Promise<string> {
+ try {
+ const result = await executor.execute<{ engine: string }>("resolve_transcribe_engine", {
+ engine,
+ model_size: modelSize,
+ });
+ return result.data?.engine ?? engine ?? "whisper-py";
+ } catch {
+ // Resolution is a best-effort cache-key optimization; a failure here
+ // should fall through to a normal (possibly-missed) cache lookup rather
+ // than block transcription.
+ return engine ?? "whisper-py";
+ }
+}
diff --git a/src/services/episode-state.test.ts b/src/services/episode-state.test.ts
new file mode 100644
index 0000000..fdaac2e
--- /dev/null
+++ b/src/services/episode-state.test.ts
@@ -0,0 +1,115 @@
+import { describe, it, expect, beforeAll, afterAll, afterEach } from "vitest";
+import { existsSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-episode-state-"));
+const savedHome = process.env.PODCLI_HOME;
+
+let EpisodeState: typeof import("./episode-state.js").EpisodeState;
+let videoA: string;
+let videoB: string;
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ ({ EpisodeState } = await import("./episode-state.js"));
+ videoA = join(tmp, "episode-a.mp4");
+ videoB = join(tmp, "episode-b.mp4");
+ writeFileSync(videoA, "a".repeat(100));
+ writeFileSync(videoB, "b".repeat(200));
+});
+
+afterAll(() => {
+ if (savedHome === undefined) delete process.env.PODCLI_HOME;
+ else process.env.PODCLI_HOME = savedHome;
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+afterEach(() => {
+ rmSync(join(tmp, "home", "episode-decisions.json"), { force: true });
+});
+
+describe("EpisodeState", () => {
+ it("has no decisions for a video that was never recorded", async () => {
+ const state = new EpisodeState();
+ expect(await state.get(videoA)).toBeNull();
+ });
+
+ it("records and reads back decisions for one video", async () => {
+ const state = new EpisodeState();
+ await state.record(videoA, { clipCount: 5, captionStyle: "hormozi" });
+ const got = await state.get(videoA);
+ expect(got?.clipCount).toBe(5);
+ expect(got?.captionStyle).toBe("hormozi");
+ });
+
+ it("merges new decisions without clearing previously recorded ones", async () => {
+ const state = new EpisodeState();
+ await state.record(videoA, { clipCount: 5 });
+ await state.record(videoA, { captionsEnabled: true });
+ const got = await state.get(videoA);
+ expect(got?.clipCount).toBe(5);
+ expect(got?.captionsEnabled).toBe(true);
+ });
+
+ it("keeps two videos' decisions independent, keyed by path and size", async () => {
+ const state = new EpisodeState();
+ await state.record(videoA, { clipCount: 5 });
+ await state.record(videoB, { clipCount: 9 });
+ expect((await state.get(videoA))?.clipCount).toBe(5);
+ expect((await state.get(videoB))?.clipCount).toBe(9);
+ });
+
+ it("lists every decision as open before anything is recorded", async () => {
+ const state = new EpisodeState();
+ const open = await state.openQuestions(videoA);
+ expect(open.length).toBeGreaterThan(0);
+ expect(open.map((q) => q.field)).toContain("clipCount");
+ });
+
+ it("drops a field from open questions once it's answered", async () => {
+ const state = new EpisodeState();
+ await state.record(videoA, { clipCount: 5 });
+ const open = await state.openQuestions(videoA);
+ expect(open.map((q) => q.field)).not.toContain("clipCount");
+ expect(open.map((q) => q.field)).toContain("captionStyle");
+ });
+
+ it("treats a video that doesn't exist on disk as having no key", async () => {
+ const state = new EpisodeState();
+ expect(await state.keyFor(join(tmp, "nope.mp4"))).toBeNull();
+ expect(await state.get(join(tmp, "nope.mp4"))).toBeNull();
+ });
+
+ it("loses no update when two record() calls race (serialized read-modify-write)", async () => {
+ const state = new EpisodeState();
+ // Both start from the same on-disk snapshot if record() doesn't serialize
+ // its read-modify-write; whichever write lands second would otherwise
+ // clobber the first field instead of merging with it.
+ await Promise.all([
+ state.record(videoA, { clipCount: 5 }),
+ state.record(videoA, { captionsEnabled: true }),
+ ]);
+ const got = await state.get(videoA);
+ expect(got?.clipCount).toBe(5);
+ expect(got?.captionsEnabled).toBe(true);
+ });
+
+ it("quarantines a corrupt decisions file instead of wiping it", async () => {
+ const state = new EpisodeState();
+ await state.record(videoA, { clipCount: 5 });
+ const decisionsPath = join(tmp, "home", "episode-decisions.json");
+ writeFileSync(decisionsPath, "{ not valid json");
+
+ const got = await state.get(videoA);
+ expect(got).toBeNull(); // corrupt file reads back as empty, not an error
+
+ const quarantined = readdirSync(join(tmp, "home")).filter((f) => f.includes(".corrupt-"));
+ expect(quarantined.length).toBe(1);
+ expect(existsSync(decisionsPath)).toBe(false); // moved aside, not left in place
+
+ // Recording after the quarantine starts a clean file rather than refusing.
+ await state.record(videoA, { clipCount: 9 });
+ expect((await state.get(videoA))?.clipCount).toBe(9);
+ });
+});
diff --git a/src/services/episode-state.ts b/src/services/episode-state.ts
new file mode 100644
index 0000000..f6da3b1
--- /dev/null
+++ b/src/services/episode-state.ts
@@ -0,0 +1,139 @@
+import { createHash } from "crypto";
+import { readFile, rename, mkdir } from "fs/promises";
+import { existsSync } from "fs";
+import { dirname, resolve } from "path";
+import { paths } from "../config/paths.js";
+import { writeFileAtomic } from "../utils/atomic-file.js";
+import { computeVideoIdentity, type VideoIdentity } from "../utils/video-identity.js";
+import { childLogger } from "../utils/logger.js";
+import type { EpisodeDecisions, OpenQuestion } from "../models/index.js";
+
+const log = childLogger("episode-state");
+
+type DecisionsFile = Record<string, EpisodeDecisions>;
+
+// One fixed question per decision field, in the order a producer would
+// actually need the answers. get_ui_state surfaces these as "open
+// questions" so the agent asks each one once per episode, not once per run.
+const QUESTIONS: Array<{ field: keyof EpisodeDecisions; question: string }> = [
+ { field: "clipCount", question: "How many clips should this episode produce?" },
+ { field: "clipDurationRange", question: "What's the target clip duration range (seconds)?" },
+ { field: "captionsEnabled", question: "Should clips have captions burned in?" },
+ { field: "captionStyle", question: "Which caption style: hormozi, karaoke, subtle, or branded?" },
+ { field: "language", question: "What language is this episode in?" },
+ { field: "thumbnailsWanted", question: "Do you want thumbnails generated for these clips?" },
+ { field: "deliveryTarget", question: "Where are these clips headed: YouTube Shorts, TikTok, Instagram, or just export?" },
+];
+
+/**
+ * Per-episode decisions, keyed by the source video's identity (path + size +
+ * mtime, the same VideoIdentity transcriptVideoMismatch checks against)
+ * rather than a content hash: cheap to compute, and good enough to tell
+ * "the same file" from "a different recording" without reading the file.
+ *
+ * Callers resolve to the original video's path before calling in when the
+ * current working video is a silence-removed derivative (a new path, size
+ * and mtime), so decisions follow the episode across that rewrite instead of
+ * keying on a file that only exists for one session.
+ *
+ * Stored independently of ui-state.json (which is global and gets overwritten
+ * by the next episode opened) so these answers outlive the session that
+ * recorded them.
+ */
+export class EpisodeState {
+ private filePath: string;
+
+ // Chains every read-modify-write onto the previous one so concurrent
+ // record() calls (e.g. two record_decisions calls firing close together)
+ // never both read the same snapshot and clobber each other's write.
+ private queue: Promise<unknown> = Promise.resolve();
+
+ constructor() {
+ this.filePath = paths.episodeDecisions;
+ }
+
+ private keyForIdentity(identity: VideoIdentity): string {
+ // Same identity helper transcriptVideoMismatch uses, so "the same video"
+ // means the same thing everywhere instead of two notions quietly drifting.
+ return createHash("sha256")
+ .update(`${identity.path}:${identity.size}:${identity.mtimeMs}`)
+ .digest("hex")
+ .slice(0, 16);
+ }
+
+ async keyFor(videoPath: string): Promise<string | null> {
+ const identity = computeVideoIdentity(resolve(videoPath));
+ if (!identity) return null; // video not found on disk, nothing to key against
+ return this.keyForIdentity(identity);
+ }
+
+ private async readAll(): Promise<DecisionsFile> {
+ if (!existsSync(this.filePath)) return {};
+ try {
+ const raw = await readFile(this.filePath, "utf-8");
+ return JSON.parse(raw) as DecisionsFile;
+ } catch (err) {
+ // A corrupt file must never silently wipe every episode's decisions.
+ // Move it aside so the data isn't lost and start fresh; the next write
+ // produces a clean file without touching the renamed original.
+ const quarantined = `${this.filePath}.corrupt-${Date.now()}`;
+ await rename(this.filePath, quarantined).catch(() => {});
+ log.warn("episode-decisions.json was corrupt; quarantined and starting fresh", {
+ err: err instanceof Error ? err.message : String(err),
+ quarantined,
+ });
+ return {};
+ }
+ }
+
+ private async writeAll(data: DecisionsFile): Promise<void> {
+ await mkdir(dirname(this.filePath), { recursive: true });
+ await writeFileAtomic(this.filePath, JSON.stringify(data, null, 2));
+ }
+
+ /** Serializes a read-modify-write step behind whatever is already queued. */
+ private enqueue<T>(step: () => Promise<T>): Promise<T> {
+ const result = this.queue.then(step, step);
+ this.queue = result.catch(() => {});
+ return result;
+ }
+
+ async get(videoPath: string): Promise<EpisodeDecisions | null> {
+ const key = await this.keyFor(videoPath);
+ if (!key) return null;
+ const all = await this.readAll();
+ return all[key] ?? null;
+ }
+
+ /** Merge in whichever fields are given; fields already answered are never cleared implicitly. */
+ async record(
+ videoPath: string,
+ decisions: Partial<Omit<EpisodeDecisions, "videoPath" | "fileSize" | "updatedAt">>,
+ ): Promise<EpisodeDecisions> {
+ const abs = resolve(videoPath);
+ return this.enqueue(async () => {
+ const identity = computeVideoIdentity(abs);
+ if (!identity) throw new Error(`Video not found: ${abs}`); // nothing to key against
+ const key = this.keyForIdentity(identity);
+
+ const all = await this.readAll();
+ const existing = all[key];
+ const merged: EpisodeDecisions = {
+ ...existing,
+ ...decisions,
+ videoPath: abs,
+ fileSize: identity.size,
+ updatedAt: Date.now(),
+ };
+ all[key] = merged;
+ await this.writeAll(all);
+ return merged;
+ });
+ }
+
+ /** Decisions relevant to the next step that have no answer recorded yet. */
+ async openQuestions(videoPath: string): Promise<OpenQuestion[]> {
+ const current = await this.get(videoPath);
+ return QUESTIONS.filter(({ field }) => current?.[field] === undefined);
+ }
+}
diff --git a/src/services/transcript-cache.test.ts b/src/services/transcript-cache.test.ts
index 990983b..89ec356 100644
--- a/src/services/transcript-cache.test.ts
+++ b/src/services/transcript-cache.test.ts
@@ -8,7 +8,9 @@ const tmp = mkdtempSync(join(tmpdir(), "podcli-cache-test-"));
process.env.PODCLI_HOME = tmp;
process.env.PODCLI_DATA = tmp;
-const { TranscriptCache, hasSpeakerLabels } = await import("./transcript-cache.js");
+const { TranscriptCache, hasSpeakerLabels, needsDiarizationRetry } = await import(
+ "./transcript-cache.js"
+);
function makeFakeVideo(name: string, content: string): string {
const p = join(tmp, name);
@@ -78,12 +80,111 @@ describe("TranscriptCache", () => {
expect(await cache.getPackedMarkdown(file)).toBeNull();
});
+ it("keys the packed view by model and language too, not engine alone", async () => {
+ const file = makeFakeVideo("packed-multi-key.mp4", "packed multi key");
+ mkdirSync(join(tmp, "packed"), { recursive: true });
+ const base = await cache.getFileHashForEngine(file, { engine: "whispercpp" });
+ const small = await cache.getFileHashForEngine(file, {
+ engine: "whispercpp",
+ model: "small",
+ language: "ka",
+ });
+ writeFileSync(join(tmp, "packed", `${base}.md`), "# base");
+ writeFileSync(join(tmp, "packed", `${small}.md`), "# small-ka");
+ expect(await cache.getPackedMarkdown(file, { engine: "whispercpp" })).toBe("# base");
+ expect(
+ await cache.getPackedMarkdown(file, { engine: "whispercpp", model: "small", language: "ka" }),
+ ).toBe("# small-ka");
+ // A combo that was never written must miss, not fall back to either.
+ expect(
+ await cache.getPackedMarkdown(file, { engine: "whispercpp", model: "medium", language: "fr" }),
+ ).toBeNull();
+ });
+
+ it("falls back to the most recently modified packed view when the caller doesn't know model/language", async () => {
+ const file = makeFakeVideo("packed-unknown-key.mp4", "packed unknown key");
+ mkdirSync(join(tmp, "packed"), { recursive: true });
+ const first = await cache.getFileHashForEngine(file, { engine: "whispercpp", model: "small" });
+ writeFileSync(join(tmp, "packed", `${first}.md`), "# first");
+ await new Promise((r) => setTimeout(r, 5));
+ const second = await cache.getFileHashForEngine(file, {
+ engine: "whispercpp",
+ model: "medium",
+ language: "ka",
+ });
+ writeFileSync(join(tmp, "packed", `${second}.md`), "# second");
+ // Caller only knows the engine, not which model/language actually ran.
+ // Deterministic rule: most recently written entry for this hash+engine.
+ expect(await cache.getPackedMarkdown(file, { engine: "whispercpp" })).toBe("# second");
+ });
+
it("get returns null when the cache file is corrupt", async () => {
const file = makeFakeVideo("corrupt-source.mp4", "any content");
const hash = await cache.getFileHash(file);
writeFileSync(join(tmp, "cache", "transcripts", `${hash}.json`), "this is not json");
expect(await cache.get(file)).toBeNull();
});
+
+ it("keys the raw cache by engine, model and language, not engine alone", async () => {
+ const file = makeFakeVideo("multi-key.mp4", "same media, different requests");
+ await cache.set(file, { ...fakeTranscript, transcript: "base-auto" }, {
+ engine: "whispercpp",
+ });
+ await cache.set(file, { ...fakeTranscript, transcript: "small-ka" }, {
+ engine: "whispercpp",
+ model: "small",
+ language: "ka",
+ });
+ expect((await cache.get(file, { engine: "whispercpp" }))?.transcript).toBe("base-auto");
+ expect(
+ (await cache.get(file, { engine: "whispercpp", model: "small", language: "ka" }))
+ ?.transcript,
+ ).toBe("small-ka");
+ // A request for a third, never-written combo must miss, not fall back to
+ // either of the above.
+ expect(
+ await cache.get(file, { engine: "whispercpp", model: "medium", language: "fr" }),
+ ).toBeNull();
+ });
+
+ it("treats base model and auto language as no suffix, for backward compatibility", async () => {
+ const file = makeFakeVideo("default-key.mp4", "legacy cache shape");
+ // A plain string engine (the pre-existing call shape) must land on the
+ // same key as the equivalent object form with default model/language.
+ await cache.set(file, { ...fakeTranscript, transcript: "legacy" }, "whispercpp");
+ expect(
+ (await cache.get(file, { engine: "whispercpp", model: "base", language: "auto" }))
+ ?.transcript,
+ ).toBe("legacy");
+ });
+
+ it("sanitizes a language tag to [a-z0-9-], rejecting path separators and odd casing", async () => {
+ const file = makeFakeVideo("lang-sanitize.mp4", "language sanitize check");
+ await cache.set(file, { ...fakeTranscript, transcript: "danger" }, {
+ language: "../../etc",
+ });
+ // The sanitized form keys the lookup: "../../etc" strips to "etc".
+ expect(
+ (await cache.get(file, { language: "etc" }))?.transcript,
+ ).toBe("danger");
+ const { readdirSync } = await import("fs");
+ const files = readdirSync(join(tmp, "cache", "transcripts"));
+ expect(files.every((f) => !f.includes("..") && !f.includes("/"))).toBe(true);
+ });
+
+ it("treats mixed-case language tags the same as their lowercase form", async () => {
+ const file = makeFakeVideo("lang-case.mp4", "case check");
+ await cache.set(file, { ...fakeTranscript, transcript: "georgian" }, { language: "KA" });
+ expect((await cache.get(file, { language: "ka" }))?.transcript).toBe("georgian");
+ });
+
+ it("writes atomically: no temp file left behind, and no partial reads", async () => {
+ const file = makeFakeVideo("atomic.mp4", "atomic write check");
+ await cache.set(file, fakeTranscript);
+ const { readdirSync } = await import("fs");
+ const files = readdirSync(join(tmp, "cache", "transcripts"));
+ expect(files.some((f) => f.endsWith(".tmp"))).toBe(false);
+ });
});
@@ -119,3 +220,30 @@ describe("hasSpeakerLabels", () => {
expect(hasSpeakerLabels({})).toBe(false);
});
});
+
+describe("needsDiarizationRetry", () => {
+ const unlabelled = { diarization_attempted: false, words: [] };
+ const attemptedButNoneFound = { diarization_attempted: true, words: [] };
+
+ it("is false when diarization wasn't requested", () => {
+ expect(needsDiarizationRetry(unlabelled, false, true)).toBe(false);
+ });
+
+ it("is false when the resolved engine can never diarize, no matter the flag", () => {
+ // whisper.cpp/omnilingual will never gain labels, so retrying forever
+ // would be the bug this flag exists to prevent.
+ expect(needsDiarizationRetry(unlabelled, true, false)).toBe(false);
+ });
+
+ it("is true when requested, possible, and never attempted", () => {
+ expect(needsDiarizationRetry(unlabelled, true, true)).toBe(true);
+ });
+
+ it("is false once diarization was attempted, even with no labels to show for it", () => {
+ expect(needsDiarizationRetry(attemptedButNoneFound, true, true)).toBe(false);
+ });
+
+ it("is false for missing cache input", () => {
+ expect(needsDiarizationRetry(null, true, true)).toBe(false);
+ });
+});
diff --git a/src/services/transcript-cache.ts b/src/services/transcript-cache.ts
index c963798..ccda03a 100644
--- a/src/services/transcript-cache.ts
+++ b/src/services/transcript-cache.ts
@@ -1,10 +1,19 @@
import { createHash } from "crypto";
-import { readFile, writeFile, mkdir } from "fs/promises";
+import { readFile, writeFile, mkdir, rename } from "fs/promises";
import { existsSync } from "fs";
import { join } from "path";
import { paths } from "../config/paths.js";
import type { TranscriptResult } from "../models/index.js";
+/** Cache key parts beyond the file hash. A bare string is shorthand for
+ * { engine: string }, the call shape every existing caller already used. */
+export interface CacheKeyParts {
+ engine?: string;
+ model?: string;
+ language?: string;
+}
+export type CacheKey = string | CacheKeyParts;
+
/**
* Caches transcripts by file hash so we don't re-transcribe
* the same podcast when creating multiple clips.
@@ -31,6 +40,26 @@ export function hasSpeakerLabels(transcript: unknown): boolean {
return labelled(t.speaker_segments) || labelled(t.words);
}
+/**
+ * Whether a cached transcript is stale with respect to a diarization
+ * request: enable_diarization is true, the resolved engine can diarize at
+ * all (see engineCanDiarize), and this cache entry never actually attempted
+ * it. Re-transcribing a whisper.cpp/omnilingual cache for missing labels
+ * would never succeed, they never diarize, so gate on diarization_attempted
+ * (backend/services/transcription.py sets it on every branch) rather than
+ * hasSpeakerLabels alone, which can't tell "never tried" from "tried and
+ * genuinely found one speaker".
+ */
+export function needsDiarizationRetry(
+ cached: unknown,
+ enableDiarization: boolean,
+ engineCanDiarize: boolean,
+): boolean {
+ if (!cached || !enableDiarization || !engineCanDiarize) return false;
+ const attempted = (cached as { diarization_attempted?: unknown }).diarization_attempted;
+ return attempted !== true;
+}
+
export class TranscriptCache {
private cacheDir: string;
@@ -86,8 +115,15 @@ export class TranscriptCache {
});
}
- async getFileHashForEngine(filePath: string, engine?: string): Promise<string> {
- return `${await this.getFileHash(filePath)}${this.engineSuffix(engine)}`;
+ /**
+ * File hash plus the full {engine, model, language} cache key suffix.
+ * The packed markdown view is now keyed the same way the raw JSON cache
+ * is (see keySuffix), so a caller that knows the exact combo it wants
+ * reads the same filename this class' own set()/write_packed wrote it
+ * under.
+ */
+ async getFileHashForEngine(filePath: string, key?: CacheKey): Promise<string> {
+ return `${await this.getFileHash(filePath)}${this.keySuffix(key)}`;
}
private engineSuffix(engine?: string): string {
@@ -98,13 +134,48 @@ export class TranscriptCache {
if (["assemblyai", "assembly-ai", "aai"].includes(value)) {
return "-assemblyai";
}
+ if (["omnilingual", "omni", "omnilingual-asr"].includes(value)) {
+ return "-omnilingual";
+ }
return "";
}
- async get(filePath: string, engine?: string): Promise<TranscriptResult | null> {
+ /**
+ * Normalize a language tag for use in a cache filename: lowercase, and
+ * strip anything outside [a-z0-9-]. The tag comes from caller-supplied
+ * input (an MCP tool argument), so without this a value like "en/../x" or
+ * one carrying path separators would land in the cache path unescaped.
+ * Matches backend/services/transcript_packer.sanitize_language exactly, so
+ * the two sides land on the same filename for the same language.
+ */
+ private sanitizeLanguage(language?: string): string {
+ return (language ?? "").trim().toLowerCase().replace(/[^a-z0-9-]/g, "");
+ }
+
+ /**
+ * Full cache key suffix: engine + model + language. A plain string argument
+ * (the pre-existing call shape) is treated as engine-only.
+ *
+ * base model and auto/empty language contribute no suffix, so a cache
+ * written before model/language were tracked (always base, whisper-py, or
+ * whatever engine was passed, auto-detected language) still reads back
+ * under the same key. Any other model or language gets its own key instead
+ * of silently colliding with that implicit default.
+ */
+ private keySuffix(key?: CacheKey): string {
+ const parts = typeof key === "string" ? { engine: key } : key ?? {};
+ const engineSuffix = this.engineSuffix(parts.engine);
+ const model = (parts.model ?? "").trim().toLowerCase();
+ const modelSuffix = model && model !== "base" ? `-m${model}` : "";
+ const language = this.sanitizeLanguage(parts.language);
+ const languageSuffix = language && language !== "auto" ? `-l${language}` : "";
+ return `${engineSuffix}${modelSuffix}${languageSuffix}`;
+ }
+
+ async get(filePath: string, key?: CacheKey): Promise<TranscriptResult | null> {
try {
const hash = await this.getFileHash(filePath);
- const cachePath = join(this.cacheDir, `${hash}${this.engineSuffix(engine)}.json`);
+ const cachePath = join(this.cacheDir, `${hash}${this.keySuffix(key)}.json`);
if (!existsSync(cachePath)) return null;
@@ -115,11 +186,15 @@ export class TranscriptCache {
}
}
- async set(filePath: string, transcript: TranscriptResult, engine?: string): Promise<void> {
+ async set(filePath: string, transcript: TranscriptResult, key?: CacheKey): Promise<void> {
await this.ensureDir();
const hash = await this.getFileHash(filePath);
- const cachePath = join(this.cacheDir, `${hash}${this.engineSuffix(engine)}.json`);
- await writeFile(cachePath, JSON.stringify(transcript), "utf-8");
+ const cachePath = join(this.cacheDir, `${hash}${this.keySuffix(key)}.json`);
+ // Write-then-rename: a reader never observes a half-written cache file,
+ // and a crash mid-write leaves only an orphaned .tmp, not a corrupt entry.
+ const tmpPath = `${cachePath}.${process.pid}.${Date.now()}.tmp`;
+ await writeFile(tmpPath, JSON.stringify(transcript), "utf-8");
+ await rename(tmpPath, cachePath);
}
/**
@@ -127,15 +202,47 @@ export class TranscriptCache {
* written by backend/services/transcript_packer.py as a side-effect of
* transcription. Returns null if not yet generated.
*/
- async getPackedMarkdown(filePath: string, engine?: string): Promise<string | null> {
+ async getPackedMarkdown(filePath: string, key?: CacheKey): Promise<string | null> {
try {
- const hash = await this.getFileHashForEngine(filePath, engine);
- return await this.readPackedByHash(hash);
+ const parts = typeof key === "string" ? { engine: key } : key ?? {};
+ if (parts.model !== undefined || parts.language !== undefined) {
+ // The caller knows exactly which combo it wants: read that key
+ // only, never silently substitute a different model/language's view.
+ const hash = await this.getFileHashForEngine(filePath, parts);
+ return await this.readPackedByHash(hash);
+ }
+ // The caller only knows the engine (e.g. get_ui_state, which has a
+ // cached transcript object but not the model/language that produced
+ // it), fall back to the same deterministic scan the Python side uses
+ // in find_cached_transcript_path.
+ const hash = await this.getFileHash(filePath);
+ const found = await this.findPackedPath(hash, parts.engine);
+ return found ? await readFile(found, "utf-8") : null;
} catch {
return null;
}
}
+ /**
+ * Port of backend/services/transcript_packer.find_cached_transcript_path,
+ * applied to the packed .md directory: prefer the engine-only key, else
+ * the most recently modified matching file for this hash+engine.
+ */
+ private async findPackedPath(hash: string, engine?: string): Promise<string | null> {
+ const engineSuffix = this.engineSuffix(engine);
+ const exact = join(paths.packed, `${hash}${engineSuffix}.md`);
+ if (existsSync(exact)) return exact;
+ if (!existsSync(paths.packed)) return null;
+ const { readdirSync, statSync } = await import("fs");
+ const prefix = `${hash}${engineSuffix}`;
+ const tailRe = /^(-m[a-z0-9]+)?(-l[a-z0-9-]+)?\.md$/;
+ const candidates = readdirSync(paths.packed)
+ .filter((name) => name.startsWith(prefix) && tailRe.test(name.slice(prefix.length)))
+ .map((name) => join(paths.packed, name));
+ if (!candidates.length) return null;
+ return candidates.sort((a, b) => statSync(b).mtimeMs - statSync(a).mtimeMs)[0];
+ }
+
/**
* Look up the packed view for a pasted transcript (no source file).
* Mirrors backend/services handle_parse_transcript's content-hash keying:
diff --git a/src/thumbnail-config-tool.test.ts b/src/thumbnail-config-tool.test.ts
new file mode 100644
index 0000000..499fd22
--- /dev/null
+++ b/src/thumbnail-config-tool.test.ts
@@ -0,0 +1,62 @@
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import { mkdtempSync, rmSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// manage_thumbnail_config talks to the studio's /api/thumbnail-config. These
+// stub that endpoint and check what set_layout sends it.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-thumbnail-config-"));
+const savedHome = process.env.PODCLI_HOME;
+const savedData = process.env.PODCLI_DATA;
+
+let createServer: typeof import("./server.js").createServer;
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ ({ createServer } = await import("./server.js"));
+});
+
+afterAll(() => {
+ if (savedHome === undefined) delete process.env.PODCLI_HOME;
+ else process.env.PODCLI_HOME = savedHome;
+ if (savedData === undefined) delete process.env.PODCLI_DATA;
+ else process.env.PODCLI_DATA = savedData;
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+afterEach(() => {
+ vi.unstubAllGlobals();
+});
+
+function handler() {
+ const server = createServer() as unknown as {
+ _registeredTools: Record<string, { handler: (args: unknown, extra: unknown) => Promise<{ content: { text: string }[] }> }>;
+ };
+ return server._registeredTools.manage_thumbnail_config.handler;
+}
+
+describe("manage_thumbnail_config set_layout", () => {
+ it("writes the layout into the template and keeps every other field", async () => {
+ const calls: { method: string; body?: string }[] = [];
+ vi.stubGlobal("fetch", vi.fn(async (_url: string, init?: RequestInit) => {
+ calls.push({ method: init?.method ?? "GET", body: init?.body as string | undefined });
+ return new Response(JSON.stringify({ accent_color: "#FF3366", layout: "single" }), { status: 200 });
+ }));
+
+ const result = await handler()({ action: "set_layout", layout: "pair" }, {});
+
+ expect(result.content[0].text).toBe("Thumbnail layout set to pair.");
+ const put = calls.find((c) => c.method === "PUT");
+ expect(JSON.parse(put!.body!)).toEqual({ accent_color: "#FF3366", layout: "pair" });
+ });
+
+ it("asks for the layout when it is missing", async () => {
+ const fetch = vi.fn();
+ vi.stubGlobal("fetch", fetch);
+ const result = await handler()({ action: "set_layout" }, {});
+ expect(result.content[0].text).toContain("'layout' is required");
+ expect(fetch).not.toHaveBeenCalled();
+ });
+});
diff --git a/src/ui-state-offline.test.ts b/src/ui-state-offline.test.ts
new file mode 100644
index 0000000..5ee62fb
--- /dev/null
+++ b/src/ui-state-offline.test.ts
@@ -0,0 +1,86 @@
+import { describe, it, expect, beforeAll, afterAll, afterEach } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// get_ui_state used to report "Web UI is not running" whenever the Web UI
+// process was down, even though the same state it would have served lives on
+// disk at paths.uiState. These tests pin the fallback: offline reads should
+// come from the state file, and the "no session yet" message should only
+// appear when there is truly nothing to read.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-ui-state-offline-"));
+const savedHome = process.env.PODCLI_HOME;
+const savedData = process.env.PODCLI_DATA;
+const savedFetch = globalThis.fetch;
+
+let createServer: typeof import("./server.js").createServer;
+let uiStatePath: string;
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ const { paths } = await import("./config/paths.js");
+ uiStatePath = paths.uiState;
+ mkdirSync(paths.home, { recursive: true });
+ ({ createServer } = await import("./server.js"));
+});
+
+afterAll(() => {
+ if (savedHome === undefined) delete process.env.PODCLI_HOME;
+ else process.env.PODCLI_HOME = savedHome;
+ if (savedData === undefined) delete process.env.PODCLI_DATA;
+ else process.env.PODCLI_DATA = savedData;
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+afterEach(() => {
+ globalThis.fetch = savedFetch;
+ rmSync(uiStatePath, { force: true });
+});
+
+function getUiStateHandler() {
+ const server = createServer() as unknown as {
+ _registeredTools: Record<string, { handler: (args: unknown, extra: unknown) => Promise<{ content: { text: string }[] }> }>;
+ };
+ return server._registeredTools["get_ui_state"].handler;
+}
+
+function stubWebUiDown() {
+ globalThis.fetch = (() => Promise.reject(new Error("fetch failed"))) as typeof fetch;
+}
+
+describe("get_ui_state offline fallback", () => {
+ it("reads session state from disk when the Web UI is not running", async () => {
+ writeFileSync(
+ uiStatePath,
+ JSON.stringify({
+ videoPath: "/videos/ep42.mp4",
+ phase: "suggesting",
+ settings: { captionStyle: "hormozi", cropStrategy: "speaker" },
+ suggestions: [],
+ deselectedIndices: [],
+ transcript: { words: ["a", "b", "c"] },
+ }),
+ );
+ stubWebUiDown();
+
+ const handler = getUiStateHandler();
+ const result = await handler({ include_transcript: false }, {});
+ const text = result.content[0].text;
+
+ expect(text).toContain("/videos/ep42.mp4");
+ expect(text).toContain("Transcript: 3 words");
+ expect(text).not.toContain("Web UI is not running");
+ });
+
+ it("falls back to start-from-scratch guidance when there is no state anywhere", async () => {
+ stubWebUiDown();
+
+ const handler = getUiStateHandler();
+ const result = await handler({ include_transcript: false }, {});
+ const text = result.content[0].text;
+
+ expect(text).toContain("Start from scratch");
+ });
+});
diff --git a/src/ui/client/ClipDetail.tsx b/src/ui/client/ClipDetail.tsx
index eb4d59c..39f9fda 100644
--- a/src/ui/client/ClipDetail.tsx
+++ b/src/ui/client/ClipDetail.tsx
@@ -1,7 +1,8 @@
import React, { useEffect, useRef, useState } from "react";
import { PageHeader } from "./Page";
import { Link, useParams, useNavigate } from "react-router-dom";
-import { api, upload, fmt, basename, labelStyle } from "./lib";
+import { api, upload, fmt, basename, labelStyle, describePeople } from "./lib";
+import { needsFrame, type ThumbnailLayout, type ThumbnailPerson } from "../../utils/thumbnail-layout";
import ClipPlayer from "./ClipPlayer";
import { BackIcon } from "./icons";
import ReframeEditor from "./ReframeEditor";
@@ -16,6 +17,8 @@ interface ThumbnailConfig {
timestamp?: number;
preview_path?: string;
variations?: string[];
+ layout?: ThumbnailLayout;
+ people?: ThumbnailPerson[];
}
interface Clip {
@@ -62,6 +65,8 @@ export default function ClipDetail() {
const [frameOpts, setFrameOpts] = useState<any[]>([]);
const [frameIdx, setFrameIdx] = useState(0);
const [selFrame, setSelFrame] = useState<{ path: string; info?: any } | null>(null);
+ const [thumbLayout, setThumbLayout] = useState<ThumbnailLayout>("single");
+ const [swapSides, setSwapSides] = useState(false);
const [busy, setBusy] = useState<string | null>(null);
const [msg, setMsg] = useState<string | null>(null);
const [msgErr, setMsgErr] = useState(false);
@@ -86,6 +91,7 @@ export default function ClipDetail() {
const tc = found.thumbnail_config || {};
setLine1(tc.line1 ?? "");
setLine2(tc.line2 ?? "");
+ setThumbLayout(tc.layout ?? "single");
}
})
.finally(() => setLoading(false));
@@ -139,17 +145,24 @@ export default function ClipDetail() {
} catch (e: any) { setMsg(`Upload failed: ${e.message}`); setMsgErr(true); } finally { setBusy(null); }
};
+ const pairLayout = thumbLayout === "pair";
+
const renderThumb = async () => {
- if (!selFrame) { setMsg("Select or upload a frame first"); setMsgErr(true); return; }
+ if (!selFrame && needsFrame(thumbLayout)) { setMsg("Select or upload a frame first"); setMsgErr(true); return; }
setBusy("render"); setMsg(null); setMsgErr(false);
try {
const r = await api(`/clips/${clip.id}/thumbnail/render`, {
method: "POST",
- body: JSON.stringify({ line1: line1 || undefined, line2: line2 || undefined, frame_path: selFrame.path, frame_info: selFrame.info }),
+ body: JSON.stringify({
+ line1: line1 || undefined, line2: line2 || undefined,
+ frame_path: selFrame?.path, frame_info: selFrame?.info,
+ layout: thumbLayout, swap: pairLayout && swapSides,
+ }),
});
if (r.error) throw new Error(r.error);
setBust(Date.now()); load();
- setMsg("Thumbnail generated"); setMsgErr(false);
+ if (r.note) { setMsg(r.note); setMsgErr(true); }
+ else { setMsg(r.people?.length ? `Thumbnail generated: ${describePeople(r.people)}` : "Thumbnail generated"); setMsgErr(false); }
} catch (e: any) { setMsg(`Generate failed: ${e.message}`); setMsgErr(true); } finally { setBusy(null); }
};
@@ -374,10 +387,19 @@ export default function ClipDetail() {
</div>
</div>
<div className="thumb-edit-controls">
+ <div style={{ display: "flex", gap: 6, flexWrap: "wrap", alignItems: "center", marginBottom: 8 }}>
+ <button className={`btn btn-ghost btn-sm ${!pairLayout ? "selected" : ""}`} onClick={() => setThumbLayout("single")} disabled={busy !== null}>One person</button>
+ <button className={`btn btn-ghost btn-sm ${pairLayout ? "selected" : ""}`} onClick={() => setThumbLayout("pair")} disabled={busy !== null}>Two people</button>
+ {pairLayout && (
+ <label className="meta" style={{ display: "flex", alignItems: "center", gap: 6 }}>
+ <input type="checkbox" checked={swapSides} onChange={(e) => setSwapSides(e.target.checked)} disabled={busy !== null} /> Swap sides
+ </label>
+ )}
+ </div>
<input type="text" value={line1} onChange={(e) => setLine1(e.target.value)} placeholder="Line 1" style={{ width: "100%" }} />
<input type="text" value={line2} onChange={(e) => setLine2(e.target.value)} placeholder="Line 2 (highlighted)" style={{ width: "100%", marginTop: 8 }} />
<div className="set-actions" style={{ marginTop: 10 }}>
- <button className="btn btn-primary btn-sm" onClick={renderThumb} disabled={busy !== null || !selFrame}>
+ <button className="btn btn-primary btn-sm" onClick={renderThumb} disabled={busy !== null || (!selFrame && needsFrame(thumbLayout))}>
{busy === "render" ? <div className="spinner sm" /> : (tc.preview_path ? "Regenerate" : "Generate")}
</button>
<button className="btn btn-ghost btn-sm" onClick={newFrame} disabled={busy !== null} title="Pull a different frame from the clip">
@@ -388,7 +410,13 @@ export default function ClipDetail() {
</button>
<input ref={fileRef} type="file" accept=".png,.jpg,.jpeg,.webp" style={{ display: "none" }} onChange={(e) => e.target.files?.[0] && uploadFrame(e.target.files[0])} />
</div>
- <div className="hint" style={{ marginTop: 8 }}>Leave line 1 and line 2 empty to auto-write the text.</div>
+ <div className="hint" style={{ marginTop: 8 }}>
+ Leave line 1 and line 2 empty to auto-write the text.
+ {pairLayout && " Two people puts the guest left and the host right, both from this clip. A selected frame is used if podcli cannot tell them apart."}
+ </div>
+ {tc.layout === "pair" && tc.people?.length ? (
+ <div className="hint" style={{ marginTop: 4 }}>Current thumbnail: {describePeople(tc.people)}</div>
+ ) : null}
</div>
</div>
diff --git a/src/ui/client/EpisodeWorkspace.jsx b/src/ui/client/EpisodeWorkspace.jsx
index 474f9b8..2cd9ec4 100644
--- a/src/ui/client/EpisodeWorkspace.jsx
+++ b/src/ui/client/EpisodeWorkspace.jsx
@@ -37,10 +37,11 @@ import AssetPicker from './AssetPicker';
import { assetSrc, useAssets } from './useAssets';
import RecentSources from './RecentSources';
import MomentTrim from './MomentTrim';
+import { playbackDuration, playbackRanges, validateHook } from '../../utils/clip-hook';
import { useDialog } from './useDialog';
import { PageHeader } from './Page';
import { buildPreviewChunks, activePreviewChunk, selectPreviewWords } from './captionChunks';
-import { findClipResult, resultBoundsKey, clipKey, buildEnergyMap, dropEnergy, clampClipIndex, resolveAssetName, formatTranscriptText } from './lib';
+import { findClipResult, resultBoundsKey, clipKey, buildEnergyMap, dropEnergy, clampClipIndex, resolveAssetName, formatTranscriptText, safeUpper, buildUiStateSyncPayload } from './lib';
// Mirrors backend/services/formats.py. A horizontal cutdown is minutes long,
// so a slider capped at 60s could not express one.
@@ -294,7 +295,7 @@ const onKeyActivate = (fn) => (e) => {
function PhoneCaptionBody({ chunk, activeWordInChunk, cfg, singleLine = false }) {
if (!chunk || !chunk.length) return null;
- const fmt = (w) => (cfg.uppercase ? w.toUpperCase() : w);
+ const fmt = (w) => (cfg.uppercase ? safeUpper(w) : w);
const renderWord = (w, i, isActive) => {
if (cfg.activePill && isActive) {
@@ -812,7 +813,7 @@ const onKeyActivate = (fn) => (e) => {
// Clip editing
const [editingClip, setEditingClip] = useState(null); // index
- const [editForm, setEditForm] = useState({ title: '', start: 0, end: 0, payoff: '', contextLine: '' });
+ const [editForm, setEditForm] = useState({ title: '', start: 0, end: 0, payoff: '', contextLine: '', hook: null });
const [editDuration, setEditDuration] = useState(0);
const editDialogRef = useDialog(editingClip !== null, () => setEditingClip(null));
const previewDialogRef = useDialog(!!previewFile, () => setPreviewFile(null));
@@ -931,16 +932,30 @@ const onKeyActivate = (fn) => (e) => {
end: clip.end_second,
payoff: clip.payoff || '',
contextLine: clip.context_line || '',
+ hook: clip.hook || null,
});
};
+ // Saving drops segments, but the server restores them while the range
+ // stays within half a second, so the hook is checked against the same.
+ const editedClip = editingClip !== null ? suggestions[editingClip] : null;
+ const editSegments = editedClip
+ && Math.abs(editedClip.start_second - editForm.start) < 0.5
+ && Math.abs(editedClip.end_second - editForm.end) < 0.5
+ ? editedClip.segments : undefined;
+ const editHookError = validateHook(editForm.hook, editForm.start, editForm.end, editSegments);
+ const setEditHook = (patch) => setEditForm(f => ({ ...f, hook: patch && { ...f.hook, ...patch } }));
+ const addEditHook = () => setEditForm(f => ({
+ ...f, hook: { start: f.start, end: Math.min(f.end, f.start + 3), mode: 'repeat' },
+ }));
+
const clampEditTime = (v) => {
const n = Number.isFinite(v) ? Math.max(0, v) : 0;
return editDuration ? Math.min(n, editDuration) : n;
};
const saveClipEdit = () => {
- if (editingClip === null || editForm.end <= editForm.start) return;
+ if (editingClip === null || editForm.end <= editForm.start || editHookError) return;
const edited = suggestions[editingClip];
const reTimed = edited && (edited.start_second !== editForm.start || edited.end_second !== editForm.end);
setSuggestions(prev => {
@@ -952,8 +967,9 @@ const onKeyActivate = (fn) => (e) => {
end_second: editForm.end,
payoff: editForm.payoff,
context_line: editForm.contextLine,
- duration: Math.round(editForm.end - editForm.start),
+ duration: Math.round(playbackDuration(editForm.start, editForm.end, editSegments, editForm.hook)),
segments: undefined,
+ hook: editForm.hook || undefined,
};
return next;
});
@@ -1002,7 +1018,7 @@ const onKeyActivate = (fn) => (e) => {
const fileData = await api('/select-file', { method: 'POST', body: JSON.stringify({ file_path: vp }) });
if (fileData.error) { setTranscribing(false); return; }
setFile(fileData);
- const engine = transcriptionEngine === 'assemblyai' ? 'assemblyai' : undefined;
+ const engine = transcriptionEngine === 'whisper' ? undefined : transcriptionEngine;
const data = await api('/transcribe', { method: 'POST', body: JSON.stringify({
file_path: vp,
model_size: whisperModel,
@@ -1063,6 +1079,9 @@ const onKeyActivate = (fn) => (e) => {
// Sync UI state to server on changes (fire-and-forget)
// Guard: don't sync until initial SSE state has been received to avoid overwriting persisted state with defaults
const prevSyncRef = useRef('');
+ // See buildUiStateSyncPayload in lib.ts: transcript rides in this same
+ // request whenever it changed, instead of a request of its own.
+ const prevTranscriptRef = useRef(null);
useEffect(() => {
if (!stateHydrated) return;
const syncable = {
@@ -1086,21 +1105,14 @@ const onKeyActivate = (fn) => (e) => {
hydrationTargetRef.current = null;
if (signature !== target) return;
}
- const state = { _source: 'ui', filePath: file?.file_path || '', ...syncable };
+ const transcriptChanged = transcript !== prevTranscriptRef.current;
+ const state = buildUiStateSyncPayload(syncable, file?.file_path || '', transcript, transcriptChanged);
const key = JSON.stringify(state);
if (key === prevSyncRef.current) return;
prevSyncRef.current = key;
+ if (transcriptChanged) prevTranscriptRef.current = transcript;
fetch('/api/ui-state', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: key }).catch(() => { });
- }, [stateHydrated, videoPath, file, silenceOriginal, silencePlan, suggestions, deselected, captionStyle, captionPosition, captionFontScale, logoPosition, cropStrategy, format, logoPath, outroPath, introPath, cleanFillers, silenceThreshold, silenceMinPause, silencePadding, phase, results, energyData]);
-
- // Sync transcript separately (large payload)
- const prevTranscriptRef = useRef(null);
- useEffect(() => {
- if (!stateHydrated) return;
- if (!transcript || transcript === prevTranscriptRef.current) return;
- prevTranscriptRef.current = transcript;
- fetch('/api/ui-state', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ _source: 'ui', transcript }) }).catch(() => { });
- }, [stateHydrated, transcript]);
+ }, [stateHydrated, videoPath, file, silenceOriginal, silencePlan, suggestions, deselected, captionStyle, captionPosition, captionFontScale, logoPosition, cropStrategy, format, logoPath, outroPath, introPath, cleanFillers, silenceThreshold, silenceMinPause, silencePadding, phase, results, energyData, transcript]);
// Sync raw transcript text so MCP can read it before pipeline runs
const prevRawRef = useRef('');
@@ -1440,6 +1452,7 @@ const onKeyActivate = (fn) => (e) => {
crop_strategy: cropStrategy,
format,
...(Array.isArray(c.segments) && c.segments.length > 0 && { keep_segments: c.segments }),
+ hook: c.hook || null,
});
const startExport = async () => {
@@ -1610,6 +1623,7 @@ const onKeyActivate = (fn) => (e) => {
title: c.title, caption_style: captionStyle, caption_position: captionPosition, caption_font_scale: captionFontScale, logo_position: logoPosition, crop_strategy: cropStrategy, format,
transcript_words: transcript?.words || [], logo_path: logoPath || undefined, outro_path: outroPath || undefined, intro_path: introPath || undefined, clean_fillers: cleanFillers || undefined,
...(Array.isArray(c.segments) && c.segments.length > 0 && { keep_segments: c.segments }),
+ hook: c.hook || null,
})
});
setRetryJobId(data.job_id);
@@ -1986,6 +2000,7 @@ const onKeyActivate = (fn) => (e) => {
<label className="field-label">Engine</label>
<select value={transcriptionEngine} onChange={e => { setTranscriptionEngine(e.target.value); setTranscript(null); setCachedTranscript(false); autoTranscribeRef.current = ''; }} disabled={isProcessing || transcribing}>
<option value="whisper">Whisper</option>
+ <option value="omnilingual">Omnilingual (1600 languages)</option>
<option value="assemblyai">AssemblyAI</option>
</select>
</div>
@@ -2000,6 +2015,11 @@ const onKeyActivate = (fn) => (e) => {
</div>
)}
</div>
+ {transcriptionEngine === 'omnilingual' && (
+ <p className="hint" style={{ marginBottom: 10 }}>
+ Runs on this machine. The first run downloads a 1 GB model. Text comes back lowercase with no punctuation.
+ </p>
+ )}
{transcriptionEngine === 'assemblyai' && (
<div style={{ marginBottom: 10 }}>
<label className="field-label">AssemblyAI API key</label>
@@ -2643,6 +2663,7 @@ const onKeyActivate = (fn) => (e) => {
)}
<div className="clip-meta">
{fmt(clip.start_second)} {'\u2192'} {fmt(clip.end_second)} {'\u00B7'} {clip.duration}s
+ {clip.hook && <> {'\u00B7'} opens with {fmt(clip.hook.start)}</>}
{energy && (
<span className={`energy-badge ${energy.level}`} title={`Energy: ${energy.score}/10`}>
{energy.level === 'high' ? <Activity size={10} /> : energy.level === 'medium' ? '~' : '○'} {energy.score.toFixed(1)}
@@ -2927,10 +2948,51 @@ const onKeyActivate = (fn) => (e) => {
{editForm.end <= editForm.start && <span style={{ color: 'var(--red)' }}> {'·'} end must be after start</span>}
</div>
</div>
+ <div className="edit-field">
+ <label>Opening hook</label>
+ {!editForm.hook ? (
+ <button className="btn btn-ghost btn-sm" onClick={addEditHook} disabled={editForm.end <= editForm.start}>
+ <Plus size={12} /> Add a spoken line to open with
+ </button>
+ ) : (
+ <>
+ <div className="time-row">
+ <div>
+ <input type="number" step="0.1" min={editForm.start} max={editForm.end} value={editForm.hook.start}
+ onChange={e => setEditHook({ start: clampEditTime(parseFloat(e.target.value)) })}
+ aria-label="Hook start in seconds" style={{ textAlign: 'center' }} />
+ <div className="hint-xs" style={{ textAlign: 'center', marginTop: 2 }}>{fmt(editForm.hook.start)}</div>
+ </div>
+ <div className="arrow"><ArrowRight size={14} /></div>
+ <div>
+ <input type="number" step="0.1" min={editForm.start} max={editForm.end} value={editForm.hook.end}
+ onChange={e => setEditHook({ end: clampEditTime(parseFloat(e.target.value)) })}
+ aria-label="Hook end in seconds" style={{ textAlign: 'center' }} />
+ <div className="hint-xs" style={{ textAlign: 'center', marginTop: 2 }}>{fmt(editForm.hook.end)}</div>
+ </div>
+ </div>
+ <div style={{ display: 'flex', gap: 8, marginTop: 8 }}>
+ <select value={editForm.hook.mode} onChange={e => setEditHook({ mode: e.target.value })}
+ aria-label="Hook mode" style={{ flex: 1 }}>
+ <option value="repeat">Repeat: plays first, then again in place</option>
+ <option value="move">Move: plays first, cut from its place</option>
+ </select>
+ <button className="btn btn-ghost btn-sm" onClick={() => setEditHook(null)}>Clear hook</button>
+ </div>
+ <div className="hint" style={{ marginTop: 6, textAlign: 'center' }}>
+ {editHookError
+ ? <span style={{ color: 'var(--red)' }}>{editHookError}</span>
+ : <>Plays {playbackRanges(editForm.start, editForm.end, editSegments, editForm.hook)
+ .map(r => `${fmt(r.start)}-${fmt(r.end)}`).join(', then ')}
+ {' \u00B7 '}{Math.round(playbackDuration(editForm.start, editForm.end, editSegments, editForm.hook))}s total</>}
+ </div>
+ </>
+ )}
+ </div>
<div className="clip-edit-actions">
<button className="btn btn-ghost btn-sm" onClick={deleteClipEdit} style={{ color: 'var(--red)', marginRight: 'auto' }}>Delete clip</button>
<button className="btn btn-ghost btn-sm" onClick={() => setEditingClip(null)}>Cancel</button>
- <button className="btn btn-primary btn-sm" onClick={saveClipEdit} disabled={!editForm.title.trim() || editForm.end <= editForm.start}>Save</button>
+ <button className="btn btn-primary btn-sm" onClick={saveClipEdit} disabled={!editForm.title.trim() || editForm.end <= editForm.start || !!editHookError}>Save</button>
</div>
</div>
</div>
diff --git a/src/ui/client/McpSetupPage.test.ts b/src/ui/client/McpSetupPage.test.ts
new file mode 100644
index 0000000..59857ad
--- /dev/null
+++ b/src/ui/client/McpSetupPage.test.ts
@@ -0,0 +1,21 @@
+import { describe, expect, it } from "vitest";
+import { shellQuote } from "./McpSetupPage";
+
+describe("shellQuote", () => {
+ it("quotes a macOS Application Support path so the space doesn't split it into two args", () => {
+ const path = "/Users/nika/Library/Application Support/podcli/mcp-server.mjs";
+ const quoted = shellQuote(path);
+ expect(quoted).toBe("'/Users/nika/Library/Application Support/podcli/mcp-server.mjs'");
+ // A naive split on whitespace (what an unquoted shell command would do)
+ // must not break the path in two.
+ expect(quoted.split(" ")).toHaveLength(2);
+ });
+
+ it("escapes an embedded single quote so the quoting itself can't be broken out of", () => {
+ expect(shellQuote("it's/a/path")).toBe(`'it'\\''s/a/path'`);
+ });
+
+ it("leaves a path with no special characters readable", () => {
+ expect(shellQuote("/opt/podcli/mcp-server.mjs")).toBe("'/opt/podcli/mcp-server.mjs'");
+ });
+});
diff --git a/src/ui/client/McpSetupPage.tsx b/src/ui/client/McpSetupPage.tsx
index b96999e..f8cefd3 100644
--- a/src/ui/client/McpSetupPage.tsx
+++ b/src/ui/client/McpSetupPage.tsx
@@ -5,6 +5,13 @@ import CopyButton from "./CopyButton";
type StatusKind = "warn" | "ok" | "err";
+// macOS installs live under "~/Library/Application Support/...", so the
+// server path reliably contains a space. Wrapped in single quotes (with any
+// embedded single quote escaped) it pastes safely into sh/bash/zsh.
+export function shellQuote(value: string): string {
+ return `'${value.replace(/'/g, `'\\''`)}'`;
+}
+
const STATUS_STYLE: Record<StatusKind, React.CSSProperties> = {
ok: { background: "var(--green-subtle)", color: "var(--green)", border: "1px solid var(--green-border)" },
err: { background: "var(--red-subtle)", color: "var(--red)", border: "1px solid var(--red-border)" },
@@ -17,6 +24,7 @@ export default function McpSetupPage() {
const [statusText, setStatusText] = useState("Checking…");
const desktopRef = useRef<HTMLPreElement>(null);
const codeRef = useRef<HTMLPreElement>(null);
+ const codexRef = useRef<HTMLPreElement>(null);
useEffect(() => {
api<any>("/integration-info")
@@ -78,6 +86,21 @@ export default function McpSetupPage() {
<pre ref={codeRef}>podcli mcp install</pre>
</div>
</div>
+
+ <div className="section">
+ <div className="section-label">Codex</div>
+ <p style={{ fontSize: 13, color: "var(--text2)", marginTop: -4 }}>
+ <code>podcli mcp install</code> registers with Codex too when the <code>codex</code> CLI is on PATH. To
+ register by hand instead:
+ </p>
+ <div className="code-block">
+ <div className="code-block-head">
+ <span>terminal</span>
+ <CopyButton className="btn btn-ghost btn-sm" style={{ padding: "3px 10px" }} getText={() => codexRef.current?.innerText ?? ""} />
+ </div>
+ <pre ref={codexRef}>{`codex mcp add podcli -- node ${shellQuote(serverPath)}`}</pre>
+ </div>
+ </div>
</div>
);
}
diff --git a/src/ui/client/ThumbnailTemplate.tsx b/src/ui/client/ThumbnailTemplate.tsx
index e7266e5..bb256ef 100644
--- a/src/ui/client/ThumbnailTemplate.tsx
+++ b/src/ui/client/ThumbnailTemplate.tsx
@@ -13,6 +13,10 @@ const GROUPS: { group: string; items: Field[] }[] = [
{ k: "bg_color", t: "color" },
{ k: "font_family", t: "text", wide: true }, { k: "font_import_url", t: "text", wide: true },
]},
+ { group: "Layout", items: [
+ { k: "layout", t: "select", opts: ["single", "pair"] },
+ { k: "pair_photo_height", t: "text" }, { k: "pair_box_y", t: "text" }, { k: "pair_divider_width", t: "num" },
+ ]},
{ group: "Frame & box", items: [
{ k: "frame_border_width", t: "num" },
{ k: "box_x", t: "text" }, { k: "box_y", t: "text" },
diff --git a/src/ui/client/lib.test.ts b/src/ui/client/lib.test.ts
index cf3615d..657716b 100644
--- a/src/ui/client/lib.test.ts
+++ b/src/ui/client/lib.test.ts
@@ -9,6 +9,9 @@ import {
clampClipIndex,
resolveAssetName,
formatTranscriptText,
+ safeUpper,
+ describePeople,
+ buildUiStateSyncPayload,
} from "./lib";
describe("fmt", () => {
@@ -212,3 +215,64 @@ describe("formatTranscriptText", () => {
);
});
});
+
+describe("safeUpper", () => {
+ it("uppercases plain ASCII text", () => {
+ expect(safeUpper("hello world")).toBe("HELLO WORLD");
+ });
+
+ it("leaves Georgian Mkhedruli text unchanged instead of switching to Mtavruli", () => {
+ const georgian = "მიშა";
+ expect(safeUpper(georgian)).toBe(georgian);
+ });
+
+ it("uppercases the Latin parts of a mixed-script string and leaves Georgian alone", () => {
+ expect(safeUpper("hello მიშა")).toBe("HELLO მიშა");
+ });
+
+ it("passes through empty strings", () => {
+ expect(safeUpper("")).toBe("");
+ });
+});
+
+describe("describePeople", () => {
+ it("names each person and the second their face came from", () => {
+ expect(describePeople([
+ { side: "left", role: "guest", from: "seats", source_time: 72 },
+ { side: "right", role: "host", from: "seats", source_time: 9 },
+ ])).toBe("guest at 1:12, host at 0:09");
+ });
+
+ it("falls back to the side when the role is unknown and skips a missing time", () => {
+ expect(describePeople([
+ { side: "left", role: null, from: "image", source_time: null },
+ { side: "right", role: null, from: "seats", source_time: 3 },
+ ])).toBe("left person, right person at 0:03");
+ });
+});
+
+describe("buildUiStateSyncPayload", () => {
+ // The studio used to sync the transcript in a request of its own, separate
+ // from videoPath and the rest of the syncable state. After silence
+ // removal, videoPath and transcript update together in one render, but
+ // the two separate fetches could still arrive at the server in either
+ // order. A videoPath-only request landing after the transcript-only one
+ // looked exactly like a bare set_video (videoPath with no transcript in
+ // the request) and cleared the transcript that had just arrived.
+
+ it("omits transcript when it hasn't changed, matching the old split-request shape", () => {
+ const payload = buildUiStateSyncPayload({ videoPath: "a.mp4" }, "a.mp4", { words: [] }, false);
+ expect(payload).not.toHaveProperty("transcript");
+ });
+
+ it("carries videoPath and a just-changed transcript in one payload, not two", () => {
+ const transcript = { words: [{ word: "hi", start: 0, end: 1 }] };
+ const payload = buildUiStateSyncPayload({ videoPath: "b.mp4" }, "b.mp4", transcript, true);
+ expect(payload).toMatchObject({ _source: "ui", videoPath: "b.mp4", transcript });
+ });
+
+ it("sends an explicit transcript: null when it changed to null, not a silent omission", () => {
+ const payload = buildUiStateSyncPayload({ videoPath: "a.mp4" }, "a.mp4", null, true);
+ expect(payload).toHaveProperty("transcript", null);
+ });
+});
diff --git a/src/ui/client/lib.ts b/src/ui/client/lib.ts
index eca88ee..21e7a5c 100644
--- a/src/ui/client/lib.ts
+++ b/src/ui/client/lib.ts
@@ -1,4 +1,5 @@
import type { CSSProperties } from "react";
+import type { ThumbnailPerson } from "../../utils/thumbnail-layout";
// Rolls into hours past 3600s: podcast timestamps ran past "78:31" without it.
export const fmt = (s: number) => {
@@ -11,6 +12,16 @@ export const fmt = (s: number) => {
return hours > 0 ? `${hours}:${mm}:${ss}` : `${minutes}:${ss}`;
};
+/** "guest at 0:12, host at 0:41": where each face on a pair thumbnail came from. */
+export function describePeople(people: ThumbnailPerson[]): string {
+ return people
+ .map((p) => {
+ const who = p.role ?? `${p.side} person`;
+ return p.source_time == null ? who : `${who} at ${fmt(p.source_time)}`;
+ })
+ .join(", ");
+}
+
export const fmtMs = (s: number) =>
`${fmt(s)}.${String(Math.floor((s % 1) * 1000)).padStart(3, "0")}`;
@@ -26,6 +37,37 @@ export const labelStyle: CSSProperties = {
display: "block",
};
+// Scripts with no case distinction whose letters Unicode nonetheless assigns
+// an uppercase mapping for display styling (Georgian Mkhedruli -> Mtavruli).
+// Kept in sync with remotion/src/text.ts and backend/utils/text.py's
+// safe_upper. This is the same fix for the studio's live caption/thumbnail
+// preview, so it doesn't show a different alphabet than the final render.
+const CASELESS_SCRIPT_RANGES: Array<[number, number]> = [
+ [0x10a0, 0x10ff], // Georgian (Mkhedruli, Asomtavruli)
+ [0x1c90, 0x1cbf], // Georgian Extended (Mtavruli)
+ [0x2d00, 0x2d2f], // Georgian Supplement
+];
+
+function isCaselessScriptChar(ch: string): boolean {
+ const cp = ch.codePointAt(0) ?? 0;
+ return CASELESS_SCRIPT_RANGES.some(([lo, hi]) => cp >= lo && cp <= hi);
+}
+
+export function safeUpper(text: string): string {
+ if (!text) return text;
+ let hasCaseless = false;
+ for (const ch of text) {
+ if (isCaselessScriptChar(ch)) {
+ hasCaseless = true;
+ break;
+ }
+ }
+ if (!hasCaseless) return text.toUpperCase();
+ return Array.from(text)
+ .map((ch) => (isCaselessScriptChar(ch) ? ch : ch.toUpperCase()))
+ .join("");
+}
+
export class ApiError extends Error {
constructor(
message: string,
@@ -303,3 +345,27 @@ export function findClipResult<T extends ClipResultRow>(
const row = results[positionalIdx];
return row && !resultBoundsKey(row) ? row : undefined;
}
+
+/**
+ * Body for the combined POST /api/ui-state sync. Transcript rides in this
+ * same request whenever it changed, rather than a request of its own:
+ * after silence removal, videoPath and transcript update together in one
+ * render, and splitting them into separate fetches raced on the server. A
+ * videoPath-only request landing after the transcript-only one looked
+ * exactly like a bare set_video (videoPath with no transcript), which
+ * clears the transcript server-side as if it were stale. One request can't
+ * race with itself.
+ */
+export function buildUiStateSyncPayload(
+ syncable: Record<string, unknown>,
+ filePath: string,
+ transcript: unknown,
+ transcriptChanged: boolean,
+): Record<string, unknown> {
+ return {
+ _source: "ui",
+ filePath,
+ ...syncable,
+ ...(transcriptChanged && { transcript }),
+ };
+}
diff --git a/src/ui/client/multicam-sync.tsx b/src/ui/client/multicam-sync.tsx
index 957f151..73db9b3 100644
--- a/src/ui/client/multicam-sync.tsx
+++ b/src/ui/client/multicam-sync.tsx
@@ -6,6 +6,7 @@ import { whoLabel } from "./multicam-types";
const STATUS: Record<string, { label: string; color: string; pill: string }> = {
reference: { label: "Reference", color: "var(--green)", pill: "pill-green" },
ok: { label: "Synced", color: "var(--green)", pill: "pill-green" },
+ review: { label: "Check sync", color: "var(--amber)", pill: "pill-amber" },
rough: { label: "Rough sync", color: "var(--amber)", pill: "pill-amber" },
manual: { label: "Set by hand", color: "var(--blue)", pill: "pill-sky" },
assumed: { label: "Starts with the others", color: "var(--amber)", pill: "pill-amber" },
@@ -100,6 +101,12 @@ function SyncRow({ source, people, onEdit, disabled }: { source: McSource; peopl
<span className="hint" style={{ color: "var(--red)" }}>{source.sync.message || "Sync this file, or type where it starts."}</span>
)}
{source.sync.status === "assumed" && <span className="hint">{source.sync.message}</span>}
+ {source.sync.status === "review" && (
+ <span className="hint" style={{ color: "var(--amber)" }}>
+ Residual {Math.round(Math.max(source.sync.residual_ms ?? 0, source.sync.residual_all_ms ?? 0))} ms
+ {source.sync.reasons?.length ? ` (${source.sync.reasons.join(", ")})` : ""}
+ </span>
+ )}
<div style={{ display: "inline-flex", gap: 6, marginLeft: "auto", alignItems: "center" }}>
{source.offset === null ? (
diff --git a/src/ui/client/multicam-types.ts b/src/ui/client/multicam-types.ts
index cfaa5e1..cdc3384 100644
--- a/src/ui/client/multicam-types.ts
+++ b/src/ui/client/multicam-types.ts
@@ -9,15 +9,19 @@ export interface McPerson {
}
export type McRole = "camera" | "mic" | "ignore";
-export type McSyncStatus = "reference" | "ok" | "rough" | "failed" | "manual" | "assumed";
+export type McSyncStatus = "reference" | "ok" | "review" | "rough" | "failed" | "manual" | "assumed";
export interface McSourceSync {
status?: McSyncStatus;
+ method?: "manual";
+ anchors?: { timeline: number; source: number }[];
score?: number;
checkpoints?: number;
residual_ms?: number;
+ residual_all_ms?: number;
drift_ppm?: number;
message?: string;
+ reasons?: string[];
}
export interface McSource {
@@ -28,12 +32,16 @@ export interface McSource {
duration: number;
has_audio: boolean;
audio_channels: number;
+ audio_stream_count?: number;
+ audio_stream_index?: number;
width?: number;
height?: number;
fps?: number;
+ fps_warning?: string;
role: McRole;
person: string;
channel_people: string[];
+ input_lut?: string;
guessed: boolean;
offset: number | null;
speed: number;
@@ -72,12 +80,23 @@ export interface McStats {
share?: Record<string, number>;
}
+export interface McValidation {
+ frames_expected: number;
+ frames_actual: number;
+ duration: number;
+ lufs: number | null;
+ true_peak: number | null;
+ warnings: string[];
+}
+
export interface McOutputs {
video?: string;
stems?: string[];
duration?: number;
+ validation?: McValidation;
premiere?: string;
fcpxml?: string;
+ color_handoff?: string;
}
export interface McSession {
@@ -117,7 +136,8 @@ export interface McSessionSummary {
export interface McPreviews {
cameras: Record<string, string>;
- looks: Record<string, string>;
+ /** Camera id to look name to still path. */
+ looks: Record<string, Record<string, string>>;
}
export interface McPreviewsResp extends McSession {
diff --git a/src/ui/web-server-outputs.test.ts b/src/ui/web-server-outputs.test.ts
new file mode 100644
index 0000000..7ba4371
--- /dev/null
+++ b/src/ui/web-server-outputs.test.ts
@@ -0,0 +1,87 @@
+import { describe, it, expect, beforeAll, afterAll, vi } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// A re-render reuses its output path, so a "_clean.mp4" variant from a
+// previous render that this pass doesn't produce could sit next to the new
+// main clip. /api/outputs used to list every "*_clean.mp4" as its own
+// unrelated-looking clip instead of folding it into the main clip's entry.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-outputs-"));
+const savedEnv: Record<string, string | undefined> = {
+ PODCLI_HOME: process.env.PODCLI_HOME,
+ PODCLI_DATA: process.env.PODCLI_DATA,
+ PODCLI_OUTPUT: process.env.PODCLI_OUTPUT,
+ PODCLI_PORT: process.env.PODCLI_PORT,
+};
+
+const PORT = 18426;
+
+vi.mock("../services/python-executor.js", async (importOriginal) => {
+ const original = await importOriginal<typeof import("../services/python-executor.js")>();
+ class StubExecutor {
+ async execute() {
+ return { data: {} };
+ }
+ }
+ return { ...original, PythonExecutor: StubExecutor };
+});
+
+let outputDir: string;
+
+async function waitForServer(): Promise<void> {
+ for (let i = 0; i < 50; i++) {
+ try {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/ui-state`);
+ if (res.ok) return;
+ } catch {
+ // not up yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ throw new Error("web server never came up");
+}
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ outputDir = join(tmp, "data", "output");
+ process.env.PODCLI_OUTPUT = outputDir;
+ process.env.PODCLI_PORT = String(PORT);
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+ mkdirSync(outputDir, { recursive: true });
+
+ writeFileSync(join(outputDir, "clip-a.mp4"), "main clip bytes");
+ writeFileSync(join(outputDir, "clip-a_clean.mp4"), "clean variant bytes");
+ writeFileSync(join(outputDir, "clip-b.mp4"), "a second clip with no clean variant");
+
+ await import("./web-server.js");
+ await waitForServer();
+}, 20_000);
+
+afterAll(() => {
+ for (const [key, value] of Object.entries(savedEnv)) {
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+describe("GET /api/outputs", () => {
+ it("folds a clip's _clean.mp4 variant into its own entry instead of listing it separately", async () => {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/outputs`);
+ const clips = (await res.json()) as Array<{ filename: string; clean_output_path?: string }>;
+
+ const filenames = clips.map((c) => c.filename);
+ expect(filenames).toContain("clip-a.mp4");
+ expect(filenames).toContain("clip-b.mp4");
+ expect(filenames).not.toContain("clip-a_clean.mp4");
+
+ const clipA = clips.find((c) => c.filename === "clip-a.mp4");
+ expect(clipA?.clean_output_path).toBe(join(outputDir, "clip-a_clean.mp4"));
+
+ const clipB = clips.find((c) => c.filename === "clip-b.mp4");
+ expect(clipB?.clean_output_path).toBeUndefined();
+ });
+});
diff --git a/src/ui/web-server-studio-edit.test.ts b/src/ui/web-server-studio-edit.test.ts
new file mode 100644
index 0000000..6989e17
--- /dev/null
+++ b/src/ui/web-server-studio-edit.test.ts
@@ -0,0 +1,113 @@
+import { describe, it, expect, beforeAll, afterAll, vi } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// The studio's "_source: 'ui'" suggestions merge in POST /api/ui-state
+// restores a clip's segments from server state when the studio's edit
+// omitted them (so a stale array scoped to the old range doesn't override
+// the new range), but it carried the clip's *old* duration forward verbatim
+// instead of recomputing it, and never ran the "this no longer matches what
+// was selected" hash check that modify_clip's own endpoint runs. A studio
+// edit after approval could silently export something other than what was
+// approved, and adding a hook to an approved clip reported a duration that
+// excluded it.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-studio-edit-"));
+const savedEnv: Record<string, string | undefined> = {
+ PODCLI_HOME: process.env.PODCLI_HOME,
+ PODCLI_DATA: process.env.PODCLI_DATA,
+ PODCLI_PORT: process.env.PODCLI_PORT,
+};
+
+const PORT = 18427;
+
+vi.mock("../services/python-executor.js", async (importOriginal) => {
+ const original = await importOriginal<typeof import("../services/python-executor.js")>();
+ class StubExecutor {
+ async execute() {
+ return { data: {} };
+ }
+ }
+ return { ...original, PythonExecutor: StubExecutor };
+});
+
+async function waitForServer(): Promise<void> {
+ for (let i = 0; i < 50; i++) {
+ try {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/ui-state`);
+ if (res.ok) return;
+ } catch {
+ // not up yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ throw new Error("web server never came up");
+}
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ process.env.PODCLI_PORT = String(PORT);
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+
+ await import("./web-server.js");
+ await waitForServer();
+}, 20_000);
+
+afterAll(() => {
+ for (const [key, value] of Object.entries(savedEnv)) {
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+const baseClip = {
+ clip_id: "c1",
+ title: "clip",
+ start_second: 10,
+ end_second: 30,
+ duration: 20,
+ reasoning: "",
+ preview_text: "",
+ segments: [{ start: 10, end: 30 }],
+};
+
+describe("POST /api/ui-state studio suggestions merge", () => {
+ it("flags changedSinceSelection and recomputes duration to include a newly added hook", async () => {
+ // Seed one suggestion from the MCP side.
+ await fetch(`http://127.0.0.1:${PORT}/api/ui-state`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ suggestions: [baseClip] }),
+ });
+
+ // Approve it: stamps a selectionHash, same as the studio's "select" action.
+ await fetch(`http://127.0.0.1:${PORT}/api/suggestions/modify`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ action: "toggle", index: 0, selected: true }),
+ });
+
+ // The studio edits the clip (adds a 3s hook) and syncs back without
+ // segments, as it does after a server round trip strips them.
+ const studioUpdate = {
+ ...baseClip,
+ segments: undefined,
+ hook: { start: 12, end: 15, mode: "repeat" },
+ };
+ await fetch(`http://127.0.0.1:${PORT}/api/ui-state`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ _source: "ui", suggestions: [studioUpdate] }),
+ });
+
+ const state = await (await fetch(`http://127.0.0.1:${PORT}/api/ui-state`)).json();
+ const clip = state.suggestions[0];
+
+ // Body (20s) + hook (3s) played first, repeat mode.
+ expect(clip.duration).toBeCloseTo(23, 1);
+ expect(clip.changedSinceSelection).toBe(true);
+ });
+});
diff --git a/src/ui/web-server-transcribe-identity.test.ts b/src/ui/web-server-transcribe-identity.test.ts
new file mode 100644
index 0000000..c381c55
--- /dev/null
+++ b/src/ui/web-server-transcribe-identity.test.ts
@@ -0,0 +1,153 @@
+import { describe, it, expect, beforeAll, afterAll, vi } from "vitest";
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// transcribe_start (POST /api/transcribe) used to update uiState.transcript
+// and uiState.videoPath without setting transcriptVideoIdentity. The next
+// create_clip/batch_create_clips call reads that identity to confirm the
+// session's transcript still belongs to the loaded video; left unset, a
+// freshly transcribed session falsely looks like it never recorded one, and
+// callers relying on it being present (or on it being cleared on a stale
+// video) see the wrong state. This test drives the real HTTP route and
+// checks the persisted ui-state.json, not just the in-memory handler logic.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-transcribe-identity-"));
+const savedEnv: Record<string, string | undefined> = {
+ PODCLI_HOME: process.env.PODCLI_HOME,
+ PODCLI_DATA: process.env.PODCLI_DATA,
+ PODCLI_PORT: process.env.PODCLI_PORT,
+};
+
+const PORT = 18424;
+
+// The web server spawns the Python backend on startup (manage_config status
+// check) and per-transcribe-request (engine resolution). Stub it so the test
+// never depends on a real backend or network call, and resolves instantly.
+vi.mock("../services/python-executor.js", async (importOriginal) => {
+ const original = await importOriginal<typeof import("../services/python-executor.js")>();
+ class StubExecutor {
+ async execute(taskType: string) {
+ if (taskType === "resolve_transcribe_engine") {
+ return { data: { engine: "whisper-py" } };
+ }
+ if (taskType === "transcribe") {
+ return { data: { engine: "whisper-py", words: [{ word: "hi", start: 0, end: 1 }] } };
+ }
+ return { data: {} };
+ }
+ }
+ return { ...original, PythonExecutor: StubExecutor };
+});
+
+let videoPath: string;
+let uiStatePath: string;
+
+async function waitForServer(): Promise<void> {
+ for (let i = 0; i < 50; i++) {
+ try {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/ui-state`);
+ if (res.ok) return;
+ } catch {
+ // not up yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ throw new Error("web server never came up");
+}
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ process.env.PODCLI_PORT = String(PORT);
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+ uiStatePath = join(process.env.PODCLI_HOME, "ui-state.json");
+
+ videoPath = join(tmp, "episode.mp4");
+ writeFileSync(videoPath, "fake video bytes");
+
+ const { TranscriptCache } = await import("../services/transcript-cache.js");
+ const cache = new TranscriptCache();
+ await cache.set(
+ videoPath,
+ { words: [{ word: "hi", start: 0, end: 1 }] } as never,
+ { engine: "whisper-py", model: "base", language: undefined },
+ );
+
+ await import("./web-server.js");
+ await waitForServer();
+}, 20_000);
+
+afterAll(() => {
+ process.env.PODCLI_HOME = savedEnv.PODCLI_HOME;
+ process.env.PODCLI_DATA = savedEnv.PODCLI_DATA;
+ process.env.PODCLI_PORT = savedEnv.PODCLI_PORT;
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+describe("POST /api/transcribe", () => {
+ it("sets transcriptVideoIdentity for the cached-transcript branch", async () => {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/transcribe`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({
+ file_path: videoPath,
+ model_size: "base",
+ enable_diarization: false,
+ }),
+ });
+ const body = await res.json();
+ expect(body.cached).toBe(true);
+
+ // /api/ui-state doesn't echo transcriptVideoIdentity, and persistState()
+ // debounces its write to disk by 500ms, so poll the persisted file.
+ let uiState: { transcriptVideoIdentity?: { path?: string } } = {};
+ for (let i = 0; i < 20; i++) {
+ try {
+ uiState = JSON.parse(readFileSync(uiStatePath, "utf-8"));
+ if (uiState.transcriptVideoIdentity) break;
+ } catch {
+ // not written yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ expect(uiState.transcriptVideoIdentity).toBeTruthy();
+ expect(uiState.transcriptVideoIdentity?.path).toBe(videoPath);
+ });
+
+ it("sets transcriptVideoIdentity for the async (uncached) transcription branch", async () => {
+ const freshVideoPath = join(tmp, "episode-2.mp4");
+ writeFileSync(freshVideoPath, "other fake video bytes");
+
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/transcribe`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({
+ file_path: freshVideoPath,
+ model_size: "base",
+ enable_diarization: false,
+ }),
+ });
+ const { job_id } = await res.json();
+
+ let job: { status?: string } = {};
+ for (let i = 0; i < 50; i++) {
+ job = await (await fetch(`http://127.0.0.1:${PORT}/api/job/${job_id}`)).json();
+ if (job.status === "done" || job.status === "error") break;
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ expect(job.status).toBe("done");
+
+ let uiState: { transcriptVideoIdentity?: { path?: string } } = {};
+ for (let i = 0; i < 20; i++) {
+ try {
+ uiState = JSON.parse(readFileSync(uiStatePath, "utf-8"));
+ if (uiState.transcriptVideoIdentity?.path === freshVideoPath) break;
+ } catch {
+ // not written yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ expect(uiState.transcriptVideoIdentity?.path).toBe(freshVideoPath);
+ });
+});
diff --git a/src/ui/web-server-video-change-broadcast.test.ts b/src/ui/web-server-video-change-broadcast.test.ts
new file mode 100644
index 0000000..242b932
--- /dev/null
+++ b/src/ui/web-server-video-change-broadcast.test.ts
@@ -0,0 +1,133 @@
+import { describe, it, expect, beforeAll, afterAll, vi } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// A bare set_video from MCP (videoPath with no transcript in the same
+// request) clears the server's transcript/suggestions/selections, since
+// they describe a recording that's no longer loaded. But the SSE broadcast
+// only echoed back whatever fields the request body itself carried. A
+// request that only sent videoPath never told the studio that transcript
+// and suggestions were cleared too, so the studio's in-memory copies stayed
+// stale and a later sync from the studio wrote them right back onto the
+// server.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-video-change-broadcast-"));
+const savedEnv: Record<string, string | undefined> = {
+ PODCLI_HOME: process.env.PODCLI_HOME,
+ PODCLI_DATA: process.env.PODCLI_DATA,
+ PODCLI_PORT: process.env.PODCLI_PORT,
+};
+
+const PORT = 18429;
+
+vi.mock("../services/python-executor.js", async (importOriginal) => {
+ const original = await importOriginal<typeof import("../services/python-executor.js")>();
+ class StubExecutor {
+ async execute() {
+ return { data: {} };
+ }
+ }
+ return { ...original, PythonExecutor: StubExecutor };
+});
+
+async function waitForServer(): Promise<void> {
+ for (let i = 0; i < 50; i++) {
+ try {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/ui-state`);
+ if (res.ok) return;
+ } catch {
+ // not up yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ throw new Error("web server never came up");
+}
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ process.env.PODCLI_PORT = String(PORT);
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+
+ await import("./web-server.js");
+ await waitForServer();
+}, 20_000);
+
+afterAll(() => {
+ for (const [key, value] of Object.entries(savedEnv)) {
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+/** Collects every "event: state-sync" payload broadcast while `during` runs. */
+async function captureStateSyncEvents(during: () => Promise<void>): Promise<Array<Record<string, unknown>>> {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/events`);
+ const reader = res.body!.getReader();
+ const decoder = new TextDecoder();
+ let buffer = "";
+ const events: Array<Record<string, unknown>> = [];
+
+ const pump = async () => {
+ for (;;) {
+ const { value, done } = await reader.read();
+ if (done) return;
+ buffer += decoder.decode(value, { stream: true });
+ let idx;
+ while ((idx = buffer.indexOf("\n\n")) !== -1) {
+ const chunk = buffer.slice(0, idx);
+ buffer = buffer.slice(idx + 2);
+ const eventLine = chunk.split("\n").find((l) => l.startsWith("event: "));
+ const dataLine = chunk.split("\n").find((l) => l.startsWith("data: "));
+ if (eventLine?.slice(7) === "state-sync" && dataLine) {
+ events.push(JSON.parse(dataLine.slice(6)));
+ }
+ }
+ }
+ };
+ const pumpPromise = pump();
+
+ await during();
+ await new Promise((r) => setTimeout(r, 200)); // let the broadcast land
+ await reader.cancel();
+ await pumpPromise.catch(() => {});
+ return events;
+}
+
+describe("POST /api/ui-state video change broadcast", () => {
+ it("broadcasts the cleared transcript and suggestions, not just videoPath", async () => {
+ const videoA = join(tmp, "episode-a.mp4");
+ const videoB = join(tmp, "episode-b.mp4");
+
+ // Seed a session on video A with a transcript and a suggestion.
+ await fetch(`http://127.0.0.1:${PORT}/api/ui-state`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({
+ videoPath: videoA,
+ transcript: { words: [{ word: "hi", start: 0, end: 1 }] },
+ suggestions: [
+ { clip_id: "c1", title: "a moment", start_second: 0, end_second: 5, duration: 5, reasoning: "", preview_text: "" },
+ ],
+ }),
+ });
+
+ const events = await captureStateSyncEvents(async () => {
+ // A bare set_video from MCP: videoPath changes, no transcript in the
+ // same request.
+ await fetch(`http://127.0.0.1:${PORT}/api/ui-state`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ videoPath: videoB }),
+ });
+ });
+
+ expect(events.length).toBeGreaterThan(0);
+ const last = events[events.length - 1];
+ expect(last.videoPath).toBe(videoB);
+ expect(last.transcript).toBeNull();
+ expect(last.suggestions).toEqual([]);
+ });
+});
diff --git a/src/ui/web-server-video-missing.test.ts b/src/ui/web-server-video-missing.test.ts
new file mode 100644
index 0000000..f3f8a6c
--- /dev/null
+++ b/src/ui/web-server-video-missing.test.ts
@@ -0,0 +1,86 @@
+import { describe, it, expect, beforeAll, afterAll, vi } from "vitest";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+
+// Loading persisted state at startup used to wipe the transcript and
+// suggestions whenever the saved videoPath didn't exist on disk (e.g. an
+// external drive is unmounted), losing a whole session for something that
+// might just be temporarily unreachable. It should keep them and flag the
+// video as missing instead.
+
+const tmp = mkdtempSync(join(tmpdir(), "podcli-video-missing-"));
+const savedEnv: Record<string, string | undefined> = {
+ PODCLI_HOME: process.env.PODCLI_HOME,
+ PODCLI_DATA: process.env.PODCLI_DATA,
+ PODCLI_PORT: process.env.PODCLI_PORT,
+};
+
+const PORT = 18428;
+
+vi.mock("../services/python-executor.js", async (importOriginal) => {
+ const original = await importOriginal<typeof import("../services/python-executor.js")>();
+ class StubExecutor {
+ async execute() {
+ return { data: {} };
+ }
+ }
+ return { ...original, PythonExecutor: StubExecutor };
+});
+
+async function waitForServer(): Promise<void> {
+ for (let i = 0; i < 50; i++) {
+ try {
+ const res = await fetch(`http://127.0.0.1:${PORT}/api/ui-state`);
+ if (res.ok) return;
+ } catch {
+ // not up yet
+ }
+ await new Promise((r) => setTimeout(r, 100));
+ }
+ throw new Error("web server never came up");
+}
+
+const missingVideoPath = join(tmp, "unmounted-drive", "episode.mp4");
+
+beforeAll(async () => {
+ process.env.PODCLI_HOME = join(tmp, "home");
+ process.env.PODCLI_DATA = join(tmp, "data");
+ process.env.PODCLI_PORT = String(PORT);
+ mkdirSync(process.env.PODCLI_HOME, { recursive: true });
+
+ writeFileSync(
+ join(process.env.PODCLI_HOME, "ui-state.json"),
+ JSON.stringify({
+ videoPath: missingVideoPath,
+ filePath: missingVideoPath,
+ transcript: { words: [{ word: "hi", start: 0, end: 1 }] },
+ suggestions: [
+ { clip_id: "c1", title: "a moment", start_second: 0, end_second: 10, duration: 10, reasoning: "", preview_text: "" },
+ ],
+ deselectedIndices: [],
+ }),
+ );
+
+ await import("./web-server.js");
+ await waitForServer();
+}, 20_000);
+
+afterAll(() => {
+ for (const [key, value] of Object.entries(savedEnv)) {
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+describe("loading persisted state with a missing video", () => {
+ it("keeps the transcript and suggestions and flags videoMissing instead of wiping them", async () => {
+ const state = await (await fetch(`http://127.0.0.1:${PORT}/api/ui-state`)).json();
+ expect(state.videoMissing).toBe(true);
+ expect(state.transcriptWordCount).toBe(1);
+ expect(state.transcript?.words?.[0]?.word).toBe("hi");
+ expect(state.suggestions).toHaveLength(1);
+ expect(state.suggestions[0].title).toBe("a moment");
+ });
+});
diff --git a/src/ui/web-server.ts b/src/ui/web-server.ts
index 4a10756..0785e8a 100644
--- a/src/ui/web-server.ts
+++ b/src/ui/web-server.ts
@@ -31,7 +31,8 @@ import { fileURLToPath } from "url";
import { v4 as uuidv4 } from "uuid";
import { PythonExecutor, terminateProcessTree } from "../services/python-executor.js";
-import { hasSpeakerLabels, TranscriptCache } from "../services/transcript-cache.js";
+import { needsDiarizationRetry, TranscriptCache } from "../services/transcript-cache.js";
+import { resolveTranscribeEngine, engineCanDiarize } from "../services/engine-resolve.js";
import { FileManager } from "../services/file-manager.js";
import { AssetManager, inferType, safeName } from "../services/asset-manager.js";
import { ClipsHistory } from "../services/clips-history.js";
@@ -40,11 +41,24 @@ import { paths, pythonEnv } from "../config/paths.js";
import { webServerPort } from "../config/server.js";
import { writeFileAtomicSync } from "../utils/atomic-file.js";
import { maxClipSeconds, validateClipRange, validateSuggestionRange } from "../utils/clip-validation.js";
+import { playbackDuration, validateHook } from "../utils/clip-hook.js";
+import { needsFrame, parseThumbnailRender, thumbnailRenderArgs } from "../utils/thumbnail-layout.js";
import { advanceProgress, tagSubmittedClip, tagSubmittedClips } from "../utils/clip-results.js";
import { DEMO_ASSETS_DIR } from "./demo-fixtures.js";
import { registerConfigIntegrationRoutes } from "../handlers/integrations.routes.js";
import { childLogger } from "../utils/logger.js";
-import { sliceTranscript, sliceWords, findContentType, findSuggestionSegments } from "../utils/transcript.js";
+import {
+ sliceTranscript,
+ sliceWords,
+ findContentType,
+ findGroundingText,
+ findSuggestionForRange,
+ findSuggestionSegments,
+ reconcileSegmentsForRange,
+} from "../utils/transcript.js";
+import { formatSrtTime, formatVttTime } from "../utils/srt-time.js";
+import { computeVideoIdentity, type VideoIdentity } from "../utils/video-identity.js";
+import { computeSelectionHash } from "../utils/selection-hash.js";
import { errMsg } from "../utils/errors.js";
import { resolveByteRange } from "../utils/http-range.js";
import {
@@ -57,6 +71,7 @@ import type {
AssetType,
BatchClipsResult,
ClipHistoryEntry,
+ ClipHook,
ClipResult,
Format,
ProgressEvent,
@@ -150,9 +165,14 @@ interface UIState {
filePath: string;
activeExportJobId: string | null;
transcript: ServerTranscript | null;
+ transcriptVideoIdentity: VideoIdentity | null;
rawTranscriptText: string;
silenceOriginal: SilenceOriginal | null;
silencePlan: SilencePlan | null;
+ // True when the persisted videoPath didn't exist at startup (e.g. an
+ // external drive is unmounted). The session (transcript, suggestions)
+ // is kept rather than wiped, since the file may well come back.
+ videoMissing: boolean;
suggestions: SuggestedClip[];
deselectedIndices: number[];
settings: {
@@ -183,12 +203,11 @@ function loadPersistedState(): UIState {
if (existsSync(paths.uiState)) {
const raw = readFileSync(paths.uiState, "utf-8");
const saved = JSON.parse(raw);
- // Validate video still exists
- if (saved.videoPath && !existsSync(saved.videoPath)) {
- saved.videoPath = "";
- saved.filePath = "";
- saved.phase = "idle";
- }
+ // A missing video (e.g. its external drive is unmounted) doesn't mean
+ // the episode is gone. The transcript and suggestions are kept as a
+ // session the file may rejoin; flag it instead so the caller can warn
+ // and skip anything that needs the file on disk right now.
+ const videoMissing = !!saved.videoPath && !existsSync(saved.videoPath);
if (saved.silenceOriginal?.videoPath && !existsSync(saved.silenceOriginal.videoPath)) {
saved.silenceOriginal = null;
}
@@ -197,9 +216,11 @@ function loadPersistedState(): UIState {
filePath: saved.filePath || "",
activeExportJobId: null,
transcript: saved.transcript || null,
+ transcriptVideoIdentity: saved.transcriptVideoIdentity || null,
rawTranscriptText: saved.rawTranscriptText || "",
silenceOriginal: saved.silenceOriginal || null,
silencePlan: saved.silencePlan || null,
+ videoMissing,
suggestions: saved.suggestions || [],
deselectedIndices: saved.deselectedIndices || [],
settings: {
@@ -239,9 +260,11 @@ function loadPersistedState(): UIState {
filePath: "",
activeExportJobId: null,
transcript: null,
+ transcriptVideoIdentity: null,
rawTranscriptText: "",
silenceOriginal: null,
silencePlan: null,
+ videoMissing: false,
suggestions: [],
deselectedIndices: [],
settings: {
@@ -390,12 +413,32 @@ function setExportState(phase: string, activeExportJobId: string | null) {
persistState();
}
-function enrichClipWithSegments<T extends { start_second: number; end_second: number; keep_segments?: Array<{ start: number; end: number }> }>(
- clip: T,
-): T {
- if (clip.keep_segments?.length) return clip;
- const segments = findSuggestionSegments(uiState.suggestions, clip.start_second, clip.end_second);
- return segments?.length ? { ...clip, keep_segments: segments } : clip;
+type ExportClip = {
+ start_second: number;
+ end_second: number;
+ keep_segments?: Array<{ start: number; end: number }>;
+ hook?: ClipHook | null;
+};
+
+/** Fill segments and hook the caller left out from the matching suggestion.
+ * An explicit hook: null stays null, so a render can opt out of one. */
+function enrichClipFromSuggestion<T extends ExportClip>(clip: T): T {
+ const match = findSuggestionForRange(uiState.suggestions, clip.start_second, clip.end_second);
+ if (!match) return clip;
+ return {
+ ...clip,
+ ...(!clip.keep_segments?.length && match.segments?.length && { keep_segments: match.segments }),
+ ...(clip.hook === undefined && match.hook && { hook: match.hook }),
+ };
+}
+
+function exportHookError(clips: ExportClip[]): string | null {
+ for (let i = 0; i < clips.length; i++) {
+ const c = clips[i];
+ const err = validateHook(c.hook, c.start_second, c.end_second, c.keep_segments);
+ if (err) return clips.length > 1 ? `Clip ${i + 1}: ${err}` : err;
+ }
+ return null;
}
function createBatchHistoryRecorder({
@@ -426,6 +469,7 @@ function createBatchHistoryRecorder({
crop_strategy?: string;
format?: Format;
keep_segments?: Array<{ start: number; end: number }>;
+ hook?: ClipHook | null;
}>;
logoPath?: string | null;
outroPath?: string | null;
@@ -443,6 +487,7 @@ function createBatchHistoryRecorder({
defaultCropStrategy,
defaultFormat,
contentTypeFor: (s, e) => findContentType(uiState.suggestions, s, e),
+ suggestions: uiState.suggestions,
});
let recordedIdx = 0;
for (const row of rows) {
@@ -459,6 +504,7 @@ function createBatchHistoryRecorder({
introPath,
cleanFillers,
keepSegments: spec?.keep_segments,
+ hook: spec?.hook,
});
} catch (err) {
log.warn(`Failed to save recipe for ${label} clip`, { err: errMsg(err) });
@@ -547,8 +593,10 @@ function clearEpisodeSessionState(): void {
allowedSourcePaths.clear();
uiState.videoPath = "";
uiState.filePath = "";
+ uiState.videoMissing = false;
uiState.activeExportJobId = null;
uiState.transcript = null;
+ uiState.transcriptVideoIdentity = null;
uiState.rawTranscriptText = "";
uiState.silenceOriginal = null;
uiState.silencePlan = null;
@@ -799,6 +847,11 @@ app.post("/api/download-video", async (req, res) => {
registerSourcePath(filePath);
uiState.videoPath = filePath;
uiState.filePath = filePath;
+ uiState.videoMissing = false;
+ // A fresh download has no transcript yet; a stale identity from
+ // whatever video was loaded before would otherwise still read as "the
+ // transcript belongs to this video" until the next transcribe call.
+ uiState.transcriptVideoIdentity = null;
uiState.lastUpdated = Date.now();
persistState();
broadcastSSE("state-sync", uiState);
@@ -991,7 +1044,7 @@ app.post("/api/import-transcript", (req, res) => {
* Uses Python backend to generate word-level timestamps.
*/
app.post("/api/parse-transcript", async (req, res) => {
- const { file_path, raw_text, total_duration, time_adjust = 0 } = req.body;
+ const { file_path, raw_text, total_duration, time_adjust = 0, language } = req.body;
if (!file_path) {
res.status(400).json({ error: "file_path is required" });
@@ -1007,6 +1060,7 @@ app.post("/api/parse-transcript", async (req, res) => {
raw_text,
total_duration: total_duration || null,
time_adjust: time_adjust || 0,
+ language: language || null,
});
if (result.data) {
@@ -1040,23 +1094,39 @@ app.post("/api/transcribe", async (req, res) => {
language,
enable_diarization = true,
num_speakers,
+ start_seconds,
+ duration_seconds,
} = req.body;
if (!file_path || !existsSync(file_path)) {
res.status(400).json({ error: "File not found" });
return;
}
- // Check cache first. A cached transcript without speakers cannot answer a
- // request for them, so serving it makes re-transcribing look like a no-op.
- const cachedRaw = await cache.get(file_path, engine);
- const cached =
- cachedRaw && enable_diarization && !hasSpeakerLabels(cachedRaw) ? null : cachedRaw;
+ // A sample is a positive window, not merely a present key, matching
+ // transcribe.handler.ts and backend/services/transcription.py exactly.
+ const isSample = (duration_seconds ?? 0) > 0 || (start_seconds ?? 0) > 0;
+
+ // Resolve before reading the cache: an unset engine is written under
+ // whatever transcribe_file actually ran (e.g. "whispercpp" on a native
+ // install), so reading with the raw unset request always misses.
+ const resolvedEngine = await resolveTranscribeEngine(executor, engine, model_size);
+ const cacheKey = { engine: resolvedEngine, model: model_size, language };
+
+ // A sample is a throwaway check on a slice of the file. Never serve or
+ // populate the session/UI state from the main cache (keyed by, and
+ // assumed to describe, the whole file).
+ const cachedRaw = isSample ? null : await cache.get(file_path, cacheKey);
+ const cached = needsDiarizationRetry(cachedRaw, enable_diarization, engineCanDiarize(resolvedEngine))
+ ? null
+ : cachedRaw;
if (cached) {
const jobId = uuidv4();
sessionTranscripts.set(file_path, cached as unknown as ServerTranscript);
uiState.transcript = cached as unknown as typeof uiState.transcript;
uiState.videoPath = file_path;
uiState.filePath = file_path;
+ uiState.videoMissing = false;
+ uiState.transcriptVideoIdentity = computeVideoIdentity(file_path);
registerSourcePath(file_path);
uiState.lastUpdated = Date.now();
persistState();
@@ -1086,7 +1156,17 @@ app.post("/api/transcribe", async (req, res) => {
executor
.execute(
"transcribe",
- { file_path, model_size, engine, assemblyai_api_key, language, enable_diarization, num_speakers },
+ {
+ file_path,
+ model_size,
+ engine,
+ assemblyai_api_key,
+ language,
+ enable_diarization,
+ num_speakers,
+ start_seconds,
+ duration_seconds,
+ },
(event) => {
job.progress = event.percent;
job.message = event.message;
@@ -1097,6 +1177,11 @@ app.post("/api/transcribe", async (req, res) => {
job.progress = 100;
job.message = "Transcription complete";
job.result = result.data;
+
+ // A sample result is a slice, not the episode. Leave the session/UI
+ // state and the main cache alone. The caller reads it via job_status.
+ if (isSample) return;
+
sessionTranscripts.set(
file_path,
result.data as unknown as ServerTranscript,
@@ -1106,6 +1191,8 @@ app.post("/api/transcribe", async (req, res) => {
uiState.transcript = result.data as unknown as typeof uiState.transcript;
uiState.videoPath = file_path;
uiState.filePath = file_path;
+ uiState.videoMissing = false;
+ uiState.transcriptVideoIdentity = computeVideoIdentity(file_path);
registerSourcePath(file_path);
uiState.lastUpdated = Date.now();
persistState();
@@ -1113,9 +1200,17 @@ app.post("/api/transcribe", async (req, res) => {
// forever: the job id lived only in the tab that started it, and nothing
// else announces that the transcript landed.
broadcastSSE("state-sync", uiState);
- // Cache it
+ // Cache it under the engine it actually ran with. A fresh resolution
+ // (or the pre-transcribe request) can differ from what transcribe_file
+ // fell back to once it tried loading the model for real.
try {
- await cache.set(file_path, result.data as unknown as TranscriptResult, engine);
+ const actualEngine =
+ (result.data as unknown as TranscriptResult | undefined)?.engine ?? resolvedEngine;
+ await cache.set(file_path, result.data as unknown as TranscriptResult, {
+ engine: actualEngine,
+ model: model_size,
+ language,
+ });
} catch (err) {
log.warn("Failed to cache transcript", { file_path, err: errMsg(err) });
}
@@ -1155,6 +1250,7 @@ app.post("/api/create-clip", async (req, res) => {
allow_ass_fallback = false,
content_type = null,
keep_segments,
+ hook,
caption_position = "auto",
caption_font_scale = 100,
logo_position = "top-left",
@@ -1233,11 +1329,17 @@ app.post("/api/create-clip", async (req, res) => {
await fileManager.ensureDirectories();
- const enriched = enrichClipWithSegments({
+ const enriched = enrichClipFromSuggestion({
start_second,
end_second,
keep_segments: Array.isArray(keep_segments) ? keep_segments : undefined,
+ hook: hook as ClipHook | null | undefined,
});
+ const hookError = exportHookError([enriched]);
+ if (hookError) {
+ res.status(400).json({ error: hookError });
+ return;
+ }
const jobId = uuidv4();
const job: JobState = {
@@ -1274,6 +1376,7 @@ app.post("/api/create-clip", async (req, res) => {
caption_font_scale: normalizedFontScale,
logo_position,
...(enriched.keep_segments?.length && { keep_segments: enriched.keep_segments }),
+ ...(enriched.hook && { hook: enriched.hook }),
},
(event) => {
job.progress = event.percent;
@@ -1304,6 +1407,7 @@ app.post("/api/create-clip", async (req, res) => {
duration: d?.duration || 0,
content_type: content_type || undefined,
transcript_slice: sliceTranscript(transcript_words, start_second, end_second),
+ ...findGroundingText(uiState.suggestions, start_second, end_second),
});
await clipsHistory.persistClipRecipe(rec, {
transcriptWords: transcript_words,
@@ -1312,6 +1416,7 @@ app.post("/api/create-clip", async (req, res) => {
introPath: intro_path,
cleanFillers: clean_fillers,
keepSegments: enriched.keep_segments,
+ hook: enriched.hook,
});
broadcastHistoryUpdated(jobId, [rec]);
} catch (err) {
@@ -1398,9 +1503,12 @@ app.post("/api/batch-clips", async (req, res) => {
await fileManager.ensureDirectories();
- const enrichedClips = clips.map((c: { start_second: number; end_second: number; keep_segments?: Array<{ start: number; end: number }> }) =>
- enrichClipWithSegments(c),
- );
+ const enrichedClips = clips.map((c: ExportClip) => enrichClipFromSuggestion(c));
+ const batchHookError = exportHookError(enrichedClips);
+ if (batchHookError) {
+ res.status(400).json({ error: batchHookError });
+ return;
+ }
const jobId = uuidv4();
const job: JobState = {
@@ -1474,6 +1582,9 @@ app.post("/api/batch-clips", async (req, res) => {
job.progress = 100;
job.message = "Batch complete!";
job.result = data;
+ // Persist here too: the studio tab may be closed (an MCP-driven
+ // export), and it only stores results when it hears job-complete.
+ uiState.results = data?.results ?? [];
// Record successful clips to history
try {
await historyRecorder.recordRemaining(result.data?.results);
@@ -1894,16 +2005,29 @@ app.get("/api/outputs", async (_req, res) => {
try {
await mkdir(paths.output, { recursive: true });
const files = await readdir(paths.output);
- const clips = files
- .filter((f) => f.endsWith(".mp4"))
+ const mp4Files = files.filter((f) => f.endsWith(".mp4"));
+ // A clean (caption-free) variant renders to "<stem>_clean.mp4" next to
+ // its main clip. Fold it into that clip's entry instead of listing it
+ // as a second, unrelated-looking clip.
+ const cleanByStem = new Map<string, string>();
+ for (const f of mp4Files) {
+ if (f.endsWith("_clean.mp4")) {
+ cleanByStem.set(f.slice(0, -"_clean.mp4".length), f);
+ }
+ }
+ const clips = mp4Files
+ .filter((f) => !f.endsWith("_clean.mp4"))
.map((f) => {
const fullPath = join(paths.output, f);
const stat = statSync(fullPath);
+ const stem = f.slice(0, -".mp4".length);
+ const cleanFilename = cleanByStem.get(stem);
return {
filename: f,
path: fullPath,
size_mb: Math.round((stat.size / (1024 * 1024)) * 100) / 100,
created: stat.mtime.toISOString(),
+ ...(cleanFilename && { clean_output_path: join(paths.output, cleanFilename) }),
};
})
.sort(
@@ -2149,15 +2273,8 @@ app.get("/api/export-transcript", (_req, res) => {
});
}
- const fmtSrt = (s: number) => {
- const h = Math.floor(s / 3600);
- const m = Math.floor((s % 3600) / 60);
- const sec = Math.floor(s % 60);
- const ms = Math.round((s % 1) * 1000);
- return `${String(h).padStart(2, "0")}:${String(m).padStart(2, "0")}:${String(sec).padStart(2, "0")},${String(ms).padStart(3, "0")}`;
- };
-
- const fmtVtt = (s: number) => fmtSrt(s).replace(",", ".");
+ const fmtSrt = formatSrtTime;
+ const fmtVtt = formatVttTime;
if (format === "vtt") {
let vtt = "WEBVTT\n\n";
@@ -2255,10 +2372,16 @@ const multicamRunning = new Map<string, { id: string; action: string }>();
type MulticamPayload = {
session_id?: string;
- outputs?: { video?: string };
+ outputs?: { video?: string; validation?: { warnings?: string[] } };
active_job?: { id: string; action: string };
};
+// A render that finished with warnings says so in the job's status line, which is what job_status shows first.
+function multicamDoneMessage(data: MulticamPayload | undefined): string {
+ const warnings = data?.outputs?.validation?.warnings || [];
+ return warnings.length ? `Done with ${warnings.length} warning(s): ${warnings.join(" ")}` : "Done";
+}
+
// Only the finished episode becomes streamable: paths in a request body are the
// caller's say-so, and registering them would let a request read any media file.
function allowMulticamPaths(data: MulticamPayload | undefined): void {
@@ -2311,7 +2434,7 @@ app.post("/api/multicam", async (req, res) => {
allowMulticamPaths(result.data);
job.status = "done";
job.progress = 100;
- job.message = "Done";
+ job.message = action === "render" || action === "pull" ? multicamDoneMessage(result.data) : "Done";
job.result = result.data;
}).catch((err) => {
job.status = "error";
@@ -2778,6 +2901,17 @@ app.post("/api/clips/:id/thumbnail/select", async (req, res) => {
res.json({ ok: true, preview_path: pick });
});
+// CLI flags grounding thumbnail headline copy in the clip's own content
+// (payoff, the question it answers, its verbatim opening line) instead of
+// just the title. Empty for clips rendered before these fields existed.
+function thumbnailGroundingArgs(clip: { payoff?: string; context_line?: string; preview_text?: string }): string[] {
+ const args: string[] = [];
+ if (clip.payoff) args.push("--payoff", clip.payoff);
+ if (clip.context_line) args.push("--context-line", clip.context_line);
+ if (clip.preview_text) args.push("--preview-text", clip.preview_text);
+ return args;
+}
+
// Candidate headline texts + face frames for the two-step thumbnail picker.
app.get("/api/clips/:id/thumbnail/options", async (req, res) => {
const clip = await clipsHistory.findById(req.params.id);
@@ -2793,6 +2927,7 @@ app.get("/api/clips/:id/thumbnail/options", async (req, res) => {
"--end", String(clip.end_second),
"--texts", String(clamp(req.query.texts, 6)),
"--frames", String(clamp(req.query.frames, 6)),
+ ...thumbnailGroundingArgs(clip),
"--", tc.text || clip.title,
]);
if (r.code !== 0) { res.status(400).json({ error: stripAnsi(r.stderr || r.stdout) || "options failed" }); return; }
@@ -2811,34 +2946,40 @@ app.post("/api/clips/:id/thumbnail/render", async (req, res) => {
const clip = await clipsHistory.findById(req.params.id);
if (!clip) { res.status(404).json({ error: "clip not found" }); return; }
const tc = clip.thumbnail_config || {};
- const { line1, line2, frame_path, frame_info } = req.body || {};
- if (!frame_path) { res.status(400).json({ error: "select a frame first" }); return; }
+ const { line1, line2, frame_path, frame_info, layout, swap } = req.body || {};
+ // The pair layout takes both people from the clip itself; a frame is only
+ // its fallback when two people cannot be told apart.
+ if (!frame_path && needsFrame(layout)) { res.status(400).json({ error: "select a frame first" }); return; }
// Only allow frames podcli itself produced (candidate frames) or the user uploaded —
// never an arbitrary server path passed through to the renderer.
- const resolvedFrame = resolveFrameInRoots(frame_path, [join(paths.output, "thumbnails", String(clip.id)), uploadDir]);
- if (!resolvedFrame) { res.status(400).json({ error: "invalid frame" }); return; }
+ const resolvedFrame = frame_path
+ ? resolveFrameInRoots(frame_path, [join(paths.output, "thumbnails", String(clip.id)), uploadDir])
+ : null;
+ if (frame_path && !resolvedFrame) { res.status(400).json({ error: "invalid frame" }); return; }
const outDir = join(paths.output, "thumbnails", String(clip.id));
await mkdir(outDir, { recursive: true });
const out = join(outDir, `thumb_${uuidv4().slice(0, 8)}.png`);
- const args = ["thumbnail-render", "--frame", resolvedFrame, "--output", out];
- if (line1) args.push(`--line1=${line1}`);
- if (line2) args.push(`--line2=${line2}`);
- if (frame_info) args.push("--frame-info", JSON.stringify(frame_info));
+ const args = thumbnailRenderArgs({
+ output: out, frame: resolvedFrame, frameInfo: frame_info,
+ line1, line2, layout, swap: !!swap, clip,
+ });
+ args.push(...thumbnailGroundingArgs(clip));
args.push("--", tc.text || clip.title);
const r = await runCli(args);
if (r.code !== 0) { res.status(400).json({ error: stripAnsi(r.stderr || r.stdout) || "render failed" }); return; }
- const jsonLine = r.stdout.trim().split("\n").reverse().find((l) => l.trim().startsWith("{"));
- let outPath = "";
- try { outPath = JSON.parse(jsonLine || "{}").path || ""; } catch { /* no path */ }
+ const { path: outPath, report } = parseThumbnailRender(r.stdout);
if (!outPath || !existsSync(outPath)) { res.status(500).json({ error: "no thumbnail produced" }); return; }
if (clip.output_path && existsSync(clip.output_path)) {
const bake = await bakeThumbnailCard(clip.output_path, outPath, tc.card_seconds || 0);
if (!bake.ok) { res.status(500).json({ error: `rendered but bake into clip failed: ${bake.error}` }); return; }
}
- const merged = { ...tc, line1: line1 || undefined, line2: line2 || undefined, preview_path: outPath, card_seconds: 1.5 };
+ const merged = {
+ ...tc, line1: line1 || undefined, line2: line2 || undefined, preview_path: outPath, card_seconds: 1.5,
+ layout: report.layout, people: report.people,
+ };
const edit = await runCli(["clips", "edit", String(clip.id), "--thumbnail-config", JSON.stringify(merged)]);
if (edit.code !== 0) { res.status(500).json({ error: stripAnsi(edit.stderr || edit.stdout) || "thumbnail metadata update failed" }); return; }
- res.json({ ok: true, preview_path: outPath });
+ res.json({ ok: true, preview_path: outPath, ...report });
});
app.get("/api/clips/:id/logo/previews", async (req, res) => {
@@ -3255,6 +3396,12 @@ app.post("/api/clips/:id/rerender", async (req, res) => {
}
// If trimmed wider/narrower, keep only words inside the new bounds.
const words = allWords.filter((w: any) => typeof w?.start !== "number" || (w.start >= startSecond && w.start < endSecond));
+ const recipeSegments = trimOnly ? undefined : (recipe.keep_segments as ExportClip["keep_segments"]);
+ // A trim that no longer holds the hook's passage renders without it.
+ const recipeHook = recipe.hook as ClipHook | undefined;
+ const hook = recipeHook && !validateHook(recipeHook, startSecond, endSecond, recipeSegments)
+ ? recipeHook
+ : undefined;
const recipeAsset = (key: "logo_path" | "outro_path" | "intro_path") => {
const value = recipe[key];
return typeof value === "string" && value.trim() ? value : undefined;
@@ -3274,7 +3421,8 @@ app.post("/api/clips/:id/rerender", async (req, res) => {
clean_fillers: trimOnly ? false : (recipe.clean_fillers !== undefined ? recipe.clean_fillers : true),
trim_opening: trimOnly ? false : undefined,
preserve_timing: trimOnly ? true : undefined,
- ...(trimOnly ? {} : recipe.keep_segments ? { keep_segments: recipe.keep_segments } : {}),
+ ...(recipeSegments ? { keep_segments: recipeSegments } : {}),
+ ...(hook && { hook }),
title: clip.title,
output_dir: dirname(clip.output_path),
});
@@ -4124,6 +4272,7 @@ app.get("/api/ui-state", (_req, res) => {
rawTranscriptText: uiState.rawTranscriptText,
silenceOriginal: uiState.silenceOriginal,
silencePlan: uiState.silencePlan,
+ videoMissing: uiState.videoMissing,
lastUpdated: uiState.lastUpdated,
});
});
@@ -4149,9 +4298,38 @@ app.post("/api/ui-state", (req, res) => {
return;
}
- if (body.videoPath !== undefined) uiState.videoPath = body.videoPath;
+ // Changing the video without a transcript arriving in the same call means
+ // the old transcript, suggestions and selections describe a recording
+ // that's no longer loaded. Carrying them forward lets create_clip burn
+ // captions and timings from the wrong video. transcribe_podcast and
+ // import_transcript always send videoPath and transcript together, so
+ // this only fires for a bare set_video.
+ const videoChanged =
+ body.videoPath !== undefined &&
+ body.videoPath !== uiState.videoPath &&
+ body.transcript === undefined;
+ if (videoChanged) {
+ uiState.transcript = null;
+ uiState.transcriptVideoIdentity = null;
+ uiState.rawTranscriptText = "";
+ uiState.suggestions = [];
+ uiState.deselectedIndices = [];
+ uiState.energyData = {};
+ uiState.silenceOriginal = null;
+ uiState.silencePlan = null;
+ }
+
+ if (body.videoPath !== undefined) {
+ uiState.videoPath = body.videoPath;
+ uiState.videoMissing = !!body.videoPath && !existsSync(body.videoPath);
+ }
if (body.filePath !== undefined) uiState.filePath = body.filePath;
- if (body.transcript !== undefined) uiState.transcript = body.transcript;
+ if (body.transcript !== undefined) {
+ uiState.transcript = body.transcript;
+ uiState.transcriptVideoIdentity = body.transcript
+ ? computeVideoIdentity(body.videoPath ?? uiState.videoPath ?? "")
+ : null;
+ }
if (body.rawTranscriptText !== undefined)
uiState.rawTranscriptText = body.rawTranscriptText;
if (body.silenceOriginal !== undefined) {
@@ -4176,20 +4354,51 @@ app.post("/api/ui-state", (req, res) => {
}
if (body.suggestions !== undefined) {
if (body._source === "ui" && Array.isArray(body.suggestions)) {
+ const previousSuggestions = uiState.suggestions;
uiState.suggestions = body.suggestions.map((incoming: SuggestedClip) => {
- if (incoming.segments?.length) return incoming;
- const segments = findSuggestionSegments(
- uiState.suggestions,
- incoming.start_second,
- incoming.end_second,
- );
- if (!segments?.length) return incoming;
- const existing = uiState.suggestions.find(
+ const existingIndex = previousSuggestions.findIndex(
(s) =>
Math.abs(s.start_second - incoming.start_second) < 0.5 &&
Math.abs(s.end_second - incoming.end_second) < 0.5,
);
- return { ...incoming, segments, duration: existing?.duration ?? incoming.duration };
+ const existing = existingIndex >= 0 ? previousSuggestions[existingIndex] : undefined;
+
+ let merged = incoming;
+ if (!incoming.segments?.length) {
+ const segments = findSuggestionSegments(
+ previousSuggestions,
+ incoming.start_second,
+ incoming.end_second,
+ );
+ if (segments?.length) {
+ // The restored segments (and any hook) change what actually
+ // plays, so the duration has to be recomputed rather than
+ // carried over from before the edit. Otherwise a clip with an
+ // opening hook reports a duration that excludes it.
+ merged = {
+ ...incoming,
+ segments,
+ duration:
+ Math.round(
+ playbackDuration(incoming.start_second, incoming.end_second, segments, incoming.hook) * 10,
+ ) / 10,
+ };
+ }
+ }
+
+ // A studio edit (retiming, segment trim, hook change, ...) lands here
+ // the same way MCP's modify_clip lands on /api/suggestions/modify;
+ // an already-approved clip needs the same "no longer matches what
+ // was selected" flag, or a studio edit after selection silently
+ // exports something other than what was approved.
+ if (existing?.selectionHash && !uiState.deselectedIndices.includes(existingIndex)) {
+ const currentHash = computeSelectionHash(merged, uiState.transcript?.words);
+ if (currentHash !== existing.selectionHash) {
+ merged = { ...merged, selectionHash: existing.selectionHash, changedSinceSelection: true };
+ }
+ }
+
+ return merged;
});
} else {
uiState.suggestions = body.suggestions;
@@ -4241,12 +4450,17 @@ app.post("/api/ui-state", (req, res) => {
broadcastSSE("state-sync", {
...(body.videoPath !== undefined && { videoPath: uiState.videoPath }),
...(body.filePath !== undefined && { filePath: uiState.filePath }),
- ...(body.suggestions !== undefined && { suggestions: uiState.suggestions }),
- ...(body.deselectedIndices !== undefined && {
+ // The request body only carried videoPath, but a bare set_video also
+ // cleared the transcript/suggestions/selections server-side; without
+ // forcing these onto the broadcast too, the studio's in-memory state
+ // never learns they were cleared and later syncs the stale ones right
+ // back.
+ ...((body.suggestions !== undefined || videoChanged) && { suggestions: uiState.suggestions }),
+ ...((body.deselectedIndices !== undefined || videoChanged) && {
deselectedIndices: uiState.deselectedIndices,
}),
...(body.phase !== undefined && { phase: uiState.phase }),
- ...(body.transcript !== undefined && { transcript: uiState.transcript }),
+ ...((body.transcript !== undefined || videoChanged) && { transcript: uiState.transcript }),
...(body.silenceOriginal !== undefined && { silenceOriginal: uiState.silenceOriginal }),
...(body.silencePlan !== undefined && { silencePlan: uiState.silencePlan }),
...(body.settings && { settings: uiState.settings }),
@@ -4295,12 +4509,20 @@ app.post("/api/suggestions/modify", (req, res) => {
res.status(400).json({ error: "selected must be a boolean for action 'toggle'" });
return;
}
+ clip = uiState.suggestions[index];
if (selected) {
uiState.deselectedIndices = uiState.deselectedIndices.filter((i) => i !== index);
- } else if (!uiState.deselectedIndices.includes(index)) {
- uiState.deselectedIndices = [...uiState.deselectedIndices, index];
+ // Stamp what's being approved so a later edit to the same clip can be
+ // caught instead of silently rendering something else.
+ clip.selectionHash = computeSelectionHash(clip, uiState.transcript?.words);
+ clip.changedSinceSelection = false;
+ } else {
+ if (!uiState.deselectedIndices.includes(index)) {
+ uiState.deselectedIndices = [...uiState.deselectedIndices, index];
+ }
+ delete clip.selectionHash;
+ delete clip.changedSinceSelection;
}
- clip = uiState.suggestions[index];
} else if (action === "update") {
const upd = updates || {};
clip = uiState.suggestions[index];
@@ -4311,8 +4533,31 @@ app.post("/api/suggestions/modify", (req, res) => {
res.status(400).json({ error: rangeError });
return;
}
+ const reTimed = nextStart !== clip.start_second || nextEnd !== clip.end_second;
+ // The old segments array is scoped to the old range; if left stale it
+ // overrides the new start/end at render time (create_clip reads
+ // keep_segments ahead of start_second/end_second).
+ const nextSegments = reTimed
+ ? reconcileSegmentsForRange(clip.segments, clip.start_second, clip.end_second, nextStart, nextEnd)
+ : clip.segments;
+ const nextHook: ClipHook | undefined =
+ upd.hook === null ? undefined : upd.hook !== undefined ? upd.hook : clip.hook;
+ const hookError = validateHook(nextHook, nextStart, nextEnd, nextSegments);
+ if (hookError) {
+ res.status(400).json({
+ error: upd.hook === undefined
+ ? `${hookError} The new range no longer holds this clip's hook. Pass hook: null to clear it, or a new hook.`
+ : hookError,
+ });
+ return;
+ }
// The energy score was measured over the old range.
- if (nextStart !== clip.start_second || nextEnd !== clip.end_second) dropEnergy(clip);
+ if (reTimed) {
+ dropEnergy(clip);
+ clip.segments = nextSegments;
+ }
+ if (nextHook) clip.hook = { start: nextHook.start, end: nextHook.end, mode: nextHook.mode };
+ else delete clip.hook;
if (typeof upd.title === "string") clip.title = upd.title;
clip.start_second = nextStart;
clip.end_second = nextEnd;
@@ -4324,10 +4569,21 @@ app.post("/api/suggestions/modify", (req, res) => {
if (typeof upd.suggested_caption_style === "string") {
clip.suggested_caption_style = upd.suggested_caption_style;
}
- clip.duration = Math.round((clip.end_second - clip.start_second) * 10) / 10;
+ clip.duration =
+ Math.round(playbackDuration(clip.start_second, clip.end_second, clip.segments, clip.hook) * 10) / 10;
const fmtTime = (s: number) =>
`${Math.floor(s / 60)}:${Math.floor(s % 60).toString().padStart(2, "0")}`;
clip.timestamp_display = `${fmtTime(clip.start_second)} → ${fmtTime(clip.end_second)}`;
+
+ // Approval was for a specific render; if this edit changed any
+ // render-relevant field on an already-selected clip, flag it instead of
+ // silently exporting something other than what got approved.
+ if (clip.selectionHash && !uiState.deselectedIndices.includes(index)) {
+ const currentHash = computeSelectionHash(clip, uiState.transcript?.words);
+ if (currentHash !== clip.selectionHash) {
+ clip.changedSinceSelection = true;
+ }
+ }
} else {
res.status(400).json({ error: `Unknown action: ${action}` });
return;
@@ -4383,6 +4639,7 @@ app.post("/api/mcp/export", async (req, res) => {
? req.body.clean_fillers !== false
: uiState.settings.cleanFillers !== false;
const keepCaptionOverlay = req.body.keep_caption_overlay === true;
+ const writeCleanVariant = req.body.write_clean_variant === true;
if (!videoPath || !existsSync(videoPath)) {
res.status(400).json({ error: "Video file not found" });
@@ -4405,7 +4662,7 @@ app.post("/api/mcp/export", async (req, res) => {
// Apply style settings to clips that don't have their own
const styledClips = clips.map((c: any) =>
- enrichClipWithSegments({
+ enrichClipFromSuggestion({
start_second: c.start_second,
end_second: c.end_second,
title: c.title || "clip",
@@ -4418,8 +4675,14 @@ app.post("/api/mcp/export", async (req, res) => {
(Array.isArray(c.keep_segments) && c.keep_segments.length > 0 && c.keep_segments) ||
(Array.isArray(c.keep_segment) && c.keep_segment.length > 0 && c.keep_segment) ||
undefined,
+ hook: c.hook,
}),
);
+ const mcpHookError = exportHookError(styledClips);
+ if (mcpHookError) {
+ res.status(400).json({ error: mcpHookError });
+ return;
+ }
const jobId = uuidv4();
const job: JobState = {
@@ -4466,6 +4729,7 @@ app.post("/api/mcp/export", async (req, res) => {
intro_path: introPath,
clean_fillers: cleanFillers,
keep_caption_overlay: keepCaptionOverlay,
+ write_clean_variant: writeCleanVariant,
face_map: uiState.transcript?.face_map,
},
(event) => {
@@ -4493,6 +4757,7 @@ app.post("/api/mcp/export", async (req, res) => {
job.progress = 100;
job.message = "Export complete!";
job.result = data;
+ uiState.results = data?.results ?? [];
// Record clips to history
try {
await historyRecorder.recordRemaining(result.data?.results);
diff --git a/src/utils/captions.test.ts b/src/utils/captions.test.ts
new file mode 100644
index 0000000..0732876
--- /dev/null
+++ b/src/utils/captions.test.ts
@@ -0,0 +1,190 @@
+import { describe, it, expect } from "vitest";
+import { selectBestCaptionTrack, parseVtt, parseJson3, cuesToWords } from "./captions.js";
+
+describe("selectBestCaptionTrack", () => {
+ it("prefers manual subtitles over automatic captions", () => {
+ const track = selectBestCaptionTrack(
+ { en: [{ ext: "vtt", url: "manual-en" }] },
+ { en: [{ ext: "vtt", url: "auto-en" }] },
+ );
+ expect(track).toEqual({ lang: "en", kind: "manual", ext: "vtt", url: "manual-en" });
+ });
+
+ it("prefers an -orig track over a plain language code (translated)", () => {
+ const track = selectBestCaptionTrack(null, {
+ ka: [{ ext: "vtt", url: "translated-ka" }],
+ "ka-orig": [{ ext: "vtt", url: "original-ka" }],
+ en: [{ ext: "vtt", url: "translated-en" }],
+ });
+ expect(track?.lang).toBe("ka-orig");
+ expect(track?.url).toBe("original-ka");
+ });
+
+ it("prefers json3 over vtt, since it times each word", () => {
+ const track = selectBestCaptionTrack(null, {
+ en: [
+ { ext: "vtt", url: "vtt-url" },
+ { ext: "json3", url: "json3-url" },
+ ],
+ });
+ expect(track?.url).toBe("json3-url");
+ });
+
+ it("prefers the vtt track when json3 is not offered", () => {
+ const track = selectBestCaptionTrack(null, {
+ en: [
+ { ext: "srv3", url: "srv3-url" },
+ { ext: "vtt", url: "vtt-url" },
+ { ext: "ttml", url: "ttml-url" },
+ ],
+ });
+ expect(track?.url).toBe("vtt-url");
+ });
+
+ it("returns null when there are no tracks at all", () => {
+ expect(selectBestCaptionTrack(null, null)).toBeNull();
+ expect(selectBestCaptionTrack({}, {})).toBeNull();
+ });
+});
+
+describe("parseVtt", () => {
+ it("parses cues with timestamps and text", () => {
+ const vtt = [
+ "WEBVTT",
+ "",
+ "00:00:01.000 --> 00:00:04.000",
+ "Hello world",
+ "",
+ "00:00:04.000 --> 00:00:07.500",
+ "second line here",
+ "",
+ ].join("\n");
+ const cues = parseVtt(vtt);
+ expect(cues).toEqual([
+ { start: 1, end: 4, text: "Hello world" },
+ { start: 4, end: 7.5, text: "second line here" },
+ ]);
+ });
+
+ it("strips inline word-highlight tags auto-captions add", () => {
+ const vtt = [
+ "WEBVTT",
+ "",
+ "00:00:01.000 --> 00:00:04.000 align:start position:0%",
+ "<00:00:01.000><c> Hello</c><00:00:01.500><c> world</c>",
+ "",
+ ].join("\n");
+ const cues = parseVtt(vtt);
+ expect(cues).toEqual([{ start: 1, end: 4, text: "Hello world" }]);
+ });
+
+ it("handles an hour component in timestamps", () => {
+ const vtt = ["WEBVTT", "", "01:00:01.000 --> 01:00:04.000", "late line", ""].join("\n");
+ const cues = parseVtt(vtt);
+ expect(cues[0].start).toBe(3601);
+ expect(cues[0].end).toBe(3604);
+ });
+
+ it("returns no cues for text with no timestamp lines", () => {
+ expect(parseVtt("WEBVTT\n\njust some text\n")).toEqual([]);
+ });
+});
+
+describe("cuesToWords", () => {
+ it("spaces words evenly across the cue, each word's end is the next word's start", () => {
+ const words = cuesToWords([{ start: 0, end: 4, text: "one two three four" }]);
+ expect(words).toEqual([
+ { word: "one", start: 0, end: 1, confidence: 1 },
+ { word: "two", start: 1, end: 2, confidence: 1 },
+ { word: "three", start: 2, end: 3, confidence: 1 },
+ { word: "four", start: 3, end: 4, confidence: 1 },
+ ]);
+ });
+
+ it("caps the last word of a cue at the cue's own end rather than leaving it open", () => {
+ const words = cuesToWords([{ start: 10, end: 11, text: "ok" }]);
+ expect(words.at(-1)?.end).toBe(11);
+ });
+
+ it("drops a rolling auto-caption cue that is a prefix of the next one", () => {
+ const words = cuesToWords([
+ { start: 0, end: 1, text: "hello" },
+ { start: 0, end: 2, text: "hello world" },
+ { start: 2, end: 4, text: "next sentence" },
+ ]);
+ expect(words.map((w) => w.word)).toEqual(["hello", "world", "next", "sentence"]);
+ });
+
+ it("keeps each line once from YouTube's rolling auto-captions", () => {
+ const vtt = [
+ "WEBVTT",
+ "",
+ "00:00:00.080 --> 00:00:02.070 align:start position:0%",
+ " ",
+ "It's<00:00:00.400><c> a</c><00:00:00.800><c> great</c><00:00:01.120><c> pleasure</c>",
+ "",
+ "00:00:02.070 --> 00:00:02.080 align:start position:0%",
+ "It's a great pleasure",
+ " ",
+ "",
+ "00:00:02.080 --> 00:00:04.550 align:start position:0%",
+ "It's a great pleasure",
+ "today<00:00:02.399><c> and</c><00:00:02.639><c> I'm</c><00:00:02.879><c> excited</c>",
+ "",
+ "00:00:04.550 --> 00:00:04.560 align:start position:0%",
+ "today and I'm excited",
+ " ",
+ "",
+ "00:00:04.560 --> 00:00:06.950 align:start position:0%",
+ "today and I'm excited",
+ "&gt;&gt; Definitely.<00:00:05.680><c> Yeah.</c>",
+ ].join("\n");
+ expect(cuesToWords(parseVtt(vtt)).map((w) => w.word)).toEqual([
+ "It's", "a", "great", "pleasure", "today", "and", "I'm", "excited", "Definitely.", "Yeah.",
+ ]);
+ });
+
+ it("concatenates words across multiple distinct cues in order", () => {
+ const words = cuesToWords([
+ { start: 0, end: 2, text: "first cue" },
+ { start: 2, end: 4, text: "second cue" },
+ ]);
+ expect(words.map((w) => w.word)).toEqual(["first", "cue", "second", "cue"]);
+ });
+
+ it("returns an empty array for no cues", () => {
+ expect(cuesToWords([])).toEqual([]);
+ });
+});
+
+describe("parseJson3", () => {
+ it("times each word from its own offset and drops speaker-change marks", () => {
+ const raw = JSON.stringify({
+ events: [
+ { tStartMs: 80, dDurationMs: 2000, segs: [{ utf8: "It's" }, { utf8: " a", tOffsetMs: 320 }, { utf8: " great", tOffsetMs: 720 }] },
+ { tStartMs: 2070, dDurationMs: 10, aAppend: 1, segs: [{ utf8: "\n" }] },
+ { tStartMs: 2080, dDurationMs: 1000, segs: [{ utf8: ">> Definitely." }] },
+ ],
+ });
+ const { words, segments } = parseJson3(raw);
+ expect(words.map((w) => w.word)).toEqual(["It's", "a", "great", "Definitely."]);
+ expect(words[1].start).toBeCloseTo(0.4);
+ expect(words[0].end).toBeCloseTo(0.4);
+ expect(segments).toHaveLength(2);
+ });
+
+ it("orders words from overlapping events and never lets one run past the next", () => {
+ const raw = JSON.stringify({
+ events: [
+ { tStartMs: 0, dDurationMs: 3000, segs: [{ utf8: "Okay." }] },
+ { tStartMs: 1000, dDurationMs: 1000, segs: [{ utf8: "Something" }, { utf8: " like", tOffsetMs: 300 }] },
+ ],
+ });
+ const { words } = parseJson3(raw);
+ expect(words.map((w) => w.word)).toEqual(["Okay.", "Something", "like"]);
+ words.forEach((w, i) => {
+ expect(w.end).toBeGreaterThanOrEqual(w.start);
+ if (i > 0) expect(w.start).toBeGreaterThanOrEqual(words[i - 1].end);
+ });
+ });
+});
diff --git a/src/utils/captions.ts b/src/utils/captions.ts
new file mode 100644
index 0000000..bf34d4e
--- /dev/null
+++ b/src/utils/captions.ts
@@ -0,0 +1,182 @@
+import type { WordTimestamp } from "../models/index.js";
+
+export interface CaptionCue {
+ start: number;
+ end: number;
+ text: string;
+}
+
+export interface CaptionTrackRef {
+ ext: string;
+ url: string;
+}
+
+export interface SelectedCaptionTrack {
+ lang: string;
+ kind: "manual" | "auto";
+ ext: string;
+ url: string;
+}
+
+/**
+ * Pick the caption track worth mining: a manual (human-written) track over
+ * an auto-generated one, and within either, the track's own original-language
+ * variant (yt-dlp exposes it as `<lang>-orig`, e.g. "ka-orig") over a plain
+ * language code, which is usually YouTube's auto-translation of some other
+ * track and drifts from what was actually said.
+ */
+export function selectBestCaptionTrack(
+ subtitles: Record<string, CaptionTrackRef[]> | undefined | null,
+ automaticCaptions: Record<string, CaptionTrackRef[]> | undefined | null,
+): SelectedCaptionTrack | null {
+ const pick = (
+ dict: Record<string, CaptionTrackRef[]> | undefined | null,
+ kind: "manual" | "auto",
+ ): SelectedCaptionTrack | null => {
+ if (!dict) return null;
+ const langs = Object.keys(dict);
+ if (langs.length === 0) return null;
+ const lang = langs.find((l) => l.endsWith("-orig")) ?? langs[0];
+ const tracks = dict[lang];
+ if (!tracks?.length) return null;
+ // json3 carries a start time per word; vtt only per cue.
+ const best = tracks.find((t) => t.ext === "json3") ?? tracks.find((t) => t.ext === "vtt") ?? tracks[0];
+ return { lang, kind, ext: best.ext, url: best.url };
+ };
+ return pick(subtitles, "manual") ?? pick(automaticCaptions, "auto");
+}
+
+/**
+ * Parse WebVTT cue text into {start, end, text}. Strips inline tags
+ * (<c>, <00:00:01.000>, voice spans) that auto-captions use for
+ * word-by-word highlighting. podcli derives its own word timing from the
+ * cue span instead, so these would only be noise in the text.
+ */
+export function parseVtt(vtt: string): CaptionCue[] {
+ const timeToSeconds = (t: string): number => {
+ const m = t.trim().match(/(?:(\d+):)?(\d+):(\d+)\.(\d+)/);
+ if (!m) return 0;
+ const [, h, mm, ss, ms] = m;
+ return (Number(h) || 0) * 3600 + Number(mm) * 60 + Number(ss) + Number(ms) / 1000;
+ };
+ const stripTags = (s: string): string => s.replace(/<[^>]*>/g, "").trim();
+
+ const blocks = vtt.replace(/\r\n/g, "\n").split(/\n\n+/);
+ const cues: CaptionCue[] = [];
+ for (const block of blocks) {
+ const lines = block.split("\n").filter((l) => l.trim().length > 0);
+ const timeLineIdx = lines.findIndex((l) => l.includes("-->"));
+ if (timeLineIdx === -1) continue;
+ const [startRaw, endRaw] = lines[timeLineIdx].split("-->");
+ const start = timeToSeconds(startRaw);
+ const end = timeToSeconds(endRaw.trim().split(/\s+/)[0] ?? endRaw);
+ const text = stripTags(lines.slice(timeLineIdx + 1).join(" "));
+ if (!text) continue;
+ cues.push({ start, end, text });
+ }
+ return cues;
+}
+
+/**
+ * One cue's words, evenly spaced across [cue.start, cue.end]. Each word's
+ * end is the next word's start; the cue's own final word is capped at
+ * cue.end rather than left open, since a caption gives no finer timing
+ * than the cue span itself.
+ */
+function cueToWords(cue: CaptionCue): WordTimestamp[] {
+ const words = captionTokens(cue.text);
+ if (words.length === 0) return [];
+ const step = (cue.end - cue.start) / words.length;
+ return words.map((word, i) => ({
+ word,
+ start: cue.start + i * step,
+ end: i === words.length - 1 ? cue.end : cue.start + (i + 1) * step,
+ // Captions carry no ASR confidence score; 1 marks it as "given", not
+ // "measured", so nothing downstream mistakes it for a low-confidence word.
+ confidence: 1,
+ }));
+}
+
+/**
+ * Convert caption cues into podcli's word-level transcript format.
+ *
+ * YouTube auto-captions roll: each cue repeats the previous cue's last line
+ * above the new one, and a ~10 ms cue between them repeats it once more.
+ * Kept verbatim, every line appears three times. Drop the carry cues, then
+ * strip from each cue the words it repeats from the end of the text so far.
+ * Manual tracks have no such overlap and pass through unchanged.
+ */
+export function cuesToWords(cues: CaptionCue[]): WordTimestamp[] {
+ const kept: CaptionCue[] = [];
+ let tail: string[] = [];
+ for (const cue of cues) {
+ if (cue.end - cue.start < 0.05) continue;
+ const words = captionTokens(cue.text);
+ let overlap = Math.min(words.length, tail.length);
+ while (overlap > 0 && tail.slice(-overlap).join(" ") !== words.slice(0, overlap).join(" ")) overlap--;
+ const fresh = words.slice(overlap);
+ tail = [...tail, ...fresh].slice(-64);
+ if (fresh.length) kept.push({ ...cue, text: fresh.join(" ") });
+ }
+ return kept.flatMap(cueToWords);
+}
+
+interface Json3Event {
+ tStartMs?: number;
+ dDurationMs?: number;
+ segs?: Array<{ utf8?: string; tOffsetMs?: number }>;
+}
+
+/**
+ * Caption text as spoken words: HTML entities decoded and YouTube's ">>"
+ * speaker-change marks dropped, since they are layout, not speech.
+ */
+export function captionTokens(text: string): string[] {
+ return text
+ .replace(/&gt;/g, ">")
+ .replace(/&lt;/g, "<")
+ .replace(/&quot;/g, '"')
+ .replace(/&#39;/g, "'")
+ .replace(/&amp;/g, "&")
+ .split(/\s+/)
+ .filter((t) => t && !/^>+$/.test(t));
+}
+
+/**
+ * Words and segments from YouTube's json3 caption format. Auto-captions give
+ * each word its own offset, so timing comes from the source rather than being
+ * spread evenly across a cue. Events can overlap (a speaker change opens a new
+ * event before the old one closes), so words are ordered by start and each
+ * ends where the next begins.
+ */
+export function parseJson3(raw: string): { words: WordTimestamp[]; segments: CaptionCue[] } {
+ const events = (JSON.parse(raw) as { events?: Json3Event[] }).events ?? [];
+ const words: WordTimestamp[] = [];
+ const segments: CaptionCue[] = [];
+ for (const ev of events) {
+ const evStart = (ev.tStartMs ?? 0) / 1000;
+ const evEnd = evStart + (ev.dDurationMs ?? 0) / 1000;
+ const segs = (ev.segs ?? []).map((seg) => ({
+ tokens: captionTokens(seg.utf8 ?? ""),
+ start: evStart + (seg.tOffsetMs ?? 0) / 1000,
+ }));
+ const evWords: WordTimestamp[] = [];
+ segs.forEach((seg, i) => {
+ const segEnd = Math.max(seg.start, Math.min(segs[i + 1]?.start ?? evEnd, evEnd));
+ const step = (segEnd - seg.start) / Math.max(seg.tokens.length, 1);
+ seg.tokens.forEach((word, k) =>
+ evWords.push({ word, start: seg.start + k * step, end: seg.start + (k + 1) * step, confidence: 1 }),
+ );
+ });
+ if (!evWords.length) continue;
+ words.push(...evWords);
+ segments.push({ start: evStart, end: evEnd, text: evWords.map((w) => w.word).join(" ") });
+ }
+ words.sort((a, b) => a.start - b.start);
+ words.forEach((w, i) => {
+ const next = words[i + 1];
+ if (next && w.end > next.start) w.end = next.start;
+ });
+ segments.sort((a, b) => a.start - b.start);
+ return { words, segments };
+}
diff --git a/src/utils/clip-hook.test.ts b/src/utils/clip-hook.test.ts
new file mode 100644
index 0000000..05317aa
--- /dev/null
+++ b/src/utils/clip-hook.test.ts
@@ -0,0 +1,79 @@
+import { describe, it, expect } from "vitest";
+import { playbackDuration, playbackRanges, validateHook } from "./clip-hook.js";
+
+describe("validateHook", () => {
+ it("accepts no hook", () => {
+ expect(validateHook(undefined, 10, 40)).toBeNull();
+ expect(validateHook(null, 10, 40)).toBeNull();
+ });
+
+ it("accepts a hook inside the range, edges included", () => {
+ expect(validateHook({ start: 10, end: 12, mode: "repeat" }, 10, 40)).toBeNull();
+ expect(validateHook({ start: 38, end: 40, mode: "move" }, 10, 40)).toBeNull();
+ });
+
+ it("rejects an end at or before the start", () => {
+ expect(validateHook({ start: 15, end: 15, mode: "repeat" }, 10, 40)).toMatch(/greater than/);
+ });
+
+ it("rejects hooks under 1s or over 15s", () => {
+ expect(validateHook({ start: 12, end: 12.5, mode: "repeat" }, 10, 40)).toMatch(/between 1 and 15/);
+ expect(validateHook({ start: 12, end: 28, mode: "repeat" }, 10, 40)).toMatch(/between 1 and 15/);
+ });
+
+ it("rejects a hook outside the range", () => {
+ expect(validateHook({ start: 8, end: 11, mode: "repeat" }, 10, 40)).toMatch(/not inside/);
+ });
+
+ it("rejects a hook in a gap between segments", () => {
+ const segments = [{ start: 10, end: 15 }, { start: 20, end: 30 }];
+ expect(validateHook({ start: 14, end: 17, mode: "repeat" }, 10, 30, segments)).toMatch(/not inside/);
+ expect(validateHook({ start: 21, end: 24, mode: "repeat" }, 10, 30, segments)).toBeNull();
+ });
+
+ it("rejects a bad mode or non-numeric bounds", () => {
+ expect(validateHook({ start: 12, end: 14, mode: "loop" }, 10, 40)).toMatch(/repeat/);
+ expect(validateHook({ start: 12, end: 14 }, 10, 40)).toMatch(/repeat/);
+ expect(validateHook({ start: "12", end: 14, mode: "move" }, 10, 40)).toMatch(/numbers/);
+ expect(validateHook("12-14", 10, 40)).toMatch(/object/);
+ });
+
+ it("rejects a move hook that would take the whole clip", () => {
+ expect(validateHook({ start: 10, end: 14, mode: "move" }, 10, 14)).toMatch(/whole clip/);
+ expect(validateHook({ start: 10, end: 14, mode: "repeat" }, 10, 14)).toBeNull();
+ });
+});
+
+describe("playbackRanges", () => {
+ const segments = [{ start: 10, end: 20 }, { start: 22, end: 30 }];
+
+ it("is the body when there is no hook", () => {
+ expect(playbackRanges(10, 30, segments)).toEqual(segments);
+ expect(playbackRanges(10, 30)).toEqual([{ start: 10, end: 30 }]);
+ });
+
+ it("puts a repeat hook first and keeps the body whole", () => {
+ expect(playbackRanges(10, 30, segments, { start: 24, end: 27, mode: "repeat" })).toEqual([
+ { start: 24, end: 27 },
+ { start: 10, end: 20 },
+ { start: 22, end: 30 },
+ ]);
+ });
+
+ it("puts a move hook first and cuts it from the body", () => {
+ expect(playbackRanges(10, 30, segments, { start: 24, end: 27, mode: "move" })).toEqual([
+ { start: 24, end: 27 },
+ { start: 10, end: 20 },
+ { start: 22, end: 24 },
+ { start: 27, end: 30 },
+ ]);
+ });
+});
+
+describe("playbackDuration", () => {
+ it("adds a repeat hook's length and leaves a move hook's total unchanged", () => {
+ expect(playbackDuration(10, 30)).toBe(20);
+ expect(playbackDuration(10, 30, undefined, { start: 12, end: 15, mode: "repeat" })).toBe(23);
+ expect(playbackDuration(10, 30, undefined, { start: 12, end: 15, mode: "move" })).toBe(20);
+ });
+});
diff --git a/src/utils/clip-hook.ts b/src/utils/clip-hook.ts
new file mode 100644
index 0000000..e51c16b
--- /dev/null
+++ b/src/utils/clip-hook.ts
@@ -0,0 +1,109 @@
+/**
+ * Opening hook: a short spoken passage from inside a clip, played first.
+ * backend/services/opening_hook.py renders it; this mirrors its rules so the
+ * MCP tools and the studio reject the same hooks and show the same order.
+ *
+ * No imports, so the studio client can bundle it as is.
+ */
+
+export type HookMode = "repeat" | "move";
+
+export interface ClipHook {
+ start: number;
+ end: number;
+ mode: HookMode;
+}
+
+export interface TimeRange {
+ start: number;
+ end: number;
+}
+
+export const MIN_HOOK_SECONDS = 1;
+export const MAX_HOOK_SECONDS = 15;
+
+// Matches _EDGE_TOLERANCE and _MIN_REMNANT_SECONDS in opening_hook.py.
+const EDGE_TOLERANCE = 0.01;
+const MIN_REMNANT_SECONDS = 0.1;
+
+function bodyRanges(start: number, end: number, segments?: TimeRange[] | null): TimeRange[] {
+ const body = (segments ?? []).filter((s) => s.end > s.start);
+ return body.length ? body : [{ start, end }];
+}
+
+function coveringSpans(ranges: TimeRange[]): TimeRange[] {
+ const spans: TimeRange[] = [];
+ for (const r of [...ranges].sort((a, b) => a.start - b.start)) {
+ const last = spans[spans.length - 1];
+ if (last && r.start <= last.end + EDGE_TOLERANCE) last.end = Math.max(last.end, r.end);
+ else spans.push({ ...r });
+ }
+ return spans;
+}
+
+/** Returns an error message, or null when the hook is renderable (or absent). */
+export function validateHook(
+ hook: unknown,
+ start: number,
+ end: number,
+ segments?: TimeRange[] | null,
+): string | null {
+ if (hook === undefined || hook === null) return null;
+ if (typeof hook !== "object") return "hook must be an object with start, end and mode";
+ const { start: hs, end: he, mode } = hook as Record<string, unknown>;
+ if (typeof hs !== "number" || !Number.isFinite(hs) || typeof he !== "number" || !Number.isFinite(he)) {
+ return "hook start and end must be numbers of seconds";
+ }
+ if (mode !== "repeat" && mode !== "move") return 'hook mode must be "repeat" or "move"';
+ if (he <= hs) return "hook end must be greater than hook start";
+ const length = he - hs;
+ if (length < MIN_HOOK_SECONDS || length > MAX_HOOK_SECONDS) {
+ return `hook runs ${length.toFixed(1)}s. It must run between ${MIN_HOOK_SECONDS} and ${MAX_HOOK_SECONDS} seconds.`;
+ }
+ const inside = coveringSpans(bodyRanges(start, end, segments)).some(
+ (s) => s.start - EDGE_TOLERANCE <= hs && he <= s.end + EDGE_TOLERANCE,
+ );
+ if (!inside) {
+ return `hook ${hs.toFixed(2)}-${he.toFixed(2)}s is not inside the clip body. Pick a passage the clip already plays.`;
+ }
+ if (mode === "move" && playbackRanges(start, end, segments, { start: hs, end: he, mode }).length < 2) {
+ return "A move hook cannot take the whole clip. Use repeat, or widen the clip.";
+ }
+ return null;
+}
+
+/**
+ * The ranges a clip plays, in order. Without a hook that is its body. With
+ * one, the hook comes first; repeat keeps the body whole and move cuts the
+ * hook out of it. The renderer may still tighten the body; this is the plan.
+ */
+export function playbackRanges(
+ start: number,
+ end: number,
+ segments?: TimeRange[] | null,
+ hook?: ClipHook | null,
+): TimeRange[] {
+ const body = bodyRanges(start, end, segments).map((s) => ({ start: s.start, end: s.end }));
+ if (!hook) return body;
+ const head = { start: hook.start, end: hook.end };
+ if (hook.mode === "repeat") return [head, ...body];
+ const rest: TimeRange[] = [];
+ for (const s of body) {
+ for (const [a, b] of [
+ [s.start, Math.min(s.end, hook.start)],
+ [Math.max(s.start, hook.end), s.end],
+ ]) {
+ if (b - a >= MIN_REMNANT_SECONDS) rest.push({ start: a, end: b });
+ }
+ }
+ return [head, ...rest];
+}
+
+export function playbackDuration(
+ start: number,
+ end: number,
+ segments?: TimeRange[] | null,
+ hook?: ClipHook | null,
+): number {
+ return playbackRanges(start, end, segments, hook).reduce((sum, r) => sum + (r.end - r.start), 0);
+}
diff --git a/src/utils/selection-hash.test.ts b/src/utils/selection-hash.test.ts
new file mode 100644
index 0000000..626f434
--- /dev/null
+++ b/src/utils/selection-hash.test.ts
@@ -0,0 +1,104 @@
+import { describe, it, expect } from "vitest";
+import { computeSelectionHash } from "./selection-hash.js";
+
+function clip(overrides: Partial<Parameters<typeof computeSelectionHash>[0]> = {}) {
+ return {
+ start_second: 10,
+ end_second: 20,
+ segments: undefined,
+ suggested_caption_style: "hormozi",
+ title: "A title",
+ ...overrides,
+ };
+}
+
+const words = [
+ { word: "one", start: 9, end: 9.5, confidence: 1 },
+ { word: "two", start: 10, end: 10.5, confidence: 1 },
+ { word: "three", start: 15, end: 15.5, confidence: 1 },
+ { word: "four", start: 20, end: 20.5, confidence: 1 },
+];
+
+describe("computeSelectionHash", () => {
+ it("is stable for the same clip and words", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash(clip(), words);
+ expect(a).toBe(b);
+ });
+
+ it("changes when the range changes", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash(clip({ end_second: 25 }), words);
+ expect(a).not.toBe(b);
+ });
+
+ it("changes when the caption style changes", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash(clip({ suggested_caption_style: "karaoke" }), words);
+ expect(a).not.toBe(b);
+ });
+
+ it("changes when the title changes", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash(clip({ title: "A different title" }), words);
+ expect(a).not.toBe(b);
+ });
+
+ it("changes when segments change", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash(
+ clip({ segments: [{ start: 10, end: 15 }] }),
+ words,
+ );
+ expect(a).not.toBe(b);
+ });
+
+ it("changes when the transcript words inside the range change", () => {
+ const a = computeSelectionHash(clip(), words);
+ const editedWords = words.map((w) =>
+ w.word === "three" ? { ...w, word: "THREE-EDITED" } : w,
+ );
+ const b = computeSelectionHash(clip(), editedWords);
+ expect(a).not.toBe(b);
+ });
+
+ it("ignores words outside the clip range", () => {
+ const a = computeSelectionHash(clip(), words);
+ const extraOutside = [...words, { word: "far-away", start: 500, end: 501, confidence: 1 }];
+ // "far-away" is outside [10, 20) only because of its start time. Adding
+ // a word inside the range should matter, outside should not.
+ const b = computeSelectionHash(clip(), extraOutside);
+ expect(a).toBe(b);
+ });
+
+ it("is unaffected by a hook field that isn't set", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash({ ...clip(), hook: undefined }, words);
+ expect(a).toBe(b);
+ });
+
+ it("changes when a hook is added", () => {
+ const a = computeSelectionHash(clip(), words);
+ const b = computeSelectionHash(
+ { ...clip(), hook: { start: 10, end: 12, mode: "repeat" } },
+ words,
+ );
+ expect(a).not.toBe(b);
+ });
+
+ it("changes when the hook's range or mode is edited", () => {
+ const hooked = { ...clip(), hook: { start: 12, end: 14, mode: "repeat" as const } };
+ const base = computeSelectionHash(hooked, words);
+ const moved = computeSelectionHash({ ...hooked, hook: { ...hooked.hook, start: 12.5 } }, words);
+ const remoded = computeSelectionHash({ ...hooked, hook: { ...hooked.hook, mode: "move" } }, words);
+ expect(moved).not.toBe(base);
+ expect(remoded).not.toBe(base);
+ });
+
+ it("changes when a hook is cleared", () => {
+ const hooked = { ...clip(), hook: { start: 12, end: 14, mode: "repeat" as const } };
+ expect(computeSelectionHash({ ...hooked, hook: undefined }, words)).not.toBe(
+ computeSelectionHash(hooked, words),
+ );
+ });
+});
diff --git a/src/utils/selection-hash.ts b/src/utils/selection-hash.ts
new file mode 100644
index 0000000..1364564
--- /dev/null
+++ b/src/utils/selection-hash.ts
@@ -0,0 +1,34 @@
+import { createHash } from "crypto";
+import type { SuggestedClip, WordTimestamp } from "../models/index.js";
+
+/**
+ * Fingerprints everything about a clip that changes what actually renders:
+ * its range, its editorial segments and hook (if any), caption style, title
+ * text, and the transcript words that fall inside its range. Selecting a
+ * clip for export stores this; if the clip is edited afterward the stored
+ * hash no longer matches, which is the signal that the export no longer
+ * describes what was approved.
+ */
+export function computeSelectionHash(
+ clip: Pick<
+ SuggestedClip,
+ "start_second" | "end_second" | "segments" | "hook" | "suggested_caption_style" | "title"
+ >,
+ transcriptWords: WordTimestamp[] | undefined | null,
+): string {
+ const wordsInRange = (transcriptWords || [])
+ .filter((w) => w.start >= clip.start_second && w.start < clip.end_second)
+ .map((w) => `${w.word}:${w.start}:${w.end}`);
+
+ const payload = JSON.stringify({
+ start: clip.start_second,
+ end: clip.end_second,
+ segments: clip.segments ?? null,
+ hook: clip.hook ?? null,
+ captionStyle: clip.suggested_caption_style ?? null,
+ title: clip.title ?? null,
+ words: wordsInRange,
+ });
+
+ return createHash("sha1").update(payload).digest("hex");
+}
diff --git a/src/utils/srt-time.test.ts b/src/utils/srt-time.test.ts
new file mode 100644
index 0000000..90995d8
--- /dev/null
+++ b/src/utils/srt-time.test.ts
@@ -0,0 +1,27 @@
+import { describe, it, expect } from "vitest";
+import { formatSrtTime, formatVttTime } from "./srt-time.js";
+
+describe("formatSrtTime", () => {
+ it("formats a plain value", () => {
+ expect(formatSrtTime(65.25)).toBe("00:01:05,250");
+ });
+
+ it("carries milliseconds that round up to 1000 into the next second", () => {
+ // 1.9996 * 1000 = 1999.6 -> rounds to 2000ms, not "01,1000".
+ expect(formatSrtTime(1.9996)).toBe("00:00:02,000");
+ });
+
+ it("carries a rounded second into the next minute", () => {
+ expect(formatSrtTime(59.9996)).toBe("00:01:00,000");
+ });
+
+ it("carries through hours", () => {
+ expect(formatSrtTime(3599.9996)).toBe("01:00:00,000");
+ });
+});
+
+describe("formatVttTime", () => {
+ it("uses a period instead of a comma", () => {
+ expect(formatVttTime(1.9996)).toBe("00:00:02.000");
+ });
+});
diff --git a/src/utils/srt-time.ts b/src/utils/srt-time.ts
new file mode 100644
index 0000000..b7793b3
--- /dev/null
+++ b/src/utils/srt-time.ts
@@ -0,0 +1,23 @@
+/**
+ * Format seconds as an SRT timestamp (HH:MM:SS,mmm).
+ *
+ * Rounds to whole milliseconds once, up front, then derives h/m/sec/ms by
+ * integer division. Rounding the seconds and milliseconds components
+ * separately lets a value like 1.9996 round seconds down to 1 and ms up to
+ * 1000, emitting the invalid "01,1000" instead of carrying into "02,000".
+ */
+export function formatSrtTime(s: number): string {
+ let totalMs = Math.round(s * 1000);
+ const h = Math.floor(totalMs / 3_600_000);
+ totalMs -= h * 3_600_000;
+ const m = Math.floor(totalMs / 60_000);
+ totalMs -= m * 60_000;
+ const sec = Math.floor(totalMs / 1000);
+ const ms = totalMs - sec * 1000;
+ return `${String(h).padStart(2, "0")}:${String(m).padStart(2, "0")}:${String(sec).padStart(2, "0")},${String(ms).padStart(3, "0")}`;
+}
+
+/** Format seconds as a WEBVTT timestamp (HH:MM:SS.mmm). */
+export function formatVttTime(s: number): string {
+ return formatSrtTime(s).replace(",", ".");
+}
diff --git a/src/utils/thumbnail-layout.test.ts b/src/utils/thumbnail-layout.test.ts
new file mode 100644
index 0000000..589db1d
--- /dev/null
+++ b/src/utils/thumbnail-layout.test.ts
@@ -0,0 +1,66 @@
+import { describe, expect, it } from "vitest";
+import { needsFrame, parseThumbnailRender, thumbnailRenderArgs } from "./thumbnail-layout.js";
+
+const clip = { source_video: "/eps/ep1.mp4", start_second: 12.5, end_second: 48 };
+
+describe("thumbnailRenderArgs", () => {
+ it("takes both people from the clip's own range for the pair layout", () => {
+ const args = thumbnailRenderArgs({ output: "/out/t.png", layout: "pair", swap: true, clip });
+ expect(args).toEqual([
+ "thumbnail-render", "--output", "/out/t.png", "--layout", "pair",
+ "--video", "/eps/ep1.mp4", "--start", "12.5", "--end", "48", "--swap",
+ ]);
+ });
+
+ it("keeps a chosen frame as the pair layout's fallback but drops its face metadata", () => {
+ const args = thumbnailRenderArgs({
+ output: "/out/t.png", frame: "/f.jpg", frameInfo: { face_x_pct: 40 }, layout: "pair", clip,
+ });
+ expect(args).toContain("--frame");
+ expect(args).not.toContain("--frame-info");
+ expect(args).not.toContain("--swap");
+ });
+
+ it("renders one face from the chosen frame without touching the video", () => {
+ const args = thumbnailRenderArgs({
+ output: "/out/t.png", frame: "/f.jpg", frameInfo: { face_x_pct: 40 }, line1: "One", line2: "Two", clip,
+ });
+ expect(args).toEqual([
+ "thumbnail-render", "--output", "/out/t.png", "--frame", "/f.jpg",
+ "--line1=One", "--line2=Two", "--frame-info", '{"face_x_pct":40}',
+ ]);
+ });
+
+ it("ignores a layout it does not know", () => {
+ expect(thumbnailRenderArgs({ output: "/o.png", frame: "/f.jpg", layout: "grid", clip })).not.toContain("--layout");
+ });
+});
+
+describe("needsFrame", () => {
+ it("lets the pair layout render without a chosen frame", () => {
+ expect(needsFrame("pair")).toBe(false);
+ expect(needsFrame("single")).toBe(true);
+ expect(needsFrame(undefined)).toBe(true);
+ });
+});
+
+describe("parseThumbnailRender", () => {
+ it("reads the path and which source second each face came from", () => {
+ const people = [
+ { side: "left", role: "guest", from: "seats", source_time: 14 },
+ { side: "right", role: "host", from: "seats", source_time: 40.5 },
+ ];
+ const out = `progress line\n${JSON.stringify({ path: "/out/t.png", layout: "pair", people })}\n`;
+ expect(parseThumbnailRender(out)).toEqual({ path: "/out/t.png", report: { layout: "pair", people } });
+ });
+
+ it("carries the reason a pair render fell back to one face", () => {
+ const note = "Only one person is on screen in this clip. The thumbnail uses the single-face layout.";
+ const out = JSON.stringify({ path: "/out/t.png", layout: "single", note });
+ expect(parseThumbnailRender(out).report).toEqual({ layout: "single", note });
+ });
+
+ it("answers an empty path for output it cannot read", () => {
+ expect(parseThumbnailRender("not json")).toEqual({ path: "", report: { layout: "single" } });
+ });
+});
diff --git a/src/utils/thumbnail-layout.ts b/src/utils/thumbnail-layout.ts
new file mode 100644
index 0000000..003622e
--- /dev/null
+++ b/src/utils/thumbnail-layout.ts
@@ -0,0 +1,81 @@
+/**
+ * Thumbnail layouts: one face, or two people side by side from the clip.
+ * backend/services/thumbnail_pair.py picks the people; this builds the
+ * `thumbnail-render` call for a clip and reads back what it reported.
+ *
+ * No imports, so the studio client can bundle it as is.
+ */
+
+export type ThumbnailLayout = "single" | "pair";
+
+/** One person on a pair-layout thumbnail and where their face came from. */
+export interface ThumbnailPerson {
+ side: "left" | "right";
+ role: "guest" | "host" | null;
+ from: "seats" | "multicam" | "image";
+ source_time: number | null; // second in the source video the face was taken at
+ camera?: string; // multicam only: the person's own camera file
+ camera_time?: number; // multicam only: the second in that file
+}
+
+/** The layout a render ended up with, and why when it fell back to one face. */
+export interface ThumbnailLayoutReport {
+ layout: ThumbnailLayout;
+ note?: string;
+ people?: ThumbnailPerson[];
+}
+
+export interface ClipThumbnailRender {
+ output: string;
+ frame?: string | null;
+ frameInfo?: unknown;
+ line1?: string;
+ line2?: string;
+ layout?: unknown;
+ swap?: boolean;
+ clip: { source_video: string; start_second: number; end_second: number };
+}
+
+/** Whether a render needs a chosen frame. The pair layout takes both people from the clip. */
+export function needsFrame(layout: unknown): boolean {
+ return layout !== "pair";
+}
+
+/** Arguments for `podcli thumbnail-render` on one clip, ahead of the grounding flags and the title. */
+export function thumbnailRenderArgs(r: ClipThumbnailRender): string[] {
+ const pair = r.layout === "pair";
+ const args = ["thumbnail-render", "--output", r.output];
+ if (r.frame) args.push("--frame", r.frame);
+ if (r.layout === "pair" || r.layout === "single") args.push("--layout", r.layout);
+ if (pair) {
+ args.push(
+ "--video", r.clip.source_video,
+ "--start", String(r.clip.start_second),
+ "--end", String(r.clip.end_second),
+ );
+ if (r.swap) args.push("--swap");
+ }
+ if (r.line1) args.push(`--line1=${r.line1}`);
+ if (r.line2) args.push(`--line2=${r.line2}`);
+ // Face metadata describes the chosen frame, which a pair render does not draw.
+ if (r.frameInfo && !pair) args.push("--frame-info", JSON.stringify(r.frameInfo));
+ return args;
+}
+
+/** The PNG path and layout report from the last JSON line `thumbnail-render` printed. */
+export function parseThumbnailRender(stdout: string): { path: string; report: ThumbnailLayoutReport } {
+ const line = stdout.trim().split("\n").reverse().find((l) => l.trim().startsWith("{"));
+ try {
+ const parsed = JSON.parse(line || "{}");
+ return {
+ path: typeof parsed.path === "string" ? parsed.path : "",
+ report: {
+ layout: parsed.layout === "pair" ? "pair" : "single",
+ ...(typeof parsed.note === "string" && { note: parsed.note }),
+ ...(Array.isArray(parsed.people) && { people: parsed.people as ThumbnailPerson[] }),
+ },
+ };
+ } catch {
+ return { path: "", report: { layout: "single" } };
+ }
+}
diff --git a/src/utils/transcript.test.ts b/src/utils/transcript.test.ts
new file mode 100644
index 0000000..58dcc2d
--- /dev/null
+++ b/src/utils/transcript.test.ts
@@ -0,0 +1,109 @@
+import { describe, it, expect } from "vitest";
+import { findGroundingText, reconcileSegmentsForRange } from "./transcript.js";
+
+describe("findGroundingText", () => {
+ const suggestions = [
+ { start_second: 10, end_second: 40, payoff: "the payoff", context_line: "the question", preview_text: "the opening line" },
+ { start_second: 100, end_second: 130 }, // no grounding fields at all
+ ];
+
+ it("returns the grounding fields of the closest-matching suggestion", () => {
+ expect(findGroundingText(suggestions, 10, 40)).toEqual({
+ payoff: "the payoff",
+ context_line: "the question",
+ preview_text: "the opening line",
+ });
+ });
+
+ it("matches within a small tolerance, not just exactly", () => {
+ expect(findGroundingText(suggestions, 11, 39)).toEqual({
+ payoff: "the payoff",
+ context_line: "the question",
+ preview_text: "the opening line",
+ });
+ });
+
+ it("returns undefined fields when the matched suggestion has none", () => {
+ expect(findGroundingText(suggestions, 100, 130)).toEqual({
+ payoff: undefined,
+ context_line: undefined,
+ preview_text: undefined,
+ });
+ });
+
+ it("returns undefined when nothing matches", () => {
+ expect(findGroundingText(suggestions, 500, 530)).toBeUndefined();
+ });
+
+ it("returns undefined for an empty or missing suggestions list", () => {
+ expect(findGroundingText([], 10, 40)).toBeUndefined();
+ expect(findGroundingText(undefined, 10, 40)).toBeUndefined();
+ expect(findGroundingText(null, 10, 40)).toBeUndefined();
+ });
+});
+
+describe("reconcileSegmentsForRange", () => {
+ it("passes through when there are no segments", () => {
+ expect(reconcileSegmentsForRange(undefined, 0, 10, 0, 10)).toBeUndefined();
+ expect(reconcileSegmentsForRange([], 0, 10, 0, 10)).toEqual([]);
+ });
+
+ it("drops a single implicit full-range segment and lets the new bounds govern", () => {
+ // suggest_clips stores a single [start, end] segment with no custom cuts;
+ // clamping it only ever shrinks, so widening or shifting the clip left a
+ // stale segment overriding the new range at render time.
+ const segments = [{ start: 10, end: 60 }];
+ expect(reconcileSegmentsForRange(segments, 10, 60, 5, 70)).toBeUndefined();
+ expect(reconcileSegmentsForRange(segments, 10, 60, 20, 80)).toBeUndefined();
+ });
+
+ it("drops segments entirely outside the new range", () => {
+ const segments = [
+ { start: 0, end: 5 },
+ { start: 20, end: 25 },
+ ];
+ expect(reconcileSegmentsForRange(segments, 0, 25, 20, 25)).toBeUndefined();
+ });
+
+ it("clamps segments straddling a new boundary", () => {
+ const segments = [
+ { start: 0, end: 10 },
+ { start: 15, end: 25 },
+ ];
+ expect(reconcileSegmentsForRange(segments, 0, 25, 5, 20)).toEqual([
+ { start: 5, end: 10 },
+ { start: 15, end: 20 },
+ ]);
+ });
+
+ it("clears a single custom segment that ends up spanning the whole new range", () => {
+ const segments = [{ start: 0, end: 30 }];
+ expect(reconcileSegmentsForRange(segments, -10, 40, 5, 20)).toBeUndefined();
+ });
+
+ it("stretches the first segment's start and the last segment's end to the new bounds", () => {
+ // The interior gap (8-12) is the editorial cut; the outer edges track
+ // the clip's own bounds, so widening the clip carries them along
+ // instead of leaving them pinned to the old range.
+ const segments = [
+ { start: 2, end: 8 },
+ { start: 12, end: 18 },
+ ];
+ expect(reconcileSegmentsForRange(segments, 2, 18, 0, 20)).toEqual([
+ { start: 0, end: 8 },
+ { start: 12, end: 20 },
+ ]);
+ });
+
+ it("drops and clamps alongside the outer-edge stretch for multi-segment edits", () => {
+ const segments = [
+ { start: 2, end: 8 },
+ { start: 12, end: 18 },
+ { start: 50, end: 60 }, // entirely outside the new range
+ ];
+ expect(reconcileSegmentsForRange(segments, 2, 60, 0, 20)).toEqual([
+ { start: 0, end: 8 },
+ { start: 12, end: 20 },
+ ]);
+ });
+});
diff --git a/src/utils/transcript.ts b/src/utils/transcript.ts
index 4a10803..ef1e585 100644
--- a/src/utils/transcript.ts
+++ b/src/utils/transcript.ts
@@ -43,6 +43,101 @@ export function findContentType(
return match?.content_type;
}
+/**
+ * The suggestion's grounding text for the clip whose range best matches
+ * [start, end]: the payoff, the question it answers, and its verbatim
+ * opening line. Thumbnail copy is written from this, not from the title
+ * alone, so it matches what the clip actually says.
+ */
+export function findGroundingText(
+ suggestions:
+ | Array<{
+ start_second: number;
+ end_second: number;
+ payoff?: string;
+ context_line?: string;
+ preview_text?: string;
+ }>
+ | undefined
+ | null,
+ start: number,
+ end: number,
+): { payoff?: string; context_line?: string; preview_text?: string } | undefined {
+ if (!suggestions || suggestions.length === 0) return undefined;
+ const match = suggestions.find(
+ (s) => Math.abs(s.start_second - start) <= 2 && Math.abs(s.end_second - end) <= 2,
+ );
+ if (!match) return undefined;
+ return { payoff: match.payoff, context_line: match.context_line, preview_text: match.preview_text };
+}
+
+/**
+ * Keep a clip's keep_segments consistent after its start/end range is edited.
+ * Without this, a stale segments array (scoped to the old range) overrides
+ * the new start_second/end_second at render time, since the generator
+ * derives the actual cut points from keep_segments when present.
+ *
+ * A single segment that already spans the entire old range is just a
+ * stand-in for "no custom cuts". It's dropped outright so the new
+ * start/end govern directly, instead of clamping it (which only ever
+ * shrinks, so widening or shifting the clip left it pinned to the old
+ * bounds: editing 10-60 to 5-70 kept [10, 60], and to 20-80 rendered
+ * [20, 60]).
+ *
+ * With two or more segments, the interior gaps are the editorial cuts;
+ * the first segment's start and the last segment's end are assumed to
+ * track the clip's own bounds, so they're stretched or trimmed to the new
+ * range rather than left at the old one. A segment entirely outside the
+ * new range is dropped, and one straddling a new boundary is clamped to it.
+ */
+export function reconcileSegmentsForRange(
+ segments: Array<{ start: number; end: number }> | undefined,
+ oldStart: number,
+ oldEnd: number,
+ nextStart: number,
+ nextEnd: number,
+): Array<{ start: number; end: number }> | undefined {
+ if (!segments?.length) return segments;
+
+ if (
+ segments.length === 1 &&
+ Math.abs(segments[0].start - oldStart) <= 0.01 &&
+ Math.abs(segments[0].end - oldEnd) <= 0.01
+ ) {
+ return undefined;
+ }
+
+ const clamped = segments
+ .map((s) => ({ start: Math.max(s.start, nextStart), end: Math.min(s.end, nextEnd) }))
+ .filter((s) => s.end > s.start);
+ if (clamped.length === 0) return undefined;
+
+ if (clamped.length > 1) {
+ clamped[0] = { ...clamped[0], start: nextStart };
+ clamped[clamped.length - 1] = { ...clamped[clamped.length - 1], end: nextEnd };
+ return clamped;
+ }
+
+ if (
+ clamped[0].start <= nextStart + 0.01 &&
+ clamped[0].end >= nextEnd - 0.01
+ ) {
+ return undefined;
+ }
+ return clamped;
+}
+
+/** The suggestion whose range matches start/end within half a second. */
+export function findSuggestionForRange<T extends { start_second: number; end_second: number }>(
+ suggestions: T[] | undefined | null,
+ start: number,
+ end: number,
+): T | undefined {
+ return suggestions?.find(
+ (s) => Math.abs(s.start_second - start) < 0.5 && Math.abs(s.end_second - end) < 0.5,
+ );
+}
+
export function findSuggestionSegments(
suggestions: Array<{
start_second: number;
@@ -52,9 +147,5 @@ export function findSuggestionSegments(
start: number,
end: number,
): Array<{ start: number; end: number }> | undefined {
- if (!suggestions?.length) return undefined;
- const match = suggestions.find(
- (s) => Math.abs(s.start_second - start) < 0.5 && Math.abs(s.end_second - end) < 0.5,
- );
- return match?.segments;
+ return findSuggestionForRange(suggestions, start, end)?.segments;
}
diff --git a/src/utils/video-identity.test.ts b/src/utils/video-identity.test.ts
new file mode 100644
index 0000000..7a8c261
--- /dev/null
+++ b/src/utils/video-identity.test.ts
@@ -0,0 +1,67 @@
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { mkdtempSync, writeFileSync, rmSync, utimesSync } from "fs";
+import { tmpdir } from "os";
+import { join } from "path";
+import { computeVideoIdentity, identitiesMatch, transcriptVideoMismatch } from "./video-identity.js";
+
+describe("video-identity", () => {
+ let dir: string;
+ let filePath: string;
+
+ beforeAll(() => {
+ dir = mkdtempSync(join(tmpdir(), "podcli-video-identity-"));
+ filePath = join(dir, "video.mp4");
+ writeFileSync(filePath, "abc");
+ });
+
+ afterAll(() => {
+ rmSync(dir, { recursive: true, force: true });
+ });
+
+ it("returns null for a missing file", () => {
+ expect(computeVideoIdentity(join(dir, "missing.mp4"))).toBeNull();
+ });
+
+ it("computes an identity for an existing file", () => {
+ const identity = computeVideoIdentity(filePath);
+ expect(identity?.path).toBe(filePath);
+ expect(identity?.size).toBe(3);
+ });
+
+ it("matches two identical identities", () => {
+ const a = computeVideoIdentity(filePath);
+ const b = computeVideoIdentity(filePath);
+ expect(identitiesMatch(a, b)).toBe(true);
+ });
+
+ it("does not match once the file is rewritten (size or mtime change)", () => {
+ const before = computeVideoIdentity(filePath);
+ writeFileSync(filePath, "a longer replacement body");
+ utimesSync(filePath, new Date(Date.now() + 5000), new Date(Date.now() + 5000));
+ const after = computeVideoIdentity(filePath);
+ expect(identitiesMatch(before, after)).toBe(false);
+ });
+
+ it("transcriptVideoMismatch is null when there's no recorded identity", () => {
+ expect(transcriptVideoMismatch(null, filePath)).toBeNull();
+ expect(transcriptVideoMismatch(undefined, filePath)).toBeNull();
+ });
+
+ it("transcriptVideoMismatch is null when the current video can't be statted", () => {
+ const stale = computeVideoIdentity(filePath);
+ expect(transcriptVideoMismatch(stale, join(dir, "gone.mp4"))).toBeNull();
+ });
+
+ it("transcriptVideoMismatch names the mismatch when the file changed", () => {
+ const stale = computeVideoIdentity(filePath);
+ writeFileSync(filePath, "totally different content, different length");
+ const msg = transcriptVideoMismatch(stale, filePath);
+ expect(msg).toMatch(/different video/);
+ expect(msg).toContain(filePath);
+ });
+
+ it("transcriptVideoMismatch is null when the file is unchanged", () => {
+ const identity = computeVideoIdentity(filePath);
+ expect(transcriptVideoMismatch(identity, filePath)).toBeNull();
+ });
+});
diff --git a/src/utils/video-identity.ts b/src/utils/video-identity.ts
new file mode 100644
index 0000000..474bacc
--- /dev/null
+++ b/src/utils/video-identity.ts
@@ -0,0 +1,53 @@
+import { statSync } from "fs";
+
+/**
+ * Identifies a specific video file on disk, not just its path, so a
+ * transcript can be tied to the exact bytes it was generated from. A path
+ * can be reused for a different recording (re-record, re-encode, swap in a
+ * trimmed version); size and mtime catch that even though the path matches.
+ */
+export interface VideoIdentity {
+ path: string;
+ size: number;
+ mtimeMs: number;
+}
+
+/** Stat a video file into a VideoIdentity, or null if it can't be read. */
+export function computeVideoIdentity(path: string): VideoIdentity | null {
+ try {
+ const stat = statSync(path);
+ return { path, size: stat.size, mtimeMs: stat.mtimeMs };
+ } catch {
+ return null;
+ }
+}
+
+/** True when two identities refer to the same file snapshot. */
+export function identitiesMatch(
+ a: VideoIdentity | null | undefined,
+ b: VideoIdentity | null | undefined,
+): boolean {
+ if (!a || !b) return false;
+ return a.path === b.path && a.size === b.size && a.mtimeMs === b.mtimeMs;
+}
+
+/**
+ * Checks whether a session's recorded transcript still belongs to the video
+ * currently loaded at videoPath. Returns an error string naming the mismatch
+ * when it doesn't, or null when it's safe to proceed (including when there's
+ * no recorded identity to check, e.g. older sessions before this check existed).
+ */
+export function transcriptVideoMismatch(
+ transcriptVideoIdentity: VideoIdentity | null | undefined,
+ videoPath: string,
+): string | null {
+ if (!transcriptVideoIdentity) return null;
+ const current = computeVideoIdentity(videoPath);
+ if (!current) return null;
+ if (identitiesMatch(transcriptVideoIdentity, current)) return null;
+ return (
+ `Session transcript belongs to a different video than the one currently set ` +
+ `(${transcriptVideoIdentity.path}). Re-transcribe or re-import a transcript for ` +
+ `${videoPath} before rendering.`
+ );
+}
diff --git a/src/utils/ytdlp-args.test.ts b/src/utils/ytdlp-args.test.ts
index 273a9a5..878c0d2 100644
--- a/src/utils/ytdlp-args.test.ts
+++ b/src/utils/ytdlp-args.test.ts
@@ -1,5 +1,14 @@
import { describe, expect, it } from "vitest";
-import { buildYtDlpArgs, isCookieBrowser, ytDlpHint } from "./ytdlp-args.js";
+import {
+ buildYtDlpArgs,
+ buildChannelListArgs,
+ buildVideoInfoArgs,
+ extractYouTubeVideoId,
+ isCookieBrowser,
+ isHttpUrl,
+ normalizeChannelUrl,
+ ytDlpHint,
+} from "./ytdlp-args.js";
const base = {
url: "https://example.com/watch?v=abc",
@@ -26,6 +35,12 @@ describe("buildYtDlpArgs", () => {
expect(buildYtDlpArgs(base).at(-1)).toBe(base.url);
});
+ it("puts -- right before the url so a dash-prefixed value can't be read as a flag", () => {
+ const args = buildYtDlpArgs(base);
+ expect(args.at(-2)).toBe("--");
+ expect(args.at(-1)).toBe(base.url);
+ });
+
it("omits cookies and extractor args when unset", () => {
const args = buildYtDlpArgs(base);
expect(args).not.toContain("--cookies-from-browser");
@@ -64,6 +79,135 @@ describe("isCookieBrowser", () => {
});
});
+describe("buildChannelListArgs", () => {
+ const channelUrl = "https://www.youtube.com/@example/videos";
+
+ it("never resolves each video's own page: flat-playlist only", () => {
+ const args = buildChannelListArgs({ channelUrl });
+ expect(args).toContain("--flat-playlist");
+ expect(args).not.toContain("--format");
+ });
+
+ it("caps the listing with --playlist-end when a limit is given", () => {
+ const args = buildChannelListArgs({ channelUrl, limit: 25 });
+ const i = args.indexOf("--playlist-end");
+ expect(args[i + 1]).toBe("25");
+ });
+
+ it("omits the limit flag entirely when none is given", () => {
+ expect(buildChannelListArgs({ channelUrl })).not.toContain("--playlist-end");
+ });
+
+ it("puts the channel url last", () => {
+ expect(buildChannelListArgs({ channelUrl }).at(-1)).toBe(channelUrl);
+ });
+
+ it("puts -- right before the channel url", () => {
+ const args = buildChannelListArgs({ channelUrl });
+ expect(args.at(-2)).toBe("--");
+ expect(args.at(-1)).toBe(channelUrl);
+ });
+});
+
+describe("buildVideoInfoArgs", () => {
+ const videoUrl = "https://www.youtube.com/watch?v=abc123";
+
+ it("skips downloading the video itself", () => {
+ expect(buildVideoInfoArgs({ videoUrl })).toContain("--skip-download");
+ });
+
+ it("dumps JSON rather than a human-readable report", () => {
+ const args = buildVideoInfoArgs({ videoUrl });
+ expect(args).toContain("--dump-json");
+ });
+
+ it("puts -- right before the video url", () => {
+ const args = buildVideoInfoArgs({ videoUrl });
+ expect(args.at(-2)).toBe("--");
+ expect(args.at(-1)).toBe(videoUrl);
+ });
+
+ it("refuses to follow a playlist/mix to its first entry", () => {
+ expect(buildVideoInfoArgs({ videoUrl })).toContain("--no-playlist");
+ });
+});
+
+describe("extractYouTubeVideoId", () => {
+ it("reads the v= query param off a watch url", () => {
+ expect(extractYouTubeVideoId("https://www.youtube.com/watch?v=abc123")).toBe("abc123");
+ });
+
+ it("reads the id off a youtu.be short url", () => {
+ expect(extractYouTubeVideoId("https://youtu.be/abc123")).toBe("abc123");
+ });
+
+ it("reads the id off shorts/embed/live path shapes", () => {
+ expect(extractYouTubeVideoId("https://www.youtube.com/shorts/abc123")).toBe("abc123");
+ expect(extractYouTubeVideoId("https://www.youtube.com/embed/abc123")).toBe("abc123");
+ expect(extractYouTubeVideoId("https://www.youtube.com/live/abc123")).toBe("abc123");
+ });
+
+ it("returns null when no video id is present, e.g. a channel url", () => {
+ expect(extractYouTubeVideoId("https://www.youtube.com/@example/videos")).toBeNull();
+ });
+});
+
+describe("normalizeChannelUrl", () => {
+ it("pins a handle channel root to the videos tab", () => {
+ expect(normalizeChannelUrl("https://www.youtube.com/@deeptechdecodedai")).toBe(
+ "https://www.youtube.com/@deeptechdecodedai/videos",
+ );
+ });
+
+ it("pins a trailing-slash handle channel root to the videos tab", () => {
+ expect(normalizeChannelUrl("https://www.youtube.com/@deeptechdecodedai/")).toBe(
+ "https://www.youtube.com/@deeptechdecodedai/videos",
+ );
+ });
+
+ it("pins a /channel/<id> root to the videos tab", () => {
+ expect(normalizeChannelUrl("https://www.youtube.com/channel/UC12345")).toBe(
+ "https://www.youtube.com/channel/UC12345/videos",
+ );
+ });
+
+ it("leaves an already-specific tab url unchanged", () => {
+ expect(normalizeChannelUrl("https://www.youtube.com/@example/shorts")).toBe(
+ "https://www.youtube.com/@example/shorts",
+ );
+ expect(normalizeChannelUrl("https://www.youtube.com/@example/videos")).toBe(
+ "https://www.youtube.com/@example/videos",
+ );
+ });
+
+ it("leaves a non-channel url (playlist, watch) unchanged", () => {
+ const playlistUrl = "https://www.youtube.com/playlist?list=PL123";
+ expect(normalizeChannelUrl(playlistUrl)).toBe(playlistUrl);
+ });
+
+ it("returns the input unchanged when it isn't a valid url", () => {
+ expect(normalizeChannelUrl("not a url")).toBe("not a url");
+ });
+});
+
+describe("isHttpUrl", () => {
+ it("accepts http and https urls", () => {
+ expect(isHttpUrl("https://www.youtube.com/watch?v=abc123")).toBe(true);
+ expect(isHttpUrl("http://example.com")).toBe(true);
+ });
+
+ it("rejects a dash-prefixed value that would be read as a yt-dlp flag", () => {
+ expect(isHttpUrl("--config-locations=/tmp/evil.conf")).toBe(false);
+ });
+
+ it("rejects non-http schemes and non-strings", () => {
+ expect(isHttpUrl("file:///etc/passwd")).toBe(false);
+ expect(isHttpUrl("not a url")).toBe(false);
+ expect(isHttpUrl(undefined)).toBe(false);
+ expect(isHttpUrl(7)).toBe(false);
+ });
+});
+
describe("ytDlpHint", () => {
it("names the setting when the site wants a signed-in session", () => {
expect(ytDlpHint("ERROR: Sign in to confirm you're not a bot")).toContain(
diff --git a/src/utils/ytdlp-args.ts b/src/utils/ytdlp-args.ts
index 12b977f..6457e04 100644
--- a/src/utils/ytdlp-args.ts
+++ b/src/utils/ytdlp-args.ts
@@ -25,6 +25,20 @@ export function isCookieBrowser(value: unknown): value is CookieBrowser {
return typeof value === "string" && (COOKIE_BROWSERS as readonly string[]).includes(value);
}
+// A bare z.string() lets a value like "--config-locations=/tmp/evil.conf" through
+// as a channel/video url, and yt-dlp reads it as another flag rather than a
+// positional argument, one that can point at a config carrying --exec. Only
+// http(s) URLs are legitimate inputs here.
+export function isHttpUrl(value: unknown): value is string {
+ if (typeof value !== "string") return false;
+ try {
+ const parsed = new URL(value);
+ return parsed.protocol === "http:" || parsed.protocol === "https:";
+ } catch {
+ return false;
+ }
+}
+
export interface YtDlpOptions {
url: string;
outputDir: string;
@@ -71,10 +85,99 @@ export function buildYtDlpArgs(opts: YtDlpOptions): string[] {
args.push("--newline", "--progress", "--progress-template", opts.progressTemplate);
}
args.push("--print", "after_move:podcli-filepath:%(filepath)s");
- args.push(opts.url);
+ // "--" forces everything after it to be read positionally, so a url that
+ // starts with a dash can never be parsed as another flag.
+ args.push("--", opts.url);
return args;
}
+export interface YtDlpListOptions {
+ channelUrl: string;
+ limit?: number;
+ cookiesFromBrowser?: string;
+}
+
+const CHANNEL_TABS = ["videos", "shorts", "streams", "playlists", "live", "podcasts", "releases"];
+
+// A bare channel root (youtube.com/@handle, /channel/ID, /c/name, /user/name)
+// lists the channel's main feed, which YouTube mixes long-form uploads and
+// Shorts into. Pinning to the /videos tab keeps the listing to uploads only,
+// matching what the tool description promises. Already-specific urls
+// (a tab, a playlist, a single video) pass through unchanged.
+export function normalizeChannelUrl(url: string): string {
+ let parsed: URL;
+ try {
+ parsed = new URL(url);
+ } catch {
+ return url;
+ }
+ const segments = parsed.pathname.split("/").filter(Boolean);
+ const isChannelRoot = segments.length === 1 && segments[0].startsWith("@");
+ const isChannelIdRoot = segments.length === 2 && ["channel", "c", "user"].includes(segments[0]);
+ if (!isChannelRoot && !isChannelIdRoot) return url;
+
+ const lastSegment = segments.at(-1) ?? "";
+ if (CHANNEL_TABS.includes(lastSegment)) return url;
+
+ parsed.pathname = `${parsed.pathname.replace(/\/+$/, "")}/videos`;
+ return parsed.toString();
+}
+
+// One JSON object per line, tab would collide with titles that contain one.
+// --flat-playlist skips resolving each video's own page, so listing a
+// channel's uploads never pulls anything beyond the playlist metadata,
+// nowhere close to a full-video download.
+export function buildChannelListArgs(opts: YtDlpListOptions): string[] {
+ const args = ["-m", "yt_dlp"];
+ args.push("--ignore-config", "--no-config-locations", "--no-plugin-dirs");
+ args.push("--flat-playlist", "--no-warnings");
+ if (opts.limit) args.push("--playlist-end", String(opts.limit));
+ if (opts.cookiesFromBrowser) args.push("--cookies-from-browser", opts.cookiesFromBrowser);
+ args.push("--print", "%(.{id,title,duration,upload_date,url})j");
+ args.push("--", opts.channelUrl);
+ return args;
+}
+
+export interface YtDlpVideoInfoOptions {
+ videoUrl: string;
+ cookiesFromBrowser?: string;
+}
+
+// Dumps one video's full metadata (including subtitle/automatic_captions
+// track URLs) as JSON, with --skip-download so this never fetches the
+// video itself, only the page and timed-text track list.
+export function buildVideoInfoArgs(opts: YtDlpVideoInfoOptions): string[] {
+ const args = ["-m", "yt_dlp"];
+ args.push("--ignore-config", "--no-config-locations", "--no-plugin-dirs");
+ args.push("--skip-download", "--no-warnings");
+ // A url pointing at a playlist or a channel's "radio" mix would otherwise
+ // dump the first entry's info instead of erroring, silently mining the
+ // wrong video's captions.
+ args.push("--no-playlist");
+ if (opts.cookiesFromBrowser) args.push("--cookies-from-browser", opts.cookiesFromBrowser);
+ args.push("--dump-json", "--", opts.videoUrl);
+ return args;
+}
+
+// Pulls the video id out of the handful of URL shapes yt-dlp/YouTube use, so
+// the id yt-dlp actually resolved can be checked against what was asked for.
+export function extractYouTubeVideoId(url: string): string | null {
+ try {
+ const parsed = new URL(url);
+ const vParam = parsed.searchParams.get("v");
+ if (vParam) return vParam;
+ const pathMatch = parsed.pathname.match(/\/(?:shorts|embed|live)\/([^/?]+)/);
+ if (pathMatch) return pathMatch[1];
+ if (parsed.hostname === "youtu.be") {
+ const id = parsed.pathname.slice(1).split("/")[0];
+ return id || null;
+ }
+ return null;
+ } catch {
+ return null;
+ }
+}
+
/** Turns a yt-dlp failure into one line naming what to do about it. */
export function ytDlpHint(stderr: string): string | null {
const s = stderr.toLowerCase();
diff --git a/src/version.ts b/src/version.ts
index 42c1912..eed9051 100644
--- a/src/version.ts
+++ b/src/version.ts
@@ -15,3 +15,15 @@ export function podcliVersion(): string {
return "0.0.0-dev";
}
+
+// PODCLI_VERSION is only set when the Go launcher spawns this process (see
+// cli/internal/engine/engine.go nodeEnv). A source checkout running `npm run
+// ui` directly never has it, so its presence tells the two install types apart.
+export function isLauncherInstall(): boolean {
+ return !!process.env.PODCLI_VERSION?.trim();
+}
+
+/** The right command to start the Web UI for whichever install type is running. */
+export function studioStartCommand(): string {
+ return isLauncherInstall() ? "podcli studio" : "npm run ui";
+}
diff --git a/tests/test_ai_fallback.py b/tests/test_ai_fallback.py
index 9c9bf02..66bf5e3 100644
--- a/tests/test_ai_fallback.py
+++ b/tests/test_ai_fallback.py
@@ -412,13 +412,19 @@ class AICliDiscoveryTests(unittest.TestCase):
self.assertEqual(found, cli)
def test_find_cli_falls_back_to_shell_lookup(self):
- with tempfile.TemporaryDirectory() as tmp:
+ with tempfile.TemporaryDirectory() as tmp, tempfile.TemporaryDirectory() as home:
cli = os.path.join(tmp, "claude")
with open(cli, "w", encoding="utf-8") as fh:
fh.write("#!/bin/sh\n")
- with mock.patch.object(ai, "_shell_lookup", return_value=cli):
- with mock.patch("shutil.which", return_value=None):
- found = ai._find_cli("claude", [])
+ # Without redirecting HOME, _find_cli's fixed lookup dirs (e.g.
+ # ~/.local/bin) hit the real filesystem. On a machine with an
+ # actual claude CLI installed there, it wins before shell lookup
+ # ever runs, and this test passes for the wrong reason.
+ with mock.patch.dict(os.environ, {"HOME": home, "PATH": ""}, clear=False):
+ with mock.patch("os.path.expanduser", side_effect=lambda p: p.replace("~", home)):
+ with mock.patch.object(ai, "_shell_lookup", return_value=cli):
+ with mock.patch("shutil.which", return_value=None):
+ found = ai._find_cli("claude", [])
self.assertEqual(found, cli)
def test_get_ai_cli_status_reports_candidates(self):
diff --git a/tests/test_assemblyai_resume.py b/tests/test_assemblyai_resume.py
new file mode 100644
index 0000000..a79688c
--- /dev/null
+++ b/tests/test_assemblyai_resume.py
@@ -0,0 +1,194 @@
+"""AssemblyAI: a restarted job must resume by polling the transcript it
+already started, not re-upload the file and start a second paid job."""
+
+import hashlib
+import os
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+import services.transcription as tr
+from services import transcribe_runs
+
+_TEST_KEY_HASH = hashlib.sha256(b"test-key").hexdigest()[:8]
+_MODEL_PART = f"diar0-spk0-us-{_TEST_KEY_HASH}"
+
+
+class AssemblyAIResumeTests(unittest.TestCase):
+ def setUp(self):
+ self._tmp_cache = tempfile.mkdtemp()
+ self._orig_run_dir = transcribe_runs.run_dir
+ # Route every receipt into a throwaway directory for this test
+ # instead of the real cache.
+ transcribe_runs.run_dir = lambda cache_root, *a, **k: self._orig_run_dir(
+ self._tmp_cache, *a, **k
+ )
+ self._tmp_file = tempfile.NamedTemporaryFile(suffix=".mp3", delete=False)
+ self._tmp_file.write(b"fake audio")
+ self._tmp_file.close()
+ self._saved_key = os.environ.get("ASSEMBLYAI_API_KEY")
+ os.environ["ASSEMBLYAI_API_KEY"] = "test-key"
+
+ def tearDown(self):
+ transcribe_runs.run_dir = self._orig_run_dir
+ os.unlink(self._tmp_file.name)
+ import shutil
+ shutil.rmtree(self._tmp_cache, ignore_errors=True)
+ if self._saved_key is None:
+ os.environ.pop("ASSEMBLYAI_API_KEY", None)
+ else:
+ os.environ["ASSEMBLYAI_API_KEY"] = self._saved_key
+
+ def _completed_response(self):
+ return {
+ "status": "completed",
+ "text": "hello world",
+ "words": [],
+ "audio_duration": 1.0,
+ "language_code": "en",
+ }
+
+ def test_first_call_uploads_and_writes_a_receipt_before_polling(self):
+ with mock.patch.object(tr, "_assemblyai_upload", return_value="https://upload/x") as upload, \
+ mock.patch.object(
+ tr, "_assemblyai_json_request",
+ side_effect=[{"id": "tid-1"}, self._completed_response()],
+ ) as req:
+ result = tr._transcribe_with_assemblyai(
+ self._tmp_file.name, language=None, enable_diarization=False,
+ num_speakers=None, progress_callback=None,
+ )
+ upload.assert_called_once()
+ self.assertEqual(result["transcript"], "hello world")
+ # POST to create + GET to poll
+ self.assertEqual(req.call_count, 2)
+ self.assertEqual(req.call_args_list[0][0][0], "POST")
+
+ def test_restarted_run_polls_the_saved_transcript_id_without_reuploading(self):
+ directory = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai", model_size=_MODEL_PART, language=None,
+ )
+ transcribe_runs.write_receipt(directory, "assemblyai.json", {"transcript_id": "tid-existing"})
+
+ with mock.patch.object(tr, "_assemblyai_upload") as upload, \
+ mock.patch.object(
+ tr, "_assemblyai_json_request", return_value=self._completed_response(),
+ ) as req:
+ result = tr._transcribe_with_assemblyai(
+ self._tmp_file.name, language=None, enable_diarization=False,
+ num_speakers=None, progress_callback=None,
+ )
+ upload.assert_not_called()
+ self.assertEqual(result["transcript"], "hello world")
+ # Only the poll GET, no POST to create a new transcript.
+ self.assertEqual(req.call_count, 1)
+ polled_url = req.call_args_list[0][0][1]
+ self.assertIn("tid-existing", polled_url)
+
+ def test_receipt_is_cleared_once_completed(self):
+ directory = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai", model_size=_MODEL_PART, language=None,
+ )
+ with mock.patch.object(tr, "_assemblyai_upload", return_value="https://upload/x"), \
+ mock.patch.object(
+ tr, "_assemblyai_json_request",
+ side_effect=[{"id": "tid-2"}, self._completed_response()],
+ ):
+ tr._transcribe_with_assemblyai(
+ self._tmp_file.name, language=None, enable_diarization=False,
+ num_speakers=None, progress_callback=None,
+ )
+ self.assertIsNone(transcribe_runs.read_receipt(directory, "assemblyai.json"))
+
+ def test_a_different_file_never_adopts_anothers_receipt(self):
+ other = tempfile.NamedTemporaryFile(suffix=".mp3", delete=False)
+ other.write(b"different fake audio, different fingerprint")
+ other.close()
+ try:
+ directory = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai", model_size=_MODEL_PART, language=None,
+ )
+ transcribe_runs.write_receipt(directory, "assemblyai.json", {"transcript_id": "tid-not-mine"})
+
+ with mock.patch.object(tr, "_assemblyai_upload", return_value="https://upload/x") as upload, \
+ mock.patch.object(
+ tr, "_assemblyai_json_request",
+ side_effect=[{"id": "tid-mine"}, self._completed_response()],
+ ):
+ tr._transcribe_with_assemblyai(
+ other.name, language=None, enable_diarization=False,
+ num_speakers=None, progress_callback=None,
+ )
+ upload.assert_called_once()
+ finally:
+ os.unlink(other.name)
+
+ def test_a_4xx_on_the_resume_get_clears_the_receipt_and_reuploads(self):
+ directory = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai", model_size=_MODEL_PART, language=None,
+ )
+ transcribe_runs.write_receipt(directory, "assemblyai.json", {"transcript_id": "tid-gone"})
+ http_error = tr.AssemblyAIHTTPError("gone", status=404)
+
+ with mock.patch.object(tr, "_assemblyai_upload", return_value="https://upload/x") as upload, \
+ mock.patch.object(
+ tr, "_assemblyai_json_request",
+ side_effect=[http_error, {"id": "tid-fresh"}, self._completed_response()],
+ ) as req:
+ result = tr._transcribe_with_assemblyai(
+ self._tmp_file.name, language=None, enable_diarization=False,
+ num_speakers=None, progress_callback=None,
+ )
+ # The stale receipt's GET fails once, then a fresh upload + create +
+ # poll succeeds, never retrying the dead transcript_id forever.
+ upload.assert_called_once()
+ self.assertEqual(result["transcript"], "hello world")
+ self.assertEqual(req.call_args_list[0][0][0], "GET")
+ self.assertIn("tid-gone", req.call_args_list[0][0][1])
+ self.assertEqual(req.call_args_list[1][0][0], "POST")
+ new_receipt = transcribe_runs.read_receipt(directory, "assemblyai.json")
+ self.assertIsNone(new_receipt) # cleared on completion, same as any other run
+
+ def test_a_5xx_on_the_resume_get_does_not_clear_the_receipt(self):
+ # A transient server error is retried by _assemblyai_json_request
+ # itself; if it still fails after retries, the receipt must survive
+ # so a later run can resume rather than paying for a new upload.
+ directory = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai", model_size=_MODEL_PART, language=None,
+ )
+ transcribe_runs.write_receipt(directory, "assemblyai.json", {"transcript_id": "tid-existing"})
+ http_error = tr.AssemblyAIHTTPError("server error", status=503)
+
+ with mock.patch.object(tr, "_assemblyai_upload") as upload, \
+ mock.patch.object(tr, "_assemblyai_json_request", side_effect=http_error):
+ with self.assertRaises(tr.AssemblyAIHTTPError):
+ tr._transcribe_with_assemblyai(
+ self._tmp_file.name, language=None, enable_diarization=False,
+ num_speakers=None, progress_callback=None,
+ )
+ upload.assert_not_called()
+ self.assertIsNotNone(transcribe_runs.read_receipt(directory, "assemblyai.json"))
+
+ def test_region_and_key_are_part_of_the_receipt_key(self):
+ # Two different API keys for the same file must never share a run
+ # directory: resuming the wrong account's transcript_id would poll
+ # (or worse, successfully return) someone else's job.
+ dir_a = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai",
+ model_size=f"diar0-spk0-us-{hashlib.sha256(b'key-a').hexdigest()[:8]}", language=None,
+ )
+ dir_b = transcribe_runs.run_dir(
+ "unused", self._tmp_file.name, "assemblyai",
+ model_size=f"diar0-spk0-us-{hashlib.sha256(b'key-b').hexdigest()[:8]}", language=None,
+ )
+ self.assertNotEqual(dir_a, dir_b)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_caption_renderer.py b/tests/test_caption_renderer.py
index fa130c4..4d0a9e4 100644
--- a/tests/test_caption_renderer.py
+++ b/tests/test_caption_renderer.py
@@ -98,5 +98,64 @@ class BrandedEventVolumeTests(unittest.TestCase):
self.assertEqual(len(ends), 1)
+class AssTextSanitizationTests(unittest.TestCase):
+ """Words containing backslash/brace characters must not corrupt the ASS
+ override-block syntax or get read as \\N/\\n/\\h control codes."""
+
+ def _residual_text(self, content: str) -> str:
+ # Strip the override blocks we generate ourselves; anything left
+ # over came from word text and must not contain raw \, { or }.
+ import re
+ stripped = re.sub(r"\{\\[^}]*\}", "", content)
+ return stripped
+
+ def test_sanitize_replaces_control_characters(self):
+ self.assertEqual(cr._sanitize_ass_text("C:\\path"), "C:\path")
+ self.assertEqual(cr._sanitize_ass_text("{weird}"), "{weird}")
+ self.assertEqual(cr._sanitize_ass_text(r"a\nb"), "a\nb")
+
+ def test_render_hormozi_escapes_backslash_and_braces(self):
+ style = get_style("hormozi")
+ words = _flowing_words(["a\\nb", "{curly}"])
+ content = cr._render_hormozi(words, style, 0.0)
+ residual = self._residual_text(content)
+ self.assertNotIn("\\n", residual)
+ self.assertNotIn("{curly}", content)
+ self.assertIn("\", content)
+ # hormozi uppercases word text; the braces are swapped regardless of case.
+ self.assertIn("{CURLY}" if style["uppercase"] else "{curly}", content)
+
+ def test_render_karaoke_escapes_backslash_and_braces(self):
+ style = get_style("karaoke")
+ words = _flowing_words(["a\\nb", "{curly}"])
+ content = cr._render_karaoke(words, style, 0.0)
+ self.assertIn("\", content)
+ self.assertIn("{curly}", content)
+
+ def test_render_subtle_escapes_backslash_and_braces(self):
+ style = get_style("subtle")
+ words = _flowing_words(["a\\nb", "{curly}"])
+ content = cr._render_subtle(words, style, 0.0)
+ self.assertIn("\", content)
+ self.assertIn("{curly}", content)
+
+ def test_render_branded_escapes_backslash_and_braces(self):
+ style = get_style("branded")
+ words = _flowing_words(["a\\nb", "{curly}"])
+ content = cr._render_branded(words, style, 0.0)
+ self.assertIn("\", content)
+ self.assertIn("{curly}", content)
+
+ def test_uppercase_does_not_turn_escaped_backslash_into_linebreak(self):
+ # Before the fix, a literal "\n" surviving into hormozi's uppercase
+ # transform became the literal two characters "\" + "N", which
+ # libass reads as a forced line break control code.
+ style = get_style("hormozi")
+ words = _flowing_words(["a\\nb"])
+ content = cr._render_hormozi(words, style, 0.0)
+ self.assertNotIn("\\N", content)
+ self.assertNotIn("\\n", content)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_cli_hook.py b/tests/test_cli_hook.py
new file mode 100644
index 0000000..6da8271
--- /dev/null
+++ b/tests/test_cli_hook.py
@@ -0,0 +1,148 @@
+"""Opening hooks in the CLI render paths.
+
+The MCP tools and the studio hand a clip's hook to the renderer. The CLI paths
+built their clip specs without it, so a hook the AI proposed in `podcli
+process` or one asked for on `podcli studio` never reached the render. These
+cover each hand-off: the AI parse keeps it, every CLI render call passes it,
+and the studio script checks and forwards it.
+"""
+
+import ast
+import json
+import os
+import sys
+import types
+import unittest
+from unittest import mock
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+import cli as cli_mod
+import clip_studio
+from services import ai_cli, ai_provider
+from services import claude_suggest as cs
+
+from test_studio_format import _run_studio, _studio_args
+
+SEGMENTS = [
+ {"start": 10.0, "end": 20.0, "text": "We talked about discipline and habits.", "speaker": "A"},
+ {"start": 20.0, "end": 60.0, "text": "The moment I realized failure was the turning point.", "speaker": "B"},
+]
+BODY = [{"start": 20.0, "end": 58.0}]
+
+
+class SuggestedHookTests(unittest.TestCase):
+ def test_keeps_a_hook_inside_the_clip(self):
+ hook = cs._suggested_hook({"hook": {"start": 40.0, "end": 44.5, "mode": "repeat"}}, 20.0, 58.0, BODY)
+ self.assertEqual(hook, {"start": 40.0, "end": 44.5, "mode": "repeat"})
+
+ def test_reads_numbers_the_model_wrote_as_strings(self):
+ hook = cs._suggested_hook({"hook": {"start": "40.04", "end": "44.5", "mode": "move"}}, 20.0, 58.0, BODY)
+ self.assertEqual(hook, {"start": 40.0, "end": 44.5, "mode": "move"})
+
+ def test_drops_a_hook_outside_the_clip(self):
+ self.assertIsNone(cs._suggested_hook({"hook": {"start": 5.0, "end": 9.0, "mode": "repeat"}}, 20.0, 58.0, BODY))
+
+ def test_drops_an_unknown_mode(self):
+ self.assertIsNone(cs._suggested_hook({"hook": {"start": 40.0, "end": 44.0, "mode": "loop"}}, 20.0, 58.0, BODY))
+
+ def test_drops_a_hook_that_is_not_an_object(self):
+ self.assertIsNone(cs._suggested_hook({"hook": "40-44"}, 20.0, 58.0, BODY))
+ self.assertIsNone(cs._suggested_hook({}, 20.0, 58.0, BODY))
+
+ def test_both_prompts_offer_the_hook(self):
+ prompt = cs._build_prompt("[20.0s] B: hello", 2, 1.0, 3)
+ self.assertIn('"hook"', prompt)
+ self.assertNotIn("{{", prompt)
+
+
+class FindMomentsCarriesHookTests(unittest.TestCase):
+ def setUp(self):
+ self._orig_chain = ai_provider._chain
+ self._orig_run = ai_cli._run_ai_command
+ payload = {"clips": [{
+ "title": "The turning point", "start_second": 20.0, "end_second": 58.0,
+ "segments": BODY, "content_type": "guest_story", "quote": "Failure was the turning point",
+ "hook": {"start": 50.0, "end": 54.0, "mode": "repeat"},
+ }]}
+ ai_provider._chain = lambda: [("cli", "/usr/bin/claude", "claude")]
+ ai_cli._run_ai_command = lambda **kw: types.SimpleNamespace(
+ returncode=0, stdout=json.dumps(payload), stderr="")
+
+ def tearDown(self):
+ ai_provider._chain = self._orig_chain
+ ai_cli._run_ai_command = self._orig_run
+
+ def test_found_moment_keeps_its_hook(self):
+ clips = cs.find_moments_from_text("the turning point", SEGMENTS, [])
+ self.assertEqual(clips[0]["hook"], {"start": 50.0, "end": 54.0, "mode": "repeat"})
+
+
+class ClipHookTests(unittest.TestCase):
+ def test_passes_a_hook_that_fits(self):
+ clip = {"start_second": 20.0, "end_second": 58.0, "segments": BODY,
+ "hook": {"start": 40.0, "end": 44.0, "mode": "move"}}
+ self.assertEqual(cli_mod._clip_hook(clip), {"start": 40.0, "end": 44.0, "mode": "move"})
+
+ def test_no_hook_is_none(self):
+ self.assertIsNone(cli_mod._clip_hook({"start_second": 20.0, "end_second": 58.0}))
+
+ def test_drops_a_hook_review_moved_out_of_the_clip(self):
+ clip = {"start_second": 20.0, "end_second": 30.0, "segments": None,
+ "hook": {"start": 40.0, "end": 44.0, "mode": "repeat"}}
+ with mock.patch("builtins.print") as printed:
+ self.assertIsNone(cli_mod._clip_hook(clip))
+ self.assertIn("Opening hook dropped", printed.call_args[0][0])
+
+ def test_every_cli_render_passes_the_hook(self):
+ with open(cli_mod.__file__, encoding="utf-8") as f:
+ tree = ast.parse(f.read())
+ calls = [
+ node for node in ast.walk(tree)
+ if isinstance(node, ast.Call) and getattr(node.func, "id", None) == "generate_clip"
+ ]
+ self.assertGreaterEqual(len(calls), 5)
+ for call in calls:
+ self.assertIn("hook", {k.arg for k in call.keywords}, f"generate_clip at line {call.lineno}")
+
+
+class StudioHookTests(unittest.TestCase):
+ HOOK = '{"start": 4.0, "end": 7.5, "mode": "repeat"}'
+
+ def test_studio_hands_the_hook_to_the_script(self):
+ cmd = _run_studio(_studio_args(hook=self.HOOK))
+ self.assertEqual(cmd[cmd.index("--hook") + 1], self.HOOK)
+
+ def test_no_hook_sends_no_flag(self):
+ self.assertNotIn("--hook", _run_studio(_studio_args()))
+
+ def test_script_accepts_a_hook_inside_the_fragment(self):
+ self.assertEqual(clip_studio._hook_arg(self.HOOK, 0.0, 10.0),
+ {"start": 4.0, "end": 7.5, "mode": "repeat"})
+
+ def test_script_stops_on_a_hook_outside_the_fragment(self):
+ with self.assertRaises(SystemExit) as stop:
+ clip_studio._hook_arg(self.HOOK, 5.0, 20.0)
+ self.assertIn("--hook", str(stop.exception))
+
+ def test_script_stops_on_malformed_json(self):
+ with self.assertRaises(SystemExit):
+ clip_studio._hook_arg("{start: 4", 0.0, 10.0)
+
+ def test_no_hook_is_none(self):
+ self.assertIsNone(clip_studio._hook_arg(None, 0.0, 10.0))
+
+ def test_fragment_render_passes_the_hook(self):
+ hook = {"start": 4.0, "end": 7.5, "mode": "move"}
+ with mock.patch("services.clip_generator.generate_clip",
+ return_value={"output_path": "/tmp/x.mp4"}) as gen:
+ clip_studio._render_fragment("v.mp4", 0.0, 10.0, [], "hormozi", "center",
+ "fragment", "/tmp", hook=hook)
+ self.assertEqual(gen.call_args.kwargs["hook"], hook)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_clip_generator.py b/tests/test_clip_generator.py
index 1f212f8..21618cc 100644
--- a/tests/test_clip_generator.py
+++ b/tests/test_clip_generator.py
@@ -519,5 +519,273 @@ class OutputPathReservationTests(unittest.TestCase):
self.assertTrue(base.endswith("same_title_short-2"))
+class BoundRangeToSourceTests(unittest.TestCase):
+ """A requested range past the end of the source must be clamped before
+ it reaches ffmpeg's -ss/-t cut, which would otherwise silently stop at
+ EOF and under-deliver instead of failing."""
+
+ def test_unknown_source_duration_leaves_range_untouched(self):
+ end, segs = cg._bound_range_to_source(10.0, 999.0, None, source_duration=0.0)
+ self.assertEqual(end, 999.0)
+ self.assertIsNone(segs)
+
+ def test_end_second_clamped_to_source_duration(self):
+ end, segs = cg._bound_range_to_source(5.0, 120.0, None, source_duration=42.0)
+ self.assertEqual(end, 42.0)
+ self.assertIsNone(segs)
+
+ def test_end_second_within_source_is_unchanged(self):
+ end, _ = cg._bound_range_to_source(5.0, 20.0, None, source_duration=42.0)
+ self.assertEqual(end, 20.0)
+
+ def test_start_at_or_past_source_duration_raises(self):
+ with self.assertRaises(ValueError):
+ cg._bound_range_to_source(42.0, 50.0, None, source_duration=42.0)
+ with self.assertRaises(ValueError):
+ cg._bound_range_to_source(50.0, 60.0, None, source_duration=42.0)
+
+ def test_keep_segments_clamped_and_out_of_range_ones_dropped(self):
+ segs_in = [
+ {"start": 5.0, "end": 10.0},
+ {"start": 35.0, "end": 60.0}, # end runs past source
+ {"start": 50.0, "end": 55.0}, # starts past source entirely
+ ]
+ end, segs = cg._bound_range_to_source(5.0, 60.0, segs_in, source_duration=42.0)
+ self.assertEqual(end, 42.0)
+ self.assertEqual(segs, [
+ {"start": 5.0, "end": 10.0},
+ {"start": 35.0, "end": 42.0},
+ ])
+
+ def test_all_keep_segments_dropped_falls_back_to_none(self):
+ segs_in = [{"start": 50.0, "end": 55.0}]
+ _, segs = cg._bound_range_to_source(5.0, 60.0, segs_in, source_duration=42.0)
+ self.assertIsNone(segs)
+
+
+ shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg/ffprobe not installed"
+)
+class SidecarsAndCleanVariantTests(unittest.TestCase):
+ """Real render, forced onto the ASS path (center crop, no Remotion/node
+ dependency) so it stays fast and deterministic. Covers the sidecar and
+ clean-variant wiring added to generate_clip, which nothing else here
+ exercises end to end."""
+
+ @classmethod
+ def setUpClass(cls):
+ cls.tmpdir = tempfile.mkdtemp(prefix="podcli-sidecar-test-")
+ cls.src = os.path.join(cls.tmpdir, "src.mp4")
+ subprocess.run(
+ [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc=size=320x240:rate=25:duration=3",
+ "-f", "lavfi", "-i", "sine=frequency=440:duration=3",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
+ cls.src,
+ ],
+ check=True, capture_output=True,
+ )
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.tmpdir, ignore_errors=True)
+
+ def test_writes_sidecars_and_clean_variant(self):
+ from services import clip_generator as cg
+
+ words = [
+ {"word": "hello", "start": 0.2, "end": 0.6, "speaker": None},
+ {"word": "world", "start": 0.7, "end": 1.1, "speaker": None},
+ {"word": "testing", "start": 1.3, "end": 1.8, "speaker": None},
+ ]
+ out_dir = os.path.join(self.tmpdir, "out")
+ result = cg.generate_clip(
+ video_path=self.src,
+ start_second=0,
+ end_second=2.5,
+ caption_style="subtle",
+ crop_strategy="center",
+ transcript_words=words,
+ title="sidecar_test",
+ output_dir=out_dir,
+ # Burned captions aren't the point of this test (that's covered
+ # by caption_renderer's own tests), turning them off here keeps
+ # this test to the render pipeline's font/compositor dependencies
+ # out of the way of the sidecar/clean-variant wiring under test.
+ # Sidecars otherwise follow `captions`, so request them
+ # explicitly for this clean (caption-free) export.
+ captions=False,
+ write_subtitles=True,
+ clean_fillers=False,
+ write_clean_variant=True,
+ )
+
+ self.assertTrue(os.path.exists(result["output_path"]))
+
+ self.assertIn("srt_path", result)
+ self.assertTrue(os.path.exists(result["srt_path"]))
+ with open(result["srt_path"], encoding="utf-8") as f:
+ srt_text = f.read()
+ self.assertIn("hello world testing", srt_text)
+
+ self.assertIn("vtt_path", result)
+ self.assertTrue(os.path.exists(result["vtt_path"]))
+ self.assertTrue(open(result["vtt_path"], encoding="utf-8").read().startswith("WEBVTT"))
+
+ self.assertIn("clean_output_path", result)
+ clean_path = result["clean_output_path"]
+ self.assertTrue(os.path.exists(clean_path))
+ self.assertNotEqual(clean_path, result["output_path"])
+
+ # Clean variant: same audio presence/rough duration, no caption burn.
+ self.assertTrue(cg.probe_has_audio_stream(clean_path))
+ main_duration = cg._get_media_duration(result["output_path"])
+ clean_duration = cg._get_media_duration(clean_path)
+ self.assertAlmostEqual(main_duration, clean_duration, delta=0.5)
+
+ def test_duration_is_content_length_not_the_file_with_intro_included(self):
+ from services import clip_generator as cg
+
+ intro_path = os.path.join(self.tmpdir, "intro.mp4")
+ subprocess.run(
+ [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc=size=320x240:rate=25:duration=1",
+ "-f", "lavfi", "-i", "sine=frequency=330:duration=1",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
+ intro_path,
+ ],
+ check=True, capture_output=True,
+ )
+
+ out_dir = os.path.join(self.tmpdir, "out_with_intro")
+ result = cg.generate_clip(
+ video_path=self.src,
+ start_second=0,
+ end_second=2.5,
+ caption_style="subtle",
+ crop_strategy="center",
+ title="intro_duration_test",
+ output_dir=out_dir,
+ captions=False,
+ clean_fillers=False,
+ intro_path=intro_path,
+ )
+ # "duration" is what clip_history and its learnings have always
+ # recorded: the content's own length, not however long the delivered
+ # file plays once an intro is prepended.
+ self.assertAlmostEqual(result["duration"], 2.5, delta=0.3)
+ self.assertIn("output_duration", result)
+ self.assertAlmostEqual(result["output_duration"], 3.5, delta=0.3)
+ self.assertGreater(result["output_duration"], result["duration"])
+
+ def test_video_only_source_does_not_require_an_audio_stream(self):
+ from services import clip_generator as cg
+
+ silent_src = os.path.join(self.tmpdir, "silent_src.mp4")
+ subprocess.run(
+ [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc=size=320x240:rate=25:duration=3",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p",
+ silent_src,
+ ],
+ check=True, capture_output=True,
+ )
+ self.assertFalse(cg.probe_has_audio_stream(silent_src))
+
+ out_dir = os.path.join(self.tmpdir, "out_silent")
+ result = cg.generate_clip(
+ video_path=silent_src,
+ start_second=0,
+ end_second=2.5,
+ caption_style="subtle",
+ crop_strategy="center",
+ title="silent_source_test",
+ output_dir=out_dir,
+ captions=False,
+ clean_fillers=False,
+ )
+ self.assertTrue(os.path.exists(result["output_path"]))
+
+ def test_no_sidecars_when_captions_off_and_not_explicitly_requested(self):
+ from services import clip_generator as cg
+
+ words = [{"word": "hello", "start": 0.2, "end": 0.6, "speaker": None}]
+ out_dir = os.path.join(self.tmpdir, "out_no_sidecars")
+ result = cg.generate_clip(
+ video_path=self.src,
+ start_second=0,
+ end_second=2.5,
+ caption_style="subtle",
+ crop_strategy="center",
+ transcript_words=words,
+ title="no_sidecar_test",
+ output_dir=out_dir,
+ captions=False,
+ clean_fillers=False,
+ )
+ self.assertTrue(os.path.exists(result["output_path"]))
+ self.assertNotIn("srt_path", result)
+ self.assertNotIn("vtt_path", result)
+ output_base, _ = os.path.splitext(result["output_path"])
+ self.assertFalse(os.path.exists(f"{output_base}.srt"))
+ self.assertFalse(os.path.exists(f"{output_base}.vtt"))
+
+ def test_rerender_clears_stale_sidecars_and_clean_variant(self):
+ from services import clip_generator as cg
+
+ words = [
+ {"word": "hello", "start": 0.2, "end": 0.6, "speaker": None},
+ {"word": "world", "start": 0.7, "end": 1.1, "speaker": None},
+ ]
+ out_dir = os.path.join(self.tmpdir, "out_rerender")
+ first = cg.generate_clip(
+ video_path=self.src,
+ start_second=0,
+ end_second=2.5,
+ caption_style="subtle",
+ crop_strategy="center",
+ transcript_words=words,
+ title="rerender_test",
+ output_dir=out_dir,
+ captions=False,
+ write_subtitles=True,
+ clean_fillers=False,
+ write_clean_variant=True,
+ )
+ output_base, _ = os.path.splitext(first["output_path"])
+ srt_path = f"{output_base}.srt"
+ vtt_path = f"{output_base}.vtt"
+ clean_path = f"{output_base}_clean.mp4"
+ self.assertTrue(os.path.exists(srt_path))
+ self.assertTrue(os.path.exists(vtt_path))
+ self.assertTrue(os.path.exists(clean_path))
+
+ # Re-render the same clip (same title/output_dir, so the same
+ # output path) without subtitles or a clean variant this time.
+ # Without cleanup, the previous render's .srt/.vtt/_clean.mp4
+ # would sit next to the new clip describing stale content.
+ cg._reserved_output_paths.discard(first["output_path"])
+ second = cg.generate_clip(
+ video_path=self.src,
+ start_second=0,
+ end_second=2.5,
+ caption_style="subtle",
+ crop_strategy="center",
+ transcript_words=words,
+ title="rerender_test",
+ output_dir=out_dir,
+ captions=False,
+ clean_fillers=False,
+ write_clean_variant=False,
+ )
+ self.assertEqual(second["output_path"], first["output_path"])
+ self.assertFalse(os.path.exists(srt_path))
+ self.assertFalse(os.path.exists(vtt_path))
+ self.assertFalse(os.path.exists(clean_path))
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_corrections.py b/tests/test_corrections.py
index 7560038..99f487c 100644
--- a/tests/test_corrections.py
+++ b/tests/test_corrections.py
@@ -95,6 +95,103 @@ class CorrectionsTests(unittest.TestCase):
_, s = corrections.apply_corrections([], segments)
self.assertEqual(s[0]["text"], "we maintain A.I. systems")
+ def test_apply_corrections_merges_multiword_word_run(self):
+ # "open AI" split across two words must merge into one caption word,
+ # not just fix the segment text and leave "open"/"AI" as-is.
+ self._set({"open AI": "OpenAI"})
+ words = [
+ {"word": "open", "start": 1.0, "end": 1.4, "speaker": "SPEAKER_00"},
+ {"word": "AI", "start": 1.4, "end": 1.8, "speaker": "SPEAKER_00"},
+ ]
+ segments = [{"text": "I love open AI"}]
+ w, s = corrections.apply_corrections(words, segments)
+ self.assertEqual([x["word"] for x in w], ["OpenAI"])
+ self.assertAlmostEqual(w[0]["start"], 1.0)
+ self.assertAlmostEqual(w[0]["end"], 1.8)
+ self.assertEqual(w[0]["speaker"], "SPEAKER_00")
+ self.assertEqual(s[0]["text"], "I love OpenAI")
+
+ def test_apply_corrections_multiword_merge_is_case_insensitive(self):
+ self._set({"open AI": "OpenAI"})
+ words = [
+ {"word": "Open", "start": 0.0, "end": 0.5},
+ {"word": "ai", "start": 0.5, "end": 1.0},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ self.assertEqual([x["word"] for x in w], ["OpenAI"])
+
+ def test_apply_corrections_multiword_merge_ignores_punctuation(self):
+ self._set({"open AI": "OpenAI"})
+ words = [
+ {"word": "open", "start": 0.0, "end": 0.5},
+ {"word": "AI.", "start": 0.5, "end": 1.0},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ # Punctuation is ignored for matching, but the last word's trailing
+ # punctuation still belongs on the merged word: "open AI." should
+ # read as "OpenAI.", not drop the sentence end.
+ self.assertEqual([x["word"] for x in w], ["OpenAI."])
+
+ def test_apply_corrections_multiword_merge_carries_last_words_trailing_punctuation(self):
+ self._set({"open AI": "OpenAI"})
+ words = [
+ {"word": "open", "start": 1.0, "end": 1.4, "speaker": "SPEAKER_00", "confidence": 0.9},
+ {"word": "AI.", "start": 1.4, "end": 1.8, "speaker": "SPEAKER_01", "confidence": 0.4},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ self.assertEqual([x["word"] for x in w], ["OpenAI."])
+ # First word's other fields carry onto the merged word...
+ self.assertEqual(w[0]["speaker"], "SPEAKER_00")
+ # ...except confidence, which takes the minimum across the run
+ # rather than silently reporting the first word's (possibly higher)
+ # confidence for a merge that includes a less-confident word.
+ self.assertAlmostEqual(w[0]["confidence"], 0.4)
+
+ def test_apply_corrections_multiword_merge_multi_word_replacement_keeps_trailing_punctuation(self):
+ self._set({"open ai": "Open AI"})
+ words = [
+ {"word": "open", "start": 0.0, "end": 1.0},
+ {"word": "ai.", "start": 1.0, "end": 2.0},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ # The trailing punctuation lands on the final replacement word only.
+ self.assertEqual([x["word"] for x in w], ["Open", "AI."])
+
+ def test_apply_corrections_multiword_replacement_splits_time_across_words(self):
+ # A multi-word replacement distributes the merged span evenly across
+ # its own word count, rather than collapsing into a single word.
+ self._set({"open ai": "Open AI"})
+ words = [
+ {"word": "open", "start": 0.0, "end": 1.0},
+ {"word": "ai", "start": 1.0, "end": 2.0},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ self.assertEqual([x["word"] for x in w], ["Open", "AI"])
+ self.assertAlmostEqual(w[0]["start"], 0.0)
+ self.assertAlmostEqual(w[0]["end"], 1.0)
+ self.assertAlmostEqual(w[1]["start"], 1.0)
+ self.assertAlmostEqual(w[1]["end"], 2.0)
+
+ def test_apply_corrections_multiword_longest_match_wins_in_words(self):
+ self._set({"open AI": "OpenAI", "AI": "A.I."})
+ words = [
+ {"word": "open", "start": 0.0, "end": 0.5},
+ {"word": "AI", "start": 0.5, "end": 1.0},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ self.assertEqual([x["word"] for x in w], ["OpenAI"])
+
+ def test_apply_corrections_leaves_non_matching_words_alone(self):
+ self._set({"open AI": "OpenAI"})
+ words = [
+ {"word": "I", "start": 0.0, "end": 0.2},
+ {"word": "love", "start": 0.2, "end": 0.5},
+ {"word": "open", "start": 0.5, "end": 0.8},
+ {"word": "AI", "start": 0.8, "end": 1.0},
+ ]
+ w, _ = corrections.apply_corrections(words, [])
+ self.assertEqual([x["word"] for x in w], ["I", "love", "OpenAI"])
+
def test_apply_corrections_returns_same_list_objects(self):
self._set({"Foo": "Bar"})
words = [{"word": "Foo"}]
diff --git a/tests/test_engine_comparison.py b/tests/test_engine_comparison.py
new file mode 100644
index 0000000..5e42cb4
--- /dev/null
+++ b/tests/test_engine_comparison.py
@@ -0,0 +1,200 @@
+"""Tests for backend.services.engine_comparison: disagreement scoring and report output."""
+
+import json
+import os
+import sys
+import tempfile
+import unittest
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import engine_comparison as ec
+
+
+class NormalizeWordTests(unittest.TestCase):
+ def test_strips_punctuation(self):
+ self.assertEqual(ec.normalize_word("hello,"), "hello")
+
+ def test_casefolds(self):
+ self.assertEqual(ec.normalize_word("HELLO"), "hello")
+
+ def test_nfkc_normalizes(self):
+ # Fullwidth "A" (U+FF21) NFKC-normalizes to ASCII "a" after casefold.
+ self.assertEqual(ec.normalize_word("A"), "a")
+
+ def test_keeps_marks_and_numbers(self):
+ self.assertEqual(ec.normalize_word("café2"), "café2")
+
+ def test_empty_punctuation_only_becomes_empty_string(self):
+ self.assertEqual(ec.normalize_word("..."), "")
+
+
+class WordLevenshteinTests(unittest.TestCase):
+ def test_identical_sequences(self):
+ self.assertEqual(ec.word_levenshtein(["a", "b"], ["a", "b"]), 0)
+
+ def test_one_substitution(self):
+ self.assertEqual(ec.word_levenshtein(["a", "b", "c"], ["a", "x", "c"]), 1)
+
+ def test_one_insertion(self):
+ self.assertEqual(ec.word_levenshtein(["a", "c"], ["a", "b", "c"]), 1)
+
+ def test_against_empty(self):
+ self.assertEqual(ec.word_levenshtein([], ["a", "b"]), 2)
+ self.assertEqual(ec.word_levenshtein(["a", "b"], []), 2)
+
+
+class DisagreementRatioTests(unittest.TestCase):
+ def test_identical_text_is_zero(self):
+ self.assertEqual(ec.disagreement_ratio("hello world", "hello world"), 0.0)
+
+ def test_punctuation_and_case_do_not_count_as_disagreement(self):
+ self.assertEqual(ec.disagreement_ratio("Hello, World!", "hello world"), 0.0)
+
+ def test_one_word_different_out_of_two(self):
+ self.assertEqual(ec.disagreement_ratio("hello world", "hello word"), 0.5)
+
+ def test_completely_different_short_texts(self):
+ self.assertEqual(ec.disagreement_ratio("hello", "goodbye"), 1.0)
+
+ def test_both_empty_is_zero_not_a_crash(self):
+ self.assertEqual(ec.disagreement_ratio("", ""), 0.0)
+
+
+class BuildWindowsTests(unittest.TestCase):
+ def test_splits_into_fixed_windows_and_flags_both_empty(self):
+ words_a = [{"word": "hello", "start": 1.0, "end": 1.5}]
+ words_b = [{"word": "hallo", "start": 1.0, "end": 1.5}]
+ windows = ec.build_windows(words_a, words_b, total_duration=45.0, window_seconds=20.0)
+ self.assertEqual(len(windows), 3)
+ self.assertEqual(windows[0]["start"], 0.0)
+ self.assertEqual(windows[0]["end"], 20.0)
+ self.assertFalse(windows[0]["both_empty"])
+ self.assertTrue(windows[1]["both_empty"]) # 20-40s: no words from either
+ self.assertEqual(windows[1]["disagreement"], 0.0)
+
+ def test_word_assigned_to_window_holding_its_midpoint(self):
+ # Word spans 19-21s, straddling the 20s window boundary; its
+ # midpoint (20.0) belongs to the second window.
+ words_a = [{"word": "straddle", "start": 19.0, "end": 21.0}]
+ windows = ec.build_windows(words_a, [], total_duration=40.0, window_seconds=20.0)
+ self.assertEqual(windows[0]["text_a"], "")
+ self.assertEqual(windows[1]["text_a"], "straddle")
+
+
+class CompareEnginesTests(unittest.TestCase):
+ def _fake_transcribe(self, words_by_engine):
+ def fn(file_path, model_size=None, engine=None, language=None,
+ enable_diarization=None, start_seconds=None, duration_seconds=None):
+ words = words_by_engine[engine]
+ duration = max((w["end"] for w in words), default=0.0)
+ return {"words": words, "duration": duration, "engine": engine}
+ return fn
+
+ def test_compares_two_engines_and_scores_overall_disagreement(self):
+ fake = self._fake_transcribe({
+ "whispercpp": [{"word": "hello", "start": 0.0, "end": 0.5}],
+ "omnilingual": [{"word": "hallo", "start": 0.0, "end": 0.5}],
+ })
+ report = ec.compare_engines(
+ "/fake/video.mp4", "whispercpp", "omnilingual",
+ duration_seconds=20.0, transcribe_fn=fake,
+ )
+ self.assertEqual(report["engine_a"], "whispercpp")
+ self.assertEqual(report["engine_b"], "omnilingual")
+ self.assertEqual(len(report["windows"]), 1)
+ self.assertEqual(report["overall_disagreement"], 1.0)
+ self.assertEqual(report["note"], "disagreement between the two engines' output, not accuracy against a transcript")
+
+ def test_identical_output_scores_zero_disagreement(self):
+ fake = self._fake_transcribe({
+ "whispercpp": [{"word": "hello", "start": 0.0, "end": 0.5}],
+ "omnilingual": [{"word": "hello", "start": 0.0, "end": 0.5}],
+ })
+ report = ec.compare_engines(
+ "/fake/video.mp4", "whispercpp", "omnilingual",
+ duration_seconds=20.0, transcribe_fn=fake,
+ )
+ self.assertEqual(report["overall_disagreement"], 0.0)
+
+ def test_writes_json_and_html_when_output_dir_given(self):
+ fake = self._fake_transcribe({
+ "whispercpp": [{"word": "hello", "start": 0.0, "end": 0.5}],
+ "omnilingual": [{"word": "hallo", "start": 0.0, "end": 0.5}],
+ })
+ with tempfile.TemporaryDirectory() as tmp:
+ report = ec.compare_engines(
+ "/fake/video.mp4", "whispercpp", "omnilingual",
+ duration_seconds=20.0, transcribe_fn=fake, output_dir=tmp,
+ )
+ self.assertTrue(os.path.exists(report["json_path"]))
+ self.assertTrue(os.path.exists(report["html_path"]))
+ with open(report["json_path"], encoding="utf-8") as f:
+ on_disk = json.load(f)
+ self.assertEqual(on_disk["overall_disagreement"], report["overall_disagreement"])
+
+
+class RenderHtmlEscapingTests(unittest.TestCase):
+ def _report(self, text_a, text_b):
+ return {
+ "engine_a": "whispercpp",
+ "engine_b": "omnilingual",
+ "file_path": "/fake/video.mp4",
+ "overall_disagreement": 0.5,
+ "windows": [
+ {"start": 0.0, "end": 20.0, "text_a": text_a, "text_b": text_b,
+ "both_empty": False, "disagreement": 0.5},
+ ],
+ }
+
+ def test_script_closing_tag_in_transcript_text_is_neutralized(self):
+ html_out = ec.render_html(self._report("</script><script>alert(1)</script>", "normal"), None)
+ self.assertNotIn("</script><script>alert(1)</script>", html_out)
+ # The data must still round-trip through JS correctly. Check the
+ # escaped marker is present instead of a raw closing tag.
+ self.assertIn("\\u003c/script", html_out)
+
+ def test_script_closing_tag_with_mixed_case_is_also_neutralized(self):
+ # A case-sensitive "</script" replacement lets "</SCRIPT>" (or any
+ # other casing) through, since HTML tag matching is case-insensitive
+ # but a literal string replace is not.
+ html_out = ec.render_html(
+ self._report("</SCRIPT><img onerror=alert(1)>", "normal"), None
+ )
+ self.assertNotIn("</SCRIPT><img onerror=alert(1)>", html_out)
+ self.assertIn("\\u003c/SCRIPT>\\u003cimg onerror=alert(1)>", html_out)
+
+ def test_html_tags_in_engine_names_are_escaped(self):
+ report = self._report("hello", "world")
+ report["engine_a"] = "<img src=x onerror=alert(1)>"
+ html_out = ec.render_html(report, None)
+ self.assertNotIn("<img src=x onerror=alert(1)>", html_out)
+ self.assertIn("&lt;img", html_out)
+
+ def test_no_audio_renders_placeholder_not_a_broken_tag(self):
+ html_out = ec.render_html(self._report("a", "b"), None)
+ self.assertIn("No sample audio", html_out)
+ self.assertNotIn("<audio", html_out)
+
+ def test_audio_path_is_wired_into_audio_tag(self):
+ html_out = ec.render_html(self._report("a", "b"), "sample.wav")
+ self.assertIn('src="sample.wav"', html_out)
+
+
+class JsonForScriptTagTests(unittest.TestCase):
+ def test_escapes_line_separator_and_paragraph_separator(self):
+ # U+2028/U+2029 are valid JSON string characters but some JS engines
+ # treat them as line terminators even inside a string literal, which
+ # can truncate the embedded JSON mid-statement.
+ out = ec._json_for_script_tag({"text": "line one
line two
line three"})
+ self.assertNotIn("
", out)
+ self.assertNotIn("
", out)
+ self.assertIn("\\u2028", out)
+ self.assertIn("\\u2029", out)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_glyph_coverage.py b/tests/test_glyph_coverage.py
new file mode 100644
index 0000000..b50d17e
--- /dev/null
+++ b/tests/test_glyph_coverage.py
@@ -0,0 +1,106 @@
+"""Tests for backend.services.glyph_coverage: pre-render font coverage check."""
+
+import os
+import shutil
+import sys
+import unittest
+from unittest import mock
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import glyph_coverage as gc
+
+
+class RemotionCoverageTests(unittest.TestCase):
+ """No real fontconfig/Remotion needed: this is a static subset table."""
+
+ def test_latin_text_is_covered(self):
+ self.assertIsNone(gc.check_remotion_font_coverage("Hello world"))
+
+ def test_georgian_text_is_covered(self):
+ self.assertIsNone(gc.check_remotion_font_coverage("მიშა"))
+
+ def test_cyrillic_text_is_flagged(self):
+ warning = gc.check_remotion_font_coverage("Привет")
+ self.assertIsNotNone(warning)
+ self.assertIn("П", warning)
+
+ def test_empty_text_is_covered(self):
+ self.assertIsNone(gc.check_remotion_font_coverage(""))
+
+ def test_whitespace_and_duplicates_are_not_double_reported(self):
+ # "мама" repeats letters; the count should reflect unique characters
+ # (2: м, а), not every occurrence across the repeated word.
+ warning = gc.check_remotion_font_coverage("мама мама")
+ self.assertIsNotNone(warning)
+ self.assertIn("2 unique character", warning)
+
+
+class AssCoverageUnitTests(unittest.TestCase):
+ """fc-match/fc-query calls mocked to isolate the coverage logic."""
+
+ def test_returns_none_when_font_cannot_be_resolved(self):
+ with mock.patch.object(gc, "resolve_ass_font_path", return_value=None):
+ self.assertIsNone(gc.check_ass_font_coverage("text", "Nonexistent Font"))
+
+ def test_returns_none_when_charset_cannot_be_read(self):
+ with mock.patch.object(gc, "resolve_ass_font_path", return_value="/fake/font.ttf"), \
+ mock.patch.object(gc, "_font_charset_ranges", return_value=None):
+ self.assertIsNone(gc.check_ass_font_coverage("text", "Some Font"))
+
+ def test_flags_characters_outside_the_charset(self):
+ with mock.patch.object(gc, "resolve_ass_font_path", return_value="/fake/font.ttf"), \
+ mock.patch.object(gc, "_font_charset_ranges", return_value=[(0x20, 0x7E)]):
+ warning = gc.check_ass_font_coverage("hello მიშა", "Some Font")
+ self.assertIsNotNone(warning)
+ self.assertIn("მ", warning)
+
+ def test_empty_text_returns_none(self):
+ self.assertIsNone(gc.check_ass_font_coverage("", "Some Font"))
+
+
+class DispatchTests(unittest.TestCase):
+ def test_use_ass_true_calls_ass_check(self):
+ with mock.patch.object(gc, "check_ass_font_coverage", return_value="ass-warning") as ass_mock, \
+ mock.patch.object(gc, "check_remotion_font_coverage") as remotion_mock:
+ result = gc.check_caption_font_coverage("text", "Font", False, use_ass=True)
+ self.assertEqual(result, "ass-warning")
+ ass_mock.assert_called_once()
+ remotion_mock.assert_not_called()
+
+ def test_use_ass_false_calls_remotion_check(self):
+ with mock.patch.object(gc, "check_ass_font_coverage") as ass_mock, \
+ mock.patch.object(gc, "check_remotion_font_coverage", return_value="remotion-warning") as remotion_mock:
+ result = gc.check_caption_font_coverage("text", "Font", False, use_ass=False)
+ self.assertEqual(result, "remotion-warning")
+ remotion_mock.assert_called_once()
+ ass_mock.assert_not_called()
+
+
+ shutil.which("fc-match") and shutil.which("fc-query"), "fontconfig not installed"
+)
+class AssCoverageRealFontconfigTests(unittest.TestCase):
+ """Arial has no Georgian glyphs on a stock macOS/Linux fontconfig setup,
+ so it's a reliable real-world case for "font resolves fine, coverage
+ doesn't"."""
+
+ def test_arial_covers_ascii(self):
+ self.assertIsNone(gc.check_ass_font_coverage("Hello world", "Arial"))
+
+ def test_arial_does_not_cover_georgian(self):
+ warning = gc.check_ass_font_coverage("hello მიშა", "Arial")
+ self.assertIsNotNone(warning)
+ self.assertIn("Arial", warning)
+
+ def test_unresolvable_font_name_does_not_raise(self):
+ self.assertIsNone(
+ gc.check_ass_font_coverage("text", "Definitely Not A Real Font XYZ")
+ )
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_learnings.py b/tests/test_learnings.py
new file mode 100644
index 0000000..ea226dd
--- /dev/null
+++ b/tests/test_learnings.py
@@ -0,0 +1,71 @@
+"""Tests for the performance-learnings digest: median aggregation and the
+small-bucket suppression that keeps a single viral or flop clip from reading
+as a trend."""
+
+import os
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services.integrations.youtube import learnings
+
+
+def _clip(content_type, retention, views=1000, ctr=5.0, caption_style="hormozi", duration=30):
+ return {
+ "title": f"clip-{content_type}-{retention}",
+ "content_type": content_type,
+ "caption_style": caption_style,
+ "duration": duration,
+ "metrics": {"retention": retention, "ctr": ctr, "views": views},
+ }
+
+
+class AggTests(unittest.TestCase):
+ def test_uses_median_not_mean(self):
+ # One outlier (90) would drag a mean way up; the median should stay
+ # anchored to the typical clip.
+ clips = [_clip("story", r) for r in [20, 22, 21, 90]]
+ out = learnings._agg(clips, lambda c: c["content_type"])
+ self.assertEqual(len(out), 1)
+ self.assertEqual(out[0]["ret"], 21.5)
+ self.assertEqual(out[0]["n"], 4)
+
+ def test_suppresses_buckets_under_minimum_size(self):
+ clips = [_clip("story", 20), _clip("story", 22), _clip("story", 24)]
+ out = learnings._agg(clips, lambda c: c["content_type"])
+ self.assertEqual(out, [])
+
+ def test_keeps_bucket_at_minimum_size(self):
+ clips = [_clip("story", r) for r in [10, 20, 30, 40]]
+ out = learnings._agg(clips, lambda c: c["content_type"])
+ self.assertEqual(len(out), 1)
+ self.assertEqual(out[0]["n"], 4)
+
+
+class WriteLearningsTests(unittest.TestCase):
+ def test_small_bucket_is_dropped_from_output(self):
+ # "video" has only 3 clips (below MIN_BUCKET_SIZE) and must not show
+ # up in the rendered "By moment type" section.
+ clips = [_clip("story", r) for r in [10, 20, 30, 40]] + [
+ _clip("video", r) for r in [60, 70, 80]
+ ]
+ with tempfile.TemporaryDirectory() as tmp:
+ with mock.patch.object(learnings, "load_clips_history", return_value=clips), mock.patch.object(
+ learnings, "paths", {"knowledge": tmp}
+ ):
+ path = learnings.write_learnings(min_clips=3)
+ self.assertIsNotNone(path)
+ with open(path, encoding="utf-8") as f:
+ content = f.read()
+ self.assertIn("story", content)
+ self.assertNotIn("video:", content)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_multicam.py b/tests/test_multicam.py
index 4ea2c34..cd81f9f 100644
--- a/tests/test_multicam.py
+++ b/tests/test_multicam.py
@@ -252,6 +252,57 @@ def test_sync_plan_render_and_export_a_three_camera_episode(sandbox):
assert abs(mc.sync_session(session, force=True).source(wide.id).offset - 2.2) < 0.005
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_a_stems_failure_does_not_strand_a_video_with_no_outputs_record(episode, monkeypatch):
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ mc.update_mapping(session, {"sources": [
+ {"id": s.id, "role": "camera", "person": "nika" if "one" in os.path.basename(s.path) else "ana"}
+ for s in session.sources if s.kind == "video"
+ ]})
+ session = mc.plan_session(mc.sync_session(session))
+ out_dir = mc._output_dir(session)
+
+ monkeypatch.setattr(mc, "_render_stems", lambda *a, **k: (_ for _ in ()).throw(RuntimeError("boom")))
+ with pytest.raises(RuntimeError, match="boom"):
+ mc.render_session(session)
+
+ # Nothing half-rendered should land at the canonical output path, and the
+ # session must not record a video it never finished publishing.
+ assert not (out_dir / "episode.mp4").exists()
+ assert not any(out_dir.glob("*.publishing"))
+ session = mc.MulticamSession.load(session.session_id)
+ assert not session.outputs.get("video")
+
+ monkeypatch.undo()
+ outputs = mc.render_session(session)
+ assert os.path.exists(outputs["video"]) and len(outputs["stems"]) == 2
+ assert not any(out_dir.glob("*.publishing"))
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_a_rerender_keeps_stable_paths_and_drops_stems_nobody_needs(episode):
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ mc.update_mapping(session, {"sources": [
+ {"id": s.id, "role": "camera", "person": "nika" if "one" in os.path.basename(s.path) else "ana"}
+ for s in session.sources if s.kind == "video"
+ ]})
+ session = mc.plan_session(mc.sync_session(session))
+ out_dir = mc._output_dir(session)
+ first = mc.render_session(session)
+ assert first["video"] == str(out_dir / "episode.mp4")
+
+ stale = out_dir / "former-guest.wav"
+ stale.write_bytes(b"old stem")
+ session = mc.MulticamSession.load(session.session_id)
+ session.outputs = {**session.outputs, "stems": [*session.outputs["stems"], str(stale)], "render_key": ""}
+ session.save()
+
+ second = mc.render_session(session)
+ assert second["video"] == first["video"]
+ assert not stale.exists()
+ assert all(os.path.exists(p) for p in second["stems"])
+
+
@pytest.mark.skipif(not shutil.which("ffmpeg"), reason="ffmpeg not installed")
def test_sync_measures_clock_drift(sandbox):
folder = sandbox / "drift"
@@ -271,6 +322,42 @@ def test_sync_measures_clock_drift(sandbox):
assert abs(moved.offset - 4.0) < 0.005
+def test_sync_review_reasons_flag_a_fit_that_looks_clean_but_isnt():
+ import services.multicam_signal as sig
+
+ clean = sig.ClockFit(offset=0.0, speed=1.0, residual_ms=5.0, checkpoints=5, total_checkpoints=5,
+ residual_all_ms=5.0, speed_fallback=False)
+ good_match = sig.CoarseMatch(lag_seconds=0.0, score=30.0, peak_ratio=0.1)
+ assert mc._sync_review_reasons(clean, good_match, overlap_seconds=1200.0) == []
+
+ # Speed was forced back to 1.0: flagged for that even though the inlier
+ # residual (5 ms) is tidy.
+ bad_fallback = sig.ClockFit(offset=0.0, speed=1.0, residual_ms=5.0, checkpoints=2, total_checkpoints=2,
+ residual_all_ms=500.0, speed_fallback=True)
+ reasons = mc._sync_review_reasons(bad_fallback, good_match, overlap_seconds=60.0)
+ assert any("implausible" in r for r in reasons)
+
+ # One checkpoint was a real outlier and got correctly dropped (4 of 5
+ # survive, under the 40% dropped threshold): residual_all_ms is large
+ # only because it still includes that outlier. The fit the sync actually
+ # used is the tidy inlier one, so this must not flag for review.
+ one_outlier_dropped = sig.ClockFit(offset=0.0, speed=1.0, residual_ms=5.0, checkpoints=4, total_checkpoints=5,
+ residual_all_ms=500.0, speed_fallback=False)
+ assert mc._sync_review_reasons(one_outlier_dropped, good_match, overlap_seconds=60.0) == []
+
+ # Too few checkpoints over a long overlap, and most checkpoints dropped.
+ sparse = sig.ClockFit(offset=0.0, speed=1.0, residual_ms=1.0, checkpoints=1, total_checkpoints=6,
+ residual_all_ms=1.0, speed_fallback=False)
+ reasons = mc._sync_review_reasons(sparse, good_match, overlap_seconds=900.0)
+ assert any("checkpoints" in r and "outliers" in r for r in reasons)
+ assert any("10+ minute overlap" in r for r in reasons)
+
+ # A weak correlation peak alone should trigger review even if the fit is tidy.
+ weak_match = sig.CoarseMatch(lag_seconds=0.0, score=30.0, peak_ratio=0.7)
+ reasons = mc._sync_review_reasons(clean, weak_match, overlap_seconds=60.0)
+ assert any("correlation" in r for r in reasons)
+
+
@pytest.mark.skipif(not shutil.which("ffmpeg"), reason="ffmpeg not installed")
def test_shared_audio_falls_back_to_diarization(sandbox, monkeypatch):
from services import speaker_detection
@@ -383,6 +470,31 @@ def test_exports_keep_stereo_sides_apart_and_honor_camera_timecode(sandbox):
assert all(Fraction(c.get("start").rstrip("s")) >= 3600 for c in cam_clips)
+def test_exports_point_a_second_audio_stream_at_the_right_track_not_channel_one(sandbox):
+ """A mic file with one mono stream per mic (a camera's second XLR input, say).
+
+ audio_stream_index picks the stream; the editor timelines must land the
+ mic's track/channel at the position that stream actually sits at in the
+ file, not always at channel 1.
+ """
+ cam = _source("cam.mp4", role="camera", person="host", offset=0.0, duration=40.0)
+ mic = _source("mic.mp4", kind="video", role="mic", person="host", offset=0.0, duration=40.0,
+ audio_stream_count=2, audio_stream_channels=[1, 1], audio_stream_index=1)
+ session = mc.MulticamSession(session_id="abc123abc905", name="ep", people=[mc.Person("host", "Host")],
+ sources=[cam, mic], reference_id="mic",
+ cuts=[{"start": 0.0, "end": 40.0, "source_id": "cam"}])
+
+ premiere = ET.parse(mc.export_xml(session, "premiere")).getroot()
+ assert premiere.findtext("./sequence/media/audio/track/clipitem/sourcetrack/trackindex") == "2"
+ mic_file = next(f for f in premiere.iter("file") if f.findtext("name") == "mic.mp4")
+ assert mic_file.findtext("media/audio/channelcount") == "2"
+
+ fcp = ET.parse(mc.export_xml(session, "fcpxml")).getroot()
+ assert {c.get("srcCh") for c in fcp.iter("audio-channel-source")} == {"2"}
+ mic_asset = next(a for a in fcp.iter("asset") if a.get("name") == "mic.mp4")
+ assert mic_asset.get("audioChannels") == "2"
+
+
# --- podcli multicam ------------------------------------------------------------
def run_cli(monkeypatch, *argv):
@@ -526,6 +638,169 @@ def test_cli_json_mode_applies_a_cut_and_builds_the_preview(episode, monkeypatch
assert "error" in json.loads(capsys.readouterr().out)
+def test_preview_stills_regenerate_after_a_nudge_instead_of_serving_a_stale_frame(episode):
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ mc.update_mapping(session, {"sources": [
+ {"id": s.id, "role": "camera", "person": "nika" if "one" in os.path.basename(s.path) else "ana"}
+ for s in session.sources if s.kind == "video"
+ ]})
+ session = mc.sync_session(session)
+ cam = next(s for s in session.sources if os.path.basename(s.path) == "cam_one.mp4")
+
+ frames = mc.previews(session, looks=True, at=cam.timeline_start() + 1.0)
+ before = frames["cameras"][cam.id]
+ look_before = frames["looks"][cam.id]["natural"]
+ assert os.path.exists(before) and os.path.exists(look_before)
+
+ # Nudging changes the offset->source mapping for the same timeline moment,
+ # so the cache key must change and a fresh still must be rendered: reusing
+ # the old file would show the pre-nudge frame.
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "nudge": 2.0}]})
+ session = mc.MulticamSession.load(session.session_id)
+ cam = session.source(cam.id)
+ frames = mc.previews(session, looks=True, at=cam.timeline_start() + 1.0)
+ after = frames["cameras"][cam.id]
+ look_after = frames["looks"][cam.id]["natural"]
+ assert after != before and os.path.exists(after)
+ assert look_after != look_before and os.path.exists(look_after)
+ # The pre-nudge stills aren't left behind: every nudge would otherwise
+ # pile up one more frame and one more look per camera, forever.
+ assert not os.path.exists(before)
+ assert not os.path.exists(look_before)
+
+
+def test_reopening_a_session_with_an_unchanged_camera_keeps_its_sync(episode):
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ session = mc.sync_session(session)
+ cam = next(s for s in session.sources if os.path.basename(s.path) == "cam_one.mp4")
+ assert cam.synced
+
+ reopened = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ assert reopened.session_id == session.session_id
+ assert reopened.source(cam.id).synced
+ assert reopened.source(cam.id).offset == cam.offset
+
+
+def test_source_changed_on_disk_treats_a_zero_zero_identity_as_unknown(tmp_path):
+ path = tmp_path / "f.mp4"
+ path.write_bytes(b"x")
+ s = _source(str(path), file_size=0, file_mtime_ns=0)
+ assert mc._source_changed_on_disk(s) is False
+
+
+def test_reopening_an_old_shape_session_backfills_identity_without_resetting_sync(episode):
+ """A session saved before file_size/file_mtime_ns existed has both at 0.
+
+ That must read as "no fingerprint recorded yet", not "the file shrank to
+ nothing": reopening it should stamp the real identity in and leave sync,
+ cuts and range exactly as they were.
+ """
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ session = mc.sync_session(session)
+ session = mc.plan_session(session)
+ cuts_before = session.cuts
+ cam = next(s for s in session.sources if os.path.basename(s.path) == "cam_one.mp4")
+ offset_before = cam.offset
+ assert cuts_before and cam.synced
+
+ # Simulate the old session shape directly on disk, as if saved before
+ # file identity was tracked.
+ for s in session.sources:
+ s.file_size = s.file_mtime_ns = 0
+ session.save()
+
+ reopened = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ assert reopened.cuts == cuts_before
+ fresh = reopened.source(cam.id)
+ assert fresh.synced and fresh.offset == offset_before
+ assert fresh.file_size != 0 and fresh.file_mtime_ns != 0
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_refresh_stale_sources_copies_every_probe_field_and_resets_its_tiles(sandbox):
+ """Re-probing a replaced file used to only copy duration/size/fps.
+
+ A call recording that gained a second audio stream, or lost the one a
+ tile's audio_stream_index pointed at, kept the stale stream count and an
+ out-of-range index; a pane cropped out of it kept showing as synced
+ because nothing told the virtual camera built from it to refresh too.
+ """
+ path = sandbox / "call.mp4"
+ _ffmpeg("-f", "lavfi", "-i", "color=s=160x90:d=2", "-f", "lavfi", "-i", "anullsrc=r=48000:cl=mono",
+ "-map", "0:v", "-map", "1:a", "-shortest", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
+ str(path))
+ src = mc.probe_source(str(path))
+ assert src.audio_stream_count == 1
+ src.role, src.person, src.offset, src.audio_stream_index = "camera", "host", 0.0, 0
+ pane = _source(str(path), role="camera", person="pane1", parent=src.id, crop=[0.0, 0.0, 0.5, 1.0],
+ offset=5.0, duration=2.0, sync={"status": "ok"})
+ session = mc.MulticamSession(session_id="abc123abc906", name="ep",
+ people=[mc.Person("host", "Host")], sources=[src, pane])
+
+ # Re-export with a second audio stream: same path, new bytes.
+ _ffmpeg("-f", "lavfi", "-i", "color=s=160x90:d=2", "-f", "lavfi", "-i", "anullsrc=r=48000:cl=mono",
+ "-f", "lavfi", "-i", "sine=f=440:r=48000:d=2", "-map", "0:v", "-map", "1:a", "-map", "2:a",
+ "-shortest", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-y", str(path))
+
+ assert mc.refresh_stale_sources(session)
+ assert src.audio_stream_count == 2
+ assert src.audio_stream_channels == [1, 1]
+ assert src.audio_stream_index == 0 # still in range, so left alone
+ assert src.offset is None and src.sync["status"] == "failed"
+ # The pane cropped from it is no longer synced either, not left showing stale.
+ assert pane.offset is None and pane.sync["status"] == "failed"
+ assert pane.sync == src.sync
+
+
+def test_reopening_a_session_detects_a_camera_replaced_on_disk(episode):
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ session = mc.sync_session(session)
+ cam_path = episode / "cam_one.mp4"
+ cam = next(s for s in session.sources if os.path.basename(s.path) == "cam_one.mp4")
+ assert cam.synced
+ old_identity = (cam.file_size, cam.file_mtime_ns)
+
+ # Re-export the same file path with different content: same name, same id
+ # (basename:size can coincide), but the bytes underneath changed.
+ _ffmpeg("-f", "lavfi", "-i", "color=c=yellow:s=160x90:r=30:d=40", "-i", str(episode.parent / "one.wav"),
+ "-shortest", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-y", str(cam_path))
+
+ reopened = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ assert reopened.session_id == session.session_id
+ fresh = reopened.source(cam.id)
+ assert (fresh.file_size, fresh.file_mtime_ns) != old_identity
+ assert not fresh.synced
+ assert fresh.sync["status"] == "failed"
+ assert reopened.cuts == []
+
+
+def test_loading_by_session_id_also_catches_a_camera_replaced_on_disk(episode, monkeypatch):
+ """The MCP tool and the CLI jump straight to a known session id instead of
+
+ reopening by folder, so they used to skip the stale-file check entirely:
+ a camera swapped out underneath an open session kept its old sync and cut.
+ """
+ import main as backend
+
+ session = mc.new_session(folder=str(episode), people=["Nika", "Ana"])
+ session = mc.sync_session(session)
+ session = mc.plan_session(session)
+ cam_path = episode / "cam_one.mp4"
+ cam = next(s for s in session.sources if os.path.basename(s.path) == "cam_one.mp4")
+ assert cam.synced and session.cuts
+
+ _ffmpeg("-f", "lavfi", "-i", "color=c=yellow:s=160x90:r=30:d=40", "-i", str(episode.parent / "one.wav"),
+ "-shortest", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-y", str(cam_path))
+
+ sent = []
+ monkeypatch.setattr(backend, "emit_result", lambda task, status, data=None, error=None: sent.append((status, data, error)))
+ backend.handle_manage_multicam("t", {"action": "show", "session_id": session.session_id})
+ assert sent[0][0] == "success", sent
+ fresh = next(s for s in sent[0][1]["sources"] if s["id"] == cam.id)
+ assert fresh["sync"]["status"] == "failed"
+ assert sent[0][1]["cuts"] == []
+
+
def test_last_person_is_the_guest_and_roles_survive_renames(episode):
session = mc.new_session(folder=str(episode), people=["Nihal", "Cameron", "Ana"])
assert [(p.name, p.role) for p in session.people] == [("Nihal", "host"), ("Cameron", "host"), ("Ana", "guest")]
@@ -605,6 +880,21 @@ def test_turn_edges_follow_sentences_not_loose_timestamps():
mc._settle_turn_edges(w, [0.0, 15.0, 15.0])
assert [x["person"] for x in w] == ["a", "a", "b"]
+ # Georgian has no letter case, so a sentence opener there can't pass an
+ # isupper() check. It must still move to the next speaker.
+ w = words([("დამიდა.", "g"), ("კარგი,", "g"),
+ ("ამიტაო", "h")])
+ mc._settle_turn_edges(w, [0.0] * len(w))
+ assert [x["person"] for x in w] == ["g", "h", "h"]
+
+
+def test_sentence_opener_detection_handles_caseless_scripts():
+ assert mc._looks_like_sentence_opener("კარგი") # Georgian, no case
+ assert mc._looks_like_sentence_opener("Right,")
+ assert not mc._looks_like_sentence_opener("right,")
+ assert not mc._looks_like_sentence_opener("123")
+ assert not mc._looks_like_sentence_opener("")
+
# --- Remote recordings ------------------------------------------------------------
@@ -952,3 +1242,669 @@ def test_a_removal_where_cameras_stop_and_start_is_bridged_by_another_camera(san
assert session.cuts[1]["end"] == pytest.approx(13.0, abs=0.04)
assert session.removals[0]["start"] == pytest.approx(10.0, abs=0.04)
assert session.removals[0]["end"] == pytest.approx(16.0, abs=0.04)
+
+
+ "fps,tc,expected",
+ [
+ # 25 fps: integer rate, plain frames/fps.
+ (25.0, "01:00:00:10", 3600 + 10 / 25),
+ # 29.97 non-drop frame: separator is ':', frame count is not adjusted,
+ # but each frame is 1001/30000 s rather than 1/30 s.
+ (29.97, "00:01:00:00", (30 * 60) * (1001 / 30000)),
+ # 29.97 drop frame: separator is ';' before the frame field. At 1 minute
+ # in, drop-frame counting has skipped 2 frame numbers versus wall time,
+ # so :00;00 lands earlier than non-drop ':00:00:00' would.
+ (29.97, "00:01:00;00", (30 * 60 - 2) * (1001 / 30000)),
+ # 29.97 drop frame at the tenth minute: drop frame skips 2 counts at
+ # the start of every minute except every tenth, so by minute 10 the
+ # cumulative skip is 2 * (10 - 1) = 18 frames, not 0.
+ (29.97, "00:10:00;00", (30 * 600 - 18) * (1001 / 30000)),
+ # 59.94 drop frame: 4 frames skipped per non-tenth minute.
+ (59.94, "00:01:00;00", (60 * 60 - 4) * (1001 / 60000)),
+ # 23.976 non-drop: nominal 24 fps grid, real frame duration 1001/24000 s.
+ (23.976, "00:00:10:00", (24 * 10) * (1001 / 24000)),
+ ],
+)
+def test_timecode_seconds_handles_ntsc_pulldown_and_drop_frame(fps, tc, expected):
+ info = {"format": {"tags": {"timecode": tc}}}
+ assert mc._timecode_seconds(info, fps, 48000) == pytest.approx(expected, abs=1e-6)
+
+
+def test_timecode_seconds_accepts_a_dot_separator_as_non_drop():
+ # Some cameras and field recorders write the non-drop separator as '.'
+ # instead of ':'. It must parse the same as the all-':' form, not read 0.0.
+ info_dot = {"format": {"tags": {"timecode": "01.00.00.10"}}}
+ info_colon = {"format": {"tags": {"timecode": "01:00:00:10"}}}
+ assert mc._timecode_seconds(info_dot, 25.0, 48000) == pytest.approx(3600 + 10 / 25, abs=1e-6)
+ assert mc._timecode_seconds(info_dot, 25.0, 48000) == mc._timecode_seconds(info_colon, 25.0, 48000)
+
+
+def test_timecode_seconds_falls_back_to_time_reference_without_embedded_timecode():
+ info = {"format": {"tags": {"time_reference": "48000"}}}
+ assert mc._timecode_seconds(info, 29.97, 48000) == pytest.approx(1.0, abs=1e-6)
+
+
+def _fake_probe(tmp_path, monkeypatch, video_stream):
+ path = tmp_path / "cam.mov"
+ path.write_bytes(b"0")
+ info = {"format": {"duration": "10.0", "tags": {}}, "streams": [{"codec_type": "video", **video_stream}]}
+ monkeypatch.setattr(mc, "get_video_info", lambda p: info)
+ return mc.probe_source(str(path))
+
+
+def test_timecode_pulldown_is_decided_from_r_frame_rate_not_the_noisy_average(tmp_path, monkeypatch):
+ """A steady 25 fps camera's avg_frame_rate can measure 24.98 by noise alone.
+
+ Deciding pulldown from that average instead of the stream's exact
+ r_frame_rate would misread a plain 25 fps file as NTSC and shift its
+ embedded start timecode by seconds.
+ """
+ src = _fake_probe(tmp_path, monkeypatch, {
+ "avg_frame_rate": "2498/100", "r_frame_rate": "25/1",
+ "tags": {"timecode": "01:00:00:10"},
+ })
+ assert src.timecode == pytest.approx(3600 + 10 / 25, abs=1e-6)
+
+
+def test_probe_source_counts_multiple_audio_streams(tmp_path, monkeypatch):
+ path = tmp_path / "cam.mxf"
+ path.write_bytes(b"0")
+ info = {"format": {"duration": "10.0", "tags": {}}, "streams": [
+ {"codec_type": "video", "avg_frame_rate": "30/1", "r_frame_rate": "30/1"},
+ {"codec_type": "audio", "channels": 1},
+ {"codec_type": "audio", "channels": 1},
+ ]}
+ monkeypatch.setattr(mc, "get_video_info", lambda p: info)
+ src = mc.probe_source(str(path))
+ assert src.audio_stream_count == 2
+ assert src.audio_stream_channels == [1, 1]
+ assert src.audio_stream_index == 0
+
+
+def test_update_mapping_validates_audio_stream_index(sandbox):
+ session = mc.MulticamSession(
+ session_id="abc123abc999", name="ep",
+ people=[mc.Person("nika", "Nika")],
+ sources=[mc.Source(id="a", path="/x/cam_a.mov", kind="video", duration=9, has_audio=True,
+ audio_channels=1, audio_stream_count=2, audio_stream_channels=[1, 2])],
+ )
+ with pytest.raises(ValueError, match="audio_stream_index"):
+ mc.update_mapping(session, {"sources": [{"id": "a", "audio_stream_index": 5}]})
+ session = mc.update_mapping(session, {"sources": [{"id": "a", "audio_stream_index": 1}]})
+ src = session.source("a")
+ assert src.audio_stream_index == 1
+ assert src.audio_channels == 2 # recomputed from the selected stream, not the first one
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_extract_reads_the_selected_audio_stream_not_always_the_first(sandbox):
+ path = sandbox / "two_streams.mp4"
+ _ffmpeg("-f", "lavfi", "-i", "color=s=160x90:d=2", "-f", "lavfi", "-i", "anullsrc=r=48000:cl=mono",
+ "-f", "lavfi", "-i", "sine=f=440:r=48000:d=2",
+ "-map", "0:v", "-map", "1:a", "-map", "2:a", "-shortest",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(path))
+ src = mc.probe_source(str(path))
+ assert src.audio_stream_count == 2
+
+ silent = mc._read_wav(mc._extract(src, sandbox / "stream0.wav"))
+ assert np.abs(silent).mean() < 50 # stream 0 is anullsrc: near silence
+
+ src.audio_stream_index = 1
+ tone = mc._read_wav(mc._extract(src, sandbox / "stream1.wav"))
+ assert np.abs(tone).mean() > 1000 # stream 1 is a 440 Hz tone
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_mix_and_stems_honor_audio_stream_index_not_always_the_first_stream(sandbox):
+ path = sandbox / "two_streams.mp4"
+ _ffmpeg("-f", "lavfi", "-i", "color=s=160x90:d=2", "-f", "lavfi", "-i", "anullsrc=r=48000:cl=mono",
+ "-f", "lavfi", "-i", "sine=f=440:r=48000:d=2",
+ "-map", "0:v", "-map", "1:a", "-map", "2:a", "-shortest",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(path))
+ mic = _source(str(path), kind="video", role="mic", person="host", offset=0.0, duration=2.0,
+ audio_stream_count=2, audio_stream_channels=[1, 1], audio_stream_index=1)
+
+ mix = sandbox / "mix.wav"
+ mc._write_mix(mc.MulticamSession(session_id="abc123abc903", name="ep", sources=[mic]),
+ mix, 0.0, 2.0, encode=("-ac", "1", "-c:a", "pcm_s16le"), feeds=[(mic, -1)])
+ assert np.abs(mc._read_wav(mix)).mean() > 1000 # the tone on stream 1, not the silence on stream 0
+
+ session = mc.MulticamSession(session_id="abc123abc904", name="ep", people=[mc.Person("host", "Host")],
+ sources=[mic])
+ stems = mc._render_stems(session, sandbox, 0.0, 2.0)
+ assert len(stems) == 1
+ assert np.abs(_read_f32(stems[0])).mean() > 0.05
+
+
+def test_probe_source_warns_on_variable_frame_rate(tmp_path, monkeypatch):
+ # avg_frame_rate (what actually played) is far below r_frame_rate (the
+ # stream's time base): frames held variable lengths.
+ src = _fake_probe(tmp_path, monkeypatch, {"avg_frame_rate": "24/1", "r_frame_rate": "60/1"})
+ assert src.fps == pytest.approx(24.0)
+ assert "variable frame rate" in src.fps_warning.lower()
+
+
+def test_probe_source_does_not_warn_on_a_steady_frame_rate(tmp_path, monkeypatch):
+ src = _fake_probe(tmp_path, monkeypatch, {"avg_frame_rate": "30000/1001", "r_frame_rate": "30000/1001"})
+ assert src.fps_warning == ""
+
+
+def test_probe_source_warns_when_falling_back_to_the_default_frame_rate(tmp_path, monkeypatch):
+ src = _fake_probe(tmp_path, monkeypatch, {"avg_frame_rate": "0/0", "r_frame_rate": "0/0"})
+ assert src.fps == 30.0
+ assert "assuming 30 fps" in src.fps_warning.lower()
+
+
+# --- drift inside a shot ----------------------------------------------------------
+
+def _drifting_session(sandbox, speed=1.0001):
+ cam = _source(str(sandbox / "cam.mp4"), role="camera", person="host", offset=0.0, speed=speed, duration=700.0)
+ mic = _source(str(sandbox / "mic.wav"), kind="audio", role="mic", person="host", offset=0.0, speed=speed,
+ duration=700.0)
+ session = mc.MulticamSession(session_id="abc123abc888", name="ep", people=[mc.Person("host", "Host")],
+ sources=[cam, mic], reference_id="mic",
+ cuts=[{"start": 10.0, "end": 610.0, "source_id": "cam"}])
+ session.save()
+ return session
+
+
+def test_a_long_shot_on_a_drifting_camera_stays_within_half_a_frame(sandbox):
+ session = _drifting_session(sandbox)
+ cam = session.source("cam")
+ fps = 30.0
+ plan = mc.render_plan(session, fps)
+ assert len(plan) > 1
+ assert session.cuts == [{"start": 10.0, "end": 610.0, "source_id": "cam"}]
+ visible = mc._merged([{"start": p.tl_start, "end": p.tl_start + p.frames / fps, "source_id": p.source_id}
+ for p in plan])
+ assert [(round(c["start"], 3), round(c["end"], 3), c["source_id"]) for c in visible] == [(10.0, 610.0, "cam")]
+ assert sum(p.frames for p in plan) == 600 * 30
+
+ worst = 0.0
+ for p in plan:
+ start = cam.source_in(p.tl_start, p.frames / fps)
+ # Played at real speed, the slip is linear across a piece, so its ends are the worst frames.
+ for k in (0, p.frames):
+ worst = max(worst, abs(start + k / fps - cam.source_time(p.tl_start + k / fps)))
+ assert worst < 0.5 / fps
+ # Mapping the whole shot at its start would have slipped 60 ms by the end.
+ assert abs(cam.source_time(10.0) + 600.0 - cam.source_time(610.0)) > 0.05
+
+
+def test_drift_split_reaches_the_render_seek_and_both_editor_timelines(sandbox):
+ session = _drifting_session(sandbox)
+ cam = session.source("cam")
+ fps = 30.0
+ plan = mc.render_plan(session, fps)
+
+ piece = plan[1]
+ command = mc._shot_command(session, cam, piece.tl_start, piece.frames, 320, 180, fps, "none")
+ seek = float(command[command.index("-ss") + 1])
+ assert seek == pytest.approx(cam.source_in(piece.tl_start, piece.frames / fps), abs=1e-4)
+
+ premiere = ET.parse(mc.export_xml(session, "premiere")).getroot()
+ assert len(premiere.findall("./sequence/media/video/track/clipitem")) == len(plan)
+ assert len(premiere.findall("./sequence/media/audio/track/clipitem")) >= len(plan)
+
+ fcp = ET.parse(mc.export_xml(session, "fcpxml")).getroot()
+ mics = [c for c in fcp.iter("asset-clip") if c.get("lane") == "-1"]
+ assert len(mics) >= len(plan)
+ for clip in mics:
+ offset, start, duration = (float(Fraction(clip.get(k).rstrip("s"))) for k in ("offset", "start", "duration"))
+ middle = 10.0 + offset + duration / 2
+ # Each mic piece is pinned at its middle; a sample of rounding is all that's left there.
+ assert abs(start + duration / 2 - session.source("mic").source_time(middle)) < 1e-3
+
+
+# --- render validation and the shot cache ------------------------------------------
+
+def _planned(folder):
+ session = mc.new_session(folder=str(folder), people=["Nika", "Ana"])
+ mc.update_mapping(session, {"sources": [
+ {"id": s.id, "role": "camera", "person": "nika" if "one" in os.path.basename(s.path) else "ana"}
+ for s in session.sources if s.kind == "video"
+ ]})
+ return mc.plan_session(mc.sync_session(session))
+
+
+def _counting_encodes(monkeypatch):
+ real = mc.proc_run
+ encodes = []
+
+ def run(cmd, **kw):
+ if "-frames:v" in cmd and "-filter_complex" in cmd or "lavfi" in cmd and "-frames:v" in cmd:
+ encodes.append(cmd)
+ return real(cmd, **kw)
+
+ monkeypatch.setattr(mc, "proc_run", run)
+ return encodes
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_render_reports_validation_and_reuses_cached_shots(episode, monkeypatch):
+ session = _planned(episode)
+ encodes = _counting_encodes(monkeypatch)
+ outputs = mc.render_session(session)
+ shots = len(mc.render_plan(session, 30.0))
+ assert len(encodes) == shots
+ v = outputs["validation"]
+ assert v["frames_expected"] == v["frames_actual"] == sum(p.frames for p in mc.render_plan(session, 30.0))
+ assert v["duration"] == pytest.approx(v["frames_expected"] / 30.0, abs=0.05)
+ assert v["lufs"] is not None and v["true_peak"] is not None
+ assert isinstance(v["warnings"], list)
+ assert mc.MulticamSession.load(session.session_id).outputs["validation"] == v
+
+ # Dropping the stems changes the render but not one shot's pixels: nothing re-encodes.
+ encodes.clear()
+ mc.render_session(session, stems=False)
+ assert encodes == []
+ # A removal re-encodes only the pieces it splits.
+ mc.set_removals(session, [{"start": 5.0, "end": 6.0}])
+ mc.render_session(session)
+ assert 0 < len(encodes) < len(mc.render_plan(session, 30.0))
+ encodes.clear()
+
+ # A cached shot that lost frames is caught by its count and encoded again.
+ cached = sorted((mc._work_dir(session.session_id) / "shots").glob("*/shot.mp4"))
+ cached[0].write_bytes(cached[0].read_bytes()[: cached[0].stat().st_size // 3])
+ session.outputs.pop("render_key")
+ again = mc.render_session(session)["validation"]
+ assert again["frames_actual"] == again["frames_expected"] == v["frames_expected"] - 30
+ assert len(encodes) == 1
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_render_warns_when_a_camera_runs_out_or_the_mix_is_short(sandbox, monkeypatch):
+ folder = sandbox / "short"
+ folder.mkdir()
+ voice = _speech(40, 9)
+ _write_wav(folder / "rec_Tr1.wav", voice)
+ _write_wav(sandbox / "cam.wav", voice * 0.6)
+ # The picture stops 2 s before the camera's own sound does.
+ _ffmpeg("-f", "lavfi", "-i", "color=c=red:s=160x90:r=30:d=38", "-i", str(sandbox / "cam.wav"),
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(folder / "cam_one.mp4"))
+ session = mc.sync_session(mc.new_session(folder=str(folder), people=["Nika"]))
+ cam = next(s for s in session.sources if s.kind == "video")
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "role": "camera", "person": "nika"}]})
+ mc.set_cuts(session, [{"start": 1.0, "end": cam.timeline_end() - 0.1, "source_id": cam.id}])
+
+ real_audio = mc._render_audio
+
+ def short_audio(session, out, start, duration, splice=None):
+ return real_audio(session, out, start, duration - 1.0, splice)
+
+ monkeypatch.setattr(mc, "_render_audio", short_audio)
+ outputs = mc.render_session(session, stems=False)
+ v = outputs["validation"]
+ assert any("cam_one.mp4 ran out" in w for w in v["warnings"])
+ assert any("shorter than the picture" in w for w in v["warnings"])
+ # The short mix no longer trims the picture to fit.
+ assert v["frames_actual"] == v["frames_expected"]
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_render_fails_on_decode_errors_or_a_wrong_frame_count(episode, monkeypatch):
+ session = _planned(episode)
+ out_dir = mc._output_dir(session)
+ monkeypatch.setattr(mc, "_decode_errors", lambda path, **k: "Invalid NAL unit size")
+ with pytest.raises(RuntimeError, match="decode"):
+ mc.render_session(session)
+ assert not (out_dir / "episode.mp4").exists()
+ monkeypatch.undo()
+
+ real = mc._frame_count
+ monkeypatch.setattr(mc, "_frame_count", lambda p: real(p) - (2 if p.name == "episode.mp4" else 0))
+ with pytest.raises(RuntimeError, match="frames"):
+ mc.render_session(session)
+ assert not (out_dir / "episode.mp4").exists()
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_decode_check_samples_a_few_short_spans_by_default_not_the_whole_file(episode, monkeypatch):
+ """A full decode of a finished episode costs minutes per hour of 1080p.
+
+ The default check must still pass a clean file, but it must do it by
+ decoding a handful of short, bounded spans, not the whole thing; asking
+ for validate="full" is the only way to get the exhaustive decode.
+ """
+ video = mc.render_session(_planned(episode), stems=False)["video"]
+ calls = []
+ real_run = mc.proc_run
+
+ def recording_run(cmd, **k):
+ calls.append(cmd)
+ return real_run(cmd, **k)
+
+ monkeypatch.setattr(mc, "proc_run", recording_run)
+ assert mc._decode_errors(video) == ""
+ decodes = [c for c in calls if c[0] == "ffmpeg"]
+ assert decodes and all("-t" in c for c in decodes), "every sampled decode must be bounded to a short span"
+ assert len(decodes) <= 5
+
+ calls.clear()
+ assert mc._decode_errors(video, full=True) == ""
+ decodes = [c for c in calls if c[0] == "ffmpeg"]
+ assert len(decodes) == 1 and "-t" not in decodes[0], "a full decode has no time bound"
+
+
+# --- audio chain --------------------------------------------------------------------
+
+def _read_f32(path, channels=1):
+ raw = subprocess.run(["ffmpeg", "-v", "error", "-i", str(path), "-f", "f32le", "-ac", str(channels), "-"],
+ check=True, capture_output=True).stdout
+ return np.frombuffer(raw, dtype=np.float32)
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_drift_correction_resamples_to_the_exact_length_without_audible_pitch_shift(sandbox):
+ speed = 1.0 + 100e-6
+ t = np.arange(60 * RATE) / RATE
+ _write_wav(sandbox / "tone.wav", np.sin(2 * np.pi * 1000 * t) * 0.6)
+ tone = _source(str(sandbox / "tone.wav"), kind="audio", role="mic", offset=0.5, speed=speed)
+ session = mc.MulticamSession(session_id="abc123abc901", name="ep", sources=[tone])
+ out = sandbox / "stretched.wav"
+ mc._write_mix(session, out, 0.0, 62.0, encode=("-c:a", "pcm_f32le"), feeds=[(tone, -1)])
+ y = _read_f32(out)
+ sounding = np.flatnonzero(np.abs(y) > 1e-3)
+ first, last = sounding[0] / RATE, (sounding[-1] + 1) / RATE
+ assert first == pytest.approx(0.5, abs=1e-3)
+ # 60 s of source lasts 60 * speed on the timeline: 6 ms longer, matched to under a millisecond.
+ assert last - first == pytest.approx(60 * speed, abs=1e-3)
+ seg = y[int(10 * RATE):int(50 * RATE)]
+ spectrum = np.abs(np.fft.rfft(seg * np.hanning(len(seg)), 1 << 23))
+ freq = np.argmax(spectrum) * RATE / (1 << 23)
+ cents = 1200 * np.log2(freq / 1000)
+ assert abs(cents) < 0.5
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_the_mix_high_passes_each_mic_before_leveling(sandbox):
+ t = np.arange(20 * RATE) / RATE
+ voice = _speech(20, 11)
+ _write_wav(sandbox / "rumble.wav", 0.5 * voice + 0.4 * np.sin(2 * np.pi * 30 * t))
+ mic = _source(str(sandbox / "rumble.wav"), kind="audio", role="mic", offset=0.0, duration=20.0)
+ session = mc.MulticamSession(session_id="abc123abc902", name="ep", sources=[mic])
+ assert mc.VOICE_CHAIN[0].startswith("highpass=f=70") and mc.VOICE_CHAIN[1].startswith("dynaudnorm")
+
+ def hum_share(x):
+ spectrum = np.abs(np.fft.rfft(x)) ** 2
+ f = np.fft.rfftfreq(len(x), 1 / RATE)
+ return spectrum[(f > 25) & (f < 35)].sum() / spectrum[(f > 500) & (f < 4000)].sum()
+
+ out = sandbox / "mixed.wav"
+ mc._write_mix(session, out, 0.0, 20.0, per_feed=mc.VOICE_CHAIN, encode=("-c:a", "pcm_f32le"), feeds=[(mic, -1)])
+ before = hum_share(_read_f32(sandbox / "rumble.wav"))
+ after = hum_share(_read_f32(out))
+ assert 10 * np.log10(before / after) > 10
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_two_pass_mix_hits_the_target_and_the_stems_sum_to_it(episode, monkeypatch):
+ session = _planned(episode)
+ work = mc._work_dir(session.session_id)
+ start, end = session.cuts[0]["start"], session.cuts[-1]["end"]
+ gain = mc._render_audio(session, work / "mix.wav", start, end - start)
+ stems = mc._render_stems(session, work, start, end - start, None, gain)
+ assert len(stems) == 2
+ mix = _read_f32(work / "mix.wav")
+ total = sum(_read_f32(p) for p in stems)
+ assert len(total) == len(mix)
+ # The limiter only touches peaks the gain pushed past the ceiling; everywhere else the stems add up exactly.
+ calm = np.abs(total) < 10 ** ((mc.TRUE_PEAK_CEILING - 1.0) / 20)
+ assert calm.mean() > 0.9
+ assert np.abs(total[calm] - mix[calm]).max() < 2e-3
+
+ lufs, _ = mc._loudness(work / "mix.wav")
+ assert lufs == pytest.approx(mc.LOUDNESS_TARGET, abs=0.5)
+ v = mc.render_session(session, stems=False)["validation"]
+ assert v["lufs"] == pytest.approx(mc.LOUDNESS_TARGET, abs=1.0)
+ assert v["true_peak"] <= mc.TRUE_PEAK_CEILING + 0.5
+ assert v["warnings"] == []
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_loudness_gain_is_measured_on_the_kept_audio_not_the_removed_stretch(sandbox):
+ """A removed stretch (a retake, a long silence) shouldn't pull the
+
+ target gain around: it never airs. Measuring it on the whole premix
+ instead of what the splice keeps asks for the wrong gain for the
+ episode anyone actually hears.
+ """
+ kept = _speech(10, 21)
+ # A removed stretch, loud enough to drag a whole-file measurement's
+ # target gain down well below what the kept 10 s alone would ask for.
+ loud_removed = 8.0 * _speech(5, 22)
+ _write_wav(sandbox / "mic.wav", np.concatenate([kept, loud_removed]))
+ mic = _source(str(sandbox / "mic.wav"), kind="audio", role="mic", offset=0.0, duration=15.0)
+ session = mc.MulticamSession(session_id="abc123abc907", name="ep", sources=[mic])
+
+ gain_whole = mc._render_audio(session, sandbox / "whole.wav", 0.0, 15.0)
+ gain_kept = mc._render_audio(session, sandbox / "kept.wav", 0.0, 15.0, splice=[(0.0, 10.0)])
+ assert gain_kept != pytest.approx(gain_whole, abs=0.5)
+
+ lufs_kept, _ = mc._loudness(sandbox / "kept.wav")
+ assert lufs_kept == pytest.approx(mc.LOUDNESS_TARGET, abs=0.5)
+ assert mc._media_duration(sandbox / "kept.wav") == pytest.approx(10.0, abs=0.05)
+
+
+# --- manual drift anchors -------------------------------------------------------------
+
+def test_anchors_fit_offset_and_drift_by_least_squares(sandbox):
+ session = _bare(sandbox)
+ cam = session.sources[0]
+ speed = 1.0 + 80e-6
+ pairs = [{"timeline": 12.0 + speed * src + jitter, "source": src}
+ for src, jitter in ((5.0, 0.0), (20.0, 0.002), (40.0, -0.002), (55.0, 0.0))]
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "anchors": pairs}]})
+ assert cam.offset == pytest.approx(12.0, abs=0.003)
+ assert cam.speed == pytest.approx(speed, abs=1e-4)
+ assert cam.sync["status"] == "manual" and cam.sync["method"] == "manual"
+ assert 0 < cam.sync["residual_ms"] < 3
+ assert [p["source"] for p in cam.sync["anchors"]] == [5.0, 20.0, 40.0, 55.0]
+
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "anchors": [{"timeline": 30.0, "source": 10.0}]}]})
+ assert (cam.offset, cam.speed) == (20.0, 1.0)
+ assert cam.sync["residual_ms"] == 0
+
+
+def test_anchors_are_refused_when_they_run_backwards_or_imply_impossible_drift(sandbox):
+ session = _bare(sandbox)
+ cam = session.sources[0]
+ for bad in (
+ [],
+ [{"timeline": 10.0}],
+ [{"timeline": 10.0, "source": 5.0}, {"timeline": 9.0, "source": 15.0}],
+ [{"timeline": 10.0, "source": 5.0}, {"timeline": 11.0, "source": 5.0}],
+ [{"timeline": 10.0, "source": 5.0}, {"timeline": 30.0, "source": 15.0}],
+ [{"timeline": 10.0, "source": 500.0}],
+ ):
+ with pytest.raises(ValueError):
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "anchors": bad}]})
+ assert cam.sync.get("method") != "manual"
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_anchors_sync_a_camera_with_no_audio_and_survive_a_resync(sandbox):
+ folder = sandbox / "mute"
+ folder.mkdir()
+ _write_wav(folder / "rec_Tr1.wav", _speech(40, 12))
+ _ffmpeg("-f", "lavfi", "-i", "color=c=red:s=160x90:r=30:d=30", "-c:v", "libx264", "-pix_fmt", "yuv420p",
+ str(folder / "cam_one.mp4"))
+ session = mc.sync_session(mc.new_session(folder=str(folder), people=["Nika"]))
+ cam = next(s for s in session.sources if s.kind == "video")
+ assert not cam.has_audio and not cam.synced
+
+ speed = 1.0 + 50e-6
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "role": "camera", "person": "nika", "anchors": [
+ {"timeline": 4.0 + speed * 2.0, "source": 2.0}, {"timeline": 4.0 + speed * 28.0, "source": 28.0}]}]})
+ assert cam.offset == pytest.approx(4.0, abs=1e-6) and cam.speed == pytest.approx(speed, abs=1e-9)
+
+ session = mc.sync_session(session)
+ cam = session.source(cam.id)
+ assert cam.sync["method"] == "manual" and cam.speed == pytest.approx(speed, abs=1e-9)
+ assert cam.offset == pytest.approx(4.0, abs=1e-6)
+ session = mc.plan_session(session)
+ assert {c["source_id"] for c in session.cuts} == {cam.id}
+ assert mc.render_plan(session, 30.0)
+
+
+# --- per-camera input LUTs ------------------------------------------------------------
+
+def _cube(path, size=2, entries=None, header=None):
+ lines = header if header is not None else ['TITLE "invert"', f"LUT_3D_SIZE {size}"]
+ grid = [f"{1 - r / (size - 1):.4f} {1 - g / (size - 1):.4f} {1 - b / (size - 1):.4f}"
+ for b in range(size) for g in range(size) for r in range(size)]
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text("\n".join(lines + grid[:entries]) + "\n", encoding="utf-8")
+ return path
+
+
+def test_input_lut_must_be_a_complete_3d_cube(sandbox):
+ session = _bare(sandbox)
+ cam = session.sources[0]
+ good = _cube(sandbox / "luts" / "invert.cube")
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "input_lut": str(good)}]})
+ assert cam.input_lut == str(good)
+ for bad, match in (
+ (str(_cube(sandbox / "luts" / "short.cube", entries=7)), "holds 7"),
+ (str(_cube(sandbox / "luts" / "flat.cube", header=["LUT_1D_SIZE 2"])), "1D"),
+ (str(_cube(sandbox / "luts" / "nosize.cube", header=[])), "LUT_3D_SIZE"),
+ ("luts/invert.cube", "absolute"),
+ (str(sandbox / "luts" / "missing.cube"), "isn't a .cube"),
+ ):
+ with pytest.raises(ValueError, match=match):
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "input_lut": bad}]})
+ mc.update_mapping(session, {"sources": [{"id": cam.id, "input_lut": ""}]})
+ assert cam.input_lut == ""
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_input_lut_colors_renders_stills_and_looks_and_exports_hand_it_off(episode):
+ session = _planned(episode)
+ red = next(s for s in session.sources if s.path.endswith("cam_one.mp4"))
+ # Spaces, a colon and brackets in the path exercise the filter graph escaping.
+ # Windows forbids a colon in a folder name, but its drive letter puts one
+ # in every path there anyway.
+ folder = "grade [v1], final" if os.name == "nt" else "grade: [v1], final"
+ lut = _cube(episode.parent / folder / "invert.cube")
+ before = mc.previews(session)["cameras"][red.id]
+ mc.update_mapping(session, {"sources": [{"id": red.id, "input_lut": str(lut)}]})
+ assert session.cuts, "a LUT is color, so the cut stays"
+
+ stills = mc.previews(session, looks=True)
+ assert stills["cameras"][red.id] != before
+ inverted = _mean_rgb(stills["cameras"][red.id], 0)
+ assert inverted[0] < 80 and inverted[1] > 150 and inverted[2] > 150
+ assert set(stills["looks"]) == {c.id for c in session.cameras()}
+ assert all(set(per) == set(mc.LOOKS) for per in stills["looks"].values())
+
+ video = mc.render_session(session, stems=False)["video"]
+ on_red = next(c for c in session.cuts if c["source_id"] == red.id)
+ pixel = _mean_rgb(video, (on_red["start"] + on_red["end"]) / 2 - session.cuts[0]["start"])
+ assert pixel[0] < 80 and pixel[2] > 150
+
+ mc.export_xml(session, "fcpxml")
+ handoff = json.loads(open(session.outputs["color_handoff"], encoding="utf-8").read())
+ assert os.path.dirname(session.outputs["color_handoff"]) == os.path.dirname(session.outputs["fcpxml"])
+ luts = {c["source_id"]: c["input_lut"] for c in handoff["cameras"]}
+ assert luts[red.id] == str(lut)
+ assert all(v is None for k, v in luts.items() if k != red.id)
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_previews_only_renders_looks_for_cameras_actually_used_in_the_cut(episode):
+ """Four look stills per camera is wasted work for a camera that's mapped
+
+ but never actually appears in the cut (an alternate angle left in,
+ say). Once a cut exists, only cameras it actually uses should get them.
+ """
+ session = _planned(episode)
+ cam_one = next(s for s in session.sources if s.path.endswith("cam_one.mp4"))
+ cam_two = next(s for s in session.sources if s.path.endswith("cam_two.mp4"))
+ mc.set_cuts(session, [{"start": session.cuts[0]["start"], "end": session.cuts[-1]["end"],
+ "source_id": cam_one.id}])
+
+ stills = mc.previews(session, looks=True)
+ assert cam_one.id in stills["looks"]
+ assert cam_two.id not in stills["looks"]
+
+
+def test_a_lut_deleted_after_mapping_fails_fast_and_names_the_camera(episode):
+ """A LUT picked at map time can later be moved or deleted on disk.
+
+ Without an upfront check, ffmpeg's own error about the missing file
+ surfaces from deep inside a shot or a still render, with nothing saying
+ which camera's LUT broke.
+ """
+ session = _planned(episode)
+ red = next(s for s in session.sources if s.path.endswith("cam_one.mp4"))
+ lut = _cube(episode.parent / "luts" / "invert.cube")
+ mc.update_mapping(session, {"sources": [{"id": red.id, "input_lut": str(lut)}]})
+
+ os.remove(lut)
+
+ with pytest.raises(ValueError, match="cam_one.mp4") as render_exc:
+ mc.render_session(session, stems=False)
+ assert "missing" in str(render_exc.value).lower()
+
+ with pytest.raises(ValueError, match="cam_one.mp4") as preview_exc:
+ mc.previews(session)
+ assert "missing" in str(preview_exc.value).lower()
+
+
+def test_lut_is_refused_on_a_mic(sandbox):
+ session = _bare(sandbox)
+ session.sources.append(_source("/x/room.wav", kind="audio", role="mic", id="room"))
+ with pytest.raises(ValueError, match="camera"):
+ mc.update_mapping(session, {"sources": [{"id": "room", "input_lut": str(_cube(sandbox / "l" / "a.cube"))}]})
+
+
+# --- fitting cameras of another size into the sequence --------------------------------
+
+def test_exports_fit_cameras_of_another_size_inside_the_sequence(sandbox):
+ wide = _source(str(sandbox / "wide.mp4"), role="camera", person="wide", offset=0.0, id="wide")
+ uhd = _source(str(sandbox / "uhd.mp4"), role="camera", person="host", offset=0.0, id="uhd", width=3840, height=2160)
+ phone = _source(str(sandbox / "phone.mp4"), role="camera", person="guest", offset=0.0, id="phone",
+ width=1080, height=1440)
+ room = _source(str(sandbox / "room.wav"), kind="audio", role="mic", offset=0.0, id="room")
+ session = mc.MulticamSession(
+ session_id="abc123abc903", name="ep", people=[mc.Person("host", "Host"), mc.Person("guest", "Guest")],
+ sources=[wide, uhd, phone, room], reference_id="room",
+ cuts=[{"start": 0.0, "end": 5.0, "source_id": "wide"}, {"start": 5.0, "end": 10.0, "source_id": "uhd"},
+ {"start": 10.0, "end": 15.0, "source_id": "phone"}])
+ session.save()
+
+ premiere = ET.parse(mc.export_xml(session, "premiere", review=True)).getroot()
+ assert premiere.findtext("./sequence/media/video/format/samplecharacteristics/width") == "1920"
+ scales: dict = {}
+ for item in premiere.iter("clipitem"):
+ effect = item.find("filter/effect")
+ if item.find("sourcetrack") is not None:
+ assert effect is None
+ continue
+ name = item.findtext("name")
+ if effect is None:
+ scales.setdefault(name, set()).add(None)
+ continue
+ assert [effect.findtext(k) for k in ("name", "effectid", "effectcategory", "effecttype", "mediatype")] == [
+ "Basic Motion", "basic", "motion", "motion", "video"]
+ param = effect.find("parameter")
+ assert param.get("authoringApp") == "PremierePro"
+ assert [param.findtext(k) for k in ("parameterid", "name", "valuemin", "valuemax")] == ["scale", "Scale", "0", "1000"]
+ scales.setdefault(name, set()).add(float(param.findtext("value")))
+ assert scales == {"wide.mp4": {None}, "uhd.mp4": {50.0}, "phone.mp4": {75.0}}
+
+ fcp = ET.parse(mc.export_xml(session, "fcpxml", review=True)).getroot()
+ for clip in fcp.iter("asset-clip"):
+ fitted = [c.get("type") for c in clip.findall("adjust-conform")]
+ expected = ["fit"] if clip.get("name") in ("uhd.mp4", "phone.mp4") else []
+ assert fitted == expected, clip.get("name")
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_a_fresh_edit_with_no_people_summarizes_before_its_first_sync(episode):
+ session = mc.new_session(folder=str(episode))
+ assert session.reference_id == ""
+ data = mc.payload(session)
+ assert data["preview"] is None
diff --git a/tests/test_multicam_cloud.py b/tests/test_multicam_cloud.py
index aed1511..4300f65 100644
--- a/tests/test_multicam_cloud.py
+++ b/tests/test_multicam_cloud.py
@@ -116,7 +116,7 @@ def test_sends_only_previews_and_renders_the_cloud_cut_here(episode, cloud, monk
cloud["edit"] = {
"state": {**sent["engine"], "people": [{"id": "nika", "name": "Nika"}, {"id": "ana", "name": "Ana Smith"}]},
"cuts": [{"start": start, "end": end, "source_id": one}],
- "removals": [{"start": 10, "end": 12}],
+ "removals": [{"start": 10, "end": 12, "reason": "retake"}],
"look": "warm",
"revision": 4,
}
@@ -126,10 +126,73 @@ def test_sends_only_previews_and_renders_the_cloud_cut_here(episode, cloud, monk
assert "Pulled the cloud edit: 1 shots" in out
session = mc.MulticamSession.load(session.session_id)
assert [p.name for p in session.people] == ["Nika", "Ana Smith"]
- assert session.look == "warm" and session.removals == [{"start": 10.0, "end": 12.0}]
+ assert session.look == "warm"
+ assert session.removals == [{"start": 10.0, "end": 12.0, "reason": "retake"}]
assert session.outputs["duration"] == pytest.approx(end - start - 2, abs=0.05)
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_push_records_the_basis_of_what_was_actually_sent_not_whatever_lands_later(episode, cloud, monkeypatch, capsys):
+ """Edits keep landing on disk while proxies encode and upload.
+
+ push() used to recompute the basis from the session reloaded after the
+ uploads finished, so a nudge that landed mid-push got baked into the
+ recorded basis as if it had been there when the cuts were made. Pull
+ would then see pushed_basis == current_basis and apply a cut that was
+ actually made against the pre-nudge placement.
+ """
+ words = [{"start": 1.0, "end": 1.4, "text": "Hello", "person": "nika"}]
+ monkeypatch.setattr(mc, "transcript", lambda *a, **k: {"words": words})
+
+ real_put = multicam_cloud._put
+ nudged = []
+
+ def nudging_put(url, path):
+ # Simulate another edit landing on disk after push captured its
+ # basis but before the upload loop (which the basis must survive) finishes.
+ if not nudged:
+ nudged.append(True)
+ session = mc.MulticamSession.load(mc.list_sessions()[0]["session_id"])
+ one = next(s for s in session.sources if s.path.endswith("cam_one.mp4"))
+ mc.update_mapping(session, {"sources": [{"id": one.id, "nudge": 1.5}]})
+ return real_put(url, path)
+
+ monkeypatch.setattr(multicam_cloud, "_put", nudging_put)
+ code = run_cli(monkeypatch, str(episode), "--people", "Nika, Ana", "--set", "cam_one=camera:nika",
+ "--set", "cam_two=camera:ana", "--cloud", "-y")
+ assert code == 0, capsys.readouterr().out
+
+ session = mc.MulticamSession.load(mc.list_sessions()[0]["session_id"])
+ pushed_basis = session.cloud["basis"]
+ sent_basis = mc.sync_basis_signature(session) # recomputed from the session as it is now, nudge included
+ assert pushed_basis != sent_basis, "the nudge that landed mid-push must not be folded into the recorded basis"
+
+ cloud["edit"] = {"state": {}, "cuts": [], "removals": [], "revision": 1}
+ with pytest.raises(multicam_cloud.MulticamCloudError, match="moved on the timeline"):
+ multicam_cloud.pull(session)
+
+
def test_pull_needs_an_edit_that_was_sent(sandbox):
with pytest.raises(multicam_cloud.MulticamCloudError, match="No multicam edit on this computer"):
multicam_cloud.resolve("11111111-2222-3333-4444-555555555555")
+
+
[email protected](not shutil.which("ffmpeg"), reason="ffmpeg not installed")
+def test_pull_refuses_when_sources_moved_since_the_edit_was_sent(episode, cloud, monkeypatch, capsys):
+ words = [{"start": 1.0, "end": 1.4, "text": "Hello", "person": "nika"}]
+ monkeypatch.setattr(mc, "transcript", lambda *a, **k: {"words": words})
+ code = run_cli(monkeypatch, str(episode), "--people", "Nika, Ana", "--set", "cam_one=camera:nika",
+ "--set", "cam_two=camera:ana", "--cloud", "-y")
+ assert code == 0, capsys.readouterr().out
+
+ session = mc.MulticamSession.load(mc.list_sessions()[0]["session_id"])
+ one = next(s for s in session.sources if s.path.endswith("cam_one.mp4"))
+ cloud["edit"] = {"state": {}, "cuts": [], "removals": [], "revision": 1}
+
+ # Nudging a source after the push moves it on the timeline, so the cuts
+ # the cloud editor made against the old position would land on the wrong
+ # footage if pulled.
+ mc.update_mapping(session, {"sources": [{"id": one.id, "nudge": 1.5}]})
+ session = mc.MulticamSession.load(session.session_id)
+ with pytest.raises(multicam_cloud.MulticamCloudError, match="moved on the timeline"):
+ multicam_cloud.pull(session)
diff --git a/tests/test_multicam_signal.py b/tests/test_multicam_signal.py
index 300a5bd..d47eee3 100644
--- a/tests/test_multicam_signal.py
+++ b/tests/test_multicam_signal.py
@@ -2,6 +2,7 @@ import os
import sys
import numpy as np
+import pytest
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
BACKEND_ROOT = os.path.join(ROOT, "backend")
@@ -10,6 +11,7 @@ if BACKEND_ROOT not in sys.path:
from services.multicam_signal import ( # noqa: E402
BOTH,
+ LEAD_IN,
SILENT,
Camera,
coarse_lag,
@@ -95,6 +97,27 @@ def test_fit_clock_recovers_drift_and_drops_an_outlier():
def test_fit_clock_rejects_implausible_speed():
fit = fit_clock([(0, 5.0, 1.0), (100, 106.0, 1.0)])
assert fit is not None and fit.speed == 1.0
+ # Falling back to speed 1.0 does not make the fit trustworthy: the 1 s
+ # drift between these two points still shows up as a large residual, and
+ # callers need that signal to flag the sync for review.
+ assert fit.speed_fallback is True
+ assert fit.residual_ms == pytest.approx(500.0, abs=1.0)
+ assert fit.residual_all_ms == pytest.approx(500.0, abs=1.0)
+
+
+def test_fit_clock_reports_total_checkpoints_and_outlier_residual():
+ speed = 1.0 + 80e-6
+ pts = [(s, 12.5 + s * speed, 1.0) for s in (60, 1200, 2400, 3600, 4800)]
+ pts.append((3000, 12.5 + 3000 * speed + 0.4, 1.0))
+ fit = fit_clock(pts)
+ assert fit is not None
+ assert fit.total_checkpoints == 6
+ assert fit.checkpoints == 5
+ # The inlier residual is tiny, but the dropped outlier was 0.4 s off: the
+ # all-checkpoints residual must still surface that.
+ assert fit.residual_ms < 0.01
+ assert fit.residual_all_ms == pytest.approx(400.0, abs=1.0)
+ assert fit.speed_fallback is False
def _two_person_levels():
@@ -150,7 +173,8 @@ def test_plan_cuts_follows_the_speaker_and_folds_short_blips():
cuts = plan_cuts(labels, ["host", "guest"], CAMS, range_start=0, range_end=27.5, min_shot=2.0)
assert [c["source_id"] for c in cuts] == ["cam_host", "cam_guest"]
assert cuts[0]["start"] == 0 and cuts[-1]["end"] == 27.5
- assert abs(cuts[1]["start"] - 17.5) < 0.02
+ # The guest starts at 17.5 after a pause, so the cut leads in to them.
+ assert abs(cuts[1]["start"] - (17.5 - LEAD_IN)) < 0.02
def test_plan_cuts_goes_wide_on_crosstalk():
@@ -261,3 +285,42 @@ def test_remote_style_returns_to_the_split_soon_after_the_answer_ends():
guests=frozenset({"guest"}), host_solo=False, guest_min=8, guest_delay=4)
solo = next(c for c in cuts if c["source_id"] == "cam_guest")
assert abs(solo["start"] - 8) < 0.05 and abs(solo["end"] - 25) < 0.05
+
+
+def test_a_cut_on_a_speech_onset_leads_in_through_silence_only():
+ # Host, a pause, then the guest: the cut moves LEAD_IN into the pause.
+ cuts = plan_cuts(_labels([(0, 6), (SILENT, 1), (1, 6)]), ["host", "guest"], CAMS,
+ range_start=0, range_end=13, min_shot=2.0)
+ assert [c["source_id"] for c in cuts] == ["cam_host", "cam_guest"]
+ assert cuts[0]["end"] == cuts[1]["start"] == pytest.approx(7.0 - LEAD_IN)
+
+ # A pause shorter than the lead-in: the cut stops at the host's last word.
+ cuts = plan_cuts(_labels([(0, 6), (SILENT, 0.05), (1, 6)]), ["host", "guest"], CAMS,
+ range_start=0, range_end=12.05, min_shot=2.0)
+ assert cuts[1]["start"] == pytest.approx(6.0)
+
+ # No pause at all: the guest's first word lands on the cut, as before.
+ cuts = plan_cuts(_labels([(0, 6), (1, 6)]), ["host", "guest"], CAMS, range_start=0, range_end=12, min_shot=2.0)
+ assert cuts[1]["start"] == pytest.approx(6.0)
+
+ # The host's shot can't drop under min_shot to make room.
+ cuts = plan_cuts(_labels([(SILENT, 1.0), (0, 1.0), (SILENT, 1.05), (1, 6)]), ["host", "guest"], CAMS,
+ range_start=1.0, range_end=9.05, min_shot=2.0)
+ assert [c["source_id"] for c in cuts] == ["cam_host", "cam_guest"]
+ assert cuts[1]["start"] == pytest.approx(3.0)
+
+ # Nor reach back before the guest's camera started rolling.
+ late = [Camera("cam_host", "host", 0, 100), Camera("cam_guest", "guest", 6.95, 100)]
+ cuts = plan_cuts(_labels([(0, 6), (SILENT, 1), (1, 6)]), ["host", "guest"], late,
+ range_start=0, range_end=13, min_shot=2.0)
+ assert cuts[1]["start"] == pytest.approx(6.95)
+
+
+def test_cuts_away_from_a_continuing_speaker_do_not_lead_in():
+ # The remote style holds the split for guest_delay into an answer; the cut
+ # to the guest falls mid-sentence, not on an onset, so it stays put.
+ labels = _labels([(0, 4), (SILENT, 1), (1, 20)])
+ cuts = plan_cuts(labels, ["host", "guest"], CAMS, range_start=0, range_end=25, min_shot=2.0,
+ guests=frozenset({"guest"}), host_solo=False, guest_min=8.0, guest_delay=4.0)
+ solo = next(c for c in cuts if c["source_id"] == "cam_guest")
+ assert solo["start"] == pytest.approx(9.0)
diff --git a/tests/test_omnilingual_model_pins.py b/tests/test_omnilingual_model_pins.py
new file mode 100644
index 0000000..e3f4da6
--- /dev/null
+++ b/tests/test_omnilingual_model_pins.py
@@ -0,0 +1,101 @@
+import io
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "backend"))
+
+from services import transcription_omnilingual as omni
+
+PROVISION_GO = os.path.join(os.path.dirname(__file__), "..", "cli", "internal", "provision", "provision.go")
+
+
+def test_python_pins_match_the_launcher():
+ go = open(PROVISION_GO, encoding="utf-8").read()
+ assert f'omnilingualRevision = "{omni.MODEL_REVISION}"' in go
+ assert omni.MODEL_REPO in go
+ for name, (sha, _) in omni.MODEL_FILES.items():
+ block = re.search(rf'"{re.escape(name)}": \{{.*?SHA256: "([0-9a-f]+)"', go, re.S)
+ assert block and block.group(1) == sha, name
+
+
+def _fake_files(monkeypatch, payloads):
+ files = {name: (omni.hashlib.sha256(data).hexdigest(), len(data)) for name, data in payloads.items()}
+ monkeypatch.setattr(omni, "MODEL_FILES", files)
+ fetched = []
+
+ def urlopen(request, timeout=0):
+ name = request.full_url.rsplit("/", 1)[-1]
+ fetched.append(name)
+ return io.BytesIO(payloads[name])
+
+ monkeypatch.setattr(omni.urllib.request, "urlopen", urlopen)
+ return fetched
+
+
+def test_downloads_missing_files_and_keeps_ones_of_the_pinned_size(tmp_path, monkeypatch):
+ fetched = _fake_files(monkeypatch, {"model.int8.onnx": b"weights", "tokens.txt": b"tok"})
+ (tmp_path / "tokens.txt").write_bytes(b"tok")
+
+ omni.ensure_model(str(tmp_path))
+
+ assert fetched == ["model.int8.onnx"]
+ assert (tmp_path / "model.int8.onnx").read_bytes() == b"weights"
+
+
+def test_replaces_an_older_model_of_a_different_size(tmp_path, monkeypatch):
+ fetched = _fake_files(monkeypatch, {"model.int8.onnx": b"new weights"})
+ (tmp_path / "model.int8.onnx").write_bytes(b"old")
+
+ omni.ensure_model(str(tmp_path))
+
+ assert fetched == ["model.int8.onnx"]
+ assert (tmp_path / "model.int8.onnx").read_bytes() == b"new weights"
+
+
+def test_a_checksum_mismatch_leaves_nothing_behind(tmp_path, monkeypatch):
+ _fake_files(monkeypatch, {"model.int8.onnx": b"weights"})
+ monkeypatch.setattr(omni, "MODEL_FILES", {"model.int8.onnx": ("0" * 64, 7)})
+
+ with pytest.raises(RuntimeError, match="checksum mismatch"):
+ omni.ensure_model(str(tmp_path))
+
+ assert os.listdir(tmp_path) == []
+
+
+def test_transcribing_with_a_custom_model_path_never_downloads(tmp_path, monkeypatch):
+ from services import transcription as tr
+
+ model = tmp_path / "model.int8.onnx"
+ tokens = tmp_path / "tokens.txt"
+ model.write_bytes(b"m")
+ tokens.write_bytes(b"t")
+ media = tmp_path / "a.wav"
+ media.write_bytes(b"x")
+ monkeypatch.setattr(tr, "_omnilingual_model", lambda: str(model))
+ monkeypatch.setattr(tr, "_omnilingual_tokens", lambda: str(tokens))
+ monkeypatch.setattr(omni, "ensure_model", lambda *a, **k: pytest.fail("downloaded beside a custom model"))
+ monkeypatch.setattr(omni, "transcribe_file", lambda *a, **k: {"transcript": "", "segments": [], "words": [], "duration": 0.0, "language": "und"})
+
+ tr._transcribe_with_omnilingual(str(media), progress_callback=None)
+
+
+def test_transcribing_with_the_managed_model_folder_ensures_the_pinned_files(tmp_path, monkeypatch):
+ from services import transcription as tr
+
+ monkeypatch.setenv("PODCLI_HOME", str(tmp_path))
+ managed = tmp_path / "models" / "omnilingual"
+ managed.mkdir(parents=True)
+ (managed / "model.int8.onnx").write_bytes(b"m")
+ (managed / "tokens.txt").write_bytes(b"t")
+ media = tmp_path / "a.wav"
+ media.write_bytes(b"x")
+ calls = []
+ monkeypatch.setattr(omni, "ensure_model", lambda d, *a, **k: calls.append(d))
+ monkeypatch.setattr(omni, "transcribe_file", lambda *a, **k: {"transcript": "", "segments": [], "words": [], "duration": 0.0, "language": "und"})
+
+ tr._transcribe_with_omnilingual(str(media), progress_callback=None)
+
+ assert calls == [str(managed)]
diff --git a/tests/test_opening_hook.py b/tests/test_opening_hook.py
new file mode 100644
index 0000000..a3765d1
--- /dev/null
+++ b/tests/test_opening_hook.py
@@ -0,0 +1,324 @@
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+import numpy as np
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import clip_generator as cg
+from services.opening_hook import (
+ keyframes_to_playback,
+ order_with_hook,
+ snap_hook_to_words,
+ validate_hook,
+)
+
+
+class ValidateHookTests(unittest.TestCase):
+ def test_none_means_no_hook(self):
+ self.assertIsNone(validate_hook(None, 10, 40))
+
+ def test_valid_hook_is_normalized(self):
+ hook = validate_hook({"start": 12, "end": 15, "mode": "move"}, 10, 40)
+ self.assertEqual(hook, {"start": 12.0, "end": 15.0, "mode": "move"})
+
+ def test_hook_on_the_clip_edges_is_inside(self):
+ self.assertIsNotNone(validate_hook({"start": 10, "end": 12, "mode": "repeat"}, 10, 40))
+ self.assertIsNotNone(validate_hook({"start": 38, "end": 40, "mode": "repeat"}, 10, 40))
+
+ def test_end_must_follow_start(self):
+ with self.assertRaisesRegex(ValueError, "greater than"):
+ validate_hook({"start": 15, "end": 15, "mode": "repeat"}, 10, 40)
+
+ def test_shorter_than_one_second_is_rejected(self):
+ with self.assertRaisesRegex(ValueError, "between 1 and 15"):
+ validate_hook({"start": 12, "end": 12.5, "mode": "repeat"}, 10, 40)
+
+ def test_longer_than_fifteen_seconds_is_rejected(self):
+ with self.assertRaisesRegex(ValueError, "between 1 and 15"):
+ validate_hook({"start": 12, "end": 28, "mode": "repeat"}, 10, 40)
+
+ def test_outside_the_clip_range_is_rejected(self):
+ with self.assertRaisesRegex(ValueError, "not inside the clip body"):
+ validate_hook({"start": 8, "end": 11, "mode": "repeat"}, 10, 40)
+ with self.assertRaisesRegex(ValueError, "not inside the clip body"):
+ validate_hook({"start": 39, "end": 41, "mode": "repeat"}, 10, 40)
+
+ def test_hook_in_a_gap_between_segments_is_rejected(self):
+ segments = [{"start": 10, "end": 15}, {"start": 20, "end": 30}]
+ with self.assertRaisesRegex(ValueError, "not inside the clip body"):
+ validate_hook({"start": 14, "end": 17, "mode": "repeat"}, 10, 30, segments)
+
+ def test_hook_across_touching_segments_is_inside(self):
+ segments = [{"start": 10, "end": 15}, {"start": 15, "end": 30}]
+ self.assertIsNotNone(
+ validate_hook({"start": 14, "end": 17, "mode": "repeat"}, 10, 30, segments)
+ )
+
+ def test_unknown_mode_is_rejected(self):
+ with self.assertRaisesRegex(ValueError, "repeat"):
+ validate_hook({"start": 12, "end": 14, "mode": "loop"}, 10, 40)
+ with self.assertRaisesRegex(ValueError, "repeat"):
+ validate_hook({"start": 12, "end": 14}, 10, 40)
+
+ def test_non_numeric_bounds_are_rejected(self):
+ with self.assertRaisesRegex(ValueError, "numbers"):
+ validate_hook({"start": "12", "end": 14, "mode": "repeat"}, 10, 40)
+ with self.assertRaisesRegex(ValueError, "numbers"):
+ validate_hook({"start": True, "end": 14, "mode": "repeat"}, 10, 40)
+
+ def test_non_object_is_rejected(self):
+ with self.assertRaisesRegex(ValueError, "object"):
+ validate_hook([12, 14], 10, 40)
+
+ def test_generate_clip_rejects_a_bad_hook_before_rendering(self):
+ with tempfile.TemporaryDirectory() as td:
+ video = os.path.join(td, "v.mp4")
+ open(video, "wb").close()
+ with mock.patch.object(cg, "cut_segment") as cut, \
+ mock.patch.object(cg, "cut_multi_segment") as cut_multi:
+ with self.assertRaisesRegex(ValueError, "not inside the clip body"):
+ cg.generate_clip(
+ video_path=video, start_second=10, end_second=20,
+ hook={"start": 25, "end": 27, "mode": "repeat"},
+ )
+ cut.assert_not_called()
+ cut_multi.assert_not_called()
+
+
+class OrderWithHookTests(unittest.TestCase):
+ body = [{"start": 10, "end": 20}, {"start": 22, "end": 30}]
+
+ def test_repeat_plays_the_hook_then_the_whole_body(self):
+ ranges = order_with_hook(self.body, {"start": 24, "end": 27, "mode": "repeat"})
+ self.assertEqual(ranges, [
+ {"start": 24, "end": 27},
+ {"start": 10, "end": 20},
+ {"start": 22, "end": 30},
+ ])
+
+ def test_move_removes_the_hook_from_the_body(self):
+ ranges = order_with_hook(self.body, {"start": 24, "end": 27, "mode": "move"})
+ self.assertEqual(ranges, [
+ {"start": 24, "end": 27},
+ {"start": 10, "end": 20},
+ {"start": 22, "end": 24},
+ {"start": 27, "end": 30},
+ ])
+
+ def test_move_drops_a_sliver_left_at_a_segment_edge(self):
+ ranges = order_with_hook(self.body, {"start": 22.05, "end": 25, "mode": "move"})
+ self.assertEqual(ranges, [
+ {"start": 22.05, "end": 25},
+ {"start": 10, "end": 20},
+ {"start": 25, "end": 30},
+ ])
+
+ def test_move_cannot_take_the_whole_body(self):
+ with self.assertRaisesRegex(ValueError, "whole clip"):
+ order_with_hook([{"start": 10, "end": 14}], {"start": 10, "end": 14, "mode": "move"})
+
+
+class SnapHookTests(unittest.TestCase):
+ words = [
+ {"word": "one", "start": 1.0, "end": 1.6},
+ {"word": "two", "start": 1.8, "end": 2.4},
+ {"word": "three", "start": 2.6, "end": 3.3},
+ ]
+
+ def test_edges_inside_words_widen_to_the_whole_word(self):
+ hook = snap_hook_to_words({"start": 1.2, "end": 3.0, "mode": "repeat"}, self.words)
+ self.assertEqual((hook["start"], hook["end"]), (1.0, 3.3))
+ self.assertEqual(hook["mode"], "repeat")
+
+ def test_edges_in_gaps_are_left_alone(self):
+ hook = snap_hook_to_words({"start": 1.7, "end": 2.5, "mode": "move"}, self.words)
+ self.assertEqual((hook["start"], hook["end"]), (1.7, 2.5))
+
+
+class KeyframesToPlaybackTests(unittest.TestCase):
+ def test_identity_for_one_range_at_the_origin(self):
+ kf = [{"t": 0.0, "x_pct": 30}, {"t": 4.0, "x_pct": 70}]
+ out = keyframes_to_playback(kf, 10.0, [{"start": 10.0, "end": 20.0}])
+ self.assertEqual(out, kf)
+
+ def test_each_range_opens_on_the_keyframe_in_force(self):
+ # Source clock: 30% from 10s, 70% from 14s. The hook 15-17 plays first
+ # at 70%; the body then replays 10-20 from 30% and switches at 4s.
+ kf = [{"t": 0.0, "x_pct": 30}, {"t": 4.0, "x_pct": 70}]
+ ranges = [{"start": 15.0, "end": 17.0}, {"start": 10.0, "end": 20.0}]
+ out = keyframes_to_playback(kf, 10.0, ranges, part_durations=[2.0, 10.0])
+ self.assertEqual(out, [
+ {"t": 0.0, "x_pct": 70},
+ {"t": 2.0, "x_pct": 30},
+ {"t": 6.0, "x_pct": 70},
+ ])
+
+
+ shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg/ffprobe not installed"
+)
+class OpeningHookRenderTests(unittest.TestCase):
+ """Real render of a source whose every second has its own colour and its
+ own tone, so the order of the output can be read back frame by frame."""
+
+ FPS = 25
+ COLORS = ["red", "green", "blue", "yellow", "cyan", "magenta", "white", "gray", "orange", "purple"]
+ RGB = {
+ "red": (255, 0, 0), "green": (0, 128, 0), "blue": (0, 0, 255),
+ "yellow": (255, 255, 0), "cyan": (0, 255, 255), "magenta": (255, 0, 255),
+ "white": (255, 255, 255), "gray": (128, 128, 128), "orange": (255, 165, 0),
+ "purple": (128, 0, 128),
+ }
+
+ @staticmethod
+ def freq(second):
+ return 300 + 100 * second
+
+ @classmethod
+ def setUpClass(cls):
+ cls.tmpdir = tempfile.mkdtemp(prefix="podcli-hook-test-")
+ cls.src = os.path.join(cls.tmpdir, "src.mp4")
+ n = len(cls.COLORS)
+ cmd = ["ffmpeg", "-y", "-loglevel", "error"]
+ for color in cls.COLORS:
+ cmd += ["-f", "lavfi", "-i", f"color=c={color}:s=320x240:r={cls.FPS}:d=1"]
+ for i in range(n):
+ cmd += ["-f", "lavfi", "-i", f"sine=frequency={cls.freq(i)}:sample_rate=44100:duration=1"]
+ pairs = "".join(f"[{i}:v][{n + i}:a]" for i in range(n))
+ cmd += [
+ "-filter_complex", f"{pairs}concat=n={n}:v=1:a=1[v][a]",
+ "-map", "[v]", "-map", "[a]",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
+ cls.src,
+ ]
+ subprocess.run(cmd, check=True, capture_output=True)
+ # One word per second, well inside it.
+ cls.words = [
+ {"word": f"w{i}", "start": i + 0.2, "end": i + 0.7, "speaker": None}
+ for i in range(n)
+ ]
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.tmpdir, ignore_errors=True)
+
+ def _render(self, mode):
+ captured = {}
+ real_write = cg.write_sidecars
+
+ def spy(words, base, *a, **kw):
+ captured["words"] = words
+ return real_write(words, base, *a, **kw)
+
+ with mock.patch.object(cg, "write_sidecars", side_effect=spy):
+ result = cg.generate_clip(
+ video_path=self.src,
+ start_second=2.0,
+ end_second=8.0,
+ caption_style="subtle",
+ crop_strategy="center",
+ transcript_words=self.words,
+ title=f"hook_{mode}",
+ output_dir=os.path.join(self.tmpdir, mode),
+ captions=False,
+ # Sidecars otherwise follow `captions`; this test inspects
+ # the sidecar words directly, so request them explicitly.
+ write_subtitles=True,
+ clean_fillers=False,
+ trim_opening=False,
+ hook={"start": 5.0, "end": 7.0, "mode": mode},
+ )
+ return result, captured["words"]
+
+ def _frame_rgb(self, path, t):
+ raw = subprocess.run(
+ ["ffmpeg", "-loglevel", "error", "-ss", f"{t:.3f}", "-i", path,
+ "-frames:v", "1", "-vf", "scale=32:32", "-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
+ check=True, capture_output=True,
+ ).stdout
+ px = np.frombuffer(raw, dtype=np.uint8).reshape(-1, 3)
+ return tuple(int(c) for c in px.mean(axis=0).round())
+
+ def _dominant_freq(self, path, t, window=0.4):
+ raw = subprocess.run(
+ ["ffmpeg", "-loglevel", "error", "-ss", f"{t:.3f}", "-i", path, "-t", f"{window}",
+ "-vn", "-ac", "1", "-ar", "16000", "-f", "s16le", "-"],
+ check=True, capture_output=True,
+ ).stdout
+ samples = np.frombuffer(raw, dtype=np.int16).astype(np.float64)
+ spectrum = np.abs(np.fft.rfft(samples * np.hanning(len(samples))))
+ freqs = np.fft.rfftfreq(len(samples), 1 / 16000)
+ return float(freqs[int(spectrum.argmax())])
+
+ def _video_duration(self, path):
+ out = subprocess.run(
+ ["ffprobe", "-v", "error", "-select_streams", "v:0", "-count_frames",
+ "-show_entries", "stream=nb_read_frames", "-of", "csv=p=0", path],
+ check=True, capture_output=True, text=True,
+ ).stdout.strip()
+ return int(out) / self.FPS
+
+ def _assert_plays(self, path, playback_t, source_second):
+ got = self._frame_rgb(path, playback_t)
+ want = self.RGB[self.COLORS[source_second]]
+ for g, w in zip(got, want):
+ self.assertLessEqual(
+ abs(g - w), 40,
+ f"at {playback_t}s the frame is {got}, expected source second "
+ f"{source_second} ({self.COLORS[source_second]} {want})",
+ )
+ self.assertAlmostEqual(
+ self._dominant_freq(path, playback_t - 0.2), self.freq(source_second), delta=15,
+ msg=f"at {playback_t}s the tone is not source second {source_second}",
+ )
+
+ def _assert_words(self, words, expected):
+ got = [(w["word"], w["start"]) for w in words]
+ self.assertEqual([g[0] for g in got], [e[0] for e in expected])
+ for (word, start), (_, want) in zip(got, expected):
+ self.assertAlmostEqual(start, want, delta=0.05, msg=f"{word} at {start}, expected {want}")
+
+ def test_repeat_plays_the_hook_first_then_the_whole_body(self):
+ result, words = self._render("repeat")
+ path = result["output_path"]
+ # Hook 5-7, then body 2-8: 2 + 6 seconds.
+ plan = [(0.5, 5), (1.5, 6), (2.5, 2), (3.5, 3), (4.5, 4), (5.5, 5), (6.5, 6), (7.5, 7)]
+ for playback_t, source_second in plan:
+ self._assert_plays(path, playback_t, source_second)
+ self.assertAlmostEqual(self._video_duration(path), 8.0, delta=1 / self.FPS)
+ self.assertAlmostEqual(result["duration"], 8.0, delta=0.1)
+ self.assertEqual(result["hook"], {"start": 5.0, "end": 7.0, "mode": "repeat"})
+ self._assert_words(words, [
+ ("w5", 0.2), ("w6", 1.2),
+ ("w2", 2.2), ("w3", 3.2), ("w4", 4.2), ("w5", 5.2), ("w6", 6.2), ("w7", 7.2),
+ ])
+ with open(result["srt_path"], encoding="utf-8") as f:
+ self.assertIn("w5 w6 w2 w3 w4 w5 w6 w7", f.read())
+
+ def test_move_plays_the_hook_first_and_drops_it_from_the_body(self):
+ result, words = self._render("move")
+ path = result["output_path"]
+ # Hook 5-7, then body 2-5 and 7-8: 2 + 3 + 1 seconds.
+ plan = [(0.5, 5), (1.5, 6), (2.5, 2), (3.5, 3), (4.5, 4), (5.5, 7)]
+ for playback_t, source_second in plan:
+ self._assert_plays(path, playback_t, source_second)
+ # Each of the three parts can round up to a whole frame on its own,
+ # depending on the ffmpeg build; captions follow the probed lengths.
+ self.assertAlmostEqual(self._video_duration(path), 6.0, delta=3 / self.FPS)
+ self.assertAlmostEqual(result["duration"], 6.0, delta=0.1)
+ self._assert_words(words, [
+ ("w5", 0.2), ("w6", 1.2), ("w2", 2.2), ("w3", 3.2), ("w4", 4.2), ("w7", 5.2),
+ ])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_signal_cache.py b/tests/test_signal_cache.py
new file mode 100644
index 0000000..10b710c
--- /dev/null
+++ b/tests/test_signal_cache.py
@@ -0,0 +1,42 @@
+import os
+import sys
+
+import pytest
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from config.paths import paths # noqa: E402
+from services import signal_cache # noqa: E402
+
+
+def sandbox(tmp_path, monkeypatch):
+ monkeypatch.setitem(paths, "cache", str(tmp_path / "cache"))
+ video = tmp_path / "video.mp4"
+ video.write_bytes(b"fake video bytes")
+ return str(video)
+
+
+def test_save_and_load_round_trip(sandbox):
+ signal_cache.save_signals(sandbox, energy_data=[{"t": 1.0, "v": 0.5}])
+ assert signal_cache.load_signals(sandbox) == {"energy_data": [{"t": 1.0, "v": 0.5}]}
+
+
+def test_load_misses_cleanly_when_nothing_cached(sandbox):
+ assert signal_cache.load_signals(sandbox) == {}
+
+
+def test_cache_path_is_salted_with_the_analyzer_version(sandbox, monkeypatch):
+ signal_cache.save_signals(sandbox, energy_data=[{"t": 1.0}])
+ path_now = signal_cache._signals_path(sandbox)
+ assert os.path.exists(path_now)
+
+ # Simulate an analyzer algorithm change: bumping the version must stop
+ # serving the old profile and must not collide with its cache file.
+ monkeypatch.setattr(signal_cache, "SIGNAL_CACHE_VERSION", signal_cache.SIGNAL_CACHE_VERSION + 1)
+ path_next = signal_cache._signals_path(sandbox)
+ assert path_next != path_now
+ assert signal_cache.load_signals(sandbox) == {}
diff --git a/tests/test_silence_removal.py b/tests/test_silence_removal.py
index bc3af06..acabe4c 100644
--- a/tests/test_silence_removal.py
+++ b/tests/test_silence_removal.py
@@ -1,14 +1,18 @@
import os
import sys
+import pytest
+
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
BACKEND_ROOT = os.path.join(ROOT, "backend")
if BACKEND_ROOT not in sys.path:
sys.path.insert(0, BACKEND_ROOT)
from services.silence_removal import (
+ _map_range,
plan_silence_removal,
probabilities_to_speech_segments,
+ remap_timed_items,
remap_transcript,
)
@@ -64,6 +68,68 @@ def test_remap_transcript_closes_removed_gaps():
assert remapped["duration"] == 3.0
+def test_zero_duration_word_inside_a_kept_range_is_kept_as_a_point():
+ keep_segments = [{"start": 0.5, "end": 2.0}, {"start": 4.5, "end": 6.0}]
+ words = [{"word": "", "start": 1.2, "end": 1.2}]
+
+ remapped = remap_timed_items(words, keep_segments, assign_by_midpoint=True)
+
+ assert len(remapped) == 1
+ assert remapped[0]["start"] == remapped[0]["end"] == 0.7
+
+
+def test_short_whole_word_inside_a_kept_range_survives_float_rounding():
+ # 10.50 - 10.49 is 0.00999... in floating point, under the 10ms floor,
+ # but the word kept its whole length, so no cut clipped it.
+ keep_segments = [{"start": 0.0, "end": 44.66}, {"start": 45.04, "end": 60.0}]
+ words = [{"word": "in", "start": 10.49, "end": 10.5}, {"word": "me", "start": 50.57, "end": 50.58}]
+
+ remapped = remap_timed_items(words, keep_segments, assign_by_midpoint=True)
+
+ assert [w["word"] for w in remapped] == ["in", "me"]
+
+
+def test_word_clipped_to_a_sliver_by_a_cut_is_dropped():
+ keep_segments = [{"start": 0.0, "end": 1.005}, {"start": 3.0, "end": 4.0}]
+ words = [{"word": "gone", "start": 1.0, "end": 1.5}]
+
+ assert remap_timed_items(words, keep_segments) == []
+
+
+def test_zero_duration_word_in_a_removed_range_is_dropped():
+ keep_segments = [{"start": 0.5, "end": 2.0}, {"start": 4.5, "end": 6.0}]
+ words = [{"word": "", "start": 3.0, "end": 3.0}]
+
+ assert remap_timed_items(words, keep_segments, assign_by_midpoint=True) == []
+
+
+def test_word_straddling_a_removed_gap_with_no_owning_side_is_dropped():
+ # Word runs 1.8-2.2 across a removed gap from 1.9-4.0. Its midpoint (2.0)
+ # falls inside that gap, so neither side owns it; it's dropped rather
+ # than stitched across the cut.
+ keep_segments = [{"start": 0.0, "end": 1.9}, {"start": 4.0, "end": 6.0}]
+ mapped = _map_range(1.8, 2.2, keep_segments, assign_by_midpoint=True)
+ assert mapped is None
+
+
+def test_word_straddling_a_cut_is_not_stitched_across_it():
+ keep_segments = [{"start": 0.0, "end": 2.0}, {"start": 2.0, "end": 4.0}]
+ # Midpoint 2.05 belongs to the second segment; the old stitching behavior
+ # would have spanned from the first segment's overlap through the
+ # second's, inflating the word's output duration across the cut.
+ mapped = _map_range(1.9, 2.2, keep_segments, assign_by_midpoint=True)
+ assert mapped == (2.0, 2.2)
+
+
+def test_segments_still_stitch_across_a_removed_gap():
+ # Sentence-level segments intentionally keep the old stitching behavior
+ # (see test_remap_transcript_closes_removed_gaps), only words are
+ # reassigned by midpoint.
+ keep_segments = [{"start": 0.5, "end": 2.0}, {"start": 4.5, "end": 6.0}]
+ mapped = _map_range(1.0, 5.4, keep_segments)
+ assert mapped == pytest.approx((0.5, 2.4))
+
+
def test_probability_hysteresis_ignores_short_noise():
probabilities = [0.0] * 5 + [0.8] * 12 + [0.0] * 8 + [0.9] * 2 + [0.0] * 8
speech = probabilities_to_speech_segments(
diff --git a/tests/test_subtitle_export.py b/tests/test_subtitle_export.py
new file mode 100644
index 0000000..0e4cdcf
--- /dev/null
+++ b/tests/test_subtitle_export.py
@@ -0,0 +1,110 @@
+"""Tests for backend.services.subtitle_export: SRT/VTT sidecar generation."""
+
+import os
+import sys
+import tempfile
+import unittest
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import subtitle_export as se
+
+
+def _word(text, start, end):
+ return {"word": text, "start": start, "end": end}
+
+
+class TimestampTests(unittest.TestCase):
+ def test_srt_timestamp_format(self):
+ self.assertEqual(se._timestamp(65.25, vtt=False), "00:01:05,250")
+
+ def test_vtt_timestamp_format(self):
+ self.assertEqual(se._timestamp(65.25, vtt=True), "00:01:05.250")
+
+ def test_rounds_milliseconds_without_overflow(self):
+ # 1.9996 * 1000 = 1999.6 -> rounds to 2000ms, carried into seconds,
+ # never the invalid "01,1000".
+ self.assertEqual(se._timestamp(1.9996, vtt=False), "00:00:02,000")
+
+ def test_negative_seconds_clamped_to_zero(self):
+ self.assertEqual(se._timestamp(-0.5, vtt=False), "00:00:00,000")
+
+
+class GroupWordsTests(unittest.TestCase):
+ def test_groups_by_words_per_line(self):
+ words = [_word(f"w{i}", i, i + 0.5) for i in range(10)]
+ groups = se._group_words(words, words_per_line=4)
+ self.assertEqual([len(g["text"].split()) for g in groups], [4, 4, 2])
+
+ def test_group_spans_first_start_to_last_end(self):
+ words = [_word("a", 1.0, 1.5), _word("b", 1.5, 2.5), _word("c", 2.5, 3.0)]
+ groups = se._group_words(words, words_per_line=3)
+ self.assertEqual(groups[0]["start"], 1.0)
+ self.assertEqual(groups[0]["end"], 3.0)
+
+ def test_skips_blank_words(self):
+ words = [_word("a", 0, 1), _word("", 1, 1), _word("b", 1, 2)]
+ groups = se._group_words(words, words_per_line=8)
+ self.assertEqual(groups[0]["text"], "a b")
+
+ def test_empty_input_yields_no_groups(self):
+ self.assertEqual(se._group_words([], words_per_line=8), [])
+
+
+class WordsToSrtVttTests(unittest.TestCase):
+ def test_words_to_srt_basic(self):
+ words = [_word("Hello", 0.0, 0.5), _word("world", 0.5, 1.0)]
+ srt = se.words_to_srt(words, words_per_line=8)
+ self.assertEqual(
+ srt,
+ "1\n00:00:00,000 --> 00:00:01,000\nHello world\n",
+ )
+
+ def test_words_to_vtt_basic(self):
+ words = [_word("Hello", 0.0, 0.5), _word("world", 0.5, 1.0)]
+ vtt = se.words_to_vtt(words, words_per_line=8)
+ self.assertEqual(
+ vtt,
+ "WEBVTT\n\n1\n00:00:00.000 --> 00:00:01.000\nHello world\n",
+ )
+
+ def test_no_words_returns_none(self):
+ self.assertIsNone(se.words_to_srt([]))
+ self.assertIsNone(se.words_to_vtt([]))
+
+ def test_multiple_lines_are_numbered_in_order(self):
+ words = [_word(f"w{i}", i, i + 0.5) for i in range(5)]
+ srt = se.words_to_srt(words, words_per_line=2)
+ self.assertIn("1\n", srt)
+ self.assertIn("2\n", srt)
+ self.assertIn("3\n", srt)
+
+
+class WriteSidecarsTests(unittest.TestCase):
+ def test_writes_both_files_and_returns_their_paths(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ base = os.path.join(tmp, "clip")
+ words = [_word("Hello", 0.0, 0.5), _word("world", 0.5, 1.0)]
+ paths = se.write_sidecars(words, base)
+
+ self.assertEqual(paths["srt_path"], f"{base}.srt")
+ self.assertEqual(paths["vtt_path"], f"{base}.vtt")
+ self.assertTrue(os.path.exists(paths["srt_path"]))
+ self.assertTrue(os.path.exists(paths["vtt_path"]))
+ with open(paths["srt_path"], encoding="utf-8") as f:
+ self.assertIn("Hello world", f.read())
+
+ def test_writes_nothing_when_there_are_no_words(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ base = os.path.join(tmp, "clip")
+ paths = se.write_sidecars([], base)
+ self.assertEqual(paths, {})
+ self.assertFalse(os.path.exists(f"{base}.srt"))
+ self.assertFalse(os.path.exists(f"{base}.vtt"))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_text_utils.py b/tests/test_text_utils.py
index 64a2187..066771b 100644
--- a/tests/test_text_utils.py
+++ b/tests/test_text_utils.py
@@ -3,7 +3,7 @@ import sys
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "backend"))
-from utils.text import clean_title, safe_filename, truncate_title # noqa: E402
+from utils.text import clean_title, safe_filename, safe_upper, truncate_title # noqa: E402
def test_clean_title_keeps_full_text():
@@ -87,3 +87,23 @@ def test_safe_filename_escapes_windows_reserved_device_names():
def test_safe_filename_leaves_names_merely_containing_reserved_words():
assert safe_filename("Conference") == "Conference"
assert safe_filename("CON artists") == "CON_artists"
+
+
+def test_safe_upper_uppercases_plain_ascii():
+ assert safe_upper("hello world") == "HELLO WORLD"
+
+
+def test_safe_upper_leaves_georgian_mkhedruli_unchanged():
+ georgian = "მიშა"
+ # Sanity check the bug this guards against: str.upper() alone does remap
+ # Mkhedruli to Mtavruli, a different alphabet, not just a different case.
+ assert georgian.upper() != georgian
+ assert safe_upper(georgian) == georgian
+
+
+def test_safe_upper_uppercases_latin_in_mixed_script_text():
+ assert safe_upper("hello მიშა") == "HELLO მიშა"
+
+
+def test_safe_upper_handles_empty_string():
+ assert safe_upper("") == ""
diff --git a/tests/test_thumbnail_ai.py b/tests/test_thumbnail_ai.py
new file mode 100644
index 0000000..50d567e
--- /dev/null
+++ b/tests/test_thumbnail_ai.py
@@ -0,0 +1,71 @@
+"""Tests for grounding thumbnail headline generation in the clip's own content
+(payoff, the question it answers, its verbatim opening line) instead of the
+title alone."""
+
+import os
+import sys
+import unittest
+from unittest import mock
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import thumbnail_ai as ta
+
+
+class GroundingContextTests(unittest.TestCase):
+ def test_empty_without_grounding(self):
+ self.assertEqual(ta._grounding_context(None), "")
+ self.assertEqual(ta._grounding_context({}), "")
+
+ def test_includes_every_field_given(self):
+ text = ta._grounding_context({
+ "payoff": "the payoff line",
+ "context_line": "the question",
+ "preview_text": "the opening line",
+ })
+ self.assertIn("the payoff line", text)
+ self.assertIn("the question", text)
+ self.assertIn("the opening line", text)
+
+ def test_omits_fields_not_given(self):
+ text = ta._grounding_context({"payoff": "only this"})
+ self.assertIn("only this", text)
+ self.assertNotIn("Question this clip answers", text)
+ self.assertNotIn("verbatim opening line", text)
+
+
+class GroundingFlowsIntoPromptTests(unittest.TestCase):
+ def test_ask_claude_for_layout_includes_grounding_in_prompt(self):
+ captured = {}
+
+ def fake_ask(prompt, timeout=30):
+ captured["prompt"] = prompt
+ return {"line1": "a", "line2": "b"}
+
+ with mock.patch.object(ta, "_ask_ai_for_json", side_effect=fake_ask):
+ ta.ask_claude_for_layout(
+ "a generic title",
+ frame_path="/tmp/frame.png",
+ grounding={"payoff": "the clip's actual payoff"},
+ )
+ self.assertIn("the clip's actual payoff", captured["prompt"])
+
+ def test_generate_headline_variations_includes_grounding_in_prompt(self):
+ captured = {}
+
+ def fake_ask(prompt, timeout=30):
+ captured["prompt"] = prompt
+ return [{"line1": "a", "line2": "b"}]
+
+ with mock.patch.object(ta, "_ask_ai_for_json", side_effect=fake_ask):
+ ta.generate_headline_variations(
+ "a generic title", 3, grounding={"preview_text": "the real opening line"}
+ )
+ self.assertIn("the real opening line", captured["prompt"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_thumbnail_html.py b/tests/test_thumbnail_html.py
index 0ae5751..2f29c07 100644
--- a/tests/test_thumbnail_html.py
+++ b/tests/test_thumbnail_html.py
@@ -14,6 +14,11 @@ from services import thumbnail_html as th
class ThumbnailHtmlTests(unittest.TestCase):
+ def test_default_font_stack_covers_georgian(self):
+ cfg = th._load_config()
+ self.assertIn("Noto Sans Georgian", cfg["font_family"])
+ self.assertIn("Noto+Sans+Georgian", cfg["font_import_url"])
+
def test_prepare_thumbnail_lines_compacts_long_sentence_title(self):
line1, line2 = th._prepare_thumbnail_lines(
"We build 10 megawatt data centers in 200 days — everyone is shocked",
@@ -177,5 +182,23 @@ class LayerTests(unittest.TestCase):
self.assertEqual(th._layer_html([]), "")
+class GeorgianCasingTests(unittest.TestCase):
+ """Georgian is caseless; uppercasing for emphasis must not swap it to a
+ different alphabet (Mkhedruli -> Mtavruli), whether that uppercasing
+ happens in Python or in the CSS the HTML carries."""
+
+ def test_build_html_uppercases_latin_but_not_georgian(self):
+ html = th._build_html("hello მიშა", "second line", config={})
+ self.assertIn("HELLO", html)
+ self.assertIn("მიშა", html)
+ self.assertNotIn("Მიშა".upper(), html)
+
+ def test_build_html_css_does_not_redundantly_uppercase(self):
+ # text-transform: uppercase in the CSS would re-break Georgian in the
+ # headless browser even after the Python-side fix above.
+ html = th._build_html("hello world", "second line", config={})
+ self.assertNotIn("text-transform: uppercase", html)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_thumbnail_pair.py b/tests/test_thumbnail_pair.py
new file mode 100644
index 0000000..dd52be2
--- /dev/null
+++ b/tests/test_thumbnail_pair.py
@@ -0,0 +1,293 @@
+"""The two-person thumbnail layout.
+
+Frames are synthetic: a red person sits left and a blue person sits right, each
+with a textured face patch so the sharpness scoring has something to read. The
+face detector is stubbed to report where those patches are, which is the one
+thing a drawn frame cannot give a real detector.
+"""
+
+import json
+import os
+from pathlib import Path
+import sys
+import tempfile
+import unittest
+from types import SimpleNamespace
+from unittest import mock
+
+import numpy as np
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+import cv2
+
+from services import multicam
+from services import thumbnail_ai
+from services import thumbnail_html as th
+from services import thumbnail_pair as tp
+
+W, H = 1920, 1080
+LEFT_X, RIGHT_X, FACE_Y, FACE = 500, 1400, 420, 200
+PANEL = (540, 1344)
+RED, BLUE = 2, 0 # BGR channel index
+
+
+def _person_frame(people=(("left", LEFT_X, RED), ("right", RIGHT_X, BLUE)), seed=0):
+ rng = np.random.default_rng(seed)
+ frame = np.full((H, W, 3), 90, dtype=np.uint8)
+ for _, x, channel in people:
+ stripe = frame[:, x - 300:x + 300]
+ stripe[:] = 20
+ stripe[:, :, channel] = 200
+ patch = frame[FACE_Y - FACE // 2:FACE_Y + FACE // 2, x - FACE // 2:x + FACE // 2]
+ patch[:, :, channel] = rng.integers(120, 255, size=patch.shape[:2], dtype=np.uint8)
+ return frame
+
+
+def _face(x, confidence=0.9):
+ return {"cx": x, "cy": FACE_Y, "fw": FACE, "fh": FACE, "confidence": confidence}
+
+
+def _dominant(path) -> int:
+ image = cv2.imread(path)
+ return int(np.argmax(image.reshape(-1, 3).mean(axis=0)))
+
+
+class SeatPairTests(unittest.TestCase):
+ def setUp(self):
+ self.tmp = tempfile.mkdtemp()
+ self.video = os.path.join(self.tmp, "episode.mp4")
+ open(self.video, "wb").close()
+ self.frames = [(float(t), _person_frame(seed=t)) for t in range(1, 7)]
+
+ def _pick(self, faces_for, **kw):
+ with mock.patch.object(tp, "_sample_frames", return_value=iter(self.frames)), \
+ mock.patch.object(tp, "_find_faces", side_effect=faces_for), \
+ mock.patch.object(multicam, "session_for_render", return_value=None):
+ return tp.pick_pair(self.video, 0.0, 8.0, os.path.join(self.tmp, "pair"), PANEL, **kw)
+
+ def _both(self, frame):
+ # The left person is clearest at 3 s, the right one at 5 s.
+ t = next(t for t, f in self.frames if f is frame)
+ return [_face(LEFT_X, 0.99 if t == 3.0 else 0.7), _face(RIGHT_X, 0.99 if t == 5.0 else 0.7)]
+
+ def test_places_two_crops_left_and_right_from_their_own_frames(self):
+ result = self._pick(self._both)
+ self.assertEqual(result["layout"], "pair")
+ left, right = result["people"]
+ self.assertEqual((left["side"], right["side"]), ("left", "right"))
+ self.assertEqual(_dominant(left["path"]), RED)
+ self.assertEqual(_dominant(right["path"]), BLUE)
+ self.assertEqual(cv2.imread(left["path"]).shape[:2], (PANEL[1], PANEL[0]))
+ self.assertEqual((left["source_time"], right["source_time"]), (3.0, 5.0))
+ self.assertEqual(result["roles"], "footage_order")
+
+ def test_swap_flips_the_sides(self):
+ result = self._pick(self._both, swap=True)
+ left, right = result["people"]
+ self.assertEqual(_dominant(left["path"]), BLUE)
+ self.assertEqual(_dominant(right["path"]), RED)
+ self.assertEqual(left["source_time"], 5.0)
+ self.assertTrue(result["swapped"])
+
+ def test_the_speaker_who_talks_more_goes_left_as_the_guest(self):
+ face_map = {"clusters": [{"center_x": LEFT_X}, {"center_x": RIGHT_X}],
+ "speaker_mappings": {"A": 0, "B": 1}}
+ segments = [{"start": 0.0, "end": 2.0, "speaker": "A"}, {"start": 2.0, "end": 8.0, "speaker": "B"}]
+ result = self._pick(self._both, face_map=face_map, segments=segments)
+ left, right = result["people"]
+ self.assertEqual((left["role"], right["role"]), ("guest", "host"))
+ self.assertEqual(_dominant(left["path"]), BLUE)
+ self.assertEqual(result["roles"], "talk_time")
+
+ def test_falls_back_to_one_face_when_only_one_person_is_on_screen(self):
+ result = self._pick(lambda frame: [_face(LEFT_X)])
+ self.assertEqual(result["layout"], "single")
+ self.assertEqual(result["people"], [])
+ self.assertIn("Only one person", result["reason"])
+ self.assertIn("single-face layout", result["reason"])
+
+ def test_falls_back_without_a_source_video(self):
+ result = tp.pick_pair(None, 0.0, 8.0, self.tmp, PANEL)
+ self.assertEqual(result["layout"], "single")
+
+
+class ImagePairTests(unittest.TestCase):
+ def setUp(self):
+ self.tmp = tempfile.mkdtemp()
+ self.red = os.path.join(self.tmp, "red.png")
+ self.blue = os.path.join(self.tmp, "blue.png")
+ cv2.imwrite(self.red, _person_frame(people=(("left", 960, RED),)))
+ cv2.imwrite(self.blue, _person_frame(people=(("left", 960, BLUE),)))
+
+ def _pick(self, **kw):
+ with mock.patch.object(tp, "_find_faces", return_value=[]):
+ return tp.pick_pair(None, None, None, os.path.join(self.tmp, "pair"), PANEL, **kw)
+
+ def test_named_images_take_their_sides(self):
+ result = self._pick(left_image=self.red, right_image=self.blue)
+ left, right = result["people"]
+ self.assertEqual(_dominant(left["path"]), RED)
+ self.assertEqual(_dominant(right["path"]), BLUE)
+ self.assertIsNone(left["source_time"])
+ self.assertEqual(result["roles"], "chosen")
+
+ def test_swap_flips_named_images(self):
+ left, right = self._pick(left_image=self.red, right_image=self.blue, swap=True)["people"]
+ self.assertEqual(_dominant(left["path"]), BLUE)
+ self.assertEqual(_dominant(right["path"]), RED)
+
+ def test_one_image_alone_is_refused(self):
+ with self.assertRaises(ValueError):
+ self._pick(left_image=self.red)
+
+
+class MulticamPairTests(unittest.TestCase):
+ def test_each_person_comes_from_their_own_camera_guest_left(self):
+ tmp = tempfile.mkdtemp()
+ video = os.path.join(tmp, "render.mp4")
+ open(video, "wb").close()
+ session = SimpleNamespace(people=[multicam.Person("host", "Nika", "host"),
+ multicam.Person("guest", "Ana", "guest")])
+ colour = {"host": RED, "guest": BLUE}
+
+ def still(_session, person_id, tl, out, width=1280):
+ cv2.imwrite(str(out), _person_frame(people=(("only", 960, colour[person_id]),)))
+ return {"path": str(out), "camera": f"/cams/{person_id}.mov", "camera_time": round(tl - 90, 3)}
+
+ with mock.patch.object(multicam, "session_for_render", return_value=session), \
+ mock.patch.object(multicam, "render_to_timeline", side_effect=lambda s, t: t + 100), \
+ mock.patch.object(multicam, "person_still", side_effect=still), \
+ mock.patch.object(tp, "_find_faces", return_value=[_face(960)]):
+ result = tp.pick_pair(video, 10.0, 30.0, os.path.join(tmp, "pair"), PANEL)
+
+ self.assertEqual(result["roles"], "multicam")
+ left, right = result["people"]
+ self.assertEqual((left["role"], left["name"]), ("guest", "Ana"))
+ self.assertEqual(_dominant(left["path"]), BLUE)
+ self.assertEqual(_dominant(right["path"]), RED)
+ self.assertEqual(left["camera"], "/cams/guest.mov")
+ self.assertTrue(10.0 <= left["source_time"] <= 30.0)
+ self.assertAlmostEqual(left["camera_time"], left["source_time"] + 10, places=1)
+
+
+class RenderToTimelineTests(unittest.TestCase):
+ def test_skips_removed_stretches(self):
+ session = multicam.MulticamSession(
+ session_id="abc123abc999", name="ep",
+ cuts=[{"start": 10.0, "end": 40.0, "source_id": "a"}],
+ removals=[{"start": 15.0, "end": 20.0}],
+ )
+ self.assertEqual(multicam.render_to_timeline(session, 2.0), 12.0)
+ self.assertEqual(multicam.render_to_timeline(session, 6.0), 21.0)
+ self.assertIsNone(multicam.render_to_timeline(session, 26.0))
+
+
+class PairHtmlTests(unittest.TestCase):
+ def setUp(self):
+ self.tmp = tempfile.mkdtemp()
+ self.left = os.path.join(self.tmp, "a.jpg")
+ self.right = os.path.join(self.tmp, "b.jpg")
+ for p in (self.left, self.right):
+ cv2.imwrite(p, np.zeros((10, 10, 3), dtype=np.uint8))
+
+ def test_two_panels_replace_the_single_photo(self):
+ people = [{"side": "left", "path": self.left}, {"side": "right", "path": self.right}]
+ html = th._build_html("Line one", "Line two", photo_path=self.left, people=people,
+ config={"pair_box_y": "86%"})
+ self.assertIn(f'<div class="pair pair-left"><img src="{Path(self.left).as_uri()}"', html)
+ self.assertIn(f'<div class="pair pair-right"><img src="{Path(self.right).as_uri()}"', html)
+ self.assertIn('class="pair-divider"', html)
+ self.assertNotIn('class="photo"', html)
+ self.assertIn("top: 86%", html)
+
+ def test_a_missing_panel_draws_the_single_photo(self):
+ people = [{"side": "left", "path": self.left}, {"side": "right", "path": "/nope.jpg"}]
+ html = th._build_html("Line one", "Line two", photo_path=self.left, people=people)
+ self.assertIn('class="photo"', html)
+ self.assertNotIn("pair-left", html)
+
+ def test_panel_size_follows_the_canvas(self):
+ self.assertEqual(th.pair_panel_size({"width": 1080, "height": 1920, "pair_photo_height": "70%"}), (540, 1344))
+
+
+class RenderVariationsPairTests(unittest.TestCase):
+ def test_pair_layout_reaches_every_variation(self):
+ tmp = tempfile.mkdtemp()
+ pair = {"layout": "pair", "roles": "footage_order", "swapped": False,
+ "people": [{"side": "left", "path": "l.jpg", "source_time": 3.0},
+ {"side": "right", "path": "r.jpg", "source_time": 5.0}]}
+ with mock.patch.object(tp, "pick_pair", return_value=pair) as pick, \
+ mock.patch.object(thumbnail_ai, "generate_thumbnail_with_template",
+ side_effect=lambda **kw: kw["output_path"]) as draw:
+ out = thumbnail_ai.render_variations(
+ "Title", tmp, video_path="v.mp4", start_second=0.0, end_second=8.0,
+ config={"variations": 2, "layout": "pair"}, line1="One", line2="Two",
+ grounding={"payoff": "You learn X"},
+ )
+ self.assertEqual(len(out["paths"]), 2)
+ self.assertIs(out["pair"], pair)
+ self.assertEqual(pick.call_args.args[1:3], (0.0, 8.0))
+ for call in draw.call_args_list:
+ self.assertEqual(call.kwargs["people"], pair["people"])
+ self.assertEqual(call.kwargs["grounding"], {"payoff": "You learn X"})
+
+ def test_single_layout_never_looks_for_two_people(self):
+ with mock.patch.object(tp, "pick_pair") as pick:
+ self.assertIsNone(thumbnail_ai.resolve_pair({"layout": "single"}, tempfile.mkdtemp(), "v.mp4", 0.0, 8.0))
+ pick.assert_not_called()
+
+
+class ThumbnailRenderCliTests(unittest.TestCase):
+ def test_result_records_each_face_and_its_source_time(self):
+ import argparse
+ import io
+ from contextlib import redirect_stdout
+
+ import cli as cli_mod
+
+ tmp = tempfile.mkdtemp()
+ pair = {"layout": "pair", "roles": "talk_time", "swapped": False, "people": [
+ {"side": "left", "role": "guest", "from": "seats", "path": "l.jpg", "source_time": 12.5},
+ {"side": "right", "role": "host", "from": "seats", "path": "r.jpg", "source_time": 31.0},
+ ]}
+ args = argparse.Namespace(
+ title="Title", frame=None, output=os.path.join(tmp, "t.png"), line1="One", line2="Two",
+ frame_info=None, logo=None, payoff=None, context_line=None, preview_text=None,
+ video="v.mp4", start=10.0, end=40.0, layout="pair", left_image=None, right_image=None, swap=False,
+ )
+ out = io.StringIO()
+ with mock.patch.object(tp, "pick_pair", return_value=pair), \
+ mock.patch.object(cli_mod, "_cached_transcript", return_value={}), \
+ mock.patch.object(thumbnail_ai, "generate_thumbnail_with_template", return_value=args.output) as draw, \
+ redirect_stdout(out):
+ cli_mod.cmd_thumbnail_render(args)
+ result = json.loads(out.getvalue().strip().splitlines()[-1])
+ self.assertEqual(result["layout"], "pair")
+ self.assertEqual([(p["side"], p["source_time"]) for p in result["people"]], [("left", 12.5), ("right", 31.0)])
+ self.assertEqual(draw.call_args.kwargs["people"], pair["people"])
+
+ def test_fallback_without_a_frame_says_why(self):
+ import argparse
+
+ import cli as cli_mod
+
+ args = argparse.Namespace(
+ title="Title", frame=None, output="/tmp/t.png", line1=None, line2=None, frame_info=None, logo=None,
+ payoff=None, context_line=None, preview_text=None, video="v.mp4", start=1.0, end=9.0,
+ layout="pair", left_image=None, right_image=None, swap=False,
+ )
+ single = {"layout": "single", "people": [], "reason": "Only one person is on screen in this clip."}
+ with mock.patch.object(tp, "pick_pair", return_value=single), \
+ mock.patch.object(cli_mod, "_cached_transcript", return_value={}), \
+ mock.patch("builtins.print") as printed, \
+ self.assertRaises(SystemExit):
+ cli_mod.cmd_thumbnail_render(args)
+ self.assertIn("Only one person", printed.call_args.args[0])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_transcribe_runs.py b/tests/test_transcribe_runs.py
new file mode 100644
index 0000000..4286e1c
--- /dev/null
+++ b/tests/test_transcribe_runs.py
@@ -0,0 +1,97 @@
+"""Tests for backend.services.transcribe_runs: per-run receipts keyed by
+{file fingerprint, engine, model, language} so a restarted job can resume."""
+
+import os
+import sys
+import tempfile
+import unittest
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import transcribe_runs as runs
+
+
+class FingerprintFileTests(unittest.TestCase):
+ def test_same_content_same_fingerprint(self):
+ with tempfile.NamedTemporaryFile(delete=False) as a, tempfile.NamedTemporaryFile(delete=False) as b:
+ a.write(b"identical bytes")
+ b.write(b"identical bytes")
+ try:
+ self.assertEqual(runs.fingerprint_file(a.name), runs.fingerprint_file(b.name))
+ finally:
+ os.unlink(a.name)
+ os.unlink(b.name)
+
+ def test_different_content_different_fingerprint(self):
+ with tempfile.NamedTemporaryFile(delete=False) as a, tempfile.NamedTemporaryFile(delete=False) as b:
+ a.write(b"content A")
+ b.write(b"content B")
+ try:
+ self.assertNotEqual(runs.fingerprint_file(a.name), runs.fingerprint_file(b.name))
+ finally:
+ os.unlink(a.name)
+ os.unlink(b.name)
+
+
+class RunKeyTests(unittest.TestCase):
+ def test_different_engine_or_model_or_language_gives_different_key(self):
+ with tempfile.NamedTemporaryFile(delete=False) as f:
+ f.write(b"same file")
+ try:
+ base = runs.run_key(f.name, "assemblyai", "base", "en")
+ self.assertNotEqual(base, runs.run_key(f.name, "omnilingual", "base", "en"))
+ self.assertNotEqual(base, runs.run_key(f.name, "assemblyai", "small", "en"))
+ self.assertNotEqual(base, runs.run_key(f.name, "assemblyai", "base", "ka"))
+ finally:
+ os.unlink(f.name)
+
+ def test_unsafe_characters_are_sanitized(self):
+ with tempfile.NamedTemporaryFile(delete=False) as f:
+ f.write(b"x")
+ try:
+ key = runs.run_key(f.name, "assemblyai/v2", "base model", "en US")
+ self.assertNotIn("/", key)
+ self.assertNotIn(" ", key)
+ finally:
+ os.unlink(f.name)
+
+
+class ReceiptRoundTripTests(unittest.TestCase):
+ def test_write_then_read_round_trips(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ runs.write_receipt(tmp, "r.json", {"transcript_id": "abc123"})
+ self.assertEqual(runs.read_receipt(tmp, "r.json"), {"transcript_id": "abc123"})
+
+ def test_missing_receipt_is_none(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ self.assertIsNone(runs.read_receipt(tmp, "missing.json"))
+
+ def test_corrupt_receipt_is_none_not_a_crash(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ path = os.path.join(tmp, "bad.json")
+ with open(path, "w", encoding="utf-8") as f:
+ f.write("not json")
+ self.assertIsNone(runs.read_receipt(tmp, "bad.json"))
+
+ def test_write_is_atomic_no_tmp_file_left_behind(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ runs.write_receipt(tmp, "r.json", {"a": 1})
+ self.assertEqual(os.listdir(tmp), ["r.json"])
+
+ def test_clear_run_removes_directory(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ run_dir = os.path.join(tmp, "a-run")
+ runs.write_receipt(run_dir, "r.json", {"a": 1})
+ self.assertTrue(os.path.isdir(run_dir))
+ runs.clear_run(run_dir)
+ self.assertFalse(os.path.exists(run_dir))
+
+ def test_clear_run_on_missing_directory_does_not_raise(self):
+ runs.clear_run("/nonexistent/path/for/sure")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_transcript_cache_key.py b/tests/test_transcript_cache_key.py
new file mode 100644
index 0000000..209ccb7
--- /dev/null
+++ b/tests/test_transcript_cache_key.py
@@ -0,0 +1,165 @@
+"""Tests for the Python port of src/services/transcript-cache.ts's keySuffix
+(cache_key_suffix), and the readers/writers built on it: this is what keeps a
+transcript written by one side of the TS/Python boundary visible to the
+other, for any (engine, model, language) combo, not just the base/auto
+default the two sides already agreed on before model/language were tracked.
+"""
+
+import json
+import os
+import shutil
+import sys
+import tempfile
+import unittest
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+from services import transcript_packer as tp
+
+
+class CacheKeySuffixTests(unittest.TestCase):
+ def setUp(self):
+ self._saved = os.environ.get("PODCLI_ENGINE")
+ os.environ.pop("PODCLI_ENGINE", None)
+
+ def tearDown(self):
+ if self._saved is None:
+ os.environ.pop("PODCLI_ENGINE", None)
+ else:
+ os.environ["PODCLI_ENGINE"] = self._saved
+
+ def test_base_model_and_auto_language_contribute_nothing(self):
+ self.assertEqual(tp.cache_key_suffix("whisper-py", "base", "auto"), "")
+ self.assertEqual(tp.cache_key_suffix("whisper-py", None, None), "")
+
+ def test_non_base_model_gets_its_own_suffix(self):
+ self.assertEqual(tp.cache_key_suffix("whisper-py", "small", None), "-msmall")
+
+ def test_non_auto_language_gets_its_own_suffix(self):
+ self.assertEqual(tp.cache_key_suffix("whisper-py", None, "ka"), "-lka")
+
+ def test_engine_model_and_language_combine_in_order(self):
+ self.assertEqual(
+ tp.cache_key_suffix("whispercpp", "small", "ka"), "-whispercpp-msmall-lka"
+ )
+
+ def test_model_is_lowercased(self):
+ self.assertEqual(tp.cache_key_suffix(None, "SMALL", None), "-msmall")
+
+ def test_sanitize_language_strips_everything_outside_a_z0_9_dash(self):
+ self.assertEqual(tp.sanitize_language("../../etc"), "etc")
+ self.assertEqual(tp.sanitize_language("KA"), "ka")
+ self.assertEqual(tp.sanitize_language(None), "")
+
+
+class FindCachedTranscriptPathTests(unittest.TestCase):
+ """The scanning fallback for a reader that only has a video path, not
+ model/language, e.g. manage_reel or _cached_face_map."""
+
+ def setUp(self):
+ self._tmp = tempfile.mkdtemp()
+ self._orig_dir = tp._transcripts_cache_dir
+ tp._transcripts_cache_dir = lambda: self._tmp
+
+ def tearDown(self):
+ tp._transcripts_cache_dir = self._orig_dir
+ shutil.rmtree(self._tmp, ignore_errors=True)
+
+ def _touch(self, name, mtime=None):
+ path = os.path.join(self._tmp, name)
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump({"marker": name}, f)
+ if mtime is not None:
+ os.utime(path, (mtime, mtime))
+ return path
+
+ def test_prefers_the_engine_only_key_when_present(self):
+ self._touch("abc123.json")
+ self._touch("abc123-msmall-lka.json")
+ found = tp.find_cached_transcript_path("abc123", engine=None)
+ self.assertEqual(os.path.basename(found), "abc123.json")
+
+ def test_falls_back_to_the_most_recently_modified_match(self):
+ self._touch("abc123-msmall.json", mtime=1000)
+ self._touch("abc123-mmedium-lka.json", mtime=2000)
+ found = tp.find_cached_transcript_path("abc123", engine=None)
+ self.assertEqual(os.path.basename(found), "abc123-mmedium-lka.json")
+
+ def test_never_crosses_into_a_different_engines_files(self):
+ # Only a whispercpp entry exists; a whisper-py (engine-only) lookup
+ # must not adopt it.
+ self._touch("abc123-whispercpp.json")
+ found = tp.find_cached_transcript_path("abc123", engine=None)
+ self.assertIsNone(found)
+
+ def test_matches_only_this_hash_not_a_hash_with_extra_suffix(self):
+ self._touch("abc123extra-msmall.json")
+ found = tp.find_cached_transcript_path("abc123", engine=None)
+ self.assertIsNone(found)
+
+ def test_no_matches_returns_none(self):
+ self.assertIsNone(tp.find_cached_transcript_path("nope", engine=None))
+
+
+class LoadSaveCachedTranscriptModelLanguageTests(unittest.TestCase):
+ def setUp(self):
+ self._tmp = tempfile.mkdtemp()
+ self._orig_dir = tp._transcripts_cache_dir
+ tp._transcripts_cache_dir = lambda: self._tmp
+ self._video = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False)
+ self._video.write(b"fake video bytes for hashing")
+ self._video.close()
+
+ def tearDown(self):
+ tp._transcripts_cache_dir = self._orig_dir
+ shutil.rmtree(self._tmp, ignore_errors=True)
+ os.unlink(self._video.name)
+
+ def test_a_non_base_model_write_is_invisible_to_a_bare_read(self):
+ # Before this fix, Python's engine-only key meant a whisper-py
+ # "small" model write landed under the same bare filename as "base",
+ # colliding instead of getting its own key.
+ tp.save_cached_transcript_for_video(
+ self._video.name, {"words": ["small"]}, engine="whisper-py", model="small"
+ )
+ tp.save_cached_transcript_for_video(
+ self._video.name, {"words": ["base"]}, engine="whisper-py", model="base"
+ )
+ small = tp.load_cached_transcript_for_video(
+ self._video.name, engine="whisper-py", model="small"
+ )
+ base = tp.load_cached_transcript_for_video(
+ self._video.name, engine="whisper-py", model="base"
+ )
+ self.assertEqual(small["words"], ["small"])
+ self.assertEqual(base["words"], ["base"])
+
+ def test_a_request_for_an_unwritten_combo_misses_rather_than_substituting(self):
+ tp.save_cached_transcript_for_video(
+ self._video.name, {"words": ["small-ka"]}, engine="whisper-py", model="small", language="ka"
+ )
+ self.assertIsNone(
+ tp.load_cached_transcript_for_video(
+ self._video.name, engine="whisper-py", model="medium", language="fr"
+ )
+ )
+
+ def test_unknown_model_language_reader_finds_the_transcript_the_session_used(self):
+ tp.save_cached_transcript_for_video(
+ self._video.name, {"words": ["small-ka"]}, engine="whisper-py", model="small", language="ka"
+ )
+ found = tp.load_cached_transcript_for_video(self._video.name, engine="whisper-py")
+ self.assertEqual(found["words"], ["small-ka"])
+
+ def test_save_writes_atomically_leaving_no_tmp_file_behind(self):
+ path = tp.save_cached_transcript_for_video(self._video.name, {"words": []})
+ self.assertTrue(os.path.exists(path))
+ leftovers = [f for f in os.listdir(self._tmp) if f.endswith(".tmp")]
+ self.assertEqual(leftovers, [])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_transcript_parser.py b/tests/test_transcript_parser.py
index 6c9f387..b99260e 100644
--- a/tests/test_transcript_parser.py
+++ b/tests/test_transcript_parser.py
@@ -135,6 +135,91 @@ class SpeakerTranscriptTests(unittest.TestCase):
# First word start near 3600s
self.assertGreaterEqual(result["words"][0]["start"], 3600.0 - 1.0)
+ def test_language_defaults_to_undetermined_not_english(self):
+ raw = "Alice (00:00)\nHello\n"
+ result = tp.parse_speaker_transcript(raw, total_duration=5.0)
+ self.assertEqual(result["language"], "und")
+
+ def test_language_passthrough(self):
+ raw = "Alice (00:00)\nHello\n"
+ result = tp.parse_speaker_transcript(raw, total_duration=5.0, language="ka")
+ self.assertEqual(result["language"], "ka")
+
+ def test_detect_and_parse_threads_language_to_each_format(self):
+ speaker_raw = "Alice (00:00)\nHello\n"
+ srt_raw = "1\n00:00:00,000 --> 00:00:01,000\nHello\n"
+ vtt_raw = "WEBVTT\n\n00:00:00.000 --> 00:00:01.000\nHello\n"
+ for raw in (speaker_raw, srt_raw, vtt_raw):
+ with self.subTest(raw=raw):
+ result = tp.detect_and_parse(raw, total_duration=5.0, language="ka")
+ self.assertEqual(result["language"], "ka")
+
+ def test_speaker_segments_apply_time_adjust_like_words_and_segments(self):
+ # Regression: speaker_segments previously used the raw block
+ # start/end, ignoring time_adjust, while words and segments applied
+ # it, so the three arrays drifted out of sync for any non-zero adjust.
+ raw = "Alice (00:10)\nHello there\n"
+ result = tp.parse_speaker_transcript(raw, total_duration=30.0, time_adjust=-2.0)
+ seg = result["speaker_segments"][0]
+ self.assertAlmostEqual(seg["start"], result["segments"][0]["start"], places=3)
+ self.assertAlmostEqual(seg["end"], result["segments"][0]["end"], places=3)
+ self.assertAlmostEqual(seg["start"], 8.0, places=3)
+
+
+class CaptionMarkupStrippingTests(unittest.TestCase):
+ """SRT/VTT override tags like {\\an8} aren't spoken text and must not
+ end up imported as literal words in the transcript."""
+
+ def test_parse_srt_strips_ass_override_blocks(self):
+ raw = (
+ "1\n"
+ "00:00:01,000 --> 00:00:04,000\n"
+ "{\\an8}Hello this is text\n"
+ )
+ result = tp.parse_srt(raw)
+ self.assertNotIn("error", result)
+ self.assertEqual(result["segments"][0]["text"], "Hello this is text")
+ self.assertNotIn("an8", result["transcript"])
+ self.assertNotIn("{", result["transcript"])
+
+ def test_parse_srt_strips_html_tags(self):
+ raw = (
+ "1\n"
+ "00:00:01,000 --> 00:00:04,000\n"
+ "<b>Hello</b> <i>world</i>\n"
+ )
+ result = tp.parse_srt(raw)
+ self.assertEqual(result["segments"][0]["text"], "Hello world")
+
+ def test_parse_srt_strips_override_block_in_trailing_block(self):
+ # No trailing blank line, so this exercises the "last block" path.
+ raw = (
+ "1\n"
+ "00:00:01,000 --> 00:00:04,000\n"
+ "{\\an8}No trailing blank line"
+ )
+ result = tp.parse_srt(raw)
+ self.assertEqual(result["segments"][0]["text"], "No trailing blank line")
+
+ def test_parse_vtt_strips_ass_override_blocks(self):
+ raw = (
+ "WEBVTT\n\n"
+ "00:00:01.000 --> 00:00:04.000\n"
+ "{\\an8}Hello this is text\n"
+ )
+ result = tp.parse_vtt(raw)
+ self.assertNotIn("error", result)
+ self.assertEqual(result["segments"][0]["text"], "Hello this is text")
+
+ def test_parse_vtt_strips_override_block_in_trailing_block(self):
+ raw = (
+ "WEBVTT\n\n"
+ "00:00:01.000 --> 00:00:04.000\n"
+ "{\\an8}No trailing blank line"
+ )
+ result = tp.parse_vtt(raw)
+ self.assertEqual(result["segments"][0]["text"], "No trailing blank line")
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_transcription_engine.py b/tests/test_transcription_engine.py
index 1343382..6e470ee 100644
--- a/tests/test_transcription_engine.py
+++ b/tests/test_transcription_engine.py
@@ -1,6 +1,7 @@
"""Engine selection: native installs auto-use whisper.cpp when openai-whisper
is absent, unless the user explicitly asked for the whisper-py engine."""
+import importlib.util
import os
import sys
import tempfile
@@ -18,9 +19,11 @@ class TranscriptionEngineTests(unittest.TestCase):
def setUp(self):
self._orig_wcpp = tr._transcribe_with_whispercpp
self._orig_aai = tr._transcribe_with_assemblyai
+ self._orig_omni = tr._transcribe_with_omnilingual
self._orig_ready = tr._whispercpp_ready
tr._transcribe_with_whispercpp = lambda *a, **k: {"engine": "whispercpp"}
tr._transcribe_with_assemblyai = lambda *a, **k: {"engine": "assemblyai"}
+ tr._transcribe_with_omnilingual = lambda *a, **k: {"engine": "omnilingual"}
tr._whispercpp_ready = lambda size: True
# Make `import whisper` fail to simulate a native (hermetic) install.
self._had_whisper = sys.modules.get("whisper", "__absent__")
@@ -33,6 +36,7 @@ class TranscriptionEngineTests(unittest.TestCase):
def tearDown(self):
tr._transcribe_with_whispercpp = self._orig_wcpp
tr._transcribe_with_assemblyai = self._orig_aai
+ tr._transcribe_with_omnilingual = self._orig_omni
tr._whispercpp_ready = self._orig_ready
if self._had_whisper == "__absent__":
sys.modules.pop("whisper", None)
@@ -59,6 +63,24 @@ class TranscriptionEngineTests(unittest.TestCase):
result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
self.assertEqual(result["engine"], "assemblyai")
+ def test_explicit_omnilingual_uses_it(self):
+ os.environ["PODCLI_ENGINE"] = "omnilingual"
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
+ self.assertEqual(result["engine"], "omnilingual")
+
+ def test_omnilingual_skips_diarization_like_whispercpp(self):
+ os.environ["PODCLI_ENGINE"] = "omnilingual"
+ result = tr.transcribe_file(self._tmp.name, model_size="base")
+ self.assertEqual(result["engine"], "omnilingual")
+ self.assertEqual(result.get("diarization_warning"), "Speaker detection disabled")
+
+ def test_unset_engine_never_auto_falls_back_to_omnilingual(self):
+ # Only whispercpp is an automatic fallback for an unset engine on a
+ # native install; omnilingual is explicit-only.
+ os.environ.pop("PODCLI_ENGINE", None)
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
+ self.assertEqual(result["engine"], "whispercpp")
+
def test_whispercpp_skips_diarization_by_default(self):
# Regression: the cpp path must not attempt torch-backed diarization even
# with the default enable_diarization=True — importing a broken torch in a
@@ -79,6 +101,348 @@ class TranscriptionEngineTests(unittest.TestCase):
with self.assertRaises(RuntimeError):
tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
+ def test_whispercpp_never_marks_diarization_attempted(self):
+ # whisper.cpp can never diarize. A cache entry from this engine must
+ # say so explicitly, or a reader asking for speaker labels would
+ # re-transcribe it forever (every run looks like a retriable miss).
+ os.environ["PODCLI_ENGINE"] = "whispercpp"
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=True)
+ self.assertFalse(result["diarization_attempted"])
+
+ def test_omnilingual_never_marks_diarization_attempted(self):
+ os.environ["PODCLI_ENGINE"] = "omnilingual"
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=True)
+ self.assertFalse(result["diarization_attempted"])
+
+ def test_assemblyai_marks_diarization_attempted_from_its_own_request_flag(self):
+ os.environ["PODCLI_ENGINE"] = "assemblyai"
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=True)
+ self.assertTrue(result["diarization_attempted"])
+
+ def test_assemblyai_without_diarization_is_not_marked_attempted(self):
+ os.environ["PODCLI_ENGINE"] = "assemblyai"
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
+ self.assertFalse(result["diarization_attempted"])
+
+
+class SampleModeTests(unittest.TestCase):
+ """start_seconds/duration_seconds should transcribe a trimmed window
+ instead of the full file, mark the result incomplete, and skip
+ diarization/face analysis (the trimmed clip has no video track for face
+ analysis to read, and a throwaway sample isn't worth either pass)."""
+
+ def setUp(self):
+ self._orig_wcpp = tr._transcribe_with_whispercpp
+ self._orig_ready = tr._whispercpp_ready
+ self._saved_engine = os.environ.pop("PODCLI_ENGINE", None)
+ os.environ["PODCLI_ENGINE"] = "whispercpp"
+ self._tmp = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False)
+ self._tmp.write(b"not a real video")
+ self._tmp.close()
+ self.extract_calls = []
+
+ def fake_extract(file_path, wav_path=None, timeout=1800, start_seconds=None, duration_seconds=None):
+ self.extract_calls.append(
+ {"file_path": file_path, "start_seconds": start_seconds, "duration_seconds": duration_seconds}
+ )
+ fd, path = tempfile.mkstemp(suffix=".wav")
+ os.close(fd)
+ return path
+
+ self._fake_extract = fake_extract
+ import services.audio_extract as audio_extract
+
+ self._orig_extract = audio_extract.extract_wav_16k_mono
+ audio_extract.extract_wav_16k_mono = fake_extract
+
+ def fake_wcpp(*args, **kwargs):
+ # The inner call must receive the sample wav as both file_path
+ # and wav_path, and diarization must be forced off.
+ self.inner_file_path = args[0] if args else kwargs.get("file_path")
+ self.inner_wav_path = kwargs.get("wav_path")
+ return {"engine": "whispercpp", "words": [], "segments": [], "duration": 40.0}
+
+ tr._transcribe_with_whispercpp = fake_wcpp
+ tr._whispercpp_ready = lambda size: True
+
+ def tearDown(self):
+ tr._transcribe_with_whispercpp = self._orig_wcpp
+ tr._whispercpp_ready = self._orig_ready
+ import services.audio_extract as audio_extract
+
+ audio_extract.extract_wav_16k_mono = self._orig_extract
+ os.unlink(self._tmp.name)
+ if self._saved_engine is None:
+ os.environ.pop("PODCLI_ENGINE", None)
+ else:
+ os.environ["PODCLI_ENGINE"] = self._saved_engine
+
+ def test_sample_window_is_passed_to_extraction(self):
+ tr.transcribe_file(
+ self._tmp.name, model_size="base", start_seconds=120.0, duration_seconds=40.0,
+ enable_diarization=False,
+ )
+ self.assertEqual(len(self.extract_calls), 1)
+ self.assertEqual(self.extract_calls[0]["start_seconds"], 120.0)
+ self.assertEqual(self.extract_calls[0]["duration_seconds"], 40.0)
+
+ def test_result_marked_incomplete_with_offset(self):
+ result = tr.transcribe_file(
+ self._tmp.name, model_size="base", start_seconds=120.0, duration_seconds=40.0,
+ enable_diarization=False,
+ )
+ self.assertFalse(result["complete"])
+ self.assertEqual(result["sample_offset_seconds"], 120.0)
+
+ def test_full_run_is_marked_neither_incomplete_nor_offset(self):
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
+ self.assertNotIn("complete", result)
+ self.assertNotIn("sample_offset_seconds", result)
+
+ def test_default_start_is_zero_when_only_duration_given(self):
+ result = tr.transcribe_file(
+ self._tmp.name, model_size="base", duration_seconds=40.0, enable_diarization=False,
+ )
+ self.assertEqual(self.extract_calls[0]["start_seconds"], 0.0)
+ self.assertEqual(result["sample_offset_seconds"], 0.0)
+
+ def test_duration_seconds_zero_is_a_full_run_not_a_sample(self):
+ # Matches src/handlers/transcribe.handler.ts and web-server.ts: a
+ # sample is a *positive* window, not merely a present key.
+ result = tr.transcribe_file(
+ self._tmp.name, model_size="base", duration_seconds=0.0, enable_diarization=False,
+ )
+ self.assertEqual(self.extract_calls, [])
+ self.assertNotIn("complete", result)
+ self.assertNotIn("sample_offset_seconds", result)
+
+ def test_start_seconds_zero_alone_is_a_full_run_not_a_sample(self):
+ result = tr.transcribe_file(
+ self._tmp.name, model_size="base", start_seconds=0.0, enable_diarization=False,
+ )
+ self.assertEqual(self.extract_calls, [])
+ self.assertNotIn("complete", result)
+
+
+class ResolveEngineInfoTests(unittest.TestCase):
+ """resolve_engine_info predicts transcribe_file's engine choice so a
+ caller can build a matching cache key before deciding to transcribe."""
+
+ def setUp(self):
+ self._had_whisper = sys.modules.get("whisper", "__absent__")
+ self._orig_ready = tr._whispercpp_ready
+ self._saved_engine = os.environ.pop("PODCLI_ENGINE", None)
+ # _whisper_py_available is memoized for the process lifetime (the
+ # package's installed-ness can't change mid-run), but that means
+ # a value cached by one test would leak into the next, which
+ # sabotages sys.modules["whisper"] to simulate availability.
+ tr._whisper_py_available.cache_clear()
+
+ def tearDown(self):
+ if self._had_whisper == "__absent__":
+ sys.modules.pop("whisper", None)
+ else:
+ sys.modules["whisper"] = self._had_whisper
+ tr._whispercpp_ready = self._orig_ready
+ tr._whisper_py_available.cache_clear()
+ if self._saved_engine is None:
+ os.environ.pop("PODCLI_ENGINE", None)
+ else:
+ os.environ["PODCLI_ENGINE"] = self._saved_engine
+
+ def test_explicit_whispercpp_resolves_as_is(self):
+ self.assertEqual(tr.resolve_engine_info("whispercpp")["engine"], "whispercpp")
+
+ def test_explicit_assemblyai_resolves_as_is(self):
+ self.assertEqual(tr.resolve_engine_info("assemblyai")["engine"], "assemblyai")
+
+ def test_unset_falls_back_to_whispercpp_on_native_install(self):
+ sys.modules["whisper"] = None # simulate "import whisper" failing
+ tr._whispercpp_ready = lambda size: True
+ self.assertEqual(tr.resolve_engine_info(None)["engine"], "whispercpp")
+
+ def test_unset_stays_whisper_py_when_whisper_importable(self):
+ sys.modules.pop("whisper", None)
+ try:
+ import whisper # noqa: F401
+ except Exception:
+ self.skipTest("openai-whisper not installed in this environment")
+ self.assertEqual(tr.resolve_engine_info(None)["engine"], "whisper-py")
+
+ def test_explicit_whisper_py_request_never_falls_back(self):
+ # Fallback only applies to an unset request; an explicit whisper-py
+ # ask should resolve as whisper-py even on a native install, matching
+ # transcribe_file which raises instead of silently substituting.
+ sys.modules["whisper"] = None
+ tr._whispercpp_ready = lambda size: True
+ self.assertEqual(tr.resolve_engine_info("whisper-py")["engine"], "whisper-py")
+
+
+class OmnilingualReceiptKeyTests(unittest.TestCase):
+ """The resumable run directory has to change whenever a resumed
+ window's receipt would no longer be valid: a different model file, or
+ a change to the window/context constants its timestamps assume."""
+
+ def setUp(self):
+ self._model = tempfile.NamedTemporaryFile(suffix=".onnx", delete=False)
+ self._model.write(b"fake model bytes")
+ self._model.close()
+ self._tokens = tempfile.NamedTemporaryFile(suffix=".txt", delete=False)
+ self._tokens.write(b"fake tokens")
+ self._tokens.close()
+ self._media = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False)
+ self._media.write(b"fake media")
+ self._media.close()
+
+ self._orig_model = tr._omnilingual_model
+ self._orig_tokens = tr._omnilingual_tokens
+ tr._omnilingual_model = lambda: self._model.name
+ tr._omnilingual_tokens = lambda: self._tokens.name
+
+ self.captured = []
+ import services.transcription_omnilingual as omni
+
+ self._orig_omni_transcribe = omni.transcribe_file
+
+ def fake_transcribe(*a, run_dir=None, **k):
+ self.captured.append(run_dir)
+ return {"transcript": "", "segments": [], "words": [], "duration": 0.0, "language": "und"}
+
+ omni.transcribe_file = fake_transcribe
+ self._omni = omni
+
+ def tearDown(self):
+ tr._omnilingual_model = self._orig_model
+ tr._omnilingual_tokens = self._orig_tokens
+ self._omni.transcribe_file = self._orig_omni_transcribe
+ for f in (self._model, self._tokens, self._media):
+ os.unlink(f.name)
+
+ def test_a_different_model_file_gets_its_own_run_dir(self):
+ tr._transcribe_with_omnilingual(self._media.name, progress_callback=None)
+ first = self.captured[0]
+
+ # Swap in a "different" model: same path, new content/mtime.
+ import time as _time
+ _time.sleep(0.01)
+ with open(self._model.name, "wb") as f:
+ f.write(b"a completely different and larger fake model")
+
+ tr._transcribe_with_omnilingual(self._media.name, progress_callback=None)
+ second = self.captured[1]
+
+ self.assertNotEqual(first, second)
+
+ def test_the_window_and_context_constants_are_baked_into_the_key(self):
+ tr._transcribe_with_omnilingual(self._media.name, progress_callback=None)
+ run_dir = self.captured[0]
+ self.assertIn(f"w{self._omni.WINDOW_SECONDS}", run_dir)
+ self.assertIn(f"c{self._omni.CONTEXT_SECONDS}", run_dir)
+
+
+class WhisperPyFallbackSharedPathTests(unittest.TestCase):
+ """resolve_engine_info (a cheap cache-key prediction) and transcribe_file
+ (about to actually transcribe) make the same fallback decision through
+ one shared function, _resolve_whisper_py_fallback, differing only in
+ whether they load the model weights to check it."""
+
+ def setUp(self):
+ self._orig_wcpp = tr._transcribe_with_whispercpp
+ self._orig_ready = tr._whispercpp_ready
+ tr._transcribe_with_whispercpp = lambda *a, **k: {"engine": "whispercpp"}
+ tr._whispercpp_ready = lambda size: True
+ self._saved_engine = os.environ.pop("PODCLI_ENGINE", None)
+ self._tmp = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
+ self._tmp.write(b"x")
+ self._tmp.close()
+
+ # Simulate "import whisper" succeeding but whisper.load_model()
+ # failing: the one gap resolve_engine_info's cheap check can't see.
+ import importlib.machinery
+
+ class _BrokenWhisperModule:
+ # find_spec requires a real module to carry __spec__. A bare
+ # object stand-in raises ValueError instead of just finding it.
+ __spec__ = importlib.machinery.ModuleSpec("whisper", loader=None)
+
+ @staticmethod
+ def load_model(size):
+ raise RuntimeError("corrupt model weights")
+
+ self._had_whisper = sys.modules.get("whisper", "__absent__")
+ sys.modules["whisper"] = _BrokenWhisperModule()
+ tr._whisper_py_available.cache_clear()
+
+ def tearDown(self):
+ tr._transcribe_with_whispercpp = self._orig_wcpp
+ tr._whispercpp_ready = self._orig_ready
+ if self._had_whisper == "__absent__":
+ sys.modules.pop("whisper", None)
+ else:
+ sys.modules["whisper"] = self._had_whisper
+ tr._whisper_py_available.cache_clear()
+ os.unlink(self._tmp.name)
+ if self._saved_engine is None:
+ os.environ.pop("PODCLI_ENGINE", None)
+ else:
+ os.environ["PODCLI_ENGINE"] = self._saved_engine
+
+ def test_transcribe_file_falls_back_to_whispercpp_on_a_load_model_failure(self):
+ result = tr.transcribe_file(self._tmp.name, model_size="base", enable_diarization=False)
+ self.assertEqual(result["engine"], "whispercpp")
+
+ def test_resolve_engine_info_cannot_see_this_failure_by_design(self):
+ # Documents the known, acceptable gap: resolve_engine_info only
+ # checks the import, so it still predicts whisper-py here even
+ # though transcribe_file will fall back, a one-time cache miss,
+ # not a wrong cache write (that's keyed by what actually ran).
+ self.assertEqual(tr.resolve_engine_info(None, "base")["engine"], "whisper-py")
+
+
+class WhisperPyAvailableMemoizationTests(unittest.TestCase):
+ """resolve_engine_info runs on every transcribe request. The
+ availability check behind it has to find_spec (not import, which would
+ pull in torch) and compute that only once per process."""
+
+ def setUp(self):
+ tr._whisper_py_available.cache_clear()
+ self._orig_find_spec = importlib.util.find_spec
+ self.calls = 0
+
+ def counting_find_spec(name, *a, **k):
+ self.calls += 1
+ return self._orig_find_spec(name, *a, **k)
+
+ importlib.util.find_spec = counting_find_spec
+
+ def tearDown(self):
+ importlib.util.find_spec = self._orig_find_spec
+ tr._whisper_py_available.cache_clear()
+
+ def test_find_spec_runs_at_most_once_across_repeated_calls(self):
+ for _ in range(5):
+ tr._whisper_py_available()
+ self.assertEqual(self.calls, 1)
+
+
+class WhisperCppModelAliasTests(unittest.TestCase):
+ """"large" alone doesn't name a real ggml file upstream (v1/v2/v3/v3-turbo
+ are separate downloads); provisioning always fetches large-v3, so the
+ model path lookup must resolve the same alias."""
+
+ def setUp(self):
+ self._saved = os.environ.pop("PODCLI_WHISPERCPP_MODEL", None)
+
+ def tearDown(self):
+ if self._saved is not None:
+ os.environ["PODCLI_WHISPERCPP_MODEL"] = self._saved
+
+ def test_large_resolves_to_large_v3(self):
+ self.assertTrue(tr._whispercpp_model("large").endswith("ggml-large-v3.bin"))
+
+ def test_medium_is_unaliased(self):
+ self.assertTrue(tr._whispercpp_model("medium").endswith("ggml-medium.bin"))
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_transcription_omnilingual.py b/tests/test_transcription_omnilingual.py
new file mode 100644
index 0000000..496cea7
--- /dev/null
+++ b/tests/test_transcription_omnilingual.py
@@ -0,0 +1,328 @@
+"""Tests for backend.services.transcription_omnilingual: the sherpa-onnx
+Omnilingual ASR CTC adapter. Mocks the recognizer; never loads the real
+model (large download) or decodes real audio."""
+
+import math
+import os
+import sys
+import unittest
+from unittest import mock
+
+ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+BACKEND_ROOT = os.path.join(ROOT, "backend")
+if BACKEND_ROOT not in sys.path:
+ sys.path.insert(0, BACKEND_ROOT)
+
+import services.transcription_omnilingual as omni
+
+
+class TokensToWordsTests(unittest.TestCase):
+ def test_splits_on_boundary_space_token(self):
+ tokens = ["h", "i", " ", "t", "h", "e", "r", "e"]
+ timestamps = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7]
+ words = omni._tokens_to_words(tokens, timestamps, decode_end=10.0)
+ self.assertEqual([w["word"] for w in words], ["hi", "there"])
+ self.assertEqual(words[0]["start"], 0.0)
+ self.assertAlmostEqual(words[0]["end"], 0.1 + omni.WORD_END_PAD_SECONDS)
+ self.assertEqual(words[1]["start"], 0.3)
+
+ def test_leading_and_trailing_boundary_tokens_produce_no_empty_words(self):
+ tokens = [" ", "h", "i", " "]
+ timestamps = [0.0, 0.1, 0.2, 0.3]
+ words = omni._tokens_to_words(tokens, timestamps, decode_end=10.0)
+ self.assertEqual([w["word"] for w in words], ["hi"])
+
+ def test_word_end_capped_at_decode_end(self):
+ tokens = ["x"]
+ timestamps = [9.995]
+ words = omni._tokens_to_words(tokens, timestamps, decode_end=10.0)
+ self.assertLessEqual(words[0]["end"], 10.0)
+
+ def test_single_token_word_has_positive_duration(self):
+ tokens = ["a"]
+ timestamps = [1.0]
+ words = omni._tokens_to_words(tokens, timestamps, decode_end=10.0)
+ self.assertGreater(words[0]["end"], words[0]["start"])
+
+
+class GroupIntoSegmentsTests(unittest.TestCase):
+ def test_splits_on_pause(self):
+ words = [
+ {"word": "hello", "start": 0.0, "end": 0.5},
+ {"word": "world", "start": 0.6, "end": 1.0},
+ {"word": "later", "start": 2.0, "end": 2.5}, # 1.0s gap >= 0.5
+ ]
+ segments = omni._group_into_segments(words)
+ self.assertEqual(len(segments), 2)
+ self.assertEqual(segments[0]["text"], "hello world")
+ self.assertEqual(segments[1]["text"], "later")
+
+ def test_no_words_produces_no_segments(self):
+ self.assertEqual(omni._group_into_segments([]), [])
+
+
+class ValidateTokensAndTimestampsTests(unittest.TestCase):
+ def test_mismatched_lengths_raise(self):
+ with self.assertRaises(ValueError):
+ omni._validate_tokens_and_timestamps(["a", "b"], [0.0], "w")
+
+ def test_nan_timestamp_raises(self):
+ with self.assertRaises(ValueError):
+ omni._validate_tokens_and_timestamps(["a"], [math.nan], "w")
+
+ def test_negative_timestamp_raises(self):
+ with self.assertRaises(ValueError):
+ omni._validate_tokens_and_timestamps(["a"], [-0.1], "w")
+
+ def test_out_of_order_timestamps_raise(self):
+ with self.assertRaises(ValueError):
+ omni._validate_tokens_and_timestamps(["a", "b"], [1.0, 0.5], "w")
+
+ def test_valid_input_does_not_raise(self):
+ omni._validate_tokens_and_timestamps(["a", "b"], [0.0, 0.1], "w")
+
+
+class FakeStream:
+ def __init__(self, tokens, timestamps):
+ self._tokens = tokens
+ self._timestamps = timestamps
+ self.result = mock.Mock(tokens=tokens, timestamps=timestamps)
+
+ def accept_waveform(self, sr, chunk):
+ pass
+
+
+class FakeRecognizer:
+ """Returns a fixed token stream per call, regardless of audio content.
+ The tests only check the windowing/assignment logic in transcribe_file,
+ not real decoding."""
+
+ def __init__(self, per_call_tokens):
+ self._calls = iter(per_call_tokens)
+
+ def create_stream(self):
+ tokens, timestamps = next(self._calls)
+ return FakeStream(tokens, timestamps)
+
+ def decode_stream(self, stream):
+ pass
+
+
+class TranscribeFileWindowingTests(unittest.TestCase):
+ def setUp(self):
+ self._orig_read_wav = omni._read_wav_mono16
+ self._orig_load = omni._load_recognizer
+ self._tmp_files = []
+
+ def tearDown(self):
+ import shutil
+
+ omni._read_wav_mono16 = self._orig_read_wav
+ omni._load_recognizer = self._orig_load
+ for p in self._tmp_files:
+ if os.path.isdir(p):
+ shutil.rmtree(p, ignore_errors=True)
+ elif os.path.exists(p):
+ os.unlink(p)
+
+ def _touch(self, name):
+ import tempfile
+
+ fd, path = tempfile.mkstemp(suffix=name)
+ os.close(fd)
+ self._tmp_files.append(path)
+ return path
+
+ def test_single_window_file_produces_expected_words(self):
+ media = self._touch(".mp4")
+ model = self._touch(".onnx")
+ tokens_file = self._touch(".txt")
+ wav = self._touch(".wav")
+
+ import numpy as np
+
+ omni._read_wav_mono16 = lambda path: (np.zeros(16000 * 5, dtype=np.float32), 16000, 5.0)
+ fake = FakeRecognizer([(["h", "i"], [0.0, 0.1])])
+ omni._load_recognizer = lambda *a, **k: fake
+
+ result = omni.transcribe_file(media, model, tokens_file, wav_path=wav)
+ self.assertEqual(result["words"][0]["word"], "hi")
+ self.assertEqual(result["duration"], 5.0)
+ self.assertEqual(result["language"], "und")
+
+ def test_context_overlap_is_not_duplicated_across_windows(self):
+ # A 25s file splits into two core windows (0-20, 20-25). Each decode
+ # call includes 1s of context from the neighbor. A word whose
+ # midpoint sits just past 20s must be claimed by window 2 only.
+ media = self._touch(".mp4")
+ model = self._touch(".onnx")
+ tokens_file = self._touch(".txt")
+ wav = self._touch(".wav")
+
+ import numpy as np
+
+ duration = 25.0
+ omni._read_wav_mono16 = lambda path: (
+ np.zeros(int(16000 * duration), dtype=np.float32), 16000, duration,
+ )
+ # Window 0 decodes [0, 21): sees a word at absolute t=20.5 (in its
+ # context tail). Window 1 decodes [19, 25): sees the same word.
+ fake = FakeRecognizer([
+ (["a", "b"], [20.5 - 0.0, 20.5 - 0.0 + 0.01]), # decode_start=0 -> absolute 20.5
+ (["a", "b"], [20.5 - 19.0, 20.5 - 19.0 + 0.01]), # decode_start=19 -> absolute 20.5
+ ])
+ omni._load_recognizer = lambda *a, **k: fake
+
+ result = omni.transcribe_file(media, model, tokens_file, wav_path=wav)
+ matching = [w for w in result["words"] if w["word"] == "ab"]
+ self.assertEqual(len(matching), 1)
+
+ def test_shifted_timestamps_for_the_same_word_at_a_seam_are_deduped(self):
+ # Same scenario as the exact-timestamp case above, but the two
+ # neighboring windows' own (overlapping) decodes timestamp the same
+ # word a little differently, as a real acoustic model can. Each
+ # window's own midpoint test alone would let both keep it.
+ media = self._touch(".mp4")
+ model = self._touch(".onnx")
+ tokens_file = self._touch(".txt")
+ wav = self._touch(".wav")
+
+ import numpy as np
+
+ duration = 25.0
+ omni._read_wav_mono16 = lambda path: (
+ np.zeros(int(16000 * duration), dtype=np.float32), 16000, duration,
+ )
+ # Window 0 decodes [0, 21): "ab" lands at absolute 19.6-19.9,
+ # inside its own core [0, 20).
+ # Window 1 decodes [19, 25): the same "ab" lands at absolute
+ # 20.05-20.3 instead, shifted ~0.4s by the different decode
+ # context, inside its own core [20, 25).
+ fake = FakeRecognizer([
+ (["a", "b"], [19.6, 19.9]), # decode_start=0 -> absolute 19.6, 19.9
+ (["a", "b"], [1.05, 1.3]), # decode_start=19 -> absolute 20.05, 20.3
+ ])
+ omni._load_recognizer = lambda *a, **k: fake
+
+ result = omni.transcribe_file(media, model, tokens_file, wav_path=wav)
+ matching = [w for w in result["words"] if w["word"] == "ab"]
+ self.assertEqual(len(matching), 1)
+ # The surviving copy is window 0's: the earlier window wins ties,
+ # since it's the one whose receipt (if any) was already written.
+ self.assertAlmostEqual(matching[0]["start"], 19.6, places=3)
+
+ def test_a_legitimate_repeated_word_away_from_any_seam_is_not_deduped(self):
+ media = self._touch(".mp4")
+ model = self._touch(".onnx")
+ tokens_file = self._touch(".txt")
+ wav = self._touch(".wav")
+
+ import numpy as np
+
+ duration = 25.0
+ omni._read_wav_mono16 = lambda path: (
+ np.zeros(int(16000 * duration), dtype=np.float32), 16000, duration,
+ )
+ # Both repeats of "no" sit well inside window 0's core, nowhere
+ # near the 20s boundary, a real stutter/repeat, not a seam echo.
+ fake = FakeRecognizer([
+ (["n", "o", " ", "n", "o"], [2.0, 2.1, 2.2, 2.3, 2.4]),
+ ([], []),
+ ])
+ omni._load_recognizer = lambda *a, **k: fake
+
+ result = omni.transcribe_file(media, model, tokens_file, wav_path=wav)
+ matching = [w for w in result["words"] if w["word"] == "no"]
+ self.assertEqual(len(matching), 2)
+
+ def test_resumes_from_receipts_without_touching_the_recognizer(self):
+ # A rerun with the same run_dir must skip every window that already
+ # has a receipt. Simulated here by pre-seeding window-0's receipt
+ # and giving the fake recognizer only enough calls for window 1.
+ media = self._touch(".mp4")
+ model = self._touch(".onnx")
+ tokens_file = self._touch(".txt")
+ wav = self._touch(".wav")
+
+ import tempfile
+ import numpy as np
+
+ from services import transcribe_runs
+
+ run_dir = tempfile.mkdtemp()
+ self._tmp_files.append(run_dir)
+ transcribe_runs.write_receipt(
+ run_dir, "window-0.json",
+ {"words": [{"word": "resumed", "start": 1.0, "end": 1.3}]},
+ )
+
+ duration = 25.0
+ omni._read_wav_mono16 = lambda path: (
+ np.zeros(int(16000 * duration), dtype=np.float32), 16000, duration,
+ )
+ # Only one call available: if the resumed window tried to decode
+ # again, create_stream() would raise StopIteration from the fake.
+ # Window 1 decodes [19, 25) (1s context back into window 0's core);
+ # these relative timestamps land at absolute 21.0-21.2, inside
+ # window 1's own core [20, 25), so it's the one that claims them.
+ fake = FakeRecognizer([(["n", "e", "w"], [2.0, 2.1, 2.2])])
+ omni._load_recognizer = lambda *a, **k: fake
+
+ result = omni.transcribe_file(media, model, tokens_file, wav_path=wav, run_dir=run_dir)
+
+ self.assertIn({"word": "resumed", "start": 1.0, "end": 1.3, "speaker": None}, result["words"])
+ self.assertTrue(any(w["word"] == "new" for w in result["words"]))
+ # The run completed fully, so its receipts are cleared.
+ self.assertFalse(os.path.exists(run_dir))
+
+ def test_a_crash_mid_file_leaves_completed_windows_resumable(self):
+ media = self._touch(".mp4")
+ model = self._touch(".onnx")
+ tokens_file = self._touch(".txt")
+ wav = self._touch(".wav")
+
+ import tempfile
+ import numpy as np
+
+ run_dir = tempfile.mkdtemp()
+ self._tmp_files.append(run_dir)
+
+ duration = 25.0
+ omni._read_wav_mono16 = lambda path: (
+ np.zeros(int(16000 * duration), dtype=np.float32), 16000, duration,
+ )
+
+ class CrashingRecognizer:
+ def __init__(self):
+ self.calls = 0
+
+ def create_stream(self):
+ self.calls += 1
+ if self.calls == 2:
+ raise RuntimeError("simulated crash on window 1")
+ return FakeStream(["a"], [0.0])
+
+ def decode_stream(self, stream):
+ pass
+
+ omni._load_recognizer = lambda *a, **k: CrashingRecognizer()
+
+ with self.assertRaises(RuntimeError):
+ omni.transcribe_file(media, model, tokens_file, wav_path=wav, run_dir=run_dir)
+
+ # Window 0's receipt survives the crash in window 1.
+ self.assertTrue(os.path.exists(os.path.join(run_dir, "window-0.json")))
+ self.assertFalse(os.path.exists(os.path.join(run_dir, "window-1.json")))
+
+ def test_raises_a_clear_error_when_method_missing_on_old_sherpa_onnx(self):
+ class _NoOmnilingualRecognizer:
+ pass
+
+ fake_sherpa = mock.Mock(OfflineRecognizer=_NoOmnilingualRecognizer)
+ with mock.patch.dict(sys.modules, {"sherpa_onnx": fake_sherpa}):
+ with self.assertRaises(RuntimeError):
+ omni._load_recognizer("/fake/model", "/fake/tokens", threads=1)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_video_cut.py b/tests/test_video_cut.py
index d9ea007..64ac575 100644
--- a/tests/test_video_cut.py
+++ b/tests/test_video_cut.py
@@ -51,12 +51,15 @@ class CutSegmentTests(unittest.TestCase):
class CutMultiSegmentTests(unittest.TestCase):
def test_single_segment_delegates_to_cut_segment(self):
- with mock.patch.object(video_cut, "cut_segment", return_value="/out.mp4") as cs:
- out = video_cut.cut_multi_segment(
+ with mock.patch.object(video_cut, "cut_segment", return_value="/out.mp4") as cs, \
+ mock.patch.object(video_cut, "probe_duration", return_value=5.0) as pd:
+ out, durations = video_cut.cut_multi_segment(
"/in.mp4", "/out.mp4", [{"start": 0, "end": 5}]
)
self.assertEqual(out, "/out.mp4")
+ self.assertEqual(durations, [5.0])
cs.assert_called_once_with("/in.mp4", "/out.mp4", 0, 5)
+ pd.assert_called_once_with("/out.mp4")
def test_multi_segment_invokes_cut_then_concat(self):
# Two-segment flow: two cut_segment calls + one proc_run for concat
@@ -75,7 +78,7 @@ class CutMultiSegmentTests(unittest.TestCase):
) as cs, mock.patch.object(
video_cut, "proc_run", return_value=_ok()
) as mocked:
- out = video_cut.cut_multi_segment(
+ out, durations = video_cut.cut_multi_segment(
"/in.mp4",
out_path,
[
@@ -84,6 +87,7 @@ class CutMultiSegmentTests(unittest.TestCase):
],
)
self.assertEqual(out, out_path)
+ self.assertEqual(len(durations), 2)
self.assertEqual(cs.call_count, 2)
# concat call
cmd = mocked.call_args.args[0]
@@ -213,5 +217,153 @@ class CutContentAccuracyTests(unittest.TestCase):
self._assert_frame_is(out, "cyan")
+ shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg/ffprobe not installed"
+)
+class OutputVerificationTests(unittest.TestCase):
+ """probe_has_audio_stream and verify_full_decode are the post-render
+ checks that catch a 0-exit ffmpeg run that still wrote a bad file."""
+
+ @classmethod
+ def setUpClass(cls):
+ cls.tmpdir = tempfile.mkdtemp(prefix="podcli-verify-test-")
+
+ cls.with_audio = os.path.join(cls.tmpdir, "with_audio.mp4")
+ subprocess.run(
+ [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-f", "lavfi", "-i", "color=c=red:s=64x64:r=25:d=1",
+ "-f", "lavfi", "-i", "sine=frequency=440:duration=1",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
+ "-shortest", cls.with_audio,
+ ],
+ check=True, capture_output=True,
+ )
+
+ cls.no_audio = os.path.join(cls.tmpdir, "no_audio.mp4")
+ subprocess.run(
+ [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-f", "lavfi", "-i", "color=c=blue:s=64x64:r=25:d=1",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p",
+ cls.no_audio,
+ ],
+ check=True, capture_output=True,
+ )
+
+ # A truncated file: valid header, but chopped mid-stream so a full
+ # decode (not just ffprobe's duration read) turns up an error.
+ cls.truncated = os.path.join(cls.tmpdir, "truncated.mp4")
+ with open(cls.with_audio, "rb") as f:
+ data = f.read()
+ with open(cls.truncated, "wb") as f:
+ f.write(data[: len(data) // 2])
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.tmpdir, ignore_errors=True)
+
+ def test_probe_has_audio_stream_true_when_present(self):
+ self.assertTrue(video_cut.probe_has_audio_stream(self.with_audio))
+
+ def test_probe_has_audio_stream_false_when_missing(self):
+ self.assertFalse(video_cut.probe_has_audio_stream(self.no_audio))
+
+ def test_probe_has_audio_stream_false_for_missing_file(self):
+ self.assertFalse(video_cut.probe_has_audio_stream("/no/such/file.mp4"))
+
+ def test_verify_full_decode_clean_file_returns_none(self):
+ self.assertIsNone(video_cut.verify_full_decode(self.with_audio))
+
+ def test_verify_full_decode_flags_truncated_file(self):
+ self.assertIsNotNone(video_cut.verify_full_decode(self.truncated))
+
+
+class VerifyFullDecodeThresholdTests(unittest.TestCase):
+ """verify_full_decode used to delete the output on any `-v error` line at
+ all, even with a 0 exit code. A single benign warning (e.g. a
+ non-monotonic DTS line from a concat/re-encode) was enough to flag a
+ clip that decoded and played fine. These exercise the counting logic
+ against a mocked proc_run so they don't depend on finding a real file
+ that reliably produces exactly one or many such lines."""
+
+ @mock.patch.object(video_cut, "proc_run")
+ def test_tolerates_a_handful_of_error_lines_on_a_clean_exit(self, mock_run):
+ mock_run.return_value = mock.Mock(returncode=0, stdout="", stderr="one benign line\n")
+ self.assertIsNone(video_cut.verify_full_decode("clip.mp4"))
+
+ @mock.patch.object(video_cut, "proc_run")
+ def test_flags_many_error_lines_even_on_a_clean_exit(self, mock_run):
+ stderr = "\n".join(f"decode error {i}" for i in range(10))
+ mock_run.return_value = mock.Mock(returncode=0, stdout="", stderr=stderr)
+ self.assertIsNotNone(video_cut.verify_full_decode("clip.mp4"))
+
+ @mock.patch.object(video_cut, "proc_run")
+ def test_nonzero_exit_always_fails_regardless_of_stderr(self, mock_run):
+ mock_run.return_value = mock.Mock(returncode=1, stdout="", stderr="")
+ self.assertIsNotNone(video_cut.verify_full_decode("clip.mp4"))
+
+ @mock.patch.object(video_cut, "proc_run")
+ def test_runs_with_threads_auto_for_a_cheaper_decode(self, mock_run):
+ mock_run.return_value = mock.Mock(returncode=0, stdout="", stderr="")
+ video_cut.verify_full_decode("clip.mp4")
+ cmd = mock_run.call_args[0][0]
+ self.assertIn("-threads", cmd)
+ self.assertEqual(cmd[cmd.index("-threads") + 1], "auto")
+
+
+ shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg/ffprobe not installed"
+)
+class MultiSegmentPartDurationTests(unittest.TestCase):
+ """Captions for a multi-cut clip are timed by summing part durations
+ (clip_generator.py). Each part's actual encoded length can be a few ms
+ off the requested end - start, and that drift compounds across cuts,
+ so the probed durations cut_multi_segment now returns must track the
+ real concatenated output, not the planned request."""
+
+ @classmethod
+ def setUpClass(cls):
+ cls.tmpdir = tempfile.mkdtemp(prefix="podcli-multiseg-test-")
+ cls.src = os.path.join(cls.tmpdir, "src.mp4")
+ subprocess.run(
+ [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc=size=64x64:rate=30:duration=20",
+ "-f", "lavfi", "-i", "sine=frequency=440:duration=20",
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac",
+ "-g", "30", "-keyint_min", "30", "-sc_threshold", "0",
+ cls.src,
+ ],
+ check=True, capture_output=True,
+ )
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.tmpdir, ignore_errors=True)
+
+ def test_sum_of_part_durations_matches_the_concatenated_output(self):
+ # Deliberately off-grid, irregular lengths so frame-snap rounding
+ # doesn't cancel out across segments.
+ segments = [
+ {"start": 0.13, "end": 0.46},
+ {"start": 1.02, "end": 1.58},
+ {"start": 2.21, "end": 2.69},
+ {"start": 3.07, "end": 3.91},
+ {"start": 4.44, "end": 4.77},
+ {"start": 5.10, "end": 5.88},
+ ]
+ out_path = os.path.join(self.tmpdir, "multi.mp4")
+ _, part_durations = video_cut.cut_multi_segment(self.src, out_path, segments)
+
+ self.assertEqual(len(part_durations), len(segments))
+ actual_total = video_cut.probe_duration(out_path)
+
+ # One frame at 30fps; the probed per-part durations summed should
+ # land within a frame of the real concatenated output.
+ frame = 1.0 / 30
+ self.assertLessEqual(abs(sum(part_durations) - actual_total), frame)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_whispercpp_adapter.py b/tests/test_whispercpp_adapter.py
index 78dea57..0fbfa4b 100644
--- a/tests/test_whispercpp_adapter.py
+++ b/tests/test_whispercpp_adapter.py
@@ -1,13 +1,20 @@
+import json
import os
import sys
+import tempfile
import unittest
+from unittest import mock
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
BACKEND_ROOT = os.path.join(ROOT, "backend")
if BACKEND_ROOT not in sys.path:
sys.path.insert(0, BACKEND_ROOT)
-from services.transcription_whispercpp import _dtw_preset_for_model, _tokens_to_words
+from services.transcription_whispercpp import (
+ _dtw_preset_for_model,
+ _tokens_to_words,
+ transcribe_file,
+)
class WhisperCppAdapterTests(unittest.TestCase):
@@ -19,6 +26,58 @@ class WhisperCppAdapterTests(unittest.TestCase):
self.assertEqual([w["word"] for w in words], ["hello", "world"])
+class LanguageHandlingTests(unittest.TestCase):
+ """whisper-cli defaults to English decoding when -l is omitted entirely, so
+ an unset language must become "-l auto" rather than no flag at all; the
+ reported language must come from whisper-cli's detected result, not the
+ (possibly "auto") param we passed in."""
+
+ def _fake_run(self, result_language, params_language):
+ def run(cmd, **kwargs):
+ of_index = cmd.index("-of")
+ out_base = cmd[of_index + 1]
+ payload = {
+ "params": {"language": params_language},
+ "result": {"language": result_language},
+ "transcription": [],
+ }
+ with open(out_base + ".json", "w", encoding="utf-8") as f:
+ json.dump(payload, f)
+ return mock.Mock(returncode=0)
+
+ return mock.Mock(side_effect=run)
+
+ def _run_transcribe(self, language, result_language, params_language):
+ with tempfile.NamedTemporaryFile(suffix=".mp3") as media, \
+ tempfile.NamedTemporaryFile(suffix=".bin") as model, \
+ tempfile.NamedTemporaryFile(suffix=".wav") as wav:
+ fake_run = self._fake_run(result_language, params_language)
+ with mock.patch("services.transcription_whispercpp.subprocess.run", fake_run):
+ result = transcribe_file(
+ media.name,
+ model.name,
+ language=language,
+ wav_path=wav.name,
+ )
+ cmd = fake_run.call_args[0][0]
+ return result, cmd
+
+ def test_unset_language_passes_auto(self):
+ _, cmd = self._run_transcribe(None, "es", "auto")
+ self.assertIn("-l", cmd)
+ self.assertEqual(cmd[cmd.index("-l") + 1], "auto")
+
+ def test_explicit_language_is_passed_through(self):
+ _, cmd = self._run_transcribe("fr", "fr", "fr")
+ self.assertEqual(cmd[cmd.index("-l") + 1], "fr")
+
+ def test_label_prefers_detected_result_language_over_param(self):
+ # Requesting auto-detect but whisper-cli actually detects Spanish: the
+ # output must be labeled "es", never the "auto" we passed as -l.
+ result, _ = self._run_transcribe(None, "es", "auto")
+ self.assertEqual(result["language"], "es")
+
+
class DtwPresetTests(unittest.TestCase):
def test_preset_tracks_the_model_file(self):
cases = {
diff --git a/tests/test_whispercpp_snap.py b/tests/test_whispercpp_snap.py
index 9f09861..d3ce5cb 100644
--- a/tests/test_whispercpp_snap.py
+++ b/tests/test_whispercpp_snap.py
@@ -5,6 +5,7 @@ silence; this keeps captions in sync.)"""
import math
import os
+import random
import struct
import sys
import tempfile
@@ -16,7 +17,55 @@ BACKEND_ROOT = os.path.join(ROOT, "backend")
if BACKEND_ROOT not in sys.path:
sys.path.insert(0, BACKEND_ROOT)
-from services.transcription_whispercpp import _snap_words_to_voiced, _voiced_intervals
+import numpy as np
+
+from services.transcription_whispercpp import _frame_rms, _snap_words_to_voiced, _voiced_intervals
+
+
+def _voiced_intervals_quadratic_reference(wav_path, bridge=0.3, thresh_ratio=0.07):
+ """The pre-fix implementation: gathers every window into an (nf, frame)
+ matrix before taking the RMS. Kept here only as a correctness oracle for
+ the O(n) rewrite in transcription_whispercpp._voiced_intervals. It must
+ never run on real audio, only the short synthetic clips in this test."""
+ import wave as wave_mod
+
+ try:
+ w = wave_mod.open(wav_path, "rb")
+ sr, width, n = w.getframerate(), w.getsampwidth(), w.getnframes()
+ raw = w.readframes(n)
+ w.close()
+ except Exception:
+ return []
+ if width != 2 or sr <= 0 or not raw:
+ return []
+ samples = np.frombuffer(raw, dtype=np.int16).astype(np.float32)
+ hop = max(1, int(sr * 0.010))
+ frame = max(hop, int(sr * 0.025))
+ if len(samples) < frame:
+ return []
+ nf = 1 + (len(samples) - frame) // hop
+ idx = np.arange(nf)[:, None] * hop + np.arange(frame)[None, :]
+ rms = np.sqrt((samples[idx] ** 2).mean(axis=1))
+ peak = float(rms.max())
+ if peak <= 0:
+ return []
+ voiced = rms > thresh_ratio * peak
+ intervals, start = [], None
+ for i, v in enumerate(voiced):
+ if v and start is None:
+ start = i * hop / sr
+ elif not v and start is not None:
+ intervals.append([start, i * hop / sr])
+ start = None
+ if start is not None:
+ intervals.append([start, nf * hop / sr])
+ merged = []
+ for iv in intervals:
+ if merged and iv[0] - merged[-1][1] <= bridge:
+ merged[-1][1] = iv[1]
+ else:
+ merged.append(iv)
+ return merged
def _make_wav(path, voiced_s=1.0, silence_s=1.0, sr=16000):
@@ -69,5 +118,108 @@ class EnergySnapTests(unittest.TestCase):
os.remove(silent)
+class VoicedIntervalsMatchesOldImplementationTests(unittest.TestCase):
+ """The O(n) running-sum rewrite has to produce the same voiced intervals
+ as the O(nf * frame) index-matrix version it replaces, on audio shaped
+ like what it actually has to handle: several voiced/silent transitions,
+ not just one clean on/off."""
+
+ def _make_mixed_wav(self, path, sr=16000, seed=0):
+ rng = random.Random(seed)
+ w = wave.open(path, "wb")
+ w.setnchannels(1)
+ w.setsampwidth(2)
+ w.setframerate(sr)
+ buf = bytearray()
+ # Alternating voiced tones and silence of varying length, plus a
+ # little noise floor during "silence" so it isn't a degenerate
+ # all-zero case.
+ for seg_i in range(6):
+ dur = 0.3 + 0.1 * seg_i
+ n = int(sr * dur)
+ if seg_i % 2 == 0:
+ for i in range(n):
+ val = int(7000 * math.sin(2 * math.pi * (180 + seg_i * 20) * i / sr))
+ buf += struct.pack("<h", val)
+ else:
+ for _ in range(n):
+ buf += struct.pack("<h", rng.randint(-50, 50))
+ w.writeframes(bytes(buf))
+ w.close()
+
+ def test_matches_quadratic_reference_on_mixed_audio(self):
+ path = tempfile.mktemp(suffix=".wav")
+ self._make_mixed_wav(path)
+ try:
+ fast = _voiced_intervals(path)
+ reference = _voiced_intervals_quadratic_reference(path)
+ finally:
+ os.remove(path)
+
+ self.assertEqual(len(fast), len(reference))
+ for (fs, fe), (rs, re) in zip(fast, reference):
+ self.assertAlmostEqual(fs, rs, places=4)
+ self.assertAlmostEqual(fe, re, places=4)
+
+ def test_matches_quadratic_reference_across_seeds(self):
+ for seed in range(3):
+ with self.subTest(seed=seed):
+ path = tempfile.mktemp(suffix=".wav")
+ self._make_mixed_wav(path, seed=seed)
+ try:
+ fast = _voiced_intervals(path)
+ reference = _voiced_intervals_quadratic_reference(path)
+ finally:
+ os.remove(path)
+ self.assertEqual(fast, reference)
+
+
+class FrameRmsLongFilePrecisionTests(unittest.TestCase):
+ """On a multi-hour file the running sum of squares climbs into the range
+ where float32 can no longer resolve one frame's contribution. A late
+ frame's RMS comes out wrong, and can cross the voiced/silent threshold
+ the wrong way. float64 has to hold exactly through the same file length."""
+
+ def test_late_frame_rms_matches_direct_computation_on_a_long_constant_signal(self):
+ sr = 16000
+ hop = int(sr * 0.010)
+ frame = int(sr * 0.025)
+ amplitude = 10000.0
+ # ~20.8 minutes at 16kHz: long enough for float32's cumulative sum
+ # of squares to lose more precision than a single frame is worth.
+ n_samples = 20_000_000
+ samples = np.full(n_samples, amplitude, dtype=np.float32)
+ nf = 1 + (n_samples - frame) // hop
+
+ rms = _frame_rms(samples, hop, frame, nf)
+
+ # A constant-amplitude signal has an exact, trivial per-frame RMS:
+ # amplitude itself, independent of position. So this is a direct
+ # computation, not another running sum that could share the bug.
+ late_frame = nf - 1
+ self.assertAlmostEqual(float(rms[late_frame]), amplitude, delta=1e-6)
+ mid_frame = nf // 2
+ self.assertAlmostEqual(float(rms[mid_frame]), amplitude, delta=1e-6)
+
+ def test_float32_cumsum_would_have_failed_the_same_assertion(self):
+ # Pins the regression: confirms the bug this test guards against is
+ # real and large, not a tolerance picked to pass trivially.
+ sr = 16000
+ hop = int(sr * 0.010)
+ frame = int(sr * 0.025)
+ amplitude = 10000.0
+ n_samples = 20_000_000
+ samples = np.full(n_samples, amplitude, dtype=np.float32)
+ nf = 1 + (n_samples - frame) // hop
+
+ sq = samples * samples
+ csum32 = np.concatenate(([0.0], np.cumsum(sq, dtype=np.float32)))
+ starts = np.arange(nf) * hop
+ window_sums = csum32[starts + frame] - csum32[starts]
+ rms32 = np.sqrt(np.maximum(window_sums, 0.0) / frame)
+
+ self.assertGreater(abs(float(rms32[nf // 2]) - amplitude), 100.0)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/vitest.config.ts b/vitest.config.ts
new file mode 100644
index 0000000..574ccf8
--- /dev/null
+++ b/vitest.config.ts
@@ -0,0 +1,10 @@
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+ test: {
+ // `npm run build` compiles src/**/*.test.ts into dist/**/*.test.js. Without
+ // this, a stale build directory makes vitest run every test twice, once
+ // against the TS source, once against whatever JS it compiled last time.
+ exclude: ["**/node_modules/**", "**/dist/**"],
+ },
+});