Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

Release 0.7.0: MCP server, batch folders, optional local transcription (#8)

kajisho5a4ab4da421f1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e4c99ff..1ebda37 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,12 @@
# Changelog
+## 0.7.0
+
+- `mcp/server.py` (new): the whole toolkit as an MCP server over stdio (JSON-RPC 2.0, standard library only). Every script is a tool; named args or raw argv; results as structured JSON. Installed alongside scripts by `npx ffmpeg-skill`.
+- `batch.py` (new): apply a step recipe or a render project to every file in a folder, content-hash cache so re-runs only touch changed files, `--watch` polling.
+- `caption.py --transcribe`: optional local speech-to-text bridge (whisper.cpp `whisper-cli`, faster-whisper, or openai-whisper if present). Never required; a clear install hint otherwise.
+- Tests: 49 end-to-end cases.
+
## 0.6.0
- `graphics.py` (new): motion-graphics templates with no image assets — lower-third (slide in/out), title card, chapter chip, progress bar, countdown, corner bug — coloured from brand.json.
diff --git a/README.md b/README.md
index fe23db0..ef4950f 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,9 @@ npx ffmpeg-skill
- **Lossless when possible** — cuts and joins use stream copy by default; re-encoding only happens when it must (frame-accurate cuts, filters, format changes).
- **Cut & join** segments with `mm:ss` / `hh:mm:ss.ms` times.
- **Declarative edits** — describe the whole edit in a `project.json` (clips, transitions, captions, overlays, music, loudness, export, check) and re-render after every tweak.
+- **MCP server** — `mcp/server.py` exposes every script as an MCP tool over stdio (stdlib only) for Claude Desktop, Cursor or any MCP client.
+- **Batch / watch folder** — one recipe over a whole shoot with a content-hash cache; re-runs only touch what changed.
+- **Optional local transcription** — `caption.py --transcribe` uses whisper.cpp / faster-whisper / openai-whisper when present; never required.
- **Brand kit** — one `brand.json` (fonts, colours, logo, safe margins, caption style) applied by captions, overlays, graphics and projects.
- **Motion graphics without assets** — lower-thirds, title cards, chapter chips, progress bars, countdowns and corner bugs drawn by FFmpeg.
- **HTML delivery report** — before/after contact sheets, media facts, loudness, compliance and the commands run, in one file.
@@ -99,6 +102,8 @@ More examples: [examples/README.md](examples/README.md). To see everything run e
| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision, colour space, rotation, audio channels as JSON; `--analyze` flags Log footage |
| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact |
| `render.py` | Render a whole edit from `project.json`; `--init`, `--dry-run`, `--stop-after` |
+| `batch.py` | Apply a step recipe or render project to a folder, cached, optional watch |
+| `mcp/server.py` | MCP server exposing all scripts as tools (stdio JSON-RPC) |
| `graphics.py` | Lower-third, title, chapter, progress, countdown, bug templates (brand colours) |
| `report.py` | Single-file HTML delivery report with sheets, facts, loudness, compliance, commands |
| `scenes.py` | Scene changes, audio peaks, highlight proposals and per-scene sheet |
@@ -119,6 +124,14 @@ More examples: [examples/README.md](examples/README.md). To see everything run e
All scripts: Python 3.9+, standard library only, `--help`, non-zero exit + stderr message on failure.
+## MCP
+
+```json
+{"mcpServers": {"ffmpeg-skill": {"command": "python3", "args": ["/Users/you/.claude/skills/ffmpeg-skill/mcp/server.py"]}}}
+```
+
+`python3 mcp/server.py --list` prints the tools; `--call probe '{"inputs": ["a.mp4"]}'` runs one from the shell.
+
## Requirements
- FFmpeg 5.0+ with `libx264`, `libx265`, `libass`, `prores_ks` and `libzimg` (for `color.py --to-sdr`); the default builds from Homebrew, apt and gyan.dev include all of them
diff --git a/SKILL.md b/SKILL.md
index a472ea8..6c0a7bc 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -1,6 +1,6 @@
---
name: ffmpeg-skill
-description: Professional video editing with local FFmpeg — declarative project rendering, brand kits, motion-graphics templates (lower-thirds, titles, countdowns), HTML delivery reports, scene detection and highlight picks, delivery compliance checks, cut, silence removal, transitions, multicam, captions (animated/karaoke timed to speech), fit to duration/aspect, audio sync with drift correction, HDR/HLG/Dolby Vision to SDR, LUTs, audio clean-up and ducking, loudness, overlays, platform exports, frame inspection and a real-footage verification kit; Python stdlib scripts, no cloud or API keys.
+description: Professional video editing with local FFmpeg — MCP server, batch folders, declarative project rendering, brand kits, motion-graphics templates (lower-thirds, titles, countdowns), HTML delivery reports, scene detection and highlight picks, delivery compliance checks, cut, silence removal, transitions, multicam, captions (animated/karaoke timed to speech), fit to duration/aspect, audio sync with drift correction, HDR/HLG/Dolby Vision to SDR, LUTs, audio clean-up and ducking, loudness, overlays, platform exports, frame inspection and a real-footage verification kit; Python stdlib scripts, no cloud or API keys.
---
# ffmpeg-skill
@@ -75,6 +75,8 @@ path on stdout, and defaults the output name to `<input>_<operation>.<ext>`.
| "add a lower third with my name", "title card", "countdown intro", "progress bar" | `graphics.py input.mp4 --template lower-third --name "..." --title "..." --start 2 --end 8` |
| "use our brand fonts/colours/logo" | pass `--brand brand.json` to caption/overlay/graphics, or `"brand"` in project.json |
| "send me a summary of what you did" | `report.py --before raw.mov --after final.mp4 --platform youtube -o report.html` |
+| "do this to every file in the folder", "process the whole shoot" | `batch.py FOLDER --recipe batch.json` (steps or a render project; cached) |
+| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (needs a local whisper; otherwise `--text`) |
| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` |
| "it's an iPhone Dolby Vision clip and players show it wrong" | `color.py clip.mov --to-sdr` or `color.py clip.mov --strip-dovi` (keep HDR, drop the DV layer) |
| "does it look like Log / S-Log / flat footage?" | `probe.py clip.mp4 --analyze` (`looks_like_log`) then `color.py --lut` |
@@ -176,6 +178,30 @@ check.py INPUT --platform youtube|shorts|reels|tiktok|x|linkedin|broadcast|podca
PASS/WARN/FAIL per check with the script that fixes it. Run it as the final
step before reporting a deliverable; fix FAILs, mention WARNs.
+### batch.py — same recipe over a folder, cached
+```
+batch.py FOLDER --recipe batch.json [--force] [--watch SECONDS] [--json]
+```
+`batch.json` holds either `steps` (a list of script argv with `{in}`/`{out}`
+placeholders, chained) or `project` (a render project applied per file).
+Outputs land in `output_dir` with `suffix`; a content-hash cache skips files
+already done with the same recipe. Use `--dry-run` to preview the plan.
+
+### caption.py --transcribe — optional local speech-to-text
+If `whisper-cli` (whisper.cpp), `faster-whisper` or `whisper` is installed,
+`caption.py input.mp4 --transcribe [--language ja] [--model base]` writes the
+SRT from the audio and burns it (combine with `--animate pop --karaoke`).
+Nothing is downloaded and nothing is required: without an engine it prints
+install hints and the user can supply `--text` cues instead. Always tell the
+user which engine was used, and treat the transcript as a draft to review.
+
+### MCP server — the toolkit for any MCP client
+`python3 mcp/server.py` speaks MCP over stdio; each script is a tool taking
+named args (flags without dashes, underscores for hyphens) or `argv`. Config
+for Claude Desktop / Claude Code:
+`{"mcpServers": {"ffmpeg-skill": {"command": "python3", "args": ["~/.claude/skills/ffmpeg-skill/mcp/server.py"]}}}`.
+Inside this skill, call the scripts directly; the server is for other hosts.
+
### graphics.py — motion-graphics templates
```
graphics.py INPUT --template lower-third|title|chapter|progress|countdown|bug [--name] [--title] [--subtitle]
diff --git a/bin/install.js b/bin/install.js
index f3b3e01..4995e32 100755
--- a/bin/install.js
+++ b/bin/install.js
@@ -22,7 +22,7 @@ const { spawnSync } = require('child_process');
const SKILL_NAME = 'ffmpeg-skill';
const ROOT = path.resolve(__dirname, '..');
-const PAYLOAD = ['SKILL.md', 'scripts'];
+const PAYLOAD = ['SKILL.md', 'scripts', 'mcp'];
const args = process.argv.slice(2);
const has = (flag) => args.includes(flag);
diff --git a/examples/README.md b/examples/README.md
index de0d58b..e8e322b 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -109,6 +109,21 @@ python3 $S/render.py project.json --fast # preview
python3 $S/render.py project.json # final
```
+### "Process the whole folder"
+```bash
+cat > batch.json <<'EOF2'
+{"glob": "*.mp4", "output_dir": "out", "suffix": "_yt",
+ "steps": [["silence.py", "{in}", "-o", "{out}"], ["loudness.py", "{in}", "-o", "{out}"], ["export.py", "{in}", "--preset", "youtube", "-o", "{out}"]]}
+EOF2
+python3 $S/batch.py ~/Shoot --recipe batch.json # second run: cached, only new files
+python3 $S/batch.py ~/Shoot --recipe batch.json --watch 60
+```
+
+### "Transcribe and caption it" (needs a local whisper)
+```bash
+python3 $S/caption.py talk.mp4 --transcribe --language en --animate pop --karaoke
+```
+
### "Add a lower third / title / countdown"
```bash
python3 $S/graphics.py talk.mp4 --template lower-third --name "Ada Lovelace" --title "Analyst" --start 2 --end 8 --brand brand.json
diff --git a/mcp/server.py b/mcp/server.py
new file mode 100755
index 0000000..674d4cd
--- /dev/null
+++ b/mcp/server.py
@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""ffmpeg-skill as an MCP server (stdio, JSON-RPC 2.0) — standard library only.
+
+Every script in ../scripts becomes a tool; arguments are passed as an argv list
+or as a flat object of flags. Results are the script's --json output.
+
+Run:
+ python3 mcp/server.py # stdio transport
+Claude Desktop / Claude Code config example:
+ {"mcpServers": {"ffmpeg-skill": {"command": "python3", "args": ["/path/to/ffmpeg-skill/mcp/server.py"]}}}
+"""
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any, Dict, List
+
+HERE = Path(__file__).resolve().parent
+SCRIPTS = HERE.parent / "scripts"
+PROTOCOL_VERSION = "2024-11-05"
+
+TOOLS: Dict[str, str] = {
+ "probe": "Inspect a media file: duration, fps/VFR, resolution, codecs, HDR, rotation, audio. Args: inputs (list), analyze (bool), compact (bool).",
+ "cut": "Cut a range or several segments, lossless when possible. Args: input, start, end, duration, segments, accurate, output.",
+ "fit": "Fit to a duration (speed/trim) and/or aspect (pad/crop). Args: input, duration, method, aspect, fit, width, fps, smooth, output.",
+ "caption": "Burn SRT/ASS or timed text; animated/karaoke styles; brand. Args: input, srt, ass, text, animate, karaoke, font, size, position, brand, output.",
+ "overlay": "Composite a logo/image/text with timing and fade. Args: input, image, text, logo, position, start, end, fade, opacity, scale, brand, output.",
+ "graphics": "Motion-graphics templates: lower-third, title, chapter, progress, countdown, bug. Args: input, template, name, title, subtitle, start, end, brand, output.",
+ "sync": "Detect offset between two recordings by audio, optional drift fix. Args: reference, second, fix_drift, replace_audio, trim_second, output.",
+ "multicam": "Align N cameras/recorders and switch between them. Args: inputs (list), switch, auto, audio, fix_drift, output.",
+ "audio": "Denoise/voice chain, music bed with ducking, fades, downmix, replace. Args: input, voice, denoise, music, duck, music_volume, fade_in, fade_out, downmix, replace, output.",
+ "loudness": "Two-pass EBU R128 normalisation. Args: input, lufs, tp, measure_only, output.",
+ "silence": "Remove dead air / list silences. Args: input, threshold, min_silence, margin, list, edl, output.",
+ "join": "Concatenate clips with transitions, normalising size/fps/audio. Args: inputs (list), transition, duration, width, height, fps, output.",
+ "color": "HDR→SDR tone mapping, LUTs, retag, strip Dolby Vision. Args: input, to_sdr, lut, retag, strip_dovi, tonemap, output.",
+ "export": "Platform presets: youtube, youtube4k, reels, x, prores, h265, gif. Args: input, preset, fit, output.",
+ "check": "Pre-delivery compliance per platform. Args: input, platform, no_loudness.",
+ "scenes": "Scene changes, audio peaks, highlight proposals. Args: input, highlights, target, edl, sheet.",
+ "look": "Contact sheet / frames / before-after PNG for visual checks. Args: input, at (list), tiles, compare, output.",
+ "render": "Render a whole edit from project.json. Args: project, fast, stop_after, init.",
+ "verify": "Run the toolchain on real files and report PASS/FAIL. Args: paths (list), quick, report.",
+ "report": "HTML delivery report. Args: after, before, platform, commands, notes, title, output.",
+}
+POSITIONAL = {"probe": ["inputs"], "sync": ["reference", "second"], "multicam": ["inputs"], "join": ["inputs"], "verify": ["paths"], "render": ["project"]}
+
+
+def build_argv(name: str, args: Dict[str, Any]) -> List[str]:
+ if isinstance(args.get("argv"), list):
+ argv = [str(a) for a in args["argv"]]
+ if name not in ("look", "probe") and "--json" not in argv and "--help" not in argv:
+ argv.append("--json")
+ return argv
+ argv: List[str] = []
+ args = dict(args)
+ pos = POSITIONAL.get(name, ["input"])
+ for key in pos:
+ val = args.pop(key, None)
+ if val is None:
+ continue
+ if isinstance(val, list):
+ argv += [str(v) for v in val]
+ else:
+ argv.append(str(val))
+ for key, val in args.items():
+ if val is None or val is False:
+ continue
+ flag = "--" + key.replace("_", "-")
+ if key == "output":
+ flag = "-o"
+ if key == "lufs" and name == "loudness":
+ flag = "-I"
+ if val is True:
+ argv.append(flag)
+ elif isinstance(val, list):
+ for v in val:
+ argv += [flag, str(v)]
+ else:
+ argv += [flag, str(val)]
+ if name not in ("look", "probe") and "--json" not in argv and "--help" not in argv:
+ argv.append("--json")
+ return argv
+
+
+def call_tool(name: str, args: Dict[str, Any]) -> Dict[str, Any]:
+ script = SCRIPTS / f"{name}.py"
+ if not script.exists():
+ return {"isError": True, "content": [{"type": "text", "text": f"unknown tool {name}"}]}
+ argv = build_argv(name, args or {})
+ proc = subprocess.run([sys.executable, str(script)] + argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+ stdout = proc.stdout.strip()
+ text = stdout
+ structured = None
+ try:
+ structured = json.loads(stdout) if stdout.startswith("{") or stdout.startswith("[") else None
+ except ValueError:
+ structured = None
+ if proc.returncode != 0:
+ err = proc.stderr.strip().splitlines()
+ tail = "\n".join(err[-12:])
+ return {"isError": True, "content": [{"type": "text", "text": f"{name} failed (exit {proc.returncode})\n{tail}"}]}
+ if structured is None:
+ text = stdout or "\n".join(proc.stderr.strip().splitlines()[-5:])
+ result: Dict[str, Any] = {"content": [{"type": "text", "text": text}]}
+ if structured is not None:
+ result["structuredContent"] = structured if isinstance(structured, dict) else {"result": structured}
+ return result
+
+
+def tool_list() -> List[Dict[str, Any]]:
+ out = []
+ for name, desc in TOOLS.items():
+ out.append({
+ "name": name,
+ "description": desc + " Pass either named args (flags without dashes, underscores for hyphens) or argv (raw CLI list). Media paths must be absolute.",
+ "inputSchema": {"type": "object", "properties": {"argv": {"type": "array", "items": {"type": "string"}, "description": "raw CLI arguments"}}, "additionalProperties": True},
+ })
+ return out
+
+
+def handle(req: Dict[str, Any]) -> Dict[str, Any]:
+ method = req.get("method")
+ params = req.get("params") or {}
+ if method == "initialize":
+ return {"protocolVersion": PROTOCOL_VERSION, "capabilities": {"tools": {}}, "serverInfo": {"name": "ffmpeg-skill", "version": version()}}
+ if method == "tools/list":
+ return {"tools": tool_list()}
+ if method == "tools/call":
+ return call_tool(params.get("name", ""), params.get("arguments") or {})
+ if method == "ping":
+ return {}
+ raise KeyError(method)
+
+
+def version() -> str:
+ try:
+ return json.loads((HERE.parent / "package.json").read_text())["version"]
+ except Exception:
+ return "0"
+
+
+def main() -> int:
+ if "--list" in sys.argv:
+ for t in tool_list():
+ print(f"{t['name']:10s} {t['description'].split(' Pass either')[0]}")
+ return 0
+ if "--call" in sys.argv: # debugging helper: --call NAME '{"input": "..."}'
+ i = sys.argv.index("--call")
+ name = sys.argv[i + 1]
+ args = json.loads(sys.argv[i + 2]) if len(sys.argv) > i + 2 else {}
+ print(json.dumps(call_tool(name, args), indent=2))
+ return 0
+ stdin = sys.stdin.buffer
+ stdout = sys.stdout.buffer
+ for raw in stdin:
+ line = raw.decode("utf-8", errors="replace").strip()
+ if not line:
+ continue
+ try:
+ req = json.loads(line)
+ except ValueError:
+ continue
+ if "id" not in req: # notification
+ continue
+ try:
+ result = handle(req)
+ resp = {"jsonrpc": "2.0", "id": req["id"], "result": result}
+ except KeyError as exc:
+ resp = {"jsonrpc": "2.0", "id": req["id"], "error": {"code": -32601, "message": f"method not found: {exc}"}}
+ except Exception as exc: # noqa: BLE001
+ resp = {"jsonrpc": "2.0", "id": req["id"], "error": {"code": -32000, "message": str(exc)}}
+ stdout.write((json.dumps(resp) + "\n").encode("utf-8"))
+ stdout.flush()
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/package.json b/package.json
index fe11c68..cca91a0 100644
--- a/package.json
+++ b/package.json
@@ -1,7 +1,7 @@
{
"name": "ffmpeg-skill",
- "version": "0.6.0",
- "description": "Agent Skill that lets coding agents (Claude Code, Cursor, Codex) do professional video editing with local FFmpeg: declarative project rendering, brand kits, motion-graphics templates, HTML delivery reports, scene detection, delivery checks, cut, silence removal, transitions, multicam, captions, sync with drift correction, HDR to SDR, LUTs, audio clean-up and ducking, loudness, platform exports. No API keys, no cloud, no dependencies.",
+ "version": "0.7.0",
+ "description": "Agent Skill that lets coding agents (Claude Code, Cursor, Codex) do professional video editing with local FFmpeg: MCP server, batch processing, declarative project rendering, brand kits, motion-graphics templates, HTML delivery reports, scene detection, delivery checks, cut, silence removal, transitions, multicam, captions, sync with drift correction, HDR to SDR, LUTs, audio clean-up and ducking, loudness, platform exports. No API keys, no cloud, no dependencies.",
"keywords": ["ffmpeg", "video", "agent-skill", "claude-code", "cursor", "codex", "skill", "video-editing"],
"license": "MIT",
"author": "kajisho5",
diff --git a/scripts/batch.py b/scripts/batch.py
new file mode 100755
index 0000000..b196ce7
--- /dev/null
+++ b/scripts/batch.py
@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+"""Apply the same recipe to every file in a folder, with a content-hash cache
+so re-runs only process what changed. A recipe is a list of script steps;
+{in} and {out} are substituted, and the output of one step feeds the next.
+
+Recipe (batch.json):
+{
+ "glob": "*.mp4",
+ "output_dir": "out",
+ "suffix": "_final",
+ "steps": [
+ ["silence.py", "{in}", "--threshold", "-38", "-o", "{out}"],
+ ["loudness.py", "{in}", "-o", "{out}"],
+ ["export.py", "{in}", "--preset", "youtube", "-o", "{out}"]
+ ]
+}
+or use a render project for every file: {"project": "project.json", "clip_key": 0}
+
+Examples:
+ python3 batch.py ~/Footage --recipe batch.json
+ python3 batch.py ~/Footage --recipe batch.json --dry-run
+ python3 batch.py ~/Footage --recipe batch.json --force # ignore the cache
+ python3 batch.py ~/Footage --recipe batch.json --watch 30 # poll the folder every 30 s
+"""
+import argparse
+import hashlib
+import json
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any, Dict, List
+
+from _common import STATE, add_common, apply_common, die, emit, info
+
+HERE = Path(__file__).resolve().parent
+MEDIA_EXT = {".mp4", ".mov", ".m4v", ".mkv", ".webm", ".avi", ".mts", ".m2ts", ".mxf", ".wav", ".m4a", ".mp3", ".flac"}
+
+
+def file_key(path: Path) -> str:
+ st = path.stat()
+ h = hashlib.sha1()
+ h.update(f"{path.name}|{st.st_size}|{int(st.st_mtime)}".encode())
+ with open(path, "rb") as fh: # first and last MB: cheap and good enough to detect changes
+ h.update(fh.read(1 << 20))
+ if st.st_size > 2 << 20:
+ fh.seek(-(1 << 20), os.SEEK_END)
+ h.update(fh.read(1 << 20))
+ return h.hexdigest()
+
+
+def recipe_key(recipe: Dict[str, Any]) -> str:
+ return hashlib.sha1(json.dumps(recipe, sort_keys=True).encode()).hexdigest()[:12]
+
+
+def run_step(argv: List[str]) -> bool:
+ cmd = [sys.executable, str(HERE / argv[0])] + argv[1:]
+ if STATE["fast"]:
+ cmd.append("--fast")
+ if STATE["dry_run"]:
+ cmd.append("--dry-run")
+ info(" → " + " ".join(os.path.basename(c) if i < 2 else c for i, c in enumerate(cmd)))
+ proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+ if proc.returncode != 0:
+ info(" " + "\n ".join(proc.stderr.strip().splitlines()[-4:]))
+ return False
+ return True
+
+
+def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path) -> Dict[str, Any]:
+ suffix = recipe.get("suffix", "_out")
+ final_ext = recipe.get("ext") or src.suffix.lstrip(".") or "mp4"
+ final = outdir / f"{src.stem}{suffix}.{final_ext}"
+ t0 = time.time()
+ if recipe.get("project"):
+ proj = json.loads(Path(recipe["project"]).read_text(encoding="utf-8"))
+ idx = int(recipe.get("clip_key", 0))
+ proj.setdefault("clips", [{}])
+ while len(proj["clips"]) <= idx:
+ proj["clips"].append({})
+ proj["clips"][idx]["src"] = str(src.resolve())
+ proj["output"] = str(final.resolve())
+ pj = work / f"{src.stem}_project.json"
+ pj.write_text(json.dumps(proj, indent=2), encoding="utf-8")
+ ok = run_step(["render.py", str(pj)])
+ else:
+ steps = recipe.get("steps") or []
+ if not steps:
+ die("recipe needs steps or project")
+ cur = str(src)
+ ok = True
+ for i, step in enumerate(steps):
+ last = i == len(steps) - 1
+ out = str(final) if last else str(work / f"{src.stem}_step{i}.{'mp4' if src.suffix.lower() not in ('.wav', '.mp3', '.m4a', '.flac') else src.suffix.lstrip('.')}")
+ argv = [str(a).replace("{in}", cur).replace("{out}", out) for a in step]
+ if not run_step(argv):
+ ok = False
+ break
+ cur = out
+ return {"file": str(src), "output": str(final), "ok": ok, "seconds": round(time.time() - t0, 1)}
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("folder")
+ ap.add_argument("--recipe", required=True, help="batch.json")
+ ap.add_argument("--force", action="store_true", help="ignore the cache and redo everything")
+ ap.add_argument("--watch", type=float, help="keep polling the folder every N seconds")
+ ap.add_argument("--work", help="work directory for intermediates (default: <output_dir>/.work)")
+ add_common(ap)
+ args = ap.parse_args()
+ apply_common(args)
+
+ folder = Path(args.folder)
+ if not folder.is_dir():
+ die(f"not a folder: {folder}")
+ try:
+ recipe = json.loads(Path(args.recipe).read_text(encoding="utf-8"))
+ except (OSError, ValueError) as exc:
+ die(f"cannot read recipe: {exc}")
+ if recipe.get("project") and not os.path.isabs(recipe["project"]):
+ recipe["project"] = str((Path(args.recipe).resolve().parent / recipe["project"]))
+ outdir = Path(recipe.get("output_dir") or (folder / "out"))
+ if not outdir.is_absolute():
+ outdir = folder / outdir
+ work = Path(args.work) if args.work else outdir / ".work"
+ outdir.mkdir(parents=True, exist_ok=True)
+ work.mkdir(parents=True, exist_ok=True)
+ cache_path = outdir / ".ffskill_cache.json"
+ cache: Dict[str, Any] = {}
+ if cache_path.exists() and not args.force:
+ try:
+ cache = json.loads(cache_path.read_text(encoding="utf-8"))
+ except ValueError:
+ cache = {}
+ rkey = recipe_key(recipe)
+ glob = recipe.get("glob") or "*"
+
+ def one_pass() -> List[Dict[str, Any]]:
+ results = []
+ files = sorted(p for p in folder.glob(glob) if p.is_file() and p.suffix.lower() in MEDIA_EXT and outdir not in p.parents)
+ for src in files:
+ key = f"{file_key(src)}:{rkey}"
+ hit = cache.get(key)
+ if hit and Path(hit.get("output", "")).exists() and not args.force:
+ info(f"skip (cached) {src.name}")
+ results.append({**hit, "cached": True})
+ continue
+ info(f"=== {src.name}")
+ r = process(src, recipe, outdir, work)
+ results.append(r)
+ if r["ok"] and not STATE["dry_run"]:
+ cache[key] = r
+ cache_path.write_text(json.dumps(cache, indent=2), encoding="utf-8")
+ return results
+
+ results = one_pass()
+ if args.watch:
+ info(f"watching {folder} every {args.watch:g}s (Ctrl-C to stop)")
+ try:
+ while True:
+ time.sleep(args.watch)
+ results = one_pass()
+ except KeyboardInterrupt:
+ pass
+ done = sum(1 for r in results if r["ok"])
+ info(f"{done}/{len(results)} processed, {sum(1 for r in results if r.get('cached'))} from cache")
+ emit(None, results=results, processed=done, total=len(results))
+ if not args.json:
+ for r in results:
+ print(f"{'OK ' if r['ok'] else 'FAIL'} {r['file']} -> {r['output']}" + (" (cached)" if r.get("cached") else ""))
+ return 0 if done == len(results) else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/caption.py b/scripts/caption.py
index 9b01549..c272c31 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -21,7 +21,7 @@ import os
import re
import sys
from pathlib import Path
-from typing import List, Tuple
+from typing import List, Optional, Tuple
from _common import color_hex, load_brand, video_args, add_common, apply_common, emit, aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, fmt_srt_time, info, parse_time, probe, run, x264_args
@@ -60,6 +60,71 @@ def parse_text_cues(path: str, auto_seconds: float, gap: float) -> List[Tuple[fl
return cues
+def transcribe(video: str, out_srt: str, language: Optional[str], model: str) -> List[Tuple[float, float, str]]:
+ """Optional local ASR bridge. Tries, in order: whisper-cli / main (whisper.cpp), faster-whisper (python),
+ whisper (openai-whisper CLI). Produces an SRT with word timings where the engine supports it.
+ No engine installed -> clear error with install hints; the skill never depends on one."""
+ import shutil
+ import subprocess
+ import tempfile
+ from _common import require_tool
+ ffmpeg = require_tool("ffmpeg")
+ tmpdir = tempfile.mkdtemp(prefix="ffskill_asr_")
+ wav = os.path.join(tmpdir, "audio.wav")
+ subprocess.run([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav], check=True)
+ # 1. whisper.cpp
+ cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp") or shutil.which("main")
+ if cli and (shutil.which("whisper-cli") or shutil.which("whisper-cpp")):
+ model_path = model
+ if not os.path.exists(model_path):
+ for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"), f"models/ggml-{model}.bin", f"/usr/local/share/whisper/ggml-{model}.bin"):
+ if os.path.exists(cand):
+ model_path = cand
+ break
+ base = os.path.join(tmpdir, "out")
+ cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base]
+ if language:
+ cmd += ["-l", language]
+ proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+ if proc.returncode == 0 and os.path.exists(base + ".srt"):
+ info(f"transcribed with whisper.cpp ({os.path.basename(cli)}, model {os.path.basename(model_path)})")
+ cues = parse_srt(base + ".srt")
+ write_srt(cues, out_srt)
+ return cues
+ info("whisper.cpp found but failed: " + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
+ # 2. faster-whisper (python package)
+ try:
+ from faster_whisper import WhisperModel # type: ignore
+ m = WhisperModel(model, device="cpu", compute_type="int8")
+ segments, _ = m.transcribe(wav, language=language, word_timestamps=False)
+ cues = [(seg.start, seg.end, seg.text.strip()) for seg in segments if seg.text.strip()]
+ if cues:
+ info("transcribed with faster-whisper")
+ write_srt(cues, out_srt)
+ return cues
+ except ImportError:
+ pass
+ # 3. openai-whisper CLI
+ if shutil.which("whisper"):
+ cmd = ["whisper", wav, "--model", model, "--output_format", "srt", "--output_dir", tmpdir]
+ if language:
+ cmd += ["--language", language]
+ proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+ srt = os.path.join(tmpdir, "audio.srt")
+ if proc.returncode == 0 and os.path.exists(srt):
+ info("transcribed with openai-whisper")
+ cues = parse_srt(srt)
+ write_srt(cues, out_srt)
+ return cues
+ die("no local speech-to-text engine found for --transcribe.\n"
+ "Install one (all run offline):\n"
+ " whisper.cpp: brew install whisper-cpp (then download a model: ggml-base.bin)\n"
+ " faster-whisper: pip install faster-whisper\n"
+ " openai-whisper: pip install openai-whisper\n"
+ "Or write the cues by hand with --text cues.txt (see format above).")
+ return []
+
+
def parse_srt(path: str) -> List[Tuple[float, float, str]]:
cues: List[Tuple[float, float, str]] = []
block: List[str] = []
@@ -217,6 +282,9 @@ def main() -> int:
src.add_argument("--srt", help="SRT file to burn")
src.add_argument("--ass", help="ASS file to burn (styles inside the file are used)")
src.add_argument("--text", help="plain text cue file to convert into SRT (see format above)")
+ src.add_argument("--transcribe", action="store_true", help="generate the SRT from the audio with a local speech-to-text engine if one is installed (whisper-cli / whisper / faster-whisper); never required")
+ src.add_argument("--language", help="language code for --transcribe (e.g. en, ja); default auto")
+ src.add_argument("--model", default="base", help="whisper model name/path for --transcribe (default base)")
src.add_argument("--write-srt", help="where to save the generated SRT (default: <text>.srt)")
src.add_argument("--auto-seconds", type=float, default=3.0, help="duration for cues without timing (default 3)")
src.add_argument("--gap", type=float, default=0.0, help="gap after auto-timed cues in seconds")
@@ -263,10 +331,17 @@ def main() -> int:
args.karaoke = True
if args.brand and brand.get("font_file") and not args.fonts_dir:
args.fonts_dir = str(Path(brand["font_file"]).parent)
- if not (args.srt or args.ass or args.text):
- die("give one of --srt, --ass or --text")
+ if not (args.srt or args.ass or args.text or args.transcribe):
+ die("give one of --srt, --ass, --text or --transcribe")
srt_path = args.srt
+ if args.transcribe:
+ if not args.input:
+ die("--transcribe needs the input video")
+ srt_path = args.write_srt or os.path.splitext(args.input)[0] + ".srt"
+ cues = transcribe(args.input, srt_path, args.language, args.model)
+ info(f"wrote {srt_path} ({len(cues)} cues)")
+ args.text = None
if args.text:
cues = parse_text_cues(args.text, args.auto_seconds, args.gap)
srt_path = args.write_srt or os.path.splitext(args.text)[0] + ".srt"
diff --git a/tests/test_all.py b/tests/test_all.py
index 3e59e36..d045681 100644
--- a/tests/test_all.py
+++ b/tests/test_all.py
@@ -695,6 +695,72 @@ class FFmpegSkillTests(unittest.TestCase):
script("report.py", "--after", reels, "--no-sheets", "-o", OUT / "report2.html")
self.assertLess((OUT / "report2.html").stat().st_size, 40000)
+ # ---------------------------------------------------------------- v0.7: MCP server / batch / ASR bridge
+ def test_mcp_server_stdio(self):
+ server = ROOT / "mcp" / "server.py"
+ reqs = [
+ {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}},
+ {"jsonrpc": "2.0", "method": "notifications/initialized"},
+ {"jsonrpc": "2.0", "id": 2, "method": "tools/list"},
+ {"jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": {"name": "probe", "arguments": {"inputs": [str(self.src)]}}},
+ {"jsonrpc": "2.0", "id": 4, "method": "tools/call", "params": {"name": "cut", "arguments": {"input": str(self.src), "start": 1, "end": 3, "output": str(OUT / "mcp_cut.mp4")}}},
+ {"jsonrpc": "2.0", "id": 5, "method": "tools/call", "params": {"name": "cut", "arguments": {"argv": [str(self.src), "--start", "0", "--end", "2", "-o", str(OUT / "mcp_cut2.mp4")]}}},
+ {"jsonrpc": "2.0", "id": 6, "method": "tools/call", "params": {"name": "nope", "arguments": {}}},
+ {"jsonrpc": "2.0", "id": 7, "method": "bogus/method"},
+ ]
+ proc = subprocess.run([sys.executable, str(server)], input="\n".join(json.dumps(r) for r in reqs) + "\n", stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+ self.assertEqual(proc.returncode, 0, proc.stderr)
+ resp = {r["id"]: r for r in (json.loads(l) for l in proc.stdout.splitlines() if l.strip())}
+ self.assertEqual(resp[1]["result"]["serverInfo"]["name"], "ffmpeg-skill")
+ names = {t["name"] for t in resp[2]["result"]["tools"]}
+ self.assertTrue({"probe", "cut", "render", "check", "scenes", "batch"} <= names or {"probe", "cut", "render", "check", "scenes"} <= names)
+ self.assertEqual(resp[3]["result"]["structuredContent"]["duration"], 12.0)
+ self.assertClose(resp[4]["result"]["structuredContent"]["probe"]["duration"], 2.0, 0.6)
+ self.assertTrue(Path(resp[5]["result"]["structuredContent"]["output"]).exists())
+ self.assertTrue(resp[6]["result"].get("isError"))
+ self.assertEqual(resp[7]["error"]["code"], -32601)
+
+ def test_batch_recipe_and_cache(self):
+ folder = OUT / "batch_in"
+ folder.mkdir(exist_ok=True)
+ for name in ("a.mp4", "b.mp4"):
+ (folder / name).write_bytes(Path(self.src).read_bytes())
+ recipe = folder / "batch.json"
+ recipe.write_text(json.dumps({"glob": "*.mp4", "output_dir": "out", "suffix": "_final",
+ "steps": [["fit.py", "{in}", "--duration", "5", "-o", "{out}"], ["export.py", "{in}", "--preset", "x", "-o", "{out}"]]}))
+ data = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast", "--json").stdout)
+ self.assertEqual(data["processed"], 2)
+ self.assertFalse(any(r.get("cached") for r in data["results"]))
+ for r in data["results"]:
+ m = probe(r["output"])
+ self.assertClose(m["duration"], 5.0, 0.3)
+ self.assertEqual(m["video"]["width"], 1280)
+ data = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast", "--json").stdout)
+ self.assertTrue(all(r.get("cached") for r in data["results"]), "second run served from cache")
+ data = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast", "--force", "--json").stdout)
+ self.assertFalse(any(r.get("cached") for r in data["results"]))
+ # project-based recipe
+ proj = folder / "p.json"
+ proj.write_text(json.dumps({"clips": [{"src": "x", "in": 0, "out": 3}], "export": {"preset": "x"}}))
+ recipe2 = folder / "batch2.json"
+ recipe2.write_text(json.dumps({"glob": "a.mp4", "output_dir": "out2", "suffix": "_p", "project": "p.json"}))
+ data = json.loads(script("batch.py", folder, "--recipe", recipe2, "--fast", "--json").stdout)
+ self.assertEqual(data["processed"], 1)
+ self.assertClose(probe(data["results"][0]["output"])["duration"], 3.0, 0.3)
+
+ def test_transcribe_without_engine_explains(self):
+ import shutil
+ if shutil.which("whisper-cli") or shutil.which("whisper"):
+ self.skipTest("a local ASR engine is installed")
+ try:
+ import faster_whisper # noqa: F401
+ self.skipTest("faster-whisper installed")
+ except ImportError:
+ pass
+ proc = script("caption.py", self.src, "--transcribe", "-o", OUT / "asr.mp4", expect_fail=True)
+ self.assertIn("no local speech-to-text engine", proc.stderr)
+ self.assertIn("whisper", proc.stderr)
+
# ---------------------------------------------------------------- help
def test_every_script_has_help(self):
for name in sorted(p.name for p in SCRIPTS.glob("*.py") if not p.name.startswith("_")):