Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

contract --json: add reencodes_video/reencodes_audio per tool (#36)

Shogo Kajiyama42a6b6a6c4f1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 67f93db..ce207a9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
## Unreleased — FFmpeg 8+ / Windows compatibility for caption and color
+- **`contract --json`: `reencodes_video`/`reencodes_audio` per tool.** Each of the 21 tools now declares, per stream type, `"always"` / `"never"` / `"conditional"` (with a `reencode_note` for the conditional ones), read from what each script's own encode/copy args actually do. Surfaces a fact that wasn't documented anywhere: `fit`, `caption`, `overlay`, `graphics`, `color`, `join`, `multicam` and `silence` always transcode audio to AAC alongside a video filter — there is no `-c:a copy` path in this codebase for a tool that also re-encodes video, so a caller cannot assume the original audio codec survives a picture-only edit. `caption.py`'s docstring now says plainly that burn-in is the only mode (no soft-subtitle mux) and always re-encodes both streams. Additive contract field; no tool's behaviour changed.
- **SKILL.md: explicit "what this skill does and does not decide".** Added a section naming what belongs to a production agent (approval, which cut is right), another skill (thumbnail composition), or nobody in this skill (content understanding, judging a highlight's interest beyond a measured proxy) -- and an explicit rule against ever falling back to a raw `ffmpeg`/`filter_complex` invocation when a request needs something none of the 21 scripts expose. The trigger description (frontmatter) is intentionally left as-is: it stays broad on purpose (any video/audio file touch), since narrowing it risks under-triggering on requests that do belong here; what changed is what the skill does once triggered. `evals/agent_prompts_24.json` gains one refusal case (`r05-no-raw-ffmpeg`) asking directly for a raw `-filter_complex` command.
- **`check.py`: plain-language `reason` on the less obvious FAILs.** Every row already had `fix` (the command that resolves it); video codec, pixel format, HDR colour and loudness FAILs now also carry `reason` ("QuickTime and iOS commonly reject video that isn't 8-bit 4:2:0") for a caller reporting the result to someone who doesn't already know why the spec value matters. Empty on PASS rows and every other check. Additive, no existing field changed.
- **`cut.py --json`: full requested-vs-actual provenance.** `expected_duration`/`duration_error_ms`/`precision`/`reencoded` already existed (0.9.1); added `requested_start`/`requested_end` (or `requested_segments` for `--segments`), `requested_duration`, `output_duration`, `duration_delta_seconds` (seconds-unit alias of `duration_error_ms`) and `mode` (`copy`/`accurate`/`hybrid` — "hybrid" means a lossless cut silently re-encoded because the keyframe snap exceeded `--tolerance`) and `keyframe_snapped`. Additive only, no existing field changed. Intended for a downstream repo (an editing skill, an agent) that wants this in its own provenance/audit trail without re-deriving it from `reencoded`+`precision`.
diff --git a/docs/contract.md b/docs/contract.md
index fe068b5..104d67d 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -64,6 +64,7 @@ One entry per tool under `tools`, sorted by id. Tool ids are stable:
| `produces_artifact` | writes a file (media, PNG, HTML, EDL) |
| `verification` | `{required, tools}`: which tools to run on the output afterwards |
| `requires_visual_verification` | the picture changed; run `ffmpeg-skill/look` and inspect the PNG |
+| `reencodes_video`, `reencodes_audio` | `"always"` / `"never"` / `"conditional"`, meaning *when that stream is present in the input* — not whether the tool touches the file at all. `"conditional"` tools (`cut`, `export`, `render`, `batch`, `verify`) carry a `reencode_note` explaining what it depends on. Several visual tools (`fit`, `caption`, `overlay`, `graphics`, `color`, `join`, `multicam`, `silence`) are `"always"` on audio too: this codebase never mixes `-c:v` re-encode with `-c:a copy` in one call, so a caller cannot assume the original audio codec survives just because only the picture changed |
| `audio_only` | accepts an audio-only input (WAV, MP3, M4A, FLAC, OGG, Opus) |
| `video_required` | refuses an input without a video stream ("input has no video stream") |
| | `join` has `audio_only: true` and `video_required: false` since 0.9.1: audio-only inputs are joined as audio (no `look` needed then); mixing audio and video inputs is refused |
diff --git a/scripts/_contract.py b/scripts/_contract.py
index 6862595..e003f51 100644
--- a/scripts/_contract.py
+++ b/scripts/_contract.py
@@ -146,6 +146,38 @@ DRY_RUN_NOTES = {
"verify": "not supported: the flag is accepted but the steps run and outputs are written",
}
+# Whether a tool re-encodes each stream *when that stream is present in the input* -- not whether
+# the tool touches the file at all. "always"/"never" are unconditional given that stream exists;
+# "conditional" means it depends on flags or on how far a lossless attempt misses (see "note").
+# Read from each script's actual encode/copy args, not from role or intent, since several tools
+# (fit, caption, overlay, graphics, color, join, multicam, silence) always transcode audio to AAC
+# alongside a video filter even though the audio itself is untouched content -- there is no
+# "-c:a copy while re-encoding video" path in this codebase, so a soft-subtitle-style passthrough
+# of the original audio codec never happens on those tools.
+REENCODE_META: Dict[str, Dict[str, str]] = {
+ "probe": dict(video="never", audio="never", note="analysis only, no artifact"),
+ "cut": dict(video="conditional", audio="conditional", note="lossless -c copy preferred; re-encodes on --accurate, a VFR source, or a keyframe snap past --tolerance (see cut.py --json: mode, keyframe_snapped)"),
+ "fit": dict(video="always", audio="always", note="always re-encodes to AAC when audio is present, even if only --fps or --aspect was asked for"),
+ "caption": dict(video="always", audio="always", note="burn-in only: no soft-subtitle mux path exists, so captions always cost a full re-encode of both streams"),
+ "overlay": dict(video="always", audio="always"),
+ "graphics": dict(video="always", audio="always"),
+ "sync": dict(video="never", audio="conditional", note="video is never touched; audio is copied or re-encoded depending on --trim-second / --replace-audio / --fix-drift"),
+ "multicam": dict(video="always", audio="always"),
+ "audio": dict(video="never", audio="always", note="video stream is always -c:v copy when present; this tool's job is the audio"),
+ "loudness": dict(video="never", audio="always"),
+ "silence": dict(video="always", audio="always", note="removing gaps requires cutting on non-keyframe boundaries"),
+ "join": dict(video="always", audio="always"),
+ "color": dict(video="always", audio="always"),
+ "export": dict(video="conditional", audio="conditional", note="--preset copy is -c:v copy -c:a copy (no re-encode); every other preset re-encodes both"),
+ "check": dict(video="never", audio="never", note="read-only, no artifact"),
+ "scenes": dict(video="never", audio="never", note="analysis only; --sheet renders a new contact-sheet PNG, not a re-encode of the source"),
+ "look": dict(video="never", audio="never", note="renders a new contact-sheet/frame PNG, not a re-encode of the source"),
+ "render": dict(video="conditional", audio="conditional", note="delegated: depends on which stages a project.json runs and how each one behaves"),
+ "batch": dict(video="conditional", audio="conditional", note="delegated: depends on which script each recipe step runs"),
+ "verify": dict(video="conditional", audio="conditional", note="delegated: runs cut/fit/caption/export/loudness/color internally as checks"),
+ "report": dict(video="never", audio="never", note="measures via look/check; produces an HTML report, not a re-encoded artifact"),
+}
+
IDEMPOTENCY = {
"bit_exact": "same inputs and flags give byte-identical output",
"content_equivalent": "same inputs and flags give the same media content; bytes may differ between encoder builds",
@@ -506,6 +538,11 @@ def tool_spec(name: str, version: str) -> Dict[str, Any]:
"mutates_input": False,
"produces_artifact": meta["produces_artifact"],
"verification": {"required": bool(meta["verify"]), "tools": [f"{SKILL_ID}/{t}" for t in meta["verify"]]},
+ # a tool with no declared entry reports "conditional" rather than guessing "always" or
+ # "never" -- same "unknown is not missing" principle as doctor's capability detection
+ "reencodes_video": REENCODE_META.get(name, {}).get("video", "conditional"),
+ "reencodes_audio": REENCODE_META.get(name, {}).get("audio", "conditional"),
+ **({"reencode_note": REENCODE_META[name]["note"]} if REENCODE_META.get(name, {}).get("note") else {}),
"requires_visual_verification": meta["visual"],
"audio_only": meta["audio_only"],
"video_required": meta["video_required"],
diff --git a/scripts/caption.py b/scripts/caption.py
index 3197615..9843ba5 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -4,6 +4,12 @@
Styling (font, size, colour, outline, position) applies to SRT input via
libass force_style. ASS files carry their own styles and are rendered as-is.
+This only burns subtitles into the picture. There is no soft-subtitle (mux a
+subtitle stream, toggleable by the player) path -- every call re-encodes the
+whole video and its audio (see contract --json: reencodes_video/reencodes_audio
+are both "always" for this tool), even for a source that only needed the
+subtitle track added.
+
Text-to-SRT input format (one cue per line, blank lines ignored):
0:00-0:03 Hello and welcome
00:00:03.500 --> 00:00:06 Second line | with a manual line break
diff --git a/tests/test_contract.py b/tests/test_contract.py
index ee7893a..f79547c 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -262,6 +262,17 @@ class ContractTests(unittest.TestCase):
skill = (ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertIn("Look: not needed", skill)
+ def test_reencodes_video_and_audio_declared_for_every_tool(self):
+ for t in self.contract["tools"]:
+ self.assertIn(t["reencodes_video"], ("always", "never", "conditional"), t["name"])
+ self.assertIn(t["reencodes_audio"], ("always", "never", "conditional"), t["name"])
+ # a handful of the least intuitive ones, checked against what the scripts actually do
+ self.assertEqual((self.tools["cut"]["reencodes_video"], self.tools["cut"]["reencodes_audio"]), ("conditional", "conditional"))
+ self.assertEqual((self.tools["export"]["reencodes_video"], self.tools["export"]["reencodes_audio"]), ("conditional", "conditional"))
+ self.assertEqual((self.tools["caption"]["reencodes_video"], self.tools["caption"]["reencodes_audio"]), ("always", "always"))
+ self.assertEqual((self.tools["loudness"]["reencodes_video"], self.tools["loudness"]["reencodes_audio"]), ("never", "always"))
+ self.assertEqual((self.tools["probe"]["reencodes_video"], self.tools["probe"]["reencodes_audio"]), ("never", "never"))
+
def test_original_preservation_and_roles(self):
for t in self.contract["tools"]:
self.assertFalse(t["mutates_input"], t["name"])