Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Gotchas and things that look right but are wrong
The long form of SKILL.md's "Things that look right but are wrong" and "Gotchas" bullets. SKILL.md keeps one line per item and points here with the anchors below; nothing here is optional reading when the job touches that area — it is the detail, not an aside.
Colour, HDR and Log
HDR and colour
SDR outputs are H.264 tagged BT.709 yuv420p. A source tagged with a BT.601 matrix
(smpte170m, bt470bg; also fcc, smpte240m) has its pixels converted to that BT.709 matrix
by every re-encode (through 2.10.0 only the label changed); color.py --retag bt709 is a
stream copy and still only relabels. Untagged sources are taken as BT.709 and left alone. An RGB
picture (PNG/JPEG still, logo, emoji PNG, PNG sequence, qtrle clip) is converted to BT.709 YUV
explicitly (through 2.10.1 FFmpeg's BT.601 default did it), and grid.py converts each cell by its
own tags. A colour the tools draw (pad and background colours, text, boxes, captions, waveform
colours) is drawn in BT.709 too; through 2.10.2 it was BT.601 under the BT.709 tag (pure red
showed as (255,24,0)). When probe.py reports
bt2020_or_hdr: true (HDR10/PQ, HLG, Dolby Vision, or BT.2020 SDR), every editing script keeps
the output on the 10-bit path (HEVC Main10, source colour tags) so nothing is silently
flattened. Decide with the user: keep HDR (fine for YouTube/phones) or run
color.py --to-sdr first for SDR-only destinations, LUT work or H.264
deliverables. Since 2.0 hdr: true (and its 1.9 twin hdr_signal) is the narrow
fact — a real PQ / HLG / Dolby Vision transfer; bt2020_or_hdr also counts a
wide-gamut SDR file (1.x's hdr), and hdr_format names that in-between case
(BT.2020 SDR). "Not HDR" is not "ready for a BT.709 deliverable": a BT.2020 SDR file
still carries wide-gamut primaries, so an SDR-only client or platform expecting BT.709 needs
color.py --to-sdr first -- export.py only retags BT.709, it does not convert (eval 24 h1).
export.py platform presets are SDR and warn on HDR input.
iPhone .mov files also carry timecode/metadata tracks; scripts map only the
first audio track, so extra tracks are dropped on re-encode. Keep ProRes masters
at source colour: export.py --preset prores does not retag.
Re-encoding an HDR (iPhone, HDR10) source through an SDR path flattens the colours. The scripts keep HDR; if you hand-write ffmpeg (never do — see SKILL.md), do not tag BT.709 on BT.2020 pixels.
Log footage
S-Log, V-Log and C-Log look grey and low-contrast but are tagged SDR. Run
probe.py --analyze; looks_like_log: true means apply the manufacturer's
.cube with color.py --lut before anything else.
Cutting
Keyframe cuts
A lossless cut.py result may start up to one GOP (often 1–10 s) earlier than
requested (start_snapped: true); the script re-encodes automatically when the
deviation exceeds 0.5 s. With --edit-list, a single-segment .mp4/.m4v/.mov copy
keeps the MP4 edit list instead: the pre-roll back to the keyframe is stored but
hidden, so the picture starts at the requested time (edit_list,
stored_preroll_seconds). A player or tool that ignores edit lists shows that
pre-roll. The end still lands on a packet boundary, a few frames long, and
those extra frames can skip. .mkv output has no edit list, and a --segments
join shows each part from its keyframe. On B-frame video an .mp4/.mov
--segments part's end moves to the first keyframe at or after it, within the
same tolerance (segment_end_snap_seconds); no-B-frame and .mkv/.ts parts
are not snapped. Every copy join is checked (frames, steps, each part's sound
against its picture) and re-cut when wrong. Open-GOP HEVC (iPhone "High Efficiency")
always re-encodes a --segments join, even with --tolerance -1, and so does a B-frame .mp4/.mov
join whose later segment starts between keyframes (start it on a keyframe, or write .mkv). If the user insists on lossless output, pass
--tolerance -1 and tell them the cut lands on the nearest earlier keyframe.
Where a copy begins is measured with the seek ffmpeg makes: on B-frame .mp4/.mov
a start a few frames before a keyframe can begin at that keyframe or one GOP
earlier (start_snapped: true either way).
av_start_skew_seconds warns when a copy's sound and picture start apart
(Core Media HEVC once gave 3.7 s of sound with no picture); it is reported, not
repaired: --accurate is the fix, or --edit-list for a lossless .mp4/.m4v/.mov. A -c copy cut on VFR or a
non-keyframe boundary produces a file that "works" but starts on a frozen or
wrong frame — respect the automatic re-encode rather than forcing the copy.
Variable frame rate
probe.py sets variable_frame_rate_suspected when r_frame_rate and
avg_frame_rate disagree (phone and screen recordings). Every re-encoding
script then adds -fps_mode cfr at the source's average rate. That flag is a
whole-file average, and a phone clip at 29.98 against a nominal 30 trips it, so
cut.py re-encodes on it by default and reports packet timestamps sampled in up
to five windows (vfr_check), with a note when they are constant;
--vfr-guard sampled switches to --accurate only when they are irregular or too
few to judge, and --vfr-guard off keeps the copy anyway. Pick the rate
explicitly with fit.py --fps 30|60 when the average is odd (e.g. 23.4 fps from
dropped frames).
Audio
Sync, multicam and drift
A sync or multicam alignment with confidence under 0.3, or an offset larger
than 60 % of the analysis window, is probably wrong: enlarge --analyze-seconds
or find a clap. multicam.py reports one confidence per camera — check all of
them, not just that the command succeeded, before trusting the cut.
sync.py/multicam.py align audio tracks to each other, never lip sync (mouth
movement vs. audio) — there is no face or mouth detection anywhere in this skill.
High confidence means the audio matched well, not that the picture looks right;
whether lip sync is correct needs a look at the video, not the reported offset.
Don't mix files with different frame rates or sample rates in one
cut.py --segments join without re-encoding (--accurate). After sync.py,
verify by running it again on the output: offset (and drift ppm with
--fix-drift) should be ~0. Recordings longer than ~10 minutes from separate
devices: always use --fix-drift.
Loudness and ambience
"Normalised" audio can still clip: check true peak, not just LUFS (check.py
does both). And do not normalise ambience or near-silence to a speech target — a
clip measured at -40 LUFS or below is room tone, wind or nothing; raising it
25 dB raises the noise, not the content. Leave the level, say so, and offer music
or narration.
Do not add a speech gate in front of the measurement either: loudnorm's EBU
R128 integrated loudness already applies the −70 LUFS absolute and −10 LU
relative gates, which drop the same quiet blocks a silencedetect pass would.
A speech-span gate was measured against the whole-file measurement on every
fixture in the repo, including one that is half digital silence, and moved the
result by at most 0.6 LU — inside check.py's own ±1 LU tolerance — for the
cost of a second full decode. That is why loudness.py has no speech-gate flag.
Text and framing
Captions, fonts and text order
Captions burned before a crop/resize land off-frame: frame changes first, then
text. Captions burned at an intermediate size and then upscaled by export.py
come out soft (a 1280x720 source fit to 9:16 is 406x720 until export scales it to
1080x1920) — fit to the delivery size first
(fit.py --width 1080 --height 1920), then caption, then export.
Cue length and timing are handled for you since 1.12: caption.py wraps every
cue to the safe area by measured width at the chosen --size, splits a cue past
--max-lines (default 2) into consecutive cues, holds a cue shorter than
--min-duration (default 1.0 s) — never past the next cue — and shifts
everything by --offset SECONDS. It reports what it changed on one cues: line
and writes the adjusted copy next to the output, never over the file you passed
in. Wrapping needs the input video (the line width comes from its real frame
size); with --write-srt alone only the timing flags apply.
Windows drawtext crashes on some real builds (#100): the drawtext tools resolve a
concrete --font-file by default, which avoids it; if one still crashes, pass
--font-file explicitly. Details: references/ci-platform-pitfalls.md.
Fonts by script
libass and drawtext draw an empty box per character they have no glyph for, and
ffmpeg still exits 0 — a video full of tofu is the classic "it worked" failure.
Since 1.12 caption.py, graphics.py and overlay.py --text detect the script
of the text they are about to draw (Japanese, Chinese, Korean, Arabic, Hebrew,
Devanagari, Bengali, Tamil, Thai, Lao, Cyrillic, Greek) and resolve a font file that covers it,
printing one line — font: /usr/share/fonts/.../wqy-zenhei.ttc (covers ko).
No font for the script is a failed job (kind: input), not a warning.
- What this machine can render:
python3 scripts/_contract.py doctor --json, fieldfonts.scripts(available/missing/unknownper language, with the file it would use). The plain-textdoctorsays the same in one line. - What fontconfig has:
fc-list ":lang=ja" file family(ja,zh-cn,ko,ar,he,hi,th,ru,el). - Install:
apt install fonts-noto-cjk fonts-noto-core, orbrew install --cask font-noto-sans-cjk font-noto-sans-arabic, or point at a file with--font-file(overlay.py,graphics.py) /--fonts-dir(caption.py). - Windows drawtext crashes on some builds (#100) when it has to resolve a font by
family name: pass
--font-fileexplicitly if one does.default_font_file()already resolves a file for you on Windows, so this bites only a hand-built drawtext call. More:references/ci-platform-pitfalls.md. - Han characters alone (no kana, no hangul) are read as Chinese. Japanese or
Korean hanja text with no kana needs
--lang ja/--lang ko(caption.py --languageis the same flag), or"lang"in brand.json. - An explicit
--font, an explicit--font-file, or a font your brand file itself names is always kept, even when fontconfig says it does not cover the script: you get one info line saying so, not a silent substitution. A brand file that never names a font is not a choice — the script still picks one. --fonts-dir(caption.py) adds faces to the search, it does not switch the check off: if nothing in the directory covers the script, one line says so and a covering font is resolved as usual.- No fontconfig is
unknown, notmissing. With nofc-liston PATH (or one that fails) coverage cannot be verified: the job runs with the font as given behind one info line, because libass and drawtext have font backends of their own. Only fontconfig answering "nothing covers this" fails the job. - RTL: libass shapes and reorders Arabic and Hebrew correctly, so
caption.py— which renders every subtitle through libass,subtitles=andass=alike — is right for them by construction.drawtext(overlay.py --text,graphics.py) depends on the build:ffmpeg -versionshowing--enable-libfribidi(and--enable-libharfbuzz) shapes and reorders RTL correctly too; a build without them draws logical order with unjoined letterforms. Nothing in the tools checks this, so on an unknown machine a caption is the safe place for Arabic/Hebrew.
Shaping is a second problem, and it is not the same one. A font that covers
the script still has to be shaped: Devanagari matras are reordered, Thai and
Lao marks re-cluster. drawtext does bidi and Arabic joining correctly on a build
compiled with fribidi (so Arabic and Hebrew are already right today), but it
never reorders or re-clusters, because it does not use harfbuzz on any build.
libass does. Since 1.15 graphics.py therefore renders a shaping script
(hi bn ta te kn ml gu pa si th lo km my) through libass automatically — one
generated <output>_gfx.ass next to the file, text_renderer: "ass" in the
JSON — and --text-render auto|ass|drawtext overrides it; --text-render drawtext with such a script is a refusal, not a wrong frame. Latin, CJK and
Arabic output is unchanged. overlay.py --text has no ASS route yet and refuses
a shaping script, naming caption.py/graphics.py.
Emoji
Colour emoji need either a libass that can draw colour glyphs or a directory of PNGs — and on most builds only the second one exists.
- An installed colour emoji font proves nothing. Noto Color Emoji installs
cleanly on Ubuntu and libass on the distro ffmpeg still renders a monochrome
outline (
Glyph 0x1F389 not found, broken font? Trying all charmaps). The only honest test is a render, which is whatdoctorruns:doctor --json→.fonts.emoji(mode:color/png/mono/none, pluslibass_colorfrom that render).contract --json --staticis the way to get the contract without running it;doctoritself always probes. - drawtext cannot load an emoji font at all. Not a degraded render: a
CBDT/sbix face fails filter initialisation outright (
Could not set font size to 48 pixels: invalid library handle, and at the font's own strikeMonocromatic (1bpp) fonts are not supported.) and no file is written. That is why the colour route is a PNG overlay, never a drawtext font. - PNGs:
--emoji-assets DIR, a directory of files named by code point in the Twemoji/Noto convention — lowercase hex joined by-:1f389.png,1f1ef-1f1f5.png,1f469-200d-1f4bb.png. Twemoji'sassets/72x72(CC-BY 4.0) and Noto Emoji'spng/128(OFL/Apache-2.0) are the two people already have. The skill never downloads anything: a missing directory iskind: input, never a silent fetch. The same directory can come frombrand.json(styles.caption.emoji_assets) orFFMPEG_SKILL_EMOJI_ASSETS. - Mixed lines are the normal case. The text stays in the ASS with an
invisible placeholder reserving exactly the emoji's box, and the PNG is
composited on top after the
ass=filter — libass keeps the shaping and the karaoke, including inside a\kfrun (the placeholder is its own zero-duration segment). - What counts as an emoji. A cluster only starts at an emoji base.
U+200D ZWJ and U+200C ZWNJ are ordinary Indic/Persian orthography (
क्षis ka + virama + ZWJ + ssa) and are glue only between two emoji; a VS16 or a skin-tone modifier after a plain letter is not an emoji either. U+FE0E (VS15) asks for the character, so❤︎is text and❤️is an emoji. - Placement tolerance. The position comes from the same averaged em table the wrap uses, so an emoji at the start or end of a line is exact and one in the middle of a Latin line drifts by the accumulated rounding of the characters before it — measured at 17 px, 0.28 em, about 3 % of the line width on a 24 px caption over a 1280-wide frame, always landing inside the reserved gap rather than on a glyph. It never leaves the safe area. An RTL line (Arabic, Hebrew) is laid out right-to-left, so the position is measured from the rendered end of the line, not the logical prefix.
- Degraded paths are honest, not silent.
mode: monomeans libass draws whatever glyph the text font has (it has a fallback chain), exits 0, and says so in awarning:line and innotes— so agraphics.pyrun that would otherwise use drawtext, which loads one font file and has no fallback chain, is routed to libass rather than reporting a monochrome glyph it would actually draw as an empty box; a run pinned to--text-render drawtextdegrades tononeand says so.mode: nonestrips the cluster from the drawn text (a missing decoration must not fail a delivery) but refuses agraphics.pytemplate whose text is only emoji — that frame would be blank.--emoji-max(default 60) caps the overlay count;--emoji-max 0means none. --animatemoves the PNG too. The overlay gets a matching alpha fade, so the emoji arrives and leaves with the line instead of popping in.overlay.py --texthas no PNG route:--emoji png|colorthere namescaption.py/graphics.pyinstead.
Reframing, fps and duration
--fit crop to reach 9:16 from 16:9 throws away 70 % of the width: a wide shot
loses people at the edges. Check the sheet; pad (bars), --crop-x/--crop-y
toward the subject, or a reframe is often the honest answer — a silent centre
crop is a guess, not a decision.
Conforming 60 fps to 30 halves the motion samples: fine for a talking head, visibly choppy for sports, gaming, drone pans. Keep 60 when the platform allows.
"Make it 60 seconds" on a 3-minute talk by speed change is unwatchable (3×); by
trim it drops two thirds of the words. Ask which, or propose a highlight cut with
scenes.py.
Platform safe zones
Every vertical app draws its own UI over the delivery: TikTok covers roughly the
bottom 22 % (description and caption block), the right 14 % (like/comment/share
column) and the top 10 % (status bar and tabs); Reels 20/12/8 %; Shorts 18/12/6 %.
The feed destinations (YouTube, X, LinkedIn, Facebook) have no persistent overlay
and use the conventional 5 % title-safe border instead. A file can pass every
check.py row and still be unreadable because the caption sits under the
description.
The fractions live in one table (scripts/_platforms.py). caption.py --platform NAME, graphics.py --platform NAME and overlay.py --platform NAME take their
margins from it, render.py --template NAME passes it to all three, and
look.py --safe tiktok shades the zones on a frame or contact sheet so you can
see what the app covers. An explicit --margin always wins.
Dimensions and rotation
yuv420p needs even width/height; fit.py and export.py round to even values
automatically. Phone footage often carries a rotation tag. probe.py reports
the stored width and height plus that rotation. A tool that places or measures
a rectangle (fit.py, crop.py, cropdetect.py, broll.py, overlay.py --scale-percent / --platform, look.py --safe) uses the displayed size,
the two swapped for 90° and 270°, because ffmpeg applies the tag before the
filter graph.
Planning
Highlights
scenes.py --highlights defaults to the loudest scenes (--rank-by audio): a
quiet but important moment (a confession, a punchline landing in silence) is
skipped, and pure crowd noise or a mic bump can outrank it. --rank-by duration
picks the longest unbroken scenes instead. Neither is "the best parts" — check
the contact sheet (--sheet) before treating the picks as final.
Chaining and speed
Anything chained by hand through three re-encodes should be one render.py
project instead, so the plan is one file and the user can change one number.
Re-encodes use x264 medium; for long files add --preset veryfast to
intermediates and keep the default for the final export. render.py does that by itself since
2.15 when its project ends in an encoding export (--intermediate-preset medium turns it off). On Apple Silicon,
--hw (or FFMPEG_SKILL_HW=1 for the machine) moves h264/hevc/prores encodes to
VideoToolbox: 2–7× faster, 1.1–2.9× the bytes at the same quality (1.9–3.5× on HDR). The variable
leaves export.py's delivery presets on the CPU, which is the usual split (GPU
intermediates, CPU delivery); render.py --hw puts the export on the GPU too. A build with no
libx264/libx265 (an LGPL ffmpeg) has no CPU delivery to keep: its tools and presets use
VideoToolbox by themselves, hw.source says auto.
Check encoder/hw in the result: a job the GPU refuses (H.264 wider than 4096)
was re-encoded on the CPU.
Audio-only files
Audio files are a first-class input, not a special case. probe.py, cut.py,
silence.py, loudness.py, audio.py, sync.py and check.py --platform podcast all accept WAV, FLAC, MP3, M4A/AAC, OGG and Opus (any container ffmpeg
can read) and write the codec that fits the output extension, so the same
commands work with talk.wav in place of talk.mp4. What changes:
- The output extension picks the format:
-o out.mp3converts,-o out.wavkeeps PCM,-o out.m4awrites AAC.audio.py in.wav -o out.mp3with no other flag is a plain conversion. cut.pystream-copies audio too, so trims land on a packet boundary (precision: packet, a few ms; the JSON reportsduration_error_ms). Pass--accuratefor a sample-exact trim:precision: samplewhen the output is PCM or FLAC,codec_framewhen a lossy codec (AAC, MP3, Opus) frames it again. A.wavoutput is always PCM, never AAC packets inside a WAV.join.pyjoins audio-only clips as audio (acrossfadeor a butt join) at one sample rate and channel layout; the output must have an audio extension. Video and audio clips cannot be mixed in one join.- An audio extension on a video input (
audio.py talk.mp4 -o talk.wav,cut.py talk.mp4 --start 1:00 --end 2:00 -o part.wav) extracts the audio; the output has no video stream.audio.py --audio-stream Npicks a track whenprobelists several underaudio_streams. Look: not neededin the report;Check:still applies for loudness (check.py file.wav --platform podcastmeasures LUFS and true peak).- Scripts that need a picture (
fit,caption,overlay,graphics,color,export,scenes,look) refuse an audio file with "input has no video stream". Say so instead of forcing a video wrapper.
Audio-only recipes
| User says (audio file) | Do |
|---|---|
| "normalise this WAV to -14 LUFS", "podcast levels" | loudness.py talk.wav -I -14 --tp -1 -o talk_norm.wav (-I -16 --tp -1.5 podcast) |
| "remove the silence from this recording" | silence.py talk.wav -o talk_tight.wav |
| "clean up the noise in this M4A" | audio.py talk.m4a --voice -o talk_clean.m4a (speech) or --denoise |
| "convert this WAV to MP3" | audio.py talk.wav -o talk.mp3 |
| "trim this audio from 00:30 to 02:00" | cut.py talk.wav --start 0:30 --end 2:00 -o talk_cut.wav (--accurate for sample-exact) |
| "join these recordings", "intro + episode + outro" | join.py intro.wav episode.m4a outro.wav -o full.flac (--transition none for a butt join) |
| "extract the audio from the video" | audio.py talk.mp4 -o talk.wav (--voice -o talk.m4a to clean it on the way) |
| "compress / limit / gate the voice" | audio.py talk.wav --compress --comp-threshold -20 --comp-ratio 4 --limit --limit-ceiling -1 -o talk_dyn.wav |
| "is this loud enough for Apple Podcasts?" | check.py talk.m4a --platform podcast |