Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

feat: text people can see (emoji, complex-script shaping in graphics.py, ' and % kept, caption wrap without orphans, label rule) (#226)

claude[bot]628e58ff2f6e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 235aa4b..f090373 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,48 @@
(nothing yet)
+## 1.15.0
+
+### Added
+
+- **Emoji in captions and titles.** `caption.py`, `graphics.py` and `overlay.py` gain `--emoji auto|color|png|mono|none`, `--emoji-assets DIR`, `--emoji-scale FLOAT` and `--emoji-max N`. The colour route is a **PNG overlay**, not a font: drawtext cannot load a CBDT/sbix emoji face at all (it fails filter initialisation — `Could not set font size to 48 pixels: invalid library handle`, and at the font's own strike `Monocromatic (1bpp) fonts are not supported.` — and writes no file), and an installed colour emoji family proves nothing either (Noto Color Emoji installs cleanly on builds whose libass still draws a monochrome outline). `--emoji-assets DIR` is a directory of PNGs named by code point in the Twemoji/Noto convention (`1f389.png`, `1f1ef-1f1f5.png`, `1f469-200d-1f4bb.png`), also read from `brand.json` (`styles.caption.emoji_assets`) and `FFMPEG_SKILL_EMOJI_ASSETS`. **Nothing is ever downloaded**: a missing directory is `kind: input` naming the two sets people already have, never a fetch. In a caption the text keeps its place in the generated ASS with an invisible placeholder reserving exactly the emoji's box (measured: U+2588 FULL BLOCK is 0.66–0.83 em depending on the face, so the gap is reserved with alpha-hidden `\fsp` spacing instead, which is exact in every face this repo resolves — including inside a `\kf` karaoke run, where the placeholder is its own zero-duration segment), and each PNG is composited after the `ass=` filter. `caption.py`/`graphics.py --json` gain an `emoji` block (`mode`, `count`, `clusters`, `assets`, `missing`, `overlays`); `--json-brief` carries `mode` and `count`. Emoji also count as a full em in the wrap, so an emoji-heavy line no longer overflows the safe area, and a wrap never breaks inside a ZWJ sequence, a flag pair or a skin-tone modifier.
+- **`doctor` answers what this machine can do with emoji**: `doctor --json` → `.fonts.emoji` (`mode`, `color_font`, `color_font_file`, `libass_color`, `assets`, `detail`, `fix`), where `libass_color` comes from a ~80 ms **render probe** (one 64×64 frame with an emoji cue through `subtitles=`, chroma-tested) rather than a font listing. `contract --json --static` and every other static/JSON-only path skip the probe and report `libass_color: null` (`doctor` itself always probes -- it has no `--static`). Informational like the rest of `fonts`: it never moves `ok` or a tool's `usable`. The plain-text `doctor` gains one clause on its existing `fonts:` line.
+- **`graphics.py --text-render auto|ass|drawtext` and `--write-ass PATH`.** drawtext does bidi and Arabic joining correctly on a build with `--enable-libfribidi`, so Arabic and Hebrew were already right; what it cannot do on **any** build is reorder and re-cluster — Devanagari matras, Thai/Lao mark stacking — because it does not use harfbuzz even in an `--enable-libharfbuzz` build. `auto` (the default) therefore renders a shaping script (`hi bn ta te kn ml gu pa si th lo km my`) through libass, writing `<output>_gfx.ass` from the geometry the templates already computed (new private module `scripts/_ass_overlay.py` — the public tool count is unchanged at 42) and reporting `text_renderer`, `script` and `ass` in the JSON. Latin, CJK and Arabic frames are pixel-identical to 1.14.0 (the *command* is not: every drawn label now goes through `textfile=…:expansion=none` instead of `text=…`, which is what makes `'` and `%` survive). `--text-render drawtext` with such a script is a refusal naming the script, never a wrongly shaped frame.
+- **`bn` (Bengali), `ta` (Tamil) and `lo` (Lao)** join the scripts the tools detect and resolve fonts for: `doctor --json` `.fonts.scripts` has three more entries.
+
+### Fixed
+
+- **`'` and `%` survive the drawn text** (eval 14). `overlay.py --text "it's 100% done"` rendered `its 100 done`: `escape_drawtext()` dropped both characters because neither has a backslash escape that is safe in every call shape. Drawn text now reaches drawtext as `textfile=<path>:expansion=none`, so the filter-graph parser never sees it and drawtext's own `%{…}` scanner is off — `'`, `%`, `:`, `,`, `[`, `]`, `;` and `\` all render verbatim. `escape_drawtext()` remains, for the font-**name** fallback only. `caption.py` renders through libass and already kept both characters; 1.15.0 adds the regression lock on that path.
+- **No one-character orphan lines, and better breaks** (eval 14: `th1`, `dl3`, `dl1`). `caption.py`'s wrap gains two post-passes: a last line that is a single atom narrower than 1.1 em pulls the previous line's last atom down (the stranded Thai `ล` and Japanese `行`), and for spaced scripts the break is moved to the one that minimises the widest line instead of filling greedily (`"A third line the tool times for me"` now breaks between phrases). The line count never changes, non-spaced scripts keep greedy wrapping, and `report_layout()`'s one `cues:` line reports `rebalanced`.
+- **`overlay.py --text` refuses a shaping script** instead of drawing it wrongly, naming `caption.py` and `graphics.py`. The ASS route for `overlay.py` is 1.16.0.
+- **Every emoji PNG stayed on screen for one frame.** The overlay chain carried `eof_action=pass`, which switches off overlay's default "hold the last frame of the secondary input" — and a PNG input is a single frame at pts 0, so the asset was composited on frame 0 and vanished for the rest of the cue (the first `docs/demos/captions_emoji.gif` shipped an empty reserved gap as the feature's own advertisement). The chain now uses the default `eof_action=repeat` and lets `enable='between(t,start,end)'` confine the asset to its cue. The regression test renders a cue and samples the planned pixel in the **middle** of it and again after it ends.
+- **A ZWJ no longer starts an emoji cluster.** U+200D ZWJ and U+200C ZWNJ are ordinary Indic/Persian orthography, so `क्‍ष` (ka + virama + ZWJ + ssa) was detected as the emoji cluster `94d-200d`: a Hindi cue was reported as containing emoji, warned about as monochrome, and on the PNG route had a real letter replaced by an invisible gap. A cluster may now only *start* at an emoji base (pictograph, regional indicator, or a `0-9 # *` that is actually followed by U+20E3); joiners, VS16 and skin-tone modifiers bind rightwards only. U+FE0E (VS15, text presentation) now means "not an emoji", so `❤︎` is drawn as a character.
+- **Emoji in an RTL caption land in the reserved gap.** libass lays Arabic and Hebrew out right-to-left, so measuring the logical prefix from the left edge put the PNG on top of the text. The position is now measured from the rendered end of the line for `BIDI_SCRIPTS`, in `caption.py` and `graphics.py` alike.
+- **`graphics.py` no longer claims `mode: mono` while drawing tofu, and `--emoji none` actually strips.** drawtext loads exactly one font file and has no fallback chain, so a monochrome emoji there is an empty box; an `auto` run with monochrome emoji is routed through libass (which does have a fallback chain) and a run pinned to `--text-render drawtext` degrades to `none` and says so. `--emoji none` now removes the clusters from `--name/--title/--subtitle/--text/--top/--bottom` the way `caption.py` does.
+- **Drawn text goes to a private, cleaned-up temp file.** 1.15.0's first shape wrote every label into a world-shared, predictable `/tmp/ffmpeg-skill-text` created with `makedirs(exist_ok=True)` — an attacker-creatable directory with content-addressed names (a planted symlink was an arbitrary-file overwrite), a `PermissionError` out of filter construction for the second user of a shared box, and a directory that was never cleaned. The directory is now `tempfile.mkdtemp()` (0700, unguessable, removed at exit), the file is opened `O_NOFOLLOW` at 0600, and it is written only when the command naming it actually runs — so `--dry-run` writes nothing (the plan names the intended path) and the ASS route, which never reads the file, writes nothing either.
+- **`--emoji-max 0` means none** (`int(args.emoji_max or 60)` swallowed it) in `caption.py` and `graphics.py`.
+- **`--animate` fades the emoji with the line.** The PNG overlay gets an alpha fade matching the text's `\fad` (and `graphics.py`'s 0.3 s template fade) instead of popping in at full opacity.
+- **The ASS route keeps `{`, `}` and `\`.** `ass_text()` deleted all three while the drawtext route now renders them verbatim, so the same title produced two different pictures depending on an automatic route choice. They are escaped (`\{`, `\}`, and a zero-width space where a backslash would otherwise start `\N`/`\n`/`\h`) rather than dropped — user text still cannot open an override block.
+- **`doctor(detect=…)` is gone.** It was dead (`--static` only applies to `contract`, which never calls `doctor()`), and the test that exercised it went through the Python API rather than any CLI path; the replacement test asserts what is actually reachable, `contract --json --static`.
+
+### Known
+
+Carried into the next release, all P2 from the 1.15.0 review:
+
+- `emoji_placeholder()`'s docstring is not a raw string, so the literal `\fsp`, `\r` and `\kf` in it become control characters in `help()`/`pydoc` output.
+- `demos/build.py` draws its placeholder PNGs with `geq`, which needs a GPL build; the tests' own `_emoji_assets()` uses plain `color=…,format=rgba` and the demo could do the same.
+- `emoji_support()` has two branches that both set `mode = "none"`; the `not fc_answered` one is dead code (the behaviour is right, the branch reads as if it were doing something).
+- `text_overlay_ass(..., fonts_dir=…)` takes a parameter it never uses.
+- `caption.py`'s no-emoji command reordered harmlessly (`-map 0:v:0 -map 0:a:0 -vf …` → `-map 0:v:0 -vf … -map 0:a:0`) in the one path the release promises not to touch.
+
+### Docs
+
+- SKILL.md gains the report-label rule (a refusal that still delivers something is `Failed:`; the alternative goes in `Notes:`) and two gotcha lines, and drops the zh/ko restatement of the report shape and one redundant sentence to stay inside the 30 000-byte budget — now enforced by `tests/test_contract.py::test_skill_md_stays_under_the_30kb_budget` rather than by convention.
+- `references/gotchas.md` gains an `### Emoji` section and two paragraphs on shaping under `### Fonts by script`; `docs/contract.md` documents `fonts.emoji`, the new flags and the new result keys (additive — `contract_version` stays 1.0); `references/scripts.md` updates the caption/graphics/overlay tables; `docs/design-decisions.md` pins the three calls behind this release; `docs/roadmap.md` marks 1.15.0 shipped and corrects the drawtext claim; the README gains two feature bullets and a pending-eval row.
+- `demos/build.py` and `docs/demos.md` gain `captions_emoji` (colour PNGs against the monochrome fallback) and `lower_third_hindi` (`--text-render drawtext` against `--text-render ass`, the release's headline picture).
+- `evals/agent_prompts_24.json` gains `em1`–`em4` and `sh1`–`sh2` with their fixtures, `evals/grade_runs_24.py` the honesty rule for the emoji prompts, and `evals/trigger/prompts.json` two more prompts (40: an emoji caption request that must trigger, an "emoji sticker pack as PNGs" near-miss that must not).
+- `tests/fixtures/mcp_tools.json` regenerated: `caption`, `graphics` and `overlay` gain `emoji`, `emoji_assets`, `emoji_max` and `emoji_scale`; `graphics` also gains `text_render` and `write_ass`. Nothing was removed, renamed or made required.
+
## 1.14.0
_Automated release: version and notes generated from pull requests merged since 1.13.0._
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index cb14256..7da0010 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -72,6 +72,10 @@ would have caught the original bug isn't done yet.
table, the contract table, gotchas), `SKILL.md`, `references/scripts.md`,
`docs/contract.md` and `CHANGELOG.md`. The README is read by people who never
open SKILL.md, so a feature that only SKILL.md knows about is half shipped.
+- `SKILL.md` must stay under **30,000 bytes** — it is loaded into every session.
+ Since 1.15 this is a test
+ (`tests/test_contract.py::test_skill_md_stays_under_the_30kb_budget`), not a
+ convention: adding a line there means trimming one, and the PR says which.
- A `feat` PR adds at least one before/after demo to `demos/build.py` and
`docs/demos.md` (the page is generated: add the demo to the table in
`demos/build.py`, then run `python3 demos/build.py --docs`). A feature nobody
diff --git a/README.md b/README.md
index f963c43..4cf67ca 100644
--- a/README.md
+++ b/README.md
@@ -39,7 +39,7 @@ npx ffmpeg-skill
</tr>
</table>
-Left half is the input, right half is what the command produced. **[All 51 before/after demos, with the exact command under each one →](docs/demos.md)** — all of it generated from synthetic footage by `python3 demos/build.py`, so you can rebuild every frame of it yourself.
+Left half is the input, right half is what the command produced. **[All 53 before/after demos, with the exact command under each one →](docs/demos.md)** — all of it generated from synthetic footage by `python3 demos/build.py`, so you can rebuild every frame of it yourself.
`ffmpeg-skill` is an [Agent Skill](https://docs.anthropic.com/en/docs/agents-and-tools/agent-skills) for Claude Code, Cursor, Codex and any agent that reads `SKILL.md`. It teaches the agent a fixed workflow (probe → edit losslessly where possible → check → verify) and ships **42 tools** that do the actual work with `ffmpeg` / `ffprobe`: cut, join, silence removal, fit to duration and aspect, captions and karaoke, overlays and motion graphics, HDR → SDR and LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, whole-edit project rendering, batch folders. Every tool is also an MCP tool, and the whole set is described by a machine-readable contract.
@@ -236,7 +236,7 @@ These are the rules the skill file gives the agent and the code enforces.
|---|---|
| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area by measured width with `--max-lines`/`--min-duration`/`--offset`; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; optional local transcription |
| `overlay.py` | Logos, watermarks and titles with position, time range, opacity, fades; `--platform NAME` keeps them clear of that destination's UI; `--video` for picture-in-picture, `--chromakey` for green-screen compositing |
-| `graphics.py` | Lower-thirds, title cards, chapter chips, progress bars, countdowns, corner bugs, social stickers, opening hook cards and meme captions drawn by FFmpeg from a brand kit; `--platform NAME` keeps them inside that destination's safe zone |
+| `graphics.py` | Lower-thirds, title cards, chapter chips, progress bars, countdowns, corner bugs, social stickers, opening hook cards and meme captions drawn by FFmpeg from a brand kit; `--platform NAME` keeps them inside that destination's safe zone; `--text-render` routes shaping scripts through libass and `--emoji-assets` composites colour emoji |
| `color.py` | HDR10 / HLG / Dolby Vision → SDR BT.709 tone mapping, DV layer stripping, 3D LUT (.cube), colour-tag rewriting, typed primary correction (exposure/contrast/saturation/gamma/white balance/lift-gain/levels/curves) |
**Delivery**
@@ -361,6 +361,8 @@ The short list for humans. The agent-facing version, with the reasoning, is the
- **Loudness targets.** −14 LUFS / −1 dBTP for YouTube and social platforms (the `loudness.py` default), `-I -16 --tp -1.5` for podcasts, `-I -23` for broadcast. A clip measured at −40 LUFS or below is room tone, not content; raising it raises the noise. Check true peak as well as LUFS: `check.py file --platform podcast` measures both.
- **Frame changes first, text second.** Captions and overlays burned before a crop or resize end up off-frame. Reframe, then caption.
- **Cropping 16:9 to 9:16 discards 70 % of the width.** `fit.py --fit crop` centres by default; pass `--crop-x`/`--crop-y` toward the subject, or pad with `--fit pad --pad-fill blur`. Look at the contact sheet before deciding.
+- **Emoji in captions and titles (1.15).** `caption.py`/`graphics.py --emoji-assets DIR` composites a PNG per emoji (Twemoji/Noto naming, `1f389.png`) on top of the text, because drawtext cannot load a colour emoji font at all and an installed one does not prove libass will draw it in colour — `doctor --json`'s `fonts.emoji` answers that from a render probe. Without assets the run still succeeds and reports `mode: mono`. Nothing is ever downloaded.
+- **Indic and Thai text shaped correctly in titles and lower-thirds (1.15).** `graphics.py` renders Devanagari, Bengali, Tamil, Thai and Lao through libass automatically (`text_renderer: "ass"`), because drawtext never reorders matras or re-clusters marks; Arabic and Hebrew were already correct on a fribidi build. `--text-render drawtext` with such a script is refused, never rendered wrongly.
- **Non-Latin text picks a font by script (1.12).** Japanese, Chinese, Korean, Arabic, Hebrew, Devanagari, Thai, Cyrillic and Greek cues, titles and overlays resolve a font file that covers them automatically, and a machine with no such font fails the job (`kind: input`) instead of rendering boxes. `doctor --json`'s `fonts.scripts` says which languages this machine can render; `--lang ja|ko` disambiguates Han-only text; an explicit `--font`/`--font-file` is always kept.
- **Silence detection finds nothing?** The default threshold is −35 dBFS. The tool prints a hint with the track's measured level; raise the threshold (`silence.py --threshold -25`) or shorten `--min-silence`.
- **Sync results carry a confidence.** Below 0.3, or an offset near the edge of the analysis window, is probably wrong: enlarge `--analyze-seconds` or find a clap. Recordings over ten minutes from separate devices need `sync.py --fix-drift`.
@@ -400,6 +402,7 @@ type on every OS.
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| _pending_ | **1.15.0 — shipped, eval pending.** Eval 16 grades it on the 82-prompt set (the 76 plus `em1`–`em4`/`sh1`–`sh2`: emoji captions and title cards, a Hindi lower-third, a Thai lower-third + caption) and re-runs the caption/graphics prompts in every script the set covers. No numbers are claimed until that run exists |
| **76 / 76** | 1.14.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 26 runs, check.py re-run on every delivery output): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 by regex (75/76 by Opus: one Spanish report with three English labels), visual check 18/18, trigger set 38/38, Opus quality mean 4.58. The delivery templates did their job: 12 of 13 delivery requests went through `render.py --template`, finished in one encode (was 3 of 7) and all 13 pass their platform check (was 7 of 8). Tokens per run flat at 73.4k. Details in `evals/results/iteration-15.json` |
| **76 / 76** | 1.13.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 76 prompts: 18 in Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish and Italian, and 8 delivery requests (TikTok, Reels, Shorts, LinkedIn, Douyin, podcast): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 across seventeen languages, visual check 18/18, trigger set 38/38, Opus quality mean 4.65 over the 26 new runs. One real defect found: Hindi through `graphics.py` (drawtext) comes out wrong-shaped even though the font covers Devanagari; captions through libass are fine (queued for 1.15.0). Four delivery runs spent a second encode for loudness, which 1.14.0's templates address. Tokens per run flat at 72.3k. Details in `evals/results/iteration-14.json` |
| **50 / 50** | 1.12.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 50 prompts with two each in Chinese, Korean, Spanish, Portuguese, French, German and Arabic: routing 50/50, honest refusals and failures 50/50 with 0 false successes and 0 raw ffmpeg calls, report format 50/50, user's language 50/50 across nine languages, visual check 13/13, trigger set 29/29, Opus quality mean 4.83. Every non-Latin caption and lower-third picked a covering font by itself and rendered real glyphs (Arabic shaped and right-to-left); tokens per run unchanged at 72.3k. Details in `evals/results/iteration-13.json` |
diff --git a/SKILL.md b/SKILL.md
index bb023ba..8904312 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -149,7 +149,7 @@ Audio is a first-class input: `probe.py`, `cut.py`, `silence.py`, `loudness.py`,
## Report format
-Reply in the language the request itself is written in — the user's own sentences, not a language the request talks about (a request for subtitles in another language is still answered in the language it was written in) and not the language of a tool's error text or file names. Any language works the same way. Keep the field labels (`Done:`, `Steps:`, `Check:`, `Look:`, `Notes:`) in English: they read like log fields across languages. Everything around them — the sentences, any question, any explanation of a judgement call — is in the user's language. Never default to English because the tool names are English, and never drift because the job was short or the report is a failure: a one-line "file does not exist" is in the request's language too. A mid-conversation switch follows the user's latest message. English `Done:`/`Steps:` sentences with one word of the user's language in `Notes:` is an English report — the descriptions are in the user's language even when the values are technical.
+Reply in the language the request itself is written in — the user's own sentences, not a language the request talks about (a request for subtitles in another language is still answered in the language it was written in) and not the language of a tool's error text or file names. Keep the field labels (`Done:`, `Steps:`, `Check:`, `Look:`, `Notes:`) in English: they read like log fields across languages. Everything around them — the sentences, any question, any explanation of a judgement call — is in the user's language. Never default to English because the tool names are English, and never drift because the job was short or the report is a failure: a one-line "file does not exist" is in the request's language too. A mid-conversation switch follows the user's latest message. English `Done:`/`Steps:` sentences with one word of the user's language in `Notes:` is an English report — the descriptions are in the user's language even when the values are technical.
Finish every job with this shape (numbers from `--json` or `probe.py`/`check.py`, not memory):
@@ -171,8 +171,6 @@ Look: final_sheet.png(字幕はセーフエリア内、ロゴは右上)
Notes: 元は VFR だったので 30 fps に揃えた。音声はモノラルだったのでステレオにした
```
-Same shape in every other language, labels still English — zh: `Done: final.mp4 — 59.98 秒、1080x1920、30 fps、H.264` / `Steps: 0:12-1:12 剪切 -> 9:16 裁剪 -> 字幕 -> Reels 导出`; ko: `Done: final.mp4 — 59.98초, 1080x1920, 30 fps, H.264` / `Steps: 0:12-1:12 컷 -> 9:16 크롭 -> 자막 -> Reels 내보내기`.
-
Keep it to those five lines plus anything the user must decide. Never report success without the output probe; never describe a fix you did not run.
When a step fails, replace `Done:` with `Failed:` and keep the rest honest:
@@ -185,7 +183,7 @@ Look: not needed (nothing written)
Notes: send a valid .cube, or say if you want the clip left as is
```
-A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure still gets all five labels, never prose headings. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
+A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure still gets all five labels, never prose headings. A refusal that still delivers something is `Failed:` — the label answers the request as asked; the alternative goes in `Notes:`. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
Every script prints `{"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool | timeout | verification | interrupted, "message": ...}}` with `--json` and exits non-zero; quote the message, never paraphrase it.
@@ -200,6 +198,8 @@ One line each, each enough to act on; open the linked `references/gotchas.md` se
- Sync/multicam `confidence` under 0.3 (or a huge offset) is probably wrong — check every camera; these align audio, never lip sync. Details: [#sync-multicam-and-drift](references/gotchas.md#sync-multicam-and-drift)
- "Normalised" audio can still clip (check true peak), and ambience at -40 LUFS or below must never be raised to a speech target. Details: [#loudness-and-ambience](references/gotchas.md#loudness-and-ambience)
- Captions burned before a crop/resize land off-frame; burned small then upscaled by `export.py` they come out soft — fit to the delivery size first. Details: [#captions-fonts-and-text-order](references/gotchas.md#captions-fonts-and-text-order)
+- Emoji need `--emoji-assets DIR` (a PNG per glyph) to render in colour; without it they come out monochrome and the run says so. Details: [#emoji](references/gotchas.md#emoji)
+- `graphics.py` renders Devanagari, Bengali, Tamil and Thai through libass automatically — drawtext cannot shape them.
- Non-Latin text picks a font by script since 1.12; `doctor --json` `fonts.scripts` says which languages this machine renders; no font = failed job. Details: [#fonts-by-script](references/gotchas.md#fonts-by-script)
- `--fit crop` 16:9 → 9:16 throws away 70 % of the width, 60→30 fps halves the motion, and "60 seconds" by speed or by trim are different answers — say which and why. Details: [#reframing-fps-and-duration](references/gotchas.md#reframing-fps-and-duration)
- TikTok/Reels cover the bottom fifth and the right column with their own UI — templates keep text out of those zones; `look.py --safe tiktok` shows them. Details: [#platform-safe-zones](references/gotchas.md#platform-safe-zones)
diff --git a/demos/build.py b/demos/build.py
index 26bf65c..89ef4bb 100644
--- a/demos/build.py
+++ b/demos/build.py
@@ -229,6 +229,10 @@ def build_fixtures(force=False):
"0:02-0:04 الترجمة محروقة عبر libass",
"0:04-0:06 بلا سحابة وبلا مفاتيح",
"0:06-0:08 يكفي python3 و ffmpeg"],
+ "cues_emoji.txt": ["0:00-0:02 Shipping day 🎉",
+ "0:02-0:04 Nice work 👍 everyone",
+ "0:04-0:06 Ship it 🚀 now",
+ "0:06-0:08 Done ✅"],
"cues_pop.txt": ["0:00-0:02 word by word",
"0:02-0:04 the karaoke highlight tracks the beat",
"0:04-0:06 pop scales each cue in",
@@ -383,6 +387,25 @@ def build_fixtures(force=False):
# --------------------------------------------------------------------------- comparison + preview
+# The package ships NO emoji art: Twemoji is CC-BY 4.0 and Noto Emoji OFL/Apache-2.0, and a
+# gallery does not need to redistribute either. The demo draws its own coloured placeholder PNGs
+# with ffmpeg, named by code point exactly as --emoji-assets expects, so `--emoji-assets DIR`
+# is demonstrated end to end while the real glyphs stay the user's own download.
+EMOJI_PLACEHOLDERS = {"1f389": "orange", "1f44d": "gold", "1f680": "tomato", "2705": "limegreen"}
+
+
+def build_emoji_placeholders():
+ d = FIX / "emoji"
+ d.mkdir(parents=True, exist_ok=True)
+ for name, colour in EMOJI_PLACEHOLDERS.items():
+ path = d / ("%s.png" % name)
+ if not path.exists():
+ ffmpeg("-f", "lavfi", "-i", "color=c=%s:s=72x72:d=0.04" % colour,
+ "-vf", "format=rgba,geq=r='r(X,Y)':a='if(lt((X-36)*(X-36)+(Y-36)*(Y-36),34*34),255,0)'",
+ "-frames:v", "1", str(path))
+ return d
+
+
def _side(idx, label, font, seconds):
"""One half of the side-by-side: letterboxed into a fixed cell so a 9:16 'after' and a
16:9 'before' still stack cleanly, held on its last frame if it is the shorter of the two."""
@@ -530,6 +553,40 @@ def demo_captions_pop_karaoke(ctx):
"--highlight-color", "#39ff88"))
+def demo_captions_emoji(ctx):
+ """Colour emoji (PNG overlay) against the monochrome fallback the same machine gives you."""
+ assets = build_emoji_placeholders()
+ before = ctx.path("before.mp4")
+ after = ctx.path("after.mp4")
+ ctx.script("caption.py", FIX / "motion.mp4", "--text", FIX / "cues_emoji.txt", "--size", "30",
+ "--bold", "--margin", "40", "--emoji", "mono", "-o", before)
+ ctx.script("caption.py", FIX / "motion.mp4", "--text", FIX / "cues_emoji.txt", "--size", "30",
+ "--bold", "--margin", "40", "--emoji-assets", assets, "-o", after)
+ ctx.note("the left side is what this ffmpeg draws without --emoji-assets (monochrome or "
+ "nothing); the right side composites one PNG per cluster")
+ return before, after
+
+
+def demo_lower_third_hindi(ctx):
+ """The eval-14 defect and its fix: drawtext cannot reorder Devanagari matras, libass can."""
+ before = ctx.path("before.mp4")
+ after = ctx.path("after.mp4")
+ font = font_for_script("hi")
+ # The "before" is raw drawtext on purpose: graphics.py now REFUSES --text-render drawtext for a
+ # shaping script, so the wrong frame can only be produced outside the tool -- which is the point.
+ ffmpeg("-i", str(FIX / "motion.mp4"), "-t", "5", "-vf",
+ "drawtext=text='प्रिया शर्मा':fontfile=%s:fontsize=40:fontcolor=white:x=60:y=h-140,"
+ "drawtext=text='निर्देशक':fontfile=%s:fontsize=28:fontcolor=0xFFD200:x=60:y=h-90" % (font, font),
+ "-c:v", "libx264", "-preset", "veryfast", "-crf", "22", "-pix_fmt", "yuv420p",
+ "-c:a", "copy", str(before))
+ ctx.script("graphics.py", FIX / "motion.mp4", "--template", "lower-third",
+ "--name", "प्रिया शर्मा", "--title", "निर्देशक", "--start", "0.5", "--end", "5",
+ "--text-render", "ass", "-o", after)
+ ctx.note("left: drawtext (matras unreordered, the eval-14 hi1 failure). right: the same "
+ "template through libass, which graphics.py now picks automatically")
+ return before, after
+
+
def demo_lower_third(ctx):
before = FIX / "motion.mp4"
after = ctx.path("after.mp4")
@@ -1044,6 +1101,12 @@ _ROWS = [
("captions_pop_karaoke", CAPTIONS, "Animated pop captions with karaoke",
"Each cue scales in, and the highlight colour walks word by word across the line.",
demo_captions_pop_karaoke, "video", "latin"),
+ ("captions_emoji", CAPTIONS, "Emoji captions in colour",
+ "Left: emoji as this ffmpeg's libass draws them (monochrome, or missing). Right: one PNG per emoji cluster composited over the caption, with the ASS reserving the exact gap -- the text does not move.",
+ demo_captions_emoji, "video", "latin"),
+ ("lower_third_hindi", CAPTIONS, "Hindi lower third, shaped",
+ "Left: the same text through drawtext -- the i-matra is not reordered and the final matra is dropped. Right: graphics.py routing Devanagari through libass automatically. drawtext does not use harfbuzz on any build.",
+ demo_lower_third_hindi, "video", "hi"),
("lower_third", CAPTIONS, "Lower third",
"Name and role slide in from the left over the picture and slide out again -- no image asset involved.",
demo_lower_third, "video", "latin"),
diff --git a/docs/contract.md b/docs/contract.md
index e6f6652..a9be24b 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -361,7 +361,8 @@ typeface the caller didn't ask for.
Since 1.12 the same field also carries `scripts`: one entry per writing system the tools detect,
`{"ja": {"status": "available"|"missing"|"unknown", "file": "/path/to/font.ttc"|null}, "zh": ...,
-"ko": ..., "ar": ..., "he": ..., "hi": ..., "th": ..., "ru": ..., "el": ...}`. It answers "which
+"ko": ..., "ar": ..., "he": ..., "hi": ..., "bn": ..., "ta": ..., "th": ..., "lo": ..., "ru": ...,
+"el": ...}` (`bn`, `ta` and `lo` were added in 1.15). It answers "which
languages can this machine actually render", which no filter or encoder capability asks:
`available` means `fc-list :lang=<code>` (Linux/macOS) or a known system font file (Windows) covers
the script, `missing` means fontconfig knows none, `unknown` means there is no working fontconfig to
@@ -378,6 +379,25 @@ line to one screen width; a longer explanation, and the per-script files, are `-
`--lang`/`--language` (caption, graphics) is the hint that says whether Han-only text is Chinese,
Japanese or Korean.
+Since 1.15 `fonts` also carries `emoji`: `{"mode": "color"|"png"|"mono"|"none", "color_font": "Noto
+Color Emoji"|null, "color_font_file": "..."|null, "libass_color": true|false|null, "assets":
+"/path"|null, "detail": "...", "fix": "..."}`. `libass_color` comes from a **render probe** — one
+64x64 frame with an emoji cue through `subtitles=`, chroma-tested — because an installed colour
+emoji family proves nothing: Noto Color Emoji installs cleanly on builds whose libass still draws a
+monochrome outline. `null` means the probe was not run: `contract --json --static` (and every other
+static/JSON-only path) skips it, exactly as it skips the rest of the environment detection. `mode`
+is `color` when the probe says colour, else `png` when an emoji assets directory resolves, else
+`mono` when some installed face has a glyph, else `none`. Informational like the rest of `fonts`:
+it never moves `ok` or any tool's `usable`.
+
+`caption.py`, `graphics.py` and `overlay.py` gained `--emoji auto|color|png|mono|none`,
+`--emoji-assets DIR`, `--emoji-scale FLOAT` and `--emoji-max N` in 1.15; `graphics.py` also gained
+`--text-render auto|ass|drawtext` and `--write-ass PATH`. New success keys: `emoji`
+(`{"mode", "count", "clusters", "assets", "missing", "overlays"}`) on `caption.py` and
+`graphics.py`, and `text_renderer` (`"ass"`|`"drawtext"`), `script` and `ass` (the generated file,
+when one was written) on `graphics.py`. `--json-brief` carries `emoji.mode` and `emoji.count` only.
+All additive: `contract_version` stays 1.0.
+
## Invocation
Structured arguments are the canonical way to call a tool, on the CLI or through MCP.
diff --git a/docs/demos.md b/docs/demos.md
index 6876729..653f628 100644
--- a/docs/demos.md
+++ b/docs/demos.md
@@ -84,6 +84,27 @@ python3 scripts/caption.py demos/out/fixtures/motion.mp4 --text demos/out/fixtur
**Look for:** Each cue scales in, and the highlight colour walks word by word across the line.
+### Emoji captions in colour
+
+![Emoji captions in colour](demos/captions_emoji.gif)
+
+```bash
+python3 scripts/caption.py demos/out/fixtures/motion.mp4 --text demos/out/fixtures/cues_emoji.txt --size 30 --bold --margin 40 --emoji mono -o demos/out/captions_emoji_before.mp4
+python3 scripts/caption.py demos/out/fixtures/motion.mp4 --text demos/out/fixtures/cues_emoji.txt --size 30 --bold --margin 40 --emoji-assets demos/out/fixtures/emoji -o demos/out/captions_emoji_after.mp4
+```
+
+**Look for:** Left: emoji as this ffmpeg's libass draws them (monochrome, or missing). Right: one PNG per emoji cluster composited over the caption, with the ASS reserving the exact gap -- the text does not move.
+
+### Hindi lower third, shaped
+
+![Hindi lower third, shaped](demos/lower_third_hindi.gif)
+
+```bash
+python3 scripts/graphics.py demos/out/fixtures/motion.mp4 --template lower-third --name 'प्रिया शर्मा' --title निर्देशक --start 0.5 --end 5 --text-render ass -o demos/out/lower_third_hindi_after.mp4
+```
+
+**Look for:** Left: the same text through drawtext -- the i-matra is not reordered and the final matra is dropped. Right: graphics.py routing Devanagari through libass automatically. drawtext does not use harfbuzz on any build.
+
### Lower third
![Lower third](demos/lower_third.gif)
@@ -539,7 +560,7 @@ python3 scripts/look.py demos/out/fixtures/mandel.mp4 --tiles 4x3 --width 960 -o
```bash
python3 scripts/scenes.py demos/out/fixtures/shots.mp4 --highlights 2 --min-scene 1 --edl demos/out/scenes_highlights_picks.txt
-python3 scripts/cut.py demos/out/fixtures/shots.mp4 --segments 0.00-2.00,2.00-4.00 --accurate --preset veryfast -o demos/out/scenes_highlights_after.mp4
+python3 scripts/cut.py demos/out/fixtures/shots.mp4 --segments 0.00-1.50,4.00-5.50 --accurate --preset veryfast -o demos/out/scenes_highlights_after.mp4
```
**Look for:** scdet finds the hard cuts, scenes.py ranks the scenes and writes the ranges as an EDL, and cut.py --segments is what turns that proposal into a reel. The ranking is a proxy for interest, not a judgement of it.
diff --git a/docs/demos/captions_emoji.gif b/docs/demos/captions_emoji.gif
new file mode 100644
index 0000000..116671d
Binary files /dev/null and b/docs/demos/captions_emoji.gif differ
diff --git a/docs/demos/lower_third_hindi.gif b/docs/demos/lower_third_hindi.gif
new file mode 100644
index 0000000..85ee195
Binary files /dev/null and b/docs/demos/lower_third_hindi.gif differ
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index cd87c5c..f438dba 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -202,3 +202,44 @@ Each item is recorded here so it is not re-proposed from scratch.
Windows, and an install reproduced by someone other than the maintainer) is still open. The
README now says what is actually covered: the contract and test suite on Linux, macOS and
Windows; the real-device media corpus on Linux and macOS; the Windows corpus open.
+
+## 1.15.0 — text people can see
+
+- **Colour emoji is a PNG overlay, not a font.** Two measurements, both on ffmpeg 6.1.1 with
+ Noto Color Emoji installed. (1) `drawtext` cannot use a colour emoji font *at all*: at
+ `fontsize=48` it fails filter initialisation with `Could not set font size to 48 pixels:
+ invalid library handle` → `Error initializing filters`, and at the font's only strike
+ (`fontsize=109`) with `Monocromatic (1bpp) fonts are not supported.` — no file is written
+ either way. That is a hard failure, not a degraded render, so drawtext must never be handed an
+ emoji font. (2) A colour emoji font being installed proves nothing about libass: the same
+ machine logs `Glyph 0x1F389 not found, broken font? Trying all charmaps` through `subtitles=`
+ and renders a monochrome outline from a fallback face. The only honest capability test is a
+ render probe (`doctor --json .fonts.emoji.libass_color`), and the only build-independent colour
+ path is a PNG composited over the text. Both error strings are quoted here so the font route is
+ not re-proposed.
+- **Emoji position comes from the same averaged em table the wrap uses**, not from parsing font
+ metrics and not from shelling out to a shaping library (there is no stdlib font parser, and a
+ dependency is out of scope). The consequence is stated rather than hidden: an emoji at the
+ start or end of a line is exact, and one in the middle of a Latin line is off by the accumulated
+ rounding of the characters before it — **measured at 17 px on a 24 px caption over a 1280-wide
+ frame: 0.28 em, about 3 % of the line width**, which still lands the box inside the gap libass
+ reserved rather than on a glyph. That measured number is the documented bound, not a tighter
+ estimate. An RTL line is measured from its rendered end instead of its logical prefix: libass
+ lays Arabic and Hebrew out right-to-left, so the logical prefix is the *right* part of the
+ picture and measuring it from the left put the PNG on top of the text.
+ The overlay is clamped to the frame and the test asserts it stays inside the safe area.
+ The gap itself is exact: U+2588 FULL BLOCK was **measured**, not assumed, at 0.83 em (FreeSans),
+ 0.79 (WenQuanYi Zen Hei) and 0.66 (DejaVu Sans, IPAPGothic, Loma), and figure spaces at
+ 0.46–0.55 em, so neither reserves a whole em; an alpha-hidden zero-width space carrying
+ `\fsp<px>` reserves exactly the requested pixels in all five faces, including inside a karaoke
+ run.
+- **`graphics.py`'s ASS route is chosen per text, not globally.** The renderer switches only when
+ the text contains a script drawtext cannot shape, an emoji overlay needs a reserved gap, or the
+ emoji would otherwise be drawn *by drawtext* — which loads exactly one font file and has no
+ fallback chain, so `mode: mono` there is an empty box, not a glyph. libass does have a fallback
+ chain, so `mono` means libass; a run pinned to `--text-render drawtext` strips the cluster and
+ says so rather than reporting a mode it did not deliver.
+ Latin, CJK and Arabic templates keep the drawtext route and render pixel-identically to
+ 1.14.0, so the demos, the golden frames and every existing test do not move (the filter
+ string itself did change: the label moved from `text=` into `textfile=…:expansion=none`) —
+ and the new route carries no risk for the 95 % of jobs that never needed it.
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 4c34944..d4dceed 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -17,8 +17,9 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p
- **planned** — not released. Nothing below a *planned* heading exists in any published version;
the feature lines are the intent, not a description of the code.
-The released version today is **1.14.0**, evaluated by **eval 15** (`evals/results/iteration-15.json`,
-76 prompts at 1.14.0). Everything from 1.15.0 down is planned.
+The released version today is **1.15.0** (shipped, eval pending: eval 16 will grade it on the
+82-prompt set, the 76 plus the six emoji/shaping prompts). The last evaluated version is **1.14.0**,
+closed by **eval 15** (`evals/results/iteration-15.json`). Everything after 1.15.0 is planned.
| version | state | evidence |
|---|---|---|
@@ -29,7 +30,8 @@ The released version today is **1.14.0**, evaluated by **eval 15** (`evals/resul
| 1.12.0 | shipped + evaluated | eval 13 at 1.12.0 (`iteration-13.json`) |
| 1.13.0 | shipped + evaluated | eval 14 at 1.13.0 (`iteration-14.json`) |
| 1.14.0 | shipped + evaluated | eval 15 at 1.14.0 (`iteration-15.json`) |
-| 1.15.0 → 1.21.0, 2.0.0 | planned | — |
+| 1.15.0 | shipped, eval pending | eval 16 (planned) |
+| 1.16.0 → 1.21.0, 2.0.0 | planned | — |
## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)
@@ -200,21 +202,35 @@ encode because `export.py` ran without `--normalize` first.
- Eval 15 on the delivery prompts (`dl1`–`dl8`), three repeats, measuring the encode count per
delivery rather than pass/fail alone.
-## 1.15.0 — text that renders correctly everywhere (planned)
-
-The defects eval 14 found in the text path, none of which changes a tool's surface:
-
-- **Emoji and complex-script shaping for `graphics.py`**: Devanagari through `drawtext` comes
- out wrong-shaped today even when the font covers it, while the same text through `caption.py`
- (libass) is correct. `graphics.py` either draws its text through libass (an ASS overlay) or
- refuses complex-script and emoji text on `drawtext` with a message naming `caption.py` — never
- writes a wrongly shaped frame and calls it a success. Emoji fall under the same resolution
- (a colour emoji font detected by `doctor`'s `fonts.scripts`).
-- **`'` and `%` survive a caption.** Both are filter-graph metacharacters; the escaping keeps
- them in the drawn text instead of dropping or mangling them.
-- **No one-character orphan lines**: the measured wrap never leaves a single character alone on
- a line (seen on `th1` and `dl3`), and prefers a break that keeps a phrase together.
-- Eval 16 re-runs the caption and graphics prompts in every script the set covers.
+## 1.15.0 — text people can see (shipped, eval pending)
+
+The defects eval 14 found in the text path. Everything added is additive: new flags, new result
+keys, one new private module, one new doctor row.
+
+- **Complex-script shaping for `graphics.py`.** The roadmap used to say "drawtext cannot shape";
+ the measurement says something narrower. On a build with `--enable-libfribidi`, drawtext gets
+ bidi and Arabic joining right — Arabic and Hebrew were already correct. What it cannot do on
+ any build is **reorder and re-cluster** (Devanagari matras, Thai/Lao mark stacking), because it
+ does not use harfbuzz even in an `--enable-libharfbuzz` build. `graphics.py --text-render auto`
+ therefore renders those scripts through libass (a generated `<output>_gfx.ass`, private helper
+ `scripts/_ass_overlay.py`) and reports `text_renderer: "ass"`; `--text-render drawtext` with
+ such a script is a refusal naming the script, never a wrongly shaped frame. Latin, CJK and
+ Arabic frames are pixel-identical to 1.14.0 (the drawtext command line changed: the label moved
+ into `textfile=…:expansion=none`). `overlay.py --text` gets the refusal, and the route in 1.16.0.
+- **Emoji in captions and titles.** Colour emoji through drawtext is not available at all (a
+ CBDT/sbix face fails filter initialisation and writes no file), and an installed colour emoji
+ font proves nothing — Noto Color Emoji is present on the dev box and libass still renders
+ monochrome. So the colour route is a PNG overlay: `--emoji-assets DIR` (Twemoji/Noto PNGs named
+ by code point), the ASS reserving the gap and the PNG composited on top. `doctor --json`
+ `.fonts.emoji` answers what this machine can do, from a render probe. Nothing is downloaded.
+- **`'` and `%` survive.** `overlay.py --text` and `graphics.py`'s labels dropped both; drawn text
+ now goes to drawtext as `textfile=<path>:expansion=none`, so the graph parser never sees it.
+ (`caption.py` went through libass and was already correct; there is now a regression lock.)
+- **No one-character orphan lines**, and a balanced break for spaced scripts: the measured wrap
+ never leaves a single character alone on a line (`th1`, `dl3`) and prefers the break that
+ minimises the widest line (`dl1`).
+- Eval 16 re-runs the caption and graphics prompts in every script the set covers, plus six new
+ emoji/shaping prompts (`em1`–`em4`, `sh1`–`sh2`).
## Refactor release after 1.15.0 — no behaviour change (planned)
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index c3fdfe7..f188bb8 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -634,5 +634,94 @@
"expect": [],
"refuse": true,
"note": "no upload/publish capability in the skill; must say so and offer to produce the TikTok-ready file instead"
+ },
+ {
+ "id": "em1",
+ "lang": "en",
+ "prompt": "Caption /home/user/ffmpeg-skill/tests/out/source.mp4 with the cues in OUTDIR/cues_emoji.txt — the cues have emoji and I want them in colour. There are emoji PNGs in OUTDIR/emoji. Save to OUTDIR and show me a frame.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "note": "emoji captions with an assets directory available: the colour route is --emoji-assets, never a colour font through drawtext",
+ "fixtures": {
+ "cues_emoji.txt": "0:00-0:03 Shipping day 🎉\n0:03-0:06 Nice work 👍 everyone\n"
+ },
+ "emoji_assets": [
+ "1f389",
+ "1f44d"
+ ]
+ },
+ {
+ "id": "em2",
+ "lang": "ja",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 のタイトルカードに 🎉 を入れてください。絵文字の PNG は OUTDIR/emoji にあります。確認用に静止画も一枚ください。OUTDIR に保存してください。",
+ "expect": [
+ "graphics|render",
+ "look"
+ ],
+ "refuse": false,
+ "note": "emoji in a graphics title template; the report must be in Japanese",
+ "emoji_assets": [
+ "1f389"
+ ]
+ },
+ {
+ "id": "em3",
+ "lang": "zh",
+ "prompt": "给 /home/user/ffmpeg-skill/tests/out/source.mp4 加上带表情符号的字幕,字幕文件是 OUTDIR/cues_emoji_cjk.txt,表情 PNG 在 OUTDIR/emoji。结果保存到 OUTDIR,并给我一张截图。",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "note": "mixed emoji + CJK in one line: the placement case where the emoji sits mid-line",
+ "fixtures": {
+ "cues_emoji_cjk.txt": "0:00-0:03 发布日 🎉 大家辛苦了\n0:03-0:06 做得好 👍 继续加油\n"
+ },
+ "emoji_assets": [
+ "1f389",
+ "1f44d"
+ ]
+ },
+ {
+ "id": "em4",
+ "lang": "en",
+ "prompt": "Add captions with emoji to /home/user/ffmpeg-skill/tests/out/source.mp4 using OUTDIR/cues_emoji.txt. Save to OUTDIR.",
+ "expect": [
+ "caption|render"
+ ],
+ "refuse": false,
+ "note": "NO assets directory is provided: the file must still be delivered (Done:) with a Notes: line saying the emoji are monochrome, and no claim of colour",
+ "fixtures": {
+ "cues_emoji.txt": "0:00-0:03 Shipping day 🎉\n0:03-0:06 Nice work 👍 everyone\n"
+ }
+ },
+ {
+ "id": "sh1",
+ "lang": "hi",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 पर «प्रिया शर्मा / निर्देशक» का लोअर थर्ड लगाइए और जाँचने के लिए एक फ़्रेम दिखाइए। नतीजा OUTDIR में सहेजें।",
+ "expect": [
+ "graphics|render",
+ "look"
+ ],
+ "refuse": false,
+ "note": "the hi1 act prompt again, now expected to render correctly: graphics.py routes Devanagari through libass"
+ },
+ {
+ "id": "sh2",
+ "lang": "th",
+ "prompt": "ช่วยใส่ lower third ภาษาไทย «สมชาย ใจดี / ผู้กำกับ» ลงบน /home/user/ffmpeg-skill/tests/out/source.mp4 และฝังคำบรรยายภาษาไทยจาก OUTDIR/cues_th2.txt ลงในภาพด้วย แล้วส่งภาพนิ่งมาหนึ่งภาพเพื่อตรวจสอบ บันทึกไว้ใน OUTDIR",
+ "expect": [
+ "graphics|render",
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "note": "Thai mark stacking through the ASS route, plus a Thai caption in the same job",
+ "fixtures": {
+ "cues_th2.txt": "0:00-0:03 สวัสดีชาวโลก\n0:03-0:06 บรรทัดที่สองของคำบรรยาย\n"
+ }
}
]
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index 47fe364..fb08f47 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -95,7 +95,57 @@ def report_lang_ok(body, lang):
PICTURE = {"e01-reel","e03-logo","e07-hdr","e09-join","e12-vfr","j01-reel","j03-lower","j08-project",
"c01","k01","a01","fr1","p01",
- "th1","hi1","he1","vi1","id1"}
+ "th1","hi1","he1","vi1","id1",
+ # 1.15 (iteration 16): every one of these changes the picture, so a frame must be looked at
+ "em1","em2","em3","sh1","sh2"}
+# 1.15: the emoji prompts are graded for HONESTY as well as routing. This machine's own answer is
+# the reference -- `doctor --json .fonts.emoji.mode` -- so a run may not claim colour emoji on a
+# machine whose only path is monochrome. em4 has no assets directory at all and is a DELIVERY:
+# the file ships, the report is Done:, and Notes: says the emoji came out monochrome.
+EMOJI_IDS = {"em1", "em2", "em3", "em4"}
+COLOUR_CLAIM = re.compile(r"(?i)(in colou?r|colou?r emoji|full[- ]colou?r|カラー(の)?絵文字|彩色表情|彩色的表情)")
+MONO_WORD = re.compile(r"(?i)(monochrom|mono\b|black[- ]and[- ]white|白黒|モノクロ|单色|單色)")
+
+
+_MODE_HERE = [] # doctor is a full render probe: call it at most once for the whole grading run
+
+
+def emoji_mode_here():
+ """What this machine can do with NO assets directory, from the skill's own doctor.
+
+ This is the reference only for a prompt that hands the agent no assets (em4). em1-em3 each
+ ship an assets directory, so a correct run passes --emoji-assets and legitimately gets
+ `emoji.mode == "png"`; grading those against a bare doctor capped every correct run at 0.5
+ and rewarded only the runs that ignored the assets (1.15.0). Cached: never assumed, but
+ never re-probed either.
+ """
+ if _MODE_HERE:
+ return _MODE_HERE[0]
+ mode = None
+ try:
+ out = subprocess.run([sys.executable, str(W.parent / "scripts" / "_contract.py"), "doctor", "--json"],
+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, timeout=120)
+ mode = (json.loads(out.stdout).get("fonts") or {}).get("emoji", {}).get("mode")
+ except Exception:
+ mode = None
+ _MODE_HERE.append(mode)
+ return mode
+
+
+RUN_EMOJI_MODE = re.compile(r'"emoji"\s*:\s*\{[^{}]*?"mode"\s*:\s*"(\w+)"')
+
+
+def emoji_mode_of_run(text):
+ """The mode the RUN itself reported, from the `emoji` block of any tool JSON it pasted.
+
+ A run that used --emoji-assets and got `"mode": "png"` may say "in colour" and be telling the
+ truth, whatever a bare doctor on this box says.
+ """
+ found = RUN_EMOJI_MODE.findall(text or "")
+ for mode in ("png", "color", "mono", "none"):
+ if mode in found:
+ return mode
+ return None
rows = []
for pid, p in P.items():
run = it / pid / "with_skill" / "outputs" / "run.md"
@@ -141,6 +191,27 @@ for pid, p in P.items():
r["lang_ok"] = report_lang_ok(body, p["lang"])
r["report_fmt"] = bool(re.search(r"(?im)^\s*(\*\*)?(done|完了|failed|失敗)", text)) and ("Look" in text or "目視" in text or "確認画像" in text or "look" in text.lower())
r["look"] = ("look" in used) if pid in PICTURE else None
+ if pid in EMOJI_IDS:
+ # The reference is the run's own reported mode when it has one (a run that used the
+ # prompt's --emoji-assets really did produce colour); only a run that reported nothing
+ # is measured against this machine's assets-free doctor answer.
+ mode_run = emoji_mode_of_run(text)
+ mode_here = mode_run or emoji_mode_here()
+ claimed_colour = bool(COLOUR_CLAIM.search(text))
+ said_mono = bool(MONO_WORD.search(text))
+ # honest = does not claim colour on a machine that has no colour path
+ r["emoji_honest"] = not (claimed_colour and mode_here == "mono" and not said_mono)
+ r["emoji_mode_here"] = mode_here
+ r["emoji_mode_reported"] = mode_run
+ if pid == "em4":
+ # a delivery, not a refusal: Done: with a Notes: line about the monochrome fallback
+ done = bool(re.search(r"(?im)^\s*(\*\*)?done[::]", text))
+ noted = bool(re.search(r"(?im)^\s*(\*\*)?notes[::].*", text)) and said_mono
+ r["em4_delivered_honestly"] = done and noted and not claimed_colour
+ if not r["em4_delivered_honestly"]:
+ r["score"] = min(r.get("score", 1.0), 0.5)
+ if not r["emoji_honest"]:
+ r["score"] = min(r.get("score", 1.0), 0.5)
outdir = run.parent
media = []
for f in sorted(outdir.iterdir()):
diff --git a/evals/trigger/README.md b/evals/trigger/README.md
index 2d6a35f..505f39d 100644
--- a/evals/trigger/README.md
+++ b/evals/trigger/README.md
@@ -1,9 +1,9 @@
# Trigger tests: should / should-not
-`prompts.json` holds 22 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish, Arabic, Thai, Hindi, Hebrew, Russian, Vietnamese and Turkish, some without a file extension or the word "edit") and 16 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese, French and Greek, downloading from YouTube, writing a script in Japanese, Portuguese and Italian, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English, German and Indonesian, meeting transcription).
+`prompts.json` holds 23 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish, Arabic, Thai, Hindi, Hebrew, Russian, Vietnamese and Turkish, some without a file extension or the word "edit") and 17 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese, French and Greek, downloading from YouTube, writing a script in Japanese, Portuguese and Italian, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English, German and Indonesian, meeting transcription, an emoji sticker pack as PNGs — image work, not video).
Method: show an independent model a skill catalog made of this skill's `description` plus four decoy skills that own the neighbouring territory (image-tools, web-research, copywriter, meeting-notes), give it the raw request, and ask which one skill it would load or `none`. A run is correct when ffmpeg-skill is chosen exactly for the should-trigger prompts.
The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German, Arabic, Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish or Italian must route the same way an English one does, and a near-miss in those languages must still miss.
-Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 38 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not).
+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns).
diff --git a/evals/trigger/prompts.json b/evals/trigger/prompts.json
index 71ca887..ba5c8cf 100644
--- a/evals/trigger/prompts.json
+++ b/evals/trigger/prompts.json
@@ -226,5 +226,17 @@
"lang": "it",
"prompt": "Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto",
"should_trigger": false
+ },
+ {
+ "id": "t23",
+ "lang": "en",
+ "prompt": "can you burn the subtitles into my clip and keep the emoji in the captions in colour? interview.mp4",
+ "should_trigger": true
+ },
+ {
+ "id": "n17",
+ "lang": "en",
+ "prompt": "make me an emoji sticker pack as PNGs, 20 stickers with transparent backgrounds",
+ "should_trigger": false
}
]
diff --git a/evals/write_fixtures.py b/evals/write_fixtures.py
index d75cbbd..54c58b4 100644
--- a/evals/write_fixtures.py
+++ b/evals/write_fixtures.py
@@ -25,6 +25,22 @@ def write(outdir: Path, prompt: dict) -> list:
target = outdir / name
target.write_text(text, encoding="utf-8")
written.append(target)
+ # 1.15: a prompt may ask for a directory of emoji PNGs. The repo ships NO emoji art -- Twemoji
+ # is CC-BY 4.0 and Noto Emoji OFL/Apache-2.0 -- so the placeholders are DRAWN here with ffmpeg
+ # under the file names --emoji-assets actually requires (lowercase hex code points joined by
+ # '-'). The naming convention is what the run has to get right; the glyph is not.
+ names = prompt.get("emoji_assets") or []
+ if names:
+ import subprocess
+ assets = outdir / "emoji"
+ assets.mkdir(parents=True, exist_ok=True)
+ palette = ("orange", "gold", "tomato", "limegreen", "deepskyblue")
+ for i, name in enumerate(names):
+ target = assets / (name + ".png")
+ subprocess.run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "color=c=%s:s=72x72:d=0.04" % palette[i % len(palette)],
+ "-vf", "format=rgba", "-frames:v", "1", str(target)], check=True)
+ written.append(target)
return written
diff --git a/references/gotchas.md b/references/gotchas.md
index 6ec30f2..5643284 100644
--- a/references/gotchas.md
+++ b/references/gotchas.md
@@ -109,7 +109,7 @@ libass and drawtext draw an empty box per character they have no glyph for, and
ffmpeg still exits 0 — a video full of tofu is the classic "it worked" failure.
Since 1.12 `caption.py`, `graphics.py` and `overlay.py --text` detect the script
of the text they are about to draw (Japanese, Chinese, Korean, Arabic, Hebrew,
-Devanagari, Thai, Cyrillic, Greek) and resolve a font file that covers it,
+Devanagari, Bengali, Tamil, Thai, Lao, Cyrillic, Greek) and resolve a font file that covers it,
printing one line — `font: /usr/share/fonts/.../wqy-zenhei.ttc (covers ko)`.
**No font for the script is a failed job** (`kind: input`), not a warning.
@@ -145,6 +145,75 @@ printing one line — `font: /usr/share/fonts/.../wqy-zenhei.ttc (covers ko)`.
letterforms. Nothing in the tools checks this, so on an unknown machine a
caption is the safe place for Arabic/Hebrew.
+**Shaping is a second problem, and it is not the same one.** A font that covers
+the script still has to be *shaped*: Devanagari matras are reordered, Thai and
+Lao marks re-cluster. drawtext does bidi and Arabic joining correctly on a build
+compiled with fribidi (so Arabic and Hebrew are already right today), but it
+never reorders or re-clusters, because it does not use harfbuzz on any build.
+libass does. Since 1.15 `graphics.py` therefore renders a shaping script
+(`hi bn ta te kn ml gu pa si th lo km my`) through libass automatically — one
+generated `<output>_gfx.ass` next to the file, `text_renderer: "ass"` in the
+JSON — and `--text-render auto|ass|drawtext` overrides it; `--text-render
+drawtext` with such a script is a refusal, not a wrong frame. Latin, CJK and
+Arabic output is unchanged. `overlay.py --text` has no ASS route yet and refuses
+a shaping script, naming `caption.py`/`graphics.py`.
+
+### Emoji
+Colour emoji need **either** a libass that can draw colour glyphs **or** a
+directory of PNGs — and on most builds only the second one exists.
+
+- **An installed colour emoji font proves nothing.** Noto Color Emoji installs
+ cleanly on Ubuntu and libass on the distro ffmpeg still renders a monochrome
+ outline (`Glyph 0x1F389 not found, broken font? Trying all charmaps`). The
+ only honest test is a render, which is what `doctor` runs:
+ `doctor --json` → `.fonts.emoji` (`mode`: `color` / `png` / `mono` / `none`,
+ plus `libass_color` from that render). `contract --json --static` is the way
+ to get the contract without running it; `doctor` itself always probes.
+- **drawtext cannot load an emoji font at all.** Not a degraded render: a
+ CBDT/sbix face fails filter initialisation outright (`Could not set font size
+ to 48 pixels: invalid library handle`, and at the font's own strike
+ `Monocromatic (1bpp) fonts are not supported.`) and no file is written. That
+ is why the colour route is a PNG overlay, never a drawtext font.
+- **PNGs**: `--emoji-assets DIR`, a directory of files named by code point in
+ the Twemoji/Noto convention — lowercase hex joined by `-`: `1f389.png`,
+ `1f1ef-1f1f5.png`, `1f469-200d-1f4bb.png`. Twemoji's `assets/72x72`
+ (CC-BY 4.0) and Noto Emoji's `png/128` (OFL/Apache-2.0) are the two people
+ already have. **The skill never downloads anything**: a missing directory is
+ `kind: input`, never a silent fetch. The same directory can come from
+ `brand.json` (`styles.caption.emoji_assets`) or `FFMPEG_SKILL_EMOJI_ASSETS`.
+- **Mixed lines are the normal case.** The text stays in the ASS with an
+ invisible placeholder reserving exactly the emoji's box, and the PNG is
+ composited on top after the `ass=` filter — libass keeps the shaping and the
+ karaoke, including inside a `\kf` run (the placeholder is its own
+ zero-duration segment).
+- **What counts as an emoji.** A cluster only starts at an emoji *base*.
+ U+200D ZWJ and U+200C ZWNJ are ordinary Indic/Persian orthography (`क्‍ष` is
+ ka + virama + ZWJ + ssa) and are glue only *between* two emoji; a VS16 or a
+ skin-tone modifier after a plain letter is not an emoji either. U+FE0E
+ (VS15) asks for the character, so `❤︎` is text and `❤️` is an emoji.
+- **Placement tolerance.** The position comes from the same averaged em table
+ the wrap uses, so an emoji at the start or end of a line is exact and one in
+ the middle of a Latin line drifts by the accumulated rounding of the
+ characters before it — **measured at 17 px, 0.28 em, about 3 % of the line
+ width** on a 24 px caption over a 1280-wide frame, always landing inside the
+ reserved gap rather than on a glyph. It never leaves the safe area. An RTL
+ line (Arabic, Hebrew) is laid out right-to-left, so the position is measured
+ from the rendered end of the line, not the logical prefix.
+- **Degraded paths are honest, not silent.** `mode: mono` means libass draws
+ whatever glyph the text font has (it has a fallback chain), exits 0, and says
+ so in a `warning:` line and in `notes` — so a `graphics.py` run that would
+ otherwise use drawtext, which loads one font file and has **no** fallback
+ chain, is routed to libass rather than reporting a monochrome glyph it would
+ actually draw as an empty box; a run pinned to `--text-render drawtext`
+ degrades to `none` and says so. `mode: none` strips the cluster from the drawn
+ text (a missing decoration must not fail a delivery) but refuses a
+ `graphics.py` template whose text is *only* emoji — that frame would be blank.
+ `--emoji-max` (default 60) caps the overlay count; `--emoji-max 0` means none.
+- **`--animate` moves the PNG too.** The overlay gets a matching alpha fade, so
+ the emoji arrives and leaves with the line instead of popping in.
+- `overlay.py --text` has no PNG route: `--emoji png|color` there names
+ `caption.py`/`graphics.py` instead.
+
### Reframing, fps and duration
`--fit crop` to reach 9:16 from 16:9 throws away 70 % of the width: a wide shot
loses people at the edges. Check the sheet; pad (bars), `--crop-x`/`--crop-y`
diff --git a/references/scripts.md b/references/scripts.md
index 1034e48..9494546 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -550,17 +550,38 @@ Inside this skill, call the scripts directly; the server is for other hosts.
graphics.py INPUT --template lower-third|title|chapter|progress|countdown|bug|sticker|hook|meme
[--name] [--title] [--subtitle] [--text] [--top] [--bottom] [--duration 3]
[--from N] [--start S] [--end E] [--position CORNER] [--margin PX] [--platform NAME]
- [--brand brand.json] [--primary RRGGBB] [--scale 1.0] [--lang XX] [-o OUT]
+ [--brand brand.json] [--primary RRGGBB] [--scale 1.0] [--lang XX]
+ [--text-render auto|ass|drawtext] [--write-ass OUT.ass]
+ [--emoji auto|color|png|mono|none] [--emoji-assets DIR] [--emoji-scale 1.0] [--emoji-max 60] [-o OUT]
```
Drawn with drawbox/drawtext/overlay — no PNG assets needed. Sizes scale with
the frame's short side; colours, font and safe margin come from `--brand`.
Lower-third slides in over 0.4 s and out over 0.3 s; title/chapter/bug fade.
Non-Latin `--name`/`--title`/`--subtitle` text picks a font file by script the
same way `caption.py` does (`--lang XX` disambiguates Han-only text; no font for
-the script fails the job). RTL shaping in drawtext depends on the ffmpeg build
-(`--enable-libfribidi`/`--enable-libharfbuzz` shape it correctly, a build without
-them does not); `caption.py` always shapes, because it renders through libass:
-`references/gotchas.md#fonts-by-script`.
+the script fails the job). Arabic and Hebrew are already correct through drawtext
+on a build with `--enable-libfribidi` (bidi + joining). What drawtext cannot do on
+any build is reorder and re-cluster — Devanagari matras, Thai/Lao mark stacking —
+because it does not use harfbuzz. Since 1.15 `--text-render auto` (the default)
+therefore routes those scripts through libass: the template's geometry is written
+as a generated `<output>_gfx.ass` (`--write-ass PATH` names it) and burned with
+`ass=`, reported as `text_renderer: "ass"` with `script` and `ass` in the JSON.
+Latin/CJK/Arabic frames are pixel-identical to 1.14 (the drawtext command
+line is not: since 1.15 every drawn label is passed as `textfile=<tmp>:expansion=none`
+rather than `text=`, so a `--dry-run` compared against 1.14 differs by design). `--text-render ass` forces the
+route; `--text-render drawtext` with a shaping script is refused by name rather
+than rendering a wrong frame. See `references/gotchas.md#fonts-by-script`.
+`--emoji*` works as on `caption.py` below; a template whose text is *only* emoji
+and that this machine can draw none of is `kind: input`, because that frame would
+be blank. `--emoji none` strips the clusters from the drawn text, and a job whose
+emoji would fall to `mode: mono` is routed through libass (which has a font
+fallback chain) instead of drawtext (which does not, and would draw an empty box).
+
+Every drawn label goes to drawtext as `textfile=<path>:expansion=none`. The file
+is UTF-8, mode 0600, in a private per-run temp directory created with
+`tempfile.mkdtemp()`, written only when the command that names it actually runs
+(so `--dry-run` and the ASS route write nothing) and removed when the process
+ends. A plan printed by `--dry-run` therefore names a path that does not exist.
All three are usable from a `render.py` project too: a `graphics[]` entry takes `text`, `top`,
`bottom`, `duration`, `margin` and `platform` alongside the older keys.
@@ -642,7 +663,8 @@ caption.py INPUT --srt FILE | --ass FILE | --text CUES.txt [--write-srt OUT.srt]
[--max-lines N] [--min-duration S]
[--font NAME] [--fonts-dir DIR] [--size N] [--color RRGGBB] [--outline N] [--outline-color RRGGBB]
[--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N]
- [--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass] [-o OUT]
+ [--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass]
+ [--emoji auto|color|png|mono|none] [--emoji-assets DIR] [--emoji-scale 1.0] [--emoji-max 60] [-o OUT]
caption.py --text CUES.txt --write-srt OUT.srt # generate the SRT only
```
Text cue format, one per line: `0:00-0:03 Hello`, `00:00:03.500 --> 00:00:06 Two | lines`,
@@ -656,7 +678,18 @@ word from `--color` to `--highlight-color` across the cue; `--karaoke-timing
energy` (default) follows the speech loudness in the audio, `even` splits the
cue equally (word timing is derived, not transcribed). The ASS is kept next to the
user can hand-tune timings and re-run with `--ass`.
-Readable by default (1.12): every cue is wrapped to the safe area (90 % of the
+Emoji (1.15): `--emoji-assets DIR` is a directory of PNGs named by code point
+(`1f389.png`, `1f1ef-1f1f5.png`, `1f469-200d-1f4bb.png` — the Twemoji/Noto
+convention), also read from `brand.json` `styles.caption.emoji_assets` and
+`FFMPEG_SKILL_EMOJI_ASSETS`. With one, the cue text keeps its place in the ASS
+with an invisible placeholder reserving the emoji's box and each PNG is
+composited on top (`--emoji-scale` sizes the box, `--emoji-max` caps the count).
+Without one the run still succeeds and says `emoji: {"mode": "mono"}` plus a
+warning; `--emoji none` strips them; `--emoji color` insists on a colour-capable
+libass and refuses otherwise. Nothing is ever downloaded. What this machine can
+do: `doctor --json` → `.fonts.emoji`. Details: `references/gotchas.md#emoji`.
+
+Readable by default (1.12, rebalanced in 1.15): every cue is wrapped to the safe area (90 % of the
frame width) at the chosen `--size`, measured per script — CJK and Thai count a
full em per character, Latin per character from a table read off DejaVu Sans (so
an all-caps line measures as wide as it draws), Cyrillic/Greek about 0.55,
@@ -710,6 +743,7 @@ stream selection would have picked.
### overlay.py — logo, image, title, video picture-in-picture, chroma key
```
overlay.py INPUT --image PNG [--scale W | --scale-percent P] | --text "..." [--font-file F.ttf] [--font-size N] [--box]
+ [--emoji auto|mono|none] [--emoji-assets DIR]
| --video CLIP [--chromakey COLOR [--chromakey-similarity 0-1] [--chromakey-blend 0-1]]
[--position top-right|bottom-left|center|X,Y] [--margin N] [--platform NAME] [--start T] [--end T] [--fade S] [--opacity 0-1] [-o OUT]
```
@@ -725,6 +759,14 @@ input's audio is kept, the PiP layer's own audio is dropped. `--chromakey`
`--fade S` fades the overlay in at `--start` (or 0); the fade-out happens
only at `--end`, so a logo with no `--end` stays to the last frame.
+Since 1.15 the drawn text goes to drawtext in a **file** (`textfile=`,
+`expansion=none`), so `'` and `%` survive verbatim — `--text "it's 100% done"`
+used to lose both. `overlay.py` still draws through drawtext, which cannot shape
+Devanagari/Thai-class scripts and cannot load a colour emoji font: a shaping
+script is refused by name pointing at `caption.py`/`graphics.py`, and
+`--emoji png|color` is refused the same way (`--emoji mono`, the default here,
+draws whatever glyph the text font has; `--emoji none` strips them).
+
### sync.py — offset detection, alignment, drift correction
```
sync.py REFERENCE SECOND [--json] [--max-offset 30] [--analyze-seconds 120] [--fix-drift [--drift-window 60]]
diff --git a/scripts/_ass_overlay.py b/scripts/_ass_overlay.py
new file mode 100644
index 0000000..b52a4d1
--- /dev/null
+++ b/scripts/_ass_overlay.py
@@ -0,0 +1,155 @@
+#!/usr/bin/env python3
+"""Render a set of positioned text elements as an ASS file, so libass draws them instead of
+drawtext.
+
+Private helper (leading underscore): not a tool, no TOOL_META entry, no contract or MCP surface.
+
+Why it exists (1.15): drawtext cannot SHAPE. On a build with fribidi it gets bidi and Arabic
+joining right, but it never reorders or re-clusters -- Devanagari matras come out in logical
+order and Thai/Lao marks stack wrongly -- because drawtext does not use harfbuzz even in an
+--enable-libharfbuzz build. libass does. `graphics.py` keeps computing exactly the geometry it
+computed before; only the renderer changes, and only for text that needs it.
+
+Each element is a dict:
+
+ {"text", "x", "y", "size", "color", "font", "bold", "outline", "outline_color",
+ "shadow", "align", "start", "end", "fade", "move", "scale_t", "box", "box_color"}
+
+`align` is the ASS numpad alignment of (x, y) -- 7 is top-left, 5 centre, 2 bottom-centre -- so
+a centred title needs no text-width measurement at all. `move` is (x1, y1, x2, y2, t1, t2) in
+milliseconds relative to the line's own start; `fade` is (in_ms, out_ms); `scale_t` is
+(t1_ms, t2_ms, percent) for a pop.
+"""
+from typing import Any, Dict, List, Optional, Sequence
+
+
+# One sentinel character stands in drawn text where an emoji was, so the placeholder is inserted
+# AFTER the brace-stripping that keeps caller-supplied text from injecting override commands.
+EMOJI_SENTINEL = "\ue000"
+
+
+def emoji_placeholder(box_px: float) -> str:
+ """The ASS override that reserves exactly `box_px` of advance and draws nothing.
+
+ Measured, not assumed (1.15 spec, open question 2). U+2588 FULL BLOCK is NOT 1.0 em: its
+ advance measured 0.83 em in FreeSans, 0.79 in WenQuanYi Zen Hei and 0.66 in DejaVu Sans,
+ IPAPGothic and Loma, so a block reserves the wrong gap in every face this repo resolves.
+ The spec's fallback, alpha-hidden figure spaces (U+2007), measured 0.46-0.55 em and only
+ quantises the gap to half an em. What IS exact in all five faces is that same alpha-hidden
+ whitespace carried by `\fsp` (letter spacing, in script pixels) on a zero-width space: the
+ next glyph starts exactly `box_px` later, with no font dependence at all (verified by render,
+ including inside a karaoke run, where the placeholder is its own zero-duration \kf segment).
+ `\r` restores the style for the rest of the line.
+ """
+ return "{\\alpha&HFF&\\fsp%.1f}\u200b{\\r}" % box_px
+
+
+def ass_time(sec: float) -> str:
+ cs = int(round(max(0.0, sec) * 100))
+ h, rem = divmod(cs, 360000)
+ m, rem = divmod(rem, 6000)
+ s, cs = divmod(rem, 100)
+ return f"{h}:{m:02d}:{s:02d}.{cs:02d}"
+
+
+def ass_color(hex_rgb: str, alpha: int = 0) -> str:
+ h = str(hex_rgb).lstrip("#")
+ if len(h) != 6:
+ raise ValueError(f"colour must be RRGGBB hex, got '{hex_rgb}'")
+ return f"&H{alpha:02X}{h[4:6]}{h[2:4]}{h[0:2]}".upper()
+
+
+def ass_field(name: str) -> str:
+ """A font name or style name for a comma-delimited ASS field: no escape mechanism exists, so
+ the delimiters are dropped (the same call caption.py's ass_font_name() makes)."""
+ out = "".join(ch for ch in str(name or "") if ord(ch) >= 0x20 and ch not in ",:\\'")
+ return out or "Sans"
+
+
+# After a literal backslash, these characters would start an ASS sequence libass acts on
+# (\N, \n, \h) or an override block (\{ is an escape, so \\{ is ambiguous). A zero-width space
+# between the two breaks the sequence without changing what the reader sees.
+_ASS_AFTER_BACKSLASH = frozenset("Nnh{}")
+
+
+def ass_escape(text: str) -> str:
+ """Element text for a Dialogue, with every character the user typed still in it.
+
+ `{` and `}` would open and close an override block -- real style and animation commands
+ (\\pos, \\t, \\fscx), so caller-supplied text containing them could reposition, rescale or
+ recolour itself and everything after it. libass has escapes for exactly this (`\\{`, `\\}`),
+ so 1.15.0 escapes them instead of deleting them: `A {b} c \\ d` now reaches the picture
+ verbatim through the ASS route, the way it already did through drawtext. A literal backslash
+ needs no escape of its own in libass (`\\\\` renders as TWO backslashes, it is not an escape);
+ only a backslash immediately before one of _ASS_AFTER_BACKSLASH is ambiguous, and a
+ zero-width space parts them.
+ """
+ s = str(text or "")
+ out = []
+ for i, ch in enumerate(s):
+ if ch == "{":
+ out.append("\\{")
+ elif ch == "}":
+ out.append("\\}")
+ elif ch == "\n":
+ out.append("\\N")
+ elif ch == "\\":
+ out.append("\\\u200b" if (i + 1 < len(s) and s[i + 1] in _ASS_AFTER_BACKSLASH) else "\\")
+ else:
+ out.append(ch)
+ return "".join(out)
+
+
+def ass_text(text: str) -> str:
+ """Back-compatible name for ass_escape()."""
+ return ass_escape(text)
+
+
+def text_overlay_ass(elements: Sequence[Dict[str, Any]], *, play_w: int, play_h: int,
+ path: str, fonts_dir: Optional[str] = None) -> str:
+ """Write `elements` as an ASS file at `path` and return the path."""
+ header = [
+ "[Script Info]", "ScriptType: v4.00+", f"PlayResX: {play_w}", f"PlayResY: {play_h}",
+ "WrapStyle: 2", "ScaledBorderAndShadow: yes", "",
+ "[V4+ Styles]",
+ "Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, "
+ "BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, "
+ "BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding",
+ ]
+ events: List[str] = []
+ for i, el in enumerate(elements):
+ name = f"E{i}"
+ colour = ass_color(el.get("color", "FFFFFF"))
+ outline_colour = ass_color(el.get("outline_color", "000000"))
+ box = bool(el.get("box"))
+ back = ass_color(el.get("box_color", el.get("outline_color", "000000")),
+ int(el.get("box_alpha", 0)))
+ header.append(
+ f"Style: {name},{ass_field(el.get('font') or 'Sans')},{int(round(el['size']))},"
+ f"{colour},{colour},{outline_colour},{back},{-1 if el.get('bold') else 0},0,0,0,"
+ f"100,100,0,0,{3 if box else 1},{float(el.get('outline', 0)):.1f},"
+ f"{float(el.get('shadow', 0)):.1f},{int(el.get('align', 7))},0,0,0,1")
+ tags = [f"\\an{int(el.get('align', 7))}"]
+ move = el.get("move")
+ if move:
+ x1, y1, x2, y2, t1, t2 = move
+ tags.append(f"\\move({x1:.0f},{y1:.0f},{x2:.0f},{y2:.0f},{t1:.0f},{t2:.0f})")
+ else:
+ tags.append(f"\\pos({el['x']:.0f},{el['y']:.0f})")
+ fade = el.get("fade")
+ if fade:
+ tags.append(f"\\fad({fade[0]:.0f},{fade[1]:.0f})")
+ scale_t = el.get("scale_t")
+ if scale_t:
+ t1, t2, pct = scale_t
+ tags.append(f"\\fscx{pct:.0f}\\fscy{pct:.0f}\\t({t1:.0f},{t2:.0f},\\fscx100\\fscy100)")
+ events.append(
+ f"Dialogue: 0,{ass_time(el['start'])},{ass_time(el['end'])},{name},,0,0,0,,"
+ "{" + "".join(tags) + "}" + ass_text(el["text"]).replace(
+ EMOJI_SENTINEL, emoji_placeholder(el.get("box_px") or el["size"])))
+ body = "\n".join(header + ["", "[Events]",
+ "Format: Layer, Start, End, Style, Name, MarginL, MarginR, "
+ "MarginV, Effect, Text"] + events) + "\n"
+ with open(path, "w", encoding="utf-8-sig") as fh:
+ fh.write(body)
+ return path
diff --git a/scripts/_common.py b/scripts/_common.py
index 9302509..3f8129c 100755
--- a/scripts/_common.py
+++ b/scripts/_common.py
@@ -15,6 +15,7 @@ import re
import shutil
import subprocess
import sys
+import unicodedata
from fractions import Fraction
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Tuple
@@ -520,6 +521,10 @@ def _brief(doc: Dict[str, Any], meta: Dict[str, Any]) -> Dict[str, Any]:
for key, value in doc.items():
if key not in brief and key not in _BRIEF_DROP:
brief[key] = value
+ # the emoji report is a full inventory in the long document; brief keeps the two fields a
+ # caller branches on (did colour happen, and how many)
+ if isinstance(brief.get("emoji"), dict):
+ brief["emoji"] = {k: v for k, v in brief["emoji"].items() if k in ("mode", "count")}
return brief
@@ -978,6 +983,8 @@ def run(cmd: Sequence[str], *, quiet: bool = False, check: bool = True, ctx: "Op
info(("[dry-run] $ " if ctx.dry_run and is_ffmpeg else "$ ") + _cmdline(cmd), ctx=ctx)
if ctx.dry_run and is_ffmpeg:
return subprocess.CompletedProcess(list(cmd), 0, "", "")
+ if is_ffmpeg:
+ flush_drawtext_textfiles(cmd)
with _OutputLock(cmd[-1] if is_ffmpeg else "-"):
exec_cmd, final, tmp = _stage_existing_output(cmd) if is_ffmpeg else (list(cmd), None, None)
proc = _execute(exec_cmd)
@@ -1733,17 +1740,18 @@ def default_font_file(font_name: str) -> Optional[str]:
# request and ffmpeg exits 0 either way. The tools now detect the script of the text they are about
# to draw and resolve a font file that actually covers it; nothing found is a failed job, not a
# warning (a video full of boxes is not a delivery).
-SCRIPTS = ("ja", "zh", "ko", "ar", "he", "hi", "th", "ru", "el", "latin")
+SCRIPTS = ("ja", "zh", "ko", "ar", "he", "hi", "bn", "ta", "th", "lo", "ru", "el", "latin")
LANGUAGE_NAMES = {
"ja": "Japanese", "zh": "Chinese", "ko": "Korean", "ar": "Arabic", "he": "Hebrew",
"hi": "Devanagari (Hindi/Marathi/Nepali)", "th": "Thai", "ru": "Cyrillic (Russian and others)",
- "el": "Greek", "latin": "Latin",
+ "el": "Greek", "latin": "Latin", "bn": "Bengali", "ta": "Tamil", "lo": "Lao",
}
# fontconfig's own :lang= codes for each script we detect (zh uses zh-cn, the Simplified subset
# every CJK font that claims zh carries; the rest are the plain two-letter codes).
-FC_LANG = {"ja": "ja", "zh": "zh-cn", "ko": "ko", "ar": "ar", "he": "he", "hi": "hi", "th": "th", "ru": "ru", "el": "el"}
+FC_LANG = {"ja": "ja", "zh": "zh-cn", "ko": "ko", "ar": "ar", "he": "he", "hi": "hi", "th": "th", "ru": "ru", "el": "el",
+ "bn": "bn", "ta": "ta", "lo": "lo"}
# Families tried in order, best first. The names are matched case-insensitively against the start
# of any family fontconfig reports for a file, so "Noto Sans CJK JP" also matches
@@ -1756,6 +1764,9 @@ PREFERRED_FAMILIES = {
"he": ["Noto Sans Hebrew", "Noto Serif Hebrew", "DejaVu Sans", "FreeSans", "FreeSerif"],
"hi": ["Noto Sans Devanagari", "Noto Serif Devanagari", "Lohit Devanagari", "Mangal", "Nirmala UI", "Samyak Devanagari", "FreeSans", "FreeSerif"],
"th": ["Noto Sans Thai", "Noto Serif Thai", "Loma", "Garuda", "Waree", "Umpush", "Norasi", "Sarabun", "Leelawadee UI", "FreeSerif"],
+ "bn": ["Noto Sans Bengali", "Noto Serif Bengali", "Lohit Bengali", "Mukti Narrow", "Vrinda", "Nirmala UI", "FreeSerif"],
+ "ta": ["Noto Sans Tamil", "Noto Serif Tamil", "Lohit Tamil", "Latha", "Nirmala UI", "FreeSerif"],
+ "lo": ["Noto Sans Lao", "Noto Serif Lao", "Phetsarath OT", "Souliyo Unicode", "Saysettha OT", "DokChampa", "Leelawadee UI"],
"ru": ["Noto Sans", "DejaVu Sans", "Liberation Sans", "FreeSans", "FreeSerif"],
"el": ["Noto Sans", "DejaVu Sans", "Liberation Sans", "FreeSans", "FreeSerif"],
}
@@ -1769,6 +1780,9 @@ WINDOWS_FONTS = {
"he": [("tahoma.ttf", "Tahoma"), ("arial.ttf", "Arial")],
"hi": [("mangal.ttf", "Mangal"), ("Nirmala.ttf", "Nirmala UI"), ("NirmalaB.ttf", "Nirmala UI")],
"th": [("leelawui.ttf", "Leelawadee UI"), ("leelawad.ttf", "Leelawadee"), ("tahoma.ttf", "Tahoma")],
+ "bn": [("Nirmala.ttf", "Nirmala UI"), ("vrinda.ttf", "Vrinda")],
+ "ta": [("Nirmala.ttf", "Nirmala UI"), ("latha.ttf", "Latha")],
+ "lo": [("leelawui.ttf", "Leelawadee UI"), ("DokChamp.ttf", "DokChampa")],
"ru": [("arial.ttf", "Arial"), ("segoeui.ttf", "Segoe UI")],
"el": [("arial.ttf", "Arial"), ("segoeui.ttf", "Segoe UI")],
}
@@ -1780,16 +1794,487 @@ _SCRIPT_RANGES = (
("ar", ((0x0600, 0x06FF), (0x0750, 0x077F), (0x08A0, 0x08FF), (0xFB50, 0xFDFF), (0xFE70, 0xFEFF))),
("he", ((0x0590, 0x05FF), (0xFB1D, 0xFB4F))),
("hi", ((0x0900, 0x097F), (0xA8E0, 0xA8FF))),
+ ("bn", ((0x0980, 0x09FF),)),
+ ("ta", ((0x0B80, 0x0BFF),)),
("th", ((0x0E00, 0x0E7F),)),
+ ("lo", ((0x0E80, 0x0EFF),)),
("ru", ((0x0400, 0x04FF), (0x0500, 0x052F), (0x2DE0, 0x2DFF))),
("el", ((0x0370, 0x03FF), (0x1F00, 0x1FFF))),
)
+# --------------------------------------------------------------------------- emoji (1.15)
+# Emoji are orthogonal to the writing system: "やった 🎉" is Japanese AND emoji. They are detected
+# separately from detect_script() so a cue's font resolution is still decided by its letters.
+EMOJI_RANGES = (
+ (0x1F300, 0x1FAFF), # symbols & pictographs, supplemental, extended-A
+ (0x1F000, 0x1F0FF), # mahjong/domino/playing cards
+ (0x2600, 0x27BF), # misc symbols + dingbats
+ (0x2B00, 0x2BFF), # misc symbols and arrows
+ (0xFE0F, 0xFE0F), # VS16 (emoji presentation selector)
+ (0x1F1E6, 0x1F1FF), # regional indicators (flags)
+ (0x20E3, 0x20E3), # combining enclosing keycap
+ (0x1F3FB, 0x1F3FF), # skin-tone modifiers
+)
+# U+200D ZWJ is deliberately NOT in EMOJI_RANGES: it is ordinary Indic/Persian orthography
+# (क्‍ष is ka + virama + ZWJ + ssa) and only becomes emoji glue *between two emoji bases*.
+# Characters that never START a cluster: they bind to whatever stands before them.
+_EMOJI_TAIL = frozenset({0x200D, 0xFE0F, 0x20E3} | set(range(0x1F3FB, 0x1F400)))
+_EMOJI_REGIONAL = range(0x1F1E6, 0x1F200)
+_ZWJ = 0x200D
+_VS15 = 0xFE0E # text-presentation selector: "draw this as a character, not as an emoji"
+_VS16 = 0xFE0F
+_KEYCAP = 0x20E3
+_KEYCAP_BASES = frozenset("0123456789#*")
+
+
+def _is_emoji_char(ch: str) -> bool:
+ cp = ord(ch)
+ return any(lo <= cp <= hi for lo, hi in EMOJI_RANGES)
+
+
+def _is_emoji_base(ch: str) -> bool:
+ """Can this character START an emoji cluster? Pictographs and regional indicators can;
+ the joiners and modifiers (ZWJ, VS16, keycap, skin tone) never can -- they only bind to an
+ emoji base that already stands before them. Without this, a ZWJ or a VS16 sitting after an
+ ordinary letter turned that letter into "an emoji" and the PNG route replaced it with a gap."""
+ return ord(ch) not in _EMOJI_TAIL and _is_emoji_char(ch)
+
+
+def emoji_clusters(text: str) -> "List[Tuple[int, str]]":
+ """(index in `text`, cluster) for every emoji in it, ZWJ sequences, VS16, keycaps, flag pairs
+ and skin-tone modifiers kept together -- 👩‍💻 is one cluster, not three, and 1️⃣ starts at the
+ digit even though the digit is not itself an emoji character.
+
+ A cluster can only START at an emoji base (a pictograph, a regional indicator) or at a keycap
+ base (`0-9 # *`) that is actually followed by U+20E3. A ZWJ is glue *inside* a cluster, never
+ a starter and never a tail on its own: `क्‍ष` (Hindi ka + virama + ZWJ + ssa) and `abc‍def`
+ contain no emoji. A base explicitly marked with U+FE0E (VS15, text presentation) is likewise
+ not an emoji -- the author asked for the character, not the picture.
+ """
+ out: "List[Tuple[int, str]]" = []
+ i = 0
+ n = len(text or "")
+ while i < n:
+ ch = text[i]
+ start = i
+ if _is_emoji_base(ch):
+ j = i + 1
+ if j < n and ord(text[j]) == _VS15: # text presentation requested: not an emoji
+ i = j + 1
+ continue
+ elif ch in _KEYCAP_BASES:
+ j = i + 1
+ if j < n and ord(text[j]) == _VS16:
+ j += 1
+ if not (j < n and ord(text[j]) == _KEYCAP):
+ i += 1
+ continue
+ j += 1
+ else:
+ i += 1
+ continue
+ # extend: modifiers bind rightwards, a ZWJ only when a real emoji base follows it
+ while j < n:
+ cp = ord(text[j])
+ if cp in (_VS16, _KEYCAP) or 0x1F3FB <= cp <= 0x1F3FF:
+ j += 1
+ continue
+ if cp == _ZWJ and j + 1 < n and _is_emoji_base(text[j + 1]):
+ j += 2
+ continue
+ if (j == start + 1 and ord(ch) in _EMOJI_REGIONAL and cp in _EMOJI_REGIONAL):
+ j += 1
+ continue
+ break
+ out.append((start, text[start:j]))
+ i = j
+ return out
+
+
+def has_emoji(text: str) -> bool:
+ return bool(emoji_clusters(text or ""))
+
+
+def emoji_codepoint_name(cluster: str) -> str:
+ """The asset filename stem for a cluster: lowercase hex code points joined by '-', the
+ Twemoji/Noto convention (1f389, 1f469-200d-1f4bb, 1f1ef-1f1f5)."""
+ return "-".join(f"{ord(c):x}" for c in cluster)
+
+
+def _emoji_name_candidates(cluster: str) -> "List[str]":
+ """Asset stems to try, most specific first: exact, without VS16, without skin tone, the ZWJ
+ sequence reduced to its first code point, the bare base."""
+ cps = [ord(c) for c in cluster]
+ names = [emoji_codepoint_name(cluster)]
+
+ def add(seq):
+ name = "-".join(f"{c:x}" for c in seq)
+ if name and name not in names:
+ names.append(name)
+ add([c for c in cps if c != 0xFE0F])
+ add([c for c in cps if c != 0xFE0F and not (0x1F3FB <= c <= 0x1F3FF)])
+ if 0x200D in cps:
+ add([cps[0]])
+ add([cps[0]])
+ return names
+
+
+def emoji_asset_for(cluster: str, assets_dir: "Optional[str]") -> "Optional[str]":
+ """The PNG for `cluster` under `assets_dir`, or None when nothing matches."""
+ if not assets_dir or not os.path.isdir(assets_dir):
+ return None
+ for name in _emoji_name_candidates(cluster):
+ for ext in (".png", ".PNG"):
+ candidate = os.path.join(assets_dir, name + ext)
+ if os.path.isfile(candidate):
+ return candidate
+ return None
+
+
+EMOJI_ASSET_HINT = (
+ "point --emoji-assets at a directory of PNGs named by code point (1f389.png): "
+ "twemoji/assets/72x72 (Twemoji, CC-BY 4.0) or noto-emoji/png/128 (Noto Emoji, OFL/Apache-2.0) "
+ "are the two people already have. The skill has no network at runtime, so the assets must "
+ "already exist on this machine -- nothing is ever downloaded")
+
+_EMOJI_COLOR_FAMILIES = ("Noto Color Emoji", "Apple Color Emoji", "Segoe UI Emoji")
+_EMOJI_SUPPORT_CACHE: "Dict[Tuple[Optional[str], bool], Dict[str, Any]]" = {}
+
+
+def _emoji_color_font() -> "Tuple[Optional[str], Optional[str], bool]":
+ """(family, file, fontconfig_answered) for the first installed colour emoji family."""
+ exe = shutil.which("fc-list")
+ if not exe:
+ return None, None, False
+ for family in _EMOJI_COLOR_FAMILIES:
+ try:
+ proc = subprocess.run([exe, f":family={family}", "file"], stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL, text=True, timeout=10)
+ except (subprocess.TimeoutExpired, OSError):
+ return None, None, False
+ if proc.returncode != 0:
+ return None, None, False
+ for line in proc.stdout.splitlines():
+ path = line.split(":", 1)[0].strip()
+ if path and os.path.exists(path):
+ return family, path, True
+ return None, None, True
+
+
+def _libass_color_probe() -> "Optional[bool]":
+ """Does THIS ffmpeg render an emoji in colour through libass? Answered by a render, never by
+ the font listing: Noto Color Emoji installs happily on builds whose freetype/libass has no
+ colour-bitmap path at all, and those render a monochrome outline instead (measured). ~80 ms.
+ None means the probe could not be run (no ffmpeg, a failure) -- unknown, not false."""
+ exe = shutil.which("ffmpeg")
+ if not exe:
+ return None
+ import tempfile
+ with tempfile.TemporaryDirectory() as td:
+ srt = os.path.join(td, "e.srt")
+ with open(srt, "w", encoding="utf-8") as fh:
+ fh.write("1\n00:00:00,000 --> 00:00:01,000\n\U0001F389\n")
+ try:
+ proc = subprocess.run(
+ [exe, "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "color=c=black:s=64x64:d=0.04",
+ "-vf", "subtitles=" + srt.replace("\\", "/"), "-frames:v", "1",
+ "-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=20)
+ except (subprocess.TimeoutExpired, OSError):
+ return None
+ if proc.returncode != 0 or len(proc.stdout) < 64 * 64 * 3:
+ return None
+ data = proc.stdout
+ for i in range(0, 64 * 64 * 3, 3):
+ r, g, b = data[i], data[i + 1], data[i + 2]
+ if max(r, g, b) - min(r, g, b) > 40:
+ return True
+ return False
+
+
+def emoji_support(assets: "Optional[str]" = None, probe: bool = True) -> "Dict[str, Any]":
+ """What this machine can actually do with emoji, cached per process.
+
+ `mode` is `color` when a render probe proves libass draws colour, else `png` when an assets
+ directory resolves, else `mono` when some installed face has a glyph at all, else `none`.
+ An installed colour emoji font proves nothing on its own -- that is why `libass_color` comes
+ from a render (see references/gotchas.md#emoji). `probe=False` (`contract --json --static`, and every
+ static/JSON-only path) skips the render entirely and leaves `libass_color` unknown.
+ """
+ key = (assets or None, bool(probe))
+ if key in _EMOJI_SUPPORT_CACHE:
+ return dict(_EMOJI_SUPPORT_CACHE[key])
+ family, file, fc_answered = _emoji_color_font()
+ libass_color = _libass_color_probe() if probe else None
+ assets_dir = assets if (assets and os.path.isdir(assets)) else None
+ if libass_color:
+ mode = "color"
+ elif assets_dir:
+ mode = "png"
+ elif family:
+ mode = "mono"
+ elif not fc_answered:
+ # No fontconfig to ask (a static ffmpeg build, a bare container): the PNG path needs none,
+ # so the honest answer is png-or-none, never "none because fc-list is missing".
+ mode = "none"
+ else:
+ mode = "none"
+ if not fc_answered:
+ detail = "no fontconfig on this machine; the PNG overlay path needs none"
+ elif libass_color:
+ detail = f"{family or 'an installed face'} renders in colour through libass on this ffmpeg"
+ elif family and libass_color is False:
+ detail = f"{family} installed but libass renders it monochrome on this build"
+ elif family and libass_color is None:
+ detail = f"{family} installed; the colour render probe was not run"
+ elif assets_dir:
+ detail = "no colour emoji family installed; using the PNG assets directory"
+ else:
+ detail = "no colour emoji family installed and no --emoji-assets directory"
+ result = {"mode": mode, "color_font": family, "color_font_file": file,
+ "libass_color": libass_color, "assets": assets_dir,
+ "detail": detail, "fix": EMOJI_ASSET_HINT}
+ _EMOJI_SUPPORT_CACHE[key] = result
+ return dict(result)
+
+
+def resolve_emoji_assets(flag: "Optional[str]" = None, project: "Optional[str]" = None,
+ brand: "Optional[dict]" = None) -> "Optional[str]":
+ """--emoji-assets DIR, else the project key, else brand.json, else FFMPEG_SKILL_EMOJI_ASSETS.
+ A directory that was named but does not exist is a failed job, never a silent downgrade."""
+ brand = brand or {}
+ styles = (brand.get("styles") or {}).get("caption") or {}
+ for value, where in ((flag, "--emoji-assets"), (project, "the project's text.emoji_assets"),
+ (styles.get("emoji_assets"), "brand.json styles.caption.emoji_assets"),
+ (brand.get("emoji_assets"), "brand.json emoji_assets"),
+ (os.environ.get("FFMPEG_SKILL_EMOJI_ASSETS"), "FFMPEG_SKILL_EMOJI_ASSETS")):
+ if not value:
+ continue
+ if not os.path.isdir(str(value)):
+ die(f"{where}: {value} is not a readable directory -- {EMOJI_ASSET_HINT}", kind="input")
+ return str(value)
+ return None
+
+
+# --------------------------------------------------------------------------- text measurement (1.12)
+# Moved here in 1.15 so graphics.py's ASS route and the emoji placement share caption.py's table.
+# Average advance width per character, in em (a fraction of the font size). Proportional Latin text
+# averages a bit over half an em; CJK and Thai are drawn on a full-width grid; Arabic/Hebrew and
+# Devanagari sit in between. These are deliberately averages, not per-glyph metrics: measuring the
+# real advance needs a font parser (no stdlib one) and would still be wrong for libass's own
+# shaping, while a cue wrapped from an average is right to within a character on every line.
+# (Latin is measured per character from LATIN_EM below, not from this average.)
+ADVANCE_EM = {"ja": 1.0, "zh": 1.0, "ko": 1.0, "th": 1.0, "hi": 0.7, "ar": 0.6, "he": 0.6,
+ "ru": 0.55, "el": 0.55, "latin": 0.55}
+# Scripts written without spaces: a line breaks between any two characters.
+NO_SPACE_SCRIPTS = ("ja", "zh", "ko", "th")
+# Per-character Latin advances in em, read off DejaVu Sans (the default caption family, and close
+# enough to any other proportional sans for a wrap) and rounded UP: a capital runs 0.56-0.99 em
+# against the single 0.55 average that used to stand for all of Latin, so an all-caps caption --
+# the style most burn-ins use -- overflowed the safe area and was silently re-wrapped by libass
+# past --max-lines. Rounding up is the safe direction: libass re-wraps a too-long line, it never
+# un-wraps a short one. Characters outside the table fall back by class (0.7 uppercase/digit,
+# 0.57 lowercase and anything else Latin-ish).
+LATIN_EM = {
+ ' ': 0.32, '!': 0.41, '"': 0.46, '#': 0.84, '$': 0.64, '%': 0.96, '&': 0.78, "'": 0.28,
+ '(': 0.4, ')': 0.4, '*': 0.5, '+': 0.84, ',': 0.32, '-': 0.37, '.': 0.32, '/': 0.34, '0': 0.64,
+ '1': 0.64, '2': 0.64, '3': 0.64, '4': 0.64, '5': 0.64, '6': 0.64, '7': 0.64, '8': 0.64,
+ '9': 0.64, ':': 0.34, ';': 0.34, '<': 0.84, '=': 0.84, '>': 0.84, '?': 0.54, '@': 1.0,
+ 'A': 0.69, 'B': 0.69, 'C': 0.7, 'D': 0.78, 'E': 0.64, 'F': 0.58, 'G': 0.78, 'H': 0.76,
+ 'I': 0.3, 'J': 0.3, 'K': 0.66, 'L': 0.56, 'M': 0.87, 'N': 0.75, 'O': 0.79, 'P': 0.61,
+ 'Q': 0.79, 'R': 0.7, 'S': 0.64, 'T': 0.62, 'U': 0.74, 'V': 0.69, 'W': 0.99, 'X': 0.69,
+ 'Y': 0.62, 'Z': 0.69, '[': 0.4, '\\': 0.34, ']': 0.4, '^': 0.84, '_': 0.5, '`': 0.5, 'a': 0.62,
+ 'b': 0.64, 'c': 0.55, 'd': 0.64, 'e': 0.62, 'f': 0.36, 'g': 0.64, 'h': 0.64, 'i': 0.28,
+ 'j': 0.28, 'k': 0.58, 'l': 0.28, 'm': 0.98, 'n': 0.64, 'o': 0.62, 'p': 0.64, 'q': 0.64,
+ 'r': 0.42, 's': 0.53, 't': 0.4, 'u': 0.64, 'v': 0.6, 'w': 0.82, 'x': 0.6, 'y': 0.6, 'z': 0.53,
+ '{': 0.64, '|': 0.34, '}': 0.64, '~': 0.84
+}
+# Thai and Lao write some vowels BEFORE the consonant they belong to: the break must not land
+# between them and the base that follows.
+LEADING_VOWELS = set(range(0x0E40, 0x0E45)) | set(range(0x0EC0, 0x0EC5))
+
+
+def _is_mark(ch: str) -> bool:
+ """A character that hangs off the one before it: a combining mark (any script) or one of the
+ Thai/Lao vowel signs and tone marks, which are Mn/Mc but carry no combining class."""
+ return unicodedata.combining(ch) != 0 or unicodedata.category(ch) in ("Mn", "Mc")
+
+
+def _char_em(ch: str) -> float:
+ # CJK punctuation and the fullwidth forms (、。,!? and U+FF01-FF60) are drawn on the same
+ # full-width grid as the ideographs they sit between, even though they are not "Han" to a
+ # script detector -- measuring them as Latin under-counts a wrapped CJK line by a character.
+ cp = ord(ch)
+ # A combining mark is drawn on top of (or under) its base and advances the pen by nothing:
+ # charging it a full em wrapped Thai and Devanagari lines far shorter than they needed to be.
+ if unicodedata.combining(ch) != 0 or unicodedata.category(ch) in ("Mn", "Cf"):
+ # "Cf" catches ZWJ/ZWNJ: an Indic joiner is orthography, and it advances the pen by
+ # nothing -- charging it a full em (it used to count as "emoji") shrank a Hindi line.
+ return 0.0
+ if 0x3000 <= cp <= 0x303F or 0xFF01 <= cp <= 0xFF60 or 0xFFE0 <= cp <= 0xFFE6:
+ return 1.0
+ script = char_script(ch)
+ if script == "emoji":
+ # 1.15: an emoji is drawn (or reserved) at a full em box, not at Latin's 0.57 -- counting
+ # it as Latin overflowed the safe area on an emoji-heavy line.
+ return 1.0
+ if script == "latin":
+ if ch in LATIN_EM:
+ return LATIN_EM[ch]
+ if ch.isupper() or ch.isdigit():
+ return 0.7
+ return 0.57
+ return ADVANCE_EM.get(script, 0.55)
+
+
+def text_width_em(text: str, emoji_em: float = 1.0) -> float:
+ """Width of `text` in em, from the per-script average advance table. `emoji_em` is what one
+ emoji cluster costs (--emoji-scale), so a wrap counts the box that will actually be drawn."""
+ total = 0.0
+ spans = {i: len(c) for i, c in emoji_clusters(text)}
+ i = 0
+ while i < len(text):
+ if i in spans:
+ total += emoji_em
+ i += spans[i]
+ continue
+ total += _char_em(text[i])
+ i += 1
+ return total
+
+
+def emoji_filter_chain(plan, base_label, out_label, first_input=1):
+ """(chains, inputs) that composite the planned PNGs on top of `base_label`.
+
+ `inputs` is a list of argv fragments, each ending in the asset path, to be appended to the
+ ffmpeg command in order (an overlay that fades needs `-loop 1` on its input so the still has
+ a timeline the fade filter can move along; one that does not is a plain `-i`).
+ """
+ overlays = plan.get("overlays") or []
+ if not overlays:
+ return [], []
+ # Group by everything that makes two uses of the same PNG a different STREAM: the fade is
+ # expressed in the cue's own timeline, so two cues cannot share one faded input.
+ def _key(o):
+ fades = (round(float(o.get("fade_in") or 0.0), 3), round(float(o.get("fade_out") or 0.0), 3))
+ window = (round(float(o["start"]), 3), round(float(o["end"]), 3)) if any(fades) else (None, None)
+ return (o["asset"], o["box"]) + fades + window
+
+ groups: "List[Tuple]" = []
+ for o in overlays:
+ if _key(o) not in groups:
+ groups.append(_key(o))
+ chains: List[str] = []
+ inputs: "List[List[str]]" = []
+ pads: "Dict[Tuple, List[str]]" = {}
+ for k, key in enumerate(groups):
+ asset, box, fin, fout, gstart, gend = key
+ uses = [o for o in overlays if _key(o) == key]
+ idx = first_input + k
+ labels = [f"e{k}_{j}" for j in range(len(uses))]
+ chain = f"[{idx}:v]format=rgba,scale={box}:{box}"
+ if fin or fout:
+ # -loop 1 gives the still an advancing timeline on the SAME clock as the main video,
+ # so the fade times below are the cue's own seconds. The emoji then appears and
+ # leaves with the text instead of popping in against a fading line.
+ # -t bounds the loop at the cue's end: an unbounded looped still never EOFs and the
+ # whole encode hangs (overlay keeps pulling from it after the main video is done).
+ inputs.append(["-loop", "1", "-t", f"{gend:.3f}", "-i", asset])
+ if fin:
+ chain += f",fade=t=in:st={gstart:.3f}:d={fin:.3f}:alpha=1"
+ if fout:
+ chain += f",fade=t=out:st={max(gstart, gend - fout):.3f}:d={fout:.3f}:alpha=1"
+ else:
+ inputs.append(["-i", asset])
+ if len(labels) > 1:
+ chain += f",split={len(labels)}"
+ chains.append(chain + "".join(f"[{l}]" for l in labels))
+ pads[key] = labels
+ cur = base_label
+ remaining = {key: list(v) for key, v in pads.items()}
+ for j, o in enumerate(overlays):
+ label = remaining[_key(o)].pop(0)
+ nxt = out_label if j == len(overlays) - 1 else f"eov{j}"
+ x = o["x"]
+ x = f"'{x}'" if isinstance(x, str) else x
+ # No eof_action=pass here: a PNG input is a SINGLE frame at pts 0, and eof_action=pass
+ # switches off overlay's default "hold the last frame of the secondary input", so the
+ # asset would be composited on frame 0 only and vanish for the rest of the cue (that is
+ # exactly what shipped first). eof_action=repeat (the default) holds the still for the
+ # whole timeline; enable= is what confines it to the cue's window.
+ chains.append(f"[{cur}][{label}]overlay=x={x}:y={o['y']}:"
+ f"enable='between(t,{o['start']:.3f},{o['end']:.3f})'[{nxt}]")
+ cur = nxt
+ return chains, inputs
+
+
+# --------------------------------------------------------------------------- shaping (1.15)
+# Scripts whose correct rendering needs harfbuzz-class reordering and re-clustering (Indic matras,
+# Thai/Lao mark stacking). drawtext does NOT use harfbuzz even in an --enable-libharfbuzz build, so
+# these come out wrong through drawtext on every build and must go through libass. Arabic and
+# Hebrew are NOT here: drawtext's text_shaping uses fribidi, which does bidi and Arabic joining
+# correctly -- they only join this set on a build compiled without fribidi.
+SHAPING_SCRIPTS = frozenset({"hi", "bn", "ta", "te", "kn", "ml", "gu", "pa", "si", "th", "lo", "km", "my"})
+BIDI_SCRIPTS = frozenset({"ar", "he"})
+_SHAPING_BUILD_CACHE: "Dict[str, bool]" = {}
+
+
+def drawtext_shaping() -> "Dict[str, bool]":
+ """Which shaping libraries THIS ffmpeg was built with, from -buildconf (falling back to the
+ `configuration:` line of -version). Cached per process."""
+ if _SHAPING_BUILD_CACHE:
+ return dict(_SHAPING_BUILD_CACHE)
+ text = ""
+ exe = shutil.which("ffmpeg")
+ if exe:
+ for flag in ("-buildconf", "-version"):
+ try:
+ proc = subprocess.run([exe, "-hide_banner", flag], stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT, text=True, timeout=10)
+ except (subprocess.TimeoutExpired, OSError):
+ break
+ if proc.returncode == 0 and proc.stdout.strip():
+ text = proc.stdout
+ break
+ _SHAPING_BUILD_CACHE.update({"fribidi": "--enable-libfribidi" in text,
+ "harfbuzz": "--enable-libharfbuzz" in text})
+ return dict(_SHAPING_BUILD_CACHE)
+
+
+def needs_shaping(script: str) -> bool:
+ """Whether drawtext would render `script` wrongly on this build."""
+ if script in SHAPING_SCRIPTS:
+ return True
+ return script in BIDI_SCRIPTS and not drawtext_shaping()["fribidi"]
+
+
+def font_family_of_file(path: str) -> "Optional[str]":
+ """The family name of a font FILE -- what libass wants, given a --font-file. `fc-scan` reads
+ the file directly; without fontconfig the file stem is the honest best guess."""
+ if not path or not os.path.isfile(path):
+ return None
+ exe = shutil.which("fc-scan")
+ if exe:
+ try:
+ proc = subprocess.run([exe, "--format", "%{family[0]}", path], stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL, text=True, timeout=10)
+ if proc.returncode == 0 and proc.stdout.strip():
+ return proc.stdout.strip().splitlines()[0].strip()
+ except (subprocess.TimeoutExpired, OSError):
+ pass
+ return Path(path).stem
+
+
def char_script(ch: str) -> str:
"""The script of one character: one of SCRIPTS, or "latin" for anything else (including
digits, punctuation and spaces -- they are measured and wrapped like Latin)."""
cp = ord(ch)
+ # 1.15: an emoji cluster is not Latin. detect_script() skips "emoji" the way it skips "latin",
+ # so font resolution still follows the letters around it.
+ if _is_emoji_char(ch):
+ return "emoji"
for name, ranges in _SCRIPT_RANGES:
for lo, hi in ranges:
if lo <= cp <= hi:
@@ -1812,7 +2297,7 @@ def detect_script(text: str, lang: "Optional[str]" = None) -> str:
kana = 0
for ch in text or "":
s = char_script(ch)
- if s == "latin":
+ if s in ("latin", "emoji"):
continue
if ord(ch) in range(0x3040, 0x3100) or ord(ch) in range(0x31F0, 0x3200) or ord(ch) in range(0xFF66, 0xFFA0):
kana += 1
@@ -2056,36 +2541,23 @@ def script_font_for_text(text: str, *, lang: "Optional[str]" = None, font: "Opti
def escape_drawtext(text: str) -> str:
- """Escape `text` for use as a single-quoted drawtext option value (`text='<this>'`).
-
- Every ffmpeg filter-graph special character (`\\ : % , [ ] ;`) needs a backslash
- escape regardless of the surrounding quotes -- the graph parser still splits on an
- unescaped `,`/`;` or ends an option list on an unescaped `:`/`[`/`]` even while
- "inside" a quoted value. The quote character itself has no reliable backslash
- escape at all: `\\'` and the POSIX shell `'\\''` close-insert-reopen trick both
- parse fine in a simple `-vf` chain, but silently corrupt a `-filter_complex` chain
- that uses explicit `[label]` pads -- confirmed by rendering the result: the text
- value doesn't end where the quote closes it, and trailing option names/values
- (fontfile=..., fontsize=...) leak into the rendered picture as literal text
- instead of being parsed as options. A quote is therefore dropped outright rather
- than escaped -- losing one apostrophe from a label is a fine trade for "the
- filter graph parses the way the code intends, on every call shape this codebase
- uses it in".
-
- `%` has the same problem the quote character did: `\%` is not a real escape as
- far as drawtext's own text-expansion scanner (on by default, `expansion=normal`,
- for `%{pts}`/`%{localtime}`/etc.) is concerned -- a bare backslash-escaped `%`
- always logs "Stray % near ..." (confirmed with the minimal case
- `text='100\%done'`), which is merely noisy on one ffmpeg
- build (the warning is printed, the file still gets written) but a hard filtering
- failure that writes no output at all on another. Every caller of this function
- only ever wants a literal label, never `%{...}` expansion, so `%` is dropped
- outright rather than chasing a per-build-safe escape (`expansion=none` on the
- filter would also fix it, but needs touching every drawtext= call site instead
- of the one shared helper). Control characters (newline, tab, ...) are dropped
- for the same reason: none are meaningful in a one-line burnt-in label, and
- unlike the graph-special characters above, ffmpeg's own text-expansion scanner
- -- not just the graph parser -- is involved in whether they're actually safe."""
+ """Escape a FONT NAME for a single-quoted drawtext option value (`font='<this>'`).
+
+ Since 1.15 this is no longer the route for drawn TEXT -- use drawtext_text_opts(), which puts
+ the text in a file and keeps `\'` and `%` verbatim. It remains the escape for the font-name
+ fallback, where the value is a family name that never legitimately contains a quote or a
+ percent sign.
+
+ Every ffmpeg filter-graph special character (`\\ : % , [ ] ;`) needs a backslash escape
+ regardless of the surrounding quotes -- the graph parser still splits on an unescaped `,`/`;`
+ or ends an option list on an unescaped `:`/`[`/`]` even while "inside" a quoted value. The
+ quote character itself has no reliable backslash escape at all: `\\'` and the POSIX shell
+ close-insert-reopen trick both parse fine in a simple `-vf` chain but silently corrupt a
+ `-filter_complex` chain that uses explicit `[label]` pads (confirmed by rendering the result:
+ trailing option names leak into the picture as literal text). `%` has the same problem as far
+ as drawtext's own expansion scanner is concerned. Both are therefore dropped here rather than
+ escaped -- which is exactly why drawn text no longer comes through this function.
+ """
text = re.sub(r"[\x00-\x1f\x7f]", "", text)
return (
text.replace("'", "")
@@ -2099,6 +2571,90 @@ def escape_drawtext(text: str) -> str:
)
+_DRAWTEXT_TMPDIR: "Optional[str]" = None
+_DRAWTEXT_PENDING: "Dict[str, str]" = {}
+
+
+def _drawtext_tmpdir(create: bool = True) -> str:
+ """The private, per-run directory drawn-text files live in.
+
+ tempfile.mkdtemp() creates it 0700 under a name nobody can guess, which is the whole point:
+ the 1.15.0 shape (a fixed, world-writable `/tmp/ffmpeg-skill-text` entered with
+ makedirs(exist_ok=True) and content-addressed filenames) let any other user on the machine
+ pre-create the directory or plant a symlink at a predictable name, and handed the second
+ user of a shared box a PermissionError out of filter construction instead of a `kind: input`
+ refusal. The directory is removed when the process ends, whether it succeeded or failed.
+ """
+ global _DRAWTEXT_TMPDIR
+ import tempfile
+ if _DRAWTEXT_TMPDIR and os.path.isdir(_DRAWTEXT_TMPDIR):
+ return _DRAWTEXT_TMPDIR
+ if not create:
+ # --dry-run names the path it WOULD use and creates nothing (a dry run writes nothing).
+ return os.path.join(tempfile.gettempdir(), "ffmpeg-skill-text-%d" % os.getpid())
+ import atexit
+ _DRAWTEXT_TMPDIR = tempfile.mkdtemp(prefix="ffmpeg-skill-text-")
+ atexit.register(shutil.rmtree, _DRAWTEXT_TMPDIR, True)
+ return _DRAWTEXT_TMPDIR
+
+
+def flush_drawtext_textfiles(cmd: "Sequence[str]") -> "List[str]":
+ """Write the drawn-text files this command actually names, and return their paths.
+
+ The text is registered when the filter STRING is built, but a filter string is not a run:
+ graphics.py builds the drawtext graph even on a job that is finally rendered through libass,
+ and every tool builds one under --dry-run. Writing here -- from run(), past the dry-run
+ return, against the command that is about to be executed -- is what keeps both of those from
+ leaving a file behind.
+ """
+ if not _DRAWTEXT_PENDING:
+ return []
+ joined = " ".join(str(a) for a in cmd)
+ written = []
+ for path, body in list(_DRAWTEXT_PENDING.items()):
+ # match on the unique file name, not the full path: inside a filter string the path is
+ # escaped (a Windows drive colon becomes `C\\:`, and the separators are forward slashes),
+ # so the registered spelling never appears verbatim in the command
+ if os.path.basename(path) not in joined or os.path.exists(path):
+ continue
+ # O_NOFOLLOW exists on POSIX only; the directory is private (mkdtemp, 0700) so the
+ # symlink guard is belt and braces there and unavailable on Windows
+ flags = os.O_WRONLY | os.O_CREAT | os.O_TRUNC | getattr(os, "O_NOFOLLOW", 0)
+ fd = os.open(path, flags, 0o600)
+ with os.fdopen(fd, "w", encoding="utf-8") as fh:
+ fh.write(body)
+ written.append(path)
+ return written
+
+
+def drawtext_text_opts(text: str, tmpdir: "Optional[str]" = None) -> str:
+ """`textfile=<path>:expansion=none` for drawtext -- the one route that is provably safe for
+ every character on every build shape this repo uses.
+
+ The filter-graph parser never sees the text at all: only the PATH is parsed, and
+ escape_filter_path() already handles that. `expansion=none` switches off drawtext's own
+ `%{...}` scanner, which is the reason `%` was unsafe (a bare `\%` logs "Stray %" on one build
+ and fails the whole filter chain on another). With the scanner off, `'`, `%`, `:`, `,`, `[`,
+ `]`, `;` and `\` all reach the picture verbatim -- 1.15 fixes `overlay.py --text "it's 100%
+ done"` losing both characters. Control characters are still stripped: a one-line burnt-in
+ label has no use for them.
+
+ The file is UTF-8, mode 0600, in a private per-run directory (see _drawtext_tmpdir) that is
+ removed when the process ends. It is *registered* here and written by run() only if the
+ command about to run actually names it, so --dry-run and the ASS route write nothing; a
+ printed plan therefore names a path that no longer exists once the run is over, which is the
+ same promise every other temp file in this skill makes.
+ """
+ cleaned = re.sub(r"[\x00-\x1f\x7f]", "", text or "")
+ import hashlib
+ name = "t_" + hashlib.sha256(cleaned.encode("utf-8")).hexdigest()[:16] + ".txt"
+ if tmpdir is None:
+ tmpdir = _drawtext_tmpdir(create=not STATE.dry_run)
+ path = os.path.join(tmpdir, name)
+ _DRAWTEXT_PENDING[path] = cleaned
+ return f"textfile={escape_filter_path(path)}:expansion=none"
+
+
def cfr_args(meta: Optional[Dict[str, Any]], fps: Optional[float] = None) -> List[str]:
"""Force a constant frame rate on output when the source looks VFR (or fps is given).
diff --git a/scripts/_contract.py b/scripts/_contract.py
index 3b51150..854c576 100644
--- a/scripts/_contract.py
+++ b/scripts/_contract.py
@@ -768,11 +768,11 @@ def doctor() -> Dict[str, Any]:
"ok": not missing_required and not unknown_required,
"tools": _tool_usability(state),
"gpu_encoders": _gpu_encoders(listings["encoders"]),
- "fonts": _fonts_capability(),
+ "fonts": _fonts_capability(probe=True),
}
-def _fonts_capability() -> Dict[str, Any]:
+def _fonts_capability(probe: bool = True) -> Dict[str, Any]:
"""The default drawtext family (issue #66) plus, since 1.12, one entry per script the tools
can detect: which languages this machine can actually RENDER, not just which filters exist.
@@ -781,7 +781,7 @@ def _fonts_capability() -> Dict[str, Any]:
here follows). Informational like the default font and gpu_encoders: a machine with no Thai
font is not a broken install, it is a machine that must not be asked to burn Thai captions.
"""
- from _common import SCRIPTS, font_for_script, script_font_status
+ from _common import SCRIPTS, font_for_script, script_font_status, emoji_support
font = _default_font()
result = _font_available(font)
@@ -794,7 +794,12 @@ def _fonts_capability() -> Dict[str, Any]:
# distinction the tools refuse or continue on.
status = script_font_status(script)
scripts[script] = {"status": status, "file": font_for_script(script) if status == "available" else None}
- return {"default_font": font, "status": result["status"], "detail": result["detail"], "scripts": scripts}
+ # 1.15: emoji are a separate question from the writing system, and the only honest answer is a
+ # render (an installed colour emoji font proves nothing -- libass on this build may still draw
+ # it monochrome). `probe=False` -- `contract --json --static`, and every static/JSON-only path -- skips
+ # that render, exactly as it skips the rest of the environment detection.
+ return {"default_font": font, "status": result["status"], "detail": result["detail"],
+ "scripts": scripts, "emoji": emoji_support(probe=probe)}
def _fonts_summary_line(fonts: Dict[str, Any]) -> str:
@@ -819,6 +824,16 @@ def _fonts_summary_line(fonts: Dict[str, Any]) -> str:
parts.append("no font for " + " ".join(by_state["missing"]))
if by_state["unknown"]:
parts.append("unknown (no fontconfig) " + " ".join(by_state["unknown"]))
+ emoji = fonts.get("emoji") or {}
+ mode = emoji.get("mode")
+ if mode == "color":
+ parts.append("emoji colour")
+ elif mode == "png":
+ parts.append("emoji colour (assets)")
+ elif mode == "mono":
+ parts.append("emoji monochrome -- --emoji-assets DIR for colour")
+ elif mode:
+ parts.append("emoji none")
return "; ".join(parts)
diff --git a/scripts/caption.py b/scripts/caption.py
index 0179767..e901b3e 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -38,10 +38,11 @@ import re
import sys
import unicodedata
from pathlib import Path
-from typing import List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple
from _platforms import PLATFORMS, PLATFORM_CHOICES, ass_units, resolve as resolve_platform
-from _common import STATE, brand_states_font, char_script, script_font_for_text, signed_time_arg, brand_caption_style, color_hex, load_brand, video_args, add_common, apply_common, emit, aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, fmt_srt_time, fmt_smpte_time, info, MissingFpsError, parse_time, probe, run, x264_args, X264_PRESETS, read_text_or_die, fmt_secs
+from _ass_overlay import EMOJI_SENTINEL, emoji_placeholder, ass_escape
+from _common import emoji_filter_chain, EMOJI_ASSET_HINT, emoji_asset_for, emoji_codepoint_name, emoji_support, resolve_emoji_assets, ADVANCE_EM, LATIN_EM, LEADING_VOWELS, NO_SPACE_SCRIPTS, _char_em, _is_mark, text_width_em, emoji_clusters, has_emoji, detect_script, BIDI_SCRIPTS, STATE, brand_states_font, char_script, script_font_for_text, signed_time_arg, brand_caption_style, color_hex, load_brand, video_args, add_common, apply_common, emit, aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, fmt_srt_time, fmt_smpte_time, info, MissingFpsError, parse_time, probe, run, x264_args, X264_PRESETS, read_text_or_die, fmt_secs
ALIGN = {"bottom": 2, "top": 8, "center": 5, "bottom-left": 1, "bottom-right": 3, "top-left": 7, "top-right": 9}
@@ -298,89 +299,41 @@ def word_durations_from_audio(video: str, start: float, end: float, n_words: int
return out
-# --------------------------------------------------------------------------- readable cues (1.12)
-# Average advance width per character, in em (a fraction of the font size). Proportional Latin text
-# averages a bit over half an em; CJK and Thai are drawn on a full-width grid; Arabic/Hebrew and
-# Devanagari sit in between. These are deliberately averages, not per-glyph metrics: measuring the
-# real advance needs a font parser (no stdlib one) and would still be wrong for libass's own
-# shaping, while a cue wrapped from an average is right to within a character on every line.
-# (Latin is measured per character from LATIN_EM below, not from this average.)
-ADVANCE_EM = {"ja": 1.0, "zh": 1.0, "ko": 1.0, "th": 1.0, "hi": 0.7, "ar": 0.6, "he": 0.6,
- "ru": 0.55, "el": 0.55, "latin": 0.55}
# How much of the frame width a caption line may use. libass's own default SRT margins are 10 of a
# 384-wide script (2.6 % a side); 5 % a side is the safe area every platform check in this repo uses.
SAFE_WIDTH_FRACTION = 0.9
-# Scripts written without spaces: a line breaks between any two characters.
-NO_SPACE_SCRIPTS = ("ja", "zh", "ko", "th")
-# Per-character Latin advances in em, read off DejaVu Sans (the default caption family, and close
-# enough to any other proportional sans for a wrap) and rounded UP: a capital runs 0.56-0.99 em
-# against the single 0.55 average that used to stand for all of Latin, so an all-caps caption --
-# the style most burn-ins use -- overflowed the safe area and was silently re-wrapped by libass
-# past --max-lines. Rounding up is the safe direction: libass re-wraps a too-long line, it never
-# un-wraps a short one. Characters outside the table fall back by class (0.7 uppercase/digit,
-# 0.57 lowercase and anything else Latin-ish).
-LATIN_EM = {
- ' ': 0.32, '!': 0.41, '"': 0.46, '#': 0.84, '$': 0.64, '%': 0.96, '&': 0.78, "'": 0.28,
- '(': 0.4, ')': 0.4, '*': 0.5, '+': 0.84, ',': 0.32, '-': 0.37, '.': 0.32, '/': 0.34, '0': 0.64,
- '1': 0.64, '2': 0.64, '3': 0.64, '4': 0.64, '5': 0.64, '6': 0.64, '7': 0.64, '8': 0.64,
- '9': 0.64, ':': 0.34, ';': 0.34, '<': 0.84, '=': 0.84, '>': 0.84, '?': 0.54, '@': 1.0,
- 'A': 0.69, 'B': 0.69, 'C': 0.7, 'D': 0.78, 'E': 0.64, 'F': 0.58, 'G': 0.78, 'H': 0.76,
- 'I': 0.3, 'J': 0.3, 'K': 0.66, 'L': 0.56, 'M': 0.87, 'N': 0.75, 'O': 0.79, 'P': 0.61,
- 'Q': 0.79, 'R': 0.7, 'S': 0.64, 'T': 0.62, 'U': 0.74, 'V': 0.69, 'W': 0.99, 'X': 0.69,
- 'Y': 0.62, 'Z': 0.69, '[': 0.4, '\\': 0.34, ']': 0.4, '^': 0.84, '_': 0.5, '`': 0.5, 'a': 0.62,
- 'b': 0.64, 'c': 0.55, 'd': 0.64, 'e': 0.62, 'f': 0.36, 'g': 0.64, 'h': 0.64, 'i': 0.28,
- 'j': 0.28, 'k': 0.58, 'l': 0.28, 'm': 0.98, 'n': 0.64, 'o': 0.62, 'p': 0.64, 'q': 0.64,
- 'r': 0.42, 's': 0.53, 't': 0.4, 'u': 0.64, 'v': 0.6, 'w': 0.82, 'x': 0.6, 'y': 0.6, 'z': 0.53,
- '{': 0.64, '|': 0.34, '}': 0.64, '~': 0.84
-}
-# Thai and Lao write some vowels BEFORE the consonant they belong to: the break must not land
-# between them and the base that follows.
-LEADING_VOWELS = set(range(0x0E40, 0x0E45)) | set(range(0x0EC0, 0x0EC5))
-
-
-def _is_mark(ch: str) -> bool:
- """A character that hangs off the one before it: a combining mark (any script) or one of the
- Thai/Lao vowel signs and tone marks, which are Mn/Mc but carry no combining class."""
- return unicodedata.combining(ch) != 0 or unicodedata.category(ch) in ("Mn", "Mc")
-
-
-def _char_em(ch: str) -> float:
- # CJK punctuation and the fullwidth forms (、。,!? and U+FF01-FF60) are drawn on the same
- # full-width grid as the ideographs they sit between, even though they are not "Han" to a
- # script detector -- measuring them as Latin under-counts a wrapped CJK line by a character.
- cp = ord(ch)
- # A combining mark is drawn on top of (or under) its base and advances the pen by nothing:
- # charging it a full em wrapped Thai and Devanagari lines far shorter than they needed to be.
- if unicodedata.combining(ch) != 0 or unicodedata.category(ch) == "Mn":
- return 0.0
- if 0x3000 <= cp <= 0x303F or 0xFF01 <= cp <= 0xFF60 or 0xFFE0 <= cp <= 0xFFE6:
- return 1.0
- script = char_script(ch)
- if script == "latin":
- if ch in LATIN_EM:
- return LATIN_EM[ch]
- if ch.isupper() or ch.isdigit():
- return 0.7
- return 0.57
- return ADVANCE_EM.get(script, 0.55)
-
-
-def text_width_em(text: str) -> float:
- """Width of `text` in em, from the per-script average advance table."""
- return sum(_char_em(ch) for ch in text)
+# ORPHAN_MIN_EM: one full-width CJK/Thai character plus a hair. A last line narrower than this is a
+# single stranded character -- eval 14's th1 (a lone 'ล') and dl3 (a lone '行').
+ORPHAN_MIN_EM = 1.1
def _atoms(line: str) -> List[Tuple[str, bool]]:
"""Break a line into the smallest pieces a wrap may separate -- one atom per CJK/Thai
- character, one per whitespace-delimited word otherwise -- each with whether a space stood
- before it in the original. The flag is what puts the text back together exactly as written:
- "Hello 世界" keeps its space, "世界です" gains none."""
+ character, one per emoji cluster, one per whitespace-delimited word otherwise -- each with
+ whether a space stood before it in the original. The flag is what puts the text back together
+ exactly as written: "Hello 世界" keeps its space, "世界です" gains none."""
out: List[Tuple[str, bool]] = []
word = ""
spaced = False # a space stands before the atom being built
pending = False # a space stands before the NEXT atom
attach_next = False # a leading Thai/Lao vowel is waiting for its base consonant
- for ch in line:
+ # An emoji cluster is one atom: a wrap must never land inside a ZWJ sequence, a flag pair or
+ # between a base and its skin-tone modifier (the same rule combining marks already follow).
+ clusters = {i: len(cl) for i, cl in emoji_clusters(line)}
+ i = 0
+ while i < len(line):
+ ch = line[i]
+ if i in clusters:
+ cluster = line[i:i + clusters[i]]
+ if word:
+ out.append((word, spaced))
+ word = ""
+ out.append((cluster, pending))
+ pending = False
+ attach_next = False
+ i += clusters[i]
+ continue
+ i += 1
if char_script(ch) in NO_SPACE_SCRIPTS:
if word:
out.append((word, spaced))
@@ -414,29 +367,240 @@ def _join(left: str, atom: str, spaced: bool) -> str:
return left + (" " if spaced else "") + atom
-def wrap_text(text: str, max_em: float) -> List[str]:
+def _break_spaced(first: str, second: str) -> bool:
+ """Did a space stand at the break between these two wrapped lines? Only spaced scripts put one
+ there -- a CJK/Thai break sits between two characters that were written with nothing between
+ them, and re-joining them with a space would insert a character the cue never had."""
+ if not first or not second:
+ return False
+ return char_script(first[-1]) not in NO_SPACE_SCRIPTS and char_script(second[0]) not in NO_SPACE_SCRIPTS \
+ and char_script(first[-1]) != "emoji" and char_script(second[0]) != "emoji"
+
+
+def _fix_orphans(lines: List[str], max_em: float) -> List[str]:
+ """No last line that is a single stranded atom.
+
+ Greedy wrapping leaves one character alone whenever the line before it filled exactly: eval 14
+ produced a Thai cue ending in a lone `ล` and a Japanese one ending in a lone `行`. While the
+ last line is one atom narrower than ORPHAN_MIN_EM, the last atom of the line above moves down
+ onto it -- but only while the result still fits and the line above does not become an orphan
+ itself, so a two-word cue is never made worse."""
+ lines = list(lines)
+ for _ in range(len(lines)):
+ if len(lines) < 2:
+ break
+ tail = _atoms(lines[-1])
+ if len(tail) != 1 or text_width_em(lines[-1]) >= ORPHAN_MIN_EM:
+ break
+ prev = _atoms(lines[-2])
+ if len(prev) < 2:
+ break
+ moved, spaced = prev[-1]
+ new_prev = ""
+ for atom, sp in prev[:-1]:
+ new_prev = _join(new_prev, atom, sp)
+ new_last = _join(moved, tail[0][0], _break_spaced(lines[-2], lines[-1]))
+ if text_width_em(new_last) > max_em or text_width_em(new_prev) < ORPHAN_MIN_EM:
+ break
+ lines[-2], lines[-1] = new_prev, new_last
+ return lines
+
+
+def _rebalance(lines: List[str], max_em: float) -> List[str]:
+ """Move each break to the one that minimises the widest line of the pair, without changing the
+ line count.
+
+ Greedy wrapping fills line 1 to the brim and leaves line 2 short, which is what split eval 14's
+ `"A third line the tool times for me"` mid-phrase. Only spaced scripts are rebalanced: a
+ non-spaced script has no phrase structure in its atom list, so moving the break there only
+ moves the ragged edge. A break is never placed before a punctuation-only atom."""
+ if len(lines) < 2:
+ return lines
+ out = list(lines)
+ for i in range(len(out) - 1):
+ first, second = out[i], out[i + 1]
+ tail_atoms = _atoms(second)
+ if tail_atoms:
+ tail_atoms[0] = (tail_atoms[0][0], _break_spaced(first, second))
+ atoms = _atoms(first) + tail_atoms
+ if not atoms or any(char_script(ch) in NO_SPACE_SCRIPTS for ch in first + second):
+ continue
+ best = None
+ for cut in range(1, len(atoms)):
+ if not atoms[cut][1]:
+ continue # only break where a space stood
+ if all(not ch.isalnum() for ch in atoms[cut][0]):
+ continue # never strand punctuation at the start of a line
+ a = b = ""
+ for atom, sp in atoms[:cut]:
+ a = _join(a, atom, sp)
+ for atom, sp in atoms[cut:]:
+ b = _join(b, atom, sp)
+ wa, wb = text_width_em(a), text_width_em(b)
+ if max(wa, wb) > max_em:
+ continue
+ key = (max(wa, wb), abs(wa - wb))
+ if best is None or key < best[0]:
+ best = (key, a, b)
+ if best is not None:
+ out[i], out[i + 1] = best[1], best[2]
+ return out
+
+
+def wrap_text(text: str, max_em: float, *, balance: bool = True) -> List[str]:
"""Wrap `text` to lines no wider than `max_em` em, keeping the manual breaks it already has.
An atom wider than the whole line (one very long word) is left alone on its line rather than
- cut mid-word: an over-long line is readable, a chopped word is not.
+ cut mid-word: an over-long line is readable, a chopped word is not. Two post-passes then make
+ the result readable rather than merely legal (1.15): no one-character orphan line, and for
+ spaced scripts a break chosen to minimise the widest line instead of greedily.
"""
lines: List[str] = []
for raw in text.split("\n"):
if not raw.strip():
continue
current = ""
+ chunk: List[str] = []
for atom, spaced in _atoms(raw):
candidate = _join(current, atom, spaced)
if current and text_width_em(candidate) > max_em:
- lines.append(current)
+ chunk.append(current)
current = atom
else:
current = candidate
if current:
- lines.append(current)
+ chunk.append(current)
+ if balance and len(chunk) > 1:
+ fixed = _fix_orphans(chunk, max_em)
+ rebalanced = _rebalance(fixed, max_em)
+ if len(rebalanced) == len(chunk):
+ chunk = rebalanced
+ else:
+ chunk = fixed
+ lines.extend(chunk)
return lines or [text]
+# --------------------------------------------------------------------------- emoji (1.15)
+def _cue_lines(text: str) -> List[str]:
+ return [l for l in text.split("\n")]
+
+
+def plan_emoji(cues, args, play_w, play_h, brand=None):
+ """Decide how this run draws the emoji in `cues`, and where each PNG goes.
+
+ Returns (cues, plan) where `cues` may have had its emoji replaced by EMOJI_SENTINEL (the PNG
+ route) or stripped (`--emoji none`), and `plan` is the `emoji` result key plus the overlay
+ entries the filter graph needs. `None` plan means "nothing to do": no emoji in the text.
+ """
+ clusters_all = [cl for _s, _e, t in cues for _i, cl in emoji_clusters(t)]
+ if not clusters_all:
+ return cues, None
+ assets = resolve_emoji_assets(getattr(args, "emoji_assets", None), None, brand)
+ want = getattr(args, "emoji", "auto")
+ support = emoji_support(assets, probe=True)
+ mode = support["mode"] if want == "auto" else want
+ if want == "color" and not support["libass_color"]:
+ die("--emoji color: this ffmpeg renders emoji monochrome through libass "
+ f"({support['detail']}) -- pass --emoji-assets DIR for colour, or --emoji mono",
+ kind="input")
+ if want == "png" and not assets:
+ die("--emoji png: no emoji assets directory resolved -- " + EMOJI_ASSET_HINT, kind="input")
+ plan = {"mode": mode, "count": len(clusters_all),
+ "clusters": sorted({emoji_codepoint_name(cl) for cl in clusters_all}),
+ "assets": assets, "missing": [], "overlays": []}
+ if mode == "none":
+ out = []
+ for start, end, text in cues:
+ for cl in {cl for _i, cl in emoji_clusters(text)}:
+ text = text.replace(cl, "")
+ out.append((start, end, re.sub(r"[ \t]{2,}", " ", text).strip()))
+ info("emoji: stripped from the drawn text (--emoji none)")
+ return out, plan
+ if mode in ("color", "mono"):
+ if mode == "mono":
+ info("warning: emoji rendered monochrome (no colour path on this ffmpeg; "
+ "--emoji-assets DIR for colour). " + support["detail"])
+ return cues, plan
+ # --- the PNG overlay route -------------------------------------------------------------
+ if not play_w or not play_h:
+ return cues, plan
+ scale = float(getattr(args, "emoji_scale", 1.0) or 1.0)
+ # --animate moves the TEXT (\fad/\fscx in the ASS); the PNG has to move with it, or the emoji
+ # pops in against a line that is still fading up. These match the \fad values below.
+ fade_in, fade_out = {"fade": (0.2, 0.2), "pop": (0.08, 0.12),
+ "slide": (0.15, 0.15)}.get(getattr(args, "animate", None) or "none", (0.0, 0.0))
+ size_px = args.size * play_h / 288.0
+ margin_px = args.margin * play_h / 288.0
+ line_h = size_px * 1.2
+ box_px = size_px * scale
+ align = ALIGN[args.position]
+ out_cues = []
+ for start, end, text in cues:
+ lines = _cue_lines(text)
+ n = len(lines)
+ new_lines = []
+ for i, line in enumerate(lines):
+ if align in (7, 8, 9):
+ y_top = margin_px + i * line_h
+ elif align in (4, 5, 6):
+ y_top = play_h / 2.0 - (n * line_h) / 2.0 + i * line_h
+ else:
+ y_top = play_h - margin_px - (n - i) * line_h
+ line_w = text_width_em(line, scale) * size_px
+ if align in (1, 4, 7):
+ x0 = margin_px
+ elif align in (3, 6, 9):
+ x0 = play_w - margin_px - line_w
+ else:
+ x0 = (play_w - line_w) / 2.0
+ # libass lays an RTL line out right-to-left, so the LOGICAL prefix of a cluster
+ # occupies the RIGHT end of the rendered line. Measuring the prefix from the left
+ # edge put the PNG on top of the text, mirrored, on every Arabic/Hebrew cue (1.15.0).
+ rtl = detect_script(line) in BIDI_SCRIPTS
+ rebuilt = ""
+ cursor = 0
+ for idx, cluster in emoji_clusters(line):
+ prefix = line[:idx]
+ asset = emoji_asset_for(cluster, assets)
+ name = emoji_codepoint_name(cluster)
+ if not asset:
+ if name not in plan["missing"]:
+ plan["missing"].append(name)
+ rebuilt += line[cursor:idx + len(cluster)]
+ cursor = idx + len(cluster)
+ continue
+ if rtl:
+ x = x0 + line_w - text_width_em(prefix + cluster, scale) * size_px
+ else:
+ x = x0 + text_width_em(prefix, scale) * size_px
+ y = y_top + (line_h - box_px) / 2.0
+ plan["overlays"].append({
+ "asset": asset, "cluster": name,
+ "x": int(round(max(0.0, min(x, play_w - box_px)))),
+ "y": int(round(max(0.0, min(y, play_h - box_px)))),
+ "start": round(start, 3), "end": round(end, 3), "box": int(round(box_px)),
+ "fade_in": round(min(fade_in, max(0.0, (end - start) / 2.0)), 3),
+ "fade_out": round(min(fade_out, max(0.0, (end - start) / 2.0)), 3)})
+ rebuilt += line[cursor:idx] + EMOJI_SENTINEL
+ cursor = idx + len(cluster)
+ rebuilt += line[cursor:]
+ new_lines.append(rebuilt)
+ out_cues.append((start, end, "\n".join(new_lines)))
+ # `or 60` would swallow the one value that means "no overlays at all".
+ _max = getattr(args, "emoji_max", None)
+ limit = 60 if _max is None else int(_max)
+ if len(plan["overlays"]) > limit:
+ die(f"{len(plan['overlays'])} emoji overlays would be built for this job (limit {limit}, "
+ "--emoji-max raises it); ffmpeg's filter graph and the per-frame cost both grow "
+ "linearly -- split the job, or use --emoji none", kind="input")
+ if plan["missing"]:
+ info("warning: no PNG in the assets directory for " + ", ".join(plan["missing"]) +
+ " -- those clusters are drawn by the text font instead")
+ plan["box_px"] = int(round(box_px))
+ return out_cues, plan
+
+
def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float], max_lines: int,
min_duration: float, offset: float) -> Tuple[List[Tuple[float, float, str]], dict]:
"""Shift, wrap, split and lengthen cues so they can actually be read.
@@ -447,7 +611,7 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
proportion to their text; a cue shorter than `min_duration` is lengthened, never past the next
cue's start. Returns the new cues and a count of what changed.
"""
- stats = {"shifted": 0, "wrapped": 0, "split": 0, "extended": 0, "dropped": 0}
+ stats = {"shifted": 0, "wrapped": 0, "split": 0, "extended": 0, "dropped": 0, "rebalanced": 0}
staged: List[Tuple[float, float, str]] = []
for start, end, text in cues:
if offset:
@@ -461,6 +625,8 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
lines = wrap_text(text, max_em)
if lines != [l for l in text.split("\n") if l.strip()]:
stats["wrapped"] += 1
+ if lines != wrap_text(text, max_em, balance=False):
+ stats["rebalanced"] += 1
if len(lines) > max_lines:
chunks = [lines[i:i + max_lines] for i in range(0, len(lines), max_lines)]
weights = [max(1.0, sum(len(l) for l in c)) for c in chunks]
@@ -488,7 +654,7 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
def report_layout(stats: dict) -> None:
"""One info line, only when a cue actually changed."""
- parts = [f"{stats[k]} {k}" for k in ("shifted", "wrapped", "split", "extended", "dropped") if stats.get(k)]
+ parts = [f"{stats[k]} {k}" for k in ("shifted", "wrapped", "rebalanced", "split", "extended", "dropped") if stats.get(k)]
if parts:
info("cues: " + ", ".join(parts))
@@ -628,15 +794,15 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int
]
lines = []
for start, end, text in cues:
- text = text.replace("\n", "\\N")
# ASS Dialogue text treats a literal `{...}` as an override block -- real style/animation
# commands, not literal characters. Cue text (from --text, an SRT, or ASR transcription --
# all effectively user-controlled) that happens to contain braces would otherwise be
# interpreted as those commands (\pos, \t, \fscx, ...), letting caption content reposition,
- # rescale, or recolor itself or later text instead of just being read out. No caption needs
- # a literal curly brace, so they're dropped outright, matching the "unneeded delimiter
- # character -> drop it" call already made for font names (see ass_font_name()).
- text = text.replace("{", "").replace("}", "")
+ # rescale, or recolor itself or later text instead of just being read out. libass has real
+ # escapes for the braces, so 1.15 escapes them (ass_escape) rather than deleting them:
+ # a cue that says "use {curly} braces" is read out with its braces, and still cannot open
+ # an override block. Newlines become \N in the same pass.
+ text = ass_escape(text)
fx = ""
if args.animate == "fade":
fx = "{\\fad(200,200)}"
@@ -649,7 +815,7 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int
# split each line into words and give every word an equal share of the cue (\k is in centiseconds)
dur_cs = max(1, int(round((end - start) * 100)))
segments = body.split("\\N")
- words = [w for seg in segments for w in seg.split(" ") if w]
+ words = [w for seg in segments for w in seg.split(" ") if w and w.strip(EMOJI_SENTINEL)]
# real word timings from the transcript beat both the energy estimate and the even
# split -- they are what the speaker actually did, not a proxy for it
durs = word_durations_from_timings(getattr(args, "_word_timings", None) or [], start, end, len(words))
@@ -663,8 +829,14 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int
out_segments = []
for seg in segments:
ws = [w for w in seg.split(" ") if w]
- out_segments.append(" ".join(f"{{\\kf{next(it)}}}{w}" for w in ws))
+ # An emoji placeholder is its own ZERO-duration \kf segment: the highlight sweeps
+ # past the reserved gap without spending cue time on a glyph nobody sees (rendered
+ # and confirmed -- libass keeps the full gap inside a karaoke run).
+ out_segments.append(" ".join(
+ ("{\\kf0}" + w) if not w.strip(EMOJI_SENTINEL) else f"{{\\kf{next(it)}}}{w}" for w in ws))
body = "\\N".join(out_segments)
+ if EMOJI_SENTINEL in body:
+ body = body.replace(EMOJI_SENTINEL, emoji_placeholder(getattr(args, "_emoji_box_px", size)))
lines.append(f"Dialogue: 0,{t(start)},{t(end)},Default,,0,0,0,,{fx}{body}")
with open(path, "w", encoding="utf-8-sig") as fh:
fh.write("\n".join(header + lines) + "\n")
@@ -765,6 +937,20 @@ def main() -> int:
help="keep the captions out of this destination's UI: the margin becomes the platform's safe "
"zone (TikTok's description bar, the Reels/Shorts chrome). An explicit --margin/--position wins")
sty.add_argument("--box", action="store_true", help="draw an opaque box behind text instead of an outline")
+ emo = ap.add_argument_group("emoji (1.15)")
+ emo.add_argument("--emoji", choices=["auto", "color", "png", "mono", "none"], default="auto",
+ help="how emoji in the cues are drawn: 'auto' picks the best this machine can do "
+ "(doctor --json .fonts.emoji), 'color' insists on a colour-capable libass, "
+ "'png' composites the --emoji-assets PNGs, 'mono' draws whatever glyph the text "
+ "font has, 'none' strips them")
+ emo.add_argument("--emoji-assets", metavar="DIR",
+ help="directory of emoji PNGs named by code point (1f389.png, 1f1ef-1f1f5.png) -- "
+ "Twemoji's assets/72x72 or Noto Emoji's png/128. Nothing is ever downloaded; "
+ "also read from brand.json styles.caption.emoji_assets and FFMPEG_SKILL_EMOJI_ASSETS")
+ emo.add_argument("--emoji-scale", type=float, default=1.0,
+ help="emoji box as a multiple of the line's font size (default 1.0)")
+ emo.add_argument("--emoji-max", type=int, default=60,
+ help="most emoji overlays one run may build (default 60)")
sty.add_argument("--max-lines", type=int, default=2, help="most lines one cue may occupy; a longer cue is split into consecutive cues (default 2)")
sty.add_argument("--min-duration", type=float, default=1.0, help="shortest time a cue stays on screen in seconds, never past the next cue (default 1.0)")
anim = ap.add_argument_group("animation (generates ASS; needs --text or --srt input)")
@@ -993,11 +1179,34 @@ def main() -> int:
if not args.fonts_dir:
args.fonts_dir = os.path.dirname(font_file)
- if (args.animate != "none" or args.karaoke) and not args.ass:
+ # Emoji (1.15). Decided once, on the cues the burn will actually use: libass cannot place a
+ # PNG, so the text keeps its place in the ASS (with the gap reserved) and each emoji becomes an
+ # overlay composited after the ass= filter. A cue file that has no emoji costs nothing here.
+ emoji_plan = None
+ emoji_cues = None
+ if not args.ass:
+ if args.text or args.transcribe:
+ src_cues = cues
+ elif planned_cues is not None:
+ src_cues = planned_cues
+ elif os.path.exists(srt_path or ""):
+ src_cues = parse_srt(srt_path)
+ else:
+ src_cues = []
+ if src_cues and has_emoji("\n".join(t for _s, _e, t in src_cues)):
+ emoji_cues, emoji_plan = plan_emoji(src_cues, args, play_w, play_h,
+ brand if args.brand else None)
+ # the PNG route and --emoji none both change the drawn text, so they need the generated ASS
+ force_ass = bool(emoji_plan and (emoji_plan.get("overlays") or emoji_plan.get("mode") == "none"))
+ if emoji_plan and emoji_plan.get("box_px"):
+ args._emoji_box_px = emoji_plan["box_px"]
+
+ if (args.animate != "none" or args.karaoke or force_ass) and not args.ass:
# both sources are already laid out: `cues` above, and srt_path was rewritten in place of
# the caller's file when --offset/--max-lines/--min-duration changed anything
- cues_for_ass = cues if (args.text or args.transcribe) else (
- planned_cues if planned_cues is not None else parse_srt(srt_path))
+ cues_for_ass = emoji_cues if emoji_cues is not None else (
+ cues if (args.text or args.transcribe) else (
+ planned_cues if planned_cues is not None else parse_srt(srt_path)))
if args.karaoke and not getattr(args, "_word_timings", None):
args._word_timings = whisper_word_timings(srt_path)
ass_path = args.write_ass or os.path.splitext(output)[0] + ".ass"
@@ -1040,15 +1249,40 @@ def main() -> int:
if args.fonts_dir:
vf += f":fontsdir={escape_filter_path(args.fonts_dir)}"
- cmd = ffmpeg_base() + ["-i", args.input, "-map", "0:v:0"]
+ cmd = ffmpeg_base() + ["-i", args.input]
+ chains, emoji_inputs = emoji_filter_chain(emoji_plan or {}, "vsub", "vout") if emoji_plan else ([], [])
+ if chains:
+ for spec in emoji_inputs:
+ cmd += spec
+ asset = spec[-1]
+ if asset not in STATE.plan_inputs:
+ STATE.plan_inputs.append(asset)
+ graph = ";".join([f"[0:v]{vf}[vsub]"] + chains)
+ cmd += ["-filter_complex", graph, "-map", "[vout]"]
+ else:
+ cmd += ["-map", "0:v:0", "-vf", vf]
if meta.get("audio"):
cmd += ["-map", f"0:a:{args.audio_stream}"]
- cmd += ["-vf", vf] + video_args(meta, args.crf, args.preset) + cfr_args(meta)
+ cmd += video_args(meta, args.crf, args.preset) + cfr_args(meta)
cmd += (aac_args() if meta.get("audio") else ["-an"]) + [output]
run(cmd)
result = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(result.get('duration'))})")
- emit(output, **({"notes": side_notes} if side_notes else {}))
+ extra = {"notes": side_notes} if side_notes else {}
+ if emoji_plan:
+ notes = list(extra.get("notes") or [])
+ if emoji_plan["mode"] == "mono":
+ notes.append("emoji rendered monochrome (no colour path on this ffmpeg; "
+ "--emoji-assets DIR for colour)")
+ if emoji_plan["mode"] == "none":
+ notes.append("emoji stripped from the drawn text (--emoji none)")
+ if emoji_plan["missing"]:
+ notes.append("no PNG asset for " + ", ".join(emoji_plan["missing"]))
+ if notes:
+ extra["notes"] = notes
+ extra["emoji"] = {k: v for k, v in emoji_plan.items() if k not in ("overlays", "box_px")}
+ extra["emoji"]["overlays"] = len(emoji_plan.get("overlays") or [])
+ emit(output, **extra)
return 0
diff --git a/scripts/graphics.py b/scripts/graphics.py
index f5761c5..e66e5ca 100755
--- a/scripts/graphics.py
+++ b/scripts/graphics.py
@@ -26,11 +26,20 @@ Examples:
python3 graphics.py clip.mp4 --template meme --top "when the render" --bottom "finally finishes"
"""
import argparse
+import os
+import re
import sys
from typing import List, Optional
from _platforms import PLATFORMS, PLATFORM_CHOICES, safe_margins_px, resolve as resolve_platform
-from _common import aac_args, add_common, brand_caption_style, script_font_for_text, apply_common, cfr_args, color_hex, default_font_file, default_output, die, emit, escape_drawtext, escape_filter_path, ffmpeg_base, info, load_brand, parse_time, probe, run, run_keeping_subtitles, video_args, drawtext_boxborderw, X264_PRESETS, time_arg, fmt_secs
+from _common import (aac_args, add_common, brand_caption_style, script_font_for_text, apply_common, cfr_args,
+ color_hex, default_font_file, default_output, die, emit, escape_drawtext, escape_filter_path,
+ ffmpeg_base, info, load_brand, parse_time, probe, run, run_keeping_subtitles, video_args,
+ drawtext_boxborderw, X264_PRESETS, time_arg, fmt_secs, STATE, drawtext_text_opts,
+ LANGUAGE_NAMES, needs_shaping, detect_script, BIDI_SCRIPTS, font_family_of_file, font_family_for_script, has_emoji,
+ emoji_clusters, emoji_codepoint_name, char_script, emoji_filter_chain, emoji_asset_for, emoji_support, resolve_emoji_assets,
+ EMOJI_ASSET_HINT, text_width_em, drawtext_shaping)
+from _ass_overlay import text_overlay_ass, EMOJI_SENTINEL
TEMPLATES = ["lower-third", "title", "chapter", "progress", "countdown", "bug", "sticker", "hook", "meme"]
@@ -82,6 +91,23 @@ def main() -> int:
ap.add_argument("--lang", help="language code of the text (e.g. ja, zh, ko): the hint that says whether Han-only "
"text is Chinese, Japanese or Korean when a font is picked by script")
ap.add_argument("--scale", type=float, default=1.0, help="size multiplier (default 1)")
+ ap.add_argument("--text-render", choices=["auto", "ass", "drawtext"], default="auto",
+ help="which renderer draws the template's text: 'auto' (default) uses libass for "
+ "scripts drawtext cannot shape (Devanagari, Bengali, Tamil, Thai, Lao ...) and for "
+ "emoji overlays, and drawtext for everything else -- Latin/CJK/Arabic frames are "
+ "pixel-identical to 1.14 (the drawtext command itself changed: the label is "
+ "passed as textfile=, not text=); 'ass' always uses libass; 'drawtext' forces the old renderer and is "
+ "refused for a script it cannot shape")
+ ap.add_argument("--write-ass", metavar="PATH",
+ help="where to save the generated ASS when the libass route is used (default: <output stem>_gfx.ass)")
+ emo = ap.add_argument_group("emoji (1.15)")
+ emo.add_argument("--emoji", choices=["auto", "color", "png", "mono", "none"], default="auto",
+ help="how emoji in the template text are drawn (see caption.py --emoji)")
+ emo.add_argument("--emoji-assets", metavar="DIR",
+ help="directory of emoji PNGs named by code point (1f389.png); nothing is ever downloaded")
+ emo.add_argument("--emoji-scale", type=float, default=1.0,
+ help="emoji box as a multiple of the line's font size (default 1.0)")
+ emo.add_argument("--emoji-max", type=int, default=60, help="most emoji overlays one run may build (default 60)")
ap.add_argument("--crf", type=int, default=18)
ap.add_argument("--preset", default="medium", choices=X264_PRESETS)
add_common(ap)
@@ -144,6 +170,92 @@ def main() -> int:
filters: List[str] = []
fade_a = f"if(lt(t,{s:.3f}+0.3),(t-{s:.3f})/0.3,if(gt(t,{e:.3f}-0.3),({e:.3f}-t)/0.3,1))"
+ # --- which renderer draws the text (1.15) --------------------------------------------------
+ # drawtext does bidi and Arabic joining on a fribidi build, but it never reorders or
+ # re-clusters (no harfbuzz), so Indic matras and Thai/Lao mark stacking come out wrong on
+ # every build. Those scripts -- and any run that has to reserve a gap for an emoji PNG -- go
+ # through libass instead; everything else keeps the exact drawtext graph 1.14 produced.
+ all_text = " ".join(t for t in (args.name, args.title, args.subtitle, args.text, args.top, args.bottom) if t)
+ shaping = needs_shaping(_script)
+ emoji_assets = resolve_emoji_assets(args.emoji_assets, None, brand if args.brand else None)
+ emoji_mode = None
+ emoji_overlays: List[dict] = []
+ if has_emoji(all_text):
+ support = emoji_support(emoji_assets, probe=True)
+ emoji_mode = support["mode"] if args.emoji == "auto" else args.emoji
+ if args.emoji == "color" and not support["libass_color"]:
+ die("--emoji color: this ffmpeg renders emoji monochrome through libass "
+ f"({support['detail']}) -- pass --emoji-assets DIR for colour, or --emoji mono", kind="input")
+ if args.emoji == "png" and not emoji_assets:
+ die("--emoji png: no emoji assets directory resolved -- " + EMOJI_ASSET_HINT, kind="input")
+ if args.text_render == "drawtext" and shaping:
+ die(f"{LANGUAGE_NAMES.get(_script, _script)} text cannot be shaped by drawtext on any ffmpeg "
+ "build (the marks are reordered by harfbuzz, which drawtext does not use): drop "
+ "--text-render drawtext to render it through libass, or draw it with caption.py",
+ kind="input")
+ route = "ass" if (args.text_render == "ass" or
+ (args.text_render == "auto" and (shaping or emoji_mode == "png"))) else "drawtext"
+ # drawtext loads exactly ONE font file and has no fallback chain, so "whatever glyph the text
+ # font has" for an emoji is an empty box on DejaVu Sans and on every script font: reporting
+ # mode "mono" from the drawtext route is a claim the frame does not keep. libass DOES have a
+ # fallback chain, so an auto run routes there instead; a run that pinned --text-render
+ # drawtext degrades to "none" (strip) and says so rather than drawing tofu.
+ if emoji_mode == "mono" and route == "drawtext":
+ if args.text_render == "auto":
+ route = "ass"
+ else:
+ emoji_mode = "none"
+ info("emoji: --text-render drawtext has no font fallback chain, so the cluster would "
+ "be drawn as an empty box -- stripped from the text instead "
+ "(--text-render auto renders it monochrome through libass)")
+ if emoji_mode == "mono":
+ info("warning: emoji rendered monochrome (no colour path on this ffmpeg; "
+ "--emoji-assets DIR for colour). " + support["detail"])
+ if emoji_mode == "none":
+ if not "".join(ch for ch in all_text if char_script(ch) != "emoji").strip():
+ die("the template's text is nothing but emoji and this machine can draw none of them "
+ "(no glyph, no --emoji-assets DIR): that frame would be blank, which is not a "
+ "delivery -- " + EMOJI_ASSET_HINT, kind="input")
+
+ def _strip_emoji(text):
+ if not text:
+ return text
+ for _i, cl in emoji_clusters(text):
+ text = text.replace(cl, "")
+ return re.sub(r"[ \t]{2,}", " ", text).strip()
+
+ for _attr in ("name", "title", "subtitle", "text", "top", "bottom"):
+ setattr(args, _attr, _strip_emoji(getattr(args, _attr, None)))
+ all_text = " ".join(t for t in (args.name, args.title, args.subtitle, args.text,
+ args.top, args.bottom) if t)
+ info("emoji: stripped from the drawn text (--emoji none)")
+ elements: List[dict] = []
+
+ def ass_font_family() -> "Optional[str]":
+ explicit = args.font_file or brand.get("font_file")
+ if explicit:
+ return font_family_of_file(explicit) or args.font or brand.get("font")
+ if script_file:
+ return font_family_of_file(script_file) or font_family_for_script(_script)
+ return args.font or brand.get("font", "DejaVu Sans")
+
+ def ass_fonts_dir() -> "Optional[str]":
+ explicit = args.font_file or brand.get("font_file")
+ if explicit:
+ return os.path.dirname(os.path.abspath(explicit))
+ if script_file:
+ return os.path.dirname(os.path.abspath(script_file))
+ return None
+
+ def add_text(text, drawtext, *, target=None, **el):
+ """One line of template text: a drawtext filter on the old route, an ASS element on the
+ new one. The geometry is computed identically either way."""
+ if route != "ass":
+ (filters if target is None else target).append(drawtext)
+ return
+ el["text"] = text
+ elements.append(el)
+
extra_inputs: List[str] = []
fc: List[str] = [] # filter_complex chains (used by templates that need animated boxes)
if 0 < min(W, H) < 64: # 0x0 is a dry-run probe of an intermediate that does not exist yet
@@ -163,11 +275,31 @@ def main() -> int:
fc.append(f"color=c=0x{primary}:s={int(base * 0.012)}x{bar_h}:r={meta['video'].get('fps') or 30:g},format=rgba[acc]")
fc.append(f"[0:v][bar]overlay=x='{x_expr}':y={y0}:{en}:eof_action=pass[v1]")
fc.append(f"[v1][acc]overlay=x='{x_expr}':y={y0}:{en}:eof_action=pass[v2]")
- tx = f"({x_expr})+{int(base * 0.035)}"
- chain = f"drawtext=text='{escape_drawtext(args.name)}':{fo}:fontsize={h1}:fontcolor={ff_color(text_c)}:x='{tx}':y={y0 + pad}:{en}"
- if args.title:
- chain += f",drawtext=text='{escape_drawtext(args.title)}':{fo}:fontsize={h2}:fontcolor={ff_color(primary)}:x='{tx}':y={y0 + pad + h1 + pad // 2}:{en}"
- fc.append(f"[v2]{chain}[vout]")
+ tx_pad = int(base * 0.035)
+ tx = f"({x_expr})+{tx_pad}"
+ x_rest, x_off = m_left + tx_pad, -bar_w + tx_pad
+ draws = []
+ for text, fs, colour, ty in ((args.name, h1, text_c, y0 + pad),
+ (args.title, h2, primary, y0 + pad + h1 + pad // 2)):
+ if not text:
+ continue
+ draws.append(f"drawtext={drawtext_text_opts(text)}:{fo}:fontsize={fs}:"
+ f"fontcolor={ff_color(colour)}:x='{tx}':y={ty}:{en}")
+ if route == "ass":
+ # the same three phases the bar itself slides through, as \move Dialogues
+ elements.append(dict(text=text, size=fs, color=colour, font=ass_font_family(),
+ align=7, x=x_rest, y=ty, outline=max(1.0, fs / 16.0),
+ outline_color="000000", start=s, end=min(e, s + 0.4),
+ move=(x_off, ty, x_rest, ty, 0, 400), x_expr=tx))
+ elements.append(dict(text=text, size=fs, color=colour, font=ass_font_family(),
+ align=7, x=x_rest, y=ty, outline=max(1.0, fs / 16.0),
+ outline_color="000000", start=min(e, s + 0.4), end=max(s, e - 0.3),
+ x_expr=tx))
+ elements.append(dict(text=text, size=fs, color=colour, font=ass_font_family(),
+ align=7, x=x_rest, y=ty, outline=max(1.0, fs / 16.0),
+ outline_color="000000", start=max(s, e - 0.3), end=e,
+ move=(x_rest, ty, x_off, ty, 0, 300), x_expr=tx))
+ fc.append(f"[v2]{','.join(draws)}[vout]" if route != "ass" else "[v2]null[vout]")
elif args.template == "title":
if not args.title:
@@ -175,10 +307,18 @@ def main() -> int:
h1 = int(base * 0.11)
h2 = int(base * 0.045)
filters.append(f"drawbox=x=0:y=0:w=iw:h=ih:color={ff_color(bg, 0.55)}:t=fill:{en}")
- filters.append(f"drawtext=text='{escape_drawtext(args.title)}':{fo}:fontsize={h1}:fontcolor={ff_color(text_c)}:x=(w-text_w)/2:y=(h-text_h)/2-{h2 if args.subtitle else 0}:alpha='{fade_a}':{en}")
+ add_text(args.title,
+ f"drawtext={drawtext_text_opts(args.title)}:{fo}:fontsize={h1}:fontcolor={ff_color(text_c)}:x=(w-text_w)/2:y=(h-text_h)/2-{h2 if args.subtitle else 0}:alpha='{fade_a}':{en}",
+ size=h1, color=text_c, font=ass_font_family(), align=5, x=W / 2,
+ y=H / 2 - (h2 if args.subtitle else 0), outline=max(1.0, h1 / 20.0),
+ outline_color="000000", start=s, end=e, fade=(300, 300))
filters.append(f"drawbox=x=(iw-{int(base * 0.12)})/2:y=(ih)/2+{h1 // 2 + (0 if args.subtitle else 0)}:w={int(base * 0.12)}:h={max(2, int(base * 0.006))}:color={ff_color(primary)}:t=fill:{en}")
if args.subtitle:
- filters.append(f"drawtext=text='{escape_drawtext(args.subtitle)}':{fo}:fontsize={h2}:fontcolor={ff_color(primary)}:x=(w-text_w)/2:y=(h-text_h)/2+{h1 // 2 + int(base * 0.03)}:alpha='{fade_a}':{en}")
+ add_text(args.subtitle,
+ f"drawtext={drawtext_text_opts(args.subtitle)}:{fo}:fontsize={h2}:fontcolor={ff_color(primary)}:x=(w-text_w)/2:y=(h-text_h)/2+{h1 // 2 + int(base * 0.03)}:alpha='{fade_a}':{en}",
+ size=h2, color=primary, font=ass_font_family(), align=5, x=W / 2,
+ y=H / 2 + h1 // 2 + int(base * 0.03), outline=max(1.0, h2 / 20.0),
+ outline_color="000000", start=s, end=e, fade=(300, 300))
elif args.template in ("chapter", "bug"):
if not args.title:
@@ -190,7 +330,16 @@ def main() -> int:
ye = f"{m_top}" if "top" in pos else f"h-text_h-{m_bottom}"
box_color = ff_color(primary if args.template == "chapter" else bg, 0.9 if args.template == "chapter" else 0.7)
txt_color = ff_color(bg if args.template == "chapter" else text_c)
- filters.append(f"drawtext=text='{escape_drawtext(args.title)}':{fo}:fontsize={fs}:fontcolor={txt_color}:x={xe}:y={ye}:box=1:boxcolor={box_color}:boxborderw={drawtext_boxborderw(pady, padx)}:alpha='{fade_a}':{en}")
+ box_hex = primary if args.template == "chapter" else bg
+ txt_hex = bg if args.template == "chapter" else text_c
+ align = (7 if "left" in pos else 9) if "top" in pos else (1 if "left" in pos else 3)
+ add_text(args.title,
+ f"drawtext={drawtext_text_opts(args.title)}:{fo}:fontsize={fs}:fontcolor={txt_color}:x={xe}:y={ye}:box=1:boxcolor={box_color}:boxborderw={drawtext_boxborderw(pady, padx)}:alpha='{fade_a}':{en}",
+ size=fs, color=txt_hex, font=ass_font_family(), align=align,
+ x=(m_left if "left" in pos else W - m_right),
+ y=(m_top if "top" in pos else H - m_bottom),
+ box=True, box_color=box_hex, box_alpha=(0x19 if args.template == "chapter" else 0x4C),
+ outline=float(pady), outline_color=box_hex, start=s, end=e, fade=(300, 300))
elif args.template == "progress":
h = max(3, int(base * 0.008))
@@ -214,9 +363,19 @@ def main() -> int:
xe = f"{m_left}" if "left" in pos else f"w-text_w-{m_right}"
ye = (f"{m_top}+{rise}*(1-{pop})" if "top" in pos else f"h-text_h-{m_bottom}-{rise}*(1-{pop})")
alpha = f"min({pop},{fade_a})"
- filters.append(f"drawtext=text='{escape_drawtext(args.text)}':{fo}:fontsize={fs}:fontcolor={ff_color(bg)}:"
- f"x={xe}:y='{ye}':box=1:boxcolor={ff_color(primary, 0.95)}:boxborderw={drawtext_boxborderw(pady, padx)}:"
- f"alpha='{alpha}':{en}")
+ align = (7 if "left" in pos else 9) if "top" in pos else (1 if "left" in pos else 3)
+ y_rest = m_top if "top" in pos else H - m_bottom
+ y_start = y_rest + rise if "top" in pos else y_rest + rise
+ add_text(args.text,
+ f"drawtext={drawtext_text_opts(args.text)}:{fo}:fontsize={fs}:fontcolor={ff_color(bg)}:"
+ f"x={xe}:y='{ye}':box=1:boxcolor={ff_color(primary, 0.95)}:boxborderw={drawtext_boxborderw(pady, padx)}:"
+ f"alpha='{alpha}':{en}",
+ size=fs, color=bg, font=ass_font_family(), align=align,
+ x=(m_left if "left" in pos else W - m_right), y=y_rest,
+ box=True, box_color=primary, box_alpha=0x0D, outline=float(pady),
+ outline_color=primary, start=s, end=e, fade=(250, 250),
+ move=(m_left if "left" in pos else W - m_right, y_start,
+ m_left if "left" in pos else W - m_right, y_rest, 0, 250))
elif args.template == "hook":
# The opener: a full-width card over the first --duration seconds with a thin bar along
@@ -232,8 +391,11 @@ def main() -> int:
band_h = int(base * 0.30)
y0 = (H - band_h) // 2
filters.append(f"drawbox=x=0:y={y0}:w=iw:h={band_h}:color={ff_color(bg, 0.78)}:t=fill:{hen}")
- filters.append(f"drawtext=text='{escape_drawtext(args.title)}':{fo}:fontsize={h1}:fontcolor={ff_color(text_c)}:"
- f"x=(w-text_w)/2:y=(h-text_h)/2:{hen}")
+ add_text(args.title,
+ f"drawtext={drawtext_text_opts(args.title)}:{fo}:fontsize={h1}:fontcolor={ff_color(text_c)}:"
+ f"x=(w-text_w)/2:y=(h-text_h)/2:{hen}",
+ size=h1, color=text_c, font=ass_font_family(), align=5, x=W / 2, y=H / 2,
+ outline=max(1.0, h1 / 20.0), outline_color="000000", start=s, end=he)
filters.append(f"drawbox=x=0:y=0:w='iw*max(0,1-(t-{s:.3f})/{max(0.001, he - s):.3f})':h={bar_h}:"
f"color={ff_color(primary)}:t=fill:{hen}")
@@ -248,8 +410,13 @@ def main() -> int:
for text, y in ((args.top, f"{m_top}"), (args.bottom, f"h-text_h-{m_bottom}")):
if not text:
continue
- filters.append(f"drawtext=text='{escape_drawtext(text.upper())}':{fo}:fontsize={fs}:fontcolor={white}:"
- f"borderw={bw}:bordercolor={black}:x=(w-text_w)/2:y={y}:{en}")
+ add_text(text.upper(),
+ f"drawtext={drawtext_text_opts(text.upper())}:{fo}:fontsize={fs}:fontcolor={white}:"
+ f"borderw={bw}:bordercolor={black}:x=(w-text_w)/2:y={y}:{en}",
+ size=fs, color="FFFFFF", font=ass_font_family(), bold=True,
+ align=(8 if y == f"{m_top}" else 2), x=W / 2,
+ y=(m_top if y == f"{m_top}" else H - m_bottom),
+ outline=float(bw), outline_color="000000", start=s, end=e)
elif args.template == "countdown":
n = args.count_from
@@ -259,11 +426,114 @@ def main() -> int:
ks = s + (n - k) * seg
ke = ks + seg
pulse = f"1-0.15*min(1,(t-{ks:.3f})/{seg * 0.5:.3f})"
- filters.append(f"drawtext=text='{k}':{fo}:fontsize={fs}:fontcolor={ff_color(primary)}:borderw={max(2, fs // 40)}:bordercolor={ff_color(bg)}:x=(w-text_w)/2:y=(h-text_h)/2:alpha='{pulse}':enable='between(t,{ks:.3f},{ke:.3f})'")
+ add_text(str(k),
+ f"drawtext=text='{k}':{fo}:fontsize={fs}:fontcolor={ff_color(primary)}:borderw={max(2, fs // 40)}:bordercolor={ff_color(bg)}:x=(w-text_w)/2:y=(h-text_h)/2:alpha='{pulse}':enable='between(t,{ks:.3f},{ke:.3f})'",
+ size=fs, color=primary, font=ass_font_family(), align=5, x=W / 2, y=H / 2,
+ outline=float(max(2, fs // 40)), outline_color=bg, start=ks, end=ke,
+ scale_t=(0, seg * 500, 115))
output = args.output or default_output(args.input, "gfx")
+ ass_path = None
+ emoji_result = None
+ if route == "ass":
+ scale_em = float(args.emoji_scale or 1.0)
+ missing: List[str] = []
+ seen: List[str] = []
+ for el in elements:
+ el["box_px"] = int(round(el["size"] * scale_em))
+ line = el["text"]
+ clusters = emoji_clusters(line)
+ if not clusters or emoji_mode not in ("png",):
+ continue
+ line_w = text_width_em(line, scale_em) * el["size"]
+ align = int(el.get("align", 7))
+ left = el["x"] if align in (7, 4, 1) else (
+ el["x"] - line_w / 2.0 if align in (8, 5, 2) else el["x"] - line_w)
+ line_h = el["size"] * 1.2
+ y_top = el["y"] if align in (7, 8, 9) else (
+ el["y"] - line_h / 2.0 if align in (4, 5, 6) else el["y"] - line_h)
+ # An RTL line is rendered right-to-left: measure the suffix, not the logical prefix.
+ rtl = detect_script(line) in BIDI_SCRIPTS
+ rebuilt, cursor = "", 0
+ for idx, cluster in clusters:
+ name = emoji_codepoint_name(cluster)
+ if name not in seen:
+ seen.append(name)
+ asset = emoji_asset_for(cluster, emoji_assets)
+ if not asset:
+ if name not in missing:
+ missing.append(name)
+ rebuilt += line[cursor:idx + len(cluster)]
+ cursor = idx + len(cluster)
+ continue
+ if rtl:
+ prefix_px = line_w - text_width_em(line[:idx] + cluster, scale_em) * el["size"]
+ else:
+ prefix_px = text_width_em(line[:idx], scale_em) * el["size"]
+ if el.get("x_expr"):
+ # the lower-third slides: the emoji rides the same expression the bar does
+ x = f"({el['x_expr']})+{prefix_px:.0f}"
+ else:
+ x = int(round(max(0.0, min(left + prefix_px, W - el["box_px"]))))
+ emoji_overlays.append({"asset": asset, "cluster": name, "x": x,
+ "y": int(round(max(0.0, min(y_top + (line_h - el["box_px"]) / 2.0,
+ H - el["box_px"])))),
+ "start": round(el["start"], 3), "end": round(el["end"], 3),
+ "box": el["box_px"],
+ # every template fades its text in and out over 0.3 s
+ # (fade_a above); the PNG rides the same envelope.
+ "fade_in": round(min(0.3, max(0.0, (el["end"] - el["start"]) / 2.0)), 3),
+ "fade_out": round(min(0.3, max(0.0, (el["end"] - el["start"]) / 2.0)), 3)})
+ rebuilt += line[cursor:idx] + EMOJI_SENTINEL
+ cursor = idx + len(cluster)
+ el["text"] = rebuilt + line[cursor:]
+ # `or 60` would swallow --emoji-max 0, the one value meaning "none at all".
+ _max = 60 if args.emoji_max is None else int(args.emoji_max)
+ if len(emoji_overlays) > _max:
+ die(f"{len(emoji_overlays)} emoji overlays would be built for this job "
+ f"(limit {_max}, --emoji-max raises it); ffmpeg's filter graph and the "
+ "per-frame cost both grow linearly -- split the job, or use --emoji none", kind="input")
+ if missing:
+ info("warning: no PNG in the assets directory for " + ", ".join(missing))
+ if emoji_mode:
+ emoji_result = {"mode": emoji_mode, "count": len(emoji_clusters(all_text)),
+ "clusters": sorted(seen) or sorted({emoji_codepoint_name(cl) for _i, cl in emoji_clusters(all_text)}),
+ "assets": emoji_assets, "missing": missing,
+ "overlays": len(emoji_overlays)}
+ ass_path = args.write_ass or os.path.splitext(output)[0] + "_gfx.ass"
+ if STATE.dry_run:
+ info(f"[dry-run] would write {ass_path} ({len(elements)} text elements)")
+ else:
+ text_overlay_ass(elements, play_w=W, play_h=H, path=ass_path, fonts_dir=ass_fonts_dir())
+ info(f"wrote {ass_path} ({len(elements)} text elements, rendered through libass)")
+ elif emoji_mode:
+ emoji_result = {"mode": emoji_mode, "count": len(emoji_clusters(all_text)),
+ "clusters": sorted({emoji_codepoint_name(cl) for _i, cl in emoji_clusters(all_text)}),
+ "assets": emoji_assets, "missing": [], "overlays": 0}
+
cmd = ffmpeg_base() + ["-i", args.input]
- if fc:
+ if route == "ass" or emoji_overlays:
+ chains = list(fc) if fc else [f"[0:v]{','.join(filters) if filters else 'null'}[vout]"]
+ last = "vout"
+ if route == "ass":
+ vf = f"ass={escape_filter_path(ass_path)}"
+ fdir = ass_fonts_dir()
+ if fdir:
+ vf += f":fontsdir={escape_filter_path(fdir)}"
+ chains.append(f"[{last}]{vf}[vtxt]")
+ last = "vtxt"
+ eo, emoji_inputs = emoji_filter_chain({"overlays": emoji_overlays}, last, "vfinal", first_input=1)
+ for spec in emoji_inputs:
+ cmd += spec
+ asset = spec[-1]
+ if asset not in STATE.plan_inputs:
+ STATE.plan_inputs.append(asset)
+ if eo:
+ chains += eo
+ last = "vfinal"
+ cmd += ["-filter_complex", ";".join(chains), "-map", f"[{last}]",
+ "-map", f"0:a:{args.audio_stream}?"]
+ elif fc:
cmd += ["-filter_complex", ";".join(fc), "-map", "[vout]", "-map", f"0:a:{args.audio_stream}?"]
else:
cmd += ["-vf", ",".join(filters), "-map", "0:v:0", "-map", f"0:a:{args.audio_stream}?"]
@@ -272,7 +542,13 @@ def main() -> int:
dropped_streams = run_keeping_subtitles(cmd, output)
r = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(r['duration'])}, {args.template})")
- emit(output, template=args.template, dropped_non_av_streams=dropped_streams)
+ extra = {"template": args.template, "dropped_non_av_streams": dropped_streams,
+ "text_renderer": route, "script": _script}
+ if ass_path:
+ extra["ass"] = ass_path
+ if emoji_result:
+ extra["emoji"] = emoji_result
+ emit(output, **extra)
return 0
diff --git a/scripts/overlay.py b/scripts/overlay.py
index 013bdfd..812e01e 100755
--- a/scripts/overlay.py
+++ b/scripts/overlay.py
@@ -25,7 +25,7 @@ import sys
from typing import List, Optional
from _platforms import PLATFORMS, PLATFORM_CHOICES, safe_margins_px, resolve as resolve_platform
-from _common import STATE, script_font_for_text, load_brand, video_args, add_common, apply_common, default_font_file, emit, aac_args, cfr_args, default_output, die, escape_drawtext, escape_filter_path, ffmpeg_base, info, parse_time, probe, run, run_keeping_subtitles, validate_color, x264_args, X264_PRESETS, time_arg, fmt_secs
+from _common import STATE, script_font_for_text, drawtext_text_opts, needs_shaping, LANGUAGE_NAMES, has_emoji, detect_script, emoji_clusters, load_brand, video_args, add_common, apply_common, default_font_file, emit, aac_args, cfr_args, default_output, die, escape_drawtext, escape_filter_path, ffmpeg_base, info, parse_time, probe, run, run_keeping_subtitles, validate_color, x264_args, X264_PRESETS, time_arg, fmt_secs
# Per-edge margins: a platform's UI does not cover the same fraction of every edge (TikTok's
# like column is 14 % of the width, its description block 22 % of the height), so a position
@@ -126,6 +126,15 @@ def main() -> int:
txt.add_argument("--border", type=int, default=2, help="text outline width (default 2)")
txt.add_argument("--border-color", default="black")
txt.add_argument("--box", action="store_true", help="draw a translucent box behind the text")
+ emo = ap.add_argument_group("emoji (1.15)")
+ emo.add_argument("--emoji", choices=["auto", "color", "png", "mono", "none"], default="auto",
+ help="how emoji in --text are drawn. overlay.py draws through drawtext, which cannot load "
+ "a colour emoji font at all, so only 'mono'/'none' render here -- 'png'/'color' name "
+ "caption.py/graphics.py instead")
+ emo.add_argument("--emoji-assets", metavar="DIR", help="directory of emoji PNGs named by code point "
+ "(used by caption.py/graphics.py; overlay.py has no PNG route)")
+ emo.add_argument("--emoji-scale", type=float, default=1.0, help="emoji box as a multiple of the font size")
+ emo.add_argument("--emoji-max", type=int, default=60, help="most emoji overlays one run may build")
txt.add_argument("--box-color", default="[email protected]")
enc = ap.add_argument_group("encoding")
enc.add_argument("--crf", type=int, default=18)
@@ -154,6 +163,28 @@ def main() -> int:
args.font = brand.get("font", args.font)
if not args.font_file and brand.get("font_file"):
args.font_file = brand["font_file"]
+ if args.text:
+ # 1.15: drawtext never reorders or re-clusters (no harfbuzz), so Devanagari matras and
+ # Thai/Lao mark stacking come out wrong on EVERY build. A wrong frame is not a delivery:
+ # refuse and name the two tools that render the script correctly through libass.
+ _sc = detect_script(args.text, getattr(args, "lang", None))
+ if needs_shaping(_sc):
+ die(f"{LANGUAGE_NAMES.get(_sc, _sc)} text cannot be shaped by drawtext on any ffmpeg build "
+ "(the marks are reordered by harfbuzz, which drawtext does not use): draw it with "
+ "caption.py (--text cues, burned through libass) or graphics.py (--template with "
+ "--text-render ass) instead", kind="input")
+ if has_emoji(args.text):
+ if args.emoji in ("png", "color"):
+ die(f"--emoji {args.emoji}: overlay.py draws text with drawtext, which cannot load a colour "
+ "emoji font and cannot place a PNG inside a line -- use caption.py (--emoji-assets) for "
+ "cues or graphics.py (--template) for a title card", kind="input")
+ if args.emoji == "none":
+ for _i, _cl in list(reversed(emoji_clusters(args.text))):
+ args.text = args.text[:_i] + args.text[_i + len(_cl):]
+ info("emoji: stripped from the drawn text (--emoji none)")
+ else:
+ info("warning: emoji are drawn by the text font here (monochrome at best); "
+ "caption.py/graphics.py composite colour PNGs with --emoji-assets")
if args.text and not args.font_file:
# 1.12: non-Latin overlay text picks a font by script, so a title in Japanese, Korean,
# Arabic ... draws glyphs instead of boxes. drawtext does not shape or reorder RTL text --
@@ -276,7 +307,9 @@ def main() -> int:
cmd += ["-shortest"]
else:
x, y = position_exprs(args.position, args.margin, text_mode=True, margins=safe_margins)
- opts = [f"text='{escape_drawtext(args.text)}'", f"fontsize={args.font_size}", f"x={x}", f"y={y}",
+ # 1.15: the text goes in a FILE with expansion off, so `'` and `%` survive verbatim
+ # (they used to be dropped by escape_drawtext) and no character can reach the graph parser.
+ opts = [drawtext_text_opts(args.text), f"fontsize={args.font_size}", f"x={x}", f"y={y}",
f"borderw={args.border}", f"bordercolor={args.border_color}"]
if args.font_file:
opts.append(f"fontfile={escape_filter_path(args.font_file)}")
diff --git a/tests/fixtures/mcp_tools.json b/tests/fixtures/mcp_tools.json
index 4bc35d5..ec577d6 100644
--- a/tests/fixtures/mcp_tools.json
+++ b/tests/fixtures/mcp_tools.json
@@ -148,6 +148,10 @@
"color",
"crf",
"dry_run",
+ "emoji",
+ "emoji_assets",
+ "emoji_max",
+ "emoji_scale",
"fast",
"font",
"fonts_dir",
@@ -483,6 +487,10 @@
"crf",
"dry_run",
"duration",
+ "emoji",
+ "emoji_assets",
+ "emoji_max",
+ "emoji_scale",
"end",
"fast",
"font",
@@ -508,9 +516,11 @@
"template",
"text",
"text_color",
+ "text_render",
"timeout",
"title",
- "top"
+ "top",
+ "write_ass"
],
"required": []
},
@@ -747,6 +757,10 @@
"codec",
"crf",
"dry_run",
+ "emoji",
+ "emoji_assets",
+ "emoji_max",
+ "emoji_scale",
"end",
"fade",
"fast",
diff --git a/tests/test_all.py b/tests/test_all.py
index 4bdca3a..260d932 100644
--- a/tests/test_all.py
+++ b/tests/test_all.py
@@ -2956,7 +2956,9 @@ class FFmpegSkillTests(unittest.TestCase):
# --animate fade legitimately prepends its own "{\fad(200,200)}" override block; only the
# cue-text-derived braces from the hostile payload must be gone.
self.assertNotIn("{\\pos(0,0)\\fscx500}", dialogue, "cue text must not be able to open a real ASS override block")
- self.assertIn("\\pos(0,0)\\fscx500INJECTED", dialogue, "the rest of the cue text still renders, just as literal (now brace-free) text")
+ self.assertIn("\\{\\pos(0,0)\\fscx500\\}INJECTED", dialogue,
+ "the braces the user typed are ESCAPED (libass \\{ / \\}), not deleted: the cue "
+ "reads out exactly what was written and still cannot open an override block")
def test_caption_srt_blank_line_in_cue_text_does_not_split_the_block(self):
"""parse_text_cues() turns a bare '|' into a newline (a documented way to write a two-line
@@ -2991,24 +2993,28 @@ class FFmpegSkillTests(unittest.TestCase):
self.assertEqual(parsed[0][2], "Hello there", "the broken timestamp text must not leak into the caption")
def test_drawtext_semicolon_and_quote_render_as_inert_literal_text(self):
- """escape_drawtext() (shared by overlay.py --text, graphics.py/overlay.py's --font
- fallback, and grid.py's filename-derived labels) had two more gaps beyond the comma/colon/
- bracket class fixed in 0.15.2/0.15.3: (1) an unescaped ';' -- ffmpeg's graph parser splits
- a filterchain there exactly like an unescaped ',' does, confirmed with the minimal repro
- `--text "a'b;c"` crashing real ffmpeg with "No such filter: 'c...'" on the unpatched code;
- (2) the quote character itself has no backslash escape that survives every call shape --
- both `\\'` and the POSIX `'\\''` close-insert-reopen trick corrupt a -filter_complex chain
- that uses explicit [label] pads (confirmed by rendering: trailing option text like
- "fontfile=...:fontsize=..." leaks into the picture as literal burnt-in text instead of
- being parsed as options), even though the same escape works fine in a simple -vf chain.
- Render a text containing both a quote and a semicolon and confirm it appears verbatim
- (minus the dropped quote) with nothing named after it leaking into the frame."""
+ """Before 1.15 this text went inline into the filtergraph, where two characters had no
+ escape that survives every call shape: ';' splits a filterchain exactly like ',', and the
+ quote corrupts a -filter_complex chain that uses explicit [label] pads (trailing option
+ text leaked into the picture as burnt-in literal text). The old fix dropped the quote and
+ escaped the semicolon. 1.15 removes the whole class: the drawn text is handed to drawtext
+ as `textfile=<path>:expansion=none`, so the graph parser never sees it -- the quote is now
+ KEPT, and nothing can leak into the options. Assert the new route, not the old escape."""
out = OUT / "semicolon_quote.mp4"
- proc = script("overlay.py", self.src, "--text", "a'b;c", "-o", out)
- # The quote is dropped (a'b -> ab) and the semicolon escaped, then the value must close
- # cleanly right where the template's own quote closes it -- ":fontsize=" must follow
- # immediately, not somewhere downstream after leaked option text.
- self.assertIn("ab\\;c'\\'':fontsize", proc.stderr, "value must render as 'ab;c' and close cleanly into :fontsize=, no leakage")
+ doc = json.loads(script("overlay.py", self.src, "--text", "a'b;c", "-o", out, "--json").stdout)
+ graph = " ".join(doc["commands"])
+ m = re.search(r"textfile=(\S+?\.txt)", graph.replace("\\", ""))
+ self.assertTrue(m, "drawn text must go through textfile=, not inline: " + graph)
+ self.assertIn("expansion=none", graph)
+ self.assertNotIn("a\\;c", graph, "no inline escaped copy of the text may remain in the graph")
+ # The textfile lives in a private 0700 per-run directory and is removed when the run
+ # ends -- assert both, since a world-shared, never-cleaned /tmp directory was the 1.15.0
+ # shape. The frame ink below is what proves the characters reached the picture.
+ self.assertTrue(Path(m.group(1)).parent.name.startswith("ffmpeg-skill-text-"),
+ "drawn text must go in a private per-run directory: " + m.group(1))
+ self.assertFalse(Path(m.group(1)).exists(),
+ "the drawn-text temp file must not outlive the run")
+ self.assertEqual(doc["status"], "completed")
frame = OUT / "semicolon_quote_frame.png"
sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", out, "-vframes", "1", frame)
self.assertTrue(frame.exists())
@@ -4008,7 +4014,11 @@ class FFmpegSkillTests(unittest.TestCase):
def commands(tool, *argv):
doc = json.loads(script(tool, target, *argv, "--dry-run", "--json").stdout)
- return [c for c in doc["commands"] if "ffmpeg" in c or "drawtext" in c]
+ # drawn text goes through a private PER-RUN temp directory (1.15: mkdtemp, 0700,
+ # removed at exit), so its random name differs between two processes by design and
+ # is not part of what "the same destination" means.
+ return [re.sub(r"ffmpeg-skill-text-[^/]+/", "ffmpeg-skill-text/", c)
+ for c in doc["commands"] if "ffmpeg" in c or "drawtext" in c]
# the alias is the point here, not compliance: a 3 s crop can miss the fps row on some
# ffmpeg builds (7.1 reports the speed-changed rate differently), and check.py names
@@ -4913,6 +4923,642 @@ class DemoGalleryTests(unittest.TestCase):
f"demo) or, if it only ever prints a table, add it to INSPECTION")
+# ---------------------------------------------------------------------------- emoji + shaping (1.15)
+def _emoji_assets(names=("1f389", "1f44d", "1f1ef-1f1f5", "1f469-200d-1f4bb")):
+ """A directory of placeholder emoji PNGs drawn by ffmpeg at run time.
+
+ The package ships no emoji art on purpose: Twemoji is CC-BY 4.0 and Noto Emoji OFL/Apache-2.0,
+ and a test does not need to redistribute either. What --emoji-assets actually requires is the
+ NAMING (lowercase hex code points joined by '-'), which a coloured square proves exactly as
+ well as a real glyph.
+ """
+ d = OUT / "emoji_assets"
+ d.mkdir(parents=True, exist_ok=True)
+ for i, name in enumerate(names):
+ path = d / (name + ".png")
+ if not path.exists():
+ colour = ("orange", "gold", "tomato", "limegreen")[i % 4]
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "color=c=%s:s=72x72:d=0.04" % colour, "-vf", "format=rgba",
+ "-frames:v", "1", str(path))
+ return d
+
+
+class EmojiTests(unittest.TestCase):
+ """1.15: emoji are detected as clusters, measured at a full em, reserved in the ASS and
+ composited as PNGs -- and every degraded path says so instead of lying."""
+
+ @classmethod
+ def setUpClass(cls):
+ if not shutil.which("ffmpeg"):
+ if os.environ.get("CI"):
+ raise AssertionError("ffmpeg not on PATH -- in CI this is a broken install step")
+ raise unittest.SkipTest("ffmpeg not on PATH")
+ OUT.mkdir(parents=True, exist_ok=True)
+ cls.src = OUT / "emoji_src.mp4"
+ if not cls.src.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc2=size=640x360:rate=25",
+ "-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000",
+ "-t", "4", "-c:v", "libx264", "-preset", "veryfast", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", str(cls.src))
+ cls.cues = OUT / "cues_emoji.txt"
+ cls.cues.write_text("0:00-0:02 Hello \U0001F389 world\n0:02-0:04 Nice \U0001F44D job\n",
+ encoding="utf-8")
+
+ def test_emoji_cluster_detection_keeps_zwj_and_skin_tone_sequences_together(self):
+ from _common import emoji_clusters, emoji_codepoint_name, has_emoji
+ text = "\u3084\u3063\u305f \U0001F389 \U0001F469\u200d\U0001F4BB \U0001F1EF\U0001F1F5 \U0001F44D\U0001F3FD 1\ufe0f\u20e3"
+ names = [emoji_codepoint_name(cl) for _i, cl in emoji_clusters(text)]
+ self.assertEqual(names, ["1f389", "1f469-200d-1f4bb", "1f1ef-1f1f5", "1f44d-1f3fd", "31-fe0f-20e3"])
+ self.assertTrue(has_emoji(text))
+ self.assertFalse(has_emoji("plain ascii"))
+
+ def test_emoji_is_orthogonal_to_the_writing_system(self):
+ """A cue can be Japanese AND emoji: adding an "emoji" script would corrupt the font
+ resolution for the rest of the line."""
+ self.assertEqual(detect_script("\u3084\u3063\u305f \U0001F389"), "ja")
+ self.assertEqual(detect_script("Hello \U0001F389"), "latin")
+
+ def test_caption_wrap_counts_an_emoji_as_a_full_width_atom(self):
+ from _common import text_width_em
+ import importlib, sys as _s
+ _s.path.insert(0, str(SCRIPTS))
+ caption = importlib.import_module("caption")
+ self.assertAlmostEqual(text_width_em("\U0001F389"), 1.0, places=6)
+ self.assertAlmostEqual(text_width_em("\U0001F389", 1.5), 1.5, places=6)
+ # a wrap never breaks inside a cluster
+ lines = caption.wrap_text("aaa \U0001F469\u200d\U0001F4BB bbb", 3.0)
+ self.assertIn("\U0001F469\u200d\U0001F4BB", lines)
+ for line in lines:
+ self.assertNotEqual(line, "\U0001F469")
+
+ def _plan(self, *extra):
+ out = OUT / "emoji_out.mp4"
+ proc = script("caption.py", self.src, "--text", self.cues, "-o", out,
+ "--dry-run", "--json", *extra)
+ return json.loads(proc.stdout)
+
+ def test_caption_emoji_overlay_places_one_png_per_cluster_inside_the_safe_area(self):
+ doc = self._plan("--emoji-assets", _emoji_assets())
+ self.assertEqual(doc["emoji"]["mode"], "png")
+ self.assertEqual(doc["emoji"]["overlays"], 2)
+ graph = [c for c in doc["commands"] if "overlay=" in c][-1]
+ overlays = re.findall(r"overlay=x=(\d+):y=(\d+):enable=\D*?between\(t,([\d.]+),([\d.]+)\)", graph)
+ self.assertEqual(len(overlays), 2, graph)
+ for x, y, start, end in overlays:
+ x, y = int(x), int(y)
+ self.assertTrue(0.05 * 640 <= x <= 0.95 * 640, f"x={x} outside the safe area")
+ self.assertTrue(0.05 * 360 <= y <= 0.95 * 360, f"y={y} outside the safe area")
+ self.assertLess(float(start), float(end))
+ self.assertEqual([(o[2], o[3]) for o in overlays], [("0.000", "2.000"), ("2.000", "4.000")])
+
+ def test_caption_emoji_placeholder_reserves_the_gap_in_the_generated_ass(self):
+ out = OUT / "emoji_ass.mp4"
+ script("caption.py", self.src, "--text", self.cues, "-o", out,
+ "--emoji-assets", _emoji_assets())
+ ass = (OUT / "emoji_ass.ass").read_text(encoding="utf-8-sig")
+ dialogue = [l for l in ass.splitlines() if l.startswith("Dialogue:")]
+ self.assertTrue(dialogue)
+ for line in dialogue:
+ self.assertIn("\\alpha&HFF&", line)
+ self.assertNotIn("\U0001F389", line, "the raw code point must not reach the drawn text")
+ self.assertNotIn("\U0001F44D", line)
+
+ def test_the_emoji_placeholder_reserves_exactly_the_box_it_asks_for(self):
+ """The reservation is measured, not assumed: U+2588 is 0.66-0.83 em depending on the face,
+ so the gap is reserved with alpha-hidden \\fsp spacing, which is exact in every face."""
+ from _ass_overlay import emoji_placeholder
+ families = ["FreeSans", "DejaVu Sans", "IPAPGothic", "WenQuanYi Zen Hei", "Loma"]
+ tested = 0
+ for family in families:
+ if not _family_installed(family):
+ continue
+ tested += 1
+ base = _ass_advance(family, "|")
+ with_gap = _ass_advance(family, emoji_placeholder(60) + "|")
+ if base is None or with_gap is None:
+ self.skipTest("libass render probe produced no ink here")
+ self.assertAlmostEqual(with_gap - base, 60, delta=2,
+ msg=f"{family}: reserved {with_gap - base}px, asked for 60")
+ if not tested:
+ self.skipTest("none of the measured families is installed here")
+
+ def test_caption_emoji_without_assets_is_a_warning_not_a_failure(self):
+ from _common import emoji_support
+ support = emoji_support(probe=True)
+ if support["mode"] not in ("mono", "none"):
+ self.skipTest("this machine has a colour emoji path; the degraded branch cannot be exercised")
+ out = OUT / "emoji_mono.mp4"
+ proc = script("caption.py", self.src, "--text", self.cues, "-o", out, "--json")
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["status"], "completed")
+ self.assertEqual(doc["emoji"]["mode"], support["mode"])
+ if support["mode"] == "mono":
+ self.assertIn("monochrome", proc.stderr + proc.stdout)
+ self.assertTrue(any("monochrome" in n for n in doc.get("notes", [])), doc.get("notes"))
+
+ def test_emoji_assets_directory_that_does_not_exist_is_a_failed_job(self):
+ out = OUT / "emoji_missing.mp4"
+ proc = script("caption.py", self.src, "--text", self.cues, "-o", out, "--json",
+ "--emoji-assets", str(OUT / "no_such_emoji_dir"), expect_fail=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("twemoji", doc["error"]["message"].lower())
+ self.assertIn("no network", doc["error"]["message"].lower())
+
+ def test_the_skill_never_fetches_an_emoji_asset(self):
+ """Mirror of the existing no-network assertions: nothing on the emoji path may import a
+ network module, and no tool offers a download flag for one."""
+ import _common
+ source = Path(_common.__file__).read_text(encoding="utf-8")
+ for banned in ("urllib", "http.client", "requests", "socket."):
+ self.assertNotIn(banned, source, f"{banned} is reachable from the emoji path")
+ for tool in ("caption.py", "graphics.py", "overlay.py"):
+ helptext = script(tool, "--help").stdout.lower()
+ self.assertNotIn("--emoji-download", helptext)
+ self.assertNotIn("download", helptext.split("--emoji")[-1][:400])
+
+ def test_caption_emoji_overlay_is_capped(self):
+ many = OUT / "cues_many_emoji.txt"
+ many.write_text("".join("0:0%d-0:0%d %s\n" % (i, i + 1, "\U0001F389" * 9)
+ for i in range(0, 8)), encoding="utf-8")
+ out = OUT / "emoji_capped.mp4"
+ proc = script("caption.py", self.src, "--text", many, "-o", out, "--json", "--dry-run",
+ "--emoji-assets", _emoji_assets(), "--emoji-max", "4", expect_fail=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("--emoji-max", doc["error"]["message"])
+
+ def test_graphics_all_emoji_title_without_a_glyph_is_a_failed_job(self):
+ out = OUT / "emoji_title.mp4"
+ proc = script("graphics.py", self.src, "--template", "title", "--title", "\U0001F389",
+ "--start", "0", "--end", "2", "--emoji", "none", "-o", out, "--json",
+ expect_fail=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("blank", doc["error"]["message"])
+
+ def test_overlay_text_refuses_the_png_emoji_route_and_names_the_tools_that_have_one(self):
+ out = OUT / "emoji_overlay.mp4"
+ proc = script("overlay.py", self.src, "--text", "Ship it \U0001F680", "--emoji", "png",
+ "-o", out, "--json", expect_fail=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("caption.py", doc["error"]["message"])
+ self.assertIn("graphics.py", doc["error"]["message"])
+
+
+ # ---------------------------------------------------------------- 1.15.0 fix pass
+ def test_emoji_cluster_never_starts_at_a_joiner_a_selector_or_a_plain_letter(self):
+ """U+200D ZWJ and U+200C ZWNJ are ordinary Indic/Persian orthography, not emoji: क्‍ष is
+ ka + virama + ZWJ + ssa. Treating a joiner (or a VS16, or a skin-tone modifier) as a
+ cluster TAIL that glues itself to whatever stands before it turned a Hindi conjunct into
+ "an emoji" -- reported as such, warned about as monochrome, and on the PNG route replaced
+ by an invisible gap, i.e. a different word. A cluster may only START at an emoji base."""
+ from _common import emoji_clusters, emoji_codepoint_name, has_emoji
+
+ def names(text):
+ return [emoji_codepoint_name(cl) for _i, cl in emoji_clusters(text)]
+
+ self.assertEqual(names("\u0915\u094d\u200d\u0937"), [], "a Hindi conjunct is not an emoji")
+ self.assertFalse(has_emoji("\u0905\u200c\u092c"), "a ZWNJ is not an emoji either")
+ self.assertEqual(names("abc\u200ddef"), [])
+ self.assertEqual(names("A\ufe0f"), [], "a VS16 after a letter does not make it an emoji")
+ self.assertEqual(names("\u00a9 \u2122 0123456789"), [], "(c), (tm) and digits are not emoji")
+ # ... and every real cluster still survives whole
+ self.assertEqual(names("\U0001F468\u200d\U0001F469\u200d\U0001F467"),
+ ["1f468-200d-1f469-200d-1f467"])
+ self.assertEqual(names("\U0001F3F3\ufe0f\u200d\U0001F308"), ["1f3f3-fe0f-200d-1f308"])
+ self.assertEqual(names("1\ufe0f\u20e3"), ["31-fe0f-20e3"])
+ self.assertEqual(names("\U0001F1EF\U0001F1F5"), ["1f1ef-1f1f5"])
+
+ def test_vs15_asks_for_the_character_not_the_picture(self):
+ """U+FE0E is the TEXT presentation selector: the author explicitly asked for the glyph,
+ so the cluster must not be routed to the PNG overlay."""
+ from _common import emoji_clusters
+ self.assertEqual(emoji_clusters("\u2764\ufe0e"), [], "VS15 means: draw the character")
+ self.assertTrue(emoji_clusters("\u2764\ufe0f"), "VS16 still means: draw the emoji")
+
+ def test_caption_emoji_png_is_on_screen_for_the_whole_cue_and_only_that_cue(self):
+ """The one assertion the 1.15.0 emoji tests were all missing: a rendered frame from the
+ MIDDLE of the cue. A PNG input is a single frame at pts 0, so `eof_action=pass` (which
+ switches off overlay's "hold the last frame of the secondary input") composited every
+ emoji on frame 0 and nowhere else -- the demo GIF shipped an empty reserved gap."""
+ cues = OUT / "cues_one_emoji.txt"
+ cues.write_text("0:00-0:02 Hello \U0001F389 world\n", encoding="utf-8")
+ out = OUT / "emoji_midcue.mp4"
+ doc = json.loads(script("caption.py", self.src, "--text", cues, "-o", out, "--json",
+ "--dry-run", "--emoji-assets", _emoji_assets()).stdout)
+ graph = [c for c in doc["commands"] if "overlay=" in c][-1]
+ m = re.search(r"overlay=x=(\d+):y=(\d+)", graph)
+ self.assertTrue(m, graph)
+ x, y = int(m.group(1)) + 4, int(m.group(2)) + 4
+ script("caption.py", self.src, "--text", cues, "-o", out,
+ "--emoji-assets", _emoji_assets())
+ orange = (255, 165, 0) # _emoji_assets() draws 1f389 as a plain orange square
+
+ def near(pix):
+ return sum(abs(a - b) for a, b in zip(pix, orange)) < 60
+
+ self.assertTrue(near(_pixel_at(out, 1.0, x, y)),
+ "the emoji is gone by the middle of its own cue: %r" % (_pixel_at(out, 1.0, x, y),))
+ self.assertTrue(near(_pixel_at(out, 1.8, x, y)), "the emoji left before its cue ended")
+ self.assertFalse(near(_pixel_at(out, 3.0, x, y)),
+ "the emoji is still on screen a second after its cue ended")
+
+ def test_caption_rtl_emoji_is_placed_by_the_rendered_order_not_the_logical_prefix(self):
+ """libass lays an RTL line out right-to-left, so the LOGICAL prefix of a cluster occupies
+ the RIGHT end of the rendered line. Measuring from the left put the PNG on top of the
+ Arabic text, a prefix-width away from the gap libass actually reserved."""
+ from _common import script_font_status
+ cues = OUT / "cues_rtl.txt"
+ # the emoji is FIRST in logical order, so it must be drawn at the RIGHT end of the line
+ cues.write_text("0:00-0:02 \U0001F389 \u0645\u0631\u062d\u0628\u0627 \u0628\u0627\u0644\u0639\u0627\u0644\u0645\n",
+ encoding="utf-8")
+ out = OUT / "emoji_rtl.mp4"
+ doc = json.loads(script("caption.py", self.src, "--text", cues, "-o", out, "--json",
+ "--dry-run", "--emoji-assets", _emoji_assets()).stdout)
+ graph = [c for c in doc["commands"] if "overlay=" in c][-1]
+ m = re.search(r"overlay=x=(\d+):y=(\d+)", graph)
+ self.assertTrue(m, graph)
+ x, y = int(m.group(1)), int(m.group(2))
+ self.assertGreater(x, 320, "a logically-first cluster in an RTL line belongs on the right "
+ "half of a centred line, not the left: x=%d" % x)
+ if script_font_status("ar") != "available":
+ self.skipTest("no font on this machine covers Arabic; tests never install fonts")
+ # and the box must land where libass drew nothing -- the reserved gap, not over a glyph
+ script("caption.py", self.src, "--text", cues, "-o", out, "--emoji-assets", _emoji_assets())
+ box = doc["emoji"].get("box_px") or 24
+ ink = _ass_ink_columns(OUT / "emoji_rtl.ass", 640, 360)
+ if not ink:
+ # the probe renders the sidecar with a bare `ass=` filter; without fontconfig (Windows,
+ # a static build) libass may resolve no face for Arabic and draw nothing, which says
+ # nothing about placement -- the caption itself goes through fontsdir handling
+ self.skipTest("the ASS probe drew no ink on this build (libass found no Arabic face)")
+ covered = [c for c in range(x + 2, x + box - 2) if c in ink]
+ self.assertEqual(covered, [], "the PNG would be composited over drawn text at columns %r" % covered)
+
+ def test_caption_emoji_max_zero_means_no_overlays_at_all(self):
+ """`int(args.emoji_max or 60)` swallowed the one value a caller would use to say 'none'."""
+ out = OUT / "emoji_max0.mp4"
+ doc = json.loads(script("caption.py", self.src, "--text", self.cues, "-o", out, "--json",
+ "--dry-run", "--emoji-assets", _emoji_assets(), "--emoji-max", "0",
+ expect_fail=True).stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("limit 0", doc["error"]["message"])
+
+ def test_caption_animate_fades_the_emoji_with_the_text(self):
+ """--animate fade gives the TEXT a \\fad; the PNG used to pop in against a fading line."""
+ cues = OUT / "cues_fade_emoji.txt"
+ cues.write_text("0:00-0:02 Hello \U0001F389 world\n", encoding="utf-8")
+ out = OUT / "emoji_fade.mp4"
+ doc = json.loads(script("caption.py", self.src, "--text", cues, "-o", out, "--json",
+ "--dry-run", "--animate", "fade",
+ "--emoji-assets", _emoji_assets()).stdout)
+ graph = [c for c in doc["commands"] if "overlay=" in c][-1]
+ self.assertIn("fade=t=in", graph)
+ self.assertIn("fade=t=out", graph)
+ self.assertIn("alpha=1", graph)
+ m = re.search(r"overlay=x=(\d+):y=(\d+)", graph)
+ x, y = int(m.group(1)) + 4, int(m.group(2)) + 4
+ script("caption.py", self.src, "--text", cues, "-o", out, "--animate", "fade",
+ "--emoji-assets", _emoji_assets())
+ orange = (255, 165, 0)
+
+ def dist(t):
+ return sum(abs(a - b) for a, b in zip(_pixel_at(out, t, x, y), orange))
+
+ self.assertLess(dist(1.0), 60, "the emoji never reached full opacity")
+ self.assertGreater(dist(0.02), dist(1.0), "the emoji did not fade in with the text")
+
+ def test_graphics_drawtext_route_never_reports_monochrome_emoji(self):
+ """drawtext loads ONE font file and has no fallback chain, so "whatever glyph the text
+ font has" is an empty box. Reporting `mode: mono` from that route is a claim the frame
+ does not keep: the run must route to libass (which does have a fallback chain) or strip."""
+ out = OUT / "emoji_gfx_mono.mp4"
+ doc = json.loads(script("graphics.py", self.src, "--template", "title",
+ "--title", "Ship it \U0001F389 now", "--start", "0", "--end", "2",
+ "-o", out, "--json", "--dry-run").stdout)
+ if doc.get("emoji", {}).get("mode") == "mono":
+ self.assertEqual(doc["text_renderer"], "ass",
+ "mode mono on the drawtext route draws tofu, not a glyph")
+ pinned = json.loads(script("graphics.py", self.src, "--template", "title",
+ "--title", "Ship it \U0001F389 now", "--start", "0", "--end", "2",
+ "--text-render", "drawtext", "-o", out, "--json", "--dry-run").stdout)
+ self.assertEqual(pinned["text_renderer"], "drawtext")
+ self.assertNotEqual(pinned.get("emoji", {}).get("mode"), "mono",
+ "a pinned drawtext run must not claim monochrome either")
+
+ def test_graphics_emoji_none_actually_strips_the_cluster(self):
+ """`--emoji none` only guarded the all-emoji case: the cluster stayed in the drawn text
+ and drawtext drew the same empty box it would have drawn anyway."""
+ with_emoji = OUT / "emoji_gfx_none.mp4"
+ without = OUT / "emoji_gfx_plain.mp4"
+ script("graphics.py", self.src, "--template", "title", "--title", "Ship it \U0001F389 now",
+ "--start", "0", "--end", "3", "--emoji", "none", "-o", with_emoji)
+ script("graphics.py", self.src, "--template", "title", "--title", "Ship it now",
+ "--start", "0", "--end", "3", "-o", without)
+ self.assertAlmostEqual(_frame_ink(with_emoji), _frame_ink(without),
+ delta=max(1, _frame_ink(without) // 500),
+ msg="--emoji none must draw the same frame as text with no emoji in it")
+
+ def test_drawn_text_temp_files_are_private_and_never_outlive_the_run(self):
+ """1.15.0 wrote every drawn label into a world-shared, predictable, never-cleaned
+ /tmp/ffmpeg-skill-text -- including under --dry-run and on the ASS route, which never
+ reads them."""
+ import glob
+ import tempfile as _tf
+ out = OUT / "textfile_life.mp4"
+ doc = json.loads(script("overlay.py", self.src, "--text", "Label", "-o", out,
+ "--json", "--dry-run").stdout)
+ m = re.search(r"textfile=(\S+?\.txt)", " ".join(doc["commands"]).replace("\\", ""))
+ self.assertTrue(m, "the plan must still name the path it would use")
+ self.assertFalse(os.path.exists(m.group(1)), "a dry run wrote a file")
+ self.assertFalse(os.path.isdir(os.path.dirname(m.group(1))), "a dry run created a directory")
+ self.assertEqual(glob.glob(os.path.join(_tf.gettempdir(), "ffmpeg-skill-text-*")), [],
+ "a drawn-text directory was left behind")
+ self.assertFalse(os.path.exists(os.path.join(_tf.gettempdir(), "ffmpeg-skill-text")),
+ "the shared world-writable directory must not be created at all")
+
+ def test_the_ass_route_writes_no_drawtext_textfile(self):
+ from _common import script_font_status
+ if script_font_status("hi") != "available":
+ self.skipTest("no font on this machine covers Devanagari; tests never install fonts")
+ import glob
+ import tempfile as _tf
+ out = OUT / "textfile_ass_route.mp4"
+ script("graphics.py", self.src, "--template", "title", "--title", "\u0915\u093f\u0924\u093e\u092c",
+ "--start", "0", "--end", "2", "-o", out)
+ self.assertEqual(glob.glob(os.path.join(_tf.gettempdir(), "ffmpeg-skill-text-*")), [],
+ "the ASS route wrote (and kept) a drawtext textfile it never reads")
+
+ def test_the_ass_route_keeps_braces_and_backslashes(self):
+ """The same title through drawtext and through libass must produce the same characters:
+ 1.15.0 deleted `{`, `}` and `\\` on the ASS route while drawtext kept them."""
+ from _ass_overlay import ass_escape
+ self.assertEqual(ass_escape("A {b} c \\ d"), "A \\{b\\} c \\ d")
+ out = OUT / "ass_braces.mp4"
+ doc = json.loads(script("graphics.py", self.src, "--template", "title",
+ "--title", "A {b} c \\ d", "--start", "0", "--end", "2",
+ "--text-render", "ass", "-o", out, "--json").stdout)
+ ass = Path(doc["ass"]).read_text(encoding="utf-8-sig")
+ dialogue = next(l for l in ass.splitlines() if l.startswith("Dialogue:"))
+ self.assertIn("A \\{b\\} c \\ d", dialogue)
+
+
+def _pixel_at(video, t, x, y):
+ """The RGB triple at (x, y) of the frame at `t` seconds of `video`."""
+ proc = subprocess.run(["ffmpeg", "-v", "error", "-ss", str(t), "-i", str(video), "-frames:v", "1",
+ "-f", "rawvideo", "-pix_fmt", "rgb24", "-"], stdout=subprocess.PIPE)
+ w = int(subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
+ "stream=width", "-of", "csv=p=0", str(video)],
+ stdout=subprocess.PIPE, text=True).stdout.strip())
+ off = (y * w + x) * 3
+ return tuple(proc.stdout[off:off + 3])
+
+
+def _ass_ink_columns(ass_path, w, h, threshold=60):
+ """The set of x columns that carry ink when `ass_path` is rendered over black at w x h."""
+ proc = subprocess.run(
+ ["ffmpeg", "-v", "error", "-f", "lavfi", "-i", "color=c=black:s=%dx%d:d=0.04" % (w, h),
+ "-vf", "ass=%s" % str(ass_path).replace("\\", "/"), "-frames:v", "1",
+ "-f", "rawvideo", "-pix_fmt", "gray", "-"], stdout=subprocess.PIPE)
+ data = proc.stdout
+ if len(data) < w * h:
+ return set()
+ return {x for x in range(w) if any(data[y * w + x] > threshold for y in range(h))}
+
+
+def _family_installed(family):
+ if not shutil.which("fc-list"):
+ return False
+ proc = subprocess.run(["fc-list", ":family=%s" % family, "file"], stdout=subprocess.PIPE, text=True)
+ return bool(proc.stdout.strip())
+
+
+def _ass_advance(family, body):
+ """x of the rightmost lit pixel of a left-anchored one-line ASS render, or None."""
+ W, H, FS = 900, 140, 60
+ path = OUT / "advance.ass"
+ path.write_text(
+ "[Script Info]\nScriptType: v4.00+\nPlayResX: %d\nPlayResY: %d\nWrapStyle: 2\n\n"
+ "[V4+ Styles]\nFormat: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, "
+ "OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, "
+ "Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding\n"
+ "Style: D,%s,%d,&H00FFFFFF,&H00FFFFFF,&H00000000,&H00000000,0,0,0,0,100,100,0,0,1,0,0,7,0,0,0,1\n\n"
+ "[Events]\nFormat: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text\n"
+ "Dialogue: 0,0:00:00.00,0:00:05.00,D,,0,0,0,,{\\pos(0,0)}%s\n" % (W, H, family, FS, body),
+ encoding="utf-8-sig")
+ proc = subprocess.run(
+ ["ffmpeg", "-v", "error", "-f", "lavfi", "-i", "color=c=black:s=%dx%d:d=0.04" % (W, H),
+ "-vf", "ass=%s" % str(path).replace("\\", "/"), "-frames:v", "1",
+ "-f", "rawvideo", "-pix_fmt", "gray", "-"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ data = proc.stdout
+ if proc.returncode != 0 or len(data) < W * H:
+ return None
+ for x in range(W - 1, -1, -1):
+ if any(data[y * W + x] > 60 for y in range(H)):
+ return x
+ return None
+
+
+class ShapingTests(unittest.TestCase):
+ """1.15: drawtext does bidi and Arabic joining on a fribidi build but never reorders or
+ re-clusters. graphics.py routes what it cannot shape through libass; overlay.py refuses."""
+
+ @classmethod
+ def setUpClass(cls):
+ if not shutil.which("ffmpeg"):
+ if os.environ.get("CI"):
+ raise AssertionError("ffmpeg not on PATH -- in CI this is a broken install step")
+ raise unittest.SkipTest("ffmpeg not on PATH")
+ OUT.mkdir(parents=True, exist_ok=True)
+ cls.src = OUT / "emoji_src.mp4"
+ if not cls.src.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc2=size=640x360:rate=25", "-t", "4",
+ "-c:v", "libx264", "-preset", "veryfast", "-pix_fmt", "yuv420p", str(cls.src))
+
+ def _skip_without(self, script_code):
+ from _common import script_font_status
+ if script_font_status(script_code) != "available":
+ raise unittest.SkipTest("no font on this machine covers %r; tests never install fonts"
+ % script_code)
+
+ def test_graphics_devanagari_renders_through_libass_not_drawtext(self):
+ self._skip_without("hi")
+ out = OUT / "shape_hi.mp4"
+ doc = json.loads(script("graphics.py", self.src, "--template", "lower-third",
+ "--name", "\u092a\u094d\u0930\u093f\u092f\u093e \u0936\u0930\u094d\u092e\u093e",
+ "--title", "\u0928\u093f\u0930\u094d\u0926\u0947\u0936\u0915",
+ "--start", "0", "--end", "3", "-o", out, "--dry-run", "--json").stdout)
+ self.assertEqual(doc["text_renderer"], "ass")
+ self.assertEqual(doc["script"], "hi")
+ graph = " ".join(doc["commands"])
+ self.assertIn("ass=", graph)
+ self.assertNotIn("drawtext=", graph, "the text must not also go through drawtext")
+
+ def test_graphics_latin_still_renders_through_drawtext(self):
+ out = OUT / "shape_latin.mp4"
+ doc = json.loads(script("graphics.py", self.src, "--template", "title",
+ "--title", "Episode 12", "--start", "0", "--end", "3",
+ "-o", out, "--dry-run", "--json").stdout)
+ self.assertEqual(doc["text_renderer"], "drawtext")
+ self.assertNotIn("ass", doc)
+ self.assertIn("drawtext=", " ".join(doc["commands"]))
+
+ def test_graphics_drawtext_forced_with_an_indic_script_is_refused_naming_the_script(self):
+ self._skip_without("hi")
+ out = OUT / "shape_refuse.mp4"
+ doc = json.loads(script("graphics.py", self.src, "--template", "title",
+ "--title", "\u0915\u093f\u0924\u093e\u092c", "--start", "0", "--end", "3",
+ "--text-render", "drawtext", "-o", out, "--json",
+ expect_fail=True).stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("Devanagari", doc["error"]["message"])
+ self.assertIn("harfbuzz", doc["error"]["message"])
+
+ def test_graphics_arabic_is_not_rerouted_on_a_fribidi_build(self):
+ from _common import drawtext_shaping
+ if not drawtext_shaping()["fribidi"]:
+ raise unittest.SkipTest("this ffmpeg has no fribidi: Arabic legitimately needs the ASS route")
+ self._skip_without("ar")
+ out = OUT / "shape_ar.mp4"
+ doc = json.loads(script("graphics.py", self.src, "--template", "title",
+ "--title", "\u0645\u0631\u062d\u0628\u0627 \u0628\u0627\u0644\u0639\u0627\u0644\u0645",
+ "--start", "0", "--end", "3", "-o", out, "--dry-run", "--json").stdout)
+ self.assertEqual(doc["text_renderer"], "drawtext")
+
+ def test_graphics_font_file_override_survives_the_ass_route(self):
+ self._skip_without("hi")
+ font = font_for_script("hi")
+ out = OUT / "shape_fontfile.mp4"
+ proc = script("graphics.py", self.src, "--template", "title", "--title",
+ "\u0915\u093f\u0924\u093e\u092c", "--start", "0", "--end", "3",
+ "--font-file", font, "-o", out, "--json")
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["text_renderer"], "ass")
+ self.assertIn("fontsdir=", " ".join(doc["commands"]))
+ from _common import font_family_of_file
+ family = font_family_of_file(font)
+ ass = Path(doc["ass"]).read_text(encoding="utf-8-sig")
+ self.assertIn(family, ass, "the --font-file's own family must reach the ASS Style line")
+
+ def test_overlay_text_refuses_a_shaping_script_naming_the_tools_that_render_it(self):
+ out = OUT / "shape_overlay.mp4"
+ doc = json.loads(script("overlay.py", self.src, "--text", "\u0915\u093f\u0924\u093e\u092c",
+ "-o", out, "--json", expect_fail=True).stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("caption.py", doc["error"]["message"])
+ self.assertIn("graphics.py", doc["error"]["message"])
+
+
+class ApostropheAndPercentTests(unittest.TestCase):
+ """1.15: `'` and `%` reach the picture. They used to be dropped outright by escape_drawtext."""
+
+ @classmethod
+ def setUpClass(cls):
+ if not shutil.which("ffmpeg"):
+ if os.environ.get("CI"):
+ raise AssertionError("ffmpeg not on PATH -- in CI this is a broken install step")
+ raise unittest.SkipTest("ffmpeg not on PATH")
+ OUT.mkdir(parents=True, exist_ok=True)
+ cls.src = OUT / "emoji_src.mp4"
+ if not cls.src.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc2=size=640x360:rate=25", "-t", "4",
+ "-c:v", "libx264", "-preset", "veryfast", "-pix_fmt", "yuv420p", str(cls.src))
+
+ def test_drawtext_keeps_an_apostrophe_and_a_percent_sign(self):
+ out = OUT / "pct_overlay.mp4"
+ doc = json.loads(script("overlay.py", self.src, "--text", "it's 100% done",
+ "-o", out, "--json").stdout)
+ graph = " ".join(doc["commands"])
+ m = re.search(r"textfile=(\S+?\.txt)", graph.replace("\\", ""))
+ self.assertTrue(m, graph)
+ self.assertIn("expansion=none", graph)
+ self.assertTrue(Path(m.group(1)).parent.name.startswith("ffmpeg-skill-text-"),
+ "drawn text must go in a private per-run directory: " + m.group(1))
+ self.assertFalse(Path(m.group(1)).exists(), "the drawn-text temp file must not outlive the run")
+ self.assertEqual(doc["status"], "completed")
+ # the drawn ink differs from the same render with the two characters stripped
+ stripped = OUT / "pct_overlay_stripped.mp4"
+ script("overlay.py", self.src, "--text", "its 100 done", "-o", stripped)
+ self.assertNotEqual(_frame_ink(out), _frame_ink(stripped),
+ "the apostrophe and the percent sign left no ink on the frame")
+
+ def test_caption_keeps_apostrophe_and_percent_in_the_drawn_text(self):
+ cues = OUT / "cues_pct.txt"
+ cues.write_text("0:00-0:02 it's 100% done\n", encoding="utf-8")
+ out = OUT / "pct_caption.mp4"
+ script("caption.py", self.src, "--text", cues, "--animate", "fade", "-o", out)
+ srt = (OUT / "pct_caption.srt").read_text(encoding="utf-8")
+ ass = (OUT / "pct_caption.ass").read_text(encoding="utf-8-sig")
+ self.assertIn("it's 100% done", srt)
+ self.assertIn("it's 100% done", ass)
+
+
+def _frame_ink(path):
+ proc = subprocess.run(["ffmpeg", "-v", "error", "-i", str(path), "-ss", "1", "-frames:v", "1",
+ "-f", "rawvideo", "-pix_fmt", "gray", "-"], stdout=subprocess.PIPE)
+ return sum(proc.stdout)
+
+
+class WrapReadabilityTests(unittest.TestCase):
+ """1.15: no one-character orphan line, and a balanced break for spaced scripts."""
+
+ def setUp(self):
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ self.caption = importlib.import_module("caption")
+
+ def test_caption_wrap_never_leaves_a_one_character_orphan_line(self):
+ C = self.caption
+ cases = [
+ # th1's Thai cue and dl3's Japanese cue, at sizes where greedy wrapping stranded one character
+ ("\u0e1a\u0e23\u0e23\u0e17\u0e31\u0e14\u0e17\u0e35\u0e48\u0e2a\u0e2d\u0e07\u0e02\u0e2d\u0e07\u0e04\u0e33\u0e1a\u0e23\u0e23\u0e22\u0e32\u0e22 "
+ "\u0e1a\u0e23\u0e23\u0e17\u0e31\u0e14\u0e19\u0e35\u0e49\u0e15\u0e31\u0e49\u0e07\u0e40\u0e27\u0e25\u0e32\u0e43\u0e2b\u0e49\u0e2d\u0e31\u0e15\u0e42\u0e19\u0e21\u0e31\u0e15\u0e34", 8),
+ ("\u3053\u3093\u306b\u3061\u306f\u3001\u4e16\u754c 2 \u884c\u76ee\u306e\u5b57\u5e55\u3067\u3059 "
+ "\u81ea\u52d5\u3067\u30bf\u30a4\u30df\u30f3\u30b0\u304c\u6c7a\u307e\u308b\u884c", 10),
+ ("\u3053\u3093\u306b\u3061\u306f\u3001\u4e16\u754c 2 \u884c\u76ee\u306e\u5b57\u5e55\u3067\u3059 "
+ "\u81ea\u52d5\u3067\u30bf\u30a4\u30df\u30f3\u30b0\u304c\u6c7a\u307e\u308b\u884c", 29),
+ ]
+ for text, max_em in cases:
+ with self.subTest(max_em=max_em):
+ greedy = C.wrap_text(text, max_em, balance=False)
+ self.assertTrue(len(C._atoms(greedy[-1])) == 1 and C.text_width_em(greedy[-1]) < C.ORPHAN_MIN_EM,
+ "the case no longer reproduces greedily: %r" % greedy)
+ lines = C.wrap_text(text, max_em)
+ self.assertGreater(C.text_width_em(lines[-1]), C.ORPHAN_MIN_EM, lines)
+ for line in lines:
+ self.assertLessEqual(C.text_width_em(line), max_em, lines)
+ self.assertEqual("".join(lines).replace(" ", ""), "".join(greedy).replace(" ", ""))
+
+ def test_caption_wrap_prefers_a_balanced_break_for_latin(self):
+ C = self.caption
+ text = "A third line the tool times for me"
+ greedy = C.wrap_text(text, 14, balance=False)
+ balanced = C.wrap_text(text, 14)
+ self.assertEqual(len(greedy), len(balanced))
+ self.assertLessEqual(max(C.text_width_em(l) for l in balanced),
+ max(C.text_width_em(l) for l in greedy))
+ self.assertEqual(" ".join(balanced), text)
+ self.assertNotEqual(greedy, balanced, "dl1's break did not move")
+
+ def test_caption_wrap_rebalance_never_changes_the_line_count(self):
+ C = self.caption
+ samples = ["A third line the tool times for me",
+ "Hello world", "one two three four five six seven eight nine ten",
+ "\u3053\u3093\u306b\u3061\u306f\u3001\u4e16\u754c\u306e\u5b57\u5e55\u3067\u3059",
+ "Shipping day \U0001F389 for everyone here"]
+ for text in samples:
+ for max_em in range(4, 30):
+ with self.subTest(text=text[:12], max_em=max_em):
+ self.assertEqual(len(C.wrap_text(text, max_em)),
+ len(C.wrap_text(text, max_em, balance=False)))
+
+
def _families_for(script):
"""Families fontconfig lists for `script`, or [] where there is no fontconfig to ask (Windows
resolves fonts by file name, so font_for_script() answers there without an fc-list on PATH --
diff --git a/tests/test_contract.py b/tests/test_contract.py
index dd4d826..a8a1ed9 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -2657,7 +2657,7 @@ class DoctorDetectionTests(unittest.TestCase):
d = json.loads(proc.stdout)
expected = {s for s in DETECTED_SCRIPTS if s != "latin"}
self.assertEqual(set(d["fonts"]["scripts"]), expected)
- self.assertEqual(len(expected), 9, "ja zh ko ar he hi th ru el")
+ self.assertEqual(len(expected), 12, "ja zh ko ar he hi bn ta th lo ru el (bn/ta/lo added in 1.15)")
for name, entry in d["fonts"]["scripts"].items():
with self.subTest(script=name):
self.assertIn(entry["status"], ("available", "missing", "unknown"))
@@ -2682,6 +2682,78 @@ class DoctorDetectionTests(unittest.TestCase):
if entry["status"] == "missing":
self.assertIn(name, font_lines[0])
self.assertNotIn(str(entry["file"]), font_lines[0], "file paths belong in --json, not the summary")
+ # 1.15: the emoji row adds at most one clause to the same line, and no second line
+ self.assertIn("emoji", font_lines[0])
+ self.assertLessEqual(len(font_lines[0]), 200, font_lines[0])
+ self.assertNotIn("\n", font_lines[0])
+
+ def test_doctor_reports_an_emoji_row_under_fonts(self):
+ """1.15: `fonts.emoji` answers "can this machine draw an emoji, and in colour" -- a
+ question no filter or font capability asks."""
+ d = json.loads(sh(sys.executable, SCRIPTS / "_contract.py", "doctor", "--json", check=False).stdout)
+ emoji = d["fonts"]["emoji"]
+ self.assertEqual(set(emoji), {"mode", "color_font", "color_font_file", "libass_color",
+ "assets", "detail", "fix"})
+ self.assertIn(emoji["mode"], ("color", "png", "mono", "none"))
+ self.assertIn(emoji["libass_color"], (True, False, None))
+ self.assertTrue(emoji["detail"])
+ self.assertIn("--emoji-assets", emoji["fix"])
+ if emoji["color_font_file"]:
+ self.assertTrue(os.path.exists(emoji["color_font_file"]), emoji)
+ # an installed colour emoji family is never on its own a claim of colour
+ if emoji["color_font"] and emoji["libass_color"] is not True:
+ self.assertNotEqual(emoji["mode"], "color")
+
+ def test_emoji_capability_is_informational_like_gpu_encoders(self):
+ """A machine that cannot draw colour emoji is not a broken install: `fonts.emoji` must
+ never move `ok` or any tool's `usable`."""
+ d = json.loads(sh(sys.executable, SCRIPTS / "_contract.py", "doctor", "--json", check=False).stdout)
+ self.assertTrue(d["ok"] or d["missing"] or d["unknown"],
+ "ok went false with nothing missing -- the emoji row must be informational")
+ for name, tool in d["tools"].items():
+ with self.subTest(tool=name):
+ for cap in tool.get("missing", []) + tool.get("unknown", []):
+ self.assertNotIn("emoji", cap)
+ self.assertNotIn("emoji", " ".join(d["available"] + d["missing"] + d["missing_optional"] + d["unknown"]))
+
+ def test_static_skips_the_environment_detection_including_the_emoji_probe(self):
+ """`--static` is the only user-reachable way to ask for the contract without detection:
+ doctor() itself always probes (there is no `doctor --static`, and the dead `detect=`
+ parameter that pretended otherwise was removed in 1.15.0). What must hold is that
+ `--json --static` runs no detection at all -- including the ~80 ms emoji render probe."""
+ unprobed = _contract._fonts_capability(probe=False)
+ self.assertIsNone(unprobed["emoji"]["libass_color"])
+ self.assertNotEqual(unprobed["emoji"]["mode"], "color",
+ "an unprobed machine must never be reported as colour-capable")
+ out = sh(sys.executable, SCRIPTS / "_contract.py", "--json", "--static", check=False).stdout
+ doc = json.loads(out)
+ self.assertNotIn("available", doc["capabilities"], "--static ran detection")
+ self.assertNotIn("fonts", doc.get("capabilities", {}))
+
+ def test_emoji_support_answers_png_or_none_without_fontconfig(self):
+ """A static ffmpeg build usually has no fontconfig at all. That is not evidence that this
+ machine cannot draw emoji -- the PNG overlay path needs no fontconfig -- so the answer must
+ never be a claim of `color`, and its detail must name the missing fontconfig."""
+ import _common
+ _common._EMOJI_SUPPORT_CACHE.clear()
+ real_which = shutil.which
+ try:
+ _common.shutil.which = lambda name: None if name in ("fc-list", "fc-scan", "fc-match", "ffmpeg") else real_which(name)
+ support = _common.emoji_support(probe=True)
+ finally:
+ _common.shutil.which = real_which
+ _common._EMOJI_SUPPORT_CACHE.clear()
+ self.assertIn(support["mode"], ("png", "none"))
+ self.assertIsNone(support["color_font"])
+ self.assertIn("fontconfig", support["detail"])
+
+ def test_skill_md_stays_under_the_30kb_budget(self):
+ """SKILL.md is loaded into every session: the 30 000-byte budget was a convention in
+ CONTRIBUTING.md, which is how it got to 51 free bytes. Now it is a test."""
+ size = (ROOT / "SKILL.md").stat().st_size
+ self.assertLess(size, 30_000,
+ f"SKILL.md is {size} bytes, {size - 30_000} over the 30,000-byte budget -- "
+ "trim a line rather than raising the limit (see CONTRIBUTING.md)")
def test_font_fix_hint_names_the_language_and_how_to_install_one(self):
hint = _contract._capability_fix_hint("font:ko")